Mailman 3 build system abstraction PEP - Distutils-SIG

build system abstraction PEP

Robert Collins

26 Oct 2015 26 Oct '15

6:01 a.m.

Since Nathaniel seems busy, I've taken the liberty of drafting a narrow PEP based on the conversations that arose from the prior discussion. It (naturally) has my unique flavor, but builds on the work Nathaniel had put together, so I've put his name as a co-author even though he hasn't seen a word of it until now :) - all errors and mistakes are therefore mine... Current draft text in rendered form at: https://gist.github.com/rbtcollins/666c12aec869237f7cf7 I've run it past Donald and he has a number of concerns - I think we'll need to discuss them here, and possibly in another hangout, to get a path forward. Cheers, Rob -- Robert Collins Distinguished Technologist HP Converged Cloud

Show replies by date

Nathaniel Smith

26 Oct 26 Oct

6:04 a.m.

Wow, uh, that's some timing we have. I'll, uh, take a look :-) On Sun, Oct 25, 2015 at 11:01 PM, Robert Collins wrote:

...

Since Nathaniel seems busy, I've taken the liberty of drafting a narrow PEP based on the conversations that arose from the prior discussion.

It (naturally) has my unique flavor, but builds on the work Nathaniel had put together, so I've put his name as a co-author even though he hasn't seen a word of it until now :) - all errors and mistakes are therefore mine...

Current draft text in rendered form at: https://gist.github.com/rbtcollins/666c12aec869237f7cf7

I've run it past Donald and he has a number of concerns - I think we'll need to discuss them here, and possibly in another hangout, to get a path forward.

Cheers, Rob

-- Robert Collins Distinguished Technologist HP Converged Cloud

-- Nathaniel J. Smith -- http://vorpus.org

Robert Collins

6:22 a.m.

On 26 October 2015 at 19:04, Nathaniel Smith wrote:

...

Wow, uh, that's some timing we have. I'll, uh, take a look :-)

Hah! Jinx. Thanks. I'll read through yours too :) -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud

Ralf Gommers

9:32 p.m.

On Mon, Oct 26, 2015 at 7:01 AM, Robert Collins wrote:

...

Since Nathaniel seems busy, I've taken the liberty of drafting a narrow PEP based on the conversations that arose from the prior discussion.

It (naturally) has my unique flavor, but builds on the work Nathaniel had put together, so I've put his name as a co-author even though he hasn't seen a word of it until now :) - all errors and mistakes are therefore mine...

Current draft text in rendered form at: https://gist.github.com/rbtcollins/666c12aec869237f7cf7

I have the feeling that I like where this PEP is going, but it's quite difficult to read. It would be very helpful to add one or two examples. Suggestions: (1) Super simple example: for a pure Python package with one dependency and a build tool which has no dependencies itself. (2) Complex example: to build a Scipy wheel on Windows with MinGW the command is ``python setup.py config --compiler=mingw32 build --compiler=mingw32 bdist_wheel``. I tried to do (1) with flit as the build tool, the pypa.yaml should include: bootstrap-requires: - requests - docutils - requirement:python_version>="3" So you have to encode flit's dependencies in your pypa.yaml, which will break as soon as flit grows a new dependency. Or did I misunderstand? Maybe (2) simply is not possible / out of scope, but I have the feeling that there'll be a need for users to be able pass stuff (via pip or some other mechanism) to the build tool. If it is out of scope, I'd be interested to see what you think are use-cases with complex requirements that are enabled by this PEP. Cheers, Ralf

...

I've run it past Donald and he has a number of concerns - I think we'll need to discuss them here, and possibly in another hangout, to get a path forward.

Cheers, Rob

-- Robert Collins Distinguished Technologist HP Converged Cloud _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Robert Collins

11:23 p.m.

On 27 October 2015 at 10:32, Ralf Gommers wrote:

...

On Mon, Oct 26, 2015 at 7:01 AM, Robert Collins wrote:

...
Since Nathaniel seems busy, I've taken the liberty of drafting a narrow PEP based on the conversations that arose from the prior discussion.

It (naturally) has my unique flavor, but builds on the work Nathaniel had put together, so I've put his name as a co-author even though he hasn't seen a word of it until now :) - all errors and mistakes are therefore mine...

Current draft text in rendered form at: https://gist.github.com/rbtcollins/666c12aec869237f7cf7

I have the feeling that I like where this PEP is going, but it's quite difficult to read. It would be very helpful to add one or two examples.

Ok, need to make it easier to read :).

...

Suggestions:

(1) Super simple example: for a pure Python package with one dependency and a build tool which has no dependencies itself.

pypy.yaml === bootstrap-requires: - flit build-tool: flit --dump-build-description === flit --dump-build-description would output something like: === build-requires: flit --dump-build-requires dist-info: flit --dump-dist-info provided-by: flit wheel: flit wheel -d $OUTPUT_DIR === The --dump- switches in this example would be needed to be added to flit (or someone else could write an adapter). Because flit doesn't have setup-requires in flit.ini today, --dump-build-requires would just output a constant string: {"schema_version": "2.0", "build_requires": []} --dump-dist-info interrogate flit.ini and marshal the metadata into a PEP 426 JSON document and output that on stdout. flit wheel would need a -d parameter that tells it where to output the wheel(pip needs this).

...

(2) Complex example: to build a Scipy wheel on Windows with MinGW the command is ``python setup.py config --compiler=mingw32 build --compiler=mingw32 bdist_wheel``.

So in this case the build tool needs to know about the compiler stuff itself- pip doesn't know. We have a way in pip to tunnel stuff down to setuptools today; thats incompatible with dynamically building wheels on the fly for 'pip install' - so I'm not sure it needs to be reflected here. We'll need some more input on that I think. ...

...

mechanism) to the build tool. If it is out of scope, I'd be interested to see what you think are use-cases with complex requirements that are enabled by this PEP.

The PEP is aimed at enabling additional build-tools to be on parity with setuptools in *pip install* invocations that go through the wheel-autobuild-path in pip. The complex examples of passing arbitrary options to setup.py currently bypasses wheel building in pip, and so can't be tackled at all :(. But we can work on including that with some thought. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud

Ionel Cristian Mărieș

27 Oct 27 Oct

12:07 a.m.

On Tue, Oct 27, 2015 at 1:23 AM, Robert Collins wrote:

...

wheel: flit wheel -d $ OUTPUT_DIR

What's expected to happen if there are multiple files produced in $OUTPUT_DIR? Is it an error? Also, doesn't this pep overlap with PEP 426? Shouldn't these ideas be incorporated there? It seems that only the command metadata is missing from PEP 426. Another thing, I've noticed there's a "develop" command in this draft. I think this is confusing, I'd expect it to be a "install" command (and that's what it is now in setuptools). It should be renamed to something more clear, like "inplace-build" and leave the installing to pip. Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro

Robert Collins

1:06 a.m.

On 27 October 2015 at 13:07, Ionel Cristian Mărieș wrote:

...

On Tue, Oct 27, 2015 at 1:23 AM, Robert Collins wrote:

...
wheel: flit wheel -d $ OUTPUT_DIR

What's expected to happen if there are multiple files produced in $OUTPUT_DIR? Is it an error?

https://github.com/pypa/pip/blob/develop/pip/wheel.py#L665 is the calling function in pip today. It picks a single output, so if a wheel build generates multiple outputs, the results will be nondeterministic. Don't do that :).

...

Also, doesn't this pep overlap with PEP 426? Shouldn't these ideas be incorporated there? It seems that only the command metadata is missing from PEP 426.

It does. I believe Nick wants to see PEP-426 broken up into small bits. The meta build system described in PEP 426 is aspirational and not reflective of what pip *actually does today*. The intent I have with this PEP is to have something that preserves the current behaviour while adding the abstraction - but there is no PEP that describes the current markers-inclusive metadata on disk (345 misses setup-requires and markers). So we may need a PEP to just bless the metadata from 426, without any of the aspirational stuff.

...

Another thing, I've noticed there's a "develop" command in this draft. I think this is confusing, I'd expect it to be a "install" command (and that's what it is now in setuptools). It should be renamed to something more clear, like "inplace-build" and leave the installing to pip.

The current thing pip calls - and all existing released pips is 'setup.py develop'. I agree that doing an in place build and having pip do the install into an environment would make sense, but it makes this more aspirational. And the value here is that this is doable *now* without e.g. an additional PEP to describe how 'develop' mode in pip and related tools should behave and interoperate. We can rev this in schema version 2. There's no prose in the PEP about how that should work, so I'll add that now. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud

Robert Collins

1:39 a.m.

On 27 October 2015 at 14:06, Robert Collins wrote:

...

We can rev this in schema version 2. There's no prose in the PEP about how that should work, so I'll add that now.

Done: https://gist.github.com/rbtcollins/666c12aec869237f7cf7#upgrades -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud

Daniel Holth

1:28 p.m.

It looks like you would normally be able to say just 'flit' in the bootstrap-requires or provided-by, unless the requirements were for build code bundled inside the sdist. On Tue, Oct 27, 2015 at 1:07 AM Robert Collins wrote:

...

On 27 October 2015 at 14:06, Robert Collins wrote:

...
We can rev this in schema version 2. There's no prose in the PEP about how that should work, so I'll add that now.

Done: https://gist.github.com/rbtcollins/666c12aec869237f7cf7#upgrades

-Rob

-- Robert Collins Distinguished Technologist HP Converged Cloud _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Robert Collins

28 Oct 28 Oct

6:22 a.m.

On 28 October 2015 at 02:28, Daniel Holth wrote:

...

It looks like you would normally be able to say just 'flit' in the bootstrap-requires or provided-by, unless the requirements were for build code bundled inside the sdist.

Right - you'd say flit in bootstrap-requires, and pip would cache based on the version of flit it chose to run. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud

Ralf Gommers

29 Oct 29 Oct

10:54 p.m.

On Tue, Oct 27, 2015 at 12:23 AM, Robert Collins wrote:

...

On 27 October 2015 at 10:32, Ralf Gommers wrote:

...

...

...
(2) Complex example: to build a Scipy wheel on Windows with MinGW the command is ``python setup.py config --compiler=mingw32 build --compiler=mingw32 bdist_wheel``.

So in this case the build tool needs to know about the compiler stuff itself- pip doesn't know. We have a way in pip to tunnel stuff down to setuptools today; thats incompatible with dynamically building wheels on the fly for 'pip install' - so I'm not sure it needs to be reflected here.

It looks like you made a start at https://github.com/rbtcollins/interoperability-peps/blob/build-system-abstra... "Instead we recommend that individual build tools should have a config file mechanism to provide such pervasive settings across all things built locally." makes sense, at least up to "across". The same settings for everything built locally isn't appropriate - should be able to have a config file for one project. Example: you may want to build with MSVC and with MinGW on Windows. Also, it seems to me like there should be a way to pass the full path of a config file to the build tool via pip. Can be easily done via an optional key "config-file" in the build tool description. Example: right now numpy distributes a site.cfg.example that users can rename to site.cfg and usually put right next to setup.py. When one uses pip, it may go off building in some tmpdir, change path envvars, etc. - so how does the build tool find that config file? Ralf

...

We'll need some more input on that I think. ...

...
mechanism) to the build tool. If it is out of scope, I'd be interested to see what you think are use-cases with complex requirements that are enabled by this PEP.

The PEP is aimed at enabling additional build-tools to be on parity with setuptools in *pip install* invocations that go through the wheel-autobuild-path in pip.

The complex examples of passing arbitrary options to setup.py currently bypasses wheel building in pip, and so can't be tackled at all :(.

But we can work on including that with some thought.

Marcus Smith

28 Oct 28 Oct

4:50 a.m.

...

Current draft text in rendered form at: https://gist.github.com/rbtcollins/666c12aec869237f7cf7

Thanks for working on this. Overall I like the idea, but have some comments/questions 1) *Please*, *please*, *please* let's start doing PEP conversations as PRs to pypa/interoperability-peps : ) There's a place in there for unnumbered PEPs. Review will be better and faster IMO as PRs. If the PR gets too cluttered with old conversations, just start clean with a new PR. We can announce the PRs here, but don't discuss them on the mailing list. The same goes for trimming down PEP426 as you mention. Let's do that as a PR to pypa/interoperability-peps to the existing document. 2) Ditto on Ralf's concerns about readability. I only really understood it after seeing the examples you gave to Ralf (they need to be in the PEP, not just in response to Ralf). There's a few places I'd want to alter the wording, but I likely won't bother, unless it's done as a PR. 3) You refer to "source distribution". What does that mean exactly in this PEP? just the current setuptools notion? 4) Although using a process interface is not necessarily a problem, I don't agree with your point on why a python interface would be unworkable. You're assuming that pip would try to import all the build tools (for every dependency it's processing) in the master process. An alternative could be that pip would have it's own tool (that runs as a subprocess in an isolated env) that knows how to load and work with python build interfaces. You could argue that a python api is an advantage, because build tools aren't forced to grow a certain cli for pip, they just have to add a shim module that conforms to the interface. But my point is not to argue for the Python API, but for you to remove an argument that from what I can tell, isn't really an argument for one or the other. 5) at one pt you said "--dump-build-requires would just output a constant string: {"schema_version": "2.0", "build_requires": []}" you meant "metadata_version", right? 6) Can you explain better why we need to manage a pypa.yaml schema version *and* a build description schema version. Why not just a version for pypa.yaml, and if anything changes (in the yaml schema or the build "description" api), then just bump the version. 7) it's unclear when pip get's to run "dist-info" and when the result might be different. For example, we've discussed that run time dependencies may get clarifed *after* the build process.... so this command might produce different results at different times? --Marcus

...

-- Robert Collins Distinguished Technologist HP Converged Cloud _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Ben Finney

5:02 a.m.

New subject: Please don't impose additional barriers to participation (was: build system abstraction PEP)

Marcus Smith writes:

...

1) *Please*, *please*, *please* let's start doing PEP conversations as PRs to pypa/interoperability-peps : )

Please keep the conversation on a mailing list where one can participate without needing to sign up with any particular service provider. Your proposal would have the effect of excluding people from the conversation if they don't agree to have a GitHub account. I think it's valuable to avoid that barrier to entry, needing only an email account. -- \ “'Tis strange, — but true; for truth is always strange; / | `\ Stranger than fiction.” —“Lord” George Gordon Noel Byron, _Don | _o__) Juan_ | Ben Finney

Robert Collins

6:19 a.m.

New subject: Please don't impose additional barriers to participation (was: build system abstraction PEP)

On 28 October 2015 at 18:02, Ben Finney wrote:

...

Marcus Smith writes:

...
1) *Please*, *please*, *please* let's start doing PEP conversations as PRs to pypa/interoperability-peps : )

Please keep the conversation on a mailing list where one can participate without needing to sign up with any particular service provider.

One has to sign up with the mailing list, so there's no functional difference there.

...

Your proposal would have the effect of excluding people from the conversation if they don't agree to have a GitHub account. I think it's valuable to avoid that barrier to entry, needing only an email account.

They can always reply to emails. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud

Ben Finney

6:36 a.m.

New subject: Please don't impose additional barriers to participation

Robert Collins writes:

...

On 28 October 2015 at 18:02, Ben Finney wrote:

...
Please keep the conversation on a mailing list where one can participate without needing to sign up with any particular service provider.

One has to sign up with the mailing list, so there's no functional difference there.

You've stated that point in the past, and it's still untrue. The functional difference is that the mailing list is provided by effectively the same organisation that one is wanting to participating in discussion with. An objector to the terms of service of the mailing list is much less likely to want to also participate as part of this community. The same is certainly not true of GitHub. Please don't make a GitHub account a requirement for participating in this community's discussions on an equal footing with other participants.

...

...
Your proposal would have the effect of excluding people from the conversation if they don't agree to have a GitHub account. I think it's valuable to avoid that barrier to entry, needing only an email account.

They can always reply to emails.

As first-class citizens, like today? Or as outsiders looking in? The proposal is to move the primary discussion forum *away* from this forum, and make GitHub participants first-class citizens of that forum. Either there is no significant advantage to having the discussion forum take place primarily on the walled garden of GitHub, in which case the proposal seems pointless on its face. Or there is some significant advantage being proposed, in which case my objections stand: don't restrict that advantage to only those who are willing to use a GitHub account to do so. -- \ “If you go flying back through time and you see somebody else | `\ flying forward into the future, it's probably best to avoid eye | _o__) contact.” —Jack Handey | Ben Finney

Marcus Smith

7:30 a.m.

New subject: Please don't impose additional barriers to participation (was: build system abstraction PEP)

my intention certainly wasn't to try to exclude anybody. for me, it's the practical matter of the PR UI being more effective than a mailing list thread (in this case referring to a gist), and that we can track proposals easier and link to them from issues (in that same repo) and other PyPA docs. also, to be clear, this isn't about thinking PyPA would sidestep the PEP approval process all together. It's about managing drafts, reviews and updates into that process. certainly, this isn't my decision alone to make this change... just stating my preference, and hoping to get others to agree. Part of my motivation for bringing it up, was do due to trying write a "PyPA Roadmap" recently for pypa.io, and wanting it to be easier to track and link to ideas that people are coming up with (ideas that don't immediately start as draft PEPs) --Marcus On Tue, Oct 27, 2015 at 10:02 PM, Ben Finney wrote:

...

Marcus Smith writes:

...
1) *Please*, *please*, *please* let's start doing PEP conversations as PRs to pypa/interoperability-peps : )

Please keep the conversation on a mailing list where one can participate without needing to sign up with any particular service provider.

Your proposal would have the effect of excluding people from the conversation if they don't agree to have a GitHub account. I think it's valuable to avoid that barrier to entry, needing only an email account.

-- \ “'Tis strange, — but true; for truth is always strange; / | `\ Stranger than fiction.” —“Lord” George Gordon Noel Byron, _Don | _o__) Juan_ | Ben Finney

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Nick Coghlan

8:39 a.m.

New subject: Please don't impose additional barriers to participation (was: build system abstraction PEP)

On 28 Oct 2015 09:32, "Marcus Smith" wrote:

...

my intention certainly wasn't to try to exclude anybody. for me, it's

the practical matter of the PR UI being more effective than a mailing list thread (in this case referring to a gist), and that we can track proposals easier and link to them from issues (in that same repo) and other PyPA docs.

...

also, to be clear, this isn't about thinking PyPA would sidestep the PEP

approval process all together. It's about managing drafts, reviews and updates into that process.

...

certainly, this isn't my decision alone to make this change... just

stating my preference, and hoping to get others to agree. Part of my motivation for bringing it up, was do due to trying write a "PyPA Roadmap" recently for pypa.io, and wanting it to be easier to track and link to ideas that people are coming up with (ideas that don't immediately start as draft PEPs) I'd certainly like us to move line by line comments (etc) to the interoperability PEPs repo. Overall design discussions would stay here on the mailing list, and if folks submitting PEPs don't want to create a GitHub account, they can send them here and we'll create the PR on their behalf. If python-dev ends up adopting GitLab for the main PEPs repo, then we should be able to move the whole process there, rather than needing to maintain a separate copy. Cheers, Nick.

...

--Marcus

On Tue, Oct 27, 2015 at 10:02 PM, Ben Finney

wrote:

...

...
Marcus Smith writes:

...
1) *Please*, *please*, *please* let's start doing PEP conversations as PRs to pypa/interoperability-peps : )

Please keep the conversation on a mailing list where one can participate without needing to sign up with any particular service provider.

Your proposal would have the effect of excluding people from the conversation if they don't agree to have a GitHub account. I think it's valuable to avoid that barrier to entry, needing only an email account.

-- \ “'Tis strange, — but true; for truth is always strange; / | `\ Stranger than fiction.” —“Lord” George Gordon Noel Byron, _Don | _o__) Juan_ | Ben Finney

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Marcus Smith

30 Oct 30 Oct

5:52 a.m.

New subject: Please don't impose additional barriers to participation (was: build system abstraction PEP)

...

If python-dev ends up adopting GitLab for the main PEPs repo, then we should be able to move the whole process there, rather than needing to maintain a separate copy.

will that be as open as pypa/interoperability-peps? if it's closed off such that only python devs can log PRs against PEPs once they're in the system, then that's no fun. If so, I'd still want to keep pypa/interoperability-peps as the front end tool for the actual change management.

Brett Cannon

3:29 p.m.

New subject: Please don't impose additional barriers to participation (was: build system abstraction PEP)

On Thu, 29 Oct 2015 at 22:53 Marcus Smith wrote:

...

...
If python-dev ends up adopting GitLab for the main PEPs repo, then we should be able to move the whole process there, rather than needing to maintain a separate copy.

will that be as open as pypa/interoperability-peps?

Proof of concept and final proposal isn't in front of me yet so it isn't known. I think Nick's hope is that it will be as open. -Brett

...

if it's closed off such that only python devs can log PRs against PEPs once they're in the system, then that's no fun. If so, I'd still want to keep pypa/interoperability-peps as the front end tool for the actual change management.

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Barry Warsaw

1 Nov 1 Nov

1:08 a.m.

New subject: Please don't impose additional barriers to participation (was: build system abstraction PEP)

On Oct 29, 2015, at 10:52 PM, Marcus Smith wrote:

...

...
If python-dev ends up adopting GitLab for the main PEPs repo, then we should be able to move the whole process there, rather than needing to maintain a separate copy.

will that be as open as pypa/interoperability-peps? if it's closed off such that only python devs can log PRs against PEPs once they're in the system, then that's no fun. If so, I'd still want to keep pypa/interoperability-peps as the front end tool for the actual change management.

The way I believe it would work via GitLab is that anybody can fork the repo, push branches to their fork, and file PRs against the PEPs repo. Pushing to the PEPs repo would be gated via privileged members of the repo's owner, either via git push or button push (i.e. "hey website, I'm chillin' on the beach with my tablet so do the merge for me!") Cheers, -Barry

Nick Coghlan

10:58 p.m.

New subject: Please don't impose additional barriers to participation (was: build system abstraction PEP)

On 1 November 2015 at 02:08, Barry Warsaw wrote:

...

On Oct 29, 2015, at 10:52 PM, Marcus Smith wrote:

...
...
If python-dev ends up adopting GitLab for the main PEPs repo, then we should be able to move the whole process there, rather than needing to maintain a separate copy.

will that be as open as pypa/interoperability-peps? if it's closed off such that only python devs can log PRs against PEPs once they're in the system, then that's no fun. If so, I'd still want to keep pypa/interoperability-peps as the front end tool for the actual change management.

The way I believe it would work via GitLab is that anybody can fork the repo, push branches to their fork, and file PRs against the PEPs repo. Pushing to the PEPs repo would be gated via privileged members of the repo's owner, either via git push or button push (i.e. "hey website, I'm chillin' on the beach with my tablet so do the merge for me!")

Exactly. The case can be made it would be more open than GitHub, since folks should be able to log in with any of the identity providers GitLab supports: http://doc.gitlab.com/ce/integration/omniauth.html One of the relevant benefits of migrating to a full repository management server (whether that's GitHub or GitLab) is that we'll be able to get away from the current manual approach to managing SSH keys in favour of people uploading their own keys after authenticating with the main web interface. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

M.-A. Lemburg

28 Oct 28 Oct

8:31 a.m.

New subject: Please don't impose additional barriers to participation

On 28.10.2015 06:02, Ben Finney wrote:

...

Marcus Smith writes:

...
1) *Please*, *please*, *please* let's start doing PEP conversations as PRs to pypa/interoperability-peps : )

Please keep the conversation on a mailing list where one can participate without needing to sign up with any particular service provider.

Your proposal would have the effect of excluding people from the conversation if they don't agree to have a GitHub account. I think it's valuable to avoid that barrier to entry, needing only an email account.

I agree with Ben. Discussions on PEPs need to happen on mailing lists, not hidden away on some issue tracker or PR ticket. PEP generally affect a large number of Python users, so it's vital to get as much feedback as possible. This is not possible using a PR approach. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Oct 28 2015)

...

...
...
Python Projects, Coaching and Consulting ... http://www.egenix.com/ Python Database Interfaces ... http://products.egenix.com/ Plone/Zope Database Interfaces ... http://zope.egenix.com/

2015-10-23: Released mxODBC Connect 2.1.5 ... http://egenix.com/go85 ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/

Ionel Cristian Mărieș

6:27 p.m.

New subject: Please don't impose additional barriers to participation

On Wed, Oct 28, 2015 at 10:31 AM, M.-A. Lemburg wrote:

...

I agree with Ben. Discussions on PEPs need to happen on mailing lists, not hidden away on some issue tracker or PR ticket.

I think some balance is needed here. Every so often discussions on the mailing list get so disorganized and slide into so many irrelevant things I can't afford to participate. I think the PR thing should at least be tried and MAL/Ben haven't provided any pertinent arguments against making a github account anyway. Plus the PyPA shown willingness to accommodate feedback from the mailinglist. I'd expect draft updates will be posted once in a while on distutils-sig for people who don't have a web browser or github account. Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro

Paul Moore

6:34 p.m.

New subject: Please don't impose additional barriers to participation

On 28 October 2015 at 18:27, Ionel Cristian Mărieș wrote:

...

On Wed, Oct 28, 2015 at 10:31 AM, M.-A. Lemburg wrote:

...
I agree with Ben. Discussions on PEPs need to happen on mailing lists, not hidden away on some issue tracker or PR ticket.

I think some balance is needed here. Every so often discussions on the mailing list get so disorganized and slide into so many irrelevant things I can't afford to participate. I think the PR thing should at least be tried and MAL/Ben haven't provided any pertinent arguments against making a github account anyway.

Plus the PyPA shown willingness to accommodate feedback from the mailinglist. I'd expect draft updates will be posted once in a while on distutils-sig for people who don't have a web browser or github account.

Note that I also prefer substantive discussions on the mailing list. I'm happy to have simple edits (typos, wording clarifications, etc) covered on the PR tracker, but for anything significant around the intent of the PEP, motivating use cases, design decisions, etc, I find the github tracker interface to be pretty awful. The lack of quoting in typical discussions makes it harder to follow threads, the need to switch from my mail client (where notifications will appear) to the github interface if I want to contribute in the "correct" form (reply by email works, but doesn't integrate cleanly in the "style" of the tracker discussion for anything but short interjections) makes it harder to contribute, and the fact that it's less conducive to "lurking" than a mailing list are concrete reasons, not related to wanting to use a github account, for preferring the mailing list. Call me an old fogey for preferring last year's technology if you must :-) Paul

Ian Cordasco

6:33 p.m.

New subject: Please don't impose additional barriers to participation

On Wed, Oct 28, 2015 at 3:31 AM, M.-A. Lemburg wrote:

...

On 28.10.2015 06:02, Ben Finney wrote:

...
Marcus Smith writes:

...
1) *Please*, *please*, *please* let's start doing PEP conversations as PRs to pypa/interoperability-peps : )

Please keep the conversation on a mailing list where one can participate without needing to sign up with any particular service provider.

Your proposal would have the effect of excluding people from the conversation if they don't agree to have a GitHub account. I think it's valuable to avoid that barrier to entry, needing only an email account.

I agree with Ben. Discussions on PEPs need to happen on mailing lists, not hidden away on some issue tracker or PR ticket.

Others may be willing to tolerate your FUD, but without concrete reasons against GitHub (other than zomg it's a proprietary service) I don't see a reason to not use the pull request flow on an open repository that is free for people to clone, fork, contribute to, etc. GitHub isn't my preferred hosting platform for git but it is the defacto standard and it's workflow ubiquitous, documented, and far more user-friendly than mailing list threads (especially when they devolve into ideology wars). Also nothing precludes mailing list discussions, so without details about your objections, I don't see why this should be held up.

Paul Moore

6:42 p.m.

New subject: Please don't impose additional barriers to participation

On 28 October 2015 at 18:33, Ian Cordasco wrote:

...

Also nothing precludes mailing list discussions, so without details about your objections, I don't see why this should be held up.

One immediate question - not at all FUD, nor intended as provocative, it's a genuine question. I thought right now I'd check if there had actually been any discussion on the PR that I'd missed [1]. But I don't have a link to the specific tracker issue, and offhand I can't recall the repo name. So how do I find the issue quickly? For discussions here, I can scan the list of message subjects in the distutils-sig folder in my email. Quickly checking on a tracker PR is harder, and so I'm *much* less likely to contribute. Result - when the PEP is posted back to the list once it's been thrashed out on the tracker, there's a *second* round of time-consuming debate as people who missed the first round join in with issues that have possibly already been discussed. I assume there's no intention of PEPs being discussed and approved, *all* without debate on distutils-sig? Paul [1] If I'm supposed to be getting notifications for comments on the PR (as a member of the PyPA group, shouldn't I be?) then it's not happening... I know I can subscribe to the PR, but I'm not clear why I should need to - I don't for pip issues, for instance...

Donald Stufft

6:44 p.m.

New subject: Please don't impose additional barriers to participation

On October 28, 2015 at 2:42:19 PM, Paul Moore (p.f.moore@gmail.com) wrote:

...

[1] If I'm supposed to be getting notifications for comments on the PR (as a member of the PyPA group, shouldn't I be?) then it's not happening... I know I can subscribe to the PR, but I'm not clear why I should need to - I don't for pip issues, for instance...

You’re probably not directly subscribed to that repo, there should be a follow button on https://github.com/pypa/interoperability-peps. You got it automatically for pip because you were explicitly added as a committer on the pip repo. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Paul Moore

8:12 p.m.

New subject: Please don't impose additional barriers to participation

On 28 October 2015 at 18:44, Donald Stufft wrote:

...

On October 28, 2015 at 2:42:19 PM, Paul Moore (p.f.moore@gmail.com) wrote:

...
[1] If I'm supposed to be getting notifications for comments on the PR (as a member of the PyPA group, shouldn't I be?) then it's not happening... I know I can subscribe to the PR, but I'm not clear why I should need to - I don't for pip issues, for instance...

You’re probably not directly subscribed to that repo, there should be a follow button on https://github.com/pypa/interoperability-peps. You got it automatically for pip because you were explicitly added as a committer on the pip repo.

No subscribe button that I can see on the repo. I could subscribe on a per-issue basis, but that's a bit different. Note - I'm actually not sure I *want* to be subscribing to issue updates like this in any case - it'd be more email filters to file the messages, more management of email groups, etc. If people *really* want to discuss on the PR, why not subscribe distutils-sig to the PR, so that list members can contribute to the discussion as well? (That's actually probably a bad idea, because the differences between mailing list etiquette and PR discussion etiquette are pretty big - but that's sort of my point, I *prefer* the mailing list format for PEP discussions, and I don't think I'm the only one...) Paul

Ralf Gommers

8:15 p.m.

New subject: Please don't impose additional barriers to participation

On Wed, Oct 28, 2015 at 9:12 PM, Paul Moore wrote:

...

On 28 October 2015 at 18:44, Donald Stufft wrote:

...
On October 28, 2015 at 2:42:19 PM, Paul Moore (p.f.moore@gmail.com) wrote:

...
[1] If I'm supposed to be getting notifications for comments on the PR (as a member of the PyPA group, shouldn't I be?) then it's not happening... I know I can subscribe to the PR, but I'm not clear why I should need to - I don't for pip issues, for instance...

You’re probably not directly subscribed to that repo, there should be a follow button on https://github.com/pypa/interoperability-peps. You got it automatically for pip because you were explicitly added as a committer on the pip repo.

No subscribe button that I can see on the repo. I could subscribe on a per-issue basis, but that's a bit different.

It's the "Watch" button on the right top of https://github.com/pypa/interoperability-peps Ralf

Paul Moore

10:22 p.m.

New subject: Please don't impose additional barriers to participation

On 28 October 2015 at 20:15, Ralf Gommers wrote:

...

It's the "Watch" button on the right top of https://github.com/pypa/interoperability-peps

Doh. Thanks. Paul

M.-A. Lemburg

6:42 p.m.

New subject: Please don't impose additional barriers to participation

On 28.10.2015 19:33, Ian Cordasco wrote:

...

On Wed, Oct 28, 2015 at 3:31 AM, M.-A. Lemburg wrote:

...
On 28.10.2015 06:02, Ben Finney wrote:

...
Marcus Smith writes:

...
1) *Please*, *please*, *please* let's start doing PEP conversations as PRs to pypa/interoperability-peps : )

Please keep the conversation on a mailing list where one can participate without needing to sign up with any particular service provider.

Your proposal would have the effect of excluding people from the conversation if they don't agree to have a GitHub account. I think it's valuable to avoid that barrier to entry, needing only an email account.

I agree with Ben. Discussions on PEPs need to happen on mailing lists, not hidden away on some issue tracker or PR ticket.

Others may be willing to tolerate your FUD, but without concrete reasons against GitHub (other than zomg it's a proprietary service) I don't see a reason to not use the pull request flow on an open repository that is free for people to clone, fork, contribute to, etc.

GitHub isn't my preferred hosting platform for git but it is the defacto standard and it's workflow ubiquitous, documented, and far more user-friendly than mailing list threads (especially when they devolve into ideology wars).

Also nothing precludes mailing list discussions, so without details about your objections, I don't see why this should be held up.

Ok, as first step, please change your tone and reread my reply. I'm not willing to tolerate your tone. Thanks. The argument here is not about Github or not, and it's not FUD. It's the same argument as is used when starting into discussions on the Python issue tracker and the reason for often moving those to the python-dev or -ideas mailing lists. Hiding away discussions is not the way to go about when discussing PEPs. Using PRs is fine when it comes to improving the language of the PEP or adding new aspects, but the reasoning for applying these changes should be based on the results of discussions on the relevant lists, distutils-sig in this case. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Oct 28 2015)

...

...
...
Python Projects, Coaching and Consulting ... http://www.egenix.com/ Python Database Interfaces ... http://products.egenix.com/ Plone/Zope Database Interfaces ... http://zope.egenix.com/

Robert Collins

6:34 a.m.

On 28 October 2015 at 17:50, Marcus Smith wrote:

...

...
Current draft text in rendered form at: https://gist.github.com/rbtcollins/666c12aec869237f7cf7

Thanks for working on this. Overall I like the idea, but have some comments/questions

1) *Please*, *please*, *please* let's start doing PEP conversations as PRs to pypa/interoperability-peps : ) There's a place in there for unnumbered PEPs. Review will be better and faster IMO as PRs. If the PR gets too cluttered with old conversations, just start clean with a new PR. We can announce the PRs here, but don't discuss them on the mailing list. The same goes for trimming down PEP426 as you mention. Let's do that as a PR to pypa/interoperability-peps to the existing document.

https://github.com/pypa/interoperability-peps/pull/54

...

2) Ditto on Ralf's concerns about readability. I only really understood it after seeing the examples you gave to Ralf (they need to be in the PEP, not just in response to Ralf). There's a few places I'd want to alter the wording, but I likely won't bother, unless it's done as a PR.

I'll work on an iteration for that, the pr is just what I had in the gist.

...

3) You refer to "source distribution". What does that mean exactly in this PEP? just the current setuptools notion?

yes. I don't change the definition of source distribution at all in this PEP.

...

4) Although using a process interface is not necessarily a problem, I don't agree with your point on why a python interface would be unworkable. You're assuming that pip would try to import all the build tools (for every dependency it's processing) in the master process. An alternative could be that pip would have it's own tool (that runs as a subprocess in an isolated env) that knows how to load and work with python build interfaces. You could argue that a python api is an advantage, because build tools aren't

That would mean that pip has to have the same exact version of it embedded in the environment that build tools will be run in, which it doesn't have today. I worry that that will be hard to manage, and hard for folk to debug.

...

forced to grow a certain cli for pip, they just have to add a shim module that conforms to the interface. But my point is not to argue for the Python API, but for you to remove an argument that from what I can tell, isn't really an argument for one or the other.

I'll review the text I have there, but the argument does seem valid to me.

...

5) at one pt you said "--dump-build-requires would just output a constant string: {"schema_version": "2.0", "build_requires": []}" you meant "metadata_version", right?

Yes, sorry.

...

6) Can you explain better why we need to manage a pypa.yaml schema version *and* a build description schema version. Why not just a version for pypa.yaml, and if anything changes (in the yaml schema or the build "description" api), then just bump the version.

I see no problem with evolving them in lockstep, but because the documents are separate, we either have to infer the version of one, or we have to be explicit on each. I'd rather avoid the chance for a bug where something tries to parse a v2 schema build description with a v1 schema parser. Making them self identifying avoids that.

...

7) it's unclear when pip get's to run "dist-info" and when the result might be different. For example, we've discussed that run time dependencies may get clarifed *after* the build process.... so this command might produce different results at different times?

Pip would run dist-info when determining the install-requires and extras for the package. That would be required to generate the actual final dependencies. It would only be run once I expect. I'll clarify that expected behaviour in the PEP. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud

Paul Moore

8:54 a.m.

On 28 October 2015 at 06:34, Robert Collins wrote:

...

...
1) *Please*, *please*, *please* let's start doing PEP conversations as PRs to pypa/interoperability-peps : ) There's a place in there for unnumbered PEPs. Review will be better and faster IMO as PRs. If the PR gets too cluttered with old conversations, just start clean with a new PR. We can announce the PRs here, but don't discuss them on the mailing list. The same goes for trimming down PEP426 as you mention. Let's do that as a PR to pypa/interoperability-peps to the existing document.

https://github.com/pypa/interoperability-peps/pull/54

I'm concerned that if the discussion all ends up on the PR, then people (like me) who follow the mailing list but aren't watching the PR discussion[1] will miss important chunks of the debate. I guess I could subscribe to the PR discussion, but the typical lack of quoting in PR conversations makes it much harder to follow threads there. I'm happy to see the reference versions and change history of PEPs in the repository, but can we not keep discussions of the principles on the public list? Otherwise, we'll have a first run of the discussion on github, then a rerun when it finally gets posted to distutils-sig... Paul [1] I'm not sure why, as part of the PyPA group on github, I don't receive notifications when PRs like this one are added. Do I need to subscribe to something that I haven't?

Marcus Smith

29 Oct 29 Oct

11:33 a.m.

...

...
4) Although using a process interface is not necessarily a problem, I don't agree with your point on why a python interface would be unworkable. You're assuming that pip would try to import all the build tools (for every dependency it's processing) in the master process. An alternative could be that pip would have it's own tool (that runs as a subprocess in an isolated env) that knows how to load and work with python build interfaces. You could argue that a python api is an advantage, because build tools aren't

That would mean that pip has to have the same exact version of it embedded in the environment that build tools will be run in, which it doesn't have today.

sorry, I can't make clear sense of your sentence here. : ) I'll just explain my point again. pip doesn't necessarily have to "interact with many different versions of the same build tool during a single invocation" if for example it's subprocessing the interactions to some "pip-build" tool that handles the imports and use of the python API. I.e. pips calls some "pip-build" too (per build), which does the import, not pip itself. and again, it's not about arguing for this idea, but just that your "in-process APIs are harder" argument doesn't decide the matter.

...

I see no problem with evolving them in lockstep,

it's unnecessarily complex IMO. if they're really in lockstep, then they're one thing it seems to me. I'd rather avoid the chance for a bug where something tries to parse a

...

v2 schema build description with a v1 schema parser.

but it won't happen? the pypa.yaml schema version would determine the parser version that's used.

...

...
7) it's unclear when pip get's to run "dist-info" and when the result might be different. For example, we've discussed that run time dependencies may get clarifed *after* the build process.... so this command might produce different results at different times?

Pip would run dist-info when determining the install-requires and extras for the package.

you're not addressing the point about how the act of building can create new run time dependencies, per the whole long discussion with Nathaniel recently (his draft deals with this matter explicitly) --Marcus

Marcus Smith

9:04 p.m.

...

pip doesn't necessarily have to "interact with many different versions of the same build tool during a single invocation" if for example it's subprocessing the interactions to some "pip-build" tool that handles the imports and use of the python API. I.e. pips calls some "pip-build" too (per build), which does the import, not pip itself.

sorry, there's 2 confusing misspells in my sentence. here it is again: pip doesn't necessarily have to "interact with many different versions of the same build tool during a single invocation" if for example it's sub-processing the interactions to some "pip-build" tool that handles the imports and use of the python API. I.e. pip calls some "pip-build" tool (per build), which does the import, not pip itself.

Robert Collins

10:56 p.m.

On 30 October 2015 at 00:33, Marcus Smith wrote:

...

...
...
4) Although using a process interface is not necessarily a problem, I don't agree with your point on why a python interface would be unworkable. You're assuming that pip would try to import all the build tools (for every dependency it's processing) in the master process. An alternative could be that pip would have it's own tool (that runs as a subprocess in an isolated env) that knows how to load and work with python build interfaces. You could argue that a python api is an advantage, because build tools aren't

That would mean that pip has to have the same exact version of it embedded in the environment that build tools will be run in, which it doesn't have today.

sorry, I can't make clear sense of your sentence here. : )

I'll just explain my point again.

pip doesn't necessarily have to "interact with many different versions of the same build tool during a single invocation" if for example it's subprocessing the interactions to some "pip-build" tool that handles the imports and use of the python API. I.e. pips calls some "pip-build" too (per build), which does the import, not pip itself.

and again, it's not about arguing for this idea, but just that your "in-process APIs are harder" argument doesn't decide the matter.

I agree that it doesn't decide the matter - I added a section to rationale about this contention. We can literally do whatever we want to do - my reading of the interactions is that its going to be less fragile overall to not have to have a Python->command process thunk that is a separate interface we have to carry around in Pip everywhere. But we can make literally anything work, so I'm probably going to just say 'Nick and Donald can decide' and not argue about this.

...

...
I see no problem with evolving them in lockstep,

it's unnecessarily complex IMO. if they're really in lockstep, then they're one thing it seems to me.

I don't understand.

...

...
I'd rather avoid the chance for a bug where something tries to parse a v2 schema build description with a v1 schema parser.

but it won't happen? the pypa.yaml schema version would determine the parser version that's used.

We have two separate documents. If we mark the schema version in one, and not in the other, then we introduce room for skew. Yes, anything that skewed would be buggy, but there'd be no clue that such a bug was happening to anyone debugging it. All my experience debugging network protocols and data storage engines tells me to be explicit about the version of anything that can *possibly* be evaluated separately.

...

...
...
7) it's unclear when pip get's to run "dist-info" and when the result might be different. For example, we've discussed that run time dependencies may get clarifed *after* the build process.... so this command might produce different results at different times?

Pip would run dist-info when determining the install-requires and extras for the package.

you're not addressing the point about how the act of building can create new run time dependencies, per the whole long discussion with Nathaniel recently (his draft deals with this matter explicitly)

I don't follow how there is an issue: the dist-info hook would be responsible for figuring that out. In the numpy ABI case it would do discovery of the ABI it would build against and then use that. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud

Robert Collins

9 Nov 9 Nov

1:28 a.m.

Now that the dependent spec is up for review, I've refreshed this build system abstraction one. Key changes: - pep-426 -> wheel METADATA files as the distribution description interface - dropped yaml for JSON as its in the stdlib I've kept the indirection via a build system rather than adopting setup.py as-is: I judge the risk for user confusion to be large enough that its worth doing that - even though we recommend a setuptools shim, forcing it on everyone is unpleasant. commit 6de544a6e6aa2d0505e2ac8eb364b8b95e27790d Author: Robert Collins Date: Wed Oct 28 19:21:48 2015 +1300 PEP for build system abstraction. diff --git a/build-system-abstraction.rst b/build-system-abstraction.rst new file mode 100644 index 0000000..b8232c8 --- /dev/null +++ b/build-system-abstraction.rst @@ -0,0 +1,419 @@ +:PEP: XX +:Title: Build system abstraction for pip/conda etc +:Version: $Revision$ +:Last-Modified: $Date$ +:Author: Robert Collins , + Nathaniel Smith +:BDFL-Delegate: Donald Stufft +:Discussions-To: distutils-sig +:Status: Draft +:Type: Standards Track +:Content-Type: text/x-rst +:Created: 26-Oct-2015 +:Post-History: XX +:Requires: The new dependency-specification PEP. + + +Abstract +======== + +This PEP specifies a programmatic interface for pip [#pip]_ and other +distribution or installation tools to use when working with Python +source trees (both the developer tree - e.g. the git tree - and source +distributions). + +The programmatic interface allows decoupling of pip from its current +hard dependency on setuptools [#setuptools]_ able for two +key reasons: + +1. It enables new build systems that may be much easier to use without + requiring them to even appear to be setuptools. + +2. It facilitates setuptools itself changing its user interface without + breaking pip, giving looser coupling. + +The programmatic interface also enables pip to install build time requirements +for packages which is an important step in getting pip to full feature parity +with the installation components of easy-install. + +As PEP-426 [#pep426]_ is draft, we cannot utilise the metadata format it +defined. However PEP-427 wheels are in wide use and fairly well specified, so +we have adopted the METADATA format from that for specifying distribution +dependencies. However something was needed for communicating bootstrap +requirements and build requirements - but a thin JSON schema is sufficient. + +Motivation +========== + +There is significant pent-up frustration in the Python packaging ecosystem +around the current lock-in between build system and pip. Breaking that lock-in +is better for pip, for setuptools, and for other build systems like flit +[#flit]_. + +Specification +============= + +Overview +-------- + +Build tools will be located by reading a file ``pypa.json`` from the root +directory of the source tree. That file describes how to get the build tool. +The build tool is then self-describing. This avoids having stale descriptions +of build tools encoded into sdists that have been uploaded to PyPI [#pypi]_. + +The interface involves executing processes rather than loading Python modules, +because tools like pip will often interact with many different versions of the +same build tool during a single invocation, and in-process APIs are much +harder to manage in that situation. + +Process interface +----------------- + +Where a process needs to be run, we need to be able to assemble the subprocess +call to make. For this we use a simple command string with in-Python variable +interpolation. Only variables defined in this PEP will be honoured: the use of +additional variables is an error. Additional variables can only be added with +a schema version increase. Basic variable expressions such as ``${PYTHON}``, +``${PYTHON:-python}``, ``${PYTHON:+PYTHON -m coverage}`` can be used. An +implementation of this is in shellvars [#shellvars]_. + +Processes will be run with the current working directory set to the root of +the source tree. + +Available variables +------------------- + +PYTHON + The Python interpreter in use. This is important to enable calling things + which are just Python entry points:: + + ${PYTHON} -m foo + +OUTPUT_DIR + Where to create requested output. If not set, outputting to the current + working directory is appropriate. This is required by ``pip`` which needs + to have wheels output in a known location. + +pypa.json +--------- + +The yaml file has the following schema. Extra keys are ignored. + +schema + The version of the schema. This PEP defines version 1. + Defaults to 1 + +bootstrap-requires + A list of dependency specifications that must be installed before + running the build tool. For instance, if using flit, then the requirements + might be:: + + bootstrap-requires: + - flit + +build-tool + A command to run to query the build tool for its complete interface. + The build tool should output on stdout a build tool description JSON + document. Stdin may not be read from, and stderr may be handled however + the calling process desires. + +build tool description +---------------------- + +The build tool description schema. Extra keys are ignored. + +schema + The version of the schema. This PEP defines version 1. + Defaults to 1 + +build-requires + Command to run to query build requirements. Build requirements are + returned as a JSON document with one key ``build_requires`` consisting of + a list of dependency specifications. Additional keys must be ignored. + +metadata + Command to run to generate metadata for the project. The metadata should + be output on stdout, stdin may not be consumed, and stderr handling is up + to the discretion of the calling process. The build-requires for the + project will be present in the Python environment when the metadata + command is run. pip would run metadata just once to determine what other + packages need to be downloaded and installed. The metadata is output as a + wheel METADATA file per PEP-427 [#pep427]_. + + Note that the metadata generated by the metadata command, and the metadata + present in a generated wheel must be identical. + +wheel + Command to run to build a wheel of the project. OUTPUT_DIR will be set and + point to an existing directory where the wheel should be output. Stdin + may not be consumed, stdout and stderr handling is at the discretion of + the calling process. The build-requires for the project will be present in + the Python environment when the wheel command is run. Only one file + should be output - if more are output then pip would pick an arbitrary one + to consume. + +develop + Command to do an in-place 'development' installation of the project. + Stdin may not be consumed, stdout and stderr handling is at the discretion + of the calling process. The build-requires for the project will be + present in the Python environment when the develop command is run. + + Not all build systems will be able to perform develop installs. If a build + system cannot do develop installs, then this key can be omitted. + + Note that when it is omitted, commands like ``pip install -e foo`` + will be unable to complete. + +provided-by + Optional distribution name that provides this build system. + This is used to facilitate caching the build tool description. If absent + then the build tool description cannot be cached, which will incure an + extra subprocess per package being built (to query the build tool). + Specifically, where the resolved bootstrap-requires results in the same + version of the named distribution being installed, the build tool + description is presumed to be identical. + +Python environments and hermetic builds +--------------------------------------- + +This specification does not prescribe whether builds should be hermetic or not. +Existing build tools like setuptools will use installed versions of build time +requirements (e.g. setuptools_scm) and only install other versions on version +conflicts or missing dependencies. However its likely that better consistency +can be created by always isolation builds and using only the specified dependencies. + +However there are nuanced problems there - such as how can users force the +avoidance of a bad version of a build requirement which meets some packages +dependencies. Future PEPs may tackle this problem, but it is not currently in +scope - it does not affect the metadata required to coordinate between build +systems and things that need to do builds. + +Upgrades +-------- + +Both 'pypa.json' and the build tool description are versioned to permit future +incompatible changes. There is a sequence dependency here. + +Upgrades to the schemas defined in this specification must proceed with the +consumers first. To ensure that consumers should refuse to operate when +'pypa.json' has a schema version that they do not recognise. + +Build tools listed in a 'pypa.json' with schema version 1 must not generate a +build tool description with a version other than 1 (or absent, as the default +is 1). + +Thus the sequence for upgrading either of schemas in a new PEP will be: + +1. Issue new PEP defining build tool description and 'pypa.json' schemas. The + 'pypa.yaml' schema version must change if either the 'pypa.yaml' schema has + changed or the build tool description schema has changed. The build tool + description schema version must change if the build tool description schema + has changed. +2. Consumers (e.g. pip) implement support for the new schema version. +3. Build tool authors implement the new schemas, and publish updated reference + 'pypa.json' files for their users. 4. Package authors opt into the new + schema when they are happy to introduce a dependency on the version of + 'pip' (and potentially other consumers) that introduced support for the new + schema version. + +The *same* process will take place for the initial deployment of this PEP:- +the propogation of the capability to use this PEP without a `setuptools shim`_ +will be largely gated by the adoption rate of the first version of pip that +supports it. + +Static metadata in sdists +------------------------- + +This PEP does not tackle the current inability to trust static metadata in +sdists. That is a separate problem to identifying and consuming the build +system that is in use in a source tree, whether it came from an sdist or not. + +Handling of compiler options +---------------------------- + +Handling of different compiler options is out of scope for this specification. + +pip currently handles compiler options by appending user supplied strings to +the command line it runs when running setuptools. This approach is sufficient +to work with the build system interface defined in this PEP, with the +exception that globally specified options will stop working globally as +different build systems evolve. That problem can be solved in pip (or conda or +other installers). + +In the long term, wheels should be able to express the difference between +wheels built with one compiler or options vs another. + +Examples +======== + +An example 'pypa.json' for using flit:: + + bootstrap-requires: + - flit + build-tool: flit --dump-build-description + +When 'pip' reads this it would prepare an environment with flit in it and +run `flit --dump-build-description` which would output something like:: + + build-requires: flit --dump-build-requires + metadata: flit --dump-metadata + provided-by: flit + wheel: flit wheel -d $OUTPUT_DIR + +The `--dump-` switches in this example would be needed to be added to +flit (or someone else could write an adapter). Because flit doesn't have +setup-requires support today, `--dump-build-requires` would just output a +constant string:: + + {"build_requires": []} + +`--dump-metadata` would interrogate `flit.ini` and marshal the metadata into +a wheel METADATA file and output that on stdout. + +flit wheel would need a `-d` parameter that tells it where to output the +wheel (pip needs this). + +Backwards Compatibility +======================= + +Older pips will remain unable to handle alternative build systems. +This is no worse than the status quo - and individual build system +projects can decide whether to include a shim ``setup.py`` or not. + +All existing build systems that can product wheels and do develop installs +should be able to run under this abstraction and will only need a specific +adapter for them constructed and published on PyPI. + +In the absence of a ``pypa.json`` file, tools like pip should assume a +setuptools build system and use setuptools commands directly. + + +Network effects +--------------- + +Projects that adopt build systems that are not setuptools compatible - that +is that they have no setup.py, or the setup.py doesn't accept commands that +existing tools try to use - will not be installable by those existing tools. + +Where those projects are used by other projects, this effect will cascade. + +In particular, because pip does not handle setup-requires today, any project +(A) that adopts a setuptools-incompatible build system and is consumed as a +setup-requirement by a second project (B) which has not itself transitioned to +having a pypa.json will make B uninstallable by any version of pip. This is +because setup.py in B will trigger easy-install when 'setup.py egg_info' is +run by pip, and that will try and fail to install A. + +As such we recommend that tools which are currently used as setup-requires +either ensure that they keep a `setuptools shim`_ or find their consumers and +get them all to upgrade to the use of a `pypa.json` in advance of moving +themselves. Pragmatically that is impossible, so the advice is to keep a +setuptools shim indefinitely - both for projects like pbr, setuptools_scm and +also projects like numpy. + +setuptools shim +--------------- + +It would be possible to write a generic setuptools shim that looks like +``setup.py`` and under the hood uses ``pypa.json`` to drive the builds. This +is not needed for pip to use the system, but would allow package authors to +use the new features while still retaining compatibility with older pip +versions. + +Rationale +========= + +This PEP started with a long mailing list thread on distutils-sig [#thread]_. +Subsequent to that a online meeting was held to debug all the positions folk +had. Minutes from that were posted to the list [#minutes]_. + +This specification is a translation of the consensus reached there into PEP +form, along with some arbitrary choices on the minor remaining questions. + +The basic heuristic for the design has to been to focus on introducing an +abstraction without requiring development not strictly tied to the +abstraction. Where the gap is small to improvements, or the cost of using the +existing interface is very high, then we've taken on having the improvement as +a dependency, but otherwise defered such to future iterations. + +We chose wheel METADATA files rather than defining a new specification, +because pip can already handle wheel .dist-info directories which encode all +the necessary data in a METADATA file. PEP-426 can't be used as it's still +draft, and defining a new metadata format, while we should do that, is a +separate problem. Using a directory on disk would not add any value to the +interface (pip has to do that today due to limitations in the setuptools +CLI). + +The use of 'develop' as a command is because there is no PEP specifying the +interoperability of things that do what 'setuptools develop' does - so we'll +need to define that before pip can take on the responsibility for doing the +'develop' step. Once thats done we can issue a successor PEP to this one. + +The use of a command line API rather than a Python API is a little +contentious. Fundamentally anything can be made to work, and Robert wants to +pick something thats sufficiently lowest common denominator that +implementation is straight forward on all sides. Picking a CLI for that makes +sense because all build systems will need a CLI for end users to use anyway. + +The choice of JSON as a file format is a compromise between several +constraints. Firstly there is no stdlib YAML interpreter, nor one for any of +the other low-friction structured file formats. Secondly, INIParser is a poor +format for a number of reasons, primarily that it has very minimal structure - +but pip's maintainers are not fond of it. JSON is in the stdlib, has +sufficient structure to permit embedding anything we want in future without +requiring embedded DSL's. + +Donald suggested using ``setup.cfg`` and the existing setuptools command line +rather than inventing something new. While that would permit interoperability +with less visible changes, it requires nearly as much engineering on the pip +side - looking for the new key in setup.cfg, implementing the non-installed +environments to run the build in. And the desire from other build system +authors not to confuse their users by delivering something that looks like but +behaves quite differently to setuptools seems like a bigger issue than pip +learning how to invoke a custom build tool. + +References +========== + +.. [#pip] pip, the recommended installer for Python packages + (http://pip.readthedocs.org/en/stable/) + +.. [#setuptools] setuptools, the defacto Python package build system + (https://pythonhosted.org/setuptools/) + +.. [#flit] flit, a simple way to put packages in PyPI + (http://flit.readthedocs.org/en/latest/) + +.. [#pypi] PyPI, the Python Package Index + (https://pypi.python.org/) + +.. [#shellvars] Shellvars, an implementation of shell variable rules for Python. + (https://github.com/testing-cabal/shellvars) + +.. [#pep426] PEP-426, Python distribution metadata. + (https://www.python.org/dev/peps/pep-0426/) + +.. [#pep427] PEP-427, Python distribution metadata. + (https://www.python.org/dev/peps/pep-0427/) + +.. [#thread] The kick-off thread. + (https://mail.python.org/pipermail/distutils-sig/2015-October/026925.html) + +.. [#minutes] The minutes. + (https://mail.python.org/pipermail/distutils-sig/2015-October/027214.html) + +Copyright +========= + +This document has been placed in the public domain. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud

Nathaniel Smith

28 Oct 28 Oct

5:03 a.m.

On Sun, Oct 25, 2015 at 11:01 PM, Robert Collins wrote:

...

Since Nathaniel seems busy, I've taken the liberty of drafting a narrow PEP based on the conversations that arose from the prior discussion.

It (naturally) has my unique flavor, but builds on the work Nathaniel had put together, so I've put his name as a co-author even though he hasn't seen a word of it until now :) - all errors and mistakes are therefore mine...

Current draft text in rendered form at: https://gist.github.com/rbtcollins/666c12aec869237f7cf7

I've run it past Donald and he has a number of concerns - I think we'll need to discuss them here, and possibly in another hangout, to get a path forward.

Now that I've had a chance to read it properly... First impression: there's surprisingly little overlap between this and my simultaneously-posted draft [1] -- my draft focuses on trying to only document the stuff that everyone seemed to agree on, includes a proposal for static metadata in sdists (since Donald seemed to be saying that he considered this a mandatory component of any proposal to update how sdists work), and tries to set out a blueprint for how to organize the remaining issues, whereas yours spends most of its time on the controversial details that I decided to skip over for this draft. Being an engineer, there are a number of these details that I'm tempted to quibble over :-), but I will restrain myself for now and wait first to see how the other thread develops and whether we do have general agreement on the points I wrote, since this seems like a complex enough topic that it could easily spiral become a conversational morass if we aren't careful to keep the different issues organized? -n [1] https://mail.python.org/pipermail/distutils-sig/2015-October/027360.html -- Nathaniel J. Smith -- http://vorpus.org

Ralf Gommers

5:58 a.m.

On Wed, Oct 28, 2015 at 6:03 AM, Nathaniel Smith wrote:

...

On Sun, Oct 25, 2015 at 11:01 PM, Robert Collins wrote:

...
Since Nathaniel seems busy, I've taken the liberty of drafting a narrow PEP based on the conversations that arose from the prior discussion.

It (naturally) has my unique flavor, but builds on the work Nathaniel had put together, so I've put his name as a co-author even though he hasn't seen a word of it until now :) - all errors and mistakes are therefore mine...

Current draft text in rendered form at: https://gist.github.com/rbtcollins/666c12aec869237f7cf7

I've run it past Donald and he has a number of concerns - I think we'll need to discuss them here, and possibly in another hangout, to get a path forward.

Now that I've had a chance to read it properly...

First impression: there's surprisingly little overlap between this and my simultaneously-posted draft [1] --

Which is good, double work has been kept to a minimum - it's like you two actually coordinated this:)

...

my draft focuses on trying to only document the stuff that everyone seemed to agree on, includes a proposal for static metadata in sdists (since Donald seemed to be saying that he considered this a mandatory component of any proposal to update how sdists work), and tries to set out a blueprint for how to organize the remaining issues, whereas yours spends most of its time on the controversial details that I decided to skip over for this draft.

Imho they're not details. The controversial parts of your draft are still mostly in the metadata part. If you'd split your draft in two, then you'd see that the first one is pretty short and the second half of it is only TBDs. And those TBDs are exactly what Robert's draft fills in. @Robert: thanks for the example, very helpful. I'll look at it in more detail later. Ralf

Nathaniel Smith

6:17 a.m.

On Oct 27, 2015 10:58 PM, "Ralf Gommers" wrote:

...

On Wed, Oct 28, 2015 at 6:03 AM, Nathaniel Smith wrote:

...
On Sun, Oct 25, 2015 at 11:01 PM, Robert Collins wrote:

...
Since Nathaniel seems busy, I've taken the liberty of drafting a narrow PEP based on the conversations that arose from the prior discussion.

It (naturally) has my unique flavor, but builds on the work Nathaniel had put together, so I've put his name as a co-author even though he hasn't seen a word of it until now :) - all errors and mistakes are therefore mine...

Current draft text in rendered form at: https://gist.github.com/rbtcollins/666c12aec869237f7cf7

I've run it past Donald and he has a number of concerns - I think we'll need to discuss them here, and possibly in another hangout, to get a path forward.

Now that I've had a chance to read it properly...

First impression: there's surprisingly little overlap between this and my simultaneously-posted draft [1] --

Which is good, double work has been kept to a minimum - it's like you two

actually coordinated this:)

...

...
my draft focuses on trying to only document the stuff that everyone seemed to agree on, includes a proposal for static metadata in sdists (since Donald seemed to be saying that he considered this a mandatory component of any proposal to update how sdists work), and tries to set out a blueprint for how to organize the remaining issues, whereas yours spends most of its time on the controversial details that I decided to skip over for this draft.

Imho they're not details. The controversial parts of your draft are still

mostly in the metadata part. If you'd split your draft in two, then you'd see that the first one is pretty short and the second half of it is only TBDs. And those TBDs are exactly what Robert's draft fills in. Sure. When I say they're details, I don't mean they're unimportant; I just mean that after that last thread got so big and tangled, I don't want to spend time debating options for JSON formats if we don't yet have agreement on the big picture of what we're trying to do, or try to debate semantics and ipc conventions etc simultaneously. -n

Robert Collins

6:35 a.m.

On 28 October 2015 at 18:03, Nathaniel Smith wrote:

...

On Sun, Oct 25, 2015 at 11:01 PM, Robert Collins wrote: ...

...
I've run it past Donald and he has a number of concerns - I think we'll need to discuss them here, and possibly in another hangout, to get a path forward.

Now that I've had a chance to read it properly...

First impression: there's surprisingly little overlap between this and my simultaneously-posted draft [1] -- my draft focuses on trying to only document the stuff that everyone seemed to agree on, includes a proposal for static metadata in sdists (since Donald seemed to be saying that he considered this a mandatory component of any proposal to update how sdists work), and tries to set out a blueprint for how to organize the remaining issues, whereas yours spends most of its time on the controversial details that I decided to skip over for this draft.

Right - I'm avoiding changing how sdists works at all: that is not necessary for introducing the ability to run arbitrary build tools.

...

Being an engineer, there are a number of these details that I'm tempted to quibble over :-), but I will restrain myself for now and wait first to see how the other thread develops and whether we do have general agreement on the points I wrote, since this seems like a complex enough topic that it could easily spiral become a conversational morass if we aren't careful to keep the different issues organized?

Agreed, separate issues, separate threads :). -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud

3085

Age (days ago)

3099

Last active (days ago)

List overview

Download

40 comments

14 participants

participants (14)

Barry Warsaw
Ben Finney
Brett Cannon
Daniel Holth
Donald Stufft
Ian Cordasco
Ionel Cristian Mărieș
M.-A. Lemburg
Marcus Smith
Nathaniel Smith
Nick Coghlan
Paul Moore
Ralf Gommers
Robert Collins

build system abstraction PEP

Marcus Smith

Marcus Smith

Marcus Smith

Marcus Smith

Marcus Smith

tags

participants (14)