Alternatively, he could have just produced a wheel from any checkout at all if the MANIFEST.in excluded a file that would otherwise have been installed. This sort of thing is why I'm an advocate that we should only build sdists from checkouts, and wheels from sdists (at the low level anyways, even if the UI allows people to appear to create a wheel straight from a checkout). On Apr 26 2016, at 10:07 am, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:

It looks like you have published a wheel built from a dirty checkout. The 1.1.3 zip sdist produces a clean wheel, as expected. ​You should use <https://pypi.python.org/pypi/check-manifest> in your release process.

This isn't really a problem with what you're doing. Rather it's an issue with the toolchain and and open question whether or not wheels should conceptually be able to be produced from a checkout, or if they should only be produced from a sdist. Problems like this are why I advocate the Checkout -> sdist -> wheel being the only path, but others feel differently. — Donald Stufft On Apr 27 2016, at 1:38 pm, Ethan Furman <ethan@stoneleaf.us> wrote:

On 04/26/2016 07:10 AM, Donald Stufft wrote:

> Alternatively, he could have just produced a wheel from any checkout at > all if the MANIFEST.in excluded a file that would otherwise have been > installed.

Yes. My MANIFEST.in starts with an 'exclude enum/*' and then includes all files it wants.

> This sort of thing is why I'm an advocate that we should only > build sdists from checkouts, and wheels from sdists (at the low level > anyways, even if the UI allows people to appear to create a wheel > straight from a checkout).

My current process is:

python3.5 setup.py sdist --format=gztar,zip bdist_wheel upload

What should I be doing instead?

\-- ~Ethan~

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

You can try to patch around the problem (and do it in every single alternative build system once we enable support for that) or you can just fix the problem at the "protocol" level once and for all. The fundamental problem isn't with MANIFEST.in, it's that some indeterminite steps need to run to produce a sdist and unless you make sure you run every single one of those steps you can have no confidence that the results of producing a wheel from a VCS checkout will be the same as installing from a sdist or building a wheel from a sdist. — Donald Stufft On Apr 27 2016, at 2:05 pm, Daniel Holth <dholth@gmail.com> wrote:

I feel differently.

Bdist_wheel just doesn't understand MANIFEST.in . Perhaps support could be adapted from bdist_egg or other bdist implementations. Wheel doesn't do everything the setuptools or distutils implementations put into their dist commands. Even something as simple as a forced clean of the dist directory on init may fix the problem?

On Wed, Apr 27, 2016, 13:52 Donald Stufft <[donald@stufft.io](mailto:donald@stufft.io)> wrote:

...

This isn't really a problem with what you're doing. Rather it's an issue with the toolchain and and open question whether or not wheels should conceptually be able to be produced from a checkout, or if they should only be

produced from a sdist. Problems like this are why I advocate the Checkout -> sdist -> wheel being the only path, but others feel differently.

...

...

— Donald Stufft

...

...

...

...

On Apr 27 2016, at 1:38 pm, Ethan Furman <[ethan@stoneleaf.us](mailto:ethan@stoneleaf.us)> wrote:

...

...

...

On 04/26/2016 07:10 AM, Donald Stufft wrote:

...

...

...

...

> Alternatively, he could have just produced a wheel from any checkout at > all if the MANIFEST.in excluded a file that would otherwise have been > installed.

...

...

...

...

Yes. My MANIFEST.in starts with an 'exclude enum/*' and then includes all files it wants.

...

...

...

...

> This sort of thing is why I'm an advocate that we should only > build sdists from checkouts, and wheels from sdists (at the low level > anyways, even if the UI allows people to appear to create a wheel > straight from a checkout).

...

...

...

...

My current process is:

...

...

...

...

python3.5 setup.py sdist --format=gztar,zip bdist_wheel upload

...

...

...

...

What should I be doing instead?

...

...

...

...

\-- ~Ethan~

...

...

...

...

_______________________________________________ Distutils-SIG maillist - [Distutils-SIG@python.org](mailto:Distutils- SIG@python.org) <https://mail.python.org/mailman/listinfo/distutils-sig>

...

...

_______________________________________________ Distutils-SIG maillist - [Distutils-SIG@python.org](mailto:Distutils- SIG@python.org) <https://mail.python.org/mailman/listinfo/distutils-sig>

No I mean the build directory. On Wed, Apr 27, 2016, 14:11 Ethan Furman <ethan@stoneleaf.us> wrote:

On 04/27/2016 11:05 AM, Daniel Holth wrote:

...

Bdist_wheel just doesn't understand MANIFEST.in . Perhaps support could be adapted from bdist_egg or other bdist implementations. Wheel doesn't do everything the setuptools or distutils implementations put into their dist commands. Even something as simple as a forced clean of the dist directory on init may fix the problem?

Um, does that mean it would destroy my files-in-progress? 'Cause that would really [insert colloquial phrase meaning extreme irritation here].

-- ~Ethan~

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On Apr 27 2016, at 2:09 pm, Ethan Furman <ethan@stoneleaf.us> wrote:

On 04/27/2016 10:52 AM, Donald Stufft wrote:

> This isn't really a problem with what you're doing. Rather it's an issue > with the toolchain and and open question whether or not wheels should > conceptually be able to be produced from a checkout, or if they should > only be produced from a sdist. Problems like this are why I advocate the > Checkout -> sdist -> wheel being the only path, but others feel differently.

As a simple user, my feelings are that the command I used should have generated three equivalent distributions, but did not. That feels like a bug. :(

I agree with you. Others don't believe it to be a big deal.

Let me rephrase my question: what command do I use to build the wheel from the sdist I just made? For bonus points: why can't setup do that for me?

You can first build the sdist, then unpack it into a directory and then run ``setup.py bdist_wheel`` inside of that directory.

Whatever the ideal situation (not using distutils at all would be a suggestion) bdist_wheel certainly has some simple bugs that make it less fun to use sometimes. On Wed, Apr 27, 2016 at 2:14 PM Alex Grönholm <alex.gronholm@nextday.fi> wrote:

Are you seriously saying that you want your bdists to include tests, documentation etc.? Most developers would not agree with you, including yours truly.

27.04.2016, 21:10, Ethan Furman kirjoitti:

...

On 04/27/2016 10:52 AM, Donald Stufft wrote:

...

This isn't really a problem with what you're doing. Rather it's an issue with the toolchain and and open question whether or not wheels should conceptually be able to be produced from a checkout, or if they should only be produced from a sdist. Problems like this are why I advocate the Checkout -> sdist -> wheel being the only path, but others feel differently.

As a simple user, my feelings are that the command I used should have generated three equivalent distributions, but did not. That feels like a bug. :(

Let me rephrase my question: what command do I use to build the wheel from the sdist I just made? For bonus points: why can't setup do that for me?

-- ~Ethan~

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

The sdist should include all the source files, including tests and documentation. In binary distributions, however, they are just dead weight. Do you want the full documentation and test suites to be installed for every single dependency when you deploy your application? I sure don't. 27.04.2016, 21:40, Ethan Furman kirjoitti:

On 04/27/2016 11:13 AM, Alex Grönholm wrote:

...

Are you seriously saying that you want your bdists to include tests, documentation etc.?

However you and I agree or disagree on what should be in a bdist, the command I ran should have produced a bdist based on the sdists I just created in the same command.

...

Most developers would not agree with you, including yours truly.

Well, we disagree. To me, the salient difference between an sdist and a bdist is whether binary artifacts are, um, already built. I certianly enjoy having docs (so I know how to use the binaries I just installed) and tests (so I can assure myself the binaries work as advertised).

If a project is big enough I can see making separate packages for docs and/or tests, but mine are small.

And whichever way we decide to do the packaging, the tools should work for us, not us for the tools.

-- ~Ethan~

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

To answer the original question, report the bug here https://bitbucket.org/pypa/wheel On Wed, Apr 27, 2016 at 3:10 PM Ethan Furman <ethan@stoneleaf.us> wrote:

On 04/27/2016 12:00 PM, Alex Grönholm wrote:

...

The sdist should include all the source files, including tests and documentation. In binary distributions, however, they are just dead weight. Do you want the full documentation and test suites to be installed for every single dependency when you deploy your application? I sure don't.

That's makes a certain amount of sense.

It also raises some questions:

- are wheels a binary-only distribution? - if yes, we still have to use distutils to make source distributions? - if no, what are the commands to make source vs binary distributions?

- is this just a matter of a properly configured setup.py? - if yes, what is that configuration?

-- ~Ethan~ _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On 04/27/2016 12:18 PM, Daniel Holth wrote:

To answer the original question, report the bug here https://bitbucket.org/pypa/wheel

What do you know, it's already there! ;) https://bitbucket.org/pypa/wheel/issues/99/cannot-exclude-directory -- ~Ethan~

To what end? 28.04.2016, 21:21, Barry Warsaw kirjoitti:

On Apr 27, 2016, at 10:00 PM, Alex Grönholm wrote:

...

The sdist should include all the source files, including tests and documentation. In binary distributions, however, they are just dead weight. Do you want the full documentation and test suites to be installed for every single dependency when you deploy your application? I sure don't. Shouldn't that be left to the package author to decide?

-Barry

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On Wed, 27 Apr 2016 at 10:53 Donald Stufft <donald@stufft.io> wrote:

This isn't really a problem with what you're doing. Rather it's an issue with the toolchain and and open question whether or not wheels should conceptually be able to be produced from a checkout, or if they should only be produced from a sdist. Problems like this are why I advocate the Checkout -> sdist -> wheel being the only path, but others feel differently.

And Daniel Holth said he did feel differently, so obviously this hasn't moved forward in terms of finding consensus. Where does all of this sit in regards to trying to separate building from installation? From my perspective as mailing list lurker, it's sitting at an impasse as Nick hasn't made a final call nor has a BDFL delegate on the topic been chosen to settle the matter (obviously if I missed something then please let me know). Could we choose a Great Decider on this topic of build/installation separation and get final RFC/PEPs written so we can get passed this impasse and move forward? -Brett

— Donald Stufft

On Apr 27 2016, at 1:38 pm, Ethan Furman <ethan@stoneleaf.us> wrote:

...

On 04/26/2016 07:10 AM, Donald Stufft wrote:

...

Alternatively, he could have just produced a wheel from any checkout at all if the MANIFEST.in excluded a file that would otherwise have been installed.

Yes. My MANIFEST.in starts with an 'exclude enum/*' and then includes all files it wants.

...

This sort of thing is why I'm an advocate that we should only build sdists from checkouts, and wheels from sdists (at the low level anyways, even if the UI allows people to appear to create a wheel straight from a checkout).

My current process is:

python3.5 setup.py sdist --format=gztar,zip bdist_wheel upload

What should I be doing instead?

-- ~Ethan~

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On 3 May 2016 at 17:47, Donald Stufft <donald@stufft.io> wrote:

It will likely get decided as part of the build system PEP, whenever that gets picked up again.

Yes, but on 15th March (https://mail.python.org/pipermail/distutils-sig/2016-March/028457.html) Robert posted

Just to set expectations: this whole process seems stalled to me; I'm going to context switch and focus on things that can move forward. Someone please ping me when its relevant to put effort in again :).

And I think that's right. The whole build system PEP issue appears stalled from a lack of someone willing (or with the time) to make a call on the approach we take. As far as I'm aware, the decision remains with Nick. With the possible exception of Donald's proposal (which AFAIK never got formally published as a PEP) everything that can be said on the other proposals has been said, and the remaining differences are ones of choice of approach rather than anything affecting capabilities. (Robert's message at https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html summarised the state of the 3 proposals at the time). I think this is something that should be resolved - we don't appear to be gaining anything by waiting, and until we have a decision on the approach that's being taken, we aren't going to get anyone writing code for their preferred option. Nick - do you have the time to pick this up? Or does it need someone to step up as BDFL-delegate? Robert, Nathaniel, do you have time to spend on a final round of discussion on this, on the assumption that the goal will be a final decision at the end of it? Donald, do you have the time and interest to complete and publish your proposal? Paul

We did separate build from install. Now we just want to be able to build without [having to emulate] distutils; just having some dependencies installed before setup.py runs would also be a great boon. I'm reading part of this conversation as "a simple bdist_wheel bug is a reason to do a lot of work standardizing file formats" which I find unfortunate. If he is still up for it let Robert implement his own PEP as the way forward for build system abstraction. The extra PEPs are just delaying action. On Tue, May 3, 2016 at 1:11 PM Paul Moore <p.f.moore@gmail.com> wrote:

On 3 May 2016 at 17:47, Donald Stufft <donald@stufft.io> wrote:

...

It will likely get decided as part of the build system PEP, whenever that gets picked up again.

Yes, but on 15th March (https://mail.python.org/pipermail/distutils-sig/2016-March/028457.html) Robert posted

...

Just to set expectations: this whole process seems stalled to me; I'm going to context switch and focus on things that can move forward. Someone please ping me when its relevant to put effort in again :).

And I think that's right. The whole build system PEP issue appears stalled from a lack of someone willing (or with the time) to make a call on the approach we take.

As far as I'm aware, the decision remains with Nick. With the possible exception of Donald's proposal (which AFAIK never got formally published as a PEP) everything that can be said on the other proposals has been said, and the remaining differences are ones of choice of approach rather than anything affecting capabilities. (Robert's message at https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html summarised the state of the 3 proposals at the time).

I think this is something that should be resolved - we don't appear to be gaining anything by waiting, and until we have a decision on the approach that's being taken, we aren't going to get anyone writing code for their preferred option.

Nick - do you have the time to pick this up? Or does it need someone to step up as BDFL-delegate? Robert, Nathaniel, do you have time to spend on a final round of discussion on this, on the assumption that the goal will be a final decision at the end of it? Donald, do you have the time and interest to complete and publish your proposal?

Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On 3 May 2016 at 15:07, Alex Grönholm <alex.gronholm@nextday.fi> wrote:

Having setuptools process the setup requirements before parsing install requirements would be a good step forward. Had that been done before, we could've just added a setup requirement for a newer setuptools to enable PEP 508 conditional requirements.

Setuptools does process setup requirements before install requirements. The "chicken and egg" issue with setuptools is that, most of the time, setup requires are needed to calculate information that is passed into the `setup()` call itself. For example information on header files coming from the C api of `numpy` which is used to build extensions. This usually means importing code from the packages in "setup requires" before setuptools has a chance to actually look at it. A simple fix would be to allow `setup()` keywords to accept functions as well as direct values and only invoke the functions when the values are actually needed, but this idea never gained traction. Of course, even if this was implemented, it wouldn't help directly with "setup requiring" a new version of setuptools itself, unless setuptools detected this situation and reinvoked setup.py from scratch. Regards, Leo 03.05.2016, 21:04, Daniel Holth kirjoitti:

We did separate build from install. Now we just want to be able to build without [having to emulate] distutils; just having some dependencies installed before setup.py runs would also be a great boon.

I'm reading part of this conversation as "a simple bdist_wheel bug is a reason to do a lot of work standardizing file formats" which I find unfortunate.

If he is still up for it let Robert implement his own PEP as the way forward for build system abstraction. The extra PEPs are just delaying action.

On Tue, May 3, 2016 at 1:11 PM Paul Moore < <p.f.moore@gmail.com> p.f.moore@gmail.com> wrote:

...

...

It will likely get decided as part of the build system PEP, whenever

On 3 May 2016 at 17:47, Donald Stufft <donald@stufft.io> wrote: that

...

gets picked up again.

Yes, but on 15th March (https://mail.python.org/pipermail/distutils-sig/2016-March/028457.html) Robert posted

...

Just to set expectations: this whole process seems stalled to me; I'm going to context switch and focus on things that can move forward. Someone please ping me when its relevant to put effort in again :).

And I think that's right. The whole build system PEP issue appears stalled from a lack of someone willing (or with the time) to make a call on the approach we take.

As far as I'm aware, the decision remains with Nick. With the possible exception of Donald's proposal (which AFAIK never got formally published as a PEP) everything that can be said on the other proposals has been said, and the remaining differences are ones of choice of approach rather than anything affecting capabilities. (Robert's message at https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html summarised the state of the 3 proposals at the time).

I think this is something that should be resolved - we don't appear to be gaining anything by waiting, and until we have a decision on the approach that's being taken, we aren't going to get anyone writing code for their preferred option.

Nick - do you have the time to pick this up? Or does it need someone to step up as BDFL-delegate? Robert, Nathaniel, do you have time to spend on a final round of discussion on this, on the assumption that the goal will be a final decision at the end of it? Donald, do you have the time and interest to complete and publish your proposal?

Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.orghttps://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

What happened is that only a half-dozen setuptools experts (I am not one of those six people) know how to write an extended command or whatever that would actually be able to take advantage of setup requirements as implemented by setuptools. Everyone else wants to "import x" at the top of setup.py and pass arguments to the setup() function. So it would be better to have the installer make that possible. On Tue, May 3, 2016 at 2:29 PM Alex Grönholm <alex.gronholm@nextday.fi> wrote:

No, setuptools parses the install requirements before acting on setup requirements. That is the source of the problem. If setuptools only parsed and acted on setup requirements before even parsing install requirements, this wouldn't be an issue.

03.05.2016, 21:26, Leonardo Rochael Almeida kirjoitti:

On 3 May 2016 at 15:07, Alex Grönholm <alex.gronholm@nextday.fi> wrote:

...

Having setuptools process the setup requirements before parsing install requirements would be a good step forward. Had that been done before, we could've just added a setup requirement for a newer setuptools to enable PEP 508 conditional requirements.

Setuptools does process setup requirements before install requirements. The "chicken and egg" issue with setuptools is that, most of the time, setup requires are needed to calculate information that is passed into the `setup()` call itself.

For example information on header files coming from the C api of `numpy` which is used to build extensions.

This usually means importing code from the packages in "setup requires" before setuptools has a chance to actually look at it.

A simple fix would be to allow `setup()` keywords to accept functions as well as direct values and only invoke the functions when the values are actually needed, but this idea never gained traction.

Of course, even if this was implemented, it wouldn't help directly with "setup requiring" a new version of setuptools itself, unless setuptools detected this situation and reinvoked setup.py from scratch.

Regards,

Leo

03.05.2016, 21:04, Daniel Holth kirjoitti:

...

We did separate build from install. Now we just want to be able to build without [having to emulate] distutils; just having some dependencies installed before setup.py runs would also be a great boon.

I'm reading part of this conversation as "a simple bdist_wheel bug is a reason to do a lot of work standardizing file formats" which I find unfortunate.

If he is still up for it let Robert implement his own PEP as the way forward for build system abstraction. The extra PEPs are just delaying action.

On Tue, May 3, 2016 at 1:11 PM Paul Moore <p.f.moore@gmail.com> wrote:

...

...

It will likely get decided as part of the build system PEP, whenever

On 3 May 2016 at 17:47, Donald Stufft <donald@stufft.io> wrote: that

...

gets picked up again.

Yes, but on 15th March (https://mail.python.org/pipermail/distutils-sig/2016-March/028457.html) Robert posted

...

Just to set expectations: this whole process seems stalled to me; I'm going to context switch and focus on things that can move forward. Someone please ping me when its relevant to put effort in again :).

And I think that's right. The whole build system PEP issue appears stalled from a lack of someone willing (or with the time) to make a call on the approach we take.

As far as I'm aware, the decision remains with Nick. With the possible exception of Donald's proposal (which AFAIK never got formally published as a PEP) everything that can be said on the other proposals has been said, and the remaining differences are ones of choice of approach rather than anything affecting capabilities. (Robert's message at https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html summarised the state of the 3 proposals at the time).

I think this is something that should be resolved - we don't appear to be gaining anything by waiting, and until we have a decision on the approach that's being taken, we aren't going to get anyone writing code for their preferred option.

Nick - do you have the time to pick this up? Or does it need someone to step up as BDFL-delegate? Robert, Nathaniel, do you have time to spend on a final round of discussion on this, on the assumption that the goal will be a final decision at the end of it? Donald, do you have the time and interest to complete and publish your proposal?

Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.orghttps://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On May 3, 2016, at 3:28 PM, Alex Grönholm <alex.gronholm@nextday.fi> wrote:

As I pointed out in my previous post, "extended commands" are not the only use case for setup_requires -- upgrading setuptools itself would enable support for PEP 508 style conditional requirements. This currently does not work because if you have such requirements in your setup(), setuptools will fail when parsing those before it even has a chance to act on the minimum setuptools requirement specification in setup_requires.

I don’t think this would work anyways without some terrible hacks since by the time setup_requires have been installed setuptools has already been imported so the *old* version will be in sys.modules. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On May 3, 2016, at 3:35 PM, Daniel Holth <dholth@gmail.com> wrote:

Who cares exactly why it doesn't work? We know how to fix it by doing something different (put build dependencies in a static file and have them installed by pip before running setup.py).

Presumably Alex would like to know why we can’t implement his suggestion. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Tue, May 3, 2016 at 10:10 AM, Paul Moore <p.f.moore@gmail.com> wrote:

On 3 May 2016 at 17:47, Donald Stufft <donald@stufft.io> wrote:

...

It will likely get decided as part of the build system PEP, whenever that gets picked up again.

Yes, but on 15th March (https://mail.python.org/pipermail/distutils-sig/2016-March/028457.html) Robert posted

...

Just to set expectations: this whole process seems stalled to me; I'm going to context switch and focus on things that can move forward. Someone please ping me when its relevant to put effort in again :).

And I think that's right. The whole build system PEP issue appears stalled from a lack of someone willing (or with the time) to make a call on the approach we take.

No, no, Nick's not the blocker. I'm the blocker! (Sorry) Donald + Robert + I had a longish conversation about this on IRC a month ago [1]. I volunteered to summarize back to the mailing list, and then I flaked -- so I guess this is that belated email :-). Here's the tentative conclusions we came to: Blocker 1 is figuring out what to do about the sdist format. The debate is between keeping something that's basically the current format, versus somehow cleaning it up (e.g. Donald's "source wheel" ideas). To move forward: - I'll write up a PEP that attempts to just document/standardize the current de facto sdist format and send it to the mailing list (basically: filename convention, PKG-INFO + a list of which fields in PKG-INFO pypi actually cares about, presence of setup.py), and adds some sort of optional-defaulting-to-1 SDist-Version (I guess in a file called SDIST by analogy with WHEEL). And also contains a rationale section explaining the trade-offs of standardizing this versus creating a new extension.) - Donald will make his case for the new extension approach on the mailing list - We beg Nick to read over both things and make a ruling so we can move on Blocker 2 is figuring out whether the new pip <-> build system "hook" interface should be command-line based (like the current draft of PEP 516) or Python function call based (like the current draft of PEP 517). It sounds like currently Donald and I are in favor of the python hooks approach, and Robert is indifferent between them and just wants to move forward, so we agreed that unless anyone objects we'll drop the command-line approach and go ahead with refining the Python function call approach. So... if you want to object then speak up now. Then there are a bunch of details to work out about what hooks to provide exactly and what their semantics should be, but hopefully once we've settled the two issues above that will be an easier discussion to have. So yeah, basically the next step is for me [2] to write up a spec for how sdists currently (really) work. -n [1] Logs (split across two pages in the log viewer): http://chat-logs.dcpython.org/day/pypa-dev/2016-03-30#18.23.46.njs http://chat-logs.dcpython.org/day/pypa-dev/2016-03-31#00.29.32.lifeless [2] Or if someone else wants to raise their hand and volunteer I wouldn't object, obviously I am a bit swamped right now :-) -- Nathaniel J. Smith -- https://vorpus.org

On 3 May 2016 at 19:28, Nathaniel Smith <njs@pobox.com> wrote:

No, no, Nick's not the blocker. I'm the blocker! (Sorry)

Donald + Robert + I had a longish conversation about this on IRC a month ago [1]. I volunteered to summarize back to the mailing list, and then I flaked -- so I guess this is that belated email :-).

Not a problem - I'm glad my mail prompted some movement, but I completely understand that other things can get in the way.

Here's the tentative conclusions we came to:

Blocker 1 is figuring out what to do about the sdist format. The debate is between keeping something that's basically the current format, versus somehow cleaning it up (e.g. Donald's "source wheel" ideas). To move forward: - I'll write up a PEP that attempts to just document/standardize the current de facto sdist format and send it to the mailing list (basically: filename convention, PKG-INFO + a list of which fields in PKG-INFO pypi actually cares about, presence of setup.py), and adds some sort of optional-defaulting-to-1 SDist-Version (I guess in a file called SDIST by analogy with WHEEL). And also contains a rationale section explaining the trade-offs of standardizing this versus creating a new extension.) - Donald will make his case for the new extension approach on the mailing list - We beg Nick to read over both things and make a ruling so we can move on

Even though I was one who wanted a properly defined sdist format, I'm inclined to be OK with a "whatever works for now" approach, so I don't see a problem with documenting and tidying up the current de facto standard. If Donald comes up with a good proposal, that's great - but if not, we can always revisit that side of things later.

Blocker 2 is figuring out whether the new pip <-> build system "hook" interface should be command-line based (like the current draft of PEP 516) or Python function call based (like the current draft of PEP 517). It sounds like currently Donald and I are in favor of the python hooks approach, and Robert is indifferent between them and just wants to move forward, so we agreed that unless anyone objects we'll drop the command-line approach and go ahead with refining the Python function call approach. So... if you want to object then speak up now.

Cool. No objections from me.

Then there are a bunch of details to work out about what hooks to provide exactly and what their semantics should be, but hopefully once we've settled the two issues above that will be an easier discussion to have.

So yeah, basically the next step is for me [2] to write up a spec for how sdists currently (really) work. [...] [2] Or if someone else wants to raise their hand and volunteer I wouldn't object, obviously I am a bit swamped right now :-)

I don't want to volunteer to take this on completely, as I'll probably not have the time either, but if I can help in any way (research, proofreading, writing parts of the document) let me know. Thanks for the update! Paul

On 4 May 2016 at 05:10, Paul Moore <p.f.moore@gmail.com> wrote:

Nick - do you have the time to pick this up? Or does it need someone to step up as BDFL-delegate? Robert, Nathaniel, do you have time to spend on a final round of discussion on this, on the assumption that the goal will be a final decision at the end of it? Donald, do you have the time and interest to complete and publish your proposal?

I'm currently going through a redundancy process @ HPE - while I remain convinced that sorting out the issues with packaging is crucial for Python going forward, my time to work on it is now rather more limited than it was. I'm not sure where I'm going to end up, nor how much work time I'll have for this going forward: it may end up being a personal-time-only thing. Planning wise, we have to work on 'personal time only' going forward - which means I"m going to be very careful about biting off more than I can chew :) Right now, I don't see any point updating the PEP at this point. The edits I'd expect to make if the conclusions I suggested in https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html are adopted are: - change to a Python API - BFDL call on the file format and name There is no need to issue a new sdist thing, because sdists today are *already* documented across PEPs 241, 314 and 345. So - if Nick is ready to rule, there is basically about an hour of editing to switch the CLI to a Python API and update the file format bits, and then we can call it provisional and encourage implementations, update the thunk implementation and so on. If there's more debate to be had, thats fine too, but editing the PEP won't achieve that. -Rob

On May 4, 2016, at 3:39 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 4 May 2016 at 16:03, Robert Collins <robertc@robertcollins.net> wrote:

...

The edits I'd expect to make if the conclusions I suggested in https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html are adopted are:

- change to a Python API - BFDL call on the file format and name

There is no need to issue a new sdist thing, because sdists today are *already* documented across PEPs 241, 314 and 345.

I already +1'ed using a Python API, but on the file name & format side, we have the following candidates and prior art floating around:

pypa.json in PEP 516 pypackage.json in PEP 517 pydist.json in PEP 426 METADATA (Key: Value) in sdists and wheels WHEEL (Key: Value) in wheels

My impression is that we're generally agreed on wanting to move from Key:Value to JSON as the baseline for interoperability formats, so my suggestion is to use the name "pybuild.json”.

I'd actually prefer not using JSON for something that is human editable/writable because I think it's a pretty poor format for that case. It is overly restrictive in what it allows (for instance, no trailing comma gets me every time) and the lack of comments I think make it a poor format for that. I think JSON is great for what gets included *IN* a sdist or a wheel, but for what sits inside of a VCS checkout that we expect human beings to edit, I think not. I'm +1 on tying this to a new extension because I feel like it fundamentally changes what it means to be a Python sdist. It eliminates the setup.py which is just about the only thing you actually can depend on existing inside of a Python sdist and there are a lot of things out there that make the assumption that Python + ".tar.gz/.zip/etc" == has a setup.py and are going to be broken from it. A new extension means those tools will ignore it and we can bake in versioning of the format itself right from the start (even if the new format looks remarkably like the old format with a different name). I also believe that we can't provide a replacement for setup.py without either purposely declaring we no longer support something that people used from it or providing a way to support that in the new, setup.py-less format. One thing I remarked to Nataniel yesterday was that it might be a good idea to drop the build system aspect of these for right now (since afaict all the invested parties are currently overloaded and/or have a lack of time) and focus soley on the part of the proposal that enables us to get a real setup_requires that doesn't involve needing to do the tricky delayed import thing that the current implementation of setup_requires needs. That would net a pretty huge win I think since people would be able to use abstractions in their setup.py (even if they still use setuptools) through a simple ``import statement``, including the ability to specify what version of setuptools they need to build. People could still implement non setuptools build systems by mimicing the setup.py interface (though it will still continue to be less then amazingly documented/defined) but some of the Numpy folks from one of the previous threads stated that mimicing setup.py wasn't really the hard part of making numpy.distutils anyways. The benefit of that is not only a smaller chunk, but also the chunk that I think (and I could be wrong) that there's no real disagreement on about how to go about doing it (besides some bikshedding things like what the filename should be). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 4 May 2016 at 23:00, Daniel Holth <dholth@gmail.com> wrote:

+1 It would be great to start with a real setup_requires and probably would not interfere with later build system abstractions at all.

If we're going to go down that path, perhaps it might make sense to just define a standard [setup_requires] section in setup.cfg? Quite a few projects already have one of those thanks to distutiils2, d2to1 and pbr, which means the pragmatic approach here might be to ask what needs to change so the qualifier can be removed from this current observation in the PBR docs: "The setup.cfg file is an ini-like file that can mostly replace the setup.py file." The build system abstraction config could then also just be another setup.cfg section. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, May 4, 2016 at 6:28 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 4 May 2016 at 23:00, Daniel Holth <dholth@gmail.com> wrote:

...

+1 It would be great to start with a real setup_requires and probably would not interfere with later build system abstractions at all.

If we're going to go down that path, perhaps it might make sense to just define a standard [setup_requires] section in setup.cfg?

Quite a few projects already have one of those thanks to distutiils2, d2to1 and pbr, which means the pragmatic approach here might be to ask what needs to change so the qualifier can be removed from this current observation in the PBR docs: "The setup.cfg file is an ini-like file that can mostly replace the setup.py file."

The build system abstraction config could then also just be another setup.cfg section.

I'm sympathetic to the general approach, but on net I think I prefer a slightly different proposal. Downsides to just standardizing [setup_requires]: - if projects have existing ones, that's actually kinda good / kinda bad -- pip has never been respecting these before, so if we suddenly start doing that then existing stuff could break. I don't know how likely this is, but experience suggests that *something* will break and make someone angry. (I'm still blinking at getting angry complaints arguing that uploading a wheel to pypi, where before there was only an sdist, should be treated as a compatibility-breaking change that requires a new version etc.) - IMO an extremely valuable aspect of this new declarative setup-requirements thing is that it gives us an opportunity to switch to enforcing the accuracy of this metadata. Right now we have a swamp we need to drain, where there's really no way to know what environment any given setup.py needs to run. Sometimes there are setup_requires, sometimes not; if there are setup_requires then sometimes they're real, sometimes they're lies. We'd like to reach the point where for a random package on pypi, either we can tell it's a legacy package, or else we can be confident that the setup-requirements it declares are actually accurate and sufficient to building the package. This is important because it unblocks ecosystem improvements like automated wheel builders, autoconfiguring CI systems, etc. And the only way we'll get there, AFAICT, is if at some point we start enforcing build isolation, so that by default a build *can't* import anything that wasn't declared as a setup-requirement. And the only way I can see for us to make this transition smoothly -- without any horrible flag days, and with a nice carrot so projects feel like we're giving them a gift instead of punishing them -- is if we make the rule be "projects that use the declarative setup-requirements feature also get isolated build environments". (Then the messaging is: "see, this helps you check that you actually set it up right! if your new metadata works for you in testing, it'll also work for your users!) But this again would mean we can't reuse the existing [setup_requires] section. - setup.cfg is kind of a terrible format for standardizing things because the only definition of the format is "read the ConfigParser source". You cannot parse setup.cfg from a non-Python language. And the *only* benefit is that it already exists; teaching pip to read pybuild.json or pybuild.toml instead would be completely trivial. So if we aren't going to try and grandfather in existing [setup_requires] sections, then we might as well switch to a better file format at the same time -- this is not the hard part. So I like the idea of splitting out the declarative setup-requirements PEP from the new build system hook interface PEP, but I think that the way we should do this is by defining a new pybuild.whatever file like the ones in PEP 516 / PEP 517 (they're identical in this regard) that *only* has schema-version and bootstrap-requirements, and then we can add the build backend key as a second step. -n -- Nathaniel J. Smith -- https://vorpus.org

On Wed, May 4, 2016 at 10:42 PM, Robert Collins <robertc@robertcollins.net> wrote:

On 5 May 2016 at 16:22, Nathaniel Smith <njs@pobox.com> wrote: ...

...

I'm sympathetic to the general approach, but on net I think I prefer a slightly different proposal.

Downsides to just standardizing [setup_requires]:

If I write a PEP, it won't be standardising setup_requires, it will be standardising bootstrap requirements. The distinction is nuance but critical:

- we don't expect setuptools to ever need to honour it (of course, it could, should it choose to, and ease of adoption may suggest that it should) - as a new feature, making it opt in allows folk to adopt it when they are ready; if it was literally just a new home for setup_requires, we may see build systems auto-populating it, and the potential swamp you describe below would then apply.

The main argument I was making there was that it needs to be a new opt-in thing, so if we're agreed on that then great :-).

...

- if projects have existing ones, that's actually kinda good / kinda bad -- pip has never been respecting these before, so if we suddenly start doing that then existing stuff could break. I don't know how likely this is, but experience suggests that *something* will break and make someone angry. (I'm still blinking at getting angry complaints arguing that uploading a wheel to pypi, where before there was only an sdist, should be treated as a compatibility-breaking change that requires a new version etc.)

Yes, things will break: anyone using this will need a new pip, by definition. Not everyone will be willing to wait 10 years before using it :).

Just to clarify (since we seem to agree): I meant that if pip starts interpreting an existing setup.cfg thing, then the new-pip/old-package situation could break, which would be bad.

...

- IMO an extremely valuable aspect of this new declarative setup-requirements thing is that it gives us an opportunity to switch to enforcing the accuracy of this metadata. Right now we have a swamp we need to drain, where there's really no way to know what environment any given setup.py needs to run. Sometimes there are setup_requires, sometimes not; if there are setup_requires then sometimes they're

Huh? I've not encountered any of this, ever. I'd love some examples to go look at. The only issue I've ever had with setup_requires is the easy_install stuff it ties into.

I don't think I've ever seen a package that had accurate setup_requires (outside the trivial case of packages where setup_requires=[] is accurate). Scientific packages in particular universally have undeclared setup requirements.

...

...

instead of punishing them -- is if we make the rule be "projects that use the declarative setup-requirements feature also get isolated build environments". (Then the messaging is: "see, this helps you check that you actually set it up right! if your new metadata works for you in testing, it'll also work for your users!) But this again would mean we can't reuse the existing [setup_requires] section.

I'm very much against forcing isolated build environments as part of this effort. I get where you are coming from, but really it conflates two entirely separate things, and you'll *utterly* break building anything with dependencies that are e.g. SWIG based unless you increase the size of the PEP by about 10-fold. (Thats not hyperbole, I think).

Okay, now's my turn to be baffled :-). I literally have no idea what you're talking about here. What would this 10x longer PEP be talking about? Why would this be needed?

Working around that will require a bunch of UX work, and its transitive: you have to expose how-to-workaround-the-fact-that-all-our-deps-are-not-installable all the way up the chain. That's probably a good thing to do (e.g. see bindep, or the aborted callout-to-system-packages we proposed after PyCon AU last year), but tying these two things together is not necessary, though I can certainly see the appeal; the main impact I see is that it will just impede adoption.

What are these things that aren't pip-installable and why isn't the solution to fix that? I definitely don't want to break things that work now, but providing new features that incentivize folks to clean up their stuff is a good thing, surely? Yeah, it means that the bootstrap-requirements stuff will take some time and cleanup to spread, but that's life. We've spent a huge amount of effort on reaching the point where pretty much everything *can* be made pip installable. Heck, *PyQt5*, which is my personal benchmark for a probably-totally-unpackageable package, announced last week that they now have binary wheels on pypi for all of Win/Mac/Linux: https://pypi.python.org/pypi/PyQt5/5.6 I want to work towards a world where this stuff just works, not keep holding the entire ecosystem back with compromise hacks to work around a minority of broken packages.

The reality, AFAICT, is that most projects with undeclared build deps today get corrected fairly quickly: a bug is filed, folk fix it, and we move on. A robotic system that isolates everything such that folk *cannot* fix it is much less usable, and I'm very much in favour of pragmatism here.

Again, in my world ~100% of packages have undeclared build deps...

...

- setup.cfg is kind of a terrible format for standardizing things because the only definition of the format is "read the ConfigParser source". You cannot parse setup.cfg from a non-Python language. And the *only* benefit is that it already exists; teaching pip to read pybuild.json or pybuild.toml instead would be completely trivial. So if we aren't going to try and grandfather in existing [setup_requires] sections, then we might as well switch to a better file format at the same time -- this is not the hard part.

The patch to read a single list-valued key out of setup.cfg is trivial and shallow. We've not managed to settle on consensus on a file format choice in a year of debate. I hold zero confidence we will going forward either. If the BDFL delegate makes a call - fine. I read Nick's earlier email in the thread as such a thing TBH :).

Oh sure, I think everyone agrees that the file format choice is not a make-or-break decision and that ultimately this is a good case for of BDFL-style pronouncement rather than endless consensus seeking. But I'm still allowed to make technical arguments for why I think the BDFL-delegate should pronounce one way or another :-).

...

So I like the idea of splitting out the declarative setup-requirements PEP from the new build system hook interface PEP, but I think that the way we should do this is by defining a new pybuild.whatever file like the ones in PEP 516 / PEP 517 (they're identical in this regard) that *only* has schema-version and bootstrap-requirements, and then we can add the build backend key as a second step.

I think the risk there is that that presumes the answer, and without the abstract build effort actually moving forward we may be years before we actually see the file being used for anything else. If ever.

I see the risk, but I think it's minimal. We're not presuming anything about the answer except that somehow it will involve adding some kind of static metadata, which seems very safe. -n -- Nathaniel J. Smith -- https://vorpus.org

On 5 May 2016 at 07:57, Robert Collins <robertc@robertcollins.net> wrote:

We've a history in this group of biting off too much and things not getting executed. We're *still* in the final phases of deploying PEP-508, and it was conceptually trivial. I'm not arguing that we shouldn't make things better, I'm arguing that tying two separate things together because we *can* seems, based on the historical record, to be unwise.

This is a very good point, and ties in nicely with Nick's comment about taking small steps to make things better than they currently are. On that basis, I'd be +1 on a simple proposal to add a new "install this stuff before we do the build" capability that sits in setup.cfg. Let's keep build isolation off the table for now. There's probably enough substantive detail (I'll do my best to avoid bikeshedding over trivia :-)) to thrash out in that simple proposal. For example, if package foo specifies that it needs a new version of setuptools to build, is it OK for "pip install foo" to automatically upgrade setuptools, or should it fail with an error "your setuptools is too old"? If it does auto-upgrade, then if the build of foo fails, is it OK that we won't be able to revert the upgrade of setuptools? How should we handle cases where a package specifies that the it needs an *older* version of setuptools? I'd expect we simply bail and report an error for that one - it should never really happen, so why waste time on "clever" solutions? Anyway, we can have these sorts of debate when we get down to details. Paul

On 5 May 2016 at 10:10, Nathaniel Smith <njs@pobox.com> wrote:

...The main thing I want to point out though, is that all of these problems you're raising are complications caused entirely by wanting to avoid build isolation in the name of simplicity. If each package gets its own isolated build environment, then it can depend on whatever it wants without any danger of collision with the ambient environment.

Understood. But that doesn't mean they can *only* be solved by build isolation. One relatively naive approach might be: 1. Collect build requirements. If they are satisfied, fine, move on to build. 2. If the only unsatisfied requirements are new installs, install them and move on to build. 3. If there are unsatisfied requirements needing an upgrade, do the upgrades (possibly require a flag for the pip invocation to allow silent upgrades), and move on to build. 4. Anything more complicated (downgrades, conflicts) report the issue and stop to let the user resolve it manually. Sure, build isolation might be better (probably is, IMO, but that's clearly a topic for some serious debate) but the whole point here is to do something incremental, and "good enough to make things better", not to stall on a completely perfect solution.

Premise 1: Without build isolation enabled by default, then in practice everyone will putter along putting up with broken builds all the time. It's *incredibly* easy to forget to declare a build dependency, it's the kind of mistake that every new user makes, and experienced users too.

That's certainly possible, but "some missing dependencies" is still better than "no way to specify dependencies at all". And all it needs is an end user to hit the problem, raise an issue, and the dependency gets added and we're closer to perfection. Why is that worse than the current status quo?

Premise 2: We can either enable build isolation together with the new static bootstrap requirements, or we can never enable build isolation at all, ever.

I have no idea why you feel that's the case. Why can't we add build isolation later? You clearly have more experience in this area than I do, so I'm genuinely trying to see what I'm missing here. You mention "we can't add it later without breaking everything". But the same applies now surely? And why can't we introduce the feature gradually - initially as opt-in via an "--isolated-build" flag, then in a later release change the default to "--isolated-build" with "--no-isolated-build" as a fallback for people who haven't fixed their builds yet, and ultimately removing "--no-isolated-build" and making isolated builds the only option? That's the standard approach we use, and I see no reason it wouldn't work here for introducing isolated builds at *any* time. Paul

On 5 May 2016 at 22:36, Daniel Holth <dholth@gmail.com> wrote:

Pedantic note

setup_requires is a setuptools parameter used to install packages after setup() is called. Even though very many people expect or want those packages to be installed before setup.py executes. I think it is reasonable to call the new feature setup_requires but some prefer to eliminate uncertainty by calling it bootstrap_requires.

The main advantage of a new feature name is that when someone searches the internet for "python bootstrap_requires", they won't find a decade+ worth of inapplicable documentation of the setuptools feature :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 5 May 2016 at 13:36, Daniel Holth <dholth@gmail.com> wrote:

Here's the kind of thing that you should expect. Someone will write

setup.cfg:

[bootstrap_requires] pbr

pip installs pbr in a directory that is added to PYTHONPATH for that build.

Ah, so we don't install into the environment, just to a temporary location which we add to PYTHONPATH. That is indeed a much better approach than what I assumed, as it avoids any messing with the user's actual environment. (We may need to be a bit more clever than just adding the temporary directory to PYTHONPATH, to ensure that what we install *overrides* anything in the user's environment, such as a newer version of setuptools, but that's just implementation details). Paul

On Thu, May 5, 2016 at 3:18 PM, Chris Barker <chris.barker@noaa.gov> wrote:

On Thu, May 5, 2016 at 2:10 AM, Nathaniel Smith <njs@pobox.com> wrote:

...

...The main thing I want to point out though, is that all of these problems you're raising are complications caused entirely by wanting to avoid build isolation in the name of simplicity. If each package gets its own isolated build environment, then it can depend on whatever it wants without any danger of collision with the ambient environment.

You do know that we're on our way to re-writing conda, now, don't you? :-)

I think we need to be careful of scope-creep...

conda did not invent the idea of creating separate python environments for different tasks :-) -n -- Nathaniel J. Smith -- https://vorpus.org

On Thu, May 5, 2016 at 4:32 PM, Nathaniel Smith <njs@pobox.com> wrote:

...

You do know that we're on our way to re-writing conda, now, don't you? :-)

I think we need to be careful of scope-creep...

conda did not invent the idea of creating separate python environments for different tasks :-)

I'm not suggesting conda invented anything -- I'm suggesting it has implemented many of the things being talked about here -- the truth is conda was designed to solve exactly the problems that scientific python packages had that pip+wheel+setuptools do not currently solve. So my point is about scope-creep -- if you want the PyPa tools to solve all these problems, then you are re-inventing conda -- better to simply adopt conda (or fork it and fix issues that I'm sure are there....) On the other hand, improving the PyPa tools while maintaining their current scope is a lovely idea -- but that means leaving isolation of build environments etc, to external tools: Docker, VMs, conda...... Actually, conda has a good lesson here: conda is about packaging, NOT building -- I was quite disappointed that conda provided NO support for cross platform building at all -- but after using it a bit I realized that that was a great decision -- if you want to support a wide variety of packages, you really need to let the package authors use whatever the heck build system they want -- you really don't want to have to implement (or even debug) everyone else's builds. And IIUC, that is the direction we are trying to go with pip now -- making pip+wheel build-system independent -- good goal. Which actually give me an idea: it seems we are very bogged down in struggles with keeping backward compatibility, and coming up with a API for an arbitrary build system. Maybe we can take a lesson from conda and essentially punt: conda works by reading a meta.yaml -- a purely declarative package configuration. The actual building is done by calling a build script -- conda itself does not need to know or care what the heck is in that build script -- all it needs to know is that it can be run. Why not keep the API that simple for pip? we could do total backward compatible, and simply say: pip will call "python setup.py install". And that's it, end of story -- individual packages could do ANYTHING in that setup.py script -- current packages would be using setuptools, but pip wouldn't need to care at all what was in there -- only know that it would install the package. And, I suppose, a setup.py bdist_wheel for building wheels, and setup.py sdist for creating source distributions), though I'm not sure that those need to be a standardized API -- or at least not the SAME standardized API. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Fri, May 6, 2016 at 9:39 AM, Donald Stufft <donald@stufft.io> wrote:

On May 6, 2016, at 11:54 AM, Chris Barker <chris.barker@noaa.gov> wrote:

So my point is about scope-creep -- if you want the PyPa tools to solve all these problems, then you are re-inventing conda -- better to simply adopt conda (or fork it and fix issues that I'm sure are there….)

Adopting Conda is unlikely to ever happen for the defaulting tooling.

I didn't put the emphasis right in that sentence -- I was not actually suggesting that conda be adopted -- what I was suggesting was the we DON'T expand the scope of pip et al to be able to do everything that conda does. That being said, I think there are some misunderstanding here that may be relevant to this discussion:

The problems that the default tooling attempt to solve are significantly harder than the problems that Conda attempts to solve

kinda-sorta -- conda is a packaging system. period, end of story (actually it is an isolated environment system, too -- is that packaging? )-- it is NOT a build system. Right now, you can invoke pip to: * Find and install a binary package (wheel) -- this conda does * Find, download, build and install install a source package -- nope, conda doesn't do anything like that. * Build and install a local package from source * Install a local package from source in "develop mode" (editable mde) -- actually conda does have that -- though I'm not sure who well it works, of it it's python specific. But the fact that this all is (apparently) done by one tool is actually a problem -- we have a tangled mess of binary and source and building and installing, and it isn't obvious to the user what they are getting ) often they don't care -- as long as it works :-) ) And, under the hood, it's all driven by setuptools, which also has duplicate and slightly different functionality that can be accidentally invoked (fun with easy_install!). So I thought the goal now was to clean up an untangle this mess, yes? and switching to it would be a regression.

only in the sense the switching to pip (and getting rid of setuptools) would be a regression....

The primary benefit of Conda’s packages over Wheels are that they have a curated repository with people who are ensuring that things build correctly and since they don’t rely on authors to do it correctly, they don’t have to wait for, or convince authors to do this fresh new thing.

That simply is not true. Let's not mix social issues from technical ones. Continuum set out to create a curated set of packages, yes -- but they didn't put the enormous amount of work into conda for no reason -- they did it because pip + wheel didn't meet their needs (and it doesn't meet a lot of our needs, either). And, as I understand it, they came to the Python community, and said something along the lines of "the packaging tooling doesn't meet the needs of the scientific community, can we contribute changes, improvements" -- and were told something along the lines of: "We don't want the standard tooling to solve those problems -- go make what you need, and then maybe we'll talk" (I can dig out the message from Travis Oliphant about this if you like) And they did go out and make something else, and they have released it as an open source project, and a number of folks are adopting it to solve their problems, too.

The problem is, none of those benefits are something that would apply if we decided to throw away the 588,074 files that are currently able to be installed on PyPI.

conda aside, I think there is a real hang-up here. yes, of course you don't want to throw away all that -- we absolutely need to maintain the current packaging, building structure to support existing projects. But that doesn't mean something new and better can't be built to be used in parallel -- as folks move forward with pacakge development, they can adopt the new system or they can keep using teh same old setup.py they have. MAybe decades later something will get depreciated. But this approach is what got us in the ugly mess to begin with. Us deciding that Conda is the thing to use isn’t going to magically create

an army of volunteers to go through and take all 80,000 packages on PyPI and ensure that we get a correctly compiled package on every platform for each version. If we could do that we could just convince everyone to go out and build binary packages, we could just do that with Wheels without requiring forklifting an entire ecosystem.

Again, conda packages and wheels are not the same thing -- there are technical differences, and those differences really are why continuum uses it for Anaconda, and why a lot of folks are adopting it for community led efforts, like conda-forge. And there are a LOT of packages on conda-forge despite it being very new and only a small handful of people contributing. https://conda-forge.github.io/feedstocks.html

While wheels are optimized for the pure Python case, there is nothing preventing them from being used to install anything else (Erlang or R or even Python itself).

OK, wheel is essentially "just" a archive -- so are conda packages -- they are the same in that regard. but the tools that build and install wheels aren't set up to support non-python. (and virtualenv vs. conda environments, etc)

The pynativelib stuff is proof of the ability to do just that— and in distutils-sig land we tend to care a lot more about how these things will affect our downstream consumers like Debian and such than Conda needs to (or should need to!).

if you split out building from package management, then the package management has no impact on Debian et al. They use their package management, and python's build system -- isn't that how it's done now? Now, this isn’t to say that Conda is bad or anything, but it’s use as a

replacement for the current ecosystem is about as interesting as suggesting we all adopt RPM, or apt-get, or Chocolatey, or Ports, or Pacman or whatever flavor of downstream you wish to insert here.

Again -- not suggesting that -- though I honestly think it's more driven by "not invented here" than real technical reasons. A couple years ago, the Python community very clearly said they didn't want to support non-python libs, etc, etc. Now apparetnly we've changed our minds, but want to keep down the road we started out on... Now I see scope creep pulling in -- we really want the "standard" python packaging system to be able to support complex packages like numpy, scipy, etc. So If the community has changed its mind, and wants to support that, then it makes sense to step back a second and ask yourselves what the architecture of the future system should look like? What do you want to support? If nothing else, there have got to be lessons that can be learned from conda (and of course, rpm, deb etc, too) BTW, if PyPa does eventually get around to re-inventing conda -- great! I was pretty unhappy when conda came out -- I was putting in real efforts to try to get wheels built for various things on OS-X, and looking at how to support my users with pip, pypi, and wheel. But in the end, it turned out that conda made a whole lot of this a whole lot easier - it really did (some of that technical, some of it social, but certainly mostly technical). So I've been putting efforts into using conda for my stuff, and supporting the community when I can. But, of course, it turns out not all of my users want to use conda, so now I need to put efforts into building wheels of my stuff, too -- and I am NOT looking forward to that. So one system to rule them all would be great. So what is the scope of all this? Do you want to support non-python C libs as first class citizens? (and Fortran, and...) Do you want to support isolated environments (for not just python) Do you want to support non-python systems as well -- R, perl, Julia, who knows? (I'm sure there are others....) If that's the long view, great -- but start thinking about it now. OK -- didn't intend that to be so long! Carry on, I'll shut up now. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Sat, May 7, 2016 at 6:51 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 7 May 2016 01:55, "Chris Barker" <chris.barker@noaa.gov> wrote:

...

So my point is about scope-creep -- if you want the PyPa tools to solve all these problems, then you are re-inventing conda -- better to simply adopt conda (or fork it and fix issues that I'm sure are there....)

conda doesn't solve these problems either - it solves the *end user* problem for data analysts (install the Python library they want to use),

I really need to make this clear -- there is NOTHING "data analyst" specific about these problems -- they do come up more in the computational programming domain, but there are any number of other application that have the same problems (pyQT has come up in this conversation, yes?) -- and we're finding conda to be a good solution for our web development needs, too -- a conda environment is kinda like a lighter-weight, platform independent docker container. And of course, there is more an more data analysis going on behind web services these days too -- any python developer is going to run into these issues eventually...

by ignoring the system integrator one (interoperate with the system integrator's existing platform level package management systems, of which we all tend to have our own with no plans for consolidation)

That's by design, though - conda was specifically created as a language independent cross-platform platform, not as a cross-platform plugin management system for Python runtimes.

exactly!

For a long time I was convinced the existence of conda and Linux containers as end user tools meant we wouldn't ever need to solve these problems at the language runtime layer, but it's since become clear to me that there's significant value in solving these problems in a way that doesn't care about how you obtained your Python runtime.

yup -- that would be the scope creep I'm referring too :-) But now I'm confused about what problems we're talking about. From above: """ interoperate with the system integrator's existing platform level package management systems """ you mean rpm, .deb, homebrew, etc? distutils did have (does it still) bdist_rpm (and, for that matter bdist_wininst) -- which I see as an attempt to inter-operate with the system integrator's platform. But I think it turns out that this is a hopeless task -- there are just too many different systems to consider and maintain -- much better to let the external systems handle that. And it got really ugly when you wanted to mingle virtualenv with rpm, etc.... The "solution" to that is to actually not do it -- use Docker or Conda if you want an isolated environment. What all this means to me is that we really do need to keep the building separate from the packaging -- so that the building tools can be used / supported by the systems integrator. But people shouldn't be using rpm to manage their system, and expect to install binary wheels... The trick is that the pip/wheel system is really handy -- certainly for Windows and OS-X that don't provide a standard system package manager. It is nice to be able to "just pip install" various packages. But the problem is that it is a half way solution -- it only works well for python-centered packages -- pure python is trivial of course, but custom written C extensions work great too -- but as soon as you get external dependencies, it all goes to heck. We can improve that -- defining the manylinux platform is a great step, but to really get there, there needs to be a way to handle third party libs (which we could probably do with the existing tools by establishing conventions), but then you get to non-python support tools: Fortran compilers, other language run times, all sorts of stuff. Oh, and then we want isolated environments, and they need to isolate these other non-python tools, so .... I think it's WAY beyond the current PEP under discussion, but we really should map out a "where we'd like to get" plan. "significant value in solving these problems in a way that doesn't care about how you obtained your Python runtime." I don't get this -- I don't think it's possible to not care about the python run-time -- we've only gotten this far because most folks have more or less standardized on the python.org builds as the standard python build. Even when you get your python elsewhere (Anaconda, for instance) they take pains to keep it compatible. I think we should count on python itself being managed by the same package manager as the packages -- or at least conforming to a standard (which I think we have with teh ABI tags, yes?) BTW, I only recently learned that there are more "other" python builds out there than I thought. Apparently the Visual Effect folks are on the habit of building their own python2.7 on Windows, so they can use a newer MS compiler, and then build their extensions to be compatible (all so you can use modern C++ in the extension code). I don't know that they are careful about setting the ABI tags, but they could be.

For a long time I was convinced the existence of conda and Linux containers as end user tools meant we wouldn't ever need to solve these problems at the language runtime layer

to an extent that is true -- but it does mean a proliferation of options, and as I think you pointed out earlier, that sucks because you need someone to support all those options -- for instance, most conda packages are built by third parties now - not the original package author. This is (very slowly) starting to change (Ned Batchelder recently asked about building conda packages for coverage...). and of course, the same is true with homebrew, and rpm, and deb, and.... So yeah, it would be nice to have ONE thing that python package developers need to do to make their work available to everyone. So I'll leave off with this: If there is even going to be one way for python package developers to provide binary packages for their users -- that system will need to support all (most) of what conda supports. Every feature in conda was added because it was needed. And THAT's what I've been meaning every time i say "reimpliment conda" -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Sun, May 8, 2016 at 5:31 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

any python developer is going to run into these issues eventually...

Aye, I know - conda was one of the systems I evaluated as a possible general purpose tool for user level package management in Fedora and derivatives (see

https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManag... for details).

Hence my comment about conda not currently solving *system integrator* problems in the general case - it's fine as a platform for running software *on top of* Fedora and for DIY integration within an organisation, but as a tool for helping to *build Fedora*, it turned out to introduce a whole slew of new problems

Sure -- I don't think it was ever intended to support rpm et. all -- it's to be used INSTEAD of rpm et. all -- for the most part, rpm has solved the problems that conda is trying to solve -- but only for rpm-based Linux systems. And I'm going to guess that Continuum didn't want to: build packages for rpm systems (which ones?) build packages for deb-based systems (which ones?) build packages for gentoo build packages for arch.. ..... build packages for homebrew build packages for cygwin build packages for Windows.. OOPS, there IS not package manager for Windows!! And, AFAICT, none of those package management systems support "environments", either. Clearly -- a tool like conda was required to meet Continuum's goals -- and those goals are very much the same as PyPa's goals, actually. (except the curated collection of packages part, but conda itself is not about the Curation...) However, the kinds of enhancements we're now considering upstream in

pip should improve things for conda users as well - just as some folks in Fedora are exploring the feasibility of automatically rebuilding the whole of PyPI as RPMs

yes -- that's a good analogy -- for python packages, conda relies entirely on distutils/setuptools/pip -- so yes, making those tools better and more flexible is great. but I'm still confused about what "the kinds of enhancements we're now considering upstream in pip" are. Here are a few: More flexibility about the build system used Easier to get metadata without actually initiating a build These are great! But I started this whole line of conversation because it seemed that there was desire for: Ability to isolate the build environment. Ability to better handle/manage non-python dependencies These are what I was referring to as mission-creep, and that overlap with conda (and other tools).

(twice, actually, anchored on /usr/bin/python and /usr/bin/python3), it should eventually be feasible to have the upstream->conda pipeline fully automated as well.

yeah -- there's been talk for ages of automatically building conda packages (on the fly, maybe) from PyPi packages. But currently on conda-forge we've decided to NOT try to do that -- it's turned out in practice that enough pypi packages end up needing some hand-tweaking to build. So teh planned workflow is now: Auto-build a conda build script for a PyPi package Test it Tweak it as required Add it to conda-forge. Then -- *maybe* write a tool that auto-updates the PyPi based packages in a chron job or whatever. So not quite a automated conda-PyPi bridge, but not a bad start. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Wed, May 4, 2016 at 11:57 PM, Robert Collins <robertc@robertcollins.net> wrote:

On 5 May 2016 at 18:32, Nathaniel Smith <njs@pobox.com> wrote:

...

On Wed, May 4, 2016 at 10:42 PM, Robert Collins ...

...

Yes, things will break: anyone using this will need a new pip, by definition. Not everyone will be willing to wait 10 years before using it :).

Just to clarify (since we seem to agree): I meant that if pip starts interpreting an existing setup.cfg thing, then the new-pip/old-package situation could break, which would be bad.

No. Old pip new package will break, new pip old package is entirely safe AFAICT.

We're talking past each other... I'm saying, *if* pip started reinterpreting some existing thing as indicating setup-requirements, *then* things would break. You're saying, pip isn't going to do that, so they won't. So we're good :-)

...

...

...

- IMO an extremely valuable aspect of this new declarative setup-requirements thing is that it gives us an opportunity to switch to enforcing the accuracy of this metadata. Right now we have a swamp we need to drain, where there's really no way to know what environment any given setup.py needs to run. Sometimes there are setup_requires, sometimes not; if there are setup_requires then sometimes they're

Huh? I've not encountered any of this, ever. I'd love some examples to go look at. The only issue I've ever had with setup_requires is the easy_install stuff it ties into.

I don't think I've ever seen a package that had accurate setup_requires (outside the trivial case of packages where setup_requires=[] is accurate). Scientific packages in particular universally have undeclared setup requirements.

Are those requirements pip installable?

Either they are, or they will be soon.

..

...

...

I'm very much against forcing isolated build environments as part of this effort. I get where you are coming from, but really it conflates two entirely separate things, and you'll *utterly* break building anything with dependencies that are e.g. SWIG based unless you increase the size of the PEP by about 10-fold. (Thats not hyperbole, I think).

Okay, now's my turn to be baffled :-). I literally have no idea what you're talking about here. What would this 10x longer PEP be talking about? Why would this be needed?

Take an i386 linux machine, and build something needing pyqt5 on it :). Currently, you apt-get/yum/dnf/etc install python-pyqt5, then run pip install. If using a virtualenv you enable system site packages.

When you introduce isolation, the build will only have the standard library + whatever is declared as a dep: and pyqt5 has no source on PyPI.

So the 10x thing is defining how the thing doing the isolation (e.g. pip) should handle things that can't be installed but are already available on the system.

And that has to tunnel all the way out to the user, because its context specific, its not an attribute of the dependencies per se (since new releases can add or remove this situation), nor of the consuming thing (same reason).

# User experience today on i386 $ pip install foo <... error: missing pyqt5 ...> $ apt install python-pyqt5 $ pip install foo # User experience with build isolation on i386 $ pip install foo <... error: missing pyqt5 ...> $ apt install python-pyqt5 $ pip install --no-isolated-environment foo It'd even be straightforward for pip to notice that the requirement that it failed to satisfy is already satisfied by the ambient environment, and suggest --no-isolated-environment as a solution.

Ultimately, its not even an interopability question: pip could do isolated builds now, if it chose, and it has no ramifications as far as PEPs etc are concerned.

That's not true. In fact, it seems dangerously naive :-/ If pip just went ahead and flipped a switch to start doing isolated builds now, then everything would burst into flame and there would be a howling mob in the bug tracker. Sure, there's no PEP saying we *can't* do that, but in practice it's utterly impossible. If we roll out this feature without build isolation, then next year we'll still be in the exact same situation we are today -- we'll have the theoretical capability of enabling build isolation, but everything would break, so in practice, we won't be able to. The reason I'm being so intense about this is that AFAICT these are all true: Premise 1: Without build isolation enabled by default, then in practice everyone will putter along putting up with broken builds all the time. It's *incredibly* easy to forget to declare a build dependency, it's the kind of mistake that every new user makes, and experienced users too. Premise 2: We can either enable build isolation together with the new static bootstrap requirements, or we can never enable build isolation at all, ever. Conclusion: If we want to ever reach a state where builds are reliable, we need to tie build isolation to the new static metadata. If you have some clever plan for how we could practically transition to build isolation without having them opt-in via a new feature, then that would be an interesting counter-argument; or an alternative plan for how to reach a point where build requirements are accurate without being enforced; or ...?

...

...

What are these things that aren't pip-installable and why isn't the solution to fix that? I definitely don't want to break things that work now, but providing new features that incentivize folks to clean up their stuff is a good thing, surely? Yeah, it means that the bootstrap-requirements stuff will take some time and cleanup to spread, but that's life.

We've a history in this group of biting off too much and things not getting executed. We're *still* in the final phases of deploying PEP-508, and it was conceptually trivial. I'm not arguing that we shouldn't make things better, I'm arguing that tying two separate things together because we *can* seems, based on the historical record, to be unwise.

My argument is not that we can, it's that we have to :-).

...

We've spent a huge amount of effort on reaching the point where pretty much everything *can* be made pip installable. Heck, *PyQt5*, which is my personal benchmark for a probably-totally-unpackageable package, announced last week that they now have binary wheels on pypi for all of Win/Mac/Linux:

https://pypi.python.org/pypi/PyQt5/5.6

I want to work towards a world where this stuff just works, not keep holding the entire ecosystem back with compromise hacks to work around a minority of broken packages.

Sure, but the underlying problem here is that manylinux is a 90% solve: its great for the common cases but it doesn't actually solve the actual baseline problem: we can't represent the actual system dependencies needed to rebuild many Python packages.

pyqt5 not having i386 is just a trivial egregious case. ARM32 and 64 is going to be super common, Power8 another one, let alone less common but still extant and used architectures like PPC, itanium, or new ones like x86_32 [If I remember the abbreviation right - no, its not i386].

(it's x32) manylinux is helpful here, but it's not necessary -- build isolation just requires that the dependencies be pip installable, could be from source or whatever. In practice the wheel cache will kick in and handle most of the work.

Solve that underlying problem - great, then isolation becomes an optimisation question for things without manylinux wheels. But if we don't solve it then isolation becomes a 'Can build X at all' question, which is qualitatively different.

More like 'can build X at all (without adding one command line option)'. And even this is only if you're in some kind of environment that X upstream doesn't support -- no developer is going to make a release of X with build isolation turned on unless build isolation works on the platforms they care about.

I'm all for solving the underlying problem, but not at the cost of *not solving* the 'any easy-install is triggered' problem for another X months while that work takes place.

...

...

The reality, AFAICT, is that most projects with undeclared build deps today get corrected fairly quickly: a bug is filed, folk fix it, and we move on. A robotic system that isolates everything such that folk *cannot* fix it is much less usable, and I'm very much in favour of pragmatism here.

Again, in my world ~100% of packages have undeclared build deps...

So - put a patch forward to pip to do isolated builds. If/when bug reports come in, we can solve them there. There's no standards body work involved in that as far as I can see....

See above... -n -- Nathaniel J. Smith -- https://vorpus.org

On Thu, May 5, 2016 at 2:47 AM, Nathaniel Smith <njs@pobox.com> wrote:

...

When you introduce isolation, the build will only have the standard library + whatever is declared as a dep: and pyqt5 has no source on PyPI.

so build isolation isolates you from arbitrary system libs? now you NEED to ship all your non-trivial dependent libs. which I guess is the point, but again, we're re-implementing conda here. And the only time where it makes sense to force this build requirement is when you are building binary wheels to put up on PyPi. So maybe part of bdist_wheel?? OH -- and you need to make sure there is an isolated test environment too -- which is a bit of pain, particularly when the tests aren't installed as part of the package. And, of course this means a version of virtualenv that isolates you from system libs -- that might be quite hard to do, actually. and again, conda :-) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

From my point of view mandatory build isolation will make building thinks slow and bad, besides being totally unrelated to the idea of a working bootstrap requirements feature.

On Thu, May 5, 2016 at 9:37 PM Robert Collins <robertc@robertcollins.net> wrote:

On 5 May 2016 at 21:47, Nathaniel Smith <njs@pobox.com> wrote:

...

On Wed, May 4, 2016 at 11:57 PM, Robert Collins ...

...

...

I don't think I've ever seen a package that had accurate setup_requires (outside the trivial case of packages where setup_requires=[] is accurate). Scientific packages in particular universally have undeclared setup requirements.

Are those requirements pip installable?

Either they are, or they will be soon.

Thats good. It occurs to me that scientific builds may be univerally broken because folk want to avoid easy-install and the high cost of double builds of things. E.g. adding bootstrap_requires will let folk fix things?

But the main question is : why are these packages staying inaccurate? Even in the absence of a systematic solution I'd expect bug reports and pull requests to converge on good dependencies.

...

...

...

So the 10x thing is defining how the thing doing the isolation (e.g. pip) should handle things that can't be installed but are already available on the system.

And that has to tunnel all the way out to the user, because its context specific, its not an attribute of the dependencies per se (since new releases can add or remove this situation), nor of the consuming thing (same reason).

# User experience today on i386 $ pip install foo <... error: missing pyqt5 ...> $ apt install python-pyqt5 $ pip install foo

# User experience with build isolation on i386 $ pip install foo <... error: missing pyqt5 ...> $ apt install python-pyqt5 $ pip install --no-isolated-environment foo

So thats one way that isolation can be controlled yes.

...

It'd even be straightforward for pip to notice that the requirement that it failed to satisfy is already satisfied by the ambient environment, and suggest --no-isolated-environment as a solution.

...

Ultimately, its not even an interopability question: pip could do isolated builds now, if it chose, and it has no ramifications as far as PEPs etc are concerned.

That's not true. In fact, it seems dangerously naive :-/

If pip just went ahead and flipped a switch to start doing isolated builds now, then everything would burst into flame and there would be a howling mob in the bug tracker. Sure, there's no PEP saying we *can't* do that, but in practice it's utterly impossible.

We've done similarly omg it could be bad changes before - e.g. introducing wheel caching. A couple of iterations later and we're in good shape. Paul seems to think we could introduce it gracefully - opt in, then default with fallback, then solely opt-out.

...

If we roll out this feature without build isolation, then next year we'll still be in the exact same situation we are today -- we'll have the theoretical capability of enabling build isolation, but everything would break, so in practice, we won't be able to.

The reason I'm being so intense about this is that AFAICT these are all true:

Premise 1: Without build isolation enabled by default, then in practice everyone will putter along putting up with broken builds all the time. It's *incredibly* easy to forget to declare a build dependency, it's the kind of mistake that every new user makes, and experienced users too.

I know lots of projects that already build in [mostly] clean environments all the time - e.g. anything using tox[*], most things that use Travis, Appveyor, Jenkins etc. Yes, if its not there by default it requires effort to opt-in, and there will be a tail of some sort. I don't disagree with the effect of the premise, but I do disagree about the intensity.

[*]: yes, to runs setup.py sdist outside the environment, so setup_requires doesn't get well exercised. IIRC tox is adding a feature to build in a venv so they will be exercised.

...

Premise 2: We can either enable build isolation together with the new static bootstrap requirements, or we can never enable build isolation at all, ever.

I don't agree with this one at all. We can enable it now if we want: yes, its a behavioural change, and we'd need to phase it in, but the logistics are pretty direct.

...

Conclusion: If we want to ever reach a state where builds are reliable, we need to tie build isolation to the new static metadata.

If you have some clever plan for how we could practically transition to build isolation without having them opt-in via a new feature, then that would be an interesting counter-argument; or an alternative plan for how to reach a point where build requirements are accurate without being enforced; or ...?

As Paul said, add it off by default, phase it in over a couple of releases. 0: opt-in 1: opt-out but when builds fail disable and retry (and warn) 2: opt out

...

...

pyqt5 not having i386 is just a trivial egregious case. ARM32 and 64 is going to be super common, Power8 another one, let alone less common but still extant and used architectures like PPC, itanium, or new ones like x86_32 [If I remember the abbreviation right - no, its not i386].

(it's x32)

Thanks :)

...

manylinux is helpful here, but it's not necessary -- build isolation just requires that the dependencies be pip installable, could be from source or whatever. In practice the wheel cache will kick in and handle most of the work.

*only* for things that honour setup.py - which the vast majority of SWIG and SWIG style extension modules do not.

...

...

Solve that underlying problem - great, then isolation becomes an optimisation question for things without manylinux wheels. But if we don't solve it then isolation becomes a 'Can build X at all' question, which is qualitatively different.

More like 'can build X at all (without adding one command line option)'. And even this is only if you're in some kind of environment that X upstream doesn't support -- no developer is going to make a release of X with build isolation turned on unless build isolation works on the platforms they care about.

There's some thinking here I don't follow: upstreams don't opt into-or-outof build isolation AFAICT, *other* than by your proposal to couple it to declaring bootstrap requirements. The idea that upstream should care about whether their thing is built in an isolated environment or not doesn't make any sense to me. Upstream *should* make sure their thing can build in an isolated fashion - and then not care anymore.

-Rob -- Robert Collins <rbtcollins@hpe.com> Distinguished Technologist HP Converged Cloud _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On Wed, May 4, 2016 at 11:57 PM, Robert Collins <robertc@robertcollins.net> wrote:

No. Old pip new package will break, new pip old package is entirely safe AFAICT.

That's the goal, yes? So I think we need to get less obsessed with backward compatibility: pip will retain (for along time) the ability to install, build, whatever any packages that it currently works for -- nothing on PyPi breaks. New versions of pip can "detect" that a given package is "new style", and it will do the right thing with that -- maybe following a completely different code path. If we take this approach, then we can design the new pip interface however we want, rather than trying to keep compatibility. but we need to be abel to provide the build tool to go with it, yes? I think the way to do that is to have a new build tool that is forked from setuptools, and is essentially the same thing (in version 1), but with the functionality that it "shouldn't" have stripped out -- i.e. no easy_install! Pacakge developers that want to adpt the new system only need to change an import ln their setup.py: from setuptools-lite import setup and then do the right things with meta-data, and it will work with new versions of pip. Right now we have a swamp

...

...

...

we need to drain,

exactly -- but rather than trying to drain that swamp, let's just move over to the dry land next door.... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On 5 May 2016 at 09:00, Marius Gedminas <marius@gedmin.as> wrote:

pip install pyqt5

You need Python 3.5, and you also need to ensure you are calling the `pip` command for Python 3.5, and not the default `pip` which may be linked to a different Python version. Try this for example: python3.5 -m pip install --user pyqt5

We've spent a huge amount of effort on reaching the point where pretty

much everything *can* be made pip installable. Heck, *PyQt5*, which is my personal benchmark for a probably-totally-unpackageable package, announced last week that they now have binary wheels on pypi for all of Win/Mac/Linux:

that's thanks to binary wheels being a simple archive -- by definition, if you can build and install it *somehow* you can make a binary wheel out of it -- which is a great thing -- so nice to have wheel completely independent of the build system. isn't that where we are trying to go -- have pip be independent of the build system?

If the BDFL delegate makes a call - fine. I read

...

Nick's earlier email in the thread as such a thing TBH :).

Oh sure, I think everyone agrees that the file format choice is not a make-or-break decision

Am I the only one that thinks we're conflating two things here? choice of file format is separate from "use setup.cfg" -- I think we should use something NEW -- rather than try to patch on top of existing setup.cfg files -- that way we're free to change the rules, and no one (and no tools) will be confused about what version of what system is in play. In short -- we should do something new, not just extend setup.cfg, and not because we want a new file format. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On 4 May 2016 at 22:33, Donald Stufft <donald@stufft.io> wrote:

..> I also believe that we can't provide a replacement for setup.py without either

purposely declaring we no longer support something that people used from it or providing a way to support that in the new, setup.py-less format.

The thunk I wrote being insufficient?

One thing I remarked to Nataniel yesterday was that it might be a good idea to drop the build system aspect of these for right now (since afaict all the invested parties are currently overloaded and/or have a lack of time) and focus soley on the part of the proposal that enables us to get a real setup_requires ...

the only reason I got involved in build system discussions was pushback 18months or so back when I implemented a proof of concept for pip that just used setup.cfg. I'd be very happy to ignore all the build system stuff and just do bootstrap requirements in setup.cfg. -Rob -- Robert Collins <rbtcollins@hpe.com> Distinguished Technologist HP Converged Cloud

Ok so, if i draft a pep for said proposal, will it die under the weight of a thousand bike sheds? On 5 May 2016 3:09 PM, "Nick Coghlan" <ncoghlan@gmail.com> wrote:

On 5 May 2016 at 06:28, Robert Collins <robertc@robertcollins.net> wrote:

...

the only reason I got involved in build system discussions was pushback 18months or so back when I implemented a proof of concept for pip that just used setup.cfg. I'd be very happy to ignore all the build system stuff and just do bootstrap requirements in setup.cfg.

I know I'm one of the folks that has historically been dubious of the "just use setup.cfg" idea, due to the assorted problems with the ini-style format not extending particularly well to tree-structured data (beyond the single level of file sections).

However, my point of view has changed over that time:

1. We've repeatedly run up against the "JSON is good for programs talking to each other, but lousy as a human-facing interface" problem 2. By way of PEPs 440 and 508, we've got a lot more experience in figuring out how to effectively bless de facto practices as properly documented standards (even when it makes the latter more complicated than we'd like) 3. The ongoing popularity of setup.cfg shows that while ini-style may not be perfect for this use case, it clearly makes it over the threshold of "good enough" 4. Folks that *really* want a different input format (whether that's YAML, TOML, or something else entirely) will still be free to treat setup.cfg as a generated file, just as setup.py can be a generated file today

The last couple of years have also given me a whole range of opportunities (outside distutils-sig) to apply the mantra "the goal is to make things better than the status quo, not to make them perfect", and that has meant getting better at distinguishing what I would do given unlimited development resources from what makes the most sense given the development resources that are actually available.

So when I ask myself now "What's the *simplest* thing we could do that will make things better than the status quo?", then the answer I come up with today is your original idea: bless setup.cfg (or at least a subset of it) as a standardised interface.

Don't get me wrong, I still think that answer has significant downsides - I've just come around to the view that "is likely to be easy to implement and adopt" are upsides that can outweigh a whole lot of downsides :)

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 6 May 2016 at 06:41, Chris Barker <chris.barker@noaa.gov> wrote:

On Wed, May 4, 2016 at 8:09 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

3. The ongoing popularity of setup.cfg shows that while ini-style may not be perfect for this use case, it clearly makes it over the threshold of "good enough"

it's only popular because it's what's there -- if we're using that standard, we could make the same argument about setuptools ;-)

That's exactly the course change we made following the release of Python 3.3 - rather than trying to replace setuptools directly, the scope of our ambitions was narrowed to eliminating the need to run "./setup.py install", while keeping setuptools as the default build system used to enable things like generating wheel files ("./setup.py bdist_wheel" doesn't work if you're importing setup from distutils.core - you have to be importing it from setuptools. If you have a setup.py file that imports from distutils.core, you have to use "pip wheel" if you want to get a wheel file)

...

So when I ask myself now "What's the *simplest* thing we could do that will make things better than the status quo?", then the answer I come up with today is your original idea: bless setup.cfg (or at least a subset of it) as a standardised interface.

IIUC, we would be changin, or at least adding to the current setup.dfg spec. So this is a change, no matter how you slice it, saying "configuration will be specified in setup.something, in some other format, is the least significant part of all this change.

And maybe it's good to keep "new style" configuration clearly separate.

Part of my motivation for suggesting re-using setup.cfg is the proliferation of packaging related config sprawl in project root directories - setup.py won't be going anywhere any time soon for most projects, some folks already have a setup.cfg (e.g. to specify universal wheel creation), and there's also MANIFEST.in to control sdist generation. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, May 4, 2016 at 3:33 AM, Donald Stufft <donald@stufft.io> wrote:

I'd actually prefer not using JSON for something that is human editable/writable because I think it's a pretty poor format for that case. It is overly restrictive in what it allows (for instance, no trailing comma gets me every time) and the lack of comments I think make it a poor format for that.

yup -- these are really annoying when JSON is used for a config format. but INI pretty much sucks, too. What about PYSON (my term) -- python literals -- could be evaluated with ast.literal_eval to be safe, and would give us comments, and trailing commas, and python's richer data types. or just plain Python -- the file would be imported and we'd specify particular variables that needed to be defined -- maybe as simple as: config = a_big_dict_with_lots_of_stuff_in_it. so it could be purely declarative, but users could also put code in there to customize the configuration on the fly, too. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On 05/04/2016 03:28 PM, Paul Moore wrote:

On 4 May 2016 at 23:11, Chris Barker wrote:

That basically repeats the mistake that was made with setup.py. We explicitly don't want an executable format for specifying build configuration.

Executable code or not, we need to be able to specify different files depending on the python version. -- ~Ethan~

On 05/04/2016 04:29 PM, Alex Grönholm wrote:

Different files for what? Something not covered by PEP 508?

Somebody will have to distill that PEP, I have only an small inkling of what it's trying to say. As for my specific use case: I have Python3-only files in my distribution, so they should only be installed on Python3 systems. Python2 systems generate useless errors. -- ~Ethan~

On 5 May 2016 at 00:58, Ethan Furman <ethan@stoneleaf.us> wrote:

Somebody will have to distill that PEP, I have only an small inkling of what it's trying to say.

The relevant point for this discussion is "you can request that particular packages are installed before the build step for your package, and precisely what gets installed can depend on the Python version". If you're not intending to use a build system other than setuptools, this is probably not relevant to you (other than to require a new enough version of setuptools that has the bug fixes you need).

As for my specific use case: I have Python3-only files in my distribution, so they should only be installed on Python3 systems. Python2 systems generate useless errors.

That's a build system issue, so not directly relevant here. Specifically, assuming it's the issue you reported here previously, it's a problem in setuptools (so when it's fixed there, with this new proposal you can specify that your build requires setuptools >= X.Y, to ensure the bug fix is present). Paul

On Wed, May 4, 2016 at 7:45 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

This configuration vs customisation distinction is probably worth spelling out for folks without a formal software engineering or computer science background, so:

fair enough -- good to be clear on the terms.

Configuration is different: you're choosing amongst a set of possibilities that have been constrained in some way, and those constraints are structurally enforced.

That's a key point here -- I guess I'm skeptical that we can have the flexibility we need with a purely configuration-based system -- we probably don't WANT to constrain the options completely. If you think about it, while distutils has it's many, many flaws, what made it possible for it to be as useful as it is, and last as long as it has because is CAN be customized -- users are NOT constrained to the built-in functionality. I suspect the idea of this thread is to keep the API to a build system constrained -- and let the build systems themselves be as customizable as the want to be. And I haven't thought it out carefully, but I have a feeling that we're going to hit a wall that way .. but maybe not.

Usually that enforcement is handled by making the configuration declarative - it's in some passive format like an ini file or JSON, and if it gets too repetitive then you introduce a config generator, rather than making the format itself more sophisticated.

OK -- that's more or less my thought -- if it's python that gets run, then you've got your config generator built in -- why not?

The big advantage of configuration over customisation is that you substantially increase the degrees of freedom in how *consumers* of that configuration are implemented - no longer do you need a full Python runtime (or whatever), you just need an ini file parser, or a JSON decoder, and then you can look at just the bits you care about for your particular use case and ignore the rest.

Sure -- but do we care? this is about python packaging -- is it too big a burden to say you need python to read the configuration? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

This is a recurring point of confusion. setup.py is not ever going away. In general it is necessary for you to be able to write software to build your software, and there is no intention to take that feature away. People repeatedly come to the conclusion that static metadata means the entire build is static. It's only the dependencies that need to be static to enable better dependency resolution in pip. The build does not need to be static. The proposed feature means you will be able to have a simpler setup.py or no setup.py it by using something like flit or pbr that are configured with .cfg or .ini. setup.py is not going away. Static metadata means the list of dependencies, author name, trove classifiers is static. Not the build itself. Enforced staticness of the build script is right out. On Thu, May 5, 2016 at 4:34 PM Alex Grönholm <alex.gronholm@nextday.fi> wrote:

I think it would be best to gather a few extreme examples of setup.py files from real world projects and figure out if they can be implemented in a declarative fashion. That at least would help us identify the pain points.

For starters, gevent's setup.py looks like it needs a fair bit of custom logic: https://github.com/gevent/gevent/blob/master/setup.py

05.05.2016, 23:30, Chris Barker kirjoitti:

On Wed, May 4, 2016 at 7:45 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

This configuration vs customisation distinction is probably worth spelling out for folks without a formal software engineering or computer science background, so:

fair enough -- good to be clear on the terms.

...

Configuration is different: you're choosing amongst a set of possibilities that have been constrained in some way, and those constraints are structurally enforced.

That's a key point here -- I guess I'm skeptical that we can have the flexibility we need with a purely configuration-based system -- we probably don't WANT to constrain the options completely. If you think about it, while distutils has it's many, many flaws, what made it possible for it to be as useful as it is, and last as long as it has because is CAN be customized -- users are NOT constrained to the built-in functionality.

I suspect the idea of this thread is to keep the API to a build system constrained -- and let the build systems themselves be as customizable as the want to be. And I haven't thought it out carefully, but I have a feeling that we're going to hit a wall that way .. but maybe not.

...

Usually that enforcement is handled by making the configuration declarative - it's in some passive format like an ini file or JSON, and if it gets too repetitive then you introduce a config generator, rather than making the format itself more sophisticated.

OK -- that's more or less my thought -- if it's python that gets run, then you've got your config generator built in -- why not?

...

The big advantage of configuration over customisation is that you substantially increase the degrees of freedom in how *consumers* of that configuration are implemented - no longer do you need a full Python runtime (or whatever), you just need an ini file parser, or a JSON decoder, and then you can look at just the bits you care about for your particular use case and ignore the rest.

Sure -- but do we care? this is about python packaging -- is it too big a burden to say you need python to read the configuration?

-CHB

--

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.orghttps://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

C extensions, py-modules, ... On Thu, May 5, 2016, 17:05 Alex Grönholm <alex.gronholm@nextday.fi> wrote:

OK, so which setup() arguments do we want to leave out of the static metadata?

05.05.2016, 23:53, Daniel Holth kirjoitti:

This is a recurring point of confusion. setup.py is not ever going away. In general it is necessary for you to be able to write software to build your software, and there is no intention to take that feature away.

People repeatedly come to the conclusion that static metadata means the entire build is static. It's only the dependencies that need to be static to enable better dependency resolution in pip. The build does not need to be static.

The proposed feature means you will be able to have a simpler setup.py or no setup.py it by using something like flit or pbr that are configured with .cfg or .ini. setup.py is not going away.

Static metadata means the list of dependencies, author name, trove classifiers is static. Not the build itself.

Enforced staticness of the build script is right out.

On Thu, May 5, 2016 at 4:34 PM Alex Grönholm <alex.gronholm@nextday.fi> wrote:

...

I think it would be best to gather a few extreme examples of setup.py files from real world projects and figure out if they can be implemented in a declarative fashion. That at least would help us identify the pain points.

For starters, gevent's setup.py looks like it needs a fair bit of custom logic: https://github.com/gevent/gevent/blob/master/setup.py

05.05.2016, 23:30, Chris Barker kirjoitti:

On Wed, May 4, 2016 at 7:45 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

This configuration vs customisation distinction is probably worth spelling out for folks without a formal software engineering or computer science background, so:

fair enough -- good to be clear on the terms.

...

Configuration is different: you're choosing amongst a set of possibilities that have been constrained in some way, and those constraints are structurally enforced.

That's a key point here -- I guess I'm skeptical that we can have the flexibility we need with a purely configuration-based system -- we probably don't WANT to constrain the options completely. If you think about it, while distutils has it's many, many flaws, what made it possible for it to be as useful as it is, and last as long as it has because is CAN be customized -- users are NOT constrained to the built-in functionality.

I suspect the idea of this thread is to keep the API to a build system constrained -- and let the build systems themselves be as customizable as the want to be. And I haven't thought it out carefully, but I have a feeling that we're going to hit a wall that way .. but maybe not.

...

Usually that enforcement is handled by making the configuration declarative - it's in some passive format like an ini file or JSON, and if it gets too repetitive then you introduce a config generator, rather than making the format itself more sophisticated.

OK -- that's more or less my thought -- if it's python that gets run, then you've got your config generator built in -- why not?

...

The big advantage of configuration over customisation is that you substantially increase the degrees of freedom in how *consumers* of that configuration are implemented - no longer do you need a full Python runtime (or whatever), you just need an ini file parser, or a JSON decoder, and then you can look at just the bits you care about for your particular use case and ignore the rest.

Sure -- but do we care? this is about python packaging -- is it too big a burden to say you need python to read the configuration?

-CHB

--

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.orghttps://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On May 6, 2016, at 12:36 PM, Brett Cannon <brett@python.org> wrote:

So who is the BDFL on this decision? It seems we need someone to stop the bikeshedding on the field name and what file is going to house this configuration data. And do we need someone to write a PEP for this proposal to have something to target?

We need someone to write the PEP and someone to offer to be BDFL for it. For this particular change the default would be Nick for BDFL but if he’s busy someone else can take it over for this PEP. Though I think we need someone writing an actual PEP first. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 6 May 2016 at 19:14, Brett Cannon <brett@python.org> wrote:

On Fri, 6 May 2016 at 09:40 Donald Stufft <donald@stufft.io> wrote:

...

On May 6, 2016, at 12:36 PM, Brett Cannon <brett@python.org> wrote:

So who is the BDFL on this decision? It seems we need someone to stop the bikeshedding on the field name and what file is going to house this configuration data. And do we need someone to write a PEP for this proposal to have something to target?

We need someone to write the PEP and someone to offer to be BDFL for it. For this particular change the default would be Nick for BDFL but if he’s busy someone else can take it over for this PEP. Though I think we need someone writing an actual PEP first.

OK, assuming the Nick will be pronouncing, who wants to write the PEP?

... and if Nick doesn't want to pronounce, I'm willing to offer to be BDFL for this one. But a PEP is the first thing. (And IMO the key point of the PEP is to be very clear on what is in scope and what isn't - the discussions have covered a *lot* of ground and being clear on what's excluded will be at least as important as stating what's in scope). Paul

On Mon, 9 May 2016 at 06:42 Nick Coghlan <ncoghlan@gmail.com> wrote:

On 7 May 2016 at 08:21, Paul Moore <p.f.moore@gmail.com> wrote:

...

On 6 May 2016 at 19:14, Brett Cannon <brett@python.org> wrote:

...

OK, assuming the Nick will be pronouncing, who wants to write the PEP?

... and if Nick doesn't want to pronounce, I'm willing to offer to be BDFL for this one. But a PEP is the first thing. (And IMO the key point of the PEP is to be very clear on what is in scope and what isn't - the discussions have covered a *lot* of ground and being clear on what's excluded will be at least as important as stating what's in scope).

Answering this specifically: I'm happy to be the arbiter-of-consensus for this one, as my schedule's pretty clear right now (at least until I head to PyCon US on the 27th).

OK, I'll list Nick as the BDFL Delegate in the PEP then. Hoping to have a draft to send to the list today or tomorrow.

On Fri, 6 May 2016 at 16:58 Nathaniel Smith <njs@pobox.com> wrote:

On Fri, May 6, 2016 at 11:14 AM, Brett Cannon <brett@python.org> wrote:

...

On Fri, 6 May 2016 at 09:40 Donald Stufft <donald@stufft.io> wrote:

...

On May 6, 2016, at 12:36 PM, Brett Cannon <brett@python.org> wrote:

So who is the BDFL on this decision? It seems we need someone to stop

...

...

bikeshedding on the field name and what file is going to house this configuration data. And do we need someone to write a PEP for this

the proposal

...

...

to have something to target?

We need someone to write the PEP and someone to offer to be BDFL for it. For this particular change the default would be Nick for BDFL but if he’s busy someone else can take it over for this PEP. Though I think we need someone writing an actual PEP first.

OK, assuming the Nick will be pronouncing, who wants to write the PEP?

And Paul also stepped forward to pronounce if Nick didn't want to, so we have the role of Great Decider covered one way or another.

I've just been writing up a comparison of the different file formats, partly in case it's useful to others and partly just for my own use in looking at them against each other and seeing how much it actually matters. This might also be reusable for the rationale/rejected-alternatives section of a PEP, if anyone wants it, or I could go ahead and add a few paragraphs to turn it into a proper PEP.

What does the PEP need to cover? 1. The syntax of the file (which based on the replies to your great overview, Nathaniel, looks to be TOML). 2. The name of the file (although I'm assuming it will be setup.* out of tradition, probably setup.toml if TOML wins the format battle). 3. What fields there will be and their semantics ... 1. Format version (so just deciding on a name -- which also includes whether it should be top-level or in a subsection -- and initial value) 2. The actual bootstrap field (so its name and what to do if a dependency is already installed but at a version that doesn't match what the bootstrap specification asks for) Am I missing anything? And since I keep pushing on this I'm willing to be a co-author on any PEP if there's no hard deadline in getting the PEP written (i.e. I can help write the prose, but I don't have the time to do the coding as this will be the fourth PEP I have going in some state; got to catch up to Nick's 35 PEPs somehow ;).

On Sat, May 7, 2016, 12:16 Chris Barker <chris.barker@noaa.gov> wrote:

On Sat, May 7, 2016 at 11:18 AM, Brett Cannon <brett@python.org> wrote:

...

What fields there will be and their semantics ...

1. Format version (so just deciding on a name -- which also includes whether it should be top-level or in a subsection -- and initial value) 2. The actual bootstrap field (so its name and what to do if a dependency is already installed but at a version that doesn't match what the bootstrap specification asks for)

Am I missing anything?

So what is this new configuration file supposed to cover?

How to specify what has to be installed to simply build a project, e.g. is setuptools needed to run setup.py, and if so what version? All the package meta-data? i.e. everything that would be needed by a

package manager to properly install the package?

or the build meta-data: everything needed by the build system to build the package?

both in one file?

And am missing something?

You're missing that you're talking several PEPs down the road. :) Right now all we are discussing is how to specify what build tools a project needs (historically setuptools, but in the future it could be flit or something else). how is this about "bootstrapping" -- to me, bootstrapping is when you need

X to build X. Isn't this just regular old configuration: you need x,y to build z?

Sure, if you don't like the term "bootstrap" then you can call it "build requirements". We have not been calling it " configuration" in a general sense as this doesn't cover how to invoke the build step (that will probably be the next PEP), just what needs to be installed to even potentially do a build. -Brett

-CHB

...

And since I keep pushing on this I'm willing to be a co-author on any PEP if there's no hard deadline in getting the PEP written (i.e. I can help write the prose, but I don't have the time to do the coding as this will be the fourth PEP I have going in some state; got to catch up to Nick's 35 PEPs somehow ;).

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

--

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

On Sun, May 8, 2016 at 5:49 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

The reason I think "bootstrap" is a better name at this point

I *really* don't want to add to the bike-shedding of the name at this point -- I really don't care. I was just trying to see if I was misunderstanding something, as it didn't seem to be bootstrapping anything to me, and I see I was kinda missing some of the scope and complexity -- thanks. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Thu, May 5, 2016 at 3:21 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

Chris Barker wrote:

OK -- that's more or less my thought -- if it's python that gets run,

...

then you've got your config generator built in -- why not?

The difference is that the config generator only gets run once, when the config info needs to get produced. If the config file itself is executable, it needs to be run each time the config info gets *used*

exactly -- maybe some run-time configuration specialization? though maybe that's exactly what we want to avoid.

-- which could be by a tool for which running arbitrary python code is awkward or undesirable.

This is python packaging, I'm still trying to see why you'd need to read the config without python available. But I think the key point is -- do we want to enforce that the config data is static -- it WILL not change depending on when or how or where it is read? If so, then yeah, no runnable code. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Thu, May 5, 2016 at 10:45 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

Even if python is available, you might not want to run arbitrary code just to install a package.

If a config file can contain executable code, then it can contain bugs. Debugging is something the developer of a package should have to do, not the user. In my experience, fixing someone else's buggy setup.py is about as much fun as pulling one's own teeth out with a blunt screwdriver.

well, in my experience, debugging my OWN buggy setup.py is about equally pleasant! But I think there is consensus here that build systems need to be customisable -- which means arbitrary code may have to be run. So I don't think this is going to help anyone avoiding dealing with buggy builds :-( or are we talking about a config file that would be delivered with a binary wheel? IN which case, yes, it should not have any executable code in it. And anyway, it seems folks want to go with static config anyway -- clearly separating configuration from customization, so I'll stop now. I'd still like to be able to use Python literals, though :-) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

tl;dr version I think you're right that terminology can be confusing. I think the definitions people typically work to are: 1. The "packaging" or "release" process - the process (run on a developer's machine) of creating files that get published for users to download and install. 2. The "install" process - the process (run on a user's machine) of taking a published file and making it available in their environment. This consists of separate steps: 2a. Optional, and to be avoided wherever possible (by distribution of wheels) - the "build" step that takes a published file and configures (compiles) it for the user's environment 2b. The "install" step (confusion alert! - the "install" step is only one step of the "install" *process*) that puts the files in the correct places on the user's machine. We're not interested in trying to dictate the "packaging" process - pip isn't involved in that process at all (see flit for a system that lets projects build releases in a completely different way). Sigh. Even the tl;dr version is too long :-) On 7 May 2016 at 01:55, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

Chris Barker wrote:

...

But I think there is consensus here that build systems need to be customisable -- which means arbitrary code may have to be run.

I think different people are using the word "build" in different ways here.

To my mind, "building" is what the developer of a package does, and a "build system" is what he uses to do it. I don't care how much arbitrary code gets run during that process.

That is correct, and I agree with you that making a build process like this declarative is not particularly useful. However...

But when I do "python setup.py install" or "pip install" or whatever the recommended way is going to be, from my point of view I'm not "building" the package, I'm *installing* it.

Unfortunately, "python setup.py install" does not work that way - it builds the project and then installs the files. So whether you want to or not, you're building. That's basically why we're trying to make "pip install foo" the canonical way of installing packages. So let's ignore "setup.py install" for the rest of this discussion. Now, for "pip install foo", *if the foo project provides a wheel compatible with your system* then what you expect is what you get - a pure install with no build step. The problem lies with projects that don't supply wheels, only source. Or unusual systems that we can't expect projects to have wheels for. Or local checkouts ("pip install ."). In those cases, it's necessary to do a build before you can install. So while we're aiming for 80% or more of the time "pip install" to do a pure install from a binary distribution, we can't avoid the fact that occasionally the install will need to run an implicit build step.

Confusion arises because the process of installation may require running a C compiler to generate extension modules. But figuring out how to do that shouldn't require running arbitrary code supplied by the developer. All the tricky stuff should have been done before the package was released.

I'm not sure I follow what you are suggesting here. Do you expect that projects should be able to publish something (it's not exactly a sdist, but it's not a wheel either as it doesn't contain everything built) should (somehow) contain simplified instructions on how to build the various C/Fortran extensions supplied in the bundle as source code? That's an interesting idea, but how would it work in practice? The bundles would need to be platform specific, I assume? Or would the user need to put all the various details of his system into a configuration file somewhere (and update that file whenever he installs a new library, updates his compiler, or whatever)? How would this cope with (for example) projects on Windows that *have* to be compiled with mingw, and not with MSVC?

If it's having trouble finding some library or header file or whatever on my system, I'd much rather have a nice, clear declarative config file that I can edit to tell it where to find them, than some overly clever piece of python code that's great when it works but a pain to unravel when it doesn't.

This sounds to me more like an attempt to replace the "build" part of distutils/setuptools with a more declarative system. While that may be a worthwhile goal (I genuinely have no opinion on that) it's largely orthogonal to the current discussions. Except in the sense that if you were to build such a system, the proposals currently on the table would allow you to ask pip to use that build system rather than setuptools: # Invented syntax, because syntax is what we *haven't* agreed on yet :-) [buildsystem] requires=gregsbuild build_command=gregsbuild --make-wheel Then if a user on a system for which the project doesn't have a binary wheel installed tries to install the project, your build system will be downloaded, and the "gregsbuild --make-wheel" command will be run to make a wheel. That's a one-off event - the wheel will be cached on the user's system for future use. I think the key point to understand is that of necessity, "pip install" runs two *steps* - one is the obvious install step, the other is a build step. We're working to reduce the number of cases where a build step is needed as far as we can, but the discussion here is about making life easier for projects who can't provide wheels for all their users, and need the build *step* (the one run on the user's machine, not the build *process* - which I'd describe as a "release" or "packaging" process, as it creates the distribution files made available to users - that runs on the developer's machine. Paul

On 7 May 2016 at 14:17, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

I don't know if there are conventions for such things on Windows. I suspect not, in which case manual input is going to be needed one way or another.

There aren't. You typically need to specify the exact locations of all non-system libraries you use when you link. Judging by the various complex systems in use (autoconf, pkg_config, ...) I don't think it's as simple as you're suggesting on Unix either. The complexities of configuring and building C libraries are a hard problem, and one that people have been trying to solve for years. It's not an area we (the Python packaging community) have any interest or intention of getting involved in. Distutils went there and provided a solution that did a fantastic job of solving the simpler parts of the problem[1], but these days, the limitations of that approach are clear - and we'd much rather enable specialists to build better (or more specific) solutions and plug them into pip, than (as non-experts) try to write those solutions for them.

That's nice, but it wouldn't help me when I encounter a package that *hadn't* been set up to use gregsbuild. :-(

And ultimately that's the key issue that pip has to deal with - we have to provide support for the many thousands of packages on PyPI and elsewhere (including in closed-source in-house projects that we aren't even theoretically capable of finding out about) that as you say haven't been set up to use gregsbuild. You suggest a declarative way of specifying compilation details. But I'm not clear how *that* would help when I encounter a project that hasn't been set up to use that new system, either. Paul [1] I was around before distutils, and it's incredibly easy these days, with such a high awareness of the limitations of distutils, to forget what a huge improvement it was over the previous situation. I strongly believe that distutils - and the way it enabled all of Python's subsequent packaging infrastructure - is one of the key reasons why Python has become as popular as it is today. The debt we owe to Greg Ward, who wrote distutils, is huge.

On 6 May 2016 at 06:30, Chris Barker <chris.barker@noaa.gov> wrote:

On Wed, May 4, 2016 at 7:45 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

Usually that enforcement is handled by making the configuration declarative - it's in some passive format like an ini file or JSON, and if it gets too repetitive then you introduce a config generator, rather than making the format itself more sophisticated.

OK -- that's more or less my thought -- if it's python that gets run, then you've got your config generator built in -- why not?

The immediate reason is because Python allows imports, and if imports are permitted in the config script, people will use them, and if they're not permitted, they'll complain about their absence. The "Python-with-imports" case is the status quo with setup.py, and we already know that's a pain because you need to set up an environment that already has the right dependencies installed to enable your module level imports in order to run the script and find out what dependencies you need to install to let you run the script in the first place. The "Python-without-imports" approach would just be confusing - while it would avoid the dependency bootstrapping problem, it would only be kinda-sorta-Python rather than actual Python. So rather than saying "the bootstrapping dependency declaration file is Python-but-not-really", it's easier to say "it's an ini-file format that can be parsed with the configparser module" or "it's JSON" (I'm ruling out any options that don't have a stdlib parser in Python 2.7) The "future benefit" reason is that it's a lot easier to be confident that reading a config file from an uploaded artifact isn't going to compromise a web service, so a future version of PyPI can readily pull information out of the config file and republish it via an API. Once you have that kind of information available via an API, you can resolve it before downloading *anything* (which is especially useful when your goal is dependency graph analysis rather than downloading the whole of PyPI and running a Python script from every package you downloaded). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 6 May 2016 at 13:15, Chris Barker <chris.barker@noaa.gov> wrote:

On Fri, May 6, 2016 at 5:41 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

[...]

...

So rather than saying "the bootstrapping dependency declaration file is Python-but-not-really", it's easier to say "it's an ini-file format that can be parsed with the configparser module" or "it's JSON" (I'm ruling out any options that don't have a stdlib parser in Python 2.7)

Last time, I promise :-)

"python literals" is perfectly well defined -- both by the language reference, and by "can be parsed by ast.literal_eval"

and it addresses the limitations of JSON and is fully declarative.

There is actually prior art for that kind of use. Odoo uses such a language for its addons system, including package dependencies. See example file here: https://github.com/OCA/maintainer-tools/blob/master/template/module/__opener... Notice the `depends` key, that lists other addons, and the `external_dependencies` key that can list both python distribution dependencies as well as external program dependencies. ISTM the reluctance in adopting this idea comes from the desire of using other programming languages to parse the bootstraping information (imagine an sdist -> deb converter implemented in go. It will have to exec python eventually, during the build process, but not before it has prepared the build environment). And even though, unlike JSON, the ConfigParser format of setup.cfg is not officially defined anywhere, there are sufficiently compatible ini parsers in other languages that the idea still has merit.

On Wed, May 4, 2016 at 3:28 PM, Paul Moore <p.f.moore@gmail.com> wrote:

...

so it could be purely declarative, but users could also put code in

On 4 May 2016 at 23:11, Chris Barker <chris.barker@noaa.gov> wrote: there to

...

customize the configuration on the fly, too.

That basically repeats the mistake that was made with setup.py. We explicitly don't want an executable format for specifying build configuration.

I don't think it's the same thing -- setup.py is supposed to actually CALL setup() when it is imported or run. It is not declarative in the least. what I'm suggesting is that the API be purely declarative: "These values will be declared" (probably a single dict of stuff) but if you make it an executable python file, then users can write code to actually create their setup. Alternatively, you make the API purely declarative, and then folks have to write external scripts that create/populate the configuration files. I guess where I'm coming from is that I'm not sure we CAN make a purely, completely declarative API to a build system -- folks are always going to find some weird corner case where they need to do something special. making the configuration file executable is an easy way to allow this. Otherwise, you need to provide hooks to plug in custom functionality, which is jsut alot more awkward, though I suppose more robust. I've found it very handy to use an executable python file for configuration for a number of my projects -- it's really nice to just throw in a little calculation for something on the fly -- even if it could be done purely declaratively Anyway, if you stick with purely declarative, you can still use Python literals as an alternative to JSON (Or INI, or, ....). Python literals are richer, more flexible, and, of course, familiar to anyone developing a python package :-) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

Just call it Steve On Wed, May 4, 2016, 16:25 Robert Collins <robertc@robertcollins.net> wrote:

On 4 May 2016 at 19:39, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

On 4 May 2016 at 16:03, Robert Collins <robertc@robertcollins.net> wrote:

...

The edits I'd expect to make if the conclusions I suggested in https://mail.python.org/pipermail/distutils-sig/2016-March/028437.html are adopted are:

- change to a Python API - BFDL call on the file format and name

There is no need to issue a new sdist thing, because sdists today are *already* documented across PEPs 241, 314 and 345.

I already +1'ed using a Python API, but on the file name & format side, we have the following candidates and prior art floating around:

pypa.json in PEP 516 pypackage.json in PEP 517 pydist.json in PEP 426 METADATA (Key: Value) in sdists and wheels WHEEL (Key: Value) in wheels

My impression is that we're generally agreed on wanting to move from Key:Value to JSON as the baseline for interoperability formats, so my suggestion is to use the name "pybuild.json".

The problem I have with pypa/pypackage/pydist is that they're all too broad - we're moving towards an explicitly multi-stage pipeline (tree -> sdist -> wheel -> installed) and additional metadata gets added at each step. The "pybuild.json" metadata specifically covers how to get from a source tree or sdist to a built wheel file, so I think it makes sense to use a name that reflects that.

I don't think we have anything resembling consensus on that pipeline idea.

pybuild.json would be fine as a name though :).

-Rob

-- Robert Collins <rbtcollins@hpe.com> Distinguished Technologist HP Converged Cloud _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On 04/27/2016 02:45 PM, Paul Moore wrote:

On 27 April 2016 at 18:39, Ethan Furman wrote:

# Upload the files once you've checked them twine upload *.whl dist/* # because setup.py upload can't upload prebuilt files

What a pain. :(

Personally, I agree with Donald that the "normal" process of building a distribution should be:

1. Build the sdist. 2. Build wheels *from* that sdist. 3. Check the built sdist and wheels. 4. Upload the files once you've confirmed they are OK.

What types of things should I be doing for (3) ? -- ~Ethan~

Actually, another thing you can do with the wheels at this point is to take them for a spin :-) Have pip in another virtualenv install it by using --find-links, or even setting your own local index. If you have tests embedded in your wheel, now would be a good time to run them. Best regards, Leo On 28 April 2016 at 05:03, Paul Moore <p.f.moore@gmail.com> wrote:

On 28 April 2016 at 00:57, Ethan Furman <ethan@stoneleaf.us> wrote:

...

What types of things should I be doing for (3) ?

You could check the contents of the wheels to ensure they don't contain unexpected files :-)

But in all honesty, probably not much. I only noted it as it's the only place you can verify the actual files you're uploading. Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Wheel has a proposal for a "wheel pack" command that might help you. On Thu, Apr 28, 2016, 12:46 Steve Dower <steve.dower@python.org> wrote:

On 27Apr2016 1445, Paul Moore wrote:

...

Personally, I agree with Donald that the "normal" process of building a distribution should be:

1. Build the sdist. 2. Build wheels *from* that sdist. 3. Check the built sdist and wheels. 4. Upload the files once you've confirmed they are OK.

The tools should make that as transparent as possible (I'm not averse to a command that builds sdist and wheels together, but such a command could easily use the sdist to build the wheels "behind the scenes") and there may be special cases (incremental builds of wheels when a full recompile is a significant cost) but those are quality of implementation details.

One extra task that I often need is the ability to separate bdist_ext from bdist_wheel, so that I can apply modifications to built modules (e.g. code signing) before packaging up the wheel.

There's probably a way to insert this step using a setuptools extension or a setup.py hack, but it's actually most convenient to have two completely separate commands (or options) for building and packaging.

Cheers, Steve _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On Tue, Apr 26, 2016 at 5:11 PM, Paul Moore <p.f.moore@gmail.com> wrote:

...

It looks like you have published a wheel built from a dirty checkout. The 1.1.3 zip sdist produces a clean wheel, as expected. You should use https://pypi.python.org/pypi/check-manifest in your release process.

Ah, yes. enum/py35_test_enum.py is not part of the source tree or sdist, but is in the "enum" package directory, so will get picked up by packages=['enum']

​So yes, technically check-manifest don't have anything to do with bdist_wheel (it does checks for sdist). But most probably it would had​ complained about unversioned file going into the sdist. Maybe there should be a similar tool for bdist_wheel, "check-wheel" anyone? What other checks could it have? Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro