Oh, BDFL, will you settle whether we should raise an exception or return
On Aug 25, 2017 8:28 PM, "Greg Ewing" <greg.ewing(a)canterbury.ac.nz> wrote:
> I mean how is opening a file different than attempting to build an sdist?
1. Opening a file is a very common operation.
2. Most file opens are expected to succeed, and if one doesn't,
the appropriate place to deal with that is almost never at the
site of the open call, but somewhere further up.
In contrast, there's probably only about one place in any
given frontend where the backend's build_sdist method gets
invoked, and it's unlikely the ability to let a not-implemented
exception bubble up from that point would be of great use.
Distutils-SIG maillist - Distutils-SIG(a)python.org
I feel that the discussion on PEP 517 is nearing an end. The
outstanding questions are pretty detail-level, and there's little
progress being made on them because the situations involved are mostly
theoretical in the absence of actual implementations. So I propose the
following plan of action:
1. Accept PEP 517 (at least provisionally).
2. Thomas to complete and publish his library of helper functions for
PEP 517 consumers (optional - but I think he said he had such a thing
3. Build PEP 517 interfaces for flit and setuptools.
4. Implement PEP 517 support in pip.
5. Review what we learned from the above.
6. If needed, revise the PEP in view of actual implementation
experience, and finalise it.
Further frontends and backends can wait for the results of this
process if they are concerned about tracking a provisional PEP.
Hopefully, it shouldn't be too long a process (I'd like to see PEP 517
in the next release of pip, if possible).
1. On any remaining questions for the PEP, let's just go for the
simplest solution for now. Let the PEP authors decide what that is,
with Nick as BDFL arbitrating if there's disagreement.
2. In my view, the PEP 517 support in pip should fully embrace the
principle in the PEP that it's the backend's job to ensure that
sdist->wheel and build_wheel produce equivalent results (when both are
possible), and should therefore *only* use build_wheel, at least until
experience demonstrates that we have to protect users against badly
3. My suspicion is that the setuptools backend will pretty clearly
demonstrate whether it's hard or easy to conform to the requirements
in the PEP, which can inform pip's stance on a "pure build_wheel"
4. If pip goes for direct wheel builds, this does mean that the
build_sdist and get_requires_for_build_sdist hooks won't be exercised
in any actual code during this process. IMO, that's OK, as those hooks
cover important (if currently theoretical) cases for validating the
"publish my project" workflow, and tools that want to address those
cases (such as tox) are likely to need the sdist hooks later.
If this seems like a reasonable plan, I'd suggest that people add
links to issues or PRs for PEP 517 implementations in flit, setuptools
and pip to this thread - as I'm sure all the participants here will
want to review the concrete implementations.
Please understand - I'm not trying to steamroller the process here, or
force particular decisions. It's just that I know I'm burned out on
the discussions, to the extent that I'm skimming mails and largely
ignoring many of the hypothetical arguments, and I'm sure others are
too. So we need to move to a point where we can discuss actual,
in-development code rather than possibilities, and I believe the above
plan gets us to that point as fast as possible.
Am 25.08.2017 um 15:00 schrieb Paul Moore:
> One thought - are the PBR and/or distutils2 teams looking at providing
> PEP 517 support? Assuming they are, have they had a change to review
> the PEP to ensure it suits their needs? And if they aren't, what is it
> about the PEP that makes them unwilling to do so?
I think this is a very good question!
I started a new thread to make this more visible.
Are there PBR/distutils2 hackers on this list here?
If yes: Do you support this pep?
If no: where can you meet PBR/distutils2 hackers?
Thomas Guettler http://www.thomas-guettler.de/
Should probably take Guido out of this email chain now. And does everyone
agree with what Nathaniel said on sys.path?
On Aug 26, 2017 11:00 AM, "Daniel Holth" <dholth(a)gmail.com> wrote:
I'm bored with that
On Sat, Aug 26, 2017, 11:59 xoviat <xoviat(a)gmail.com> wrote:
> Is everyone on board with that?
> On Aug 26, 2017 4:29 AM, "Paul Moore" <p.f.moore(a)gmail.com> wrote:
>> On 26 August 2017 at 03:17, Guido van Rossum <guido(a)python.org> wrote:
>> > In pretty much any other context, if you have an operation that returns
>> > regular value or an error value, the error value should be None.
>> > include e.g. returning a non-negative int or -1 for errors, or True for
>> > success and False for errors.)
>> So, given that build_sdist returns the path of the newly built sdist,
>> the correct way to signal "I didn't manage to build a sdist" would be
>> to return None.
>> Now that it's put this way, it seems glaringly obvious to me that this
>> is the correct thing to do.
> Distutils-SIG maillist - Distutils-SIG(a)python.org
The setup.py of numpy looks big and complicated: https://github.com/numpy/numpy/blob/master/setup.py
I guess there are even more complicated setup.py files.
Do you think it will be possible in the future to make setup.py conditionless (no "if", no "else", no function calls)?
I mean that the setuptools/distutils/whatever libraries will handle all needed conditions?
This would mean setup.py would be a pure data structure which could be encoded in json or yaml.
What do you think?
Thomas Guettler http://www.thomas-guettler.de/
Nathaniel seems to be busy with other things at the moment, so I hope he
won't mind me passing on this list of things he'd like to resolve with
the draft PEP. I'll quote his comments and put my responses inline.
> - needs some sort of NotImplemented(Error) handling specified
We've gone round return-value vs exception a few times, but I don't
think that debate is particularly heated, so it probably just needs
someone to say "this is what we're doing". I can be that someone if
needs be. But does anyone feel strongly about it?
> - The out-of-tree build example (that makes an sdist and then builds
> it) seems like exactly the kind of implementation that Donald says he
> doesn't want to see? OTOH I think Nick said he wants to see a more
> elaborated specification of out-of-tree build strategies with this
> specifically as one of them.
> - Nick has suggested that the to-be-defined NotImplemented(Error)
> handling can be used by build_wheel to indicate that it can't do
> out-of-tree builds, so maybe the frontend should try an in-tree build
> instead. I don't know how to convert this into a real spec, though --
> like in general, if I'm a frontend and I call `hook(*args, **kwargs)`
> and it says "can't do that", then how do I know what the problem is
> and what I should try instead?
> - What happens if prepare_build_metadata returns NotImplemented /
> raises NotImplementedError?
> - I don't understand how out-of-tree builds and prepare_build_metadata
> are supposed to interact.
> - Also, AFAICT the out-of-tree build feature has no motivation
> anymore. The only mentioned use case is to support incremental build
> features in automatic build pipelines like Travis, but there are a
> number of these build pipelines, and lots of existing build systems
> that support out-of-tree builds, and AFAICT no-one actually uses them
> together like this. (I think it's because most build systems'
> out-of-tree build features are designed on the assumption that there's
> a human running the show who will deal with any edge cases.)
I believe the out-of-tree build option Nathaniel is referring to is the
build_directory parameter to build_wheel(). His proposed APIs remove
that parameter, and elsewhere in his email he describes that no-one
seems to want it: everyone thinks someone else wants it.
By my understanding, the reasons for including the build_directory
1. Without an explicit build directory, the developers of pip are
concerned that build backends will modify the source tree and cause
issues which users report as bugs to pip.
2. A frontend-controlled build directory could be used for caching
intermediate build artifacts. This was a secondary argument after we had
the idea, and we've never really fleshed out how we expect it to work.
There's also a concern that if the first round of frontends always use
an empty tempdir, backends will end up relying on that. I think this
second argument is a weak one unless we spend the time to figure out the
Do other people see the situation in a similar way? How might we move
forwards on this?
> - needs some sort of conclusion to the srcdir-on-sys.path issue.
While Nathaniel is in the minority regarding srcdir-on-sys.path, he
argues that most of us are assuming a default position without really
thinking through it, which is certainly true of me. I don't feel
strongly about this topic, but I do want to come to a conclusion. As
Nathaniel does feel strongly about it, does anyone object to either:
A. Leaving the source dir off sys.path when loading the hooks, and
making a special backend which exposes hooks from the CWD.
B. Adding another key in the TOML file to control whether hooks can be
loaded from the source dir.
I can live with either, but I prefer A, because it means fewer options
to describe in the spec.
---------- Forwarded message ----------
From: "xoviat" <xoviat(a)gmail.com>
Date: Aug 24, 2017 8:39 AM
Subject: Re: [Distutils] PEP 517 again
To: "Thomas Kluyver" <thomas(a)kluyver.me.uk>
That's actually the general argument against exceptions and why golang
doesn't have them. I have not seen notimplemented used in the wild ever
Also, regarding the build_dir argument, it won't be used in pip. The pip
devs may have said that but it's too much work to support now and most of
them seem to have a shortage of time. Pip will be copying to a temporary
directory for a long time.
On Aug 24, 2017 8:28 AM, "Thomas Kluyver" <thomas(a)kluyver.me.uk> wrote:
> On Thu, Aug 24, 2017, at 02:21 PM, xoviat wrote:
> I mean is this golang or Python? In Python, you raise notimplementederror.
> But there's a NotImplemented singleton in Python as well. The argument for
> using a return value is that the hook code has to deliberately return that,
> whereas a NotImplementedError could bubble up from some internal call, in
> which case it should really be registered as an error.
---------- Forwarded message ----------
From: xoviat <xoviat(a)gmail.com>
Date: 2017-08-24 14:26 GMT-05:00
Subject: Re: [Distutils] Fwd: Re: PEP 517 again
To: Thomas Kluyver <thomas(a)kluyver.me.uk>
And also, are Nick and Paul okay with this?
2017-08-24 14:24 GMT-05:00 xoviat <xoviat(a)gmail.com>:
> So in summary:
> - NotImplementedError
> - Remove build_dir and specify that no modifications can be made to the
> source directory
> On the third issue:
> I understand where Nathaniel is coming from in regards to sys.path. But
> there should at least be an option for this because of for example, numpy:
> numpy builds itself with it's own build system that is defined in the
> source directory. If you remove '' from sys.path, then numpy cannot build
> itself because there is no other numpy.distutils package.
> 2017-08-24 14:00 GMT-05:00 Thomas Kluyver <thomas(a)kluyver.me.uk>:
>> On Thu, Aug 24, 2017, at 07:30 PM, Leonardo Rochael Almeida wrote:
>> It looks like a lot of trouble for a feature that is designed to solve a
>> problem for a very thin intersection of the Venn diagram of people who:
>> a. wants to quickly iterate while hacking a package
>> b. doesn't want to bother learning how to trigger the incremental build
>> system of that package or how to install it in editable mode.
>> For better or worse, editable installs exist and don't need to be defined
>> by PEP 517 just to handle this case. I'm not convinced it's worth the
>> additional complexity of the spec just for that.
>> This is a good point. Given that developers can use the build tool
>> directly for development purposes, rather than going through the PEP 517
>> interface, we don't need to prioritise that use case for the spec.
>> So for the time being, I'd prefer to say that the build_wheel() hook
>> should not create or modify files in the source directory. Backends may
>> keep their own caches elsewhere to speed up repeated builds, but any more
>> provisions for developer use cases should be left for a future PEP. And
>> developers can invoke enscons/flit/... directly in the meantime.
>> Distutils-SIG maillist - Distutils-SIG(a)python.org
First time emailer, so please be kind. Also, if this is not the right
mailing list for PyPA talk, I apologize. Please point me in the right
direction if so (Brett Canon pointed me here). The main reason I have
emailed here is I believe it may be PEP time to standardize the JSON
metadata that PyPI makes available, like what was done for the `'simple
API` described in PEP503.
I've been doing a bit of work on `bandersnatch` (I didn't name it), which
is a PEP 381 mirroring package and wanted to enhance it to also mirror the
handy JSON metadata PyPI generates and makes available @
I've done a PR on bandersnatch as a POC that mirrors both the PyPI
directory structure (URL/pypi/PKG_NAME/json) and created a standardizable
URL/json/PKG_NAME that the former symlinks to (to be served by NGINX / some
other proxy). I'm also contemplating naming the directory 'metadata' rather
than JSON so if some new hotness / we want to change the format down the
line we're not stuck with json as the dirname. This PR can be found here:
My main use case is to write a very simple async 'verifier' tool that will
crawl all the JSON files and then ensure the packages directory on each of
my internal mirrors (I have a mirror per region / datacenter) have all the
files they should. I sync centrally (to save resource on the PyPI
infrastructure) and then rsync out all the diffs to each region /
datacenter, and under some failure scenarios I could miss a file or many.
So I feel using JSON pulled down from the authoritative source will allow
an async job to verify the MD5 of all the package files on each mirror.
What are peoples thoughts here? Is it worth a PEP similar to PEP503 going
forward? Can people enhance / share some thoughts on this idea.
Cooper Ry Lees
Previously, the attempt to move setuptools off of vendored dependencies
failed because it was not done correctly: install_requires was set to the
vendored dependencies but not setup_requires, which would have been
required to correctly specify dependencies. However, setup_requires would
have introduced circular dependencies that are impossible to correctly
resolve, so that would have simply shifted an impossible problem onto pip.
The principle issue then, is that setuptools requires setuptools to build
itself. And although, this problem was previously not worth solving because
of 'setup.py install', it is now not a difficult problem to solve if we
rethink slightly what should be included in the setuptools respository:
- Rather than re-generating egg_info on each setup.py invocation, we can
cache dist-info in the setuptools respository.
- We can implement a simple entry_points script that updates dist-info
based on the contents of setup.py, which is only runnable of course after
setuptools is installed
- We can implement the reference build backend for setuptools: simply copy
the files and dist-info into a new compliant zip archive
Viola! Setuptools is no longer required to build setuptools, and the
installation process is fully compliant with Python PEPs.