[Distutils] PEP 517 again

[Nathaniel Smith]

On Fri, Aug 25, 2017 at 2:26 PM, xoviat <xoviat at gmail.com> wrote:
>> I'm more or less persuaded by Nathaniel's argument that the source
>> directory shouldn't be on sys.path
>
> I do too. There should be an option in pyproject.toml to disable this
> behavior though so that numpy can build itself.

My original proposal was to leave the srcdir off of sys.path, and then
have a key in pyproject.toml like:

[build-system]
backend-python-path = ["."]

Frontends would then do something like:

os.chdir(srcdir)
# This line is new to handle the above proposal:
sys.path[:0] = [abspath(p) for p in
config["build-system"].get("backend-python-path", [])]
backend = resolve_entrypoint(config["build-system"]["build-backend"])

I don't have a strong opinion on whether we put this into PEP 517
(it's pretty light weight and doesn't interact with any other features
AFAICT), or make it a followup PEP, or start out by deferring this
option to a dedicated build backend, like:

[build-system]
requires = ["override_backend_path"]
build-backend = "override_backend_path"

[tool.override_backend_path]
python-path = ["."]
real-backend = "my_awesome:backend"

These are all pretty similar.

I think the big question for debate is: should sys.path be
configurable, or not configurable? IIUC, the main argument for putting
the source directory on the path was that extra configuration options
are annoying, so we don't want one of those, but we do want to support
in-tree backends (even though we expect that most projects won't use
this), so we had better put the srcdir on sys.path.

My feeling is that in practice, though, that the "no configuration,
srcdir always on sys.path" approach is not going to hold up. So first,
obviously, the hack above works just fine to make it configurable even
if we don't put it in any PEP, so.... in fact it's configurable no
matter what. Plus, sooner or later, someone's going to say "hey
distutils-sig, I have a build backend package that I want to be able
to bootstrap itself, AND I want to put my package inside src/ for
reasons [1], wouldn't it be nice if I could put src/ on sys.path
without jumping through hoops?". Or someone will release a new version
of their build backend that adds a new dependency, or one of their
transitive dependencies will release a new version that adds a new
dependency, and it collides with some already-released package that
uses that build-backend, and the project suffering the collision gets
annoyed at being told they need to rearrange their source tree
(retroactively, in already released versions!). And they'll come here
and say "hey distutils-sig, can solve this problem once and for all?".
And we'll be like... uh, fixing this would take what, <5 lines of new
code in pip? Kinda hard to argue with that.

So... it'll be configurable, one way or another.

And if it's configurable... then the question is about whether the
default configuration: is it srcdir on sys.path ("opt-out"), or srcdir
not on sys.path ("opt-in"). And it seems to me that in this case, all
the standard criteria say it should be opt-in.

If it's opt-in, then everyone using build backends distributed on PyPI
-- which we expect to be the vast majority of project -- never have to
think about it and it does the right thing, with no risk of collisions
or anything. In fact the only people who have to think about it are
the ones implementing in-tree backends, and if you're already like,
writing a whole backend and configuring your pyproject.toml to invoke
it, then asking you to add one more line of configuration is really
not a big deal.

OTOH if it's opt-out, then it becomes Yet Another Bad Packaging
Default, where conscientious package authors will fret about the risk
of collisions and write blog posts about how every project needs to
make sure to opt-out of this as a Best Practice, and I am so, so tired
of those.

-n

[1] https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure

-- 
Nathaniel J. Smith -- https://vorpus.org