On May 8, 2016, at 9:13 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
So let's reduce our scope to: "We want *current users* of d2to1 and pbr to be able to declare those dependencies to pip and other installation tools in a way that avoids the implicit invocation of easy_install by setuptools"
I think it's less pbr and d2to1 that we're designing for here (although they'll benefit from it) but more like numpy.distutils. The reasoning is that pbr and such have already been able to architecture themselves so that they work within the current system. One of the goals here is to enable projects that haven't been able to (or can't reasonably) be written to work as a setuptools extension to be used here without requiring a manual pre-installation step. The other side of this is anytime you have dependencies that aren't installed by pip (such as setup_requires being installed by setuptools) you end up with a confusing situation where settings don't get passed down into the "inner" installer (like ``--index-url``) or when they have different resolution algorithms (like setuptools supporting pre-releases by default) or when they support different formats (like pip not supporting eggs, but setuptools not supporting wheels). However, while I think that a new format is more work just to make sure we don't back ourselves into a corner, I also don't think it's so much more work that it's really worth it to skip doing it now. Their isn't a whole lot of ways to solve the very narrow problem of "we need some dependencies installed prior to building". We need a field in a file that takes a list of PEP 508 style specifiers. We can skip any of the build system support right now and add it in later (and when we add it in, we can just make it so that a lack of a declaration is an implicit setuptool). So the only real things we need to decide on are: 1) What format to use? For this I think that it's been pretty clearly that TOML has had the greatest amount of support here and I think it represents the best set of trade offs for us. 2) What do we want to call this file? This one is pure bikeshed but also a bit important since this is one of the things that will be the hardest to change once we actually pick the name. Ideas I've seen so far (using the toml extension) are: * pypa.toml - This one is specific to us which is good, but it's also a bit bad in that I don't think it's very widely known what the PyPA even is and I think that the PyPA works better if it's just sort of the thing in the background. * pybuild.toml - This one might be a bit too oriented towards building rather than all of the possible uses of this file, but the bigger problem with it I think is that hte name clashes with pybuild on Debian which is their tool used for building Python packages. * pip.toml - Also specific to us which is good, but I think it's a bit too overly specific maybe? One of our goals has been to make this stuff not dependent on a specific implementation so that we can replace it if we need to (as we more or less replaced distutils with setuptools, or easy_install with pip) however this also isn't exactly depending on an implementation-- just the name of a particular implementation. It does have the benefit that for a lot of people they associate "pip" with everything packaging related (e.g. I see people calling them "pip packages" now). * pymeta.toml - This one might be reasonable, it's a bit generic but at least it has the ``py`` prefix to tie it to us. Not much to say about it otherwise. Overall from those names, I think I probably like pymeta.toml the best (or maybe just ``meta.toml`` if we don't like the py prefix) but maybe other people have ideas/opinions that I haven't seen yet. 3) How do we structure the file and what keys do we use for just this part of the overall feature we're hoping to eventually get (and what semantics do we give these keys). We could just call it ``setup_requires``, but I think that's a bad name for something we're not planning on getting rid of since one of the eventual goals is to make ``setup.py`` itself optional. Another option is ``build_requires`` but that isn't quite right because we don't *just* need these dependencies for the build step (``setup.py bdist_wheel``) but also for the metadata step (``setup.py egg_info``). Although perhaps it doesn't really matter. Another option is to call them ``bootstrap_requires`` though it's a bit wonky to cram *build* requirements into that. I guess if I were deciding this I would just call it build requirements because the nature of ``setup.py`` is that pretty much anything you need to execute ``setup.py`` is going to be a requirement for all ways you invoke ``setup.py`` (metadata and build and the legacyish direct to install). We could then say that if a project is missing this new file, then there is an implicit ``build_requires`` of ``["setuptools", "wheel"]`` but that as soon as someone defines this file that they are expected to accurately list all of their build dependencies in it. Overall, my suggestion here would be to have a file called ``pymeta.toml`` (or ``meta.toml``) and have it look like:: [dependencies] build = [ "setuptools", "wheel>=1.0", ] If at some point we decide we need to add a bootstrap_requires (and then the ability to add dynamic build requirements) we can do that by just saying that if you plan on having dynamic build requirements, you need to omit the build key under the [dependencies] section. This same thing could be used for other kinds of dependencies too (I've come around to the idea that you can't always declare a static set of dependencies in a sdist, but you can in a wheel) so that at some point in the future we could add additional keys like ``runtime`` to this where, if declared, will be assumed to be a static set of dependencies for that type of dependency. This also doesn't prevent us from moving more of the metadata to this file in the future either. For an example, we could (at some point in the future, not now-- we shouldn't get bogged down in this now, it's just an example of this not tying us down too much) add something like:: [package] name = "mycoolpackage" version = "1.0" [dependencies] build = ["setuptools", "wheel>=1.0"] runtime = ["requests==2.*"] ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA