<div dir="ltr">I think that would be very handy, especially making setuptools not a special case. You could get it down to 3 lines in setup.cfg, a file that already exists and already gathers random settings.<div><br></div><div><div>Case 1, pip both requires a single package, and looks for an entry point inside that package to build.</div></div><div><br></div><div>[bootstrap]</div><div>build-with:buildamajig</div><div><br></div><div>buildamajig package:</div><div><br></div><div>class Builder:</div><div> def metadata(target="directory"): ...</div><div> def wheel(): ...</div><div><br></div><div>In buildamajig's setup.py:</div><div><br></div><div>entry_points = { "pip.builder" : "builder = buildamajig:Builder" }</div><div><br></div><div><br></div><div>Case 2</div><div><br></div><div>[bootstrap]</div><div>requires=pbr</div><div><br></div><div>pip makes sure pbr is importable inside setup.py, but uses setuptools to build the package. (Implicit build-with=pip_setuptools_adapter).</div><div><br></div><div><br></div><div>Case 3</div><div><br></div><div>[bootstrap]</div><div>build-with=buildtool:Foobar</div><div><br></div><div>pip does not require a dependency, but noticing the : in the argument executes "from buildtool import Foobar ..." with the setup.cfg directory on sys.path; alternative: different key "builder=module:Object". Useful for embedded build scripts or anywhere entry points would be impractical.</div><div><br></div><div><br></div><div>Case 4</div><div><div><br></div><div>[bootstrap]</div><div>requires=random-dependency</div></div><div>build-with=thing:Klass</div><div><br></div><div>Legal to add requirements whether or not setup.py is used.</div><div><br></div><div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Thu, Oct 29, 2015 at 1:32 PM Donald Stufft <<a href="mailto:donald@stufft.io" target="_blank">donald@stufft.io</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello!<br>
<br>
So I've been reading the various threads (well trying to, there's a lot going<br>
on in all of them) and I'm trying to reconile them all in my head and sorting<br>
out how things are going to look if we add them. I'm worried about adding more<br>
complication to an already underspecified format and making it a lot harder for<br>
tooling to work with these things.<br>
<br>
I'm wondering if there isn't a smaller step we can take towards allowing better<br>
interopability with diffrent build systems while also making the situation<br>
clearer when someone is still using distutils or setuptools. For the sake of<br>
argument, let's say we do something like this:<br>
<br>
Standardize a key in setup.cfg that can be used for setup_requires. This will<br>
be a static list of items which need to be installed in order to execute the<br>
setup.py. Unlike the current key (which is inside of setup.py) you don't have<br>
to deal with problems of trying to defer the import of the items or dealing<br>
with setuptools at all since your setup.py will not be invoked until after<br>
those items are installed. For people who want to continue to use setuptools,<br>
this file is trivial to parse with the standard library, so they can actually<br>
parse it and stick it into a setup_requires as well which will have the benefit<br>
that the tool calling the setup.py can handle those dependencies but still have<br>
it working on older versions of those tools without requiring duplication.<br>
<br>
This can also help work to make setuptools less special. Right now you<br>
basically have to just manually install setuptools prior to installing anything<br>
that uses it (or at all for something like pip when installing from sdist).<br>
With this we could pretty easily make it so that the rule is something like<br>
"If there is a setup.cfg:setup_requires, then assume that everything that is<br>
needed to execute the setup.py is listed there, else assume an implicit<br>
'setuptools' requirement". This means that end users won't need to have<br>
setuptools installed at all, because we'll treat it (mostly) like just another<br>
build tool. This would also (for the first time) make it possible for things to<br>
depend on a specific version of setuptools instead of having to support every<br>
version of setuptools ever.<br>
<br>
I think that could solve the problem of "bootstrapping" the requirements to<br>
execute a setup.cfg and extract that from being implicitly handled by<br>
setuptools. That still doesn't handle the problem of making it possible to<br>
actually invoke the now installed build system.<br>
<br>
I think that what we should do is figure out the minimal interface we need from<br>
a ``setup.py`` based on what already exists. This would be a much smaller API<br>
surface than what exists today and wouldn't (ideally) include anything that is<br>
an implementation detail of setuptools. We would also need to standard on what<br>
flags an arguments the various commands of setup.py MUST accept. This would of<br>
course not require a setup.py to implement _only_ that particular interface so<br>
additional commands that setuptools already provide can stay, they just won't<br>
be required. Off the top of my head, I think we'd probably want to have the<br>
``bdist_wheel``, ``sdist``, and ``build`` commands. We would also need<br>
something like ``egg_info``, but I think we could probably diverge a bit from<br>
what setuptools does now and make it ``dist_info`` so it's backed by the same<br>
standard that Wheel is already. I think that these four commands can support<br>
everything we actually _need_ to support. This isn't everything we actually<br>
use today, but since this would be opt-in, we can actually be more restrictive<br>
about what build/installation tools would call and what level of fallback we<br>
need to support for them.<br>
<br>
The way it would work when what we have available is a sdist is something like<br>
this:<br>
<br>
We download a sdist and we notice that there is a setup.cfg:setup_requires.<br>
This toggles a stricter mode of operation where we no longer attempt to do as<br>
many fallbacks or hacks to try and work around various broken shims with<br>
setuptools. We read this key and install those items needed to execute the<br>
setup.py. Once we do that, then pip would invoke ``setup.py bdist_wheel`` and<br>
build a wheel from that [1]. Once we have a wheel built, we'll feed that data<br>
back into the resolver [2] and use the runtime dependency information from<br>
within that wheel to continue resolving the dependencies.<br>
<br>
OR<br>
<br>
We have an "arbitrary directory" (VCS, local FS, whatever) on disk that is not<br>
being installed in editable. In this case we'll call ``setup.py sdist`` first,<br>
then feed that into the above.<br>
<br>
OR<br>
<br>
We have an "arbitrary directory" (VCS, local FS, whatever) on disk that is<br>
being installed as an editable. In this case, we'll call<br>
``setup.py build --inplace`` first, then do something to ensure that the<br>
inplace directory is on sys.path. This is currently undefined because I don't<br>
know exactly what we'd need to do to make this work, but I think it should be<br>
possible and will be more consistent. We'll probably also need something like<br>
``setup.py egg_info`` or ``setup.py dist_info``, but balancing backwards compat<br>
with removing setuptools specifics is something we'd need to figure out.<br>
<br>
So why do I think something like this would be better?<br>
<br>
* It uses interfaces that are already there (for the most part) which means<br>
that it's easier for people to adapt their current tooling to it, and to do<br>
it in a way that will work with existing legacy packages.<br>
<br>
* It defines an interface that build systems must adhere too, instead of<br>
defining an interface that lets you query for how you actually interact with<br>
the build system (sort of a plugin system). This removes indirection and in<br>
my experience, interfaces are generally less problematic then "plugins".<br>
<br>
* It doesn't drastically change the mechanics of how the underlying thing works<br>
so it's a much smaller iteration on existing concepts rather than throwing<br>
everything out and starting from scratch.<br>
<br>
* It moves us towards an overall installation "path" that I think will be<br>
benefical for us and will help to reduce the combinatorial explosion of ways<br>
that a set of dependencies can be installed.<br>
<br>
* It will be easier to integrate setuptools into this new system (since it<br>
largely already implements it) which means that it's easier for us to move<br>
projects to upgrade piece meal to this new system since it would only require<br>
dropping a ``setup.cfg`` with a ``setup_requires``. This also means we could<br>
adjust setuptools (like adding a dist_info command) and have the implicit<br>
setup.cfg:setup_requires be 'setuptools>=somever' instead of just<br>
'setuptools'.<br>
<br>
What this doesn't solve:<br>
<br>
* This essentially doesn't solve any of the dynamic vs static metadata issues<br>
with the legacy sdist format, but that's OK because it's just a small<br>
layering (a new setup.cfg feature) of a new feature combined with just<br>
standardizing what already exists. Drastically altering the sdist format to<br>
try and shoe-horn static metadata in there is probably not something that is<br>
going to work well in practice (and needs a more thought out, newer format).<br>
<br>
* Dynamically specified _build_ requirements (or really, build requirements at<br>
all other than via setup_requires). You can sort of kludge dynamically<br>
specified build requirements by making a sort of meta-package that you put in<br>
your static setup_requires that generats dynamic runtime requirements. I<br>
think keeping this step simple though and work on enabling breaking the<br>
dependency on setuptools/distutils in this step and then waiting to see if<br>
that is "enough" or if we need to layer on additional features (like dynamic<br>
or otherwise seperately declared build requirements).<br>
<br>
<br>
[1] This isn't related to the Wheel cache. In this stricter mode we would only<br>
ever build a wheel and install from that. If caching is on then we'll save<br>
that wheel and reuse it next time, if caching is not on then we'll just<br>
throw that wheel away at the end of the run.<br>
<br>
[2] Pretending for a minute that we have a real resolver.<br>
<br>
-----------------<br>
Donald Stufft<br>
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA<br>
<br>
<br>
_______________________________________________<br>
Distutils-SIG maillist - <a href="mailto:Distutils-SIG@python.org" target="_blank">Distutils-SIG@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/distutils-sig" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/distutils-sig</a><br>
</blockquote></div></div></div>