<div dir="ltr"><div>This is a nice idea.</div><div><br></div><div>It makes it easy to fix problematic `setup.py` files like this [1]:</div><div><br></div><div>```</div><div><div>from distutils.core import setup, Extension</div><div>import numpy</div><div><br></div><div>setup(ext_modules=[Extension("_cos_doubles",</div><div> sources=["cos_doubles.c", "cos_doubles.i"],</div><div> include_dirs=[numpy.get_include()])])</div></div><div><br></div><div>```</div><div>[1] Documented here <a href="http://www.scipy-lectures.org/advanced/interfacing_with_c/interfacing_with_c.html">http://www.scipy-lectures.org/advanced/interfacing_with_c/interfacing_with_c.html</a></div><div><br></div><div>Just add a setup.cfg with `numpy` in the `setup_requires` key and done. No more `ImportError: No module named numpy` on an empty virtualenv.</div><div><br></div><div>A small layer on top of this idea could also mean a fix to people/systems who still do `setup.py install` (Debian current tooling was mentioned).</div><div><br></div><div>Just change the `setup.py` above to:</div><div><br></div><div>```</div><div><div>from setuptools import setup, Extension, pip_bootstrap</div><div>pip_bootstrap('setup.cfg')</div><div>import numpy</div><div><br></div><div>setup(ext_modules=[Extension("_cos_doubles",</div><div> sources=["cos_doubles.c", "cos_doubles.i"],</div><div> include_dirs=[numpy.get_include()])])</div></div><div><br></div><div>```</div><div class="gmail_extra"><br></div><div class="gmail_extra">The `pip_boostrap` function would:</div><div class="gmail_extra"><br></div><div class="gmail_extra"> - Install pip as if it was a `setup_requires` (i.e., it gets dumped in the local directory, or somewhere more hygienic, but not in `site-packages`), maybe using `get-pip` so we can actually remove the `install` functionality from setuptools in the future</div><div class="gmail_extra"><br></div><div class="gmail_extra"> - run `pip bootstrap setup.cfg`, so that pip installs `numpy` and any other setup_requires declared in setup.cfg (dumped in the local directory like today or somewhere more hygienic, but placing them on sys.path)</div><div class="gmail_extra"><br></div><div class="gmail_extra"> - proceed as usual</div><div class="gmail_extra"><br></div><div class="gmail_extra">if `pip_bootstrap` was actually a (sufficiently magical) module instead of a function, even the second line above could be omitted.</div><div class="gmail_extra"><br></div><div class="gmail_extra">Cheers,</div><div class="gmail_extra"><br></div><div class="gmail_extra">Leo</div><div class="gmail_extra"><br><div class="gmail_quote">On 29 October 2015 at 15:31, Donald Stufft <span dir="ltr"><<a href="mailto:donald@stufft.io" target="_blank">donald@stufft.io</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hello!<br>
<br>
So I've been reading the various threads (well trying to, there's a lot going<br>
on in all of them) and I'm trying to reconile them all in my head and sorting<br>
out how things are going to look if we add them. I'm worried about adding more<br>
complication to an already underspecified format and making it a lot harder for<br>
tooling to work with these things.<br>
<br>
I'm wondering if there isn't a smaller step we can take towards allowing better<br>
interopability with diffrent build systems while also making the situation<br>
clearer when someone is still using distutils or setuptools. For the sake of<br>
argument, let's say we do something like this:<br>
<br>
Standardize a key in setup.cfg that can be used for setup_requires. This will<br>
be a static list of items which need to be installed in order to execute the<br>
setup.py. Unlike the current key (which is inside of setup.py) you don't have<br>
to deal with problems of trying to defer the import of the items or dealing<br>
with setuptools at all since your setup.py will not be invoked until after<br>
those items are installed. For people who want to continue to use setuptools,<br>
this file is trivial to parse with the standard library, so they can actually<br>
parse it and stick it into a setup_requires as well which will have the benefit<br>
that the tool calling the setup.py can handle those dependencies but still have<br>
it working on older versions of those tools without requiring duplication.<br>
<br>
This can also help work to make setuptools less special. Right now you<br>
basically have to just manually install setuptools prior to installing anything<br>
that uses it (or at all for something like pip when installing from sdist).<br>
With this we could pretty easily make it so that the rule is something like<br>
"If there is a setup.cfg:setup_requires, then assume that everything that is<br>
needed to execute the setup.py is listed there, else assume an implicit<br>
'setuptools' requirement". This means that end users won't need to have<br>
setuptools installed at all, because we'll treat it (mostly) like just another<br>
build tool. This would also (for the first time) make it possible for things to<br>
depend on a specific version of setuptools instead of having to support every<br>
version of setuptools ever.<br>
<br>
I think that could solve the problem of "bootstrapping" the requirements to<br>
execute a setup.cfg and extract that from being implicitly handled by<br>
setuptools. That still doesn't handle the problem of making it possible to<br>
actually invoke the now installed build system.<br>
<br>
I think that what we should do is figure out the minimal interface we need from<br>
a ``setup.py`` based on what already exists. This would be a much smaller API<br>
surface than what exists today and wouldn't (ideally) include anything that is<br>
an implementation detail of setuptools. We would also need to standard on what<br>
flags an arguments the various commands of setup.py MUST accept. This would of<br>
course not require a setup.py to implement _only_ that particular interface so<br>
additional commands that setuptools already provide can stay, they just won't<br>
be required. Off the top of my head, I think we'd probably want to have the<br>
``bdist_wheel``, ``sdist``, and ``build`` commands. We would also need<br>
something like ``egg_info``, but I think we could probably diverge a bit from<br>
what setuptools does now and make it ``dist_info`` so it's backed by the same<br>
standard that Wheel is already. I think that these four commands can support<br>
everything we actually _need_ to support. This isn't everything we actually<br>
use today, but since this would be opt-in, we can actually be more restrictive<br>
about what build/installation tools would call and what level of fallback we<br>
need to support for them.<br>
<br>
The way it would work when what we have available is a sdist is something like<br>
this:<br>
<br>
We download a sdist and we notice that there is a setup.cfg:setup_requires.<br>
This toggles a stricter mode of operation where we no longer attempt to do as<br>
many fallbacks or hacks to try and work around various broken shims with<br>
setuptools. We read this key and install those items needed to execute the<br>
setup.py. Once we do that, then pip would invoke ``setup.py bdist_wheel`` and<br>
build a wheel from that [1]. Once we have a wheel built, we'll feed that data<br>
back into the resolver [2] and use the runtime dependency information from<br>
within that wheel to continue resolving the dependencies.<br>
<br>
OR<br>
<br>
We have an "arbitrary directory" (VCS, local FS, whatever) on disk that is not<br>
being installed in editable. In this case we'll call ``setup.py sdist`` first,<br>
then feed that into the above.<br>
<br>
OR<br>
<br>
We have an "arbitrary directory" (VCS, local FS, whatever) on disk that is<br>
being installed as an editable. In this case, we'll call<br>
``setup.py build --inplace`` first, then do something to ensure that the<br>
inplace directory is on sys.path. This is currently undefined because I don't<br>
know exactly what we'd need to do to make this work, but I think it should be<br>
possible and will be more consistent. We'll probably also need something like<br>
``setup.py egg_info`` or ``setup.py dist_info``, but balancing backwards compat<br>
with removing setuptools specifics is something we'd need to figure out.<br>
<br>
So why do I think something like this would be better?<br>
<br>
* It uses interfaces that are already there (for the most part) which means<br>
that it's easier for people to adapt their current tooling to it, and to do<br>
it in a way that will work with existing legacy packages.<br>
<br>
* It defines an interface that build systems must adhere too, instead of<br>
defining an interface that lets you query for how you actually interact with<br>
the build system (sort of a plugin system). This removes indirection and in<br>
my experience, interfaces are generally less problematic then "plugins".<br>
<br>
* It doesn't drastically change the mechanics of how the underlying thing works<br>
so it's a much smaller iteration on existing concepts rather than throwing<br>
everything out and starting from scratch.<br>
<br>
* It moves us towards an overall installation "path" that I think will be<br>
benefical for us and will help to reduce the combinatorial explosion of ways<br>
that a set of dependencies can be installed.<br>
<br>
* It will be easier to integrate setuptools into this new system (since it<br>
largely already implements it) which means that it's easier for us to move<br>
projects to upgrade piece meal to this new system since it would only require<br>
dropping a ``setup.cfg`` with a ``setup_requires``. This also means we could<br>
adjust setuptools (like adding a dist_info command) and have the implicit<br>
setup.cfg:setup_requires be 'setuptools>=somever' instead of just<br>
'setuptools'.<br>
<br>
What this doesn't solve:<br>
<br>
* This essentially doesn't solve any of the dynamic vs static metadata issues<br>
with the legacy sdist format, but that's OK because it's just a small<br>
layering (a new setup.cfg feature) of a new feature combined with just<br>
standardizing what already exists. Drastically altering the sdist format to<br>
try and shoe-horn static metadata in there is probably not something that is<br>
going to work well in practice (and needs a more thought out, newer format).<br>
<br>
* Dynamically specified _build_ requirements (or really, build requirements at<br>
all other than via setup_requires). You can sort of kludge dynamically<br>
specified build requirements by making a sort of meta-package that you put in<br>
your static setup_requires that generats dynamic runtime requirements. I<br>
think keeping this step simple though and work on enabling breaking the<br>
dependency on setuptools/distutils in this step and then waiting to see if<br>
that is "enough" or if we need to layer on additional features (like dynamic<br>
or otherwise seperately declared build requirements).<br>
<br>
<br>
[1] This isn't related to the Wheel cache. In this stricter mode we would only<br>
ever build a wheel and install from that. If caching is on then we'll save<br>
that wheel and reuse it next time, if caching is not on then we'll just<br>
throw that wheel away at the end of the run.<br>
<br>
[2] Pretending for a minute that we have a real resolver.<br>
<br>
-----------------<br>
Donald Stufft<br>
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA<br>
<br>
<br>
_______________________________________________<br>
Distutils-SIG maillist - <a href="mailto:Distutils-SIG@python.org">Distutils-SIG@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/distutils-sig" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/distutils-sig</a><br>
</blockquote></div><br></div></div>