> So both the abstract build system PEP and donalds setup.py interface
> depend on having a bootstrap dependency list written into a file in
> the source tree.
your build PEP said stuff like this "Additional data *may* be included,
but the ``build_requires`` and ``metadata_version`` keys must be present"
that leads me to think you need more than just this specification of a
I read the RoadMap (Thank you Marcus Smith) and came across this:
> An effort to integrate PyPI with the “The Update Framework” (TUF). This is specified in PEP458
I see a trend to immutable systems everywhere. Updates are a pain. Building
new systems is easier. With current hardware and good software it is easier
to build new systems instead of updating existing systems.
It is like from pets to cattle:
- pets: you give them names and care for them (do updates)
- cattle: you give them numbers and if they get ill you get rid of them.
Maybe I am missing something. But why is there an
effort to create "The Update Framework”, and why integrate
it with pypi?
This is kind-of related to the othe thread:
"Remove distutils, was: ..."
But more specific, so I thought I'd start a new one.
Here are my thoughts:
We had distutils -- there was a lot it didn't do that the "Masses" needed,
so setuptools was born. It proved to be useful to a lot of people, and grew
a large userbase....
But a lot was "wrong" with setuptools -- most prominently (in my mind
anyway) that it put too many kind-sorta orthogonal stuff into one package:
building, installing, distributing, managing version, managing
dependencies, managing non-python resources, (and others??). And we didn't
like how easy-install installed things :-)
So distribute, and pip, and wheel, and now a new backward compatible
setuptools was born.
But we still have a bunch of should be orthogonal stuff tangled up
together. In particular, I find that I often find easy-install getting
invoked when I don't want ot to, and I get those darn eggs scattered all
over the place, and easy_install.pth, and ????
I think if I am really careful about what I invoke when, this could be
avoided, but the reality is that I've been dealing with this for years, and
am trying desperately to do things the "right, modern" way, and I still get
ugliness. I seriously doubt that I am alone.
So -- here's my thought:
I think we have it pretty well mapped out what functionality belongs where:
one system for building packages (setuptools?)
one system for installing packages and managing dependencies (pip)
one system (really standard) for metadata and distributing packages (wheel)
[I'm just whipping this out off the top of my head, I'm sure we'd need to
more clearly define what belongs where]
So why not have a setuptools-lite that only does the building? We need to
keep the full over-burdened setuptools around, because lot sof folks are
using those features. But for those of us that are doing something fairly
new, and don't want to use stuff that setuptools "shouldn't" be doing, I'd
love to have a setuptools-lite that only did what I really need, and was
guaranteed NOT to accidentally introduce easy_install, etc...
This seems to me to be a way to go forward -- as it is we'll have people
using setuptools features that they "shouldn't" forever, and never be able
to move to a cleaner system.
Or maybe a flag:
That would make the dependencies easier -- i.e. pip depends on some of
setuptools being there -- hard to say that it needs either setuptools OR
Maybe I'm missing something here, but if the goal is for there to be one
way to do things, let's have a tool chain that only does things one way.....
Christopher Barker, Ph.D.
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
So I've been reading the various threads (well trying to, there's a lot going
on in all of them) and I'm trying to reconile them all in my head and sorting
out how things are going to look if we add them. I'm worried about adding more
complication to an already underspecified format and making it a lot harder for
tooling to work with these things.
I'm wondering if there isn't a smaller step we can take towards allowing better
interopability with diffrent build systems while also making the situation
clearer when someone is still using distutils or setuptools. For the sake of
argument, let's say we do something like this:
Standardize a key in setup.cfg that can be used for setup_requires. This will
be a static list of items which need to be installed in order to execute the
setup.py. Unlike the current key (which is inside of setup.py) you don't have
to deal with problems of trying to defer the import of the items or dealing
with setuptools at all since your setup.py will not be invoked until after
those items are installed. For people who want to continue to use setuptools,
this file is trivial to parse with the standard library, so they can actually
parse it and stick it into a setup_requires as well which will have the benefit
that the tool calling the setup.py can handle those dependencies but still have
it working on older versions of those tools without requiring duplication.
This can also help work to make setuptools less special. Right now you
basically have to just manually install setuptools prior to installing anything
that uses it (or at all for something like pip when installing from sdist).
With this we could pretty easily make it so that the rule is something like
"If there is a setup.cfg:setup_requires, then assume that everything that is
needed to execute the setup.py is listed there, else assume an implicit
'setuptools' requirement". This means that end users won't need to have
setuptools installed at all, because we'll treat it (mostly) like just another
build tool. This would also (for the first time) make it possible for things to
depend on a specific version of setuptools instead of having to support every
version of setuptools ever.
I think that could solve the problem of "bootstrapping" the requirements to
execute a setup.cfg and extract that from being implicitly handled by
setuptools. That still doesn't handle the problem of making it possible to
actually invoke the now installed build system.
I think that what we should do is figure out the minimal interface we need from
a ``setup.py`` based on what already exists. This would be a much smaller API
surface than what exists today and wouldn't (ideally) include anything that is
an implementation detail of setuptools. We would also need to standard on what
flags an arguments the various commands of setup.py MUST accept. This would of
course not require a setup.py to implement _only_ that particular interface so
additional commands that setuptools already provide can stay, they just won't
be required. Off the top of my head, I think we'd probably want to have the
``bdist_wheel``, ``sdist``, and ``build`` commands. We would also need
something like ``egg_info``, but I think we could probably diverge a bit from
what setuptools does now and make it ``dist_info`` so it's backed by the same
standard that Wheel is already. I think that these four commands can support
everything we actually _need_ to support. This isn't everything we actually
use today, but since this would be opt-in, we can actually be more restrictive
about what build/installation tools would call and what level of fallback we
need to support for them.
The way it would work when what we have available is a sdist is something like
We download a sdist and we notice that there is a setup.cfg:setup_requires.
This toggles a stricter mode of operation where we no longer attempt to do as
many fallbacks or hacks to try and work around various broken shims with
setuptools. We read this key and install those items needed to execute the
setup.py. Once we do that, then pip would invoke ``setup.py bdist_wheel`` and
build a wheel from that . Once we have a wheel built, we'll feed that data
back into the resolver  and use the runtime dependency information from
within that wheel to continue resolving the dependencies.
We have an "arbitrary directory" (VCS, local FS, whatever) on disk that is not
being installed in editable. In this case we'll call ``setup.py sdist`` first,
then feed that into the above.
We have an "arbitrary directory" (VCS, local FS, whatever) on disk that is
being installed as an editable. In this case, we'll call
``setup.py build --inplace`` first, then do something to ensure that the
inplace directory is on sys.path. This is currently undefined because I don't
know exactly what we'd need to do to make this work, but I think it should be
possible and will be more consistent. We'll probably also need something like
``setup.py egg_info`` or ``setup.py dist_info``, but balancing backwards compat
with removing setuptools specifics is something we'd need to figure out.
So why do I think something like this would be better?
* It uses interfaces that are already there (for the most part) which means
that it's easier for people to adapt their current tooling to it, and to do
it in a way that will work with existing legacy packages.
* It defines an interface that build systems must adhere too, instead of
defining an interface that lets you query for how you actually interact with
the build system (sort of a plugin system). This removes indirection and in
my experience, interfaces are generally less problematic then "plugins".
* It doesn't drastically change the mechanics of how the underlying thing works
so it's a much smaller iteration on existing concepts rather than throwing
everything out and starting from scratch.
* It moves us towards an overall installation "path" that I think will be
benefical for us and will help to reduce the combinatorial explosion of ways
that a set of dependencies can be installed.
* It will be easier to integrate setuptools into this new system (since it
largely already implements it) which means that it's easier for us to move
projects to upgrade piece meal to this new system since it would only require
dropping a ``setup.cfg`` with a ``setup_requires``. This also means we could
adjust setuptools (like adding a dist_info command) and have the implicit
setup.cfg:setup_requires be 'setuptools>=somever' instead of just
What this doesn't solve:
* This essentially doesn't solve any of the dynamic vs static metadata issues
with the legacy sdist format, but that's OK because it's just a small
layering (a new setup.cfg feature) of a new feature combined with just
standardizing what already exists. Drastically altering the sdist format to
try and shoe-horn static metadata in there is probably not something that is
going to work well in practice (and needs a more thought out, newer format).
* Dynamically specified _build_ requirements (or really, build requirements at
all other than via setup_requires). You can sort of kludge dynamically
specified build requirements by making a sort of meta-package that you put in
your static setup_requires that generats dynamic runtime requirements. I
think keeping this step simple though and work on enabling breaking the
dependency on setuptools/distutils in this step and then waiting to see if
that is "enough" or if we need to layer on additional features (like dynamic
or otherwise seperately declared build requirements).
 This isn't related to the Wheel cache. In this stricter mode we would only
ever build a wheel and install from that. If caching is on then we'll save
that wheel and reuse it next time, if caching is not on then we'll just
throw that wheel away at the end of the run.
 Pretending for a minute that we have a real resolver.
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
This was shown to me today, and I thought I’d share it here since it pretty much applies 100% to what we’re doing on this list too (other than the fact the examples are from Java) - https://www.youtube.com/watch?v=2y5Pv4yN0b0
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Caught up on distutils-sig this morning - I don't have any additional
comments on Robert's and Nathaniel's draft PEPs except to say "I really
like the look of where they're going, and look forward to reading the next
iterations on them that take into account the feedback already received" :)
The idea of treating extras as subdistributions with their own wheel
metadata also sounds promising (and analogous to building multiple binary
RPMs from a single source RPM).
>From an sdist metadata perspective, though, I think the right thing to do
is to descope PEP 426 to just the stuff we *need* for the build system
improvements, and defer everything else (e.g. JSON-LD, SPDX, richer
dependency semantics, etc) to a future metadata 3.0 proposal (or
potentially metadata extensions, or 2.x format updates).
The one major enhancement I think would be worth keeping is the metadata
extension system (including mandatory extension support), since that
provides a way to experiment with ideas that we may later standardise in
I'm not sure when I'd have time to work on that myself though, so I'm
definitely open to expressions of interest in taking a hatchet to the PEP.
Nathaniel, Robert, perhaps you'd be interested in that as part of the build