Mailman 3 October 2015 - Distutils-SIG

A smaller step towards de-specializing setuptools/distutils
by Donald Stufft 02 Nov '15

02 Nov '15

Hello! So I've been reading the various threads (well trying to, there's a lot going on in all of them) and I'm trying to reconile them all in my head and sorting out how things are going to look if we add them. I'm worried about adding more complication to an already underspecified format and making it a lot harder for tooling to work with these things. I'm wondering if there isn't a smaller step we can take towards allowing better interopability with diffrent build systems while also making the situation clearer when someone is still using distutils or setuptools. For the sake of argument, let's say we do something like this: Standardize a key in setup.cfg that can be used for setup_requires. This will be a static list of items which need to be installed in order to execute the setup.py. Unlike the current key (which is inside of setup.py) you don't have to deal with problems of trying to defer the import of the items or dealing with setuptools at all since your setup.py will not be invoked until after those items are installed. For people who want to continue to use setuptools, this file is trivial to parse with the standard library, so they can actually parse it and stick it into a setup_requires as well which will have the benefit that the tool calling the setup.py can handle those dependencies but still have it working on older versions of those tools without requiring duplication. This can also help work to make setuptools less special. Right now you basically have to just manually install setuptools prior to installing anything that uses it (or at all for something like pip when installing from sdist). With this we could pretty easily make it so that the rule is something like "If there is a setup.cfg:setup_requires, then assume that everything that is needed to execute the setup.py is listed there, else assume an implicit 'setuptools' requirement". This means that end users won't need to have setuptools installed at all, because we'll treat it (mostly) like just another build tool. This would also (for the first time) make it possible for things to depend on a specific version of setuptools instead of having to support every version of setuptools ever. I think that could solve the problem of "bootstrapping" the requirements to execute a setup.cfg and extract that from being implicitly handled by setuptools. That still doesn't handle the problem of making it possible to actually invoke the now installed build system. I think that what we should do is figure out the minimal interface we need from a ``setup.py`` based on what already exists. This would be a much smaller API surface than what exists today and wouldn't (ideally) include anything that is an implementation detail of setuptools. We would also need to standard on what flags an arguments the various commands of setup.py MUST accept. This would of course not require a setup.py to implement _only_ that particular interface so additional commands that setuptools already provide can stay, they just won't be required. Off the top of my head, I think we'd probably want to have the ``bdist_wheel``, ``sdist``, and ``build`` commands. We would also need something like ``egg_info``, but I think we could probably diverge a bit from what setuptools does now and make it ``dist_info`` so it's backed by the same standard that Wheel is already. I think that these four commands can support everything we actually _need_ to support. This isn't everything we actually use today, but since this would be opt-in, we can actually be more restrictive about what build/installation tools would call and what level of fallback we need to support for them. The way it would work when what we have available is a sdist is something like this: We download a sdist and we notice that there is a setup.cfg:setup_requires. This toggles a stricter mode of operation where we no longer attempt to do as many fallbacks or hacks to try and work around various broken shims with setuptools. We read this key and install those items needed to execute the setup.py. Once we do that, then pip would invoke ``setup.py bdist_wheel`` and build a wheel from that [1]. Once we have a wheel built, we'll feed that data back into the resolver [2] and use the runtime dependency information from within that wheel to continue resolving the dependencies. OR We have an "arbitrary directory" (VCS, local FS, whatever) on disk that is not being installed in editable. In this case we'll call ``setup.py sdist`` first, then feed that into the above. OR We have an "arbitrary directory" (VCS, local FS, whatever) on disk that is being installed as an editable. In this case, we'll call ``setup.py build --inplace`` first, then do something to ensure that the inplace directory is on sys.path. This is currently undefined because I don't know exactly what we'd need to do to make this work, but I think it should be possible and will be more consistent. We'll probably also need something like ``setup.py egg_info`` or ``setup.py dist_info``, but balancing backwards compat with removing setuptools specifics is something we'd need to figure out. So why do I think something like this would be better? * It uses interfaces that are already there (for the most part) which means that it's easier for people to adapt their current tooling to it, and to do it in a way that will work with existing legacy packages. * It defines an interface that build systems must adhere too, instead of defining an interface that lets you query for how you actually interact with the build system (sort of a plugin system). This removes indirection and in my experience, interfaces are generally less problematic then "plugins". * It doesn't drastically change the mechanics of how the underlying thing works so it's a much smaller iteration on existing concepts rather than throwing everything out and starting from scratch. * It moves us towards an overall installation "path" that I think will be benefical for us and will help to reduce the combinatorial explosion of ways that a set of dependencies can be installed. * It will be easier to integrate setuptools into this new system (since it largely already implements it) which means that it's easier for us to move projects to upgrade piece meal to this new system since it would only require dropping a ``setup.cfg`` with a ``setup_requires``. This also means we could adjust setuptools (like adding a dist_info command) and have the implicit setup.cfg:setup_requires be 'setuptools>=somever' instead of just 'setuptools'. What this doesn't solve: * This essentially doesn't solve any of the dynamic vs static metadata issues with the legacy sdist format, but that's OK because it's just a small layering (a new setup.cfg feature) of a new feature combined with just standardizing what already exists. Drastically altering the sdist format to try and shoe-horn static metadata in there is probably not something that is going to work well in practice (and needs a more thought out, newer format). * Dynamically specified _build_ requirements (or really, build requirements at all other than via setup_requires). You can sort of kludge dynamically specified build requirements by making a sort of meta-package that you put in your static setup_requires that generats dynamic runtime requirements. I think keeping this step simple though and work on enabling breaking the dependency on setuptools/distutils in this step and then waiting to see if that is "enough" or if we need to layer on additional features (like dynamic or otherwise seperately declared build requirements). [1] This isn't related to the Wheel cache. In this stricter mode we would only ever build a wheel and install from that. If caching is on then we'll save that wheel and reuse it next time, if caching is not on then we'll just throw that wheel away at the end of the run. [2] Pretending for a minute that we have a real resolver. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

10 29

Brian Goetz - Stewardship: the Sobering Parts
by Donald Stufft 01 Nov '15

01 Nov '15

This was shown to me today, and I thought I’d share it here since it pretty much applies 100% to what we’re doing on this list too (other than the fact the examples are from Java) - https://www.youtube.com/watch?v=2y5Pv4yN0b0 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

4 3

Removing the aspirational aspects of PEP 426
by Nick Coghlan 31 Oct '15

31 Oct '15

Caught up on distutils-sig this morning - I don't have any additional comments on Robert's and Nathaniel's draft PEPs except to say "I really like the look of where they're going, and look forward to reading the next iterations on them that take into account the feedback already received" :) The idea of treating extras as subdistributions with their own wheel metadata also sounds promising (and analogous to building multiple binary RPMs from a single source RPM). >From an sdist metadata perspective, though, I think the right thing to do is to descope PEP 426 to just the stuff we *need* for the build system improvements, and defer everything else (e.g. JSON-LD, SPDX, richer dependency semantics, etc) to a future metadata 3.0 proposal (or potentially metadata extensions, or 2.x format updates). The one major enhancement I think would be worth keeping is the metadata extension system (including mandatory extension support), since that provides a way to experiment with ideas that we may later standardise in 3.0. I'm not sure when I'd have time to work on that myself though, so I'm definitely open to expressions of interest in taking a hatchet to the PEP. Nathaniel, Robert, perhaps you'd be interested in that as part of the build interface standardisation? Regards, Nick.

3 4

metadata in sdists (was Re: Second draft of a plan for a new source tree / sdist format)
by Nathaniel Smith 30 Oct '15

30 Oct '15

On Wed, Oct 28, 2015 at 5:32 AM, Daniel Holth <dholth(a)gmail.com> wrote: > Nathaniel, > > I'm not sure what the software is supposed to do with fine grained dynamic > metadata that would make very much sense to the end user. I think you could > probably get away with a single flag Dynamic: true / false. Iff true, pip > runs the dist-info command after installing bootstrap dependencies. You > could still complain if the name & version changed. Of course in a VCS > checkout or during development you probably always want the > regenerate-metadata behavior. So, right, the draft I just posted proposes that sdists should contain static wheel-style metadata where possible, with a fine-grained mechanism for marking specific parts as being dynamically generated at build time. The different motivations that I'm trying to balance here are: - Donald wants the fields that PyPI cares about (name, version, summary, long_description, author, homepage, trove classifiers, etc.) to *always* be statically present in sdists, even if there is some other metadata that can't be made static (i.e., install-requirements) - Robert wants the (name, version, install-requirements) information to be available statically as often as possible, because that's the information that the resolver needs to know when it's considering trying to install some (name, version) pair, and he wants this information to be cheap to access because the resolver may have to backtrack and consider many different versions of the same package. Installing build-requirements and then running egg-info/dist-info is already pretty expensive, so it would be good if install-requirements could be static in the 99% of cases where they are known. - I'm still holding out some small hope of killing off the "egg-info"/"dist-info" step entirely, because any time you have two different operations and it's a bug if they can get out of sync (in this case: the dist-info step, and then the actual wheel building step), then maintaining and testing is a hassle and eventually that bug will happen. Basically it's a violation of DRY -- you have two sources of the same data and they're both supposed to be authoritative. And we know from experience with egg_info that there will be strong temptations for people to cheat, which creates all kinds of headaches. So how can we balance these different design goals? Obviously a new sdist format should provide static metadata for the fields PyPI cares about -- we basically require this already for sdists, we just encode it in a weird out-of-band way during the PyPI upload instead of recording it authoritatively in the sdist itself. Then that leaves a 2x2 space of plausible design options: we can either support dist-info as a separate operation from building, or not; and we can make a best-effort to provide static install-requirements when they're available, or we can not bother and just never provide static install-requirements. None of this matters for 'pip install <source directory>' None of this matters for installations using wheels. None of this matters for installations where the requirements are straightforward to satisfy (e.g. cases where the current pseudo-resolver works okay). None of this affects correctness -- it's purely an optimization. But maybe it's an important optimization in certain specific cases. The case where this matters is like: suppose you just did 'pip install scikit-learn', and scikit-learn requires scipy and numpy. And scipy also requires numpy, so our dependency graph forms a triangle. And let's say that scikit-learn is happy with any version of scipy, but scikit-learn and scipy both have versioned requirements on numpy. So for example, scikit-learn might require numpy (< 1.9). Meanwhile, the latest version of scipy requires numpy (>= 1.9)... so there's no version of numpy that satisfies both at once, which means we can't use this version of scipy. So you have to consider the next-to-latest version of scipy. But then it turns out that it also requires numpy (>= 1.9). So then you have to consider the next-to-next-to-latest version of scipy, which turns out to require numpy (>= 1.8), so phew, we can use numpy = 1.8 to satisfy both of these at once and we're done. And, to make this maximally annoying, it turns out that there is no scipy wheel for your platform, so all this checking of scipy's install-requires has to be done based on whatever information is inside the scipy sdist. And finally, the reason we use scipy in this example is that building scipy turns out to be really slow (~8 minutes on my laptop), definitely an outlier among python packages. There's three ways this can play out: Option 1, "fast": if the scipy sdists have static install-requires metadata, then this is pretty trivial. You might have to download the sdists for the first two non-working versions, but once you have the files you can immediately tell from the metadata that they aren't going to be usable and move on. (Of course, scipy itself probably *can't* have static install-requires metadata, but most packages can.) Option 2, "slow": if the scipy sdists don't have static install-requires metadata, but do support the dist-info operation, then you do something like: - download and unpack scipy-XX.zip - discover that it build-requires numpy. If there's no scipy wheel for your platform, there's presumably no numpy wheel either, so we'll have to build it. - download and build numpy, and install your newly built numpy into scipy's build environment (~2 minutes?) - run scipy's dist-info (a few seconds?) - go back to the top and repeat for the next candidate Option 3, "slowest": if you don't have static install-requires *and* you don't have the dist-info operation, then it's identical to the "slow" option, except instead of dist-info you have to actually run the full scipy build at each step, so it takes ~10 minutes per cycle instead of ~2 minutes per cycle (or maybe a bit less depending on whether the build-requirements work out so that you can re-use the same build of numpy for both attempts, etc.) Now, our four design options break down like this: no dist-info / never record install-requirements in sdist: 100% of sdists are in the "slowest" path no dist-info / record install-requirements in sdist when available: 95% of sdists are on "fast" path, 5% (the ones with dynamic install-requirements, like scipy) are on the "slowest" path dist-info / never record install-requirements in sdist (= current sdist format): 100% of sdists are in the "slow" path dist-info / record install-requirements in sdist when available: 95% of sdists are on "fast" path, 5% are on the "slow" path *** Summary *** There are two different ways to speed up resolution of dependencies involving sdists. static install-requirements are super fast when they work, and their complexity cost is low but non-zero. (We know we want to include lots of other metadata in sdists, and we already have standard ways to represent lots of metadata + install-requirements, so the only new thing is the __SDIST_DYNAMIC__ bit.) dist-info is slower and always works, but has a higher complexity cost. (Every build backend needs to implement this extra operation, and it's a bit fragile.) If you think that: - automagic building from sdists is an important case to support seamlessly and will remain so indefinitely, and - these kinds of complicated conflicts are common, and - it's important to optimize them to be resolved as fast as possible, then you'll want dist-info + static install-requirements. (IIUC this is Robert's position.) OTOH if you think that: - automagic building of sdists is a mixed blessing at best (at least in this particular example, what mostly actually happens is that people get angry and swear at their computer because they didn't want 'pip install scikit-learn' to go rebuild scipy and numpy, argh why is it my laptop suddenly swapping to death / argh I don't actually have a functional toolchain installed because I'm on windows / argh the resulting builds will seem to work at first but in fact be horribly slow because my BLAS is in funny place where it won't be autodetected and so they'll fall back on unoptimized routines / ... see also Paul's requests to just turn off sdist building entirely), and - by the time these new formats are implemented and in wide use in a few years then the increased maturity of the wheel ecosystem + Linux wheels on PyPI + Donald's planned automated build servers will mean that automagic building of sdists will only be rarely needed practice, and - these kinds of pathological requirements conflicts are going to be pretty rare and anyway when they do arise then you don't care whether it takes 60 minutes or merely 20 to sort out scikit-learn's dependencies because you're probably going to hit Control-C either way... then you might prefer the no-dist-info + no static install-requirements approach, because it simplifies the build system / packaging stuff :-). (That was the reasoning behind my original proposal.) And if you like that second argument, but have gotten push-back from people like Robert, then you might try writing up a proposal for static metadata in sdists, in the hopes that no-dist-info / yes-static-metadata might be an acceptable compromise between the above two positions :-). This discussion is important structurally is because it's where the design of a new build system interface becomes coupled to the design of a new sdist format: if you want to get a new build system interface done as soon as possible and don't want to touch the static metadata stuff, then the choice is between "no dist-info / no static metadata" versus "yes dist-info / no static-metadata", and if these are your options then it pushes you strongly towards supporting a dist-info operation. But then you risk saddling all build systems with the requirement to implement this dist-info thing, even if it later turns out that static sdist metadata makes it unnecessary. As mentioned in the other thread about promoting extras to first-class packages, there is still some hope that we might be able to get away with having full static metadata in all sdists, in which case the dist-info operation becomes completely vestigial -- but that discussion is going to be much more involved and certainly requires the addition of a real resolver for pip. So we probably don't want to block all progress on better build system while waiting for that. One possible compromise to streamline things might be: - give up on specifying a new sdist format for now (sorry Donald!) -- it's a good idea, we have some idea how to do it, but treat it as a separate project - include a dist-info operation in the new standard build system interface... - ...but make it optional. The cost of making it optional is that pip would need to be prepared with a fallback if the dist-info operation is not available -- so the get-install-requirements step would look like: def get_install_requirements(source_directory): if supports_dist_info(source_directory): dist_info_directory = run_dist_info_hook(source_directory) return dist_info_to_install_requirements(dist_info_directory) else: wheel = run_wheel_build_hook(source_directory) return wheel_to_install_requirements(wheel) Compared to a design where dist-info is mandatory, this is some extra complexity added to pip, but not a huge amount -- the above is pseudo-code, but aside from supports_dist_info all the functions it calls are ones that need to exist anyway, and the extra logic is neatly encapsulated within get_install_requirements. The minor advantage of doing this is that if some project/build system doesn't need this logic they don't need to implement it (e.g. for flit, which only supports pure python packages, there's absolutely no use -- it can build a wheel as fast as it can generate just the dist-info, and it doesn't even matter because projects using flit will ~always have wheels available and the resolver will ~never need to look at an sdist). The major advantage is that if later on it turns out that dist-info is useless (e.g. because sdists start shipping with full static metadata), then making it optional now means that we'll have the option of just dropping the support from build systems and from pip without breaking backcompat. (Compare to the situation where we make it mandatory now: since there will be versions of pip in the wild that blow up if the operation is not provided, every build system may be stuck providing some sort of dist-info support for a decade or whatever, even if it's not needed.) Alternatively we could just leave dist-info out for now, and implement it later as an optional optimization that a build system can provide if it turns out to matter :-). This would definitely help move things forward more quickly, because I wouldn't be surprised if the most complicated part of implementing Robert's current spec turned out to be the code needed to validate that dist-info and build-wheel actually return matching metadata :-). You absolutely need to do this by default, because otherwise we'll be back where we are now with people intentionally playing horrible fragile tricks, and weird accidental inconsistencies going undetected and creating subtle bugs. But to implement this checking you need some complex flow control to make sure that the eventual build-wheel step has access to the original metadata, plus the code for checking metadata-equality might be non-trivial... -n -- Nathaniel J. Smith -- http://vorpus.org

2 1

wacky idea about reifying extras
by Nathaniel Smith 30 Oct '15

30 Oct '15

Hi all, I had a wacky idea and I can't tell if it's brilliant, or ridiculous, or both. Which makes sense given that I had a temperature of 39.3 when I thought of it, but even after getting better and turning it over in my mind for a while I still can't tell, so, I figured I'd put it out there and see what y'all thought :-). The initial motivation was to try and clean up an apparent infelicity in how extras work. After poking at it for a bit, I realized that it might accidentally solve a bunch of other problems, including removing the need for dynamic metadata in sdists (!). If it can be made to work at all... Motivating problem ------------------ "Extras" are pretty handy: e.g., if you just want the basic ipython REPL, you can do ``pip install ipython``, but if you want the fancy HTML notebook interface that's implemented in the same source code base but has lots of extra heavy dependencies, then you can do ``pip install ipython[notebook]`` instead. Currently, extras are implemented via dedicated metadata fields inside each package -- when you ``pip install ipython[notebook]``, it downloads ipython-XX.whl, and when it reads the metadata about this package's dependencies and entry points, the ``notebook`` tag flips on some extra dependencies and extra entry points. This worked pretty well for ``easy_install``'s purposes, since ``easy_install`` mostly cared about installing things; there was no concern for upgrading. But people do want to upgrade, and as ``pip`` gets better at this, extras are going to start causing problems. Hopefully soon, pip will get a proper resolver, and then an ``upgrade-all`` command like other package managers have. Once this happens, it's entirely possible -- indeed, a certainty in the long run -- that if you do:: $ pip install ipython[notebook] # wait a week $ pip upgrade-all you will no longer have the notebook installed, because the new version of ipython added a new dependency to the "notebook" extra, and since there's no record that you ever installed that extra, this new dependency won't be installed when you upgrade. I'm not sure what happens to any entry points or scripts that were part of ipython[notebook] but not ipython -- I'm guessing they'd still be present, but broken? If you want to upgrade while keeping the notebook around, then ``upgrade-all`` is useless to you; you have to manually keep a list of all packages-with-extras you have installed and explicitly pass them to the upgrade command every time. Which is terrible UX. Supporting extras in this manner also ramifies complexity through the system: e.g., the specification of entry points becomes more complex because you need a way to make them conditional on particular extras, PEP 426 proposes special mechanisms to allow package A to declare a dependency on extra B of package C, etc. And extras also have minor but annoying limitations, e.g. there's no mechanism provided to store a proper description of what an extra provides and why you might want it. Solution (?): promoting extras to first-class packages ------------------------------------------------------ There's an obvious possible solution, inspired by how other systems (e.g. Debian) handle this situation: promote ``ipython[notebook]`` to a full-fledged package, that happens to contain no files of its own, but which gets its own dependencies and other metadata. What would this package look like? Something like:: Name: ipython[notebook] Version: 4.0.0 Requires-Dist: ipython (= 4.0.0) Requires-Dist: extra_dependency_1 Requires-Dist: extra_dependency_2 Requires-Dist: ... The ``notebook`` extra extends IPython with an HTML interface to... Installing it needs to automatically trigger the installation of ipython, so it should depend on ipython. It needs to be upgraded in sync with ``ipython``, so this dependency should be an exact version dependency -- that way, upgrading ``ipython`` will (once we have a real resolver!) force an upgrade of ``ipython[notebook]`` and vice-versa. Then of course we also need to include the extra's unique dependencies, and whatever else we want (e.g. a description). What would need to happen to get there from here? AFAICT a relatively small set of changes would actually suffice: **PyPI:** starts allowing the upload of wheels named like ``BASE[EXTRA]-VERSION-COMPAT.whl``. They get special handling, though: who-ever owns ``BASE`` gets to do whatever they like with names like ``BASE[EXTRA]``, that's an established rule, so wheels following this naming scheme would be treated like other artifacts associated with the (BASE, VERSION) release. In particular, the uploader would need to have write permission to the ``BASE`` name, and it would remain impossible to register top-level distribution names containing square brackets. **setuptools:** Continues to provide "extra" metadata inside the METADATA file just as it does now (for backwards compatibility with old versions of pip that encounter new packages). In addition, though, the egg-info command would starts generating .egg-info directories for each defined extra (according to the schema described above), the bdist_wheel command would start generating a wheel file for each defined extra, etc. **pip:** Uses a new and different mechanism for looking up packages with extras: - when asked to fulfill a requirement for ``BASE[EXTRA1,EXTRA2,...] (> X)``, it should expand this to ``BASE[EXTRA1] (> X), BASE[EXTRA2] (> X), ...``, and then attempt to find wheels with those actual names - backcompat case: if we fail to find a BASE[EXTRA] wheel, then fall back to fetching a wheel named BASE and attempt to use the "extra" metadata inside it to generate BASE[EXTRA], and install this (this is morally similar to the fallback logic where if it can't find foo.whl it tries to generate it from the foo.zip sdist) - Optionally, PyPI itself could auto-generate these wheels for legacy versions (since they can be generated automatically from static wheel metadata), thus guaranteeing that this path would never be needed, and then pip could disable this fallback path... but I guess it would still need it to handle non-PyPI indexes. - if this fails, then it falls back to fetching an sdist named BASE (*not* BASE[EXTRA]) and attempting to build it (while making sure to inject a version of setuptools that's recent enough to include the above changes). **PEP 426:** can delete all the special case stuff for extras, because they are no longer special cases, and there's no backcompat needed for a format that is not yet in use. **twine and other workflows:** ``twine upload dist/*`` continues to do the right thing (now including the new extra wheels). Other workflows might need the obvious tweaking to include the new wheel files. So this seems surprisingly achievable (except for the obvious glaring problem that I missed but someone is about to point out?), would improve correctness in the face of upgrades, and simplifies our conceptual models, and provides a more solid basis for future improvements (e.g. if in the future we add better tracking of which packages were manually installed, then this will automatically apply to extras as well, since they are just packages). But wait there's more --------------------- **Non-trivial plugins:** Once we've done this, suddenly extras become much more powerful. Right now it's a hard constraint that extras can only add new dependencies and entry points, not contain any code. But this is rather artificial -- from the user's point of view, 'foo[bar]' just means 'I want a version of foo that has the bar feature', they don't care whether this requires installing some extra foo-to-bar shim code. With extras as first-class packages, it becomes possible to use this naming scheme for things like plugins or compiled extensions that add actual code. **Build variants:** People keep asking for us to provide numpy builds against Intel's super-fancy closed-source (but freely redistributable) math library, MKL. It will never be the case that 'pip install numpy' will automatically give you the MKL-ified version, because see above re: "closed source". But we could provide a numpy[mkl]-{VERSION}-{COMPAT}.whl with metadata like:: Name: numpy[mkl] Conflicts: numpy Provides: numpy which acts as a drop-in replacement for the regular numpy for those who explicitly request it via ``pip install numpy[mkl]``. This involves two new concepts on top of the ones above: Conflicts: is missing from the current metadata standards but (I think?) trivial to implement in any real resolver. It means "I can't be installed at the same time as something else which matches this requirement". In a sense, it's actually an even more primitive concept than a versioned requirement -- Requires: foo (> 2.0) is equivalent to Requires: foo + Conflicts: foo (<= 2.0), but there's no way to expand an arbitrary Conflicts in terms of Requires. (A minor but important wrinkle: the word "else" is important there; you need a special case saying that a package never conflicts with itself. But I think that's the only tricky bit.) Provides: is trickier -- there's some vestigial support in the current standards and even in pip, but AFAICT it hasn't really been worked out properly. The semantics are obvious enough (Provides: numpy means that this package counts as being numpy; there's some subtleties around what version of numpy it should count as but I think that can be worked out), but it opens a can of worms, because you don't want to allow things like:: Name: numpy Provides: django But once you have the concept of a namespace for multiple distributions from the same project, then you can limit Provides: so that it's only legal if the provider distribution and the provided distribution have the same BASE. This solves the social problem (PyPI knows that numpy[mkl] and numpy are 'owned' by the same people, so this Provides: is OK), and provides algorithmic benefits (if you're trying to find some package that provides foo[extra] out of a flat directory of random distributions, then you only have to examine wheels and sdists that have BASE=foo). The other advantage to having the package be ``numpy[mkl]`` instead of ``numpy-mkl`` is that it correctly encodes that the sdist is ``numpy.zip``, not ``numpy-mkl.zip`` -- the rules we derived to match how extras work now are actually exactly what we want here too. **ABI tracking:** This also solves another use case entirely: the numpy ABI tracking problem (which is probably the single #1 problem the numerical crowd has with current packaging, because it actually prevents us making basic improvements to our code -- the reason I've been making a fuss about other things first is that until now I couldn't figure out any tractable way to solve this problem, but now I have hope). Once you have Provides: and a namespace to use with it, then you can immediately start using "pure virtual" packages to keep track of which ABIs are provided by a single distribution, and determine that these two packages are consistent: Name: numpy Version: 1.9.2 Provides: numpy[abi-2] Provides: numpy[abi-3] Name: scipy Depends: numpy Depends: numpy[abi-2] (AFAICT this would actually make pip *better* than conda as far as numpy's needs are concerned.) The build variants and virtual packages bits also work neatly together. If SciPy wants to provide builds against multiple versions of numpy during the transition period between two ABIs, then these are build variants exactly like numpy[mkl]. For their 0.17.0 release they can upload:: scipy-0.17.0.zip scipy[numpy-abi-2]-0.17.0.whl scipy[numpy-abi-3]-0.17.0.whl (And again, it would be ridiculous to have to register scipy-numpy-abi-2, scipy-numpy-abi-3, etc. on PyPI, and upload separate sdists for each of them. Note that there's nothing magical about the names -- those are just arbitrary tags chosen by the project; what pip would care about is that one of the wheels' metadata says Requires-Dist: numpy[abi-2] and the other says Requires-Dist: numpy[abi-3].) So far as I can tell, these build variant cases therefore cover *all of the situations that were discussed in the previous thread* as reasons why we can't necessarily provide static metadata for an sdist. The numpy sdist can't statically declare a single set of install dependencies for the resulting wheel... but it could give you a menu, and say that it knows how to build numpy.whl, numpy[mkl].whl, or numpy[external-blas].whl, and tell you what the dependencies will be in each case. (And maybe it's also possible to make numpy[custom] by manually editing some configuration file or whatever, but pip would never be called upon to do this so it doesn't need the static metadata.) So I think this would be sufficient to let us start providing full static metadata inside sdists? (Concretely, I imagine that the way this would work is that when we define the new sdist hooks, one of the arguments that pip would pass in when running the build system would be a list of the extras that it's hoping to see, e.g. "the user asked for numpy[mkl], please configure yourself accordingly". For legacy setuptools builds that just use traditional extras, this could safely be ignored.) TL;DR ----- If we: - implement a real resolver, and - add a notion of a per-project namespace of distribution names, that are collected under the same PyPI registration and come from the same sdist, and - add Conflicts:, and Provides:, then we can elegantly solve a collection of important and difficult problems, and we can retroactively pun the old extras system onto the new system in a way that preserves 100% compatibility with all existing packages. I think? What do you think? -n -- Nathaniel J. Smith -- http://vorpus.org

8 11

Second draft of a plan for a new source tree / sdist format
by Nathaniel Smith 29 Oct '15

29 Oct '15

Hi all, Here's a second round of text towards making a build-system independent interface between pip and source trees/sdists. My idea this time is to take a divide-and-conquer approach: this text tries to summarize all the stuff that it seemed like we had mostly reached consensus on in the previous thread + call, with blank chunks marked "TBD" where there are specific points that still need To Be Determined. So my hope is that everyone will read what's here and agree that it's great as far as it goes, and then we can go through and fill in each missing piece one at a time. ------ PEP: ?? Title: A build-system independent format for source trees and source distributions Version: $Revision$ Last-Modified: $Date$ Author: Nathaniel J. Smith <njs(a)pobox.com> Status: Draft Type: Standards-Track Content-Type: text/x-rst Created: 30-Sep-2015 Post-History: 1 Oct 2015, 25 Oct 2015 Discussions-To: <distutils-sig(a)python.org> Abstract ======== Distutils delenda est. Extended abstract ================= While ``distutils`` / ``setuptools`` have taken us a long way, they suffer from three serious problems: (a) they're missing important features like autoconfiguration and usable build-time dependency declaration, (b) extending them is quirky, complicated, and fragile, (c) it's very difficult to use anything else, because they provide the standard interface for installing python packages expected by both users and installation tools like ``pip``. Previous efforts (e.g. distutils2 or setuptools itself) have attempted to solve problems (a) and/or (b). We propose to solve (c). The goal of this PEP is get distutils-sig out of the business of being a gatekeeper for Python build systems. If you want to use distutils, great; if you want to use something else, then that should be easy to do using standardized methods. The difficulty of interfacing with distutils means that there aren't many such systems right now, but to give a sense of what we're thinking about see `flit <https://github.com/takluyver/flit>`_ or `bento <https://cournape.github.io/Bento/>`_. Fortunately, wheels have now solved many of the hard problems here -- e.g. it's no longer necessary that a build system also know about every possible installation configuration -- so pretty much all we really need from a build system is that it have some way to spit out standard-compliant wheels. We therefore propose a new, relatively minimal interface for installation tools like ``pip`` to interact with package source trees and source distributions. In addition, we propose a wheel-inspired static metadata format for sdists, suitable for tools like PyPI and pip's resolver. Terminology and goals ===================== A *source tree* is something like a VCS checkout. We need a standard interface for installing from this format, to support usages like ``pip install some-directory/``. A *source distribution* is a static snapshot representing a particular release of some source code, like ``lxml-3.4.4.zip``. Source distributions serve many purposes: they form an archival record of releases, they provide a stupid-simple de facto standard for tools that want to ingest and process large corpora of code, possibly written in many languages (e.g. code search), they act as the input to downstream packaging systems like Debian/Fedora/Conda/..., and so forth. In the Python ecosystem they additionally have a particularly important role to play, because packaging tools like ``pip`` are able to use source distributions to fulfill binary dependencies, e.g. if there is a distribution ``foo.whl`` which declares a dependency on ``bar``, then we need to support the case where ``pip install bar`` or ``pip install foo`` automatically locates the sdist for ``bar``, downloads it, builds it, and installs the resulting package. Source distributions are also known as "sdists" for short. Source trees ============ We retroactively declare the legacy source tree format involving ``setup.py`` to be "version 0". We don't try to specify it further; its de facto specification is encoded in the source code and documentation of ``distutils``, ``setuptools``, ``pip``, and other tools. A "version 1" (or greater) source tree is any directory which contains a file named ``pypackage.cfg``, which will -- in some manner whose details are TBD -- describe the package's build dependencies and how to invoke the build system. This mechanism: - Will allow for both static and dynamic specification of build dependencies - Will have some degree of isolation of different builds from each other, so that it will be possible for a single run of pip to install one package that build-depends on ``foo = 1.1`` and another package that build-depends on ``foo = 1.2``. - Will leave the actual installation of the package in the hands of the build/installation tool (i.e. individual package build systems will not need to know about things like --user versus --global or make decisions about when and how to modify .pth files) [TBD: the exact set of operations to be supported and their detailed semantics] [TBD: should builds be performed in a fully isolated environment, or should they get access to packages that are already installed in the target install environment? The former simplifies a number of things, but Robert was skeptical it would be possible.] [TBD: the form of the communication channel between an installation tool like ``pip`` and the build system, over which these operations are requested] [TBD: the syntactic details of the configuration file format itself. We can change the name too if we want, I just think it's useful to have a single name to refer to it for now, and this is the last and least interesting thing to figure out.] Source distributions ==================== [possibly this should get split off into a separate PEP, but I'll keep it together for now for ease of discussion] A "version 1" (or greater) source distribution is a file meeting the following criteria: - It MUST have a name of the form: {PACKAGE}-{VERSION}.{EXT}, where {PACKAGE} is the package name, {VERSION} is a PEP 440-compliant version number, and {EXT} is a compliant archive format. The set of compliant archive formats is: zip, [TBD] [QUESTION: should we continue to allow .tar.gz and friends? In practice by "allow" I mean something like "accept new-style sdists on PyPI in this format". I'm inclined not to -- zip is the most universally supported format around, it allows file-based random access (unlike tar-based things) which is useful for pulling out metadata without decompressing the whole thing, and standardizing on one format dodges distracting and pointless discussions about which format to use, i.e. it's TOOWTDI-compliant. Of course pip is free to continue to support other archive formats when passed explicitly on the command line. Any objections?] Similar to wheels, the archive is Unicode, and the filenames inside the archive are encoded in UTF-8. - When unpacked, it MUST contain a single directory directory tree named ``{PACKAGE}-{VERSION}``. - This directory tree MUST be a valid version 1 (or greater) source tree as defined above. - It MUST additionally contain a directory named ``{PACKAGE}-{VERSION}.sdist-info`` (notice the ``s``), with the following contents: - ``SDIST``: Mandatory. Same record-oriented format as a wheel's ``WHEEL`` file, but with different fields:: SDist-Version: 1.0 Generator: setuptools sdist 20.1 ``SDist-Version`` is the version number of this specification. Software that processes sdists should warn if ``SDist-Version`` is greater than the version it supports, and must fail if ``SDist-Version`` has a greater major version than the version it supports. ``Generator`` is the name and optionally the version of the software that produced the archive. - ``RECORD``: Mandatory. A list of all files contained in the sdist (except for the RECORD file itself and any signature files) together with their hashes, as specified in PEP 427. - ``RECORD.jws``, ``RECORD.p7s``: Optional. Signature files as specified in PEP 427. - ``METADATA``: Mandatory. Metadata version 1.1 or greater format metadata, with an additional rule that fields may contain the special sentinel value ``__SDIST_DYNAMIC__``, which indicates that the value of this field cannot be determined until build time. If a "multiple use field" is present with the value ``__SDIST_DYNAMIC__``, then this field MUST occur exactly once, e.g.:: # Okay: Requires-Dist: lxml (> 3.3) Requires-Dist: requests # no Requires-Dist lines at all is okay # (meaning: this package's requirements are the empty set) # Okay, requirements will be determined at build time: Requires-Dist: __SDIST_DYNAMIC__ # NOT okay: Requires-Dist: lxml (> 3.3) Requires-Dist: __SDIST_DYNAMIC__ (The use of a special token allows us to distinguish between multiple use fields whose value is statically the empty list versus one whose value is dynamic; it also allows us to distinguish between optional fields which are statically not present versus ones whose value is dynamic.) When this sdist is built, the resulting wheel MUST have metadata which is identical to the metadata present in this file, except that any fields with value ``__SDIST_DYNAMIC__`` in the sdist may have arbitrary values in the wheel. A valid sdist MUST NOT use the ``__SDIST_DYNAMIC__`` mechanism for the package name or version (i.e., these must be given statically), and these MUST match the {PACKAGE} and {VERSION} of the sdist as described above. [TBD: do we want to forbid the use of dynamic metadata for any other fields? I assume PyPI will enforce some stricter rules at least, but I don't know if we want to make that part of the spec, or just part of PyPI's administrative rules.] This is intentionally a close analogue of a wheel's ``.dist-info`` directory; intention is that as future metadata standards are defined, the specifications for the ``.sdist-info`` and ``.dist-info`` directories will evolve in synchrony. Evolutionary notes ================== A goal here is to make it as simple as possible to convert old-style sdists to new-style sdists. (E.g., this is one motivation for supporting dynamic build requirements.) The ideal would be that there would be a single static pypackage.cfg that could be dropped into any "version 0" VCS checkout to convert it to the new shiny. This is probably not 100% possible, but we can get close, and it's important to keep track of how close we are... hence this section. A rough plan would be: Create a build system package (``setuptools_pypackage`` or whatever) that knows how to speak whatever hook language we come up with, and convert them into setuptools calls. This will probably require some sort of hooking or monkeypatching to setuptools to provide a way to extract the ``setup_requires=`` argument when needed, and to provide a new version of the sdist command that generates the new-style format. This all seems doable and sufficient for a large proportion of packages (though obviously we'll want to prototype such a system before we finalize anything here). (Alternatively, these changes could be made to setuptools itself rather than going into a separate package.) But there remain two obstacles that mean we probably won't be able to automatically upgrade packages to the new format: 1) There currently exist packages which insist on particular packages being available in their environment before setup.py is executed. This means that if we decide to execute build scripts in an isolated virtualenv-like environment, then projects will need to check whether they do this, and if so then when upgrading to the new system they will have to start explicitly declaring these dependencies (either via ``setup_requires=`` or via static declaration in ``pypackage.cfg``). 2) There currently exist packages which do not declare consistent metadata (e.g. ``egg_info`` and ``bdist_wheel`` might get different ``install_requires=``). When upgrading to the new system, projects will have to evaluate whether this applies to them, and if so they will need to either stop doing that, or else add ``__SDIST_DYNAMIC__`` annotations at appropriate places. We'll also presumably need some API for packages to describe which parts of the METADATA file should be marked ``__SDIST_DYNAMIC__``, for the packages that need it (a new argument to ``setup()`` or some setting in ``setup.cfg`` or something). -- Nathaniel J. Smith -- http://vorpus.org

7 11

flit adapter for the current sdist format
by Daniel Holth 29 Oct '15

29 Oct '15

OK, so I wanted to see what it would take to create current-format sdists, defined as anything that has a setup.py that pip can install by running said setup.py. The following script takes a flit package (in the current directory and builds a pip-installable sdist. Tested against https://github.com/pohmelie/aioftp https://gist.github.com/dholth/e17f243085dcc539813f The first script is the archiver and the second script is the setup.py embedded into the sdist by the archiver. The embedded setup.py does an isolated install of flit into the setup-requires directory, if required, and uses flit to implement bdist_wheel. The trick is that the only thing pip really needs from egg_info is the current requirements, easily generated from wheel-style .dist-info. It might need slightly more metadata to pass on pypi and the argument handling is inflexible, but it demonstrates most of what would be required for a backwards compatible shim. If our new sdist format is an even easier way to adapt new build systems than writing this trivial setup.py shim, then we might have something.

2 1

Environment markers: ready for prime time?
by Marius Gedminas 28 Oct '15

28 Oct '15

Hi! pip 7 enables wheel caching by default, which is good. Wheel caching means you can't compute dynamic dependencies any more -- i.e. things like setup( ... install_requires=[...] + (['enum34'] if sys.version_info[0] < 3 else []), ...) will cause problems. As far as I understand, you're supposed to use environment markers instead. Problem 1: where's the documentation? E.g. https://python-packaging-user-guide.readthedocs.org/en/latest/distributing/ has no mention of the word "marker". Try to google "setuptools environment marker" (and how is a user going to discover the magical keyword they need to google is "environment marker")? Try to find anything resembling documentation on the 1st page. The best I could find was http://docs.openstack.org/developer/pbr/#environment-markers which only works if you use pbr. Even the spec (https://www.python.org/dev/peps/pep-0426/#environment-markers) only shows how the markers are supposed to appear in the JSON metadata. No clue is provided for poor writers of setup.py files. I somehow discovered the syntax once (I don't remember how -- most likely kind people in #pypa spoon-fed me), but now I'm cargo-culting my existing setup.py files that already use environment markers. Problem 2: what are the minimum versions of the tools that your users must have before you can rely on environment markers? - setuptools >= 0.7 ("Added experimental environment marker support") - wheel >= 0.24 (if you have wheel 0.23 or older, environment markers are silently broken and have fun figuring out why: https://github.com/pypa/pip/issues/2870). - does the pip version matter at all? I think not; please correct me if I'm wrong. Some official answers from the hard-working PyPA visionaries would be welcome. Marius Gedminas -- I once held a little hand That made my sad heart sing. Twas the loveliest hand I'd ever held, Four Aces and a King

4 3

New PyPUG Tutorials
by Marcus Smith 28 Oct '15

28 Oct '15

As some of you may already know, Nicole from the Warehouse team has initiated an effort to improve our packaging tutorials, and have them linked in the new Warehouse UI (https://github.com/pypa/warehouse/issues/729 ). Although originally the idea was to create additional tutorials (but still maintain consistency with the "Python Packaging User Guide", aka "PyPUG"), after further discussions, we've decided to just improve the 2 existing PyPUG sections that we have (for installation and distribution), with the goal being to improve the accessibilty for new users, and in general to improve the quality and overall style and experience. Nicole has already gathered a group of volunteer writers, and there's a plan in place to manage the process from planning to release. To have the most success, the writers will certainly need feedback from subject matter experts, so the process will include 2 stages where we specifically ask for feedback from PyPA-Dev and Distutils-Sig: 1) To validate the initial proposal that covers the scope of the changes, and 2) to review the actual PRs to PyPUG for accuracy, when it's time for merging. I'll post again with more details as those stages occur. As someone who has slowly chipped away on the PyPUG for the last 2 years, I'm excited about the prospect of being able to make a big jump in quality very quickly. --Marcus P.S. for anyone here who wants to be on the core writing team, see https://github.com/pypa/warehouse/issues/729

3 2

Re: [Distutils] Remove distutils, was: red, green, refactor ...
by Thomas Güttler 27 Oct '15

27 Oct '15

Am 21.10.2015 um 13:28 schrieb Ionel Cristian Mărieș: > > On Wed, Oct 21, 2015 at 1:42 PM, Thomas Güttler <guettliml(a)thomas-guettler.de <mailto:guettliml@thomas-guettler.de>> wrote: > > > Why not deprecate and remove distutils? > > > Should we or can we? There was an attempt at that, called distutils2. I'd love to hear what people have learnt from that. > > And if now-days walking around with pants instead of only underwear is the way to go, should we still have underwear > under the pants? I think yes :-) In other words, setuptools uses distutils. ok, at the moment setuptools uses distutils. Why not melt them together into **one** underwear-pants-module? Regards, Thomas Güttler -- Thomas Guettler http://www.thomas-guettler.de/

15 37