Mailman 3 July 2017 - Distutils-SIG

Manual upload of RPM files no longer works
by Anthony Tuininga 26 Jul '17

26 Jul '17

All, I just tried to upload some RPM files using the "Files" section in PyPI. I get the following response: Gone (This API has been deprecated and removed from legacy PyPI in favor of using the APIs available in the new PyPI.org implementation of PyPI (located at https://pypi.org/). For more information about migrating your use of this API to PyPI.org, please see https://packaging.python.org/guides/migrating-to-pypi-org/#uploading. For more information about the sunsetting of this API, please see https://mail.python.org/pipermail/distutils-sig/2017-June/030766.html) I have used twine for wheels and source distributions without any difficulty. But it refuses to support RPM files: ValueError: Unknown distribution format: 'cx_Oracle-6.0rc2-py26-1.x86_64.rpm' Is there any way to get these uploaded now? I can't use bdist_rpm upload and I'd like to verify the files *before* they are uploaded, not afterwards! Thoughts? Anthony

3 2

Installing modules
by HARISHYAM KUMAR 26 Jul '17

26 Jul '17

No any program run in python 2.7 Please suggest me how to down load modules to run required programs Sent from Mail for Windows 10

1 0

Maintenance of the "wheel" package
by Alex Grönholm 25 Jul '17

25 Jul '17

Looking at the commit history, the maintenance of "wheel" has more or less stalled since 2016. I would like to volunteer myself as a co-maintainer of this project, if allowed. Here's my list of things I would like to do with it: * Move the project from Bitbucket to the PyPA organization on Github * Add a setup() keyword (and documentation for it) for license_file (currently only usable via the deprecated [metadata] section in setup.cfg) * Refactor the test suite to make better use of py.test * Clean up code style for PEP 8 compliance

2 2

Getting dependecies of package from PyPiJSON
by Krzysiek Płachno 21 Jul '17

21 Jul '17

Hi! I'm contributing to XWiki open source project http://www.xwiki.org/. XWiki platform is written in Java and apart from many wonderful features allows for scripting in python on its pages. My current task is to enable installing packages form PyPi repository so that they can be later used in scripts. The installment is done in custom way making python packages available for Java classloader. To make downloaded package working it's needed to install also dependencies. Is it possible to get dependencies information directly from PyPiJSON API? (e.g. by adding some request parameter or header in GET request) Or is it possible to this data in any other way (apart from downloading package)? Thanks for help in advance! Best, Krzysztof

7 12

requirements for variable python
by Robin Becker 19 Jul '17

19 Jul '17

Is there a way to build a universal wheel which needs an additional module when python 2.7 is being used. Currently the run requirements for the module are python-dateutil jsonschema but in addition jsonschema needs functools32 for python 2.7. Is there a way to get the extra requirement used when python 2.7 is the installer? Or does this package not qualify for universalness. -- Robin Becker

2 2

Pip not working
by Michael Afolabi 19 Jul '17

19 Jul '17

Hello Python, I have been trying to install pip onto my windows laptop but even though I have python 3.6. It still will not run. Every time I type pip it returns by saying that it is not recognized as an internal or external command. Also when i type in python -m pip install, it returns with python.exe: No module named pip. I was wondering if you can provide me a solution to this problem Thanking you, Mike

2 2

Re: [Distutils] Notes on pip's requirements for PEP 517
by Nathaniel Smith 18 Jul '17

18 Jul '17

On Jul 17, 2017 8:48 AM, "Paul Moore" <p.f.moore(a)gmail.com> wrote: On 17 July 2017 at 14:45, Nathaniel Smith <njs(a)pobox.com> wrote: > On Mon, Jul 17, 2017 at 6:42 AM, Daniel Holth <dholth(a)gmail.com> wrote: >> A minor detail for pip strategy option #2 is that sdists do not have to have >> PKG-INFO. > > ...should they be required to have PKG-INFO? This seems like a > relevant question regardless of whether pip goes with "strategy option > #2" :-). A note of caution here - this way lies (at least some level) of defining an official sdist format. I'm not against that (specifically, I'd be happy if we did that, just formalising what setuptools currently does in some usable form, and leave any sort of "sdist 2.0" debate for when we've all got our breath back after this discussion) but I'll note that we went round this loop very early on here (which resulted in the PEP being written in terms of "source trees") so we need to be careful not to go back over old ground. There's already a short paragraph that gives some minimal requirements on sdists; we needed it for build_sdist. Now that I look again though, it turns out that it already mandates the presence of PKG-INFO. -n

2 1

Notes on pip's requirements for PEP 517
by Nathaniel Smith 18 Jul '17

18 Jul '17

Hi all, I happened to talk to Donald on IRC last night, and said I'd write up some notes on that discussion. (He's also seen these notes, but is partially-offline dealing with jury duty.) [Edit: but apparently still replying to email on the list, so some of this repeats things he's posted since I sent it to him to look at, but oh well :-). That's life in the fast-paced world of F/OSS development I guess.] One thing we tried to get a handle on is what, exactly, are constraints that PEP 517 is trying to deal with. As far as pip goes, these are some of the situations that Donald's worried about handling gracefully when it comes to 'pip install .': a) The source tree is on read-only media b) People do an editable install in a source tree, and then follow it up with a regular install ('pip install -e . && pip install .'). Apparently this can cause problems because of the residual metadata from the editable install confusing setuptools. c) Someone does 'pip install .' on a complex package, then discovers that while the build succeeded, it's missing some optional feature because some necessary system library was missing, so they install that library and re-try the 'pip install .'. Will the build backend notice that the library has appeared, or does it only do environment sniffing the first time it's run in a given tree? d) Mounting the same source tree into multiple different docker containers: 'docker run ubuntu -v .:/io pip install /io && docker run alpine -v .:/io install /io'. Will the build backend notice that these two systems have incompatible C toolchains and C ABIs, so you can't share .o files between them? In principle a build system could notice this by detecting that the compiler and system headers have changed, but in traditional build systems it's common to intentionally *not* include these in your build cache key, because you don't want to rebuild the world after every apt upgrade. (For example, gcc provides the -M switch to generate make rules to trigger rebuilds on header changes, and the -MM switch to do the same but ignoring changes in system headers; lots of projects choose to use -MM. This is related to the thing where incremental builds are traditionally a bit sloppy and intended for experts only.) e) Build systems that have different sources of truth for what goes into an sdist versus what goes into a wheel, and thus can easily end up in a situation where a direct VCS->wheel build gives a different result than a VCS->sdist->wheel build. The most prominent offender here is distutils/setuptools with its MANIFEST.in mess, but Donald is nervous that other systems might also reproduce this mistake. f) And finally Donald feels "it's just more hygenic to have ``pip install .`` not modify ., similiarly to how [he] would be upset if ``pip install foo-1.0.tar.gz`` modified foo-1.0.tar.gz in some way". Of course, no system can avoid every problem; the overall goal here is harm reduction and minimizing spurious bugs filed on pip, not perfection. None of these cases arise for 'pip install name' or 'pip install sdist.tar.gz'; it's really only 'pip install .' on an user-provided source tree. ---------- For reference, here's my analysis of how these particular desiderata relate to some possible approaches: - Pip's current system for handling builds from existing source trees (copytree + setup.py bdist_wheel -or- setup.py install) handles (a), (c), (d), (f), but not (b) or (e), unless someone has previously done an in-place build in the tree, in which case it handles (a) and (f) but not (b), (c), (d), or (e). Which is kind of unfortunate, since (a) and (f) are probably the least important items. - When sdist->unpack->wheel is possible, it automatically handles all of these cases. - If a build backend *only* does in-place builds (like meson) and *does not support* in-place or editable installs, then having an out-of-place build_wheel hook automatically takes care of everything except (e). - Otherwise, an out-of-place build_wheel hook acts handles (a), (c), (d), (f) but not (b) or (e), unless someone has previously done an in-place build in the tree, in which case it handles (a) and (f) but not (b), (c), (d), or (e) (assuming that the build system can get confused between artifacts left behind by in-place builds when doing an out-of-place build, which appears to be a common feature of existing systems). Notice that this means that as far as this score card is concerned, out-of-place build_wheel is identical to pip's current copytree + in-place build_wheel. The key insight here is that an out-of-tree build is nicer for the *next* person to use this source tree, but when pip runs what it cares about is whether the *last* build was in-tree or out-of-tree, and that's not something that it has any control over. - If you have a system that supports in-place builds but not build_sdist, then copytree + a "clean build" hook would cover everything except (e). Notice that, for the same reason as above, it doesn't matter whether the clean build is in-tree or out-of-tree, just that it has the ability to avoid any interference from junk left inside the tree by any previous build. ---------- Anyway. Back to things we discussed. As far as we understand, the way that flit comes into this is that flit cannot generate an sdist: - from a VCS directory when the VCS tools are unavailable - from an unpacked sdist (Thomas, is that right?) So on that basis, we considered a few different strategies that pip might take for handling 'pip install .': Pip Strategy Option 1: First make sure that the setuptools backend build_wheel hook takes responsibility for doing the sdist->unpack->wheel dance when running from a VCS checkout. And then, now that we don't have to worry about setuptools messing things up, make pip's strategy be to just do a plain in-place or out-of-place build_wheel. As you can probably guess from the list above, Donald didn't feel comfortable with this, at least to start with -- he thinks that it might be possible for pip to transition to this in the future if it turns out that new backends are generally of high quality, but until we have more experience, he's nervous. This makes sense to me too -- I think we can plausibly mandate that build backends take care of (e) (MANIFEST.in problems -- that's basically the idea of making setuptools's build backend responsible for doing the sdist->unpac->wheel dance), and (a) (read-only source tree) is a rare edge case that can reasonably be handled by a special case in the frontend, but all the other items actually are regular occurences in modern build systems. Really he wants pip to go via sdist 99% of the time, because that's the case that Just Works, and then have whatever fallback is necessary for the 1%. In some sense it doesn't even matter that much what the fallback is, because if this rare edge case is a bit slow or a bit more error prone, well... it's a rare edge case. Even for flit, 99% of the time people will just be installing wheels, not downloading and manually unpacking sdists, so as long as it *works* then it doesn't have to be perfect. Anyway, this leads to the last two possible strategies we discussed: Pip Strategy Option 2: Check to see if a PKG-INFO file is present. If so, do an in-place build_wheel; otherwise, do build_sdist->unpack->build_wheel. The key thing here is that because it's pip checking for the PKG-INFO instead of querying the build backend, then every source tree has to either *be* an sdist, or be able *produce* an sdist; there's no fallback for non-sdist trees that can't produce sdists. (At least if you want to support 'pip install .') Pip Strategy Option 3: Ask the build backend whether it can do a build_sdist. If so, do build_sdist->unpack->build_wheel; otherwise, do copytree->in-place build_wheel, or out-of-place build_wheel, or whatever, it doesn't matter that much to pip. As far as Donald is concerned, either of these options would be fine. For "option 2" (having pip check for PKG-INFO), the main potential flaw we noted is that while it's OK with flit not being able to build an sdist from an sdist, it will error out in the case where you try to do 'pip install .' on a flit VCS checkout and flit can't find the VCS tools. Donald is fine with that, on the grounds that oh well, if you're working with a VCS checkout and need VCS tools to get consistent result then printing an error message telling people to install them is a fine outcome. (In particular, he observes that flit *can* produce sdist-inconsistent results in this case: if there's a .py file present in the source but not added to the VCS, and you do build_wheel, then he thinks that the .py file will be installed iff the VCS tools are unavailable, and would never be included in any sdist. Thomas, is this right?) However, he thinks that Thomas objected to raising an error in this case, and wants to allow flit to work. (Thomas, is that right?) So possibly that's an argument for preferring "option 3" (ask the backend if it can do an sdist). Another point about option 2 that we didn't discuss but that I'll mention now is that it does kinda enforce that all build backends support building sdists in general, which might be good or bad depending on what you think about that. ----------- A few more general points that Donald made: - He doesn't care much about in-place vs out-of-place in general, we can provide either or both as far as he's concerned. It doesn't matter to pip. - But, if we do support both in-place and out-of-place, then he strongly feels that there should *not* be any semantic difference between them "because it just adds another "path" and possible inconsistency", plus if a build backend is able to do something smart it should just do it all the time. So I think this is a point of disagreement between Donald and Nick. - If pip is going to go with "option 3" (i.e., attempting to do build_sdist->unpack->build_wheel, unless the build backend says it can't), then he strongly feels that this fallback should be triggered by some *explicit* signal from the backend saying that build_sdist is not supported; in general he wants build_sdist raising an error to be an immediate failure, rather than triggering the fallback path. Basically what he's saying is that he wants something like my NotImplemented proposal or equivalent. Regarding this last point: I want to point out this would also mean that when building via pip, the fallback could *only* be triggered by backends that explicitly ask for it, so they're the only ones who could possibly end up exposing their users to any weirdness in the fallback path, and it's up to them to figure out how to deal with it. In particular, it means that setuptools and flit are both handled fine regardless of what we decide about in-place vs. out-of-place, because setuptools will always go via the sdist path and thus be built from a pristine directory, and flit doesn't care about in-place vs. out-of-place anyway. Basically, as long as pip has some kind of build_sdist and build_wheel and build_sdist can signal "not implemented" in a clean way, then pip is happy. My analysis above suggests that beyond this, there's *one* possible extension that pip might want: In the future, it might turn out that there is a build backend that is like flit in that it can't always generate an sdist, but is like setuptools in that it can produce broken results when run in a unclean directory. If that happens, and if we decide that it's important for such a backend to play nicely with pip *and* to support incremental builds *and* to do both of these through the PEP 517 hook interface, then it might be useful to have something like a "make_clean_then_build_wheel" hook for pip to call. My feeling is we can probably defer this for now though? This is like 3 edge cases on top of each other and we don't even know if such a backend will ever exist. ---------- So none of the above actually has much to do with the in-place vs out-of-place debate; basically the conclusion is that pip doesn't care. But we still have to pick something. Donald and I both have some preference for only having one way to do it; given that the semantics are supposed to be identical, why bother making everyone implement both? So: should we start with in-place (and possibly add out-of-place as an option later for specific use cases like build pipelines), or should we start by mandating out-of-place? Neither of us have a particularly strong opinion on this. I guess I have mild preference for starting with in-place, because: - Out-of-place is more complicated to implement than in-place, and it's going to be difficult to explain to build system authors why we're forcing them to do this extra work for some obscure cases they might not care about and that we haven't articulated well. - It might seem like only supporting out-of-place builds would simplify pip's life, because the problems above that are caused by in-place build detritus would go away. But this isn't really true, because even if we don't *expose* in-place build functionality through the PEP 517 hooks, then most build systems are still going to implement this, and that means we still need to be prepared to handle the case where the user has done an in-place build, which is the tricky one. Plus, editable installs intrinsically leave detritus in the source tree. - For me, one of the major goals of PEP 517 is to make life easier for *really* gnarly projects -- and here I'm not thinking of like numpy or scipy, they have it easy; I'm thinking of the folks who have, like, a dozen C different libraries with build systems from the 80s vendored inside their source tree. I'm nervous that forcing these folks with complicated embedded multi-package builds to support out-of-tree builds would be a significant burden and present an obstacle to uptake. Out-of-place builds are definitely a more advanced feature and the direction good build systems move in, but I don't want to leave anyone behind. - We have zero users for this functionality right now, which is usually not a good situation for trying to write a spec. - I don't think it will be a big deal to add out-of-place support later if we want to. But really it's all that other stuff that's important to sort out first. -- Nathaniel J. Smith -- https://vorpus.org

4 4

A possible refactor/streamlining of PEP 517
by Nathaniel Smith 18 Jul '17

18 Jul '17

Hi all, I just attempted an experimental refactor/streamlining of PEP 517, to match what I think it should look like :-). I haven't submitted it as a PR to the PEPs repository yet since I don't know if others will agree with the changes, but I've pasted the full text below, or you can see the text online at: https://github.com/njsmith/peps/blob/517-refactor-streamline/pep-0517.txt and the diff at: https://github.com/python/peps/compare/master...njsmith:517-refactor-stream… Briefly, the changes are: - Rearrange text into (hopefully) better signposted sections with better organization - Clarify a number of details that have come up in discussion (e.g., be more explicit that the hooks are run with the process working directory set to the source tree, and why) - Drop prepare_wheel_metadata and prepare_wheel_build_files (for now); add detailed rationale for why we might want to add them back later. - Add an "extensions" hook namespace to allow prototyping of future extensions. - Rename get_build_*_requires -> get_requires_for_build_* to make the naming parallelism more obvious - Add the option to declare an operation unsupported by returning NotImplemented - Instead of defining a default value for get_requires_for_build_*, make it mandatory for get_require_for_build_* and build_* to appear together; this seems simpler now that we have multiple high-level operations defined in the same PEP, and also simplifies the definition of the NotImplemented semantics. - Update title to better match the scope we ended up with - Add a TODO to decide how to handle backends that don't want to have multiple hooks called from the same process, including some discussion of the options. --------------------- PEP: 517 Title: Supporting non-setup.py-based build backends in pyproject.toml Version: $Revision$ Last-Modified: $Date$ Author: Nathaniel J. Smith <njs(a)pobox.com>, Thomas Kluyver <thomas(a)kluyver.me.uk> BDFL-Delegate: Nick Coghlan <ncoghlan(a)gmail.com> Discussions-To: <distutils-sig(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30-Sep-2015 Post-History: 1 Oct 2015, 25 Oct 2015, 1 July 2017 ========== Abstract ========== While ``distutils`` / ``setuptools`` have taken us a long way, they suffer from three serious problems: (a) they're missing important features like usable build-time dependency declaration, autoconfiguration, and even basic ergonomic niceties like `DRY <https://en.wikipedia.org/wiki/Don%27t_repeat_yourself>`_-compliant version number management, and (b) extending them is difficult, so while there do exist various solutions to the above problems, they're often quirky, fragile, and expensive to maintain, and yet (c) it's very difficult to use anything else, because distutils/setuptools provide the standard interface for installing packages expected by both users and installation tools like ``pip``. Previous efforts (e.g. distutils2 or setuptools itself) have attempted to solve problems (a) and/or (b). This proposal aims to solve (c). The goal of this PEP is get distutils-sig out of the business of being a gatekeeper for Python build systems. If you want to use distutils, great; if you want to use something else, then that should be easy to do using standardized methods. The difficulty of interfacing with distutils means that there aren't many such systems right now, but to give a sense of what we're thinking about see `flit <https://github.com/takluyver/flit>`_ or `bento <https://cournape.github.io/Bento/>`_. Fortunately, wheels have now solved many of the hard problems here -- e.g. it's no longer necessary that a build system also know about every possible installation configuration -- so pretty much all we really need from a build system is that it have some way to spit out standard-compliant wheels and sdists. We therefore propose a new, relatively minimal interface for installation tools like ``pip`` to interact with package source trees and source distributions. ========================== Reversion to Draft Status ========================== While this PEP was provisionally accepted for implementation in `pip` and other tools, some additional concerns were subsequently raised around adequately supporting out of tree builds. It has been reverted to Draft status while those concerns are being resolved. ======================= Terminology and goals ======================= A *source tree* is something like a VCS checkout. We need a standard interface for installing from this format, to support usages like ``pip install some-directory/``. A *source distribution* is a static snapshot representing a particular release of some source code, like ``lxml-3.4.4.tar.gz``. Source distributions serve many purposes: they form an archival record of releases, they provide a stupid-simple de facto standard for tools that want to ingest and process large corpora of code, possibly written in many languages (e.g. code search), they act as the input to downstream packaging systems like Debian/Fedora/Conda/..., and so forth. In the Python ecosystem they additionally have a particularly important role to play, because packaging tools like ``pip`` are able to use source distributions to fulfill binary dependencies, e.g. if there is a distribution ``foo.whl`` which declares a dependency on ``bar``, then we need to support the case where ``pip install bar`` or ``pip install foo`` automatically locates the sdist for ``bar``, downloads it, builds it, and installs the resulting package. Source distributions are also known as *sdists* for short. A *build frontend* is a tool that users might run that takes arbitrary source trees or source distributions and builds wheels from them. The actual building is done by each source tree's *build backend*. In a command like ``pip wheel some-directory/``, pip is acting as a build frontend. An *integration frontend* is a tool that users might run that takes a set of package requirements (e.g. a requirements.txt file) and attempts to update a working environment to satisfy those requirements. This may require locating, building, and installing a combination of wheels and sdists. In a command like ``pip install lxml==2.4.0``, pip is acting as an integration frontend. ============== Source trees ============== There is an existing, legacy source tree format involving ``setup.py``. We don't try to specify it further; its de facto specification is encoded in the source code and documentation of ``distutils``, ``setuptools``, ``pip``, and other tools. We'll refer to it as the ``setup.py``\-style. Here we define a new style of source tree based around the ``pyproject.toml`` file defined in PEP 518, extending the ``[build-system]`` table in that file with one additional key, ``build-backend``. Here's an example of how it would look:: [build-system] # Defined by PEP 518: requires = ["flit"] # Defined by this PEP: build-backend = "flit.api:main" ``build-backend`` is a string naming a Python object that will be used to perform the build (see below for details). This is formatted following the same ``module:object`` syntax as a ``setuptools`` entry point. For instance, if the string is ``"flit.api:main"`` as in the example above, this object would be looked up by executing the equivalent of:: import flit.api backend = flit.api.main It's also legal to leave out the ``:object`` part, e.g. :: build-backend = "flit.api" which acts like:: import flit.api backend = flit.api Formally, the string should satisfy this grammar:: identifier = (letter | '_') (letter | '_' | digit)* module_path = identifier ('.' identifier)* object_path = identifier ('.' identifier)* entry_point = module_path (':' object_path)? And we import ``module_path`` and then lookup ``module_path.object_path`` (or just ``module_path`` if ``object_path`` is missing). If the ``pyproject.toml`` file is absent, or the ``build-backend`` key is missing, the source tree is not using this specification, and tools should fall back to running ``setup.py``. Where the ``build-backend`` key exists, it takes precedence over ``setup.py``, and source trees need not include ``setup.py`` at all. Projects may still wish to include a ``setup.py`` for compatibility with tools that do not use this spec. ========================= Build backend interface ========================= The build backend object is expected to have callable attributes called "hooks", which the build frontend can use to perform various actions. The two high-level actions defined by this spec are creation of an sdist (analogous to the legacy ``setup.py sdist`` command) and building of a wheel (analogous to the legacy ``setup.py bdist_wheel`` command). We additionally define a namespace for tool-specific hooks, which may be useful for prototyping future extensions to this specification. General rules for all hooks --------------------------- Finding the source tree ~~~~~~~~~~~~~~~~~~~~~~~ All hooks are run with the process working directory set to the root of the source tree (i.e., the directory containing ``pyproject.toml``). To find the source tree, hooks should call ``os.getpwd()`` or equivalent. Rationale: the process working directory has to be set to something, and if we were to leave it up to the build frontend to pick, then packages developers would accidentally write code that assumes a particular answer here (example: ``long_desc = open("README.rst").read()``), and this code would break when used with other build frontends. So it's important that we standardize a value for all build frontends to use consistently. And this is the obvious thing to specify it as, especially because it's compatible with popular and long-standing conventions like calling ``open("README.rst").read()``. Then, given that we've decided to standardize on working directory = source directory, it makes sense to say that this is the *only* way that this information is passed, because providing a second redundant way (example: as an explicit argument to hooks) would only increase the possiblity of error without any benefit. Lifecycle ~~~~~~~~~ XX TODO: do we want to require frontends to use a new process for every hook call, or do we want to require backends to support multiple calls from the same process? Apparently scons and setuptools both can get cranky if you try to invoke them twice from the same process, so *someone* will be spawning extra processes here; the question is where to put that responsibility. The basic trade-off is that making it the backend's responsibility has better best-case performance if both the frontend and backend are able to re-use a single host process; but, if common frontends end up using new processes for each hook call for other reasons, then in practice either backends will end up spawning unnecessary extra processes, or else will end up with poorly tested paths when multiple hooks are run in the same process. Given that ``get_build_*_requires`` → ``build_*`` in general requires changing the Python environment, it doesn't necessarily make sense to run these in the same process anyway. However, there's an important special case where it does: when ``get_build_*_requires`` returns ``[]``. And this is probably the overwhelmingly most common case. Does it even matter? Windows is notoriously slow at spawning subprocesses. As a quick test, I tried measuring the time to spawn CPython 3.6 + import a package on a Windows 10 VM running on my laptop. ``python3.6 -c "import flit"`` was about 300 ms per call; ``python3.6 -c "import setuptools"`` was about 600 ms per call. We could also potentially get fancy and have a flag to let the frontend and backend negotiate this (e.g. ``process_reuse_safe`` as an opt-in flag). This could also be added later as an extension, as long as we initially default to requiring separate processes for each hook. Calling conventions ~~~~~~~~~~~~~~~~~~~ Hooks MAY be called with positional or keyword arguments, so backends implementing them MUST be careful to make sure that their signatures – including argument names – exactly match those specified here. Output ~~~~~~ Hooks MAY print arbitrary informational text on stdout and stderr. They MUST NOT read from stdin, and the build frontend MAY close stdin before invoking the hooks. The build frontend may capture stdout and/or stderr from the backend. If the backend detects that an output stream is not a terminal/console (e.g. ``not sys.stdout.isatty()``), it SHOULD ensure that any output it writes to that stream is UTF-8 encoded. The build frontend MUST NOT fail if captured output is not valid UTF-8, but it MAY not preserve all the information in that case (e.g. it may decode using the *replace* error handler in Python). If the output stream is a terminal, the build backend is responsible for presenting its output accurately, as for any program running in a terminal. If a hook raises any exception, or causes the process to terminate, then this indicates that the operation has failed. User-specified configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ All hooks take a standard ``config_settings`` argument. This argument is an arbitrary dictionary provided as an "escape hatch" for users to pass ad-hoc configuration into individual package builds. Build backends MAY assign any semantics they like to this dictionary. Build frontends SHOULD provide some mechanism for users to specify arbitrary string-key/string-value pairs to be placed in this dictionary. For example, they might support some syntax like ``--package-config CC=gcc``. Build frontends MAY also provide arbitrary other mechanisms for users to place entries in this dictionary. For example, ``pip`` might choose to map a mix of modern and legacy command line arguments like:: pip install \ --package-config CC=gcc \ --global-option="--some-global-option" \ --build-option="--build-option1" \ --build-option="--build-option2" into a ``config_settings`` dictionary like:: { "CC": "gcc", "--global-option": ["--some-global-option"], "--build-option": ["--build-option1", "--build-option2"], } Of course, it's up to users to make sure that they pass options which make sense for the particular build backend and package that they are building. Hook execution environment ~~~~~~~~~~~~~~~~~~~~~~~~~~ One of the responsibilities of a build frontend is to set up the Python environment in which the build backend will run. We do not require that any particular "virtual environment" mechanism be used; a build frontend might use virtualenv, or venv, or no special mechanism at all. But whatever mechanism is used MUST meet the following criteria: - All requirements specified by the project's build-requirements must be available for import from Python. In particular, the distributions specified in the ``pyproject.toml`` key ``build-system.requires`` must be made available to all hooks. Some hooks have additional requirements documented below. - This must remain true even for new Python subprocesses spawned by the build environment, e.g. code like:: import sys, subprocess subprocess.check_call([sys.executable, ...]) must spawn a Python process which has access to all the project's build-requirements. For example, this is necessary to support build backends that want to run legacy ``setup.py`` scripts in a subprocess. - All command-line scripts provided by the build-required packages must be present in the build environment's PATH. For example, if a project declares a build-requirement on `flit <https://flit.readthedocs.org/en/latest/>`__, then the following must work as a mechanism for running the flit command-line tool:: import subprocess subprocess.check_call(["flit", ...]) A build backend MUST be prepared to function in any environment which meets the above criteria. In particular, it MUST NOT assume that it has access to any packages except those that are present in the stdlib, or that are explicitly declared as build-requirements. Building an sdist ----------------- Building an sdist involves three phases: 1. The frontend calls the backend's ``get_requires_for_build_sdist`` hook to query for any extra requirements that are needed for the sdist build. 2. The frontend obtains those requirements. For example, it might download them from PyPI and install them into some kind of virtual environment. 3. The frontend calls the backend's ``build_sdist`` hook to create the sdist. If either hook is missing, or returns the built-in constant ``NotImplemented``. (Note that this is the object ``NotImplemented``, *not* the string ``"NotImplemented"``), then this indicates that this backend does not support building an sdist from this source tree. For example, some build backends might only support building sdists from a VCS checkout, and not from an unpacked sdist. If this occurs then the frontend should respond in whatever way it feels is appropriate. For example, it might display an error to the user. get_requires_for_build_sdist ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :: def get_requires_for_build_sdist(config_settings): ... Computes any additional requirements needed for ``build_sdist``. Returns: a list of strings containing PEP 508 dependency specifications, or ``NotImplemented``. Execution environment: everything specified by the ``build-system.requires`` key in ``pyproject.toml``. Example:: def get_requires_for_build_sdist(config_settings): return ["cython"] Or if there are no additional requirements beyond those specified in ``pyproject.toml``:: def get_requires_for_build_sdist(config_settings): return [] build_sdist ~~~~~~~~~~~ :: def build_sdist(sdist_directory, config_settings): ... Builds a ``.tar.gz`` source distribution and places it in the specified ``sdist_directory``. Returns: The basename (not the full path) of the new ``.tar.gz`` file, as a unicode string, or ``NotImplemented``. Execution environment: everything specified by the ``build-system.requires`` key in ``pyproject.toml`` and by the return value of ``get_requires_for_build_sdist``. Notes: A .tar.gz source distribution (sdist) is named like ``{name}-{version}.tar.gz`` (for example: ``foo-1.0.tar.gz``), and contains a single top-level directory called ``{name}-{version}`` (for example: ``foo-1.0``), which contains the source files of the package. This directory must also contain the ``pyproject.toml`` from the build directory, and a PKG-INFO file containing metadata in the format described in `PEP 345 <https://www.python.org/dev/peps/pep-0345/>`_. Although historically zip files have also been used as sdists, this hook should produce a gzipped tarball. This is already the more common format for sdists, and having a consistent format makes for simpler tooling, so build backends MUST generate ``.tar.gz`` sdists. The generated tarball should use the modern POSIX.1-2001 pax tar format, which specifies UTF-8 based file names. This is not yet the default for the tarfile module shipped with Python 3.6, so backends using the tarfile module need to explicitly pass ``format=tarfile.PAX_FORMAT``. Building a wheel ---------------- The interface for building a wheel is exactly analogous to that for building an sdist: the same three phases, the same interpretation of ``NotImplemented``, etc., except of course that at the end it produces a wheel instead of an sdist. get_requires_for_build_wheel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :: def get_requires_for_build_wheel(config_settings): ... Computes any additional requirements needed for ``build_wheel``. Returns: a list of strings containing PEP 508 dependency specifications, or ``NotImplemented``. Execution environment: everything specified by the ``build-system.requires`` key in ``pyproject.toml``. Example:: def get_requires_for_build_wheel(config_settings): return ["wheel >= 0.25", "setuptools"] build_wheel ~~~~~~~~~~~ :: def build_wheel(wheel_directory, config_settings): ... Builds a ``.whl`` binary distribution, and places it in the specified ``wheel_directory``. Returns: the basename (not the full path) of the new ``.whl``, as a unicode string, or ``NotImplemented``. Execution environment: everything specified by the ``build-system.requires`` key in ``pyproject.toml`` and by the return value of ``get_requires_for_build_wheel``. Note: If you unpack an sdist named ``{name}-{version}.tar.gz``, and then build a wheel from it, then the resulting wheel MUST be named ``{name}-{version}-{compat-info}.whl``. Extensions ---------- Particular frontends and backends MAY coordinate to define additional hooks beyond those described here, but they MUST NOT claim top-level attributes on the build backend object to do so; these attributes are reserved for future PEPs. Backends MAY provide a ``extensions`` dict, and the semantics of the object at ``BACKEND.extensions["XX"]`` can be defined by the project that owns the name ``XX`` on PyPI. For example, the pip project could choose to define extension hooks like:: BACKEND.extensions["pip"].get_wheel_metadata or:: BACKEND.extensions["pip"]["prepare_build_files"] ===================================================== Recommendations for build frontends (non-normative) ===================================================== A build frontend MAY use any mechanism for setting up a build environment that meets the above criteria. For example, simply installing all build-requirements into the global environment would be sufficient to build any compliant package -- but this would be sub-optimal for a number of reasons. This section contains non-normative advice to frontend implementors. A build frontend SHOULD, by default, create an isolated environment for each build, containing only the standard library and any explicitly requested build-dependencies. This has two benefits: - It allows for a single installation run to build multiple packages that have contradictory build-requirements. E.g. if package1 build-requires pbr==1.8.1, and package2 build-requires pbr==1.7.2, then these cannot both be installed simultaneously into the global environment -- which is a problem when the user requests ``pip install package1 package2``. Or if the user already has pbr==1.8.1 installed in their global environment, and a package build-requires pbr==1.7.2, then downgrading the user's version would be rather rude. - It acts as a kind of public health measure to maximize the number of packages that actually do declare accurate build-dependencies. We can write all the strongly worded admonitions to package authors we want, but if build frontends don't enforce isolation by default, then we'll inevitably end up with lots of packages on PyPI that build fine on the original author's machine and nowhere else, which is a headache that no-one needs. However, there will also be situations where build-requirements are problematic in various ways. For example, a package author might accidentally leave off some crucial requirement despite our best efforts; or, a package might declare a build-requirement on ``foo >= 1.0`` which worked great when 1.0 was the latest version, but now 1.1 is out and it has a showstopper bug; or, the user might decide to build a package against numpy==1.7 -- overriding the package's preferred numpy==1.8 -- to guarantee that the resulting build will be compatible at the C ABI level with an older version of numpy (even if this means the resulting build is unsupported upstream). Therefore, build frontends SHOULD provide some mechanism for users to override the above defaults. For example, a build frontend could have a ``--build-with-system-site-packages`` option that causes the ``--system-site-packages`` option to be passed to virtualenv-or-equivalent when creating build environments, or a ``--build-requirements-override=my-requirements.txt`` option that overrides the project's normal build-requirements. The general principle here is that we want to enforce hygiene on package *authors*, while still allowing *end-users* to open up the hood and apply duct tape when necessary. =================================== Comparison to competing proposals =================================== The primary difference between this and competing proposals (in particular, PEP 516) is that our build backend is defined via a Python hook-based interface rather than a command-line based interface. We do *not* expect that this will, by itself, intrinsically reduce the complexity calling into the backend, because build frontends will in any case want to run hooks inside a child -- this is important to isolate the build frontend itself from the backend code and to better control the build backends execution environment. So under both proposals, there will need to be some code in ``pip`` to spawn a subprocess and talk to some kind of command-line/IPC interface, and there will need to be some code in the subprocess that knows how to parse these command line arguments and call the actual build backend implementation. So this diagram applies to all proposals equally:: +-----------+ +---------------+ +----------------+ | frontend | -spawn-> | child cmdline | -Python-> | backend | | (pip) | | interface | | implementation | +-----------+ +---------------+ +----------------+ The key difference between the two approaches is how these interface boundaries map onto project structure:: .-= This PEP =-. +-----------+ +---------------+ | +----------------+ | frontend | -spawn-> | child cmdline | -Python-> | backend | | (pip) | | interface | | | implementation | +-----------+ +---------------+ | +----------------+ | |______________________________________| | Owned by pip, updated in lockstep | | | PEP-defined interface boundary Changes here require distutils-sig .-= Alternative =-. +-----------+ | +---------------+ +----------------+ | frontend | -spawn-> | child cmdline | -Python-> | backend | | (pip) | | | interface | | implementation | +-----------+ | +---------------+ +----------------+ | | |____________________________________________| | Owned by build backend, updated in lockstep | PEP-defined interface boundary Changes here require distutils-sig By moving the PEP-defined interface boundary into Python code, we gain three key advantages. **First**, because there will likely be only a small number of build frontends (``pip``, and... maybe a few others?), while there will likely be a long tail of custom build backends (since these are chosen separately by each package to match their particular build requirements), the actual diagrams probably look more like:: .-= This PEP =-. +-----------+ +---------------+ +----------------+ | frontend | -spawn-> | child cmdline | -Python+> | backend | | (pip) | | interface | | | implementation | +-----------+ +---------------+ | +----------------+ | | +----------------+ +> | backend | | | implementation | | +----------------+ : : .-= Alternative =-. +-----------+ +---------------+ +----------------+ | frontend | -spawn+> | child cmdline | -Python-> | backend | | (pip) | | | interface | | implementation | +-----------+ | +---------------+ +----------------+ | | +---------------+ +----------------+ +> | child cmdline | -Python-> | backend | | | interface | | implementation | | +---------------+ +----------------+ : : That is, this PEP leads to less total code in the overall ecosystem. And in particular, it reduces the barrier to entry of making a new build system. For example, this is a complete, working build backend:: # mypackage_custom_build_backend.py import os.path import pathlib def get_requires_for_build_wheel(config_settings): return ["wheel"] def build_wheel(wheel_directory, config_settings): from wheel.archive import archive_wheelfile filename = "mypackage-0.1-py2.py3-none-any" path = os.path.join(wheel_directory, filename) archive_wheelfile(path, "src/") return filename def _exclude_hidden_and_special_files(archive_entry): """Tarfile filter to exclude hidden and special files from the archive""" if entry.isfile() or entry.isdir(): if not os.path.basename(archive_entry.name).startswith("."): return archive_entry return None def get_requires_for_build_sdist(config_settings): return [] def build_sdist(sdist_dir, config_settings): sdist_subdir = "mypackage-0.1" sdist_path = pathlib.Path(sdist_dir) / (sdist_subdir + ".tar.gz") sdist = tarfile.open(sdist_path, "w:gz", format=tarfile.PAX_FORMAT) # Tar up the whole directory, minus hidden and special files sdist.add(os.getcwd(), arcname=sdist_subdir, filter=_exclude_hidden_and_special_files) return sdist_subdir + ".tar.gz" Of course, this is a *terrible* build backend: it requires the user to have manually set up the wheel metadata in ``src/mypackage-0.1.dist-info/``; when the version number changes it must be manually updated in multiple places... but it works, and more features could be added incrementally. Much experience suggests that large successful projects often originate as quick hacks (e.g., Linux -- "just a hobby, won't be big and professional"; `IPython/Jupyter <https://en.wikipedia.org/wiki/IPython#Grants_and_awards>`_ -- `a grad student's ``$PYTHONSTARTUP`` file <http://blog.fperez.org/2012/01/ipython-notebook-historical.html>`_), so if our goal is to encourage the growth of a vibrant ecosystem of good build tools, it's important to minimize the barrier to entry. **Second**, because Python provides a simpler yet richer structure for describing interfaces, we remove unnecessary complexity from the specification -- and specifications are the worst place for complexity, because changing specifications requires painful consensus-building across many stakeholders. In the command-line interface approach, we have to come up with ad hoc ways to map multiple different kinds of inputs into a single linear command line (e.g. how do we avoid collisions between user-specified configuration arguments and PEP-defined arguments? how do we specify optional arguments? when working with a Python interface these questions have simple, obvious answers). When spawning and managing subprocesses, there are many fiddly details that must be gotten right, subtle cross-platform differences, and some of the most obvious approaches -- e.g., using stdout to return data for the ``build_requires`` operation -- can create unexpected pitfalls (e.g., what happens when computing the build requirements requires spawning some child processes, and these children occasionally print an error message to stdout? obviously a careful build backend author can avoid this problem, but the most obvious way of defining a Python interface removes this possibility entirely, because the hook return value is clearly demarcated). In general, the need to isolate build backends into their own process means that we can't remove IPC complexity entirely -- but by placing both sides of the IPC channel under the control of a single project, we make it much cheaper to fix bugs in the IPC interface than if fixing bugs requires coordinated agreement and coordinated changes across the ecosystem. **Third**, and most crucially, the Python hook approach gives us much more powerful options for evolving this specification in the future. For concreteness, imagine that next year we add a new ``build_wheel2`` hook, which replaces the current ``build_wheel2`` hook with something that adds new features (for example, the ability to build multiple wheels from the same source tree). In order to manage the transition, we want it to be possible for build frontends to transparently use ``build_wheel2`` when available and fall back onto ``build_wheel`` otherwise; and we want it to be possible for build backends to define both methods, for compatibility with both old and new build frontends. Furthermore, our mechanism should also fulfill two more goals: (a) If new versions of e.g. ``pip`` and ``flit`` are both updated to support the new interface, then this should be sufficient for it to be used; in particular, it should *not* be necessary for every project that *uses* ``flit`` to update its individual ``pyproject.toml`` file. (b) We do not want to have to spawn extra processes just to perform this negotiation, because process spawns can easily become a bottleneck when deploying large multi-package stacks on some platforms (Windows). In the interface described here, all of these goals are easy to achieve. Because ``pip`` controls the code that runs inside the child process, it can easily write it to do something like:: command, backend, args = parse_command_line_args(...) if command == "build_wheel": if hasattr(backend, "build_wheel2"): backend.build_wheel2(...) elif hasattr(backend, "build_wheel"): backend.build_wheel(...) else: # error handling In the alternative where the public interface boundary is placed at the subprocess call, this is not possible -- either we need to spawn an extra process just to query what interfaces are supported (as was included in an earlier draft of PEP 516, an alternative to this), or else we give up on autonegotiation entirely (as in the current version of that PEP), meaning that any changes in the interface will require N individual packages to update their ``pyproject.toml`` files before any change can go live, and that any changes will necessarily be restricted to new releases. ==================== Evolutionary notes ==================== A goal here is to make it as simple as possible to convert old-style sdists to new-style sdists. (E.g., this is one motivation for supporting dynamic build requirements.) The ideal would be that there would be a single static ``pyproject.toml`` that could be dropped into any "version 0" VCS checkout to convert it to the new shiny. This is probably not 100% possible, but we can get close, and it's important to keep track of how close we are... hence this section. A rough plan would be: Create a build system package (``setuptools_pypackage`` or whatever) that knows how to speak whatever hook language we come up with, and convert them into calls to ``setup.py``. This will probably require some sort of hooking or monkeypatching to setuptools to provide a way to extract the ``setup_requires=`` argument when needed, and to provide a new version of the sdist command that generates the new-style format. This all seems doable and sufficient for a large proportion of packages (though obviously we'll want to prototype such a system before we finalize anything here). (Alternatively, these changes could be made to setuptools itself rather than going into a separate package.) But there remain two obstacles that mean we probably won't be able to automatically upgrade packages to the new format: 1) There currently exist packages which insist on particular packages being available in their environment before setup.py is executed. This means that if we decide to execute build scripts in an isolated virtualenv-like environment, then projects will need to check whether they do this, and if so then when upgrading to the new system they will have to start explicitly declaring these dependencies (either via ``setup_requires=`` or via static declaration in ``pyproject.toml``). 2) There currently exist packages which do not declare consistent metadata (e.g. ``egg_info`` and ``bdist_wheel`` might get different ``install_requires=``). When upgrading to the new system, projects will have to evaluate whether this applies to them, and if so they will need to stop doing that. ================================ Rejected and deferred features ================================ A number of potential extra features were discussed beyond the above. For the most part the decision was made that it was better to defer trying to implement these until we had more experience with the basic interface, and to provide a minimal extension interface (the ``extensions`` dictionary) that will allow us to prototype these features before standardizing them. Specifically: * Editable installs: This PEP originally specified another hook, ``install_editable``, to do an editable install (as with ``pip install -e``). It was removed due to the complexity of the topic, but may be specified in a later PEP. Briefly, the questions to be answered include: what reasonable ways existing of implementing an 'editable install'? Should the backend or the frontend pick how to make an editable install? And if the frontend does, what does it need from the backend to do so. * Getting wheel metadata from a source tree without building a wheel: it's believed that when pip adds a backtracking constraint solver for package dependencies, it may be useful to add a hook to query a source tree to get metadata about the wheel that it *would* generate, if it were asked to build a wheel. Specifically, the kind of situation where it's anticipated that this might come up is: 1. Package A depends on B and C==1.0 2. B is only available as an sdist 3. We fetch the sdist for the latest version of B, build it into a wheel, and then discover that it depends on C==1.5, which means that it isn't compatible with this version of A. 4. We fetch the sdist for the latest-but-one version of B, build it into a wheel, and then discover that it depends on C==1.4, which means that it isn't compatible with this version of A. 5. We fetch the sdist for the latest-but-two version of B... The idea would be that we could reduce (but not eliminate) the cost of steps 3, 4, 5, ... if there were a way to query a build backend to find out the requirements without actually building a wheel, which is a potentially expensive operation. Of course, these repeated fetches are expensive no matter what we do, so the ideal solution would be to provide wheels for B, so that none of this needs to be done at all. And for many packages (for example, pure Python packages), building a wheel is nearly as cheap as fetching the metadata. And building a wheel also has the advantage of giving us something we can store in the wheel cache for next time. But perhaps this is still a good idea for packages that are particularly slow to build (for example, complex packages like scipy or qt). It was eventually decided to defer this for now, since it adds non-trivial complexity for build backends (the metadata fetching phase and the wheel building phase run at different times, yet have to produce consistent results), and until pip's backtracking resolver is actually implemented, we're only guessing at the value of this optimization and the exact semantics it will require. * A specialized hook for copying a source tree into a new source tree: in certain cases, like when installing directly from a local VCS checkout, pip prefers to copy the source tree to a temporary directory before building it. This provides some protection against build systems that can give incorrect results when repeatedly building in the same tree. Historically, pip has accomplished this copy using a simple ``shutil.copytree``, but this causes various problems, like copying large git checkouts or intermediate artifacts from previous in-place builds. In the future, therefore, pip might move to a multi-step process like: 1. Create an sdist from the VCS checkout 2. Unpack this sdist into a temporary directory. 3. Build a wheel from the unpacked sdist. 4. Install the wheel. Even better, this provides some guarantee that going from VCS checkout → sdist → wheel will produce identical results to going directly from VCS checkout → wheel. However, this runs into a potential problem: what if this particular combination of source tree + build backend can't actually build an sdist? (For example, `flit <http://flit.readthedocs.org/>`__ may have this limitation for certain trees unpacked from sdists.) Therefore, we considered adding an optional hook like ``prepare_temporary_tree_for_build_wheel`` that would copy the required source files into a specified temporary directory. But: * Such a hook would add non-trivial complexity to this spec: it requires us to promote the idea of an "out of tree build" to a first class concept, and specify which kinds of trees are required to support which operations, etc. * A major motivation for doing the build-sdist-unpack-sdist dance in the first place is that we don't trust the backend code to produce the same result when building from a VCS checkout as when building from an sdist, but if we don't trust the backend then it seems odd to add a special hook that puts the backend in charge of doing the dance. * If sdist creation is unsupported, then pip can fall back on a ``shutil.copytree`` strategy in just a few lines of code * And in fact, for the one known case where this might be a problem (unpacked sdist using flit), ``shutil.copytree`` is essentially optimal * Though in fact for flit, this is still a pointless expense – doing an in-place build is perfectly safe and even more efficient. * Plus projects using flit always have wheels, so this will essentially never even come up in the first place * And pip hasn't even implemented the sdist optimization for legacy ``setup.py``\-based projects yet, so we have no operational experience to refer to and it might turn out there are some unknown-unknowns that we'll want to take into account before standardizing an optimization for it here. And since this would be an optional hook anyway, it's just as easy to add later once the exact parameters are better understood. * There was some discussion of extending these hooks to allow a single source tree to produce multiple wheels. But this is a complex enough topic that it clearly requires its own PEP. * We also discussed making the wheel and sdist hooks build unpacked directories containing the same contents as their respective archives. In some cases this could avoid the need to pack and unpack an archive, but this seems like premature optimisation. It's advantageous for tools to work with archives as the canonical interchange formats (especially for wheels, where the archive format is already standardised). Close control of archive creation is important for reproducible builds. And it's not clear that tasks requiring an unpacked distribution will be more common than those requiring an archive. =========== Copyright =========== This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -n -- Nathaniel J. Smith -- https://vorpus.org

14 161

Including soabi tag in --user path?
by Nathaniel Smith 16 Jul '17

16 Jul '17

Hi all, A user on the pywavelets mailing list seems to have gotten themselves into a mess: AFAICT they did 'pip install --user pywavelets' using a ucs4 python, and then started a ucs2 python and tried to import it, which of course failed because these two python builds have incompatible ABIs, even though they share the same --user directory. In retrospect, this seems like an obvious oversight: right now the --user path (on Linux) is something like ~/.local/lib/python3.5/site-packages Wouldn't it make more sense for it to be something like ~/.local/lib/cpython-35m-x86_64-linux-gnu/site-packages so that Pythons with incompatible ABIs can't see each other's packages? (That's sysconfig.get_config_var("SOABI").) Obviously this can't really be fixed for old versions, but is this a change that should be made in 3.7? -n -- Nathaniel J. Smith -- https://vorpus.org

4 11