Mailman 3 February 2016 - Distutils-SIG

Re: [Distutils] deprecating pip install --target
by Jon Parrott 29 Feb '16

29 Feb '16

Currently, Google App Engine recommends that users use "--target" to vendor packages: https://cloud.google.com/appengine/docs/python/tools/libraries27?hl=en#vend… Ostensibly we also support virtualenvs with only pure-python packages. > Alternatively, it should be possible to *detect* the problem cases, so > why not do that, and reject them? Effectively, reduce the scope of > --target to pure-python wheels only. This would be acceptable for the App Engine use case.

1 0

Can't build C++11 libraries with distutils.command.build_clib because it lacks extra_compile_args
by Matthias Goerner 25 Feb '16

25 Feb '16

Hi! I am trying to use distutils.command's build_clib to compile C++11 code but had to monkey patch build_clib to support the extra_compile_args "-std=c++11", see my_build_libraries in http://sageregina.unhyperbolic.org/setup.py . As you can see, it was only adding the line extra_postargs = build_info.get('extra_compile_args')) to objects = self.compiler.compile( sources, output_dir=self.build_temp, macros=macros, include_dirs=include_dirs, ...) in build_clib.py. Could you make this change to distutils so that is available for everyone and I can eventually stop monkey-patching? Best wishes, Matthias Goerner

2 1

how about -c config.ini
by Carl Karsten 22 Feb '16

22 Feb '16

What are the chances of being able to pass the full path to a config file as a command line option? https://docs.python.org/2/install/index.html#location-and-names-of-config-f… setup.cfg (3) ... I.e., in the current directory (usually the location of the setup script). This, only when running in a script in Travis it may not be obvious what the current dir is, and being able to point to the desired config file would be ... desired. -- Carl K

1 0

multiple backports of ipaddress and a world of pain
by Chris Withers 20 Feb '16

20 Feb '16

Hi All, (Apologies for copying in the maintainers of the two backports and django-netfields directly, I'm not sure you're on this distutils list...) This is painful and horrible, and I wish pip would prevent modules/packages with the same name being installed by different distributions at the same time, but even if it did, that would just force something to happen rather than this: So, RHEL7, for worse or worse, ships with Python 2.7.5. That means to keep pip happy, you need to do these dances in all the virtualenvs you create: http://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarn… http://urllib3.readthedocs.org/en/latest/security.html#pyopenssl One of those extra packages drags in this backport: https://pypi.python.org/pypi/ipaddress Yay! Now we have a happy pip talking to both PyPI and our internal DevPI server! Right, so in a Django project I need to use https://pypi.python.org/pypi/django-netfields. This, however, chooses this backport instead: https://pypi.python.org/pypi/py2-ipaddress So, now we have two packages installing ipaddress.py, except they're two very different versions and make different assumptions about what to do with Python 2 strings. What should happen here? (other than me crying a lot...) Chris

4 7

Question about package metainfo shown on PyPI
by Lele Gaifax 19 Feb '16

19 Feb '16

Hi all, a friend showed me the case of the package "django-ddp": it's PyPI page[1] contains a section "Requires Distributions", apparently matching package' setup.py declarations[2]. Since very few other PyPI pages contain that information, surely not those describing my packages, even if their setup declarations look very similar to the d-ddp one. Is there some magic at work here, or is it just that Django's people manually tweak the settings of their packages directly thru the admin pages on PyPI? Thanks for any clarification, ciao, lele. [1] https://pypi.python.org/pypi/django-ddp/0.19.1 [2] https://github.com/django-ddp/django-ddp/blob/develop/setup.py#L206 -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele(a)metapensiero.it | -- Fortunato Depero, 1929.

1 0

PEP 516 and pypa.json
by Ethan Furman 18 Feb '16

18 Feb '16

Greetings! I saw PEP-0516 go through check-ins, and had a question about the pypa.json portion of the proposal -- namely, why are we using a .json file? I presume this is a file that will be created by hand, and while json is not as ugly as xml, it's certainly not pretty. Can we not use an .ini file, or a white-space significant format? The latter should be familiar, if not comfortable, to any Python programmer. -- ~Ethan~

3 3

abstract build system PEP update
by Robert Collins 18 Feb '16

18 Feb '16

I've just pushed this up. - provide space for config of other things in pypa.json via convention - document PATH as a reliable variable - capture the sdist discussion - remove --root from develop: it isn't needed after discussion on IRC. -Rob diff --git a/build-system-abstraction.rst b/build-system-abstraction.rst index a6e4712..56464f1 100644 --- a/build-system-abstraction.rst +++ b/build-system-abstraction.rst @@ -68,12 +68,15 @@ modelled on pip's existing use of the setuptools setup.py interface. pypa.json --------- -The file ``pypa.json`` acts as neutron configuration file for pip and other +The file ``pypa.json`` acts as neutral configuration file for pip and other tools that want to build source trees to consult for configuration. The absence of a ``pypa.json`` file in a Python source tree implies a setuptools or setuptools compatible build system. -The JSON has the following schema. Extra keys are ignored. +The JSON has the following schema. Extra keys are ignored, which permits the +use of ``pypa.json`` as a configuration file for other related tools. If doing +that the chosen keys must be namespaced - e.g. ``flit`` with keys under that +rather than (say) ``build`` or other generic keys. schema The version of the schema. This PEP defines version "1". Defaults to "1" @@ -130,6 +133,9 @@ Available environment variables These variables are set by the caller of the build system and will always be available. +PATH + The standard system path. + PYTHON As for format variables. @@ -176,7 +182,7 @@ wheel -d OUTPUT_DIR flit wheel -d /tmp/pip-build_1234 -develop [--prefix PREFIX] [--root ROOT] +develop [--prefix PREFIX] Command to do an in-place 'development' installation of the project. Stdout and stderr have no semantic meaning. @@ -185,8 +191,11 @@ develop [--prefix PREFIX] [--root ROOT] that doing so will cause use operations like ``pip install -e foo`` to fail. - The prefix option is used for defining an alternative prefix within the - installation root. + The prefix option is used for defining an alternative prefix for the + installation. While setuptools has ``--root`` and ``--user`` options, + they can be done equivalently using ``--prefix``, and pip or other + tools that accept ``--root`` or ``--user`` options should translate + appropriately. The root option is used to define an alternative root within which the command should operate. @@ -403,6 +412,13 @@ the setuptools shim is in use (with older pip versions), or an environment marker ready pip is in use. The setuptools shim can take care of exploiting the difference older pip versions require. +We discussed having an sdist verb. The main driver for this was to make sure +that build systems were able to produce sdists that pip can build - but this is +circular: the whole point of this PEP is to let pip consume such sdists +reliably and without requiring an implementation of setuptools. Further, while +most everyone agrees that encouraging sdists to be uploaded to PyPI, there +wasn't complete consensus on that. + References ========== -- Robert Collins <rbtcollins(a)hpe.com> Distinguished Technologist HP Converged Cloud

9 20

[final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions
by Nathaniel Smith 18 Feb '16

18 Feb '16

Hi all, I think this is ready for pronouncement now -- thanks to everyone for all their feedback over the last few weeks! The only change relative to the last posting is that we rewrote the section on "Platform detection for installers", to switch to letting distributors explicitly control manylinux1 compatibility by means of a _manylinux module. -n --- PEP: 513 Title: A Platform Tag for Portable Linux Built Distributions Version: $Revision$ Last-Modified: $Date$ Author: Robert T. McGibbon <rmcgibbo(a)gmail.com>, Nathaniel J. Smith <njs(a)pobox.com> BDFL-Delegate: Nick Coghlan <ncoghlan(a)gmail.com> Discussions-To: Distutils SIG <distutils-sig(a)python.org> Status: Draft Type: Informational Content-Type: text/x-rst Created: 19-Jan-2016 Post-History: 19-Jan-2016, 25-Jan-2016, 29-Jan-2016 Abstract ======== This PEP proposes the creation of a new platform tag for Python package built distributions, such as wheels, called ``manylinux1_{x86_64,i386}`` with external dependencies limited to a standardized, restricted subset of the Linux kernel and core userspace ABI. It proposes that PyPI support uploading and distributing wheels with this platform tag, and that ``pip`` support downloading and installing these packages on compatible platforms. Rationale ========= Currently, distribution of binary Python extensions for Windows and OS X is straightforward. Developers and packagers build wheels [1]_ [2]_, which are assigned platform tags such as ``win32`` or ``macosx_10_6_intel``, and upload these wheels to PyPI. Users can download and install these wheels using tools such as ``pip``. For Linux, the situation is much more delicate. In general, compiled Python extension modules built on one Linux distribution will not work on other Linux distributions, or even on different machines running the same Linux distribution with different system libraries installed. Build tools using PEP 425 platform tags [3]_ do not track information about the particular Linux distribution or installed system libraries, and instead assign all wheels the too-vague ``linux_i386`` or ``linux_x86_64`` tags. Because of this ambiguity, there is no expectation that ``linux``-tagged built distributions compiled on one machine will work properly on another, and for this reason, PyPI has not permitted the uploading of wheels for Linux. It would be ideal if wheel packages could be compiled that would work on *any* linux system. But, because of the incredible diversity of Linux systems -- from PCs to Android to embedded systems with custom libcs -- this cannot be guaranteed in general. Instead, we define a standard subset of the kernel+core userspace ABI that, in practice, is compatible enough that packages conforming to this standard will work on *many* linux systems, including essentially all of the desktop and server distributions in common use. We know this because there are companies who have been distributing such widely-portable pre-compiled Python extension modules for Linux -- e.g. Enthought with Canopy [4]_ and Continuum Analytics with Anaconda [5]_. Building on the compability lessons learned from these companies, we thus define a baseline ``manylinux1`` platform tag for use by binary Python wheels, and introduce the implementation of preliminary tools to aid in the construction of these ``manylinux1`` wheels. Key Causes of Inter-Linux Binary Incompatibility ================================================ To properly define a standard that will guarantee that wheel packages meeting this specification will operate on *many* linux platforms, it is necessary to understand the root causes which often prevent portability of pre-compiled binaries on Linux. The two key causes are dependencies on shared libraries which are not present on users' systems, and dependencies on particular versions of certain core libraries like ``glibc``. External Shared Libraries ------------------------- Most desktop and server linux distributions come with a system package manager (examples include ``APT`` on Debian-based systems, ``yum`` on ``RPM``-based systems, and ``pacman`` on Arch linux) that manages, among other responsibilities, the installation of shared libraries installed to system directories such as ``/usr/lib``. Most non-trivial Python extensions will depend on one or more of these shared libraries, and thus function properly only on systems where the user has the proper libraries (and the proper versions thereof), either installed using their package manager, or installed manually by setting certain environment variables such as ``LD_LIBRARY_PATH`` to notify the runtime linker of the location of the depended-upon shared libraries. Versioning of Core Shared Libraries ----------------------------------- Even if the developers a Python extension module wish to use no external shared libraries, the modules will generally have a dynamic runtime dependency on the GNU C library, ``glibc``. While it is possible, statically linking ``glibc`` is usually a bad idea because certain important C functions like ``dlopen()`` cannot be called from code that statically links ``glibc``. A runtime shared library dependency on a system-provided ``glibc`` is unavoidable in practice. The maintainers of the GNU C library follow a strict symbol versioning scheme for backward compatibility. This ensures that binaries compiled against an older version of ``glibc`` can run on systems that have a newer ``glibc``. The opposite is generally not true -- binaries compiled on newer Linux distributions tend to rely upon versioned functions in ``glibc`` that are not available on older systems. This generally prevents wheels compiled on the latest Linux distributions from being portable. The ``manylinux1`` policy ========================= For these reasons, to achieve broad portability, Python wheels * should depend only on an extremely limited set of external shared libraries; and * should depend only on "old" symbol versions in those external shared libraries; and * should depend only on a widely-compatible kernel ABI. To be eligible for the ``manylinux1`` platform tag, a Python wheel must therefore both (a) contain binary executables and compiled code that links *only* to libraries (other than the appropriate ``libpython`` library, which is always a permitted dependency consistent with the PEP 425 ABI tag) with SONAMEs included in the following list: :: libpanelw.so.5 libncursesw.so.5 libgcc_s.so.1 libstdc++.so.6 libm.so.6 libdl.so.2 librt.so.1 libcrypt.so.1 libc.so.6 libnsl.so.1 libutil.so.1 libpthread.so.0 libX11.so.6 libXext.so.6 libXrender.so.1 libICE.so.6 libSM.so.6 libGL.so.1 libgobject-2.0.so.0 libgthread-2.0.so.0 libglib-2.0.so.0 and (b), work on a stock CentOS 5.11 [6]_ system that contains the system package manager's provided versions of these libraries. Because CentOS 5 is only available for x86_64 and i386 architectures, these are the only architectures currently supported by the ``manylinux1`` policy. On Debian-based systems, these libraries are provided by the packages :: libncurses5 libgcc1 libstdc++6 libc6 libx11-6 libxext6 libxrender1 libice6 libsm6 libgl1-mesa-glx libglib2.0-0 On RPM-based systems, these libraries are provided by the packages :: ncurses libgcc libstdc++ glibc libXext libXrender libICE libSM mesa-libGL glib2 This list was compiled by checking the external shared library dependencies of the Canopy [4]_ and Anaconda [5]_ distributions, which both include a wide array of the most popular Python modules and have been confirmed in practice to work across a wide swath of Linux systems in the wild. Many of the permitted system libraries listed above use symbol versioning schemes for backward compatibility. The latest symbol versions provided with the CentOS 5.11 versions of these libraries are: :: GLIBC_2.5 CXXABI_3.4.8 GLIBCXX_3.4.9 GCC_4.2.0 Therefore, as a consequence of requirement (b), any wheel that depends on versioned symbols from the above shared libraries may depend only on symbols with the following versions: :: GLIBC <= 2.5 CXXABI <= 3.4.8 GLIBCXX <= 3.4.9 GCC <= 4.2.0 These recommendations are the outcome of the relevant discussions in January 2016 [7]_, [8]_. Note that in our recommendations below, we do not suggest that ``pip`` or PyPI should attempt to check for and enforce the details of this policy (just as they don't check for and enforce the details of existing platform tags like ``win32``). The text above is provided (a) as advice to package builders, and (b) as a method for allocating blame if a given wheel doesn't work on some system: if it satisfies the policy above, then this is a bug in the spec or the installation tool; if it does not satisfy the policy above, then it's a bug in the wheel. One useful consequence of this approach is that it leaves open the possibility of further updates and tweaks as we gain more experience, e.g., we could have a "manylinux 1.1" policy which targets the same systems and uses the same ``manylinux1`` platform tag (and thus requires no further changes to ``pip`` or PyPI), but that adjusts the list above to remove libraries that have turned out to be problematic or add libraries that have turned out to be safe. Compilation of Compliant Wheels =============================== The way glibc, libgcc, and libstdc++ manage their symbol versioning means that in practice, the compiler toolchains that most developers use to do their daily work are incapable of building ``manylinux1``-compliant wheels. Therefore we do not attempt to change the default behavior of ``pip wheel`` / ``bdist_wheel``: they will continue to generate regular ``linux_*`` platform tags, and developers who wish to use them to generate ``manylinux1``-tagged wheels will have to change the tag as a second post-processing step. To support the compilation of wheels meeting the ``manylinux1`` standard, we provide initial drafts of two tools. Docker Image ------------ The first tool is a Docker image based on CentOS 5.11, which is recommended as an easy to use self-contained build box for compiling ``manylinux1`` wheels [9]_. Compiling on a more recently-released linux distribution will generally introduce dependencies on too-new versioned symbols. The image comes with a full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.2) as well as the latest releases of Python and ``pip``. Auditwheel ---------- The second tool is a command line executable called ``auditwheel`` [10]_ that may aid in package maintainers in dealing with third-party external dependencies. There are at least three methods for building wheels that use third-party external libraries in a way that meets the above policy. 1. The third-party libraries can be statically linked. 2. The third-party shared libraries can be distributed in separate packages on PyPI which are depended upon by the wheel. 3. The third-party shared libraries can be bundled inside the wheel libraries, linked with a relative path. All of these are valid option which may be effectively used by different packages and communities. Statically linking generally requires package-specific modifications to the build system, and distributing third-party dependencies on PyPI may require some coordination of the community of users of the package. As an often-automatic alternative to these options, we introduce ``auditwheel``. The tool inspects all of the ELF files inside a wheel to check for dependencies on versioned symbols or external shared libraries, and verifies conformance with the ``manylinux1`` policy. This includes the ability to add the new platform tag to conforming wheels. More importantly, ``auditwheel`` has the ability to automatically modify wheels that depend on external shared libraries by copying those shared libraries from the system into the wheel itself, and modifying the appropriate ``RPATH`` entries such that these libraries will be picked up at runtime. This accomplishes a similar result as if the libraries had been statically linked without requiring changes to the build system. Packagers are advised that bundling, like static linking, may implicate copyright concerns. Bundled Wheels on Linux ======================= While we acknowledge many approaches for dealing with third-party library dependencies within ``manylinux1`` wheels, we recognize that the ``manylinux1`` policy encourages bundling external dependencies, a practice which runs counter to the package management policies of many linux distributions' system package managers [11]_, [12]_. The primary purpose of this is cross-distro compatibility. Furthermore, ``manylinux1`` wheels on PyPI occupy a different niche than the Python packages available through the system package manager. The decision in this PEP to encourage departure from general Linux distribution unbundling policies is informed by the following concerns: 1. In these days of automated continuous integration and deployment pipelines, publishing new versions and updating dependencies is easier than it was when those policies were defined. 2. ``pip`` users remain free to use the ``"--no-binary"`` option if they want to force local builds rather than using pre-built wheel files. 3. The popularity of modern container based deployment and "immutable infrastructure" models involve substantial bundling at the application layer anyway. 4. Distribution of bundled wheels through PyPI is currently the norm for Windows and OS X. 5. This PEP doesn't rule out the idea of offering more targeted binaries for particular Linux distributions in the future. The model described in this PEP is most ideally suited for cross-platform Python packages, because it means they can reuse much of the work that they're already doing to make static Windows and OS X wheels. We recognize that it is less optimal for Linux-specific packages that might prefer to interact more closely with Linux's unique package management functionality and only care about targeting a small set of particular distos. Security Implications --------------------- One of the advantages of dependencies on centralized libraries in Linux is that bugfixes and security updates can be deployed system-wide, and applications which depend on these libraries will automatically feel the effects of these patches when the underlying libraries are updated. This can be particularly important for security updates in packages engaged in communication across the network or cryptography. ``manylinux1`` wheels distributed through PyPI that bundle security-critical libraries like OpenSSL will thus assume responsibility for prompt updates in response disclosed vulnerabilities and patches. This closely parallels the security implications of the distribution of binary wheels on Windows that, because the platform lacks a system package manager, generally bundle their dependencies. In particular, because it lacks a stable ABI, OpenSSL cannot be included in the ``manylinux1`` profile. Platform Detection for Installers ================================= Above, we defined what it means for a *wheel* to be ``manylinux1``-compatible. Here we discuss what it means for a *Python installation* to be ``manylinux1``-compatible. In particular, this is important for tools like ``pip`` to know when deciding whether or not they should consider ``manylinux1``-tagged wheels for installation. Because the ``manylinux1`` profile is already known to work for the many thousands of users of popular commercial Python distributions, we suggest that installation tools should error on the side of assuming that a system *is* compatible, unless there is specific reason to think otherwise. We know of three main sources of potential incompatibility that are likely to arise in practice: * Eventually, in the future, there may exist distributions that break compatibility with this profile (e.g., if one of the libraries in the profile changes its ABI in a backwards-incompatible way) * A linux distribution that is too old (e.g. RHEL 4) * A linux distribution that does not use ``glibc`` (e.g. Alpine Linux, which is based on musl ``libc``, or Android) Therefore, we propose a two-pronged approach. To catch the first case, we standardize a mechanism for a Python distributor to signal that a particular Python install definitely is or is not compatible with ``manylinux1``: this is done by installing a module named ``_manylinux``, and setting its ``manylinux1_compatible`` attribute. We do not propose adding any such module to the standard library -- this is merely a well-known name by which distributors and installation tools can rendezvous. However, if a distributor does add this module, *they should add it to the standard library* rather than to a ``site-packages/`` directory, because the standard library is inherited by virtualenvs (which we want), and ``site-packages/`` in general is not. Then, to handle the latter two cases for existing Python distributions, we suggest a simple and reliable method to check for the presence and version of ``glibc`` (basically using it as a "clock" for the overall age of the distribution). Specifically, the algorithm we propose is:: def is_manylinux1_compatible(): # Only Linux, and only x86-64 / i386 from distutils.util import get_platform if get_platform() not in ["linux_x86_64", "linux_i386"]: return False # Check for presence of _manylinux module try: import _manylinux return bool(_manylinux.manylinux1_compatible) except (ImportError, AttributeError): # Fall through to heuristic check below pass # Check glibc version. CentOS 5 uses glibc 2.5. return have_compatible_glibc(2, 5) def have_compatible_glibc(major, minimum_minor): import ctypes process_namespace = ctypes.CDLL(None) try: gnu_get_libc_version = process_namespace.gnu_get_libc_version except AttributeError: # Symbol doesn't exist -> therefore, we are not linked to # glibc. return False # Call gnu_get_libc_version, which returns a string like "2.5". gnu_get_libc_version.restype = ctypes.c_char_p version_str = gnu_get_libc_version() # py2 / py3 compatibility: if not isinstance(version_str, str): version_str = version_str.decode("ascii") # Parse string and check against requested version. version = [int(piece) for piece in version_str.split(".")] assert len(version) == 2 if major != version[0]: return False if minimum_minor > version[1]: return False return True **Rejected alternatives:** We also considered using a configuration file, e.g. ``/etc/python/compatibility.cfg``. The problem with this is that a single filesystem might contain many different interpreter environments, each with their own ABI profile -- the ``manylinux1`` compatibility of a system-installed x86_64 CPython might not tell us much about the ``manylinux1`` compatibility of a user-installed i386 PyPy. Locating this configuration information within the Python environment itself ensures that it remains attached to the correct binary, and dramatically simplifies lookup code. We also considered using a more elaborate structure, like a list of all platform tags that should be considered compatible, together with their preference ordering, for example: ``_binary_compat.compatible = ["manylinux1_x86_64", "centos5_x86_64", "linux_x86_64"]``. However, this introduces several complications. For example, we want to be able to distinguish between the state of "doesn't support ``manylinux1``" (or eventually ``manylinux2``, etc.) versus "doesn't specify either way whether it supports ``manylinux1``", which is not entirely obvious in the above representation; and, it's not at all clear what features are really needed vis a vis preference ordering given that right now the only possible platform tags are ``manylinux1`` and ``linux``. So we're deferring a more complete solution here for a separate PEP, when / if Linux gets more platform tags. For the library compatibility check, we also considered much more elaborate checks (e.g. checking the kernel version, searching for and checking the versions of all the individual libraries listed in the ``manylinux1`` profile, etc.), but ultimately decided that this would be more likely to introduce confusing bugs than actually help the user. (For example: different distributions vary in where they actually put these libraries, and if our checking code failed to use the correct path search then it could easily return incorrect answers.) PyPI Support ============ PyPI should permit wheels containing the ``manylinux1`` platform tag to be uploaded. PyPI should not attempt to formally verify that wheels containing the ``manylinux1`` platform tag adhere to the ``manylinux1`` policy described in this document. This verification tasks should be left to other tools, like ``auditwheel``, that are developed separately. Rejected Alternatives ===================== One alternative would be to provide separate platform tags for each Linux distribution (and each version thereof), e.g. ``RHEL6``, ``ubuntu14_10``, ``debian_jessie``, etc. Nothing in this proposal rules out the possibility of adding such platform tags in the future, or of further extensions to wheel metadata that would allow wheels to declare dependencies on external system-installed packages. However, such extensions would require substantially more work than this proposal, and still might not be appreciated by package developers who would prefer not to have to maintain multiple build environments and build multiple wheels in order to cover all the common Linux distributions. Therefore we consider such proposals to be out-of-scope for this PEP. Future updates ============== We anticipate that at some point in the future there will be a ``manylinux2`` specifying a more modern baseline environment (perhaps based on CentOS 6), and someday a ``manylinux3`` and so forth, but we defer specifying these until we have more experience with the initial ``manylinux1`` proposal. References ========== .. [1] PEP 0427 -- The Wheel Binary Package Format 1.0 (https://www.python.org/dev/peps/pep-0427/) .. [2] PEP 0491 -- The Wheel Binary Package Format 1.9 (https://www.python.org/dev/peps/pep-0491/) .. [3] PEP 425 -- Compatibility Tags for Built Distributions (https://www.python.org/dev/peps/pep-0425/) .. [4] Enthought Canopy Python Distribution (https://store.enthought.com/downloads/) .. [5] Continuum Analytics Anaconda Python Distribution (https://www.continuum.io/downloads) .. [6] CentOS 5.11 Release Notes (https://wiki.centos.org/Manuals/ReleaseNotes/CentOS5.11) .. [7] manylinux-discuss mailing list discussion (https://groups.google.com/forum/#!topic/manylinux-discuss/-4l3rrjfr9U) .. [8] distutils-sig discussion (https://mail.python.org/pipermail/distutils-sig/2016-January/027997.html) .. [9] manylinux1 docker image (https://quay.io/repository/manylinux/manylinux) .. [10] auditwheel tool (https://pypi.python.org/pypi/auditwheel) .. [11] Fedora Bundled Software Policy (https://fedoraproject.org/wiki/Bundled_Software_policy) .. [12] Debian Policy Manual -- 4.13: Convenience copies of code (https://www.debian.org/doc/debian-policy/ch-source.html#s-embeddedfiles) Copyright ========= This document has been placed into the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- Nathaniel J. Smith -- https://vorpus.org

20 65

Alternative build system abstraction PEP
by Nathaniel Smith 17 Feb '16

17 Feb '16

Hi all, Finally found the time to sit down and take the various drafts I've sent of this to the list before, add a more detailed rationale section, and turn it into a pull request: https://github.com/pypa/interoperability-peps/pull/63 I've also included the text below for reference, in case people want to quote-and-comment here rather than in the PR. The main differences from Robert's proposal are described in the sections "Build backend interface" and "Comparison to competing proposals". -n --- PEP: ?? Title: A build-system independent format for source trees Version: $Revision$ Last-Modified: $Date$ Author: Nathaniel J. Smith <njs(a)pobox.com> Status: Draft Type: Standards-Track Content-Type: text/x-rst Created: 30-Sep-2015 Post-History: 1 Oct 2015, 25 Oct 2015 Discussions-To: <distutils-sig(a)python.org> ========== Abstract ========== While ``distutils`` / ``setuptools`` have taken us a long way, they suffer from three serious problems: (a) they're missing important features like usable build-time dependency declaration, autoconfiguration, and even basic ergonomic niceties like `DRY <https://en.wikipedia.org/wiki/Don%27t_repeat_yourself>`_-compliant version number management, and (b) extending them is difficult, so while there do exist various solutions to the above problems, they're often quirky, fragile, and expensive to maintain, and yet (c) it's very difficult to use anything else, because distutils/setuptools provide the standard interface for installing packages expected by both users and installation tools like ``pip``. Previous efforts (e.g. distutils2 or setuptools itself) have attempted to solve problems (a) and/or (b). This proposal aims to solve (c). The goal of this PEP is get distutils-sig out of the business of being a gatekeeper for Python build systems. If you want to use distutils, great; if you want to use something else, then that should be easy to do using standardized methods. The difficulty of interfacing with distutils means that there aren't many such systems right now, but to give a sense of what we're thinking about see `flit <https://github.com/takluyver/flit>`_ or `bento <https://cournape.github.io/Bento/>`_. Fortunately, wheels have now solved many of the hard problems here -- e.g. it's no longer necessary that a build system also know about every possible installation configuration -- so pretty much all we really need from a build system is that it have some way to spit out standard-compliant wheels and sdists. We therefore propose a new, relatively minimal interface for installation tools like ``pip`` to interact with package source trees and source distributions. ======================= Terminology and goals ======================= A *source tree* is something like a VCS checkout. We need a standard interface for installing from this format, to support usages like ``pip install some-directory/``. A *source distribution* is a static snapshot representing a particular release of some source code, like ``lxml-3.4.4.zip``. Source distributions serve many purposes: they form an archival record of releases, they provide a stupid-simple de facto standard for tools that want to ingest and process large corpora of code, possibly written in many languages (e.g. code search), they act as the input to downstream packaging systems like Debian/Fedora/Conda/..., and so forth. In the Python ecosystem they additionally have a particularly important role to play, because packaging tools like ``pip`` are able to use source distributions to fulfill binary dependencies, e.g. if there is a distribution ``foo.whl`` which declares a dependency on ``bar``, then we need to support the case where ``pip install bar`` or ``pip install foo`` automatically locates the sdist for ``bar``, downloads it, builds it, and installs the resulting package. Source distributions are also known as *sdists* for short. A *build frontend* is a tool that users might run that takes arbitrary source trees or source distributions and builds wheels from them. The actual building is done by each source tree's *build backend*. In a command like ``pip wheel some-directory/``, pip is acting as a build frontend. An *integration frontend* is a tool that users might run that takes a set of package requirements (e.g. a requirements.txt file) and attempts to update a working environment to satisfy those requirements. This may require locating, building, and installing a combination of wheels and sdists. In a command like ``pip install lxml==2.4.0``, pip is acting as an integration frontend. ============== Source trees ============== We retroactively declare the legacy source tree format involving ``setup.py`` to be "version 0". We don't try to specify it further; its de facto specification is encoded in the source code and documentation of ``distutils``, ``setuptools``, ``pip``, and other tools. A "version 1" (or greater) source tree is any directory which contains a file named ``pypackage.json``. (If a tree contains both ``pypackage.json`` and ``setup.py`` then it is a version 1+ source tree and the ``setup.py`` is ignored; this allows packages to include a ``setup.py`` for compatibility with old build frontends, while using the new system with new build frontends.) This file has the following schema. Extra keys are ignored. schema The version of the schema. This PEP defines version "1". Defaults to "1" when absent. All tools reading the file MUST error on an unrecognised schema version. bootstrap_requires Optional list of PEP 508 dependency specifications that the build frontend must ensure are available before invoking the build backend. For instance, if using flit, then the requirements might be:: "bootstrap_requires": ["flit"] build_backend A mandatory string naming a Python object that will be used to perform the build (see below for details). This is formatted following the same ``module:object`` syntax as a ``setuptools`` entry point. For instance, if using flit, then the build system might be specified as:: "build_system": "flit.api:main" and this object would be looked up by executing the equivalent of:: import flit.api backend = flit.api.main It's also legal to leave out the ``:object`` part, e.g. :: "build_system": "flit.api" which acts like:: import flit.api backend = flit.api Formally, the string should satisfy this grammar:: identifier = (letter | '_') (letter | '_' | digit)* module_path = identifier ('.' identifier)* object_path = identifier ('.' identifier)* entry_point = module_path (':' object_path)? And we import ``module_path`` and then lookup ``module_path.object_path`` (or just ``module_path`` if ``object_path`` is missing). ========================= Build backend interface ========================= The build backend object is expected to have attributes which provide some or all of the following hooks. The common ``config_settings`` argument is described after the individual hooks:: def get_build_requires(config_settings): ... This hook MUST return an additional list of strings containing PEP 508 dependency specifications, above and beyond those specified in the ``pypackage.json`` file. Example:: def get_build_requires(config_settings): return ["wheel >= 0.25", "setuptools"] Optional. If not defined, the default implementation is equivalent to ``return []``. :: def get_wheel_metadata(metadata_directory, config_settings): ... Must create a ``.dist-info`` directory containing wheel metadata inside the specified ``metadata_directory`` (i.e., creates a directory like ``{metadata_directory}/{package}-{version}.dist-info/``. This directory MUST be a valid ``.dist-info`` directory as defined in the wheel specification, except that it need not contain ``RECORD`` or signatures. The hook MAY also create other files inside this directory, and a build frontend MUST ignore such files; the intention here is that in cases where the metadata depends on build-time decisions, the build backend may need to record these decisions in some convenient format for re-use by the actual wheel-building step. Return value is ignored. Optional. If a build frontend needs this information and the method is not defined, it should call ``build_wheel`` and look at the resulting metadata directly. :: def build_wheel(wheel_directory, config_settings, metadata_directory=None): ... Must build a ``.whl`` file, and place it in the specified ``wheel_directory``. If the build frontend has previously called ``get_wheel_metadata`` and depends on the wheel resulting from this call to have metadata matching this earlier call, then it should provide the path to the previous ``metadata_directory`` as an argument. If this argument is provided, then ``build_wheel`` MUST produce a wheel with identical metadata. The directory passed in by the build frontend MUST be identical to the directory created by ``get_wheel_metadata``, including any unrecognized files it created. Mandatory. :: def install_editable(prefix, config_settings, metadata_directory=None): ... Must perform whatever actions are necessary to install the current project into the Python installation at ``install_prefix`` in an "editable" fashion. This is intentionally underspecified, because it's included as a stopgap to avoid regressing compared to the current equally underspecified setuptools ``develop`` command; hopefully a future PEP will replace this hook with something that works better and is better specified. (Unfortunately, cleaning up editable installs to actually work well and be well-specified turns out to be a large and difficult job, so we prefer not to do a half-way job here.) For the meaning and requirements of the ``metadata_directory`` argument, see ``build_wheel`` above. Optional. If not defined, then this build backend does not support editable builds. :: config_settings This argument, which is passed to all hooks, is an arbitrary dictionary provided as an "escape hatch" for users to pass ad-hoc configuration into individual package builds. Build backends MAY assign any semantics they like to this dictionary. Build frontends SHOULD provide some mechanism for users to specify arbitrary string-key/string-value pairs to be placed in this dictionary. For example, they might support some syntax like ``--package-config CC=gcc``. Build frontends MAY also provide arbitrary other mechanisms for users to place entries in this dictionary. For example, ``pip`` might choose to map a mix of modern and legacy command line arguments like:: pip install \ --package-config CC=gcc \ --global-option="--some-global-option" \ --build-option="--build-option1" \ --build-option="--build-option2" into a ``config_settings`` dictionary like:: { "CC": "gcc", "--global-option": ["--some-global-option"], "--build-option": ["--build-option1", "--build-option2"], } Of course, it's up to users to make sure that they pass options which make sense for the particular build backend and package that they are building. All hooks are run with working directory set to the root of the source tree, and MAY print arbitrary informational text on stdout and stderr. They MUST NOT read from stdin, and the build frontend MAY close stdin before invoking the hooks. If a hook raises an exception, or causes the process to terminate, then this indicates an error. Build environment ================= One of the responsibilities of a build frontend is to set up the Python environment in which the build backend will run. We do not require that any particular "virtual environment" mechanism be used; a build frontend might use virtualenv, or venv, or no special mechanism at all. But whatever mechanism is used MUST meet the following criteria: - All requirements specified by the project's build-requirements must be available for import from Python. In particular: - The ``get_build_requires`` hook is executed in an environment which contains the bootstrap requirements specified in the ``pypackage.json`` file. - All other hooks are executed in an environment which contains both the bootstrap requirements specified in the ``pypackage.json`` hook and those specified by the ``get_build_requires`` hook. - This must remain true even for new Python subprocesses spawned by the build environment, e.g. code like:: import sys, subprocess subprocess.check_call([sys.executable, ...]) must spawn a Python process which has access to all the project's build-requirements. This is necessary e.g. for build backends that want to run legacy ``setup.py`` scripts in a subprocess. - All command-line scripts provided by the build-required packages must be present in the build environment's PATH. For example, if a project declares a build-requirement on `flit <https://flit.readthedocs.org/en/latest/>`_, then the following must work as a mechanism for running the flit command-line tool:: import subprocess subprocess.check_call(["flit", ...]) A build backend MUST be prepared to function in any environment which meets the above criteria. In particular, it MUST NOT assume that it has access to any packages except those that are present in the stdlib, or that are explicitly declared as build-requirements. Recommendations for build frontends (non-normative) --------------------------------------------------- A build frontend MAY use any mechanism for setting up a build environment that meets the above criteria. For example, simply installing all build-requirements into the global environment would be sufficient to build any compliant package -- but this would be sub-optimal for a number of reasons. This section contains non-normative advice to frontend implementors. A build frontend SHOULD, by default, create an isolated environment for each build, containing only the standard library and any explicitly requested build-dependencies. This has two benefits: - It allows for a single installation run to build multiple packages that have contradictory build-requirements. E.g. if package1 build-requires pbr==1.8.1, and package2 build-requires pbr==1.7.2, then these cannot both be installed simultaneously into the global environment -- which is a problem when the user requests ``pip install package1 package2``. Or if the user already has pbr==1.8.1 installed in their global environment, and a package build-requires pbr==1.7.2, then downgrading the user's version would be rather rude. - It acts as a kind of public health measure to maximize the number of packages that actually do declare accurate build-dependencies. We can write all the strongly worded admonitions to package authors we want, but if build frontends don't enforce isolation by default, then we'll inevitably end up with lots of packages on PyPI that build fine on the original author's machine and nowhere else, which is a headache that no-one needs. However, there will also be situations where build-requirements are problematic in various ways. For example, a package author might accidentally leave off some crucial requirement despite our best efforts; or, a package might declare a build-requirement on `foo >= 1.0` which worked great when 1.0 was the latest version, but now 1.1 is out and it has a showstopper bug; or, the user might decide to build a package against numpy==1.7 -- overriding the package's preferred numpy==1.8 -- to guarantee that the resulting build will be compatible at the C ABI level with an older version of numpy (even if this means the resulting build is unsupported upstream). Therefore, build frontends SHOULD provide some mechanism for users to override the above defaults. For example, a build frontend could have a ``--build-with-system-site-packages`` option that causes the ``--system-site-packages`` option to be passed to virtualenv-or-equivalent when creating build environments, or a ``--build-requirements-override=my-requirements.txt`` option that overrides the project's normal build-requirements. The general principle here is that we want to enforce hygiene on package *authors*, while still allowing *end-users* to open up the hood and apply duct tape when necessary. ====================== Source distributions ====================== For now, we continue with the legacy sdist format which is mostly undefined, but basically comes down to: a file named ``{NAME}-{VERSION}.{EXT}``, which unpacks into a buildable source tree called ``{NAME}-{VERSION}/``. Traditionally these have always contained "version 0" source trees; we now allow them to also contain version 1+ source trees. Integration frontends require that an sdist named ``{NAME}-{VERSION}.{EXT}`` will generate a wheel named ``{NAME}-{VERSION}-{COMPAT-INFO}.whl``. =================================== Comparison to competing proposals =================================== The primary difference between this and competing proposals (`in particular <https://github.com/pypa/interoperability-peps/pull/54/files>`_) is that our build backend is defined via a Python hook-based interface rather than a command-line based interface. We do *not* expect that this will, by itself, intrinsically reduce the complexity calling into the backend, because build frontends will in any case want to run hooks inside a child -- this is important to isolate the build frontend itself from the backend code and to better control the build backends execution environment. So under both proposals, there will need to be some code in ``pip`` to spawn a subprocess and talk to some kind of command-line/IPC interface, and there will need to be some code in the subprocess that knows how to parse these command line arguments and call the actual build backend implementation. So this diagram applies to all proposals equally:: +-----------+ +---------------+ +----------------+ | frontend | -spawn-> | child cmdline | -Python-> | backend | | (pip) | | interface | | implementation | +-----------+ +---------------+ +----------------+ The key difference between the two approaches is how these interface boundaries map onto project structure:: .-= This PEP =-. +-----------+ +---------------+ | +----------------+ | frontend | -spawn-> | child cmdline | -Python-> | backend | | (pip) | | interface | | | implementation | +-----------+ +---------------+ | +----------------+ | |______________________________________| | Owned by pip, updated in lockstep | | | PEP-defined interface boundary Changes here require distutils-sig .-= Alternative =-. +-----------+ | +---------------+ +----------------+ | frontend | -spawn-> | child cmdline | -Python-> | backend | | (pip) | | | interface | | implementation | +-----------+ | +---------------+ +----------------+ | | |____________________________________________| | Owned by build backend, updated in lockstep | PEP-defined interface boundary Changes here require distutils-sig By moving the PEP-defined interface boundary into Python code, we gain three key advantages. **First**, because there will likely be only a small number of build frontends (``pip``, and... maybe a few others?), while there will likely be a long tail of custom build backends (since these are chosen separately by each package to match their particular build requirements), the actual diagrams probably look more like:: .-= This PEP =-. +-----------+ +---------------+ +----------------+ | frontend | -spawn-> | child cmdline | -Python+> | backend | | (pip) | | interface | | | implementation | +-----------+ +---------------+ | +----------------+ | | +----------------+ +> | backend | | | implementation | | +----------------+ : : .-= Alternative =-. +-----------+ +---------------+ +----------------+ | frontend | -spawn+> | child cmdline | -Python-> | backend | | (pip) | | | interface | | implementation | +-----------+ | +---------------+ +----------------+ | | +---------------+ +----------------+ +> | child cmdline | -Python-> | backend | | | interface | | implementation | | +---------------+ +----------------+ : : That is, this PEP leads to less total code in the overall ecosystem. And in particular, it reduces the barrier to entry of making a new build system. For example, this is a complete, working build backend:: # mypackage_custom_build_backend.py import os.path def get_build_requires(config_settings, config_directory): return ["wheel"] def build_wheel(wheel_directory, config_settings, config_directory=None): from wheel.archive import archive_wheelfile path = os.path.join(wheel_directory, "mypackage-0.1-py2.py3-none-any") archive_wheelfile(path, "src/") Of course, this is a *terrible* build backend: it requires the user to have manually set up the wheel metadata in ``src/mypackage-0.1.dist-info/``; when the version number changes it must be manually updated in multiple places; it doesn't implement the metadata or develop hooks, ... but it works, and these features can be added incrementally. Much experience suggests that large successful projects often originate as quick hacks (e.g., Linux -- "just a hobby, won't be big and professional"; `IPython/Jupyter <https://en.wikipedia.org/wiki/IPython#Grants_and_awards>`_ -- `a grad student's ``$PYTHONSTARTUP`` file <http://blog.fperez.org/2012/01/ipython-notebook-historical.html>`_), so if our goal is to encourage the growth of a vibrant ecosystem of good build tools, it's important to minimize the barrier to entry. **Second**, because Python provides a simpler yet richer structure for describing interfaces, we remove unnecessary complexity from the specification -- and specifications are the worst place for complexity, because changing specifications requires painful consensus-building across many stakeholders. In the command-line interface approach, we have to come up with ad hoc ways to map multiple different kinds of inputs into a single linear command line (e.g. how do we avoid collisions between user-specified configuration arguments and PEP-defined arguments? how do we specify optional arguments? when working with a Python interface these questions have simple, obvious answers). When spawning and managing subprocesses, there are many fiddly details that must be gotten right, subtle cross-platform differences, and some of the most obvious approaches -- e.g., using stdout to return data for the ``build_requires`` operation -- can create unexpected pitfalls (e.g., what happens when computing the build requirements requires spawning some child processes, and these children occasionally print an error message to stdout? obviously a careful build backend author can avoid this problem, but the most obvious way of defining a Python interface removes this possibility entirely, because the hook return value is clearly demarcated). In general, the need to isolate build backends into their own process means that we can't remove IPC complexity entirely -- but by placing both sides of the IPC channel under the control of a single project, we make it much much cheaper to fix bugs in the IPC interface than if fixing bugs requires coordinated agreement and coordinated changes across the ecosystem. **Third**, and most crucially, the Python hook approach gives us much more powerful options for evolving this specification in the future. For concreteness, imagine that next year we add a new ``install_editable2`` hook, which replaces the current ``install_editable`` hook with something better specified. In order to manage the transition, we want it to be possible for build frontends to transparently use ``install_editable2`` when available and fall back onto ``install_editable`` otherwise; and we want it to be possible for build backends to define both methods, for compatibility with both old and new build frontends. Furthermore, our mechanism should also fulfill two more goals: (a) If new versions of e.g. ``pip`` and ``flit`` are both updated to support the new interface, then this should be sufficient for it to be used; in particular, it should *not* be necessary for every project that *uses* ``flit`` to update its individual ``pypackage.json`` file. (b) We do not want to have to spawn extra processes just to perform this negotation, because process spawns can easily become a bottleneck when deploying large multi-package stacks on some platforms (Windows). In the interface described here, all of these goals are easy to achieve. Because ``pip`` controls the code that runs inside the child process, it can easily write it to do something like:: command, backend, args = parse_command_line_args(...) if command == "do_editable_install": if hasattr(backend, "install_editable2"): backend.install_editable2(...) elif hasattr(backend, "install_editable"): backend.install_editable(...) else: # error handling In the alternative where the public interface boundary is placed at the subprocess call, this is not possible -- either we need to spawn an extra process just to query what interfaces are supported (as was included in an earlier version of `this alternative PEP <https://github.com/pypa/interoperability-peps/pull/54/files>`_), or else we give up on autonegotiation entirely (as in the current version of that PEP), meaning that any changes in the interface will require N individual packages to update their ``pypackage.json`` files before any change can go live, and that any changes will necessarily be restricted to new releases. One specific consequence of this is that in this PEP, we're able to make the ``get_wheel_metadata`` command optional. In our design, this can easily be worked around by a tool like ``pip``, which can put code in its subprocess runner like:: def get_wheel_metadata(output_dir, config_settings): if hasattr(backend, "get_wheel_metadata"): backend.get_wheel_metadata(output_dir, config_settings) else: backend.build_wheel(output_dir, config_settings) touch(output_dir / "PIP_ALREADY_BUILT_WHEELS") unzip_metadata(output_dir/*.whl) def build_wheel(output_dir, config_settings, metadata_dir): if os.path.exists(metadata_dir / "PIP_ALREADY_BUILT_WHEELS"): copy(metadata_dir / *.whl, output_dir) else: backend.build_wheel(output_dir, config_settings, metadata_dir) and thus expose a totally uniform interface to the rest of ``pip``, with no extra subprocess calls, no duplicated builds, etc. But obviously this is the kind of code that you only want to write as part of a private, within-project interface. (And, of course, making the ``metadata`` command optional is one piece of lowering the barrier to entry, as discussed above.) Other differences ----------------- Besides the key command line versus Python hook difference described above, there are a few other differences in this proposal: * Metadata command is optional (as described above). * We return metadata as a directory, rather than a single METADATA file. This aligns better with the way that in practice wheel metadata is distributed across multiple files (e.g. entry points), and gives us more options in the future. (For example, instead of following the PEP 426 proposal of switching the format of METADATA to JSON, we might decide to keep the existing METADATA the way it is for backcompat, while adding new extensions as JSON "sidecar" files inside the same directory. Or maybe not; the point is it keeps our options more open.) * We provide a mechanism for passing information between the metadata step and the wheel building step. I guess everyone probably will agree this is a good idea? * We call our config file ``pypackage.json`` instead of ``pypa.json``. This is because it describes a package, rather than describing a packaging authority. But really, who cares. * We provide more detailed recommendations about the build environment, but these aren't normative anyway. ==================== Evolutionary notes ==================== A goal here is to make it as simple as possible to convert old-style sdists to new-style sdists. (E.g., this is one motivation for supporting dynamic build requirements.) The ideal would be that there would be a single static pypackage.cfg that could be dropped into any "version 0" VCS checkout to convert it to the new shiny. This is probably not 100% possible, but we can get close, and it's important to keep track of how close we are... hence this section. A rough plan would be: Create a build system package (``setuptools_pypackage`` or whatever) that knows how to speak whatever hook language we come up with, and convert them into calls to ``setup.py``. This will probably require some sort of hooking or monkeypatching to setuptools to provide a way to extract the ``setup_requires=`` argument when needed, and to provide a new version of the sdist command that generates the new-style format. This all seems doable and sufficient for a large proportion of packages (though obviously we'll want to prototype such a system before we finalize anything here). (Alternatively, these changes could be made to setuptools itself rather than going into a separate package.) But there remain two obstacles that mean we probably won't be able to automatically upgrade packages to the new format: 1) There currently exist packages which insist on particular packages being available in their environment before setup.py is executed. This means that if we decide to execute build scripts in an isolated virtualenv-like environment, then projects will need to check whether they do this, and if so then when upgrading to the new system they will have to start explicitly declaring these dependencies (either via ``setup_requires=`` or via static declaration in ``pypackage.cfg``). 2) There currently exist packages which do not declare consistent metadata (e.g. ``egg_info`` and ``bdist_wheel`` might get different ``install_requires=``). When upgrading to the new system, projects will have to evaluate whether this applies to them, and if so they will need to stop doing that. =========== Copyright =========== This document has been placed in the public domain. -- Nathaniel J. Smith -- https://vorpus.org

4 3

devpi-server/web-3.0: generalized mirroring, speed, new backends
by holger krekel 12 Feb '16

12 Feb '16

The 3.0 releases of devpi-server and devpi-web, the python packaging and work flow system for handling release files, documentation, testing and staging, bring several major improvements: - Due to popular demand we now support generalized mirroring, i.e. you can create mirror indexes which proxy and cache release files from other pypi servers. Even if the mirror goes down, pip-installing will continue to work with your devpi-server instance. Previously we only supported mirroring of pypi.python.org. Using it is simple: http://doc.devpi.net/3.0/userman/devpi_indices.html#mirror-index - For our enterprise clients we majorly worked on improving the speed of serving simple pages which is now several times faster with private indexes. We now also support multiple worker processes both on master and replica sites. http://doc.devpi.net/3.0/adminman/server.html#multiple-server-instances - For our enterprise clients we also introduced a new backend architecture which allows to store server state in sqlite or postgres (which is supported through a separately released plugin). The default remains to use the "sqlite" backend and store files on the filesystem. See http://doc.devpi.net/3.0/adminman/server.html#storage-backend-selection - we started a new "admin" manual for devpi-server which describes features relating to server configuration, replication and security aspects. It's a bit work-in-progress but should already be helpful. http://doc.devpi.net/3.0/adminman/ - A few option names changed and we also released devpi-client-2.5 where we took great care to keep it forward and backward compatible so it should run against devpi-server-2.1 and upwards all the way to 3.0. - The "3.0" major release number increase means that you will need to run through an export/import cycle to upgrade your devpi-2.X installation. For more details, see the changelog and the referenced documentation with the main entry point here: http://doc.devpi.net Many thanks to my partner Florian Schulze and to the several companies who funded parts of the work on 3.0. We are especially grateful for their support to not only cover their own direct issues but also support community driven demands. I'd also like to express my gratitude to Rackspace and Jesse Noller who provide VMs for our open source work and which help a lot with the testing of our releases. We are open towards entering more support contracts to make sure you get what you need out of devpi, tox and pytest which together provide a mature tool chain for professional python development. And speaking of showing support, if you or your company is interested to donate to or attend the largest python testing sprint in history with a particular focus to pytest or tox, please see https://www.indiegogo.com/projects/python-testing-sprint-mid-2016/ have fun, holger krekel, http://merlinux.eu server-3.0.0 (2016-02-12) ------------------------- - dropped support for python2.6 - block most ascii symbols for user and index names except ``-.@_``. unicode characters are fine. - add ``--no-root-pypi`` option which prevents the creation of the ``root/pypi`` mirror instance on first startup. - added optional ``title`` and ``description`` options to users and indexes. - new indexes have no bases by default anymore. If you want to be able to install pypi packages, then you have to explicitly add ``root/pypi`` to the ``bases`` option of your index. - added optional ``custom_data`` option to users. - generalized mirroring to allow adding mirror indexes other than only PyPI - renamed ``pypi_whitelist`` to ``mirror_whitelist`` - speed up simple-page serving for private indexes. A private index with 200 release files should now be some 5 times faster. - internally use normalized project names everywhere, simplifying code and slightly speeding up some operations. - change {name} in route_urls to {project} to disambiguate. This is potentially incompatible for plugins which have registered on existing route_urls. - use "project" variable naming consistently in APIs - drop calling of devpi_pypi_initial hook in favor of the new "devpi_mirror_initialnames(stage, projectnames)" hook which is called when a mirror is initialized. - introduce new "devpiserver_stage_created(stage)" hook which is called for each index which is created. - simplify and unify internal mirroring code some more with "normal" stage handling. - don't persist the list of mirrored project names anymore but rely on a per-process RAM cache and the fact that neither the UI nor pip/easy_install typically need the projectnames list, anyway. - introduce new "devpiserver_storage_backend" hook which allows plugins to provide custom storage backends. When there is more than one backend available, the "--storage" option becomes required for startup. - introduce new "--requests-only" option to start devpi-server in "worker" mode. It can be used both for master and replica sites. It starts devpi-server without event processing and replication threads and thus depends on respective "main" instances (those not using "--request-only") to perform event and hook processing. Each worker instance needs to share the filesystem with a main instance. Worker instances can not serve the "/+status" URL which must always be routed to the main instance. - add more info when importing data. Thanks Marc Abramowitz for the PR. web-3.0.0 (2016-02-12) ---------------------- - dropped support for python2.6 - index.pt, root.pt, style.css: added title and description to users and indexes. - root.pt, style.css: more compact styling of user/index overview using flexbox, resulting in three columns at most sizes - cleanup previously unpacked documentation to remove obsolete files. - store hash of doczip with the unpacked data to avoid unpacking if the data already exists. - project.pt, version.pt: renamed ``pypi_whitelist`` related things to ``mirror_whitelist``. - require and adapt to devpi-server-3.0.0 which always uses normalized project names internally and offers new hooks. devpi-web-3.0.0 is incompatible to devpi-server-2.X. - doc.pt, macros.pt, style.css, docview.js: use scrollbar of documentation iframe, so documentation that contains dynamically resizing elements works correctly. For that to work, the search from and navigation was moved into a wrapping div with class ``header``, so it can overlap the top of the iframe. 2.5.0 (2016-02-08) ------------------ - the ``user`` command now behaves slightly more like ``index`` to show current user settings and modify them. - fix issue309: print server versions with ``devpi --version`` if available. This is only supported on Python 3.x because of shortcomings in older argparse versions for Python 2.x. - fix issue310: with --set-cfg the ``index`` setting in the ``[search]`` section would be set multiple times. - fix getjson to work when no index but a server is selected - allow full urls for getjson - "devpi quickstart" is not documented anymore and will be removed in a later release. -- about me: http://holgerkrekel.net/about-me/ contracting: http://merlinux.eu

3 2