Mailman 3 February 2018 - Distutils-SIG

Documenting project_urls to suggesting thanking or funding mechanisms
by Philip James 15 Feb '18

15 Feb '18

Hello distutils-sig! This email comes to you by way of an issue I filed ( https://github.com/pypa/setuptools/issues/1276), the tl:dr; of which is: I would like it to be easier for packages and package maintainers to indicate how they want to be thanked or funded for their work. I created https://pypi.org/project/thanks/ to help with this, but right now it's reading from a JSON blob maintained by the library. I would prefer to read from the packages themselves, and the `project_urls` dict seems like a good place to do so. Here is an example from the thanks project: https://github.com/phildini/thanks/commit/a4e549338eb3e3c70b1dd5628b38dcbdb… What I am planning to do: - Update the documentation and sample project with a `Project-URL: Funding` item, to encourage maintainers to add this information. What I would love to know from this list: - What do we think of the name "Funding"? Would you prefer "Thanks"? Cheers, and thank you for your time. Philip

2 2

pypi.python Error 403 Forbidden
by Heiko L. 14 Feb '18

14 Feb '18

hallo, I try to install a software from pypi.python.org and receive following errmsg: urllib2.HTTPError: HTTP Error 403: Forbidden and $ wget http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg HTTP request sent, awaiting response... 403 SSL is required After a few hours I found the following article: https://mail.python.org/pipermail/distutils-sig/2017-October/031712.html ...you can no longer access /simple/ and /packages/ over HTTP and you will have to directly go to HTTPS.... - It is true? - Is this the right place if I have any questions? regards Heiko

3 2

Fwd: Warehouse: package manager features & question about advertising
by Sumana Harihareswara 14 Feb '18

14 Feb '18

Forwarded from pypa-dev https://groups.google.com/forum/#!topic/pypa-dev/xQb5RvDb5rc - the weekly Warehouse update. -------- Forwarded Message -------- Subject: Warehouse: package manager features & question about advertising Date: Tue, 13 Feb 2018 23:15:10 -0500 From: Sumana Harihareswara <sh(a)changeset.nyc> Here's your weekly update on Warehouse, powering the new PyPI.[0] Perhaps the biggest news is that the pace of our progress is making us optimistic; we expect to finish all the issues in the first milestone next week, which means Warehouse will have all the essential features package maintainers need.[1] When we get there, we'll be asking some active maintainers to take some time and poke at the site (in the browser and using the APIs) to let us know of any bugs or confusion. In the past week, we've made a ton of progress on, for instance, viewing releases[2] and managing user emails.[3] You can try those out right now at the pre-production site.[4] And the PyPI footer has various policies properly linked in the footer now -- thanks for your advice, PSF![5] Plus, a fix to human-friendly time indicators.[6] Also: Ever wonder how Twine is structured?[7] How does core metadata with multiple email addresses look?[8] And we continued our work on making our credentials handling for Kubernetes more robust.[9] Part of our work is setting up Warehouse on a good foundation for future work, so we spent some time sorting out stuff like: what API documentation do we need?[10] There's a new GitHub label for issues that ask: what APIs do we need?[11] And we restarted the discussion: How much work should we put into Warehouse localisation?[12] Luke Sneeringer volunteered to work on two-factor auth and PyPI API keys, which is great![13] As usual, the notes from our weekly meeting are on the Packaging Working Group wiki.[14] We've also introduced an overview of Warehouse's near-term progress using the GitHub "Projects" feature[15], in case you want to see what we're working on and what's next in a bit more detail than the roadmap.[16] Folks who want to help: we have several good first contribution issues[17] and a guide to getting started[18]. Also, as we prepare for future publicity pushes, please let me know (replying offlist is probably best): where should we advertise to reach occasional and non-Anglophone programmers?[19] Thanks to Mozilla and the PSF for their support for the PyPI & Warehouse work![20][21] [0] https://github.com/pypa/warehouse/ [1] https://github.com/pypa/warehouse/milestone/8 [2] https://github.com/pypa/warehouse/pull/2879 [3] https://github.com/pypa/warehouse/pull/2904 [4] https://pypi.org/ [5] https://github.com/pypa/warehouse/issues/1989 [6] https://github.com/pypa/warehouse/pull/2924 [7] https://github.com/pypa/twine/pull/296 [8] https://github.com/pypa/python-packaging-user-guide/pull/429 [9] https://github.com/cabotage/cabotage-app/commits/master [10] https://github.com/pypa/warehouse/issues/2913 [11] https://github.com/pypa/warehouse/labels/APIs%2Ffeeds [12] https://github.com/pypa/warehouse/issues/1453 [13] https://github.com/pypa/warehouse/issues/994 [14] https://wiki.python.org/psf/PackagingWG/2018-02-12-Warehouse [15] https://github.com/pypa/warehouse/projects/1 [16] https://wiki.python.org/psf/WarehouseRoadmap [17] https://github.com/pypa/warehouse/issues?q=is%3Aissue+is%3Aopen+label%3A%22… [18] https://warehouse.readthedocs.io/development/getting-started/ [19] https://ask.metafilter.com/319055/How-do-I-reach-occasional-and-non-Angloph… [20] https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html [21] https://blog.mozilla.org/blog/2018/01/23/moss-q4-supporting-python-ecosyste… -- Sumana Harihareswara Warehouse project manager Changeset Consulting https://changeset.nyc

1 0

draft PEP: manylinux2
by Mark Williams 13 Feb '18

13 Feb '18

Hi everyone! The manylinux1 platform tag has been tremendously useful, but unfortunately it's showing its age: https://mail.python.org/pipermail/distutils-sig/2017-April/030360.html https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html Nathaniel identified a list of things to do for its successor, manylinux2: https://mail.python.org/pipermail/distutils-sig/2017-April/030361.html Please find below a draft PEP for manylinux2 that attempts to address these issues. I've also opened a PR against python/peps: https://github.com/python/peps/pull/565 Docker images for x86_64 (and soon i686) are available to test drive: https://hub.docker.com/r/markrwilliams/manylinux2/tags/ Thanks! ---- PEP: 9999 Title: The manylinux2 Platform Tag Version: $Revision$ Last-Modified: $Date$ Author: Mark Williams <mrw(a)enotuniq.org> BDFL-Delegate: Nick Coghlan <ncoghlan(a)gmail.com> Discussions-To: Distutils SIG <distutils-sig(a)python.org> Status: Active Type: Informational Content-Type: text/x-rst Created: Post-History: Resolution: Abstract ======== This PEP proposes the creation of a ``manylinux2`` platform tag to succeed the ``manylinux1`` tag introduced by PEP 513 [1]_. It also proposes that PyPI and ``pip`` both be updated to support uploading, downloading, and installing ``manylinux2`` distributions on compatible platforms. Rationale ========= True to its name, the ``manylinux1`` platform tag has made the installation of binary extension modules a reality on many Linux systems. Libraries like ``cryptography`` [2]_ and ``numpy`` [3]_ are more accessible to Python developers now that their installation on common architectures does not depend on fragile development environments and build toolchains. ``manylinux1`` wheels achieve their portability by allowing the extension modules they contain to link against only a small set of system-level shared libraries that export versioned symbols old enough to benefit from backwards-compatibility policies. Extension modules in a ``manylinux1`` wheel that rely on ``glibc``, for example, must be built against version 2.5 or earlier; they may then be run systems that provide more recent ``glibc`` version that still export the required symbols at version 2.5. PEP 513 drew its whitelisted shared libraries and their symbol versions from CentOS 5.11, which was the oldest supported CentOS release at the time of its writing. Unfortunately, CentOS 5.11 reached its end-of-life on March 31st, 2017 with a clear warning against its continued use. [4]_ No further updates, such as security patches, will be made available. This means that its packages will remain at obsolete versions that hamper the efforts of Python software packagers who use the ``manylinux1`` Docker image. CentOS 6.9 is now the oldest supported CentOS release, and will receive maintenance updates through November 30th, 2020. [5]_ We propose that a new PEP 425-style [6]_ platform tag called ``manylinux2`` be derived from CentOS 6.9 and that the ``manylinux`` toolchain, PyPI, and ``pip`` be updated to support it. The ``manylinux2`` policy ========================= The following criteria determine a ``linux`` wheel's eligibility for the ``manylinux2`` tag: 1. The wheel may only contain binary executables and shared objects compiled for one of the two architectures supported by CentOS 6.9: x86_64 or i686. [5]_ 2. The wheel's binary executables or shared objects may not link against externally-provided libraries except those in the following whitelist: :: libgcc_s.so.1 libstdc++.so.6 libm.so.6 libdl.so.2 librt.so.1 libcrypt.so.1 libc.so.6 libnsl.so.1 libutil.so.1 libpthread.so.0 libresolv.so.2 libX11.so.6 libXext.so.6 libXrender.so.1 libICE.so.6 libSM.so.6 libGL.so.1 libgobject-2.0.so.0 libgthread-2.0.so.0 libglib-2.0.so.0 This list is identical to the externally-provided libraries whitelisted for ``manylinux1``, minus ``libncursesw.so.5`` and ``libpanelw.so.5``. [7]_ ``libpythonX.Y`` remains ineligible for inclusion for the same reasons outlined in PEP 513. On Debian-based systems, these libraries are provided by the packages: ============ ======================================================= Package Libraries ============ ======================================================= libc6 libdl.so.2, libresolv.so.2, librt.so.1, libc.so.6, libpthread.so.0, libm.so.6, libutil.so.1, libcrypt.so.1, libnsl.so.1 libgcc1 libgcc_s.so.1 libgl1 libGL.so.1 libglib2.0-0 libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0 libice6 libICE.so.6 libsm6 libSM.so.6 libstdc++6 libstdc++.so.6 libx11-6 libX11.so.6 libxext6 libXext.so.6 libxrender1 libXrender.so.1 ============ ======================================================= On RPM-based systems, they are provided by these packages: ============ ======================================================= Package Libraries ============ ======================================================= glib2 libglib-2.0.so.0, libgthread-2.0.so.0, libgobject-2.0.so.0 glibc libresolv.so.2, libutil.so.1, libnsl.so.1, librt.so.1, libcrypt.so.1, libpthread.so.0, libdl.so.2, libm.so.6, libc.so.6 libICE libICE.so.6 libX11 libX11.so.6 libXext: libXext.so.6 libXrender libXrender.so.1 libgcc: libgcc_s.so.1 libstdc++ libstdc++.so.6 mesa libGL.so.1 ============ ======================================================= 3. If the wheel contains binary executables or shared objects linked against any whitelisted libraries that also export versioned symbols, they may only depend on the following maximum versions:: GLIBC_2.12 CXXABI_1.3.3 GLIBCXX_3.4.13 GCC_4.3.0 As an example, ``manylinux2`` wheels may include binary artifacts that require ``glibc`` symbols at version ``GLIBC_2.4``, because this an earlier version than the maximum of ``GLIBC_2.12``. 4. If a wheel is built for any version of CPython 2 or CPython versions 3.0 up to and including 3.2, it *must* include a CPython ABI tag indicating its Unicode ABI. A ``manylinux2`` wheel built against Python 2, then, must include either the ``cpy27mu`` tag indicating it was built against an interpreter with the UCS-4 ABI or the ``cpy27m`` tag indicating an interpeter with the UCS-2 ABI. *[Citation for UCS ABI tags?]* 5. A wheel *must not* require the ``PyFPE_jbuf`` symbol. This is achieved by building it against a Python compiled *without* the ``--with-fpectl`` ``configure`` flag. Compilation of Compliant Wheels =============================== Like ``manylinux1``, the ``auditwheel`` tool adds ```manylinux2`` platform tags to ``linux`` wheels built by ``pip wheel`` or ``bdist_wheel`` a ``manylinux2`` Docker container. Docker Images ------------- ``manylinux2`` Docker images based on CentOS 6.9 x86_64 and i686 are provided for building binary ``linux`` wheels that can reliably be converted to ``manylinux2`` wheels. [8]_ These images come with a full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.2) as well as the latest releases of Python and ``pip``. Compatibility with kernels that lack ``vsyscall`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A Docker container assumes that its userland is compatible with its host's kernel. Unfortunately, an increasingly common kernel configuration breaks breaks this assumption for x86_64 CentOS 6.9 Docker images. Versions 2.14 and earlier of ``glibc`` require the kernel provide an archaic system call optimization known as ``vsyscall`` on x86_64. [9]_ To effect the optimization, the kernel maps a read-only page of frequently-called system calls -- most notably ``time(2)`` -- into each process at a fixed memory location. ``glibc`` then invokes these system calls by dereferencing a function pointer to the appropriate offset into the ``vsyscall`` page and calling it. This avoids the overhead associated with invoking the kernel that affects normal system call invocation. ``vsyscall`` has long been deprecated in favor of an equivalent mechanism known as vDSO, or "virtual dynamic shared object", in which the kernel instead maps a relocatable virtual shared object containing the optimized system calls into each process. [10]_ The ``vsyscall`` page has serious security implications because it does not participate in address space layout randomization (ASLR). Its predictable location and contents make it a useful source of gadgets used in return-oriented programming attacks. [11]_ At the same time, its elimination breaks the x86_64 ABI, because ``glibc`` versions that depend on ``vsyscall`` suffer from segmentation faults when attempting to dereference a system call pointer into a non-existent page. As a compromise, Linux 3.1 implemented an "emulated" ``vsyscall`` that reduced the executable code, and thus the material for ROP gadgets, mapped into the process. [12]_ ``vsyscall=emulated`` has been the default configuration in most distribution's kernels for many years. Unfortunately, ``vsyscall`` emulation still exposes predicatable code at a reliable memory location, and continues to be useful for return-oriented programming. [13]_ Because most distributions have now upgraded to ``glibc`` versions that do not depend on ``vsyscall``, they are beginning to ship kernels that do not support ``vsyscall`` at all. [14]_ CentOS 5.11 and 6.9 both include versions of ``glibc`` that depend on the ``vsyscall`` page (2.5 and 2.12.2 respectively), so containers based on either cannot run under kernels provided with many distribution's upcoming releases. [15]_ Continuum Analytics faces a related problem with its conda software suite, and as they point out, this will pose a significant obstacle to using these tools in hosted services. [16]_ If Travis CI, for example, begins running jobs under a kernel that does not provide the ``vsyscall`` interface, Python packagers will not be able to use our Docker images there to build ``manylinux`` wheels. [17]_ We have derived a patch from the ``glibc`` git repository that backports the removal of all dependencies on ``vsyscall`` to the version of ``glibc`` included with our ``manylinux2`` image. [18]_ Rebuilding ``glibc``, and thus building ``manylinux2`` image itself, still requires a host kernel that provides the ``vsyscall`` mechanism, but the resulting image can be both run on hosts that provide it and those that do not. Because the ``vsyscall`` interface is an optimization that is only applied to running processes, the ``manylinux2`` wheels built with this modified image should be identical to those built on an unmodified CentOS 6.9 system. Also, the ``vsyscall`` problem applies only to x86_64; it is not part of the i686 ABI. Auditwheel ---------- The ``auditwheel`` tool has also been updated to produce ``manylinux2`` wheels. [19]_ Its behavior and purpose are otherwise unchanged from PEP 513. Platform Detection for Installers ================================= Platforms may define a ``manylinux2_compatible`` boolean attribute on the ``_manylinux`` module described in PEP 513. A platform is considered incompatible with ``manylinux2`` if the attribute is ``False``. Backwards compatibility with ``manylinux1`` wheels ================================================== As explained in PEP 513, the specified symbol versions for ``manylinux1`` whitelisted libraries constitute an *upper bound*. The same is true for the symbol versions defined for ``manylinux2`` in this PEP. As a result, ``manylinux1`` wheels are considered ``manylinux2`` wheels. A ``pip`` that recognizes the ``manylinux2`` platform tag will thus install ``manylinux1`` wheels for ``manylinux2`` platforms -- even when explicitly set -- when no ``manylinux2`` wheels are available. [20]_ PyPI Support ============ PyPI should permit wheels containing the ``manylinux2`` platform tag to be uploaded in the same way that it permits ``manylinux1``. It should not attempt to verify the compatibility of ``manylinux2`` wheels. References ========== .. [1] PEP 513 -- A Platform Tag for Portable Linux Built Distributions (https://www.python.org/dev/peps/pep-0513/) .. [2] pyca/cryptography (https://cryptography.io/) .. [3] numpy (https://numpy.org) .. [4] CentOS 5.11 EOL announcement (https://lists.centos.org/pipermail/centos-announce/2017-April/022350.html) .. [5] CentOS Product Specifications (https://web.archive.org/web/20180108090257/https://wiki.centos.org/About/Pr…) .. [6] PEP 425 -- Compatibility Tags for Built Distributions (https://www.python.org/dev/peps/pep-0425/) .. [7] ncurses 5 -> 6 transition means we probably need to drop some libraries from the manylinux whitelist (https://github.com/pypa/manylinux/issues/94) .. [8] manylinux2 Docker images (https://hub.docker.com/r/markrwilliams/manylinux2/) .. [9] On vsyscalls and the vDSO (https://lwn.net/Articles/446528/) .. [10] vdso(7) (http://man7.org/linux/man-pages/man7/vdso.7.html) .. [11] Framing Signals -- A Return to Portable Shellcode (http://www.cs.vu.nl/~herbertb/papers/srop_sp14.pdf) .. [12] ChangeLog-3.1 (https://www.kernel.org/pub/linux/kernel/v3.x/ChangeLog-3.1) .. [13] Project Zero: Three bypasses and a fix for one of Flash's Vector.<*> mitigations (https://googleprojectzero.blogspot.com/2015/08/three-bypasses-and-fix-for-o…) .. [14] linux: activate CONFIG_LEGACY_VSYSCALL_NONE ? (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=852620) .. [15] [Wheel-builders] Heads-up re: new kernel configurations breaking the manylinux docker image (https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html) .. [16] Due to glibc 2.12 limitation, static executables that use time(), cpuinfo() and maybe a few others cannot be run on systems that do not support or use `vsyscall=emulate` (https://github.com/ContinuumIO/anaconda-issues/issues/8203) .. [17] Travis CI (https://travis-ci.org/) .. [18] remove-vsyscall.patch https://github.com/markrwilliams/manylinux/commit/e9493d55471d153089df3aafc… .. [19] auditwheel manylinux2 branch (https://github.com/markrwilliams/auditwheel/tree/manylinux2) .. [20] pip manylinux2 branch https://github.com/markrwilliams/pip/commits/manylinux2 Copyright ========= This document has been placed into the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- Mark Williams mrw(a)enotuniq.org

13 28

Installed Extras Metadata
by Pradyun Gedam 11 Feb '18

11 Feb '18

Hello! I hope everyone's had a great start to 2018! :) A few months back, while working on pip, I had noticed an oddity about extras. Installing a package with extras would not store information about the fact that the extras were requested. This means, later, it is not possible to know which extra-based optional dependencies of a package have to be considered when verifying that the packages are compatible with each other. This information is relavant for resolution/validation since without it, it is not possible to know which the extra-requirements to care about. As an example, installing ``requests[security]`` and then uninstalling ``PyOpenSSL`` leaves you in a state where you don't really satisfy what was asked for but there's no way to detect that either. Thus, obviously, I'm interested in making pip to be able to store this information. As I understand, this is done needs to be specified in a PEP and/or on PyPUG's specification page. To that end, here's seeding proposal for the discussion: a new `extras-requested.txt` file in the .dist-info directory, storing the extra names in a one-per-line format. Cheers! Pradyun

5 12

Re: [Distutils] draft PEP: manylinux2
by Alex Walters 08 Feb '18

08 Feb '18

-----Original Message----- From: Alex Walters [mailto:tritium-list@sdamon.com] Sent: Wednesday, February 7, 2018 12:21 AM To: 'Janzert' <janzert(a)janzert.com> Subject: RE: [Distutils] draft PEP: manylinux2 > -----Original Message----- > From: Distutils-SIG [mailto:distutils-sig-bounces+tritium- > list=sdamon.com(a)python.org] On Behalf Of Janzert > Sent: Tuesday, February 6, 2018 3:33 PM > To: Distutils-Sig(a)Python.Org > Subject: Re: [Distutils] draft PEP: manylinux2 > > On 2/5/2018 16:01, Nick Coghlan wrote: > > Compare: > > > > - manylinux1 vs manylinux2 vs manylinux3 > > - manylinux2007 vs manylinux2010 vs manylinux2014 > > > > I'll leave this just as a data point (anecdote) from someone that isn't > heavily involved with linux sysadmin or python packaging. Feel free to > make of it what you like. I generally run debian stable and occasionally > ubuntu lts on servers and the latest ubuntu for my workstation. > > If I were looking to install a package and one of the binaries available > is manylinux2010 I probably completely pass over that option and don't > even attempt using it. My assumption would be anything that old probably > isn't going to work on my 2016 or newer OS. Whereas manylinux(1, 2, 3) I > would think has a good chance of working on any reasonably modern linux. > > If not for having read the discussion here I would have interpreted a > date, especially a date that's the better part of a decade in the past, > completely the wrong way. > > Having said that, I'm pretty sure that pip should in general be handling > this decision for me and doing the right thing anyway? So it probably > doesn't matter too much. > > Janzert > This is a really good point. Since pip is the main interface to packages for end users anyways, we can call it manylinux8675309 and it wouldn't really matter to users - the name only really matters to package maintainers, not users. And because of that, manylinux2010, manylinux2014, etc makes more sense. A package maintainer is expected to be more educated about these matters, and that naming scheme is more useful to them. "Whats the oldest linux system my code will run on?" is a very likely question a maintainer would have when building binary packages, and the year-based naming scheme is the logical answer. +1 to manylinux2010, -0 manylinux2 > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG(a)python.org > https://mail.python.org/mailman/listinfo/distutils-sig

4 4

Building a Python package build service for warehouse
by Jannis Gebauer 07 Feb '18

07 Feb '18

Hi! I’m currently working on a package build server. My goal is to produce useful additional meta data for all packages available on PyPi. This includes: - Transitive dependencies - Is the package installable under Python 3? - Various automated “code quality” tests like pylint, pyflakes, pep8, mccabe etc. - Automated security tests - (possibly changelogs, commit logs) - Licenses! The main idea is to run the build process in a restricted “sandbox” docker container that pulls the package from PyPi, installs it and runs a couple of tools on it. Code is still pretty rough, nothing to look at at the moment I’m afraid. Is there any interest in working on this together? Maybe even with the goal to make it an open API that can be consumed by warehouse et al.? Interested in any thoughts on this! Cheers, Jannis P.S: I’m currently crunching trough the data on a 96 CPU cluster. There’s an API available, but it’s sitting behind HTTP Basic Auth as it is basically an endpoint for remote code execution (and throws lots of 500s :D). Send me a mail to jay(a)pyup.io <mailto:jay@pyup.io> if you want to play around with it.

3 2

Moving PEP 541 (PyPI's name management policy) forward
by Nick Coghlan 07 Feb '18

07 Feb '18

Hi folks, As Sumana mentioned in her latest summary email, we've been looking at PEP 541 again, and figuring out how best to manage the process for getting that approved so that the folks handling name management requests can start relying on it. The key challenge with it has been the fact that this document is essentially defining a new operational policy for the Python Software Foundation (as the legal entity behind PyPI), and given that it relates to naming things on the internet, we can reasonably expect it to prove potentially contentious (and perhaps even turn litigious if someone *really* doesn't like a decision that was made in accordance with the policy). Approving that kind of policy would be a significant burden to place on the shoulders of a single volunteer, so we discussed it in the PSF's Packaging working group [1] (which decides how best to allocate any funding the PSF receives or allocates specifically to support the packaging infrastructure), and amended the approval process for this particular PEP as follows: - the BDFL-Delegate is now Mark Mangoba, in his role as the PSF's IT manager - rather than approving the PEP directly, Mark will instead put it forward for a vote in the Packaging Working Group (similar to the way we handle funding allocation decisions) - input from the PSF's General Counsel will be explicitly requested prior to the vote The commit making those changes can be found here: https://github.com/python/peps/commit/6c356f0472b38e69e8a59a2dee1f275b61aea… distutils-sig will remain the mailing list for discussing the actual contents of the policy (and any future amendments to it) - the change in approach just makes it clearer that the policy needs to be explicitly adopted by the PSF as an entity, rather than by an individual volunteer. Regards, Nick. [1] https://wiki.python.org/psf/PackagingWG -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Fwd: Warehouse update: still on track, new features
by Sumana Harihareswara 07 Feb '18

07 Feb '18

Our weekly Warehouse update just went out to pypa-dev and is included below. -------- Forwarded Message -------- Subject: Warehouse update: still on track, new features Date: Tue, 6 Feb 2018 22:09:12 -0500 From: Sumana Harihareswara <sh(a)changeset.nyc> To: pypa-dev <pypa-dev(a)googlegroups.com> Here's your weekly update on Warehouse, powering the new PyPI.[0] You can see some noticeable improvements to Warehouse right now compared to last week. There's a mobile UI for managing projects[1], and a project owner can now delete a project.[2] We also have several CSS tweaks and other continuing design improvements -- we're lucky to be working with Nicole on this.[3] Less visibly, we have further Kubernetes security work by Ernest in cabotage[4] and Dustin's work on a generic token service[5]. We're still on track to hit the Maintainer MVP milestone at the end of this month.[6] On the documentation and outreach side, Laura and I have been preparing to contact very active maintainers when we hit that milestone, and we've been improving the packaging user guide,[7] and working a bit on Twine (e.g., documentation for using python-keyring with Twine to avoid having to use a .pypirc).[8] Thanks to Jon Wayne Parrott for fixing an issue Dustin spotted[9] so that pypa.io gets fresh updates again.[10] In PEP progress, PEP 541 is moving forward again, with a pull request for a change in BDFL-Delegate.[11] As usual, meeting notes from our weekly discussion are on the wiki.[12] And if you want to get started contributing to Warehouse, Ernest wants to help you and give you stickers, and has 30-minute 1:1 slots available.[13] Right now we have eleven open issues marked as good for newcomers.[14] Thanks to Mozilla for their support for the PyPI & Warehouse work, and thanks to the PSF for facilitating and supporting this work![15][16] [0] https://pypi.org/ [1] https://github.com/pypa/warehouse/pull/2865 [2] https://github.com/pypa/warehouse/pull/2821 [3] http://whoisnicoleharris.com/warehouse/ [4] https://github.com/cabotage/cabotage-app/commits/master [5] https://github.com/pypa/warehouse/pull/2864 [6] https://github.com/pypa/warehouse/milestone/8 [7] https://github.com/pypa/python-packaging-user-guide/pull/426 [8] https://github.com/pypa/python-packaging-user-guide/issues/297#issuecomment… [9] https://groups.google.com/forum/#!topic/pypa-dev/jzXR3A3E-dw [10] https://www.pypa.io/en/latest/roadmap/ [11] https://github.com/python/peps/pull/566 [12] https://wiki.python.org/psf/PackagingWG/2018-02-05-Warehouse [13] https://twitter.com/EWDurbin/status/955415184339849217 [14] https://github.com/pypa/warehouse/issues?q=is%3Aissue+is%3Aopen+label%3A%22… [15] https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html [16] https://blog.mozilla.org/blog/2018/01/23/moss-q4-supporting-python-ecosyste… -- Sumana Harihareswara Warehouse project manager Changeset Consulting https://changeset.nyc -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

1 0

draft PEP: manylinux2
by Mark Williams 02 Feb '18

02 Feb '18

Hi everyone! The manylinux1 platform tag has been tremendously useful, but unfortunately it's showing its age: https://mail.python.org/pipermail/distutils-sig/2017-April/030360.html https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html Nathaniel identified a list of things to do for its successor, manylinux2: https://mail.python.org/pipermail/distutils-sig/2017-April/030361.html Please find below a draft PEP for manylinux2 that attempts to address these issues. I've also opened a PR against python/peps: https://github.com/python/peps/pull/565 Docker images for x86_64 and i686 are available to test drive: https://hub.docker.com/r/markrwilliams/manylinux2/tags/ Thanks! ---- PEP: 9999 Title: The manylinux2 Platform Tag Version: $Revision$ Last-Modified: $Date$ Author: Mark Williams <mrw(a)enotuniq.org> BDFL-Delegate: Nick Coghlan <ncoghlan(a)gmail.com> Discussions-To: Distutils SIG <distutils-sig(a)python.org> Status: Active Type: Informational Content-Type: text/x-rst Created: Post-History: Resolution: Abstract ======== This PEP proposes the creation of a ``manylinux2`` platform tag to succeed the ``manylinux1`` tag introduced by PEP 513 [1]_. It also proposes that PyPI and ``pip`` both be updated to support uploading, downloading, and installing ``manylinux2`` distributions on compatible platforms. Rationale ========= True to its name, the ``manylinux1`` platform tag has made the installation of binary extension modules a reality on many Linux systems. Libraries like ``cryptography`` [2]_ and ``numpy`` [3]_ are more accessible to Python developers now that their installation on common architectures does not depend on fragile development environments and build toolchains. ``manylinux1`` wheels achieve their portability by allowing the extension modules they contain to link against only a small set of system-level shared libraries that export versioned symbols old enough to benefit from backwards-compatibility policies. Extension modules in a ``manylinux1`` wheel that rely on ``glibc``, for example, must be built against version 2.5 or earlier; they may then be run systems that provide more recent ``glibc`` version that still export the required symbols at version 2.5. PEP 513 drew its whitelisted shared libraries and their symbol versions from CentOS 5.11, which was the oldest supported CentOS release at the time of its writing. Unfortunately, CentOS 5.11 reached its end-of-life on March 31st, 2017 with a clear warning against its continued use. [4]_ No further updates, such as security patches, will be made available. This means that its packages will remain at obsolete versions that hamper the efforts of Python software packagers who use the ``manylinux1`` Docker image. CentOS 6.9 is now the oldest supported CentOS release, and will receive maintenance updates through November 30th, 2020. [5]_ We propose that a new PEP 425-style [6]_ platform tag called ``manylinux2`` be derived from CentOS 6.9 and that the ``manylinux`` toolchain, PyPI, and ``pip`` be updated to support it. The ``manylinux2`` policy ========================= The following criteria determine a ``linux`` wheel's eligibility for the ``manylinux2`` tag: 1. The wheel may only contain binary executables and shared objects compiled for one of the two architectures supported by CentOS 6.9: x86_64 or i686. [5]_ 2. The wheel's binary executables or shared objects may not link against externally-provided libraries except those in the following whitelist: :: libgcc_s.so.1 libstdc++.so.6 libm.so.6 libdl.so.2 librt.so.1 libcrypt.so.1 libc.so.6 libnsl.so.1 libutil.so.1 libpthread.so.0 libresolv.so.2 libX11.so.6 libXext.so.6 libXrender.so.1 libICE.so.6 libSM.so.6 libGL.so.1 libgobject-2.0.so.0 libgthread-2.0.so.0 libglib-2.0.so.0 This list is identical to the externally-provided libraries whitelisted for ``manylinux1``, minus ``libncursesw.so.5`` and ``libpanelw.so.5``. [7]_ ``libpythonX.Y`` remains ineligible for inclusion for the same reasons outlined in PEP 513. On Debian-based systems, these libraries are provided by the packages: ============ ======================================================= Package Libraries ============ ======================================================= libc6 libdl.so.2, libresolv.so.2, librt.so.1, libc.so.6, libpthread.so.0, libm.so.6, libutil.so.1, libcrypt.so.1, libnsl.so.1 libgcc1 libgcc_s.so.1 libgl1 libGL.so.1 libglib2.0-0 libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0 libice6 libICE.so.6 libsm6 libSM.so.6 libstdc++6 libstdc++.so.6 libx11-6 libX11.so.6 libxext6 libXext.so.6 libxrender1 libXrender.so.1 ============ ======================================================= On RPM-based systems, they are provided by these packages: ============ ======================================================= Package Libraries ============ ======================================================= glib2 libglib-2.0.so.0, libgthread-2.0.so.0, libgobject-2.0.so.0 glibc libresolv.so.2, libutil.so.1, libnsl.so.1, librt.so.1, libcrypt.so.1, libpthread.so.0, libdl.so.2, libm.so.6, libc.so.6 libICE libICE.so.6 libX11 libX11.so.6 libXext: libXext.so.6 libXrender libXrender.so.1 libgcc: libgcc_s.so.1 libstdc++ libstdc++.so.6 mesa libGL.so.1 ============ ======================================================= 3. If the wheel contains binary executables or shared objects linked against any whitelisted libraries that also export versioned symbols, they may only depend on the following maximum versions:: GLIBC_2.12 CXXABI_1.3.3 GLIBCXX_3.4.13 GCC_4.3.0 As an example, ``manylinux2`` wheels may include binary artifacts that require ``glibc`` symbols at version ``GLIBC_2.4``, because this an earlier version than the maximum of ``GLIBC_2.12``. 4. If a wheel is built for any version of CPython 2 or CPython versions 3.0 up to and including 3.2, it *must* include a CPython ABI tag indicating its Unicode ABI. A ``manylinux2`` wheel built against Python 2, then, must include either the ``cpy27mu`` tag indicating it was built against an interpreter with the UCS-4 ABI or the ``cpy27m`` tag indicating an interpeter with the UCS-2 ABI. *[Citation for UCS ABI tags?]* 5. A wheel *must not* require the ``PyFPE_jbuf`` symbol. This is achieved by building it against a Python compiled *without* the ``--with-fpectl`` ``configure`` flag. Compilation of Compliant Wheels =============================== Like ``manylinux1``, the ``auditwheel`` tool adds ```manylinux2`` platform tags to ``linux`` wheels built by ``pip wheel`` or ``bdist_wheel`` a ``manylinux2`` Docker container. Docker Images ------------- ``manylinux2`` Docker images based on CentOS 6.9 x86_64 and i686 are provided for building binary ``linux`` wheels that can reliably be converted to ``manylinux2`` wheels. [8]_ These images come with a full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.2) as well as the latest releases of Python and ``pip``. Compatibility with kernels that lack ``vsyscall`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A Docker container assumes that its userland is compatible with its host's kernel. Unfortunately, an increasingly common kernel configuration breaks breaks this assumption for x86_64 CentOS 6.9 Docker images. Versions 2.14 and earlier of ``glibc`` require the kernel provide an archaic system call optimization known as ``vsyscall`` on x86_64. [9]_ To effect the optimization, the kernel maps a read-only page of frequently-called system calls -- most notably ``time(2)`` -- into each process at a fixed memory location. ``glibc`` then invokes these system calls by dereferencing a function pointer to the appropriate offset into the ``vsyscall`` page and calling it. This avoids the overhead associated with invoking the kernel that affects normal system call invocation. ``vsyscall`` has long been deprecated in favor of an equivalent mechanism known as vDSO, or "virtual dynamic shared object", in which the kernel instead maps a relocatable virtual shared object containing the optimized system calls into each process. [10]_ The ``vsyscall`` page has serious security implications because it does not participate in address space layout randomization (ASLR). Its predictable location and contents make it a useful source of gadgets used in return-oriented programming attacks. [11]_ At the same time, its elimination breaks the x86_64 ABI, because ``glibc`` versions that depend on ``vsyscall`` suffer from segmentation faults when attempting to dereference a system call pointer into a non-existent page. As a compromise, Linux 3.1 implemented an "emulated" ``vsyscall`` that reduced the executable code, and thus the material for ROP gadgets, mapped into the process. [12]_ ``vsyscall=emulated`` has been the default configuration in most distribution's kernels for many years. Unfortunately, ``vsyscall`` emulation still exposes predicatable code at a reliable memory location, and continues to be useful for return-oriented programming. [13]_ Because most distributions have now upgraded to ``glibc`` versions that do not depend on ``vsyscall``, they are beginning to ship kernels that do not support ``vsyscall`` at all. [14]_ CentOS 5.11 and 6.9 both include versions of ``glibc`` that depend on the ``vsyscall`` page (2.5 and 2.12.2 respectively), so containers based on either cannot run under kernels provided with many distribution's upcoming releases. [15]_ Continuum Analytics faces a related problem with its conda software suite, and as they point out, this will pose a significant obstacle to using these tools in hosted services. [16]_ If Travis CI, for example, begins running jobs under a kernel that does not provide the ``vsyscall`` interface, Python packagers will not be able to use our Docker images there to build ``manylinux`` wheels. [17]_ We have derived a patch from the ``glibc`` git repository that backports the removal of all dependencies on ``vsyscall`` to the version of ``glibc`` included with our ``manylinux2`` image. [18]_ Rebuilding ``glibc``, and thus building ``manylinux2`` image itself, still requires a host kernel that provides the ``vsyscall`` mechanism, but the resulting image can be both run on hosts that provide it and those that do not. Because the ``vsyscall`` interface is an optimization that is only applied to running processes, the ``manylinux2`` wheels built with this modified image should be identical to those built on an unmodified CentOS 6.9 system. Also, the ``vsyscall`` problem applies only to x86_64; it is not part of the i686 ABI. Auditwheel ---------- The ``auditwheel`` tool has also been updated to produce ``manylinux2`` wheels. [19]_ Its behavior and purpose are otherwise unchanged from PEP 513. Platform Detection for Installers ================================= Platforms may define a ``manylinux2_compatible`` boolean attribute on the ``_manylinux`` module described in PEP 513. A platform is considered incompatible with ``manylinux2`` if the attribute is ``False``. Backwards compatibility with ``manylinux1`` wheels ================================================== As explained in PEP 513, the specified symbol versions for ``manylinux1`` whitelisted libraries constitute an *upper bound*. The same is true for the symbol versions defined for ``manylinux2`` in this PEP. As a result, ``manylinux1`` wheels are considered ``manylinux2`` wheels. A ``pip`` that recognizes the ``manylinux2`` platform tag will thus install ``manylinux1`` wheels for ``manylinux2`` platforms -- even when explicitly set -- when no ``manylinux2`` wheels are available. [20]_ PyPI Support ============ PyPI should permit wheels containing the ``manylinux2`` platform tag to be uploaded in the same way that it permits ``manylinux1``. It should not attempt to verify the compatibility of ``manylinux2`` wheels. References ========== .. [1] PEP 513 -- A Platform Tag for Portable Linux Built Distributions (https://www.python.org/dev/peps/pep-0513/) .. [2] pyca/cryptography (https://cryptography.io/) .. [3] numpy (https://numpy.org) .. [4] CentOS 5.11 EOL announcement (https://lists.centos.org/pipermail/centos-announce/2017-April/022350.html) .. [5] CentOS Product Specifications (https://web.archive.org/web/20180108090257/https://wiki.centos.org/About/Pr…) .. [6] PEP 425 -- Compatibility Tags for Built Distributions (https://www.python.org/dev/peps/pep-0425/) .. [7] ncurses 5 -> 6 transition means we probably need to drop some libraries from the manylinux whitelist (https://github.com/pypa/manylinux/issues/94) .. [8] manylinux2 Docker images (https://hub.docker.com/r/markrwilliams/manylinux2/) .. [9] On vsyscalls and the vDSO (https://lwn.net/Articles/446528/) .. [10] vdso(7) (http://man7.org/linux/man-pages/man7/vdso.7.html) .. [11] Framing Signals -- A Return to Portable Shellcode (http://www.cs.vu.nl/~herbertb/papers/srop_sp14.pdf) .. [12] ChangeLog-3.1 (https://www.kernel.org/pub/linux/kernel/v3.x/ChangeLog-3.1) .. [13] Project Zero: Three bypasses and a fix for one of Flash's Vector.<*> mitigations (https://googleprojectzero.blogspot.com/2015/08/three-bypasses-and-fix-for-o…) .. [14] linux: activate CONFIG_LEGACY_VSYSCALL_NONE ? (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=852620) .. [15] [Wheel-builders] Heads-up re: new kernel configurations breaking the manylinux docker image (https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html) .. [16] Due to glibc 2.12 limitation, static executables that use time(), cpuinfo() and maybe a few others cannot be run on systems that do not support or use `vsyscall=emulate` (https://github.com/ContinuumIO/anaconda-issues/issues/8203) .. [17] Travis CI (https://travis-ci.org/) .. [18] remove-vsyscall.patch https://github.com/markrwilliams/manylinux/commit/e9493d55471d153089df3aafc… .. [19] auditwheel manylinux2 branch (https://github.com/markrwilliams/auditwheel/tree/manylinux2) .. [20] pip manylinux2 branch https://github.com/markrwilliams/pip/commits/manylinux2 Copyright ========= This document has been placed into the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- Mark Williams mrw(a)twistedmatrix.com

1 0