Distutils-SIG May 2018

distutils-sig@python.org

31 participants
17 discussions

Surviving a Compromise of PyPI - PEP 458 and 480

by Vladimir Diaz

Hello everyone, I am a research programmer at the NYU School of Engineering. My colleagues (Trishank Kuppusamy and Justin Cappos) and I are requesting community feedback on our proposal, "Surviving a Compromise of PyPI." The two-stage proposal can be reviewed online at: PEP 458 http://legacy.python.org/dev/peps/pep-0458/ PEP 480 http://legacy.python.org/dev/peps/pep-0480/ Summary of the Proposal: "Surviving a Compromise of PyPI" proposes how the Python Package Index (PyPI) can be amended to better protect end users from altered or malicious packages, and to minimize the extent of PyPI compromises against affected users. The proposed integration allows package managers such as pip to be more secure against various types of security attacks on PyPI and defend end users from attackers responding to package requests. Specifically, these PEPs describe how PyPI processes should be adapted to generate and incorporate repository metadata, which are signed text files that describe the packages and metadata available on PyPI. Package managers request (along with the packages) the metadata on PyPI to verify the authenticity of packages before they are installed. The changes to PyPI and tools will be minimal by leveraging a library, The Update Framework <https://github.com/theupdateframework/tuf>, that generates and transparently validates the relevant metadata. The first stage of the proposal (PEP 458 <http://legacy.python.org/dev/peps/pep-0458/>) uses a basic security model that supports verification of PyPI packages signed with cryptographic keys stored on PyPI, requires no action from developers and end users, and protects against malicious CDNs and public mirrors. To support continuous delivery of uploaded packages, PyPI administrators sign for uploaded packages with an online key stored on PyPI infrastructure. This level of security prevents packages from being accidentally or deliberately tampered with by a mirror or a CDN because the mirror or CDN will not have any of the keys required to sign for projects. The second stage of the proposal (PEP 480 <http://legacy.python.org/dev/peps/pep-0480/>) is an extension to the basic security model (discussed in PEP 458) that supports end-to-end verification of signed packages. End-to-end signing allows both PyPI and developers to sign for the packages that are downloaded by end users. If the PyPI infrastructure were to be compromised, attackers would be unable to serve malicious versions of these packages without access to the project's developer key. As in PEP 458, no additional action is required by end users. However, PyPI administrators will need to periodically (perhaps every few months) sign metadata with an offline key. PEP 480 also proposes an easy-to-use key management solution for developers, how to interface with a potential build farm on PyPI infrastructure, and discusses the security benefits of end-to-end signing. The second stage of the proposal simultaneously supports real-time project registration and developer signatures, and when configured to maximize security on PyPI, less than 1% of end users will be at risk even if an attacker controls PyPI and goes undetected for a month. We thank Nick Coghlan and Donald Stufft for their valuable contributions, and Giovanni Bajo and Anatoly Techtonik for their feedback. Thanks, PEP 458 & 480 authors.

1 month, 2 weeks

proposing Python package index upload API spec (potential PEP)

by Sumana Harihareswara

As a new Twine maintainer I've been running into questions like: * Now that Warehouse doesn't use "register" anymore, can we deprecate it from distutils, setuptools, and twine? Are any other package indexes or upload tools using it? https://github.com/pypa/twine/issues/311 * It would be nice if Twine could depend on a package index providing an HTTP 201 response in response to a successful upload, and fail on 200 (a response some non-package-index servers will give to an arbitrary POST request). I do not see specifications to guide me here, e.g., in the official guidance on hosting one's own package index https://packaging.python.org/guides/hosting-your-own-index/ . PEP 301 was long enough ago that it's due an update, and PEP 503 only concerns browsing and download, not upload. I suggest that I write a PEP specifying an API for uploading to a Python package index. This PEP would partially supersede PEP 301 and would document the Warehouse reference implementation. I would write it in collaboration with the Warehouse maintainers who will develop the reference implementation per pypa/warehouse/issues/284 and maybe add a header referring to compliance with this new standard. And I would consult with the maintainers of packaging and distribution tools such as zest.releaser, flit, poetry, devpi, pypiserver, etc. Per Nick Coghlan's formulation, my specific goal here would be close to: > Documenting what the current upload API between twine & warehouse actually is, similar to the way PEP 503 focused on describing the status quo, without making any changes to it. That way, other servers (like devpi) and other upload clients have the info they need to help ensure interoperability. Since Warehouse is trying to redo its various APIs in the next several months, I think it might be more useful to document and work with the new upload API, but I'm open to feedback on this. After a little conversation here on distutils-sig, I believe my steps would be: 1. start a very early PEP draft with lots of To Be Determined blanks, submit as a PR to the python/peps repo, and share it with distutils-sig 2. ping maintainers of related tools 3. discuss with others at the packaging sprints https://wiki.python.org/psf/PackagingSprints next week 4. revise and get consensus, preferably mostly on this list 5. finalize PEP and get PEP accepted by BDFL-Delegate 6. coordinate with PyPA, maintainers of `distutils`, maintainers of packaging and distribution tools, and documentation maintainers to implement PEP compliance Thoughts are welcome. I originally posted this at https://github.com/pypa/packaging-problems/issues/128 . -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

4 months, 2 weeks

Re: [Distutils] [Python-ideas] Pypi private repo's

by Alex Walters

I am fairly sure if you give the PyPA that suggestion, they will just deflate at the thought of the workload. Besides, we already offer private repos for free, several ways ranging from devpi to python -m SimpleHTTPServer in a specially created directory. From: Python-ideas <python-ideas-bounces+tritium-list=sdamon.com(a)python.org> On Behalf Of Nick Humrich Sent: Wednesday, April 4, 2018 12:26 PM To: python-ideas(a)python.org Subject: [Python-ideas] Pypi private repo's I am sure this has been discussed before, and this might not even be the best place for this discussion, but I just wanted to make sure this has been thought about. What if pypi.org <http://pypi.org> supported private repos at a cost, similar to npm? This would be able to help support the cost of pypi, and hopefully make it better/more reliable, thus in turn improving the python community. If this discussion should happen somewhere else, let me know. Nick

1 year, 5 months

Oct 27-28: Bloomberg sponsoring packaging sprint

by Sumana Harihareswara

The weekend of October 27-28, simultaneously in London, UK and New York City, USA, Bloomberg will host a Python packaging and distribution tools event. Please mark your calendars! If you live in North America or Europe and would need assistance to attend this as a mentor/helper, watch for more details in July. If you live outside of the US or UK and would need an invitation letter to get a visa to travel to one of these sprints, please write to Kevin P. Fleming at Bloomberg, kpfleming AT bloomberg DOT net, and he'll start setting you up. Details: Thanks to Bloomberg for their generosity. They're already a Platinum PSF sponsor, and they'll host this, pay for a maintainers'/mentors' dinner the night before, provide clusters of cloud virtual machines for the attendees to use, and book and pay for some contributors' lodging and travel. This'll be an opportunity to advance Python packaging/distro tools, teach new contributors (including many Bloomberg employees), and yeah, if you want to get to know Bloomberg for career reasons, that too. :) We hope mentors can arrive Thursday night 25 Oct, do prep, setup, and dinner on Friday, then participate Sat-Sun, then leave Sunday evening or Monday. We'll be putting more details on these lists (distutils-sig and pypa-dev) and at https://wiki.python.org/psf/PackagingSprints . Thanks to Bloomberg folks Mario Corchero and Henry Kleynhans in London and Kevin P. Fleming in New York City for coordinating this, and thanks especially to Mario and to Paul Ganssle for suggesting it! -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

1 year, 6 months

Summary of PyPI overhaul in new LWN article

by Sumana Harihareswara

Today, LWN published my new article "A new package index for Python". https://lwn.net/Articles/751458/ In it, I discuss security, policy, UX and developer experience changes in the 15+ years since PyPI's founding, new features (and deprecated old features) in Warehouse, and future plans. Plus: screenshots! If you aren't already an LWN subscriber, you can use this subscriber link for the next week to read the article despite the LWN paywall. https://lwn.net/SubscriberLink/751458/81b2759e7025d6b9/ This summary should help occasional Python programmers -- and frequent Pythonists who don't follow packaging/distro discussions closely -- understand why a new application is necessary, what's new, what features are going away, and what to expect in the near future. I also hope it catches the attention of downstreams that ought to migrate. -- Sumana Harihareswara Warehouse project manager Changeset Consulting https://changeset.nyc

1 year, 8 months

sudo pip install: install pip files into /usr/local on Linux?

by Victor Stinner

Hi, pip is currently not well integrated on Linux: it conflicts with the system package manager like apt or rpm. When pip writes files into /usr, it can replace files written by the system package manager and so create different kind of issues. For example, if you check the system integry, you will likely see that some Python files have been modified. I would like to open a discussion to see how each Linux vendor handles the issue, and see if a common solution can be designed. Debian uses /usr for apt-get install and /usr/local for distutils and "sudo pip". Fedora decided to change pip to install files into /usr/local by default, instead of /usr, so "sudo pip install" doesn't replace files installed by dnf (Fedora package manager): https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe It gives you 3 main places to install Python code: /usr (managed by dnf), /usr/local (managed by sudo pip), $HOME/.local (managed by pip --user). Would it make sense to make the Fedora/Debian change upstream? At least, give an opt-in option for Linux vendors to use /usr/local? I propose to make the change upstream because there are still issues, and I don't want to be alone to have to fix them :-) It should be easier if we agree on a filesystem layout and an implementation, so we can collaborate on issues! Issues with the current Fedora implementation: (1) When Python is embedded in an application, there is an issue with the current heuristic to decide if /usr/local should be added to sys.path: https://bugzilla.redhat.com/show_bug.cgi?id=1532287 (2) On Fedora, "sudo pip install -U" currently removes old code from /usr and install the new one in /usr/local. We should leave /usr unchanged, since only dnf should touch /usr. https://bugzilla.redhat.com/show_bug.cgi?id=1550368#c24 The implementation is made of a single patch on the Python site module: https://src.fedoraproject.org/rpms/python3/blob/master/f/00251-change-user-… -- There are two issues related to the "sudo pip" change, but they already exist when pip is installed in $HOME/.local: (3) Priority issue between PATH and PYTHONPATH directories. When the user runs "pip", the pip binary may come from /usr, /usr/local or $HOME/.local/bin, but the Python pip module ("import pip") may come from a different path. Which binary and which module should be used? Obvisouly, users can replace these two environment variables... (4) Related to (3). Running "pip" may run pip binary of one pip version, but pick the "pip" Python module of another pip version. For example, pip9 binary from /usr/bin/pip, but pip10 module from /usr/local. Fedora works around issue (4) with a downstream patch on pip: https://src.fedoraproject.org/rpms/python-pip/blob/master/f/pip9-allow-pip1… -- I don't well well how Linux distributions handle the issue with "sudo pip". So don't hesitate to correct me if I'm wrong :-) My goal is just to start a discussion about a common "upstream" solution. Victor

1 year, 10 months

Excluding specific Python modules from wheels

by Michal Petrucha

Hello everyone, I've been looking for a way to ensure that certain modules don't end up in a wheel, while the rest of the package they reside in does. If I only cared about sdist, I could add a MANIFEST.in, in which I'd exclude those specific files, however, unfortunately, MANIFEST.in has no effect on bdists (at least of the wheel kind). The use case is that our application auto-generates a parser and lexer with ply, and that parser might not work with different versions of ply. Since we don't have a whole lot of control over what version users have installed in their environments, we'd like to generate those modules in the target environment. I took a deep dive into distutils and setuptools, and as far as I can see, any Python modules residing inside a package listed in the packages argument to setup() are included in the distribution unconditionally. Searching this mailing list only reveals a short thread from nine years ago [1] without any solution... For now, the easiest hacky solution for me is to add a couple of os.remove calls to setup.py, but I'm not a big fan of setup.py messing with the source tree. And as a follow-up question, is there any post-installation hook that we could use to trigger regeneration of those files? Any ideas? Michal [1]: https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/…

1 year, 10 months

appveyor updateb broken by changes to pip

by Robin Becker

Not exactly sure if this is the issue, but my appveyor test script contains testenv\scripts\pip.exe install --cache-dir=dist --find-links file:dist reportlab where in turn reportlab setup.py contained install_requires=['pillow>=2.4.0']. After an upgrade sometime between Apr 26 and May 18 this started to fail because pip could no longer find a pillow>=2.4. > "=============================== Python27 TEST START" > 941c:\Python%PYTHON%\python.exe -mvirtualenv testenv > 942New python executable in C:\projects\reportlab\testenv\Scripts\python.exe > 943Installing setuptools, pip, wheel...done. > 944testenv\scripts\pip.exe install --cache-dir=dist --find-links file:dist reportlab > 945Looking in links: file:dist > 946Collecting reportlab > 947Collecting pillow>=2.4.0 (from reportlab) > 948 Could not find a version that satisfies the requirement pillow>=2.4.0 (from reportlab) (from versions: ) > 949No matching distribution found for pillow>=2.4.0 (from reportlab) > 950Command exited with code 1 > 951myenv\scripts\python.exe -mrl_ci_tools email --subject="!!!!! APPVEYOR %APPVEYOR_PROJECT_NAME% build failed" --faddr=rl-wheelbuilder --body="see https://ci.appveyor.com/project/MrRLBitBucket/%PROJECT_NAME%/history" I had assumed that pip would fail to find pillow in the dist folder and then move on to somewhere more useful eg pypi, but with pip 10.0.1 this seems not to happen. It's easy to fix with a separate install of the, but what is the correct way to indicate dependencies which will allow installation from my freshly built wheel? -- Robin Becker

1 year, 10 months

Manual PIP upgrade for TLS 1.2 support fails.

by edlipsongm＠gmail.com

I'm trying to install python 3.5.5 on RHEL without root. I used the referenced --prefix to identify where the install is (per webpage). This worked in 3.5.4. PIP installs, but fails to install other modules. Searching for a solution I found a reference to an external way to upgrade PIP to a version where this problem is supposed to be resolved. I obtainde get_pip.py and ran it, but it too fails (below) for TLS issues. Policy says I have to obscure anything related to my company, hence the paths have been edited in this email. Are there any other solutions? bash-3.2$ which python <installpathObscuredbyPolicy>/Python-3.5.5/python -bash-3.2$ python <homepathObscuredbyPolicy >/python/get-pip.py Collecting pip /tmp/tmp9qehrweu/pip.zip/pip/_vendor/urllib3/util/ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:719)'),)': /simple/pip/ Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:719)'),)': /simple/pip/ Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:719)'),)': /simple/pip/ Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:719)'),)': /simple/pip/ Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:719)'),)': /simple/pip/ Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError(SSLError(1, '[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:719)'),)) - skipping Could not find a version that satisfies the requirement pip (from versions: ) No matching distribution found for pip Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError(SSLError(1, '[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:719)'),)) - skipping

1 year, 10 months

requirements.txt or not requirements.txt?

by Carles Sala Cladellas

Hello here! TL;DR: Should I use requirements.txt, or should I have my dependencies only listed inside setup.py? Long version: The two most common ways to have dependencies listed in a python package are either having them listed both in setup.py and requirements.txt files, or having them only listed in requirements.txt and then import them using some `parse_requirements` methods. In the projects which I am currently working on there are five lists of requirements: - requirements.txt - requirements_test.txt - requirements_dev.txt - install_requires inside setup.py - tests_require inside setup.py I wanted to remove duplicities, so I thought about removing the setup.py lists and parse the dependencies from the requirements*.txt files. But then I came across this, https://stackoverflow.com/a/28842733, which recommends using a `dev` entry in `extras_require` and then using `pip install -e .[dev]` to install them, which could be easily extended to also have a `pip install -e .[test]`. I find this a very clean option, since it reduces the amount of files the repository and makes setup.py and makes setup.py self-contained. Are any known drawbacks to this approach? Is it a good way to go? Regards, -- Carles Sala Cladellas carles(a)pythiac.com / +34 666 278 679 https://www.pythiac.com Pythia Consulting - Data Science as a Service

1 year, 10 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

Distutils-SIG May 2018