I am a research programmer at the NYU School of Engineering. My colleagues
(Trishank Kuppusamy and Justin Cappos) and I are requesting community
feedback on our proposal, "Surviving a Compromise of PyPI." The two-stage
proposal can be reviewed online at:
Summary of the Proposal:
"Surviving a Compromise of PyPI" proposes how the Python Package Index
(PyPI) can be amended to better protect end users from altered or malicious
packages, and to minimize the extent of PyPI compromises against affected
users. The proposed integration allows package managers such as pip to be
more secure against various types of security attacks on PyPI and defend
end users from attackers responding to package requests. Specifically,
these PEPs describe how PyPI processes should be adapted to generate and
incorporate repository metadata, which are signed text files that describe
the packages and metadata available on PyPI. Package managers request
(along with the packages) the metadata on PyPI to verify the authenticity
of packages before they are installed. The changes to PyPI and tools will
be minimal by leveraging a library, The Update Framework
<https://github.com/theupdateframework/tuf>, that generates and
transparently validates the relevant metadata.
The first stage of the proposal (PEP 458
<http://legacy.python.org/dev/peps/pep-0458/>) uses a basic security model
that supports verification of PyPI packages signed with cryptographic keys
stored on PyPI, requires no action from developers and end users, and
protects against malicious CDNs and public mirrors. To support continuous
delivery of uploaded packages, PyPI administrators sign for uploaded
packages with an online key stored on PyPI infrastructure. This level of
security prevents packages from being accidentally or deliberately tampered
with by a mirror or a CDN because the mirror or CDN will not have any of
the keys required to sign for projects.
The second stage of the proposal (PEP 480
<http://legacy.python.org/dev/peps/pep-0480/>) is an extension to the basic
security model (discussed in PEP 458) that supports end-to-end verification
of signed packages. End-to-end signing allows both PyPI and developers to
sign for the packages that are downloaded by end users. If the PyPI
infrastructure were to be compromised, attackers would be unable to serve
malicious versions of these packages without access to the project's
developer key. As in PEP 458, no additional action is required by end
users. However, PyPI administrators will need to periodically (perhaps
every few months) sign metadata with an offline key. PEP 480 also proposes
an easy-to-use key management solution for developers, how to interface
with a potential build farm on PyPI infrastructure, and discusses the
security benefits of end-to-end signing. The second stage of the proposal
simultaneously supports real-time project registration and developer
signatures, and when configured to maximize security on PyPI, less than 1%
of end users will be at risk even if an attacker controls PyPI and goes
undetected for a month.
We thank Nick Coghlan and Donald Stufft for their valuable contributions,
and Giovanni Bajo and Anatoly Techtonik for their feedback.
PEP 458 & 480 authors.
I'm working on tools to convert python packaging data to rpm spec
files. Most of the conversions are relatively straightforward, but
there's an odd corner case with ">" and versions ending in ".*". That
combination is parsed as a valid specification, but oddly, it seems to
behave the same as ">= version", as demonstrated below. Is this
intentional, or is it a byproduct of parsing ">1.0.*" as a
Should I convert ">1.0.*" into ">=1.0" and mimic the current behavior,
or into something else, like "> 1.0", or ">= 1.1"?
>>> pkg_resources._vendor.packaging.requirements.Requirement('cairocffi > 1.0.*,< 1.0.1').specifier._specs
>>> '1.7' in pkg_resources.Requirement.parse('foo>1.7.*')
On behalf of the PyPA, I am pleased to announce that pip 20.0 has been
This release contains quite a lot of changes (you can see the full list here
The highlights for this release are:
- Shorter filenames in output, when installing from PyPI.
- Automatic fall back to user installs, when main site-packages directory
writable and user installs are allowed.
- Significantly improved wheel building and wheel caching.
- Significantly reworked PEP 425 (binary compatibility tag) computation.
- Better handling of system packages in virtual environments created with
`venv` (PEP 405).
- Older pip entry points have been reinstated temporarily, to aid users with
stale wrapper scripts.
We expect that upgrading to this version of pip, would be painless for most
users. We recommend that users test the release before deploying in
as with any other software release.
The pip development team is extremely grateful to everyone in the community
their contributions. Thanks to everyone who put so much effort into the new
release. Many of the contributions came from community members, whether in
form of code, participation in design discussions and/or bug reports.
I've just posted a final progress report on Discourse about the last
month of Open Tech Fund-supported progress on PyPI's localization and
accessibility features. Including a screenshot and a bar graph!
We've finished our OTF-funded accessibility & internationalization work.
And sometime this month people will be able to use PyPI in Brazilian
Portugese and Japanese!
PyPI project manager
Greetings one and all!
Each year millions of eyes around the world eagerly wait at their
computer with baited breath to learn the timing of the most fabulous,
extravagant, decadent and hyperbolically oversold event of the year: The
Python Packaging Summit. While most of the details aren't finalized (and
in fact basically no details are finalized), in the interest of giving
those who would like to attend a chance at some additional notice on
their travel plans, we have established that the date of the summit:
*Thursday, 16 April 2020* (the day before the first PyCon talk day).
For the past two years we've unofficially held our mini-summit on the
first day of PyCon Sprints, but given that this summit is of particular
interest to maintainers - many of whom would like to spend the first day
of sprints actually sprinting - we have decided to run the summit on the
tutorials day parallel to the Education Summit. We have applied for a
spot in the Hatchery program, but even if we are not accepted as an
official PyCon evenat, we will make arrangements to run the summit
off-site on that day.
For those unaware of the nature of the summit, you may want to pay
attention to the "hyperbolically oversold" portion of my description
less than the "decadent" and "extravagant" portions. The purpose of the
summit is to get a bunch of people interested in Python Packaging
together to discuss and coordinate on important topics in packaging and
how we can move them forward. If you are on the fence about joining and
want to know more, here are some detailed notes taken at the summit last
As I mentioned earlier, we have finalized essentially *no* details of
this summit other than the date. I do not know how many people will be
able to attend or by what mechanism we will prioritize attendance at the
summit if there is limited space.
In the coming months please keep an eye out for:
* Registration details (if such a thing is deemed necessary)
* A call for topic suggestions
* Voting on how to prioritize topic suggestions
Looking forward to seeing you all there!
(There's a copy of this message on stack overlfow at
I'm putting a little time into getting my treap module (like a dict, but
always sorted by key) working with distutils.
I want it to be able to compile and install the Cython version from an
included .c file, or to fall back on a pure python version if that fails.
I'm currently using:
ext_modules=[Extension('pyx_treap', ['pyx_treap.c'], optional=True)],
description='Python implementation of treaps,
...plus a few more options to setup that don't seem relevant.
If I run python3.6 setup.py build, I get:
copying treap.py -> build/lib.linux-x86_64-3.6
copying py_treap.py -> build/lib.linux-x86_64-3.6
copying nest.py -> build/lib.linux-x86_64-3.6
building 'pyx_treap' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall
-g-fstack-protector-strong -Wformat -Werror=format-security
-Wdate-time-D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -c
pyx_treap.c:4:10: fatal error: Python.h: No such file or directory
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
So it seems like despite using Extension('pyx_treap', ['pyx_treap.c'],
optional=True), the cython version is not optional.
Am I missing something? Why doesn't it just continue with the pure python
I realize I could install python3.6-dev or similar to eliminate the error,
but I don't want to assume the user knows that.
Is there an ETA for when setuptools is going to break the test subcommand? It would be useful for me to know how long I have to stop using tests_require and to help OS package maintainers migrate to a new way of running tests.
Thanks for any info!