Hello everyone,
I am a research programmer at the NYU School of Engineering. My colleagues
(Trishank Kuppusamy and Justin Cappos) and I are requesting community
feedback on our proposal, "Surviving a Compromise of PyPI." The two-stage
proposal can be reviewed online at:
PEP 458
http://legacy.python.org/dev/peps/pep-0458/
PEP 480
http://legacy.python.org/dev/peps/pep-0480/
Summary of the Proposal:
"Surviving a Compromise of PyPI" proposes how the Python Package Index
(PyPI) can be amended to better protect end users from altered or malicious
packages, and to minimize the extent of PyPI compromises against affected
users. The proposed integration allows package managers such as pip to be
more secure against various types of security attacks on PyPI and defend
end users from attackers responding to package requests. Specifically,
these PEPs describe how PyPI processes should be adapted to generate and
incorporate repository metadata, which are signed text files that describe
the packages and metadata available on PyPI. Package managers request
(along with the packages) the metadata on PyPI to verify the authenticity
of packages before they are installed. The changes to PyPI and tools will
be minimal by leveraging a library, The Update Framework
<https://github.com/theupdateframework/tuf>, that generates and
transparently validates the relevant metadata.
The first stage of the proposal (PEP 458
<http://legacy.python.org/dev/peps/pep-0458/>) uses a basic security model
that supports verification of PyPI packages signed with cryptographic keys
stored on PyPI, requires no action from developers and end users, and
protects against malicious CDNs and public mirrors. To support continuous
delivery of uploaded packages, PyPI administrators sign for uploaded
packages with an online key stored on PyPI infrastructure. This level of
security prevents packages from being accidentally or deliberately tampered
with by a mirror or a CDN because the mirror or CDN will not have any of
the keys required to sign for projects.
The second stage of the proposal (PEP 480
<http://legacy.python.org/dev/peps/pep-0480/>) is an extension to the basic
security model (discussed in PEP 458) that supports end-to-end verification
of signed packages. End-to-end signing allows both PyPI and developers to
sign for the packages that are downloaded by end users. If the PyPI
infrastructure were to be compromised, attackers would be unable to serve
malicious versions of these packages without access to the project's
developer key. As in PEP 458, no additional action is required by end
users. However, PyPI administrators will need to periodically (perhaps
every few months) sign metadata with an offline key. PEP 480 also proposes
an easy-to-use key management solution for developers, how to interface
with a potential build farm on PyPI infrastructure, and discusses the
security benefits of end-to-end signing. The second stage of the proposal
simultaneously supports real-time project registration and developer
signatures, and when configured to maximize security on PyPI, less than 1%
of end users will be at risk even if an attacker controls PyPI and goes
undetected for a month.
We thank Nick Coghlan and Donald Stufft for their valuable contributions,
and Giovanni Bajo and Anatoly Techtonik for their feedback.
Thanks,
PEP 458 & 480 authors.
As a new Twine maintainer I've been running into questions like:
* Now that Warehouse doesn't use "register" anymore, can we deprecate it from distutils, setuptools, and twine? Are any other package indexes or upload tools using it? https://github.com/pypa/twine/issues/311
* It would be nice if Twine could depend on a package index providing an HTTP 201 response in response to a successful upload, and fail on 200 (a response some non-package-index servers will give to an arbitrary POST request).
I do not see specifications to guide me here, e.g., in the official guidance on hosting one's own package index https://packaging.python.org/guides/hosting-your-own-index/ . PEP 301 was long enough ago that it's due an update, and PEP 503 only concerns browsing and download, not upload.
I suggest that I write a PEP specifying an API for uploading to a Python package index. This PEP would partially supersede PEP 301 and would document the Warehouse reference implementation. I would write it in collaboration with the Warehouse maintainers who will develop the reference implementation per pypa/warehouse/issues/284 and maybe add a header referring to compliance with this new standard. And I would consult with the maintainers of packaging and distribution tools such as zest.releaser, flit, poetry, devpi, pypiserver, etc.
Per Nick Coghlan's formulation, my specific goal here would be close to:
> Documenting what the current upload API between twine & warehouse actually is, similar to the way PEP 503 focused on describing the status quo, without making any changes to it. That way, other servers (like devpi) and other upload clients have the info they need to help ensure interoperability.
Since Warehouse is trying to redo its various APIs in the next several months, I think it might be more useful to document and work with the new upload API, but I'm open to feedback on this.
After a little conversation here on distutils-sig, I believe my steps would be:
1. start a very early PEP draft with lots of To Be Determined blanks, submit as a PR to the python/peps repo, and share it with distutils-sig
2. ping maintainers of related tools
3. discuss with others at the packaging sprints https://wiki.python.org/psf/PackagingSprints next week
4. revise and get consensus, preferably mostly on this list
5. finalize PEP and get PEP accepted by BDFL-Delegate
6. coordinate with PyPA, maintainers of `distutils`, maintainers of packaging and distribution tools, and documentation maintainers to implement PEP compliance
Thoughts are welcome. I originally posted this at https://github.com/pypa/packaging-problems/issues/128 .
--
Sumana Harihareswara
Changeset Consulting
https://changeset.nyc
Work has started on the Open Technology Fund-supported project to improve Warehouse security, accessibility, and internationalization. More details in today's progress report:
https://discuss.python.org/t/pypi-security-work-multifactor-auth-progress-h…
best,
Sumana Harihareswara
Warehouse project manager
Changeset Consulting
I'm making a python package for something python Gedit/Pluma/Xed plugin.
Like other lots of other packages that use Gtk, I use glib-compile-schemas [1] at install time.
I do this by using a custom setup.py- similar to this file this from flux:
https://github.com/xflux-gui/fluxgui/blob/master/setup.py#L81(other app, like Meld to something similar)
It looks like there are a lot of changes happening to setuptools, will this continue to work or will it break one day ?
This isn't the only file I'd like to generate at install time, but might be the hardest to work around if I can't.
The other custom action I'm going looking at is setting `--install-lib` to the editors plugin directory (this varies depending on: Operating system and is the user root ?)
I'm using use pip for this, as the install story for most python plugins for these editors is nonexistant - people have to copy the files to some place manually, and instructions usually miss out Windows and OSX.
[1] glib-compile-schemas takes an XML file and generates a binary GVariant file.
Hello everybody,
my name is Richard Neumann.
I started and maintain, amongst other projects, "mcipc" (
https://github.com/conqp/mcipc).
Now, some user named wangzhizhou (https://pypi.org/user/wangzhizhou/)
published this project without consulting me first on PyPI.
Since mcipc is Free Software, I do not really have a problem with the
publication itself, but that I am linked on the PyPI project page (
https://pypi.org/project/mcipc/) as "Author" and "Maintainer" with my
company's email address, that is neither part of the respective repo
nor of my GitHub account (I assume that they googled my name and got it
from my company's website).
I do not wish to have my private software projects liked to my
company's email account, but have no means on contacting the publisher
on PyPI nor uploading an updated version of the package's metadata
since I am not the owner of the PyPI project.
Is it possible for an administrator to at least change my email address
on PyPI to my private one?
Best regards,
Richard
And now for something completely different
I've implemented PEP517 support for the SCons-powered build system
"enscons", quite possibly the most different pip build backend
available. It should be able to build itself and rsalette (another
enscons-using package) using pep517. Note the newly required
env.Default() that enscons needs to return the target filename to pip.
Due to SCons limitations enscons may not be able to build .tar.gz
sdists on Windows.
enscons is an experimental Python build system that runs on top of
SCons, a Makefile replacement. It is able to build C. It should be
much easier to customize than a setuptools-based system. I originally
wrote it so that I could experiment with the wheel format without
having to hack on setuptools.
It might be useful to you if you have a package that has a complex
dependency tree of generated code between the source and whatever ends
up in the wheel, if you want to build more than one wheel from a
single source tree, or if you have other needs that just don't fit
into setuptools.
enscons also has a setup.py emulation that supports "setup.py develop".
The pep517 package is pretty cool. Thanks!
I would call these pep517 powered builds "pluggable".
Daniel Holth
Hi all,
Apologies if this is not the proper channel for the following question.
I have a project that I would like to post on PyPI that clashes with an existing project that I believe may be abandoned (https://pypi.org/project/toybox/#description). According to this PEP (https://www.python.org/dev/peps/pep-0541/) it appears that the appropriate course of action is to attempt to contact the owner of this namespace and find out if they will transfer ownership to me. In the section of the PEP that discusses prior art, there is a reference to cc'ing folks from npm when analogous issues arise over in JS land. It isn't clear to me from this PEP or its resolution what the proper procedure for pursuing ownership of a claimed, but possibly abandoned namespace is for PyPI. Is there an email address I should cc when contacting the namespace owner? How does one initiate an official request for an abandoned namespace?
Thanks!
Emma
--
Emma Tosch
PhD Candidate, College of Information & Computer Sciences
University of Massachusetts Amherst
etosch(a)cs.umass.edu http://cs.umass.edu/~etosch
I've recently released version 0.2.9 of distlib on PyPI [1]. For newcomers,distlib is a library of packaging functionality which is intended to beusable as the basis for third-party packaging tools.
The main changes in this release are as follows:
* Updated default PyPI URL to https://pypi.org/pypi.
* Relaxed metadata format checks to ignore 'Provides'.
* Fixed #33, #34: simplified script template.
* Fixed #115: Relaxed check for '..' in wheel archive entries by not checking filename parts, only directory segments.
* Fixed #116: Corrected parsing of credentials from URLs.
* Fixed #122: Skipped entries in archive entries ending with '/' (directories) when verifying or installing.
* Commented out Disqus comment section in documentation.
A more detailed change log is available at [2].
Please try it out, and if you find any problems or have any suggestions for improvements,please give some feedback using the issue tracker! [3]
Regards,
Vinay Sajip
[1] https://pypi.org/project/distlib/0.2.9/[2]https://distlib.readthedocs.io/en/latest/overview.html#change-log-for-distl…https://bitbucket.org/pypa/distlib/issues/new
