[Catalog-sig] Use user-specific site-packages by default?

[Daniel Holth]

On Tue, Feb 5, 2013 at 9:06 AM, Holger Krekel <holger.krekel at gmail.com>wrote:

>
> On Tue, Feb 5, 2013 at 2:13 PM, Lennart Regebro <regebro at gmail.com> wrote:
>
>> On Tue, Feb 5, 2013 at 2:02 PM, Holger Krekel <holger.krekel at gmail.com>
>> wrote:
>> > Dropping the crawling over external pages needs _much_ more than just a
>> few
>> > months deprecation warnings, rather years.   There are many packages out
>> > there, and it would break people's installations.
>>
>> No it won't. Nothing gets uninstalled. What stops working is
>> installing those packages with pip/easy_install. And that will start
>> again as soon as the maintainer uploads the last version to PyPI,
>> which she/he is likely to do quite quickly after people start
>> complaining.
>>
>>
> I wouldn't assume that maintainers are easily reachable.  I've contacted
> at least three people of different (>1K downloads) packages which never
> responded.
>
> And of course, i didn't mean to imply that already installed packages
> would suddenly break. Rather that installation instructions like "use pip
> install X" will just fail with some dependency "Y" not getting installed.
> Or getting installed in some random lower version which might contain evil
> bugs (including security bugs).   For exmaple, the referenced "lockfile"
> project has a "0.2" release on pypi, but is currently at 0.9.
>
>
>> > I certainly agree, though, that the current client-side crawling is a
>> > nuisance and makes for unreliability of installation procedures.  I
>> think we
>> > should move the crawling to the server side and cache packages.
>>
>> That will mean that a man in the middle-attack might poison PyPI's
>> cache. I don't think that's a feasible path forward.
>>
>>
> Like i said (you snipped that part of the mail), it's a matter of policy.
> Externally available packages could be downloaded at once, and not on
> demand.   Such a download and checksumming could be repeated over a period
> of time and from different machines.  Of course a remotely stored package
> could already be compromised - but such a possibility always exists (even
> if an author signs a package with PGP - his machine might be infiltrated,
> or the Jenkins build systems performing automated releases etc.).
>
> Packages does not need to be "cached", as they are not supposed to
>> change. If you change the package you should really release a new
>> version. (Unless you made a mistake and discovered it before anyone
>> actually downloaded it). So what you are proposing is really that PyPI
>> downloads the package from an untrusted source, if the maintainer
>> doesn't upload it. I prefer that we demand that the maintainer upload
>> it.
>>
>>
> I actually think it might make sense to forbid referencing external files
> for _future_ pypi uploads (except "#egg=" references probably).   The
> maintainer trying to do that, then gets a clear error and instructions how
> to proceed.   She is just trying to get something out, so we have her
> attention.
>
> Changing pip/distribute-easy_install defaults to require an option for
> installing packages coming from link rel-types of "download" or "homepage"
> might make sense as well.
>
> In the end, however, none of this prevents MITM attacks between a
> downloader and pypi.python.org.  Or between the uploader and
> pypi.python.org (using basic auth over http often).  Signing methods like
> https://wiki.archlinux.org/index.php/Pacman-key are key.  If a signature
> is available (also at a download_url site), then we can exclude undetected
> tampering.  And there might not be a need to break currently working
> package releases.
>
> It certainly makes sense to fortify python packaging and installation
> procedures, but i'd like a bit more of a systematic view on it, including
> reviews from security-focused people and a somewhat incremental verified
> approach to turn it real and used.
>
> best,
> holger
>

As long as you are trusting PyPI itself, a PyPI-hosted/signed/timestamped
SHA2 hash of the file to be downloaded from an external host would be
enough to detect tampering over time.

pip could come with a copy of PyPI's ssl certificate, verifying that it was
identical to the expected cert rather than signed by one of 100s of trusted
CAs.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20130205/4d2fa936/attachment.html>