From pmiscml at gmail.com  Sun Jun  1 17:02:18 2014
From: pmiscml at gmail.com (Paul Sokolovsky)
Date: Sun, 1 Jun 2014 18:02:18 +0300
Subject: [Distutils] Support for multiple PyPI publishing identities is
 rather convoluted
Message-ID: <20140601180218.7e94e300@x34f>

Hello,

My usecase is: I work on different projects in parallel, with different
roles. For example, I work on community project and publish packages on
behalf of it, and I publish personal packages too. Obviously, I want to
have 2 separate PyPI publishing accounts for those roles. Also, I don't
want to cleanup after dumb mistakes, so want to explicitly specify an
identity to use for each publishing operation, and get an error if I
don't.

It took me some trial and error to arrive at following .pypirc:

======
[distutils]
index-servers =
    use--repository-switch!
    pfalcon
    micropython-lib

[micropython-lib]
reposytory: http://www.python.org/pypi
username:...
password:...

[pfalcon]
reposytory: http://www.python.org/pypi
username:...
password:...

[use--repository-switch!]
username:foo
password:foo
======

However, using standard 3-fold idiom of "python setup.py sdist register
upload" (suffixed with "-r pfalcon"), it didn't work, throwing random
"Server response (401): basic auth failed". It took me even more trial
and error to figure out that I needed to specify -r after each of
register & upload:

python setup.py sdist register -r pfalcon upload -r pfalcon


So, the questions are: Why so complicated? Am I doing something wrong
and there's easier way to achieve described setup?
(https://docs.python.org/2/distutils/packageindex.html#the-pypirc-file
describes a case of 2 different servers, and that's what I started with,
but it took bunch of attempts to add 3rd "captive" account, and then
make it all work without throwing random errors like "AssertionError:
unsupported schema").


Thanks,
 Paul                          mailto:pmiscml at gmail.com

From noah at coderanger.net  Sun Jun  1 21:10:01 2014
From: noah at coderanger.net (Noah Kantrowitz)
Date: Sun, 1 Jun 2014 12:10:01 -0700
Subject: [Distutils] Support for multiple PyPI publishing identities is
	rather convoluted
In-Reply-To: <20140601180218.7e94e300@x34f>
References: <20140601180218.7e94e300@x34f>
Message-ID: <0AA89EF0-F326-4235-B0D8-2CB9DAD324FF@coderanger.net>


On Jun 1, 2014, at 8:02 AM, Paul Sokolovsky <pmiscml at gmail.com> wrote:

> Hello,
> 
> My usecase is: I work on different projects in parallel, with different
> roles. For example, I work on community project and publish packages on
> behalf of it, and I publish personal packages too. Obviously, I want to
> have 2 separate PyPI publishing accounts for those roles. Also, I don't
> want to cleanup after dumb mistakes, so want to explicitly specify an
> identity to use for each publishing operation, and get an error if I
> don't.

PyPI has an ACL system to make this unnecessary. You can use a single account, and for the community project just grant multiple people access.

--Noah

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140601/21f4507e/attachment.sig>

From noah at coderanger.net  Sun Jun  1 21:38:55 2014
From: noah at coderanger.net (Noah Kantrowitz)
Date: Sun, 1 Jun 2014 12:38:55 -0700
Subject: [Distutils] Support for multiple PyPI publishing identities is
	rather convoluted
In-Reply-To: <20140601223038.6b73d9be@x34f>
References: <20140601180218.7e94e300@x34f>
 <0AA89EF0-F326-4235-B0D8-2CB9DAD324FF@coderanger.net>
 <20140601223038.6b73d9be@x34f>
Message-ID: <69B562A3-23DE-4EB2-B6E7-8FEC4FBF9922@coderanger.net>


On Jun 1, 2014, at 12:30 PM, Paul Sokolovsky <pmiscml at gmail.com> wrote:

> Hello,
> 
> On Sun, 1 Jun 2014 12:10:01 -0700
> Noah Kantrowitz <noah at coderanger.net> wrote:
> 
>> 
>> On Jun 1, 2014, at 8:02 AM, Paul Sokolovsky <pmiscml at gmail.com> wrote:
>> 
>>> Hello,
>>> 
>>> My usecase is: I work on different projects in parallel, with
>>> different roles. For example, I work on community project and
>>> publish packages on behalf of it, and I publish personal packages
>>> too. Obviously, I want to have 2 separate PyPI publishing accounts
>>> for those roles. Also, I don't want to cleanup after dumb mistakes,
>>> so want to explicitly specify an identity to use for each
>>> publishing operation, and get an error if I don't.
>> 
>> PyPI has an ACL system to make this unnecessary. You can use a
>> single account, and for the community project just grant multiple
>> people access.
> 
> Unnecessary what exactly? On my packages' PyPI pages, I want to have 
> "Package Index Owner: pfalcon", and on other packages' pages, I don't
> want to have "pfalcon" (and want to have another specific username).
> Having it otherwise would be misrepresentation of package origin.
> 
> If "single account" can do that (that would be a surprise), I'd
> appreciate a link to materials I can read up on the matter.

If you didn't want to show up as the owner you would need to use the other account once to register it, but after that just grant your normal user access and use that for day-to-day releases.

--Noah

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140601/4e6097d0/attachment.sig>

From pmiscml at gmail.com  Sun Jun  1 21:30:38 2014
From: pmiscml at gmail.com (Paul Sokolovsky)
Date: Sun, 1 Jun 2014 22:30:38 +0300
Subject: [Distutils] Support for multiple PyPI publishing identities is
 rather convoluted
In-Reply-To: <0AA89EF0-F326-4235-B0D8-2CB9DAD324FF@coderanger.net>
References: <20140601180218.7e94e300@x34f>
 <0AA89EF0-F326-4235-B0D8-2CB9DAD324FF@coderanger.net>
Message-ID: <20140601223038.6b73d9be@x34f>

Hello,

On Sun, 1 Jun 2014 12:10:01 -0700
Noah Kantrowitz <noah at coderanger.net> wrote:

> 
> On Jun 1, 2014, at 8:02 AM, Paul Sokolovsky <pmiscml at gmail.com> wrote:
> 
> > Hello,
> > 
> > My usecase is: I work on different projects in parallel, with
> > different roles. For example, I work on community project and
> > publish packages on behalf of it, and I publish personal packages
> > too. Obviously, I want to have 2 separate PyPI publishing accounts
> > for those roles. Also, I don't want to cleanup after dumb mistakes,
> > so want to explicitly specify an identity to use for each
> > publishing operation, and get an error if I don't.
> 
> PyPI has an ACL system to make this unnecessary. You can use a
> single account, and for the community project just grant multiple
> people access.

Unnecessary what exactly? On my packages' PyPI pages, I want to have 
"Package Index Owner: pfalcon", and on other packages' pages, I don't
want to have "pfalcon" (and want to have another specific username).
Having it otherwise would be misrepresentation of package origin.

If "single account" can do that (that would be a surprise), I'd
appreciate a link to materials I can read up on the matter.





-- 
Best regards,
 Paul                          mailto:pmiscml at gmail.com

From pronghornparrot at yahoo.com  Mon Jun  2 23:37:53 2014
From: pronghornparrot at yahoo.com (John Smith)
Date: Mon, 2 Jun 2014 14:37:53 -0700 (PDT)
Subject: [Distutils] pypi suggestion
Message-ID: <1401745073.61866.YahooMailNeo@web161504.mail.bf1.yahoo.com>

pypi really needs a way to sort packages by popularity.? Sorting by other factors, such as author, code size, pure python/compiled, etc would be a bonus,? But something to put most "important"/popular, etc. packages first would be great.
?
I apologize if it's there and I just don't see it :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140602/91c85cc3/attachment.html>

From wesleyh93 at gmail.com  Tue Jun  3 00:03:06 2014
From: wesleyh93 at gmail.com (Wesley Huang)
Date: Mon, 2 Jun 2014 18:03:06 -0400
Subject: [Distutils] Installing Statsmodel
Message-ID: <CAMEWEwFU_RWhtp0RxcqNLjJLxTr4PGtZzJaJuFcumP_6+Mx=Mw@mail.gmail.com>

Hi there,

I am using a Mac and trying to install the package called statsmodel. Based
on available downloads, it seems i can only get it if i have a PC? Or if
not true, can you link me to where I'd be able to download it?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140602/57f83ff4/attachment.html>

From noah at coderanger.net  Tue Jun  3 02:49:33 2014
From: noah at coderanger.net (Noah Kantrowitz)
Date: Mon, 2 Jun 2014 17:49:33 -0700
Subject: [Distutils] pypi suggestion
In-Reply-To: <1401745073.61866.YahooMailNeo@web161504.mail.bf1.yahoo.com>
References: <1401745073.61866.YahooMailNeo@web161504.mail.bf1.yahoo.com>
Message-ID: <3B1637F7-7627-4162-AFB4-D57804AA5B12@coderanger.net>

Step one, define "popular" in numeric terms.

--Noah

On Jun 2, 2014, at 2:37 PM, John Smith <pronghornparrot at yahoo.com.dmarc.invalid> wrote:

> pypi really needs a way to sort packages by popularity.  Sorting by other factors, such as author, code size, pure python/compiled, etc would be a bonus,  But something to put most "important"/popular, etc. packages first would be great.
>  
> I apologize if it's there and I just don't see it :)
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140602/f713ff0d/attachment.sig>

From ncoghlan at gmail.com  Tue Jun  3 04:19:22 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 3 Jun 2014 12:19:22 +1000
Subject: [Distutils] pypi suggestion
In-Reply-To: <3B1637F7-7627-4162-AFB4-D57804AA5B12@coderanger.net>
References: <1401745073.61866.YahooMailNeo@web161504.mail.bf1.yahoo.com>
 <3B1637F7-7627-4162-AFB4-D57804AA5B12@coderanger.net>
Message-ID: <CADiSq7dkS=VY_6=StPyy_ov+x+XrzT9K7uQDWZr73=2QneRa1A@mail.gmail.com>

On 3 June 2014 10:49, Noah Kantrowitz <noah at coderanger.net> wrote:
> Step one, define "popular" in numeric terms.

"Most depended on", but that's currently a pain to extract. Making it
easier to do that kind of analysis is actually one of my goals for
metadata 2.0.

Sites like http://pythonwheels.com/ use "most downloaded" because it's
readily available, rather than because it's a particularly good metric
- that's part of the reason why PyPI doesn't offer sorting by download
count natively (the other part is because the current focus of PyPI
development is the migration from the current legacy code base to
Warehouse).

One key issue with "most downloaded" is the fact that it overweights
direct downloads from PyPI by automated testing and deployment
systems. Anyone using a local caching proxy (which is the recommended
approach), relying on a Python redistributor (like a Linux distro, or
folks like ActiveState/Enthought/Continuum) or otherwise using third
party Windows installers doesn't show up in the download stats.

Cheers,
Nick.

P.S. For folks using Django, grids on Django Packages (e.g.
https://www.djangopackages.com/grids/g/rest/) can be a much better way
to find packages to use than looking directly on PyPI. I believe there
are OpenComparison based sites for some other Python subcommunities as
well.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From matt at notevencode.com  Tue Jun  3 07:51:34 2014
From: matt at notevencode.com (Matthew Iversen)
Date: Tue, 03 Jun 2014 15:51:34 +1000
Subject: [Distutils] Installing Statsmodel
In-Reply-To: <CAMEWEwFU_RWhtp0RxcqNLjJLxTr4PGtZzJaJuFcumP_6+Mx=Mw@mail.gmail.com>
References: <CAMEWEwFU_RWhtp0RxcqNLjJLxTr4PGtZzJaJuFcumP_6+Mx=Mw@mail.gmail.com>
Message-ID: <538D6266.7030501@notevencode.com>

On 03/06/14 08:03, Wesley Huang wrote:
> Hi there,
>
> I am using a Mac and trying to install the package called statsmodel.
> Based on available downloads, it seems i can only get it if i have a
> PC? Or if not true, can you link me to where I'd be able to download it?
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
This is a binary package that depends on numpy, scipy, pandas and
cython, at the least.

These packages are also binary packages, that are best compiled for you
by someone who knows how to make efficient binaries of them.

The easiest way to get these packages probably begins by looking at

http://www.scipy.org/install.html
http://www.scipy.org/scipylib/download.html

-- 
Matt Iversen // matt at notevencode.com
PGP: 0xc046e8a874522973 // 2F04 3DCC D6E6 D5AC D262  2E0B C046 E8A8 7452 2973

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140603/3081f5ae/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140603/3081f5ae/attachment.sig>

From vinay_sajip at yahoo.co.uk  Tue Jun  3 17:24:39 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Tue, 3 Jun 2014 16:24:39 +0100 (BST)
Subject: [Distutils] pypi suggestion
In-Reply-To: <CADiSq7dkS=VY_6=StPyy_ov+x+XrzT9K7uQDWZr73=2QneRa1A@mail.gmail.com>
References: <1401745073.61866.YahooMailNeo@web161504.mail.bf1.yahoo.com>
 <3B1637F7-7627-4162-AFB4-D57804AA5B12@coderanger.net>
 <CADiSq7dkS=VY_6=StPyy_ov+x+XrzT9K7uQDWZr73=2QneRa1A@mail.gmail.com>
Message-ID: <1401809079.40617.YahooMailNeo@web172401.mail.ir2.yahoo.com>

----- Original Message -----

From: Nick Coghlan <ncoghlan at gmail.com>


> "Most depended on", but that's currently a pain to extract. Making it
> easier to do that kind of analysis is actually one of my goals for
> metadata 2.0.




Here's my list of the top 25, based on the PyPI metadata that I maintain (not rigorously tested, but seems plausible):

6265:setuptools 
2371:django 
2178:requests 
1007:zope.interface 
1003:numpy 
873:pyyaml 
860:lxml 
784:simplejson 
758:zope.component 
734:six 
679:flask 
678:nose 
677:argparse 
633:sqlalchemy 
569:jinja2 
551:python-dateutil 
471:zope.schema 
456:sphinx 
433:pytz 
427:mock 
414:scipy 
405:distribute 
396:zc.buildout 
364:docopt 
363:pil 


The number against each dependency represents the number of distinct PyPI projects which list the dependency as a run-time requirement (:run: or :meta:).


It's possible that some projects declare things as run-time dependencies when they really mean build-time: I'm not sure that there are really over 400 projects which are Sphinx plugins / extensions.

Just throwing this out there as it may be of interest.

Regards,

Vinay Sajip

From p.f.moore at gmail.com  Tue Jun  3 20:35:55 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 3 Jun 2014 19:35:55 +0100
Subject: [Distutils] pypi suggestion
In-Reply-To: <1401809079.40617.YahooMailNeo@web172401.mail.ir2.yahoo.com>
References: <1401745073.61866.YahooMailNeo@web161504.mail.bf1.yahoo.com>
 <3B1637F7-7627-4162-AFB4-D57804AA5B12@coderanger.net>
 <CADiSq7dkS=VY_6=StPyy_ov+x+XrzT9K7uQDWZr73=2QneRa1A@mail.gmail.com>
 <1401809079.40617.YahooMailNeo@web172401.mail.ir2.yahoo.com>
Message-ID: <CACac1F-ZX8dmysqj690fnDo5eyMGWGUsysObK6sVG8Ki1x+K2A@mail.gmail.com>

On 3 June 2014 16:24, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>
> It's possible that some projects declare things as run-time dependencies when they really mean build-time: I'm not sure that there are really over 400 projects which are Sphinx plugins / extensions.

zc.buildout seems like an unusual runtime dependency, but as you say
that's more likely incorrect metadata than anything else (and even as
a build-time dependency it's a measure of zc.buildout's popularity)

Also, how does this measure optional dependencies? Something like
html5lib can use lxml if it's present, but works without it. (That's
of interest because lxml is a pain to build and would benefit from
wheels, so anything that depends on it would also benefit if lxml
wheels were available).

> Just throwing this out there as it may be of interest.

Definitely. Interesting (not surprising, though) that pyyaml is so
high, and that docopt is on there. Also notable that pil is in there.
(Does your data distinguish pil and pillow? I understood the common
view these days was to use pillow in place of pil).

Paul

From vinay_sajip at yahoo.co.uk  Tue Jun  3 23:39:28 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Tue, 3 Jun 2014 22:39:28 +0100 (BST)
Subject: [Distutils] pypi suggestion
In-Reply-To: <CACac1F-ZX8dmysqj690fnDo5eyMGWGUsysObK6sVG8Ki1x+K2A@mail.gmail.com>
References: <1401745073.61866.YahooMailNeo@web161504.mail.bf1.yahoo.com>	<3B1637F7-7627-4162-AFB4-D57804AA5B12@coderanger.net>	<CADiSq7dkS=VY_6=StPyy_ov+x+XrzT9K7uQDWZr73=2QneRa1A@mail.gmail.com>	<1401809079.40617.YahooMailNeo@web172401.mail.ir2.yahoo.com>
 <CACac1F-ZX8dmysqj690fnDo5eyMGWGUsysObK6sVG8Ki1x+K2A@mail.gmail.com>
Message-ID: <1401831568.82355.YahooMailNeo@web172402.mail.ir2.yahoo.com>



----- Original Message -----

> Also, how does this measure optional dependencies?

I capture extras information where provided, but it all depends on what's declared in the metadata (via install_requires, setup_requires, test_requires). For example, html5lib doesn't declare any dependencies other than six, so that's all my code would know about.


> Interesting (not surprising, though) that pyyaml is so
> high, and that docopt is on there. Also notable that pil is in there.
> (Does your data distinguish pil and pillow? I understood the common
> view these days was to use pillow in place of pil).




The data hasn't been thoroughly validated - PyPI is a vast repository of information. While I've checked the sanity of my metadata for projects that I work on / use, it's too large a body of data to check everything, and it's certainly possible that there are a few bugs lurking in my metadata "scraping" code.

The code doesn't do anything clever like aliasing PIL/Pillow or distribute/setuptools - it's the bare dependencies as declared in zillions of setup.py files.

Regards,

Vinay Sajip

From reinout at vanrees.org  Thu Jun  5 10:30:41 2014
From: reinout at vanrees.org (Reinout van Rees)
Date: Thu, 05 Jun 2014 10:30:41 +0200
Subject: [Distutils] Packaging buildout-based project for release
In-Reply-To: <CAPDm-FjVkAZBwDGVQi+SQ=4KD+NPwLdue8rk_g-3gRz-Aipv+Q@mail.gmail.com>
References: <538623CB.1050504@oracle.com>
 <CAPDm-FjVkAZBwDGVQi+SQ=4KD+NPwLdue8rk_g-3gRz-Aipv+Q@mail.gmail.com>
Message-ID: <lmp9ri$20l$1@ger.gmane.org>

On 29-05-14 00:03, Jim Fulton wrote:
> I wrote zc.sourcerelease to help solve this problem.

Yes, that's a good option for what Travis asked. I've only used it two 
or three times, but it Just Works.

If afterwards you need to turn it (or something else) into a package, 
I've found FPM (https://github.com/jordansissel/fpm) to be real handy. 
Whether you need .rpm or .deb, it makes it easy to quickly create a 
proper package without necessarily building up the whole machinery 
that's otherwise needed.


Reinout

-- 
Reinout van Rees                    http://reinout.vanrees.org/
reinout at vanrees.org             http://www.nelen-schuurmans.nl/
"If you're not sure what to do, make something. -- Paul Graham"


From donald at stufft.io  Fri Jun  6 04:08:44 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 5 Jun 2014 22:08:44 -0400
Subject: [Distutils] PEP 470 Round 2 - Using Multi Index Support for
	External to PyPI Package File Hosting
Message-ID: <0D7BEAFC-7987-46C8-8A9A-7931BE4DA36B@stufft.io>

Here's round 2 of PEP 470.

You can see it online at https://python.org/dev/peps/pep-0470/ or below.

Notable changes:

- Ensure it's obvious this strictly deals with the installer API and does not
  affect a project's ability to register their project on PyPI for human
  consumptions.

- Mention that the functional mechanisms that make it possible for an end user
  to specify the additional locations have existed for a long time across many
  versions of the installers.

- Explicitly mention that the installer changes from PEP 438 should be
  deprecated and removed as part of this PEP.

- Explicitly mention pythonhosted.org as a location that authors can use to
  host an index if they do not wish to purchase a TLS certificate or host
  additional infrastructure.

- Include that a link to PyPI ToS should be included in the emails sent to
  authors to remind them of the PyPI ToS.

- Special case PIL as it is an outlier in terms of impact.

- Fill out the impact sections further to provide more detail


Abstract
========

This PEP proposes that the official means of having an installer locate and
find package files which are hosted externally to PyPI become the use of
multi index support instead of the practice of using external links on the
simple installer API.

It is important to remember that this is **not** about forcing anyone to host
their files on PyPI. If someone does not wish to do so they will never be under
any obligation too. They can still list their project in PyPI as an index, and
the tooling will still allow them to host it elsewhere.

This PEP strictly is concerned with the Simple Installer API and how automated
installers interact with PyPI, it has no bearing on the informational pages
which are primarily for human consumption.


Rationale
=========

There is a long history documented in PEP 438 that explains why externally
hosted files exist today in the state that they do on PyPI. For the sake of
brevity I will not duplicate that and instead urge readers to first take a look
at PEP 438 for background.

There are currently two primary ways for a project to make itself available
without directly hosting the package files on PyPI. They can either include
links to the package files in the simpler installer API or they can publish
a custom package index which contains their project.


Custom Additional Index
-----------------------

Each installer which speaks to PyPI offers a mechanism for the user invoking
that installer to provide additional custom locations to search for files
during the dependency resolution phase. For pip these locations can be
configured per invocation, per shell environment, per requirements file, per
virtual environment, and per user. The mechanism for specifying additional
locations have existed within pip and setuptools for many years, by comparison
the mechanisms in PEP 438 and any other new mechanism will have existed for
only a short period of time (if they exist at all currently).

The use of additional indexes instead of external links on the simple
installer API provides a simple clean interface which is consistent with the
way most Linux package systems work (apt-get, yum, etc). More importantly it
works the same even for projects which are commercial or otherwise have their
access restricted in some form (private networks, password, IP ACLs etc)
while the external links method only realistically works for projects which
do not have their access restricted.

Compared to the complex rules which a project must be aware of to prevent
themselves from being considered unsafely hosted setting up an index is fairly
trivial and in the simplest case does not require anything more than a
filesystem and a standard web server such as Nginx or Twisted Web. Even if
using simple static hosting without autoindexing support, it is still
straightforward to generate appropriate index pages as static HTML.

Example Index with Twisted Web
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1. Create a root directory for your index, for the purposes of the example
   I'll assume you've chosen ``/var/www/index.example.com/``.
2. Inside of this root directory, create a directory for each project such
   as ``mkdir -p /var/www/index.example.com/{foo,bar,other}/``.
3. Place the package files for each project in their respective folder,
   creating paths like ``/var/www/index.example.com/foo/foo-1.0.tar.gz``.
4. Configure Twisted Web to serve the root directory, ideally with TLS.

::

    $ twistd -n web --path /var/www/index.example.com/


Examples of Additional indexes with pip
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Invocation:**

::

    $ pip install --extra-index-url https://pypi.example.com/ foobar

**Shell Environment:**

::

    $ export PIP_EXTRA_INDEX_URL=https://pypi.example.com/
    $ pip install foobar

**Requirements File:**

::

    $ echo "--extra-index-url https://pypi.example.com/\nfoobar" > requirements.txt
    $ pip install -r requirements.txt

**Virtual Environment:**

::

    $ python -m venv myvenv
    $ echo "[global]\nextra-index-url = https://pypi.example.com/" > myvenv/pip.conf
    $ myvenv/bin/pip install foobar

**User:**

::

    $ echo "[global]\nextra-index-url = https://pypi.example.com/" >~/.pip/pip.conf
    $ pip install foobar


External Links on the Simple Installer API
------------------------------------------

PEP 438 proposed a system of classifying file links as either internal,
external, or unsafe. It recommended that by default only internal links would
be installed by an installer however users could opt into external links on
either a global or a per package basis. Additionally they could also opt into
unsafe links on a per package basis.

This system has turned out to be *extremely* unfriendly towards the end users
and it is the position of this PEP that the situation has become untenable. The
situation as provided by PEP 438 requires an end user to be aware not only of
the difference between internal, external, and unsafe, but also to be aware of
what hosting mode the package they are trying to install is in, what links are
available on that project's /simple/ page, whether or not those links have
a properly formatted hash fragment, and what links are available from pages
linked to from that project's /simple/ page.

There are a number of common confusion/pain points with this system that I
have witnessed:

* Users unaware what the simple installer api is at all or how an installer
  locates installable files.
* Users unaware that even if the simple api links to a file, if it does
  not include a ``#md5=...`` fragment that it will be counted as unsafe.
* Users unaware that an installer can look at pages linked from the
  simple api to determine additional links, or that any links found in this
  fashion are considered unsafe.
* Users are unaware and often surprised that PyPI supports hosting your files
  someplace other than PyPI at all.

In addition to that, the information that an installer is able to provide
when an installation fails is pretty minimal. We are able to detect if there
are externally hosted files directly linked from the simple installer api,
however we cannot detect if there are files hosted on a linked page without
fetching that page and doing so would cause a massive performance hit just to
see if there might be a file there so that a better error message could be
provided.

Finally very few projects have properly linked to their external files so that
they can be safely downloaded and verified. At the time of this writing there
are a total of 65 projects which have files that are only available externally
and are safely hosted.

The end result of all of this, is that with PEP 438, when a user attempts to
install a file that is not hosted on PyPI typically the steps they follow are:

1. First, they attempt to install it normally, using ``pip install foobar``.
   This fails because the file is not hosted on PyPI and PEP 438 has us default
   to only hosted on PyPI. If pip detected any externally hosted files or other
   pages that we *could* have attempted to find other files at it will give an
   error message suggesting that they try ``--allow-external foobar``.
2. They then attempt to install their package using
   ``pip install --allow-external foobar foobar``. If they are lucky foobar is
   one of the packages which is hosted externally and safely and this will
   succeed. If they are unlucky they will get a different error message
   suggesting that they *also* try ``--allow-unverified foobar``.
3. They then attempt to install their package using
   ``pip install --allow-external foobar --allow-unverified foobar foobar``
   and this finally works.

This is the same basic steps that practically everyone goes through every time
they try to install something that is not hosted on PyPI. If they are lucky it'll
only take them two steps, but typically it requires three steps. Worse there is
no real indication to these people why one package might install after two
but most require three. Even worse than that most of them will never get an
externally hosted package that does not take three steps, so they will be
increasingly annoyed and frustrated at the intermediate step and will likely
eventually just start skipping it.


External Index Discovery
========================

One of the problems with using an additional index is one of discovery. Users
will not generally be aware that an additional index is required at all much
less where that index can be found. Projects can attempt to convey this
information using their description on the PyPI page however that excludes
people who discover their project organically through ``pip search``.

To support projects that wish to externally host their files and to enable
users to easily discover what additional indexes are required, PyPI will gain
the ability for projects to register external index URLs and additionally an
associated comment for each. These URLs will be made available on the simple
page however they will not be linked or provided in a form that older
installers will automatically search them.

When an installer fetches the simple page for a project, if it finds this
additional meta-data and it cannot find any files for that project in it's
configured URLs then it should use this data to tell the user how to add one
or more of the additional URLs to search in. This message should include any
comments that the project has included to enable them to communicate to the
user and provide hints as to which URL they might want if some are only
useful or compatible with certain platforms or situations. When the installer
has implemented the auto discovery mechanisms they should also deprecate any
of the mechanisms added for PEP 438 (such as ``--allow-external``) for removal
at the end of the deprecation period proposed by the PEP.

This feature *must* be added to PyPI prior to starting the deprecation and
removal process for link spidering.


Deprecation and Removal of Link Spidering
=========================================

A new hosting mode will be added to PyPI. This hosting mode will be called
``pypi-only`` and will be in addition to the three that PEP 438 has already
given us which are ``pypi-explicit``, ``pypi-scrape``, ``pypi-scrape-crawl``.
This new hosting mode will modify a project's simple api page so that it only
lists the files which are directly hosted on PyPI and will not link to anything
else.

Upon acceptance of this PEP and the addition of the ``pypi-only`` mode, all new
projects will by defaulted to the PyPI only mode and they will be locked to
this mode and unable to change this particular setting. ``pypi-only`` projects
will still be able to register external index URLs as described above - the
"pypi-only" refers only to the download links that are published directly on
PyPI.

An email will then be sent out to all of the projects which are hosted only on
PyPI informing them that in one month their project will be automatically
converted to the ``pypi-only`` mode. A month after these emails have been sent
any of those projects which were emailed, which still are hosted only on PyPI
will have their mode set to ``pypi-only``.

After that switch, an email will be sent to projects which rely on hosting
external to PyPI. This email will warn these projects that externally hosted
files have been deprecated on PyPI and that in 6 months from the time of that
email that all external links will be removed from the installer APIs. This
email *must* include instructions for converting their projects to be hosted
on PyPI and *must* include links to a script or package that will enable them
to enter their PyPI credentials and package name and have it automatically
download and re-host all of their files on PyPI. This email *must also*
include instructions for setting up their own index page and registering that
with PyPI, including the fact that they can use pythonhosted.org as a host
for an index page without requiring them to host any additional infrastructure
or purchase a TLS certificate. This email must also contain a link to the Terms
of Service for PyPI as many users may have signed up a long time ago and may
not recall what those terms are.

Five months after the initial email, another email must be sent to any projects
still relying on external hosting. This email will include all of the same
information that the first email contained, except that the removal date will
be one month away instead of six.

Finally a month later all projects will be switched to the ``pypi-only`` mode
and PyPI will be modified to remove the externally linked files functionality.
At this point in time any installers should finally remove any of the
deprecated PEP 438 functionality such as ``--allow-external`` and
``--allow-unverified`` in pip.


PIL
---

It's obvious from the numbers below that the vast bulk of the impact come from
the PIL project. On 2014-05-17 an email was sent to the contact for PIL
inquiring whether or not they would be willing to upload to PyPI. A response
has not been received as of yet (2014-06-05) nor has any change in the hosting
happened. Due to the popularity of PIL this PEP also proposes that during the
deprecation period that PyPI Administrators will set the PIL download URL as
the external index for that project. Allowing the users of PIL to take
advantage of the auto discovery mechanisms although the project has seemingly
become unmaintained.


Impact
======

The largest impact of this is going to be projects where the maintainers are
no longer maintaining the project, for one reason or another. For these
projects it's unlikely that a maintainer will arrive to set the external index
metadata which would allow the auto discovery mechanism to find it.

Looking at the numbers factoring out PIL (which has been special cased above)
the actual impact should be quite low, with it affecting just 6.9% of projects
which host only externally or 2.8% which have their latest version hosted
externally. This represents a mere 3883 unique IP addresses. The break down of
this is that of those 3883 addresses, 100% of them installed something that
could not be verified while only 3% installed something which could be.


Projects Which Rely on Externally Hosted files
----------------------------------------------

This is determined by crawling the simple index and looking for installable
files using a similar detection method as pip and setuptools use. The "latest"
version is determined using ``pkg_resources.parse_version`` sort order and it
is used to show whether or not the latest version is hosted externally or only
old versions are.

============ ======= ================ =================== =======
\             PyPI    External (old)   External (latest)   Total
============ ======= ================ =================== =======
 **Safe**     38716   31               35                  38782
 **Unsafe**   0       1659             1169                2828
 **Total**    38716   1690             1204                41610
============ ======= ================ =================== =======


Top Externally Hosted Projects by Requests
------------------------------------------

This is determined by looking at the number of requests the
``/simple/<project>/`` page had gotten in a single day. The total number of
requests during that day was 17,960,467.

============================== ========
Project                        Requests
============================== ========
PIL                            13470
mysql-connector-python         321
salesforce-python-toolkit      54
pyodbc                         50
elementtree                    44
atfork                         39
RBTools                        29
django-contrib-requestprovider 28
wadofstuff-django-serializers  23
Pygame                         21
============================== ========


Top Externally Hosted Projects by Unique IPs
--------------------------------------------

This is determined by looking at the IP addresses of requests the
``/simple/<project>/`` page had gotten in a single day. The total number of
unique IP addresses during that day was 105,587.

============================== ==========
Project                        Unique IPs
============================== ==========
PIL                            3515
mysql-connector-python         117
pyodbc                         34
elementtree                    21
RBTools                        19
egenix-mx-base                 16
Pygame                         14
salesforce-python-toolkit      13
django-contrib-requestprovider 12
wxPython                       11
python-apt                     10
============================== ==========


Rejected Proposals
==================

Keep the current classification system but adjust the options
-------------------------------------------------------------

This PEP rejects several related proposals which attempt to fix some of the
usability problems with the current system but while still keeping the
general gist of PEP 438.

This includes:

* Default to allowing safely externally hosted files, but disallow unsafely
  hosted.
* Default to disallowing safely externally hosted files with only a global
  flag to enable them, but disallow unsafely hosted.

These proposals are rejected because:

* The classification "system" is complex, hard to explain, and requires an
  intimate knowledge of how the simple API works in order to be able to reason
  about which classification is required. This is reflected in the fact that
  the code to implement it is complicated and hard to understand as well.

* People are generally surprised that PyPI allows externally linking to files
  and doesn't require people to host on PyPI. In contrast most of them are
  familiar with the concept of multiple software repositories such as is in
  use by many OSs.

* PyPI is fronted by a globally distributed CDN which has improved the
  reliability and speed for end users. It is unlikely that any particular
  external host has something comparable. This can lead to extremely bad
  performance for end users when the external host is located in different
  parts of the world or does not generally have good connectivity.

  As a data point, many users reported sub DSL speeds and latency when
  accessing PyPI from parts of Europe and Asia prior to the use of the CDN.

* PyPI has monitoring and an on-call rotation of sysadmins whom can respond to
  downtime quickly, thus enabling a quicker response to downtime. Again it is
  unlikely that any particular external host will have this. This can lead
  to single packages in a dependency chain being un-installable. This will
  often confuse users, who often times have no idea that this package relies
  on an external host, and they cannot figure out why PyPI appears to be up
  but the installer cannot find a package.

* PyPI supports mirroring, both for private organizations and public mirrors.
  The legal terms of uploading to PyPI ensure that mirror operators, both
  public and private, have the right to distribute the software found on PyPI.
  However software that is hosted externally does not have this, causing
  private organizations to need to investigate each package individually and
  manually to determine if the license allows them to mirror it.

  For public mirrors this essentially means that these externally hosted
  packages *cannot* be reasonably mirrored. This is particularly troublesome
  in countries such as China where the bandwidth to outside of China is
  highly congested making a mirror within China often times a massively better
  experience.

* Installers have no method to determine if they should expect any particular
  URL to be available or not. It is not unusual for the simple API to reference
  old packages and URLs which have long since stopped working. This causes
  installers to have to assume that it is OK for any particular URL to not be
  accessible. This causes problems where an URL is temporarily down or
  otherwise unavailable (a common cause of this is using a copy of Python
  linked against a really ancient copy of OpenSSL which is unable to verify
  the SSL certificate on PyPI) but it *should* be expected to be up. In this
  case installers will typically silently ignore this URL and later the user
  will get a confusing error stating that the installer couldn't find any
  versions instead of getting the real error message indicating that the URL
  was unavailable.

* In the long run, global opt in flags like ``--allow-all-external`` will
  become little annoyances that developers cargo cult around in order to make
  their installer work. When they run into a project that requires it they
  will most likely simply add it to their configuration file for that installer
  and continue on with whatever they were actually trying to do. This will
  continue until they try to install their requirements on another computer
  or attempt to deploy to a server where their install will fail again until
  they add the "make it work" flag in their configuration file.


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140605/f70c6e87/attachment-0001.sig>

From holger at merlinux.eu  Fri Jun  6 10:13:48 2014
From: holger at merlinux.eu (holger krekel)
Date: Fri, 6 Jun 2014 08:13:48 +0000
Subject: [Distutils] PEP 470 Round 2 - Using Multi Index Support for
 External to PyPI Package File Hosting
In-Reply-To: <0D7BEAFC-7987-46C8-8A9A-7931BE4DA36B@stufft.io>
References: <0D7BEAFC-7987-46C8-8A9A-7931BE4DA36B@stufft.io>
Message-ID: <20140606081348.GG30349@merlinux.eu>

Hi Donald,

1. you published numbers where <4K or <300 discounting PIL would be
   affected by PEP470.  You also say that the main reason for deprecating
   PEP438 is that it confused users.  Did it confuse other users than those few?

2. I don't see a valid precise reasoning why PEP438, just agreed on and 
   implemented last year, needs deprecation.  It boosted everyone
   everyone's install experiences (independently from the CDN which
   brought another boost) as usage of crawling dramatically dropped 
   and thus brings us into the exact situation PEP438 already hinted at:

   "Deprecation of hosting modes to eventually only allow the
   pypi-explicit mode is NOT REGULATED by this PEP but is expected to
   become feasible some time after successful implementation of the
   transition phases described in this PEP. It is expected that
   deprecation requires a new process to deal with abandoned packages
   because of unreachable maintainers for still popular packages."

   We should follow through and discuss removing crawling and 
   how to deal with abandoned packages.  On the PyPI side, what 
   would remain are two kind of links:

   - pypi internally hosted
   - registered safe external links to release files

   The resulting situation is:

   easy: users have an already existing option to consider to allow externals.
   
   safe: All links served from pypi have checksums. Project maintainers need
         to register hashed links to their new release files.
         
   clean: Pip could eventually remove support for crawling/related options.
   
   This is all easy to do, reduces user confusion and makes pip
   and pypi simpler and less suprising.

   I don't see this approach discussed or seriously considered in the PEP,
   also not in its "rejection reasons".

   By contrast, PEP470 would require many users to learn about
   specifying other indexes and what that means.  For you and me
   and many here on the list it may be a no-brainer but trust me,
   for many users (i've done ten trainings touching the topic now)
   this is not a natural concept at all.  "pip install --allow-all-externals" 
   is far easier to convey than specifying extra per-project indexes and 
   what it means if the install fails (wrong URL? Index noch reachable?
   Release file not found?).

3. PEP470 makes life a lot harder for devpi-server, currently used
   by many companies for serving their private indexes.  With PEP438 and
   almost no external crawling left, devpi-server can rely on seeing
   changes through the PEP381 API.  By contrast, with projects hosted on
   additional per-project external indexes, it requires polling to see
   changes because releases may not be registered with PyPI anymore (and
   there is no way to enforce that IISIC).  IOW, PEP470 is a serious
   regression here as it doesn't allow getting notified on new releasefiles.

best,
holger

On Thu, Jun 05, 2014 at 22:08 -0400, Donald Stufft wrote:
> Here's round 2 of PEP 470.
> 
> You can see it online at https://python.org/dev/peps/pep-0470/ or below.
> 
> Notable changes:
> 
> - Ensure it's obvious this strictly deals with the installer API and does not
>   affect a project's ability to register their project on PyPI for human
>   consumptions.
> 
> - Mention that the functional mechanisms that make it possible for an end user
>   to specify the additional locations have existed for a long time across many
>   versions of the installers.
> 
> - Explicitly mention that the installer changes from PEP 438 should be
>   deprecated and removed as part of this PEP.
> 
> - Explicitly mention pythonhosted.org as a location that authors can use to
>   host an index if they do not wish to purchase a TLS certificate or host
>   additional infrastructure.
> 
> - Include that a link to PyPI ToS should be included in the emails sent to
>   authors to remind them of the PyPI ToS.
> 
> - Special case PIL as it is an outlier in terms of impact.
> 
> - Fill out the impact sections further to provide more detail
> 
> 
> Abstract
> ========
> 
> This PEP proposes that the official means of having an installer locate and
> find package files which are hosted externally to PyPI become the use of
> multi index support instead of the practice of using external links on the
> simple installer API.
> 
> It is important to remember that this is **not** about forcing anyone to host
> their files on PyPI. If someone does not wish to do so they will never be under
> any obligation too. They can still list their project in PyPI as an index, and
> the tooling will still allow them to host it elsewhere.
> 
> This PEP strictly is concerned with the Simple Installer API and how automated
> installers interact with PyPI, it has no bearing on the informational pages
> which are primarily for human consumption.
> 
> 
> Rationale
> =========
> 
> There is a long history documented in PEP 438 that explains why externally
> hosted files exist today in the state that they do on PyPI. For the sake of
> brevity I will not duplicate that and instead urge readers to first take a look
> at PEP 438 for background.
> 
> There are currently two primary ways for a project to make itself available
> without directly hosting the package files on PyPI. They can either include
> links to the package files in the simpler installer API or they can publish
> a custom package index which contains their project.
> 
> 
> Custom Additional Index
> -----------------------
> 
> Each installer which speaks to PyPI offers a mechanism for the user invoking
> that installer to provide additional custom locations to search for files
> during the dependency resolution phase. For pip these locations can be
> configured per invocation, per shell environment, per requirements file, per
> virtual environment, and per user. The mechanism for specifying additional
> locations have existed within pip and setuptools for many years, by comparison
> the mechanisms in PEP 438 and any other new mechanism will have existed for
> only a short period of time (if they exist at all currently).
> 
> The use of additional indexes instead of external links on the simple
> installer API provides a simple clean interface which is consistent with the
> way most Linux package systems work (apt-get, yum, etc). More importantly it
> works the same even for projects which are commercial or otherwise have their
> access restricted in some form (private networks, password, IP ACLs etc)
> while the external links method only realistically works for projects which
> do not have their access restricted.
> 
> Compared to the complex rules which a project must be aware of to prevent
> themselves from being considered unsafely hosted setting up an index is fairly
> trivial and in the simplest case does not require anything more than a
> filesystem and a standard web server such as Nginx or Twisted Web. Even if
> using simple static hosting without autoindexing support, it is still
> straightforward to generate appropriate index pages as static HTML.
> 
> Example Index with Twisted Web
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> 1. Create a root directory for your index, for the purposes of the example
>    I'll assume you've chosen ``/var/www/index.example.com/``.
> 2. Inside of this root directory, create a directory for each project such
>    as ``mkdir -p /var/www/index.example.com/{foo,bar,other}/``.
> 3. Place the package files for each project in their respective folder,
>    creating paths like ``/var/www/index.example.com/foo/foo-1.0.tar.gz``.
> 4. Configure Twisted Web to serve the root directory, ideally with TLS.
> 
> ::
> 
>     $ twistd -n web --path /var/www/index.example.com/
> 
> 
> Examples of Additional indexes with pip
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> **Invocation:**
> 
> ::
> 
>     $ pip install --extra-index-url https://pypi.example.com/ foobar
> 
> **Shell Environment:**
> 
> ::
> 
>     $ export PIP_EXTRA_INDEX_URL=https://pypi.example.com/
>     $ pip install foobar
> 
> **Requirements File:**
> 
> ::
> 
>     $ echo "--extra-index-url https://pypi.example.com/\nfoobar" > requirements.txt
>     $ pip install -r requirements.txt
> 
> **Virtual Environment:**
> 
> ::
> 
>     $ python -m venv myvenv
>     $ echo "[global]\nextra-index-url = https://pypi.example.com/" > myvenv/pip.conf
>     $ myvenv/bin/pip install foobar
> 
> **User:**
> 
> ::
> 
>     $ echo "[global]\nextra-index-url = https://pypi.example.com/" >~/.pip/pip.conf
>     $ pip install foobar
> 
> 
> External Links on the Simple Installer API
> ------------------------------------------
> 
> PEP 438 proposed a system of classifying file links as either internal,
> external, or unsafe. It recommended that by default only internal links would
> be installed by an installer however users could opt into external links on
> either a global or a per package basis. Additionally they could also opt into
> unsafe links on a per package basis.
> 
> This system has turned out to be *extremely* unfriendly towards the end users
> and it is the position of this PEP that the situation has become untenable. The
> situation as provided by PEP 438 requires an end user to be aware not only of
> the difference between internal, external, and unsafe, but also to be aware of
> what hosting mode the package they are trying to install is in, what links are
> available on that project's /simple/ page, whether or not those links have
> a properly formatted hash fragment, and what links are available from pages
> linked to from that project's /simple/ page.
> 
> There are a number of common confusion/pain points with this system that I
> have witnessed:
> 
> * Users unaware what the simple installer api is at all or how an installer
>   locates installable files.
> * Users unaware that even if the simple api links to a file, if it does
>   not include a ``#md5=...`` fragment that it will be counted as unsafe.
> * Users unaware that an installer can look at pages linked from the
>   simple api to determine additional links, or that any links found in this
>   fashion are considered unsafe.
> * Users are unaware and often surprised that PyPI supports hosting your files
>   someplace other than PyPI at all.
> 
> In addition to that, the information that an installer is able to provide
> when an installation fails is pretty minimal. We are able to detect if there
> are externally hosted files directly linked from the simple installer api,
> however we cannot detect if there are files hosted on a linked page without
> fetching that page and doing so would cause a massive performance hit just to
> see if there might be a file there so that a better error message could be
> provided.
> 
> Finally very few projects have properly linked to their external files so that
> they can be safely downloaded and verified. At the time of this writing there
> are a total of 65 projects which have files that are only available externally
> and are safely hosted.
> 
> The end result of all of this, is that with PEP 438, when a user attempts to
> install a file that is not hosted on PyPI typically the steps they follow are:
> 
> 1. First, they attempt to install it normally, using ``pip install foobar``.
>    This fails because the file is not hosted on PyPI and PEP 438 has us default
>    to only hosted on PyPI. If pip detected any externally hosted files or other
>    pages that we *could* have attempted to find other files at it will give an
>    error message suggesting that they try ``--allow-external foobar``.
> 2. They then attempt to install their package using
>    ``pip install --allow-external foobar foobar``. If they are lucky foobar is
>    one of the packages which is hosted externally and safely and this will
>    succeed. If they are unlucky they will get a different error message
>    suggesting that they *also* try ``--allow-unverified foobar``.
> 3. They then attempt to install their package using
>    ``pip install --allow-external foobar --allow-unverified foobar foobar``
>    and this finally works.
> 
> This is the same basic steps that practically everyone goes through every time
> they try to install something that is not hosted on PyPI. If they are lucky it'll
> only take them two steps, but typically it requires three steps. Worse there is
> no real indication to these people why one package might install after two
> but most require three. Even worse than that most of them will never get an
> externally hosted package that does not take three steps, so they will be
> increasingly annoyed and frustrated at the intermediate step and will likely
> eventually just start skipping it.
> 
> 
> External Index Discovery
> ========================
> 
> One of the problems with using an additional index is one of discovery. Users
> will not generally be aware that an additional index is required at all much
> less where that index can be found. Projects can attempt to convey this
> information using their description on the PyPI page however that excludes
> people who discover their project organically through ``pip search``.
> 
> To support projects that wish to externally host their files and to enable
> users to easily discover what additional indexes are required, PyPI will gain
> the ability for projects to register external index URLs and additionally an
> associated comment for each. These URLs will be made available on the simple
> page however they will not be linked or provided in a form that older
> installers will automatically search them.
> 
> When an installer fetches the simple page for a project, if it finds this
> additional meta-data and it cannot find any files for that project in it's
> configured URLs then it should use this data to tell the user how to add one
> or more of the additional URLs to search in. This message should include any
> comments that the project has included to enable them to communicate to the
> user and provide hints as to which URL they might want if some are only
> useful or compatible with certain platforms or situations. When the installer
> has implemented the auto discovery mechanisms they should also deprecate any
> of the mechanisms added for PEP 438 (such as ``--allow-external``) for removal
> at the end of the deprecation period proposed by the PEP.
> 
> This feature *must* be added to PyPI prior to starting the deprecation and
> removal process for link spidering.
> 
> 
> Deprecation and Removal of Link Spidering
> =========================================
> 
> A new hosting mode will be added to PyPI. This hosting mode will be called
> ``pypi-only`` and will be in addition to the three that PEP 438 has already
> given us which are ``pypi-explicit``, ``pypi-scrape``, ``pypi-scrape-crawl``.
> This new hosting mode will modify a project's simple api page so that it only
> lists the files which are directly hosted on PyPI and will not link to anything
> else.
> 
> Upon acceptance of this PEP and the addition of the ``pypi-only`` mode, all new
> projects will by defaulted to the PyPI only mode and they will be locked to
> this mode and unable to change this particular setting. ``pypi-only`` projects
> will still be able to register external index URLs as described above - the
> "pypi-only" refers only to the download links that are published directly on
> PyPI.
> 
> An email will then be sent out to all of the projects which are hosted only on
> PyPI informing them that in one month their project will be automatically
> converted to the ``pypi-only`` mode. A month after these emails have been sent
> any of those projects which were emailed, which still are hosted only on PyPI
> will have their mode set to ``pypi-only``.
> 
> After that switch, an email will be sent to projects which rely on hosting
> external to PyPI. This email will warn these projects that externally hosted
> files have been deprecated on PyPI and that in 6 months from the time of that
> email that all external links will be removed from the installer APIs. This
> email *must* include instructions for converting their projects to be hosted
> on PyPI and *must* include links to a script or package that will enable them
> to enter their PyPI credentials and package name and have it automatically
> download and re-host all of their files on PyPI. This email *must also*
> include instructions for setting up their own index page and registering that
> with PyPI, including the fact that they can use pythonhosted.org as a host
> for an index page without requiring them to host any additional infrastructure
> or purchase a TLS certificate. This email must also contain a link to the Terms
> of Service for PyPI as many users may have signed up a long time ago and may
> not recall what those terms are.
> 
> Five months after the initial email, another email must be sent to any projects
> still relying on external hosting. This email will include all of the same
> information that the first email contained, except that the removal date will
> be one month away instead of six.
> 
> Finally a month later all projects will be switched to the ``pypi-only`` mode
> and PyPI will be modified to remove the externally linked files functionality.
> At this point in time any installers should finally remove any of the
> deprecated PEP 438 functionality such as ``--allow-external`` and
> ``--allow-unverified`` in pip.
> 
> 
> PIL
> ---
> 
> It's obvious from the numbers below that the vast bulk of the impact come from
> the PIL project. On 2014-05-17 an email was sent to the contact for PIL
> inquiring whether or not they would be willing to upload to PyPI. A response
> has not been received as of yet (2014-06-05) nor has any change in the hosting
> happened. Due to the popularity of PIL this PEP also proposes that during the
> deprecation period that PyPI Administrators will set the PIL download URL as
> the external index for that project. Allowing the users of PIL to take
> advantage of the auto discovery mechanisms although the project has seemingly
> become unmaintained.
> 
> 
> Impact
> ======
> 
> The largest impact of this is going to be projects where the maintainers are
> no longer maintaining the project, for one reason or another. For these
> projects it's unlikely that a maintainer will arrive to set the external index
> metadata which would allow the auto discovery mechanism to find it.
> 
> Looking at the numbers factoring out PIL (which has been special cased above)
> the actual impact should be quite low, with it affecting just 6.9% of projects
> which host only externally or 2.8% which have their latest version hosted
> externally. This represents a mere 3883 unique IP addresses. The break down of
> this is that of those 3883 addresses, 100% of them installed something that
> could not be verified while only 3% installed something which could be.
> 
> 
> Projects Which Rely on Externally Hosted files
> ----------------------------------------------
> 
> This is determined by crawling the simple index and looking for installable
> files using a similar detection method as pip and setuptools use. The "latest"
> version is determined using ``pkg_resources.parse_version`` sort order and it
> is used to show whether or not the latest version is hosted externally or only
> old versions are.
> 
> ============ ======= ================ =================== =======
> \             PyPI    External (old)   External (latest)   Total
> ============ ======= ================ =================== =======
>  **Safe**     38716   31               35                  38782
>  **Unsafe**   0       1659             1169                2828
>  **Total**    38716   1690             1204                41610
> ============ ======= ================ =================== =======
> 
> 
> Top Externally Hosted Projects by Requests
> ------------------------------------------
> 
> This is determined by looking at the number of requests the
> ``/simple/<project>/`` page had gotten in a single day. The total number of
> requests during that day was 17,960,467.
> 
> ============================== ========
> Project                        Requests
> ============================== ========
> PIL                            13470
> mysql-connector-python         321
> salesforce-python-toolkit      54
> pyodbc                         50
> elementtree                    44
> atfork                         39
> RBTools                        29
> django-contrib-requestprovider 28
> wadofstuff-django-serializers  23
> Pygame                         21
> ============================== ========
> 
> 
> Top Externally Hosted Projects by Unique IPs
> --------------------------------------------
> 
> This is determined by looking at the IP addresses of requests the
> ``/simple/<project>/`` page had gotten in a single day. The total number of
> unique IP addresses during that day was 105,587.
> 
> ============================== ==========
> Project                        Unique IPs
> ============================== ==========
> PIL                            3515
> mysql-connector-python         117
> pyodbc                         34
> elementtree                    21
> RBTools                        19
> egenix-mx-base                 16
> Pygame                         14
> salesforce-python-toolkit      13
> django-contrib-requestprovider 12
> wxPython                       11
> python-apt                     10
> ============================== ==========
> 
> 
> Rejected Proposals
> ==================
> 
> Keep the current classification system but adjust the options
> -------------------------------------------------------------
> 
> This PEP rejects several related proposals which attempt to fix some of the
> usability problems with the current system but while still keeping the
> general gist of PEP 438.
> 
> This includes:
> 
> * Default to allowing safely externally hosted files, but disallow unsafely
>   hosted.
> * Default to disallowing safely externally hosted files with only a global
>   flag to enable them, but disallow unsafely hosted.
> 
> These proposals are rejected because:
> 
> * The classification "system" is complex, hard to explain, and requires an
>   intimate knowledge of how the simple API works in order to be able to reason
>   about which classification is required. This is reflected in the fact that
>   the code to implement it is complicated and hard to understand as well.
> 
> * People are generally surprised that PyPI allows externally linking to files
>   and doesn't require people to host on PyPI. In contrast most of them are
>   familiar with the concept of multiple software repositories such as is in
>   use by many OSs.
> 
> * PyPI is fronted by a globally distributed CDN which has improved the
>   reliability and speed for end users. It is unlikely that any particular
>   external host has something comparable. This can lead to extremely bad
>   performance for end users when the external host is located in different
>   parts of the world or does not generally have good connectivity.
> 
>   As a data point, many users reported sub DSL speeds and latency when
>   accessing PyPI from parts of Europe and Asia prior to the use of the CDN.
> 
> * PyPI has monitoring and an on-call rotation of sysadmins whom can respond to
>   downtime quickly, thus enabling a quicker response to downtime. Again it is
>   unlikely that any particular external host will have this. This can lead
>   to single packages in a dependency chain being un-installable. This will
>   often confuse users, who often times have no idea that this package relies
>   on an external host, and they cannot figure out why PyPI appears to be up
>   but the installer cannot find a package.
> 
> * PyPI supports mirroring, both for private organizations and public mirrors.
>   The legal terms of uploading to PyPI ensure that mirror operators, both
>   public and private, have the right to distribute the software found on PyPI.
>   However software that is hosted externally does not have this, causing
>   private organizations to need to investigate each package individually and
>   manually to determine if the license allows them to mirror it.
> 
>   For public mirrors this essentially means that these externally hosted
>   packages *cannot* be reasonably mirrored. This is particularly troublesome
>   in countries such as China where the bandwidth to outside of China is
>   highly congested making a mirror within China often times a massively better
>   experience.
> 
> * Installers have no method to determine if they should expect any particular
>   URL to be available or not. It is not unusual for the simple API to reference
>   old packages and URLs which have long since stopped working. This causes
>   installers to have to assume that it is OK for any particular URL to not be
>   accessible. This causes problems where an URL is temporarily down or
>   otherwise unavailable (a common cause of this is using a copy of Python
>   linked against a really ancient copy of OpenSSL which is unable to verify
>   the SSL certificate on PyPI) but it *should* be expected to be up. In this
>   case installers will typically silently ignore this URL and later the user
>   will get a confusing error stating that the installer couldn't find any
>   versions instead of getting the real error message indicating that the URL
>   was unavailable.
> 
> * In the long run, global opt in flags like ``--allow-all-external`` will
>   become little annoyances that developers cargo cult around in order to make
>   their installer work. When they run into a project that requires it they
>   will most likely simply add it to their configuration file for that installer
>   and continue on with whatever they were actually trying to do. This will
>   continue until they try to install their requirements on another computer
>   or attempt to deploy to a server where their install will fail again until
>   they add the "make it work" flag in their configuration file.
> 
> 
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 



> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140606/cf60366a/attachment-0001.sig>

From donald at stufft.io  Fri Jun  6 13:55:05 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 6 Jun 2014 07:55:05 -0400
Subject: [Distutils] PEP 470 Round 2 - Using Multi Index Support for
	External to PyPI Package File Hosting
In-Reply-To: <20140606081348.GG30349@merlinux.eu>
References: <0D7BEAFC-7987-46C8-8A9A-7931BE4DA36B@stufft.io>
 <20140606081348.GG30349@merlinux.eu>
Message-ID: <18F8F7BC-11E0-462F-AAB8-635114DEF05C@stufft.io>


On Jun 6, 2014, at 4:13 AM, holger krekel <holger at merlinux.eu> wrote:

> Hi Donald,
> 
> 1. you published numbers where <4K or <300 discounting PIL would be
>   affected by PEP470.  You also say that the main reason for deprecating
>   PEP438 is that it confused users.  Did it confuse other users than those few?

It confused more of than the current numbers because at the onset more projects
relied on it than does now. Currently PIL is the primary instigator for people?s
confusion that I personally see.

> 
> 2. I don't see a valid precise reasoning why PEP438, just agreed on and 
>   implemented last year, needs deprecation.  It boosted everyone
>   everyone's install experiences (independently from the CDN which
>   brought another boost) as usage of crawling dramatically dropped 
>   and thus brings us into the exact situation PEP438 already hinted at:
> 
>   "Deprecation of hosting modes to eventually only allow the
>   pypi-explicit mode is NOT REGULATED by this PEP but is expected to
>   become feasible some time after successful implementation of the
>   transition phases described in this PEP. It is expected that
>   deprecation requires a new process to deal with abandoned packages
>   because of unreachable maintainers for still popular packages."
> 
>   We should follow through and discuss removing crawling and 
>   how to deal with abandoned packages.  On the PyPI side, what 
>   would remain are two kind of links:
> 
>   - pypi internally hosted
>   - registered safe external links to release files
> 
>   The resulting situation is:
> 
>   easy: users have an already existing option to consider to allow externals.
> 
>   safe: All links served from pypi have checksums. Project maintainers need
>         to register hashed links to their new release files.
> 
>   clean: Pip could eventually remove support for crawling/related options.
> 
>   This is all easy to do, reduces user confusion and makes pip
>   and pypi simpler and less suprising.
> 
>   I don't see this approach discussed or seriously considered in the PEP,
>   also not in its "rejection reasons?.

The reasons are listed in the PEP, though I can make it more explicit that
it is for this as well.

* People are generally surprised that PyPI allows externally linking to files
  and doesn't require people to host on PyPI. In contrast most of them are
  familiar with the concept of multiple software repositories such as is in
  use by many OSs.

* PyPI is fronted by a globally distributed CDN which has improved the
  reliability and speed for end users. It is unlikely that any particular
  external host has something comparable. This can lead to extremely bad
  performance for end users when the external host is located in different
  parts of the world or does not generally have good connectivity.

  As a data point, many users reported sub DSL speeds and latency when
  accessing PyPI from parts of Europe and Asia prior to the use of the CDN.

* PyPI has monitoring and an on-call rotation of sysadmins whom can respond to
  downtime quickly, thus enabling a quicker response to downtime. Again it is
  unlikely that any particular external host will have this. This can lead
  to single packages in a dependency chain being un-installable. This will
  often confuse users, who often times have no idea that this package relies
  on an external host, and they cannot figure out why PyPI appears to be up
  but the installer cannot find a package.

* PyPI supports mirroring, both for private organizations and public mirrors.
  The legal terms of uploading to PyPI ensure that mirror operators, both
  public and private, have the right to distribute the software found on PyPI.
  However software that is hosted externally does not have this, causing
  private organizations to need to investigate each package individually and
  manually to determine if the license allows them to mirror it.

  For public mirrors this essentially means that these externally hosted
  packages *cannot* be reasonably mirrored. This is particularly troublesome
  in countries such as China where the bandwidth to outside of China is
  highly congested making a mirror within China often times a massively better
  experience.

* In the long run, global opt in flags like ``--allow-all-external`` will
  become little annoyances that developers cargo cult around in order to make
  their installer work. When they run into a project that requires it they
  will most likely simply add it to their configuration file for that installer
  and continue on with whatever they were actually trying to do. This will
  continue until they try to install their requirements on another computer
  or attempt to deploy to a server where their install will fail again until
  they add the "make it work" flag in their configuration file.

Implied but not explicitly called out reason (I?ll add this):

* The URL classification only works for a certain subset of projects, however
  it does not allow for any project which needs additional restrictions such
  as Access Controls. This means that there would be two methods of doing the
  same thing, linking to a file safely and hosting an index. Hosting an index
  works in all situations and by relying on this we make for a more consistent
  experience no matter the reason for external hosting.

Not implied, but I?ll add as well:

* The safe external hosting option hampers the ability of PyPI to upgrade it's
  security infrastructure. For instance if MD5 becomes broken in the future
  there will be no way for PyPI to upgrade the hashes of the projects which
  rely on safe external hosting via MD5 while files that are hosted on PyPI
  can simply be processed over with a new hash function.

Not going to add:

Ultimately we're looking at maintaining an additional feature, both in PyPI
and in the installers, which almost nobody is actually taking advantage of. A
year ago I was hopeful that perhaps people would take advantage of it and maybe
that would solve some of the issues. However it's now been an entire year and
the buy-in for that feature is minuscule. I do not believe that it makes sense
to pay the cost for continuing to have that feature.

It costs time and effort to ensure that these features to not break. It also
adds to the cognitive burden of using the installers. We require end users to
learn two different ways of specifying they wish to allow external indexes
and require them to understand when or why they'd use one or the other. It is
my opinion that this makes the end user experience wholly worse.

> 
>   By contrast, PEP470 would require many users to learn about
>   specifying other indexes and what that means.  For you and me
>   and many here on the list it may be a no-brainer but trust me,
>   for many users (i've done ten trainings touching the topic now)
>   this is not a natural concept at all.  "pip install --allow-all-externals" 
>   is far easier to convey than specifying extra per-project indexes and 
>   what it means if the install fails (wrong URL? Index noch reachable?
>   Release file not found?).

Some projects host their files on PyPI, some files do not, if the thing you're
trying to install doesn't host on PyPI you'll have to tell pip where it's
hosted. I don't really believe this is a difficult concept.

> 
> 3. PEP470 makes life a lot harder for devpi-server, currently used
>   by many companies for serving their private indexes.  With PEP438 and
>   almost no external crawling left, devpi-server can rely on seeing
>   changes through the PEP381 API.  By contrast, with projects hosted on
>   additional per-project external indexes, it requires polling to see
>   changes because releases may not be registered with PyPI anymore (and
>   there is no way to enforce that IISIC).  IOW, PEP470 is a serious
>   regression here as it doesn't allow getting notified on new release files.

That only works for the set of projects which are currently safely externally
hosted, which again is tiny. It does not work at all for projects which are
hosted externally nor does it work for projects which require some sort of
ACL.

Additionally I don't believe devpi *can* actually do that, as it has no way
of knowing if it's even legal for it to be mirroring the files if they are
not hosted on PyPI. This is one of the points of the PEP, there is a known
legal right to distribute files hosted on PyPI, no such right is promised for
any file not hosted on PyPI. It's completely possible that devpi is doing
something that it has no legal right to do in those cases.

> 
> best,
> holger
> 
> On Thu, Jun 05, 2014 at 22:08 -0400, Donald Stufft wrote:
>> Here's round 2 of PEP 470.
>> 
>> You can see it online at https://python.org/dev/peps/pep-0470/ or below.
>> 
>> Notable changes:
>> 
>> - Ensure it's obvious this strictly deals with the installer API and does not
>>  affect a project's ability to register their project on PyPI for human
>>  consumptions.
>> 
>> - Mention that the functional mechanisms that make it possible for an end user
>>  to specify the additional locations have existed for a long time across many
>>  versions of the installers.
>> 
>> - Explicitly mention that the installer changes from PEP 438 should be
>>  deprecated and removed as part of this PEP.
>> 
>> - Explicitly mention pythonhosted.org as a location that authors can use to
>>  host an index if they do not wish to purchase a TLS certificate or host
>>  additional infrastructure.
>> 
>> - Include that a link to PyPI ToS should be included in the emails sent to
>>  authors to remind them of the PyPI ToS.
>> 
>> - Special case PIL as it is an outlier in terms of impact.
>> 
>> - Fill out the impact sections further to provide more detail
>> 
>> 
>> Abstract
>> ========
>> 
>> This PEP proposes that the official means of having an installer locate and
>> find package files which are hosted externally to PyPI become the use of
>> multi index support instead of the practice of using external links on the
>> simple installer API.
>> 
>> It is important to remember that this is **not** about forcing anyone to host
>> their files on PyPI. If someone does not wish to do so they will never be under
>> any obligation too. They can still list their project in PyPI as an index, and
>> the tooling will still allow them to host it elsewhere.
>> 
>> This PEP strictly is concerned with the Simple Installer API and how automated
>> installers interact with PyPI, it has no bearing on the informational pages
>> which are primarily for human consumption.
>> 
>> 
>> Rationale
>> =========
>> 
>> There is a long history documented in PEP 438 that explains why externally
>> hosted files exist today in the state that they do on PyPI. For the sake of
>> brevity I will not duplicate that and instead urge readers to first take a look
>> at PEP 438 for background.
>> 
>> There are currently two primary ways for a project to make itself available
>> without directly hosting the package files on PyPI. They can either include
>> links to the package files in the simpler installer API or they can publish
>> a custom package index which contains their project.
>> 
>> 
>> Custom Additional Index
>> -----------------------
>> 
>> Each installer which speaks to PyPI offers a mechanism for the user invoking
>> that installer to provide additional custom locations to search for files
>> during the dependency resolution phase. For pip these locations can be
>> configured per invocation, per shell environment, per requirements file, per
>> virtual environment, and per user. The mechanism for specifying additional
>> locations have existed within pip and setuptools for many years, by comparison
>> the mechanisms in PEP 438 and any other new mechanism will have existed for
>> only a short period of time (if they exist at all currently).
>> 
>> The use of additional indexes instead of external links on the simple
>> installer API provides a simple clean interface which is consistent with the
>> way most Linux package systems work (apt-get, yum, etc). More importantly it
>> works the same even for projects which are commercial or otherwise have their
>> access restricted in some form (private networks, password, IP ACLs etc)
>> while the external links method only realistically works for projects which
>> do not have their access restricted.
>> 
>> Compared to the complex rules which a project must be aware of to prevent
>> themselves from being considered unsafely hosted setting up an index is fairly
>> trivial and in the simplest case does not require anything more than a
>> filesystem and a standard web server such as Nginx or Twisted Web. Even if
>> using simple static hosting without autoindexing support, it is still
>> straightforward to generate appropriate index pages as static HTML.
>> 
>> Example Index with Twisted Web
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> 
>> 1. Create a root directory for your index, for the purposes of the example
>>   I'll assume you've chosen ``/var/www/index.example.com/``.
>> 2. Inside of this root directory, create a directory for each project such
>>   as ``mkdir -p /var/www/index.example.com/{foo,bar,other}/``.
>> 3. Place the package files for each project in their respective folder,
>>   creating paths like ``/var/www/index.example.com/foo/foo-1.0.tar.gz``.
>> 4. Configure Twisted Web to serve the root directory, ideally with TLS.
>> 
>> ::
>> 
>>    $ twistd -n web --path /var/www/index.example.com/
>> 
>> 
>> Examples of Additional indexes with pip
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> 
>> **Invocation:**
>> 
>> ::
>> 
>>    $ pip install --extra-index-url https://pypi.example.com/ foobar
>> 
>> **Shell Environment:**
>> 
>> ::
>> 
>>    $ export PIP_EXTRA_INDEX_URL=https://pypi.example.com/
>>    $ pip install foobar
>> 
>> **Requirements File:**
>> 
>> ::
>> 
>>    $ echo "--extra-index-url https://pypi.example.com/\nfoobar" > requirements.txt
>>    $ pip install -r requirements.txt
>> 
>> **Virtual Environment:**
>> 
>> ::
>> 
>>    $ python -m venv myvenv
>>    $ echo "[global]\nextra-index-url = https://pypi.example.com/" > myvenv/pip.conf
>>    $ myvenv/bin/pip install foobar
>> 
>> **User:**
>> 
>> ::
>> 
>>    $ echo "[global]\nextra-index-url = https://pypi.example.com/" >~/.pip/pip.conf
>>    $ pip install foobar
>> 
>> 
>> External Links on the Simple Installer API
>> ------------------------------------------
>> 
>> PEP 438 proposed a system of classifying file links as either internal,
>> external, or unsafe. It recommended that by default only internal links would
>> be installed by an installer however users could opt into external links on
>> either a global or a per package basis. Additionally they could also opt into
>> unsafe links on a per package basis.
>> 
>> This system has turned out to be *extremely* unfriendly towards the end users
>> and it is the position of this PEP that the situation has become untenable. The
>> situation as provided by PEP 438 requires an end user to be aware not only of
>> the difference between internal, external, and unsafe, but also to be aware of
>> what hosting mode the package they are trying to install is in, what links are
>> available on that project's /simple/ page, whether or not those links have
>> a properly formatted hash fragment, and what links are available from pages
>> linked to from that project's /simple/ page.
>> 
>> There are a number of common confusion/pain points with this system that I
>> have witnessed:
>> 
>> * Users unaware what the simple installer api is at all or how an installer
>>  locates installable files.
>> * Users unaware that even if the simple api links to a file, if it does
>>  not include a ``#md5=...`` fragment that it will be counted as unsafe.
>> * Users unaware that an installer can look at pages linked from the
>>  simple api to determine additional links, or that any links found in this
>>  fashion are considered unsafe.
>> * Users are unaware and often surprised that PyPI supports hosting your files
>>  someplace other than PyPI at all.
>> 
>> In addition to that, the information that an installer is able to provide
>> when an installation fails is pretty minimal. We are able to detect if there
>> are externally hosted files directly linked from the simple installer api,
>> however we cannot detect if there are files hosted on a linked page without
>> fetching that page and doing so would cause a massive performance hit just to
>> see if there might be a file there so that a better error message could be
>> provided.
>> 
>> Finally very few projects have properly linked to their external files so that
>> they can be safely downloaded and verified. At the time of this writing there
>> are a total of 65 projects which have files that are only available externally
>> and are safely hosted.
>> 
>> The end result of all of this, is that with PEP 438, when a user attempts to
>> install a file that is not hosted on PyPI typically the steps they follow are:
>> 
>> 1. First, they attempt to install it normally, using ``pip install foobar``.
>>   This fails because the file is not hosted on PyPI and PEP 438 has us default
>>   to only hosted on PyPI. If pip detected any externally hosted files or other
>>   pages that we *could* have attempted to find other files at it will give an
>>   error message suggesting that they try ``--allow-external foobar``.
>> 2. They then attempt to install their package using
>>   ``pip install --allow-external foobar foobar``. If they are lucky foobar is
>>   one of the packages which is hosted externally and safely and this will
>>   succeed. If they are unlucky they will get a different error message
>>   suggesting that they *also* try ``--allow-unverified foobar``.
>> 3. They then attempt to install their package using
>>   ``pip install --allow-external foobar --allow-unverified foobar foobar``
>>   and this finally works.
>> 
>> This is the same basic steps that practically everyone goes through every time
>> they try to install something that is not hosted on PyPI. If they are lucky it'll
>> only take them two steps, but typically it requires three steps. Worse there is
>> no real indication to these people why one package might install after two
>> but most require three. Even worse than that most of them will never get an
>> externally hosted package that does not take three steps, so they will be
>> increasingly annoyed and frustrated at the intermediate step and will likely
>> eventually just start skipping it.
>> 
>> 
>> External Index Discovery
>> ========================
>> 
>> One of the problems with using an additional index is one of discovery. Users
>> will not generally be aware that an additional index is required at all much
>> less where that index can be found. Projects can attempt to convey this
>> information using their description on the PyPI page however that excludes
>> people who discover their project organically through ``pip search``.
>> 
>> To support projects that wish to externally host their files and to enable
>> users to easily discover what additional indexes are required, PyPI will gain
>> the ability for projects to register external index URLs and additionally an
>> associated comment for each. These URLs will be made available on the simple
>> page however they will not be linked or provided in a form that older
>> installers will automatically search them.
>> 
>> When an installer fetches the simple page for a project, if it finds this
>> additional meta-data and it cannot find any files for that project in it's
>> configured URLs then it should use this data to tell the user how to add one
>> or more of the additional URLs to search in. This message should include any
>> comments that the project has included to enable them to communicate to the
>> user and provide hints as to which URL they might want if some are only
>> useful or compatible with certain platforms or situations. When the installer
>> has implemented the auto discovery mechanisms they should also deprecate any
>> of the mechanisms added for PEP 438 (such as ``--allow-external``) for removal
>> at the end of the deprecation period proposed by the PEP.
>> 
>> This feature *must* be added to PyPI prior to starting the deprecation and
>> removal process for link spidering.
>> 
>> 
>> Deprecation and Removal of Link Spidering
>> =========================================
>> 
>> A new hosting mode will be added to PyPI. This hosting mode will be called
>> ``pypi-only`` and will be in addition to the three that PEP 438 has already
>> given us which are ``pypi-explicit``, ``pypi-scrape``, ``pypi-scrape-crawl``.
>> This new hosting mode will modify a project's simple api page so that it only
>> lists the files which are directly hosted on PyPI and will not link to anything
>> else.
>> 
>> Upon acceptance of this PEP and the addition of the ``pypi-only`` mode, all new
>> projects will by defaulted to the PyPI only mode and they will be locked to
>> this mode and unable to change this particular setting. ``pypi-only`` projects
>> will still be able to register external index URLs as described above - the
>> "pypi-only" refers only to the download links that are published directly on
>> PyPI.
>> 
>> An email will then be sent out to all of the projects which are hosted only on
>> PyPI informing them that in one month their project will be automatically
>> converted to the ``pypi-only`` mode. A month after these emails have been sent
>> any of those projects which were emailed, which still are hosted only on PyPI
>> will have their mode set to ``pypi-only``.
>> 
>> After that switch, an email will be sent to projects which rely on hosting
>> external to PyPI. This email will warn these projects that externally hosted
>> files have been deprecated on PyPI and that in 6 months from the time of that
>> email that all external links will be removed from the installer APIs. This
>> email *must* include instructions for converting their projects to be hosted
>> on PyPI and *must* include links to a script or package that will enable them
>> to enter their PyPI credentials and package name and have it automatically
>> download and re-host all of their files on PyPI. This email *must also*
>> include instructions for setting up their own index page and registering that
>> with PyPI, including the fact that they can use pythonhosted.org as a host
>> for an index page without requiring them to host any additional infrastructure
>> or purchase a TLS certificate. This email must also contain a link to the Terms
>> of Service for PyPI as many users may have signed up a long time ago and may
>> not recall what those terms are.
>> 
>> Five months after the initial email, another email must be sent to any projects
>> still relying on external hosting. This email will include all of the same
>> information that the first email contained, except that the removal date will
>> be one month away instead of six.
>> 
>> Finally a month later all projects will be switched to the ``pypi-only`` mode
>> and PyPI will be modified to remove the externally linked files functionality.
>> At this point in time any installers should finally remove any of the
>> deprecated PEP 438 functionality such as ``--allow-external`` and
>> ``--allow-unverified`` in pip.
>> 
>> 
>> PIL
>> ---
>> 
>> It's obvious from the numbers below that the vast bulk of the impact come from
>> the PIL project. On 2014-05-17 an email was sent to the contact for PIL
>> inquiring whether or not they would be willing to upload to PyPI. A response
>> has not been received as of yet (2014-06-05) nor has any change in the hosting
>> happened. Due to the popularity of PIL this PEP also proposes that during the
>> deprecation period that PyPI Administrators will set the PIL download URL as
>> the external index for that project. Allowing the users of PIL to take
>> advantage of the auto discovery mechanisms although the project has seemingly
>> become unmaintained.
>> 
>> 
>> Impact
>> ======
>> 
>> The largest impact of this is going to be projects where the maintainers are
>> no longer maintaining the project, for one reason or another. For these
>> projects it's unlikely that a maintainer will arrive to set the external index
>> metadata which would allow the auto discovery mechanism to find it.
>> 
>> Looking at the numbers factoring out PIL (which has been special cased above)
>> the actual impact should be quite low, with it affecting just 6.9% of projects
>> which host only externally or 2.8% which have their latest version hosted
>> externally. This represents a mere 3883 unique IP addresses. The break down of
>> this is that of those 3883 addresses, 100% of them installed something that
>> could not be verified while only 3% installed something which could be.
>> 
>> 
>> Projects Which Rely on Externally Hosted files
>> ----------------------------------------------
>> 
>> This is determined by crawling the simple index and looking for installable
>> files using a similar detection method as pip and setuptools use. The "latest"
>> version is determined using ``pkg_resources.parse_version`` sort order and it
>> is used to show whether or not the latest version is hosted externally or only
>> old versions are.
>> 
>> ============ ======= ================ =================== =======
>> \             PyPI    External (old)   External (latest)   Total
>> ============ ======= ================ =================== =======
>> **Safe**     38716   31               35                  38782
>> **Unsafe**   0       1659             1169                2828
>> **Total**    38716   1690             1204                41610
>> ============ ======= ================ =================== =======
>> 
>> 
>> Top Externally Hosted Projects by Requests
>> ------------------------------------------
>> 
>> This is determined by looking at the number of requests the
>> ``/simple/<project>/`` page had gotten in a single day. The total number of
>> requests during that day was 17,960,467.
>> 
>> ============================== ========
>> Project                        Requests
>> ============================== ========
>> PIL                            13470
>> mysql-connector-python         321
>> salesforce-python-toolkit      54
>> pyodbc                         50
>> elementtree                    44
>> atfork                         39
>> RBTools                        29
>> django-contrib-requestprovider 28
>> wadofstuff-django-serializers  23
>> Pygame                         21
>> ============================== ========
>> 
>> 
>> Top Externally Hosted Projects by Unique IPs
>> --------------------------------------------
>> 
>> This is determined by looking at the IP addresses of requests the
>> ``/simple/<project>/`` page had gotten in a single day. The total number of
>> unique IP addresses during that day was 105,587.
>> 
>> ============================== ==========
>> Project                        Unique IPs
>> ============================== ==========
>> PIL                            3515
>> mysql-connector-python         117
>> pyodbc                         34
>> elementtree                    21
>> RBTools                        19
>> egenix-mx-base                 16
>> Pygame                         14
>> salesforce-python-toolkit      13
>> django-contrib-requestprovider 12
>> wxPython                       11
>> python-apt                     10
>> ============================== ==========
>> 
>> 
>> Rejected Proposals
>> ==================
>> 
>> Keep the current classification system but adjust the options
>> -------------------------------------------------------------
>> 
>> This PEP rejects several related proposals which attempt to fix some of the
>> usability problems with the current system but while still keeping the
>> general gist of PEP 438.
>> 
>> This includes:
>> 
>> * Default to allowing safely externally hosted files, but disallow unsafely
>>  hosted.
>> * Default to disallowing safely externally hosted files with only a global
>>  flag to enable them, but disallow unsafely hosted.
>> 
>> These proposals are rejected because:
>> 
>> * The classification "system" is complex, hard to explain, and requires an
>>  intimate knowledge of how the simple API works in order to be able to reason
>>  about which classification is required. This is reflected in the fact that
>>  the code to implement it is complicated and hard to understand as well.
>> 
>> * People are generally surprised that PyPI allows externally linking to files
>>  and doesn't require people to host on PyPI. In contrast most of them are
>>  familiar with the concept of multiple software repositories such as is in
>>  use by many OSs.
>> 
>> * PyPI is fronted by a globally distributed CDN which has improved the
>>  reliability and speed for end users. It is unlikely that any particular
>>  external host has something comparable. This can lead to extremely bad
>>  performance for end users when the external host is located in different
>>  parts of the world or does not generally have good connectivity.
>> 
>>  As a data point, many users reported sub DSL speeds and latency when
>>  accessing PyPI from parts of Europe and Asia prior to the use of the CDN.
>> 
>> * PyPI has monitoring and an on-call rotation of sysadmins whom can respond to
>>  downtime quickly, thus enabling a quicker response to downtime. Again it is
>>  unlikely that any particular external host will have this. This can lead
>>  to single packages in a dependency chain being un-installable. This will
>>  often confuse users, who often times have no idea that this package relies
>>  on an external host, and they cannot figure out why PyPI appears to be up
>>  but the installer cannot find a package.
>> 
>> * PyPI supports mirroring, both for private organizations and public mirrors.
>>  The legal terms of uploading to PyPI ensure that mirror operators, both
>>  public and private, have the right to distribute the software found on PyPI.
>>  However software that is hosted externally does not have this, causing
>>  private organizations to need to investigate each package individually and
>>  manually to determine if the license allows them to mirror it.
>> 
>>  For public mirrors this essentially means that these externally hosted
>>  packages *cannot* be reasonably mirrored. This is particularly troublesome
>>  in countries such as China where the bandwidth to outside of China is
>>  highly congested making a mirror within China often times a massively better
>>  experience.
>> 
>> * Installers have no method to determine if they should expect any particular
>>  URL to be available or not. It is not unusual for the simple API to reference
>>  old packages and URLs which have long since stopped working. This causes
>>  installers to have to assume that it is OK for any particular URL to not be
>>  accessible. This causes problems where an URL is temporarily down or
>>  otherwise unavailable (a common cause of this is using a copy of Python
>>  linked against a really ancient copy of OpenSSL which is unable to verify
>>  the SSL certificate on PyPI) but it *should* be expected to be up. In this
>>  case installers will typically silently ignore this URL and later the user
>>  will get a confusing error stating that the installer couldn't find any
>>  versions instead of getting the real error message indicating that the URL
>>  was unavailable.
>> 
>> * In the long run, global opt in flags like ``--allow-all-external`` will
>>  become little annoyances that developers cargo cult around in order to make
>>  their installer work. When they run into a project that requires it they
>>  will most likely simply add it to their configuration file for that installer
>>  and continue on with whatever they were actually trying to do. This will
>>  continue until they try to install their requirements on another computer
>>  or attempt to deploy to a server where their install will fail again until
>>  they add the "make it work" flag in their configuration file.
>> 
>> 
>> -----------------
>> Donald Stufft
>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>> 
> 
> 
> 
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
> 


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140606/cd6f3dd3/attachment-0001.sig>

From donald at stufft.io  Fri Jun  6 13:58:04 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 6 Jun 2014 07:58:04 -0400
Subject: [Distutils] PEP 470 Round 2 - Using Multi Index Support for
	External to PyPI Package File Hosting
In-Reply-To: <18F8F7BC-11E0-462F-AAB8-635114DEF05C@stufft.io>
References: <0D7BEAFC-7987-46C8-8A9A-7931BE4DA36B@stufft.io>
 <20140606081348.GG30349@merlinux.eu>
 <18F8F7BC-11E0-462F-AAB8-635114DEF05C@stufft.io>
Message-ID: <AA69EF34-33C1-4259-8E23-D86CC1A003EC@stufft.io>

I?ve updated the PEP:

http://hg.python.org/peps/rev/3128e9d38937


files:
 pep-0470.txt |  15 +++++++++++++++
 1 files changed, 15 insertions(+), 0 deletions(-)


diff --git a/pep-0470.txt b/pep-0470.txt
--- a/pep-0470.txt
+++ b/pep-0470.txt
@@ -389,6 +389,9 @@
  hosted.
* Default to disallowing safely externally hosted files with only a global
  flag to enable them, but disallow unsafely hosted.
+* Continue on the suggested path of PEP 438 and remove the option to unsafely
+  host externally but continue to allow the option to safely host externally.
+

These proposals are rejected because:

@@ -454,6 +457,18 @@
  or attempt to deploy to a server where their install will fail again until
  they add the "make it work" flag in their configuration file.

+* The URL classification only works for a certain subset of projects, however
+  it does not allow for any project which needs additional restrictions such
+  as Access Controls. This means that there would be two methods of doing the
+  same thing, linking to a file safely and hosting an index. Hosting an index
+  works in all situations and by relying on this we make for a more consistent
+  experience no matter the reason for external hosting.
+
+* The safe external hosting option hampers the ability of PyPI to upgrade it's
+  security infrastructure. For instance if MD5 becomes broken in the future
+  there will be no way for PyPI to upgrade the hashes of the projects which
+  rely on safe external hosting via MD5 while files that are hosted on PyPI
+  can simply be processed over with a new hash function.

Copyright
=========

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140606/6cc4c509/attachment.sig>

From holger at merlinux.eu  Fri Jun  6 15:41:27 2014
From: holger at merlinux.eu (holger krekel)
Date: Fri, 6 Jun 2014 13:41:27 +0000
Subject: [Distutils] PEP 470 Round 2 - Using Multi Index Support for
 External to PyPI Package File Hosting
In-Reply-To: <18F8F7BC-11E0-462F-AAB8-635114DEF05C@stufft.io>
References: <0D7BEAFC-7987-46C8-8A9A-7931BE4DA36B@stufft.io>
 <20140606081348.GG30349@merlinux.eu>
 <18F8F7BC-11E0-462F-AAB8-635114DEF05C@stufft.io>
Message-ID: <20140606134127.GK30349@merlinux.eu>

On Fri, Jun 06, 2014 at 07:55 -0400, Donald Stufft wrote:
> 
> On Jun 6, 2014, at 4:13 AM, holger krekel <holger at merlinux.eu> wrote:
> 
> > Hi Donald,
> > 
> > 1. you published numbers where <4K or <300 discounting PIL would be
> >   affected by PEP470.  You also say that the main reason for deprecating
> >   PEP438 is that it confused users.  Did it confuse other users than those few?
> 
> It confused more of than the current numbers because at the onset more
> projects relied on it than does now. Currently PIL is the primary
> instigator for people?s confusion that I personally see.

So currently we don't have many confused users anymore.  Doesn't
this take away a good part of the reasoning behind PEP470?

In the following i use "PEP438f" to speak about a hypothetical
follow-up PEP as outlined in my previous mail.  I volunteer to write
it and present it as an alternative should we not reach some 
form of conclusion together.

> > 2. I don't see a valid precise reasoning why PEP438, just agreed on and 
> >   implemented last year, needs deprecation.  It boosted everyone
> >   everyone's install experiences (independently from the CDN which
> >   brought another boost) as usage of crawling dramatically dropped 
> >   and thus brings us into the exact situation PEP438 already hinted at:
> > 
> >   "Deprecation of hosting modes to eventually only allow the
> >   pypi-explicit mode is NOT REGULATED by this PEP but is expected to
> >   become feasible some time after successful implementation of the
> >   transition phases described in this PEP. It is expected that
> >   deprecation requires a new process to deal with abandoned packages
> >   because of unreachable maintainers for still popular packages."
> > 
> >   We should follow through and discuss removing crawling and 
> >   how to deal with abandoned packages.  On the PyPI side, what 
> >   would remain are two kind of links:
> > 
> >   - pypi internally hosted
> >   - registered safe external links to release files
> > 
> >   The resulting situation is:
> > 
> >   easy: users have an already existing option to consider to allow externals.
> > 
> >   safe: All links served from pypi have checksums. Project maintainers need
> >         to register hashed links to their new release files.
> > 
> >   clean: Pip could eventually remove support for crawling/related options.
> > 
> >   This is all easy to do, reduces user confusion and makes pip
> >   and pypi simpler and less suprising.
> > 
> >   I don't see this approach discussed or seriously considered in the PEP,
> >   also not in its "rejection reasons?.
> 
> The reasons are listed in the PEP, though I can make it more explicit that
> it is for this as well.
> 
> * People are generally surprised that PyPI allows externally linking to files
>   and doesn't require people to host on PyPI. In contrast most of them are
>   familiar with the concept of multiple software repositories such as is in
>   use by many OSs.

"People are generally surprised" is a rather subjective statement.
Wrt to PEP470 we might have at least 65 projects and many more users being 
annoyed rather than just surprised at the sudden change in direction.
Especially if there are no compelling arguments.

> * PyPI is fronted by a globally distributed CDN which has improved the
>   reliability and speed for end users. It is unlikely that any particular
>   external host has something comparable. This can lead to extremely bad
>   performance for end users when the external host is located in different
>   parts of the world or does not generally have good connectivity.
> 
>   As a data point, many users reported sub DSL speeds and latency when
>   accessing PyPI from parts of Europe and Asia prior to the use of the CDN.
>
> * PyPI has monitoring and an on-call rotation of sysadmins whom can respond to
>   downtime quickly, thus enabling a quicker response to downtime. Again it is
>   unlikely that any particular external host will have this. This can lead
>   to single packages in a dependency chain being un-installable. This will
>   often confuse users, who often times have no idea that this package relies
>   on an external host, and they cannot figure out why PyPI appears to be up
>   but the installer cannot find a package.

Sorry but both points have not much to do with the discussion.  If
anything, they speak *against* PEP470 because users would need to rely
on project specific external index sites to even know which releases
exist.  With PEP438 you know that a certain release file must exist and
the installer clearly says "i could not download release file X from
URL".  Works today.

Also the external index could be temporarily broken and serve not the newest
files.  The integrity and reliability of external indexes would generally
not be covered by the CDN and PyPI's on-rotation admins so instead of
speaking for PEP470 they speak against it.

> * PyPI supports mirroring, both for private organizations and public mirrors.
>   The legal terms of uploading to PyPI ensure that mirror operators, both
>   public and private, have the right to distribute the software found on PyPI.
>   However software that is hosted externally does not have this, causing
>   private organizations to need to investigate each package individually and
>   manually to determine if the license allows them to mirror it.
> 
>   For public mirrors this essentially means that these externally hosted
>   packages *cannot* be reasonably mirrored. This is particularly troublesome
>   in countries such as China where the bandwidth to outside of China is
>   highly congested making a mirror within China often times a massively better
>   experience.

With PEP438 today, PyPI merely hosts a link with a checksum, not the file
itself.  Mirrors also just mirror that link but not the file.
How exactly is managing and mirroring a link a problem here?

> * In the long run, global opt in flags like ``--allow-all-external`` will
>   become little annoyances that developers cargo cult around in order to make
>   their installer work. When they run into a project that requires it they
>   will most likely simply add it to their configuration file for that installer
>   and continue on with whatever they were actually trying to do. This will
>   continue until they try to install their requirements on another computer
>   or attempt to deploy to a server where their install will fail again until
>   they add the "make it work" flag in their configuration file.

Well, with PEP470 developers would have to cargo cult around one or 
more "--extra-index-url SOME_URL" options.  Also:

- if a project wants to later change its index URL, existing users option
  will break and the project has no way to communicate it.  With PEP438f
  you can just update your hashes and be done.  
  
- if a project-specific index resides on a domain that changes ownership 
  (e.g. project abandoned, maintainer gone), user are suddenly vulnerable
  without knowing it. With PEP438f the trusted PyPI infrastructure
  provides a checksummed link so new DNS owners can not break into
  the user's computer as they can do with PEP470.

- if projects host on HTTP instead of HTTPS, users are 
  vulnerable against MITM attacks whereas PEP438f provides a checksummed
  link from PyPI which cannot be easily man-in-the-middled.

> Implied but not explicitly called out reason (I?ll add this):
> 
> * The URL classification only works for a certain subset of projects, however
>   it does not allow for any project which needs additional restrictions such
>   as Access Controls. This means that there would be two methods of doing the
>   same thing, linking to a file safely and hosting an index. Hosting an index
>   works in all situations and by relying on this we make for a more consistent
>   experience no matter the reason for external hosting.

Once you care for ACLs for indexes and releases you have a number
of issues to consider, it's hardly related to PEP470/PEP438.

> Not implied, but I?ll add as well:
> 
> * The safe external hosting option hampers the ability of PyPI to upgrade it's
>   security infrastructure. For instance if MD5 becomes broken in the future
>   there will be no way for PyPI to upgrade the hashes of the projects which
>   rely on safe external hosting via MD5 while files that are hosted on PyPI
>   can simply be processed over with a new hash function.

This is a good point.  So we should make sure that it's easy to re-upload
with different hashes and maybe store some more hashes (sha256 etc.) with
an uploaded file and have pypi decide which one is used.

> Not going to add:
> 
> Ultimately we're looking at maintaining an additional feature, both in PyPI
> and in the installers, which almost nobody is actually taking advantage of. A
> year ago I was hopeful that perhaps people would take advantage of it and maybe
> that would solve some of the issues. However it's now been an entire year and
> the buy-in for that feature is minuscule. I do not believe that it makes sense
> to pay the cost for continuing to have that feature.

PEP438 and most of us urged people to use internal hosting and that's
what people did.  That we have 65 projects using safe external hosting
means we succeeded and there is a clear safe path to host your files
externally.  Why did you expect more people to use external hosting
when PyPI's current hosting is perceived as good and we told everyone
to use it?

> It costs time and effort to ensure that these features to not break. It also
> adds to the cognitive burden of using the installers. We require end users to
> learn two different ways of specifying they wish to allow external indexes
> and require them to understand when or why they'd use one or the other. It is
> my opinion that this makes the end user experience wholly worse.

However, we have a rather settled situation now and can easily 
clean it up further: crawling can go, pypi-crawl can go, a number 
of options go.  All easily done with PEP438f.

> >   By contrast, PEP470 would require many users to learn about
> >   specifying other indexes and what that means.  For you and me
> >   and many here on the list it may be a no-brainer but trust me,
> >   for many users (i've done ten trainings touching the topic now)
> >   this is not a natural concept at all.  "pip install --allow-all-externals" 
> >   is far easier to convey than specifying extra per-project indexes and 
> >   what it means if the install fails (wrong URL? Index noch reachable?
> >   Release file not found?).
> 
> Some projects host their files on PyPI, some files do not, if the thing you're
> trying to install doesn't host on PyPI you'll have to tell pip where it's
> hosted. I don't really believe this is a difficult concept.

Well, with PEP470 you need to tell and explain users:
"--extra-index-url <some_url_you_need_to_find_out>" versus
"--allow-all-externals" with PEP438f.

I am pretty sure the second would win most usability contests.

> > 
> > 3. PEP470 makes life a lot harder for devpi-server, currently used
> >   by many companies for serving their private indexes.  With PEP438 and
> >   almost no external crawling left, devpi-server can rely on seeing
> >   changes through the PEP381 API.  By contrast, with projects hosted on
> >   additional per-project external indexes, it requires polling to see
> >   changes because releases may not be registered with PyPI anymore (and
> >   there is no way to enforce that IISIC).  IOW, PEP470 is a serious
> >   regression here as it doesn't allow getting notified on new release files.
> 
> That only works for the set of projects which are currently safely externally
> hosted, which again is tiny.

Well, but it works.  With PEP470 it would not work anymore.  That was my point.
And the question is anyway not how many projects but how many users are using
externally hosted files.

> It does not work at all for projects which are
> hosted externally nor does it work for projects which require some sort of
> ACL.

I don't quite understand why you are talking about ACLs in context of our
discussion.

> Additionally I don't believe devpi *can* actually do that, as it has no way
> of knowing if it's even legal for it to be mirroring the files if they are
> not hosted on PyPI. This is one of the points of the PEP, there is a known
> legal right to distribute files hosted on PyPI, no such right is promised for
> any file not hosted on PyPI. It's completely possible that devpi is doing
> something that it has no legal right to do in those cases.

If I use devpi on my laptop or in an organisation I am using devpi as a
private http cache and not serving my cached files to 3rd parties.

best,
holger

> > 
> > best,
> > holger
> > 
> > On Thu, Jun 05, 2014 at 22:08 -0400, Donald Stufft wrote:
> >> Here's round 2 of PEP 470.
> >> 
> >> You can see it online at https://python.org/dev/peps/pep-0470/ or below.
> >> 
> >> Notable changes:
> >> 
> >> - Ensure it's obvious this strictly deals with the installer API and does not
> >>  affect a project's ability to register their project on PyPI for human
> >>  consumptions.
> >> 
> >> - Mention that the functional mechanisms that make it possible for an end user
> >>  to specify the additional locations have existed for a long time across many
> >>  versions of the installers.
> >> 
> >> - Explicitly mention that the installer changes from PEP 438 should be
> >>  deprecated and removed as part of this PEP.
> >> 
> >> - Explicitly mention pythonhosted.org as a location that authors can use to
> >>  host an index if they do not wish to purchase a TLS certificate or host
> >>  additional infrastructure.
> >> 
> >> - Include that a link to PyPI ToS should be included in the emails sent to
> >>  authors to remind them of the PyPI ToS.
> >> 
> >> - Special case PIL as it is an outlier in terms of impact.
> >> 
> >> - Fill out the impact sections further to provide more detail
> >> 
> >> 
> >> Abstract
> >> ========
> >> 
> >> This PEP proposes that the official means of having an installer locate and
> >> find package files which are hosted externally to PyPI become the use of
> >> multi index support instead of the practice of using external links on the
> >> simple installer API.
> >> 
> >> It is important to remember that this is **not** about forcing anyone to host
> >> their files on PyPI. If someone does not wish to do so they will never be under
> >> any obligation too. They can still list their project in PyPI as an index, and
> >> the tooling will still allow them to host it elsewhere.
> >> 
> >> This PEP strictly is concerned with the Simple Installer API and how automated
> >> installers interact with PyPI, it has no bearing on the informational pages
> >> which are primarily for human consumption.
> >> 
> >> 
> >> Rationale
> >> =========
> >> 
> >> There is a long history documented in PEP 438 that explains why externally
> >> hosted files exist today in the state that they do on PyPI. For the sake of
> >> brevity I will not duplicate that and instead urge readers to first take a look
> >> at PEP 438 for background.
> >> 
> >> There are currently two primary ways for a project to make itself available
> >> without directly hosting the package files on PyPI. They can either include
> >> links to the package files in the simpler installer API or they can publish
> >> a custom package index which contains their project.
> >> 
> >> 
> >> Custom Additional Index
> >> -----------------------
> >> 
> >> Each installer which speaks to PyPI offers a mechanism for the user invoking
> >> that installer to provide additional custom locations to search for files
> >> during the dependency resolution phase. For pip these locations can be
> >> configured per invocation, per shell environment, per requirements file, per
> >> virtual environment, and per user. The mechanism for specifying additional
> >> locations have existed within pip and setuptools for many years, by comparison
> >> the mechanisms in PEP 438 and any other new mechanism will have existed for
> >> only a short period of time (if they exist at all currently).
> >> 
> >> The use of additional indexes instead of external links on the simple
> >> installer API provides a simple clean interface which is consistent with the
> >> way most Linux package systems work (apt-get, yum, etc). More importantly it
> >> works the same even for projects which are commercial or otherwise have their
> >> access restricted in some form (private networks, password, IP ACLs etc)
> >> while the external links method only realistically works for projects which
> >> do not have their access restricted.
> >> 
> >> Compared to the complex rules which a project must be aware of to prevent
> >> themselves from being considered unsafely hosted setting up an index is fairly
> >> trivial and in the simplest case does not require anything more than a
> >> filesystem and a standard web server such as Nginx or Twisted Web. Even if
> >> using simple static hosting without autoindexing support, it is still
> >> straightforward to generate appropriate index pages as static HTML.
> >> 
> >> Example Index with Twisted Web
> >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >> 
> >> 1. Create a root directory for your index, for the purposes of the example
> >>   I'll assume you've chosen ``/var/www/index.example.com/``.
> >> 2. Inside of this root directory, create a directory for each project such
> >>   as ``mkdir -p /var/www/index.example.com/{foo,bar,other}/``.
> >> 3. Place the package files for each project in their respective folder,
> >>   creating paths like ``/var/www/index.example.com/foo/foo-1.0.tar.gz``.
> >> 4. Configure Twisted Web to serve the root directory, ideally with TLS.
> >> 
> >> ::
> >> 
> >>    $ twistd -n web --path /var/www/index.example.com/
> >> 
> >> 
> >> Examples of Additional indexes with pip
> >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >> 
> >> **Invocation:**
> >> 
> >> ::
> >> 
> >>    $ pip install --extra-index-url https://pypi.example.com/ foobar
> >> 
> >> **Shell Environment:**
> >> 
> >> ::
> >> 
> >>    $ export PIP_EXTRA_INDEX_URL=https://pypi.example.com/
> >>    $ pip install foobar
> >> 
> >> **Requirements File:**
> >> 
> >> ::
> >> 
> >>    $ echo "--extra-index-url https://pypi.example.com/\nfoobar" > requirements.txt
> >>    $ pip install -r requirements.txt
> >> 
> >> **Virtual Environment:**
> >> 
> >> ::
> >> 
> >>    $ python -m venv myvenv
> >>    $ echo "[global]\nextra-index-url = https://pypi.example.com/" > myvenv/pip.conf
> >>    $ myvenv/bin/pip install foobar
> >> 
> >> **User:**
> >> 
> >> ::
> >> 
> >>    $ echo "[global]\nextra-index-url = https://pypi.example.com/" >~/.pip/pip.conf
> >>    $ pip install foobar
> >> 
> >> 
> >> External Links on the Simple Installer API
> >> ------------------------------------------
> >> 
> >> PEP 438 proposed a system of classifying file links as either internal,
> >> external, or unsafe. It recommended that by default only internal links would
> >> be installed by an installer however users could opt into external links on
> >> either a global or a per package basis. Additionally they could also opt into
> >> unsafe links on a per package basis.
> >> 
> >> This system has turned out to be *extremely* unfriendly towards the end users
> >> and it is the position of this PEP that the situation has become untenable. The
> >> situation as provided by PEP 438 requires an end user to be aware not only of
> >> the difference between internal, external, and unsafe, but also to be aware of
> >> what hosting mode the package they are trying to install is in, what links are
> >> available on that project's /simple/ page, whether or not those links have
> >> a properly formatted hash fragment, and what links are available from pages
> >> linked to from that project's /simple/ page.
> >> 
> >> There are a number of common confusion/pain points with this system that I
> >> have witnessed:
> >> 
> >> * Users unaware what the simple installer api is at all or how an installer
> >>  locates installable files.
> >> * Users unaware that even if the simple api links to a file, if it does
> >>  not include a ``#md5=...`` fragment that it will be counted as unsafe.
> >> * Users unaware that an installer can look at pages linked from the
> >>  simple api to determine additional links, or that any links found in this
> >>  fashion are considered unsafe.
> >> * Users are unaware and often surprised that PyPI supports hosting your files
> >>  someplace other than PyPI at all.
> >> 
> >> In addition to that, the information that an installer is able to provide
> >> when an installation fails is pretty minimal. We are able to detect if there
> >> are externally hosted files directly linked from the simple installer api,
> >> however we cannot detect if there are files hosted on a linked page without
> >> fetching that page and doing so would cause a massive performance hit just to
> >> see if there might be a file there so that a better error message could be
> >> provided.
> >> 
> >> Finally very few projects have properly linked to their external files so that
> >> they can be safely downloaded and verified. At the time of this writing there
> >> are a total of 65 projects which have files that are only available externally
> >> and are safely hosted.
> >> 
> >> The end result of all of this, is that with PEP 438, when a user attempts to
> >> install a file that is not hosted on PyPI typically the steps they follow are:
> >> 
> >> 1. First, they attempt to install it normally, using ``pip install foobar``.
> >>   This fails because the file is not hosted on PyPI and PEP 438 has us default
> >>   to only hosted on PyPI. If pip detected any externally hosted files or other
> >>   pages that we *could* have attempted to find other files at it will give an
> >>   error message suggesting that they try ``--allow-external foobar``.
> >> 2. They then attempt to install their package using
> >>   ``pip install --allow-external foobar foobar``. If they are lucky foobar is
> >>   one of the packages which is hosted externally and safely and this will
> >>   succeed. If they are unlucky they will get a different error message
> >>   suggesting that they *also* try ``--allow-unverified foobar``.
> >> 3. They then attempt to install their package using
> >>   ``pip install --allow-external foobar --allow-unverified foobar foobar``
> >>   and this finally works.
> >> 
> >> This is the same basic steps that practically everyone goes through every time
> >> they try to install something that is not hosted on PyPI. If they are lucky it'll
> >> only take them two steps, but typically it requires three steps. Worse there is
> >> no real indication to these people why one package might install after two
> >> but most require three. Even worse than that most of them will never get an
> >> externally hosted package that does not take three steps, so they will be
> >> increasingly annoyed and frustrated at the intermediate step and will likely
> >> eventually just start skipping it.
> >> 
> >> 
> >> External Index Discovery
> >> ========================
> >> 
> >> One of the problems with using an additional index is one of discovery. Users
> >> will not generally be aware that an additional index is required at all much
> >> less where that index can be found. Projects can attempt to convey this
> >> information using their description on the PyPI page however that excludes
> >> people who discover their project organically through ``pip search``.
> >> 
> >> To support projects that wish to externally host their files and to enable
> >> users to easily discover what additional indexes are required, PyPI will gain
> >> the ability for projects to register external index URLs and additionally an
> >> associated comment for each. These URLs will be made available on the simple
> >> page however they will not be linked or provided in a form that older
> >> installers will automatically search them.
> >> 
> >> When an installer fetches the simple page for a project, if it finds this
> >> additional meta-data and it cannot find any files for that project in it's
> >> configured URLs then it should use this data to tell the user how to add one
> >> or more of the additional URLs to search in. This message should include any
> >> comments that the project has included to enable them to communicate to the
> >> user and provide hints as to which URL they might want if some are only
> >> useful or compatible with certain platforms or situations. When the installer
> >> has implemented the auto discovery mechanisms they should also deprecate any
> >> of the mechanisms added for PEP 438 (such as ``--allow-external``) for removal
> >> at the end of the deprecation period proposed by the PEP.
> >> 
> >> This feature *must* be added to PyPI prior to starting the deprecation and
> >> removal process for link spidering.
> >> 
> >> 
> >> Deprecation and Removal of Link Spidering
> >> =========================================
> >> 
> >> A new hosting mode will be added to PyPI. This hosting mode will be called
> >> ``pypi-only`` and will be in addition to the three that PEP 438 has already
> >> given us which are ``pypi-explicit``, ``pypi-scrape``, ``pypi-scrape-crawl``.
> >> This new hosting mode will modify a project's simple api page so that it only
> >> lists the files which are directly hosted on PyPI and will not link to anything
> >> else.
> >> 
> >> Upon acceptance of this PEP and the addition of the ``pypi-only`` mode, all new
> >> projects will by defaulted to the PyPI only mode and they will be locked to
> >> this mode and unable to change this particular setting. ``pypi-only`` projects
> >> will still be able to register external index URLs as described above - the
> >> "pypi-only" refers only to the download links that are published directly on
> >> PyPI.
> >> 
> >> An email will then be sent out to all of the projects which are hosted only on
> >> PyPI informing them that in one month their project will be automatically
> >> converted to the ``pypi-only`` mode. A month after these emails have been sent
> >> any of those projects which were emailed, which still are hosted only on PyPI
> >> will have their mode set to ``pypi-only``.
> >> 
> >> After that switch, an email will be sent to projects which rely on hosting
> >> external to PyPI. This email will warn these projects that externally hosted
> >> files have been deprecated on PyPI and that in 6 months from the time of that
> >> email that all external links will be removed from the installer APIs. This
> >> email *must* include instructions for converting their projects to be hosted
> >> on PyPI and *must* include links to a script or package that will enable them
> >> to enter their PyPI credentials and package name and have it automatically
> >> download and re-host all of their files on PyPI. This email *must also*
> >> include instructions for setting up their own index page and registering that
> >> with PyPI, including the fact that they can use pythonhosted.org as a host
> >> for an index page without requiring them to host any additional infrastructure
> >> or purchase a TLS certificate. This email must also contain a link to the Terms
> >> of Service for PyPI as many users may have signed up a long time ago and may
> >> not recall what those terms are.
> >> 
> >> Five months after the initial email, another email must be sent to any projects
> >> still relying on external hosting. This email will include all of the same
> >> information that the first email contained, except that the removal date will
> >> be one month away instead of six.
> >> 
> >> Finally a month later all projects will be switched to the ``pypi-only`` mode
> >> and PyPI will be modified to remove the externally linked files functionality.
> >> At this point in time any installers should finally remove any of the
> >> deprecated PEP 438 functionality such as ``--allow-external`` and
> >> ``--allow-unverified`` in pip.
> >> 
> >> 
> >> PIL
> >> ---
> >> 
> >> It's obvious from the numbers below that the vast bulk of the impact come from
> >> the PIL project. On 2014-05-17 an email was sent to the contact for PIL
> >> inquiring whether or not they would be willing to upload to PyPI. A response
> >> has not been received as of yet (2014-06-05) nor has any change in the hosting
> >> happened. Due to the popularity of PIL this PEP also proposes that during the
> >> deprecation period that PyPI Administrators will set the PIL download URL as
> >> the external index for that project. Allowing the users of PIL to take
> >> advantage of the auto discovery mechanisms although the project has seemingly
> >> become unmaintained.
> >> 
> >> 
> >> Impact
> >> ======
> >> 
> >> The largest impact of this is going to be projects where the maintainers are
> >> no longer maintaining the project, for one reason or another. For these
> >> projects it's unlikely that a maintainer will arrive to set the external index
> >> metadata which would allow the auto discovery mechanism to find it.
> >> 
> >> Looking at the numbers factoring out PIL (which has been special cased above)
> >> the actual impact should be quite low, with it affecting just 6.9% of projects
> >> which host only externally or 2.8% which have their latest version hosted
> >> externally. This represents a mere 3883 unique IP addresses. The break down of
> >> this is that of those 3883 addresses, 100% of them installed something that
> >> could not be verified while only 3% installed something which could be.
> >> 
> >> 
> >> Projects Which Rely on Externally Hosted files
> >> ----------------------------------------------
> >> 
> >> This is determined by crawling the simple index and looking for installable
> >> files using a similar detection method as pip and setuptools use. The "latest"
> >> version is determined using ``pkg_resources.parse_version`` sort order and it
> >> is used to show whether or not the latest version is hosted externally or only
> >> old versions are.
> >> 
> >> ============ ======= ================ =================== =======
> >> \             PyPI    External (old)   External (latest)   Total
> >> ============ ======= ================ =================== =======
> >> **Safe**     38716   31               35                  38782
> >> **Unsafe**   0       1659             1169                2828
> >> **Total**    38716   1690             1204                41610
> >> ============ ======= ================ =================== =======
> >> 
> >> 
> >> Top Externally Hosted Projects by Requests
> >> ------------------------------------------
> >> 
> >> This is determined by looking at the number of requests the
> >> ``/simple/<project>/`` page had gotten in a single day. The total number of
> >> requests during that day was 17,960,467.
> >> 
> >> ============================== ========
> >> Project                        Requests
> >> ============================== ========
> >> PIL                            13470
> >> mysql-connector-python         321
> >> salesforce-python-toolkit      54
> >> pyodbc                         50
> >> elementtree                    44
> >> atfork                         39
> >> RBTools                        29
> >> django-contrib-requestprovider 28
> >> wadofstuff-django-serializers  23
> >> Pygame                         21
> >> ============================== ========
> >> 
> >> 
> >> Top Externally Hosted Projects by Unique IPs
> >> --------------------------------------------
> >> 
> >> This is determined by looking at the IP addresses of requests the
> >> ``/simple/<project>/`` page had gotten in a single day. The total number of
> >> unique IP addresses during that day was 105,587.
> >> 
> >> ============================== ==========
> >> Project                        Unique IPs
> >> ============================== ==========
> >> PIL                            3515
> >> mysql-connector-python         117
> >> pyodbc                         34
> >> elementtree                    21
> >> RBTools                        19
> >> egenix-mx-base                 16
> >> Pygame                         14
> >> salesforce-python-toolkit      13
> >> django-contrib-requestprovider 12
> >> wxPython                       11
> >> python-apt                     10
> >> ============================== ==========
> >> 
> >> 
> >> Rejected Proposals
> >> ==================
> >> 
> >> Keep the current classification system but adjust the options
> >> -------------------------------------------------------------
> >> 
> >> This PEP rejects several related proposals which attempt to fix some of the
> >> usability problems with the current system but while still keeping the
> >> general gist of PEP 438.
> >> 
> >> This includes:
> >> 
> >> * Default to allowing safely externally hosted files, but disallow unsafely
> >>  hosted.
> >> * Default to disallowing safely externally hosted files with only a global
> >>  flag to enable them, but disallow unsafely hosted.
> >> 
> >> These proposals are rejected because:
> >> 
> >> * The classification "system" is complex, hard to explain, and requires an
> >>  intimate knowledge of how the simple API works in order to be able to reason
> >>  about which classification is required. This is reflected in the fact that
> >>  the code to implement it is complicated and hard to understand as well.
> >> 
> >> * People are generally surprised that PyPI allows externally linking to files
> >>  and doesn't require people to host on PyPI. In contrast most of them are
> >>  familiar with the concept of multiple software repositories such as is in
> >>  use by many OSs.
> >> 
> >> * PyPI is fronted by a globally distributed CDN which has improved the
> >>  reliability and speed for end users. It is unlikely that any particular
> >>  external host has something comparable. This can lead to extremely bad
> >>  performance for end users when the external host is located in different
> >>  parts of the world or does not generally have good connectivity.
> >> 
> >>  As a data point, many users reported sub DSL speeds and latency when
> >>  accessing PyPI from parts of Europe and Asia prior to the use of the CDN.
> >> 
> >> * PyPI has monitoring and an on-call rotation of sysadmins whom can respond to
> >>  downtime quickly, thus enabling a quicker response to downtime. Again it is
> >>  unlikely that any particular external host will have this. This can lead
> >>  to single packages in a dependency chain being un-installable. This will
> >>  often confuse users, who often times have no idea that this package relies
> >>  on an external host, and they cannot figure out why PyPI appears to be up
> >>  but the installer cannot find a package.
> >> 
> >> * PyPI supports mirroring, both for private organizations and public mirrors.
> >>  The legal terms of uploading to PyPI ensure that mirror operators, both
> >>  public and private, have the right to distribute the software found on PyPI.
> >>  However software that is hosted externally does not have this, causing
> >>  private organizations to need to investigate each package individually and
> >>  manually to determine if the license allows them to mirror it.
> >> 
> >>  For public mirrors this essentially means that these externally hosted
> >>  packages *cannot* be reasonably mirrored. This is particularly troublesome
> >>  in countries such as China where the bandwidth to outside of China is
> >>  highly congested making a mirror within China often times a massively better
> >>  experience.
> >> 
> >> * Installers have no method to determine if they should expect any particular
> >>  URL to be available or not. It is not unusual for the simple API to reference
> >>  old packages and URLs which have long since stopped working. This causes
> >>  installers to have to assume that it is OK for any particular URL to not be
> >>  accessible. This causes problems where an URL is temporarily down or
> >>  otherwise unavailable (a common cause of this is using a copy of Python
> >>  linked against a really ancient copy of OpenSSL which is unable to verify
> >>  the SSL certificate on PyPI) but it *should* be expected to be up. In this
> >>  case installers will typically silently ignore this URL and later the user
> >>  will get a confusing error stating that the installer couldn't find any
> >>  versions instead of getting the real error message indicating that the URL
> >>  was unavailable.
> >> 
> >> * In the long run, global opt in flags like ``--allow-all-external`` will
> >>  become little annoyances that developers cargo cult around in order to make
> >>  their installer work. When they run into a project that requires it they
> >>  will most likely simply add it to their configuration file for that installer
> >>  and continue on with whatever they were actually trying to do. This will
> >>  continue until they try to install their requirements on another computer
> >>  or attempt to deploy to a server where their install will fail again until
> >>  they add the "make it work" flag in their configuration file.
> >> 
> >> 
> >> -----------------
> >> Donald Stufft
> >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> >> 
> > 
> > 
> > 
> >> _______________________________________________
> >> Distutils-SIG maillist  -  Distutils-SIG at python.org
> >> https://mail.python.org/mailman/listinfo/distutils-sig
> > 
> 
> 
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 



From donald at stufft.io  Fri Jun  6 16:25:57 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 6 Jun 2014 10:25:57 -0400
Subject: [Distutils] PEP 470 Round 2 - Using Multi Index Support for
	External to PyPI Package File Hosting
In-Reply-To: <20140606134127.GK30349@merlinux.eu>
References: <0D7BEAFC-7987-46C8-8A9A-7931BE4DA36B@stufft.io>
 <20140606081348.GG30349@merlinux.eu>
 <18F8F7BC-11E0-462F-AAB8-635114DEF05C@stufft.io>
 <20140606134127.GK30349@merlinux.eu>
Message-ID: <9BCE568E-665E-466C-9C5B-1EF80F6C2728@stufft.io>


On Jun 6, 2014, at 9:41 AM, holger krekel <holger at merlinux.eu> wrote:

> On Fri, Jun 06, 2014 at 07:55 -0400, Donald Stufft wrote:
>> 
>> On Jun 6, 2014, at 4:13 AM, holger krekel <holger at merlinux.eu> wrote:
>> 
>>> Hi Donald,
>>> 
>>> 1. you published numbers where <4K or <300 discounting PIL would be
>>>  affected by PEP470.  You also say that the main reason for deprecating
>>>  PEP438 is that it confused users.  Did it confuse other users than those few?
>> 
>> It confused more of than the current numbers because at the onset more
>> projects relied on it than does now. Currently PIL is the primary
>> instigator for people?s confusion that I personally see.
> 
> So currently we don't have many confused users anymore.  Doesn't
> this take away a good part of the reasoning behind PEP470?

No.

> 
> In the following i use "PEP438f" to speak about a hypothetical
> follow-up PEP as outlined in my previous mail.  I volunteer to write
> it and present it as an alternative should we not reach some 
> form of conclusion together.
> 
>>> 2. I don't see a valid precise reasoning why PEP438, just agreed on and 
>>>  implemented last year, needs deprecation.  It boosted everyone
>>>  everyone's install experiences (independently from the CDN which
>>>  brought another boost) as usage of crawling dramatically dropped 
>>>  and thus brings us into the exact situation PEP438 already hinted at:
>>> 
>>>  "Deprecation of hosting modes to eventually only allow the
>>>  pypi-explicit mode is NOT REGULATED by this PEP but is expected to
>>>  become feasible some time after successful implementation of the
>>>  transition phases described in this PEP. It is expected that
>>>  deprecation requires a new process to deal with abandoned packages
>>>  because of unreachable maintainers for still popular packages."
>>> 
>>>  We should follow through and discuss removing crawling and 
>>>  how to deal with abandoned packages.  On the PyPI side, what 
>>>  would remain are two kind of links:
>>> 
>>>  - pypi internally hosted
>>>  - registered safe external links to release files
>>> 
>>>  The resulting situation is:
>>> 
>>>  easy: users have an already existing option to consider to allow externals.
>>> 
>>>  safe: All links served from pypi have checksums. Project maintainers need
>>>        to register hashed links to their new release files.
>>> 
>>>  clean: Pip could eventually remove support for crawling/related options.
>>> 
>>>  This is all easy to do, reduces user confusion and makes pip
>>>  and pypi simpler and less suprising.
>>> 
>>>  I don't see this approach discussed or seriously considered in the PEP,
>>>  also not in its "rejection reasons?.
>> 
>> The reasons are listed in the PEP, though I can make it more explicit that
>> it is for this as well.
>> 
>> * People are generally surprised that PyPI allows externally linking to files
>>  and doesn't require people to host on PyPI. In contrast most of them are
>>  familiar with the concept of multiple software repositories such as is in
>>  use by many OSs.
> 
> "People are generally surprised" is a rather subjective statement.
> Wrt to PEP470 we might have at least 65 projects and many more users being 
> annoyed rather than just surprised at the sudden change in direction.
> Especially if there are no compelling arguments.
> 
>> * PyPI is fronted by a globally distributed CDN which has improved the
>>  reliability and speed for end users. It is unlikely that any particular
>>  external host has something comparable. This can lead to extremely bad
>>  performance for end users when the external host is located in different
>>  parts of the world or does not generally have good connectivity.
>> 
>>  As a data point, many users reported sub DSL speeds and latency when
>>  accessing PyPI from parts of Europe and Asia prior to the use of the CDN.
>> 
>> * PyPI has monitoring and an on-call rotation of sysadmins whom can respond to
>>  downtime quickly, thus enabling a quicker response to downtime. Again it is
>>  unlikely that any particular external host will have this. This can lead
>>  to single packages in a dependency chain being un-installable. This will
>>  often confuse users, who often times have no idea that this package relies
>>  on an external host, and they cannot figure out why PyPI appears to be up
>>  but the installer cannot find a package.
> 
> Sorry but both points have not much to do with the discussion.  If
> anything, they speak *against* PEP470 because users would need to rely
> on project specific external index sites to even know which releases
> exist.  With PEP438 you know that a certain release file must exist and
> the installer clearly says "i could not download release file X from
> URL".  Works today.
> 
> Also the external index could be temporarily broken and serve not the newest
> files.  The integrity and reliability of external indexes would generally
> not be covered by the CDN and PyPI's on-rotation admins so instead of
> speaking for PEP470 they speak against it.

The point is, end users are *aware* they are relying on something external
and they are aware exactly what external items they are relying on. With PEP 470
people can correctly assume that a pip install is going to be covered by the CDN
and our sysadmins unless they?ve gone out of their way to add an additional index.

Part of this, is that I consider ?allow-all-external to be a UX failure. I originally thought
it would be alright, but given actual experience with it in the real world, I see constant
confusion about what it does.

> 
>> * PyPI supports mirroring, both for private organizations and public mirrors.
>>  The legal terms of uploading to PyPI ensure that mirror operators, both
>>  public and private, have the right to distribute the software found on PyPI.
>>  However software that is hosted externally does not have this, causing
>>  private organizations to need to investigate each package individually and
>>  manually to determine if the license allows them to mirror it.
>> 
>>  For public mirrors this essentially means that these externally hosted
>>  packages *cannot* be reasonably mirrored. This is particularly troublesome
>>  in countries such as China where the bandwidth to outside of China is
>>  highly congested making a mirror within China often times a massively better
>>  experience.
> 
> With PEP438 today, PyPI merely hosts a link with a checksum, not the file
> itself.  Mirrors also just mirror that link but not the file.
> How exactly is managing and mirroring a link a problem here?

The problem isn?t in the link, it?s in that you can?t automatically mirror the file,
which means that the primary way for an individual or organization to take
control over their own uptime is hampered by the fact that they cannot rely
on the PyPI ToS to allow them to mirror the file.

> 
>> * In the long run, global opt in flags like ``--allow-all-external`` will
>>  become little annoyances that developers cargo cult around in order to make
>>  their installer work. When they run into a project that requires it they
>>  will most likely simply add it to their configuration file for that installer
>>  and continue on with whatever they were actually trying to do. This will
>>  continue until they try to install their requirements on another computer
>>  or attempt to deploy to a server where their install will fail again until
>>  they add the "make it work" flag in their configuration file.
> 
> Well, with PEP470 developers would have to cargo cult around one or 
> more "--extra-index-url SOME_URL" options.  Also:

Ah, but you see the nature of ``?extra-index-url`` and ``?find-links`` is that
they *feel* more per-projecty. They are more likely to be added to the
requirements file because they are project specific than a global pip
configuration file. This isn?t cargo culting, this is configuring your project. The
problem with ``?allow-all-external`` is that lends itself more to being added
to the configuration file for the user (~/.pip/pip.conf) instead of the per
project requirements file. In one of the earlier discussions at least one person
stated that the first time they came across an externally hosted file they
would be likely to just chuck the ?allow all external? bit into their pip.conf, making
it unlikely that they will even realize if they are depending or using on something
that requires external hosting.

> 
> - if a project wants to later change its index URL, existing users option
>  will break and the project has no way to communicate it.  With PEP438f
>  you can just update your hashes and be done.  

The project absolutely has a way to communicate it, first they can update
their metadata in PyPI which will trigger pip, when it cannot find anything
to satisfy that requirement, to say ?hey you want to use this extra index?.
Secondly this is all HTTP, and HTTP has a very old and very well tested
way to communicate a change in the URL, the 301 redirect.

> 
> - if a project-specific index resides on a domain that changes ownership 
>  (e.g. project abandoned, maintainer gone), user are suddenly vulnerable
>  without knowing it. With PEP438f the trusted PyPI infrastructure
>  provides a checksummed link so new DNS owners can not break into
>  the user's computer as they can do with PEP470.

Sure, but the most likely outcome is that the domain goes missing at first,
and thanks to PEP 470, pip hard fails because it can?t find that index instead
of soft failing and the person removes the offending index and they are
perfectly safe. The window of attack is very small and will get smaller as
we implement other pieces such as package signing.

> 
> - if projects host on HTTP instead of HTTPS, users are 
>  vulnerable against MITM attacks whereas PEP438f provides a checksummed
>  link from PyPI which cannot be easily man-in-the-middled.

Which is why pip warns if you?re using HTTP instead of HTTPS, this will also
end up being mitigated by package signing.

> 
>> Implied but not explicitly called out reason (I?ll add this):
>> 
>> * The URL classification only works for a certain subset of projects, however
>>  it does not allow for any project which needs additional restrictions such
>>  as Access Controls. This means that there would be two methods of doing the
>>  same thing, linking to a file safely and hosting an index. Hosting an index
>>  works in all situations and by relying on this we make for a more consistent
>>  experience no matter the reason for external hosting.
> 
> Once you care for ACLs for indexes and releases you have a number
> of issues to consider, it's hardly related to PEP470/PEP438.

It is related, because it means that the exact same mechanisms can be used,
people don?t have to learn two different ways of specifying externally hosted
projects. In fact it also teaches them how to specify mirrors and the like as well
something that any devpi user is already going to have to learn how to do.

> 
>> Not implied, but I?ll add as well:
>> 
>> * The safe external hosting option hampers the ability of PyPI to upgrade it's
>>  security infrastructure. For instance if MD5 becomes broken in the future
>>  there will be no way for PyPI to upgrade the hashes of the projects which
>>  rely on safe external hosting via MD5 while files that are hosted on PyPI
>>  can simply be processed over with a new hash function.
> 
> This is a good point.  So we should make sure that it's easy to re-upload
> with different hashes and maybe store some more hashes (sha256 etc.) with
> an uploaded file and have pypi decide which one is used.

That assumes the maintainer is maintaining their software anymore.

> 
>> Not going to add:
>> 
>> Ultimately we're looking at maintaining an additional feature, both in PyPI
>> and in the installers, which almost nobody is actually taking advantage of. A
>> year ago I was hopeful that perhaps people would take advantage of it and maybe
>> that would solve some of the issues. However it's now been an entire year and
>> the buy-in for that feature is minuscule. I do not believe that it makes sense
>> to pay the cost for continuing to have that feature.
> 
> PEP438 and most of us urged people to use internal hosting and that's
> what people did.  That we have 65 projects using safe external hosting
> means we succeeded and there is a clear safe path to host your files
> externally.  Why did you expect more people to use external hosting
> when PyPI's current hosting is perceived as good and we told everyone
> to use it?

No, I expected more people to move to safe external vs staying with the unsafe
external. Strictly speaking there are only 35 projects which safely host their
*latest* version. Looking at the latest version is important because it is an
indicator as to what people plan on doing in the future. The 31 projects which
have *old* files hosted appear to all be projects where maybe one or two
releases were hosted externally (or perhaps only 1 file out of the release).

To put it bluntly, even if we call it 65 projects, that is not enough of a buy in
in my opinion to maintain this feature. People were given the chance to use
it, not enough did. It?s time to simplify the options.

> 
>> It costs time and effort to ensure that these features to not break. It also
>> adds to the cognitive burden of using the installers. We require end users to
>> learn two different ways of specifying they wish to allow external indexes
>> and require them to understand when or why they'd use one or the other. It is
>> my opinion that this makes the end user experience wholly worse.
> 
> However, we have a rather settled situation now and can easily 
> clean it up further: crawling can go, pypi-crawl can go, a number 
> of options go.  All easily done with PEP438f.
> 
>>>  By contrast, PEP470 would require many users to learn about
>>>  specifying other indexes and what that means.  For you and me
>>>  and many here on the list it may be a no-brainer but trust me,
>>>  for many users (i've done ten trainings touching the topic now)
>>>  this is not a natural concept at all.  "pip install --allow-all-externals" 
>>>  is far easier to convey than specifying extra per-project indexes and 
>>>  what it means if the install fails (wrong URL? Index noch reachable?
>>>  Release file not found?).
>> 
>> Some projects host their files on PyPI, some files do not, if the thing you're
>> trying to install doesn't host on PyPI you'll have to tell pip where it's
>> hosted. I don't really believe this is a difficult concept.
> 
> Well, with PEP470 you need to tell and explain users:
> "--extra-index-url <some_url_you_need_to_find_out>" versus
> "--allow-all-externals" with PEP438f.
> 
> I am pretty sure the second would win most usability contests.

I disagree. ?allow-all-external is bad as I mentioned above, it leads
people to chuck it in their user config and forget about it whereas
?extra-index-url leads people to include it in their requirements.txt.

Additionally it?s not ?some url you need to find out? it?s ?some url
that we?re going to tell you?. This was detailed in the PEP, projects
would register what that URL is and then pip can present an error
message like:

Files for foo are located on an another index, add ?external-index-url https://example.com/index/
to locate them.

This will work massively better than the per project option from PEP 438
as well. People get confused and don?t realize that they need to specify
the thing they are trying to install twice (pip install ?allow-external foo foo).

> 
>>> 
>>> 3. PEP470 makes life a lot harder for devpi-server, currently used
>>>  by many companies for serving their private indexes.  With PEP438 and
>>>  almost no external crawling left, devpi-server can rely on seeing
>>>  changes through the PEP381 API.  By contrast, with projects hosted on
>>>  additional per-project external indexes, it requires polling to see
>>>  changes because releases may not be registered with PyPI anymore (and
>>>  there is no way to enforce that IISIC).  IOW, PEP470 is a serious
>>>  regression here as it doesn't allow getting notified on new release files.
>> 
>> That only works for the set of projects which are currently safely externally
>> hosted, which again is tiny.
> 
> Well, but it works.  With PEP470 it would not work anymore.  That was my point.
> And the question is anyway not how many projects but how many users are using
> externally hosted files.

And the answer is, barely any. Roughly 116 unique IP addresses on the day I examined
which represents 0.1% of the total number of unique IP addresses on that day. To be quite
honest it?s more likely that those projects were grabbed by something automated that
was canning the whole thing rather than an end user actually trying to install them. But
even if it is the case that they were end users, 0.1% is not enough of a buy in to justify
this special case to exist. We have a mechanism for specifying external locations to
fetch from, it?s existed for a long time, we don?t need a special one for an ultra minority.

> 
>> It does not work at all for projects which are
>> hosted externally nor does it work for projects which require some sort of
>> ACL.
> 
> I don't quite understand why you are talking about ACLs in context of our
> discussion.

As I said above, because it matters, the less concepts people have to understand
the easier the tooling is to use. If there are two different ways to host externally
then that?s two different concepts users have to learn.

> 
>> Additionally I don't believe devpi *can* actually do that, as it has no way
>> of knowing if it's even legal for it to be mirroring the files if they are
>> not hosted on PyPI. This is one of the points of the PEP, there is a known
>> legal right to distribute files hosted on PyPI, no such right is promised for
>> any file not hosted on PyPI. It's completely possible that devpi is doing
>> something that it has no legal right to do in those cases.
> 
> If I use devpi on my laptop or in an organisation I am using devpi as a
> private http cache and not serving my cached files to 3rd parties.

This would depend greatly upon the license of the theoretical library. IANAL
but it is my understanding that it?s completely possible and even reasonable
for there to exist licenses where this is not allowed. The point being that
by blurring the lines it makes it more difficult for people who have to make sure
they are following the license terms because they have something to lose if
they don?t whereas providing clear separation makes it so much easier.

> 
> best,
> holger
> 
>>> 
>>> best,
>>> holger
>>> 
>>> On Thu, Jun 05, 2014 at 22:08 -0400, Donald Stufft wrote:
>>>> Here's round 2 of PEP 470.
>>>> 
>>>> You can see it online at https://python.org/dev/peps/pep-0470/ or below.
>>>> 
>>>> Notable changes:
>>>> 
>>>> - Ensure it's obvious this strictly deals with the installer API and does not
>>>> affect a project's ability to register their project on PyPI for human
>>>> consumptions.
>>>> 
>>>> - Mention that the functional mechanisms that make it possible for an end user
>>>> to specify the additional locations have existed for a long time across many
>>>> versions of the installers.
>>>> 
>>>> - Explicitly mention that the installer changes from PEP 438 should be
>>>> deprecated and removed as part of this PEP.
>>>> 
>>>> - Explicitly mention pythonhosted.org as a location that authors can use to
>>>> host an index if they do not wish to purchase a TLS certificate or host
>>>> additional infrastructure.
>>>> 
>>>> - Include that a link to PyPI ToS should be included in the emails sent to
>>>> authors to remind them of the PyPI ToS.
>>>> 
>>>> - Special case PIL as it is an outlier in terms of impact.
>>>> 
>>>> - Fill out the impact sections further to provide more detail
>>>> 
>>>> 
>>>> Abstract
>>>> ========
>>>> 
>>>> This PEP proposes that the official means of having an installer locate and
>>>> find package files which are hosted externally to PyPI become the use of
>>>> multi index support instead of the practice of using external links on the
>>>> simple installer API.
>>>> 
>>>> It is important to remember that this is **not** about forcing anyone to host
>>>> their files on PyPI. If someone does not wish to do so they will never be under
>>>> any obligation too. They can still list their project in PyPI as an index, and
>>>> the tooling will still allow them to host it elsewhere.
>>>> 
>>>> This PEP strictly is concerned with the Simple Installer API and how automated
>>>> installers interact with PyPI, it has no bearing on the informational pages
>>>> which are primarily for human consumption.
>>>> 
>>>> 
>>>> Rationale
>>>> =========
>>>> 
>>>> There is a long history documented in PEP 438 that explains why externally
>>>> hosted files exist today in the state that they do on PyPI. For the sake of
>>>> brevity I will not duplicate that and instead urge readers to first take a look
>>>> at PEP 438 for background.
>>>> 
>>>> There are currently two primary ways for a project to make itself available
>>>> without directly hosting the package files on PyPI. They can either include
>>>> links to the package files in the simpler installer API or they can publish
>>>> a custom package index which contains their project.
>>>> 
>>>> 
>>>> Custom Additional Index
>>>> -----------------------
>>>> 
>>>> Each installer which speaks to PyPI offers a mechanism for the user invoking
>>>> that installer to provide additional custom locations to search for files
>>>> during the dependency resolution phase. For pip these locations can be
>>>> configured per invocation, per shell environment, per requirements file, per
>>>> virtual environment, and per user. The mechanism for specifying additional
>>>> locations have existed within pip and setuptools for many years, by comparison
>>>> the mechanisms in PEP 438 and any other new mechanism will have existed for
>>>> only a short period of time (if they exist at all currently).
>>>> 
>>>> The use of additional indexes instead of external links on the simple
>>>> installer API provides a simple clean interface which is consistent with the
>>>> way most Linux package systems work (apt-get, yum, etc). More importantly it
>>>> works the same even for projects which are commercial or otherwise have their
>>>> access restricted in some form (private networks, password, IP ACLs etc)
>>>> while the external links method only realistically works for projects which
>>>> do not have their access restricted.
>>>> 
>>>> Compared to the complex rules which a project must be aware of to prevent
>>>> themselves from being considered unsafely hosted setting up an index is fairly
>>>> trivial and in the simplest case does not require anything more than a
>>>> filesystem and a standard web server such as Nginx or Twisted Web. Even if
>>>> using simple static hosting without autoindexing support, it is still
>>>> straightforward to generate appropriate index pages as static HTML.
>>>> 
>>>> Example Index with Twisted Web
>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>> 
>>>> 1. Create a root directory for your index, for the purposes of the example
>>>>  I'll assume you've chosen ``/var/www/index.example.com/``.
>>>> 2. Inside of this root directory, create a directory for each project such
>>>>  as ``mkdir -p /var/www/index.example.com/{foo,bar,other}/``.
>>>> 3. Place the package files for each project in their respective folder,
>>>>  creating paths like ``/var/www/index.example.com/foo/foo-1.0.tar.gz``.
>>>> 4. Configure Twisted Web to serve the root directory, ideally with TLS.
>>>> 
>>>> ::
>>>> 
>>>>   $ twistd -n web --path /var/www/index.example.com/
>>>> 
>>>> 
>>>> Examples of Additional indexes with pip
>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>> 
>>>> **Invocation:**
>>>> 
>>>> ::
>>>> 
>>>>   $ pip install --extra-index-url https://pypi.example.com/ foobar
>>>> 
>>>> **Shell Environment:**
>>>> 
>>>> ::
>>>> 
>>>>   $ export PIP_EXTRA_INDEX_URL=https://pypi.example.com/
>>>>   $ pip install foobar
>>>> 
>>>> **Requirements File:**
>>>> 
>>>> ::
>>>> 
>>>>   $ echo "--extra-index-url https://pypi.example.com/\nfoobar" > requirements.txt
>>>>   $ pip install -r requirements.txt
>>>> 
>>>> **Virtual Environment:**
>>>> 
>>>> ::
>>>> 
>>>>   $ python -m venv myvenv
>>>>   $ echo "[global]\nextra-index-url = https://pypi.example.com/" > myvenv/pip.conf
>>>>   $ myvenv/bin/pip install foobar
>>>> 
>>>> **User:**
>>>> 
>>>> ::
>>>> 
>>>>   $ echo "[global]\nextra-index-url = https://pypi.example.com/" >~/.pip/pip.conf
>>>>   $ pip install foobar
>>>> 
>>>> 
>>>> External Links on the Simple Installer API
>>>> ------------------------------------------
>>>> 
>>>> PEP 438 proposed a system of classifying file links as either internal,
>>>> external, or unsafe. It recommended that by default only internal links would
>>>> be installed by an installer however users could opt into external links on
>>>> either a global or a per package basis. Additionally they could also opt into
>>>> unsafe links on a per package basis.
>>>> 
>>>> This system has turned out to be *extremely* unfriendly towards the end users
>>>> and it is the position of this PEP that the situation has become untenable. The
>>>> situation as provided by PEP 438 requires an end user to be aware not only of
>>>> the difference between internal, external, and unsafe, but also to be aware of
>>>> what hosting mode the package they are trying to install is in, what links are
>>>> available on that project's /simple/ page, whether or not those links have
>>>> a properly formatted hash fragment, and what links are available from pages
>>>> linked to from that project's /simple/ page.
>>>> 
>>>> There are a number of common confusion/pain points with this system that I
>>>> have witnessed:
>>>> 
>>>> * Users unaware what the simple installer api is at all or how an installer
>>>> locates installable files.
>>>> * Users unaware that even if the simple api links to a file, if it does
>>>> not include a ``#md5=...`` fragment that it will be counted as unsafe.
>>>> * Users unaware that an installer can look at pages linked from the
>>>> simple api to determine additional links, or that any links found in this
>>>> fashion are considered unsafe.
>>>> * Users are unaware and often surprised that PyPI supports hosting your files
>>>> someplace other than PyPI at all.
>>>> 
>>>> In addition to that, the information that an installer is able to provide
>>>> when an installation fails is pretty minimal. We are able to detect if there
>>>> are externally hosted files directly linked from the simple installer api,
>>>> however we cannot detect if there are files hosted on a linked page without
>>>> fetching that page and doing so would cause a massive performance hit just to
>>>> see if there might be a file there so that a better error message could be
>>>> provided.
>>>> 
>>>> Finally very few projects have properly linked to their external files so that
>>>> they can be safely downloaded and verified. At the time of this writing there
>>>> are a total of 65 projects which have files that are only available externally
>>>> and are safely hosted.
>>>> 
>>>> The end result of all of this, is that with PEP 438, when a user attempts to
>>>> install a file that is not hosted on PyPI typically the steps they follow are:
>>>> 
>>>> 1. First, they attempt to install it normally, using ``pip install foobar``.
>>>>  This fails because the file is not hosted on PyPI and PEP 438 has us default
>>>>  to only hosted on PyPI. If pip detected any externally hosted files or other
>>>>  pages that we *could* have attempted to find other files at it will give an
>>>>  error message suggesting that they try ``--allow-external foobar``.
>>>> 2. They then attempt to install their package using
>>>>  ``pip install --allow-external foobar foobar``. If they are lucky foobar is
>>>>  one of the packages which is hosted externally and safely and this will
>>>>  succeed. If they are unlucky they will get a different error message
>>>>  suggesting that they *also* try ``--allow-unverified foobar``.
>>>> 3. They then attempt to install their package using
>>>>  ``pip install --allow-external foobar --allow-unverified foobar foobar``
>>>>  and this finally works.
>>>> 
>>>> This is the same basic steps that practically everyone goes through every time
>>>> they try to install something that is not hosted on PyPI. If they are lucky it'll
>>>> only take them two steps, but typically it requires three steps. Worse there is
>>>> no real indication to these people why one package might install after two
>>>> but most require three. Even worse than that most of them will never get an
>>>> externally hosted package that does not take three steps, so they will be
>>>> increasingly annoyed and frustrated at the intermediate step and will likely
>>>> eventually just start skipping it.
>>>> 
>>>> 
>>>> External Index Discovery
>>>> ========================
>>>> 
>>>> One of the problems with using an additional index is one of discovery. Users
>>>> will not generally be aware that an additional index is required at all much
>>>> less where that index can be found. Projects can attempt to convey this
>>>> information using their description on the PyPI page however that excludes
>>>> people who discover their project organically through ``pip search``.
>>>> 
>>>> To support projects that wish to externally host their files and to enable
>>>> users to easily discover what additional indexes are required, PyPI will gain
>>>> the ability for projects to register external index URLs and additionally an
>>>> associated comment for each. These URLs will be made available on the simple
>>>> page however they will not be linked or provided in a form that older
>>>> installers will automatically search them.
>>>> 
>>>> When an installer fetches the simple page for a project, if it finds this
>>>> additional meta-data and it cannot find any files for that project in it's
>>>> configured URLs then it should use this data to tell the user how to add one
>>>> or more of the additional URLs to search in. This message should include any
>>>> comments that the project has included to enable them to communicate to the
>>>> user and provide hints as to which URL they might want if some are only
>>>> useful or compatible with certain platforms or situations. When the installer
>>>> has implemented the auto discovery mechanisms they should also deprecate any
>>>> of the mechanisms added for PEP 438 (such as ``--allow-external``) for removal
>>>> at the end of the deprecation period proposed by the PEP.
>>>> 
>>>> This feature *must* be added to PyPI prior to starting the deprecation and
>>>> removal process for link spidering.
>>>> 
>>>> 
>>>> Deprecation and Removal of Link Spidering
>>>> =========================================
>>>> 
>>>> A new hosting mode will be added to PyPI. This hosting mode will be called
>>>> ``pypi-only`` and will be in addition to the three that PEP 438 has already
>>>> given us which are ``pypi-explicit``, ``pypi-scrape``, ``pypi-scrape-crawl``.
>>>> This new hosting mode will modify a project's simple api page so that it only
>>>> lists the files which are directly hosted on PyPI and will not link to anything
>>>> else.
>>>> 
>>>> Upon acceptance of this PEP and the addition of the ``pypi-only`` mode, all new
>>>> projects will by defaulted to the PyPI only mode and they will be locked to
>>>> this mode and unable to change this particular setting. ``pypi-only`` projects
>>>> will still be able to register external index URLs as described above - the
>>>> "pypi-only" refers only to the download links that are published directly on
>>>> PyPI.
>>>> 
>>>> An email will then be sent out to all of the projects which are hosted only on
>>>> PyPI informing them that in one month their project will be automatically
>>>> converted to the ``pypi-only`` mode. A month after these emails have been sent
>>>> any of those projects which were emailed, which still are hosted only on PyPI
>>>> will have their mode set to ``pypi-only``.
>>>> 
>>>> After that switch, an email will be sent to projects which rely on hosting
>>>> external to PyPI. This email will warn these projects that externally hosted
>>>> files have been deprecated on PyPI and that in 6 months from the time of that
>>>> email that all external links will be removed from the installer APIs. This
>>>> email *must* include instructions for converting their projects to be hosted
>>>> on PyPI and *must* include links to a script or package that will enable them
>>>> to enter their PyPI credentials and package name and have it automatically
>>>> download and re-host all of their files on PyPI. This email *must also*
>>>> include instructions for setting up their own index page and registering that
>>>> with PyPI, including the fact that they can use pythonhosted.org as a host
>>>> for an index page without requiring them to host any additional infrastructure
>>>> or purchase a TLS certificate. This email must also contain a link to the Terms
>>>> of Service for PyPI as many users may have signed up a long time ago and may
>>>> not recall what those terms are.
>>>> 
>>>> Five months after the initial email, another email must be sent to any projects
>>>> still relying on external hosting. This email will include all of the same
>>>> information that the first email contained, except that the removal date will
>>>> be one month away instead of six.
>>>> 
>>>> Finally a month later all projects will be switched to the ``pypi-only`` mode
>>>> and PyPI will be modified to remove the externally linked files functionality.
>>>> At this point in time any installers should finally remove any of the
>>>> deprecated PEP 438 functionality such as ``--allow-external`` and
>>>> ``--allow-unverified`` in pip.
>>>> 
>>>> 
>>>> PIL
>>>> ---
>>>> 
>>>> It's obvious from the numbers below that the vast bulk of the impact come from
>>>> the PIL project. On 2014-05-17 an email was sent to the contact for PIL
>>>> inquiring whether or not they would be willing to upload to PyPI. A response
>>>> has not been received as of yet (2014-06-05) nor has any change in the hosting
>>>> happened. Due to the popularity of PIL this PEP also proposes that during the
>>>> deprecation period that PyPI Administrators will set the PIL download URL as
>>>> the external index for that project. Allowing the users of PIL to take
>>>> advantage of the auto discovery mechanisms although the project has seemingly
>>>> become unmaintained.
>>>> 
>>>> 
>>>> Impact
>>>> ======
>>>> 
>>>> The largest impact of this is going to be projects where the maintainers are
>>>> no longer maintaining the project, for one reason or another. For these
>>>> projects it's unlikely that a maintainer will arrive to set the external index
>>>> metadata which would allow the auto discovery mechanism to find it.
>>>> 
>>>> Looking at the numbers factoring out PIL (which has been special cased above)
>>>> the actual impact should be quite low, with it affecting just 6.9% of projects
>>>> which host only externally or 2.8% which have their latest version hosted
>>>> externally. This represents a mere 3883 unique IP addresses. The break down of
>>>> this is that of those 3883 addresses, 100% of them installed something that
>>>> could not be verified while only 3% installed something which could be.
>>>> 
>>>> 
>>>> Projects Which Rely on Externally Hosted files
>>>> ----------------------------------------------
>>>> 
>>>> This is determined by crawling the simple index and looking for installable
>>>> files using a similar detection method as pip and setuptools use. The "latest"
>>>> version is determined using ``pkg_resources.parse_version`` sort order and it
>>>> is used to show whether or not the latest version is hosted externally or only
>>>> old versions are.
>>>> 
>>>> ============ ======= ================ =================== =======
>>>> \             PyPI    External (old)   External (latest)   Total
>>>> ============ ======= ================ =================== =======
>>>> **Safe**     38716   31               35                  38782
>>>> **Unsafe**   0       1659             1169                2828
>>>> **Total**    38716   1690             1204                41610
>>>> ============ ======= ================ =================== =======
>>>> 
>>>> 
>>>> Top Externally Hosted Projects by Requests
>>>> ------------------------------------------
>>>> 
>>>> This is determined by looking at the number of requests the
>>>> ``/simple/<project>/`` page had gotten in a single day. The total number of
>>>> requests during that day was 17,960,467.
>>>> 
>>>> ============================== ========
>>>> Project                        Requests
>>>> ============================== ========
>>>> PIL                            13470
>>>> mysql-connector-python         321
>>>> salesforce-python-toolkit      54
>>>> pyodbc                         50
>>>> elementtree                    44
>>>> atfork                         39
>>>> RBTools                        29
>>>> django-contrib-requestprovider 28
>>>> wadofstuff-django-serializers  23
>>>> Pygame                         21
>>>> ============================== ========
>>>> 
>>>> 
>>>> Top Externally Hosted Projects by Unique IPs
>>>> --------------------------------------------
>>>> 
>>>> This is determined by looking at the IP addresses of requests the
>>>> ``/simple/<project>/`` page had gotten in a single day. The total number of
>>>> unique IP addresses during that day was 105,587.
>>>> 
>>>> ============================== ==========
>>>> Project                        Unique IPs
>>>> ============================== ==========
>>>> PIL                            3515
>>>> mysql-connector-python         117
>>>> pyodbc                         34
>>>> elementtree                    21
>>>> RBTools                        19
>>>> egenix-mx-base                 16
>>>> Pygame                         14
>>>> salesforce-python-toolkit      13
>>>> django-contrib-requestprovider 12
>>>> wxPython                       11
>>>> python-apt                     10
>>>> ============================== ==========
>>>> 
>>>> 
>>>> Rejected Proposals
>>>> ==================
>>>> 
>>>> Keep the current classification system but adjust the options
>>>> -------------------------------------------------------------
>>>> 
>>>> This PEP rejects several related proposals which attempt to fix some of the
>>>> usability problems with the current system but while still keeping the
>>>> general gist of PEP 438.
>>>> 
>>>> This includes:
>>>> 
>>>> * Default to allowing safely externally hosted files, but disallow unsafely
>>>> hosted.
>>>> * Default to disallowing safely externally hosted files with only a global
>>>> flag to enable them, but disallow unsafely hosted.
>>>> 
>>>> These proposals are rejected because:
>>>> 
>>>> * The classification "system" is complex, hard to explain, and requires an
>>>> intimate knowledge of how the simple API works in order to be able to reason
>>>> about which classification is required. This is reflected in the fact that
>>>> the code to implement it is complicated and hard to understand as well.
>>>> 
>>>> * People are generally surprised that PyPI allows externally linking to files
>>>> and doesn't require people to host on PyPI. In contrast most of them are
>>>> familiar with the concept of multiple software repositories such as is in
>>>> use by many OSs.
>>>> 
>>>> * PyPI is fronted by a globally distributed CDN which has improved the
>>>> reliability and speed for end users. It is unlikely that any particular
>>>> external host has something comparable. This can lead to extremely bad
>>>> performance for end users when the external host is located in different
>>>> parts of the world or does not generally have good connectivity.
>>>> 
>>>> As a data point, many users reported sub DSL speeds and latency when
>>>> accessing PyPI from parts of Europe and Asia prior to the use of the CDN.
>>>> 
>>>> * PyPI has monitoring and an on-call rotation of sysadmins whom can respond to
>>>> downtime quickly, thus enabling a quicker response to downtime. Again it is
>>>> unlikely that any particular external host will have this. This can lead
>>>> to single packages in a dependency chain being un-installable. This will
>>>> often confuse users, who often times have no idea that this package relies
>>>> on an external host, and they cannot figure out why PyPI appears to be up
>>>> but the installer cannot find a package.
>>>> 
>>>> * PyPI supports mirroring, both for private organizations and public mirrors.
>>>> The legal terms of uploading to PyPI ensure that mirror operators, both
>>>> public and private, have the right to distribute the software found on PyPI.
>>>> However software that is hosted externally does not have this, causing
>>>> private organizations to need to investigate each package individually and
>>>> manually to determine if the license allows them to mirror it.
>>>> 
>>>> For public mirrors this essentially means that these externally hosted
>>>> packages *cannot* be reasonably mirrored. This is particularly troublesome
>>>> in countries such as China where the bandwidth to outside of China is
>>>> highly congested making a mirror within China often times a massively better
>>>> experience.
>>>> 
>>>> * Installers have no method to determine if they should expect any particular
>>>> URL to be available or not. It is not unusual for the simple API to reference
>>>> old packages and URLs which have long since stopped working. This causes
>>>> installers to have to assume that it is OK for any particular URL to not be
>>>> accessible. This causes problems where an URL is temporarily down or
>>>> otherwise unavailable (a common cause of this is using a copy of Python
>>>> linked against a really ancient copy of OpenSSL which is unable to verify
>>>> the SSL certificate on PyPI) but it *should* be expected to be up. In this
>>>> case installers will typically silently ignore this URL and later the user
>>>> will get a confusing error stating that the installer couldn't find any
>>>> versions instead of getting the real error message indicating that the URL
>>>> was unavailable.
>>>> 
>>>> * In the long run, global opt in flags like ``--allow-all-external`` will
>>>> become little annoyances that developers cargo cult around in order to make
>>>> their installer work. When they run into a project that requires it they
>>>> will most likely simply add it to their configuration file for that installer
>>>> and continue on with whatever they were actually trying to do. This will
>>>> continue until they try to install their requirements on another computer
>>>> or attempt to deploy to a server where their install will fail again until
>>>> they add the "make it work" flag in their configuration file.
>>>> 
>>>> 
>>>> -----------------
>>>> Donald Stufft
>>>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>>>> 
>>> 
>>> 
>>> 
>>>> _______________________________________________
>>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>> 
>> 
>> 
>> -----------------
>> Donald Stufft
>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140606/1d29f85f/attachment-0001.sig>

From edbrannin at gmail.com  Fri Jun  6 21:30:38 2014
From: edbrannin at gmail.com (edbrannin at gmail.com)
Date: Fri, 6 Jun 2014 15:30:38 -0400
Subject: [Distutils] "ImportError: No module named mkdocs" but I can import
	it
Message-ID: <CAEWvxjyG8BuqPozP0vFPrnGY6CG1vAK4-1YmS-LA492fNcAOvg@mail.gmail.com>

Hello,

I'm trying to switch an opensource project from setup(scripts=) to
setup(entry_points=dict(console_scripts=)) to enable Windows support for
its executable script.

If you are feeling generous, you can download my changes and attempt to run
them from this branch:
https://github.com/edbrannin/mkdocs/tree/windows-runnable
Zip version:
https://github.com/edbrannin/mkdocs/archive/windows-runnable.zip

When I run "python setup.py install", it installs the mkdocs module to
lib/site-packages/mkdocs and I can import it from a Python shell, but when
I run the "mkdocs" command I get this error:

C:\>mkdocs Traceback (most recent call last): File
"C:\Python27\Scripts\mkdocs-script.py", line 9, in <module>
load_entry_point('mkdocs==0.9', 'console_scripts', 'mkdocs')() File
"C:\Python27\lib\site-packages\distribute-0.6.34-py2.7.egg\pkg_resources.py",
line 343, in load_entry_point return
get_distribution(dist).load_entry_point(group, name) File
"C:\Python27\lib\site-packages\distribute-0.6.34-py2.7.egg\pkg_resources.py",
line 2307, in load_entry_point return ep.load() File
"C:\Python27\lib\site-packages\distribute-0.6.34-py2.7.egg\pkg_resources.py",
line 2013, in load entry = __import__(self.module_name,
globals(),globals(), ['__name__']) ImportError: No module named mkdocs
There are more details at this StackOverflow question:
http://stackoverflow.com/q/24067318/25625

What am I doing wrong here?

Thank you for reading this far,
-Ed Brannin
edbrannin at gmail.com
cell: 585-261-0279

P.S. For the archives, the exact code revision I'm having trouble with is
at
https://github.com/edbrannin/mkdocs/tree/b924f2e09e511e8bda35465bc0e103a3401c3bf8

P.P.S. I had tried to send this earlier, before I'd subscribed to the list.
 Sorry if both copies of this email eventually get delivered.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140606/1f4c4b0f/attachment.html>

From wichert at wiggy.net  Sat Jun  7 01:31:41 2014
From: wichert at wiggy.net (Wichert Akkerman)
Date: Sat, 7 Jun 2014 01:31:41 +0200
Subject: [Distutils] "ImportError: No module named mkdocs" but I can
	import it
In-Reply-To: <CAEWvxjyG8BuqPozP0vFPrnGY6CG1vAK4-1YmS-LA492fNcAOvg@mail.gmail.com>
References: <CAEWvxjyG8BuqPozP0vFPrnGY6CG1vAK4-1YmS-LA492fNcAOvg@mail.gmail.com>
Message-ID: <7834DDE5-1A8C-4B7E-83F2-3EB673A9255F@wiggy.net>

On 06 Jun 2014, at 21:30, edbrannin at gmail.com wrote:
> When I run "python setup.py install", it installs the mkdocs module to lib/site-packages/mkdocs and I can import it from a Python shell, but when I run the "mkdocs" command I get this error:
> 
> C:\>mkdocs
> Traceback (most recent call last):
>   File "C:\Python27\Scripts\mkdocs-script.py", line 9, in <module>
>     load_entry_point('mkdocs==0.9', 'console_scripts', 'mkdocs')()
>   File "C:\Python27\lib\site-packages\distribute-0.6.34-py2.7.egg\pkg_resources.py", line 343, in load_entry_point
>     return get_distribution(dist).load_entry_point(group, name)
>   File "C:\Python27\lib\site-packages\distribute-0.6.34-py2.7.egg\pkg_resources.py", line 2307, in load_entry_point
>     return ep.load()
>   File "C:\Python27\lib\site-packages\distribute-0.6.34-py2.7.egg\pkg_resources.py", line 2013, in load
>     entry = __import__(self.module_name, globals(),globals(), ['__name__'])
> ImportError: No module named mkdocs
> 
> 
mkdocs/mkdocs is not a .py file. Try renaming that to mkdocs/mkdocs.py . Your import test did not try to import the exact same thing as the console script uses: that uses mkdocs.mkdocs.main_entry_point, and you only tried to import the upper mkdocs package.

Wichert.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140607/dc7f357d/attachment.html>

From wichert at wiggy.net  Sat Jun  7 01:31:41 2014
From: wichert at wiggy.net (Wichert Akkerman)
Date: Sat, 7 Jun 2014 01:31:41 +0200
Subject: [Distutils] "ImportError: No module named mkdocs" but I can
	import it
In-Reply-To: <CAEWvxjyG8BuqPozP0vFPrnGY6CG1vAK4-1YmS-LA492fNcAOvg@mail.gmail.com>
References: <CAEWvxjyG8BuqPozP0vFPrnGY6CG1vAK4-1YmS-LA492fNcAOvg@mail.gmail.com>
Message-ID: <7834DDE5-1A8C-4B7E-83F2-3EB673A9255F@wiggy.net>

On 06 Jun 2014, at 21:30, edbrannin at gmail.com wrote:
> When I run "python setup.py install", it installs the mkdocs module to lib/site-packages/mkdocs and I can import it from a Python shell, but when I run the "mkdocs" command I get this error:
> 
> C:\>mkdocs
> Traceback (most recent call last):
>   File "C:\Python27\Scripts\mkdocs-script.py", line 9, in <module>
>     load_entry_point('mkdocs==0.9', 'console_scripts', 'mkdocs')()
>   File "C:\Python27\lib\site-packages\distribute-0.6.34-py2.7.egg\pkg_resources.py", line 343, in load_entry_point
>     return get_distribution(dist).load_entry_point(group, name)
>   File "C:\Python27\lib\site-packages\distribute-0.6.34-py2.7.egg\pkg_resources.py", line 2307, in load_entry_point
>     return ep.load()
>   File "C:\Python27\lib\site-packages\distribute-0.6.34-py2.7.egg\pkg_resources.py", line 2013, in load
>     entry = __import__(self.module_name, globals(),globals(), ['__name__'])
> ImportError: No module named mkdocs
> 
> 
mkdocs/mkdocs is not a .py file. Try renaming that to mkdocs/mkdocs.py . Your import test did not try to import the exact same thing as the console script uses: that uses mkdocs.mkdocs.main_entry_point, and you only tried to import the upper mkdocs package.

Wichert.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140607/dc7f357d/attachment-0001.html>

From ncoghlan at gmail.com  Sat Jun  7 01:46:08 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 7 Jun 2014 09:46:08 +1000
Subject: [Distutils] PEP 470 Round 2 - Using Multi Index Support for
 External to PyPI Package File Hosting
In-Reply-To: <9BCE568E-665E-466C-9C5B-1EF80F6C2728@stufft.io>
References: <0D7BEAFC-7987-46C8-8A9A-7931BE4DA36B@stufft.io>
 <20140606081348.GG30349@merlinux.eu>
 <18F8F7BC-11E0-462F-AAB8-635114DEF05C@stufft.io>
 <20140606134127.GK30349@merlinux.eu>
 <9BCE568E-665E-466C-9C5B-1EF80F6C2728@stufft.io>
Message-ID: <CADiSq7ftZV6iOxeHhDRBdn-yvt19ZK7bN=UPbZsyOF5PVcPp_A@mail.gmail.com>

On 7 Jun 2014 06:08, "Donald Stufft" <donald at stufft.io> wrote:
>
>
> On Jun 6, 2014, at 9:41 AM, holger krekel <holger at merlinux.eu> wrote:
> >
> > Once you care for ACLs for indexes and releases you have a number
> > of issues to consider, it's hardly related to PEP470/PEP438.
>
> It is related, because it means that the exact same mechanisms can be
used,
> people don?t have to learn two different ways of specifying externally
hosted
> projects. In fact it also teaches them how to specify mirrors and the
like as well
> something that any devpi user is already going to have to learn how to do.

This is the key benefit of PEP 470 from my perspective: some aspects of the
Python packaging ecosystem suffer from a bad case of "too many ways to do
it", and if we're ever going to fix that, we need to be ruthless in culling
redundant concepts.

Specifying custom indexes is a feature with a lot of use cases - local
mirrors and private indexes being two of the big ones. By contrast,
external references from the simple API duplicate a small subset of the
custom index functionality in a way that introduces a whole slew of new
concepts that still need to be documented and learned, even if the advice
is "don't use that, use custom indexes instead".

As far as dev-pi goes, if it's only mirroring links rather than externally
hosted files today, then in the future, it will still automatically mirror
the external index URLs. Dependency update scanners could follow those
links automatically, even if pip install doesn't check them by default.

One other nice consequence of PEP 470 should make it easier for
organisations to flag and investigate cases where they're relying on an
upstream source other than PyPI, regardless of whether they care about the
details of their dependencies' hosting for speed, reliability or legal
reasons.

>From a migration perspective, how hard would it be to automate generation
of a custom index page on pythonhosted.org for projects currently relying
on external references? That would still let us make the client changes
without needing to special case PIL.

Also, it occurred to me that while the latest/any split matters for new
users, we still need to consider the impact on projects which have pinned
dependencies on older versions of packages that were previously externally
hosted, but have moved to PyPI for more recent releases. I still think
dropping the external reference feature from the simple API in favour of
improving the custom index support is the right to do, but a couple of
*client side* examples of handling the migration could help clarify the
consequences for the existing users that may be affected.

For example, perhaps we should keep "--allow-all-external", but have it
mean that pip automatically adds new custom index URLs given for the
requested packages. Even if it emitted a deprecation warning, clients using
it would keep working in the face of the proposed changes to the simple API
link handling.

Regards,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140607/81bcaa4c/attachment-0001.html>

From donald at stufft.io  Sat Jun  7 01:58:51 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 6 Jun 2014 19:58:51 -0400
Subject: [Distutils] PEP 470 Round 2 - Using Multi Index Support for
	External to PyPI Package File Hosting
In-Reply-To: <CADiSq7ftZV6iOxeHhDRBdn-yvt19ZK7bN=UPbZsyOF5PVcPp_A@mail.gmail.com>
References: <0D7BEAFC-7987-46C8-8A9A-7931BE4DA36B@stufft.io>
 <20140606081348.GG30349@merlinux.eu>
 <18F8F7BC-11E0-462F-AAB8-635114DEF05C@stufft.io>
 <20140606134127.GK30349@merlinux.eu>
 <9BCE568E-665E-466C-9C5B-1EF80F6C2728@stufft.io>
 <CADiSq7ftZV6iOxeHhDRBdn-yvt19ZK7bN=UPbZsyOF5PVcPp_A@mail.gmail.com>
Message-ID: <EBCF19CA-C365-41CB-9F78-9BD304ABEA6C@stufft.io>


On Jun 6, 2014, at 7:46 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> 
> On 7 Jun 2014 06:08, "Donald Stufft" <donald at stufft.io> wrote:
> >
> >
> > On Jun 6, 2014, at 9:41 AM, holger krekel <holger at merlinux.eu> wrote:
> > >
> > > Once you care for ACLs for indexes and releases you have a number
> > > of issues to consider, it's hardly related to PEP470/PEP438.
> >
> > It is related, because it means that the exact same mechanisms can be used,
> > people don?t have to learn two different ways of specifying externally hosted
> > projects. In fact it also teaches them how to specify mirrors and the like as well
> > something that any devpi user is already going to have to learn how to do.
> 
> This is the key benefit of PEP 470 from my perspective: some aspects of the Python packaging ecosystem suffer from a bad case of "too many ways to do it", and if we're ever going to fix that, we need to be ruthless in culling redundant concepts.
> 
> Specifying custom indexes is a feature with a lot of use cases - local mirrors and private indexes being two of the big ones. By contrast, external references from the simple API duplicate a small subset of the custom index functionality in a way that introduces a whole slew of new concepts that still need to be documented and learned, even if the advice is "don't use that, use custom indexes instead".
> 
> As far as dev-pi goes, if it's only mirroring links rather than externally hosted files today, then in the future, it will still automatically mirror the external index URLs. Dependency update scanners could follow those links automatically, even if pip install doesn't check them by default.
> 
> One other nice consequence of PEP 470 should make it easier for organisations to flag and investigate cases where they're relying on an upstream source other than PyPI, regardless of whether they care about the details of their dependencies' hosting for speed, reliability or legal reasons.
> 
> From a migration perspective, how hard would it be to automate generation of a custom index page on pythonhosted.org for projects currently relying on external references? That would still let us make the client changes without needing to special case PIL.
> 
> 

Not very difficult. My current crawl script could generate a minimal one with some minor modifications (it?d have to save the whole URL instead of just the filename) and would take about 3 hours to process. This process would also weed out links which have died and the like. Downside would be these files wouldn?t be verifies so it would be external + unsafe index since we don?t have hash information to make them safe. Of course this would be the case for PIL anyways which easily makes up most of this traffic so this could just end up in the wash as far as how ?safe? it is.
> Also, it occurred to me that while the latest/any split matters for new users, we still need to consider the impact on projects which have pinned dependencies on older versions of packages that were previously externally hosted, but have moved to PyPI for more recent releases. I still think dropping the external reference feature from the simple API in favour of improving the custom index support is the right to do, but a couple of *client side* examples of handling the migration could help clarify the consequences for the existing users that may be affected.
> 
> 

Right, this was one of the reasons my old numbers had a split at 50%, part of the idea was that a project with less than some percent of it?s files hosted on PyPI had a smaller ?breakage? surface, even for old pinned versions. I can get these numbers too again if they?d be useful, though I?m not sure if they should go in the PEP or not, it?s already kind of heavy on the numbers I think and I?m not sure additional numbers would be more or less confusing.

What do you mean by client side examples of handling the migration? I?m assuming you mean something other than the examples which show how to utilize the new indexes?
> For example, perhaps we should keep "--allow-all-external", but have it mean that pip automatically adds new custom index URLs given for the requested packages. Even if it emitted a deprecation warning, clients using it would keep working in the face of the proposed changes to the simple API link handling.
> 
Well it?d actually expand what ?allow-all-external means, since it?d also allow those unsafely hosted files. I?m not sure it?d be a good idea to silently (or with a warning even) upgrade an option from a ?do this to allow all safely hosted files? to a ?do this to allow a whole bunch of legacy and unsafely hosted files?. The one upside to that is we?d direct link to files instead of relying on scraping so you?d have to actually rely on an unsafe file to be at risk, but it still makes me nervous.

It?s possible we could add a flag for this, but I?m not sure how useful it?d be since it?d only be in pip 1.6+ and unless people upgrade to that immediately they?ll have to specify the old ?extra-index-url // ?find-links method anyways.

I guess the question would be, how bad would it be to have pip install ?extra-index-url https://legacy-unsafe.pythonhosted.org/ PIL vs pip install ?allow-legacy-unsafe PIL.
> Regards,
> Nick.
> 


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140606/7b76eba4/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140606/7b76eba4/attachment.sig>

From holger at merlinux.eu  Sat Jun  7 18:28:47 2014
From: holger at merlinux.eu (holger krekel)
Date: Sat, 7 Jun 2014 16:28:47 +0000
Subject: [Distutils] PEP 470 Round 2 - Using Multi Index Support for
 External to PyPI Package File Hosting
In-Reply-To: <CADiSq7ftZV6iOxeHhDRBdn-yvt19ZK7bN=UPbZsyOF5PVcPp_A@mail.gmail.com>
References: <0D7BEAFC-7987-46C8-8A9A-7931BE4DA36B@stufft.io>
 <20140606081348.GG30349@merlinux.eu>
 <18F8F7BC-11E0-462F-AAB8-635114DEF05C@stufft.io>
 <20140606134127.GK30349@merlinux.eu>
 <9BCE568E-665E-466C-9C5B-1EF80F6C2728@stufft.io>
 <CADiSq7ftZV6iOxeHhDRBdn-yvt19ZK7bN=UPbZsyOF5PVcPp_A@mail.gmail.com>
Message-ID: <20140607162847.GM30349@merlinux.eu>

On Sat, Jun 07, 2014 at 09:46 +1000, Nick Coghlan wrote:
> On 7 Jun 2014 06:08, "Donald Stufft" <donald at stufft.io> wrote:
> >
> >
> > On Jun 6, 2014, at 9:41 AM, holger krekel <holger at merlinux.eu> wrote:
> > >
> > > Once you care for ACLs for indexes and releases you have a number
> > > of issues to consider, it's hardly related to PEP470/PEP438.
> >
> > It is related, because it means that the exact same mechanisms can be
> used,
> > people don?t have to learn two different ways of specifying externally
> hosted
> > projects. In fact it also teaches them how to specify mirrors and the
> like as well
> > something that any devpi user is already going to have to learn how to do.
> 
> This is the key benefit of PEP 470 from my perspective: some aspects of the
> Python packaging ecosystem suffer from a bad case of "too many ways to do
> it", and if we're ever going to fix that, we need to be ruthless in culling
> redundant concepts.
>
> Specifying custom indexes is a feature with a lot of use cases - local
> mirrors and private indexes being two of the big ones. By contrast,
> external references from the simple API duplicate a small subset of the
> custom index functionality in a way that introduces a whole slew of new
> concepts that still need to be documented and learned, even if the advice
> is "don't use that, use custom indexes instead".

Fair point from a UX design perspective -- trying to minimze the concepts
you have to learn.  However, IMO many python users feel far from needing to
know about configuring indexes with pip.  When they try to install a
project with an external reference they will none-theless with PEP470 need 
to know about indices and according options, failure modes etc.  They
will also usually depend on crawling other index sites every time they
perform an install with these options.  

And i think we all agreed at one point that client-side crawling is not
he greatest thing on earth.  Linux distros have an "update" phase
collecting infos from the repos, and a separate install phase.  So you
don't need to go to the remote sites to get index information at
install-time.  With pip you do it at every install.

And, maybe most importantly, for the integrity of their install they
will depend on the operators of this external index.  DNS-Takeover, MITM
or targetted server breakins will not only compromise the server hosting
the index but also compromise all users and companies using that index.
With a pypi-managed checksummed release link the worst that can happen
is that the release file is not there.  We can leverage the integrity of 
PyPI's usually more solid operations to help users not getting something
malicious in the future because they decided at one point to rely on an
external index now turned evil.

> As far as dev-pi goes, if it's only mirroring links rather than externally
> hosted files today, then in the future, it will still automatically mirror
> the external index URLs. Dependency update scanners could follow those
> links automatically, even if pip install doesn't check them by default.

Yes but it's work to get that right.  Simply having checksummed links
from pypi makes things a lot simpler.

best, need to shop for a barbecue now :)
holger

> One other nice consequence of PEP 470 should make it easier for
> organisations to flag and investigate cases where they're relying on an
> upstream source other than PyPI, regardless of whether they care about the
> details of their dependencies' hosting for speed, reliability or legal
> reasons.
>
> >From a migration perspective, how hard would it be to automate generation
> of a custom index page on pythonhosted.org for projects currently relying
> on external references? That would still let us make the client changes
> without needing to special case PIL.
> 
> Also, it occurred to me that while the latest/any split matters for new
> users, we still need to consider the impact on projects which have pinned
> dependencies on older versions of packages that were previously externally
> hosted, but have moved to PyPI for more recent releases. I still think
> dropping the external reference feature from the simple API in favour of
> improving the custom index support is the right to do, but a couple of
> *client side* examples of handling the migration could help clarify the
> consequences for the existing users that may be affected.
> 
> For example, perhaps we should keep "--allow-all-external", but have it
> mean that pip automatically adds new custom index URLs given for the
> requested packages. Even if it emitted a deprecation warning, clients using
> it would keep working in the face of the proposed changes to the simple API
> link handling.
>
> Regards,
> Nick.

From donald at stufft.io  Sat Jun  7 22:09:57 2014
From: donald at stufft.io (Donald Stufft)
Date: Sat, 7 Jun 2014 16:09:57 -0400
Subject: [Distutils] PEP 470 Round 2 - Using Multi Index Support for
	External to PyPI Package File Hosting
In-Reply-To: <CALeMXf636_hukCTxWKaGx_Mf2k3UncJPvXKROcZ_-82NFgxfqQ@mail.gmail.com>
References: <0D7BEAFC-7987-46C8-8A9A-7931BE4DA36B@stufft.io>
 <20140606081348.GG30349@merlinux.eu>
 <18F8F7BC-11E0-462F-AAB8-635114DEF05C@stufft.io>
 <20140606134127.GK30349@merlinux.eu>
 <9BCE568E-665E-466C-9C5B-1EF80F6C2728@stufft.io>
 <CALeMXf636_hukCTxWKaGx_Mf2k3UncJPvXKROcZ_-82NFgxfqQ@mail.gmail.com>
Message-ID: <274F285A-9428-40D0-B464-609F7A43CB30@stufft.io>


On Jun 7, 2014, at 4:06 PM, PJ Eby <pje at telecommunity.com> wrote:

> 
> On Fri, Jun 6, 2014 at 10:25 AM, Donald Stufft <donald at stufft.io> wrote:
> I expected more people to move to safe external vs staying with the unsafe
> external.
> 
> Is there a tool that makes this *easy*?  I'm not aware of one.
> 
> (Ideally, something like a replacement for setup.py upload that generates the download URLs and sends them off to PyPI, so that all one needs is a setup_requires for the tool, a setup.cfg with the hosting prefix, and a run of "setup.py register bdist_whatever uplink" to get the links set up.)
> 


I know of:

https://warehouse.python.org/project/bitbucket-distutils/
https://warehouse.python.org/project/github-distutils/

But other than that, no. I assume most people who won?t upload to PyPI
are also unlikely to upload to github or bitbucket.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140607/e0d34c53/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140607/e0d34c53/attachment.sig>

From pje at telecommunity.com  Sat Jun  7 22:06:47 2014
From: pje at telecommunity.com (PJ Eby)
Date: Sat, 7 Jun 2014 16:06:47 -0400
Subject: [Distutils] PEP 470 Round 2 - Using Multi Index Support for
 External to PyPI Package File Hosting
In-Reply-To: <9BCE568E-665E-466C-9C5B-1EF80F6C2728@stufft.io>
References: <0D7BEAFC-7987-46C8-8A9A-7931BE4DA36B@stufft.io>
 <20140606081348.GG30349@merlinux.eu>
 <18F8F7BC-11E0-462F-AAB8-635114DEF05C@stufft.io>
 <20140606134127.GK30349@merlinux.eu>
 <9BCE568E-665E-466C-9C5B-1EF80F6C2728@stufft.io>
Message-ID: <CALeMXf636_hukCTxWKaGx_Mf2k3UncJPvXKROcZ_-82NFgxfqQ@mail.gmail.com>

On Fri, Jun 6, 2014 at 10:25 AM, Donald Stufft <donald at stufft.io> wrote:

> I expected more people to move to safe external vs staying with the unsafe
> external.
>

Is there a tool that makes this *easy*?  I'm not aware of one.

(Ideally, something like a replacement for setup.py upload that generates
the download URLs and sends them off to PyPI, so that all one needs is a
setup_requires for the tool, a setup.cfg with the hosting prefix, and a run
of "setup.py register bdist_whatever uplink" to get the links set up.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140607/b39e2a75/attachment.html>

From vladimir.v.diaz at gmail.com  Sat Jun  7 21:53:53 2014
From: vladimir.v.diaz at gmail.com (Vladimir Diaz)
Date: Sat, 7 Jun 2014 15:53:53 -0400
Subject: [Distutils] PEP 470 Round 2 - Using Multi Index Support for
 External to PyPI Package File Hosting
In-Reply-To: <9BCE568E-665E-466C-9C5B-1EF80F6C2728@stufft.io>
References: <0D7BEAFC-7987-46C8-8A9A-7931BE4DA36B@stufft.io>
 <20140606081348.GG30349@merlinux.eu>
 <18F8F7BC-11E0-462F-AAB8-635114DEF05C@stufft.io>
 <20140606134127.GK30349@merlinux.eu>
 <9BCE568E-665E-466C-9C5B-1EF80F6C2728@stufft.io>
Message-ID: <CAOyQwLhQbo2rf_wwyi0uM0sg8umPsvR08kq7itT3QMW6q_WvkQ@mail.gmail.com>

On Fri, Jun 6, 2014 at 10:25 AM, Donald Stufft <donald at stufft.io> wrote:

>
> On Jun 6, 2014, at 9:41 AM, holger krekel <holger at merlinux.eu> wrote:
>
> > On Fri, Jun 06, 2014 at 07:55 -0400, Donald Stufft wrote:
> >>
> >> On Jun 6, 2014, at 4:13 AM, holger krekel <holger at merlinux.eu> wrote:
> >>
> >>> Hi Donald,
> >>>
> >>> 1. you published numbers where <4K or <300 discounting PIL would be
> >>>  affected by PEP470.  You also say that the main reason for deprecating
> >>>  PEP438 is that it confused users.  Did it confuse other users than
> those few?
> >>
> >> It confused more of than the current numbers because at the onset more
> >> projects relied on it than does now. Currently PIL is the primary
> >> instigator for people?s confusion that I personally see.
> >
> > So currently we don't have many confused users anymore.  Doesn't
> > this take away a good part of the reasoning behind PEP470?
>
> No.
>
> >
> > In the following i use "PEP438f" to speak about a hypothetical
> > follow-up PEP as outlined in my previous mail.  I volunteer to write
> > it and present it as an alternative should we not reach some
> > form of conclusion together.
> >
> >>> 2. I don't see a valid precise reasoning why PEP438, just agreed on and
> >>>  implemented last year, needs deprecation.  It boosted everyone
> >>>  everyone's install experiences (independently from the CDN which
> >>>  brought another boost) as usage of crawling dramatically dropped
> >>>  and thus brings us into the exact situation PEP438 already hinted at:
> >>>
> >>>  "Deprecation of hosting modes to eventually only allow the
> >>>  pypi-explicit mode is NOT REGULATED by this PEP but is expected to
> >>>  become feasible some time after successful implementation of the
> >>>  transition phases described in this PEP. It is expected that
> >>>  deprecation requires a new process to deal with abandoned packages
> >>>  because of unreachable maintainers for still popular packages."
> >>>
> >>>  We should follow through and discuss removing crawling and
> >>>  how to deal with abandoned packages.  On the PyPI side, what
> >>>  would remain are two kind of links:
> >>>
> >>>  - pypi internally hosted
> >>>  - registered safe external links to release files
> >>>
> >>>  The resulting situation is:
> >>>
> >>>  easy: users have an already existing option to consider to allow
> externals.
> >>>
> >>>  safe: All links served from pypi have checksums. Project maintainers
> need
> >>>        to register hashed links to their new release files.
> >>>
> >>>  clean: Pip could eventually remove support for crawling/related
> options.
> >>>
> >>>  This is all easy to do, reduces user confusion and makes pip
> >>>  and pypi simpler and less suprising.
> >>>
> >>>  I don't see this approach discussed or seriously considered in the
> PEP,
> >>>  also not in its "rejection reasons?.
> >>
> >> The reasons are listed in the PEP, though I can make it more explicit
> that
> >> it is for this as well.
> >>
> >> * People are generally surprised that PyPI allows externally linking to
> files
> >>  and doesn't require people to host on PyPI. In contrast most of them
> are
> >>  familiar with the concept of multiple software repositories such as is
> in
> >>  use by many OSs.
> >
> > "People are generally surprised" is a rather subjective statement.
> > Wrt to PEP470 we might have at least 65 projects and many more users
> being
> > annoyed rather than just surprised at the sudden change in direction.
> > Especially if there are no compelling arguments.
> >
> >> * PyPI is fronted by a globally distributed CDN which has improved the
> >>  reliability and speed for end users. It is unlikely that any particular
> >>  external host has something comparable. This can lead to extremely bad
> >>  performance for end users when the external host is located in
> different
> >>  parts of the world or does not generally have good connectivity.
> >>
> >>  As a data point, many users reported sub DSL speeds and latency when
> >>  accessing PyPI from parts of Europe and Asia prior to the use of the
> CDN.
> >>
> >> * PyPI has monitoring and an on-call rotation of sysadmins whom can
> respond to
> >>  downtime quickly, thus enabling a quicker response to downtime. Again
> it is
> >>  unlikely that any particular external host will have this. This can
> lead
> >>  to single packages in a dependency chain being un-installable. This
> will
> >>  often confuse users, who often times have no idea that this package
> relies
> >>  on an external host, and they cannot figure out why PyPI appears to be
> up
> >>  but the installer cannot find a package.
> >
> > Sorry but both points have not much to do with the discussion.  If
> > anything, they speak *against* PEP470 because users would need to rely
> > on project specific external index sites to even know which releases
> > exist.  With PEP438 you know that a certain release file must exist and
> > the installer clearly says "i could not download release file X from
> > URL".  Works today.
> >
> > Also the external index could be temporarily broken and serve not the
> newest
> > files.  The integrity and reliability of external indexes would generally
> > not be covered by the CDN and PyPI's on-rotation admins so instead of
> > speaking for PEP470 they speak against it.
>
> The point is, end users are *aware* they are relying on something external
> and they are aware exactly what external items they are relying on. With
> PEP 470
> people can correctly assume that a pip install is going to be covered by
> the CDN
> and our sysadmins unless they?ve gone out of their way to add an
> additional index.
>
> Part of this, is that I consider ?allow-all-external to be a UX failure. I
> originally thought
> it would be alright, but given actual experience with it in the real
> world, I see constant
> confusion about what it does.
>
> >
> >> * PyPI supports mirroring, both for private organizations and public
> mirrors.
> >>  The legal terms of uploading to PyPI ensure that mirror operators, both
> >>  public and private, have the right to distribute the software found on
> PyPI.
> >>  However software that is hosted externally does not have this, causing
> >>  private organizations to need to investigate each package individually
> and
> >>  manually to determine if the license allows them to mirror it.
> >>
> >>  For public mirrors this essentially means that these externally hosted
> >>  packages *cannot* be reasonably mirrored. This is particularly
> troublesome
> >>  in countries such as China where the bandwidth to outside of China is
> >>  highly congested making a mirror within China often times a massively
> better
> >>  experience.
> >
> > With PEP438 today, PyPI merely hosts a link with a checksum, not the file
> > itself.  Mirrors also just mirror that link but not the file.
> > How exactly is managing and mirroring a link a problem here?
>
> The problem isn?t in the link, it?s in that you can?t automatically mirror
> the file,
> which means that the primary way for an individual or organization to take
> control over their own uptime is hampered by the fact that they cannot rely
> on the PyPI ToS to allow them to mirror the file.
>
> >
> >> * In the long run, global opt in flags like ``--allow-all-external``
> will
> >>  become little annoyances that developers cargo cult around in order to
> make
> >>  their installer work. When they run into a project that requires it
> they
> >>  will most likely simply add it to their configuration file for that
> installer
> >>  and continue on with whatever they were actually trying to do. This
> will
> >>  continue until they try to install their requirements on another
> computer
> >>  or attempt to deploy to a server where their install will fail again
> until
> >>  they add the "make it work" flag in their configuration file.
> >
> > Well, with PEP470 developers would have to cargo cult around one or
> > more "--extra-index-url SOME_URL" options.  Also:
>
> Ah, but you see the nature of ``?extra-index-url`` and ``?find-links`` is
> that
> they *feel* more per-projecty. They are more likely to be added to the
> requirements file because they are project specific than a global pip
> configuration file. This isn?t cargo culting, this is configuring your
> project. The
> problem with ``?allow-all-external`` is that lends itself more to being
> added
> to the configuration file for the user (~/.pip/pip.conf) instead of the per
> project requirements file. In one of the earlier discussions at least one
> person
> stated that the first time they came across an externally hosted file they
> would be likely to just chuck the ?allow all external? bit into their
> pip.conf, making
> it unlikely that they will even realize if they are depending or using on
> something
> that requires external hosting.
>
> >
> > - if a project wants to later change its index URL, existing users option
> >  will break and the project has no way to communicate it.  With PEP438f
> >  you can just update your hashes and be done.
>
> The project absolutely has a way to communicate it, first they can update
> their metadata in PyPI which will trigger pip, when it cannot find anything
> to satisfy that requirement, to say ?hey you want to use this extra index?.
> Secondly this is all HTTP, and HTTP has a very old and very well tested
> way to communicate a change in the URL, the 301 redirect.
>
> >
> > - if a project-specific index resides on a domain that changes ownership
> >  (e.g. project abandoned, maintainer gone), user are suddenly vulnerable
> >  without knowing it. With PEP438f the trusted PyPI infrastructure
> >  provides a checksummed link so new DNS owners can not break into
> >  the user's computer as they can do with PEP470.
>
> Sure, but the most likely outcome is that the domain goes missing at first,
> and thanks to PEP 470, pip hard fails because it can?t find that index
> instead
> of soft failing and the person removes the offending index and they are
> perfectly safe. The window of attack is very small and will get smaller as
> we implement other pieces such as package signing.
>

I agree.  Package signing, rather than a link containing a hash sum, seems
like the more secure approach long-term.  Perhaps at some point in the
future external projects can sign and include signatures in metadata
instead, which may be uploaded along with packages (either to PyPI or an
external host.)

A signed metadata file can specify things like a timestamp or version
number, expiry date, and the key signing method.  PyPI can use the extra
information to verify packages, determine the state of the external
project, and also prevent certain attacks against a simple signature or
hash approach.

Cryptography software has a long history of bad user experience design, but
I do not see why it cannot be as user friendly as producing file hashes.

>
> - if projects host on HTTP instead of HTTPS, users are
>  vulnerable against MITM attacks whereas PEP438f provides a checksummed
>  link from PyPI which cannot be easily man-in-the-middled.

Which is why pip warns if you?re using HTTP instead of HTTPS, this will also
> end up being mitigated by package signing.
>
> >
> >> Implied but not explicitly called out reason (I?ll add this):
> >>
> >> * The URL classification only works for a certain subset of projects,
> however
> >>  it does not allow for any project which needs additional restrictions
> such
> >>  as Access Controls. This means that there would be two methods of
> doing the
> >>  same thing, linking to a file safely and hosting an index. Hosting an
> index
> >>  works in all situations and by relying on this we make for a more
> consistent
> >>  experience no matter the reason for external hosting.
> >
> > Once you care for ACLs for indexes and releases you have a number
> > of issues to consider, it's hardly related to PEP470/PEP438.
>
> It is related, because it means that the exact same mechanisms can be used,
> people don?t have to learn two different ways of specifying externally
> hosted
> projects. In fact it also teaches them how to specify mirrors and the like
> as well
> something that any devpi user is already going to have to learn how to do.
>
> >
> >> Not implied, but I?ll add as well:
> >>
> >> * The safe external hosting option hampers the ability of PyPI to
> upgrade it's
> >>  security infrastructure. For instance if MD5 becomes broken in the
> future
> >>  there will be no way for PyPI to upgrade the hashes of the projects
> which
> >>  rely on safe external hosting via MD5 while files that are hosted on
> PyPI
> >>  can simply be processed over with a new hash function.
> >
> > This is a good point.  So we should make sure that it's easy to re-upload
> > with different hashes and maybe store some more hashes (sha256 etc.) with
> > an uploaded file and have pypi decide which one is used.
>
> That assumes the maintainer is maintaining their software anymore.
>
> >
> >> Not going to add:
> >>
> >> Ultimately we're looking at maintaining an additional feature, both in
> PyPI
> >> and in the installers, which almost nobody is actually taking advantage
> of. A
> >> year ago I was hopeful that perhaps people would take advantage of it
> and maybe
> >> that would solve some of the issues. However it's now been an entire
> year and
> >> the buy-in for that feature is minuscule. I do not believe that it
> makes sense
> >> to pay the cost for continuing to have that feature.
> >
> > PEP438 and most of us urged people to use internal hosting and that's
> > what people did.  That we have 65 projects using safe external hosting
> > means we succeeded and there is a clear safe path to host your files
> > externally.  Why did you expect more people to use external hosting
> > when PyPI's current hosting is perceived as good and we told everyone
> > to use it?
>
> No, I expected more people to move to safe external vs staying with the
> unsafe
> external. Strictly speaking there are only 35 projects which safely host
> their
> *latest* version. Looking at the latest version is important because it is
> an
> indicator as to what people plan on doing in the future. The 31 projects
> which
> have *old* files hosted appear to all be projects where maybe one or two
> releases were hosted externally (or perhaps only 1 file out of the
> release).
>
> To put it bluntly, even if we call it 65 projects, that is not enough of a
> buy in
> in my opinion to maintain this feature. People were given the chance to use
> it, not enough did. It?s time to simplify the options.
>
> >
> >> It costs time and effort to ensure that these features to not break. It
> also
> >> adds to the cognitive burden of using the installers. We require end
> users to
> >> learn two different ways of specifying they wish to allow external
> indexes
> >> and require them to understand when or why they'd use one or the other.
> It is
> >> my opinion that this makes the end user experience wholly worse.
> >
> > However, we have a rather settled situation now and can easily
> > clean it up further: crawling can go, pypi-crawl can go, a number
> > of options go.  All easily done with PEP438f.
> >
> >>>  By contrast, PEP470 would require many users to learn about
> >>>  specifying other indexes and what that means.  For you and me
> >>>  and many here on the list it may be a no-brainer but trust me,
> >>>  for many users (i've done ten trainings touching the topic now)
> >>>  this is not a natural concept at all.  "pip install
> --allow-all-externals"
> >>>  is far easier to convey than specifying extra per-project indexes and
> >>>  what it means if the install fails (wrong URL? Index noch reachable?
> >>>  Release file not found?).
> >>
> >> Some projects host their files on PyPI, some files do not, if the thing
> you're
> >> trying to install doesn't host on PyPI you'll have to tell pip where
> it's
> >> hosted. I don't really believe this is a difficult concept.
> >
> > Well, with PEP470 you need to tell and explain users:
> > "--extra-index-url <some_url_you_need_to_find_out>" versus
> > "--allow-all-externals" with PEP438f.
> >
> > I am pretty sure the second would win most usability contests.
>
> I disagree. ?allow-all-external is bad as I mentioned above, it leads
> people to chuck it in their user config and forget about it whereas
> ?extra-index-url leads people to include it in their requirements.txt.
>
> Additionally it?s not ?some url you need to find out? it?s ?some url
> that we?re going to tell you?. This was detailed in the PEP, projects
> would register what that URL is and then pip can present an error
> message like:
>
> Files for foo are located on an another index, add ?external-index-url
> https://example.com/index/
> to locate them.
>

The other day I was pip installing some project and mistyped a command.
pip returned an error message and was smart about determining the command I
actually wanted.  I do not know how long this has been available, but if
the index URL is provided in a similar fashion, then I am a big fan of this
approach.


> This will work massively better than the per project option from PEP 438
> as well. People get confused and don?t realize that they need to specify
> the thing they are trying to install twice (pip install ?allow-external
> foo foo).
>
> >
> >>>
> >>> 3. PEP470 makes life a lot harder for devpi-server, currently used
> >>>  by many companies for serving their private indexes.  With PEP438 and
> >>>  almost no external crawling left, devpi-server can rely on seeing
> >>>  changes through the PEP381 API.  By contrast, with projects hosted on
> >>>  additional per-project external indexes, it requires polling to see
> >>>  changes because releases may not be registered with PyPI anymore (and
> >>>  there is no way to enforce that IISIC).  IOW, PEP470 is a serious
> >>>  regression here as it doesn't allow getting notified on new release
> files.
> >>
> >> That only works for the set of projects which are currently safely
> externally
> >> hosted, which again is tiny.
> >
> > Well, but it works.  With PEP470 it would not work anymore.  That was my
> point.
> > And the question is anyway not how many projects but how many users are
> using
> > externally hosted files.
>
> And the answer is, barely any. Roughly 116 unique IP addresses on the day
> I examined
> which represents 0.1% of the total number of unique IP addresses on that
> day. To be quite
> honest it?s more likely that those projects were grabbed by something
> automated that
> was canning the whole thing rather than an end user actually trying to
> install them. But
> even if it is the case that they were end users, 0.1% is not enough of a
> buy in to justify
> this special case to exist. We have a mechanism for specifying external
> locations to
> fetch from, it?s existed for a long time, we don?t need a special one for
> an ultra minority.
>
> >
> >> It does not work at all for projects which are
> >> hosted externally nor does it work for projects which require some sort
> of
> >> ACL.
> >
> > I don't quite understand why you are talking about ACLs in context of our
> > discussion.
>
> As I said above, because it matters, the less concepts people have to
> understand
> the easier the tooling is to use. If there are two different ways to host
> externally
> then that?s two different concepts users have to learn.
>
> >
> >> Additionally I don't believe devpi *can* actually do that, as it has no
> way
> >> of knowing if it's even legal for it to be mirroring the files if they
> are
> >> not hosted on PyPI. This is one of the points of the PEP, there is a
> known
> >> legal right to distribute files hosted on PyPI, no such right is
> promised for
> >> any file not hosted on PyPI. It's completely possible that devpi is
> doing
> >> something that it has no legal right to do in those cases.
> >
> > If I use devpi on my laptop or in an organisation I am using devpi as a
> > private http cache and not serving my cached files to 3rd parties.
>
> This would depend greatly upon the license of the theoretical library.
> IANAL
> but it is my understanding that it?s completely possible and even
> reasonable
> for there to exist licenses where this is not allowed. The point being that
> by blurring the lines it makes it more difficult for people who have to
> make sure
> they are following the license terms because they have something to lose if
> they don?t whereas providing clear separation makes it so much easier.
>
> >
> > best,
> > holger
> >
> >>>
> >>> best,
> >>> holger
> >>>
> >>> On Thu, Jun 05, 2014 at 22:08 -0400, Donald Stufft wrote:
> >>>> Here's round 2 of PEP 470.
> >>>>
> >>>> You can see it online at https://python.org/dev/peps/pep-0470/ or
> below.
> >>>>
> >>>> Notable changes:
> >>>>
> >>>> - Ensure it's obvious this strictly deals with the installer API and
> does not
> >>>> affect a project's ability to register their project on PyPI for human
> >>>> consumptions.
> >>>>
> >>>> - Mention that the functional mechanisms that make it possible for an
> end user
> >>>> to specify the additional locations have existed for a long time
> across many
> >>>> versions of the installers.
> >>>>
> >>>> - Explicitly mention that the installer changes from PEP 438 should be
> >>>> deprecated and removed as part of this PEP.
> >>>>
> >>>> - Explicitly mention pythonhosted.org as a location that authors can
> use to
> >>>> host an index if they do not wish to purchase a TLS certificate or
> host
> >>>> additional infrastructure.
> >>>>
> >>>> - Include that a link to PyPI ToS should be included in the emails
> sent to
> >>>> authors to remind them of the PyPI ToS.
> >>>>
> >>>> - Special case PIL as it is an outlier in terms of impact.
> >>>>
> >>>> - Fill out the impact sections further to provide more detail
> >>>>
> >>>>
> >>>> Abstract
> >>>> ========
> >>>>
> >>>> This PEP proposes that the official means of having an installer
> locate and
> >>>> find package files which are hosted externally to PyPI become the use
> of
> >>>> multi index support instead of the practice of using external links
> on the
> >>>> simple installer API.
> >>>>
> >>>> It is important to remember that this is **not** about forcing anyone
> to host
> >>>> their files on PyPI. If someone does not wish to do so they will
> never be under
> >>>> any obligation too. They can still list their project in PyPI as an
> index, and
> >>>> the tooling will still allow them to host it elsewhere.
> >>>>
> >>>> This PEP strictly is concerned with the Simple Installer API and how
> automated
> >>>> installers interact with PyPI, it has no bearing on the informational
> pages
> >>>> which are primarily for human consumption.
> >>>>
> >>>>
> >>>> Rationale
> >>>> =========
> >>>>
> >>>> There is a long history documented in PEP 438 that explains why
> externally
> >>>> hosted files exist today in the state that they do on PyPI. For the
> sake of
> >>>> brevity I will not duplicate that and instead urge readers to first
> take a look
> >>>> at PEP 438 for background.
> >>>>
> >>>> There are currently two primary ways for a project to make itself
> available
> >>>> without directly hosting the package files on PyPI. They can either
> include
> >>>> links to the package files in the simpler installer API or they can
> publish
> >>>> a custom package index which contains their project.
> >>>>
> >>>>
> >>>> Custom Additional Index
> >>>> -----------------------
> >>>>
> >>>> Each installer which speaks to PyPI offers a mechanism for the user
> invoking
> >>>> that installer to provide additional custom locations to search for
> files
> >>>> during the dependency resolution phase. For pip these locations can be
> >>>> configured per invocation, per shell environment, per requirements
> file, per
> >>>> virtual environment, and per user. The mechanism for specifying
> additional
> >>>> locations have existed within pip and setuptools for many years, by
> comparison
> >>>> the mechanisms in PEP 438 and any other new mechanism will have
> existed for
> >>>> only a short period of time (if they exist at all currently).
> >>>>
> >>>> The use of additional indexes instead of external links on the simple
> >>>> installer API provides a simple clean interface which is consistent
> with the
> >>>> way most Linux package systems work (apt-get, yum, etc). More
> importantly it
> >>>> works the same even for projects which are commercial or otherwise
> have their
> >>>> access restricted in some form (private networks, password, IP ACLs
> etc)
> >>>> while the external links method only realistically works for projects
> which
> >>>> do not have their access restricted.
> >>>>
> >>>> Compared to the complex rules which a project must be aware of to
> prevent
> >>>> themselves from being considered unsafely hosted setting up an index
> is fairly
> >>>> trivial and in the simplest case does not require anything more than a
> >>>> filesystem and a standard web server such as Nginx or Twisted Web.
> Even if
> >>>> using simple static hosting without autoindexing support, it is still
> >>>> straightforward to generate appropriate index pages as static HTML.
> >>>>
> >>>> Example Index with Twisted Web
> >>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >>>>
> >>>> 1. Create a root directory for your index, for the purposes of the
> example
> >>>>  I'll assume you've chosen ``/var/www/index.example.com/``.
> >>>> 2. Inside of this root directory, create a directory for each project
> such
> >>>>  as ``mkdir -p /var/www/index.example.com/{foo,bar,other}/``.
> >>>> 3. Place the package files for each project in their respective
> folder,
> >>>>  creating paths like ``/var/www/
> index.example.com/foo/foo-1.0.tar.gz``.
> >>>> 4. Configure Twisted Web to serve the root directory, ideally with
> TLS.
> >>>>
> >>>> ::
> >>>>
> >>>>   $ twistd -n web --path /var/www/index.example.com/
> >>>>
> >>>>
> >>>> Examples of Additional indexes with pip
> >>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >>>>
> >>>> **Invocation:**
> >>>>
> >>>> ::
> >>>>
> >>>>   $ pip install --extra-index-url https://pypi.example.com/ foobar
> >>>>
> >>>> **Shell Environment:**
> >>>>
> >>>> ::
> >>>>
> >>>>   $ export PIP_EXTRA_INDEX_URL=https://pypi.example.com/
> >>>>   $ pip install foobar
> >>>>
> >>>> **Requirements File:**
> >>>>
> >>>> ::
> >>>>
> >>>>   $ echo "--extra-index-url https://pypi.example.com/\nfoobar" >
> requirements.txt
> >>>>   $ pip install -r requirements.txt
> >>>>
> >>>> **Virtual Environment:**
> >>>>
> >>>> ::
> >>>>
> >>>>   $ python -m venv myvenv
> >>>>   $ echo "[global]\nextra-index-url = https://pypi.example.com/" >
> myvenv/pip.conf
> >>>>   $ myvenv/bin/pip install foobar
> >>>>
> >>>> **User:**
> >>>>
> >>>> ::
> >>>>
> >>>>   $ echo "[global]\nextra-index-url = https://pypi.example.com/"
> >~/.pip/pip.conf
> >>>>   $ pip install foobar
> >>>>
> >>>>
> >>>> External Links on the Simple Installer API
> >>>> ------------------------------------------
> >>>>
> >>>> PEP 438 proposed a system of classifying file links as either
> internal,
> >>>> external, or unsafe. It recommended that by default only internal
> links would
> >>>> be installed by an installer however users could opt into external
> links on
> >>>> either a global or a per package basis. Additionally they could also
> opt into
> >>>> unsafe links on a per package basis.
> >>>>
> >>>> This system has turned out to be *extremely* unfriendly towards the
> end users
> >>>> and it is the position of this PEP that the situation has become
> untenable. The
> >>>> situation as provided by PEP 438 requires an end user to be aware not
> only of
> >>>> the difference between internal, external, and unsafe, but also to be
> aware of
> >>>> what hosting mode the package they are trying to install is in, what
> links are
> >>>> available on that project's /simple/ page, whether or not those links
> have
> >>>> a properly formatted hash fragment, and what links are available from
> pages
> >>>> linked to from that project's /simple/ page.
> >>>>
> >>>> There are a number of common confusion/pain points with this system
> that I
> >>>> have witnessed:
> >>>>
> >>>> * Users unaware what the simple installer api is at all or how an
> installer
> >>>> locates installable files.
> >>>> * Users unaware that even if the simple api links to a file, if it
> does
> >>>> not include a ``#md5=...`` fragment that it will be counted as unsafe.
> >>>> * Users unaware that an installer can look at pages linked from the
> >>>> simple api to determine additional links, or that any links found in
> this
> >>>> fashion are considered unsafe.
> >>>> * Users are unaware and often surprised that PyPI supports hosting
> your files
> >>>> someplace other than PyPI at all.
> >>>>
> >>>> In addition to that, the information that an installer is able to
> provide
> >>>> when an installation fails is pretty minimal. We are able to detect
> if there
> >>>> are externally hosted files directly linked from the simple installer
> api,
> >>>> however we cannot detect if there are files hosted on a linked page
> without
> >>>> fetching that page and doing so would cause a massive performance hit
> just to
> >>>> see if there might be a file there so that a better error message
> could be
> >>>> provided.
> >>>>
> >>>> Finally very few projects have properly linked to their external
> files so that
> >>>> they can be safely downloaded and verified. At the time of this
> writing there
> >>>> are a total of 65 projects which have files that are only available
> externally
> >>>> and are safely hosted.
> >>>>
> >>>> The end result of all of this, is that with PEP 438, when a user
> attempts to
> >>>> install a file that is not hosted on PyPI typically the steps they
> follow are:
> >>>>
> >>>> 1. First, they attempt to install it normally, using ``pip install
> foobar``.
> >>>>  This fails because the file is not hosted on PyPI and PEP 438 has us
> default
> >>>>  to only hosted on PyPI. If pip detected any externally hosted files
> or other
> >>>>  pages that we *could* have attempted to find other files at it will
> give an
> >>>>  error message suggesting that they try ``--allow-external foobar``.
> >>>> 2. They then attempt to install their package using
> >>>>  ``pip install --allow-external foobar foobar``. If they are lucky
> foobar is
> >>>>  one of the packages which is hosted externally and safely and this
> will
> >>>>  succeed. If they are unlucky they will get a different error message
> >>>>  suggesting that they *also* try ``--allow-unverified foobar``.
> >>>> 3. They then attempt to install their package using
> >>>>  ``pip install --allow-external foobar --allow-unverified foobar
> foobar``
> >>>>  and this finally works.
> >>>>
> >>>> This is the same basic steps that practically everyone goes through
> every time
> >>>> they try to install something that is not hosted on PyPI. If they are
> lucky it'll
> >>>> only take them two steps, but typically it requires three steps.
> Worse there is
> >>>> no real indication to these people why one package might install
> after two
> >>>> but most require three. Even worse than that most of them will never
> get an
> >>>> externally hosted package that does not take three steps, so they
> will be
> >>>> increasingly annoyed and frustrated at the intermediate step and will
> likely
> >>>> eventually just start skipping it.
> >>>>
> >>>>
> >>>> External Index Discovery
> >>>> ========================
> >>>>
> >>>> One of the problems with using an additional index is one of
> discovery. Users
> >>>> will not generally be aware that an additional index is required at
> all much
> >>>> less where that index can be found. Projects can attempt to convey
> this
> >>>> information using their description on the PyPI page however that
> excludes
> >>>> people who discover their project organically through ``pip search``.
> >>>>
> >>>> To support projects that wish to externally host their files and to
> enable
> >>>> users to easily discover what additional indexes are required, PyPI
> will gain
> >>>> the ability for projects to register external index URLs and
> additionally an
> >>>> associated comment for each. These URLs will be made available on the
> simple
> >>>> page however they will not be linked or provided in a form that older
> >>>> installers will automatically search them.
> >>>>
> >>>> When an installer fetches the simple page for a project, if it finds
> this
> >>>> additional meta-data and it cannot find any files for that project in
> it's
> >>>> configured URLs then it should use this data to tell the user how to
> add one
> >>>> or more of the additional URLs to search in. This message should
> include any
> >>>> comments that the project has included to enable them to communicate
> to the
> >>>> user and provide hints as to which URL they might want if some are
> only
> >>>> useful or compatible with certain platforms or situations. When the
> installer
> >>>> has implemented the auto discovery mechanisms they should also
> deprecate any
> >>>> of the mechanisms added for PEP 438 (such as ``--allow-external``)
> for removal
> >>>> at the end of the deprecation period proposed by the PEP.
> >>>>
> >>>> This feature *must* be added to PyPI prior to starting the
> deprecation and
> >>>> removal process for link spidering.
> >>>>
> >>>>
> >>>> Deprecation and Removal of Link Spidering
> >>>> =========================================
> >>>>
> >>>> A new hosting mode will be added to PyPI. This hosting mode will be
> called
> >>>> ``pypi-only`` and will be in addition to the three that PEP 438 has
> already
> >>>> given us which are ``pypi-explicit``, ``pypi-scrape``,
> ``pypi-scrape-crawl``.
> >>>> This new hosting mode will modify a project's simple api page so that
> it only
> >>>> lists the files which are directly hosted on PyPI and will not link
> to anything
> >>>> else.
> >>>>
> >>>> Upon acceptance of this PEP and the addition of the ``pypi-only``
> mode, all new
> >>>> projects will by defaulted to the PyPI only mode and they will be
> locked to
> >>>> this mode and unable to change this particular setting. ``pypi-only``
> projects
> >>>> will still be able to register external index URLs as described above
> - the
> >>>> "pypi-only" refers only to the download links that are published
> directly on
> >>>> PyPI.
> >>>>
> >>>> An email will then be sent out to all of the projects which are
> hosted only on
> >>>> PyPI informing them that in one month their project will be
> automatically
> >>>> converted to the ``pypi-only`` mode. A month after these emails have
> been sent
> >>>> any of those projects which were emailed, which still are hosted only
> on PyPI
> >>>> will have their mode set to ``pypi-only``.
> >>>>
> >>>> After that switch, an email will be sent to projects which rely on
> hosting
> >>>> external to PyPI. This email will warn these projects that externally
> hosted
> >>>> files have been deprecated on PyPI and that in 6 months from the time
> of that
> >>>> email that all external links will be removed from the installer
> APIs. This
> >>>> email *must* include instructions for converting their projects to be
> hosted
> >>>> on PyPI and *must* include links to a script or package that will
> enable them
> >>>> to enter their PyPI credentials and package name and have it
> automatically
> >>>> download and re-host all of their files on PyPI. This email *must
> also*
> >>>> include instructions for setting up their own index page and
> registering that
> >>>> with PyPI, including the fact that they can use pythonhosted.org as
> a host
> >>>> for an index page without requiring them to host any additional
> infrastructure
> >>>> or purchase a TLS certificate. This email must also contain a link to
> the Terms
> >>>> of Service for PyPI as many users may have signed up a long time ago
> and may
> >>>> not recall what those terms are.
> >>>>
> >>>> Five months after the initial email, another email must be sent to
> any projects
> >>>> still relying on external hosting. This email will include all of the
> same
> >>>> information that the first email contained, except that the removal
> date will
> >>>> be one month away instead of six.
> >>>>
> >>>> Finally a month later all projects will be switched to the
> ``pypi-only`` mode
> >>>> and PyPI will be modified to remove the externally linked files
> functionality.
> >>>> At this point in time any installers should finally remove any of the
> >>>> deprecated PEP 438 functionality such as ``--allow-external`` and
> >>>> ``--allow-unverified`` in pip.
> >>>>
> >>>>
> >>>> PIL
> >>>> ---
> >>>>
> >>>> It's obvious from the numbers below that the vast bulk of the impact
> come from
> >>>> the PIL project. On 2014-05-17 an email was sent to the contact for
> PIL
> >>>> inquiring whether or not they would be willing to upload to PyPI. A
> response
> >>>> has not been received as of yet (2014-06-05) nor has any change in
> the hosting
> >>>> happened. Due to the popularity of PIL this PEP also proposes that
> during the
> >>>> deprecation period that PyPI Administrators will set the PIL download
> URL as
> >>>> the external index for that project. Allowing the users of PIL to take
> >>>> advantage of the auto discovery mechanisms although the project has
> seemingly
> >>>> become unmaintained.
> >>>>
> >>>>
> >>>> Impact
> >>>> ======
> >>>>
> >>>> The largest impact of this is going to be projects where the
> maintainers are
> >>>> no longer maintaining the project, for one reason or another. For
> these
> >>>> projects it's unlikely that a maintainer will arrive to set the
> external index
> >>>> metadata which would allow the auto discovery mechanism to find it.
> >>>>
> >>>> Looking at the numbers factoring out PIL (which has been special
> cased above)
> >>>> the actual impact should be quite low, with it affecting just 6.9% of
> projects
> >>>> which host only externally or 2.8% which have their latest version
> hosted
> >>>> externally. This represents a mere 3883 unique IP addresses. The
> break down of
> >>>> this is that of those 3883 addresses, 100% of them installed
> something that
> >>>> could not be verified while only 3% installed something which could
> be.
> >>>>
> >>>>
> >>>> Projects Which Rely on Externally Hosted files
> >>>> ----------------------------------------------
> >>>>
> >>>> This is determined by crawling the simple index and looking for
> installable
> >>>> files using a similar detection method as pip and setuptools use. The
> "latest"
> >>>> version is determined using ``pkg_resources.parse_version`` sort
> order and it
> >>>> is used to show whether or not the latest version is hosted
> externally or only
> >>>> old versions are.
> >>>>
> >>>> ============ ======= ================ =================== =======
> >>>> \             PyPI    External (old)   External (latest)   Total
> >>>> ============ ======= ================ =================== =======
> >>>> **Safe**     38716   31               35                  38782
> >>>> **Unsafe**   0       1659             1169                2828
> >>>> **Total**    38716   1690             1204                41610
> >>>> ============ ======= ================ =================== =======
> >>>>
> >>>>
> >>>> Top Externally Hosted Projects by Requests
> >>>> ------------------------------------------
> >>>>
> >>>> This is determined by looking at the number of requests the
> >>>> ``/simple/<project>/`` page had gotten in a single day. The total
> number of
> >>>> requests during that day was 17,960,467.
> >>>>
> >>>> ============================== ========
> >>>> Project                        Requests
> >>>> ============================== ========
> >>>> PIL                            13470
> >>>> mysql-connector-python         321
> >>>> salesforce-python-toolkit      54
> >>>> pyodbc                         50
> >>>> elementtree                    44
> >>>> atfork                         39
> >>>> RBTools                        29
> >>>> django-contrib-requestprovider 28
> >>>> wadofstuff-django-serializers  23
> >>>> Pygame                         21
> >>>> ============================== ========
> >>>>
> >>>>
> >>>> Top Externally Hosted Projects by Unique IPs
> >>>> --------------------------------------------
> >>>>
> >>>> This is determined by looking at the IP addresses of requests the
> >>>> ``/simple/<project>/`` page had gotten in a single day. The total
> number of
> >>>> unique IP addresses during that day was 105,587.
> >>>>
> >>>> ============================== ==========
> >>>> Project                        Unique IPs
> >>>> ============================== ==========
> >>>> PIL                            3515
> >>>> mysql-connector-python         117
> >>>> pyodbc                         34
> >>>> elementtree                    21
> >>>> RBTools                        19
> >>>> egenix-mx-base                 16
> >>>> Pygame                         14
> >>>> salesforce-python-toolkit      13
> >>>> django-contrib-requestprovider 12
> >>>> wxPython                       11
> >>>> python-apt                     10
> >>>> ============================== ==========
> >>>>
> >>>>
> >>>> Rejected Proposals
> >>>> ==================
> >>>>
> >>>> Keep the current classification system but adjust the options
> >>>> -------------------------------------------------------------
> >>>>
> >>>> This PEP rejects several related proposals which attempt to fix some
> of the
> >>>> usability problems with the current system but while still keeping the
> >>>> general gist of PEP 438.
> >>>>
> >>>> This includes:
> >>>>
> >>>> * Default to allowing safely externally hosted files, but disallow
> unsafely
> >>>> hosted.
> >>>> * Default to disallowing safely externally hosted files with only a
> global
> >>>> flag to enable them, but disallow unsafely hosted.
> >>>>
> >>>> These proposals are rejected because:
> >>>>
> >>>> * The classification "system" is complex, hard to explain, and
> requires an
> >>>> intimate knowledge of how the simple API works in order to be able to
> reason
> >>>> about which classification is required. This is reflected in the fact
> that
> >>>> the code to implement it is complicated and hard to understand as
> well.
> >>>>
> >>>> * People are generally surprised that PyPI allows externally linking
> to files
> >>>> and doesn't require people to host on PyPI. In contrast most of them
> are
> >>>> familiar with the concept of multiple software repositories such as
> is in
> >>>> use by many OSs.
> >>>>
> >>>> * PyPI is fronted by a globally distributed CDN which has improved the
> >>>> reliability and speed for end users. It is unlikely that any
> particular
> >>>> external host has something comparable. This can lead to extremely bad
> >>>> performance for end users when the external host is located in
> different
> >>>> parts of the world or does not generally have good connectivity.
> >>>>
> >>>> As a data point, many users reported sub DSL speeds and latency when
> >>>> accessing PyPI from parts of Europe and Asia prior to the use of the
> CDN.
> >>>>
> >>>> * PyPI has monitoring and an on-call rotation of sysadmins whom can
> respond to
> >>>> downtime quickly, thus enabling a quicker response to downtime. Again
> it is
> >>>> unlikely that any particular external host will have this. This can
> lead
> >>>> to single packages in a dependency chain being un-installable. This
> will
> >>>> often confuse users, who often times have no idea that this package
> relies
> >>>> on an external host, and they cannot figure out why PyPI appears to
> be up
> >>>> but the installer cannot find a package.
> >>>>
> >>>> * PyPI supports mirroring, both for private organizations and public
> mirrors.
> >>>> The legal terms of uploading to PyPI ensure that mirror operators,
> both
> >>>> public and private, have the right to distribute the software found
> on PyPI.
> >>>> However software that is hosted externally does not have this, causing
> >>>> private organizations to need to investigate each package
> individually and
> >>>> manually to determine if the license allows them to mirror it.
> >>>>
> >>>> For public mirrors this essentially means that these externally hosted
> >>>> packages *cannot* be reasonably mirrored. This is particularly
> troublesome
> >>>> in countries such as China where the bandwidth to outside of China is
> >>>> highly congested making a mirror within China often times a massively
> better
> >>>> experience.
> >>>>
> >>>> * Installers have no method to determine if they should expect any
> particular
> >>>> URL to be available or not. It is not unusual for the simple API to
> reference
> >>>> old packages and URLs which have long since stopped working. This
> causes
> >>>> installers to have to assume that it is OK for any particular URL to
> not be
> >>>> accessible. This causes problems where an URL is temporarily down or
> >>>> otherwise unavailable (a common cause of this is using a copy of
> Python
> >>>> linked against a really ancient copy of OpenSSL which is unable to
> verify
> >>>> the SSL certificate on PyPI) but it *should* be expected to be up. In
> this
> >>>> case installers will typically silently ignore this URL and later the
> user
> >>>> will get a confusing error stating that the installer couldn't find
> any
> >>>> versions instead of getting the real error message indicating that
> the URL
> >>>> was unavailable.
> >>>>
> >>>> * In the long run, global opt in flags like ``--allow-all-external``
> will
> >>>> become little annoyances that developers cargo cult around in order
> to make
> >>>> their installer work. When they run into a project that requires it
> they
> >>>> will most likely simply add it to their configuration file for that
> installer
> >>>> and continue on with whatever they were actually trying to do. This
> will
> >>>> continue until they try to install their requirements on another
> computer
> >>>> or attempt to deploy to a server where their install will fail again
> until
> >>>> they add the "make it work" flag in their configuration file.
> >>>>
> >>>>
> >>>> -----------------
> >>>> Donald Stufft
> >>>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9
> 3372 DCFA
> >>>>
> >>>
> >>>
> >>>
> >>>> _______________________________________________
> >>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
> >>>> https://mail.python.org/mailman/listinfo/distutils-sig
> >>>
> >>
> >>
> >> -----------------
> >> Donald Stufft
> >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140607/bdcc8c51/attachment-0001.html>

From wichert at wiggy.net  Tue Jun 10 11:40:09 2014
From: wichert at wiggy.net (Wichert Akkerman)
Date: Tue, 10 Jun 2014 11:40:09 +0200
Subject: [Distutils] PyPI lost IPv6 support?
Message-ID: <5A7FB6E4-E7FF-4ECD-B65F-0216EFC1CB25@wiggy.net>

I just noticed that my uploads to PyPI are now using IPv4 instead of IPv6. Looking closer it looks like PyPI is not reachable over IPv6 at all anymore, which is somewhat disappointing. Was dropping IPv6 a deliberate choice, or an unfortunate side-effect of switching to Fastly?s CDN?

Regards,
Wichert.

From noah at coderanger.net  Tue Jun 10 12:20:05 2014
From: noah at coderanger.net (Noah Kantrowitz)
Date: Tue, 10 Jun 2014 03:20:05 -0700
Subject: [Distutils] PyPI lost IPv6 support?
In-Reply-To: <5A7FB6E4-E7FF-4ECD-B65F-0216EFC1CB25@wiggy.net>
References: <5A7FB6E4-E7FF-4ECD-B65F-0216EFC1CB25@wiggy.net>
Message-ID: <A9B1FAC8-45A6-4CD6-AC9F-0105CD7238CE@coderanger.net>

Both, supporting IPv6 is not a priority and so no extra work will be done for it. This is true across the board for all PSF services.

--Noah

On Jun 10, 2014, at 2:40 AM, Wichert Akkerman <wichert at wiggy.net> wrote:

> I just noticed that my uploads to PyPI are now using IPv4 instead of IPv6. Looking closer it looks like PyPI is not reachable over IPv6 at all anymore, which is somewhat disappointing. Was dropping IPv6 a deliberate choice, or an unfortunate side-effect of switching to Fastly?s CDN?
> 
> Regards,
> Wichert.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140610/cb3c459f/attachment.sig>

From daniele.sluijters at gmail.com  Tue Jun 10 13:44:01 2014
From: daniele.sluijters at gmail.com (Daniele Sluijters)
Date: Tue, 10 Jun 2014 13:44:01 +0200
Subject: [Distutils] PyPI lost IPv6 support?
In-Reply-To: <A9B1FAC8-45A6-4CD6-AC9F-0105CD7238CE@coderanger.net>
References: <5A7FB6E4-E7FF-4ECD-B65F-0216EFC1CB25@wiggy.net>
 <A9B1FAC8-45A6-4CD6-AC9F-0105CD7238CE@coderanger.net>
Message-ID: <CACdcYG0m0FtoyyWYJ4ZqYxNYOv53GJ-cbcYxOD1OZcuX4z1p-A@mail.gmail.com>

IPv6 (on AMS-IX) currently accounts for 20G/s of traffic at peak times
compared to 2.8T/s overall. That said, the amount of traffic that IPv6
generates has almost quadrupled since August last year. Stating that
IPv6 is not on the PSF's list of priorities is, as far as I'm
concerned, short sighted. It doesn't have to work tomorrow but it
should be on the list of things to get to.

PyPi not being available over IPv6 anymore is not "we're not willing
to do extra work to enable it", it's a regression. I understand that
this is not something the PSF can solve but it should at least push
Fastly to roll out IPv6.

-- 
Daniele Sluijters

On 10 June 2014 12:20, Noah Kantrowitz <noah at coderanger.net> wrote:
> Both, supporting IPv6 is not a priority and so no extra work will be done for it. This is true across the board for all PSF services.
>
> --Noah
>
> On Jun 10, 2014, at 2:40 AM, Wichert Akkerman <wichert at wiggy.net> wrote:
>
>> I just noticed that my uploads to PyPI are now using IPv4 instead of IPv6. Looking closer it looks like PyPI is not reachable over IPv6 at all anymore, which is somewhat disappointing. Was dropping IPv6 a deliberate choice, or an unfortunate side-effect of switching to Fastly?s CDN?
>>
>> Regards,
>> Wichert.
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
Daniele Sluijters

From donald at stufft.io  Tue Jun 10 14:59:42 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 10 Jun 2014 08:59:42 -0400
Subject: [Distutils] PyPI lost IPv6 support?
In-Reply-To: <CACdcYG0m0FtoyyWYJ4ZqYxNYOv53GJ-cbcYxOD1OZcuX4z1p-A@mail.gmail.com>
References: <5A7FB6E4-E7FF-4ECD-B65F-0216EFC1CB25@wiggy.net>
 <A9B1FAC8-45A6-4CD6-AC9F-0105CD7238CE@coderanger.net>
 <CACdcYG0m0FtoyyWYJ4ZqYxNYOv53GJ-cbcYxOD1OZcuX4z1p-A@mail.gmail.com>
Message-ID: <C2AD4704-0C75-4171-9933-D4C58C033493@stufft.io>


On Jun 10, 2014, at 7:44 AM, Daniele Sluijters <daniele.sluijters at gmail.com> wrote:

> IPv6 (on AMS-IX) currently accounts for 20G/s of traffic at peak times
> compared to 2.8T/s overall. That said, the amount of traffic that IPv6
> generates has almost quadrupled since August last year. Stating that
> IPv6 is not on the PSF's list of priorities is, as far as I'm
> concerned, short sighted. It doesn't have to work tomorrow but it
> should be on the list of things to get to.
> 
> PyPi not being available over IPv6 anymore is not "we're not willing
> to do extra work to enable it", it's a regression. I understand that
> this is not something the PSF can solve but it should at least push
> Fastly to roll out IPv6.
> 

I'm pretty sure the thing is that there are many more important things that are
of a higher priority than IPv6 support. While IPv6 support is nice to have, it
also doesn't generally matter unless there is someone trying to connect to
PyPI who has no IPv4 connectivity. I don't believe it to be likely that there
is many, if any, people who do not have IPv4 connectivity else they'd be unable
to reach vast parts of the internet.

I'm sure at some point Fastly will have IPv6 connectivity, at which point there
won't be any major reason *not* to turn it on.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140610/f3e11d9a/attachment.sig>

From ncoghlan at gmail.com  Tue Jun 10 15:09:37 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 10 Jun 2014 23:09:37 +1000
Subject: [Distutils] PyPI lost IPv6 support?
In-Reply-To: <C2AD4704-0C75-4171-9933-D4C58C033493@stufft.io>
References: <5A7FB6E4-E7FF-4ECD-B65F-0216EFC1CB25@wiggy.net>
 <A9B1FAC8-45A6-4CD6-AC9F-0105CD7238CE@coderanger.net>
 <CACdcYG0m0FtoyyWYJ4ZqYxNYOv53GJ-cbcYxOD1OZcuX4z1p-A@mail.gmail.com>
 <C2AD4704-0C75-4171-9933-D4C58C033493@stufft.io>
Message-ID: <CADiSq7fEO3LdfeFL+gBV7xgkpoYD2dmJCB2763iR_pwzq_tE=g@mail.gmail.com>

On 10 June 2014 22:59, Donald Stufft <donald at stufft.io> wrote:
> On Jun 10, 2014, at 7:44 AM, Daniele Sluijters <daniele.sluijters at gmail.com> wrote:
>> PyPi not being available over IPv6 anymore is not "we're not willing
>> to do extra work to enable it", it's a regression. I understand that
>> this is not something the PSF can solve but it should at least push
>> Fastly to roll out IPv6.
>
> I'm pretty sure the thing is that there are many more important things that are
> of a higher priority than IPv6 support. While IPv6 support is nice to have, it
> also doesn't generally matter unless there is someone trying to connect to
> PyPI who has no IPv4 connectivity. I don't believe it to be likely that there
> is many, if any, people who do not have IPv4 connectivity else they'd be unable
> to reach vast parts of the internet.

The thread prompted me to go back and check the status of IPv4
availability, at least in APAC, since IANA originally ran out of /8
blocks a few years back. It turns out APNIC is still parcelling out
their last /8 block, and the IANA's reclamation of unused IPv4 blocks
also freed up some addresses for reallocation. (I tried to look up the
status for Africa as well, but didn't find any clear statement in a
quick search).

> I'm sure at some point Fastly will have IPv6 connectivity, at which point there
> won't be any major reason *not* to turn it on.

It's something we'll want to keep an eye on, but yeah, at this point
in time, when connecting an IPv6-only system to the internet, PyPI is
likely to be long way down the "it isn't working" priority list.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From wichert at wiggy.net  Tue Jun 10 15:22:34 2014
From: wichert at wiggy.net (Wichert Akkerman)
Date: Tue, 10 Jun 2014 15:22:34 +0200
Subject: [Distutils] PyPI lost IPv6 support?
In-Reply-To: <CADiSq7fEO3LdfeFL+gBV7xgkpoYD2dmJCB2763iR_pwzq_tE=g@mail.gmail.com>
References: <5A7FB6E4-E7FF-4ECD-B65F-0216EFC1CB25@wiggy.net>
 <A9B1FAC8-45A6-4CD6-AC9F-0105CD7238CE@coderanger.net>
 <CACdcYG0m0FtoyyWYJ4ZqYxNYOv53GJ-cbcYxOD1OZcuX4z1p-A@mail.gmail.com>
 <C2AD4704-0C75-4171-9933-D4C58C033493@stufft.io>
 <CADiSq7fEO3LdfeFL+gBV7xgkpoYD2dmJCB2763iR_pwzq_tE=g@mail.gmail.com>
Message-ID: <39F4F579-0792-4E6C-B7CF-5868C5A54A2B@wiggy.net>


On 10 Jun 2014, at 15:09, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 10 June 2014 22:59, Donald Stufft <donald at stufft.io> wrote:
>> On Jun 10, 2014, at 7:44 AM, Daniele Sluijters <daniele.sluijters at gmail.com> wrote:
>>> PyPi not being available over IPv6 anymore is not "we're not willing
>>> to do extra work to enable it", it's a regression. I understand that
>>> this is not something the PSF can solve but it should at least push
>>> Fastly to roll out IPv6.
>> 
>> I'm pretty sure the thing is that there are many more important things that are
>> of a higher priority than IPv6 support. While IPv6 support is nice to have, it
>> also doesn't generally matter unless there is someone trying to connect to
>> PyPI who has no IPv4 connectivity. I don't believe it to be likely that there
>> is many, if any, people who do not have IPv4 connectivity else they'd be unable
>> to reach vast parts of the internet.
> 
> The thread prompted me to go back and check the status of IPv4
> availability, at least in APAC, since IANA originally ran out of /8
> blocks a few years back. It turns out APNIC is still parcelling out
> their last /8 block, and the IANA's reclamation of unused IPv4 blocks
> also freed up some addresses for reallocation. (I tried to look up the
> status for Africa as well, but didn't find any clear statement in a
> quick search).

If I remember correctly there are some mobile networks in Asia who only do IPv6 internally. Gandi offers IPv6-only servers that are cheaper than servers with ipv4 connectivity. So while right now not having IPv4 connectivity is unlikely, it does happen and will only become more common. People are also introducing monstrosities like carrier-grade NAT to delay the inevitable, but we really should not encourage that madness and just add IPv6. It generally is very easy to do.

> It's something we'll want to keep an eye on, but yeah, at this point
> in time, when connecting an IPv6-only system to the internet, PyPI is
> likely to be long way down the "it isn't working" priority list.

I have an ipv6-only VM, and it works wonderfully: it can send email, pull Debian updates, serve IPv6 websites and it has my remote backups and git-annex repositories.

Wichert.

From ncoghlan at gmail.com  Tue Jun 10 15:47:25 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 10 Jun 2014 23:47:25 +1000
Subject: [Distutils] PyPI lost IPv6 support?
In-Reply-To: <39F4F579-0792-4E6C-B7CF-5868C5A54A2B@wiggy.net>
References: <5A7FB6E4-E7FF-4ECD-B65F-0216EFC1CB25@wiggy.net>
 <A9B1FAC8-45A6-4CD6-AC9F-0105CD7238CE@coderanger.net>
 <CACdcYG0m0FtoyyWYJ4ZqYxNYOv53GJ-cbcYxOD1OZcuX4z1p-A@mail.gmail.com>
 <C2AD4704-0C75-4171-9933-D4C58C033493@stufft.io>
 <CADiSq7fEO3LdfeFL+gBV7xgkpoYD2dmJCB2763iR_pwzq_tE=g@mail.gmail.com>
 <39F4F579-0792-4E6C-B7CF-5868C5A54A2B@wiggy.net>
Message-ID: <CADiSq7dsvoYKKOFra2oYPyU-S-mwqXk0NHfign7t-k3i=m2GMQ@mail.gmail.com>

On 10 June 2014 23:22, Wichert Akkerman <wichert at wiggy.net> wrote:
>
> If I remember correctly there are some mobile networks in Asia who only do IPv6 internally. Gandi offers IPv6-only servers that are cheaper than servers with ipv4 connectivity. So while right now not having IPv4 connectivity is unlikely, it does happen and will only become more common. People are also introducing monstrosities like carrier-grade NAT to delay the inevitable, but we really should not encourage that madness and just add IPv6. It generally is very easy to do.

The challenge is that PyPI now runs behind a donated CDN service, and
our vendor doesn't offer IPv6 yet:
https://fastly.zendesk.com/entries/30549708-Do-you-support-IPv6-

That means that, for the time being "the PyPI CDN is generously
donated by Fastly" trumps "the PyPI CDN supports IPv6" - IPv6 support
isn't currently high enough on the priority list for us to be willing
to turn down Fastly's offer. That trade-off may change some day, but I
expect Fastly will have already added IPv6 support before we reach
that point.

>> It's something we'll want to keep an eye on, but yeah, at this point
>> in time, when connecting an IPv6-only system to the internet, PyPI is
>> likely to be long way down the "it isn't working" priority list.
>
> I have an ipv6-only VM, and it works wonderfully: it can send email, pull Debian updates, serve IPv6 websites and it has my remote backups and git-annex repositories.

I was thinking of the client case, but you're right, in a server
context, IPv6 only is far more likely to be viable already.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From wichert at wiggy.net  Tue Jun 10 16:00:44 2014
From: wichert at wiggy.net (Wichert Akkerman)
Date: Tue, 10 Jun 2014 16:00:44 +0200
Subject: [Distutils] PyPI lost IPv6 support?
In-Reply-To: <CADiSq7dsvoYKKOFra2oYPyU-S-mwqXk0NHfign7t-k3i=m2GMQ@mail.gmail.com>
References: <5A7FB6E4-E7FF-4ECD-B65F-0216EFC1CB25@wiggy.net>
 <A9B1FAC8-45A6-4CD6-AC9F-0105CD7238CE@coderanger.net>
 <CACdcYG0m0FtoyyWYJ4ZqYxNYOv53GJ-cbcYxOD1OZcuX4z1p-A@mail.gmail.com>
 <C2AD4704-0C75-4171-9933-D4C58C033493@stufft.io>
 <CADiSq7fEO3LdfeFL+gBV7xgkpoYD2dmJCB2763iR_pwzq_tE=g@mail.gmail.com>
 <39F4F579-0792-4E6C-B7CF-5868C5A54A2B@wiggy.net>
 <CADiSq7dsvoYKKOFra2oYPyU-S-mwqXk0NHfign7t-k3i=m2GMQ@mail.gmail.com>
Message-ID: <0A122086-17B3-4E0F-8101-DDAB23B693BE@wiggy.net>

On 10 Jun 2014, at 15:47, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 10 June 2014 23:22, Wichert Akkerman <wichert at wiggy.net> wrote:
>> 
>> If I remember correctly there are some mobile networks in Asia who only do IPv6 internally. Gandi offers IPv6-only servers that are cheaper than servers with ipv4 connectivity. So while right now not having IPv4 connectivity is unlikely, it does happen and will only become more common. People are also introducing monstrosities like carrier-grade NAT to delay the inevitable, but we really should not encourage that madness and just add IPv6. It generally is very easy to do.
> 
> The challenge is that PyPI now runs behind a donated CDN service, and
> our vendor doesn't offer IPv6 yet:
> https://fastly.zendesk.com/entries/30549708-Do-you-support-IPv6-

I saw that and was very disappointed by Fastly. It means I won't consider using them for anything I do at this point in time.
 
> That means that, for the time being "the PyPI CDN is generously
> donated by Fastly" trumps "the PyPI CDN supports IPv6" - IPv6 support
> isn't currently high enough on the priority list for us to be willing
> to turn down Fastly's offer. That trade-off may change some day, but I
> expect Fastly will have already added IPv6 support before we reach
> that point.

Is the PSF willing to ask push Fastly a little bit on this, so we at least have timeframe?

>>> It's something we'll want to keep an eye on, but yeah, at this point
>>> in time, when connecting an IPv6-only system to the internet, PyPI is
>>> likely to be long way down the "it isn't working" priority list.
>> 
>> I have an ipv6-only VM, and it works wonderfully: it can send email, pull Debian updates, serve IPv6 websites and it has my remote backups and git-annex repositories.
> 
> I was thinking of the client case, but you're right, in a server
> context, IPv6 only is far more likely to be viable already.

It?s very viable right now I?ld say. If I remember correctly Cloudflare has a free service where they add an IPv4 frontend for IPv6-only servers so you can still provide service to IPv4-only users.

Wichert.

From donald at stufft.io  Tue Jun 10 16:20:23 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 10 Jun 2014 10:20:23 -0400
Subject: [Distutils] PyPI lost IPv6 support?
In-Reply-To: <0A122086-17B3-4E0F-8101-DDAB23B693BE@wiggy.net>
References: <5A7FB6E4-E7FF-4ECD-B65F-0216EFC1CB25@wiggy.net>
 <A9B1FAC8-45A6-4CD6-AC9F-0105CD7238CE@coderanger.net>
 <CACdcYG0m0FtoyyWYJ4ZqYxNYOv53GJ-cbcYxOD1OZcuX4z1p-A@mail.gmail.com>
 <C2AD4704-0C75-4171-9933-D4C58C033493@stufft.io>
 <CADiSq7fEO3LdfeFL+gBV7xgkpoYD2dmJCB2763iR_pwzq_tE=g@mail.gmail.com>
 <39F4F579-0792-4E6C-B7CF-5868C5A54A2B@wiggy.net>
 <CADiSq7dsvoYKKOFra2oYPyU-S-mwqXk0NHfign7t-k3i=m2GMQ@mail.gmail.com>
 <0A122086-17B3-4E0F-8101-DDAB23B693BE@wiggy.net>
Message-ID: <020D7FFE-18B1-4D57-807C-0C35EE463C24@stufft.io>


On Jun 10, 2014, at 10:00 AM, Wichert Akkerman <wichert at wiggy.net> wrote:

> On 10 Jun 2014, at 15:47, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On 10 June 2014 23:22, Wichert Akkerman <wichert at wiggy.net> wrote:
>>> 
>>> If I remember correctly there are some mobile networks in Asia who only do IPv6 internally. Gandi offers IPv6-only servers that are cheaper than servers with ipv4 connectivity. So while right now not having IPv4 connectivity is unlikely, it does happen and will only become more common. People are also introducing monstrosities like carrier-grade NAT to delay the inevitable, but we really should not encourage that madness and just add IPv6. It generally is very easy to do.
>> 
>> The challenge is that PyPI now runs behind a donated CDN service, and
>> our vendor doesn't offer IPv6 yet:
>> https://fastly.zendesk.com/entries/30549708-Do-you-support-IPv6-
> 
> I saw that and was very disappointed by Fastly. It means I won't consider using them for anything I do at this point in time.

Eh, like anything it's a value judgment. Fastly has been great for us and it
has made scaling PyPI massively easier and just better overall. I wouldn't
want to try to handle the ~78TB and ~891M requests that PyPI got in May (or
~108TB/~1.1B requests for all of Python.org via Fastly) without Fastly,
especially not for something like IPv6.

> 
>> That means that, for the time being "the PyPI CDN is generously
>> donated by Fastly" trumps "the PyPI CDN supports IPv6" - IPv6 support
>> isn't currently high enough on the priority list for us to be willing
>> to turn down Fastly's offer. That trade-off may change some day, but I
>> expect Fastly will have already added IPv6 support before we reach
>> that point.
> 
> Is the PSF willing to ask push Fastly a little bit on this, so we at least have timeframe?

We don't need to push them, we have a real amicable relationship :)

I just popped into their IRC channel and asked, they said that IPv6 is on their
radar and planned for the future but they do not have an ETA.

We're unlikely to do anything but wait for Fastly at this point. Perhaps if
push came to shove we'd setup a ipv6.pypi.python.org which doesn't route
through Fastly but we're not setup for that currently and it'd take a bit of
effort to do so. Effort we'd much rather spend elsewhere right now.

Of course PyPI supports mirroring, so if someone believes that ipv6 support
is important for package installs all it takes is a server with some bandwidth,
~80GB of HD space, and an IPv6 address to host a IPv6 mirror. It's like it
wouldn't get near as much traffic as PyPI itself does so it wouldn't require
as much as PyPI does to power it.

> 
>>>> It's something we'll want to keep an eye on, but yeah, at this point
>>>> in time, when connecting an IPv6-only system to the internet, PyPI is
>>>> likely to be long way down the "it isn't working" priority list.
>>> 
>>> I have an ipv6-only VM, and it works wonderfully: it can send email, pull Debian updates, serve IPv6 websites and it has my remote backups and git-annex repositories.
>> 
>> I was thinking of the client case, but you're right, in a server
>> context, IPv6 only is far more likely to be viable already.
> 
> It?s very viable right now I?ld say. If I remember correctly Cloudflare has a free service where they add an IPv4 frontend for IPv6-only servers so you can still provide service to IPv4-only users.
> 
> Wichert.


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140610/65c90bb4/attachment.sig>

From daniele.sluijters at gmail.com  Tue Jun 10 16:00:17 2014
From: daniele.sluijters at gmail.com (Daniele Sluijters)
Date: Tue, 10 Jun 2014 16:00:17 +0200
Subject: [Distutils] PyPI lost IPv6 support?
In-Reply-To: <CADiSq7dsvoYKKOFra2oYPyU-S-mwqXk0NHfign7t-k3i=m2GMQ@mail.gmail.com>
References: <5A7FB6E4-E7FF-4ECD-B65F-0216EFC1CB25@wiggy.net>
 <A9B1FAC8-45A6-4CD6-AC9F-0105CD7238CE@coderanger.net>
 <CACdcYG0m0FtoyyWYJ4ZqYxNYOv53GJ-cbcYxOD1OZcuX4z1p-A@mail.gmail.com>
 <C2AD4704-0C75-4171-9933-D4C58C033493@stufft.io>
 <CADiSq7fEO3LdfeFL+gBV7xgkpoYD2dmJCB2763iR_pwzq_tE=g@mail.gmail.com>
 <39F4F579-0792-4E6C-B7CF-5868C5A54A2B@wiggy.net>
 <CADiSq7dsvoYKKOFra2oYPyU-S-mwqXk0NHfign7t-k3i=m2GMQ@mail.gmail.com>
Message-ID: <CACdcYG0s82Wh-ej6de=-3DNX71nhOQd7ALVp4n_bCqCktccMBg@mail.gmail.com>

Looks like Latin America has reached the point of no return:
http://www.lacnic.net/en/web/anuncios/2014-no-hay-mas-direcciones-ipv4-en-lac
(article is in English)

On 10 June 2014 15:47, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 10 June 2014 23:22, Wichert Akkerman <wichert at wiggy.net> wrote:
>>
>> If I remember correctly there are some mobile networks in Asia who only do IPv6 internally. Gandi offers IPv6-only servers that are cheaper than servers with ipv4 connectivity. So while right now not having IPv4 connectivity is unlikely, it does happen and will only become more common. People are also introducing monstrosities like carrier-grade NAT to delay the inevitable, but we really should not encourage that madness and just add IPv6. It generally is very easy to do.
>
> The challenge is that PyPI now runs behind a donated CDN service, and
> our vendor doesn't offer IPv6 yet:
> https://fastly.zendesk.com/entries/30549708-Do-you-support-IPv6-
>
> That means that, for the time being "the PyPI CDN is generously
> donated by Fastly" trumps "the PyPI CDN supports IPv6" - IPv6 support
> isn't currently high enough on the priority list for us to be willing
> to turn down Fastly's offer. That trade-off may change some day, but I
> expect Fastly will have already added IPv6 support before we reach
> that point.
>
>>> It's something we'll want to keep an eye on, but yeah, at this point
>>> in time, when connecting an IPv6-only system to the internet, PyPI is
>>> likely to be long way down the "it isn't working" priority list.
>>
>> I have an ipv6-only VM, and it works wonderfully: it can send email, pull Debian updates, serve IPv6 websites and it has my remote backups and git-annex repositories.
>
> I was thinking of the client case, but you're right, in a server
> context, IPv6 only is far more likely to be viable already.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia



-- 
Daniele Sluijters

From chris.barker at noaa.gov  Wed Jun 11 17:35:06 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 11 Jun 2014 08:35:06 -0700
Subject: [Distutils] ctypes and shared libs, and wheels, oh my!
Message-ID: <CALGmxE+q_4p00UjoATpBH_B_sGfpfqMfqnNjdCe3_uGPaYVwnQ@mail.gmail.com>

Folks,

I'm trying to help figure out how to do binary wheels for a package that
relies on ctypes and a bundles shared lib (dll, .so. etc)

The trick here is that the python code is quite version and platform
independent: py2 and py3, version 2.7 and 3.3+ (I think)

(it's py_enchant, if anyone is interested:
http://pythonhosted.org/pyenchant/)

So the trick is that the binary wheel will be platform dependent, but not
the code itself, so ideally we'd have one wheel, that for instance (and teh
case at hand) should work on any OS-X box version 10.6 and above, with a
any of python2.7, 3.3, 3.4 (an up?)

Usually, a binary wheel involves  compiled extensions, and thus is tied to
a particular python version -- so this is an odd case.

We tried:

pyenchant-1.6.6-py2.py3-none-macosx_10_6_intel.whl

which seems to be saying: any version of python2 or python 3, but only on
macosx 10.6

but trying to install that on my machine (py2.7, os-x 10.6) gives:

pyenchant-1.6.6-py2.py3-none-macosx_10_6_intel.whl is not a supported wheel
on this platform.

(side note: it would be really great if that could be a more useful message
-- what part of the file name didn't match? I know that's a trick, as there
is a whole pile of heuristics to go through, but maybe a way to dump that
process would be helpful...)

Now, this may, in fat be tied to CPython (I have no idea if ctypes
is available on pypy or jython or IronPython...). So I tried:

pyenchant-1.6.6-cp27-none-macosx_10_6_intel.whl

that does, indeed, install on my system.

Also:

pyenchant-1.6.6-cp27.cp33-none-macosx_10_6_intel.whl

works.

As 2.7 is really the only py2 that much matters, no biggie, but is there a
way to get 3.3 and 3.4 and ??? all at once (I don't have py3 on that
machine, so didn't test that...)

So: how should this be done? Is the above the best option there is?

Thanks,
  -Chris














-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140611/56e913cb/attachment.html>

From dholth at gmail.com  Wed Jun 11 18:09:59 2014
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 11 Jun 2014 12:09:59 -0400
Subject: [Distutils] ctypes and shared libs, and wheels, oh my!
In-Reply-To: <CALGmxE+q_4p00UjoATpBH_B_sGfpfqMfqnNjdCe3_uGPaYVwnQ@mail.gmail.com>
References: <CALGmxE+q_4p00UjoATpBH_B_sGfpfqMfqnNjdCe3_uGPaYVwnQ@mail.gmail.com>
Message-ID: <CAG8k2+4VUs7Yd=XXVr8wzXmromGWXsO3qaaXCNHooAOz8Fi6hg@mail.gmail.com>

This is in the bug tracker already. We need to add the py2-none-arch tags
etc. to Pip's list.
On Jun 11, 2014 12:05 PM, "Chris Barker" <chris.barker at noaa.gov> wrote:

> Folks,
>
> I'm trying to help figure out how to do binary wheels for a package that
> relies on ctypes and a bundles shared lib (dll, .so. etc)
>
> The trick here is that the python code is quite version and platform
> independent: py2 and py3, version 2.7 and 3.3+ (I think)
>
> (it's py_enchant, if anyone is interested:
> http://pythonhosted.org/pyenchant/)
>
> So the trick is that the binary wheel will be platform dependent, but not
> the code itself, so ideally we'd have one wheel, that for instance (and teh
> case at hand) should work on any OS-X box version 10.6 and above, with a
> any of python2.7, 3.3, 3.4 (an up?)
>
> Usually, a binary wheel involves  compiled extensions, and thus is tied to
> a particular python version -- so this is an odd case.
>
> We tried:
>
> pyenchant-1.6.6-py2.py3-none-macosx_10_6_intel.whl
>
> which seems to be saying: any version of python2 or python 3, but only on
> macosx 10.6
>
> but trying to install that on my machine (py2.7, os-x 10.6) gives:
>
> pyenchant-1.6.6-py2.py3-none-macosx_10_6_intel.whl is not a supported
> wheel on this platform.
>
> (side note: it would be really great if that could be a more useful
> message -- what part of the file name didn't match? I know that's a trick,
> as there is a whole pile of heuristics to go through, but maybe a way to
> dump that process would be helpful...)
>
> Now, this may, in fat be tied to CPython (I have no idea if ctypes
> is available on pypy or jython or IronPython...). So I tried:
>
> pyenchant-1.6.6-cp27-none-macosx_10_6_intel.whl
>
> that does, indeed, install on my system.
>
> Also:
>
> pyenchant-1.6.6-cp27.cp33-none-macosx_10_6_intel.whl
>
> works.
>
> As 2.7 is really the only py2 that much matters, no biggie, but is there a
> way to get 3.3 and 3.4 and ??? all at once (I don't have py3 on that
> machine, so didn't test that...)
>
> So: how should this be done? Is the above the best option there is?
>
> Thanks,
>   -Chris
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140611/9c79feba/attachment.html>

From chris.barker at noaa.gov  Wed Jun 11 18:32:54 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 11 Jun 2014 09:32:54 -0700
Subject: [Distutils] ctypes and shared libs, and wheels, oh my!
In-Reply-To: <CAG8k2+4VUs7Yd=XXVr8wzXmromGWXsO3qaaXCNHooAOz8Fi6hg@mail.gmail.com>
References: <CALGmxE+q_4p00UjoATpBH_B_sGfpfqMfqnNjdCe3_uGPaYVwnQ@mail.gmail.com>
 <CAG8k2+4VUs7Yd=XXVr8wzXmromGWXsO3qaaXCNHooAOz8Fi6hg@mail.gmail.com>
Message-ID: <CALGmxELHducPqRmvKQkXkRt95cS1iSa2tRPjWTteUiRfiWNXnw@mail.gmail.com>

On Wed, Jun 11, 2014 at 9:09 AM, Daniel Holth <dholth at gmail.com> wrote:

> This is in the bug tracker already. We need to add the py2-none-arch tags
> etc. to Pip's list.
>
Great, thanks. Is the idea that:

pyenchant-1.6.6-py2.py3-none-macosx_10_6_intel.whl

should have worked? And will in some future version?

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140611/ca6893fa/attachment-0001.html>

From dholth at gmail.com  Wed Jun 11 18:50:27 2014
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 11 Jun 2014 12:50:27 -0400
Subject: [Distutils] ctypes and shared libs, and wheels, oh my!
In-Reply-To: <CALGmxELHducPqRmvKQkXkRt95cS1iSa2tRPjWTteUiRfiWNXnw@mail.gmail.com>
References: <CALGmxE+q_4p00UjoATpBH_B_sGfpfqMfqnNjdCe3_uGPaYVwnQ@mail.gmail.com>
 <CAG8k2+4VUs7Yd=XXVr8wzXmromGWXsO3qaaXCNHooAOz8Fi6hg@mail.gmail.com>
 <CALGmxELHducPqRmvKQkXkRt95cS1iSa2tRPjWTteUiRfiWNXnw@mail.gmail.com>
Message-ID: <CAG8k2+5CJMqKkE7HEGC=-82WR9Bys-VPBVO5cWcU3Z_1UR_6Qw@mail.gmail.com>

If you insert those tags into this list, in the
pip.pep425tags.get_supported() function:
https://github.com/pypa/pip/blob/develop/pip/pep425tags.py#L38 then
your wheel will become installable.

I'd probably put it right after these:
https://github.com/pypa/pip/blob/develop/pip/pep425tags.py#L78

On Wed, Jun 11, 2014 at 12:32 PM, Chris Barker <chris.barker at noaa.gov> wrote:
> On Wed, Jun 11, 2014 at 9:09 AM, Daniel Holth <dholth at gmail.com> wrote:
>>
>> This is in the bug tracker already. We need to add the py2-none-arch tags
>> etc. to Pip's list.
>
> Great, thanks. Is the idea that:
>
> pyenchant-1.6.6-py2.py3-none-macosx_10_6_intel.whl
>
> should have worked? And will in some future version?
>
> -Chris
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov

From chris.barker at noaa.gov  Thu Jun 12 00:01:53 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 11 Jun 2014 15:01:53 -0700
Subject: [Distutils] ctypes and shared libs, and wheels, oh my!
In-Reply-To: <CAG8k2+5CJMqKkE7HEGC=-82WR9Bys-VPBVO5cWcU3Z_1UR_6Qw@mail.gmail.com>
References: <CALGmxE+q_4p00UjoATpBH_B_sGfpfqMfqnNjdCe3_uGPaYVwnQ@mail.gmail.com>
 <CAG8k2+4VUs7Yd=XXVr8wzXmromGWXsO3qaaXCNHooAOz8Fi6hg@mail.gmail.com>
 <CALGmxELHducPqRmvKQkXkRt95cS1iSa2tRPjWTteUiRfiWNXnw@mail.gmail.com>
 <CAG8k2+5CJMqKkE7HEGC=-82WR9Bys-VPBVO5cWcU3Z_1UR_6Qw@mail.gmail.com>
Message-ID: <CALGmxEJ49veHZE=h6CWdsX_M1Ab10OMArGJZayLm6XmJpB8E3w@mail.gmail.com>

On Wed, Jun 11, 2014 at 9:50 AM, Daniel Holth <dholth at gmail.com> wrote:

> If you insert those tags into this list, in the
> pip.pep425tags.get_supported() function:
> https://github.com/pypa/pip/blob/develop/pip/pep425tags.py#L38 then
> your wheel will become installable.
>
> I'd probably put it right after these:
> https://github.com/pypa/pip/blob/develop/pip/pep425tags.py#L78


OK, thanks -- that's going to take some poking around.

Maybe once we have the build actually working, I'll take a look at that.

-Chris





>  On Wed, Jun 11, 2014 at 12:32 PM, Chris Barker <chris.barker at noaa.gov>
> wrote:
> > On Wed, Jun 11, 2014 at 9:09 AM, Daniel Holth <dholth at gmail.com> wrote:
> >>
> >> This is in the bug tracker already. We need to add the py2-none-arch
> tags
> >> etc. to Pip's list.
> >
> > Great, thanks. Is the idea that:
> >
> > pyenchant-1.6.6-py2.py3-none-macosx_10_6_intel.whl
> >
> > should have worked? And will in some future version?
> >
> > -Chris
> >
> >
> > --
> >
> > Christopher Barker, Ph.D.
> > Oceanographer
> >
> > Emergency Response Division
> > NOAA/NOS/OR&R            (206) 526-6959   voice
> > 7600 Sand Point Way NE   (206) 526-6329   fax
> > Seattle, WA  98115       (206) 526-6317   main reception
> >
> > Chris.Barker at noaa.gov
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140611/e34f1bdd/attachment.html>

From robin at reportlab.com  Thu Jun 12 11:21:57 2014
From: robin at reportlab.com (Robin Becker)
Date: Thu, 12 Jun 2014 10:21:57 +0100
Subject: [Distutils] wheels and installation time
Message-ID: <53997135.3090003@chamonix.reportlab.co.uk>

In the past reportlab used .exe packages for the various windows distributions. 
However, we recently decided to use wheels. With patches to support cross 
platform compilation I can build these on win xp win32 for win32 & amd64. The 
wheels themselves work fine.

However, in the past we used to support installation time options eg create a 
start menu folder where some of our scripts could be easily found, desktop icons 
etc etc. I looked in the pip 1.5.6 code, but could not see any place where 
installation time scripts could be built. Is there a way to get scripts run at 
installation time?
-- 
Robin Becker

From donald at stufft.io  Thu Jun 12 18:01:11 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 12 Jun 2014 12:01:11 -0400
Subject: [Distutils] wheels and installation time
In-Reply-To: <53997135.3090003@chamonix.reportlab.co.uk>
References: <53997135.3090003@chamonix.reportlab.co.uk>
Message-ID: <1DACC6BA-BED6-4925-A15D-33B2AE240BC7@stufft.io>

Not currently. There?s an open issue about how to handle that within Wheels as a few projects need those kinds of hooks.

On Jun 12, 2014, at 5:21 AM, Robin Becker <robin at reportlab.com> wrote:

> In the past reportlab used .exe packages for the various windows distributions. However, we recently decided to use wheels. With patches to support cross platform compilation I can build these on win xp win32 for win32 & amd64. The wheels themselves work fine.
> 
> However, in the past we used to support installation time options eg create a start menu folder where some of our scripts could be easily found, desktop icons etc etc. I looked in the pip 1.5.6 code, but could not see any place where installation time scripts could be built. Is there a way to get scripts run at installation time?
> -- 
> Robin Becker
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140612/90722678/attachment.sig>

From ncoghlan at gmail.com  Fri Jun 13 00:37:40 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 13 Jun 2014 08:37:40 +1000
Subject: [Distutils] wheels and installation time
In-Reply-To: <1DACC6BA-BED6-4925-A15D-33B2AE240BC7@stufft.io>
References: <53997135.3090003@chamonix.reportlab.co.uk>
 <1DACC6BA-BED6-4925-A15D-33B2AE240BC7@stufft.io>
Message-ID: <CADiSq7cYL6XNdkXSM_+Z0RtwBeY1-FVnnY4fh2PpgXXq91tu4A@mail.gmail.com>

On 13 Jun 2014 02:01, "Donald Stufft" <donald at stufft.io> wrote:
>
> Not currently. There?s an open issue about how to handle that within
Wheels as a few projects need those kinds of hooks.

To elaborate a little further on that, install time scripts are one of the
main motivators for "required extensions" in PEP 426 - it means that
deciding on an install hook design can be postponed until *after* the
initial metadata 2.0 definition, so there's an opportunity to experiment
with different approaches before blessing one as the official way of
requesting them.

Cheers,
Nick.

>
> On Jun 12, 2014, at 5:21 AM, Robin Becker <robin at reportlab.com> wrote:
>
> > In the past reportlab used .exe packages for the various windows
distributions. However, we recently decided to use wheels. With patches to
support cross platform compilation I can build these on win xp win32 for
win32 & amd64. The wheels themselves work fine.
> >
> > However, in the past we used to support installation time options eg
create a start menu folder where some of our scripts could be easily found,
desktop icons etc etc. I looked in the pip 1.5.6 code, but could not see
any place where installation time scripts could be built. Is there a way to
get scripts run at installation time?
> > --
> > Robin Becker
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
>
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
DCFA
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140613/a6cf60e7/attachment.html>

From dholth at gmail.com  Fri Jun 13 14:29:06 2014
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 13 Jun 2014 08:29:06 -0400
Subject: [Distutils] ctypes and shared libs, and wheels, oh my!
In-Reply-To: <539ADCCE.30203@rfk.id.au>
References: <CALGmxE+q_4p00UjoATpBH_B_sGfpfqMfqnNjdCe3_uGPaYVwnQ@mail.gmail.com>
 <CAG8k2+4VUs7Yd=XXVr8wzXmromGWXsO3qaaXCNHooAOz8Fi6hg@mail.gmail.com>
 <539ADCCE.30203@rfk.id.au>
Message-ID: <CAG8k2+6Ma7UPUctL1zu-b6Ru_phffgte8DojmDj3Gvv8xyMgFQ@mail.gmail.com>

https://github.com/pypa/pip/issues/1870

On Fri, Jun 13, 2014 at 7:13 AM, Ryan Kelly <ryan at rfk.id.au> wrote:
> On 12/06/2014 2:09 AM, Daniel Holth wrote:
>> This is in the bug tracker already. We need to add the py2-none-arch
>> tags etc. to Pip's list.
>
> Thanks Daniel, can you please link me to the specific bug?  I couldn't
> find it in the pip issue tracker, but I could easily have been looking
> wrong.
>
>   Cheers,
>
>     Ryan
>
>> On Jun 11, 2014 12:05 PM, "Chris Barker" <chris.barker at noaa.gov
>> <mailto:chris.barker at noaa.gov>> wrote:
>>
>>     Folks,
>>
>>     I'm trying to help figure out how to do binary wheels for a package
>>     that relies on ctypes and a bundles shared lib (dll, .so. etc)
>>
>>     The trick here is that the python code is quite version and platform
>>     independent: py2 and py3, version 2.7 and 3.3+ (I think)
>>
>>     (it's py_enchant, if anyone is
>>     interested: http://pythonhosted.org/pyenchant/)
>>
>>     So the trick is that the binary wheel will be platform dependent,
>>     but not the code itself, so ideally we'd have one wheel, that for
>>     instance (and teh case at hand) should work on any OS-X box version
>>     10.6 and above, with a any of python2.7, 3.3, 3.4 (an up?)
>>
>>     Usually, a binary wheel involves  compiled extensions, and thus is
>>     tied to a particular python version -- so this is an odd case.
>>
>>     We tried:
>>
>>     pyenchant-1.6.6-py2.py3-none-macosx_10_6_intel.whl
>>
>>     which seems to be saying: any version of python2 or python 3, but
>>     only on macosx 10.6
>>
>>     but trying to install that on my machine (py2.7, os-x 10.6) gives:
>>
>>     pyenchant-1.6.6-py2.py3-none-macosx_10_6_intel.whl is not a
>>     supported wheel on this platform.
>>
>>     (side note: it would be really great if that could be a more useful
>>     message -- what part of the file name didn't match? I know that's a
>>     trick, as there is a whole pile of heuristics to go through, but
>>     maybe a way to dump that process would be helpful...)
>>
>>     Now, this may, in fat be tied to CPython (I have no idea if ctypes
>>     is available on pypy or jython or IronPython...). So I tried:
>>
>>     pyenchant-1.6.6-cp27-none-macosx_10_6_intel.whl
>>
>>     that does, indeed, install on my system.
>>
>>     Also:
>>
>>     pyenchant-1.6.6-cp27.cp33-none-macosx_10_6_intel.whl
>>
>>     works.
>>
>>     As 2.7 is really the only py2 that much matters, no biggie, but is
>>     there a way to get 3.3 and 3.4 and ??? all at once (I don't have py3
>>     on that machine, so didn't test that...)
>>
>>     So: how should this be done? Is the above the best option there is?
>>
>>     Thanks,
>>       -Chris
>

From jaraco at jaraco.com  Sat Jun 14 16:25:02 2014
From: jaraco at jaraco.com (Jason R. Coombs)
Date: Sat, 14 Jun 2014 14:25:02 +0000
Subject: [Distutils] Setuptools 4.0 and 4.0.1 superseded by 3.8.1
Message-ID: <715c52d434ab49e187add4a86fdd2a6f@BLUPR06MB434.namprd06.prod.outlook.com>

Due to the regression reported in [Setuptools #218]( https://bitbucket.org/pypa/setuptools/issue/218/setuppy-develop-fails-with-some-packages), I've removed all versions of Setuptools 4.x and closed that branch.

I will continue to explore the underlying issue, [Setuptools #210]( https://bitbucket.org/pypa/setuptools/issue/210) to seek a proper solution, possibly without backward incompatibility.

Please use Setuptools 3.8.1 or later. If another backward-incompatible release is made for anything other than issue 210, it will tagged as Setuptools 5.0, bypassing 4.0 altogether.

Regards,
Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140614/266b066c/attachment.html>

From fungi at yuggoth.org  Sat Jun 14 16:53:32 2014
From: fungi at yuggoth.org (Jeremy Stanley)
Date: Sat, 14 Jun 2014 14:53:32 +0000
Subject: [Distutils] Setuptools 4.0 and 4.0.1 superseded by 3.8.1
In-Reply-To: <715c52d434ab49e187add4a86fdd2a6f@BLUPR06MB434.namprd06.prod.outlook.com>
References: <715c52d434ab49e187add4a86fdd2a6f@BLUPR06MB434.namprd06.prod.outlook.com>
Message-ID: <20140614145331.GA1251@yuggoth.org>

On 2014-06-14 14:25:02 +0000 (+0000), Jason R. Coombs wrote:
[...]
> I?ve removed all versions of Setuptools 4.x and closed that
> branch. [...] Please use Setuptools 3.8.1 or later.
[...]

Are most of the PyPI mirror tools out there smart enough to remove
packages which have been deleted from PyPI? People who are running
their own mirrors should probably check and take action as needed.
(I know I'm going to need to either delete copies from our CI
mirror, or more likely blacklist 4.x in all our requirements files
since I can't know the state of anyone else's mirrors.)
-- 
{ PGP( 48F9961143495829 ); FINGER( fungi at cthulhu.yuggoth.org );
WWW( http://fungi.yuggoth.org/ ); IRC( fungi at irc.yuggoth.org#ccl );
WHOIS( STANL3-ARIN ); MUD( kinrui at katarsis.mudpy.org:6669 ); }

From nad at acm.org  Sat Jun 14 20:15:40 2014
From: nad at acm.org (Ned Deily)
Date: Sat, 14 Jun 2014 11:15:40 -0700
Subject: [Distutils] Setuptools 4.0 and 4.0.1 superseded by 3.8.1
References: <715c52d434ab49e187add4a86fdd2a6f@BLUPR06MB434.namprd06.prod.outlook.com>
 <20140614145331.GA1251@yuggoth.org>
Message-ID: <nad-B42ADC.11154014062014@news.gmane.org>

In article <20140614145331.GA1251 at yuggoth.org>,
 Jeremy Stanley <fungi at yuggoth.org> wrote:

> On 2014-06-14 14:25:02 +0000 (+0000), Jason R. Coombs wrote:
> [...]
> > I?ve removed all versions of Setuptools 4.x and closed that
> > branch. [...] Please use Setuptools 3.8.1 or later.
> [...]
> 
> Are most of the PyPI mirror tools out there smart enough to remove
> packages which have been deleted from PyPI? People who are running
> their own mirrors should probably check and take action as needed.
> (I know I'm going to need to either delete copies from our CI
> mirror, or more likely blacklist 4.x in all our requirements files
> since I can't know the state of anyone else's mirrors.)

It's not just PyPI.  Third-party distributors have already pushed 4.x 
out into the wild, for example, MacPorts.  It's not practical to make 
the whole world revert once that happens.  If it's a serious enough 
regression, I suggest you release a 4.x+ that is really a 3.8.x.

-- 
 Ned Deily,
 nad at acm.org


From donald at stufft.io  Sat Jun 14 20:16:44 2014
From: donald at stufft.io (Donald Stufft)
Date: Sat, 14 Jun 2014 14:16:44 -0400
Subject: [Distutils] Setuptools 4.0 and 4.0.1 superseded by 3.8.1
In-Reply-To: <nad-B42ADC.11154014062014@news.gmane.org>
References: <715c52d434ab49e187add4a86fdd2a6f@BLUPR06MB434.namprd06.prod.outlook.com>
 <20140614145331.GA1251@yuggoth.org>
 <nad-B42ADC.11154014062014@news.gmane.org>
Message-ID: <0F3BB363-B0AB-48E9-A4FF-F2D6A3E66EE6@stufft.io>

A 5.0 has been released which resolves this.

On Jun 14, 2014, at 2:15 PM, Ned Deily <nad at acm.org> wrote:

> In article <20140614145331.GA1251 at yuggoth.org>,
> Jeremy Stanley <fungi at yuggoth.org> wrote:
> 
>> On 2014-06-14 14:25:02 +0000 (+0000), Jason R. Coombs wrote:
>> [...]
>>> I?ve removed all versions of Setuptools 4.x and closed that
>>> branch. [...] Please use Setuptools 3.8.1 or later.
>> [...]
>> 
>> Are most of the PyPI mirror tools out there smart enough to remove
>> packages which have been deleted from PyPI? People who are running
>> their own mirrors should probably check and take action as needed.
>> (I know I'm going to need to either delete copies from our CI
>> mirror, or more likely blacklist 4.x in all our requirements files
>> since I can't know the state of anyone else's mirrors.)
> 
> It's not just PyPI.  Third-party distributors have already pushed 4.x 
> out into the wild, for example, MacPorts.  It's not practical to make 
> the whole world revert once that happens.  If it's a serious enough 
> regression, I suggest you release a 4.x+ that is really a 3.8.x.
> 
> -- 
> Ned Deily,
> nad at acm.org
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140614/2c693d59/attachment.sig>

From egor.berdnikov at gmail.com  Sat Jun 14 11:07:06 2014
From: egor.berdnikov at gmail.com (Egor Berdnikov)
Date: Sat, 14 Jun 2014 16:07:06 +0700
Subject: [Distutils] Is there a chance for directory globs supported in
	'data_files' ?
Message-ID: <CAKoyyFVoO8LzL1s+6cAzuOC_p+d=2Fe6X2caAQ3da7yOu448=w@mail.gmail.com>

Hi all,
Some time ago there was a movement towards deprecating MANIFEST.in and
adding directory support to setup()'s 'data_files'.  Is there actually
something against implementing directory globs in 'data_files'?

I'm a little bit lost among current 'sdist' and 'bdist_rpm' options
regarding directories. I'm searching for a solution for 'bdist_rpm'
errors like this one:
    "can't copy 'lib/directory/*': doesn't exist or not a regular file".

Related part of the documentation, a mail list discussion, and a bug:
https://docs.python.org/2/distutils/setupscript.html#installing-additional-files
https://mail.python.org/pipermail/distutils-sig/2009-April/011308.html
http://bugs.python.org/issue5302

-- 
Egor

From Erik.Purins at imgtec.com  Fri Jun 13 21:36:18 2014
From: Erik.Purins at imgtec.com (Erik Purins)
Date: Fri, 13 Jun 2014 19:36:18 +0000
Subject: [Distutils] windows permissions on bdist egg scripts, data?
Message-ID: <CFC0BCE0.26182%erik.purins@imgtec.com>

Hey there,

I?m using python 2.7/setuptools 3.6 to build an egg of my code (checked into perforce) using my setup.py. This egg gets installed into a virtualenv.

On osx/darwin this works fine.

On windows, the bdist creates scripts and data whose permissions are read-only (attrib+r), and they cannot overwrite themselves in a subsequent build of that egg. Similarly, re-installing a new egg using virtualenv fails because the installed egg?s scripts and data are not writable.

I thought it might be that when checked out from perforce, the permissions stayed read-only during the build of the egg, so I set all the files in my module to be read-write before building. Even when the original source files? permissions are all set to read-write, the generated bdist egg script/data content end up becoming read-only.

Couple of questions.

  1.  Since it works fine on osx, it seems like a bug in the windows implementation of setuptools. Is this known behavior, is there a reason for setting this on scripts/data on windows?
  2.  Is there a hook in the bdist egg build process I can use to set the attributes of the files correctly when the bdist egg is made? How/where should I define this in my setup.py?

Kind regards,

-e
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140613/5fd40b5b/attachment-0001.html>

From lunasspecto at gmail.com  Sun Jun 15 02:25:23 2014
From: lunasspecto at gmail.com (lunasspecto at gmail.com)
Date: Sat, 14 Jun 2014 20:25:23 -0400
Subject: [Distutils] setuptools wish list
In-Reply-To: <CACyo8_8XSncuMb9-Gyf87+2oN5hjyQxVLDgq-ZviD__oP47daw@mail.gmail.com>
References: <CACyo8_8XSncuMb9-Gyf87+2oN5hjyQxVLDgq-ZviD__oP47daw@mail.gmail.com>
Message-ID: <CACyo8__S2Q4dbmbSD=+5H=PJmvtmWds_1+OvOxpnH2BdmBC80Q@mail.gmail.com>

Hi folks. I'm new to the list and don't mean to impose, but I've been
following some of the recent developments of the PyPA and I'm wondering if
the unreasonable expectations I have for Python packaging are about to come
true, or if there's a simple workaround to achieve some of these goals in
the meantime.

In building the Python 3 application I'm currently working on I have the
end goal of supporting Windows users, Debian-based system users, and maybe
Fedora users with installable distributions they can install without
worrying about dependencies. I won't consider my packaging efforts to be
successful if somebody using one of these systems has to manually install
Python 3.4 or PyQt5 before they can install my application. Ultimately I'd
also like to package my application as a standalone OS X app, but I don't
have access to an OS X system right now.


 After some wrestling with cx_Freeze in Python 3.4 on Windows 7 in a
VirtualBox machine, I've been able to put together a satisfactory Windows
test distribution bundled with all its dependencies. setuptools will
produce an rpm package, but I wouldn't know how to make sure the package
depends on PyQt 5 and the QScintilla plugin for PyQt 5, so Fedora users can
install my application without having to know they need these dependencies.
I can't list PyQt5 or PyQt5.Qsci in the "install-requires" setuptools
option because, as far as I understand, this option is only for listing
packages that can be installed by pip. I have managed to set up a Debian
packaging infrastructure such that I can build a Debian binary package with
the command "dh --with python3 binary-indep", but I haven't figured out how
to include an XDG menu entry in the package, and I consider this a
deal-breaker. Of course, setuptools doesn't currently handle XDG menus at
all.


 So my wish list for setuptools or a similar tool is:

*

Automated XDG menu entry generation

*

Some way to handle dependencies that can't be installed by pip

*

Automated generation of Debian binary packages good enough to be accepted
into the repositories

*

Automated generation of rpm packages with all the right dependencies (For
all I know this could exist already, but considering the way dependencies
are currently handled by setuptools I really doubt it)

*

In a perfect universe, easy cross-building functionality, so I could
produce a standalone Windows executable or an OS X app from the comfort of
Xubuntu.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140614/307e5d79/attachment.html>

From ryan at rfk.id.au  Fri Jun 13 13:13:18 2014
From: ryan at rfk.id.au (Ryan Kelly)
Date: Fri, 13 Jun 2014 21:13:18 +1000
Subject: [Distutils] ctypes and shared libs, and wheels, oh my!
In-Reply-To: <CAG8k2+4VUs7Yd=XXVr8wzXmromGWXsO3qaaXCNHooAOz8Fi6hg@mail.gmail.com>
References: <CALGmxE+q_4p00UjoATpBH_B_sGfpfqMfqnNjdCe3_uGPaYVwnQ@mail.gmail.com>
 <CAG8k2+4VUs7Yd=XXVr8wzXmromGWXsO3qaaXCNHooAOz8Fi6hg@mail.gmail.com>
Message-ID: <539ADCCE.30203@rfk.id.au>

On 12/06/2014 2:09 AM, Daniel Holth wrote:
> This is in the bug tracker already. We need to add the py2-none-arch
> tags etc. to Pip's list.

Thanks Daniel, can you please link me to the specific bug?  I couldn't
find it in the pip issue tracker, but I could easily have been looking
wrong.

  Cheers,

    Ryan

> On Jun 11, 2014 12:05 PM, "Chris Barker" <chris.barker at noaa.gov
> <mailto:chris.barker at noaa.gov>> wrote:
> 
>     Folks,
> 
>     I'm trying to help figure out how to do binary wheels for a package
>     that relies on ctypes and a bundles shared lib (dll, .so. etc)
> 
>     The trick here is that the python code is quite version and platform
>     independent: py2 and py3, version 2.7 and 3.3+ (I think)
> 
>     (it's py_enchant, if anyone is
>     interested: http://pythonhosted.org/pyenchant/)
> 
>     So the trick is that the binary wheel will be platform dependent,
>     but not the code itself, so ideally we'd have one wheel, that for
>     instance (and teh case at hand) should work on any OS-X box version
>     10.6 and above, with a any of python2.7, 3.3, 3.4 (an up?)
> 
>     Usually, a binary wheel involves  compiled extensions, and thus is
>     tied to a particular python version -- so this is an odd case.
> 
>     We tried:
> 
>     pyenchant-1.6.6-py2.py3-none-macosx_10_6_intel.whl 
> 
>     which seems to be saying: any version of python2 or python 3, but
>     only on macosx 10.6
> 
>     but trying to install that on my machine (py2.7, os-x 10.6) gives:
> 
>     pyenchant-1.6.6-py2.py3-none-macosx_10_6_intel.whl is not a
>     supported wheel on this platform.
> 
>     (side note: it would be really great if that could be a more useful
>     message -- what part of the file name didn't match? I know that's a
>     trick, as there is a whole pile of heuristics to go through, but
>     maybe a way to dump that process would be helpful...)
> 
>     Now, this may, in fat be tied to CPython (I have no idea if ctypes
>     is available on pypy or jython or IronPython...). So I tried:
> 
>     pyenchant-1.6.6-cp27-none-macosx_10_6_intel.whl 
> 
>     that does, indeed, install on my system.
> 
>     Also:
> 
>     pyenchant-1.6.6-cp27.cp33-none-macosx_10_6_intel.whl
> 
>     works.
> 
>     As 2.7 is really the only py2 that much matters, no biggie, but is
>     there a way to get 3.3 and 3.4 and ??? all at once (I don't have py3
>     on that machine, so didn't test that...)
> 
>     So: how should this be done? Is the above the best option there is?
> 
>     Thanks,
>       -Chris


From ncoghlan at gmail.com  Sun Jun 15 05:42:47 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 15 Jun 2014 13:42:47 +1000
Subject: [Distutils] setuptools wish list
In-Reply-To: <CACyo8__S2Q4dbmbSD=+5H=PJmvtmWds_1+OvOxpnH2BdmBC80Q@mail.gmail.com>
References: <CACyo8_8XSncuMb9-Gyf87+2oN5hjyQxVLDgq-ZviD__oP47daw@mail.gmail.com>
 <CACyo8__S2Q4dbmbSD=+5H=PJmvtmWds_1+OvOxpnH2BdmBC80Q@mail.gmail.com>
Message-ID: <CADiSq7enpXj=WB+eoOdRmhvJd0APtHcuSOiubZ3Fz+-zqkwJ+g@mail.gmail.com>

On 15 June 2014 10:25, lunasspecto at gmail.com <lunasspecto at gmail.com> wrote:
> So my wish list for setuptools or a similar tool is:
>
> *
>
> Automated XDG menu entry generation
>
> *
>
> Some way to handle dependencies that can't be installed by pip
>
> *
>
> Automated generation of Debian binary packages good enough to be accepted
> into the repositories
>
> *
>
> Automated generation of rpm packages with all the right dependencies (For
> all I know this could exist already, but considering the way dependencies
> are currently handled by setuptools I really doubt it)

These are all dependent on having platform specific extensions in
metadata 2.0 to allow the platform specific metadata tweaks to be
added without needing to manually duplicate the upstream metadata. The
main focus there at the moment is in getting the PEP 440 versioning
spec to a releasable state (Donald Stufft has been working on ensuring
that can be integrated sensibly into pip, and there's a major update
to all the draft metadata 2.0 PEPs coming soon based on that work).

Once 440 is resolved satisfactorily, the focus will shift to PEP 426
itself, along with the initial iteration of the "standard extensions"
in 459. With install hooks postponed for the time being (in favour of
the more general "required extension" mechanism) most of the open
questions there are actually around PEP 440, although the proposed
design of the "constraints" based install time integrity checking
model will need review.

> *
> In a perfect universe, easy cross-building functionality, so I could produce
> a standalone Windows executable or an OS X app from the comfort of Xubuntu.

A build farm integrated into PyPI is also on the wishlist, but behind
the migration to Warehouse on the list of things to be worked on (it
also depends on the improved handling of build dependencies that is
part of the metadata 2.0 definition). Donald could provide a better
update than I can in terms of the current status of the Warehouse
migration, and the remaining blockers.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Jun 29 06:40:55 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 29 Jun 2014 14:40:55 +1000
Subject: [Distutils] zc.buildout & Docker container images
Message-ID: <CADiSq7cMzwMcYqeAndKudAEeBCao-OViwZCe4yhfXBF9dMcC0g@mail.gmail.com>

Hey all (but especially the zc.buildout users),

With zc.buildout being used to define repeatable application
deployments, and Docker images acting as a way to snapshot entire
execution environments for ease of deployment on any environment than
can run a container (which is basically any modern Linux at this
point, or will be once CentOS 7 is available), has any experimented
with using zc.buildout from a Dockerfile to define an image?

If yes, is it written up anywhere?

If not, is anyone intrigued enough by the idea to considering trying
it out and writing up the results as a blog post somewhere?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From jim at zope.com  Sun Jun 29 16:29:28 2014
From: jim at zope.com (Jim Fulton)
Date: Sun, 29 Jun 2014 10:29:28 -0400
Subject: [Distutils] zc.buildout & Docker container images
In-Reply-To: <CADiSq7cMzwMcYqeAndKudAEeBCao-OViwZCe4yhfXBF9dMcC0g@mail.gmail.com>
References: <CADiSq7cMzwMcYqeAndKudAEeBCao-OViwZCe4yhfXBF9dMcC0g@mail.gmail.com>
Message-ID: <CAPDm-FgVGDOAN2f5Sypv=j+LELk-oCrdZF777om52DWXftDTmw@mail.gmail.com>

On Sun, Jun 29, 2014 at 12:40 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Hey all (but especially the zc.buildout users),
>
> With zc.buildout being used to define repeatable application
> deployments, and Docker images acting as a way to snapshot entire
> execution environments for ease of deployment on any environment than
> can run a container (which is basically any modern Linux at this
> point, or will be once CentOS 7 is available), has any experimented
> with using zc.buildout from a Dockerfile to define an image?

Yes.

> If yes, is it written up anywhere?

No. :)

A challenge for us has been that our apps typically register themselves
with ZooKeeper and, until recently, this has been really awkward
with Docker. Now that you can share host networking with containers,
I look forward to making more use of Docker in the future.

For now, we've packaged a number of legacy applications,
including svn.zope.org, as docker containers.

> If not, is anyone intrigued enough by the idea to considering trying
> it out and writing up the results as a blog post somewhere?

Absolutely, although I'm not sure when.

In some ways, docker potentially reduces the need for tools
like buildout and virtualenv to provide isolation. With a docker
image, you don't share the system Python with other apps, so
it's not as problematic to use it.

You also don't need tools to automate deployment of production
configurations when an application is deployed, as this is mostly done
when building an image.  The isolation provided by docker containers
also allows configuration to be simpler. There's still benefit of having
a system like buildout with Python recipes to automate assembly
(often including non-python bits like JavaScript libraries) and
configuration.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From batoot2022 at gmail.com  Sun Jun 29 03:12:07 2014
From: batoot2022 at gmail.com (batoot zilzel)
Date: Sun, 29 Jun 2014 04:12:07 +0300
Subject: [Distutils] question
Message-ID: <CABmZAW3dFttAumWdmq9_XQ=W-HcAX6+27q_XnocZb82pyYkzUA@mail.gmail.com>

hi am blind and am python programmer but i can't write the codes from
python idl should i use notepat ++ and some plugins to write and run
the codes cause the python idlis not accessible with blind and with
the screen reader i hope to hear from you  as soon as   thanks regards

From brett at python.org  Mon Jun 30 14:30:41 2014
From: brett at python.org (Brett Cannon)
Date: Mon, 30 Jun 2014 12:30:41 +0000
Subject: [Distutils]  question
References: <CABmZAW3dFttAumWdmq9_XQ=W-HcAX6+27q_XnocZb82pyYkzUA@mail.gmail.com>
Message-ID: <CAP1=2W606yNzfTwi834OBZ6HR4WkkajhzGYEs_yiga26j3_U9w@mail.gmail.com>

On 1404130668299, batoot zilzel <batoot2022 at gmail.com> wrote:

> hi am blind and am python programmer but i can't write the codes from
> python idl should i use notepat ++ and some plugins to write and run
> the codes cause the python idlis not accessible with blind and with
> the screen reader i hope to hear from you  as soon as   thanks regards
>

The best place for this kind of question is the python-list or python-help
mailing lists. But to answer your question, you can write Python in
anything; IDLE is there just as a convenience and is not required.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140630/ce349072/attachment.html>

From ncoghlan at gmail.com  Mon Jun 30 17:56:02 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 1 Jul 2014 01:56:02 +1000
Subject: [Distutils] zc.buildout & Docker container images
In-Reply-To: <CAPDm-FgVGDOAN2f5Sypv=j+LELk-oCrdZF777om52DWXftDTmw@mail.gmail.com>
References: <CADiSq7cMzwMcYqeAndKudAEeBCao-OViwZCe4yhfXBF9dMcC0g@mail.gmail.com>
 <CAPDm-FgVGDOAN2f5Sypv=j+LELk-oCrdZF777om52DWXftDTmw@mail.gmail.com>
Message-ID: <CADiSq7dZidd9Fn7bvrTksE2T9Km+qmxXYwzT9tOfWhng4uHztQ@mail.gmail.com>

On 29 Jun 2014 07:29, "Jim Fulton" <jim at zope.com> wrote:
>
> You also don't need tools to automate deployment of production
> configurations when an application is deployed, as this is mostly done
> when building an image.  The isolation provided by docker containers
> also allows configuration to be simpler. There's still benefit of having
> a system like buildout with Python recipes to automate assembly
> (often including non-python bits like JavaScript libraries) and
> configuration.

Yeah, it's the "you still need a way to define what goes into the image"
part that intrigues me with respect to combining tools like zc.buildout
with Docker.

It's an aspect I've seen folks sometimes miss in the discussions around
containers in general - the existing tech for repeatable deployments
doesn't necessarily go away, you just run it at image build time rather
than deployment time.

Cheers,
Nick.

>
> Jim
>
> --
> Jim Fulton
> http://www.linkedin.com/in/jimfulton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140701/b4482a88/attachment.html>