From xoviat at gmail.com  Sun Oct  1 13:53:22 2017
From: xoviat at gmail.com (xoviat)
Date: Sun, 1 Oct 2017 12:53:22 -0500
Subject: [Distutils] Extracting distutils into setuptools
In-Reply-To: <C504A254-8B0C-4848-A37E-3EC610FE0FD3@stufft.io>
References: <CAKXb8mH-zQ0hK=cGAbbh6UySaEFzHVkk4nkwigmpNbRoQUm4TQ@mail.gmail.com>
 <CADiSq7fiCdPYoPP3-BEPFWEUQnFbJv7scmtBcd-aJYHadfzscQ@mail.gmail.com>
 <d4c82132-f2ca-2d4e-fc5a-c5a31c7f3270@ubuntu.com>
 <CADiSq7e9OGrTaarFeEHfHEVXJJ=Uh6_P31rE7LDimVPE5SVHZQ@mail.gmail.com>
 <EC9383EE-59AD-45F9-8657-98312531DD29@stufft.io>
 <7714677f-0443-2cef-b465-4e600be11133@python.org>
 <98F10EB7-60CF-4C06-9CB4-B68B32B8D4B1@stufft.io>
 <CAKXb8mHSazJ_FGuojw1cCCDmY2yWy+jNMt4W3Q07Vahg83F4vw@mail.gmail.com>
 <97EEEF28-6C43-4325-90A3-EC5F6395BFEF@stufft.io>
 <CAKXb8mEEAUDHWiX0gwQnxiCBMB_2Bx-Ow7VEhpgdKRojcP9_Ng@mail.gmail.com>
 <7EB1D0E7-941E-4261-A38F-4904251EA4FC@stufft.io>
 <CAKXb8mHZn8CoX46RPkhCQYnZnsWki9LKZ67N4krdgk7t0fDUhQ@mail.gmail.com>
 <CAKXb8mGYgSoDT4BsnQZw9qNid60o9KhxuG=wLtARZfSdbw++Hw@mail.gmail.com>
 <C504A254-8B0C-4848-A37E-3EC610FE0FD3@stufft.io>
Message-ID: <CAKXb8mH7zg7MF2h_7ePR94zKOTiTh2UZszdcg0rQ+NO+NGc+CQ@mail.gmail.com>

After thinking again about that possibilities that we've discussed here, I
realized that a previously proposed alternative would eliminate external
build-time dependencies and allow us to merge setuptools with distutils: an
"ensuresetuptools" module. This was proposed by @zooba, but basically the
idea would be to bundle a wheel of setuptools (setuptools is
py2.py3.none-any) that Python could install without requiring network
access or other modules. If distutils is required during the build process,
then this idea should conform to all of the requirements proposed here and
take distutils off of the CPython release schedule.

Although it may seem more complicated at first, once implemented,
maintenance from the CPython side would be minimal (a bot could update the
wheel, although I'm not sure whether this is done with ensurepip), there is
precedent for this (with ensurepip), end-users could continue to use
distutils without any modifications to their scripts (ensuresetuptools
could be run during the installation process, and even if it isn't, then a
single command line could install distutils/setuptools), and it would allow
simplified setuptools maintenance (no monkeypatching).

2017-09-30 20:14 GMT-05:00 Donald Stufft <donald at stufft.io>:

> I think that the CPython builds a python executable, then uses that built
> executable to finish the installation.
>
>
> On Sep 30, 2017, at 9:11 PM, xoviat <xoviat at gmail.com> wrote:
>
> It would be nice to know whether this information is correct, or whether I
> hold an invalid belief.
>
> 2017-09-30 20:09 GMT-05:00 xoviat <xoviat at gmail.com>:
>
>> I have personally not built Python myself (though I've built many an
>> extension), but what I can say is that I got the idea from Larry Hastings.
>> According to him (this if for the Gilectomy fork):
>>
>> "Second, as you hack on the Gilectomy you may break your "python"
>> executable rather badly. This is of course expected. However, the python
>> Makefile itself depends on having a working local python interpreter, so
>> when you break that you often break your build too."
>>
>> 2017-09-30 19:59 GMT-05:00 Donald Stufft <donald at stufft.io>:
>>
>>>
>>>
>>> On Sep 30, 2017, at 3:52 PM, xoviat <xoviat at gmail.com> wrote:
>>>
>>> I don't think CPython needs to bundle all of its build-time
>>> dependencies. That principle doesn't really apply to other Python programs
>>> nor most other programs in general. AFAIK, CPython already has a build-time
>>> dependency on another, external, Python, so it wouldn't be too much to
>>> require the external Python to have setuptools installed with something
>>> like pyproject.toml (other programming languages usually bootstrap
>>> themselves with previous versions of the language along with some
>>> associated build tools).
>>>
>>>
>>> As far as I can tell, CPython does *not* have a build time dependency on
>>> having Python available. I just spun up a bare alpine linux container and
>>> compiled CPython `master` branch on it. As far as I can tell the only
>>> Python that exists in this container is the one I just compiled.
>>>
>>> That means that in order for CPython to depend on distutils to build as
>>> you indicate, it would also need to start depending on an existing version
>>> of Python being available. I don?t think that?s a great idea. I think
>>> Python should not depend on Python to build.
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171001/8b909a35/attachment.html>

From donald at stufft.io  Sun Oct  1 14:55:39 2017
From: donald at stufft.io (Donald Stufft)
Date: Sun, 1 Oct 2017 14:55:39 -0400
Subject: [Distutils] Extracting distutils into setuptools
In-Reply-To: <CAKXb8mH7zg7MF2h_7ePR94zKOTiTh2UZszdcg0rQ+NO+NGc+CQ@mail.gmail.com>
References: <CAKXb8mH-zQ0hK=cGAbbh6UySaEFzHVkk4nkwigmpNbRoQUm4TQ@mail.gmail.com>
 <CADiSq7fiCdPYoPP3-BEPFWEUQnFbJv7scmtBcd-aJYHadfzscQ@mail.gmail.com>
 <d4c82132-f2ca-2d4e-fc5a-c5a31c7f3270@ubuntu.com>
 <CADiSq7e9OGrTaarFeEHfHEVXJJ=Uh6_P31rE7LDimVPE5SVHZQ@mail.gmail.com>
 <EC9383EE-59AD-45F9-8657-98312531DD29@stufft.io>
 <7714677f-0443-2cef-b465-4e600be11133@python.org>
 <98F10EB7-60CF-4C06-9CB4-B68B32B8D4B1@stufft.io>
 <CAKXb8mHSazJ_FGuojw1cCCDmY2yWy+jNMt4W3Q07Vahg83F4vw@mail.gmail.com>
 <97EEEF28-6C43-4325-90A3-EC5F6395BFEF@stufft.io>
 <CAKXb8mEEAUDHWiX0gwQnxiCBMB_2Bx-Ow7VEhpgdKRojcP9_Ng@mail.gmail.com>
 <7EB1D0E7-941E-4261-A38F-4904251EA4FC@stufft.io>
 <CAKXb8mHZn8CoX46RPkhCQYnZnsWki9LKZ67N4krdgk7t0fDUhQ@mail.gmail.com>
 <CAKXb8mGYgSoDT4BsnQZw9qNid60o9KhxuG=wLtARZfSdbw++Hw@mail.gmail.com>
 <C504A254-8B0C-4848-A37E-3EC610FE0FD3@stufft.io>
 <CAKXb8mH7zg7MF2h_7ePR94zKOTiTh2UZszdcg0rQ+NO+NGc+CQ@mail.gmail.com>
Message-ID: <CE63276E-77B7-4DD5-897B-947A1C59D3E0@stufft.io>



> On Oct 1, 2017, at 1:53 PM, xoviat <xoviat at gmail.com> wrote:
> 
> After thinking again about that possibilities that we've discussed here, I realized that a previously proposed alternative would eliminate external build-time dependencies and allow us to merge setuptools with distutils: an "ensuresetuptools" module. This was proposed by @zooba, but basically the idea would be to bundle a wheel of setuptools (setuptools is py2.py3.none-any) that Python could install without requiring network access or other modules. If distutils is required during the build process, then this idea should conform to all of the requirements proposed here and take distutils off of the CPython release schedule.


This isn?t as easy as ensurepip, because ensurepip can wait until the end of the build process when the entire Python installation has been built. The same isn?t true for a hypothetical ensuresetuptools module. This is because we end up in a circular dependency, if installing a wheel requires a C extension (like say zlib) then we can?t install that wheel prior to building zlib, but if we need to install that wheel to build zlib then we end up stuck.

I?m not sure what all c-extensions are used in the process of installing a wheel? certainly zlib is but we could maybe build a special wheel that only uses the STORED algorithm and doesn?t do compression (does the zipfile module work with zlib doesn?t exist?). I?m going to guess there are others though and no idea if they are able to be avoided.

I still think a better idea if we want to go down that route is to modify the CPython build process to not depend on distutils at all.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171001/75589eeb/attachment.html>

From ncoghlan at gmail.com  Mon Oct  2 01:31:16 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 2 Oct 2017 15:31:16 +1000
Subject: [Distutils] Reproducible builds (Sdist)
In-Reply-To: <1506715336.3233190.1122879608.5840B3BD@webmail.messagingengine.com>
References: <CANJQusVp8sBOX3n0OamU3v8RJjOym2ovyNBKqJyxxgoKthce=Q@mail.gmail.com>
 <1506715336.3233190.1122879608.5840B3BD@webmail.messagingengine.com>
Message-ID: <CADiSq7cFxZRKqJqZ5=n21+Q_cQqpU4xtVFV+tBPrKms7SAKdbw@mail.gmail.com>

On 30 September 2017 at 06:02, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> On Fri, Sep 29, 2017, at 07:16 PM, Matthias Bussonnier wrote:
>> Second; is there a convention to store the SDE value ? I don't seem to
>> be able to find one. It is nice to have reproducible build; but if
>> it's a pain for reproducers to find the SDE value that highly decrease
>> the value of SDE build.
>
> Does it make sense to add a new optional metadata field to store the
> value of SOURCE_DATE_EPOCH if it's set when a distribution is built? I
> guess it could cause problems if unpacking & repacking a tarball means
> that its metadata is no longer accurate, though.

For distro level reproducible build purposes, we typically treat the
published tarball *as* the original sources, and don't really worry
about the question of "Can we reproduce that tarball, from that VCS
tree?".

This stems from the original model of open source distribution, where
publication *was* a matter of putting a tarball up on a website
somewhere, and it was an open question as to whether or not the
publisher was even using a version control system at all (timeline:
RCS=1982, CVS=1986, SVN=2000, git/hg=2005, with Linux distributions
getting their start in the early-to-mid 1990's).

So SOURCE_DATE_EPOCH gets applied *after* unpacking the original
tarball, rather than being used to *create* the tarball (we already
know when the publisher created it, since that's part of the tarball
metadata).

Python's sdists mess with that assumption a bit, since it's fairly
common to include generated C files that aren't part of the original
source tree, and Cython explicitly recommends doing so in order to
avoid requiring Cython as a build time dependency:
http://docs.cython.org/en/latest/src/reference/compilation.html#distributing-cython-modules

So in many ways, this isn't the problem that SOURCE_DATE_EPOCH on its
own is designed to solve - instead, it's asking the question of "How
do I handle the case where my nominal source archive is itself a built
artifact?", which means you not only need to record source timestamps
of the original inputs you used to build the artifact (which the
version control system will give you), you also need to record details
of the build tools used (e.g. using a different version of Cython will
generate different code, and hence different "source" archives), and
decide what to do with any timestamps on the *output* artifacts you
generate (e.g. you may decide to force them to match the commit date
from the VCS).

So saying "SOURCE_DATE_EPOCH will be set to the VCS commit date when
creating an sdist" would be a reasonable thing for an sdist creation
tool to decide to do, and combined with something like `Pipfile.lock`
in `pipenv`, or a `dev-requirements.txt` with fully pinned versions,
*would* go a long way towards giving you reproducible sdist archives.

However, it's not a problem to be solved by adding anything to the
produced sdist: it's a property of the publishing tools that create
sdists to aim to ensure that given the same inputs, on a different
machine, at a different time, you will nevertheless still get the same
result.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Mon Oct  2 02:03:09 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 2 Oct 2017 16:03:09 +1000
Subject: [Distutils] Extracting distutils into setuptools
In-Reply-To: <7EB1D0E7-941E-4261-A38F-4904251EA4FC@stufft.io>
References: <CAKXb8mH-zQ0hK=cGAbbh6UySaEFzHVkk4nkwigmpNbRoQUm4TQ@mail.gmail.com>
 <CADiSq7fiCdPYoPP3-BEPFWEUQnFbJv7scmtBcd-aJYHadfzscQ@mail.gmail.com>
 <d4c82132-f2ca-2d4e-fc5a-c5a31c7f3270@ubuntu.com>
 <CADiSq7e9OGrTaarFeEHfHEVXJJ=Uh6_P31rE7LDimVPE5SVHZQ@mail.gmail.com>
 <EC9383EE-59AD-45F9-8657-98312531DD29@stufft.io>
 <7714677f-0443-2cef-b465-4e600be11133@python.org>
 <98F10EB7-60CF-4C06-9CB4-B68B32B8D4B1@stufft.io>
 <CAKXb8mHSazJ_FGuojw1cCCDmY2yWy+jNMt4W3Q07Vahg83F4vw@mail.gmail.com>
 <97EEEF28-6C43-4325-90A3-EC5F6395BFEF@stufft.io>
 <CAKXb8mEEAUDHWiX0gwQnxiCBMB_2Bx-Ow7VEhpgdKRojcP9_Ng@mail.gmail.com>
 <7EB1D0E7-941E-4261-A38F-4904251EA4FC@stufft.io>
Message-ID: <CADiSq7e3EDhkXfzJfJAkdeOd521+u3ZBXye036TXJWj-3SYQeQ@mail.gmail.com>

On 1 October 2017 at 10:59, Donald Stufft <donald at stufft.io> wrote:
>
>
> On Sep 30, 2017, at 3:52 PM, xoviat <xoviat at gmail.com> wrote:
>
> I don't think CPython needs to bundle all of its build-time dependencies.
> That principle doesn't really apply to other Python programs nor most other
> programs in general. AFAIK, CPython already has a build-time dependency on
> another, external, Python, so it wouldn't be too much to require the
> external Python to have setuptools installed with something like
> pyproject.toml (other programming languages usually bootstrap themselves
> with previous versions of the language along with some associated build
> tools).
>
>
> As far as I can tell, CPython does *not* have a build time dependency on
> having Python available. I just spun up a bare alpine linux container and
> compiled CPython `master` branch on it. As far as I can tell the only Python
> that exists in this container is the one I just compiled.
>
> That means that in order for CPython to depend on distutils to build as you
> indicate, it would also need to start depending on an existing version of
> Python being available. I don?t think that?s a great idea. I think Python
> should not depend on Python to build.

CPython's self-bootstrapping is a bit complicated :)

>From a regular checkout, the way it works is:

1. We build CPython itself, including all frozen and builtin modules,
since the source code for the frozen modules is checked in, as is the
code generated by Argument Clinic
2. We use distutils & that just built CPython to build a bunch of
optional extension modules

For working on the compiler itself, we also have an extra step where
we build CPython with no import system, use that to refreeze importlib
with the modified compiler, then rebuild CPython with the recompiled
import system.

Breaking the code to run the code generators can thus break your
build, although that should be less common now that we require
execution of the code generators to be requested explicitly (and rely
on CI to check when they haven't been run when they should).

However, there's *nothing* in that arrangement which says we can't use
a virtual environment to do the optional extension module builds, and
plenty of good arguments to say that we *should* be using a virtual
environment for those.

And if we move them into a virtual environment, then *those builds*
will still have access to distutils, regardless of whether we're
picking it up from Lib/distutils (which is what we currently do), or
from an installed setuptools (which is what I'd propose we switch to
for 3.8+).

As far as PEP 4 goes, I think the constraint that places on us is that
*after installation of CPython*, then "import distutils" has to
continue working, just as it does today.

I *don't* think it places any requirement on us to ensure that
"Lib/distutils" is a thing that exists in the CPython source repo -
I'd consider it entirely acceptable for us to install it into
site-packages with `pip` (PEP 453 style), and have it be missing by
default from things like the embeddable Windows distribution.

As far as the intended pay-off goes, this is entirely aimed at reduce
the level of work for the current setuptools maintainers, as they have
a complex bug triage process that requires them to do something like
this:

1. Figure out if something is a distutils bug or a setuptools bug
2. If it's a distutils bug, figure out if setuptools can work around it
3. Regardless of whether setuptools can work around it or not, file a
bug against distutils on b.p.o
4. Decide which branches to fix the distutils bug on
5. If setuptools can't work around it, tell people they'll have to
upgrade CPython to fix their problem

setuptools forking distutils wholesale simplifies that whole triage &
process to:

1. It doesn't matter if the problem is a distutils bug or a setuptools
bug, fix it in the setuptools code base
2. Ship a new release of setuptools to make the fix available to users

However, if setuptools *does* fork distutils wholesale, then the
already low level of contributions to distutils maintenance are likely
to drop to essentially zero, at which point the question becomes "How
can we switch *CPython itself* to using and providing the
independently upgradable setuptools fork instead of the Lib/distutils
version?"

And there, PEP 453 has already established a relevant precedent: we
can bundle setuptools as a wheel file, with a helper module to request
installation of that wheel file's contents into the current
environment.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Mon Oct  2 02:18:51 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 2 Oct 2017 16:18:51 +1000
Subject: [Distutils] Extracting distutils into setuptools
In-Reply-To: <CE63276E-77B7-4DD5-897B-947A1C59D3E0@stufft.io>
References: <CAKXb8mH-zQ0hK=cGAbbh6UySaEFzHVkk4nkwigmpNbRoQUm4TQ@mail.gmail.com>
 <CADiSq7fiCdPYoPP3-BEPFWEUQnFbJv7scmtBcd-aJYHadfzscQ@mail.gmail.com>
 <d4c82132-f2ca-2d4e-fc5a-c5a31c7f3270@ubuntu.com>
 <CADiSq7e9OGrTaarFeEHfHEVXJJ=Uh6_P31rE7LDimVPE5SVHZQ@mail.gmail.com>
 <EC9383EE-59AD-45F9-8657-98312531DD29@stufft.io>
 <7714677f-0443-2cef-b465-4e600be11133@python.org>
 <98F10EB7-60CF-4C06-9CB4-B68B32B8D4B1@stufft.io>
 <CAKXb8mHSazJ_FGuojw1cCCDmY2yWy+jNMt4W3Q07Vahg83F4vw@mail.gmail.com>
 <97EEEF28-6C43-4325-90A3-EC5F6395BFEF@stufft.io>
 <CAKXb8mEEAUDHWiX0gwQnxiCBMB_2Bx-Ow7VEhpgdKRojcP9_Ng@mail.gmail.com>
 <7EB1D0E7-941E-4261-A38F-4904251EA4FC@stufft.io>
 <CAKXb8mHZn8CoX46RPkhCQYnZnsWki9LKZ67N4krdgk7t0fDUhQ@mail.gmail.com>
 <CAKXb8mGYgSoDT4BsnQZw9qNid60o9KhxuG=wLtARZfSdbw++Hw@mail.gmail.com>
 <C504A254-8B0C-4848-A37E-3EC610FE0FD3@stufft.io>
 <CAKXb8mH7zg7MF2h_7ePR94zKOTiTh2UZszdcg0rQ+NO+NGc+CQ@mail.gmail.com>
 <CE63276E-77B7-4DD5-897B-947A1C59D3E0@stufft.io>
Message-ID: <CADiSq7cr+mGECSXp4JvRK7rpQM2=07jfnH9dPwugOKCrkkoOpQ@mail.gmail.com>

On 2 October 2017 at 04:55, Donald Stufft <donald at stufft.io> wrote:
> This isn?t as easy as ensurepip, because ensurepip can wait until the end of
> the build process when the entire Python installation has been built. The
> same isn?t true for a hypothetical ensuresetuptools module. This is because
> we end up in a circular dependency, if installing a wheel requires a C
> extension (like say zlib) then we can?t install that wheel prior to building
> zlib, but if we need to install that wheel to build zlib then we end up
> stuck.
>
> I?m not sure what all c-extensions are used in the process of installing a
> wheel? certainly zlib is but we could maybe build a special wheel that only
> uses the STORED algorithm and doesn?t do compression (does the zipfile
> module work with zlib doesn?t exist?). I?m going to guess there are others
> though and no idea if they are able to be avoided.
>
> I still think a better idea if we want to go down that route is to modify
> the CPython build process to not depend on distutils at all.

The main reason for using distutils is to reduce the need to define
different build instructions for different platforms for things like
curses, sqlite, cdecimal, dbm, Tcl/Tk, etc. (And even when we do need
to define separate instructions for Windows, we still get to avoid
running those builds through autotools)

That said, we do have other extension modules that *are* handled by
the platform specific build instructions, and it should be relatively
straightforward to migrate anything that `ensuresetuptools` needed
down to that layer (and it's likely only `zlib` that falls into that
category).

Cheers,
Nick.

P.S. I'm honestly not sure how well CPython in general works without
zlib: while it's nominally optional in the build process, and we have
a "test.support.requires_zlib" decorator, I don't think we run any
buildbots without it, and we statically link it on Windows and Mac OS
X.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From bussonniermatthias at gmail.com  Mon Oct  2 11:20:18 2017
From: bussonniermatthias at gmail.com (Matthias Bussonnier)
Date: Mon, 2 Oct 2017 08:20:18 -0700
Subject: [Distutils] Reproducible builds (Sdist)
In-Reply-To: <CADiSq7cFxZRKqJqZ5=n21+Q_cQqpU4xtVFV+tBPrKms7SAKdbw@mail.gmail.com>
References: <CANJQusVp8sBOX3n0OamU3v8RJjOym2ovyNBKqJyxxgoKthce=Q@mail.gmail.com>
 <1506715336.3233190.1122879608.5840B3BD@webmail.messagingengine.com>
 <CADiSq7cFxZRKqJqZ5=n21+Q_cQqpU4xtVFV+tBPrKms7SAKdbw@mail.gmail.com>
Message-ID: <CANJQusWK8nJDUYmrXr_dB5usaE7oAYqzGND3=QjOhspBZYj2PQ@mail.gmail.com>

Hi all,

On Fri, Sep 29, 2017 at 12:04 PM, Jakub Wilk <jwilk at jwilk.net> wrote:
> It not enough to normalize timestamps. You need to normalize permissions and
> ownership, too.
>
> (I'm using https://pypi.python.org/pypi/distutils644 for normalizing
> permissions/ownership in my own packages.)
>
Thanks Jakub this will be helpful for me;

> Yeah, I don't believe distutils honors SOURCE_DATE_EPOCH at the moment.
>
>> Second; is there a convention to store the SDE value ?
>
> In the changelog.

I'll consider that as well;


On Sun, Oct 1, 2017 at 10:31 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 30 September 2017 at 06:02, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
>> On Fri, Sep 29, 2017, at 07:16 PM, Matthias Bussonnier wrote:
>
> For distro level reproducible build purposes, we typically treat the
> published tarball *as* the original sources, and don't really worry
> about the question of "Can we reproduce that tarball, from that VCS
> tree?".

Thanks for the detail explanation Nick, even if this was not the
original goal of SDE,
I would still like to have it reproducible build of sdist even if my package
does not have source generation like Cython;  I'll embed the timestamp in the
commit for now;  and see if I can also extract the timestamp from the
commit log.
AFAICT it's `git log -1 --pretty=format:%ct` if it's of interest to anyone.

My interest in this is to have CI to build the sdist, and make sure independant
machines can get the same artifact in order to have a potentially distributed
agreement on what the sdist is.

Is there any plan (or would it be accepted), to try to upstream patches like
distutils644 Jakub linked to ?

Thanks,
-- 
Matthias

From ncoghlan at gmail.com  Tue Oct  3 11:06:07 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 4 Oct 2017 01:06:07 +1000
Subject: [Distutils] Migrating interoperability specs to
 packaging.python.org
In-Reply-To: <CADiSq7do60dUVa-9DXYTx4HMUEvCYoKEQTQb4+5-_zHoHgnW9g@mail.gmail.com>
References: <CADiSq7do60dUVa-9DXYTx4HMUEvCYoKEQTQb4+5-_zHoHgnW9g@mail.gmail.com>
Message-ID: <CADiSq7eFFpwrDMCerdLYGAqvc4DzuwJ7MXRmUx4+Q+gzO9eLvg@mail.gmail.com>

On 4 September 2017 at 23:33, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Some time ago, I started the process [1] of adjusting how
> distutils-sig uses the PEP process so that the reference
> specifications will live on packaging.python.org, and we use the PEP
> process to manage *changes* to those specifications, rather than
> serving as the specifications themselves (that is, adopting a process
> closer to the URL-centric way the Python language reference is
> managed, rather than using the RFCstyle PEP-number-centric model the
> way we do now).

I'm happy to report that Dustin Ingram has started making some
progress on this change, with
https://packaging.python.org/specifications/ now broken out into a set
of distinct subpages with appropriate URLs, and
https://packaging.python.org/specifications/core-metadata/ now
coversing *all* the fields that can appear in PKG-INFO and METADATA
files, not just the ones that are post-1.2 additions.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed Oct  4 00:11:45 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 4 Oct 2017 14:11:45 +1000
Subject: [Distutils] Migrating interoperability specs to
 packaging.python.org
In-Reply-To: <CADiSq7do60dUVa-9DXYTx4HMUEvCYoKEQTQb4+5-_zHoHgnW9g@mail.gmail.com>
References: <CADiSq7do60dUVa-9DXYTx4HMUEvCYoKEQTQb4+5-_zHoHgnW9g@mail.gmail.com>
Message-ID: <CADiSq7fS72X1pz_+-NdwWL1svpA2E1dALPq=3k73ao43qob4-Q@mail.gmail.com>

On 4 September 2017 at 23:33, Nick Coghlan <ncoghlan at gmail.com> wrote:
> However, I'm also wondering if it may still be worthwhile writing a
> metadata 1.3 PEP that does the following things:
>
> 1. Explicitly notes the addition of the two new fields
> 2. Describes the process change for packaging interoperability specifications
> 3. Defines a canonical transformation between the human-readable
> key:value format and a more automation friendly JSON format

Discussing this idea further with Dustin, I now think it's conflating
two different activities (one related to process management, one
related to actually updating the metadata specification).

So I'd suggest that any metadata 1.3 PEP restrict itself to only
covering the first point (more formally documenting the two fields
added since metadata 1.2) and the last point (defining a standard
translation from the Key:Value format to JSON and back).

For the process management question around improving traceability in
our spec management processes, I'm wondering if it may make sense to
write a meta-PEP called just "The Python Packaging Authority" that
attempts to better articulate what that term actually encompasses:

* The published interoperability specifications at
https://packaging.python.org/specifications/
* The process for updating those specifications at
https://www.pypa.io/en/latest/specifications/
* Ecosystem & specification level design discussions on distutils-sig
* Funding and resource allocation authority delegated from the PSF
Board to the Python Packaging Working Group (overseen by the PSF's
Director of Operations & Infrastructure Manager)
* PyPI operating authority delegated from the PSF Board to the PSF
Infrastructure team (overseen by the PSF's Director of Operations &
Infrastructure Manager)

Such a meta-PEP could also clearly document the limits of Guido's
original design delegation back at the 2013 Python Language Summit
(i.e. as soon as we need or want to change anything in CPython or the
standard library, then that's a change that needs to go through the
regular PEP process, not the PyPA/distutils-sig specific variant of
it)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From milindha at gmail.com  Tue Oct  3 15:27:48 2017
From: milindha at gmail.com (Milind Rangnekar)
Date: Tue, 3 Oct 2017 14:27:48 -0500
Subject: [Distutils] PIP not recognized in python shell
Message-ID: <CAKpe-28iObu5LvtBNVGOE1=n=Y9p9wkfaPP-igNzRT=mtJYAYw@mail.gmail.com>

Hello,

>From window's 10 command prompt, I can install pip and other packages:

C:\>pip

Usage:
  pip <command> [options]

Commands:
  install                     Install packages.
  uninstall                   Uninstall packages.
  freeze                      Output installed packages in requirements
format.
  list                        List installed packages.
  show                        Show information about installed packages.
  search                      Search PyPI for packages.
  wheel                       Build wheels from your requirements.
  zip                         DEPRECATED. Zip individual packages.
  unzip                       DEPRECATED. Unzip individual packages.
  help                        Show help for commands.

General Options:
  -h, --help                  Show help.
  --isolated                  Run pip in an isolated mode, ignoring
                              environment variables and user configuration.
  -v, --verbose               Give more output. Option is additive, and can
be
                              used up to 3 times.
  -V, --version               Show version and exit.
  -q, --quiet                 Give less output.
  --log <path>                Path to a verbose appending log.
  --proxy <proxy>             Specify a proxy in the form
                              [user:passwd@]proxy.server:port.
  --retries <retries>         Maximum number of retries each connection
should
                              attempt (default 5 times).
  --timeout <sec>             Set the socket timeout (default 15 seconds).
  --exists-action <action>    Default action when a path already exists:
                              (s)witch, (i)gnore, (w)ipe, (b)ackup.
  --trusted-host <hostname>   Mark this host as trusted, even though it does
                              not have valid or any HTTPS.
  --cert <path>               Path to alternate CA bundle.
  --client-cert <path>        Path to SSL client certificate, a single file
                              containing the private key and the certificate
                              in PEM format.
  --cache-dir <dir>           Store the cache data in <dir>.
  --no-cache-dir              Disable the cache.
  --disable-pip-version-check
                              Don't periodically check PyPI to determine
                              whether a new version of pip is available for
                              download. Implied with --no-index.

C:\>python -m pip install beautifulsoup4
You are using pip version 6.0.8, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting beautifulsoup4
  Downloading beautifulsoup4-4.6.0-py3-none-any.whl (86kB)
    100% |################################| 90kB 1.5MB/s
Installing collected packages: beautifulsoup4

Successfully installed beautifulsoup4-4.6.0

C:\>
 However, PIP is  not recognized in Python 3.4.3 Shell.

>>> pip install
SyntaxError: invalid syntax
>>> pip
Traceback (most recent call last):
  File "<pyshell#11>", line 1, in <module>
    pip
NameError: name 'pip' is not defined
>>> python -m pip install beautifulsoup4
SyntaxError: invalid syntax


Please advise.

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171003/322eb718/attachment.html>

From ncoghlan at gmail.com  Wed Oct  4 08:40:04 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 4 Oct 2017 22:40:04 +1000
Subject: [Distutils] PIP not recognized in python shell
In-Reply-To: <CAKpe-28iObu5LvtBNVGOE1=n=Y9p9wkfaPP-igNzRT=mtJYAYw@mail.gmail.com>
References: <CAKpe-28iObu5LvtBNVGOE1=n=Y9p9wkfaPP-igNzRT=mtJYAYw@mail.gmail.com>
Message-ID: <CADiSq7eHBOf4ePy8+99iEjPsav7FvNrWn087pChacpxNUbhVvA@mail.gmail.com>

On 4 October 2017 at 05:27, Milind Rangnekar <milindha at gmail.com> wrote:
>  However, PIP is  not recognized in Python 3.4.3 Shell.
>
>>>> pip install
> SyntaxError: invalid syntax
>>>> pip
> Traceback (most recent call last):
>   File "<pyshell#11>", line 1, in <module>
>     pip
> NameError: name 'pip' is not defined
>>>> python -m pip install beautifulsoup4
> SyntaxError: invalid syntax

This is expected behaviour, as the Python REPL is for running Python
code, not for running command line executables.

"pip" and "python" don't work in the Python REPL for the same reason
that attempting to run "dir", "cd" or "systeminfo" doesn't work -
they're shell commands rather than Python code.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From wes.turner at gmail.com  Wed Oct  4 09:58:04 2017
From: wes.turner at gmail.com (Wes Turner)
Date: Wed, 4 Oct 2017 08:58:04 -0500
Subject: [Distutils] PIP not recognized in python shell
In-Reply-To: <CADiSq7eHBOf4ePy8+99iEjPsav7FvNrWn087pChacpxNUbhVvA@mail.gmail.com>
References: <CAKpe-28iObu5LvtBNVGOE1=n=Y9p9wkfaPP-igNzRT=mtJYAYw@mail.gmail.com>
 <CADiSq7eHBOf4ePy8+99iEjPsav7FvNrWn087pChacpxNUbhVvA@mail.gmail.com>
Message-ID: <CACfEFw-7L1N3xf61zYSQKy28E86fBuqkYtHdKADdz4Czy4wDAA@mail.gmail.com>

With IPython:

!pip install -U pip

Or, with Python:

>> subprocess.check_call(['./pip', 'install'])
>> subprocess.check_call('pip install', shell=True) #*

https://docs.python.org/3/library/subprocess.html#security-considerations

... http://sarge.readthedocs.io/en/latest/tutorial.html

On Wednesday, October 4, 2017, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 4 October 2017 at 05:27, Milind Rangnekar <milindha at gmail.com
> <javascript:;>> wrote:
> >  However, PIP is  not recognized in Python 3.4.3 Shell.
> >
> >>>> pip install
> > SyntaxError: invalid syntax
> >>>> pip
> > Traceback (most recent call last):
> >   File "<pyshell#11>", line 1, in <module>
> >     pip
> > NameError: name 'pip' is not defined
> >>>> python -m pip install beautifulsoup4
> > SyntaxError: invalid syntax
>
> This is expected behaviour, as the Python REPL is for running Python
> code, not for running command line executables.
>
> "pip" and "python" don't work in the Python REPL for the same reason
> that attempting to run "dir", "cd" or "systeminfo" doesn't work -
> they're shell commands rather than Python code.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com <javascript:;>   |   Brisbane,
> Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org <javascript:;>
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171004/32c29ece/attachment-0001.html>

From g.rodola at gmail.com  Sat Oct  7 03:12:38 2017
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Sat, 7 Oct 2017 09:12:38 +0200
Subject: [Distutils] pythonhosted.org doc upload no longer works
In-Reply-To: <CAFYqXL-wFYS80-LSaoVB0WvjnHq4mVfP=OxJNLAS4jTWGgN6jA@mail.gmail.com>
References: <CAFYqXL9AXXTdgLfEwBbnh1DejZ1sRqNGp1Dh=o19Rgmqzvokxw@mail.gmail.com>
 <CADiSq7dPLj4J2TgScLscCr8QM3BasP9q9wsarKJecyARJq4BPw@mail.gmail.com>
 <CAF4280LMiC14PjPkACFFxsyshanVHkO5PX0azUdRvr-6cguOmQ@mail.gmail.com>
 <CAFYqXL-9i9MJjTX1Za22L4FqV5pZap1Qihv5Y909eZc0rNbZTg@mail.gmail.com>
 <C5823E59-AB8C-4DA8-AD4A-AE970B062061@coderanger.net>
 <CAFYqXL9kZDc+UicOq_W8POZ+nT5W+dBgFSSZ7NMkUSGAGE2xLA@mail.gmail.com>
 <CAFYqXL-wKDhG8h00esnXUuRXYRQXEdMmGZbx+FLgKvPLXUyaUQ@mail.gmail.com>
 <CAFYqXL-uHDVZ=Yd0puWy=rA4GFNa8aGk2xSLu3ZSxgNvXu7ocA@mail.gmail.com>
 <CAFYqXL9Eq+BrcaX4YBpJutBVU5A5ncYiFXCW=K8xzmnO-hwFNg@mail.gmail.com>
 <CACfEFw9K_4tnEFyVY1fABUp_HnWdFuB7a3513Tg1TyPxmfTZag@mail.gmail.com>
 <CAFYqXL-wFYS80-LSaoVB0WvjnHq4mVfP=OxJNLAS4jTWGgN6jA@mail.gmail.com>
Message-ID: <CAFYqXL8dZWo4KdVq2GQU-N137DrOLxVChXuX39TEC95P64eDDQ@mail.gmail.com>

Any news about this please? This is becoming a problem as my outdated doc
keeps being around and I have no way to remove it.

On Wed, Sep 13, 2017 at 2:39 PM, Giampaolo Rodola' <g.rodola at gmail.com>
wrote:

> Thanks. Can somebody please upload this HTML page on
> https://pythonhosted.org/psutil/?
>
> <!doctype html>
> <html>
>     <head>
>         <meta http-equiv="REFRESH" content="0; url=
> http://psutil.readthedocs.io" />
>         <title>Redirecting to http://psutil.readthedocs.io</title>
>     </head>
>     <body>
>     <h1>This page is outdated</h1>
>     <p>
>     Redirecting to <a href="http://psutil.readthedocs.io" rel="me">
> http://psutil.readthedocs.io</a>
>     </p>
>     </body>
> </html>
>
>
> On Fri, Sep 8, 2017 at 5:50 AM, Wes Turner <wes.turner at gmail.com> wrote:
>
>> Here's an http-equiv REFRESH index.html:
>>
>> https://github.com/westurner/resume/blob/master/index.html
>>
>> Is there a
>> Sphinx extension to generate redirect pages?
>>
>> On Thursday, September 7, 2017, Giampaolo Rodola' <g.rodola at gmail.com>
>> wrote:
>>
>>> Whops! I meant: http://pythonhosted.org/pyftpdlib/
>>>
>>> On Thu, Sep 7, 2017 at 5:28 PM, Giampaolo Rodola' <g.rodola at gmail.com>
>>> wrote:
>>>
>>>> ...and also http://pyftpdlib.readthedocs.io/
>>>> <http://psutil.readthedocs.io/> if possible; thanks a lot!
>>>>
>>>> On Thu, Sep 7, 2017 at 4:10 PM, Giampaolo Rodola' <g.rodola at gmail.com>
>>>> wrote:
>>>>
>>>>> OK I updated the new doc at http://psutil.readthedocs.io/
>>>>> Is it possible to redirect the old doc here?
>>>>>
>>>>> On Tue, Sep 5, 2017 at 3:31 PM, Giampaolo Rodola' <g.rodola at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> A bare redirect would also be fine. It's a shame setup.py classifiers
>>>>>> / metadata does not have a "documentation" field otherwise
>>>>>> pythonhosted.org/{project_name}
>>>>>> <http://pythonhosted.org/%7Bproject_name%7D> could have
>>>>>> automatically redirected to the project doc.
>>>>>> Perhaps https://pypi.python.org/pypi?:action=pkg_edit&name={
>>>>>> project_name} could provide a form where this can be specified.
>>>>>> What do you think?
>>>>>>
>>>>>> On Tue, Sep 5, 2017 at 12:49 PM, Noah Kantrowitz <noah at coderanger.net
>>>>>> > wrote:
>>>>>>
>>>>>>> This was discussed several years ago in
>>>>>>> https://mail.python.org/pipermail/distutils-sig/2015-May/026381.html
>>>>>>> and a few other threads. The final phases went out earlier this year. I
>>>>>>> don't think there is any plan to re-enable uploads to pythonhosted at this
>>>>>>> time. If you want a one-off redirect change or just having the old files
>>>>>>> removed, we can probably do that though, but I very much defer to Donald
>>>>>>> and others on that :)
>>>>>>>
>>>>>>> --Noah
>>>>>>>
>>>>>>> > On Sep 4, 2017, at 9:43 PM, Giampaolo Rodola' <g.rodola at gmail.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > I think it's wise to revert that commit. It seems pythonhosted
>>>>>>> only suggested to migrate to RTD but there never was an official shutdown
>>>>>>> date or warning (either via direct email or message on the web page).
>>>>>>> >
>>>>>>> > On Tue, Sep 5, 2017 at 12:19 PM, Berker Peksa? <
>>>>>>> berker.peksag at gmail.com> wrote:
>>>>>>> > On Mon, Sep 4, 2017 at 4:56 PM, Nick Coghlan <ncoghlan at gmail.com>
>>>>>>> wrote:
>>>>>>> > > On 2 September 2017 at 15:34, Giampaolo Rodola' <
>>>>>>> g.rodola at gmail.com> wrote:
>>>>>>> > >> I know it was deprecated long ago in favor of readthedocs but I
>>>>>>> kept
>>>>>>> > >> postponing it and my doc is still hosted on
>>>>>>> > >> https://pythonhosted.org/psutil/.
>>>>>>> > >
>>>>>>> > > While we've talked about deprecating it, it *hasn't* been
>>>>>>> deprecated.
>>>>>>> > > Looking at https://github.com/pypa/pypi-l
>>>>>>> egacy/commits/production, I'm
>>>>>>> > > not seeing anything obvious that would have caused problems with
>>>>>>> docs
>>>>>>> > > management, but that's probably still the best issue tracker to
>>>>>>> use to
>>>>>>> > > report the bug.
>>>>>>> >
>>>>>>> > See the 'doc_upload' handler at
>>>>>>> > https://github.com/pypa/pypi-legacy/commit/1598e6ea0f7fb0393
>>>>>>> 891f6c6bcbf84c191834a0e#diff-19fadc30e1b17100568adbd8c6c3cc13R2804
>>>>>>> > I've collected all information I found at
>>>>>>> > https://github.com/pypa/pypi-legacy/issues/672#issuecomment-
>>>>>>> 316125918
>>>>>>> > Please correct me if I missed anything.
>>>>>>> >
>>>>>>> > --Berker
>>>>>>> > _______________________________________________
>>>>>>> > Distutils-SIG maillist  -  Distutils-SIG at python.org
>>>>>>> > https://mail.python.org/mailman/listinfo/distutils-sig
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > --
>>>>>>> > Giampaolo - http://grodola.blogspot.com
>>>>>>> >
>>>>>>> > _______________________________________________
>>>>>>> > Distutils-SIG maillist  -  Distutils-SIG at python.org
>>>>>>> > https://mail.python.org/mailman/listinfo/distutils-sig
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>>>>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Giampaolo - http://grodola.blogspot.com
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Giampaolo - http://grodola.blogspot.com
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Giampaolo - http://grodola.blogspot.com
>>>>
>>>>
>>>
>>>
>>> --
>>> Giampaolo - http://grodola.blogspot.com
>>>
>>>
>
>
> --
> Giampaolo - http://grodola.blogspot.com
>
>


-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171007/3a37f52b/attachment.html>

From xoviat at gmail.com  Mon Oct  9 18:35:05 2017
From: xoviat at gmail.com (xoviat)
Date: Mon, 9 Oct 2017 17:35:05 -0500
Subject: [Distutils] Claiming Ownership of Existing Package
Message-ID: <CAKXb8mH7wNhAoRV0GeXSpoJaNpH=XK_jgTTSZuc+atciANZsCA@mail.gmail.com>

All:

I would like to claim ownership of the "flint" package on pypi so that I
can upload a new package with the same name. The existing project page
states that it is abandoned.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171009/0f255d0f/attachment.html>

From bill at baddogconsulting.com  Tue Oct 10 11:35:23 2017
From: bill at baddogconsulting.com (Bill Deegan)
Date: Tue, 10 Oct 2017 08:35:23 -0700
Subject: [Distutils] Claim/remove pypi package named sc0ns
Message-ID: <CAEyG4CEDn7LiDDgYn-Ow5K5-SNUsfFy_wq3ryPwU9=+ikJ2R5w@mail.gmail.com>

Greetings,

There's a package named sc0ns which is an ancient packaging of a SCons
release:

https://pypi.python.org/pypi/sc0ns

vs.

https://pypi.python.org/pypi/SCons

Is it possible to have the sc0ns package removed/hidden?
And/or is there a way to contact that package owner to request same?

Thanks,
Bill Deegan
SCons Project Co-Manager
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171010/d5a678c6/attachment.html>

From xoviat at gmail.com  Tue Oct 10 11:53:39 2017
From: xoviat at gmail.com (xoviat)
Date: Tue, 10 Oct 2017 10:53:39 -0500
Subject: [Distutils] Claim/remove pypi package named sc0ns
In-Reply-To: <CAEyG4CEDn7LiDDgYn-Ow5K5-SNUsfFy_wq3ryPwU9=+ikJ2R5w@mail.gmail.com>
References: <CAEyG4CEDn7LiDDgYn-Ow5K5-SNUsfFy_wq3ryPwU9=+ikJ2R5w@mail.gmail.com>
Message-ID: <CAKXb8mEYdJumNvOiUrkOT+3S31JdVbkp0+8r83V4G7_4ty728A@mail.gmail.com>

If I understand correctly, the guidelines say that its possible to
remove/take over a package if the owner doesn't dispute it. If the owner
disputes it, then I don't think it's possible. Please let me know whether
this assumption is correct.

2017-10-10 10:35 GMT-05:00 Bill Deegan <bill at baddogconsulting.com>:

> Greetings,
>
> There's a package named sc0ns which is an ancient packaging of a SCons
> release:
>
> https://pypi.python.org/pypi/sc0ns
>
> vs.
>
> https://pypi.python.org/pypi/SCons
>
> Is it possible to have the sc0ns package removed/hidden?
> And/or is there a way to contact that package owner to request same?
>
> Thanks,
> Bill Deegan
> SCons Project Co-Manager
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171010/ecec2659/attachment.html>

From thomas at kluyver.me.uk  Wed Oct 18 10:52:00 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Wed, 18 Oct 2017 15:52:00 +0100
Subject: [Distutils] Entry points: specifying and caching
Message-ID: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>

We're increasingly using entry points in Jupyter to help integrate
third-party components. This brings up a couple of things that I'd like
to do:

1. Specification

As far as I know, there's no document describing the details of entry
points; it's a de-facto standard established by setuptools. It seems to
work quite well, but it's worth writing down what is unofficially
standardised. I would like to see a document on
https://packaging.python.org/specifications/ saying:

- Where build tools should put entry points in wheels
- Where entry points live in installed distributions
- The file format (including allowed characters, case sensitivity...)

I guess I'm volunteering to write this, although if someone else wants
to, don't let me stop you. ;-)

I'd also be happy to hear that I'm wrong, that this specification
already exists somewhere. If it does, can we add a link from
https://packaging.python.org/specifications/ ?

2. Caching

"There are only two hard problems in computer science: cache
invalidation, naming things, and off-by-one errors"

I know that caching is going to make things more complex, but at present
a scan of available entry points requires a stat() for every installed
package, plus open()+read()+parse for every installed package that
provides entry points. This doesn't scale well, especially on spinning
hard drives. By eliminating a call to pygments which caused an entry
points scan, we cut the cold-start time of IPython almost in half on one
HDD system (11s -> 6s; PR 10859).

As packaging improves, the trend is to break functionality into more,
smaller packages, which is only going to make this worse (though I hope
we never end up with a left-pad package ;-). Caching could allow entry
points to be used in places where the current performance penalty is too
much.

I envisage a cache working something like this:
- Each directory on sys.path can have a cache file, e.g.
'entry-points.json'
- I suggest JSON because Python can parse it efficiently, and it's not
intended to be directly edited by humans. Other options? SQLite? Does
someone want to do performance comparisons?
- There is a command to scan all packages in a directory and build the
cache file
- After an install tool (e.g. pip) has added/removed packages from a
directory, it should call that command to rebuild the cache.
- A second command goes through all directories on sys.path and rebuilds
their cache files - this lets the user rebuild caches if something has
gone wrong.
- Applications looking for entry points can choose from a range of
behaviours depending on how important accuracy and performance are. E.g.
ignore all caches, only use caches, use caches for directories where
they exist, or try caches first and then scan packages if a key is
missing.

In the best case, when the caches exist and you trust them, loading them
would cost one set of filesystem operations per sys.path entry, rather
than per package.

Thanks,
Thomas

From doug at doughellmann.com  Wed Oct 18 12:48:04 2017
From: doug at doughellmann.com (Doug Hellmann)
Date: Wed, 18 Oct 2017 12:48:04 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
Message-ID: <1508345133-sup-1461@lrrr.local>

Excerpts from Thomas Kluyver's message of 2017-10-18 15:52:00 +0100:
> We're increasingly using entry points in Jupyter to help integrate
> third-party components. This brings up a couple of things that I'd like
> to do:
> 
> 1. Specification
> 
> As far as I know, there's no document describing the details of entry
> points; it's a de-facto standard established by setuptools. It seems to
> work quite well, but it's worth writing down what is unofficially
> standardised. I would like to see a document on
> https://packaging.python.org/specifications/ saying:
> 
> - Where build tools should put entry points in wheels
> - Where entry points live in installed distributions
> - The file format (including allowed characters, case sensitivity...)
> 
> I guess I'm volunteering to write this, although if someone else wants
> to, don't let me stop you. ;-)
> 
> I'd also be happy to hear that I'm wrong, that this specification
> already exists somewhere. If it does, can we add a link from
> https://packaging.python.org/specifications/ ?

I've always used the setuptools documentation as a reference. Are you
suggesting moving that information to a different location to
allow/encourage other tools to implement it as a standard?

> 2. Caching
> 
> "There are only two hard problems in computer science: cache
> invalidation, naming things, and off-by-one errors"
> 
> I know that caching is going to make things more complex, but at present
> a scan of available entry points requires a stat() for every installed
> package, plus open()+read()+parse for every installed package that
> provides entry points. This doesn't scale well, especially on spinning
> hard drives. By eliminating a call to pygments which caused an entry
> points scan, we cut the cold-start time of IPython almost in half on one
> HDD system (11s -> 6s; PR 10859).
> 
> As packaging improves, the trend is to break functionality into more,
> smaller packages, which is only going to make this worse (though I hope
> we never end up with a left-pad package ;-). Caching could allow entry
> points to be used in places where the current performance penalty is too
> much.
> 
> I envisage a cache working something like this:
> - Each directory on sys.path can have a cache file, e.g.
> 'entry-points.json'
> - I suggest JSON because Python can parse it efficiently, and it's not
> intended to be directly edited by humans. Other options? SQLite? Does
> someone want to do performance comparisons?
> - There is a command to scan all packages in a directory and build the
> cache file
> - After an install tool (e.g. pip) has added/removed packages from a
> directory, it should call that command to rebuild the cache.
> - A second command goes through all directories on sys.path and rebuilds
> their cache files - this lets the user rebuild caches if something has
> gone wrong.
> - Applications looking for entry points can choose from a range of
> behaviours depending on how important accuracy and performance are. E.g.
> ignore all caches, only use caches, use caches for directories where
> they exist, or try caches first and then scan packages if a key is
> missing.
> 
> In the best case, when the caches exist and you trust them, loading them
> would cost one set of filesystem operations per sys.path entry, rather
> than per package.
> 
> Thanks,
> Thomas

We've run into similar issues in some applications I work on. I had
intended to implement a caching layer within stevedore
(https://docs.openstack.org/stevedore/latest/) as a first step for
experimenting with approaches, but I would be happy to collaborate on
something further upstream if there's interest.

Doug

From p.f.moore at gmail.com  Wed Oct 18 12:59:55 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 18 Oct 2017 17:59:55 +0100
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1508345133-sup-1461@lrrr.local>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <1508345133-sup-1461@lrrr.local>
Message-ID: <CACac1F9GtauCppSi5geo_JiY2tpj0m+RXC9xX02FvqfOTCWbVg@mail.gmail.com>

On 18 October 2017 at 17:48, Doug Hellmann <doug at doughellmann.com> wrote:
> Excerpts from Thomas Kluyver's message of 2017-10-18 15:52:00 +0100:
>> We're increasingly using entry points in Jupyter to help integrate
>> third-party components. This brings up a couple of things that I'd like
>> to do:
>>
>> 1. Specification
>>
>> As far as I know, there's no document describing the details of entry
>> points; it's a de-facto standard established by setuptools. It seems to
>> work quite well, but it's worth writing down what is unofficially
>> standardised. I would like to see a document on
>> https://packaging.python.org/specifications/ saying:
>>
>> - Where build tools should put entry points in wheels
>> - Where entry points live in installed distributions
>> - The file format (including allowed characters, case sensitivity...)
>>
>> I guess I'm volunteering to write this, although if someone else wants
>> to, don't let me stop you. ;-)
>>
>> I'd also be happy to hear that I'm wrong, that this specification
>> already exists somewhere. If it does, can we add a link from
>> https://packaging.python.org/specifications/ ?
>
> I've always used the setuptools documentation as a reference. Are you
> suggesting moving that information to a different location to
> allow/encourage other tools to implement it as a standard?

I've never used entry points myself (other than the console script
entry points supported by packaging) but a quick Google search found
http://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discovery-of-services-and-plugins
as the only obvious candidate for documentation (and a bit later I
thought of looking under pkg_resources and found
http://setuptools.readthedocs.io/en/latest/pkg_resources.html#entry-points).
This doesn't really say how the entry point data is stored in the
project metadata, so it's not clear how I'd read that data in my own
code (the answer is of course to use pkg_resources, but the point of
documenting it as a standard is to allow alternative implementations).
Also, it's not clear how a tool like flit might implement entry points
- again, because the specifications don't describe how the metadata is
stored.

+1 from me on moving the entry point specification to
https://packaging.python.org/specifications/

Paul

From dholth at gmail.com  Wed Oct 18 14:06:40 2017
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 18 Oct 2017 18:06:40 +0000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CACac1F9GtauCppSi5geo_JiY2tpj0m+RXC9xX02FvqfOTCWbVg@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <1508345133-sup-1461@lrrr.local>
 <CACac1F9GtauCppSi5geo_JiY2tpj0m+RXC9xX02FvqfOTCWbVg@mail.gmail.com>
Message-ID: <CAG8k2+5XFwuUN7nW3zmmHhqtOgt809ULJPRZPf7xMzxasjAEXA@mail.gmail.com>

http://setuptools.readthedocs.io/en/latest/formats.html?highlight=entry_points.txt#entry-points-txt-entry-point-plugin-metadata

http://setuptools.readthedocs.io/en/latest/pkg_resources.html?highlight=pkg_resources#creating-and-parsing

It is not very complicated. It looks like the characters are mostly 'python
identifier' rules with a little bit of 'package name' rules.

I am also concerned about the amount of parsing on startup. A hard problem
for certain, since no one likes outdated cache problems either. It is also
unpleasant to have too much code with a runtime dependency on 'packaging'.

On Wed, Oct 18, 2017 at 1:00 PM Paul Moore <p.f.moore at gmail.com> wrote:

> On 18 October 2017 at 17:48, Doug Hellmann <doug at doughellmann.com> wrote:
> > Excerpts from Thomas Kluyver's message of 2017-10-18 15:52:00 +0100:
> >> We're increasingly using entry points in Jupyter to help integrate
> >> third-party components. This brings up a couple of things that I'd like
> >> to do:
> >>
> >> 1. Specification
> >>
> >> As far as I know, there's no document describing the details of entry
> >> points; it's a de-facto standard established by setuptools. It seems to
> >> work quite well, but it's worth writing down what is unofficially
> >> standardised. I would like to see a document on
> >> https://packaging.python.org/specifications/ saying:
> >>
> >> - Where build tools should put entry points in wheels
> >> - Where entry points live in installed distributions
> >> - The file format (including allowed characters, case sensitivity...)
> >>
> >> I guess I'm volunteering to write this, although if someone else wants
> >> to, don't let me stop you. ;-)
> >>
> >> I'd also be happy to hear that I'm wrong, that this specification
> >> already exists somewhere. If it does, can we add a link from
> >> https://packaging.python.org/specifications/ ?
> >
> > I've always used the setuptools documentation as a reference. Are you
> > suggesting moving that information to a different location to
> > allow/encourage other tools to implement it as a standard?
>
> I've never used entry points myself (other than the console script
> entry points supported by packaging) but a quick Google search found
>
> http://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discovery-of-services-and-plugins
> as the only obvious candidate for documentation (and a bit later I
> thought of looking under pkg_resources and found
> http://setuptools.readthedocs.io/en/latest/pkg_resources.html#entry-points
> ).
> This doesn't really say how the entry point data is stored in the
> project metadata, so it's not clear how I'd read that data in my own
> code (the answer is of course to use pkg_resources, but the point of
> documenting it as a standard is to allow alternative implementations).
> Also, it's not clear how a tool like flit might implement entry points
> - again, because the specifications don't describe how the metadata is
> stored.
>
> +1 from me on moving the entry point specification to
> https://packaging.python.org/specifications/
>
> Paul
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171018/795729dd/attachment-0001.html>

From alex.gronholm at nextday.fi  Wed Oct 18 14:18:19 2017
From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=)
Date: Wed, 18 Oct 2017 21:18:19 +0300
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CAG8k2+5XFwuUN7nW3zmmHhqtOgt809ULJPRZPf7xMzxasjAEXA@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <1508345133-sup-1461@lrrr.local>
 <CACac1F9GtauCppSi5geo_JiY2tpj0m+RXC9xX02FvqfOTCWbVg@mail.gmail.com>
 <CAG8k2+5XFwuUN7nW3zmmHhqtOgt809ULJPRZPf7xMzxasjAEXA@mail.gmail.com>
Message-ID: <d049b6da-b4fa-acbe-f8ac-90ca34e50010@nextday.fi>

Daniel Holth kirjoitti 18.10.2017 klo 21:06:
> http://setuptools.readthedocs.io/en/latest/formats.html?highlight=entry_points.txt#entry-points-txt-entry-point-plugin-metadata 
>
>
> http://setuptools.readthedocs.io/en/latest/pkg_resources.html?highlight=pkg_resources#creating-and-parsing
>
> It is not very complicated. It looks like the characters are mostly 
> 'python identifier' rules with a little bit of 'package name' rules.
>
> I am also concerned about the amount of parsing on startup. A hard 
> problem for certain, since no one likes outdated cache problems 
> either. It is also unpleasant to have too much code with a runtime 
> dependency on 'packaging'.
Wasn't someone working on implementing pkg_resources in the standard 
library at some point?
>
> On Wed, Oct 18, 2017 at 1:00 PM Paul Moore <p.f.moore at gmail.com 
> <mailto:p.f.moore at gmail.com>> wrote:
>
>     On 18 October 2017 at 17:48, Doug Hellmann <doug at doughellmann.com
>     <mailto:doug at doughellmann.com>> wrote:
>     > Excerpts from Thomas Kluyver's message of 2017-10-18 15:52:00 +0100:
>     >> We're increasingly using entry points in Jupyter to help integrate
>     >> third-party components. This brings up a couple of things that
>     I'd like
>     >> to do:
>     >>
>     >> 1. Specification
>     >>
>     >> As far as I know, there's no document describing the details of
>     entry
>     >> points; it's a de-facto standard established by setuptools. It
>     seems to
>     >> work quite well, but it's worth writing down what is unofficially
>     >> standardised. I would like to see a document on
>     >> https://packaging.python.org/specifications/ saying:
>     >>
>     >> - Where build tools should put entry points in wheels
>     >> - Where entry points live in installed distributions
>     >> - The file format (including allowed characters, case
>     sensitivity...)
>     >>
>     >> I guess I'm volunteering to write this, although if someone
>     else wants
>     >> to, don't let me stop you. ;-)
>     >>
>     >> I'd also be happy to hear that I'm wrong, that this specification
>     >> already exists somewhere. If it does, can we add a link from
>     >> https://packaging.python.org/specifications/ ?
>     >
>     > I've always used the setuptools documentation as a reference.
>     Are you
>     > suggesting moving that information to a different location to
>     > allow/encourage other tools to implement it as a standard?
>
>     I've never used entry points myself (other than the console script
>     entry points supported by packaging) but a quick Google search found
>     http://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discovery-of-services-and-plugins
>     as the only obvious candidate for documentation (and a bit later I
>     thought of looking under pkg_resources and found
>     http://setuptools.readthedocs.io/en/latest/pkg_resources.html#entry-points).
>     This doesn't really say how the entry point data is stored in the
>     project metadata, so it's not clear how I'd read that data in my own
>     code (the answer is of course to use pkg_resources, but the point of
>     documenting it as a standard is to allow alternative implementations).
>     Also, it's not clear how a tool like flit might implement entry points
>     - again, because the specifications don't describe how the metadata is
>     stored.
>
>     +1 from me on moving the entry point specification to
>     https://packaging.python.org/specifications/
>
>     Paul
>     _______________________________________________
>     Distutils-SIG maillist? - Distutils-SIG at python.org
>     <mailto:Distutils-SIG at python.org>
>     https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171018/ec551d97/attachment.html>

From thomas at kluyver.me.uk  Wed Oct 18 14:42:27 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Wed, 18 Oct 2017 19:42:27 +0100
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CACac1F9GtauCppSi5geo_JiY2tpj0m+RXC9xX02FvqfOTCWbVg@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <1508345133-sup-1461@lrrr.local>
 <CACac1F9GtauCppSi5geo_JiY2tpj0m+RXC9xX02FvqfOTCWbVg@mail.gmail.com>
Message-ID: <1508352147.3591995.1143262104.64043008@webmail.messagingengine.com>

On Wed, Oct 18, 2017, at 05:59 PM, Paul Moore wrote:
> > I've always used the setuptools documentation as a reference. Are you
> > suggesting moving that information to a different location to
> > allow/encourage other tools to implement it as a standard?
> 
> I've never used entry points myself (other than the console script
> entry points supported by packaging) but a quick Google search found
> http://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discovery-of-services-and-plugins
> as the only obvious candidate for documentation (and a bit later I
> thought of looking under pkg_resources and found
> http://setuptools.readthedocs.io/en/latest/pkg_resources.html#entry-points).
> This doesn't really say how the entry point data is stored in the
> project metadata, so it's not clear how I'd read that data in my own
> code (the answer is of course to use pkg_resources, but the point of
> documenting it as a standard is to allow alternative implementations).

I have in fact made an alternative implementation (PyPI package
entrypoints) by 'reverse engineering' the format. A simple text-based
format doesn't really justify the term 'reverse engineering', but for
instance it wasn't obvious to me that the names were case sensitive,
whereas Python's standard config parser treats keys as case-insensitive.

Daniel:
> http://setuptools.readthedocs.io/en/latest/formats.html?highlight=entry_points.txt#entry-points-txt-entry-point-plugin-metadata

Thanks, this link is closer than any I found to a specification. There
are docs on how to create entry points in setup.py and how to use them
with pkg_resources, but that's the only bit I've seen that describes the
interchange file format.

I think we can probably expand on it a bit, though! I'll try to put
together something for packaging.python.org.

Thomas

From p.f.moore at gmail.com  Wed Oct 18 14:53:48 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 18 Oct 2017 19:53:48 +0100
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1508352147.3591995.1143262104.64043008@webmail.messagingengine.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <1508345133-sup-1461@lrrr.local>
 <CACac1F9GtauCppSi5geo_JiY2tpj0m+RXC9xX02FvqfOTCWbVg@mail.gmail.com>
 <1508352147.3591995.1143262104.64043008@webmail.messagingengine.com>
Message-ID: <CACac1F8=6ui+4OgaegOK0kTvE3AfZbog-+VLAATZnL0Jfcxzog@mail.gmail.com>

On 18 October 2017 at 19:42, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> On Wed, Oct 18, 2017, at 05:59 PM, Paul Moore wrote:
>> > I've always used the setuptools documentation as a reference. Are you
>> > suggesting moving that information to a different location to
>> > allow/encourage other tools to implement it as a standard?
>>
>> I've never used entry points myself (other than the console script
>> entry points supported by packaging) but a quick Google search found
>> http://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discovery-of-services-and-plugins
>> as the only obvious candidate for documentation (and a bit later I
>> thought of looking under pkg_resources and found
>> http://setuptools.readthedocs.io/en/latest/pkg_resources.html#entry-points).
>> This doesn't really say how the entry point data is stored in the
>> project metadata, so it's not clear how I'd read that data in my own
>> code (the answer is of course to use pkg_resources, but the point of
>> documenting it as a standard is to allow alternative implementations).
>
> I have in fact made an alternative implementation (PyPI package
> entrypoints) by 'reverse engineering' the format. A simple text-based
> format doesn't really justify the term 'reverse engineering', but for
> instance it wasn't obvious to me that the names were case sensitive,
> whereas Python's standard config parser treats keys as case-insensitive.
>
> Daniel:
>> http://setuptools.readthedocs.io/en/latest/formats.html?highlight=entry_points.txt#entry-points-txt-entry-point-plugin-metadata
>
> Thanks, this link is closer than any I found to a specification. There
> are docs on how to create entry points in setup.py and how to use them
> with pkg_resources, but that's the only bit I've seen that describes the
> interchange file format.

Agreed, I hadn't found that, either.

> I think we can probably expand on it a bit, though! I'll try to put
> together something for packaging.python.org.

One thing that immediately strikes me is that the encoding of the file
is unspecified...
Paul

From dholth at gmail.com  Wed Oct 18 15:06:33 2017
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 18 Oct 2017 19:06:33 +0000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <d049b6da-b4fa-acbe-f8ac-90ca34e50010@nextday.fi>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <1508345133-sup-1461@lrrr.local>
 <CACac1F9GtauCppSi5geo_JiY2tpj0m+RXC9xX02FvqfOTCWbVg@mail.gmail.com>
 <CAG8k2+5XFwuUN7nW3zmmHhqtOgt809ULJPRZPf7xMzxasjAEXA@mail.gmail.com>
 <d049b6da-b4fa-acbe-f8ac-90ca34e50010@nextday.fi>
Message-ID: <CAG8k2+5ER2gBxLgL7Oqs8MHiitOiJF9Dv3W06km=4f-GRganug@mail.gmail.com>

On Wed, Oct 18, 2017 at 2:18 PM Alex Gr?nholm <alex.gronholm at nextday.fi>
wrote:

> Daniel Holth kirjoitti 18.10.2017 klo 21:06:
>
>
> http://setuptools.readthedocs.io/en/latest/formats.html?highlight=entry_points.txt#entry-points-txt-entry-point-plugin-metadata
>
>
> http://setuptools.readthedocs.io/en/latest/pkg_resources.html?highlight=pkg_resources#creating-and-parsing
>
> It is not very complicated. It looks like the characters are mostly
> 'python identifier' rules with a little bit of 'package name' rules.
>
> I am also concerned about the amount of parsing on startup. A hard problem
> for certain, since no one likes outdated cache problems either. It is also
> unpleasant to have too much code with a runtime dependency on 'packaging'.
>
> Wasn't someone working on implementing pkg_resources in the standard
> library at some point?
>

I'm just saying it is good to avoid importing it unless you really need to.
Same reason we removed it from entry point script wrappers.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171018/aa232f33/attachment.html>

From dholth at gmail.com  Wed Oct 18 15:07:22 2017
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 18 Oct 2017 19:07:22 +0000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CACac1F8=6ui+4OgaegOK0kTvE3AfZbog-+VLAATZnL0Jfcxzog@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <1508345133-sup-1461@lrrr.local>
 <CACac1F9GtauCppSi5geo_JiY2tpj0m+RXC9xX02FvqfOTCWbVg@mail.gmail.com>
 <1508352147.3591995.1143262104.64043008@webmail.messagingengine.com>
 <CACac1F8=6ui+4OgaegOK0kTvE3AfZbog-+VLAATZnL0Jfcxzog@mail.gmail.com>
Message-ID: <CAG8k2+6LUi4+oA8RrctW1wSzSM8VK_9Lm2FWmgdnVXj6M4qH9A@mail.gmail.com>

On Wed, Oct 18, 2017 at 2:57 PM Paul Moore <p.f.moore at gmail.com> wrote:

> On 18 October 2017 at 19:42, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> > On Wed, Oct 18, 2017, at 05:59 PM, Paul Moore wrote:
> >> > I've always used the setuptools documentation as a reference. Are you
> >> > suggesting moving that information to a different location to
> >> > allow/encourage other tools to implement it as a standard?
> >>
> >> I've never used entry points myself (other than the console script
> >> entry points supported by packaging) but a quick Google search found
> >>
> http://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discovery-of-services-and-plugins
> >> as the only obvious candidate for documentation (and a bit later I
> >> thought of looking under pkg_resources and found
> >>
> http://setuptools.readthedocs.io/en/latest/pkg_resources.html#entry-points
> ).
> >> This doesn't really say how the entry point data is stored in the
> >> project metadata, so it's not clear how I'd read that data in my own
> >> code (the answer is of course to use pkg_resources, but the point of
> >> documenting it as a standard is to allow alternative implementations).
> >
> > I have in fact made an alternative implementation (PyPI package
> > entrypoints) by 'reverse engineering' the format. A simple text-based
> > format doesn't really justify the term 'reverse engineering', but for
> > instance it wasn't obvious to me that the names were case sensitive,
> > whereas Python's standard config parser treats keys as case-insensitive.
> >
> > Daniel:
> >>
> http://setuptools.readthedocs.io/en/latest/formats.html?highlight=entry_points.txt#entry-points-txt-entry-point-plugin-metadata
> >
> > Thanks, this link is closer than any I found to a specification. There
> > are docs on how to create entry points in setup.py and how to use them
> > with pkg_resources, but that's the only bit I've seen that describes the
> > interchange file format.
>
> Agreed, I hadn't found that, either.
>
> > I think we can probably expand on it a bit, though! I'll try to put
> > together something for packaging.python.org.
>
> One thing that immediately strikes me is that the encoding of the file
> is unspecified...
> Paul
>

Now that's an easy one to clear up, since there is only one worthwhile
encoding.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171018/0c79a6e4/attachment.html>

From ncoghlan at gmail.com  Wed Oct 18 20:53:31 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 19 Oct 2017 10:53:31 +1000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <d049b6da-b4fa-acbe-f8ac-90ca34e50010@nextday.fi>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <1508345133-sup-1461@lrrr.local>
 <CACac1F9GtauCppSi5geo_JiY2tpj0m+RXC9xX02FvqfOTCWbVg@mail.gmail.com>
 <CAG8k2+5XFwuUN7nW3zmmHhqtOgt809ULJPRZPf7xMzxasjAEXA@mail.gmail.com>
 <d049b6da-b4fa-acbe-f8ac-90ca34e50010@nextday.fi>
Message-ID: <CADiSq7etRRq5h6jo1fOZ4ckO8W0jU3UdGLXMOaFzhLHX2ScCNQ@mail.gmail.com>

On 19 October 2017 at 04:18, Alex Gr?nholm <alex.gronholm at nextday.fi> wrote:

> Daniel Holth kirjoitti 18.10.2017 klo 21:06:
>
> http://setuptools.readthedocs.io/en/latest/formats.html?
> highlight=entry_points.txt#entry-points-txt-entry-point-plugin-metadata
>
> http://setuptools.readthedocs.io/en/latest/pkg_resources.
> html?highlight=pkg_resources#creating-and-parsing
>
> It is not very complicated. It looks like the characters are mostly
> 'python identifier' rules with a little bit of 'package name' rules.
>
> I am also concerned about the amount of parsing on startup. A hard problem
> for certain, since no one likes outdated cache problems either. It is also
> unpleasant to have too much code with a runtime dependency on 'packaging'.
>
> Wasn't someone working on implementing pkg_resources in the standard
> library at some point?
>

The idea has been raised, but we've been hesitant for the same reason we're
inclined to take distutils out: packaging APIs need to be free to evolve in
line with packaging interoperability standards, rather than with the Python
language definition.

Barry Warsaw & Brett Cannon recently mentioned something to me about
working on a potential runtime alternative to pkg_resources that could be
installed without also installing setuptools, but I don't know any of the
specifics (and I'm not sure either of them follows distutils-sig).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171019/11ada528/attachment.html>

From dholth at gmail.com  Wed Oct 18 22:16:29 2017
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 19 Oct 2017 02:16:29 +0000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CADiSq7etRRq5h6jo1fOZ4ckO8W0jU3UdGLXMOaFzhLHX2ScCNQ@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <1508345133-sup-1461@lrrr.local>
 <CACac1F9GtauCppSi5geo_JiY2tpj0m+RXC9xX02FvqfOTCWbVg@mail.gmail.com>
 <CAG8k2+5XFwuUN7nW3zmmHhqtOgt809ULJPRZPf7xMzxasjAEXA@mail.gmail.com>
 <d049b6da-b4fa-acbe-f8ac-90ca34e50010@nextday.fi>
 <CADiSq7etRRq5h6jo1fOZ4ckO8W0jU3UdGLXMOaFzhLHX2ScCNQ@mail.gmail.com>
Message-ID: <CAG8k2+4HP5h5V5RM=SoRv6E_cLqdx55YDU6jK+tkSW9Th-O3sw@mail.gmail.com>

We said "you won't have to install setuptools" but actually "you don't have
to use it" is good enough. If you had 2 pkg-resources implementations
running you might wind up scanning sys.path extra times...

On Wed, Oct 18, 2017, 20:53 Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 19 October 2017 at 04:18, Alex Gr?nholm <alex.gronholm at nextday.fi>
> wrote:
>
>> Daniel Holth kirjoitti 18.10.2017 klo 21:06:
>>
>>
>> http://setuptools.readthedocs.io/en/latest/formats.html?highlight=entry_points.txt#entry-points-txt-entry-point-plugin-metadata
>>
>>
>> http://setuptools.readthedocs.io/en/latest/pkg_resources.html?highlight=pkg_resources#creating-and-parsing
>>
>> It is not very complicated. It looks like the characters are mostly
>> 'python identifier' rules with a little bit of 'package name' rules.
>>
>> I am also concerned about the amount of parsing on startup. A hard
>> problem for certain, since no one likes outdated cache problems either. It
>> is also unpleasant to have too much code with a runtime dependency on
>> 'packaging'.
>>
>> Wasn't someone working on implementing pkg_resources in the standard
>> library at some point?
>>
>
> The idea has been raised, but we've been hesitant for the same reason
> we're inclined to take distutils out: packaging APIs need to be free to
> evolve in line with packaging interoperability standards, rather than with
> the Python language definition.
>
> Barry Warsaw & Brett Cannon recently mentioned something to me about
> working on a potential runtime alternative to pkg_resources that could be
> installed without also installing setuptools, but I don't know any of the
> specifics (and I'm not sure either of them follows distutils-sig).
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171019/67599d97/attachment-0001.html>

From ncoghlan at gmail.com  Wed Oct 18 23:51:00 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 19 Oct 2017 13:51:00 +1000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CAG8k2+4HP5h5V5RM=SoRv6E_cLqdx55YDU6jK+tkSW9Th-O3sw@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <1508345133-sup-1461@lrrr.local>
 <CACac1F9GtauCppSi5geo_JiY2tpj0m+RXC9xX02FvqfOTCWbVg@mail.gmail.com>
 <CAG8k2+5XFwuUN7nW3zmmHhqtOgt809ULJPRZPf7xMzxasjAEXA@mail.gmail.com>
 <d049b6da-b4fa-acbe-f8ac-90ca34e50010@nextday.fi>
 <CADiSq7etRRq5h6jo1fOZ4ckO8W0jU3UdGLXMOaFzhLHX2ScCNQ@mail.gmail.com>
 <CAG8k2+4HP5h5V5RM=SoRv6E_cLqdx55YDU6jK+tkSW9Th-O3sw@mail.gmail.com>
Message-ID: <CADiSq7eFnQ3rx=brHLBuU-yZoXZthvjBSS613tw6aAxE_kvEsg@mail.gmail.com>

On 19 October 2017 at 12:16, Daniel Holth <dholth at gmail.com> wrote:

> We said "you won't have to install setuptools" but actually "you don't
> have to use it" is good enough. If you had 2 pkg-resources implementations
> running you might wind up scanning sys.path extra times...
>
True, but that's where Thomas's suggestion of attempting to define a
standardised caching convention comes in: right now, there's no middle
ground between "you must use pkg_resources" and "every helper library must
scan for the raw entry-point metadata itself".

If there's a defined common caching mechanism, and support for it is added
to new versions of pkg_resources, then the design constraint becomes "If
you end up using multiple entry-point scanners, you'll want a recent
setuptools/pkg_resource, so you don't waste too much time on repeated
metadata scans".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171019/4fe8a1c4/attachment.html>

From wes.turner at gmail.com  Thu Oct 19 04:02:24 2017
From: wes.turner at gmail.com (Wes Turner)
Date: Thu, 19 Oct 2017 04:02:24 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CADiSq7eFnQ3rx=brHLBuU-yZoXZthvjBSS613tw6aAxE_kvEsg@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <1508345133-sup-1461@lrrr.local>
 <CACac1F9GtauCppSi5geo_JiY2tpj0m+RXC9xX02FvqfOTCWbVg@mail.gmail.com>
 <CAG8k2+5XFwuUN7nW3zmmHhqtOgt809ULJPRZPf7xMzxasjAEXA@mail.gmail.com>
 <d049b6da-b4fa-acbe-f8ac-90ca34e50010@nextday.fi>
 <CADiSq7etRRq5h6jo1fOZ4ckO8W0jU3UdGLXMOaFzhLHX2ScCNQ@mail.gmail.com>
 <CAG8k2+4HP5h5V5RM=SoRv6E_cLqdx55YDU6jK+tkSW9Th-O3sw@mail.gmail.com>
 <CADiSq7eFnQ3rx=brHLBuU-yZoXZthvjBSS613tw6aAxE_kvEsg@mail.gmail.com>
Message-ID: <CACfEFw_N1JTVTwhCYfKJNO4T_tgF-QSisdQ+eYzZtbdE5GTgLQ@mail.gmail.com>

def get_env_json_path():
  directory = $VIRTUAL_ENV || ?
  return os.path.join(directory, ENV_JSON_FILENAME)

def on_install(pkg_json):
  env_json_path = get_env_json_path()
  env_json = json.load(env_json_path)
  env_json['pkgs?][pkgname] = pkg_json
  with open(env_json_path, 'w') as f:
    f.write(env_json)

def read_cached_entry_points():
  env_json_path = get_env_json_path()
  env_json = json.load(env_json_path)
  entry_points = flatten(**{ pkg['entry_points'] for pkg in
env_json['pigs']})
  return entry_points


Would this introduce a need for a new and confusing rescan_metadata()
(pkg.on_install() for pkg in pkgs)?

On Wednesday, October 18, 2017, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 19 October 2017 at 12:16, Daniel Holth <dholth at gmail.com
> <javascript:_e(%7B%7D,'cvml','dholth at gmail.com');>> wrote:
>
>> We said "you won't have to install setuptools" but actually "you don't
>> have to use it" is good enough. If you had 2 pkg-resources implementations
>> running you might wind up scanning sys.path extra times...
>>
> True, but that's where Thomas's suggestion of attempting to define a
> standardised caching convention comes in: right now, there's no middle
> ground between "you must use pkg_resources" and "every helper library must
> scan for the raw entry-point metadata itself".
>
> If there's a defined common caching mechanism, and support for it is added
> to new versions of pkg_resources, then the design constraint becomes "If
> you end up using multiple entry-point scanners, you'll want a recent
> setuptools/pkg_resource, so you don't waste too much time on repeated
> metadata scans".
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com
> <javascript:_e(%7B%7D,'cvml','ncoghlan at gmail.com');>   |   Brisbane,
> Australia
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171019/e2312af7/attachment.html>

From donald at stufft.io  Thu Oct 19 11:10:27 2017
From: donald at stufft.io (Donald Stufft)
Date: Thu, 19 Oct 2017 11:10:27 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
Message-ID: <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>


> On Oct 18, 2017, at 10:52 AM, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> 
> 
> 1. Specification
> 


I?m in favor, although one question I guess is whether it should be a a PEP or an ad hoc spec. Given (2) it should *probably* be a a PEP (since without (2), its just another file in the .dist-info directory and that doesn?t actually need standardized at all). I don?t think that this will be a very controversial PEP though, and should be pretty easy.


> 
> 2. Caching


I?m also in favor of this. Although I would suggest SQLite rather than a JSON file for the primary reason being that a JSON file isn?t multiprocess safe without being careful (and possibly introducing locking) whereas SQLite has already solved that problem.

One possible further enhancement to your proposal is to try and think of a way to have a singular cache, since we can include the sys.path entry as part of the data inside the cache, having a singular cache means we can reduce the the number of files we have to open down to a single file. The biggest problem I see with this, is it opens up questions about how we handle things like user installs? so maybe a cache DB per sys.path entry is the best way. I think we could use something like SQLite?s ATTACH DATABASE command to add multiple DBs to the same SQLite connection to be able to query across all of the entries with a single query. One downside to this is that SQLite is an optional module in Python so it may not exist, although we could implement that so that we just bypass the cache always in that case (and probably raise a warning?) so things continue to work, they will just be slower.

I know that Twisted has used a cache file for awhile for plugins (so a similiar use case) so I wonder if they would have any opinions or insight into this as well.



From thomas at kluyver.me.uk  Thu Oct 19 12:14:43 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Thu, 19 Oct 2017 17:14:43 +0100
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
Message-ID: <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>

On Thu, Oct 19, 2017, at 04:10 PM, Donald Stufft wrote:
> I?m in favor, although one question I guess is whether it should be a a
> PEP or an ad hoc spec. Given (2) it should *probably* be a a PEP (since
> without (2), its just another file in the .dist-info directory and that
> doesn?t actually need standardized at all). I don?t think that this will
> be a very controversial PEP though, and should be pretty easy.

I have opened a PR to document what is already there, without adding any
new features. I think this is worth doing even if we don't change
anything, since it's a de-facto standard used for different tools to
interact.

https://github.com/pypa/python-packaging-user-guide/pull/390

We can still write a PEP for caching if necessary.

> I?m also in favor of this. Although I would suggest SQLite rather than a
> JSON file for the primary reason being that a JSON file isn?t
> multiprocess safe without being careful (and possibly introducing
> locking) whereas SQLite has already solved that problem.

SQLite was actually my first thought, but from experience in Jupyter &
IPython I'm wary of it - its built-in locking does not work well over
NFS, and it's easy to corrupt the database. I think careful use of
atomic writing can be more reliable (though that has given us some
problems too).

That may be easier if there's one cache per user, though - we can
perhaps try to store it somewhere that's not NFS.

Thomas

From donald at stufft.io  Thu Oct 19 14:09:51 2017
From: donald at stufft.io (Donald Stufft)
Date: Thu, 19 Oct 2017 14:09:51 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
Message-ID: <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>



> On Oct 19, 2017, at 12:14 PM, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> 
> On Thu, Oct 19, 2017, at 04:10 PM, Donald Stufft wrote:
>> I?m in favor, although one question I guess is whether it should be a a
>> PEP or an ad hoc spec. Given (2) it should *probably* be a a PEP (since
>> without (2), its just another file in the .dist-info directory and that
>> doesn?t actually need standardized at all). I don?t think that this will
>> be a very controversial PEP though, and should be pretty easy.
> 
> I have opened a PR to document what is already there, without adding any
> new features. I think this is worth doing even if we don't change
> anything, since it's a de-facto standard used for different tools to
> interact.
> 
> https://github.com/pypa/python-packaging-user-guide/pull/390
> 
> We can still write a PEP for caching if necessary.

I think documenting what?s there is a reasonable goal, but if we?re going to add caching we should just PEP the whole thing changing it from a defect standard to an actual standard + caching. Generally we should only use non-PEP ?specs? in places where we?re just trying to document what exists already, but where we?re not really happy with the current solution or we plan to alter it eventually.

For this, I think the entry points solution is generally a good one with some alterations (namely, the addition of caching)?. Although now that I think about it, maybe this isn?t really a packaging problem at all and I?m not sure that it benefits from standardization at all.

So stepping back a second, here?s what entrypoints provides today:

1. A way to implement a interface that some other package can provide implementations for.
2. A way to specify script wrappers that will be automatically generated.
3. A way to define extras that must be installed in order for a particular entry point to be available.

Off the bat I?m going to say we don?t need to worry about (2) in this hypothetical system, because I think the fact it is implemented currently via this system is mostly a historic accident, and it?s not something we should be looking at in the future. Script wrappers should have some dedicated metadata, not piggybacking off of the plugin system.

For (3) I don?t believe that what extras were installed is recorded anywhere, so I?m going to guess that this works by looking up what extras are *available* for a particular package and then seeing if all of the requirements of that distribution are satisfied. Assuming that?s the case then that?s not really something that requires deep integration with the packaging toolchain, it just needs the APIs to look those things up.

Finally we come to (1), which is in my opinion the meet of what you?re hoping to achieve here (and what most people are using entry points for outside of console scripts. What I notice about (1) is that it really has absolutely nothing to do with packaging at all. It would likely use some of the APIs provided by the packaging toolchain (for instance, the ability to add custom files to a .dist-info directory, the ability to iterate over installed packages, etc) but as a whole pip, setuptools, twine, PyPI, etc none of these things need to know anything about it.

EXCEPT, for the fact that with the desire to cache things, it would be beneficial to ?hook? into the lifecycle of a package install. However I know that there are other plugin systems out there that would like to also be able to do that (Twisted Plugins come to mind) and that I think outside of plugin systems, such a mechanism is likely to be useful in general for other cases.

So heres a different idea that is a bit more ambitious but that I think is a better overall idea. Let entrypoints be a setuptools thing, and lets define some key lifecycle hooks during the installation of a package and some mechanism in the metadata to let other tools subscribe to those hooks. Then  a caching layer could be written for setuptools entrypoints to make that faster without requiring standardization, but also a whole new, better plugin system could to, Twisted plugins could benefit, etc [1].

One thing that I like about all of our work recently in packaging is a lot of it has been about making it so there isn?t just one standard set of tools, and I think that providing lifecycle hooks is another step along that path.

> 
>> I?m also in favor of this. Although I would suggest SQLite rather than a
>> JSON file for the primary reason being that a JSON file isn?t
>> multiprocess safe without being careful (and possibly introducing
>> locking) whereas SQLite has already solved that problem.
> 
> SQLite was actually my first thought, but from experience in Jupyter &
> IPython I'm wary of it - its built-in locking does not work well over
> NFS, and it's easy to corrupt the database. I think careful use of
> atomic writing can be more reliable (though that has given us some
> problems too).
> 
> That may be easier if there's one cache per user, though - we can
> perhaps try to store it somewhere that's not NFS.
> 


I don?t have a lot of experience using SQLite in this way so it?s entirely possible it?s not as robust as we want/need it to be. I?m not wedded to this idea (but then if we do what I said above, this idea becomes something for any individual implementation of plugins to decide and we don?t need to pick a standard here at all!).


[1] I realize the irony in saying a plugin system isn?t a packaging problem, so let?s define a plugin system for packaging hooks, but I think it can be very simple and not something designed to be reusable outside of that context and speed is less of a concern, etc.


From p.f.moore at gmail.com  Thu Oct 19 14:28:34 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 19 Oct 2017 19:28:34 +0100
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
Message-ID: <CACac1F9nA-N4k-zSbLfLxEcpfEVSSXuMhpEKEzr0F5PkwreWrw@mail.gmail.com>

On 19 October 2017 at 19:09, Donald Stufft <donald at stufft.io> wrote:
>
> So heres a different idea that is a bit more ambitious but that I think is a better overall idea. Let entrypoints be a setuptools thing, and lets define some key lifecycle hooks during the installation of a package and some mechanism in the metadata to let other tools subscribe to those hooks. Then  a caching layer could be written for setuptools entrypoints to make that faster without requiring standardization, but also a whole new, better plugin system could to, Twisted plugins could benefit, etc [1].

I think this is a nice idea, and like you say could likely enable a
number of interesting use cases. However...

>
> One thing that I like about all of our work recently in packaging is a lot of it has been about making it so there isn?t just one standard set of tools, and I think that providing lifecycle hooks is another step along that path.

While I agree with this, one thing I have noticed with recent work is
that standardising existing things has typically been relatively
painless and stress-free. But designing new mechanisms generally ends
up with huge threads, heated debates, and people burning out on the
whole thing. We've had a couple of cases of that recently, and in
particular Thomas has endured the big PEP 517 debate, so I'm inclined
to say we should take a rest from new designs for a while, and keep
the scope here limited.

We can go back and hit packaging system hooks later, it's not like the
idea will go away. And the breathing space will also give people time
to actually implement the recent PEPs, and consolidate the gains we've
already made.

Paul

From dholth at gmail.com  Thu Oct 19 14:34:10 2017
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 19 Oct 2017 18:34:10 +0000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
Message-ID: <CAG8k2+5pXYPEU-dG70T-XHL1o-X_k40zLrBko6gDEsM3sSpq+Q@mail.gmail.com>

I prefer a single more generic mechanism that packaging happens to use
instead of making special mechanisms for scripts or other callables that
packaging might some day be interested in. One API, I can type
pkg_resources.iter_entry_points('console_scripts') to enumerate the scripts
and perhaps invoke them without the wrappers, or I can look other plugins.

+1 on simply documenting what we have first.

How long does pkg_resources take to import for you folks?

On Thu, Oct 19, 2017 at 2:10 PM Donald Stufft <donald at stufft.io> wrote:

>
>
> > On Oct 19, 2017, at 12:14 PM, Thomas Kluyver <thomas at kluyver.me.uk>
> wrote:
> >
> > On Thu, Oct 19, 2017, at 04:10 PM, Donald Stufft wrote:
> >> I?m in favor, although one question I guess is whether it should be a a
> >> PEP or an ad hoc spec. Given (2) it should *probably* be a a PEP (since
> >> without (2), its just another file in the .dist-info directory and that
> >> doesn?t actually need standardized at all). I don?t think that this will
> >> be a very controversial PEP though, and should be pretty easy.
> >
> > I have opened a PR to document what is already there, without adding any
> > new features. I think this is worth doing even if we don't change
> > anything, since it's a de-facto standard used for different tools to
> > interact.
> >
> > https://github.com/pypa/python-packaging-user-guide/pull/390
> >
> > We can still write a PEP for caching if necessary.
>
> I think documenting what?s there is a reasonable goal, but if we?re going
> to add caching we should just PEP the whole thing changing it from a defect
> standard to an actual standard + caching. Generally we should only use
> non-PEP ?specs? in places where we?re just trying to document what exists
> already, but where we?re not really happy with the current solution or we
> plan to alter it eventually.
>
> For this, I think the entry points solution is generally a good one with
> some alterations (namely, the addition of caching)?. Although now that I
> think about it, maybe this isn?t really a packaging problem at all and I?m
> not sure that it benefits from standardization at all.
>
> So stepping back a second, here?s what entrypoints provides today:
>
> 1. A way to implement a interface that some other package can provide
> implementations for.
> 2. A way to specify script wrappers that will be automatically generated.
> 3. A way to define extras that must be installed in order for a particular
> entry point to be available.
>
> Off the bat I?m going to say we don?t need to worry about (2) in this
> hypothetical system, because I think the fact it is implemented currently
> via this system is mostly a historic accident, and it?s not something we
> should be looking at in the future. Script wrappers should have some
> dedicated metadata, not piggybacking off of the plugin system.
>
> For (3) I don?t believe that what extras were installed is recorded
> anywhere, so I?m going to guess that this works by looking up what extras
> are *available* for a particular package and then seeing if all of the
> requirements of that distribution are satisfied. Assuming that?s the case
> then that?s not really something that requires deep integration with the
> packaging toolchain, it just needs the APIs to look those things up.
>
> Finally we come to (1), which is in my opinion the meet of what you?re
> hoping to achieve here (and what most people are using entry points for
> outside of console scripts. What I notice about (1) is that it really has
> absolutely nothing to do with packaging at all. It would likely use some of
> the APIs provided by the packaging toolchain (for instance, the ability to
> add custom files to a .dist-info directory, the ability to iterate over
> installed packages, etc) but as a whole pip, setuptools, twine, PyPI, etc
> none of these things need to know anything about it.
>
> EXCEPT, for the fact that with the desire to cache things, it would be
> beneficial to ?hook? into the lifecycle of a package install. However I
> know that there are other plugin systems out there that would like to also
> be able to do that (Twisted Plugins come to mind) and that I think outside
> of plugin systems, such a mechanism is likely to be useful in general for
> other cases.
>
> So heres a different idea that is a bit more ambitious but that I think is
> a better overall idea. Let entrypoints be a setuptools thing, and lets
> define some key lifecycle hooks during the installation of a package and
> some mechanism in the metadata to let other tools subscribe to those hooks.
> Then  a caching layer could be written for setuptools entrypoints to make
> that faster without requiring standardization, but also a whole new, better
> plugin system could to, Twisted plugins could benefit, etc [1].
>
> One thing that I like about all of our work recently in packaging is a lot
> of it has been about making it so there isn?t just one standard set of
> tools, and I think that providing lifecycle hooks is another step along
> that path.
>
> >
> >> I?m also in favor of this. Although I would suggest SQLite rather than a
> >> JSON file for the primary reason being that a JSON file isn?t
> >> multiprocess safe without being careful (and possibly introducing
> >> locking) whereas SQLite has already solved that problem.
> >
> > SQLite was actually my first thought, but from experience in Jupyter &
> > IPython I'm wary of it - its built-in locking does not work well over
> > NFS, and it's easy to corrupt the database. I think careful use of
> > atomic writing can be more reliable (though that has given us some
> > problems too).
> >
> > That may be easier if there's one cache per user, though - we can
> > perhaps try to store it somewhere that's not NFS.
> >
>
>
> I don?t have a lot of experience using SQLite in this way so it?s entirely
> possible it?s not as robust as we want/need it to be. I?m not wedded to
> this idea (but then if we do what I said above, this idea becomes something
> for any individual implementation of plugins to decide and we don?t need to
> pick a standard here at all!).
>
>
> [1] I realize the irony in saying a plugin system isn?t a packaging
> problem, so let?s define a plugin system for packaging hooks, but I think
> it can be very simple and not something designed to be reusable outside of
> that context and speed is less of a concern, etc.
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171019/ff1f83fd/attachment-0001.html>

From donald at stufft.io  Thu Oct 19 14:51:05 2017
From: donald at stufft.io (Donald Stufft)
Date: Thu, 19 Oct 2017 14:51:05 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CACac1F9nA-N4k-zSbLfLxEcpfEVSSXuMhpEKEzr0F5PkwreWrw@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <CACac1F9nA-N4k-zSbLfLxEcpfEVSSXuMhpEKEzr0F5PkwreWrw@mail.gmail.com>
Message-ID: <162854A4-4A6A-496F-9613-0F6769D7C4E7@stufft.io>


> On Oct 19, 2017, at 2:28 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> While I agree with this, one thing I have noticed with recent work is
> that standardising existing things has typically been relatively
> painless and stress-free. But designing new mechanisms generally ends
> up with huge threads, heated debates, and people burning out on the
> whole thing. We've had a couple of cases of that recently, and in
> particular Thomas has endured the big PEP 517 debate, so I'm inclined
> to say we should take a rest from new designs for a while, and keep
> the scope here limited.

So I?m generally fine with keeping the scope limited, but for the same reason as I think the real solution is what I defined above, I think this isn?t/shouldn?t be a packaging standard and is a setuptools feature and should be documented/live there. If setuptools wants to enable people to directly manipulate those files they can document the standard of those files, if they want to treat it as internal and you?re expected to use their APIs then they can.

Essentially, I don?t think that a plugin system should be within the domain of distutils-sig or the PyPA and the only reason we?re even thinking of it as one is because (a) historically setuptools _had_ a plugin system and (b) we lack lifecycle hooks. I?m loathe to move the documentation for a setuptools specific feature out of their documentation because I think it muddies the water further.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171019/1f1b6130/attachment.html>

From thomas at kluyver.me.uk  Thu Oct 19 14:54:44 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Thu, 19 Oct 2017 19:54:44 +0100
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
Message-ID: <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>

On Thu, Oct 19, 2017, at 07:09 PM, Donald Stufft wrote:
> So heres a different idea that is a bit more ambitious but that I think
> is a better overall idea. Let entrypoints be a setuptools thing, and lets
> define some key lifecycle hooks during the installation of a package and
> some mechanism in the metadata to let other tools subscribe to those
> hooks.

I'd like to document the existing mechanism as previously suggested. Not
least because I've already written the PR ;-).

I don't think this needs to be controversial. They are a de-facto
packaging standard, whether or not that's theoretically necessary.
There's more than one tool that can create them (setuptools, flit), and
more than one that can consume them (pkg_resources, entrypoints). Lots
of packages use them, and they're not going anywhere soon. Describing
the format properly seems like a clear win.

For caching, I'm happy enough to work on a more general PEP to define
packaging hooks, so long as that isn't going to be as long a discussion
as PEP 517.

Daniel:
> How long does pkg_resources take to import for you folks?

About 0.5s on my laptop with an SSD, about 5s on a machine with a
spinning hard drive. This is simulating a cold start on both; it's much
quicker once the OS caches it in memory.

Thomas

From donald at stufft.io  Thu Oct 19 15:01:51 2017
From: donald at stufft.io (Donald Stufft)
Date: Thu, 19 Oct 2017 15:01:51 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
Message-ID: <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>


> On Oct 19, 2017, at 2:54 PM, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> 
> I don't think this needs to be controversial. They are a de-facto
> packaging standard, whether or not that's theoretically necessary.
> There's more than one tool that can create them (setuptools, flit), and
> more than one that can consume them (pkg_resources, entrypoints). Lots
> of packages use them, and they're not going anywhere soon. Describing
> the format properly seems like a clear win.


I disagree they are a packaging standard and I think it would be crummy to define it as one. I believe it is a setuptools feature, that flit and entrypoints wants to integrate with a setuptools feature is fine, but that doesn?t make it a packaging standard just because it came from setuptools. I agree that describing the format properly is a clear win, but I believe it belongs in the setuptools documentation.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171019/c01ae2d1/attachment-0001.html>

From thomas at kluyver.me.uk  Thu Oct 19 15:15:58 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Thu, 19 Oct 2017 20:15:58 +0100
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
Message-ID: <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>

On Thu, Oct 19, 2017, at 08:01 PM, Donald Stufft wrote:
> 
>> On Oct 19, 2017, at 2:54 PM, Thomas Kluyver
>> <thomas at kluyver.me.uk> wrote:>> 
>> I don't think this needs to be controversial. They are a de-facto
>> packaging standard, whether or not that's theoretically necessary.
>> There's more than one tool that can create them (setuptools,
>> flit), and>> more than one that can consume them (pkg_resources,
>> entrypoints). Lots>> of packages use them, and they're not going anywhere soon. Describing>> the format properly seems like a clear win.
> 
> 
> I disagree they are a packaging standard and I think it would be
> crummy to define it as one. I believe it is a setuptools feature, that
> flit and entrypoints wants to integrate with a setuptools feature is
> fine, but that doesn?t make it a packaging standard just because it
> came from setuptools. I agree that describing the format properly is a
> clear win, but I believe it belongs in the setuptools documentation.
pip and distlib also independently read this format without going
through setuptools. It's a de-facto standard already.  Entry points are
also the most common way for packages to install command-line scripts,
and the most effective way to do so across different platforms. So it's
essential that install tools do understand this.
Much of our packaging standards were built out of setuptools features
anyway - why pretend that this is different?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171019/efa888b3/attachment.html>

From donald at stufft.io  Thu Oct 19 15:29:32 2017
From: donald at stufft.io (Donald Stufft)
Date: Thu, 19 Oct 2017 15:29:32 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
Message-ID: <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>



> On Oct 19, 2017, at 3:15 PM, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> 
> On Thu, Oct 19, 2017, at 08:01 PM, Donald Stufft wrote:
>> 
>>> On Oct 19, 2017, at 2:54 PM, Thomas Kluyver <thomas at kluyver.me.uk <mailto:thomas at kluyver.me.uk>> wrote:
>>> 
>>> I don't think this needs to be controversial. They are a de-facto
>>> packaging standard, whether or not that's theoretically necessary.
>>> There's more than one tool that can create them (setuptools, flit), and
>>> more than one that can consume them (pkg_resources, entrypoints). Lots
>>> of packages use them, and they're not going anywhere soon. Describing
>>> the format properly seems like a clear win.
>> 
>> 
>> 
>> I disagree they are a packaging standard and I think it would be crummy to define it as one. I believe it is a setuptools feature, that flit and entrypoints wants to integrate with a setuptools feature is fine, but that doesn?t make it a packaging standard just because it came from setuptools. I agree that describing the format properly is a clear win, but I believe it belongs in the setuptools documentation.
> 
> pip and distlib also independently read this format without going through setuptools. It's a de-facto standard already.  Entry points are also the most common way for packages to install command-line scripts, and the most effective way to do so across different platforms. So it's essential that install tools do understand this.

It?s only essential in that we support a very limited subset specifically for console scripts, which long term we should be extracting from entry points and using something dedicated to that. Generating script wrappers is a packaging concern, and if this proposal was about documenting the console_scripts key in an entry_points.txt file to trigger a console script being generated, then that?s fine with me.

> 
> Much of our packaging standards were built out of setuptools features anyway - why pretend that this is different?

Because it is? A generic plugin mechanism is not a packaging feature any more then a HTTP client is a packaging feature, but setuptools contains one of those too. Since setuptools was in large part a packaging library, it will of course contain many packaging features that we?re going to standardize on, but something being in setuptools does not in fact make it a packaging feature in and of itself.

As an example of another setuptools feature that isn?t a packaging feature, I also would be against adding the resource APIs in a packaging standard because they?re not a packaging feature either, they?re a python import module feature (which is why Brett Cannon and Barry are adding them to importlib instead of trying to make a packaging PEP for them).

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171019/7693c4e0/attachment.html>

From thomas at kluyver.me.uk  Thu Oct 19 15:55:07 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Thu, 19 Oct 2017 20:55:07 +0100
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
Message-ID: <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>

On Thu, Oct 19, 2017, at 08:29 PM, Donald Stufft wrote:
> Because it is? A generic plugin mechanism is not a packaging feature
> any more then a HTTP client is a packaging feature, but setuptools
> contains one of those too. Since setuptools was in large part a
> packaging library, it will of course contain many packaging features
> that we?re going to standardize on, but something being in setuptools
> does not in fact make it a packaging feature in and of itself.
My argument is not that it's in setuptools, it's that

1. It's already processed by multiple packaging tools
2. Any tool producing wheels which include command line tools basically
   has to use entry points (or include a bunch of redundant complexity
   to make command-line wrappers). It's a de-facto part of the wheel
   spec, at least until a replacement is devised - and since it works,
   replacing for semantic cleanliness is not a priority.
You're quite right that a plugin system doesn't need to be a packaging
standard. But that ship has sailed. It's already a standard format for
packaging, the only question is whether it's documented. Practicality
beats purity.
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171019/574da3ef/attachment-0001.html>

From donald at stufft.io  Thu Oct 19 16:04:38 2017
From: donald at stufft.io (Donald Stufft)
Date: Thu, 19 Oct 2017 16:04:38 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
Message-ID: <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>



> On Oct 19, 2017, at 3:55 PM, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> 
> On Thu, Oct 19, 2017, at 08:29 PM, Donald Stufft wrote:
>> Because it is? A generic plugin mechanism is not a packaging feature any more then a HTTP client is a packaging feature, but setuptools contains one of those too. Since setuptools was in large part a packaging library, it will of course contain many packaging features that we?re going to standardize on, but something being in setuptools does not in fact make it a packaging feature in and of itself.
> 
> My argument is not that it's in setuptools, it's that
> 
> 1. It's already processed by multiple packaging tools
> 2. Any tool producing wheels which include command line tools basically has to use entry points (or include a bunch of redundant complexity to make command-line wrappers). It's a de-facto part of the wheel spec, at least until a replacement is devised - and since it works, replacing for semantic cleanliness is not a priority.
> 
> You're quite right that a plugin system doesn't need to be a packaging standard. But that ship has sailed. It's already a standard format for packaging, the only question is whether it's documented. Practicality beats purity.


Like I said, I?m perfectly fine documenting that if you add an entry_points.txt to the .dist-info directory, that is an INI file that contains a section named ?console_scripts? and define what is valid inside of the console_scripts section that it will generate script wrappers, then fine. But we should leave any other section in this entry_points.txt file as undefined in packaging terms, and point people towards setuptools for more information about it if they want to know anything more than what we need for packaging.

I am against fully speccing out or adding more features to entry points as part of a packaging standardization effort.

From donald at stufft.io  Thu Oct 19 16:34:42 2017
From: donald at stufft.io (Donald Stufft)
Date: Thu, 19 Oct 2017 16:34:42 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
Message-ID: <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>


> On Oct 19, 2017, at 4:04 PM, Donald Stufft <donald at stufft.io> wrote:
> 
> Like I said, I?m perfectly fine documenting that if you add an entry_points.txt to the .dist-info directory, that is an INI file that contains a section named ?console_scripts? and define what is valid inside of the console_scripts section that it will generate script wrappers, then fine. But we should leave any other section in this entry_points.txt file as undefined in packaging terms, and point people towards setuptools for more information about it if they want to know anything more than what we need for packaging.

To be more specific here, the hypothetical thing we would be documenting/standardizing here is console entry points and script wrappers, not a generic plugin system. So console scripts would be the focus of the documentation.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171019/3b2f7def/attachment.html>

From thomas at kluyver.me.uk  Thu Oct 19 16:36:18 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Thu, 19 Oct 2017 21:36:18 +0100
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
Message-ID: <1508445378.1368777.1144644376.068AB497@webmail.messagingengine.com>

On Thu, Oct 19, 2017, at 09:04 PM, Donald Stufft wrote:
> Like I said, I?m perfectly fine documenting that if you add an
> entry_points.txt to the .dist-info directory, that is an INI file that
> contains a section named ?console_scripts? and define what is valid
> inside of the console_scripts section that it will generate script
> wrappers, then fine. But we should leave any other section in this
> entry_points.txt file as undefined in packaging terms, and point people
> towards setuptools for more information about it if they want to know
> anything more than what we need for packaging.

I don't see any advantage in describing the file format but then
pretending that there's only section in it. We're not prescribing any
particular meaning or use for other sections, but it seems bizarre to
not describe the possibilities. console_scripts is just one use case.

Also, entry points in general kind of are a packaging thing. You specify
them in packaging metadata, both for setuptools and flit, and the
packaging tools write entry_points.txt. It's not the only way to create
a plugin system, but it's the way this one was created.

I honestly don't get the resistance to documenting this as a whole. I'm
not proposing something that will add a new maintenance burden; it's a
description of something that's already there. Can't we save the energy
for discussing a real change or new thing?

Thomas

From donald at stufft.io  Thu Oct 19 16:57:25 2017
From: donald at stufft.io (Donald Stufft)
Date: Thu, 19 Oct 2017 16:57:25 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1508445378.1368777.1144644376.068AB497@webmail.messagingengine.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <1508445378.1368777.1144644376.068AB497@webmail.messagingengine.com>
Message-ID: <27A027C1-85EA-4C11-9762-658DCA027430@stufft.io>



> On Oct 19, 2017, at 4:36 PM, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> 
> On Thu, Oct 19, 2017, at 09:04 PM, Donald Stufft wrote:
>> Like I said, I?m perfectly fine documenting that if you add an
>> entry_points.txt to the .dist-info directory, that is an INI file that
>> contains a section named ?console_scripts? and define what is valid
>> inside of the console_scripts section that it will generate script
>> wrappers, then fine. But we should leave any other section in this
>> entry_points.txt file as undefined in packaging terms, and point people
>> towards setuptools for more information about it if they want to know
>> anything more than what we need for packaging.
> 
> I don't see any advantage in describing the file format but then
> pretending that there's only section in it. We're not prescribing any
> particular meaning or use for other sections, but it seems bizarre to
> not describe the possibilities. console_scripts is just one use case.

Because the feature is unrelated to packaging other than the fact we currently utilize it for console_scripts. A spec to standardize console_scripts is a good thing, a spec to standardize an almost entirely unrelated feature for packaging is a bad thing. 

> 
> Also, entry points in general kind of are a packaging thing. You specify
> them in packaging metadata, both for setuptools and flit, and the
> packaging tools write entry_points.txt. It's not the only way to create
> a plugin system, but it's the way this one was created.

You can describe lots of things in the packaging metadata, because one of the features of the packaging metadata is you can add arbitrary files to the dist-info directory. Entrypoints are one such file that some projects add to that directory, but there are other examples and jsut becuause it involves adding files to that, does not mean it belongs to ?packaging?.

> 
> I honestly don't get the resistance to documenting this as a whole. I'm
> not proposing something that will add a new maintenance burden; it's a
> description of something that's already there. Can't we save the energy
> for discussing a real change or new thing?
> 

I don?t get the resistance to documenting this where it belongs. Its not any more difficult to document things in the setuptools repository than it is to document it in the packaging specs repository.


From tseaver at palladion.com  Thu Oct 19 17:26:27 2017
From: tseaver at palladion.com (Tres Seaver)
Date: Thu, 19 Oct 2017 17:26:27 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <27A027C1-85EA-4C11-9762-658DCA027430@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <1508445378.1368777.1144644376.068AB497@webmail.messagingengine.com>
 <27A027C1-85EA-4C11-9762-658DCA027430@stufft.io>
Message-ID: <osb59u$3j9$1@blaine.gmane.org>

On 10/19/2017 04:57 PM, Donald Stufft wrote:

> Because the feature is unrelated to packaging other than the fact we
> currently utilize it for console_scripts.
That seems like an odd perspective.  Console scripts may be the only bit of
entry points which is used *by the packaging system* at installation time,
but an system composed of separately-installable packages providing shared
services needs some way of querying those services at runtime, which is
what all the *other* uses of entry points represent.  Having the packaging
system register those services at installation time (even if it doesn't
care otherwise about them) seems pretty reasonable to me.


Tres.
-- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com


From donald at stufft.io  Thu Oct 19 17:33:55 2017
From: donald at stufft.io (Donald Stufft)
Date: Thu, 19 Oct 2017 17:33:55 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <osb59u$3j9$1@blaine.gmane.org>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <1508445378.1368777.1144644376.068AB497@webmail.messagingengine.com>
 <27A027C1-85EA-4C11-9762-658DCA027430@stufft.io>
 <osb59u$3j9$1@blaine.gmane.org>
Message-ID: <D3637C7E-4C72-4674-87A9-6D65CB6CCDCC@stufft.io>


> On Oct 19, 2017, at 5:26 PM, Tres Seaver <tseaver at palladion.com> wrote:
> 
> Having the packaging
> system register those services at installation time (even if it doesn't
> care otherwise about them) seems pretty reasonable to me.

It does register them at installation time, using an entirely generic feature of ?you can add any file you want to a dist-info directory and we will preserve it?. It doesn?t need to know anything else about them other then it?s a file that needs preserved.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171019/4f037a2c/attachment.html>

From wes.turner at gmail.com  Thu Oct 19 20:44:03 2017
From: wes.turner at gmail.com (Wes Turner)
Date: Thu, 19 Oct 2017 20:44:03 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <D3637C7E-4C72-4674-87A9-6D65CB6CCDCC@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <1508445378.1368777.1144644376.068AB497@webmail.messagingengine.com>
 <27A027C1-85EA-4C11-9762-658DCA027430@stufft.io>
 <osb59u$3j9$1@blaine.gmane.org>
 <D3637C7E-4C72-4674-87A9-6D65CB6CCDCC@stufft.io>
Message-ID: <CACfEFw90KKcVXpdxG86h8wbnw6M1LW7qNFM_MsjbKA9Unw8YBg@mail.gmail.com>

On Thursday, October 19, 2017, Donald Stufft <donald at stufft.io> wrote:

>
> On Oct 19, 2017, at 5:26 PM, Tres Seaver <tseaver at palladion.com
> <javascript:_e(%7B%7D,'cvml','tseaver at palladion.com');>> wrote:
>
> Having the packaging
> system register those services at installation time (even if it doesn't
> care otherwise about them) seems pretty reasonable to me.
>
>
> It does register them at installation time, using an entirely generic
> feature of ?you can add any file you want to a dist-info directory and we
> will preserve it?. It doesn?t need to know anything else about them other
> then it?s a file that needs preserved.
>

When I think of 'register at installation time', I think of adding them to
a single { locked JSON || SQLite DB || ...}; because that's the only way
there'd be a performance advantage?

Why would we write a .txt, transform it to {JSON || SQL INSERTS}, and then
write it to a central registrar?

(BTW, pipsi does console script entry points with isolated virtualenvs
linked into from ~/.local/bin (which is generally user-writable)).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171019/48a7f77e/attachment.html>

From ncoghlan at gmail.com  Fri Oct 20 00:42:09 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 20 Oct 2017 14:42:09 +1000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
Message-ID: <CADiSq7daBpo8TK2cRW=ou7puQsMgWHObBEFBv48C1KOWnEmUhw@mail.gmail.com>

On 20 October 2017 at 02:14, Thomas Kluyver <thomas at kluyver.me.uk> wrote:

> On Thu, Oct 19, 2017, at 04:10 PM, Donald Stufft wrote:
> > I?m in favor, although one question I guess is whether it should be a a
> > PEP or an ad hoc spec. Given (2) it should *probably* be a a PEP (since
> > without (2), its just another file in the .dist-info directory and that
> > doesn?t actually need standardized at all). I don?t think that this will
> > be a very controversial PEP though, and should be pretty easy.
>
> I have opened a PR to document what is already there, without adding any
> new features. I think this is worth doing even if we don't change
> anything, since it's a de-facto standard used for different tools to
> interact.
>
> https://github.com/pypa/python-packaging-user-guide/pull/390
>
> We can still write a PEP for caching if necessary.
>

+1 for that approach (PR for the status quo, PEP for a shared metadata
caching design) from me

Making the status quo more discoverable is valuable in its own right, and
the only decisions we'll need to make for that are terminology
clarification ones, not interoperability ones (this isn't like PEP 440 or
508 where we actually thought some of the default setuptools behaviour was
slightly incorrect and wanted to change it).

Figuring out a robust cross-platform network-file-system-tolerant metadata
caching design on the other hand is going to be hard, and as Donald
suggests, the right ecosystem level solution might be to define
install-time hooks for package installation operations.


> > I?m also in favor of this. Although I would suggest SQLite rather than a
> > JSON file for the primary reason being that a JSON file isn?t
> > multiprocess safe without being careful (and possibly introducing
> > locking) whereas SQLite has already solved that problem.
>
> SQLite was actually my first thought, but from experience in Jupyter &
> IPython I'm wary of it - its built-in locking does not work well over
> NFS, and it's easy to corrupt the database. I think careful use of
> atomic writing can be more reliable (though that has given us some
> problems too).
>
> That may be easier if there's one cache per user, though - we can
> perhaps try to store it somewhere that's not NFS.
>

I'm wondering if rather than jumping straight to a PEP, it may make sense
to instead initially pursue this idea as a *non-*standard, implementation
dependent thing specific to the "entrypoints" project. There are a *lot* of
challenges to be taken into account for a truly universal metadata caching
design, and it would be easy to fall into the trap of coming up with a
design so complex that nobody can realistically implement it.

Specifically, I'm thinking of a usage model along the lines of the
updatedb/locate pair on *nix systems: `locate` gives you access to very
fast searches of your filesystem, but it *doesn't* try to automagically
keeps its indexes up to date. Instead, refreshing the indexes is handled by
`updatedb`, and you can either rely on that being run automatically in a
cron job, or else force an update with `sudo updatedb` when you want to use
`locate`.

For a project like entrypoints, what that might look like is that at
*runtime*, you may implement a reasonably fast "cache freshness check",
where you scanned the mtime of all the sys.path entries, and compared those
to the mtime of the cache. If the cache looks up to date, then cool,
otherwise emit a warning about the stale metadata cache, and then bypass it.

The entrypoints project itself could then expose a
`refresh-entrypoints-cache` command that could start out only supporting
virtual environments, and then extend to per-user caching, and then finally
(maybe) consider whether or not it wanted to support installation-wide
caches (with the extra permissions management and cross-process and
cross-system coordination that may imply).

Such an approach would also tie in nicely with Donald's suggestion of
reframing the ecosystem level question as "How should the entrypoints
project request that 'refresh-entrypoints-cache' be run after every package
installation or removal operation?", which in turn would integrate nicely
with things like RPM file triggers (where the system `pip` package could
set a file trigger that arranged for any properly registered Python package
installation plugins to be run for every modification to site-packages
while still appropriately managing the risk of running arbitrary code with
elevated privileges)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/efd13dfe/attachment-0001.html>

From ncoghlan at gmail.com  Fri Oct 20 01:32:18 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 20 Oct 2017 15:32:18 +1000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
Message-ID: <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>

On 20 October 2017 at 06:34, Donald Stufft <donald at stufft.io> wrote:

>
> On Oct 19, 2017, at 4:04 PM, Donald Stufft <donald at stufft.io> wrote:
>
> Like I said, I?m perfectly fine documenting that if you add an
> entry_points.txt to the .dist-info directory, that is an INI file that
> contains a section named ?console_scripts? and define what is valid inside
> of the console_scripts section that it will generate script wrappers, then
> fine. But we should leave any other section in this entry_points.txt file
> as undefined in packaging terms, and point people towards setuptools for
> more information about it if they want to know anything more than what we
> need for packaging.
>
>
> To be more specific here, the hypothetical thing we would be
> documenting/standardizing here is console entry points and script wrappers,
> not a generic plugin system. So console scripts would be the focus of the
> documentation.
>

We've already effectively blessed console_scripts as a standard approach:
https://packaging.python.org/tutorials/distributing-packages/#entry-points

The specific problem that blessing creates is that we currently only define:

- a way for publishers to specify console_scripts via setuptools
- a way for installers to find console_scripts using pkg_resources

That's *very* similar to the problem we had with dependency declarations:
only setuptools knew how to write them, and only easy_install knew how to
read them.

Beyond the specific example of console_scripts, there are also multiple
subecosystems where both publishers and subscribers remain locked into the
setuptools/pkg_resources combination because they use entry points for
their plugin management. This means that if you want to write a pytest
plugin, for example, the only officially supported way to do so is to use
setuptools in order to publish the relevant entry point definitions:
https://docs.pytest.org/en/latest/writing_plugins.html#setuptools-entry-points

If we want to enable pytest plugin authors to use other build systems like
flit, then those build systems need a defined interoperability format
that's compatible with what pytest is expecting to see (i.e. entry point
definitions that pkg_resources knows how to read).

We ended up solving the previous tight publisher/installer coupling problem
for dependency management *not* by coming up with completely new metadata
formats, but rather by better specifying the ones that setuptools already
knew how to emit, such that most publishers didn't need to change anything,
and even when there were slight differences between the way setuptools
worked and the agreed interoperability standards, other tools could readily
translate setuptools output into the standardised form (e.g. egg_info ->
PEP 376 dist-info directories and wheel metadata).

The difference in this case is that:

1. entry_points.txt is already transported reliably through the whole
packaging toolchain
2. It is the existing interoperability format for `console_scripts`
definitions
3. Unlike setup.cfg & pyproject.toml, actual humans never touch it - it's
written and read solely by software

This means that the interoperability problems we actually care about
solving (allowing non-setuptools based publishing tools to specify
console_scripts and other pkg_resources entry points, and allowing
non-pkg_resources based consumers to read pkg_resources entry point
metadata, including console_scripts) can both be solved *just* by properly
specifying the existing de facto format.

So standardising on entry_points.txt isn't a matter of "because setuptools
does it", it's because formalising it is the least-effort solution to what
we actually want to enable: making setuptools optional on the publisher
side (even if you need to publish entry point metadata), and making
pkg_resources optional on the consumer side (even if you need to read entry
point metadata).

I do agree that the metadata caching problem is best tackled as a specific
motivating example for supporting packaging installation and uninstallation
hooks, but standardising the entry points format still helps us with that:
it means we can just define "python.install_hooks" as a new entry point
category, and spend our energy on defining the semantics and APIs of the
hooks themselves, rather than having to worry about defining a new format
for how publishers will declare how to run the hooks, or how installers
will find out which hooks have been installed locally.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/daa12df1/attachment.html>

From ncoghlan at gmail.com  Fri Oct 20 01:48:47 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 20 Oct 2017 15:48:47 +1000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <D3637C7E-4C72-4674-87A9-6D65CB6CCDCC@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <1508445378.1368777.1144644376.068AB497@webmail.messagingengine.com>
 <27A027C1-85EA-4C11-9762-658DCA027430@stufft.io>
 <osb59u$3j9$1@blaine.gmane.org>
 <D3637C7E-4C72-4674-87A9-6D65CB6CCDCC@stufft.io>
Message-ID: <CADiSq7dA8Ty6r2+dvs5gyncyAkf28hhb17edg9OLbi3ds7wA-g@mail.gmail.com>

On 20 October 2017 at 07:33, Donald Stufft <donald at stufft.io> wrote:

>
> On Oct 19, 2017, at 5:26 PM, Tres Seaver <tseaver at palladion.com> wrote:
>
> Having the packaging
> system register those services at installation time (even if it doesn't
> care otherwise about them) seems pretty reasonable to me.
>
>
> It does register them at installation time, using an entirely generic
> feature of ?you can add any file you want to a dist-info directory and we
> will preserve it?. It doesn?t need to know anything else about them other
> then it?s a file that needs preserved.
>

That's all the *installer* needs to know. Publishing tools like flit need
to know the internal format in order to replicate the effect of
https://packaging.python.org/tutorials/distributing-packages/#console-scripts
and to interoperate with any other pkg_resources based plugin ecosystem.

I personally find it useful to think of entry points as a pub/sub
communications channel between package authors and other runtime components.

When you use the entry points syntax to declare a pytest plugin as a
publisher, your intended subscriber is pytest, and pytest defines the
possible messages. Ditto for any other entry points based plugin management
system.

Installers are mostly just a relay link in that pub/sub channel - they take
the entry point announcement messages in the sdist or wheel archive, and
pass them along to the installation database.

The one exception to the "installers as passive relay" behaviour is that
when you specify "console_scripts", your intended subscribers *are* package
installation tools, and your message is "I'd like an executable wrapper for
these entry points, please".

Right now, the only documented publishing API for that pub/sub channel is
setuptools.setup(), and the only documented subscription API is
pkg_resources. Documenting the file format explicitly changes that dynamic,
such that any publisher that produces a compliant `entry_points.txt` file
will be supported by pkg_resources, and any consumer that can read a
compliant `entry_points.txt` file will be supported by setuptools.setup()

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/b35edbb0/attachment-0001.html>

From thomas at kluyver.me.uk  Fri Oct 20 02:40:02 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Fri, 20 Oct 2017 07:40:02 +0100
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CADiSq7daBpo8TK2cRW=ou7puQsMgWHObBEFBv48C1KOWnEmUhw@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <CADiSq7daBpo8TK2cRW=ou7puQsMgWHObBEFBv48C1KOWnEmUhw@mail.gmail.com>
Message-ID: <1508481602.1501299.1145057848.33EF8D37@webmail.messagingengine.com>

On Fri, Oct 20, 2017, at 05:42 AM, Nick Coghlan wrote:
> I'm wondering if rather than jumping straight to a PEP, it may make
> sense to instead initially pursue this idea as a *non-*standard,
> implementation dependent thing specific to the "entrypoints" project.
> There are a *lot* of challenges to be taken into account for a truly
> universal metadata caching design, and it would be easy to fall into
> the trap of coming up with a design so complex that nobody can
> realistically implement it.
I'd be happy to tackle it like that. Donald's proposed hooks for package
installation and uninstallation would provide all the necessary
interoperation between different tools. As and when it's working, the
cache format can be documented for other consumers to use.
> Right now, the only documented publishing API for that pub/sub channel
> is setuptools.setup(), and the only documented subscription API is
> pkg_resources. Documenting the file format explicitly changes that
> dynamic, such that any publisher that produces a compliant
> `entry_points.txt` file will be supported by pkg_resources, and any
> consumer that can read a compliant `entry_points.txt` file will be
> supported by setuptools.setup()
Yup, this is very much what I'd like :-)

Thanks,
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/5ac0bd5a/attachment.html>

From thomas at kluyver.me.uk  Fri Oct 20 02:43:00 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Fri, 20 Oct 2017 07:43:00 +0100
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1508481602.1501299.1145057848.33EF8D37@webmail.messagingengine.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <CADiSq7daBpo8TK2cRW=ou7puQsMgWHObBEFBv48C1KOWnEmUhw@mail.gmail.com>
 <1508481602.1501299.1145057848.33EF8D37@webmail.messagingengine.com>
Message-ID: <1508481780.1501981.1145060440.19C88CB6@webmail.messagingengine.com>

I would also be happy to add a section to the document describing the
specific use of entry points for defining scripts to install.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/5dda99aa/attachment.html>

From donald at stufft.io  Fri Oct 20 06:48:44 2017
From: donald at stufft.io (Donald Stufft)
Date: Fri, 20 Oct 2017 06:48:44 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
Message-ID: <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>


> On Oct 20, 2017, at 1:32 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> If we want to enable pytest plugin authors to use other build systems like flit, then those build systems need a defined interoperability format that's compatible with what pytest is expecting to see (i.e. entry point definitions that pkg_resources knows how to read).
> 

This is thinking about it wrong IMO.

We could just as easily say if we want tools like flit to be able to package Twisted plugins then those build systems need a defined interoperability format that is compatible with what Twisted and that ecosystem is expecting.The *ONLY* reason we should care at all about defining entry points as a packaging feature is console scripts, so we should limit our standardization to that. PBR has a runtime feature too where it inserts metadata into the .dist-info directory at build time and then a runtime API that reads that.. should we standardize that too?

I?m *not* saying that flit doesn?t nee to know how to generate entry points if a entry points using project wants to use flit, but what I am saying is that entry points isn?t a packaging specification. It?s a setuptools feature that should live within setuptools.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/b272fc67/attachment.html>

From ncoghlan at gmail.com  Fri Oct 20 06:49:12 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 20 Oct 2017 20:49:12 +1000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1508481780.1501981.1145060440.19C88CB6@webmail.messagingengine.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <CADiSq7daBpo8TK2cRW=ou7puQsMgWHObBEFBv48C1KOWnEmUhw@mail.gmail.com>
 <1508481602.1501299.1145057848.33EF8D37@webmail.messagingengine.com>
 <1508481780.1501981.1145060440.19C88CB6@webmail.messagingengine.com>
Message-ID: <CADiSq7c=9nW9E1QBCec6NjSumVo5EeeBhTa9yjUGxLB2b_+-yA@mail.gmail.com>

On 20 October 2017 at 16:43, Thomas Kluyver <thomas at kluyver.me.uk> wrote:

> I would also be happy to add a section to the document describing the
> specific use of entry points for defining scripts to install.
>

Yeah, it would make sense to include that, as well as reserving the
"console_scripts" name on PyPI so we abide by our own "Only rely on a
category name if you or one of your dependencies controls it on PyPI"
guideline.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/b405e206/attachment-0001.html>

From ncoghlan at gmail.com  Fri Oct 20 07:02:43 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 20 Oct 2017 21:02:43 +1000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
Message-ID: <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>

On 20 October 2017 at 20:48, Donald Stufft <donald at stufft.io> wrote:

>
> On Oct 20, 2017, at 1:32 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> If we want to enable pytest plugin authors to use other build systems like
> flit, then those build systems need a defined interoperability format
> that's compatible with what pytest is expecting to see (i.e. entry point
> definitions that pkg_resources knows how to read).
>
>
> This is thinking about it wrong IMO.
>
> We could just as easily say if we want tools like flit to be able to
> package Twisted plugins then those build systems need a defined
> interoperability format that is compatible with what Twisted and that
> ecosystem is expecting.
>

Twisted already defines plugin discovery in an inherently
packaging-friendly way, since it's based on import names rather than
packaging metadata. Other plugin management systems like straight.plugins
are similar: they use Python's import system as their pub/sub channel to
advertise plugin availability, and accept the limitation that this means
all plugin APIs will be module level ones rather than being individual
classes or callables.


> The *ONLY* reason we should care at all about defining entry points as a
> packaging feature is console scripts, so we should limit our
> standardization to that. PBR has a runtime feature too where it inserts
> metadata into the .dist-info directory at build time and then a runtime API
> that reads that.. should we standardize that too?
>

No, because PBR isn't the defacto default build system that pip injects
into setup.py execution by default. That's the one point where the "de
facto standard" status of setuptools is relevant to the question of whether
the entry_points.txt format is a PyPA interoperability standard: it is,
because providing a functionally equivalent capability is required for
publishers to be able to transparently switch from setuptools to something
else without their end users noticing the difference.

Sure we *could* say "We don't want to standardise on that one, we want to
define a different one", but I think entry points are good enough for our
purposes, so inventing something different wouldn't be a good use of
anyone's time (see also: the perpetually deferred status of PEP 426).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/afbb3376/attachment.html>

From donald at stufft.io  Fri Oct 20 07:15:44 2017
From: donald at stufft.io (Donald Stufft)
Date: Fri, 20 Oct 2017 07:15:44 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
Message-ID: <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>



> On Oct 20, 2017, at 7:02 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
>  That's the one point where the "de facto standard" status of setuptools is relevant to the question of whether the entry_points.txt format is a PyPA interoperability standard: it is, because providing a functionally equivalent capability is required for publishers to be able to transparently switch from setuptools to something else without their end users noticing the difference.


Nope. Because this isn?t a packaging feature. It?s a runtime feature of setuptools, and we do everyone a disservice by trying to move this into the purview of distutils-sig just because setuptools included a feature once. Just because setuptools included a feature does *NOT* make it a packaging related feature.

Tell you what, I?ll drop everything today and write up a PEP that adds metadata for console scripts to the packaging metadata where it belongs, so we can move console_scripts entry point to a legacy footnote as far as packaging systems go. Then we can discuss whether an arbitrary plugin system is actually a packaging related spec (it?s not) on it?s own merits.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/e737db03/attachment.html>

From donald at stufft.io  Fri Oct 20 07:25:18 2017
From: donald at stufft.io (Donald Stufft)
Date: Fri, 20 Oct 2017 07:25:18 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
Message-ID: <D331EFEE-7263-49A5-A466-17441CAF5D66@stufft.io>


> On Oct 20, 2017, at 1:32 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> 3. Unlike setup.cfg & pyproject.toml, actual humans never touch it - it's written and read solely by software

This is wrong BTW, humans can and do effectively write entry_points.txt, it?s a supported feature of setuptools to do:

setuptools.setup(
    entry_points=???
        [my_cool_entrypoint]
        athing = the.thing:bar
    ???,
)


This is documented and I have run into a number of projects that do this.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/ec76586f/attachment-0001.html>

From thomas at kluyver.me.uk  Fri Oct 20 07:31:50 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Fri, 20 Oct 2017 12:31:50 +0100
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
Message-ID: <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>

On Fri, Oct 20, 2017, at 12:15 PM, Donald Stufft wrote:
> Tell you what, I?ll drop everything today and write up a PEP...

Donald, why are you so determined that this spec should not be created?
Your time is enormously valuable, so why would you drop everything to
write a PEP which implies changes to tooling, simply so that we don't
document the status quo? Even if we do make that change, there are
thousands of existing packages using the existing de-facto standard, so
it would still be valuable to document it.
If it makes things easier, I'll host the spec on my own site and add a
'see also' from the specs page of the packaging user guide (because I
think people would expect it to be there, even if it's not the 'right'
place). But I don't think anyone else has expressed any objection to
putting the spec there.
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/fc5c6b74/attachment.html>

From donald at stufft.io  Fri Oct 20 07:50:09 2017
From: donald at stufft.io (Donald Stufft)
Date: Fri, 20 Oct 2017 07:50:09 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>
Message-ID: <5F9E5F93-7AF4-4D05-956F-A4367F89CD96@stufft.io>


> On Oct 20, 2017, at 7:31 AM, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> 
> On Fri, Oct 20, 2017, at 12:15 PM, Donald Stufft wrote:
>> Tell you what, I?ll drop everything today and write up a PEP...
> 
> Donald, why are you so determined that this spec should not be created? Your time is enormously valuable, so why would you drop everything to write a PEP which implies changes to tooling, simply so that we don't document the status quo? Even if we do make that change, there are thousands of existing packages using the existing de-facto standard, so it would still be valuable to document it.
> 
> If it makes things easier, I'll host the spec on my own site and add a 'see also' from the specs page of the packaging user guide (because I think people would expect it to be there, even if it's not the 'right' place). But I don't think anyone else has expressed any objection to putting the spec there.
> 
> Thomas


I mean, it?s a PEP I was already planning on writing at some point, because I?ve *never* liked the fact that our console script support was reliant on a setuptools feature so all I?d be doing is re-prioritizing work I was already planning on doing. I?m also completely happy with documenting the status quo, which from a packaging stand point means documenting console_scripts? it doesn?t mean pulling in an entire setuptools feature. I?m not even against documenting the entire feature, *if* it?s done inside of setuptools where it belongs.

What I am against, is moving the entire entry points feature from a setuptools feature to a packaging standard. It is at best, tangental to packaging since outside of console_scripts it?s only real relation is that it uses features of the packaging ecosystem and happened to come from setuptools (but it could have just as easily been written externally to setuptools). Making it a packaging standard comes with several implications:

* Since it is a packaging standard, then it is expected that all packaging tools will be updated to work with it.
* We?re explicitly saying that this is the one true way of solving this problem in the Python ecosystem.
* We stifle innovation (hell just including it in setutools at all does this, but we can?t unopen that can of worms).
* We make it actively harder to improve the feature (since once it?s part of the purview of packaging standards, all of distutils-sig gets to weigh in on improvements).

I don?t get why anyone would want to saddle all of the extra implications and work that comes with being a packaging standard on a feature that isn?t one and doesn?t need to be one. We are at our best when our efforts are on generalized mechanisms that allow features such as entry points to be implemented on top of us, rather than trying to pull in every tangential feature under the sun into our domain.


From thomas at kluyver.me.uk  Fri Oct 20 07:57:31 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Fri, 20 Oct 2017 12:57:31 +0100
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <5F9E5F93-7AF4-4D05-956F-A4367F89CD96@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>
 <5F9E5F93-7AF4-4D05-956F-A4367F89CD96@stufft.io>
Message-ID: <1508500651.117367.1145301232.4A2422BF@webmail.messagingengine.com>

On Fri, Oct 20, 2017, at 12:50 PM, Donald Stufft wrote:
> * Since it is a packaging standard, then it is expected that all
> packaging tools will be updated to work with it.

Where packaging tools need to know about it,  they already have to.
Where they don't, writing a standard doesn't imply that every tool has
to implement it. Documenting it doesn't change either case, it just
makes life easier for tools that do need to use it.

> * We?re explicitly saying that this is the one true way of solving this
> problem in the Python ecosystem.

I don't buy that at all. We're saying that it exists, and this is what
it is.

> * We stifle innovation (hell just including it in setutools at all does
> this, but we can?t unopen that can of worms).

I don't think that's true to any significant extent. Having a standard
does not stop people coming up with something better.

> * We make it actively harder to improve the feature (since once it?s part
> of the purview of packaging standards, all of distutils-sig gets to weigh
> in on improvements).

It hasn't changed in years, as far as I know, and it's so widely used
that any change is likely to break a load of stuff anyway. As we've
already discussed for caching, we can improve by building *on top* of it
relatively easily. And ultimately I think that bringing it out into
daylight leads to a healthier future than leaving it under the stone
marked 'setuptools''.


From ncoghlan at gmail.com  Fri Oct 20 08:06:03 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 20 Oct 2017 22:06:03 +1000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
Message-ID: <CADiSq7e81J-S-=8TYuHrUt5SDcoCitv_aExCNA2ytyoVcSdg1w@mail.gmail.com>

On 20 October 2017 at 21:15, Donald Stufft <donald at stufft.io> wrote:

> Tell you what, I?ll drop everything today and write up a PEP that adds
> metadata for console scripts to the packaging metadata where it belongs,
>

Donald, you're making the same mistake I did with PEP 426: interoperability
specifications are useless without a commitment from tooling developers to
actually provide implementations that know how to read and write them. And
since any new format you come up with won't be supported by existing pip
and pkg_resources installations, there won't be any incentive for
publishers to start using it, which means there's no incentives for runtime
libraries to learn how to read it, etc, etc.

In this case, we already have a perfectly serviceable format
(entry_points.txt), a reference publisher (setuptools.setup) and a
reference consumer (pkg_resources). The fact that the reference consumer is
pkg_resources rather than pip doesn't suddenly take this outside the domain
of responsibility of distutils-sig as a whole - it only takes it outside
the domain of responsibility of PyPI.

So if you want to say it is neither pip's nor PyPI's responsibility to say
anything one way or the other about the entry points format (beyond whether
or not they're used to declare console scripts in a way that pip
understands), then I agree with you entirely. This spec isn't something you
personally need to worry about, since it doesn't impact any of the tools
you work on (aside from giving pip's existing console_scripts
implementation a firmer foundation from an interoperability perpsective).

So the core of our disagreement is whether or not interfaces involving pip
and PyPI represent the limits of distutil-sig's responsibility. They don't,
and that's reflected in the fact we have a split standing delegation from
Guido (one initially to Richard Jones and later to you for changes that
affect PyPI, and one to me for packaging ecosystem interoperability
specifications in general)


> so we can move console_scripts entry point to a legacy footnote as far as
> packaging systems go. Then we can discuss whether an arbitrary plugin
> system is actually a packaging related spec (it?s not) on it?s own merits.
>

Instructing publishing system developers on how to publish pkg_resources
compatible entry points is indeed a Python packaging ecosystem level
concern.

Whether that capability survives into a hypothetical future metadata
specification (whether that's PEP 426+459 or something else entirely) would
then be a different question, but it isn't one we need to worry about right
now (since it purely affects internal interoperability file formats that
only automated scripts and folks maintaining those scripts need to care
about, and we'd expect entry_points.txt and PKG-INFO to coexist alongside
any new format for a *long* time).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/df46e972/attachment-0001.html>

From donald at stufft.io  Fri Oct 20 08:10:06 2017
From: donald at stufft.io (Donald Stufft)
Date: Fri, 20 Oct 2017 08:10:06 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1508500651.117367.1145301232.4A2422BF@webmail.messagingengine.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>
 <5F9E5F93-7AF4-4D05-956F-A4367F89CD96@stufft.io>
 <1508500651.117367.1145301232.4A2422BF@webmail.messagingengine.com>
Message-ID: <6BBFF1A1-F05D-496A-8BEB-FB65D7554823@stufft.io>



> On Oct 20, 2017, at 7:57 AM, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> 
> On Fri, Oct 20, 2017, at 12:50 PM, Donald Stufft wrote:
>> * Since it is a packaging standard, then it is expected that all
>> packaging tools will be updated to work with it.
> 
> Where packaging tools need to know about it,  they already have to.
> Where they don't, writing a standard doesn't imply that every tool has
> to implement it. Documenting it doesn't change either case, it just
> makes life easier for tools that do need to use it.

Packaging tools shouldn?t be expected to know anything about it other than the console_scripts feature (which shouldn?t exist as an entry point, but currently does for historical reasons).

Publishing tools should have a way for additional files that the publishing tool wasn?t even aware might exist someday to get added to the metadata directory, installation tools should preserve those files when installing them. With those two generic features, then entry points (and other things!) can be written on top of the ecosystem *without* needing to standardize on one solution for one particular non-packaging problem.

If a publishing tool doesn?t want to provide that mechanism, then that is fine, but that limits their audience (in the same way that not building C extensions limits their audience, people who need that capability won?t be able to use them).

> 
>> * We?re explicitly saying that this is the one true way of solving this
>> problem in the Python ecosystem.
> 
> I don't buy that at all. We're saying that it exists, and this is what
> it is.

It?s literally the way all of our packaging standards are written. Don?t use eggs, wheels are the one true way, don?t use YOLO versions, PEP 440 is the one true way, don?t add arbitrary extensions to the simple repo format, PEP 503 API Is the one true way, etc etc etc.

> 
>> * We stifle innovation (hell just including it in setutools at all does
>> this, but we can?t unopen that can of worms).
> 
> I don't think that's true to any significant extent. Having a standard
> does not stop people coming up with something better.

It doesn?t actively prevent someone from coming up with something better no, but what it does do is add a pretty huge barrier to entry for someone who wanted to come up with something better. It?s the same way that something being added to the stdlib stifles competition. When something is ?the standard?, it discourages people from even trying to make something better? or if they do make other people from trying it, unless ?the standard? is really bad.

> 
>> * We make it actively harder to improve the feature (since once it?s part
>> of the purview of packaging standards, all of distutils-sig gets to weigh
>> in on improvements).
> 
> It hasn't changed in years, as far as I know, and it's so widely used
> that any change is likely to break a load of stuff anyway. As we've
> already discussed for caching, we can improve by building *on top* of it
> relatively easily. And ultimately I think that bringing it out into
> daylight leads to a healthier future than leaving it under the stone
> marked 'setuptools''.
> 


If I could guess, I?d say it hasn?t changed in years because setuptools has had bigger things to work on and not enough time to do it in.

From ncoghlan at gmail.com  Fri Oct 20 08:14:13 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 20 Oct 2017 22:14:13 +1000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1508500651.117367.1145301232.4A2422BF@webmail.messagingengine.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>
 <5F9E5F93-7AF4-4D05-956F-A4367F89CD96@stufft.io>
 <1508500651.117367.1145301232.4A2422BF@webmail.messagingengine.com>
Message-ID: <CADiSq7fN8Bb5xj0f3-CTZ6UczULX8da=XxyoBwp7X4We8_S_Nw@mail.gmail.com>

On 20 October 2017 at 21:57, Thomas Kluyver <thomas at kluyver.me.uk> wrote:

> On Fri, Oct 20, 2017, at 12:50 PM, Donald Stufft wrote:
> > * We stifle innovation (hell just including it in setutools at all does
> > this, but we can?t unopen that can of worms).
>
> I don't think that's true to any significant extent. Having a standard
> does not stop people coming up with something better.
>

entry_points.txt will be hard to change for similar reasons to why PKG-INFO
is hard to change, but that challenge exists regardless of whether we
consider it a setuptools/pkg_resources feature or an ecosystem level
standard, since it relates to coupling between metadata publishers and
consumers of that metadata.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/ee2989e0/attachment.html>

From donald at stufft.io  Fri Oct 20 08:18:45 2017
From: donald at stufft.io (Donald Stufft)
Date: Fri, 20 Oct 2017 08:18:45 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CADiSq7e81J-S-=8TYuHrUt5SDcoCitv_aExCNA2ytyoVcSdg1w@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <CADiSq7e81J-S-=8TYuHrUt5SDcoCitv_aExCNA2ytyoVcSdg1w@mail.gmail.com>
Message-ID: <393D9291-2205-45F5-A9F4-1CAFAF1593CF@stufft.io>



> On Oct 20, 2017, at 8:06 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 20 October 2017 at 21:15, Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
> Tell you what, I?ll drop everything today and write up a PEP that adds metadata for console scripts to the packaging metadata where it belongs,
> 
> Donald, you're making the same mistake I did with PEP 426: interoperability specifications are useless without a commitment from tooling developers to actually provide implementations that know how to read and write them. And since any new format you come up with won't be supported by existing pip and pkg_resources installations, there won't be any incentive for publishers to start using it, which means there's no incentives for runtime libraries to learn how to read it, etc, etc.

Not particularly no.

I can promise you 100% that pip will support it in the next version once I write it. I can also promise you that setuptools will have a PR to support it (not pkg_resources, because console scripts are a install time feature not a runtime feature), and I assume Jason would be happy to merge it.

So there?s commitment from at least one tool.

The ?existing installations? is horse shit, because existing implementations won?t support *any* new feature of anything so it can literally be used as a justification for doing nothing about anything except standardizing what already exists. I guess we shouldn?t have done PEP 517 or PEP 518 because, by your logic here, since it won?t be supported by existing tooling, there won?t be any incentive for people to use it ever.

> 
> In this case, we already have a perfectly serviceable format (entry_points.txt), a reference publisher (setuptools.setup) and a reference consumer (pkg_resources). The fact that the reference consumer is pkg_resources rather than pip doesn't suddenly take this outside the domain of responsibility of distutils-sig as a whole - it only takes it outside the domain of responsibility of PyPI.
> 
> So if you want to say it is neither pip's nor PyPI's responsibility to say anything one way or the other about the entry points format (beyond whether or not they're used to declare console scripts in a way that pip understands), then I agree with you entirely. This spec isn't something you personally need to worry about, since it doesn't impact any of the tools you work on (aside from giving pip's existing console_scripts implementation a firmer foundation from an interoperability perpsective).

My objection has absolutely nothing to do with whether pip is the consumer or not. My objection is entirely based on the fact that a plugin system is no .a packaging related feature and it doesn?t become one because a packaging tool once added a plugin system.

> 
> So the core of our disagreement is whether or not interfaces involving pip and PyPI represent the limits of distutil-sig's responsibility. They don't, and that's reflected in the fact we have a split standing delegation from Guido (one initially to Richard Jones and later to you for changes that affect PyPI, and one to me for packaging ecosystem interoperability specifications in general)

No that?s not the core of our disagreement. The core of our disagreement is whether random runtime features suddenly become a packaging concern because they were implemented by one packaging tool once.

>  
> so we can move console_scripts entry point to a legacy footnote as far as packaging systems go. Then we can discuss whether an arbitrary plugin system is actually a packaging related spec (it?s not) on it?s own merits.
> 
> Instructing publishing system developers on how to publish pkg_resources compatible entry points is indeed a Python packaging ecosystem level concern. 

No it?s really not.

> 
> Whether that capability survives into a hypothetical future metadata specification (whether that's PEP 426+459 or something else entirely) would then be a different question, but it isn't one we need to worry about right now (since it purely affects internal interoperability file formats that only automated scripts and folks maintaining those scripts need to care about, and we'd expect entry_points.txt and PKG-INFO to coexist alongside any new format for a *long* time).
> 
> Cheers,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncoghlan at gmail.com <mailto:ncoghlan at gmail.com>   |   Brisbane, Australia

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/0ec8faa6/attachment-0001.html>

From ncoghlan at gmail.com  Fri Oct 20 08:23:03 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 20 Oct 2017 22:23:03 +1000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <6BBFF1A1-F05D-496A-8BEB-FB65D7554823@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>
 <5F9E5F93-7AF4-4D05-956F-A4367F89CD96@stufft.io>
 <1508500651.117367.1145301232.4A2422BF@webmail.messagingengine.com>
 <6BBFF1A1-F05D-496A-8BEB-FB65D7554823@stufft.io>
Message-ID: <CADiSq7dmmoB_Hj_Q7ZurMJ1f5e1g=K4oAQQnx5xFucccxUyWhQ@mail.gmail.com>

On 20 October 2017 at 22:10, Donald Stufft <donald at stufft.io> wrote:

> If I could guess, I?d say it hasn?t changed in years because setuptools
> has had bigger things to work on and not enough time to do it in.


Then you'd be wrong - it hasn't changed in years because it's a sensible,
simple solution to the problem of declaring integration points between
independently distributed pieces of software that allows the installed
integration points to be listed *without* importing the software providing
them (unlike most import based plugin systems).

And yes, I know you're attempting to claim that "declaring integration
points between independently distributed pieces of software" isn't
something that's a packaging ecosystem level to concern.

It is an ecosystem level concern, but we haven't had to worry about it
previously, because entry points haven't had problems to be fixed the way
that other aspects of setuptools have (lack of uninstall support in
easy_install, lack of filesystem layout metadata in eggs, ordering quirks
in the versioning scheme). For entry points, by contrast, the only missing
piece is explicit documentation of the file format used in distribution
archives and the installation database.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/7c614852/attachment.html>

From thomas at kluyver.me.uk  Fri Oct 20 08:30:05 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Fri, 20 Oct 2017 13:30:05 +0100
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <393D9291-2205-45F5-A9F4-1CAFAF1593CF@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <CADiSq7e81J-S-=8TYuHrUt5SDcoCitv_aExCNA2ytyoVcSdg1w@mail.gmail.com>
 <393D9291-2205-45F5-A9F4-1CAFAF1593CF@stufft.io>
Message-ID: <1508502605.122931.1145330736.2B2C0A1E@webmail.messagingengine.com>

On Fri, Oct 20, 2017, at 01:18 PM, Donald Stufft wrote:
> I guess we shouldn?t have done PEP 517 or PEP 518 because, by your
> logic here, since it won?t be supported by existing tooling, there
> won?t be any incentive for people to use it ever.
I see this as having a similar purpose to those PEPs: reducing
dependence on setuptools. The difference is that for building packages,
pip explicitly uses setuptools, so the practical way forward was to
define an alternative to achieve the same ends. For this, the existing
mechanism does not directly rely on setuptools, so it's sufficient to
document it so that other tools can confidently produce and consume it.
I also get annoyed at times by arguments that it's not worth improving
something because it will be a long time before the change is useful.
But I don't think that's what Nick is saying here.
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/401bbaa3/attachment.html>

From ncoghlan at gmail.com  Fri Oct 20 08:34:16 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 20 Oct 2017 22:34:16 +1000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <393D9291-2205-45F5-A9F4-1CAFAF1593CF@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <CADiSq7e81J-S-=8TYuHrUt5SDcoCitv_aExCNA2ytyoVcSdg1w@mail.gmail.com>
 <393D9291-2205-45F5-A9F4-1CAFAF1593CF@stufft.io>
Message-ID: <CADiSq7c9bZvNwcVq4MtggzEGScyhTYboO3a0L=60b_TTdZ_S5g@mail.gmail.com>

On 20 October 2017 at 22:18, Donald Stufft <donald at stufft.io> wrote:

>
> The ?existing installations? is horse shit, because existing
> implementations won?t support *any* new feature of anything so it can
> literally be used as a justification for doing nothing about anything
> except standardizing what already exists. I guess we shouldn?t have done
> PEP 517 or PEP 518 because, by your logic here, since it won?t be supported
> by existing tooling, there won?t be any incentive for people to use it ever.
>

No, because PEP 517 and 518 actually change the UX for *publishers* away
from setup.py to pyproject.toml + whatever build system they choose, while
allowing the definition of a *common* setup.py shim for compatibility with
older clients.

By contrast, it's relatively rare for people to edit entry_points.txt by
hand - it's typically a generated file, just like PKG-INFO.

For any *new* console_scripts replacement, you're also going to have define
how to translate it back to entry_points.txt for compatibility with older
pip installations, and that means you're also going to have to define how
to do that without conflicting with any other pkg_resources entry points
already declared by a package.

Those two characteristics mean that entry_points.txt has a lot more in
common with PKG-INFO than it does with setup.py, and that similarity is
further enhanced by the fact that it's a pretty easy format to document.

> So if you want to say it is neither pip's nor PyPI's responsibility to say
> anything one way or the other about the entry points format (beyond whether
> or not they're used to declare console scripts in a way that pip
> understands), then I agree with you entirely. This spec isn't something you
> personally need to worry about, since it doesn't impact any of the tools
> you work on (aside from giving pip's existing console_scripts
> implementation a firmer foundation from an interoperability perpsective).
>
>
> My objection has absolutely nothing to do with whether pip is the consumer
> or not. My objection is entirely based on the fact that a plugin system is
> no .a packaging related feature and it doesn?t become one because a
> packaging tool once added a plugin system.
>

You're acting like you believe you have veto power over this topic. You
don't - it's not a PyPI related concern, and it doesn't require any changes
to pip or warehouse.

I'd certainly be *happier* if you were only -0 rather than -1, but your
disapproval won't prevent me from accepting Thomas's PR either way.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/f3102f28/attachment-0001.html>

From donald at stufft.io  Fri Oct 20 08:36:53 2017
From: donald at stufft.io (Donald Stufft)
Date: Fri, 20 Oct 2017 08:36:53 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CADiSq7dmmoB_Hj_Q7ZurMJ1f5e1g=K4oAQQnx5xFucccxUyWhQ@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>
 <5F9E5F93-7AF4-4D05-956F-A4367F89CD96@stufft.io>
 <1508500651.117367.1145301232.4A2422BF@webmail.messagingengine.com>
 <6BBFF1A1-F05D-496A-8BEB-FB65D7554823@stufft.io>
 <CADiSq7dmmoB_Hj_Q7ZurMJ1f5e1g=K4oAQQnx5xFucccxUyWhQ@mail.gmail.com>
Message-ID: <5BD2E6C3-2D0C-45FC-A15D-66ACDBC6B33D@stufft.io>


> On Oct 20, 2017, at 8:23 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 20 October 2017 at 22:10, Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
> If I could guess, I?d say it hasn?t changed in years because setuptools has had bigger things to work on and not enough time to do it in.
> 
> Then you'd be wrong - it hasn't changed in years because it's a sensible, simple solution to the problem of declaring integration points between independently distributed pieces of software that allows the installed integration points to be listed *without* importing the software providing them (unlike most import based plugin systems).

I mean no I?m not.

Entry points have a lot of problems and I know of multiple systems that have either moved away from them, had to hack around how bad they are, have refused to implement them because of previous pain felt by them, are looking for ways to eliminate them, or which just regret ever supporting them.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/3326200e/attachment.html>

From donald at stufft.io  Fri Oct 20 08:38:37 2017
From: donald at stufft.io (Donald Stufft)
Date: Fri, 20 Oct 2017 08:38:37 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CADiSq7c9bZvNwcVq4MtggzEGScyhTYboO3a0L=60b_TTdZ_S5g@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <CADiSq7e81J-S-=8TYuHrUt5SDcoCitv_aExCNA2ytyoVcSdg1w@mail.gmail.com>
 <393D9291-2205-45F5-A9F4-1CAFAF1593CF@stufft.io>
 <CADiSq7c9bZvNwcVq4MtggzEGScyhTYboO3a0L=60b_TTdZ_S5g@mail.gmail.com>
Message-ID: <1DFD4CFC-A4C2-40A7-A120-CC4B3B64E6E4@stufft.io>



> On Oct 20, 2017, at 8:34 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> You're acting like you believe you have veto power over this topic. You don't - it's not a PyPI related concern, and it doesn't require any changes to pip or warehouse.
> 
> I'd certainly be *happier* if you were only -0 rather than -1, but your disapproval won't prevent me from accepting Thomas's PR either way.


I?m acting like I have an opinion. You?re obviously free to accept something that I think is a bad idea, that doesn?t mean I should just shut up and not voice my concerns or objections and I?d appreciate it if you didn?t imply that I should.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/fe9ac904/attachment.html>

From thomas at kluyver.me.uk  Fri Oct 20 08:41:15 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Fri, 20 Oct 2017 13:41:15 +0100
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <5BD2E6C3-2D0C-45FC-A15D-66ACDBC6B33D@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>
 <5F9E5F93-7AF4-4D05-956F-A4367F89CD96@stufft.io>
 <1508500651.117367.1145301232.4A2422BF@webmail.messagingengine.com>
 <6BBFF1A1-F05D-496A-8BEB-FB65D7554823@stufft.io>
 <CADiSq7dmmoB_Hj_Q7ZurMJ1f5e1g=K4oAQQnx5xFucccxUyWhQ@mail.gmail.com>
 <5BD2E6C3-2D0C-45FC-A15D-66ACDBC6B33D@stufft.io>
Message-ID: <1508503275.124767.1145341384.76E2EDA6@webmail.messagingengine.com>

On Fri, Oct 20, 2017, at 01:36 PM, Donald Stufft wrote:
> Entry points have a lot of problems and I know of multiple systems
> that have either moved away from them, had to hack around how bad they
> are, have refused to implement them because of previous pain felt by
> them, are looking for ways to eliminate them, or which just regret
> ever supporting them.
The fate of the PR notwithstanding, I'd be interested in hearing more
about what problems projects have experienced with entry points, if you
have time to describe some examples. We're looking at using them in more
places than we already do, so it would be useful to hear about drawbacks
we might not have thought about, and about what other options projects
have moved to.
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/f8303d4c/attachment.html>

From wes.turner at gmail.com  Fri Oct 20 08:58:09 2017
From: wes.turner at gmail.com (Wes Turner)
Date: Fri, 20 Oct 2017 08:58:09 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1508503275.124767.1145341384.76E2EDA6@webmail.messagingengine.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>
 <5F9E5F93-7AF4-4D05-956F-A4367F89CD96@stufft.io>
 <1508500651.117367.1145301232.4A2422BF@webmail.messagingengine.com>
 <6BBFF1A1-F05D-496A-8BEB-FB65D7554823@stufft.io>
 <CADiSq7dmmoB_Hj_Q7ZurMJ1f5e1g=K4oAQQnx5xFucccxUyWhQ@mail.gmail.com>
 <5BD2E6C3-2D0C-45FC-A15D-66ACDBC6B33D@stufft.io>
 <1508503275.124767.1145341384.76E2EDA6@webmail.messagingengine.com>
Message-ID: <CACfEFw_saF04BBGGFR+CSbhJksx3JBoJr0m_RpBF+gtuVTgT1g@mail.gmail.com>

On Friday, October 20, 2017, Thomas Kluyver <thomas at kluyver.me.uk> wrote:

> On Fri, Oct 20, 2017, at 01:36 PM, Donald Stufft wrote:
>
> Entry points have a lot of problems and I know of multiple systems that
> have either moved away from them, had to hack around how bad they are, have
> refused to implement them because of previous pain felt by them, are
> looking for ways to eliminate them, or which just regret ever supporting
> them.
>
>
> The fate of the PR notwithstanding, I'd be interested in hearing more
> about what problems projects have experienced with entry points, if you
> have time to describe some examples. We're looking at using them in more
> places than we already do, so it would be useful to hear about drawbacks we
> might not have thought about, and about what other options projects have
> moved to.
>
> Thomas
>

 What were the issues with setuptools entry points here (in 2014, when you
two were opposed to adding them to sendibly list ipython extensions)?

https://github.com/ipython/ipython/pull/4673

https://github.com/ipython/ipython/compare/master...westurner:setuptools_entry_points
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/e8aa8c85/attachment-0001.html>

From thomas at kluyver.me.uk  Fri Oct 20 09:05:45 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Fri, 20 Oct 2017 14:05:45 +0100
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CACfEFw_saF04BBGGFR+CSbhJksx3JBoJr0m_RpBF+gtuVTgT1g@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>
 <5F9E5F93-7AF4-4D05-956F-A4367F89CD96@stufft.io>
 <1508500651.117367.1145301232.4A2422BF@webmail.messagingengine.com>
 <6BBFF1A1-F05D-496A-8BEB-FB65D7554823@stufft.io>
 <CADiSq7dmmoB_Hj_Q7ZurMJ1f5e1g=K4oAQQnx5xFucccxUyWhQ@mail.gmail.com>
 <5BD2E6C3-2D0C-45FC-A15D-66ACDBC6B33D@stufft.io>
 <1508503275.124767.1145341384.76E2EDA6@webmail.messagingengine.com>
 <CACfEFw_saF04BBGGFR+CSbhJksx3JBoJr0m_RpBF+gtuVTgT1g@mail.gmail.com>
Message-ID: <1508504745.130424.1145361984.4D7C6A18@webmail.messagingengine.com>

On Fri, Oct 20, 2017, at 01:58 PM, Wes Turner wrote:
> What were the issues with setuptools entry points here (in 2014, when
> you two were opposed to adding them to sendibly list ipython
> extensions)?
I'm impressed by your memory! The main issue then was that it implied
that extension authors would have to use setuptools.
Setuptools has got much better since then, we have better tools and
norms for dealing with its rough edges, and there are usable alternative
tools that can be used to distribute entrypoints. But the description
I've written up is still basically trying to solve the same problem: an
application should be able to use entry points without forcing all
plugins to use setuptools.
Thomas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/f427ac02/attachment.html>

From donald at stufft.io  Fri Oct 20 09:19:37 2017
From: donald at stufft.io (Donald Stufft)
Date: Fri, 20 Oct 2017 09:19:37 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1508503275.124767.1145341384.76E2EDA6@webmail.messagingengine.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>
 <5F9E5F93-7AF4-4D05-956F-A4367F89CD96@stufft.io>
 <1508500651.117367.1145301232.4A2422BF@webmail.messagingengine.com>
 <6BBFF1A1-F05D-496A-8BEB-FB65D7554823@stufft.io>
 <CADiSq7dmmoB_Hj_Q7ZurMJ1f5e1g=K4oAQQnx5xFucccxUyWhQ@mail.gmail.com>
 <5BD2E6C3-2D0C-45FC-A15D-66ACDBC6B33D@stufft.io>
 <1508503275.124767.1145341384.76E2EDA6@webmail.messagingengine.com>
Message-ID: <F0F05ABC-788F-49AF-9905-E98F7FDDB988@stufft.io>

On Oct 20, 2017, at 8:41 AM, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> 
> On Fri, Oct 20, 2017, at 01:36 PM, Donald Stufft wrote:
>> Entry points have a lot of problems and I know of multiple systems that have either moved away from them, had to hack around how bad they are, have refused to implement them because of previous pain felt by them, are looking for ways to eliminate them, or which just regret ever supporting them.
> 
> The fate of the PR notwithstanding, I'd be interested in hearing more about what problems projects have experienced with entry points, if you have time to describe some examples. We're looking at using them in more places than we already do, so it would be useful to hear about drawbacks we might not have thought about, and about what other options projects have moved to.
> 

One that I was helping someone debug just the other day is that they?re super non-debuggable and the behavior when you have two things providing the same entry point is basically ???? (If I remember correctly, the behavior is that the first thing found is the one that ?wins?, which means the ordering of sys.path and the names of the projects supply it is what determines it). This got exposed to the end user that they installed something that they thought was going to add support for something, but which silently did nothing because two different project happened to pick the same name for their entry point (not the group, it was two things providing plugins for the same system).

Of course there is the perennial entrypoints are super slow, which is partially the fault of pkg_resources does a bunch of import time logic, but also because scanning sys.path for all installed stuff is just slow.

They?re also somewhat fragile since they rely on the packaging metadata system at runtime, and a number of tools exclude that information (often times things that deploy stuff as a tarball/zipfile) which causes regular issues to be opened up for these projects when they get used in those environments.

Those are the ones I remember because they come up regularly (and people regularly come to me with issues with any project related to packaging in any way even for non packaging related features in those projects). I?m pretty sure there were more of them that I?ve encountered and seen projects encounter, but I can?t remember them to be sure.

I?m more familiar with why console_scripts entry point is not great and why we should stop using it since I regularly try to re-read all of pip?s issues and a lot of it?s issues are documented there.


From p.f.moore at gmail.com  Fri Oct 20 09:22:03 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 20 Oct 2017 14:22:03 +0100
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
Message-ID: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>

We're in the process of starting to plan for a release of pip (the
long-awaited pip 10). We're likely still a month or two away from a
release, but now is the time for people to start ensuring that
everything works for them. One key change in the new version will be
that all of the internal APIs of pip will no longer be available, so
any code that currently calls functions in the "pip" namespace will
break. Calling pip's internal APIs has never been supported, and
always carried a risk of such breakage, so projects doing so should,
in theory, be prepared for such things. However, reality is not always
that simple, and we are aware that people will need time to deal with
the implications.

Just in case it's not clear, simply finding where the internal APIs
have moved to and calling them under the new names is *not* what
people should do. We can't stop people calling the internal APIs,
obviously, but the idea of this change is to give people the incentive
to find a supported approach, not just to annoy people who are doing
things we don't want them to ;-)

So please - if you're calling pip's internals in your code, take the
opportunity *now* to check out the in-development version of pip, and
ensure your project will still work when pip 10 is released.

And many thanks to anyone else who helps by testing out the new
version, as well :-)

Thanks,
Paul

From matthew.brett at gmail.com  Fri Oct 20 09:26:47 2017
From: matthew.brett at gmail.com (Matthew Brett)
Date: Fri, 20 Oct 2017 14:26:47 +0100
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
Message-ID: <CAH6Pt5o58J8b801UR-VWU2w_Ahmc2VpcOc3ey8dhPPnfFeGZGQ@mail.gmail.com>

Hi,

On Fri, Oct 20, 2017 at 2:22 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> We're in the process of starting to plan for a release of pip (the
> long-awaited pip 10). We're likely still a month or two away from a
> release, but now is the time for people to start ensuring that
> everything works for them. One key change in the new version will be
> that all of the internal APIs of pip will no longer be available, so
> any code that currently calls functions in the "pip" namespace will
> break. Calling pip's internal APIs has never been supported, and
> always carried a risk of such breakage, so projects doing so should,
> in theory, be prepared for such things. However, reality is not always
> that simple, and we are aware that people will need time to deal with
> the implications.
>
> Just in case it's not clear, simply finding where the internal APIs
> have moved to and calling them under the new names is *not* what
> people should do. We can't stop people calling the internal APIs,
> obviously, but the idea of this change is to give people the incentive
> to find a supported approach, not just to annoy people who are doing
> things we don't want them to ;-)
>
> So please - if you're calling pip's internals in your code, take the
> opportunity *now* to check out the in-development version of pip, and
> ensure your project will still work when pip 10 is released.
>
> And many thanks to anyone else who helps by testing out the new
> version, as well :-)

Thanks for the heads-up.

Will y'all be doing a PyPI pre-release so we can test with `pip
install --pre -U pip`?

Cheers,

Matthew

From p.f.moore at gmail.com  Fri Oct 20 09:30:26 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 20 Oct 2017 14:30:26 +0100
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <CAH6Pt5o58J8b801UR-VWU2w_Ahmc2VpcOc3ey8dhPPnfFeGZGQ@mail.gmail.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <CAH6Pt5o58J8b801UR-VWU2w_Ahmc2VpcOc3ey8dhPPnfFeGZGQ@mail.gmail.com>
Message-ID: <CACac1F8bZD8CgF8-9JYvaRD+SdW4J+MTE66TvXrJN8MGBQpaEA@mail.gmail.com>

On 20 October 2017 at 14:26, Matthew Brett <matthew.brett at gmail.com> wrote:
> Thanks for the heads-up.
>
> Will y'all be doing a PyPI pre-release so we can test with `pip
> install --pre -U pip`?

We've not yet decided on that. Traditionally I don't think we have
done so, but I'm inclined to think it's a good idea. It might not be
until noticeably closer to the release, though...

Paul

From ncoghlan at gmail.com  Fri Oct 20 09:35:40 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 20 Oct 2017 23:35:40 +1000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <F0F05ABC-788F-49AF-9905-E98F7FDDB988@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>
 <5F9E5F93-7AF4-4D05-956F-A4367F89CD96@stufft.io>
 <1508500651.117367.1145301232.4A2422BF@webmail.messagingengine.com>
 <6BBFF1A1-F05D-496A-8BEB-FB65D7554823@stufft.io>
 <CADiSq7dmmoB_Hj_Q7ZurMJ1f5e1g=K4oAQQnx5xFucccxUyWhQ@mail.gmail.com>
 <5BD2E6C3-2D0C-45FC-A15D-66ACDBC6B33D@stufft.io>
 <1508503275.124767.1145341384.76E2EDA6@webmail.messagingengine.com>
 <F0F05ABC-788F-49AF-9905-E98F7FDDB988@stufft.io>
Message-ID: <CADiSq7dE8ZxPZRj0RChEc_65kMpQNUzos_gN-hd6FBu_cLPQ_Q@mail.gmail.com>

On 20 October 2017 at 23:19, Donald Stufft <donald at stufft.io> wrote:

> One that I was helping someone debug just the other day is that they?re
> super non-debuggable and the behavior when you have two things providing
> the same entry point is basically ???? (If I remember correctly, the
> behavior is that the first thing found is the one that ?wins?, which means
> the ordering of sys.path and the names of the projects supply it is what
> determines it). This got exposed to the end user that they installed
> something that they thought was going to add support for something, but
> which silently did nothing because two different project happened to pick
> the same name for their entry point (not the group, it was two things
> providing plugins for the same system).
>

While I agree with this, I think that's a combination of pkg_resources
itself being hard to debug in general, and the fact that pkg_resources
doesn't clearly define the semantics of how it resolves name conflicts
within an entry point group - as far as I know, it's largely an accident of
implementation.

The interoperability spec is going to state that conflict resolution when
the same name within a group is declared by multiple packages is the
responsibility of the group consumer, so documenting the format should
actually improve this situation, since it allows for the development of
competing conflict resolution strategies in different runtime libraries.


> Of course there is the perennial entrypoints are super slow, which is
> partially the fault of pkg_resources does a bunch of import time logic, but
> also because scanning sys.path for all installed stuff is just slow.
>

Similar to the above, one of the goals of documenting the entry point file
format is to permit libraries to compete in the development of effective
entrypoint metadata caching strategies without needing to bless any
particular one a priori, and without trying to manage experimental cache
designs across the huge pkg_resources install base.


> They?re also somewhat fragile since they rely on the packaging metadata
> system at runtime, and a number of tools exclude that information (often
> times things that deploy stuff as a tarball/zipfile) which causes regular
> issues to be opened up for these projects when they get used in those
> environments.
>

This is true, and one of the main pragmatic benefits of adopting one of the
purely import based plugin management systems. However, this problem will
impact all packaging metadata based plugin management solutions, regardless
of whether they use an existing file format or a new one.


> Those are the ones I remember because they come up regularly (and people
> regularly come to me with issues with any project related to packaging in
> any way even for non packaging related features in those projects). I?m
> pretty sure there were more of them that I?ve encountered and seen projects
> encounter, but I can?t remember them to be sure.
>
> I?m more familiar with why console_scripts entry point is not great and
> why we should stop using it since I regularly try to re-read all of pip?s
> issues and a lot of it?s issues are documented there.
>

I'm sympathetic to that, but I think even in that case, clearly documenting
the format as an interoperability specification will help tease out which
of those are due to the file format itself, and which are due to
setuptools.setup specifically.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/047d0efb/attachment.html>

From donald at stufft.io  Fri Oct 20 09:42:20 2017
From: donald at stufft.io (Donald Stufft)
Date: Fri, 20 Oct 2017 09:42:20 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CADiSq7dE8ZxPZRj0RChEc_65kMpQNUzos_gN-hd6FBu_cLPQ_Q@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>
 <5F9E5F93-7AF4-4D05-956F-A4367F89CD96@stufft.io>
 <1508500651.117367.1145301232.4A2422BF@webmail.messagingengine.com>
 <6BBFF1A1-F05D-496A-8BEB-FB65D7554823@stufft.io>
 <CADiSq7dmmoB_Hj_Q7ZurMJ1f5e1g=K4oAQQnx5xFucccxUyWhQ@mail.gmail.com>
 <5BD2E6C3-2D0C-45FC-A15D-66ACDBC6B33D@stufft.io>
 <1508503275.124767.1145341384.76E2EDA6@webmail.messagingengine.com>
 <F0F05ABC-788F-49AF-9905-E98F7FDDB988@stufft.io>
 <CADiSq7dE8ZxPZRj0RChEc_65kMpQNUzos_gN-hd6FBu_cLPQ_Q@mail.gmail.com>
Message-ID: <A2AB937D-D8E0-44E2-8471-3E2D1F433929@stufft.io>



> On Oct 20, 2017, at 9:35 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 20 October 2017 at 23:19, Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
> One that I was helping someone debug just the other day is that they?re super non-debuggable and the behavior when you have two things providing the same entry point is basically ???? (If I remember correctly, the behavior is that the first thing found is the one that ?wins?, which means the ordering of sys.path and the names of the projects supply it is what determines it). This got exposed to the end user that they installed something that they thought was going to add support for something, but which silently did nothing because two different project happened to pick the same name for their entry point (not the group, it was two things providing plugins for the same system).
> 
> While I agree with this, I think that's a combination of pkg_resources itself being hard to debug in general, and the fact that pkg_resources doesn't clearly define the semantics of how it resolves name conflicts within an entry point group - as far as I know, it's largely an accident of implementation.
> 
> The interoperability spec is going to state that conflict resolution when the same name within a group is declared by multiple packages is the responsibility of the group consumer, so documenting the format should actually improve this situation, since it allows for the development of competing conflict resolution strategies in different runtime libraries.

I think it makes it *worse*, because now the behavior isn?t just a entrypoints weirdness, but now it changes based on which runtime library you use (which isn?t something that end users are likely to have much insight into) and it represents a footgun that package authors are unlikely to be aware of. If mycoolentrypointslib comes out that is faster, but changes some subtle behavior like this it?ll break people, but that is unlikely going to be an effect that people expect to happen just because they switched between two things both implementing the same standard.

So effectively this means that not only is the fact you?re using entrypoints part of your API, but now which entry point library you?re using at runtime is now also part of your API.

>  
> Of course there is the perennial entrypoints are super slow, which is partially the fault of pkg_resources does a bunch of import time logic, but also because scanning sys.path for all installed stuff is just slow.
> 
> Similar to the above, one of the goals of documenting the entry point file format is to permit libraries to compete in the development of effective entrypoint metadata caching strategies without needing to bless any particular one a priori, and without trying to manage experimental cache designs across the huge pkg_resources install base.

That goal can be achieved if it?s documented in setuptools.

>  
> They?re also somewhat fragile since they rely on the packaging metadata system at runtime, and a number of tools exclude that information (often times things that deploy stuff as a tarball/zipfile) which causes regular issues to be opened up for these projects when they get used in those environments.
> 
> This is true, and one of the main pragmatic benefits of adopting one of the purely import based plugin management systems. However, this problem will impact all packaging metadata based plugin management solutions, regardless of whether they use an existing file format or a new one.
>  
> Those are the ones I remember because they come up regularly (and people regularly come to me with issues with any project related to packaging in any way even for non packaging related features in those projects). I?m pretty sure there were more of them that I?ve encountered and seen projects encounter, but I can?t remember them to be sure.
> 
> I?m more familiar with why console_scripts entry point is not great and why we should stop using it since I regularly try to re-read all of pip?s issues and a lot of it?s issues are documented there.
> 
> I'm sympathetic to that, but I think even in that case, clearly documenting the format as an interoperability specification will help tease out which of those are due to the file format itself, and which are due to setuptools.setup specifically.

All of the ones I?m aware of are due to the file format itself, because they exist even without setuptools being involved at all.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/93f37493/attachment-0001.html>

From ronaldoussoren at mac.com  Fri Oct 20 08:42:56 2017
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Fri, 20 Oct 2017 14:42:56 +0200
Subject: [Distutils] pythonhosted.org doc upload no longer works
In-Reply-To: <CAFYqXL8dZWo4KdVq2GQU-N137DrOLxVChXuX39TEC95P64eDDQ@mail.gmail.com>
References: <CAFYqXL9AXXTdgLfEwBbnh1DejZ1sRqNGp1Dh=o19Rgmqzvokxw@mail.gmail.com>
 <CADiSq7dPLj4J2TgScLscCr8QM3BasP9q9wsarKJecyARJq4BPw@mail.gmail.com>
 <CAF4280LMiC14PjPkACFFxsyshanVHkO5PX0azUdRvr-6cguOmQ@mail.gmail.com>
 <CAFYqXL-9i9MJjTX1Za22L4FqV5pZap1Qihv5Y909eZc0rNbZTg@mail.gmail.com>
 <C5823E59-AB8C-4DA8-AD4A-AE970B062061@coderanger.net>
 <CAFYqXL9kZDc+UicOq_W8POZ+nT5W+dBgFSSZ7NMkUSGAGE2xLA@mail.gmail.com>
 <CAFYqXL-wKDhG8h00esnXUuRXYRQXEdMmGZbx+FLgKvPLXUyaUQ@mail.gmail.com>
 <CAFYqXL-uHDVZ=Yd0puWy=rA4GFNa8aGk2xSLu3ZSxgNvXu7ocA@mail.gmail.com>
 <CAFYqXL9Eq+BrcaX4YBpJutBVU5A5ncYiFXCW=K8xzmnO-hwFNg@mail.gmail.com>
 <CACfEFw9K_4tnEFyVY1fABUp_HnWdFuB7a3513Tg1TyPxmfTZag@mail.gmail.com>
 <CAFYqXL-wFYS80-LSaoVB0WvjnHq4mVfP=OxJNLAS4jTWGgN6jA@mail.gmail.com>
 <CAFYqXL8dZWo4KdVq2GQU-N137DrOLxVChXuX39TEC95P64eDDQ@mail.gmail.com>
Message-ID: <3FC71839-40AD-497B-8ADD-A343CDB2336A@mac.com>


> On 7 Oct 2017, at 09:12, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> 
> Any news about this please? This is becoming a problem as my outdated doc keeps being around and I have no way to remove it. 

I have the same problem with my projects: the documentation on pythonhosted.org can no longer be updated, and it is also not possible to change that documentation into a link to a new location.

I have new documentation on Read the Docs, but Google searches still prefer the old location and I would therefore prefer to have redirects on python host.org.

BTW. Has there been any documentation beyond this list about this change? 

Ronald

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/50fbd6b2/attachment.html>

From ncoghlan at gmail.com  Fri Oct 20 09:52:14 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 20 Oct 2017 23:52:14 +1000
Subject: [Distutils] pythonhosted.org doc upload no longer works
In-Reply-To: <3FC71839-40AD-497B-8ADD-A343CDB2336A@mac.com>
References: <CAFYqXL9AXXTdgLfEwBbnh1DejZ1sRqNGp1Dh=o19Rgmqzvokxw@mail.gmail.com>
 <CADiSq7dPLj4J2TgScLscCr8QM3BasP9q9wsarKJecyARJq4BPw@mail.gmail.com>
 <CAF4280LMiC14PjPkACFFxsyshanVHkO5PX0azUdRvr-6cguOmQ@mail.gmail.com>
 <CAFYqXL-9i9MJjTX1Za22L4FqV5pZap1Qihv5Y909eZc0rNbZTg@mail.gmail.com>
 <C5823E59-AB8C-4DA8-AD4A-AE970B062061@coderanger.net>
 <CAFYqXL9kZDc+UicOq_W8POZ+nT5W+dBgFSSZ7NMkUSGAGE2xLA@mail.gmail.com>
 <CAFYqXL-wKDhG8h00esnXUuRXYRQXEdMmGZbx+FLgKvPLXUyaUQ@mail.gmail.com>
 <CAFYqXL-uHDVZ=Yd0puWy=rA4GFNa8aGk2xSLu3ZSxgNvXu7ocA@mail.gmail.com>
 <CAFYqXL9Eq+BrcaX4YBpJutBVU5A5ncYiFXCW=K8xzmnO-hwFNg@mail.gmail.com>
 <CACfEFw9K_4tnEFyVY1fABUp_HnWdFuB7a3513Tg1TyPxmfTZag@mail.gmail.com>
 <CAFYqXL-wFYS80-LSaoVB0WvjnHq4mVfP=OxJNLAS4jTWGgN6jA@mail.gmail.com>
 <CAFYqXL8dZWo4KdVq2GQU-N137DrOLxVChXuX39TEC95P64eDDQ@mail.gmail.com>
 <3FC71839-40AD-497B-8ADD-A343CDB2336A@mac.com>
Message-ID: <CADiSq7e6_oynApo5zok9_uB9mTv86M7ro_NbVEzbcREzWOrQ1w@mail.gmail.com>

On 20 October 2017 at 22:42, Ronald Oussoren <ronaldoussoren at mac.com> wrote:

>
> On 7 Oct 2017, at 09:12, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
>
> Any news about this please? This is becoming a problem as my outdated doc
> keeps being around and I have no way to remove it.
>
>
> I have the same problem with my projects: the documentation on
> pythonhosted.org can no longer be updated, and it is also not possible to
> change that documentation into a link to a new location.
>
> I have new documentation on Read the Docs, but Google searches still
> prefer the old location and I would therefore prefer to have redirects on
> python host.org.
>
> BTW. Has there been any documentation beyond this list about this change?
>

I don't believe so, as I believe the change from "deprecated" to "no longer
works" was a side effect of disabling uploads via the old service.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/6a08b842/attachment.html>

From ronaldoussoren at mac.com  Fri Oct 20 09:13:51 2017
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Fri, 20 Oct 2017 15:13:51 +0200
Subject: [Distutils] Documentation link on PyPI.org
In-Reply-To: <CAP1=2W6L79ZuPQ=pCsPGpOaTLf6GdpMUMpDxbDBMF-P6jireUg@mail.gmail.com>
References: <DF28CD64-C6F6-488A-AF45-593EB4FC1B1B@mac.com>
 <CACfEFw8nOUqn63akZJcY9xCUX6d=Cv22FvQiqz04cV=dhLF9-g@mail.gmail.com>
 <7E1ABA81-9C79-4D02-8135-6AA409B1A540@mac.com>
 <CAP1=2W6L79ZuPQ=pCsPGpOaTLf6GdpMUMpDxbDBMF-P6jireUg@mail.gmail.com>
Message-ID: <B31953AD-9784-4C49-ACB4-44DA3329FA3F@mac.com>


> On 28 Aug 2017, at 00:12, Brett Cannon <brett at python.org> wrote:
> 
> If you search the archive of this mailing list you will notice I asked this exact question about a month or 2 ago (I think). The answer I got was it is used on PyPI.org, but I don't know how to set it with setuptools (flit will probably add support after PEP 517).

To get back to this: I?m probably doing something wrong, but I haven?t managed yet to make anything show up on PyPI.org <http://pypi.org/> when I use ?Project-URL? in metadata. 

There?s a pyobjc-core 4.0.1b1 wheel on PyPI.org <http://pypi.org/> that includes two Project-URL lines in its metadata, but that doesn?t add anything to the sidebar. I?ve manually verified that the metadata is present in the wheel file (as I?m using a replacement egg_info command in my setup.py file to generate this part of the metadata).

Ronald

P.S. I tried to upload to test.pypi.org <http://test.pypi.org/>, but that didn?t work due to an SSL error:

$ twine upload -r testpypi dist/pyobjc_core-4.0.1b1-cp27-cp27m-macosx_10_6_intel.whl 
Uploading distributions to https://test.pypi.org/legacy/
Uploading pyobjc_core-4.0.1b1-cp27-cp27m-macosx_10_6_intel.whl
SSLError: HTTPSConnectionPool(host='test.pypi.org', port=443): Max retries exceeded with url: /legacy/ (Caused by SSLError(SSLError(1, u'[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661)'),))

> 
> On Sun, Aug 27, 2017, 13:01 Ronald Oussoren <ronaldoussoren at mac.com <mailto:ronaldoussoren at mac.com>> wrote:
>> On 27 Aug 2017, at 17:33, Wes Turner <wes.turner at gmail.com <mailto:wes.turner at gmail.com>> wrote:
>> 
>> You can add a URL in the long_description (e.g. to https://read-the-docs.readthedocs.org/ <https://read-the-docs.readthedocs.org/> )
> 
> I know, but that adds the URL to the project description and not the sidebar. PEP 345 appears to specify metadata that could be used for this, and warehouse seems to contain to make use of this (or at least a database model that allows for specifying more links than just the default one). 
> 
> Adding links to the sidebar would be nicer with the new design on PyPI.org <http://pypi.org/>, but there doesn?t seem to be a clean way to add ?Project-URL? metadata as described in PEP 345 to a wheel file. I could tweak my setup.py files to add Project-URL metadata, but before I do that I?d like to confirm that (a) that metadata is actually used by warehouse and (b) there is no cleaner way to add this metadata using setuptools.
> 
> Ronald
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org <mailto:Distutils-SIG at python.org>
> https://mail.python.org/mailman/listinfo/distutils-sig <https://mail.python.org/mailman/listinfo/distutils-sig>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/5c74f33b/attachment-0001.html>

From p.f.moore at gmail.com  Fri Oct 20 10:20:01 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 20 Oct 2017 15:20:01 +0100
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <BEF8CE18-C084-4556-A7A6-BFED5633654C@me.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <BEF8CE18-C084-4556-A7A6-BFED5633654C@me.com>
Message-ID: <CACac1F-p7uoVb=4o0Zue5TNQi2ZopOFkGayK2sVjAwfuN6cntA@mail.gmail.com>

On 20 October 2017 at 14:55, Jannis Gebauer <ja.geb at me.com> wrote:
> Thanks for the heads-up, Paul.
>
> I?m currently using `pip.get_installed_distributions` and as far as I can
> see that has moved into `_internal`, too:
> https://github.com/pypa/pip/blob/master/src/pip/_internal/utils/misc.py#L333
>
> Any recommendations?

See https://github.com/pypa/pip/pull/4743

Unfortunately, the "latest" docs build doesn't seem to reflect this (I
don't know why).

I guess you probably want something from pkg_resources? Depends
precisely what you're trying to do, I guess.
Paul

From wes.turner at gmail.com  Fri Oct 20 10:41:02 2017
From: wes.turner at gmail.com (Wes Turner)
Date: Fri, 20 Oct 2017 10:41:02 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <A2AB937D-D8E0-44E2-8471-3E2D1F433929@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>
 <5F9E5F93-7AF4-4D05-956F-A4367F89CD96@stufft.io>
 <1508500651.117367.1145301232.4A2422BF@webmail.messagingengine.com>
 <6BBFF1A1-F05D-496A-8BEB-FB65D7554823@stufft.io>
 <CADiSq7dmmoB_Hj_Q7ZurMJ1f5e1g=K4oAQQnx5xFucccxUyWhQ@mail.gmail.com>
 <5BD2E6C3-2D0C-45FC-A15D-66ACDBC6B33D@stufft.io>
 <1508503275.124767.1145341384.76E2EDA6@webmail.messagingengine.com>
 <F0F05ABC-788F-49AF-9905-E98F7FDDB988@stufft.io>
 <CADiSq7dE8ZxPZRj0RChEc_65kMpQNUzos_gN-hd6FBu_cLPQ_Q@mail.gmail.com>
 <A2AB937D-D8E0-44E2-8471-3E2D1F433929@stufft.io>
Message-ID: <CACfEFw8UUtXf-vS0xd9+vT042GJvJ7fOOFw7QANpwUGeFV0Rrg@mail.gmail.com>

On Friday, October 20, 2017, Donald Stufft <donald at stufft.io> wrote:

>
>
> On Oct 20, 2017, at 9:35 AM, Nick Coghlan <ncoghlan at gmail.com
> <javascript:_e(%7B%7D,'cvml','ncoghlan at gmail.com');>> wrote:
>
> On 20 October 2017 at 23:19, Donald Stufft <donald at stufft.io
> <javascript:_e(%7B%7D,'cvml','donald at stufft.io');>> wrote:
>
>> One that I was helping someone debug just the other day is that they?re
>> super non-debuggable and the behavior when you have two things providing
>> the same entry point is basically ???? (If I remember correctly, the
>> behavior is that the first thing found is the one that ?wins?, which means
>> the ordering of sys.path and the names of the projects supply it is what
>> determines it). This got exposed to the end user that they installed
>> something that they thought was going to add support for something, but
>> which silently did nothing because two different project happened to pick
>> the same name for their entry point (not the group, it was two things
>> providing plugins for the same system).
>>
>
> While I agree with this, I think that's a combination of pkg_resources
> itself being hard to debug in general, and the fact that pkg_resources
> doesn't clearly define the semantics of how it resolves name conflicts
> within an entry point group - as far as I know, it's largely an accident of
> implementation.
>
> The interoperability spec is going to state that conflict resolution when
> the same name within a group is declared by multiple packages is the
> responsibility of the group consumer, so documenting the format should
> actually improve this situation, since it allows for the development of
> competing conflict resolution strategies in different runtime libraries.
>
>
> I think it makes it *worse*, because now the behavior isn?t just a
> entrypoints weirdness, but now it changes based on which runtime library
> you use (which isn?t something that end users are likely to have much
> insight into) and it represents a footgun that package authors are unlikely
> to be aware of. If mycoolentrypointslib comes out that is faster, but
> changes some subtle behavior like this it?ll break people, but that is
> unlikely going to be an effect that people expect to happen just because
> they switched between two things both implementing the same standard.
>
> So effectively this means that not only is the fact you?re using
> entrypoints part of your API, but now which entry point library you?re
> using at runtime is now also part of your API.
>

When should the check for duplicate entry points occur?

- At on_install() time (+1)
- At runtime

Is a sys.path-like OrderedDict preemptive strategy preferable or just as
dangerous as importlib?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/690c7268/attachment.html>

From ja.geb at me.com  Fri Oct 20 09:55:48 2017
From: ja.geb at me.com (Jannis Gebauer)
Date: Fri, 20 Oct 2017 15:55:48 +0200
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
Message-ID: <BEF8CE18-C084-4556-A7A6-BFED5633654C@me.com>

Thanks for the heads-up, Paul.

I?m currently using `pip.get_installed_distributions` and as far as I can see that has moved into `_internal`, too: https://github.com/pypa/pip/blob/master/src/pip/_internal/utils/misc.py#L333 <https://github.com/pypa/pip/blob/master/src/pip/_internal/utils/misc.py#L333>

Any recommendations?

Thanks,

Jannis

> On 20. Oct 2017, at 15:22, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> We're in the process of starting to plan for a release of pip (the
> long-awaited pip 10). We're likely still a month or two away from a
> release, but now is the time for people to start ensuring that
> everything works for them. One key change in the new version will be
> that all of the internal APIs of pip will no longer be available, so
> any code that currently calls functions in the "pip" namespace will
> break. Calling pip's internal APIs has never been supported, and
> always carried a risk of such breakage, so projects doing so should,
> in theory, be prepared for such things. However, reality is not always
> that simple, and we are aware that people will need time to deal with
> the implications.
> 
> Just in case it's not clear, simply finding where the internal APIs
> have moved to and calling them under the new names is *not* what
> people should do. We can't stop people calling the internal APIs,
> obviously, but the idea of this change is to give people the incentive
> to find a supported approach, not just to annoy people who are doing
> things we don't want them to ;-)
> 
> So please - if you're calling pip's internals in your code, take the
> opportunity *now* to check out the in-development version of pip, and
> ensure your project will still work when pip 10 is released.
> 
> And many thanks to anyone else who helps by testing out the new
> version, as well :-)
> 
> Thanks,
> Paul
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/49c8c964/attachment.html>

From alex.gronholm at nextday.fi  Fri Oct 20 11:14:25 2017
From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=)
Date: Fri, 20 Oct 2017 18:14:25 +0300
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <BEF8CE18-C084-4556-A7A6-BFED5633654C@me.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <BEF8CE18-C084-4556-A7A6-BFED5633654C@me.com>
Message-ID: <bf64eb12-93ed-6139-317f-83b69ad6ff4f@nextday.fi>

Perhaps pkg_resources.find_distributions()?

http://setuptools.readthedocs.io/en/latest/pkg_resources.html#getting-or-creating-distributions


Jannis Gebauer kirjoitti 20.10.2017 klo 16:55:
> Thanks for the heads-up, Paul.
>
> I?m currently using `pip.get_installed_distributions` and as far as I 
> can see that has moved into `_internal`, too: 
> https://github.com/pypa/pip/blob/master/src/pip/_internal/utils/misc.py#L333
>
> Any recommendations?
>
> Thanks,
>
> Jannis
>
>> On 20. Oct 2017, at 15:22, Paul Moore <p.f.moore at gmail.com 
>> <mailto:p.f.moore at gmail.com>> wrote:
>>
>> We're in the process of starting to plan for a release of pip (the
>> long-awaited pip 10). We're likely still a month or two away from a
>> release, but now is the time for people to start ensuring that
>> everything works for them. One key change in the new version will be
>> that all of the internal APIs of pip will no longer be available, so
>> any code that currently calls functions in the "pip" namespace will
>> break. Calling pip's internal APIs has never been supported, and
>> always carried a risk of such breakage, so projects doing so should,
>> in theory, be prepared for such things. However, reality is not always
>> that simple, and we are aware that people will need time to deal with
>> the implications.
>>
>> Just in case it's not clear, simply finding where the internal APIs
>> have moved to and calling them under the new names is *not* what
>> people should do. We can't stop people calling the internal APIs,
>> obviously, but the idea of this change is to give people the incentive
>> to find a supported approach, not just to annoy people who are doing
>> things we don't want them to ;-)
>>
>> So please - if you're calling pip's internals in your code, take the
>> opportunity *now* to check out the in-development version of pip, and
>> ensure your project will still work when pip 10 is released.
>>
>> And many thanks to anyone else who helps by testing out the new
>> version, as well :-)
>>
>> Thanks,
>> Paul
>> _______________________________________________
>> Distutils-SIG maillist ?- Distutils-SIG at python.org 
>> <mailto:Distutils-SIG at python.org>
>> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/9175a27c/attachment-0001.html>

From wes.turner at gmail.com  Fri Oct 20 11:51:21 2017
From: wes.turner at gmail.com (Wes Turner)
Date: Fri, 20 Oct 2017 11:51:21 -0400
Subject: [Distutils] Documentation link on PyPI.org
In-Reply-To: <B31953AD-9784-4C49-ACB4-44DA3329FA3F@mac.com>
References: <DF28CD64-C6F6-488A-AF45-593EB4FC1B1B@mac.com>
 <CACfEFw8nOUqn63akZJcY9xCUX6d=Cv22FvQiqz04cV=dhLF9-g@mail.gmail.com>
 <7E1ABA81-9C79-4D02-8135-6AA409B1A540@mac.com>
 <CAP1=2W6L79ZuPQ=pCsPGpOaTLf6GdpMUMpDxbDBMF-P6jireUg@mail.gmail.com>
 <B31953AD-9784-4C49-ACB4-44DA3329FA3F@mac.com>
Message-ID: <CACfEFw9DxjC0oX2Xp9pHfcZrX4v6F6rrOH5cNsxmQtjWd-hEDQ@mail.gmail.com>

On Friday, October 20, 2017, Ronald Oussoren <ronaldoussoren at mac.com> wrote:

>
> On 28 Aug 2017, at 00:12, Brett Cannon <brett at python.org
> <javascript:_e(%7B%7D,'cvml','brett at python.org');>> wrote:
>
> If you search the archive of this mailing list you will notice I asked
> this exact question about a month or 2 ago (I think). The answer I got was
> it is used on PyPI.org, but I don't know how to set it with setuptools
> (flit will probably add support after PEP 517).
>
>
> To get back to this: I?m probably doing something wrong, but I haven?t
> managed yet to make anything show up on PyPI.org when I use ?Project-URL?
> in metadata.
>
> There?s a pyobjc-core 4.0.1b1 wheel on PyPI.org that includes two
> Project-URL lines in its metadata, but that doesn?t add anything to the
> sidebar. I?ve manually verified that the metadata is present in the wheel
> file (as I?m using a replacement egg_info command in my setup.py file to
> generate this part of the metadata).
>

https://github.com/pypa/warehouse/search?utf8=?&q="project-url"&type=Issues

https://github.com/pypa/warehouse/search?q="Project-Url"&type=Commits&utf8=?


>
> Ronald
>
> P.S. I tried to upload to test.pypi.org, but that didn?t work due to an
> SSL error:
>
> $ twine upload -r testpypi dist/pyobjc_core-4.0.1b1-cp27-
> cp27m-macosx_10_6_intel.whl
> Uploading distributions to https://test.pypi.org/legacy/
> Uploading pyobjc_core-4.0.1b1-cp27-cp27m-macosx_10_6_intel.whl
> SSLError: HTTPSConnectionPool(host='test.pypi.org', port=443): Max
> retries exceeded with url: /legacy/ (Caused by SSLError(SSLError(1, u'[SSL:
> TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661)'),))
>

https://www.google.com/search?q=TLSV1_ALERT_PROTOCOL_VERSION

Looks like TLSv1 (TLS 1.0) is deprecated.

https://www.google.com/search?q=TLSV1_ALERT_PROTOCOL_VERSION+twine

https://github.com/pypa/twine/issues/273




>
>
> On Sun, Aug 27, 2017, 13:01 Ronald Oussoren <ronaldoussoren at mac.com
> <javascript:_e(%7B%7D,'cvml','ronaldoussoren at mac.com');>> wrote:
>
>> On 27 Aug 2017, at 17:33, Wes Turner <wes.turner at gmail.com
>> <javascript:_e(%7B%7D,'cvml','wes.turner at gmail.com');>> wrote:
>>
>> You can add a URL in the long_description (e.g. to https://read-the-docs.
>> readthedocs.org/ )
>>
>>
>> I know, but that adds the URL to the project description and not the
>> sidebar. PEP 345 appears to specify metadata that could be used for this,
>> and warehouse seems to contain to make use of this (or at least a database
>> model that allows for specifying more links than just the default one).
>>
>> Adding links to the sidebar would be nicer with the new design on
>> PyPI.org <http://pypi.org/>, but there doesn?t seem to be a clean way to
>> add ?Project-URL? metadata as described in PEP 345 to a wheel file. I could
>> tweak my setup.py files to add Project-URL metadata, but before I do that
>> I?d like to confirm that (a) that metadata is actually used by warehouse
>> and (b) there is no cleaner way to add this metadata using setuptools.
>>
>> Ronald
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> <javascript:_e(%7B%7D,'cvml','Distutils-SIG at python.org');>
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/5b7dc1dc/attachment.html>

From ronaldoussoren at mac.com  Fri Oct 20 12:04:44 2017
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Fri, 20 Oct 2017 18:04:44 +0200
Subject: [Distutils] Documentation link on PyPI.org
In-Reply-To: <CACfEFw9DxjC0oX2Xp9pHfcZrX4v6F6rrOH5cNsxmQtjWd-hEDQ@mail.gmail.com>
References: <DF28CD64-C6F6-488A-AF45-593EB4FC1B1B@mac.com>
 <CACfEFw8nOUqn63akZJcY9xCUX6d=Cv22FvQiqz04cV=dhLF9-g@mail.gmail.com>
 <7E1ABA81-9C79-4D02-8135-6AA409B1A540@mac.com>
 <CAP1=2W6L79ZuPQ=pCsPGpOaTLf6GdpMUMpDxbDBMF-P6jireUg@mail.gmail.com>
 <B31953AD-9784-4C49-ACB4-44DA3329FA3F@mac.com>
 <CACfEFw9DxjC0oX2Xp9pHfcZrX4v6F6rrOH5cNsxmQtjWd-hEDQ@mail.gmail.com>
Message-ID: <B8439D3C-419F-474F-A64E-94845F58E291@mac.com>


> On 20 Oct 2017, at 17:51, Wes Turner <wes.turner at gmail.com> wrote:
> 
> 
> 
> On Friday, October 20, 2017, Ronald Oussoren <ronaldoussoren at mac.com <mailto:ronaldoussoren at mac.com>> wrote:
> 
>> On 28 Aug 2017, at 00:12, Brett Cannon <brett at python.org <javascript:_e(%7B%7D,'cvml','brett at python.org');>> wrote:
>> 
>> If you search the archive of this mailing list you will notice I asked this exact question about a month or 2 ago (I think). The answer I got was it is used on PyPI.org <http://pypi.org/>, but I don't know how to set it with setuptools (flit will probably add support after PEP 517).
> 
> To get back to this: I?m probably doing something wrong, but I haven?t managed yet to make anything show up on PyPI.org <http://pypi.org/> when I use ?Project-URL? in metadata. 
> 
> There?s a pyobjc-core 4.0.1b1 wheel on PyPI.org <http://pypi.org/> that includes two Project-URL lines in its metadata, but that doesn?t add anything to the sidebar. I?ve manually verified that the metadata is present in the wheel file (as I?m using a replacement egg_info command in my setup.py file to generate this part of the metadata).
> 
> https://github.com/pypa/warehouse/search?utf8=?&q= <https://github.com/pypa/warehouse/search?utf8=%E2%9C%93&q=>"project-url"&type=Issues
> 
> https://github.com/pypa/warehouse/search?q= <https://github.com/pypa/warehouse/search?q=>"Project-Url"&type=Commits&utf8=?

I?m note sure what you?re trying to say here, but <https://github.com/pypa/warehouse/pull/878/files <https://github.com/pypa/warehouse/pull/878/files>> suggests that Project-URL is supported in the PyPI.org <http://pypi.org/> UI (assuming that it is running a version of the software from after merging this pull request).  But for some reason this doesn?t work for me, and I don?t know why. Either I?m doing something wrong, or the code receiving wheel files does not yet store project-urls information in the database.

>  
> 
> Ronald
> 
> P.S. I tried to upload to test.pypi.org <http://test.pypi.org/>, but that didn?t work due to an SSL error:
> 
> $ twine upload -r testpypi dist/pyobjc_core-4.0.1b1-cp27-cp27m-macosx_10_6_intel.whl 
> Uploading distributions to https://test.pypi.org/legacy/ <https://test.pypi.org/legacy/>
> Uploading pyobjc_core-4.0.1b1-cp27-cp27m-macosx_10_6_intel.whl
> SSLError: HTTPSConnectionPool(host='test.pypi.org <http://test.pypi.org/>', port=443): Max retries exceeded with url: /legacy/ (Caused by SSLError(SSLError(1, u'[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661)'),))
> 
> https://www.google.com/search?q=TLSV1_ALERT_PROTOCOL_VERSION <https://www.google.com/search?q=TLSV1_ALERT_PROTOCOL_VERSION>
> 
> Looks like TLSv1 (TLS 1.0) is deprecated.
> 
> https://www.google.com/search?q=TLSV1_ALERT_PROTOCOL_VERSION+twine <https://www.google.com/search?q=TLSV1_ALERT_PROTOCOL_VERSION+twine>
> 
> https://github.com/pypa/twine/issues/273 <https://github.com/pypa/twine/issues/273>

Hmm? that issue means I should be slightly worried, the VM I?m using for uploading releases might stop working for this some time in the future. Hopefully upgrading the python installation on that will avoid that.

Ronald

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/0649755c/attachment.html>

From xoviat at gmail.com  Fri Oct 20 13:34:44 2017
From: xoviat at gmail.com (xoviat)
Date: Fri, 20 Oct 2017 12:34:44 -0500
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <CACac1F8bZD8CgF8-9JYvaRD+SdW4J+MTE66TvXrJN8MGBQpaEA@mail.gmail.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <CAH6Pt5o58J8b801UR-VWU2w_Ahmc2VpcOc3ey8dhPPnfFeGZGQ@mail.gmail.com>
 <CACac1F8bZD8CgF8-9JYvaRD+SdW4J+MTE66TvXrJN8MGBQpaEA@mail.gmail.com>
Message-ID: <CAKXb8mGdmDK8ydx8Jmx0Qm7yd_UQN9e+NP2DORUGxrMnZDv0vg@mail.gmail.com>

+1 on pre-release wheels. I've seen the process in action
with SciPy, and it helped to catch at least one significant bug.

2017-10-20 8:30 GMT-05:00 Paul Moore <p.f.moore at gmail.com>:

> On 20 October 2017 at 14:26, Matthew Brett <matthew.brett at gmail.com>
> wrote:
> > Thanks for the heads-up.
> >
> > Will y'all be doing a PyPI pre-release so we can test with `pip
> > install --pre -U pip`?
>
> We've not yet decided on that. Traditionally I don't think we have
> done so, but I'm inclined to think it's a good idea. It might not be
> until noticeably closer to the release, though...
>
> Paul
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/059c9658/attachment.html>

From alex.gronholm at nextday.fi  Fri Oct 20 13:36:39 2017
From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=)
Date: Fri, 20 Oct 2017 20:36:39 +0300
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <CAKXb8mGdmDK8ydx8Jmx0Qm7yd_UQN9e+NP2DORUGxrMnZDv0vg@mail.gmail.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <CAH6Pt5o58J8b801UR-VWU2w_Ahmc2VpcOc3ey8dhPPnfFeGZGQ@mail.gmail.com>
 <CACac1F8bZD8CgF8-9JYvaRD+SdW4J+MTE66TvXrJN8MGBQpaEA@mail.gmail.com>
 <CAKXb8mGdmDK8ydx8Jmx0Qm7yd_UQN9e+NP2DORUGxrMnZDv0vg@mail.gmail.com>
Message-ID: <68a08c34-29cd-3dbb-652d-9961604a43ee@nextday.fi>

Yeah, +1 from me too. Pip is one project where people will highly likely 
try out the pre-release versions.


xoviat kirjoitti 20.10.2017 klo 20:34:
> +1 on pre-release wheels. I've seen the process in action
> with SciPy, and it helped to catch at least one significant bug.
>
> 2017-10-20 8:30 GMT-05:00 Paul Moore <p.f.moore at gmail.com 
> <mailto:p.f.moore at gmail.com>>:
>
>     On 20 October 2017 at 14:26, Matthew Brett
>     <matthew.brett at gmail.com <mailto:matthew.brett at gmail.com>> wrote:
>     > Thanks for the heads-up.
>     >
>     > Will y'all be doing a PyPI pre-release so we can test with `pip
>     > install --pre -U pip`?
>
>     We've not yet decided on that. Traditionally I don't think we have
>     done so, but I'm inclined to think it's a good idea. It might not be
>     until noticeably closer to the release, though...
>
>     Paul
>     _______________________________________________
>     Distutils-SIG maillist? - Distutils-SIG at python.org
>     <mailto:Distutils-SIG at python.org>
>     https://mail.python.org/mailman/listinfo/distutils-sig
>     <https://mail.python.org/mailman/listinfo/distutils-sig>
>
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/b90731d0/attachment-0001.html>

From brett at python.org  Fri Oct 20 13:53:13 2017
From: brett at python.org (Brett Cannon)
Date: Fri, 20 Oct 2017 17:53:13 +0000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CADiSq7etRRq5h6jo1fOZ4ckO8W0jU3UdGLXMOaFzhLHX2ScCNQ@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <1508345133-sup-1461@lrrr.local>
 <CACac1F9GtauCppSi5geo_JiY2tpj0m+RXC9xX02FvqfOTCWbVg@mail.gmail.com>
 <CAG8k2+5XFwuUN7nW3zmmHhqtOgt809ULJPRZPf7xMzxasjAEXA@mail.gmail.com>
 <d049b6da-b4fa-acbe-f8ac-90ca34e50010@nextday.fi>
 <CADiSq7etRRq5h6jo1fOZ4ckO8W0jU3UdGLXMOaFzhLHX2ScCNQ@mail.gmail.com>
Message-ID: <CAP1=2W5RunG--6zn_gThDazqnoZO-5Pnbso_xK1tuiywvMzRDg@mail.gmail.com>

On Wed, 18 Oct 2017 at 17:54 Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 19 October 2017 at 04:18, Alex Gr?nholm <alex.gronholm at nextday.fi>
> wrote:
>
>> Daniel Holth kirjoitti 18.10.2017 klo 21:06:
>>
>>
>> http://setuptools.readthedocs.io/en/latest/formats.html?highlight=entry_points.txt#entry-points-txt-entry-point-plugin-metadata
>>
>>
>> http://setuptools.readthedocs.io/en/latest/pkg_resources.html?highlight=pkg_resources#creating-and-parsing
>>
>> It is not very complicated. It looks like the characters are mostly
>> 'python identifier' rules with a little bit of 'package name' rules.
>>
>> I am also concerned about the amount of parsing on startup. A hard
>> problem for certain, since no one likes outdated cache problems either. It
>> is also unpleasant to have too much code with a runtime dependency on
>> 'packaging'.
>>
>> Wasn't someone working on implementing pkg_resources in the standard
>> library at some point?
>>
>
> The idea has been raised, but we've been hesitant for the same reason
> we're inclined to take distutils out: packaging APIs need to be free to
> evolve in line with packaging interoperability standards, rather than with
> the Python language definition.
>
> Barry Warsaw & Brett Cannon recently mentioned something to me about
> working on a potential runtime alternative to pkg_resources that could be
> installed without also installing setuptools, but I don't know any of the
> specifics (and I'm not sure either of them follows distutils-sig).
>

I've been following distutils-sig for a couple of years now. :)

And what Barry and I are working on is only a subset of pkg_resources,
specifically the reading of data files included in a package. We aren't
touching any other aspect of pkg_resources.

Heck, until this discussion, "entry points" == "console scripts" for me so
I don't really know what y'all are talking about standardizing when it
comes to plug-in systems and metadata. Having said that, I do understand
why Donald doesn't want to just go ahead and standardize something by
giving it the level of a spec on packaging.python.org just because it's out
there. But since entry points seem to be used widely enough, having them
written down appropriately also seems reasonable.

As a compromise, could entry points be documented as Thomas is suggesting,
but have a note at the top saying something along the lines of "entry
points are considered a setuptools-specific feature, but their wide spread
use warrants a clear understanding of how they function for other packaging
tools choose on their own to also support them"? Basically acknowledge
there are ad-hoc, folk standards in the community that a decent chunk of
people rely on and thus docs would be helpful, but don't need to be
promoted to full-on, everyone-implements standard.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/074b6afc/attachment.html>

From noah at coderanger.net  Fri Oct 20 13:42:41 2017
From: noah at coderanger.net (Noah Kantrowitz)
Date: Fri, 20 Oct 2017 10:42:41 -0700
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
Message-ID: <FD3FC9A7-5C92-488A-80E6-238B201E5230@coderanger.net>

So as someone on the tooling side, is there any kind of install dry-run yet? I've got https://github.com/poise/poise-python/blob/master/lib/poise_python/resources/python_package.rb#L34-L78 which touches a toooon of internals. Basically I need a way to know exactly what versions `pip install` would have used in a given situation without actually changing the system. Happy for a better solution!

--Noah

> On Oct 20, 2017, at 6:22 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> We're in the process of starting to plan for a release of pip (the
> long-awaited pip 10). We're likely still a month or two away from a
> release, but now is the time for people to start ensuring that
> everything works for them. One key change in the new version will be
> that all of the internal APIs of pip will no longer be available, so
> any code that currently calls functions in the "pip" namespace will
> break. Calling pip's internal APIs has never been supported, and
> always carried a risk of such breakage, so projects doing so should,
> in theory, be prepared for such things. However, reality is not always
> that simple, and we are aware that people will need time to deal with
> the implications.
> 
> Just in case it's not clear, simply finding where the internal APIs
> have moved to and calling them under the new names is *not* what
> people should do. We can't stop people calling the internal APIs,
> obviously, but the idea of this change is to give people the incentive
> to find a supported approach, not just to annoy people who are doing
> things we don't want them to ;-)
> 
> So please - if you're calling pip's internals in your code, take the
> opportunity *now* to check out the in-development version of pip, and
> ensure your project will still work when pip 10 is released.
> 
> And many thanks to anyone else who helps by testing out the new
> version, as well :-)
> 
> Thanks,
> Paul
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


From doug at doughellmann.com  Fri Oct 20 14:16:48 2017
From: doug at doughellmann.com (Doug Hellmann)
Date: Fri, 20 Oct 2017 14:16:48 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CACfEFw8UUtXf-vS0xd9+vT042GJvJ7fOOFw7QANpwUGeFV0Rrg@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>
 <5F9E5F93-7AF4-4D05-956F-A4367F89CD96@stufft.io>
 <1508500651.117367.1145301232.4A2422BF@webmail.messagingengine.com>
 <6BBFF1A1-F05D-496A-8BEB-FB65D7554823@stufft.io>
 <CADiSq7dmmoB_Hj_Q7ZurMJ1f5e1g=K4oAQQnx5xFucccxUyWhQ@mail.gmail.com>
 <5BD2E6C3-2D0C-45FC-A15D-66ACDBC6B33D@stufft.io>
 <1508503275.124767.1145341384.76E2EDA6@webmail.messagingengine.com>
 <F0F05ABC-788F-49AF-9905-E98F7FDDB988@stufft.io>
 <CADiSq7dE8ZxPZRj0RChEc_65kMpQNUzos_gN-hd6FBu_cLPQ_Q@mail.gmail.com>
 <A2AB937D-D8E0-44E2-8471-3E2D1F433929@stufft.io>
 <CACfEFw8UUtXf-vS0xd9+vT042GJvJ7fOOFw7QANpwUGeFV0Rrg@mail.gmail.com>
Message-ID: <1508523033-sup-6338@lrrr.local>

Excerpts from Wes Turner's message of 2017-10-20 10:41:02 -0400:
> On Friday, October 20, 2017, Donald Stufft <donald at stufft.io> wrote:
> 
> >
> >
> > On Oct 20, 2017, at 9:35 AM, Nick Coghlan <ncoghlan at gmail.com
> > <javascript:_e(%7B%7D,'cvml','ncoghlan at gmail.com');>> wrote:
> >
> > On 20 October 2017 at 23:19, Donald Stufft <donald at stufft.io
> > <javascript:_e(%7B%7D,'cvml','donald at stufft.io');>> wrote:
> >
> >> One that I was helping someone debug just the other day is that they?re
> >> super non-debuggable and the behavior when you have two things providing
> >> the same entry point is basically ???? (If I remember correctly, the
> >> behavior is that the first thing found is the one that ?wins?, which means
> >> the ordering of sys.path and the names of the projects supply it is what
> >> determines it). This got exposed to the end user that they installed
> >> something that they thought was going to add support for something, but
> >> which silently did nothing because two different project happened to pick
> >> the same name for their entry point (not the group, it was two things
> >> providing plugins for the same system).
> >>
> >
> > While I agree with this, I think that's a combination of pkg_resources
> > itself being hard to debug in general, and the fact that pkg_resources
> > doesn't clearly define the semantics of how it resolves name conflicts
> > within an entry point group - as far as I know, it's largely an accident of
> > implementation.
> >
> > The interoperability spec is going to state that conflict resolution when
> > the same name within a group is declared by multiple packages is the
> > responsibility of the group consumer, so documenting the format should
> > actually improve this situation, since it allows for the development of
> > competing conflict resolution strategies in different runtime libraries.
> >
> >
> > I think it makes it *worse*, because now the behavior isn?t just a
> > entrypoints weirdness, but now it changes based on which runtime library
> > you use (which isn?t something that end users are likely to have much
> > insight into) and it represents a footgun that package authors are unlikely
> > to be aware of. If mycoolentrypointslib comes out that is faster, but
> > changes some subtle behavior like this it?ll break people, but that is
> > unlikely going to be an effect that people expect to happen just because
> > they switched between two things both implementing the same standard.
> >
> > So effectively this means that not only is the fact you?re using
> > entrypoints part of your API, but now which entry point library you?re
> > using at runtime is now also part of your API.
> >
> 
> When should the check for duplicate entry points occur?
> 
> - At on_install() time (+1)
> - At runtime
> 
> Is a sys.path-like OrderedDict preemptive strategy preferable or just as
> dangerous as importlib?

Having "duplicate" entry points is not necessarily an error. It's
a different usage pattern.  The semantics of dropping a named plugin
into a namespace are defined by the application and plugin-point.
Please do not build assumptions about uniqueness into the underlying
implementation.

The stevedore library wraps up pkg_resources with several such
patterns. For example, it supports "give me all of the plugins in
a namespace" (find all the extensions to your app), "give me all
of the plugins named $name in a namespace" (find the hooks for a
specific event defined by the app), and "give me *the* plugin named
$name in a namespace" (load a driver for talking to a backend).

https://docs.openstack.org/stevedore/latest/reference/index.html

Doug

From doug at doughellmann.com  Fri Oct 20 14:24:29 2017
From: doug at doughellmann.com (Doug Hellmann)
Date: Fri, 20 Oct 2017 14:24:29 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CADiSq7daBpo8TK2cRW=ou7puQsMgWHObBEFBv48C1KOWnEmUhw@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <CADiSq7daBpo8TK2cRW=ou7puQsMgWHObBEFBv48C1KOWnEmUhw@mail.gmail.com>
Message-ID: <1508523732-sup-6024@lrrr.local>

Excerpts from Nick Coghlan's message of 2017-10-20 14:42:09 +1000:
> On 20 October 2017 at 02:14, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> 
> > On Thu, Oct 19, 2017, at 04:10 PM, Donald Stufft wrote:
> > > I?m in favor, although one question I guess is whether it should be a a
> > > PEP or an ad hoc spec. Given (2) it should *probably* be a a PEP (since
> > > without (2), its just another file in the .dist-info directory and that
> > > doesn?t actually need standardized at all). I don?t think that this will
> > > be a very controversial PEP though, and should be pretty easy.
> >
> > I have opened a PR to document what is already there, without adding any
> > new features. I think this is worth doing even if we don't change
> > anything, since it's a de-facto standard used for different tools to
> > interact.
> >
> > https://github.com/pypa/python-packaging-user-guide/pull/390
> >
> > We can still write a PEP for caching if necessary.
> >
> 
> +1 for that approach (PR for the status quo, PEP for a shared metadata
> caching design) from me
> 
> Making the status quo more discoverable is valuable in its own right, and
> the only decisions we'll need to make for that are terminology
> clarification ones, not interoperability ones (this isn't like PEP 440 or
> 508 where we actually thought some of the default setuptools behaviour was
> slightly incorrect and wanted to change it).
> 
> Figuring out a robust cross-platform network-file-system-tolerant metadata
> caching design on the other hand is going to be hard, and as Donald
> suggests, the right ecosystem level solution might be to define
> install-time hooks for package installation operations.
> 
> > > I?m also in favor of this. Although I would suggest SQLite rather than a
> > > JSON file for the primary reason being that a JSON file isn?t
> > > multiprocess safe without being careful (and possibly introducing
> > > locking) whereas SQLite has already solved that problem.
> >
> > SQLite was actually my first thought, but from experience in Jupyter &
> > IPython I'm wary of it - its built-in locking does not work well over
> > NFS, and it's easy to corrupt the database. I think careful use of
> > atomic writing can be more reliable (though that has given us some
> > problems too).
> >
> > That may be easier if there's one cache per user, though - we can
> > perhaps try to store it somewhere that's not NFS.
> >
> 
> I'm wondering if rather than jumping straight to a PEP, it may make sense
> to instead initially pursue this idea as a *non-*standard, implementation
> dependent thing specific to the "entrypoints" project. There are a *lot* of
> challenges to be taken into account for a truly universal metadata caching
> design, and it would be easy to fall into the trap of coming up with a
> design so complex that nobody can realistically implement it.
> 
> Specifically, I'm thinking of a usage model along the lines of the
> updatedb/locate pair on *nix systems: `locate` gives you access to very
> fast searches of your filesystem, but it *doesn't* try to automagically
> keeps its indexes up to date. Instead, refreshing the indexes is handled by
> `updatedb`, and you can either rely on that being run automatically in a
> cron job, or else force an update with `sudo updatedb` when you want to use
> `locate`.
> 
> For a project like entrypoints, what that might look like is that at
> *runtime*, you may implement a reasonably fast "cache freshness check",
> where you scanned the mtime of all the sys.path entries, and compared those
> to the mtime of the cache. If the cache looks up to date, then cool,
> otherwise emit a warning about the stale metadata cache, and then bypass it.
> 
> The entrypoints project itself could then expose a
> `refresh-entrypoints-cache` command that could start out only supporting
> virtual environments, and then extend to per-user caching, and then finally
> (maybe) consider whether or not it wanted to support installation-wide
> caches (with the extra permissions management and cross-process and
> cross-system coordination that may imply).
> 
> Such an approach would also tie in nicely with Donald's suggestion of
> reframing the ecosystem level question as "How should the entrypoints
> project request that 'refresh-entrypoints-cache' be run after every package
> installation or removal operation?", which in turn would integrate nicely
> with things like RPM file triggers (where the system `pip` package could
> set a file trigger that arranged for any properly registered Python package
> installation plugins to be run for every modification to site-packages
> while still appropriately managing the risk of running arbitrary code with
> elevated privileges)
> 
> Cheers,
> Nick.
> 

I have been trying to find time to do something like that within
stevedore for a while to solve some client-side startup performance
issues with the OpenStack client. I would be happy to help add it
to entrypoints instead and use it from there.

Thomas, please me know how I can help.

Doug

From donald at stufft.io  Fri Oct 20 14:34:30 2017
From: donald at stufft.io (Donald Stufft)
Date: Fri, 20 Oct 2017 14:34:30 -0400
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <CACac1F8bZD8CgF8-9JYvaRD+SdW4J+MTE66TvXrJN8MGBQpaEA@mail.gmail.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <CAH6Pt5o58J8b801UR-VWU2w_Ahmc2VpcOc3ey8dhPPnfFeGZGQ@mail.gmail.com>
 <CACac1F8bZD8CgF8-9JYvaRD+SdW4J+MTE66TvXrJN8MGBQpaEA@mail.gmail.com>
Message-ID: <FA3BCDCB-E7D0-4A06-8E08-5419BA9C0B2C@stufft.io>


> On Oct 20, 2017, at 9:30 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 20 October 2017 at 14:26, Matthew Brett <matthew.brett at gmail.com> wrote:
>> Thanks for the heads-up.
>> 
>> Will y'all be doing a PyPI pre-release so we can test with `pip
>> install --pre -U pip`?
> 
> We've not yet decided on that. Traditionally I don't think we have
> done so, but I'm inclined to think it's a good idea. It might not be
> until noticeably closer to the release, though?
> 

I used to cut pre-releases for pip, and after awhile I gave up on doing them because it felt like nobody ever actually reported any issues with them anyways, and it wasn?t until we cut the final release that we started finding bugs with them. I don?t have any problem with us starting to issue them again though and seeing if we start catching issues earlier this time.



From xoviat at gmail.com  Fri Oct 20 14:35:46 2017
From: xoviat at gmail.com (xoviat)
Date: Fri, 20 Oct 2017 13:35:46 -0500
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <FD3FC9A7-5C92-488A-80E6-238B201E5230@coderanger.net>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <FD3FC9A7-5C92-488A-80E6-238B201E5230@coderanger.net>
Message-ID: <CAKXb8mG+XRrxacgmcLrOyG=8794te5SsvJKWMjdxOq4WphWTYg@mail.gmail.com>

There's no dry-run functionality that I know of so far. However, you could
use the following:

pip install --prefix=tmpdir

This command is actually about the same speed as a proper implementation,
because we can't actually know what we're installing until we build the
requirements.

2017-10-20 12:42 GMT-05:00 Noah Kantrowitz <noah at coderanger.net>:

> So as someone on the tooling side, is there any kind of install dry-run
> yet? I've got https://github.com/poise/poise-python/blob/master/lib/
> poise_python/resources/python_package.rb#L34-L78 which touches a toooon
> of internals. Basically I need a way to know exactly what versions `pip
> install` would have used in a given situation without actually changing the
> system. Happy for a better solution!
>
> --Noah
>
> > On Oct 20, 2017, at 6:22 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> >
> > We're in the process of starting to plan for a release of pip (the
> > long-awaited pip 10). We're likely still a month or two away from a
> > release, but now is the time for people to start ensuring that
> > everything works for them. One key change in the new version will be
> > that all of the internal APIs of pip will no longer be available, so
> > any code that currently calls functions in the "pip" namespace will
> > break. Calling pip's internal APIs has never been supported, and
> > always carried a risk of such breakage, so projects doing so should,
> > in theory, be prepared for such things. However, reality is not always
> > that simple, and we are aware that people will need time to deal with
> > the implications.
> >
> > Just in case it's not clear, simply finding where the internal APIs
> > have moved to and calling them under the new names is *not* what
> > people should do. We can't stop people calling the internal APIs,
> > obviously, but the idea of this change is to give people the incentive
> > to find a supported approach, not just to annoy people who are doing
> > things we don't want them to ;-)
> >
> > So please - if you're calling pip's internals in your code, take the
> > opportunity *now* to check out the in-development version of pip, and
> > ensure your project will still work when pip 10 is released.
> >
> > And many thanks to anyone else who helps by testing out the new
> > version, as well :-)
> >
> > Thanks,
> > Paul
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/8afa03d9/attachment.html>

From thomas at kluyver.me.uk  Fri Oct 20 14:37:45 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Fri, 20 Oct 2017 19:37:45 +0100
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1508523732-sup-6024@lrrr.local>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <CADiSq7daBpo8TK2cRW=ou7puQsMgWHObBEFBv48C1KOWnEmUhw@mail.gmail.com>
 <1508523732-sup-6024@lrrr.local>
Message-ID: <1508524665.2432978.1145710912.161590AD@webmail.messagingengine.com>

On Fri, Oct 20, 2017, at 07:24 PM, Doug Hellmann wrote:
> I have been trying to find time to do something like that within
> stevedore for a while to solve some client-side startup performance
> issues with the OpenStack client. I would be happy to help add it
> to entrypoints instead and use it from there.
> 
> Thomas, please me know how I can help.

Thanks Doug! For starters, I'd be interested to hear any plans you have
for how to tackle caching, or any thoughts you have on the rough plan I
described before. If you're happy with the concepts, I'll have a go at
implementing it. I'll probably consider it experimental until there's a
hooks mechanism to trigger rebuilding the cache when packages are
installed or uninstalled.

Thomas

From marius at gedmin.as  Fri Oct 20 14:31:20 2017
From: marius at gedmin.as (Marius Gedminas)
Date: Fri, 20 Oct 2017 21:31:20 +0300
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <6BBFF1A1-F05D-496A-8BEB-FB65D7554823@stufft.io>
References: <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>
 <5F9E5F93-7AF4-4D05-956F-A4367F89CD96@stufft.io>
 <1508500651.117367.1145301232.4A2422BF@webmail.messagingengine.com>
 <6BBFF1A1-F05D-496A-8BEB-FB65D7554823@stufft.io>
Message-ID: <20171020183120.tsclaohbddpsmuz6@platonas>

On Fri, Oct 20, 2017 at 08:10:06AM -0400, Donald Stufft wrote:
> Packaging tools shouldn?t be expected to know anything about it other
> than the console_scripts feature

Please do not forget about gui_scripts entry points!

Marius Gedminas
-- 
What can I do with Python that I can't do with C#?  You can go home on time at
the end of the day.
        -- Daniel Klein
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/e4720924/attachment.sig>

From thomas at kluyver.me.uk  Fri Oct 20 14:42:27 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Fri, 20 Oct 2017 19:42:27 +0100
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <20171020183120.tsclaohbddpsmuz6@platonas>
References: <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>
 <5F9E5F93-7AF4-4D05-956F-A4367F89CD96@stufft.io>
 <1508500651.117367.1145301232.4A2422BF@webmail.messagingengine.com>
 <6BBFF1A1-F05D-496A-8BEB-FB65D7554823@stufft.io>
 <20171020183120.tsclaohbddpsmuz6@platonas>
Message-ID: <1508524947.2434008.1145719048.584C4252@webmail.messagingengine.com>

On Fri, Oct 20, 2017, at 07:31 PM, Marius Gedminas wrote:
> Please do not forget about gui_scripts entry points!

I haven't forgotten about them in the draft spec:
https://github.com/pypa/python-packaging-user-guide/pull/390/files#diff-089b079de062f6fdb759bb719b79e6c8R121

From xoviat at gmail.com  Fri Oct 20 14:43:59 2017
From: xoviat at gmail.com (xoviat)
Date: Fri, 20 Oct 2017 13:43:59 -0500
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <36779048-39F8-41E7-B1D2-A01FBFCFA284@coderanger.net>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <FD3FC9A7-5C92-488A-80E6-238B201E5230@coderanger.net>
 <CAKXb8mG+XRrxacgmcLrOyG=8794te5SsvJKWMjdxOq4WphWTYg@mail.gmail.com>
 <36779048-39F8-41E7-B1D2-A01FBFCFA284@coderanger.net>
Message-ID: <CAKXb8mG0dbyFYmxHtofuCrfijMBgdhXE9VFG9rR1N_FYiz_Nig@mail.gmail.com>

A correct dry-run implementation will do about the same amount of work as
installing to a temporary directory right now. In the future, that could be
optimized, but any patch to the finder doesn't actually detect the
requirements correctly (as they're not necessarily known until after the
wheels are built).

2017-10-20 13:41 GMT-05:00 Noah Kantrowitz <noah at coderanger.net>:

> Installing to a temp dir is really not an option for automated tooling (if
> nothing else, it takes way too long). `pip list --outdated` does already
> get fairly close to this (and doesn't install anything I suspect you can
> actually get a lot closer than you think) but it calculates for all
> packages (read: is slow) and doesn't give a good way to restrict things
> (hence that hack-y script which is a modified version of the pip list
> code). This is 100% a hard requirement for config management systems and if
> not fixed in pip, will require continued use of internal APIs. I would
> recommend just making pip list take a set of install-compatible
> names/version patterns and apply that as a filter in a similar way to what
> I've done there.
>
> --Noah
>
> > On Oct 20, 2017, at 11:35 AM, xoviat <xoviat at gmail.com> wrote:
> >
> > There's no dry-run functionality that I know of so far. However, you
> could use the following:
> >
> > pip install --prefix=tmpdir
> >
> > This command is actually about the same speed as a proper
> implementation, because we can't actually know what we're installing until
> we build the requirements.
> >
> > 2017-10-20 12:42 GMT-05:00 Noah Kantrowitz <noah at coderanger.net>:
> > So as someone on the tooling side, is there any kind of install dry-run
> yet? I've got https://github.com/poise/poise-python/blob/master/lib/
> poise_python/resources/python_package.rb#L34-L78 which touches a toooon
> of internals. Basically I need a way to know exactly what versions `pip
> install` would have used in a given situation without actually changing the
> system. Happy for a better solution!
> >
> > --Noah
> >
> > > On Oct 20, 2017, at 6:22 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> > >
> > > We're in the process of starting to plan for a release of pip (the
> > > long-awaited pip 10). We're likely still a month or two away from a
> > > release, but now is the time for people to start ensuring that
> > > everything works for them. One key change in the new version will be
> > > that all of the internal APIs of pip will no longer be available, so
> > > any code that currently calls functions in the "pip" namespace will
> > > break. Calling pip's internal APIs has never been supported, and
> > > always carried a risk of such breakage, so projects doing so should,
> > > in theory, be prepared for such things. However, reality is not always
> > > that simple, and we are aware that people will need time to deal with
> > > the implications.
> > >
> > > Just in case it's not clear, simply finding where the internal APIs
> > > have moved to and calling them under the new names is *not* what
> > > people should do. We can't stop people calling the internal APIs,
> > > obviously, but the idea of this change is to give people the incentive
> > > to find a supported approach, not just to annoy people who are doing
> > > things we don't want them to ;-)
> > >
> > > So please - if you're calling pip's internals in your code, take the
> > > opportunity *now* to check out the in-development version of pip, and
> > > ensure your project will still work when pip 10 is released.
> > >
> > > And many thanks to anyone else who helps by testing out the new
> > > version, as well :-)
> > >
> > > Thanks,
> > > Paul
> > > _______________________________________________
> > > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > > https://mail.python.org/mailman/listinfo/distutils-sig
> >
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/b5f9a7bc/attachment.html>

From noah at coderanger.net  Fri Oct 20 14:41:44 2017
From: noah at coderanger.net (Noah Kantrowitz)
Date: Fri, 20 Oct 2017 11:41:44 -0700
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <CAKXb8mG+XRrxacgmcLrOyG=8794te5SsvJKWMjdxOq4WphWTYg@mail.gmail.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <FD3FC9A7-5C92-488A-80E6-238B201E5230@coderanger.net>
 <CAKXb8mG+XRrxacgmcLrOyG=8794te5SsvJKWMjdxOq4WphWTYg@mail.gmail.com>
Message-ID: <36779048-39F8-41E7-B1D2-A01FBFCFA284@coderanger.net>

Installing to a temp dir is really not an option for automated tooling (if nothing else, it takes way too long). `pip list --outdated` does already get fairly close to this (and doesn't install anything I suspect you can actually get a lot closer than you think) but it calculates for all packages (read: is slow) and doesn't give a good way to restrict things (hence that hack-y script which is a modified version of the pip list code). This is 100% a hard requirement for config management systems and if not fixed in pip, will require continued use of internal APIs. I would recommend just making pip list take a set of install-compatible names/version patterns and apply that as a filter in a similar way to what I've done there.

--Noah

> On Oct 20, 2017, at 11:35 AM, xoviat <xoviat at gmail.com> wrote:
> 
> There's no dry-run functionality that I know of so far. However, you could use the following:
> 
> pip install --prefix=tmpdir
> 
> This command is actually about the same speed as a proper implementation, because we can't actually know what we're installing until we build the requirements.
> 
> 2017-10-20 12:42 GMT-05:00 Noah Kantrowitz <noah at coderanger.net>:
> So as someone on the tooling side, is there any kind of install dry-run yet? I've got https://github.com/poise/poise-python/blob/master/lib/poise_python/resources/python_package.rb#L34-L78 which touches a toooon of internals. Basically I need a way to know exactly what versions `pip install` would have used in a given situation without actually changing the system. Happy for a better solution!
> 
> --Noah
> 
> > On Oct 20, 2017, at 6:22 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> >
> > We're in the process of starting to plan for a release of pip (the
> > long-awaited pip 10). We're likely still a month or two away from a
> > release, but now is the time for people to start ensuring that
> > everything works for them. One key change in the new version will be
> > that all of the internal APIs of pip will no longer be available, so
> > any code that currently calls functions in the "pip" namespace will
> > break. Calling pip's internal APIs has never been supported, and
> > always carried a risk of such breakage, so projects doing so should,
> > in theory, be prepared for such things. However, reality is not always
> > that simple, and we are aware that people will need time to deal with
> > the implications.
> >
> > Just in case it's not clear, simply finding where the internal APIs
> > have moved to and calling them under the new names is *not* what
> > people should do. We can't stop people calling the internal APIs,
> > obviously, but the idea of this change is to give people the incentive
> > to find a supported approach, not just to annoy people who are doing
> > things we don't want them to ;-)
> >
> > So please - if you're calling pip's internals in your code, take the
> > opportunity *now* to check out the in-development version of pip, and
> > ensure your project will still work when pip 10 is released.
> >
> > And many thanks to anyone else who helps by testing out the new
> > version, as well :-)
> >
> > Thanks,
> > Paul
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
> 


From xoviat at gmail.com  Fri Oct 20 15:05:10 2017
From: xoviat at gmail.com (xoviat)
Date: Fri, 20 Oct 2017 14:05:10 -0500
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <BA22AD5F-A411-44ED-9544-267DE5656F9D@coderanger.net>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <FD3FC9A7-5C92-488A-80E6-238B201E5230@coderanger.net>
 <CAKXb8mG+XRrxacgmcLrOyG=8794te5SsvJKWMjdxOq4WphWTYg@mail.gmail.com>
 <36779048-39F8-41E7-B1D2-A01FBFCFA284@coderanger.net>
 <CAKXb8mG0dbyFYmxHtofuCrfijMBgdhXE9VFG9rR1N_FYiz_Nig@mail.gmail.com>
 <BA22AD5F-A411-44ED-9544-267DE5656F9D@coderanger.net>
Message-ID: <CAKXb8mFJhPkQkAnPZdayPYAP+=j-0GfYmmd9J-iUbmJF3XeJPA@mail.gmail.com>

The "installing collected packages" output is not always correct. The
"successfully installed [package list]" is always correct
(see pypa/pip/issues/4724 for an example).

I understand your requirements, but is the practical implication that pip
should offer a command that outputs packages that are potentially
incorrect? What happens when someone relies on that and then files a bug
report?

2017-10-20 13:48 GMT-05:00 Noah Kantrowitz <noah at coderanger.net>:

> While I understand that pip itself has to be very careful about edge cases
> and all the pathological things you can do in setup.py, as a higher-level
> tooling author my priorities are on the happy path UX and speed is a big
> factor there. So yes, using PackageFinder is potentially inaccurate, but
> it's also _usually_ accurate :) Anyways, if there is true concern that
> finder-based approaches are too risky, probably don't offer it in the pip
> list output.
>
> --Noah
>
> > On Oct 20, 2017, at 11:43 AM, xoviat <xoviat at gmail.com> wrote:
> >
> > A correct dry-run implementation will do about the same amount of work
> as installing to a temporary directory right now. In the future, that could
> be optimized, but any patch to the finder doesn't actually detect the
> requirements correctly (as they're not necessarily known until after the
> wheels are built).
> >
> > 2017-10-20 13:41 GMT-05:00 Noah Kantrowitz <noah at coderanger.net>:
> > Installing to a temp dir is really not an option for automated tooling
> (if nothing else, it takes way too long). `pip list --outdated` does
> already get fairly close to this (and doesn't install anything I suspect
> you can actually get a lot closer than you think) but it calculates for all
> packages (read: is slow) and doesn't give a good way to restrict things
> (hence that hack-y script which is a modified version of the pip list
> code). This is 100% a hard requirement for config management systems and if
> not fixed in pip, will require continued use of internal APIs. I would
> recommend just making pip list take a set of install-compatible
> names/version patterns and apply that as a filter in a similar way to what
> I've done there.
> >
> > --Noah
> >
> > > On Oct 20, 2017, at 11:35 AM, xoviat <xoviat at gmail.com> wrote:
> > >
> > > There's no dry-run functionality that I know of so far. However, you
> could use the following:
> > >
> > > pip install --prefix=tmpdir
> > >
> > > This command is actually about the same speed as a proper
> implementation, because we can't actually know what we're installing until
> we build the requirements.
> > >
> > > 2017-10-20 12:42 GMT-05:00 Noah Kantrowitz <noah at coderanger.net>:
> > > So as someone on the tooling side, is there any kind of install
> dry-run yet? I've got https://github.com/poise/
> poise-python/blob/master/lib/poise_python/resources/python_
> package.rb#L34-L78 which touches a toooon of internals. Basically I need
> a way to know exactly what versions `pip install` would have used in a
> given situation without actually changing the system. Happy for a better
> solution!
> > >
> > > --Noah
> > >
> > > > On Oct 20, 2017, at 6:22 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> > > >
> > > > We're in the process of starting to plan for a release of pip (the
> > > > long-awaited pip 10). We're likely still a month or two away from a
> > > > release, but now is the time for people to start ensuring that
> > > > everything works for them. One key change in the new version will be
> > > > that all of the internal APIs of pip will no longer be available, so
> > > > any code that currently calls functions in the "pip" namespace will
> > > > break. Calling pip's internal APIs has never been supported, and
> > > > always carried a risk of such breakage, so projects doing so should,
> > > > in theory, be prepared for such things. However, reality is not
> always
> > > > that simple, and we are aware that people will need time to deal with
> > > > the implications.
> > > >
> > > > Just in case it's not clear, simply finding where the internal APIs
> > > > have moved to and calling them under the new names is *not* what
> > > > people should do. We can't stop people calling the internal APIs,
> > > > obviously, but the idea of this change is to give people the
> incentive
> > > > to find a supported approach, not just to annoy people who are doing
> > > > things we don't want them to ;-)
> > > >
> > > > So please - if you're calling pip's internals in your code, take the
> > > > opportunity *now* to check out the in-development version of pip, and
> > > > ensure your project will still work when pip 10 is released.
> > > >
> > > > And many thanks to anyone else who helps by testing out the new
> > > > version, as well :-)
> > > >
> > > > Thanks,
> > > > Paul
> > > > _______________________________________________
> > > > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > > > https://mail.python.org/mailman/listinfo/distutils-sig
> > >
> > > _______________________________________________
> > > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > > https://mail.python.org/mailman/listinfo/distutils-sig
> > >
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/d77bca29/attachment.html>

From noah at coderanger.net  Fri Oct 20 14:48:59 2017
From: noah at coderanger.net (Noah Kantrowitz)
Date: Fri, 20 Oct 2017 11:48:59 -0700
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <CAKXb8mG0dbyFYmxHtofuCrfijMBgdhXE9VFG9rR1N_FYiz_Nig@mail.gmail.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <FD3FC9A7-5C92-488A-80E6-238B201E5230@coderanger.net>
 <CAKXb8mG+XRrxacgmcLrOyG=8794te5SsvJKWMjdxOq4WphWTYg@mail.gmail.com>
 <36779048-39F8-41E7-B1D2-A01FBFCFA284@coderanger.net>
 <CAKXb8mG0dbyFYmxHtofuCrfijMBgdhXE9VFG9rR1N_FYiz_Nig@mail.gmail.com>
Message-ID: <BA22AD5F-A411-44ED-9544-267DE5656F9D@coderanger.net>

While I understand that pip itself has to be very careful about edge cases and all the pathological things you can do in setup.py, as a higher-level tooling author my priorities are on the happy path UX and speed is a big factor there. So yes, using PackageFinder is potentially inaccurate, but it's also _usually_ accurate :) Anyways, if there is true concern that finder-based approaches are too risky, probably don't offer it in the pip list output.

--Noah

> On Oct 20, 2017, at 11:43 AM, xoviat <xoviat at gmail.com> wrote:
> 
> A correct dry-run implementation will do about the same amount of work as installing to a temporary directory right now. In the future, that could be optimized, but any patch to the finder doesn't actually detect the requirements correctly (as they're not necessarily known until after the wheels are built).
> 
> 2017-10-20 13:41 GMT-05:00 Noah Kantrowitz <noah at coderanger.net>:
> Installing to a temp dir is really not an option for automated tooling (if nothing else, it takes way too long). `pip list --outdated` does already get fairly close to this (and doesn't install anything I suspect you can actually get a lot closer than you think) but it calculates for all packages (read: is slow) and doesn't give a good way to restrict things (hence that hack-y script which is a modified version of the pip list code). This is 100% a hard requirement for config management systems and if not fixed in pip, will require continued use of internal APIs. I would recommend just making pip list take a set of install-compatible names/version patterns and apply that as a filter in a similar way to what I've done there.
> 
> --Noah
> 
> > On Oct 20, 2017, at 11:35 AM, xoviat <xoviat at gmail.com> wrote:
> >
> > There's no dry-run functionality that I know of so far. However, you could use the following:
> >
> > pip install --prefix=tmpdir
> >
> > This command is actually about the same speed as a proper implementation, because we can't actually know what we're installing until we build the requirements.
> >
> > 2017-10-20 12:42 GMT-05:00 Noah Kantrowitz <noah at coderanger.net>:
> > So as someone on the tooling side, is there any kind of install dry-run yet? I've got https://github.com/poise/poise-python/blob/master/lib/poise_python/resources/python_package.rb#L34-L78 which touches a toooon of internals. Basically I need a way to know exactly what versions `pip install` would have used in a given situation without actually changing the system. Happy for a better solution!
> >
> > --Noah
> >
> > > On Oct 20, 2017, at 6:22 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> > >
> > > We're in the process of starting to plan for a release of pip (the
> > > long-awaited pip 10). We're likely still a month or two away from a
> > > release, but now is the time for people to start ensuring that
> > > everything works for them. One key change in the new version will be
> > > that all of the internal APIs of pip will no longer be available, so
> > > any code that currently calls functions in the "pip" namespace will
> > > break. Calling pip's internal APIs has never been supported, and
> > > always carried a risk of such breakage, so projects doing so should,
> > > in theory, be prepared for such things. However, reality is not always
> > > that simple, and we are aware that people will need time to deal with
> > > the implications.
> > >
> > > Just in case it's not clear, simply finding where the internal APIs
> > > have moved to and calling them under the new names is *not* what
> > > people should do. We can't stop people calling the internal APIs,
> > > obviously, but the idea of this change is to give people the incentive
> > > to find a supported approach, not just to annoy people who are doing
> > > things we don't want them to ;-)
> > >
> > > So please - if you're calling pip's internals in your code, take the
> > > opportunity *now* to check out the in-development version of pip, and
> > > ensure your project will still work when pip 10 is released.
> > >
> > > And many thanks to anyone else who helps by testing out the new
> > > version, as well :-)
> > >
> > > Thanks,
> > > Paul
> > > _______________________________________________
> > > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > > https://mail.python.org/mailman/listinfo/distutils-sig
> >
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> >
> 
> 


From doug at doughellmann.com  Fri Oct 20 15:26:46 2017
From: doug at doughellmann.com (Doug Hellmann)
Date: Fri, 20 Oct 2017 15:26:46 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1508524665.2432978.1145710912.161590AD@webmail.messagingengine.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <CADiSq7daBpo8TK2cRW=ou7puQsMgWHObBEFBv48C1KOWnEmUhw@mail.gmail.com>
 <1508523732-sup-6024@lrrr.local>
 <1508524665.2432978.1145710912.161590AD@webmail.messagingengine.com>
Message-ID: <1508525235-sup-7030@lrrr.local>

Excerpts from Thomas Kluyver's message of 2017-10-20 19:37:45 +0100:
> On Fri, Oct 20, 2017, at 07:24 PM, Doug Hellmann wrote:
> > I have been trying to find time to do something like that within
> > stevedore for a while to solve some client-side startup performance
> > issues with the OpenStack client. I would be happy to help add it
> > to entrypoints instead and use it from there.
> > 
> > Thomas, please me know how I can help.
> 
> Thanks Doug! For starters, I'd be interested to hear any plans you have
> for how to tackle caching, or any thoughts you have on the rough plan I
> described before. If you're happy with the concepts, I'll have a go at
> implementing it. I'll probably consider it experimental until there's a
> hooks mechanism to trigger rebuilding the cache when packages are
> installed or uninstalled.
> 
> Thomas

I assumed that the user loading the plugins might not be able to
write to any of the directories on sys.path (aside from "." and we
don't want to put a cache file there), so my plan was to build the
cache the first time entry points were scanned and use appdirs [1]
to pick a cache location specific to the user.  I thought I would
use the value of sys.path as a string (joining the paths together
with a separator of some sort) to create a hash for the cache file
ID. Some of that may be obviated if we assume a setuptools hook that
lets us update the cache(s) when a package is installed.

I also thought I'd provide a command line tool to generate the cache
just in case it became corrupted or if someone wanted to update it
by hand for some other reason, similar to Nick's locate/updatedb
parallel UX example (and re-reading your email, I see you mention this,
too).

I hadn't gone as far as deciding on a file format, but sqlite, JSON,
and INI (definitely something built-in) were all on my mind.  I
planned to see if we would actually gain enough of a boost just by
placing a separate file for each dist in a single cache directory,
rather than trying to merge everything into one file. In addition
to eliminating the concurrency issue, that approach might have the
additional benefit of simplifying operating system packages, because
they could just add a new file to the package instead of having to
run a command to update the cache when a package was installed (if
the file is the same format as entry_points.txt but with a different
name, that's even simpler since it's just a copy of a file that
will already be available during packaging).

Your idea of having a cache file per directory on sys.path is also
interesting, though I have to admit I'm not familiar enough with
the import machinery to know if it's easy to determine the containing
directory for a dist to find the right cache to update. I am
interested in hearing more details about what you planned there.

I would also like to compare the performance of a few approaches
(1 file per sys.path hash using INI, JSON, and sqlite; one file per
entry on sys.path using the same formats) using a significant number
of plugins (~100?) before we decide.

I agree with your statement in the original email that applications
should be able to disable the cache. I'm not sure it makes sense
to have a mode that only reads from a cache, but I may just not see
the use case for that.

What's our next step?

Doug

[1] https://pypi.python.org/pypi/appdirs/1.4.3

From wes.turner at gmail.com  Fri Oct 20 15:59:18 2017
From: wes.turner at gmail.com (Wes Turner)
Date: Fri, 20 Oct 2017 15:59:18 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1508523033-sup-6338@lrrr.local>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>
 <5F9E5F93-7AF4-4D05-956F-A4367F89CD96@stufft.io>
 <1508500651.117367.1145301232.4A2422BF@webmail.messagingengine.com>
 <6BBFF1A1-F05D-496A-8BEB-FB65D7554823@stufft.io>
 <CADiSq7dmmoB_Hj_Q7ZurMJ1f5e1g=K4oAQQnx5xFucccxUyWhQ@mail.gmail.com>
 <5BD2E6C3-2D0C-45FC-A15D-66ACDBC6B33D@stufft.io>
 <1508503275.124767.1145341384.76E2EDA6@webmail.messagingengine.com>
 <F0F05ABC-788F-49AF-9905-E98F7FDDB988@stufft.io>
 <CADiSq7dE8ZxPZRj0RChEc_65kMpQNUzos_gN-hd6FBu_cLPQ_Q@mail.gmail.com>
 <A2AB937D-D8E0-44E2-8471-3E2D1F433929@stufft.io>
 <CACfEFw8UUtXf-vS0xd9+vT042GJvJ7fOOFw7QANpwUGeFV0Rrg@mail.gmail.com>
 <1508523033-sup-6338@lrrr.local>
Message-ID: <CACfEFw9A-CiLfFOE10gNHM9c7nSMg4gaZEeWeEBi5iaUt+0s_A@mail.gmail.com>

On Friday, October 20, 2017, Doug Hellmann <doug at doughellmann.com> wrote:

> Excerpts from Wes Turner's message of 2017-10-20 10:41:02 -0400:
> > On Friday, October 20, 2017, Donald Stufft <donald at stufft.io
> <javascript:;>> wrote:
> >
> > >
> > >
> > > On Oct 20, 2017, at 9:35 AM, Nick Coghlan <ncoghlan at gmail.com
> <javascript:;>
> > > <javascript:_e(%7B%7D,'cvml','ncoghlan at gmail.com <javascript:;>');>>
> wrote:
> > >
> > > On 20 October 2017 at 23:19, Donald Stufft <donald at stufft.io
> <javascript:;>
> > > <javascript:_e(%7B%7D,'cvml','donald at stufft.io <javascript:;>');>>
> wrote:
> > >
> > >> One that I was helping someone debug just the other day is that
> they?re
> > >> super non-debuggable and the behavior when you have two things
> providing
> > >> the same entry point is basically ???? (If I remember correctly, the
> > >> behavior is that the first thing found is the one that ?wins?, which
> means
> > >> the ordering of sys.path and the names of the projects supply it is
> what
> > >> determines it). This got exposed to the end user that they installed
> > >> something that they thought was going to add support for something,
> but
> > >> which silently did nothing because two different project happened to
> pick
> > >> the same name for their entry point (not the group, it was two things
> > >> providing plugins for the same system).
> > >>
> > >
> > > While I agree with this, I think that's a combination of pkg_resources
> > > itself being hard to debug in general, and the fact that pkg_resources
> > > doesn't clearly define the semantics of how it resolves name conflicts
> > > within an entry point group - as far as I know, it's largely an
> accident of
> > > implementation.
> > >
> > > The interoperability spec is going to state that conflict resolution
> when
> > > the same name within a group is declared by multiple packages is the
> > > responsibility of the group consumer, so documenting the format should
> > > actually improve this situation, since it allows for the development of
> > > competing conflict resolution strategies in different runtime
> libraries.
> > >
> > >
> > > I think it makes it *worse*, because now the behavior isn?t just a
> > > entrypoints weirdness, but now it changes based on which runtime
> library
> > > you use (which isn?t something that end users are likely to have much
> > > insight into) and it represents a footgun that package authors are
> unlikely
> > > to be aware of. If mycoolentrypointslib comes out that is faster, but
> > > changes some subtle behavior like this it?ll break people, but that is
> > > unlikely going to be an effect that people expect to happen just
> because
> > > they switched between two things both implementing the same standard.
> > >
> > > So effectively this means that not only is the fact you?re using
> > > entrypoints part of your API, but now which entry point library you?re
> > > using at runtime is now also part of your API.
> > >
> >
> > When should the check for duplicate entry points occur?
> >
> > - At on_install() time (+1)
> > - At runtime
> >
> > Is a sys.path-like OrderedDict preemptive strategy preferable or just as
> > dangerous as importlib?
>
> Having "duplicate" entry points is not necessarily an error. It's
> a different usage pattern.  The semantics of dropping a named plugin
> into a namespace are defined by the application and plugin-point.
> Please do not build assumptions about uniqueness into the underlying
> implementation.


I think that, at least with console_scripts, we already assume uniqueness:
if there's another package which provides a 'pip' console_script, for
example, there's not yet an error message?

Would it be helpful to at least spec that iterated entrypoints are in
sys.path order? And then what about entrypoints coming from the same path
in sys.path: alphabetical? Whatever hash randomization does with it?

Whenever I feel unsure about my data model, I tend to sometimes read the
OWL spec: here, the OWL spec has owl:cardinality OR owl:minCardinality OR
owl:maxCardinality. Some entrypoints may have 0, only one, or n "instances"?

We should throw an error if a given console_script entrypoint has more than
one "instance" (exceeds maxCardinality xsd:string = 1).


>
> The stevedore library wraps up pkg_resources with several such
> patterns. For example, it supports "give me all of the plugins in
> a namespace" (find all the extensions to your app), "give me all
> of the plugins named $name in a namespace" (find the hooks for a
> specific event defined by the app), and "give me *the* plugin named
> $name in a namespace" (load a driver for talking to a backend).
>
> https://docs.openstack.org/stevedore/latest/reference/index.html


https://github.com/openstack/stevedore/blob/master/stevedore/extension.py

https://github.com/openstack/stevedore/blob/master/stevedore/tests/test_extension.py

These tests mention saving discovered entry points in a cache?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/8b0c7993/attachment.html>

From njs at pobox.com  Fri Oct 20 16:41:03 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 20 Oct 2017 13:41:03 -0700
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
Message-ID: <CAPJVwBnVP_tMKz+ZQLQ9BbkymbrO3msm-zk3X7uyxHmuDWLSfw@mail.gmail.com>

On Oct 19, 2017 11:10, "Donald Stufft" <donald at stufft.io> wrote:


EXCEPT, for the fact that with the desire to cache things, it would be
beneficial to ?hook? into the lifecycle of a package install. However I
know that there are other plugin systems out there that would like to also
be able to do that (Twisted Plugins come to mind) and that I think outside
of plugin systems, such a mechanism is likely to be useful in general for
other cases.

So heres a different idea that is a bit more ambitious but that I think is
a better overall idea. Let entrypoints be a setuptools thing, and lets
define some key lifecycle hooks during the installation of a package and
some mechanism in the metadata to let other tools subscribe to those hooks.
Then  a caching layer could be written for setuptools entrypoints to make
that faster without requiring standardization, but also a whole new, better
plugin system could to, Twisted plugins could benefit, etc [1].


In this hypothetical system, how do installers like pip find the list of
hooks to call? By looking up an entrypoint? (Sorry if this was discussed
downthread; I didn't see it but I admit I only skimmed.)

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/f417321c/attachment-0001.html>

From doug at doughellmann.com  Fri Oct 20 16:50:29 2017
From: doug at doughellmann.com (Doug Hellmann)
Date: Fri, 20 Oct 2017 16:50:29 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CAPJVwBnVP_tMKz+ZQLQ9BbkymbrO3msm-zk3X7uyxHmuDWLSfw@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <CAPJVwBnVP_tMKz+ZQLQ9BbkymbrO3msm-zk3X7uyxHmuDWLSfw@mail.gmail.com>
Message-ID: <1508532378-sup-8768@lrrr.local>

Excerpts from Nathaniel Smith's message of 2017-10-20 13:41:03 -0700:
> On Oct 19, 2017 11:10, "Donald Stufft" <donald at stufft.io> wrote:
> 
> 
> EXCEPT, for the fact that with the desire to cache things, it would be
> beneficial to ?hook? into the lifecycle of a package install. However I
> know that there are other plugin systems out there that would like to also
> be able to do that (Twisted Plugins come to mind) and that I think outside
> of plugin systems, such a mechanism is likely to be useful in general for
> other cases.
> 
> So heres a different idea that is a bit more ambitious but that I think is
> a better overall idea. Let entrypoints be a setuptools thing, and lets
> define some key lifecycle hooks during the installation of a package and
> some mechanism in the metadata to let other tools subscribe to those hooks.
> Then  a caching layer could be written for setuptools entrypoints to make
> that faster without requiring standardization, but also a whole new, better
> plugin system could to, Twisted plugins could benefit, etc [1].

Having post-install and pre-uninstall hooks should be sufficient for
updating a cache, assuming the hook could be given enough information
about the thing being manipulated to probe for whatever data it
needs.

> In this hypothetical system, how do installers like pip find the list of
> hooks to call? By looking up an entrypoint? (Sorry if this was discussed
> downthread; I didn't see it but I admit I only skimmed.)

That's how I would expect it to work. Using setuptools most likely?
That would mean that other plugin systems would have to provide one
setuptools plugin to hook into the installer to build a lookup
cache, but the actual plugins wouldn't have to use setuptools for
anything.

Doug

From dholth at gmail.com  Fri Oct 20 16:50:17 2017
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 20 Oct 2017 20:50:17 +0000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CAPJVwBnVP_tMKz+ZQLQ9BbkymbrO3msm-zk3X7uyxHmuDWLSfw@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <CAPJVwBnVP_tMKz+ZQLQ9BbkymbrO3msm-zk3X7uyxHmuDWLSfw@mail.gmail.com>
Message-ID: <CAG8k2+5Bcfi3QBwOsPyEKhMpNxHKwrSLJoCYing+XbUTm-wRPg@mail.gmail.com>

I like the idea of lifecycle hooks but I worry about the malware problem;
would there need to be a blacklist / whitelist / disable system?
(ignore-scripts=true is now a recommended part of anyone's npm
configuration) That is why we have avoided any kind of (package specific)
hooks to wheel. However hooks would be a very elegant way to avoid worrying
about core pip functionality since it wouldn't be core functionality.

On Fri, Oct 20, 2017 at 4:41 PM Nathaniel Smith <njs at pobox.com> wrote:

> On Oct 19, 2017 11:10, "Donald Stufft" <donald at stufft.io> wrote:
>
>
> EXCEPT, for the fact that with the desire to cache things, it would be
> beneficial to ?hook? into the lifecycle of a package install. However I
> know that there are other plugin systems out there that would like to also
> be able to do that (Twisted Plugins come to mind) and that I think outside
> of plugin systems, such a mechanism is likely to be useful in general for
> other cases.
>
> So heres a different idea that is a bit more ambitious but that I think is
> a better overall idea. Let entrypoints be a setuptools thing, and lets
> define some key lifecycle hooks during the installation of a package and
> some mechanism in the metadata to let other tools subscribe to those hooks.
> Then  a caching layer could be written for setuptools entrypoints to make
> that faster without requiring standardization, but also a whole new, better
> plugin system could to, Twisted plugins could benefit, etc [1].
>
>
> In this hypothetical system, how do installers like pip find the list of
> hooks to call? By looking up an entrypoint? (Sorry if this was discussed
> downthread; I didn't see it but I admit I only skimmed.)
>
> -n
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/2064c189/attachment.html>

From r1chardj0n3s at gmail.com  Fri Oct 20 18:53:42 2017
From: r1chardj0n3s at gmail.com (Richard Jones)
Date: Sat, 21 Oct 2017 09:53:42 +1100
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
Message-ID: <CAHrZfZDKVvXNR-H0BfC9CrKjzp==UqcemxtU9AB_0bCuZAcMww@mail.gmail.com>

Hiya Paul,

There's a bunch of tooling out there using pip's internals to extending
pip's functionality. Could you please provide a some reasoning as to why
they're all going to be broken at pip 10, and possibly some guidance on how
to get that functionality back?


Cheers,

     Richard


On 21 October 2017 at 00:22, Paul Moore <p.f.moore at gmail.com> wrote:

> We're in the process of starting to plan for a release of pip (the
> long-awaited pip 10). We're likely still a month or two away from a
> release, but now is the time for people to start ensuring that
> everything works for them. One key change in the new version will be
> that all of the internal APIs of pip will no longer be available, so
> any code that currently calls functions in the "pip" namespace will
> break. Calling pip's internal APIs has never been supported, and
> always carried a risk of such breakage, so projects doing so should,
> in theory, be prepared for such things. However, reality is not always
> that simple, and we are aware that people will need time to deal with
> the implications.
>
> Just in case it's not clear, simply finding where the internal APIs
> have moved to and calling them under the new names is *not* what
> people should do. We can't stop people calling the internal APIs,
> obviously, but the idea of this change is to give people the incentive
> to find a supported approach, not just to annoy people who are doing
> things we don't want them to ;-)
>
> So please - if you're calling pip's internals in your code, take the
> opportunity *now* to check out the in-development version of pip, and
> ensure your project will still work when pip 10 is released.
>
> And many thanks to anyone else who helps by testing out the new
> version, as well :-)
>
> Thanks,
> Paul
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171021/045363bd/attachment.html>

From xoviat at gmail.com  Fri Oct 20 19:02:22 2017
From: xoviat at gmail.com (xoviat)
Date: Fri, 20 Oct 2017 18:02:22 -0500
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <CAHrZfZDKVvXNR-H0BfC9CrKjzp==UqcemxtU9AB_0bCuZAcMww@mail.gmail.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <CAHrZfZDKVvXNR-H0BfC9CrKjzp==UqcemxtU9AB_0bCuZAcMww@mail.gmail.com>
Message-ID: <CAKXb8mGiUJe0Y0bjtLpqWTJ61JA0xjhuVYLSDj57_C3ivSZGwQ@mail.gmail.com>

The short answer is that someone complained about the pip API not doing
what they wanted it to do, so we broke the API to make it clear to everyone
that the API is not something that should be depended on.

2017-10-20 17:53 GMT-05:00 Richard Jones <r1chardj0n3s at gmail.com>:

> Hiya Paul,
>
> There's a bunch of tooling out there using pip's internals to extending
> pip's functionality. Could you please provide a some reasoning as to why
> they're all going to be broken at pip 10, and possibly some guidance on how
> to get that functionality back?
>
>
> Cheers,
>
>      Richard
>
>
> On 21 October 2017 at 00:22, Paul Moore <p.f.moore at gmail.com> wrote:
>
>> We're in the process of starting to plan for a release of pip (the
>> long-awaited pip 10). We're likely still a month or two away from a
>> release, but now is the time for people to start ensuring that
>> everything works for them. One key change in the new version will be
>> that all of the internal APIs of pip will no longer be available, so
>> any code that currently calls functions in the "pip" namespace will
>> break. Calling pip's internal APIs has never been supported, and
>> always carried a risk of such breakage, so projects doing so should,
>> in theory, be prepared for such things. However, reality is not always
>> that simple, and we are aware that people will need time to deal with
>> the implications.
>>
>> Just in case it's not clear, simply finding where the internal APIs
>> have moved to and calling them under the new names is *not* what
>> people should do. We can't stop people calling the internal APIs,
>> obviously, but the idea of this change is to give people the incentive
>> to find a supported approach, not just to annoy people who are doing
>> things we don't want them to ;-)
>>
>> So please - if you're calling pip's internals in your code, take the
>> opportunity *now* to check out the in-development version of pip, and
>> ensure your project will still work when pip 10 is released.
>>
>> And many thanks to anyone else who helps by testing out the new
>> version, as well :-)
>>
>> Thanks,
>> Paul
>>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171020/2b68f684/attachment-0001.html>

From ncoghlan at gmail.com  Sat Oct 21 02:59:52 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 21 Oct 2017 16:59:52 +1000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CAG8k2+5Bcfi3QBwOsPyEKhMpNxHKwrSLJoCYing+XbUTm-wRPg@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <CAPJVwBnVP_tMKz+ZQLQ9BbkymbrO3msm-zk3X7uyxHmuDWLSfw@mail.gmail.com>
 <CAG8k2+5Bcfi3QBwOsPyEKhMpNxHKwrSLJoCYing+XbUTm-wRPg@mail.gmail.com>
Message-ID: <CADiSq7fbu68be22=-FCzrO9-K-SRHmZ4vYhOsoT+N3UGQ6sW9A@mail.gmail.com>

On 21 October 2017 at 06:50, Daniel Holth <dholth at gmail.com> wrote:

> I like the idea of lifecycle hooks but I worry about the malware problem;
> would there need to be a blacklist / whitelist / disable system?
> (ignore-scripts=true is now a recommended part of anyone's npm
> configuration) That is why we have avoided any kind of (package specific)
> hooks to wheel. However hooks would be a very elegant way to avoid worrying
> about core pip functionality since it wouldn't be core functionality.
>

Yeah, here's the gist of what I had in mind regarding the malware problem
(i.e. aiming to ensure we don't get all of setup.py's problems back again):

- a package's own install hooks do *not* get called for that package
- hooks only run by default inside a virtualenv as a regular user
- outside a virtualenv, the default is "hooks don't get run at all"
- when running with elevated privileges, the default is "hooks don't get
run at all"

There are still some open questions with it (like what to do with hooks
defined in packages that get implicitly coinstalled as a dependency), and
having the default behaviour depend on both "venv or not" and "superuser or
not" may prove confusing, but it would avoid a number of the things we
dislike about install-time setup.py invocation.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171021/e6ca104d/attachment.html>

From ncoghlan at gmail.com  Sat Oct 21 03:15:54 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 21 Oct 2017 17:15:54 +1000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <A2AB937D-D8E0-44E2-8471-3E2D1F433929@stufft.io>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>
 <5F9E5F93-7AF4-4D05-956F-A4367F89CD96@stufft.io>
 <1508500651.117367.1145301232.4A2422BF@webmail.messagingengine.com>
 <6BBFF1A1-F05D-496A-8BEB-FB65D7554823@stufft.io>
 <CADiSq7dmmoB_Hj_Q7ZurMJ1f5e1g=K4oAQQnx5xFucccxUyWhQ@mail.gmail.com>
 <5BD2E6C3-2D0C-45FC-A15D-66ACDBC6B33D@stufft.io>
 <1508503275.124767.1145341384.76E2EDA6@webmail.messagingengine.com>
 <F0F05ABC-788F-49AF-9905-E98F7FDDB988@stufft.io>
 <CADiSq7dE8ZxPZRj0RChEc_65kMpQNUzos_gN-hd6FBu_cLPQ_Q@mail.gmail.com>
 <A2AB937D-D8E0-44E2-8471-3E2D1F433929@stufft.io>
Message-ID: <CADiSq7dMKvLjkxyx+NXkSkSYkk53qcWnpajxBiQg1co_gtm0MA@mail.gmail.com>

On 20 October 2017 at 23:42, Donald Stufft <donald at stufft.io> wrote:

> On Oct 20, 2017, at 9:35 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> The interoperability spec is going to state that conflict resolution when
> the same name within a group is declared by multiple packages is the
> responsibility of the group consumer, so documenting the format should
> actually improve this situation, since it allows for the development of
> competing conflict resolution strategies in different runtime libraries.
>
> I think it makes it *worse*, because now the behavior isn?t just a
> entrypoints weirdness, but now it changes based on which runtime library
> you use (which isn?t something that end users are likely to have much
> insight into) and it represents a footgun that package authors are unlikely
> to be aware of. If mycoolentrypointslib comes out that is faster, but
> changes some subtle behavior like this it?ll break people, but that is
> unlikely going to be an effect that people expect to happen just because
> they switched between two things both implementing the same standard.
>
> So effectively this means that not only is the fact you?re using
> entrypoints part of your API, but now which entry point library you?re
> using at runtime is now also part of your API.
>

The semantics of conflict resolution across different projects is a concern
that mainly affects app developers a large established plugin base, and
even with pkg_resources the question of whether or not multiple projects
re-using the same entrypoint name is a problem depends on how the
application uses that information.

With console_scripts and gui_scripts, name conflicts can definitely be a
problem, since different projects will end up fighting over the same
filename for their executable script wrapper.

For other use cases (like some of the ones Doug described for stevedore),
it's less of a concern, because the names never get collapsed into a single
flat namespace the way script wrappers do.

Cheers,
Nick.

P.S. Thanks for your comments on the PR - they're helping to make sure we
accurately capture the status quo. I'm also going to file an issue on the
setuptools issue tracker to make sure Jason is aware of what we're doing,
and get his explicit OK with the idea of making the format a PyPA
interoperability specification (if he isn't, we'll demote Thomas's document
to being a guide for tool developers aiming for pkg_resources
interoperability).

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171021/dff07201/attachment.html>

From ncoghlan at gmail.com  Sat Oct 21 03:27:01 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 21 Oct 2017 17:27:01 +1000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1508525235-sup-7030@lrrr.local>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <CADiSq7daBpo8TK2cRW=ou7puQsMgWHObBEFBv48C1KOWnEmUhw@mail.gmail.com>
 <1508523732-sup-6024@lrrr.local>
 <1508524665.2432978.1145710912.161590AD@webmail.messagingengine.com>
 <1508525235-sup-7030@lrrr.local>
Message-ID: <CADiSq7fTd3Re0Mi9LzpWywsGAh3FJoY3_VjscSMJV5OtF7KPzg@mail.gmail.com>

On 21 October 2017 at 05:26, Doug Hellmann <doug at doughellmann.com> wrote:

> I would also like to compare the performance of a few approaches
> (1 file per sys.path hash using INI, JSON, and sqlite; one file per
> entry on sys.path using the same formats) using a significant number
> of plugins (~100?) before we decide.
>

If you can manage it, you'll want to run at least some of those tests with
the plugins and their metadata mounted via a network drive. When the import
system switched from multiple stat calls to cached os.listdir() lookups,
SSD and spinning disk imports received a minor speedup, but NFS imports
improved *dramatically* (folks reported order of magnitude improvements,
along the lines of startup times dropping from 2-3 seconds to 200-300 ms).

I'd expect to see a similar pattern here - inefficient file access patterns
can be tolerable with an SSD, and even spinning disks, but the higher
latency involved in accessing network drives will make you pay for it.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171021/be2cb549/attachment-0001.html>

From wes.turner at gmail.com  Sat Oct 21 04:04:46 2017
From: wes.turner at gmail.com (Wes Turner)
Date: Sat, 21 Oct 2017 04:04:46 -0400
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CADiSq7dMKvLjkxyx+NXkSkSYkk53qcWnpajxBiQg1co_gtm0MA@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>
 <5F9E5F93-7AF4-4D05-956F-A4367F89CD96@stufft.io>
 <1508500651.117367.1145301232.4A2422BF@webmail.messagingengine.com>
 <6BBFF1A1-F05D-496A-8BEB-FB65D7554823@stufft.io>
 <CADiSq7dmmoB_Hj_Q7ZurMJ1f5e1g=K4oAQQnx5xFucccxUyWhQ@mail.gmail.com>
 <5BD2E6C3-2D0C-45FC-A15D-66ACDBC6B33D@stufft.io>
 <1508503275.124767.1145341384.76E2EDA6@webmail.messagingengine.com>
 <F0F05ABC-788F-49AF-9905-E98F7FDDB988@stufft.io>
 <CADiSq7dE8ZxPZRj0RChEc_65kMpQNUzos_gN-hd6FBu_cLPQ_Q@mail.gmail.com>
 <A2AB937D-D8E0-44E2-8471-3E2D1F433929@stufft.io>
 <CADiSq7dMKvLjkxyx+NXkSkSYkk53qcWnpajxBiQg1co_gtm0MA@mail.gmail.com>
Message-ID: <CACfEFw-0Njtu2Jm+AUSkycJN=+YQ1Xh7rn7Nnd658RzQ19LNpg@mail.gmail.com>

On Saturday, October 21, 2017, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 20 October 2017 at 23:42, Donald Stufft <donald at stufft.io
> <javascript:_e(%7B%7D,'cvml','donald at stufft.io');>> wrote:
>
>> On Oct 20, 2017, at 9:35 AM, Nick Coghlan <ncoghlan at gmail.com
>> <javascript:_e(%7B%7D,'cvml','ncoghlan at gmail.com');>> wrote:
>> The interoperability spec is going to state that conflict resolution when
>> the same name within a group is declared by multiple packages is the
>> responsibility of the group consumer, so documenting the format should
>> actually improve this situation, since it allows for the development of
>> competing conflict resolution strategies in different runtime libraries.
>>
>> I think it makes it *worse*, because now the behavior isn?t just a
>> entrypoints weirdness, but now it changes based on which runtime library
>> you use (which isn?t something that end users are likely to have much
>> insight into) and it represents a footgun that package authors are unlikely
>> to be aware of. If mycoolentrypointslib comes out that is faster, but
>> changes some subtle behavior like this it?ll break people, but that is
>> unlikely going to be an effect that people expect to happen just because
>> they switched between two things both implementing the same standard.
>>
>> So effectively this means that not only is the fact you?re using
>> entrypoints part of your API, but now which entry point library you?re
>> using at runtime is now also part of your API.
>>
>
> The semantics of conflict resolution across different projects is a
> concern that mainly affects app developers a large established plugin base,
> and even with pkg_resources the question of whether or not multiple
> projects re-using the same entrypoint name is a problem depends on how the
> application uses that information.
>
> With console_scripts and gui_scripts, name conflicts can definitely be a
> problem, since different projects will end up fighting over the same
> filename for their executable script wrapper.
>
> For other use cases (like some of the ones Doug described for stevedore),
> it's less of a concern, because the names never get collapsed into a single
> flat namespace the way script wrappers do.
>
> Cheers,
> Nick.
>
> P.S. Thanks for your comments on the PR - they're helping to make sure we
> accurately capture the status quo. I'm also going to file an issue on the
> setuptools issue tracker to make sure Jason is aware of what we're doing,
> and get his explicit OK with the idea of making the format a PyPA
> interoperability specification (if he isn't, we'll demote Thomas's document
> to being a guide for tool developers aiming for pkg_resources
> interoperability).
>

What are the URIs for this PR and issue?


>
> --
> Nick Coghlan   |   ncoghlan at gmail.com
> <javascript:_e(%7B%7D,'cvml','ncoghlan at gmail.com');>   |   Brisbane,
> Australia
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171021/948b2c83/attachment.html>

From ncoghlan at gmail.com  Sat Oct 21 04:21:44 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 21 Oct 2017 18:21:44 +1000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CACfEFw-0Njtu2Jm+AUSkycJN=+YQ1Xh7rn7Nnd658RzQ19LNpg@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>
 <5F9E5F93-7AF4-4D05-956F-A4367F89CD96@stufft.io>
 <1508500651.117367.1145301232.4A2422BF@webmail.messagingengine.com>
 <6BBFF1A1-F05D-496A-8BEB-FB65D7554823@stufft.io>
 <CADiSq7dmmoB_Hj_Q7ZurMJ1f5e1g=K4oAQQnx5xFucccxUyWhQ@mail.gmail.com>
 <5BD2E6C3-2D0C-45FC-A15D-66ACDBC6B33D@stufft.io>
 <1508503275.124767.1145341384.76E2EDA6@webmail.messagingengine.com>
 <F0F05ABC-788F-49AF-9905-E98F7FDDB988@stufft.io>
 <CADiSq7dE8ZxPZRj0RChEc_65kMpQNUzos_gN-hd6FBu_cLPQ_Q@mail.gmail.com>
 <A2AB937D-D8E0-44E2-8471-3E2D1F433929@stufft.io>
 <CADiSq7dMKvLjkxyx+NXkSkSYkk53qcWnpajxBiQg1co_gtm0MA@mail.gmail.com>
 <CACfEFw-0Njtu2Jm+AUSkycJN=+YQ1Xh7rn7Nnd658RzQ19LNpg@mail.gmail.com>
Message-ID: <CADiSq7eGRc5tia9oRXhur05D9AbUWnftPVE_5vAXDjBdiT0qng@mail.gmail.com>

On 21 October 2017 at 18:04, Wes Turner <wes.turner at gmail.com> wrote:
> On Saturday, October 21, 2017, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> I'm also going to file an issue on the setuptools issue tracker to make
sure Jason is aware of what we're doing, and get his explicit OK with the
idea of making the format a PyPA interoperability specification (if he
isn't, we'll demote Thomas's document to being a guide for tool developers
aiming for pkg_resources interoperability).
>
> What are the URIs for this PR and issue?

New setuptools issue: https://github.com/pypa/setuptools/issues/1179 (I
hadn't filed it yet when I wrote the previous comment)
Thomas's PR: https://github.com/pypa/python-packaging-user-guide/pull/390

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171021/a10f2a60/attachment.html>

From p.f.moore at gmail.com  Sat Oct 21 06:03:21 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 21 Oct 2017 11:03:21 +0100
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <CAHrZfZDKVvXNR-H0BfC9CrKjzp==UqcemxtU9AB_0bCuZAcMww@mail.gmail.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <CAHrZfZDKVvXNR-H0BfC9CrKjzp==UqcemxtU9AB_0bCuZAcMww@mail.gmail.com>
Message-ID: <CACac1F-Mv+-aWauJNH_fZFhyhn3ReaEnze24SxWoXAX-ww0WGg@mail.gmail.com>

On 20 October 2017 at 23:53, Richard Jones <r1chardj0n3s at gmail.com> wrote:
> Hiya Paul,
>
> There's a bunch of tooling out there using pip's internals to extending
> pip's functionality. Could you please provide a some reasoning as to why
> they're all going to be broken at pip 10, and possibly some guidance on how
> to get that functionality back?

Hi Richard,
There was a change to the pip docs that clarified the status of pip's
internal code. The PR for that is at
https://github.com/pypa/pip/pull/4743 but unfortunately it appears
that the docs build has been failing so it hasn't yet made it to the
formal pip docs site.

To summarise, pip has *never* supported the use of its internal APIs
by external code. Over time, we've had a steady trickle of people
raising issues when their code broke because of doing so, and it
usually turned out to be because they violated assumptions made by the
pip code - such as that it's running in a single-threaded application,
or it has sole control over the logging subsystem, or even just that
you can run your own code after calling a pip API. We've explained
this position regularly on those issues, but as is typical, people
don't manage to find similar issues when raising new ones, so we spent
a lot of time repeating ourselves.

Coming up to pip 10, there's been a *lot* of internal work going on,
and it's fairly likely that a decent chunk of code using pip's
internal APIs will break anyway. We don't document internals changes,
so we faced the possibility of an extended period of people raising
issues saying "you broke my code" and us having no better response
than "you shouldn't do that", which would likely hinder adoption of
pip 10, and cause problems for the ecosystem as a whole. Rather than
do this, we decided to make a clean compatibility break, where we
could send out a clear message - "everything's moved, if that matters
to you, then you were using unsupported functionality and you should
find a better way". The breakage is still there (and certainly we made
it affect more people, as there are no doubt some people who would
have survived the pip 10 release unscathed if we hadn't done this) but
at least it's clearly defined and contained.

As to alternatives, we don't have all the answers here but I can offer
the following suggestions:

1. There are a number of external packages that provide functionality
equivalent to what pip does - packaging, wheel, distlib, pkg_resources
are the ones I'm aware of. These are designed as libraries and so *do*
provide supported APIs.
2. We're making a strong push to standardise *all* of the external
interfaces that pip uses, so you should be far more able to write your
own code if that's necessary, without worrying that it'll work
differently than pip does, or that things will suddenly change and
break your code.
3. You can call pip as a subprocess - that's always been supported and
will remain so. We've added some automation-friendly features there
(such as json output format for "pip list") and we'd be happy to add
more if people want to submit PRs.

Likely the biggest problems will be for people who call into the pip
resolver and build APIs, as I don't know of any alternatives out
there. But they were *definitely* breaking anyway, as we've made major
changes to that code (and will be making more).

Also, I should note that we didn't take this decision lightly. We
don't have any particular objection in principle to having a supported
stable pip API, it's just that we don't have anything even close to
the resources needed to define a supported API, maintain it with
acceptable backward compatibility guarantees, and support users who
will inevitably be using it in unexpected and creative ways (we don't
even have the resources to support the limited use of pip's internals
that we see today). Also, pip was never designed originally as a
library, so we *would* have to design that API from scratch. As I
alluded to above, the existing internals code makes some strong
assumptions about how it's called - assumptions that would be
unacceptable in library code, but are fine in an application's
internal code.

Paul

PS People who want to, can of course hunt out the new equivalents of
the code they were using, and just switch. It's not like we can stop
them. But the new names make it clear that they shouldn't be doing
this, so there's an obvious warning there.
PPS Please disregard xoviat's response. This is something we've been
considering for a while, and most definitely not a spur of the moment
decision. It's unfortunate that he was the one most immediately
affected by the change when we made it, but that was just bad timing
(we didn't suddenly do this just because "someone complained").

From ronaldoussoren at mac.com  Sat Oct 21 06:09:19 2017
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Sat, 21 Oct 2017 12:09:19 +0200
Subject: [Distutils] pythonhosted.org doc upload no longer works
In-Reply-To: <CADiSq7e6_oynApo5zok9_uB9mTv86M7ro_NbVEzbcREzWOrQ1w@mail.gmail.com>
References: <CAFYqXL9AXXTdgLfEwBbnh1DejZ1sRqNGp1Dh=o19Rgmqzvokxw@mail.gmail.com>
 <CADiSq7dPLj4J2TgScLscCr8QM3BasP9q9wsarKJecyARJq4BPw@mail.gmail.com>
 <CAF4280LMiC14PjPkACFFxsyshanVHkO5PX0azUdRvr-6cguOmQ@mail.gmail.com>
 <CAFYqXL-9i9MJjTX1Za22L4FqV5pZap1Qihv5Y909eZc0rNbZTg@mail.gmail.com>
 <C5823E59-AB8C-4DA8-AD4A-AE970B062061@coderanger.net>
 <CAFYqXL9kZDc+UicOq_W8POZ+nT5W+dBgFSSZ7NMkUSGAGE2xLA@mail.gmail.com>
 <CAFYqXL-wKDhG8h00esnXUuRXYRQXEdMmGZbx+FLgKvPLXUyaUQ@mail.gmail.com>
 <CAFYqXL-uHDVZ=Yd0puWy=rA4GFNa8aGk2xSLu3ZSxgNvXu7ocA@mail.gmail.com>
 <CAFYqXL9Eq+BrcaX4YBpJutBVU5A5ncYiFXCW=K8xzmnO-hwFNg@mail.gmail.com>
 <CACfEFw9K_4tnEFyVY1fABUp_HnWdFuB7a3513Tg1TyPxmfTZag@mail.gmail.com>
 <CAFYqXL-wFYS80-LSaoVB0WvjnHq4mVfP=OxJNLAS4jTWGgN6jA@mail.gmail.com>
 <CAFYqXL8dZWo4KdVq2GQU-N137DrOLxVChXuX39TEC95P64eDDQ@mail.gmail.com>
 <3FC71839-40AD-497B-8ADD-A343CDB2336A@mac.com>
 <CADiSq7e6_oynApo5zok9_uB9mTv86M7ro_NbVEzbcREzWOrQ1w@mail.gmail.com>
Message-ID: <46D4CF3B-C332-4C88-ACD9-5891B2C9C265@mac.com>


> On 20 Oct 2017, at 15:52, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 20 October 2017 at 22:42, Ronald Oussoren <ronaldoussoren at mac.com <mailto:ronaldoussoren at mac.com>> wrote:
> 
>> On 7 Oct 2017, at 09:12, Giampaolo Rodola' <g.rodola at gmail.com <mailto:g.rodola at gmail.com>> wrote:
>> 
>> Any news about this please? This is becoming a problem as my outdated doc keeps being around and I have no way to remove it. 
> 
> I have the same problem with my projects: the documentation on pythonhosted.org <http://pythonhosted.org/> can no longer be updated, and it is also not possible to change that documentation into a link to a new location.
> 
> I have new documentation on Read the Docs, but Google searches still prefer the old location and I would therefore prefer to have redirects on python host.org <http://host.org/>.
> 
> BTW. Has there been any documentation beyond this list about this change? 
> 
> I don't believe so, as I believe the change from "deprecated" to "no longer works" was a side effect of disabling uploads via the old service.

That?s too bad as this caused problems for a number of people.  I am aware that the PyPI team is severely understaffed at best, but getting hit by this is still annoying.  As far as I know even the deprecation was not communicated widely, sending mail to maintainers of packages that use the feature would have been nice.

I?m currently stuck with old documentation on the site that has Google ranking, and no way to change anything about that. A way to replace the documentation by a redirect would have been nice (and was something that I wanted to do anyways, I?ve moved documentation to Read the Docs because that way the website gets updated automatically leaving me more time to do useful work)

Ronald


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171021/6fd0ca4c/attachment.html>

From ncoghlan at gmail.com  Sat Oct 21 07:15:17 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 21 Oct 2017 21:15:17 +1000
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <CACac1F-Mv+-aWauJNH_fZFhyhn3ReaEnze24SxWoXAX-ww0WGg@mail.gmail.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <CAHrZfZDKVvXNR-H0BfC9CrKjzp==UqcemxtU9AB_0bCuZAcMww@mail.gmail.com>
 <CACac1F-Mv+-aWauJNH_fZFhyhn3ReaEnze24SxWoXAX-ww0WGg@mail.gmail.com>
Message-ID: <CADiSq7df8CvGQB24UTb+uQxHunrsQHU97Nc6kmOSr+wY99Xg4Q@mail.gmail.com>

On 21 October 2017 at 20:03, Paul Moore <p.f.moore at gmail.com> wrote:

> Likely the biggest problems will be for people who call into the pip
> resolver and build APIs, as I don't know of any alternatives out
> there. But they were *definitely* breaking anyway, as we've made major
> changes to that code (and will be making more).
>
> Also, I should note that we didn't take this decision lightly. We
> don't have any particular objection in principle to having a supported
> stable pip API, it's just that we don't have anything even close to
> the resources needed to define a supported API, maintain it with
> acceptable backward compatibility guarantees, and support users who
> will inevitably be using it in unexpected and creative ways (we don't
> even have the resources to support the limited use of pip's internals
> that we see today). Also, pip was never designed originally as a
> library, so we *would* have to design that API from scratch. As I
> alluded to above, the existing internals code makes some strong
> assumptions about how it's called - assumptions that would be
> unacceptable in library code, but are fine in an application's
> internal code.
>

(Note: this is entirely speculative, and I have no idea how hard it would
be, so please read it as the question it's intended to be)

Do you know if there any key APIs (like installation) that could be turned
into wrappers around pip CLI calls in order to mitigate some of the impact?

The reason I ask is because it's unlikely anyone else is going to
understand how to emulate the previous functionality better than the pip
devs would, and if there's an API for those particular invocations, than
they can be covered directly by pip's test suite.

Plus if there are previous API capabilities that *can't* currently be
emulated via the CLI, then the pip devs are the folks in the best position
to add the necessary CLI enhancements (such as the ones Noah asked about
for doing a more selective `pip list`).

If that's an approach you might be amenable to, then a 10.0 pre-release
could be a good time to solicit PRs from folks that were using particular
APIs and would be prepared to invest time in defining comparable CLI
wrappers for them.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171021/9783b786/attachment.html>

From r1chardj0n3s at gmail.com  Sat Oct 21 07:43:33 2017
From: r1chardj0n3s at gmail.com (Richard Jones)
Date: Sat, 21 Oct 2017 22:43:33 +1100
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <CACac1F-Mv+-aWauJNH_fZFhyhn3ReaEnze24SxWoXAX-ww0WGg@mail.gmail.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <CAHrZfZDKVvXNR-H0BfC9CrKjzp==UqcemxtU9AB_0bCuZAcMww@mail.gmail.com>
 <CACac1F-Mv+-aWauJNH_fZFhyhn3ReaEnze24SxWoXAX-ww0WGg@mail.gmail.com>
Message-ID: <CAHrZfZBJcQGZ+OV9xbOT_3=qgQORw4areutOs+FcGSF1Xx6GJw@mail.gmail.com>

Thanks for writing that detailed explanation, Paul (and all your other hard
work!)


     Richard


On 21 October 2017 at 21:03, Paul Moore <p.f.moore at gmail.com> wrote:

> On 20 October 2017 at 23:53, Richard Jones <r1chardj0n3s at gmail.com> wrote:
> > Hiya Paul,
> >
> > There's a bunch of tooling out there using pip's internals to extending
> > pip's functionality. Could you please provide a some reasoning as to why
> > they're all going to be broken at pip 10, and possibly some guidance on
> how
> > to get that functionality back?
>
> Hi Richard,
> There was a change to the pip docs that clarified the status of pip's
> internal code. The PR for that is at
> https://github.com/pypa/pip/pull/4743 but unfortunately it appears
> that the docs build has been failing so it hasn't yet made it to the
> formal pip docs site.
>
> To summarise, pip has *never* supported the use of its internal APIs
> by external code. Over time, we've had a steady trickle of people
> raising issues when their code broke because of doing so, and it
> usually turned out to be because they violated assumptions made by the
> pip code - such as that it's running in a single-threaded application,
> or it has sole control over the logging subsystem, or even just that
> you can run your own code after calling a pip API. We've explained
> this position regularly on those issues, but as is typical, people
> don't manage to find similar issues when raising new ones, so we spent
> a lot of time repeating ourselves.
>
> Coming up to pip 10, there's been a *lot* of internal work going on,
> and it's fairly likely that a decent chunk of code using pip's
> internal APIs will break anyway. We don't document internals changes,
> so we faced the possibility of an extended period of people raising
> issues saying "you broke my code" and us having no better response
> than "you shouldn't do that", which would likely hinder adoption of
> pip 10, and cause problems for the ecosystem as a whole. Rather than
> do this, we decided to make a clean compatibility break, where we
> could send out a clear message - "everything's moved, if that matters
> to you, then you were using unsupported functionality and you should
> find a better way". The breakage is still there (and certainly we made
> it affect more people, as there are no doubt some people who would
> have survived the pip 10 release unscathed if we hadn't done this) but
> at least it's clearly defined and contained.
>
> As to alternatives, we don't have all the answers here but I can offer
> the following suggestions:
>
> 1. There are a number of external packages that provide functionality
> equivalent to what pip does - packaging, wheel, distlib, pkg_resources
> are the ones I'm aware of. These are designed as libraries and so *do*
> provide supported APIs.
> 2. We're making a strong push to standardise *all* of the external
> interfaces that pip uses, so you should be far more able to write your
> own code if that's necessary, without worrying that it'll work
> differently than pip does, or that things will suddenly change and
> break your code.
> 3. You can call pip as a subprocess - that's always been supported and
> will remain so. We've added some automation-friendly features there
> (such as json output format for "pip list") and we'd be happy to add
> more if people want to submit PRs.
>
> Likely the biggest problems will be for people who call into the pip
> resolver and build APIs, as I don't know of any alternatives out
> there. But they were *definitely* breaking anyway, as we've made major
> changes to that code (and will be making more).
>
> Also, I should note that we didn't take this decision lightly. We
> don't have any particular objection in principle to having a supported
> stable pip API, it's just that we don't have anything even close to
> the resources needed to define a supported API, maintain it with
> acceptable backward compatibility guarantees, and support users who
> will inevitably be using it in unexpected and creative ways (we don't
> even have the resources to support the limited use of pip's internals
> that we see today). Also, pip was never designed originally as a
> library, so we *would* have to design that API from scratch. As I
> alluded to above, the existing internals code makes some strong
> assumptions about how it's called - assumptions that would be
> unacceptable in library code, but are fine in an application's
> internal code.
>
> Paul
>
> PS People who want to, can of course hunt out the new equivalents of
> the code they were using, and just switch. It's not like we can stop
> them. But the new names make it clear that they shouldn't be doing
> this, so there's an obvious warning there.
> PPS Please disregard xoviat's response. This is something we've been
> considering for a while, and most definitely not a spur of the moment
> decision. It's unfortunate that he was the one most immediately
> affected by the change when we made it, but that was just bad timing
> (we didn't suddenly do this just because "someone complained").
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171021/ccc15032/attachment-0001.html>

From p.f.moore at gmail.com  Sat Oct 21 08:34:37 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 21 Oct 2017 13:34:37 +0100
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <CADiSq7df8CvGQB24UTb+uQxHunrsQHU97Nc6kmOSr+wY99Xg4Q@mail.gmail.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <CAHrZfZDKVvXNR-H0BfC9CrKjzp==UqcemxtU9AB_0bCuZAcMww@mail.gmail.com>
 <CACac1F-Mv+-aWauJNH_fZFhyhn3ReaEnze24SxWoXAX-ww0WGg@mail.gmail.com>
 <CADiSq7df8CvGQB24UTb+uQxHunrsQHU97Nc6kmOSr+wY99Xg4Q@mail.gmail.com>
Message-ID: <CACac1F_PkcHekc-2qkas5J6w0xUumjuVJCCMcCGXPF4Ft1zmSw@mail.gmail.com>

On 21 October 2017 at 12:15, Nick Coghlan <ncoghlan at gmail.com> wrote:
> (Note: this is entirely speculative, and I have no idea how hard it would
> be, so please read it as the question it's intended to be)

No problem - I don't know myself how hard some of this would be, either ;-)

> Do you know if there any key APIs (like installation) that could be turned
> into wrappers around pip CLI calls in order to mitigate some of the impact?

The obvious one is pip.main(). That's the one a lot of people use, but
it's easily replaceable by a simple subprocess call. That's actually
one of the reasons this was so frustrating to us - the bug reports we
got were often from people doing things they didn't need to, that they
could handle trivially via a supported approach.

Otherwise, no. We've had little or no feedback on the tracker from
people using more complex internals, so our working assumption has
been there's very little that can't be handled via the CLI or existing
packages. Feedback so far from this mail hints that maybe we were
wrong, but it's still hard to know if it's one or two key projects, or
a whole range of people that we've yet to hear from. I'm pretty sure,
for example, that pipenv uses internals, either directly or via one of
their dependencies, but we've not seen any feedback from them yet.

> The reason I ask is because it's unlikely anyone else is going to understand
> how to emulate the previous functionality better than the pip devs would,
> and if there's an API for those particular invocations, than they can be
> covered directly by pip's test suite.

Understood. We understand *how*, but don't know what is needed. One of
the points of all this is to tease out such requirements. We'd hope to
get them addressed by including them into *other* packages like
distlib or packaging, but that's mostly just a matter of where the API
goes, not what is needed. It does help us avoid tying fixes to pip's
release cycle, though.

It's also worth noting that the pip devs are possibly way too deep
into how pip does things, rather than what the standards say. Getting
others to implement libraries based on the published standards would
help immensely to ensure that we're not falling into the
"implementation defined" trap of the community being stuck with having
to use pip because no-one else knows how to do what pip does.

> Plus if there are previous API capabilities that *can't* currently be
> emulated via the CLI, then the pip devs are the folks in the best position
> to add the necessary CLI enhancements (such as the ones Noah asked about for
> doing a more selective `pip list`).

Oh, sure - apart from the aforementioned "pip.main()", most
capabilities of the internal API are *not* easily emulated by the CLI.

But 3rd party libraries are just as much an option. Remember, the
issue here isn't so much about designing an API as about exposing (and
therefore locking in stone) internal implementation details of how pip
does things. And the pip devs are arguably in the *worst* position to
handle that option, precisely because they know so much about how pip
does things.

> If that's an approach you might be amenable to, then a 10.0 pre-release
> could be a good time to solicit PRs from folks that were using particular
> APIs and would be prepared to invest time in defining comparable CLI
> wrappers for them.

Well, I get your point here, but the implication of this is that we
have to design and build an API before we can release pip 10. Calling
it CLI wrappers (and implementing it that way) doesn't do anything to
ease the work needed on design, or the maintenance burden of providing
stability. We're already getting a lot of pressure to release pip 10,
and trying to do that would push it way further off.

To be blunt, no-one on the pip team is really interested in trying to
provide or support a stable API at this point in time. We have our
hands full, as everyone is aware, and this is way down all our
priorities. Community-submitted PRs would help, but there's still work
in agreeing a design, reviewing those PRs, and maintaining them
(exposing details from the internal APIs limits how much we can change
those internals later, which is something we have to consider).

What would be ideal would be for the community to build
standards-based libraries that satisfied the needs that people
currently handle by using pip internals. Heck, we could even consider
vendoring such libraries inside of pip and saving ourselves a bunch of
complexity ;-) The packaging, pkg_resources and distlib libraries are
examples of this happening. But it doesn't seem to be a popular route,
unfortunately. And those particular libraries are all maintained by
PyPA members, so don't really count as examples of the community
getting involved...

The most likely alternative solution would be to revert the internal
API moves. If that happened, I'd still prefer to lose pip.main, as
that's the major pain point for us and the easiest for users to solve,
but we could put the rest back. But it would be naive to assume that
as a result people like Noah would be unaffected - they'd actually be
worse off, as the internal APIs they use would probably break anyway
(the installer/resolver stuff has been heavily modified for pip 10)
but not all would do so in obvious ways.

Beyond this, I don't know.

Paul

From xoviat at gmail.com  Sat Oct 21 14:03:02 2017
From: xoviat at gmail.com (xoviat)
Date: Sat, 21 Oct 2017 13:03:02 -0500
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <CADiSq7df8CvGQB24UTb+uQxHunrsQHU97Nc6kmOSr+wY99Xg4Q@mail.gmail.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <CAHrZfZDKVvXNR-H0BfC9CrKjzp==UqcemxtU9AB_0bCuZAcMww@mail.gmail.com>
 <CACac1F-Mv+-aWauJNH_fZFhyhn3ReaEnze24SxWoXAX-ww0WGg@mail.gmail.com>
 <CADiSq7df8CvGQB24UTb+uQxHunrsQHU97Nc6kmOSr+wY99Xg4Q@mail.gmail.com>
Message-ID: <CAKXb8mE9CwRyeMdYjpTS=AGRtf+TDFVm=yyw65mUy6pC8uaH5w@mail.gmail.com>

Nick:

That's generally a good idea, but one significant problem that can occur is
that the Python import system will cache certain libraries, people will run
"pip install," and then they will expect such libraries to be available. I
don't even know exactly how the caching for the import system works, so I
don't want to go and make claims about it that may be incorrect (maybe you
do?). However, it is important to keep that in mind when considering an API.

2017-10-21 6:15 GMT-05:00 Nick Coghlan <ncoghlan at gmail.com>:

> On 21 October 2017 at 20:03, Paul Moore <p.f.moore at gmail.com> wrote:
>
>> Likely the biggest problems will be for people who call into the pip
>> resolver and build APIs, as I don't know of any alternatives out
>> there. But they were *definitely* breaking anyway, as we've made major
>> changes to that code (and will be making more).
>>
>> Also, I should note that we didn't take this decision lightly. We
>> don't have any particular objection in principle to having a supported
>> stable pip API, it's just that we don't have anything even close to
>> the resources needed to define a supported API, maintain it with
>> acceptable backward compatibility guarantees, and support users who
>> will inevitably be using it in unexpected and creative ways (we don't
>> even have the resources to support the limited use of pip's internals
>> that we see today). Also, pip was never designed originally as a
>> library, so we *would* have to design that API from scratch. As I
>> alluded to above, the existing internals code makes some strong
>> assumptions about how it's called - assumptions that would be
>> unacceptable in library code, but are fine in an application's
>> internal code.
>>
>
> (Note: this is entirely speculative, and I have no idea how hard it would
> be, so please read it as the question it's intended to be)
>
> Do you know if there any key APIs (like installation) that could be turned
> into wrappers around pip CLI calls in order to mitigate some of the impact?
>
> The reason I ask is because it's unlikely anyone else is going to
> understand how to emulate the previous functionality better than the pip
> devs would, and if there's an API for those particular invocations, than
> they can be covered directly by pip's test suite.
>
> Plus if there are previous API capabilities that *can't* currently be
> emulated via the CLI, then the pip devs are the folks in the best position
> to add the necessary CLI enhancements (such as the ones Noah asked about
> for doing a more selective `pip list`).
>
> If that's an approach you might be amenable to, then a 10.0 pre-release
> could be a good time to solicit PRs from folks that were using particular
> APIs and would be prepared to invest time in defining comparable CLI
> wrappers for them.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171021/b34f430f/attachment.html>

From alex.gronholm at nextday.fi  Sat Oct 21 14:11:18 2017
From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=)
Date: Sat, 21 Oct 2017 21:11:18 +0300
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <CACac1F-Mv+-aWauJNH_fZFhyhn3ReaEnze24SxWoXAX-ww0WGg@mail.gmail.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <CAHrZfZDKVvXNR-H0BfC9CrKjzp==UqcemxtU9AB_0bCuZAcMww@mail.gmail.com>
 <CACac1F-Mv+-aWauJNH_fZFhyhn3ReaEnze24SxWoXAX-ww0WGg@mail.gmail.com>
Message-ID: <17248240-7b4b-f114-d274-a0a0accb13e1@nextday.fi>

Paul Moore kirjoitti 21.10.2017 klo 13:03:
> On 20 October 2017 at 23:53, Richard Jones <r1chardj0n3s at gmail.com> wrote:
>> Hiya Paul,
>>
>> There's a bunch of tooling out there using pip's internals to extending
>> pip's functionality. Could you please provide a some reasoning as to why
>> they're all going to be broken at pip 10, and possibly some guidance on how
>> to get that functionality back?
> Hi Richard,
> There was a change to the pip docs that clarified the status of pip's
> internal code. The PR for that is at
> https://github.com/pypa/pip/pull/4743 but unfortunately it appears
> that the docs build has been failing so it hasn't yet made it to the
> formal pip docs site.
>
> To summarise, pip has *never* supported the use of its internal APIs
> by external code. Over time, we've had a steady trickle of people
> raising issues when their code broke because of doing so, and it
> usually turned out to be because they violated assumptions made by the
> pip code - such as that it's running in a single-threaded application,
> or it has sole control over the logging subsystem, or even just that
> you can run your own code after calling a pip API. We've explained
> this position regularly on those issues, but as is typical, people
> don't manage to find similar issues when raising new ones, so we spent
> a lot of time repeating ourselves.
>
> Coming up to pip 10, there's been a *lot* of internal work going on,
> and it's fairly likely that a decent chunk of code using pip's
> internal APIs will break anyway. We don't document internals changes,
> so we faced the possibility of an extended period of people raising
> issues saying "you broke my code" and us having no better response
> than "you shouldn't do that", which would likely hinder adoption of
> pip 10, and cause problems for the ecosystem as a whole. Rather than
> do this, we decided to make a clean compatibility break, where we
> could send out a clear message - "everything's moved, if that matters
> to you, then you were using unsupported functionality and you should
> find a better way". The breakage is still there (and certainly we made
> it affect more people, as there are no doubt some people who would
> have survived the pip 10 release unscathed if we hadn't done this) but
> at least it's clearly defined and contained.
>
> As to alternatives, we don't have all the answers here but I can offer
> the following suggestions:
>
> 1. There are a number of external packages that provide functionality
> equivalent to what pip does - packaging, wheel, distlib, pkg_resources
> are the ones I'm aware of. These are designed as libraries and so *do*
> provide supported APIs.
I need to correct you here: wheel does *NOT* have a public API!

It did previously have some documented functions but it was not really 
well thought out and we recently decided to remove all traces of API 
documentation so as to not cause any confusion about it.
> 2. We're making a strong push to standardise *all* of the external
> interfaces that pip uses, so you should be far more able to write your
> own code if that's necessary, without worrying that it'll work
> differently than pip does, or that things will suddenly change and
> break your code.
> 3. You can call pip as a subprocess - that's always been supported and
> will remain so. We've added some automation-friendly features there
> (such as json output format for "pip list") and we'd be happy to add
> more if people want to submit PRs.
>
> Likely the biggest problems will be for people who call into the pip
> resolver and build APIs, as I don't know of any alternatives out
> there. But they were *definitely* breaking anyway, as we've made major
> changes to that code (and will be making more).
>
> Also, I should note that we didn't take this decision lightly. We
> don't have any particular objection in principle to having a supported
> stable pip API, it's just that we don't have anything even close to
> the resources needed to define a supported API, maintain it with
> acceptable backward compatibility guarantees, and support users who
> will inevitably be using it in unexpected and creative ways (we don't
> even have the resources to support the limited use of pip's internals
> that we see today). Also, pip was never designed originally as a
> library, so we *would* have to design that API from scratch. As I
> alluded to above, the existing internals code makes some strong
> assumptions about how it's called - assumptions that would be
> unacceptable in library code, but are fine in an application's
> internal code.
>
> Paul
>
> PS People who want to, can of course hunt out the new equivalents of
> the code they were using, and just switch. It's not like we can stop
> them. But the new names make it clear that they shouldn't be doing
> this, so there's an obvious warning there.
> PPS Please disregard xoviat's response. This is something we've been
> considering for a while, and most definitely not a spur of the moment
> decision. It's unfortunate that he was the one most immediately
> affected by the change when we made it, but that was just bad timing
> (we didn't suddenly do this just because "someone complained").
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


From brett at python.org  Sat Oct 21 14:15:55 2017
From: brett at python.org (Brett Cannon)
Date: Sat, 21 Oct 2017 18:15:55 +0000
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <CAKXb8mE9CwRyeMdYjpTS=AGRtf+TDFVm=yyw65mUy6pC8uaH5w@mail.gmail.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <CAHrZfZDKVvXNR-H0BfC9CrKjzp==UqcemxtU9AB_0bCuZAcMww@mail.gmail.com>
 <CACac1F-Mv+-aWauJNH_fZFhyhn3ReaEnze24SxWoXAX-ww0WGg@mail.gmail.com>
 <CADiSq7df8CvGQB24UTb+uQxHunrsQHU97Nc6kmOSr+wY99Xg4Q@mail.gmail.com>
 <CAKXb8mE9CwRyeMdYjpTS=AGRtf+TDFVm=yyw65mUy6pC8uaH5w@mail.gmail.com>
Message-ID: <CAP1=2W4WrYuYLDYZ5gRjfZqzaxoBxeWMjot3jKXw-Q8gJQVT-Q@mail.gmail.com>

On Sat, 21 Oct 2017 at 11:03 xoviat <xoviat at gmail.com> wrote:

> Nick:
>
> That's generally a good idea, but one significant problem that can occur
> is that the Python import system will cache certain libraries, people will
> run "pip install," and then they will expect such libraries to be
> available. I don't even know exactly how the caching for the import system
> works, so I don't want to go and make claims about it that may be incorrect
> (maybe you do?). However, it is important to keep that in mind when
> considering an API.
>

Nick does know how the import system works, but to specifically address
this point, as long as the module isn't already imported it's fine. If it
has, though, then you will need to do a reload instead of an import, and
even then that doesn't necessarily work the way some people want it to.

-Brett


>
> 2017-10-21 6:15 GMT-05:00 Nick Coghlan <ncoghlan at gmail.com>:
>
>> On 21 October 2017 at 20:03, Paul Moore <p.f.moore at gmail.com> wrote:
>>
>>> Likely the biggest problems will be for people who call into the pip
>>> resolver and build APIs, as I don't know of any alternatives out
>>> there. But they were *definitely* breaking anyway, as we've made major
>>> changes to that code (and will be making more).
>>>
>>> Also, I should note that we didn't take this decision lightly. We
>>> don't have any particular objection in principle to having a supported
>>> stable pip API, it's just that we don't have anything even close to
>>> the resources needed to define a supported API, maintain it with
>>> acceptable backward compatibility guarantees, and support users who
>>> will inevitably be using it in unexpected and creative ways (we don't
>>> even have the resources to support the limited use of pip's internals
>>> that we see today). Also, pip was never designed originally as a
>>> library, so we *would* have to design that API from scratch. As I
>>> alluded to above, the existing internals code makes some strong
>>> assumptions about how it's called - assumptions that would be
>>> unacceptable in library code, but are fine in an application's
>>> internal code.
>>>
>>
>> (Note: this is entirely speculative, and I have no idea how hard it would
>> be, so please read it as the question it's intended to be)
>>
>> Do you know if there any key APIs (like installation) that could be
>> turned into wrappers around pip CLI calls in order to mitigate some of the
>> impact?
>>
>> The reason I ask is because it's unlikely anyone else is going to
>> understand how to emulate the previous functionality better than the pip
>> devs would, and if there's an API for those particular invocations, than
>> they can be covered directly by pip's test suite.
>>
>> Plus if there are previous API capabilities that *can't* currently be
>> emulated via the CLI, then the pip devs are the folks in the best position
>> to add the necessary CLI enhancements (such as the ones Noah asked about
>> for doing a more selective `pip list`).
>>
>> If that's an approach you might be amenable to, then a 10.0 pre-release
>> could be a good time to solicit PRs from folks that were using particular
>> APIs and would be prepared to invest time in defining comparable CLI
>> wrappers for them.
>>
>> Cheers,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171021/9b700442/attachment-0001.html>

From donald at stufft.io  Sat Oct 21 14:26:26 2017
From: donald at stufft.io (Donald Stufft)
Date: Sat, 21 Oct 2017 14:26:26 -0400
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <CAP1=2W4WrYuYLDYZ5gRjfZqzaxoBxeWMjot3jKXw-Q8gJQVT-Q@mail.gmail.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <CAHrZfZDKVvXNR-H0BfC9CrKjzp==UqcemxtU9AB_0bCuZAcMww@mail.gmail.com>
 <CACac1F-Mv+-aWauJNH_fZFhyhn3ReaEnze24SxWoXAX-ww0WGg@mail.gmail.com>
 <CADiSq7df8CvGQB24UTb+uQxHunrsQHU97Nc6kmOSr+wY99Xg4Q@mail.gmail.com>
 <CAKXb8mE9CwRyeMdYjpTS=AGRtf+TDFVm=yyw65mUy6pC8uaH5w@mail.gmail.com>
 <CAP1=2W4WrYuYLDYZ5gRjfZqzaxoBxeWMjot3jKXw-Q8gJQVT-Q@mail.gmail.com>
Message-ID: <78D405DD-0A8A-4F5F-A02E-153ACA21958D@stufft.io>



> On Oct 21, 2017, at 2:15 PM, Brett Cannon <brett at python.org> wrote:
> 
> as long as the module isn't already imported it's fine.

Negative imports get cached too don?t they?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171021/946d18a5/attachment.html>

From brett at python.org  Sat Oct 21 14:36:56 2017
From: brett at python.org (Brett Cannon)
Date: Sat, 21 Oct 2017 18:36:56 +0000
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <78D405DD-0A8A-4F5F-A02E-153ACA21958D@stufft.io>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <CAHrZfZDKVvXNR-H0BfC9CrKjzp==UqcemxtU9AB_0bCuZAcMww@mail.gmail.com>
 <CACac1F-Mv+-aWauJNH_fZFhyhn3ReaEnze24SxWoXAX-ww0WGg@mail.gmail.com>
 <CADiSq7df8CvGQB24UTb+uQxHunrsQHU97Nc6kmOSr+wY99Xg4Q@mail.gmail.com>
 <CAKXb8mE9CwRyeMdYjpTS=AGRtf+TDFVm=yyw65mUy6pC8uaH5w@mail.gmail.com>
 <CAP1=2W4WrYuYLDYZ5gRjfZqzaxoBxeWMjot3jKXw-Q8gJQVT-Q@mail.gmail.com>
 <78D405DD-0A8A-4F5F-A02E-153ACA21958D@stufft.io>
Message-ID: <CAP1=2W7d7-UFze3smEveypZgBh0dP8RYaO+PjJwpuiD1OEBsEA@mail.gmail.com>

On Sat, 21 Oct 2017 at 11:26 Donald Stufft <donald at stufft.io> wrote:

>
>
> On Oct 21, 2017, at 2:15 PM, Brett Cannon <brett at python.org> wrote:
>
> as long as the module isn't already imported it's fine.
>
>
> Negative imports get cached too don?t they?
>

Yes, that too. :) (aside: don't reply to technical emails while you're
waiting to head out the door for a weekend away, you will forget details ;)
.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171021/adfe3477/attachment.html>

From p.f.moore at gmail.com  Sat Oct 21 15:28:37 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 21 Oct 2017 20:28:37 +0100
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <17248240-7b4b-f114-d274-a0a0accb13e1@nextday.fi>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <CAHrZfZDKVvXNR-H0BfC9CrKjzp==UqcemxtU9AB_0bCuZAcMww@mail.gmail.com>
 <CACac1F-Mv+-aWauJNH_fZFhyhn3ReaEnze24SxWoXAX-ww0WGg@mail.gmail.com>
 <17248240-7b4b-f114-d274-a0a0accb13e1@nextday.fi>
Message-ID: <CACac1F-fO-6MjiqfBFWfC-74Y-Y+v2hqzgjQTK3RahUKHf3gOg@mail.gmail.com>

On 21 October 2017 at 19:11, Alex Gr?nholm <alex.gronholm at nextday.fi> wrote:

> I need to correct you here: wheel does *NOT* have a public API!

Ah, sorry. I hadn't realised that.

> It did previously have some documented functions but it was not really well
> thought out and we recently decided to remove all traces of API
> documentation so as to not cause any confusion about it.

That's probably what I was remembering.

Paul

From ncoghlan at gmail.com  Sat Oct 21 22:30:44 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 22 Oct 2017 12:30:44 +1000
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <CAKXb8mE9CwRyeMdYjpTS=AGRtf+TDFVm=yyw65mUy6pC8uaH5w@mail.gmail.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <CAHrZfZDKVvXNR-H0BfC9CrKjzp==UqcemxtU9AB_0bCuZAcMww@mail.gmail.com>
 <CACac1F-Mv+-aWauJNH_fZFhyhn3ReaEnze24SxWoXAX-ww0WGg@mail.gmail.com>
 <CADiSq7df8CvGQB24UTb+uQxHunrsQHU97Nc6kmOSr+wY99Xg4Q@mail.gmail.com>
 <CAKXb8mE9CwRyeMdYjpTS=AGRtf+TDFVm=yyw65mUy6pC8uaH5w@mail.gmail.com>
Message-ID: <CADiSq7ccUoAcxhKc_23fny5TAHiN2bN5Gv4xod13sSPfJHoPgw@mail.gmail.com>

On 22 October 2017 at 04:03, xoviat <xoviat at gmail.com> wrote:

> Nick:
>
> That's generally a good idea, but one significant problem that can occur
> is that the Python import system will cache certain libraries, people will
> run "pip install," and then they will expect such libraries to be
> available. I don't even know exactly how the caching for the import system
> works, so I don't want to go and make claims about it that may be incorrect
> (maybe you do?). However, it is important to keep that in mind when
> considering an API.
>

Yep, since we switched to the implementation that uses fewer stat calls,
you need to call `importlib.invalidate_caches()` to be sure your current
process will see anything you just installed. However, for modules you
haven't previously imported, that's independent of how you installed them -
the problem is that the granularity on the import system's automated cache
invalidation depends on the granularity of the filesystem's mtime records
for directories, so you either have to sleep for a few seconds (since the
mtime resolution on filesystems like FAT & FAT32 is only a couple of
seconds), or else force the cache invalidation.

The problem with the sys.modules cache and already imported libraries is
different: for those, you either need to use
`importlib.reload(existing_module)` to force an in-place update of the
existing namespace, or else `del sys.modules[existing_module.__name__]` to
force creation of a new module without affecting old references to it.

It's those subtleties that keep "Restart all Python processes if you expect
them to see components you just installed" in place as our default
guidance: it's the only option that's guaranteed to work in all cases.
While hot reloading can be made to work, it has assorted caveats and
restrictions that you need to learn in order to do it correctly and
reliably (many of them are actually pretty similar to the self-imposed
restrictions needed to make lazy loading work reliably).

However, none of that impacts the question of whether `pip.main()` runs
code in the current process or implicitly runs it in a subprocess - `pip`
doesn't import the modules it installs either way, so it will all look the
same as far as the import system is concerned.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171022/a1df9572/attachment-0001.html>

From donald at stufft.io  Sun Oct 22 02:10:06 2017
From: donald at stufft.io (Donald Stufft)
Date: Sun, 22 Oct 2017 02:10:06 -0400
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <CADiSq7ccUoAcxhKc_23fny5TAHiN2bN5Gv4xod13sSPfJHoPgw@mail.gmail.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <CAHrZfZDKVvXNR-H0BfC9CrKjzp==UqcemxtU9AB_0bCuZAcMww@mail.gmail.com>
 <CACac1F-Mv+-aWauJNH_fZFhyhn3ReaEnze24SxWoXAX-ww0WGg@mail.gmail.com>
 <CADiSq7df8CvGQB24UTb+uQxHunrsQHU97Nc6kmOSr+wY99Xg4Q@mail.gmail.com>
 <CAKXb8mE9CwRyeMdYjpTS=AGRtf+TDFVm=yyw65mUy6pC8uaH5w@mail.gmail.com>
 <CADiSq7ccUoAcxhKc_23fny5TAHiN2bN5Gv4xod13sSPfJHoPgw@mail.gmail.com>
Message-ID: <440E8DF4-5E3C-4D1A-BBE1-89B04D07E6FF@stufft.io>


> On Oct 21, 2017, at 10:30 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> However, none of that impacts the question of whether `pip.main()` runs code in the current process or implicitly runs it in a subprocess - `pip` doesn't import the modules it installs either way, so it will all look the same as far as the import system is concerned.


That of course also ignores things like ?foo.py optionally imports bar.py if it is available?, with something like:

try:
    import bar
except ImportError:
    bar = None


If you then did ``import foo``, noticed the optional features powered by bar weren?t added, you would also have to reload ``foo`` (and track down any references to it!). Reloading modules in Python is hard :/


The additional thing here is that the import system isn?t the only cache at play either? pkg_resources builds a cache of installed packages at import time too and that has to get invalidated somehow as well. This took pip like 3 or 4 versions to get right because we tried to detect what version of pip was installed _after_ we installed all the versions in order to stop doing the outdated warning with ``pip install -U pip`` and I?m still not entirely convinced we?re not erroneously spitting out the warning in some cases.

The end result of all of this is unless you?re really really careful and design things just right and know exactly how not only yourself, but all your dependencies are written (and track how they change in different versions!) and also how things that use _you_ are going to import you and your dependencies? it?s basically playing whack a mole and is almost never worth the effort.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171022/24503f14/attachment.html>

From elvis.stansvik at orexplore.com  Sun Oct 22 03:52:02 2017
From: elvis.stansvik at orexplore.com (Elvis Stansvik)
Date: Sun, 22 Oct 2017 09:52:02 +0200
Subject: [Distutils] Announcement: Pip 10 is coming,
 and will move all internal APIs
In-Reply-To: <CACac1F_PkcHekc-2qkas5J6w0xUumjuVJCCMcCGXPF4Ft1zmSw@mail.gmail.com>
References: <CACac1F8phs-jvpsCKWUnxeuUr5rRmze6SCNsnw95JGFW-jSaZA@mail.gmail.com>
 <CAHrZfZDKVvXNR-H0BfC9CrKjzp==UqcemxtU9AB_0bCuZAcMww@mail.gmail.com>
 <CACac1F-Mv+-aWauJNH_fZFhyhn3ReaEnze24SxWoXAX-ww0WGg@mail.gmail.com>
 <CADiSq7df8CvGQB24UTb+uQxHunrsQHU97Nc6kmOSr+wY99Xg4Q@mail.gmail.com>
 <CACac1F_PkcHekc-2qkas5J6w0xUumjuVJCCMcCGXPF4Ft1zmSw@mail.gmail.com>
Message-ID: <CAE-7VJnuXf8zayaJG8JDCr+deJ-+9tDEDFe6=5e6up_RkyGcdw@mail.gmail.com>

2017-10-21 14:34 GMT+02:00 Paul Moore <p.f.moore at gmail.com>:
> On 21 October 2017 at 12:15, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> (Note: this is entirely speculative, and I have no idea how hard it would
>> be, so please read it as the question it's intended to be)
>
> No problem - I don't know myself how hard some of this would be, either ;-)
>
>> Do you know if there any key APIs (like installation) that could be turned
>> into wrappers around pip CLI calls in order to mitigate some of the impact?
>
> The obvious one is pip.main(). That's the one a lot of people use, but
> it's easily replaceable by a simple subprocess call. That's actually
> one of the reasons this was so frustrating to us - the bug reports we
> got were often from people doing things they didn't need to, that they
> could handle trivially via a supported approach.
>
> Otherwise, no. We've had little or no feedback on the tracker from
> people using more complex internals, so our working assumption has
> been there's very little that can't be handled via the CLI or existing
> packages. Feedback so far from this mail hints that maybe we were
> wrong, but it's still hard to know if it's one or two key projects, or
> a whole range of people that we've yet to hear from. I'm pretty sure,
> for example, that pipenv uses internals, either directly or via one of
> their dependencies, but we've not seen any feedback from them yet.

Another one that immediately comes to mind is pip-tools [1], which I
think is quite widely used.

But I just checked, and they filed a "check out how to deal with pip
10" issue two days ago [2] (I'm guessing in response to this thread).

Elvis

[1] https://github.com/jazzband/pip-tools
[2] https://github.com/jazzband/pip-tools/issues/580

>
>> The reason I ask is because it's unlikely anyone else is going to understand
>> how to emulate the previous functionality better than the pip devs would,
>> and if there's an API for those particular invocations, than they can be
>> covered directly by pip's test suite.
>
> Understood. We understand *how*, but don't know what is needed. One of
> the points of all this is to tease out such requirements. We'd hope to
> get them addressed by including them into *other* packages like
> distlib or packaging, but that's mostly just a matter of where the API
> goes, not what is needed. It does help us avoid tying fixes to pip's
> release cycle, though.
>
> It's also worth noting that the pip devs are possibly way too deep
> into how pip does things, rather than what the standards say. Getting
> others to implement libraries based on the published standards would
> help immensely to ensure that we're not falling into the
> "implementation defined" trap of the community being stuck with having
> to use pip because no-one else knows how to do what pip does.
>
>> Plus if there are previous API capabilities that *can't* currently be
>> emulated via the CLI, then the pip devs are the folks in the best position
>> to add the necessary CLI enhancements (such as the ones Noah asked about for
>> doing a more selective `pip list`).
>
> Oh, sure - apart from the aforementioned "pip.main()", most
> capabilities of the internal API are *not* easily emulated by the CLI.
>
> But 3rd party libraries are just as much an option. Remember, the
> issue here isn't so much about designing an API as about exposing (and
> therefore locking in stone) internal implementation details of how pip
> does things. And the pip devs are arguably in the *worst* position to
> handle that option, precisely because they know so much about how pip
> does things.
>
>> If that's an approach you might be amenable to, then a 10.0 pre-release
>> could be a good time to solicit PRs from folks that were using particular
>> APIs and would be prepared to invest time in defining comparable CLI
>> wrappers for them.
>
> Well, I get your point here, but the implication of this is that we
> have to design and build an API before we can release pip 10. Calling
> it CLI wrappers (and implementing it that way) doesn't do anything to
> ease the work needed on design, or the maintenance burden of providing
> stability. We're already getting a lot of pressure to release pip 10,
> and trying to do that would push it way further off.
>
> To be blunt, no-one on the pip team is really interested in trying to
> provide or support a stable API at this point in time. We have our
> hands full, as everyone is aware, and this is way down all our
> priorities. Community-submitted PRs would help, but there's still work
> in agreeing a design, reviewing those PRs, and maintaining them
> (exposing details from the internal APIs limits how much we can change
> those internals later, which is something we have to consider).
>
> What would be ideal would be for the community to build
> standards-based libraries that satisfied the needs that people
> currently handle by using pip internals. Heck, we could even consider
> vendoring such libraries inside of pip and saving ourselves a bunch of
> complexity ;-) The packaging, pkg_resources and distlib libraries are
> examples of this happening. But it doesn't seem to be a popular route,
> unfortunately. And those particular libraries are all maintained by
> PyPA members, so don't really count as examples of the community
> getting involved...
>
> The most likely alternative solution would be to revert the internal
> API moves. If that happened, I'd still prefer to lose pip.main, as
> that's the major pain point for us and the easiest for users to solve,
> but we could put the rest back. But it would be naive to assume that
> as a result people like Noah would be unaffected - they'd actually be
> worse off, as the internal APIs they use would probably break anyway
> (the installer/resolver stuff has been heavily modified for pip 10)
> but not all would do so in obvious ways.
>
> Beyond this, I don't know.
>
> Paul
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

From mlaing at post.harvard.edu  Sun Oct 22 19:10:40 2017
From: mlaing at post.harvard.edu (Michael Laing)
Date: Sun, 22 Oct 2017 16:10:40 -0700
Subject: [Distutils] Re PEP 508
Message-ID: <8C8E65AA-3ED1-469B-9ECA-402BC81C40DB@ml1.net>

I have a correction and a couple of possible improvements to the parsley grammar.

Is there a convenient way to submit them?

Thanks,

ml

From wes.turner at gmail.com  Sun Oct 22 20:22:45 2017
From: wes.turner at gmail.com (Wes Turner)
Date: Sun, 22 Oct 2017 20:22:45 -0400
Subject: [Distutils] Re PEP 508
In-Reply-To: <8C8E65AA-3ED1-469B-9ECA-402BC81C40DB@ml1.net>
References: <8C8E65AA-3ED1-469B-9ECA-402BC81C40DB@ml1.net>
Message-ID: <CACfEFw8ZEk+KyF=wy94JxrBoytLM4kg=drxs=5X40FyJCda7YA@mail.gmail.com>

https://docs.python.org/devguide/pullrequest.html

https://github.com/python/peps/issues/new

https://github.com/python/peps/blob/master/pep-0508.txt


On Sunday, October 22, 2017, Michael Laing <mlaing at post.harvard.edu> wrote:

> I have a correction and a couple of possible improvements to the parsley
> grammar.
>
> Is there a convenient way to submit them?
>
> Thanks,
>
> ml
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org <javascript:;>
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171022/fcffdf9b/attachment.html>

From ncoghlan at gmail.com  Sun Oct 22 23:47:25 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 23 Oct 2017 13:47:25 +1000
Subject: [Distutils] Re PEP 508
In-Reply-To: <8C8E65AA-3ED1-469B-9ECA-402BC81C40DB@ml1.net>
References: <8C8E65AA-3ED1-469B-9ECA-402BC81C40DB@ml1.net>
Message-ID: <CADiSq7fiwy2NUmc=S0mnsWnERzVD37pA=Xo4qLgAA=M_8Kq9_g@mail.gmail.com>

On 23 October 2017 at 09:10, Michael Laing <mlaing at post.harvard.edu> wrote:

> I have a correction and a couple of possible improvements to the parsley
> grammar.
>
> Is there a convenient way to submit them?
>

While we suffer from a fair bit of tracker sprawl which makes it hard to
work out where to report problems,
https://github.com/pypa/interoperability-peps/issues is still the best
option for possible bugs in the packaging specs themselves.

Any subsequent PRs would go to the main PEPs repo at
https://github.com/python/peps (we started the interoperability PEPs repo
back when the main PEPs repo was on hg.python.org, and hence didn't offer a
native PR workflow of its own)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171023/75f323be/attachment.html>

From guettliml at thomas-guettler.de  Tue Oct 24 06:34:28 2017
From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=)
Date: Tue, 24 Oct 2017 12:34:28 +0200
Subject: [Distutils] The software update framework
Message-ID: <5a167fa9-5c32-017c-4377-05b1941a6d0e@thomas-guettler.de>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171024/c033f019/attachment.html>

From ncoghlan at gmail.com  Tue Oct 24 08:56:44 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 24 Oct 2017 22:56:44 +1000
Subject: [Distutils] The software update framework
In-Reply-To: <5a167fa9-5c32-017c-4377-05b1941a6d0e@thomas-guettler.de>
References: <5a167fa9-5c32-017c-4377-05b1941a6d0e@thomas-guettler.de>
Message-ID: <CADiSq7crxWgpdnaZeuv51d43o+0x8qy10b5TELcsZ5FzB90XBQ@mail.gmail.com>

On 24 October 2017 at 20:34, Thomas G?ttler <guettliml at thomas-guettler.de>
wrote:

> I stumbled over this page: https://theupdateframework.github.io/
>
For folks that haven't read them before, note that TUF is also the basis
for the SSL/TLS independent package signing proposals in PEPs 458 & 480:

* https://www.python.org/dev/peps/pep-0458/ (PyPI -> end user signing)
* https://www.python.org/dev/peps/pep-0480/ (publisher -> end user signing)

Actually pursuing that idea is contingent on our being comfortable that the
related key management activities will be on a sustainable footing, though:
http://www.curiousefficiency.org/posts/2016/09/python-packaging-ecosystem.html#making-pypi-security-independent-of-ssl-tls

Cheers,
Nick.

P.S. TUF is in the news a bit this week, as both it and the related content
signing project, Notary, were just accepted as community projects hosted by
the Cloud Native Computing Foundation:
https://thenewstack.io/cncf-brings-security-cloud-native-stack-notary-tuf-adoption/

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171024/8857f0c2/attachment.html>

From tomuxiong at gmx.com  Tue Oct 24 09:01:14 2017
From: tomuxiong at gmx.com (Thomas Nyberg)
Date: Tue, 24 Oct 2017 15:01:14 +0200
Subject: [Distutils] How do native namespaces work? (And do they play well
 with setuptools?)
Message-ID: <eba0a97d-2db1-9894-fd4c-6f914afac46a@gmx.com>

Hello,

I previously posted this to python-list as well as reddit:

	https://mail.python.org/pipermail/python-list/2017-October/726935.html
	https://www.reddit.com/r/learnpython/comments/78emwr/how_to_native_namespaces_work_in_python_35/

So far (as pointed out in the reddit link I posted) it seems like maybe
I can't use native namespaces in conjunction with setuptools and must
instead use pip. Is this correct?

Here is my original message which contains all the details:

I'm trying to understand native namespaces. I'm currently using python
3.5 as packaged in debian 9. I've been following the instructions here:

	https://packaging.python.org/guides/packaging-namespace-packages/#native-namespace-packages

Those instructions link to the following example:

	https://github.com/pypa/sample-namespace-packages/tree/master/native

I presume I'm doing something silly, but I can't get that to work. I've
tried creating a virtual environment as:

--------------------------------------------------
$ python3 -m venv venv
$ source venv/bin/activate
--------------------------------------------------

_Note_: The virtual environment is active, but I'm stripping the (venv)
off the prompt below to shorten the lines.

And then I've gone into those repo folders and run their setup.py files
(e.g. for namespace a):

--------------------------------------------------
$ cd sample-namespace-packages/native/pkg_a/
$ python3 setup.py install
--------------------------------------------------

Then if I try to run their sample verification file, things fail:

--------------------------------------------------
$ cat sample-namespace-packages/verify_packages.py
from example_pkg import a
from example_pkg import b

print(a.name)
print(a.__path__)
print(b.name)
print(b.__path__)
$ python3 sample-namespace-packages/verify_packages.py
Traceback (most recent call last):
  File "sample-namespace-packages/verify_packages.py", line 1, in <module>
    from example_pkg import a
ImportError: No module named 'example_pkg'
--------------------------------------------------

Things seem to be installing:

--------------------------------------------------
$ ls venv/lib/python3.5/site-packages/
easy-install.pth           pkg_resources
easy_install.py            pkg_resources-0.0.0.dist-info
example_pkg_a-1-py3.5.egg  __pycache__
example_pkg_b-1-py3.5.egg  setuptools
pip                        setuptools-32.3.1.dist-info
pip-9.0.1.dist-info
--------------------------------------------------

Am I missing something totally obvious here? Does anyone here see what
I'm doing wrong? Thanks for any help!

Cheers,
Thomas

From alex.gronholm at nextday.fi  Wed Oct 25 10:54:22 2017
From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=)
Date: Wed, 25 Oct 2017 17:54:22 +0300
Subject: [Distutils] How do native namespaces work? (And do they play
 well with setuptools?)
In-Reply-To: <eba0a97d-2db1-9894-fd4c-6f914afac46a@gmx.com>
References: <eba0a97d-2db1-9894-fd4c-6f914afac46a@gmx.com>
Message-ID: <f22a5a69-9424-aa06-a8fa-ce72325fdfc6@nextday.fi>

What do you mean by "using"? You mean installing them (for which you 
should never use easy_install anyway, btw), or packaging them? The only 
limitation regarding packaging them is that you currently need to 
explicitly list them in your packaging configuration, as find_packages() 
does not yet support namespace packages.


Thomas Nyberg kirjoitti 24.10.2017 klo 16:01:
> Hello,
>
> I previously posted this to python-list as well as reddit:
>
> 	https://mail.python.org/pipermail/python-list/2017-October/726935.html
> 	https://www.reddit.com/r/learnpython/comments/78emwr/how_to_native_namespaces_work_in_python_35/
>
> So far (as pointed out in the reddit link I posted) it seems like maybe
> I can't use native namespaces in conjunction with setuptools and must
> instead use pip. Is this correct?
>
> Here is my original message which contains all the details:
>
> I'm trying to understand native namespaces. I'm currently using python
> 3.5 as packaged in debian 9. I've been following the instructions here:
>
> 	https://packaging.python.org/guides/packaging-namespace-packages/#native-namespace-packages
>
> Those instructions link to the following example:
>
> 	https://github.com/pypa/sample-namespace-packages/tree/master/native
>
> I presume I'm doing something silly, but I can't get that to work. I've
> tried creating a virtual environment as:
>
> --------------------------------------------------
> $ python3 -m venv venv
> $ source venv/bin/activate
> --------------------------------------------------
>
> _Note_: The virtual environment is active, but I'm stripping the (venv)
> off the prompt below to shorten the lines.
>
> And then I've gone into those repo folders and run their setup.py files
> (e.g. for namespace a):
>
> --------------------------------------------------
> $ cd sample-namespace-packages/native/pkg_a/
> $ python3 setup.py install
> --------------------------------------------------
>
> Then if I try to run their sample verification file, things fail:
>
> --------------------------------------------------
> $ cat sample-namespace-packages/verify_packages.py
> from example_pkg import a
> from example_pkg import b
>
> print(a.name)
> print(a.__path__)
> print(b.name)
> print(b.__path__)
> $ python3 sample-namespace-packages/verify_packages.py
> Traceback (most recent call last):
>    File "sample-namespace-packages/verify_packages.py", line 1, in <module>
>      from example_pkg import a
> ImportError: No module named 'example_pkg'
> --------------------------------------------------
>
> Things seem to be installing:
>
> --------------------------------------------------
> $ ls venv/lib/python3.5/site-packages/
> easy-install.pth           pkg_resources
> easy_install.py            pkg_resources-0.0.0.dist-info
> example_pkg_a-1-py3.5.egg  __pycache__
> example_pkg_b-1-py3.5.egg  setuptools
> pip                        setuptools-32.3.1.dist-info
> pip-9.0.1.dist-info
> --------------------------------------------------
>
> Am I missing something totally obvious here? Does anyone here see what
> I'm doing wrong? Thanks for any help!
>
> Cheers,
> Thomas
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


From tomuxiong at gmx.com  Wed Oct 25 15:05:23 2017
From: tomuxiong at gmx.com (Thomas Nyberg)
Date: Wed, 25 Oct 2017 21:05:23 +0200
Subject: [Distutils] How do native namespaces work? (And do they play
 well with setuptools?)
In-Reply-To: <f22a5a69-9424-aa06-a8fa-ce72325fdfc6@nextday.fi>
References: <eba0a97d-2db1-9894-fd4c-6f914afac46a@gmx.com>
 <f22a5a69-9424-aa06-a8fa-ce72325fdfc6@nextday.fi>
Message-ID: <3ddca944-6387-1394-8acd-89e57fb917e4@gmx.com>

On 10/25/2017 04:54 PM, Alex Gr?nholm wrote:
> What do you mean by "using"?

Are you referring to where I say "I'm currently using python 3.5 as
packaged in debian 9"? (I can't find the string "using" anywhere else.)
I just mean that that is the version of python I have installed.

>                              You mean installing them (for which you
> should never use easy_install anyway, btw), or packaging them? The only
> limitation regarding packaging them is that you currently need to
> explicitly list them in your packaging configuration, as find_packages()
> does not yet support namespace packages.

At this point I mean installing them using the `setup.py` script that is
in that repo (i.e. executing `python3 setup.py install`). I mean this
file:
https://github.com/pypa/sample-namespace-packages/blob/master/native/pkg_a/setup.py)
That file is using setuptools.

Maybe the best way to formulate my question is: Should the steps that I
put in the previous email result in error or not? I.e. is that a bug or
expected behavior.

Thanks for the help.

Cheers,
Thomas

From alex.gronholm at nextday.fi  Wed Oct 25 17:27:22 2017
From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=)
Date: Thu, 26 Oct 2017 00:27:22 +0300
Subject: [Distutils] How do native namespaces work? (And do they play
 well with setuptools?)
In-Reply-To: <3ddca944-6387-1394-8acd-89e57fb917e4@gmx.com>
References: <eba0a97d-2db1-9894-fd4c-6f914afac46a@gmx.com>
 <f22a5a69-9424-aa06-a8fa-ce72325fdfc6@nextday.fi>
 <3ddca944-6387-1394-8acd-89e57fb917e4@gmx.com>
Message-ID: <6fa406cd-0c8a-33f0-fa55-b39dec025bdc@nextday.fi>

Ok, I get you now. So, what you should be doing is using pip to install 
them ("pip install ."). Directly doing "setup.py install" will install 
.egg files which are a thing of the past. Do this and the verify script 
will work fine.

I hope this answers your question.


Thomas Nyberg kirjoitti 25.10.2017 klo 22:05:
> On 10/25/2017 04:54 PM, Alex Gr?nholm wrote:
>> What do you mean by "using"?
> Are you referring to where I say "I'm currently using python 3.5 as
> packaged in debian 9"? (I can't find the string "using" anywhere else.)
> I just mean that that is the version of python I have installed.
>
>>                               You mean installing them (for which you
>> should never use easy_install anyway, btw), or packaging them? The only
>> limitation regarding packaging them is that you currently need to
>> explicitly list them in your packaging configuration, as find_packages()
>> does not yet support namespace packages.
> At this point I mean installing them using the `setup.py` script that is
> in that repo (i.e. executing `python3 setup.py install`). I mean this
> file:
> https://github.com/pypa/sample-namespace-packages/blob/master/native/pkg_a/setup.py)
> That file is using setuptools.
>
> Maybe the best way to formulate my question is: Should the steps that I
> put in the previous email result in error or not? I.e. is that a bug or
> expected behavior.
>
> Thanks for the help.
>
> Cheers,
> Thomas
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


From ncoghlan at gmail.com  Wed Oct 25 23:05:30 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 26 Oct 2017 13:05:30 +1000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CADiSq7eGRc5tia9oRXhur05D9AbUWnftPVE_5vAXDjBdiT0qng@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <1508439284.1276939.1144539120.2C07E959@webmail.messagingengine.com>
 <0CA249EC-2F2D-46E5-97E4-C7613D309637@stufft.io>
 <1508440558.1349027.1144567248.70B76A6E@webmail.messagingengine.com>
 <329299D5-E3E2-4422-A1B9-615E434334CB@stufft.io>
 <1508442907.1358802.1144608536.3A6D9403@webmail.messagingengine.com>
 <6C066EB1-2DE2-4EC0-8F86-345103F52AAC@stufft.io>
 <FF51B1D1-9B54-4C32-908C-170A7AC569FB@stufft.io>
 <CADiSq7d2SY1X6mKwqFm_XwKnft3djhm=kU80JMJWPLU4qEyt=w@mail.gmail.com>
 <A9A13F49-62D5-4966-9BBB-A218B584BC5D@stufft.io>
 <CADiSq7fywTbL8-s8O__fiB1PrrEHWMpPD0HUuWR7NqVupKg=zg@mail.gmail.com>
 <98E4A9CC-8A72-46DB-949F-ABE3B7C1157A@stufft.io>
 <1508499110.3322597.1145281088.44F5B2AE@webmail.messagingengine.com>
 <5F9E5F93-7AF4-4D05-956F-A4367F89CD96@stufft.io>
 <1508500651.117367.1145301232.4A2422BF@webmail.messagingengine.com>
 <6BBFF1A1-F05D-496A-8BEB-FB65D7554823@stufft.io>
 <CADiSq7dmmoB_Hj_Q7ZurMJ1f5e1g=K4oAQQnx5xFucccxUyWhQ@mail.gmail.com>
 <5BD2E6C3-2D0C-45FC-A15D-66ACDBC6B33D@stufft.io>
 <1508503275.124767.1145341384.76E2EDA6@webmail.messagingengine.com>
 <F0F05ABC-788F-49AF-9905-E98F7FDDB988@stufft.io>
 <CADiSq7dE8ZxPZRj0RChEc_65kMpQNUzos_gN-hd6FBu_cLPQ_Q@mail.gmail.com>
 <A2AB937D-D8E0-44E2-8471-3E2D1F433929@stufft.io>
 <CADiSq7dMKvLjkxyx+NXkSkSYkk53qcWnpajxBiQg1co_gtm0MA@mail.gmail.com>
 <CACfEFw-0Njtu2Jm+AUSkycJN=+YQ1Xh7rn7Nnd658RzQ19LNpg@mail.gmail.com>
 <CADiSq7eGRc5tia9oRXhur05D9AbUWnftPVE_5vAXDjBdiT0qng@mail.gmail.com>
Message-ID: <CADiSq7dVuO0cEa7ri1+tj+ncZ-0g_o2R3wbHrxV4k-D=WqNm0w@mail.gmail.com>

On 21 October 2017 at 18:21, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 21 October 2017 at 18:04, Wes Turner <wes.turner at gmail.com> wrote:
> > On Saturday, October 21, 2017, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >> I'm also going to file an issue on the setuptools issue tracker to make
> sure Jason is aware of what we're doing, and get his explicit OK with the
> idea of making the format a PyPA interoperability specification (if he
> isn't, we'll demote Thomas's document to being a guide for tool developers
> aiming for pkg_resources interoperability).
> >
> > What are the URIs for this PR and issue?
>
> New setuptools issue: https://github.com/pypa/setuptools/issues/1179 (I
> hadn't filed it yet when I wrote the previous comment)
> Thomas's PR: https://github.com/pypa/python-packaging-user-guide/pull/390
>

With Jason's +1 on the setuptools issue, I've gone ahead and hit the merge
button on Thomas's PR:
https://github.com/pypa/python-packaging-user-guide/commit/34c37f0e66821127a8cbe59fa1f33dca0cf20d97

The spec is now available here
https://packaging.python.org/specifications/entry-points/, and
clarifications and corrections can be submitted as follow-up PRs (as for
other PyPA specifications).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171026/371f9e14/attachment.html>

From guettliml at thomas-guettler.de  Thu Oct 26 03:40:07 2017
From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=)
Date: Thu, 26 Oct 2017 09:40:07 +0200
Subject: [Distutils] The software update framework
In-Reply-To: <CAPA-EquKWYcUkEnnHLCjvAF0SXpyiv9Lgkyyep1uE3oYW_Z=-w@mail.gmail.com>
References: <5a167fa9-5c32-017c-4377-05b1941a6d0e@thomas-guettler.de>
 <CAPA-EquKWYcUkEnnHLCjvAF0SXpyiv9Lgkyyep1uE3oYW_Z=-w@mail.gmail.com>
Message-ID: <e621694a-da1a-d336-a1c0-1fb722d44a4b@thomas-guettler.de>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171026/f60f593f/attachment.html>

From njs at pobox.com  Thu Oct 26 03:57:12 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 26 Oct 2017 00:57:12 -0700
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CADiSq7fbu68be22=-FCzrO9-K-SRHmZ4vYhOsoT+N3UGQ6sW9A@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <CAPJVwBnVP_tMKz+ZQLQ9BbkymbrO3msm-zk3X7uyxHmuDWLSfw@mail.gmail.com>
 <CAG8k2+5Bcfi3QBwOsPyEKhMpNxHKwrSLJoCYing+XbUTm-wRPg@mail.gmail.com>
 <CADiSq7fbu68be22=-FCzrO9-K-SRHmZ4vYhOsoT+N3UGQ6sW9A@mail.gmail.com>
Message-ID: <CAPJVwB=QauirbT_VkC3dbBjiSkhQEjiL0tK=JGFh83FtAGhuZw@mail.gmail.com>

On Fri, Oct 20, 2017 at 11:59 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Yeah, here's the gist of what I had in mind regarding the malware problem
> (i.e. aiming to ensure we don't get all of setup.py's problems back again):
>
> - a package's own install hooks do *not* get called for that package

Doesn't that break the entry point caching use case that started this
whole discussion? When you first install the caching package, then it
has to immediately build the cache for the first time.

I don't really have the time or interest to dig into this (I know
there are legitimate use cases for entry points but I'm very wary of
any feature where package A starts doing something different because
package B was installed). But, I just wanted to throw out that I see
at least two reasons we might want to "bake in" the caching as part of
our PEPified metadata:

- if we do want to add "install hooks", then we need some way for a
package to declare it has an install hook and for pip-or-whoever to
find it. The natural way would be to use an entry point, which means
entry points are in some sense "more fundamental" than install hooks.

- in general, the only thing that can update an entry-point cache is
the package that's doing the install, at the time it runs. In
particular, consider an environment with some packages installed in
/usr, some in /usr/local, some in ~/.local/. Really you want one cache
in each location, and then to have dpkg/rpm responsible for updating
the /usr cache (this is something they're familiar with, it's
isomorphic to stuff like /etc/ld.so.cache), 'sudo pip' responsible for
updating the /usr/local cache, and 'pip --user' responsible for
updating the ~/.local/ cache. If we go the install hook route instead,
then when I do 'pip install --user entry_point_cacher' then there's no
way that it'll ever have the permissions to write to /usr, and maybe
not to /usr/local either depending on how you want to handle the
interaction between 'sudo pip' and ~/.local/ install hooks, so it
just... won't actually work as a caching tool. Similarly, it's
probably easier to convince conda to regenerate a single standard
entry point cache after installing a conda package, than it would be
to convince them to run generic wheel install hooks when not even
installing wheels.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From thomas at kluyver.me.uk  Thu Oct 26 04:33:45 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Thu, 26 Oct 2017 09:33:45 +0100
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CADiSq7fbu68be22=-FCzrO9-K-SRHmZ4vYhOsoT+N3UGQ6sW9A@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <CAPJVwBnVP_tMKz+ZQLQ9BbkymbrO3msm-zk3X7uyxHmuDWLSfw@mail.gmail.com>
 <CAG8k2+5Bcfi3QBwOsPyEKhMpNxHKwrSLJoCYing+XbUTm-wRPg@mail.gmail.com>
 <CADiSq7fbu68be22=-FCzrO9-K-SRHmZ4vYhOsoT+N3UGQ6sW9A@mail.gmail.com>
Message-ID: <1509006825.179130.1151509872.4F40FCCB@webmail.messagingengine.com>

On Sat, Oct 21, 2017, at 07:59 AM, Nick Coghlan wrote:
> Yeah, here's the gist of what I had in mind regarding the malware
> problem (i.e. aiming to ensure we don't get all of setup.py's problems
> back again):> 
> - a package's own install hooks do *not* get called for that package
> - hooks only run by default inside a virtualenv as a regular user
> - outside a virtualenv, the default is "hooks don't get run at all"

This one would make caching much less useful for me, because I install a
lot of stuff with 'pip install --user'.
I'm not really sure how useful this protection is. A malicious
package can shadow common module names and command names, so once
it's installed, it has an excellent chance of getting to run code,
even without hooks. And virtualenvs are not a security boundary -
malware installed in a virtualenv is just as bad as malware installed
with --user.
Moving away from running 'setup.py' to install stuff protects us against
packages doing silly things like running pip in a subprocess, but it
provides very little protection against deliberately malicious packages.
If we're going to do package install hooks, let's not cripple them by
trying to introduce security that doesn't really achieve much.
Nathaniel raises the point that it may be easier to convince other
package managers to regenerate an entry points cache than to call
arbitrary Python hooks on install. I guess the key question here is: how
many other use cases can we see for package install/uninstall hooks, and
how would those work with other packaging systems?
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171026/375b1938/attachment.html>

From ncoghlan at gmail.com  Thu Oct 26 08:15:16 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 26 Oct 2017 22:15:16 +1000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1509006825.179130.1151509872.4F40FCCB@webmail.messagingengine.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <CAPJVwBnVP_tMKz+ZQLQ9BbkymbrO3msm-zk3X7uyxHmuDWLSfw@mail.gmail.com>
 <CAG8k2+5Bcfi3QBwOsPyEKhMpNxHKwrSLJoCYing+XbUTm-wRPg@mail.gmail.com>
 <CADiSq7fbu68be22=-FCzrO9-K-SRHmZ4vYhOsoT+N3UGQ6sW9A@mail.gmail.com>
 <1509006825.179130.1151509872.4F40FCCB@webmail.messagingengine.com>
Message-ID: <CADiSq7eg7=TOq022LPOX6JXLGyv1tpfbvDmPdPROga0WHsOryw@mail.gmail.com>

On 26 October 2017 at 18:33, Thomas Kluyver <thomas at kluyver.me.uk> wrote:

> Nathaniel raises the point that it may be easier to convince other package
> managers to regenerate an entry points cache than to call arbitrary Python
> hooks on install.
>

At least for RPM, we have file triggers now, whereby system packages can
register a hook to say "Any time another package touches a file under <path
of interest> I want to know about it".

That means the exact semantics of any RPM integration would likely end up
just living in a file trigger, so it wouldn't matter to much whether that
trigger was "refresh these predefined caches" or "run any installed hooks
based on the defined Python level metadata".

However, I expect it would be much easier to define a "optionally export
data for caching in a more efficient key value store" API than it would be
to define an API for arbitrary pre-/post- [un]install hooks. In particular,
a caching API is much easier to *repair*, since the "source of truth"
remains the installation DB itself - the cache is just to speed up runtime
lookups.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171026/f4c96bbc/attachment.html>

From dholth at gmail.com  Thu Oct 26 10:57:34 2017
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 26 Oct 2017 14:57:34 +0000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CADiSq7eg7=TOq022LPOX6JXLGyv1tpfbvDmPdPROga0WHsOryw@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <CAPJVwBnVP_tMKz+ZQLQ9BbkymbrO3msm-zk3X7uyxHmuDWLSfw@mail.gmail.com>
 <CAG8k2+5Bcfi3QBwOsPyEKhMpNxHKwrSLJoCYing+XbUTm-wRPg@mail.gmail.com>
 <CADiSq7fbu68be22=-FCzrO9-K-SRHmZ4vYhOsoT+N3UGQ6sW9A@mail.gmail.com>
 <1509006825.179130.1151509872.4F40FCCB@webmail.messagingengine.com>
 <CADiSq7eg7=TOq022LPOX6JXLGyv1tpfbvDmPdPROga0WHsOryw@mail.gmail.com>
Message-ID: <CAG8k2+5jrvnknC17GOVBkOz=uCZCf2T492==zsJNHzd6qGnE8A@mail.gmail.com>

I agree. The "malware" problem is really a "how do I understand which hooks
run in each environment" problem. The hooks could slow down or confuse,
frustrate people in ways that were unrelated to any malicious intent.

The caching could just be a more efficient, lossless representation of the
*.dist/egg-info data model.
Would something as simple as a file per sys.path with the 'last modified by
installer' date be helpful? You could check those to determine whether your
cache was out of date.

Another option would be to try to investigate whether the per-sys-path
operations that 'import x' has to do anyway can be cached and shared with
pkg_resources?

On Thu, Oct 26, 2017 at 8:21 AM Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 26 October 2017 at 18:33, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
>
>> Nathaniel raises the point that it may be easier to convince other
>> package managers to regenerate an entry points cache than to call arbitrary
>> Python hooks on install.
>>
>
> At least for RPM, we have file triggers now, whereby system packages can
> register a hook to say "Any time another package touches a file under <path
> of interest> I want to know about it".
>
> That means the exact semantics of any RPM integration would likely end up
> just living in a file trigger, so it wouldn't matter to much whether that
> trigger was "refresh these predefined caches" or "run any installed hooks
> based on the defined Python level metadata".
>
> However, I expect it would be much easier to define a "optionally export
> data for caching in a more efficient key value store" API than it would be
> to define an API for arbitrary pre-/post- [un]install hooks. In particular,
> a caching API is much easier to *repair*, since the "source of truth"
> remains the installation DB itself - the cache is just to speed up runtime
> lookups.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171026/ee72f616/attachment.html>

From donald at stufft.io  Thu Oct 26 11:11:43 2017
From: donald at stufft.io (Donald Stufft)
Date: Thu, 26 Oct 2017 11:11:43 -0400
Subject: [Distutils] Disabling non HTTPS access to APIs on PyPI
Message-ID: <9C55AB34-E5F8-4355-B8F0-2896E6922EA4@stufft.io>

Historically PyPI was only available over either HTTP or unvalidated HTTPS, and over time we?ve been pushing more and more traffic onto HTTPS. In Warehouse the decision was made to *not* redirect ?API? URLs from HTTP to HTTPS, but to rather return an error accessing them from HTTP. This is because while logged in views have HSTS to ensure HTTPS in the browser (and with humans manually entering them into the URL bar regularly they are more error prone) APIs which are typically accessed by automated clients with an URL configured or hardcoded typically do not respect HSTS, so if you had a script that did ``curl http://pypi.python.org/simple/`` <http://pypi.python.org/simple/%60%60>, it would silently get redirects to https and appear to ?work?, but you wouldn?t get any of the security properties of TLS because an attacker would just intercept the request prior to the redirect happening.

Today I?ve backported this changed to the current production deployment of PyPI, which means that you can no longer access /simple/ and /packages/ over HTTP and you will have to directly go to HTTPS. For most people this should have no effect, because most tooling should be defaulting to HTTPS anyways, however if you?re using a significantly old version of tooling, it may still be defaulting to the HTTP url and will now stop functioning.

The recommended remediation is to upgrade your tooling to versions that support verified TLS connections and which default to the proper HTTPS URLs.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171026/fc67a7cd/attachment-0001.html>

From thomas at kluyver.me.uk  Thu Oct 26 11:45:59 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Thu, 26 Oct 2017 16:45:59 +0100
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CAG8k2+5jrvnknC17GOVBkOz=uCZCf2T492==zsJNHzd6qGnE8A@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <CAPJVwBnVP_tMKz+ZQLQ9BbkymbrO3msm-zk3X7uyxHmuDWLSfw@mail.gmail.com>
 <CAG8k2+5Bcfi3QBwOsPyEKhMpNxHKwrSLJoCYing+XbUTm-wRPg@mail.gmail.com>
 <CADiSq7fbu68be22=-FCzrO9-K-SRHmZ4vYhOsoT+N3UGQ6sW9A@mail.gmail.com>
 <1509006825.179130.1151509872.4F40FCCB@webmail.messagingengine.com>
 <CADiSq7eg7=TOq022LPOX6JXLGyv1tpfbvDmPdPROga0WHsOryw@mail.gmail.com>
 <CAG8k2+5jrvnknC17GOVBkOz=uCZCf2T492==zsJNHzd6qGnE8A@mail.gmail.com>
Message-ID: <1509032759.1288413.1151944200.22A79128@webmail.messagingengine.com>



On Thu, Oct 26, 2017, at 03:57 PM, Daniel Holth wrote:
> Would something as simple as a file per sys.path with the 'last
> modified by installer' date be helpful? You could check those to
> determine whether your cache was out of date.
I wonder if we could use the directory mtime for this? It's only really
useful if we can be confident that all installer tools will update it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171026/bd580791/attachment.html>

From nate at bx.psu.edu  Thu Oct 26 14:10:26 2017
From: nate at bx.psu.edu (Nate Coraor)
Date: Thu, 26 Oct 2017 14:10:26 -0400
Subject: [Distutils] Disabling non HTTPS access to APIs on PyPI
In-Reply-To: <9C55AB34-E5F8-4355-B8F0-2896E6922EA4@stufft.io>
References: <9C55AB34-E5F8-4355-B8F0-2896E6922EA4@stufft.io>
Message-ID: <CALT861HBnmh=CSzYZc0r+w2-BM5BqXF8+PMF2AeuD8Yu6dCsFQ@mail.gmail.com>

Heads up to anyone running pypiserver, this breaks the default redirection
from pypiserver to PyPI. There's an open issue[1] to fix it. In the
meantime, you can use the pypiserver option `--fallback-url
https://pypi.python.org/simple/` to work around it.

--nate

[1] https://github.com/pypiserver/pypiserver/issues/179

On Thu, Oct 26, 2017 at 11:11 AM, Donald Stufft <donald at stufft.io> wrote:

> Historically PyPI was only available over either HTTP or unvalidated
> HTTPS, and over time we?ve been pushing more and more traffic onto HTTPS.
> In Warehouse the decision was made to *not* redirect ?API? URLs from HTTP
> to HTTPS, but to rather return an error accessing them from HTTP. This is
> because while logged in views have HSTS to ensure HTTPS in the browser (and
> with humans manually entering them into the URL bar regularly they are more
> error prone) APIs which are typically accessed by automated clients with an
> URL configured or hardcoded typically do not respect HSTS, so if you had a
> script that did ``curl http://pypi.python.org/simple/``, it would
> silently get redirects to https and appear to ?work?, but you wouldn?t get
> any of the security properties of TLS because an attacker would just
> intercept the request prior to the redirect happening.
>
> Today I?ve backported this changed to the current production deployment of
> PyPI, which means that you can no longer access /simple/ and /packages/
> over HTTP and you will have to directly go to HTTPS. For most people this
> should have no effect, because most tooling should be defaulting to HTTPS
> anyways, however if you?re using a significantly old version of tooling, it
> may still be defaulting to the HTTP url and will now stop functioning.
>
> The recommended remediation is to upgrade your tooling to versions that
> support verified TLS connections and which default to the proper HTTPS URLs.
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171026/57c23a14/attachment.html>

From ncoghlan at gmail.com  Fri Oct 27 00:02:52 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 27 Oct 2017 14:02:52 +1000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <1509032759.1288413.1151944200.22A79128@webmail.messagingengine.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <CAPJVwBnVP_tMKz+ZQLQ9BbkymbrO3msm-zk3X7uyxHmuDWLSfw@mail.gmail.com>
 <CAG8k2+5Bcfi3QBwOsPyEKhMpNxHKwrSLJoCYing+XbUTm-wRPg@mail.gmail.com>
 <CADiSq7fbu68be22=-FCzrO9-K-SRHmZ4vYhOsoT+N3UGQ6sW9A@mail.gmail.com>
 <1509006825.179130.1151509872.4F40FCCB@webmail.messagingengine.com>
 <CADiSq7eg7=TOq022LPOX6JXLGyv1tpfbvDmPdPROga0WHsOryw@mail.gmail.com>
 <CAG8k2+5jrvnknC17GOVBkOz=uCZCf2T492==zsJNHzd6qGnE8A@mail.gmail.com>
 <1509032759.1288413.1151944200.22A79128@webmail.messagingengine.com>
Message-ID: <CADiSq7eEmTXRWNKAqH-2c-NQyS_kD3fE65prwSxqTOrhtfZLFw@mail.gmail.com>

On 27 October 2017 at 01:45, Thomas Kluyver <thomas at kluyver.me.uk> wrote:

>
>
> On Thu, Oct 26, 2017, at 03:57 PM, Daniel Holth wrote:
>
> Would something as simple as a file per sys.path with the 'last modified
> by installer' date be helpful? You could check those to determine whether
> your cache was out of date.
>
>
> I wonder if we could use the directory mtime for this? It's only really
> useful if we can be confident that all installer tools will update it.
>

There are lots of options for this, and one thing worth keeping in mind is
compatibility with the "monolithic system package" model, where the entire
preconfigured virtual environment gets archived, and then dropped into
place on the target system. In such cases, filesystem level mtimes may
change *without* the entry point cache actually being out of date.

In that context, it's worth keeping in mind what the actual goals of the
cache will be:

1. The entry point cache should ideally reflect the state of installed
components in a given execution environment at the time of access. If this
is not true, installing a component may require explicit cache
invalidation/rebuilding to get things back to a consistent state (similar
to the way a call to importlib.invalidate_caches() is needed to reliably
see filesystem changes)
2. Checking for available entry points in a given group should be
consistently cheap (ideally O(1)), rather than scaling with the number of
packages installed or the number of sys.path entries

Given those goals, there are a number of different points in time where the
cache can be generated, each with different trade-offs between how reliably
fresh the cache is, and how frequently you have to rebuild the cache.

Option 1: in-memory cache

* Pro: consistent with the way importlib caches work
* Pro: automatically adjusts to sys.path changes
* Pro: will likely be needed regardless to handle per-path-entry caches
with other methods
* Con: every process incurs at least 1 linear DB read
* Con: zero pay-off if you only query one entry point group
* Con: requires explicit invalidation to pick up filesystem changes (but
can hook into importlib.invalidate_caches())

Option 2: temporary (or persistent) per-user-session cache

* Pro: only the first query per path entry per user session incurs a linear
DB read
* Pro: given persistent cache dirs (e.g. XDG_CACHE_HOME, ~/.cache) even
that overhead can be avoided
* Pro: sys.path directory mtimes are sufficient for cache invalidation
(subject to filesystem timestamp granularity)
* Pro: zero elevated privileges needed (cache would be stored in a per-user
directory tree)
* Con: interprocess locking likely needed to avoid the "thundering herd"
cache update problem [1]
* Con: if a non-persistent storage location is used, zero benefit over an
in-memory cache for throwaway environments (e.g. container startup)
* Con: cost of the cache freshness check will still scale linearly with the
number of sys.path entries

Option 3: persistent per-path-entry cache

* Pro: assuming cache freshness means zero runtime queries incur a linear
DB read (cache creation becomes an install time cost)
* Con: if you don't assume cache freshness, you need option 1 or 2 anyway,
and the install time cache just speeds up that first linear read
* Con: filesystem access control requires either explicit cache refresh or
implicit metadata caching support in installers
* Con: sys.path directory mtimes are no longer sufficient for cache
invalidation (due to potential for directory relocation)
* Con: interprocess locking arguably still needed to avoid the "thundering
herd" cache update problem (just between installers rather than runtime
processes)

Given those trade-offs, I think it would probably make the most sense to
start out by exploring a combination of options 1 & 2, and then only
explore option 3 based on demonstrated performance problems with a
per-user-session caching model. My rationale for that is that even in an
image based "immutable infrastructure" deployment model, it's often
entirely feasible to preseed runtime caches as part of the build process,
and in cases where that *isn't* possible, you're likely also going to have
trouble generating per-path-entry caches.

Cheers,
Nick.

[1] https://en.wikipedia.org/wiki/Thundering_herd_problem

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171027/2534aa7c/attachment.html>

From ncoghlan at gmail.com  Fri Oct 27 00:23:11 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 27 Oct 2017 14:23:11 +1000
Subject: [Distutils] Fwd: [Python-Dev] PEP 561: Distributing and Packaging
 Type Information
In-Reply-To: <CAMzjp8FvOGZ3e=Cgo+qCpgv2TFgDVBu95KEUzQyPhEMpR=ODiA@mail.gmail.com>
References: <CAMzjp8FvOGZ3e=Cgo+qCpgv2TFgDVBu95KEUzQyPhEMpR=ODiA@mail.gmail.com>
Message-ID: <CADiSq7fE2Ea9_5a=jYYRjzqa7iWOCyJ0TqsaEcQBrrbtdn+K0A@mail.gmail.com>

Hi folks,

Forwarding this PEP from python-dev, since there's a small piece of it that
falls into distutils-sig's area of responsibility: claiming the "Typing"
key in the distribution metadata to allow projects to state whether or not
they natively provide type annotations.

This seems like a reasonable addition to me, and it doesn't have any
specific implications for packaging tools beyond the existing convention of
"ignore-but-preserve any metadata keys you don't understand".

Cheers,
Nick.

---------- Forwarded message ----------
From: Ethan Smith <ethan at ethanhs.me>
Date: 27 October 2017 at 08:42
Subject: [Python-Dev] PEP 561: Distributing and Packaging Type Information
To: python-dev at python.org


Hello all,

I have completed an implementation for PEP 561, and believe it is time to
share the PEP and implementation with python-dev

Python-ideas threads:
 * PEP 561: Distributing and Packaging Type Information
 * PEP 561 v2 - Packaging Static Type Information
 * PEP 561: Distributing Type Information V3

The live version is here: https://www.python.org/dev/peps/pep-0561/

As always, duplicated below.

Ethan Smith

---------------------------------------------------

PEP: 561
Title: Distributing and Packaging Type Information
Author: Ethan Smith <ethan at ethanhs.me>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 09-Sep-2017
Python-Version: 3.7
Post-History:


Abstract
========

PEP 484 introduced type hinting to Python, with goals of making typing
gradual and easy to adopt. Currently, typing information must be distributed
manually. This PEP provides a standardized means to package and distribute
type information and an ordering for type checkers to resolve modules and
collect this information for type checking using existing packaging
architecture.


Rationale
=========

Currently, package authors wish to distribute code that has
inline type information. However, there is no standard method to distribute
packages with inline type annotations or syntax that can simultaneously
be used at runtime and in type checking. Additionally, if one wished to
ship typing information privately the only method would be via setting
``MYPYPATH`` or the equivalent to manually point to stubs. If the package
can be released publicly, it can be added to typeshed [1]_. However, this
does not scale and becomes a burden on the maintainers of typeshed.
Additionally, it ties bugfixes to releases of the tool using typeshed.

PEP 484 has a brief section on distributing typing information. In this
section [2]_ the PEP recommends using ``shared/typehints/pythonX.Y/`` for
shipping stub files. However, manually adding a path to stub files for each
third party library does not scale. The simplest approach people have taken
is to add ``site-packages`` to their ``MYPYPATH``, but this causes type
checkers to fail on packages that are highly dynamic (e.g. sqlalchemy
and Django).


Specification
=============

There are several motivations and methods of supporting typing in a package.
This PEP recognizes three (3) types of packages that may be created:

1. The package maintainer would like to add type information inline.

2. The package maintainer would like to add type information via stubs.

3. A third party would like to share stub files for a package, but the
   maintainer does not want to include them in the source of the package.

This PEP aims to support these scenarios and make them simple to add to
packaging and deployment.

The two major parts of this specification are the packaging specifications
and the resolution order for resolving module type information. The packaging
spec is based on and extends PEP 345 metadata. The type checking spec is
meant to replace the ``shared/typehints/pythonX.Y/`` spec of PEP 484 [2]_.

New third party stub libraries are encouraged to distribute stubs via the
third party packaging proposed in this PEP in place of being added to
typeshed. Typeshed will remain in use, but if maintainers are found, third
party stubs in typeshed are encouraged to be split into their own package.

Packaging Type Information
--------------------------
In order to make packaging and distributing type information as simple and
easy as possible, the distribution of type information, and typed Python code
is done through existing packaging frameworks. This PEP adds a new item to the
``*.distinfo/METADATA`` file to contain metadata about a package's support for
typing. The new item is optional, but must have a name of ``Typed`` and have a
value of either ``inline`` or ``stubs``, if present.

Metadata Examples::

    Typed: inline
    Typed: stubs


Stub Only Packages
''''''''''''''''''

For package maintainers wishing to ship stub files containing all of their
type information, it is prefered that the ``*.pyi`` stubs are alongside the
corresponding ``*.py`` files. However, the stubs may be put in a sub-folder
of the Python sources, with the same name the ``*.py`` files are in. For
example, the ``flyingcircus`` package would have its stubs in the folder
``flyingcircus/flyingcircus/``. This path is chosen so that if stubs are
not found in ``flyingcircus/`` the type checker may treat the subdirectory as
a normal package. The normal resolution order of checking ``*.pyi`` before
``*.py`` will be maintained.

Third Party Stub Packages
'''''''''''''''''''''''''

Third parties seeking to distribute stub files are encouraged to contact the
maintainer of the package about distribution alongside the package. If the
maintainer does not wish to maintain or package stub files or type information
inline, then a "third party stub package" should be created. The structure is
similar, but slightly different from that of stub only packages. If the stubs
are for the library ``flyingcircus`` then the package should be named
``flyingcircus-stubs`` and the stub files should be put in a sub-directory
named ``flyingcircus``. This allows the stubs to be checked as if they were in
a regular package.

In addition, the third party stub package should indicate which version(s)
of the runtime package are supported by indicating the runtime package's
version(s) through the normal dependency data. For example, if there was a
stub package ``flyingcircus-stubs``, it can indicate the versions of the
runtime ``flyingcircus`` package supported through ``install_requires``
in distutils based tools, or the equivalent in other packaging tools.

Type Checker Module Resolution Order
------------------------------------

The following is the order that type checkers supporting this PEP should
resolve modules containing type information:

1. User code - the files the type checker is running on.

2. Stubs or Python source manually put in the beginning of the path. Type
   checkers should provide this to allow the user complete control of which
   stubs to use, and patch broken stubs/inline types from packages.

3. Third party stub packages - these packages can supersede the installed
   untyped packages. They can be found at ``pkg-stubs`` for package ``pkg``,
   however it is encouraged to check the package's metadata using packaging
   query APIs such as ``pkg_resources`` to assure that the package is meant
   for type checking, and is compatible with the installed version.

4. Inline packages - finally, if there is nothing overriding the installed
   package, and it opts into type checking.

5. Typeshed (if used) - Provides the stdlib types and several third party
   libraries

Type checkers that check a different Python version than the version they run
on must find the type information in the ``site-packages``/``dist-packages``
of that Python version. This can be queried e.g.
``pythonX.Y -c 'import site; print(site.getsitepackages())'``. It is
also recommended
that the type checker allow for the user to point to a particular Python
binary, in case it is not in the path.

To check if a package has opted into type checking, type checkers are
recommended to use the ``pkg_resources`` module to query the package
metadata. If the ``typed`` package metadata has ``None`` as its value, the
package has not opted into type checking, and the type checker should skip
that package.


Implementation
==============

A CPython branch with a modified distutils supporting the ``typed`` setup
keyword lives here: [impl]_. In addition, a sample package with inline types is
available [typed_pkg]_, as well as a sample package [pkg_checker]_
which reads the metadata
of installed packages and reports on their status as either not typed, inline
typed, or a stub package.


Acknowledgements
================

This PEP would not have been possible without the ideas, feedback, and support
of Ivan Levkivskyi, Jelle Zijlstra, Nick Coghlan, Daniel F Moisset, and
Guido van Rossum.


Version History
===============

* 2017-10-26

    * Added implementation references.
    * Added acknowledgements and version history.

* 2017-10-06

    * Rewritten to use .distinfo/METADATA over a distutils specific command.
    * Clarify versioning of third party stub packages.

* 2017-09-11

    * Added information about current solutions and typeshed.
    * Clarify rationale.


References
==========
.. [1] Typeshed (https://github.com/python/typeshed)

.. [2] PEP 484, Storing and Distributing Stub Files
   (https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-files)

.. [impl] CPython sample implementation
   (https://github.com/ethanhs/cpython/tree/typeddist)

.. [typed_pkg] Sample typed package
   (https://github.com/ethanhs/sample-typed-package)

.. [pkg_checker] Sample package checker
   (https://github.com/ethanhs/check_typedpkg)

Copyright
=========

This document has been placed in the public domain.




..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:


_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/
ncoghlan%40gmail.com




-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171027/301a91c5/attachment-0001.html>

From njs at pobox.com  Fri Oct 27 04:10:54 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 27 Oct 2017 01:10:54 -0700
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CADiSq7eEmTXRWNKAqH-2c-NQyS_kD3fE65prwSxqTOrhtfZLFw@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <CAPJVwBnVP_tMKz+ZQLQ9BbkymbrO3msm-zk3X7uyxHmuDWLSfw@mail.gmail.com>
 <CAG8k2+5Bcfi3QBwOsPyEKhMpNxHKwrSLJoCYing+XbUTm-wRPg@mail.gmail.com>
 <CADiSq7fbu68be22=-FCzrO9-K-SRHmZ4vYhOsoT+N3UGQ6sW9A@mail.gmail.com>
 <1509006825.179130.1151509872.4F40FCCB@webmail.messagingengine.com>
 <CADiSq7eg7=TOq022LPOX6JXLGyv1tpfbvDmPdPROga0WHsOryw@mail.gmail.com>
 <CAG8k2+5jrvnknC17GOVBkOz=uCZCf2T492==zsJNHzd6qGnE8A@mail.gmail.com>
 <1509032759.1288413.1151944200.22A79128@webmail.messagingengine.com>
 <CADiSq7eEmTXRWNKAqH-2c-NQyS_kD3fE65prwSxqTOrhtfZLFw@mail.gmail.com>
Message-ID: <CAPJVwBn7OWz0tzvqSKekyYCkm=GeXJp9OHRzpxNLMifhHBQXKg@mail.gmail.com>

On Thu, Oct 26, 2017 at 9:02 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Option 2: temporary (or persistent) per-user-session cache
>
> * Pro: only the first query per path entry per user session incurs a linear
> DB read
> * Pro: given persistent cache dirs (e.g. XDG_CACHE_HOME, ~/.cache) even that
> overhead can be avoided
> * Pro: sys.path directory mtimes are sufficient for cache invalidation
> (subject to filesystem timestamp granularity)

Timestamp granularity is a solvable problem. You just have to be
careful not to write out the cache unless the directory mtime is
sufficiently far in the past, like 10 seconds old, say. (This is an
old trick that VCSes use to make commands like 'git status'
fast-and-reliable.)

This does mean you can get in a weird state where if the directory
mtime somehow gets set to the future, then start time starts sucking
because caching goes away.

Note also that you'll want to explicitly write the observed directory
mtime to the cache file, rather than comparing it to the cache file's
mtime, to avoid the race condition where the directory gets modified
just after we scan it but before we write out the cache.

> * Pro: zero elevated privileges needed (cache would be stored in a per-user
> directory tree)
> * Con: interprocess locking likely needed to avoid the "thundering herd"
> cache update problem [1]

Interprocess filesystem locking is going to be far more painful than
any problem it might solve. Seriously. At least on Unix, the right
approach is to go ahead and regenerate the cache, and then atomically
write it to the given place, and if someone else overwrites it a few
milliseconds later then oh well.

I guess on Windows locking might be OK, given that it has no atomic
writes and less gratuitously broken filesystem locking. But you'd
still want to make sure you never block when acquiring the lock; if
the lock is already taken because someone else is in the middle of
updating the cache, then you need to fall back on doing a linear scan.
This is explicitly *not* avoiding the thundering herd problem, because
it's more important to avoid the "one process got stuck and now
everyone else freezes on startup waiting for it" problem.

> * Con: if a non-persistent storage location is used, zero benefit over an
> in-memory cache for throwaway environments (e.g. container startup)

You also have to be careful about whether you have a writeable storage
location at all, and if so whether you have the right permissions. (It
might be bad if 'sudo somescript.py' leaves me with root-owned cache
files in /home/njs/.cache/.)

Filesystems are just a barrel of fun.

> * Con: cost of the cache freshness check will still scale linearly with the
> number of sys.path entries
>
> Option 3: persistent per-path-entry cache
>
> * Pro: assuming cache freshness means zero runtime queries incur a linear DB
> read (cache creation becomes an install time cost)
> * Con: if you don't assume cache freshness, you need option 1 or 2 anyway,
> and the install time cache just speeds up that first linear read
> * Con: filesystem access control requires either explicit cache refresh or
> implicit metadata caching support in installers
> * Con: sys.path directory mtimes are no longer sufficient for cache
> invalidation (due to potential for directory relocation)

Not sure what problem you're thinking of here? In this model we
wouldn't be using mtimes for cache invalidation anyway, because it'd
be the responsibility of those modifying the directory to update the
cache. And if you rename a whole directory, that doesn't affect its
mtime anyway?

> * Con: interprocess locking arguably still needed to avoid the "thundering
> herd" cache update problem (just between installers rather than runtime
> processes)

If two installers are trying to rearrange the same directory at the
same time then they can conflict in lots of ways. For the most part
people get away with it because doing multiple 'pip install' runs in
parallel is generally considered a Bad Idea and unlikely to happen by
accident; and if it is a problem then we should add locking anyway
(like dpkg and rpm already do), regardless of the cache update part.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From ncoghlan at gmail.com  Fri Oct 27 08:34:12 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 27 Oct 2017 22:34:12 +1000
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CAPJVwBn7OWz0tzvqSKekyYCkm=GeXJp9OHRzpxNLMifhHBQXKg@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <CAPJVwBnVP_tMKz+ZQLQ9BbkymbrO3msm-zk3X7uyxHmuDWLSfw@mail.gmail.com>
 <CAG8k2+5Bcfi3QBwOsPyEKhMpNxHKwrSLJoCYing+XbUTm-wRPg@mail.gmail.com>
 <CADiSq7fbu68be22=-FCzrO9-K-SRHmZ4vYhOsoT+N3UGQ6sW9A@mail.gmail.com>
 <1509006825.179130.1151509872.4F40FCCB@webmail.messagingengine.com>
 <CADiSq7eg7=TOq022LPOX6JXLGyv1tpfbvDmPdPROga0WHsOryw@mail.gmail.com>
 <CAG8k2+5jrvnknC17GOVBkOz=uCZCf2T492==zsJNHzd6qGnE8A@mail.gmail.com>
 <1509032759.1288413.1151944200.22A79128@webmail.messagingengine.com>
 <CADiSq7eEmTXRWNKAqH-2c-NQyS_kD3fE65prwSxqTOrhtfZLFw@mail.gmail.com>
 <CAPJVwBn7OWz0tzvqSKekyYCkm=GeXJp9OHRzpxNLMifhHBQXKg@mail.gmail.com>
Message-ID: <CADiSq7ekxAKdumV0V8Rsfg88PaaeeMxM8aKPNSkHP04iQcMWHw@mail.gmail.com>

On 27 October 2017 at 18:10, Nathaniel Smith <njs at pobox.com> wrote:

> On Thu, Oct 26, 2017 at 9:02 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > Option 2: temporary (or persistent) per-user-session cache
> >
> > * Pro: only the first query per path entry per user session incurs a
> linear
> > DB read
> > * Pro: given persistent cache dirs (e.g. XDG_CACHE_HOME, ~/.cache) even
> that
> > overhead can be avoided
> > * Pro: sys.path directory mtimes are sufficient for cache invalidation
> > (subject to filesystem timestamp granularity)
>
> Timestamp granularity is a solvable problem. You just have to be
> careful not to write out the cache unless the directory mtime is
> sufficiently far in the past, like 10 seconds old, say. (This is an
> old trick that VCSes use to make commands like 'git status'
> fast-and-reliable.)
>

Yeah, we just recently fixed a bug related to that in pyc file caching (If
you managed to modify and reload a source file multiple times in the same
second we could end up missing the later edits. The fix was to check the
source timestamp didn't match the current timestamp before actually
updating the cached copy on the filesystem)


> This does mean you can get in a weird state where if the directory
> mtime somehow gets set to the future, then start time starts sucking
> because caching goes away.
>

For pyc files, we're able to avoid that by looking for cache
*inconsistency* without making any assumptions about which direction time
moves - as long as the source timestamp recorded in the file pyc doesn't
match the source file's mtime, we'll refresh the cache.

This is necessary to cope with things like version controlled directories,
where directory mtimes can easily go backwards because you switched
branches or reverted to an earlier version.


>
> Note also that you'll want to explicitly write the observed directory
> mtime to the cache file, rather than comparing it to the cache file's
> mtime, to avoid the race condition where the directory gets modified
> just after we scan it but before we write out the cache.
>
> > * Pro: zero elevated privileges needed (cache would be stored in a
> per-user
> > directory tree)
> > * Con: interprocess locking likely needed to avoid the "thundering herd"
> > cache update problem [1]
>
> Interprocess filesystem locking is going to be far more painful than
> any problem it might solve. Seriously. At least on Unix, the right
> approach is to go ahead and regenerate the cache, and then atomically
> write it to the given place, and if someone else overwrites it a few
> milliseconds later then oh well.
>

Aye, limiting the handling for this to the use of atomic writes is likely
an entirely reasonable approach to take.


> I guess on Windows locking might be OK, given that it has no atomic
> writes and less gratuitously broken filesystem locking.


The os module has atomic write support on Windows in 3.x now:
https://docs.python.org/3/library/os.html#os.replace

So the only problematic case is 2.7 on WIndows, and for that Christian
Heimes backported pyosreplace here: https://pypi.org/project/pyosreplace/

(The "may be non-atomic" case is the same situation where it will fail
outright on POSIX systems: when you're attempting to do the rename across
filesystems. If you stay within the same directory, which you want to do
anyway for permissions inheritance and automatic file labeling, it's
atomic).

But you'd
> still want to make sure you never block when acquiring the lock; if
> the lock is already taken because someone else is in the middle of
> updating the cache, then you need to fall back on doing a linear scan.
> This is explicitly *not* avoiding the thundering herd problem, because
> it's more important to avoid the "one process got stuck and now
> everyone else freezes on startup waiting for it" problem.
>

Fair point.


> > * Con: if a non-persistent storage location is used, zero benefit over an
> > in-memory cache for throwaway environments (e.g. container startup)
>
> You also have to be careful about whether you have a writeable storage
> location at all, and if so whether you have the right permissions. (It
> might be bad if 'sudo somescript.py' leaves me with root-owned cache
> files in /home/njs/.cache/.)
>
> Filesystems are just a barrel of fun.
>

C'mon, who doesn't enjoy debugging SELinux file labeling problems arising
from mounting symlinked host directories into Docker containers running as
root internally? :)


>
> > * Con: cost of the cache freshness check will still scale linearly with
> the
> > number of sys.path entries
> >
> > Option 3: persistent per-path-entry cache
> >
> > * Pro: assuming cache freshness means zero runtime queries incur a
> linear DB
> > read (cache creation becomes an install time cost)
> > * Con: if you don't assume cache freshness, you need option 1 or 2
> anyway,
> > and the install time cache just speeds up that first linear read
> > * Con: filesystem access control requires either explicit cache refresh
> or
> > implicit metadata caching support in installers
> > * Con: sys.path directory mtimes are no longer sufficient for cache
> > invalidation (due to potential for directory relocation)
>
> Not sure what problem you're thinking of here? In this model we
> wouldn't be using mtimes for cache invalidation anyway, because it'd
> be the responsibility of those modifying the directory to update the
> cache. And if you rename a whole directory, that doesn't affect its
> mtime anyway?
>

Your second sentence is what I meant - whether the cache is still valid or
not is less about the mtime, and more about what other actions have been
performed. (It's much closer to the locate/updatedb model, where the
runtime part just assumes the cache is valid, and it's somebody else's
problem to ensure that assumption is reasonably valid)


> > * Con: interprocess locking arguably still needed to avoid the
> "thundering
> > herd" cache update problem (just between installers rather than runtime
> > processes)
>
> If two installers are trying to rearrange the same directory at the
> same time then they can conflict in lots of ways. For the most part
> people get away with it because doing multiple 'pip install' runs in
> parallel is generally considered a Bad Idea and unlikely to happen by
> accident; and if it is a problem then we should add locking anyway
> (like dpkg and rpm already do), regardless of the cache update part.
>

Another fair point.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171027/4bfa462f/attachment-0001.html>

From erik.m.bray at gmail.com  Fri Oct 27 11:15:47 2017
From: erik.m.bray at gmail.com (Erik Bray)
Date: Fri, 27 Oct 2017 17:15:47 +0200
Subject: [Distutils] Disabling non HTTPS access to APIs on PyPI
In-Reply-To: <9C55AB34-E5F8-4355-B8F0-2896E6922EA4@stufft.io>
References: <9C55AB34-E5F8-4355-B8F0-2896E6922EA4@stufft.io>
Message-ID: <CAOTD34Ynij8c+ROzMT2o4MArE2feHwOvoeu_fph89G0rMdt_nQ@mail.gmail.com>

On Thu, Oct 26, 2017 at 5:11 PM, Donald Stufft <donald at stufft.io> wrote:
> Historically PyPI was only available over either HTTP or unvalidated HTTPS,
> and over time we?ve been pushing more and more traffic onto HTTPS. In
> Warehouse the decision was made to *not* redirect ?API? URLs from HTTP to
> HTTPS, but to rather return an error accessing them from HTTP. This is
> because while logged in views have HSTS to ensure HTTPS in the browser (and
> with humans manually entering them into the URL bar regularly they are more
> error prone) APIs which are typically accessed by automated clients with an
> URL configured or hardcoded typically do not respect HSTS, so if you had a
> script that did ``curl http://pypi.python.org/simple/``, it would silently
> get redirects to https and appear to ?work?, but you wouldn?t get any of the
> security properties of TLS because an attacker would just intercept the
> request prior to the redirect happening.
>
> Today I?ve backported this changed to the current production deployment of
> PyPI, which means that you can no longer access /simple/ and /packages/ over
> HTTP and you will have to directly go to HTTPS. For most people this should
> have no effect, because most tooling should be defaulting to HTTPS anyways,
> however if you?re using a significantly old version of tooling, it may still
> be defaulting to the HTTP url and will now stop functioning.
>
> The recommended remediation is to upgrade your tooling to versions that
> support verified TLS connections and which default to the proper HTTPS URLs.

+1

This will probably (unfortunately) break some things for some people,
which is worrying.  But it is the right thing to do and good advice in
general.

From tritium-list at sdamon.com  Fri Oct 27 14:24:29 2017
From: tritium-list at sdamon.com (Alex Walters)
Date: Fri, 27 Oct 2017 14:24:29 -0400
Subject: [Distutils] Disabling non HTTPS access to APIs on PyPI
In-Reply-To: <CAOTD34Ynij8c+ROzMT2o4MArE2feHwOvoeu_fph89G0rMdt_nQ@mail.gmail.com>
References: <9C55AB34-E5F8-4355-B8F0-2896E6922EA4@stufft.io>
 <CAOTD34Ynij8c+ROzMT2o4MArE2feHwOvoeu_fph89G0rMdt_nQ@mail.gmail.com>
Message-ID: <12e701d34f50$d4dd3080$7e979180$@sdamon.com>



> -----Original Message-----
> From: Distutils-SIG [mailto:distutils-sig-bounces+tritium-
> list=sdamon.com at python.org] On Behalf Of Erik Bray
> Sent: Friday, October 27, 2017 11:16 AM
> To: Distutils <distutils-sig at python.org>
> Subject: Re: [Distutils] Disabling non HTTPS access to APIs on PyPI
> 
> On Thu, Oct 26, 2017 at 5:11 PM, Donald Stufft <donald at stufft.io> wrote:
> > Historically PyPI was only available over either HTTP or unvalidated HTTPS,
> > and over time we?ve been pushing more and more traffic onto HTTPS. In
> > Warehouse the decision was made to *not* redirect ?API? URLs from HTTP
> to
> > HTTPS, but to rather return an error accessing them from HTTP. This is
> > because while logged in views have HSTS to ensure HTTPS in the browser
> (and
> > with humans manually entering them into the URL bar regularly they are
> more
> > error prone) APIs which are typically accessed by automated clients with an
> > URL configured or hardcoded typically do not respect HSTS, so if you had a
> > script that did ``curl http://pypi.python.org/simple/``, it would silently
> > get redirects to https and appear to ?work?, but you wouldn?t get any of
> the
> > security properties of TLS because an attacker would just intercept the
> > request prior to the redirect happening.
> >
> > Today I?ve backported this changed to the current production deployment
> of
> > PyPI, which means that you can no longer access /simple/ and /packages/
> over
> > HTTP and you will have to directly go to HTTPS. For most people this should
> > have no effect, because most tooling should be defaulting to HTTPS
> anyways,
> > however if you?re using a significantly old version of tooling, it may still
> > be defaulting to the HTTP url and will now stop functioning.
> >
> > The recommended remediation is to upgrade your tooling to versions that
> > support verified TLS connections and which default to the proper HTTPS
> URLs.
> 
> +1
> 
> This will probably (unfortunately) break some things for some people,
> which is worrying.  But it is the right thing to do and good advice in
> general.

Might want to post a message on the front page of pypi.python.org when this becomes effective (if it's not there already.)

> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


From alex.hha at gmail.com  Fri Oct 27 12:54:27 2017
From: alex.hha at gmail.com (Alex Domoradov)
Date: Fri, 27 Oct 2017 19:54:27 +0300
Subject: [Distutils] Disabling non HTTPS access to APIs on PyPI
Message-ID: <CAK90gp6mU__+6HopzL+=8RxogkaztPh1fdoE53=Y42rnBTWrjA@mail.gmail.com>

> This will probably (unfortunately) break some things for some people,
> which is worrying.  But it is the right thing to do and good advice in
> general.

Did I miss something, but I can't install some package. I use docker
and get the following

$ cat Dockerfile
FROM ubuntu:16.04

RUN apt-get update
RUN apt-get install -y python-pip

RUN pip install --upgrade pip

$ docker run -it --rm pip bash

root at a633a43c5df6:/# pip -V
pip 9.0.1 from /usr/local/lib/python2.7/dist-packages (python 2.7)

root at a633a43c5df6:/# pip install google-appengine
Collecting google-appengine
  Downloading google-appengine-1.5.1.tar.gz (897kB)
    100% |################################| 901kB 745kB/s
    Complete output from command python setup.py egg_info:
    Downloading
http://pypi.python.org/packages/source/d/distribute/distribute-0.6.14.tar.gz
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-hd5GWR/google-appengine/setup.py", line 3,
in <module>
        ez_setup.use_setuptools()
      File "/usr/local/lib/python2.7/dist-packages/ez_setup.py", line
145, in use_setuptools
        return _do_download(version, download_base, to_dir, download_delay)
      File "/usr/local/lib/python2.7/dist-packages/ez_setup.py", line
124, in _do_download
        to_dir, download_delay)
      File "/usr/local/lib/python2.7/dist-packages/ez_setup.py", line
193, in download_setuptools
        src = urlopen(url)
      File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
        return opener.open(url, data, timeout)
      File "/usr/lib/python2.7/urllib2.py", line 435, in open
        response = meth(req, response)
      File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
        'http', request, response, code, msg, hdrs)
      File "/usr/lib/python2.7/urllib2.py", line 473, in error
        return self._call_chain(*args)
      File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
        result = func(*args)
      File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
        raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
*    urllib2.HTTPError: HTTP Error 403: SSL is required*

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in
/tmp/pip-build-hd5GWR/google-appengine/

Do I need to make some additional upgrades? I just thought that pip is
the only one tool that I have to upgrade
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171027/d0050c8b/attachment.html>

From njs at vorpus.org  Fri Oct 27 15:22:37 2017
From: njs at vorpus.org (Nathaniel Smith)
Date: Fri, 27 Oct 2017 12:22:37 -0700
Subject: [Distutils] Disabling non HTTPS access to APIs on PyPI
In-Reply-To: <CAPJVwB=JvFyzDA2r77BxSjCMhURHouHKSXW1GmPRjs=gzdQxAA@mail.gmail.com>
References: <CAK90gp6mU__+6HopzL+=8RxogkaztPh1fdoE53=Y42rnBTWrjA@mail.gmail.com>
 <CAPJVwB=JvFyzDA2r77BxSjCMhURHouHKSXW1GmPRjs=gzdQxAA@mail.gmail.com>
Message-ID: <CAPJVwBmvZODgFAwBC+UG+jnLWLNDnEnSG6B_tLmq-E-CmB1ZbQ@mail.gmail.com>

On Oct 27, 2017 11:49, "Alex Domoradov" <alex.hha at gmail.com> wrote:

RUN pip install --upgrade pip

Try upgrading setuptools here too.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171027/4c0cb0fd/attachment.html>

From leorochael at gmail.com  Fri Oct 27 16:05:17 2017
From: leorochael at gmail.com (Leonardo Rochael Almeida)
Date: Fri, 27 Oct 2017 18:05:17 -0200
Subject: [Distutils] Disabling non HTTPS access to APIs on PyPI
In-Reply-To: <CAPJVwBmvZODgFAwBC+UG+jnLWLNDnEnSG6B_tLmq-E-CmB1ZbQ@mail.gmail.com>
References: <CAK90gp6mU__+6HopzL+=8RxogkaztPh1fdoE53=Y42rnBTWrjA@mail.gmail.com>
 <CAPJVwB=JvFyzDA2r77BxSjCMhURHouHKSXW1GmPRjs=gzdQxAA@mail.gmail.com>
 <CAPJVwBmvZODgFAwBC+UG+jnLWLNDnEnSG6B_tLmq-E-CmB1ZbQ@mail.gmail.com>
Message-ID: <CAMxX_oMstS2vHCre_pE7TT-5Zj98bdetnxzNJ4344GS1rBgQcg@mail.gmail.com>

By the way, some pages on pypi itself are pointing to http:

See, e.g. the "Download URL:" link here:

https://pypi.python.org/pypi/google-appengine

On 27 October 2017 at 17:22, Nathaniel Smith <njs at vorpus.org> wrote:

> On Oct 27, 2017 11:49, "Alex Domoradov" <alex.hha at gmail.com> wrote:
>
> RUN pip install --upgrade pip
>
> Try upgrading setuptools here too.
>
> -n
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171027/fcd85e64/attachment-0001.html>

From p.f.moore at gmail.com  Fri Oct 27 16:17:37 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 27 Oct 2017 21:17:37 +0100
Subject: [Distutils] Disabling non HTTPS access to APIs on PyPI
In-Reply-To: <CAMxX_oMstS2vHCre_pE7TT-5Zj98bdetnxzNJ4344GS1rBgQcg@mail.gmail.com>
References: <CAK90gp6mU__+6HopzL+=8RxogkaztPh1fdoE53=Y42rnBTWrjA@mail.gmail.com>
 <CAPJVwB=JvFyzDA2r77BxSjCMhURHouHKSXW1GmPRjs=gzdQxAA@mail.gmail.com>
 <CAPJVwBmvZODgFAwBC+UG+jnLWLNDnEnSG6B_tLmq-E-CmB1ZbQ@mail.gmail.com>
 <CAMxX_oMstS2vHCre_pE7TT-5Zj98bdetnxzNJ4344GS1rBgQcg@mail.gmail.com>
Message-ID: <CACac1F-Mrja6dhLsQKuwA-ZZwzokRtyxq5icE4oxRHgapjpuPw@mail.gmail.com>

On 27 October 2017 at 21:05, Leonardo Rochael Almeida
<leorochael at gmail.com> wrote:
> By the way, some pages on pypi itself are pointing to http:
>
> See, e.g. the "Download URL:" link here:
>
> https://pypi.python.org/pypi/google-appengine

That's the project itself doing that, I believe (it's metadata from
setup.py) so you'd need to report it to them.

Paul

From p.f.moore at gmail.com  Fri Oct 27 17:41:44 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 27 Oct 2017 22:41:44 +0100
Subject: [Distutils] Disabling non HTTPS access to APIs on PyPI
In-Reply-To: <CAK90gp5zFsvYPEcRyf7sa18+qXADqOayH2ey_n28ou-uMpJJ0Q@mail.gmail.com>
References: <CAK90gp6mU__+6HopzL+=8RxogkaztPh1fdoE53=Y42rnBTWrjA@mail.gmail.com>
 <CAPJVwB=JvFyzDA2r77BxSjCMhURHouHKSXW1GmPRjs=gzdQxAA@mail.gmail.com>
 <CAPJVwBmvZODgFAwBC+UG+jnLWLNDnEnSG6B_tLmq-E-CmB1ZbQ@mail.gmail.com>
 <CAMxX_oMstS2vHCre_pE7TT-5Zj98bdetnxzNJ4344GS1rBgQcg@mail.gmail.com>
 <CACac1F-Mrja6dhLsQKuwA-ZZwzokRtyxq5icE4oxRHgapjpuPw@mail.gmail.com>
 <CAK90gp5zFsvYPEcRyf7sa18+qXADqOayH2ey_n28ou-uMpJJ0Q@mail.gmail.com>
Message-ID: <CACac1F-0KRhA1y25-5ap1SvyrsWcxUkUkL_1nVaxzLRk9mVF1A@mail.gmail.com>

On 27 October 2017 at 22:22, Alex Domoradov <alex.hha at gmail.com> wrote:
> I got it. And what I should do with old system? For e.g. we still use ubuntu
> 12.04. Is there any way to upgrade pip/setuptools?

Well, if Ubuntu aren't offering an upgrade, you can do

pip install -i https://pypi.python.org/simple/ ...

Or install your own copy of pip/setuptools, I guess (get-pip --user,
see https://pip.pypa.io/en/stable/installing/#installing-with-get-pip-py).
Paul

From njs at pobox.com  Fri Oct 27 17:56:15 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 27 Oct 2017 14:56:15 -0700
Subject: [Distutils] Entry points: specifying and caching
In-Reply-To: <CADiSq7ekxAKdumV0V8Rsfg88PaaeeMxM8aKPNSkHP04iQcMWHw@mail.gmail.com>
References: <1508338320.2029237.1142961520.46FEDFAA@webmail.messagingengine.com>
 <63B0944B-5389-481B-A4F6-CDC528019685@stufft.io>
 <1508429683.2435957.1144376800.6FC6F0E6@webmail.messagingengine.com>
 <90A0BB1E-0C20-496A-9220-B68624CA975F@stufft.io>
 <CAPJVwBnVP_tMKz+ZQLQ9BbkymbrO3msm-zk3X7uyxHmuDWLSfw@mail.gmail.com>
 <CAG8k2+5Bcfi3QBwOsPyEKhMpNxHKwrSLJoCYing+XbUTm-wRPg@mail.gmail.com>
 <CADiSq7fbu68be22=-FCzrO9-K-SRHmZ4vYhOsoT+N3UGQ6sW9A@mail.gmail.com>
 <1509006825.179130.1151509872.4F40FCCB@webmail.messagingengine.com>
 <CADiSq7eg7=TOq022LPOX6JXLGyv1tpfbvDmPdPROga0WHsOryw@mail.gmail.com>
 <CAG8k2+5jrvnknC17GOVBkOz=uCZCf2T492==zsJNHzd6qGnE8A@mail.gmail.com>
 <1509032759.1288413.1151944200.22A79128@webmail.messagingengine.com>
 <CADiSq7eEmTXRWNKAqH-2c-NQyS_kD3fE65prwSxqTOrhtfZLFw@mail.gmail.com>
 <CAPJVwBn7OWz0tzvqSKekyYCkm=GeXJp9OHRzpxNLMifhHBQXKg@mail.gmail.com>
 <CADiSq7ekxAKdumV0V8Rsfg88PaaeeMxM8aKPNSkHP04iQcMWHw@mail.gmail.com>
Message-ID: <CAPJVwB=goVj_oXhi19sFz24irNj9ZSvR9jKXYZ7XgYyLCVhUKA@mail.gmail.com>

On Fri, Oct 27, 2017 at 5:34 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 27 October 2017 at 18:10, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> On Thu, Oct 26, 2017 at 9:02 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> > Option 2: temporary (or persistent) per-user-session cache
>> >
>> > * Pro: only the first query per path entry per user session incurs a
>> > linear
>> > DB read
>> > * Pro: given persistent cache dirs (e.g. XDG_CACHE_HOME, ~/.cache) even
>> > that
>> > overhead can be avoided
>> > * Pro: sys.path directory mtimes are sufficient for cache invalidation
>> > (subject to filesystem timestamp granularity)
>>
>> Timestamp granularity is a solvable problem. You just have to be
>> careful not to write out the cache unless the directory mtime is
>> sufficiently far in the past, like 10 seconds old, say. (This is an
>> old trick that VCSes use to make commands like 'git status'
>> fast-and-reliable.)
>
>
> Yeah, we just recently fixed a bug related to that in pyc file caching (If
> you managed to modify and reload a source file multiple times in the same
> second we could end up missing the later edits. The fix was to check the
> source timestamp didn't match the current timestamp before actually updating
> the cached copy on the filesystem)
>
>>
>> This does mean you can get in a weird state where if the directory
>> mtime somehow gets set to the future, then start time starts sucking
>> because caching goes away.
>
>
> For pyc files, we're able to avoid that by looking for cache *inconsistency*
> without making any assumptions about which direction time moves - as long as
> the source timestamp recorded in the file pyc doesn't match the source
> file's mtime, we'll refresh the cache.
>
> This is necessary to cope with things like version controlled directories,
> where directory mtimes can easily go backwards because you switched branches
> or reverted to an earlier version.

Yeah, this is a good idea, but it doesn't address the reason why some
systems refuse to update their caches when they see mtimes in the
future. The motivation there is that if the mtime is in the future,
then it's possible that at some point in the future, the mtime will
match the current time, and then if the directory is modified at that
moment, the cache will become silently invalid.

It's not clear how important this really is; you have to get somewhat
unlucky, and if you're seeing timestamps from the future then
timekeeping has obviously broken down somehow and nothing based on
mtimes can be reliable without reliable timekeeping. (For example,
even if the mtime seems to be in the past, the clock could get set
backwards and now the same mtime is in the future after all.) But
that's the reasoning I've seen.

> The os module has atomic write support on Windows in 3.x now:
> https://docs.python.org/3/library/os.html#os.replace
>
> So the only problematic case is 2.7 on WIndows, and for that Christian
> Heimes backported pyosreplace here: https://pypi.org/project/pyosreplace/
>
> (The "may be non-atomic" case is the same situation where it will fail
> outright on POSIX systems: when you're attempting to do the rename across
> filesystems. If you stay within the same directory, which you want to do
> anyway for permissions inheritance and automatic file labeling, it's
> atomic).

I've never been able to tell whether this is trustworthy or not; MS
documents the rename-across-filesystems case as an *example* of a case
where it's non-atomic, and doesn't document any atomicity guarantees
either way. Is it really atomic on FAT filesystems? On network
filesystems? (Do all versions of CIFS even give a way to express file
replacement as a single operation?) But there's folklore saying it's
OK...

I guess in this case atomicity wouldn't be that crucial anyway though.

>> > Option 3: persistent per-path-entry cache
>> >
>> > * Pro: assuming cache freshness means zero runtime queries incur a
>> > linear DB
>> > read (cache creation becomes an install time cost)
>> > * Con: if you don't assume cache freshness, you need option 1 or 2
>> > anyway,
>> > and the install time cache just speeds up that first linear read
>> > * Con: filesystem access control requires either explicit cache refresh
>> > or
>> > implicit metadata caching support in installers
>> > * Con: sys.path directory mtimes are no longer sufficient for cache
>> > invalidation (due to potential for directory relocation)
>>
>> Not sure what problem you're thinking of here? In this model we
>> wouldn't be using mtimes for cache invalidation anyway, because it'd
>> be the responsibility of those modifying the directory to update the
>> cache. And if you rename a whole directory, that doesn't affect its
>> mtime anyway?
>
>
> Your second sentence is what I meant - whether the cache is still valid or
> not is less about the mtime, and more about what other actions have been
> performed. (It's much closer to the locate/updatedb model, where the runtime
> part just assumes the cache is valid, and it's somebody else's problem to
> ensure that assumption is reasonably valid)

Yeah. Which is probably the big issue with your third approach: it'll
probably work great if all installers are updated to properly manage
the cache. Explicit cache invalidation is fast and reliable and avoids
all these mtime shenanigans... if everyone implements it properly. But
currently there's lots of software that feels free to randomly dump
stuff into sys.path and doesn't know about the cache invalidation
thing (e.g. old versions of pip and setuptools), and that's a disaster
in a pure explicit invalidation model.

I guess in practice the solution for the transition period would be to
also store the mtime in the cache, so you can at least detect with
high probability when someone has used a legacy installer, and yell at
them to stop doing that. Though this might then cause problems if
people do stuff like zip up their site-packages and then unzip it
somewhere else, updating the mtimes in the process.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From alex.hha at gmail.com  Fri Oct 27 17:22:32 2017
From: alex.hha at gmail.com (Alex Domoradov)
Date: Sat, 28 Oct 2017 00:22:32 +0300
Subject: [Distutils] Disabling non HTTPS access to APIs on PyPI
In-Reply-To: <CACac1F-Mrja6dhLsQKuwA-ZZwzokRtyxq5icE4oxRHgapjpuPw@mail.gmail.com>
References: <CAK90gp6mU__+6HopzL+=8RxogkaztPh1fdoE53=Y42rnBTWrjA@mail.gmail.com>
 <CAPJVwB=JvFyzDA2r77BxSjCMhURHouHKSXW1GmPRjs=gzdQxAA@mail.gmail.com>
 <CAPJVwBmvZODgFAwBC+UG+jnLWLNDnEnSG6B_tLmq-E-CmB1ZbQ@mail.gmail.com>
 <CAMxX_oMstS2vHCre_pE7TT-5Zj98bdetnxzNJ4344GS1rBgQcg@mail.gmail.com>
 <CACac1F-Mrja6dhLsQKuwA-ZZwzokRtyxq5icE4oxRHgapjpuPw@mail.gmail.com>
Message-ID: <CAK90gp5zFsvYPEcRyf7sa18+qXADqOayH2ey_n28ou-uMpJJ0Q@mail.gmail.com>

I got it. And what I should do with old system? For e.g. we still use
ubuntu 12.04. Is there any way to upgrade pip/setuptools?

On Fri, Oct 27, 2017 at 11:17 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 27 October 2017 at 21:05, Leonardo Rochael Almeida
> <leorochael at gmail.com> wrote:
> > By the way, some pages on pypi itself are pointing to http:
> >
> > See, e.g. the "Download URL:" link here:
> >
> > https://pypi.python.org/pypi/google-appengine
>
> That's the project itself doing that, I believe (it's metadata from
> setup.py) so you'd need to report it to them.
>
> Paul
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171028/c66e1864/attachment-0001.html>

From wes.turner at gmail.com  Fri Oct 27 22:24:16 2017
From: wes.turner at gmail.com (Wes Turner)
Date: Fri, 27 Oct 2017 22:24:16 -0400
Subject: [Distutils] Disabling non HTTPS access to APIs on PyPI
In-Reply-To: <CACac1F-0KRhA1y25-5ap1SvyrsWcxUkUkL_1nVaxzLRk9mVF1A@mail.gmail.com>
References: <CAK90gp6mU__+6HopzL+=8RxogkaztPh1fdoE53=Y42rnBTWrjA@mail.gmail.com>
 <CAPJVwB=JvFyzDA2r77BxSjCMhURHouHKSXW1GmPRjs=gzdQxAA@mail.gmail.com>
 <CAPJVwBmvZODgFAwBC+UG+jnLWLNDnEnSG6B_tLmq-E-CmB1ZbQ@mail.gmail.com>
 <CAMxX_oMstS2vHCre_pE7TT-5Zj98bdetnxzNJ4344GS1rBgQcg@mail.gmail.com>
 <CACac1F-Mrja6dhLsQKuwA-ZZwzokRtyxq5icE4oxRHgapjpuPw@mail.gmail.com>
 <CAK90gp5zFsvYPEcRyf7sa18+qXADqOayH2ey_n28ou-uMpJJ0Q@mail.gmail.com>
 <CACac1F-0KRhA1y25-5ap1SvyrsWcxUkUkL_1nVaxzLRk9mVF1A@mail.gmail.com>
Message-ID: <CACfEFw8pxFBJdXP5mKz3hei8_6whRA1Rs7ubG7oV1e+HgXBvKw@mail.gmail.com>

On Friday, October 27, 2017, Paul Moore <p.f.moore at gmail.com> wrote:

> On 27 October 2017 at 22:22, Alex Domoradov <alex.hha at gmail.com
> <javascript:;>> wrote:
> > I got it. And what I should do with old system? For e.g. we still use
> ubuntu
> > 12.04. Is there any way to upgrade pip/setuptools?
>
> Well, if Ubuntu aren't offering an upgrade, you can do
>
> pip install -i https://pypi.python.org/simple/ ...
>
> Or install your own copy of pip/setuptools, I guess (get-pip --user,
> see https://pip.pypa.io/en/stable/installing/#installing-with-get-pip-py).
> Paul


Does this upgrade system pip, regardless of the package checksums:

$ sudo pip install -i https://pypi.python.org/simple/ -U pip setuptools


If that doesn't work (or **isn't advisable because `sudo pip` is
dangerous**),
you can also configure the index URL with a config file or an environment
variable:

https://pip.pypa.io/en/latest/user_guide/#config-file lists the paths for
Windows, MacOS, and Linux.

/etc/pip.conf
~/.pip/pip.conf
$VIRTUAL_ENV/pip.conf

```
[global]

index-url = https://pypi.python.org/simple/
```

$ export PIP_INDEX_URL="https://pypi.python.org/simple/"

Setuptools (easy_install) uses ~/.pydistutils.cfg :

```
[easy_install]
index_url = https://pypi.python.org/simple/
```

Buildout uses buildout.cfg and ~/.buildout/default.cfg :

```
[buildout]
index = https://pypi.python.org/simple/
```

"What to do when PyPi goes down" (2010)
https://jacobian.org/writing/when-pypi-goes-down/

This URL also works as the -i/--index-url?

https://pypi.org/simple/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171027/677cc1d3/attachment.html>

From wes.turner at gmail.com  Fri Oct 27 22:29:26 2017
From: wes.turner at gmail.com (Wes Turner)
Date: Fri, 27 Oct 2017 22:29:26 -0400
Subject: [Distutils] Disabling non HTTPS access to APIs on PyPI
In-Reply-To: <9C55AB34-E5F8-4355-B8F0-2896E6922EA4@stufft.io>
References: <9C55AB34-E5F8-4355-B8F0-2896E6922EA4@stufft.io>
Message-ID: <CACfEFw8k4_Ksry5pqAbHv9iMd--n0-NMhgbsBqGD4fgSCDhZbA@mail.gmail.com>

- Are there issue tickets which contain the search-indexed ERROR_STRINGS
users may encounter due to this change?

- Does it make sense to add an update regarding this necessary security
upgrade to https://status.python.org (which can be subscribed to and
followed on http://www.twitter.com/PythonStatus )?

On Thursday, October 26, 2017, Donald Stufft <donald at stufft.io> wrote:

> Historically PyPI was only available over either HTTP or unvalidated
> HTTPS, and over time we?ve been pushing more and more traffic onto HTTPS.
> In Warehouse the decision was made to *not* redirect ?API? URLs from HTTP
> to HTTPS, but to rather return an error accessing them from HTTP. This is
> because while logged in views have HSTS to ensure HTTPS in the browser (and
> with humans manually entering them into the URL bar regularly they are more
> error prone) APIs which are typically accessed by automated clients with an
> URL configured or hardcoded typically do not respect HSTS, so if you had a
> script that did ``curl http://pypi.python.org/simple/``, it would
> silently get redirects to https and appear to ?work?, but you wouldn?t get
> any of the security properties of TLS because an attacker would just
> intercept the request prior to the redirect happening.
>
> Today I?ve backported this changed to the current production deployment of
> PyPI, which means that you can no longer access /simple/ and /packages/
> over HTTP and you will have to directly go to HTTPS. For most people this
> should have no effect, because most tooling should be defaulting to HTTPS
> anyways, however if you?re using a significantly old version of tooling, it
> may still be defaulting to the HTTP url and will now stop functioning.
>
> The recommended remediation is to upgrade your tooling to versions that
> support verified TLS connections and which default to the proper HTTPS URLs.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171027/986ee8ab/attachment.html>

From vinay_sajip at yahoo.co.uk  Sat Oct 28 08:04:21 2017
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 28 Oct 2017 12:04:21 +0000 (UTC)
Subject: [Distutils] ANN: distlib 0.2.6 released on PyPI
References: <462443690.8909932.1509192261626.ref@mail.yahoo.com>
Message-ID: <462443690.8909932.1509192261626@mail.yahoo.com>

I've just released version 0.2.6 of distlib on PyPI [1]. For newcomers,distlib is a library of packaging functionality which is intended to beusable as the basis for third-party packaging tools.
The main changes in this release are as follows:
* Fixed #99: Updated to handle a case where sys.getfilesystemencoding()? returns None.
* Fixed #97: Eliminated a crash in EggInfoDistribution.list_distinfo_files()? which was caused by trying to open a non-existent file.
* Fixed #96: SimpleScrapingLocator no longer fails prematurely when scraping? links due to invalid versions.
* Improved error messages issued when interpreting markers.
* Improved the shebangs written into installed scripts when the interpreter? path is very long or contains spaces (to cater for a limitation in shebang? line parsing on Linux).
* Updated launcher binaries.
A more detailed change log is available at [2].
Please try it out, and if you find any problems or have any suggestions forimprovements, please give some feedback using the issue tracker! [3]
Regards,

Vinay Sajip
[1] https://pypi.python.org/pypi/distlib/0.2.6[2] https://goo.gl/M3kQzR[3] https://bitbucket.org/pypa/distlib/issues/new

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171028/7465b515/attachment.html>

From alex.gronholm at nextday.fi  Sun Oct 29 15:06:36 2017
From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=)
Date: Sun, 29 Oct 2017 21:06:36 +0200
Subject: [Distutils] Wheel 1.0 roadmap
Message-ID: <d12460f9-209a-fa51-880f-f94476d93d3c@nextday.fi>

I am planning for a 1.0.0 release of the "wheel" library. I would like 
to start using semver from this point onwards, which in the case of 
wheel means that its command line interface should be well defined and 
remain backwards compatible. As part of this effort, I've rewritten the 
documentation (currently in the "docs-update" branch on Github) to 
conform to the PyPA guidelines. Wheel also had some generated API 
documentation on ReadTheDocs, but as discussed privately with Daniel 
Holth and Nick Coghlan, wheel should not have a public API going forward 
so I've deleted that documentation.

I've also taken a hard look at wheel's features and would like to remove 
those which I consider to be either useless or harmful. I've added these 
tasks as issues on Github.

All the issues that I'd like to get resolved by 1.0.0 have been tagged 
with the proper milestone marker here: 
https://github.com/pypa/wheel/milestone/1

Feedback is very welcome!

ps. Daniel, if you're reading this, would you mind giving the new docs a 
once-over? Also, if you can suggest where to put the "story" page, I'll 
link it back to the main index file.


From wes.turner at gmail.com  Sun Oct 29 17:01:30 2017
From: wes.turner at gmail.com (Wes Turner)
Date: Sun, 29 Oct 2017 17:01:30 -0400
Subject: [Distutils] Wheel 1.0 roadmap
In-Reply-To: <d12460f9-209a-fa51-880f-f94476d93d3c@nextday.fi>
References: <d12460f9-209a-fa51-880f-f94476d93d3c@nextday.fi>
Message-ID: <CACfEFw9mYXAgMbAzX9GQcE_kacQ+Aw-=9nx1q3qhHk==dhmgfg@mail.gmail.com>

REQ: feedback re: "Remove or deprecate wheel signing features #196"
https://github.com/pypa/wheel/issues/196

Is the current implementation incomplete without signature verification?
According to the spec?

```
The spec includes this feature. So, even though this verify() function is
incomplete, it would be wrong to just remove it without also removing it
from the spec.

- https://www.python.org/dev/peps/pep-0427/#signed-wheel-files
- https://www.python.org/dev/peps/pep-0491/#signed-wheel-files

I don't have the information needed to explain what completely implemented
signatures are useful for. Does the spec explain this?

> A wheel installer is not required to understand digital signatures but
MUST verify the hashes in RECORD against the extracted file contents. When
the installer checks file hashes against RECORD, a separate signature
checker only needs to establish that RECORD matches the signature.
```

On Sunday, October 29, 2017, Alex Gr?nholm <alex.gronholm at nextday.fi> wrote:

> I am planning for a 1.0.0 release of the "wheel" library. I would like to
> start using semver from this point onwards, which in the case of wheel
> means that its command line interface should be well defined and remain
> backwards compatible. As part of this effort, I've rewritten the
> documentation (currently in the "docs-update" branch on Github) to conform
> to the PyPA guidelines. Wheel also had some generated API documentation on
> ReadTheDocs, but as discussed privately with Daniel Holth and Nick Coghlan,
> wheel should not have a public API going forward so I've deleted that
> documentation.
>
> I've also taken a hard look at wheel's features and would like to remove
> those which I consider to be either useless or harmful. I've added these
> tasks as issues on Github.
>
> All the issues that I'd like to get resolved by 1.0.0 have been tagged
> with the proper milestone marker here: https://github.com/pypa/wheel/
> milestone/1
>
> Feedback is very welcome!
>
> ps. Daniel, if you're reading this, would you mind giving the new docs a
> once-over? Also, if you can suggest where to put the "story" page, I'll
> link it back to the main index file.
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171029/cdf205cd/attachment.html>

From p.f.moore at gmail.com  Mon Oct 30 04:43:26 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 30 Oct 2017 08:43:26 +0000
Subject: [Distutils] Wheel 1.0 roadmap
In-Reply-To: <CACfEFw9mYXAgMbAzX9GQcE_kacQ+Aw-=9nx1q3qhHk==dhmgfg@mail.gmail.com>
References: <d12460f9-209a-fa51-880f-f94476d93d3c@nextday.fi>
 <CACfEFw9mYXAgMbAzX9GQcE_kacQ+Aw-=9nx1q3qhHk==dhmgfg@mail.gmail.com>
Message-ID: <CACac1F_C_xEtDsk9MsAe12vRSePM_RDZKdH=s-4dNzoZEpKrPg@mail.gmail.com>

On 29 October 2017 at 21:01, Wes Turner <wes.turner at gmail.com> wrote:
> REQ: feedback re: "Remove or deprecate wheel signing features #196"
> https://github.com/pypa/wheel/issues/196
>
> Is the current implementation incomplete without signature verification?
> According to the spec?

I've never used or felt the need for this feature. I won't miss it.
Paul

From ncoghlan at gmail.com  Mon Oct 30 10:20:34 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 31 Oct 2017 00:20:34 +1000
Subject: [Distutils] Wheel 1.0 roadmap
In-Reply-To: <CACac1F_C_xEtDsk9MsAe12vRSePM_RDZKdH=s-4dNzoZEpKrPg@mail.gmail.com>
References: <d12460f9-209a-fa51-880f-f94476d93d3c@nextday.fi>
 <CACfEFw9mYXAgMbAzX9GQcE_kacQ+Aw-=9nx1q3qhHk==dhmgfg@mail.gmail.com>
 <CACac1F_C_xEtDsk9MsAe12vRSePM_RDZKdH=s-4dNzoZEpKrPg@mail.gmail.com>
Message-ID: <CADiSq7cGtbFYUh6AnLRzpz5eG4YYO-ucDG7dAXUv8p=oWDiJhw@mail.gmail.com>

On 30 October 2017 at 18:43, Paul Moore <p.f.moore at gmail.com> wrote:

> On 29 October 2017 at 21:01, Wes Turner <wes.turner at gmail.com> wrote:
> > REQ: feedback re: "Remove or deprecate wheel signing features #196"
> > https://github.com/pypa/wheel/issues/196
> >
> > Is the current implementation incomplete without signature verification?
> > According to the spec?
>
> I've never used or felt the need for this feature. I won't miss it.
>

In practice, most folks are relying on checking the archive hashes as their
integrity check, rather than checking the individual file hashes in RECORD
(and then signing the RECORD file), since that lets them completely avoid
worrying about the problem of establishing trust in an initial set of
public keys.

For folks that do want signatures on their build server -> deployment
system connections (which is the problem this features aims to help with),
they're currently more likely to use external GPG signatures (the way Linux
distros and some container registries do) or a system like Notary/TUF (the
way the Docker registry does), than they are a Python-specific format.

So I think it would be reasonable for the wheel project maintainers to say
they don't want to be responsible for ensuring that their signing
implementation provides meaningful security assurances, and deprecate and
remove it. We'd then update PEP 427 with a note saying that the signing
feature has been deprecated in the reference implementation, and may be
removed from a future version of the specification.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171031/b035a2bc/attachment-0001.html>

From a.badger at gmail.com  Mon Oct 30 13:41:58 2017
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Mon, 30 Oct 2017 10:41:58 -0700
Subject: [Distutils] What's the use case of testpypi?
Message-ID: <CABVPEKpYmmw_ZSD7dVa2eXOQwTFhQLSLpjk3LhqRqG749T+CiQ@mail.gmail.com>

When we locked down pypi to prevent uploading an sdist to overwrite a
previous I remember that some people wanted a brief window to check
for brown paper bag issues and be able to upload a new tarball in that
window if needed.  IIRC, those people were told to use testpypi for
that sort of thing.  Upload potential tarball to testpypi.  If it
works, go ahead and rerun to upload to the real pypi.  If it doesn't,
fix it and try again.

This past week I decided to try out that workflow for a project that
I'm managing the releases for and ran into a snag.  testpypi has the
same behaviour as production pypi wherein you can only upload a given
sdist once and afterwards it's no longer allowed.  For the use case
above this is problematic.  It essentially changes the idea of "test
your release on testpypi before making a release" into "You can have
to chances to get it right if you use testpypi" which, although better
than uploading directly to pypi, still leaves a lot of room for error
(let's face it: I know that if I'm stuck releasing late at night due
to time constraints and make a mistake, chances are better than normal
that my fix won't be the perfection that my ordinary code is and could
have other showstopper bugs that I'd want my testing to catch as well
;-)

Is this something that we could change for testpypi?  It could be
implemented in many ways: straight overwrite, being able to destroy a
version so that it seems to never have existed, or being able to
destroy and recreate a package so that it has no uploaded sdists
recorded.

On the other side of the usefulness of enabling the testing use case
above, such a change would be a difference between testpypi and
production pypi meaning that it would no longer be testing exactly the
same functionality as will be deployed in production.  I'm not sure if
that's a more important consideration or not.  I figured that unless I
asked, I would never know the answer :-)

Thanks,
-Toshio

From mariatta.wijaya at gmail.com  Mon Oct 30 14:01:52 2017
From: mariatta.wijaya at gmail.com (Mariatta Wijaya)
Date: Mon, 30 Oct 2017 11:01:52 -0700
Subject: [Distutils] What's the use case of testpypi?
In-Reply-To: <CABVPEKpYmmw_ZSD7dVa2eXOQwTFhQLSLpjk3LhqRqG749T+CiQ@mail.gmail.com>
References: <CABVPEKpYmmw_ZSD7dVa2eXOQwTFhQLSLpjk3LhqRqG749T+CiQ@mail.gmail.com>
Message-ID: <CAGbohnaQVZVPH2Cn9GiQNdJsntczwOTzA1Wj=_qv7m8xM8hMpQ@mail.gmail.com>

TestPyPI can be useful for those who are not package maintainers, but want
to learn about Python Packaging and need a 'playground' to experiment with
it.


Mariatta Wijaya

On Mon, Oct 30, 2017 at 10:41 AM, Toshio Kuratomi <a.badger at gmail.com>
wrote:

> When we locked down pypi to prevent uploading an sdist to overwrite a
> previous I remember that some people wanted a brief window to check
> for brown paper bag issues and be able to upload a new tarball in that
> window if needed.  IIRC, those people were told to use testpypi for
> that sort of thing.  Upload potential tarball to testpypi.  If it
> works, go ahead and rerun to upload to the real pypi.  If it doesn't,
> fix it and try again.
>
> This past week I decided to try out that workflow for a project that
> I'm managing the releases for and ran into a snag.  testpypi has the
> same behaviour as production pypi wherein you can only upload a given
> sdist once and afterwards it's no longer allowed.  For the use case
> above this is problematic.  It essentially changes the idea of "test
> your release on testpypi before making a release" into "You can have
> to chances to get it right if you use testpypi" which, although better
> than uploading directly to pypi, still leaves a lot of room for error
> (let's face it: I know that if I'm stuck releasing late at night due
> to time constraints and make a mistake, chances are better than normal
> that my fix won't be the perfection that my ordinary code is and could
> have other showstopper bugs that I'd want my testing to catch as well
> ;-)
>
> Is this something that we could change for testpypi?  It could be
> implemented in many ways: straight overwrite, being able to destroy a
> version so that it seems to never have existed, or being able to
> destroy and recreate a package so that it has no uploaded sdists
> recorded.
>
> On the other side of the usefulness of enabling the testing use case
> above, such a change would be a difference between testpypi and
> production pypi meaning that it would no longer be testing exactly the
> same functionality as will be deployed in production.  I'm not sure if
> that's a more important consideration or not.  I figured that unless I
> asked, I would never know the answer :-)
>
> Thanks,
> -Toshio
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171030/d6214f9d/attachment.html>

From donald at stufft.io  Mon Oct 30 14:12:28 2017
From: donald at stufft.io (Donald Stufft)
Date: Mon, 30 Oct 2017 14:12:28 -0400
Subject: [Distutils] What's the use case of testpypi?
In-Reply-To: <CABVPEKpYmmw_ZSD7dVa2eXOQwTFhQLSLpjk3LhqRqG749T+CiQ@mail.gmail.com>
References: <CABVPEKpYmmw_ZSD7dVa2eXOQwTFhQLSLpjk3LhqRqG749T+CiQ@mail.gmail.com>
Message-ID: <F8D26948-185E-4A61-A715-3AD4A5CA5577@stufft.io>


> On Oct 30, 2017, at 1:41 PM, Toshio Kuratomi <a.badger at gmail.com> wrote:
> 
> I figured that unless I
> asked, I would never know the answer :-)
> 


TestPyPI has a couple of use cases, and it?s not very good at any of them.

One use case is to serve as a staging site for production PyPI to test changes before they go to live. If I had to pick any use case as it?s ?core? use case, it would be this one, it?s currently optimized for it and we?re unlikely to make changes that deviate from it. Unfortunately it?s not really good at this because staging sites are generally not a great way to prevent regression (versus something like automated testing), so with Warehouse TestPyPI is no longer a staging site for the production code, and they are deployed at the same time.

Another use case is the one you specified, someone wants to test their releases prior to uploading to the ?real? PyPI. TestPyPI is not great at this use case both because, as you noted, it has the same ?you can only upload a file once? semantics as PyPI does, but also because it doesn?t have a full copy of everything on PyPI (and worse, it potentially has broken or insecure or malicious versions of your dependencies) so pip might either be blatantly broken OR produce subtly broken installs. Long term the solution that I believe has merit for this, is the ability to, on regular PyPI, ?stage? an upload to a temporary location that would automatically be deleted after some period of time, and that once you were satisfied with it, could be published/committed and thus be available to normal ``pip install`` invocations. This not only would allow testing a release beforehand, but it would also allow projects that produce many different artifacts across a variety of platforms to collect them all and upload them and make them all available immediately instead of one at a time. You can see more about this idea/use case at https://github.com/pypa/warehouse/issues/726 <https://github.com/pypa/warehouse/issues/726>.

The third relevant use case of PyPI is people simply messing around or doing testing of a generalized workflow tool but not specific to a specific package. This could also include people who are learning how to produce a package, but using a trivial throw away project (the infamous printer of nested lists is an example of this). TestPyPI can be used for this as well, but again it?s not really good at it. The key difference here is that ideally these projects are either ephemeral or namespaced to the specific user. I don?t have a great idea for how best to handle this, the immediate thought is a secondary deployment of Warehouse (such as what TestPyPI is now), that just periodically and automatically purges any projects > N days old to give it a more of a ?sandbox? vibe and use case. Given the aforementioned staging feature, we would likely otherwise keep the same ruleset as exists on the production PyPI (and ideally, we?d be re-using credentials too, instead of it literally being a complete secondary deployment). That idea isn?t set in stone and I haven?t done a lot of thinking about it yet, but discussion of that particular use case and ideas to fix it would take place in https://github.com/pypa/warehouse/issues/2286 <https://github.com/pypa/warehouse/issues/2286>.

So longer term, I hope to shut down TestPyPI and replace it with features and/or deployments more suited to whatever specific task/use case we?re hoping to have served by it, rather than just sort of being poor at everything.

- dstufft
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171030/b718fc0f/attachment-0001.html>

From opensource at ronnypfannschmidt.de  Mon Oct 30 15:16:02 2017
From: opensource at ronnypfannschmidt.de (RonnyPfannschmidt)
Date: Mon, 30 Oct 2017 20:16:02 +0100
Subject: [Distutils] [proposal] shared distribution installations
Message-ID: <8e2d6eaa-cfac-dde2-0b4c-50015211447f@ronnypfannschmidt.de>

Hi everyone,

since a while now various details of installing python packages in
virtualenvs caused me grief

a) typically each tox folder in a project is massive, and has a lot of
duplicate files, recreating them, managing and iterating them takes
quite a while
b) for nicely separated deployments, each virtualenv for an application
takes a few hundred megabytes - that quickly can saturate disk space
even if a reasonable amount was reserved
c) installation and recreation of virtualenvs with the same set of
packages takes quite a while (even with pip caches this is slow, and
there is no good reason to avoid making it completely instantaneous)

in order to elevate those issues i would like to propose a new
installation layout,
where instead of storing each distribution in every python all
distributions would share a storage, and each individual environment
would only have references to the packages that where
"installed/activated" for them

this would massively reduce time required to create the contents of the
environments and also the space required

since blindly expanding sys.path would lead to similar performance
issues as where seen with setuptools/buildout multi-version installs,
this mechanism would also need a element on sys.meta_path that handles
inexpensive dispatch to the toplevels and metadata files of each
packages (off hand i would assume linear walking of hundreds of entries
simply isn't that effective)

however there would be need for some experimentation to see what
tradeoff is sensible there

I hope this mail will spark enough discussion to enable the creation of
a PEP and a prototype.


Best, Ronny




From freddyrietdijk at fridh.nl  Mon Oct 30 15:35:41 2017
From: freddyrietdijk at fridh.nl (Freddy Rietdijk)
Date: Mon, 30 Oct 2017 20:35:41 +0100
Subject: [Distutils] [proposal] shared distribution installations
In-Reply-To: <8e2d6eaa-cfac-dde2-0b4c-50015211447f@ronnypfannschmidt.de>
References: <8e2d6eaa-cfac-dde2-0b4c-50015211447f@ronnypfannschmidt.de>
Message-ID: <CAOQtOH2EmPdKmTbNq_BbGOm+25Phr8G1JOgAf_oT4bpr6z163w@mail.gmail.com>

Hi Ronny,

What you describe here is, as you know, basically what the Nix package
manager does. You could create something similar specifically for Python,
like e.g. `ied` is for Node [2], or Spack, which is written in Python. But
then how are you going to deal with other system libraries, and impurities?
And you will have to deal with them, because depending on how you configure
Python packages that depend on them (say a `numpy`), their output will be
different. Or would you choose to ignore this?

Freddy

[1] https://nixos.org/nix/
[2] https://github.com/alexanderGugel/ied
[3] https://spack.io/

On Mon, Oct 30, 2017 at 8:16 PM, RonnyPfannschmidt <
opensource at ronnypfannschmidt.de> wrote:

> Hi everyone,
>
> since a while now various details of installing python packages in
> virtualenvs caused me grief
>
> a) typically each tox folder in a project is massive, and has a lot of
> duplicate files, recreating them, managing and iterating them takes
> quite a while
> b) for nicely separated deployments, each virtualenv for an application
> takes a few hundred megabytes - that quickly can saturate disk space
> even if a reasonable amount was reserved
> c) installation and recreation of virtualenvs with the same set of
> packages takes quite a while (even with pip caches this is slow, and
> there is no good reason to avoid making it completely instantaneous)
>
> in order to elevate those issues i would like to propose a new
> installation layout,
> where instead of storing each distribution in every python all
> distributions would share a storage, and each individual environment
> would only have references to the packages that where
> "installed/activated" for them
>
> this would massively reduce time required to create the contents of the
> environments and also the space required
>
> since blindly expanding sys.path would lead to similar performance
> issues as where seen with setuptools/buildout multi-version installs,
> this mechanism would also need a element on sys.meta_path that handles
> inexpensive dispatch to the toplevels and metadata files of each
> packages (off hand i would assume linear walking of hundreds of entries
> simply isn't that effective)
>
> however there would be need for some experimentation to see what
> tradeoff is sensible there
>
> I hope this mail will spark enough discussion to enable the creation of
> a PEP and a prototype.
>
>
> Best, Ronny
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171030/fa59d346/attachment.html>

From fungi at yuggoth.org  Mon Oct 30 15:47:10 2017
From: fungi at yuggoth.org (Jeremy Stanley)
Date: Mon, 30 Oct 2017 19:47:10 +0000
Subject: [Distutils] Wheel 1.0 roadmap
In-Reply-To: <CADiSq7cGtbFYUh6AnLRzpz5eG4YYO-ucDG7dAXUv8p=oWDiJhw@mail.gmail.com>
References: <d12460f9-209a-fa51-880f-f94476d93d3c@nextday.fi>
 <CACfEFw9mYXAgMbAzX9GQcE_kacQ+Aw-=9nx1q3qhHk==dhmgfg@mail.gmail.com>
 <CACac1F_C_xEtDsk9MsAe12vRSePM_RDZKdH=s-4dNzoZEpKrPg@mail.gmail.com>
 <CADiSq7cGtbFYUh6AnLRzpz5eG4YYO-ucDG7dAXUv8p=oWDiJhw@mail.gmail.com>
Message-ID: <20171030194710.GB28203@yuggoth.org>

On 2017-10-31 00:20:34 +1000 (+1000), Nick Coghlan wrote:
[...]
> For folks that do want signatures on their build server -> deployment
> system connections (which is the problem this features aims to help with),
> they're currently more likely to use external GPG signatures (the way Linux
> distros and some container registries do) or a system like Notary/TUF (the
> way the Docker registry does), than they are a Python-specific format.
[...]

Agreed. For the hundreds of projects we publish on PyPI we have our
release automation generate detached OpenPGP signatures of sdsits
and wheels, and serve those signatures from our own release info
site since PyPI also seems to not want to support arbitrary
signature uploads over the long term. This satisfies the requests we
get from distribution package maintainers to provide proof of
provenance for our release artifacts; our release managers and
community infrastructure sysadmins sign the per-cycle release
automation keys, and regularly participate in key signing with
distro package maintainers in-person at conferences to establish a
sufficient web of trust. I understand this is probably untenable for
smaller projects, but at our scale it works fairly well (also easier
to generalize beyond merely Python-based software).

I'll be honest, when designing our artifact signing automation I
didn't even know the wheel spec suggested it should be a feature,
but without having consistent integration in other tooling for
signed sdists too it wouldn't have been much help to us anyway.
-- 
Jeremy Stanley
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171030/7585c60e/attachment.sig>

From opensource at ronnypfannschmidt.de  Mon Oct 30 17:09:51 2017
From: opensource at ronnypfannschmidt.de (Ronny Pfannschmidt)
Date: Mon, 30 Oct 2017 22:09:51 +0100
Subject: [Distutils] [proposal] shared distribution installations
In-Reply-To: <CAOQtOH2EmPdKmTbNq_BbGOm+25Phr8G1JOgAf_oT4bpr6z163w@mail.gmail.com>
References: <8e2d6eaa-cfac-dde2-0b4c-50015211447f@ronnypfannschmidt.de>
 <CAOQtOH2EmPdKmTbNq_BbGOm+25Phr8G1JOgAf_oT4bpr6z163w@mail.gmail.com>
Message-ID: <1509397791.27574.8.camel@ronnypfannschmidt.de>

Hi Freddy,

im well aware what nix currently does for python packages and suffered
my fair share from it.

What i want to do is simply store those wheels that pip would first
generate and unpack into each environment into a location where each
environment shares the unpacked files more directly

im not going to expand uppon my perceived shortcomings of nix as i know
,it since its irrelevant to this discussion and not something i have
the time and motivation to fix.

as far as impurities go, the behaviour i aim for would be moslty like
virtualenv but without the file duplication.

I beleive nix could also benefit from parts of such a mechanism.

-- Ronny


Am Montag, den 30.10.2017, 20:35 +0100 schrieb Freddy Rietdijk:
> Hi Ronny,
> 
> What you describe here is, as you know, basically what the Nix
> package manager does. You could create something similar specifically
> for Python, like e.g. `ied` is for Node [2], or Spack, which is
> written in Python. But then how are you going to deal with other
> system libraries, and impurities? And you will have to deal with
> them, because depending on how you configure Python packages that
> depend on them (say a `numpy`), their output will be different. Or
> would you choose to ignore this? 
> 
> Freddy
> 
> [1] https://nixos.org/nix/
> [2] https://github.com/alexanderGugel/ied
> [3] https://spack.io/
> 
> On Mon, Oct 30, 2017 at 8:16 PM, RonnyPfannschmidt <opensource at ronnyp
> fannschmidt.de> wrote:
> > Hi everyone,
> > 
> > since a while now various details of installing python packages in
> > virtualenvs caused me grief
> > 
> > a) typically each tox folder in a project is massive, and has a lot
> > of
> > duplicate files, recreating them, managing and iterating them takes
> > quite a while
> > b) for nicely separated deployments, each virtualenv for an
> > application
> > takes a few hundred megabytes - that quickly can saturate disk
> > space
> > even if a reasonable amount was reserved
> > c) installation and recreation of virtualenvs with the same set of
> > packages takes quite a while (even with pip caches this is slow,
> > and
> > there is no good reason to avoid making it completely
> > instantaneous)
> > 
> > in order to elevate those issues i would like to propose a new
> > installation layout,
> > where instead of storing each distribution in every python all
> > distributions would share a storage, and each individual
> > environment
> > would only have references to the packages that where
> > "installed/activated" for them
> > 
> > this would massively reduce time required to create the contents of
> > the
> > environments and also the space required
> > 
> > since blindly expanding sys.path would lead to similar performance
> > issues as where seen with setuptools/buildout multi-version
> > installs,
> > this mechanism would also need a element on sys.meta_path that
> > handles
> > inexpensive dispatch to the toplevels and metadata files of each
> > packages (off hand i would assume linear walking of hundreds of
> > entries
> > simply isn't that effective)
> > 
> > however there would be need for some experimentation to see what
> > tradeoff is sensible there
> > 
> > I hope this mail will spark enough discussion to enable the
> > creation of
> > a PEP and a prototype.
> > 
> > 
> > Best, Ronny
> > 
> > 
> > 
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> 
> 

From thomas at kluyver.me.uk  Mon Oct 30 18:23:03 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Mon, 30 Oct 2017 22:23:03 +0000
Subject: [Distutils] [proposal] shared distribution installations
In-Reply-To: <8e2d6eaa-cfac-dde2-0b4c-50015211447f@ronnypfannschmidt.de>
References: <8e2d6eaa-cfac-dde2-0b4c-50015211447f@ronnypfannschmidt.de>
Message-ID: <1509402183.552259.1156144160.3FEFC4F3@webmail.messagingengine.com>

On Mon, Oct 30, 2017, at 07:16 PM, RonnyPfannschmidt wrote:
> in order to elevate those issues i would like to propose a new
> installation layout,
> where instead of storing each distribution in every python all
> distributions would share a storage, and each individual environment
> would only have references to the packages that where
> "installed/activated" for them

This is also essentially what conda does - the references being in the
form of hard links. The mechanism has some drawbacks of its own - like
if a file somehow gets modified, it's harder to fix it, because removing
the environment no longer removes the files.

Thomas

From chris.barker at noaa.gov  Mon Oct 30 20:55:58 2017
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Mon, 30 Oct 2017 17:55:58 -0700
Subject: [Distutils] [proposal] shared distribution installations
In-Reply-To: <1509397791.27574.8.camel@ronnypfannschmidt.de>
References: <8e2d6eaa-cfac-dde2-0b4c-50015211447f@ronnypfannschmidt.de>
 <CAOQtOH2EmPdKmTbNq_BbGOm+25Phr8G1JOgAf_oT4bpr6z163w@mail.gmail.com>
 <1509397791.27574.8.camel@ronnypfannschmidt.de>
Message-ID: <-7116111737088143840@unknownmsgid>

, the behaviour i aim for would be moslty like
virtualenv but without the file duplication.


For what it?s worth, conda environments use hard links where possible, so
limiting duplication...

Maybe conda would solve your problem...

-CHB







I beleive nix could also benefit from parts of such a mechanism.

-- Ronny


Am Montag, den 30.10.2017, 20:35 +0100 schrieb Freddy Rietdijk:

Hi Ronny,


What you describe here is, as you know, basically what the Nix

package manager does. You could create something similar specifically

for Python, like e.g. `ied` is for Node [2], or Spack, which is

written in Python. But then how are you going to deal with other

system libraries, and impurities? And you will have to deal with

them, because depending on how you configure Python packages that

depend on them (say a `numpy`), their output will be different. Or

would you choose to ignore this?


Freddy


[1] https://nixos.org/nix/

[2] https://github.com/alexanderGugel/ied

[3] https://spack.io/


On Mon, Oct 30, 2017 at 8:16 PM, RonnyPfannschmidt <opensource at ronnyp

fannschmidt.de> wrote:

Hi everyone,


since a while now various details of installing python packages in

virtualenvs caused me grief


a) typically each tox folder in a project is massive, and has a lot

of

duplicate files, recreating them, managing and iterating them takes

quite a while

b) for nicely separated deployments, each virtualenv for an

application

takes a few hundred megabytes - that quickly can saturate disk

space

even if a reasonable amount was reserved

c) installation and recreation of virtualenvs with the same set of

packages takes quite a while (even with pip caches this is slow,

and

there is no good reason to avoid making it completely

instantaneous)


in order to elevate those issues i would like to propose a new

installation layout,

where instead of storing each distribution in every python all

distributions would share a storage, and each individual

environment

would only have references to the packages that where

"installed/activated" for them


this would massively reduce time required to create the contents of

the

environments and also the space required


since blindly expanding sys.path would lead to similar performance

issues as where seen with setuptools/buildout multi-version

installs,

this mechanism would also need a element on sys.meta_path that

handles

inexpensive dispatch to the toplevels and metadata files of each

packages (off hand i would assume linear walking of hundreds of

entries

simply isn't that effective)


however there would be need for some experimentation to see what

tradeoff is sensible there


I hope this mail will spark enough discussion to enable the

creation of

a PEP and a prototype.



Best, Ronny




_______________________________________________

Distutils-SIG maillist  -  Distutils-SIG at python.org

https://mail.python.org/mailman/listinfo/distutils-sig



_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG at python.org
https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171030/d042f28e/attachment.html>

From ich at ronnypfannschmidt.de  Tue Oct 31 00:24:53 2017
From: ich at ronnypfannschmidt.de (Ronny Pfannschmidt)
Date: Tue, 31 Oct 2017 05:24:53 +0100
Subject: [Distutils] [proposal] shared distribution installations
In-Reply-To: <1509402183.552259.1156144160.3FEFC4F3@webmail.messagingengine.com>
References: <8e2d6eaa-cfac-dde2-0b4c-50015211447f@ronnypfannschmidt.de>
 <1509402183.552259.1156144160.3FEFC4F3@webmail.messagingengine.com>
Message-ID: <1509423893.27574.10.camel@ronnypfannschmidt.de>

I would like to explicitly avoid Hardlink Farms
because those still have "Logical" duplication
i'd like to bind in the new paths without having it look like each
virtualenv is 400-1000 mb of distinct data

-- Ronny

Am Montag, den 30.10.2017, 22:23 +0000 schrieb Thomas Kluyver:
> On Mon, Oct 30, 2017, at 07:16 PM, RonnyPfannschmidt wrote:
> > in order to elevate those issues i would like to propose a new
> > installation layout,
> > where instead of storing each distribution in every python all
> > distributions would share a storage, and each individual
> > environment
> > would only have references to the packages that where
> > "installed/activated" for them
> 
> This is also essentially what conda does - the references being in
> the
> form of hard links. The mechanism has some drawbacks of its own -
> like
> if a file somehow gets modified, it's harder to fix it, because
> removing
> the environment no longer removes the files.
> 
> Thomas
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

From ncoghlan at gmail.com  Tue Oct 31 02:25:08 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 31 Oct 2017 16:25:08 +1000
Subject: [Distutils] What's the use case of testpypi?
In-Reply-To: <CABVPEKpYmmw_ZSD7dVa2eXOQwTFhQLSLpjk3LhqRqG749T+CiQ@mail.gmail.com>
References: <CABVPEKpYmmw_ZSD7dVa2eXOQwTFhQLSLpjk3LhqRqG749T+CiQ@mail.gmail.com>
Message-ID: <CADiSq7ckpABeu9aCK3q1RTpMTQfdjKjy_Xi4KSZvnWf7=zAY2w@mail.gmail.com>

On 31 October 2017 at 03:41, Toshio Kuratomi <a.badger at gmail.com> wrote:

> When we locked down pypi to prevent uploading an sdist to overwrite a
> previous I remember that some people wanted a brief window to check
> for brown paper bag issues and be able to upload a new tarball in that
> window if needed.  IIRC, those people were told to use testpypi for
> that sort of thing.  Upload potential tarball to testpypi.  If it
> works, go ahead and rerun to upload to the real pypi.  If it doesn't,
> fix it and try again.
>

Ideally we'd be recommending
https://devpi.net/docs/devpi/devpi/stable/%2Bd/index.html to folks looking
to develop a robust pre-release artifact testing workflow.

While we mention it a couple of times on packaging.python.org [1,2], and
include it in the packaging related Non-PyPA Projects list [3], we don't
really emphasise that it makes a much better platform for pre-release
testing and private experimentation than PyPI itself does (see
https://devpi.net/ for an example of a deployed instance with multiple
distinct user namespaces).

Given Donald's comment about the current test PyPI not being particularly
good at any of its roles, perhaps it would make sense to aim to replace it
with a periodically purged DevPi instance?

Cheers,
Nick.

[1]
https://packaging.python.org/guides/index-mirrors-and-caches/?highlight=devpi#caching-with-devpi
[2]
https://packaging.python.org/guides/hosting-your-own-index/?highlight=devpi
[3]
https://packaging.python.org/key_projects/?highlight=devpi#non-pypa-projects

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171031/246f21c4/attachment-0001.html>

From ncoghlan at gmail.com  Tue Oct 31 03:22:03 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 31 Oct 2017 17:22:03 +1000
Subject: [Distutils] [proposal] shared distribution installations
In-Reply-To: <8e2d6eaa-cfac-dde2-0b4c-50015211447f@ronnypfannschmidt.de>
References: <8e2d6eaa-cfac-dde2-0b4c-50015211447f@ronnypfannschmidt.de>
Message-ID: <CADiSq7dO=x=twKtd+2Ce_JzOTouGObgg=ZvdLd6RPs66r1_Q_w@mail.gmail.com>

On 31 October 2017 at 05:16, RonnyPfannschmidt <
opensource at ronnypfannschmidt.de> wrote:

> Hi everyone,
>
> since a while now various details of installing python packages in
> virtualenvs caused me grief
>
> a) typically each tox folder in a project is massive, and has a lot of
> duplicate files, recreating them, managing and iterating them takes
> quite a while
> b) for nicely separated deployments, each virtualenv for an application
> takes a few hundred megabytes - that quickly can saturate disk space
> even if a reasonable amount was reserved
> c) installation and recreation of virtualenvs with the same set of
> packages takes quite a while (even with pip caches this is slow, and
> there is no good reason to avoid making it completely instantaneous)
>
> in order to elevate those issues i would like to propose a new
> installation layout,
> where instead of storing each distribution in every python all
> distributions would share a storage, and each individual environment
> would only have references to the packages that where
> "installed/activated" for them
>

I've spent a fair bit of time pondering this problem (since distros care
about it in relation to ease of security updates), and the combination of
Python's import semantics with the PEP 376 installation database semantics
makes it fairly tricky to improve. Fortunately, the pth-file mechanism
provides an escape hatch that makes it possible to transparently experiment
with difference approaches.

At the venv management layer, pew already supports a model similar to that
offered by the Flatpak application container format [1]: instead of
attempting to share everything, pew permits a limited form of "virtual
environment inheritance", via "pew add $(pew dir
<named-venv-to-depend-on>)" (which injects a *.pth file that appends the
other venv's site-packages directory to sys.path). Those inherited runtimes
then become the equivalent of the runtime layer in Flatpak: applications
will automatically pick up new versions of the runtime, so the runtime
maintainers are expected to strictly preserve backwards compatibility, and
when that isn't possible, provide a new parallel-installable version, so
apps using both the old and the new runtime can happily run side-by-side.

The idea behind that approach is to trade-off a bit of inflexibility in the
exact versions of some of your dependencies for the benefit of a reduction
in data duplication on systems running multiple applications or
environments: instead of specifying your full dependency set, you'd instead
only specify that you depended on a particular common computational
environment being available, plus whatever you needed that isn't part of
the assumed platform.

As semi-isolated-applications-with-a-shared-runtime mechanisms like Flatpak
gain popularity (vs fully isolated application & service silos), I'd expect
this model to start making more of an appearance in the Linux distro world,
as it's a natural way of mapping per-application venvs to the shared
runtime model, and it doesn't require any changes to installers or
applications to support it.

However, there's another approach that specifically tackles the content
duplication problem, which would require a new installation layout as you
suggest, but could still rely on *.pth files to make it implicitly
compatible with existing packages and applications and existing Python
runtime versions.

That approach is to create an install tree somewhere that looks like this:

    _shared-packages/
        <normalised-package-name>/
            <release-version>/
                <version-details>.dist-info/
                <installed-files>

Instead of installing full packages directly into a venv the way pip does,
an installer that worked this way would instead manage a
<normalised-package-name>.pth file that indicated
"_shared-packages/<normalised-package-name>/<release-version>" should be
added to sys.path. Each shared package directory could include references
back to all of the venvs where it has been installed, allowing it to be
removed when either all of those have been updated to a new version, or
else removed entirely. This is actually a *lot* like the way
pkg_resources.requires() and self-contained egg directories work, but with
the version selection shifted to the venv's site-packages directory, rather
than happening implicitly in Python code on application startup.

An interesting point about this layout is that it would be amenable to a
future enhancement that allowed for more relaxed MAJOR and MAJOR.MINOR
qualifiers on the install directory references, permitting transparently
shared maintenance and security updates.

The big downside of this layout is that it means you lose the ability to
just bundle up an entire directory and unpack it on a different machine to
get a probably-mostly-working environment. This means that while it's
likely better for managing lots of environments on a single workstation
(due to the reduced file duplication), it's likely to be worse for folks
that work on only a handful of different projects at any given point in
time (and I say that as someone with ~140 different local repository clones
across my ~/devel, ~/fedoradevel and ~/rhdevel directories).

Cheers,
Nick.

[1] http://docs.flatpak.org/en/latest/introduction.html#how-it-works

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171031/18c05817/attachment.html>

From marius at gedmin.as  Mon Oct 30 09:21:32 2017
From: marius at gedmin.as (Marius Gedminas)
Date: Mon, 30 Oct 2017 15:21:32 +0200
Subject: [Distutils] Disabling non HTTPS access to APIs on PyPI
In-Reply-To: <CAK90gp5zFsvYPEcRyf7sa18+qXADqOayH2ey_n28ou-uMpJJ0Q@mail.gmail.com>
References: <CAK90gp6mU__+6HopzL+=8RxogkaztPh1fdoE53=Y42rnBTWrjA@mail.gmail.com>
 <CAPJVwB=JvFyzDA2r77BxSjCMhURHouHKSXW1GmPRjs=gzdQxAA@mail.gmail.com>
 <CAPJVwBmvZODgFAwBC+UG+jnLWLNDnEnSG6B_tLmq-E-CmB1ZbQ@mail.gmail.com>
 <CAMxX_oMstS2vHCre_pE7TT-5Zj98bdetnxzNJ4344GS1rBgQcg@mail.gmail.com>
 <CACac1F-Mrja6dhLsQKuwA-ZZwzokRtyxq5icE4oxRHgapjpuPw@mail.gmail.com>
 <CAK90gp5zFsvYPEcRyf7sa18+qXADqOayH2ey_n28ou-uMpJJ0Q@mail.gmail.com>
Message-ID: <20171030132132.a75o6yrkgzbk3td6@platonas>

On Sat, Oct 28, 2017 at 12:22:32AM +0300, Alex Domoradov wrote:
> I got it. And what I should do with old system? For e.g. we still use ubuntu
> 12.04. Is there any way to upgrade pip/setuptools?

If you're using Ubuntu 12.04, then presumably you're paying Canoncial
for extended support, so ask them to provide a pip/setuptools SRU.

(If you're not paying Canonical, then you're not getting security
updates and should upgrade ASAP.)

Marius Gedminas
-- 
Favorite MS-DOS error message: "Drive C: not ready, close door."
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171030/87f641e5/attachment.sig>

From leorochael at gmail.com  Tue Oct 31 08:13:36 2017
From: leorochael at gmail.com (Leonardo Rochael Almeida)
Date: Tue, 31 Oct 2017 10:13:36 -0200
Subject: [Distutils] [proposal] shared distribution installations
In-Reply-To: <CADiSq7dO=x=twKtd+2Ce_JzOTouGObgg=ZvdLd6RPs66r1_Q_w@mail.gmail.com>
References: <8e2d6eaa-cfac-dde2-0b4c-50015211447f@ronnypfannschmidt.de>
 <CADiSq7dO=x=twKtd+2Ce_JzOTouGObgg=ZvdLd6RPs66r1_Q_w@mail.gmail.com>
Message-ID: <CAMxX_oO_0DL1HPrPSbmnvuW1wDtDNjJRbSs0MokUm29ChFs9-Q@mail.gmail.com>

Hi,

On 31 October 2017 at 05:22, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 31 October 2017 at 05:16, RonnyPfannschmidt <
> opensource at ronnypfannschmidt.de> wrote:
>
>> Hi everyone,
>>
>> since a while now various details of installing python packages in
>> virtualenvs caused me grief
>>
>> a) typically each tox folder in a project is massive, and has a lot of
>> duplicate files, recreating them, managing and iterating them takes
>> quite a while
>> b) for nicely separated deployments, each virtualenv for an application
>> takes a few hundred megabytes - that quickly can saturate disk space
>> even if a reasonable amount was reserved
>> c) installation and recreation of virtualenvs with the same set of
>> packages takes quite a while (even with pip caches this is slow, and
>> there is no good reason to avoid making it completely instantaneous)
>>
>
Those are issues that buildout has solved long before pip was even around,
but they rely on sys.path expansion that Ronny found objectionable due to
performance issues.

I don't think the performance issues are that problematic (and wasn't there
some work on Python 3 that made import faster even with long sys.paths?).


> [...]
>
> However, there's another approach that specifically tackles the content
> duplication problem, which would require a new installation layout as you
> suggest, but could still rely on *.pth files to make it implicitly
> compatible with existing packages and applications and existing Python
> runtime versions.
>
> That approach is to create an install tree somewhere that looks like this:
>
>     _shared-packages/
>         <normalised-package-name>/
>             <release-version>/
>                 <version-details>.dist-info/
>                 <installed-files>
>
> Instead of installing full packages directly into a venv the way pip does,
> an installer that worked this way would instead manage a
> <normalised-package-name>.pth file that indicated
> "_shared-packages/<normalised-package-name>/<release-version>" should be
> added to sys.path.
>

This solution is nice, but preserves the long sys.path that Ronny wanted to
avoid in the first place.

Another detail that needs mentioning is that, for .pth based sys.path
manipulation to work, the <installed-files> would need to be all the files
from purelib and platlib directories from wheels mashed together instead of
a simple unpacking of the wheel (though I guess the .pth file could add
both purelib and platlib subfolders to sys.path...)

Another possibility that avoids the issue of long.syspath is to use this
layout but with symlink farms instead of either sys.path manipulation or
conda-like hard-linking.

Symlinks would preserve better filesystem size visibility that Ronny wanted
while allowing the layout above to contain wheels that were simply unzipped.

In Windows, where symlinks require admin privileges (though this is changing
<https://blogs.windows.com/buildingapps/2016/12/02/symlinks-windows-10/>),
an option could be provided for using hard links instead (which never
require elevated privileges).

Using symlinks into the above layout preserves all advantages and drawbacks
Nick mentioned other than the sys.path expansion.

Regards,

Leo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171031/8a951982/attachment-0001.html>

From fungi at yuggoth.org  Tue Oct 31 08:49:13 2017
From: fungi at yuggoth.org (Jeremy Stanley)
Date: Tue, 31 Oct 2017 12:49:13 +0000
Subject: [Distutils] What's the use case of testpypi?
In-Reply-To: <CADiSq7ckpABeu9aCK3q1RTpMTQfdjKjy_Xi4KSZvnWf7=zAY2w@mail.gmail.com>
References: <CABVPEKpYmmw_ZSD7dVa2eXOQwTFhQLSLpjk3LhqRqG749T+CiQ@mail.gmail.com>
 <CADiSq7ckpABeu9aCK3q1RTpMTQfdjKjy_Xi4KSZvnWf7=zAY2w@mail.gmail.com>
Message-ID: <20171031124912.GD28203@yuggoth.org>

On 2017-10-31 16:25:08 +1000 (+1000), Nick Coghlan wrote:
> Ideally we'd be recommending
> https://devpi.net/docs/devpi/devpi/stable/%2Bd/index.html to folks looking
> to develop a robust pre-release artifact testing workflow.
> 
> While we mention it a couple of times on packaging.python.org [1,2], and
> include it in the packaging related Non-PyPA Projects list [3], we don't
> really emphasise that it makes a much better platform for pre-release
> testing and private experimentation than PyPI itself does (see
> https://devpi.net/ for an example of a deployed instance with multiple
> distinct user namespaces).
> 
> Given Donald's comment about the current test PyPI not being particularly
> good at any of its roles, perhaps it would make sense to aim to replace it
> with a periodically purged DevPi instance?
[...]

Do the two share enough common code for successful uploading to a
devpi instance to be indicative of whether PyPI will accept or
reject on the grounds of, e.g., invalid trove classifiers (this one
in particular has been the most common preventable but otherwise
untestable upload failure our community encounters).
-- 
Jeremy Stanley
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171031/773f649f/attachment.sig>

From vano at mail.mipt.ru  Tue Oct 31 02:28:10 2017
From: vano at mail.mipt.ru (Ivan Pozdeev)
Date: Tue, 31 Oct 2017 09:28:10 +0300
Subject: [Distutils] Fwd: Re: [Python-ideas] Add processor generation to
 wheel metadata
In-Reply-To: <CAPJVwBnwxW0R-xtSDu9wPSyvb5WDk3VHsQc4+p+FyjehE1jivw@mail.gmail.com>
References: <CAPJVwBnwxW0R-xtSDu9wPSyvb5WDk3VHsQc4+p+FyjehE1jivw@mail.gmail.com>
Message-ID: <4d49183c-d1f5-136b-f185-87fa19575bc3@mail.mipt.ru>

Forwarding here as suggested.

-------- Forwarded Message --------
Subject: 	Re: [Python-ideas] Add processor generation to wheel metadata
Date: 	Mon, 30 Oct 2017 23:02:08 -0700
From: 	Nathaniel Smith <njs at pobox.com>
To: 	Ivan Pozdeev <vano at mail.mipt.ru>
CC: 	python-ideas at python.org <python-ideas at python.org>



On Mon, Oct 30, 2017 at 5:45 AM, Ivan Pozdeev via Python-ideas
<python-ideas at python.org> wrote:
> Generally, packages are compiled for the same processor generation as the
> corresponding Python.
> But not always -- e.g. NumPy opted for SSE2 even for Py2 to work around some
> compiler bug
> (https://github.com/numpy/numpy/issues/6428).
> I was bitten by that at an old machine once and found out that there is no
> way for `pip' to have checked for that.
> Besides, performance-oriented packages like the one mentioned could probably
> benefit from newer instructions.

You should probably resend this to distutils-sig instead of
python-ideas -- that's where discussions about python packaging
happen. (Python-ideas is more for discussions about the language
itself.)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org


From thomas at kluyver.me.uk  Tue Oct 31 09:09:34 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Tue, 31 Oct 2017 13:09:34 +0000
Subject: [Distutils] What's the use case of testpypi?
In-Reply-To: <20171031124912.GD28203@yuggoth.org>
References: <CABVPEKpYmmw_ZSD7dVa2eXOQwTFhQLSLpjk3LhqRqG749T+CiQ@mail.gmail.com>
 <CADiSq7ckpABeu9aCK3q1RTpMTQfdjKjy_Xi4KSZvnWf7=zAY2w@mail.gmail.com>
 <20171031124912.GD28203@yuggoth.org>
Message-ID: <1509455374.4074121.1156779656.78AAC65B@webmail.messagingengine.com>

On Tue, Oct 31, 2017, at 12:49 PM, Jeremy Stanley wrote:
> Do the two share enough common code for successful uploading to a
> devpi instance to be indicative of whether PyPI will accept or
> reject on the grounds of, e.g., invalid trove classifiers (this one
> in particular has been the most common preventable but otherwise
> untestable upload failure our community encounters).

It's feasible for tools to test this before upload. Flit does this. If
anyone wants to adapt the code to use in other tools, it's here:

https://github.com/takluyver/flit/blob/ca081197df229a0df07e98c6dece13e62e2bdcd1/flit/inifile.py#L63-L108

It's definitely useful to have a testing site based on the same code as
PyPI, but I hope that with smarter tools, building invalid packages
should be less common.

Thomas

From ncoghlan at gmail.com  Tue Oct 31 09:47:52 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 31 Oct 2017 23:47:52 +1000
Subject: [Distutils] [proposal] shared distribution installations
In-Reply-To: <CAMxX_oO_0DL1HPrPSbmnvuW1wDtDNjJRbSs0MokUm29ChFs9-Q@mail.gmail.com>
References: <8e2d6eaa-cfac-dde2-0b4c-50015211447f@ronnypfannschmidt.de>
 <CADiSq7dO=x=twKtd+2Ce_JzOTouGObgg=ZvdLd6RPs66r1_Q_w@mail.gmail.com>
 <CAMxX_oO_0DL1HPrPSbmnvuW1wDtDNjJRbSs0MokUm29ChFs9-Q@mail.gmail.com>
Message-ID: <CADiSq7dW9MoCdp_cS9BB+emOVPuegpOzV35v5ftJGAY1OYvhDg@mail.gmail.com>

On 31 October 2017 at 22:13, Leonardo Rochael Almeida <leorochael at gmail.com>
wrote:

>
> Those are issues that buildout has solved long before pip was even around,
> but they rely on sys.path expansion that Ronny found objectionable due to
> performance issues.
>

The combination of network drives and lots of sys.path entries could lead
to *awful* startup times with the old stat-based import model (which Python
2.7 still uses by default).

The import system in Python 3.3+ relies on cached os.listdir() results
instead, and after we switched to that, we received at least one report
from a HPC operator of batch jobs that used to take 100+ seconds to start
when importing modules from NFS dropped down to startup times measured in
hundreds of milliseconds - most of the time was previously being lost to
network round trips for failed stat calls that just reported that the file
didn't exist. Even on spinning disks, the new import system gained back
most of the speed that was lost in the switch from low level C to more
maintainable and portable Python code.

An org that runs large rendering farms also reported significantly
improving their batch job startup times in 2.7 by switching to importlib2
(which backports the Py3 import implementation).


> I don't think the performance issues are that problematic (and wasn't
> there some work on Python 3 that made import faster even with long
> sys.paths?).
>

As soon as you combined the old import model with network drives, your
startup times could quickly become intolerable, even with short sys.path
entries - failing imports, and imports that get satisfied later in the path
just end up taking too long.

I wouldn't call it a *completely* solved problem in Py3 (there are still
some application startup related activities that scale linearly with the
length of sys.path), but the worst offender (X stat calls by Y sys.path
entries, taking Z milliseconds per call) is gone.


> On 31 October 2017 at 05:22, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>
>> [...]
>>
>> However, there's another approach that specifically tackles the content
>> duplication problem, which would require a new installation layout as you
>> suggest, but could still rely on *.pth files to make it implicitly
>> compatible with existing packages and applications and existing Python
>> runtime versions.
>>
>> That approach is to create an install tree somewhere that looks like this:
>>
>>     _shared-packages/
>>         <normalised-package-name>/
>>             <release-version>/
>>                 <version-details>.dist-info/
>>                 <installed-files>
>>
>> Instead of installing full packages directly into a venv the way pip
>> does, an installer that worked this way would instead manage a
>> <normalised-package-name>.pth file that indicated
>> "_shared-packages/<normalised-package-name>/<release-version>" should be
>> added to sys.path.
>>
>
> This solution is nice, but preserves the long sys.path that Ronny wanted
> to avoid in the first place.
>
> Another detail that needs mentioning is that, for .pth based sys.path
> manipulation to work, the <installed-files> would need to be all the files
> from purelib and platlib directories from wheels mashed together instead of
> a simple unpacking of the wheel (though I guess the .pth file could add
> both purelib and platlib subfolders to sys.path...)
>

Virtual environments already tend to mash those file types together anyway
- it's mainly Linux system packages that separate them out.


> Another possibility that avoids the issue of long.syspath is to use this
> layout but with symlink farms instead of either sys.path manipulation or
> conda-like hard-linking.
>
> Symlinks would preserve better filesystem size visibility that Ronny
> wanted while allowing the layout above to contain wheels that were simply
> unzipped.
>

Yeah, one thing I really like about that install layout is that it
separates the question of "the installed package layout" from how that
package gets linked into a virtual environment. If you're only doing exact
version matches, then you can use symlinks quite happily, since you don't
need to cope with the name of the "dist-info" directory changing. However,
if you're going to allow for transparent maintenance updates (and hence
version number changes in the dist-info directory name), then you need a
*.pth file.


> In Windows, where symlinks require admin privileges (though this is
> changing
> <https://blogs.windows.com/buildingapps/2016/12/02/symlinks-windows-10/>),
> an option could be provided for using hard links instead (which never
> require elevated privileges).
>

Huh, interesting - I never knew that Windows offered unprivileged hard link
support. I wonder if the venv module could be updated to offer that as an
alternative to copying when symlinks aren't available.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171031/bdf7ec25/attachment-0001.html>

From fungi at yuggoth.org  Tue Oct 31 09:53:03 2017
From: fungi at yuggoth.org (Jeremy Stanley)
Date: Tue, 31 Oct 2017 13:53:03 +0000
Subject: [Distutils] What's the use case of testpypi?
In-Reply-To: <1509455374.4074121.1156779656.78AAC65B@webmail.messagingengine.com>
References: <CABVPEKpYmmw_ZSD7dVa2eXOQwTFhQLSLpjk3LhqRqG749T+CiQ@mail.gmail.com>
 <CADiSq7ckpABeu9aCK3q1RTpMTQfdjKjy_Xi4KSZvnWf7=zAY2w@mail.gmail.com>
 <20171031124912.GD28203@yuggoth.org>
 <1509455374.4074121.1156779656.78AAC65B@webmail.messagingengine.com>
Message-ID: <20171031135303.GE28203@yuggoth.org>

On 2017-10-31 13:09:34 +0000 (+0000), Thomas Kluyver wrote:
[...]
> If anyone wants to adapt the code to use in other tools, it's
> here:
> 
> https://github.com/takluyver/flit/blob/ca08119/flit/inifile.py#L63-L108
[...]

Thanks for the pointer! That would make for a really great
stand-alone linting utility or maybe a flake8 plug-in (particularly
if it vendored a copy of the classifiers list and included an
option/flag to skip downloading).
-- 
Jeremy Stanley
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171031/46aa20f8/attachment.sig>

From ncoghlan at gmail.com  Tue Oct 31 09:53:53 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 31 Oct 2017 23:53:53 +1000
Subject: [Distutils] What's the use case of testpypi?
In-Reply-To: <20171031124912.GD28203@yuggoth.org>
References: <CABVPEKpYmmw_ZSD7dVa2eXOQwTFhQLSLpjk3LhqRqG749T+CiQ@mail.gmail.com>
 <CADiSq7ckpABeu9aCK3q1RTpMTQfdjKjy_Xi4KSZvnWf7=zAY2w@mail.gmail.com>
 <20171031124912.GD28203@yuggoth.org>
Message-ID: <CADiSq7e=6zqmwA-sLUAjRHQbKN=8-5ThF-1Q3PAnfRDeknFX2A@mail.gmail.com>

On 31 October 2017 at 22:49, Jeremy Stanley <fungi at yuggoth.org> wrote:

> On 2017-10-31 16:25:08 +1000 (+1000), Nick Coghlan wrote:
> > Ideally we'd be recommending
> > https://devpi.net/docs/devpi/devpi/stable/%2Bd/index.html to folks
> looking
> > to develop a robust pre-release artifact testing workflow.
> >
> > While we mention it a couple of times on packaging.python.org [1,2], and
> > include it in the packaging related Non-PyPA Projects list [3], we don't
> > really emphasise that it makes a much better platform for pre-release
> > testing and private experimentation than PyPI itself does (see
> > https://devpi.net/ for an example of a deployed instance with multiple
> > distinct user namespaces).
> >
> > Given Donald's comment about the current test PyPI not being particularly
> > good at any of its roles, perhaps it would make sense to aim to replace
> it
> > with a periodically purged DevPi instance?
> [...]
>
> Do the two share enough common code for successful uploading to a
> devpi instance to be indicative of whether PyPI will accept or
> reject on the grounds of, e.g., invalid trove classifiers (this one
> in particular has been the most common preventable but otherwise
> untestable upload failure our community encounters).
>

Even if they don't currently, that's likely an easier problem to fix than
working out how to intermingle devpi's "pre-release developer sandbox" user
experience with PyPI's "authoritative release publishing platform" user
experience in one service, especially since the Legacy PyPI -> Warehouse
migration is already encouraging several of those validation features to be
extracted out to helper libraries.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171031/e174de25/attachment.html>

From vano at mail.mipt.ru  Tue Oct 31 09:56:06 2017
From: vano at mail.mipt.ru (Ivan Pozdeev)
Date: Tue, 31 Oct 2017 16:56:06 +0300
Subject: [Distutils] Fwd: Re: [Python-ideas] Add processor generation to
 wheel metadata
In-Reply-To: <4d49183c-d1f5-136b-f185-87fa19575bc3@mail.mipt.ru>
References: <CAPJVwBnwxW0R-xtSDu9wPSyvb5WDk3VHsQc4+p+FyjehE1jivw@mail.gmail.com>
 <4d49183c-d1f5-136b-f185-87fa19575bc3@mail.mipt.ru>
Message-ID: <1fd22c98-32f3-3b0d-2dd1-8b90bf65083a@mail.mipt.ru>

Missed the fact that Nathaniel didn't quote the entire original message. 
Here it is:

------------------------------------------------------------------------

Generally, packages are compiled for the same processor generation as 
the corresponding Python.
But not always -- e.g. NumPy opted for SSE2 even for Py2 to work around 
some compiler bug
(https://github.com/numpy/numpy/issues/6428).
I was bitten by that at an old machine once and found out that there is 
no way for `pip' to have checked for that.
Besides, performance-oriented packages like the one mentioned could 
probably benefit from newer instructions.

Regarding identifiers:
gcc, cl and clang all have their private directories of generation 
identifiers:
https://gcc.gnu.org/onlinedocs/gcc-4.7.1/gcc/i386-and-x86_002d64-Options.html 

https://msdn.microsoft.com/en-us/library/7t5yh4fd.aspx
https://clang.llvm.org/doxygen/Driver_2ToolChains_2Arch_2X86_8cpp_source.html 


Linux packages typically use gcc's ones. Clang generally follows in 
gcc's footsteps and accepts cl's IDs, too, as aliases.

So, using the IDs of whatever compiler is used to build the package 
(i.e. most likely the canonical compiler for CPython for that platform) 
looks like the simple&stupid(r) way - we can just take the value of the 
"march" argument.


The tricky part is mapping the system's processor to an ID when checking 
compatibility: the logic will have to keep a directory (that's the job 
of `wheel' package maintainers though, I guess).
I also guess that there are cases where there's no such thing as _the_ 
system's processor.

-- 
Regards,
Ivan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171031/b6f3aee4/attachment.html>

From ncoghlan at gmail.com  Tue Oct 31 10:01:39 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 1 Nov 2017 00:01:39 +1000
Subject: [Distutils] What's the use case of testpypi?
In-Reply-To: <CADiSq7e=6zqmwA-sLUAjRHQbKN=8-5ThF-1Q3PAnfRDeknFX2A@mail.gmail.com>
References: <CABVPEKpYmmw_ZSD7dVa2eXOQwTFhQLSLpjk3LhqRqG749T+CiQ@mail.gmail.com>
 <CADiSq7ckpABeu9aCK3q1RTpMTQfdjKjy_Xi4KSZvnWf7=zAY2w@mail.gmail.com>
 <20171031124912.GD28203@yuggoth.org>
 <CADiSq7e=6zqmwA-sLUAjRHQbKN=8-5ThF-1Q3PAnfRDeknFX2A@mail.gmail.com>
Message-ID: <CADiSq7f3BeuKgCe9GoR3PYew3C2ot5TZwaqHAZL2ocW1p8vmTg@mail.gmail.com>

On 31 October 2017 at 23:53, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 31 October 2017 at 22:49, Jeremy Stanley <fungi at yuggoth.org> wrote:
>
>> On 2017-10-31 16:25:08 +1000 (+1000), Nick Coghlan wrote:
>> > Ideally we'd be recommending
>> > https://devpi.net/docs/devpi/devpi/stable/%2Bd/index.html to folks
>> looking
>> > to develop a robust pre-release artifact testing workflow.
>> >
>> > While we mention it a couple of times on packaging.python.org [1,2],
>> and
>> > include it in the packaging related Non-PyPA Projects list [3], we don't
>> > really emphasise that it makes a much better platform for pre-release
>> > testing and private experimentation than PyPI itself does (see
>> > https://devpi.net/ for an example of a deployed instance with multiple
>> > distinct user namespaces).
>> >
>> > Given Donald's comment about the current test PyPI not being
>> particularly
>> > good at any of its roles, perhaps it would make sense to aim to replace
>> it
>> > with a periodically purged DevPi instance?
>> [...]
>>
>> Do the two share enough common code for successful uploading to a
>> devpi instance to be indicative of whether PyPI will accept or
>> reject on the grounds of, e.g., invalid trove classifiers (this one
>> in particular has been the most common preventable but otherwise
>> untestable upload failure our community encounters).
>>
>
> Even if they don't currently, that's likely an easier problem to fix than
> working out how to intermingle devpi's "pre-release developer sandbox" user
> experience with PyPI's "authoritative release publishing platform" user
> experience in one service, especially since the Legacy PyPI -> Warehouse
> migration is already encouraging several of those validation features to be
> extracted out to helper libraries.
>

Taking the classifiers list as a specific example: while that's *currently*
stored in the live PyPI database, there isn't anything in particular that
says that that needs to be the authoritative source of the list - it could
be redefined as a cache of a list that PyPI picks up from somewhere else.

It could, for example, be moved into a static data file in the `packaging`
project, and have updates be deployed to the live service by way of new
releases of that project. Tools could then decide for themselves whether to
depend on the static list in the packaging project, or query PyPI to get
the list for whichever version of the library PyPI itself was using.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171101/bad38549/attachment-0001.html>

From wes.turner at gmail.com  Tue Oct 31 12:23:40 2017
From: wes.turner at gmail.com (Wes Turner)
Date: Tue, 31 Oct 2017 12:23:40 -0400
Subject: [Distutils] [Python-ideas] Add processor generation to wheel
 metadata
In-Reply-To: <1fd22c98-32f3-3b0d-2dd1-8b90bf65083a@mail.mipt.ru>
References: <CAPJVwBnwxW0R-xtSDu9wPSyvb5WDk3VHsQc4+p+FyjehE1jivw@mail.gmail.com>
 <4d49183c-d1f5-136b-f185-87fa19575bc3@mail.mipt.ru>
 <1fd22c98-32f3-3b0d-2dd1-8b90bf65083a@mail.mipt.ru>
Message-ID: <CACfEFw_bGuuY-2pyiUxrLbegtRuZHOhSdpvaMtk9npgQJivi7w@mail.gmail.com>

Maybe the anaconda team has some insight on a standard way to capture (&
configure) compiler versions and flags in package metadata?

From
https://www.anaconda.com/blog/developer-blog/announcing-the-release-of-anaconda-distribution-5-0/
:

> The Anaconda 5.0 release used very modern compilers to rebuild almost
everything (~99.5%) provided in the installers for x86 Linux and MacOS.
This enables Anaconda users to get the benefits of the latest compilers?
still allowing support for older operating systems?back to MacOS 10.9 and
CentOS 6. Our own builds of GCC 7.2 (Linux) and Clang 4.0.1 (MacOS) are
used, and every reasonable security flag has been enabled. CFLAGS and
CXXFLAGS are no longer managed by each package; instead compiler activation
sets them globally.
>
> The packages built with the new compilers are in a different channel from
packages built the old way, and as we build out this new channel, we will
eventually be able to change the default experience to only using these
packages. Interested in using this approach to build your own conda
packages? Stay tuned for a more developer-focused blog post!

On Tuesday, October 31, 2017, Ivan Pozdeev via Distutils-SIG <
distutils-sig at python.org> wrote:

> Missed the fact that Nathaniel didn't quote the entire original message.
> Here it is:
> ------------------------------
>
> Generally, packages are compiled for the same processor generation as the
> corresponding Python.
> But not always -- e.g. NumPy opted for SSE2 even for Py2 to work around
> some compiler bug
> (https://github.com/numpy/numpy/issues/6428).
> I was bitten by that at an old machine once and found out that there is no
> way for `pip' to have checked for that.
> Besides, performance-oriented packages like the one mentioned could
> probably benefit from newer instructions.
>
> Regarding identifiers:
> gcc, cl and clang all have their private directories of generation
> identifiers:
> https://gcc.gnu.org/onlinedocs/gcc-4.7.1/gcc/i386-
> and-x86_002d64-Options.html
> https://msdn.microsoft.com/en-us/library/7t5yh4fd.aspx
> https://clang.llvm.org/doxygen/Driver_2ToolChains_
> 2Arch_2X86_8cpp_source.html
>
> Linux packages typically use gcc's ones. Clang generally follows in gcc's
> footsteps and accepts cl's IDs, too, as aliases.
>
> So, using the IDs of whatever compiler is used to build the package (i.e.
> most likely the canonical compiler for CPython for that platform) looks
> like the simple&stupid(r) way - we can just take the value of the "march"
> argument.
>
>
> The tricky part is mapping the system's processor to an ID when checking
> compatibility: the logic will have to keep a directory (that's the job of
> `wheel' package maintainers though, I guess).
> I also guess that there are cases where there's no such thing as _the_
> system's processor.
>
> --
> Regards,
> Ivan
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171031/317a5bbf/attachment.html>

From alex.hha at gmail.com  Tue Oct 31 16:47:16 2017
From: alex.hha at gmail.com (Alex Domoradov)
Date: Tue, 31 Oct 2017 22:47:16 +0200
Subject: [Distutils] Disabling non HTTPS access to APIs on PyPI
In-Reply-To: <20171030132132.a75o6yrkgzbk3td6@platonas>
References: <CAK90gp6mU__+6HopzL+=8RxogkaztPh1fdoE53=Y42rnBTWrjA@mail.gmail.com>
 <CAPJVwB=JvFyzDA2r77BxSjCMhURHouHKSXW1GmPRjs=gzdQxAA@mail.gmail.com>
 <CAPJVwBmvZODgFAwBC+UG+jnLWLNDnEnSG6B_tLmq-E-CmB1ZbQ@mail.gmail.com>
 <CAMxX_oMstS2vHCre_pE7TT-5Zj98bdetnxzNJ4344GS1rBgQcg@mail.gmail.com>
 <CACac1F-Mrja6dhLsQKuwA-ZZwzokRtyxq5icE4oxRHgapjpuPw@mail.gmail.com>
 <CAK90gp5zFsvYPEcRyf7sa18+qXADqOayH2ey_n28ou-uMpJJ0Q@mail.gmail.com>
 <20171030132132.a75o6yrkgzbk3td6@platonas>
Message-ID: <CAK90gp5P9iH5CJ5PmvS5HP+qhZoJ-UwZh0XUmo7LfyLCgQu6xw@mail.gmail.com>

Yes, I know that. But unfortunately it's a legacy, so there is no way :(

I have resolved my issue with the following line in Dockerfile

RUN echo "[easy_install]\n\nindex-url = https://pypi.python.org/simple/" >
~/.pydistutils.cfg

On Mon, Oct 30, 2017 at 3:21 PM, Marius Gedminas <marius at gedmin.as> wrote:

> On Sat, Oct 28, 2017 at 12:22:32AM +0300, Alex Domoradov wrote:
> > I got it. And what I should do with old system? For e.g. we still use
> ubuntu
> > 12.04. Is there any way to upgrade pip/setuptools?
>
> If you're using Ubuntu 12.04, then presumably you're paying Canoncial
> for extended support, so ask them to provide a pip/setuptools SRU.
>
> (If you're not paying Canonical, then you're not getting security
> updates and should upgrade ASAP.)
>
> Marius Gedminas
> --
> Favorite MS-DOS error message: "Drive C: not ready, close door."
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171031/34e57326/attachment.html>