From kdreyer at redhat.com  Mon May  1 23:00:18 2017
From: kdreyer at redhat.com (Ken Dreyer)
Date: Mon, 1 May 2017 21:00:18 -0600
Subject: [Distutils] package ownership transfer: kobo
Message-ID: <CALqRxCyr-CfvC9QBQpm2DwYO5tttnNm0Wg-=zqp_95fONubOzg@mail.gmail.com>

Hi folks,

Would someone please give me access to upload new versions of kobo to PyPI?

I've filed the transfer request [1] a while back, and several users
are asking for new versions. [2]

Someone uploaded a very old version four years ago, and I'd like to
make sure the PyPI copy is up-to-date.

- Ken


[1] https://sourceforge.net/p/pypi/support-requests/632/
[2] https://github.com/release-engineering/kobo/issues/24

From robin at reportlab.com  Thu May  4 05:12:14 2017
From: robin at reportlab.com (Robin Becker)
Date: Thu, 4 May 2017 10:12:14 +0100
Subject: [Distutils] environment replication
Message-ID: <55e67724-da43-d235-eb22-71f2b5876529@chamonix.reportlab.co.uk>

I have a bunch of virtual environments which need replicating from one machine 
to another. In the past rsync was OK, but now with slightly different 
environments I have been using pip freeze to get a list of the installed 
requirements and pip install to carry out changes to the remote environment.

Is there a way to resolve issues where pip reports a different package to the 
one installed eg


the original requirements say

git+http://github.com/django-cumulus/django-cumulus.git at master#egg=django-cumulus

but pip freeze will only say


django-cumulus===1.0.19-22-g506121d

ie we lost the actual installation notation. Given that the git location is 
actually unstable ie the revision can change it seems quite hard to get 
reproducibility in an automatic fashion.

g506121d is derived from the repo hash 
(https://github.com/django-cumulus/django-cumulus/commit/506121d8669e1cce8843d9f76f2767324f8b1e78), 
but I see no way to guess that from the version string unless there some 
standard decoding I can do of the revision.
-- 
Robin Becker

From wes.turner at gmail.com  Thu May  4 07:40:20 2017
From: wes.turner at gmail.com (Wes Turner)
Date: Thu, 4 May 2017 06:40:20 -0500
Subject: [Distutils] environment replication
In-Reply-To: <55e67724-da43-d235-eb22-71f2b5876529@chamonix.reportlab.co.uk>
References: <55e67724-da43-d235-eb22-71f2b5876529@chamonix.reportlab.co.uk>
Message-ID: <CACfEFw-oKX9T5KnRj+vRhuu5z3Q1HGnjJzXr_rQ6uURS_gHHNw@mail.gmail.com>

On Thursday, May 4, 2017, Robin Becker <robin at reportlab.com> wrote:

> I have a bunch of virtual environments which need replicating from one
> machine to another. In the past rsync was OK, but now with slightly
> different environments I have been using pip freeze to get a list of the
> installed requirements and pip install to carry out changes to the remote
> environment.
>
> Is there a way to resolve issues where pip reports a different package to
> the one installed eg
>
>
After you've installed as editable with `pip install - git+
https://GitHub.com/project/team`, pip freeze is not displaying the source
url?

Or did you manually git clone and then `python setup.py install` or `pip
install -e`?



>
> the original requirements say
>
> git+http://github.com/django-cumulus/django-cumulus.git at mast
> er#egg=django-cumulus
>
> but pip freeze will only say
>
>
> django-cumulus===1.0.19-22-g506121d
>
> ie we lost the actual installation notation. Given that the git location
> is actually unstable ie the revision can change it seems quite hard to get
> reproducibility in an automatic fashion.
>
> g506121d is derived from the repo hash (https://github.com/django-cum
> ulus/django-cumulus/commit/506121d8669e1cce8843d9f76f2767324f8b1e78), but
> I see no way to guess that from the version string unless there some
> standard decoding I can do of the revision.


- SemVer.org has a filename field for (previous) git hashes
- https://github.com/kennethreitz/pipenv


> --
> Robin Becker
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170504/720a7705/attachment.html>

From robin at reportlab.com  Thu May  4 09:48:20 2017
From: robin at reportlab.com (Robin Becker)
Date: Thu, 4 May 2017 14:48:20 +0100
Subject: [Distutils] environment replication
In-Reply-To: <CACfEFw-oKX9T5KnRj+vRhuu5z3Q1HGnjJzXr_rQ6uURS_gHHNw@mail.gmail.com>
References: <55e67724-da43-d235-eb22-71f2b5876529@chamonix.reportlab.co.uk>
 <CACfEFw-oKX9T5KnRj+vRhuu5z3Q1HGnjJzXr_rQ6uURS_gHHNw@mail.gmail.com>
Message-ID: <2d792c07-3400-244c-cd9f-f187eb966da2@chamonix.reportlab.co.uk>




On 04/05/2017 12:40, Wes Turner wrote:
> On Thursday, May 4, 2017, Robin Becker <robin at reportlab.com> wrote:
>
>> I have a bunch of virtual environments which need replicating from one
>> machine to another. In the past rsync was OK, but now with slightly
>> different environments I have been using pip freeze to get a list of the
>> installed requirements and pip install to carry out changes to the remote
>> environment.
>>
>> Is there a way to resolve issues where pip reports a different package to
>> the one installed eg
>>
>>
> After you've installed as editable with `pip install - git+
> https://GitHub.com/project/team`, pip freeze is not displaying the source
> url?
>
> Or did you manually git clone and then `python setup.py install` or `pip
> install -e`?
>
>

Unfortunately I didn't do the original install; developers hacking on live 
servers :(, but the requirements.txt file shows just

git+http://github.com/django-cumulus/django-cumulus.git at master#egg=django-cumulus

so I must assume that was how the installation took place. I don't think the 
intent is to edit anything so the developers don't care.

As a test I did the install as above and then I see pip freeze respond with

django-cumulus===1.0.19-22-g506121d


If I try the method you suggest with -e flag then I do get something more

-e 
git+http://github.com/django-cumulus/django-cumulus.git at 506121d8669e1cce8843d9f76f2767324f8b1e78#egg=django_cumulus

that is certainly more usable, but is not ideal as the hacker/developers will 
tend to hack the installation :(

thanks
>
>>
>> the original requirements say
>>
........

-- 
Robin Becker

From vinay_sajip at yahoo.co.uk  Sat May  6 15:33:26 2017
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 6 May 2017 19:33:26 +0000 (UTC)
Subject: [Distutils] ANN: distlib 0.2.5 released on PyPI
References: <1881383229.7179475.1494099206223.ref@mail.yahoo.com>
Message-ID: <1881383229.7179475.1494099206223@mail.yahoo.com>

I've just released version 0.2.5 of distlib on PyPI [1]. For newcomers,
distlib is a library of packaging functionality which is intended to be
usable as the basis for third-party packaging tools.

The main changes in this release are as follows:

* Changed regular expressions to be compatible with 3.6 as regards escape sequences.

* Closed some resource leaks related to XML-RPC proxies.

* Changed wheel processing to look for metadata in metadata.json as well as pydist.json.


* Updated requirement and marker parsing to be compatible with PEP 508.
* Made downloadability a factor in scoring URLs for preferences.


* Removed Python 2.6 from the support list. 

A more detailed change log is available at [2].

Please try it out, and if you find any problems or have any suggestions for
improvements, please give some feedback using the issue tracker! [3]

Regards,


Vinay Sajip

[1] https://pypi.python.org/pypi/distlib/0.2.5
[2] https://goo.gl/M3kQzR
[3] https://bitbucket.org/pypa/distlib/issues/new

From nalini.tamilmani at gmail.com  Tue May  9 05:18:29 2017
From: nalini.tamilmani at gmail.com (Kamalini)
Date: Tue, 9 May 2017 14:48:29 +0530
Subject: [Distutils] Python modules -reg.,
Message-ID: <CAOaWuwqDLvK++O-SmPVuNguOo0h1bRpu2g=WFG3=LPLvnr096w@mail.gmail.com>

Dear sir,

I am Nalini working as Assistant Professor in Velammal Institute of
Technology,chennai.

I am working in python 3.4.4. I would like to import excel for research
purpose.

Kindly guide me to install supporting packages.

Thanks,

S. Nalini
9894812725
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170509/c46b5fc8/attachment.html>

From brett at python.org  Tue May  9 12:04:04 2017
From: brett at python.org (Brett Cannon)
Date: Tue, 09 May 2017 16:04:04 +0000
Subject: [Distutils] Python modules -reg.,
In-Reply-To: <CAOaWuwqDLvK++O-SmPVuNguOo0h1bRpu2g=WFG3=LPLvnr096w@mail.gmail.com>
References: <CAOaWuwqDLvK++O-SmPVuNguOo0h1bRpu2g=WFG3=LPLvnr096w@mail.gmail.com>
Message-ID: <CAP1=2W50imcEswZFH7r1mpKTiNo7bHjE-Uv3hF9f76urxw0Rfg@mail.gmail.com>

This list is for discussing the development of how to install packages in
Python. For suggestions of what to install, I would ask the python-list
mailing list.

On Tue, 9 May 2017 at 02:20 Kamalini <nalini.tamilmani at gmail.com> wrote:

> Dear sir,
>
> I am Nalini working as Assistant Professor in Velammal Institute of
> Technology,chennai.
>
> I am working in python 3.4.4. I would like to import excel for research
> purpose.
>
> Kindly guide me to install supporting packages.
>
> Thanks,
>
> S. Nalini
> 9894812725 <(989)%20481-2725>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170509/e8a96adc/attachment.html>

From p.f.moore at gmail.com  Wed May 10 11:25:42 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 10 May 2017 16:25:42 +0100
Subject: [Distutils] A little bit of fun with package dependencies
Message-ID: <CACac1F9FhaBd+GdbAyd1vKWTxF-KGM9JofS7kGc3Wxy3pG466g@mail.gmail.com>

I was playing with pkg_resources today, and came up with the following
little script. It's nothing special, but I thought it might be of
interest. It uses pkg_resources and networkx to create a graph of
dependencies between the packages installed in your Python
environment.

The output is a GraphML file, that can be loaded into a graph display
tool (I used a tool called yEd). You could probably also output in
other formats, like GraphViz.

I don't know how interesting this will be to others, but I thought I'd
share it - enjoy!

Paul

import networkx as nx
from pkg_resources import Environment

e = Environment()
g = nx.DiGraph()
for name in sorted(e):
    dists = e[name]
    for dist in dists:
        reqs = dist.requires()
        for r in reqs:
            g.add_edge(dist.project_name, r.name)

for n in g.nodes():
    g.node[n]['Label'] = n

nx.write_graphml(g, 'deps.graphml')

From robin at reportlab.com  Fri May 12 07:22:03 2017
From: robin at reportlab.com (Robin Becker)
Date: Fri, 12 May 2017 12:22:03 +0100
Subject: [Distutils] appveyor
Message-ID: <c7a26ce3-10bf-343d-0bcb-02181d012ac5@chamonix.reportlab.co.uk>

I'm looking at building/testing reportlab with appveyor, but need some 
assistance building the C extensions in particular freetype.

I looked at the code for pillow in

https://github.com/python-pillow/Pillow/tree/master/winbuild

along with

https://github.com/python-pillow/Pillow/blob/master/appveyor.yml


but I'm not sure I really understand it terribly well.

In particular does appveyor support 3.5 & 3.6? The config.py compilers list 
seems to refer to versions 7.0 and 7.1, but that doesn't seem to be the 
visualc++ version at all.

Looking at the appveyor.yml do people really reveal their ftp or s3 tokens? Or 
is it more reasonable to upload to a github acccount.
-puzzled-ly yrs-
Robin Becker

From thomas at kluyver.me.uk  Fri May 12 08:24:56 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Fri, 12 May 2017 13:24:56 +0100
Subject: [Distutils] appveyor
In-Reply-To: <c7a26ce3-10bf-343d-0bcb-02181d012ac5@chamonix.reportlab.co.uk>
References: <c7a26ce3-10bf-343d-0bcb-02181d012ac5@chamonix.reportlab.co.uk>
Message-ID: <1494591896.2022718.974415784.0950552C@webmail.messagingengine.com>

On Fri, May 12, 2017, at 12:22 PM, Robin Becker wrote:
> Looking at the appveyor.yml do people really reveal their ftp or s3
> tokens? 

If it's like Travis, any values under 'secure:' are encrypted with a
project-specific key generated by the CI service, so you can have them
in appveyor.yml without revealing them to anyone other than Appveyor.

Thomas

From marius at gedmin.as  Sun May 14 07:08:53 2017
From: marius at gedmin.as (Marius Gedminas)
Date: Sun, 14 May 2017 14:08:53 +0300
Subject: [Distutils] appveyor
In-Reply-To: <c7a26ce3-10bf-343d-0bcb-02181d012ac5@chamonix.reportlab.co.uk>
References: <c7a26ce3-10bf-343d-0bcb-02181d012ac5@chamonix.reportlab.co.uk>
Message-ID: <20170514110853.62rjpjrgqg5kq7sz@platonas>

On Fri, May 12, 2017 at 12:22:03PM +0100, Robin Becker wrote:
> I'm looking at building/testing reportlab with appveyor, but need some
> assistance building the C extensions in particular freetype.
> 
> I looked at the code for pillow in
> 
> https://github.com/python-pillow/Pillow/tree/master/winbuild
> 
> along with
> 
> https://github.com/python-pillow/Pillow/blob/master/appveyor.yml
> 
> but I'm not sure I really understand it terribly well.
> 
> In particular does appveyor support 3.5 & 3.6?

Yes.  Here's how zope.interface builds binary wheels containing C
extension modules for all supported Python versions and uploads them to
PyPI: https://github.com/zopefoundation/zope.interface/blob/master/appveyor.yml

I like how it's short and simple, with only one strange thing I don't
understand (the echo > vcvars64.bat thing).

> Looking at the appveyor.yml do people really reveal their ftp or s3 tokens?
> Or is it more reasonable to upload to a github acccount.

https://www.appveyor.com/docs/build-configuration/#secure-variables

HTH,
Marius Gedminas
-- 
Despite all appearances, your boss is a thinking, feeling, human being.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170514/252354e8/attachment.sig>

From p.f.moore at gmail.com  Mon May 15 08:13:58 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 15 May 2017 13:13:58 +0100
Subject: [Distutils] appveyor
In-Reply-To: <20170514110853.62rjpjrgqg5kq7sz@platonas>
References: <c7a26ce3-10bf-343d-0bcb-02181d012ac5@chamonix.reportlab.co.uk>
 <20170514110853.62rjpjrgqg5kq7sz@platonas>
Message-ID: <CACac1F9tzu7XXowxRPS-aSct=Wackja0DV-qL2oMjPhGyJdzLA@mail.gmail.com>

On 14 May 2017 at 12:08, Marius Gedminas <marius at gedmin.as> wrote:
> I like how it's short and simple, with only one strange thing I don't
> understand (the echo > vcvars64.bat thing).

I think that's patching round the fact that the MS batch files aren't
100% consistent, and you need the SDK compilers for 64-bit builds on
one of the earlier Visual Studio free versions. Sorry I can't recall
the details, but it's basically that.

But I agree, that looks like an impressively simple Appveyor setup.
I'll have to steal it :-)

Paul

From robin at reportlab.com  Mon May 15 12:22:04 2017
From: robin at reportlab.com (Robin Becker)
Date: Mon, 15 May 2017 17:22:04 +0100
Subject: [Distutils] appveyor
In-Reply-To: <20170514110853.62rjpjrgqg5kq7sz@platonas>
References: <c7a26ce3-10bf-343d-0bcb-02181d012ac5@chamonix.reportlab.co.uk>
 <20170514110853.62rjpjrgqg5kq7sz@platonas>
Message-ID: <d72db1a6-d09c-1a12-3006-33727c6637d7@chamonix.reportlab.co.uk>

On 14/05/2017 12:08, Marius Gedminas wrote:
...........
>
> Yes.  Here's how zope.interface builds binary wheels containing C
> extension modules for all supported Python versions and uploads them to
> PyPI: https://github.com/zopefoundation/zope.interface/blob/master/appveyor.yml
>
> I like how it's short and simple, with only one strange thing I don't
> understand (the echo > vcvars64.bat thing).
>
>> Looking at the appveyor.yml do people really reveal their ftp or s3 tokens?
>> Or is it more reasonable to upload to a github acccount.
>
> https://www.appveyor.com/docs/build-configuration/#secure-variables
>
> HTH,
> Marius Gedminas
......
thanks for this Marius.
-- 
Robin Becker

From sylvain.corlay at gmail.com  Mon May 15 13:29:15 2017
From: sylvain.corlay at gmail.com (Sylvain Corlay)
Date: Mon, 15 May 2017 19:29:15 +0200
Subject: [Distutils] On vendoring binaries and headers with pypi packages
Message-ID: <CAK=Phk7eqmPHxm2+nrHzEy2fTvuZZFG-RMzmVJ3c3Ttt_nTrFQ@mail.gmail.com>

Hi,

When distributing python packages that depend on non-python components, the
typical way of doing this with general-purpose package managers (linux
distributions, conda) is to place binaries, headers, configuration and
other artefacts for these components in their respective natural locations
under the installation prefix (e.g. PREFIX/bin/, PREFIX/include/,
PREFIX/etc/,  PREFIX/share/).

Now, if I have a python package that depends on such a thing (e.g. pyzmq vs
zmq), the pypi wheel for pyzmq will vendor a binary for libzmq as package
data.

For the case of the header files, there is the distutils `install_headers`
directive, but the target directory is (typically) under
PREFIX/include/PythonX.Y, which is probably only a good choices for headers
of a C extension linked with this specific python installation.

Would it make sense, when vendoring packages packages that don't depend on
the python version in a wheel to use `data_files` instead, and target
respectively the `bin`, `include`, `etc` and `share` subdirectories of the
prefix?

Specifically, for the headers, one could do


    data_files      = [
        ('include/foobar', [list of the foobar header files])),
    ],

Note: I was just using zmq as an example.

Thanks,

Sylvain
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170515/affca5e3/attachment.html>

From sylvain.corlay at gmail.com  Mon May 15 13:58:14 2017
From: sylvain.corlay at gmail.com (Sylvain Corlay)
Date: Mon, 15 May 2017 19:58:14 +0200
Subject: [Distutils] On vendoring binaries and headers with pypi packages
In-Reply-To: <CAK=Phk7eqmPHxm2+nrHzEy2fTvuZZFG-RMzmVJ3c3Ttt_nTrFQ@mail.gmail.com>
References: <CAK=Phk7eqmPHxm2+nrHzEy2fTvuZZFG-RMzmVJ3c3Ttt_nTrFQ@mail.gmail.com>
Message-ID: <CAK=Phk75VUOhuANLDYy0rroDA8wdHxoXEZ_pKJGVB76=LLpwHA@mail.gmail.com>

The goal in this approach would be to place headers, and binaries in the
same location as if they were installed with a separate package.

On Mon, May 15, 2017 at 7:29 PM, Sylvain Corlay <sylvain.corlay at gmail.com>
wrote:

> Hi,
>
> When distributing python packages that depend on non-python components,
> the typical way of doing this with general-purpose package managers (linux
> distributions, conda) is to place binaries, headers, configuration and
> other artefacts for these components in their respective natural locations
> under the installation prefix (e.g. PREFIX/bin/, PREFIX/include/,
> PREFIX/etc/,  PREFIX/share/).
>
> Now, if I have a python package that depends on such a thing (e.g. pyzmq
> vs zmq), the pypi wheel for pyzmq will vendor a binary for libzmq as
> package data.
>
> For the case of the header files, there is the distutils `install_headers`
> directive, but the target directory is (typically) under
> PREFIX/include/PythonX.Y, which is probably only a good choices for headers
> of a C extension linked with this specific python installation.
>
> Would it make sense, when vendoring packages packages that don't depend on
> the python version in a wheel to use `data_files` instead, and target
> respectively the `bin`, `include`, `etc` and `share` subdirectories of the
> prefix?
>
> Specifically, for the headers, one could do
>
>
>     data_files      = [
>         ('include/foobar', [list of the foobar header files])),
>     ],
>
> Note: I was just using zmq as an example.
>
> Thanks,
>
> Sylvain
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170515/85f690be/attachment.html>

From njs at pobox.com  Mon May 15 15:47:40 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 15 May 2017 12:47:40 -0700
Subject: [Distutils] On vendoring binaries and headers with pypi packages
In-Reply-To: <CAK=Phk7eqmPHxm2+nrHzEy2fTvuZZFG-RMzmVJ3c3Ttt_nTrFQ@mail.gmail.com>
References: <CAK=Phk7eqmPHxm2+nrHzEy2fTvuZZFG-RMzmVJ3c3Ttt_nTrFQ@mail.gmail.com>
Message-ID: <CAPJVwBmsQ2ia5vL9XjEH1MRXG=Rc35JsoaG94FEEKL9ZhBYsVg@mail.gmail.com>

What are you trying to accomplish by putting include files into virtualenvs?

On May 15, 2017 10:29 AM, "Sylvain Corlay" <sylvain.corlay at gmail.com> wrote:

> Hi,
>
> When distributing python packages that depend on non-python components,
> the typical way of doing this with general-purpose package managers (linux
> distributions, conda) is to place binaries, headers, configuration and
> other artefacts for these components in their respective natural locations
> under the installation prefix (e.g. PREFIX/bin/, PREFIX/include/,
> PREFIX/etc/,  PREFIX/share/).
>
> Now, if I have a python package that depends on such a thing (e.g. pyzmq
> vs zmq), the pypi wheel for pyzmq will vendor a binary for libzmq as
> package data.
>
> For the case of the header files, there is the distutils `install_headers`
> directive, but the target directory is (typically) under
> PREFIX/include/PythonX.Y, which is probably only a good choices for headers
> of a C extension linked with this specific python installation.
>
> Would it make sense, when vendoring packages packages that don't depend on
> the python version in a wheel to use `data_files` instead, and target
> respectively the `bin`, `include`, `etc` and `share` subdirectories of the
> prefix?
>
> Specifically, for the headers, one could do
>
>
>     data_files      = [
>         ('include/foobar', [list of the foobar header files])),
>     ],
>
> Note: I was just using zmq as an example.
>
> Thanks,
>
> Sylvain
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170515/4019e714/attachment-0001.html>

From sylvain.corlay at gmail.com  Mon May 15 17:42:39 2017
From: sylvain.corlay at gmail.com (Sylvain Corlay)
Date: Mon, 15 May 2017 23:42:39 +0200
Subject: [Distutils] On vendoring binaries and headers with pypi packages
In-Reply-To: <CAPJVwBmsQ2ia5vL9XjEH1MRXG=Rc35JsoaG94FEEKL9ZhBYsVg@mail.gmail.com>
References: <CAK=Phk7eqmPHxm2+nrHzEy2fTvuZZFG-RMzmVJ3c3Ttt_nTrFQ@mail.gmail.com>
 <CAPJVwBmsQ2ia5vL9XjEH1MRXG=Rc35JsoaG94FEEKL9ZhBYsVg@mail.gmail.com>
Message-ID: <CAK=Phk5C09xa+gRD1Wa-nk7Ei-bAvPeJzPcV=y5uO9tzFvmXfA@mail.gmail.com>

Let's say that you have an anaconda installation, and you install the
latest pyzmq from pypi instead of from the conda recipe.

I would like the vendored libzmq to be placed under `PREFIX/bin` and the
headers under `PREFIX/include`, just as if I had either

 - installed zmq with `cmake -D CMAKE_INSTALL_PREFIX=PREFIX` and `make
install`
 - or installed the zeromq from the conda package.

Similarly, for a system-wide install on linux distribution, the same would
hold with PREFIX=/usr

On Mon, May 15, 2017 at 9:47 PM, Nathaniel Smith <njs at pobox.com> wrote:

> What are you trying to accomplish by putting include files into
> virtualenvs?
>
> On May 15, 2017 10:29 AM, "Sylvain Corlay" <sylvain.corlay at gmail.com>
> wrote:
>
>> Hi,
>>
>> When distributing python packages that depend on non-python components,
>> the typical way of doing this with general-purpose package managers (linux
>> distributions, conda) is to place binaries, headers, configuration and
>> other artefacts for these components in their respective natural locations
>> under the installation prefix (e.g. PREFIX/bin/, PREFIX/include/,
>> PREFIX/etc/,  PREFIX/share/).
>>
>> Now, if I have a python package that depends on such a thing (e.g. pyzmq
>> vs zmq), the pypi wheel for pyzmq will vendor a binary for libzmq as
>> package data.
>>
>> For the case of the header files, there is the distutils
>> `install_headers` directive, but the target directory is (typically) under
>> PREFIX/include/PythonX.Y, which is probably only a good choices for headers
>> of a C extension linked with this specific python installation.
>>
>> Would it make sense, when vendoring packages packages that don't depend
>> on the python version in a wheel to use `data_files` instead, and target
>> respectively the `bin`, `include`, `etc` and `share` subdirectories of the
>> prefix?
>>
>> Specifically, for the headers, one could do
>>
>>
>>     data_files      = [
>>         ('include/foobar', [list of the foobar header files])),
>>     ],
>>
>> Note: I was just using zmq as an example.
>>
>> Thanks,
>>
>> Sylvain
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170515/230cfcd5/attachment.html>

From njs at pobox.com  Mon May 15 17:45:58 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 15 May 2017 14:45:58 -0700
Subject: [Distutils] On vendoring binaries and headers with pypi packages
In-Reply-To: <CAK=Phk5C09xa+gRD1Wa-nk7Ei-bAvPeJzPcV=y5uO9tzFvmXfA@mail.gmail.com>
References: <CAK=Phk7eqmPHxm2+nrHzEy2fTvuZZFG-RMzmVJ3c3Ttt_nTrFQ@mail.gmail.com>
 <CAPJVwBmsQ2ia5vL9XjEH1MRXG=Rc35JsoaG94FEEKL9ZhBYsVg@mail.gmail.com>
 <CAK=Phk5C09xa+gRD1Wa-nk7Ei-bAvPeJzPcV=y5uO9tzFvmXfA@mail.gmail.com>
Message-ID: <CAPJVwBn1isUfrVm3nzU95+-L34MQ+bs-zRM14joX=G99_GjUew@mail.gmail.com>

No, I'm trying to ask, like, why you want that. Presumably you have some
end goal in mind that you think this will help accomplish, and I think it
would be helpful to know what that is :-).

On May 15, 2017 2:42 PM, "Sylvain Corlay" <sylvain.corlay at gmail.com> wrote:

> Let's say that you have an anaconda installation, and you install the
> latest pyzmq from pypi instead of from the conda recipe.
>
> I would like the vendored libzmq to be placed under `PREFIX/bin` and the
> headers under `PREFIX/include`, just as if I had either
>
>  - installed zmq with `cmake -D CMAKE_INSTALL_PREFIX=PREFIX` and `make
> install`
>  - or installed the zeromq from the conda package.
>
> Similarly, for a system-wide install on linux distribution, the same would
> hold with PREFIX=/usr
>
> On Mon, May 15, 2017 at 9:47 PM, Nathaniel Smith <njs at pobox.com> wrote:
>
>> What are you trying to accomplish by putting include files into
>> virtualenvs?
>>
>> On May 15, 2017 10:29 AM, "Sylvain Corlay" <sylvain.corlay at gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> When distributing python packages that depend on non-python components,
>>> the typical way of doing this with general-purpose package managers (linux
>>> distributions, conda) is to place binaries, headers, configuration and
>>> other artefacts for these components in their respective natural locations
>>> under the installation prefix (e.g. PREFIX/bin/, PREFIX/include/,
>>> PREFIX/etc/,  PREFIX/share/).
>>>
>>> Now, if I have a python package that depends on such a thing (e.g. pyzmq
>>> vs zmq), the pypi wheel for pyzmq will vendor a binary for libzmq as
>>> package data.
>>>
>>> For the case of the header files, there is the distutils
>>> `install_headers` directive, but the target directory is (typically) under
>>> PREFIX/include/PythonX.Y, which is probably only a good choices for headers
>>> of a C extension linked with this specific python installation.
>>>
>>> Would it make sense, when vendoring packages packages that don't depend
>>> on the python version in a wheel to use `data_files` instead, and target
>>> respectively the `bin`, `include`, `etc` and `share` subdirectories of the
>>> prefix?
>>>
>>> Specifically, for the headers, one could do
>>>
>>>
>>>     data_files      = [
>>>         ('include/foobar', [list of the foobar header files])),
>>>     ],
>>>
>>> Note: I was just using zmq as an example.
>>>
>>> Thanks,
>>>
>>> Sylvain
>>>
>>>
>>> _______________________________________________
>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170515/78ed8788/attachment.html>

From sylvain.corlay at gmail.com  Mon May 15 18:07:17 2017
From: sylvain.corlay at gmail.com (Sylvain Corlay)
Date: Tue, 16 May 2017 00:07:17 +0200
Subject: [Distutils] On vendoring binaries and headers with pypi packages
In-Reply-To: <CAPJVwBn1isUfrVm3nzU95+-L34MQ+bs-zRM14joX=G99_GjUew@mail.gmail.com>
References: <CAK=Phk7eqmPHxm2+nrHzEy2fTvuZZFG-RMzmVJ3c3Ttt_nTrFQ@mail.gmail.com>
 <CAPJVwBmsQ2ia5vL9XjEH1MRXG=Rc35JsoaG94FEEKL9ZhBYsVg@mail.gmail.com>
 <CAK=Phk5C09xa+gRD1Wa-nk7Ei-bAvPeJzPcV=y5uO9tzFvmXfA@mail.gmail.com>
 <CAPJVwBn1isUfrVm3nzU95+-L34MQ+bs-zRM14joX=G99_GjUew@mail.gmail.com>
Message-ID: <CAK=Phk6n5eLH19RwWnhbuUbu=7prWC9XvLVMVZ3LaQSg6gXEEw@mail.gmail.com>

OK thanks!

I want to create a pypi package for xtensor and xtensor-python, and would
like it to place the headers in the same location as with the conda package
or Debian packages.

(Which are essentially built with a regular
`cmake -D CMAKE_INSTALL_PREFIX=PREFIX; make install`)

The conda package puts things under `CONDA_PREFIX/include/xtensor[-python]`
and the Debian package `/usr/include/xtensor[-python]`.

It seems that wheel's data_files can achieve that very simply, and was
wondering if there was any drawback in this approach.

Sylvain


On Mon, May 15, 2017 at 11:45 PM, Nathaniel Smith <njs at pobox.com> wrote:

> No, I'm trying to ask, like, why you want that. Presumably you have some
> end goal in mind that you think this will help accomplish, and I think it
> would be helpful to know what that is :-).
>
> On May 15, 2017 2:42 PM, "Sylvain Corlay" <sylvain.corlay at gmail.com>
> wrote:
>
>> Let's say that you have an anaconda installation, and you install the
>> latest pyzmq from pypi instead of from the conda recipe.
>>
>> I would like the vendored libzmq to be placed under `PREFIX/bin` and the
>> headers under `PREFIX/include`, just as if I had either
>>
>>  - installed zmq with `cmake -D CMAKE_INSTALL_PREFIX=PREFIX` and `make
>> install`
>>  - or installed the zeromq from the conda package.
>>
>> Similarly, for a system-wide install on linux distribution, the same
>> would hold with PREFIX=/usr
>>
>> On Mon, May 15, 2017 at 9:47 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>>> What are you trying to accomplish by putting include files into
>>> virtualenvs?
>>>
>>> On May 15, 2017 10:29 AM, "Sylvain Corlay" <sylvain.corlay at gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> When distributing python packages that depend on non-python components,
>>>> the typical way of doing this with general-purpose package managers (linux
>>>> distributions, conda) is to place binaries, headers, configuration and
>>>> other artefacts for these components in their respective natural locations
>>>> under the installation prefix (e.g. PREFIX/bin/, PREFIX/include/,
>>>> PREFIX/etc/,  PREFIX/share/).
>>>>
>>>> Now, if I have a python package that depends on such a thing (e.g.
>>>> pyzmq vs zmq), the pypi wheel for pyzmq will vendor a binary for libzmq as
>>>> package data.
>>>>
>>>> For the case of the header files, there is the distutils
>>>> `install_headers` directive, but the target directory is (typically) under
>>>> PREFIX/include/PythonX.Y, which is probably only a good choices for headers
>>>> of a C extension linked with this specific python installation.
>>>>
>>>> Would it make sense, when vendoring packages packages that don't depend
>>>> on the python version in a wheel to use `data_files` instead, and target
>>>> respectively the `bin`, `include`, `etc` and `share` subdirectories of the
>>>> prefix?
>>>>
>>>> Specifically, for the headers, one could do
>>>>
>>>>
>>>>     data_files      = [
>>>>         ('include/foobar', [list of the foobar header files])),
>>>>     ],
>>>>
>>>> Note: I was just using zmq as an example.
>>>>
>>>> Thanks,
>>>>
>>>> Sylvain
>>>>
>>>>
>>>> _______________________________________________
>>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>>
>>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170516/db57721b/attachment-0001.html>

From p.f.moore at gmail.com  Mon May 15 18:09:41 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 15 May 2017 23:09:41 +0100
Subject: [Distutils] On vendoring binaries and headers with pypi packages
In-Reply-To: <CAK=Phk5C09xa+gRD1Wa-nk7Ei-bAvPeJzPcV=y5uO9tzFvmXfA@mail.gmail.com>
References: <CAK=Phk7eqmPHxm2+nrHzEy2fTvuZZFG-RMzmVJ3c3Ttt_nTrFQ@mail.gmail.com>
 <CAPJVwBmsQ2ia5vL9XjEH1MRXG=Rc35JsoaG94FEEKL9ZhBYsVg@mail.gmail.com>
 <CAK=Phk5C09xa+gRD1Wa-nk7Ei-bAvPeJzPcV=y5uO9tzFvmXfA@mail.gmail.com>
Message-ID: <CACac1F86Wm-dnrAxXMVv63JPv-ohKB9-nLQACjpS6za8VBiJag@mail.gmail.com>

On 15 May 2017 at 22:42, Sylvain Corlay <sylvain.corlay at gmail.com> wrote:
> Let's say that you have an anaconda installation, and you install the latest
> pyzmq from pypi instead of from the conda recipe.

One of the key features of conda over pip is that it manages not just
Python packages, but also non-Python dependencies. So you seem to be
asking "if I choose to use the approach that doesn't handle non-Python
dependencies, how can I make it handle non-Python dependencies in the
same way as the solution which does, but which I chose not to use"...

So I'd have to ask, why aren't you able to use the conda build? Making
pip handle non-Python dependencies is a major piece of work that's
been discussed on various occasions on this list, but typically the
debate has boiled down to "given that pip has limited developer
resource, why should we spend it on implementing something that isn't
core to pip and is already handled well by tools like conda?"

Paul.

From sylvain.corlay at gmail.com  Mon May 15 18:44:52 2017
From: sylvain.corlay at gmail.com (Sylvain Corlay)
Date: Tue, 16 May 2017 00:44:52 +0200
Subject: [Distutils] On vendoring binaries and headers with pypi packages
In-Reply-To: <CACac1F86Wm-dnrAxXMVv63JPv-ohKB9-nLQACjpS6za8VBiJag@mail.gmail.com>
References: <CAK=Phk7eqmPHxm2+nrHzEy2fTvuZZFG-RMzmVJ3c3Ttt_nTrFQ@mail.gmail.com>
 <CAPJVwBmsQ2ia5vL9XjEH1MRXG=Rc35JsoaG94FEEKL9ZhBYsVg@mail.gmail.com>
 <CAK=Phk5C09xa+gRD1Wa-nk7Ei-bAvPeJzPcV=y5uO9tzFvmXfA@mail.gmail.com>
 <CACac1F86Wm-dnrAxXMVv63JPv-ohKB9-nLQACjpS6za8VBiJag@mail.gmail.com>
Message-ID: <CAK=Phk4S8dydD9vadETyO2LmM5v6=L0b5vakc1R24hEhHxu1mA@mail.gmail.com>

So I actually do use conda (and also maintain dozens of conda-forge
recipes). but I also need to package things on pypi and would like it to be
as consistent as possible with what general purpose package managers do.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170516/9136901c/attachment.html>

From dholth at gmail.com  Tue May 16 13:22:06 2017
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 16 May 2017 17:22:06 +0000
Subject: [Distutils] On vendoring binaries and headers with pypi packages
In-Reply-To: <CAK=Phk4S8dydD9vadETyO2LmM5v6=L0b5vakc1R24hEhHxu1mA@mail.gmail.com>
References: <CAK=Phk7eqmPHxm2+nrHzEy2fTvuZZFG-RMzmVJ3c3Ttt_nTrFQ@mail.gmail.com>
 <CAPJVwBmsQ2ia5vL9XjEH1MRXG=Rc35JsoaG94FEEKL9ZhBYsVg@mail.gmail.com>
 <CAK=Phk5C09xa+gRD1Wa-nk7Ei-bAvPeJzPcV=y5uO9tzFvmXfA@mail.gmail.com>
 <CACac1F86Wm-dnrAxXMVv63JPv-ohKB9-nLQACjpS6za8VBiJag@mail.gmail.com>
 <CAK=Phk4S8dydD9vadETyO2LmM5v6=L0b5vakc1R24hEhHxu1mA@mail.gmail.com>
Message-ID: <CAG8k2+7EJxNiOvEnTAcuckPKs_ieQeaCWa=hrqx4SE=BDfdhCg@mail.gmail.com>

I started a proposal to add additional install paths to wheel. Maybe
something like this would be useful for your application, would you be
interested in finishing the proposal? If I was still working on it then I
would decouple it from wheel, come up with a good API for finding the
installed files afterwards, and possibly come up with a way to define
custom path prefixes using string templates.
https://www.python.org/dev/peps/pep-0491/#install-paths

Several people broke data_files by accident, long story, setup.py once was
able to reliably install things anywhere but between virtualenv and wheel
etc. this feature has been broken for many years, in other words you can
try to install things with data_files but they will be installed in
unpredictable locations depending on the installer. Unfortunately the
current less flexible data_files is sometimes regarded as a feature.

On Mon, May 15, 2017 at 6:49 PM Sylvain Corlay <sylvain.corlay at gmail.com>
wrote:

> So I actually do use conda (and also maintain dozens of conda-forge
> recipes). but I also need to package things on pypi and would like it to be
> as consistent as possible with what general purpose package managers do.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170516/86ab7f18/attachment.html>

From thomas at kluyver.me.uk  Fri May 19 10:18:03 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Fri, 19 May 2017 15:18:03 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
Message-ID: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>

Hi,

I'd like to make another push for PEP 517, which would make it possible
to build wheels from a source tree with other build tools, without
needing setup.py.

https://www.python.org/dev/peps/pep-0517/

Last time this was discussed we made a couple of minor changes to the
PEP, but we didn't want to accept another packaging related PEP until
PEP 518 was implemented in pip. I'm pleased to say that that
implementation has just been merged:
https://github.com/pypa/pip/pull/4144 .

I have an implementation of a build backend according to the current PEP
517 spec in a pull request on flit:
https://github.com/takluyver/flit/pull/77/files#diff-6df5630df9568322a453f03159fc8fe8

Thanks,
Thomas

From p.f.moore at gmail.com  Fri May 19 10:41:30 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 19 May 2017 15:41:30 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
Message-ID: <CACac1F_V2DcYdzZga8EWYCcLjXvFWC5aeoGaRQZcL3vcRL8TNg@mail.gmail.com>

On 19 May 2017 at 15:18, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> Hi,
>
> I'd like to make another push for PEP 517, which would make it possible
> to build wheels from a source tree with other build tools, without
> needing setup.py.

A point that came up recently while dealing with a pip issue.

"""
All hooks are run with working directory set to the root of the source
tree, and MAY print arbitrary informational text on stdout and stderr.
"""

Can we specify what encoding the informational text must be written
in? At the moment pip has problems dealing with non-ASCII locales
because it captures the build output and then displays it on error.
This involves a decode/encode step (on Python 3) or printing arbitrary
bytes to stdout (on Python 2). And at the moment we get UnicodeErrors
if there's a mismatch. I've patched it to use errors=replace, but we
still risk mojibake.

Ideally, we should specify an encoding that hooks will use for output
- but that's somewhat difficult as many build tools will want to do
things like run compilers which could do arbitrarily silly things. I
believe this is less of a problem on Unix (where there's a
well-managed convention), but on Windows there's an "OEM" codepage for
console programs, and an "ANSI" codepage for Windows programs - but
not all programs use the same one - some console programs such as
Mingw, I think, and Python itself if stdout is redirected (see
https://docs.python.org/3.6/library/sys.html#sys.stdout) use the ANSI
codepage.

So we may have to fudge the situation a bit. (Maybe something like
"Install tools MAY assume a specific encoding for the output, and MAY
produce corrupted output if the build tool does not use that encoding,
but install tools MUST NOT fail with an encoding error just because
the encodings don't match").

But I don't think we should leave the situation completely unspecified.

Paul.

From thomas at kluyver.me.uk  Fri May 19 11:16:23 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Fri, 19 May 2017 16:16:23 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CACac1F_V2DcYdzZga8EWYCcLjXvFWC5aeoGaRQZcL3vcRL8TNg@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F_V2DcYdzZga8EWYCcLjXvFWC5aeoGaRQZcL3vcRL8TNg@mail.gmail.com>
Message-ID: <1495206983.270969.982170216.5CE4EF05@webmail.messagingengine.com>

On Fri, May 19, 2017, at 03:41 PM, Paul Moore wrote:
> Can we specify what encoding the informational text must be written
> in?

Sure, that makes sense. What about:

All hooks are run with working directory set to the root of the source
tree, and MAY print arbitrary informational text on stdout and stderr.
This text SHOULD be UTF-8 encoded, but as building may invoke other
processes, install tools MUST NOT fail if the data they receive is not
valid UTF-8; though in this case the display of the output may be
corrupted.

Do we also want to recommend that install tools set
PYTHONIOENCODING=utf-8 when invoking build tools? Or leave this up to
the build tools?

Thomas

From p.f.moore at gmail.com  Fri May 19 11:31:44 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 19 May 2017 16:31:44 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <1495206983.270969.982170216.5CE4EF05@webmail.messagingengine.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F_V2DcYdzZga8EWYCcLjXvFWC5aeoGaRQZcL3vcRL8TNg@mail.gmail.com>
 <1495206983.270969.982170216.5CE4EF05@webmail.messagingengine.com>
Message-ID: <CACac1F82jZpeW+DHaXPCxVoKc-egFzxC3iBiGNh0gKik1qJ3hQ@mail.gmail.com>

On 19 May 2017 at 16:16, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> On Fri, May 19, 2017, at 03:41 PM, Paul Moore wrote:
>> Can we specify what encoding the informational text must be written
>> in?
>
> Sure, that makes sense. What about:
>
> All hooks are run with working directory set to the root of the source
> tree, and MAY print arbitrary informational text on stdout and stderr.
> This text SHOULD be UTF-8 encoded, but as building may invoke other
> processes, install tools MUST NOT fail if the data they receive is not
> valid UTF-8; though in this case the display of the output may be
> corrupted.

Looks good, although whether UTF-8 is viable on Windows is something
I'll have to think about.

> Do we also want to recommend that install tools set
> PYTHONIOENCODING=utf-8 when invoking build tools? Or leave this up to
> the build tools?

Good question. At the moment, the only 2 cases I know of are
setuptools/distutils and flit. For setuptools, I'm pretty sure there's
no handling of subprocesses, it just fires them off and lets them
write to the console - so there's nothing to even ensure a consistent
encoding. We may have to allow for special casing with setuptools, as
I doubt anyone's going to put in the effort to add a transcoding layer
in there.

For flit, would having the install tool set PYTHONIOENCODING help?

I don't know immediately what I'd do if I were designing a brand new
build tool that called out to a 3rd party compiler. Let me think about
it.
Paul

From thomas at kluyver.me.uk  Fri May 19 11:37:11 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Fri, 19 May 2017 16:37:11 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CACac1F82jZpeW+DHaXPCxVoKc-egFzxC3iBiGNh0gKik1qJ3hQ@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F_V2DcYdzZga8EWYCcLjXvFWC5aeoGaRQZcL3vcRL8TNg@mail.gmail.com>
 <1495206983.270969.982170216.5CE4EF05@webmail.messagingengine.com>
 <CACac1F82jZpeW+DHaXPCxVoKc-egFzxC3iBiGNh0gKik1qJ3hQ@mail.gmail.com>
Message-ID: <1495208231.274988.982196352.471ADC61@webmail.messagingengine.com>

On Fri, May 19, 2017, at 04:31 PM, Paul Moore wrote:
> For flit, would having the install tool set PYTHONIOENCODING help?

If install tools were meant to set PYTHONIOENCODING, I probably wouldn't
do anything else in flit's code. Python should then take care of
ensuring that any output is UTF-8 encoded, and flit doesn't currently
invoke any separate processes to do the build.

Thomas

From dholth at gmail.com  Fri May 19 11:53:04 2017
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 19 May 2017 15:53:04 +0000
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <1495208231.274988.982196352.471ADC61@webmail.messagingengine.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F_V2DcYdzZga8EWYCcLjXvFWC5aeoGaRQZcL3vcRL8TNg@mail.gmail.com>
 <1495206983.270969.982170216.5CE4EF05@webmail.messagingengine.com>
 <CACac1F82jZpeW+DHaXPCxVoKc-egFzxC3iBiGNh0gKik1qJ3hQ@mail.gmail.com>
 <1495208231.274988.982196352.471ADC61@webmail.messagingengine.com>
Message-ID: <CAG8k2+6OBoT78OF8YMh=bsTS6bFYUiXcFfbHRYXnTh+0s+jkJA@mail.gmail.com>

Congrats on getting 518 in.

On Fri, May 19, 2017, 11:37 Thomas Kluyver <thomas at kluyver.me.uk> wrote:

> On Fri, May 19, 2017, at 04:31 PM, Paul Moore wrote:
> > For flit, would having the install tool set PYTHONIOENCODING help?
>
> If install tools were meant to set PYTHONIOENCODING, I probably wouldn't
> do anything else in flit's code. Python should then take care of
> ensuring that any output is UTF-8 encoded, and flit doesn't currently
> invoke any separate processes to do the build.
>
> Thomas
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170519/091b42ce/attachment-0001.html>

From p.f.moore at gmail.com  Fri May 19 12:17:25 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 19 May 2017 17:17:25 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CAG8k2+6OBoT78OF8YMh=bsTS6bFYUiXcFfbHRYXnTh+0s+jkJA@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F_V2DcYdzZga8EWYCcLjXvFWC5aeoGaRQZcL3vcRL8TNg@mail.gmail.com>
 <1495206983.270969.982170216.5CE4EF05@webmail.messagingengine.com>
 <CACac1F82jZpeW+DHaXPCxVoKc-egFzxC3iBiGNh0gKik1qJ3hQ@mail.gmail.com>
 <1495208231.274988.982196352.471ADC61@webmail.messagingengine.com>
 <CAG8k2+6OBoT78OF8YMh=bsTS6bFYUiXcFfbHRYXnTh+0s+jkJA@mail.gmail.com>
Message-ID: <CACac1F9vTXrFPpZNwHq4STa81tz6OaCKdOou-nRLsbDA3TLikg@mail.gmail.com>

On 19 May 2017 at 16:53, Daniel Holth <dholth at gmail.com> wrote:
> Congrats on getting 518 in.

Agreed, by the way. That's a big step!
Paul

From thomas at kluyver.me.uk  Fri May 19 12:19:54 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Fri, 19 May 2017 17:19:54 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CACac1F9vTXrFPpZNwHq4STa81tz6OaCKdOou-nRLsbDA3TLikg@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F_V2DcYdzZga8EWYCcLjXvFWC5aeoGaRQZcL3vcRL8TNg@mail.gmail.com>
 <1495206983.270969.982170216.5CE4EF05@webmail.messagingengine.com>
 <CACac1F82jZpeW+DHaXPCxVoKc-egFzxC3iBiGNh0gKik1qJ3hQ@mail.gmail.com>
 <1495208231.274988.982196352.471ADC61@webmail.messagingengine.com>
 <CAG8k2+6OBoT78OF8YMh=bsTS6bFYUiXcFfbHRYXnTh+0s+jkJA@mail.gmail.com>
 <CACac1F9vTXrFPpZNwHq4STa81tz6OaCKdOou-nRLsbDA3TLikg@mail.gmail.com>
Message-ID: <1495210794.285488.982246064.51BDE4B2@webmail.messagingengine.com>

On Fri, May 19, 2017, at 05:17 PM, Paul Moore wrote:
> On 19 May 2017 at 16:53, Daniel Holth <dholth at gmail.com> wrote:
> > Congrats on getting 518 in.
> 
> Agreed, by the way. That's a big step!

Thanks both. It does feel like an achievement. :-)

From ncoghlan at gmail.com  Sat May 20 02:54:09 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 20 May 2017 16:54:09 +1000
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
Message-ID: <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>

On 20 May 2017 at 00:18, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> Hi,
>
> I'd like to make another push for PEP 517, which would make it possible
> to build wheels from a source tree with other build tools, without
> needing setup.py.
>
> https://www.python.org/dev/peps/pep-0517/
>
> Last time this was discussed we made a couple of minor changes to the
> PEP, but we didn't want to accept another packaging related PEP until
> PEP 518 was implemented in pip. I'm pleased to say that that
> implementation has just been merged:
> https://github.com/pypa/pip/pull/4144 .

Huzzah, and congratulations! :)

Regarding the encoding question, I agree with your recommendation with
one key amendment to account for the 16-bit console APIs on Windows:

* on platforms with 8-bit standard streams (e.g. Linux, Mac OS X),
build systems SHOULD emit UTF-8 encoded output
* on platforms with 16-bit standard streams (e.g. Windows), build
systems SHOULD emit UTF-16-LE encoded output
* on platforms that offer both, build systems SHOULD use the 16-bit
streams to match the default behaviour of CPython 3.6+
* install tools MUST NOT fail the build solely due to improperly
encoded output, but are otherwise free to handle the situation as they
see fit

Folks on Python 3.5 and earlier on Windows may still have problems
given that guidance (since that uses the 8-bit stream interfaces with
the Windows native encodings by default), but that's also a large part
of why CPython's behaviour on Windows was changed in 3.6 :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat May 20 03:00:22 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 20 May 2017 17:00:22 +1000
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <1495206983.270969.982170216.5CE4EF05@webmail.messagingengine.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F_V2DcYdzZga8EWYCcLjXvFWC5aeoGaRQZcL3vcRL8TNg@mail.gmail.com>
 <1495206983.270969.982170216.5CE4EF05@webmail.messagingengine.com>
Message-ID: <CADiSq7cTTvOKuUiumJiRrghCD1++Y+80qY_7No77=hE+ymXwTg@mail.gmail.com>

On 20 May 2017 at 01:16, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> On Fri, May 19, 2017, at 03:41 PM, Paul Moore wrote:
>> Can we specify what encoding the informational text must be written
>> in?
>
> Sure, that makes sense. What about:
>
> All hooks are run with working directory set to the root of the source
> tree, and MAY print arbitrary informational text on stdout and stderr.
> This text SHOULD be UTF-8 encoded, but as building may invoke other
> processes, install tools MUST NOT fail if the data they receive is not
> valid UTF-8; though in this case the display of the output may be
> corrupted.
>
> Do we also want to recommend that install tools set
> PYTHONIOENCODING=utf-8 when invoking build tools? Or leave this up to
> the build tools?

Setting PYTHONIOENCODING=utf-8:strict would potentially fail the
"don't fail hard on misencoded output" requirement, and setting
anything else is dubious from a potential data loss or compatibility
point of view (as there's no "surrogateescape" error handler in Python
2).

For use cases like distro package building, we'd also like to inherit
the surrounding build environment, so explictly requiring installation
tools to alter it at the Python level doesn't strike me as ideal.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From thomas at kluyver.me.uk  Sat May 20 04:03:52 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Sat, 20 May 2017 09:03:52 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>
Message-ID: <1495267432.2610777.982827576.60026133@webmail.messagingengine.com>

On Sat, May 20, 2017, at 07:54 AM, Nick Coghlan wrote:
> * on platforms with 8-bit standard streams (e.g. Linux, Mac OS X),
> build systems SHOULD emit UTF-8 encoded output
> * on platforms with 16-bit standard streams (e.g. Windows), build
> systems SHOULD emit UTF-16-LE encoded output

I'm quite prepared to accept that I'm mistaken, but my understanding is
that *standard streams* are 8-bit on Windows as well. The 16-bit thing
that Python 3.6 does, as I understand it, is to bypass standard streams
when it detects that they're connected to a console, and use a Windows
API call to write text to the console directly as UTF-16.

If so, when stdout/stderr are pipes, which I assume is how pip captures
the output from build processes, there's no particular reason to send
UTF-16 data just because it's Windows.

Thomas


From p.f.moore at gmail.com  Sat May 20 05:11:19 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 20 May 2017 10:11:19 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <1495267432.2610777.982827576.60026133@webmail.messagingengine.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>
 <1495267432.2610777.982827576.60026133@webmail.messagingengine.com>
Message-ID: <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>

On 20 May 2017 at 09:03, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> On Sat, May 20, 2017, at 07:54 AM, Nick Coghlan wrote:
>> * on platforms with 8-bit standard streams (e.g. Linux, Mac OS X),
>> build systems SHOULD emit UTF-8 encoded output
>> * on platforms with 16-bit standard streams (e.g. Windows), build
>> systems SHOULD emit UTF-16-LE encoded output
>
> I'm quite prepared to accept that I'm mistaken, but my understanding is
> that *standard streams* are 8-bit on Windows as well. The 16-bit thing
> that Python 3.6 does, as I understand it, is to bypass standard streams
> when it detects that they're connected to a console, and use a Windows
> API call to write text to the console directly as UTF-16.
>
> If so, when stdout/stderr are pipes, which I assume is how pip captures
> the output from build processes, there's no particular reason to send
> UTF-16 data just because it's Windows.

That's my understanding too. The standard streams are still byte
streams with an encoding. It's just that the underlying IO when the
final destination is the console, is done by the Windows Unicode APIs.
Because of this, when the output is the console the stream can accept
any unicode character and so an encoding of UTF8 is specified (and
yes, AIUI there is a translation Unicode string -> UTF-8 bytes ->
Unicode console API). For non-console output, though, the standard
streams are still byte streams and the platform behaviour is
respected, so we use the ANSI codepage (calling this the platform
standard glosses over the fact that there are two standard codepages,
ANSI and OEM, and tools don't always make the same choice when faced
with piped output). Long story short, UTF-16 is irrelevant here.

The docs for 3.6 say "Under Windows, if the stream is interactive
(that is, if its isatty() method returns True), the console codepage
is used, otherwise the ANSI code page". This is out of date (it was
true for 3.5 and earlier). In 3.6+ utf-8 is used for interactive
streams rather than the console codepage:

>py -c "import sys; print(sys.stdout.encoding, file=sys.stderr)"
utf-8
>py -c "import sys; print(sys.stdout.encoding, file=sys.stderr)" >$null
cp1252

The bigger question, though, is to what extent we want to mandate that
build tools that run external tools such as compilers take
responsibility for the encoding of the output of those tools (rather
than simply passing the output through to the output stream
unmodified). And if we do want to, whether we want to allow an
exception for setuptools/distutils.

Also, a question regarding Unix - do we really want to mandate UTF-8
even if the system locale is set to something else? Won't that mean
that build tools have the same problem with compilers generating
output in the encoding the tool wants that we already have on Windows?

My feeling is:

1. Build systems SHOULD emit output encoded in the preferred locale
encoding (normally UTF-8 on Unix, ANSI on Windows).
2. Build systems should ideally check the encoding used by external
tools that they run and transcode to the correct encoding if necessary
- but this is a quality of implementation matter.
3. Install tools MUST NOT fail if build tools produce output with the
wrong encoding, but MUST correctly reproduce build tool output if the
build tools do produce the right encoding.

My biggest concern with this is that I believe that Visual C produces
output in the OEM codepage even when output to a pipe. Actually I just
did some experiments (VS 2015), and it's even worse than that. The
compiler (cl) seems to use the OEM code page when writing to a pipe,
but the linker uses the ANSI code page. This means that a command like
"cl a?bc" produces output on (a piped) stdout that contains mixed
encodings. Given this situation, I think we have to simply give up and
take the view that the Visual C tools are simply broken in this
regard, and we shouldn't worry about them. So I'm inclined therefore
to drop point (2) from the 3 above.

Paul

From ncoghlan at gmail.com  Sat May 20 11:20:19 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 21 May 2017 01:20:19 +1000
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>
 <1495267432.2610777.982827576.60026133@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
Message-ID: <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>

Good point regarding the fact that the Windows 16-bit APIs only come
into play for interactive sessions (even in 3.6+), while for PEP 517
we're specifically interested in the 8-bit pipes used to communicate
with build subprocesses launched by an installation tool.

On 20 May 2017 at 19:11, Paul Moore <p.f.moore at gmail.com> wrote:
> The bigger question, though, is to what extent we want to mandate that
> build tools that run external tools such as compilers take
> responsibility for the encoding of the output of those tools (rather
> than simply passing the output through to the output stream
> unmodified). And if we do want to, whether we want to allow an
> exception for setuptools/distutils.
>
> Also, a question regarding Unix - do we really want to mandate UTF-8
> even if the system locale is set to something else? Won't that mean
> that build tools have the same problem with compilers generating
> output in the encoding the tool wants that we already have on Windows?

Yeah, I think that problem was starting to occur to me, hence the
reference to handling RPM and DEB build environments.

At least for non-Windows systems, I see two possible recommendations:

1. We advise installation tools to use binary streams to communicate
with build tools, and treat the results as opaque binary data. If it
needs to be written out to the installation tool's own streams, then
use the binary level APIs for those interfaces to inject the build
tool output directly, rather than decoding and re-encoding it first.

2. We advise installation tools to adopt a PEP 538 style solution,
where they mostly just trust the result of
locale.getpreferredencoding() *unless*
"codecs.lookup(locale.getpreferredencoding()).name == 'ascii'". In the
latter case, we'd advise them to set LC_CTYPE (and potentially LANG)
appropriately for the running OS. Regardless of whether or not that
locale coercion was needed, we'd recommend setting "replace" or
"backslashreplace" when decoding the stream output from the
subprocess.

At the specification level, I think option 1 probably makes the most
sense - we'd be advising insallation tools that they're free to kick
any mojibake problems further down the automation pipeline if they
don't want to worry about it. It's also the only one of the two
recommendations we can readily make cross platform.

At a quality-of-implementation level, there's a lot of potential value
in option 2 (at least on non-Windows systems) - we just wouldn't
require or recommend it at the level of the interoperability
specifications.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From brett at python.org  Sat May 20 11:29:46 2017
From: brett at python.org (Brett Cannon)
Date: Sat, 20 May 2017 15:29:46 +0000
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <1495210794.285488.982246064.51BDE4B2@webmail.messagingengine.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F_V2DcYdzZga8EWYCcLjXvFWC5aeoGaRQZcL3vcRL8TNg@mail.gmail.com>
 <1495206983.270969.982170216.5CE4EF05@webmail.messagingengine.com>
 <CACac1F82jZpeW+DHaXPCxVoKc-egFzxC3iBiGNh0gKik1qJ3hQ@mail.gmail.com>
 <1495208231.274988.982196352.471ADC61@webmail.messagingengine.com>
 <CAG8k2+6OBoT78OF8YMh=bsTS6bFYUiXcFfbHRYXnTh+0s+jkJA@mail.gmail.com>
 <CACac1F9vTXrFPpZNwHq4STa81tz6OaCKdOou-nRLsbDA3TLikg@mail.gmail.com>
 <1495210794.285488.982246064.51BDE4B2@webmail.messagingengine.com>
Message-ID: <CAP1=2W5Qk11tP8LPQnCRD0tdQ7=pYdY45i+-Npo9OvfPqvbLsA@mail.gmail.com>

On Fri, May 19, 2017, 09:20 Thomas Kluyver, <thomas at kluyver.me.uk> wrote:

> On Fri, May 19, 2017, at 05:17 PM, Paul Moore wrote:
> > On 19 May 2017 at 16:53, Daniel Holth <dholth at gmail.com> wrote:
> > > Congrats on getting 518 in.
> >
> > Agreed, by the way. That's a big step!
>
> Thanks both. It does feel like an achievement. :-)
>

As it should! Thanks for bringing the PEP to life!

-brett


_______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170520/71c7996c/attachment.html>

From steve.dower at python.org  Sat May 20 12:36:04 2017
From: steve.dower at python.org (Steve Dower)
Date: Sat, 20 May 2017 09:36:04 -0700
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>
 <1495267432.2610777.982827576.60026133@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
Message-ID: <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>

On 20May2017 0820, Nick Coghlan wrote:
> Good point regarding the fact that the Windows 16-bit APIs only come
> into play for interactive sessions (even in 3.6+), while for PEP 517
> we're specifically interested in the 8-bit pipes used to communicate
> with build subprocesses launched by an installation tool.

I need to catch up on the PEP (and thanks Brett for alerting me to the 
thread), but this comment in particular cements the mental diagram I 
have right now:

(build UI) <--> (build tool) <--> (compiler)
( Python ) <--> (  Python  ) <--> (anything)

I'll probably read the PEP closely and see that this is entirely 
incorrect, but if it's right:

* encoding for text between the build UI and build tool should just be 
specified once for all platforms (i.e. use UTF-8).
* encoding for text between build tool and the compiler depends on the 
compiler

In general, since most subprocesses (at least on Windows) do not have 
customizable encodings, the tool that launches them needs to know what 
the encoding is. Since we don't live in a Python 3.6 world quite yet, 
that means the tool should read raw bytes from the compiler and encode 
them to UTF-8.

The encoding between the tool and UI is essentially irrelevant - the UI 
is going to transform the data anyway for display, and the tool is going 
to have to transform it from the compilation tools, so the best we can 
do is pick the most likely encoding to avoid too many operations. UTF-8 
is probably that.

That's my 0.02AUD based on a vague memory of the PEP and this thread. As 
I get time today (at PyCon) to read up on it I may post amendments, but 
in general I'm +100 on "just pick an encoding and make the 
implementations transcode".

Cheers,
Steve


From luis at huntingbears.com.ve  Fri May 19 22:01:01 2017
From: luis at huntingbears.com.ve (=?UTF-8?Q?Luis_Alejandro_Mart=C3=ADnez_Faneyth?=)
Date: Fri, 19 May 2017 22:01:01 -0400
Subject: [Distutils] Introducing PyPIContents
Message-ID: <CAHagVwYw_eoQL7DtdVX1yQLZuNzzkMwn1Ji3dpi_hKMvP3j=ew@mail.gmail.com>

Hi everyone,

I'm new to this list but I've been reading some threads in the archive.

Around february, an idea about indexing modules from PyPI packages was
brought up. I've been working on something similar for quite a while.

PyPIContents is an index of PyPI packages that lists its modules and
command line scripts in JSON format, like this:


[
...

    "1337": {
        "cmdline": [],
        "modules": [
            "1337",
            "1337.1337"
        ],
        "version": "1.0.0"
    },

...

]


You can check it out here:

https://github.com/LuisAlejandro/pypicontents

And some use cases:

https://github.com/LuisAlejandro/pypicontents#use-cases

The actual index lives here, its around 60MB:

https://raw.githubusercontent.com/LuisAlejandro/pypicontents/contents/pypi.json

Is updated daily with the help of Travis:

https://github.com/LuisAlejandro/pypicontents/blob/contents/.travis.yml

Anyway, I hope is useful and I'll be around for any comments or questions.

Cheers!


Luis Alejandro Mart?nez Faneyth
Blog: http://huntingbears.com.ve
Github: http://github.com/LuisAlejandro
Twitter: http://twitter.com/LuisAlejandro

CODE IS POETRY
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170519/e7ecee1c/attachment.html>

From radon.neon at gmail.com  Sat May 20 10:45:24 2017
From: radon.neon at gmail.com (Radon Rosborough)
Date: Sat, 20 May 2017 07:45:24 -0700
Subject: [Distutils] Finally making distlib handle spaces
Message-ID: <CADB4rJGZ-XtdnZxsvqqe0dFFLWrh8PPy0M2qtsbM37kBfpG6vg@mail.gmail.com>

Hi all,

Since at least 2011, virtualenv has not supported spaces in paths.
This has bitten many people, including myself, and caused numerous
issues over the years [1] [2] [3] [4] [5] [6] [7].

However, as was discussed in [8], the issue lies not with virtualenv
but with distlib, via pip. It would be possible for pip to use the
existing distlib interface to hack around the problem, but I believe
the current behavior of distlib is erroneous when it comes to spaces
in paths. I therefore believe it would be more appropriate to fix the
problem in distlib.

Two separate patches [9] [10] that solve the problem in distlib were
posted in January by Harald Nordgren. However, they were declined
pending a discussion on distutils-sig [11]. As far as I can tell, no
such discussion was ever started. However, the issue remains, and we
have a clear solution proposal to consider, so I'd like to kick it off
now. In the remainder of this email, I'll explain the problem and
surrounding context in detail, and why I think the solution proposed
by Harald (or some variation) is a good path forward for distlib. I
look forward to hearing your thoughts on the matter.

==========================
The behavior of virtualenv
==========================

The following is written for:

    $ python --version
    Python 2.7.13
    $ pip --version
    pip 9.0.1 from /usr/local/lib/python2.7/site-packages (python 2.7)
    $ virtualenv --version
    15.1.0

Creating a virtualenv is done as follows:

    $ virtualenv venv
    New python executable in /private/tmp/path with spaces/venv/bin/python2.7
    Also creating executable in /private/tmp/path with spaces/venv/bin/python
    Installing setuptools, pip, wheel...done.

This creates a directory structure looking as follows under the venv
directory:

    ??? bin
    ?  ??? activate
    ?  ??? activate.csh
    ?  ??? activate.fish
    ?  ??? activate_this.py
    ?  ??? easy_install
    ?  ??? easy_install-2.7
    ?  ??? pip
    ?  ??? pip2
    ?  ??? pip2.7
    ?  ??? python -> ./bin/python2.7
    ?  ??? python-config
    ?  ??? python2 -> ./bin/python2.7
    ?  ??? python2.7
    ?  ??? wheel
    ??? include
    ?  ??? ...
    ??? lib
    ?  ??? ...
    ??? ...

The idea is that one can call the pip and python executables
inside the virtualenv, instead of the system ones. Like so:

    $ venv/bin/python --version
    Python 2.7.13
    $ venv/bin/pip --version
    zsh: venv/bin/pip: bad interpreter: "/private/tmp/path: no such
file or directory

Unfortunately, as you can see, pip doesn't work at all! Why does this
happen? While the python executable is a native binary, pip is
actually just a Python script, which is specified to be run by the
accompanying virtualenv python executable. Here are the contents:

    #!"/private/tmp/path with spaces/venv/bin/python2.7"

    # -*- coding: utf-8 -*-
    import re
    import sys

    from pip import main

    if __name__ == '__main__':
        sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
        sys.exit(main())

The issue is that the python binary is specified using a shebang,
which is known [12] [13] to be fragile, OS-dependent, and error-prone
regarding paths that are long or that contain spaces or non-ASCII
characters. In particular, the quoting of the shebang does not work on
most operating systems, including macOS, which is what I ran this test
on.

======================
virtualenv and distlib
======================

The issue is complicated by the fact that there are several different
libraries at play. To perform the installation of pip and wheel into
the virtualenv, virtualenv calls into pip [14]. The 'pip install'
command then uses the subroutine library 'wheel.py', which generates
the stub scripts using distlib's ScriptMaker [15]. It is actually
distlib which generates the shebang, although this can be overridden
by setting the 'executable' property of the ScriptMaker object [16]
[17]. Any patch to fix the virtualenv problem would therefore need to
be in either pip (as a consumer of the shebang-generation interface)
or distlib (as the provider of that interface). The problem cannot be
addressed by virtualenv without doing something like using pip/distlib
to generate the scripts and then fixing them after the fact (this has
been proposed [26] [29], but I consider it a hack).

==================
Proposed solutions
==================

There has been extended discussion about this issue, especially in
[2]. Essentially, the solutions proposed fall into four categories:

(1) Don't change anything; end users can work around the issue. For
    example, they can place their virtualenvs in a different directory
    than their project, or change their username to avoid having
    spaces or non-ASCII characters.

(2) Don't fix the bug, but add a warning to virtualenv. If we
    absolutely can't fix the bug (which I strongly believe not to be
    the case), then this would be the next best thing to do. See [27]
    [30] [31].

(3) Attempt to escape the shebang, for example by using backslashes or
    quotes. The latter has already been implemented. Unfortunately,
    this does not work on most operating systems, since shebangs are
    interpreted by the kernel in a rather primitive way.

(4) Patch either pip or distlib to use a different strategy for
    dispatching the python binary to be used. For example, make the
    pip script into a shell script with '#!/usr/bin/env sh' that then
    invokes the real pip script using the appropriate python binary [9];
    this works because even POSIX sh supports escaping arbitrary
    characters in executable paths. Alternatively, use a clever hack
    to make the pip script executable as both a Python script and as a
    shell script [10]. This idea originated from [19].

I argue that approaches (1), (2), and (3) are inadequate, and that (4)
or some variation thereof is the best path forward for distlib.
Regarding (4), note that in addition to the second pull request [10],
there is another, possibly cleaner implementation of the same idea at
[18], by the same author. Since this issue has been discussed since
2011 without being fixed, I obviously need to justify my position.

===================================
Justifications and counterarguments
===================================

> There's an easy workaround for the end user, so there's no need to
> change things on the pip/distlib side.

The workaround may not necessarily be "easy". For example, there have
been reports of this problem being unavoidable on macOS in certain
circumstances, since the drive name ("Macintosh HD") has a space in it
[1]. People working in corporate environments may not be able to
simply move their virtualenvs to a different directory (possibly one
outside their home directory, if their username has a space or
non-ASCII character in it). These issues really do happen, and really
do inconvenience people [20].

However, setting these cases aside, the workaround is indeed not too
hard for most people. The question is: do we really want virtualenv to
have this behavior? To me, it honestly seems embarrassing that such an
important tool, and a cornerstone of the Python ecosystem, can't
handle something as simple as spaces in directory names, even more
than six years after the bug was first reported. If there were a bug
in the development version of Git that prevented it from working at
all when you had your repository in a path with spaces, nobody would
even consider shipping a release! In 2017, people use Unicode and
spaces in their filesystems, and they expect their tools to handle
this.

In my opinion, end-user workarounds should be reserved for those cases
where it is absolutely impossible for the software to solve the
problem itself without introducing even worse problems. This is simply
not the case with the distlib/pip path bug. Patches that present
elegant ways to solve the issue without breaking backward
compatibility are available [9] [10]. There is no risk of breaking
programs that rely on the previous behavior, since the previous
behavior was a complete inability for virtualenv to function *at all*,
even for 'pip --version'!

> People who use long paths and/or paths with spaces and/or paths with
> non-ASCII characters in them are rare.

No, that's not true [1] [2] [3] [4] [5] [6] [7] [23] [24] [28].
Considering the tiny fraction of people running into bugs who actually
report them, I'd say quite a few people run into this issue. It's
pretty rare that I'm able to scrounge up this number of issues
all leading to a single bug. (I've also run into it, on multiple
occasions over several years, by the way.)

> The issue with spaces is a limitation of the way in which the kernel
> parses shebangs, and this can't be fixed.

This is technically true [13], but irrelevant [21]. The implications
of this fact are that virtualenv cannot solve this issue *if it
continues to directly use shebangs for interpreter dispatch*. However,
there are many other ways to dispatch interpreters [9] [10], none of
which have the same limitations of the shebang. Thus the limitations
of shebangs have no bearing on this issue, unless it turns out that
it's completely impossible to dispatch scripts in any other way while
maintaining compatibility with all supported platforms, which seems
extremely unlikely to me.

> We don't want to make the code more complicated unless there's a
> good reason for it.

I agree. In my opinion, the complete failure of virtualenv to operate
in a common real-world use case is a good reason. However, beyond
this, we can be even more conservative. In Harald's alternate patch
[18], the polyglot script hack is only used in situations where the
path is too long for a shebang (conditional on the OS) or it contains
spaces. Thus, it's only possible for the patch to improve things,
since it is only activated in cases where virtualenv was completely
nonfunctional before.

Look at software like Bash. It has implicit wordsplitting, which makes
dealing with spaces in paths difficult. However, it is *extremely*
well-understood that it is a best practice to disable wordsplitting by
quoting arguments [22]. That is, modern software is expected to handle
spaces and other special characters in paths. Failing to do so leads
to hard-to-trace bugs and potential security vulnerabilities.

> We don't want to change the default behavior of distlib. pip should
> handle its shebang needs using the ScriptMaker.executable property,
> which was provided for this purpose.

Now we are getting somewhere. This is one of two solutions proposed
which would actually fix the bug. However, I feel that it is the wrong
way to go about it.

In my opinion, distlib (and specifically ScriptMaker) are providing an
interface. This interface specifies that when you use distlib to
generate a script, and then run the script, it will invoke the
executable that you told it to. This is not currently the case when
there are spaces in the path leading to the executable. In my opinion,
this is a *bug* in that distlib fails to fulfill its specification.
Yes, other applications that use distlib could work around this bug,
but it is fundamentally a bug in distlib, and should be fixed there.
The ability to support spaces in paths should not be a special feature
that must be hacked in (separately, resulting in code duplication and
increased maintenance burden) to every application that wants to use
distlib to generate executables.

If we really want to go that way, though, Harald has made pull
requests for both pip [25] and virtualenv [26].

==========
Next steps
==========

I'd love to hear people's thoughts on the issues raised here. If we
decide to go forward with patching distlib, I think the logical next
step would be to look at Harald's patches and see if they need to
modified to work correctly on all the systems that Python/pip support.
As I mentioned earlier, the last comment on [9] was a request for a
discussion to be started on distutils-sig, as this email intends. What
do people think about [9], [10], [18]?

Best regards,
Radon Rosborough

[1]: http://stackoverflow.com/q/15472430/3538165
[2]: https://github.com/pypa/virtualenv/issues/53
[3]: https://github.com/pypa/virtualenv/issues/994
[4]: https://github.com/pypa/pip/issues/923
[5]: https://bugs.python.org/issue20622
[6]: https://groups.google.com/forum/#!topic/virtualenvwrapper/vqW3fedTgKc
[7]: https://bugs.launchpad.net/virtualenv/+bug/241581
[8]: https://github.com/pypa/virtualenv/issues/53#issuecomment-302005361
[9]: https://bitbucket.org/pypa/distlib/pull-requests/31
[10]: https://bitbucket.org/pypa/distlib/pull-requests/32
[11]: https://bitbucket.org/pypa/distlib/pull-requests/31/#comment-29795586
[12]: https://bitbucket.org/pypa/distlib/pull-requests/31/#comment-29734103
[13]: https://lists.gnu.org/archive/html/bug-bash/2008-05/msg00052.html
[14]: https://github.com/pypa/virtualenv/blob/c1ef9e29bfda9f5b128476d0c6d865ffe681b3fb/virtualenv.py#L856-L882
[15]: https://github.com/pypa/pip/blob/7c4353e6834a38b36a34234db865e13a589c9e5f/pip/wheel.py#L383-L427
[16]: https://bitbucket.org/pypa/distlib/src/216bd857431686493d37eb081614d91c1356fa79/distlib/scripts.py#-141:143
[17]: http://pythonhosted.org/distlib/tutorial.html#specifying-a-custom-executable-for-shebangs
[18]: https://bitbucket.org/pypa/distlib/pull-requests/33
[19]: https://hg.mozilla.org/mozilla-central/file/tip/mach
[20]: https://bitbucket.org/pypa/distlib/pull-requests/31/#comment-29735496
[21]: https://github.com/pypa/virtualenv/issues/53#issuecomment-302019457
[22]: http://mywiki.wooledge.org/BashGuide/Practices#Quoting
[23]: https://codedump.io/share/7xYRbcZCNihE/1/using-virtualenv-with-spaces-in-a-path
[24]: https://github.com/pypa/virtualenv/issues/997
[25]: https://github.com/pypa/pip/pull/4237
[26]: https://github.com/pypa/virtualenv/pull/1004
[27]: https://github.com/pypa/virtualenv/issues/1039
[28]: https://github.com/pypa/virtualenv/issues/1014
[29]: https://github.com/pypa/virtualenv/pull/910
[30]: http://bugs.python.org/issue28446
[31]: https://github.com/pypa/virtualenv/issues/53#issuecomment-282466994

From thomas at kluyver.me.uk  Sat May 20 13:11:33 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Sat, 20 May 2017 18:11:33 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>
 <1495267432.2610777.982827576.60026133@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
Message-ID: <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>

On Sat, May 20, 2017, at 05:36 PM, Steve Dower wrote:
> I'll probably read the PEP closely and see that this is entirely 
> incorrect, but if it's right:
> 
> * encoding for text between the build UI and build tool should just be 
> specified once for all platforms (i.e. use UTF-8).

+1

> * encoding for text between build tool and the compiler depends on the 
> compiler
> 
> In general, since most subprocesses (at least on Windows) do not have 
> customizable encodings, the tool that launches them needs to know what 
> the encoding is. Since we don't live in a Python 3.6 world quite yet, 
> that means the tool should read raw bytes from the compiler and encode 
> them to UTF-8.

I half agree, but:
- Build tools may not 100% know what encoding output will be produced,
especially if the developer can supply a custom command for the build
tool to run.
- It's possible for data on a pipe to be binary data with no meaning as
text.
- As a lazy developer, I don't want to read stdout/stderr from a
subprocess only to spit it back to my own stdout/stderr. I'd much rather
just launch the subprocess and let it use the same stdout/stderr as my
build tool.

So I think it's most practical to recommend that build tools produce
UTF-8 (if not sys.stdout.isatty()), but let build tool developers decide
how far they go to comply with that.

Thomas

From thomas at kluyver.me.uk  Sat May 20 13:23:09 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Sat, 20 May 2017 18:23:09 +0100
Subject: [Distutils] Introducing PyPIContents
In-Reply-To: <CAHagVwYw_eoQL7DtdVX1yQLZuNzzkMwn1Ji3dpi_hKMvP3j=ew@mail.gmail.com>
References: <CAHagVwYw_eoQL7DtdVX1yQLZuNzzkMwn1Ji3dpi_hKMvP3j=ew@mail.gmail.com>
Message-ID: <1495300989.1548172.983098880.1DFE0265@webmail.messagingengine.com>

Hi Luis,

Awesome, thanks for this :-). It was me posting before about
indexing PyPI.
I'm intrigued: how do you keep it up to date using Travis? When I
looked into this, I was pretty sure you need to download every package
to index it. Do you have some way to only download the new releases? Or
is Travis able to download every package every day? Or have you found
another way round it?
Does the index only include the latest version of each package, or does
it also include older versions? The wifi on the train I'm on at the
moment isn't fast enough to download 60 MB to find out. ;-)
Does your indexing tool prefer to use wheels or sdists? Is it capable of
using either for packages which don't have both available? Do you do
anything to cope with modules which may be included for one platform but
not another?
I'm excited to see someone actually doing this!

Thomas


On Sat, May 20, 2017, at 03:01 AM, Luis Alejandro Mart?nez Faneyth wrote:> Hi everyone,
> 
> I'm new to this list but I've been reading some threads in the
> archive.> 
> Around february, an idea about indexing modules from PyPI packages was
> brought up. I've been working on something similar for quite a while.> 
> PyPIContents is an index of PyPI packages that lists its modules and
> command line scripts in JSON format, like this:> 
> 
> [
> ...    "1337": { "cmdline": [], "modules": [ "1337", "1337.1337" ],
> "version": "1.0.0" }, ... ]
>
>> You can check it out here:
> 
> https://github.com/LuisAlejandro/pypicontents
> 
> And some use cases:
> 
> https://github.com/LuisAlejandro/pypicontents#use-cases
> 
> The actual index lives here, its around 60MB:
> 
> https://raw.githubusercontent.com/LuisAlejandro/pypicontents/contents/pypi.json> 
> Is updated daily with the help of Travis:
> 
> https://github.com/LuisAlejandro/pypicontents/blob/contents/.travis.yml> 
> Anyway, I hope is useful and I'll be around for any comments or
> questions.> 
> Cheers!
> 
> 
> 
> Luis Alejandro Mart?nez Faneyth
> Blog: http://huntingbears.com.ve
> Github: http://github.com/LuisAlejandro
> Twitter: http://twitter.com/LuisAlejandro
> 
> CODE IS POETRY
> 
> _________________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170520/fd173a12/attachment.html>

From luis at huntingbears.com.ve  Sat May 20 14:29:33 2017
From: luis at huntingbears.com.ve (=?UTF-8?Q?Luis_Alejandro_Mart=C3=ADnez_Faneyth?=)
Date: Sat, 20 May 2017 14:29:33 -0400
Subject: [Distutils] Introducing PyPIContents
In-Reply-To: <1495300989.1548172.983098880.1DFE0265@webmail.messagingengine.com>
References: <CAHagVwYw_eoQL7DtdVX1yQLZuNzzkMwn1Ji3dpi_hKMvP3j=ew@mail.gmail.com>
 <1495300989.1548172.983098880.1DFE0265@webmail.messagingengine.com>
Message-ID: <CAHagVwYGtAqLA9QWQZBUmjc124+fZAbXp=T986y8LP-HCfeFaA@mail.gmail.com>

Hi Thomas,

2017-05-20 13:23 GMT-04:00 Thomas Kluyver <thomas at kluyver.me.uk>:

> Hi Luis,
>
> Awesome, thanks for this :-). It was me posting before about indexing PyPI.
>
> I'm intrigued: how do you keep it up to date using Travis? When I looked
> into this, I was pretty sure you need to download every package to index
> it. Do you have some way to only download the new releases? Or is Travis
> able to download every package every day? Or have you found another way
> round it?
>

?I divided the index processing alphabetically, so that each letter ?is
processed in a separate travis job. I also placed memory and time limits to
avoid abusing Travis. The first run it has to download each package until
it reaches the maximum time limit for each job, which is 40min. The next
time, the script will only process packages that have been updated since
the last run.



> Does the index only include the latest version of each package, or does it
> also include older versions? The wifi on the train I'm on at the moment
> isn't fast enough to download 60 MB to find out. ;-)
>

?It only includes the current versions.?



>
> Does your indexing tool prefer to use wheels or sdists? Is it capable of
> using either for packages which don't have both available? Do you do
> anything to cope with modules which may be included for one platform but
> not another?
>

?It supports ['.whl', '.egg', '.zip', '.tgz', '.tar.gz', '.tar.bz2'?]
formats, and it extracts the data using any available.

I wasn't aware of the fact that some modules may be on one platform and not
in another. I guess there's room for improvement.



>
> I'm excited to see someone actually doing this!
>

?Thank you. I made this because I wanted to have an app that guessed python
dependencies from code by scaning module imports and ?then looking up the
Index. That app is called Pip Sala Bim and you can check it out here:

https://github.com/LuisAlejandro/pipsalabim



>
>
> Thomas
>
>
> On Sat, May 20, 2017, at 03:01 AM, Luis Alejandro Mart?nez Faneyth wrote:
>
> Hi everyone,
>
> I'm new to this list but I've been reading some threads in the archive.
>
> Around february, an idea about indexing modules from PyPI packages was
> brought up. I've been working on something similar for quite a while.
>
> PyPIContents is an index of PyPI packages that lists its modules and
> command line scripts in JSON format, like this:
>
>
> [
> ...
>
>     "1337": {
>         "cmdline": [],
>         "modules": [
>             "1337",
>             "1337.1337"
>         ],
>         "version": "1.0.0"
>     },
>
> ...
>
> ]
>
>
> You can check it out here:
>
> https://github.com/LuisAlejandro/pypicontents
>
> And some use cases:
>
> https://github.com/LuisAlejandro/pypicontents#use-cases
>
> The actual index lives here, its around 60MB:
>
> https://raw.githubusercontent.com/LuisAlejandro/pypicontents
> /contents/pypi.json
>
> Is updated daily with the help of Travis:
>
> https://github.com/LuisAlejandro/pypicontents/blob/contents/.travis.yml
>
> Anyway, I hope is useful and I'll be around for any comments or questions.
>
> Cheers!
>
>
>
> Luis Alejandro Mart?nez Faneyth
> Blog: http://huntingbears.com.ve
> Github: http://github.com/LuisAlejandro
> Twitter: http://twitter.com/LuisAlejandro
>
> CODE IS POETRY
>
> *_______________________________________________*
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>


-- 
Luis Alejandro Mart?nez Faneyth
Blog: http://huntingbears.com.ve
Github: http://github.com/LuisAlejandro
Twitter: http://twitter.com/LuisAlejandro

CODE IS POETRY
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170520/09d2672b/attachment-0001.html>

From steve.dower at python.org  Sat May 20 14:36:38 2017
From: steve.dower at python.org (Steve Dower)
Date: Sat, 20 May 2017 11:36:38 -0700
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>
 <1495267432.2610777.982827576.60026133@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
Message-ID: <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>

On 20May2017 1011, Thomas Kluyver wrote:
> On Sat, May 20, 2017, at 05:36 PM, Steve Dower wrote:
>> In general, since most subprocesses (at least on Windows) do not have
>> customizable encodings, the tool that launches them needs to know what
>> the encoding is. Since we don't live in a Python 3.6 world quite yet,
>> that means the tool should read raw bytes from the compiler and encode
>> them to UTF-8.
>
> I half agree, but:
> - Build tools may not 100% know what encoding output will be produced,
> especially if the developer can supply a custom command for the build
> tool to run.

In this case, the whole thing breaks down anyway. UI can't be expected 
to reliably display text from an unknown encoding - at some point it has 
to be forced into a known quantity, and IMHO the code closest to the 
tool should do it.

> - It's possible for data on a pipe to be binary data with no meaning as
> text.

Sure, but it cannot be rendered unless you choose an encoding. All you 
can do is dump it to a file (and let a file editor choose an encoding).

> - As a lazy developer, I don't want to read stdout/stderr from a
> subprocess only to spit it back to my own stdout/stderr. I'd much rather
> just launch the subprocess and let it use the same stdout/stderr as my
> build tool.

One of the open issues against distutils is that it does this. We can 
allow it, but a well-defined tool should capture the output and pass it 
to the UI component rather than bypassing the UI component.

> So I think it's most practical to recommend that build tools produce
> UTF-8 (if not sys.stdout.isatty()), but let build tool developers decide
> how far they go to comply with that.

Require that build tools either send UTF-8 to the UI component, or write 
bytes to a file and call it a build output. I see no benefit in 
requiring both the build tool and the UI tool to guess what the text 
encoding is.

Cheers,
Steve

From p.f.moore at gmail.com  Sat May 20 15:59:26 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 20 May 2017 20:59:26 +0100
Subject: [Distutils] Finally making distlib handle spaces
In-Reply-To: <CADB4rJGZ-XtdnZxsvqqe0dFFLWrh8PPy0M2qtsbM37kBfpG6vg@mail.gmail.com>
References: <CADB4rJGZ-XtdnZxsvqqe0dFFLWrh8PPy0M2qtsbM37kBfpG6vg@mail.gmail.com>
Message-ID: <CACac1F_cYCUbL6Ma0Go_E7ctJPj76ufB2uZmw9Ec82kTBtd7TQ@mail.gmail.com>

On 20 May 2017 at 15:45, Radon Rosborough <radon.neon at gmail.com> wrote:
> ===================================
> Justifications and counterarguments
> ===================================
[...]

While you make a good case countering the issues you quote, I think
there's an important issue you haven't addressed, and I'd like to hear
your comments on it:

The current behaviour fails, as you note, but it does so in a
"standard" way - shebang behaviour (and its limitations) is
well-known. At the moment, your proposal is just to use "an
alternative" launch process. Without a specific proposal, it's
impossible to judge whether the solution is better than the current
situation. And indeed, any proposed solution needs to demonstrate that
it doesn't have security vulnerabilities or other issues that make it
just as much a problem as the status quo (if not more).

For example, I would have thought that "#!/usr/bin/env sh" runs the
risk of picking up a malicious sh executable injected into the user's
PATH. Also, different systems use different sh implementations - so
care would need to be taken to only code in the lowest common
denominator syntax.

I suggest that the next step needs to be to propose a specific
implementation of the wrapper. The specifics can then be debated, but
you'd need a pretty solid proposal - otherwise the discussion will
descend into multiple proposals and bikeshedding, and is likely to
stall without achieving anything.

One other point, which I'd hope is obvious. On Windows (where shebang
processing is handled by the wrapper exe, and is well defined and
robust) there should be no change to the current behaviour.

Paul

From p.f.moore at gmail.com  Sat May 20 16:05:53 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 20 May 2017 21:05:53 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>
 <1495267432.2610777.982827576.60026133@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
Message-ID: <CACac1F_mjeA7B6+7HqiiL_gqcT5BWAP_dKEWY7smVA8BoP-G6A@mail.gmail.com>

On 20 May 2017 at 19:36, Steve Dower <steve.dower at python.org> wrote:
>
>> - As a lazy developer, I don't want to read stdout/stderr from a
>> subprocess only to spit it back to my own stdout/stderr. I'd much rather
>> just launch the subprocess and let it use the same stdout/stderr as my
>> build tool.
>
>
> One of the open issues against distutils is that it does this. We can allow
> it, but a well-defined tool should capture the output and pass it to the UI
> component rather than bypassing the UI component.

I'm a little concerned if we're going to end up with a proposal that
means that distutils is in violation of the spec unless this issue is
fixed. I'm not sure if that's where we're headed, but I wanted to be
clear here - is PEP 517 intended to encompass distutils/setuptools, or
are we treating them as a legacy case, that pip should special-case?

Paul

From p.f.moore at gmail.com  Sat May 20 16:15:35 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 20 May 2017 21:15:35 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>
 <1495267432.2610777.982827576.60026133@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
Message-ID: <CACac1F-Ao1GLs0BL8369d7uxQ3ssWHh1m6Lb6UnAC-S=h5c_Zg@mail.gmail.com>

On 20 May 2017 at 17:36, Steve Dower <steve.dower at python.org> wrote:
> In general, since most subprocesses (at least on Windows) do not have
> customizable encodings, the tool that launches them needs to know what the
> encoding is. Since we don't live in a Python 3.6 world quite yet, that means
> the tool should read raw bytes from the compiler and encode them to UTF-8.

Did you spot my point that Visual C produces output that's a mixture
of OEM and ANSI codepages?

The example I used was:

OEM code page 850, ANSI codepage 1252 (standard British English Windows)

Visual Studio 2015

cl a?b >output.file

The output uses CP850 (in the cl error message) and CP1252 (in the
link error) for the ? sign.

When run from the command line without redirection, the output is in a
consistent encoding. It's only when you redirect the output (I
redirected to a file, I assume a pipe would be the same) that you get
the problem.

I'd be very surprised if build tool developers got this sort of edge
case correct without at least some guidance from the PEP on the sorts
of things they need to consider. You suggest "read raw bytes and
encode them to UTF-8" - but you don't encode bytes, you encode
strings, so you still need to convert those bytes to a string first,
and there's no encoding you can reliably use for this. You need to use
"errors=replace" to ensure you can handle inconsistently encoded data
without getting an exception.

Paul

From radon.neon at gmail.com  Sat May 20 17:29:39 2017
From: radon.neon at gmail.com (Radon Rosborough)
Date: Sat, 20 May 2017 14:29:39 -0700
Subject: [Distutils] Finally making distlib handle spaces
In-Reply-To: <CACac1F_cYCUbL6Ma0Go_E7ctJPj76ufB2uZmw9Ec82kTBtd7TQ@mail.gmail.com>
References: <CADB4rJGZ-XtdnZxsvqqe0dFFLWrh8PPy0M2qtsbM37kBfpG6vg@mail.gmail.com>
 <CACac1F_cYCUbL6Ma0Go_E7ctJPj76ufB2uZmw9Ec82kTBtd7TQ@mail.gmail.com>
Message-ID: <CADB4rJEoOxYUzgSys4mQnQtZMTCFFgcdcJ0kxL=-MKX5LNfosA@mail.gmail.com>

> The current behaviour fails, as you note, but it does so in a
> "standard" way - shebang behaviour (and its limitations) is
> well-known.

I agree with this, but in my opinion the shebang is simply an
implementation detail of virtualenv. I would like to quote @JDLH from
[1]: "There is nothing about the value provided by virtualenv that
demands it use the shebang." If the shebang were fundamentally
necessary to provide the functionality of virtualenv, it would make
sense for virtualenv to inherit the shebang's restrictions. But this
is not the case, so I think that the shebang should be considered an
implementation detail that the end user should not need to be aware
of.

> At the moment, your proposal is just to use "an alternative" launch
> process. Without a specific proposal, it's impossible to judge
> whether the solution is better than the current situation.

We already have three specific patches which provide alternative
launch processes: [2], [3], and [4]. I feel like that should be
specific enough to start a discussion. In fact, Vinay specifically
requested a discussion about [2] be raised on distutils-sig [5]. The
only reason that no action has been taken is that nobody started that
discussion (until now).

> I would have thought that "#!/usr/bin/env sh" runs the risk of
> picking up a malicious sh executable injected into the user's PATH.

That's certainly a valid concern. Does this happen in the real world?
I feel like if you have a malicious sh executable on your PATH, you're
going to have a lot more problems than just from virtualenv. But this
is a good reason that we may want to restrict the patch to only take
effect when using the shebang directly would fail.

> Also, different systems use different sh implementations - so care
> would need to be taken to only code in the lowest common denominator
> syntax.

Can we assume POSIX compatibility? Even if not, we're not doing
anything complicated, only passing some arguments to a command. Surely
that can be done in pretty much any shell one can find.

> multiple proposals and bikeshedding

Although we have three implementations, my personal preference is for
[4]. This is because:

* It avoids the need for creating new files.
* It only takes effect when necessary (i.e. when the shebang won't
  work).
* The code is fairly clean.

> On Windows (where shebang processing is handled by the wrapper exe,
> and is well defined and robust) there should be no change to the
> current behaviour.

Agreed.

[1]: https://github.com/pypa/virtualenv/issues/53#issuecomment-302019457
[2]: https://bitbucket.org/pypa/distlib/pull-requests/31
[3]: https://bitbucket.org/pypa/distlib/pull-requests/32
[4]: https://bitbucket.org/pypa/distlib/pull-requests/33
[5]: https://bitbucket.org/pypa/distlib/pull-requests/31#comment-29795586

From thomas at kluyver.me.uk  Sat May 20 17:36:33 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Sat, 20 May 2017 22:36:33 +0100
Subject: [Distutils] Introducing PyPIContents
In-Reply-To: <CAHagVwYGtAqLA9QWQZBUmjc124+fZAbXp=T986y8LP-HCfeFaA@mail.gmail.com>
References: <CAHagVwYw_eoQL7DtdVX1yQLZuNzzkMwn1Ji3dpi_hKMvP3j=ew@mail.gmail.com>
 <1495300989.1548172.983098880.1DFE0265@webmail.messagingengine.com>
 <CAHagVwYGtAqLA9QWQZBUmjc124+fZAbXp=T986y8LP-HCfeFaA@mail.gmail.com>
Message-ID: <1495316193.1607990.983225000.11AB3CCB@webmail.messagingengine.com>

On Sat, May 20, 2017, at 07:29 PM, Luis Alejandro Mart?nez Faneyth wrote:> It supports ['.whl', '.egg', '.zip', '.tgz', '.tar.gz', '.tar.bz2']
> formats, and it extracts the data using any available.
Nice! If there are multiple of those formats present, does it get the
data from just one? Or does it get data from all of them and combine
it somehow?
> 
> I wasn't aware of the fact that some modules may be on one platform
> and not in another. I guess there's room for improvement.
It probably doesn't matter for most cases, but since setup.py runs
arbitrary code, it's possible for it to install different modules in
different situations - or even select modules at random, if you really
want to confuse tools like this. ;-)
This is why my own efforts at indexing focused on wheels - you can be
sure of exactly what a wheel contains. My wheel-indexing tool 'wheeldex'
is here, if there's any code or ideas there that you can use:https://github.com/takluyver/wheeldex

> Thank you. I made this because I wanted to have an app that guessed
> python dependencies from code by scaning module imports and then
> looking up the Index. That app is called Pip Sala Bim and you can
> check it out here:> 
> https://github.com/LuisAlejandro/pipsalabim

Neat, that's precisely one of the use cases I was thinking of for an
index. The other thing I'm interested in is providing an interface to
install modules by their import name rather than their PyPI name; I
think your index should work for that as well. I'll dig into the code of
both PyPIContents and Pip Sala Bim more soon.
Thanks,
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170520/ea46954a/attachment.html>

From steve.dower at python.org  Sat May 20 19:42:44 2017
From: steve.dower at python.org (Steve Dower)
Date: Sat, 20 May 2017 16:42:44 -0700
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CACac1F-Ao1GLs0BL8369d7uxQ3ssWHh1m6Lb6UnAC-S=h5c_Zg@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>
 <1495267432.2610777.982827576.60026133@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <CACac1F-Ao1GLs0BL8369d7uxQ3ssWHh1m6Lb6UnAC-S=h5c_Zg@mail.gmail.com>
Message-ID: <7c106af7-1758-4a48-4133-f857b39e7203@python.org>

On 20May2017 1315, Paul Moore wrote:
> On 20 May 2017 at 17:36, Steve Dower <steve.dower at python.org> wrote:
>> In general, since most subprocesses (at least on Windows) do not have
>> customizable encodings, the tool that launches them needs to know what the
>> encoding is. Since we don't live in a Python 3.6 world quite yet, that means
>> the tool should read raw bytes from the compiler and encode them to UTF-8.
>
> Did you spot my point that Visual C produces output that's a mixture
> of OEM and ANSI codepages?

[SNIP]

Yes, and it's a perfect example of why the MSVC-specific wrapper should 
be the one to deal with tool encodings. If you forward unencoded bytes 
like this back to the UI, it will have to deal with the mixed encoding.

> I'd be very surprised if build tool developers got this sort of edge
> case correct without at least some guidance from the PEP on the sorts
> of things they need to consider. You suggest "read raw bytes and
> encode them to UTF-8" - but you don't encode bytes, you encode
> strings, so you still need to convert those bytes to a string first,
> and there's no encoding you can reliably use for this. You need to use
> "errors=replace" to ensure you can handle inconsistently encoded data
> without getting an exception.

The "read raw bytes and [transcode] them" comment was meant to be that 
sort of help. I didn't go as far as writing 
`output.decode(output_encoding, errors="replace").encode("utf-8", 
errors="replace")`, but that's basically what I meant to imply. The 
build tool developer is the *only* developer who can get this right, and 
if they can't, then they have to figure out the most appropriate way to 
work around the fact that they can't.

As for defining distutils as incompatible with the PEP, I'm okay with 
that. Updating distutils to use subprocess for launching tools rather 
than spawnv would be a very good start (and likely a good contribution 
for a new contributor), but allowing build tools to continue to be 
written badly is not worthwhile.

Cheers,
Steve


From donald at stufft.io  Sat May 20 19:48:06 2017
From: donald at stufft.io (Donald Stufft)
Date: Sat, 20 May 2017 19:48:06 -0400
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CACac1F_mjeA7B6+7HqiiL_gqcT5BWAP_dKEWY7smVA8BoP-G6A@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>
 <1495267432.2610777.982827576.60026133@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <CACac1F_mjeA7B6+7HqiiL_gqcT5BWAP_dKEWY7smVA8BoP-G6A@mail.gmail.com>
Message-ID: <8653C5A9-D01B-4E62-8C67-922F17891C06@stufft.io>


> On May 20, 2017, at 4:05 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> I'm a little concerned if we're going to end up with a proposal that
> means that distutils is in violation of the spec unless this issue is
> fixed. I'm not sure if that's where we're headed, but I wanted to be
> clear here - is PEP 517 intended to encompass distutils/setuptools, or
> are we treating them as a legacy case, that pip should special-case?


I don?t think distutils/setuptools are going to be compatible out of the box anyways, because it?s API is tied to setup.py. Whatever adapter is written to adapt it to PEP 517 can handle any semantic differences as well.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170520/9f75512f/attachment-0001.html>

From ncoghlan at gmail.com  Sun May 21 00:30:32 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 21 May 2017 14:30:32 +1000
Subject: [Distutils] Finally making distlib handle spaces
In-Reply-To: <CADB4rJEoOxYUzgSys4mQnQtZMTCFFgcdcJ0kxL=-MKX5LNfosA@mail.gmail.com>
References: <CADB4rJGZ-XtdnZxsvqqe0dFFLWrh8PPy0M2qtsbM37kBfpG6vg@mail.gmail.com>
 <CACac1F_cYCUbL6Ma0Go_E7ctJPj76ufB2uZmw9Ec82kTBtd7TQ@mail.gmail.com>
 <CADB4rJEoOxYUzgSys4mQnQtZMTCFFgcdcJ0kxL=-MKX5LNfosA@mail.gmail.com>
Message-ID: <CADiSq7ccaxH4=_CmPTjFc9k+kZ4Hjm1G4Y_be6jfZoQ-3uw=EQ@mail.gmail.com>

On 21 May 2017 at 07:29, Radon Rosborough <radon.neon at gmail.com> wrote:
>> The current behaviour fails, as you note, but it does so in a
>> "standard" way - shebang behaviour (and its limitations) is
>> well-known.
>
> I agree with this, but in my opinion the shebang is simply an
> implementation detail of virtualenv. I would like to quote @JDLH from
> [1]: "There is nothing about the value provided by virtualenv that
> demands it use the shebang." If the shebang were fundamentally
> necessary to provide the functionality of virtualenv, it would make
> sense for virtualenv to inherit the shebang's restrictions. But this
> is not the case, so I think that the shebang should be considered an
> implementation detail that the end user should not need to be aware
> of.

I agree with this way of looking at the problem, so my perspective would be:

1. Don't change anything on Windows (since that already uses the
custom 'py' dispatcher)
2. Don't change anything for cases where platform provided shebang
dispatch is trusted to be correct (i.e. no quoting of the shebang line
is needed)
3. Change the cases that currently quote the shebang line to instead
invoke a custom dispatch script running in /bin/sh

I also agree that distlib is the right level to implement the change -
this isn't about people wanting custom behaviour, it's about distlib's
default dispatch mechanism having been found not to work in certain
cases, so it makes sense to automatically switch to an alternative
that *does* work.

The custom dispatcher doesn't need to solve 100% of the currently
failing cases - it just needs to solve some of them, and provide a
foundation for future iterations on the design and implementation to
handle more.

Cheers,
Nick.

P.S. I was originally going to ask "Can we use Python to implement the
dispatch instead?", but then realised there are actually lots of messy
complications with that around getting program metadata and
environmental details right that sh deals with natively. So /bin/sh is
a better way to go.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun May 21 01:36:22 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 21 May 2017 15:36:22 +1000
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>
 <1495267432.2610777.982827576.60026133@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
Message-ID: <CADiSq7eXEHVDMCBLc1_-pZBi_DWtwMnfDUy6n=2scsee-5=Z3Q@mail.gmail.com>

On 21 May 2017 at 02:36, Steve Dower <steve.dower at python.org> wrote:
> On 20May2017 0820, Nick Coghlan wrote:
>>
>> Good point regarding the fact that the Windows 16-bit APIs only come
>> into play for interactive sessions (even in 3.6+), while for PEP 517
>> we're specifically interested in the 8-bit pipes used to communicate
>> with build subprocesses launched by an installation tool.
>
>
> I need to catch up on the PEP (and thanks Brett for alerting me to the
> thread), but this comment in particular cements the mental diagram I have
> right now:
>
> (build UI) <--> (build tool) <--> (compiler)
> ( Python ) <--> (  Python  ) <--> (anything)
>
> I'll probably read the PEP closely and see that this is entirely incorrect,
> but if it's right:
>
> * encoding for text between the build UI and build tool should just be
> specified once for all platforms (i.e. use UTF-8).
> * encoding for text between build tool and the compiler depends on the
> compiler

Alas, it isn't quite that simple. Let's take the current de facto standard case:


    (user console/CI build log) <-> pip <-> setup.py
(distutils/setuptools) <-> 3rd party tool

Key usability feature:

* when requested, informational messages from 3rd party tools SHOULD
be made available to the end user for debugging purposes

Ideal outcome:

* everything that makes it to the user console or CI build log is
readable by the end user

Essential requirement:

* encoding problems in informational messages emitted by 3rd party
tools MUST NOT cause the build to fail

Now, the easiest way to handle the essential requirement as the author
of an installation or build tool is to choose not to deal with it:
instead, you just treat the output from further downstream as opaque
binary data, and let the user console/CI build log layer deal with any
encoding problems as they see fit. You may end up with some build
failures that are a pain to debug because you're getting nonsense from
the build pipeline, but you won't fail your build *because* some
particular build tool emitted improperly encoded nonsense.

That all changes if we *require* UTF-8 on the link between the
installation tool (e.g. pip) and the build tool (e.g. setup.py). If we
do that:

* the installation tool can't just pass along build tool output to the
user console or CI build log any more, it has a nominal obligation to
try to interpret it as UTF-8
* the build tool (or build tool shim) can't just pass along 3rd party
tool output to the installation tool any more, it has a nominal
obligation to try to get it to emit UTF-8

Now, *particular* installation and build tools may want to strongly
encourage the use of UTF-8 in an effort to get closer to the ideal
outcome, but that isn't the key objective of PEP 517: the key
objective of PEP 517 is to make it easier to use *general purpose*
build systems that happen to be implemented in Python (like waf,
scons, and meson) to handle complex build scenarios, while also
allowing the use of simpler Python-only build systems (like flit) for
distribution of pure Python projects.

That said, the PEP *could* explicitly define a short list of
behaviours that we consider reasonable in an installation tool:

1. Treat the informational output from the build tool as an opaque binary stream
2. Treat the informational output from the build tool as a text stream
encoded using locale.getpreferredencoding(), and decode it using the
backslashreplace error handler
3. Treat the informational output from the build tool as a UTF-8
encoded text stream, and decode it using the backslashreplace error
handler

We'd just need to caveat the latter two options with the fact that
they'll give you a cryptic error message on Python 3.4 and earlier
(including Python 2):

>>> b"\xf0\x01\x02\x03".decode("utf-8", "backslashreplace")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ncoghlan/devel/py27/Lib/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
TypeError: don't know how to handle UnicodeDecodeError in error callback

I had to look that up on Stack Overflow myself, but what it's trying
to say is that until Python 3.5, "backslashreplace" only worked for
encoding, not for decoding.

That means that for earlier versions, you'd need to define your own
custom error handler as described in
http://stackoverflow.com/questions/25442954/how-should-i-decode-bytes-using-ascii-without-losing-any-junk-bytes-if-xmlch/25443356#25443356

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From njs at pobox.com  Sun May 21 03:18:05 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Sun, 21 May 2017 00:18:05 -0700
Subject: [Distutils] Finally making distlib handle spaces
In-Reply-To: <CADiSq7ccaxH4=_CmPTjFc9k+kZ4Hjm1G4Y_be6jfZoQ-3uw=EQ@mail.gmail.com>
References: <CADB4rJGZ-XtdnZxsvqqe0dFFLWrh8PPy0M2qtsbM37kBfpG6vg@mail.gmail.com>
 <CACac1F_cYCUbL6Ma0Go_E7ctJPj76ufB2uZmw9Ec82kTBtd7TQ@mail.gmail.com>
 <CADB4rJEoOxYUzgSys4mQnQtZMTCFFgcdcJ0kxL=-MKX5LNfosA@mail.gmail.com>
 <CADiSq7ccaxH4=_CmPTjFc9k+kZ4Hjm1G4Y_be6jfZoQ-3uw=EQ@mail.gmail.com>
Message-ID: <CAPJVwB=iZMs3OgB0dk+QQfRyYfLsR_1rNaBjD4LHabA-yqCthQ@mail.gmail.com>

On May 20, 2017 9:30 PM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:

On 21 May 2017 at 07:29, Radon Rosborough <radon.neon at gmail.com> wrote:
>> The current behaviour fails, as you note, but it does so in a
>> "standard" way - shebang behaviour (and its limitations) is
>> well-known.
>
> I agree with this, but in my opinion the shebang is simply an
> implementation detail of virtualenv. I would like to quote @JDLH from
> [1]: "There is nothing about the value provided by virtualenv that
> demands it use the shebang." If the shebang were fundamentally
> necessary to provide the functionality of virtualenv, it would make
> sense for virtualenv to inherit the shebang's restrictions. But this
> is not the case, so I think that the shebang should be considered an
> implementation detail that the end user should not need to be aware
> of.

I agree with this way of looking at the problem, so my perspective would be:

1. Don't change anything on Windows (since that already uses the
custom 'py' dispatcher)
2. Don't change anything for cases where platform provided shebang
dispatch is trusted to be correct (i.e. no quoting of the shebang line
is needed)
3. Change the cases that currently quote the shebang line to instead
invoke a custom dispatch script running in /bin/sh


I agree that using /bin/sh seems more sensible. I don't understand why the
proposal used /use/bin/env, and would be curious to hear if there's a
rationale...


I also agree that distlib is the right level to implement the change -
this isn't about people wanting custom behaviour, it's about distlib's
default dispatch mechanism having been found not to work in certain
cases, so it makes sense to automatically switch to an alternative
that *does* work.


+1


The custom dispatcher doesn't need to solve 100% of the currently
failing cases - it just needs to solve some of them, and provide a
foundation for future iterations on the design and implementation to
handle more.

Cheers,
Nick.

P.S. I was originally going to ask "Can we use Python to implement the
dispatch instead?", but then realised there are actually lots of messy
complications with that around getting program metadata and
environmental details right that sh deals with natively. So /bin/sh is
a better way to go.


And using python is a non starter anyway because we may not have access to
any python interpreter at a path that can be mentioned in a shebang line!

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170521/4cb0ddb4/attachment-0001.html>

From Humpback at outlook.com  Sun May 21 05:08:06 2017
From: Humpback at outlook.com (Jost now)
Date: Sun, 21 May 2017 09:08:06 +0000
Subject: [Distutils] python 3.6.1
Message-ID: <SG2PR0601MB18197650B398871001B20FBFD7FB0@SG2PR0601MB1819.apcprd06.prod.outlook.com>

My Python does not recognise pip or pip3. I should be installed in the 64 bit executable I think.

Did I forget something? (I first installed the 32 bit version by mistake but deinstalled it)

I also installed VScode, and in that the DON Jayamanne python editor. Is that only an editor?, It does  Diagnostics but does not seen to be able to execute python.

I should also install a driver to my INVIDIA I was told, but told I could do that later, not sure how.

Any advice please.

PS: I always have the same problem with downloading programs, thet does not seem to be any guides that actually Works?


Sendt fra Mail<https://go.microsoft.com/fwlink/?LinkId=550986> til Windows 10

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170521/bc56d01d/attachment.html>

From jwilk at jwilk.net  Sat May 20 17:56:22 2017
From: jwilk at jwilk.net (Jakub Wilk)
Date: Sat, 20 May 2017 23:56:22 +0200
Subject: [Distutils] Finally making distlib handle spaces
In-Reply-To: <CADB4rJEoOxYUzgSys4mQnQtZMTCFFgcdcJ0kxL=-MKX5LNfosA@mail.gmail.com>
References: <CADB4rJGZ-XtdnZxsvqqe0dFFLWrh8PPy0M2qtsbM37kBfpG6vg@mail.gmail.com>
 <CACac1F_cYCUbL6Ma0Go_E7ctJPj76ufB2uZmw9Ec82kTBtd7TQ@mail.gmail.com>
 <CADB4rJEoOxYUzgSys4mQnQtZMTCFFgcdcJ0kxL=-MKX5LNfosA@mail.gmail.com>
Message-ID: <20170520215622.v2mh52gidlfj2lh6@jwilk.net>

* Radon Rosborough <radon.neon at gmail.com>, 2017-05-20, 14:29:
>>I would have thought that "#!/usr/bin/env sh" runs the risk of picking up a 
>>malicious sh executable injected into the user's PATH.
>That's certainly a valid concern. Does this happen in the real world? I feel 
>like if you have a malicious sh executable on your PATH, you're going to have 
>a lot more problems than just from virtualenv.

Right. It's safe to assume that all elements of PATH are trusted. If they're 
not, all bets are off.

That said, I'm surprised that "#!/usr/bin/env sh" was proposed instead of the 
more conventional "#!/bin/sh". I don't really see any advantages of the former.

-- 
Jakub Wilk

From sebastienturr at gmail.com  Sat May 20 18:45:45 2017
From: sebastienturr at gmail.com (=?UTF-8?Q?S=C3=A9bastien_Turrel?=)
Date: Sun, 21 May 2017 00:45:45 +0200
Subject: [Distutils] installing pip or at least modules on mac
Message-ID: <CANLsdJFNaDpgJegtArMcFWRvpsqPitixZ4YpsJTvbK1wnn1N6w@mail.gmail.com>

Hello from France !
I am using python 3,5 since month and i am no problems to install modules
on my window Os but i can't do it on my Mac.
I got Sierra Os, maybe is that a problem ?
i mean, i can't find pip on my mac, after search, can't find it, i spent
hours on internet to find a way to install pip and evey times it's not
working, maybe can you help me ?
thanks a lot !

-- 
S?bastien TURREL
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170521/c8be3090/attachment-0001.html>

From sebastienturr at gmail.com  Sat May 20 19:01:44 2017
From: sebastienturr at gmail.com (=?UTF-8?Q?S=C3=A9bastien_Turrel?=)
Date: Sun, 21 May 2017 01:01:44 +0200
Subject: [Distutils] installing pip or at least modules on mac
In-Reply-To: <CANLsdJFNaDpgJegtArMcFWRvpsqPitixZ4YpsJTvbK1wnn1N6w@mail.gmail.com>
References: <CANLsdJFNaDpgJegtArMcFWRvpsqPitixZ4YpsJTvbK1wnn1N6w@mail.gmail.com>
Message-ID: <CANLsdJFRNATjq_M-uXP4Lh7Bamr0N_muREvhoxSbXJQDH+Bzdg@mail.gmail.com>

iMac-de-Sebastien:3.5 sebastienturrel$ cd bin
iMac-de-Sebastien:bin sebastienturrel$ ls
2to3            pydoc3            python3.5-config
2to3-3.5        pydoc3.5        python3.5m
easy_install-3.5    python3            python3.5m-config
idle3            python3-32        pyvenv
idle3.5            python3-config        pyvenv-3.5
pip3            python3.5
pip3.5            python3.5-32
iMac-de-Sebastien:bin sebastienturrel$ pip3.5

Usage:
  pip <command> [options]

Commands:
  install                     Install packages.
  download                    Download packages.
  uninstall                   Uninstall packages.
  freeze                      Output installed packages in requirements
format.
  list                        List installed packages.
  show                        Show information about installed packages.
  check                       Verify installed packages have compatible
dependencies.
  search                      Search PyPI for packages.
  wheel                       Build wheels from your requirements.
  hash                        Compute hashes of package archives.
  completion                  A helper command used for command completion.
  help                        Show help for commands.

General Options:
  -h, --help                  Show help.
  --isolated                  Run pip in an isolated mode, ignoring
                              environment variables and user configuration.
  -v, --verbose               Give more output. Option is additive, and can
be
                              used up to 3 times.
  -V, --version               Show version and exit.
  -q, --quiet                 Give less output. Option is additive, and can
be
                              used up to 3 times (corresponding to WARNING,
                              ERROR, and CRITICAL logging levels).
  --log <path>                Path to a verbose appending log.
  --proxy <proxy>             Specify a proxy in the form
                              [user:passwd@]proxy.server:port.
  --retries <retries>         Maximum number of retries each connection
should
                              attempt (default 5 times).
  --timeout <sec>             Set the socket timeout (default 15 seconds).
  --exists-action <action>    Default action when a path already exists:
                              (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort.
  --trusted-host <hostname>   Mark this host as trusted, even though it does
                              not have valid or any HTTPS.
  --cert <path>               Path to alternate CA bundle.
  --client-cert <path>        Path to SSL client certificate, a single file
                              containing the private key and the certificate
                              in PEM format.
  --cache-dir <dir>           Store the cache data in <dir>.
  --no-cache-dir              Disable the cache.
  --disable-pip-version-check
                              Don't periodically check PyPI to determine
                              whether a new version of pip is available for
                              download. Implied with --no-index.
iMac-de-Sebastien:bin sebastienturrel$ install PyPDF2
usage: install [-bCcpSsv] [-B suffix] [-f flags] [-g group] [-m mode]
               [-o owner] file1 file2
       install [-bCcpSsv] [-B suffix] [-f flags] [-g group] [-m mode]
               [-o owner] file1 ... fileN directory
       install -d [-v] [-g group] [-m mode] [-o owner] directory ...
iMac-de-Sebastien:bin sebastienturrel$ -m install PyPDF2
-bash: -m: command not found
iMac-de-Sebastien:bin sebastienturrel$ install PyPDf2
$usage: install [-bCcpSsv] [-B suffix] [-f flags] [-g group] [-m mode]
               [-o owner] file1 file2
       install [-bCcpSsv] [-B suffix] [-f flags] [-g group] [-m mode]
               [-o owner] file1 ... fileN directory
       install -d [-v] [-g group] [-m mode] [-o owner] directory ...
iMac-de-Sebastien:bin sebastienturrel$ -M
-bash: -M: command not found
iMac-de-Sebastien:bin sebastienturrel$ -m
-bash: -m: command not found
iMac-de-Sebastien:bin sebastienturrel$ install -o PyPDF2
usage: install [-bCcpSsv] [-B suffix] [-f flags] [-g group] [-m mode]
               [-o owner] file1 file2
       install [-bCcpSsv] [-B suffix] [-f flags] [-g group] [-m mode]
               [-o owner] file1 ... fileN directory
       install -d [-v] [-g group] [-m mode] [-o owner] directory ...
iMac-de-Sebastien:bin sebastienturrel$


2017-05-21 0:45 GMT+02:00 S?bastien Turrel <sebastienturr at gmail.com>:

> Hello from France !
> I am using python 3,5 since month and i am no problems to install modules
> on my window Os but i can't do it on my Mac.
> I got Sierra Os, maybe is that a problem ?
> i mean, i can't find pip on my mac, after search, can't find it, i spent
> hours on internet to find a way to install pip and evey times it's not
> working, maybe can you help me ?
> thanks a lot !
>
> --
> S?bastien TURREL
>



-- 
S?bastien TURREL
06 88 79 77 11
17 rue des Marguettes
75012 Paris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170521/7e8ea71f/attachment-0001.html>

From matthew.brett at gmail.com  Sun May 21 07:23:19 2017
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sun, 21 May 2017 12:23:19 +0100
Subject: [Distutils] installing pip or at least modules on mac
In-Reply-To: <CANLsdJFRNATjq_M-uXP4Lh7Bamr0N_muREvhoxSbXJQDH+Bzdg@mail.gmail.com>
References: <CANLsdJFNaDpgJegtArMcFWRvpsqPitixZ4YpsJTvbK1wnn1N6w@mail.gmail.com>
 <CANLsdJFRNATjq_M-uXP4Lh7Bamr0N_muREvhoxSbXJQDH+Bzdg@mail.gmail.com>
Message-ID: <CAH6Pt5oVc9FXVCJKzwVjgKmX2bsS+ZdqWLhgV-LS=nShj8JXOg@mail.gmail.com>

Hi,

On Sun, May 21, 2017 at 12:01 AM, S?bastien Turrel
<sebastienturr at gmail.com> wrote:
> iMac-de-Sebastien:3.5 sebastienturrel$ cd bin
> iMac-de-Sebastien:bin sebastienturrel$ ls
> 2to3            pydoc3            python3.5-config
> 2to3-3.5        pydoc3.5        python3.5m
> easy_install-3.5    python3            python3.5m-config
> idle3            python3-32        pyvenv
> idle3.5            python3-config        pyvenv-3.5
> pip3            python3.5
> pip3.5            python3.5-32
> iMac-de-Sebastien:bin sebastienturrel$ pip3.5
>
> Usage:
>   pip <command> [options]
>
> Commands:
>   install                     Install packages.
>   download                    Download packages.
>   uninstall                   Uninstall packages.
>   freeze                      Output installed packages in requirements
> format.
>   list                        List installed packages.
>   show                        Show information about installed packages.
>   check                       Verify installed packages have compatible
> dependencies.
>   search                      Search PyPI for packages.
>   wheel                       Build wheels from your requirements.
>   hash                        Compute hashes of package archives.
>   completion                  A helper command used for command completion.
>   help                        Show help for commands.
>
> General Options:
>   -h, --help                  Show help.
>   --isolated                  Run pip in an isolated mode, ignoring
>                               environment variables and user configuration.
>   -v, --verbose               Give more output. Option is additive, and can
> be
>                               used up to 3 times.
>   -V, --version               Show version and exit.
>   -q, --quiet                 Give less output. Option is additive, and can
> be
>                               used up to 3 times (corresponding to WARNING,
>                               ERROR, and CRITICAL logging levels).
>   --log <path>                Path to a verbose appending log.
>   --proxy <proxy>             Specify a proxy in the form
>                               [user:passwd@]proxy.server:port.
>   --retries <retries>         Maximum number of retries each connection
> should
>                               attempt (default 5 times).
>   --timeout <sec>             Set the socket timeout (default 15 seconds).
>   --exists-action <action>    Default action when a path already exists:
>                               (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort.
>   --trusted-host <hostname>   Mark this host as trusted, even though it does
>                               not have valid or any HTTPS.
>   --cert <path>               Path to alternate CA bundle.
>   --client-cert <path>        Path to SSL client certificate, a single file
>                               containing the private key and the certificate
>                               in PEM format.
>   --cache-dir <dir>           Store the cache data in <dir>.
>   --no-cache-dir              Disable the cache.
>   --disable-pip-version-check
>                               Don't periodically check PyPI to determine
>                               whether a new version of pip is available for
>                               download. Implied with --no-index.
> iMac-de-Sebastien:bin sebastienturrel$ install PyPDF2

Here you're running the 'install' system command.  What you want is to
run the pip 'install' subcommand:

On my machine:

$ pip install PyPDF2
Collecting PyPDF2
  Downloading PyPDF2-1.26.0.tar.gz (77kB)
    100% |????????????????????????????????| 81kB 824kB/s
Installing collected packages: PyPDF2
  Running setup.py install for PyPDF2 ... done
Successfully installed PyPDF2-1.26.0

Best,

Matthew

From steve.dower at python.org  Sun May 21 09:34:09 2017
From: steve.dower at python.org (Steve Dower)
Date: Sun, 21 May 2017 06:34:09 -0700
Subject: [Distutils] python 3.6.1
In-Reply-To: <SG2PR0601MB18197650B398871001B20FBFD7FB0@SG2PR0601MB1819.apcprd06.prod.outlook.com>
References: <SG2PR0601MB18197650B398871001B20FBFD7FB0@SG2PR0601MB1819.apcprd06.prod.outlook.com>
Message-ID: <E1dCQzx-0008FR-KI@se2-syd.hostedmail.net.au>

Hi Jost

For Windows, those commands are typically not available by default (on PATH) because it is difficult to automatically add them.

You should be able to use ?py.exe -m pip? as a substitute for ?pip?. Or you can modify your default PATH environment variable to include your Python directory and the Scripts directory inside of it.

VS Code is basically just an editor. It does not have functionality to help with packages (the full Visual Studio 2017 does if you select the Python option when installing).

Cheers,
Steve

Top-posted from my Windows phone

From: Jost now
Sent: Sunday, May 21, 2017 4:18
To: distutils-sig at python.org
Subject: [Distutils] python 3.6.1

My Python does not recognise pip or pip3. I should be installed in the 64 bit executable I think.

Did I forget something? (I first installed the 32 bit version by mistake but deinstalled it)

I also installed VScode, and in that the DON Jayamanne python editor. Is that only an editor?, It does ?Diagnostics but does not seen to be able to execute python.

I should also install a driver to my INVIDIA I was told, but told I could do that later, not sure how.

Any advice please.

PS: I always have the same problem with downloading programs, thet does not seem to be any guides that actually Works?


Sendt fra Mail til Windows 10


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170521/d1034817/attachment.html>

From thomas at kluyver.me.uk  Mon May 22 05:56:56 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Mon, 22 May 2017 10:56:56 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>
 <1495267432.2610777.982827576.60026133@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
Message-ID: <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>

On Sat, May 20, 2017, at 07:36 PM, Steve Dower wrote:
> Require that build tools either send UTF-8 to the UI component, or write 
> bytes to a file and call it a build output. I see no benefit in 
> requiring both the build tool and the UI tool to guess what the text 
> encoding is.

I'm not proposing that the install tool should try to guess the
encoding, but I think a well written install tool shouldn't crash if the
build output doesn't match the encoding it expects. Even if the spec
says that the build output MUST be UTF-8 encoded, build tools can have
bugs, and you don't want want the install to fail just because the log
isn't correctly encoded.

Hence, I think a 'SHOULD' is appropriate for this part of the spec:

- To install tool authors, it is clear that they can display the output
as UTF-8 so long as they don't crash if it's invalid.
- To build tool authors, it's clear that they can't pass the buck to
install tool authors if output gets jumbled because it's not UTF-8.

Thomas

From p.f.moore at gmail.com  Mon May 22 06:19:40 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 22 May 2017 11:19:40 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>
 <1495267432.2610777.982827576.60026133@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
Message-ID: <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>

On 22 May 2017 at 10:56, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> On Sat, May 20, 2017, at 07:36 PM, Steve Dower wrote:
>> Require that build tools either send UTF-8 to the UI component, or write
>> bytes to a file and call it a build output. I see no benefit in
>> requiring both the build tool and the UI tool to guess what the text
>> encoding is.
>
> I'm not proposing that the install tool should try to guess the
> encoding, but I think a well written install tool shouldn't crash if the
> build output doesn't match the encoding it expects. Even if the spec
> says that the build output MUST be UTF-8 encoded, build tools can have
> bugs, and you don't want want the install to fail just because the log
> isn't correctly encoded.
>
> Hence, I think a 'SHOULD' is appropriate for this part of the spec:
>
> - To install tool authors, it is clear that they can display the output
> as UTF-8 so long as they don't crash if it's invalid.
> - To build tool authors, it's clear that they can't pass the buck to
> install tool authors if output gets jumbled because it's not UTF-8.

I'd say that it's not so much just "well written" install tools. I'd
say that install tools MUST NOT crash if build tool output isn't in
the expected encoding. On the other hand, the encoding agreement
implies that if build tools *do* send data in the correct encoding
then they are entitled to expect that it will be displayed accurately
to the end user.

Output can be garbled in two ways:

1. The build tool does not (or cannot) ensure that its output is in
the standard-mandated encoding.
2. The install tool cannot display the full range of characters
representable in the standard-mandated encoding.

Neither of these should cause a failure. Well written install tools
should warn in the case of (1) - "I have been passed data that I don't
understand, I'll do my best to display it but can't guarantee the
output won't be garbled". In the case of (2), though, that's "as
expected" - if your OS settings mean you can't display certain
characters, you shouldn't be surprised if your install tool replaces
them with a placeholder.

On an implementation note, this boils down to something like the
following in the install tool:

    # Step 1
    try:
        data = decode build output using STD_ENCODING
    except UnicodeDecodeError:
        warn "Data is not in expected encoding"
        data = decode using STD_ENCODING with errors=<some form of replacement>

    # Step 2
    data = data.encode(MY_OUTPUT_ENCODING, errors=<some form of
replacement>).decode(MY_OUTPUT_ENCODING)

    # We now have subprocess output that's safe to display if requested.

As a side note, I find step 2 "sanitise my string to ensure it can be
safely output" to be a pretty common operation - possibly because
Python's standard IO streams raise exceptions on unicode errors - and
I'm surprised there isn't a better way to spell it than the
encode/decode pair above.

Paul

From thomas at kluyver.me.uk  Mon May 22 06:22:15 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Mon, 22 May 2017 11:22:15 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>
 <1495267432.2610777.982827576.60026133@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
Message-ID: <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>

I have made a PR against the PEP with my best take on the encoding
situation:
https://github.com/python/peps/pull/264/files

On Mon, May 22, 2017, at 11:19 AM, Paul Moore wrote:
> On 22 May 2017 at 10:56, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> > On Sat, May 20, 2017, at 07:36 PM, Steve Dower wrote:
> >> Require that build tools either send UTF-8 to the UI component, or write
> >> bytes to a file and call it a build output. I see no benefit in
> >> requiring both the build tool and the UI tool to guess what the text
> >> encoding is.
> >
> > I'm not proposing that the install tool should try to guess the
> > encoding, but I think a well written install tool shouldn't crash if the
> > build output doesn't match the encoding it expects. Even if the spec
> > says that the build output MUST be UTF-8 encoded, build tools can have
> > bugs, and you don't want want the install to fail just because the log
> > isn't correctly encoded.
> >
> > Hence, I think a 'SHOULD' is appropriate for this part of the spec:
> >
> > - To install tool authors, it is clear that they can display the output
> > as UTF-8 so long as they don't crash if it's invalid.
> > - To build tool authors, it's clear that they can't pass the buck to
> > install tool authors if output gets jumbled because it's not UTF-8.
> 
> I'd say that it's not so much just "well written" install tools. I'd
> say that install tools MUST NOT crash if build tool output isn't in
> the expected encoding. On the other hand, the encoding agreement
> implies that if build tools *do* send data in the correct encoding
> then they are entitled to expect that it will be displayed accurately
> to the end user.
> 
> Output can be garbled in two ways:
> 
> 1. The build tool does not (or cannot) ensure that its output is in
> the standard-mandated encoding.
> 2. The install tool cannot display the full range of characters
> representable in the standard-mandated encoding.
> 
> Neither of these should cause a failure. Well written install tools
> should warn in the case of (1) - "I have been passed data that I don't
> understand, I'll do my best to display it but can't guarantee the
> output won't be garbled". In the case of (2), though, that's "as
> expected" - if your OS settings mean you can't display certain
> characters, you shouldn't be surprised if your install tool replaces
> them with a placeholder.
> 
> On an implementation note, this boils down to something like the
> following in the install tool:
> 
>     # Step 1
>     try:
>         data = decode build output using STD_ENCODING
>     except UnicodeDecodeError:
>         warn "Data is not in expected encoding"
>         data = decode using STD_ENCODING with errors=<some form of
>         replacement>
> 
>     # Step 2
>     data = data.encode(MY_OUTPUT_ENCODING, errors=<some form of
> replacement>).decode(MY_OUTPUT_ENCODING)
> 
>     # We now have subprocess output that's safe to display if requested.
> 
> As a side note, I find step 2 "sanitise my string to ensure it can be
> safely output" to be a pretty common operation - possibly because
> Python's standard IO streams raise exceptions on unicode errors - and
> I'm surprised there isn't a better way to spell it than the
> encode/decode pair above.
> 
> Paul

From p.f.moore at gmail.com  Mon May 22 07:02:14 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 22 May 2017 12:02:14 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>
 <1495267432.2610777.982827576.60026133@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
Message-ID: <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>

On 22 May 2017 at 11:22, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> I have made a PR against the PEP with my best take on the encoding
> situation:
> https://github.com/python/peps/pull/264/files

LGTM.

The only reservation I have is that the choice of UTF-8 means that on
Windows, build backends pretty much have to explicitly manage tool
output (as they are pretty much certain *not* to output in UTF-8).
Build backend writers that aren't aware of this issue (most likely
because their main platform is not Windows) could very easily choose
to just pass through the raw bytes, and as a result *all* non-ASCII
output would be garbled on non-UTF-8 systems.

Would locale.getpreferredencoding() not be a better choice here? I
know it has issues in some situations on Unix, but are they worse than
the issues UTF-8 would cause on Windows? After all it's the encoding
used by subprocess.Popen in "universal newlines" mode...

Paul

From thomas at kluyver.me.uk  Mon May 22 07:28:17 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Mon, 22 May 2017 12:28:17 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>
 <1495267432.2610777.982827576.60026133@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
Message-ID: <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>

On Mon, May 22, 2017, at 12:02 PM, Paul Moore wrote:
> The only reservation I have is that the choice of UTF-8 means that on
> Windows, build backends pretty much have to explicitly manage tool
> output (as they are pretty much certain *not* to output in UTF-8).
> Build backend writers that aren't aware of this issue (most likely
> because their main platform is not Windows) could very easily choose
> to just pass through the raw bytes, and as a result *all* non-ASCII
> output would be garbled on non-UTF-8 systems.
> 
> Would locale.getpreferredencoding() not be a better choice here? I
> know it has issues in some situations on Unix, but are they worse than
> the issues UTF-8 would cause on Windows? After all it's the encoding
> used by subprocess.Popen in "universal newlines" mode...

What if it wants to send a character which can't be encoded in the
locale encoding? It's quite easy on Windows to end up with a character
that you can't encode as cp1252. If the build tool uses .encode(loc_enc,
'replace'), then you've lost information even before it gets to the
install tool.

It's 2017, I really don't want to go down the 'locale specified
encoding' route again. UTF-8 everywhere!

One affordance I'd consider is a recommendation to install tools that if
captured output is not valid UTF-8, they dump the raw bytes to a file so
that no information is lost. I'm not sure if that recommendation needs
to be in the spec itself, though.

Thomas

From ncoghlan at gmail.com  Mon May 22 07:29:41 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 22 May 2017 21:29:41 +1000
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>
 <1495267432.2610777.982827576.60026133@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
Message-ID: <CADiSq7ehpZVjtxPbr6cXev9kiy9KGP68FhAEaopsePMzdgNa=Q@mail.gmail.com>

On 22 May 2017 at 21:02, Paul Moore <p.f.moore at gmail.com> wrote:
> On 22 May 2017 at 11:22, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
>> I have made a PR against the PEP with my best take on the encoding
>> situation:
>> https://github.com/python/peps/pull/264/files
>
> LGTM.
>
> The only reservation I have is that the choice of UTF-8 means that on
> Windows, build backends pretty much have to explicitly manage tool
> output (as they are pretty much certain *not* to output in UTF-8).
> Build backend writers that aren't aware of this issue (most likely
> because their main platform is not Windows) could very easily choose
> to just pass through the raw bytes, and as a result *all* non-ASCII
> output would be garbled on non-UTF-8 systems.
>
> Would locale.getpreferredencoding() not be a better choice here? I
> know it has issues in some situations on Unix, but are they worse than
> the issues UTF-8 would cause on Windows? After all it's the encoding
> used by subprocess.Popen in "universal newlines" mode...

+1 from me for locale.getpreferredencoding() as the default - not only
is it a more suitable default on Windows, it's also the best way to do
the right thing in GB.18030 locales, and as far as I'm aware, handling
that correctly is still a requirement for selling commercial software
into China (that's why I chose it as the main non-UTF-8 example
encoding in PEP 538).

If Python tools want to specifically detect the use of 7-bit ASCII and
override *that* to be UTF-8, then the relevant snippet is:

    def get_stream_encoding():
        nominal = locale.getpreferredencoding()
        if codecs.lookup(nominal).name == "ascii":
            return "utf-8"
        return nominal

That's effectively the same model that PEP 538 and 540 are proposing
be applied by default for the standard streams, so it would also
interoperate well with Python 3.7+.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Mon May 22 08:06:14 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 22 May 2017 22:06:14 +1000
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>
 <1495267432.2610777.982827576.60026133@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
Message-ID: <CADiSq7ew0E_TVVAMs_eAVdCw_zsEZnrd9_Pie5u0qb6WDFbJXA@mail.gmail.com>

On 22 May 2017 at 21:28, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> On Mon, May 22, 2017, at 12:02 PM, Paul Moore wrote:
>> The only reservation I have is that the choice of UTF-8 means that on
>> Windows, build backends pretty much have to explicitly manage tool
>> output (as they are pretty much certain *not* to output in UTF-8).
>> Build backend writers that aren't aware of this issue (most likely
>> because their main platform is not Windows) could very easily choose
>> to just pass through the raw bytes, and as a result *all* non-ASCII
>> output would be garbled on non-UTF-8 systems.
>>
>> Would locale.getpreferredencoding() not be a better choice here? I
>> know it has issues in some situations on Unix, but are they worse than
>> the issues UTF-8 would cause on Windows? After all it's the encoding
>> used by subprocess.Popen in "universal newlines" mode...
>
> What if it wants to send a character which can't be encoded in the
> locale encoding? It's quite easy on Windows to end up with a character
> that you can't encode as cp1252. If the build tool uses .encode(loc_enc,
> 'replace'), then you've lost information even before it gets to the
> install tool.

The counterargument is that there's plenty of text that *can* be
correctly encoded in cp1252 (especially in Europe and LATAM) that will
be rendered incorrectly if the installation tool attempts to interpret
it as UTF-8. CPython itself will also display explicitly UTF-8 encoded
text incorrectly on a Windows console in versions prior to 3.6.

> It's 2017, I really don't want to go down the 'locale specified
> encoding' route again. UTF-8 everywhere!

"UTF-8 everywhere" is fine for network services that only need to talk
to other network services, command line applications, and web
browsers, but even in 2017 it's still a problematic assumption on
client devices running Windows or Linux.

Rather than the locale specified encoding being broken in general, the
key recurring problem we've found with it on *nix systems relates to
the fact that glibc still defaults to ASCII in the C locale - "assume
ASCII really means UTF-8" is enough to solve that problem *without*
breaking compatibility with cp1252 and non-UTF-8 universal encodings.

The other recurring problem is cp1252 itself on Windows, which suffers
from the fact that there isn't a nice environment variable based way
to change the active code page when invoking a subprocess, and also
that cp65001 (the UTF-8 code page) isn't really properly supported in
Python 2.7 (although you can inject a custom search function to alias
it to utf-8 [1]).

Even in that case though, mandating "though shalt treat the streams as
UTF-8" in the spec doesn't *solve* those problems - it just means
we're specifying a behaviour that we know will provide a poor
developer experience on Windows, rather than alerting tool developers
to the fact that this is something they're going to need to be aware
of.

Cheers,
Nick.

[1] http://neurocline.github.io/dev/2016/10/13/python-utf8-windows.html

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From p.f.moore at gmail.com  Mon May 22 09:15:09 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 22 May 2017 14:15:09 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>
 <1495267432.2610777.982827576.60026133@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
Message-ID: <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>

On 22 May 2017 at 12:28, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> What if it wants to send a character which can't be encoded in the
> locale encoding? It's quite easy on Windows to end up with a character
> that you can't encode as cp1252. If the build tool uses .encode(loc_enc,
> 'replace'), then you've lost information even before it gets to the
> install tool.
>
> It's 2017, I really don't want to go down the 'locale specified
> encoding' route again. UTF-8 everywhere!

Hang on. Can we take a step back here? I just re-read the PEP and
remembered (!) that hooks are *in-process* Python entry points (I've
been working with pip's current backend-as-subprocess model, and mixed
up in my mind the original 2 proposals here). I think this encoding
debate may be a red herring.

If a hook is being called as a Python method call, then it can print
what it likes to stdout and stderr. And it's the backend's
responsibility to ensure that it never fails when printing - so the
*backend* has to deal with the fact that anything it wants to print
must be representable in sys.stdout.encoding, with the default (raise
an exception) error handling. Given this fact, and the fact that
sys.stdout and sys.stderr are *text* output streams, build frontends
like pip can reasonably just replace sys.std{out,err} (for example
with a StringIO object) to get hook output. There's no encoding issue
for frontends, they just capture the text sent to the stdio streams.

The rules needed for *backends* are then:

1. Backends MUST NOT write to raw IO channels, all output MUST go via
sys.stdout and sys.stderr. Build frontends MAY redirect these streams
to post-process them, but are not required to do so. As a consequence:

  1a. Backends MUST be prepared to deal with the possibility that
those IO streams have the limitations of the platform IO streams
(e.g., limited subset of Unicode allowed, fails with an exception when
invalid characters are written).

  1b. Backends MUST capture and manage the output from any
subprocesses they spawn (so that they can follow the other rules).

  1c. Backends cannot assume that they can write output that the user
will see - frontends may suppress or modify any output passed on
stdout. Conversely, backends should not bypass the ability of
frontends to capture stdout, as frontends are responsible for user
interaction.

Some of those MUSTs could be replaced by SHOULD, if we want to allow
backends to write directly to the screen. But that is likely to
corrupt the UI of the frontend, so I'm inclined to say that we don't
allow that.

Paul

From ncoghlan at gmail.com  Mon May 22 10:23:29 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 23 May 2017 00:23:29 +1000
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>
 <1495267432.2610777.982827576.60026133@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
Message-ID: <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>

On 22 May 2017 at 23:15, Paul Moore <p.f.moore at gmail.com> wrote:
> On 22 May 2017 at 12:28, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
>> What if it wants to send a character which can't be encoded in the
>> locale encoding? It's quite easy on Windows to end up with a character
>> that you can't encode as cp1252. If the build tool uses .encode(loc_enc,
>> 'replace'), then you've lost information even before it gets to the
>> install tool.
>>
>> It's 2017, I really don't want to go down the 'locale specified
>> encoding' route again. UTF-8 everywhere!
>
> Hang on. Can we take a step back here? I just re-read the PEP and
> remembered (!) that hooks are *in-process* Python entry points (I've
> been working with pip's current backend-as-subprocess model, and mixed
> up in my mind the original 2 proposals here). I think this encoding
> debate may be a red herring.

No, that's discussed here:
https://www.python.org/dev/peps/pep-0517/#comparison-to-competing-proposals

Even though PEP 517 defines a Python API for build backends to
implement, it still expects installation tools to wrap a subprocess
call around the backend invocation.

Frontends needs to do that in order to protect *their own* process
state from bugs and design quirks in backend implementations:

- no monkeypatching of parent process modules
- no changes to the standard stream configuration
- no persistent locale changes
- no environment variable changes
- no manipulation of any other process global state
- calling sys.exit() won't cryptically crash the entire installation process
- memory leaks won't cryptically crash the entire installation process
- infinite loops won't *necessarily* crash the entire installation
process (if the build has a timeout on it)
- installation tools running with elevated privileges can readily run
the build process with reduced privileges
- installation tools can also readily run the build process in a
chroot or containerised environment

And in the context of this thread, it gives the frontend complete
control over the stream output from not only the backend itself, but
any child processes that it launches.

That said, the whole "The build backend still runs in a subprocess"
aspect should probably be separated out into its own section
"Isolating build backends from frontend process state", rather than
solely being covered in the "Comparison to PEP 516?" section, as it's
a key aspect of the design - we expect each installation tool to
provide its own CLI shim for calling build backends, rather than
requiring all installation tools to use the same one.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Mon May 22 10:40:32 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 23 May 2017 00:40:32 +1000
Subject: [Distutils] Last call for PEP 516 champions
Message-ID: <CADiSq7fDUb_asZ8-bmGhEhEQDnJ+vC6F2n5=N3CMUibSbZjTTg@mail.gmail.com>

Hi folks,

The restarted discussion on PEP 517 meant I realised that we hadn't
officially decided between its Python API based approach and PEP 516's
approach of using the backend CLI as the standardised interface (akin
to the current setup.py approach).

My current intention is to reject PEP 516's CLI standardisation
approach on the following grounds:

- PEP 517 makes a convincing case for the benefits of the Python API
based approach within the Python ecosystem
- the difficulties encountered in evolving the setup.py CLI over time
lend significant weight to the notion that a Python level API will be
easier to update without breaking backwards compatibility
- PEP 517 still advises front-ends to isolate back-end invocation
behind a subprocess boundary due to all of the other practical
benefits that brings, it just makes the specifics of that invocation
an implementation detail of the front-end tool
- third party tools that want an implementation language independent
CLI abstraction over the Python ecosystem (including builds) can just
use pip itself (or another standards-compliant frontend)

This isn't particularly urgent, but I also don't see a lot of reason
for an extended discussion, so if I don't hear a convincing
counterargument in the meantime, I'll mark the PEP as rejected at the
beginning of June :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From p.f.moore at gmail.com  Mon May 22 11:03:11 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 22 May 2017 16:03:11 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7cg9ostMW1c3C8O_5zbGoMOK8EaLEFgA0m4aSE=MBppTA@mail.gmail.com>
 <1495267432.2610777.982827576.60026133@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
Message-ID: <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>

On 22 May 2017 at 15:23, Nick Coghlan <ncoghlan at gmail.com> wrote:
> No, that's discussed here:
> https://www.python.org/dev/peps/pep-0517/#comparison-to-competing-proposals
>
> Even though PEP 517 defines a Python API for build backends to
> implement, it still expects installation tools to wrap a subprocess
> call around the backend invocation.

OK, but is it not acceptable for the child cmdline process (owned by
pip) to capture the backend implementation's stdout using reassignment
of sys.stdout? I assume, from your response, that it's *not*
acceptable to do that - but that needs to be documented somewhere.
Specifically, that the child cmdline is not allowed to do something
like:

    out = io.StringIO
    sys.stdout = out
    build_backend.hook()
    print(out.getvalue(), encoding="UTF-8")

(Which would otherwise be a very simple way to get guaranteed UTF-8 as
the encoding across the process boundary - but it does so by imposing
basically the rules I stated on the backend).

> That said, the whole "The build backend still runs in a subprocess"
> aspect should probably be separated out into its own section
> "Isolating build backends from frontend process state", rather than
> solely being covered in the "Comparison to PEP 516?" section, as it's
> a key aspect of the design - we expect each installation tool to
> provide its own CLI shim for calling build backends, rather than
> requiring all installation tools to use the same one.

Strong +1. And that section needs to be very clear on issues like
this, covering what the shim is allowed to do. As the point of the
shim is to protect the backend from frontend state, I'm OK with the
general principle that the shim must do "as little as possible" before
calling the hook - but "reset sys.stdout to protect against encoding
errors" could easily be seen as within the realm of acceptable
behaviour (as it stops hooks writing arbitrary Unicode to a standard
output that the shim knows is limited).

I'm happy enough with the idea that pip won't do anything silly in its
CLI shim, but we don't want to get into the "implementation as the
standard" situation where a backend is allowed to do anything that
pip's shim can cope with...

Paul

From steve.dower at python.org  Mon May 22 13:38:30 2017
From: steve.dower at python.org (Steve Dower)
Date: Mon, 22 May 2017 10:38:30 -0700
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
Message-ID: <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>

On 22May2017 0803, Paul Moore wrote:
> On 22 May 2017 at 15:23, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> No, that's discussed here:
>> https://www.python.org/dev/peps/pep-0517/#comparison-to-competing-proposals
>>
>> Even though PEP 517 defines a Python API for build backends to
>> implement, it still expects installation tools to wrap a subprocess
>> call around the backend invocation.
>
> OK, but is it not acceptable for the child cmdline process (owned by
> pip) to capture the backend implementation's stdout using reassignment
> of sys.stdout? I assume, from your response, that it's *not*
> acceptable to do that - but that needs to be documented somewhere.
> Specifically, that the child cmdline is not allowed to do something
> like:
>
>     out = io.StringIO
>     sys.stdout = out
>     build_backend.hook()
>     print(out.getvalue(), encoding="UTF-8")
>
> (Which would otherwise be a very simple way to get guaranteed UTF-8 as
> the encoding across the process boundary - but it does so by imposing
> basically the rules I stated on the backend).

Okay, I think I get the problem now. We expect backends to let child 
subprocesses just spit out whatever *they* want onto the same stdout/stderr.

I'm really not a fan of forcing front ends to clean up that mess, and so 
I'd still suggest that the backend "tool" be a script to launch the 
actual tool and do the conversion to UTF-8.

Perhaps the middle ground is to specify encoding='utf-8', 
errors='anything but strict' for front-ends, and well-behaved backends 
should do the work to transcode when it is known to be necessary for the 
tools they run. (i.e. frontends do not crash, backends have a simple 
rule for avoiding loss of data).

Cheers,
Steve

From p.f.moore at gmail.com  Mon May 22 15:53:02 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 22 May 2017 20:53:02 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
Message-ID: <CACac1F9keSPkD_aZztyy_K6OA=papJ8iCRyh-kiCR-xoMoY57A@mail.gmail.com>

On 22 May 2017 at 18:38, Steve Dower <steve.dower at python.org> wrote:
> Okay, I think I get the problem now. We expect backends to let child
> subprocesses just spit out whatever *they* want onto the same stdout/stderr.

s/expect/allow/

The paranoid in me suspects "expect" is also true, though :-)

> I'm really not a fan of forcing front ends to clean up that mess, and so I'd
> still suggest that the backend "tool" be a script to launch the actual tool
> and do the conversion to UTF-8.

What you're referring to as the backend "tool" being a script, is what
the PEP refers to as a "shim" (as Nick pointed out to me) and is
considered part of the front end. The back end is a set of Python APIs
which are called by the front end (in any real life front end, via the
front end's shim script).

> Perhaps the middle ground is to specify encoding='utf-8', errors='anything
> but strict' for front-ends, and well-behaved backends should do the work to
> transcode when it is known to be necessary for the tools they run. (i.e.
> frontends do not crash, backends have a simple rule for avoiding loss of
> data).

For front ends, "never crash" is essential. But "produce as readable
as possible data" is also a high priority. Consider for example a
Russian user with a series of directories named in Russian. If the
tools write an error using his local 8-bit encoding, and the front end
assumes UTF-8, then all of the high-bit characters in his directory
names would be replaced. Deciphering an error message like "File
???????/?????/?????.c: unexpected EOF" is problematic... :-(

The model assumes that most front-ends would call the backend via a
subprocess "shim" that was maintained by the front end project. But
the expectation here seems to be that the backend is allowed to write
directly to the stdio streams of its process (or at least, to let the
tools it calls do so). So the shim *cannot* control the encoding of
the data received by the frontend, and so the encoding has to be
agreed between backend and frontend. The basic question is how the
responsibility for dealing with data in an uncertain encoding is
allocated.

It seems to me there are 2 schools of thought:

1. There are likely to be fewer front ends than back ends, and so the
front end(s) (basically, pip) should deal with the problem. Also,
backends are more likely to be written by developers who are looking
at very specific scenarios, and asking them to handle all the
complexities of robust multilingual coding is raising the bar on
writing a backend too high.

2. The backend is where the problem lies, and so the backend should
address the issue. Furthermore, a well-established principle in
dealing with encodings is to convert to strings right at the boundary
of the application, and in this case the backend is the only code that
has access to that boundary.

(I tend towards (2), but I honestly can't say to what extent that's
because it makes it "someone else's problem" for me ;-))

As you say, the middle ground here is that front ends must never
crash, and back ends should (but aren't required to) produce output in
a specified encoding (I still prefer the locale encoding as that has
the best chance of avoiding the ????/???? issue). That's more or less
what pip has to deal with now (and not that far off (1)), and my
current attempt to address that situation is at
https://github.com/pypa/pip/pull/4486 for what it's worth.

A couple of final thoughts. I would expect that testing the handling
of encodings is likely to be an important issue (at least, I expect
there'll be bugs, and adding tests to make sure they get properly
fixed will be important). Handling tool output encoding in the backend
is likely to involve relatively low level interface functions, where
the inputs and outputs can be relatively easily mocked. So I would
expect backend unit testing of encoding handling would be relatively
straightforward. Conversely, testing front end handling of encoding
issues is very tricky - it's necessary to set up system state to
persuade the build tools to produce the data you want to test against
(it feels like integration testing rather than unit testing). Also,
fixing encoding issues in the backend decouples the fix from pip's
release cycle, which is probably a good thing (unless the backend is
not well maintained, but that's an issue in itself).

Paul

From matt at nycresistor.com  Mon May 22 15:39:46 2017
From: matt at nycresistor.com (Matt Joyce)
Date: Mon, 22 May 2017 15:39:46 -0400
Subject: [Distutils] setuptools 33.1.1 - 35 issue with source package
 install from pypi
Message-ID: <CAP_sDUFNJViyox9GbjYqVnxBnDBJh60sXg-dHv7JFYLtd0z6mA@mail.gmail.com>

https://stackoverflow.com/questions/44120045/something-is-breaking-my-package-deployment

has full info on what i am seeing, but it suffices to say:

I use setuptools in my setup.py
I iterate a requirements.txt file to satisfy dependencies.
I publish as sdist to pypi

my package used to work fine as it currently is on pypi.  however as of
late i've been seeing the python setup.py invocation ( from pip install
python-symphony ) fail with a couple different failure scenarios.

one is it attempting to install what looks like a binary wheel ( that
doesn't exist on pypi or anywhere ) that ends up being a bunch of blank
stubs.   the other is a situation where the package looks 100% correct in
site-packages, but none of the module components are registered when you
try to use the imported module.

pip install --isolated --no-cache-dir python-symphony seems to avoid the
issue.  but it also tends to do weird stuff like break bpython by messing
up the pygments install.

curious if anyone was familiar with some sort of bug that i am stumbling
into here.

-Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170522/7c1d4234/attachment.html>

From steve.dower at python.org  Mon May 22 18:36:25 2017
From: steve.dower at python.org (Steve Dower)
Date: Mon, 22 May 2017 15:36:25 -0700
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CACac1F9keSPkD_aZztyy_K6OA=papJ8iCRyh-kiCR-xoMoY57A@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CACac1F9keSPkD_aZztyy_K6OA=papJ8iCRyh-kiCR-xoMoY57A@mail.gmail.com>
Message-ID: <e0a2d8ee-d832-ece7-b300-766fbba0db2d@python.org>

On 22May2017 1253, Paul Moore wrote:
> It seems to me there are 2 schools of thought:
>
> 1. There are likely to be fewer front ends than back ends, and so the
> front end(s) (basically, pip) should deal with the problem. Also,
> backends are more likely to be written by developers who are looking
> at very specific scenarios, and asking them to handle all the
> complexities of robust multilingual coding is raising the bar on
> writing a backend too high.
>
> 2. The backend is where the problem lies, and so the backend should
> address the issue. Furthermore, a well-established principle in
> dealing with encodings is to convert to strings right at the boundary
> of the application, and in this case the backend is the only code that
> has access to that boundary.
>
> (I tend towards (2), but I honestly can't say to what extent that's
> because it makes it "someone else's problem" for me ;-))

I also tend towards 2, and I assume I am one of the more likely people 
to write the part that invokes Microsoft's cl.exe/link.exe :)

Is the front end going to be directly invoking those tools? I would 
assume not, otherwise it won't be cross platform.

Since the shim belongs to the front end, I've essentially been ignoring 
it. The shim can invoke another part of the build tool, but that is not 
going to be cl.exe/link.exe either.

At some point there will be a script that runs the tools directly. I 
have been referring to that as the backend, and it is the part that 
should handle capturing and transcoding the output. Everything from 
there can be utf8:replace to prevent crashing, but we can't say "the 
frontend can handle all encodings", and shouldn't say "the frontend will 
only use bad encodings".

IMHO, #2 is definitely the right way to go. Yes, the platform specific 
code now has to worry about the encoding, but... the encoding is 
platform specific? So... that seems exactly right? :) Maybe I'm still 
missing something here, but I'm totally happy to leave it to Thomas to 
decide (which I think he has, but I haven't gotten to looking at that PR 
yet).

Cheers,
Steve

From thomas at kluyver.me.uk  Mon May 22 19:13:58 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Tue, 23 May 2017 00:13:58 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <e0a2d8ee-d832-ece7-b300-766fbba0db2d@python.org>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CACac1F9keSPkD_aZztyy_K6OA=papJ8iCRyh-kiCR-xoMoY57A@mail.gmail.com>
 <e0a2d8ee-d832-ece7-b300-766fbba0db2d@python.org>
Message-ID: <1495494838.1867642.985189456.4B61C99B@webmail.messagingengine.com>

On Mon, May 22, 2017, at 11:36 PM, Steve Dower wrote:
> IMHO, #2 is definitely the right way to go. Yes, the platform specific 
> code now has to worry about the encoding, but... the encoding is 
> platform specific? So... that seems exactly right? :) Maybe I'm still 
> missing something here, but I'm totally happy to leave it to Thomas to 
> decide (which I think he has, but I haven't gotten to looking at that PR 
> yet).

I think I broadly agree with this as well. My reservation is that the
build backend might be running a subprocess which produces output in an
*unknown* encoding, especially if it allows the package author or the
end user to configure a command to run. If it doesn't know the encoding,
I'd rather get the raw bytes from the subprocess in the log (e.g. dumped
to a file), rather than attempting to transcode them to UTF-8 - the
conversion risks losing information, and even if it doesn't, it makes it
harder to work out what was really meant.

I feel like we're spending a lot of energy on a point that's not really
central to the PEP, though. I think we've established that there's a
potential for bugs and mojibake whatever we put in the spec. So I'd like
to put something relatively simple and move on. I still stand by my PR,
which amounts to "backends try to make it UTF-8, frontends don't crash
if it isn't". I might be persuaded to add a recommendation that
frontends dump the bytes to a file if they're not UTF-8, so the user can
pull it apart if necessary.

Thomas

From ncoghlan at gmail.com  Tue May 23 00:11:05 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 23 May 2017 14:11:05 +1000
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
Message-ID: <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>

On 23 May 2017 at 03:38, Steve Dower <steve.dower at python.org> wrote:
> Okay, I think I get the problem now. We expect backends to let child
> subprocesses just spit out whatever *they* want onto the same stdout/stderr.
>
> I'm really not a fan of forcing front ends to clean up that mess, and so I'd
> still suggest that the backend "tool" be a script to launch the actual tool
> and do the conversion to UTF-8.

One of the key premises of PEP 517 is that there will be relatively
few front ends (pip, possibly easy_install, ???), but a relatively
large number of backends (one per build system - at least
distutils/setuptools, distutils2, flit, encons, likely eventually
meson, waf, and yotta, and potentially even C/C++ build systems like
autotools, CMake, etc).

So it makes sense to put the implementation burden for important
aspects of the UX on the part that PyPA has the most influence over
(the front-end), rather than considering it reasonable for front-end
developers to point fingers and say "That UX failure in the tool we
provide isn't *our* fault, it's the fault of the build backend
developers for not complying with the interoperability specification
properly").

Once we make that core assumption about where the responsibility for
the end user experience resides, then the absolutely *minimum*
behavioural requirements that can be placed on build backends are:

- respect the locale encoding
- emit informational messages on stdout
- emit error messages on stderr

What we can then also do is to recommend that *front-ends* do the
following when invoking their build backend CLI shims:

1. Implement the C locale -> UTF-8 based locale coercion defined in
PEP 538 when launching the subprocess
2. Implement a similar coercion for Windows, where cp1252 being active
in the parent process prompts a call to "'chcp cp65001'" inside the
subprocess before the build backend itself actually starts running

That leaves build backend authors with the freedom to assume that they
*don't* need to worry about stream encoding issues, since giving them
access to properly configured streams is the front end's
responsibility.

> Perhaps the middle ground is to specify encoding='utf-8', errors='anything
> but strict' for front-ends, and well-behaved backends should do the work to
> transcode when it is known to be necessary for the tools they run. (i.e.
> frontends do not crash, backends have a simple rule for avoiding loss of
> data).

In PEP 517's architecture, the front-end developers are also
responsible for the CLI that's running inside the backend subprocess.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From p.f.moore at gmail.com  Tue May 23 04:08:28 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 23 May 2017 09:08:28 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
Message-ID: <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>

On 23 May 2017 at 05:11, Nick Coghlan <ncoghlan at gmail.com> wrote:
> What we can then also do is to recommend that *front-ends* do the
> following when invoking their build backend CLI shims:
>
> 1. Implement the C locale -> UTF-8 based locale coercion defined in
> PEP 538 when launching the subprocess
> 2. Implement a similar coercion for Windows, where cp1252 being active
> in the parent process prompts a call to "'chcp cp65001'" inside the
> subprocess before the build backend itself actually starts running

I'm a fairly strong -1 on doing "chcp 65001" on Windows. It puts the
backend into a position of running under a relatively non-standard
environment, and therefore runs the risk of provoking issues. If a
build tool has issues as a result of the changed codepage, who's
responsible for dealing with the bug? The backend, that manages the
tool, or the frontend, that set the codepage? One of the big issues we
have is that very few people have expertise in this area (encodings on
Windows) and so keeping the environment as "standard" as we can
ensures that we can make best use of the limited expertise we have.

I agree with Thomas that we're probably reaching a point of
diminishing returns. But on one point I remain in dispute - I strongly
prefer using the locale encoding as the assumed encoding for the
output stream rather than UTF-8. Also (although this is a quality of
implementation issue) I think that the frontend (i.e.the shim) should
*not* make any changes to the global environment that the backend runs
in.

Paul

From thomas at kluyver.me.uk  Tue May 23 04:56:53 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Tue, 23 May 2017 09:56:53 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
 <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>
Message-ID: <1495529813.1777650.985588336.3D9322A0@webmail.messagingengine.com>

On Tue, May 23, 2017, at 09:08 AM, Paul Moore wrote:
> I strongly
> prefer using the locale encoding as the assumed encoding for the
> output stream rather than UTF-8.

I may have missed it, but has anyone proposed what it should do if it
wants to send characters which can't be encoded in the locale encoding?
Paths on Windows are handled natively as UTF-16, as I understand it, so
it's entirely possible for them to contain characters which can't be
represented in, say, CP1252.

Given this, and the workarounds Nick has pointed out are necessary for
systems where the locale thinks it's ASCII, I still think that
specifying "UTF-8" is a better option than trying to work with locale
encodings. We're building a new spec for new tools in 2017, let's not
prolong the pain of platform-dependent default encodings further.

Thomas

From p.f.moore at gmail.com  Tue May 23 06:04:27 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 23 May 2017 11:04:27 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <1495529813.1777650.985588336.3D9322A0@webmail.messagingengine.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
 <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>
 <1495529813.1777650.985588336.3D9322A0@webmail.messagingengine.com>
Message-ID: <CACac1F_jZ6LqkSZLdA-c8h922H7AgMpR19ooT3SCB=sqk5G+7w@mail.gmail.com>

On 23 May 2017 at 09:56, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> I may have missed it, but has anyone proposed what it should do if it
> wants to send characters which can't be encoded in the locale encoding?

No, it's not been mentioned - the focus has been on running build
tools like a compiler. Best answer I can give is to use a
(backslash)replace error handler. I agree this is suboptimal, but see
below.

> Paths on Windows are handled natively as UTF-16, as I understand it, so
> it's entirely possible for them to contain characters which can't be
> represented in, say, CP1252.

Agreed. In practice, the vast bulk of the issues reported for pip seem
to be to do with filename characters or localised messages using the
ANSI/OEM codepages, though. But I agree that in theory this is an
issue.

> Given this, and the workarounds Nick has pointed out are necessary for
> systems where the locale thinks it's ASCII, I still think that
> specifying "UTF-8" is a better option than trying to work with locale
> encodings. We're building a new spec for new tools in 2017, let's not
> prolong the pain of platform-dependent default encodings further.

However, if we do this then we have a situation where existing build
tools (compilers, etc) that we have to support still use platform
dependent encodings. That's a reality that we can't wish away. And the
majority of real-life issues reported on pip are with compilation
errors. So do we require backends that run these tools to ensure that
they transcode the output, or do we risk significant output
corruption, because (essentially) every high-bit character in the
compiler output will be replaced as it's invalid UTF-8?

I agree 100% that UTF-8 is in theory the right thing. My focus is on
the practical aspects of minimising the risks of repeating the sorts
of actual issues that we have seen in the past on pip, though, and
"don't require backends that run compilers to transcode the output"
seems to me to be the most likely route to achieve that.

Having said that, I won't be the one writing those backends - if
people like Steve are OK with transcoding (or dealing with pip issues
saying "I can't read the compiler output" being passed back to them as
backend issues) then I'm not going to argue against UTF-8.

Paul

From thomas at kluyver.me.uk  Tue May 23 07:36:22 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Tue, 23 May 2017 12:36:22 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CACac1F_jZ6LqkSZLdA-c8h922H7AgMpR19ooT3SCB=sqk5G+7w@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
 <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>
 <1495529813.1777650.985588336.3D9322A0@webmail.messagingengine.com>
 <CACac1F_jZ6LqkSZLdA-c8h922H7AgMpR19ooT3SCB=sqk5G+7w@mail.gmail.com>
Message-ID: <1495539382.1810627.985733704.7CD7CB52@webmail.messagingengine.com>

On Tue, May 23, 2017, at 11:04 AM, Paul Moore wrote:
> However, if we do this then we have a situation where existing build
> tools (compilers, etc) that we have to support still use platform
> dependent encodings. That's a reality that we can't wish away. And the
> majority of real-life issues reported on pip are with compilation
> errors. So do we require backends that run these tools to ensure that
> they transcode the output, or do we risk significant output
> corruption, because (essentially) every high-bit character in the
> compiler output will be replaced as it's invalid UTF-8?

As you described earlier, though, even using a locale dependent encoding
doesn't really avoid this problem, because of tools using OEM vs ANSI
codepages on Windows. And if PYTHONIOENCODING is set, Python processes
will use that over the locale encoding. I think we're ultimately better
off specifying a consistent encoding rather than trying to guess about
it.

I'm also thinking of all the bugs I've seen (and written) by assuming
open() in text mode defaults to UTF-8 encoding - as it does on the Linux
and Mac computers many open source developers use, but not on Windows,
nor in all Linux configurations.

So I'd recommend that backends running processes for which they know the
encoding should transcode it to UTF-8. I expect we can make standard
utility functions to wait for a subprocess to finish while reading,
transcoding, and repeating its output.

I'm still not sure what the backend should do when it runs something for
which it doesn't know the output encoding. The possibilities are either:

- Take a best guess and transcode it to UTF-8, which may risk losing
some information, but keeps the output as valid UTF-8
- Pass through the raw bytes, ensuring that no information is lost, but
leaving it up to the frontend/user to deal with that.

Thomas

From p.f.moore at gmail.com  Tue May 23 07:56:40 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 23 May 2017 12:56:40 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <1495539382.1810627.985733704.7CD7CB52@webmail.messagingengine.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
 <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>
 <1495529813.1777650.985588336.3D9322A0@webmail.messagingengine.com>
 <CACac1F_jZ6LqkSZLdA-c8h922H7AgMpR19ooT3SCB=sqk5G+7w@mail.gmail.com>
 <1495539382.1810627.985733704.7CD7CB52@webmail.messagingengine.com>
Message-ID: <CACac1F9OedvrYtEUAKDb55d6pYN2XCV68nQtC_kwKFXcpq7-1w@mail.gmail.com>

On 23 May 2017 at 12:36, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> As you described earlier, though, even using a locale dependent encoding
> doesn't really avoid this problem, because of tools using OEM vs ANSI
> codepages on Windows. And if PYTHONIOENCODING is set, Python processes
> will use that over the locale encoding. I think we're ultimately better
> off specifying a consistent encoding rather than trying to guess about
> it.

Agreed it doesn't avoid the problem. But it does minimise it. I don't
see any huge advantage in having a consistent encoding across
platforms though - having a consistent *rule*, yes, but "use the
locale encoding" is such a rule as well.

> I'm also thinking of all the bugs I've seen (and written) by assuming
> open() in text mode defaults to UTF-8 encoding - as it does on the Linux
> and Mac computers many open source developers use, but not on Windows,
> nor in all Linux configurations.

So based on your proposal, won't you introduce similar bugs by using
print() without sorting out encodings? Unless (see below) you assume
that the frontend sorts it out for you.

> So I'd recommend that backends running processes for which they know the
> encoding should transcode it to UTF-8. I expect we can make standard
> utility functions to wait for a subprocess to finish while reading,
> transcoding, and repeating its output.

Yes, subprocesses that produce a known encoding are trivial to deal
with. But remembering that you *need* to deal with them less so. My
concern here is the same one as you quote above - assuming that
subprocess returns UTF-8 encoded bytes, because it does on Linux and
Mac.

> I'm still not sure what the backend should do when it runs something for
> which it doesn't know the output encoding. The possibilities are either:
>
> - Take a best guess and transcode it to UTF-8, which may risk losing
> some information, but keeps the output as valid UTF-8
> - Pass through the raw bytes, ensuring that no information is lost, but
> leaving it up to the frontend/user to deal with that.

There's never a good answer here. The "correct" answer is to do
research and establish what encoding the tool uses, but that's often
stupidly difficult.

But if you genuinely don't know (or worse, know there is no consistent
encoding) I'm not sure I see how passing unknown bytes onto the
frontend, which by necessity has less context to guess what those
bytes might mean, is the right answer. The frontend is better able to
know what it wants to *do* with those bytes, but "convert them to text
for the user to see" is the only real answer here IMO (sure, dumping
the raw bytes to a file might be an option, but I imagine it'll be a
relatively uncommon choice).

At the end of the day, there is no perfect answer here. Someone is
going to have to make a judgement call, and as the PEP author, I guess
that's you. So at this point I'll stop badgering you and leave it up
to you to decide what the consensus is. Thanks for listening to my
points, though.

Paul

From thomas at kluyver.me.uk  Tue May 23 08:41:49 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Tue, 23 May 2017 13:41:49 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CACac1F9OedvrYtEUAKDb55d6pYN2XCV68nQtC_kwKFXcpq7-1w@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
 <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>
 <1495529813.1777650.985588336.3D9322A0@webmail.messagingengine.com>
 <CACac1F_jZ6LqkSZLdA-c8h922H7AgMpR19ooT3SCB=sqk5G+7w@mail.gmail.com>
 <1495539382.1810627.985733704.7CD7CB52@webmail.messagingengine.com>
 <CACac1F9OedvrYtEUAKDb55d6pYN2XCV68nQtC_kwKFXcpq7-1w@mail.gmail.com>
Message-ID: <1495543309.1825097.985759744.0BA2FA81@webmail.messagingengine.com>

On Tue, May 23, 2017, at 12:56 PM, Paul Moore wrote:
> So based on your proposal, won't you introduce similar bugs by using
> print() without sorting out encodings? Unless (see below) you assume
> that the frontend sorts it out for you.

If you strictly follow the locale encoding, you need to sort it out in
Python anyway, in case the stdout encoding has been overridden by
PYTHONIOENCODING, or PYTHONSTARTUP, or the infernal .pth files. I accept
that those are corner cases, though.

> Yes, subprocesses that produce a known encoding are trivial to deal
> with. But remembering that you *need* to deal with them less so. My
> concern here is the same one as you quote above - assuming that
> subprocess returns UTF-8 encoded bytes, because it does on Linux and
> Mac.

I agree, that is a concern.

> But if you genuinely don't know (or worse, know there is no consistent
> encoding) I'm not sure I see how passing unknown bytes onto the
> frontend, which by necessity has less context to guess what those
> bytes might mean, is the right answer. The frontend is better able to
> know what it wants to *do* with those bytes, but "convert them to text
> for the user to see" is the only real answer here IMO (sure, dumping
> the raw bytes to a file might be an option, but I imagine it'll be a
> relatively uncommon choice).

I was indeed thinking of dumping them to a file. It's not very user
friendly, but it means the information is there if you need it. I
suspect that regardless of the locale, technical information like code
and filesystem paths will often contain enough ASCII that a human can
interpret them even if non-ASCII characters are wrongly encoded. So I
hope that needing to reverse-engineer the encoding will be relatively
rare.

The appeal of this is that it follows "in the face of ambiguity, refuse
the temptation to guess". If the backend guesses the encoding
incorrectly, the frontend gets valid UTF-8, but is no better able to
display it meaningfully, and you then need to go through
decode-encode-decode to recover the original text, even if no data was
lost.

Another option: if the backend runs a subprocess with unknown output
encoding, it redirects that output to a temp file and prints the path in
its own output. Then there's a better chance that the unknown encoding
is at least consistent within the file, so tools can do encoding
detection on it.

> At the end of the day, there is no perfect answer here. Someone is
> going to have to make a judgement call, and as the PEP author, I guess
> that's you. So at this point I'll stop badgering you and leave it up
> to you to decide what the consensus is. Thanks for listening to my
> points, though.

I know what I think, but I don't feel like there's a consensus as yet.

Can I take a quick poll of what people following this topic think?

Q1: Default encoding for captured build stdout/stderr
a. UTF-8 (consistent, can represent any character)
b. Locale default (convenient if backend runs subprocesses which produce
output in the locale encoding)

Q2: Handling unknown encodings from subprocesses
a. Backend should ensure all output is valid in the target encoding
(Q1), though it may not be accurate.
b. Unknown output may be passed on as bytes without transcoding, so the
frontend can e.g. dump it to a file.

I'm currently 1:a, 2:?a .

Thomas

From p.f.moore at gmail.com  Tue May 23 10:16:14 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 23 May 2017 15:16:14 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <1495543309.1825097.985759744.0BA2FA81@webmail.messagingengine.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
 <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>
 <1495529813.1777650.985588336.3D9322A0@webmail.messagingengine.com>
 <CACac1F_jZ6LqkSZLdA-c8h922H7AgMpR19ooT3SCB=sqk5G+7w@mail.gmail.com>
 <1495539382.1810627.985733704.7CD7CB52@webmail.messagingengine.com>
 <CACac1F9OedvrYtEUAKDb55d6pYN2XCV68nQtC_kwKFXcpq7-1w@mail.gmail.com>
 <1495543309.1825097.985759744.0BA2FA81@webmail.messagingengine.com>
Message-ID: <CACac1F_SRgovO6ktF6jkHp+MsRqzA9h5v82ekxmVZmyD5vqAkQ@mail.gmail.com>

On 23 May 2017 at 13:41, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> Can I take a quick poll of what people following this topic think?
>
> Q1: Default encoding for captured build stdout/stderr
> a. UTF-8 (consistent, can represent any character)
> b. Locale default (convenient if backend runs subprocesses which produce
> output in the locale encoding)
>
> Q2: Handling unknown encodings from subprocesses
> a. Backend should ensure all output is valid in the target encoding
> (Q1), though it may not be accurate.
> b. Unknown output may be passed on as bytes without transcoding, so the
> frontend can e.g. dump it to a file.
>
> I'm currently 1:a, 2:?a .

You probably know this, but I'm 1: b, 2: mild preference for a, but
not too bothered. If the answer to 1 is a, though, I strongly prefer
2: a.

Paul

From ncoghlan at gmail.com  Tue May 23 11:20:47 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 24 May 2017 01:20:47 +1000
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <1495543309.1825097.985759744.0BA2FA81@webmail.messagingengine.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
 <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>
 <1495529813.1777650.985588336.3D9322A0@webmail.messagingengine.com>
 <CACac1F_jZ6LqkSZLdA-c8h922H7AgMpR19ooT3SCB=sqk5G+7w@mail.gmail.com>
 <1495539382.1810627.985733704.7CD7CB52@webmail.messagingengine.com>
 <CACac1F9OedvrYtEUAKDb55d6pYN2XCV68nQtC_kwKFXcpq7-1w@mail.gmail.com>
 <1495543309.1825097.985759744.0BA2FA81@webmail.messagingengine.com>
Message-ID: <CADiSq7e_NcQq5DAvD3WBvVp5BOv5vV2rsaMtK9ZxkEU4pttxzg@mail.gmail.com>

On 23 May 2017 at 22:41, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> On Tue, May 23, 2017, at 12:56 PM, Paul Moore wrote:
> Can I take a quick poll of what people following this topic think?
>
> Q1: Default encoding for captured build stdout/stderr
> a. UTF-8 (consistent, can represent any character)
> b. Locale default (convenient if backend runs subprocesses which produce
> output in the locale encoding)
>
> Q2: Handling unknown encodings from subprocesses
> a. Backend should ensure all output is valid in the target encoding
> (Q1), though it may not be accurate.
> b. Unknown output may be passed on as bytes without transcoding, so the
> frontend can e.g. dump it to a file.

Up to this point, I've been in favour of both 1b and 2b, since they're
the main options that allow a build backend to get itself out of the
way entirely and let the front-end deal with the problem rather than
having to figure out encoding issues for themselves. pip's already has
to deal with the "arbitrarily encoded data" problem for the current
setup.py invocation, and whatever solution is adopted there should
suffice for PEP 517 as well.

If PEP 426 taught me anything, it was that if you weren't planning to
write something yourself, and didn't have the budget to pay someone
else to write it for you, your best bet is to adhere as closely to the
status quo as you can while still incorporating the 100% essential
changes that you actually need. (A Zen of Python style aphorism for
that: "The right way and the easy way should be the same way")

To be honest, I still think that's likely to be the right way to go
for PEP 517, and will take some convincing that we're going to be able
to persuade future backend developers that personally couldn't care
less about encoding issues to adopt anything more complex.

However, I also realised that there's a potential third way to handle
this problem: design a Python level API that allows front ends to use
more structured data formats (e.g. JSON) for communication between the
frontend and their backend shim.

In particular, I'm thinking we could move the current
"config_settings" dict onto something like a "build context" object
that, *even in Python 2*, offers a Unicode "outstream" and
"errstream", which the backend is then expected to use rather than
writing to sys.stdout/err directly. That context could also provide a
Python 3 style "run()" API for subprocess invocation that implemented
the preferred stream handling behaviour for subprocess invocation
(including applying the "backslashreplace" error handler regardless of
version)

That way, instead of trying to hit build backend developers with a
fairly flimsy stick ("Thou shalt comply with the specification or some
other open source developers may say mildly disapproving things about
you on the internet"), we'd instead be offering them the easy way out
of letting the front-end provided build context deal with all the
messy encoding issues.

Taking that approach of just defining a helper API and expecting build
backends to either use it or emulate it gives us some quite attractive
properties:

- backends themselves deal entirely in Unicode, not bytes
- frontends get full control of the communication format used between
the frontend and its backend shim - they're not restricted to plain
text
- the Python 2/3 differences can be handled in the frontend CLI shims,
rather than every backend needing to do it
- we don't need to enshrine any particular encoding handling behaviour
in the spec, we can let it be a quality of implementation issue for
the front-end tools
- platform specific tools can make platform specific choices
- tools can adapt to new platforms without requiring a specification update
- tools can update their default behaviour as other considerations
change (e.g. the possible introduction of locale coercion and
PYTHONUTF8 mode in 3.7)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From p.f.moore at gmail.com  Tue May 23 11:39:36 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 23 May 2017 16:39:36 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CADiSq7e_NcQq5DAvD3WBvVp5BOv5vV2rsaMtK9ZxkEU4pttxzg@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
 <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>
 <1495529813.1777650.985588336.3D9322A0@webmail.messagingengine.com>
 <CACac1F_jZ6LqkSZLdA-c8h922H7AgMpR19ooT3SCB=sqk5G+7w@mail.gmail.com>
 <1495539382.1810627.985733704.7CD7CB52@webmail.messagingengine.com>
 <CACac1F9OedvrYtEUAKDb55d6pYN2XCV68nQtC_kwKFXcpq7-1w@mail.gmail.com>
 <1495543309.1825097.985759744.0BA2FA81@webmail.messagingengine.com>
 <CADiSq7e_NcQq5DAvD3WBvVp5BOv5vV2rsaMtK9ZxkEU4pttxzg@mail.gmail.com>
Message-ID: <CACac1F-aK-q_hO57SLK=fKgDuB4z5ca4pymYkQnZNHs=AywwEA@mail.gmail.com>

On 23 May 2017 at 16:20, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Taking that approach of just defining a helper API and expecting build
> backends to either use it or emulate it gives us some quite attractive
> properties:

Making the output data part of a structured API (and by implication,
saying that backends shouldn't be writing to stdout directly at all)
would definitely improve the situation, IMO. Frankly, it seems likely
that the only real way we're going to get backend developers to
consider encodings is by having the "build output" as a string value
passed back via the API, rather than implied in the fact that backends
can write to stdout/err. It also squarely places the responsibility
for dealing with the question of displaying full-range Unicode output
to the user onto the frontend.

However, it's a relatively big change to the PEP and there's a risk
that by endlessly reaching for perfection, we miss the chance to get
the PEP in at all (another lesson we should probably learn from PEP
426!)

Paul

From ncoghlan at gmail.com  Tue May 23 12:16:42 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 24 May 2017 02:16:42 +1000
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CACac1F-aK-q_hO57SLK=fKgDuB4z5ca4pymYkQnZNHs=AywwEA@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
 <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>
 <1495529813.1777650.985588336.3D9322A0@webmail.messagingengine.com>
 <CACac1F_jZ6LqkSZLdA-c8h922H7AgMpR19ooT3SCB=sqk5G+7w@mail.gmail.com>
 <1495539382.1810627.985733704.7CD7CB52@webmail.messagingengine.com>
 <CACac1F9OedvrYtEUAKDb55d6pYN2XCV68nQtC_kwKFXcpq7-1w@mail.gmail.com>
 <1495543309.1825097.985759744.0BA2FA81@webmail.messagingengine.com>
 <CADiSq7e_NcQq5DAvD3WBvVp5BOv5vV2rsaMtK9ZxkEU4pttxzg@mail.gmail.com>
 <CACac1F-aK-q_hO57SLK=fKgDuB4z5ca4pymYkQnZNHs=AywwEA@mail.gmail.com>
Message-ID: <CADiSq7dPYtRJudm1Q63=t-06_ZGwvmMX1hL3L3fh9=9LVvucEg@mail.gmail.com>

On 24 May 2017 at 01:39, Paul Moore <p.f.moore at gmail.com> wrote:
> On 23 May 2017 at 16:20, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Taking that approach of just defining a helper API and expecting build
>> backends to either use it or emulate it gives us some quite attractive
>> properties:
>
> Making the output data part of a structured API (and by implication,
> saying that backends shouldn't be writing to stdout directly at all)
> would definitely improve the situation, IMO. Frankly, it seems likely
> that the only real way we're going to get backend developers to
> consider encodings is by having the "build output" as a string value
> passed back via the API, rather than implied in the fact that backends
> can write to stdout/err. It also squarely places the responsibility
> for dealing with the question of displaying full-range Unicode output
> to the user onto the frontend.
>
> However, it's a relatively big change to the PEP and there's a risk
> that by endlessly reaching for perfection, we miss the chance to get
> the PEP in at all (another lesson we should probably learn from PEP
> 426!)

Yep, and that's also why I want to avoid trying to use it to improve
the encoding handling situation - pip and other tools have to deal
with the current mess regardless, and there's already likely to be
some significant churn in this space as a result of the changes Victor
and I have proposed for Python 3.7.

As a result, I think adding in additional requirements here runs a
significant risk of requiring build backend developers to do
additional work to achieve nominal spec compliance without actually
simplifying anything in practice for frontend developers.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From p.f.moore at gmail.com  Tue May 23 12:32:20 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 23 May 2017 17:32:20 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CADiSq7dPYtRJudm1Q63=t-06_ZGwvmMX1hL3L3fh9=9LVvucEg@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
 <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>
 <1495529813.1777650.985588336.3D9322A0@webmail.messagingengine.com>
 <CACac1F_jZ6LqkSZLdA-c8h922H7AgMpR19ooT3SCB=sqk5G+7w@mail.gmail.com>
 <1495539382.1810627.985733704.7CD7CB52@webmail.messagingengine.com>
 <CACac1F9OedvrYtEUAKDb55d6pYN2XCV68nQtC_kwKFXcpq7-1w@mail.gmail.com>
 <1495543309.1825097.985759744.0BA2FA81@webmail.messagingengine.com>
 <CADiSq7e_NcQq5DAvD3WBvVp5BOv5vV2rsaMtK9ZxkEU4pttxzg@mail.gmail.com>
 <CACac1F-aK-q_hO57SLK=fKgDuB4z5ca4pymYkQnZNHs=AywwEA@mail.gmail.com>
 <CADiSq7dPYtRJudm1Q63=t-06_ZGwvmMX1hL3L3fh9=9LVvucEg@mail.gmail.com>
Message-ID: <CACac1F-y4VNDpwSPwCFysgdsqF1QBxaz11m6bgTAwAXmE-svmQ@mail.gmail.com>

On 23 May 2017 at 17:16, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Yep, and that's also why I want to avoid trying to use it to improve
> the encoding handling situation - pip and other tools have to deal
> with the current mess regardless, and there's already likely to be
> some significant churn in this space as a result of the changes Victor
> and I have proposed for Python 3.7.

Encoding issues have been around in pip for many years, with little or
no progress. We might be getting a handle on things now (Thomas'
initial email in this thread was very timely - the fact that I was in
the middle of working on the encoding issue in pip was the only reason
I picked up on the need for clarity in the PEP) but I'd be very
cautious about saying we've got it solved until we have the latest
changes in a released version of pip and we get some feedback (or
silence, more likely) from international users.

One of the reasons I made the point about ease of testing earlier in
the thread is that we've found it's extremely difficult to pin down
the root of the reported problems in pip - the route that badly
encoded data takes from build tool to pip's output is pretty
convoluted. Anything that adds clear-cut boundaries at which we can
make guarantees about the integrity of the data will help a lot with
this in the future.

Paul

From thomas at kluyver.me.uk  Tue May 23 13:04:53 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Tue, 23 May 2017 18:04:53 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CADiSq7e_NcQq5DAvD3WBvVp5BOv5vV2rsaMtK9ZxkEU4pttxzg@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
 <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>
 <1495529813.1777650.985588336.3D9322A0@webmail.messagingengine.com>
 <CACac1F_jZ6LqkSZLdA-c8h922H7AgMpR19ooT3SCB=sqk5G+7w@mail.gmail.com>
 <1495539382.1810627.985733704.7CD7CB52@webmail.messagingengine.com>
 <CACac1F9OedvrYtEUAKDb55d6pYN2XCV68nQtC_kwKFXcpq7-1w@mail.gmail.com>
 <1495543309.1825097.985759744.0BA2FA81@webmail.messagingengine.com>
 <CADiSq7e_NcQq5DAvD3WBvVp5BOv5vV2rsaMtK9ZxkEU4pttxzg@mail.gmail.com>
Message-ID: <1495559093.2774864.986098160.74BDFA33@webmail.messagingengine.com>

On Tue, May 23, 2017, at 04:20 PM, Nick Coghlan wrote:
> Up to this point, I've been in favour of both 1b and 2b, since they're

Noted.

> However, I also realised that there's a potential third way to handle
> this problem: design a Python level API that allows front ends to use
> more structured data formats (e.g. JSON) for communication between the
> frontend and their backend shim.
> 
> In particular, I'm thinking we could move the current
> "config_settings" dict onto something like a "build context" object
> that, *even in Python 2*, offers a Unicode "outstream" and
> "errstream", which the backend is then expected to use rather than
> writing to sys.stdout/err directly. That context could also provide a
> Python 3 style "run()" API for subprocess invocation that implemented
> the preferred stream handling behaviour for subprocess invocation
> (including applying the "backslashreplace" error handler regardless of
> version)

I'm not really compelled by this so far:

- It's more complexity for build tools - instead of just producing
output as usual, now they have to pass around a context object and
direct output to it.
- What does the frontend do if there is output on stdout/stderr anyway?
Throw it away? Let it go straight to the terminal? Reprimand the backend
for not using the streams in the build context? Or try to include it as
part of build output anyway?
- I don't see how it solves the issue with subprocesses producing
unknown encodings. The output bytes still need to be interpreted
somehow.

I'll propose a variant of an idea I described already: the frontend
could provide the backend with a fresh temp directory. If the backend
needs to run other processes, it can redirect the output into a file in
that temp directory. Then you have files with an unknown encoding,  but
each file will hopefully have one encoding, and you can use a tool like
chardet to guess what it is.

Thomas


From chris.jerdonek at gmail.com  Tue May 23 20:22:47 2017
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Tue, 23 May 2017 17:22:47 -0700
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <1495559093.2774864.986098160.74BDFA33@webmail.messagingengine.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
 <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>
 <1495529813.1777650.985588336.3D9322A0@webmail.messagingengine.com>
 <CACac1F_jZ6LqkSZLdA-c8h922H7AgMpR19ooT3SCB=sqk5G+7w@mail.gmail.com>
 <1495539382.1810627.985733704.7CD7CB52@webmail.messagingengine.com>
 <CACac1F9OedvrYtEUAKDb55d6pYN2XCV68nQtC_kwKFXcpq7-1w@mail.gmail.com>
 <1495543309.1825097.985759744.0BA2FA81@webmail.messagingengine.com>
 <CADiSq7e_NcQq5DAvD3WBvVp5BOv5vV2rsaMtK9ZxkEU4pttxzg@mail.gmail.com>
 <1495559093.2774864.986098160.74BDFA33@webmail.messagingengine.com>
Message-ID: <CAOTb1wcpjyKy5b-O9squhNdk8Rn_aLCs=24056xb2zLwVr-cCw@mail.gmail.com>

A couple comments:

1) Would it make sense to provide a way for build tools to specify
what encoding they use (e.g. if not using the default), instead of
changing their encoding to conform to a standard? It seems like that
could be easier, although I know this doesn't address problems like
non-conforming tools.

2) In terms of debugging, in cases where there are encoding-related
errors, it would help if the overall system made it easy to pinpoint
which parts of the system are at fault (using good error handling,
diagnostic messages, etc).

--Chris

On Tue, May 23, 2017 at 10:04 AM, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> On Tue, May 23, 2017, at 04:20 PM, Nick Coghlan wrote:
>> Up to this point, I've been in favour of both 1b and 2b, since they're
>
> Noted.
>
>> However, I also realised that there's a potential third way to handle
>> this problem: design a Python level API that allows front ends to use
>> more structured data formats (e.g. JSON) for communication between the
>> frontend and their backend shim.
>>
>> In particular, I'm thinking we could move the current
>> "config_settings" dict onto something like a "build context" object
>> that, *even in Python 2*, offers a Unicode "outstream" and
>> "errstream", which the backend is then expected to use rather than
>> writing to sys.stdout/err directly. That context could also provide a
>> Python 3 style "run()" API for subprocess invocation that implemented
>> the preferred stream handling behaviour for subprocess invocation
>> (including applying the "backslashreplace" error handler regardless of
>> version)
>
> I'm not really compelled by this so far:
>

From ncoghlan at gmail.com  Wed May 24 03:11:15 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 24 May 2017 17:11:15 +1000
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <1495559093.2774864.986098160.74BDFA33@webmail.messagingengine.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
 <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>
 <1495529813.1777650.985588336.3D9322A0@webmail.messagingengine.com>
 <CACac1F_jZ6LqkSZLdA-c8h922H7AgMpR19ooT3SCB=sqk5G+7w@mail.gmail.com>
 <1495539382.1810627.985733704.7CD7CB52@webmail.messagingengine.com>
 <CACac1F9OedvrYtEUAKDb55d6pYN2XCV68nQtC_kwKFXcpq7-1w@mail.gmail.com>
 <1495543309.1825097.985759744.0BA2FA81@webmail.messagingengine.com>
 <CADiSq7e_NcQq5DAvD3WBvVp5BOv5vV2rsaMtK9ZxkEU4pttxzg@mail.gmail.com>
 <1495559093.2774864.986098160.74BDFA33@webmail.messagingengine.com>
Message-ID: <CADiSq7ed8ewOp=yoavRfPMma+C0z=bo+iQxhQ=ziFfjXv0yseQ@mail.gmail.com>

On 24 May 2017 at 03:04, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> I'll propose a variant of an idea I described already: the frontend
> could provide the backend with a fresh temp directory. If the backend
> needs to run other processes, it can redirect the output into a file in
> that temp directory. Then you have files with an unknown encoding,  but
> each file will hopefully have one encoding, and you can use a tool like
> chardet to guess what it is.

That's actually pretty similar to the way tools like mock (the chroot
based RPM builder) work. That way, build backends could choose
between:

- use pipes to stream output from the tools they call, deal with
encoding issues themselves
- redirect output to a suitable named file in the tool log directory

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From thomas at kluyver.me.uk  Wed May 24 06:29:47 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Wed, 24 May 2017 11:29:47 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CAOTb1wcpjyKy5b-O9squhNdk8Rn_aLCs=24056xb2zLwVr-cCw@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
 <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>
 <1495529813.1777650.985588336.3D9322A0@webmail.messagingengine.com>
 <CACac1F_jZ6LqkSZLdA-c8h922H7AgMpR19ooT3SCB=sqk5G+7w@mail.gmail.com>
 <1495539382.1810627.985733704.7CD7CB52@webmail.messagingengine.com>
 <CACac1F9OedvrYtEUAKDb55d6pYN2XCV68nQtC_kwKFXcpq7-1w@mail.gmail.com>
 <1495543309.1825097.985759744.0BA2FA81@webmail.messagingengine.com>
 <CADiSq7e_NcQq5DAvD3WBvVp5BOv5vV2rsaMtK9ZxkEU4pttxzg@mail.gmail.com>
 <1495559093.2774864.986098160.74BDFA33@webmail.messagingengine.com>
 <CAOTb1wcpjyKy5b-O9squhNdk8Rn_aLCs=24056xb2zLwVr-cCw@mail.gmail.com>
Message-ID: <1495621787.530876.986950088.6E287FBD@webmail.messagingengine.com>

On Wed, May 24, 2017, at 01:22 AM, Chris Jerdonek wrote:
> 1) Would it make sense to provide a way for build tools to specify
> what encoding they use (e.g. if not using the default), instead of
> changing their encoding to conform to a standard? It seems like that
> could be easier, although I know this doesn't address problems like
> non-conforming tools.

Interesting idea, but I'm not convinced it actually makes anything
easier. You still have the same issues if the backend runs a subprocess
which doesn't produce output in the expected encoding. And there would
be some small amount of added complexity to communicate the encoding to
the frontend.

> 2) In terms of debugging, in cases where there are encoding-related
> errors, it would help if the overall system made it easy to pinpoint
> which parts of the system are at fault (using good error handling,
> diagnostic messages, etc).

Agreed.

Nick:
> That's actually pretty similar to the way tools like mock (the chroot
> based RPM builder) work. That way, build backends could choose
> between:
> 
> - use pipes to stream output from the tools they call, deal with
> encoding issues themselves
> - redirect output to a suitable named file in the tool log directory

Do you know if that system works well for mock? Shall I try to draft a
spec of something like this for PEP 517?

Thomas

From ncoghlan at gmail.com  Thu May 25 08:26:41 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 May 2017 22:26:41 +1000
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <1495621787.530876.986950088.6E287FBD@webmail.messagingengine.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
 <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>
 <1495529813.1777650.985588336.3D9322A0@webmail.messagingengine.com>
 <CACac1F_jZ6LqkSZLdA-c8h922H7AgMpR19ooT3SCB=sqk5G+7w@mail.gmail.com>
 <1495539382.1810627.985733704.7CD7CB52@webmail.messagingengine.com>
 <CACac1F9OedvrYtEUAKDb55d6pYN2XCV68nQtC_kwKFXcpq7-1w@mail.gmail.com>
 <1495543309.1825097.985759744.0BA2FA81@webmail.messagingengine.com>
 <CADiSq7e_NcQq5DAvD3WBvVp5BOv5vV2rsaMtK9ZxkEU4pttxzg@mail.gmail.com>
 <1495559093.2774864.986098160.74BDFA33@webmail.messagingengine.com>
 <CAOTb1wcpjyKy5b-O9squhNdk8Rn_aLCs=24056xb2zLwVr-cCw@mail.gmail.com>
 <1495621787.530876.986950088.6E287FBD@webmail.messagingengine.com>
Message-ID: <CADiSq7dw1FiGE2oMmqwexK4635Vc5YrwUZyZvt6VvkSkGQLXNg@mail.gmail.com>

On 24 May 2017 at 20:29, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> Nick:
>> That's actually pretty similar to the way tools like mock (the chroot
>> based RPM builder) work. That way, build backends could choose
>> between:
>>
>> - use pipes to stream output from the tools they call, deal with
>> encoding issues themselves
>> - redirect output to a suitable named file in the tool log directory
>
> Do you know if that system works well for mock? Shall I try to draft a
> spec of something like this for PEP 517?

I'm genuinely unsure. The main downside of the directory based
approach is that it doesn't play well with CI systems in general -
those are typically set up to capture the standard streams, and if you
want to capture other artifacts, you either have to stream them
anyway, or else you have to use a CI specific upload mechanism to keep
them around.

I guess what we could do is have a "debug log directory" as part of
the defined interface between the frontends and the build backends,
and then the exact UX of dealing with those build logs would then be
something for frontends to define (e.g. offering an option to
automatically stream the logs after a failed build, with appropriate
headers and footers around each file)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From waynejwerner at gmail.com  Thu May 25 09:04:58 2017
From: waynejwerner at gmail.com (Wayne Werner)
Date: Thu, 25 May 2017 13:04:58 +0000
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CADiSq7dw1FiGE2oMmqwexK4635Vc5YrwUZyZvt6VvkSkGQLXNg@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
 <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>
 <1495529813.1777650.985588336.3D9322A0@webmail.messagingengine.com>
 <CACac1F_jZ6LqkSZLdA-c8h922H7AgMpR19ooT3SCB=sqk5G+7w@mail.gmail.com>
 <1495539382.1810627.985733704.7CD7CB52@webmail.messagingengine.com>
 <CACac1F9OedvrYtEUAKDb55d6pYN2XCV68nQtC_kwKFXcpq7-1w@mail.gmail.com>
 <1495543309.1825097.985759744.0BA2FA81@webmail.messagingengine.com>
 <CADiSq7e_NcQq5DAvD3WBvVp5BOv5vV2rsaMtK9ZxkEU4pttxzg@mail.gmail.com>
 <1495559093.2774864.986098160.74BDFA33@webmail.messagingengine.com>
 <CAOTb1wcpjyKy5b-O9squhNdk8Rn_aLCs=24056xb2zLwVr-cCw@mail.gmail.com>
 <1495621787.530876.986950088.6E287FBD@webmail.messagingengine.com>
 <CADiSq7dw1FiGE2oMmqwexK4635Vc5YrwUZyZvt6VvkSkGQLXNg@mail.gmail.com>
Message-ID: <CAPM86NcWCxM8sLyQu4r-KQix_fm5diFeJqjG8XDP_-5EHekYBQ@mail.gmail.com>

FWIW, I was just reading an article about writing libraries to just operate
on streams and totally ignore stdout/stdin/file io, and just leave the IO
to something else.

It may be a good idea to define the spec as purely operating on byte and
text streams, then leave where those streams go as an implementation
detail. That way for CI systems they could dump to stdout/stdin and other
systems  could do something different.

-W

On Thu, May 25, 2017, 7:27 AM Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 24 May 2017 at 20:29, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> > Nick:
> >> That's actually pretty similar to the way tools like mock (the chroot
> >> based RPM builder) work. That way, build backends could choose
> >> between:
> >>
> >> - use pipes to stream output from the tools they call, deal with
> >> encoding issues themselves
> >> - redirect output to a suitable named file in the tool log directory
> >
> > Do you know if that system works well for mock? Shall I try to draft a
> > spec of something like this for PEP 517?
>
> I'm genuinely unsure. The main downside of the directory based
> approach is that it doesn't play well with CI systems in general -
> those are typically set up to capture the standard streams, and if you
> want to capture other artifacts, you either have to stream them
> anyway, or else you have to use a CI specific upload mechanism to keep
> them around.
>
> I guess what we could do is have a "debug log directory" as part of
> the defined interface between the frontends and the build backends,
> and then the exact UX of dealing with those build logs would then be
> something for frontends to define (e.g. offering an option to
> automatically stream the logs after a failed build, with appropriate
> headers and footers around each file)
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170525/a956fdb2/attachment.html>

From p.f.moore at gmail.com  Thu May 25 09:27:28 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 25 May 2017 14:27:28 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CADiSq7dw1FiGE2oMmqwexK4635Vc5YrwUZyZvt6VvkSkGQLXNg@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
 <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>
 <1495529813.1777650.985588336.3D9322A0@webmail.messagingengine.com>
 <CACac1F_jZ6LqkSZLdA-c8h922H7AgMpR19ooT3SCB=sqk5G+7w@mail.gmail.com>
 <1495539382.1810627.985733704.7CD7CB52@webmail.messagingengine.com>
 <CACac1F9OedvrYtEUAKDb55d6pYN2XCV68nQtC_kwKFXcpq7-1w@mail.gmail.com>
 <1495543309.1825097.985759744.0BA2FA81@webmail.messagingengine.com>
 <CADiSq7e_NcQq5DAvD3WBvVp5BOv5vV2rsaMtK9ZxkEU4pttxzg@mail.gmail.com>
 <1495559093.2774864.986098160.74BDFA33@webmail.messagingengine.com>
 <CAOTb1wcpjyKy5b-O9squhNdk8Rn_aLCs=24056xb2zLwVr-cCw@mail.gmail.com>
 <1495621787.530876.986950088.6E287FBD@webmail.messagingengine.com>
 <CADiSq7dw1FiGE2oMmqwexK4635Vc5YrwUZyZvt6VvkSkGQLXNg@mail.gmail.com>
Message-ID: <CACac1F-VY_P5k=muzB=EBrAypTAGS-Rk43ZzT5bCLcbAJ3t-ig@mail.gmail.com>

On 25 May 2017 at 13:26, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 24 May 2017 at 20:29, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
>> Nick:
>>> That's actually pretty similar to the way tools like mock (the chroot
>>> based RPM builder) work. That way, build backends could choose
>>> between:
>>>
>>> - use pipes to stream output from the tools they call, deal with
>>> encoding issues themselves
>>> - redirect output to a suitable named file in the tool log directory
>>
>> Do you know if that system works well for mock? Shall I try to draft a
>> spec of something like this for PEP 517?
>
> I'm genuinely unsure. The main downside of the directory based
> approach is that it doesn't play well with CI systems in general -
> those are typically set up to capture the standard streams, and if you
> want to capture other artifacts, you either have to stream them
> anyway, or else you have to use a CI specific upload mechanism to keep
> them around.
>
> I guess what we could do is have a "debug log directory" as part of
> the defined interface between the frontends and the build backends,
> and then the exact UX of dealing with those build logs would then be
> something for frontends to define (e.g. offering an option to
> automatically stream the logs after a failed build, with appropriate
> headers and footers around each file)

To me, this feels like a lot of potentially unnecessary complexity. At
the moment pip's UI works around "run a build, get some output,
display the output if the situation warrants (i.e., there was an
error)". The only stumbling block is over transferring that output
from backend to frontend where we need to consider text/bytes issues.

We're now talking about potentially managing a directory containing
logs, do we need to persist log files, should we display the file
content or just the filename, etc.

I'd be concerned here that we risk making the frontend UI a lot more
complex for little actual benefit. I'd rather we stick with the
current model, where a backend just has some output to pass through to
the frontend. Let's get a solution that works for that before adding
extra complexity, or we'll never get the PEP signed off.

Paul

From thomas at kluyver.me.uk  Thu May 25 10:04:13 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Thu, 25 May 2017 15:04:13 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CACac1F-VY_P5k=muzB=EBrAypTAGS-Rk43ZzT5bCLcbAJ3t-ig@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
 <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>
 <1495529813.1777650.985588336.3D9322A0@webmail.messagingengine.com>
 <CACac1F_jZ6LqkSZLdA-c8h922H7AgMpR19ooT3SCB=sqk5G+7w@mail.gmail.com>
 <1495539382.1810627.985733704.7CD7CB52@webmail.messagingengine.com>
 <CACac1F9OedvrYtEUAKDb55d6pYN2XCV68nQtC_kwKFXcpq7-1w@mail.gmail.com>
 <1495543309.1825097.985759744.0BA2FA81@webmail.messagingengine.com>
 <CADiSq7e_NcQq5DAvD3WBvVp5BOv5vV2rsaMtK9ZxkEU4pttxzg@mail.gmail.com>
 <1495559093.2774864.986098160.74BDFA33@webmail.messagingengine.com>
 <CAOTb1wcpjyKy5b-O9squhNdk8Rn_aLCs=24056xb2zLwVr-cCw@mail.gmail.com>
 <1495621787.530876.986950088.6E287FBD@webmail.messagingengine.com>
 <CADiSq7dw1FiGE2oMmqwexK4635Vc5YrwUZyZvt6VvkSkGQLXNg@mail.gmail.com>
 <CACac1F-VY_P5k=muzB=EBrAypTAGS-Rk43ZzT5bCLcbAJ3t-ig@mail.gmail.com>
Message-ID: <1495721053.3484266.988324184.59F35AE9@webmail.messagingengine.com>

On Thu, May 25, 2017, at 02:27 PM, Paul Moore wrote:
> I'd be concerned here that we risk making the frontend UI a lot more
> complex for little actual benefit. I'd rather we stick with the
> current model, where a backend just has some output to pass through to
> the frontend. Let's get a solution that works for that before adding
> extra complexity, or we'll never get the PEP signed off.

I'm inclined to agree that we're overcomplicating things. But if we
can't agree on which simple-but-imperfect option to take, maybe it's
worth trying to work out something more complex.

My proposed addition to the PEP so far says this:

The build frontend may capture stdout and/or stderr from the backend. If
the backend detects that an output stream is not a terminal/console
(e.g. ``not sys.stdout.isatty()``), it SHOULD ensure that any output it
writes to that stream is UTF-8 encoded. The build frontend MUST NOT fail
if captured output is not valid UTF-8, but it MAY not preserve all the
information in that case (e.g. it may decode using the *replace* error
handler in Python). If the output stream is a terminal, the build
backend is responsible for presenting its output accurately, as for any
program running in a terminal.

We could add a paragraph like this:

The backend may do some operations, such as running subprocesses, which
produce output in an unknown encoding. To handle such output, the build
frontend MAY (?) create an empty directory, and set the environment
variable PEP517_BUILD_LOGS to the path of this directory for the
backend. If this environment variable is set, the backend MAY create any
number of files inside this directory containing additional output. This
is designed to allow the use of encoding detection tools on this output.
If files are created in this directory, frontends SHOULD display its
location in their output, and MAY display the contents of the files.

That's not a massive amount more complexity for the spec, but it does
add a moderate burden to frontend & backend implementations which want
to properly support it.

If you're being purist about it, displaying a path on a Unix based
system is producing output in an unknown encoding, since filenames in
Unix are bytes. I don't imagine many tools are going to go that far,
though.

Thomas

From ncoghlan at gmail.com  Thu May 25 10:38:20 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 26 May 2017 00:38:20 +1000
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <1495721053.3484266.988324184.59F35AE9@webmail.messagingengine.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
 <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>
 <1495529813.1777650.985588336.3D9322A0@webmail.messagingengine.com>
 <CACac1F_jZ6LqkSZLdA-c8h922H7AgMpR19ooT3SCB=sqk5G+7w@mail.gmail.com>
 <1495539382.1810627.985733704.7CD7CB52@webmail.messagingengine.com>
 <CACac1F9OedvrYtEUAKDb55d6pYN2XCV68nQtC_kwKFXcpq7-1w@mail.gmail.com>
 <1495543309.1825097.985759744.0BA2FA81@webmail.messagingengine.com>
 <CADiSq7e_NcQq5DAvD3WBvVp5BOv5vV2rsaMtK9ZxkEU4pttxzg@mail.gmail.com>
 <1495559093.2774864.986098160.74BDFA33@webmail.messagingengine.com>
 <CAOTb1wcpjyKy5b-O9squhNdk8Rn_aLCs=24056xb2zLwVr-cCw@mail.gmail.com>
 <1495621787.530876.986950088.6E287FBD@webmail.messagingengine.com>
 <CADiSq7dw1FiGE2oMmqwexK4635Vc5YrwUZyZvt6VvkSkGQLXNg@mail.gmail.com>
 <CACac1F-VY_P5k=muzB=EBrAypTAGS-Rk43ZzT5bCLcbAJ3t-ig@mail.gmail.com>
 <1495721053.3484266.988324184.59F35AE9@webmail.messagingengine.com>
Message-ID: <CADiSq7f6kJBBE_nUHgzmof2Xs6t7RGXny25QaduWiNqnH13v3w@mail.gmail.com>

On 26 May 2017 at 00:04, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> On Thu, May 25, 2017, at 02:27 PM, Paul Moore wrote:
>> I'd be concerned here that we risk making the frontend UI a lot more
>> complex for little actual benefit. I'd rather we stick with the
>> current model, where a backend just has some output to pass through to
>> the frontend. Let's get a solution that works for that before adding
>> extra complexity, or we'll never get the PEP signed off.
>
> I'm inclined to agree that we're overcomplicating things. But if we
> can't agree on which simple-but-imperfect option to take, maybe it's
> worth trying to work out something more complex.
>
> My proposed addition to the PEP so far says this:
>
> The build frontend may capture stdout and/or stderr from the backend. If
> the backend detects that an output stream is not a terminal/console
> (e.g. ``not sys.stdout.isatty()``), it SHOULD ensure that any output it
> writes to that stream is UTF-8 encoded. The build frontend MUST NOT fail
> if captured output is not valid UTF-8, but it MAY not preserve all the
> information in that case (e.g. it may decode using the *replace* error
> handler in Python). If the output stream is a terminal, the build
> backend is responsible for presenting its output accurately, as for any
> program running in a terminal.
>
> We could add a paragraph like this:
>
> The backend may do some operations, such as running subprocesses, which
> produce output in an unknown encoding. To handle such output, the build
> frontend MAY (?) create an empty directory, and set the environment
> variable PEP517_BUILD_LOGS to the path of this directory for the
> backend. If this environment variable is set, the backend MAY create any
> number of files inside this directory containing additional output. This
> is designed to allow the use of encoding detection tools on this output.
> If files are created in this directory, frontends SHOULD display its
> location in their output, and MAY display the contents of the files.

Seeing it like this pushes me from "Eh, maybe?" to "No, definitely not" :)

So that gets us to the point where we're agreeing that your suggested
addition to the PEP is basically right, with the only remaining
question being whether or not we're happy with the section that says
"it SHOULD ensure that any output it writes to that stream is UTF-8
encoded".

For a Python with locale coercion enabled, we're going to get that by
default, so such environments will comply without backend developers
doing anything in particular. Frontends may also decide to implement
their own PEP 538 style locale coercion for the backend build process
when they're running in a non-UTF-8 locale - specifying UTF-8 as a
SHOULD in the PEP gives them implied permission to do that.

So I don't think this is going to place any undue burden on backend
developers for *nix systems - frontends will probably want to
implement PEP 538 style locale coercion for LC_CTYPE to handle cases
where tools rely on the default stream encoding, but I think that's
fine.

That leaves Windows, and there I'm prepared to defer to Steve Dower's
opinion that it's better to deal with the encoding challenges of
consuming the output from MSVC in the build backend, rather than
expecting the frontend to deal with it. We also have a precedent now
in pip's legacy subprocess handling for what doing that reliably looks
like, so it shouldn't be hard for backend implementors to re-use that
approach as needed.

So I'm inclined to accept the encoding amendment, and then
provisionally accept the overall PEP pending implementation in pip.

I'll give others a couple of days to comment further, but assuming
nothing else comes up, I'll go ahead and do that on the weekend :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From p.f.moore at gmail.com  Thu May 25 10:56:54 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 25 May 2017 15:56:54 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CADiSq7f6kJBBE_nUHgzmof2Xs6t7RGXny25QaduWiNqnH13v3w@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
 <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>
 <1495529813.1777650.985588336.3D9322A0@webmail.messagingengine.com>
 <CACac1F_jZ6LqkSZLdA-c8h922H7AgMpR19ooT3SCB=sqk5G+7w@mail.gmail.com>
 <1495539382.1810627.985733704.7CD7CB52@webmail.messagingengine.com>
 <CACac1F9OedvrYtEUAKDb55d6pYN2XCV68nQtC_kwKFXcpq7-1w@mail.gmail.com>
 <1495543309.1825097.985759744.0BA2FA81@webmail.messagingengine.com>
 <CADiSq7e_NcQq5DAvD3WBvVp5BOv5vV2rsaMtK9ZxkEU4pttxzg@mail.gmail.com>
 <1495559093.2774864.986098160.74BDFA33@webmail.messagingengine.com>
 <CAOTb1wcpjyKy5b-O9squhNdk8Rn_aLCs=24056xb2zLwVr-cCw@mail.gmail.com>
 <1495621787.530876.986950088.6E287FBD@webmail.messagingengine.com>
 <CADiSq7dw1FiGE2oMmqwexK4635Vc5YrwUZyZvt6VvkSkGQLXNg@mail.gmail.com>
 <CACac1F-VY_P5k=muzB=EBrAypTAGS-Rk43ZzT5bCLcbAJ3t-ig@mail.gmail.com>
 <1495721053.3484266.988324184.59F35AE9@webmail.messagingengine.com>
 <CADiSq7f6kJBBE_nUHgzmof2Xs6t7RGXny25QaduWiNqnH13v3w@mail.gmail.com>
Message-ID: <CACac1F_unWSTHT25Uq14-FbeAFTOW+wvx3TC036F=BThHYjO=Q@mail.gmail.com>

On 25 May 2017 at 15:38, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Seeing it like this pushes me from "Eh, maybe?" to "No, definitely not" :)

Agreed. Given that it's stated as optional for frontends to support
it, I'd be arguing against pip bothering (as it seems like too much
complexity) - so I'd rather leave it out until another frontend comes
along. If at that point there's a need, we can always revise the PEP.

> So I'm inclined to accept the encoding amendment, and then
> provisionally accept the overall PEP pending implementation in pip.

Me too. (Assuming I understand Steve's comments on backends, and he's
comfortable with the idea that backends need to capture and manage
MSVC output for presentation to the frontend).

Paul

From steve.dower at python.org  Thu May 25 12:32:02 2017
From: steve.dower at python.org (Steve Dower)
Date: Thu, 25 May 2017 09:32:02 -0700
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CACac1F_unWSTHT25Uq14-FbeAFTOW+wvx3TC036F=BThHYjO=Q@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F_jZ6LqkSZLdA-c8h922H7AgMpR19ooT3SCB=sqk5G+7w@mail.gmail.com>
 <1495539382.1810627.985733704.7CD7CB52@webmail.messagingengine.com>
 <CACac1F9OedvrYtEUAKDb55d6pYN2XCV68nQtC_kwKFXcpq7-1w@mail.gmail.com>
 <1495543309.1825097.985759744.0BA2FA81@webmail.messagingengine.com>
 <CADiSq7e_NcQq5DAvD3WBvVp5BOv5vV2rsaMtK9ZxkEU4pttxzg@mail.gmail.com>
 <1495559093.2774864.986098160.74BDFA33@webmail.messagingengine.com>
 <CAOTb1wcpjyKy5b-O9squhNdk8Rn_aLCs=24056xb2zLwVr-cCw@mail.gmail.com>
 <1495621787.530876.986950088.6E287FBD@webmail.messagingengine.com>
 <CADiSq7dw1FiGE2oMmqwexK4635Vc5YrwUZyZvt6VvkSkGQLXNg@mail.gmail.com>
 <CACac1F-VY_P5k=muzB=EBrAypTAGS-Rk43ZzT5bCLcbAJ3t-ig@mail.gmail.com>
 <1495721053.3484266.988324184.59F35AE9@webmail.messagingengine.com>
 <CADiSq7f6kJBBE_nUHgzmof2Xs6t7RGXny25QaduWiNqnH13v3w@mail.gmail.com>
 <CACac1F_unWSTHT25Uq14-FbeAFTOW+wvx3TC036F=BThHYjO=Q@mail.gmail.com>
Message-ID: <1ccc5ddc-c809-a0eb-92b5-2816be76706a@python.org>

On 25May2017 0756, Paul Moore wrote:
> On 25 May 2017 at 15:38, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> So I'm inclined to accept the encoding amendment, and then
>> provisionally accept the overall PEP pending implementation in pip.
>
> Me too. (Assuming I understand Steve's comments on backends, and he's
> comfortable with the idea that backends need to capture and manage
> MSVC output for presentation to the frontend).

Sounds like you understood my comments :) +1 overall (-0 on a formal way 
to pass logs via the disk)

As I mentioned at one point, there's a bug against the CPython test 
suite that the distutils tests show too much console output, which is 
because distutils currently just lets MSVC write directly to the 
console. To fix it, we need to capture the output and then conditionally 
display it, at which point transcoding from ANSI to UTF-8 with 'replace' 
is trivial, and saves the front end (in this case, the test suite) from 
having to guess. So it is something that the backend around MSVC needs 
to do regardless, and if the PEP says "send me UTF-8" then it's one less 
thing for the backend developer to guess.

Cheers,
Steve

From jonwayne at google.com  Thu May 25 22:03:39 2017
From: jonwayne at google.com (Jon Wayne Parrott)
Date: Fri, 26 May 2017 02:03:39 +0000
Subject: [Distutils] Proposal: moving PyPA projects to pydoctheme
Message-ID: <CAN6JEy=usF+Z8FvjMV2BAkohJu5umvec6=bNX-+t6nZyoWGjcw@mail.gmail.com>

o/ Hello everyone,
I've been working on the Packaging User Guide and various discussions have
come up about the theme (
https://github.com/pypa/python-packaging-user-guide/issues/304) as well as
the common brand for PyPA projects (
https://github.com/pypa/python-packaging-user-guide/issues/62).

I'm proposing we switch PyPA projects (namely pypa.io, PyPUG, distlib, pip,
setuptools, virtualenv, warehouse, and wheel) to match the upstream CPython
docs for Python 3 (referred to as "pydoctheme").

Switching from the current readthedocs theme has a couple of advantages:

* Higher contrast and sans-serif fonts means better readability and
accessibility.
* Consistency with Python re-enforces that these are "official"/"blessed"
tools & documentation.
* A central shared theme among these projects allows us to make consistent
identity modifications across projects easily.

This work has been started on PyPUG (
https://github.com/pypa/python-packaging-user-guide/pull/305#issuecomment-304169735).
I have staged a build of PyPUG using the new theme here (
http://temp.theadora.io/pypug-pydoctheme/index.html). Please take a look
and comment on github with any concerns, and by all means tell me I'm crazy
for trying to do this. :)

If the primary maintainers of these projects all agree, I will create the
theme package and submit PRs to all the projects to do this migration.
You'll only need to approve. From what I understand those people are
@dstufft, @pfmoore, @jaraco, @vsajip, @dholth, and @ncoglan, but please let
me know if I missed anyone (I'm still new!)

Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170526/3cc58ab5/attachment.html>

From njs at pobox.com  Fri May 26 01:26:22 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 25 May 2017 22:26:22 -0700
Subject: [Distutils] Proposal: moving PyPA projects to pydoctheme
In-Reply-To: <CAN6JEy=usF+Z8FvjMV2BAkohJu5umvec6=bNX-+t6nZyoWGjcw@mail.gmail.com>
References: <CAN6JEy=usF+Z8FvjMV2BAkohJu5umvec6=bNX-+t6nZyoWGjcw@mail.gmail.com>
Message-ID: <CAPJVwBmyp+Q6zaGny8k7cZ_otQ3avDA=piXL0JXABaY7nA10LA@mail.gmail.com>

On Thu, May 25, 2017 at 7:03 PM, Jon Wayne Parrott via Distutils-SIG
<distutils-sig at python.org> wrote:
> o/ Hello everyone,
> I've been working on the Packaging User Guide and various discussions have
> come up about the theme
> (https://github.com/pypa/python-packaging-user-guide/issues/304) as well as
> the common brand for PyPA projects
> (https://github.com/pypa/python-packaging-user-guide/issues/62).
>
> I'm proposing we switch PyPA projects (namely pypa.io, PyPUG, distlib, pip,
> setuptools, virtualenv, warehouse, and wheel) to match the upstream CPython
> docs for Python 3 (referred to as "pydoctheme").
>
> Switching from the current readthedocs theme has a couple of advantages:
>
> * Higher contrast and sans-serif fonts means better readability and
> accessibility.
> * Consistency with Python re-enforces that these are "official"/"blessed"
> tools & documentation.
> * A central shared theme among these projects allows us to make consistent
> identity modifications across projects easily.

Personally, I always use the RTD theme for my projects because it has
a subtle but major advantage over every other Sphinx theme: it always
provides an overview of the entire manual in the sidebar, which makes
navigation much easier. With other themes I always feel like I'm lost.

Compare the sidebar here:
    https://packaging.python.org/current/
Versus here:
    http://temp.theadora.io/pypug-pydoctheme/current.html

I don't care either way about the RTD styling, pydoctheme looks nice
too, but IMO this specific feature is a pretty major usability win for
most projects. I believe it involves some disgusting hacks inside the
RTD theme code.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From jonwayne at google.com  Fri May 26 02:08:50 2017
From: jonwayne at google.com (Jon Wayne Parrott)
Date: Fri, 26 May 2017 06:08:50 +0000
Subject: [Distutils] Proposal: moving PyPA projects to pydoctheme
In-Reply-To: <CAPJVwBmyp+Q6zaGny8k7cZ_otQ3avDA=piXL0JXABaY7nA10LA@mail.gmail.com>
References: <CAN6JEy=usF+Z8FvjMV2BAkohJu5umvec6=bNX-+t6nZyoWGjcw@mail.gmail.com>
 <CAPJVwBmyp+Q6zaGny8k7cZ_otQ3avDA=piXL0JXABaY7nA10LA@mail.gmail.com>
Message-ID: <CAN6JEymxBf+ac2p4fWMWj_kHu_B=DA3M-ySm9CsvDst6sc2DpQ@mail.gmail.com>

This is great feedback Nathaniel. :)

Luckily, no nasty hacks involved. I've updated the pypa-theme to support
both global and local toc so that projects can pick whichever is most
appropriate for them. See the commit here
https://github.com/pypa/python-packaging-user-guide/pull/305/commits/12afb52f76fc33ad43438ec6b58c7333a6186b07
and
I've restaged the preview here
http://temp.theadora.io/pypug-pydoctheme/current.html#installation-tool-recommendations
(caching
might make it take a second, but visit any other page and you should see
it).

Let me know if that addresses your concerns and if you have any more!

On Thu, May 25, 2017 at 10:26 PM Nathaniel Smith <njs at pobox.com> wrote:

> On Thu, May 25, 2017 at 7:03 PM, Jon Wayne Parrott via Distutils-SIG
> <distutils-sig at python.org> wrote:
> > o/ Hello everyone,
> > I've been working on the Packaging User Guide and various discussions
> have
> > come up about the theme
> > (https://github.com/pypa/python-packaging-user-guide/issues/304) as
> well as
> > the common brand for PyPA projects
> > (https://github.com/pypa/python-packaging-user-guide/issues/62).
> >
> > I'm proposing we switch PyPA projects (namely pypa.io, PyPUG, distlib,
> pip,
> > setuptools, virtualenv, warehouse, and wheel) to match the upstream
> CPython
> > docs for Python 3 (referred to as "pydoctheme").
> >
> > Switching from the current readthedocs theme has a couple of advantages:
> >
> > * Higher contrast and sans-serif fonts means better readability and
> > accessibility.
> > * Consistency with Python re-enforces that these are "official"/"blessed"
> > tools & documentation.
> > * A central shared theme among these projects allows us to make
> consistent
> > identity modifications across projects easily.
>
> Personally, I always use the RTD theme for my projects because it has
> a subtle but major advantage over every other Sphinx theme: it always
> provides an overview of the entire manual in the sidebar, which makes
> navigation much easier. With other themes I always feel like I'm lost.
>
> Compare the sidebar here:
>     https://packaging.python.org/current/
> Versus here:
>     http://temp.theadora.io/pypug-pydoctheme/current.html
>
> I don't care either way about the RTD styling, pydoctheme looks nice
> too, but IMO this specific feature is a pretty major usability win for
> most projects. I believe it involves some disgusting hacks inside the
> RTD theme code.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170526/568341a6/attachment-0001.html>

From p.f.moore at gmail.com  Fri May 26 02:45:53 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 26 May 2017 07:45:53 +0100
Subject: [Distutils] Proposal: moving PyPA projects to pydoctheme
In-Reply-To: <CAPJVwBmyp+Q6zaGny8k7cZ_otQ3avDA=piXL0JXABaY7nA10LA@mail.gmail.com>
References: <CAN6JEy=usF+Z8FvjMV2BAkohJu5umvec6=bNX-+t6nZyoWGjcw@mail.gmail.com>
 <CAPJVwBmyp+Q6zaGny8k7cZ_otQ3avDA=piXL0JXABaY7nA10LA@mail.gmail.com>
Message-ID: <CACac1F_Ava8PVo23ZK-GmAV9OJs5a5BANF=j2OM+Oyw0KaCY3w@mail.gmail.com>

On 26 May 2017 at 06:26, Nathaniel Smith <njs at pobox.com> wrote:
> I don't care either way about the RTD styling, pydoctheme looks nice
> too, but IMO this specific feature is a pretty major usability win for
> most projects. I believe it involves some disgusting hacks inside the
> RTD theme code.

This was a concern for me, too - thanks for bringing it up - and
honestly, my only real reservation about the new theme. It looks like
Jon has addressed it, so personally, I'm happy with the proposed new
theme now.

Paul

From njs at pobox.com  Fri May 26 03:06:39 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 26 May 2017 00:06:39 -0700
Subject: [Distutils] Proposal: moving PyPA projects to pydoctheme
In-Reply-To: <CAN6JEymxBf+ac2p4fWMWj_kHu_B=DA3M-ySm9CsvDst6sc2DpQ@mail.gmail.com>
References: <CAN6JEy=usF+Z8FvjMV2BAkohJu5umvec6=bNX-+t6nZyoWGjcw@mail.gmail.com>
 <CAPJVwBmyp+Q6zaGny8k7cZ_otQ3avDA=piXL0JXABaY7nA10LA@mail.gmail.com>
 <CAN6JEymxBf+ac2p4fWMWj_kHu_B=DA3M-ySm9CsvDst6sc2DpQ@mail.gmail.com>
Message-ID: <CAPJVwBm+EEmRUJ2yib_w8=gbpUdSvB6KaV2x94O98_d6wCZ69A@mail.gmail.com>

Further nitpick: the new version shows the whole ToC, but doesn't do
anything to mark which part of it corresponds to the current page (cf
the RTD theme's grey box), which makes it hard to orient oneself when
looking at it. Some sort of "you are here" indicator would help :-)

On Thu, May 25, 2017 at 11:08 PM, Jon Wayne Parrott <jonwayne at google.com> wrote:
> This is great feedback Nathaniel. :)
>
> Luckily, no nasty hacks involved. I've updated the pypa-theme to support
> both global and local toc so that projects can pick whichever is most
> appropriate for them. See the commit here
> https://github.com/pypa/python-packaging-user-guide/pull/305/commits/12afb52f76fc33ad43438ec6b58c7333a6186b07
> and I've restaged the preview here
> http://temp.theadora.io/pypug-pydoctheme/current.html#installation-tool-recommendations
> (caching might make it take a second, but visit any other page and you
> should see it).
>
> Let me know if that addresses your concerns and if you have any more!
>
> On Thu, May 25, 2017 at 10:26 PM Nathaniel Smith <njs at pobox.com> wrote:
>>
>> On Thu, May 25, 2017 at 7:03 PM, Jon Wayne Parrott via Distutils-SIG
>> <distutils-sig at python.org> wrote:
>> > o/ Hello everyone,
>> > I've been working on the Packaging User Guide and various discussions
>> > have
>> > come up about the theme
>> > (https://github.com/pypa/python-packaging-user-guide/issues/304) as well
>> > as
>> > the common brand for PyPA projects
>> > (https://github.com/pypa/python-packaging-user-guide/issues/62).
>> >
>> > I'm proposing we switch PyPA projects (namely pypa.io, PyPUG, distlib,
>> > pip,
>> > setuptools, virtualenv, warehouse, and wheel) to match the upstream
>> > CPython
>> > docs for Python 3 (referred to as "pydoctheme").
>> >
>> > Switching from the current readthedocs theme has a couple of advantages:
>> >
>> > * Higher contrast and sans-serif fonts means better readability and
>> > accessibility.
>> > * Consistency with Python re-enforces that these are
>> > "official"/"blessed"
>> > tools & documentation.
>> > * A central shared theme among these projects allows us to make
>> > consistent
>> > identity modifications across projects easily.
>>
>> Personally, I always use the RTD theme for my projects because it has
>> a subtle but major advantage over every other Sphinx theme: it always
>> provides an overview of the entire manual in the sidebar, which makes
>> navigation much easier. With other themes I always feel like I'm lost.
>>
>> Compare the sidebar here:
>>     https://packaging.python.org/current/
>> Versus here:
>>     http://temp.theadora.io/pypug-pydoctheme/current.html
>>
>> I don't care either way about the RTD styling, pydoctheme looks nice
>> too, but IMO this specific feature is a pretty major usability win for
>> most projects. I believe it involves some disgusting hacks inside the
>> RTD theme code.
>>
>> -n
>>
>> --
>> Nathaniel J. Smith -- https://vorpus.org



-- 
Nathaniel J. Smith -- https://vorpus.org

From donald at stufft.io  Fri May 26 07:20:14 2017
From: donald at stufft.io (Donald Stufft)
Date: Fri, 26 May 2017 07:20:14 -0400
Subject: [Distutils] Proposal: moving PyPA projects to pydoctheme
In-Reply-To: <CAN6JEy=usF+Z8FvjMV2BAkohJu5umvec6=bNX-+t6nZyoWGjcw@mail.gmail.com>
References: <CAN6JEy=usF+Z8FvjMV2BAkohJu5umvec6=bNX-+t6nZyoWGjcw@mail.gmail.com>
Message-ID: <F3F649BE-BFE4-44B2-84D4-EB9D400B9565@stufft.io>

This looks fine to me.

> On May 25, 2017, at 10:03 PM, Jon Wayne Parrott via Distutils-SIG <distutils-sig at python.org> wrote:
> 
> o/ Hello everyone,
> I've been working on the Packaging User Guide and various discussions have come up about the theme (https://github.com/pypa/python-packaging-user-guide/issues/304 <https://github.com/pypa/python-packaging-user-guide/issues/304>) as well as the common brand for PyPA projects (https://github.com/pypa/python-packaging-user-guide/issues/62 <https://github.com/pypa/python-packaging-user-guide/issues/62>).
> 
> I'm proposing we switch PyPA projects (namely pypa.io <http://pypa.io/>, PyPUG, distlib, pip, setuptools, virtualenv, warehouse, and wheel) to match the upstream CPython docs for Python 3 (referred to as "pydoctheme").
> 
> Switching from the current readthedocs theme has a couple of advantages:
> 
> * Higher contrast and sans-serif fonts means better readability and accessibility.
> * Consistency with Python re-enforces that these are "official"/"blessed" tools & documentation.
> * A central shared theme among these projects allows us to make consistent identity modifications across projects easily.
> 
> This work has been started on PyPUG (https://github.com/pypa/python-packaging-user-guide/pull/305#issuecomment-304169735 <https://github.com/pypa/python-packaging-user-guide/pull/305#issuecomment-304169735>). I have staged a build of PyPUG using the new theme here (http://temp.theadora.io/pypug-pydoctheme/index.html <http://temp.theadora.io/pypug-pydoctheme/index.html>). Please take a look and comment on github with any concerns, and by all means tell me I'm crazy for trying to do this. :)
> 
> If the primary maintainers of these projects all agree, I will create the theme package and submit PRs to all the projects to do this migration. You'll only need to approve. From what I understand those people are @dstufft, @pfmoore, @jaraco, @vsajip, @dholth, and @ncoglan, but please let me know if I missed anyone (I'm still new!)
> 
> Thanks!
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170526/f532787c/attachment.html>

From ncoghlan at gmail.com  Fri May 26 08:13:53 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 26 May 2017 22:13:53 +1000
Subject: [Distutils] Proposal: moving PyPA projects to pydoctheme
In-Reply-To: <CAPJVwBm+EEmRUJ2yib_w8=gbpUdSvB6KaV2x94O98_d6wCZ69A@mail.gmail.com>
References: <CAN6JEy=usF+Z8FvjMV2BAkohJu5umvec6=bNX-+t6nZyoWGjcw@mail.gmail.com>
 <CAPJVwBmyp+Q6zaGny8k7cZ_otQ3avDA=piXL0JXABaY7nA10LA@mail.gmail.com>
 <CAN6JEymxBf+ac2p4fWMWj_kHu_B=DA3M-ySm9CsvDst6sc2DpQ@mail.gmail.com>
 <CAPJVwBm+EEmRUJ2yib_w8=gbpUdSvB6KaV2x94O98_d6wCZ69A@mail.gmail.com>
Message-ID: <CADiSq7expwQKToHZTJ9p_jTVTOz-b=Fpn-rAOOPN5cEzhaT+eA@mail.gmail.com>

Thanks for working on this, Jon!

On 26 May 2017 at 17:06, Nathaniel Smith <njs at pobox.com> wrote:
> Further nitpick: the new version shows the whole ToC, but doesn't do
> anything to mark which part of it corresponds to the current page (cf
> the RTD theme's grey box), which makes it hard to orient oneself when
> looking at it. Some sort of "you are here" indicator would help :-)

While the subtree expansion does provide a bit of that, I do agree
that some additional contrast between the expanded section and the
rest of the ToC would be beneficial.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From jonwayne at google.com  Fri May 26 12:32:32 2017
From: jonwayne at google.com (Jon Wayne Parrott)
Date: Fri, 26 May 2017 16:32:32 +0000
Subject: [Distutils] Proposal: moving PyPA projects to pydoctheme
In-Reply-To: <CADiSq7expwQKToHZTJ9p_jTVTOz-b=Fpn-rAOOPN5cEzhaT+eA@mail.gmail.com>
References: <CAN6JEy=usF+Z8FvjMV2BAkohJu5umvec6=bNX-+t6nZyoWGjcw@mail.gmail.com>
 <CAPJVwBmyp+Q6zaGny8k7cZ_otQ3avDA=piXL0JXABaY7nA10LA@mail.gmail.com>
 <CAN6JEymxBf+ac2p4fWMWj_kHu_B=DA3M-ySm9CsvDst6sc2DpQ@mail.gmail.com>
 <CAPJVwBm+EEmRUJ2yib_w8=gbpUdSvB6KaV2x94O98_d6wCZ69A@mail.gmail.com>
 <CADiSq7expwQKToHZTJ9p_jTVTOz-b=Fpn-rAOOPN5cEzhaT+eA@mail.gmail.com>
Message-ID: <CAN6JEymq=4om9jHfnxtrz9bsAqdKE7gOQ6nWMTufdv9HgvwBZg@mail.gmail.com>

Awesome, more good feedback. I've made some stylesheet adjustments to
highlight both the current page (by bolding) and the current section (by
coloring links). Let me know on the PR if you have alternative
recommendations (leaving out the current section coloring is actually fine,
IMO). Staged again here (
http://temp.theadora.io/pypug-pydoctheme/self_hosted_repository.html).

Since we mostly seem to all be in agreement, can someone who's an admin on
the PyPA org create the pypa-theme repository for me?

On Fri, May 26, 2017 at 5:13 AM Nick Coghlan <ncoghlan at gmail.com> wrote:

> Thanks for working on this, Jon!
>
> On 26 May 2017 at 17:06, Nathaniel Smith <njs at pobox.com> wrote:
> > Further nitpick: the new version shows the whole ToC, but doesn't do
> > anything to mark which part of it corresponds to the current page (cf
> > the RTD theme's grey box), which makes it hard to orient oneself when
> > looking at it. Some sort of "you are here" indicator would help :-)
>
> While the subtree expansion does provide a bit of that, I do agree
> that some additional contrast between the expanded section and the
> rest of the ToC would be beneficial.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170526/f72547ab/attachment.html>

From contact at ionelmc.ro  Sat May 27 07:51:13 2017
From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=)
Date: Sat, 27 May 2017 14:51:13 +0300
Subject: [Distutils] Proposal: moving PyPA projects to pydoctheme
In-Reply-To: <CAN6JEymq=4om9jHfnxtrz9bsAqdKE7gOQ6nWMTufdv9HgvwBZg@mail.gmail.com>
References: <CAN6JEy=usF+Z8FvjMV2BAkohJu5umvec6=bNX-+t6nZyoWGjcw@mail.gmail.com>
 <CAPJVwBmyp+Q6zaGny8k7cZ_otQ3avDA=piXL0JXABaY7nA10LA@mail.gmail.com>
 <CAN6JEymxBf+ac2p4fWMWj_kHu_B=DA3M-ySm9CsvDst6sc2DpQ@mail.gmail.com>
 <CAPJVwBm+EEmRUJ2yib_w8=gbpUdSvB6KaV2x94O98_d6wCZ69A@mail.gmail.com>
 <CADiSq7expwQKToHZTJ9p_jTVTOz-b=Fpn-rAOOPN5cEzhaT+eA@mail.gmail.com>
 <CAN6JEymq=4om9jHfnxtrz9bsAqdKE7gOQ6nWMTufdv9HgvwBZg@mail.gmail.com>
Message-ID: <CANkHFr_zjjhHbLp_Yi03bRFVbY0BWPF7Qx7vfZc-ir7wN5NXng@mail.gmail.com>

Interestingly this doesn't have the "sticky" sidebar (eg: it moves as you
scroll page) as the py3 docs have. This is my attempt to package it into a
reusable theme btw:
http://ionelmc.github.io/sphinx-py3doc-enhanced-theme/bare/reference/foo.bar.html

You probably don't want to use that directly but maybe there is stuff you
can just copy from it.


Thanks,
-- Ionel Cristian M?rie?, http://blog.ionelmc.ro

On Fri, May 26, 2017 at 7:32 PM, Jon Wayne Parrott via Distutils-SIG <
distutils-sig at python.org> wrote:

> Awesome, more good feedback. I've made some stylesheet adjustments to
> highlight both the current page (by bolding) and the current section (by
> coloring links). Let me know on the PR if you have alternative
> recommendations (leaving out the current section coloring is actually fine,
> IMO). Staged again here (http://temp.theadora.io/
> pypug-pydoctheme/self_hosted_repository.html).
>
> Since we mostly seem to all be in agreement, can someone who's an admin on
> the PyPA org create the pypa-theme repository for me?
>
> On Fri, May 26, 2017 at 5:13 AM Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> Thanks for working on this, Jon!
>>
>> On 26 May 2017 at 17:06, Nathaniel Smith <njs at pobox.com> wrote:
>> > Further nitpick: the new version shows the whole ToC, but doesn't do
>> > anything to mark which part of it corresponds to the current page (cf
>> > the RTD theme's grey box), which makes it hard to orient oneself when
>> > looking at it. Some sort of "you are here" indicator would help :-)
>>
>> While the subtree expansion does provide a bit of that, I do agree
>> that some additional contrast between the expanded section and the
>> rest of the ToC would be beneficial.
>>
>> Cheers,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170527/a26ccb82/attachment.html>

From thomas at kluyver.me.uk  Sun May 28 05:30:53 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Sun, 28 May 2017 10:30:53 +0100
Subject: [Distutils] PEP 517 - specifying build system in pyproject.toml
In-Reply-To: <CADiSq7f6kJBBE_nUHgzmof2Xs6t7RGXny25QaduWiNqnH13v3w@mail.gmail.com>
References: <1495203483.257055.982092736.4B991B04@webmail.messagingengine.com>
 <CACac1F8Ccc1rqUw03vV_qScGzSA=+n036cBimtBWjUPMugXXZQ@mail.gmail.com>
 <CADiSq7fQcgjWApcss0T+NoF5fxTGk+Vb+Ena_31fZDwj4_QX4Q@mail.gmail.com>
 <8525147c-7e9e-9bc6-a271-cc05119f0065@python.org>
 <1495300293.1545181.983089392.3DFD1E55@webmail.messagingengine.com>
 <16ede617-26f8-44cc-68d4-6bb447aaa994@python.org>
 <1495447016.3889583.984359600.2A083F74@webmail.messagingengine.com>
 <CACac1F8O1rMiX3R5HHatusZrWsWdDw-4BmmseVELKduMz-kcoQ@mail.gmail.com>
 <1495448535.3894766.984386928.4C1D80D0@webmail.messagingengine.com>
 <CACac1F9GC5SkXNBxxURpP_E454zckuBi45nQvOQZt3xCB7wv5A@mail.gmail.com>
 <1495452497.3907821.984434000.02E3E514@webmail.messagingengine.com>
 <CACac1F_QmQqy4Us93xAaDp_vYjOg3nOAw7C57oRj7_R4BkKpUg@mail.gmail.com>
 <CADiSq7cRV7PgLeu6dJ_S5er1HZPanUmiuWTxc=ndpH=50kW7PA@mail.gmail.com>
 <CACac1F8gce6fFTXHYRQVGHRnajA0Lbkh28sjAuO7b6SZQtrPtA@mail.gmail.com>
 <dbbb120f-50b3-3483-16a1-fe6f7245cd9d@python.org>
 <CADiSq7dRFwO9NBhsd+XneQO2J7Q6xCMDO4S9a79VSNvsbfHR+g@mail.gmail.com>
 <CACac1F8WvXnOn=jTL3WH+RmHtATs_6fX4r9H+t5vQmiLP5bHgg@mail.gmail.com>
 <1495529813.1777650.985588336.3D9322A0@webmail.messagingengine.com>
 <CACac1F_jZ6LqkSZLdA-c8h922H7AgMpR19ooT3SCB=sqk5G+7w@mail.gmail.com>
 <1495539382.1810627.985733704.7CD7CB52@webmail.messagingengine.com>
 <CACac1F9OedvrYtEUAKDb55d6pYN2XCV68nQtC_kwKFXcpq7-1w@mail.gmail.com>
 <1495543309.1825097.985759744.0BA2FA81@webmail.messagingengine.com>
 <CADiSq7e_NcQq5DAvD3WBvVp5BOv5vV2rsaMtK9ZxkEU4pttxzg@mail.gmail.com>
 <1495559093.2774864.986098160.74BDFA33@webmail.messagingengine.com>
 <CAOTb1wcpjyKy5b-O9squhNdk8Rn_aLCs=24056xb2zLwVr-cCw@mail.gmail.com>
 <1495621787.530876.986950088.6E287FBD@webmail.messagingengine.com>
 <CADiSq7dw1FiGE2oMmqwexK4635Vc5YrwUZyZvt6VvkSkGQLXNg@mail.gmail.com>
 <CACac1F-VY_P5k=muzB=EBrAypTAGS-Rk43ZzT5bCLcbAJ3t-ig@mail.gmail.com>
 <1495721053.3484266.988324184.59F35AE9@webmail.messagingengine.com>
 <CADiSq7f6kJBBE_nUHgzmof2Xs6t7RGXny25QaduWiNqnH13v3w@mail.gmail.com>
Message-ID: <1495963853.2552223.990857144.1D2BA042@webmail.messagingengine.com>

On Thu, May 25, 2017, at 03:38 PM, Nick Coghlan wrote:
> Seeing it like this pushes me from "Eh, maybe?" to "No, definitely not"
> [on the log directory] :)

That's fine by me. It does feel like unwanted extra complexity for both
backends and frontends. And backends dealing with output in an unknown
encoding can still choose to write it to a file and log "full output in
/tmp/blah" if they want - they don't need a spec for that.

Thomas

From ncoghlan at gmail.com  Sun May 28 21:34:28 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 29 May 2017 11:34:28 +1000
Subject: [Distutils] Last call for PEP 516 champions
In-Reply-To: <CADiSq7fDUb_asZ8-bmGhEhEQDnJ+vC6F2n5=N3CMUibSbZjTTg@mail.gmail.com>
References: <CADiSq7fDUb_asZ8-bmGhEhEQDnJ+vC6F2n5=N3CMUibSbZjTTg@mail.gmail.com>
Message-ID: <CADiSq7cnkEat-u-00A1NCN510gFRLADWFsyL4xOFHmo4KgKk_Q@mail.gmail.com>

On 23 May 2017 at 00:40, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Hi folks,
>
> The restarted discussion on PEP 517 meant I realised that we hadn't
> officially decided between its Python API based approach and PEP 516's
> approach of using the backend CLI as the standardised interface (akin
> to the current setup.py approach).
>
> My current intention is to reject PEP 516's CLI standardisation
> approach on the following grounds:
>
> - PEP 517 makes a convincing case for the benefits of the Python API
> based approach within the Python ecosystem
> - the difficulties encountered in evolving the setup.py CLI over time
> lend significant weight to the notion that a Python level API will be
> easier to update without breaking backwards compatibility
> - PEP 517 still advises front-ends to isolate back-end invocation
> behind a subprocess boundary due to all of the other practical
> benefits that brings, it just makes the specifics of that invocation
> an implementation detail of the front-end tool
> - third party tools that want an implementation language independent
> CLI abstraction over the Python ecosystem (including builds) can just
> use pip itself (or another standards-compliant frontend)
>
> This isn't particularly urgent, but I also don't see a lot of reason
> for an extended discussion, so if I don't hear a convincing
> counterargument in the meantime, I'll mark the PEP as rejected at the
> beginning of June :)

Since I'm about to accept PEP 517, I'm bringing this rejection forward
a couple of days, and officially declaring PEP 516 rejected in favour
of PEP 517, with the subprocess CLI being considered a private
implementation detail of front-end tools.

Thanks Robert and Nathaniel for your work on PEP 516 and your
participation in the related discussions :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun May 28 23:40:31 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 29 May 2017 13:40:31 +1000
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
Message-ID: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>

Hi folks,

I am hereby provisionally accepting PEP 517 as our CLI independent
interface for build backends, allowing interested parties to move
forward with implementing it as part of pip and other tools.

As with other provisionally accepted PyPA interoperability
specifications, it won't be considered completely settled until it has
been implemented and released in pip and we have at least one
iteration of end user feedback.

Thanks to Nathaniel & Thomas for their work on the PEP, and to
everyone that participated in the related discussions!

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From donald at stufft.io  Mon May 29 01:37:39 2017
From: donald at stufft.io (Donald Stufft)
Date: Mon, 29 May 2017 01:37:39 -0400
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
Message-ID: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>


> On May 28, 2017, at 11:40 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> Hi folks,
> 
> I am hereby provisionally accepting PEP 517 as our CLI independent
> interface for build backends, allowing interested parties to move
> forward with implementing it as part of pip and other tools.
> 
> As with other provisionally accepted PyPA interoperability
> specifications, it won't be considered completely settled until it has
> been implemented and released in pip and we have at least one
> iteration of end user feedback.
> 
> Thanks to Nathaniel & Thomas for their work on the PEP, and to
> everyone that participated in the related discussions!
> 


Bleh, I had it on my stack to respond to PEP 517, but life has been super hectic so I hadn?t gotten around to it.

Here are a few thoughts FWIW:

1. Using the {package-name}-{package-version}.dist-info in the get_wheel_metadata() metadata is a mistake I think. In pip currently we have a bug we have not yet been able to track down because there is nothing systematically preventing both foobar-1.0.dist-info and foobar-2.0.distinfo from existing side by side in a build directory (or inside a wheel for that matter). Thus I think this naming scheme is a nuisance and we shouldn?t propagate it any further. I would just use something like DIST-INFO/ which will completely side step this issue. The only reason I can think of to use the current scheme is to make it easier to shutil.copytree it into the wheel, but handling that case is trivial.

2. As I mentioned in previous discussions on this, I think that this interface *needs* some mechanism to ask it to build a sdist. Ideally this would either fail or recreate the sdist when being run from something that is already an sdist. In pip when we?re doing ``pip install .`` we currently copy tree the entire thing into a temporary directory, and we run the build out of there. For a variety of reasons we?re going to keep build isolation, but the current mechanism is painful because it also grabs things like .tox, .git, etc. At one point we tried to simply filter those out but it broke some packages expectations. The planned (which is on my long list of things to do?) mechanism to fix this is to create a sdist from a ``.`` and then install from that (building an intermediate wheel as well). This also solves another class of bugs that people run into where ``pip install .`` and ``python setup.py sdist && pip install dist/*`` give different results. As written, this PEP prevents that from happening (and thus, when I implement it, I?ll only be able to implement it for old style sdists, and will need to tell people to continue to use old style if they don?t want pip to grab random directories from ``.``).


Other than that, it looks fine, and #2 is the one that I think is going to be the bigger issue in pip.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170529/f7f7a276/attachment.html>

From njs at pobox.com  Mon May 29 03:05:23 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 29 May 2017 00:05:23 -0700
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
Message-ID: <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>

On Sun, May 28, 2017 at 10:37 PM, Donald Stufft <donald at stufft.io> wrote:
>
> Bleh, I had it on my stack to respond to PEP 517, but life has been super
> hectic so I hadn?t gotten around to it.
>
> Here are a few thoughts FWIW:
>
> 1. Using the {package-name}-{package-version}.dist-info in the
> get_wheel_metadata() metadata is a mistake I think. In pip currently we have
> a bug we have not yet been able to track down because there is nothing
> systematically preventing both foobar-1.0.dist-info and foobar-2.0.distinfo
> from existing side by side in a build directory (or inside a wheel for that
> matter). Thus I think this naming scheme is a nuisance and we shouldn?t
> propagate it any further. I would just use something like DIST-INFO/ which
> will completely side step this issue. The only reason I can think of to use
> the current scheme is to make it easier to shutil.copytree it into the
> wheel, but handling that case is trivial.

The rationale for this is to leave the door open to in the future
allowing the same sdist to build multiple wheels. Obviously that would
require a whole 'nother PEP, but I keep running into cases where this
is a blocker so I think it will happen eventually, and in any case
don't want to make new barriers...

For get_wheel_metadata() in particular there are several options
though... we could call it DIST-INFO/ and then later declare that
DIST-INFO2/, DIST-INFO3/, etc. are also valid and pip will look at all
of them.

{package-name}.dist-info might also be reasonable, both here and in
actual installs...

In general get_wheel_metadata is an optimization for the backtracking
resolver (that doesn't exist yet) to more efficiently handle cases
where there are complex constraints and no built wheels (both of which
we hope will be rare). Robert thinks it's very important, and he knows
more about that bit of the code than I do, but if it becomes an issue
we could even defer get_wheel_metadata to a future PEP.

> 2. As I mentioned in previous discussions on this, I think that this
> interface *needs* some mechanism to ask it to build a sdist. Ideally this
> would either fail or recreate the sdist when being run from something that
> is already an sdist. In pip when we?re doing ``pip install .`` we currently
> copy tree the entire thing into a temporary directory, and we run the build
> out of there. For a variety of reasons we?re going to keep build isolation,
> but the current mechanism is painful because it also grabs things like .tox,
> .git, etc. At one point we tried to simply filter those out but it broke
> some packages expectations. The planned (which is on my long list of things
> to do?) mechanism to fix this is to create a sdist from a ``.`` and then
> install from that (building an intermediate wheel as well). This also solves
> another class of bugs that people run into where ``pip install .`` and
> ``python setup.py sdist && pip install dist/*`` give different results. As
> written, this PEP prevents that from happening (and thus, when I implement
> it, I?ll only be able to implement it for old style sdists, and will need to
> tell people to continue to use old style if they don?t want pip to grab
> random directories from ``.``).
>
>
> Other than that, it looks fine, and #2 is the one that I think is going to
> be the bigger issue in pip.

I think there's some pip bug somewhere discussing this, where Ralf
Gommers and I point out that this is a complete showstopper for
projects with complex and expensive builds (like scipy). If 'pip
install .' is going to replace 'setup.py install', then it needs to
support incremental builds, and the way
setup.py-and-almost-every-other-build-tool do this currently is by
reusing the working directory across builds.

I think there's some hope for making both of us happy, though, because
our concern is mostly about the
install-this-potentially-dirty-working-directory case, and your
concern is mostly about the build-this-package-for-release case
(right)?

Right now pip doesn't really have a good way to expressing the latter.
'pip install directory/' is relatively unambiguously saying that I
want a local install of some potentially-locally-modified files, and
while it might involve a temporary wheel internally there's no need to
expose this in any way (and e.g. it certainly shouldn't be cached), so
I think it's OK if this builds in-place and risks giving different
results than 'pip install sdist.tar.gz'. (Also note that in the most
common case where a naive user might use this accidentally, where
they've downloaded an sdist, unpacked it manually, and then run 'pip
install .', they *do* give the same results -- the potential for
trouble only comes when someone runs 'pip install .' multiple times in
the same directory.)

The other option would be to say that 'pip install .' is *not* the
preferred way to build-and-install from a working directory, and
there's some other command you have to use instead (maybe
build-system-specific) if you want efficient builds. This seems
unreasonably unfriendly to me, though; 'pip install .' is really the
obvious thing, so it should also be the one you want for common usage.

OTOH the closest thing to a do-a-release-build command currently is
'pip wheel', but it actually has the wrong semantics for making
release builds, because it downloads and builds wheels for the entire
dependency chain. I don't care if 'pip wheel directory/' copies the
directory or makes an sdist or what. If you want to build from an
sdist here it's fine with me :-). But I'm also uncertain of the value,
because 'pip wheel directory/' is a somewhat weird command to be
running in the first place, and it's rare and heavy-weight enough that
the shutil.copy() thing seems as good as anything.

And what seems like it's really missing is a command to like...
generate an sdist, build it into a wheel, and then leave them both
somewhere twine can see them. Maybe 'pip release .'? This would
definitely require a build-an-sdist hook in the build backend, and to
me it seems like the Right way to address your concern. In PEP 517 as
currently written this is punted to build-backend-specific tooling,
and I agree that that's unpleasant. But given that 'pip release'
doesn't even exist yet, maybe it makes more sense to defer the
build-an-sdist hook to a new PEP that we can write alongside 'pip
release'?

It might also make sense to have a 'pip ci .' that makes a release
build and installs it for testing...

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From thomas at kluyver.me.uk  Mon May 29 03:57:55 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Mon, 29 May 2017 08:57:55 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>
Message-ID: <1496044675.3814377.991545912.5F9435FF@webmail.messagingengine.com>

On Mon, May 29, 2017, at 08:05 AM, Nathaniel Smith wrote:
> And what seems like it's really missing is a command to like...
> generate an sdist, build it into a wheel, and then leave them both
> somewhere twine can see them. Maybe 'pip release .'?

FWIW, we've just had a discussion over naming for similar functionality
in flit. We've ended up going with 'flit build' to build sdist+wheel,
and 'flit publish' to also upload them to PyPI.

Thomas

From p.f.moore at gmail.com  Mon May 29 05:19:07 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 29 May 2017 10:19:07 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>
Message-ID: <CACac1F8UvthfRgV=bPQTM+dTcnZvZYDTJ3CKo8=5Ke1J4L8jmQ@mail.gmail.com>

On 29 May 2017 at 08:05, Nathaniel Smith <njs at pobox.com> wrote:
> Right now pip doesn't really have a good way to expressing the latter.
> 'pip install directory/' is relatively unambiguously saying that I
> want a local install of some potentially-locally-modified files, and
> while it might involve a temporary wheel internally there's no need to
> expose this in any way (and e.g. it certainly shouldn't be cached), so
> I think it's OK if this builds in-place and risks giving different
> results than 'pip install sdist.tar.gz'. (Also note that in the most
> common case where a naive user might use this accidentally, where
> they've downloaded an sdist, unpacked it manually, and then run 'pip
> install .', they *do* give the same results -- the potential for
> trouble only comes when someone runs 'pip install .' multiple times in
> the same directory.)

I think that the key thing here is that as things stand, pip needs a
means to copy an existing "source tree", as efficiently as possible.
For local directories (source checkouts, typically) there's a lot of
clutter that isn't needed to replicate the "source tree" aspect of the
directory - but we can't reliably determine what is clutter and what
isn't.

Whether that copying is a good idea, in the face of the need to do
incremental builds, is something of an open question - clearly we
can't do something that closes the door on incremental builds, but
equally the overhead of copying unwanted data is huge at the moment,
and we can't ignore that.

Talking about a "build a sdist" operation brings a whole load of
questions about whether there should be a sdist format, what about
sdist 2.0, etc into the mix. So maybe we should avoid all that, and
say that pip[1] needs a "copy a source tree" operation. Backends
SHOULD implement that by skipping any unneeded files in the source
tree,but can fall back to a simple copy if they wish. In fact, we
could make the operation optional and have *pip* fall back to copying
if necessary. It would then be a backend quality of implementation
issue if builds are slow because multi-megabyte git trees get copied
unnecessarily.

(This operation might also help tools like setuptools-scm that need
git information to work - the backend could extract that information
on a "copy" operation and pit it somewhere static for the build).

Paul

[1] As pip is currently the only frontend, "what pip needs right now"
is the only non-theoretical indication we have of what frontends might
need to do, so we should be cautious about dismissing this as "pip
shouldn't work like this", IMO.

From ralf.gommers at gmail.com  Mon May 29 06:46:24 2017
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Mon, 29 May 2017 22:46:24 +1200
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CACac1F8UvthfRgV=bPQTM+dTcnZvZYDTJ3CKo8=5Ke1J4L8jmQ@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>
 <CACac1F8UvthfRgV=bPQTM+dTcnZvZYDTJ3CKo8=5Ke1J4L8jmQ@mail.gmail.com>
Message-ID: <CABL7CQjuPNCymVBDXbJmg6jJjaMShVJLTS22ApAr7Qs76Hta0g@mail.gmail.com>

> I think there's some pip bug somewhere discussing this ....

https://github.com/pypa/pip/issues/2195
https://github.com/pypa/pip/pull/3219
plus some long mailing list threads IIRC

On Mon, May 29, 2017 at 9:19 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 29 May 2017 at 08:05, Nathaniel Smith <njs at pobox.com> wrote:
> > Right now pip doesn't really have a good way to expressing the latter.

> 'pip install directory/' is relatively unambiguously saying that I
> > want a local install of some potentially-locally-modified files, and
> > while it might involve a temporary wheel internally there's no need to
> > expose this in any way (and e.g. it certainly shouldn't be cached), so
> > I think it's OK if this builds in-place and risks giving different
> > results than 'pip install sdist.tar.gz'. (Also note that in the most
> > common case where a naive user might use this accidentally, where
> > they've downloaded an sdist, unpacked it manually, and then run 'pip
> > install .', they *do* give the same results -- the potential for
> > trouble only comes when someone runs 'pip install .' multiple times in
> > the same directory.)
>
> I think that the key thing here is that as things stand, pip needs a
> means to copy an existing "source tree", as efficiently as possible.
> For local directories (source checkouts, typically) there's a lot of
> clutter that isn't needed to replicate the "source tree" aspect of the
> directory - but we can't reliably determine what is clutter and what
> isn't.
>
> Whether that copying is a good idea, in the face of the need to do
> incremental builds, is something of an open question - clearly we
> can't do something that closes the door on incremental builds, but
> equally the overhead of copying unwanted data is huge at the moment,
> and we can't ignore that.
>
> Talking about a "build a sdist" operation brings a whole load of
> questions about whether there should be a sdist format, what about
> sdist 2.0, etc into the mix. So maybe we should avoid all that, and
> say that pip[1] needs a "copy a source tree" operation. Backends
> SHOULD implement that by skipping any unneeded files in the source
> tree,but can fall back to a simple copy if they wish. In fact, we
> could make the operation optional and have *pip* fall back to copying
> if necessary. It would then be a backend quality of implementation
> issue if builds are slow because multi-megabyte git trees get copied
> unnecessarily.
>

Doesn't that just move the problem from pip to backends? It's still a
choice between:
(1) making no copy (good for in-place builds and also fine for pbr & co,
but needs education or a "pip release" type command)
(2) making a full copy like now including .git, .vagrant, etc. (super
inefficient)
(3) making an efficient copy (will likely still break pbr and
setuptools-scm, *and* break in-place builds)

(This operation might also help tools like setuptools-scm that need
> git information to work - the backend could extract that information
> on a "copy" operation and pit it somewhere static for the build).
>

If the backend can do it, so can pip right?

Ralf



>
> Paul
>
> [1] As pip is currently the only frontend, "what pip needs right now"
> is the only non-theoretical indication we have of what frontends might
> need to do, so we should be cautious about dismissing this as "pip
> shouldn't work like this", IMO.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170529/ce7dc561/attachment.html>

From thomas at kluyver.me.uk  Mon May 29 06:57:19 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Mon, 29 May 2017 11:57:19 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CACac1F8UvthfRgV=bPQTM+dTcnZvZYDTJ3CKo8=5Ke1J4L8jmQ@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>
 <CACac1F8UvthfRgV=bPQTM+dTcnZvZYDTJ3CKo8=5Ke1J4L8jmQ@mail.gmail.com>
Message-ID: <1496055439.3859148.991675096.1F85F451@webmail.messagingengine.com>

On Mon, May 29, 2017, at 10:19 AM, Paul Moore wrote:
> I think that the key thing here is that as things stand, pip needs a
> means to copy an existing "source tree", as efficiently as possible.
> For local directories (source checkouts, typically) there's a lot of
> clutter that isn't needed to replicate the "source tree" aspect of the
> directory - but we can't reliably determine what is clutter and what
> isn't.

This sounds similar to the question we had in flit about making sdists -
we don't want to reinvent MANIFEST.in, but we also don't want to include
random clutter that might be present in the source tree.

The compromise we've gone for in flit is to ask the VCS which files it
is tracking and only include those, on the grounds that it should be
possible to treat a fresh clone/checkout of the repository as the
source. But this probably wouldn't work for what pip wants, because
there are two big limitations:

1. Flit can only build an sdist from a VCS checkout
2. Flit needs to know about the VCS you're using

From p.f.moore at gmail.com  Mon May 29 08:05:32 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 29 May 2017 13:05:32 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CABL7CQjuPNCymVBDXbJmg6jJjaMShVJLTS22ApAr7Qs76Hta0g@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>
 <CACac1F8UvthfRgV=bPQTM+dTcnZvZYDTJ3CKo8=5Ke1J4L8jmQ@mail.gmail.com>
 <CABL7CQjuPNCymVBDXbJmg6jJjaMShVJLTS22ApAr7Qs76Hta0g@mail.gmail.com>
Message-ID: <CACac1F8MVAYwDzDHBJB7STYAUjZMuw4HcmFBbzN2LaB+7uOaaQ@mail.gmail.com>

On 29 May 2017 at 11:46, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>> Talking about a "build a sdist" operation brings a whole load of
>> questions about whether there should be a sdist format, what about
>> sdist 2.0, etc into the mix. So maybe we should avoid all that, and
>> say that pip[1] needs a "copy a source tree" operation. Backends
>> SHOULD implement that by skipping any unneeded files in the source
>> tree,but can fall back to a simple copy if they wish. In fact, we
>> could make the operation optional and have *pip* fall back to copying
>> if necessary. It would then be a backend quality of implementation
>> issue if builds are slow because multi-megabyte git trees get copied
>> unnecessarily.
>
>
> Doesn't that just move the problem from pip to backends? It's still a choice
> between:
> (1) making no copy (good for in-place builds and also fine for pbr & co, but
> needs education or a "pip release" type command)
> (2) making a full copy like now including .git, .vagrant, etc. (super
> inefficient)
> (3) making an efficient copy (will likely still break pbr and
> setuptools-scm, *and* break in-place builds)

Yes, that's precisely what it does. The point being that a backend is
the only code that can know what files are needed for the build, and
which ones can safely not be copied.

>
>> (This operation might also help tools like setuptools-scm that need
>> git information to work - the backend could extract that information
>> on a "copy" operation and pit it somewhere static for the build).
>
>
> If the backend can do it, so can pip right?

No (see above). pip has to make the safe "copy everything" assumption,
and that has a significant cost.

Paul

From donald at stufft.io  Mon May 29 10:26:14 2017
From: donald at stufft.io (Donald Stufft)
Date: Mon, 29 May 2017 10:26:14 -0400
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>
Message-ID: <17880636-D7D8-46B0-802D-925E9207CC2D@stufft.io>


> On May 29, 2017, at 3:05 AM, Nathaniel Smith <njs at pobox.com> wrote:
> 
> I think there's some pip bug somewhere discussing this, where Ralf
> Gommers and I point out that this is a complete showstopper for
> projects with complex and expensive builds (like scipy). If 'pip
> install .' is going to replace 'setup.py install', then it needs to
> support incremental builds, and the way
> setup.py-and-almost-every-other-build-tool do this currently is by
> reusing the working directory across builds.


Wouldn?t supporting incremental builds the way ccache does work just fine? Have a per build tool cache directory somewhere that stores cached build output for each individual file keyed off a hash or something? (For that matter, if someone wants incremental rebuilds, couldn?t they just *use* ccache as their CC?).

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170529/ed97ca39/attachment.html>

From donald at stufft.io  Mon May 29 10:32:22 2017
From: donald at stufft.io (Donald Stufft)
Date: Mon, 29 May 2017 10:32:22 -0400
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>
Message-ID: <3D5FAB07-0E44-4C83-B537-8469399F8388@stufft.io>


> On May 29, 2017, at 3:05 AM, Nathaniel Smith <njs at pobox.com> wrote:
> 
> {package-name}.dist-info might also be reasonable, both here and in
> actual installs...
> 
> In general get_wheel_metadata is an optimization for the backtracking
> resolver (that doesn't exist yet) to more efficiently handle cases
> where there are complex constraints and no built wheels (both of which
> we hope will be rare). Robert thinks it's very important, and he knows
> more about that bit of the code than I do, but if it becomes an issue
> we could even defer get_wheel_metadata to a future PEP.


I?d have to think about just {package-name}.dist-info. That would work in the install case (though that would require a whole other PEP so not super relevant to this PEP). The ultimate question is, when pip goes looking for this directory, does it know what it?s looking for or not. If it doesn?t know what it?s looking for, then it has to resort to looking for everything that matches a pattern, and if more than one thing matches said pattern, it has no idea which one it is supposed to use, and it just has to error out.

Perhaps either just using DIST-INFO today, and if we ever support multiple wheels from a single sdist expose a new API that can handle multiple directories and returns the name of said directories. The other option would be to modify get_wheel_data so it returns the name of the dist-info directory that was created, so pip knows what directory it needs to look for.

We?re hopefully going to get a resolver over this summer, we have a GSoC student who is working on just that.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170529/4e0a5672/attachment-0001.html>

From njs at pobox.com  Mon May 29 15:09:48 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 29 May 2017 12:09:48 -0700
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <17880636-D7D8-46B0-802D-925E9207CC2D@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>
 <17880636-D7D8-46B0-802D-925E9207CC2D@stufft.io>
Message-ID: <CAPJVwBk_oB_iknx97Zy+i1jm03DUJf=Zu1AsbovXZuNF+9QFHw@mail.gmail.com>

On Mon, May 29, 2017 at 7:26 AM, Donald Stufft <donald at stufft.io> wrote:
>
> On May 29, 2017, at 3:05 AM, Nathaniel Smith <njs at pobox.com> wrote:
>
> > I think there's some pip bug somewhere discussing this, where Ralf
> > Gommers and I point out that this is a complete showstopper for
> > projects with complex and expensive builds (like scipy). If 'pip
> > install .' is going to replace 'setup.py install', then it needs to
> > support incremental builds, and the way
> > setup.py-and-almost-every-other-build-tool do this currently is by
> > reusing the working directory across builds.
>
>
>
> Wouldn?t supporting incremental builds the way ccache does work just fine?
> Have a per build tool cache directory somewhere that stores cached build
> output for each individual file keyed off a hash or something? (For that
> matter, if someone wants incremental rebuilds, couldn?t they just *use*
> ccache as their CC?).

With a random numpy checkout on my laptop and a fully-primed ccache,
some wall-clock timings:

no-op incremental build (python setup.py build): 1.186 seconds

python setup.py sdist: 3.213 seconds
unpack resulting tarball: 0.136 seconds
python setup.py build in unpacked tree: 7.696 seconds

So ccache makes the sdist-and-build a mere 10x slower than an in-place
incremental build.

ccache is great, but it isn't magic. It can't make copying files
faster (notice we're already 3x slower before we even start
building!), it doesn't speed up linking, and you still need to spawn
all those processes and hash all that source code instead of just
making some stat() calls.

Also, this is on Linux. The numbers would look much worse on Windows,
given that it generally has much higher overhead for unpacking
tarballs and spawning lots of processes, and also given that ccache
doesn't support MSVC!

Also also, notice elsewhere in the thread where Thomas notes that flit
can't build an sdist from an unpacked sdist. It seems like 'pip
install unpacked-sdist/' is an important use case to support...

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From donald at stufft.io  Mon May 29 15:50:28 2017
From: donald at stufft.io (Donald Stufft)
Date: Mon, 29 May 2017 15:50:28 -0400
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CAPJVwBk_oB_iknx97Zy+i1jm03DUJf=Zu1AsbovXZuNF+9QFHw@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>
 <17880636-D7D8-46B0-802D-925E9207CC2D@stufft.io>
 <CAPJVwBk_oB_iknx97Zy+i1jm03DUJf=Zu1AsbovXZuNF+9QFHw@mail.gmail.com>
Message-ID: <1B9440C4-CE80-492A-B6E5-ADC89B67579E@stufft.io>


> On May 29, 2017, at 3:09 PM, Nathaniel Smith <njs at pobox.com> wrote:
> 
> On Mon, May 29, 2017 at 7:26 AM, Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
>> 
>> On May 29, 2017, at 3:05 AM, Nathaniel Smith <njs at pobox.com> wrote:
>> 
>>> I think there's some pip bug somewhere discussing this, where Ralf
>>> Gommers and I point out that this is a complete showstopper for
>>> projects with complex and expensive builds (like scipy). If 'pip
>>> install .' is going to replace 'setup.py install', then it needs to
>>> support incremental builds, and the way
>>> setup.py-and-almost-every-other-build-tool do this currently is by
>>> reusing the working directory across builds.
>> 
>> 
>> 
>> Wouldn?t supporting incremental builds the way ccache does work just fine?
>> Have a per build tool cache directory somewhere that stores cached build
>> output for each individual file keyed off a hash or something? (For that
>> matter, if someone wants incremental rebuilds, couldn?t they just *use*
>> ccache as their CC?).
> 
> With a random numpy checkout on my laptop and a fully-primed ccache,
> some wall-clock timings:
> 
> no-op incremental build (python setup.py build): 1.186 seconds
> 
> python setup.py sdist: 3.213 seconds
> unpack resulting tarball: 0.136 seconds
> python setup.py build in unpacked tree: 7.696 seconds
> 
> So ccache makes the sdist-and-build a mere 10x slower than an in-place
> incremental build.
> 
> ccache is great, but it isn't magic. It can't make copying files
> faster (notice we're already 3x slower before we even start
> building!), it doesn't speed up linking, and you still need to spawn
> all those processes and hash all that source code instead of just
> making some stat() calls.
> 
> Also, this is on Linux. The numbers would look much worse on Windows,
> given that it generally has much higher overhead for unpacking
> tarballs and spawning lots of processes, and also given that ccache
> doesn't support MSVC!


To be honest, I?m not hardly going to feel particularly bad if one of the most compilation heavy packages that exist takes a whole 10 seconds to install from a VCS checkout. Particularly when I assume that the build tool can be even smarter here than ccache is able to be to reduce the setup.py build step back down to the no-op incremental build case.

I mean, unless numpy is doing something different, the default distutils incremental build stuff is incredibly dumb, it just stores the build output in a directory (by default it?s located in ./build/) and compares the mtime of a list of source files with the mtime of the target file, and if the sources files are newer, it recompiles it. If you replace mtime with blake2 (or similar) then you can trivially support the exact same thing just storing the built target files in some user directory cache instead. Hell, we *might* even be able to preserve mtime (if we?re not already? we might be! But I?d need to dig into it) so literally the only thing that would need to change is instead of storing the built artifacts in ./build/ you store them in ~/.cache/my-cool-build-tool/{project-name}. Bonus points: this means you get incremental speeds even when building from a sdist from PyPI that doesn?t have wheels and hasn?t changed those files either.

I?m of the opinion that first you need to make it *correct*, then you can try to make it *fast*. It is my opinion that a installer that shits random debris into your current directory is not correct. It?s kind of silly that we have to have a ?random pip/distutils/setuptools? crap chunk of stuff to add to .gitignore to basically every Python package in existence. Nevermind the random stuff that doesn?t currently get written there, but will if we stop copying files out of the path and into a temporary location (I?m sure everyone wants a pip-egg-info directory in their current directory).

I?m also of the opinion that avoiding foot guns is more important than shooting for the fastest operation possible. I regularly (sometimes multiple times a week!, but often every week or two) see people tripping up on the fact that ``git clone ? && pip install .`` does something different than ``git clone ? && python setup.py sdist && pip install dist/*``. Files suddenly go missing and they have no idea why. If they?re lucky, they?ll figure out they need to modify some combination of package_data, data_files, and MANIFEST.in to make it work, if they?re not lucky they just sit there dumbfounded at it.


> 
> Also also, notice elsewhere in the thread where Thomas notes that flit
> can't build an sdist from an unpacked sdist. It seems like 'pip
> install unpacked-sdist/' is an important use case to support?
> 

If the build tool gives us a mechanism to determine if something is an unpacked sdist or not so we can fallback to just copying in that case, that is fine with me. The bad case is generally only going to be hit on VCS checkouts or other not sdist kinds of source trees. 


?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170529/c36f5617/attachment-0001.html>

From njs at pobox.com  Mon May 29 15:56:57 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 29 May 2017 12:56:57 -0700
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <1B9440C4-CE80-492A-B6E5-ADC89B67579E@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>
 <17880636-D7D8-46B0-802D-925E9207CC2D@stufft.io>
 <CAPJVwBk_oB_iknx97Zy+i1jm03DUJf=Zu1AsbovXZuNF+9QFHw@mail.gmail.com>
 <1B9440C4-CE80-492A-B6E5-ADC89B67579E@stufft.io>
Message-ID: <CAPJVwBnAbFFQHPMOOdbRz4-a4V48EP281CZYZtx6uSY+PrZ=Nw@mail.gmail.com>

On Mon, May 29, 2017 at 12:50 PM, Donald Stufft <donald at stufft.io> wrote:
>
> To be honest, I?m not hardly going to feel particularly bad if one of the
> most compilation heavy packages that exist takes a whole 10 seconds to
> install from a VCS checkout.

Rebuild latency is *really* important. People get really cranky at me
when I argue that we should get rid of "editable installs", which
create much greater problems for . I think I'm entitled to be cranky

> Particularly when I assume that the build tool
> can be even smarter here than ccache is able to be to reduce the setup.py
> build step back down to the no-op incremental build case.
>
> I mean, unless numpy is doing something different, the default distutils
> incremental build stuff is incredibly dumb, it just stores the build output
> in a directory (by default it?s located in ./build/) and compares the mtime
> of a list of source files with the mtime of the target file, and if the
> sources files are newer, it recompiles it. If you replace mtime with blake2
> (or similar) then you can trivially support the exact same thing just
> storing the built target files in some user directory cache instead.

> Hell,
> we *might* even be able to preserve mtime (if we?re not already? we might
> be! But I?d need to dig into it) so literally the only thing that would need
> to change is instead of storing the built artifacts in ./build/ you store
> them in ~/.cache/my-cool-build-tool/{project-name}. Bonus points: this means
> you get incremental speeds even when building from a sdist from PyPI that
> doesn?t have wheels and hasn?t changed those files either.
>
> I?m of the opinion that first you need to make it *correct*, then you can
> try to make it *fast*. It is my opinion that a installer that shits random
> debris into your current directory is not correct. It?s kind of silly that
> we have to have a ?random pip/distutils/setuptools? crap chunk of stuff to
> add to .gitignore to basically every Python package in existence. Nevermind
> the random stuff that doesn?t currently get written there, but will if we
> stop copying files out of the path and into a temporary location (I?m sure
> everyone wants a pip-egg-info directory in their current directory).
>
> I?m also of the opinion that avoiding foot guns is more important than
> shooting for the fastest operation possible. I regularly (sometimes multiple
> times a week!, but often every week or two) see people tripping up on the
> fact that ``git clone ? && pip install .`` does something different than
> ``git clone ? && python setup.py sdist && pip install dist/*``. Files
> suddenly go missing and they have no idea why. If they?re lucky, they?ll
> figure out they need to modify some combination of package_data, data_files,
> and MANIFEST.in to make it work, if they?re not lucky they just sit there
> dumbfounded at it.
>
>
>
> Also also, notice elsewhere in the thread where Thomas notes that flit
> can't build an sdist from an unpacked sdist. It seems like 'pip
> install unpacked-sdist/' is an important use case to support?
>
>
> If the build tool gives us a mechanism to determine if something is an
> unpacked sdist or not so we can fallback to just copying in that case, that
> is fine with me. The bad case is generally only going to be hit on VCS
> checkouts or other not sdist kinds of source trees.
>
>
> ?
> Donald Stufft
>
>
>



-- 
Nathaniel J. Smith -- https://vorpus.org

From p.f.moore at gmail.com  Mon May 29 16:01:07 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 29 May 2017 21:01:07 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CAPJVwBk_oB_iknx97Zy+i1jm03DUJf=Zu1AsbovXZuNF+9QFHw@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>
 <17880636-D7D8-46B0-802D-925E9207CC2D@stufft.io>
 <CAPJVwBk_oB_iknx97Zy+i1jm03DUJf=Zu1AsbovXZuNF+9QFHw@mail.gmail.com>
Message-ID: <CACac1F_xRr59an0JKz0y13GGXex8mWa6d09=Z3GVHf3wK4nQ1w@mail.gmail.com>

On 29 May 2017 at 20:09, Nathaniel Smith <njs at pobox.com> wrote:
> With a random numpy checkout on my laptop and a fully-primed ccache,
> some wall-clock timings:
>
> no-op incremental build (python setup.py build): 1.186 seconds
>
> python setup.py sdist: 3.213 seconds
> unpack resulting tarball: 0.136 seconds
> python setup.py build in unpacked tree: 7.696 seconds
>
> So ccache makes the sdist-and-build a mere 10x slower than an in-place
> incremental build.

These numbers are useful to know, but ignore the fact that the two
operations could give completely different results (because files in
the current directory that don't get put into the sdist could affect
the build). I have no particular opinion on whether "do an inplace
build" is something pip should offer (although it seems like something
a developer would need, and I don't see why "setup.py bdist_wheel"
followed by "pip install the_wheel" isn't sufficient in that case[1]).
But for end user installs, and for building wheels for distribution,
I'd argue that isolated builds are essential.

Paul

[1] Of course, this ignores the case of editable builds, but again, I
don't think that they should dictate the behaviour of the normal case.

From njs at pobox.com  Mon May 29 16:04:39 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 29 May 2017 13:04:39 -0700
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CAPJVwBnAbFFQHPMOOdbRz4-a4V48EP281CZYZtx6uSY+PrZ=Nw@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>
 <17880636-D7D8-46B0-802D-925E9207CC2D@stufft.io>
 <CAPJVwBk_oB_iknx97Zy+i1jm03DUJf=Zu1AsbovXZuNF+9QFHw@mail.gmail.com>
 <1B9440C4-CE80-492A-B6E5-ADC89B67579E@stufft.io>
 <CAPJVwBnAbFFQHPMOOdbRz4-a4V48EP281CZYZtx6uSY+PrZ=Nw@mail.gmail.com>
Message-ID: <CAPJVwBmi4qrfPB+8hws535rrN2TM6Q=vaEZ-E3bXSqoDMAnxHQ@mail.gmail.com>

Ugh, sorry, fat-fingered that. Actual reply below...

On Mon, May 29, 2017 at 12:56 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Mon, May 29, 2017 at 12:50 PM, Donald Stufft <donald at stufft.io> wrote:
>>
>> To be honest, I?m not hardly going to feel particularly bad if one of the
>> most compilation heavy packages that exist takes a whole 10 seconds to
>> install from a VCS checkout.

Rebuild latency is *really* important. People get really cranky at me
when I argue that we should get rid of "editable installs", which
create much greater problems for maintaining consistent environments,
and that's only saving like 1 second of latency. I think I'm entitled
to be cranky if your response is "well suck it up and maybe rewrite
all your build tools".

NumPy really isn't that compilation heavy either... it's all C, which
is pretty quick. SciPy is *much* slower, for example, as is pretty
much any project using C++.

>> Particularly when I assume that the build tool
>> can be even smarter here than ccache is able to be to reduce the setup.py
>> build step back down to the no-op incremental build case.
>>
>> I mean, unless numpy is doing something different, the default distutils
>> incremental build stuff is incredibly dumb, it just stores the build output
>> in a directory (by default it?s located in ./build/) and compares the mtime
>> of a list of source files with the mtime of the target file, and if the
>> sources files are newer, it recompiles it. If you replace mtime with blake2
>> (or similar) then you can trivially support the exact same thing just
>> storing the built target files in some user directory cache instead.

Cache management is not a trivial problem.

And it actually doesn't matter, because we definitely can't silently
dump stuff into some user directory. An important feature of storing
temporary artifacts in the source tree is that it means that if
someone downloads the source, plays around with it a bit, and deletes
it, then it's actually gone. We can't squirrel away a few hundred
megabytes of data in some hidden directory that will hang around for
years after the user stops using numpy.

>> Hell,
>> we *might* even be able to preserve mtime (if we?re not already? we might
>> be! But I?d need to dig into it) so literally the only thing that would need
>> to change is instead of storing the built artifacts in ./build/ you store
>> them in ~/.cache/my-cool-build-tool/{project-name}. Bonus points: this means
>> you get incremental speeds even when building from a sdist from PyPI that
>> doesn?t have wheels and hasn?t changed those files either.
>>
>> I?m of the opinion that first you need to make it *correct*, then you can
>> try to make it *fast*. It is my opinion that a installer that shits random
>> debris into your current directory is not correct. It?s kind of silly that
>> we have to have a ?random pip/distutils/setuptools? crap chunk of stuff to
>> add to .gitignore to basically every Python package in existence. Nevermind
>> the random stuff that doesn?t currently get written there, but will if we
>> stop copying files out of the path and into a temporary location (I?m sure
>> everyone wants a pip-egg-info directory in their current directory).
>>
>> I?m also of the opinion that avoiding foot guns is more important than
>> shooting for the fastest operation possible. I regularly (sometimes multiple
>> times a week!, but often every week or two) see people tripping up on the
>> fact that ``git clone ? && pip install .`` does something different than
>> ``git clone ? && python setup.py sdist && pip install dist/*``. Files
>> suddenly go missing and they have no idea why. If they?re lucky, they?ll
>> figure out they need to modify some combination of package_data, data_files,
>> and MANIFEST.in to make it work, if they?re not lucky they just sit there
>> dumbfounded at it.

Yeah, setuptools is kinda sucky this way. But this is fixable with
better build systems. And before we can get better build systems, we
need buy-in from devs. And saying "sorry, we're unilaterally screwing
up your recompile times because we don't care" is not a good way to
get there :-(

>>
>>
>> Also also, notice elsewhere in the thread where Thomas notes that flit
>> can't build an sdist from an unpacked sdist. It seems like 'pip
>> install unpacked-sdist/' is an important use case to support?
>>
>>
>> If the build tool gives us a mechanism to determine if something is an
>> unpacked sdist or not so we can fallback to just copying in that case, that
>> is fine with me. The bad case is generally only going to be hit on VCS
>> checkouts or other not sdist kinds of source trees.

I guess numpy could just claim that all VCS checkouts are actually
unpacked sdists...?

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From p.f.moore at gmail.com  Mon May 29 16:13:53 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 29 May 2017 21:13:53 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CAPJVwBmi4qrfPB+8hws535rrN2TM6Q=vaEZ-E3bXSqoDMAnxHQ@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>
 <17880636-D7D8-46B0-802D-925E9207CC2D@stufft.io>
 <CAPJVwBk_oB_iknx97Zy+i1jm03DUJf=Zu1AsbovXZuNF+9QFHw@mail.gmail.com>
 <1B9440C4-CE80-492A-B6E5-ADC89B67579E@stufft.io>
 <CAPJVwBnAbFFQHPMOOdbRz4-a4V48EP281CZYZtx6uSY+PrZ=Nw@mail.gmail.com>
 <CAPJVwBmi4qrfPB+8hws535rrN2TM6Q=vaEZ-E3bXSqoDMAnxHQ@mail.gmail.com>
Message-ID: <CACac1F_NiG42Rrgq4vrAhD=BmgJ50zu=TC-VkLNgFVVKXkToPw@mail.gmail.com>

On 29 May 2017 at 21:04, Nathaniel Smith <njs at pobox.com> wrote:
>
> I guess numpy could just claim that all VCS checkouts are actually
> unpacked sdists...?

Well, my proposal was to allow backends to decide what needs to be
copied. So yes, you could do that under what I suggested (with the
co-operation of the build backend). Legacy setuptools builds wouldn't
provide a "this is what to copy" API, so we'd fall back to copy
everything (or maybe "build sdist", but we could add a get-out clause
in one of the config files, that said we're not allowed to do that on
a per-project basis).

Paul

From donald at stufft.io  Mon May 29 16:48:30 2017
From: donald at stufft.io (Donald Stufft)
Date: Mon, 29 May 2017 16:48:30 -0400
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CAPJVwBmi4qrfPB+8hws535rrN2TM6Q=vaEZ-E3bXSqoDMAnxHQ@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>
 <17880636-D7D8-46B0-802D-925E9207CC2D@stufft.io>
 <CAPJVwBk_oB_iknx97Zy+i1jm03DUJf=Zu1AsbovXZuNF+9QFHw@mail.gmail.com>
 <1B9440C4-CE80-492A-B6E5-ADC89B67579E@stufft.io>
 <CAPJVwBnAbFFQHPMOOdbRz4-a4V48EP281CZYZtx6uSY+PrZ=Nw@mail.gmail.com>
 <CAPJVwBmi4qrfPB+8hws535rrN2TM6Q=vaEZ-E3bXSqoDMAnxHQ@mail.gmail.com>
Message-ID: <70D7A8ED-143A-4B06-A1CC-4181F678CF50@stufft.io>


> On May 29, 2017, at 4:04 PM, Nathaniel Smith <njs at pobox.com> wrote:
> 
> Ugh, sorry, fat-fingered that. Actual reply below...
> 
> On Mon, May 29, 2017 at 12:56 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> On Mon, May 29, 2017 at 12:50 PM, Donald Stufft <donald at stufft.io> wrote:
>>> 
>>> To be honest, I?m not hardly going to feel particularly bad if one of the
>>> most compilation heavy packages that exist takes a whole 10 seconds to
>>> install from a VCS checkout.
> 
> Rebuild latency is *really* important. People get really cranky at me
> when I argue that we should get rid of "editable installs", which
> create much greater problems for maintaining consistent environments,
> and that's only saving like 1 second of latency. I think I'm entitled
> to be cranky if your response is "well suck it up and maybe rewrite
> all your build tools?.

Well, distutils literally already has support for storing the ?cache" someplace other than the current directory, the current directory is just the default. So ?rewrite all your build tools? is fairly hyperbolic, it?s really just ?change the default of your build tools?. See for example: https://gist.github.com/dstufft/a577c3c9d54a3bb3b88e9b20ba86c625 <https://gist.github.com/dstufft/a577c3c9d54a3bb3b88e9b20ba86c625> which shows that Numpy &tc are already capable of this.

Hell, the build backend could create an unpacked sdist in the target directory instead of an actual sdist that is already packed into a tarball, tools like twine could add a ``twine sdist`` command that just called the ?create unpacked sdist? API and then just tar?d up the directory into the sdist. A quick rudimentary test on my machine (using ``python setup.py sdist ?formats=`` in a numpy checkout [1]) suggests that this entire process takes ~0.7s which the copy operation on that same check out (shutil.copytree) also takes ~0.7s. That also eliminates the need to untar so unless someone is doing something in their sdist creation step that takes a significant amount of time, generating an unpacked sdist is really not any more time consuming than copying the files.

> 
> NumPy really isn't that compilation heavy either... it's all C, which
> is pretty quick. SciPy is *much* slower, for example, as is pretty
> much any project using C++.
> 
>>> Particularly when I assume that the build tool
>>> can be even smarter here than ccache is able to be to reduce the setup.py
>>> build step back down to the no-op incremental build case.
>>> 
>>> I mean, unless numpy is doing something different, the default distutils
>>> incremental build stuff is incredibly dumb, it just stores the build output
>>> in a directory (by default it?s located in ./build/) and compares the mtime
>>> of a list of source files with the mtime of the target file, and if the
>>> sources files are newer, it recompiles it. If you replace mtime with blake2
>>> (or similar) then you can trivially support the exact same thing just
>>> storing the built target files in some user directory cache instead.
> 
> Cache management is not a trivial problem.
> 
> And it actually doesn't matter, because we definitely can't silently
> dump stuff into some user directory. An important feature of storing
> temporary artifacts in the source tree is that it means that if
> someone downloads the source, plays around with it a bit, and deletes
> it, then it's actually gone. We can't squirrel away a few hundred
> megabytes of data in some hidden directory that will hang around for
> years after the user stops using numpy.


I mean, you absolutely can do that. We store temporary wheels and HTTP responses silently in pip and have for years. I don?t think *anyone* has *ever* complained about it. I think macOS even explicitly will clean up stuff from ~/Library/Caches when it hasn?t been used in awhile. If you use the standard cache locations for Linux then IIRC similar systems exist for Linux too. In exchange for ?I can delete the directory and it?s just all gone?, you get ?faster builds in more scenarios, including straight from PyPI?s sdists?. If I were a user I?d care a lot more about the second then the first.

But even if I grant you that you can?t just do that silently, then go ahead and make it opt in. For people who need it, a simple boolean in a config file seems to be pretty low cost to me.


Combine this user cache with generating an unpacked sdist instead of copying the directory tree, and you get:

1) Safety from weirdness that comes from ``pip install`` from a sdist versus a VCS.
2) Not crapping up ./ with random debris from the installation process.
3) Fast incremental builds that even help speed up installs from PyPI etc (assuming we use something like blake2 to compute hashes for the files).

And you lose:

1) Deleting a clone doesn?t delete the cache directory, but your OS might already be managing this directory anyways.

Seems like an obvious trade off to me.


> 
>>> Hell,
>>> we *might* even be able to preserve mtime (if we?re not already? we might
>>> be! But I?d need to dig into it) so literally the only thing that would need
>>> to change is instead of storing the built artifacts in ./build/ you store
>>> them in ~/.cache/my-cool-build-tool/{project-name}. Bonus points: this means
>>> you get incremental speeds even when building from a sdist from PyPI that
>>> doesn?t have wheels and hasn?t changed those files either.
>>> 
>>> I?m of the opinion that first you need to make it *correct*, then you can
>>> try to make it *fast*. It is my opinion that a installer that shits random
>>> debris into your current directory is not correct. It?s kind of silly that
>>> we have to have a ?random pip/distutils/setuptools? crap chunk of stuff to
>>> add to .gitignore to basically every Python package in existence. Nevermind
>>> the random stuff that doesn?t currently get written there, but will if we
>>> stop copying files out of the path and into a temporary location (I?m sure
>>> everyone wants a pip-egg-info directory in their current directory).
>>> 
>>> I?m also of the opinion that avoiding foot guns is more important than
>>> shooting for the fastest operation possible. I regularly (sometimes multiple
>>> times a week!, but often every week or two) see people tripping up on the
>>> fact that ``git clone ? && pip install .`` does something different than
>>> ``git clone ? && python setup.py sdist && pip install dist/*``. Files
>>> suddenly go missing and they have no idea why. If they?re lucky, they?ll
>>> figure out they need to modify some combination of package_data, data_files,
>>> and MANIFEST.in to make it work, if they?re not lucky they just sit there
>>> dumbfounded at it.
> 
> Yeah, setuptools is kinda sucky this way. But this is fixable with
> better build systems. And before we can get better build systems, we
> need buy-in from devs. And saying "sorry, we're unilaterally screwing
> up your recompile times because we don't care" is not a good way to
> get there :-(


I don?t think it has anything to do with setuptools TBH other than the fact that it?s interface for declaring what does and doesn?t go into a sdist is kind of crummy. This problem is going to exist as long as you have any mechanism for having some files not be included inside of a sdist.


> 
>>> 
>>> 
>>> Also also, notice elsewhere in the thread where Thomas notes that flit
>>> can't build an sdist from an unpacked sdist. It seems like 'pip
>>> install unpacked-sdist/' is an important use case to support?
>>> 
>>> 
>>> If the build tool gives us a mechanism to determine if something is an
>>> unpacked sdist or not so we can fallback to just copying in that case, that
>>> is fine with me. The bad case is generally only going to be hit on VCS
>>> checkouts or other not sdist kinds of source trees.
> 
> I guess numpy could just claim that all VCS checkouts are actually
> unpacked sdists??

I mean, ``pip install .`` is still going to ``cp -r`` that VCS checkout into a temporary location if you do that, and making sure that the invariant of ``python setup.py build && pip install .`` doesn?t trigger a recompile isn?t going to be something that I would want pip to start doing. So would it _work_ for this use case? Possibly? Is it supported? Nope, if it breaks you get to keep both pieces.


?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170529/5bf423c7/attachment.html>

From donald at stufft.io  Mon May 29 16:50:09 2017
From: donald at stufft.io (Donald Stufft)
Date: Mon, 29 May 2017 16:50:09 -0400
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <70D7A8ED-143A-4B06-A1CC-4181F678CF50@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>
 <17880636-D7D8-46B0-802D-925E9207CC2D@stufft.io>
 <CAPJVwBk_oB_iknx97Zy+i1jm03DUJf=Zu1AsbovXZuNF+9QFHw@mail.gmail.com>
 <1B9440C4-CE80-492A-B6E5-ADC89B67579E@stufft.io>
 <CAPJVwBnAbFFQHPMOOdbRz4-a4V48EP281CZYZtx6uSY+PrZ=Nw@mail.gmail.com>
 <CAPJVwBmi4qrfPB+8hws535rrN2TM6Q=vaEZ-E3bXSqoDMAnxHQ@mail.gmail.com>
 <70D7A8ED-143A-4B06-A1CC-4181F678CF50@stufft.io>
Message-ID: <5527FC15-7747-435E-A6B3-0B42BC6B407B@stufft.io>


> On May 29, 2017, at 4:48 PM, Donald Stufft <donald at stufft.io> wrote:
> 
> A quick rudimentary test on my machine (using ``python setup.py sdist ?formats=`` in a numpy checkout [1]) suggests that this entire process takes ~0.7s which the copy operation on that same check out (shutil.copytree) also takes ~0.7s.


I forgot to add the [1] here, but it?s basically that `python setup.py sdist ?formats=` actually errors out, because an empty str is not a valid format, but it errors out *after* distutils has prepped an unpacked directory and the only things left are (A) taring up that unpacked directory into the desired formats and (B) deleting that unpacked directory. Neither of which we want to do in this case.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170529/ee8da8e8/attachment-0001.html>

From ncoghlan at gmail.com  Tue May 30 01:36:00 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 30 May 2017 15:36:00 +1000
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
Message-ID: <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>

On 29 May 2017 at 15:37, Donald Stufft <donald at stufft.io> wrote:
>
> On May 28, 2017, at 11:40 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> Hi folks,
>
> I am hereby provisionally accepting PEP 517 as our CLI independent
> interface for build backends, allowing interested parties to move
> forward with implementing it as part of pip and other tools.
>
> As with other provisionally accepted PyPA interoperability
> specifications, it won't be considered completely settled until it has
> been implemented and released in pip and we have at least one
> iteration of end user feedback.
>
> Thanks to Nathaniel & Thomas for their work on the PEP, and to
> everyone that participated in the related discussions!
>
>
>
> Bleh, I had it on my stack to respond to PEP 517, but life has been super
> hectic so I hadn?t gotten around to it.
>
> Here are a few thoughts FWIW:
>
> 1. Using the {package-name}-{package-version}.dist-info in the
> get_wheel_metadata() metadata is a mistake I think. In pip currently we have
> a bug we have not yet been able to track down because there is nothing
> systematically preventing both foobar-1.0.dist-info and foobar-2.0.distinfo
> from existing side by side in a build directory (or inside a wheel for that
> matter). Thus I think this naming scheme is a nuisance and we shouldn?t
> propagate it any further. I would just use something like DIST-INFO/ which
> will completely side step this issue. The only reason I can think of to use
> the current scheme is to make it easier to shutil.copytree it into the
> wheel, but handling that case is trivial.

I think this is an ambiguity in the current spec, since the name of
the metadata directory is wholly controlled by the front-end, but I
now see that "This directory MUST be a valid .dist-info directory as
defined in the wheel specification, except that it need not contain
RECORD or signatures." can be taken as suggesting it is required to
follow the directory naming scheme and hence backends can parse it
accordingly to find out the name of the wheel being built.

That's not the way I actually read it when accepting the PEP, so I
think it should be clarified to say:

* the *name* of the directory is arbitrary, and entirely up to the
build frontend. Build backends MUST NOT attempt to parse it for
information.
* the *contents* of the directory, as produced by the build backend,
MUST be as defined in the wheel specification (only omitting RECORD
and signatures)

If we want to later introduce an RPM style "multiple wheels from a
single sdist" model, that should be its own PEP, with its own
additions to pyproject.toml to define the available alternate wheel
files (so that tools like pyp2rpm and conda skeleton can read it and
adjust their output accordingly) and to the build backend API (to
request building an alternate wheel rather than the default one that's
named after the sdist).

> 2. As I mentioned in previous discussions on this, I think that this
> interface *needs* some mechanism to ask it to build a sdist. Ideally this
> would either fail or recreate the sdist when being run from something that
> is already an sdist. In pip when we?re doing ``pip install .`` we currently
> copy tree the entire thing into a temporary directory, and we run the build
> out of there. For a variety of reasons we?re going to keep build isolation,
> but the current mechanism is painful because it also grabs things like .tox,
> .git, etc. At one point we tried to simply filter those out but it broke
> some packages expectations. The planned (which is on my long list of things
> to do?) mechanism to fix this is to create a sdist from a ``.`` and then
> install from that (building an intermediate wheel as well). This also solves
> another class of bugs that people run into where ``pip install .`` and
> ``python setup.py sdist && pip install dist/*`` give different results. As
> written, this PEP prevents that from happening (and thus, when I implement
> it, I?ll only be able to implement it for old style sdists, and will need to
> tell people to continue to use old style if they don?t want pip to grab
> random directories from ``.``).

As with multiplexed wheel creation, I think "give me a filtered source
tree to use as an out-of-tree build root" can be a separate follow-up
PEP (I'll even propose a name for the API:
"export_sdist_source_tree").

My rationale for seeing it that way, is that while PEP 517 requires
that the current working directory correspond to the root of the
source tree, it *doesn't* require that the source tree be the
unfiltered contents of a VCS checkout. It can't, since the source tree
might have come from an sdist, and MANIFEST.in and friends already
allow that to be a filtered subset of the full VCS contents.

And when it comes to *publishing* an sdist, folks are free to call
their build backend directly for that, rather than necessarily going
through pip to do it.

That said, I *do* think it makes sense for pip to offer an option to
use out-of-tree builds when running `pip install` (even if it isn't
the default), and that means extending the build backend API to cover
creating an exported sdist tree directly from a VCS checkout or other
local directory without necessarily making the sdist first.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From chris.jerdonek at gmail.com  Tue May 30 02:22:51 2017
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Mon, 29 May 2017 23:22:51 -0700
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CAPJVwBmi4qrfPB+8hws535rrN2TM6Q=vaEZ-E3bXSqoDMAnxHQ@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>
 <17880636-D7D8-46B0-802D-925E9207CC2D@stufft.io>
 <CAPJVwBk_oB_iknx97Zy+i1jm03DUJf=Zu1AsbovXZuNF+9QFHw@mail.gmail.com>
 <1B9440C4-CE80-492A-B6E5-ADC89B67579E@stufft.io>
 <CAPJVwBnAbFFQHPMOOdbRz4-a4V48EP281CZYZtx6uSY+PrZ=Nw@mail.gmail.com>
 <CAPJVwBmi4qrfPB+8hws535rrN2TM6Q=vaEZ-E3bXSqoDMAnxHQ@mail.gmail.com>
Message-ID: <CAOTb1wf_LazV6jWX=T-Kqe29pYuujQbuhM-_k1YLFHreA659-w@mail.gmail.com>

On Mon, May 29, 2017 at 1:04 PM, Nathaniel Smith <njs at pobox.com> wrote:
> Ugh, sorry, fat-fingered that. Actual reply below...
>
> On Mon, May 29, 2017 at 12:56 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> On Mon, May 29, 2017 at 12:50 PM, Donald Stufft <donald at stufft.io> wrote:
>>>
>>> To be honest, I?m not hardly going to feel particularly bad if one of the
>>> most compilation heavy packages that exist takes a whole 10 seconds to
>>> install from a VCS checkout.
>
> Rebuild latency is *really* important. People get really cranky at me
> when I argue that we should get rid of "editable installs", which
> create much greater problems for maintaining consistent environments,
> and that's only saving like 1 second of latency.

I don't recall this being discussed. Is support for editable installs
being considered for removal? Either way, what is the argument out of
curiosity?

--Chris

From donald at stufft.io  Tue May 30 03:07:52 2017
From: donald at stufft.io (Donald Stufft)
Date: Tue, 30 May 2017 03:07:52 -0400
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
Message-ID: <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>

It can require that it is either unfiltered or an unpacked sdist since that is how a lot of build time projects treat it now. They handle a sdist differently from a vcs source, for example pbr. Swapping out a call to setup.py for an internal shim that calls a Python API doesn't change anything here, randomly filtering out some files from a vcs will break random projects. We ether only suppprt copying the whole directory or we add support for something that enables this in the PEP. There is basically no middle ground that isn't worse than one of those two options for PEP 517 style packages. 

I also don't think that creating an sdist should be an optional part of the build interface, but things added in later PEPs can only be added as optional, not mandatory. There is already automation that relies on handling sdist- for example the Travis deployment to PyPI code path- that will be unable to support this new standard without either ONLY supporting wheels, or needing to add custom code for each individual build tool (unlikely to happen). The effect of which will be that either people simply can't use this spec without it or we incentivize releasing only wheels to PyPI instead of wheels and a sdist. 

I don't see anyway really for this PEP to move forward without support for sdists without causing major regressions. 

Sent from my iPhone

> On May 30, 2017, at 1:36 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> My rationale for seeing it that way, is that while PEP 517 requires
> that the current working directory correspond to the root of the
> source tree, it *doesn't* require that the source tree be the
> unfiltered contents of a VCS checkout. It can't, since the source tree
> might have come from an sdist, and MANIFEST.in and friends already
> allow that to be a filtered subset of the full VCS contents.


From freddyrietdijk at fridh.nl  Tue May 30 05:27:31 2017
From: freddyrietdijk at fridh.nl (Freddy Rietdijk)
Date: Tue, 30 May 2017 11:27:31 +0200
Subject: [Distutils] Introducing PyPIContents
In-Reply-To: <CAHagVwYw_eoQL7DtdVX1yQLZuNzzkMwn1Ji3dpi_hKMvP3j=ew@mail.gmail.com>
References: <CAHagVwYw_eoQL7DtdVX1yQLZuNzzkMwn1Ji3dpi_hKMvP3j=ew@mail.gmail.com>
Message-ID: <CAOQtOH39u9vR3FYMPVBUTL7N12dBB0-87EtkE4xu-VKucCQXgQ@mail.gmail.com>

This looks very nice. The readme states that monkeypatching is used to
extract this info from the `setup()` call. Is `setup.cfg` also taken into
account?

What would it take to split of this part into a separate module/package so
that we have one function taking a source directory and returning the
contents of that call? I would be very interested in extracting not just
available modules but also dependencies.

On Sat, May 20, 2017 at 4:01 AM, Luis Alejandro Mart?nez Faneyth <
luis at huntingbears.com.ve> wrote:

> Hi everyone,
>
> I'm new to this list but I've been reading some threads in the archive.
>
> Around february, an idea about indexing modules from PyPI packages was
> brought up. I've been working on something similar for quite a while.
>
> PyPIContents is an index of PyPI packages that lists its modules and
> command line scripts in JSON format, like this:
>
>
> [
> ...
>
>     "1337": {
>         "cmdline": [],
>         "modules": [
>             "1337",
>             "1337.1337"
>         ],
>         "version": "1.0.0"
>     },
>
> ...
>
> ]
>
>
> You can check it out here:
>
> https://github.com/LuisAlejandro/pypicontents
>
> And some use cases:
>
> https://github.com/LuisAlejandro/pypicontents#use-cases
>
> The actual index lives here, its around 60MB:
>
> https://raw.githubusercontent.com/LuisAlejandro/
> pypicontents/contents/pypi.json
>
> Is updated daily with the help of Travis:
>
> https://github.com/LuisAlejandro/pypicontents/blob/contents/.travis.yml
>
> Anyway, I hope is useful and I'll be around for any comments or questions.
>
> Cheers!
>
>
> Luis Alejandro Mart?nez Faneyth
> Blog: http://huntingbears.com.ve
> Github: http://github.com/LuisAlejandro
> Twitter: http://twitter.com/LuisAlejandro
>
> CODE IS POETRY
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170530/ed7ac079/attachment-0001.html>

From ncoghlan at gmail.com  Tue May 30 06:34:08 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 30 May 2017 20:34:08 +1000
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
Message-ID: <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>

On 30 May 2017 at 17:07, Donald Stufft <donald at stufft.io> wrote:
> It can require that it is either unfiltered or an unpacked sdist since that is how a lot of build time projects treat it now. They handle a sdist differently from a vcs source, for example pbr. Swapping out a call to setup.py for an internal shim that calls a Python API doesn't change anything here, randomly filtering out some files from a vcs will break random projects. We ether only suppprt copying the whole directory or we add support for something that enables this in the PEP. There is basically no middle ground that isn't worse than one of those two options for PEP 517 style packages.
>
> I also don't think that creating an sdist should be an optional part of the build interface, but things added in later PEPs can only be added as optional, not mandatory. There is already automation that relies on handling sdist- for example the Travis deployment to PyPI code path- that will be unable to support this new standard without either ONLY supporting wheels, or needing to add custom code for each individual build tool (unlikely to happen). The effect of which will be that either people simply can't use this spec without it or we incentivize releasing only wheels to PyPI instead of wheels and a sdist.
>
> I don't see anyway really for this PEP to move forward without support for sdists without causing major regressions.

Is your concern that there's no explicit statement in the PEP saying
that build backends MUST NOT assume they will have access to version
control metadata directories in the source tree, since that source
tree may have come from an sdist rather than VCS checkout?

Aside from that possibility, I otherwise don't follow this chain of
reasoning, as I don't see how PEP 517 has any impact whatsoever on the
sdist build step.

Status quo:

- pip (et al) can use setup.py to build from an unfiltered source tree
- pip (et al) can use setup.py to build from an sdist
- creation of the sdist is up to the software publisher, so if pip or
another frontend wants to do an out of tree build, it copies the
entire unfiltered tree

Post PEP 517:

- pip (et al) can use setup.py to build from an unfiltered source tree
- pip (et al) can use setup.py to build from an sdist
- pip (et al) can use the pyproject.toml build-backend setting to
build from an unfiltered source tree
- pip (et al) can use the pyproject.toml build-backend setting to
build from an sdist
- creation of the sdist is still up to the software publisher, so if
pip or another frontend wants to do an out of tree build, it still
copies the entire unfiltered tree

Post a to-be-written sdist-source-tree-export PEP:

- pip (et al) can use setup.py to build from an unfiltered source tree
- pip (et al) can use setup.py to build from an sdist
- pip (et al) can use the pyproject.toml build-backend setting to
build from an unfiltered source tree
- pip (et al) can use the pyproject.toml build-backend setting to
build from an sdist
- pip (et al) can use a new pyproject.toml setting defined in that PEP
("source-filter" perhaps?) to export a filtered version of a source
tree, otherwise they fall back on copying the entire unfiltered tree
(similar to the way build-backend falls back to setuptools & wheel if
not otherwise specified)

That approach would decouple the export backends from the build
backends, so we might even end up with a common VCS-aware source
exporter that projects could nominate (e.g. by adding this
functionality to twine), rather than every build backend having to
define its own source export logic.

Note that I'm also fine with pip as a project saying that it will only
ship support for the build-backend interface once the source filtering
interface is also defined and implemented.

I'm just saying that I don't see a close enough link between "here is
how to build this component from source" and "here is how to export a
filtered source tree for this component from an unfiltered VCS
checkout" for it to make sense to define them as part of the same
backend API. The only connection I'm aware of is that it makes sense
for projects to ensure that their source filtering when creating an
sdist isn't leaving out any files needed by their build process, but
that's no different from making sure that your build process produces
a wheel file that actually works when installed.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From donald at stufft.io  Tue May 30 07:37:03 2017
From: donald at stufft.io (Donald Stufft)
Date: Tue, 30 May 2017 07:37:03 -0400
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
Message-ID: <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>


> On May 30, 2017, at 6:34 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 30 May 2017 at 17:07, Donald Stufft <donald at stufft.io> wrote:
>> It can require that it is either unfiltered or an unpacked sdist since that is how a lot of build time projects treat it now. They handle a sdist differently from a vcs source, for example pbr. Swapping out a call to setup.py for an internal shim that calls a Python API doesn't change anything here, randomly filtering out some files from a vcs will break random projects. We ether only suppprt copying the whole directory or we add support for something that enables this in the PEP. There is basically no middle ground that isn't worse than one of those two options for PEP 517 style packages.
>> 
>> I also don't think that creating an sdist should be an optional part of the build interface, but things added in later PEPs can only be added as optional, not mandatory. There is already automation that relies on handling sdist- for example the Travis deployment to PyPI code path- that will be unable to support this new standard without either ONLY supporting wheels, or needing to add custom code for each individual build tool (unlikely to happen). The effect of which will be that either people simply can't use this spec without it or we incentivize releasing only wheels to PyPI instead of wheels and a sdist.
>> 
>> I don't see anyway really for this PEP to move forward without support for sdists without causing major regressions.
> 
> Is your concern that there's no explicit statement in the PEP saying
> that build backends MUST NOT assume they will have access to version
> control metadata directories in the source tree, since that source
> tree may have come from an sdist rather than VCS checkout?
> 
> Aside from that possibility, I otherwise don't follow this chain of
> reasoning, as I don't see how PEP 517 has any impact whatsoever on the
> sdist build step.

My concern is that it?s literally impossible for the most common tooling outside of setuptools that acts at the build stage to function if we just randomly start filtering out files. For example let?s take a look at setuptools_scm, it assumes that one of two cases are true:

1) I am in a VCS checkout where I can run ``git`` commands to compute my version number, as well as what files should be added to the sdist, installed, etc.
2) I am in a sdist where the above information was ?baked? into the sdist at sdist creation time and thus no longer requires access to the .git/ directory.

Those are the only two situations where it works. The ?bad? case for performance reasons comes from the fact that a VCS checkout often times has A LOT of files that don?t need to be copied over typically, but that we do because when we attempted to filter things out previously it broke. These files can make the entire ``pip install .`` take over a minute on a slow hard drive. One of the biggest offenders is .tox/, but another big offender is .git. Another common offender is large chunks of demo data that doesn?t get added to the sdist.

> 
> Status quo:
> 
> - pip (et al) can use setup.py to build from an unfiltered source tree
> - pip (et al) can use setup.py to build from an sdist
> - creation of the sdist is up to the software publisher, so if pip or
> another frontend wants to do an out of tree build, it copies the
> entire unfiltered tree
> 
> Post PEP 517:
> 
> - pip (et al) can use setup.py to build from an unfiltered source tree
> - pip (et al) can use setup.py to build from an sdist
> - pip (et al) can use the pyproject.toml build-backend setting to
> build from an unfiltered source tree
> - pip (et al) can use the pyproject.toml build-backend setting to
> build from an sdist
> - creation of the sdist is still up to the software publisher, so if
> pip or another frontend wants to do an out of tree build, it still
> copies the entire unfiltered tree


Status quo is also that Travis CI, Gem Fury, etc can produce and upload a sdist using ``setup.py sdist``. Pip is not the only consumer of setup.py that needs to be able to operate on a VCS and do things with it. Ignoring this just means that we solve the problem of standardizing access for pip?s current use case, and tell these other use cases to go pound sand.


> 
> Post a to-be-written sdist-source-tree-export PEP:
> 
> - pip (et al) can use setup.py to build from an unfiltered source tree
> - pip (et al) can use setup.py to build from an sdist
> - pip (et al) can use the pyproject.toml build-backend setting to
> build from an unfiltered source tree
> - pip (et al) can use the pyproject.toml build-backend setting to
> build from an sdist
> - pip (et al) can use a new pyproject.toml setting defined in that PEP
> ("source-filter" perhaps?) to export a filtered version of a source
> tree, otherwise they fall back on copying the entire unfiltered tree
> (similar to the way build-backend falls back to setuptools & wheel if
> not otherwise specified)
> 
> That approach would decouple the export backends from the build
> backends, so we might even end up with a common VCS-aware source
> exporter that projects could nominate (e.g. by adding this
> functionality to twine), rather than every build backend having to
> define its own source export logic.
> 
> Note that I'm also fine with pip as a project saying that it will only
> ship support for the build-backend interface once the source filtering
> interface is also defined and implemented.
> 
> I'm just saying that I don't see a close enough link between "here is
> how to build this component from source" and "here is how to export a
> filtered source tree for this component from an unfiltered VCS
> checkout" for it to make sense to define them as part of the same
> backend API. The only connection I'm aware of is that it makes sense
> for projects to ensure that their source filtering when creating an
> sdist isn't leaving out any files needed by their build process, but
> that's no different from making sure that your build process produces
> a wheel file that actually works when installed.
> 

I don?t think there is any value in decoupling the generation of what goes into an sdist from the tool that builds them. If we did that, I suspect that 100% of the time the exact same tool is going to be used to handle both anyways (or people just won?t bother to put in the extra effort to produce sdists). I think trying to split them up serves only to make the entire toolchain harder and more complicated for people who aren?t stepped in packaging lore to figure out what goes where and what does what. The fact that we have different mechanisms just to control what goes into a sdist (MANIFEST.in) vs what gets installed (package_data) already confuses people, further splitting these two steps apart is only going to make that worse.

Keeping the two together also completely sidesteps the problems around ?well what if only the sdist tool is defined but the build tool isn?t?? Or ?what if only the build tool is defined but the sdist tool isn?t??.

The only value I can even think of, is that some of the code is going to be re-usable, but we already have a perfectly serviceable way of allowing code re-use: publish a library and have end tools consume it. 


?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170530/3c7a4a8b/attachment-0001.html>

From donald at stufft.io  Tue May 30 07:40:13 2017
From: donald at stufft.io (Donald Stufft)
Date: Tue, 30 May 2017 07:40:13 -0400
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
Message-ID: <4F4194BD-7963-48A9-9136-7013EA82440B@stufft.io>


> On May 30, 2017, at 6:34 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> Note that I'm also fine with pip as a project saying that it will only
> ship support for the build-backend interface once the source filtering
> interface is also defined and implemented.


As an addendum, this seems entirely silly to me as well. The packaging landscape is already littered with unimplemented PEPs that confuse people, adding another one seems like a waste of everyone?s time, and if this one is dependent on another one, it seems silly not to define them together.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170530/ba6e2d65/attachment.html>

From donald at stufft.io  Tue May 30 10:57:50 2017
From: donald at stufft.io (Donald Stufft)
Date: Tue, 30 May 2017 10:57:50 -0400
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
Message-ID: <3934203D-8475-4165-988C-53423CB62AB6@stufft.io>


> On May 30, 2017, at 7:37 AM, Donald Stufft <donald at stufft.io> wrote:
> 
>> 
>> 
>> Post a to-be-written sdist-source-tree-export PEP:
>> 
>> - pip (et al) can use setup.py to build from an unfiltered source tree
>> - pip (et al) can use setup.py to build from an sdist
>> - pip (et al) can use the pyproject.toml build-backend setting to
>> build from an unfiltered source tree
>> - pip (et al) can use the pyproject.toml build-backend setting to
>> build from an sdist
>> - pip (et al) can use a new pyproject.toml setting defined in that PEP
>> ("source-filter" perhaps?) to export a filtered version of a source
>> tree, otherwise they fall back on copying the entire unfiltered tree
>> (similar to the way build-backend falls back to setuptools & wheel if
>> not otherwise specified)
>> 
>> That approach would decouple the export backends from the build
>> backends, so we might even end up with a common VCS-aware source
>> exporter that projects could nominate (e.g. by adding this
>> functionality to twine), rather than every build backend having to
>> define its own source export logic.
>> 
>> Note that I'm also fine with pip as a project saying that it will only
>> ship support for the build-backend interface once the source filtering
>> interface is also defined and implemented.
>> 
>> I'm just saying that I don't see a close enough link between "here is
>> how to build this component from source" and "here is how to export a
>> filtered source tree for this component from an unfiltered VCS
>> checkout" for it to make sense to define them as part of the same
>> backend API. The only connection I'm aware of is that it makes sense
>> for projects to ensure that their source filtering when creating an
>> sdist isn't leaving out any files needed by their build process, but
>> that's no different from making sure that your build process produces
>> a wheel file that actually works when installed.
>> 
> 
> I don?t think there is any value in decoupling the generation of what goes into an sdist from the tool that builds them. If we did that, I suspect that 100% of the time the exact same tool is going to be used to handle both anyways (or people just won?t bother to put in the extra effort to produce sdists). I think trying to split them up serves only to make the entire toolchain harder and more complicated for people who aren?t stepped in packaging lore to figure out what goes where and what does what. The fact that we have different mechanisms just to control what goes into a sdist (MANIFEST.in) vs what gets installed (package_data) already confuses people, further splitting these two steps apart is only going to make that worse.
> 
> Keeping the two together also completely sidesteps the problems around ?well what if only the sdist tool is defined but the build tool isn?t?? Or ?what if only the build tool is defined but the sdist tool isn?t??.
> 
> The only value I can even think of, is that some of the code is going to be re-usable, but we already have a perfectly serviceable way of allowing code re-use: publish a library and have end tools consume it. 
> 



I think my other thing here is that I don?t even think as written you *can* separate the two concepts. The reason being they?re both going to need the same metadata (like name, version, etc). If you want to have two separate tools handling then, that the build wheel step should accept *only* an sdist (unpacked or otherwise) as the ?input? to this build tool and it should not accept a VCS input at all. The wheel build tool can then read the metadata like name, version, etc from whatever metadata exists in the hypothetical sdist format. Otherwise you?ll need two different to both implement the same scheme for determining version (since you?d be able to run both options from a VCS checkout).

Now I?m perfectly fine mandating that builds go VCS -> sdist -> wheel -> install [1], but that?s not what the current PEP does and making it do that would be significantly more effort since you?d have to spend more time hammering out what metadata is ?sdist metadata? and what metadata is ?Wheel metadata? and I really don?t think that particular rabbit hole is an effective use of our time.

[1] Indeed I think this being the only path (besides of course editable) is a overall good thing and I?d support it, but even if we did I still would think that separate tools for this is a waste of time which can also be added after the fact with a simple shim package if people do end up finding it very useful.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170530/63e0a61a/attachment-0001.html>

From thomas at kluyver.me.uk  Tue May 30 11:36:53 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Tue, 30 May 2017 16:36:53 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <3934203D-8475-4165-988C-53423CB62AB6@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <3934203D-8475-4165-988C-53423CB62AB6@stufft.io>
Message-ID: <1496158613.2017782.992977864.39E1269A@webmail.messagingengine.com>

I'm struggling to understand the objections here. As I understand the
PEP, the input to building a wheel is a source tree. That may come
from an unpacked sdist or a VCS checkout; hopefully those contain
basically the same files, give or take some unimportant generated
files in an sdist.
This seems to work for building wheels with setup.py (as pip already
does), and it's not a problem for flit. So why does pip need to know how
to make an sdist?
I understand the concern about copying many unnecessary files if the
source tree is copied into a build directory, though I'm not entirely
sure why we'd need to do that. I don't think the source tree is often so
big as to be problematic, though. My clone of the IPython repository,
with many years of active git history and a build of the Sphinx docs, is
still under 50 MB. Most repos I work with are much smaller. If we define
optional mechanisms to filter which files are copied, a fallback to
copying the whole directory sounds acceptable.
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170530/eddef298/attachment.html>

From donald at stufft.io  Tue May 30 12:17:28 2017
From: donald at stufft.io (Donald Stufft)
Date: Tue, 30 May 2017 12:17:28 -0400
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <1496158613.2017782.992977864.39E1269A@webmail.messagingengine.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <3934203D-8475-4165-988C-53423CB62AB6@stufft.io>
 <1496158613.2017782.992977864.39E1269A@webmail.messagingengine.com>
Message-ID: <0F7EBE84-64D3-480B-B31F-2D66E4404CFB@stufft.io>


> On May 30, 2017, at 11:36 AM, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> 
> I'm struggling to understand the objections here. As I understand the PEP, the input to building a wheel is a source tree. That may come from an unpacked sdist or a VCS checkout; hopefully those contain basically the same files, give or take some unimportant generated files in an sdist.

I?m struggling to understand the objection to adding a mechanism for creating an unpacked sdist. Presumably all of the build tools are planning on supporting sdist creation, so the only real additional effort is to expose that interface which should be fairly minimal. The only argument I can think of against adding support for generating sdists is if the build tools don?t plan on implementing sdist creation, in which case I personally have no interest in supporting that build tool.

They quite often do *not* contain the same files, like the example I keep going back to which is a .git directory that can be used to compute the version in a VCS checkout, which no longer exists in a sdist (and instead a pre-baked version number exists). Another example is pyca/cryptography (https://github.com/pyca/cryptography <https://github.com/pyca/cryptography>) where the root of the package actually contains a whole other package inside of it (under the vectors directory) which does not exist in the sdist. The behavior of the setup.py changes based on whether the vectors directory exists or not (e.g. inside a tarball or a VCS directory).

> 
> This seems to work for building wheels with setup.py (as pip already does), and it's not a problem for flit. So why does pip need to know how to make an sdist?

Because it?s the most reasonable way to not hit fairly large slow downs on big repositories. You can see:

https://github.com/pypa/pip/issues/2195 <https://github.com/pypa/pip/issues/2195>

https://github.com/pypa/pip/pull/2196 <https://github.com/pypa/pip/pull/2196>

https://github.com/pypa/pip/pull/2535 <https://github.com/pypa/pip/pull/2535> (the last comment on that one is nice? current behavior is taking over 5 minutes for pip to copy the data)

https://github.com/pypa/pip/pull/3176 <https://github.com/pypa/pip/pull/3176>

https://github.com/pypa/pip/pull/3615 <https://github.com/pypa/pip/pull/3615>

This is a long standing issue with pip that people hit with semi regularity? refusing to fix it is user hostile. Personally I don?t really have much interest in seeing something land in pip that prevents fixing issues that we?re currently seeing? the other pip devs may disagree with me, but as it stands I would be -1 on implementing this PEP as it stands without additional work (either in a stand alone PEP, or as part of this PEP, though I prefer as part of this PEP).

It?s also not just about pip, as I?ve mentioned there are other tooling that is relying on the ability to create a sdist, and refusing to accommodate them is something I am not thrilled about us doing. If every build tool is going to implement it?s own command to build a sdist, how is something like TravisCI supposed to handle building sdists automatically on push as they do currently? How is gem fury supposed to continue to build packages to host when you ``git push fury``? ?Sorry you don?t get to or you have to handle every build tool directly? is not a answer I can support.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170530/de470f35/attachment.html>

From thomas at kluyver.me.uk  Tue May 30 12:29:02 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Tue, 30 May 2017 17:29:02 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <0F7EBE84-64D3-480B-B31F-2D66E4404CFB@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <3934203D-8475-4165-988C-53423CB62AB6@stufft.io>
 <1496158613.2017782.992977864.39E1269A@webmail.messagingengine.com>
 <0F7EBE84-64D3-480B-B31F-2D66E4404CFB@stufft.io>
Message-ID: <1496161742.2030540.993054856.2EE7EF69@webmail.messagingengine.com>

What about saying that the copying step, if necessary, is part of the
build backend's responsibilities? I.e. pip doesn't copy the whole
directory to a temporary build location, but the build backend may
decide to do that at its discretion when it's asked to build a wheel.
pip would continue to handle this for setup.py builds.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170530/1e4467ec/attachment.html>

From donald at stufft.io  Tue May 30 12:43:48 2017
From: donald at stufft.io (Donald Stufft)
Date: Tue, 30 May 2017 12:43:48 -0400
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <1496161742.2030540.993054856.2EE7EF69@webmail.messagingengine.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <3934203D-8475-4165-988C-53423CB62AB6@stufft.io>
 <1496158613.2017782.992977864.39E1269A@webmail.messagingengine.com>
 <0F7EBE84-64D3-480B-B31F-2D66E4404CFB@stufft.io>
 <1496161742.2030540.993054856.2EE7EF69@webmail.messagingengine.com>
Message-ID: <C9C32AAD-8768-49A1-9506-F999F2343E8D@stufft.io>


> On May 30, 2017, at 12:29 PM, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> 
> What about saying that the copying step, if necessary, is part of the build backend's responsibilities? I.e. pip doesn't copy the whole directory to a temporary build location, but the build backend may decide to do that at its discretion when it's asked to build a wheel. pip would continue to handle this for setup.py builds.


That still leaves the other use cases for building sdists unsatisfied. In addition it?ll likely be pip that gets the bug reports when some backend inevitably doesn?t copy those files and then leaves random debris laying about (including things like read only file systems where random crap *can?t* be written to the ``.`` directory or mounting the same package in multiple docker containers that would cause different things to be pooped into the ``.`` directory).

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170530/d0fc8aaf/attachment-0001.html>

From brett at python.org  Tue May 30 14:17:36 2017
From: brett at python.org (Brett Cannon)
Date: Tue, 30 May 2017 18:17:36 +0000
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <C9C32AAD-8768-49A1-9506-F999F2343E8D@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <3934203D-8475-4165-988C-53423CB62AB6@stufft.io>
 <1496158613.2017782.992977864.39E1269A@webmail.messagingengine.com>
 <0F7EBE84-64D3-480B-B31F-2D66E4404CFB@stufft.io>
 <1496161742.2030540.993054856.2EE7EF69@webmail.messagingengine.com>
 <C9C32AAD-8768-49A1-9506-F999F2343E8D@stufft.io>
Message-ID: <CAP1=2W4BcKw_yQZS_ZEKgA1j1bjcabfQJy_hu844gA7s=-14iQ@mail.gmail.com>

Just to make sure I'm following this correctly, Donald is asking for:

   - A way for pip to ask back-ends what files should be in an sdist from a
   source checkout or to make an actual sdist
      - Because sdists are a thing and so we should support them properly
      - To make it so building wheels interact with just the files from an
      sdist instead of skipping that and going straight from a source checkout
   - Have this be a part of PEP 517 or at least a requirement for back-ends
   to support so it doesn't get left out

Am I missing anything?

On Tue, 30 May 2017 at 09:48 Donald Stufft <donald at stufft.io> wrote:

>
> On May 30, 2017, at 12:29 PM, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
>
> What about saying that the copying step, if necessary, is part of the
> build backend's responsibilities? I.e. pip doesn't copy the whole directory
> to a temporary build location, but the build backend may decide to do that
> at its discretion when it's asked to build a wheel. pip would continue to
> handle this for setup.py builds.
>
>
>
> That still leaves the other use cases for building sdists unsatisfied. In
> addition it?ll likely be pip that gets the bug reports when some backend
> inevitably doesn?t copy those files and then leaves random debris laying
> about (including things like read only file systems where random crap
> *can?t* be written to the ``.`` directory or mounting the same package in
> multiple docker containers that would cause different things to be pooped
> into the ``.`` directory).
>
> ?
>
> Donald Stufft
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170530/42d0635b/attachment.html>

From p.f.moore at gmail.com  Tue May 30 14:23:38 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 30 May 2017 19:23:38 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <0F7EBE84-64D3-480B-B31F-2D66E4404CFB@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <3934203D-8475-4165-988C-53423CB62AB6@stufft.io>
 <1496158613.2017782.992977864.39E1269A@webmail.messagingengine.com>
 <0F7EBE84-64D3-480B-B31F-2D66E4404CFB@stufft.io>
Message-ID: <CACac1F95LPda7C+VFhfY+eaWV4XSKuBvo_-1hqJs_we91cbbJQ@mail.gmail.com>

On 30 May 2017 at 17:17, Donald Stufft <donald at stufft.io> wrote:
> This is a long standing issue with pip that people hit with semi regularity?
> refusing to fix it is user hostile. Personally I don?t really have much
> interest in seeing something land in pip that prevents fixing issues that
> we?re currently seeing? the other pip devs may disagree with me, but as it
> stands I would be -1 on implementing this PEP as it stands without
> additional work (either in a stand alone PEP, or as part of this PEP, though
> I prefer as part of this PEP).

Just to chime in as "another pip developer" I agree that this is
something we need to solve, and I'm -1 on anything that makes doing so
harder.

I assume that no-one is trying to insist that pip shouldn't do the
build in a temporary directory? That's existing pip behaviour and
writing a PEP that doesn't support it isn't going to get very far. So
the question is, what does the PEP need to do? Not saying anything
means that pip can't implement the PEP without abandoning any hope of
improving copy times (at least until a follow-up PEP is agreed). So
PEP 517 languishes unused until we resolve that issue and write that
follow-up PEP. That seems pointless. Surely it's better to cover the
issue now.

Maybe all we need to do is to make it the backend's job, and say "the
backend MUST copy the source tree to an isolated temporary directory
when building". But as Donald says, that means that tools to build
sdists have to replicate that logic[1]. So surely it's better to
factor out the "define the set of files needed for a build" into a
build backend API that both tools can use?

Paul

[1] There's an implied assumption here that we need tools to build
sdists - either backends do it themselves, or something else does. I
think both Donald and I take that as a given, because if you don't
have source bundles (call them that if you don't like the term sdist)
on PyPI, then people whose systems don't match the supplied wheels are
out of luck. Also, from a policy point of view, I'd be bothered by
PyPI being used to distribute binary only packages - we're an open
source community, after all.

From p.f.moore at gmail.com  Tue May 30 14:27:23 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 30 May 2017 19:27:23 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CAP1=2W4BcKw_yQZS_ZEKgA1j1bjcabfQJy_hu844gA7s=-14iQ@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <3934203D-8475-4165-988C-53423CB62AB6@stufft.io>
 <1496158613.2017782.992977864.39E1269A@webmail.messagingengine.com>
 <0F7EBE84-64D3-480B-B31F-2D66E4404CFB@stufft.io>
 <1496161742.2030540.993054856.2EE7EF69@webmail.messagingengine.com>
 <C9C32AAD-8768-49A1-9506-F999F2343E8D@stufft.io>
 <CAP1=2W4BcKw_yQZS_ZEKgA1j1bjcabfQJy_hu844gA7s=-14iQ@mail.gmail.com>
Message-ID: <CACac1F-sQaupFMgTi+Dbky=X0KGAcnJbg6PExH1Ci=2m73rtjQ@mail.gmail.com>

On 30 May 2017 at 19:17, Brett Cannon <brett at python.org> wrote:
> Just to make sure I'm following this correctly, Donald is asking for:
>
> A way for pip to ask back-ends what files should be in an sdist from a
> source checkout or to make an actual sdist
>
> Because sdists are a thing and so we should support them properly
> To make it so building wheels interact with just the files from an sdist
> instead of skipping that and going straight from a source checkout
>
> Have this be a part of PEP 517 or at least a requirement for back-ends to
> support so it doesn't get left out
>
> Am I missing anything?

That's it basically. The only nuance is that there's a debate over the
idea of a "sdist" - the term carries some baggage in terms of implying
a need for a standard, static metadata, etc.

In this context, all we need is the list of files to include, so
backends that don't want to buy into the idea of a "sdist" format yet
(IIRC, flit may be in this position) can just say "these are the
files" and not worry about the whole question of a standard sdist
format.

Paul

From donald at stufft.io  Tue May 30 14:40:03 2017
From: donald at stufft.io (Donald Stufft)
Date: Tue, 30 May 2017 14:40:03 -0400
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CAP1=2W4BcKw_yQZS_ZEKgA1j1bjcabfQJy_hu844gA7s=-14iQ@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <3934203D-8475-4165-988C-53423CB62AB6@stufft.io>
 <1496158613.2017782.992977864.39E1269A@webmail.messagingengine.com>
 <0F7EBE84-64D3-480B-B31F-2D66E4404CFB@stufft.io>
 <1496161742.2030540.993054856.2EE7EF69@webmail.messagingengine.com>
 <C9C32AAD-8768-49A1-9506-F999F2343E8D@stufft.io>
 <CAP1=2W4BcKw_yQZS_ZEKgA1j1bjcabfQJy_hu844gA7s=-14iQ@mail.gmail.com>
Message-ID: <CD5A1BEC-E50F-404D-BE0F-E7C9F1627CAC@stufft.io>


> On May 30, 2017, at 2:17 PM, Brett Cannon <brett at python.org> wrote:
> 
> Just to make sure I'm following this correctly, Donald is asking for:
> A way for pip to ask back-ends what files should be in an sdist from a source checkout or to make an actual sdist
> Because sdists are a thing and so we should support them properly
> To make it so building wheels interact with just the files from an sdist instead of skipping that and going straight from a source checkout
> Have this be a part of PEP 517 or at least a requirement for back-ends to support so it doesn't get left out
> Am I missing anything?
> 


More or less. If I had my druthers we?d just add another mandatory method to the API, something like:

    def build_sdist_tree(sdist_directory, config_settings) -> None:
        ?

All this does is assemble the would be sdist tree into sdist_directory, handling any sdist creation time steps WITHOUT actually tar+gz?ing up this tree. My tests show that this is basically as fast as the copytree option in the simple case (which makes sense, that?s basically all it is) and is way faster than actually assembling the entire sdist tarball and all.

To handle creation, we can either have ``twine sdist`` or something like that which just calls the above API and then runs ``shutil.make_archive()`` on the final directory.

When building a wheel, pip can then just inside of this directory, call into the wheel metadata/wheel build API and ultimately install that wheel. It may even make sense for the build_wheel API to produce an ?unpacked wheel? as well and again let something like ``twine wheel`` handle running ``shutil.make_archive()`` on it. Similar to the sdist case, this would further reduce the install time by avoiding the need to zip and then immediately unzip inside of pip.

One unrelated thing I just noticed, I don?t think the PEP states how pip is supposed to communicate to this API what it?s actually building. Is it just assumed to be ``os.curdir()``? Ideally I think we?d add another parameter to all of these functions that is the directory containing the ?input? to each of these.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170530/37ec3fe3/attachment-0001.html>

From brett at python.org  Tue May 30 19:58:54 2017
From: brett at python.org (Brett Cannon)
Date: Tue, 30 May 2017 23:58:54 +0000
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CD5A1BEC-E50F-404D-BE0F-E7C9F1627CAC@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <3934203D-8475-4165-988C-53423CB62AB6@stufft.io>
 <1496158613.2017782.992977864.39E1269A@webmail.messagingengine.com>
 <0F7EBE84-64D3-480B-B31F-2D66E4404CFB@stufft.io>
 <1496161742.2030540.993054856.2EE7EF69@webmail.messagingengine.com>
 <C9C32AAD-8768-49A1-9506-F999F2343E8D@stufft.io>
 <CAP1=2W4BcKw_yQZS_ZEKgA1j1bjcabfQJy_hu844gA7s=-14iQ@mail.gmail.com>
 <CD5A1BEC-E50F-404D-BE0F-E7C9F1627CAC@stufft.io>
Message-ID: <CAP1=2W59bMj6wh3Atn4+hNN7unAvb7nJzxdmLvUpYCce=2-XnQ@mail.gmail.com>

On Tue, 30 May 2017 at 11:40 Donald Stufft <donald at stufft.io> wrote:

>
> On May 30, 2017, at 2:17 PM, Brett Cannon <brett at python.org> wrote:
>
> Just to make sure I'm following this correctly, Donald is asking for:
>
>    - A way for pip to ask back-ends what files should be in an sdist from
>    a source checkout or to make an actual sdist
>       - Because sdists are a thing and so we should support them properly
>       - To make it so building wheels interact with just the files from
>       an sdist instead of skipping that and going straight from a source checkout
>    - Have this be a part of PEP 517 or at least a requirement for
>    back-ends to support so it doesn't get left out
>
> Am I missing anything?
>
>
>
> More or less. If I had my druthers we?d just add another mandatory method
> to the API, something like:
>
>     def build_sdist_tree(sdist_directory, config_settings) -> None:
>         ?
>
> All this does is assemble the would be sdist tree into sdist_directory,
> handling any sdist creation time steps WITHOUT actually tar+gz?ing up this
> tree. My tests show that this is basically as fast as the copytree option
> in the simple case (which makes sense, that?s basically all it is) and is
> way faster than actually assembling the entire sdist tarball and all.
>

So the back-ends then need to provide a way for users to specify what files
to go into the sdist (e.g. the MANIFEST.in case). Obviously a fallback is
to just copy everything or just what is used to build the wheel (overly
board or overly selective, respectively), but I doubt tools will want to do
either in general and so they will need to come up with some solution to
specify extra files like READMEs and stuff.


>
> To handle creation, we can either have ``twine sdist`` or something like
> that which just calls the above API and then runs ``shutil.make_archive()``
> on the final directory.
>

Do we want to say that sdist creation is the front-end's job? If so that is
different from PEP 517 where the back-end packages zip up the wheel versus
simply providing all the data and files that make up a wheel as seems to be
suggested in the sdist case here.


>
> When building a wheel, pip can then just inside of this directory, call
> into the wheel metadata/wheel build API and ultimately install that wheel.
> It may even make sense for the build_wheel API to produce an ?unpacked
> wheel? as well and again let something like ``twine wheel`` handle running
> ``shutil.make_archive()`` on it. Similar to the sdist case, this would
> further reduce the install time by avoiding the need to zip and then
> immediately unzip inside of pip.
>

If we're going to go down a route where back-ends simply provide the files
and the front-ends do the packaging then it makes sense to do that for
wheels as well in PEP 517, separating the back-ends to create the
components of the final artifact while the front-ends handle the bundling
up for all of it into its final form. This makes the back-ends purely a
compiler-and-metadata thing and the front-ends a manage-final-files thing.

-Brett


>
> One unrelated thing I just noticed, I don?t think the PEP states how pip
> is supposed to communicate to this API what it?s actually building. Is it
> just assumed to be ``os.curdir()``? Ideally I think we?d add another
> parameter to all of these functions that is the directory containing the
> ?input? to each of these.
>
> ?
>
> Donald Stufft
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170530/101f06ed/attachment.html>

From ncoghlan at gmail.com  Wed May 31 04:17:55 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 31 May 2017 18:17:55 +1000
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
Message-ID: <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>

On 30 May 2017 at 21:37, Donald Stufft <donald at stufft.io> wrote:
> I don?t think there is any value in decoupling the generation of what goes
> into an sdist from the tool that builds them. If we did that, I suspect that
> 100% of the time the exact same tool is going to be used to handle both
> anyways (or people just won?t bother to put in the extra effort to produce
> sdists). I think trying to split them up serves only to make the entire
> toolchain harder and more complicated for people who aren?t stepped in
> packaging lore to figure out what goes where and what does what. The fact
> that we have different mechanisms just to control what goes into a sdist
> (MANIFEST.in) vs what gets installed (package_data) already confuses people,
> further splitting these two steps apart is only going to make that worse.
>
> Keeping the two together also completely sidesteps the problems around ?well
> what if only the sdist tool is defined but the build tool isn?t?? Or ?what
> if only the build tool is defined but the sdist tool isn?t??.

I don't have a strong opinion either way, so I'd also be fine if the
second PEP defined a new optional method for build backends to
implement rather than a new project.toml setting.

So I'd say go ahead and write a new PEP that depends on PEP 517, and
defines the source tree export API you'd like build backends to
provide.

PEP 517 is already fairly complex just in covering the build step, and
trying to also explain the considerations involved in handling the
sdist source export step would likely make it unreadable.

The only constraints I'd place on that proposal up front are:

- it needs to be consistent with the accepted PEP 517 interface
- if the build backend doesn't provide the source tree export method,
then the new PEP should require frontends to fall back to the current
"copy everything" behaviour

Beyond that, since you know what you're looking for, and neither
Thomas nor I fully understand that yet, it makes far more sense you to
write it, and for us to review it as a separate PEP, rather than
trying to incorporate both the additional proposal and its rationale
into PEP 517.

Then one PR to pip can implement support for both PEPs, and everyone
will be happy.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed May 31 04:24:49 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 31 May 2017 18:24:49 +1000
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CAP1=2W4BcKw_yQZS_ZEKgA1j1bjcabfQJy_hu844gA7s=-14iQ@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <3934203D-8475-4165-988C-53423CB62AB6@stufft.io>
 <1496158613.2017782.992977864.39E1269A@webmail.messagingengine.com>
 <0F7EBE84-64D3-480B-B31F-2D66E4404CFB@stufft.io>
 <1496161742.2030540.993054856.2EE7EF69@webmail.messagingengine.com>
 <C9C32AAD-8768-49A1-9506-F999F2343E8D@stufft.io>
 <CAP1=2W4BcKw_yQZS_ZEKgA1j1bjcabfQJy_hu844gA7s=-14iQ@mail.gmail.com>
Message-ID: <CADiSq7fdh5vPqHYyap+0-eZRO8_CZM4bjbHs7D2aZp7Lz_zTvw@mail.gmail.com>

On 31 May 2017 at 04:17, Brett Cannon <brett at python.org> wrote:
> Just to make sure I'm following this correctly, Donald is asking for:
>
> A way for pip to ask back-ends what files should be in an sdist from a
> source checkout or to make an actual sdist
>
> Because sdists are a thing and so we should support them properly
> To make it so building wheels interact with just the files from an sdist
> instead of skipping that and going straight from a source checkout
>
> Have this be a part of PEP 517 or at least a requirement for back-ends to
> support so it doesn't get left out
>
> Am I missing anything?

Yes: there are apparently additional design considerations here that
Thomas and I *aren't aware of* because they relate to internal details
of how pip works that we've never personally needed to worry about.

Rather than trying to wedge the explanation of those considerations
and their consequences into the existing structure of PEP 517 (which I
consider to be a clear and complete discussion of a particular problem
and an appropriate solution), I'd prefer to see them laid out clearly
in their own PEP that describes the status quo of how a pip driven
build works in practice, how PEP 517 alters that status quo, and the
necessary enhancements to the build backend API needed to deal with it
effectively.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From donald at stufft.io  Wed May 31 07:56:23 2017
From: donald at stufft.io (Donald Stufft)
Date: Wed, 31 May 2017 07:56:23 -0400
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
Message-ID: <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>


> On May 31, 2017, at 4:17 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 30 May 2017 at 21:37, Donald Stufft <donald at stufft.io> wrote:
>> I don?t think there is any value in decoupling the generation of what goes
>> into an sdist from the tool that builds them. If we did that, I suspect that
>> 100% of the time the exact same tool is going to be used to handle both
>> anyways (or people just won?t bother to put in the extra effort to produce
>> sdists). I think trying to split them up serves only to make the entire
>> toolchain harder and more complicated for people who aren?t stepped in
>> packaging lore to figure out what goes where and what does what. The fact
>> that we have different mechanisms just to control what goes into a sdist
>> (MANIFEST.in) vs what gets installed (package_data) already confuses people,
>> further splitting these two steps apart is only going to make that worse.
>> 
>> Keeping the two together also completely sidesteps the problems around ?well
>> what if only the sdist tool is defined but the build tool isn?t?? Or ?what
>> if only the build tool is defined but the sdist tool isn?t??.
> 
> I don't have a strong opinion either way, so I'd also be fine if the
> second PEP defined a new optional method for build backends to
> implement rather than a new project.toml setting.
> 
> So I'd say go ahead and write a new PEP that depends on PEP 517, and
> defines the source tree export API you'd like build backends to
> provide.
> 
> PEP 517 is already fairly complex just in covering the build step, and
> trying to also explain the considerations involved in handling the
> sdist source export step would likely make it unreadable.
> 
> The only constraints I'd place on that proposal up front are:
> 
> - it needs to be consistent with the accepted PEP 517 interface
> - if the build backend doesn't provide the source tree export method,
> then the new PEP should require frontends to fall back to the current
> "copy everything" behaviour
> 
> Beyond that, since you know what you're looking for, and neither
> Thomas nor I fully understand that yet, it makes far more sense you to
> write it, and for us to review it as a separate PEP, rather than
> trying to incorporate both the additional proposal and its rationale
> into PEP 517.
> 
> Then one PR to pip can implement support for both PEPs, and everyone
> will be happy.
> 

I don?t think an *optional* interface is the correct way to implement this. I?m not entirely sure why folks seem to be ignoring the fact that this is not just for pip, but well, this isn?t just for pip. There are a wide range of popular ecosystem projects depending on the ability to produce a sdist from a given VCS tree, and the current proposal leaves those projects out in the cold.

I also think that the current interface needs some tweaks, some to make it clearer what is being operated on, and some to reduce installation time (basically to avoid a round trip through zip needlessly).

Given that, instead of a separate PEP I?ve just implemented the changes I want to see as a PR which is quite short, it?s a +55 -22 diff and can be viewed at https://github.com/python/peps/pull/275 <https://github.com/python/peps/pull/275> .


?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170531/275730f4/attachment.html>

From anee.vesna at gmail.com  Tue May 30 19:24:15 2017
From: anee.vesna at gmail.com (Aneesh Kona)
Date: Tue, 30 May 2017 18:24:15 -0500
Subject: [Distutils] Installation problem
Message-ID: <CADPEtBY1no0iSRwwuhtji=u+1ayjzjY0p9riGfa7aRLzZ59fHQ@mail.gmail.com>

Hi,
I'm getting an error while installing software,
I did installed firstly but did in another in local disk,
but I want to install again, now it shows 'fatal error,' can you please let
me fix it, so that I can install again in C disk,
plz hand me in this particular,
Thanks.

Venkata Aneesh Kona
Graduate Assistant
Department of Mining Engineering and Management
South Dakota School of Mines & Technology
@: venkataaneesh.kona at mines.sdsmt.edu
ph: 6823042622
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170530/5c7e95fe/attachment.html>

From ncoghlan at gmail.com  Wed May 31 09:24:11 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 31 May 2017 23:24:11 +1000
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
Message-ID: <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>

On 31 May 2017 at 21:56, Donald Stufft <donald at stufft.io> wrote:
> I don?t think an *optional* interface is the correct way to implement this.
> I?m not entirely sure why folks seem to be ignoring the fact that this is
> not just for pip, but well, this isn?t just for pip. There are a wide range
> of popular ecosystem projects depending on the ability to produce a sdist
> from a given VCS tree, and the current proposal leaves those projects out in
> the cold.
>
> I also think that the current interface needs some tweaks, some to make it
> clearer what is being operated on, and some to reduce installation time
> (basically to avoid a round trip through zip needlessly).
>
> Given that, instead of a separate PEP I?ve just implemented the changes I
> want to see as a PR which is quite short, it?s a +55 -22 diff and can be
> viewed at https://github.com/python/peps/pull/275 .

Cool, thanks for doing that - I'll let you thrash out the details with
Thomas as the PEP author, and then take a look once you're in
agreement on the amendments you want to make.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From thomas at kluyver.me.uk  Wed May 31 09:39:05 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Wed, 31 May 2017 14:39:05 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
Message-ID: <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>

On Wed, May 31, 2017, at 02:24 PM, Nick Coghlan wrote:
> Cool, thanks for doing that - I'll let you thrash out the details with
> Thomas as the PEP author, and then take a look once you're in
> agreement on the amendments you want to make.

I've had a quick look over the PR, and my main thought is that if we're
going to specify a 'build sdist' step, we need to specify more precisely
what an sdist actually means. The PEP as currently written expects a
'source tree' as the input for building a wheel, but does not really
need to specify sdists, beyond being a possible source of a source tree.
If the backend needs to produce an sdist from a source tree, I think we
need a clearer picture of what the expected result looks like.

Thomas

From leorochael at gmail.com  Wed May 31 09:42:56 2017
From: leorochael at gmail.com (Leonardo Rochael Almeida)
Date: Wed, 31 May 2017 10:42:56 -0300
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CAPJVwBkHn6vubc=pKoh_g0vmNCcm3rNd_s0P1UFSnA5ezC=P9g@mail.gmail.com>
Message-ID: <CAMxX_oOfKi7vv1-QSnCX2g63uS270EeBBLCVNk-z3j5fWAQAOQ@mail.gmail.com>

Hi,

On 29 May 2017 at 04:05, Nathaniel Smith <njs at pobox.com> wrote:

> On Sun, May 28, 2017 at 10:37 PM, Donald Stufft <donald at stufft.io> wrote:
> >
> > [...]
> >
> > 1. Using the {package-name}-{package-version}.dist-info in the
> > get_wheel_metadata() metadata is a mistake I think. In pip currently we
> have
> > a bug we have not yet been able to track down because there is nothing
> > systematically preventing both foobar-1.0.dist-info and
> foobar-2.0.distinfo
> > from existing side by side in a build directory (or inside a wheel for
> that
> > matter). Thus I think this naming scheme is a nuisance and we shouldn?t
> > propagate it any further. I would just use something like DIST-INFO/
> which
> > will completely side step this issue. The only reason I can think of to
> use
> > the current scheme is to make it easier to shutil.copytree it into the
> > wheel, but handling that case is trivial.
>
> The rationale for this is to leave the door open to in the future
> allowing the same sdist to build multiple wheels. [...]
>
> For get_wheel_metadata() in particular there are several options
> though... we could call it DIST-INFO/ and then later declare that
> DIST-INFO2/, DIST-INFO3/, etc. are also valid and pip will look at all
> of them.
>
> {package-name}.dist-info might also be reasonable, both here and in
> actual installs... [...]
>

Wasn't there a thread here some time ago about switching from:
 - `{package-name}-{package-version}.dist-info`

to:

 - `{package-name}.dist-info`

in all tools?

(while accepting the old format, for old, already built or already
installed wheels, of course)

If I remember correctly, there wasn't any complaints about doing this
(multi-version installs where a setuptools/egg-info thing anyway and
wouldn't be affected by this), and it's perfectly compatible with building
multiple wheels from a single source package.

What was needed to move forward with this change?

Regards,

Leo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170531/821dfc19/attachment.html>

From p.f.moore at gmail.com  Wed May 31 09:57:54 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 31 May 2017 14:57:54 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
Message-ID: <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>

On 31 May 2017 at 14:39, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> On Wed, May 31, 2017, at 02:24 PM, Nick Coghlan wrote:
>> Cool, thanks for doing that - I'll let you thrash out the details with
>> Thomas as the PEP author, and then take a look once you're in
>> agreement on the amendments you want to make.
>
> I've had a quick look over the PR, and my main thought is that if we're
> going to specify a 'build sdist' step, we need to specify more precisely
> what an sdist actually means. The PEP as currently written expects a
> 'source tree' as the input for building a wheel, but does not really
> need to specify sdists, beyond being a possible source of a source tree.
> If the backend needs to produce an sdist from a source tree, I think we
> need a clearer picture of what the expected result looks like.

My feeling is that in the current context we're talking about "minimal
set of files needed to build the wheel". That's not precisely a sdist
(because it ignores additional files that the user might want to
include like README, COPYRIGHT, or tests) so doesn't satisfy the use
case for something like "twine sdist" - but I don't see how we can do
that without getting sucked into the full "sdist 2.0" debate, which
really is off-topic for this PEP.

I'll take a proper look at the PR and add any further comments there.

Paul

From thomas at kluyver.me.uk  Wed May 31 10:00:17 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Wed, 31 May 2017 15:00:17 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
Message-ID: <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>

On Wed, May 31, 2017, at 02:57 PM, Paul Moore wrote:
> My feeling is that in the current context we're talking about "minimal
> set of files needed to build the wheel". That's not precisely a sdist
> (because it ignores additional files that the user might want to
> include like README, COPYRIGHT, or tests) so doesn't satisfy the use
> case for something like "twine sdist" - but I don't see how we can do
> that without getting sucked into the full "sdist 2.0" debate, which
> really is off-topic for this PEP.

I would be fine with specifying a hook to copy only the files needed to
build the wheel, but if we do that, let's not call it 'sdist' or
anything that suggests that.

From thomas at kluyver.me.uk  Wed May 31 10:05:12 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Wed, 31 May 2017 15:05:12 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
 <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
Message-ID: <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>

On Wed, May 31, 2017, at 03:00 PM, Thomas Kluyver wrote:
> I would be fine with specifying a hook to copy only the files needed to
> build the wheel, but if we do that, let's not call it 'sdist' or
> anything that suggests that.

Also, if this is all we're after, I'd like to push again for making it
optional - for flit, the resulting wheel will always be the same whether
you copy the files somewhere else first or just build from the original
source tree. It's less to implement and more performant if flit can just
build the wheel directly, and skip the copying step.

Thomas

From rmcgibbo at gmail.com  Wed May 31 10:19:52 2017
From: rmcgibbo at gmail.com (Robert T. McGibbon)
Date: Wed, 31 May 2017 10:19:52 -0400
Subject: [Distutils] Installation problem
In-Reply-To: <CADPEtBY1no0iSRwwuhtji=u+1ayjzjY0p9riGfa7aRLzZ59fHQ@mail.gmail.com>
References: <CADPEtBY1no0iSRwwuhtji=u+1ayjzjY0p9riGfa7aRLzZ59fHQ@mail.gmail.com>
Message-ID: <CAN4+E8H0MN1UKOBL0Qt2h+uP2=qvOJ+f0b5=B9dBJ+MBb4A+ug@mail.gmail.com>

This isn't actually the right place to ask for installation help, Aneesh.
Your best bet would be the python-list or python-help mailing list.

-Robert

On Tue, May 30, 2017 at 7:24 PM, Aneesh Kona <anee.vesna at gmail.com> wrote:

> Hi,
> I'm getting an error while installing software,
> I did installed firstly but did in another in local disk,
> but I want to install again, now it shows 'fatal error,' can you please
> let me fix it, so that I can install again in C disk,
> plz hand me in this particular,
> Thanks.
>
> Venkata Aneesh Kona
> Graduate Assistant
> Department of Mining Engineering and Management
> South Dakota School of Mines & Technology
> @: venkataaneesh.kona at mines.sdsmt.edu
> ph: 6823042622 <(682)%20304-2622>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>


-- 
-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170531/b0a323cf/attachment-0001.html>

From p.f.moore at gmail.com  Wed May 31 11:16:41 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 31 May 2017 16:16:41 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
 <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
 <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>
Message-ID: <CACac1F-pOZXxS84H+_B4_tQPTBGi9JZgyrH6V-B5GD9ktmXOgg@mail.gmail.com>

On 31 May 2017 at 15:05, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> On Wed, May 31, 2017, at 03:00 PM, Thomas Kluyver wrote:
>> I would be fine with specifying a hook to copy only the files needed to
>> build the wheel, but if we do that, let's not call it 'sdist' or
>> anything that suggests that.

I agree - I've been trying to push away from using the term "sdist"
myself, as there's too much history with it, most of which isn't (IMO)
relevant in this context.

> Also, if this is all we're after, I'd like to push again for making it
> optional - for flit, the resulting wheel will always be the same whether
> you copy the files somewhere else first or just build from the original
> source tree. It's less to implement and more performant if flit can just
> build the wheel directly, and skip the copying step.

Hmm. The proposed API (whether it's "create a sdist" or "tell me what
files to copy") is intended so that a frontend can use the following
process:

1. Create a temp directory
2. Call a hook to get the "source"
3. Put the source in the temp directory (unpack, copy, ask the hook to
put the files in here directly, whatever)
4. Call the "build wheel" hook on the temp directory

Pip basically does this currently, and it's probably out of scope for
this PEP to look at changing that. The current discussion is really
only about whether we need a hook at step 2, or whether "copy the
whole source tree" is sufficient. Clearly, in one sense, it is
sufficient, because that's what pip has to do at the moment, but we
have outstanding bug reports flagging that our copy step is sometimes
severely inefficient, and without the proposed hook, we have no way of
improving things. The benefit of a formal hook is that it gives the
backend control of the process, but that only helps if the tools that
cause us issues currently, like setuptools_scm, work with the backend,
specifically to control what gets copied so that it's sufficient.

Backends like flit probably have no need for this flexibility, but
should probably follow some form of standard rule like "exclude VCS
and tox control directories". It may make sense to implement that rule
in the frontend (on the usual "one frontend, multiple backends" basis)
but I'm inclined to think that choice should be opt-in, not make it
the default if a backend doesn't implement the hook. I'm prepared to
be convinced otherwise, though.

By the way, in theory with this workflow, there's some pretty bizarre
misbehaviours that backends could implement. Consider a backend that
does "build sdist" by preprocessing the current directory, and writing
a sdist that uses a completely different backend (e.g., a
"setuptools_scm" backend that writes sdists that freeze the version
data and use flit to build). Then step (4) needs to re-read the
backend before running the "build wheel" hook. That's actually a
not-unreasonable way to handle tools like setuptools_scm, so we should
probably take a view on whether it's allowable. Specifically, there's
a choice the spec needs to make between

1. Frontends MAY determine the backend once and assume that it won't
change while processing a project
   1a. Hooks MUST NOT modify the pyproject.toml metadata specifying
the backend details, and MUST copy that file unchanged if asked to
copy it

vs

2. Frontends MUST allow for the possibility that pyproject.toml could
change after a hook is called

Ever get the feeling that you shouldn't have looked too closely at
something? :-)

Paul

From donald at stufft.io  Wed May 31 11:31:32 2017
From: donald at stufft.io (Donald Stufft)
Date: Wed, 31 May 2017 11:31:32 -0400
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CACac1F-pOZXxS84H+_B4_tQPTBGi9JZgyrH6V-B5GD9ktmXOgg@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
 <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
 <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>
 <CACac1F-pOZXxS84H+_B4_tQPTBGi9JZgyrH6V-B5GD9ktmXOgg@mail.gmail.com>
Message-ID: <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io>

The hook is also so a tool like tox or TravisCI or twine can produce a sdist that can be uploaded to PyPI or similar. 

Sent from my iPhone

> On May 31, 2017, at 11:16 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> Hmm. The proposed API (whether it's "create a sdist" or "tell me what
> files to copy") is intended so that a frontend can use the following
> process:
> 
> 1. Create a temp directory
> 2. Call a hook to get the "source"
> 3. Put the source in the temp directory (unpack, copy, ask the hook to
> put the files in here directly, whatever)
> 4. Call the "build wheel" hook on the temp directory


From thomas at kluyver.me.uk  Wed May 31 11:53:09 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Wed, 31 May 2017 16:53:09 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
 <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
 <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>
 <CACac1F-pOZXxS84H+_B4_tQPTBGi9JZgyrH6V-B5GD9ktmXOgg@mail.gmail.com>
 <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io>
Message-ID: <1496245989.926144.994309856.6D666DF2@webmail.messagingengine.com>

On Wed, May 31, 2017, at 04:31 PM, Donald Stufft wrote:
> The hook is also so a tool like tox or TravisCI or twine can produce a
> sdist that can be uploaded to PyPI or similar. 

This seems like a distinct operation from 'prepare the files needed to
build a wheel', although they are related. For instance, sdists
typically contain docs (in source form), but you don't need these to
build a wheel. It would be fairly easy for Flit to identify the files
needed to build a wheel, but it can only identify the files needed for
an sdist with the help of a VCS.

Would you be happy with a compromise where PEP 517 defines a (possibly
optional) hook like 'prepare_build_files', and we leave a standardised
interface for preparing an sdist to be hashed out in a later PR?

Thomas

From dholth at gmail.com  Wed May 31 12:01:56 2017
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 31 May 2017 16:01:56 +0000
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <1496245989.926144.994309856.6D666DF2@webmail.messagingengine.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
 <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
 <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>
 <CACac1F-pOZXxS84H+_B4_tQPTBGi9JZgyrH6V-B5GD9ktmXOgg@mail.gmail.com>
 <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io>
 <1496245989.926144.994309856.6D666DF2@webmail.messagingengine.com>
Message-ID: <CAG8k2+7MneK65FHctcvCddNHBL+mv+7nSZBet_qbg4wxWyFmtQ@mail.gmail.com>

Sounds like we have bad support for out of ? builds

On Wed, May 31, 2017, 11:53 Thomas Kluyver <thomas at kluyver.me.uk> wrote:

> On Wed, May 31, 2017, at 04:31 PM, Donald Stufft wrote:
> > The hook is also so a tool like tox or TravisCI or twine can produce a
> > sdist that can be uploaded to PyPI or similar.
>
> This seems like a distinct operation from 'prepare the files needed to
> build a wheel', although they are related. For instance, sdists
> typically contain docs (in source form), but you don't need these to
> build a wheel. It would be fairly easy for Flit to identify the files
> needed to build a wheel, but it can only identify the files needed for
> an sdist with the help of a VCS.
>
> Would you be happy with a compromise where PEP 517 defines a (possibly
> optional) hook like 'prepare_build_files', and we leave a standardised
> interface for preparing an sdist to be hashed out in a later PR?
>
> Thomas
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170531/75f05a3f/attachment.html>

From p.f.moore at gmail.com  Wed May 31 12:33:57 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 31 May 2017 17:33:57 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
 <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
 <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>
 <CACac1F-pOZXxS84H+_B4_tQPTBGi9JZgyrH6V-B5GD9ktmXOgg@mail.gmail.com>
 <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io>
Message-ID: <CACac1F-vgULOYhAo0JPAzsxfeBMow1RqhH2=Z3jJEWNOSxhXpw@mail.gmail.com>

On 31 May 2017 at 16:31, Donald Stufft <donald at stufft.io> wrote:
> The hook is also so a tool like tox or TravisCI or twine can produce a sdist that can be uploaded to PyPI or similar.

Understood. The part that the backend can do (these are the files
needed for the build process) is done via the hook. The rest (add
files requested by the user, generate metadata, ...) is backend
independent (or covered by a separate hook/not specified yet in the
case of metadata) and so should be done by the frontend tool.

Paul

PS When did Travis become a frontend? I'd have assumed that to produce
a sdist, Travis would *invoke* a frontend tool, such as "twine sdist".
Or am I unaware of some capability of Travis (highly likely)?

From donald at stufft.io  Wed May 31 13:03:55 2017
From: donald at stufft.io (Donald Stufft)
Date: Wed, 31 May 2017 13:03:55 -0400
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CACac1F-vgULOYhAo0JPAzsxfeBMow1RqhH2=Z3jJEWNOSxhXpw@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
 <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
 <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>
 <CACac1F-pOZXxS84H+_B4_tQPTBGi9JZgyrH6V-B5GD9ktmXOgg@mail.gmail.com>
 <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io>
 <CACac1F-vgULOYhAo0JPAzsxfeBMow1RqhH2=Z3jJEWNOSxhXpw@mail.gmail.com>
Message-ID: <FC2711EF-F610-41F4-BEF8-6D51380527DD@stufft.io>


> On May 31, 2017, at 12:33 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 31 May 2017 at 16:31, Donald Stufft <donald at stufft.io> wrote:
>> The hook is also so a tool like tox or TravisCI or twine can produce a sdist that can be uploaded to PyPI or similar.
> 
> Understood. The part that the backend can do (these are the files
> needed for the build process) is done via the hook. The rest (add
> files requested by the user, generate metadata, ...) is backend
> independent (or covered by a separate hook/not specified yet in the
> case of metadata) and so should be done by the frontend tool.

I don?t think it?s backend independent though. You?re going to have different mechanisms for handling these things in different backends, for example one piece of the metadata is the version. Some projects will be fine with a static version, some projects are going to want to automatically deduce it using a VCS. Trying to cram all of these into a single tool falls into the same problem that PEP 517 is trying to solve.

> 
> Paul
> 
> PS When did Travis become a frontend? I'd have assumed that to produce
> a sdist, Travis would *invoke* a frontend tool, such as "twine sdist".
> Or am I unaware of some capability of Travis (highly likely)?


No you?re correct, it currently just invokes ``setup.py sdist bdist_wheel``. The hook is needed so that Travis can have a singular tool to invoke (likely twine?) instead of needing to determine if it needs to invoke flit or setuptools or mytotallyradbuildthing. The thing I?m trying to express (and doing poorly it seems :( ) is that generating a sdist is an important thing to have be possible, and it needs to be done in a way that it can be invoked generically.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170531/9e76ce4f/attachment.html>

From thomas at kluyver.me.uk  Wed May 31 13:59:23 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Wed, 31 May 2017 18:59:23 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <FC2711EF-F610-41F4-BEF8-6D51380527DD@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
 <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
 <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>
 <CACac1F-pOZXxS84H+_B4_tQPTBGi9JZgyrH6V-B5GD9ktmXOgg@mail.gmail.com>
 <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io>
 <CACac1F-vgULOYhAo0JPAzsxfeBMow1RqhH2=Z3jJEWNOSxhXpw@mail.gmail.com>
 <FC2711EF-F610-41F4-BEF8-6D51380527DD@stufft.io>
Message-ID: <1496253563.733118.994457104.19DF8B31@webmail.messagingengine.com>

On Wed, May 31, 2017, at 06:03 PM, Donald Stufft wrote:
> generating a sdist is an important thing to have be possible, and it
> needs to be done in a way that it can be invoked generically.
I agree that it needs to be possible to make an sdist, but can we leave
the generic interface for it to be standardised later? At the moment, I
don't see an urgent need for a build-system independent 'make an sdist'
command, and it's wrapped up in this whole question of what an sdist is
and what you make it from (e.g. flit can make an sdist from a VCS, but
not from another sdist).
For pip, we seem to be using sdist as a proxy for 'get the files we need
to build this thing'. That's much easier to specify and causes less
controversy.
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170531/7ca839f9/attachment-0001.html>

From p.f.moore at gmail.com  Wed May 31 14:01:51 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 31 May 2017 19:01:51 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <FC2711EF-F610-41F4-BEF8-6D51380527DD@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
 <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
 <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>
 <CACac1F-pOZXxS84H+_B4_tQPTBGi9JZgyrH6V-B5GD9ktmXOgg@mail.gmail.com>
 <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io>
 <CACac1F-vgULOYhAo0JPAzsxfeBMow1RqhH2=Z3jJEWNOSxhXpw@mail.gmail.com>
 <FC2711EF-F610-41F4-BEF8-6D51380527DD@stufft.io>
Message-ID: <CACac1F8+utdgQYHYNVu=NceV5Z+yUrMDWjMR+jdP3Yo7i8fxvQ@mail.gmail.com>

On 31 May 2017 at 18:03, Donald Stufft <donald at stufft.io> wrote:
> No you?re correct, it currently just invokes ``setup.py sdist bdist_wheel``.
> The hook is needed so that Travis can have a singular tool to invoke (likely
> twine?) instead of needing to determine if it needs to invoke flit or
> setuptools or mytotallyradbuildthing. The thing I?m trying to express (and
> doing poorly it seems :( ) is that generating a sdist is an important thing
> to have be possible, and it needs to be done in a way that it can be invoked
> generically.

I don't think that's either unclear or in dispute. The question here
is whether "produce a sdist" is in scope for this particular PEP.

The problem is that you're proposing using a "build a sdist" hook as
the means for pip to do its "copy the source to an isolated build
directory" step. Currently, doing that fails because tools like
setuptools_scm work differently when run from a VCS checkout instead
of a sdist. The long term plan for pip is something we've always
described in terms of going via a sdist, but there's lots of details
we haven't thrashed out yet. I don't think we're at a point where we
can insist that in a post-PEP 517 world, we can switch straight to
building via a sdist. However, I agree with you that we want PEP 517
to assist us with moving in that direction, and we *definitely* don't
want to get into a situation like we're in now, where a PEP 517
compliant backend can leave us with no option better than "copy the
whole source tree".

That's why I'm focusing at the moment on asking that PEP 517 require
backends to specify the minimal set of files needed to do the build.
With that we can create isolated build directories efficiently, and
those files are going to be an essential component of building a
sdist. They aren't *sufficient* to build a sdist by themselves, but
specifying in a PEP how we do build a sdist is a perfectly reasonable
next step, and can be done independently of PEP 517 (and indeed of pip
- there's no "pip sdist" command, so pip doesn't even have a stake in
that debate).

One other point that I think we're glossing over here. The only reason
that I know of why current setuptools builds can't be isolated via a
"build sdist" step, is because of helpers like setuptools_scm. I don't
know much about such tools, and I don't know where they'd fit in a
landscape of PEP 517 backends. Would setuptools_scm be a backend
itself, or would it be a plugin for other backends (would backends
want to *have* a plugin system for things like this)? The only
concrete backends I know of are flit and
something-that-will-do-what-setuptools-does-involving-compilers.
Neither of those need a "tell me what files constitute a build" hook -
they'd be fine with pip (or a library like packaging) implementing a
"copy everything except VCS directories, tox directories, etc"
heuristic[1]. So the cases where we *need* a hook are those where an
as-yet undefined "use VCS information to generate wheel metadata"
backends (or backend plugin) is involved - and nobody knows what shape
that will take, so we're guessing. My feeling is thatsuch a tool
should be a backend that wraps another backend, and modifies the
generate metadata hook. On that basis, it's perfectly capable of also
modifying the "specify the build files" hook. But that's nothing more
than a guess at this point.

Thoughts?

Paul

[1] They can do a better job than that, certainly, which is why the
hook is a good thing, but it's not a showstopper. The key point is
that they need *less* than the obvious heuristic would give them, not
more.

From thomas at kluyver.me.uk  Wed May 31 14:13:43 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Wed, 31 May 2017 19:13:43 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CACac1F8+utdgQYHYNVu=NceV5Z+yUrMDWjMR+jdP3Yo7i8fxvQ@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
 <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
 <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>
 <CACac1F-pOZXxS84H+_B4_tQPTBGi9JZgyrH6V-B5GD9ktmXOgg@mail.gmail.com>
 <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io>
 <CACac1F-vgULOYhAo0JPAzsxfeBMow1RqhH2=Z3jJEWNOSxhXpw@mail.gmail.com>
 <FC2711EF-F610-41F4-BEF8-6D51380527DD@stufft.io>
 <CACac1F8+utdgQYHYNVu=NceV5Z+yUrMDWjMR+jdP3Yo7i8fxvQ@mail.gmail.com>
Message-ID: <1496254423.737105.994476664.0B1099C1@webmail.messagingengine.com>

I've made an alternative PR against PEP 517 which defines a
prepare_build_files hook, defined in terms of copying the files needed
to build a package rather than making an sdist.

https://github.com/python/peps/pull/277

From donald at stufft.io  Wed May 31 14:40:00 2017
From: donald at stufft.io (Donald Stufft)
Date: Wed, 31 May 2017 14:40:00 -0400
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CACac1F8+utdgQYHYNVu=NceV5Z+yUrMDWjMR+jdP3Yo7i8fxvQ@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
 <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
 <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>
 <CACac1F-pOZXxS84H+_B4_tQPTBGi9JZgyrH6V-B5GD9ktmXOgg@mail.gmail.com>
 <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io>
 <CACac1F-vgULOYhAo0JPAzsxfeBMow1RqhH2=Z3jJEWNOSxhXpw@mail.gmail.com>
 <FC2711EF-F610-41F4-BEF8-6D51380527DD@stufft.io>
 <CACac1F8+utdgQYHYNVu=NceV5Z+yUrMDWjMR+jdP3Yo7i8fxvQ@mail.gmail.com>
Message-ID: <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io>


> On May 31, 2017, at 2:01 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 31 May 2017 at 18:03, Donald Stufft <donald at stufft.io> wrote:
>> No you?re correct, it currently just invokes ``setup.py sdist bdist_wheel``.
>> The hook is needed so that Travis can have a singular tool to invoke (likely
>> twine?) instead of needing to determine if it needs to invoke flit or
>> setuptools or mytotallyradbuildthing. The thing I?m trying to express (and
>> doing poorly it seems :( ) is that generating a sdist is an important thing
>> to have be possible, and it needs to be done in a way that it can be invoked
>> generically.
> 
> I don't think that's either unclear or in dispute. The question here
> is whether "produce a sdist" is in scope for this particular PEP.
> 
> The problem is that you're proposing using a "build a sdist" hook as
> the means for pip to do its "copy the source to an isolated build
> directory" step. Currently, doing that fails because tools like
> setuptools_scm work differently when run from a VCS checkout instead
> of a sdist. The long term plan for pip is something we've always
> described in terms of going via a sdist, but there's lots of details
> we haven't thrashed out yet. I don't think we're at a point where we
> can insist that in a post-PEP 517 world, we can switch straight to
> building via a sdist. However, I agree with you that we want PEP 517
> to assist us with moving in that direction, and we *definitely* don't
> want to get into a situation like we're in now, where a PEP 517
> compliant backend can leave us with no option better than "copy the
> whole source tree?.

I don?t think we can start telling projects if they using a PEP 517 they can delete their ``setup.py`` and live in the brave new world (assuming all of their tools have been updated to PEP 517) when doing so is removing a ?standard? interface for producing a sdist. Either a replacement for setup.py should support the things we want to keep from setup.py and explicitly unsupport things we don?t want, or I don?t think that thing is actually a replacement for setup.py and I don?t think we should support it.

Taking pip completely off the table a second, let?s take a look at tox. Tox?s default mode of operation is to produce a sdist. Now let?s say I?m writing a project that I want to use PEP 517 and get rid of setup.py, except now tox is broken with no path forward because PEP 517 doesn?t define how to produce a sdist. 

The same story is true for TravisCI?s PyPI deployment pipeline, as soon as any project starts depending on PEP 517, we completely break that feature for them without a path for them to fix it (besides writing a PEP of course).

The same is true for Gem Fury?s private PyPI repositories where you can ``git push fury`` and have them build a sdist automatically for you.

This same pattern is over and over and over again, projects depend on the ability to produce a sdist for any Python project. PEP 517 says that people can delete their setup.py but doesn?t provide the mechanism for producing a sdist, thus breaking parts of the ecosystem. Simply changing the PEP to say ?ok you can?t delete your setup.py yet? isn?t acceptable yet either, because then you have two competing build systems both who think they should be in charge, which makes the entire process *more* confusing for the end users than just baking the concept of sdist generation into PEP 517.

Now, independently of that, pip needs a way to take an arbitrary directory that might contain a git clone with a bunch of extraneous files in it, or it might also just be a sdist that was already unpacked. For a variety of reasons we want to copy this directory into a temporary location, but doing a blind copy of everything can trigger a bad path where a simple ``pip install .`` can take a long time (up to minutes long have been reported in the wild) trying to copy the entire directory, including files that we don?t even need or want. We need some mechanism for copying these files over, and it just so happens that the exact same process needs to occur when computing what files going into a sdist, and since I believe that for completely unrelated reasons, computing a sdist *must* be a part of any attempt to replace setup.py, reusing that simplifies the process of creating a PEP 517 backend (since having to only implement build_sdist is simpler than having to implement build_sdist AND copy_files_for_build).

In addition to all of the above, we currently have like 7 different ?paths? installation can go through on the process of going from a VCS checkout/developer copy to a installed distribution, we have:

1) VCS Checkout -> Installed
2) VCS Checkout -> Sdist -> Installed
3) VCS Checkout -> Wheel -> Installed
4) VCS Checkout -> Sdist -> Wheel -> Installed
5) VCS Checkout -> Editable Install
6) VCS Checkout -> Sdist -> Editable Install

Unless you?re careful to have your packaging done exactly correct, each of those 6 can end up having different (and often times surprising behavior) that regular end users who are new to packaging (or hell, even old hands) hit with some regularity. One of my long term goals is try and reduce the number of those paths down, which will make it more likely that people are not surprised by edge cases in how their own uses are calling ``pip install`` and will ultimately provide a more enjoyable experience using pip. We obviously cannot reduce the number of supported methods down to 1, but we can reduce them down to:

A) VCS Checkout -> Sdist -> Wheel -> Installed
B) VCS Checkout -> Editable Install

Implementing build_sdist in PEP 517 and using that to handle copying the files from what could either be a VCS checkout OR an unpacked sdist, means that we eliminate (1) and (3) from the first list. Ensuring that we only ever install a PEP 517 style project by always using build_wheel after having used build_sdist then eliminates (2). We?re not dealing with editable installs here (and it kind of pains me we aren?t, but they?re a much bigger topic so I think it?s unavoidable) but preventing an editable install of an sdist would eliminate (6) from above, leaving us with just two paths (and the second path requiring an explicit flag to opt into, rather than being implicit by nature of what you?re passing into pip and what other libraries you have installed).

In addition to all of the above, any part of building a sdist that is more complicated than ?copy some files?, these build backends are already going to have to support by nature of the fact we?re expecting them to generate wheel metadata. The wheel metadata has to include the version number, so if someone wants to dynamically compute the version number from git, a PEP 517 backend must already handle that or it simply won?t work.

Finally, we?re should/are assuming that these build projects are going to be capable of producing sdists. Thus they already have to implement 99% of build_sdist anyways, and the only additional effort on their part is just the glue code that wires up their internal mechanism for producing sdists to the API that allows a standard mechanism for calling those mechanisms. Hopefully it is not controversial that a build tool *must* be capable of producing a sdist, since otherwise we?re throwing away support for any non Windows/macOS/Linux platform. Implementing a custom ?copy these files? is *more* effort than exposing the mechanism that they should already have.

So yes, one of the things I want to do with this hook is copy the source files to an isolated directory, but that?s not the *only* thing I want to do with that hook, and when I see single solution that can solve multiple problems, I always vastly prefer that over a single solution that only solves a single problem.


?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170531/a6a0de04/attachment-0001.html>

From dholth at gmail.com  Wed May 31 15:02:19 2017
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 31 May 2017 19:02:19 +0000
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
 <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
 <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>
 <CACac1F-pOZXxS84H+_B4_tQPTBGi9JZgyrH6V-B5GD9ktmXOgg@mail.gmail.com>
 <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io>
 <CACac1F-vgULOYhAo0JPAzsxfeBMow1RqhH2=Z3jJEWNOSxhXpw@mail.gmail.com>
 <FC2711EF-F610-41F4-BEF8-6D51380527DD@stufft.io>
 <CACac1F8+utdgQYHYNVu=NceV5Z+yUrMDWjMR+jdP3Yo7i8fxvQ@mail.gmail.com>
 <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io>
Message-ID: <CAG8k2+51nXFf1=+_0P313hoS-xsy7nw3zKF+LX4isZwExYdRUg@mail.gmail.com>

In my experience tools think an archive is an sdist if it is named
properly, contains PKG-INFO with a minimum number of fields

Metadata-Version: 1.1Name: bobVersion: 1.0


and setup.py. setuptools sdists also contain .egg-info but it is unnecessary.


On Wed, May 31, 2017 at 2:41 PM Donald Stufft <donald at stufft.io> wrote:

> On May 31, 2017, at 2:01 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>
> On 31 May 2017 at 18:03, Donald Stufft <donald at stufft.io> wrote:
>
> No you?re correct, it currently just invokes ``setup.py sdist
> bdist_wheel``.
> The hook is needed so that Travis can have a singular tool to invoke
> (likely
> twine?) instead of needing to determine if it needs to invoke flit or
> setuptools or mytotallyradbuildthing. The thing I?m trying to express (and
> doing poorly it seems :( ) is that generating a sdist is an important thing
> to have be possible, and it needs to be done in a way that it can be
> invoked
> generically.
>
>
> I don't think that's either unclear or in dispute. The question here
> is whether "produce a sdist" is in scope for this particular PEP.
>
> The problem is that you're proposing using a "build a sdist" hook as
> the means for pip to do its "copy the source to an isolated build
> directory" step. Currently, doing that fails because tools like
> setuptools_scm work differently when run from a VCS checkout instead
> of a sdist. The long term plan for pip is something we've always
> described in terms of going via a sdist, but there's lots of details
> we haven't thrashed out yet. I don't think we're at a point where we
> can insist that in a post-PEP 517 world, we can switch straight to
> building via a sdist. However, I agree with you that we want PEP 517
> to assist us with moving in that direction, and we *definitely* don't
> want to get into a situation like we're in now, where a PEP 517
> compliant backend can leave us with no option better than "copy the
> whole source tree?.
>
>
> I don?t think we can start telling projects if they using a PEP 517 they
> can delete their ``setup.py`` and live in the brave new world (assuming all
> of their tools have been updated to PEP 517) when doing so is removing a
> ?standard? interface for producing a sdist. Either a replacement for
> setup.py should support the things we want to keep from setup.py and
> explicitly unsupport things we don?t want, or I don?t think that thing is
> actually a replacement for setup.py and I don?t think we should support it.
>
> Taking pip completely off the table a second, let?s take a look at tox.
> Tox?s default mode of operation is to produce a sdist. Now let?s say I?m
> writing a project that I want to use PEP 517 and get rid of setup.py,
> except now tox is broken with no path forward because PEP 517 doesn?t
> define how to produce a sdist.
>
> The same story is true for TravisCI?s PyPI deployment pipeline, as soon as
> any project starts depending on PEP 517, we completely break that feature
> for them without a path for them to fix it (besides writing a PEP of
> course).
>
> The same is true for Gem Fury?s private PyPI repositories where you can
> ``git push fury`` and have them build a sdist automatically for you.
>
> This same pattern is over and over and over again, projects depend on the
> ability to produce a sdist for any Python project. PEP 517 says that people
> can delete their setup.py but doesn?t provide the mechanism for producing a
> sdist, thus breaking parts of the ecosystem. Simply changing the PEP to say
> ?ok you can?t delete your setup.py yet? isn?t acceptable yet either,
> because then you have two competing build systems both who think they
> should be in charge, which makes the entire process *more* confusing for
> the end users than just baking the concept of sdist generation into PEP 517.
>
> Now, independently of that, pip needs a way to take an arbitrary directory
> that might contain a git clone with a bunch of extraneous files in it, or
> it might also just be a sdist that was already unpacked. For a variety of
> reasons we want to copy this directory into a temporary location, but doing
> a blind copy of everything can trigger a bad path where a simple ``pip
> install .`` can take a long time (up to minutes long have been reported in
> the wild) trying to copy the entire directory, including files that we
> don?t even need or want. We need some mechanism for copying these files
> over, and it just so happens that the exact same process needs to occur
> when computing what files going into a sdist, and since I believe that for
> completely unrelated reasons, computing a sdist *must* be a part of any
> attempt to replace setup.py, reusing that simplifies the process of
> creating a PEP 517 backend (since having to only implement build_sdist is
> simpler than having to implement build_sdist AND copy_files_for_build).
>
> In addition to all of the above, we currently have like 7 different
> ?paths? installation can go through on the process of going from a VCS
> checkout/developer copy to a installed distribution, we have:
>
> 1) VCS Checkout -> Installed
> 2) VCS Checkout -> Sdist -> Installed
> 3) VCS Checkout -> Wheel -> Installed
> 4) VCS Checkout -> Sdist -> Wheel -> Installed
> 5) VCS Checkout -> Editable Install
> 6) VCS Checkout -> Sdist -> Editable Install
>
> Unless you?re careful to have your packaging done exactly correct, each of
> those 6 can end up having different (and often times surprising behavior)
> that regular end users who are new to packaging (or hell, even old hands)
> hit with some regularity. One of my long term goals is try and reduce the
> number of those paths down, which will make it more likely that people are
> not surprised by edge cases in how their own uses are calling ``pip
> install`` and will ultimately provide a more enjoyable experience using
> pip. We obviously cannot reduce the number of supported methods down to 1,
> but we can reduce them down to:
>
> A) VCS Checkout -> Sdist -> Wheel -> Installed
> B) VCS Checkout -> Editable Install
>
> Implementing build_sdist in PEP 517 and using that to handle copying the
> files from what could either be a VCS checkout OR an unpacked sdist, means
> that we eliminate (1) and (3) from the first list. Ensuring that we only
> ever install a PEP 517 style project by always using build_wheel after
> having used build_sdist then eliminates (2). We?re not dealing with
> editable installs here (and it kind of pains me we aren?t, but they?re a
> much bigger topic so I think it?s unavoidable) but preventing an editable
> install of an sdist would eliminate (6) from above, leaving us with just
> two paths (and the second path requiring an explicit flag to opt into,
> rather than being implicit by nature of what you?re passing into pip and
> what other libraries you have installed).
>
> In addition to all of the above, any part of building a sdist that is more
> complicated than ?copy some files?, these build backends are already going
> to have to support by nature of the fact we?re expecting them to generate
> wheel metadata. The wheel metadata has to include the version number, so if
> someone wants to dynamically compute the version number from git, a PEP 517
> backend must already handle that or it simply won?t work.
>
> Finally, we?re should/are assuming that these build projects are going to
> be capable of producing sdists. Thus they already have to implement 99% of
> build_sdist anyways, and the only additional effort on their part is just
> the glue code that wires up their internal mechanism for producing sdists
> to the API that allows a standard mechanism for calling those mechanisms.
> Hopefully it is not controversial that a build tool *must* be capable of
> producing a sdist, since otherwise we?re throwing away support for any non
> Windows/macOS/Linux platform. Implementing a custom ?copy these files? is
> *more* effort than exposing the mechanism that they should already have.
>
> So yes, one of the things I want to do with this hook is copy the source
> files to an isolated directory, but that?s not the *only* thing I want to
> do with that hook, and when I see single solution that can solve multiple
> problems, I always vastly prefer that over a single solution that only
> solves a single problem.
>
>
> ?
>
> Donald Stufft
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170531/dc2999a3/attachment.html>

From thomas at kluyver.me.uk  Wed May 31 15:12:27 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Wed, 31 May 2017 20:12:27 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
 <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
 <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>
 <CACac1F-pOZXxS84H+_B4_tQPTBGi9JZgyrH6V-B5GD9ktmXOgg@mail.gmail.com>
 <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io>
 <CACac1F-vgULOYhAo0JPAzsxfeBMow1RqhH2=Z3jJEWNOSxhXpw@mail.gmail.com>
 <FC2711EF-F610-41F4-BEF8-6D51380527DD@stufft.io>
 <CACac1F8+utdgQYHYNVu=NceV5Z+yUrMDWjMR+jdP3Yo7i8fxvQ@mail.gmail.com>
 <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io>
Message-ID: <1496257947.754070.994532544.49214D71@webmail.messagingengine.com>

On Wed, May 31, 2017, at 07:40 PM, Donald Stufft wrote:
> Taking pip completely off the table a second, let?s take a look at
> tox. Tox?s default mode of operation is to produce a sdist. Now let?s
> say I?m writing a project that I want to use PEP 517 and get rid of
> setup.py, except now tox is broken with no path forward because PEP
> 517 doesn?t define how to produce a sdist.> 
> The same story is true for TravisCI?s PyPI deployment pipeline, as
> soon as any project starts depending on PEP 517, we completely break
> that feature for them without a path for them to fix it (besides
> writing a PEP of course).> 
> The same is true for Gem Fury?s private PyPI repositories where
> you can ``git push fury`` and have them build a sdist
> automatically for you.
These tools are all things that the developers of the project choose to
use, however. I don't use them, so I'm happy enough to get rid of
setup.py and not have a standard interface to building sdists.
Developers who do use them will want to stick with setup.py until
there's a standard way to build an sdist - or a tool like tox may add
support for going via wheels instead of via sdist.
So PEP 517 may not be useful to *everyone* without standardising a way
to build sdists, but it is still useful for many projects, and I don't
think it prevents a later PEP from standardising a way to build sdists.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170531/7643cfe9/attachment-0001.html>

From donald at stufft.io  Wed May 31 15:20:51 2017
From: donald at stufft.io (Donald Stufft)
Date: Wed, 31 May 2017 15:20:51 -0400
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <1496257947.754070.994532544.49214D71@webmail.messagingengine.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
 <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
 <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>
 <CACac1F-pOZXxS84H+_B4_tQPTBGi9JZgyrH6V-B5GD9ktmXOgg@mail.gmail.com>
 <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io>
 <CACac1F-vgULOYhAo0JPAzsxfeBMow1RqhH2=Z3jJEWNOSxhXpw@mail.gmail.com>
 <FC2711EF-F610-41F4-BEF8-6D51380527DD@stufft.io>
 <CACac1F8+utdgQYHYNVu=NceV5Z+yUrMDWjMR+jdP3Yo7i8fxvQ@mail.gmail.com>
 <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io>
 <1496257947.754070.994532544.49214D71@webmail.messagingengine.com>
Message-ID: <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io>


> On May 31, 2017, at 3:12 PM, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> 
> On Wed, May 31, 2017, at 07:40 PM, Donald Stufft wrote:
>> Taking pip completely off the table a second, let?s take a look at tox. Tox?s default mode of operation is to produce a sdist. Now let?s say I?m writing a project that I want to use PEP 517 and get rid of setup.py, except now tox is broken with no path forward because PEP 517 doesn?t define how to produce a sdist. 
>> 
>> The same story is true for TravisCI?s PyPI deployment pipeline, as soon as any project starts depending on PEP 517, we completely break that feature for them without a path for them to fix it (besides writing a PEP of course).
>> 
>> The same is true for Gem Fury?s private PyPI repositories where you can ``git push fury`` and have them build a sdist automatically for you.
> 
> These tools are all things that the developers of the project choose to use, however. I don't use them, so I'm happy enough to get rid of setup.py and not have a standard interface to building sdists. Developers who do use them will want to stick with setup.py until there's a standard way to build an sdist - or a tool like tox may add support for going via wheels instead of via sdist.
> 
> So PEP 517 may not be useful to *everyone* without standardising a way to build sdists, but it is still useful for many projects, and I don't think it prevents a later PEP from standardising a way to build sdists.


The most likely outcome if PEP 517 is implemented as defined and people who aren?t steeped in packaging lore hear about is, is they get excited about being able to kill setup.py, they implement it, they find out some tool they depend on doesn?t work and can?t work with it, they get discouraged and start filling issues. Ideally those issues are filed on the tool that implemented PEP 517, but most likely it will be filed on tox, Travis, GemFury, etc. 

I am struggling to figure out where there is opposition to simply exposing something in a standard way, that you were already planning on implementing anyways.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170531/fa784d52/attachment.html>

From p.f.moore at gmail.com  Wed May 31 15:38:08 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 31 May 2017 20:38:08 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
 <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
 <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>
 <CACac1F-pOZXxS84H+_B4_tQPTBGi9JZgyrH6V-B5GD9ktmXOgg@mail.gmail.com>
 <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io>
 <CACac1F-vgULOYhAo0JPAzsxfeBMow1RqhH2=Z3jJEWNOSxhXpw@mail.gmail.com>
 <FC2711EF-F610-41F4-BEF8-6D51380527DD@stufft.io>
 <CACac1F8+utdgQYHYNVu=NceV5Z+yUrMDWjMR+jdP3Yo7i8fxvQ@mail.gmail.com>
 <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io>
 <1496257947.754070.994532544.49214D71@webmail.messagingengine.com>
 <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io>
Message-ID: <CACac1F_NUmuP157ALOfGAq0wgjii2rcf3dtHtdJbyYbWcZeUnQ@mail.gmail.com>

On 31 May 2017 at 20:20, Donald Stufft <donald at stufft.io> wrote:
> The most likely outcome if PEP 517 is implemented as defined and people who
> aren?t steeped in packaging lore hear about is, is they get excited about
> being able to kill setup.py, they implement it, they find out some tool they
> depend on doesn?t work and can?t work with it, they get discouraged and
> start filling issues. Ideally those issues are filed on the tool that
> implemented PEP 517, but most likely it will be filed on tox, Travis,
> GemFury, etc.
>
> I am struggling to figure out where there is opposition to simply exposing
> something in a standard way, that you were already planning on implementing
> anyways.

There's a lot of baggage associated with the term sdist.

As a suggestion - if backends supplied a prepare_build_files hook,
someone could write a pretty trivial tool that called that hook. Then
call the build_wheel_metadata hook to get some details to put into
PKG-INFO, zip the result up and call it a sdist. You could dump a
setup.py replacement in there that used PEP 517 hooks to implement the
setup.py interface, if you wanted.

Given how vaguely defined a sdist is, it would be hard to argue that
the result is *not* a sdist. I'm not sure how much further you're
going to insist on going. You no longer create a sdist using "setup.py
sdist", sure. But at some point the tools have to deal with setup.py
going away, so I don't see how that's a requirement forever.

If you really think we need to cover these use cases solidly (and you
have a point, I'm not saying they are irrelevant at all) then maybe we
need to get input from the tox/travis/gemfury devs, to see what they
actually think, rather than trying to guess their requirements?

Paul

PS None of this means I am in any way in favour of making it seem like
we're OK with projects not providing sdists (in some form or other).
We're an open source community, and I think publishing sources
alongside your binaries, is key to that. A link to an external source
code repository isn't sufficient, IMO (the repo could die, like Google
code did).

From thomas at kluyver.me.uk  Wed May 31 15:45:27 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Wed, 31 May 2017 20:45:27 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
 <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
 <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>
 <CACac1F-pOZXxS84H+_B4_tQPTBGi9JZgyrH6V-B5GD9ktmXOgg@mail.gmail.com>
 <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io>
 <CACac1F-vgULOYhAo0JPAzsxfeBMow1RqhH2=Z3jJEWNOSxhXpw@mail.gmail.com>
 <FC2711EF-F610-41F4-BEF8-6D51380527DD@stufft.io>
 <CACac1F8+utdgQYHYNVu=NceV5Z+yUrMDWjMR+jdP3Yo7i8fxvQ@mail.gmail.com>
 <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io>
 <1496257947.754070.994532544.49214D71@webmail.messagingengine.com>
 <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io>
Message-ID: <1496259927.763203.994563560.15C47384@webmail.messagingengine.com>

On Wed, May 31, 2017, at 08:20 PM, Donald Stufft wrote:
> I am struggling to figure out where there is opposition to simply
> exposing something in a standard way, that you were already planning
> on implementing anyways.
I have issues with it because:
1. Building a *release-quality* sdist is a complicated topic in its own
   right, and I'd like to move forwards with what we've already defined
   for building wheels without getting mired in that debate.2. I think it's a mistake to conflate "get the files we need to build
   this project" with "make a source distribution", and I don't want one
   hook to be used for both operations. Flit can do the former very
   easily, and in situations where it cannot do the latter.  a) If pip uses the hook for the former purpose, and I implement it
     with that in mind, it will give poor results if other tools use it
     to release an sdist.  b) If the hook properly makes an sdist, it will fail in situations
     where it can do what pip needs, and it will be unnecessarily slow
     even where it succeeds.
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170531/0f327844/attachment-0001.html>

From donald at stufft.io  Wed May 31 16:07:16 2017
From: donald at stufft.io (Donald Stufft)
Date: Wed, 31 May 2017 16:07:16 -0400
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CACac1F_NUmuP157ALOfGAq0wgjii2rcf3dtHtdJbyYbWcZeUnQ@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
 <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
 <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>
 <CACac1F-pOZXxS84H+_B4_tQPTBGi9JZgyrH6V-B5GD9ktmXOgg@mail.gmail.com>
 <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io>
 <CACac1F-vgULOYhAo0JPAzsxfeBMow1RqhH2=Z3jJEWNOSxhXpw@mail.gmail.com>
 <FC2711EF-F610-41F4-BEF8-6D51380527DD@stufft.io>
 <CACac1F8+utdgQYHYNVu=NceV5Z+yUrMDWjMR+jdP3Yo7i8fxvQ@mail.gmail.com>
 <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io>
 <1496257947.754070.994532544.49214D71@webmail.messagingengine.com>
 <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io>
 <CACac1F_NUmuP157ALOfGAq0wgjii2rcf3dtHtdJbyYbWcZeUnQ@mail.gmail.com>
Message-ID: <50D252B7-06C6-4096-9915-37F34F55363F@stufft.io>


> On May 31, 2017, at 3:38 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 31 May 2017 at 20:20, Donald Stufft <donald at stufft.io> wrote:
>> The most likely outcome if PEP 517 is implemented as defined and people who
>> aren?t steeped in packaging lore hear about is, is they get excited about
>> being able to kill setup.py, they implement it, they find out some tool they
>> depend on doesn?t work and can?t work with it, they get discouraged and
>> start filling issues. Ideally those issues are filed on the tool that
>> implemented PEP 517, but most likely it will be filed on tox, Travis,
>> GemFury, etc.
>> 
>> I am struggling to figure out where there is opposition to simply exposing
>> something in a standard way, that you were already planning on implementing
>> anyways.
> 
> There's a lot of baggage associated with the term sdist.

I mean, PEP 517 explicitly redefines ?sdist? to mean ?a tarball that includes a pyproject.toml, setup.py not required?, so by accepting PEP 517 we?re accepting that a sdist is either the thing that we conventionally call a sdist, or a thing that is similar to it, but instead of a setup.py it contains a pyproject.toml. That?s from https://www.python.org/dev/peps/pep-0517/#source-distributions <https://www.python.org/dev/peps/pep-0517/#source-distributions> which states:

For now, we continue with the legacy sdist format which is mostly
undefined, but basically comes down to: a file named
``{NAME}-{VERSION}.{EXT}``, which unpacks into a buildable source tree
called ``{NAME}-{VERSION}/``. Traditionally these have always
contained ``setup.py``\-style source trees; we now allow them to also
contain ``pyproject.toml``\-style source trees.

Integration frontends require that an sdist named
``{NAME}-{VERSION}.{EXT}`` will generate a wheel named
``{NAME}-{VERSION}-{COMPAT-INFO}.whl``.

If we want to more rigorously define a sdist that?s fine, we can go down that rabbit hole, if we want to remove that and say for something to be a sdist they still have to have a setup.py that supports the expected commands and oh by the way you can use this new thing in PEP 517 to declare statically an alternative build tool that some setup_requires tool will use to replace bdist_wheel, then that?s fine too. Hell, If it wants to get rid of the sdist terminology completely and make a new format called a source wheel or a bagofiles or whatever, that?s fine too.

The PEP right now seems to want it both ways, it wants to declare this thing without the conventional interfaces is a sdist, while ignoring the fact that people are using those conventional interfaces. For me, if the PEP wants it?s new thing to be a sdist, then it needs to handle that case and anything else doesn?t sit right with me.

> 
> As a suggestion - if backends supplied a prepare_build_files hook,
> someone could write a pretty trivial tool that called that hook. Then
> call the build_wheel_metadata hook to get some details to put into
> PKG-INFO, zip the result up and call it a sdist. You could dump a
> setup.py replacement in there that used PEP 517 hooks to implement the
> setup.py interface, if you wanted.
> 
> Given how vaguely defined a sdist is, it would be hard to argue that
> the result is *not* a sdist. I'm not sure how much further you're
> going to insist on going. You no longer create a sdist using "setup.py
> sdist", sure. But at some point the tools have to deal with setup.py
> going away, so I don't see how that's a requirement forever.

That isn?t really the same though, prepare_build_files hook is presumably not going to be including things like the LICENSE file, documentation which are things you?d want in an sdist, but which would be non-obvious to include in the said prepare_build_files hook and is likely going to be vastly different than a sdist produced by ``myhypotheticalbuildtool sdist``, unless prepare_build_files hook is basically exactly the same as build_sdist in all but name.

I?m not stating that we need to support ``setup.py sdist`` forever, I?m saying we need to support a generic way to build a sdist to replace ``setup.py sdist``.

> 
> If you really think we need to cover these use cases solidly (and you
> have a point, I'm not saying they are irrelevant at all) then maybe we
> need to get input from the tox/travis/gemfury devs, to see what they
> actually think, rather than trying to guess their requirements?

I do think we need to cover them solidly yes.

I?m happy to try and bring them in, but as far as Travis/GemFury goes, I think their use case is pretty simple, given a python project using the hypothetical PEP 517, they want to produce a sdist and either serve it directly (GemFury) or publish it to PyPI (Travis). If we don?t add build_sdist, and every project implements their own mechanism for generating a sdist, then either they, or some wrapper tool has to know and understand every possible build tool they might use and how to get it to build a sdist (OR they have to start adding configruration for people to instruct them how to build a sdist for their specific project using whatever tool they?re using).

As far as Tox goes, I will poke them, but I?m pretty sure the answer is wanting to test the thing that is being installed as it would be installed from PyPI, to try and mitigate packaging errors where you forget a file in your MANIFEST.in or so.

> 
> Paul
> 
> PS None of this means I am in any way in favour of making it seem like
> we're OK with projects not providing sdists (in some form or other).
> We're an open source community, and I think publishing sources
> alongside your binaries, is key to that. A link to an external source
> code repository isn't sufficient, IMO (the repo could die, like Google
> code did).


Unfortunately, that is exactly how I think PEP 517 will end up, since it only requires building wheels, people are going to implement ones that only build wheels, and we have nothing to indicate to them that they generally shouldn?t do that (without a specific reason). On the contrary, the indications point to sdist not being important enough to even be given an option to build them at all, requiring tool authors to go out of their way to decide to add it.


?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170531/f9939165/attachment.html>

From donald at stufft.io  Wed May 31 16:16:07 2017
From: donald at stufft.io (Donald Stufft)
Date: Wed, 31 May 2017 16:16:07 -0400
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <1496259927.763203.994563560.15C47384@webmail.messagingengine.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
 <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
 <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>
 <CACac1F-pOZXxS84H+_B4_tQPTBGi9JZgyrH6V-B5GD9ktmXOgg@mail.gmail.com>
 <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io>
 <CACac1F-vgULOYhAo0JPAzsxfeBMow1RqhH2=Z3jJEWNOSxhXpw@mail.gmail.com>
 <FC2711EF-F610-41F4-BEF8-6D51380527DD@stufft.io>
 <CACac1F8+utdgQYHYNVu=NceV5Z+yUrMDWjMR+jdP3Yo7i8fxvQ@mail.gmail.com>
 <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io>
 <1496257947.754070.994532544.49214D71@webmail.messagingengine.com>
 <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io>
 <1496259927.763203.994563560.15C47384@webmail.messagingengine.com>
Message-ID: <AD07342A-C7AC-457D-86C3-DA453B26E638@stufft.io>


> On May 31, 2017, at 3:45 PM, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> 
> On Wed, May 31, 2017, at 08:20 PM, Donald Stufft wrote:
>> I am struggling to figure out where there is opposition to simply exposing something in a standard way, that you were already planning on implementing anyways.
> 
> I have issues with it because:
> 1. Building a *release-quality* sdist is a complicated topic in its own right, and I'd like to move forwards with what we've already defined for building wheels without getting mired in that debate.

How you build the release-quality sdist isn?t really of concern of PEP 517 any more than building a release quality wheel is, it?s up to the build tool to implement that as it makes sense for them.

> 2. I think it's a mistake to conflate "get the files we need to build this project" with "make a source distribution", and I don't want one hook to be used for both operations. Flit can do the former very easily, and in situations where it cannot do the latter.
>   a) If pip uses the hook for the former purpose, and I implement it with that in mind, it will give poor results if other tools use it to release an sdist.
>   b) If the hook properly makes an sdist, it will fail in situations where it can do what pip needs, and it will be unnecessarily slow even where it succeeds.

I could see this as an argument that the PEP should have *both* a build_sdist and a prepare_build_files hook, if you don?t think that the build_sdist hook is suitable on it?s own. I?m not sure how I feel about that off the top of my head, but I *think* I would be okay adding the mandatory build_sdist command and an optional prepare_build_files hook, with the semantics being that if prepare_build_files is NOT defined, then it is acceptable for a tool like pip to use build_sdist for this purpose and if prepare_build_files IS defined, then the resulting wheel from build_wheel should not meaningfully differ from what would have been produced from build_sdist + build_wheel.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170531/f67da721/attachment-0001.html>

From thomas at kluyver.me.uk  Wed May 31 17:13:29 2017
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Wed, 31 May 2017 22:13:29 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <AD07342A-C7AC-457D-86C3-DA453B26E638@stufft.io>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
 <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
 <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>
 <CACac1F-pOZXxS84H+_B4_tQPTBGi9JZgyrH6V-B5GD9ktmXOgg@mail.gmail.com>
 <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io>
 <CACac1F-vgULOYhAo0JPAzsxfeBMow1RqhH2=Z3jJEWNOSxhXpw@mail.gmail.com>
 <FC2711EF-F610-41F4-BEF8-6D51380527DD@stufft.io>
 <CACac1F8+utdgQYHYNVu=NceV5Z+yUrMDWjMR+jdP3Yo7i8fxvQ@mail.gmail.com>
 <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io>
 <1496257947.754070.994532544.49214D71@webmail.messagingengine.com>
 <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io>
 <1496259927.763203.994563560.15C47384@webmail.messagingengine.com>
 <AD07342A-C7AC-457D-86C3-DA453B26E638@stufft.io>
Message-ID: <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com>

On Wed, May 31, 2017, at 09:16 PM, Donald Stufft wrote:
> How you build the release-quality sdist isn?t really of concern of PEP
> 517 any more than building a release quality wheel is, it?s up to the
> build tool to implement that as it makes sense for them.
But if we have a hook for building something called an sdist, we need to
define what an sdist is. The definition you're referring to is a
functional definition sufficient for a tool that only wants to install
it, but it doesn't cover what an sdist means or how it fits into
workflows.
> I could see this as an argument that the PEP should have *both* a
> build_sdist and a prepare_build_files hook, if you don?t think that
> the build_sdist hook is suitable on it?s own.
I would prefer that to the the current status of one hook that tries to
cover both use cases.
I'd still rather specify prepare_build_files in this PEP, and leave the
sdist hook to a later PEP. I don't see a great need for a standard
interface to building an sdist: the developer doing that by calling
their build tool directly seems adequate for the majority of use cases.
Thomas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170531/a1a3af13/attachment.html>

From jonwayne at google.com  Wed May 31 18:06:20 2017
From: jonwayne at google.com (Jon Wayne Parrott)
Date: Wed, 31 May 2017 22:06:20 +0000
Subject: [Distutils] Proposal: moving PyPA projects to pydoctheme
In-Reply-To: <CANkHFr_zjjhHbLp_Yi03bRFVbY0BWPF7Qx7vfZc-ir7wN5NXng@mail.gmail.com>
References: <CAN6JEy=usF+Z8FvjMV2BAkohJu5umvec6=bNX-+t6nZyoWGjcw@mail.gmail.com>
 <CAPJVwBmyp+Q6zaGny8k7cZ_otQ3avDA=piXL0JXABaY7nA10LA@mail.gmail.com>
 <CAN6JEymxBf+ac2p4fWMWj_kHu_B=DA3M-ySm9CsvDst6sc2DpQ@mail.gmail.com>
 <CAPJVwBm+EEmRUJ2yib_w8=gbpUdSvB6KaV2x94O98_d6wCZ69A@mail.gmail.com>
 <CADiSq7expwQKToHZTJ9p_jTVTOz-b=Fpn-rAOOPN5cEzhaT+eA@mail.gmail.com>
 <CAN6JEymq=4om9jHfnxtrz9bsAqdKE7gOQ6nWMTufdv9HgvwBZg@mail.gmail.com>
 <CANkHFr_zjjhHbLp_Yi03bRFVbY0BWPF7Qx7vfZc-ir7wN5NXng@mail.gmail.com>
Message-ID: <CAN6JEykDzxMvjtoPTjF8p4Ab-m_XuavP0T6pwKosJQQYQuZL0w@mail.gmail.com>

I'm not seeing the sticky sidebar in the Python 3 docs?
https://docs.python.org/3/library/stdtypes.html

On Sat, May 27, 2017 at 4:51 AM Ionel Cristian M?rie? <contact at ionelmc.ro>
wrote:

> Interestingly this doesn't have the "sticky" sidebar (eg: it moves as you
> scroll page) as the py3 docs have. This is my attempt to package it into a
> reusable theme btw:
> http://ionelmc.github.io/sphinx-py3doc-enhanced-theme/bare/reference/foo.bar.html
>
> You probably don't want to use that directly but maybe there is stuff you
> can just copy from it.
>
>
> Thanks,
> -- Ionel Cristian M?rie?, http://blog.ionelmc.ro
>
> On Fri, May 26, 2017 at 7:32 PM, Jon Wayne Parrott via Distutils-SIG <
> distutils-sig at python.org> wrote:
>
>> Awesome, more good feedback. I've made some stylesheet adjustments to
>> highlight both the current page (by bolding) and the current section (by
>> coloring links). Let me know on the PR if you have alternative
>> recommendations (leaving out the current section coloring is actually fine,
>> IMO). Staged again here (
>> http://temp.theadora.io/pypug-pydoctheme/self_hosted_repository.html).
>>
>> Since we mostly seem to all be in agreement, can someone who's an admin
>> on the PyPA org create the pypa-theme repository for me?
>>
>> On Fri, May 26, 2017 at 5:13 AM Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>>> Thanks for working on this, Jon!
>>>
>>> On 26 May 2017 at 17:06, Nathaniel Smith <njs at pobox.com> wrote:
>>> > Further nitpick: the new version shows the whole ToC, but doesn't do
>>> > anything to mark which part of it corresponds to the current page (cf
>>> > the RTD theme's grey box), which makes it hard to orient oneself when
>>> > looking at it. Some sort of "you are here" indicator would help :-)
>>>
>>> While the subtree expansion does provide a bit of that, I do agree
>>> that some additional contrast between the expanded section and the
>>> rest of the ToC would be beneficial.
>>>
>>> Cheers,
>>> Nick.
>>>
>>> --
>>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170531/987d06c3/attachment.html>

From p.f.moore at gmail.com  Wed May 31 18:48:14 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 31 May 2017 23:48:14 +0100
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
 <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
 <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>
 <CACac1F-pOZXxS84H+_B4_tQPTBGi9JZgyrH6V-B5GD9ktmXOgg@mail.gmail.com>
 <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io>
 <CACac1F-vgULOYhAo0JPAzsxfeBMow1RqhH2=Z3jJEWNOSxhXpw@mail.gmail.com>
 <FC2711EF-F610-41F4-BEF8-6D51380527DD@stufft.io>
 <CACac1F8+utdgQYHYNVu=NceV5Z+yUrMDWjMR+jdP3Yo7i8fxvQ@mail.gmail.com>
 <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io>
 <1496257947.754070.994532544.49214D71@webmail.messagingengine.com>
 <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io>
 <1496259927.763203.994563560.15C47384@webmail.messagingengine.com>
 <AD07342A-C7AC-457D-86C3-DA453B26E638@stufft.io>
 <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com>
Message-ID: <CACac1F9wNGDVKMbv0HZB8ezw9sqFk9OzAtP3bxHFb-wrTYViFg@mail.gmail.com>

On 31 May 2017 at 22:13, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
>
> But if we have a hook for building something called an sdist, we need to
> define what an sdist is.

OK, so can we do that?

At the moment, we have a de facto definition of a sdist - it's
something with a setup.py, some metadata defined by the metadata PEPs
(but implemented prettly loosely, so people don't actually rely on it)
and the files needed to build the project. Plus other stuff like
LICENSE that's important, but defined by the project saying it should
be included. Consumers of sdists are allowed to unpack them and run
any of the setup.py commands on them. They can in theory inspect the
metadata, but in practice don't. Call that the legacy sdist format.

What do consumers of the sdist format want to do? I don't actually
know, but my guess is that they just want to be able to install the
sdist. We presumably don't want to preserve the setup.py interface so
they need to allow for a new interface. What's wrong with "pip install
<file>"? They also want to publish the sdist to PyPI - so they need to
name it according to the current convention. Anything else?

Call this the post-PEP 517 setup.py. It's still not fully
standardised, it's an underspecified de facto standard, but "something
that follows current naming conventions and can be installed via pip
install filename" could be something that will do for now, until we
want to fully standardise sdist 2.0, with static metadata and all that
stuff. And as an additional benefit, all legacy sdists already conform
to this "spec".

I 100% agree that the current vagueness around what a sdist is, and
what tools can expect to do with them, is horribly unsatisfactory. But
to make any progress we have to discard the "exposes a setup.py
interface" rule. That's all we need for now. Longer term, we need a
formal spec. But *for now*, can we manage by replacing the setup.py
interface with an "installable by pip" interface? Or does anyone have
an alternative "good enough for now" definition of a sdist we can
agree on?

If we can do this, we can move forward. Otherwise, I fear this
discussion is going to stall with another "try to solve all the
problems at once" deadlock.

Paul

From fred at fdrake.net  Wed May 31 18:58:12 2017
From: fred at fdrake.net (Fred Drake)
Date: Wed, 31 May 2017 18:58:12 -0400
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CACac1F9wNGDVKMbv0HZB8ezw9sqFk9OzAtP3bxHFb-wrTYViFg@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
 <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
 <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>
 <CACac1F-pOZXxS84H+_B4_tQPTBGi9JZgyrH6V-B5GD9ktmXOgg@mail.gmail.com>
 <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io>
 <CACac1F-vgULOYhAo0JPAzsxfeBMow1RqhH2=Z3jJEWNOSxhXpw@mail.gmail.com>
 <FC2711EF-F610-41F4-BEF8-6D51380527DD@stufft.io>
 <CACac1F8+utdgQYHYNVu=NceV5Z+yUrMDWjMR+jdP3Yo7i8fxvQ@mail.gmail.com>
 <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io>
 <1496257947.754070.994532544.49214D71@webmail.messagingengine.com>
 <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io>
 <1496259927.763203.994563560.15C47384@webmail.messagingengine.com>
 <AD07342A-C7AC-457D-86C3-DA453B26E638@stufft.io>
 <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com>
 <CACac1F9wNGDVKMbv0HZB8ezw9sqFk9OzAtP3bxHFb-wrTYViFg@mail.gmail.com>
Message-ID: <CAFT4OTHzHBtneRUUA9FROC6chU_t6w12QyYJv=4az6nXXRrMoQ@mail.gmail.com>

On Wed, May 31, 2017 at 6:48 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> What do consumers of the sdist format want to do? I don't actually know,
...
> They also want to publish the sdist to PyPI - so they need to
> name it according to the current convention.

I think we can rule this out for consumers of the sdist; only the
provider cares about this.  If you started from an sdist, you don't
need to publish it to PyPI (though you may want to "publish" to some
repository within your own organization).


  -Fred

-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein

From wes.turner at gmail.com  Wed May 31 19:40:08 2017
From: wes.turner at gmail.com (Wes Turner)
Date: Wed, 31 May 2017 18:40:08 -0500
Subject: [Distutils] Proposal: moving PyPA projects to pydoctheme
In-Reply-To: <CANkHFr_zjjhHbLp_Yi03bRFVbY0BWPF7Qx7vfZc-ir7wN5NXng@mail.gmail.com>
References: <CAN6JEy=usF+Z8FvjMV2BAkohJu5umvec6=bNX-+t6nZyoWGjcw@mail.gmail.com>
 <CAPJVwBmyp+Q6zaGny8k7cZ_otQ3avDA=piXL0JXABaY7nA10LA@mail.gmail.com>
 <CAN6JEymxBf+ac2p4fWMWj_kHu_B=DA3M-ySm9CsvDst6sc2DpQ@mail.gmail.com>
 <CAPJVwBm+EEmRUJ2yib_w8=gbpUdSvB6KaV2x94O98_d6wCZ69A@mail.gmail.com>
 <CADiSq7expwQKToHZTJ9p_jTVTOz-b=Fpn-rAOOPN5cEzhaT+eA@mail.gmail.com>
 <CAN6JEymq=4om9jHfnxtrz9bsAqdKE7gOQ6nWMTufdv9HgvwBZg@mail.gmail.com>
 <CANkHFr_zjjhHbLp_Yi03bRFVbY0BWPF7Qx7vfZc-ir7wN5NXng@mail.gmail.com>
Message-ID: <CACfEFw_q5iqU58UzsSoXkqjzxfwKSeXhUfP4YHa3d9PoNOHZdw@mail.gmail.com>

On Saturday, May 27, 2017, Ionel Cristian M?rie? via Distutils-SIG <
distutils-sig at python.org> wrote:

> Interestingly this doesn't have the "sticky" sidebar (eg: it moves as you
> scroll page) as the py3 docs have. This is my attempt to package it into a
> reusable theme btw: http://ionelmc.github.io/sphinx-py3doc-enhanced-theme/
> bare/reference/foo.bar.html
>
> You probably don't want to use that directly but maybe there is stuff you
> can just copy from it.
>

Here's a (somewhat hackish) sticky sidebar w/:

- sphinxjp.themes.basicstrap
  - https://pypi.python.org/pypi/sphinxjp.themes.basicstrap/
  - responsive
    - toggle-able mobile TOC
- (toggle) show :visited links
- (toggle) open external links in new tab
- jquery.scrollTo the current TOC heading
- sphinxcontrib-srclinks
  - https://pypi.python.org/pypi/sphinxcontrib-srclinks
  - project homepage, src hyperlinks
  - git/hg clone commands


Example: https://wrdrd.github.io/docs/tools/#sphinx

https://github.com/wrdrd/docs/blob/master/Makefile
- make localjs
  https://github.com/wrdrd/docs/tree/master/docs/_static/js
 - newtab.js - open links in a new tab (and store the setting in a cookie)
 - sidenav-affix.js - sticky sidebar
 - sidenav-scrollto.js - indicate current TOC heading (and scroll the TOC
entry into view only of it's out of the viewport)

If any of these are useful, please feel free or just let me know and I'll
go ahead and package them up.

... https://github.com/yoloseem/awesome-sphinxdoc


>
> Thanks,
> -- Ionel Cristian M?rie?, http://blog.ionelmc.ro
>
> On Fri, May 26, 2017 at 7:32 PM, Jon Wayne Parrott via Distutils-SIG <
> distutils-sig at python.org
> <javascript:_e(%7B%7D,'cvml','distutils-sig at python.org');>> wrote:
>
>> Awesome, more good feedback. I've made some stylesheet adjustments to
>> highlight both the current page (by bolding) and the current section (by
>> coloring links). Let me know on the PR if you have alternative
>> recommendations (leaving out the current section coloring is actually fine,
>> IMO). Staged again here (http://temp.theadora.io/pypug
>> -pydoctheme/self_hosted_repository.html).
>>
>> Since we mostly seem to all be in agreement, can someone who's an admin
>> on the PyPA org create the pypa-theme repository for me?
>>
>> On Fri, May 26, 2017 at 5:13 AM Nick Coghlan <ncoghlan at gmail.com
>> <javascript:_e(%7B%7D,'cvml','ncoghlan at gmail.com');>> wrote:
>>
>>> Thanks for working on this, Jon!
>>>
>>> On 26 May 2017 at 17:06, Nathaniel Smith <njs at pobox.com
>>> <javascript:_e(%7B%7D,'cvml','njs at pobox.com');>> wrote:
>>> > Further nitpick: the new version shows the whole ToC, but doesn't do
>>> > anything to mark which part of it corresponds to the current page (cf
>>> > the RTD theme's grey box), which makes it hard to orient oneself when
>>> > looking at it. Some sort of "you are here" indicator would help :-)
>>>
>>> While the subtree expansion does provide a bit of that, I do agree
>>> that some additional contrast between the expanded section and the
>>> rest of the ToC would be beneficial.
>>>
>>> Cheers,
>>> Nick.
>>>
>>> --
>>> Nick Coghlan   |   ncoghlan at gmail.com
>>> <javascript:_e(%7B%7D,'cvml','ncoghlan at gmail.com');>   |   Brisbane,
>>> Australia
>>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> <javascript:_e(%7B%7D,'cvml','Distutils-SIG at python.org');>
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170531/da365592/attachment.html>

From donald at stufft.io  Wed May 31 20:08:51 2017
From: donald at stufft.io (Donald Stufft)
Date: Wed, 31 May 2017 20:08:51 -0400
Subject: [Distutils] Provisionally accepting PEP 517's declarative build
 system interface
In-Reply-To: <CACac1F9wNGDVKMbv0HZB8ezw9sqFk9OzAtP3bxHFb-wrTYViFg@mail.gmail.com>
References: <CADiSq7fuSAng3xg1iTCt=Nkp5WS_v98y9P4_rehLfFsku3qG5g@mail.gmail.com>
 <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io>
 <CADiSq7cmhB03gm8m7hBnr-oid-D-VK+uBXy-R39PKvmO78n9=A@mail.gmail.com>
 <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io>
 <CADiSq7f5Fk4i1Nfv2+NswDSeQt6eZsNSxr8_4SS-BGctXBr9nw@mail.gmail.com>
 <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io>
 <CADiSq7eaURMoH7FfiKjVPsuJ6N7Rt=borkaSMiL7ELpinK=y0w@mail.gmail.com>
 <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io>
 <CADiSq7dd7e8A1xOGwC5U19wDnU8QYosq1-fvCoFW5PTGCMQ9+A@mail.gmail.com>
 <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com>
 <CACac1F_FMzPSNO2-UP8U+HDhRR3J83U6vSRjeorAXRDe75o4oQ@mail.gmail.com>
 <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com>
 <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com>
 <CACac1F-pOZXxS84H+_B4_tQPTBGi9JZgyrH6V-B5GD9ktmXOgg@mail.gmail.com>
 <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io>
 <CACac1F-vgULOYhAo0JPAzsxfeBMow1RqhH2=Z3jJEWNOSxhXpw@mail.gmail.com>
 <FC2711EF-F610-41F4-BEF8-6D51380527DD@stufft.io>
 <CACac1F8+utdgQYHYNVu=NceV5Z+yUrMDWjMR+jdP3Yo7i8fxvQ@mail.gmail.com>
 <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io>
 <1496257947.754070.994532544.49214D71@webmail.messagingengine.com>
 <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io>
 <1496259927.763203.994563560.15C47384@webmail.messagingengine.com>
 <AD07342A-C7AC-457D-86C3-DA453B26E638@stufft.io>
 <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com>
 <CACac1F9wNGDVKMbv0HZB8ezw9sqFk9OzAtP3bxHFb-wrTYViFg@mail.gmail.com>
Message-ID: <0F77923A-B251-4A44-BC72-D717E0AD6CBC@stufft.io>


> On May 31, 2017, at 6:48 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 31 May 2017 at 22:13, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
>> 
>> But if we have a hook for building something called an sdist, we need to
>> define what an sdist is.
> 
> OK, so can we do that?
> 
> At the moment, we have a de facto definition of a sdist - it's
> something with a setup.py, some metadata defined by the metadata PEPs
> (but implemented prettly loosely, so people don't actually rely on it)
> and the files needed to build the project. Plus other stuff like
> LICENSE that's important, but defined by the project saying it should
> be included. Consumers of sdists are allowed to unpack them and run
> any of the setup.py commands on them. They can in theory inspect the
> metadata, but in practice don't. Call that the legacy sdist format.
> 
> What do consumers of the sdist format want to do? I don't actually
> know, but my guess is that they just want to be able to install the
> sdist. We presumably don't want to preserve the setup.py interface so
> they need to allow for a new interface. What's wrong with "pip install
> <file>"? They also want to publish the sdist to PyPI - so they need to
> name it according to the current convention. Anything else?
> 
> Call this the post-PEP 517 setup.py. It's still not fully
> standardised, it's an underspecified de facto standard, but "something
> that follows current naming conventions and can be installed via pip
> install filename" could be something that will do for now, until we
> want to fully standardise sdist 2.0, with static metadata and all that
> stuff. And as an additional benefit, all legacy sdists already conform
> to this "spec".
> 
> I 100% agree that the current vagueness around what a sdist is, and
> what tools can expect to do with them, is horribly unsatisfactory. But
> to make any progress we have to discard the "exposes a setup.py
> interface" rule. That's all we need for now. Longer term, we need a
> formal spec. But *for now*, can we manage by replacing the setup.py
> interface with an "installable by pip" interface? Or does anyone have
> an alternative "good enough for now" definition of a sdist we can
> agree on?
> 
> If we can do this, we can move forward. Otherwise, I fear this
> discussion is going to stall with another "try to solve all the
> problems at once" deadlock.
> 
> Paul


I think I?m -0 on spelling it out as ?it?s whatever pip can install? rather than just codifying what the defacto rules for that is, something like:

?

A sdist is a .tar.gz or a .zip file with a directory structure like (along with whatever additional files the project needs in the sdist):

.
??? {name}-{version}
    ??? PKG-INFO
    ??? setup.py OR pyproject.toml


If a sdist contains a pyproject.toml file that contains a build-system.build-backend key, then it is a PEP 517 style sdist and MUST be processed using the API as defined in PEP 517. Otherwise it is a legacy distutils/setuptools style sdist and MUST be processed by calling setup.py. PEP 517 sdists MAY contain a setup.py for compatibility with tooling that does not yet understand PEP 517.

PKG-INFO should loosely be a PEP 345 style METADATA file and the errata located at https://packaging.python.org/specifications/#package-distribution-metadata <https://packaging.python.org/specifications/#package-distribution-metadata>.

A sdist MUST following the {name}-{version}.{ext} naming scheme, where {ext} MUST be either .tar.gz or .zip matching the respective container/compression format being used. Both {name} and {version} MUST have any - characters escaped to a _ to match the escaping done by Wheel. Thus a sdist for a project named foo-bar with version 1.0-2 which is using a .tar.gz container for the sdist would produce a file named foo_bar-1.0_2.tar.gz.

?

I think this should cover the case of actually making the project pip installable (assuming of course the setup.py or build backend doesn?t do something silly like always sys.exit(1) instead of produce the expected outcome) as well as allow twine to upload a sdist produced by this (since it reads the PKG-INFO file). The delta from the defacto standard today is basically swapping the setup.py for pyproject.toml and the escaping of the filename [1]. This should not require much more of the backends than producing a wheel does, since the PKG-INFO file is essentially just the METADATA file from within a wheel (although if people are dynamically generating dependencies or something they may want to omit them rather than give misleading information in PKG-INFO).

A future PEP can solve the problem of a sdist 2.0 that has a nicer interface than that or better metadata or whatever. This just represents a fairly minimal evolution of what currently exists today to support the changes needed for PEP 517.

[1] We don?t _need_ to do this, but currently you can?t tell if foo-1-1.tar.gz is (foo-1, 1) or (foo, 1-1) and moving to mandate escaped names can try to solve that problem going into the future using the heuristic of if there is > 1 dash character in the filename, then it was not escaped and we have to fall back to the somewhat error prone context sensitive parsing of the filename. Certainly we could say that it?s out of scope for PEP 517 and leave it at that, but since it?s such a minor change I felt it wouldn?t be a big deal to add it here.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170531/df252923/attachment-0001.html>

From contact at ionelmc.ro  Wed May 31 20:54:24 2017
From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=)
Date: Thu, 1 Jun 2017 03:54:24 +0300
Subject: [Distutils] Proposal: moving PyPA projects to pydoctheme
In-Reply-To: <CAN6JEykDzxMvjtoPTjF8p4Ab-m_XuavP0T6pwKosJQQYQuZL0w@mail.gmail.com>
References: <CAN6JEy=usF+Z8FvjMV2BAkohJu5umvec6=bNX-+t6nZyoWGjcw@mail.gmail.com>
 <CAPJVwBmyp+Q6zaGny8k7cZ_otQ3avDA=piXL0JXABaY7nA10LA@mail.gmail.com>
 <CAN6JEymxBf+ac2p4fWMWj_kHu_B=DA3M-ySm9CsvDst6sc2DpQ@mail.gmail.com>
 <CAPJVwBm+EEmRUJ2yib_w8=gbpUdSvB6KaV2x94O98_d6wCZ69A@mail.gmail.com>
 <CADiSq7expwQKToHZTJ9p_jTVTOz-b=Fpn-rAOOPN5cEzhaT+eA@mail.gmail.com>
 <CAN6JEymq=4om9jHfnxtrz9bsAqdKE7gOQ6nWMTufdv9HgvwBZg@mail.gmail.com>
 <CANkHFr_zjjhHbLp_Yi03bRFVbY0BWPF7Qx7vfZc-ir7wN5NXng@mail.gmail.com>
 <CAN6JEykDzxMvjtoPTjF8p4Ab-m_XuavP0T6pwKosJQQYQuZL0w@mail.gmail.com>
Message-ID: <CANkHFr8q2_=Vefg1hfyuu5_a7WamGyekQin8246ZcofjUM0NfA@mail.gmail.com>

On Thu, Jun 1, 2017 at 1:06 AM, Jon Wayne Parrott <jonwayne at google.com>
wrote:

> I'm not seeing the sticky sidebar in the Python 3 docs?


Hmmmm. It appears that sidebar.js is part of the sphinx base theme, not
sure what's going on in the python3 docs anymore ...?



Thanks,
-- Ionel Cristian M?rie?, http://blog.ionelmc.ro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170601/7a12b31f/attachment.html>

From anee.vesna at gmail.com  Wed May 31 11:02:15 2017
From: anee.vesna at gmail.com (Aneesh Kona)
Date: Wed, 31 May 2017 15:02:15 +0000
Subject: [Distutils] Installation problem
In-Reply-To: <CAN4+E8H0MN1UKOBL0Qt2h+uP2=qvOJ+f0b5=B9dBJ+MBb4A+ug@mail.gmail.com>
References: <CADPEtBY1no0iSRwwuhtji=u+1ayjzjY0p9riGfa7aRLzZ59fHQ@mail.gmail.com>
 <CAN4+E8H0MN1UKOBL0Qt2h+uP2=qvOJ+f0b5=B9dBJ+MBb4A+ug@mail.gmail.com>
Message-ID: <CADPEtBbo1+Wy96Dx6Jy1_6K3xQbatp+6dYfK=+UnBTVvfiLgoQ@mail.gmail.com>

Got it, thanks!
I fixed it.
On Wed, May 31, 2017 at 9:19 AM Robert T. McGibbon <rmcgibbo at gmail.com>
wrote:

> This isn't actually the right place to ask for installation help, Aneesh.
> Your best bet would be the python-list or python-help mailing list.
>
> -Robert
>
> On Tue, May 30, 2017 at 7:24 PM, Aneesh Kona <anee.vesna at gmail.com> wrote:
>
>> Hi,
>> I'm getting an error while installing software,
>> I did installed firstly but did in another in local disk,
>> but I want to install again, now it shows 'fatal error,' can you please
>> let me fix it, so that I can install again in C disk,
>> plz hand me in this particular,
>> Thanks.
>>
>> Venkata Aneesh Kona
>> Graduate Assistant
>> Department of Mining Engineering and Management
>> South Dakota School of Mines & Technology
>> @: venkataaneesh.kona at mines.sdsmt.edu
>> ph: 6823042622 <(682)%20304-2622>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
>
> --
> -Robert
>
-- 
Sent from Aneesh Iphone
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170531/4baaff72/attachment-0001.html>