From ncoghlan at gmail.com  Mon Aug  1 01:42:34 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 1 Aug 2016 15:42:34 +1000
Subject: [Distutils] cffi & Py_LIMITED_API
In-Reply-To: <CAG8k2+7RwPic+eUe3ir-LFnAua8714JiJ93JB8+D4MPKfpkz3A@mail.gmail.com>
References: <CAG8k2+7RwPic+eUe3ir-LFnAua8714JiJ93JB8+D4MPKfpkz3A@mail.gmail.com>
Message-ID: <CADiSq7cmQBVpESUi8w94-Ao=5ughpZymbKM-gmwWt6xKbWMpVQ@mail.gmail.com>

On 1 August 2016 at 05:27, Daniel Holth <dholth at gmail.com> wrote:
> The next version of cffi will contain small changes to generate code
> compliant with Python's Py_LIMITED_API:
> https://bitbucket.org/cffi/cffi/commits/8f867f5a869f
>
> (although cffi itself is not, the extensions it generates will be).

Very cool!

> If we also add an appropriate supported tag to pip ~= cp3.abi3.manylinux1
> and provide a way to name the generated DLL's appropriately, it may become
> possible to reduce the burden of distributing cffi extensions, especially
> for Windows. One compiled artifact should work on Python 3.2 and above.

Aye, that would be very promising. I guess the first step would be to
document the current steps involved in doing this manually? And then
figure out what tweaks would be needed to setuptools and pip to allow
it to be automated?

As an initial stab at that:

Status quo, on publication side:

- require minimum cffi version 1.8
- build with setuptools
- postprocessing step to rename shared library/DLL
- postprocessing step to regenerate renamed whl file with renamed SO/DLL

On consumption side:

- requires "cp3.abi3" to be included at appropriate points in
compatibility tag list

Which would make the necessary changes be:

- updating setuptools to somehow be Py_LIMITED_API aware when naming
built extensions
- updating wheel to somehow be Py_LIMITED_API aware when naming whl files
- if necessary, updating pip's compatibility tag list

Of those changes, only the "somehow be Py_LIMITED_API aware" sounds
potentially tricky to me, as I'd be surprised if there was any way
around requiring a new explicit setting in either setup.py or
setup.cfg.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From dholth at gmail.com  Mon Aug  1 09:36:06 2016
From: dholth at gmail.com (Daniel Holth)
Date: Mon, 01 Aug 2016 13:36:06 +0000
Subject: [Distutils] cffi & Py_LIMITED_API
In-Reply-To: <CADiSq7cmQBVpESUi8w94-Ao=5ughpZymbKM-gmwWt6xKbWMpVQ@mail.gmail.com>
References: <CAG8k2+7RwPic+eUe3ir-LFnAua8714JiJ93JB8+D4MPKfpkz3A@mail.gmail.com>
 <CADiSq7cmQBVpESUi8w94-Ao=5ughpZymbKM-gmwWt6xKbWMpVQ@mail.gmail.com>
Message-ID: <CAG8k2+5Hy8iFnLJvJeU593X=vOnGbBQuPi00qM3jCTFqaw9D3g@mail.gmail.com>

On Mon, Aug 1, 2016 at 1:42 AM Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 1 August 2016 at 05:27, Daniel Holth <dholth at gmail.com> wrote:
> > The next version of cffi will contain small changes to generate code
> > compliant with Python's Py_LIMITED_API:
> > https://bitbucket.org/cffi/cffi/commits/8f867f5a869f
> >
> > (although cffi itself is not, the extensions it generates will be).
>
> Very cool!
>
> > If we also add an appropriate supported tag to pip ~= cp3.abi3.manylinux1
> > and provide a way to name the generated DLL's appropriately, it may
> become
> > possible to reduce the burden of distributing cffi extensions, especially
> > for Windows. One compiled artifact should work on Python 3.2 and above.
>
> Aye, that would be very promising. I guess the first step would be to
> document the current steps involved in doing this manually? And then
> figure out what tweaks would be needed to setuptools and pip to allow
> it to be automated?
>
> As an initial stab at that:
>
> Status quo, on publication side:
>
> - require minimum cffi version 1.8
> - build with setuptools
> - postprocessing step to rename shared library/DLL
> - postprocessing step to regenerate renamed whl file with renamed SO/DLL
>

There wouldn't necessarily need to be renaming. build_ext command
determines the DLL extension. It could be patched or modified to read an
"I'm ABI3" flag on the Extension() object.

We could pass an ABI3 flag to bdist_wheel in the same way we ask for
universal 'py2.py3-none-any'. To be set if the wheel contained only ABI3
extensions, and ignored on py2.


> On consumption side:
>
> - requires "cp3.abi3" to be included at appropriate points in
> compatibility tag list
>
> Which would make the necessary changes be:
>
> - updating setuptools to somehow be Py_LIMITED_API aware when naming
> built extensions
> - updating wheel to somehow be Py_LIMITED_API aware when naming whl files
> - if necessary, updating pip's compatibility tag list
>
> Of those changes, only the "somehow be Py_LIMITED_API aware" sounds
> potentially tricky to me, as I'd be surprised if there was any way
> around requiring a new explicit setting in either setup.py or
> setup.cfg.
>

Any alternative to an explicit flag that I can think of would be too
magical to consider, as poor as trying to inspect library symbols for
manylinux1 compat in bdist_wheel itself.


> Regards,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160801/3a71f2ea/attachment.html>

From ncoghlan at gmail.com  Mon Aug  1 10:02:47 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 2 Aug 2016 00:02:47 +1000
Subject: [Distutils] cffi & Py_LIMITED_API
In-Reply-To: <CAG8k2+5Hy8iFnLJvJeU593X=vOnGbBQuPi00qM3jCTFqaw9D3g@mail.gmail.com>
References: <CAG8k2+7RwPic+eUe3ir-LFnAua8714JiJ93JB8+D4MPKfpkz3A@mail.gmail.com>
 <CADiSq7cmQBVpESUi8w94-Ao=5ughpZymbKM-gmwWt6xKbWMpVQ@mail.gmail.com>
 <CAG8k2+5Hy8iFnLJvJeU593X=vOnGbBQuPi00qM3jCTFqaw9D3g@mail.gmail.com>
Message-ID: <CADiSq7en+27PK_V=SG=+1v5cXKjf805TPyEo37O4uOGyzcYBZA@mail.gmail.com>

On 1 August 2016 at 23:36, Daniel Holth <dholth at gmail.com> wrote:
> On Mon, Aug 1, 2016 at 1:42 AM Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Status quo, on publication side:
>>
>> - require minimum cffi version 1.8
>> - build with setuptools
>> - postprocessing step to rename shared library/DLL
>> - postprocessing step to regenerate renamed whl file with renamed SO/DLL
>
> There wouldn't necessarily need to be renaming.

Aye, that description was for the process if there weren't any changes
to the toolchain.

> build_ext command determines
> the DLL extension. It could be patched or modified to read an "I'm ABI3"
> flag on the Extension() object.
>
> We could pass an ABI3 flag to bdist_wheel in the same way we ask for
> universal 'py2.py3-none-any'. To be set if the wheel contained only ABI3
> extensions, and ignored on py2.

The general idea sounds good to me, but as a slight bikeshed on the
flag name, perhaps "cpabi3"?

That's a mash-up of the 'cp' interpreter code for CPython, with the
'abi3' stable ABI tag.

Longer term, we may want to allow people to version that (as new APIs
may sometimes be added to the stable ABI, which you gain access to at
the C level by setting Py_LIMITED_API to the corresponding CPython hex
version rather than just defining it [1]), but as a starting point
enabling access to the initial 3.2 stable ABI used by cffi should be
sufficient.

>> Of those changes, only the "somehow be Py_LIMITED_API aware" sounds
>> potentially tricky to me, as I'd be surprised if there was any way
>> around requiring a new explicit setting in either setup.py or
>> setup.cfg.
>
>
> Any alternative to an explicit flag that I can think of would be too magical
> to consider, as poor as trying to inspect library symbols for manylinux1
> compat in bdist_wheel itself.

Aye, that was my conclusion as well. Once my brain started going
"Well, we could scan the source for..." I just went "Ewww, no, we can
just ask people to set a new flag" :)

However, would it make sense for the new flag to also implicitly
define Py_LIMITED_API in the compile flags when building the
extension, even if it isn't otherwise specified in the extension's
source code? That way folks would still only need to define their
intent to use the flag in one place, it's just that that place would
be their build instructions rather than the extension module source
code.

Cheers,
Nick.

[1] https://docs.python.org/3/c-api/stable.html

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From steve.dower at python.org  Mon Aug  1 12:00:14 2016
From: steve.dower at python.org (Steve Dower)
Date: Mon, 1 Aug 2016 09:00:14 -0700
Subject: [Distutils] cffi & Py_LIMITED_API
In-Reply-To: <CADiSq7en+27PK_V=SG=+1v5cXKjf805TPyEo37O4uOGyzcYBZA@mail.gmail.com>
References: <CAG8k2+7RwPic+eUe3ir-LFnAua8714JiJ93JB8+D4MPKfpkz3A@mail.gmail.com>
 <CADiSq7cmQBVpESUi8w94-Ao=5ughpZymbKM-gmwWt6xKbWMpVQ@mail.gmail.com>
 <CAG8k2+5Hy8iFnLJvJeU593X=vOnGbBQuPi00qM3jCTFqaw9D3g@mail.gmail.com>
 <CADiSq7en+27PK_V=SG=+1v5cXKjf805TPyEo37O4uOGyzcYBZA@mail.gmail.com>
Message-ID: <4db88636-e654-056b-0f3a-445de41ddb07@python.org>

On 01Aug2016 0702, Nick Coghlan wrote:
> On 1 August 2016 at 23:36, Daniel Holth <dholth at gmail.com> wrote:
>> build_ext command determines
>> the DLL extension. It could be patched or modified to read an "I'm ABI3"
>> flag on the Extension() object.
>>
>> We could pass an ABI3 flag to bdist_wheel in the same way we ask for
>> universal 'py2.py3-none-any'. To be set if the wheel contained only ABI3
>> extensions, and ignored on py2.
>
> The general idea sounds good to me, but as a slight bikeshed on the
> flag name, perhaps "cpabi3"?
>
> That's a mash-up of the 'cp' interpreter code for CPython, with the
> 'abi3' stable ABI tag.
>
> Longer term, we may want to allow people to version that (as new APIs
> may sometimes be added to the stable ABI, which you gain access to at
> the C level by setting Py_LIMITED_API to the corresponding CPython hex
> version rather than just defining it [1]), but as a starting point
> enabling access to the initial 3.2 stable ABI used by cffi should be
> sufficient.

The DLL tag on Windows will have to just be ".pyd" if you want to 
support back prior to 3.5. In 3.5 you can use ".cp35-win32.pyd" or 
".cp35-win_amd64.pyd" and the importer will prefer that DLL over a plain 
".pyd", but my proposal to also support ".cp3-${PLAT}.pyd" here didn't 
make it.

The wheel tag is more important than the DLL tag. (Possibly you're not 
even suggesting changing the DLL name at all and just passing the flag 
through for the build option? Hard to tell from the discussion.)

Cheers,
Steve

From dholth at gmail.com  Mon Aug  1 12:24:35 2016
From: dholth at gmail.com (Daniel Holth)
Date: Mon, 01 Aug 2016 16:24:35 +0000
Subject: [Distutils] cffi & Py_LIMITED_API
In-Reply-To: <4db88636-e654-056b-0f3a-445de41ddb07@python.org>
References: <CAG8k2+7RwPic+eUe3ir-LFnAua8714JiJ93JB8+D4MPKfpkz3A@mail.gmail.com>
 <CADiSq7cmQBVpESUi8w94-Ao=5ughpZymbKM-gmwWt6xKbWMpVQ@mail.gmail.com>
 <CAG8k2+5Hy8iFnLJvJeU593X=vOnGbBQuPi00qM3jCTFqaw9D3g@mail.gmail.com>
 <CADiSq7en+27PK_V=SG=+1v5cXKjf805TPyEo37O4uOGyzcYBZA@mail.gmail.com>
 <4db88636-e654-056b-0f3a-445de41ddb07@python.org>
Message-ID: <CAG8k2+5x6aFOg=gPfdYCoOYKKF7Ev34nYr5f0buPQK+SyK=QGA@mail.gmail.com>

On Mon, Aug 1, 2016 at 12:01 PM Steve Dower <steve.dower at python.org> wrote:

> On 01Aug2016 0702, Nick Coghlan wrote:
> > On 1 August 2016 at 23:36, Daniel Holth <dholth at gmail.com> wrote:
> >> build_ext command determines
> >> the DLL extension. It could be patched or modified to read an "I'm ABI3"
> >> flag on the Extension() object.
> >>
> >> We could pass an ABI3 flag to bdist_wheel in the same way we ask for
> >> universal 'py2.py3-none-any'. To be set if the wheel contained only ABI3
> >> extensions, and ignored on py2.
> >
> > The general idea sounds good to me, but as a slight bikeshed on the
> > flag name, perhaps "cpabi3"?
> >
> > That's a mash-up of the 'cp' interpreter code for CPython, with the
> > 'abi3' stable ABI tag.
> >
> > Longer term, we may want to allow people to version that (as new APIs
> > may sometimes be added to the stable ABI, which you gain access to at
> > the C level by setting Py_LIMITED_API to the corresponding CPython hex
> > version rather than just defining it [1]), but as a starting point
> > enabling access to the initial 3.2 stable ABI used by cffi should be
> > sufficient.
>
> The DLL tag on Windows will have to just be ".pyd" if you want to
> support back prior to 3.5. In 3.5 you can use ".cp35-win32.pyd" or
> ".cp35-win_amd64.pyd" and the importer will prefer that DLL over a plain
> ".pyd", but my proposal to also support ".cp3-${PLAT}.pyd" here didn't
> make it.
>
> The wheel tag is more important than the DLL tag. (Possibly you're not
> even suggesting changing the DLL name at all and just passing the flag
> through for the build option? Hard to tell from the discussion.)
>

I'm sure we are interpolating from Linux, where you'd be looking at
extensions .cpython-35m-x86_64-linux-gnu.so versus .abi3.so. But any DLL
extension that the target CPythons will agree to load, plus an appropriate
wheel tag, should be sufficient for our use case.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160801/9b8103a8/attachment.html>

From chris at withers.org  Mon Aug  1 17:34:07 2016
From: chris at withers.org (Chris Withers)
Date: Mon, 1 Aug 2016 22:34:07 +0100
Subject: [Distutils] latest setuptools release breaks Python 2 installs?
Message-ID: <8a9cee09-341b-80b2-5dcf-d51cfe4829be@withers.org>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160801/22457ac9/attachment.html>

From chris.jerdonek at gmail.com  Mon Aug  1 17:40:01 2016
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Mon, 1 Aug 2016 14:40:01 -0700
Subject: [Distutils] latest setuptools release breaks Python 2 installs?
In-Reply-To: <8a9cee09-341b-80b2-5dcf-d51cfe4829be@withers.org>
References: <8a9cee09-341b-80b2-5dcf-d51cfe4829be@withers.org>
Message-ID: <CAOTb1wexU-Bu2oN5cGS9sJui4UwhU5vaeRBCJqCvHdR=dq7B-A@mail.gmail.com>

The issue tracker on PyPA's GitHub repo for the proejct seems alive and well:

https://github.com/pypa/setuptools/issues

--Chris

On Mon, Aug 1, 2016 at 2:34 PM, Chris Withers <chris at withers.org> wrote:
> Hi Guys,
>
> Where's the best place to report setuptools issues now?
>
> I see 25.1.2 was release today, and I've just had nightly builds of one of
> my packages fail as follows:
>
> Installing docs.
> Getting distribution for 'sphinx'.
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py",
> line 2271, in main
>
>   File "/opt/python/2.7.9/lib/python2.7/distutils/core.py", line 151, in
> setup
>     dist.run_commands()
>   File "/opt/python/2.7.9/lib/python2.7/distutils/dist.py", line 953, in
> run_commands
>     self.run_command(cmd)
>   File "/opt/python/2.7.9/lib/python2.7/distutils/dist.py", line 972, in
> run_command
>     cmd_obj.run()
>   File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py",
> line 409, in run
>
>   File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py",
> line 645, in easy_install
>
>   File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py",
> line 694, in install_item
>
>   File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py",
> line 845, in install_eggs
>
>   File "build/bdist.linux-x86_64/egg/setuptools/archive_util.py", line 52,
> in unpack_archive
>   File "build/bdist.linux-x86_64/egg/setuptools/archive_util.py", line 147,
> in unpack_tarfile
>   File "/home/travis/virtualenv/python2.7.9/lib/python2.7/posixpath.py",
> line 80, in join
>     path += '/' + b
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xcc in position 8:
> ordinal not in range(128)
> An error occurred when trying to install Sphinx 1.4.5. Look above this
> message for any errors that were output by easy_install.
> While:
>   Installing docs.
>   Getting distribution for 'sphinx'.
> Error: Couldn't install: Sphinx 1.4.5
>
> The full log is here:
>
> https://travis-ci.org/cjw296/nose_fixes/jobs/149012793
>
> cheers,
>
> Chris
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>

From chris at withers.org  Mon Aug  1 17:42:17 2016
From: chris at withers.org (Chris Withers)
Date: Mon, 1 Aug 2016 22:42:17 +0100
Subject: [Distutils] latest setuptools release breaks Python 2 installs?
In-Reply-To: <CAOTb1wexU-Bu2oN5cGS9sJui4UwhU5vaeRBCJqCvHdR=dq7B-A@mail.gmail.com>
References: <8a9cee09-341b-80b2-5dcf-d51cfe4829be@withers.org>
 <CAOTb1wexU-Bu2oN5cGS9sJui4UwhU5vaeRBCJqCvHdR=dq7B-A@mail.gmail.com>
Message-ID: <fb0e4ec3-f1db-af4d-2bd9-560f46efc1c6@withers.org>

Yep, file:

https://github.com/pypa/setuptools/issues/709

Chris


On 01/08/2016 22:40, Chris Jerdonek wrote:
> The issue tracker on PyPA's GitHub repo for the proejct seems alive and well:
>
> https://github.com/pypa/setuptools/issues
>
> --Chris
>
> On Mon, Aug 1, 2016 at 2:34 PM, Chris Withers <chris at withers.org> wrote:
>> Hi Guys,
>>
>> Where's the best place to report setuptools issues now?
>>
>> I see 25.1.2 was release today, and I've just had nightly builds of one of
>> my packages fail as follows:
>>
>> Installing docs.
>> Getting distribution for 'sphinx'.
>> Traceback (most recent call last):
>>    File "<string>", line 1, in <module>
>>    File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py",
>> line 2271, in main
>>
>>    File "/opt/python/2.7.9/lib/python2.7/distutils/core.py", line 151, in
>> setup
>>      dist.run_commands()
>>    File "/opt/python/2.7.9/lib/python2.7/distutils/dist.py", line 953, in
>> run_commands
>>      self.run_command(cmd)
>>    File "/opt/python/2.7.9/lib/python2.7/distutils/dist.py", line 972, in
>> run_command
>>      cmd_obj.run()
>>    File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py",
>> line 409, in run
>>
>>    File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py",
>> line 645, in easy_install
>>
>>    File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py",
>> line 694, in install_item
>>
>>    File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py",
>> line 845, in install_eggs
>>
>>    File "build/bdist.linux-x86_64/egg/setuptools/archive_util.py", line 52,
>> in unpack_archive
>>    File "build/bdist.linux-x86_64/egg/setuptools/archive_util.py", line 147,
>> in unpack_tarfile
>>    File "/home/travis/virtualenv/python2.7.9/lib/python2.7/posixpath.py",
>> line 80, in join
>>      path += '/' + b
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xcc in position 8:
>> ordinal not in range(128)
>> An error occurred when trying to install Sphinx 1.4.5. Look above this
>> message for any errors that were output by easy_install.
>> While:
>>    Installing docs.
>>    Getting distribution for 'sphinx'.
>> Error: Couldn't install: Sphinx 1.4.5
>>
>> The full log is here:
>>
>> https://travis-ci.org/cjw296/nose_fixes/jobs/149012793
>>
>> cheers,
>>
>> Chris
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>


From donald at stufft.io  Mon Aug  1 18:26:51 2016
From: donald at stufft.io (Donald Stufft)
Date: Mon, 1 Aug 2016 18:26:51 -0400
Subject: [Distutils] test.pypi.org is now active
Message-ID: <68ADCCB2-D4EE-4A2F-8D18-1987CF4F37C4@stufft.io>

Just an FYI, test.pypi.org is now active (this is the Warehouse instance backed by Test PyPI). Once https://github.com/pypa/warehouse/pull/1416 is deployed and merged test.pypi.io will start to redirect to test.pypi.org.

No changes to pypi.io have occurred yet, we?re still waiting on the EV Certificate to be deployed for that.

?
Donald Stufft




From annaraven at gmail.com  Mon Aug  1 18:39:43 2016
From: annaraven at gmail.com (Anna Ravenscroft)
Date: Mon, 1 Aug 2016 15:39:43 -0700
Subject: [Distutils] test.pypi.org is now active
In-Reply-To: <68ADCCB2-D4EE-4A2F-8D18-1987CF4F37C4@stufft.io>
References: <68ADCCB2-D4EE-4A2F-8D18-1987CF4F37C4@stufft.io>
Message-ID: <CA+nrBM9aR=50aMkaRZsiUSuK8uu0EcQScnqzzY4oaVWyV7npeg@mail.gmail.com>

So for testing, we direct testpypi (in the pypirc) to test.pypi.org, right?

On Mon, Aug 1, 2016 at 3:26 PM, Donald Stufft <donald at stufft.io> wrote:

> Just an FYI, test.pypi.org is now active (this is the Warehouse instance
> backed by Test PyPI). Once https://github.com/pypa/warehouse/pull/1416 is
> deployed and merged test.pypi.io will start to redirect to test.pypi.org.
>
> No changes to pypi.io have occurred yet, we?re still waiting on the EV
> Certificate to be deployed for that.
>
> ?
> Donald Stufft
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
cordially,
Anna
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160801/489ad8df/attachment.html>

From donald at stufft.io  Mon Aug  1 18:41:54 2016
From: donald at stufft.io (Donald Stufft)
Date: Mon, 1 Aug 2016 18:41:54 -0400
Subject: [Distutils] test.pypi.org is now active
In-Reply-To: <CA+nrBM9aR=50aMkaRZsiUSuK8uu0EcQScnqzzY4oaVWyV7npeg@mail.gmail.com>
References: <68ADCCB2-D4EE-4A2F-8D18-1987CF4F37C4@stufft.io>
 <CA+nrBM9aR=50aMkaRZsiUSuK8uu0EcQScnqzzY4oaVWyV7npeg@mail.gmail.com>
Message-ID: <099120E5-23DE-47F2-89B0-F2805CAA232C@stufft.io>

Yea, instead of testpypi.python.org you?d use test.pypi.org. Specifically for uploading you?d use something like:


[testpypiorg]
repository:https://test.pypi.org/legacy/
username: <username>
password: <password>

For downloading it?d be something like test.pypi.org/simple/


> On Aug 1, 2016, at 6:39 PM, Anna Ravenscroft <annaraven at gmail.com> wrote:
> 
> So for testing, we direct testpypi (in the pypirc) to test.pypi.org <http://test.pypi.org/>, right?
> 
> On Mon, Aug 1, 2016 at 3:26 PM, Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
> Just an FYI, test.pypi.org <http://test.pypi.org/> is now active (this is the Warehouse instance backed by Test PyPI). Once https://github.com/pypa/warehouse/pull/1416 <https://github.com/pypa/warehouse/pull/1416> is deployed and merged test.pypi.io <http://test.pypi.io/> will start to redirect to test.pypi.org <http://test.pypi.org/>.
> 
> No changes to pypi.io <http://pypi.io/> have occurred yet, we?re still waiting on the EV Certificate to be deployed for that.
> 
> ?
> Donald Stufft
> 
> 
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org <mailto:Distutils-SIG at python.org>
> https://mail.python.org/mailman/listinfo/distutils-sig <https://mail.python.org/mailman/listinfo/distutils-sig>
> 
> 
> 
> -- 
> cordially,
> Anna


?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160801/67295f0b/attachment-0001.html>

From annaraven at gmail.com  Mon Aug  1 18:41:41 2016
From: annaraven at gmail.com (Anna Ravenscroft)
Date: Mon, 1 Aug 2016 15:41:41 -0700
Subject: [Distutils] test.pypi.org is now active
In-Reply-To: <CA+nrBM9aR=50aMkaRZsiUSuK8uu0EcQScnqzzY4oaVWyV7npeg@mail.gmail.com>
References: <68ADCCB2-D4EE-4A2F-8D18-1987CF4F37C4@stufft.io>
 <CA+nrBM9aR=50aMkaRZsiUSuK8uu0EcQScnqzzY4oaVWyV7npeg@mail.gmail.com>
Message-ID: <CA+nrBM9+XLyGoS56WmVwGrZiMSTEyCoBsSevzKCZNcpWPR2R+A@mail.gmail.com>

To be specific, update .pypirc like this:

[testpypi]
repository=https://test.pypi.org/


On Mon, Aug 1, 2016 at 3:39 PM, Anna Ravenscroft <annaraven at gmail.com>
wrote:

> So for testing, we direct testpypi (in the pypirc) to test.pypi.org,
> right?
>
> On Mon, Aug 1, 2016 at 3:26 PM, Donald Stufft <donald at stufft.io> wrote:
>
>> Just an FYI, test.pypi.org is now active (this is the Warehouse instance
>> backed by Test PyPI). Once https://github.com/pypa/warehouse/pull/1416
>> is deployed and merged test.pypi.io will start to redirect to
>> test.pypi.org.
>>
>> No changes to pypi.io have occurred yet, we?re still waiting on the EV
>> Certificate to be deployed for that.
>>
>> ?
>> Donald Stufft
>>
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
>
>
> --
> cordially,
> Anna
>



-- 
cordially,
Anna
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160801/0e596efd/attachment.html>

From annaraven at gmail.com  Mon Aug  1 18:52:44 2016
From: annaraven at gmail.com (Anna Ravenscroft)
Date: Mon, 1 Aug 2016 15:52:44 -0700
Subject: [Distutils] test.pypi.org is now active
In-Reply-To: <099120E5-23DE-47F2-89B0-F2805CAA232C@stufft.io>
References: <68ADCCB2-D4EE-4A2F-8D18-1987CF4F37C4@stufft.io>
 <CA+nrBM9aR=50aMkaRZsiUSuK8uu0EcQScnqzzY4oaVWyV7npeg@mail.gmail.com>
 <099120E5-23DE-47F2-89B0-F2805CAA232C@stufft.io>
Message-ID: <CA+nrBM-vZipdy2V7axBowVOnQhzammUtWMinos_0=w6AXC0ATA@mail.gmail.com>

Great. Just making sure. Updating the chapter now.

On Mon, Aug 1, 2016 at 3:41 PM, Donald Stufft <donald at stufft.io> wrote:

> Yea, instead of testpypi.python.org you?d use test.pypi.org. Specifically
> for uploading you?d use something like:
>
>
> [testpypiorg]
> repository:https://test.pypi.org/legacy/
> username: <username>
> password: <password>
>
> For downloading it?d be something like test.pypi.org/simple/
>
>
> On Aug 1, 2016, at 6:39 PM, Anna Ravenscroft <annaraven at gmail.com> wrote:
>
> So for testing, we direct testpypi (in the pypirc) to test.pypi.org,
> right?
>
> On Mon, Aug 1, 2016 at 3:26 PM, Donald Stufft <donald at stufft.io> wrote:
>
>> Just an FYI, test.pypi.org is now active (this is the Warehouse instance
>> backed by Test PyPI). Once https://github.com/pypa/warehouse/pull/1416
>> is deployed and merged test.pypi.io will start to redirect to
>> test.pypi.org.
>>
>> No changes to pypi.io have occurred yet, we?re still waiting on the EV
>> Certificate to be deployed for that.
>>
>> ?
>> Donald Stufft
>>
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
>
>
> --
> cordially,
> Anna
>
>
>
> ?
> Donald Stufft
>
>
>
>


-- 
cordially,
Anna
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160801/076db6d0/attachment.html>

From ncoghlan at gmail.com  Tue Aug  2 04:02:23 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 2 Aug 2016 18:02:23 +1000
Subject: [Distutils] cffi & Py_LIMITED_API
In-Reply-To: <CAG8k2+5x6aFOg=gPfdYCoOYKKF7Ev34nYr5f0buPQK+SyK=QGA@mail.gmail.com>
References: <CAG8k2+7RwPic+eUe3ir-LFnAua8714JiJ93JB8+D4MPKfpkz3A@mail.gmail.com>
 <CADiSq7cmQBVpESUi8w94-Ao=5ughpZymbKM-gmwWt6xKbWMpVQ@mail.gmail.com>
 <CAG8k2+5Hy8iFnLJvJeU593X=vOnGbBQuPi00qM3jCTFqaw9D3g@mail.gmail.com>
 <CADiSq7en+27PK_V=SG=+1v5cXKjf805TPyEo37O4uOGyzcYBZA@mail.gmail.com>
 <4db88636-e654-056b-0f3a-445de41ddb07@python.org>
 <CAG8k2+5x6aFOg=gPfdYCoOYKKF7Ev34nYr5f0buPQK+SyK=QGA@mail.gmail.com>
Message-ID: <CADiSq7cLo_D9BqunxJBU768-q8c+7FFVQaFowF0JrQWp_-cnhw@mail.gmail.com>

On 2 August 2016 at 02:24, Daniel Holth <dholth at gmail.com> wrote:
> On Mon, Aug 1, 2016 at 12:01 PM Steve Dower <steve.dower at python.org> wrote:
>> The DLL tag on Windows will have to just be ".pyd" if you want to
>> support back prior to 3.5. In 3.5 you can use ".cp35-win32.pyd" or
>> ".cp35-win_amd64.pyd" and the importer will prefer that DLL over a plain
>> ".pyd", but my proposal to also support ".cp3-${PLAT}.pyd" here didn't
>> make it.
>>
>> The wheel tag is more important than the DLL tag. (Possibly you're not
>> even suggesting changing the DLL name at all and just passing the flag
>> through for the build option? Hard to tell from the discussion.)
>
> I'm sure we are interpolating from Linux, where you'd be looking at
> extensions .cpython-35m-x86_64-linux-gnu.so versus .abi3.so. But any DLL
> extension that the target CPythons will agree to load, plus an appropriate
> wheel tag, should be sufficient for our use case.

Right, on *nix the SO names already include the SOABI details by
default, so we really do want to change those to something less
explicit, but I'm not sure it's even possible to using a completely
unqualified SO name even if we wanted to.

On Windows, if we have to rely on dropping any ABI indicated from the
DLL entirely, we can leave with - pip should keep things from
colliding too badly.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From donald at stufft.io  Wed Aug  3 17:44:28 2016
From: donald at stufft.io (Donald Stufft)
Date: Wed, 3 Aug 2016 17:44:28 -0400
Subject: [Distutils] pypi.org fully deployed
Message-ID: <1C3C1592-2958-43D2-80B2-C2B0BB6DA6C4@stufft.io>

Just another FYI, other than maybe some intermittent certificate failures as Fastly fully rolls out the new certificate, pypi.org should be fully deployed now in place of pypi.io. Old URLs will continue to work and redirect to pypi.org.

?
Donald Stufft




From ncoghlan at gmail.com  Thu Aug  4 08:03:41 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 4 Aug 2016 22:03:41 +1000
Subject: [Distutils] pypi.org fully deployed
In-Reply-To: <1C3C1592-2958-43D2-80B2-C2B0BB6DA6C4@stufft.io>
References: <1C3C1592-2958-43D2-80B2-C2B0BB6DA6C4@stufft.io>
Message-ID: <CADiSq7eoCOurfzQj93QYhKjPk-_fM45vsQ7iTBBtL8_xmuDhcg@mail.gmail.com>

On 4 August 2016 at 07:44, Donald Stufft <donald at stufft.io> wrote:
> Just another FYI, other than maybe some intermittent certificate failures as Fastly fully rolls out the new certificate, pypi.org should be fully deployed now in place of pypi.io.

Very cool! I actually just updated the hostname in my linux.conf.au
2017 talk proposal earlier today :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From gweatherby at uchc.edu  Thu Aug  4 14:36:56 2016
From: gweatherby at uchc.edu (Weatherby,Gerard)
Date: Thu, 4 Aug 2016 18:36:56 +0000
Subject: [Distutils] pip3 subversion user
Message-ID: <668bd013c37e4e939e5a6965770fb035@NSO-ITEXC-MB26.uchc.net>

Apologies in advance if this is the wrong list for the question; if so,
a pointer to the correct one would be appreciated.

I'm trying to user the subversion option of pip3 to install a package; I
can use
sudo pip3 install svn+https://path_to_code , but this does not work
because our subversion repository requires a subversion user name, and I
don' t know to pass it through pip3 to subversion.

Tried:
sudo pip3 install --user gweatherby svn+https://path_to_code, but it
still prompted for "root" password.

-Gerard

From chris.barker at noaa.gov  Thu Aug  4 19:09:12 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 4 Aug 2016 16:09:12 -0700
Subject: [Distutils] pip3 subversion user
In-Reply-To: <668bd013c37e4e939e5a6965770fb035@NSO-ITEXC-MB26.uchc.net>
References: <668bd013c37e4e939e5a6965770fb035@NSO-ITEXC-MB26.uchc.net>
Message-ID: <CALGmxELSFFnQC2o3mPu8h+6W8+997dqKJhdZdo8vcuvO_Do6+Q@mail.gmail.com>

On Thu, Aug 4, 2016 at 11:36 AM, Weatherby,Gerard <gweatherby at uchc.edu>
wrote:

> Apologies in advance if this is the wrong list for the question; if so,
> a pointer to the correct one would be appreciated.
>
> I'm trying to user the subversion option of pip3 to install a package; I
> can use
> sudo pip3 install svn+https://path_to_code , but this does not work
> because our subversion repository requires a subversion user name, and I
> don' t know to pass it through pip3 to subversion.
>
> Tried:
> sudo pip3 install --user gweatherby svn+https://path_to_code, but it
> still prompted for "root" password.
>

try:

sudo pip3 install  svn+https://gweatherby at path_to_code
<https://path_to_code/>

I have no idea if it will work, but worth a shot


-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160804/2fd46845/attachment.html>

From solipsis at pitrou.net  Tue Aug  9 05:13:38 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 9 Aug 2016 11:13:38 +0200
Subject: [Distutils] setuptools 25.1.6 broke numpy.distutils on Windows
Message-ID: <20160809111338.7dc7c239@fsol>


Hello,

Just a heads-up that latest setuptools is incompatible with
numpy.distutils on Windows:
https://github.com/pypa/setuptools/issues/728

The whole scientific community will suffer until this issue is solved
one way or the other (either on the setuptools side, or on the Numpy
side, or both).

Regards

Antoine.



From steve.dower at python.org  Tue Aug  9 12:15:31 2016
From: steve.dower at python.org (Steve Dower)
Date: Tue, 9 Aug 2016 09:15:31 -0700
Subject: [Distutils] setuptools 25.1.6 broke numpy.distutils on Windows
In-Reply-To: <20160809111338.7dc7c239@fsol>
References: <20160809111338.7dc7c239@fsol>
Message-ID: <b6f748b8-2db2-320e-534a-f0421d1d03a8@python.org>

On 09Aug2016 0213, Antoine Pitrou wrote:
> Just a heads-up that latest setuptools is incompatible with
> numpy.distutils on Windows:
> https://github.com/pypa/setuptools/issues/728

For those who don't want to dig into the issues and pull requests, it 
seems that multiple libraries were patching different parts of distutils 
and adding extra quotes to the command line. This was apparently only 
occurring with Python 3.5.

The main reason I want to add to this post is that *Python 3.5 is still 
taking bugfixes*. If there is a problem in distutils, please report it 
to bugs.python.org first so we can fix it, and then look into 
monkeypatching the older versions.

Cheers,
Steve

From postings at andreaskrueger.de  Tue Aug  9 18:19:44 2016
From: postings at andreaskrueger.de (Dr. Andreas Krueger)
Date: Tue, 9 Aug 2016 23:19:44 +0100
Subject: [Distutils] msvc9compiler.py - vcvarsall.bat is living one folder
 higher - VS90COMNTOOLS
Message-ID: <c84aeb2e-e4dd-f5d2-e2a4-c74b2197032e@andreaskrueger.de>

I could not find a github repo, otherwise I would have filed this as an
"issue" there.


The subfolder "VC" is wrong in msvc9compiler.py

- at least on my system (win7-64bit, py2.7.12, anaconda 2.4.1, conda 4.1.11)


So in your "msvc9compiler.py"

instead of            
    productdir = os.path.join(toolsdir, os.pardir, os.pardir, "VC")

it needs to be
    productdir = os.path.join(toolsdir, os.pardir, os.pardir)


Why do I say that?
When googling the error message:  "DistutilsPlatformError: Unable to
find vcvarsall.bat", it is recommended "everywhere" on the web, to
download "VCForPython27.msi", Microsoft Visual C++ Compiler for Python
2.7  http://aka.ms/vcpython27


BUT then

    vcvarsall.bat

ends up being in

    "C:\Users\Andreas\AppData\Local\Programs\Common\Microsoft\Visual C++
for Python\9.0\vcvarsall.bat"

and NOT in

    "C:\Users\Andreas\AppData\Local\Programs\Common\Microsoft\Visual C++
for Python\9.0\VC\vcvarsall.bat"

where YOUR code is expecting it.


Perhaps you actually want to check BOTH places, to keep compatibility
with some older versions / different setups? 
Because I suppose, your code had worked, at some stage in the past.


Thanks a lot!
                           Andreas


P.S.:
I had put some debugging prints to understand your code better, and this
is what it outputs:

toolsdir=
C:\Users\Andreas\AppData\Local\Programs\Common\Microsoft\Visual C++ for
Python\9.0\WinSDK\Bin\
productdir=
C:\Users\Andreas\AppData\Local\Programs\Common\Microsoft\Visual C++ for
Python\9.0\WinSDK\Bin\..\..\VC
productdir=
C:\Users\Andreas\AppData\Local\Programs\Common\Microsoft\Visual C++ for
Python\9.0\VC
vcvarsall=
C:\Users\Andreas\AppData\Local\Programs\Common\Microsoft\Visual C++ for
Python\9.0\VC\vcvarsall.bat

and that last path is wrong.


I could repair it, but only by patching your "msvc9compiler.py":

            productdir = os.path.join(toolsdir, os.pardir, os.pardir,
"VC")    -->
            productdir = os.path.join(toolsdir, os.pardir, os.pardir)

now it does find it:

toolsdir=
C:\Users\Andreas\AppData\Local\Programs\Common\Microsoft\Visual C++ for
Python\9.0\WinSDK\Bin\
productdir=
C:\Users\Andreas\AppData\Local\Programs\Common\Microsoft\Visual C++ for
Python\9.0\WinSDK\Bin\..\..
productdir=
C:\Users\Andreas\AppData\Local\Programs\Common\Microsoft\Visual C++ for
Python\9.0
vcvarsall=
C:\Users\Andreas\AppData\Local\Programs\Common\Microsoft\Visual C++ for
Python\9.0\vcvarsall.bat




From meradj.aghdam at gmail.com  Tue Aug  9 10:56:11 2016
From: meradj.aghdam at gmail.com (Meradj Aghdam)
Date: Tue, 9 Aug 2016 16:56:11 +0200
Subject: [Distutils] Python package inquiry
Message-ID: <CAL7XVEZhZJWZ4x2ouyd_DTnXtr4fZkSb=X-Lz39X5qBBsV3p6Q@mail.gmail.com>

To whom it may concern,

Hi,

I am new to Python and I am struggling to install Python packages. I have
followed the instructions however it does not seem to be working. I am
trying to install the package through the Python(Command Line), the package
is economics, and when I put

python -m pip install Economics

It does not work and the error that it is a syntax error. I am using Python
2.7. Would you mind kindly let me know how I can install packages?

Your help if very much appreciated,

Kind regards,

Meradj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160809/70800aad/attachment.html>

From steve.dower at python.org  Tue Aug  9 19:53:00 2016
From: steve.dower at python.org (Steve Dower)
Date: Tue, 9 Aug 2016 16:53:00 -0700
Subject: [Distutils] msvc9compiler.py - vcvarsall.bat is living one
 folder higher - VS90COMNTOOLS
In-Reply-To: <c84aeb2e-e4dd-f5d2-e2a4-c74b2197032e@andreaskrueger.de>
References: <c84aeb2e-e4dd-f5d2-e2a4-c74b2197032e@andreaskrueger.de>
Message-ID: <b2da2469-e828-7d74-dce0-c00cfc47df4a@python.org>

On 09Aug2016 1519, Dr. Andreas Krueger wrote:
> I could not find a github repo, otherwise I would have filed this as an
> "issue" there.
>
>
> The subfolder "VC" is wrong in msvc9compiler.py
>
> - at least on my system (win7-64bit, py2.7.12, anaconda 2.4.1, conda 4.1.11)
>
>
> So in your "msvc9compiler.py"
>
> instead of
>     productdir = os.path.join(toolsdir, os.pardir, os.pardir, "VC")
>
> it needs to be
>     productdir = os.path.join(toolsdir, os.pardir, os.pardir)
>
>
> Why do I say that?
> When googling the error message:  "DistutilsPlatformError: Unable to
> find vcvarsall.bat", it is recommended "everywhere" on the web, to
> download "VCForPython27.msi", Microsoft Visual C++ Compiler for Python
> 2.7  http://aka.ms/vcpython27
>
>
> BUT then
>
>     vcvarsall.bat
>
> ends up being in
>
>     "C:\Users\Andreas\AppData\Local\Programs\Common\Microsoft\Visual C++
> for Python\9.0\vcvarsall.bat"
>
> and NOT in
>
>     "C:\Users\Andreas\AppData\Local\Programs\Common\Microsoft\Visual C++
> for Python\9.0\VC\vcvarsall.bat"
>
> where YOUR code is expecting it.
>
>
> Perhaps you actually want to check BOTH places, to keep compatibility
> with some older versions / different setups?
> Because I suppose, your code had worked, at some stage in the past.
>
>
> Thanks a lot!
>                            Andreas
>

There's actually also a requirement for a relatively recent version 
(>=6.0) of setuptools in order to build correctly using the Visual C++ 
Compiler for Python. This will allow the compiler to be found without 
using the environment variable.

I suggest ensuring that setuptools is up to date, and either import it 
in the setup.py (you don't have to do anything except import it) or use 
pip to install your packages.

Cheers,
Steve



From annaraven at gmail.com  Tue Aug  9 19:53:23 2016
From: annaraven at gmail.com (Anna Ravenscroft)
Date: Tue, 9 Aug 2016 16:53:23 -0700
Subject: [Distutils] Python package inquiry
In-Reply-To: <CAL7XVEZhZJWZ4x2ouyd_DTnXtr4fZkSb=X-Lz39X5qBBsV3p6Q@mail.gmail.com>
References: <CAL7XVEZhZJWZ4x2ouyd_DTnXtr4fZkSb=X-Lz39X5qBBsV3p6Q@mail.gmail.com>
Message-ID: <CA+nrBM-mooGY9LvO=AZxxTN-yJ2h8f_GFdn8pgCTVCTu9dMK8w@mail.gmail.com>

Hi Meradj,

Are you in the Python interpreter when you type that? If so, that's your
problem. You need to be at your normal command prompt. Open a new shell or
terminal and type that command first.

If you've already started Python and are in the interpreter, that would
indeed, give you a syntax error.

On Tue, Aug 9, 2016 at 7:56 AM, Meradj Aghdam <meradj.aghdam at gmail.com>
wrote:

> To whom it may concern,
>
> Hi,
>
> I am new to Python and I am struggling to install Python packages. I have
> followed the instructions however it does not seem to be working. I am
> trying to install the package through the Python(Command Line), the package
> is economics, and when I put
>
> python -m pip install Economics
>
> It does not work and the error that it is a syntax error. I am using
> Python 2.7. Would you mind kindly let me know how I can install packages?
>
> Your help if very much appreciated,
>
> Kind regards,
>
> Meradj
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>


-- 
cordially,
Anna
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160809/0672c8a5/attachment-0001.html>

From graffatcolmingov at gmail.com  Tue Aug  9 19:55:29 2016
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Tue, 9 Aug 2016 18:55:29 -0500
Subject: [Distutils] Python package inquiry
In-Reply-To: <CAL7XVEZhZJWZ4x2ouyd_DTnXtr4fZkSb=X-Lz39X5qBBsV3p6Q@mail.gmail.com>
References: <CAL7XVEZhZJWZ4x2ouyd_DTnXtr4fZkSb=X-Lz39X5qBBsV3p6Q@mail.gmail.com>
Message-ID: <CAN-Kwu0zbiAGA7uKJEnyBzX7Rndvs_+qSCqxsL3KfHoFYkGpMg@mail.gmail.com>

Can you tell us which instructions you followed and what the exact output
of that command is? Also, what version of Python are you using (python -V)?

On Aug 9, 2016 6:40 PM, "Meradj Aghdam" <meradj.aghdam at gmail.com> wrote:

> To whom it may concern,
>
> Hi,
>
> I am new to Python and I am struggling to install Python packages. I have
> followed the instructions however it does not seem to be working. I am
> trying to install the package through the Python(Command Line), the package
> is economics, and when I put
>
> python -m pip install Economics
>
> It does not work and the error that it is a syntax error. I am using
> Python 2.7. Would you mind kindly let me know how I can install packages?
>
> Your help if very much appreciated,
>
> Kind regards,
>
> Meradj
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160809/898f0cdc/attachment.html>

From postings at andreaskrueger.de  Tue Aug  9 20:29:03 2016
From: postings at andreaskrueger.de (postings at andreaskrueger.de)
Date: Wed, 10 Aug 2016 01:29:03 +0100
Subject: [Distutils] msvc9compiler.py - vcvarsall.bat is living one
 folder higher - VS90COMNTOOLS
In-Reply-To: <b2da2469-e828-7d74-dce0-c00cfc47df4a@python.org>
References: <c84aeb2e-e4dd-f5d2-e2a4-c74b2197032e@andreaskrueger.de>
 <b2da2469-e828-7d74-dce0-c00cfc47df4a@python.org>
Message-ID: <524b0a5d-8ce7-d8f1-e294-c33c60f20d99@andreaskrueger.de>


Hi Steve,

Thanks for your superfast answer.


> There's actually also a requirement for a relatively recent version
(>=6.0) of setuptools
>  in order to build correctly using the Visual C++ Compiler for Python.

This is the installed version

 setuptools:     25.1.6-py27_0

and 25>6 ?



> This will allow the compiler to be found without using the environment
> variable.

Then perhaps consider to completely remove/deprecate the route via
environment variable?
Because there are tons of workarounds & manuals out there which actually
point towards using the environment variable.

Or fix it.



> I suggest ensuring that setuptools is up to date

So the setuptools included in the newest anaconda is not new enough?



> , and either import it in the setup.py (you don't have to do anything
> except import it) or use pip to install your packages.
Not sure I understand.
I have been installing mc-stan.org - via 'conda install -c patricksnape
pystan'. It worked only after (solving other problems, and) patching
your distutils.



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160810/4fd4d8e0/attachment.html>

From steve.dower at python.org  Wed Aug 10 11:53:17 2016
From: steve.dower at python.org (Steve Dower)
Date: Wed, 10 Aug 2016 08:53:17 -0700
Subject: [Distutils] msvc9compiler.py - vcvarsall.bat is living
 onefolder higher - VS90COMNTOOLS
In-Reply-To: <524b0a5d-8ce7-d8f1-e294-c33c60f20d99@andreaskrueger.de>
References: <c84aeb2e-e4dd-f5d2-e2a4-c74b2197032e@andreaskrueger.de>
 <b2da2469-e828-7d74-dce0-c00cfc47df4a@python.org>
 <524b0a5d-8ce7-d8f1-e294-c33c60f20d99@andreaskrueger.de>
Message-ID: <E1bXVpH-0006II-2W@se2-syd.hostedmail.net.au>

There may be a bug in setuptools. We've already seen one reported and fixed in the last week, so possibly this is related.

Issues in setuptools go to their github page (don't have the link handy on my phone). Nothing has changed in distutils in over a year here and it's been working just fine, so I'd look at what they're doing in setuptools.

Cheers,
Steve

Top-posted from my Windows Phone

-----Original Message-----
From: "postings at andreaskrueger.de" <postings at andreaskrueger.de>
Sent: ?8/?10/?2016 6:34
To: "distutils-sig at python.org" <distutils-sig at python.org>
Subject: Re: [Distutils] msvc9compiler.py - vcvarsall.bat is living onefolder higher - VS90COMNTOOLS



Hi Steve,
Thanks for your superfast answer.


> There's actually also a requirement for a relatively recent version (>=6.0) of setuptools
>  in order to build correctly using the Visual C++ Compiler for Python. 
This is the installed version

 setuptools:     25.1.6-py27_0
and 25>6 ?




This will allow the compiler to be found without using the environment variable. 


Then perhaps consider to completely remove/deprecate the route via environment variable? 
Because there are tons of workarounds & manuals out there which actually point towards using the environment variable.

Or fix it.




I suggest ensuring that setuptools is up to date

So the setuptools included in the newest anaconda is not new enough?




, and either import it in the setup.py (you don't have to do anything except import it) or use pip to install your packages. 

Not sure I understand. 
I have been installing mc-stan.org - via 'conda install -c patricksnape pystan'. It worked only after (solving other problems, and) patching your distutils. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160810/7a611ce9/attachment.html>

From claude.marinier at lmco.com  Thu Aug 11 17:51:45 2016
From: claude.marinier at lmco.com (Marinier, Claude)
Date: Thu, 11 Aug 2016 21:51:45 +0000
Subject: [Distutils] license for setuptools
Message-ID: <1AE01599B99F8A4EA0609B55B478D77FE80ABC@HCXDSPM1.ca.lmco.com>

Good afternoon (well it's afternoon here in the EDT zone),

I am in the process of requesting the installation of Python 3 with matplotlib. The company needs to approve licenses but I cannot find the license for setuptools. The description here<https://pypi.python.org/pypi/setuptools> says it uses an MIT license but I cannot confirm this. On github, the file setup.py says the same thing.

License :: OSI Approved :: MIT License

Could the maintainer please add an explicit license file.

I would really like to use matplotlib but will not get approval unless we can confirm the license.

Thank you.

--
Claude Marinier


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160811/60bbc7f3/attachment.html>

From noah at coderanger.net  Thu Aug 11 23:36:45 2016
From: noah at coderanger.net (Noah Kantrowitz)
Date: Fri, 12 Aug 2016 13:36:45 +1000
Subject: [Distutils] license for setuptools
In-Reply-To: <1AE01599B99F8A4EA0609B55B478D77FE80ABC@HCXDSPM1.ca.lmco.com>
References: <1AE01599B99F8A4EA0609B55B478D77FE80ABC@HCXDSPM1.ca.lmco.com>
Message-ID: <3BFD6955-C3FB-409B-86E7-113E8CC88C7F@coderanger.net>

Hi there, this list is for the discussion of Python's core packaging tools like distutils. We have no control over the packages made or distributed with them. You would have to contact the matplotlib authors, not us.

--Noah

> On Aug 12, 2016, at 7:51 AM, Marinier, Claude <claude.marinier at lmco.com> wrote:
> 
> Good afternoon (well it?s afternoon here in the EDT zone),
> 
> I am in the process of requesting the installation of Python 3 with matplotlib. The company needs to approve licenses but I cannot find the license for setuptools. The description here says it uses an MIT license but I cannot confirm this. On github, the file setup.py says the same thing.
> 
> License :: OSI Approved :: MIT License
> 
> Could the maintainer please add an explicit license file.
> 
> I would really like to use matplotlib but will not get approval unless we can confirm the license.
> 
> Thank you.
> 
> --
> Claude Marinier
> 
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160812/9f02451d/attachment.sig>

From ubernostrum at gmail.com  Fri Aug 12 00:10:44 2016
From: ubernostrum at gmail.com (James Bennett)
Date: Thu, 11 Aug 2016 21:10:44 -0700
Subject: [Distutils] license for setuptools
In-Reply-To: <3BFD6955-C3FB-409B-86E7-113E8CC88C7F@coderanger.net>
References: <1AE01599B99F8A4EA0609B55B478D77FE80ABC@HCXDSPM1.ca.lmco.com>
 <3BFD6955-C3FB-409B-86E7-113E8CC88C7F@coderanger.net>
Message-ID: <CAL13Cg_dRtdBc7H6bDL7xaw-MNy+-rtDSRn=LdQ4koTK4s-6Wg@mail.gmail.com>

I think the question was really about setuptools, which does not have a
license file in its repository:

https://github.com/pypa/setuptools

On Thu, Aug 11, 2016 at 8:36 PM, Noah Kantrowitz <noah at coderanger.net>
wrote:

> Hi there, this list is for the discussion of Python's core packaging tools
> like distutils. We have no control over the packages made or distributed
> with them. You would have to contact the matplotlib authors, not us.
>
> --Noah
>
> > On Aug 12, 2016, at 7:51 AM, Marinier, Claude <claude.marinier at lmco.com>
> wrote:
> >
> > Good afternoon (well it?s afternoon here in the EDT zone),
> >
> > I am in the process of requesting the installation of Python 3 with
> matplotlib. The company needs to approve licenses but I cannot find the
> license for setuptools. The description here says it uses an MIT license
> but I cannot confirm this. On github, the file setup.py says the same thing.
> >
> > License :: OSI Approved :: MIT License
> >
> > Could the maintainer please add an explicit license file.
> >
> > I would really like to use matplotlib but will not get approval unless
> we can confirm the license.
> >
> > Thank you.
> >
> > --
> > Claude Marinier
> >
> >
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160811/0d5ba9fc/attachment.html>

From glyph at twistedmatrix.com  Fri Aug 12 00:11:12 2016
From: glyph at twistedmatrix.com (Glyph)
Date: Thu, 11 Aug 2016 21:11:12 -0700
Subject: [Distutils] license for setuptools
In-Reply-To: <3BFD6955-C3FB-409B-86E7-113E8CC88C7F@coderanger.net>
References: <1AE01599B99F8A4EA0609B55B478D77FE80ABC@HCXDSPM1.ca.lmco.com>
 <3BFD6955-C3FB-409B-86E7-113E8CC88C7F@coderanger.net>
Message-ID: <00D987AA-C8AF-457E-BCEE-78AEA11F4C85@twistedmatrix.com>

The OP specifically asked about setuptools, which is on-topic. In large corporate environments it is standard to have a process that evaluates every single dependency, so setuptools does end up in the bucket with matplotlib since you need it to install.

> On Aug 11, 2016, at 8:36 PM, Noah Kantrowitz <noah at coderanger.net> wrote:
> 
> Hi there, this list is for the discussion of Python's core packaging tools like distutils. We have no control over the packages made or distributed with them. You would have to contact the matplotlib authors, not us.
> 
> --Noah
> 
>> On Aug 12, 2016, at 7:51 AM, Marinier, Claude <claude.marinier at lmco.com> wrote:
>> 
>> Good afternoon (well it?s afternoon here in the EDT zone),
>> 
>> I am in the process of requesting the installation of Python 3 with matplotlib. The company needs to approve licenses but I cannot find the license for setuptools. The description here says it uses an MIT license but I cannot confirm this. On github, the file setup.py says the same thing.
>> 
>> License :: OSI Approved :: MIT License
>> 
>> Could the maintainer please add an explicit license file.
>> 
>> I would really like to use matplotlib but will not get approval unless we can confirm the license.
>> 
>> Thank you.
>> 
>> --
>> Claude Marinier
>> 
>> 
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
> 
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


From graffatcolmingov at gmail.com  Fri Aug 12 07:16:10 2016
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Fri, 12 Aug 2016 06:16:10 -0500
Subject: [Distutils] license for setuptools
In-Reply-To: <00D987AA-C8AF-457E-BCEE-78AEA11F4C85@twistedmatrix.com>
References: <1AE01599B99F8A4EA0609B55B478D77FE80ABC@HCXDSPM1.ca.lmco.com>
 <3BFD6955-C3FB-409B-86E7-113E8CC88C7F@coderanger.net>
 <00D987AA-C8AF-457E-BCEE-78AEA11F4C85@twistedmatrix.com>
Message-ID: <CAN-Kwu1TE8up2Ay_KQxs-kUJWq-w2DBai=hanaPugZ4wk02LWQ@mail.gmail.com>

It seems that in our attempt to help Claude, both Glyph and I have
found: https://github.com/pypa/setuptools/issues/612

On Thu, Aug 11, 2016 at 11:11 PM, Glyph <glyph at twistedmatrix.com> wrote:
> The OP specifically asked about setuptools, which is on-topic. In large corporate environments it is standard to have a process that evaluates every single dependency, so setuptools does end up in the bucket with matplotlib since you need it to install.
>
>> On Aug 11, 2016, at 8:36 PM, Noah Kantrowitz <noah at coderanger.net> wrote:
>>
>> Hi there, this list is for the discussion of Python's core packaging tools like distutils. We have no control over the packages made or distributed with them. You would have to contact the matplotlib authors, not us.
>>
>> --Noah
>>
>>> On Aug 12, 2016, at 7:51 AM, Marinier, Claude <claude.marinier at lmco.com> wrote:
>>>
>>> Good afternoon (well it?s afternoon here in the EDT zone),
>>>
>>> I am in the process of requesting the installation of Python 3 with matplotlib. The company needs to approve licenses but I cannot find the license for setuptools. The description here says it uses an MIT license but I cannot confirm this. On github, the file setup.py says the same thing.
>>>
>>> License :: OSI Approved :: MIT License
>>>
>>> Could the maintainer please add an explicit license file.
>>>
>>> I would really like to use matplotlib but will not get approval unless we can confirm the license.
>>>
>>> Thank you.
>>>
>>> --
>>> Claude Marinier
>>>
>>>
>>> _______________________________________________
>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

From thedizzle at gmail.com  Thu Aug 11 22:45:20 2016
From: thedizzle at gmail.com (Eric Dill)
Date: Fri, 12 Aug 2016 02:45:20 +0000
Subject: [Distutils] license for setuptools
In-Reply-To: <1AE01599B99F8A4EA0609B55B478D77FE80ABC@HCXDSPM1.ca.lmco.com>
References: <1AE01599B99F8A4EA0609B55B478D77FE80ABC@HCXDSPM1.ca.lmco.com>
Message-ID: <CANrsn24uiEBdjkffMRV99Wpqxvf5TLrzkFfNHQ=Hb6VMe9Fo7A@mail.gmail.com>

Hi Claude,

There was a recent discussion of the lack of a license file in setuptools
here: https://github.com/pypa/setuptools/issues/612 and another important
discussion here: https://github.com/pypa/setuptools/issues/132.  This is
probably the most relevant quotable bit from those two issues, from Jason
Coombs (the primary developer of setuptools:

"The [License :: OSI Approved :: MIT License] classifier isn't a suggestion
but a declaration and follows the distutils guide for declaring a license.
I consider inclusion of a license file redundant and error prone."

Hopefully this resolves your issue about not having an explicit license
file.

Best,

Eric

On Thu, Aug 11, 2016 at 10:20 PM Marinier, Claude <claude.marinier at lmco.com>
wrote:

> Good afternoon (well it?s afternoon here in the EDT zone),
>
>
>
> I am in the process of requesting the installation of Python 3 with
> matplotlib. The company needs to approve licenses but I cannot find the
> license for setuptools. The description here
> <https://pypi.python.org/pypi/setuptools> says it uses an MIT license but
> I cannot confirm this. On github, the file setup.py says the same thing.
>
>
>
> License :: OSI Approved :: MIT License
>
>
>
> Could the maintainer please add an explicit license file.
>
>
>
> I would really like to use matplotlib but will not get approval unless we
> can confirm the license.
>
>
>
> Thank you.
>
>
>
> --
>
> Claude Marinier
>
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160812/94de38d9/attachment.html>

From geoffspear at gmail.com  Fri Aug 12 09:00:45 2016
From: geoffspear at gmail.com (Geoffrey Spear)
Date: Fri, 12 Aug 2016 09:00:45 -0400
Subject: [Distutils] license for setuptools
In-Reply-To: <CANrsn24uiEBdjkffMRV99Wpqxvf5TLrzkFfNHQ=Hb6VMe9Fo7A@mail.gmail.com>
References: <1AE01599B99F8A4EA0609B55B478D77FE80ABC@HCXDSPM1.ca.lmco.com>
 <CANrsn24uiEBdjkffMRV99Wpqxvf5TLrzkFfNHQ=Hb6VMe9Fo7A@mail.gmail.com>
Message-ID: <CAGifb9GcbufMe0tytM536V5sHmNYxYPUEyrPsYyzb3yZXEVfRA@mail.gmail.com>

It seems a bit silly to claim that a license that contains the sentence
"The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software" shouldn't have a copy
included with the software it applies to since it literally says you need
it.

(IANAL)

On Thu, Aug 11, 2016 at 10:45 PM, Eric Dill <thedizzle at gmail.com> wrote:

> Hi Claude,
>
> There was a recent discussion of the lack of a license file in setuptools
> here: https://github.com/pypa/setuptools/issues/612 and another important
> discussion here: https://github.com/pypa/setuptools/issues/132.  This is
> probably the most relevant quotable bit from those two issues, from Jason
> Coombs (the primary developer of setuptools:
>
> "The [License :: OSI Approved :: MIT License] classifier isn't a
> suggestion but a declaration and follows the distutils guide for declaring
> a license. I consider inclusion of a license file redundant and error
> prone."
>
> Hopefully this resolves your issue about not having an explicit license
> file.
>
> Best,
>
> Eric
>
> On Thu, Aug 11, 2016 at 10:20 PM Marinier, Claude <
> claude.marinier at lmco.com> wrote:
>
>> Good afternoon (well it?s afternoon here in the EDT zone),
>>
>>
>>
>> I am in the process of requesting the installation of Python 3 with
>> matplotlib. The company needs to approve licenses but I cannot find the
>> license for setuptools. The description here
>> <https://pypi.python.org/pypi/setuptools> says it uses an MIT license
>> but I cannot confirm this. On github, the file setup.py says the same thing.
>>
>>
>>
>> License :: OSI Approved :: MIT License
>>
>>
>>
>> Could the maintainer please add an explicit license file.
>>
>>
>>
>> I would really like to use matplotlib but will not get approval unless we
>> can confirm the license.
>>
>>
>>
>> Thank you.
>>
>>
>>
>> --
>>
>> Claude Marinier
>>
>>
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160812/6544970d/attachment-0001.html>

From graffatcolmingov at gmail.com  Fri Aug 12 09:24:00 2016
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Fri, 12 Aug 2016 08:24:00 -0500
Subject: [Distutils] license for setuptools
In-Reply-To: <CAGifb9GcbufMe0tytM536V5sHmNYxYPUEyrPsYyzb3yZXEVfRA@mail.gmail.com>
References: <1AE01599B99F8A4EA0609B55B478D77FE80ABC@HCXDSPM1.ca.lmco.com>
 <CANrsn24uiEBdjkffMRV99Wpqxvf5TLrzkFfNHQ=Hb6VMe9Fo7A@mail.gmail.com>
 <CAGifb9GcbufMe0tytM536V5sHmNYxYPUEyrPsYyzb3yZXEVfRA@mail.gmail.com>
Message-ID: <CAN-Kwu2JHi9ooMKF24A7G7PALAYBe8KsQEX-twpQXuXRa7VUKw@mail.gmail.com>

Thanks for that Geoffrey. There's a PR to add it as Jason decided to
accept it. I think we can all relax now. Okay?

On Fri, Aug 12, 2016 at 8:00 AM, Geoffrey Spear <geoffspear at gmail.com> wrote:
> It seems a bit silly to claim that a license that contains the sentence "The
> above copyright notice and this permission notice shall be included in all
> copies or substantial portions of the Software" shouldn't have a copy
> included with the software it applies to since it literally says you need
> it.
>
> (IANAL)
>
> On Thu, Aug 11, 2016 at 10:45 PM, Eric Dill <thedizzle at gmail.com> wrote:
>>
>> Hi Claude,
>>
>> There was a recent discussion of the lack of a license file in setuptools
>> here: https://github.com/pypa/setuptools/issues/612 and another important
>> discussion here: https://github.com/pypa/setuptools/issues/132.  This is
>> probably the most relevant quotable bit from those two issues, from Jason
>> Coombs (the primary developer of setuptools:
>>
>> "The [License :: OSI Approved :: MIT License] classifier isn't a
>> suggestion but a declaration and follows the distutils guide for declaring a
>> license. I consider inclusion of a license file redundant and error prone."
>>
>> Hopefully this resolves your issue about not having an explicit license
>> file.
>>
>> Best,
>>
>> Eric
>>
>> On Thu, Aug 11, 2016 at 10:20 PM Marinier, Claude
>> <claude.marinier at lmco.com> wrote:
>>>
>>> Good afternoon (well it?s afternoon here in the EDT zone),
>>>
>>>
>>>
>>> I am in the process of requesting the installation of Python 3 with
>>> matplotlib. The company needs to approve licenses but I cannot find the
>>> license for setuptools. The description here says it uses an MIT license but
>>> I cannot confirm this. On github, the file setup.py says the same thing.
>>>
>>>
>>>
>>> License :: OSI Approved :: MIT License
>>>
>>>
>>>
>>> Could the maintainer please add an explicit license file.
>>>
>>>
>>>
>>> I would really like to use matplotlib but will not get approval unless we
>>> can confirm the license.
>>>
>>>
>>>
>>> Thank you.
>>>
>>>
>>>
>>> --
>>>
>>> Claude Marinier
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>

From chris.barker at noaa.gov  Fri Aug 12 13:01:13 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 12 Aug 2016 10:01:13 -0700
Subject: [Distutils] license for setuptools
In-Reply-To: <3BFD6955-C3FB-409B-86E7-113E8CC88C7F@coderanger.net>
References: <1AE01599B99F8A4EA0609B55B478D77FE80ABC@HCXDSPM1.ca.lmco.com>
 <3BFD6955-C3FB-409B-86E7-113E8CC88C7F@coderanger.net>
Message-ID: <CALGmxE+7cETeXeb=697AEsDhjm=biVY79oX2g4wOk3FFgThGiQ@mail.gmail.com>

it looks like the OP is looking for the license to setuptools, not
matplotlib.

(MPL's licence file is here:
https://github.com/matplotlib/matplotlib/blob/master/LICENSE/LICENSE)

but setuptools does not appear to have one:

https://github.com/pypa/setuptools

and it is a pypa package, so this is kind the right list for it -- though
I"d post an issue on gitHub.

BTW, I notice that the Python Packaging User Guide doesn't recommend a
LICENSE file -- it probably should.

NOTE to the OP: almost every python package depends on setuptools, so this
really has nothing to do with Matplotlib :-)

-CHB


On Thu, Aug 11, 2016 at 8:36 PM, Noah Kantrowitz <noah at coderanger.net>
wrote:

> Hi there, this list is for the discussion of Python's core packaging tools
> like distutils. We have no control over the packages made or distributed
> with them. You would have to contact the matplotlib authors, not us.
>
> --Noah
>
> > On Aug 12, 2016, at 7:51 AM, Marinier, Claude <claude.marinier at lmco.com>
> wrote:
> >
> > Good afternoon (well it?s afternoon here in the EDT zone),
> >
> > I am in the process of requesting the installation of Python 3 with
> matplotlib. The company needs to approve licenses but I cannot find the
> license for setuptools. The description here says it uses an MIT license
> but I cannot confirm this. On github, the file setup.py says the same thing.
> >
> > License :: OSI Approved :: MIT License
> >
> > Could the maintainer please add an explicit license file.
> >
> > I would really like to use matplotlib but will not get approval unless
> we can confirm the license.
> >
> > Thank you.
> >
> > --
> > Claude Marinier
> >
> >
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160812/e1a82771/attachment.html>

From chris.barker at noaa.gov  Fri Aug 12 13:02:22 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 12 Aug 2016 10:02:22 -0700
Subject: [Distutils] license for setuptools
In-Reply-To: <CALGmxE+7cETeXeb=697AEsDhjm=biVY79oX2g4wOk3FFgThGiQ@mail.gmail.com>
References: <1AE01599B99F8A4EA0609B55B478D77FE80ABC@HCXDSPM1.ca.lmco.com>
 <3BFD6955-C3FB-409B-86E7-113E8CC88C7F@coderanger.net>
 <CALGmxE+7cETeXeb=697AEsDhjm=biVY79oX2g4wOk3FFgThGiQ@mail.gmail.com>
Message-ID: <CALGmxELSkusXDrkH7Ms7ozQSndhh5-dH2qaMZX3BPRQitTq+Pg@mail.gmail.com>

Sorry for the noise  -- I was way too late to this game -- blib in my
email, I guess.

-CHB


On Fri, Aug 12, 2016 at 10:01 AM, Chris Barker <chris.barker at noaa.gov>
wrote:

> it looks like the OP is looking for the license to setuptools, not
> matplotlib.
>
> (MPL's licence file is here: https://github.com/
> matplotlib/matplotlib/blob/master/LICENSE/LICENSE)
>
> but setuptools does not appear to have one:
>
> https://github.com/pypa/setuptools
>
> and it is a pypa package, so this is kind the right list for it -- though
> I"d post an issue on gitHub.
>
> BTW, I notice that the Python Packaging User Guide doesn't recommend a
> LICENSE file -- it probably should.
>
> NOTE to the OP: almost every python package depends on setuptools, so this
> really has nothing to do with Matplotlib :-)
>
> -CHB
>
>
> On Thu, Aug 11, 2016 at 8:36 PM, Noah Kantrowitz <noah at coderanger.net>
> wrote:
>
>> Hi there, this list is for the discussion of Python's core packaging
>> tools like distutils. We have no control over the packages made or
>> distributed with them. You would have to contact the matplotlib authors,
>> not us.
>>
>> --Noah
>>
>> > On Aug 12, 2016, at 7:51 AM, Marinier, Claude <claude.marinier at lmco.com>
>> wrote:
>> >
>> > Good afternoon (well it?s afternoon here in the EDT zone),
>> >
>> > I am in the process of requesting the installation of Python 3 with
>> matplotlib. The company needs to approve licenses but I cannot find the
>> license for setuptools. The description here says it uses an MIT license
>> but I cannot confirm this. On github, the file setup.py says the same thing.
>> >
>> > License :: OSI Approved :: MIT License
>> >
>> > Could the maintainer please add an explicit license file.
>> >
>> > I would really like to use matplotlib but will not get approval unless
>> we can confirm the license.
>> >
>> > Thank you.
>> >
>> > --
>> > Claude Marinier
>> >
>> >
>> > _______________________________________________
>> > Distutils-SIG maillist  -  Distutils-SIG at python.org
>> > https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160812/21cb6761/attachment.html>

From glyph at twistedmatrix.com  Fri Aug 12 13:34:21 2016
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Fri, 12 Aug 2016 10:34:21 -0700
Subject: [Distutils] license for setuptools
In-Reply-To: <CAN-Kwu2JHi9ooMKF24A7G7PALAYBe8KsQEX-twpQXuXRa7VUKw@mail.gmail.com>
References: <1AE01599B99F8A4EA0609B55B478D77FE80ABC@HCXDSPM1.ca.lmco.com>
 <CANrsn24uiEBdjkffMRV99Wpqxvf5TLrzkFfNHQ=Hb6VMe9Fo7A@mail.gmail.com>
 <CAGifb9GcbufMe0tytM536V5sHmNYxYPUEyrPsYyzb3yZXEVfRA@mail.gmail.com>
 <CAN-Kwu2JHi9ooMKF24A7G7PALAYBe8KsQEX-twpQXuXRa7VUKw@mail.gmail.com>
Message-ID: <49749632-512B-4F11-B6DE-C98E7C9E944E@twistedmatrix.com>

Thanks for submitting this PR, Ian! :)

> On Aug 12, 2016, at 6:24 AM, Ian Cordasco <graffatcolmingov at gmail.com> wrote:
> 
> Thanks for that Geoffrey. There's a PR to add it as Jason decided to
> accept it. I think we can all relax now. Okay?
> 
> On Fri, Aug 12, 2016 at 8:00 AM, Geoffrey Spear <geoffspear at gmail.com> wrote:
>> It seems a bit silly to claim that a license that contains the sentence "The
>> above copyright notice and this permission notice shall be included in all
>> copies or substantial portions of the Software" shouldn't have a copy
>> included with the software it applies to since it literally says you need
>> it.
>> 
>> (IANAL)
>> 
>> On Thu, Aug 11, 2016 at 10:45 PM, Eric Dill <thedizzle at gmail.com> wrote:
>>> 
>>> Hi Claude,
>>> 
>>> There was a recent discussion of the lack of a license file in setuptools
>>> here: https://github.com/pypa/setuptools/issues/612 and another important
>>> discussion here: https://github.com/pypa/setuptools/issues/132.  This is
>>> probably the most relevant quotable bit from those two issues, from Jason
>>> Coombs (the primary developer of setuptools:
>>> 
>>> "The [License :: OSI Approved :: MIT License] classifier isn't a
>>> suggestion but a declaration and follows the distutils guide for declaring a
>>> license. I consider inclusion of a license file redundant and error prone."
>>> 
>>> Hopefully this resolves your issue about not having an explicit license
>>> file.
>>> 
>>> Best,
>>> 
>>> Eric
>>> 
>>> On Thu, Aug 11, 2016 at 10:20 PM Marinier, Claude
>>> <claude.marinier at lmco.com> wrote:
>>>> 
>>>> Good afternoon (well it?s afternoon here in the EDT zone),
>>>> 
>>>> 
>>>> 
>>>> I am in the process of requesting the installation of Python 3 with
>>>> matplotlib. The company needs to approve licenses but I cannot find the
>>>> license for setuptools. The description here says it uses an MIT license but
>>>> I cannot confirm this. On github, the file setup.py says the same thing.
>>>> 
>>>> 
>>>> 
>>>> License :: OSI Approved :: MIT License
>>>> 
>>>> 
>>>> 
>>>> Could the maintainer please add an explicit license file.
>>>> 
>>>> 
>>>> 
>>>> I would really like to use matplotlib but will not get approval unless we
>>>> can confirm the license.
>>>> 
>>>> 
>>>> 
>>>> Thank you.
>>>> 
>>>> 
>>>> 
>>>> --
>>>> 
>>>> Claude Marinier
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>> 
>>> 
>>> _______________________________________________
>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>> 
>> 
>> 
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


From prometheus235 at gmail.com  Fri Aug 12 14:52:39 2016
From: prometheus235 at gmail.com (Nick Timkovich)
Date: Fri, 12 Aug 2016 13:52:39 -0500
Subject: [Distutils] license for setuptools
In-Reply-To: <49749632-512B-4F11-B6DE-C98E7C9E944E@twistedmatrix.com>
References: <1AE01599B99F8A4EA0609B55B478D77FE80ABC@HCXDSPM1.ca.lmco.com>
 <CANrsn24uiEBdjkffMRV99Wpqxvf5TLrzkFfNHQ=Hb6VMe9Fo7A@mail.gmail.com>
 <CAGifb9GcbufMe0tytM536V5sHmNYxYPUEyrPsYyzb3yZXEVfRA@mail.gmail.com>
 <CAN-Kwu2JHi9ooMKF24A7G7PALAYBe8KsQEX-twpQXuXRa7VUKw@mail.gmail.com>
 <49749632-512B-4F11-B6DE-C98E7C9E944E@twistedmatrix.com>
Message-ID: <CAHkxivfxM2jbJWq4dbV9GLq3q_foKSG9byNbX0Lytc_ZgwYg3g@mail.gmail.com>

Might be wandering away from simply tacking on a license, but some related
legalese: Is there some canned contributor license agreement (CLA) that
could also be applied to make clear that contributors must license their
contributions to the project(s) in kind? Python has it's own (
https://www.python.org/psf/contrib/) but that seems very formal. Is there a
simpler one that just makes it implicit that 'submitting a patch/PR = you
agree to license that code to the project for redistribution" or whatever.

On Fri, Aug 12, 2016 at 12:34 PM, Glyph Lefkowitz <glyph at twistedmatrix.com>
wrote:

> Thanks for submitting this PR, Ian! :)
>
> > On Aug 12, 2016, at 6:24 AM, Ian Cordasco <graffatcolmingov at gmail.com>
> wrote:
> >
> > Thanks for that Geoffrey. There's a PR to add it as Jason decided to
> > accept it. I think we can all relax now. Okay?
> >
> > On Fri, Aug 12, 2016 at 8:00 AM, Geoffrey Spear <geoffspear at gmail.com>
> wrote:
> >> It seems a bit silly to claim that a license that contains the sentence
> "The
> >> above copyright notice and this permission notice shall be included in
> all
> >> copies or substantial portions of the Software" shouldn't have a copy
> >> included with the software it applies to since it literally says you
> need
> >> it.
> >>
> >> (IANAL)
> >>
> >> On Thu, Aug 11, 2016 at 10:45 PM, Eric Dill <thedizzle at gmail.com>
> wrote:
> >>>
> >>> Hi Claude,
> >>>
> >>> There was a recent discussion of the lack of a license file in
> setuptools
> >>> here: https://github.com/pypa/setuptools/issues/612 and another
> important
> >>> discussion here: https://github.com/pypa/setuptools/issues/132.  This
> is
> >>> probably the most relevant quotable bit from those two issues, from
> Jason
> >>> Coombs (the primary developer of setuptools:
> >>>
> >>> "The [License :: OSI Approved :: MIT License] classifier isn't a
> >>> suggestion but a declaration and follows the distutils guide for
> declaring a
> >>> license. I consider inclusion of a license file redundant and error
> prone."
> >>>
> >>> Hopefully this resolves your issue about not having an explicit license
> >>> file.
> >>>
> >>> Best,
> >>>
> >>> Eric
> >>>
> >>> On Thu, Aug 11, 2016 at 10:20 PM Marinier, Claude
> >>> <claude.marinier at lmco.com> wrote:
> >>>>
> >>>> Good afternoon (well it?s afternoon here in the EDT zone),
> >>>>
> >>>>
> >>>>
> >>>> I am in the process of requesting the installation of Python 3 with
> >>>> matplotlib. The company needs to approve licenses but I cannot find
> the
> >>>> license for setuptools. The description here says it uses an MIT
> license but
> >>>> I cannot confirm this. On github, the file setup.py says the same
> thing.
> >>>>
> >>>>
> >>>>
> >>>> License :: OSI Approved :: MIT License
> >>>>
> >>>>
> >>>>
> >>>> Could the maintainer please add an explicit license file.
> >>>>
> >>>>
> >>>>
> >>>> I would really like to use matplotlib but will not get approval
> unless we
> >>>> can confirm the license.
> >>>>
> >>>>
> >>>>
> >>>> Thank you.
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>>
> >>>> Claude Marinier
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
> >>>> https://mail.python.org/mailman/listinfo/distutils-sig
> >>>
> >>>
> >>> _______________________________________________
> >>> Distutils-SIG maillist  -  Distutils-SIG at python.org
> >>> https://mail.python.org/mailman/listinfo/distutils-sig
> >>>
> >>
> >>
> >> _______________________________________________
> >> Distutils-SIG maillist  -  Distutils-SIG at python.org
> >> https://mail.python.org/mailman/listinfo/distutils-sig
> >>
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160812/7d49b5fd/attachment.html>

From glyph at twistedmatrix.com  Fri Aug 12 14:59:28 2016
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Fri, 12 Aug 2016 11:59:28 -0700
Subject: [Distutils] license for setuptools
In-Reply-To: <CAHkxivfxM2jbJWq4dbV9GLq3q_foKSG9byNbX0Lytc_ZgwYg3g@mail.gmail.com>
References: <1AE01599B99F8A4EA0609B55B478D77FE80ABC@HCXDSPM1.ca.lmco.com>
 <CANrsn24uiEBdjkffMRV99Wpqxvf5TLrzkFfNHQ=Hb6VMe9Fo7A@mail.gmail.com>
 <CAGifb9GcbufMe0tytM536V5sHmNYxYPUEyrPsYyzb3yZXEVfRA@mail.gmail.com>
 <CAN-Kwu2JHi9ooMKF24A7G7PALAYBe8KsQEX-twpQXuXRa7VUKw@mail.gmail.com>
 <49749632-512B-4F11-B6DE-C98E7C9E944E@twistedmatrix.com>
 <CAHkxivfxM2jbJWq4dbV9GLq3q_foKSG9byNbX0Lytc_ZgwYg3g@mail.gmail.com>
Message-ID: <3D354736-2660-4BE7-B666-CD0C07D89803@twistedmatrix.com>

Yes, it's called "the apache license" - and I'm pretty sure setuptools rejected requiring that for good reason :-).

CLAs are a much bigger issue and are generally quite controversial.  Having one is nice if you need to do some kind of legal action but they are also a big impediment to contribution, and tracking them is a nightmare.  Not to mention the fact that there's almost no point unless you can go back and get one from everyone who has ever contributed.

-g

> On Aug 12, 2016, at 11:52 AM, Nick Timkovich <prometheus235 at gmail.com> wrote:
> 
> Might be wandering away from simply tacking on a license, but some related legalese: Is there some canned contributor license agreement (CLA) that could also be applied to make clear that contributors must license their contributions to the project(s) in kind? Python has it's own (https://www.python.org/psf/contrib/ <https://www.python.org/psf/contrib/>) but that seems very formal. Is there a simpler one that just makes it implicit that 'submitting a patch/PR = you agree to license that code to the project for redistribution" or whatever.
> 
> On Fri, Aug 12, 2016 at 12:34 PM, Glyph Lefkowitz <glyph at twistedmatrix.com <mailto:glyph at twistedmatrix.com>> wrote:
> Thanks for submitting this PR, Ian! :)
> 
> > On Aug 12, 2016, at 6:24 AM, Ian Cordasco <graffatcolmingov at gmail.com <mailto:graffatcolmingov at gmail.com>> wrote:
> >
> > Thanks for that Geoffrey. There's a PR to add it as Jason decided to
> > accept it. I think we can all relax now. Okay?
> >
> > On Fri, Aug 12, 2016 at 8:00 AM, Geoffrey Spear <geoffspear at gmail.com <mailto:geoffspear at gmail.com>> wrote:
> >> It seems a bit silly to claim that a license that contains the sentence "The
> >> above copyright notice and this permission notice shall be included in all
> >> copies or substantial portions of the Software" shouldn't have a copy
> >> included with the software it applies to since it literally says you need
> >> it.
> >>
> >> (IANAL)
> >>
> >> On Thu, Aug 11, 2016 at 10:45 PM, Eric Dill <thedizzle at gmail.com <mailto:thedizzle at gmail.com>> wrote:
> >>>
> >>> Hi Claude,
> >>>
> >>> There was a recent discussion of the lack of a license file in setuptools
> >>> here: https://github.com/pypa/setuptools/issues/612 <https://github.com/pypa/setuptools/issues/612> and another important
> >>> discussion here: https://github.com/pypa/setuptools/issues/132 <https://github.com/pypa/setuptools/issues/132>.  This is
> >>> probably the most relevant quotable bit from those two issues, from Jason
> >>> Coombs (the primary developer of setuptools:
> >>>
> >>> "The [License :: OSI Approved :: MIT License] classifier isn't a
> >>> suggestion but a declaration and follows the distutils guide for declaring a
> >>> license. I consider inclusion of a license file redundant and error prone."
> >>>
> >>> Hopefully this resolves your issue about not having an explicit license
> >>> file.
> >>>
> >>> Best,
> >>>
> >>> Eric
> >>>
> >>> On Thu, Aug 11, 2016 at 10:20 PM Marinier, Claude
> >>> <claude.marinier at lmco.com <mailto:claude.marinier at lmco.com>> wrote:
> >>>>
> >>>> Good afternoon (well it?s afternoon here in the EDT zone),
> >>>>
> >>>>
> >>>>
> >>>> I am in the process of requesting the installation of Python 3 with
> >>>> matplotlib. The company needs to approve licenses but I cannot find the
> >>>> license for setuptools. The description here says it uses an MIT license but
> >>>> I cannot confirm this. On github, the file setup.py says the same thing.
> >>>>
> >>>>
> >>>>
> >>>> License :: OSI Approved :: MIT License
> >>>>
> >>>>
> >>>>
> >>>> Could the maintainer please add an explicit license file.
> >>>>
> >>>>
> >>>>
> >>>> I would really like to use matplotlib but will not get approval unless we
> >>>> can confirm the license.
> >>>>
> >>>>
> >>>>
> >>>> Thank you.
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>>
> >>>> Claude Marinier
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Distutils-SIG maillist  -  Distutils-SIG at python.org <mailto:Distutils-SIG at python.org>
> >>>> https://mail.python.org/mailman/listinfo/distutils-sig <https://mail.python.org/mailman/listinfo/distutils-sig>
> >>>
> >>>
> >>> _______________________________________________
> >>> Distutils-SIG maillist  -  Distutils-SIG at python.org <mailto:Distutils-SIG at python.org>
> >>> https://mail.python.org/mailman/listinfo/distutils-sig <https://mail.python.org/mailman/listinfo/distutils-sig>
> >>>
> >>
> >>
> >> _______________________________________________
> >> Distutils-SIG maillist  -  Distutils-SIG at python.org <mailto:Distutils-SIG at python.org>
> >> https://mail.python.org/mailman/listinfo/distutils-sig <https://mail.python.org/mailman/listinfo/distutils-sig>
> >>
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org <mailto:Distutils-SIG at python.org>
> > https://mail.python.org/mailman/listinfo/distutils-sig <https://mail.python.org/mailman/listinfo/distutils-sig>
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org <mailto:Distutils-SIG at python.org>
> https://mail.python.org/mailman/listinfo/distutils-sig <https://mail.python.org/mailman/listinfo/distutils-sig>
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160812/3d8be67e/attachment-0001.html>

From ncoghlan at gmail.com  Sun Aug 14 09:40:25 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 14 Aug 2016 23:40:25 +1000
Subject: [Distutils] license for setuptools
In-Reply-To: <CADiSq7dAgE=KxSuhzMBxEPeNKRetPxJ+WitUXk-SNS_pUJzbAw@mail.gmail.com>
References: <1AE01599B99F8A4EA0609B55B478D77FE80ABC@HCXDSPM1.ca.lmco.com>
 <CANrsn24uiEBdjkffMRV99Wpqxvf5TLrzkFfNHQ=Hb6VMe9Fo7A@mail.gmail.com>
 <CAGifb9GcbufMe0tytM536V5sHmNYxYPUEyrPsYyzb3yZXEVfRA@mail.gmail.com>
 <CAN-Kwu2JHi9ooMKF24A7G7PALAYBe8KsQEX-twpQXuXRa7VUKw@mail.gmail.com>
 <49749632-512B-4F11-B6DE-C98E7C9E944E@twistedmatrix.com>
 <CAHkxivfxM2jbJWq4dbV9GLq3q_foKSG9byNbX0Lytc_ZgwYg3g@mail.gmail.com>
 <CADiSq7eU3dsd8F8yXUYt5=cfsW32whSc0NwUchVkUS1UCRGpbQ@mail.gmail.com>
 <CADiSq7dAgE=KxSuhzMBxEPeNKRetPxJ+WitUXk-SNS_pUJzbAw@mail.gmail.com>
Message-ID: <CADiSq7d781wqGB6jUzC-bChO5E4trgGwVNoW04oCWYWS4nqDHQ@mail.gmail.com>

On 13 Aug 2016 04:53, "Nick Timkovich" <prometheus235 at gmail.com> wrote:
>
> Might be wandering away from simply tacking on a license, but some
related legalese: Is there some canned contributor license agreement (CLA)
that could also be applied to make clear that contributors must license
their contributions to the project(s) in kind? Python has it's own (
https://www.python.org/psf/contrib/) but that seems very formal. Is there a
simpler one that just makes it implicit that 'submitting a patch/PR = you
agree to license that code to the project for redistribution" or whatever.

"Licence in = licence out" is the implied default for contributions when
there's no formal contributor agreement to specify otherwise. (I'm not
aware of any case law in any given jurisdiction to back that up, but it's a
sufficiently well established practice that it's highly unlikely to ever be
legally challenged)

The PSF has a CLA for CPython because the licensing history and hence the
associated licence are rather messy, and not something the PSF can use to
accept contributions.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160814/30f807cb/attachment.html>

From apparitionsec at gmail.com  Sun Aug 14 00:28:14 2016
From: apparitionsec at gmail.com (hyp3rlinx)
Date: Sun, 14 Aug 2016 00:28:14 -0400
Subject: [Distutils] Pip maintainers contact
Message-ID: <CAFD2FDNvBzP0=3AQuC6oH2p6f5uVikwtNTvZZFpZAnAjc2WbeA@mail.gmail.com>

Can somebody here please tell me the contact email for maintainers of pip.

Thank you,
John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160814/e9f26a97/attachment.html>

From robertc at robertcollins.net  Mon Aug 15 12:31:35 2016
From: robertc at robertcollins.net (Robert Collins)
Date: Mon, 15 Aug 2016 09:31:35 -0700
Subject: [Distutils] Pip maintainers contact
In-Reply-To: <CAFD2FDNvBzP0=3AQuC6oH2p6f5uVikwtNTvZZFpZAnAjc2WbeA@mail.gmail.com>
References: <CAFD2FDNvBzP0=3AQuC6oH2p6f5uVikwtNTvZZFpZAnAjc2WbeA@mail.gmail.com>
Message-ID: <CAJ3HoZ3PvHtYVxZ5iQFzyEKMUbtwtMofaZknTUTwGCCpSDyAjQ@mail.gmail.com>

There is no email contact per se. The project tacker is here -
github.com/pypa/pip and there is a google group here -
https://groups.google.com/forum/#!forum/pypa-dev

On 13 August 2016 at 21:28, hyp3rlinx <apparitionsec at gmail.com> wrote:
> Can somebody here please tell me the contact email for maintainers of pip.
>
> Thank you,
> John
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>

From donald at stufft.io  Mon Aug 15 15:09:03 2016
From: donald at stufft.io (Donald Stufft)
Date: Mon, 15 Aug 2016 15:09:03 -0400
Subject: [Distutils] Deprecating little used file types/extensions on PyPI?
Message-ID: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>

Hello!

I'd like to restrict what folks can upload to PyPI in an effort to help narrow
the scope down and to enable more a more consistent experience for everyone.

First off, we currently allow people to upload sdist, bdist_wheel, bdist_egg,
bdist_dmg, bdist_dumb, bdist_msi, bdist_rpm, and bdist_wininst. However I think
that we should try to get rid of support for most of these. Just for reference
currently the number of files uploaded for each type of file looks like:

* sdist: 506,585
* bdist_wheel: 81,207
* bdist_egg: 48,282
* bdist_wininst: 14,002
* bdist_dumb: 5,502
* bdist_msi: 497
* bdist_rpm: 464
* bdist_dmg: 45

Out of all of these, I think that we can easily remove bdist_dmg, bdist_rpm,
and bdist_dumb. I also believe that there is a strong case for removing
bdist_msi and bdist_wininst. I also think we should consider removing
bdist_egg.

First of all, when I say "remove", I mean disallow new uploads, but do not
delete the existing files.

Looking at each file type:

I think that bdist_dumb is a pretty easy one to remove. It's format is such
that it's basically not a very useful format to begin with. It takes the full
path to the files and stores them in the repository. So If I install something
to /opt/mycoolproject/lib/python3.5/site-packages/froblib/ then it will have
paths that look like opt/mycoolproject/lib/python3.5/site-packages/froblib/...
I think this is obviously not very useful and not many people have uploaded any
bdist_dumb files at all. They are also problematic because they have the same
file extension as sdists, so pip doesn't really have a great way to
differentiate between bdist_dumbs and sdists except by trying to guess if it
contains one of distutils's adhoc platform tags.

Looking at bdist_rpm, practically nobody has ever used it with a total of 45
files ever uploaded for it. It's not super useful to be able to upload rpms to
PyPI since it's not an RPM repository so people have to manually download them
and then install them manually. It's also a bit weird to have support for RPMs
but not for all of the other package formats that people might want.

Next we have bdist_dmg, bdist_msi, and bdist_winist. I'm lumping these together
because they're all OS specific installers for OSs that don't already have some
sort of repository. This lack of a repository format for them means that random
downloads are already the norm for people using these systems. For these, I
think the usage numbers for bdist_dmg and bdist_msi easily suggest that they
are not very important to continue to support, but it would be weird to
eliminate them without also elminating bdist_wininst. The wininst format has
the most usage out of all of the seldom used formats, however when we look at
the downloads for the last 30 days only 0.42% of the downloads were for wininst
files, so I think that it's pretty safe to remove them. I think in the past,
these were more important due to the lack of real binary packages on Windows,
but I think in 2016 we have wheel, and Wheel is a better solution. If however
we want to keep them, then I think it's pretty safe to remove them from our
/simple/ pages and any future repository pages and modify our mirroring tooling
to stop mirroring them. IOW, to treat them as some sort of "additional upload"
rather than release uploads to PyPI.

Finally, bdist_egg is quite possibly the trickiest one to justify. A fair
number of people still upload eggs, even though we have the wheel format.
However, I think that we should (and generally do) consider eggs to be
deprecated and while we don't want to break existing packages by removing them,
we should block further uploads for them. Looking again at the download numbers
eggs represented only 1.8% of total downloads in the last 30 days while wheels
represented 41% and sdists represented 56%. Further more, I think we should do
this with the expectation that any new repository API will no longer include
egg files in them, and will just be sdists and wheels.

Doing all of this would leave us with:

* The /simple/ repository only having sdists, wheels, and eggs and disallowing
  new eggs to be uploaded.
* The hypothetical repository 2.0 api only having sdists and wheels.
* *MAYBE* Having "related files" that people could upload things like
  Windows/OSX Installers.

On a related but different note, I'd also like to restrict the acceptable
extensions for sdists. Currently distutils allows people to generate .tar,
.tar.gz, .tgz, .tar.bz2, .tbz, .zip, .tar.xz, .tar.Z and possibly others. This
is a bit problematic because each of those types requires a different set of
optional libraries (which may or may not exist depending on Python version) so
you can't really use a lot of them (for example, while .tar.xz might give you
better compression, it's also not usable before Python 3).

Looking at numbers again, the current number of projects for each file ext are:

* .tar.gz: 444,338
* .zip: 58,774
* .tar.bz2: 3,265
* .tgz: 217
* Everything Else: 0

These results are not particularly surprising since .tar.gz is the default
format that distutils creates. What I would like to do is, similarly to above,
simply stop allowing new uploads for anything but .tar.gz. This will have a few
positive effects:

* It will make it so that (going forward) there is only a single one of the
  optional C libraries that Python can be compiled against that is required for
  someone to install something from PyPI using pip. This would be the zlib
  library which is available almost universally.

* It will reduce confusion because people will not be able to upload two
  different source releases for the same version (e.g. example-1.0.tar.gz and
  example-1.0.zip).

* It will making tooling around PyPI easier, because it'll only have to deal
  with one file extension for sdists.


Thoughts?

?
Donald Stufft




From alex.gronholm at nextday.fi  Mon Aug 15 15:15:36 2016
From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=)
Date: Mon, 15 Aug 2016 22:15:36 +0300
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
Message-ID: <e6e3de37-9cb3-11bb-5ce5-688bbc1e9d99@nextday.fi>

+1 for disallowing uploads of anything else than sdists and wheels.


15.08.2016, 22:09, Donald Stufft kirjoitti:
> Hello!
>
> I'd like to restrict what folks can upload to PyPI in an effort to help narrow
> the scope down and to enable more a more consistent experience for everyone.
>
> First off, we currently allow people to upload sdist, bdist_wheel, bdist_egg,
> bdist_dmg, bdist_dumb, bdist_msi, bdist_rpm, and bdist_wininst. However I think
> that we should try to get rid of support for most of these. Just for reference
> currently the number of files uploaded for each type of file looks like:
>
> * sdist: 506,585
> * bdist_wheel: 81,207
> * bdist_egg: 48,282
> * bdist_wininst: 14,002
> * bdist_dumb: 5,502
> * bdist_msi: 497
> * bdist_rpm: 464
> * bdist_dmg: 45
>
> Out of all of these, I think that we can easily remove bdist_dmg, bdist_rpm,
> and bdist_dumb. I also believe that there is a strong case for removing
> bdist_msi and bdist_wininst. I also think we should consider removing
> bdist_egg.
>
> First of all, when I say "remove", I mean disallow new uploads, but do not
> delete the existing files.
>
> Looking at each file type:
>
> I think that bdist_dumb is a pretty easy one to remove. It's format is such
> that it's basically not a very useful format to begin with. It takes the full
> path to the files and stores them in the repository. So If I install something
> to /opt/mycoolproject/lib/python3.5/site-packages/froblib/ then it will have
> paths that look like opt/mycoolproject/lib/python3.5/site-packages/froblib/...
> I think this is obviously not very useful and not many people have uploaded any
> bdist_dumb files at all. They are also problematic because they have the same
> file extension as sdists, so pip doesn't really have a great way to
> differentiate between bdist_dumbs and sdists except by trying to guess if it
> contains one of distutils's adhoc platform tags.
>
> Looking at bdist_rpm, practically nobody has ever used it with a total of 45
> files ever uploaded for it. It's not super useful to be able to upload rpms to
> PyPI since it's not an RPM repository so people have to manually download them
> and then install them manually. It's also a bit weird to have support for RPMs
> but not for all of the other package formats that people might want.
>
> Next we have bdist_dmg, bdist_msi, and bdist_winist. I'm lumping these together
> because they're all OS specific installers for OSs that don't already have some
> sort of repository. This lack of a repository format for them means that random
> downloads are already the norm for people using these systems. For these, I
> think the usage numbers for bdist_dmg and bdist_msi easily suggest that they
> are not very important to continue to support, but it would be weird to
> eliminate them without also elminating bdist_wininst. The wininst format has
> the most usage out of all of the seldom used formats, however when we look at
> the downloads for the last 30 days only 0.42% of the downloads were for wininst
> files, so I think that it's pretty safe to remove them. I think in the past,
> these were more important due to the lack of real binary packages on Windows,
> but I think in 2016 we have wheel, and Wheel is a better solution. If however
> we want to keep them, then I think it's pretty safe to remove them from our
> /simple/ pages and any future repository pages and modify our mirroring tooling
> to stop mirroring them. IOW, to treat them as some sort of "additional upload"
> rather than release uploads to PyPI.
>
> Finally, bdist_egg is quite possibly the trickiest one to justify. A fair
> number of people still upload eggs, even though we have the wheel format.
> However, I think that we should (and generally do) consider eggs to be
> deprecated and while we don't want to break existing packages by removing them,
> we should block further uploads for them. Looking again at the download numbers
> eggs represented only 1.8% of total downloads in the last 30 days while wheels
> represented 41% and sdists represented 56%. Further more, I think we should do
> this with the expectation that any new repository API will no longer include
> egg files in them, and will just be sdists and wheels.
>
> Doing all of this would leave us with:
>
> * The /simple/ repository only having sdists, wheels, and eggs and disallowing
>    new eggs to be uploaded.
> * The hypothetical repository 2.0 api only having sdists and wheels.
> * *MAYBE* Having "related files" that people could upload things like
>    Windows/OSX Installers.
>
> On a related but different note, I'd also like to restrict the acceptable
> extensions for sdists. Currently distutils allows people to generate .tar,
> .tar.gz, .tgz, .tar.bz2, .tbz, .zip, .tar.xz, .tar.Z and possibly others. This
> is a bit problematic because each of those types requires a different set of
> optional libraries (which may or may not exist depending on Python version) so
> you can't really use a lot of them (for example, while .tar.xz might give you
> better compression, it's also not usable before Python 3).
>
> Looking at numbers again, the current number of projects for each file ext are:
>
> * .tar.gz: 444,338
> * .zip: 58,774
> * .tar.bz2: 3,265
> * .tgz: 217
> * Everything Else: 0
>
> These results are not particularly surprising since .tar.gz is the default
> format that distutils creates. What I would like to do is, similarly to above,
> simply stop allowing new uploads for anything but .tar.gz. This will have a few
> positive effects:
>
> * It will make it so that (going forward) there is only a single one of the
>    optional C libraries that Python can be compiled against that is required for
>    someone to install something from PyPI using pip. This would be the zlib
>    library which is available almost universally.
>
> * It will reduce confusion because people will not be able to upload two
>    different source releases for the same version (e.g. example-1.0.tar.gz and
>    example-1.0.zip).
>
> * It will making tooling around PyPI easier, because it'll only have to deal
>    with one file extension for sdists.
>
>
> Thoughts?
>
> ?
> Donald Stufft
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


From graffatcolmingov at gmail.com  Mon Aug 15 15:22:51 2016
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Mon, 15 Aug 2016 14:22:51 -0500
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
Message-ID: <CAN-Kwu0FsKz7sJgiwZzRoQMzuxBjpBVpEsTXE-XH=B1=6gzOnw@mail.gmail.com>

On Mon, Aug 15, 2016 at 2:09 PM, Donald Stufft <donald at stufft.io> wrote:
> Hello!
>
> I'd like to restrict what folks can upload to PyPI in an effort to help narrow
> the scope down and to enable more a more consistent experience for everyone.
>
> First off, we currently allow people to upload sdist, bdist_wheel, bdist_egg,
> bdist_dmg, bdist_dumb, bdist_msi, bdist_rpm, and bdist_wininst. However I think
> that we should try to get rid of support for most of these. Just for reference
> currently the number of files uploaded for each type of file looks like:
>
> * sdist: 506,585
> * bdist_wheel: 81,207
> * bdist_egg: 48,282
> * bdist_wininst: 14,002
> * bdist_dumb: 5,502
> * bdist_msi: 497
> * bdist_rpm: 464
> * bdist_dmg: 45
>
> Out of all of these, I think that we can easily remove bdist_dmg, bdist_rpm,
> and bdist_dumb. I also believe that there is a strong case for removing
> bdist_msi and bdist_wininst. I also think we should consider removing
> bdist_egg.
>
> First of all, when I say "remove", I mean disallow new uploads, but do not
> delete the existing files.
>
> Looking at each file type:
>
> I think that bdist_dumb is a pretty easy one to remove. It's format is such
> that it's basically not a very useful format to begin with. It takes the full
> path to the files and stores them in the repository. So If I install something
> to /opt/mycoolproject/lib/python3.5/site-packages/froblib/ then it will have
> paths that look like opt/mycoolproject/lib/python3.5/site-packages/froblib/...
> I think this is obviously not very useful and not many people have uploaded any
> bdist_dumb files at all. They are also problematic because they have the same
> file extension as sdists, so pip doesn't really have a great way to
> differentiate between bdist_dumbs and sdists except by trying to guess if it
> contains one of distutils's adhoc platform tags.
>
> Looking at bdist_rpm, practically nobody has ever used it with a total of 45
> files ever uploaded for it. It's not super useful to be able to upload rpms to
> PyPI since it's not an RPM repository so people have to manually download them
> and then install them manually. It's also a bit weird to have support for RPMs
> but not for all of the other package formats that people might want.
>
> Next we have bdist_dmg, bdist_msi, and bdist_winist. I'm lumping these together
> because they're all OS specific installers for OSs that don't already have some
> sort of repository. This lack of a repository format for them means that random
> downloads are already the norm for people using these systems. For these, I
> think the usage numbers for bdist_dmg and bdist_msi easily suggest that they
> are not very important to continue to support, but it would be weird to
> eliminate them without also elminating bdist_wininst. The wininst format has
> the most usage out of all of the seldom used formats, however when we look at
> the downloads for the last 30 days only 0.42% of the downloads were for wininst
> files, so I think that it's pretty safe to remove them. I think in the past,
> these were more important due to the lack of real binary packages on Windows,
> but I think in 2016 we have wheel, and Wheel is a better solution. If however
> we want to keep them, then I think it's pretty safe to remove them from our
> /simple/ pages and any future repository pages and modify our mirroring tooling
> to stop mirroring them. IOW, to treat them as some sort of "additional upload"
> rather than release uploads to PyPI.
>
> Finally, bdist_egg is quite possibly the trickiest one to justify. A fair
> number of people still upload eggs, even though we have the wheel format.
> However, I think that we should (and generally do) consider eggs to be
> deprecated and while we don't want to break existing packages by removing them,
> we should block further uploads for them. Looking again at the download numbers
> eggs represented only 1.8% of total downloads in the last 30 days while wheels
> represented 41% and sdists represented 56%. Further more, I think we should do
> this with the expectation that any new repository API will no longer include
> egg files in them, and will just be sdists and wheels.
>
> Doing all of this would leave us with:
>
> * The /simple/ repository only having sdists, wheels, and eggs and disallowing
>   new eggs to be uploaded.
> * The hypothetical repository 2.0 api only having sdists and wheels.
> * *MAYBE* Having "related files" that people could upload things like
>   Windows/OSX Installers.
>
> On a related but different note, I'd also like to restrict the acceptable
> extensions for sdists. Currently distutils allows people to generate .tar,
> .tar.gz, .tgz, .tar.bz2, .tbz, .zip, .tar.xz, .tar.Z and possibly others. This
> is a bit problematic because each of those types requires a different set of
> optional libraries (which may or may not exist depending on Python version) so
> you can't really use a lot of them (for example, while .tar.xz might give you
> better compression, it's also not usable before Python 3).
>
> Looking at numbers again, the current number of projects for each file ext are:
>
> * .tar.gz: 444,338
> * .zip: 58,774
> * .tar.bz2: 3,265
> * .tgz: 217
> * Everything Else: 0
>
> These results are not particularly surprising since .tar.gz is the default
> format that distutils creates. What I would like to do is, similarly to above,
> simply stop allowing new uploads for anything but .tar.gz. This will have a few
> positive effects:
>
> * It will make it so that (going forward) there is only a single one of the
>   optional C libraries that Python can be compiled against that is required for
>   someone to install something from PyPI using pip. This would be the zlib
>   library which is available almost universally.
>
> * It will reduce confusion because people will not be able to upload two
>   different source releases for the same version (e.g. example-1.0.tar.gz and
>   example-1.0.zip).
>
> * It will making tooling around PyPI easier, because it'll only have to deal
>   with one file extension for sdists.
>
>
> Thoughts?

My only thought is how we convey this message to users. I wonder if it
would be beneficial to have Twine cut a release that warns users when
they are uploading something that will be unsupported, then have
Warehouse/PyPI start returning a 415 (Unsupported media type)
approximately a few weeks/month later.

I'm +1 for restricting the kinds of things people can upload though.

From thomas at kluyver.me.uk  Mon Aug 15 15:27:17 2016
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Mon, 15 Aug 2016 20:27:17 +0100
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
Message-ID: <1471289237.729994.696062937.74ECDBC7@webmail.messagingengine.com>

On Mon, Aug 15, 2016, at 08:09 PM, Donald Stufft wrote:
> Finally, bdist_egg is quite possibly the trickiest one to justify. A fair
> number of people still upload eggs, even though we have the wheel format.
> However, I think that we should (and generally do) consider eggs to be
> deprecated and while we don't want to break existing packages by removing
> them,
> we should block further uploads for them. Looking again at the download
> numbers
> eggs represented only 1.8% of total downloads in the last 30 days while
> wheels
> represented 41% and sdists represented 56%.

Is it possible to get the numbers for the proportion of uploads
represented by different formats?

I think this simplification is a good idea, but I expect there will be
some people frustrated when their egg uploads no longer work.

We still upload .zip sdists for some of our projects, in addition to
.tar.gz, but I think that's a habit left over from the days when it was
common to download and unpack sdists manually to run setup.py. I don't
think we lose anything if we drop the .zip downloads now.

From donald at stufft.io  Mon Aug 15 15:28:24 2016
From: donald at stufft.io (Donald Stufft)
Date: Mon, 15 Aug 2016 15:28:24 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAN-Kwu0FsKz7sJgiwZzRoQMzuxBjpBVpEsTXE-XH=B1=6gzOnw@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAN-Kwu0FsKz7sJgiwZzRoQMzuxBjpBVpEsTXE-XH=B1=6gzOnw@mail.gmail.com>
Message-ID: <C17D25ED-BD13-4FFA-BC7B-6D9C509927BA@stufft.io>


> On Aug 15, 2016, at 3:22 PM, Ian Cordasco <graffatcolmingov at gmail.com> wrote:
> 
> My only thought is how we convey this message to users. I wonder if it
> would be beneficial to have Twine cut a release that warns users when
> they are uploading something that will be unsupported, then have
> Warehouse/PyPI start returning a 415 (Unsupported media type)
> approximately a few weeks/month later.

I wouldn?t be opposed to something like this, though I?m not entirely sure
it?s going to be super useful. I?m not sure if a 415 is the correct response
code since these are being sent as binary blobs as far as HTTP is concerned,
but the Content-Type of the HTTP request will still be correct, but just the
data inside it will be wrong, so I think that?s solidly a 400 error? Not that
it?s super important, that?s an implementation detail :)

> 
> I'm +1 for restricting the kinds of things people can upload though.


?
Donald Stufft




From alex.gronholm at nextday.fi  Mon Aug 15 15:30:50 2016
From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=)
Date: Mon, 15 Aug 2016 22:30:50 +0300
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <C17D25ED-BD13-4FFA-BC7B-6D9C509927BA@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAN-Kwu0FsKz7sJgiwZzRoQMzuxBjpBVpEsTXE-XH=B1=6gzOnw@mail.gmail.com>
 <C17D25ED-BD13-4FFA-BC7B-6D9C509927BA@stufft.io>
Message-ID: <e0b555d8-c66a-304a-799b-7aec9ee75b2d@nextday.fi>

15.08.2016, 22:28, Donald Stufft kirjoitti:
>> On Aug 15, 2016, at 3:22 PM, Ian Cordasco <graffatcolmingov at gmail.com> wrote:
>>
>> My only thought is how we convey this message to users. I wonder if it
>> would be beneficial to have Twine cut a release that warns users when
>> they are uploading something that will be unsupported, then have
>> Warehouse/PyPI start returning a 415 (Unsupported media type)
>> approximately a few weeks/month later.
> I wouldn?t be opposed to something like this, though I?m not entirely sure
> it?s going to be super useful. I?m not sure if a 415 is the correct response
> code since these are being sent as binary blobs as far as HTTP is concerned,
> but the Content-Type of the HTTP request will still be correct, but just the
> data inside it will be wrong, so I think that?s solidly a 400 error? Not that
> it?s super important, that?s an implementation detail :)
I think it'll be more important to let the users know exactly *why* 
their uploads failed (and what to do about it) when the change finally 
takes place.
>
>> I'm +1 for restricting the kinds of things people can upload though.
>
> ?
> Donald Stufft
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


From graffatcolmingov at gmail.com  Mon Aug 15 15:33:11 2016
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Mon, 15 Aug 2016 14:33:11 -0500
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <e0b555d8-c66a-304a-799b-7aec9ee75b2d@nextday.fi>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAN-Kwu0FsKz7sJgiwZzRoQMzuxBjpBVpEsTXE-XH=B1=6gzOnw@mail.gmail.com>
 <C17D25ED-BD13-4FFA-BC7B-6D9C509927BA@stufft.io>
 <e0b555d8-c66a-304a-799b-7aec9ee75b2d@nextday.fi>
Message-ID: <CAN-Kwu0Ni2MuOV_5-M2mVw5W7wtUqTnDaKSwrUDq9RrmgRrF=g@mail.gmail.com>

On Mon, Aug 15, 2016 at 2:30 PM, Alex Gr?nholm <alex.gronholm at nextday.fi> wrote:
> 15.08.2016, 22:28, Donald Stufft kirjoitti:
>>>
>>> On Aug 15, 2016, at 3:22 PM, Ian Cordasco <graffatcolmingov at gmail.com>
>>> wrote:
>>>
>>> My only thought is how we convey this message to users. I wonder if it
>>> would be beneficial to have Twine cut a release that warns users when
>>> they are uploading something that will be unsupported, then have
>>> Warehouse/PyPI start returning a 415 (Unsupported media type)
>>> approximately a few weeks/month later.
>>
>> I wouldn?t be opposed to something like this, though I?m not entirely sure
>> it?s going to be super useful. I?m not sure if a 415 is the correct
>> response
>> code since these are being sent as binary blobs as far as HTTP is
>> concerned,
>> but the Content-Type of the HTTP request will still be correct, but just
>> the
>> data inside it will be wrong, so I think that?s solidly a 400 error? Not
>> that
>> it?s super important, that?s an implementation detail :)
>
> I think it'll be more important to let the users know exactly *why* their
> uploads failed (and what to do about it) when the change finally takes
> place.

Right. I'm not tied to 415 at all. I just want something firm that
people not using twine can understand. PyPI currently basically only
ever returns 400s and Warehouse has improved things. I'm just hoping
we find the right thing early on (and that may be a 400) but I just
want something obvious.

From donald at stufft.io  Mon Aug 15 15:40:07 2016
From: donald at stufft.io (Donald Stufft)
Date: Mon, 15 Aug 2016 15:40:07 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <1471289237.729994.696062937.74ECDBC7@webmail.messagingengine.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <1471289237.729994.696062937.74ECDBC7@webmail.messagingengine.com>
Message-ID: <42876D1B-D8DF-4E55-8A25-4B124DCB8107@stufft.io>


> On Aug 15, 2016, at 3:27 PM, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> 
> On Mon, Aug 15, 2016, at 08:09 PM, Donald Stufft wrote:
>> Finally, bdist_egg is quite possibly the trickiest one to justify. A fair
>> number of people still upload eggs, even though we have the wheel format.
>> However, I think that we should (and generally do) consider eggs to be
>> deprecated and while we don't want to break existing packages by removing
>> them,
>> we should block further uploads for them. Looking again at the download
>> numbers
>> eggs represented only 1.8% of total downloads in the last 30 days while
>> wheels
>> represented 41% and sdists represented 56%.
> 
> Is it possible to get the numbers for the proportion of uploads
> represented by different formats?

I?m not sure what you mean exactly, the numbers in my original email:

* sdist: 506,585
* bdist_wheel: 81,207
* bdist_egg: 48,282
* bdist_wininst: 14,002
* bdist_dumb: 5,502
* bdist_msi: 497
* bdist_rpm: 464
* bdist_dmg: 45

represent the total files uploaded for each file type. Do you just want
them converted to percentages? If so they?re:

* sdist: 77.1%
* bdist_wheel: 12.3%
* bdist_egg: 7.3%
* bdist_wininst: 2.1%
* bdist_dumb: 0.8%
* bdist_msi: 0.07%
* bdist_rpm: 0.07%
* bdist_dmg: 0.006%

Or were you wondering for the last 30 days like I did for downloads? If
so then:

* sdist: 15601 (66%)
* bdist_wheel: 6398 (27%)
* bdist_egg: 1076 (4.5%)
* bdist_dumb: 195 (0.8%)
* bdist_wininst: 167 (0.7%)
* bdist_rpm: 38 (0.1%)
* bdist_msi: 9 (0.03%)
* Everything Else: 0 (0%)

?
Donald Stufft




From glyph at twistedmatrix.com  Mon Aug 15 16:17:55 2016
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Mon, 15 Aug 2016 13:17:55 -0700
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
Message-ID: <A553FA54-D9F5-44E2-A077-5AE81E45A22B@twistedmatrix.com>


> On Aug 15, 2016, at 12:09 PM, Donald Stufft <donald at stufft.io> wrote:
> 
> Next we have bdist_dmg, bdist_msi, and bdist_winist. I'm lumping these together
> because they're all OS specific installers for OSs that don't already have some
> sort of repository. This lack of a repository format for them means that random
> downloads are already the norm for people using these systems. For these, I
> think the usage numbers for bdist_dmg and bdist_msi easily suggest that they
> are not very important to continue to support, but it would be weird to
> eliminate them without also elminating bdist_wininst. The wininst format has
> the most usage out of all of the seldom used formats, however when we look at
> the downloads for the last 30 days only 0.42% of the downloads were for wininst
> files, so I think that it's pretty safe to remove them. I think in the past,
> these were more important due to the lack of real binary packages on Windows,
> but I think in 2016 we have wheel, and Wheel is a better solution. If however
> we want to keep them, then I think it's pretty safe to remove them from our
> /simple/ pages and any future repository pages and modify our mirroring tooling
> to stop mirroring them. IOW, to treat them as some sort of "additional upload"
> rather than release uploads to PyPI.

I think you have a better handle on this than I do, but I did just want to provide a little input.  I think we should be cautious in the way these are disabled, because it's already hard enough to produce user-facing software with Python. We don't want to throw up yet another roadblock to creating a layperson-friendly download.  If they're not used now, it's not necessarily an indication of whether we _want_ them to be used in the future.

Also, since these formats aren't readily 'pip install'-able, and are really only suitable for applications anyway, perhaps the download numbers are skewed?  Automated systems doing 1000s of builds per day are likely to massively inflate download counts even if they're used by a far smaller number of users.

Anyway, like I said: not an expert here, just wanted to make sure the "python for desktop software (even if it's not used much right now)" angle is considered as well.

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160815/a5cb6335/attachment.html>

From barry at python.org  Mon Aug 15 16:20:58 2016
From: barry at python.org (Barry Warsaw)
Date: Mon, 15 Aug 2016 16:20:58 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
Message-ID: <20160815162058.601acb68@anarchist.wooz.org>

On Aug 15, 2016, at 03:09 PM, Donald Stufft wrote:

>Out of all of these, I think that we can easily remove bdist_dmg, bdist_rpm,
>and bdist_dumb. I also believe that there is a strong case for removing
>bdist_msi and bdist_wininst. I also think we should consider removing
>bdist_egg.

+1 for all of these, or maybe with +0 for bdist_egg in the short term.  My
only concern is that the deprecation period has been long enough, but I do
agree that they should eventually be prohibited.

>On a related but different note, I'd also like to restrict the acceptable
>extensions for sdists. Currently distutils allows people to generate .tar,
>.tar.gz, .tgz, .tar.bz2, .tbz, .zip, .tar.xz, .tar.Z and possibly
>others. This is a bit problematic because each of those types requires a
>different set of optional libraries (which may or may not exist depending on
>Python version) so you can't really use a lot of them (for example, while
>.tar.xz might give you better compression, it's also not usable before Python
>3).
>
>Looking at numbers again, the current number of projects for each file ext
>are:
>
>* .tar.gz: 444,338
>* .zip: 58,774
>* .tar.bz2: 3,265
>* .tgz: 217
>* Everything Else: 0

I'm a little less certain about this.  It's probably okay, although since the
.tgz format is the same as .tar.gz, the tools argument doesn't hold for that
extension.  I know compression nerds like to debate the merits of gz, bz2, and
xz, but I doubt with the size of most of these files, it's going to make any
difference.

FWIW, restricting the extensions would simplify Debian's template for the
PyPI debian/watch file (e.g. the pattern we use to download packages):

https://pypi.debian.net/nose2/nose2-(.+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))

(Ignore the pypi.debian.net redirector.)

That would be a mild win.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160815/fa4e20cb/attachment.sig>

From donald at stufft.io  Mon Aug 15 16:56:37 2016
From: donald at stufft.io (Donald Stufft)
Date: Mon, 15 Aug 2016 16:56:37 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <A553FA54-D9F5-44E2-A077-5AE81E45A22B@twistedmatrix.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <A553FA54-D9F5-44E2-A077-5AE81E45A22B@twistedmatrix.com>
Message-ID: <A87B1D05-77C2-4695-8B06-0BA7E03FE4D7@stufft.io>

My main thought regarding this is that bdist_dmg != all dmg files (similarly for msi and wininst). These are specific files created by distutils without a standard or without the needed work to make them truly what users should be using. I also think they are a different class of upload, the general use case for PyPI's current file uploads are for automated installs (as evidenced by the simple API and mirroring).

If we want to enable dmg, msi, etc uploads that are not the bdist_* variety for automated tooling, then we could do something like "related files" people can upload which don't get mirrored for pip and which don't show up in the repo API. Since they will be classified differently we could also do better work around the ux of discovering them and separate them from the 50 wheels that some projects end up uploading and make them more obviously visible. I don't know if pypi as a distribution for _end user_ (vs developer/power user) software makes sense or not, but if it does we should support it better than accidentally via distutils. 

Sent from my iPhone

> On Aug 15, 2016, at 4:17 PM, Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:
> 
> Anyway, like I said: not an expert here, just wanted to make sure the "python for desktop software (even if it's not used much right now)" angle is considered as well.


From donald at stufft.io  Mon Aug 15 17:39:10 2016
From: donald at stufft.io (Donald Stufft)
Date: Mon, 15 Aug 2016 17:39:10 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <20160815162058.601acb68@anarchist.wooz.org>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <20160815162058.601acb68@anarchist.wooz.org>
Message-ID: <80205719-72B4-4E53-8602-CFB3FBDAA00E@stufft.io>


> On Aug 15, 2016, at 4:20 PM, Barry Warsaw <barry at python.org> wrote:
> 
> On Aug 15, 2016, at 03:09 PM, Donald Stufft wrote:
> 
>> Out of all of these, I think that we can easily remove bdist_dmg, bdist_rpm,
>> and bdist_dumb. I also believe that there is a strong case for removing
>> bdist_msi and bdist_wininst. I also think we should consider removing
>> bdist_egg.
> 
> +1 for all of these, or maybe with +0 for bdist_egg in the short term.  My
> only concern is that the deprecation period has been long enough, but I do
> agree that they should eventually be prohibited.


Yea, egg is sort of the stickiest thing here, but I also think it?s a good
idea, at least in the long term. We can play around with timescales as it
suits us of course.

>> On a related but different note, I'd also like to restrict the acceptable
>> extensions for sdists. Currently distutils allows people to generate .tar,
>> .tar.gz, .tgz, .tar.bz2, .tbz, .zip, .tar.xz, .tar.Z and possibly
>> others. This is a bit problematic because each of those types requires a
>> different set of optional libraries (which may or may not exist depending on
>> Python version) so you can't really use a lot of them (for example, while
>> .tar.xz might give you better compression, it's also not usable before Python
>> 3).
>> 
>> Looking at numbers again, the current number of projects for each file ext
>> are:
>> 
>> * .tar.gz: 444,338
>> * .zip: 58,774
>> * .tar.bz2: 3,265
>> * .tgz: 217
>> * Everything Else: 0
> 
> I'm a little less certain about this.  It's probably okay, although since the
> .tgz format is the same as .tar.gz, the tools argument doesn't hold for that
> extension.  I know compression nerds like to debate the merits of gz, bz2, and
> xz, but I doubt with the size of most of these files, it's going to make any
> difference.

Right, it?s a few reasons rolled up together:

* Single C library makes it easier to explain to people what they need capability
  wise to install a library (and the fact zlib is near universal doesn?t hurt).

* One extension (versus many) makes expectations easier, like your debian/watch
  file, but also things like ensuring a project has only *one* sdist per version
  instead of allowing 12.

* Variances breed bugs, particularly when you have disparate systems all working
  on this data. For example, your nose2 debian/watch file has a bug, it doesn?t
  support nose2-1.0.tar which is currently a supported by distutils/PyPI/pip. Of
  course the more variances you have the more likely you are to get bugs where
  different systems support different extensions, so narrowing it down to only
  .tar.gz and .tgz is pretty safe, but practically nobody uses .tgz anyways so
  why bother?

> 
> FWIW, restricting the extensions would simplify Debian's template for the
> PyPI debian/watch file (e.g. the pattern we use to download packages):
> 
> https://pypi.debian.net/nose2/nose2-(.+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz)))
> 
> (Ignore the pypi.debian.net redirector.)
> 
> That would be a mild win.

Yea :D This is exactly the sort of thing that I think this makes better. There is
a lot of tooling built up around PyPI and the less flexible PyPI is the easier
those tools will have working with PyPI in terms of data. Of course, it is a
balancing act, but I think this is a pretty safe place to skew towards mandating
certain things.

?
Donald Stufft




From barry at python.org  Mon Aug 15 17:45:45 2016
From: barry at python.org (Barry Warsaw)
Date: Mon, 15 Aug 2016 17:45:45 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <80205719-72B4-4E53-8602-CFB3FBDAA00E@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <20160815162058.601acb68@anarchist.wooz.org>
 <80205719-72B4-4E53-8602-CFB3FBDAA00E@stufft.io>
Message-ID: <20160815174545.4182deb8@subdivisions.wooz.org>

On Aug 15, 2016, at 05:39 PM, Donald Stufft wrote:

>so narrowing it down to only .tar.gz and .tgz is pretty safe, but practically
>nobody uses .tgz anyways so why bother?

I agree with all of your reasoning.  FWIW, I personally use .tgz all the time
when making backups of things locally, but rarely use it for publicly
consumable artifacts.  And since the sdists setupdistutilstools generates will
only be .tar.gz, I think it's fine to disallow .tgz.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160815/26d30167/attachment.sig>

From glyph at twistedmatrix.com  Mon Aug 15 19:12:26 2016
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Mon, 15 Aug 2016 16:12:26 -0700
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <A87B1D05-77C2-4695-8B06-0BA7E03FE4D7@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <A553FA54-D9F5-44E2-A077-5AE81E45A22B@twistedmatrix.com>
 <A87B1D05-77C2-4695-8B06-0BA7E03FE4D7@stufft.io>
Message-ID: <D7DC1980-706A-47C0-864C-47D2E85E4E63@twistedmatrix.com>


> On Aug 15, 2016, at 1:56 PM, Donald Stufft <donald at stufft.io> wrote:
> 
> My main thought regarding this is that bdist_dmg != all dmg files (similarly for msi and wininst). These are specific files created by distutils without a standard or without the needed work to make them truly what users should be using. I also think they are a different class of upload, the general use case for PyPI's current file uploads are for automated installs (as evidenced by the simple API and mirroring).

I guess I'm just a little confused - are we talking about just hiding them from some parts of the API or disallowing their upload entirely?

If we're talking about the literal output of bdist_dmg and bdist_rpm I probably agree that they're almost useless.

> If we want to enable dmg, msi, etc uploads that are not the bdist_* variety for automated tooling, then we could do something like "related files" people can upload which don't get mirrored for pip and which don't show up in the repo API. Since they will be classified differently we could also do better work around the ux of discovering them and separate them from the 50 wheels that some projects end up uploading and make them more obviously visible. I don't know if pypi as a distribution for _end user_ (vs developer/power user) software makes sense or not, but if it does we should support it better than accidentally via distutils. 

My concern here is that if someone has a hacky workaround working with the current system, it might be better to add support for the new thing ("related files") before killing the old thing.  If the plan is to do them both anyway, wouldn't it be better to do it in that order?  As a community (and I mean the broader open source community here, not distutils-sig; if anything distutils is way better about this) we have an unfortunate habit of killing potentially-useful-but-sub-optimal stuff, wandering off for half a decade, and then only adding the better thing after the fact.

-glyph

From donald at stufft.io  Mon Aug 15 19:30:30 2016
From: donald at stufft.io (Donald Stufft)
Date: Mon, 15 Aug 2016 19:30:30 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <D7DC1980-706A-47C0-864C-47D2E85E4E63@twistedmatrix.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <A553FA54-D9F5-44E2-A077-5AE81E45A22B@twistedmatrix.com>
 <A87B1D05-77C2-4695-8B06-0BA7E03FE4D7@stufft.io>
 <D7DC1980-706A-47C0-864C-47D2E85E4E63@twistedmatrix.com>
Message-ID: <7CBDC9E5-FEF5-4C54-9393-E2ACCC5EC60E@stufft.io>


> On Aug 15, 2016, at 7:12 PM, Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:
> 
> 
>> On Aug 15, 2016, at 1:56 PM, Donald Stufft <donald at stufft.io> wrote:
>> 
>> My main thought regarding this is that bdist_dmg != all dmg files (similarly for msi and wininst). These are specific files created by distutils without a standard or without the needed work to make them truly what users should be using. I also think they are a different class of upload, the general use case for PyPI's current file uploads are for automated installs (as evidenced by the simple API and mirroring).
> 
> I guess I'm just a little confused - are we talking about just hiding them from some parts of the API or disallowing their upload entirely?
> 
> If we're talking about the literal output of bdist_dmg and bdist_rpm I probably agree that they're almost useless.

Right now we?re basically talking about the literal output of bdist_dmg and bdist_rpm, because their outputs are the only thing we actually support uploading. Our current checks aren?t very stringent (or in some cases, exist at all) so it?s *possible* someone is constructing an actual user friendly .dmg and uploading it pretending to be a ?bdist_dmg?, that?s not really a supported operation.

Ideally I?d like to disallow their upload completely and hide them from the API, but if the current use case of bdist_dmg, bdist_rpm, bdist_wininst, etc is useful enough to keep around, then I?d like to just hide them from the API.

> 
>> If we want to enable dmg, msi, etc uploads that are not the bdist_* variety for automated tooling, then we could do something like "related files" people can upload which don't get mirrored for pip and which don't show up in the repo API. Since they will be classified differently we could also do better work around the ux of discovering them and separate them from the 50 wheels that some projects end up uploading and make them more obviously visible. I don't know if pypi as a distribution for _end user_ (vs developer/power user) software makes sense or not, but if it does we should support it better than accidentally via distutils. 
> 
> My concern here is that if someone has a hacky workaround working with the current system, it might be better to add support for the new thing ("related files") before killing the old thing.  If the plan is to do them both anyway, wouldn't it be better to do it in that order?  As a community (and I mean the broader open source community here, not distutils-sig; if anything distutils is way better about this) we have an unfortunate habit of killing potentially-useful-but-sub-optimal stuff, wandering off for half a decade, and then only adding the better thing after the fact.

I don?t know if it makes sense to add the hypothetical new thing. Is PyPI the right place to distribute a .dmg that say, ships Python itself, some C libraries, some Python libraries, some bash scripts, and some static files? I don?t know. Currently I think the answer is ?no?, PyPI is a repository of software _for_ Python, not a repository of software that just happens to be written in Python. If the ultimate end goal of a .dmg is that people no longer need to worry about what language the thing they are installing is written in, it seems weird for them to go to PyPI for a .dmg for one app, npmjs.org for a .dmg for another app, and so on.

Now, if it *does* make sense to support generalized uploads for applications written in Python with some sort of system level packaging (dmg, deb, conda, whatever) then we should figure out a way to actually support that, and support it well. As it is, I don?t have any evidence that the files that are currently being uploaded to PyPI are anything but ?wheels, but in dmg format? (e.g. binary packages containing a single library). I don?t want to put a bunch of effort and work into making this well supported if there isn?t some evidence that folks will actually use it (I suspect a minimum we?d need some buy in from the authors of tooling to make these self contained packages).


?
Donald Stufft




From glyph at twistedmatrix.com  Mon Aug 15 19:35:40 2016
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Mon, 15 Aug 2016 16:35:40 -0700
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <7CBDC9E5-FEF5-4C54-9393-E2ACCC5EC60E@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <A553FA54-D9F5-44E2-A077-5AE81E45A22B@twistedmatrix.com>
 <A87B1D05-77C2-4695-8B06-0BA7E03FE4D7@stufft.io>
 <D7DC1980-706A-47C0-864C-47D2E85E4E63@twistedmatrix.com>
 <7CBDC9E5-FEF5-4C54-9393-E2ACCC5EC60E@stufft.io>
Message-ID: <728A8C9F-2BC3-4508-B9FF-BC6FF482BD99@twistedmatrix.com>


> On Aug 15, 2016, at 4:30 PM, Donald Stufft <donald at stufft.io> wrote:
> 
>> 
>> On Aug 15, 2016, at 7:12 PM, Glyph Lefkowitz <glyph at twistedmatrix.com <mailto:glyph at twistedmatrix.com>> wrote:
>> 
>> 
>>> On Aug 15, 2016, at 1:56 PM, Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
>>> 
>>> My main thought regarding this is that bdist_dmg != all dmg files (similarly for msi and wininst). These are specific files created by distutils without a standard or without the needed work to make them truly what users should be using. I also think they are a different class of upload, the general use case for PyPI's current file uploads are for automated installs (as evidenced by the simple API and mirroring).
>> 
>> I guess I'm just a little confused - are we talking about just hiding them from some parts of the API or disallowing their upload entirely?
>> 
>> If we're talking about the literal output of bdist_dmg and bdist_rpm I probably agree that they're almost useless.
> 
> Right now we?re basically talking about the literal output of bdist_dmg and bdist_rpm, because their outputs are the only thing we actually support uploading. Our current checks aren?t very stringent (or in some cases, exist at all) so it?s *possible* someone is constructing an actual user friendly .dmg and uploading it pretending to be a ?bdist_dmg?, that?s not really a supported operation.
> 
> Ideally I?d like to disallow their upload completely and hide them from the API, but if the current use case of bdist_dmg, bdist_rpm, bdist_wininst, etc is useful enough to keep around, then I?d like to just hide them from the API.
> 
>> 
>>> If we want to enable dmg, msi, etc uploads that are not the bdist_* variety for automated tooling, then we could do something like "related files" people can upload which don't get mirrored for pip and which don't show up in the repo API. Since they will be classified differently we could also do better work around the ux of discovering them and separate them from the 50 wheels that some projects end up uploading and make them more obviously visible. I don't know if pypi as a distribution for _end user_ (vs developer/power user) software makes sense or not, but if it does we should support it better than accidentally via distutils. 
>> 
>> My concern here is that if someone has a hacky workaround working with the current system, it might be better to add support for the new thing ("related files") before killing the old thing.  If the plan is to do them both anyway, wouldn't it be better to do it in that order?  As a community (and I mean the broader open source community here, not distutils-sig; if anything distutils is way better about this) we have an unfortunate habit of killing potentially-useful-but-sub-optimal stuff, wandering off for half a decade, and then only adding the better thing after the fact.
> 
> I don?t know if it makes sense to add the hypothetical new thing. Is PyPI the right place to distribute a .dmg that say, ships Python itself, some C libraries, some Python libraries, some bash scripts, and some static files? I don?t know. Currently I think the answer is ?no?, PyPI is a repository of software _for_ Python, not a repository of software that just happens to be written in Python. If the ultimate end goal of a .dmg is that people no longer need to worry about what language the thing they are installing is written in, it seems weird for them to go to PyPI for a .dmg for one app, npmjs.org <http://npmjs.org/> for a .dmg for another app, and so on.
> 
> Now, if it *does* make sense to support generalized uploads for applications written in Python with some sort of system level packaging (dmg, deb, conda, whatever) then we should figure out a way to actually support that, and support it well. As it is, I don?t have any evidence that the files that are currently being uploaded to PyPI are anything but ?wheels, but in dmg format? (e.g. binary packages containing a single library). I don?t want to put a bunch of effort and work into making this well supported if there isn?t some evidence that folks will actually use it (I suspect a minimum we?d need some buy in from the authors of tooling to make these self contained packages).

In the absence of any _actual_ person doing the thing that I suggested might be happening, where a bdist_dmg is actually a for-real dmg, then it's probably not worth doing.  I just thought that this might be a reason to give the data a second look.

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160815/a917dcc1/attachment.html>

From ncoghlan at gmail.com  Mon Aug 15 20:02:57 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 16 Aug 2016 10:02:57 +1000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
Message-ID: <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>

On 16 Aug 2016 05:09, "Donald Stufft" <donald at stufft.io> wrote:
>
> Hello!
>
> I'd like to restrict what folks can upload to PyPI in an effort to help
narrow
> the scope down and to enable more a more consistent experience for
everyone.
>
> First off, we currently allow people to upload sdist, bdist_wheel,
bdist_egg,
> bdist_dmg, bdist_dumb, bdist_msi, bdist_rpm, and bdist_wininst. However I
think
> that we should try to get rid of support for most of these. Just for
reference
> currently the number of files uploaded for each type of file looks like:
>
> * sdist: 506,585
> * bdist_wheel: 81,207
> * bdist_egg: 48,282
> * bdist_wininst: 14,002
> * bdist_dumb: 5,502
> * bdist_msi: 497
> * bdist_rpm: 464
> * bdist_dmg: 45
>
> Out of all of these, I think that we can easily remove bdist_dmg,
bdist_rpm,
> and bdist_dumb. I also believe that there is a strong case for removing
> bdist_msi and bdist_wininst. I also think we should consider removing
> bdist_egg.
>
> First of all, when I say "remove", I mean disallow new uploads, but do not
> delete the existing files.

General +1 from me for data driven design simplification.

As far as ideas for *how* to go about it go, I think there a few user
categories to consider:

* new user making their first project: we can deprecate legacy formats
aggressively here (since there shouldn't be many, if any, backwards
compatibility considerations for either automated workflows or people's
habits)

* new release of existing project: here is where we want to actively warn
project maintainers using the old formats that PyPI's behaviour will be
changing *before* we force them to change their workflows and habits

*  new project by existing maintainer: here, I'd be inclined to flag
maintainer accounts at the start of the deprecation period based on whether
or not they're currently using the legacy formats on any of their projects
- that is, the migration period would be applied at the *user* level, in
addition to the project level. If someone has never used the legacy
formats, they can be treated like a new user immediately, and will
presumably never even notice the change.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160816/e3c38ede/attachment-0001.html>

From p.f.moore at gmail.com  Tue Aug 16 05:41:06 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 16 Aug 2016 10:41:06 +0100
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
Message-ID: <CACac1F8LSnH_7=vORDxWdcqMs2=hZ9Vi=MrQ7uF4m6tWy3Fyiw@mail.gmail.com>

On 15 August 2016 at 20:09, Donald Stufft <donald at stufft.io> wrote:
>
> First off, we currently allow people to upload sdist, bdist_wheel, bdist_egg,
> bdist_dmg, bdist_dumb, bdist_msi, bdist_rpm, and bdist_wininst. However I think
> that we should try to get rid of support for most of these. Just for reference
> currently the number of files uploaded for each type of file looks like:
>
> * sdist: 506,585
> * bdist_wheel: 81,207
> * bdist_egg: 48,282
> * bdist_wininst: 14,002
> * bdist_dumb: 5,502
> * bdist_msi: 497
> * bdist_rpm: 464
> * bdist_dmg: 45
>
> Out of all of these, I think that we can easily remove bdist_dmg, bdist_rpm,
> and bdist_dumb. I also believe that there is a strong case for removing
> bdist_msi and bdist_wininst. I also think we should consider removing
> bdist_egg.

Overall +1 from me for this.

Some thoughts:

1. It would be nice to provide some transition process for the
bdist_wininst case, as these are automatically convertible (with some
caveats) to wheel using "wheel convert". I'm not suggesting that we do
an auto-convert, but a way of getting the message out to projects that
use wininst that there is a migration path might be worth it (although
it's equally possible that the projects in question are all
unmaintained, in which case there's not much point in bothering).
2. Do we understand the remaining use cases for eggs? AIUI, buildout
uses it, and I get the impression that there are other specialist
groups that still use the egg format (the plone community?). While I
don't think having two binary distribution formats is good for the
ecosystem (it's confusing for users, and splits effort), I think we
should be make sure we have those use cases covered (or at least have
a plan on how we handle the situation) before deprecating the egg
format.

Paul

From dholth at gmail.com  Tue Aug 16 08:50:07 2016
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 16 Aug 2016 12:50:07 +0000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CACac1F8LSnH_7=vORDxWdcqMs2=hZ9Vi=MrQ7uF4m6tWy3Fyiw@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CACac1F8LSnH_7=vORDxWdcqMs2=hZ9Vi=MrQ7uF4m6tWy3Fyiw@mail.gmail.com>
Message-ID: <CAG8k2+61eY685hZPxoK6H34gn1oxz1sOFC1aom_0fEDbmjhjWw@mail.gmail.com>

Wheel should be updated to support the egg use case before egg is removed.
IIUC this would mostly mean officially supporting 'unzipped wheel' as a
thing you can add to PYTHONPATH, possibly with some additional restrictions
for the specific wheel. We could go a little further and officially support
zipped wheels "if zip safe". We could implement wheel2egg to complement
egg2wheel?

The main reason wheel did not support these use cases from the beginning is
that it was quicker not to. Also, it is a bit confusing for 'unzipped
directory in the (egg/wheel) format', 'archive', and 'plugin you add to
PYTHONPATH' to all have the same name 'eggs'.

We would need to work with the buildout developers on this. Buildout uses a
flat directory full of unzipped eggs, not exactly the uploading to pypi
case, and when installing console_scripts, makes sure the necessary eggs
are added to sys.path in the script itself. The exact mechanism varies
depending on the version of buildout used.

zope.interface is an important package with up-to-date eggs.
https://pypi.python.org/pypi/zope.interface

On Tue, Aug 16, 2016 at 5:41 AM Paul Moore <p.f.moore at gmail.com> wrote:

> On 15 August 2016 at 20:09, Donald Stufft <donald at stufft.io> wrote:
> >
> > First off, we currently allow people to upload sdist, bdist_wheel,
> bdist_egg,
> > bdist_dmg, bdist_dumb, bdist_msi, bdist_rpm, and bdist_wininst. However
> I think
> > that we should try to get rid of support for most of these. Just for
> reference
> > currently the number of files uploaded for each type of file looks like:
> >
> > * sdist: 506,585
> > * bdist_wheel: 81,207
> > * bdist_egg: 48,282
> > * bdist_wininst: 14,002
> > * bdist_dumb: 5,502
> > * bdist_msi: 497
> > * bdist_rpm: 464
> > * bdist_dmg: 45
> >
> > Out of all of these, I think that we can easily remove bdist_dmg,
> bdist_rpm,
> > and bdist_dumb. I also believe that there is a strong case for removing
> > bdist_msi and bdist_wininst. I also think we should consider removing
> > bdist_egg.
>
> Overall +1 from me for this.
>
> Some thoughts:
>
> 1. It would be nice to provide some transition process for the
> bdist_wininst case, as these are automatically convertible (with some
> caveats) to wheel using "wheel convert". I'm not suggesting that we do
> an auto-convert, but a way of getting the message out to projects that
> use wininst that there is a migration path might be worth it (although
> it's equally possible that the projects in question are all
> unmaintained, in which case there's not much point in bothering).
> 2. Do we understand the remaining use cases for eggs? AIUI, buildout
> uses it, and I get the impression that there are other specialist
> groups that still use the egg format (the plone community?). While I
> don't think having two binary distribution formats is good for the
> ecosystem (it's confusing for users, and splits effort), I think we
> should be make sure we have those use cases covered (or at least have
> a plan on how we handle the situation) before deprecating the egg
> format.
>
> Paul
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160816/57aa5a14/attachment.html>

From leorochael at gmail.com  Tue Aug 16 11:15:52 2016
From: leorochael at gmail.com (Leonardo Rochael Almeida)
Date: Tue, 16 Aug 2016 12:15:52 -0300
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAG8k2+61eY685hZPxoK6H34gn1oxz1sOFC1aom_0fEDbmjhjWw@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CACac1F8LSnH_7=vORDxWdcqMs2=hZ9Vi=MrQ7uF4m6tWy3Fyiw@mail.gmail.com>
 <CAG8k2+61eY685hZPxoK6H34gn1oxz1sOFC1aom_0fEDbmjhjWw@mail.gmail.com>
Message-ID: <CAMxX_oOEKxNy45LgTwuQogSDoLrTLD_VUY2QHH0A8orO15Phig@mail.gmail.com>

Thanks Daniel,

A few corrections and considerations below:

On 16 August 2016 at 09:50, Daniel Holth <dholth at gmail.com> wrote:

> Wheel should be updated to support the egg use case before egg is removed.
> IIUC this would mostly mean officially supporting 'unzipped wheel' as a
> thing you can add to PYTHONPATH, possibly with some additional restrictions
> for the specific wheel. We could go a little further and officially support
> zipped wheels "if zip safe". We could implement wheel2egg to complement
> egg2wheel?
>

Specifically, buildout right now has setuptools as its only dependence, and
buildout uses it to locate, download and build sdist dependencies, or
directly download and use .egg dependencies, which are important for
Windows platforms in the communities that use buildout and depend on
compiled extensions.

It also uses setuptools for reading entry points of packages for internal
use (buildout recipes and extensions), and for figuring out console scripts
to install in "bin"

(this last part is actually the job of the `zc.recipe.egg` recipe, which is
buildout's pip replacement that predates pip by a few years, but it's the
only recipe that is developed in the same repo as buildout itself, so we'll
treat them as one for this discussion).

This also means that so far buildout has been missing out on wheels. It
hasn't been much of a problem since almost all packages also have an sdist
that can be turned into an egg by setuptools, but if the trend of
sdist-less wheels increases, this could quickly become a problem.

It also means that, buildout is missing out on manylinux...

A wheel2egg might be nice, but unless it's integrated into setuptools
proper, it would mean adding another dependency to buildout and the
developers would rather depend on a single library for talking to PyPI and
making packages appear as "directories addable to `sys.path`.

The main reason wheel did not support these use cases from the beginning is
> that it was quicker not to. Also, it is a bit confusing for 'unzipped
> directory in the (egg/wheel) format', 'archive', and 'plugin you add to
> PYTHONPATH' to all have the same name 'eggs'.
>
> We would need to work with the buildout developers on this. Buildout uses
> a flat directory full of unzipped eggs
>

Not necessarily unzipped. Although buildout unzips all sdists and eggs it
installs, if it's in the "eggs" directory, has the right name (e.g.
suds_jurko-0.6-py2.7.egg), and can be added to sys.path, that's all
buildout cares about, but if it's not yet there, buildout will use
setuptools to install it as an "unzipped egg" (there used to be a switch to
install eggs as zipped, or rather, as unzipped, zipped being the default,
but nobody liked the eggs zipped so they threw that foot gun away).


> , not exactly the uploading to pypi case, and when installing
> console_scripts, makes sure the necessary eggs are added to sys.path in the
> script itself. The exact mechanism varies depending on the version of
> buildout used.
>

This hasn't changed for a while. The mechanism is, basically, to create a
script in "bin/" that appends all (recursive) dependencies of the script
from the "eggs" directory into `sys.path` then call the entry point
declared in the `console_scripts`.

The buildout configuration can specify both the name that the console
scripts will be installed, and also specify some code to be run immediately
before the entry point,  which is useful for pre-setting command line
arguments, for example. All of this is done using setuptools APIs.

This and the fact that buildout has a plethora of extensions and powerful
declarative configuration language that allows for dependency management
between different "parts" of the configuration (e.g. coordinating ports and
addresses between services like Varnish and Plone, or adding the paths of
the script and configuation file from Plone into a supervisord
configuration) means that, like many things coming out of the brilliant
mind of Jim Fulton, it's considered overly complex by many, but those who
use it would find it rather difficult to replace if it's no longer
functional.

zope.interface is an important package with up-to-date eggs.
> https://pypi.python.org/pypi/zope.interface
>

And it's used by Plone and Pyramid, to mention two mainstream communities
that would be affected by a lack of binary eggs on Windows, for example.

To sumarize, if we could convince setuptools to treat wheels on PyPI for
download and installation the same way it treats eggs, or if we could
replace setuptools with something that did, then buildout would be onboard
the deprecation of eggs.

Cheers,

PS: In the buildout community, we never really understood the impetus for
replacing egg as a format, which is really not all that complex and not all
the different from wheel as it stands, instead of just formalizing it in a
PEP. We got the feeling that, in the movement of getting rid of setuptools
as an "installation" dependency, we ended up throwing out the baby with the
bathwater...

On Tue, Aug 16, 2016 at 5:41 AM Paul Moore <p.f.moore at gmail.com> wrote:
>
>> On 15 August 2016 at 20:09, Donald Stufft <donald at stufft.io> wrote:
>> >
>> > First off, we currently allow people to upload sdist, bdist_wheel,
>> bdist_egg,
>> > bdist_dmg, bdist_dumb, bdist_msi, bdist_rpm, and bdist_wininst. However
>> I think
>> > that we should try to get rid of support for most of these. Just for
>> reference
>> > currently the number of files uploaded for each type of file looks like:
>> >
>> > * sdist: 506,585
>> > * bdist_wheel: 81,207
>> > * bdist_egg: 48,282
>> > * bdist_wininst: 14,002
>> > * bdist_dumb: 5,502
>> > * bdist_msi: 497
>> > * bdist_rpm: 464
>> > * bdist_dmg: 45
>> >
>> > Out of all of these, I think that we can easily remove bdist_dmg,
>> bdist_rpm,
>> > and bdist_dumb. I also believe that there is a strong case for removing
>> > bdist_msi and bdist_wininst. I also think we should consider removing
>> > bdist_egg.
>>
>> Overall +1 from me for this.
>>
>> Some thoughts:
>>
>> 1. It would be nice to provide some transition process for the
>> bdist_wininst case, as these are automatically convertible (with some
>> caveats) to wheel using "wheel convert". I'm not suggesting that we do
>> an auto-convert, but a way of getting the message out to projects that
>> use wininst that there is a migration path might be worth it (although
>> it's equally possible that the projects in question are all
>> unmaintained, in which case there's not much point in bothering).
>> 2. Do we understand the remaining use cases for eggs? AIUI, buildout
>> uses it, and I get the impression that there are other specialist
>> groups that still use the egg format (the plone community?). While I
>> don't think having two binary distribution formats is good for the
>> ecosystem (it's confusing for users, and splits effort), I think we
>> should be make sure we have those use cases covered (or at least have
>> a plan on how we handle the situation) before deprecating the egg
>> format.
>>
>> Paul
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160816/acfdb66d/attachment-0001.html>

From brett at python.org  Tue Aug 16 11:51:39 2016
From: brett at python.org (Brett Cannon)
Date: Tue, 16 Aug 2016 15:51:39 +0000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
Message-ID: <CAP1=2W4p75=7STAs+EAo+dFY5sQvp8Gei_dnqVjW55QO6G6v+A@mail.gmail.com>

+1 on the whole idea. Trying to continue to nudge the community towards
more standardized approaches in packaging is always a good thing. :) I only
have one data point in relation to sdists file formats.

On Mon, 15 Aug 2016 at 12:09 Donald Stufft <donald at stufft.io> wrote:

> [SNIP]
>
> Looking at numbers again, the current number of projects for each file ext
> are:
>
> * .tar.gz: 444,338
> * .zip: 58,774
> * .tar.bz2: 3,265
> * .tgz: 217
> * Everything Else: 0
>

One thing to remember is that Windows can't read tar files natively while
it can for zip files. Now you can easily download tools on Windows to read
tar files and thanks to Bash on Windows you even have it included once you
turn that feature on.

The other point is we have a zip importer in Python but not a .tar.gz one. I
don't know how often anyone actually downloads a zip file directly from
PyPI and then tack it on to their sys.path for importing, but that is
currently possible.

I doubt either of these points are important enough to continue to support
zip files for sdists, but I just wanted to point it out. At worst this is
something to think about if we ever do formalize the sdist format and come
up with a custom file extension.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160816/8229f81f/attachment.html>

From donald at stufft.io  Tue Aug 16 12:06:49 2016
From: donald at stufft.io (Donald Stufft)
Date: Tue, 16 Aug 2016 12:06:49 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAG8k2+61eY685hZPxoK6H34gn1oxz1sOFC1aom_0fEDbmjhjWw@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CACac1F8LSnH_7=vORDxWdcqMs2=hZ9Vi=MrQ7uF4m6tWy3Fyiw@mail.gmail.com>
 <CAG8k2+61eY685hZPxoK6H34gn1oxz1sOFC1aom_0fEDbmjhjWw@mail.gmail.com>
Message-ID: <70914ED5-7A06-4A27-AF14-33C37C6C7BE7@stufft.io>


> On Aug 16, 2016, at 8:50 AM, Daniel Holth <dholth at gmail.com> wrote:
> 
> Wheel should be updated to support the egg use case before egg is removed. IIUC this would mostly mean officially supporting 'unzipped wheel' as a thing you can add to PYTHONPATH, possibly with some additional restrictions for the specific wheel. We could go a little further and officially support zipped wheels "if zip safe". We could implement wheel2egg to complement egg2wheel?


I don?t think Wheel should officially supported unzip wheels as a thing you can add to PYTHONPATH nor do I think we should officially support zipped wheels being added to PYTHONPATH. Neither of those things are going to work universally and setuptools has gross heuristics to try and figure out when they will and won?t work (which regularly break or report inaccurately). Wheel is improved by remaining focused on being a format for distributing and installed via an installer, not one that tries to do all of the things like Egg did.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160816/d87820ef/attachment.html>

From donald at stufft.io  Tue Aug 16 12:10:01 2016
From: donald at stufft.io (Donald Stufft)
Date: Tue, 16 Aug 2016 12:10:01 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAMxX_oOEKxNy45LgTwuQogSDoLrTLD_VUY2QHH0A8orO15Phig@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CACac1F8LSnH_7=vORDxWdcqMs2=hZ9Vi=MrQ7uF4m6tWy3Fyiw@mail.gmail.com>
 <CAG8k2+61eY685hZPxoK6H34gn1oxz1sOFC1aom_0fEDbmjhjWw@mail.gmail.com>
 <CAMxX_oOEKxNy45LgTwuQogSDoLrTLD_VUY2QHH0A8orO15Phig@mail.gmail.com>
Message-ID: <C4CDB2B3-4600-478E-8230-438A6B6F3261@stufft.io>


> On Aug 16, 2016, at 11:15 AM, Leonardo Rochael Almeida <leorochael at gmail.com> wrote:
> 
> Specifically, buildout right now has setuptools as its only dependence, and buildout uses it to locate, download and build sdist dependencies, or directly download and use .egg dependencies, which are important for Windows platforms in the communities that use buildout and depend on compiled extensions.


Ah, I knew I was forgetting something. I think we should hold off on preventing egg uploads until setuptools can download and install wheels.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160816/8c0fefc9/attachment.html>

From donald at stufft.io  Tue Aug 16 12:15:01 2016
From: donald at stufft.io (Donald Stufft)
Date: Tue, 16 Aug 2016 12:15:01 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAP1=2W4p75=7STAs+EAo+dFY5sQvp8Gei_dnqVjW55QO6G6v+A@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAP1=2W4p75=7STAs+EAo+dFY5sQvp8Gei_dnqVjW55QO6G6v+A@mail.gmail.com>
Message-ID: <81AB32E5-8C5F-4BEC-BC4E-F6D5BC04B8B0@stufft.io>


> On Aug 16, 2016, at 11:51 AM, Brett Cannon <brett at python.org> wrote:
> 
> One thing to remember is that Windows can't read tar files natively while it can for zip files. Now you can easily download tools on Windows to read tar files and thanks to Bash on Windows you even have it included once you turn that feature on.

This is true, but I think that using .tar.gz by default still makes sense because it?s still the vast bulk of what people actually release to PyPI. So it represents the status quo and switching to zip as the default would break a lot of things.

> 
> The other point is we have a zip importer in Python but not a .tar.gz one. I don't know how often anyone actually downloads a zip file directly from PyPI and then tack it on to their sys.path for importing, but that is currently possible.

A sdist is not an acceptable format for adding to sys.path. While in many, simple cases, it will ?just work?, that?s more of an implementation detail than anything else. There are many projects which simply do not run or error out if you do this. I don?t think worrying about something that sort of works, sometimes, is a big deal.

> 
> I doubt either of these points are important enough to continue to support zip files for sdists, but I just wanted to point it out. At worst this is something to think about if we ever do formalize the sdist format and come up with a custom file extension.

If we make a sdist 2.0 with a new format, I do think it makes sense to make it a zipfile like wheel already is (which reduces the internal formats down from 2 to 1), not for the reasons above though, just for consistency with wheel.


?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160816/d95006a5/attachment-0001.html>

From dholth at gmail.com  Tue Aug 16 12:37:05 2016
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 16 Aug 2016 16:37:05 +0000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <70914ED5-7A06-4A27-AF14-33C37C6C7BE7@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CACac1F8LSnH_7=vORDxWdcqMs2=hZ9Vi=MrQ7uF4m6tWy3Fyiw@mail.gmail.com>
 <CAG8k2+61eY685hZPxoK6H34gn1oxz1sOFC1aom_0fEDbmjhjWw@mail.gmail.com>
 <70914ED5-7A06-4A27-AF14-33C37C6C7BE7@stufft.io>
Message-ID: <CAG8k2+5GxzqtCZcJ8K-xPgAJANfGf8=HcsTzmR44HUJKqSFG7A@mail.gmail.com>

On Tue, Aug 16, 2016 at 12:06 PM Donald Stufft <donald at stufft.io> wrote:

>
> On Aug 16, 2016, at 8:50 AM, Daniel Holth <dholth at gmail.com> wrote:
>
> Wheel should be updated to support the egg use case before egg is removed.
> IIUC this would mostly mean officially supporting 'unzipped wheel' as a
> thing you can add to PYTHONPATH, possibly with some additional restrictions
> for the specific wheel. We could go a little further and officially support
> zipped wheels "if zip safe". We could implement wheel2egg to complement
> egg2wheel?
>
>
>
> I don?t think Wheel should officially supported unzip wheels as a thing
> you can add to PYTHONPATH nor do I think we should officially support
> zipped wheels being added to PYTHONPATH. Neither of those things are going
> to work universally and setuptools has gross heuristics to try and figure
> out when they will and won?t work (which regularly break or report
> inaccurately). Wheel is improved by remaining focused on being a format for
> distributing and installed via an installer, not one that tries to do all
> of the things like Egg did.
>

So this is how I envision "installing a wheel to its own directory".

1. Unzip the wheel into its own directory.
2. If there are any nested *.data/purelib, platlib, make sure they are also
unzipped into the root of the unzipped wheel instead of their archive paths.
3. If there are scripts and you want them, install those too.

If 2 and 3 don't apply to you, you are done almost before you've started.
What's missing?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160816/784432f5/attachment.html>

From alex.gronholm at nextday.fi  Tue Aug 16 12:43:07 2016
From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=)
Date: Tue, 16 Aug 2016 19:43:07 +0300
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAG8k2+5GxzqtCZcJ8K-xPgAJANfGf8=HcsTzmR44HUJKqSFG7A@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CACac1F8LSnH_7=vORDxWdcqMs2=hZ9Vi=MrQ7uF4m6tWy3Fyiw@mail.gmail.com>
 <CAG8k2+61eY685hZPxoK6H34gn1oxz1sOFC1aom_0fEDbmjhjWw@mail.gmail.com>
 <70914ED5-7A06-4A27-AF14-33C37C6C7BE7@stufft.io>
 <CAG8k2+5GxzqtCZcJ8K-xPgAJANfGf8=HcsTzmR44HUJKqSFG7A@mail.gmail.com>
Message-ID: <29a003b8-e319-ee01-582a-8d529d49cca3@nextday.fi>

16.08.2016, 19:37, Daniel Holth kirjoitti:
> On Tue, Aug 16, 2016 at 12:06 PM Donald Stufft <donald at stufft.io 
> <mailto:donald at stufft.io>> wrote:
>
>
>>     On Aug 16, 2016, at 8:50 AM, Daniel Holth <dholth at gmail.com
>>     <mailto:dholth at gmail.com>> wrote:
>>
>>     Wheel should be updated to support the egg use case before egg is
>>     removed. IIUC this would mostly mean officially supporting
>>     'unzipped wheel' as a thing you can add to PYTHONPATH, possibly
>>     with some additional restrictions for the specific wheel. We
>>     could go a little further and officially support zipped wheels
>>     "if zip safe". We could implement wheel2egg to complement egg2wheel?
>
>
>     I don?t think Wheel should officially supported unzip wheels as a
>     thing you can add to PYTHONPATH nor do I think we should
>     officially support zipped wheels being added to PYTHONPATH.
>     Neither of those things are going to work universally and
>     setuptools has gross heuristics to try and figure out when they
>     will and won?t work (which regularly break or report
>     inaccurately). Wheel is improved by remaining focused on being a
>     format for distributing and installed via an installer, not one
>     that tries to do all of the things like Egg did.
>
>
> So this is how I envision "installing a wheel to its own directory".
>
> 1. Unzip the wheel into its own directory.
> 2. If there are any nested *.data/purelib, platlib, make sure they are 
> also unzipped into the root of the unzipped wheel instead of their 
> archive paths.
> 3. If there are scripts and you want them, install those too.
>
> If 2 and 3 don't apply to you, you are done almost before you've 
> started. What's missing?
>
How does one go about installing console_scripts this way?
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160816/896c6c4c/attachment.html>

From dholth at gmail.com  Tue Aug 16 12:46:29 2016
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 16 Aug 2016 16:46:29 +0000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <81AB32E5-8C5F-4BEC-BC4E-F6D5BC04B8B0@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAP1=2W4p75=7STAs+EAo+dFY5sQvp8Gei_dnqVjW55QO6G6v+A@mail.gmail.com>
 <81AB32E5-8C5F-4BEC-BC4E-F6D5BC04B8B0@stufft.io>
Message-ID: <CAG8k2+4=taMGyDHUtnhh17Na_hJVt3tYiWDtXpZ+ZSwUk5vHnw@mail.gmail.com>

On Tue, Aug 16, 2016 at 12:15 PM Donald Stufft <donald at stufft.io> wrote:

> On Aug 16, 2016, at 11:51 AM, Brett Cannon <brett at python.org> wrote:
>
> One thing to remember is that Windows can't read tar files natively while
> it can for zip files. Now you can easily download tools on Windows to read
> tar files and thanks to Bash on Windows you even have it included once you
> turn that feature on.
>
>
> This is true, but I think that using .tar.gz by default still makes sense
> because it?s still the vast bulk of what people actually release to PyPI.
> So it represents the status quo and switching to zip as the default would
> break a lot of things.
>
>
> The other point is we have a zip importer in Python but not a .tar.gz one. I
> don't know how often anyone actually downloads a zip file directly from
> PyPI and then tack it on to their sys.path for importing, but that is
> currently possible.
>
>
> A sdist is not an acceptable format for adding to sys.path. While in many,
> simple cases, it will ?just work?, that?s more of an implementation detail
> than anything else. There are many projects which simply do not run or
> error out if you do this. I don?t think worrying about something that sort
> of works, sometimes, is a big deal.
>
>
> I doubt either of these points are important enough to continue to support
> zip files for sdists, but I just wanted to point it out. At worst this is
> something to think about if we ever do formalize the sdist format and come
> up with a custom file extension.
>
>
> If we make a sdist 2.0 with a new format, I do think it makes sense to
> make it a zipfile like wheel already is (which reduces the internal formats
> down from 2 to 1), not for the reasons above though, just for consistency
> with wheel.
>

ZIP is a fantastically designed file format. JAR, IPA (iPhone), all are ZIP
files. The only thing I sometimes wonder about for wheel is whether it
would be worth the trouble to support greater compression with something
like .zip.xz (an uncompressed ZIP inside a wrapper, possibly another ZIP)
since ZIP compresses each file individually and does not compress its
metadata. But most packages are quite small.

ZIP's greatest weakness is also its greatest strength, as ZIP allows random
access to its members and very fast access to its metadata. This is why it
makes sense to have a ZIP importer but not a .tar.gz importer.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160816/d8422a76/attachment-0001.html>

From dholth at gmail.com  Tue Aug 16 12:48:56 2016
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 16 Aug 2016 16:48:56 +0000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <29a003b8-e319-ee01-582a-8d529d49cca3@nextday.fi>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CACac1F8LSnH_7=vORDxWdcqMs2=hZ9Vi=MrQ7uF4m6tWy3Fyiw@mail.gmail.com>
 <CAG8k2+61eY685hZPxoK6H34gn1oxz1sOFC1aom_0fEDbmjhjWw@mail.gmail.com>
 <70914ED5-7A06-4A27-AF14-33C37C6C7BE7@stufft.io>
 <CAG8k2+5GxzqtCZcJ8K-xPgAJANfGf8=HcsTzmR44HUJKqSFG7A@mail.gmail.com>
 <29a003b8-e319-ee01-582a-8d529d49cca3@nextday.fi>
Message-ID: <CAG8k2+6V5c2a_dag6G2uKYJ=6WzJO96V5JbdqxLcyEMHGFv1Gw@mail.gmail.com>

On Tue, Aug 16, 2016 at 12:43 PM Alex Gr?nholm <alex.gronholm at nextday.fi>
wrote:

> 16.08.2016, 19:37, Daniel Holth kirjoitti:
>
> On Tue, Aug 16, 2016 at 12:06 PM Donald Stufft <donald at stufft.io> wrote:
>
>>
>> On Aug 16, 2016, at 8:50 AM, Daniel Holth <dholth at gmail.com> wrote:
>>
>> Wheel should be updated to support the egg use case before egg is
>> removed. IIUC this would mostly mean officially supporting 'unzipped wheel'
>> as a thing you can add to PYTHONPATH, possibly with some additional
>> restrictions for the specific wheel. We could go a little further and
>> officially support zipped wheels "if zip safe". We could implement
>> wheel2egg to complement egg2wheel?
>>
>>
>>
>> I don?t think Wheel should officially supported unzip wheels as a thing
>> you can add to PYTHONPATH nor do I think we should officially support
>> zipped wheels being added to PYTHONPATH. Neither of those things are going
>> to work universally and setuptools has gross heuristics to try and figure
>> out when they will and won?t work (which regularly break or report
>> inaccurately). Wheel is improved by remaining focused on being a format for
>> distributing and installed via an installer, not one that tries to do all
>> of the things like Egg did.
>>
>
> So this is how I envision "installing a wheel to its own directory".
>
> 1. Unzip the wheel into its own directory.
> 2. If there are any nested *.data/purelib, platlib, make sure they are
> also unzipped into the root of the unzipped wheel instead of their archive
> paths.
> 3. If there are scripts and you want them, install those too.
>
> If 2 and 3 don't apply to you, you are done almost before you've started.
> What's missing?
>
> How does one go about installing console_scripts this way?
>

No difference. Read entry_points.txt and generate script wrappers for all
of the listed console_scripts. In buildout's case it adds all of the
dependencies to sys.path in the generated console_script wrapper.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160816/38e0c8b5/attachment.html>

From graffatcolmingov at gmail.com  Tue Aug 16 12:49:44 2016
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Tue, 16 Aug 2016 11:49:44 -0500
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAG8k2+4=taMGyDHUtnhh17Na_hJVt3tYiWDtXpZ+ZSwUk5vHnw@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAP1=2W4p75=7STAs+EAo+dFY5sQvp8Gei_dnqVjW55QO6G6v+A@mail.gmail.com>
 <81AB32E5-8C5F-4BEC-BC4E-F6D5BC04B8B0@stufft.io>
 <CAG8k2+4=taMGyDHUtnhh17Na_hJVt3tYiWDtXpZ+ZSwUk5vHnw@mail.gmail.com>
Message-ID: <CAN-Kwu0E1zF7OB23n8Jb80FsPQ-A7cnoB=cQRHjprbKGeT2jmg@mail.gmail.com>

So perhaps I'm missing something, but why are we talking about trying
to shoehorn a legacy design into Wheel? Why wouldn't we leave
bdist_egg alone and start trying to find a better way to replace it?
We could avoid over-complicating Wheel and we could spend time
polishing whatever we come up with.

From dholth at gmail.com  Tue Aug 16 12:58:25 2016
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 16 Aug 2016 16:58:25 +0000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAN-Kwu0E1zF7OB23n8Jb80FsPQ-A7cnoB=cQRHjprbKGeT2jmg@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAP1=2W4p75=7STAs+EAo+dFY5sQvp8Gei_dnqVjW55QO6G6v+A@mail.gmail.com>
 <81AB32E5-8C5F-4BEC-BC4E-F6D5BC04B8B0@stufft.io>
 <CAG8k2+4=taMGyDHUtnhh17Na_hJVt3tYiWDtXpZ+ZSwUk5vHnw@mail.gmail.com>
 <CAN-Kwu0E1zF7OB23n8Jb80FsPQ-A7cnoB=cQRHjprbKGeT2jmg@mail.gmail.com>
Message-ID: <CAG8k2+4VXkG6jvMze+QRWp9fbNGFk9-Pp6U5e5cPrrs8a51eYw@mail.gmail.com>

On Tue, Aug 16, 2016 at 12:50 PM Ian Cordasco <graffatcolmingov at gmail.com>
wrote:

> So perhaps I'm missing something, but why are we talking about trying
> to shoehorn a legacy design into Wheel? Why wouldn't we leave
> bdist_egg alone and start trying to find a better way to replace it?
> We could avoid over-complicating Wheel and we could spend time
> polishing whatever we come up with.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


It is because wheel is almost identical to egg, but the name 'newegg' was
already in use by a major online retailer. Possibly adding the one missing
egg feature to wheel would save someone some time. Maybe you would want to
give 'extracted wheel in its own directory' a different name, or no name,
to avoid confusion.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160816/ad18c498/attachment.html>

From leorochael at gmail.com  Tue Aug 16 12:58:38 2016
From: leorochael at gmail.com (Leonardo Rochael Almeida)
Date: Tue, 16 Aug 2016 13:58:38 -0300
Subject: [Distutils] Future of setuptools and buildout
Message-ID: <CAMxX_oP-8R=Gr4Jp9P8CXNSB7muuq=2z8E0K_V7qCvFodJTJOQ@mail.gmail.com>

Spinning this off into its own thread...

On 16 August 2016 at 13:10, Donald Stufft <donald at stufft.io> wrote:

>
> On Aug 16, 2016, at 11:15 AM, Leonardo Rochael Almeida <
> leorochael at gmail.com> wrote:
>
> Specifically, buildout right now has setuptools as its only dependence,
> and buildout uses it to locate, download and build sdist dependencies, or
> directly download and use .egg dependencies, which are important for
> Windows platforms in the communities that use buildout and depend on
> compiled extensions.
>
>
>
> Ah, I knew I was forgetting something. I think we should hold off on
> preventing egg uploads until setuptools can download and install wheels.
>

Which brings us to a question that I'm meaning to ask for a while.

It looks like we're close to removing all mentions of setuptools in pip.
When this happens, it looks like pressure is going to start to mount on
setuptools to drop the ability to install packages and limit itself on
being just a build tool.

It makes sense that it should be so, considering that there would be no
incentive to keep around two distinct implementations of how to locate,
download and install stuff, one in pip (or whatever library pip uses for
locating and installing packages) and another in setuptools.

In this future, I can imagine setuptools will simply complain loudly when
the packages it requires to do its work are not installed yet, perhaps with
a plugin mechanism for allowing another package to automatically resolve
and installing packages, (like today setuptools-git extends setuptools to
recognize files from git to be included in a package).

And people will promptly write a pip implementation of this plugin (or
whatever it is that pip will use as a library to do it)... thereby
inverting the dependency between setuptools and pip.

When this future arrives, buildout will be out of a mechanism for locating,
downloading and installing packages (not counting old versions of
setuptools, of course).

Questions:

1. Is the future outlined above considered likely?

2. If this future arrives, what package should buildout be ported to use?
It should provide:

 * The ability to locate packages and their versions, resolve their
dependencies, and download them, whether they're sdists or wheels (or eggs,
while we still have them)
 * Install each package separately into it's own directory that, if added
to `sys.path`, allows this same package (or perhaps another one) to read
information from entry points so that buildout can locate its own
extensions and install console scripts.

Regards,

Leo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160816/f4a734a0/attachment-0001.html>

From donald at stufft.io  Tue Aug 16 13:00:19 2016
From: donald at stufft.io (Donald Stufft)
Date: Tue, 16 Aug 2016 13:00:19 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAG8k2+4=taMGyDHUtnhh17Na_hJVt3tYiWDtXpZ+ZSwUk5vHnw@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAP1=2W4p75=7STAs+EAo+dFY5sQvp8Gei_dnqVjW55QO6G6v+A@mail.gmail.com>
 <81AB32E5-8C5F-4BEC-BC4E-F6D5BC04B8B0@stufft.io>
 <CAG8k2+4=taMGyDHUtnhh17Na_hJVt3tYiWDtXpZ+ZSwUk5vHnw@mail.gmail.com>
Message-ID: <A11B0DCB-4525-4B24-A439-FDE5CB5B8E7A@stufft.io>


> On Aug 16, 2016, at 12:46 PM, Daniel Holth <dholth at gmail.com> wrote:
> 
> On Tue, Aug 16, 2016 at 12:15 PM Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
>> On Aug 16, 2016, at 11:51 AM, Brett Cannon <brett at python.org <mailto:brett at python.org>> wrote:
>> 
>> One thing to remember is that Windows can't read tar files natively while it can for zip files. Now you can easily download tools on Windows to read tar files and thanks to Bash on Windows you even have it included once you turn that feature on.
> 
> This is true, but I think that using .tar.gz by default still makes sense because it?s still the vast bulk of what people actually release to PyPI. So it represents the status quo and switching to zip as the default would break a lot of things.
> 
>> 
>> The other point is we have a zip importer in Python but not a .tar.gz one. I don't know how often anyone actually downloads a zip file directly from PyPI and then tack it on to their sys.path for importing, but that is currently possible.
> 
> A sdist is not an acceptable format for adding to sys.path. While in many, simple cases, it will ?just work?, that?s more of an implementation detail than anything else. There are many projects which simply do not run or error out if you do this. I don?t think worrying about something that sort of works, sometimes, is a big deal.
> 
>> 
>> I doubt either of these points are important enough to continue to support zip files for sdists, but I just wanted to point it out. At worst this is something to think about if we ever do formalize the sdist format and come up with a custom file extension.
> 
> If we make a sdist 2.0 with a new format, I do think it makes sense to make it a zipfile like wheel already is (which reduces the internal formats down from 2 to 1), not for the reasons above though, just for consistency with wheel.
> 
> ZIP is a fantastically designed file format. JAR, IPA (iPhone), all are ZIP files. The only thing I sometimes wonder about for wheel is whether it would be worth the trouble to support greater compression with something like .zip.xz (an uncompressed ZIP inside a wrapper, possibly another ZIP) since ZIP compresses each file individually and does not compress its metadata. But most packages are quite small.

I don?t think it would be worth it. Right now there?s only 1 optional c library required (zlib) to install from wheels. If we deprecate all of the other things then there?s only 1 optional c library required to install from sdists too (zlib). At best it would make sense to use gzip, but since we?re already use ZIP_DEFLATE I?m not sure the extra complexity is worth it.

> 
> ZIP's greatest weakness is also its greatest strength, as ZIP allows random access to its members and very fast access to its metadata. This is why it makes sense to have a ZIP importer but not a .tar.gz importer.


?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160816/61e53a46/attachment.html>

From p.f.moore at gmail.com  Tue Aug 16 13:00:17 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 16 Aug 2016 18:00:17 +0100
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAN-Kwu0E1zF7OB23n8Jb80FsPQ-A7cnoB=cQRHjprbKGeT2jmg@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAP1=2W4p75=7STAs+EAo+dFY5sQvp8Gei_dnqVjW55QO6G6v+A@mail.gmail.com>
 <81AB32E5-8C5F-4BEC-BC4E-F6D5BC04B8B0@stufft.io>
 <CAG8k2+4=taMGyDHUtnhh17Na_hJVt3tYiWDtXpZ+ZSwUk5vHnw@mail.gmail.com>
 <CAN-Kwu0E1zF7OB23n8Jb80FsPQ-A7cnoB=cQRHjprbKGeT2jmg@mail.gmail.com>
Message-ID: <CACac1F9FdEY8h8xXgpKmG__o33+qXd8eaT7zwx3-vUVmd-88dQ@mail.gmail.com>

On 16 August 2016 at 17:49, Ian Cordasco <graffatcolmingov at gmail.com> wrote:
> So perhaps I'm missing something, but why are we talking about trying
> to shoehorn a legacy design into Wheel? Why wouldn't we leave
> bdist_egg alone and start trying to find a better way to replace it?
> We could avoid over-complicating Wheel and we could spend time
> polishing whatever we come up with.

+1

Wheel should remain a distribution format (with running from wheel,
zipped or unzipped, an intentional but unsupported benefit - get-pip
and virtualenv use the feature, but it should be treated as "use at
your own risk").

Egg as distribution format is deprecated and we shouldn't look to
encourage it, but egg as runtime format is fine if people want/need it
(I don't see the value myself, but the work needed to modify something
like buildout to move away from it probably isn't worth it).
Downloading a wheel and converting it to a runtime egg format is
perfectly possible. Whether buildout's preference for not adding
dependencies is sufficient to push setuptools to add support for that
route, I can't say. But I'm perfectly OK with saying that in due
course, we'll stop supporting eggs for distribution (e.g., in terms of
hosting on PyPI) and then provide a period for the likes of
buildout/setuptools to decide how to migrate. If they choose not to,
then fine - we have to decide whether buildout is important enough to
unilaterally block the change. But I don't think we should rush on
this - let's just publish the statement of intent, and then give
projects like buildout the time they need to react.

Paul

From donald at stufft.io  Tue Aug 16 13:01:08 2016
From: donald at stufft.io (Donald Stufft)
Date: Tue, 16 Aug 2016 13:01:08 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAG8k2+4VXkG6jvMze+QRWp9fbNGFk9-Pp6U5e5cPrrs8a51eYw@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAP1=2W4p75=7STAs+EAo+dFY5sQvp8Gei_dnqVjW55QO6G6v+A@mail.gmail.com>
 <81AB32E5-8C5F-4BEC-BC4E-F6D5BC04B8B0@stufft.io>
 <CAG8k2+4=taMGyDHUtnhh17Na_hJVt3tYiWDtXpZ+ZSwUk5vHnw@mail.gmail.com>
 <CAN-Kwu0E1zF7OB23n8Jb80FsPQ-A7cnoB=cQRHjprbKGeT2jmg@mail.gmail.com>
 <CAG8k2+4VXkG6jvMze+QRWp9fbNGFk9-Pp6U5e5cPrrs8a51eYw@mail.gmail.com>
Message-ID: <D7EA310C-7987-4CC9-BF66-8D8651373277@stufft.io>


> On Aug 16, 2016, at 12:58 PM, Daniel Holth <dholth at gmail.com> wrote:
> 
> On Tue, Aug 16, 2016 at 12:50 PM Ian Cordasco <graffatcolmingov at gmail.com <mailto:graffatcolmingov at gmail.com>> wrote:
> So perhaps I'm missing something, but why are we talking about trying
> to shoehorn a legacy design into Wheel? Why wouldn't we leave
> bdist_egg alone and start trying to find a better way to replace it?
> We could avoid over-complicating Wheel and we could spend time
> polishing whatever we come up with.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org <mailto:Distutils-SIG at python.org>
> https://mail.python.org/mailman/listinfo/distutils-sig <https://mail.python.org/mailman/listinfo/distutils-sig>
> 
> It is because wheel is almost identical to egg, but the name 'newegg' was already in use by a major online retailer. Possibly adding the one missing egg feature to wheel would save someone some time. Maybe you would want to give 'extracted wheel in its own directory' a different name, or no name, to avoid confusion.

Installing a wheel to a single directory is fine. It?s no longer a wheel at that point, it is an installed distribution.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160816/bff3fd02/attachment-0001.html>

From dholth at gmail.com  Tue Aug 16 13:11:10 2016
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 16 Aug 2016 17:11:10 +0000
Subject: [Distutils] Future of setuptools and buildout
In-Reply-To: <CAMxX_oP-8R=Gr4Jp9P8CXNSB7muuq=2z8E0K_V7qCvFodJTJOQ@mail.gmail.com>
References: <CAMxX_oP-8R=Gr4Jp9P8CXNSB7muuq=2z8E0K_V7qCvFodJTJOQ@mail.gmail.com>
Message-ID: <CAG8k2+7z3mpev62SLM9VXwh8KZekGLtSYc-vZ+Nk+Ui=2sJ+7A@mail.gmail.com>

There are two proposals to add pluggable build systems to Python sdists,
and one of them will probably be implemented. You add a pyproject.toml to
the root of your sdist, which the installer uses to install dependencies
that setup.py itself needs to run. Second, you also tell pip which build
system you are using, once you have done that, setup.py is no longer
necessary - instead, pip could install and then invoke flit for example.

This isn't quite the situation you've outlined as setuptools will likely
continue to have all its features. Instead, packages that don't use
setuptools will also proliferate. It causes about the same problem for
buildout without breaking older packages.

In this scenario buildout will need to be able to install wheels however it
wishes to do so because that is the output of the pluggable build system. I
don't have advice for you on the implementation, except perhaps it could
call out to pip.

On Tue, Aug 16, 2016 at 12:59 PM Leonardo Rochael Almeida <
leorochael at gmail.com> wrote:

> Spinning this off into its own thread...
>
> On 16 August 2016 at 13:10, Donald Stufft <donald at stufft.io> wrote:
>
>>
>> On Aug 16, 2016, at 11:15 AM, Leonardo Rochael Almeida <
>> leorochael at gmail.com> wrote:
>>
>> Specifically, buildout right now has setuptools as its only dependence,
>> and buildout uses it to locate, download and build sdist dependencies, or
>> directly download and use .egg dependencies, which are important for
>> Windows platforms in the communities that use buildout and depend on
>> compiled extensions.
>>
>>
>>
>> Ah, I knew I was forgetting something. I think we should hold off on
>> preventing egg uploads until setuptools can download and install wheels.
>>
>
> Which brings us to a question that I'm meaning to ask for a while.
>
> It looks like we're close to removing all mentions of setuptools in pip.
> When this happens, it looks like pressure is going to start to mount on
> setuptools to drop the ability to install packages and limit itself on
> being just a build tool.
>
> It makes sense that it should be so, considering that there would be no
> incentive to keep around two distinct implementations of how to locate,
> download and install stuff, one in pip (or whatever library pip uses for
> locating and installing packages) and another in setuptools.
>
> In this future, I can imagine setuptools will simply complain loudly when
> the packages it requires to do its work are not installed yet, perhaps with
> a plugin mechanism for allowing another package to automatically resolve
> and installing packages, (like today setuptools-git extends setuptools to
> recognize files from git to be included in a package).
>
> And people will promptly write a pip implementation of this plugin (or
> whatever it is that pip will use as a library to do it)... thereby
> inverting the dependency between setuptools and pip.
>
> When this future arrives, buildout will be out of a mechanism for
> locating, downloading and installing packages (not counting old versions of
> setuptools, of course).
>
> Questions:
>
> 1. Is the future outlined above considered likely?
>
> 2. If this future arrives, what package should buildout be ported to use?
> It should provide:
>
>  * The ability to locate packages and their versions, resolve their
> dependencies, and download them, whether they're sdists or wheels (or eggs,
> while we still have them)
>  * Install each package separately into it's own directory that, if added
> to `sys.path`, allows this same package (or perhaps another one) to read
> information from entry points so that buildout can locate its own
> extensions and install console scripts.
>
> Regards,
>
> Leo
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160816/6083a2fa/attachment.html>

From donald at stufft.io  Tue Aug 16 13:12:25 2016
From: donald at stufft.io (Donald Stufft)
Date: Tue, 16 Aug 2016 13:12:25 -0400
Subject: [Distutils] Future of setuptools and buildout
In-Reply-To: <CAMxX_oP-8R=Gr4Jp9P8CXNSB7muuq=2z8E0K_V7qCvFodJTJOQ@mail.gmail.com>
References: <CAMxX_oP-8R=Gr4Jp9P8CXNSB7muuq=2z8E0K_V7qCvFodJTJOQ@mail.gmail.com>
Message-ID: <EED02033-FB4E-4B25-A37B-C09B4CDF5F31@stufft.io>


> On Aug 16, 2016, at 12:58 PM, Leonardo Rochael Almeida <leorochael at gmail.com> wrote:
> 
> Spinning this off into its own thread...
> 
> On 16 August 2016 at 13:10, Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
> 
>> On Aug 16, 2016, at 11:15 AM, Leonardo Rochael Almeida <leorochael at gmail.com <mailto:leorochael at gmail.com>> wrote:
>> 
>> Specifically, buildout right now has setuptools as its only dependence, and buildout uses it to locate, download and build sdist dependencies, or directly download and use .egg dependencies, which are important for Windows platforms in the communities that use buildout and depend on compiled extensions.
> 
> 
> Ah, I knew I was forgetting something. I think we should hold off on preventing egg uploads until setuptools can download and install wheels.
> 
> Which brings us to a question that I'm meaning to ask for a while.
> 
> It looks like we're close to removing all mentions of setuptools in pip. When this happens, it looks like pressure is going to start to mount on setuptools to drop the ability to install packages and limit itself on being just a build tool.
> 
> It makes sense that it should be so, considering that there would be no incentive to keep around two distinct implementations of how to locate, download and install stuff, one in pip (or whatever library pip uses for locating and installing packages) and another in setuptools.

So one of our end goals here (and why we?re so stuck on standards for everything, and not ad hoc things) is to make it entirely possible to have multiple distinct implementations of any part of the tool chain without having to try and copy each other?s features/implementation details to work. There?s one standard for what is correct and as long as you follow that, then you should work.

With that in mind, I certainly hope to get to a point where at the very least, setuptools make the conscious choice to continue to implement these features, rather than needing to do it because pip depends on it. Whether it makes sense for setuptools to continue to support them or not is something that the setuptools devs (particularly Jason) is going to be better suited to answer, but for my own opinion I hope we can get to a point that setuptools is just a build tool.

> 
> In this future, I can imagine setuptools will simply complain loudly when the packages it requires to do its work are not installed yet, perhaps with a plugin mechanism for allowing another package to automatically resolve and installing packages, (like today setuptools-git extends setuptools to recognize files from git to be included in a package).
> 
> And people will promptly write a pip implementation of this plugin (or whatever it is that pip will use as a library to do it)... thereby inverting the dependency between setuptools and pip.
> 
> When this future arrives, buildout will be out of a mechanism for locating, downloading and installing packages (not counting old versions of setuptools, of course).
> 
> Questions:
> 
> 1. Is the future outlined above considered likely?

I suspect that maybe, at some point it might be, but I think that setuptools will likely keep it?s support for these things for a good long while to avoid breaking things for people. At least until a satisfactory answer for what someone who is currently using setuptools should use emerges.

> 
> 2. If this future arrives, what package should buildout be ported to use? It should provide:
> 
>  * The ability to locate packages and their versions, resolve their dependencies, and download them, whether they're sdists or wheels (or eggs, while we still have them)
>  * Install each package separately into it's own directory that, if added to `sys.path`, allows this same package (or perhaps another one) to read information from entry points so that buildout can locate its own extensions and install console scripts.
> 


So you have a few choices, right now you can use pip in a subprocess (pip does not expose a programmatic API) or continue to use setuptools.

One of the things we?ve been trying to do with pip is instead of adding/exposing parts of pip via a programmatic API, is to instead extract them out into their own dedicated libraries that then pip consumes. This has the benefit that setuptools can depend on those things, instead of pip itself, but also that other people can come along and piece meal use the pieces that make sense for them. If someone has a great new idea for a pip-killer that does everything better, then they can do that, and reuse these libraries to help make their thing better right out of the starting gate.

So we could work on that effort too, I think the main thing holding it back is both developer time, as well as solid use cases so we can design the API around them. 

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160816/88832cb0/attachment-0001.html>

From p.f.moore at gmail.com  Tue Aug 16 13:18:32 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 16 Aug 2016 18:18:32 +0100
Subject: [Distutils] Future of setuptools and buildout
In-Reply-To: <CAMxX_oP-8R=Gr4Jp9P8CXNSB7muuq=2z8E0K_V7qCvFodJTJOQ@mail.gmail.com>
References: <CAMxX_oP-8R=Gr4Jp9P8CXNSB7muuq=2z8E0K_V7qCvFodJTJOQ@mail.gmail.com>
Message-ID: <CACac1F81T8DbC_vOqhPbj0naJ-UhmCn6ub37qM9MBfDmzMD3Gg@mail.gmail.com>

On 16 August 2016 at 17:58, Leonardo Rochael Almeida
<leorochael at gmail.com> wrote:
> Questions:
>
> 1. Is the future outlined above considered likely?

I can't speak for setuptools, but certainly pip is intended to be the
canonical installer for Python. It's not impossible for competing
tools like the download/install features in setuptools to remain, but
like you say, I'd expect the trend to be against them.

> 2. If this future arrives, what package should buildout be ported to use?

That's really for buildout to decide.

> It should provide:
>
>  * The ability to locate packages and their versions, resolve their
> dependencies, and download them, whether they're sdists or wheels (or eggs,
> while we still have them)

pip will do this (no support for eggs exists or will be added, but
sdists and wheels yes). Note that pip should be run as a command line
tool - we don't support use of pip's internal API from an existing
Python process. If an in-process API is important, though, it wouldn't
be impossible to discuss adding that.

>  * Install each package separately into it's own directory that, if added to
> `sys.path`, allows this same package (or perhaps another one) to read
> information from entry points so that buildout can locate its own extensions
> and install console scripts.

That's not an installer feature. So pip doesn't provide anything
specific for that. Rather, that's a runtime support facility that is
tied to Python's import mechanisms. Setuptools (or rather,
pkg_resources?) may well continue to provide that, but laying out the
files present in a wheel in the format you need to support your
requirements would have to be added - maybe setuptools would be
interested in providing such a facility, I don't know. But it's not
exactly hard, at the most basic level you're probably only talking
about unzipping the wheel and writing a bit of metadata based on
what's specified in the wheel. Heck, a "wheel to egg" function
shouldn't be that hard to write, and then you could simply use the
existing runtime egg support code.

However, I don't believe that mandating that a single package should
provide both requirements is necessary, or helpful. The two features
are different - one of the features of setuptools that many people
(myself included) disliked, was the fact that it mixed "install time"
and "run time" features in one package, making it hard to avoid having
install time capabilities in a runtime-only environment.

Having said this, if someone were to propose, design and standardise a
*runtime* format for whatever you're hinting at as "install into its
own directory ... read information", and get it accepted as an
installation layout standard, then it might make sense to add support
to pip for installing files using that layout (analogous to the
"--target" flag for pip install). Runtime support would still have to
come from elsewhere, though. The design ideas for that spec may even
be based on the ideas developed for egg. But the key focus is moving
away from adhoc, implementation-defined standards, to properly
specified and documented standards. So step 1 would be to get such a
standard accepted, *then* look at proposing pip add it as an
alternative installation layout. (And I'd recommend not calling that
standard "egg" format - like it or not, the historical baggage
associated with that name will make it harder than you'd like to get
adoption of a new standard).

Hope this helps,
Paul

From ncoghlan at gmail.com  Wed Aug 17 01:21:52 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 17 Aug 2016 15:21:52 +1000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAMxX_oOEKxNy45LgTwuQogSDoLrTLD_VUY2QHH0A8orO15Phig@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CACac1F8LSnH_7=vORDxWdcqMs2=hZ9Vi=MrQ7uF4m6tWy3Fyiw@mail.gmail.com>
 <CAG8k2+61eY685hZPxoK6H34gn1oxz1sOFC1aom_0fEDbmjhjWw@mail.gmail.com>
 <CAMxX_oOEKxNy45LgTwuQogSDoLrTLD_VUY2QHH0A8orO15Phig@mail.gmail.com>
Message-ID: <CADiSq7cR9Dg6XxvTwp94hEFNNEOkbQebYi1e5Jp+Wf2H-H6-xg@mail.gmail.com>

On 17 August 2016 at 01:15, Leonardo Rochael Almeida
<leorochael at gmail.com> wrote:
> PS: In the buildout community, we never really understood the impetus for
> replacing egg as a format, which is really not all that complex and not all
> the different from wheel as it stands, instead of just formalizing it in a
> PEP. We got the feeling that, in the movement of getting rid of setuptools
> as an "installation" dependency, we ended up throwing out the baby with the
> bathwater...

The majority of the problems with eggs are due to the fact that they
were designed to operate *by default* as the plugin management system
for Chandler. buildout uses them in a similar way, so they're a good
fit there.

However, they're *not* a good fit for other cases, hence
"--single-version-externally-managed" and all of the other ways in
which pip switches the defaults to be more developer centric, and less
deployment focused. Most of the problems people ran into with the
defaults were actually related to the way easy_install and
pkg_resources worked rather than due to the egg format itself, but the
lack of clear documentation and specifications meant that the broader
adoption process for programmatic dependency management in general
needed to be based on a more modular reimplementation of similar ideas
with more explicit interoperability interfaces, rather than adopting
the setuptools/easy_install/pkg_resources implementation of those
ideas wholesale.

This means that the situation we've managed to get to now is that we
have wheels as a satisfactory replacement for the "binary
distribution" use case, but we haven't even *tried* to replace eggs
for the "standalone path entry" use case.

That gives us a few options for possible ways forward:

- define a new egg-inspired "*.pypath" format for directories and zip
files that are designed to be placed directly on sys.path, rather than
installed
- just bless eggs as the recommended standalone importable format, and
offer a wheel2egg conversion utility and perhaps an "--as-egg" install
option in pip
- as with the previous, but bake the wheel2egg conversion into
setuptools/easy_install rather than into pip

At least for now, I think that last option probably makes the most
sense, as it moves easy_install, buildout, and other egg based tools
into a similar position to conda, PyPM and the Linux distros: a
downstream community with its own binary distribution format, but
associated tooling to also make it easy for participants in that
ecosystem to consume pre-built wheel files if they want to.

While rebranding eggs (or a close derivative) as "pypath entries"
could help make them more self-explanatory (especially when it comes
to explaining how the zipped form differs from wheel files), doing so
wouldn't actually give us any more architectural flexibility in
practice than the last option.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed Aug 17 01:40:47 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 17 Aug 2016 15:40:47 +1000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CADiSq7cR9Dg6XxvTwp94hEFNNEOkbQebYi1e5Jp+Wf2H-H6-xg@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CACac1F8LSnH_7=vORDxWdcqMs2=hZ9Vi=MrQ7uF4m6tWy3Fyiw@mail.gmail.com>
 <CAG8k2+61eY685hZPxoK6H34gn1oxz1sOFC1aom_0fEDbmjhjWw@mail.gmail.com>
 <CAMxX_oOEKxNy45LgTwuQogSDoLrTLD_VUY2QHH0A8orO15Phig@mail.gmail.com>
 <CADiSq7cR9Dg6XxvTwp94hEFNNEOkbQebYi1e5Jp+Wf2H-H6-xg@mail.gmail.com>
Message-ID: <CADiSq7c-sfEjArH_yDEFBeJCPx1Mt_af86rOHCLbvUCJUZAQeg@mail.gmail.com>

On 17 August 2016 at 15:21, Nick Coghlan <ncoghlan at gmail.com> wrote:
> While rebranding eggs (or a close derivative) as "pypath entries"
> could help make them more self-explanatory (especially when it comes
> to explaining how the zipped form differs from wheel files), doing so
> wouldn't actually give us any more architectural flexibility in
> practice than the last option.

That said, if folks *did* want to go down this path, then an important
use case to consider is "single archive applications" suitable for use
in services like AWS Lambda and Azure Cloud Functions, where you get a
language runtime to play with, but need to provide everything you want
to run in that environment as a single pre-bundled archive. Similar
problems also show up when building rich client applications for
Windows, Mac OS X, Android and iOS.

This means you want tooling that targets the direct dependency
bundling model, rather than the runtime environment configuration
model favoured by pip and virtualenv.

So this particular challenge *shouldn't* be framed as just a backwards
compatibility and migration concern for buildout et al - the "all
inclusive application bundle" approach is still an important use case,
even though it isn't the lowest common denominator cross-platform
dependency specification, retrieval and installation problem handled
by the default toolchain.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From thomas at kluyver.me.uk  Wed Aug 17 04:31:36 2016
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Wed, 17 Aug 2016 09:31:36 +0100
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <42876D1B-D8DF-4E55-8A25-4B124DCB8107@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <1471289237.729994.696062937.74ECDBC7@webmail.messagingengine.com>
 <42876D1B-D8DF-4E55-8A25-4B124DCB8107@stufft.io>
Message-ID: <1471422696.1122472.697735209.671C09DA@webmail.messagingengine.com>

On Mon, Aug 15, 2016, at 08:40 PM, Donald Stufft wrote:
> Or were you wondering for the last 30 days like I did for downloads? If
> so then:
> 
> * sdist: 15601 (66%)
> * bdist_wheel: 6398 (27%)
> * bdist_egg: 1076 (4.5%)
> * bdist_dumb: 195 (0.8%)
> * bdist_wininst: 167 (0.7%)
> * bdist_rpm: 38 (0.1%)
> * bdist_msi: 9 (0.03%)
> * Everything Else: 0 (0%)

Sorry for the lack of clarity, I was indeed looking for these numbers
for recent uploads.

This suggests there is a small but non-negligible group still uploading
eggs. Is it worth looking at what projects are doing this, and
potentially contacting some authors to understand why?

From claude.marinier at lmco.com  Mon Aug 15 08:35:11 2016
From: claude.marinier at lmco.com (Marinier, Claude)
Date: Mon, 15 Aug 2016 12:35:11 +0000
Subject: [Distutils] license for setuptools
Message-ID: <1AE01599B99F8A4EA0609B55B478D77FE876F9@HCXDSPM1.ca.lmco.com>

> Date: Fri, 12 Aug 2016 02:45:20 +0000
> From: Eric Dill <thedizzle at gmail.com>
> Subject: Re: [Distutils] license for setuptools
>
> Hi Claude,
>
> There was a recent discussion of the lack of a license file in setuptools
> here: https://github.com/pypa/setuptools/issues/612 and another important
> discussion here: https://github.com/pypa/setuptools/issues/132.  This is
> probably the most relevant quotable bit from those two issues, from Jason
> Coombs (the primary developer of setuptools:
>
> "The [License :: OSI Approved :: MIT License] classifier isn't a suggestion but a
> declaration and follows the distutils guide for declaring a license.
> I consider inclusion of a license file redundant and error prone."
>
> Hopefully this resolves your issue about not having an explicit license file.

Thank you Eric and the others. I will use this information for the request.

-- 
Claude Marinier



From graffatcolmingov at gmail.com  Wed Aug 17 09:06:43 2016
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Wed, 17 Aug 2016 08:06:43 -0500
Subject: [Distutils] license for setuptools
In-Reply-To: <1AE01599B99F8A4EA0609B55B478D77FE876F9@HCXDSPM1.ca.lmco.com>
References: <1AE01599B99F8A4EA0609B55B478D77FE876F9@HCXDSPM1.ca.lmco.com>
Message-ID: <CAN-Kwu10tXQq0EKhaESV+C=gV7E26RjANk5T+gtj8mNcY=zDSw@mail.gmail.com>

On Mon, Aug 15, 2016 at 7:35 AM, Marinier, Claude
<claude.marinier at lmco.com> wrote:
>> Date: Fri, 12 Aug 2016 02:45:20 +0000
>> From: Eric Dill <thedizzle at gmail.com>
>> Subject: Re: [Distutils] license for setuptools
>>
>> Hi Claude,
>>
>> There was a recent discussion of the lack of a license file in setuptools
>> here: https://github.com/pypa/setuptools/issues/612 and another important
>> discussion here: https://github.com/pypa/setuptools/issues/132.  This is
>> probably the most relevant quotable bit from those two issues, from Jason
>> Coombs (the primary developer of setuptools:
>>
>> "The [License :: OSI Approved :: MIT License] classifier isn't a suggestion but a
>> declaration and follows the distutils guide for declaring a license.
>> I consider inclusion of a license file redundant and error prone."
>>
>> Hopefully this resolves your issue about not having an explicit license file.
>
> Thank you Eric and the others. I will use this information for the request.

Claude,

Setuptools now ships a LICENSE file. I'm not sure if there's a release
available with it, but it's now in the SCM tree.

From steve.dower at python.org  Wed Aug 17 09:23:02 2016
From: steve.dower at python.org (Steve Dower)
Date: Wed, 17 Aug 2016 06:23:02 -0700
Subject: [Distutils] Deprecating little used file types/extensions
 onPyPI?
In-Reply-To: <CADiSq7c-sfEjArH_yDEFBeJCPx1Mt_af86rOHCLbvUCJUZAQeg@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CACac1F8LSnH_7=vORDxWdcqMs2=hZ9Vi=MrQ7uF4m6tWy3Fyiw@mail.gmail.com>
 <CAG8k2+61eY685hZPxoK6H34gn1oxz1sOFC1aom_0fEDbmjhjWw@mail.gmail.com>
 <CAMxX_oOEKxNy45LgTwuQogSDoLrTLD_VUY2QHH0A8orO15Phig@mail.gmail.com>
 <CADiSq7cR9Dg6XxvTwp94hEFNNEOkbQebYi1e5Jp+Wf2H-H6-xg@mail.gmail.com>
 <CADiSq7c-sfEjArH_yDEFBeJCPx1Mt_af86rOHCLbvUCJUZAQeg@mail.gmail.com>
Message-ID: <E1ba0oa-0008VQ-UT@se2-syd.hostedmail.net.au>

Do you mean like zipapp and *.pyz files?

Top-posted from my Windows Phone

-----Original Message-----
From: "Nick Coghlan" <ncoghlan at gmail.com>
Sent: ?8/?16/?2016 22:42
To: "Leonardo Rochael Almeida" <leorochael at gmail.com>
Cc: "distutils sig" <distutils-sig at python.org>
Subject: Re: [Distutils] Deprecating little used file types/extensions onPyPI?

On 17 August 2016 at 15:21, Nick Coghlan <ncoghlan at gmail.com> wrote:
> While rebranding eggs (or a close derivative) as "pypath entries"
> could help make them more self-explanatory (especially when it comes
> to explaining how the zipped form differs from wheel files), doing so
> wouldn't actually give us any more architectural flexibility in
> practice than the last option.

That said, if folks *did* want to go down this path, then an important
use case to consider is "single archive applications" suitable for use
in services like AWS Lambda and Azure Cloud Functions, where you get a
language runtime to play with, but need to provide everything you want
to run in that environment as a single pre-bundled archive. Similar
problems also show up when building rich client applications for
Windows, Mac OS X, Android and iOS.

This means you want tooling that targets the direct dependency
bundling model, rather than the runtime environment configuration
model favoured by pip and virtualenv.

So this particular challenge *shouldn't* be framed as just a backwards
compatibility and migration concern for buildout et al - the "all
inclusive application bundle" approach is still an important use case,
even though it isn't the lowest common denominator cross-platform
dependency specification, retrieval and installation problem handled
by the default toolchain.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG at python.org
https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160817/9b3b9ab1/attachment.html>

From ncoghlan at gmail.com  Wed Aug 17 11:25:28 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 18 Aug 2016 01:25:28 +1000
Subject: [Distutils] Deprecating little used file types/extensions
 onPyPI?
In-Reply-To: <E1ba0oa-0008VP-Ny@se2-syd.hostedmail.net.au>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CACac1F8LSnH_7=vORDxWdcqMs2=hZ9Vi=MrQ7uF4m6tWy3Fyiw@mail.gmail.com>
 <CAG8k2+61eY685hZPxoK6H34gn1oxz1sOFC1aom_0fEDbmjhjWw@mail.gmail.com>
 <CAMxX_oOEKxNy45LgTwuQogSDoLrTLD_VUY2QHH0A8orO15Phig@mail.gmail.com>
 <CADiSq7cR9Dg6XxvTwp94hEFNNEOkbQebYi1e5Jp+Wf2H-H6-xg@mail.gmail.com>
 <CADiSq7c-sfEjArH_yDEFBeJCPx1Mt_af86rOHCLbvUCJUZAQeg@mail.gmail.com>
 <E1ba0oa-0008VP-Ny@se2-syd.hostedmail.net.au>
Message-ID: <CADiSq7d5Era6MGXKgyKrZ_xaruzf8kuSYWqoMK1XdtY0xWtREw@mail.gmail.com>

On 17 August 2016 at 23:23, Steve Dower <steve.dower at python.org> wrote:
> Do you mean like zipapp and *.pyz files?

Pretty much, yeah, but with a more fleshed out workflow that accounts
for dependency bundling as well - you'd have a relatively normal
Python app with a setup.py and/or requirements.txt, but be able to
readily package the whole thing up for distribution.

This kind of thing is *possible* today if you already know what you're
doing, and at least the AWS Lambda docs are pretty decent when it
comes to describing how to make it work in their (non-zipapp) usage
model: http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html
(I haven't checked the Azure Compute Function docs, and Google Compute
Function is currently JavaScript only)

However, these kinds of approaches are nowhere near as well fleshed
out from an automation perspective as traditional Linux web service
deployment - note that there's no mention in the AWS Lambda docs of
using a requirements file rather than individual dependency
installations.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From wes.turner at gmail.com  Wed Aug 17 13:49:15 2016
From: wes.turner at gmail.com (Wes Turner)
Date: Wed, 17 Aug 2016 12:49:15 -0500
Subject: [Distutils] Deprecating little used file types/extensions on PyPI?
Message-ID: <CACfEFw9ejuPURmnLvmE12yLGmeLOKJrUPi=Dac3Ec1HBsGyyLA@mail.gmail.com>

PEX - Python Executable

- It's a ZIP with an executable header and entry points.

- PEX probably solves for the AWS Lambda case
- PEX is unsupported by PyPI (e.g. sadly doesnt work with DevPi
application-level package repository caching)

http://pex.readthedocs.io/en/stable/

https://github.com/pantsbuild/pex

- PantsBuild builds PEX packages (Twitter)
- Buck Build builds PEX packages (Facebook)

Wheel adds a manifest of checksums (which should be signed, correct?)
-
https://bitbucket.org/pypa/wheel/issues/168/code-to-add-a-wheel-manifest-to-any-zip

Wheel does not solve for everything eggs still solve for. Neither are
perfect:
- https://bitbucket.org/pypa/wheel/issues?status=new&status=open
  - Wheel doesn't support environment markers in requirements.txt
  -
-
https://bitbucket.org/pypa/wheel/issues/168/code-to-add-a-wheel-manifest-to-any-zip


What is the need to reduce/simplify the supported package types? Each have
their strengths.

"Suddenly they wouldn't accept archives which execute arbitrary code
anymore "

I say support them all and don't waste time with complex deprecation and
migration strategies when the code already works.

What's it hurting to continue legacy support? (E.g., specifically,  instead
of throwing a distribution stat up and claiming need to eliminate
everything legacy or exotic,
what are the costs for each option here?

Approaches:
- deprecate legacy and goad migration
- continue supporting a diverse array of python packaging needs

Costs:
- end user developer time
- pypa developer time
- pypa server time

On Wednesday, August 17, 2016, Nick Coghlan <ncoghlan at gmail.com
<javascript:_e(%7B%7D,'cvml','ncoghlan at gmail.com');>> wrote:

> On 17 August 2016 at 23:23, Steve Dower <steve.dower at python.org> wrote:
> > Do you mean like zipapp and *.pyz files?
>
> Pretty much, yeah, but with a more fleshed out workflow that accounts
> for dependency bundling as well - you'd have a relatively normal
> Python app with a setup.py and/or requirements.txt, but be able to
> readily package the whole thing up for distribution.
>
> This kind of thing is *possible* today if you already know what you're
> doing, and at least the AWS Lambda docs are pretty decent when it
> comes to describing how to make it work in their (non-zipapp) usage
> model: http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-
> how-to-create-deployment-package.html
> (I haven't checked the Azure Compute Function docs, and Google Compute
> Function is currently JavaScript only)
>
> However, these kinds of approaches are nowhere near as well fleshed
> out from an automation perspective as traditional Linux web service
> deployment - note that there's no mention in the AWS Lambda docs of
> using a requirements file rather than individual dependency
> installations.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160817/b549cfa7/attachment.html>

From p.f.moore at gmail.com  Wed Aug 17 17:57:52 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 17 Aug 2016 22:57:52 +0100
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CACfEFw9ejuPURmnLvmE12yLGmeLOKJrUPi=Dac3Ec1HBsGyyLA@mail.gmail.com>
References: <CACfEFw9ejuPURmnLvmE12yLGmeLOKJrUPi=Dac3Ec1HBsGyyLA@mail.gmail.com>
Message-ID: <CACac1F8t9eYkctAHU5oHOXXx-VSfifDsYnAq=STPbL48to2UAA@mail.gmail.com>

On 17 August 2016 at 18:49, Wes Turner <wes.turner at gmail.com> wrote:
> PEX - Python Executable
>
> - It's a ZIP with an executable header and entry points.
>
> - PEX probably solves for the AWS Lambda case
> - PEX is unsupported by PyPI (e.g. sadly doesnt work with DevPi
> application-level package repository caching)

I'm not quite sure what point you're making here, but pex doesn't
support Windows, which is a major downside for anything intended as a
standard solution for Python (it is of course fine for the pex project
to not support Windows, but that's a different matter).

Paul

From chris.barker at noaa.gov  Wed Aug 17 18:05:28 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 17 Aug 2016 15:05:28 -0700
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAP1=2W4p75=7STAs+EAo+dFY5sQvp8Gei_dnqVjW55QO6G6v+A@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAP1=2W4p75=7STAs+EAo+dFY5sQvp8Gei_dnqVjW55QO6G6v+A@mail.gmail.com>
Message-ID: <CALGmxELCSGjF+jnnWk+R1YicaepT+FBHO7WSxG9NHhwrwgJL8g@mail.gmail.com>

On Tue, Aug 16, 2016 at 8:51 AM, Brett Cannon <brett at python.org> wrote:

> One thing to remember is that Windows can't read tar files natively while
> it can for zip files. Now you can easily download tools on Windows to read
> tar files and thanks to Bash on Windows you even have it included once you
> turn that feature on.
>
> The other point is we have a zip importer in Python but not a .tar.gz one. I
> don't know how often anyone actually downloads a zip file directly from
> PyPI and then tack it on to their sys.path for importing, but that is
> currently possible.
>
> I doubt either of these points are important enough to continue to support
> zip files for sdists,
>

I do -- what advantage does tar.gz have over zip? Zip's always been
supported on any platform I've used...

But both are in python's stdlib, so I suppose everyone has it that has
Python.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160817/cb58a440/attachment.html>

From chris.barker at noaa.gov  Wed Aug 17 18:06:33 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 17 Aug 2016 15:06:33 -0700
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <C4CDB2B3-4600-478E-8230-438A6B6F3261@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CACac1F8LSnH_7=vORDxWdcqMs2=hZ9Vi=MrQ7uF4m6tWy3Fyiw@mail.gmail.com>
 <CAG8k2+61eY685hZPxoK6H34gn1oxz1sOFC1aom_0fEDbmjhjWw@mail.gmail.com>
 <CAMxX_oOEKxNy45LgTwuQogSDoLrTLD_VUY2QHH0A8orO15Phig@mail.gmail.com>
 <C4CDB2B3-4600-478E-8230-438A6B6F3261@stufft.io>
Message-ID: <CALGmxE+cujN1OrJdQOxcRx2JgaJjMMVihTffxLy6X1+wG5D0zg@mail.gmail.com>

On Tue, Aug 16, 2016 at 9:10 AM, Donald Stufft <donald at stufft.io> wrote:
>
> Ah, I knew I was forgetting something. I think we should hold off on
> preventing egg uploads until setuptools can download and install wheels.
>

Aren't we trying to get to the point where setuptools doesn't download and
install anything ?

Sigh.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160817/b0aa953c/attachment-0001.html>

From donald at stufft.io  Wed Aug 17 18:08:17 2016
From: donald at stufft.io (Donald Stufft)
Date: Wed, 17 Aug 2016 18:08:17 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CALGmxELCSGjF+jnnWk+R1YicaepT+FBHO7WSxG9NHhwrwgJL8g@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAP1=2W4p75=7STAs+EAo+dFY5sQvp8Gei_dnqVjW55QO6G6v+A@mail.gmail.com>
 <CALGmxELCSGjF+jnnWk+R1YicaepT+FBHO7WSxG9NHhwrwgJL8g@mail.gmail.com>
Message-ID: <35AFECEF-EED5-4ED9-BD4B-9B6CBF303646@stufft.io>


> On Aug 17, 2016, at 6:05 PM, Chris Barker <chris.barker at noaa.gov> wrote:
> 
> I do -- what advantage does tar.gz have over zip?



The two formats are roughly isomorphic in terms of technical reasons to pick one over the other, but distutils defaults to .tar.gz and most uploads to PyPI are .tar.gz.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160817/0edfd634/attachment.html>

From chris.barker at noaa.gov  Wed Aug 17 18:18:47 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 17 Aug 2016 15:18:47 -0700
Subject: [Distutils] Future of setuptools and buildout
In-Reply-To: <EED02033-FB4E-4B25-A37B-C09B4CDF5F31@stufft.io>
References: <CAMxX_oP-8R=Gr4Jp9P8CXNSB7muuq=2z8E0K_V7qCvFodJTJOQ@mail.gmail.com>
 <EED02033-FB4E-4B25-A37B-C09B4CDF5F31@stufft.io>
Message-ID: <CALGmxEJZBb=eakpUPWwqy+Wa=K+EoCvqi+sPgf4cPx-4+1m0yA@mail.gmail.com>

>
> Which brings us to a question that I'm meaning to ask for a while.
>
> It looks like we're close to removing all mentions of setuptools in pip.
> When this happens, it looks like pressure is going to start to mount on
> setuptools to drop the ability to install packages and limit itself on
> being just a build tool.
>
>
Yes, please!

I argued  a while back for a setuptools-lite -- it would do all the things
setuptools does that we think it _should_ do, and not do any of the others
-- like it wouldn't easy-install anything EVER.

most likely it would be a fork of the setuptools code with a bunch of stuff
disabled. Or maybe even a setting in setuptools itself:

import setuptools
setuptools.disable_legacy

from setuptools import setup, find_packages, ...

Why????

Folks rely on the various "features" of setuptools, some important ones
like buildout. So it's going to be a long time before we can deprecate all
that in setuptools itself.

But we're trying for a future with better separation of concerns --
building, installing, packaging, run-time management.

But as it stands now, people kind of HAVE to use setuptools to get desired
build behaviour, but then their users an accidentally invoke features they
don't want -- to teh point where pip goes in and monkey patches the darn
thing.

IF there were a setuptools_lite, user could simply do:

import setuptools_lite as setuptools

and they'd instantly get an error if they were using depreciated features,
and their end users would never accidently easy install stuff :-)

Anyway, this seem like a path forward, without having to wait for the
future fabulous pluggable build system .....

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160817/8768b1d5/attachment.html>

From dholth at gmail.com  Wed Aug 17 21:12:54 2016
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 18 Aug 2016 01:12:54 +0000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <35AFECEF-EED5-4ED9-BD4B-9B6CBF303646@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAP1=2W4p75=7STAs+EAo+dFY5sQvp8Gei_dnqVjW55QO6G6v+A@mail.gmail.com>
 <CALGmxELCSGjF+jnnWk+R1YicaepT+FBHO7WSxG9NHhwrwgJL8g@mail.gmail.com>
 <35AFECEF-EED5-4ED9-BD4B-9B6CBF303646@stufft.io>
Message-ID: <CAG8k2+6q9s=MjZ-0_bzfwCOxvxVkTz-_yh+j4L6jXnx+d-w9-Q@mail.gmail.com>

Now I remember that just the other week I had to switch a project to zip
sdists, because I was having trouble doing .tar.gz on Windows. Perhaps the
greater availability of tools to deal with zip is the reason why 10% of
sdists choose that format.

On Wed, Aug 17, 2016 at 6:08 PM Donald Stufft <donald at stufft.io> wrote:

>
> On Aug 17, 2016, at 6:05 PM, Chris Barker <chris.barker at noaa.gov> wrote:
>
> I do -- what advantage does tar.gz have over zip?
>
>
>
> The two formats are roughly isomorphic in terms of technical reasons to
> pick one over the other, but distutils defaults to .tar.gz and most uploads
> to PyPI are .tar.gz.
>
> ?
>
> Donald Stufft
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160818/096f99ba/attachment.html>

From dholth at gmail.com  Wed Aug 17 21:45:00 2016
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 18 Aug 2016 01:45:00 +0000
Subject: [Distutils] Future of setuptools and buildout
In-Reply-To: <CALGmxEJZBb=eakpUPWwqy+Wa=K+EoCvqi+sPgf4cPx-4+1m0yA@mail.gmail.com>
References: <CAMxX_oP-8R=Gr4Jp9P8CXNSB7muuq=2z8E0K_V7qCvFodJTJOQ@mail.gmail.com>
 <EED02033-FB4E-4B25-A37B-C09B4CDF5F31@stufft.io>
 <CALGmxEJZBb=eakpUPWwqy+Wa=K+EoCvqi+sPgf4cPx-4+1m0yA@mail.gmail.com>
Message-ID: <CAG8k2+7_DbueMsrokPMwbrzhzq8-M2t8QSOZN1-O3PJKvCh17Q@mail.gmail.com>

And a while back I argued against setuptools-lite, because I thought it
would not solve the poor extensibility problem that stems from its basic
distutils derived design... which includes all the classes and subclasses
that have to work together to do its thing. My own thinking is that
setuptools is fine, except that when you need to extend it you have a big
problem; so make it easy to replace.

For example, setuptools & distutils can't configure the compiler for
building extensions, without literally calling build_ext.run(). How
inconvenient.

Does pip monkey patch setuptools? I thought it just forced setuptools even
if a package only asked for distutils.

So you are suggesting setuptools_lite would not honor the setup_requires =
[] and test_requires = [] and perhaps "setup.py install" would also throw
an error? What else? I would suggest that to effectively argue the idea
requires reading the source code and enumerating exactly which complexity
goes away with the _lite version.

One "lite" version of setuptools that I personally think would be cool
would be pkg_resources as a separate package on pypi. Setuptools would
require it as a dependency.

My own setuptools-free C extension compiling demo continues to be pip
install -e hg+https://bitbucket.org/dholth/cryptacular#egg=cryptacular , in
a virtualenv.

On Wed, Aug 17, 2016 at 6:19 PM Chris Barker <chris.barker at noaa.gov> wrote:

> Which brings us to a question that I'm meaning to ask for a while.
>>
>> It looks like we're close to removing all mentions of setuptools in pip.
>> When this happens, it looks like pressure is going to start to mount on
>> setuptools to drop the ability to install packages and limit itself on
>> being just a build tool.
>>
>>
> Yes, please!
>
> I argued  a while back for a setuptools-lite -- it would do all the things
> setuptools does that we think it _should_ do, and not do any of the others
> -- like it wouldn't easy-install anything EVER.
>
> most likely it would be a fork of the setuptools code with a bunch of
> stuff disabled. Or maybe even a setting in setuptools itself:
>
> import setuptools
> setuptools.disable_legacy
>
> from setuptools import setup, find_packages, ...
>
> Why????
>
> Folks rely on the various "features" of setuptools, some important ones
> like buildout. So it's going to be a long time before we can deprecate all
> that in setuptools itself.
>
> But we're trying for a future with better separation of concerns --
> building, installing, packaging, run-time management.
>
> But as it stands now, people kind of HAVE to use setuptools to get desired
> build behaviour, but then their users an accidentally invoke features they
> don't want -- to teh point where pip goes in and monkey patches the darn
> thing.
>
> IF there were a setuptools_lite, user could simply do:
>
> import setuptools_lite as setuptools
>
> and they'd instantly get an error if they were using depreciated features,
> and their end users would never accidently easy install stuff :-)
>
> Anyway, this seem like a path forward, without having to wait for the
> future fabulous pluggable build system .....
>
> -Chris
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160818/7294b5ef/attachment-0001.html>

From ncoghlan at gmail.com  Wed Aug 17 21:51:56 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 18 Aug 2016 11:51:56 +1000
Subject: [Distutils] Future of setuptools and buildout
In-Reply-To: <CALGmxEJZBb=eakpUPWwqy+Wa=K+EoCvqi+sPgf4cPx-4+1m0yA@mail.gmail.com>
References: <CAMxX_oP-8R=Gr4Jp9P8CXNSB7muuq=2z8E0K_V7qCvFodJTJOQ@mail.gmail.com>
 <EED02033-FB4E-4B25-A37B-C09B4CDF5F31@stufft.io>
 <CALGmxEJZBb=eakpUPWwqy+Wa=K+EoCvqi+sPgf4cPx-4+1m0yA@mail.gmail.com>
Message-ID: <CADiSq7fD4bDRh4Ubx=LpgOV1CacjZWMLSYRF0vG4gdV7eHdqPg@mail.gmail.com>

On 18 August 2016 at 08:18, Chris Barker <chris.barker at noaa.gov> wrote:
> IF there were a setuptools_lite, user could simply do:
>
> import setuptools_lite as setuptools
>
> and they'd instantly get an error if they were using depreciated features,
> and their end users would never accidently easy install stuff :-)
>
> Anyway, this seem like a path forward, without having to wait for the future
> fabulous pluggable build system .....

The problem with this is a pragmatic one rather than a philosophical
one, in that setuptools is just plain *hard* to work on, and doesn't
have a particularly robust test suite, so you can easily break other
people's usage while trying to fix the particular issue you personally
care about.

Hence the ongoing efforts to let people more easily use tools that
*aren't* setuptools, rather than attempting to disentangle and
modularise setuptools itself - the architectural challenges are such
that attempting to fix them would inevitably break existing usage,
while we can get most of the same benefits more transparently through
things like declarative dependencies for setup.py execution.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed Aug 17 22:10:48 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 18 Aug 2016 12:10:48 +1000
Subject: [Distutils] Future of setuptools and buildout
In-Reply-To: <CAG8k2+7_DbueMsrokPMwbrzhzq8-M2t8QSOZN1-O3PJKvCh17Q@mail.gmail.com>
References: <CAMxX_oP-8R=Gr4Jp9P8CXNSB7muuq=2z8E0K_V7qCvFodJTJOQ@mail.gmail.com>
 <EED02033-FB4E-4B25-A37B-C09B4CDF5F31@stufft.io>
 <CALGmxEJZBb=eakpUPWwqy+Wa=K+EoCvqi+sPgf4cPx-4+1m0yA@mail.gmail.com>
 <CAG8k2+7_DbueMsrokPMwbrzhzq8-M2t8QSOZN1-O3PJKvCh17Q@mail.gmail.com>
Message-ID: <CADiSq7emcK5s=-CtoNEdQpy4FZjv7O8HYEgcYrCK4ftuyTxNzw@mail.gmail.com>

On 18 August 2016 at 11:45, Daniel Holth <dholth at gmail.com> wrote:
> One "lite" version of setuptools that I personally think would be cool would
> be pkg_resources as a separate package on pypi. Setuptools would require it
> as a dependency.

I finally managed to relocate a discussion Jason, Donald and I had
about this a couple of months ago:
https://github.com/ncoghlan/pkg_resources_shim/issues/1

(The idea there was to make "pip install pkg_resources" just work by
implicitly installing setuptools - at the moment running that command
errors out, since Donald reserved the name on PyPI to prevent anyone
using it as an attack vector against folks actually looking for
setuptools)

Creating a standalone "pkg_resources" project turns out to be
relatively straightforward, but *removing* pkg_resources from
setuptools turns out to create significant setuptools bootstrapping
problems. That means the standalone project would need to use a
different name, and any associated bugs would subsequently need to be
fixed in two different projects, which seems like a bad idea.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From barry at python.org  Thu Aug 18 17:53:13 2016
From: barry at python.org (Barry Warsaw)
Date: Thu, 18 Aug 2016 17:53:13 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CACac1F8LSnH_7=vORDxWdcqMs2=hZ9Vi=MrQ7uF4m6tWy3Fyiw@mail.gmail.com>
 <CAG8k2+61eY685hZPxoK6H34gn1oxz1sOFC1aom_0fEDbmjhjWw@mail.gmail.com>
 <CAMxX_oOEKxNy45LgTwuQogSDoLrTLD_VUY2QHH0A8orO15Phig@mail.gmail.com>
 <CADiSq7cR9Dg6XxvTwp94hEFNNEOkbQebYi1e5Jp+Wf2H-H6-xg@mail.gmail.com>
Message-ID: <20160818175313.107ce103@anarchist.wooz.org>

On Aug 17, 2016, at 03:21 PM, Nick Coghlan wrote:

>This means that the situation we've managed to get to now is that we
>have wheels as a satisfactory replacement for the "binary
>distribution" use case, but we haven't even *tried* to replace eggs
>for the "standalone path entry" use case.

I'm very sure I don't understand something, because I thought wheels were just
fine for the import-from-sys.path use case.  I mean, pip does this and in
Debian, we have a program (dirtbike) that turns installed package's file
system layouts back into wheels so they can be put on sys.path and imported
from.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160818/835b70ba/attachment.sig>

From chris.barker at noaa.gov  Thu Aug 18 17:54:58 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 18 Aug 2016 14:54:58 -0700
Subject: [Distutils] Future of setuptools and buildout
In-Reply-To: <CAG8k2+7_DbueMsrokPMwbrzhzq8-M2t8QSOZN1-O3PJKvCh17Q@mail.gmail.com>
References: <CAMxX_oP-8R=Gr4Jp9P8CXNSB7muuq=2z8E0K_V7qCvFodJTJOQ@mail.gmail.com>
 <EED02033-FB4E-4B25-A37B-C09B4CDF5F31@stufft.io>
 <CALGmxEJZBb=eakpUPWwqy+Wa=K+EoCvqi+sPgf4cPx-4+1m0yA@mail.gmail.com>
 <CAG8k2+7_DbueMsrokPMwbrzhzq8-M2t8QSOZN1-O3PJKvCh17Q@mail.gmail.com>
Message-ID: <CALGmxE+mMuJ8P5PP9bbSuUQEx06PwH3hEdo7aYr_RBbY3_NZ3g@mail.gmail.com>

On Wed, Aug 17, 2016 at 6:45 PM, Daniel Holth <dholth at gmail.com> wrote:

> And a while back I argued against setuptools-lite, because I thought it
> would not solve the poor extensibility problem that stems from its basic
> distutils derived design... which includes all the classes and subclasses
> that have to work together to do its thing. My own thinking is that
> setuptools is fine, except that when you need to extend it you have a big
> problem; so make it easy to replace.
>

I never had any fantasies that setuptool-lite would improve on this in a
way shape or form -- indeed, a better build system is likely to need to be
built from scratch, or on a totally different framework, anyway.


> Does pip monkey patch setuptools? I thought it just forced setuptools even
> if a package only asked for distutils.
>

good point -- not really a monkey patch, but it does force particular
configuration.


> So you are suggesting setuptools_lite would not honor the setup_requires =
> [] and test_requires = []
>

essentially, yes. Though It would be nice if that data could be available
for pip -- and I"m not sure how that is done at this point.


> and perhaps "setup.py install" would also throw an error?
>

Probably not -- but it could only install from local source -- like
distutils. Though if we could re-work it to only do develop mode and build
wheels, that may be good -- you'd need pip to install it.


> What else? I would suggest that to effectively argue the idea requires
> reading the source code and enumerating exactly which complexity goes away
> with the _lite version.
>

I don't know that reading the source is the issue -- I'm thinking entirely
in terms of stripping out the API, so there may be things that need to stay
in, but we don't expose them. though maybe even that's harder than I think.

But yes -- enumerating what would and wouldn't be supported would be a good
start. But here is the guideline:

It seems we have a (maybe vague) vision of what we want the tool ecosystem
to look like that involves separation of concerns, such that each component
is replaceable without having to change the others:

The concerns:
  - building a package
  - installing a package
  - dependency management
  - run-time version management
  - plugin handling, etc.
  - resource managment

setuptools does all of these, yes? but think of these in terms of when they
come into play:

build time:
   - building a package
install time: installing a package;
  - installing a package
  - dependency management
run time
  - run-time version management
  - plugin handling, etc.

so we have:

 - The mythical build tool
 - pip
 - maybe some of the setuptools spun off -- like pkg_resources

So why aren't we there yet?

One reason is that setuptools the only thing out there that does the
building and some of the run-time stuff. So people use it. And now you
can't replace any of it without making a mess.

However, pip_wheel have indeed addressed much of the install-time and
dependency management issues. So folks use them, and they work well.

So what problem do I want setuptool-lite to solve?

I actually have spent a lot of time using conda to do my package installing
and dependency management. So I want a way to build a python package that
JUST builds the package.

In practice, I've found (most) of the flags and work arounds to diable
etuptools "features", but really?

python setup.py install --single-version-externally-managed

Oh right, that's not enough, I need a record file:

setup.py install --single-version-externally-managed  --record record.txt

Then I want to install in develop mode:

python setup.py develop

OOPS! now it's easy installing a bunch of dependencies! arrgg!

OK, -- no-deps it is.

Google around, this kind of thing is a pain in a lot of people's butts
--even though it call all be worked around.

So my vision of setuptool-lite is that is simply does not do the things
that vision of a build tool should not do.

i.e.:

never easy-install anything!
don't try to resolve dependencies

others??

One "lite" version of setuptools that I personally think would be cool
> would be pkg_resources as a separate package on pypi. Setuptools would
> require it as a dependency.
>

Yes, I'd love to disable pkg_resources too, but couldn't until it was spun
off.

In fact, years ago, long before pip, setuptools drove me crazy with how it
combined run-time, build time and intall time stuff. A serious problem when
I wanted to bundle something up with py2app (or py2exe, or...).

So what would be accomplished?

package builders could simply put:

from setuptools_lite import setup

(etc.)

and then their user would never accidentally easy install anything, etc.
And it would serve as documentation / notification if a user was using a
what-should-be-deprecated feature.

By the way, this is totally acknowledged to be a hacky, temporary mess that
might only help a bit to get us closer from being locked into setuptools.

(though maybe it would be a place to work through some of the issues with a
build API)

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160818/920b832f/attachment.html>

From donald at stufft.io  Thu Aug 18 17:58:21 2016
From: donald at stufft.io (Donald Stufft)
Date: Thu, 18 Aug 2016 17:58:21 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <20160818175313.107ce103@anarchist.wooz.org>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CACac1F8LSnH_7=vORDxWdcqMs2=hZ9Vi=MrQ7uF4m6tWy3Fyiw@mail.gmail.com>
 <CAG8k2+61eY685hZPxoK6H34gn1oxz1sOFC1aom_0fEDbmjhjWw@mail.gmail.com>
 <CAMxX_oOEKxNy45LgTwuQogSDoLrTLD_VUY2QHH0A8orO15Phig@mail.gmail.com>
 <CADiSq7cR9Dg6XxvTwp94hEFNNEOkbQebYi1e5Jp+Wf2H-H6-xg@mail.gmail.com>
 <20160818175313.107ce103@anarchist.wooz.org>
Message-ID: <497B82D9-2452-4AF9-8CC4-9BE76F423A82@stufft.io>


> On Aug 18, 2016, at 5:53 PM, Barry Warsaw <barry at python.org> wrote:
> 
> On Aug 17, 2016, at 03:21 PM, Nick Coghlan wrote:
> 
>> This means that the situation we've managed to get to now is that we
>> have wheels as a satisfactory replacement for the "binary
>> distribution" use case, but we haven't even *tried* to replace eggs
>> for the "standalone path entry" use case.
> 
> I'm very sure I don't understand something, because I thought wheels were just
> fine for the import-from-sys.path use case.  I mean, pip does this and in
> Debian, we have a program (dirtbike) that turns installed package's file
> system layouts back into wheels so they can be put on sys.path and imported
> from.


It ?Works?, sometimes, if the thing in the wheel doesn?t do something that
might cause that to break. Thus it will work sometimes, but often times it
won?t. It?s not an officially supported feature, but more of a thing where
some decisions were made not to preclude it from working entirely.

As an example, your Debian dirtbike?d pip only actually fully works because
you?ve patched requests. If you didn?t have the patches you?ve applied to
requests, it wouldn?t work. Additionally, pip only works at all because we?ve
been careful to *only* depends on pure Python items that generally work when
zipped (besides the requests thing, which we end up working around in get-pip.py
and just flat out avoiding in ensurepip).

Basically, we don?t go out of our way to prevent it, but if you do it and it
breaks, you get to keep both pieces.

?
Donald Stufft




From njs at pobox.com  Thu Aug 18 18:10:43 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 18 Aug 2016 15:10:43 -0700
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CADiSq7cR9Dg6XxvTwp94hEFNNEOkbQebYi1e5Jp+Wf2H-H6-xg@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CACac1F8LSnH_7=vORDxWdcqMs2=hZ9Vi=MrQ7uF4m6tWy3Fyiw@mail.gmail.com>
 <CAG8k2+61eY685hZPxoK6H34gn1oxz1sOFC1aom_0fEDbmjhjWw@mail.gmail.com>
 <CAMxX_oOEKxNy45LgTwuQogSDoLrTLD_VUY2QHH0A8orO15Phig@mail.gmail.com>
 <CADiSq7cR9Dg6XxvTwp94hEFNNEOkbQebYi1e5Jp+Wf2H-H6-xg@mail.gmail.com>
Message-ID: <CAPJVwBkG5O1dyN+tZiGdbHVip9ihYEaQSB+d+M0Qe4m=pnUA5Q@mail.gmail.com>

On Tue, Aug 16, 2016 at 10:21 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 17 August 2016 at 01:15, Leonardo Rochael Almeida
> <leorochael at gmail.com> wrote:
>> PS: In the buildout community, we never really understood the impetus for
>> replacing egg as a format, which is really not all that complex and not all
>> the different from wheel as it stands, instead of just formalizing it in a
>> PEP. We got the feeling that, in the movement of getting rid of setuptools
>> as an "installation" dependency, we ended up throwing out the baby with the
>> bathwater...
>
> The majority of the problems with eggs are due to the fact that they
> were designed to operate *by default* as the plugin management system
> for Chandler. buildout uses them in a similar way, so they're a good
> fit there.
>
> However, they're *not* a good fit for other cases, hence
> "--single-version-externally-managed" and all of the other ways in
> which pip switches the defaults to be more developer centric, and less
> deployment focused. Most of the problems people ran into with the
> defaults were actually related to the way easy_install and
> pkg_resources worked rather than due to the egg format itself, but the
> lack of clear documentation and specifications meant that the broader
> adoption process for programmatic dependency management in general
> needed to be based on a more modular reimplementation of similar ideas
> with more explicit interoperability interfaces, rather than adopting
> the setuptools/easy_install/pkg_resources implementation of those
> ideas wholesale.
>
> This means that the situation we've managed to get to now is that we
> have wheels as a satisfactory replacement for the "binary
> distribution" use case, but we haven't even *tried* to replace eggs
> for the "standalone path entry" use case.
>
> That gives us a few options for possible ways forward:
>
> - define a new egg-inspired "*.pypath" format for directories and zip
> files that are designed to be placed directly on sys.path, rather than
> installed
> - just bless eggs as the recommended standalone importable format, and
> offer a wheel2egg conversion utility and perhaps an "--as-egg" install
> option in pip
> - as with the previous, but bake the wheel2egg conversion into
> setuptools/easy_install rather than into pip

I'm not at all familiar with buildout's internals, but from the
description upthread it sounds like it already has to have fairly
intimate knowledge of and control over the layout of the directory
it's managing (so that it can do things like generate sys.path
manipulation code inside console scripts, etc).

Would it make sense to declare that buildout has its own directory
layout and conventions that it uses (which happen to be suspiciously
similar to the legacy .egg format), and that buildout is responsible
for figuring out how to consume standardized formats like .whl and put
them into the layout it wants? I don't think there's much appetite for
trying to standardize .egg, and the advantage is that it moves control
of the format to the people who actually use it -- distutils-sig is
not really the best place to be trying to do anything with eggs at
this point AFAICT. It might even make sense to split out some bits of
setuptools into buildout.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From njs at pobox.com  Thu Aug 18 18:12:59 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 18 Aug 2016 15:12:59 -0700
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
Message-ID: <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>

On Mon, Aug 15, 2016 at 5:02 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> *  new project by existing maintainer: here, I'd be inclined to flag
> maintainer accounts at the start of the deprecation period based on whether
> or not they're currently using the legacy formats on any of their projects -
> that is, the migration period would be applied at the *user* level, in
> addition to the project level. If someone has never used the legacy formats,
> they can be treated like a new user immediately, and will presumably never
> even notice the change.

Like any invisible state, there's some potential for confusion here --
"how come I can upload these files but my co-maintainer can't?" Not
necessarily a fatal flaw, but something to be aware of.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From ncoghlan at gmail.com  Thu Aug 18 21:25:34 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 19 Aug 2016 11:25:34 +1000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <497B82D9-2452-4AF9-8CC4-9BE76F423A82@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CACac1F8LSnH_7=vORDxWdcqMs2=hZ9Vi=MrQ7uF4m6tWy3Fyiw@mail.gmail.com>
 <CAG8k2+61eY685hZPxoK6H34gn1oxz1sOFC1aom_0fEDbmjhjWw@mail.gmail.com>
 <CAMxX_oOEKxNy45LgTwuQogSDoLrTLD_VUY2QHH0A8orO15Phig@mail.gmail.com>
 <CADiSq7cR9Dg6XxvTwp94hEFNNEOkbQebYi1e5Jp+Wf2H-H6-xg@mail.gmail.com>
 <20160818175313.107ce103@anarchist.wooz.org>
 <497B82D9-2452-4AF9-8CC4-9BE76F423A82@stufft.io>
Message-ID: <CADiSq7fafki9-j1T9x88CECM4beQZ7ORH=MVWf4V5yiF11+05A@mail.gmail.com>

On 19 August 2016 at 07:58, Donald Stufft <donald at stufft.io> wrote:
>> On Aug 18, 2016, at 5:53 PM, Barry Warsaw <barry at python.org> wrote:
>> I'm very sure I don't understand something, because I thought wheels were just
>> fine for the import-from-sys.path use case.  I mean, pip does this and in
>> Debian, we have a program (dirtbike) that turns installed package's file
>> system layouts back into wheels so they can be put on sys.path and imported
>> from.
>
> It ?Works?, sometimes, if the thing in the wheel doesn?t do something that
> might cause that to break. Thus it will work sometimes, but often times it
> won?t. It?s not an officially supported feature, but more of a thing where
> some decisions were made not to preclude it from working entirely.
>
> As an example, your Debian dirtbike?d pip only actually fully works because
> you?ve patched requests. If you didn?t have the patches you?ve applied to
> requests, it wouldn?t work. Additionally, pip only works at all because we?ve
> been careful to *only* depends on pure Python items that generally work when
> zipped (besides the requests thing, which we end up working around in get-pip.py
> and just flat out avoiding in ensurepip).
>
> Basically, we don?t go out of our way to prevent it, but if you do it and it
> breaks, you get to keep both pieces.

+1

There's actually a somewhat analogous capability in the standard
library's test suite: test.support.import_fresh_module

This is a way of doing imports that effectively bypasses the normal
sys.modules cache, which lets us test both pure Python implementations
and C accelerator modules in the same run of the test suite.

It's also a marvelous foot gun that can mess up your Python process
state in a spectacularly hard to debug way if you don't know *exactly*
what you're doing.

However, it's just Python code, so anyone that wants to can go look at
how we do it, and do the same thing themselves, they're just on their
own when it comes to figuring out the precise boundaries of what does
and doesn't work, and which modules are written in such a way that
they will tolerate that kind of handling :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Thu Aug 18 21:40:19 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 19 Aug 2016 11:40:19 +1000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
Message-ID: <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>

On 19 August 2016 at 08:12, Nathaniel Smith <njs at pobox.com> wrote:
> On Mon, Aug 15, 2016 at 5:02 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> *  new project by existing maintainer: here, I'd be inclined to flag
>> maintainer accounts at the start of the deprecation period based on whether
>> or not they're currently using the legacy formats on any of their projects -
>> that is, the migration period would be applied at the *user* level, in
>> addition to the project level. If someone has never used the legacy formats,
>> they can be treated like a new user immediately, and will presumably never
>> even notice the change.
>
> Like any invisible state, there's some potential for confusion here --
> "how come I can upload these files but my co-maintainer can't?" Not
> necessarily a fatal flaw, but something to be aware of.

I wasn't suggesting the state would be invisible, more something akin
to the PEP 438 settings:

- existing projects with no legacy formats uploaded get "No legacy
formats" flagged
- existing projects with legacy formats uploaded get "Legacy formats" flagged
- existing maintainers with projects in the second category get a "Can
enable legacy formats" flag

By default, new projects would have the "No legacy formats" flag
(regardless of who created them), but existing maintainers with the
"Can enable legacy formats" marker would be able to change that. Once
a maintainer flipped the setting for a given project, then anyone
maintaining that project could upload the legacy formats, but wouldn't
automatically gain the ability to change the setting on other
projects.

This approach would mean we could be fairly aggressive in pushing
*new* users towards the newer formats (since there'd be a sharp
historical line beyond which new accounts simply can't enable the
legacy formats for their projects), *without* breaking the workflows
of existing projects and maintainers (at least, not yet).

Alternatively, we could simply not worry about a user level flag, and
just have a project level flag that's set to "No legacy formats" by
default for new projects - new users won't have any incentive to
change it, while existing users can change it (at least for the time
being) if that suits their workflow.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From wes.turner at gmail.com  Fri Aug 19 00:29:43 2016
From: wes.turner at gmail.com (Wes Turner)
Date: Thu, 18 Aug 2016 23:29:43 -0500
Subject: [Distutils] Future of setuptools and buildout
In-Reply-To: <CAG8k2+7z3mpev62SLM9VXwh8KZekGLtSYc-vZ+Nk+Ui=2sJ+7A@mail.gmail.com>
References: <CAMxX_oP-8R=Gr4Jp9P8CXNSB7muuq=2z8E0K_V7qCvFodJTJOQ@mail.gmail.com>
 <CAG8k2+7z3mpev62SLM9VXwh8KZekGLtSYc-vZ+Nk+Ui=2sJ+7A@mail.gmail.com>
Message-ID: <CACfEFw9i__SXM2G-r6R+XUnU4bn-vmv74Q0mbGS6D=Vc_SKBTQ@mail.gmail.com>

Re: buildout and pip and wheel

"Add support for installing wheels"
https://github.com/buildout/buildout/issues/144

It's been awhile since I've worked with buildout (for Zope 2, Plone,
AppEngine zipimports).

This reads #egg= links from pip requirements files:

-
https://github.com/collective/collective.recipe.pip/blob/master/collective/recipe/pip/__init__.py

This installs packages with buildout and pip (instead of zc.recipe.egg)
into the eggs/ directory. IDK if it supports wheels? (it specifies --egg):

-
https://github.com/k4ml/mk.recipe.pip/blob/master/mk/recipe/pip/__init__.py

... this builds a usable merged namespace with symlinks to eggs:

- https://github.com/collective/collective.recipe.omelette/


On Tuesday, August 16, 2016, Daniel Holth <dholth at gmail.com> wrote:

> There are two proposals to add pluggable build systems to Python sdists,
> and one of them will probably be implemented. You add a pyproject.toml to
> the root of your sdist, which the installer uses to install dependencies
> that setup.py itself needs to run. Second, you also tell pip which build
> system you are using, once you have done that, setup.py is no longer
> necessary - instead, pip could install and then invoke flit for example.
>
> This isn't quite the situation you've outlined as setuptools will likely
> continue to have all its features. Instead, packages that don't use
> setuptools will also proliferate. It causes about the same problem for
> buildout without breaking older packages.
>
> In this scenario buildout will need to be able to install wheels however
> it wishes to do so because that is the output of the pluggable build
> system. I don't have advice for you on the implementation, except perhaps
> it could call out to pip.
>
> On Tue, Aug 16, 2016 at 12:59 PM Leonardo Rochael Almeida <
> leorochael at gmail.com
> <javascript:_e(%7B%7D,'cvml','leorochael at gmail.com');>> wrote:
>
>> Spinning this off into its own thread...
>>
>> On 16 August 2016 at 13:10, Donald Stufft <donald at stufft.io
>> <javascript:_e(%7B%7D,'cvml','donald at stufft.io');>> wrote:
>>
>>>
>>> On Aug 16, 2016, at 11:15 AM, Leonardo Rochael Almeida <
>>> leorochael at gmail.com
>>> <javascript:_e(%7B%7D,'cvml','leorochael at gmail.com');>> wrote:
>>>
>>> Specifically, buildout right now has setuptools as its only dependence,
>>> and buildout uses it to locate, download and build sdist dependencies, or
>>> directly download and use .egg dependencies, which are important for
>>> Windows platforms in the communities that use buildout and depend on
>>> compiled extensions.
>>>
>>>
>>>
>>> Ah, I knew I was forgetting something. I think we should hold off on
>>> preventing egg uploads until setuptools can download and install wheels.
>>>
>>
>> Which brings us to a question that I'm meaning to ask for a while.
>>
>> It looks like we're close to removing all mentions of setuptools in pip.
>> When this happens, it looks like pressure is going to start to mount on
>> setuptools to drop the ability to install packages and limit itself on
>> being just a build tool.
>>
>> It makes sense that it should be so, considering that there would be no
>> incentive to keep around two distinct implementations of how to locate,
>> download and install stuff, one in pip (or whatever library pip uses for
>> locating and installing packages) and another in setuptools.
>>
>> In this future, I can imagine setuptools will simply complain loudly when
>> the packages it requires to do its work are not installed yet, perhaps with
>> a plugin mechanism for allowing another package to automatically resolve
>> and installing packages, (like today setuptools-git extends setuptools to
>> recognize files from git to be included in a package).
>>
>> And people will promptly write a pip implementation of this plugin (or
>> whatever it is that pip will use as a library to do it)... thereby
>> inverting the dependency between setuptools and pip.
>>
>> When this future arrives, buildout will be out of a mechanism for
>> locating, downloading and installing packages (not counting old versions of
>> setuptools, of course).
>>
>> Questions:
>>
>> 1. Is the future outlined above considered likely?
>>
>> 2. If this future arrives, what package should buildout be ported to use?
>> It should provide:
>>
>>  * The ability to locate packages and their versions, resolve their
>> dependencies, and download them, whether they're sdists or wheels (or eggs,
>> while we still have them)
>>  * Install each package separately into it's own directory that, if added
>> to `sys.path`, allows this same package (or perhaps another one) to read
>> information from entry points so that buildout can locate its own
>> extensions and install console scripts.
>>
>> Regards,
>>
>> Leo
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> <javascript:_e(%7B%7D,'cvml','Distutils-SIG at python.org');>
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160818/26c9710a/attachment.html>

From wes.turner at gmail.com  Fri Aug 19 01:17:19 2016
From: wes.turner at gmail.com (Wes Turner)
Date: Fri, 19 Aug 2016 00:17:19 -0500
Subject: [Distutils] Future of setuptools and buildout
In-Reply-To: <CALGmxE+mMuJ8P5PP9bbSuUQEx06PwH3hEdo7aYr_RBbY3_NZ3g@mail.gmail.com>
References: <CAMxX_oP-8R=Gr4Jp9P8CXNSB7muuq=2z8E0K_V7qCvFodJTJOQ@mail.gmail.com>
 <EED02033-FB4E-4B25-A37B-C09B4CDF5F31@stufft.io>
 <CALGmxEJZBb=eakpUPWwqy+Wa=K+EoCvqi+sPgf4cPx-4+1m0yA@mail.gmail.com>
 <CAG8k2+7_DbueMsrokPMwbrzhzq8-M2t8QSOZN1-O3PJKvCh17Q@mail.gmail.com>
 <CALGmxE+mMuJ8P5PP9bbSuUQEx06PwH3hEdo7aYr_RBbY3_NZ3g@mail.gmail.com>
Message-ID: <CACfEFw9aPsX-+yHkNBmbXfUkWkC-7ayZcQyOLWgm+3w14GYhkg@mail.gmail.com>

On Thu, Aug 18, 2016 at 4:54 PM, Chris Barker <chris.barker at noaa.gov> wrote:

> On Wed, Aug 17, 2016 at 6:45 PM, Daniel Holth <dholth at gmail.com> wrote:
>
>> And a while back I argued against setuptools-lite, because I thought it
>> would not solve the poor extensibility problem that stems from its basic
>> distutils derived design... which includes all the classes and subclasses
>> that have to work together to do its thing. My own thinking is that
>> setuptools is fine, except that when you need to extend it you have a big
>> problem; so make it easy to replace.
>>
>
> I never had any fantasies that setuptool-lite would improve on this in a
> way shape or form -- indeed, a better build system is likely to need to be
> built from scratch, or on a totally different framework, anyway.
>
>
>> Does pip monkey patch setuptools? I thought it just forced setuptools
>> even if a package only asked for distutils.
>>
>
> good point -- not really a monkey patch, but it does force particular
> configuration.
>
>
>> So you are suggesting setuptools_lite would not honor the setup_requires
>> = [] and test_requires = []
>>
>
> essentially, yes. Though It would be nice if that data could be available
> for pip -- and I"m not sure how that is done at this point.
>
>
>> and perhaps "setup.py install" would also throw an error?
>>
>
> Probably not -- but it could only install from local source -- like
> distutils. Though if we could re-work it to only do develop mode and build
> wheels, that may be good -- you'd need pip to install it.
>
>
>> What else? I would suggest that to effectively argue the idea requires
>> reading the source code and enumerating exactly which complexity goes away
>> with the _lite version.
>>
>
> I don't know that reading the source is the issue -- I'm thinking entirely
> in terms of stripping out the API, so there may be things that need to stay
> in, but we don't expose them. though maybe even that's harder than I think.
>
> But yes -- enumerating what would and wouldn't be supported would be a
> good start. But here is the guideline:
>
> It seems we have a (maybe vague) vision of what we want the tool ecosystem
> to look like that involves separation of concerns, such that each component
> is replaceable without having to change the others:
>
> The concerns:
>   - building a package
>   - installing a package
>   - dependency management
>   - run-time version management
>   - plugin handling, etc.
>   - resource managment
>
> setuptools does all of these, yes? but think of these in terms of when
> they come into play:
>
> build time:
>    - building a package
>

- building c extensions
- building NumPy (fortran,)


> install time: installing a package;
>   - installing a package
>   - dependency management
>
run time
>   - run-time version management
>   - plugin handling, etc.
>

- running tests
  - (It's much faster/easier/consistent/reproducible to run tests when all
of the dependencies are bundled into one executable ZIP (e.g. PEX))


>
> so we have:
>
>  - The mythical build tool
>

- TBH, I don't know much about pyproject.toml
  - "PEP 518 -- Specifying Minimum Build System Requirements for Python
Projects"
     https://www.python.org/dev/peps/pep-0518/

And then there are a number of build tools which work with multiple
languages (because just solving for Python doesn't get it):

- Conda (Python)
  - meta.yaml, build.sh, build.bat *
  - http://conda.pydata.org/docs/build_tutorials.html
  -
http://conda.pydata.org/docs/build_tutorials/pkgs.html#conda-build-skeleton
(pypkg -> conda)
  - http://conda.pydata.org/docs/building/meta-yaml.html#build-section
  - http://conda.pydata.org/docs/bdist_conda.html (condapkg -> pypkg)

- Blaze (Java, Python?; Google) -> Open Sourced as Bazel

- Pants (Python; Twitter):
  - BUILD files
  - https://pantsbuild.github.io/python_readme.html
  - https://pantsbuild.github.io/3rdparty_py.html (pip requirements.txt)
  - https://pantsbuild.github.io/scala.html
  - https://pantsbuild.github.io/go_readme.html

- Buck (Java, Facebook):
  - https://buckbuild.com/rule/prebuilt_python_library.html (.egg)
  - https://buckbuild.com/rule/python_binary.html
  - https://buckbuild.com/rule/python_library.html
  - https://buckbuild.com/rule/python_test.html
  - https://buckbuild.com/rule/cxx_library.html
  - https://buckbuild.com/rule/go_library.html
  - https://buckbuild.com/rule/rust_library.html

- Bazel (Java, Python; Google)
  - http://bazel.io/docs/be/python.html
     - http://bazel.io/docs/be/python.html#py_binary
     - http://bazel.io/docs/be/python.html#py_library
     - http://bazel.io/docs/be/python.html#py_test
  - https://bazel.io/docs/be/c-cpp.html
  - https://github.com/bazelbuild/rules_go


... PEX is based on PEP 441:

- "PEP 441 -- Improving Python ZIP Application Support"
  https://www.python.org/dev/peps/pep-0441/
- https://github.com/pantsbuild/pex/blob/master/docs/whatispex.rst

- Again, wheel includes a file manifest in the .whl .zip with SHA256
checksums:
   https://bitbucket.org/pypa/wheel/src/cf4e2d98ecb1/wscript#wscript-69

  - AFAIU, if you specify different install paths, there's no way to do
something like **debsums** / **rpm -V**
    (which fpm handles well)


>  - pip
>  - maybe some of the setuptools spun off -- like pkg_resources
>
> So why aren't we there yet?
>
> One reason is that setuptools the only thing out there that does the
> building and some of the run-time stuff. So people use it. And now you
> can't replace any of it without making a mess.
>
> However, pip_wheel have indeed addressed much of the install-time and
> dependency management issues. So folks use them, and they work well.
>
> So what problem do I want setuptool-lite to solve?
>
> I actually have spent a lot of time using conda to do my package
> installing and dependency management. So I want a way to build a python
> package that JUST builds the package.
>
> In practice, I've found (most) of the flags and work arounds to diable
> etuptools "features", but really?
>
> python setup.py install --single-version-externally-managed
>
> Oh right, that's not enough, I need a record file:
>
> setup.py install --single-version-externally-managed  --record record.txt
>
> Then I want to install in develop mode:
>
> python setup.py develop
>
> OOPS! now it's easy installing a bunch of dependencies! arrgg!
>
> OK, -- no-deps it is.
>
> Google around, this kind of thing is a pain in a lot of people's butts
> --even though it call all be worked around.
>
> So my vision of setuptool-lite is that is simply does not do the things
> that vision of a build tool should not do.
>
> i.e.:
>
> never easy-install anything!
> don't try to resolve dependencies
>
> others??
>
> One "lite" version of setuptools that I personally think would be cool
>> would be pkg_resources as a separate package on pypi. Setuptools would
>> require it as a dependency.
>>
>
> Yes, I'd love to disable pkg_resources too, but couldn't until it was spun
> off.
>
> In fact, years ago, long before pip, setuptools drove me crazy with how it
> combined run-time, build time and intall time stuff. A serious problem when
> I wanted to bundle something up with py2app (or py2exe, or...).
>
> So what would be accomplished?
>
> package builders could simply put:
>
> from setuptools_lite import setup
>
> (etc.)
>
> and then their user would never accidentally easy install anything, etc.
> And it would serve as documentation / notification if a user was using a
> what-should-be-deprecated feature.
>
> By the way, this is totally acknowledged to be a hacky, temporary mess
> that might only help a bit to get us closer from being locked into
> setuptools.
>
> (though maybe it would be a place to work through some of the issues with
> a build API)
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160819/cf163238/attachment-0001.html>

From tritium-list at sdamon.com  Fri Aug 19 01:32:02 2016
From: tritium-list at sdamon.com (tritium-list at sdamon.com)
Date: Fri, 19 Aug 2016 01:32:02 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
Message-ID: <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>



> -----Original Message-----
> From: Distutils-SIG [mailto:distutils-sig-bounces+tritium-
> list=sdamon.com at python.org] On Behalf Of Nick Coghlan
> Sent: Thursday, August 18, 2016 9:40 PM
> To: Nathaniel Smith <njs at pobox.com>
> Cc: DistUtils mailing list <distutils-sig at python.org>
> Subject: Re: [Distutils] Deprecating little used file types/extensions on
PyPI?
> 
> I wasn't suggesting the state would be invisible, more something akin
> to the PEP 438 settings:
> 
> - existing projects with no legacy formats uploaded get "No legacy
> formats" flagged
> - existing projects with legacy formats uploaded get "Legacy formats"
flagged
> - existing maintainers with projects in the second category get a "Can
> enable legacy formats" flag
> 
> By default, new projects would have the "No legacy formats" flag
> (regardless of who created them), but existing maintainers with the
> "Can enable legacy formats" marker would be able to change that. Once
> a maintainer flipped the setting for a given project, then anyone
> maintaining that project could upload the legacy formats, but wouldn't
> automatically gain the ability to change the setting on other
> projects.
> 
> This approach would mean we could be fairly aggressive in pushing
> *new* users towards the newer formats (since there'd be a sharp
> historical line beyond which new accounts simply can't enable the
> legacy formats for their projects), *without* breaking the workflows
> of existing projects and maintainers (at least, not yet).
> 
> Alternatively, we could simply not worry about a user level flag, and
> just have a project level flag that's set to "No legacy formats" by
> default for new projects - new users won't have any incentive to
> change it, while existing users can change it (at least for the time
> being) if that suits their workflow.

Well... what if I am a new user, opening a new project to work with legacy
systems?  How is it fair to me that I didn't get to the game in time to have
my files accepted and the old-timers get to have theirs accepted?  If we
shut it off for everyone, its fair.  If we let anyone turn it back on, its
fair.  I think this is exemplary of a trend on this sig - there is a
contingent that wants to assume things about the intent of project
maintainers, and I think that's the wrong thing to do.

If we want to trim the acceptable formats for distribution to make pypi et.
al. easier to maintain, then that's fine.  Do it.  Or don't do it.  Don't
selectively do it.

> Cheers,
> Nick.
> 


From ncoghlan at gmail.com  Fri Aug 19 02:53:00 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 19 Aug 2016 16:53:00 +1000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
Message-ID: <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>

On 19 August 2016 at 15:32,  <tritium-list at sdamon.com> wrote:
> From: Distutils-SIG [mailto:distutils-sig-bounces+tritium-
>> Alternatively, we could simply not worry about a user level flag, and
>> just have a project level flag that's set to "No legacy formats" by
>> default for new projects - new users won't have any incentive to
>> change it, while existing users can change it (at least for the time
>> being) if that suits their workflow.
>
> Well... what if I am a new user, opening a new project to work with legacy
> systems?

*If* we went down the sunsetting path I suggested (which is by no
means a given, as even though I think it's an option worth discussing,
it still adds complexity and controversy for debatable benefit), then
the expectation would be that anyone in this situation will either be
explicitly tasked with modernising their infrastructure, or else
working with an existing maintainer of the legacy infrastructure that
can configure these newly created legacy projects appropriately.

Folks that *don't* have a previous maintainer to help out and *also*
don't have permission to modernise their infrastructure to align with
current community expectations would then have to go to battle with
their Finance department (since an active unwillingness to modernise
toolchains almost always indicates the presence of a Finance
department) to argue for investment in tooling modernisation, rather
than expecting the open source community to help them workaround their
organisational dysfunction indefinitely.

> How is it fair to me that I didn't get to the game in time to have
> my files accepted and the old-timers get to have theirs accepted?

The UX concern with leaving it enabled as a config option for everyone
is that we potentially miss out on the main goals of the change:
simplification of the new user experience, and simplification of
future tooling development.

Instead, we may accidentally make the new user experience *more
complicated*, as there's now another project config option for them to
learn about. Configuration options should always be considered bad by
default - they complicate test matrices, they complicate maintenance
(of both the service itself and of other tools that interoperate with
it), they complicate documentation, and they complicate the learning
process. Sometimes their benefits outweigh those inherent
disadvantages, but "This behaviour will not be configurable" remains
the default position (otherwise we'd never get anything done in
software at all)

Now, sometimes providing a config option to re-enable deprecated
legacy behaviour is a useful transition tool, but there still needs to
be clear guidance given to authors of documentation and developers of
related tools that there's no need for them to cover or support that
legacy behaviour if they don't want to.

> If we
> shut it off for everyone, its fair.  If we let anyone turn it back on, its
> fair.  I think this is exemplary of a trend on this sig - there is a
> contingent that wants to assume things about the intent of project
> maintainers, and I think that's the wrong thing to do.

I don't - as the design authority for PyPI, we're responsible for a
core part of the user experience of newcomers to the Python ecosystem,
and we need to take that responsibility seriously.

The key implication of that responsibility is an obligation to keep a
close eye on the overall cognitive complexity of our toolchain, which
is still mindbendingly complicated, despite the significant
improvements over the past few years.

In particular, we need to pay attention to the cases where low feature
usage metrics from PyPI indicate the possible presence of legacy
features that are making the new user experience and the toolchain
maintenance process more complicated without providing a sufficient
pay-off in increased power and flexibility for software publishers to
justify those complications.

> If we want to trim the acceptable formats for distribution to make pypi et.
> al. easier to maintain, then that's fine.  Do it.  Or don't do it.  Don't
> selectively do it.

Unilaterally turning the feature off would be extraordinarily hostile
to current users - grace periods and sunset clauses are standard
features of change management processes for a reason, even when they
come at the cost of additional implementation complexity.

That said, allowing new users to opt-in to the feature (for as long as
it sticks around) would be simpler than preventing them, so it's
likely a better option from a pure maintainability perspective,
without even considering whether or not it's fair to new users to
prevent them from opting in to practices we're in the process of
removing from the toolchain.

In my view, the purpose of PyPI is to provide Pythonistas with a
straighforward mechanism to publish open source software if they
choose to do so. Fairness in an abstract sense doesn't really enter
into it for me - the question I ask myself is "Will this change have a
long term net impact of making it easier for more Pythonistas to
publish and readily reuse more open source software?".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From donald at stufft.io  Fri Aug 19 08:14:34 2016
From: donald at stufft.io (Donald Stufft)
Date: Fri, 19 Aug 2016 08:14:34 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
Message-ID: <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>


> On Aug 19, 2016, at 2:53 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> Unilaterally turning the feature off would be extraordinarily hostile
> to current users - grace periods and sunset clauses are standard
> features of change management processes for a reason, even when they
> come at the cost of additional implementation complexity.


This wouldn?t be a new way of handling this, when we implemented PEP 470
new projects immediately got set to the ?only files on PyPI mode? and had
no ability to change to a different mode (no reason to present an option
that was going away). Emails were sent to maintains on all projects that
had an existing external link and told that in X months all their external
links would be removed (which was implemented just as switching them to the
?only files on PyPI mode?.

Therefore, given what?s been discussed thus far, my proposal would be:

Add a hidden flag, ?legacy file support? on a per project basis. All new projects
have this flag switched off, any existing project that has not ever uploaded a
file that would use this flag has it switched off, everyone else has it switched
on. Emails get sent out to maintainers of projects where it is still switched on
and they?re told ?In 3 months you?ll no longer be able to upload legacy file types,
but all existing files uploaded will continue to exist?. 

Files that would fall under legacy file type:

* All types except sdist, bdist_wheel, and bdist_egg [1].
* All sdist extensions besides .tar.gz.


That prevents new (and existing) projects from getting to a point where they
depend on something that is going away when they previously didn?t and gives
existing projects a chance to update their automation and what not to handle
this scenario.


[1] We can tackle egg at a later point, when setuptools either has support for Wheels
    or is less needed.

?
Donald Stufft




From dholth at gmail.com  Fri Aug 19 08:46:33 2016
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 19 Aug 2016 12:46:33 +0000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
Message-ID: <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>

Check out this Camden Bench. https://en.wikipedia.org/wiki/Camden_bench . The
creators of the bench enumerated about 23 specific undesirable behaviors
that they feel normal benches allow (like sleeping and skateboarding) and
came up with a lump of concrete that you can sit on.

On Fri, Aug 19, 2016 at 8:14 AM Donald Stufft <donald at stufft.io> wrote:

>
> > On Aug 19, 2016, at 2:53 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >
> > Unilaterally turning the feature off would be extraordinarily hostile
> > to current users - grace periods and sunset clauses are standard
> > features of change management processes for a reason, even when they
> > come at the cost of additional implementation complexity.
>
>
> This wouldn?t be a new way of handling this, when we implemented PEP 470
> new projects immediately got set to the ?only files on PyPI mode? and had
> no ability to change to a different mode (no reason to present an option
> that was going away). Emails were sent to maintains on all projects that
> had an existing external link and told that in X months all their external
> links would be removed (which was implemented just as switching them to the
> ?only files on PyPI mode?.
>
> Therefore, given what?s been discussed thus far, my proposal would be:
>
> Add a hidden flag, ?legacy file support? on a per project basis. All new
> projects
> have this flag switched off, any existing project that has not ever
> uploaded a
> file that would use this flag has it switched off, everyone else has it
> switched
> on. Emails get sent out to maintainers of projects where it is still
> switched on
> and they?re told ?In 3 months you?ll no longer be able to upload legacy
> file types,
> but all existing files uploaded will continue to exist?.
>
> Files that would fall under legacy file type:
>
> * All types except sdist, bdist_wheel, and bdist_egg [1].
> * All sdist extensions besides .tar.gz.
>
>
> That prevents new (and existing) projects from getting to a point where
> they
> depend on something that is going away when they previously didn?t and
> gives
> existing projects a chance to update their automation and what not to
> handle
> this scenario.
>
>
> [1] We can tackle egg at a later point, when setuptools either has support
> for Wheels
>     or is less needed.
>
> ?
> Donald Stufft
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160819/fb14a8e2/attachment-0001.html>

From donald at stufft.io  Fri Aug 19 08:58:29 2016
From: donald at stufft.io (Donald Stufft)
Date: Fri, 19 Aug 2016 08:58:29 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
Message-ID: <35956E82-37A7-40C5-AEE5-B4D8495FB0F7@stufft.io>


> On Aug 19, 2016, at 8:46 AM, Daniel Holth <dholth at gmail.com> wrote:
> 
> Check out this Camden Bench. https://en.wikipedia.org/wiki/Camden_bench <https://en.wikipedia.org/wiki/Camden_bench> . The creators of the bench enumerated about 23 specific undesirable behaviors that they feel normal benches allow (like sleeping and skateboarding) and came up with a lump of concrete that you can sit on.


Ok?

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160819/529e111b/attachment.html>

From ubernostrum at gmail.com  Fri Aug 19 09:36:52 2016
From: ubernostrum at gmail.com (James Bennett)
Date: Fri, 19 Aug 2016 06:36:52 -0700
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
Message-ID: <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>

Every new configuration option added to a piece of software represents two
opportunities:

1. An opportunity to introduce exciting new bugs.

2. An opportunity to introduce exciting new ways for users to click the
thing that does the exact opposite of what they wanted to do.

The proliferation of different package formats is not a sign of innovation;
it's a sign of the desperation induced by how bad Python packaging was, and
for how long. Now that things are in a much better state, it's time to
remove the combinatorial explosion of opportunities for bugs and end-user
mistakes, and be able to say clearly "here's how you package and distribute
Python code".

Comparing that to to "hostile architecture" is petulant and inappropriate
and you know it.


On Fri, Aug 19, 2016 at 5:46 AM, Daniel Holth <dholth at gmail.com> wrote:

> Check out this Camden Bench. https://en.wikipedia.org/wiki/Camden_bench . The
> creators of the bench enumerated about 23 specific undesirable behaviors
> that they feel normal benches allow (like sleeping and skateboarding) and
> came up with a lump of concrete that you can sit on.
>
> On Fri, Aug 19, 2016 at 8:14 AM Donald Stufft <donald at stufft.io> wrote:
>
>>
>> > On Aug 19, 2016, at 2:53 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> >
>> > Unilaterally turning the feature off would be extraordinarily hostile
>> > to current users - grace periods and sunset clauses are standard
>> > features of change management processes for a reason, even when they
>> > come at the cost of additional implementation complexity.
>>
>>
>> This wouldn?t be a new way of handling this, when we implemented PEP 470
>> new projects immediately got set to the ?only files on PyPI mode? and had
>> no ability to change to a different mode (no reason to present an option
>> that was going away). Emails were sent to maintains on all projects that
>> had an existing external link and told that in X months all their external
>> links would be removed (which was implemented just as switching them to
>> the
>> ?only files on PyPI mode?.
>>
>> Therefore, given what?s been discussed thus far, my proposal would be:
>>
>> Add a hidden flag, ?legacy file support? on a per project basis. All new
>> projects
>> have this flag switched off, any existing project that has not ever
>> uploaded a
>> file that would use this flag has it switched off, everyone else has it
>> switched
>> on. Emails get sent out to maintainers of projects where it is still
>> switched on
>> and they?re told ?In 3 months you?ll no longer be able to upload legacy
>> file types,
>> but all existing files uploaded will continue to exist?.
>>
>> Files that would fall under legacy file type:
>>
>> * All types except sdist, bdist_wheel, and bdist_egg [1].
>> * All sdist extensions besides .tar.gz.
>>
>>
>> That prevents new (and existing) projects from getting to a point where
>> they
>> depend on something that is going away when they previously didn?t and
>> gives
>> existing projects a chance to update their automation and what not to
>> handle
>> this scenario.
>>
>>
>> [1] We can tackle egg at a later point, when setuptools either has
>> support for Wheels
>>     or is less needed.
>>
>> ?
>> Donald Stufft
>>
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160819/6573aec7/attachment.html>

From dholth at gmail.com  Fri Aug 19 10:23:22 2016
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 19 Aug 2016 14:23:22 +0000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
Message-ID: <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>

The only issue I have with the proposal is the removal of .zip sdists
because I would personally be inconvenienced by the removal of .zip.
Consider that "setup.py sdist" produces .zip by default on Windows.

On Fri, Aug 19, 2016 at 9:36 AM James Bennett <ubernostrum at gmail.com> wrote:

> Every new configuration option added to a piece of software represents two
> opportunities:
>
> 1. An opportunity to introduce exciting new bugs.
>
> 2. An opportunity to introduce exciting new ways for users to click the
> thing that does the exact opposite of what they wanted to do.
>
> The proliferation of different package formats is not a sign of
> innovation; it's a sign of the desperation induced by how bad Python
> packaging was, and for how long. Now that things are in a much better
> state, it's time to remove the combinatorial explosion of opportunities for
> bugs and end-user mistakes, and be able to say clearly "here's how you
> package and distribute Python code".
>
> Comparing that to to "hostile architecture" is petulant and inappropriate
> and you know it.
>
>
> On Fri, Aug 19, 2016 at 5:46 AM, Daniel Holth <dholth at gmail.com> wrote:
>
>> Check out this Camden Bench. https://en.wikipedia.org/wiki/Camden_bench
>>  . The creators of the bench enumerated about 23 specific undesirable
>> behaviors that they feel normal benches allow (like sleeping and
>> skateboarding) and came up with a lump of concrete that you can sit on.
>>
>> On Fri, Aug 19, 2016 at 8:14 AM Donald Stufft <donald at stufft.io> wrote:
>>
>>>
>>> > On Aug 19, 2016, at 2:53 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> >
>>> > Unilaterally turning the feature off would be extraordinarily hostile
>>> > to current users - grace periods and sunset clauses are standard
>>> > features of change management processes for a reason, even when they
>>> > come at the cost of additional implementation complexity.
>>>
>>>
>>> This wouldn?t be a new way of handling this, when we implemented PEP 470
>>> new projects immediately got set to the ?only files on PyPI mode? and had
>>> no ability to change to a different mode (no reason to present an option
>>> that was going away). Emails were sent to maintains on all projects that
>>> had an existing external link and told that in X months all their
>>> external
>>> links would be removed (which was implemented just as switching them to
>>> the
>>> ?only files on PyPI mode?.
>>>
>>> Therefore, given what?s been discussed thus far, my proposal would be:
>>>
>>> Add a hidden flag, ?legacy file support? on a per project basis. All new
>>> projects
>>> have this flag switched off, any existing project that has not ever
>>> uploaded a
>>> file that would use this flag has it switched off, everyone else has it
>>> switched
>>> on. Emails get sent out to maintainers of projects where it is still
>>> switched on
>>> and they?re told ?In 3 months you?ll no longer be able to upload legacy
>>> file types,
>>> but all existing files uploaded will continue to exist?.
>>>
>>> Files that would fall under legacy file type:
>>>
>>> * All types except sdist, bdist_wheel, and bdist_egg [1].
>>> * All sdist extensions besides .tar.gz.
>>>
>>>
>>> That prevents new (and existing) projects from getting to a point where
>>> they
>>> depend on something that is going away when they previously didn?t and
>>> gives
>>> existing projects a chance to update their automation and what not to
>>> handle
>>> this scenario.
>>>
>>>
>>> [1] We can tackle egg at a later point, when setuptools either has
>>> support for Wheels
>>>     or is less needed.
>>>
>>> ?
>>> Donald Stufft
>>>
>>>
>>>
>>> _______________________________________________
>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160819/f5b5db03/attachment-0001.html>

From donald at stufft.io  Fri Aug 19 10:27:18 2016
From: donald at stufft.io (Donald Stufft)
Date: Fri, 19 Aug 2016 10:27:18 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
Message-ID: <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>


> On Aug 19, 2016, at 10:23 AM, Daniel Holth <dholth at gmail.com> wrote:
> 
> Consider that "setup.py sdist" produces .zip by default on Windows.


This is an important consideration that nobody mentioned thus far? I have not used Windows to release a project to PyPI.. ever. So We should absolutely start by switching that to defaulting to .tar.gz.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160819/191336de/attachment.html>

From p.f.moore at gmail.com  Fri Aug 19 10:34:25 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 19 Aug 2016 15:34:25 +0100
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
Message-ID: <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>

On 19 August 2016 at 15:27, Donald Stufft <donald at stufft.io> wrote:
> This is an important consideration that nobody mentioned thus far? I have
> not used Windows to release a project to PyPI.. ever. So We should
> absolutely start by switching that to defaulting to .tar.gz.

I had forgotten this fact, but yes, zip is the standard for sdists on Windows.

I would suggest that we need to allow both .zip and .tar.gz for
sdists. Specifically, how would we handle people building sdists for
upload using Python 2.7 on Windows? Would we backport this change?

(It doesn't matter whether we say ".tar.gz is the standard, what do we
do about Windows users?" or ".zip is the standard, what do we do about
Unix users?" the problems are the same).

Paul

From donald at stufft.io  Fri Aug 19 11:14:36 2016
From: donald at stufft.io (Donald Stufft)
Date: Fri, 19 Aug 2016 11:14:36 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
Message-ID: <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>


> On Aug 19, 2016, at 10:34 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 19 August 2016 at 15:27, Donald Stufft <donald at stufft.io> wrote:
>> This is an important consideration that nobody mentioned thus far? I have
>> not used Windows to release a project to PyPI.. ever. So We should
>> absolutely start by switching that to defaulting to .tar.gz.
> 
> I had forgotten this fact, but yes, zip is the standard for sdists on Windows.
> 
> I would suggest that we need to allow both .zip and .tar.gz for
> sdists. Specifically, how would we handle people building sdists for
> upload using Python 2.7 on Windows? Would we backport this change?

Not sure if we?ll be able to back port it to 2.7 itself, but it?s a trivial
change to make inside of setuptools as well. Here?s a PR to setuptools that
will adjust the default: https://github.com/pypa/setuptools/pull/748.

While I think it?s OK to allow .zip in the interim, I do think we should be
trying to move towards only allowing a single format. I honestly don?t care
what that format is, but I think it?ll be easier to move Windows users onto
.tar.gz than it will be to move everyone else to .tar.gz (smaller user base,
Python and setuptools are not ?ingrained? into the System like is the case
on RHEL/CentOS so easier upgrades, etc).

My motivation on this is trying to normalize things across platforms and to
fix persistent sort of weirdness things that I see coming up over and over.
One such one is people getting different results (for whatever reason) when
a project has two sdists available. So I?ve been working on trying to make
that the case, and it would greatly simplify things if there were only one
filename acceptable for a single (Project, Version)?s sdist. I also think it
helps make other tooling around PyPI easier to write, and it helps ensure
that the dependencies to install something are minimal (though not particularly
relevant for zip/gztar since they both use zlib).

> 
> (It doesn't matter whether we say ".tar.gz is the standard, what do we
> do about Windows users?" or ".zip is the standard, what do we do about
> Unix users?" the problems are the same).
> 
> Paul


?
Donald Stufft




From chris.barker at noaa.gov  Fri Aug 19 12:46:30 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 19 Aug 2016 09:46:30 -0700
Subject: [Distutils] Future of setuptools and buildout
In-Reply-To: <CACfEFw9aPsX-+yHkNBmbXfUkWkC-7ayZcQyOLWgm+3w14GYhkg@mail.gmail.com>
References: <CAMxX_oP-8R=Gr4Jp9P8CXNSB7muuq=2z8E0K_V7qCvFodJTJOQ@mail.gmail.com>
 <EED02033-FB4E-4B25-A37B-C09B4CDF5F31@stufft.io>
 <CALGmxEJZBb=eakpUPWwqy+Wa=K+EoCvqi+sPgf4cPx-4+1m0yA@mail.gmail.com>
 <CAG8k2+7_DbueMsrokPMwbrzhzq8-M2t8QSOZN1-O3PJKvCh17Q@mail.gmail.com>
 <CALGmxE+mMuJ8P5PP9bbSuUQEx06PwH3hEdo7aYr_RBbY3_NZ3g@mail.gmail.com>
 <CACfEFw9aPsX-+yHkNBmbXfUkWkC-7ayZcQyOLWgm+3w14GYhkg@mail.gmail.com>
Message-ID: <CALGmxEJ0K8AmRS3GhGg+bV67iEhzXcMCgqgpaBthRua=naer3g@mail.gmail.com>

On Thu, Aug 18, 2016 at 10:17 PM, Wes Turner <wes.turner at gmail.com> wrote:

> setuptools does all of these, yes? but think of these in terms of when
>> they come into play:
>>
>> build time:
>>    - building a package
>>
>
> - building c extensions
> - building NumPy (fortran,)
>

exactly! which is WHY we want/need more flexibility in build tools --
distutils is a mess to extend to do other things. (though is does do a fine
job with pure C extensions, which is what it was designed for).



> install time: installing a package;
>>   - installing a package
>>   - dependency management
>>
> run time
>>   - run-time version management
>>   - plugin handling, etc.
>>
>
> - running tests
>   - (It's much faster/easier/consistent/reproducible to run tests when
> all of the dependencies are bundled into one executable ZIP (e.g. PEX))
>

hmm - it seems most of us use a test-running for this -- so I put that in a
separate category. But you need the package management tool to be able to
easily set up an environment for testing -- i.e install the package and all
its dependencies, then run the tests. I don't know anything about PEX, but
I'd really NOT want the test runner to rely on any particular installation
setup.

And, as it happend, test running is one thing setuptools does not
currently, do so, we have pytest and nose and ??? that actually ARE
separate pluggable test running systems.


>
>> so we have:
>>
>>  - The mythical build tool
>>
>
> - TBH, I don't know much about pyproject.toml
>   - "PEP 518 -- Specifying Minimum Build System Requirements for Python
> Projects"
>      https://www.python.org/dev/peps/pep-0518/
>

pyproject.toml is NOT a build tool at all -- but it is a step towards being
able to support other build and packaging tools.

so maybe version 2 of setuptools_lite should rely on pyproject.toml.


> And then there are a number of build tools which work with multiple
> languages (because just solving for Python doesn't get it):
>
> - Conda (Python)
>

conda IS NOT a build tool -- I was pretty disappointed when I realized
that, but what can you do? cross-platform, cross-language build tools are a
really hard problem -- no one wants to write another one :-). And this is a
good thing -- what we WANT is for the packaging tool and the build tools to
be separate.


>   - meta.yaml, build.sh, build.bat *
>

the meta.yaml provides all the meta-data (kind like a higher level
pyproject.toml, and then simply invokes the build scripts. but you need to
write those build scripts -- conda provides literally no help with that.
most of them simply invoke whatever build system the library you are trying
to [package already uses -- setuptools, make, cmake, ....

However, since you brought up testing earlier -- conda does have a nice
system for testing -- it isn't a test runner, but it does provide isolated
environments (a job of the installer???) -- so conda-build can:

setup an environment for building
build the package
Then:

setup an environment for testing
installed the package that was just built
invoke a test-runner

This is REALLY nice -- you can have all your test code run with the built
package, with the environment that you have specified, on the platform you
have built for (on).

This has caught all sorts of stupid errors for me -- forgetting a
dependency, etc...

Whether that should be built into the packaging tool I suppose is
debateable. But maybe it's not -- conda-build is a separate tool -- it
relies on the conda system, but it does it's own thing. conda packages are
fairly simple archives -- it's very possible to create them with other
tools.

- Blaze (Java, Python?; Google) -> Open Sourced as Bazel
>
- Pants (Python; Twitter):
> - Buck (Java, Facebook):
> - Bazel (Java, Python; Google)
>

Maybe the alternative to setuptools_lite is to set up one of these other
tools to work well (easily) for building python packages. If it can do
everything that setuptools can do (that we want setuptools to do), and just
as easily, maybe that's the next step forward.

but if that isn't happening soon, then a setuptools_lite would be a useful
step forward.

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160819/52444dca/attachment.html>

From chris.barker at noaa.gov  Fri Aug 19 12:53:53 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 19 Aug 2016 09:53:53 -0700
Subject: [Distutils] What role to eggs still play?
Message-ID: <CALGmxE+=4-gqgbAR1uGODKR1C-zjrSCEyBXOqp-2x_ypAeb8aw@mail.gmail.com>

Hi all,

starting a new thread, but this is related to the setuptols-_lite
discussion, and the legacy formats discussion. In another thread Donald had
a footnote:


[1] We can tackle egg at a later point, when setuptools either has support
> for Wheels
>     or is less needed.


So I'm wondering -- does anything else (other than setuptools) depend on
eggs in any way? I know why I pip install stuff, I (always?) get egg-ish
things installed:

.egg-info

directories and all that. Honestly, I'm confused -- is that making an
actual egg? or is that name simply a legacy name for package meta data?

In any case, does pip, or anything else, require it?

For my part, I find it annoying, name aside -- it seems that all the
package meta-data should be there in the package source already
(pypacakge.toml?)

-Chris







-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160819/e61c5665/attachment.html>

From p.f.moore at gmail.com  Fri Aug 19 13:14:49 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 19 Aug 2016 18:14:49 +0100
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
Message-ID: <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>

On 19 August 2016 at 16:14, Donald Stufft <donald at stufft.io> wrote:
> Not sure if we?ll be able to back port it to 2.7 itself, but it?s a trivial
> change to make inside of setuptools as well. Here?s a PR to setuptools that
> will adjust the default: https://github.com/pypa/setuptools/pull/748.

Hmm. So the question becomes, do we care about supporting users
building sdists without setuptools? It's not like this is a scenario
controlled by pip, which auto-injects setuptools. So I guess we'd need
to also publicise the --formats=gztar flag (at
https://packaging.python.org/distributing/#source-distributions if
nowhere else).

So, the plan becomes:

1. Change Python 3.6 to default to .tar.gz on Windows
2. Change setuptoos to default to .tar.gz, to catch users of older versions
3. Document how to create a source distribution as "python setup.py
sdist (add the --formats=gztar flag if you're on Windows and using a
Python older than 3.6 and not using setuptools)" in the PUG
4. Update https://docs.python.org/3/distutils/sourcedist.html for 3.6
to note the change in default.

For (3) soften the laboured conditional comment as much as you like,
but I think we need something to help people who don't understand
what's going on. For (1) and (4), maybe we don't target 3.6 (given
setuptools, it's not urgent) but I think we should change the core, if
only to avoid surprises for people who don't include setuptools in
their setup.py (maybe relying on pip injecting it for things like
wheel builds), and so that distutils isn't (in effect) deliberately
wrong going forward.

> While I think it?s OK to allow .zip in the interim, I do think we should be
> trying to move towards only allowing a single format.

Your arguments are compelling, I have no problem with the conclusion -
we just need a little more thought on the transition.

Paul

From donald at stufft.io  Fri Aug 19 13:20:34 2016
From: donald at stufft.io (Donald Stufft)
Date: Fri, 19 Aug 2016 13:20:34 -0400
Subject: [Distutils] What role to eggs still play?
In-Reply-To: <CALGmxE+=4-gqgbAR1uGODKR1C-zjrSCEyBXOqp-2x_ypAeb8aw@mail.gmail.com>
References: <CALGmxE+=4-gqgbAR1uGODKR1C-zjrSCEyBXOqp-2x_ypAeb8aw@mail.gmail.com>
Message-ID: <B242E8FD-102B-4D27-B4AB-845BD93EECC2@stufft.io>


> On Aug 19, 2016, at 12:53 PM, Chris Barker <chris.barker at noaa.gov> wrote:
> 
> Hi all,
> 
> starting a new thread, but this is related to the setuptols-_lite discussion, and the legacy formats discussion. In another thread Donald had a footnote:
> 
> 
> [1] We can tackle egg at a later point, when setuptools either has support for Wheels
>     or is less needed.
> 
> So I'm wondering -- does anything else (other than setuptools) depend on eggs in any way? I know why I pip install stuff, I (always?) get egg-ish things installed:
> 
> .egg-info
> 
> directories and all that. Honestly, I'm confused -- is that making an actual egg? or is that name simply a legacy name for package meta data?
> 
> In any case, does pip, or anything else, require it?
> 
> For my part, I find it annoying, name aside -- it seems that all the package meta-data should be there in the package source already (pypacakge.toml?)
> 


Part of the problems with ?egg? is that the word ?egg? ends up being used for slightly similar, but somewhat different things. The .egg-info is the installation metadata (and the newer format uses .dist-info instead of .egg-info). That particular footnote was speaking about .egg files themselves, uploaded to PyPI.


?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160819/9ea39a3f/attachment.html>

From donald at stufft.io  Fri Aug 19 13:21:42 2016
From: donald at stufft.io (Donald Stufft)
Date: Fri, 19 Aug 2016 13:21:42 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
Message-ID: <56CA7DFC-F489-4110-976E-8D852C5AD0B8@stufft.io>


> On Aug 19, 2016, at 1:14 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 19 August 2016 at 16:14, Donald Stufft <donald at stufft.io> wrote:
>> Not sure if we?ll be able to back port it to 2.7 itself, but it?s a trivial
>> change to make inside of setuptools as well. Here?s a PR to setuptools that
>> will adjust the default: https://github.com/pypa/setuptools/pull/748.
> 
> Hmm. So the question becomes, do we care about supporting users
> building sdists without setuptools? It's not like this is a scenario
> controlled by pip, which auto-injects setuptools. So I guess we'd need
> to also publicise the --formats=gztar flag (at
> https://packaging.python.org/distributing/#source-distributions if
> nowhere else).
> 
> So, the plan becomes:
> 
> 1. Change Python 3.6 to default to .tar.gz on Windows
> 2. Change setuptoos to default to .tar.gz, to catch users of older versions
> 3. Document how to create a source distribution as "python setup.py
> sdist (add the --formats=gztar flag if you're on Windows and using a
> Python older than 3.6 and not using setuptools)" in the PUG
> 4. Update https://docs.python.org/3/distutils/sourcedist.html for 3.6
> to note the change in default.
> 
> For (3) soften the laboured conditional comment as much as you like,
> but I think we need something to help people who don't understand
> what's going on. For (1) and (4), maybe we don't target 3.6 (given
> setuptools, it's not urgent) but I think we should change the core, if
> only to avoid surprises for people who don't include setuptools in
> their setup.py (maybe relying on pip injecting it for things like
> wheel builds), and so that distutils isn't (in effect) deliberately
> wrong going forward.


FWIW, they can also drop a setup.cfg in their project too that looks
like: https://bpaste.net/show/90dd1280eba6 and then forget about it, which
makes it somewhat less painful, since they won?t have to remember to do it
on a per project basis past the first time. I believe that can also be put
in ~/.pydistutils.cfg (or w/e that file is again?) to change the default on
a per user basis.


>> While I think it?s OK to allow .zip in the interim, I do think we should be
>> trying to move towards only allowing a single format.
> 
> Your arguments are compelling, I have no problem with the conclusion -
> we just need a little more thought on the transition.
> 

Sure! I have some other things to do today, but I?ll try and figure out something
a bit more fleshed out for the transition given this new (to me) information.

?
Donald Stufft




From dholth at gmail.com  Fri Aug 19 13:25:26 2016
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 19 Aug 2016 17:25:26 +0000
Subject: [Distutils] What role to eggs still play?
In-Reply-To: <CALGmxE+=4-gqgbAR1uGODKR1C-zjrSCEyBXOqp-2x_ypAeb8aw@mail.gmail.com>
References: <CALGmxE+=4-gqgbAR1uGODKR1C-zjrSCEyBXOqp-2x_ypAeb8aw@mail.gmail.com>
Message-ID: <CAG8k2+4W9Y3qhpBBhDXcO6wPDkjJUDDJdcQ=YZGEidOXXcbxdw@mail.gmail.com>

Eggs are the only way to add a zipped distribution to PYTHONPATH and have
setuptools find the metadata (the Python code can be found with or without
the metadata; setuptools does not discover *.dist-info inside zip). Eggs
are used by buildout, especially in the unzipped into a directory form. And
they could still be used for their originally designed use as a plugin
format.

One of the smaller problems with eggs is that everything had the same name.
*.egg-info or EGG-INFO is the predecessor of the *.dist-info format
designed in PEP-376. You get *.egg-info whenever you install something with
setuptools without going through bdist_wheel.

On Fri, Aug 19, 2016 at 12:54 PM Chris Barker <chris.barker at noaa.gov> wrote:

> Hi all,
>
> starting a new thread, but this is related to the setuptols-_lite
> discussion, and the legacy formats discussion. In another thread Donald had
> a footnote:
>
>
> [1] We can tackle egg at a later point, when setuptools either has support
>> for Wheels
>>     or is less needed.
>
>
> So I'm wondering -- does anything else (other than setuptools) depend on
> eggs in any way? I know why I pip install stuff, I (always?) get egg-ish
> things installed:
>
> .egg-info
>
> directories and all that. Honestly, I'm confused -- is that making an
> actual egg? or is that name simply a legacy name for package meta data?
>
> In any case, does pip, or anything else, require it?
>
> For my part, I find it annoying, name aside -- it seems that all the
> package meta-data should be there in the package source already
> (pypacakge.toml?)
>
> -Chris
>
>
>
>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160819/10098cb2/attachment.html>

From dholth at gmail.com  Fri Aug 19 13:39:22 2016
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 19 Aug 2016 17:39:22 +0000
Subject: [Distutils] What role to eggs still play?
In-Reply-To: <CAG8k2+4W9Y3qhpBBhDXcO6wPDkjJUDDJdcQ=YZGEidOXXcbxdw@mail.gmail.com>
References: <CALGmxE+=4-gqgbAR1uGODKR1C-zjrSCEyBXOqp-2x_ypAeb8aw@mail.gmail.com>
 <CAG8k2+4W9Y3qhpBBhDXcO6wPDkjJUDDJdcQ=YZGEidOXXcbxdw@mail.gmail.com>
Message-ID: <CAG8k2+5UYQ=dJBuVr-SjFU_=+h7GR60pp7bBwZNnrxx-QV0D9g@mail.gmail.com>

About the toml file... the *-info metadata is a compiled artifact,
according to all the existing Python packages. Most sdists even have a
*.egg-info directory. It is inconvenient if you want to know the true
dependencies without running setup.py. I think we are stuck with it, and
it's not all bad - if there is some useful metadata that doesn't affect the
dependency resolver, and it is "too static" or too cumbersome to write out
by hand, there's a place to do that.

On Fri, Aug 19, 2016 at 1:25 PM Daniel Holth <dholth at gmail.com> wrote:

> Eggs are the only way to add a zipped distribution to PYTHONPATH and have
> setuptools find the metadata (the Python code can be found with or without
> the metadata; setuptools does not discover *.dist-info inside zip). Eggs
> are used by buildout, especially in the unzipped into a directory form. And
> they could still be used for their originally designed use as a plugin
> format.
>
> One of the smaller problems with eggs is that everything had the same
> name. *.egg-info or EGG-INFO is the predecessor of the *.dist-info format
> designed in PEP-376. You get *.egg-info whenever you install something
> with setuptools without going through bdist_wheel.
>
> On Fri, Aug 19, 2016 at 12:54 PM Chris Barker <chris.barker at noaa.gov>
> wrote:
>
>> Hi all,
>>
>> starting a new thread, but this is related to the setuptols-_lite
>> discussion, and the legacy formats discussion. In another thread Donald had
>> a footnote:
>>
>>
>> [1] We can tackle egg at a later point, when setuptools either has
>>> support for Wheels
>>>     or is less needed.
>>
>>
>> So I'm wondering -- does anything else (other than setuptools) depend on
>> eggs in any way? I know why I pip install stuff, I (always?) get egg-ish
>> things installed:
>>
>> .egg-info
>>
>> directories and all that. Honestly, I'm confused -- is that making an
>> actual egg? or is that name simply a legacy name for package meta data?
>>
>> In any case, does pip, or anything else, require it?
>>
>> For my part, I find it annoying, name aside -- it seems that all the
>> package meta-data should be there in the package source already
>> (pypacakge.toml?)
>>
>> -Chris
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR&R            (206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115       (206) 526-6317   main reception
>>
>> Chris.Barker at noaa.gov
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160819/3ef76446/attachment-0001.html>

From p.f.moore at gmail.com  Fri Aug 19 13:48:34 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 19 Aug 2016 18:48:34 +0100
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <56CA7DFC-F489-4110-976E-8D852C5AD0B8@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <56CA7DFC-F489-4110-976E-8D852C5AD0B8@stufft.io>
Message-ID: <CACac1F9rapf067cr1AUm1nabKBdNjKZZjU5261+sUSz3nbXc1g@mail.gmail.com>

On 19 August 2016 at 18:21, Donald Stufft <donald at stufft.io> wrote:
> FWIW, they can also drop a setup.cfg in their project too that looks
> like: https://bpaste.net/show/90dd1280eba6 and then forget about it, which
> makes it somewhat less painful, since they won?t have to remember to do it
> on a per project basis past the first time. I believe that can also be put
> in ~/.pydistutils.cfg (or w/e that file is again?) to change the default on
> a per user basis.

Good point, that's a much better idea.

> Sure! I have some other things to do today, but I?ll try and figure out
> something a bit more fleshed out for the transition given this new (to
> me) information.

Cool. As I say, it was new to me as well, which I'm sad about as I'm
supposed to be the "remember Windows" voice around here ;-)

FWIW, even apart from the process of simplifying things for PyPI, I
think this is a good move just to eliminate the annoying
platform-dependent default behaviour.

Paul

From leorochael at gmail.com  Fri Aug 19 14:00:15 2016
From: leorochael at gmail.com (Leonardo Rochael Almeida)
Date: Fri, 19 Aug 2016 15:00:15 -0300
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
Message-ID: <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>

On 19 August 2016 at 14:14, Paul Moore <p.f.moore at gmail.com> wrote:

> [...]
>
> So, the plan becomes:
>
> 1. Change Python 3.6 to default to .tar.gz on Windows
> 2. Change setuptoos to default to .tar.gz, to catch users of older versions
> 3. Document how to create a source distribution as "python setup.py
> sdist (add the --formats=gztar flag if you're on Windows and using a
> Python older than 3.6 and not using setuptools)" in the PUG
> 4. Update https://docs.python.org/3/distutils/sourcedist.html for 3.6
> to note the change in default.
>
> For (3) soften the laboured conditional comment as much as you like,
> but I think we need something to help people who don't understand
> what's going on. For (1) and (4), maybe we don't target 3.6 (given
> setuptools, it's not urgent) but I think we should change the core, if
> only to avoid surprises for people who don't include setuptools in
> their setup.py (maybe relying on pip injecting it for things like
> wheel builds), and so that distutils isn't (in effect) deliberately
> wrong going forward.
>
> > While I think it?s OK to allow .zip in the interim, I do think we should
> be
> > trying to move towards only allowing a single format.
>
> Your arguments are compelling, I have no problem with the conclusion -
> we just need a little more thought on the transition.
>

If we're going through all this trouble, isn't it better just to jump to
.zip files like every other distribution format in existence?

Sure, more people will be affected this way than just the folks releasing
on Windows, but given the shortcuts for setting the sdist format per
project or per home directory that Donald mentioned, I think the collective
effort in the migration will be smaller than the continuous effort of
explaining to newcomers that the reason we use a .tar.gz based format for
sdists versus a .zip based format for wheels is some historical accident,
specially if we plan to change sdists back to .zip format in the future...

Regards,

Leo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160819/bcf72ca3/attachment.html>

From donald at stufft.io  Fri Aug 19 15:49:24 2016
From: donald at stufft.io (Donald Stufft)
Date: Fri, 19 Aug 2016 15:49:24 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
Message-ID: <DAA65B69-8246-4E3A-A482-5BF77E60FD44@stufft.io>


> On Aug 19, 2016, at 2:00 PM, Leonardo Rochael Almeida <leorochael at gmail.com> wrote:
> 
> ure, more people will be affected this way than just the folks releasing on Windows, but given the shortcuts for setting the sdist format per project or per home directory that Donald mentioned, I think the collective effort in the migration will be smaller than the continuous effort of explaining to newcomers that the reason we use a .tar.gz based format for sdists versus a .zip based format for wheels is some historical accident, specially if we plan to change sdists back to .zip format in the future...


I don?t think I?ve ever had anyone ask me why people (generally) use .tar.gz for sdists and why wheels use zip, though I don?t do much beginner stuff. Do you have some sort of evidence or data to suggest that this is a problem that people are experiencing or are you theorizing that folks *might* get confused by this?

I think the effort changing non-Windows platform is going to be a lot more effort than changing Windows platform for a few reasons:

* There are less people releasing on Windows than on non-Windows, the more people you need to migrate to a new thing, the longer you can expect it to take.

* Windows does not come with Python, thus people are generally free to upgrade their Python or setuptools installation at all [1] meanwhile Python and setuptools tends to be a core part of non-Windows OSs where upgrading one or the other can ?void the warranty? and cause breakage to the entire OS, combine this with things like CentOS, RHEL, Ubuntu LTS and we lengthen the time that people are using these older tools by years, maybe even a decade.

* More people are using .tar.gz than .zip, which means changing from .tar.gz is more likely to cause issues (under the assumption that any change can cause issue, and the more people who have some sort of change occur to them, the more likely an issue is to occur).

Oh, and TIL that anyone who has Python 3.4+ installed has a command line tool for extracting ``.tar.gz`` files [2]


[1] Yes, some people *might* not have administrator access on their box and be unable to do that, but whoever *is* responsible for those boxes can.

[2] https://docs.python.org/3.4/library/tarfile.html#command-line-interface <https://docs.python.org/3.4/library/tarfile.html#command-line-interface>

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160819/38469b06/attachment-0001.html>

From chris.barker at noaa.gov  Fri Aug 19 18:41:03 2016
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Fri, 19 Aug 2016 15:41:03 -0700
Subject: [Distutils] What role to eggs still play?
In-Reply-To: <CAG8k2+5UYQ=dJBuVr-SjFU_=+h7GR60pp7bBwZNnrxx-QV0D9g@mail.gmail.com>
References: <CALGmxE+=4-gqgbAR1uGODKR1C-zjrSCEyBXOqp-2x_ypAeb8aw@mail.gmail.com>
 <CAG8k2+4W9Y3qhpBBhDXcO6wPDkjJUDDJdcQ=YZGEidOXXcbxdw@mail.gmail.com>
 <CAG8k2+5UYQ=dJBuVr-SjFU_=+h7GR60pp7bBwZNnrxx-QV0D9g@mail.gmail.com>
Message-ID: <3388930217277504347@unknownmsgid>

Thanks, I think I'm getting it.

About the toml file... the *-info metadata is a compiled artifact,
according to all the existing Python packages. Most sdists even have a
*.egg-info directory.


If it's a compiled artifact, then shouldn't it NOT be in a source dist?

It is inconvenient if you want to know the true dependencies without
running setup.py.


Isn't that what the toml file is for?

I think we are stuck with it, and it's not all bad - if there is some
useful metadata that doesn't affect the dependency resolver, and it is "too
static" or too cumbersome to write out by hand, there's a place to do that.


I'm trying to imagine what that info would be,  but I suppose there could
be meta data about a package that is generated at build time -- maybe info
about how it was built, for instance.

Hmm -- maybe you could put info in there about non-Python shared libs it's
linked to, for instance.

Eggs are the only way to add a zipped distribution to PYTHONPATH and have
> setuptools find the metadata
>

Can pip find it in a zipped package? Remember,  I don't care to support
setuptools only features anyway :-)

. Eggs are used by buildout, especially in the unzipped into a directory
> form
>
And they could still be used for their originally designed use as a plugin
> format.
>

If pkg_resources gets spun off, would it support that?

 *.egg-info or EGG-INFO is the predecessor of the *.dist-info format
> designed in PEP-376. You get *.egg-info whenever you install something
> with setuptools without going through bdist_wheel.
>

So setuptools_lite would write a  dist_info.

I take it pip looks for both?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160819/b8d15e10/attachment.html>

From dholth at gmail.com  Fri Aug 19 18:54:16 2016
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 19 Aug 2016 22:54:16 +0000
Subject: [Distutils] What role to eggs still play?
In-Reply-To: <3388930217277504347@unknownmsgid>
References: <CALGmxE+=4-gqgbAR1uGODKR1C-zjrSCEyBXOqp-2x_ypAeb8aw@mail.gmail.com>
 <CAG8k2+4W9Y3qhpBBhDXcO6wPDkjJUDDJdcQ=YZGEidOXXcbxdw@mail.gmail.com>
 <CAG8k2+5UYQ=dJBuVr-SjFU_=+h7GR60pp7bBwZNnrxx-QV0D9g@mail.gmail.com>
 <3388930217277504347@unknownmsgid>
Message-ID: <CAG8k2+631wZMyrnWGnRd1gNTckTgJMUhx4-kS+jDioDa0pFvZg@mail.gmail.com>

Pip uses pkg-resources to enumerate the installed packages.

An advantage of building the installation metadata is when you decide to
change the format, like by adding toml for example ?

On Fri, Aug 19, 2016, 18:41 Chris Barker - NOAA Federal <
chris.barker at noaa.gov> wrote:

> Thanks, I think I'm getting it.
>
> About the toml file... the *-info metadata is a compiled artifact,
> according to all the existing Python packages. Most sdists even have a
> *.egg-info directory.
>
>
> If it's a compiled artifact, then shouldn't it NOT be in a source dist?
>
> It is inconvenient if you want to know the true dependencies without
> running setup.py.
>
>
> Isn't that what the toml file is for?
>
> I think we are stuck with it, and it's not all bad - if there is some
> useful metadata that doesn't affect the dependency resolver, and it is "too
> static" or too cumbersome to write out by hand, there's a place to do that.
>
>
> I'm trying to imagine what that info would be,  but I suppose there could
> be meta data about a package that is generated at build time -- maybe info
> about how it was built, for instance.
>
> Hmm -- maybe you could put info in there about non-Python shared libs it's
> linked to, for instance.
>
> Eggs are the only way to add a zipped distribution to PYTHONPATH and have
>> setuptools find the metadata
>>
>
> Can pip find it in a zipped package? Remember,  I don't care to support
> setuptools only features anyway :-)
>
> . Eggs are used by buildout, especially in the unzipped into a directory
>> form
>>
> And they could still be used for their originally designed use as a plugin
>> format.
>>
>
> If pkg_resources gets spun off, would it support that?
>
>  *.egg-info or EGG-INFO is the predecessor of the *.dist-info format
>> designed in PEP-376. You get *.egg-info whenever you install something
>> with setuptools without going through bdist_wheel.
>>
>
> So setuptools_lite would write a  dist_info.
>
> I take it pip looks for both?
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160819/8501007d/attachment.html>

From nad at python.org  Fri Aug 19 16:33:03 2016
From: nad at python.org (Ned Deily)
Date: Fri, 19 Aug 2016 16:33:03 -0400
Subject: [Distutils] Proposed new Distutils API for compiler flag detection
 (Issue26689)
Message-ID: <A45B9ADE-2FB4-4453-AACE-8B1595C009E3@python.org>

Some months ago, Sylvain brought up a couple of proposals for Distutils.  The second proposal received some discussion but it appears that the first one got lost.  Here it is:

> From sylvain.corlay at gmail.com  Wed May 25 12:01:51 2016
> From: sylvain.corlay at gmail.com (Sylvain Corlay)
> Date: Wed, 25 May 2016 12:01:51 -0400
> Subject: [Distutils] Distutils improvements regarding header installation
>  and building C extension modules
> Message-ID: <CAK=Phk4tg_S2mVS2eAkLK514Aq_gq26yEg_vWwAEz8NnstypMg at mail.gmail.com>
> 
> Hello everyone,
> 
> This is my first post here so, apologies if I am breaking any rules.
> 
> Lately, I have been filing a few bug reports and patches to distutils on
> bugs.python.org that all concern the installation and build of C++
> extensions.
> 
> *1) The distutils.ccompiler has_flag method.*
> (http://bugs.python.org/issue26689)
> 
> When building C++ extension modules that require a certain compiler flag
> (such as enabling C++11 features), you may want to check if the compiler
> has such a flag available.
> 
> I proposed a patch adding a `has_flag` method to ccompiler. It is an
> equivalent to cmake' s CHECK_CXX_COMPILER_FLAG.
> 
> The implementation is similar to the one of has_function which by the way
> has a pending patch by minrk in issue (http://bugs.python.org/issue25544).

On python-dev and in the bug tracker, Sylvain has understandably asked for a review with an eye to adding this new feature to Python 3.6 whose feature code cutoff is scheduled for a few weeks from now.  As release manager, I am not opposed in general to adding new features to Distutils but I think we should be very cautious about modifying or adding new Distutils APIs, given that many third-party distribution authors want to support their packages on multiple versions.  So I want to make sure that there is some agreement that adding this new API starting with 3.6 is a good thing to do rather than having it go in under the radar.  If there are technical review issues with the implementation, it would probably be better to give those directly on the bug tracker.

Opinions?

Thanks!
--Ned

--
  Ned Deily
  nad at python.org -- []


From brett at python.org  Sat Aug 20 12:33:43 2016
From: brett at python.org (Brett Cannon)
Date: Sat, 20 Aug 2016 16:33:43 +0000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <DAA65B69-8246-4E3A-A482-5BF77E60FD44@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <DAA65B69-8246-4E3A-A482-5BF77E60FD44@stufft.io>
Message-ID: <CAP1=2W6TdvnTFu4hqonU3dO_xAxvJX=hoigSGEPNY0YV_MB5mg@mail.gmail.com>

On Fri, 19 Aug 2016 at 12:53 Donald Stufft <donald at stufft.io> wrote:

>
> On Aug 19, 2016, at 2:00 PM, Leonardo Rochael Almeida <
> leorochael at gmail.com> wrote:
>
> ure, more people will be affected this way than just the folks releasing
> on Windows, but given the shortcuts for setting the sdist format per
> project or per home directory that Donald mentioned, I think the collective
> effort in the migration will be smaller than the continuous effort of
> explaining to newcomers that the reason we use a .tar.gz based format for
> sdists versus a .zip based format for wheels is some historical accident,
> specially if we plan to change sdists back to .zip format in the future...
>
>
>
> I don?t think I?ve ever had anyone ask me why people (generally) use
> .tar.gz for sdists and why wheels use zip, though I don?t do much beginner
> stuff. Do you have some sort of evidence or data to suggest that this is a
> problem that people are experiencing or are you theorizing that folks
> *might* get confused by this?
>
> I think the effort changing non-Windows platform is going to be a lot more
> effort than changing Windows platform for a few reasons:
>
> * There are less people releasing on Windows than on non-Windows, the more
> people you need to migrate to a new thing, the longer you can expect it to
> take.
>
> * Windows does not come with Python, thus people are generally free to
> upgrade their Python or setuptools installation at all [1] meanwhile Python
> and setuptools tends to be a core part of non-Windows OSs where upgrading
> one or the other can ?void the warranty? and cause breakage to the entire
> OS, combine this with things like CentOS, RHEL, Ubuntu LTS and we lengthen
> the time that people are using these older tools by years, maybe even a
> decade.
>
> * More people are using .tar.gz than .zip, which means changing from
> .tar.gz is more likely to cause issues (under the assumption that any
> change can cause issue, and the more people who have some sort of change
> occur to them, the more likely an issue is to occur).
>
> Oh, and TIL that anyone who has Python 3.4+ installed has a command line
> tool for extracting ``.tar.gz`` files [2]
>

So I think you're both right, but at different time scales. :) I think
Donald is right that the short-term time scale of "now" by suggesting we
just go with tar.gz since it has the numbers. But I think Leonardo's point
of general alignment with zip for packaging overall is good for the
"formally define sdist" time scale and we potentially introduce an .sdist
file extension.

IOW I think we're all starting to talk in circles and since no one has
screamed "the sky is falling" against the proposal, it's probably time to
start formalizing a plan.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160820/38ec09e9/attachment.html>

From ncoghlan at gmail.com  Sat Aug 20 13:30:18 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 21 Aug 2016 03:30:18 +1000
Subject: [Distutils] Future of setuptools and buildout
In-Reply-To: <CALGmxEJ0K8AmRS3GhGg+bV67iEhzXcMCgqgpaBthRua=naer3g@mail.gmail.com>
References: <CAMxX_oP-8R=Gr4Jp9P8CXNSB7muuq=2z8E0K_V7qCvFodJTJOQ@mail.gmail.com>
 <EED02033-FB4E-4B25-A37B-C09B4CDF5F31@stufft.io>
 <CALGmxEJZBb=eakpUPWwqy+Wa=K+EoCvqi+sPgf4cPx-4+1m0yA@mail.gmail.com>
 <CAG8k2+7_DbueMsrokPMwbrzhzq8-M2t8QSOZN1-O3PJKvCh17Q@mail.gmail.com>
 <CALGmxE+mMuJ8P5PP9bbSuUQEx06PwH3hEdo7aYr_RBbY3_NZ3g@mail.gmail.com>
 <CACfEFw9aPsX-+yHkNBmbXfUkWkC-7ayZcQyOLWgm+3w14GYhkg@mail.gmail.com>
 <CALGmxEJ0K8AmRS3GhGg+bV67iEhzXcMCgqgpaBthRua=naer3g@mail.gmail.com>
Message-ID: <CADiSq7eX6_sEKB5kA4dHgszb--9nqhFhmk=sGy-R3934OHaFWg@mail.gmail.com>

On 20 August 2016 at 02:46, Chris Barker <chris.barker at noaa.gov> wrote:
> Maybe the alternative to setuptools_lite is to set up one of these other
> tools to work well (easily) for building python packages. If it can do
> everything that setuptools can do (that we want setuptools to do), and just
> as easily, maybe that's the next step forward.
>
> but if that isn't happening soon, then a setuptools_lite would be a useful
> step forward.

Enabling folks to more easily use existing build systems like Scons,
Waf, Meson, CMake, et al is one of the main goals of pyproject.toml.

Daniel Holth has a working Scons proof of concept at
https://pypi.python.org/pypi/enscons that actually works independently
of pyproject.toml support in pip by using a boilerplate setup.py to do
the bootstrapping and invocation of the build system.

It's why we don't particularly want to replace distutils/setuptools
with any build system in particular - the world actually has a surfeit
of cross-platform build systems, and many of them are written in
Python and are hence quite friendly to being bootstrapped for use in
building Python extension modules.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From jim at jimfulton.info  Sat Aug 20 13:45:44 2016
From: jim at jimfulton.info (Jim Fulton)
Date: Sat, 20 Aug 2016 13:45:44 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAMxX_oOEKxNy45LgTwuQogSDoLrTLD_VUY2QHH0A8orO15Phig@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CACac1F8LSnH_7=vORDxWdcqMs2=hZ9Vi=MrQ7uF4m6tWy3Fyiw@mail.gmail.com>
 <CAG8k2+61eY685hZPxoK6H34gn1oxz1sOFC1aom_0fEDbmjhjWw@mail.gmail.com>
 <CAMxX_oOEKxNy45LgTwuQogSDoLrTLD_VUY2QHH0A8orO15Phig@mail.gmail.com>
Message-ID: <CAPDm-Fh-B4PYvXddeDxoSFfrdBb_KCjEZ9LMNXz_9rT3zDMh9Q@mail.gmail.com>

On Tue, Aug 16, 2016 at 11:15 AM, Leonardo Rochael Almeida <
leorochael at gmail.com> wrote:

...


> A wheel2egg might be nice, but unless it's integrated into setuptools
> proper, it would mean adding another dependency to buildout and the
> developers would rather depend on a single library for talking to PyPI
>

I'm not sure this is as important as it once was.  Depending on setuptools
and it's bootstrapping mechanism easier, but setuptools' bootstrapping
functionality is going away. We've discussed giving up on allowing buildout
to bootstrap itself without modifying the Python install, although I still
rather like that feature.

Jim

-- 
Jim Fulton
http://jimfulton.info
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160820/adcbfdc9/attachment.html>

From ncoghlan at gmail.com  Sat Aug 20 13:46:57 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 21 Aug 2016 03:46:57 +1000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAP1=2W6TdvnTFu4hqonU3dO_xAxvJX=hoigSGEPNY0YV_MB5mg@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <DAA65B69-8246-4E3A-A482-5BF77E60FD44@stufft.io>
 <CAP1=2W6TdvnTFu4hqonU3dO_xAxvJX=hoigSGEPNY0YV_MB5mg@mail.gmail.com>
Message-ID: <CADiSq7e4+jt9QjP8z3rVMqN5KCcPyTo7RNq7DPxB06x-e6bh8Q@mail.gmail.com>

On 21 August 2016 at 02:33, Brett Cannon <brett at python.org> wrote:
> On Fri, 19 Aug 2016 at 12:53 Donald Stufft <donald at stufft.io> wrote:
> So I think you're both right, but at different time scales. :) I think
> Donald is right that the short-term time scale of "now" by suggesting we
> just go with tar.gz since it has the numbers. But I think Leonardo's point
> of general alignment with zip for packaging overall is good for the
> "formally define sdist" time scale and we potentially introduce an .sdist
> file extension.

I think tarballs are a better long term release archiving format at
any time scale - they just assume you're going to untar them before
trying to do anything useful with them, which is a reasonable workflow
requirement to impose for sdists. While relatively few people actually
need to work directly with tarballs in that context, that's just
because they've faded into the background of various build systems
that produce some other format (e.g. RPMs, deb archives, Docker
container layers).

By contrast, for Python's built formats like wheel and egg, it's
useful to be able to easily access their contents *without* unpacking
them first, so it makes sense to go with the format that's friendlier
to that model (i.e. zip).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From cournape at gmail.com  Sat Aug 20 13:55:12 2016
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 20 Aug 2016 18:55:12 +0100
Subject: [Distutils] Future of setuptools and buildout
In-Reply-To: <CADiSq7eX6_sEKB5kA4dHgszb--9nqhFhmk=sGy-R3934OHaFWg@mail.gmail.com>
References: <CAMxX_oP-8R=Gr4Jp9P8CXNSB7muuq=2z8E0K_V7qCvFodJTJOQ@mail.gmail.com>
 <EED02033-FB4E-4B25-A37B-C09B4CDF5F31@stufft.io>
 <CALGmxEJZBb=eakpUPWwqy+Wa=K+EoCvqi+sPgf4cPx-4+1m0yA@mail.gmail.com>
 <CAG8k2+7_DbueMsrokPMwbrzhzq8-M2t8QSOZN1-O3PJKvCh17Q@mail.gmail.com>
 <CALGmxE+mMuJ8P5PP9bbSuUQEx06PwH3hEdo7aYr_RBbY3_NZ3g@mail.gmail.com>
 <CACfEFw9aPsX-+yHkNBmbXfUkWkC-7ayZcQyOLWgm+3w14GYhkg@mail.gmail.com>
 <CALGmxEJ0K8AmRS3GhGg+bV67iEhzXcMCgqgpaBthRua=naer3g@mail.gmail.com>
 <CADiSq7eX6_sEKB5kA4dHgszb--9nqhFhmk=sGy-R3934OHaFWg@mail.gmail.com>
Message-ID: <CAGY4rcUWkkZmg+4g=FgyCCknru20v4PewmLSO5Zd1h_xh5NzmQ@mail.gmail.com>

On Sat, Aug 20, 2016 at 6:30 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 20 August 2016 at 02:46, Chris Barker <chris.barker at noaa.gov> wrote:
> > Maybe the alternative to setuptools_lite is to set up one of these other
> > tools to work well (easily) for building python packages. If it can do
> > everything that setuptools can do (that we want setuptools to do), and
> just
> > as easily, maybe that's the next step forward.
> >
> > but if that isn't happening soon, then a setuptools_lite would be a
> useful
> > step forward.
>
> Enabling folks to more easily use existing build systems like Scons,
> Waf, Meson, CMake, et al is one of the main goals of pyproject.toml.
>
> Daniel Holth has a working Scons proof of concept at
> https://pypi.python.org/pypi/enscons that actually works independently
> of pyproject.toml support in pip by using a boilerplate setup.py to do
> the bootstrapping and invocation of the build system.
>
> It's why we don't particularly want to replace distutils/setuptools
> with any build system in particular - the world actually has a surfeit
> of cross-platform build systems, and many of them are written in
> Python and are hence quite friendly to being bootstrapped for use in
> building Python extension modules.
>

Indeed. Having been involved w/ the mess that is numpy.distutils, this was
(and still is) the major issue w/ distutils for the scientific/pydata crowd.

Conflating packaging and building is an impediment to deploy software to
different platforms (build is generally better handled by developers,
packaging by platform-specific packagers).

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160820/133720b6/attachment.html>

From jim at jimfulton.info  Sat Aug 20 14:00:16 2016
From: jim at jimfulton.info (Jim Fulton)
Date: Sat, 20 Aug 2016 14:00:16 -0400
Subject: [Distutils] Why I like eggs (or similar mechanisms) and my thoughts
 on future of buildout
Message-ID: <CAPDm-FjTfzBSnFkd39hrkNqQEKcL4RW5qD-MFfMda+p=mym20A@mail.gmail.com>

Buildout was designed to automate project development and deployment tasks.
This encompasses assembly of software packages (Python, JavaScript, etc.)
and generation of support artifacts such as configuration files and scripts.

I never set out to make buildout a Python package installer.

I adopted setuptools because it enabled most of what I needed in terms of
fetching and assembling Python packages.  Adopting eggs was a by product of
adopting setuptools.

Some basic goals of buildout:


   - Isolation from system-provided Python installations.  This was a
   result of problems experienced when deploying applications for customers
   whos Python environments were different from ours (Zope Corporation's).

   (Thankfully, Docker mitigates these issues quite a bit.)


   - Repeatability -- Given the same buildout configuration file, running
   buildouts in separate workspaces should product the same result for a given
   platform, regardless of their history.

   One consequence of this is that it's important to be able uninstall
   things cleanly, both artifacts generated by buildout and packages installed
   by it.

   - Tight control over program dependencies.  This means that different
   programs (scripts) in a buildout could depend on different versions and
   maybe more importantly, dependencies didn't leak from one program to
   another.

   Expanding on this last point, it's common for libraries to change their
   behavior depending on whether another library is importable.  With a shared
   set of packages, installing a new program could cause introduction of
   packages that change the behavior of existing programs.  This would be bad.


These really weren't goals of setuptools, which tried to fit into
site-package-based installs and ironically resorted to unsavory techniques
to wedge eggs in using elaborate .pth files.

With buildout, I chose to use eggs differently.  I simply generate scripts
with dependencies explicitly listed in the Python path.  This is very easy
to reason about.  It makes it easy to see what's installed by looking at
the generated path.  This is similar to the way Java uses class paths to
assemble jar files, which is fitting given that the design of eggs took at
least some inspiration from jar files.

I'm not a fan of Java, but I think class paths + jar files was something it
got right.  Each program has it's own distinct environment.  If you need to
add/remove/update a dependency, just change the path.  Want to share
installed components between programs?  It's easy because only one
installed component needs to be stored and different programs point to it
via their paths.

Pip's approach OTOH, leaves me skeptical.  When installing a package, it
has to unpack it and install its files into a shared directory structure
where conflicts are likely.  How does it handle these conflicts? IDK. I
should, but I don't.  I have the impression that uninstalling things can be
problematic, but maybe that's been fixed. At best, this is a lot of
complexity to maintain, at worse, uninstalls/reinstalls leave things behind
that make buildout's goal of repeatability impossible to achieve.

For isolation, pip relies on virtualenv.  This has always struck me as an
extremely heavy-handed approach. I'm quite surprised that the Python
community settled on it.  But whatever. A few packages (e.g. NumPy) really
depend on the shape of the environment they're installed into so can't be
installed with buildout, but can be installed with pip+virtualenv.

The buildout developers have discussed options for the future. We know
there's a reckoning coming, but so far, thankfully, we've been able to put
it off, but we don't want to be a burden on the rest of the Python
community. (Seriously, thank you for not breaking us. :) )

We've debated just invoking virtualenv and pip to assemble Python
applications.  A model we've been discussing is to let buildout recipes do
this.  No change is needed to buildout.  There's at least one and probably
multiple recipes that do this, although I haven't used them myself. In this
model, a buildout could install different virtualenvs for different
programs, allowing dependencies to be kept distinct.  I still worry about
the integrity of these virtualenvs over time as dependencies are added,
removed, and updated.

If I could have my way, the path of distinct package directories approach
would still be an option for buildout, as I think it's superior.  I'm
hopeful that it will be possible to use wheels this way, assuming that eggs
are retired.

I would also prefer that there be one library, or set of complementary
libraries, to find, download and install packages.  I normally like
competition, but this is such a boring and difficult domain that I don't
really see there being interest in developing multiple solutions. Honestly,
I'd be nervous if, in the long run, buildout and pip installed things
differently, especially given security considerations.

In the long run, I suspect it would be best for some buildout developers to
offer pip PRs that abstract functionality into libraries that buildout
could call (and that pip called internally), although it sounds like this
may already be happening without our help.

Jim

-- 
Jim Fulton
http://jimfulton.info
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160820/f4e3c55b/attachment.html>

From ncoghlan at gmail.com  Sat Aug 20 15:02:39 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 21 Aug 2016 05:02:39 +1000
Subject: [Distutils] Why I like eggs (or similar mechanisms) and my
 thoughts on future of buildout
In-Reply-To: <CAPDm-FjTfzBSnFkd39hrkNqQEKcL4RW5qD-MFfMda+p=mym20A@mail.gmail.com>
References: <CAPDm-FjTfzBSnFkd39hrkNqQEKcL4RW5qD-MFfMda+p=mym20A@mail.gmail.com>
Message-ID: <CADiSq7fVGNdJFrX3xqN437QiwyE3p4Kv_sNAtv7KGtSb97k0SA@mail.gmail.com>

On 21 August 2016 at 04:00, Jim Fulton <jim at jimfulton.info> wrote:
> These really weren't goals of setuptools, which tried to fit into
> site-package-based installs and ironically resorted to unsavory techniques
> to wedge eggs in using elaborate .pth files.

Right, and pip took that approach further by making the site-packages
friendly approach the *default* approach, rather than a selectable
option.

This has had the effect of making pip not only useful for component
management in its own right, but also viable as a tool for assembling
*downstream* packages from Python source projects.

> With buildout, I chose to use eggs differently.  I simply generate scripts
> with dependencies explicitly listed in the Python path.  This is very easy
> to reason about.  It makes it easy to see what's installed by looking at the
> generated path.  This is similar to the way Java uses class paths to
> assemble jar files, which is fitting given that the design of eggs took at
> least some inspiration from jar files.
>
> I'm not a fan of Java, but I think class paths + jar files was something it
> got right.  Each program has it's own distinct environment.  If you need to
> add/remove/update a dependency, just change the path.  Want to share
> installed components between programs?  It's easy because only one installed
> component needs to be stored and different programs point to it via their
> paths.

Agreed, but similar to both conda and JARs themselves, this improved
isolation made buildout *less* useful to folks working on distro
packages that actually *wanted* to be installing their Python
components into the system Python installation.

> Pip's approach OTOH, leaves me skeptical.  When installing a package, it has
> to unpack it and install its files into a shared directory structure where
> conflicts are likely.  How does it handle these conflicts? IDK. I should,
> but I don't.

It doesn't, since we don't have full file manifests in our metadata -
if you inadvertently install both python-openid and python3-openid
into the same virtualenv, they'll trample over each other's files.

> I have the impression that uninstalling things can be
> problematic, but maybe that's been fixed.

Uninstallation is fine, as we *do* have a full file manifest after a
component has been installed.

> At best, this is a lot of
> complexity to maintain, at worse, uninstalls/reinstalls leave things behind
> that make buildout's goal of repeatability impossible to achieve.
>
> For isolation, pip relies on virtualenv.  This has always struck me as an
> extremely heavy-handed approach. I'm quite surprised that the Python
> community settled on it.  But whatever.

The pay-off for the pip model comes in the fact that using venv is
*optional* in a way that isn't generally true for other more
specifically app focused systems like conda and buildout. That doesn't
make conda and buildout wrong - it means they cut out a particular
rare-in-number-but-large-in-influence use case (Linux based operating
system development) in order to better focus on a more targeted set of
use cases (data analysis for conda, network service development for
buildout)

> A few packages (e.g. NumPy) really
> depend on the shape of the environment they're installed into so can't be
> installed with buildout, but can be installed with pip+virtualenv.

The distinction also goes the other way - pip can be used to add
capabilities to an existing Python installation without fundamentally
changing the architecture of that installation, including how it
decides to handle (or not handle, as the case may be) separating
different applications from each other.

The key thing that pip's downstream platform tend to bring to the
table is different answers to that application isolation problem,
while pip just handles dependency management and build system
invocation:

- virtualenv mainly just manipulates sys.path to amend where
site-packages is found
- Linux distros offer their system package management (including full
preinstall file manifests and associated conflict detection) as well
as chroots and Linux containers (most famously, Docker)
- *nix systems have also long offered the "modules" environment
management utility (especially popular in HPC)
- buildout has tailored per-application sys.path definitions
- conda has its own environment management tooling

> The buildout developers have discussed options for the future. We know
> there's a reckoning coming, but so far, thankfully, we've been able to put
> it off, but we don't want to be a burden on the rest of the Python
> community. (Seriously, thank you for not breaking us. :) )

I don't think buildout's a burden. While there's no logical reason for
environment isolation to be tightly coupled to dependency management
and build system invocation, when designing buildout you didn't have a
choice - setuptools was realistically the only game in town for those
two pieces, so it made sense to structure buildout around that, and
then diverge only on the isolation management side of things.

> We've debated just invoking virtualenv and pip to assemble Python
> applications.  A model we've been discussing is to let buildout recipes do
> this.  No change is needed to buildout.  There's at least one and probably
> multiple recipes that do this, although I haven't used them myself. In this
> model, a buildout could install different virtualenvs for different
> programs, allowing dependencies to be kept distinct.  I still worry about
> the integrity of these virtualenvs over time as dependencies are added,
> removed, and updated.

As a competing approach to isolation management, I doubt it would make
much sense for buildout to adopt virtualenv - it seems more logical to
me to keep your current isolation model (which has a lot to recommend
it), and instead look just to swapping in pip to replace setuptools
and easy_install for the dependency management and build system
invocation pieces.

> If I could have my way, the path of distinct package directories approach
> would still be an option for buildout, as I think it's superior.

I don't see any reason for buildout to drop the per-application
sys.path customisation approach in favour of venv's - there's nothing
wrong with it, and it avoids several of the problems that can arise
with a proliferation of venv's.

>  I'm
> hopeful that it will be possible to use wheels this way, assuming that eggs
> are retired.

Wheels can already be used this way - the "officially not supported"
aspect is using them as sys.path entries without unpacking them first.
However, eggs also won't be retired until there's a comparable
documented format that officially supports usage with zipimport (as
opposed to wheel's status where it works if you know how to make it
work, but there aren't any formal guarantees that what works today in
that respect will continue working tomorrow)

> I would also prefer that there be one library, or set of complementary
> libraries, to find, download and install packages.  I normally like
> competition, but this is such a boring and difficult domain that I don't
> really see there being interest in developing multiple solutions. Honestly,
> I'd be nervous if, in the long run, buildout and pip installed things
> differently, especially given security considerations.

For several of the core pieces shared between pip and Warehouse,
Donald has already broken out https://packaging.pypa.io

> In the long run, I suspect it would be best for some buildout developers to
> offer pip PRs that abstract functionality into libraries that buildout could
> call (and that pip called internally), although it sounds like this may
> already be happening without our help.

Only for the pip/Warehouse common components, since Donald is driving
the extraction of common requirements for those two projects.

However, there's another important activity along similar lines that
doesn't have anyone that I'm aware of actively pursuing it, which is
pulling more of the pieces of pip's PyPI client and local installation
management behaviour out into a more readily re-usable form.

While Vinay Sajip's distlib (https://pypi.python.org/pypi/distlib )
already covers a lot of that, what's currently missing is folks
looking at the common capabilities of pip and distlib to ensure that
they're actually behaving the same way, with robust test suites to
ensure they're also following the relevant standards (or else that we
update the relevant standards to match what people are actually
doing).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From jim at jimfulton.info  Sat Aug 20 15:46:50 2016
From: jim at jimfulton.info (Jim Fulton)
Date: Sat, 20 Aug 2016 15:46:50 -0400
Subject: [Distutils] Why I like eggs (or similar mechanisms) and my
 thoughts on future of buildout
In-Reply-To: <CADiSq7fVGNdJFrX3xqN437QiwyE3p4Kv_sNAtv7KGtSb97k0SA@mail.gmail.com>
References: <CAPDm-FjTfzBSnFkd39hrkNqQEKcL4RW5qD-MFfMda+p=mym20A@mail.gmail.com>
 <CADiSq7fVGNdJFrX3xqN437QiwyE3p4Kv_sNAtv7KGtSb97k0SA@mail.gmail.com>
Message-ID: <CAPDm-FhW6+t6FQMx5Wi33VSPukLLyk8D0AuWUsgGRCaTwEsRDQ@mail.gmail.com>

On Sat, Aug 20, 2016 at 3:02 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

...

> I have the impression that uninstalling things can be
>
> problematic, but maybe that's been fixed.
>
> Uninstallation is fine, as we *do* have a full file manifest after a
> component has been installed.
>

So, if package A and B install the same file, X, and then A is uninstalled,
is X still there?  If then B is uninstalled, does X go away?  Does this
machinery depend on whether X has the the same contents in A and B?

...

Jim

-- 
Jim Fulton
http://jimfulton.info
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160820/9141caef/attachment.html>

From dholth at gmail.com  Sat Aug 20 16:56:14 2016
From: dholth at gmail.com (Daniel Holth)
Date: Sat, 20 Aug 2016 20:56:14 +0000
Subject: [Distutils] Why I like eggs (or similar mechanisms) and my
 thoughts on future of buildout
In-Reply-To: <CAPDm-FhW6+t6FQMx5Wi33VSPukLLyk8D0AuWUsgGRCaTwEsRDQ@mail.gmail.com>
References: <CAPDm-FjTfzBSnFkd39hrkNqQEKcL4RW5qD-MFfMda+p=mym20A@mail.gmail.com>
 <CADiSq7fVGNdJFrX3xqN437QiwyE3p4Kv_sNAtv7KGtSb97k0SA@mail.gmail.com>
 <CAPDm-FhW6+t6FQMx5Wi33VSPukLLyk8D0AuWUsgGRCaTwEsRDQ@mail.gmail.com>
Message-ID: <CAG8k2+5TkVnEb3Y5iPufOXSP35LBSjsqS6XSBM+Pj8n8pgFqVg@mail.gmail.com>

We have manifests before and after in wheel but good conflict management is
still theoretical. I think someone had a prototype years ago. For example
checking checksums on uninstall and maybe not uninstalling modified
[configuration] files.

On Sat, Aug 20, 2016, 15:47 Jim Fulton <jim at jimfulton.info> wrote:

>
>
> On Sat, Aug 20, 2016 at 3:02 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> ...
>
> > I have the impression that uninstalling things can be
>>
> > problematic, but maybe that's been fixed.
>>
>> Uninstallation is fine, as we *do* have a full file manifest after a
>> component has been installed.
>>
>
> So, if package A and B install the same file, X, and then A is
> uninstalled, is X still there?  If then B is uninstalled, does X go away?
> Does this machinery depend on whether X has the the same contents in A and
> B?
>
> ...
>
> Jim
>
> --
> Jim Fulton
> http://jimfulton.info
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160820/e7745932/attachment.html>

From njs at pobox.com  Sat Aug 20 17:31:45 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Sat, 20 Aug 2016 14:31:45 -0700
Subject: [Distutils] Why I like eggs (or similar mechanisms) and my
 thoughts on future of buildout
In-Reply-To: <CAPDm-FjTfzBSnFkd39hrkNqQEKcL4RW5qD-MFfMda+p=mym20A@mail.gmail.com>
References: <CAPDm-FjTfzBSnFkd39hrkNqQEKcL4RW5qD-MFfMda+p=mym20A@mail.gmail.com>
Message-ID: <CAPJVwBn1SwHcYkg7HA9b4TqyL=OR_mE=rb-quJmQJyaH0Q3dbg@mail.gmail.com>

On Aug 20, 2016 11:00 AM, "Jim Fulton" <jim at jimfulton.info> wrote:
>
[...]
> With buildout, I chose to use eggs differently.  I simply generate
scripts with dependencies explicitly listed in the Python path.  This is
very easy to reason about.  It makes it easy to see what's installed by
looking at the generated path.  This is similar to the way Java uses class
paths to assemble jar files, which is fitting given that the design of eggs
took at least some inspiration from jar files.
>
> I'm not a fan of Java, but I think class paths + jar files was something
it got right.  Each program has it's own distinct environment.  If you need
to add/remove/update a dependency, just change the path.  Want to share
installed components between programs?  It's easy because only one
installed component needs to be stored and different programs point to it
via their paths.

Wheels are a pretty simple and straightforward format. They've got some
metadata, and then the are a set of directories with labels attached: "this
directory needs to go on sys.path", "this directory has scripts that should
have shebang fixups applied and then go on $PATH", etc. (The spec is not
long: https://www.python.org/dev/peps/pep-0427/)

My first guess at how buildout could handle wheels elegantly would be:

- we have one piece of code that abstracts the operation of "please give me
a wheel for package X version Y that works on this system". This then
contacts pypi, builds sdists if it has to, whatever. It could be in the
form of an invocation of pip, or it could be a call into some common
library that gets factored out of pip and used by both pip and buildout,
whatever.

- pip uses this code to get a wheel, and once it has a wheel, it installs
it into an environment in the same way it does today.

- buildout *also* uses this code to get a wheel, and once it has a wheel,
it does its own thing: maybe its own dependency resolution, and definitely
its own install strategy, where it unpacks the wheel's directories into its
own preferred layout, does sys.path munging on the wheel's scripts, etc.

AFAICT there's nothing in the wheel format that ties you to the
virtualenv-style directory layout.

The disadvantage is that someone would have to do the work to factor out
that get-a-wheel code, and to write/maintain custom wheel installation code
in buildout, instead of just invoking setuptools. The advantage would be
the now you aren't dependent on setuptools...

> The buildout developers have discussed options for the future. We know
there's a reckoning coming, but so far, thankfully, we've been able to put
it off, but we don't want to be a burden on the rest of the Python
community. (Seriously, thank you for not breaking us. :) )

As a heads up, there is unfortunately another source of breakage heading
down the pipe: the PEP 516/517 plans for making it easy to use
non-distutils-based build systems are heavily tied to wheels, so if/when
the spec gets finished and projects start adopting alternative build
systems, you may see an increasing proliferation of projects that cannot
build eggs or be easy_install'ed.

(Ref: https://www.python.org/dev/peps/pep-0517/)

It's not clear when this will happen exactly -- I'm in the midst of dealing
with some medical stuff, so haven't had much/any time for distutils-sig
stuff recently, but sooner or later this is the direction we want to go.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160820/020b3fbb/attachment.html>

From ralf.gommers at gmail.com  Sat Aug 20 17:42:42 2016
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sun, 21 Aug 2016 09:42:42 +1200
Subject: [Distutils] Proposed new Distutils API for compiler flag
 detection (Issue26689)
In-Reply-To: <A45B9ADE-2FB4-4453-AACE-8B1595C009E3@python.org>
References: <A45B9ADE-2FB4-4453-AACE-8B1595C009E3@python.org>
Message-ID: <CABL7CQi86ORbhQUy11g4ipiEWMKFYtzeeYrb1JkKomsVPa8uKQ@mail.gmail.com>

On Sat, Aug 20, 2016 at 8:33 AM, Ned Deily <nad at python.org> wrote:

> Some months ago, Sylvain brought up a couple of proposals for Distutils.
> The second proposal received some discussion but it appears that the first
> one got lost.  Here it is:
>
> > From sylvain.corlay at gmail.com  Wed May 25 12:01:51 2016
> > From: sylvain.corlay at gmail.com (Sylvain Corlay)
> > Date: Wed, 25 May 2016 12:01:51 -0400
> > Subject: [Distutils] Distutils improvements regarding header installation
> >  and building C extension modules
> > Message-ID: <CAK=Phk4tg_S2mVS2eAkLK514Aq_gq26yEg_vWwAEz8NnstypMg at mail.
> gmail.com>
> >
> > Hello everyone,
> >
> > This is my first post here so, apologies if I am breaking any rules.
> >
> > Lately, I have been filing a few bug reports and patches to distutils on
> > bugs.python.org that all concern the installation and build of C++
> > extensions.
> >
> > *1) The distutils.ccompiler has_flag method.*
> > (http://bugs.python.org/issue26689)
> >
> > When building C++ extension modules that require a certain compiler flag
> > (such as enabling C++11 features), you may want to check if the compiler
> > has such a flag available.
> >
> > I proposed a patch adding a `has_flag` method to ccompiler. It is an
> > equivalent to cmake' s CHECK_CXX_COMPILER_FLAG.
> >
> > The implementation is similar to the one of has_function which by the way
> > has a pending patch by minrk in issue (http://bugs.python.org/issue25544
> ).
>
> On python-dev and in the bug tracker, Sylvain has understandably asked for
> a review with an eye to adding this new feature to Python 3.6 whose feature
> code cutoff is scheduled for a few weeks from now.  As release manager, I
> am not opposed in general to adding new features to Distutils but I think
> we should be very cautious about modifying or adding new Distutils APIs,
> given that many third-party distribution authors want to support their
> packages on multiple versions.  So I want to make sure that there is some
> agreement that adding this new API starting with 3.6 is a good thing to do
> rather than having it go in under the radar.


I'd rather see that kind of thing added to setuptools. We're already having
to deal with setuptools as a moving target, so if distutils becomes one
again as well that means more testing with combinations of different Python
and setuptools versions. Imho distutils changes should be bugfix and
essential maintenance only.


> If there are technical review issues with the implementation, it would
> probably be better to give those directly on the bug tracker.
>

The usual one for disutils: it's a patch with zero tests and zero docs. It
looks pretty safe to add, but still....

Ralf




>
> Opinions?
>
> Thanks!
> --Ned
>
> --
>   Ned Deily
>   nad at python.org -- []
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160821/b316ecaf/attachment-0001.html>

From sylvain.corlay at gmail.com  Sat Aug 20 18:09:17 2016
From: sylvain.corlay at gmail.com (Sylvain Corlay)
Date: Sun, 21 Aug 2016 00:09:17 +0200
Subject: [Distutils] Proposed new Distutils API for compiler flag
 detection (Issue26689)
In-Reply-To: <CABL7CQi86ORbhQUy11g4ipiEWMKFYtzeeYrb1JkKomsVPa8uKQ@mail.gmail.com>
References: <A45B9ADE-2FB4-4453-AACE-8B1595C009E3@python.org>
 <CABL7CQi86ORbhQUy11g4ipiEWMKFYtzeeYrb1JkKomsVPa8uKQ@mail.gmail.com>
Message-ID: <CAK=Phk5YV_Ax34_DRV9b4ch4T7GNMZ2=E4DH=QP0DzeAA8v83A@mail.gmail.com>

Hi Ralf,

On Sat, Aug 20, 2016 at 11:42 PM, Ralf Gommers <ralf.gommers at gmail.com>
wrote:

>
>> On python-dev and in the bug tracker, Sylvain has understandably asked
>> for a review with an eye to adding this new feature to Python 3.6 whose
>> feature code cutoff is scheduled for a few weeks from now.  As release
>> manager, I am not opposed in general to adding new features to Distutils
>> but I think we should be very cautious about modifying or adding new
>> Distutils APIs, given that many third-party distribution authors want to
>> support their packages on multiple versions.  So I want to make sure that
>> there is some agreement that adding this new API starting with 3.6 is a
>> good thing to do rather than having it go in under the radar.
>
>
> I'd rather see that kind of thing added to setuptools. We're already
> having to deal with setuptools as a moving target, so if distutils becomes
> one again as well that means more testing with combinations of different
> Python and setuptools versions. Imho distutils changes should be bugfix and
> essential maintenance only.
>
>

Having the `has_flag` in a different location from `has_function` would be
weird in my opinion.

Sylvain
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160821/b7e8a106/attachment.html>

From wes.turner at gmail.com  Sat Aug 20 18:24:33 2016
From: wes.turner at gmail.com (Wes Turner)
Date: Sat, 20 Aug 2016 17:24:33 -0500
Subject: [Distutils] Why I like eggs (or similar mechanisms) and my
 thoughts on future of buildout
In-Reply-To: <CAG8k2+5TkVnEb3Y5iPufOXSP35LBSjsqS6XSBM+Pj8n8pgFqVg@mail.gmail.com>
References: <CAPDm-FjTfzBSnFkd39hrkNqQEKcL4RW5qD-MFfMda+p=mym20A@mail.gmail.com>
 <CADiSq7fVGNdJFrX3xqN437QiwyE3p4Kv_sNAtv7KGtSb97k0SA@mail.gmail.com>
 <CAPDm-FhW6+t6FQMx5Wi33VSPukLLyk8D0AuWUsgGRCaTwEsRDQ@mail.gmail.com>
 <CAG8k2+5TkVnEb3Y5iPufOXSP35LBSjsqS6XSBM+Pj8n8pgFqVg@mail.gmail.com>
Message-ID: <CACfEFw-ZuZVfyPYm9L+1tL93qAq4AvxRLWp9H9W-uVMGRP0uKA@mail.gmail.com>

On Saturday, August 20, 2016, Daniel Holth <dholth at gmail.com> wrote:

> We have manifests before and after in wheel but good conflict management
> is still theoretical. I think someone had a prototype years ago. For
> example checking checksums on uninstall and maybe not uninstalling modified
> [configuration] files.
>
- Uninstalling X would break Y and Z
- Because there are no additional transitive dependencies, uninstalling X
should also uninstall Y and Z
- ./etc/x.cfg is modified, don't remove it on_uninstall



>
> On Sat, Aug 20, 2016, 15:47 Jim Fulton <jim at jimfulton.info
> <javascript:_e(%7B%7D,'cvml','jim at jimfulton.info');>> wrote:
>
>>
>>
>> On Sat, Aug 20, 2016 at 3:02 PM, Nick Coghlan <ncoghlan at gmail.com
>> <javascript:_e(%7B%7D,'cvml','ncoghlan at gmail.com');>> wrote:
>>
>> ...
>>
>> > I have the impression that uninstalling things can be
>>>
>> > problematic, but maybe that's been fixed.
>>>
>>> Uninstallation is fine, as we *do* have a full file manifest after a
>>> component has been installed.
>>>
>>
>> So, if package A and B install the same file, X, and then A is
>> uninstalled, is X still there?  If then B is uninstalled, does X go away?
>> Does this machinery depend on whether X has the the same contents in A and
>> B?
>>
>> ...
>>
>> Jim
>>
>> --
>> Jim Fulton
>> http://jimfulton.info
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> <javascript:_e(%7B%7D,'cvml','Distutils-SIG at python.org');>
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160820/8c6adce5/attachment.html>

From wes.turner at gmail.com  Sat Aug 20 18:31:32 2016
From: wes.turner at gmail.com (Wes Turner)
Date: Sat, 20 Aug 2016 17:31:32 -0500
Subject: [Distutils] Why I like eggs (or similar mechanisms) and my
 thoughts on future of buildout
In-Reply-To: <CAPJVwBn1SwHcYkg7HA9b4TqyL=OR_mE=rb-quJmQJyaH0Q3dbg@mail.gmail.com>
References: <CAPDm-FjTfzBSnFkd39hrkNqQEKcL4RW5qD-MFfMda+p=mym20A@mail.gmail.com>
 <CAPJVwBn1SwHcYkg7HA9b4TqyL=OR_mE=rb-quJmQJyaH0Q3dbg@mail.gmail.com>
Message-ID: <CACfEFw-zuh5egvM_0L_ThZDYqq-06XqbS0mpFVFVwTROcSoGrA@mail.gmail.com>

On Saturday, August 20, 2016, Nathaniel Smith <njs at pobox.com> wrote:

> On Aug 20, 2016 11:00 AM, "Jim Fulton" <jim at jimfulton.info
> <javascript:_e(%7B%7D,'cvml','jim at jimfulton.info');>> wrote:
> >
> [...]
> > With buildout, I chose to use eggs differently.  I simply generate
> scripts with dependencies explicitly listed in the Python path.  This is
> very easy to reason about.  It makes it easy to see what's installed by
> looking at the generated path.  This is similar to the way Java uses class
> paths to assemble jar files, which is fitting given that the design of eggs
> took at least some inspiration from jar files.
> >
> > I'm not a fan of Java, but I think class paths + jar files was something
> it got right.  Each program has it's own distinct environment.  If you need
> to add/remove/update a dependency, just change the path.  Want to share
> installed components between programs?  It's easy because only one
> installed component needs to be stored and different programs point to it
> via their paths.
>
> Wheels are a pretty simple and straightforward format. They've got some
> metadata, and then the are a set of directories with labels attached: "this
> directory needs to go on sys.path", "this directory has scripts that should
> have shebang fixups applied and then go on $PATH", etc. (The spec is not
> long: https://www.python.org/dev/peps/pep-0427/)
>
> My first guess at how buildout could handle wheels elegantly would be:
>
> - we have one piece of code that abstracts the operation of "please give
> me a wheel for package X version Y that works on this system". This then
> contacts pypi, builds sdists if it has to, whatever. It could be in the
> form of an invocation of pip, or it could be a call into some common
> library that gets factored out of pip and used by both pip and buildout,
> whatever.
>
> - pip uses this code to get a wheel, and once it has a wheel, it installs
> it into an environment in the same way it does today.
>
> - buildout *also* uses this code to get a wheel, and once it has a wheel,
> it does its own thing: maybe its own dependency resolution, and definitely
> its own install strategy, where it unpacks the wheel's directories into its
> own preferred layout, does sys.path munging on the wheel's scripts, etc.
>
> AFAICT there's nothing in the wheel format that ties you to the
> virtualenv-style directory layout.
>
> The disadvantage is that someone would have to do the work to factor out
> that get-a-wheel code, and to write/maintain custom wheel installation code
> in buildout, instead of just invoking setuptools. The advantage would be
> the now you aren't dependent on setuptools...
>
> > The buildout developers have discussed options for the future. We know
> there's a reckoning coming, but so far, thankfully, we've been able to put
> it off, but we don't want to be a burden on the rest of the Python
> community. (Seriously, thank you for not breaking us. :) )
>

I may be optimistic, but should adding buildout support for wheel be more
complicated than shelling out to pip with the correct, uhm, --prefix etc?

... recipes for buildout & pip, omelette symlinks:
http://www.mail-archive.com/distutils-sig at python.org/msg24964.html
Re: [Distutils] Future of setuptools and buildout
<http://www.mail-archive.com/search?l=distutils-sig at python.org&q=subject:%22Re%5C%3A+%5C%5BDistutils%5C%5D+Future+of+setuptools+and+buildout%22&o=newest>

Re: buildout and pip and wheel

"Add support for installing
wheels"https://github.com/buildout/buildout/issues/144






> As a heads up, there is unfortunately another source of breakage heading
> down the pipe: the PEP 516/517 plans for making it easy to use
> non-distutils-based build systems are heavily tied to wheels, so if/when
> the spec gets finished and projects start adopting alternative build
> systems, you may see an increasing proliferation of projects that cannot
> build eggs or be easy_install'ed.
>
> (Ref: https://www.python.org/dev/peps/pep-0517/)
>
> It's not clear when this will happen exactly -- I'm in the midst of
> dealing with some medical stuff, so haven't had much/any time for
> distutils-sig stuff recently, but sooner or later this is the direction we
> want to go.
>
> -n
>

So these will all still install with pip?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160820/d956e612/attachment-0001.html>

From chris.barker at noaa.gov  Sat Aug 20 18:40:09 2016
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Sat, 20 Aug 2016 18:40:09 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
Message-ID: <-1940544014921201238@unknownmsgid>

>
> If we're going through all this trouble, isn't it better just to jump to .zip files like every other distribution format in existence?

Yes. :-)

-CHB

From wes.turner at gmail.com  Sat Aug 20 19:59:30 2016
From: wes.turner at gmail.com (Wes Turner)
Date: Sat, 20 Aug 2016 18:59:30 -0500
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAP1=2W6TdvnTFu4hqonU3dO_xAxvJX=hoigSGEPNY0YV_MB5mg@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <DAA65B69-8246-4E3A-A482-5BF77E60FD44@stufft.io>
 <CAP1=2W6TdvnTFu4hqonU3dO_xAxvJX=hoigSGEPNY0YV_MB5mg@mail.gmail.com>
Message-ID: <CACfEFw9Pifh_mBhu46n3pBrRZJ+Lj3_g2D0VKAntvvq7kO0sMQ@mail.gmail.com>

On Saturday, August 20, 2016, Brett Cannon <brett at python.org> wrote:

>
>
> On Fri, 19 Aug 2016 at 12:53 Donald Stufft <donald at stufft.io
> <javascript:_e(%7B%7D,'cvml','donald at stufft.io');>> wrote:
>
>>
>>
>> Oh, and TIL that anyone who has Python 3.4+ installed has a command line
>> tool for extracting ``.tar.gz`` files [2]
>>
>
> So I think you're both right, but at different time scales. :) I think
> Donald is right that the short-term time scale of "now" by suggesting we
> just go with tar.gz since it has the numbers. But I think Leonardo's point
> of general alignment with zip for packaging overall is good for the
> "formally define sdist" time scale and we potentially introduce an .sdist
> file extension.
>

How about, as a convention,
.sdist.zip
.sdist.tar.gz

So that file type associations with archive programs still work?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160820/a1fb7bc5/attachment.html>

From dholth at gmail.com  Sat Aug 20 20:15:35 2016
From: dholth at gmail.com (Daniel Holth)
Date: Sun, 21 Aug 2016 00:15:35 +0000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CACfEFw9Pifh_mBhu46n3pBrRZJ+Lj3_g2D0VKAntvvq7kO0sMQ@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <DAA65B69-8246-4E3A-A482-5BF77E60FD44@stufft.io>
 <CAP1=2W6TdvnTFu4hqonU3dO_xAxvJX=hoigSGEPNY0YV_MB5mg@mail.gmail.com>
 <CACfEFw9Pifh_mBhu46n3pBrRZJ+Lj3_g2D0VKAntvvq7kO0sMQ@mail.gmail.com>
Message-ID: <CAG8k2+6ExRxYNsYt0vFC3KNy1LPLgg9hyt7qYrobzEFbgEDz0g@mail.gmail.com>

I don't think we are defining a new file type today.

On Sat, Aug 20, 2016 at 8:03 PM Wes Turner <wes.turner at gmail.com> wrote:

>
>
> On Saturday, August 20, 2016, Brett Cannon <brett at python.org> wrote:
>
>> On Fri, 19 Aug 2016 at 12:53 Donald Stufft <donald at stufft.io> wrote:
>>
>
>>>
>>> Oh, and TIL that anyone who has Python 3.4+ installed has a command line
>>> tool for extracting ``.tar.gz`` files [2]
>>>
>>
>> So I think you're both right, but at different time scales. :) I think
>> Donald is right that the short-term time scale of "now" by suggesting we
>> just go with tar.gz since it has the numbers. But I think Leonardo's point
>> of general alignment with zip for packaging overall is good for the
>> "formally define sdist" time scale and we potentially introduce an .sdist
>> file extension.
>>
>
> How about, as a convention,
> .sdist.zip
> .sdist.tar.gz
>
> So that file type associations with archive programs still work?
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160821/526e9dcd/attachment.html>

From dholth at gmail.com  Sat Aug 20 20:32:57 2016
From: dholth at gmail.com (Daniel Holth)
Date: Sun, 21 Aug 2016 00:32:57 +0000
Subject: [Distutils] cffi & Py_LIMITED_API
In-Reply-To: <CADiSq7cLo_D9BqunxJBU768-q8c+7FFVQaFowF0JrQWp_-cnhw@mail.gmail.com>
References: <CAG8k2+7RwPic+eUe3ir-LFnAua8714JiJ93JB8+D4MPKfpkz3A@mail.gmail.com>
 <CADiSq7cmQBVpESUi8w94-Ao=5ughpZymbKM-gmwWt6xKbWMpVQ@mail.gmail.com>
 <CAG8k2+5Hy8iFnLJvJeU593X=vOnGbBQuPi00qM3jCTFqaw9D3g@mail.gmail.com>
 <CADiSq7en+27PK_V=SG=+1v5cXKjf805TPyEo37O4uOGyzcYBZA@mail.gmail.com>
 <4db88636-e654-056b-0f3a-445de41ddb07@python.org>
 <CAG8k2+5x6aFOg=gPfdYCoOYKKF7Ev34nYr5f0buPQK+SyK=QGA@mail.gmail.com>
 <CADiSq7cLo_D9BqunxJBU768-q8c+7FFVQaFowF0JrQWp_-cnhw@mail.gmail.com>
Message-ID: <CAG8k2+4k-GJSRxPqU3JGn-BZ910mvg08mnuSjDhq-jJJMsc2-Q@mail.gmail.com>

Coded up in https://bitbucket.org/pypa/wheel/pull-requests/69 and
https://github.com/pypa/pip/pull/3922 and supported in setuptools 26.

It's a multi-step process, unfortunately, but if you do it correctly then
you should need a lot fewer wheels:

0. Use only the functions from the limited api, for example, by using cffi
to generate your C extensions. #define Py_LIMITED_API to the desired value.
Python 3.3 is a good minimum, adding some useful functions forgotten in
Python 3.2's limited api.
1. In setup.py, specify Extension(..., py_limited_api=True) to use .abi3.so
filenames on your C extensions
2. In setup.cfg, specify [bdist_wheel] py_limited_api=cp32 # or greater,
tagging the wheel for compatibility with that version of CPython, or later.
This argument is ignored outside of CPython 3.
3. A new enough version of pip recognizes the wheel tag

Your extension uses the limited API, setuptools' Extension() changes the
filename so that more than just the exact version of Python can find it,
wheel adds a cp33-abi3-manylinux1_x86 tag, pip finds it, and you no longer
have to compile new wheels just because a new version of CPython came out.

On Tue, Aug 2, 2016 at 4:02 AM Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 2 August 2016 at 02:24, Daniel Holth <dholth at gmail.com> wrote:
> > On Mon, Aug 1, 2016 at 12:01 PM Steve Dower <steve.dower at python.org>
> wrote:
> >> The DLL tag on Windows will have to just be ".pyd" if you want to
> >> support back prior to 3.5. In 3.5 you can use ".cp35-win32.pyd" or
> >> ".cp35-win_amd64.pyd" and the importer will prefer that DLL over a plain
> >> ".pyd", but my proposal to also support ".cp3-${PLAT}.pyd" here didn't
> >> make it.
> >>
> >> The wheel tag is more important than the DLL tag. (Possibly you're not
> >> even suggesting changing the DLL name at all and just passing the flag
> >> through for the build option? Hard to tell from the discussion.)
> >
> > I'm sure we are interpolating from Linux, where you'd be looking at
> > extensions .cpython-35m-x86_64-linux-gnu.so versus .abi3.so. But any DLL
> > extension that the target CPythons will agree to load, plus an
> appropriate
> > wheel tag, should be sufficient for our use case.
>
> Right, on *nix the SO names already include the SOABI details by
> default, so we really do want to change those to something less
> explicit, but I'm not sure it's even possible to using a completely
> unqualified SO name even if we wanted to.
>
> On Windows, if we have to rely on dropping any ABI indicated from the
> DLL entirely, we can leave with - pip should keep things from
> colliding too badly.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160821/31a20c18/attachment-0001.html>

From ncoghlan at gmail.com  Sat Aug 20 23:07:25 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 21 Aug 2016 13:07:25 +1000
Subject: [Distutils] Why I like eggs (or similar mechanisms) and my
 thoughts on future of buildout
In-Reply-To: <CAPDm-FhW6+t6FQMx5Wi33VSPukLLyk8D0AuWUsgGRCaTwEsRDQ@mail.gmail.com>
References: <CAPDm-FjTfzBSnFkd39hrkNqQEKcL4RW5qD-MFfMda+p=mym20A@mail.gmail.com>
 <CADiSq7fVGNdJFrX3xqN437QiwyE3p4Kv_sNAtv7KGtSb97k0SA@mail.gmail.com>
 <CAPDm-FhW6+t6FQMx5Wi33VSPukLLyk8D0AuWUsgGRCaTwEsRDQ@mail.gmail.com>
Message-ID: <CADiSq7dLmOSpi+8ympVN0yiph5snxkBC5bED69S20wrfopZsPQ@mail.gmail.com>

On 21 August 2016 at 05:46, Jim Fulton <jim at jimfulton.info> wrote:
> On Sat, Aug 20, 2016 at 3:02 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> > I have the impression that uninstalling things can be
>> > problematic, but maybe that's been fixed.
>>
>> Uninstallation is fine, as we *do* have a full file manifest after a
>> component has been installed.
>
> So, if package A and B install the same file, X, and then A is uninstalled,
> is X still there?  If then B is uninstalled, does X go away?  Does this
> machinery depend on whether X has the the same contents in A and B?

If A and B install to the same path, B will overwrite A's version, and
if either is uninstalled, they'll delete whichever version is
currently there. Since "the same path" in this case usually refers to
either a script or a Python module, the unhandled conflict is really
at install time - we'd want something closer to the RPM/deb style "A
file already exists at that destination, so we're not installing the
requested package at all", rather than a Windows-style reference
counting system.

As Daniel notes, this kind of check is actually already possible when
installing from a wheel file today, it just requires someone with the
roundtuits to add it to pip (perhaps via a new vendorable package for
working with Python installation metadata and detecting conflicts
between what's already installed and a wheel being considered for
installation).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Aug 20 23:22:57 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 21 Aug 2016 13:22:57 +1000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <-1940544014921201238@unknownmsgid>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
Message-ID: <CADiSq7f-wOshNskMA5mcfKVnAVYhEP6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>

On 21 August 2016 at 08:40, Chris Barker - NOAA Federal
<chris.barker at noaa.gov> wrote:
>>
>> If we're going through all this trouble, isn't it better just to jump to .zip files like every other distribution format in existence?
>
> Yes. :-)

zip is popular for *user facing* distribution formats, where the file
format is exposed directly to end users, and end user applications
(e.g. zipimport, ODF).

That's not where tarballs shine, so it's not where they show up: tar
is a backend infrastructure format, used for tool-to-tool
communication, rather than user-to-user.

The trajectory of distutils-sig for the past few years has been
towards a world where sdists are primarily a tool-to-tool format for
data interchange between publishing tools (e.g. twine), installation
tools (e.g. pip) and redistribution tools (e.g. conda, RPM, deb,
buildout), with wheels (and potentially eggs, or an equally zipimport
friendly future derivative) being the preferred formats for working
directly with Python software in a destination environment (whether
that's a system integrator's build farm, a production software
deployment, or a Python user's local system).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Aug 20 23:31:29 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 21 Aug 2016 13:31:29 +1000
Subject: [Distutils] cffi & Py_LIMITED_API
In-Reply-To: <CAG8k2+4k-GJSRxPqU3JGn-BZ910mvg08mnuSjDhq-jJJMsc2-Q@mail.gmail.com>
References: <CAG8k2+7RwPic+eUe3ir-LFnAua8714JiJ93JB8+D4MPKfpkz3A@mail.gmail.com>
 <CADiSq7cmQBVpESUi8w94-Ao=5ughpZymbKM-gmwWt6xKbWMpVQ@mail.gmail.com>
 <CAG8k2+5Hy8iFnLJvJeU593X=vOnGbBQuPi00qM3jCTFqaw9D3g@mail.gmail.com>
 <CADiSq7en+27PK_V=SG=+1v5cXKjf805TPyEo37O4uOGyzcYBZA@mail.gmail.com>
 <4db88636-e654-056b-0f3a-445de41ddb07@python.org>
 <CAG8k2+5x6aFOg=gPfdYCoOYKKF7Ev34nYr5f0buPQK+SyK=QGA@mail.gmail.com>
 <CADiSq7cLo_D9BqunxJBU768-q8c+7FFVQaFowF0JrQWp_-cnhw@mail.gmail.com>
 <CAG8k2+4k-GJSRxPqU3JGn-BZ910mvg08mnuSjDhq-jJJMsc2-Q@mail.gmail.com>
Message-ID: <CADiSq7e+jqfBoZ5ws+dR8-O1REmvO_tsip5xAwGWzY1KK55GbA@mail.gmail.com>

On 21 August 2016 at 10:32, Daniel Holth <dholth at gmail.com> wrote:
> Coded up in https://bitbucket.org/pypa/wheel/pull-requests/69 and
> https://github.com/pypa/pip/pull/3922 and supported in setuptools 26.
>
> It's a multi-step process, unfortunately, but if you do it correctly then
> you should need a lot fewer wheels:
>
> 0. Use only the functions from the limited api, for example, by using cffi
> to generate your C extensions. #define Py_LIMITED_API to the desired value.
> Python 3.3 is a good minimum, adding some useful functions forgotten in
> Python 3.2's limited api.
> 1. In setup.py, specify Extension(..., py_limited_api=True) to use .abi3.so
> filenames on your C extensions
> 2. In setup.cfg, specify [bdist_wheel] py_limited_api=cp32 # or greater,
> tagging the wheel for compatibility with that version of CPython, or later.
> This argument is ignored outside of CPython 3.
> 3. A new enough version of pip recognizes the wheel tag
>
> Your extension uses the limited API, setuptools' Extension() changes the
> filename so that more than just the exact version of Python can find it,
> wheel adds a cp33-abi3-manylinux1_x86 tag, pip finds it, and you no longer
> have to compile new wheels just because a new version of CPython came out.

Very nice! Would you have the time to draft some initial additions to
https://packaging.python.org/extensions/ to help us point folks
towards how to do this? (And why they'd want to - i.e. not needing to
rebuild their extension modules for each new release of CPython)

Cheers,
Nick.

P.S. No worries if not - if nobody else beats me to it, I should be
able to take a look at updating that page sometime after 3.6b1 next
month.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From njs at pobox.com  Sun Aug 21 02:40:20 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Sat, 20 Aug 2016 23:40:20 -0700
Subject: [Distutils] Proposed new Distutils API for compiler flag
 detection (Issue26689)
In-Reply-To: <CAK=Phk5YV_Ax34_DRV9b4ch4T7GNMZ2=E4DH=QP0DzeAA8v83A@mail.gmail.com>
References: <A45B9ADE-2FB4-4453-AACE-8B1595C009E3@python.org>
 <CABL7CQi86ORbhQUy11g4ipiEWMKFYtzeeYrb1JkKomsVPa8uKQ@mail.gmail.com>
 <CAK=Phk5YV_Ax34_DRV9b4ch4T7GNMZ2=E4DH=QP0DzeAA8v83A@mail.gmail.com>
Message-ID: <CAPJVwBnOu6i3f-_XspZtaMR8Va68suHVb9p9HiVQ6Cp8-CFFTQ@mail.gmail.com>

On Aug 20, 2016 3:09 PM, "Sylvain Corlay" <sylvain.corlay at gmail.com> wrote:
>
> Hi Ralf,
>
> On Sat, Aug 20, 2016 at 11:42 PM, Ralf Gommers <ralf.gommers at gmail.com>
wrote:
>>>
>>>
>>> On python-dev and in the bug tracker, Sylvain has understandably asked
for a review with an eye to adding this new feature to Python 3.6 whose
feature code cutoff is scheduled for a few weeks from now.  As release
manager, I am not opposed in general to adding new features to Distutils
but I think we should be very cautious about modifying or adding new
Distutils APIs, given that many third-party distribution authors want to
support their packages on multiple versions.  So I want to make sure that
there is some agreement that adding this new API starting with 3.6 is a
good thing to do rather than having it go in under the radar.
>>
>>
>> I'd rather see that kind of thing added to setuptools. We're already
having to deal with setuptools as a moving target, so if distutils becomes
one again as well that means more testing with combinations of different
Python and setuptools versions. Imho distutils changes should be bugfix and
essential maintenance only.
>>
>
>
> Having the `has_flag` in a different location from `has_function` would
be weird in my opinion.

I think the point though is that in your proposal, has_flag is in distutils
3.6, but has_function is in distutils 2.6, 2.7, 3.2, 3.3, 3.4, 3.5, 3.6. So
they're still in different places in practice, and if your new feature is
only in distutils 3.6 then almost no one will be able to use it within the
next 3+ years. Otoh putting it in setuptools makes it uniformly usable for
everyone immediately.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160820/19e10215/attachment-0001.html>

From robertc at robertcollins.net  Sun Aug 21 04:21:26 2016
From: robertc at robertcollins.net (Robert Collins)
Date: Sun, 21 Aug 2016 20:21:26 +1200
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CADiSq7f-wOshNskMA5mcfKVnAVYhEP6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CADiSq7f-wOshNskMA5mcfKVnAVYhEP6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>
Message-ID: <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>

On 21 August 2016 at 15:22, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 21 August 2016 at 08:40, Chris Barker - NOAA Federal
> <chris.barker at noaa.gov> wrote:
>>>
>>> If we're going through all this trouble, isn't it better just to jump to .zip files like every other distribution format in existence?
>>
>> Yes. :-)
>
> zip is popular for *user facing* distribution formats, where the file
> format is exposed directly to end users, and end user applications
> (e.g. zipimport, ODF).
>
> That's not where tarballs shine, so it's not where they show up: tar
> is a backend infrastructure format, used for tool-to-tool
> communication, rather than user-to-user.
>
> The trajectory of distutils-sig for the past few years has been
> towards a world where sdists are primarily a tool-to-tool format for
> data interchange between publishing tools (e.g. twine), installation
> tools (e.g. pip) and redistribution tools (e.g. conda, RPM, deb,
> buildout), with wheels (and potentially eggs, or an equally zipimport
> friendly future derivative) being the preferred formats for working
> directly with Python software in a destination environment (whether
> that's a system integrator's build farm, a production software
> deployment, or a Python user's local system).

Jar's are zip's (~= wheels). *source* jars are also zips. (~= sdists).
I'm struggling to think of any two ways that .tar.gz is better than
.zip. The compression is ~=. .tar.lzma and friends can be radically
smaller whereas .zip has per-element compression, so thats one way,
but .tar.gz has no other benefits (it has other differences, but none
are relevant in the context of the sdist use cases AFAICT).

As far as the switching consts Donald raises, here's my take:
 - we're going to break a lot of peoples workflow no matter what: 10%
of a lot is a lot.
 - all setuptools in the while /can/ produce .zip's AIUI, so the
issues w.r.t. the long tail of Linux distros is entirely
uninteresting: the folk depending on the default behaviour will need
to be explict - but thats going to be the case for *everyone* [because
setuptools on windows is inconsistent with setuptools on Linux]: we
are basically telling everyone they have to hardcode to .zip until the
long tail of existing installs *everywhere* has gone away.
 - unzip and friends are as available on Linux distros as tar is.

tl;dr: I think standardising on .tar.gz would be a rather shortsighted
thing to do, given how many Windows users Python has and how much of a
different supporting .zip makes for workflow on that platform - with
no negative impacts on any other platform.

-Rob

From sylvain.corlay at gmail.com  Sun Aug 21 05:18:28 2016
From: sylvain.corlay at gmail.com (Sylvain Corlay)
Date: Sun, 21 Aug 2016 11:18:28 +0200
Subject: [Distutils] Proposed new Distutils API for compiler flag
 detection (Issue26689)
In-Reply-To: <CAPJVwBnOu6i3f-_XspZtaMR8Va68suHVb9p9HiVQ6Cp8-CFFTQ@mail.gmail.com>
References: <A45B9ADE-2FB4-4453-AACE-8B1595C009E3@python.org>
 <CABL7CQi86ORbhQUy11g4ipiEWMKFYtzeeYrb1JkKomsVPa8uKQ@mail.gmail.com>
 <CAK=Phk5YV_Ax34_DRV9b4ch4T7GNMZ2=E4DH=QP0DzeAA8v83A@mail.gmail.com>
 <CAPJVwBnOu6i3f-_XspZtaMR8Va68suHVb9p9HiVQ6Cp8-CFFTQ@mail.gmail.com>
Message-ID: <CAK=Phk4_XgZu_z_qu_mHFKE96V2c-dJHdixJKtrckBjQ2L88Tw@mail.gmail.com>

On Sun, Aug 21, 2016 at 8:40 AM, Nathaniel Smith <njs at pobox.com> wrote:

> > Having the `has_flag` in a different location from `has_function` would
> be weird in my opinion.
>
> I think the point though is that in your proposal, has_flag is in
> distutils 3.6, but has_function is in distutils 2.6, 2.7, 3.2, 3.3, 3.4,
> 3.5, 3.6. So they're still in different places in practice, and if your new
> feature is only in distutils 3.6 then almost no one will be able to use it
> within the next 3+ years. Otoh putting it in setuptools makes it uniformly
> usable for everyone immediately.
>
> -n
>

With this reasoning, nothing should ever be added to the standard library.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160821/34cff228/attachment.html>

From p.f.moore at gmail.com  Sun Aug 21 06:51:38 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 21 Aug 2016 11:51:38 +0100
Subject: [Distutils] Why I like eggs (or similar mechanisms) and my
 thoughts on future of buildout
In-Reply-To: <CAPJVwBn1SwHcYkg7HA9b4TqyL=OR_mE=rb-quJmQJyaH0Q3dbg@mail.gmail.com>
References: <CAPDm-FjTfzBSnFkd39hrkNqQEKcL4RW5qD-MFfMda+p=mym20A@mail.gmail.com>
 <CAPJVwBn1SwHcYkg7HA9b4TqyL=OR_mE=rb-quJmQJyaH0Q3dbg@mail.gmail.com>
Message-ID: <CACac1F8S+eM7US1ffPvq3X-Q7TaAKUcFxLYMbNFX91b9wdLzwg@mail.gmail.com>

On 20 August 2016 at 22:31, Nathaniel Smith <njs at pobox.com> wrote:
> Wheels are a pretty simple and straightforward format. They've got some
> metadata, and then the are a set of directories with labels attached: "this
> directory needs to go on sys.path", "this directory has scripts that should
> have shebang fixups applied and then go on $PATH", etc. (The spec is not
> long: https://www.python.org/dev/peps/pep-0427/)
>
> My first guess at how buildout could handle wheels elegantly would be:
>
> - we have one piece of code that abstracts the operation of "please give me
> a wheel for package X version Y that works on this system".

This can be abstracted at the moment as "pip wheel <requirement>".
That will deliver a wheel to the current directory which is exactly
what is described above. It probably doesn't meet buildout's
repeatability requirements (as it could invoke a build step if there's
no published wheel, and build steps can do anything they like -
--only-binary would be a way to avoid that, at the cost of failing if
a project didn't publish a wheel) but it would likely work as a
prototype.

I don't think there's anything inherent in buildout's isolation model
of using sys.path entries for each dependency, that is incompatible
with pip/wheels. If buildout unpacks eggs, it can do the same with
wheels. If it uses zipped eggs, then it can *probably* do the same
with wheels, although there's no "zip safe" mechanism to declare a
wheel as safe for this use.

Paul

From p.f.moore at gmail.com  Sun Aug 21 06:57:34 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 21 Aug 2016 11:57:34 +0100
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CADiSq7f-wOshNskMA5mcfKVnAVYhEP6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
Message-ID: <CACac1F81hMj1Ra1+_ZQy5uyBfjHDX8Dfeh7tkLiCDr87=R0Cwg@mail.gmail.com>

On 21 August 2016 at 09:21, Robert Collins <robertc at robertcollins.net> wrote:
>
> tl;dr: I think standardising on .tar.gz would be a rather shortsighted
> thing to do, given how many Windows users Python has and how much of a
> different supporting .zip makes for workflow on that platform - with
> no negative impacts on any other platform.

One thing that has (IIRC) come up in a pip bug in the past - how do
tar and zip format fare in terms of Unicode support? IIRC, older
versions of (I think) tar format don't include an encoding, nor do
they mandate UTF-8, so they have the potential to break when used
cross-platform. Sorry, I can't recall exact details.

I think it's important that whatever format we mandate works with full
Unicode filenames, and the available user tools support that.

Paul

From njs at pobox.com  Sun Aug 21 07:07:24 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Sun, 21 Aug 2016 04:07:24 -0700
Subject: [Distutils] Why I like eggs (or similar mechanisms) and my
 thoughts on future of buildout
In-Reply-To: <CACac1F8S+eM7US1ffPvq3X-Q7TaAKUcFxLYMbNFX91b9wdLzwg@mail.gmail.com>
References: <CAPDm-FjTfzBSnFkd39hrkNqQEKcL4RW5qD-MFfMda+p=mym20A@mail.gmail.com>
 <CAPJVwBn1SwHcYkg7HA9b4TqyL=OR_mE=rb-quJmQJyaH0Q3dbg@mail.gmail.com>
 <CACac1F8S+eM7US1ffPvq3X-Q7TaAKUcFxLYMbNFX91b9wdLzwg@mail.gmail.com>
Message-ID: <CAPJVwB=23F68+=Q3L_p3S7+7tw_NcRRudMYDNfG8cZB3Hp-AiQ@mail.gmail.com>

On Sun, Aug 21, 2016 at 3:51 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 20 August 2016 at 22:31, Nathaniel Smith <njs at pobox.com> wrote:
>> Wheels are a pretty simple and straightforward format. They've got some
>> metadata, and then the are a set of directories with labels attached: "this
>> directory needs to go on sys.path", "this directory has scripts that should
>> have shebang fixups applied and then go on $PATH", etc. (The spec is not
>> long: https://www.python.org/dev/peps/pep-0427/)
>>
>> My first guess at how buildout could handle wheels elegantly would be:
>>
>> - we have one piece of code that abstracts the operation of "please give me
>> a wheel for package X version Y that works on this system".
>
> This can be abstracted at the moment as "pip wheel <requirement>".
> That will deliver a wheel to the current directory which is exactly
> what is described above.

Or possibly "pip wheel --no-deps <requirement>" or something similar.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From dholth at gmail.com  Sun Aug 21 09:39:55 2016
From: dholth at gmail.com (Daniel Holth)
Date: Sun, 21 Aug 2016 13:39:55 +0000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CACac1F81hMj1Ra1+_ZQy5uyBfjHDX8Dfeh7tkLiCDr87=R0Cwg@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CADiSq7f-wOshNskMA5mcfKVnAVYhEP6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CACac1F81hMj1Ra1+_ZQy5uyBfjHDX8Dfeh7tkLiCDr87=R0Cwg@mail.gmail.com>
Message-ID: <CAG8k2+4tbO0-++zJyZREwrPCGgrVdzf4FVpc24yJ91ZeVbmZcw@mail.gmail.com>

Targz is about 3/4 the size of zip for a bag of Python dists I tested. Zip
inside a second zip would provide the same compression. No word on the
Weissman score.

Tar isn't exactly a single format. For Unicode try POSIX-1.2001 aka pax
format tar. Python defaults to gnutar. Zip has good Unicode support.

Exactly how much simpler is exactly one file format?

On Sun, Aug 21, 2016, 06:57 Paul Moore <p.f.moore at gmail.com> wrote:

> On 21 August 2016 at 09:21, Robert Collins <robertc at robertcollins.net>
> wrote:
> >
> > tl;dr: I think standardising on .tar.gz would be a rather shortsighted
> > thing to do, given how many Windows users Python has and how much of a
> > different supporting .zip makes for workflow on that platform - with
> > no negative impacts on any other platform.
>
> One thing that has (IIRC) come up in a pip bug in the past - how do
> tar and zip format fare in terms of Unicode support? IIRC, older
> versions of (I think) tar format don't include an encoding, nor do
> they mandate UTF-8, so they have the potential to break when used
> cross-platform. Sorry, I can't recall exact details.
>
> I think it's important that whatever format we mandate works with full
> Unicode filenames, and the available user tools support that.
>
> Paul
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160821/e40d00aa/attachment.html>

From donald at stufft.io  Sun Aug 21 11:28:01 2016
From: donald at stufft.io (Donald Stufft)
Date: Sun, 21 Aug 2016 11:28:01 -0400
Subject: [Distutils] Proposed new Distutils API for compiler flag
 detection (Issue26689)
In-Reply-To: <CAK=Phk4_XgZu_z_qu_mHFKE96V2c-dJHdixJKtrckBjQ2L88Tw@mail.gmail.com>
References: <A45B9ADE-2FB4-4453-AACE-8B1595C009E3@python.org>
 <CABL7CQi86ORbhQUy11g4ipiEWMKFYtzeeYrb1JkKomsVPa8uKQ@mail.gmail.com>
 <CAK=Phk5YV_Ax34_DRV9b4ch4T7GNMZ2=E4DH=QP0DzeAA8v83A@mail.gmail.com>
 <CAPJVwBnOu6i3f-_XspZtaMR8Va68suHVb9p9HiVQ6Cp8-CFFTQ@mail.gmail.com>
 <CAK=Phk4_XgZu_z_qu_mHFKE96V2c-dJHdixJKtrckBjQ2L88Tw@mail.gmail.com>
Message-ID: <34793887-E8A4-4694-ADD2-04F45F789651@stufft.io>


> On Aug 21, 2016, at 5:18 AM, Sylvain Corlay <sylvain.corlay at gmail.com> wrote:
> 
> With this reasoning, nothing should ever be added to the standard library.


Packaging is a bit different than other things because the network effect is much more prominent. There?s no real way to say, install a backport if you need one, you just have to kind of wait until every has upgraded which is unlikely other bits of the standard library. In addition, people writing projects in Python that are designed to be distributed, they tend to need to work across many versions of Python, while someone writing a project for themselves only need to worry about whatever version of Python they are deploying to. So while the new statistics module is, even without a backport, immediately useful to people developing their own projects for a recent version of Python, something in distutils is not useful for package authors until it is the *minimum* version of Python they support.

This generally makes the reward for changing distutils very small, particularly with the 3.x split because very few authors are willing to drop 2.7 much less go straight to 3.6 (or whatever) and for people making their own, internal projects, distutils isn?t generally used a whole lot there either.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160821/e7d3f5ea/attachment.html>

From jim at jimfulton.info  Sun Aug 21 12:16:42 2016
From: jim at jimfulton.info (Jim Fulton)
Date: Sun, 21 Aug 2016 12:16:42 -0400
Subject: [Distutils] Why I like eggs (or similar mechanisms) and my
 thoughts on future of buildout
In-Reply-To: <CADiSq7dLmOSpi+8ympVN0yiph5snxkBC5bED69S20wrfopZsPQ@mail.gmail.com>
References: <CAPDm-FjTfzBSnFkd39hrkNqQEKcL4RW5qD-MFfMda+p=mym20A@mail.gmail.com>
 <CADiSq7fVGNdJFrX3xqN437QiwyE3p4Kv_sNAtv7KGtSb97k0SA@mail.gmail.com>
 <CAPDm-FhW6+t6FQMx5Wi33VSPukLLyk8D0AuWUsgGRCaTwEsRDQ@mail.gmail.com>
 <CADiSq7dLmOSpi+8ympVN0yiph5snxkBC5bED69S20wrfopZsPQ@mail.gmail.com>
Message-ID: <CAPDm-Fg=0dg-5kzxGH8YR89NgnikadLQFvxNjymbYa=egMhtzA@mail.gmail.com>

On Sat, Aug 20, 2016 at 11:07 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 21 August 2016 at 05:46, Jim Fulton <jim at jimfulton.info> wrote:
> > On Sat, Aug 20, 2016 at 3:02 PM, Nick Coghlan <ncoghlan at gmail.com>
> wrote:
> >> > I have the impression that uninstalling things can be
> >> > problematic, but maybe that's been fixed.
> >>
> >> Uninstallation is fine, as we *do* have a full file manifest after a
> >> component has been installed.
> >
> > So, if package A and B install the same file, X, and then A is
> uninstalled,
> > is X still there?  If then B is uninstalled, does X go away?  Does this
> > machinery depend on whether X has the the same contents in A and B?
>
> If A and B install to the same path, B will overwrite A's version, and
> if either is uninstalled, they'll delete whichever version is
> currently there. Since "the same path" in this case usually refers to
> either a script or a Python module, the unhandled conflict is really
> at install time - we'd want something closer to the RPM/deb style "A
> file already exists at that destination, so we're not installing the
> requested package at all", rather than a Windows-style reference
> counting system.
>
> As Daniel notes, this kind of check is actually already possible when
> installing from a wheel file today, it just requires someone with the
> roundtuits to add it to pip (perhaps via a new vendorable package for
> working with Python installation metadata and detecting conflicts
> between what's already installed and a wheel being considered for
> installation).
>

The most common source of conflicts, I expect, are namespace package
__init__.py files. You wouldn't want to reject installing a package with a
conflicting __init__.py file in a namespace package.  I realize that PEP420
and the demise of Python 2 should make this case go away, someday, although
PEP420 doesn't prohibit __init__.py files in namespace packages.

OK, I'm done beating this horse. It was dead a while ago. :)

Jim

-- 
Jim Fulton
http://jimfulton.info
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160821/dee67be3/attachment.html>

From ncoghlan at gmail.com  Sun Aug 21 12:18:19 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 22 Aug 2016 02:18:19 +1000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CADiSq7f-wOshNskMA5mcfKVnAVYhEP6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
Message-ID: <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>

On 21 August 2016 at 18:21, Robert Collins <robertc at robertcollins.net> wrote:
> tl;dr: I think standardising on .tar.gz would be a rather shortsighted
> thing to do, given how many Windows users Python has and how much of a
> different supporting .zip makes for workflow on that platform - with
> no negative impacts on any other platform.

Once we have a universal default, unilaterally changing that default
gets easier, rather than harder - the main precedent being set here is
that we *don't want* the default format to be platform dependent, we
want there to be just one default.

My core assumption in recommending starting with tar.gz as that
universal default is that Windows users will mostly be interacting
with sdists through tooling that is closely linked to or at least
originated in the Python ecosystem (easy_install, pip, PyPM, conda,
Enthought Canopy, buildout, Christoph Gohlke's Windows installers),
and that double-clicking a Python sdist in an Explorer window is not
something most folks are in the habit of doing.

By contrast, I *know* Linux distros are in the habit of pulling
release tarballs from PyPI and feeding them directly into their
release pipelines, so the potential for unanticipated breakage seems
much higher there, and more likely to fall on community distros rather
than on commercial Python redistributors (simply because there are
more volunteers working on Linux based tooling than there are on
WIndows developer tools).

Now, it may be that first assumption is wrong, and that folks actually
are relying on the sdist-as-zip behaviour on Windows in the same way
folks rely on the sdist-as-tarball behaviour on Linux. If that's the
case, then the Python 3.6 beta period will be the point where we
really want the Windows-supporting redistributors and folks doing
development and deployment on Windows to kick the tires of this change
and report any problems that arise - it may be that we end up needing
to find a different way to close the current format loophole that
allows two different sdists to be uploaded for the same version.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From jim at jimfulton.info  Sun Aug 21 12:19:12 2016
From: jim at jimfulton.info (Jim Fulton)
Date: Sun, 21 Aug 2016 12:19:12 -0400
Subject: [Distutils] Why I like eggs (or similar mechanisms) and my
 thoughts on future of buildout
In-Reply-To: <CACfEFw-zuh5egvM_0L_ThZDYqq-06XqbS0mpFVFVwTROcSoGrA@mail.gmail.com>
References: <CAPDm-FjTfzBSnFkd39hrkNqQEKcL4RW5qD-MFfMda+p=mym20A@mail.gmail.com>
 <CAPJVwBn1SwHcYkg7HA9b4TqyL=OR_mE=rb-quJmQJyaH0Q3dbg@mail.gmail.com>
 <CACfEFw-zuh5egvM_0L_ThZDYqq-06XqbS0mpFVFVwTROcSoGrA@mail.gmail.com>
Message-ID: <CAPDm-FgZ4RyCZOOOMeVav5W=spM=Vm7POs11z2Ch67D=4CwHkQ@mail.gmail.com>

On Sat, Aug 20, 2016 at 6:31 PM, Wes Turner <wes.turner at gmail.com> wrote:
...

> I may be optimistic, but should adding buildout support for wheel be more
> complicated than shelling out to pip with the correct, uhm, --prefix etc?
>

I'm open to shelling out, but pessimistic that it will turn out well. I
started with that approach initially with easy_install and it fell apart
quickly.  But when we get into it... who knows?

Jim

-- 
Jim Fulton
http://jimfulton.info
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160821/280efdce/attachment.html>

From donald at stufft.io  Sun Aug 21 12:29:05 2016
From: donald at stufft.io (Donald Stufft)
Date: Sun, 21 Aug 2016 12:29:05 -0400
Subject: [Distutils] Why I like eggs (or similar mechanisms) and my
 thoughts on future of buildout
In-Reply-To: <CAPDm-FgZ4RyCZOOOMeVav5W=spM=Vm7POs11z2Ch67D=4CwHkQ@mail.gmail.com>
References: <CAPDm-FjTfzBSnFkd39hrkNqQEKcL4RW5qD-MFfMda+p=mym20A@mail.gmail.com>
 <CAPJVwBn1SwHcYkg7HA9b4TqyL=OR_mE=rb-quJmQJyaH0Q3dbg@mail.gmail.com>
 <CACfEFw-zuh5egvM_0L_ThZDYqq-06XqbS0mpFVFVwTROcSoGrA@mail.gmail.com>
 <CAPDm-FgZ4RyCZOOOMeVav5W=spM=Vm7POs11z2Ch67D=4CwHkQ@mail.gmail.com>
Message-ID: <6A400F5A-3BC0-4855-8015-AB36D5B98279@stufft.io>


> On Aug 21, 2016, at 12:19 PM, Jim Fulton <jim at jimfulton.info> wrote:
> 
> I'm open to shelling out, but pessimistic that it will turn out well. I started with that approach initially with easy_install and it fell apart quickly.  But when we get into it... who knows?


Shelling out is currently the only exposed ?API? that pip has, we?re not opposed to adding extra APIs though. Our current approach has been to wait and see for people to come out with specific use cases they have for an API and then work together to figure out what API we can create that satisfies that. Thus far we?ve accomplished this by creating new libraries that aren?t pip and moving functionality out of pip (and setuptools) and into those libraries, and then making pip/setuptools consume those. This has generally worked pretty well I think, as it?s easier to be careful not to accidentally expose some terrible internal details of pip as public API when it?s a new, carefully designed thing, and we can make working with those libraries better than it is to simply expose some part of pip. We generally pair this along with defining things in PEPs so that these new libraries don?t become the new distutils/setuptools/pip (e.g., implementation defined standards) which should ideally allow anyone to create a from scratch implementation and have it interopt just fine.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160821/244b842e/attachment-0001.html>

From jim at jimfulton.info  Sun Aug 21 12:33:50 2016
From: jim at jimfulton.info (Jim Fulton)
Date: Sun, 21 Aug 2016 12:33:50 -0400
Subject: [Distutils] Why I like eggs (or similar mechanisms) and my
 thoughts on future of buildout
In-Reply-To: <6A400F5A-3BC0-4855-8015-AB36D5B98279@stufft.io>
References: <CAPDm-FjTfzBSnFkd39hrkNqQEKcL4RW5qD-MFfMda+p=mym20A@mail.gmail.com>
 <CAPJVwBn1SwHcYkg7HA9b4TqyL=OR_mE=rb-quJmQJyaH0Q3dbg@mail.gmail.com>
 <CACfEFw-zuh5egvM_0L_ThZDYqq-06XqbS0mpFVFVwTROcSoGrA@mail.gmail.com>
 <CAPDm-FgZ4RyCZOOOMeVav5W=spM=Vm7POs11z2Ch67D=4CwHkQ@mail.gmail.com>
 <6A400F5A-3BC0-4855-8015-AB36D5B98279@stufft.io>
Message-ID: <CAPDm-FiN6X8eEWsWMMAkgwgfyA4PwdG0ANX949UreiFSLwYPvA@mail.gmail.com>

On Sun, Aug 21, 2016 at 12:29 PM, Donald Stufft <donald at stufft.io> wrote:

>
> On Aug 21, 2016, at 12:19 PM, Jim Fulton <jim at jimfulton.info> wrote:
>
> I'm open to shelling out, but pessimistic that it will turn out well. I
> started with that approach initially with easy_install and it fell apart
> quickly.  But when we get into it... who knows?
>
>
>
> Shelling out is currently the only exposed ?API? that pip has, we?re not
> opposed to adding extra APIs though. Our current approach has been to wait
> and see for people to come out with specific use cases they have for an API
> and then work together to figure out what API we can create that satisfies
> that. Thus far we?ve accomplished this by creating new libraries that
> aren?t pip and moving functionality out of pip (and setuptools) and into
> those libraries, and then making pip/setuptools consume those. This has
> generally worked pretty well I think, as it?s easier to be careful not to
> accidentally expose some terrible internal details of pip as public API
> when it?s a new, carefully designed thing, and we can make working with
> those libraries better than it is to simply expose some part of pip. We
> generally pair this along with defining things in PEPs so that these new
> libraries don?t become the new distutils/setuptools/pip (e.g.,
> implementation defined standards) which should ideally allow anyone to
> create a from scratch implementation and have it interopt just fine.
>

Sounds reasonable. (I'd seen similar statements before, which I'd alluded
to in another message.) Thanks.

Jim

-- 
Jim Fulton
http://jimfulton.info
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160821/f8bb11ab/attachment.html>

From thomas at kluyver.me.uk  Sun Aug 21 12:38:34 2016
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Sun, 21 Aug 2016 17:38:34 +0100
Subject: [Distutils] Why I like eggs (or similar mechanisms) and my
 thoughts on future of buildout
In-Reply-To: <CAPDm-FgZ4RyCZOOOMeVav5W=spM=Vm7POs11z2Ch67D=4CwHkQ@mail.gmail.com>
References: <CAPDm-FjTfzBSnFkd39hrkNqQEKcL4RW5qD-MFfMda+p=mym20A@mail.gmail.com>
 <CAPJVwBn1SwHcYkg7HA9b4TqyL=OR_mE=rb-quJmQJyaH0Q3dbg@mail.gmail.com>
 <CACfEFw-zuh5egvM_0L_ThZDYqq-06XqbS0mpFVFVwTROcSoGrA@mail.gmail.com>
 <CAPDm-FgZ4RyCZOOOMeVav5W=spM=Vm7POs11z2Ch67D=4CwHkQ@mail.gmail.com>
Message-ID: <1471797514.3489821.701637033.18173DDB@webmail.messagingengine.com>

On Sun, Aug 21, 2016, at 05:19 PM, Jim Fulton wrote:
> I'm open to shelling out, but pessimistic that it will turn out well.
> I started with that approach initially with easy_install and it fell
> apart quickly.  But when we get into it... who knows?

I think shelling out is a reasonable approach for certain operations
if the tool you're shelling out to is designed for it. When we write
command-line programs, we often think about users typing in commands
at a terminal, and design things that are flexible and somewhat
smart. If shelling out is to be an API, the commands need to be
predictable and dumb.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160821/bec58d8f/attachment.html>

From ncoghlan at gmail.com  Sun Aug 21 12:44:25 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 22 Aug 2016 02:44:25 +1000
Subject: [Distutils] Why I like eggs (or similar mechanisms) and my
 thoughts on future of buildout
In-Reply-To: <1471797514.3489821.701637033.18173DDB@webmail.messagingengine.com>
References: <CAPDm-FjTfzBSnFkd39hrkNqQEKcL4RW5qD-MFfMda+p=mym20A@mail.gmail.com>
 <CAPJVwBn1SwHcYkg7HA9b4TqyL=OR_mE=rb-quJmQJyaH0Q3dbg@mail.gmail.com>
 <CACfEFw-zuh5egvM_0L_ThZDYqq-06XqbS0mpFVFVwTROcSoGrA@mail.gmail.com>
 <CAPDm-FgZ4RyCZOOOMeVav5W=spM=Vm7POs11z2Ch67D=4CwHkQ@mail.gmail.com>
 <1471797514.3489821.701637033.18173DDB@webmail.messagingengine.com>
Message-ID: <CADiSq7fxPzam-H9mRVZQAY6D13ZP6t2XmQ462o520WqVY3PpeA@mail.gmail.com>

On 22 August 2016 at 02:38, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> I think shelling out is a reasonable approach for certain operations if the
> tool you're shelling out to is designed for it. When we write command-line
> programs, we often think about users typing in commands at a terminal, and
> design things that are flexible and somewhat smart. If shelling out is to be
> an API, the commands need to be predictable and dumb.

And even then providing a decent error reporting API can be quite
difficult compared to the in-Python error reporting options with full
exception support.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From sylvain.corlay at gmail.com  Sun Aug 21 14:08:32 2016
From: sylvain.corlay at gmail.com (Sylvain Corlay)
Date: Sun, 21 Aug 2016 20:08:32 +0200
Subject: [Distutils] Proposed new Distutils API for compiler flag
 detection (Issue26689)
In-Reply-To: <34793887-E8A4-4694-ADD2-04F45F789651@stufft.io>
References: <A45B9ADE-2FB4-4453-AACE-8B1595C009E3@python.org>
 <CABL7CQi86ORbhQUy11g4ipiEWMKFYtzeeYrb1JkKomsVPa8uKQ@mail.gmail.com>
 <CAK=Phk5YV_Ax34_DRV9b4ch4T7GNMZ2=E4DH=QP0DzeAA8v83A@mail.gmail.com>
 <CAPJVwBnOu6i3f-_XspZtaMR8Va68suHVb9p9HiVQ6Cp8-CFFTQ@mail.gmail.com>
 <CAK=Phk4_XgZu_z_qu_mHFKE96V2c-dJHdixJKtrckBjQ2L88Tw@mail.gmail.com>
 <34793887-E8A4-4694-ADD2-04F45F789651@stufft.io>
Message-ID: <CAK=Phk4YRKgbH7Qxy2j2smuHpqGEpaNFgFtzEoHtH4qZaz92qw@mail.gmail.com>

Although we are reaching a tipping point where a lot of projects are
announcing the end of Python 2 support as of a certain date.

Whatever is in the latest version of Python 3 when it will be considered a
sane decision to have a Python 3-only library will be considered standard.

On Sun, Aug 21, 2016 at 5:28 PM, Donald Stufft <donald at stufft.io> wrote:

>
> On Aug 21, 2016, at 5:18 AM, Sylvain Corlay <sylvain.corlay at gmail.com>
> wrote:
>
> With this reasoning, nothing should ever be added to the standard library.
>
>
>
> Packaging is a bit different than other things because the network effect
> is much more prominent. There?s no real way to say, install a backport if
> you need one, you just have to kind of wait until every has upgraded which
> is unlikely other bits of the standard library. In addition, people writing
> projects in Python that are designed to be distributed, they tend to need
> to work across many versions of Python, while someone writing a project for
> themselves only need to worry about whatever version of Python they are
> deploying to. So while the new statistics module is, even without a
> backport, immediately useful to people developing their own projects for a
> recent version of Python, something in distutils is not useful for package
> authors until it is the *minimum* version of Python they support.
>
> This generally makes the reward for changing distutils very small,
> particularly with the 3.x split because very few authors are willing to
> drop 2.7 much less go straight to 3.6 (or whatever) and for people making
> their own, internal projects, distutils isn?t generally used a whole lot
> there either.
>
> ?
> Donald Stufft
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160821/c0aefc84/attachment-0001.html>

From donald at stufft.io  Sun Aug 21 14:10:47 2016
From: donald at stufft.io (Donald Stufft)
Date: Sun, 21 Aug 2016 14:10:47 -0400
Subject: [Distutils] Proposed new Distutils API for compiler flag
 detection (Issue26689)
In-Reply-To: <CAK=Phk4YRKgbH7Qxy2j2smuHpqGEpaNFgFtzEoHtH4qZaz92qw@mail.gmail.com>
References: <A45B9ADE-2FB4-4453-AACE-8B1595C009E3@python.org>
 <CABL7CQi86ORbhQUy11g4ipiEWMKFYtzeeYrb1JkKomsVPa8uKQ@mail.gmail.com>
 <CAK=Phk5YV_Ax34_DRV9b4ch4T7GNMZ2=E4DH=QP0DzeAA8v83A@mail.gmail.com>
 <CAPJVwBnOu6i3f-_XspZtaMR8Va68suHVb9p9HiVQ6Cp8-CFFTQ@mail.gmail.com>
 <CAK=Phk4_XgZu_z_qu_mHFKE96V2c-dJHdixJKtrckBjQ2L88Tw@mail.gmail.com>
 <34793887-E8A4-4694-ADD2-04F45F789651@stufft.io>
 <CAK=Phk4YRKgbH7Qxy2j2smuHpqGEpaNFgFtzEoHtH4qZaz92qw@mail.gmail.com>
Message-ID: <63D859EB-B20C-4413-ACF4-93EB22C7A7FD@stufft.io>

We?re reaching a point where *some* projects are announcing the end of Python 2 support as of a certain date, but let us not forget that Python 2.7 is still an order of magnitude more than any other version of Python in terms of downloads from PyPI.

> On Aug 21, 2016, at 2:08 PM, Sylvain Corlay <sylvain.corlay at gmail.com> wrote:
> 
> Although we are reaching a tipping point where a lot of projects are announcing the end of Python 2 support as of a certain date.
> 
> Whatever is in the latest version of Python 3 when it will be considered a sane decision to have a Python 3-only library will be considered standard.
> 
> On Sun, Aug 21, 2016 at 5:28 PM, Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
> 
>> On Aug 21, 2016, at 5:18 AM, Sylvain Corlay <sylvain.corlay at gmail.com <mailto:sylvain.corlay at gmail.com>> wrote:
>> 
>> With this reasoning, nothing should ever be added to the standard library.
> 
> 
> Packaging is a bit different than other things because the network effect is much more prominent. There?s no real way to say, install a backport if you need one, you just have to kind of wait until every has upgraded which is unlikely other bits of the standard library. In addition, people writing projects in Python that are designed to be distributed, they tend to need to work across many versions of Python, while someone writing a project for themselves only need to worry about whatever version of Python they are deploying to. So while the new statistics module is, even without a backport, immediately useful to people developing their own projects for a recent version of Python, something in distutils is not useful for package authors until it is the *minimum* version of Python they support.
> 
> This generally makes the reward for changing distutils very small, particularly with the 3.x split because very few authors are willing to drop 2.7 much less go straight to 3.6 (or whatever) and for people making their own, internal projects, distutils isn?t generally used a whole lot there either.
> 
> ?
> Donald Stufft
> 
> 
> 
> 


?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160821/57ba15e6/attachment.html>

From robertc at robertcollins.net  Sun Aug 21 15:54:55 2016
From: robertc at robertcollins.net (Robert Collins)
Date: Mon, 22 Aug 2016 07:54:55 +1200
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAJ3HoZ0Azm7nbDUS=3XQVPxu95WaufWyE2D41yztMWRXA-GNyQ@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CADiSq7f-wOshNskMA5mcfKVnAVYhEP6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAJ3HoZ2G3+fcK2h4z7R2zUU_k-YhM4OLYefmZ71J7M4jmxMpww@mail.gmail.com>
 <CAJ3HoZ0Azm7nbDUS=3XQVPxu95WaufWyE2D41yztMWRXA-GNyQ@mail.gmail.com>
Message-ID: <CAJ3HoZ3Q28RbdNsOPO+a4HdmQpWuH=OaSEgEz=fCVzCyTDu-3Q@mail.gmail.com>

On 22 Aug 2016 4:18 AM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
>
> On 21 August 2016 at 18:21, Robert Collins <robertc at robertcollins.net>
wrote:
> > tl;dr: I think standardising on .tar.gz would be a rather shortsighted
> > thing to do, given how many Windows users Python has and how much of a
> > different supporting .zip makes for workflow on that platform - with
> > no negative impacts on any other platform.
>
> Once we have a universal default, unilaterally changing that default
> gets easier, rather than harder - the main precedent being set here is
> that we *don't want* the default format to be platform dependent, we
> want there to be just one default.
>
> My core assumption in recommending starting with tar.gz as that
> universal default is that Windows users will mostly be interacting
> with sdists through tooling that is closely linked to or at least
> originated in the Python ecosystem (easy_install, pip, PyPM, conda,
> Enthought Canopy, buildout, Christoph Gohlke's Windows installers),
> and that double-clicking a Python sdist in an Explorer window is not
> something most folks are in the habit of doing.

It is certainly wrong for me, when I'm using Windows, and when I've
shoulder surfed others also. E.g. at pycon sprints.

> By contrast, I *know* Linux distros are in the habit of pulling
> release tarballs from PyPI and feeding them directly into their
> release pipelines, so the potential for unanticipated breakage seems
> much higher there, and more likely to fall on community distros rather
> than on commercial Python redistributors (simply because there are
> more volunteers working on Linux based tooling than there are on
> WIndows developer tools).

Those pipelines are mature, poly format capable and centralized, containing
the damage.

-Rob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160822/8692d65a/attachment.html>

From robertc at robertcollins.net  Sun Aug 21 16:02:12 2016
From: robertc at robertcollins.net (Robert Collins)
Date: Mon, 22 Aug 2016 08:02:12 +1200
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CADiSq7f-wOshNskMA5mcfKVnAVYhEP6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
Message-ID: <CAJ3HoZ3dHhHUGytFNPFasoky6pjBRKkUMACNYfqe1hDBA0C88w@mail.gmail.com>

On 22 Aug 2016 4:18 AM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
>
>
> Once we have a universal default, unilaterally changing that default
> gets easier, rather than harder - the main precedent being set here is
> that we *don't want* the default format to be platform dependent, we
> want there to be just one default.

Sorry for the separate reply, phone typing.

Monocultures are less flexible, so I'm assuming once we have Just One
format that bit rot in tooling will creep in and it will be harder and
harder to change, not easier.

-Rob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160822/f33d5fdb/attachment-0001.html>

From ralf.gommers at gmail.com  Sun Aug 21 16:31:13 2016
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Mon, 22 Aug 2016 08:31:13 +1200
Subject: [Distutils] Proposed new Distutils API for compiler flag
 detection (Issue26689)
In-Reply-To: <63D859EB-B20C-4413-ACF4-93EB22C7A7FD@stufft.io>
References: <A45B9ADE-2FB4-4453-AACE-8B1595C009E3@python.org>
 <CABL7CQi86ORbhQUy11g4ipiEWMKFYtzeeYrb1JkKomsVPa8uKQ@mail.gmail.com>
 <CAK=Phk5YV_Ax34_DRV9b4ch4T7GNMZ2=E4DH=QP0DzeAA8v83A@mail.gmail.com>
 <CAPJVwBnOu6i3f-_XspZtaMR8Va68suHVb9p9HiVQ6Cp8-CFFTQ@mail.gmail.com>
 <CAK=Phk4_XgZu_z_qu_mHFKE96V2c-dJHdixJKtrckBjQ2L88Tw@mail.gmail.com>
 <34793887-E8A4-4694-ADD2-04F45F789651@stufft.io>
 <CAK=Phk4YRKgbH7Qxy2j2smuHpqGEpaNFgFtzEoHtH4qZaz92qw@mail.gmail.com>
 <63D859EB-B20C-4413-ACF4-93EB22C7A7FD@stufft.io>
Message-ID: <CABL7CQgggpKpO5UG1vXfDFo1eR3ccyNRn7AcVt7OB=YKey+CCA@mail.gmail.com>

On Mon, Aug 22, 2016 at 6:10 AM, Donald Stufft <donald at stufft.io> wrote:

> We?re reaching a point where *some* projects are announcing the end of
> Python 2 support as of a certain date, but let us not forget that Python
> 2.7 is still an order of magnitude more than any other version of Python in
> terms of downloads from PyPI.
>

Even in 5 years from now, when almost all projects have dropped support for
Python 2.7, the reasoning remains the same. Projects will then support 3 or
4 Python 3.x versions, so still any new API added to distutils cannot be
used by those projects for 3-4 years. It does not make much sense to add
new things to distutils, with the exception maybe of something needed
particularly for a new Python version (like MSVC 2015 support in Python
3.5).

On top of that there are technical reasons (don't want to test combinations
of python+setuptools that both change per release) and organizational ones
(distutils maintenance is terrible, many simple bugfix patches don't get
merged for ages, setuptools at least fixes regressions quite fast).

I'm not sure if there's an official policy on adding new things to
distutils, but if not then this request is a good time to make one.
Assuming of course that the setuptools devs are willing to merge features
like the one from Sylvain.

Ralf



>
> On Aug 21, 2016, at 2:08 PM, Sylvain Corlay <sylvain.corlay at gmail.com>
> wrote:
>
> Although we are reaching a tipping point where a lot of projects are
> announcing the end of Python 2 support as of a certain date.
>
> Whatever is in the latest version of Python 3 when it will be considered a
> sane decision to have a Python 3-only library will be considered standard.
>
> On Sun, Aug 21, 2016 at 5:28 PM, Donald Stufft <donald at stufft.io> wrote:
>
>>
>> On Aug 21, 2016, at 5:18 AM, Sylvain Corlay <sylvain.corlay at gmail.com>
>> wrote:
>>
>> With this reasoning, nothing should ever be added to the standard library.
>>
>>
>>
>> Packaging is a bit different than other things because the network effect
>> is much more prominent. There?s no real way to say, install a backport if
>> you need one, you just have to kind of wait until every has upgraded which
>> is unlikely other bits of the standard library. In addition, people writing
>> projects in Python that are designed to be distributed, they tend to need
>> to work across many versions of Python, while someone writing a project for
>> themselves only need to worry about whatever version of Python they are
>> deploying to. So while the new statistics module is, even without a
>> backport, immediately useful to people developing their own projects for a
>> recent version of Python, something in distutils is not useful for package
>> authors until it is the *minimum* version of Python they support.
>>
>> This generally makes the reward for changing distutils very small,
>> particularly with the 3.x split because very few authors are willing to
>> drop 2.7 much less go straight to 3.6 (or whatever) and for people making
>> their own, internal projects, distutils isn?t generally used a whole lot
>> there either.
>>
>> ?
>> Donald Stufft
>>
>>
>>
>>
>
>
> ?
> Donald Stufft
>
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160822/dd437bdd/attachment.html>

From njs at pobox.com  Sun Aug 21 19:03:56 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Sun, 21 Aug 2016 16:03:56 -0700
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CADiSq7f-wOshNskMA5mcfKVnAVYhEP6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
Message-ID: <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>

On Aug 21, 2016 9:18 AM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
>
[...]
> By contrast, I *know* Linux distros are in the habit of pulling
> release tarballs from PyPI and feeding them directly into their
> release pipelines, so the potential for unanticipated breakage seems
> much higher there, and more likely to fall on community distros rather
> than on commercial Python redistributors (simply because there are
> more volunteers working on Linux based tooling than there are on
> WIndows developer tools).

But those release pipelines today have no problem handling the ~10% of
sdists that are zips. For my personal projects actually I've always
released .zip sdists exclusively (just because it seemed like a more
universally convenient format than .tar.gz), and those packages have shown
up in distros and no one has ever complained. Are any distros really
hardcoding an assumption that .tar.gz is that only possible sdist format?

(I'm agnostic on the overall question of whether to prefer .zip or .tar.gz
as the standard -- both seem reasonable with their own tradeoffs.)

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160821/46652d04/attachment.html>

From sylvain.corlay at gmail.com  Mon Aug 22 02:15:42 2016
From: sylvain.corlay at gmail.com (Sylvain Corlay)
Date: Mon, 22 Aug 2016 08:15:42 +0200
Subject: [Distutils] Proposed new Distutils API for compiler flag
 detection (Issue26689)
In-Reply-To: <CABL7CQgggpKpO5UG1vXfDFo1eR3ccyNRn7AcVt7OB=YKey+CCA@mail.gmail.com>
References: <A45B9ADE-2FB4-4453-AACE-8B1595C009E3@python.org>
 <CABL7CQi86ORbhQUy11g4ipiEWMKFYtzeeYrb1JkKomsVPa8uKQ@mail.gmail.com>
 <CAK=Phk5YV_Ax34_DRV9b4ch4T7GNMZ2=E4DH=QP0DzeAA8v83A@mail.gmail.com>
 <CAPJVwBnOu6i3f-_XspZtaMR8Va68suHVb9p9HiVQ6Cp8-CFFTQ@mail.gmail.com>
 <CAK=Phk4_XgZu_z_qu_mHFKE96V2c-dJHdixJKtrckBjQ2L88Tw@mail.gmail.com>
 <34793887-E8A4-4694-ADD2-04F45F789651@stufft.io>
 <CAK=Phk4YRKgbH7Qxy2j2smuHpqGEpaNFgFtzEoHtH4qZaz92qw@mail.gmail.com>
 <63D859EB-B20C-4413-ACF4-93EB22C7A7FD@stufft.io>
 <CABL7CQgggpKpO5UG1vXfDFo1eR3ccyNRn7AcVt7OB=YKey+CCA@mail.gmail.com>
Message-ID: <CAK=Phk6Qe=UWvKH-d2UbxJG9QSnDjHURwfz0rNPZ1GsxoYTr4A@mail.gmail.com>

Hi,

On Sun, Aug 21, 2016 at 10:31 PM, Ralf Gommers <ralf.gommers at gmail.com>
 wrote:

>
> On top of that there are technical reasons (don't want to test
> combinations of python + setuptools that both change per release) and
> organizational ones (distutils maintenance is terrible, many simple bugfix
> patches don't get merged for ages, setuptools at least fixes regressions
> quite fast).
>
> I'm not sure if there's an official policy on adding new things to
> distutils, but if not then this request is a good time to make one.
> Assuming of course that the setuptools devs are willing to merge features
> like the one from Sylvain.
>
>
I find this worrying that the main arguments to not include a patch would
be that

 - this part of the standard library is not very maintained (things don't
get merged)
 - earlier versions of won't have it

The former is a bad sign for a standard library and the latter is inherent
to any new feature. Whether it is a policy or not to not add new features
to distutils it is clear that a code base that does not evolve is dead.

How about, instead, we continue improving it?

Sylvain
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160822/617f3893/attachment-0001.html>

From thomas at kluyver.me.uk  Mon Aug 22 02:50:56 2016
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Mon, 22 Aug 2016 07:50:56 +0100
Subject: [Distutils] Proposed new Distutils API for compiler flag
 detection (Issue26689)
In-Reply-To: <CAK=Phk6Qe=UWvKH-d2UbxJG9QSnDjHURwfz0rNPZ1GsxoYTr4A@mail.gmail.com>
References: <A45B9ADE-2FB4-4453-AACE-8B1595C009E3@python.org>
 <CABL7CQi86ORbhQUy11g4ipiEWMKFYtzeeYrb1JkKomsVPa8uKQ@mail.gmail.com>
 <CAK=Phk5YV_Ax34_DRV9b4ch4T7GNMZ2=E4DH=QP0DzeAA8v83A@mail.gmail.com>
 <CAPJVwBnOu6i3f-_XspZtaMR8Va68suHVb9p9HiVQ6Cp8-CFFTQ@mail.gmail.com>
 <CAK=Phk4_XgZu_z_qu_mHFKE96V2c-dJHdixJKtrckBjQ2L88Tw@mail.gmail.com>
 <34793887-E8A4-4694-ADD2-04F45F789651@stufft.io>
 <CAK=Phk4YRKgbH7Qxy2j2smuHpqGEpaNFgFtzEoHtH4qZaz92qw@mail.gmail.com>
 <63D859EB-B20C-4413-ACF4-93EB22C7A7FD@stufft.io>
 <CABL7CQgggpKpO5UG1vXfDFo1eR3ccyNRn7AcVt7OB=YKey+CCA@mail.gmail.com>
 <CAK=Phk6Qe=UWvKH-d2UbxJG9QSnDjHURwfz0rNPZ1GsxoYTr4A@mail.gmail.com>
Message-ID: <1471848656.3689082.702076721.53722769@webmail.messagingengine.com>

On Mon, Aug 22, 2016, at 07:15 AM, Sylvain Corlay wrote:
> I find this worrying that the main arguments to not include a patch
> would be that
>
>  - this part of the standard library is not very maintained (things
>    don't get merged)
>  - earlier versions of won't have it

Would it make sense to add it to both distutils and setuptools? The
standard library continues to evolve, projects that require Python 3.6
wouldn't need to use setuptools, but we could start using it sooner.
There's obviously some cost in code duplication; I haven't looked at the
code in question, so I don't know how bad this is.

I've run into this argument before when trying to change things in non-packaging-
related parts of the stdlib, and I agree with Sylvain that it's
fundamentally problematic. If we're trying to improve the stdlib, we're
obviously taking a long view, but that's how we ensure the stdlib is
still useful in a few years time. This goes for packaging tools as much
as anything else.

I already have projects where I'm happy to require Python >=3.4, so
being able to depend on Python 3.6 is not such a distant prospect.

Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160822/fb80777e/attachment.html>

From tritium-list at sdamon.com  Mon Aug 22 03:23:16 2016
From: tritium-list at sdamon.com (tritium-list at sdamon.com)
Date: Mon, 22 Aug 2016 03:23:16 -0400
Subject: [Distutils] Proposed new Distutils API for compiler flag
 detection (Issue26689)
In-Reply-To: <CAK=Phk6Qe=UWvKH-d2UbxJG9QSnDjHURwfz0rNPZ1GsxoYTr4A@mail.gmail.com>
References: <A45B9ADE-2FB4-4453-AACE-8B1595C009E3@python.org>
 <CABL7CQi86ORbhQUy11g4ipiEWMKFYtzeeYrb1JkKomsVPa8uKQ@mail.gmail.com>
 <CAK=Phk5YV_Ax34_DRV9b4ch4T7GNMZ2=E4DH=QP0DzeAA8v83A@mail.gmail.com>
 <CAPJVwBnOu6i3f-_XspZtaMR8Va68suHVb9p9HiVQ6Cp8-CFFTQ@mail.gmail.com>
 <CAK=Phk4_XgZu_z_qu_mHFKE96V2c-dJHdixJKtrckBjQ2L88Tw@mail.gmail.com>
 <34793887-E8A4-4694-ADD2-04F45F789651@stufft.io>
 <CAK=Phk4YRKgbH7Qxy2j2smuHpqGEpaNFgFtzEoHtH4qZaz92qw@mail.gmail.com>
 <63D859EB-B20C-4413-ACF4-93EB22C7A7FD@stufft.io>
 <CABL7CQgggpKpO5UG1vXfDFo1eR3ccyNRn7AcVt7OB=YKey+CCA@mail.gmail.com>
 <CAK=Phk6Qe=UWvKH-d2UbxJG9QSnDjHURwfz0rNPZ1GsxoYTr4A@mail.gmail.com>
Message-ID: <0f0301d1fc46$0d5ec540$281c4fc0$@hotmail.com>

Changing packaging by updating the standard library will fail.  It?s been attempted.

 

The inherent problem is you need to fix packaging for people already using python, which means if you add a feature to the standard library, only the people who always run the latest and greatest can ever use the feature.  In a world where we are talking about Python 3.6/3.7 and python 2.7 is by far the most used version of python (and python usage is split pretty evenly between 3.4 and 3.5 IIRC), no one will use new packaging features in the standard library.

 

Putting something in setuptools means it will actually be used.  I think some of the goals of this sig is to be able to completely sunset distutils and replace it with much better solutions (plural) that all speak the same protocol.

 

From: Distutils-SIG [mailto:distutils-sig-bounces+tritium-list=sdamon.com at python.org] On Behalf Of Sylvain Corlay
Sent: Monday, August 22, 2016 2:16 AM
To: Ralf Gommers <ralf.gommers at gmail.com>
Cc: distutils-sig <distutils-sig at python.org>
Subject: Re: [Distutils] Proposed new Distutils API for compiler flag detection (Issue26689)

 

Hi,

On Sun, Aug 21, 2016 at 10:31 PM, Ralf Gommers <ralf.gommers at gmail.com <mailto:ralf.gommers at gmail.com> > wrote:


On top of that there are technical reasons (don't want to test combinations of python + setuptools that both change per release) and organizational ones (distutils maintenance is terrible, many simple bugfix patches don't get merged for ages, setuptools at least fixes regressions quite fast).

 

I'm not sure if there's an official policy on adding new things to distutils, but if not then this request is a good time to make one. Assuming of course that the setuptools devs are willing to merge features like the one from Sylvain.

 

 

I find this worrying that the main arguments to not include a patch would be that

 

 - this part of the standard library is not very maintained (things don't get merged)

 - earlier versions of won't have it

 

The former is a bad sign for a standard library and the latter is inherent to any new feature. Whether it is a policy or not to not add new features to distutils it is clear that a code base that does not evolve is dead.

 

How about, instead, we continue improving it?

 

Sylvain

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160822/83542d7b/attachment.html>

From sylvain.corlay at gmail.com  Mon Aug 22 04:25:04 2016
From: sylvain.corlay at gmail.com (Sylvain Corlay)
Date: Mon, 22 Aug 2016 10:25:04 +0200
Subject: [Distutils] Proposed new Distutils API for compiler flag
 detection (Issue26689)
In-Reply-To: <0f0301d1fc46$0d5ec540$281c4fc0$@hotmail.com>
References: <A45B9ADE-2FB4-4453-AACE-8B1595C009E3@python.org>
 <CABL7CQi86ORbhQUy11g4ipiEWMKFYtzeeYrb1JkKomsVPa8uKQ@mail.gmail.com>
 <CAK=Phk5YV_Ax34_DRV9b4ch4T7GNMZ2=E4DH=QP0DzeAA8v83A@mail.gmail.com>
 <CAPJVwBnOu6i3f-_XspZtaMR8Va68suHVb9p9HiVQ6Cp8-CFFTQ@mail.gmail.com>
 <CAK=Phk4_XgZu_z_qu_mHFKE96V2c-dJHdixJKtrckBjQ2L88Tw@mail.gmail.com>
 <34793887-E8A4-4694-ADD2-04F45F789651@stufft.io>
 <CAK=Phk4YRKgbH7Qxy2j2smuHpqGEpaNFgFtzEoHtH4qZaz92qw@mail.gmail.com>
 <63D859EB-B20C-4413-ACF4-93EB22C7A7FD@stufft.io>
 <CABL7CQgggpKpO5UG1vXfDFo1eR3ccyNRn7AcVt7OB=YKey+CCA@mail.gmail.com>
 <CAK=Phk6Qe=UWvKH-d2UbxJG9QSnDjHURwfz0rNPZ1GsxoYTr4A@mail.gmail.com>
 <0f0301d1fc46$0d5ec540$281c4fc0$@hotmail.com>
Message-ID: <CAK=Phk4Dfd1BNaC3OoTO=g8f1xqij2an8N9UuOuC0DLcenzgEA@mail.gmail.com>

I think that Thomas' proposal makes sense.

I would be ok to also add it to setuptools so that it can be used sooner by
projects that don't require python 3.6.

Sylvain

On Mon, Aug 22, 2016 at 9:23 AM, <tritium-list at sdamon.com> wrote:

> Changing packaging by updating the standard library will fail.  It?s been
> attempted.
>
>
>
> The inherent problem is you need to fix packaging for people already using
> python, which means if you add a feature to the standard library, only the
> people who always run the latest and greatest can ever use the feature.  In
> a world where we are talking about Python 3.6/3.7 and python 2.7 is by far
> the most used version of python (and python usage is split pretty evenly
> between 3.4 and 3.5 IIRC), no one will use new packaging features in the
> standard library.
>
>
>
> Putting something in setuptools means it will actually be used.  I think
> some of the goals of this sig is to be able to completely sunset distutils
> and replace it with much better solutions (plural) that all speak the same
> protocol.
>
>
>
> *From:* Distutils-SIG [mailto:distutils-sig-bounces+tritium-list=
> sdamon.com at python.org] *On Behalf Of *Sylvain Corlay
> *Sent:* Monday, August 22, 2016 2:16 AM
> *To:* Ralf Gommers <ralf.gommers at gmail.com>
> *Cc:* distutils-sig <distutils-sig at python.org>
> *Subject:* Re: [Distutils] Proposed new Distutils API for compiler flag
> detection (Issue26689)
>
>
>
> Hi,
>
> On Sun, Aug 21, 2016 at 10:31 PM, Ralf Gommers <ralf.gommers at gmail.com
> > wrote:
>
>
> On top of that there are technical reasons (don't want to test
> combinations of python + setuptools that both change per release) and
> organizational ones (distutils maintenance is terrible, many simple bugfix
> patches don't get merged for ages, setuptools at least fixes regressions
> quite fast).
>
>
>
> I'm not sure if there's an official policy on adding new things to
> distutils, but if not then this request is a good time to make one.
> Assuming of course that the setuptools devs are willing to merge features
> like the one from Sylvain.
>
>
>
>
>
> I find this worrying that the main arguments to not include a patch would
> be that
>
>
>
>  - this part of the standard library is not very maintained (things don't
> get merged)
>
>  - earlier versions of won't have it
>
>
>
> The former is a bad sign for a standard library and the latter is inherent
> to any new feature. Whether it is a policy or not to not add new features
> to distutils it is clear that a code base that does not evolve is dead.
>
>
>
> How about, instead, we continue improving it?
>
>
>
> Sylvain
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160822/2009c55f/attachment-0001.html>

From cosimo.lupo at daltonmaag.com  Mon Aug 22 05:25:07 2016
From: cosimo.lupo at daltonmaag.com (Cosimo Lupo)
Date: Mon, 22 Aug 2016 11:25:07 +0200
Subject: [Distutils] unittest2 issue tracker?
Message-ID: <CAH+Pg9q9Er2M2DC7Gj_=Ls3Ug4f=jEVVHqG+Gxzg44CPHzTazQ@mail.gmail.com>

Hello,

Sorry if this is not the right place to ask this sorts of questions.

The PyPI page for unittest2 package links to
https://code.google.com/archive/p/unittest-ext/issues as its "issue
tracker", however the Google Code pages seems to be read-only nowadays, so
I can't open new issues there.

I've found an auto-exported Github clone for unittest-ext and opened a new
issue there: https://github.com/testing-cabal/unittest-ext/issues/99

However I'm not sure it's the right one, as there doesn't seem to be much
activity in there.

It would be good to know where I should report bugs for unittest2.
Thank you!

-- 

*Cosimo Lupo*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160822/f715120c/attachment.html>

From p.f.moore at gmail.com  Mon Aug 22 06:08:33 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 22 Aug 2016 11:08:33 +0100
Subject: [Distutils] unittest2 issue tracker?
In-Reply-To: <CAH+Pg9q9Er2M2DC7Gj_=Ls3Ug4f=jEVVHqG+Gxzg44CPHzTazQ@mail.gmail.com>
References: <CAH+Pg9q9Er2M2DC7Gj_=Ls3Ug4f=jEVVHqG+Gxzg44CPHzTazQ@mail.gmail.com>
Message-ID: <CACac1F8O5OD9YkW3ihuP+fAVDozhorAxStp8e9ttW=XiZq15Eg@mail.gmail.com>

On 22 August 2016 at 10:25, Cosimo Lupo <cosimo.lupo at daltonmaag.com> wrote:
> Sorry if this is not the right place to ask this sorts of questions.
>
> The PyPI page for unittest2 package links to
> https://code.google.com/archive/p/unittest-ext/issues as its "issue
> tracker", however the Google Code pages seems to be read-only nowadays, so I
> can't open new issues there.
>
> I've found an auto-exported Github clone for unittest-ext and opened a new
> issue there: https://github.com/testing-cabal/unittest-ext/issues/99
>
> However I'm not sure it's the right one, as there doesn't seem to be much
> activity in there.
>
> It would be good to know where I should report bugs for unittest2.

As unittest2 is a backport of the newer features of (stdlib) unittest,
I'd suggest reporting issues on the Python tracker at
http://bugs.python.org/

Paul

From cosimo.lupo at daltonmaag.com  Mon Aug 22 06:11:33 2016
From: cosimo.lupo at daltonmaag.com (Cosimo Lupo)
Date: Mon, 22 Aug 2016 12:11:33 +0200
Subject: [Distutils] unittest2 issue tracker?
In-Reply-To: <CACac1F8O5OD9YkW3ihuP+fAVDozhorAxStp8e9ttW=XiZq15Eg@mail.gmail.com>
References: <CAH+Pg9q9Er2M2DC7Gj_=Ls3Ug4f=jEVVHqG+Gxzg44CPHzTazQ@mail.gmail.com>
 <CACac1F8O5OD9YkW3ihuP+fAVDozhorAxStp8e9ttW=XiZq15Eg@mail.gmail.com>
Message-ID: <CAH+Pg9qP-ZCJS70Th8O4COu4EB00YswTx2i3tpGtsH+4bWv2tg@mail.gmail.com>

Thanks for your reply, I'll try that then.
Cheers,

Cosimo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160822/6e12d319/attachment.html>

From ncoghlan at gmail.com  Mon Aug 22 07:14:40 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 22 Aug 2016 21:14:40 +1000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CADiSq7f-wOshNskMA5mcfKVnAVYhEP6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
Message-ID: <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>

On 22 August 2016 at 09:03, Nathaniel Smith <njs at pobox.com> wrote:
> On Aug 21, 2016 9:18 AM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
>>
> [...]
>> By contrast, I *know* Linux distros are in the habit of pulling
>> release tarballs from PyPI and feeding them directly into their
>> release pipelines, so the potential for unanticipated breakage seems
>> much higher there, and more likely to fall on community distros rather
>> than on commercial Python redistributors (simply because there are
>> more volunteers working on Linux based tooling than there are on
>> WIndows developer tools).
>
> But those release pipelines today have no problem handling the ~10% of
> sdists that are zips. For my personal projects actually I've always released
> .zip sdists exclusively (just because it seemed like a more universally
> convenient format than .tar.gz), and those packages have shown up in distros
> and no one has ever complained. Are any distros really hardcoding an
> assumption that .tar.gz is that only possible sdist format?

Thanks, that's good info that shows I was clearly being unduly
pessimistic about toolchain compatibility. It means the only salient
technical difference we're aware of between the two formats is that
the distutils and setuptools default settings on .tar.gz currently
result in smaller archives than the default settings for .zip.

In that case, perhaps the way to go for sdists (at least for now)
would be to continue allowing both .tar.gz and .zip, but disallow
uploading both of them for any given release?

The only remaining thing holding me back from being gung-ho about "OK,
let's just standardise on .zip, then" is the filesize consideration,
since that means implicitly asking Fastly to donate more sponsored
bandwidth, and likewise for anyone else running private PyPI mirrors,
etc.

Then sdist 2.0 (whenever that ends up bugging someone enough for them
to set out to define it) can be defined as a zip format, but with
compression applied to match or better the file size of the current
.tar.gz files.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From barry at python.org  Mon Aug 22 10:12:11 2016
From: barry at python.org (Barry Warsaw)
Date: Mon, 22 Aug 2016 10:12:11 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CADiSq7f-wOshNskMA5mcfKVnAVYhEP6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
Message-ID: <20160822101211.6b93ff84@anarchist.wooz.org>

On Aug 22, 2016, at 09:14 PM, Nick Coghlan wrote:

>In that case, perhaps the way to go for sdists (at least for now)
>would be to continue allowing both .tar.gz and .zip, but disallow
>uploading both of them for any given release?

I'd prefer allowing uploading of both formats for now, with mild encouragement
to at least include zips.

I'm fairly certain the Debian Python ecosystem in general can consume zip
sdists without problem, although there may be some individual packages that
need some adjustment.  I don't have any sense of large an impact it would be
to switch sdists from tar.gz to zip, but allowing for some overlap with
"gentle nudges" toward the preferred format would allow us to do the analysis
and give us some time to plan a transition.

Personally, I don't much care if the sdist format is tar.gz or zip, and it
does make sense to promote one archive format for both source and binary
distributions.  I mildly prefer tar.gz but for pretty shallow reasons
(i.e. not good enough :): one tool to rule them all (tar(1) both packs and
unpacks, while zip needs two commands), and I've been using tar for way longer
than zip so I'm a little more comfortable with it.

What I'd want most of all is a clearly documented transition plan, i.e. a PEP
that I can point other people to and say "we need to make sure all our tooling
is in place by date X because that's when .tar.gz is going away".

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160822/de4ed5b4/attachment-0001.sig>

From dholth at gmail.com  Mon Aug 22 10:29:23 2016
From: dholth at gmail.com (Daniel Holth)
Date: Mon, 22 Aug 2016 14:29:23 +0000
Subject: [Distutils] greater compression on pypi?
In-Reply-To: <20160822101211.6b93ff84@anarchist.wooz.org>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CADiSq7f-wOshNskMA5mcfKVnAVYhEP6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <20160822101211.6b93ff84@anarchist.wooz.org>
Message-ID: <CAG8k2+6_sSnwD0b88F8dgdon7dVr17Fhg_o9N2LYShvg=-FnTQ@mail.gmail.com>

Some obvious ideas about how to enable greater compression for pypi, should
anyone be motivated enough to do so.

1. If it's a zip, nested zips like so,

setup.py
README
(metadata)
data.zip

The metadata is easy to get to, and everything else requires a second
unpack operation. data.zip is stored, and only compressed by the outer
.zip. This could be done in a backwards compatible way.

Wheel could be revised to put everything except *.dist-info inside a zipped
*.data directory.

2. Sign the uncompressed data

Check hashes and signatures against the .tar file instead of .tar.gz when
doing pip install ... #sha256=nnn. For zip, check against a hash of all the
hashes of the uncompressed members.

3. Go crazy

pypi is now free to re-compress without additional input from the
publisher. Both .gz and .lzma versions etc. could be offered.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160822/73e5b5d6/attachment.html>

From donald at stufft.io  Mon Aug 22 10:46:14 2016
From: donald at stufft.io (Donald Stufft)
Date: Mon, 22 Aug 2016 10:46:14 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CADiSq7f-wOshNskMA5mcfKVnAVYhEP6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
Message-ID: <C6713523-CB23-4F2B-8B2E-0E29AA8EB006@stufft.io>


> On Aug 22, 2016, at 7:14 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> Thanks, that's good info that shows I was clearly being unduly
> pessimistic about toolchain compatibility. It means the only salient
> technical difference we're aware of between the two formats is that
> the distutils and setuptools default settings on .tar.gz currently
> result in smaller archives than the default settings for .zip.


I?m less worried about the Linux toolchains and I?m more worried about
the adhoc toolchains created by all the various publishers on PyPI. It?s
not a wide stretch to imagine release scripts that hard code in an
assumption on either .tar.gz _or_ .zip and picking one or the other will
inevitably break these people (albeit with a fairly simple fix in the
typical case). Picking the lesser used format just increases the number
of people who might end up having their release scripts or other misc
scripts broken because of it.

?
Donald Stufft




From barry at python.org  Mon Aug 22 10:53:51 2016
From: barry at python.org (Barry Warsaw)
Date: Mon, 22 Aug 2016 10:53:51 -0400
Subject: [Distutils] greater compression on pypi?
In-Reply-To: <CAG8k2+6_sSnwD0b88F8dgdon7dVr17Fhg_o9N2LYShvg=-FnTQ@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CADiSq7f-wOshNskMA5mcfKVnAVYhEP6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <20160822101211.6b93ff84@anarchist.wooz.org>
 <CAG8k2+6_sSnwD0b88F8dgdon7dVr17Fhg_o9N2LYShvg=-FnTQ@mail.gmail.com>
Message-ID: <20160822105351.75dc76bf@anarchist.wooz.org>

On Aug 22, 2016, at 02:29 PM, Daniel Holth wrote:

>1. If it's a zip, nested zips like so,
>
>setup.py
>README
>(metadata)
>data.zip

That would be much more disruptive to downstream consumers of sdists.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160822/058faf02/attachment.sig>

From donald at stufft.io  Mon Aug 22 10:59:15 2016
From: donald at stufft.io (Donald Stufft)
Date: Mon, 22 Aug 2016 10:59:15 -0400
Subject: [Distutils] greater compression on pypi?
In-Reply-To: <CAG8k2+6_sSnwD0b88F8dgdon7dVr17Fhg_o9N2LYShvg=-FnTQ@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CADiSq7f-wOshNskMA5mcfKVnAVYhEP6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <20160822101211.6b93ff84@anarchist.wooz.org>
 <CAG8k2+6_sSnwD0b88F8dgdon7dVr17Fhg_o9N2LYShvg=-FnTQ@mail.gmail.com>
Message-ID: <2E017907-0286-4C2A-B157-0F2F8E690EF7@stufft.io>


> On Aug 22, 2016, at 10:29 AM, Daniel Holth <dholth at gmail.com> wrote:
> 
> pypi is now free to re-compress without additional input from the publisher. Both .gz and .lzma versions etc. could be offered.


I am not opposed to changes to the file format to enable better compression, that being said I?m not particularly motivated by the ability to have PyPI re-compress things. It?s not something I can see PyPI ever taking advantage of. In addition to that, you can?t really just change the meaning of the existing things, like the #sha256=?. because you?ll break every released version of pip to date (for example).

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160822/b1840694/attachment.html>

From dholth at gmail.com  Mon Aug 22 11:12:08 2016
From: dholth at gmail.com (Daniel Holth)
Date: Mon, 22 Aug 2016 15:12:08 +0000
Subject: [Distutils] greater compression on pypi?
In-Reply-To: <2E017907-0286-4C2A-B157-0F2F8E690EF7@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CADiSq7f-wOshNskMA5mcfKVnAVYhEP6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <20160822101211.6b93ff84@anarchist.wooz.org>
 <CAG8k2+6_sSnwD0b88F8dgdon7dVr17Fhg_o9N2LYShvg=-FnTQ@mail.gmail.com>
 <2E017907-0286-4C2A-B157-0F2F8E690EF7@stufft.io>
Message-ID: <CAG8k2+5YQGjEyHrCfpzdpiP=asuWfCfDyYjHRFT84ZJmRFrLqw@mail.gmail.com>

In the example setup.py would at least do the nested unzipping, but you
would not be able to patch before running setup.py. Also expect disruption
from future sdists that don't have setup.py. Both changes if implemented
would surely be opt-in per dist and would not break everything all at once.
Of course there would have to be a different #sha256= tag for a hash of the
uncompressed contents. The 'multiple versions with different compression
algorithms' idea assumes some consumers don't have .xz and would prefer the
.gz version for example. FYI zipfile in Python 3 supports lzma compression.

On Mon, Aug 22, 2016 at 10:59 AM Donald Stufft <donald at stufft.io> wrote:

>
> On Aug 22, 2016, at 10:29 AM, Daniel Holth <dholth at gmail.com> wrote:
>
> pypi is now free to re-compress without additional input from the
> publisher. Both .gz and .lzma versions etc. could be offered.
>
>
>
> I am not opposed to changes to the file format to enable better
> compression, that being said I?m not particularly motivated by the ability
> to have PyPI re-compress things. It?s not something I can see PyPI ever
> taking advantage of. In addition to that, you can?t really just change the
> meaning of the existing things, like the #sha256=?. because you?ll break
> every released version of pip to date (for example).
>
> ?
>
> Donald Stufft
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160822/412d932b/attachment-0001.html>

From donald at stufft.io  Mon Aug 22 11:13:54 2016
From: donald at stufft.io (Donald Stufft)
Date: Mon, 22 Aug 2016 11:13:54 -0400
Subject: [Distutils] greater compression on pypi?
In-Reply-To: <CAG8k2+5YQGjEyHrCfpzdpiP=asuWfCfDyYjHRFT84ZJmRFrLqw@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CADiSq7f-wOshNskMA5mcfKVnAVYhEP6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <20160822101211.6b93ff84@anarchist.wooz.org>
 <CAG8k2+6_sSnwD0b88F8dgdon7dVr17Fhg_o9N2LYShvg=-FnTQ@mail.gmail.com>
 <2E017907-0286-4C2A-B157-0F2F8E690EF7@stufft.io>
 <CAG8k2+5YQGjEyHrCfpzdpiP=asuWfCfDyYjHRFT84ZJmRFrLqw@mail.gmail.com>
Message-ID: <C39B42D9-AB1D-497C-AC75-353A4CA66A3D@stufft.io>


> On Aug 22, 2016, at 11:12 AM, Daniel Holth <dholth at gmail.com> wrote:
> 
>  The 'multiple versions with different compression algorithms' idea assumes some consumers don't have .xz and would prefer the .gz version for example.

The more variants a file has, the less likely a cache hit is for any given request. The goal is to reduce the number of variants not increase them.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160822/c7a59369/attachment.html>

From ncoghlan at gmail.com  Mon Aug 22 11:49:48 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 23 Aug 2016 01:49:48 +1000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <C6713523-CB23-4F2B-8B2E-0E29AA8EB006@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CADiSq7f-wOshNskMA5mcfKVnAVYhEP6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <C6713523-CB23-4F2B-8B2E-0E29AA8EB006@stufft.io>
Message-ID: <CADiSq7cSX=yzHvVxsJmk831nBR4c=TZKkCv7v652G0=uvfSiCg@mail.gmail.com>

On 23 August 2016 at 00:46, Donald Stufft <donald at stufft.io> wrote:
>
>> On Aug 22, 2016, at 7:14 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> Thanks, that's good info that shows I was clearly being unduly
>> pessimistic about toolchain compatibility. It means the only salient
>> technical difference we're aware of between the two formats is that
>> the distutils and setuptools default settings on .tar.gz currently
>> result in smaller archives than the default settings for .zip.
>
> I?m less worried about the Linux toolchains and I?m more worried about
> the adhoc toolchains created by all the various publishers on PyPI. It?s
> not a wide stretch to imagine release scripts that hard code in an
> assumption on either .tar.gz _or_ .zip and picking one or the other will
> inevitably break these people (albeit with a fairly simple fix in the
> typical case). Picking the lesser used format just increases the number
> of people who might end up having their release scripts or other misc
> scripts broken because of it.

Yeah, that's fair. To summarise where we're at right now:

- the change to always use .tar.gz by default has been made in
distutils for 3.6, so if we want to do something else (like continue
allowing both .tar.gz and .zip and keep the Windows default as .zip),
we need to make that clear before the final beta in November
(*reverting* a behaviour change like that is permitted within the beta
period - otherwise the beta period wouldn't be particularly useful,
since we couldn't respond to community feedback)

- folks seem to be broadly in favour of standardising on ".zip"
whenever "formally define a new iteration of the sdist format" makes
it to the top of the collective todo list

- however, if we were to consolidate on a single sdist format *right
now* (regardless of whether that's .tar.gz or .zip), the concern is
that we risk breaking ad hoc automation and other tooling without a
compelling sales pitch to the folks that need to update their
toolchains about how the change will make *their* lives better

Furthermore, at a language ecosystem level, we don't particularly want
to inflict more disruption on either *nix users *or* Windows users
that are sufficiently engaged with the Python community to be
publishing their own software on PyPI - we still have that everpresent
"change overload" concern, especially for folks that have been
affected by both Python 3 *and* the packaging tooling changes in
recent years.

So I think the next step would be to summarise the changes to release
file hosting support and the permitted extensions in a PEP, with
details along the following lines:

Permitted formats:

- sdist (*.zip, *.tar.gz)
- bdist_wheel (*.whl)
- bdist_egg (*.egg)
- bdist_wininst (*.exe)
- bdist_msi (*.msi)

Dropped formats:

- bdist_dumb
- bdist_rpm (*.rpm)
- bdist_dmg (*.dmg)

Possible future design changes:

- defining sdist 2.0 (as an explicitly defined zip-based format)
- defining a successor to the egg format for the non-wheel use cases
- encouraging use of Windows Package Management over manual installer execution

The rationale for why the Windows formats get to stay when the other
platform specific formats are being dropped is implied by that last
line: we're expecting users on other platforms to be more comfortable
with using platform specific tooling to manage platform specific
formats (e.g. the system package manager on Linux, homebrew on Mac OS
X).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From p.f.moore at gmail.com  Mon Aug 22 12:17:23 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 22 Aug 2016 17:17:23 +0100
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CADiSq7cSX=yzHvVxsJmk831nBR4c=TZKkCv7v652G0=uvfSiCg@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CADiSq7f-wOshNskMA5mcfKVnAVYhEP6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <C6713523-CB23-4F2B-8B2E-0E29AA8EB006@stufft.io>
 <CADiSq7cSX=yzHvVxsJmk831nBR4c=TZKkCv7v652G0=uvfSiCg@mail.gmail.com>
Message-ID: <CACac1F8yPbiUseSyzt=yxiLgPBLoPQLWEutot9AUrBW=LTxZug@mail.gmail.com>

On 22 August 2016 at 16:49, Nick Coghlan <ncoghlan at gmail.com> wrote:
> - encouraging use of Windows Package Management over manual installer execution
>
> The rationale for why the Windows formats get to stay when the other
> platform specific formats are being dropped is implied by that last
> line: we're expecting users on other platforms to be more comfortable
> with using platform specific tooling to manage platform specific
> formats (e.g. the system package manager on Linux, homebrew on Mac OS
> X).

Could you clarify what you mean by this? Current usage on Windows (for
people who aren't already using wheels) is to run a bdist_msi or
bsidt_wininst installer.

You seen to be suggesting "encouraging use of Windows Package
Management" over this approach, but I'm not sure what precisely that
means?
Paul

From wes.turner at gmail.com  Mon Aug 22 13:55:24 2016
From: wes.turner at gmail.com (Wes Turner)
Date: Mon, 22 Aug 2016 12:55:24 -0500
Subject: [Distutils] greater compression on pypi?
In-Reply-To: <CAG8k2+6_sSnwD0b88F8dgdon7dVr17Fhg_o9N2LYShvg=-FnTQ@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CADiSq7f-wOshNskMA5mcfKVnAVYhEP6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <20160822101211.6b93ff84@anarchist.wooz.org>
 <CAG8k2+6_sSnwD0b88F8dgdon7dVr17Fhg_o9N2LYShvg=-FnTQ@mail.gmail.com>
Message-ID: <CACfEFw8rATtFRFA0ha61-LNOU6wqAWrBykde1c3o8JNfGGAQ4Q@mail.gmail.com>

On Monday, August 22, 2016, Daniel Holth <dholth at gmail.com> wrote:

> Some obvious ideas about how to enable greater compression for pypi,
> should anyone be motivated enough to do so.
>
> 1. If it's a zip, nested zips like so,
>
> setup.py
> README
> (metadata)
> data.zip
>
> The metadata is easy to get to, and everything else requires a second
> unpack operation. data.zip is stored, and only compressed by the outer
> .zip. This could be done in a backwards compatible way.
>
> Wheel could be revised to put everything except *.dist-info inside a
> zipped *.data directory.
>

"""

PEX is just a class that manages requirements (often embedded within PEX
files as egg distributions in the .deps directory) and autoimports them
into the sys.path, then executes a prescribed entry point.

If you read the code closely, you'll notice that it relies upon
monkeypatching zipimport. Inside the twitter.common.python library we've
provided a recursive zip importer derived from Google's pure Python
zipimport
<http://code.google.com/appengine/articles/django10_zipimport.html>module
that allows for depending upon eggs within eggs or zips (and so forth) so
that PEX files need not extract egg dependencies to disk a priori. This
even extends to C extensions (.so and .dylib files) which are written to
disk long enough to be dlopened before being unlinked.
 """ - https://pantsbuild.github.io/pex_design.html#pex-__main__py

find_eggs_in_zip and find_wheels_in_zip in
https://github.com/pantsbuild/pex/blob/master/pex/finders.py
may be helpful for building a recursive zip importer
(Which may or may not reduce total pypi bandwidth because DEFLATE)


> 2. Sign the uncompressed data
>
> Check hashes and signatures against the .tar file instead of .tar.gz when
> doing pip install ... #sha256=nnn. For zip, check against a hash of all the
> hashes of the uncompressed members.
>
> 3. Go crazy
>
> pypi is now free to re-compress without additional input from the
> publisher. Both .gz and .lzma versions etc. could be offered.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160822/fba822c7/attachment.html>

From wes.turner at gmail.com  Mon Aug 22 13:59:51 2016
From: wes.turner at gmail.com (Wes Turner)
Date: Mon, 22 Aug 2016 12:59:51 -0500
Subject: [Distutils] greater compression on pypi?
In-Reply-To: <C39B42D9-AB1D-497C-AC75-353A4CA66A3D@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CADiSq7f-wOshNskMA5mcfKVnAVYhEP6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <20160822101211.6b93ff84@anarchist.wooz.org>
 <CAG8k2+6_sSnwD0b88F8dgdon7dVr17Fhg_o9N2LYShvg=-FnTQ@mail.gmail.com>
 <2E017907-0286-4C2A-B157-0F2F8E690EF7@stufft.io>
 <CAG8k2+5YQGjEyHrCfpzdpiP=asuWfCfDyYjHRFT84ZJmRFrLqw@mail.gmail.com>
 <C39B42D9-AB1D-497C-AC75-353A4CA66A3D@stufft.io>
Message-ID: <CACfEFw-fG6oqu5becTHMEzixA4iOXy8nV8J9iEOy6nMQn0kKyQ@mail.gmail.com>

These docs explain the challeges of supporting caching with ETags, Gzip,
and conditional requests:

https://chibisov.github.io/drf-extensions/docs/#gzipped-etags

On Monday, August 22, 2016, Donald Stufft <donald at stufft.io> wrote:

>
> On Aug 22, 2016, at 11:12 AM, Daniel Holth <dholth at gmail.com
> <javascript:_e(%7B%7D,'cvml','dholth at gmail.com');>> wrote:
>
>  The 'multiple versions with different compression algorithms' idea
> assumes some consumers don't have .xz and would prefer the .gz version for
> example.
>
>
> The more variants a file has, the less likely a cache hit is for any given
> request. The goal is to reduce the number of variants not increase them.
>
> ?
> Donald Stufft
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160822/85d65f92/attachment.html>

From robertc at robertcollins.net  Mon Aug 22 14:22:45 2016
From: robertc at robertcollins.net (Robert Collins)
Date: Tue, 23 Aug 2016 06:22:45 +1200
Subject: [Distutils] unittest2 issue tracker?
In-Reply-To: <CAH+Pg9q9Er2M2DC7Gj_=Ls3Ug4f=jEVVHqG+Gxzg44CPHzTazQ@mail.gmail.com>
References: <CAH+Pg9q9Er2M2DC7Gj_=Ls3Ug4f=jEVVHqG+Gxzg44CPHzTazQ@mail.gmail.com>
Message-ID: <CAJ3HoZ2_qE-TRCbjB+kRXBZgDgOKNQ_n92xVC+XqdN6M+wcs0g@mail.gmail.com>

That is the right place for issues with the backport; note that as a
backport new features and issues that also exist in CPython's
master/tip branch should be reported to the python bugtracker, where
Paul pointed you.

Cheers,
Rob

On 22 August 2016 at 21:25, Cosimo Lupo <cosimo.lupo at daltonmaag.com> wrote:
> Hello,
>
> Sorry if this is not the right place to ask this sorts of questions.
>
> The PyPI page for unittest2 package links to
> https://code.google.com/archive/p/unittest-ext/issues as its "issue
> tracker", however the Google Code pages seems to be read-only nowadays, so I
> can't open new issues there.
>
> I've found an auto-exported Github clone for unittest-ext and opened a new
> issue there: https://github.com/testing-cabal/unittest-ext/issues/99
>
> However I'm not sure it's the right one, as there doesn't seem to be much
> activity in there.
>
> It would be good to know where I should report bugs for unittest2.
> Thank you!
>
> --
>
> Cosimo Lupo
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>

From dholth at gmail.com  Mon Aug 22 15:14:42 2016
From: dholth at gmail.com (Daniel Holth)
Date: Mon, 22 Aug 2016 19:14:42 +0000
Subject: [Distutils] greater compression on pypi?
In-Reply-To: <CACfEFw-fG6oqu5becTHMEzixA4iOXy8nV8J9iEOy6nMQn0kKyQ@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CADiSq7f-wOshNskMA5mcfKVnAVYhEP6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <20160822101211.6b93ff84@anarchist.wooz.org>
 <CAG8k2+6_sSnwD0b88F8dgdon7dVr17Fhg_o9N2LYShvg=-FnTQ@mail.gmail.com>
 <2E017907-0286-4C2A-B157-0F2F8E690EF7@stufft.io>
 <CAG8k2+5YQGjEyHrCfpzdpiP=asuWfCfDyYjHRFT84ZJmRFrLqw@mail.gmail.com>
 <C39B42D9-AB1D-497C-AC75-353A4CA66A3D@stufft.io>
 <CACfEFw-fG6oqu5becTHMEzixA4iOXy8nV8J9iEOy6nMQn0kKyQ@mail.gmail.com>
Message-ID: <CAG8k2+4xyKF+7qnw7ZYgJVjvkJVxLT=fEJbyeCbDrOD2VCbjDQ@mail.gmail.com>

Thanks for sharing! I'm not planning on implementing any of this any time
soon if ever but wanted to share the ideas.

On Mon, Aug 22, 2016, 13:59 Wes Turner <wes.turner at gmail.com> wrote:

> These docs explain the challeges of supporting caching with ETags, Gzip,
> and conditional requests:
>
> https://chibisov.github.io/drf-extensions/docs/#gzipped-etags
>
> On Monday, August 22, 2016, Donald Stufft <donald at stufft.io> wrote:
>
>>
>> On Aug 22, 2016, at 11:12 AM, Daniel Holth <dholth at gmail.com> wrote:
>>
>>  The 'multiple versions with different compression algorithms' idea
>> assumes some consumers don't have .xz and would prefer the .gz version for
>> example.
>>
>>
>> The more variants a file has, the less likely a cache hit is for any
>> given request. The goal is to reduce the number of variants not increase
>> them.
>>
>> ?
>> Donald Stufft
>>
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160822/b9a2f0ae/attachment.html>

From tritium-list at sdamon.com  Mon Aug 22 16:53:17 2016
From: tritium-list at sdamon.com (tritium-list at sdamon.com)
Date: Mon, 22 Aug 2016 16:53:17 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CADiSq7cSX=yzHvVxsJmk831nBR4c=TZKkCv7v652G0=uvfSiCg@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid> <CADiSq7f-wOshNskMA5mcfKVnAVYhEP
 6enV2+3S=Q+daP6dO1Qw@mail.gmail.com>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <C6713523-CB23-4F2B-8B2E-0E29AA8EB006@stufft.io>
 <CADiSq7cSX=yzHvVxsJmk831nBR4c=TZKkCv7v652G0=uvfSiCg@mail.gmail.com>
Message-ID: <100501d1fcb7$35ab0470$a1010d50$@hotmail.com>

> The rationale for why the Windows formats get to stay when the other
> platform specific formats are being dropped is implied by that last
> line: we're expecting users on other platforms to be more comfortable
> with using platform specific tooling to manage platform specific
> formats (e.g. the system package manager on Linux, homebrew on Mac OS
> X).
> 
> Cheers,
> Nick.

I'm a heavy Windows user.  Are you aware of a system package manager that I am not?  There's nuget (vs), choco (third party) and the Windows Store ...


From ncoghlan at gmail.com  Mon Aug 22 23:36:16 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 23 Aug 2016 13:36:16 +1000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <100501d1fcb7$35ab0470$a1010d50$@hotmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <C6713523-CB23-4F2B-8B2E-0E29AA8EB006@stufft.io>
 <CADiSq7cSX=yzHvVxsJmk831nBR4c=TZKkCv7v652G0=uvfSiCg@mail.gmail.com>
 <100501d1fcb7$35ab0470$a1010d50$@hotmail.com>
Message-ID: <CADiSq7fSaXSOgDsF5VCYt8v1Sr2kUXR2KWaXnP2dgpO7F_=PjA@mail.gmail.com>

On 23 August 2016 at 06:53,  <tritium-list at sdamon.com> wrote:
>> The rationale for why the Windows formats get to stay when the other
>> platform specific formats are being dropped is implied by that last
>> line: we're expecting users on other platforms to be more comfortable
>> with using platform specific tooling to manage platform specific
>> formats (e.g. the system package manager on Linux, homebrew on Mac OS
>> X).
>>
>> Cheers,
>> Nick.
>
> I'm a heavy Windows user.  Are you aware of a system package manager that I am not?  There's nuget (vs), choco (third party) and the Windows Store ...

I meant choco (community archive) and PackageManagement (system
integration, formerly known as OneGet):
https://blogs.technet.microsoft.com/packagemanagement/2015/04/28/introducing-packagemanagement-in-windows-10/

There's also the option of using conda and conda-forge as an option a
bit closer to what you get using homebrew on Mac OS X.

The reason I think we need to keep bdist_wininst and bdist_msi support
on PyPI for the time being is simply the fact that the idea of
software repositories and automated software management (rather than
downloading EXEs and MSIs through your web browser and running them)
is still a relatively novel concept on Windows, so it's going to take
time for it to be as accepted amongst Windows-focused Pythonistas as
comparable tools are amongst *nix focused developers.

However, now that Microsoft is actually providing support for
automated software management, I also think we should be encouraging
people to move in that direction, and if we're successful in that,
then eventually we'll be able to drop bdist_wininst and bdist_msi as
well, and nobody will miss them (since they're getting them from the
Chocalatey repo instead).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Mon Aug 22 23:46:17 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 23 Aug 2016 13:46:17 +1000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
Message-ID: <CADiSq7f-S=RbsSJHpRfkwqmeQWuoU+HcSbnyQJoMGTpj3JtniQ@mail.gmail.com>

On 16 August 2016 at 05:09, Donald Stufft <donald at stufft.io> wrote:
> Hello!
>
> I'd like to restrict what folks can upload to PyPI in an effort to help narrow
> the scope down and to enable more a more consistent experience for everyone.
>
> First off, we currently allow people to upload sdist, bdist_wheel, bdist_egg,
> bdist_dmg, bdist_dumb, bdist_msi, bdist_rpm, and bdist_wininst. However I think
> that we should try to get rid of support for most of these. Just for reference
> currently the number of files uploaded for each type of file looks like:
>
> * sdist: 506,585
> * bdist_wheel: 81,207
> * bdist_egg: 48,282
> * bdist_wininst: 14,002
> * bdist_dumb: 5,502
> * bdist_msi: 497
> * bdist_rpm: 464
> * bdist_dmg: 45
>
> Out of all of these, I think that we can easily remove bdist_dmg, bdist_rpm,
> and bdist_dumb. I also believe that there is a strong case for removing
> bdist_msi and bdist_wininst. I also think we should consider removing
> bdist_egg.
>
> First of all, when I say "remove", I mean disallow new uploads, but do not
> delete the existing files.

[snip]

> Next we have bdist_dmg, bdist_msi, and bdist_winist. I'm lumping these together
> because they're all OS specific installers for OSs that don't already have some
> sort of repository. This lack of a repository format for them means that random
> downloads are already the norm for people using these systems. For these, I
> think the usage numbers for bdist_dmg and bdist_msi easily suggest that they
> are not very important to continue to support, but it would be weird to
> eliminate them without also elminating bdist_wininst. The wininst format has
> the most usage out of all of the seldom used formats, however when we look at
> the downloads for the last 30 days only 0.42% of the downloads were for wininst
> files, so I think that it's pretty safe to remove them.

Based on the discussion about Windows package management, I realised
that "percentage of overall downloads" is probably the wrong way to
measure the importance of the platform specific downloads, since those
percentages will be suppressed by the overwhelming dominance of sdist
and wheel downloads for sdist-only and sdist+wheel projects.

Instead, I think the numbers we need to make good data driven
decisions about publisher and end user impact here are:

- for projects publishing bdist_egg, what percentage of downloads are bdist_egg?
- for projects publishing bdist_wininst, what percentage of downloads
are bdist_wininst?
- ditto for bdist_msi and bdist_dmg

For bdist_rpm, the Fedora/CentOS/RHEL ecosystem has the Fedora COPR
service at http://copr.fedoraproject.org/ now, and openSuSE has long
offered http://openbuildservice.org/ to build and publish RPMs for a
range of RPM based distributions, so I'm already happy for us to end
bdist_rpm support on PyPI and direct people that want to publish RPMs
to those services instead.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From p.f.moore at gmail.com  Tue Aug 23 05:36:35 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 23 Aug 2016 10:36:35 +0100
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CADiSq7fSaXSOgDsF5VCYt8v1Sr2kUXR2KWaXnP2dgpO7F_=PjA@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <C6713523-CB23-4F2B-8B2E-0E29AA8EB006@stufft.io>
 <CADiSq7cSX=yzHvVxsJmk831nBR4c=TZKkCv7v652G0=uvfSiCg@mail.gmail.com>
 <100501d1fcb7$35ab0470$a1010d50$@hotmail.com>
 <CADiSq7fSaXSOgDsF5VCYt8v1Sr2kUXR2KWaXnP2dgpO7F_=PjA@mail.gmail.com>
Message-ID: <CACac1F9m2jXVZyWCpQxc7xGup-Wt2Qouf6J37x=Wh91nHgGPVg@mail.gmail.com>

On 23 August 2016 at 04:36, Nick Coghlan <ncoghlan at gmail.com> wrote:
> I meant choco (community archive) and PackageManagement (system
> integration, formerly known as OneGet):
> https://blogs.technet.microsoft.com/packagemanagement/2015/04/28/introducing-packagemanagement-in-windows-10/

Neither Chocolatey nor OneGet do any form of repackaging or vendor
management, in the way that Linux distros do, though.

OneGet isn't a package manager - it's a "package manager manager" in
that it simply provides a mechanism to unify *other* package managers
(such as NuGet or Chocolatey) under a single command set.

Chocolatey *is* a package manager, but for projects like Python it
simply provides a script that says "Locate the Python MSI from this
URL, download and install it" - so nothing more than automated
instructions for what users are doing right now. And for Python
packages, it similarly just packages up a script for silent install.
There's very few such bundles for Python packages that I could find at
the moment, but if & when they do get contributed, I'd pretty much
hope that all they did was "pip install foo==1.2.3" (with a bit of
dependency and error checking).

So I don't think that in the medium term there's going to be much
practical change in the state of things on Windows:

- Users install Python from the published python.org installers
- Users install packages using pip and wheels from PyPI
- Plus some exceptions, where people need to use sdists, or
independently published wheels, or worse still, wininst/msi installers
because that's all available

Whether that process is manual, or hidden behind some form of scripted
process, won't alter the underlying infrastructure.

I don't see any sign of *anyone* working on a curated distribution for
Windows along the lines of Linux distros or Homebrew. (Unless you
count cross-platform stacks like conda, which IMO are a different
scenario than "system" Python installs).

Paul

From ncoghlan at gmail.com  Tue Aug 23 07:25:20 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 23 Aug 2016 21:25:20 +1000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CACac1F9m2jXVZyWCpQxc7xGup-Wt2Qouf6J37x=Wh91nHgGPVg@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <C6713523-CB23-4F2B-8B2E-0E29AA8EB006@stufft.io>
 <CADiSq7cSX=yzHvVxsJmk831nBR4c=TZKkCv7v652G0=uvfSiCg@mail.gmail.com>
 <100501d1fcb7$35ab0470$a1010d50$@hotmail.com>
 <CADiSq7fSaXSOgDsF5VCYt8v1Sr2kUXR2KWaXnP2dgpO7F_=PjA@mail.gmail.com>
 <CACac1F9m2jXVZyWCpQxc7xGup-Wt2Qouf6J37x=Wh91nHgGPVg@mail.gmail.com>
Message-ID: <CADiSq7eEbAaUzvwr7CGTtjYAym_tS855cJAun0=2ZzTLVWodsg@mail.gmail.com>

On 23 August 2016 at 19:36, Paul Moore <p.f.moore at gmail.com> wrote:
> So I don't think that in the medium term there's going to be much
> practical change in the state of things on Windows:
>
> - Users install Python from the published python.org installers
> - Users install packages using pip and wheels from PyPI
> - Plus some exceptions, where people need to use sdists, or
> independently published wheels, or worse still, wininst/msi installers
> because that's all available
>
> Whether that process is manual, or hidden behind some form of scripted
> process, won't alter the underlying infrastructure.
>
> I don't see any sign of *anyone* working on a curated distribution for
> Windows along the lines of Linux distros or Homebrew. (Unless you
> count cross-platform stacks like conda, which IMO are a different
> scenario than "system" Python installs).

OK, cool - that gives us all the more reason to retain bdist_wininst
and bdist_msi hosting support. However, I do think it makes sense for
us to say up front that we'll reconsider that decision if something
akin to homebrew gains traction amongst developers running Windows the
way homebrew has amongst open source users running Mac OS X.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Tue Aug 23 10:12:13 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 23 Aug 2016 16:12:13 +0200
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <C6713523-CB23-4F2B-8B2E-0E29AA8EB006@stufft.io>
 <CADiSq7cSX=yzHvVxsJmk831nBR4c=TZKkCv7v652G0=uvfSiCg@mail.gmail.com>
 <100501d1fcb7$35ab0470$a1010d50$@hotmail.com>
 <CADiSq7fSaXSOgDsF5VCYt8v1Sr2kUXR2KWaXnP2dgpO7F_=PjA@mail.gmail.com>
 <CACac1F9m2jXVZyWCpQxc7xGup-Wt2Qouf6J37x=Wh91nHgGPVg@mail.gmail.com>
Message-ID: <20160823161213.0c174fe8@fsol>

On Tue, 23 Aug 2016 10:36:35 +0100
Paul Moore <p.f.moore at gmail.com> wrote:
> 
> I don't see any sign of *anyone* working on a curated distribution for
> Windows along the lines of Linux distros or Homebrew. (Unless you
> count cross-platform stacks like conda, which IMO are a different
> scenario than "system" Python installs).

Under Windows, there isn't much moral difference between a conda install
(really a Miniconda or Anaconda install) and a python.org Python
install.  You can even install Anaconda or Miniconda system-wide.

(Miniconda is a minimal install of Python + conda, while Anaconda is a
pretty large selection of packages in addition - though not the entire
official repo contents, and not counting community packages)

Regards

Antoine.



From donald at stufft.io  Tue Aug 23 10:33:07 2016
From: donald at stufft.io (Donald Stufft)
Date: Tue, 23 Aug 2016 10:33:07 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CADiSq7eEbAaUzvwr7CGTtjYAym_tS855cJAun0=2ZzTLVWodsg@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <C6713523-CB23-4F2B-8B2E-0E29AA8EB006@stufft.io>
 <CADiSq7cSX=yzHvVxsJmk831nBR4c=TZKkCv7v652G0=uvfSiCg@mail.gmail.com>
 <100501d1fcb7$35ab0470$a1010d50$@hotmail.com>
 <CADiSq7fSaXSOgDsF5VCYt8v1Sr2kUXR2KWaXnP2dgpO7F_=PjA@mail.gmail.com>
 <CACac1F9m2jXVZyWCpQxc7xGup-Wt2Qouf6J37x=Wh91nHgGPVg@mail.gmail.com>
 <CADiSq7eEbAaUzvwr7CGTtjYAym_tS855cJAun0=2ZzTLVWodsg@mail.gmail.com>
Message-ID: <32DA07E2-7DF2-412D-AFE5-FFFF05BA57C6@stufft.io>


> On Aug 23, 2016, at 7:25 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> OK, cool - that gives us all the more reason to retain bdist_wininst
> and bdist_msi hosting support. However, I do think it makes sense for
> us to say up front that we'll reconsider that decision if something
> akin to homebrew gains traction amongst developers running Windows the
> way homebrew has amongst open source users running Mac OS X.


I still don?t think there?s a whole lot of benefit to retaining them even
now. In the last 30 days, 90% of the downloads of bdist_wininst were
generated by things that I know for a fact to be mirroring clients (almost
all entirely bandersnatch). The next highest source of downloads was coming
from setuptools, at 7%. Over 75% of the downloads from setuptools are for
coverage.py, which tells me that it?s likely being triggered by test_requires
and would be covered by teaching setuptools how to wheel instead.

For bdist_msi, 96% of all downloads come from things we know to be mirroring
clients.

For bdist_dmg, 97% of all downloads come from things we know to be mirroring
clients.

For bdist_egg, 80% of all downloads come from things we know to be mirroring
clients.

For reference:

For sdist, 30% of all downloads come from things we know to be mirroring
clients.

For bdist_wheel, 6% of all downloads come from things we know to be mirroring
clients.

It?s hard to get per project numbers for these (or at least, it takes a more
complex query than I can manage with my head here). However, I think it?s
pretty telling that when you start looking at other formats, not only is the
primary consumer tools that just indiscriminately download everything from PyPI,
but almost *all* of the consumers of those files are tools that just
indiscriminately download everything. Unless there are users of those mirrors who
follow vastly different usage patterns than what we see on PyPI itself, the primary
purpose of bdist_wininst, bdist_msi, bdist_dmg, etc on PyPI is to consume disk space
and bandwidth via the mirroring infrastructure.

I?d also like to note, that the numbers above are conservative on what they
consider to be a ?mirroring client?. For instance, devpi used to use the default
requests user-agent, and we see downloads via the requests user agent, but did not
count them as mirroring clients because it could be some other script doing the
downloading.

?
Donald Stufft




From p.f.moore at gmail.com  Tue Aug 23 10:49:43 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 23 Aug 2016 15:49:43 +0100
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <20160823161213.0c174fe8@fsol>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <C6713523-CB23-4F2B-8B2E-0E29AA8EB006@stufft.io>
 <CADiSq7cSX=yzHvVxsJmk831nBR4c=TZKkCv7v652G0=uvfSiCg@mail.gmail.com>
 <100501d1fcb7$35ab0470$a1010d50$@hotmail.com>
 <CADiSq7fSaXSOgDsF5VCYt8v1Sr2kUXR2KWaXnP2dgpO7F_=PjA@mail.gmail.com>
 <CACac1F9m2jXVZyWCpQxc7xGup-Wt2Qouf6J37x=Wh91nHgGPVg@mail.gmail.com>
 <20160823161213.0c174fe8@fsol>
Message-ID: <CACac1F81-FoNwz2rJjRKsWo_cQsPtJ2e+AjwuXMp-Ekduzc4oQ@mail.gmail.com>

On 23 August 2016 at 15:12, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Tue, 23 Aug 2016 10:36:35 +0100
> Paul Moore <p.f.moore at gmail.com> wrote:
>>
>> I don't see any sign of *anyone* working on a curated distribution for
>> Windows along the lines of Linux distros or Homebrew. (Unless you
>> count cross-platform stacks like conda, which IMO are a different
>> scenario than "system" Python installs).
>
> Under Windows, there isn't much moral difference between a conda install
> (really a Miniconda or Anaconda install) and a python.org Python
> install.  You can even install Anaconda or Miniconda system-wide.
>
> (Miniconda is a minimal install of Python + conda, while Anaconda is a
> pretty large selection of packages in addition - though not the entire
> official repo contents, and not counting community packages)

Agreed - sorry, I was responding more to Nick's implication around
"system" management of Python, conda already has a well-established
management process, but AIUI, it's largely independent of PyPI/pip
(although it interoperates with those). So in the context of Windows
package managers, conda is no more relevant there than it is in
relation to (say) rpm or deb.

Realistically, though, I'd expect people wanting a Python stack on
Windows to gravitate more and more towards distributions like Anaconda
(maybe less so outside its core area of scientific use) with the
remaining users sticking to the python.org builds and pip.

However, I'm not that keen on continuing support for bdist_wininst and
bdist_msi. I'd rather put the effort into making pip and wheels a
compelling solution for such users. IMO, it already is for command
line use (the more so as more hard to build packages start providing
wheels). The place where pip falls down is for people who want to
stick to a GUI. For those people, I believe Idle is likely to be
getting an interface to pip, and I'd like to see other tools like PTVS
and PyCharm get similar capabilities (if they don't have them
already). That may mean exposing an interface to pip's install
mechanisms that's better than "use subprocess to call pip", but we'd
need some concrete use cases to work out the best form of such
interfaces.

On the matter of Chocolatey, it should be pretty trivial to create
recipes for installing Python packages via pip. So I'd see pip+wheel
as remaining the standard interface pretty much indefinitely.

Paul

From wes.turner at gmail.com  Tue Aug 23 12:45:16 2016
From: wes.turner at gmail.com (Wes Turner)
Date: Tue, 23 Aug 2016 11:45:16 -0500
Subject: [Distutils] Deprecating little used file types/extensions on PyPI?
In-Reply-To: <32DA07E2-7DF2-412D-AFE5-FFFF05BA57C6@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <C6713523-CB23-4F2B-8B2E-0E29AA8EB006@stufft.io>
 <CADiSq7cSX=yzHvVxsJmk831nBR4c=TZKkCv7v652G0=uvfSiCg@mail.gmail.com>
 <100501d1fcb7$35ab0470$a1010d50$@hotmail.com>
 <CADiSq7fSaXSOgDsF5VCYt8v1Sr2kUXR2KWaXnP2dgpO7F_=PjA@mail.gmail.com>
 <CACac1F9m2jXVZyWCpQxc7xGup-Wt2Qouf6J37x=Wh91nHgGPVg@mail.gmail.com>
 <CADiSq7eEbAaUzvwr7CGTtjYAym_tS855cJAun0=2ZzTLVWodsg@mail.gmail.com>
 <32DA07E2-7DF2-412D-AFE5-FFFF05BA57C6@stufft.io>
Message-ID: <CACfEFw9n_gn=vm_n5vbcc0StxbfWpz6OxN8L2zx=VuQeMBCaug@mail.gmail.com>

On Tuesday, August 23, 2016, Donald Stufft <donald at stufft.io
<javascript:_e(%7B%7D,'cvml','donald at stufft.io');>> wrote:

>
>
> It?s hard to get per project numbers for these (or at least, it takes a
> more
> complex query than I can manage with my head here). However, I think it?s
> pretty telling that when you start looking at other formats, not only is
> the
> primary consumer tools that just indiscriminately download everything from
> PyPI,
> but almost *all* of the consumers of those files are tools that just
> indiscriminately download everything. Unless there are users of those
> mirrors who
> follow vastly different usage patterns than what we see on PyPI itself,
> the primary
> purpose of bdist_wininst, bdist_msi, bdist_dmg, etc on PyPI is to consume
> disk space
> and bandwidth via the mirroring infrastructure.


Other ways to reduce mirroring bandwidth:

Caching proxies:

- devpi (proxy cache only what is needed)
  http://doc.devpi.net/latest/quickstart-pypimirror.html
- "pypi proxy" https://www.google.com/search?q=pypiproxy

- nginx caching proxy
  - https://gist.github.com/dctrwatson/5785638
    - this would need to be updated for the pypi.org warehouse routes
    - it may be worth linking to these in the/some mirroring infrastructure
docs

Pip --offline switch

- Internet was down the other day and i knew the packages were locally
cached but I couldn't figure out how to make pip work w/ local offline
dependency resolution

More efficient mirroring implementations:

- rsync
- bup
- DRPM


... delta downloads would be more efficient here:
https://github.com/pypa/warehouse/issues/347
"Add API endpoint to get latest version of all projects"




> I?d also like to note, that the numbers above are conservative on what they
> consider to be a ?mirroring client?. For instance, devpi used to use the
> default
> requests user-agent, and we see downloads via the requests user agent, but
> did not
> count them as mirroring clients because it could be some other script
> doing the
> downloading.
>
> ?
> Donald Stufft
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160823/ca53e2e3/attachment.html>

From donald at stufft.io  Tue Aug 23 12:46:51 2016
From: donald at stufft.io (Donald Stufft)
Date: Tue, 23 Aug 2016 12:46:51 -0400
Subject: [Distutils] PEP 527 - Removing Un(der)used file types/extensions on
 PyPI
Message-ID: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>

Since it seemed like there was enough here for a proper PEP I went ahead and
write one up, which is now PEP 527. The tl;dr of it is that:

* Everything but sdist, bdist_wheel, and bdist_egg get deprecated.
* The only allowed extension for sdist is ``.tar.gz``.
* Phased in Deprecation.

I've included the text below, or you can see it online at
https://www.python.org/dev/peps/pep-0527/ once the PEP website is updated.


---------------------------------------------


Abstract
========

This PEP recommends deprecating, and ultimately removing, support for uploading
certain unused or under used file types and extensions to PyPI. In particular
it recommends disallowing further uploads of any files of the types
``bdist_dumb``, ``bdist_rpm``, ``bdist_dmg``, ``bdist_msi``, and
``bdist_wininst``, leaving PyPI to only accept new uploads of the ``sdist``,
``bdist_wheel``, and ``bdist_egg`` file types.

In addition, this PEP proposes removing support for new uploads of sdists using
the ``.tar``, ``.tar.bz2``, ``.tar.xz``, ``.zip``, ``.tar.Z``, ``.tgz``,
``.tbz``, and any other extension besides ``.tar.gz``.



Rationale
=========

File Formats
------------

Currently PyPI supports the following file types:

* ``sdist``
* ``bdist_wheel``
* ``bdist_egg``
* ``bdist_wininst``
* ``bdist_msi``
* ``bdist_dmg``
* ``bdist_rpm``
* ``bdist_dumb``

However, these different types of files have varying amounts of usefulness or
general use in the ecosystem. Continuing to support them adds a maintenance
burden on PyPI as well as tool authors and incurs a cost in both bandwidth and
disk space not only on PyPI itself, but also on any mirrors of PyPI.

bdist_dumb
~~~~~~~~~~

As it's name implies, ``bdist_dumb`` is not a very complex format, however it
is so simple as to be worthless for actual usage.

For instance, if you're using something like pyenv on macOS and you're building
a library using Python 3.5, then ``bdist_dumb`` will produce a ``.tar.gz`` file
named something like ``exampleproject-1.0.macosx-10.11-x86_64.tar.gz``. Right
off the bat this file name is somewhat difficult to differentiate from an
``sdist`` since they both use the same file extension (and with the legacy pre
PEP 440 versions, ``1.0-macosx-10.11-x86_64`` is a valid, although quite silly,
version number). However, once you open up the created ``.tar.gz``, you'd find
that there is no metadata inside that could be used for things like dependency
discovery and in fact, it is quite simply a tarball containing hardcoded paths
to wherever files would have been installed on the computer creating the
``bdist_dumb``. Going back to our pyenv on macOS example, this means that if I
created it, it would contain files like:

``Users/dstufft/.pyenv/versions/3.5.2/lib/python3.5/site-packages/example.py``


bdist_rpm
~~~~~~~~~

The ``bdist_rpm`` format on PyPI allows people to upload ``.rpm`` files for
end users to manually download by hand and then manually install by hand.
However, the common usage of ``rpm`` is with a specially designed repository
that allows automatic installation of dependencies, upgrades, etc which PyPI
does not provide. Thus, it is a type of file that is barely being used on PyPI
with only ~460 files of this type having been uploaded to PyPI (out a total of
662,544).

In addition, services like `COPR <https://copr.fedorainfracloud.org/>`_ provide
a better supported mechanism for publishing and using RPM files than we're ever
likely to get on PyPI.


bdist_dmg, bdist_msi, and bdist_wininst
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``bdist_dmg``, ``bdist_msi``, and ``bdist_winist`` formats are similar in
that they are an OS specific installer that will only install a library into an
environment and are not designed for real user facing installs of applications
(which would require things like bundling a Python interpreter and the like).

Out of these three, the usage for ``bdist_dmg`` and ``bdist_msi`` is very low,
with only ~500 ``bdist_msi`` files and ~50 ``bdist_dmg`` files having been
uploaded to PyPI. The ``bdist_wininst`` format has more use, with ~14,000 files
having ever been uploaded to PyPI.

It's quite easy to look at the low usage of ``bdist_dmg`` and ``bdist_msi`` and
conclude that removing them will be fairly low impact, however
``bdist_wininst`` has several orders of magnitude more usage. This is somewhat
misleading though, because although it has more people *uploading* those files
the actual usage of those uploaded files is fairly low. Taking a look at the
previous 30 days, we can see that 90% of all downloads of ``bdist_winist``
files from PyPI were generated by the mirroring infrastructure and 7% of them
were generated by setuptools (which can currently be better covered by
``bdist_egg`` files).

Given the small number of files uploaded for ``bdist_dmg`` and ``bdist_msi``
and that ``bdist_wininst`` is largely existing to either consume bandwidth and
disk space via the mirroring infrastructure *or* could be trivially replaced
with ``bdist_egg``, this PEP proposes to include these three formats in the
list of those to be disallowed.


File Extensions
---------------

Currently ``sdist`` supports a wide variety of file extensions like `.tar.gz``,
``.tar``, ``.tar.bz2``, ``.tar.xz``, ``.zip``, ``.tar.Z``, ``.tgz``, and
``.tbz``. However, of those the only extensions which get anything more than
negligable usage is ``.tar.gz`` with 444,338 sdists currently, ``.zip`` with
58,774 sdists currently, and ``.tar.bz2`` with 3,265 sdists currently.

Having multiple formats accepted requires tooling both within PyPI and outside
of PyPI to handle all of the various extensions that *might* be used (even if
nobody is currently using them). This doesn't only affect PyPI, but ripples out
throughout the ecosystem. In addition, the different formats all have different
requirements for what optional C libraries Python was linked against and
different requirements for what versions of Python they support. In addition,
multiple formats also create a weird situation where there may be two
``sdist`` files for a particular project/release with subtly different content.

It's easy to advocate that anything outside of ``.tar.gz``, ``.zip``, and
``.tar.bz2`` should be disallowed. Outside of a tiny handful, nobody has
actively been uploading these other types of files in the ~15 years of PyPI's
existence so they've obviously not been particularly useful. In addition, while
``.tar.xz`` is theoretically a nicer format than the other ``.tar.*`` formats
due to the better compression ratio achieved by LZMA, it is only available in
Python 3.3+ and has an optional dependency on the lzma C library.

Looking at the three extensions we *do* have in current use, it's also fairly
easy to conclude that ``.tar.bz2`` can be disallowed as well. It has a fairly
small number of files ever uploaded with it and it requires an additional
optional C library to handle the bzip2 compression.

Finally we get down to ``.tar.gz`` and ``.zip``. Looking at the pure numbers
for these two, we can see that ``.tar.gz`` is by far the most uploaded format,
with 444,338 total uploaded compared to ``.zip``'s 58,774 and on POSIX
operating systems ``.tar.gz`` is also the default produced by all currently
released versions of Python and setuptools. In addition, these two file types
both use the same C library (``zlib``) which is also required for
``bdist_wheel`` and ``bdist_egg``. The two wrinkles with deciding between
``.tar.gz`` and ``.zip`` is that while on POSIX operating systems ``.tar.gz``
is the default, on Windows ``.zip`` is the default and the ``bdist_wheel``
format also uses zip.

This PEP proposes that we drop the use of ``.zip`` extensions for sdists on
PyPI and standardize around ``.tar.gz``. For both extensions there are going to
be automation designed by end users which are making assumptions about what the
file extension produced by the ``sdist`` command will be. Changing either
default will break some number of those, so by changing the default of ``.zip``
to ``.tar.gz`` we minimize the amount of breakage by taking the smaller number
of users and making them match the larger number. In addition, it's more likely
to see Windows users upgrade their setuptools and Python releases on a faster
timescale than POSIX users. POSIX users often get their Python and setuptools
from their OS vendor and are discouraged or actively prevented from upgrading
them outside of complete OS upgrades while Windows users *must* install Python
and setuptools on their own, and thus are more able to upgrade those pieces
without triggering a complete OS upgrade.

While it is true that switching to ``.zip`` would align ``sdist`` with
``bdist_wheel`` in terms of format, this is not a very large benefit because
both formats are able to be manipulated with the Python standard library just
as easily and both require the same C library (``zlib``). It is also true that
Windows has support for ``.zip`` files out of the box but requires third party
software for ``.tar.gz``, however only 0.6% of downloads for sdists on PyPI are
initiated by browsers and we can assume that only a fraction of those 0.6% are
Windows users who want to manually extract the file and do not have a means of
extracting a ``.tar.gz``, particularly since Python itself can be used to
extract a ``.tar.gz`` via the command line since version 3.4. In addition, the
use of ``.tar.gz`` will result in smaller sdists which will reduce the amount
of bandwidth and disk space consumed by ``sdist`` files.


Removal Process
===============

This PEP does **NOT** propose removing any existing files from PyPI, only
disallowing new ones from being uploaded. This restriction will be phased in on
a per-project basis to allow projects to adjust to the new restrictions where
applicable.

First, any *existing* projects will be flagged to allow legacy file types to be
uploaded, and any project without that flag (i.e. new projects) will not be
able to upload anything but ``sdist`` with a ``.tar.gz`` extension,
``bdist_wheel``, and ``bdist_egg``. Then, any existing projects that have never
uploaded a file that requires the legacy file type flag will have that flag
removed, also making them fall under the new restrictions. Finally, an email
will be generated to the maintainers of all projects still given the legacy
flag, which will inform them of the upcoming new restrictions on uploads and
tell them that these restrictions will be applied to future uploads to their
projects starting in 1 month. This email should also contain work arounds for
older versions of Python/setuptools on Windows, to get a ``.tar.gz`` by
default. Finally, after 1 month all projects will have the legacy file type
flag removed, and support for uploading these types of files will cease to
exist on PyPI.

This plan should provide minimal disruption since it does not remove any
existing files, and the types of files it does prevent from being uploaded are
either not particularly useful (or used) types of files *or* they can continue
to upload a similar type of file with a slight change to their process.


?
Donald Stufft




From donald at stufft.io  Tue Aug 23 12:48:28 2016
From: donald at stufft.io (Donald Stufft)
Date: Tue, 23 Aug 2016 12:48:28 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAP1=2W4rDBRO2w5qbqpwzr49uHZyM6+7icWgNf2adp6r+SdVHg@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <C6713523-CB23-4F2B-8B2E-0E29AA8EB006@stufft.io>
 <CADiSq7cSX=yzHvVxsJmk831nBR4c=TZKkCv7v652G0=uvfSiCg@mail.gmail.com>
 <100501d1fcb7$35ab0470$a1010d50$@hotmail.com>
 <CADiSq7fSaXSOgDsF5VCYt8v1Sr2kUXR2KWaXnP2dgpO7F_=PjA@mail.gmail.com>
 <CACac1F9m2jXVZyWCpQxc7xGup-Wt2Qouf6J37x=Wh91nHgGPVg@mail.gmail.com>
 <CADiSq7eEbAaUzvwr7CGTtjYAym_tS855cJAun0=2ZzTLVWodsg@mail.gmail.com>
 <32DA07E2-7DF2-412D-AFE5-FFFF05BA57C6@stufft.io>
 <CAP1=2W4rDBRO2w5qbqpwzr49uHZyM6+7icWgNf2adp6r+SdVHg@mail.gmail.com>
Message-ID: <33234E1F-5DBA-4A85-BB01-1A49AE0831EB@stufft.io>

See https://mail.python.org/pipermail/distutils-sig/2016-August/029542.html <https://mail.python.org/pipermail/distutils-sig/2016-August/029542.html> for a PEP!

> On Aug 23, 2016, at 12:37 PM, Brett Cannon <brett at python.org> wrote:
> 
> 
> 
> On Tue, 23 Aug 2016 at 07:33 Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
> 
> > On Aug 23, 2016, at 7:25 AM, Nick Coghlan <ncoghlan at gmail.com <mailto:ncoghlan at gmail.com>> wrote:
> >
> > OK, cool - that gives us all the more reason to retain bdist_wininst
> > and bdist_msi hosting support. However, I do think it makes sense for
> > us to say up front that we'll reconsider that decision if something
> > akin to homebrew gains traction amongst developers running Windows the
> > way homebrew has amongst open source users running Mac OS X.
> 
> 
> I still don?t think there?s a whole lot of benefit to retaining them even
> now. In the last 30 days, 90% of the downloads of bdist_wininst were
> generated by things that I know for a fact to be mirroring clients (almost
> all entirely bandersnatch). The next highest source of downloads was coming
> from setuptools, at 7%. Over 75% of the downloads from setuptools are for
> coverage.py, which tells me that it?s likely being triggered by test_requires
> and would be covered by teaching setuptools how to wheel instead.
> 
> For bdist_msi, 96% of all downloads come from things we know to be mirroring
> clients.
> 
> For bdist_dmg, 97% of all downloads come from things we know to be mirroring
> clients.
> 
> For bdist_egg, 80% of all downloads come from things we know to be mirroring
> clients.
> 
> For reference:
> 
> For sdist, 30% of all downloads come from things we know to be mirroring
> clients.
> 
> For bdist_wheel, 6% of all downloads come from things we know to be mirroring
> clients.
> 
> It?s hard to get per project numbers for these (or at least, it takes a more
> complex query than I can manage with my head here). However, I think it?s
> pretty telling that when you start looking at other formats, not only is the
> primary consumer tools that just indiscriminately download everything from PyPI,
> but almost *all* of the consumers of those files are tools that just
> indiscriminately download everything. Unless there are users of those mirrors who
> follow vastly different usage patterns than what we see on PyPI itself, the primary
> purpose of bdist_wininst, bdist_msi, bdist_dmg, etc on PyPI is to consume disk space
> and bandwidth via the mirroring infrastructure.
> 
> I?d also like to note, that the numbers above are conservative on what they
> consider to be a ?mirroring client?. For instance, devpi used to use the default
> requests user-agent, and we see downloads via the requests user agent, but did not
> count them as mirroring clients because it could be some other script doing the
> downloading.
> 
> I should also mention I have never come across anyone at Microsoft use the bdist_msi or bdist_winst installers (I've added Steve to this email in case my experience is wrong). Everyone I have encountered either uses conda or pip+wheels (hence why I keep poking the sdist and build API ideas as I want to give Christophe Gohlke something else to do with his time than provide wheels on Windows).


?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160823/a6c984be/attachment-0001.html>

From brett at python.org  Tue Aug 23 12:37:58 2016
From: brett at python.org (Brett Cannon)
Date: Tue, 23 Aug 2016 16:37:58 +0000
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <32DA07E2-7DF2-412D-AFE5-FFFF05BA57C6@stufft.io>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <C6713523-CB23-4F2B-8B2E-0E29AA8EB006@stufft.io>
 <CADiSq7cSX=yzHvVxsJmk831nBR4c=TZKkCv7v652G0=uvfSiCg@mail.gmail.com>
 <100501d1fcb7$35ab0470$a1010d50$@hotmail.com>
 <CADiSq7fSaXSOgDsF5VCYt8v1Sr2kUXR2KWaXnP2dgpO7F_=PjA@mail.gmail.com>
 <CACac1F9m2jXVZyWCpQxc7xGup-Wt2Qouf6J37x=Wh91nHgGPVg@mail.gmail.com>
 <CADiSq7eEbAaUzvwr7CGTtjYAym_tS855cJAun0=2ZzTLVWodsg@mail.gmail.com>
 <32DA07E2-7DF2-412D-AFE5-FFFF05BA57C6@stufft.io>
Message-ID: <CAP1=2W4rDBRO2w5qbqpwzr49uHZyM6+7icWgNf2adp6r+SdVHg@mail.gmail.com>

On Tue, 23 Aug 2016 at 07:33 Donald Stufft <donald at stufft.io> wrote:

>
> > On Aug 23, 2016, at 7:25 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >
> > OK, cool - that gives us all the more reason to retain bdist_wininst
> > and bdist_msi hosting support. However, I do think it makes sense for
> > us to say up front that we'll reconsider that decision if something
> > akin to homebrew gains traction amongst developers running Windows the
> > way homebrew has amongst open source users running Mac OS X.
>
>
> I still don?t think there?s a whole lot of benefit to retaining them even
> now. In the last 30 days, 90% of the downloads of bdist_wininst were
> generated by things that I know for a fact to be mirroring clients (almost
> all entirely bandersnatch). The next highest source of downloads was coming
> from setuptools, at 7%. Over 75% of the downloads from setuptools are for
> coverage.py, which tells me that it?s likely being triggered by
> test_requires
> and would be covered by teaching setuptools how to wheel instead.
>
> For bdist_msi, 96% of all downloads come from things we know to be
> mirroring
> clients.
>
> For bdist_dmg, 97% of all downloads come from things we know to be
> mirroring
> clients.
>
> For bdist_egg, 80% of all downloads come from things we know to be
> mirroring
> clients.
>
> For reference:
>
> For sdist, 30% of all downloads come from things we know to be mirroring
> clients.
>
> For bdist_wheel, 6% of all downloads come from things we know to be
> mirroring
> clients.
>
> It?s hard to get per project numbers for these (or at least, it takes a
> more
> complex query than I can manage with my head here). However, I think it?s
> pretty telling that when you start looking at other formats, not only is
> the
> primary consumer tools that just indiscriminately download everything from
> PyPI,
> but almost *all* of the consumers of those files are tools that just
> indiscriminately download everything. Unless there are users of those
> mirrors who
> follow vastly different usage patterns than what we see on PyPI itself,
> the primary
> purpose of bdist_wininst, bdist_msi, bdist_dmg, etc on PyPI is to consume
> disk space
> and bandwidth via the mirroring infrastructure.
>
> I?d also like to note, that the numbers above are conservative on what they
> consider to be a ?mirroring client?. For instance, devpi used to use the
> default
> requests user-agent, and we see downloads via the requests user agent, but
> did not
> count them as mirroring clients because it could be some other script
> doing the
> downloading.
>

I should also mention I have never come across anyone at Microsoft use the
bdist_msi or bdist_winst installers (I've added Steve to this email in case
my experience is wrong). Everyone I have encountered either uses conda or
pip+wheels (hence why I keep poking the sdist and build API ideas as I want
to give Christophe Gohlke something else to do with his time than provide
wheels on Windows).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160823/db4c039b/attachment.html>

From mal at egenix.com  Tue Aug 23 15:03:21 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 23 Aug 2016 21:03:21 +0200
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
Message-ID: <57BC9DF9.50408@egenix.com>

On 23.08.2016 18:46, Donald Stufft wrote:
> Since it seemed like there was enough here for a proper PEP I went ahead and
> write one up, which is now PEP 527. The tl;dr of it is that:
> 
> * Everything but sdist, bdist_wheel, and bdist_egg get deprecated.

-1 on removing bdist_wininst and bdist_msi. If PyPI is supposed
to retain the status of the main website to go search for Python
package downloads, it needs to be able to provide ways of hosting
all distribution types which are supported by distutils, including
ones which target platform configuration management system such as
the Windows one.

The number of downloads is really irrelevant for this kind of
argument. Since the PEP proposes to keep the existing uploads
around, I also don't follow the argument of reduced maintenance.
PyPI will still have to host and support downloading those file
types.

To me, all this sounds a lot like eventually turning PyPI into a
pip package index, which no longer serves the original intent of
a Python package index. I think that's taking a wrong turn in the
development of such an index.

IMO, we should aim to reunite separate indexes such as the
one used for conda or the win32 index maintained by
Christoph Golke back into PyPI, not create even more
separation by removing platform specific formats.

> * The only allowed extension for sdist is ``.tar.gz``.

Strong -1 on this part. .tar.gz may be a good choice for Unix,
but it definitely isn't for Windows. Even for Unix, .zip files
have the advantage of not messing up file ownerships and
permissions.

> * Phased in Deprecation.
> 
> I've included the text below, or you can see it online at
> https://www.python.org/dev/peps/pep-0527/ once the PEP website is updated.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Aug 23 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From donald at stufft.io  Tue Aug 23 15:57:39 2016
From: donald at stufft.io (Donald Stufft)
Date: Tue, 23 Aug 2016 15:57:39 -0400
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <57BC9DF9.50408@egenix.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <57BC9DF9.50408@egenix.com>
Message-ID: <446CCA2A-6325-4161-96F8-9FAE2EAFE677@stufft.io>


> On Aug 23, 2016, at 3:03 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> 
> On 23.08.2016 18:46, Donald Stufft wrote:
>> Since it seemed like there was enough here for a proper PEP I went ahead and
>> write one up, which is now PEP 527. The tl;dr of it is that:
>> 
>> * Everything but sdist, bdist_wheel, and bdist_egg get deprecated.
> 
> -1 on removing bdist_wininst and bdist_msi. If PyPI is supposed
> to retain the status of the main website to go search for Python
> package downloads, it needs to be able to provide ways of hosting
> all distribution types which are supported by distutils, including
> ones which target platform configuration management system such as
> the Windows one.

I could not disagree more that PyPI needs to support hosting all
distribution types that distutils supports. For one, we?re quickly
getting to a place where distutils is no longer special, so what
distutils does is just an implementation detail of one particular
tool. But more importantly, these formats are not all equally useful,
to take a look at the most obvious one, we have bdist_dumb which
creates a tarball with zero metadata and hardcoded paths like:

./Users/dstufft/.pyenv/versions/3.5.2/lib/python3.5/site-packages/cryptography/__about__.py
./Users/dstufft/.pyenv/versions/3.5.2/lib/python3.5/site-packages/cryptography/__pycache__/__about__.cpython-35.pyc

If I uploaded this file, it is useful to basically nobody except
people who happened to have the username ?dstufft?, on macOS and
are using pyenv and they?ve installed CPython 3.5.2. However, not
only is it useless it?s actively harmful, because a bdist_dumb has
a filename that is also a valid sdist filename if you?re using the
legacy version parser (which we have to do for backwards
compatibility). This ends up with installers needing to support weird
hacks to actually interact with PyPI instead of being able to handle
it in a much smoother and simpler way. For example, we have gross
hacks like:

    
    if "macosx10" in link.path and ext == '.zip':
        self._log_skipped_link(link, 'macosx10 one')
        return

To try and work around cases where we found popular packages shipping
a bdist_dumb that we couldn?t differentiate from a sdist.

Now, maybe you agree with me, bdist_dumb is well, dumb and we could
probably do without it, but if you do then you also agree with me
that the premise that ?PyPI must support everything distutils does?
is flawed and we can then discuss which formats make sense to support
and which don?t.

> 
> The number of downloads is really irrelevant for this kind of
> argument. Since the PEP proposes to keep the existing uploads
> around, I also don't follow the argument of reduced maintenance.
> PyPI will still have to host and support downloading those file
> types.

Continuing to support a file that already exists is trivial, as far
as PyPI is concerned at that point it?s nothing more than a binary
blob that is accessible at a particular URL. However, PyPI does need
to do work when a file is uploaded to PyPI. For instance, it needs
to verify that the file being uploaded is valid, it needs to ensure
that it?s for the project it claims to be for, etc. To do this, PyPI
has to know things about the file format itself, and what it can
expect from it. One bug that has cropped up from time to time again
is people accidentally uploading a package that inside it contains
version say ?1.0?, but when they registered it with PyPI they told
PyPI it was version ?1.0a1? or something like that, which causes a lot
of the tooling to do subtly weird and broken things. PyPI should be
double checking the internal metadata of these files, but it can?t
do that unless it can expect that metadata to exist in those files
and it has to implement it for each file type (and then, that has to
be maintained).

The number of downloads *is* relevant though, because it allows us
to gauge how many people are utilizing these files to see what the
breakage looks like to try and make a decision about where my time
is spent. If very few people are utilizing something, then it?s very
likely not worth my time to try and continue to support it (and I
don?t see anyone else leaping forward to commit to maintaining support
for these things in PyPI).

> 
> To me, all this sounds a lot like eventually turning PyPI into a
> pip package index, which no longer serves the original intent of
> a Python package index. I think that's taking a wrong turn in the
> development of such an index.

Considering PyPI didn?t originally allow uploading files at all, I
don?t see how disallowing uploading some files is somehow breaking
the original intent. That being said, PyPI has *two* sort of related
functionalities, one is to allow people to discover projects, which
has nothing to do with uploading files (and indeed, originally this
was all PyPI did, there was no concept of file downloads) and for that
nothing is changing, the other side of that is of a repository for
pip/setuptools/etc and for that, you need file uploads and downloads.
Nobody is stopping anyone from linking to any sort of file they want
to in their project description or in their project URLs.

> 
> IMO, we should aim to reunite separate indexes such as the
> one used for conda or the win32 index maintained by
> Christoph Golke back into PyPI, not create even more
> separation by removing platform specific formats.

I have zero interest in turning PyPI into a conda, rpm, deb, or any
other system level package manager index and it would only be over
my very strenuous objection that such a thing ever got added. It is
not appropriate for PyPI at all.

As far as Christoph?s work, there?s nothing that he?s doing that
couldn?t exist on PyPI today *except* that he is not the author of
those packages and thus is providing unofficial, third party binaries,
which again is not appropriate for on PyPI itself.

> 
>> * The only allowed extension for sdist is ``.tar.gz``.
> 
> Strong -1 on this part. .tar.gz may be a good choice for Unix,
> but it definitely isn't for Windows. Even for Unix, .zip files
> have the advantage of not messing up file ownerships and
> permissions.

Both .tar.gz and .zip have advantages and disadvantages, it?s trivial
to sit there and go back and forth about one or the other. However,
having > 1 extensions makes PyPI and pip?s job harder so we should
pick just one and standardize on that. I think it should be .tar.gz
because anything else is a larger change for no real benefit.

?
Donald Stufft




From glyph at twistedmatrix.com  Tue Aug 23 16:08:33 2016
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Tue, 23 Aug 2016 13:08:33 -0700
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <57BC9DF9.50408@egenix.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <57BC9DF9.50408@egenix.com>
Message-ID: <72E135DE-80B3-4C16-99EC-B30B43D77EF2@twistedmatrix.com>


> On Aug 23, 2016, at 12:03 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> 
> On 23.08.2016 18:46, Donald Stufft wrote:
>> Since it seemed like there was enough here for a proper PEP I went ahead and
>> write one up, which is now PEP 527. The tl;dr of it is that:
>> 
>> * Everything but sdist, bdist_wheel, and bdist_egg get deprecated.
> 
> -1 on removing bdist_wininst and bdist_msi. If PyPI is supposed
> to retain the status of the main website to go search for Python
> package downloads, it needs to be able to provide ways of hosting
> all distribution types which are supported by distutils, including
> ones which target platform configuration management system such as
> the Windows one.

I started off at maybe -0 myself for removing format support from PyPI, but reading this rationale for preserving these misfeatures has made me a strong +1!

> The number of downloads is really irrelevant for this kind of
> argument.

It's totally relevant.  The packaging community has limited resources, and we should use those resources to serve the users that actually exist, not pretend people.  The data available from the current system is important to reality-check our assumptions about which sorts of people are in fact real.

That's not to say that "downloads" is the perfect metric, but it's what we've got to work with, and so we have to pay attention to it unless something better is proposed.

> Since the PEP proposes to keep the existing uploads
> around, I also don't follow the argument of reduced maintenance.
> PyPI will still have to host and support downloading those file
> types.

When the person responsible for the vast majority of the maintenance burden says "this will increase / decrease the maintenance burden" then I tend to believe them.  Perhaps this section should be better-motivated in the PEP but I would be _extremely_ surprised if Donald were wrong about this particular point.

> To me, all this sounds a lot like eventually turning PyPI into a
> pip package index, which no longer serves the original intent of
> a Python package index. I think that's taking a wrong turn in the
> development of such an index.

A "pip package index" - that would be fantastic!  Right now, the most confusing thing about the Python ecosystem is the vast diversity of general-purpose installation tools; manual invocation of setup.py, easy_install, pip install, manual Windows installers... if PyPI can centralize all of this stuff around a single installer and get some good, hard and fast de-facto standards around how it's done, that would be much better for user experience.  Frankly it would be better for future installers as well, since it would be possible to calibrate for "just works" much better than we can currently.

But of course, it raises the question: why do you think it is a bad thing?

In particular, bdist_wininst and bdist_msi (which Twisted supported for a long time, and still builds, so it's not like I don't understand their benefits and history!) are incompatible with virtualenvs, and make development under Windows harder, especially as compared to binary eggs.  The presence of these builds confuses users and creates more problems than it solves in every interaction I've had with onboarding people onto Python projects in the last couple of years.

> IMO, we should aim to reunite separate indexes such as the
> one used for conda or the win32 index maintained by
> Christoph Golke back into PyPI, not create even more
> separation by removing platform specific formats.

Absolutely not.  This would be a disaster.  Conda is a general-purpose cross-language package distribution environment.  It hosts packages for C dependencies.  If we want to host packages for Conda, we should also be hosting packages for competing projects at that scope of the ecosystem, which means at the very least adding Homebrew, MacPorts, and maybe Chocolatey and NuGet support too.

In other words, Conda made several different choices about its architecture specifically because it wants to serve a slightly distinct audience from the broader PyPI, and that's fine!  We should not feel pressure to standardize and force everyone into a one-size-fits-all model.  Conda seems to be doing fine and getting plenty of adoption on its own, and not hurting PyPI's success at all.

There are several Steam games and Mac apps written in Python too, but I would hope that it's obvious why we should not be running a competitor to the Steam Store, Mac App Store, or Ubuntu Software Center.  If people feel that PyPI ought to be a general-purpose CDN for anything vaguely Python-adjacent, we might need a more general anti-goals meta-PEP that specifically rules out consideration of this sort of scope creep for PyPI's design in the future.

>> * The only allowed extension for sdist is ``.tar.gz``.
> 
> Strong -1 on this part. .tar.gz may be a good choice for Unix,
> but it definitely isn't for Windows. Even for Unix, .zip files
> have the advantage of not messing up file ownerships and
> permissions.

This is the one part I'm still not totally sure about though.  .zip has the nice feature of allowing random access (which is why we have 'zipimport' but not 'tarimport', despite 'tar' being the more popular archive format with the general audience).  In this case I feel like there's enough usage of both to be worth hanging on to, even if it is a bit more work.

-g

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160823/8b744f50/attachment.html>

From graffatcolmingov at gmail.com  Tue Aug 23 16:13:05 2016
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Tue, 23 Aug 2016 15:13:05 -0500
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <57BC9DF9.50408@egenix.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <57BC9DF9.50408@egenix.com>
Message-ID: <CAN-Kwu3rKtVRXgqBtkvhsKpB-Eie04LMP=wL-49bk+JrsvaoxA@mail.gmail.com>

On Tue, Aug 23, 2016 at 2:03 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> On 23.08.2016 18:46, Donald Stufft wrote:
>> Since it seemed like there was enough here for a proper PEP I went ahead and
>> write one up, which is now PEP 527. The tl;dr of it is that:
>>
>> * Everything but sdist, bdist_wheel, and bdist_egg get deprecated.
>
> -1 on removing bdist_wininst and bdist_msi. If PyPI is supposed
> to retain the status of the main website to go search for Python
> package downloads, it needs to be able to provide ways of hosting
> all distribution types which are supported by distutils, including
> ones which target platform configuration management system such as
> the Windows one.
>
> The number of downloads is really irrelevant for this kind of
> argument. Since the PEP proposes to keep the existing uploads
> around, I also don't follow the argument of reduced maintenance.
> PyPI will still have to host and support downloading those file
> types.
>
> To me, all this sounds a lot like eventually turning PyPI into a
> pip package index, which no longer serves the original intent of
> a Python package index. I think that's taking a wrong turn in the
> development of such an index.
>
> IMO, we should aim to reunite separate indexes such as the
> one used for conda or the win32 index maintained by
> Christoph Golke back into PyPI, not create even more
> separation by removing platform specific formats.

I disagree. As it is, tools like Twine only introspect the more common
file formats to upload them to PyPI and has not had a single user
complain about it. Given that Twine is advertised as the preferred
method to upload to PyPI, you might expect that this would have been
requested already. Alas it has not. No one using modern tooling for
package management is uploading these file types.

Even if the statistic of downloads (which actually is valid) doesn't
sway you, maybe this will. Alternatively, maybe you will help maintain
support for these file types in Warehouse?

From glyph at twistedmatrix.com  Tue Aug 23 16:29:32 2016
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Tue, 23 Aug 2016 13:29:32 -0700
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <72E135DE-80B3-4C16-99EC-B30B43D77EF2@twistedmatrix.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <57BC9DF9.50408@egenix.com>
 <72E135DE-80B3-4C16-99EC-B30B43D77EF2@twistedmatrix.com>
Message-ID: <F1C330BA-7C22-496B-A6E1-FF95AAC9E050@twistedmatrix.com>


> On Aug 23, 2016, at 1:08 PM, Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:
> 
> ... especially as compared to binary eggs ...

Wheels.  especially as compared to binary wheels.

-g
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160823/c06596e7/attachment-0001.html>

From steve.dower at python.org  Tue Aug 23 16:30:14 2016
From: steve.dower at python.org (Steve Dower)
Date: Tue, 23 Aug 2016 13:30:14 -0700
Subject: [Distutils] Deprecating little used file types/extensions
 onPyPI?
In-Reply-To: <CACac1F81-FoNwz2rJjRKsWo_cQsPtJ2e+AjwuXMp-Ekduzc4oQ@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <C6713523-CB23-4F2B-8B2E-0E29AA8EB006@stufft.io>
 <CADiSq7cSX=yzHvVxsJmk831nBR4c=TZKkCv7v652G0=uvfSiCg@mail.gmail.com>
 <100501d1fcb7$35ab0470$a1010d50$@hotmail.com>
 <CADiSq7fSaXSOgDsF5VCYt8v1Sr2kUXR2KWaXnP2dgpO7F_=PjA@mail.gmail.com>
 <CACac1F9m2jXVZyWCpQxc7xGup-Wt2Qouf6J37x=Wh91nHgGPVg@mail.gmail.com>
 <20160823161213.0c174fe8@fsol>
 <CACac1F81-FoNwz2rJjRKsWo_cQsPtJ2e+AjwuXMp-Ekduzc4oQ@mail.gmail.com>
Message-ID: <E1bcILS-00005f-Er@se2-syd.hostedmail.net.au>

We could also extend the py.exe launcher to resolve a matching Python install for a wheel and run pip to install it. Then double-clicking a wheel file would do something useful.

Having a standard UI would be better, though non-trivial.

Top-posted from my Windows Phone

-----Original Message-----
From: "Paul Moore" <p.f.moore at gmail.com>
Sent: ?8/?23/?2016 7:51
To: "Antoine Pitrou" <solipsis at pitrou.net>
Cc: "Distutils" <Distutils-Sig at python.org>
Subject: Re: [Distutils] Deprecating little used file types/extensions onPyPI?

On 23 August 2016 at 15:12, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Tue, 23 Aug 2016 10:36:35 +0100
> Paul Moore <p.f.moore at gmail.com> wrote:
>>
>> I don't see any sign of *anyone* working on a curated distribution for
>> Windows along the lines of Linux distros or Homebrew. (Unless you
>> count cross-platform stacks like conda, which IMO are a different
>> scenario than "system" Python installs).
>
> Under Windows, there isn't much moral difference between a conda install
> (really a Miniconda or Anaconda install) and a python.org Python
> install.  You can even install Anaconda or Miniconda system-wide.
>
> (Miniconda is a minimal install of Python + conda, while Anaconda is a
> pretty large selection of packages in addition - though not the entire
> official repo contents, and not counting community packages)

Agreed - sorry, I was responding more to Nick's implication around
"system" management of Python, conda already has a well-established
management process, but AIUI, it's largely independent of PyPI/pip
(although it interoperates with those). So in the context of Windows
package managers, conda is no more relevant there than it is in
relation to (say) rpm or deb.

Realistically, though, I'd expect people wanting a Python stack on
Windows to gravitate more and more towards distributions like Anaconda
(maybe less so outside its core area of scientific use) with the
remaining users sticking to the python.org builds and pip.

However, I'm not that keen on continuing support for bdist_wininst and
bdist_msi. I'd rather put the effort into making pip and wheels a
compelling solution for such users. IMO, it already is for command
line use (the more so as more hard to build packages start providing
wheels). The place where pip falls down is for people who want to
stick to a GUI. For those people, I believe Idle is likely to be
getting an interface to pip, and I'd like to see other tools like PTVS
and PyCharm get similar capabilities (if they don't have them
already). That may mean exposing an interface to pip's install
mechanisms that's better than "use subprocess to call pip", but we'd
need some concrete use cases to work out the best form of such
interfaces.

On the matter of Chocolatey, it should be pretty trivial to create
recipes for installing Python packages via pip. So I'd see pip+wheel
as remaining the standard interface pretty much indefinitely.

Paul
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG at python.org
https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160823/518b7443/attachment.html>

From steve.dower at python.org  Tue Aug 23 16:40:17 2016
From: steve.dower at python.org (Steve Dower)
Date: Tue, 23 Aug 2016 13:40:17 -0700
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CAP1=2W4rDBRO2w5qbqpwzr49uHZyM6+7icWgNf2adp6r+SdVHg@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <C6713523-CB23-4F2B-8B2E-0E29AA8EB006@stufft.io>
 <CADiSq7cSX=yzHvVxsJmk831nBR4c=TZKkCv7v652G0=uvfSiCg@mail.gmail.com>
 <100501d1fcb7$35ab0470$a1010d50$@hotmail.com>
 <CADiSq7fSaXSOgDsF5VCYt8v1Sr2kUXR2KWaXnP2dgpO7F_=PjA@mail.gmail.com>
 <CACac1F9m2jXVZyWCpQxc7xGup-Wt2Qouf6J37x=Wh91nHgGPVg@mail.gmail.com>
 <CADiSq7eEbAaUzvwr7CGTtjYAym_tS855cJAun0=2ZzTLVWodsg@mail.gmail.com>
 <32DA07E2-7DF2-412D-AFE5-FFFF05BA57C6@stufft.io>
 <CAP1=2W4rDBRO2w5qbqpwzr49uHZyM6+7icWgNf2adp6r+SdVHg@mail.gmail.com>
Message-ID: <f735785e-51fd-9b97-087f-6fc00f3dd903@python.org>

On 23Aug2016 0937, Brett Cannon wrote:
> I should also mention I have never come across anyone at Microsoft use
> the bdist_msi or bdist_winst installers (I've added Steve to this email
> in case my experience is wrong).

In large part this is because I've gotten to them first :)

Personally I don't like bdist_msi or bdist_winst as neither of them 
correctly sets dependencies on the underlying Python install, so 
removing Python does not remove the package. But I can see why people 
may prefer to use them.

I see no harm in not installing them via pip though. They should be 
downloadable "associated files" at minimum, and I don't have any opinion 
about making them available via index data (except that installers 
should be free to ignore them).

Cheers,
Steve

From donald at stufft.io  Tue Aug 23 16:44:35 2016
From: donald at stufft.io (Donald Stufft)
Date: Tue, 23 Aug 2016 16:44:35 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <f735785e-51fd-9b97-087f-6fc00f3dd903@python.org>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid>
 <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <C6713523-CB23-4F2B-8B2E-0E29AA8EB006@stufft.io>
 <CADiSq7cSX=yzHvVxsJmk831nBR4c=TZKkCv7v652G0=uvfSiCg@mail.gmail.com>
 <100501d1fcb7$35ab0470$a1010d50$@hotmail.com>
 <CADiSq7fSaXSOgDsF5VCYt8v1Sr2kUXR2KWaXnP2dgpO7F_=PjA@mail.gmail.com>
 <CACac1F9m2jXVZyWCpQxc7xGup-Wt2Qouf6J37x=Wh91nHgGPVg@mail.gmail.com>
 <CADiSq7eEbAaUzvwr7CGTtjYAym_tS855cJAun0=2ZzTLVWodsg@mail.gmail.com>
 <32DA07E2-7DF2-412D-AFE5-FFFF05BA57C6@stufft.io>
 <CAP1=2W4rDBRO2w5qbqpwzr49uHZyM6+7icWgNf2adp6r+SdVHg@mail.gmail.com>
 <f735785e-51fd-9b97-087f-6fc00f3dd903@python.org>
Message-ID: <4337D756-5EB6-4459-8FE5-38F67E3F89F6@stufft.io>


> On Aug 23, 2016, at 4:40 PM, Steve Dower <steve.dower at python.org> wrote:
> 
> On 23Aug2016 0937, Brett Cannon wrote:
>> I should also mention I have never come across anyone at Microsoft use
>> the bdist_msi or bdist_winst installers (I've added Steve to this email
>> in case my experience is wrong).
> 
> In large part this is because I've gotten to them first :)
> 
> Personally I don't like bdist_msi or bdist_winst as neither of them correctly sets dependencies on the underlying Python install, so removing Python does not remove the package. But I can see why people may prefer to use them.

I mean, we have data that suggests that people do *not* prefer to use them since 90% of the downloads for those files are mirrors just mirroring them and 7% is just setuptools tearing the .exe apart to treat it similarly to an .egg (for wininst, for msi it?s like 97% mirroring and only like 400 of them uploaded ever anyways).

> 
> I see no harm in not installing them via pip though. They should be downloadable "associated files" at minimum, and I don't have any opinion about making them available via index data (except that installers should be free to ignore them).
> 
> Cheers,
> Steve


?
Donald Stufft




From tritium-list at sdamon.com  Tue Aug 23 16:48:22 2016
From: tritium-list at sdamon.com (tritium-list at sdamon.com)
Date: Tue, 23 Aug 2016 16:48:22 -0400
Subject: [Distutils] Deprecating little used file types/extensions on
 PyPI?
In-Reply-To: <CADiSq7eEbAaUzvwr7CGTtjYAym_tS855cJAun0=2ZzTLVWodsg@mail.gmail.com>
References: <B6D94B10-5569-48AF-92FD-31597F659D20@stufft.io>
 <CADiSq7chkdb6aik5df0BD0DGoPuWmQHnZ7SV5X5DGhWN23DQZw@mail.gmail.com>
 <CAPJVwB=d2k0NYyjVwH1aRhbSt53iEC4iTEi8+xz3O8e1mM36rg@mail.gmail.com>
 <CADiSq7fwjOLu8qanGR=FsnOLWUM4hnivDABpN_05QkEhhf1Tyg@mail.gmail.com>
 <06b201d1f9db$07e75450$17b5fcf0$@hotmail.com>
 <CADiSq7fs29YOEYorx_ahJCROnqXW2JxpQuwOZvC5wdBGuOKZ_w@mail.gmail.com>
 <EF0631A7-42BD-40EF-9199-FA030EB6BFC8@stufft.io>
 <CAG8k2+4eQhG1yxFEfmDPyPSm1WdHEJScJFn28cYvuuRevkufxg@mail.gmail.com>
 <CAL13Cg-vvfYYZgifkYs3tbS-6k7-M9TViqWwntnQuxz1iTsmzg@mail.gmail.com>
 <CAG8k2+4e056PXO--i7Ld1U7DbMkfKireLj4-1PY7WKDxOKt+bA@mail.gmail.com>
 <675729B6-4D72-4DAB-AB1F-F4B7B5877F28@stufft.io>
 <CACac1F-3FNPepHC=Zxu8kJw0X4T+fpSqUn_FhRwgFLXrvg7V9A@mail.gmail.com>
 <D1E23A78-FF04-4FF8-A4D2-B7F75CA07901@stufft.io>
 <CACac1F9CGKMWtz3i-otis_X0UV9+euvi0bX9LN3wriXixNwhqg@mail.gmail.com>
 <CAMxX_oPnyE2OJVELrGRF+XVyU6ufcUJ83P2dwfUY4OhtNbQpEw@mail.gmail.com>
 <-1940544014921201238@unknownmsgid> <CAJ3HoZ3u+=JxYmipLo5mbXQdabe1u_
 8yqWXi7imkJxiBCYpMrQ@mail.gmail.com>
 <CADiSq7cEgh5xJjXO=cuG01oraiMDvGH+dN7KA8wmmziAMZEYPQ@mail.gmail.com>
 <CAPJVwBnovg0wDGnnrELws08DxkfTgShetc_eK9DtzWFFCM=jwA@mail.gmail.com>
 <CADiSq7dwNyPSGhH4mOQw6TswkzeOa_GHknXfEow1A+bAodixYw@mail.gmail.com>
 <C6713523-CB23-4F2B-8B2E-0E29AA8EB006@stufft.io>
 <CADiSq7cSX=yzHvVxsJmk831nBR4c=TZKkCv7v652G0=uvfSiCg@mail.gmail.com>
 <100501d1fcb7$35ab0470$a1010d50$@hotmail.com>
 <CADiSq7fSaXSOgDsF5VCYt8v1Sr2kUXR2KWaXnP2dgpO7F_=PjA@mail.gmail.com>
 <CACac1F9m2jXVZyWCpQxc7xGup-Wt2Qouf6J37x=Wh91nHgGPVg@mail.gmail.com>
 <CADiSq7eEbAaUzvwr7CGTtjYAym_tS855cJAun0=2ZzTLVWodsg@mail.gmail.com>
Message-ID: <00de01d1fd7f$b0a93c50$11fbb4f0$@hotmail.com>

As a heavy windows user, I think I can say for the vast majority of windows users on python (that aren?t brand spanking new at python...)

BURN BDIST_WININST!
BURN BDIST_MSI!

> -----Original Message-----
> From: Nick Coghlan [mailto:ncoghlan at gmail.com]
> Sent: Tuesday, August 23, 2016 7:25 AM
> To: Paul Moore <p.f.moore at gmail.com>
> Cc: Alexander Walters <tritium-list at sdamon.com>; distutils-sig <distutils-
> sig at python.org>
> Subject: Re: [Distutils] Deprecating little used file types/extensions on PyPI?
> 
> On 23 August 2016 at 19:36, Paul Moore <p.f.moore at gmail.com> wrote:
> > So I don't think that in the medium term there's going to be much
> > practical change in the state of things on Windows:
> >
> > - Users install Python from the published python.org installers
> > - Users install packages using pip and wheels from PyPI
> > - Plus some exceptions, where people need to use sdists, or
> > independently published wheels, or worse still, wininst/msi installers
> > because that's all available
> >
> > Whether that process is manual, or hidden behind some form of scripted
> > process, won't alter the underlying infrastructure.
> >
> > I don't see any sign of *anyone* working on a curated distribution for
> > Windows along the lines of Linux distros or Homebrew. (Unless you
> > count cross-platform stacks like conda, which IMO are a different
> > scenario than "system" Python installs).
> 
> OK, cool - that gives us all the more reason to retain bdist_wininst
> and bdist_msi hosting support. However, I do think it makes sense for
> us to say up front that we'll reconsider that decision if something
> akin to homebrew gains traction amongst developers running Windows the
> way homebrew has amongst open source users running Mac OS X.
> 
> Cheers,
> Nick.
> 
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From tritium-list at sdamon.com  Tue Aug 23 17:00:35 2016
From: tritium-list at sdamon.com (tritium-list at sdamon.com)
Date: Tue, 23 Aug 2016 17:00:35 -0400
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
Message-ID: <011801d1fd81$64fe4910$2efadb30$@hotmail.com>

I am gung ho on everything in this pep except the sdist format section.

Yes, tar, tar.bz2, tar.xz, tar.Z, .tgz, and tbz can probably go without much contest.  I agree with some of the arguments in the inspiring thread that brought this up, though, that if there is one, and only one, sdist format, it should indeed be .zip, NOT .tar.gz.  If dropping tar.gz is too big of a pill to swallow, then both tar.gz and zip should survive, with an encouragement to use zip in the future.

> 
> File Extensions
> ---------------
> 
> Currently ``sdist`` supports a wide variety of file extensions like `.tar.gz``,
> ``.tar``, ``.tar.bz2``, ``.tar.xz``, ``.zip``, ``.tar.Z``, ``.tgz``, and
> ``.tbz``. However, of those the only extensions which get anything more than
> negligable usage is ``.tar.gz`` with 444,338 sdists currently, ``.zip`` with
> 58,774 sdists currently, and ``.tar.bz2`` with 3,265 sdists currently.
> 
> Having multiple formats accepted requires tooling both within PyPI and
> outside
> of PyPI to handle all of the various extensions that *might* be used (even if
> nobody is currently using them). This doesn't only affect PyPI, but ripples out
> throughout the ecosystem. In addition, the different formats all have
> different
> requirements for what optional C libraries Python was linked against and
> different requirements for what versions of Python they support. In
> addition,
> multiple formats also create a weird situation where there may be two
> ``sdist`` files for a particular project/release with subtly different content.
> 
> It's easy to advocate that anything outside of ``.tar.gz``, ``.zip``, and
> ``.tar.bz2`` should be disallowed. Outside of a tiny handful, nobody has
> actively been uploading these other types of files in the ~15 years of PyPI's
> existence so they've obviously not been particularly useful. In addition, while
> ``.tar.xz`` is theoretically a nicer format than the other ``.tar.*`` formats
> due to the better compression ratio achieved by LZMA, it is only available in
> Python 3.3+ and has an optional dependency on the lzma C library.
> 
> Looking at the three extensions we *do* have in current use, it's also fairly
> easy to conclude that ``.tar.bz2`` can be disallowed as well. It has a fairly
> small number of files ever uploaded with it and it requires an additional
> optional C library to handle the bzip2 compression.
> 
> Finally we get down to ``.tar.gz`` and ``.zip``. Looking at the pure numbers
> for these two, we can see that ``.tar.gz`` is by far the most uploaded format,
> with 444,338 total uploaded compared to ``.zip``'s 58,774 and on POSIX
> operating systems ``.tar.gz`` is also the default produced by all currently
> released versions of Python and setuptools. In addition, these two file types
> both use the same C library (``zlib``) which is also required for
> ``bdist_wheel`` and ``bdist_egg``. The two wrinkles with deciding between
> ``.tar.gz`` and ``.zip`` is that while on POSIX operating systems ``.tar.gz``
> is the default, on Windows ``.zip`` is the default and the ``bdist_wheel``
> format also uses zip.
> 
> This PEP proposes that we drop the use of ``.zip`` extensions for sdists on
> PyPI and standardize around ``.tar.gz``. For both extensions there are going
> to
> be automation designed by end users which are making assumptions about
> what the
> file extension produced by the ``sdist`` command will be. Changing either
> default will break some number of those, so by changing the default of
> ``.zip``
> to ``.tar.gz`` we minimize the amount of breakage by taking the smaller
> number
> of users and making them match the larger number. In addition, it's more
> likely
> to see Windows users upgrade their setuptools and Python releases on a
> faster
> timescale than POSIX users. POSIX users often get their Python and
> setuptools
> from their OS vendor and are discouraged or actively prevented from
> upgrading
> them outside of complete OS upgrades while Windows users *must* install
> Python
> and setuptools on their own, and thus are more able to upgrade those pieces
> without triggering a complete OS upgrade.
> 
> While it is true that switching to ``.zip`` would align ``sdist`` with
> ``bdist_wheel`` in terms of format, this is not a very large benefit because
> both formats are able to be manipulated with the Python standard library
> just
> as easily and both require the same C library (``zlib``). It is also true that
> Windows has support for ``.zip`` files out of the box but requires third party
> software for ``.tar.gz``, however only 0.6% of downloads for sdists on PyPI
> are
> initiated by browsers and we can assume that only a fraction of those 0.6%
> are
> Windows users who want to manually extract the file and do not have a
> means of
> extracting a ``.tar.gz``, particularly since Python itself can be used to
> extract a ``.tar.gz`` via the command line since version 3.4. In addition, the
> use of ``.tar.gz`` will result in smaller sdists which will reduce the amount
> of bandwidth and disk space consumed by ``sdist`` files.
> 
> 


From dholth at gmail.com  Tue Aug 23 17:05:11 2016
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 23 Aug 2016 21:05:11 +0000
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <CAN-Kwu3rKtVRXgqBtkvhsKpB-Eie04LMP=wL-49bk+JrsvaoxA@mail.gmail.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <57BC9DF9.50408@egenix.com>
 <CAN-Kwu3rKtVRXgqBtkvhsKpB-Eie04LMP=wL-49bk+JrsvaoxA@mail.gmail.com>
Message-ID: <CAG8k2+5N5nScb7wLoV8T_G6USeD1UEjzY9pPjZGMAPUNPfa56A@mail.gmail.com>

I would be sad to see .zip go. I would rather the rule be '0 or 1 sdists'.
For my own selfish reasons I am trying to generate sdists with SCons, and
it happens to expect the tar command but can already generate zips cross
platform, so I will need to patch SCons' archiver to comply with the PEP.
Sure, it only affects 2 packages, but they are mine.

ZIP is interesting from the required C extensions angle because you can
compress individual members with LZMA for example, breaking the 'only
requires zlib' constraint; zipimport would not like those either. If the
package already required Python 3.6 would this be a problem? Or the sdist
could require the go compiler, and not work, but at least the packaging
tool would not be the failure point.

Wheel comes with a converter for bdist_wininst. They are conceptually very
similar, both zip format, but bdist_wininst prepends an .exe installer.

On Tue, Aug 23, 2016 at 4:13 PM Ian Cordasco <graffatcolmingov at gmail.com>
wrote:

> On Tue, Aug 23, 2016 at 2:03 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> > On 23.08.2016 18:46, Donald Stufft wrote:
> >> Since it seemed like there was enough here for a proper PEP I went
> ahead and
> >> write one up, which is now PEP 527. The tl;dr of it is that:
> >>
> >> * Everything but sdist, bdist_wheel, and bdist_egg get deprecated.
> >
> > -1 on removing bdist_wininst and bdist_msi. If PyPI is supposed
> > to retain the status of the main website to go search for Python
> > package downloads, it needs to be able to provide ways of hosting
> > all distribution types which are supported by distutils, including
> > ones which target platform configuration management system such as
> > the Windows one.
> >
> > The number of downloads is really irrelevant for this kind of
> > argument. Since the PEP proposes to keep the existing uploads
> > around, I also don't follow the argument of reduced maintenance.
> > PyPI will still have to host and support downloading those file
> > types.
> >
> > To me, all this sounds a lot like eventually turning PyPI into a
> > pip package index, which no longer serves the original intent of
> > a Python package index. I think that's taking a wrong turn in the
> > development of such an index.
> >
> > IMO, we should aim to reunite separate indexes such as the
> > one used for conda or the win32 index maintained by
> > Christoph Golke back into PyPI, not create even more
> > separation by removing platform specific formats.
>
> I disagree. As it is, tools like Twine only introspect the more common
> file formats to upload them to PyPI and has not had a single user
> complain about it. Given that Twine is advertised as the preferred
> method to upload to PyPI, you might expect that this would have been
> requested already. Alas it has not. No one using modern tooling for
> package management is uploading these file types.
>
> Even if the statistic of downloads (which actually is valid) doesn't
> sway you, maybe this will. Alternatively, maybe you will help maintain
> support for these file types in Warehouse?
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160823/dd746897/attachment.html>

From donald at stufft.io  Tue Aug 23 17:17:31 2016
From: donald at stufft.io (Donald Stufft)
Date: Tue, 23 Aug 2016 17:17:31 -0400
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <CAG8k2+5N5nScb7wLoV8T_G6USeD1UEjzY9pPjZGMAPUNPfa56A@mail.gmail.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <57BC9DF9.50408@egenix.com>
 <CAN-Kwu3rKtVRXgqBtkvhsKpB-Eie04LMP=wL-49bk+JrsvaoxA@mail.gmail.com>
 <CAG8k2+5N5nScb7wLoV8T_G6USeD1UEjzY9pPjZGMAPUNPfa56A@mail.gmail.com>
Message-ID: <BE702123-DDA4-408F-88F6-2C4A8F66AF81@stufft.io>


> On Aug 23, 2016, at 5:05 PM, Daniel Holth <dholth at gmail.com> wrote:
> 
> I would be sad to see .zip go. I would rather the rule be '0 or 1 sdists'. For my own selfish reasons I am trying to generate sdists with SCons, and it happens to expect the tar command but can already generate zips cross platform, so I will need to patch SCons' archiver to comply with the PEP. Sure, it only affects 2 packages, but they are mine.

It will be 0 or 1 sdists, having two sdists for a single version is not going to be a thing. However, standardizing around a single extension makes implementing that easier. It also makes it easier for people consuming things from PyPI because there is less variation in what they?re going to get. As time goes on, if we standardize on a single format, all of the older formats will end up just being old cruft that?s laying around, that nobody really installs or uses so  it doesn?t really matter if the tooling handles them or not.

> 
> ZIP is interesting from the required C extensions angle because you can compress individual members with LZMA for example, breaking the 'only requires zlib' constraint; zipimport would not like those either. If the package already required Python 3.6 would this be a problem? Or the sdist could require the go compiler, and not work, but at least the packaging tool would not be the failure point.

I already plan to disallow uploads of wheels and sdists (if we keep .zip sdists) that use any storage methods besides ZIP_STORED and ZIP_DEFLATED. At least .tar.xz tells you on the tin if your current Python can support extracting it, compressing something in a zip file with LZMA is just a russian roulette of ?will this extract or will it not??. As far as whether or not depending on Python 3.6 means it?s no really an issue, that doesn?t solve the problem because the lzma module is an *optional* module, so any particular instance of Python 3.6 may or may not have it available.

> 
> Wheel comes with a converter for bdist_wininst. They are conceptually very similar, both zip format, but bdist_wininst prepends an .exe installer.
> 
> On Tue, Aug 23, 2016 at 4:13 PM Ian Cordasco <graffatcolmingov at gmail.com <mailto:graffatcolmingov at gmail.com>> wrote:
> On Tue, Aug 23, 2016 at 2:03 PM, M.-A. Lemburg <mal at egenix.com <mailto:mal at egenix.com>> wrote:
> > On 23.08.2016 18:46, Donald Stufft wrote:
> >> Since it seemed like there was enough here for a proper PEP I went ahead and
> >> write one up, which is now PEP 527. The tl;dr of it is that:
> >>
> >> * Everything but sdist, bdist_wheel, and bdist_egg get deprecated.
> >
> > -1 on removing bdist_wininst and bdist_msi. If PyPI is supposed
> > to retain the status of the main website to go search for Python
> > package downloads, it needs to be able to provide ways of hosting
> > all distribution types which are supported by distutils, including
> > ones which target platform configuration management system such as
> > the Windows one.
> >
> > The number of downloads is really irrelevant for this kind of
> > argument. Since the PEP proposes to keep the existing uploads
> > around, I also don't follow the argument of reduced maintenance.
> > PyPI will still have to host and support downloading those file
> > types.
> >
> > To me, all this sounds a lot like eventually turning PyPI into a
> > pip package index, which no longer serves the original intent of
> > a Python package index. I think that's taking a wrong turn in the
> > development of such an index.
> >
> > IMO, we should aim to reunite separate indexes such as the
> > one used for conda or the win32 index maintained by
> > Christoph Golke back into PyPI, not create even more
> > separation by removing platform specific formats.
> 
> I disagree. As it is, tools like Twine only introspect the more common
> file formats to upload them to PyPI and has not had a single user
> complain about it. Given that Twine is advertised as the preferred
> method to upload to PyPI, you might expect that this would have been
> requested already. Alas it has not. No one using modern tooling for
> package management is uploading these file types.
> 
> Even if the statistic of downloads (which actually is valid) doesn't
> sway you, maybe this will. Alternatively, maybe you will help maintain
> support for these file types in Warehouse?
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org <mailto:Distutils-SIG at python.org>
> https://mail.python.org/mailman/listinfo/distutils-sig <https://mail.python.org/mailman/listinfo/distutils-sig>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160823/a6ff91aa/attachment-0001.html>

From njs at pobox.com  Tue Aug 23 19:54:37 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 23 Aug 2016 16:54:37 -0700
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <446CCA2A-6325-4161-96F8-9FAE2EAFE677@stufft.io>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <57BC9DF9.50408@egenix.com> <446CCA2A-6325-4161-96F8-9FAE2EAFE677@stufft.io>
Message-ID: <CAPJVwB=oxw23cP4qBccH8Y0845fmpR8-fySHuH2mWeeDSSmJxw@mail.gmail.com>

On Aug 23, 2016 12:57 PM, "Donald Stufft" <donald at stufft.io> wrote:
>
[...]
> However, PyPI does need
> to do work when a file is uploaded to PyPI. For instance, it needs
> to verify that the file being uploaded is valid, it needs to ensure
> that it?s for the project it claims to be for, etc. To do this, PyPI
> has to know things about the file format itself, and what it can
> expect from it. One bug that has cropped up from time to time again
> is people accidentally uploading a package that inside it contains
> version say ?1.0?, but when they registered it with PyPI they told
> PyPI it was version ?1.0a1? or something like that, which causes a lot
> of the tooling to do subtly weird and broken things. PyPI should be
> double checking the internal metadata of these files, but it can?t
> do that unless it can expect that metadata to exist in those files
> and it has to implement it for each file type (and then, that has to
> be maintained).

Am I understanding correctly that PyPI needs to start peeking inside sdists
but hasn't started doing that yet? If that's correct, then I just want to
double check that the cost of implementing this upcoming feature has been
factored into the .zip-vs-.tar.gz discussion, because code for peeking
inside .tar.gz files is presumably harder to write and more expensive to
run than code for peeking inside .zip files. (But maybe only negligibly
harder, I haven't tried writing such code myself, and uploads are
relatively rare compared to downloads.) I guess the worst case would be if
it turns out pypi needs to look at multiple files inside each sdist, where
.tar.gz access becomes quadratic unless you're very clever.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160823/3d8ed182/attachment.html>

From donald at stufft.io  Tue Aug 23 19:59:23 2016
From: donald at stufft.io (Donald Stufft)
Date: Tue, 23 Aug 2016 19:59:23 -0400
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <CAPJVwB=oxw23cP4qBccH8Y0845fmpR8-fySHuH2mWeeDSSmJxw@mail.gmail.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <57BC9DF9.50408@egenix.com> <446CCA2A-6325-4161-96F8-9FAE2EAFE677@stufft.io>
 <CAPJVwB=oxw23cP4qBccH8Y0845fmpR8-fySHuH2mWeeDSSmJxw@mail.gmail.com>
Message-ID: <AF99826E-8A19-4B37-9BFC-B2F28BB3C7F2@stufft.io>


> On Aug 23, 2016, at 7:54 PM, Nathaniel Smith <njs at pobox.com> wrote:
> 
> On Aug 23, 2016 12:57 PM, "Donald Stufft" <donald at stufft.io <mailto:donald at stufft.io>> wrote:
> >
> [...]
> > However, PyPI does need
> > to do work when a file is uploaded to PyPI. For instance, it needs
> > to verify that the file being uploaded is valid, it needs to ensure
> > that it?s for the project it claims to be for, etc. To do this, PyPI
> > has to know things about the file format itself, and what it can
> > expect from it. One bug that has cropped up from time to time again
> > is people accidentally uploading a package that inside it contains
> > version say ?1.0?, but when they registered it with PyPI they told
> > PyPI it was version ?1.0a1? or something like that, which causes a lot
> > of the tooling to do subtly weird and broken things. PyPI should be
> > double checking the internal metadata of these files, but it can?t
> > do that unless it can expect that metadata to exist in those files
> > and it has to implement it for each file type (and then, that has to
> > be maintained).
> 
> Am I understanding correctly that PyPI needs to start peeking inside sdists but hasn't started doing that yet? If that's correct, then I just want to double check that the cost of implementing this upcoming feature has been factored into the .zip-vs-.tar.gz discussion, because code for peeking inside .tar.gz files is presumably harder to write and more expensive to run than code for peeking inside .zip files. (But maybe only negligibly harder, I haven't tried writing such code myself, and uploads are relatively rare compared to downloads.) I guess the worst case would be if it turns out pypi needs to look at multiple files inside each sdist, where .tar.gz access becomes quadratic unless you're very clever.
> 
> -n
> 
Yes, though I?m not real worried about the time it takes, uploading happens something like 700 times a day, so being a touch slower isn?t the worst thing in the world, particularly if it means that our disk space or bandwidth needs are less.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160823/2d86c8ca/attachment.html>

From wes.turner at gmail.com  Tue Aug 23 23:23:49 2016
From: wes.turner at gmail.com (Wes Turner)
Date: Tue, 23 Aug 2016 22:23:49 -0500
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <CAPJVwB=oxw23cP4qBccH8Y0845fmpR8-fySHuH2mWeeDSSmJxw@mail.gmail.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <57BC9DF9.50408@egenix.com> <446CCA2A-6325-4161-96F8-9FAE2EAFE677@stufft.io>
 <CAPJVwB=oxw23cP4qBccH8Y0845fmpR8-fySHuH2mWeeDSSmJxw@mail.gmail.com>
Message-ID: <CACfEFw9kgHtAtW6ZcPk4iCWx61dQhUD_Fe+=wtDy9bjuU+qrzw@mail.gmail.com>

On Tuesday, August 23, 2016, Nathaniel Smith <njs at pobox.com> wrote:

> On Aug 23, 2016 12:57 PM, "Donald Stufft" <donald at stufft.io
> <javascript:_e(%7B%7D,'cvml','donald at stufft.io');>> wrote:
> >
> [...]
> > However, PyPI does need
> > to do work when a file is uploaded to PyPI. For instance, it needs
> > to verify that the file being uploaded is valid, it needs to ensure
> > that it?s for the project it claims to be for, etc. To do this, PyPI
> > has to know things about the file format itself, and what it can
> > expect from it. One bug that has cropped up from time to time again
> > is people accidentally uploading a package that inside it contains
> > version say ?1.0?, but when they registered it with PyPI they told
> > PyPI it was version ?1.0a1? or something like that, which causes a lot
> > of the tooling to do subtly weird and broken things. PyPI should be
> > double checking the internal metadata of these files, but it can?t
> > do that unless it can expect that metadata to exist in those files
> > and it has to implement it for each file type (and then, that has to
> > be maintained).
>
> Am I understanding correctly that PyPI needs to start peeking inside
> sdists but hasn't started doing that yet? If that's correct, then I just
> want to double check that the cost of implementing this upcoming feature
> has been factored into the .zip-vs-.tar.gz discussion, because code for
> peeking inside .tar.gz files is presumably harder to write and more
> expensive to run than code for peeking inside .zip files. (But maybe only
> negligibly harder, I haven't tried writing such code myself, and uploads
> are relatively rare compared to downloads.) I guess the worst case would be
> if it turns out pypi needs to look at multiple files inside each sdist,
> where .tar.gz access becomes quadratic unless you're very clever.
>
-1 on .tar.gz as the one true sdist.
  - "You require 7zip to open >< 3.4 pkgs":
     http://www.7-zip.org/download.html

- This should probably be done with containers and tasks? (Inspecting and
parsing the compressed metadata)

- .tar.gz optionally persists +x, setuid, setgid permission bits (!?)

- zipimport and wheel already depend on ZIP

Also, regarding legacy support and future maintenance costs,
- who added this functionality? Can they be contacted to modernize it?
- it's not like we're going to be able to eliminate the code paths
entirely, so "yeah that's not maintained anymore" is worse better than
"someone should make those slow tests faster"

+1 on pushing for just .tar.gz as an extension






> -n
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160823/e8161024/attachment-0001.html>

From ncoghlan at gmail.com  Wed Aug 24 03:06:32 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 24 Aug 2016 17:06:32 +1000
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
Message-ID: <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>

On 24 August 2016 at 02:46, Donald Stufft <donald at stufft.io> wrote:
> Abstract
> ========
>
> This PEP recommends deprecating, and ultimately removing, support for uploading
> certain unused or under used file types and extensions to PyPI. In particular
> it recommends disallowing further uploads of any files of the types
> ``bdist_dumb``, ``bdist_rpm``, ``bdist_dmg``, ``bdist_msi``, and
> ``bdist_wininst``, leaving PyPI to only accept new uploads of the ``sdist``,
> ``bdist_wheel``, and ``bdist_egg`` file types.

+1 on this part. To address MAL's concerns, you may want to state
explicitly that this proposal is based around a notion of Python
package publication as a multi-level distribution network where:

* PyPI itself only supports publication in venv-compatible
Python-specific formats (sdist, wheel, egg)
* PyPA (the development community) actively encourages
community-driven and commercial conversions to downstream formats for
particular target environments:

- the conda cross-platform data analysis ecosystem (conda-forge)
- the deb based Linux ecosystem (Debian, Ubuntu, etc)
- the RPM based Linux ecosystem (Fedora, openSuSE, Mageia, etc)
- the homebrew, MacPorts and fink ecosystems for Mac OS X
- the Windows Package Management ecosystem (NuGet, Chocalatey, etc)
- 3rd party creation of Windows MSIs and installers (e.g. Christoph
Gohlke's work at http://www.lfd.uci.edu/~gohlke/pythonlibs/ )
- other commercial redistribution formats (ActiveState's PyPM,
Enthought Canopy, etc)
- other open source community redistribution formats (Nix, Gentoo,
Arch, *BSD, etc)

This division of responsibility means that we (PyPA) don't need to be
experts in every platform management system out there, and neither do
folks wanting to publish software through PyPI. However, we also
remain open to enhancement proposals for the standard upstream formats
that allow for easier automation of the conversions to downstream
formats.

> In addition, this PEP proposes removing support for new uploads of sdists using
> the ``.tar``, ``.tar.bz2``, ``.tar.xz``, ``.zip``, ``.tar.Z``, ``.tgz``,
> ``.tbz``, and any other extension besides ``.tar.gz``.

I think in this case, campaigning to remove ".zip" support will prove
to be a false economy, as anything you'd gain from simplifying the
toolchain implementation you'll lose arguing with folks that would
prefer for .zip to be the default instead of .tar.gz.

So while I agree with your argument that settling on a single sdist
archive format would be better, I think the political cost of pushing
for that will be too high (and overshadow the other benefits of this
proposal).

By contrast, allowing both .tar.gz and .zip still achieves the goal of
minimising external binary dependencies (since they both just need
zlib), and we can also still keep the change where setuptools and
distutils default to generating .tar.gz regardless of platform (so we
optimise for size reduction by default, but folks that really prefer
to publish their sdists as zip archives will remain free to do so).

I also see a concrete *practical* benefit to pursuing this option
(mainly prompted by Robert Collins's observations in the other
thread): when we get around to defining sdist 2.0, we're likely to
make that a zip based format for consistency with wheel, so we don't
really want people to be actively dropping zip support from their
toolchains. If we continue to allow both .tar.gz and .zip, we
essentially get that for free, and then at some point in the future,
we can push for convergence on a new ".sdist" format for distributing
annotated source archives (which could even explore ideas like a
tarball-in-zip approach that offers random access to the metadata
files, but still focuses on compression for the source archive
itself).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From sylvain.corlay at gmail.com  Wed Aug 24 07:27:49 2016
From: sylvain.corlay at gmail.com (Sylvain Corlay)
Date: Wed, 24 Aug 2016 13:27:49 +0200
Subject: [Distutils] Proposed new Distutils API for compiler flag
 detection (Issue26689)
In-Reply-To: <CAK=Phk4Dfd1BNaC3OoTO=g8f1xqij2an8N9UuOuC0DLcenzgEA@mail.gmail.com>
References: <A45B9ADE-2FB4-4453-AACE-8B1595C009E3@python.org>
 <CABL7CQi86ORbhQUy11g4ipiEWMKFYtzeeYrb1JkKomsVPa8uKQ@mail.gmail.com>
 <CAK=Phk5YV_Ax34_DRV9b4ch4T7GNMZ2=E4DH=QP0DzeAA8v83A@mail.gmail.com>
 <CAPJVwBnOu6i3f-_XspZtaMR8Va68suHVb9p9HiVQ6Cp8-CFFTQ@mail.gmail.com>
 <CAK=Phk4_XgZu_z_qu_mHFKE96V2c-dJHdixJKtrckBjQ2L88Tw@mail.gmail.com>
 <34793887-E8A4-4694-ADD2-04F45F789651@stufft.io>
 <CAK=Phk4YRKgbH7Qxy2j2smuHpqGEpaNFgFtzEoHtH4qZaz92qw@mail.gmail.com>
 <63D859EB-B20C-4413-ACF4-93EB22C7A7FD@stufft.io>
 <CABL7CQgggpKpO5UG1vXfDFo1eR3ccyNRn7AcVt7OB=YKey+CCA@mail.gmail.com>
 <CAK=Phk6Qe=UWvKH-d2UbxJG9QSnDjHURwfz0rNPZ1GsxoYTr4A@mail.gmail.com>
 <0f0301d1fc46$0d5ec540$281c4fc0$@hotmail.com>
 <CAK=Phk4Dfd1BNaC3OoTO=g8f1xqij2an8N9UuOuC0DLcenzgEA@mail.gmail.com>
Message-ID: <CAK=Phk5CnTF88tt0=65bPbcvMZCL3uA+eBhrDPE3t70DXCKgWQ@mail.gmail.com>

Hi,
Any feedback on this?
Sylvain

On Mon, Aug 22, 2016 at 10:25 AM, Sylvain Corlay <sylvain.corlay at gmail.com>
wrote:

> I think that Thomas' proposal makes sense.
>
> I would be ok to also add it to setuptools so that it can be used sooner
> by projects that don't require python 3.6.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160824/f2a64748/attachment.html>

From reinout at vanrees.org  Wed Aug 24 10:31:52 2016
From: reinout at vanrees.org (Reinout van Rees)
Date: Wed, 24 Aug 2016 16:31:52 +0200
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
Message-ID: <npkb4n$qrv$1@blaine.gmane.org>

Op 24-08-16 om 09:06 schreef Nick Coghlan:
> I think in this case, campaigning to remove ".zip" support will prove
> to be a false economy, as anything you'd gain from simplifying the
> toolchain implementation you'll lose arguing with folks that would
> prefer for .zip to be the default instead of .tar.gz.

While it might sound very strange: at one time (2008) it helped us that 
**both** .zip and .tar.gz were allowed

There was a bug in python 2.4's build-in tarfile module that failed to 
extract files with a path that was exactly 100 characters long. 99 fine, 
101 fine, 100 would fail to be extracted.

See http://maurits.vanrees.org/weblog/archive/2008/09/scrambled-eggs

In order to support broken 2.4 pythons, zest.releaser explicitly created 
.zips for a long time :-) If I look at the changelog, the issue still 
cropped up from time to time in 2012, four years later.

We stopped forcing .zip in version 3.56 (2015-03-18).


So...

a) It came in handy to have both .zip and .tar.gz once :-)

b) zest.releaser might have **massively inflated the number** of zip 
files in the statistics. zest.releaser is used a lot in the zope/plone 
corner of pypi and that's a very active corner of pypi. Donald, perhaps 
you could look at the top 10 of zip packages to see if it has a plone smell?



Reinout

-- 
Reinout van Rees                          http://reinout.vanrees.org/
reinout at vanrees.org                   http://www.nelen-schuurmans.nl/
"Learning history by destroying artifacts is a time-honored atrocity"


From alex.gronholm at nextday.fi  Wed Aug 24 10:40:40 2016
From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=)
Date: Wed, 24 Aug 2016 17:40:40 +0300
Subject: [Distutils] What role to eggs still play?
In-Reply-To: <CAG8k2+4W9Y3qhpBBhDXcO6wPDkjJUDDJdcQ=YZGEidOXXcbxdw@mail.gmail.com>
References: <CALGmxE+=4-gqgbAR1uGODKR1C-zjrSCEyBXOqp-2x_ypAeb8aw@mail.gmail.com>
 <CAG8k2+4W9Y3qhpBBhDXcO6wPDkjJUDDJdcQ=YZGEidOXXcbxdw@mail.gmail.com>
Message-ID: <afe1862e-6427-42cc-b2a1-f73ab5742671@nextday.fi>

19.08.2016, 20:25, Daniel Holth kirjoitti:
> Eggs are the only way to add a zipped distribution to PYTHONPATH and 
> have setuptools find the metadata (the Python code can be found with 
> or without the metadata; setuptools does not discover *.dist-info 
> inside zip). Eggs are used by buildout, especially in the unzipped 
> into a directory form. And they could still be used for their 
> originally designed use as a plugin format.
What is not clear to me is who needs this. Is this not what virtualenvs 
are for? Or could wheels not be used for the same purpose? What am I 
missing?
>
> One of the smaller problems with eggs is that everything had the same 
> name. *.egg-info or EGG-INFO is the predecessor of the *.dist-info 
> format designed in PEP-376. You get *.egg-info whenever you install 
> something with setuptools without going through bdist_wheel.
>
> On Fri, Aug 19, 2016 at 12:54 PM Chris Barker <chris.barker at noaa.gov 
> <mailto:chris.barker at noaa.gov>> wrote:
>
>     Hi all,
>
>     starting a new thread, but this is related to the setuptols-_lite
>     discussion, and the legacy formats discussion. In another thread
>     Donald had a footnote:
>
>
>         [1] We can tackle egg at a later point, when setuptools either
>         has support for Wheels
>             or is less needed.
>
>
>     So I'm wondering -- does anything else (other than setuptools)
>     depend on eggs in any way? I know why I pip install stuff, I
>     (always?) get egg-ish things installed:
>
>     .egg-info
>
>     directories and all that. Honestly, I'm confused -- is that making
>     an actual egg? or is that name simply a legacy name for package
>     meta data?
>
>     In any case, does pip, or anything else, require it?
>
>     For my part, I find it annoying, name aside -- it seems that all
>     the package meta-data should be there in the package source
>     already (pypacakge.toml?)
>
>     -Chris
>
>
>
>
>
>
>
>     -- 
>
>     Christopher Barker, Ph.D.
>     Oceanographer
>
>     Emergency Response Division
>     NOAA/NOS/OR&R            (206) 526-6959   voice
>     7600 Sand Point Way NE   (206) 526-6329   fax
>     Seattle, WA  98115       (206) 526-6317   main reception
>
>     Chris.Barker at noaa.gov <mailto:Chris.Barker at noaa.gov>
>     _______________________________________________
>     Distutils-SIG maillist  - Distutils-SIG at python.org
>     <mailto:Distutils-SIG at python.org>
>     https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160824/14406118/attachment-0001.html>

From alex.gronholm at nextday.fi  Wed Aug 24 10:41:24 2016
From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=)
Date: Wed, 24 Aug 2016 17:41:24 +0300
Subject: [Distutils] What role to eggs still play?
In-Reply-To: <3388930217277504347@unknownmsgid>
References: <CALGmxE+=4-gqgbAR1uGODKR1C-zjrSCEyBXOqp-2x_ypAeb8aw@mail.gmail.com>
 <CAG8k2+4W9Y3qhpBBhDXcO6wPDkjJUDDJdcQ=YZGEidOXXcbxdw@mail.gmail.com>
 <CAG8k2+5UYQ=dJBuVr-SjFU_=+h7GR60pp7bBwZNnrxx-QV0D9g@mail.gmail.com>
 <3388930217277504347@unknownmsgid>
Message-ID: <5b8f3ce4-0b59-93d8-47e9-0cf243ffabb0@nextday.fi>

20.08.2016, 01:41, Chris Barker - NOAA Federal kirjoitti:
> Thanks, I think I'm getting it.
>
>> About the toml file... the *-info metadata is a compiled artifact, 
>> according to all the existing Python packages. Most sdists even have 
>> a *.egg-info directory.
>
> If it's a compiled artifact, then shouldn't it NOT be in a source dist?
>
>> It is inconvenient if you want to know the true dependencies without 
>> running setup.py.
>
> Isn't that what the toml file is for?
Isn't the toml file for specifying build dependencies, rather then 
runtime dependencies?
>
>> I think we are stuck with it, and it's not all bad - if there is some 
>> useful metadata that doesn't affect the dependency resolver, and it 
>> is "too static" or too cumbersome to write out by hand, there's a 
>> place to do that.
>
> I'm trying to imagine what that info would be,  but I suppose there 
> could be meta data about a package that is generated at build time -- 
> maybe info about how it was built, for instance.
>
> Hmm -- maybe you could put info in there about non-Python shared libs 
> it's linked to, for instance.
>
>>     Eggs are the only way to add a zipped distribution to PYTHONPATH
>>     and have setuptools find the metadata
>>
>
> Can pip find it in a zipped package? Remember,  I don't care to 
> support setuptools only features anyway :-)
>
>>     . Eggs are used by buildout, especially in the unzipped into a
>>     directory form
>>
>>     And they could still be used for their originally designed use as
>>     a plugin format.
>>
>
> If pkg_resources gets spun off, would it support that?
>
>>      *.egg-info or EGG-INFO is the predecessor of the *.dist-info
>>     format designed in PEP-376. You get *.egg-info whenever you
>>     install something with setuptools without going through bdist_wheel.
>>
>
> So setuptools_lite would write a  dist_info.
>
> I take it pip looks for both?
>
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160824/7c06e43b/attachment.html>

From dholth at gmail.com  Wed Aug 24 11:56:21 2016
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 24 Aug 2016 15:56:21 +0000
Subject: [Distutils] What role to eggs still play?
In-Reply-To: <5b8f3ce4-0b59-93d8-47e9-0cf243ffabb0@nextday.fi>
References: <CALGmxE+=4-gqgbAR1uGODKR1C-zjrSCEyBXOqp-2x_ypAeb8aw@mail.gmail.com>
 <CAG8k2+4W9Y3qhpBBhDXcO6wPDkjJUDDJdcQ=YZGEidOXXcbxdw@mail.gmail.com>
 <CAG8k2+5UYQ=dJBuVr-SjFU_=+h7GR60pp7bBwZNnrxx-QV0D9g@mail.gmail.com>
 <3388930217277504347@unknownmsgid>
 <5b8f3ce4-0b59-93d8-47e9-0cf243ffabb0@nextday.fi>
Message-ID: <CAG8k2+5hOOtiKh5hQAjNDpujwUNyazZA0N=JSC0vH4A7g70zKg@mail.gmail.com>

On Wed, Aug 24, 2016 at 10:41 AM Alex Gr?nholm <alex.gronholm at nextday.fi>
wrote:

> 20.08.2016, 01:41, Chris Barker - NOAA Federal kirjoitti:
>
> Thanks, I think I'm getting it.
>
> About the toml file... the *-info metadata is a compiled artifact,
> according to all the existing Python packages. Most sdists even have a
> *.egg-info directory.
>
>
> If it's a compiled artifact, then shouldn't it NOT be in a source dist?
>
> It is inconvenient if you want to know the true dependencies without
> running setup.py.
>
>
> Isn't that what the toml file is for?
>
> Isn't the toml file for specifying build dependencies, rather then runtime
> dependencies?
>

It is only build dependencies now. Maybe he also anticipates the migration
of static setup() arguments into toml e.g.
https://bitbucket.org/dholth/pysdl2-cffi/src/tip/pyproject.toml . Not that
this would necessarily be in a standard format...

19.08.2016, 20:25, Daniel Holth kirjoitti:

Eggs are the only way to add a zipped distribution to PYTHONPATH and have
setuptools find the metadata (the Python code can be found with or without
the metadata; setuptools does not discover *.dist-info inside zip). Eggs
are used by buildout, especially in the unzipped into a directory form. And
they could still be used for their originally designed use as a plugin
format.

Alex kirjoitti: "What is not clear to me is who needs this. Is this not
what virtualenvs are for? Or could wheels not be used for the same purpose?
What am I missing?"

What you are missing is that while they could in theory be used for the
same purpose, in practice there is a large group of Plone & buildout users
who are not using wheels for that. (And the wheels would have to be
unzipped into their own individual directories, but buildout almost always
unzips eggs lately too.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160824/b8e31e0e/attachment.html>

From pombredanne at nexb.com  Wed Aug 24 12:22:18 2016
From: pombredanne at nexb.com (Philippe Ombredanne)
Date: Wed, 24 Aug 2016 18:22:18 +0200
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
Message-ID: <CAOFm3uHME26nURs_Sc3FbQU4oJ=0VXJhSZJnQQsernAh+gU7_Q@mail.gmail.com>

On Tue, Aug 23, 2016 at 10:08 PM, Glyph Lefkowitz
<glyph at twistedmatrix.com> wrote:
> In particular, bdist_wininst and bdist_msi (which Twisted supported for a
> long time, and still builds, so it's not like I don't understand their
> benefits and history!) are incompatible with virtualenvs, and make
> development under Windows harder, especially as compared to binary eggs.
> The presence of these builds confuses users and creates more problems than
> it solves in every interaction I've had with onboarding people onto Python
> projects in the last couple of years.

I can only agree. A strong +1 to deprecate bdist_wininst and bdist_msi ....
from Pypi ... and from Python everywhere if it is be possible.

I build and distribute a somewhat engaged open source app (scancode-toolkit)
and earlier version used eggs and now this is all wheels in virtualenvs.
This runs on linux/mac/win with vendored wheels.
I cannot fathom _any_ benefit from using  Windows installers in a modern
Python  environment like this and other where pip and virtualenv rule.

Like Glyph, I think these these installers only create confusion at
first and are the
source of troubles down the road even if they may seem like a friendly solution
for win noobs on the surface. Every time I met someone using them, this was a
source of mess in their Python setup.
They made sense in 1999, but are likely best forgotten today.
And if this was dropped elsewhere beside Pypi, there could quite a rafter of
arcane code to drop from distutils and setuptools.... a good thing IMHO.
-- 
Cordially
Philippe Ombredanne

From annaraven at gmail.com  Wed Aug 24 12:24:17 2016
From: annaraven at gmail.com (Anna Ravenscroft)
Date: Wed, 24 Aug 2016 09:24:17 -0700
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
Message-ID: <CA+nrBM9bCXG8_yiG6PbGWYo00UQqZHMz9kO+HeweoG9yg4kwxg@mail.gmail.com>

On Wed, Aug 24, 2016 at 12:06 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 24 August 2016 at 02:46, Donald Stufft <donald at stufft.io> wrote:
> > Abstract
> > ========
> >
> > This PEP recommends deprecating, and ultimately removing, support for
> uploading
> > certain unused or under used file types and extensions to PyPI. In
> particular
> > it recommends disallowing further uploads of any files of the types
> > ``bdist_dumb``, ``bdist_rpm``, ``bdist_dmg``, ``bdist_msi``, and
> > ``bdist_wininst``, leaving PyPI to only accept new uploads of the
> ``sdist``,
> > ``bdist_wheel``, and ``bdist_egg`` file types.
>
> +1 on this part. To address MAL's concerns, you may want to state
> explicitly that this proposal is based around a notion of Python
> package publication as a multi-level distribution network where:
>
> * PyPI itself only supports publication in venv-compatible
> Python-specific formats (sdist, wheel, egg)
> * PyPA (the development community) actively encourages
> community-driven and commercial conversions to downstream formats for
> particular target environments:
>
> - the conda cross-platform data analysis ecosystem (conda-forge)
> - the deb based Linux ecosystem (Debian, Ubuntu, etc)
> - the RPM based Linux ecosystem (Fedora, openSuSE, Mageia, etc)
> - the homebrew, MacPorts and fink ecosystems for Mac OS X
> - the Windows Package Management ecosystem (NuGet, Chocalatey, etc)
> - 3rd party creation of Windows MSIs and installers (e.g. Christoph
> Gohlke's work at http://www.lfd.uci.edu/~gohlke/pythonlibs/ )
> - other commercial redistribution formats (ActiveState's PyPM,
> Enthought Canopy, etc)
> - other open source community redistribution formats (Nix, Gentoo,
> Arch, *BSD, etc)
>

+1


>
> This division of responsibility means that we (PyPA) don't need to be
> experts in every platform management system out there, and neither do
> folks wanting to publish software through PyPI.



It also improves the situation for folks trying to document the situation
(like book authors...) by making things much clearer.


> However, we also
> remain open to enhancement proposals for the standard upstream formats
> that allow for easier automation of the conversions to downstream
> formats.
>
> > In addition, this PEP proposes removing support for new uploads of
> sdists using
> > the ``.tar``, ``.tar.bz2``, ``.tar.xz``, ``.zip``, ``.tar.Z``, ``.tgz``,
> > ``.tbz``, and any other extension besides ``.tar.gz``.
>
> I think in this case, campaigning to remove ".zip" support will prove
> to be a false economy, as anything you'd gain from simplifying the
> toolchain implementation you'll lose arguing with folks that would
> prefer for .zip to be the default instead of .tar.gz.
>

+1

-- 
cordially,
Anna
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160824/973cb67f/attachment.html>

From leorochael at gmail.com  Wed Aug 24 13:28:35 2016
From: leorochael at gmail.com (Leonardo Rochael Almeida)
Date: Wed, 24 Aug 2016 14:28:35 -0300
Subject: [Distutils] What role to eggs still play?
In-Reply-To: <CAG8k2+5hOOtiKh5hQAjNDpujwUNyazZA0N=JSC0vH4A7g70zKg@mail.gmail.com>
References: <CALGmxE+=4-gqgbAR1uGODKR1C-zjrSCEyBXOqp-2x_ypAeb8aw@mail.gmail.com>
 <CAG8k2+4W9Y3qhpBBhDXcO6wPDkjJUDDJdcQ=YZGEidOXXcbxdw@mail.gmail.com>
 <CAG8k2+5UYQ=dJBuVr-SjFU_=+h7GR60pp7bBwZNnrxx-QV0D9g@mail.gmail.com>
 <3388930217277504347@unknownmsgid>
 <5b8f3ce4-0b59-93d8-47e9-0cf243ffabb0@nextday.fi>
 <CAG8k2+5hOOtiKh5hQAjNDpujwUNyazZA0N=JSC0vH4A7g70zKg@mail.gmail.com>
Message-ID: <CAMxX_oMNm9vqMDqvwzRYZoAkE76KH0rmVLbj0SxcgB4_H7+hRw@mail.gmail.com>

On 24 August 2016 at 12:56, Daniel Holth <dholth at gmail.com> wrote:

> On Wed, Aug 24, 2016 at 10:41 AM Alex Gr?nholm <alex.gronholm at nextday.fi>
> wrote:
>
>> [...]
>>
>
> 19.08.2016, 20:25, Daniel Holth kirjoitti:
>
> Eggs are the only way to add a zipped distribution to PYTHONPATH and have
> setuptools find the metadata (the Python code can be found with or without
> the metadata; setuptools does not discover *.dist-info inside zip). Eggs
> are used by buildout, especially in the unzipped into a directory form. And
> they could still be used for their originally designed use as a plugin
> format.
>
> Alex kirjoitti: "What is not clear to me is who needs this. Is this not
> what virtualenvs are for? Or could wheels not be used for the same purpose?
> What am I missing?"
>

There is some mixing of concerns here, but to clarify: wheels in general
CANNOT be blindly dropped on PYTHONPATH and be expected to work.

Only wheels that DO NOT contain an entry like `wheel_name-1.0.data/purelib`
or `wheel_name-1.0.data/platlib` can be dropped on PYTHONPATH to make their
code available to a python process.

But that only works for python code. `setuptools`/`pkg_resources` will
currently be unable to read entry points (in particular, console_scripts)
from such wheels dropped in the PYTHONPATH.

So, they cannot be used in general for the purpose of dropping them on the
PYTHONPATH. But the buildout communities (incl. Plone) do not exactly
require wheels dropped on the PYTHONPATH. See below?

What you are missing is that while they could in theory be used for the
> same purpose, in practice there is a large group of Plone & buildout users
> who are not using wheels for that. (And the wheels would have to be
> unzipped into their own individual directories, but buildout almost always
> unzips eggs lately too.)
>

In particular, buildout users on Windows, who cannot easily compile sdists
that contain C extensions and which are currently uploaded as eggs to PyPI
(see lxml and zope.interface as examples of highly used packages in this
regard).

As for wheels, not only would these wheels have to be unzipped, but their
`purelib`/`platlib` contents, if present, copied to the "root" of where
they were unpacked, and the `wheel_name-1.0.dist-info` directory renamed to
`EGG-INFO`

But the most important thing of all, these wheels would have to participate
in the dependency resolution of other packages along with sdists in the
middle of a buildout run.

The last aspect means that having a canonical piece of code that knows how
to pick up a wheel from PyPI and unpack it correctly somewhere in the
PYTHONPATH is not enough.

(and nobody has offered such canonical a piece of software yet, except the
not really useful "call out to pip as a subprocess").

What is needed is for setuptools to be modified to treat wheels the same
way it currently treats bdist_wininst files for installation: as a binary
lump that it knows how to download from PyPI, unpack and convert into an
egg.

On the other hand, if the above happens, we can block .egg files from
upload to PyPI along with the rest of the formats. Wouldn't it be nice?

Any other option for adding wheels support to buildout (calling out to pip,
using distlib,... ) will involve either a lot of code rewrite [1] or the
need to rewrite many currently functioning buildout configurations [2].

[1] to make the `zc.recipe.egg` part of buildout behave exactly like it was
still using setuptools, and pkg_resources would still need to be taught how
to read .dist-info directories.
[2] to use something else than `zc.recipe.egg` to install Python packages
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160824/efb58a94/attachment-0001.html>

From dholth at gmail.com  Wed Aug 24 13:37:20 2016
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 24 Aug 2016 17:37:20 +0000
Subject: [Distutils] What role to eggs still play?
In-Reply-To: <CAMxX_oMNm9vqMDqvwzRYZoAkE76KH0rmVLbj0SxcgB4_H7+hRw@mail.gmail.com>
References: <CALGmxE+=4-gqgbAR1uGODKR1C-zjrSCEyBXOqp-2x_ypAeb8aw@mail.gmail.com>
 <CAG8k2+4W9Y3qhpBBhDXcO6wPDkjJUDDJdcQ=YZGEidOXXcbxdw@mail.gmail.com>
 <CAG8k2+5UYQ=dJBuVr-SjFU_=+h7GR60pp7bBwZNnrxx-QV0D9g@mail.gmail.com>
 <3388930217277504347@unknownmsgid>
 <5b8f3ce4-0b59-93d8-47e9-0cf243ffabb0@nextday.fi>
 <CAG8k2+5hOOtiKh5hQAjNDpujwUNyazZA0N=JSC0vH4A7g70zKg@mail.gmail.com>
 <CAMxX_oMNm9vqMDqvwzRYZoAkE76KH0rmVLbj0SxcgB4_H7+hRw@mail.gmail.com>
Message-ID: <CAG8k2+7jy_BtK7GK5Q_a8mp6yRC2cghtm=MH_GXRCeAO7XCpSg@mail.gmail.com>

On Wed, Aug 24, 2016 at 1:28 PM Leonardo Rochael Almeida <
leorochael at gmail.com> wrote:

> On 24 August 2016 at 12:56, Daniel Holth <dholth at gmail.com> wrote:
>
>> On Wed, Aug 24, 2016 at 10:41 AM Alex Gr?nholm <alex.gronholm at nextday.fi>
>> wrote:
>>
> [...]
>>>
>>
>> 19.08.2016, 20:25, Daniel Holth kirjoitti:
>>
>> Eggs are the only way to add a zipped distribution to PYTHONPATH and have
>> setuptools find the metadata (the Python code can be found with or without
>> the metadata; setuptools does not discover *.dist-info inside zip). Eggs
>> are used by buildout, especially in the unzipped into a directory form. And
>> they could still be used for their originally designed use as a plugin
>> format.
>>
>> Alex kirjoitti: "What is not clear to me is who needs this. Is this not
>> what virtualenvs are for? Or could wheels not be used for the same purpose?
>> What am I missing?"
>>
>
> There is some mixing of concerns here, but to clarify: wheels in general
> CANNOT be blindly dropped on PYTHONPATH and be expected to work.
>
> Only wheels that DO NOT contain an entry like
> `wheel_name-1.0.data/purelib` or `wheel_name-1.0.data/platlib` can be
> dropped on PYTHONPATH to make their code available to a python process.
>
> But that only works for python code. `setuptools`/`pkg_resources` will
> currently be unable to read entry points (in particular, console_scripts)
> from such wheels dropped in the PYTHONPATH.
>
> So, they cannot be used in general for the purpose of dropping them on the
> PYTHONPATH. But the buildout communities (incl. Plone) do not exactly
> require wheels dropped on the PYTHONPATH. See below?
>
> What you are missing is that while they could in theory be used for the
>> same purpose, in practice there is a large group of Plone & buildout users
>> who are not using wheels for that. (And the wheels would have to be
>> unzipped into their own individual directories, but buildout almost always
>> unzips eggs lately too.)
>>
>
> In particular, buildout users on Windows, who cannot easily compile sdists
> that contain C extensions and which are currently uploaded as eggs to PyPI
> (see lxml and zope.interface as examples of highly used packages in this
> regard).
>
> As for wheels, not only would these wheels have to be unzipped, but their
> `purelib`/`platlib` contents, if present, copied to the "root" of where
> they were unpacked, and the `wheel_name-1.0.dist-info` directory renamed to
> `EGG-INFO`
>

pkg_resources has been able to present an identical interface to EGG-INFO
and dist-info directories for longer than pip has known about wheels. Entry
points will also work fine. buildout would of course have to generate the
console_scripts, which it always does anyway.

Does anyone have an example of how to produce wheels with both purelib and
platlib using setuptools?

But the most important thing of all, these wheels would have to participate
> in the dependency resolution of other packages along with sdists in the
> middle of a buildout run.
>

Yes.

The last aspect means that having a canonical piece of code that knows how
> to pick up a wheel from PyPI and unpack it correctly somewhere in the
> PYTHONPATH is not enough.
>
> (and nobody has offered such canonical a piece of software yet, except the
> not really useful "call out to pip as a subprocess").
>
> What is needed is for setuptools to be modified to treat wheels the same
> way it currently treats bdist_wininst files for installation: as a binary
> lump that it knows how to download from PyPI, unpack and convert into an
> egg.
>
> On the other hand, if the above happens, we can block .egg files from
> upload to PyPI along with the rest of the formats. Wouldn't it be nice?
>
> Any other option for adding wheels support to buildout (calling out to
> pip, using distlib,... ) will involve either a lot of code rewrite [1] or
> the need to rewrite many currently functioning buildout configurations [2].
>
> [1] to make the `zc.recipe.egg` part of buildout behave exactly like it
> was still using setuptools, and pkg_resources would still need to be taught
> how to read .dist-info directories.
> [2] to use something else than `zc.recipe.egg` to install Python packages
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160824/4955eff8/attachment.html>

From donald at stufft.io  Wed Aug 24 16:48:26 2016
From: donald at stufft.io (Donald Stufft)
Date: Wed, 24 Aug 2016 16:48:26 -0400
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
Message-ID: <00B6BA52-2E9E-45E4-907F-1054D8AABEF0@stufft.io>

Ok, I?ve updated the PEP. You can see a diff at https://github.com/python/peps/pull/74/files or the entire PEP at https://www.python.org/dev/peps/pep-0527/ once that updated. However the primary differences are:

* Added Nick Coghlan as the BDFL Delegate (Thanks Nick!).
* Expanded upon rationale for dropping everything but sdist/wheel/egg.
* Allow an sdist to be either .tar.gz or .zip, but still drop all other formats.

Hopefully that can adequately address Marc-Andre?s concerns for the removal of the other file types, and the inclusion of .zip will address other folk?s concerns about losing .zip.

?
Donald Stufft




From ralf.gommers at gmail.com  Thu Aug 25 00:41:43 2016
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Thu, 25 Aug 2016 16:41:43 +1200
Subject: [Distutils] Proposed new Distutils API for compiler flag
 detection (Issue26689)
In-Reply-To: <1471848656.3689082.702076721.53722769@webmail.messagingengine.com>
References: <A45B9ADE-2FB4-4453-AACE-8B1595C009E3@python.org>
 <CABL7CQi86ORbhQUy11g4ipiEWMKFYtzeeYrb1JkKomsVPa8uKQ@mail.gmail.com>
 <CAK=Phk5YV_Ax34_DRV9b4ch4T7GNMZ2=E4DH=QP0DzeAA8v83A@mail.gmail.com>
 <CAPJVwBnOu6i3f-_XspZtaMR8Va68suHVb9p9HiVQ6Cp8-CFFTQ@mail.gmail.com>
 <CAK=Phk4_XgZu_z_qu_mHFKE96V2c-dJHdixJKtrckBjQ2L88Tw@mail.gmail.com>
 <34793887-E8A4-4694-ADD2-04F45F789651@stufft.io>
 <CAK=Phk4YRKgbH7Qxy2j2smuHpqGEpaNFgFtzEoHtH4qZaz92qw@mail.gmail.com>
 <63D859EB-B20C-4413-ACF4-93EB22C7A7FD@stufft.io>
 <CABL7CQgggpKpO5UG1vXfDFo1eR3ccyNRn7AcVt7OB=YKey+CCA@mail.gmail.com>
 <CAK=Phk6Qe=UWvKH-d2UbxJG9QSnDjHURwfz0rNPZ1GsxoYTr4A@mail.gmail.com>
 <1471848656.3689082.702076721.53722769@webmail.messagingengine.com>
Message-ID: <CABL7CQjWvNogHV39YMxTpv4wfVAEOo+ZC4LcPnynArVi5hVDqg@mail.gmail.com>

On Mon, Aug 22, 2016 at 6:50 PM, Thomas Kluyver <thomas at kluyver.me.uk>
wrote:

> On Mon, Aug 22, 2016, at 07:15 AM, Sylvain Corlay wrote:
>
> I find this worrying that the main arguments to not include a patch would
> be that
>
>  - this part of the standard library is not very maintained (things don't
> get merged)
>  - earlier versions of won't have it
>
>
> Would it make sense to add it to both distutils and setuptools? The
> standard library continues to evolve, projects that require Python 3.6
> wouldn't need to use setuptools, but we could start using it sooner.
>

I don't have a problem with this, at least it avoids the main issues I
pointed out. Although I also don't see much benefit of adding the code to
distutils as well, given that the non-setuptools use is effectively
deprecated (by not adding support for new PEPs in distutils for example)
and less and less relevant every year.


> There's obviously some cost in code duplication; I haven't looked at the
> code in question, so I don't know how bad this is.
>

This patch is pretty short and understandable, so not bad.


> I've run into this argument before when trying to change things in
> non-packaging-related parts of the stdlib, and I agree with Sylvain that
> it's fundamentally problematic. If we're trying to improve the stdlib,
> we're obviously taking a long view, but that's how we ensure the stdlib is
> still useful in a few years time. This goes for packaging tools as much as
> anything else.
>

This I don't agree with - packaging is fundamentally different for the
reasons Donald gave.

Ralf


> I already have projects where I'm happy to require Python >=3.4, so being
> able to depend on Python 3.6 is not such a distant prospect.
>
> Thomas
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160825/8277ceaa/attachment-0001.html>

From ncoghlan at gmail.com  Thu Aug 25 07:30:07 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 Aug 2016 21:30:07 +1000
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <00B6BA52-2E9E-45E4-907F-1054D8AABEF0@stufft.io>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
 <00B6BA52-2E9E-45E4-907F-1054D8AABEF0@stufft.io>
Message-ID: <CADiSq7cbtdBpE__dnGfBa-3xRhSJBeo087xifo2vyU2qsPtvGA@mail.gmail.com>

On 25 August 2016 at 06:48, Donald Stufft <donald at stufft.io> wrote:
> Ok, I?ve updated the PEP. You can see a diff at https://github.com/python/peps/pull/74/files or the entire PEP at https://www.python.org/dev/peps/pep-0527/ once that updated. However the primary differences are:
>
> * Added Nick Coghlan as the BDFL Delegate (Thanks Nick!).
> * Expanded upon rationale for dropping everything but sdist/wheel/egg.
> * Allow an sdist to be either .tar.gz or .zip, but still drop all other formats.
>
> Hopefully that can adequately address Marc-Andre?s concerns for the removal of the other file types, and the inclusion of .zip will address other folk?s concerns about losing .zip.

Thanks for those updates.

My one remaining question would be whether or not we have any projects
that are in the habit of uploading both .zip and .tar.gz for the same
release - if there are, the restriction to only one sdist per release
will need to be included in the notification message to maintainers.
(The introduction of that restriction could probably be clearer in the
PEP text as well - previously it was implied by the single extension,
but it may need to be called out more explicitly now that there are
two permitted options).

Aside from that, I'm inclined to give folks another few days to raise
questions and concerns, and then consider the PEP for acceptance early
next week.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From thomas at kluyver.me.uk  Thu Aug 25 07:46:22 2016
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Thu, 25 Aug 2016 12:46:22 +0100
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <CADiSq7cbtdBpE__dnGfBa-3xRhSJBeo087xifo2vyU2qsPtvGA@mail.gmail.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
 <00B6BA52-2E9E-45E4-907F-1054D8AABEF0@stufft.io>
 <CADiSq7cbtdBpE__dnGfBa-3xRhSJBeo087xifo2vyU2qsPtvGA@mail.gmail.com>
Message-ID: <1472125582.3670703.705766377.28264F24@webmail.messagingengine.com>

On Thu, Aug 25, 2016, at 12:30 PM, Nick Coghlan wrote:
> My one remaining question would be whether or not we have any projects
> that are in the habit of uploading both .zip and .tar.gz for the same
> release - if there are, the restriction to only one sdist per release

IPython's release script uploads both sdist formats:
https://pypi.python.org/pypi/ipython

Several of our other IPython/Jupyter packages have both formats on PyPI.
I don't know how unusual we are.

From donald at stufft.io  Thu Aug 25 07:57:52 2016
From: donald at stufft.io (Donald Stufft)
Date: Thu, 25 Aug 2016 07:57:52 -0400
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <CADiSq7cbtdBpE__dnGfBa-3xRhSJBeo087xifo2vyU2qsPtvGA@mail.gmail.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
 <00B6BA52-2E9E-45E4-907F-1054D8AABEF0@stufft.io>
 <CADiSq7cbtdBpE__dnGfBa-3xRhSJBeo087xifo2vyU2qsPtvGA@mail.gmail.com>
Message-ID: <7176BF59-16B6-4F17-A7E4-626B5CAE13AD@stufft.io>


> On Aug 25, 2016, at 7:30 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 25 August 2016 at 06:48, Donald Stufft <donald at stufft.io> wrote:
>> Ok, I?ve updated the PEP. You can see a diff at https://github.com/python/peps/pull/74/files or the entire PEP at https://www.python.org/dev/peps/pep-0527/ once that updated. However the primary differences are:
>> 
>> * Added Nick Coghlan as the BDFL Delegate (Thanks Nick!).
>> * Expanded upon rationale for dropping everything but sdist/wheel/egg.
>> * Allow an sdist to be either .tar.gz or .zip, but still drop all other formats.
>> 
>> Hopefully that can adequately address Marc-Andre?s concerns for the removal of the other file types, and the inclusion of .zip will address other folk?s concerns about losing .zip.
> 
> Thanks for those updates.
> 
> My one remaining question would be whether or not we have any projects
> that are in the habit of uploading both .zip and .tar.gz for the same
> release - if there are, the restriction to only one sdist per release
> will need to be included in the notification message to maintainers.
> (The introduction of that restriction could probably be clearer in the
> PEP text as well - previously it was implied by the single extension,
> but it may need to be called out more explicitly now that there are
> two permitted options).

My querying shows that we have a total of 359 projects that have ever
uploaded two sdists for a single release. Looking at the number of
releases that have 2+ sdists per project, it appears like there are a
handful that do it regularly, and then the vast bulk have only done it
once or twice (see: https://bpaste.net/show/fdfce63673ec).

I?ve gone ahead and explicitly call out the restriction of a single
sdist per release in the PEP, diff can be found online at
https://github.com/python/peps/pull/75/files.

> 
> Aside from that, I'm inclined to give folks another few days to raise
> questions and concerns, and then consider the PEP for acceptance early
> next week.
> 

Sounds good.


?
Donald Stufft




From mal at egenix.com  Thu Aug 25 11:46:03 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 25 Aug 2016 17:46:03 +0200
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <7176BF59-16B6-4F17-A7E4-626B5CAE13AD@stufft.io>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
 <00B6BA52-2E9E-45E4-907F-1054D8AABEF0@stufft.io>
 <CADiSq7cbtdBpE__dnGfBa-3xRhSJBeo087xifo2vyU2qsPtvGA@mail.gmail.com>
 <7176BF59-16B6-4F17-A7E4-626B5CAE13AD@stufft.io>
Message-ID: <57BF12BB.50101@egenix.com>

+sources of truth for a single version. Having multiple sdists often
times can
+account for strange bugs that only expose themselves based on which
sdist that
+the person used.

You may not be aware, but developers that work on both Windows
and Unix often have two sets of source code packages: one using
Windows line ends, the other using Unix ones.

The Windows ones can also include code which is only relevant
on Windows while the Unix one includes parts that are only used
on Unix, so having two sets (ZIP for Windows and .tar.gz for Unix)
is a natural way to distribute your source code for those two
target systems.

Standardizing on two sdist formats is fine, but artificially
limiting this to just one sdist upload removes useful
functionality.


On 25.08.2016 13:57, Donald Stufft wrote:
> 
>> On Aug 25, 2016, at 7:30 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> On 25 August 2016 at 06:48, Donald Stufft <donald at stufft.io> wrote:
>>> Ok, I?ve updated the PEP. You can see a diff at https://github.com/python/peps/pull/74/files or the entire PEP at https://www.python.org/dev/peps/pep-0527/ once that updated. However the primary differences are:
>>>
>>> * Added Nick Coghlan as the BDFL Delegate (Thanks Nick!).
>>> * Expanded upon rationale for dropping everything but sdist/wheel/egg.
>>> * Allow an sdist to be either .tar.gz or .zip, but still drop all other formats.
>>>
>>> Hopefully that can adequately address Marc-Andre?s concerns for the removal of the other file types, and the inclusion of .zip will address other folk?s concerns about losing .zip.
>>
>> Thanks for those updates.
>>
>> My one remaining question would be whether or not we have any projects
>> that are in the habit of uploading both .zip and .tar.gz for the same
>> release - if there are, the restriction to only one sdist per release
>> will need to be included in the notification message to maintainers.
>> (The introduction of that restriction could probably be clearer in the
>> PEP text as well - previously it was implied by the single extension,
>> but it may need to be called out more explicitly now that there are
>> two permitted options).
> 
> My querying shows that we have a total of 359 projects that have ever
> uploaded two sdists for a single release. Looking at the number of
> releases that have 2+ sdists per project, it appears like there are a
> handful that do it regularly, and then the vast bulk have only done it
> once or twice (see: https://bpaste.net/show/fdfce63673ec).
> 
> I?ve gone ahead and explicitly call out the restriction of a single
> sdist per release in the PEP, diff can be found online at
> https://github.com/python/peps/pull/75/files.
> 
>>
>> Aside from that, I'm inclined to give folks another few days to raise
>> questions and concerns, and then consider the PEP for acceptance early
>> next week.
>>
> 
> Sounds good.
> 
> 
> ?
> Donald Stufft
> 
> 
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Aug 25 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From p.f.moore at gmail.com  Thu Aug 25 11:54:07 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 25 Aug 2016 16:54:07 +0100
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <57BF12BB.50101@egenix.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
 <00B6BA52-2E9E-45E4-907F-1054D8AABEF0@stufft.io>
 <CADiSq7cbtdBpE__dnGfBa-3xRhSJBeo087xifo2vyU2qsPtvGA@mail.gmail.com>
 <7176BF59-16B6-4F17-A7E4-626B5CAE13AD@stufft.io> <57BF12BB.50101@egenix.com>
Message-ID: <CACac1F9egXugsMqjpH=19m52inym804pX0VdqDz88SCnFc=4wQ@mail.gmail.com>

On 25 August 2016 at 16:46, M.-A. Lemburg <mal at egenix.com> wrote:
> The Windows ones can also include code which is only relevant
> on Windows while the Unix one includes parts that are only used
> on Unix, so having two sets (ZIP for Windows and .tar.gz for Unix)
> is a natural way to distribute your source code for those two
> target systems.

> Standardizing on two sdist formats is fine, but artificially
> limiting this to just one sdist upload removes useful
> functionality.

While I can't speak for others, I would be extremely confused (and
annoyed) if 2 sdist files for the same release of a single Python
package, using different archive formats, contained different
contents. I'm not aware of any package on PyPI that does this [1], and
I'd hope that remains the case (which it will, trivially, under the
new proposal to only allow one sdist file per release).

Paul

[1] If there are any, I'd be happy to file a bug report against them ;-)

From solipsis at pitrou.net  Thu Aug 25 12:03:43 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 25 Aug 2016 18:03:43 +0200
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
 <00B6BA52-2E9E-45E4-907F-1054D8AABEF0@stufft.io>
 <CADiSq7cbtdBpE__dnGfBa-3xRhSJBeo087xifo2vyU2qsPtvGA@mail.gmail.com>
 <7176BF59-16B6-4F17-A7E4-626B5CAE13AD@stufft.io>
 <57BF12BB.50101@egenix.com>
Message-ID: <20160825180343.731c205a@fsol>

On Thu, 25 Aug 2016 17:46:03 +0200
"M.-A. Lemburg" <mal at egenix.com> wrote:

> +sources of truth for a single version. Having multiple sdists often
> times can
> +account for strange bugs that only expose themselves based on which
> sdist that
> +the person used.
> 
> You may not be aware, but developers that work on both Windows
> and Unix often have two sets of source code packages: one using
> Windows line ends, the other using Unix ones.

"often"? According to which sources? :-)

I do work on both Windows and Unix (though Linux is my primary
development platform), and I've never had a separate source code
package for Windows (with differing files, line endings or whatnot).

I also agree that having multiple sdists for a single release is a
recipe for weird issues.

Regards

Antoine.



From donald at stufft.io  Thu Aug 25 12:11:40 2016
From: donald at stufft.io (Donald Stufft)
Date: Thu, 25 Aug 2016 12:11:40 -0400
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <57BF12BB.50101@egenix.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
 <00B6BA52-2E9E-45E4-907F-1054D8AABEF0@stufft.io>
 <CADiSq7cbtdBpE__dnGfBa-3xRhSJBeo087xifo2vyU2qsPtvGA@mail.gmail.com>
 <7176BF59-16B6-4F17-A7E4-626B5CAE13AD@stufft.io> <57BF12BB.50101@egenix.com>
Message-ID: <9D748E37-F7FD-4464-9FAE-3F2A5365E0ED@stufft.io>


> On Aug 25, 2016, at 11:46 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> 
> You may not be aware, but developers that work on both Windows
> and Unix often have two sets of source code packages: one using
> Windows line ends, the other using Unix ones.
> 
> The Windows ones can also include code which is only relevant
> on Windows while the Unix one includes parts that are only used
> on Unix, so having two sets (ZIP for Windows and .tar.gz for Unix)
> is a natural way to distribute your source code for those two
> target systems.
> 
> Standardizing on two sdist formats is fine, but artificially
> limiting this to just one sdist upload removes useful
> functionality.

Well, except this doesn?t actually work in practice and anyone
doing this has a broken sdist. Neither pip nor setuptools is going
to consider a ``.tar.gz`` as a ?Linux? sdist nor a ``.zip`` as a
?Windows? sdist and which one they happen to pick is basically an
implementation detail (in the latest version of pip, it?ll use
whichever one appears first in the /simple/<whatever>/ page.

In fact people doing this (which I am entirely sure is not common,
but I?m sure there is someone doing it) are likely to be the cause
of the weird issues that this part of the PEP is attempting to
prevent. Someone who happens to get the ?Windows? sdist on Linux
and then is incredibly confused when it doesn?t have the Linux pieces
they need (and remember, none of the tooling supports the idea that
.tar.gz is for Linux and .zip is for Windows, so who knows which one
they?ll get). Of course, if the only difference is Windows vs Unix
line endings it?ll be ~roughly fine because Unix line endings work
fine on Windows, and Windows line endings work fine on Unix, but
shipping line ending differences to the correct OS is not a good
enough reason to keep this.

?
Donald Stufft




From donald at stufft.io  Thu Aug 25 12:11:40 2016
From: donald at stufft.io (Donald Stufft)
Date: Thu, 25 Aug 2016 12:11:40 -0400
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <57BF12BB.50101@egenix.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
 <00B6BA52-2E9E-45E4-907F-1054D8AABEF0@stufft.io>
 <CADiSq7cbtdBpE__dnGfBa-3xRhSJBeo087xifo2vyU2qsPtvGA@mail.gmail.com>
 <7176BF59-16B6-4F17-A7E4-626B5CAE13AD@stufft.io> <57BF12BB.50101@egenix.com>
Message-ID: <E4070B13-A67E-4F70-9A2C-568D97211DFB@stufft.io>


> On Aug 25, 2016, at 11:46 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> 
> You may not be aware, but developers that work on both Windows
> and Unix often have two sets of source code packages: one using
> Windows line ends, the other using Unix ones.
> 
> The Windows ones can also include code which is only relevant
> on Windows while the Unix one includes parts that are only used
> on Unix, so having two sets (ZIP for Windows and .tar.gz for Unix)
> is a natural way to distribute your source code for those two
> target systems.
> 
> Standardizing on two sdist formats is fine, but artificially
> limiting this to just one sdist upload removes useful
> functionality.

Well, except this doesn?t actually work in practice and anyone
doing this has a broken sdist. Neither pip nor setuptools is going
to consider a ``.tar.gz`` as a ?Linux? sdist nor a ``.zip`` as a
?Windows? sdist and which one they happen to pick is basically an
implementation detail (in the latest version of pip, it?ll use
whichever one appears first in the /simple/<whatever>/ page.

In fact people doing this (which I am entirely sure is not common,
but I?m sure there is someone doing it) are likely to be the cause
of the weird issues that this part of the PEP is attempting to
prevent. Someone who happens to get the ?Windows? sdist on Linux
and then is incredibly confused when it doesn?t have the Linux pieces
they need (and remember, none of the tooling supports the idea that
.tar.gz is for Linux and .zip is for Windows, so who knows which one
they?ll get). Of course, if the only difference is Windows vs Unix
line endings it?ll be ~roughly fine because Unix line endings work
fine on Windows, and Windows line endings work fine on Unix, but
shipping line ending differences to the correct OS is not a good
enough reason to keep this.

?
Donald Stufft




From wes.turner at gmail.com  Thu Aug 25 12:15:01 2016
From: wes.turner at gmail.com (Wes Turner)
Date: Thu, 25 Aug 2016 11:15:01 -0500
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <9D748E37-F7FD-4464-9FAE-3F2A5365E0ED@stufft.io>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
 <00B6BA52-2E9E-45E4-907F-1054D8AABEF0@stufft.io>
 <CADiSq7cbtdBpE__dnGfBa-3xRhSJBeo087xifo2vyU2qsPtvGA@mail.gmail.com>
 <7176BF59-16B6-4F17-A7E4-626B5CAE13AD@stufft.io> <57BF12BB.50101@egenix.com>
 <9D748E37-F7FD-4464-9FAE-3F2A5365E0ED@stufft.io>
Message-ID: <CACfEFw_bU_ZkzPpEJ1--=n1vb_RF8fNfw86KUkzhM5j9T2CWew@mail.gmail.com>

On Thu, Aug 25, 2016 at 11:11 AM, Donald Stufft <donald at stufft.io> wrote:

>
> > On Aug 25, 2016, at 11:46 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> >
> > You may not be aware, but developers that work on both Windows
> > and Unix often have two sets of source code packages: one using
> > Windows line ends, the other using Unix ones.
> >
> > The Windows ones can also include code which is only relevant
> > on Windows while the Unix one includes parts that are only used
> > on Unix, so having two sets (ZIP for Windows and .tar.gz for Unix)
> > is a natural way to distribute your source code for those two
> > target systems.
> >
> > Standardizing on two sdist formats is fine, but artificially
> > limiting this to just one sdist upload removes useful
> > functionality.
>
> Well, except this doesn?t actually work in practice and anyone
> doing this has a broken sdist. Neither pip nor setuptools is going
> to consider a ``.tar.gz`` as a ?Linux? sdist nor a ``.zip`` as a
> ?Windows? sdist and which one they happen to pick is basically an
> implementation detail (in the latest version of pip, it?ll use
> whichever one appears first in the /simple/<whatever>/ page.
>

Is that because there wasn't an OrderedDict
(either in the data producer (pypi, warehouse) or in the data consumer
(pip, easy_install)?


>
> In fact people doing this (which I am entirely sure is not common,
> but I?m sure there is someone doing it) are likely to be the cause
> of the weird issues that this part of the PEP is attempting to
> prevent. Someone who happens to get the ?Windows? sdist on Linux
> and then is incredibly confused when it doesn?t have the Linux pieces
> they need (and remember, none of the tooling supports the idea that
> .tar.gz is for Linux and .zip is for Windows, so who knows which one
> they?ll get). Of course, if the only difference is Windows vs Unix
> line endings it?ll be ~roughly fine because Unix line endings work
> fine on Windows, and Windows line endings work fine on Unix, but
> shipping line ending differences to the correct OS is not a good
> enough reason to keep this.
>
> ?
> Donald Stufft
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160825/f58c18c4/attachment-0001.html>

From ncoghlan at gmail.com  Thu Aug 25 12:29:59 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 26 Aug 2016 02:29:59 +1000
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <1472125582.3670703.705766377.28264F24@webmail.messagingengine.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
 <00B6BA52-2E9E-45E4-907F-1054D8AABEF0@stufft.io>
 <CADiSq7cbtdBpE__dnGfBa-3xRhSJBeo087xifo2vyU2qsPtvGA@mail.gmail.com>
 <1472125582.3670703.705766377.28264F24@webmail.messagingengine.com>
Message-ID: <CADiSq7fRoP73jw8v7bLqB+HT5tYnevvKiT6SsZUC--8KuVp+aA@mail.gmail.com>

On 25 August 2016 at 21:46, Thomas Kluyver <thomas at kluyver.me.uk> wrote:
> On Thu, Aug 25, 2016, at 12:30 PM, Nick Coghlan wrote:
>> My one remaining question would be whether or not we have any projects
>> that are in the habit of uploading both .zip and .tar.gz for the same
>> release - if there are, the restriction to only one sdist per release
>
> IPython's release script uploads both sdist formats:
> https://pypi.python.org/pypi/ipython
>
> Several of our other IPython/Jupyter packages have both formats on PyPI.

Could you give a bit more detail on how you came to be publishing
both? The main thing we're trying to avoid is missing a practical use
case for the status quo where folks can upload both - if it's just an
artifact of Windows and *nix having different default formats, then
the convergence in distutils and setuptools will fix it implicitly,
but if it's a deliberate design decision, then we need to check if
that's based on a misunderstanding of how pip/easy_install/et al
consume the two formats.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From dholth at gmail.com  Thu Aug 25 12:26:11 2016
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 25 Aug 2016 16:26:11 +0000
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <CACfEFw_bU_ZkzPpEJ1--=n1vb_RF8fNfw86KUkzhM5j9T2CWew@mail.gmail.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
 <00B6BA52-2E9E-45E4-907F-1054D8AABEF0@stufft.io>
 <CADiSq7cbtdBpE__dnGfBa-3xRhSJBeo087xifo2vyU2qsPtvGA@mail.gmail.com>
 <7176BF59-16B6-4F17-A7E4-626B5CAE13AD@stufft.io> <57BF12BB.50101@egenix.com>
 <9D748E37-F7FD-4464-9FAE-3F2A5365E0ED@stufft.io>
 <CACfEFw_bU_ZkzPpEJ1--=n1vb_RF8fNfw86KUkzhM5j9T2CWew@mail.gmail.com>
Message-ID: <CAG8k2+4CY6_o=OdcPTsbEV0D++ptWaD3d+B5Amby_jRSXm+uFA@mail.gmail.com>

Wheels have a sort order depending on the tag that is independent of the
order present in pypi. Sdists don't have anything like that.

On Thu, Aug 25, 2016, 12:15 Wes Turner <wes.turner at gmail.com> wrote:

> On Thu, Aug 25, 2016 at 11:11 AM, Donald Stufft <donald at stufft.io> wrote:
>
>>
>> > On Aug 25, 2016, at 11:46 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>> >
>> > You may not be aware, but developers that work on both Windows
>> > and Unix often have two sets of source code packages: one using
>> > Windows line ends, the other using Unix ones.
>> >
>> > The Windows ones can also include code which is only relevant
>> > on Windows while the Unix one includes parts that are only used
>> > on Unix, so having two sets (ZIP for Windows and .tar.gz for Unix)
>> > is a natural way to distribute your source code for those two
>> > target systems.
>> >
>> > Standardizing on two sdist formats is fine, but artificially
>> > limiting this to just one sdist upload removes useful
>> > functionality.
>>
>> Well, except this doesn?t actually work in practice and anyone
>> doing this has a broken sdist. Neither pip nor setuptools is going
>> to consider a ``.tar.gz`` as a ?Linux? sdist nor a ``.zip`` as a
>> ?Windows? sdist and which one they happen to pick is basically an
>> implementation detail (in the latest version of pip, it?ll use
>> whichever one appears first in the /simple/<whatever>/ page.
>>
>
> Is that because there wasn't an OrderedDict
> (either in the data producer (pypi, warehouse) or in the data consumer
> (pip, easy_install)?
>
>
>>
>> In fact people doing this (which I am entirely sure is not common,
>> but I?m sure there is someone doing it) are likely to be the cause
>> of the weird issues that this part of the PEP is attempting to
>> prevent. Someone who happens to get the ?Windows? sdist on Linux
>> and then is incredibly confused when it doesn?t have the Linux pieces
>> they need (and remember, none of the tooling supports the idea that
>> .tar.gz is for Linux and .zip is for Windows, so who knows which one
>> they?ll get). Of course, if the only difference is Windows vs Unix
>> line endings it?ll be ~roughly fine because Unix line endings work
>> fine on Windows, and Windows line endings work fine on Unix, but
>> shipping line ending differences to the correct OS is not a good
>> enough reason to keep this.
>>
>> ?
>> Donald Stufft
>>
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160825/f5aaa189/attachment.html>

From mal at egenix.com  Thu Aug 25 12:43:42 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 25 Aug 2016 18:43:42 +0200
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <20160825180343.731c205a@fsol>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
 <00B6BA52-2E9E-45E4-907F-1054D8AABEF0@stufft.io>
 <CADiSq7cbtdBpE__dnGfBa-3xRhSJBeo087xifo2vyU2qsPtvGA@mail.gmail.com>
 <7176BF59-16B6-4F17-A7E4-626B5CAE13AD@stufft.io> <57BF12BB.50101@egenix.com>
 <20160825180343.731c205a@fsol>
Message-ID: <57BF203E.8090200@egenix.com>

On 25.08.2016 18:03, Antoine Pitrou wrote:
> On Thu, 25 Aug 2016 17:46:03 +0200
> "M.-A. Lemburg" <mal at egenix.com> wrote:
> 
>> +sources of truth for a single version. Having multiple sdists often
>> times can
>> +account for strange bugs that only expose themselves based on which
>> sdist that
>> +the person used.
>>
>> You may not be aware, but developers that work on both Windows
>> and Unix often have two sets of source code packages: one using
>> Windows line ends, the other using Unix ones.
> 
> "often"? According to which sources? :-)

I should have written: "sometimes" :-) It's not very common for
Python packages. It was rather common for cross platform open source
applications before everyone switched to pointing folks to github
for source downloads some years ago.

I suppose the main reason to ship different archives on PyPI is to
address different tooling needs when manually downloading source
archives to e.g. have a look inside for some code review before
starting to use a package.

Aside: If pip is considered the only user of PyPI, I do wonder,
why we bother having a user readable download index at all ;-)

> I do work on both Windows and Unix (though Linux is my primary
> development platform), and I've never had a separate source code
> package for Windows (with differing files, line endings or whatnot).
> 
> I also agree that having multiple sdists for a single release is a
> recipe for weird issues.

It can be an issue, but mostly due to permission problems you can
find with .tar.gz files which do not happen with ZIP files.

OTOH, .tar.gz files do handle symlinks well, which ZIP files
often don't (or at least not per default).

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Aug 25 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From ncoghlan at gmail.com  Thu Aug 25 12:57:11 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 26 Aug 2016 02:57:11 +1000
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <57BF203E.8090200@egenix.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
 <00B6BA52-2E9E-45E4-907F-1054D8AABEF0@stufft.io>
 <CADiSq7cbtdBpE__dnGfBa-3xRhSJBeo087xifo2vyU2qsPtvGA@mail.gmail.com>
 <7176BF59-16B6-4F17-A7E4-626B5CAE13AD@stufft.io> <57BF12BB.50101@egenix.com>
 <20160825180343.731c205a@fsol> <57BF203E.8090200@egenix.com>
Message-ID: <CADiSq7e=r=QLCowRDmOZbObTADaMCQ+T6Y+bSKYd6aa=7V_QbA@mail.gmail.com>

On 26 August 2016 at 02:43, M.-A. Lemburg <mal at egenix.com> wrote:
> Aside: If pip is considered the only user of PyPI, I do wonder,
> why we bother having a user readable download index at all ;-)

I see the smiley, but this actually has a real answer :)

The manual download links aren't particularly useful (that's why
Warehouse moves them to a subpage and encourages the use of "pip
install" instead), but the user readable pages themselves are relevant
to folks searching for dependencies that might solve their problems
(whether using PyPI's native search, or a general internet search
engine).

And once you have a project information page, not having any manual
download links *at all* would be sufficiently strange that their
absence could reasonably be expected to puzzle people, so it makes
sense to just provide them, rather than attempting to explain why
they're not there.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From dholth at gmail.com  Thu Aug 25 13:10:43 2016
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 25 Aug 2016 17:10:43 +0000
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <CADiSq7e=r=QLCowRDmOZbObTADaMCQ+T6Y+bSKYd6aa=7V_QbA@mail.gmail.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
 <00B6BA52-2E9E-45E4-907F-1054D8AABEF0@stufft.io>
 <CADiSq7cbtdBpE__dnGfBa-3xRhSJBeo087xifo2vyU2qsPtvGA@mail.gmail.com>
 <7176BF59-16B6-4F17-A7E4-626B5CAE13AD@stufft.io> <57BF12BB.50101@egenix.com>
 <20160825180343.731c205a@fsol> <57BF203E.8090200@egenix.com>
 <CADiSq7e=r=QLCowRDmOZbObTADaMCQ+T6Y+bSKYd6aa=7V_QbA@mail.gmail.com>
Message-ID: <CAG8k2+7UJHoWuXNtGFgj3pRH=CYHg=Bf4icNgtas3T0f7LsRaQ@mail.gmail.com>

The funny thing about a feature like the manual download link is that no,
99.99% of the time it is not useful, but when you need it, you probably
really need it. Probably to debug an issue with pip, or to grab a package
pip would not normally download for your platform, or to make sure
something has been uploaded correctly. I would not be surprised if the
upload feature gets even less use than the manual downloads.

On Thu, Aug 25, 2016 at 12:57 PM Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 26 August 2016 at 02:43, M.-A. Lemburg <mal at egenix.com> wrote:
> > Aside: If pip is considered the only user of PyPI, I do wonder,
> > why we bother having a user readable download index at all ;-)
>
> I see the smiley, but this actually has a real answer :)
>
> The manual download links aren't particularly useful (that's why
> Warehouse moves them to a subpage and encourages the use of "pip
> install" instead), but the user readable pages themselves are relevant
> to folks searching for dependencies that might solve their problems
> (whether using PyPI's native search, or a general internet search
> engine).
>
> And once you have a project information page, not having any manual
> download links *at all* would be sufficiently strange that their
> absence could reasonably be expected to puzzle people, so it makes
> sense to just provide them, rather than attempting to explain why
> they're not there.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160825/940f4f60/attachment.html>

From donald at stufft.io  Thu Aug 25 13:12:18 2016
From: donald at stufft.io (Donald Stufft)
Date: Thu, 25 Aug 2016 13:12:18 -0400
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <57BF203E.8090200@egenix.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
 <00B6BA52-2E9E-45E4-907F-1054D8AABEF0@stufft.io>
 <CADiSq7cbtdBpE__dnGfBa-3xRhSJBeo087xifo2vyU2qsPtvGA@mail.gmail.com>
 <7176BF59-16B6-4F17-A7E4-626B5CAE13AD@stufft.io> <57BF12BB.50101@egenix.com>
 <20160825180343.731c205a@fsol> <57BF203E.8090200@egenix.com>
Message-ID: <38080F87-373A-42B4-A96B-95ABE8B0F32E@stufft.io>


> On Aug 25, 2016, at 12:43 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> 
> Aside: If pip is considered the only user of PyPI, I do wonder,
> why we bother having a user readable download index at all ;-)
> 


It?s not considered the *only* user, but it is the primary user of
the files that get uploaded to PyPI because, well it is the primary
thing downloading from PyPI by a significant margin. Here?s some
data for the last 30 days:

Total Downloads: 543,475,640

1.  pip:            346,682,962 (64%)
2.  bandersnatch:   112,405,165 (21%)
3.  null:           40,803,919  (8%)
4.  setuptools:     31,638,840  (6%)
5.  distribute:     6,661,875   (1%)
6.  Browser:        2,247,006   (0.4%)
7.  requests:       1,296,017   (0.3%)
8.  pex:            736,028     (0.2%)
9.  devpi:          357,957     (0.06%)
10. z3c.pypimirror: 294,985     (0.05%)
11. Artifactory:    169,241     (0.03%)
12. OS:             103,323     (0.02%)
13. Homebrew:       42,252      (0.01%)
14. conda:          36,070      (0.01%)

The bandersnatch downloads also likely also represents largely pip
downloads (or zero downloads) and are very unlikely to represent
manual downloads. In here null is basically anything we couldn?t
figure out from our UA parsing what it was, generally that means
older pip?s where we used the default urllib2 user agent but it
could be people with custom urllib2 based scripts or something. Also
things like curl/wget and such get rolled into ?Browser?.

This data is all available for anyone to query in BigQuery if they
have a google account. One of the things I generally try to do is
make data based decisions with PyPI/pip/etc which is why I?ve put
a fair amount of effort into creating a data pipeline that we can
query and ask questions of instead of trying to guess. Of course,
no data is perfect so some amount of guessing is unavoidable, but
at least we can make educated guesses. In this, the data unequivocally
shows that automatic downloads through some sort of tooling is the
primary use case for files uploaded to PyPI, and thus decisions
should be made to optimize that particular use case.

Given all of that, the new PyPI design de-emphasizes the downloads
in the UI (without removing it) while trying to put a higher emphasis
on the links provided by the project itself (and of course, it?s
description). If we were to add some type of file where the primary
purpose was for users to manually download it, rather than some tool,
then we?d want to rework that so that we separated these two different
kinds of files, the ones aimed mostly at tooling vs the ones aimed
mostly at humans, and properly emphasis them.

?
Donald Stufft




From thomas at kluyver.me.uk  Fri Aug 26 03:08:40 2016
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Fri, 26 Aug 2016 08:08:40 +0100
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <CADiSq7fRoP73jw8v7bLqB+HT5tYnevvKiT6SsZUC--8KuVp+aA@mail.gmail.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
 <00B6BA52-2E9E-45E4-907F-1054D8AABEF0@stufft.io>
 <CADiSq7cbtdBpE__dnGfBa-3xRhSJBeo087xifo2vyU2qsPtvGA@mail.gmail.com>
 <1472125582.3670703.705766377.28264F24@webmail.messagingengine.com>
 <CADiSq7fRoP73jw8v7bLqB+HT5tYnevvKiT6SsZUC--8KuVp+aA@mail.gmail.com>
Message-ID: <1472195320.2011099.706680353.2CCB88A9@webmail.messagingengine.com>

On Thu, Aug 25, 2016, at 05:29 PM, Nick Coghlan wrote:
> Could you give a bit more detail on how you came to be publishing
> both? The main thing we're trying to avoid is missing a practical use
> case for the status quo where folks can upload both - if it's just an
> artifact of Windows and *nix having different default formats, then
> the convergence in distutils and setuptools will fix it implicitly,
> but if it's a deliberate design decision, then we need to check if
> that's based on a misunderstanding of how pip/easy_install/et al
> consume the two formats.

I think that script was created before my time in the project. I'd guess
it's just a historical artefact, but Fernando might know more.

Fernando: is there a reason we publish both .zip and .tar.gz sdists for
each release? PyPA is thinking of only allowing one sdist per release.

From ralf.gommers at gmail.com  Fri Aug 26 16:57:59 2016
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sat, 27 Aug 2016 08:57:59 +1200
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <1472195320.2011099.706680353.2CCB88A9@webmail.messagingengine.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
 <00B6BA52-2E9E-45E4-907F-1054D8AABEF0@stufft.io>
 <CADiSq7cbtdBpE__dnGfBa-3xRhSJBeo087xifo2vyU2qsPtvGA@mail.gmail.com>
 <1472125582.3670703.705766377.28264F24@webmail.messagingengine.com>
 <CADiSq7fRoP73jw8v7bLqB+HT5tYnevvKiT6SsZUC--8KuVp+aA@mail.gmail.com>
 <1472195320.2011099.706680353.2CCB88A9@webmail.messagingengine.com>
Message-ID: <CABL7CQgfOXku9NYFD4=jQkY9vTPTW27tGkhHGPLdPs7EjWbLYQ@mail.gmail.com>

On Fri, Aug 26, 2016 at 7:08 PM, Thomas Kluyver <thomas at kluyver.me.uk>
wrote:

> On Thu, Aug 25, 2016, at 05:29 PM, Nick Coghlan wrote:
> > Could you give a bit more detail on how you came to be publishing
> > both? The main thing we're trying to avoid is missing a practical use
> > case for the status quo where folks can upload both - if it's just an
> > artifact of Windows and *nix having different default formats, then
> > the convergence in distutils and setuptools will fix it implicitly,
> > but if it's a deliberate design decision, then we need to check if
> > that's based on a misunderstanding of how pip/easy_install/et al
> > consume the two formats.
>

For Numpy and Scipy we also publish both, that's just because Windows users
often prefer .zip. I don't see an issue with dropping one of the two.

Ralf


>
> I think that script was created before my time in the project. I'd guess
> it's just a historical artefact, but Fernando might know more.
>
> Fernando: is there a reason we publish both .zip and .tar.gz sdists for
> each release? PyPA is thinking of only allowing one sdist per release.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160827/821a7e4f/attachment.html>

From fperez at lbl.gov  Fri Aug 26 17:09:09 2016
From: fperez at lbl.gov (Fernando Perez)
Date: Fri, 26 Aug 2016 14:09:09 -0700
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <CABL7CQgfOXku9NYFD4=jQkY9vTPTW27tGkhHGPLdPs7EjWbLYQ@mail.gmail.com>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
 <00B6BA52-2E9E-45E4-907F-1054D8AABEF0@stufft.io>
 <CADiSq7cbtdBpE__dnGfBa-3xRhSJBeo087xifo2vyU2qsPtvGA@mail.gmail.com>
 <1472125582.3670703.705766377.28264F24@webmail.messagingengine.com>
 <CADiSq7fRoP73jw8v7bLqB+HT5tYnevvKiT6SsZUC--8KuVp+aA@mail.gmail.com>
 <1472195320.2011099.706680353.2CCB88A9@webmail.messagingengine.com>
 <CABL7CQgfOXku9NYFD4=jQkY9vTPTW27tGkhHGPLdPs7EjWbLYQ@mail.gmail.com>
Message-ID: <CAFFEDuk1DSuFTjembQANOgZtDWNj7VfkVE6Bkm1TR3G=Shj6MA@mail.gmail.com>

Same here, having both is purely historical artifact, dating back almost 15
years to the days of windows xp and somewhat awkward zip support on Linux.

No problem with modernizing this as needed.

Sent from a mobile device. .

On Aug 26, 2016 13:58, "Ralf Gommers" <ralf.gommers at gmail.com> wrote:

>
>
> On Fri, Aug 26, 2016 at 7:08 PM, Thomas Kluyver <thomas at kluyver.me.uk>
> wrote:
>
>> On Thu, Aug 25, 2016, at 05:29 PM, Nick Coghlan wrote:
>> > Could you give a bit more detail on how you came to be publishing
>> > both? The main thing we're trying to avoid is missing a practical use
>> > case for the status quo where folks can upload both - if it's just an
>> > artifact of Windows and *nix having different default formats, then
>> > the convergence in distutils and setuptools will fix it implicitly,
>> > but if it's a deliberate design decision, then we need to check if
>> > that's based on a misunderstanding of how pip/easy_install/et al
>> > consume the two formats.
>>
>
> For Numpy and Scipy we also publish both, that's just because Windows
> users often prefer .zip. I don't see an issue with dropping one of the two.
>
> Ralf
>
>
>>
>> I think that script was created before my time in the project. I'd guess
>> it's just a historical artefact, but Fernando might know more.
>>
>> Fernando: is there a reason we publish both .zip and .tar.gz sdists for
>> each release? PyPA is thinking of only allowing one sdist per release.
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160826/e77ed700/attachment.html>

From dholth at gmail.com  Sat Aug 27 20:31:10 2016
From: dholth at gmail.com (Daniel Holth)
Date: Sun, 28 Aug 2016 00:31:10 +0000
Subject: [Distutils] enscons 0.5.0 distutils/setuptools alternative
Message-ID: <CAG8k2+4aZ8c9LiotQ_jaTxO7VPjZBBK1_S3yD6XRZ+OkBhnCYA@mail.gmail.com>

I've just released version 0.5.0 of my enscons project, a SCons-powered
distutils alternative. Now with a bit of documentation and some examples.
If you are interested in not using distutils, please give it a try.

Thanks,

Daniel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160828/378cbc50/attachment.html>

From sylvain.corlay at gmail.com  Sun Aug 28 06:34:17 2016
From: sylvain.corlay at gmail.com (Sylvain Corlay)
Date: Sun, 28 Aug 2016 12:34:17 +0200
Subject: [Distutils] Proposed new Distutils API for compiler flag
 detection (Issue26689)
In-Reply-To: <CABL7CQjWvNogHV39YMxTpv4wfVAEOo+ZC4LcPnynArVi5hVDqg@mail.gmail.com>
References: <A45B9ADE-2FB4-4453-AACE-8B1595C009E3@python.org>
 <CABL7CQi86ORbhQUy11g4ipiEWMKFYtzeeYrb1JkKomsVPa8uKQ@mail.gmail.com>
 <CAK=Phk5YV_Ax34_DRV9b4ch4T7GNMZ2=E4DH=QP0DzeAA8v83A@mail.gmail.com>
 <CAPJVwBnOu6i3f-_XspZtaMR8Va68suHVb9p9HiVQ6Cp8-CFFTQ@mail.gmail.com>
 <CAK=Phk4_XgZu_z_qu_mHFKE96V2c-dJHdixJKtrckBjQ2L88Tw@mail.gmail.com>
 <34793887-E8A4-4694-ADD2-04F45F789651@stufft.io>
 <CAK=Phk4YRKgbH7Qxy2j2smuHpqGEpaNFgFtzEoHtH4qZaz92qw@mail.gmail.com>
 <63D859EB-B20C-4413-ACF4-93EB22C7A7FD@stufft.io>
 <CABL7CQgggpKpO5UG1vXfDFo1eR3ccyNRn7AcVt7OB=YKey+CCA@mail.gmail.com>
 <CAK=Phk6Qe=UWvKH-d2UbxJG9QSnDjHURwfz0rNPZ1GsxoYTr4A@mail.gmail.com>
 <1471848656.3689082.702076721.53722769@webmail.messagingengine.com>
 <CABL7CQjWvNogHV39YMxTpv4wfVAEOo+ZC4LcPnynArVi5hVDqg@mail.gmail.com>
Message-ID: <CAK=Phk7By+hftXRODUtvJyduuSxcrOePh50M_ov6isFo_21N1A@mail.gmail.com>

Hi All,

The beta deadline for new features is approaching dangerously.

I agree with Thomas that being able to require Python 3.6 for a project
does not appear so distant for me (as soon as it is a Python 3 project
only).

Any chance to get this through? Checking support for language features will
be more and more required since new version of the C++ standard are
becoming more frequent. I understand that it is not an issue for a project
like numpy, but this is a check I do in every single one of my extension
projects.

Thanks,

Sylvain

On Thu, Aug 25, 2016 at 6:41 AM, Ralf Gommers <ralf.gommers at gmail.com>
wrote:

>
> On Mon, Aug 22, 2016 at 6:50 PM, Thomas Kluyver <thomas at kluyver.me.uk>
> wrote:
>
>> There's obviously some cost in code duplication; I haven't looked at the
>> code in question, so I don't know how bad this is.
>>
>
> This patch is pretty short and understandable, so not bad.
>
>
>> I've run into this argument before when trying to change things in
>> non-packaging-related parts of the stdlib, and I agree with Sylvain that
>> it's fundamentally problematic. If we're trying to improve the stdlib,
>> we're obviously taking a long view, but that's how we ensure the stdlib is
>> still useful in a few years time. This goes for packaging tools as much as
>> anything else.
>>
>
> This I don't agree with - packaging is fundamentally different for the
> reasons Donald gave.
>
> Ralf
>
>
>> I already have projects where I'm happy to require Python >=3.4, so being
>> able to depend on Python 3.6 is not such a distant prospect.
>>
>> Thomas
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160828/084a1ce1/attachment.html>

From steve.dower at python.org  Sun Aug 28 10:34:20 2016
From: steve.dower at python.org (Steve Dower)
Date: Sun, 28 Aug 2016 07:34:20 -0700
Subject: [Distutils] Proposed new Distutils API for compiler
 flagdetection (Issue26689)
In-Reply-To: <CAK=Phk7By+hftXRODUtvJyduuSxcrOePh50M_ov6isFo_21N1A@mail.gmail.com>
References: <A45B9ADE-2FB4-4453-AACE-8B1595C009E3@python.org>
 <CABL7CQi86ORbhQUy11g4ipiEWMKFYtzeeYrb1JkKomsVPa8uKQ@mail.gmail.com>
 <CAK=Phk5YV_Ax34_DRV9b4ch4T7GNMZ2=E4DH=QP0DzeAA8v83A@mail.gmail.com>
 <CAPJVwBnOu6i3f-_XspZtaMR8Va68suHVb9p9HiVQ6Cp8-CFFTQ@mail.gmail.com>
 <CAK=Phk4_XgZu_z_qu_mHFKE96V2c-dJHdixJKtrckBjQ2L88Tw@mail.gmail.com>
 <34793887-E8A4-4694-ADD2-04F45F789651@stufft.io>
 <CAK=Phk4YRKgbH7Qxy2j2smuHpqGEpaNFgFtzEoHtH4qZaz92qw@mail.gmail.com>
 <63D859EB-B20C-4413-ACF4-93EB22C7A7FD@stufft.io>
 <CABL7CQgggpKpO5UG1vXfDFo1eR3ccyNRn7AcVt7OB=YKey+CCA@mail.gmail.com>
 <CAK=Phk6Qe=UWvKH-d2UbxJG9QSnDjHURwfz0rNPZ1GsxoYTr4A@mail.gmail.com>
 <1471848656.3689082.702076721.53722769@webmail.messagingengine.com>
 <CABL7CQjWvNogHV39YMxTpv4wfVAEOo+ZC4LcPnynArVi5hVDqg@mail.gmail.com>
 <CAK=Phk7By+hftXRODUtvJyduuSxcrOePh50M_ov6isFo_21N1A@mail.gmail.com>
Message-ID: <E1be27C-0005op-Tw@se2-syd.hostedmail.net.au>

Some of the core development team will be sprinting full time for the week leading up to beta 1, so expect a lot of things to get added then.

My main concern with this is compatibility rather than the interface, but as a new feature I think it's best to be only in setuptools. Distutils is minimal support for building and as long as it works I don't see any reason to change it given the ease of getting alternatives.

Cheers,
Steve

Top-posted from my Windows Phone

-----Original Message-----
From: "Sylvain Corlay" <sylvain.corlay at gmail.com>
Sent: ?8/?28/?2016 3:35
To: "Ralf Gommers" <ralf.gommers at gmail.com>
Cc: "DistUtils mailing list" <distutils-sig at python.org>
Subject: Re: [Distutils] Proposed new Distutils API for compiler flagdetection (Issue26689)

Hi All,


The beta deadline for new features is approaching dangerously.


I agree with Thomas that being able to require Python 3.6 for a project does not appear so distant for me (as soon as it is a Python 3 project only).


Any chance to get this through? Checking support for language features will be more and more required since new version of the C++ standard are becoming more frequent. I understand that it is not an issue for a project like numpy, but this is a check I do in every single one of my extension projects.

Thanks,


Sylvain



On Thu, Aug 25, 2016 at 6:41 AM, Ralf Gommers <ralf.gommers at gmail.com> wrote:



On Mon, Aug 22, 2016 at 6:50 PM, Thomas Kluyver <thomas at kluyver.me.uk> wrote:

There's obviously some cost in code duplication; I haven't looked at the code in question, so I don't know how bad this is.



This patch is pretty short and understandable, so not bad.




I've run into this argument before when trying to change things in non-packaging-related parts of the stdlib, and I agree with Sylvain that it's fundamentally problematic. If we're trying to improve the stdlib, we're obviously taking a long view, but that's how we ensure the stdlib is still useful in a few years time. This goes for packaging tools as much as anything else.



This I don't agree with - packaging is fundamentally different for the reasons Donald gave.


Ralf




I already have projects where I'm happy to require Python >=3.4, so being able to depend on Python 3.6 is not such a distant prospect.



Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160828/5e884852/attachment-0001.html>

From sylvain.corlay at gmail.com  Sun Aug 28 11:44:36 2016
From: sylvain.corlay at gmail.com (Sylvain Corlay)
Date: Sun, 28 Aug 2016 17:44:36 +0200
Subject: [Distutils] Proposed new Distutils API for compiler
 flagdetection (Issue26689)
In-Reply-To: <E1be27C-0005oo-Je@se2-syd.hostedmail.net.au>
References: <A45B9ADE-2FB4-4453-AACE-8B1595C009E3@python.org>
 <CABL7CQi86ORbhQUy11g4ipiEWMKFYtzeeYrb1JkKomsVPa8uKQ@mail.gmail.com>
 <CAK=Phk5YV_Ax34_DRV9b4ch4T7GNMZ2=E4DH=QP0DzeAA8v83A@mail.gmail.com>
 <CAPJVwBnOu6i3f-_XspZtaMR8Va68suHVb9p9HiVQ6Cp8-CFFTQ@mail.gmail.com>
 <CAK=Phk4_XgZu_z_qu_mHFKE96V2c-dJHdixJKtrckBjQ2L88Tw@mail.gmail.com>
 <34793887-E8A4-4694-ADD2-04F45F789651@stufft.io>
 <CAK=Phk4YRKgbH7Qxy2j2smuHpqGEpaNFgFtzEoHtH4qZaz92qw@mail.gmail.com>
 <63D859EB-B20C-4413-ACF4-93EB22C7A7FD@stufft.io>
 <CABL7CQgggpKpO5UG1vXfDFo1eR3ccyNRn7AcVt7OB=YKey+CCA@mail.gmail.com>
 <CAK=Phk6Qe=UWvKH-d2UbxJG9QSnDjHURwfz0rNPZ1GsxoYTr4A@mail.gmail.com>
 <1471848656.3689082.702076721.53722769@webmail.messagingengine.com>
 <CABL7CQjWvNogHV39YMxTpv4wfVAEOo+ZC4LcPnynArVi5hVDqg@mail.gmail.com>
 <CAK=Phk7By+hftXRODUtvJyduuSxcrOePh50M_ov6isFo_21N1A@mail.gmail.com>
 <E1be27C-0005oo-Je@se2-syd.hostedmail.net.au>
Message-ID: <CAK=Phk6xLoctkemStM6Lr_3_mbgKYrNCy2o05xk+yV9cmDxmAA@mail.gmail.com>

Regarding compatibility, I have been using this on a variety of platforms
and compilers for some time already.

I think that monkey-patching distutils.ccompiler from setuptools is a dirty
solution and should only be provided as a patch for earlier versions of
python. At the moment, setuptools does little on ccompiler except for a
specific msvc patch regarding locating msvcvarsall.bat.

I understand that it is not a concern with C, but this feature becomes
crucial when building extensions with C++ since new flavors of the language
are emerging every couple of years now. It becomes important to be able to
check and output meaningful error messages when the compiler does not
support C++[11/14/17] features if they are needed for an extension.

Sylvain



On Sun, Aug 28, 2016 at 4:34 PM, Steve Dower <steve.dower at python.org> wrote:

> Some of the core development team will be sprinting full time for the week
> leading up to beta 1, so expect a lot of things to get added then.
>
> My main concern with this is compatibility rather than the interface, but
> as a new feature I think it's best to be only in setuptools. Distutils is
> minimal support for building and as long as it works I don't see any reason
> to change it given the ease of getting alternatives.
>
> Cheers,
> Steve
>
> Top-posted from my Windows Phone
> ------------------------------
> From: Sylvain Corlay <sylvain.corlay at gmail.com>
> Sent: ?8/?28/?2016 3:35
> To: Ralf Gommers <ralf.gommers at gmail.com>
> Cc: DistUtils mailing list <distutils-sig at python.org>
> Subject: Re: [Distutils] Proposed new Distutils API for compiler
> flagdetection (Issue26689)
>
> Hi All,
>
> The beta deadline for new features is approaching dangerously.
>
> I agree with Thomas that being able to require Python 3.6 for a project
> does not appear so distant for me (as soon as it is a Python 3 project
> only).
>
> Any chance to get this through? Checking support for language features
> will be more and more required since new version of the C++ standard are
> becoming more frequent. I understand that it is not an issue for a project
> like numpy, but this is a check I do in every single one of my extension
> projects.
>
> Thanks,
>
> Sylvain
>
> On Thu, Aug 25, 2016 at 6:41 AM, Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
>
>>
>> On Mon, Aug 22, 2016 at 6:50 PM, Thomas Kluyver <thomas at kluyver.me.uk>
>> wrote:
>>
>>> There's obviously some cost in code duplication; I haven't looked at the
>>> code in question, so I don't know how bad this is.
>>>
>>
>> This patch is pretty short and understandable, so not bad.
>>
>>
>>> I've run into this argument before when trying to change things in
>>> non-packaging-related parts of the stdlib, and I agree with Sylvain that
>>> it's fundamentally problematic. If we're trying to improve the stdlib,
>>> we're obviously taking a long view, but that's how we ensure the stdlib is
>>> still useful in a few years time. This goes for packaging tools as much as
>>> anything else.
>>>
>>
>> This I don't agree with - packaging is fundamentally different for the
>> reasons Donald gave.
>>
>> Ralf
>>
>>
>>> I already have projects where I'm happy to require Python >=3.4, so
>>> being able to depend on Python 3.6 is not such a distant prospect.
>>>
>>> Thomas
>>>
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160828/9c0fdcd7/attachment.html>

From ncoghlan at gmail.com  Sun Aug 28 12:18:56 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 29 Aug 2016 02:18:56 +1000
Subject: [Distutils] Proposed new Distutils API for compiler
 flagdetection (Issue26689)
In-Reply-To: <CAK=Phk6xLoctkemStM6Lr_3_mbgKYrNCy2o05xk+yV9cmDxmAA@mail.gmail.com>
References: <A45B9ADE-2FB4-4453-AACE-8B1595C009E3@python.org>
 <CABL7CQi86ORbhQUy11g4ipiEWMKFYtzeeYrb1JkKomsVPa8uKQ@mail.gmail.com>
 <CAK=Phk5YV_Ax34_DRV9b4ch4T7GNMZ2=E4DH=QP0DzeAA8v83A@mail.gmail.com>
 <CAPJVwBnOu6i3f-_XspZtaMR8Va68suHVb9p9HiVQ6Cp8-CFFTQ@mail.gmail.com>
 <CAK=Phk4_XgZu_z_qu_mHFKE96V2c-dJHdixJKtrckBjQ2L88Tw@mail.gmail.com>
 <34793887-E8A4-4694-ADD2-04F45F789651@stufft.io>
 <CAK=Phk4YRKgbH7Qxy2j2smuHpqGEpaNFgFtzEoHtH4qZaz92qw@mail.gmail.com>
 <63D859EB-B20C-4413-ACF4-93EB22C7A7FD@stufft.io>
 <CABL7CQgggpKpO5UG1vXfDFo1eR3ccyNRn7AcVt7OB=YKey+CCA@mail.gmail.com>
 <CAK=Phk6Qe=UWvKH-d2UbxJG9QSnDjHURwfz0rNPZ1GsxoYTr4A@mail.gmail.com>
 <1471848656.3689082.702076721.53722769@webmail.messagingengine.com>
 <CABL7CQjWvNogHV39YMxTpv4wfVAEOo+ZC4LcPnynArVi5hVDqg@mail.gmail.com>
 <CAK=Phk7By+hftXRODUtvJyduuSxcrOePh50M_ov6isFo_21N1A@mail.gmail.com>
 <E1be27C-0005oo-Je@se2-syd.hostedmail.net.au>
 <CAK=Phk6xLoctkemStM6Lr_3_mbgKYrNCy2o05xk+yV9cmDxmAA@mail.gmail.com>
Message-ID: <CADiSq7dKSA3mAMDV+RPgMQKyEqEuYouVL-TCEEfvziyxOHgrhQ@mail.gmail.com>

On 29 August 2016 at 01:44, Sylvain Corlay <sylvain.corlay at gmail.com> wrote:
> Regarding compatibility, I have been using this on a variety of platforms
> and compilers for some time already.
>
> I think that monkey-patching distutils.ccompiler from setuptools is a dirty
> solution and should only be provided as a patch for earlier versions of
> python. At the moment, setuptools does little on ccompiler except for a
> specific msvc patch regarding locating msvcvarsall.bat.
>
> I understand that it is not a concern with C, but this feature becomes
> crucial when building extensions with C++ since new flavors of the language
> are emerging every couple of years now. It becomes important to be able to
> check and output meaningful error messages when the compiler does not
> support C++[11/14/17] features if they are needed for an extension.

+1 from me - we don't have compiler version requirements in
declarative metadata (and are highly unlikely to ever do so), so it
makes sense to make it as easy as possible for extension module
authors to make their build scripts fail at least somewhat gracefully.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From brett at python.org  Sun Aug 28 13:05:17 2016
From: brett at python.org (Brett Cannon)
Date: Sun, 28 Aug 2016 17:05:17 +0000
Subject: [Distutils] What is the official position on distutils?
Message-ID: <CAP1=2W4ZXcC5CO5oqKcrsEhYjRvrYqe7o1mnKnOJ5+sCkh9k8w@mail.gmail.com>

The discussion of Sylvain's proposed changes to distutils suggests that
there isn't a clear-cut agreement or position of this SIG -- and thus
Python -- on changes to distutils, its future, etc. Is there an official
position I'm not aware of? If not, could we get one so we know how to
handle any more proposed changes to distutils?

For me personally (and to start this conversation if necessary), I view
distutils as the package in the stdlib that handles building extension
modules in the stdlib for Python. That view means that if something doesn't
service that explicit goal then it shouldn't go into distutils. Now that we
are moving down the road towards making the build step configurable I'm
fine with saying that distutils is not expected to work for people other
than Python itself and others can use setuptools, enscons, or other
projects while we continue to improve the situation to where the build
system is just something you specify in pyproject.toml to generate your
wheel. IOW distutils is only exposed publicly because Python doesn't hide
anything, but making it useful for the general case of direct usage by
people is a non-goal of the package.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160828/81d44b62/attachment.html>

From robertc at robertcollins.net  Sun Aug 28 15:43:11 2016
From: robertc at robertcollins.net (Robert Collins)
Date: Mon, 29 Aug 2016 07:43:11 +1200
Subject: [Distutils] What is the official position on distutils?
In-Reply-To: <CAP1=2W4ZXcC5CO5oqKcrsEhYjRvrYqe7o1mnKnOJ5+sCkh9k8w@mail.gmail.com>
References: <CAP1=2W4ZXcC5CO5oqKcrsEhYjRvrYqe7o1mnKnOJ5+sCkh9k8w@mail.gmail.com>
Message-ID: <CAJ3HoZ0her_jptgiqFwR7OHXSa+FZx1QNk7BD0TN7p3g94FhkA@mail.gmail.com>

My sense of the current position is:

 - distutils exists for both Python and the broader ecosystem to build
both extension modules and package distributions
 - we recognise that its able to do anything, but many folk find it
hard to work with - or are already invested in other build tools
 - we're afraid of changing it in ways that cause incompatibilities
with e.g. the numpy eco systems heavy customisations, and we're not
sure we can recognise whether a given change is one of those
 - and we're recommending that everyone move away from directly using
distutils to using setuptoools - but setuptools *is* distutils in many
many ways, so this doesn't loosen our dependency and stability
considerations around distutils

So basically the normal 'module goes to the stdlib to freeze' effect.

I haven't read about Sylvain's proposed changes yet - ETHINGSTODO :(.

I can happily agree with a position that making distutils directly
usable is a non-goal... but insofar as making it more usable makes
setuptools more usable, that would be a goal :/.

-Rob




On 29 August 2016 at 05:05, Brett Cannon <brett at python.org> wrote:
> The discussion of Sylvain's proposed changes to distutils suggests that
> there isn't a clear-cut agreement or position of this SIG -- and thus Python
> -- on changes to distutils, its future, etc. Is there an official position
> I'm not aware of? If not, could we get one so we know how to handle any more
> proposed changes to distutils?
>
> For me personally (and to start this conversation if necessary), I view
> distutils as the package in the stdlib that handles building extension
> modules in the stdlib for Python. That view means that if something doesn't
> service that explicit goal then it shouldn't go into distutils. Now that we
> are moving down the road towards making the build step configurable I'm fine
> with saying that distutils is not expected to work for people other than
> Python itself and others can use setuptools, enscons, or other projects
> while we continue to improve the situation to where the build system is just
> something you specify in pyproject.toml to generate your wheel. IOW
> distutils is only exposed publicly because Python doesn't hide anything, but
> making it useful for the general case of direct usage by people is a
> non-goal of the package.
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>

From p.f.moore at gmail.com  Sun Aug 28 18:36:43 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 28 Aug 2016 23:36:43 +0100
Subject: [Distutils] What is the official position on distutils?
In-Reply-To: <CAP1=2W4ZXcC5CO5oqKcrsEhYjRvrYqe7o1mnKnOJ5+sCkh9k8w@mail.gmail.com>
References: <CAP1=2W4ZXcC5CO5oqKcrsEhYjRvrYqe7o1mnKnOJ5+sCkh9k8w@mail.gmail.com>
Message-ID: <CACac1F8NXxGbKCPT8X1cviC1BJQ8MsPSZ4zPsPv6XCeEn3QyMQ@mail.gmail.com>

On 28 August 2016 at 18:05, Brett Cannon <brett at python.org> wrote:
> The discussion of Sylvain's proposed changes to distutils suggests that
> there isn't a clear-cut agreement or position of this SIG -- and thus Python
> -- on changes to distutils, its future, etc. Is there an official position
> I'm not aware of? If not, could we get one so we know how to handle any more
> proposed changes to distutils?
>
> For me personally (and to start this conversation if necessary), I view
> distutils as the package in the stdlib that handles building extension
> modules in the stdlib for Python. That view means that if something doesn't
> service that explicit goal then it shouldn't go into distutils. Now that we
> are moving down the road towards making the build step configurable I'm fine
> with saying that distutils is not expected to work for people other than
> Python itself and others can use setuptools, enscons, or other projects
> while we continue to improve the situation to where the build system is just
> something you specify in pyproject.toml to generate your wheel. IOW
> distutils is only exposed publicly because Python doesn't hide anything, but
> making it useful for the general case of direct usage by people is a
> non-goal of the package.

FWIW, my view is:

* distutils handles building of Python packages, both in the stdlib and outside
* in practice, setuptools is almost always used with distutils, and
any proposed change to distutils could be made to setuptools just as
easily
* patches for distutils typically suffer from a lack of maintainers
motivated to work on them
* there are a few developers with a strong investment in distutils,
who have code that is deeply entwined with the internals of distutils,
and changes to distutils come with a high risk of causing issues for
those people
* these developers *don't* use setuptools, so changes in setuptools
don't tend to suffer from the same level of backward compatibility
constraints
* implementing changes in setuptools is *far* easier than doing so in
distutils, for the above reasons
* changes in setuptools can be used with any version of Python,
distutils changes are only available to people with no need to support
anything but the latest version (which in practice is a vanishingly
small group)
* as a result, there would have to be an *extremely* compelling reason
to make a change in distutils rather than in setuptools - sufficient
to justify the additional risk, the extra developer effort needed, and
the fact that any such change is only going to benefit users of newer
versions of Python

The reality is simply that distutils is in a somewhat unhappy place,
with too few interested maintainers and serious compatibility problems
to deal with, but that's not exactly news to anyone. I don't think
that a formal statement will help much. Past history has shown that
making pronouncements like "distutils is frozen" has not helped anyone
- so we're just trying to work with the status quo.

With regard to Sylvain's proposed changes, I haven't really seen any
arguments justifying making the change in distutils rather than in
setuptools - the last suggestion I saw was to make the change in
*both* places, but I don't see any particular reason (beyond abstract
"it makes more sense to have the change in distutils" arguments) for
not just targeting setuptools. I may be missing something (as I'm not
personally in a position where I'd be comfortable reviewing the
change) but it seems to me that it's a bit of a non-issue - just
implement in setuptools and move on. The rhetoric about "this means
that distutils is dead" doesn't feel particularly helpful.

Paul

From sylvain.corlay at gmail.com  Sun Aug 28 19:55:09 2016
From: sylvain.corlay at gmail.com (Sylvain Corlay)
Date: Mon, 29 Aug 2016 01:55:09 +0200
Subject: [Distutils] What is the official position on distutils?
In-Reply-To: <CACac1F8NXxGbKCPT8X1cviC1BJQ8MsPSZ4zPsPv6XCeEn3QyMQ@mail.gmail.com>
References: <CAP1=2W4ZXcC5CO5oqKcrsEhYjRvrYqe7o1mnKnOJ5+sCkh9k8w@mail.gmail.com>
 <CACac1F8NXxGbKCPT8X1cviC1BJQ8MsPSZ4zPsPv6XCeEn3QyMQ@mail.gmail.com>
Message-ID: <CAK=Phk4XYgn45_2vJqzGciZJKFx4aAdGzMqZQ7o3f3bPHQLH4A@mail.gmail.com>

Distutils seems to be the de-facto standard for building extension modules
for Python and it is used for most of the third-party extensions out there.
I don?t think that it is reasonable to declare that it is now only meant
for Python itself. I actually see a contradiction in pointing out some lack
of involvement in the library to oppose well needed improvements,
especially for a tool that has so much adoption.

As we discussed earlier, even though it is not a concern with C, checking
for availability of a compiler flag becomes crucial when building
extensions with C++ since new flavors of the language emerge every couple
of years now. It is important to be able to output meaningful error
messages when the compiler does not support C++[11/14/17] features if they
are needed for a given extension. This is a new aspect of the landscape in
this area.

Finally, adding this method is a very straightforward change. `has_flag`
simply comes aside `has_function` as a method of ccompiler. I don't see a
more natural place for it. It would be a very weird design decision in my
opinion to not add it there, and instead to add it to distutils.ccompiler
by monkeypatching it in setuptools.

Thanks,

Sylvain

On Mon, Aug 29, 2016 at 12:36 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 28 August 2016 at 18:05, Brett Cannon <brett at python.org> wrote:
> > The discussion of Sylvain's proposed changes to distutils suggests that
> > there isn't a clear-cut agreement or position of this SIG -- and thus
> Python
> > -- on changes to distutils, its future, etc. Is there an official
> position
> > I'm not aware of? If not, could we get one so we know how to handle any
> more
> > proposed changes to distutils?
> >
> > For me personally (and to start this conversation if necessary), I view
> > distutils as the package in the stdlib that handles building extension
> > modules in the stdlib for Python. That view means that if something
> doesn't
> > service that explicit goal then it shouldn't go into distutils. Now that
> we
> > are moving down the road towards making the build step configurable I'm
> fine
> > with saying that distutils is not expected to work for people other than
> > Python itself and others can use setuptools, enscons, or other projects
> > while we continue to improve the situation to where the build system is
> just
> > something you specify in pyproject.toml to generate your wheel. IOW
> > distutils is only exposed publicly because Python doesn't hide anything,
> but
> > making it useful for the general case of direct usage by people is a
> > non-goal of the package.
>
> FWIW, my view is:
>
> * distutils handles building of Python packages, both in the stdlib and
> outside
> * in practice, setuptools is almost always used with distutils, and
> any proposed change to distutils could be made to setuptools just as
> easily
> * patches for distutils typically suffer from a lack of maintainers
> motivated to work on them
> * there are a few developers with a strong investment in distutils,
> who have code that is deeply entwined with the internals of distutils,
> and changes to distutils come with a high risk of causing issues for
> those people
> * these developers *don't* use setuptools, so changes in setuptools
> don't tend to suffer from the same level of backward compatibility
> constraints
> * implementing changes in setuptools is *far* easier than doing so in
> distutils, for the above reasons
> * changes in setuptools can be used with any version of Python,
> distutils changes are only available to people with no need to support
> anything but the latest version (which in practice is a vanishingly
> small group)
> * as a result, there would have to be an *extremely* compelling reason
> to make a change in distutils rather than in setuptools - sufficient
> to justify the additional risk, the extra developer effort needed, and
> the fact that any such change is only going to benefit users of newer
> versions of Python
>
> The reality is simply that distutils is in a somewhat unhappy place,
> with too few interested maintainers and serious compatibility problems
> to deal with, but that's not exactly news to anyone. I don't think
> that a formal statement will help much. Past history has shown that
> making pronouncements like "distutils is frozen" has not helped anyone
> - so we're just trying to work with the status quo.
>
> With regard to Sylvain's proposed changes, I haven't really seen any
> arguments justifying making the change in distutils rather than in
> setuptools - the last suggestion I saw was to make the change in
> *both* places, but I don't see any particular reason (beyond abstract
> "it makes more sense to have the change in distutils" arguments) for
> not just targeting setuptools. I may be missing something (as I'm not
> personally in a position where I'd be comfortable reviewing the
> change) but it seems to me that it's a bit of a non-issue - just
> implement in setuptools and move on. The rhetoric about "this means
> that distutils is dead" doesn't feel particularly helpful.
>
> Paul
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160829/fe619bce/attachment.html>

From ncoghlan at gmail.com  Sun Aug 28 22:28:51 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 29 Aug 2016 12:28:51 +1000
Subject: [Distutils] What is the official position on distutils?
In-Reply-To: <CAP1=2W4ZXcC5CO5oqKcrsEhYjRvrYqe7o1mnKnOJ5+sCkh9k8w@mail.gmail.com>
References: <CAP1=2W4ZXcC5CO5oqKcrsEhYjRvrYqe7o1mnKnOJ5+sCkh9k8w@mail.gmail.com>
Message-ID: <CADiSq7fTtAM4V3Jv4qcM5D=9c0vs+ASh8Jt6zucGF81pn3Lt2w@mail.gmail.com>

On 29 August 2016 at 03:05, Brett Cannon <brett at python.org> wrote:
> The discussion of Sylvain's proposed changes to distutils suggests that
> there isn't a clear-cut agreement or position of this SIG -- and thus Python
> -- on changes to distutils, its future, etc. Is there an official position
> I'm not aware of? If not, could we get one so we know how to handle any more
> proposed changes to distutils?

I agree it's not clear, and we should be more explicit about it.

> For me personally (and to start this conversation if necessary), I view
> distutils as the package in the stdlib that handles building extension
> modules in the stdlib for Python. That view means that if something doesn't
> service that explicit goal then it shouldn't go into distutils.

I think that's a reasonable place to draw the line for larger features
and major refactorings, especially with pip injecting setuptools for
"pip wheel" and "pip install", even if the setup.py itself only
imports distutils.

However, in cases where distutils *does* provide a particular piece of
functionality (e.g. the recent switch to emitting sdists as "tar.gz"
by default, regardless of platform), then we should maintain a
"distutils first" policy, make the change to distutils for future
versions of Python, and then backport it to earlier versions via
setuptools.

The grey area is cases like the one Sylvain is proposing, where it's a
straightforward API addition to distutils with no backwards
compatibility implications, and where we (as in distutils-sig/PyPA)
want to encourage people to write build scripts a certain way (in this
case, failing with a useful error message when particular compiler
features aren't available, rather than confronting end users with a
cryptic C/C++ compiler error).

For these cases, I think the value of including the change in
distutils isn't so much a practical software development one, as it is
an advertising opportunity: the What's New for 3.6 can not only
mention the addition and recommendation for use, it can point people
to setuptools as a version independent way of accessing the same
capability. There are still a *lot* of people out there for whom the
standard library *is* Python, and the feature releases are one of our
best opportunities to reach those folks with notifications that the
world has changed around them, and they may need to adapt their
development practices accordingly.

The backwards compatibility situation for CPython feature releases
itself also isn't as dire as it used to be - there are only a few
tightly coupled distutils extensions (with setuptools and
numpy.distutils being the most high profile ones), and it's rare these
days for us to get through a beta period without finding and resolving
any problems that come up (it's also worth remembering that the
original major problem wasn't with a feature release, it was with a
setuptools-breaking internal refactoring that landed in a maintenance
release, so there was no beta period to find and fix compatibility
problems).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From nad at python.org  Sun Aug 28 23:12:13 2016
From: nad at python.org (Ned Deily)
Date: Sun, 28 Aug 2016 23:12:13 -0400
Subject: [Distutils] What is the official position on distutils?
In-Reply-To: <CADiSq7fTtAM4V3Jv4qcM5D=9c0vs+ASh8Jt6zucGF81pn3Lt2w@mail.gmail.com>
References: <CAP1=2W4ZXcC5CO5oqKcrsEhYjRvrYqe7o1mnKnOJ5+sCkh9k8w@mail.gmail.com>
 <CADiSq7fTtAM4V3Jv4qcM5D=9c0vs+ASh8Jt6zucGF81pn3Lt2w@mail.gmail.com>
Message-ID: <C074A35F-BF6D-4D1B-857F-56AC101E7C61@python.org>

On Aug 28, 2016, at 22:28, Nick Coghlan <ncoghlan at gmail.com> wrote:
[...]
> The grey area is cases like the one Sylvain is proposing, where it's a
> straightforward API addition to distutils with no backwards
> compatibility implications, and where we (as in distutils-sig/PyPA)
> want to encourage people to write build scripts a certain way (in this
> case, failing with a useful error message when particular compiler
> features aren't available, rather than confronting end users with a
> cryptic C/C++ compiler error).
> 
> For these cases, I think the value of including the change in
> distutils isn't so much a practical software development one, as it is
> an advertising opportunity: the What's New for 3.6 can not only
> mention the addition and recommendation for use, it can point people
> to setuptools as a version independent way of accessing the same
> capability. There are still a *lot* of people out there for whom the
> standard library *is* Python, and the feature releases are one of our
> best opportunities to reach those folks with notifications that the
> world has changed around them, and they may need to adapt their
> development practices accordingly.

I rebooted the discussion of Sylvain's http://bugs.python.org/issue26689 here because I think decisions about Distutils API changes rightly fall under the purview of the PyPA / Distutils SIG.  With my 3.6 release manager hat on, I'd like to make sure there is agreement here before accepting this change for the release.  So, my understanding is that an acceptable course of action, as outlined by Nick, would be to see the "has_flag" method supplied by Distutils for 3.6 *and* by setuptools for use in older releases.  If so, I think the following things need to happen:

1. Someone from the PyPA (Nick?) should formally review and approve Sylvain's patch on the bug tracker.

2. Someone needs to open a setuptools issue and get agreement on the setuptools changes.

3. Someone needs to write the doc changes for 3.6, e.g. the What's New "advertising" and the changes to the Distutils documentation "Distributing Python Modules (Legacy version)" (Doc/distuils).  Anywhere else?

Is that a fair summary?

Thanks for everyone's input on this!
--Ned

--
  Ned Deily
  nad at python.org -- []


From m.van.rees at zestsoftware.nl  Mon Aug 29 04:45:51 2016
From: m.van.rees at zestsoftware.nl (Maurits van Rees)
Date: Mon, 29 Aug 2016 10:45:51 +0200
Subject: [Distutils] PEP 527 - Removing Un(der)used file
 types/extensions on PyPI
In-Reply-To: <7176BF59-16B6-4F17-A7E4-626B5CAE13AD@stufft.io>
References: <D4CE73D5-0165-47B3-BB33-15BD89590ADF@stufft.io>
 <CADiSq7esu6t-jgxgoP1eoaJzsYWR=i+71=JCM6Uw4X43Ux-=EA@mail.gmail.com>
 <00B6BA52-2E9E-45E4-907F-1054D8AABEF0@stufft.io>
 <CADiSq7cbtdBpE__dnGfBa-3xRhSJBeo087xifo2vyU2qsPtvGA@mail.gmail.com>
 <7176BF59-16B6-4F17-A7E4-626B5CAE13AD@stufft.io>
Message-ID: <nq0snt$e9h$1@blaine.gmane.org>

Op 25/08/16 om 13:57 schreef Donald Stufft:
>
>> On Aug 25, 2016, at 7:30 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> On 25 August 2016 at 06:48, Donald Stufft <donald at stufft.io> wrote:
>>> Ok, I?ve updated the PEP. You can see a diff at https://github.com/python/peps/pull/74/files or the entire PEP at https://www.python.org/dev/peps/pep-0527/ once that updated. However the primary differences are:
>>>
>>> * Added Nick Coghlan as the BDFL Delegate (Thanks Nick!).
>>> * Expanded upon rationale for dropping everything but sdist/wheel/egg.
>>> * Allow an sdist to be either .tar.gz or .zip, but still drop all other formats.
>>>
>>> Hopefully that can adequately address Marc-Andre?s concerns for the removal of the other file types, and the inclusion of .zip will address other folk?s concerns about losing .zip.
>>
>> Thanks for those updates.
>>
>> My one remaining question would be whether or not we have any projects
>> that are in the habit of uploading both .zip and .tar.gz for the same
>> release - if there are, the restriction to only one sdist per release
>> will need to be included in the notification message to maintainers.
>> (The introduction of that restriction could probably be clearer in the
>> PEP text as well - previously it was implied by the single extension,
>> but it may need to be called out more explicitly now that there are
>> two permitted options).
>
> My querying shows that we have a total of 359 projects that have ever
> uploaded two sdists for a single release. Looking at the number of
> releases that have 2+ sdists per project, it appears like there are a
> handful that do it regularly, and then the vast bulk have only done it
> once or twice (see: https://bpaste.net/show/fdfce63673ec).

I miss setuptools in this list.  For the latest 26.0.0 there is a wheel, 
a tar.gz and a zip, and that has been the case for a while.

https://bootstrap.pypa.io/ez_setup.py still explicitly looks for a zip 
file.  Occasionally the package maintainers forget this and an issue is 
raised, though apparently this has been automated to avoid such a error:
https://github.com/pypa/setuptools/issues/549

That said, I occasionally have erroneously uploaded a zip file when a 
tarball for that same version was already there, or the other way 
around.  They contained potentially different code: either I forgot to 
do a 'git pull' before releasing, or the previous uploader forgot to do 
a 'git push'.  Allowing only one sdist would have prevented this.  But 
really this is our own mistake.

I think I have always discovered this and removed the new distribution 
and uploaded a full new version to avoid problems.  I don't recognize 
any packages in the list.


-- 
Maurits van Rees: http://maurits.vanrees.org/
Zest Software: http://zestsoftware.nl


From anntzer.lee at gmail.com  Mon Aug 29 13:29:09 2016
From: anntzer.lee at gmail.com (Antony Lee)
Date: Mon, 29 Aug 2016 10:29:09 -0700
Subject: [Distutils] setup_requires: the obvious option(?)
Message-ID: <CAGRr6BG6RanD999ck=sYAPfO4grNku3pFUskE+kOECT-poCPsw@mail.gmail.com>

Hi all,

The `setup_requires` option to `setup()` is well-known to suffer from
multiple issues.  Most importantly, as it is a keyword argument to
`setup()`, it appears too late for modules that may need to be imported for
the build to occur (e.g., Cython, for which support must explicitly
provided by setuptools itself rather than by letting Cython hook into it);
additionally, there are various contorsions that people go to to avoid some
`setup_requires` when not building the package (such as checking the value
of `sys.argv`).  `setup_requires` also uses `easy_install` rather than
`pip`, but I do not see why this could not be fixed; let's focus on the
first issue instead.

If `setup_requires` appears too late to be useful, the obvious(?) option is
to move it earlier: provide a function, say, `setuptools.setup_requires()`,
that should be called *before* `setup()` itself, e.g.:

    from setuptools import setup, setup_requires
    setup_requires("numpy", needed_for=["build_ext"])
    try:
        import numpy as np
    except ImportError:
        np = None
    setup(..., include_dirs=[np.get_include()] if np else [])

When `setup.py` is invoked, either directly or by pip, upon the call to
`setup_requires()`, if `sys.argv[0]` is in the `needed_for` kwarg, and at
least one requirement is missing, `setup_requires()` calls asks pip to
install the required packages (similarly to `
https://bitbucket.org/dholth/setup-requires`) in a temporary directory, and
the whole Python process replaces itself (in the `os.execv()` sense) by a
new call to `python setup.py` with this temporary directory prepended to
the PYTHONPATH.  In this new process, the arguments to `setup_requires()`
are now available and we can proceed to the rest of `setup.py`.

I feel like this idea is natural enough that someone must already have come
up with it... but I may be missing something :-)

Best,
Antony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160829/8ffc9798/attachment.html>

From brett at python.org  Mon Aug 29 15:52:09 2016
From: brett at python.org (Brett Cannon)
Date: Mon, 29 Aug 2016 19:52:09 +0000
Subject: [Distutils] What is the official position on distutils?
In-Reply-To: <CAK=Phk4XYgn45_2vJqzGciZJKFx4aAdGzMqZQ7o3f3bPHQLH4A@mail.gmail.com>
References: <CAP1=2W4ZXcC5CO5oqKcrsEhYjRvrYqe7o1mnKnOJ5+sCkh9k8w@mail.gmail.com>
 <CACac1F8NXxGbKCPT8X1cviC1BJQ8MsPSZ4zPsPv6XCeEn3QyMQ@mail.gmail.com>
 <CAK=Phk4XYgn45_2vJqzGciZJKFx4aAdGzMqZQ7o3f3bPHQLH4A@mail.gmail.com>
Message-ID: <CAP1=2W7L70oAim3Fbyw=Qi+D+OnpGn=B_Om22mBGFX2ho0peaA@mail.gmail.com>

On Sun, 28 Aug 2016 at 16:55 Sylvain Corlay <sylvain.corlay at gmail.com>
wrote:

> Distutils seems to be the de-facto standard for building extension modules
> for Python and it is used for most of the third-party extensions out there.
> I don?t think that it is reasonable to declare that it is now only meant
> for Python itself.
>

Sorry, I didn't mean to suggest distutils was suddenly going to change or
disappear, but I did want a direction/focus for what we want distutils to
become long-term as we have this discussion of what to do with distutils
any time a change is proposed. At this point distutils is practically a
black box of code as no one wants to touch out of fear of breaking the
world and that's not sustainable.


> I actually see a contradiction in pointing out some lack of involvement in
> the library to oppose well needed improvements, especially for a tool that
> has so much adoption.
>

It's not a contradiction because new features != maintenance. And the usage
of distutils makes its lack of maintenance even more scary. I guess my key
point from what I said in my previous email is that we shouldn't
perpetually ignore the fact that distutils is not to a level of support or
quality that any of us are happy with and yet so many people rely on.

For me this means we should prioritize getting our build API defined and
sdists standardized so that distutils (or a replacement) can be in the
stdlib to build Python itself only, and after that you get your build tool
from PyPI. And if that means setuptools swallows/subsumes distutils then
that's fine with me, but my key point is we are not maintaining distutils
in a way I think any of us would choose to now and so I'm trying to prod
this question to start thinking about how to fix the maintenance problem
long-term.


>
> As we discussed earlier, even though it is not a concern with C, checking
> for availability of a compiler flag becomes crucial when building
> extensions with C++ since new flavors of the language emerge every couple
> of years now. It is important to be able to output meaningful error
> messages when the compiler does not support C++[11/14/17] features if they
> are needed for a given extension. This is a new aspect of the landscape in
> this area.
>
> Finally, adding this method is a very straightforward change. `has_flag`
> simply comes aside `has_function` as a method of ccompiler. I don't see a
> more natural place for it. It would be a very weird design decision in my
> opinion to not add it there, and instead to add it to distutils.ccompiler
> by monkeypatching it in setuptools.
>

Honestly I have no comment on your feature, Sylvain, and I'm sorry your
proposal happens to be the catalyst to this discussion. I'm just trying to
get a general alignment from the PyPA group as the "distutils problem"
comes up and has the same points made every time with no general decision
on how to handle it long-term.

-Brett


>
> Thanks,
>
> Sylvain
>
> On Mon, Aug 29, 2016 at 12:36 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>
>> On 28 August 2016 at 18:05, Brett Cannon <brett at python.org> wrote:
>> > The discussion of Sylvain's proposed changes to distutils suggests that
>> > there isn't a clear-cut agreement or position of this SIG -- and thus
>> Python
>> > -- on changes to distutils, its future, etc. Is there an official
>> position
>> > I'm not aware of? If not, could we get one so we know how to handle any
>> more
>> > proposed changes to distutils?
>> >
>> > For me personally (and to start this conversation if necessary), I view
>> > distutils as the package in the stdlib that handles building extension
>> > modules in the stdlib for Python. That view means that if something
>> doesn't
>> > service that explicit goal then it shouldn't go into distutils. Now
>> that we
>> > are moving down the road towards making the build step configurable I'm
>> fine
>> > with saying that distutils is not expected to work for people other than
>> > Python itself and others can use setuptools, enscons, or other projects
>> > while we continue to improve the situation to where the build system is
>> just
>> > something you specify in pyproject.toml to generate your wheel. IOW
>> > distutils is only exposed publicly because Python doesn't hide
>> anything, but
>> > making it useful for the general case of direct usage by people is a
>> > non-goal of the package.
>>
>> FWIW, my view is:
>>
>> * distutils handles building of Python packages, both in the stdlib and
>> outside
>> * in practice, setuptools is almost always used with distutils, and
>> any proposed change to distutils could be made to setuptools just as
>> easily
>> * patches for distutils typically suffer from a lack of maintainers
>> motivated to work on them
>> * there are a few developers with a strong investment in distutils,
>> who have code that is deeply entwined with the internals of distutils,
>> and changes to distutils come with a high risk of causing issues for
>> those people
>> * these developers *don't* use setuptools, so changes in setuptools
>> don't tend to suffer from the same level of backward compatibility
>> constraints
>> * implementing changes in setuptools is *far* easier than doing so in
>> distutils, for the above reasons
>> * changes in setuptools can be used with any version of Python,
>> distutils changes are only available to people with no need to support
>> anything but the latest version (which in practice is a vanishingly
>> small group)
>> * as a result, there would have to be an *extremely* compelling reason
>> to make a change in distutils rather than in setuptools - sufficient
>> to justify the additional risk, the extra developer effort needed, and
>> the fact that any such change is only going to benefit users of newer
>> versions of Python
>>
>> The reality is simply that distutils is in a somewhat unhappy place,
>> with too few interested maintainers and serious compatibility problems
>> to deal with, but that's not exactly news to anyone. I don't think
>> that a formal statement will help much. Past history has shown that
>> making pronouncements like "distutils is frozen" has not helped anyone
>> - so we're just trying to work with the status quo.
>>
>> With regard to Sylvain's proposed changes, I haven't really seen any
>> arguments justifying making the change in distutils rather than in
>> setuptools - the last suggestion I saw was to make the change in
>> *both* places, but I don't see any particular reason (beyond abstract
>> "it makes more sense to have the change in distutils" arguments) for
>> not just targeting setuptools. I may be missing something (as I'm not
>> personally in a position where I'd be comfortable reviewing the
>> change) but it seems to me that it's a bit of a non-issue - just
>> implement in setuptools and move on. The rhetoric about "this means
>> that distutils is dead" doesn't feel particularly helpful.
>>
>> Paul
>>
> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160829/9b2e7236/attachment-0001.html>

From brett at python.org  Mon Aug 29 19:34:01 2016
From: brett at python.org (Brett Cannon)
Date: Mon, 29 Aug 2016 23:34:01 +0000
Subject: [Distutils] Best way to handle packaging a project that only
 applies to certain versions of Python
Message-ID: <CAP1=2W79-b+x_c_6pSz54A1NAcgDzub4s2o3-7SmDPpzyyD4rQ@mail.gmail.com>

Someone has asked that I do a new release of importlib that includes a
LICENSE file on PyPI: https://pypi.org/project/importlib/. Historically I
have had the setup.py simply not include any Python code when built on
versions of Python that include importlib in the stdlib itself:
https://github.com/brettcannon/importlib/blob/master/setup.py .

But now I would like to do a wheel. Is there some way I'm not thinking of
to have a wheel that will leave out code or not install itself if a certain
version of Python is used? Or will the user have to specify a proper Python
requirement in their requirements.txt to get that kind of experience?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160829/0a9926bc/attachment.html>

From donald at stufft.io  Mon Aug 29 19:45:59 2016
From: donald at stufft.io (Donald Stufft)
Date: Mon, 29 Aug 2016 19:45:59 -0400
Subject: [Distutils] Best way to handle packaging a project that only
 applies to certain versions of Python
In-Reply-To: <CAP1=2W79-b+x_c_6pSz54A1NAcgDzub4s2o3-7SmDPpzyyD4rQ@mail.gmail.com>
References: <CAP1=2W79-b+x_c_6pSz54A1NAcgDzub4s2o3-7SmDPpzyyD4rQ@mail.gmail.com>
Message-ID: <10B80953-ED2A-4856-ACE7-01E6C3A6A996@stufft.io>


> On Aug 29, 2016, at 7:34 PM, Brett Cannon <brett at python.org> wrote:
> 
> Someone has asked that I do a new release of importlib that includes a LICENSE file on PyPI: https://pypi.org/project/importlib/ <https://pypi.org/project/importlib/>. Historically I have had the setup.py simply not include any Python code when built on versions of Python that include importlib in the stdlib itself: https://github.com/brettcannon/importlib/blob/master/setup.py <https://github.com/brettcannon/importlib/blob/master/setup.py> .
> 
> But now I would like to do a wheel. Is there some way I'm not thinking of to have a wheel that will leave out code or not install itself if a certain version of Python is used? Or will the user have to specify a proper Python requirement in their requirements.txt to get that kind of experience?
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


If your setup.py produces different output on different versions of Python, you?ll need multiple wheels.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160829/cb6216be/attachment.html>

From dholth at gmail.com  Mon Aug 29 20:08:26 2016
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 30 Aug 2016 00:08:26 +0000
Subject: [Distutils] Best way to handle packaging a project that only
 applies to certain versions of Python
In-Reply-To: <10B80953-ED2A-4856-ACE7-01E6C3A6A996@stufft.io>
References: <CAP1=2W79-b+x_c_6pSz54A1NAcgDzub4s2o3-7SmDPpzyyD4rQ@mail.gmail.com>
 <10B80953-ED2A-4856-ACE7-01E6C3A6A996@stufft.io>
Message-ID: <CAG8k2+7UWWKbC9VcbDNkbVZixFe=23814JQ9fPV7fZf44zBYHQ@mail.gmail.com>

Let's say you have different code for python 3.3 and python 3.4+. Tag one
wheel py33-none-any and the second py34-none-any. The second wheel is
preferred on python 3.4 and above, but ignored by 3.3. The py3 tag wouldn't
work well here.

Order of preference on 3.5 is: ('py35', 'none', 'any'), ('py3', 'none',
'any'), ('py34', 'none', 'any'), ('py33', 'none', 'any'), ('py32', 'none',
'any'), ('py31', 'none', 'any'), ('py30', 'none', 'any')

Enscons would let you control the wheel tag directly but doesn't yet let
you build multiple wheels with a single invocation.

On Mon, Aug 29, 2016, 19:46 Donald Stufft <donald at stufft.io> wrote:

>
> On Aug 29, 2016, at 7:34 PM, Brett Cannon <brett at python.org> wrote:
>
> Someone has asked that I do a new release of importlib that includes a
> LICENSE file on PyPI: https://pypi.org/project/importlib/. Historically I
> have had the setup.py simply not include any Python code when built on
> versions of Python that include importlib in the stdlib itself:
> https://github.com/brettcannon/importlib/blob/master/setup.py .
>
> But now I would like to do a wheel. Is there some way I'm not thinking of
> to have a wheel that will leave out code or not install itself if a certain
> version of Python is used? Or will the user have to specify a proper Python
> requirement in their requirements.txt to get that kind of experience?
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
> If your setup.py produces different output on different versions of
> Python, you?ll need multiple wheels.
>
> ?
>
> Donald Stufft
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160830/b4a08866/attachment.html>

From thomas at kluyver.me.uk  Tue Aug 30 02:20:58 2016
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Tue, 30 Aug 2016 07:20:58 +0100
Subject: [Distutils] Best way to handle packaging a project that only
 applies to certain versions of Python
In-Reply-To: <CAG8k2+7UWWKbC9VcbDNkbVZixFe=23814JQ9fPV7fZf44zBYHQ@mail.gmail.com>
References: <CAP1=2W79-b+x_c_6pSz54A1NAcgDzub4s2o3-7SmDPpzyyD4rQ@mail.gmail.com>
 <10B80953-ED2A-4856-ACE7-01E6C3A6A996@stufft.io>
 <CAG8k2+7UWWKbC9VcbDNkbVZixFe=23814JQ9fPV7fZf44zBYHQ@mail.gmail.com>
Message-ID: <1472538058.2676293.709983225.01370E6A@webmail.messagingengine.com>

There is also the 'python requires' metadata, which can now be provided
via setuptools, but I think pip 8.2 needs to be released before that is
respected.


On Tue, Aug 30, 2016, at 01:08 AM, Daniel Holth wrote:
> Let's say you have different code for python 3.3 and python 3.4+. Tag
> one wheel py33-none-any and the second py34-none-any. The second wheel
> is preferred on python 3.4 and above, but ignored by 3.3. The py3 tag
> wouldn't work well here.
> Order of preference on 3.5 is: ('py35', 'none', 'any'), ('py3',
> 'none', 'any'), ('py34', 'none', 'any'), ('py33', 'none', 'any'),
> ('py32', 'none', 'any'), ('py31', 'none', 'any'), ('py30',
> 'none', 'any')
> Enscons would let you control the wheel tag directly but doesn't yet
> let you build multiple wheels with a single invocation.
>
> On Mon, Aug 29, 2016, 19:46 Donald Stufft <donald at stufft.io> wrote:
>>
>>> On Aug 29, 2016, at 7:34 PM, Brett Cannon <brett at python.org> wrote:
>>>
>>> Someone has asked that I do a new release of importlib that includes
>>> a LICENSE file on PyPI: https://pypi.org/project/importlib/.
>>> Historically I have had the setup.py simply not include any Python
>>> code when built on versions of Python that include importlib in the
>>> stdlib itself:
>>> https://github.com/brettcannon/importlib/blob/master/setup.py .
>>>
>>> But now I would like to do a wheel. Is there some way I'm not
>>> thinking of to have a wheel that will leave out code or not install
>>> itself if a certain version of Python is used? Or will the user have
>>> to specify a proper Python requirement in their requirements.txt to
>>> get that kind of experience?
>>> _______________________________________________
>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>> If your setup.py produces different output on different versions of
>> Python, you?ll need multiple wheels.
>>
>> ?
>>
>> Donald Stufft
>> _______________________________________________
>>  Distutils-SIG maillist  - Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
> _________________________________________________
> Distutils-SIG maillist  - Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160830/9b92748c/attachment-0001.html>

From solipsis at pitrou.net  Tue Aug 30 07:14:16 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 30 Aug 2016 13:14:16 +0200
Subject: [Distutils] What is the official position on distutils?
References: <CAP1=2W4ZXcC5CO5oqKcrsEhYjRvrYqe7o1mnKnOJ5+sCkh9k8w@mail.gmail.com>
 <CACac1F8NXxGbKCPT8X1cviC1BJQ8MsPSZ4zPsPv6XCeEn3QyMQ@mail.gmail.com>
Message-ID: <20160830131416.05aa315f@fsol>

On Sun, 28 Aug 2016 23:36:43 +0100
Paul Moore <p.f.moore at gmail.com> wrote:
> * as a result, there would have to be an *extremely* compelling reason
> to make a change in distutils rather than in setuptools - sufficient
> to justify the additional risk, the extra developer effort needed, and
> the fact that any such change is only going to benefit users of newer
> versions of Python

There is a problem with this: distutils and setuptools don't document
or specify their internal APIs.  Yet if you want to do something
advanced, you have to hook into the command classes and either call
their APIs or extend them.

Using distutils, this is already tedious, since you have to go read its
source code and try to figure out what happens (it's rarely obvious).

Using setuptools, this is even worse, though, because setuptools builds
on top of distutils by modifying the command classes, and so you have
to figure out the effect that monkeypatching has on the overall
behaviour (in addition to figure out what happens in distutils).


I don't know if that's bothersome enough to compound the fact that
changes made in setuptools can benefit all Python versions, but I think
it's useful to keep it in mind.


(that lack of programmability is really distutils' failure, of course,
and setuptools inherited that failure by inheriting distutils'
architecture.)

Regards

Antoine.



From erik.m.bray at gmail.com  Tue Aug 30 08:08:37 2016
From: erik.m.bray at gmail.com (Erik Bray)
Date: Tue, 30 Aug 2016 14:08:37 +0200
Subject: [Distutils] setup_requires: the obvious option(?)
In-Reply-To: <CAGRr6BG6RanD999ck=sYAPfO4grNku3pFUskE+kOECT-poCPsw@mail.gmail.com>
References: <CAGRr6BG6RanD999ck=sYAPfO4grNku3pFUskE+kOECT-poCPsw@mail.gmail.com>
Message-ID: <CAOTD34ZKZ2teQWoyZT4Jnp6usTU7n+-088=hYqD4FP5kNtPrvw@mail.gmail.com>

On Mon, Aug 29, 2016 at 7:29 PM, Antony Lee <anntzer.lee at gmail.com> wrote:
> Hi all,
>
> The `setup_requires` option to `setup()` is well-known to suffer from
> multiple issues.  Most importantly, as it is a keyword argument to
> `setup()`, it appears too late for modules that may need to be imported for
> the build to occur (e.g., Cython, for which support must explicitly provided
> by setuptools itself rather than by letting Cython hook into it);
> additionally, there are various contorsions that people go to to avoid some
> `setup_requires` when not building the package (such as checking the value
> of `sys.argv`).  `setup_requires` also uses `easy_install` rather than
> `pip`, but I do not see why this could not be fixed; let's focus on the
> first issue instead.
>
> If `setup_requires` appears too late to be useful, the obvious(?) option is
> to move it earlier: provide a function, say, `setuptools.setup_requires()`,
> that should be called *before* `setup()` itself, e.g.:
>
>     from setuptools import setup, setup_requires
>     setup_requires("numpy", needed_for=["build_ext"])
>     try:
>         import numpy as np
>     except ImportError:
>         np = None
>     setup(..., include_dirs=[np.get_include()] if np else [])

I mean this sort of already exists but it's spelled:

from setuptools import Distribution
Distribution({'setup_requires': ['numpy'])

Granted it's non-obvious and doesn't have the needed_for flag, which I
like.  It's not entirely clear how needed_for would work though.  For
example, what if the package you're requiring provides the command
that you need that package to run?

The same can be done by subclassing commands, and there can be some
corner cases where that gets extra tricky (Cython comes to mind).

> When `setup.py` is invoked, either directly or by pip, upon the call to
> `setup_requires()`, if `sys.argv[0]` is in the `needed_for` kwarg, and at
> least one requirement is missing, `setup_requires()` calls asks pip to
> install the required packages (similarly to
> `https://bitbucket.org/dholth/setup-requires`) in a temporary directory, and
> the whole Python process replaces itself (in the `os.execv()` sense) by a
> new call to `python setup.py` with this temporary directory prepended to the
> PYTHONPATH.  In this new process, the arguments to `setup_requires()` are
> now available and we can proceed to the rest of `setup.py`.
>
> I feel like this idea is natural enough that someone must already have come
> up with it... but I may be missing something :-)

I'm glad you mentioned Daniel Holth's setup-requires hack.  Although I
haven't used it myself directly I generally like the concept.

Yeah, setup_requires is a mess, but I'd be skeptical of solving the
problem by depending on any new features in setuptools :/

Best,
Erik

From p.f.moore at gmail.com  Tue Aug 30 08:34:22 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 30 Aug 2016 13:34:22 +0100
Subject: [Distutils] What is the official position on distutils?
In-Reply-To: <20160830131416.05aa315f@fsol>
References: <CAP1=2W4ZXcC5CO5oqKcrsEhYjRvrYqe7o1mnKnOJ5+sCkh9k8w@mail.gmail.com>
 <CACac1F8NXxGbKCPT8X1cviC1BJQ8MsPSZ4zPsPv6XCeEn3QyMQ@mail.gmail.com>
 <20160830131416.05aa315f@fsol>
Message-ID: <CACac1F-ZE5K=8Hji_ehihbno=zYmncgxgfwG4vtwQWu4__-LaA@mail.gmail.com>

On 30 August 2016 at 12:14, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sun, 28 Aug 2016 23:36:43 +0100
> Paul Moore <p.f.moore at gmail.com> wrote:
>> * as a result, there would have to be an *extremely* compelling reason
>> to make a change in distutils rather than in setuptools - sufficient
>> to justify the additional risk, the extra developer effort needed, and
>> the fact that any such change is only going to benefit users of newer
>> versions of Python
>
> There is a problem with this: distutils and setuptools don't document
> or specify their internal APIs.  Yet if you want to do something
> advanced, you have to hook into the command classes and either call
> their APIs or extend them.
>
> Using distutils, this is already tedious, since you have to go read its
> source code and try to figure out what happens (it's rarely obvious).
>
> Using setuptools, this is even worse, though, because setuptools builds
> on top of distutils by modifying the command classes, and so you have
> to figure out the effect that monkeypatching has on the overall
> behaviour (in addition to figure out what happens in distutils).
>
>
> I don't know if that's bothersome enough to compound the fact that
> changes made in setuptools can benefit all Python versions, but I think
> it's useful to keep it in mind.
>
>
> (that lack of programmability is really distutils' failure, of course,
> and setuptools inherited that failure by inheriting distutils'
> architecture.)

>From what I know, anyone who is actually hacking into the internal
APIs to that level tends to use only distutils (probably because as
you say, setuptools is even worse) and therefore falls into my
category of "people affected by distutils changes who won't mind if a
change is made to setuptools". Admittedly, that's a self-perpetuating
situation...

The fact remains that (as far as it's possible to tell) very few
people do dive into distutils to that level. At least, not many people
seem to have problems caused by the current "leave distutils alone,
make changes in setuptools" approach. Ideally, someone would be
working on alternative solutions to break the remaining few's tight
dependency on distutils, but with the effort they have invested in it,
there doesn't seem to be much interest in re-engineering something
that works.

Paul

From solipsis at pitrou.net  Tue Aug 30 08:38:30 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 30 Aug 2016 14:38:30 +0200
Subject: [Distutils] What is the official position on distutils?
In-Reply-To: <CACac1F-ZE5K=8Hji_ehihbno=zYmncgxgfwG4vtwQWu4__-LaA@mail.gmail.com>
References: <CAP1=2W4ZXcC5CO5oqKcrsEhYjRvrYqe7o1mnKnOJ5+sCkh9k8w@mail.gmail.com>
 <CACac1F8NXxGbKCPT8X1cviC1BJQ8MsPSZ4zPsPv6XCeEn3QyMQ@mail.gmail.com>
 <20160830131416.05aa315f@fsol>
 <CACac1F-ZE5K=8Hji_ehihbno=zYmncgxgfwG4vtwQWu4__-LaA@mail.gmail.com>
Message-ID: <20160830143830.19d0da0d@fsol>

On Tue, 30 Aug 2016 13:34:22 +0100
Paul Moore <p.f.moore at gmail.com> wrote:
> 
> From what I know, anyone who is actually hacking into the internal
> APIs to that level tends to use only distutils (probably because as
> you say, setuptools is even worse) and therefore falls into my
> category of "people affected by distutils changes who won't mind if a
> change is made to setuptools".

Except when, as you point out, pip injects setuptools nilly-willy when
running setup.py?

Regards

Antoine.

From dholth at gmail.com  Tue Aug 30 08:56:01 2016
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 30 Aug 2016 12:56:01 +0000
Subject: [Distutils] setup_requires: the obvious option(?)
In-Reply-To: <CAOTD34ZKZ2teQWoyZT4Jnp6usTU7n+-088=hYqD4FP5kNtPrvw@mail.gmail.com>
References: <CAGRr6BG6RanD999ck=sYAPfO4grNku3pFUskE+kOECT-poCPsw@mail.gmail.com>
 <CAOTD34ZKZ2teQWoyZT4Jnp6usTU7n+-088=hYqD4FP5kNtPrvw@mail.gmail.com>
Message-ID: <CAG8k2+7Wdc7KWL4UHLkw07Nmnyo+Knfe1NYuGqO+ahz=mQxvWw@mail.gmail.com>

On Tue, Aug 30, 2016 at 8:09 AM Erik Bray <erik.m.bray at gmail.com> wrote:

> On Mon, Aug 29, 2016 at 7:29 PM, Antony Lee <anntzer.lee at gmail.com> wrote:
> > Hi all,
> >
> > The `setup_requires` option to `setup()` is well-known to suffer from
> > multiple issues.  Most importantly, as it is a keyword argument to
> > `setup()`, it appears too late for modules that may need to be imported
> for
> > the build to occur (e.g., Cython, for which support must explicitly
> provided
> > by setuptools itself rather than by letting Cython hook into it);
> > additionally, there are various contorsions that people go to to avoid
> some
> > `setup_requires` when not building the package (such as checking the
> value
> > of `sys.argv`).  `setup_requires` also uses `easy_install` rather than
> > `pip`, but I do not see why this could not be fixed; let's focus on the
> > first issue instead.
> >
> > If `setup_requires` appears too late to be useful, the obvious(?) option
> is
> > to move it earlier: provide a function, say,
> `setuptools.setup_requires()`,
> > that should be called *before* `setup()` itself, e.g.:
> >
> >     from setuptools import setup, setup_requires
> >     setup_requires("numpy", needed_for=["build_ext"])
> >     try:
> >         import numpy as np
> >     except ImportError:
> >         np = None
> >     setup(..., include_dirs=[np.get_include()] if np else [])
>
> I mean this sort of already exists but it's spelled:
>
> from setuptools import Distribution
> Distribution({'setup_requires': ['numpy'])
>
> Granted it's non-obvious and doesn't have the needed_for flag, which I
> like.  It's not entirely clear how needed_for would work though.  For
> example, what if the package you're requiring provides the command
> that you need that package to run?
>
> The same can be done by subclassing commands, and there can be some
> corner cases where that gets extra tricky (Cython comes to mind).
>
> > When `setup.py` is invoked, either directly or by pip, upon the call to
> > `setup_requires()`, if `sys.argv[0]` is in the `needed_for` kwarg, and at
> > least one requirement is missing, `setup_requires()` calls asks pip to
> > install the required packages (similarly to
> > `https://bitbucket.org/dholth/setup-requires`
> <https://bitbucket.org/dholth/setup-requires>) in a temporary directory,
> and
> > the whole Python process replaces itself (in the `os.execv()` sense) by a
> > new call to `python setup.py` with this temporary directory prepended to
> the
> > PYTHONPATH.  In this new process, the arguments to `setup_requires()` are
> > now available and we can proceed to the rest of `setup.py`.
> >
> > I feel like this idea is natural enough that someone must already have
> come
> > up with it... but I may be missing something :-)
>
> I'm glad you mentioned Daniel Holth's setup-requires hack.  Although I
> haven't used it myself directly I generally like the concept.
>
> Yeah, setup_requires is a mess, but I'd be skeptical of solving the
> problem by depending on any new features in setuptools :/


In June I updated my setup_requires hack to be a PEP 518 implementation. It
reads build-system.requires out of pyproject.toml and installs them into an
isolated directory if necessary. The basic feature is 14 lines of code not
counting blanks, and you use it by prepending the code to your setup.py.
Once pip implements PEP 518 that code becomes a no-op.

Antony is right that the main problem with setup_requires is that it
happens too late, or that you have to write a setuptools extension to use
it. Most people do not know that it is possible to write a setuptools
extension, let alone want to write one.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160830/73a2b913/attachment-0001.html>

From p.f.moore at gmail.com  Tue Aug 30 09:44:10 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 30 Aug 2016 14:44:10 +0100
Subject: [Distutils] What is the official position on distutils?
In-Reply-To: <20160830143830.19d0da0d@fsol>
References: <CAP1=2W4ZXcC5CO5oqKcrsEhYjRvrYqe7o1mnKnOJ5+sCkh9k8w@mail.gmail.com>
 <CACac1F8NXxGbKCPT8X1cviC1BJQ8MsPSZ4zPsPv6XCeEn3QyMQ@mail.gmail.com>
 <20160830131416.05aa315f@fsol>
 <CACac1F-ZE5K=8Hji_ehihbno=zYmncgxgfwG4vtwQWu4__-LaA@mail.gmail.com>
 <20160830143830.19d0da0d@fsol>
Message-ID: <CACac1F8-=nch9jNprYJJX-Fb4iPJrEBP=uinmum4iU_4R9dOrA@mail.gmail.com>

On 30 August 2016 at 13:38, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Tue, 30 Aug 2016 13:34:22 +0100
> Paul Moore <p.f.moore at gmail.com> wrote:
>>
>> From what I know, anyone who is actually hacking into the internal
>> APIs to that level tends to use only distutils (probably because as
>> you say, setuptools is even worse) and therefore falls into my
>> category of "people affected by distutils changes who won't mind if a
>> change is made to setuptools".
>
> Except when, as you point out, pip injects setuptools nilly-willy when
> running setup.py?

I'm confused. People who hack into distutils internals can't cope with
having setuptools injected (because their hacks conflict with
setuptools' hacks). That's my understanding of the situation, but it's
based mainly on what I've seen of people saying "Don't break my
APIs!!!" when distutils changes have been proposed in the past. The
projects involved are already outside of the current
pip/setuptools/wheel infrastructure (they have their own install
processes, they don't host on PyPI, they are not pip-installable or
whatever). So injecting setuptools isn't relevant, because they are
outside that particular ecosystem.

What I'm trying to say (badly, it seems - for which I apologise) is
that if we change setuptools, only people working in the current
ecosystem are affected, and those projects have typically been OK with
the level of change going on in setuptools. People outside of the
pip/setuptools ecosystem rely on distutils, and appear to be heavier
users of the internal distutils APIs. As a result they appear to be
typically much more conservative in their requirements (precisely
because they had to work with an undocumented internal API). So
there's a much higher barrier to changing distutils, and making
changes in setuptools instead is less likely to break existing
projects (at a cost of making such changes not accessible to
non-setuptools projects).

This is largely off-topic by now, though. I wasn't trying to derail
the discussion about policy on distutils, so I'll drop this subthread
now, with apologies to anyone who found it a distraction from the main
point.

Paul

From solipsis at pitrou.net  Tue Aug 30 10:09:40 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 30 Aug 2016 16:09:40 +0200
Subject: [Distutils] What is the official position on distutils?
In-Reply-To: <CACac1F8-=nch9jNprYJJX-Fb4iPJrEBP=uinmum4iU_4R9dOrA@mail.gmail.com>
References: <CAP1=2W4ZXcC5CO5oqKcrsEhYjRvrYqe7o1mnKnOJ5+sCkh9k8w@mail.gmail.com>
 <CACac1F8NXxGbKCPT8X1cviC1BJQ8MsPSZ4zPsPv6XCeEn3QyMQ@mail.gmail.com>
 <20160830131416.05aa315f@fsol>
 <CACac1F-ZE5K=8Hji_ehihbno=zYmncgxgfwG4vtwQWu4__-LaA@mail.gmail.com>
 <20160830143830.19d0da0d@fsol>
 <CACac1F8-=nch9jNprYJJX-Fb4iPJrEBP=uinmum4iU_4R9dOrA@mail.gmail.com>
Message-ID: <20160830160940.12b5f17f@fsol>

On Tue, 30 Aug 2016 14:44:10 +0100
Paul Moore <p.f.moore at gmail.com> wrote:
> On 30 August 2016 at 13:38, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > On Tue, 30 Aug 2016 13:34:22 +0100
> > Paul Moore <p.f.moore at gmail.com> wrote:  
> >>
> >> From what I know, anyone who is actually hacking into the internal
> >> APIs to that level tends to use only distutils (probably because as
> >> you say, setuptools is even worse) and therefore falls into my
> >> category of "people affected by distutils changes who won't mind if a
> >> change is made to setuptools".  
> >
> > Except when, as you point out, pip injects setuptools nilly-willy when
> > running setup.py?  
> 
> I'm confused. People who hack into distutils internals can't cope with
> having setuptools injected (because their hacks conflict with
> setuptools' hacks).

I'm not sure what you mean by that.  I'm sure there are situations
where it works (I can't remember for sure whether it did the last time
I did such "hacking").

The thing is, more and more problem expect packages to install with
"pip install ." and they report bugs for that (the PyPA's official
discours is of course partly responsible for this, since it's
insisting so heavily on using pip and discouraging direct calls to
setup.py).

And the day people expect all libraries to be available as wheel files,
it will probably become worse, because I don't know how to produce
wheel files using plain distutils.

Regards

Antoine.

From brett at python.org  Tue Aug 30 12:25:25 2016
From: brett at python.org (Brett Cannon)
Date: Tue, 30 Aug 2016 16:25:25 +0000
Subject: [Distutils] Best way to handle packaging a project that only
 applies to certain versions of Python
In-Reply-To: <1472538058.2676293.709983225.01370E6A@webmail.messagingengine.com>
References: <CAP1=2W79-b+x_c_6pSz54A1NAcgDzub4s2o3-7SmDPpzyyD4rQ@mail.gmail.com>
 <10B80953-ED2A-4856-ACE7-01E6C3A6A996@stufft.io>
 <CAG8k2+7UWWKbC9VcbDNkbVZixFe=23814JQ9fPV7fZf44zBYHQ@mail.gmail.com>
 <1472538058.2676293.709983225.01370E6A@webmail.messagingengine.com>
Message-ID: <CAP1=2W7xC2PfoR0-tn_e4_GSTDKmfsT7aeQXT5UKA8MAPN8QvQ@mail.gmail.com>

All of this seems like too much worry for a project that only applies to
Python 2.6 and earlier or Python 3.0. :) Thanks for the suggestions
everyone, but I'm just going to let the folks stuck on legacy versions deal
with the lack of wheels.

On Mon, 29 Aug 2016 at 23:21 Thomas Kluyver <thomas at kluyver.me.uk> wrote:

> There is also the 'python requires' metadata, which can now be provided
> via setuptools, but I think pip 8.2 needs to be released before that is
> respected.
>
>
> On Tue, Aug 30, 2016, at 01:08 AM, Daniel Holth wrote:
>
> Let's say you have different code for python 3.3 and python 3.4+. Tag one
> wheel py33-none-any and the second py34-none-any. The second wheel is
> preferred on python 3.4 and above, but ignored by 3.3. The py3 tag wouldn't
> work well here.
>
> Order of preference on 3.5 is: ('py35', 'none', 'any'), ('py3', 'none',
> 'any'), ('py34', 'none', 'any'), ('py33', 'none', 'any'), ('py32', 'none',
> 'any'), ('py31', 'none', 'any'), ('py30', 'none', 'any')
>
> Enscons would let you control the wheel tag directly but doesn't yet let
> you build multiple wheels with a single invocation.
>
> On Mon, Aug 29, 2016, 19:46 Donald Stufft <donald at stufft.io> wrote:
>
>
> On Aug 29, 2016, at 7:34 PM, Brett Cannon <brett at python.org> wrote:
>
> Someone has asked that I do a new release of importlib that includes a
> LICENSE file on PyPI: https://pypi.org/project/importlib/. Historically I
> have had the setup.py simply not include any Python code when built on
> versions of Python that include importlib in the stdlib itself:
> https://github.com/brettcannon/importlib/blob/master/setup.py .
>
> But now I would like to do a wheel. Is there some way I'm not thinking of
> to have a wheel that will leave out code or not install itself if a certain
> version of Python is used? Or will the user have to specify a proper Python
> requirement in their requirements.txt to get that kind of experience?
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
> If your setup.py produces different output on different versions of
> Python, you?ll need multiple wheels.
>
> ?
>
> Donald Stufft
> _______________________________________________
> Distutils-SIG maillist  - Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
> *_______________________________________________*
> Distutils-SIG maillist  - Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160830/b989741d/attachment.html>

From anntzer.lee at gmail.com  Tue Aug 30 12:51:14 2016
From: anntzer.lee at gmail.com (Antony Lee)
Date: Tue, 30 Aug 2016 09:51:14 -0700
Subject: [Distutils] setup_requires: the obvious option(?)
In-Reply-To: <CAOTD34ZKZ2teQWoyZT4Jnp6usTU7n+-088=hYqD4FP5kNtPrvw@mail.gmail.com>
References: <CAGRr6BG6RanD999ck=sYAPfO4grNku3pFUskE+kOECT-poCPsw@mail.gmail.com>
 <CAOTD34ZKZ2teQWoyZT4Jnp6usTU7n+-088=hYqD4FP5kNtPrvw@mail.gmail.com>
Message-ID: <CAGRr6BGyqEBDRMNsTMRRfMw7Ty=Uc_9R-pfXU-w3Y=Lk_=dKbA@mail.gmail.com>

2016-08-30 5:08 GMT-07:00 Erik Bray <erik.m.bray at gmail.com>:

> I mean this sort of already exists but it's spelled:
>
> from setuptools import Distribution
> Distribution({'setup_requires': ['numpy'])
>
> Granted it's non-obvious and doesn't have the needed_for flag, which I
> like.  It's not entirely clear how needed_for would work though.  For
> example, what if the package you're requiring provides the command
> that you need that package to run?
>

needed_for just textually checks sys.argv[1], and does so before the call
to setup() itself happens, so that's not a problem.

The same can be done by subclassing commands, and there can be some
> corner cases where that gets extra tricky (Cython comes to mind).


I personally don't like subclassing commands at all (it's not very
composable -- what happens if two projects both attempt to subclass
build_ext?).  But even then, how is Cython an issue?

I'm glad you mentioned Daniel Holth's setup-requires hack.  Although I
> haven't used it myself directly I generally like the concept.
>

I am not really a fan of PEP518 in general.  Basically, the idea of
setup.py is that declarative languages are not sufficient to express a
build system (and AFAICT this is always going to be the case for
expressing, say, compiler flags for extensions), so I'd rather just accept
that and stick everything in setup.py instead of adding more parameter
files.  What if someone wants dynamic build dependencies?


> Yeah, setup_requires is a mess, but I'd be skeptical of solving the
> problem by depending on any new features in setuptools :/
>

This could also go into pip (perhaps a better solution now that pip is
de-facto stdlib (via ensurepip))... we'll need something new somewhere
anyways.


> Best,
> Erik
>

Best,

Antony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160830/5e0dc333/attachment.html>

From thomas at kluyver.me.uk  Tue Aug 30 13:13:22 2016
From: thomas at kluyver.me.uk (Thomas Kluyver)
Date: Tue, 30 Aug 2016 18:13:22 +0100
Subject: [Distutils] setup_requires: the obvious option(?)
In-Reply-To: <CAGRr6BGyqEBDRMNsTMRRfMw7Ty=Uc_9R-pfXU-w3Y=Lk_=dKbA@mail.gmail.com>
References: <CAGRr6BG6RanD999ck=sYAPfO4grNku3pFUskE+kOECT-poCPsw@mail.gmail.com>
 <CAOTD34ZKZ2teQWoyZT4Jnp6usTU7n+-088=hYqD4FP5kNtPrvw@mail.gmail.com>
 <CAGRr6BGyqEBDRMNsTMRRfMw7Ty=Uc_9R-pfXU-w3Y=Lk_=dKbA@mail.gmail.com>
Message-ID: <1472577202.3918166.710584489.4F002F6A@webmail.messagingengine.com>

On Tue, Aug 30, 2016, at 05:51 PM, Antony Lee wrote:
> I am not really a fan of PEP518 in general.  Basically, the idea of
> setup.py is that declarative languages are not sufficient to express a
> build system (and AFAICT this is always going to be the case for
> expressing, say, compiler flags for extensions), so I'd rather just
> accept that and stick everything in setup.py instead of adding more
> parameter files.  What if someone wants dynamic build dependencies?

Dynamic build deps aren't precluded - the idea is that the build system
can discover additional dependencies when it runs, while the static build-
system field specifies just what's required to run the build system
itself.  However, the build system interface was split out into separate
PEPs (517 & 516 are alternatives) to allow 518 to go forwards.

I take totally the opposite view: we should make as much metadata as
possible declarative, even though we know we can't define a totally
general build system with purely declarative information.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160830/95465bde/attachment-0001.html>

From ncoghlan at gmail.com  Tue Aug 30 13:24:28 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 31 Aug 2016 03:24:28 +1000
Subject: [Distutils] What is the official position on distutils?
In-Reply-To: <CACac1F8-=nch9jNprYJJX-Fb4iPJrEBP=uinmum4iU_4R9dOrA@mail.gmail.com>
References: <CAP1=2W4ZXcC5CO5oqKcrsEhYjRvrYqe7o1mnKnOJ5+sCkh9k8w@mail.gmail.com>
 <CACac1F8NXxGbKCPT8X1cviC1BJQ8MsPSZ4zPsPv6XCeEn3QyMQ@mail.gmail.com>
 <20160830131416.05aa315f@fsol>
 <CACac1F-ZE5K=8Hji_ehihbno=zYmncgxgfwG4vtwQWu4__-LaA@mail.gmail.com>
 <20160830143830.19d0da0d@fsol>
 <CACac1F8-=nch9jNprYJJX-Fb4iPJrEBP=uinmum4iU_4R9dOrA@mail.gmail.com>
Message-ID: <CADiSq7do-K3SuQoiyYy-xFg87DEJ2Xkr3LfKPeYUArGtg7FNvQ@mail.gmail.com>

On 30 August 2016 at 23:44, Paul Moore <p.f.moore at gmail.com> wrote:
> On 30 August 2016 at 13:38, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> On Tue, 30 Aug 2016 13:34:22 +0100
>> Paul Moore <p.f.moore at gmail.com> wrote:
>>>
>>> From what I know, anyone who is actually hacking into the internal
>>> APIs to that level tends to use only distutils (probably because as
>>> you say, setuptools is even worse) and therefore falls into my
>>> category of "people affected by distutils changes who won't mind if a
>>> change is made to setuptools".
>>
>> Except when, as you point out, pip injects setuptools nilly-willy when
>> running setup.py?
>
> I'm confused. People who hack into distutils internals can't cope with
> having setuptools injected (because their hacks conflict with
> setuptools' hacks). That's my understanding of the situation, but it's
> based mainly on what I've seen of people saying "Don't break my
> APIs!!!" when distutils changes have been proposed in the past. The
> projects involved are already outside of the current
> pip/setuptools/wheel infrastructure (they have their own install
> processes, they don't host on PyPI, they are not pip-installable or
> whatever). So injecting setuptools isn't relevant, because they are
> outside that particular ecosystem.

No, this isn't the case - if you want to reliably generate wheels (as
bdist_wheel doesn't work with plain distutils), and if you want the
installdb metadata to always be generated, then you need to use pip
and its forced setuptools injection in order to get plain distutils
projects to behave the way you want.

This means that other setup.py helper projects like numpy.distutils
need to be setuptools-tolerant so they can support folks using pip
rather than calling setup.py directly.

That consequence further means that there's isn't substantially lower
risk in only changing setuptools without changing distutils - in fact,
by introducing a further unhelpful discrepancy between the two, it
increases the risk of code that works with plain distutils on 3.6+
breaking when setuptools is injected.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From dholth at gmail.com  Tue Aug 30 14:20:49 2016
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 30 Aug 2016 18:20:49 +0000
Subject: [Distutils] What is the official position on distutils?
In-Reply-To: <CADiSq7do-K3SuQoiyYy-xFg87DEJ2Xkr3LfKPeYUArGtg7FNvQ@mail.gmail.com>
References: <CAP1=2W4ZXcC5CO5oqKcrsEhYjRvrYqe7o1mnKnOJ5+sCkh9k8w@mail.gmail.com>
 <CACac1F8NXxGbKCPT8X1cviC1BJQ8MsPSZ4zPsPv6XCeEn3QyMQ@mail.gmail.com>
 <20160830131416.05aa315f@fsol>
 <CACac1F-ZE5K=8Hji_ehihbno=zYmncgxgfwG4vtwQWu4__-LaA@mail.gmail.com>
 <20160830143830.19d0da0d@fsol>
 <CACac1F8-=nch9jNprYJJX-Fb4iPJrEBP=uinmum4iU_4R9dOrA@mail.gmail.com>
 <CADiSq7do-K3SuQoiyYy-xFg87DEJ2Xkr3LfKPeYUArGtg7FNvQ@mail.gmail.com>
Message-ID: <CAG8k2+5sK72wY2aEgQ1W=4_Zm8+2PGu4YLVz188vhcOxFUp_Ew@mail.gmail.com>

Sylvain's feature seems totally harmless to me. If he wants to detect
features in Python 3.6 and above, more power to him. I love it.

If only it was sufficient to "read" the code for distutils. You will be
single-stepping it in a debugger, I guarantee it.

So let's imagine you wanted to create wheels with distutils without
installing setuptols. Technically you could write a tool do to that without
changing distutils by processing the output of "setup.py install" and
converting distutils' PKG-INFO to METADATA. Please note, the resulting
package has zero dependency information, dependencies are a setuptools-only
feature. It makes me sad to think about how completely drab and pointless
this would be. No offense meant.

Distutils does provide some useful features. If you subclass the build_ext
command and remove the "execute the compiler" step, you can use it to get
the compiler flags expected for C extensions. It also knows how to find the
right version of MSVC on Windows. Very handy:

class no_build_ext(build_ext):    def build_extension(self, ext):
  def noop_spawn(*args):            pass        self.compiler.spawn =
noop_spawn        build_ext.build_extension(self, ext)


IMO a useful, forward-looking refactor of distutils would take these kinds
of features "one weird trick to link Python modules" of which there are
several, and expose them independently of the command class system. Then
you could feed it into CMake or whatever and build some extensions. No I
don't know exactly how that would work.

It's also unimaginable that we would ever stop needing a working version of
distutils & setuptools, given that we have hundreds of thousands of
existing distributions that use it.

Daniel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160830/fa2d4f87/attachment.html>

From dholth at gmail.com  Tue Aug 30 14:32:14 2016
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 30 Aug 2016 18:32:14 +0000
Subject: [Distutils] setup_requires: the obvious option(?)
In-Reply-To: <1472577202.3918166.710584489.4F002F6A@webmail.messagingengine.com>
References: <CAGRr6BG6RanD999ck=sYAPfO4grNku3pFUskE+kOECT-poCPsw@mail.gmail.com>
 <CAOTD34ZKZ2teQWoyZT4Jnp6usTU7n+-088=hYqD4FP5kNtPrvw@mail.gmail.com>
 <CAGRr6BGyqEBDRMNsTMRRfMw7Ty=Uc_9R-pfXU-w3Y=Lk_=dKbA@mail.gmail.com>
 <1472577202.3918166.710584489.4F002F6A@webmail.messagingengine.com>
Message-ID: <CAG8k2+5HeFSGZ3x_XHkctrQnHdOxLkpiYOnbAMJE_jgqnVXhYQ@mail.gmail.com>

On Tue, Aug 30, 2016 at 1:13 PM Thomas Kluyver <thomas at kluyver.me.uk> wrote:

> On Tue, Aug 30, 2016, at 05:51 PM, Antony Lee wrote:
>
> I am not really a fan of PEP518 in general.  Basically, the idea of
> setup.py is that declarative languages are not sufficient to express a
> build system (and AFAICT this is always going to be the case for
> expressing, say, compiler flags for extensions), so I'd rather just accept
> that and stick everything in setup.py instead of adding more parameter
> files.  What if someone wants dynamic build dependencies?
>
>
> Dynamic build deps aren't precluded - the idea is that the build system
> can discover additional dependencies when it runs, while the static
> build-system field specifies just what's required to run the build system
> itself.  However, the build system interface was split out into separate
> PEPs (517 & 516 are alternatives) to allow 518 to go forwards.
>
> I take totally the opposite view: we should make as much metadata as
> possible declarative, even though we know we can't define a totally general
> build system with purely declarative information.
>

This comes up over and over again because we've been living with this
system for long enough that the build script and metadata are together both
in setup.py and in people's brains. But there is a whole lot of stuff that
makes perfect sense in a static file. For example: name, version, packages,
install_requires, extras_require, description, license, classifiers,
keywords, author, url, entry_points. The only thing that would cause
trouble is if the system had no available build script.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160830/e244d788/attachment.html>

From donald at stufft.io  Tue Aug 30 16:06:23 2016
From: donald at stufft.io (Donald Stufft)
Date: Tue, 30 Aug 2016 16:06:23 -0400
Subject: [Distutils] setup_requires: the obvious option(?)
In-Reply-To: <CAG8k2+5HeFSGZ3x_XHkctrQnHdOxLkpiYOnbAMJE_jgqnVXhYQ@mail.gmail.com>
References: <CAGRr6BG6RanD999ck=sYAPfO4grNku3pFUskE+kOECT-poCPsw@mail.gmail.com>
 <CAOTD34ZKZ2teQWoyZT4Jnp6usTU7n+-088=hYqD4FP5kNtPrvw@mail.gmail.com>
 <CAGRr6BGyqEBDRMNsTMRRfMw7Ty=Uc_9R-pfXU-w3Y=Lk_=dKbA@mail.gmail.com>
 <1472577202.3918166.710584489.4F002F6A@webmail.messagingengine.com>
 <CAG8k2+5HeFSGZ3x_XHkctrQnHdOxLkpiYOnbAMJE_jgqnVXhYQ@mail.gmail.com>
Message-ID: <E49BCC9E-000D-4FFF-8248-FDF89DE49F8F@stufft.io>


> On Aug 30, 2016, at 2:32 PM, Daniel Holth <dholth at gmail.com> wrote:
> 
> name, version, packages, install_requires, extras_require, description, license, classifiers, keywords, author, url, entry_points. 

Out of these, a number of them are regularly dynamic for people?s setup.py as is. The version number is often dynamically computed based on things like git tags, packages can be computed based on Python version, install_requires and extras_requires regularly get computed based on things like Python version, OS, etc (although environment markers eases some of this) but also things like ?We support Numpy >= for building from source, but once you?ve built from source you only support Numpy >= VERSION_YOU_BUILT_AGAINST?.

Outside of ?name?, it?s not entirely unreasonable to find reasons why a lot of things need to be dynamic. Although there?s a difference in what needs to be dynamic when pulling from a VCS and when pulling from a sdist, and currently there?s not really a whole lot of difference in terms of capability or how they are handled.

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160830/71008bbf/attachment-0001.html>

From dholth at gmail.com  Tue Aug 30 17:04:33 2016
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 30 Aug 2016 21:04:33 +0000
Subject: [Distutils] setup_requires: the obvious option(?)
In-Reply-To: <E49BCC9E-000D-4FFF-8248-FDF89DE49F8F@stufft.io>
References: <CAGRr6BG6RanD999ck=sYAPfO4grNku3pFUskE+kOECT-poCPsw@mail.gmail.com>
 <CAOTD34ZKZ2teQWoyZT4Jnp6usTU7n+-088=hYqD4FP5kNtPrvw@mail.gmail.com>
 <CAGRr6BGyqEBDRMNsTMRRfMw7Ty=Uc_9R-pfXU-w3Y=Lk_=dKbA@mail.gmail.com>
 <1472577202.3918166.710584489.4F002F6A@webmail.messagingengine.com>
 <CAG8k2+5HeFSGZ3x_XHkctrQnHdOxLkpiYOnbAMJE_jgqnVXhYQ@mail.gmail.com>
 <E49BCC9E-000D-4FFF-8248-FDF89DE49F8F@stufft.io>
Message-ID: <CAG8k2+6CBQR_upjnWyaQVED-Ng2WnU+OegpR-oPTvhcKLuvnOA@mail.gmail.com>

On Tue, Aug 30, 2016 at 4:06 PM Donald Stufft <donald at stufft.io> wrote:

>
> On Aug 30, 2016, at 2:32 PM, Daniel Holth <dholth at gmail.com> wrote:
>
> name, version, packages, install_requires, extras_require, description,
> license, classifiers, keywords, author, url, entry_points.
>
>
> Out of these, a number of them are regularly dynamic for people?s setup.py
> as is. The version number is often dynamically computed based on things
> like git tags, packages can be computed based on Python version,
> install_requires and extras_requires regularly get computed based on things
> like Python version, OS, etc (although environment markers eases some of
> this) but also things like ?We support Numpy >= for building from source,
> but once you?ve built from source you only support Numpy >=
> VERSION_YOU_BUILT_AGAINST?.
>
> Outside of ?name?, it?s not entirely unreasonable to find reasons why a
> lot of things need to be dynamic. Although there?s a difference in what
> needs to be dynamic when pulling from a VCS and when pulling from a sdist,
> and currently there?s not really a whole lot of difference in terms of
> capability or how they are handled.
>

Of course pip continues to call egg_info before trusting the metadata from
any sdist and 90k pypi projects say this isn't changing. Once you need
dynamic static metadata, madness.

In other systems I've worked on I sometimes have make-like rules that
automatically rebuild static metadata depending on other files, like
copying a version number between a .json and an .xml file - reprogramming
the system that uses the .xml file is not an option. For example a rule
could watch certain files in the .git directory to regenerate a version
number automatically as part of the build if .git changed, and do nothing
if the .git directory was absent per a tarball dist. This works pretty
well. Once you have a system that's easy to customize with make-like rules
there are all sorts of trivial build or housekeeping tasks you might decide
to do which would never be considered in a more difficult to customize
system.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160830/41a2c83e/attachment.html>

From ncoghlan at gmail.com  Tue Aug 30 23:44:14 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 31 Aug 2016 13:44:14 +1000
Subject: [Distutils] setup_requires: the obvious option(?)
In-Reply-To: <CAG8k2+6CBQR_upjnWyaQVED-Ng2WnU+OegpR-oPTvhcKLuvnOA@mail.gmail.com>
References: <CAGRr6BG6RanD999ck=sYAPfO4grNku3pFUskE+kOECT-poCPsw@mail.gmail.com>
 <CAOTD34ZKZ2teQWoyZT4Jnp6usTU7n+-088=hYqD4FP5kNtPrvw@mail.gmail.com>
 <CAGRr6BGyqEBDRMNsTMRRfMw7Ty=Uc_9R-pfXU-w3Y=Lk_=dKbA@mail.gmail.com>
 <1472577202.3918166.710584489.4F002F6A@webmail.messagingengine.com>
 <CAG8k2+5HeFSGZ3x_XHkctrQnHdOxLkpiYOnbAMJE_jgqnVXhYQ@mail.gmail.com>
 <E49BCC9E-000D-4FFF-8248-FDF89DE49F8F@stufft.io>
 <CAG8k2+6CBQR_upjnWyaQVED-Ng2WnU+OegpR-oPTvhcKLuvnOA@mail.gmail.com>
Message-ID: <CADiSq7c-SQrhtsprzeYDv-AfBPqMQc4oLCW1m8n31bWKjPyMUg@mail.gmail.com>

On 31 August 2016 at 07:04, Daniel Holth <dholth at gmail.com> wrote:
> In other systems I've worked on I sometimes have make-like rules that
> automatically rebuild static metadata depending on other files, like copying
> a version number between a .json and an .xml file - reprogramming the system
> that uses the .xml file is not an option. For example a rule could watch
> certain files in the .git directory to regenerate a version number
> automatically as part of the build if .git changed, and do nothing if the
> .git directory was absent per a tarball dist. This works pretty well. Once
> you have a system that's easy to customize with make-like rules there are
> all sorts of trivial build or housekeeping tasks you might decide to do
> which would never be considered in a more difficult to customize system.

CPython's own build process takes this to extremes by sometimes
bootstrapping a version of Python without an import system in order to
refreeze importlib before continuing with building the normal version.
Argument Clinic (which generates C function argument handling
preambles from specially formatted comments) similarly needs a
pre-existing Python build in order to run.

However, we successfully hide that complexity from folks that just
want to build their own Python from source by checking in the
generated files.

Similarly, it wouldn't astonish me if we eventually see an emergent
practice of people writing pyproject.toml.in files for complex
projects, in order to move some particular forms of complexity away
from build time and towards development time - this would be a similar
practice to folks using autoconf to generate a project's C Makefile.

Party of the beauty of the pyproject.toml format is that as
publication and installation system maintainers, we don't need to care
about this problem beyond ensuring that "maintain it by hand" is a
viable and fully supported option for the majority of projects - as
long as the resulting file gets checked in, folks are free to
autogenerate from another data source if they choose to do so.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia