From burkhardameier at gmail.com  Sun May  1 04:47:16 2016
From: burkhardameier at gmail.com (Burkhard Meier)
Date: Sun, 1 May 2016 01:47:16 -0700
Subject: [Python-Dev] Memory
Message-ID: <CACKxkAwt90YC-_Szh71G9SZrc3N9ZhD6VFD-xcygHnbaiYabww@mail.gmail.com>

~
Die Scheibe klirrt
der Spontie kichert
~
hoffentlich
Alliiance versichert...
~
https://youtu.be/m3SjCzA71eM
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160501/ba157536/attachment.html>

From rymg19 at gmail.com  Sun May  1 09:08:22 2016
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Sun, 1 May 2016 08:08:22 -0500
Subject: [Python-Dev] Memory
In-Reply-To: <CACKxkAwt90YC-_Szh71G9SZrc3N9ZhD6VFD-xcygHnbaiYabww@mail.gmail.com>
References: <CACKxkAwt90YC-_Szh71G9SZrc3N9ZhD6VFD-xcygHnbaiYabww@mail.gmail.com>
Message-ID: <CAO41-mP0+ko0e2qcPX_zqsFFM0pCzg8=R5Ec08TwVz6u=xm0rw@mail.gmail.com>

Well, I put this in Google Translate...and got this:

The disk clatters
the Spontie giggles
~
hopefully
alliance insures ...

Not sure if this a useless post or Translate just being weird. Leaning
towards the latter...

--
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
On May 1, 2016 3:48 AM, "Burkhard Meier" <burkhardameier at gmail.com> wrote:

> ~
> Die Scheibe klirrt
> der Spontie kichert
> ~
> hoffentlich
> Alliiance versichert...
> ~
> https://youtu.be/m3SjCzA71eM
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160501/c7073865/attachment.html>

From brett at python.org  Sun May  1 12:23:55 2016
From: brett at python.org (Brett Cannon)
Date: Sun, 01 May 2016 16:23:55 +0000
Subject: [Python-Dev] Bug in 2to3 concerning import statements?
In-Reply-To: <57255A70.3000404@canterbury.ac.nz>
References: <57255A70.3000404@canterbury.ac.nz>
Message-ID: <CAP1=2W7h02hVtOsa9KZgmGwDgRC3ui=c+pDbOvQdwO9n47OXPA@mail.gmail.com>

On Sat, 30 Apr 2016 at 21:51 Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:

> It seems that 2to3 is a bit simplistic when it comes to
> translating import statements. I have a module GUI.py2exe
> containing:
>
>     import py2exe.mf as modulefinder
>
> 2to3 translates this into:
>
>     from . import py2exe.mf as modulefinder
>
> which is a syntax error.
>
> It looks like 2to3 is getting confused by the fact that
> there is both a submodule and a top-level module here
> called py2exe.


That sounds right. 2to3 has to inspect the environment to figure out if the
import was meant to be relative or absolute and it's getting confused.


> But the original can only be an absolute
> import because it has a dot in it, so 2to3 shouldn't be
> translating it into a relative one.
>

2to3 is nowhere near that sophisticated for tweaking imports :)


>
> Putting "from __future__ import absolute_import" at the
> top fixes it, but I shouldn't have to do that, should I?
>

There's all sorts of weird stuff going on in that import, like having a dot
in the `from` part of the import instead of doing `from .py2exe import mf
as modulefinder`. Try that tweak and see if that solves your SyntaxError
for some reason.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160501/3111d4f6/attachment.html>

From steve at pearwood.info  Sun May  1 12:51:38 2016
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 2 May 2016 02:51:38 +1000
Subject: [Python-Dev] Memory
In-Reply-To: <CAO41-mP0+ko0e2qcPX_zqsFFM0pCzg8=R5Ec08TwVz6u=xm0rw@mail.gmail.com>
References: <CACKxkAwt90YC-_Szh71G9SZrc3N9ZhD6VFD-xcygHnbaiYabww@mail.gmail.com>
 <CAO41-mP0+ko0e2qcPX_zqsFFM0pCzg8=R5Ec08TwVz6u=xm0rw@mail.gmail.com>
Message-ID: <20160501165136.GD13497@ando.pearwood.info>

On Sun, May 01, 2016 at 08:08:22AM -0500, Ryan Gonzalez wrote:
> Well, I put this in Google Translate...and got this:
> 
> The disk clatters
> the Spontie giggles
> ~
> hopefully
> alliance insures ...
> 
> Not sure if this a useless post or Translate just being weird. Leaning
> towards the latter...

You missed the link to David Bowie's "Heroes" on YouTube.

I'm pretty sure that this is a useless post.

Burkhard, please speak English on this list, as best as you are able. 
And please avoid rambling cryptic posts and topics which have nothing to 
do with the development of CPython. If you want to share random 
thoughts, you should get a blog or a Twitter account, don't post them 
here.


Thank you.



-- 
Steve

From cmkleffner at gmail.com  Sun May  1 13:41:07 2016
From: cmkleffner at gmail.com (Carl Kleffner)
Date: Sun, 1 May 2016 19:41:07 +0200
Subject: [Python-Dev] Memory
In-Reply-To: <20160501165136.GD13497@ando.pearwood.info>
References: <CACKxkAwt90YC-_Szh71G9SZrc3N9ZhD6VFD-xcygHnbaiYabww@mail.gmail.com>
 <CAO41-mP0+ko0e2qcPX_zqsFFM0pCzg8=R5Ec08TwVz6u=xm0rw@mail.gmail.com>
 <20160501165136.GD13497@ando.pearwood.info>
Message-ID: <CAGGsPMx16Bv9KZok8W=x47JkF2o+9KSTUATyQwnz-mAmfeu7Fg@mail.gmail.com>

AFAIK Burkhard A. Meier is the author of the Python GUI Programming
Cookbook (Packtpub). I wonder if someone has hacked is gmail account?

2016-05-01 18:51 GMT+02:00 Steven D'Aprano <steve at pearwood.info>:

> On Sun, May 01, 2016 at 08:08:22AM -0500, Ryan Gonzalez wrote:
> > Well, I put this in Google Translate...and got this:
> >
> > The disk clatters
> > the Spontie giggles
> > ~
> > hopefully
> > alliance insures ...
> >
> > Not sure if this a useless post or Translate just being weird. Leaning
> > towards the latter...
>
> You missed the link to David Bowie's "Heroes" on YouTube.
>
> I'm pretty sure that this is a useless post.
>
> Burkhard, please speak English on this list, as best as you are able.
> And please avoid rambling cryptic posts and topics which have nothing to
> do with the development of CPython. If you want to share random
> thoughts, you should get a blog or a Twitter account, don't post them
> here.
>
>
> Thank you.
>
>
>
> --
> Steve
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/cmkleffner%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160501/01c32a2d/attachment.html>

From guido at python.org  Sun May  1 13:48:32 2016
From: guido at python.org (Guido van Rossum)
Date: Sun, 1 May 2016 10:48:32 -0700
Subject: [Python-Dev] Memory
In-Reply-To: <CAGGsPMx16Bv9KZok8W=x47JkF2o+9KSTUATyQwnz-mAmfeu7Fg@mail.gmail.com>
References: <CACKxkAwt90YC-_Szh71G9SZrc3N9ZhD6VFD-xcygHnbaiYabww@mail.gmail.com>
 <CAO41-mP0+ko0e2qcPX_zqsFFM0pCzg8=R5Ec08TwVz6u=xm0rw@mail.gmail.com>
 <20160501165136.GD13497@ando.pearwood.info>
 <CAGGsPMx16Bv9KZok8W=x47JkF2o+9KSTUATyQwnz-mAmfeu7Fg@mail.gmail.com>
Message-ID: <CAP7+vJLy9TZtB_4-HX228DiiXgO7R8r9yKRaUyMv91E++b+_zA@mail.gmail.com>

That would certainly be the most favorable interpretation.

On Sun, May 1, 2016 at 10:41 AM, Carl Kleffner <cmkleffner at gmail.com> wrote:

> AFAIK Burkhard A. Meier is the author of the Python GUI Programming
> Cookbook (Packtpub). I wonder if someone has hacked is gmail account?
>
> 2016-05-01 18:51 GMT+02:00 Steven D'Aprano <steve at pearwood.info>:
>
>> On Sun, May 01, 2016 at 08:08:22AM -0500, Ryan Gonzalez wrote:
>> > Well, I put this in Google Translate...and got this:
>> >
>> > The disk clatters
>> > the Spontie giggles
>> > ~
>> > hopefully
>> > alliance insures ...
>> >
>> > Not sure if this a useless post or Translate just being weird. Leaning
>> > towards the latter...
>>
>> You missed the link to David Bowie's "Heroes" on YouTube.
>>
>> I'm pretty sure that this is a useless post.
>>
>> Burkhard, please speak English on this list, as best as you are able.
>> And please avoid rambling cryptic posts and topics which have nothing to
>> do with the development of CPython. If you want to share random
>> thoughts, you should get a blog or a Twitter account, don't post them
>> here.
>>
>>
>> Thank you.
>>
>>
>>
>> --
>> Steve
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/cmkleffner%40gmail.com
>>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160501/067abd95/attachment.html>

From burkhardameier at gmail.com  Sun May  1 15:19:50 2016
From: burkhardameier at gmail.com (Burkhard Meier)
Date: Sun, 1 May 2016 12:19:50 -0700
Subject: [Python-Dev] Memory
In-Reply-To: <CAP7+vJLy9TZtB_4-HX228DiiXgO7R8r9yKRaUyMv91E++b+_zA@mail.gmail.com>
References: <CACKxkAwt90YC-_Szh71G9SZrc3N9ZhD6VFD-xcygHnbaiYabww@mail.gmail.com>
 <CAO41-mP0+ko0e2qcPX_zqsFFM0pCzg8=R5Ec08TwVz6u=xm0rw@mail.gmail.com>
 <20160501165136.GD13497@ando.pearwood.info>
 <CAGGsPMx16Bv9KZok8W=x47JkF2o+9KSTUATyQwnz-mAmfeu7Fg@mail.gmail.com>
 <CAP7+vJLy9TZtB_4-HX228DiiXgO7R8r9yKRaUyMv91E++b+_zA@mail.gmail.com>
Message-ID: <CACKxkAytXYrEmeXqoTaO06ja1q1_f503Kzc3EF3c7h-Ps-uHbw@mail.gmail.com>

Sorry.  Somebody take the keyboard away from me after midnight.
I think Google Chrome does have that feature.

Definitely Not related to Python dev...

Google translate doesn't correctly understand everything.

My apologies  (again ) for this unrelated post.

And, yes, I am the author of the "Python GUI programming cookbook "

Sorry,
Burkhard


On Sunday, May 1, 2016, Guido van Rossum <guido at python.org> wrote:

> That would certainly be the most favorable interpretation.
>
> On Sun, May 1, 2016 at 10:41 AM, Carl Kleffner <cmkleffner at gmail.com
> <javascript:_e(%7B%7D,'cvml','cmkleffner at gmail.com');>> wrote:
>
>> AFAIK Burkhard A. Meier is the author of the Python GUI Programming
>> Cookbook (Packtpub). I wonder if someone has hacked is gmail account?
>>
>> 2016-05-01 18:51 GMT+02:00 Steven D'Aprano <steve at pearwood.info
>> <javascript:_e(%7B%7D,'cvml','steve at pearwood.info');>>:
>>
>>> On Sun, May 01, 2016 at 08:08:22AM -0500, Ryan Gonzalez wrote:
>>> > Well, I put this in Google Translate...and got this:
>>> >
>>> > The disk clatters
>>> > the Spontie giggles
>>> > ~
>>> > hopefully
>>> > alliance insures ...
>>> >
>>> > Not sure if this a useless post or Translate just being weird. Leaning
>>> > towards the latter...
>>>
>>> You missed the link to David Bowie's "Heroes" on YouTube.
>>>
>>> I'm pretty sure that this is a useless post.
>>>
>>> Burkhard, please speak English on this list, as best as you are able.
>>> And please avoid rambling cryptic posts and topics which have nothing to
>>> do with the development of CPython. If you want to share random
>>> thoughts, you should get a blog or a Twitter account, don't post them
>>> here.
>>>
>>>
>>> Thank you.
>>>
>>>
>>>
>>> --
>>> Steve
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> <javascript:_e(%7B%7D,'cvml','Python-Dev at python.org');>
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/cmkleffner%40gmail.com
>>>
>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> <javascript:_e(%7B%7D,'cvml','Python-Dev at python.org');>
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160501/64c926ea/attachment.html>

From burkhardameier at gmail.com  Sun May  1 17:11:09 2016
From: burkhardameier at gmail.com (Burkhard Meier)
Date: Sun, 1 May 2016 14:11:09 -0700
Subject: [Python-Dev] Looking for a Python programming mentor
Message-ID: <CACKxkAwwf_gfNtarbF2qhEqun3BAmqaLhL8R-LXDr08k9RFQ4g@mail.gmail.com>

I don't need Sigmund Freud...

Burkhard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160501/6c89428c/attachment.html>

From brett at python.org  Sun May  1 17:16:40 2016
From: brett at python.org (Brett Cannon)
Date: Sun, 01 May 2016 21:16:40 +0000
Subject: [Python-Dev] Looking for a Python programming mentor
In-Reply-To: <CACKxkAwwf_gfNtarbF2qhEqun3BAmqaLhL8R-LXDr08k9RFQ4g@mail.gmail.com>
References: <CACKxkAwwf_gfNtarbF2qhEqun3BAmqaLhL8R-LXDr08k9RFQ4g@mail.gmail.com>
Message-ID: <CAP1=2W4DKM9ne_CVxNyFDXNi3wef-=eAXN=QWQQ8uw3oKv-Z9g@mail.gmail.com>

If you mean for Python in general, then Python+list or python-tutor are a
more appropriate mailing lists. If you mean for core development then
core-workflow is where to ask for help.

On Sun, May 1, 2016, 14:11 Burkhard Meier <burkhardameier at gmail.com> wrote:

> I don't need Sigmund Freud...
>
> Burkhard
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160501/625ba673/attachment.html>

From greg.ewing at canterbury.ac.nz  Sun May  1 19:42:35 2016
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 02 May 2016 11:42:35 +1200
Subject: [Python-Dev] Bug in 2to3 concerning import statements?
In-Reply-To: <CAP1=2W7h02hVtOsa9KZgmGwDgRC3ui=c+pDbOvQdwO9n47OXPA@mail.gmail.com>
References: <57255A70.3000404@canterbury.ac.nz>
 <CAP1=2W7h02hVtOsa9KZgmGwDgRC3ui=c+pDbOvQdwO9n47OXPA@mail.gmail.com>
Message-ID: <5726946B.9050806@canterbury.ac.nz>

Brett Cannon wrote:

> There's all sorts of weird stuff going on in that import, like having a 
> dot in the `from` part of the import instead of doing `from .py2exe 
> import mf as modulefinder`.

If I did that, it would try to import mf from the
py2exe submodule rather than the global one.

In hindsight it was probably a bad idea to name
the submodule py2exe, but the code worked as
intended.

My point is that, regardless of anything else,
2to3 shouldn't be generating syntactically
incorrect code. I can understand if it's not
considered worth fixing, but it's still a bug.

-- 
Greg

From guido at python.org  Sun May  1 20:41:19 2016
From: guido at python.org (Guido van Rossum)
Date: Sun, 1 May 2016 17:41:19 -0700
Subject: [Python-Dev] Bug in 2to3 concerning import statements?
In-Reply-To: <5726946B.9050806@canterbury.ac.nz>
References: <57255A70.3000404@canterbury.ac.nz>
 <CAP1=2W7h02hVtOsa9KZgmGwDgRC3ui=c+pDbOvQdwO9n47OXPA@mail.gmail.com>
 <5726946B.9050806@canterbury.ac.nz>
Message-ID: <CAP7+vJK2OMK3mMBUExvhbm0=_fQv31bQSKHZy=WBhdmZU=31rA@mail.gmail.com>

I agree it's a bug. May I encourage you to file it as such on
bugs.python.org? 2to3 is part of the stdlib.

On Sun, May 1, 2016 at 4:42 PM, Greg Ewing <greg.ewing at canterbury.ac.nz>
wrote:

> Brett Cannon wrote:
>
> There's all sorts of weird stuff going on in that import, like having a
>> dot in the `from` part of the import instead of doing `from .py2exe import
>> mf as modulefinder`.
>>
>
> If I did that, it would try to import mf from the
> py2exe submodule rather than the global one.
>
> In hindsight it was probably a bad idea to name
> the submodule py2exe, but the code worked as
> intended.
>
> My point is that, regardless of anything else,
> 2to3 shouldn't be generating syntactically
> incorrect code. I can understand if it's not
> considered worth fixing, but it's still a bug.
>
>
> --
> Greg
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160501/d73e88fa/attachment-0001.html>

From benjamin at python.org  Tue May  3 12:07:21 2016
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 03 May 2016 09:07:21 -0700
Subject: [Python-Dev] 2.7.12 schedule
Message-ID: <1462291641.3972729.596903881.4CD8087B@webmail.messagingengine.com>

I have been reminded that we are due for a 2.7.12 release. I think
Larry's 3.5.2 schedule is perfectly cromulent and am going to piggyback.
Therefore, 2.7.12rc1 will be on June 11th, and the final will be on June
25.

Servus,
Benjamin

From stefan at bytereef.org  Tue May  3 15:06:12 2016
From: stefan at bytereef.org (Stefan Krah)
Date: Tue, 3 May 2016 19:06:12 +0000 (UTC)
Subject: [Python-Dev] Yearly PyPI breakage
Message-ID: <loom.20160503T205904-379@post.gmane.org>


Hello,

Could someone enlighten me which hoops I have to jump through
this year in order to keep pip downloads working?

Collecting cdecimal
  Could not find a version that satisfies the requirement cdecimal (from
versions: )
No matching distribution found for cdecimal
You are using pip version 7.1.2, however version 8.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.


If this continues, I'm going to release a premium version that's
50% faster and only available from bytereef.org or Anaconda.



Stefan Krah


From lukasz at langa.pl  Tue May  3 15:13:52 2016
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Tue, 3 May 2016 12:13:52 -0700
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <loom.20160503T205904-379@post.gmane.org>
References: <loom.20160503T205904-379@post.gmane.org>
Message-ID: <3339680B-83AB-4CFB-B6B2-95AC6D055AA3@langa.pl>

Why don?t you just *host* the files on PyPI?

> On May 3, 2016, at 12:06 PM, Stefan Krah <stefan at bytereef.org> wrote:
> 
> 
> Hello,
> 
> Could someone enlighten me which hoops I have to jump through
> this year in order to keep pip downloads working?
> 
> Collecting cdecimal
>  Could not find a version that satisfies the requirement cdecimal (from
> versions: )
> No matching distribution found for cdecimal
> You are using pip version 7.1.2, however version 8.1.1 is available.
> You should consider upgrading via the 'pip install --upgrade pip' command.
> 
> 
> If this continues, I'm going to release a premium version that's
> 50% faster and only available from bytereef.org or Anaconda.
> 
> 
> 
> Stefan Krah
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/lukasz%40langa.pl

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160503/23693591/attachment.sig>

From brett at python.org  Tue May  3 15:23:41 2016
From: brett at python.org (Brett Cannon)
Date: Tue, 03 May 2016 19:23:41 +0000
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <loom.20160503T205904-379@post.gmane.org>
References: <loom.20160503T205904-379@post.gmane.org>
Message-ID: <CAP1=2W7rEd5Pv0EgN-u1MQ2iLp+Uw0rnA3Vu3p_zdb4X_ZPs5Q@mail.gmail.com>

On Tue, 3 May 2016 at 12:06 Stefan Krah <stefan at bytereef.org> wrote:

>
> Hello,
>
> Could someone enlighten me which hoops I have to jump through
> this year in order to keep pip downloads working?
>
> Collecting cdecimal
>   Could not find a version that satisfies the requirement cdecimal (from
> versions: )
> No matching distribution found for cdecimal
> You are using pip version 7.1.2, however version 8.1.1 is available.
> You should consider upgrading via the 'pip install --upgrade pip' command.
>

The distutils-sig mailing is the best place to try and get help.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160503/4dca7416/attachment.html>

From donald at stufft.io  Tue May  3 15:28:11 2016
From: donald at stufft.io (Donald Stufft)
Date: Tue, 3 May 2016 15:28:11 -0400
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <loom.20160503T205904-379@post.gmane.org>
References: <loom.20160503T205904-379@post.gmane.org>
Message-ID: <A837D6F7-09C7-4EF1-80D1-920E99312B3D@stufft.io>


> On May 3, 2016, at 3:06 PM, Stefan Krah <stefan at bytereef.org> wrote:
> 
> 
> Hello,
> 
> Could someone enlighten me which hoops I have to jump through
> this year in order to keep pip downloads working?
> 
> 


This is off topic for python-dev, but https://www.python.org/dev/peps/pep-0470/#i-can-t-host-my-project-on-pypi-because-of-x-what-should-i-do .

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160503/ebe9cb7b/attachment.sig>

From njs at pobox.com  Tue May  3 16:09:45 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 3 May 2016 13:09:45 -0700
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <loom.20160503T205904-379@post.gmane.org>
References: <loom.20160503T205904-379@post.gmane.org>
Message-ID: <CAPJVwBm45K=UNvB-SQKaZC67xhJ80inwT9rLdAD9WsVmFsNnSA@mail.gmail.com>

On Tue, May 3, 2016 at 12:06 PM, Stefan Krah <stefan at bytereef.org> wrote:
>
> Hello,
>
> Could someone enlighten me which hoops I have to jump through
> this year in order to keep pip downloads working?
>
> Collecting cdecimal
>   Could not find a version that satisfies the requirement cdecimal (from
> versions: )
> No matching distribution found for cdecimal
> You are using pip version 7.1.2, however version 8.1.1 is available.
> You should consider upgrading via the 'pip install --upgrade pip' command.
>
>
> If this continues, I'm going to release a premium version that's
> 50% faster and only available from bytereef.org or Anaconda.

There's no point in making threats -- you're threatening the air. PyPI
is maintained by one overloaded developer (Donald Stufft, sponsored by
HPE as part of their openstack work) with help from a few overloaded,
burned-out volunteers. Everyone wants PyPI to be awesome and useful;
your frustration is totally understandable, and lots of us share it.
But the limitation here is that we have ~one person running a website
that serves ~100,000,000 requests/day, and running as fast as they can
just to keep things from breaking more than they already have. As long
as that's the case it doesn't matter what "should" happen, there's
no-one to do it.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From stefan at bytereef.org  Tue May  3 16:42:46 2016
From: stefan at bytereef.org (Stefan Krah)
Date: Tue, 3 May 2016 20:42:46 +0000 (UTC)
Subject: [Python-Dev] Yearly PyPI breakage
References: <loom.20160503T205904-379@post.gmane.org>
 <CAPJVwBm45K=UNvB-SQKaZC67xhJ80inwT9rLdAD9WsVmFsNnSA@mail.gmail.com>
Message-ID: <loom.20160503T222238-321@post.gmane.org>

Nathaniel Smith <njs <at> pobox.com> writes:
> > If this continues, I'm going to release a premium version that's
> > 50% faster and only available from bytereef.org or Anaconda.
> 
> There's no point in making threats -- you're threatening the air. PyPI
> is maintained by one overloaded developer (Donald Stufft, sponsored by
> HPE as part of their openstack work) with help from a few overloaded,
> burned-out volunteers. Everyone wants PyPI to be awesome and useful;
> your frustration is totally understandable, and lots of us share it.
> But the limitation here is that we have ~one person running a website
> that serves ~100,000,000 requests/day, and running as fast as they can
> just to keep things from breaking more than they already have. As long
> as that's the case it doesn't matter what "should" happen, there's
> no-one to do it.

This wasn't so much of a threat, it was resignation. At some point it
seems like the rational thing to do.

I don't fully understand your explanation. Judging by the link that
Donald posted (thanks!) it seems that PEP 470 introduced extra work
for him that would not have been present had things been left in place.


Also, I did not get any notification and now that I searched for
PEP 470 it seems that it wasn't announced here:

  https://mail.python.org/pipermail/python-dev/2015-October/141838.html


But if the majority prefers PyPI that way, I'll stop arguing.



Stefan Krah






























From lukasz at langa.pl  Tue May  3 17:21:28 2016
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Tue, 3 May 2016 14:21:28 -0700
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <loom.20160503T222238-321@post.gmane.org>
References: <loom.20160503T205904-379@post.gmane.org>
 <CAPJVwBm45K=UNvB-SQKaZC67xhJ80inwT9rLdAD9WsVmFsNnSA@mail.gmail.com>
 <loom.20160503T222238-321@post.gmane.org>
Message-ID: <27B20312-EC4B-44DA-A1F8-090FDC4F42BD@langa.pl>


> On May 3, 2016, at 1:42 PM, Stefan Krah <stefan at bytereef.org> wrote:
> 
> I don't fully understand your explanation. Judging by the link that
> Donald posted (thanks!) it seems that PEP 470 introduced extra work
> for him that would not have been present had things been left in place.

IIRC the PyPI maintainers were constantly nagged about ?PyPI reliability issues? that were instead external hosting issues. Everybody was affected every now and then whenever tummy.com or other external servers for popular packages were down. Or at least I know *I was*. Way too often.

> But if the majority prefers PyPI that way, I'll stop arguing.

I?m not sure what you mean here but if you want to argue for reverting PEP 470, I wouldn?t hold my breath.

--
?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160503/ec9b683d/attachment.sig>

From stefan at bytereef.org  Tue May  3 17:38:13 2016
From: stefan at bytereef.org (Stefan Krah)
Date: Tue, 3 May 2016 21:38:13 +0000 (UTC)
Subject: [Python-Dev] Yearly PyPI breakage
References: <loom.20160503T205904-379@post.gmane.org>
 <CAPJVwBm45K=UNvB-SQKaZC67xhJ80inwT9rLdAD9WsVmFsNnSA@mail.gmail.com>
 <loom.20160503T222238-321@post.gmane.org>
 <27B20312-EC4B-44DA-A1F8-090FDC4F42BD@langa.pl>
Message-ID: <loom.20160503T233139-95@post.gmane.org>

?ukasz Langa <lukasz <at> langa.pl> writes:
> > I don't fully understand your explanation. Judging by the link that
> > Donald posted (thanks!) it seems that PEP 470 introduced extra work
> > for him that would not have been present had things been left in place.
> 
> IIRC the PyPI maintainers were constantly nagged about ?PyPI reliability
issues? that were instead
> external hosting issues. Everybody was affected every now and then
whenever tummy.com or other external
> servers for popular packages were down. Or at least I know *I was*. Way
too often.

But making them completely unreachable does not increase reliability. :)


> > But if the majority prefers PyPI that way, I'll stop arguing.
> 
> I?m not sure what you mean here but if you want to argue for reverting PEP
470, I wouldn?t hold my breath.

No, I mean what we're doing now.


Stefan Krah


From glyph at twistedmatrix.com  Tue May  3 20:56:13 2016
From: glyph at twistedmatrix.com (Glyph)
Date: Tue, 3 May 2016 17:56:13 -0700
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <loom.20160503T233139-95@post.gmane.org>
References: <loom.20160503T205904-379@post.gmane.org>
 <CAPJVwBm45K=UNvB-SQKaZC67xhJ80inwT9rLdAD9WsVmFsNnSA@mail.gmail.com>
 <loom.20160503T222238-321@post.gmane.org>
 <27B20312-EC4B-44DA-A1F8-090FDC4F42BD@langa.pl>
 <loom.20160503T233139-95@post.gmane.org>
Message-ID: <9CAEBE52-101F-4CFD-82AF-4F0A6BAD829C@twistedmatrix.com>

On May 3, 2016, at 2:38 PM, Stefan Krah <stefan at bytereef.org> wrote:
> 
> But making them completely unreachable does not increase reliability. :)

But it does increase security.

The other motivation, besides reliability, listed in this section <https://www.python.org/dev/peps/pep-0470/#my-users-have-a-worse-experience-with-this-pep-than-before-how-do-i-explain-that>, is that:

"transparently including external links [is] a security hazard (given that in most cases it allowed a MITM to execute arbitrary Python code on the end users machine)".

And, indeed, the URL presently listed on PyPI for the cdecimal upload is an unverified http URL.  This means that any evil barista with access to a coffee-shop wifi router could instantly execute user-privileged code on any Python programmer's laptop if they were to `pip install? this externally hosted package, which is one of the reasons why neither `pip? nor `pypi? allow such a thing any more.

Please believe me when I say I do not mean the following to be insulting - information security is incredibly confusing, difficult, and rapidly evolving, and I don't blame you for getting it wrong - but maintaining a popular package in this way is dangerously irresponsible.  There are solid social reasons to centralize the control of the default package repository in the hands of dedicated experts who can scale their security expertise to a large audience, so that package authors like you and I don't need to do this in order to prevent Python from gaining a reputation as a vector for malware; this package is a case in point.

Separately from the issue of how PyPI works, even if you have some reason you need to host it externally (which I seriously doubt), please take the trouble to set up a server with properly verified TLS, or use a '.github.io' hostname that can be verified that way.

In the meanwhile, just to demonstrate that it's a trivial amount of work to just host it on PyPI, I checked out this package via a verified mechanism ("git clone https://github.com/bytereef/bytereef.github.io") and created a new pypi-cdecimal package <https://pypi.python.org/pypi/pypi-cdecimal <https://pypi.python.org/pypi/pypi-cdecimal>>, via editing the setup.py to change the name, 'python setup.py register', 'python setup.py sdist', 'pip wheel' (for some reason direct 'python setup.py bdist_wheel' didn't work), and 'twine upload'.  `pip install pypi-cdecimal? should now work and get you an importable `cdecimal?, and if you happen to be lucky enough to run the same OS version I am, you won't even need to build C code.  cdecimal users may wish to retrieve it via this mechanism until there's a secure way to get the proper upstream distribution.

If anyone wants package-index access to this name to upload Windows or manylinux wheels just let me know; however, as this is just a proof of concept, I do not intend to maintain it long-term.

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160503/6a6020aa/attachment-0001.html>

From tjreedy at udel.edu  Tue May  3 22:07:45 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 3 May 2016 22:07:45 -0400
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <9CAEBE52-101F-4CFD-82AF-4F0A6BAD829C@twistedmatrix.com>
References: <loom.20160503T205904-379@post.gmane.org>
 <CAPJVwBm45K=UNvB-SQKaZC67xhJ80inwT9rLdAD9WsVmFsNnSA@mail.gmail.com>
 <loom.20160503T222238-321@post.gmane.org>
 <27B20312-EC4B-44DA-A1F8-090FDC4F42BD@langa.pl>
 <loom.20160503T233139-95@post.gmane.org>
 <9CAEBE52-101F-4CFD-82AF-4F0A6BAD829C@twistedmatrix.com>
Message-ID: <ngblhl$7c8$1@ger.gmane.org>

On 5/3/2016 8:56 PM, Glyph wrote:

> setup.py bdist_wheel' didn't work), and 'twine upload'.  `pip install
> pypi-cdecimal? should now work and get you an importable `cdecimal?, and
> if you happen to be lucky enough to run the same OS version I am, you
> won't even need to build C code.  cdecimal users may wish to retrieve it
> via this mechanism until there's a secure way to get the proper upstream
> distribution.
>
> If anyone wants package-index access to this name to upload Windows or
> manylinux wheels just let me know; however, as this is just a proof of
> concept, I do not intend to maintain it long-term.

For Windows, http://www.lfd.uci.edu/~gohlke/pythonlibs/ has 32 and 64 
bit builds of cdecimal 2.3 for 2.7, 3.4, and 3.5.

(Is cdecimal substantially different from the _decimal added in 3.5?)

-- 
Terry Jan Reedy



From tritium-list at sdamon.com  Tue May  3 23:44:02 2016
From: tritium-list at sdamon.com (tritium-list at sdamon.com)
Date: Tue, 3 May 2016 23:44:02 -0400
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <ngblhl$7c8$1@ger.gmane.org>
References: <loom.20160503T205904-379@post.gmane.org>
 <CAPJVwBm45K=UNvB-SQKaZC67xhJ80inwT9rLdAD9WsVmFsNnSA@mail.gmail.com>
 <loom.20160503T222238-321@post.gmane.org>
 <27B20312-EC4B-44DA-A1F8-090FDC4F42BD@langa.pl>
 <loom.20160503T233139-95@post.gmane.org>
 <9CAEBE52-101F-4CFD-82AF-4F0A6BAD829C@twistedmatrix.com>
 <ngblhl$7c8$1@ger.gmane.org>
Message-ID: <001e01d1a5b7$33978020$9ac68060$@hotmail.com>

> 
> (Is cdecimal substantially different from the _decimal added in 3.5?)
> 
AFAICT, they are unrelated codebases that do about the same thing with the same amount of performance, with the main exception that _decimal in 3.5 does not require one to change their import (or to compile the package themselves.)


From stefan at bytereef.org  Wed May  4 00:15:10 2016
From: stefan at bytereef.org (Stefan Krah)
Date: Wed, 4 May 2016 04:15:10 +0000 (UTC)
Subject: [Python-Dev] Yearly PyPI breakage
References: <loom.20160503T205904-379@post.gmane.org>
 <CAPJVwBm45K=UNvB-SQKaZC67xhJ80inwT9rLdAD9WsVmFsNnSA@mail.gmail.com>
 <loom.20160503T222238-321@post.gmane.org>
 <27B20312-EC4B-44DA-A1F8-090FDC4F42BD@langa.pl>
 <loom.20160503T233139-95@post.gmane.org>
 <9CAEBE52-101F-4CFD-82AF-4F0A6BAD829C@twistedmatrix.com>
Message-ID: <loom.20160504T060620-46@post.gmane.org>


> [cut overlong post]

Glyph,

nice sneaky way to try to divert from the original issue. Your whole post
is invalidated by the simple fact that the URL was protected by a hash
(which I repeatedly asked to be upgraded to sha256).

This was the official scheme promoted by PEP-438, which you should know.  
But of course your actual intention here is character assassination, 
pretending to "rescue" cdecimal and trying to divert from the fact that
the transition to PEP 470 was handled suboptimally.


The very reason for this thread is that the security was silently disabled
WITHOUT me getting a notification.  What is on PyPI *now* is not what I
configured!


Please believe me when I say I do not mean the following to be insulting --
people who have done *actual* cryptography to varying degrees often tend
to focus on the important parts and aren't impressed by regurgitating
catch phrases like SSL and man-in-the-middle:

    http://cr.yp.to/ecdh.html


The amount of security "experts" in the Python community that pontificate
on any occasion is pretty annoying.  What do you think djb thinks of Twisted?


> If anyone wants package-index access to this name to upload Windows or
manylinux wheels just let me know; however, as this is just a proof of
concept, I do not intend to maintain it long-term.

That apparently all you can do:  Move bits from place A to place B and not 
care how long it took to produce them.

You are a real hero.



Stefan Krah





From tritium-list at sdamon.com  Wed May  4 02:39:51 2016
From: tritium-list at sdamon.com (tritium-list at sdamon.com)
Date: Wed, 4 May 2016 02:39:51 -0400
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <loom.20160504T060620-46@post.gmane.org>
References: <loom.20160503T205904-379@post.gmane.org>
 <CAPJVwBm45K=UNvB-SQKaZC67xhJ80inwT9rLdAD9WsVmFsNnSA@mail.gmail.com>
 <loom.20160503T222238-321@post.gmane.org>
 <27B20312-EC4B-44DA-A1F8-090FDC4F42BD@langa.pl>
 <loom.20160503T233139-95@post.gmane.org>
 <9CAEBE52-101F-4CFD-82AF-4F0A6BAD829C@twistedmatrix.com>
 <loom.20160504T060620-46@post.gmane.org>
Message-ID: <003c01d1a5cf$c33133d0$49939b70$@hotmail.com>

Are you for real?  I honestly do not understand your hostility.

You posted a mean-spirited complaint about a policy that is nearly exactly
two years old, to the wrong list, and call out the people calmly trying to
explain what happened and why, and how you can mitigate it for your own work
and organization.  What do you intend to accomplish?

* PyPI is no longer and index, it is a repository
* The decision to disable the index-only features of PyPI were made 2 years
ago, including pep438 - plenty of time to make alternate arrangements.
* The tooling for hosting your own repository is available, should you
actually need to host the files outside of PyPI
* The tooling exists to use other indexes
* The tooling exists to host your own index that serves your own packages
(that you develop or third party packages that you package for your own
use), that defaults to PyPI for packages not in your own repository

I understand that you are upset that a feature you used was removed; posting
with hostility to a list of people who do not even have control over the
repository is not a legitimate way to solve your problems.

> -----Original Message-----
> From: Python-Dev [mailto:python-dev-bounces+tritium-
> list=sdamon.com at python.org] On Behalf Of Stefan Krah
> Sent: Wednesday, May 04, 2016 00:15
> To: python-dev at python.org
> Subject: Re: [Python-Dev] Yearly PyPI breakage
> 
> 
> > [cut overlong post]
> 
> Glyph,
> 
> nice sneaky way to try to divert from the original issue. Your whole post
> is invalidated by the simple fact that the URL was protected by a hash
> (which I repeatedly asked to be upgraded to sha256).
> 
> This was the official scheme promoted by PEP-438, which you should know.
> But of course your actual intention here is character assassination,
> pretending to "rescue" cdecimal and trying to divert from the fact that
> the transition to PEP 470 was handled suboptimally.
> 
> 
> The very reason for this thread is that the security was silently disabled
> WITHOUT me getting a notification.  What is on PyPI *now* is not what I
> configured!
> 
> 
> Please believe me when I say I do not mean the following to be insulting
--
> people who have done *actual* cryptography to varying degrees often tend
> to focus on the important parts and aren't impressed by regurgitating
> catch phrases like SSL and man-in-the-middle:
> 
>     http://cr.yp.to/ecdh.html
> 
> 
> The amount of security "experts" in the Python community that pontificate
> on any occasion is pretty annoying.  What do you think djb thinks of
Twisted?
> 
> 
> > If anyone wants package-index access to this name to upload Windows or
> manylinux wheels just let me know; however, as this is just a proof of
> concept, I do not intend to maintain it long-term.
> 
> That apparently all you can do:  Move bits from place A to place B and not
> care how long it took to produce them.
> 
> You are a real hero.
> 
> 
> 
> Stefan Krah
> 
> 
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium-
> list%40sdamon.com


From stefan at bytereef.org  Wed May  4 03:23:50 2016
From: stefan at bytereef.org (Stefan Krah)
Date: Wed, 4 May 2016 07:23:50 +0000 (UTC)
Subject: [Python-Dev] Yearly PyPI breakage
References: <loom.20160503T205904-379@post.gmane.org>
 <CAPJVwBm45K=UNvB-SQKaZC67xhJ80inwT9rLdAD9WsVmFsNnSA@mail.gmail.com>
 <loom.20160503T222238-321@post.gmane.org>
 <27B20312-EC4B-44DA-A1F8-090FDC4F42BD@langa.pl>
 <loom.20160503T233139-95@post.gmane.org>
 <9CAEBE52-101F-4CFD-82AF-4F0A6BAD829C@twistedmatrix.com>
 <loom.20160504T060620-46@post.gmane.org>
 <003c01d1a5cf$c33133d0$49939b70$@hotmail.com>
Message-ID: <loom.20160504T092021-174@post.gmane.org>

 <tritium-list <at> sdamon.com> writes:
> You posted a mean-spirited complaint about a policy that is nearly exactly
> two years old, to the wrong list, and call out the people calmly trying to
> explain what happened and why, and how you can mitigate it for your own work

Yeah, right.  This is distutils-sig level.  Read Glyph's "calm" and "nice"
post again.  Also note that *I* had already bowed out of the thread when
Glyph unnecessarily decided to spread complete misinformation.


Stefan Krah






From glyph at twistedmatrix.com  Wed May  4 03:38:05 2016
From: glyph at twistedmatrix.com (Glyph)
Date: Wed, 4 May 2016 00:38:05 -0700
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <loom.20160504T060620-46@post.gmane.org>
References: <loom.20160503T205904-379@post.gmane.org>
 <CAPJVwBm45K=UNvB-SQKaZC67xhJ80inwT9rLdAD9WsVmFsNnSA@mail.gmail.com>
 <loom.20160503T222238-321@post.gmane.org>
 <27B20312-EC4B-44DA-A1F8-090FDC4F42BD@langa.pl>
 <loom.20160503T233139-95@post.gmane.org>
 <9CAEBE52-101F-4CFD-82AF-4F0A6BAD829C@twistedmatrix.com>
 <loom.20160504T060620-46@post.gmane.org>
Message-ID: <53C056CE-390B-4473-A383-DBC2599A5071@twistedmatrix.com>

On May 3, 2016, at 9:15 PM, Stefan Krah <stefan at bytereef.org> wrote:
> 
>> [cut overlong post]
> 
> Glyph,
> 
> nice sneaky way to try to divert from the original issue.

The original issue, as I understood it, at the start of the thread was "which hoops I have to jump through this year in order to keep pip downloads working".  So I showed you the hoops.

Your question implied, to me, that you were not presently aware of how easy it is to simply build and upload your packages with sdist and twine.  After years and years of horrible setuptools bugs, it certainly seemed plausible to me that if you had long-standing experience with python packaging, you might have perhaps developed a well-deserved low opinion of them in the past.  Therefore, you might not be aware of relatively recent improvements to the packaging ecosystem which made this problem trivial to solve.

My intent was, therefore, simply to demonstrate that things have improved, and that this was not a hard thing for you to do and could be resolved with a minimum of fuss.

I confess that before posting I was made aware that you'd had some personality conflicts with some PyPI maintainers in the past.  But that sentence was about the extent and detail level of my understanding.  I was not aware to what extent, and the reason I jumped into this particular thread, when I rarely participate in python-dev, was that I hoped a simple explanation of the facts of the matter from someone you hadn't previously interacted with could address your concerns.

> Your whole post is invalidated by the simple fact that the URL was protected by a hash (which I repeatedly asked to be upgraded to sha256).

Based only on previous discussion here, I had no way to know either of those things.  You didn't reference it in the post I was replying to, or in your original post.  And, as you say later, PyPI's download URL doesn't include the hash any more, so it wasn't there for me to observe.  (There were some manual instructions in your package description but no automated tooling will honor that.)  In any case, fragment hashes are not really a suitable general-purpose mechanism as they are only honored by specific tools (like pip) whereas HTTPS verification ought to be universally supported, so IMHO it is a good thing that PyPI is discouraging their use for this purpose.

> This was the official scheme promoted by PEP-438, which you should know.  But of course your actual intention here is character assassination, pretending to "rescue" cdecimal

In the "overlong" post that you elided, I specifically said I didn't intend to maintain it for long. If this wasn't clear, what I meant to say by that comment was that I would keep the index entry available until you had the opportunity to upload some sdists and wheels yourself to PyPI.  If you don't intend to, I am not the right person to "rescue" the package; someone else who is more invested in cdecimal should provide an alternate PyPI entry, or take over this one.

> and trying to divert from the fact that
> the transition to PEP 470 was handled suboptimally.

I don't see any need to divert attention from this fact, because you appear to be in a minority of one in considering it so.

> The very reason for this thread is that the security was silently disabled WITHOUT me getting a notification.  What is on PyPI *now* is not what I configured!

If that was the reason for the thread, you would have been better served by making that specific complaint rather than asking for information, and then yelling at the people who provided it to you.  You might also consider reporting these issues to an appropriate forum, since python-dev is not the bugtracker for PyPI.  You can find that here: <https://bitbucket.org/pypa/pypi/issues <https://bitbucket.org/pypa/pypi/issues>>.  You might also want to continue this thread on distutils-sig; I'm sorry for contributing to the noise on python-dev, but I thought getting a high-profile package such as cdecimal integrated into the modern packaging ecosystem would be worth the off-topic digression.

> [various spurious and irrelevant ad-hominem attacks redacted]


Perhaps naively, given the level of hostility on display here, I still hope that you might see the wisdom in simply uploading build artifacts to PyPI.  But I won't try to convince you further.

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160504/56d40b9b/attachment.html>

From torkvemada at sorokdva.net  Wed May  4 14:37:40 2016
From: torkvemada at sorokdva.net (=?UTF-8?B?0JLRgdC10LLQvtC70L7QtCDQktC10LvQuNGH0LrQvg==?=)
Date: Wed, 4 May 2016 21:37:40 +0300
Subject: [Python-Dev] Restore functions pickling
Message-ID: <CAJGNwtV7H4Mf0wx9vLX_6VP+a7TV_4ZALE16r8jO2tYiCuqcBw@mail.gmail.com>

Hello,

I was recently porting my application from Py2 to Py3 and encountered
error, when I accidentally became unable to define my own pickling
mechanism for functions.
Before that I was using it to pickle pure lambdas, registering my own
handler for cPickle with copy_reg, but after [1] (and release 3.4) it seems
that feature has been removed.

As it mentioned in commit message, this was done for compatibility with
pure-python pickle. So I have to ask, may be it would be better to revert
the commit and add the corresponding feature to pure-python pickle
implementation?


[1] https://hg.python.org/cpython/rev/6bd1f0a27e8e

-- 
Best wishes,
Vsevolod Velichko
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160504/3da6e9fc/attachment.html>

From brett at python.org  Wed May  4 14:55:32 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 04 May 2016 18:55:32 +0000
Subject: [Python-Dev] Restore functions pickling
In-Reply-To: <CAJGNwtV7H4Mf0wx9vLX_6VP+a7TV_4ZALE16r8jO2tYiCuqcBw@mail.gmail.com>
References: <CAJGNwtV7H4Mf0wx9vLX_6VP+a7TV_4ZALE16r8jO2tYiCuqcBw@mail.gmail.com>
Message-ID: <CAP1=2W4aicQ3p+Dn6+TGZ99PvEMSJdqbxGVS7E8wgZp9KnpvOQ@mail.gmail.com>

On Wed, 4 May 2016 at 11:39 ???????? ??????? <torkvemada at sorokdva.net>
wrote:

> Hello,
>
> I was recently porting my application from Py2 to Py3 and encountered
> error, when I accidentally became unable to define my own pickling
> mechanism for functions.
> Before that I was using it to pickle pure lambdas, registering my own
> handler for cPickle with copy_reg, but after [1] (and release 3.4) it seems
> that feature has been removed.
>
> As it mentioned in commit message, this was done for compatibility with
> pure-python pickle. So I have to ask, may be it would be better to revert
> the commit and add the corresponding feature to pure-python pickle
> implementation?
>
>
> [1] https://hg.python.org/cpython/rev/6bd1f0a27e8e
>

If you want to provide a patch to implement function pickling in pure
Python code and make this work with both pickle.py and _pickle.c we might
be up for accepting the patch. You can also open an issue requesting
pickling support for functions at bugs.python.org to keep track of the
request and see if anyone else is interested in the feature.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160504/3f3e9bdf/attachment.html>

From torkvemada at sorokdva.net  Wed May  4 20:23:15 2016
From: torkvemada at sorokdva.net (=?UTF-8?B?0JLRgdC10LLQvtC70L7QtCDQktC10LvQuNGH0LrQvg==?=)
Date: Thu, 5 May 2016 03:23:15 +0300
Subject: [Python-Dev] Restore functions pickling
In-Reply-To: <CAP1=2W4aicQ3p+Dn6+TGZ99PvEMSJdqbxGVS7E8wgZp9KnpvOQ@mail.gmail.com>
References: <CAJGNwtV7H4Mf0wx9vLX_6VP+a7TV_4ZALE16r8jO2tYiCuqcBw@mail.gmail.com>
 <CAP1=2W4aicQ3p+Dn6+TGZ99PvEMSJdqbxGVS7E8wgZp9KnpvOQ@mail.gmail.com>
Message-ID: <CAJGNwtUarWDs_vhK1GpMPHLmNFsW3jtKhrD28GkhWVBZXm-h5A@mail.gmail.com>

Super, since you are not against this feature, I created the issue with the
pach[1].
Hopefully you or one of your colleagues will have some time to review it.

(I signed the contributor form, changes just not reached the bugtracker yet)

[1] http://bugs.python.org/issue26959

2016-05-04 21:55 GMT+03:00 Brett Cannon <brett at python.org>:

>
>
> On Wed, 4 May 2016 at 11:39 ???????? ??????? <torkvemada at sorokdva.net>
> wrote:
>
>> Hello,
>>
>> I was recently porting my application from Py2 to Py3 and encountered
>> error, when I accidentally became unable to define my own pickling
>> mechanism for functions.
>> Before that I was using it to pickle pure lambdas, registering my own
>> handler for cPickle with copy_reg, but after [1] (and release 3.4) it seems
>> that feature has been removed.
>>
>> As it mentioned in commit message, this was done for compatibility with
>> pure-python pickle. So I have to ask, may be it would be better to revert
>> the commit and add the corresponding feature to pure-python pickle
>> implementation?
>>
>>
>> [1] https://hg.python.org/cpython/rev/6bd1f0a27e8e
>>
>
> If you want to provide a patch to implement function pickling in pure
> Python code and make this work with both pickle.py and _pickle.c we might
> be up for accepting the patch. You can also open an issue requesting
> pickling support for functions at bugs.python.org to keep track of the
> request and see if anyone else is interested in the feature.
>



-- 
Best wishes,
Vsevolod Velichko
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160505/1261cee9/attachment.html>

From ncoghlan at gmail.com  Wed May  4 21:28:29 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 5 May 2016 11:28:29 +1000
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <loom.20160503T205904-379@post.gmane.org>
References: <loom.20160503T205904-379@post.gmane.org>
Message-ID: <CADiSq7ciVgXSStb+353HCF2_kr__DU4qSnHk+vBP31NZW-rK4g@mail.gmail.com>

On 4 May 2016 at 05:06, Stefan Krah <stefan at bytereef.org> wrote:
>
> Hello,
>
> Could someone enlighten me which hoops I have to jump through
> this year in order to keep pip downloads working?

Stefan,

I know you're not happy with myself and the other distutils-sig folks
regarding the decision to deprecate and remove automatic link
spidering, nor with the PSF regarding the current Terms of Service
around uploads to PyPI, but that doesn't make it OK to start off-topic
threads on python-dev just because you're a CPython core developer in
addition to being a PyPI user.

It *definitely* doesn't make it OK to accuse people of conspiring
against you when they answer your question in good faith, just because
their answer is the official distutils-sig/PyPA one (which was
approved through the PEP process in PEP 470).

Depending on which aspect is most bothering you right now, potential
courses of action worth pursuing include:

- writing to psf-legal to let them know whether or not Van Lindberg's
draft updates to the Terms of Service would be sufficient to make you
comfortable with uploading cdecimal to PyPI in addition to bundling it
with the standard library under your existing Contributor Licensing
Agreement: https://bitbucket.org/vanl/pypi/src/default/templates/confirm.pt

- writing to distutils-sig to see what opportunities exist to assist
with the migration from the fragile legacy PyPI codebase to the new
Warehouse implementation (which is a technical debt reduction activity
currently blocking a lot of other enhancements to the PyPI ecosystem)

- writing to the PSF Board asking us to increase the PSF's level of
investment in PyPI infrastructure support (while we're already aware
of that need, we're also the right venue to escalate these kinds of
concerns for folks that don't have the time or inclination to get
directly involved in improving the situation themselves)

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed May  4 22:12:19 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 5 May 2016 12:12:19 +1000
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <001e01d1a5b7$33978020$9ac68060$@hotmail.com>
References: <loom.20160503T205904-379@post.gmane.org>
 <CAPJVwBm45K=UNvB-SQKaZC67xhJ80inwT9rLdAD9WsVmFsNnSA@mail.gmail.com>
 <loom.20160503T222238-321@post.gmane.org>
 <27B20312-EC4B-44DA-A1F8-090FDC4F42BD@langa.pl>
 <loom.20160503T233139-95@post.gmane.org>
 <9CAEBE52-101F-4CFD-82AF-4F0A6BAD829C@twistedmatrix.com>
 <ngblhl$7c8$1@ger.gmane.org>
 <001e01d1a5b7$33978020$9ac68060$@hotmail.com>
Message-ID: <CADiSq7dHS9_GMjbii=EzxQEcLSWdaR7LzhdUgzX+_Hejc6K+dg@mail.gmail.com>

On 4 May 2016 at 13:44,  <tritium-list at sdamon.com> wrote:
>>
>> (Is cdecimal substantially different from the _decimal added in 3.5?)
>>
> AFAICT, they are unrelated codebases that do about the same thing with the same amount of performance, with the main exception that _decimal in 3.5 does not require one to change their import (or to compile the package themselves.)

cdecimal and the standard library's _decimal module are built around
the same decimal arithmetic library (libmpdec), and Stefan is the
maintainer for all of them. Similar to other standard library modules
with a PyPI counterpart, end users can choose between using the
standard library version (and avoiding the external dependency) and
using the independently updated version (and gaining increased
consistency across Python versions, including availability on 2.7).

More info on that can be found in the Python 3.3 What's New doc:
https://docs.python.org/3/whatsnew/3.3.html#new-decimal

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From d.srivastava15 at gmail.com  Thu May  5 04:24:04 2016
From: d.srivastava15 at gmail.com (Deepak Srivastava)
Date: Thu, 5 May 2016 13:54:04 +0530
Subject: [Python-Dev] Need help in OCR implementation with Python3.5.1 or
 pyCharm Community edition
References: <A8B5776C3825FD4A80CFDCF411A8CC5C04CAF23E8C@WMUCV491.wwg00m.rootdom.net>
Message-ID: <576BF72E-99C2-44BF-9AB7-5058ECACA237@gmail.com>


> Dear All,
> I am new to python and very much excited to learn this technology.
> I have done setup of python 3.5.1 with PyCharm community edition on windows 7(64bit service pack 1).
> I am trying to execute some piece of code but it fails . Requesting you to please help.
>  
> My Piece of code is
> import pytesseract
> from PIL import Image
> from PIL import ImageFilter
> import os
> print (os.getcwd())
> url="Polish.png"
> def process_image(url):
>     image = _get_image(url)
>     image = image.filter(ImageFilter.SHARPEN)
> #    image = image.convert('1')
>     print (pytesseract.image_to_string(image))
> 
> def _get_image(url):
>     return Image.open(url)
> 
> process_image(url)
>  
> Error which I am getting ,
> C:\Users\Administrator\AppData\Local\Programs\Python\Python35-32\python.exe C:/Temp/Opt/src/ocr.py
> C:\Temp\Opt\src
> in subprocess
> None
> 0
> None
> None
> <subprocess.STARTUPINFO object at 0x02EA4BF0>
> Traceback (most recent call last):
>   File "C:/Temp/Opt/src/ocr.py", line 17, in <module>
>     process_image(url)
>   File "C:/Temp/Opt/src/ocr.py", line 12, in process_image
>     print (pytesseract.image_to_string(image))
>   File "C:\Users\Administrator\AppData\Roaming\Python\Python35\site-packages\pytesseract\pytesseract.py", line 161, in image_to_string
>     config=config)
>   File "C:\Users\Administrator\AppData\Roaming\Python\Python35\site-packages\pytesseract\pytesseract.py", line 94, in run_tesseract
>     stderr=subprocess.PIPE)
>   File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35-32\lib\subprocess.py", line 950, in __init__
>     restore_signals, start_new_session)
>   File "C:\Users\Administrator\AppData\Local\Programs\Python\Python35-32\lib\subprocess.py", line 1226, in _execute_child
>     startupinfo)
> FileNotFoundError: [WinError 2] The system cannot find the file specified
>  
> Process finished with exit code 1
>  
> Thanks and Regards
> Deepak Srivastava
>  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160505/e41c3c1c/attachment.html>

From rymg19 at gmail.com  Thu May  5 11:26:47 2016
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Thu, 5 May 2016 10:26:47 -0500
Subject: [Python-Dev] Need help in OCR implementation with Python3.5.1
 or pyCharm Community edition
In-Reply-To: <576BF72E-99C2-44BF-9AB7-5058ECACA237@gmail.com>
References: <A8B5776C3825FD4A80CFDCF411A8CC5C04CAF23E8C@WMUCV491.wwg00m.rootdom.net>
 <576BF72E-99C2-44BF-9AB7-5058ECACA237@gmail.com>
Message-ID: <CAO41-mMN2GdhjMFV_azNExQcVSOt9RqvqbHtc2bFr5U4012vSg@mail.gmail.com>

On Thu, May 5, 2016 at 3:24 AM, Deepak Srivastava <d.srivastava15 at gmail.com>
wrote:

>
> Dear All,
>
> I am new to python and very much excited to learn this technology.
>
> I have done setup of python 3.5.1 with PyCharm community edition on
> windows 7(64bit service pack 1).
>
> I am trying to execute some piece of code but it fails . Requesting you to
> please help.
>
>
>
> My Piece of code is
>
> *import *pytesseract
> *from *PIL *import *Image
> *from *PIL *import *ImageFilter
> *import *os
> print (os.getcwd())
> url=
> *"Polish.png"**def *process_image(*url*):
>     image = _get_image(*url*)
>     image = image.filter(ImageFilter.SHARPEN)
> #    image = image.convert('1')
>     print (pytesseract.image_to_string(image))
>
> *def *_get_image(*url*):
>     *return *Image.open(*url*)
>
> process_image(url)
>
>
>
> Error which I am getting ,
>
> *C:\Users\Administrator\AppData\Local\Programs\Python\Python35-32\python.exe
> C:/Temp/Opt/src/ocr.py*
>
> *C:\Temp\Opt\src*
>
> *in subprocess*
>
> *None*
>
> *0*
>
> *None*
>
> *None*
>
> *<subprocess.STARTUPINFO object at 0x02EA4BF0>*
>
> *Traceback (most recent call last):*
>
> *  File "C:/Temp/Opt/src/ocr.py", line 17, in <module>*
>
> *    process_image(url)*
>
> *  File "C:/Temp/Opt/src/ocr.py", line 12, in process_image*
>
> *    print (pytesseract.image_to_string(image))*
>
> *  File
> "C:\Users\Administrator\AppData\Roaming\Python\Python35\site-packages\pytesseract\pytesseract.py",
> line 161, in image_to_string*
>
> *    config=config)*
>
> *  File
> "C:\Users\Administrator\AppData\Roaming\Python\Python35\site-packages\pytesseract\pytesseract.py",
> line 94, in run_tesseract*
>
> *    stderr=subprocess.PIPE)*
>
> *  File
> "C:\Users\Administrator\AppData\Local\Programs\Python\Python35-32\lib\subprocess.py",
> line 950, in __init__*
>
> *    restore_signals, start_new_session)*
>
> *  File
> "C:\Users\Administrator\AppData\Local\Programs\Python\Python35-32\lib\subprocess.py",
> line 1226, in _execute_child*
>
> *    startupinfo)*
>
> *FileNotFoundError: [WinError 2] The system cannot find the file specified*
>
>
>
> *Process finished with exit code 1*
>
>
>
> Thanks and Regards
>
> Deepak Srivastava
>
>
>
>
Questions like this are better suited for python-list. Regardless, you need
to install Tesseract first:

https://github.com/tesseract-ocr/tesseract/wiki

That should fix the error.


>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
>
>


-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160505/47ed053c/attachment-0001.html>

From python at mrabarnett.plus.com  Thu May  5 12:23:02 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Thu, 5 May 2016 17:23:02 +0100
Subject: [Python-Dev] Need help in OCR implementation with Python3.5.1
 or pyCharm Community edition
In-Reply-To: <CAO41-mMN2GdhjMFV_azNExQcVSOt9RqvqbHtc2bFr5U4012vSg@mail.gmail.com>
References: <A8B5776C3825FD4A80CFDCF411A8CC5C04CAF23E8C@WMUCV491.wwg00m.rootdom.net>
 <576BF72E-99C2-44BF-9AB7-5058ECACA237@gmail.com>
 <CAO41-mMN2GdhjMFV_azNExQcVSOt9RqvqbHtc2bFr5U4012vSg@mail.gmail.com>
Message-ID: <dee067bd-0f1f-ac59-6657-e986c488b953@mrabarnett.plus.com>

On 2016-05-05 16:26, Ryan Gonzalez wrote:
> On Thu, May 5, 2016 at 3:24 AM, Deepak Srivastava 
> <d.srivastava15 at gmail.com <mailto:d.srivastava15 at gmail.com>> wrote:
>
>
>>     Dear All,
>>
>>     I am new to python and very much excited to learn this technology.
>>
>>     I have done setup of python 3.5.1 with PyCharm community edition
>>     on windows 7(64bit service pack 1).
>>
>>     I am trying to execute some piece of code but it fails .
>>     Requesting you to please help.
>>
>>     My Piece of code is
>>
>>     /import /pytesseract
>>     /from /PIL /import /Image
>>     /from /PIL /import /ImageFilter
>>     /import /os
>>     print (os.getcwd())
>>     url=*"Polish.png"
>>     */def /process_image(/url/):
>>     image = _get_image(/url/)
>>         image = image.filter(ImageFilter.SHARPEN)
>>     #    image = image.convert('1')
>>     print (pytesseract.image_to_string(image))
>>
>>     /def /_get_image(/url/):
>>     /return /Image.open(/url/)
>>
>>     process_image(url)
>>
>>     Error which I am getting ,
>>
>>     *C:\Users\Administrator\AppData\Local\Programs\Python\Python35-32\python.exe
>>     C:/Temp/Opt/src/ocr.py*
>>
>>     *C:\Temp\Opt\src*
>>
>>     *in subprocess*
>>
>>     *None*
>>
>>     *0*
>>
>>     *None*
>>
>>     *None*
>>
>>     *<subprocess.STARTUPINFO object at 0x02EA4BF0>*
>>
>>     *Traceback (most recent call last):*
>>
>>     *  File "C:/Temp/Opt/src/ocr.py", line 17, in <module>*
>>
>>     *process_image(url)*
>>
>>     *  File "C:/Temp/Opt/src/ocr.py", line 12, in process_image*
>>
>>     *    print (pytesseract.image_to_string(image))*
>>
>>     *  File
>>     "C:\Users\Administrator\AppData\Roaming\Python\Python35\site-packages\pytesseract\pytesseract.py",
>>     line 161, in image_to_string*
>>
>>     *config=config)*
>>
>>     *  File
>>     "C:\Users\Administrator\AppData\Roaming\Python\Python35\site-packages\pytesseract\pytesseract.py",
>>     line 94, in run_tesseract*
>>
>>     *stderr=subprocess.PIPE)*
>>
>>     *  File
>>     "C:\Users\Administrator\AppData\Local\Programs\Python\Python35-32\lib\subprocess.py",
>>     line 950, in __init__*
>>
>>     *restore_signals, start_new_session)*
>>
>>     *  File
>>     "C:\Users\Administrator\AppData\Local\Programs\Python\Python35-32\lib\subprocess.py",
>>     line 1226, in _execute_child*
>>
>>     *    startupinfo)*
>>
>>     *FileNotFoundError: [WinError 2] The system cannot find the file
>>     specified*
>>
>>     **
>>
>>     *Process finished with exit code 1*
>>
>>     Thanks and Regards
>>
>>     Deepak Srivastava
>>
>
> Questions like this are better suited for python-list. Regardless, you 
> need to install Tesseract first:
>
> https://github.com/tesseract-ocr/tesseract/wiki
>
> That should fix the error.
It looks to me that it's the url="Polish.png" that's the real cause of 
the problem. It's not clear where the file "Polish.png" is actually 
stored. It would be better to use a full file path.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160505/e78f32ee/attachment.html>

From steve at pearwood.info  Thu May  5 12:51:35 2016
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 6 May 2016 02:51:35 +1000
Subject: [Python-Dev] Need help in OCR implementation with Python3.5.1
 or pyCharm Community edition
In-Reply-To: <dee067bd-0f1f-ac59-6657-e986c488b953@mrabarnett.plus.com>
References: <A8B5776C3825FD4A80CFDCF411A8CC5C04CAF23E8C@WMUCV491.wwg00m.rootdom.net>
 <576BF72E-99C2-44BF-9AB7-5058ECACA237@gmail.com>
 <CAO41-mMN2GdhjMFV_azNExQcVSOt9RqvqbHtc2bFr5U4012vSg@mail.gmail.com>
 <dee067bd-0f1f-ac59-6657-e986c488b953@mrabarnett.plus.com>
Message-ID: <20160505165135.GQ12028@ando.pearwood.info>

On Thu, May 05, 2016 at 05:23:02PM +0100, MRAB wrote:
> On 2016-05-05 16:26, Ryan Gonzalez wrote:
> >On Thu, May 5, 2016 at 3:24 AM, Deepak Srivastava 

[ ... about three or four pages of quoting ... ]

> >Questions like this are better suited for python-list. [...]
> It looks to me that it's the url="Polish.png" [...]

Guys, it's bad enough that you're engaging in problem-solving mode for 
an off-topic question, but could you at least trim your replies first? 
There's no need to quote the entire body of the OP's post (three or four 
pages worth) only to add a two or three line comment to the end.

Have pity on your readers and use the delete key on some of the 
unnecessary text, please?


-- 
Steve

From zreed at fastmail.com  Thu May  5 14:28:22 2016
From: zreed at fastmail.com (zreed at fastmail.com)
Date: Thu, 05 May 2016 13:28:22 -0500
Subject: [Python-Dev] Wordcode: new regular bytecode using 16-bit units
Message-ID: <1462472902.2822020.599321113.046AABAF@webmail.messagingengine.com>

Here is something I wrote because I was also unsatisfied with byteplay's
API: https://github.com/zachariahreed/byteasm. Maybe it's useful in a
discussion of "minimum viable" api for bytecode manipulation.




From rymg19 at gmail.com  Thu May  5 15:18:10 2016
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Thu, 5 May 2016 14:18:10 -0500
Subject: [Python-Dev] Need help in OCR implementation with Python3.5.1
 or pyCharm Community edition
In-Reply-To: <dee067bd-0f1f-ac59-6657-e986c488b953@mrabarnett.plus.com>
References: <A8B5776C3825FD4A80CFDCF411A8CC5C04CAF23E8C@WMUCV491.wwg00m.rootdom.net>
 <576BF72E-99C2-44BF-9AB7-5058ECACA237@gmail.com>
 <CAO41-mMN2GdhjMFV_azNExQcVSOt9RqvqbHtc2bFr5U4012vSg@mail.gmail.com>
 <dee067bd-0f1f-ac59-6657-e986c488b953@mrabarnett.plus.com>
Message-ID: <CAO41-mMm6SDNRhj+RyCqcrLYHK7Ga3DmVk7yYvxzCC2AQ-H2jw@mail.gmail.com>

Well, the stack trace was pointing to the line that called Tesseract, so I
figured that was the problem.

--
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
On May 5, 2016 11:24 AM, "MRAB" <python at mrabarnett.plus.com> wrote:
>
> It looks to me that it's the url="Polish.png" that's the real cause of
the problem. It's not clear where the file "Polish.png" is actually stored.
It would be better to use a full file path.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160505/926936be/attachment.html>

From stefan at bytereef.org  Thu May  5 18:22:01 2016
From: stefan at bytereef.org (Stefan Krah)
Date: Thu, 5 May 2016 22:22:01 +0000 (UTC)
Subject: [Python-Dev] Yearly PyPI breakage
References: <loom.20160503T205904-379@post.gmane.org>
 <CADiSq7ciVgXSStb+353HCF2_kr@mail.gmail.com>
Message-ID: <loom.20160506T001122-774@post.gmane.org>

Nick Coghlan <ncoghlan <at> gmail.com> writes:
> I know you're not happy with myself and the other distutils-sig folks
> regarding the decision to deprecate and remove automatic link
> spidering,

More accurately: Not happy with the removal of the checksummed "explicit"
mode. What I would have preferred is a FreeBSD-like ports system. FreeBSD
has been used in huge and secure installations, so the I don't buy the
reliability and security arguments that are used in favor of centralization.
But centralization seems to be a goal in and of itself these days (and
that isn't limited to Python).


> nor with the PSF regarding the current Terms of Service
> around uploads to PyPI, but that doesn't make it OK to start off-topic
> threads on python-dev just because you're a CPython core developer in
> addition to being a PyPI user.

Alexander thought otherwise:

  https://mail.python.org/pipermail/python-dev/2015-October/141840.html

"Given that ensurepip is part of stdlib, I am not sure this is an accurate
statement.  Even if it was, did you make any effort to discuss the proposal
outside of a small group subscribed to distutils ML?"

I completely agree with that.


Fredrik Lundh is also affected (and might not have received any mail,
same as me):

  https://pypi.python.org/pypi/PIL


> It *definitely* doesn't make it OK to accuse people of conspiring
> against you when they answer your question in good faith, just because
> their answer is the official distutils-sig/PyPA one (which was
> approved through the PEP process in PEP 470).

I'm not sure what you are referring to. Donald posted a link to PEP 470,
in my response to Nathaniel I acknowledged this. In my exchange with
?ukasz we both came to the conclusion (I think) that further discussion
would be futile.  IMO all responses from Brett, Donald, Nathaniel and
?ukasz were reasonable and I haven't accused them of conspiring in the
slightest.  I see that the PEP was accepted by Paul Moore.  I couldn't
even dream of accusing Paul Moore of any kind of conspiracy.  He's one
of the most reasonable (and *genuinely* polite) people on these mailing
lists.


Or are you referring to a super-condescending flame bait where someone
cloned a private website, assumed general ignorance and then proceeded
to offer a hostile fork to anyone who would be interested?

Well, I accepted the flame bait.


> - writing to psf-legal to let them know whether or not Van Lindberg's
> draft updates to the Terms of Service would be sufficient to make you
> comfortable with uploading cdecimal to PyPI in addition to bundling it
> with the standard library under your existing Contributor Licensing
> Agreement: https://bitbucket.org/vanl/pypi/src/default/templates

Okay, so there was recent progress here. This is actual news to me.

Do you remember what kind of derision I went through for just suggesting
something like that a couple of years ago?  Yes, you supported me, but
what about the others?  Or that Van Lindberg was in the merry group of
Twitter heroes who were gloating about the fact that (in their opinion)
I could not do anything about the first hostile fork?  Technically, he
didn't gloat, but suddenly legal advice was apparently readily available.

  https://pypi.python.org/pypi/m3-cdecimal


I can assure you that CoCs or "diversity statements" won't help you at all.


Stefan Krah





From breamoreboy at yahoo.co.uk  Thu May  5 18:58:18 2016
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Thu, 5 May 2016 23:58:18 +0100
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <loom.20160506T001122-774@post.gmane.org>
References: <loom.20160503T205904-379@post.gmane.org>
 <CADiSq7ciVgXSStb+353HCF2_kr@mail.gmail.com>
 <loom.20160506T001122-774@post.gmane.org>
Message-ID: <nggj6t$p5d$1@ger.gmane.org>

On 05/05/2016 23:22, Stefan Krah wrote
>
> Fredrik Lundh is also affected (and might not have received any mail,
> same as me):
>
>   https://pypi.python.org/pypi/PIL
>

He might be, but clearly the Python community as a whole is not 
impacted.  From what I see the latest version of PIL that is available 
is 1.1.6, which requires Python 1.5.2 or higher, and has the following 
stats:-

0 downloads in the last day
0 downloads in the last week
0 downloads in the last month

I wish I could vent my feelings regarding your comments earlier in this 
thread but I won't, as apparently core developers can say what they like 
with no comeback, whereas plebs like me can and will get hammered by the 
Python community, despite having explained that a combination of 
anxiety, depression, chronic fatigue syndrome, insomnia, autism and 
diplopia makes life rather difficult.  Unless of course you have similar 
problems, in which case please say so.  The rest of the community might 
not understand, I certainly will.

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


From chris.barker at noaa.gov  Thu May  5 19:03:58 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 5 May 2016 16:03:58 -0700
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <nggj6t$p5d$1@ger.gmane.org>
References: <loom.20160503T205904-379@post.gmane.org>
 <CADiSq7ciVgXSStb+353HCF2_kr@mail.gmail.com>
 <loom.20160506T001122-774@post.gmane.org> <nggj6t$p5d$1@ger.gmane.org>
Message-ID: <CALGmxEJTdDJFEirCob4DsivevH2OKy8DNwXE1+ZRuaQAZtc3WA@mail.gmail.com>

>
> Fredrik Lundh is also affected (and might not have received any mail,
>> same as me):
>>
>>   https://pypi.python.org/pypi/PIL
>>
>>
> He might be, but clearly the Python community as a whole is not impacted.
> From what I see the latest version of PIL that is available is 1.1.6, which
> requires Python 1.5.2 or higher,


Indeed -- Fredrik never made any effort to support pypi, pip, etc. --
that's why the Pillow fork was started in the first place.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160505/a011fcc4/attachment.html>

From barry at python.org  Thu May  5 19:06:57 2016
From: barry at python.org (Barry Warsaw)
Date: Thu, 5 May 2016 18:06:57 -0500
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <nggj6t$p5d$1@ger.gmane.org>
References: <loom.20160503T205904-379@post.gmane.org>
 <CADiSq7ciVgXSStb+353HCF2_kr@mail.gmail.com>
 <loom.20160506T001122-774@post.gmane.org>
 <nggj6t$p5d$1@ger.gmane.org>
Message-ID: <20160505180657.7be52a5e.barry@wooz.org>

On May 05, 2016, at 11:58 PM, Mark Lawrence via Python-Dev wrote:

>On 05/05/2016 23:22, Stefan Krah wrote
>>
>> Fredrik Lundh is also affected (and might not have received any mail,
>> same as me):
>>
>>   https://pypi.python.org/pypi/PIL

Maybe, but then there's the friendly fork:

https://pypi.python.org/pypi/Pillow/3.2.0

-Barry

From donald at stufft.io  Thu May  5 19:07:52 2016
From: donald at stufft.io (Donald Stufft)
Date: Thu, 5 May 2016 19:07:52 -0400
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <loom.20160506T001122-774@post.gmane.org>
References: <loom.20160503T205904-379@post.gmane.org>
 <CADiSq7ciVgXSStb+353HCF2_kr@mail.gmail.com>
 <loom.20160506T001122-774@post.gmane.org>
Message-ID: <A0AE3631-D44B-413F-A4AA-D0D2984BF84A@stufft.io>


> On May 5, 2016, at 6:22 PM, Stefan Krah <stefan at bytereef.org> wrote:
> 
> Nick Coghlan <ncoghlan <at> gmail.com> writes:
>> I know you're not happy with myself and the other distutils-sig folks
>> regarding the decision to deprecate and remove automatic link
>> spidering,
> 
> More accurately: Not happy with the removal of the checksummed "explicit"
> mode. What I would have preferred is a FreeBSD-like ports system. FreeBSD
> has been used in huge and secure installations, so the I don't buy the
> reliability and security arguments that are used in favor of centralization.
> But centralization seems to be a goal in and of itself these days (and
> that isn't limited to Python).


Let me say outright that there is nothing wrong with a well designed, ports
like system. However, PyPI was not that and the vast bulk of packages were not
using it in that fashion. At the time of PEP 470 being written a mere 59 out of
65,232 (at the time) projects were utilizing the feature that allowed people to
safely do a ports like system. That's roughly 0.09% of all of the projects on
PyPI which meant those projects were in a very serious minority. Thus it is
only natural that the expectations end users had when installing software was
that it was coming from PyPI, not from some external host and they made
decisions based around those expectations and when those expectations were
violated they came to the people operating PyPI and developing pip and were
at best very confused, and at worst irate at *us*.

It's also true that there are benefits and cons to both a repository centric
approach and a ports style approach, however since we were attempting to do
both in one repository it typically meant that we got all of the cons of both
systems, but we couldn't ever take advantage of any of the unique benefits of
either system since it would only apply to some fraction of the projects. This
meant that improvements that would help people installing 99.11% of projects
would end up getting blocked by 0.09% of projects.

At the end of the day, attempting to both things in one repository just wasn't
working out very well in the general case, so we needed (IMO) to pick one way
and stop supporting the other way. I personally think it's a pretty obvious
choice to go the way that we did, given that the overwhelming majority of
projects were already being used that way.

That being said, I don't think the characterization of "centralizing" is really
accurate here. People are still free to host their projects wherever they want
and pip (and easy_install) can be used to install them. All it requires is
telling pip where it needs to look in addition to the default location (PyPI),
which can be configured via the CLI, environment variables, or a number of
configuration files at the system, user, and virtual environment level. You're
free to continue providing a page on PyPI to enable discovery of your project
and you can include in that page instructions on how to install your project.


> 
> 
>> nor with the PSF regarding the current Terms of Service
>> around uploads to PyPI, but that doesn't make it OK to start off-topic
>> threads on python-dev just because you're a CPython core developer in
>> addition to being a PyPI user.
> 
> Alexander thought otherwise:
> 
>  https://mail.python.org/pipermail/python-dev/2015-October/141840.html
> 
> "Given that ensurepip is part of stdlib, I am not sure this is an accurate
> statement.  Even if it was, did you make any effort to discuss the proposal
> outside of a small group subscribed to distutils ML?"
> 
> I completely agree with that.

We've sort of hijacked the PEP process to do something that it wasn't really
meant to do. Changes like PEP 470 have nothing to do with what Python itself
does and thus it's really outside of the realm of where python-dev really needs
a say or notification. In Alexander's example of PEP 495 it was talking about
adding a new API to Python itself, thus it makes sense that python-dev should
be told about that PEP, and in cases where we needed to change the standard
library we've done that discussion on python-dev (such as with PEP 453).

The python-dev list is not the list to notify people of changes to things in
all corners of the Python ecosystem. If people want to get involved or keep
abreast of the changes in the external projects of PyPI, pip, setuptools, etc
then they should subscribe to the places that those changes get discussed--
which is distutils-sig. Packaging already has enough problems with getting
bogged down in "why wasn't I consulted" and bikeshedding, I don't think that
inviting folks (and nothing against python-dev here) a chance to try and
re-legislate an agreed upon change after the fact is going to provide much in
the way of benefits and will instead just contribute further to stymying
efforts to move packaging forward. If it was decided that we really need all
PEPs announced on python-dev and python-dev given a chance to weigh in, I would
advocate for packaging to stop using the PEP process all together and create
our own process to mirror the PEP process because again, we're not using the
PEP process because these things naturally fall under what the PEP process was
designed to cover, we're using it because it was there already and close enough
that we could hijack it and it was moderately easier to just do that. If the
downsides of using the PEP process become greater than the effort to come up
with our own process then I think it's a no brainer to just do our own thing.

> 
> 
> Fredrik Lundh is also affected (and might not have received any mail,
> same as me):
> 
>  https://pypi.python.org/pypi/PIL

I've personally emailed the PIL offer several times through various email
addresses over several years and have never gotten a response. At this point I
can only assume either they are willfully ignoring those messages or they've
completely changed their contact information in a way that I have not been able
to find updated contacted information for at all.

> Technically, he
> didn't gloat, but suddenly legal advice was apparently readily available.
> 
>  https://pypi.python.org/pypi/m3-cdecimal
> 


I'm not going to respond to the rest of this, because I don't think it's
possible for you and me to have a constructive discussion on these other
issues. However, I do want to point out that the PSF is the legal entity behind
PyPI itself and Van is both a board member and a lawyer. I don't recall what
exactly was discussed with who about m3-cdecimal, but it's completely
reasonable that when a legal question comes up with PyPI that we consult the
PSF board members, particularly the ones who are lawyers. I do recall that the
last time m3-cdecimal came up [1] you (again) brought up issues you had with
PyPI in an inappropriate venue and as far as I know, you never actually used
any of the contact channels that you were offered to try and resolve the
issue that you had with a project on PyPI.


[1] https://bugs.python.org/issue22483

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160505/881c096a/attachment.sig>

From stefan at bytereef.org  Thu May  5 19:13:24 2016
From: stefan at bytereef.org (Stefan Krah)
Date: Thu, 5 May 2016 23:13:24 +0000 (UTC)
Subject: [Python-Dev] Yearly PyPI breakage
References: <loom.20160503T205904-379@post.gmane.org>
 <CADiSq7ciVgXSStb+353HCF2_kr@mail.gmail.com>
 <loom.20160506T001122-774@post.gmane.org> <nggj6t$p5d$1@ger.gmane.org>
 <CALGmxEJTdDJFEirCob4DsivevH2OKy8DNwXE1+ZRuaQAZtc3WA@mail.gmail.com>
Message-ID: <loom.20160506T011049-193@post.gmane.org>

Chris Barker <chris.barker <at> noaa.gov> writes:
> Indeed -- Fredrik never made any effort to support pypi, pip, etc. --
that's why the Pillow fork was started in the first place.

Maybe, but he created PIL, so thank you, Fredrik!


Stefan Krah


From breamoreboy at yahoo.co.uk  Thu May  5 19:24:57 2016
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Fri, 6 May 2016 00:24:57 +0100
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <20160505180657.7be52a5e.barry@wooz.org>
References: <loom.20160503T205904-379@post.gmane.org>
 <CADiSq7ciVgXSStb+353HCF2_kr@mail.gmail.com>
 <loom.20160506T001122-774@post.gmane.org> <nggj6t$p5d$1@ger.gmane.org>
 <20160505180657.7be52a5e.barry@wooz.org>
Message-ID: <nggkos$c2t$1@ger.gmane.org>

On 06/05/2016 00:06, Barry Warsaw wrote:
> On May 05, 2016, at 11:58 PM, Mark Lawrence via Python-Dev wrote:
>
>> On 05/05/2016 23:22, Stefan Krah wrote
>>>
>>> Fredrik Lundh is also affected (and might not have received any mail,
>>> same as me):
>>>
>>>   https://pypi.python.org/pypi/PIL
>
> Maybe, but then there's the friendly fork:
>
> https://pypi.python.org/pypi/Pillow/3.2.0
>
> -Barry
>

Where is the relevance to my words that you've snipped?

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


From dw+python-dev at hmmz.org  Thu May  5 19:42:30 2016
From: dw+python-dev at hmmz.org (David Wilson)
Date: Thu, 5 May 2016 23:42:30 +0000
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <loom.20160503T205904-379@post.gmane.org>
References: <loom.20160503T205904-379@post.gmane.org>
Message-ID: <20160505234230.GB14936@k3>

This is mostly just observational, and not meant primarily as criticism
of the fabulous work of Donald and others (ignoring pypa, also the
prompt, reliable, and skilled support responses common on such places as
IRC), however I can't help but notice that PyPI governance seems to come
under fire vastly more often than similar and much more popular
packaging systems, and some choices that have been made particularly in
recent years have caused a noticeable amount of dissent with what might
be considered the typical developer.

When a contributor to the core language is having repeat issues
maintaining some basic element of the function of the packaging system,
might it be fair to reflect on how changes to those functions are being
managed?

There are PEPs covering a great deal of the work done to PyPI recently,
but, and I say this as someone who has bumped into friction with the
packaging tooling in the relatively recent past, even I despite my
motivations to the contrary, I have not read most of them. It seems the
current process is observed by few, does not sufficiently address the
range of traditional use cases that were possible in the past, and the
first the common user learns of a change is when pip (after insisting it
must be upgraded) fails to function as it previously did.

The usual course then is some complaint, that leads to distutils-sig,
which ultimately leads to pointing at some design work that was only
observed by perhaps 50 people max that turns out had some edge cases
that hurt in a common use case.

Is there something to contemplate in here? I dislike posting questions
instead of answers, but it seems apparent there is a problem here and it
continues to remain unaddressed.


David


On Tue, May 03, 2016 at 07:06:12PM +0000, Stefan Krah wrote:
> 
> Hello,
> 
> Could someone enlighten me which hoops I have to jump through
> this year in order to keep pip downloads working?
> 
> Collecting cdecimal
>   Could not find a version that satisfies the requirement cdecimal (from
> versions: )
> No matching distribution found for cdecimal
> You are using pip version 7.1.2, however version 8.1.1 is available.
> You should consider upgrading via the 'pip install --upgrade pip' command.
> 
> 
> If this continues, I'm going to release a premium version that's
> 50% faster and only available from bytereef.org or Anaconda.
> 
> 
> 
> Stefan Krah
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/dw%2Bpython-dev%40hmmz.org

From brett at python.org  Thu May  5 20:03:48 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 06 May 2016 00:03:48 +0000
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <20160505234230.GB14936@k3>
References: <loom.20160503T205904-379@post.gmane.org>
 <20160505234230.GB14936@k3>
Message-ID: <CAP1=2W4KWbnPJiUwWASDYyiX_QYFBb611G8JJ4Esi_PY-kuzOQ@mail.gmail.com>

On Thu, 5 May 2016 at 16:54 David Wilson <dw+python-dev at hmmz.org> wrote:

> This is mostly just observational, and not meant primarily as criticism
> of the fabulous work of Donald and others (ignoring pypa, also the
> prompt, reliable, and skilled support responses common on such places as
> IRC), however I can't help but notice that PyPI governance seems to come
> under fire vastly more often than similar and much more popular
> packaging systems, and some choices that have been made particularly in
> recent years have caused a noticeable amount of dissent with what might
> be considered the typical developer.
>
> When a contributor to the core language is having repeat issues
> maintaining some basic element of the function of the packaging system,
> might it be fair to reflect on how changes to those functions are being
> managed?
>
> There are PEPs covering a great deal of the work done to PyPI recently,
> but, and I say this as someone who has bumped into friction with the
> packaging tooling in the relatively recent past, even I despite my
> motivations to the contrary, I have not read most of them. It seems the
> current process is observed by few, does not sufficiently address the
> range of traditional use cases that were possible in the past, and the
> first the common user learns of a change is when pip (after insisting it
> must be upgraded) fails to function as it previously did.
>
> The usual course then is some complaint, that leads to distutils-sig,
> which ultimately leads to pointing at some design work that was only
> observed by perhaps 50 people max that turns out had some edge cases
> that hurt in a common use case.
>
> Is there something to contemplate in here? I dislike posting questions
> instead of answers, but it seems apparent there is a problem here and it
> continues to remain unaddressed.
>

This is whole thread is off-topic precisely because all of this is
discussed -- in the open -- on distutils-sig and decided there. If people
feel changes need to be made like broadcasting to a wider audience when a
change occurs, then please bring it up on distutils-sig. But if people
choose not to participate then they are implicitly delegating decision
powers to those who do participate (which is totally fine and I'm not
implying that  if people don't participate they are doing something wrong).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160506/fafa560e/attachment.html>

From brett at python.org  Thu May  5 20:13:11 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 06 May 2016 00:13:11 +0000
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <nggj6t$p5d$1@ger.gmane.org>
References: <loom.20160503T205904-379@post.gmane.org>
 <CADiSq7ciVgXSStb+353HCF2_kr@mail.gmail.com>
 <loom.20160506T001122-774@post.gmane.org> <nggj6t$p5d$1@ger.gmane.org>
Message-ID: <CAP1=2W4qjH_ocPrSh-O_GHZ56VTcV4LNybDfKtDRzkPNunQF0A@mail.gmail.com>

On Thu, 5 May 2016 at 16:00 Mark Lawrence via Python-Dev <
python-dev at python.org> wrote:

> On 05/05/2016 23:22, Stefan Krah wrote
> >
> > Fredrik Lundh is also affected (and might not have received any mail,
> > same as me):
> >
> >   https://pypi.python.org/pypi/PIL
> >
>
> He might be, but clearly the Python community as a whole is not
> impacted.  From what I see the latest version of PIL that is available
> is 1.1.6, which requires Python 1.5.2 or higher, and has the following
> stats:-
>
> 0 downloads in the last day
> 0 downloads in the last week
> 0 downloads in the last month
>
> I wish I could vent my feelings regarding your comments earlier in this
> thread but I won't, as apparently core developers can say what they like
> with no comeback,


I don't think that's fair. Several people pointed out that Stefan's initial
email was off-topic and somewhat rude, but was probably given some slack
due to the fact that having deployments to the Cheeseshop fail can be
frustrating (leeway I think anyone posting here would have received).

At this point I think there's nothing new to be said and unless someone
wants to take a more drastic step like a formal CoC complaint or calling
for the heads of the management of distutils-sig on spikes, this thread has
run its course.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160506/8e1709c4/attachment.html>

From stefan at bytereef.org  Thu May  5 20:24:57 2016
From: stefan at bytereef.org (Stefan Krah)
Date: Fri, 6 May 2016 00:24:57 +0000 (UTC)
Subject: [Python-Dev] Yearly PyPI breakage
References: <loom.20160503T205904-379@post.gmane.org>
 <CADiSq7ciVgXSStb+353HCF2_kr@mail.gmail.com>
 <loom.20160506T001122-774@post.gmane.org>
 <A0AE3631-D44B-413F-A4AA-D0D2984BF84A@stufft.io>
Message-ID: <loom.20160506T011632-733@post.gmane.org>

Donald Stufft <donald <at> stufft.io> writes:
> > Technically, he
> > didn't gloat, but suddenly legal advice was apparently readily available.
> > 
> >  https://pypi.python.org/pypi/m3-cdecimal
> > 
> 
> I'm not going to respond to the rest of this, because I don't think it's
> possible for you and me to have a constructive discussion on these other
> issues. However, I do want to point out that the PSF is the legal entity
behind
> PyPI itself and Van is both a board member and a lawyer. I don't recall what
> exactly was discussed with who about m3-cdecimal, but it's completely
> reasonable that when a legal question comes up with PyPI that we consult the
> PSF board members, particularly the ones who are lawyers.

It is not reasonable at all.  A normal human being would try to consider
first if an author has moral rights as opposed to legal rights and
maybe pick up the phone or use private email instead of escalating
everything to Twitter.

Also, what was discussed was not whether the PSF had any right
to remove m3-cdecimal but rather whether *I* had any rights to
*demand* the removal.

People concluded gleefully that I hadn't any (I still think they're
mistaken and an enterprising Oracle lawyer could come to a different
conclusion, but that's besides the point). 



Stefan Krah








From dw+python-dev at hmmz.org  Thu May  5 20:35:27 2016
From: dw+python-dev at hmmz.org (David Wilson)
Date: Fri, 6 May 2016 00:35:27 +0000
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <CAP1=2W4KWbnPJiUwWASDYyiX_QYFBb611G8JJ4Esi_PY-kuzOQ@mail.gmail.com>
References: <loom.20160503T205904-379@post.gmane.org>
 <20160505234230.GB14936@k3>
 <CAP1=2W4KWbnPJiUwWASDYyiX_QYFBb611G8JJ4Esi_PY-kuzOQ@mail.gmail.com>
Message-ID: <20160506003527.GA30649@k3>

On Fri, May 06, 2016 at 12:03:48AM +0000, Brett Cannon wrote:

>     Is there something to contemplate in here? I dislike posting questions
>     instead of answers, but it seems apparent there is a problem here and it
>     continues to remain unaddressed.

> This is whole thread is off-topic precisely because all of this is
> discussed -- in the open -- on distutils-sig and decided there. If
> people feel changes need to be made like broadcasting to a wider
> audience when a change occurs, then please bring it up on
> distutils-sig.

I respectfully disagree, as this has been the default response applied
for years, and it seems friction and dissemination have not been
improved by it. Packaging is not some adjunct technicality, anyone
learning Python in the past few years at least has been taught pip
within the first week.


> But if people choose not to participate then they are implicitly
> delegating decision powers to those who do participate

I believe this is also practically rhetorical in nature. I've watched
the wars on distutils-sig for many years now, and the general strategy
is that beyond minor outside influence, the process there is occupied by
few individuals who are resistant to outside change. Outside influence
is regularly met with essay-length reponses and tangential minutia until
the energy of the challenge is expended.

As an example, one common argument is that "Donald is overworked",
however as an example, I offered a very long time ago to implement full
text indexing for PyPI search. At the time I belive I was told such
things weren't necessary, only to learn a few years later that Donald
himself implemented the same function, and it suffers from huge latency
and accuracy issues in the meantime. The solution to those problems is
of course the ever-delayed rewrite.

Over on distutils-sig, one will learn that a large amount of effort has
been poured into a rewrite of PyPI (an effort going on years now),
however the original codebase was not far from rescue (I had a local
copy almost entirely ported to Flask in a few days). There is no reason
why this effort nor any other (like full text search) should be used, as
it often is, as an argument in the decisionmaking process that largely
governs how PyPI and pip have worked in the recent years, yet it only
takes a few glances at the archives to demonstrate that it regularly is.


David

From stefan at bytereef.org  Thu May  5 21:28:20 2016
From: stefan at bytereef.org (Stefan Krah)
Date: Fri, 6 May 2016 01:28:20 +0000 (UTC)
Subject: [Python-Dev] Yearly PyPI breakage
References: <loom.20160503T205904-379@post.gmane.org>
 <20160505234230.GB14936@k3>
 <CAP1=2W4KWbnPJiUwWASDYyiX_QYFBb611G8JJ4Esi_PY-kuzOQ@mail.gmail.com>
Message-ID: <loom.20160506T031555-668@post.gmane.org>

Brett Cannon <brett <at> python.org> writes:
> This is whole thread is off-topic precisely because all of this is
discussed -- in the open -- on distutils-sig and decided there. If people
feel changes need to be made like broadcasting to a wider audience when a
change occurs, then please bring it up on distutils-sig. But if people
choose not to participate then they are implicitly delegating decision
powers to those who do participate (which is totally fine and I'm not
implying that ?if people don't participate they are doing something wrong).

Participating there, especially for a non-native speaker, is basically
a full-time job. You're met with a large number of extremely lengthy
posts that would require an incredible amount of time to respond to
in a careful manner.

And ultimately it is the persons that Guido delegated the authority to
who decide everything. I think many people have concluded that the
influence/time ratio is too low to be worth it.


Also I don't know any other development group who is a) that quick
in trying to suppress any "off-topic" discussions and b) constantly
uses venues outside of the Python mailing lists to steer and
manipulate public opinion.


Stefan Krah

From donald at stufft.io  Thu May  5 22:31:48 2016
From: donald at stufft.io (Donald Stufft)
Date: Thu, 5 May 2016 22:31:48 -0400
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <20160506003527.GA30649@k3>
References: <loom.20160503T205904-379@post.gmane.org>
 <20160505234230.GB14936@k3>
 <CAP1=2W4KWbnPJiUwWASDYyiX_QYFBb611G8JJ4Esi_PY-kuzOQ@mail.gmail.com>
 <20160506003527.GA30649@k3>
Message-ID: <9F53FF72-6CCB-482A-B4EB-E8165E404362@stufft.io>


> On May 5, 2016, at 8:35 PM, David Wilson <dw+python-dev at hmmz.org> wrote:
> 
> On Fri, May 06, 2016 at 12:03:48AM +0000, Brett Cannon wrote:
> 
>>    Is there something to contemplate in here? I dislike posting questions
>>    instead of answers, but it seems apparent there is a problem here and it
>>    continues to remain unaddressed.
> 
>> This is whole thread is off-topic precisely because all of this is
>> discussed -- in the open -- on distutils-sig and decided there. If
>> people feel changes need to be made like broadcasting to a wider
>> audience when a change occurs, then please bring it up on
>> distutils-sig.
> 
> I respectfully disagree, as this has been the default response applied
> for years, and it seems friction and dissemination have not been
> improved by it. Packaging is not some adjunct technicality, anyone
> learning Python in the past few years at least has been taught pip
> within the first week.
> 
> 
>> But if people choose not to participate then they are implicitly
>> delegating decision powers to those who do participate
> 
> I believe this is also practically rhetorical in nature. I've watched
> the wars on distutils-sig for many years now, and the general strategy
> is that beyond minor outside influence, the process there is occupied by
> few individuals who are resistant to outside change. Outside influence
> is regularly met with essay-length reponses and tangential minutia until
> the energy of the challenge is expended.

Honestly, I just don't think this is an honest characterization. It is true
that in general there are few people who bother to put in the effort to take a
proposal from start to finish including actually writing the code to make such
a thing happen.

The problem of packaging is a particularly hard one where it's difficult to
make trade offs because unlike other systems, people are sort of locked into
using whatever the popular system is. If asyncio doesn't suite everyone that's
fine it's not hard for them to go and switch and use Twisted, Tornado, gevent,
eventlet, curio, etc etc but it's not very realistic for someone to opt out of
the packaging ecosystem.

In addition, it's much like something like HTTP or SMTP or the such where once
you add some feature, it becomes incredibly difficult to ever remove it if you
end up needing to (case in point, this thread and off site hosting), so we tend
to over scrutinize changes wherever we can to try and make sure we *really*
understand what the tradeoffs we're making are. Just as an example, it took a
year and a half for PEP 440 to be standardized which is for something as bite
sized as version numbers and PEP 440 was a continuation of the stalled PEP 386.
The most time consuming part of PEP 440 was trying our new rules against every
single version that existed on PyPI and minimizing the breakage.

Even then, we've had several recent PEPs go through (manylinux1, environment
markers, requirements) from people willing to do so.

> 
> As an example, one common argument is that "Donald is overworked",
> however as an example, I offered a very long time ago to implement full
> text indexing for PyPI search. At the time I belive I was told such
> things weren't necessary, only to learn a few years later that Donald
> himself implemented the same function, and it suffers from huge latency
> and accuracy issues in the meantime. The solution to those problems is
> of course the ever-delayed rewrite.

I don't remember the specific details around your proposal, but I'm pretty sure
that *I* never told you that "such things were not necessary" since I've never
held that opinion to my memory. What I have found [1][2] (to try and refresh my
memory) are threads from you that went largely unanswered in April of 2013,
which was right at the time I was heavily focused on getting PyPI setup behind
Fastly and wasn't really paying attention to much else. I don't see anything
else from you about it until September of 2015 when you offered your help again
and at that point I told you that we had already switched to using
Elasticsearch. In those two years since your initial offer, and then your
follow up in 2015 I do not see any pull request to the PyPI code base from you
(which, assuming they were reasonable would have been merged), however I do
see the PR [3] from Ernest (No, I didn't implement the search) which got merged
and deployed.

My experience is that people are often willing to *offer* help with PyPI, but
then they quickly disappear once they start to actually try and hack on it's
code base and realize how difficult it is to work with. That experience tends
to mean that I don't really get super excited when someone shows interest in
helping because it rarely actually manifests. I think we've had more people
contribute to Warehouse in the last year than we've *ever* had contribute to
the legacy PyPI code base, which I think says a lot about the decision to
switch.

> 
> Over on distutils-sig, one will learn that a large amount of effort has
> been poured into a rewrite of PyPI (an effort going on years now),
> however the original codebase was not far from rescue (I had a local
> copy almost entirely ported to Flask in a few days). There is no reason
> why this effort nor any other (like full text search) should be used, as
> it often is, as an argument in the decisionmaking process that largely
> governs how PyPI and pip have worked in the recent years, yet it only
> takes a few glances at the archives to demonstrate that it regularly is.
> 


I find the statement that "the original code base is not far from rescue" a bit
interesting, since you had previously stated [2] that:

    Again PyPI has been growing organically for a very long time, and
    dumping even more features in there doesn't seem a great idea. I
    looked at retrofitting PyPI with Flask, but there is simply too much
    custom code to be sure things won't be broken by doing it in a hurry.

In any case, yes we're rewriting PyPI and it's been something that I've put a
lot of effort into. That also means that I'm not particularly enthused about
spending a bunch of time and effort working on legacy PyPI because doing so
is incredibly demotivating to me to the point that the more time I spent in
that code base the more I want to quit all together. Even given that I still
have and would review PRs to that code base, it's just that very few people
ever suffer through that code base long enough to actually submit one.

I don't believe we've ever told someone that something can't happen because of
Warehouse, only that *I* won't implement something until after Warehouse. That
often times means that something won't happen until after Warehouse because of
the severe shortage of people with enough time and motivation to work on this
stuff but if someone did step up more things would get done.

[1] https://mail.python.org/pipermail/distutils-sig/2013-April/020553.html
[2] https://groups.google.com/d/msg/pypa-dev/ZjUNkczsKos/SzwNOckisXUJ
[3] https://bitbucket.org/pypa/pypi/pull-requests/81/implement-an-elasticsearch-index-for-pypi/diff

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160505/73f40a84/attachment.sig>

From josephnunn at gmail.com  Fri May  6 05:42:06 2016
From: josephnunn at gmail.com (Joseph Nunn)
Date: Fri, 6 May 2016 09:42:06 +0000 (UTC)
Subject: [Python-Dev] Wrong OSX platform in sysconfig.py causing
 installation problems
Message-ID: <loom.20160506T111745-702@post.gmane.org>

Hello Everyone, my first post.  I would also like to subscribe but 
haven't seen how yet.  

So I've an issue with how the 3.5.1 OSX 
package installs on OSX.  The problem was first made apparent 
when trying to create a virtual environment and `pip install ZODB`, 
which failed as clang and the linker could not find either the 
include or lib directories for a dependency.  By adding a -v option
to the compiler options I was able to see in the  compiler flags 
the following:

-isysroot /Developer/SDKs/MacOSX10.6.sdk

Which was not present in the compiler flags of other people 
who I was talking with.  The isysroot flag changes the location 
of includes and libs from the system default, which would be 
why they can't be found.  After grepping around a bit I found 
this file:

/Library/Frameworks/Python.framework/Versions/3.5
/lib/python3.5/_sysconfigdata.py

Within which are many mentions of this flag, as so:

'CONFIGURE_CFLAGS': '-arch i386 -arch x86_64 -isysroot 
/Developer/SDKs/MacOSX10.6.sdk -g',
'CONFIGURE_CFLAGS_NODIST': '-Werror=declaration-after
-statement',
'CONFIGURE_CPPFLAGS': 
'-isysroot /Developer/SDKs/MacOSX10.6.sdk',
'CONFIGURE_LDFLAGS': '-arch i386 -arch x86_64 -isysroot 
/Developer/SDKs/MacOSX10.6.sdk -g',
'CONFIG_ARGS': "'-C' '--enable-framework' '--enable-
universalsdk=/Developer/SDKs/MacOSX10.6.sdk' 
'--with-universal-archs=intel' 
'--with-computed-gotos' '--without-ensurepip' 'LDFLAGS=-g'
'CFLAGS=-g' 

'CC=gcc-4.2'",

among others.

By running Python 3.5's sysconfig.py as a module, via 
`python -m sysconfig`, I see the following at the beginning 
of the output:

Platform: "macosx-10.6-intel"
Python version: "3.5"
Current installation scheme: "posix_prefix"

And when I simply go into python interactive mode, import 
sysconfig, and type 

sysconfig.get_platform() it returns macosx-10.6-intel.

I believe somehow the platform being reported is incorrect, 
but looking at my environment, I do not have any kind of 
compiler flags or anything else like that set 

in my bash init files.

Can someone who has a working OSX 3.5 installation check 
their reported platform to see what it is and if anyone has an 
idea as to how this information is getting into the installation 
and where I might change it please let me know. 

Sincerely,

Joseph
josephnunn at gmail.com


From nad at python.org  Fri May  6 07:20:12 2016
From: nad at python.org (Ned Deily)
Date: Fri, 6 May 2016 07:20:12 -0400
Subject: [Python-Dev] Wrong OSX platform in sysconfig.py causing
 installation problems
In-Reply-To: <loom.20160506T111745-702@post.gmane.org>
References: <loom.20160506T111745-702@post.gmane.org>
Message-ID: <905F2C35-855E-4733-9023-4B30EFF228B2@python.org>


On May 6, 2016, at 05:42, Joseph Nunn <josephnunn at gmail.com> wrote:
> Hello Everyone, my first post.  I would also like to subscribe but 
> haven't seen how yet.  
> 
> So I've an issue with how the 3.5.1 OSX 
> package installs on OSX.  The problem was first made apparent 
> when trying to create a virtual environment and `pip install ZODB`, 
> which failed as clang and the linker could not find either the 
> include or lib directories for a dependency.  By adding a -v option
> to the compiler options I was able to see in the  compiler flags 
> the following:
> 
> -isysroot /Developer/SDKs/MacOSX10.6.sdk
> 
> Which was not present in the compiler flags of other people 
> who I was talking with. [...]

Hi Joseph!

Welcome!  Note this list is for the development of Python itself, not for usage or installation issues.  The Python Developer's Guide (https://docs.python.org/devguide/) including its FAQ gives information about mailing lists and other support forums to get help with general Python issues.  That said, without more information, it would be difficult to know exactly what problem you are running into but, since you mention missing headers, my guess is that you need to install Apple's Command Line Tools as described here:

https://docs.python.org/devguide/setup.html#build-dependencies

Installing Xcode by itself is not sufficient.  The sysroot and flag values you note in _sysconfigdata.py are normal and there for compatibility with installing on older versions of OS X; they should not cause a problem on newer releases.  For what it's worth, I was able to install ZODB with the current python.org 3.5.1 on OS X 10.11.4 using either venv or virtualenv.

Good luck!

--Ned

P.S. Mailing list info is available here: https://mail.python.org.  The bug tracker is here: https://bugs.python.org

--
  Ned Deily
  nad at python.org -- []


From larry at hastings.org  Fri May  6 11:34:08 2016
From: larry at hastings.org (Larry Hastings)
Date: Fri, 6 May 2016 08:34:08 -0700
Subject: [Python-Dev] 3.4.4 Release Schedule
In-Reply-To: <1462291641.3972729.596903881.4CD8087B@webmail.messagingengine.com>
References: <1462291641.3972729.596903881.4CD8087B@webmail.messagingengine.com>
Message-ID: <572CB970.3030601@hastings.org>

On 05/03/2016 09:07 AM, Benjamin Peterson wrote:
> 2.7.12rc1 will be on June 11th, and the final will be on June 25.


That's perfectly cromulent!  I'm going to piggyback on that for 3.4.  
Therefore, 3.4.4 rc1 will be on June 11th, and 3.4.4 final will be on 
June 25.

Reminder: 3.4.4 will be a source-only release.


June's coming up Python,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160506/9b4f8bac/attachment.html>

From larry at hastings.org  Fri May  6 11:40:52 2016
From: larry at hastings.org (Larry Hastings)
Date: Fri, 6 May 2016 08:40:52 -0700
Subject: [Python-Dev] 3.4.5 Release Schedule (not 3.4.4, whoops)
In-Reply-To: <572CB970.3030601@hastings.org>
References: <1462291641.3972729.596903881.4CD8087B@webmail.messagingengine.com>
 <572CB970.3030601@hastings.org>
Message-ID: <572CBB04.8090501@hastings.org>


On 05/06/2016 08:34 AM, Larry Hastings wrote:
> That's perfectly cromulent!  I'm going to piggyback on that for 3.4.  
> Therefore, 3.4.4 rc1 will be on June 11th, and 3.4.4 final will be on 
> June 25.

Whoops, a whole bunch of off-by-one bugs.

First, I'll be releasing 3.4.5 in June.  3.4.4 was released in December 
2015.  D'oh!

Second, Benjamin and I have slightly different approaches--I like to tag 
on Saturday and release on Sunday.  Therefore 3.4.5 rc1 will be released 
June 12th, and 3.4.5 final will be released June 26th. (3.4.5 rc1 will 
be tagged Saturday June 11, 3.4.5 final will be tagged June 25th.)

The release schedule PEP has all the correct numbers and dates and such.


That'll teach me to send these emails while half-asleep,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160506/affa1986/attachment.html>

From status at bugs.python.org  Fri May  6 12:08:31 2016
From: status at bugs.python.org (Python tracker)
Date: Fri,  6 May 2016 18:08:31 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20160506160831.0491F567BA@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2016-04-29 - 2016-05-06)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5504 (+29)
  closed 33223 (+56)
  total  38727 (+85)

Open issues with patches: 2388 


Issues opened (58)
==================

#24225: Idlelib: changing file names
http://bugs.python.org/issue24225  reopened by serhiy.storchaka

#26156: Bad name into power operator syntax
http://bugs.python.org/issue26156  reopened by r.david.murray

#26163: FAIL: test_hash_effectiveness (test.test_set.TestFrozenSet)
http://bugs.python.org/issue26163  reopened by berker.peksag

#26888: Multiple memory leaks after raw Py_Initialize and Py_Finalize.
http://bugs.python.org/issue26888  opened by Aleksander Gajewski

#26889: Improve Doc/library/xmlrpc.client.rst
http://bugs.python.org/issue26889  opened by serhiy.storchaka

#26890: inspect.getsource gets source copy on disk even when module ha
http://bugs.python.org/issue26890  opened by sebastien.bourdeauducq

#26891: CPython doesn't work when you disable refcounting
http://bugs.python.org/issue26891  opened by larry

#26892: debuglevel not honored in urllib
http://bugs.python.org/issue26892  opened by Chi Hsuan Yen

#26894: Readline not aborting line edition on sigint
http://bugs.python.org/issue26894  opened by memeplex

#26896: mix-up with the terms 'importer', 'finder', 'loader' in the im
http://bugs.python.org/issue26896  opened by Oren Milman

#26897: Clarify Popen stdin, stdout, stderr
http://bugs.python.org/issue26897  opened by Yclept.Nemo

#26899: struct.pack_into(), struct.unpack_from() don't document suppor
http://bugs.python.org/issue26899  opened by pfalcon

#26900: Exclude the private API from the stable API
http://bugs.python.org/issue26900  opened by serhiy.storchaka

#26901: Argument Clinic test is broken
http://bugs.python.org/issue26901  opened by serhiy.storchaka

#26903: ProcessPoolExecutor(max_workers=64) crashes on Windows
http://bugs.python.org/issue26903  opened by diogocp

#26904: Difflib quick_ratio() could use Counter()
http://bugs.python.org/issue26904  opened by Michael Cuthbert

#26906: format(object.__reduce__) fails intermittently
http://bugs.python.org/issue26906  opened by ztane

#26907: Add missing getsockopt constants
http://bugs.python.org/issue26907  opened by christian.heimes

#26909: Serious performance loss (10 times) when NOT using .drain()
http://bugs.python.org/issue26909  opened by mmarkk

#26911: lib2to3/tests/pytree_idempotency.py has a broken import
http://bugs.python.org/issue26911  opened by imz

#26912: test/test_email/torture_test.py (and test_asian_codecs.py) has
http://bugs.python.org/issue26912  opened by imz

#26917: unicodedata.normalize(): bug in Hangul Composition
http://bugs.python.org/issue26917  opened by arigo

#26919: android: test_cmd_line fails
http://bugs.python.org/issue26919  opened by xdegaye

#26920: android: test_sys fails
http://bugs.python.org/issue26920  opened by xdegaye

#26923: asyncio.gather drops cancellation
http://bugs.python.org/issue26923  opened by JohannesEbke

#26924: android: test_concurrent_futures fails
http://bugs.python.org/issue26924  opened by xdegaye

#26925: android: test_multiprocessing_main_handling fails
http://bugs.python.org/issue26925  opened by xdegaye

#26926: Large files are not supported on Android
http://bugs.python.org/issue26926  opened by xdegaye

#26928: android: test_site fails
http://bugs.python.org/issue26928  opened by xdegaye

#26929: android: test_strptime fails
http://bugs.python.org/issue26929  opened by xdegaye

#26930: Upgrade installers to OpenSSL 1.0.2h
http://bugs.python.org/issue26930  opened by alex

#26931: android: test_distutils fails
http://bugs.python.org/issue26931  opened by xdegaye

#26934: android: test_faulthandler fails
http://bugs.python.org/issue26934  opened by xdegaye

#26935: android: test_os fails
http://bugs.python.org/issue26935  opened by xdegaye

#26936: android: test_socket fails
http://bugs.python.org/issue26936  opened by xdegaye

#26937: android: test_tarfile fails
http://bugs.python.org/issue26937  opened by xdegaye

#26938: android: test_concurrent_futures hangs on armv7
http://bugs.python.org/issue26938  opened by xdegaye

#26939: android: test_functools hangs on armv7
http://bugs.python.org/issue26939  opened by xdegaye

#26940: android: test_importlib hangs on armv7
http://bugs.python.org/issue26940  opened by xdegaye

#26941: android: test_threading hangs on armv7
http://bugs.python.org/issue26941  opened by xdegaye

#26942: android: test_ctypes crashes on armv7
http://bugs.python.org/issue26942  opened by xdegaye

#26944: android: test_posix fails
http://bugs.python.org/issue26944  opened by xdegaye

#26945: difflib.HtmlDiff().make_file() treat few change as whole line 
http://bugs.python.org/issue26945  opened by Fairuz Zack

#26947: Documentation improvement needed
http://bugs.python.org/issue26947  opened by mPython

#26948: Simplify PyImport_ImportModuleLevelObject: avoid temporary tup
http://bugs.python.org/issue26948  opened by haypo

#26949: IDLE restarts when one debugs code raising SystemExit
http://bugs.python.org/issue26949  opened by ppperry

#26951: Unintuitive error when using generator expression in class pro
http://bugs.python.org/issue26951  opened by corey

#26952: argparse help formatter crashes
http://bugs.python.org/issue26952  opened by Endre

#26954: Add Guido's rejection notice to the "with" FAQ
http://bugs.python.org/issue26954  opened by steven.daprano

#26955: Implement equivalent to `pip.locations.distutils_scheme` in di
http://bugs.python.org/issue26955  opened by sylvain.corlay

#26959: pickle: respect dispatch for functions again
http://bugs.python.org/issue26959  opened by torkve

#26960: urllib hangs when closing connection
http://bugs.python.org/issue26960  opened by Chris Beaumont

#26961: Provide an API to help debug super().__getattribute__ results
http://bugs.python.org/issue26961  opened by ncoghlan

#26964: Incorrect documentation for `-u`-flag
http://bugs.python.org/issue26964  opened by olavmrk

#26967: argparse: allow_abbrev=False stops -vv from working
http://bugs.python.org/issue26967  opened by meinersbur

#26968: glob.glob incorrect results under windows when pathname exists
http://bugs.python.org/issue26968  opened by kwarzecha7

#26969: ascynio should provide a policy to address pass-loop-everywher
http://bugs.python.org/issue26969  opened by Ilya.Kulakov

#26970: Replace OpenSSL's CPRNG with system entropy source
http://bugs.python.org/issue26970  opened by christian.heimes



Most recent 15 issues with no replies (15)
==========================================

#26964: Incorrect documentation for `-u`-flag
http://bugs.python.org/issue26964

#26960: urllib hangs when closing connection
http://bugs.python.org/issue26960

#26955: Implement equivalent to `pip.locations.distutils_scheme` in di
http://bugs.python.org/issue26955

#26954: Add Guido's rejection notice to the "with" FAQ
http://bugs.python.org/issue26954

#26942: android: test_ctypes crashes on armv7
http://bugs.python.org/issue26942

#26937: android: test_tarfile fails
http://bugs.python.org/issue26937

#26936: android: test_socket fails
http://bugs.python.org/issue26936

#26935: android: test_os fails
http://bugs.python.org/issue26935

#26934: android: test_faulthandler fails
http://bugs.python.org/issue26934

#26930: Upgrade installers to OpenSSL 1.0.2h
http://bugs.python.org/issue26930

#26920: android: test_sys fails
http://bugs.python.org/issue26920

#26909: Serious performance loss (10 times) when NOT using .drain()
http://bugs.python.org/issue26909

#26907: Add missing getsockopt constants
http://bugs.python.org/issue26907

#26903: ProcessPoolExecutor(max_workers=64) crashes on Windows
http://bugs.python.org/issue26903

#26899: struct.pack_into(), struct.unpack_from() don't document suppor
http://bugs.python.org/issue26899



Most recent 15 issues waiting for review (15)
=============================================

#26970: Replace OpenSSL's CPRNG with system entropy source
http://bugs.python.org/issue26970

#26964: Incorrect documentation for `-u`-flag
http://bugs.python.org/issue26964

#26960: urllib hangs when closing connection
http://bugs.python.org/issue26960

#26959: pickle: respect dispatch for functions again
http://bugs.python.org/issue26959

#26948: Simplify PyImport_ImportModuleLevelObject: avoid temporary tup
http://bugs.python.org/issue26948

#26939: android: test_functools hangs on armv7
http://bugs.python.org/issue26939

#26931: android: test_distutils fails
http://bugs.python.org/issue26931

#26926: Large files are not supported on Android
http://bugs.python.org/issue26926

#26924: android: test_concurrent_futures fails
http://bugs.python.org/issue26924

#26917: unicodedata.normalize(): bug in Hangul Composition
http://bugs.python.org/issue26917

#26906: format(object.__reduce__) fails intermittently
http://bugs.python.org/issue26906

#26904: Difflib quick_ratio() could use Counter()
http://bugs.python.org/issue26904

#26900: Exclude the private API from the stable API
http://bugs.python.org/issue26900

#26896: mix-up with the terms 'importer', 'finder', 'loader' in the im
http://bugs.python.org/issue26896

#26892: debuglevel not honored in urllib
http://bugs.python.org/issue26892



Top 10 most discussed issues (10)
=================================

#26156: Bad name into power operator syntax
http://bugs.python.org/issue26156  12 msgs

#26906: format(object.__reduce__) fails intermittently
http://bugs.python.org/issue26906  11 msgs

#26219: implement per-opcode cache in ceval
http://bugs.python.org/issue26219  10 msgs

#26929: android: test_strptime fails
http://bugs.python.org/issue26929   9 msgs

#26939: android: test_functools hangs on armv7
http://bugs.python.org/issue26939   9 msgs

#26858: android: setting SO_REUSEPORT fails
http://bugs.python.org/issue26858   7 msgs

#26926: Large files are not supported on Android
http://bugs.python.org/issue26926   7 msgs

#26765: Factor out common bytes and bytearray implementation
http://bugs.python.org/issue26765   6 msgs

#26814: [WIP] Add a new _PyObject_FastCall() function which avoids the
http://bugs.python.org/issue26814   6 msgs

#26917: unicodedata.normalize(): bug in Hangul Composition
http://bugs.python.org/issue26917   6 msgs



Issues closed (58)
==================

#10435: Document unicode C-API in reST
http://bugs.python.org/issue10435  closed by berker.peksag

#12135: The spawn function should return stderr.
http://bugs.python.org/issue12135  closed by berker.peksag

#13173: Default values for string.Template
http://bugs.python.org/issue13173  closed by serhiy.storchaka

#13436: compile() doesn't work on ImportFrom with level=None
http://bugs.python.org/issue13436  closed by berker.peksag

#14882: Link/Compile Error on Sun Sparc Solaris10 with gcc3.4.3----pyt
http://bugs.python.org/issue14882  closed by berker.peksag

#18688: Document undocumented Unicode object API
http://bugs.python.org/issue18688  closed by berker.peksag

#18841: math.isfinite fails with Decimal sNAN
http://bugs.python.org/issue18841  closed by berker.peksag

#18916: Various out-of-date Lock text in 3.2+
http://bugs.python.org/issue18916  closed by berker.peksag

#18956: Document useful functions in ???pydoc??? module
http://bugs.python.org/issue18956  closed by berker.peksag

#20120: Percent-signs (%) in .pypirc should not be interpolated
http://bugs.python.org/issue20120  closed by lukasz.langa

#20554: Use specific asserts in optparse test
http://bugs.python.org/issue20554  closed by serhiy.storchaka

#20971: HTML output of difflib
http://bugs.python.org/issue20971  closed by berker.peksag

#21367: multiprocessing.JoinableQueue requires new kwarg
http://bugs.python.org/issue21367  closed by berker.peksag

#21746: urlparse.BaseResult no longer exists
http://bugs.python.org/issue21746  closed by berker.peksag

#23960: PyErr_SetImportError doesn't clean up on some errors
http://bugs.python.org/issue23960  closed by berker.peksag

#24114: ctypes.utils uninitialized variable 'paths'
http://bugs.python.org/issue24114  closed by meador.inge

#24950: FAIL: test_expanduser when $HOME=/
http://bugs.python.org/issue24950  closed by serhiy.storchaka

#25177: OverflowError in statistics.mean when summing large floats
http://bugs.python.org/issue25177  closed by berker.peksag

#25757: Subclasses of property lose docstring
http://bugs.python.org/issue25757  closed by berker.peksag

#25974: Fix statistics.py after the Decimal.as_integer_ratio() change
http://bugs.python.org/issue25974  closed by berker.peksag

#26002: make statistics.median_grouped more efficient
http://bugs.python.org/issue26002  closed by berker.peksag

#26710: ConfigParser: Values in DEFAULT section override defaults pass
http://bugs.python.org/issue26710  closed by Marc.Abramowitz

#26711: Fix comparison of plistlib.Data
http://bugs.python.org/issue26711  closed by serhiy.storchaka

#26756: fileinput handling of unicode errors from standard input
http://bugs.python.org/issue26756  closed by serhiy.storchaka

#26811: segfault due to null pointer in tuple
http://bugs.python.org/issue26811  closed by serhiy.storchaka

#26849: android does not support versioning in SONAME
http://bugs.python.org/issue26849  closed by xdegaye

#26864: urllib.request no_proxy check differs from curl
http://bugs.python.org/issue26864  closed by martin.panter

#26873: xmlrpclib raises when trying to convert an int to string when 
http://bugs.python.org/issue26873  closed by serhiy.storchaka

#26886: Cross-compiling python:3.5.x  fails with "Parser/pgen: Parser/
http://bugs.python.org/issue26886  closed by Peter L2

#26887: Erratum in https://docs.python.org/2.6/library/multiprocessing
http://bugs.python.org/issue26887  closed by zach.ware

#26893: ValueError exception raised when using IntEnum with an attribu
http://bugs.python.org/issue26893  closed by berker.peksag

#26895: regex matching on bytes considers zero byte as end
http://bugs.python.org/issue26895  closed by serhiy.storchaka

#26898: Typo in the documentation of math.isclose()
http://bugs.python.org/issue26898  closed by berker.peksag

#26902: Argument Clinic incorrectly works with custom converter and re
http://bugs.python.org/issue26902  closed by larry

#26905: from multiprocessing.pool import Pool bug on linux
http://bugs.python.org/issue26905  closed by SilentGhost

#26908: "1and 0" evaluated a zero, instead of SyntaxError
http://bugs.python.org/issue26908  closed by berker.peksag

#26910: dictionary literal should not allow duplicate keys
http://bugs.python.org/issue26910  closed by r.david.murray

#26913: statistics.mean of bools
http://bugs.python.org/issue26913  closed by r.david.murray

#26914: Fix formatting of lists in PEP 420
http://bugs.python.org/issue26914  closed by georg.brandl

#26915: Test identity first in membership operation of ItemsView, Valu
http://bugs.python.org/issue26915  closed by rhettinger

#26916: Word duplications in PEPs
http://bugs.python.org/issue26916  closed by serhiy.storchaka

#26918: android: test_pipes fails
http://bugs.python.org/issue26918  closed by serhiy.storchaka

#26921: Incorrect usage of a/an articles in PEPs
http://bugs.python.org/issue26921  closed by serhiy.storchaka

#26922: build from fresh checkout fails
http://bugs.python.org/issue26922  closed by pitrou

#26927: android: test_mmap fails
http://bugs.python.org/issue26927  closed by serhiy.storchaka

#26932: RTLD_* macros are not defined on Android
http://bugs.python.org/issue26932  closed by serhiy.storchaka

#26933: android: test_posixpath fails
http://bugs.python.org/issue26933  closed by serhiy.storchaka

#26943: datetime.datetime.strptime() raises an exception
http://bugs.python.org/issue26943  closed by SilentGhost

#26946: os.path.realpath.__name__ is 'abspath'
http://bugs.python.org/issue26946  closed by brett.cannon

#26950: Typo in unittest.mock > Autospeccing documentation
http://bugs.python.org/issue26950  closed by berker.peksag

#26953: Failed building wheel for greenlet
http://bugs.python.org/issue26953  closed by berker.peksag

#26956: About Idle-x  version not updated for  1.13
http://bugs.python.org/issue26956  closed by terry.reedy

#26957: Docs: remove duplicate "the" in description of `%z` in datetim
http://bugs.python.org/issue26957  closed by berker.peksag

#26958: Suggestion to imporve queue.full reliability
http://bugs.python.org/issue26958  closed by rhettinger

#26962: curses documenation- keys for Page Up and Page Down are revers
http://bugs.python.org/issue26962  closed by berker.peksag

#26963: compile() expected string without null bytes
http://bugs.python.org/issue26963  closed by serhiy.storchaka

#26965: spam
http://bugs.python.org/issue26965  closed by ned.deily

#26966: Spam
http://bugs.python.org/issue26966  closed by berker.peksag

From amk at amk.ca  Fri May  6 13:11:34 2016
From: amk at amk.ca (A.M. Kuchling)
Date: Fri, 6 May 2016 13:11:34 -0400
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <9F53FF72-6CCB-482A-B4EB-E8165E404362@stufft.io>
References: <loom.20160503T205904-379@post.gmane.org>
 <20160505234230.GB14936@k3>
 <CAP1=2W4KWbnPJiUwWASDYyiX_QYFBb611G8JJ4Esi_PY-kuzOQ@mail.gmail.com>
 <20160506003527.GA30649@k3>
 <9F53FF72-6CCB-482A-B4EB-E8165E404362@stufft.io>
Message-ID: <20160506171134.GA841@DATLANDREWK.local>

On Thu, May 05, 2016 at 10:31:48PM -0400, Donald Stufft wrote:
> I don't believe we've ever told someone that something can't happen because of
> Warehouse, only that *I* won't implement something until after Warehouse. That
> often times means that something won't happen until after Warehouse because of
> the severe shortage of people with enough time and motivation to work on this
> stuff but if someone did step up more things would get done.

Could the PSF help with this, whether by paying for Donald's (or a
consultant's) time to whatever do final implementation, polishing,
testing, or sysadmin work is required?

--amk

From josephnunn at gmail.com  Fri May  6 13:43:12 2016
From: josephnunn at gmail.com (Joseph Lee Nunn III)
Date: Fri, 6 May 2016 10:43:12 -0700
Subject: [Python-Dev] Wrong OSX platform in sysconfig.py causing
 installation problems
In-Reply-To: <905F2C35-855E-4733-9023-4B30EFF228B2@python.org>
References: <loom.20160506T111745-702@post.gmane.org>
 <905F2C35-855E-4733-9023-4B30EFF228B2@python.org>
Message-ID: <F36B85F7-3553-4BC7-8847-6189B56736E4@gmail.com>

Then I have found the right list to post on, for my problem is a bug in the Python 3.5.1 distribution.  Albeit one which will only manifest in certain situations.

The presence of numerous `-isysroot /Developer/SDKs/MacOSX10.6.sdk` directives in the following file

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/_sysconfigdata.py

Will cause errors for people who have kept around the old /Developer directory tree, and old SDKs like 10.6, for building backwards compatible old code.  Newer versions of the SDKs are now kept inside the Xcode application itself.  And then they will only have a problem when trying to install a package with C extensions, which in my case was Persistence, a dependency of ZODB.  And even more so, ONLY when trying to pip install said package into a virtual environment created by 3.5?s pyvenv command, if installing the package into the system wide Python there will be no error.

Such is this corner-case hell I have found myself, until I decided to dump my older /Developer directory tree and SDKs, as those were for a project long defunct.  You see the -isysroot flag has the unusual behavior where instead of reporting an error when it cannot find the newly specified system root it simply has no effect.  So if you don?t have the SDK specified in the location specified then the flag does nothing, as I assume is the case for 99% of people. 

I understand the need to make Python 3.5 backwards compatible, but doing it this way would appear to have the side effect of breaking the ability to install packages using C extensions in virtual environments, or at least it did for me.

Joseph Nunn
josephnunn at gmail.com
joseph.nunn at uci.edu


> On May 06 , 2016, at 4:20 AM, Ned Deily <nad at python.org> wrote:
> 
> 
> On May 6, 2016, at 05:42, Joseph Nunn <josephnunn at gmail.com> wrote:
>> Hello Everyone, my first post.  I would also like to subscribe but 
>> haven't seen how yet.  
>> 
>> So I've an issue with how the 3.5.1 OSX 
>> package installs on OSX.  The problem was first made apparent 
>> when trying to create a virtual environment and `pip install ZODB`, 
>> which failed as clang and the linker could not find either the 
>> include or lib directories for a dependency.  By adding a -v option
>> to the compiler options I was able to see in the  compiler flags 
>> the following:
>> 
>> -isysroot /Developer/SDKs/MacOSX10.6.sdk
>> 
>> Which was not present in the compiler flags of other people 
>> who I was talking with. [...]
> 
> Hi Joseph!
> 
> Welcome!  Note this list is for the development of Python itself, not for usage or installation issues.  The Python Developer's Guide (https://docs.python.org/devguide/) including its FAQ gives information about mailing lists and other support forums to get help with general Python issues.  That said, without more information, it would be difficult to know exactly what problem you are running into but, since you mention missing headers, my guess is that you need to install Apple's Command Line Tools as described here:
> 
> https://docs.python.org/devguide/setup.html#build-dependencies
> 
> Installing Xcode by itself is not sufficient.  The sysroot and flag values you note in _sysconfigdata.py are normal and there for compatibility with installing on older versions of OS X; they should not cause a problem on newer releases.  For what it's worth, I was able to install ZODB with the current python.org 3.5.1 on OS X 10.11.4 using either venv or virtualenv.
> 
> Good luck!
> 
> --Ned
> 
> P.S. Mailing list info is available here: https://mail.python.org.  The bug tracker is here: https://bugs.python.org
> 
> --
>  Ned Deily
>  nad at python.org -- []
> 


From donald at stufft.io  Fri May  6 15:31:40 2016
From: donald at stufft.io (Donald Stufft)
Date: Fri, 6 May 2016 15:31:40 -0400
Subject: [Python-Dev] Yearly PyPI breakage
In-Reply-To: <20160506171134.GA841@DATLANDREWK.local>
References: <loom.20160503T205904-379@post.gmane.org>
 <20160505234230.GB14936@k3>
 <CAP1=2W4KWbnPJiUwWASDYyiX_QYFBb611G8JJ4Esi_PY-kuzOQ@mail.gmail.com>
 <20160506003527.GA30649@k3> <9F53FF72-6CCB-482A-B4EB-E8165E404362@stufft.io>
 <20160506171134.GA841@DATLANDREWK.local>
Message-ID: <2D6D5D97-3695-450D-9C3E-60E8EFFE7D5B@stufft.io>


> On May 6, 2016, at 1:11 PM, A.M. Kuchling <amk at amk.ca> wrote:
> 
> On Thu, May 05, 2016 at 10:31:48PM -0400, Donald Stufft wrote:
>> I don't believe we've ever told someone that something can't happen because of
>> Warehouse, only that *I* won't implement something until after Warehouse. That
>> often times means that something won't happen until after Warehouse because of
>> the severe shortage of people with enough time and motivation to work on this
>> stuff but if someone did step up more things would get done.
> 
> Could the PSF help with this, whether by paying for Donald's (or a
> consultant's) time to whatever do final implementation, polishing,
> testing, or sysadmin work is required?


Personally, my time is already paid for by HPE to work on this stuff but I?m only one person. The PSF could pay for others to help with that though.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160506/a14dd6b7/attachment.sig>

From camillamon at gmail.com  Fri May  6 17:14:32 2016
From: camillamon at gmail.com (Camilla)
Date: Fri, 6 May 2016 22:14:32 +0100
Subject: [Python-Dev] Anyone want to lead the sprints at PyCon US 2016?
In-Reply-To: <CAP1=2W4Sb94TPeViPwTmOEkCNikjm4+BCVM32E4wF-j54-tyFw@mail.gmail.com>
References: <CADx+GQPKnvWWB1e_kL6Hqh7skuDiVRBMK8szs5QGS4NGt-rnJg@mail.gmail.com>
 <CAP1=2W4sDeTTZWUxkoieD+bJprXFK=5NfBkLUg=vopgNFPSMYg@mail.gmail.com>
 <CAP1=2W4Sb94TPeViPwTmOEkCNikjm4+BCVM32E4wF-j54-tyFw@mail.gmail.com>
Message-ID: <CAHh6JByvUwZ6HhTAGo19Ng0Mc=R-3-nwJb744ppLEWRPfP5qwQ@mail.gmail.com>

I was thinking about holding a Patch Review Party/Sprint, which would
provide people unfamiliar with the Python dev process a way to contribute
to the project and get familiar with running tests, applying patches and so
forth. I have a list of easy-ish patches that I wanted to take a look at
and I could expand that and use those as a starting for people who don't
have any particular bug tracker issues in mind.
I'm not a patch review guru by any means, though. Also not sure if this is
a good idea or if this is just late night caffeine talking.


2016-04-28 20:07 GMT+01:00 Brett Cannon <brett at python.org>:

> No one stepped forward to lead the sprints this year, so I will put myself
> as the sprint leader and lean on everyone else who appears to help. :)
>
>
> On Tue, 5 Apr 2016 at 09:36 Brett Cannon <brett at python.org> wrote:
>
>> The call has started to go out for sprint groups to list themselves
>> online. Anyone want to specifically lead the core sprint this year? If no
>> one specifically does then I will sign us up and do my usual thing of
>> pointing people at the devguide and encourage people to ask questions but
>> not do a lot of hand-holding (I'm expecting to be busy either working on
>> GitHub migration stuff or doing other things that I have been neglecting
>> due to my GitHub migration work).
>>
>> ---------- Forwarded message ---------
>> From: Ewa Jodlowska <ewa at python.org>
>> Date: Mon, 4 Apr 2016 at 07:14
>> Subject: [PSF-Community] Sprinting at PyCon US 2016
>> To: <psf-community at python.org>
>>
>>
>> Are you coming to PyCon US? Have you thought about sprinting?
>>
>> The coding Sprints are the hidden gem of PyCon, up to 4 days (June 2-5)
>> of coding with many Python projects and their maintainers. And if you're
>> coming to PyCon, taking part in the Sprints is easy!
>>
>> You don?t need to change your registration* to join the Sprints. There?s
>> no additional registration fee, and you even get lunch. You do need to
>> cover the additional lodging and other meals, but that?s it. If you?ve
>> booked a room through the PyCon registration system, you'll need to contact
>> the registration team at pycon2016 at cteusa.com as soon as possible to
>> request the extra nights. The sprinting itself (along with lunch every day)
>> is free, so your only expenses are your room and other meals.
>>
>> If you're interested in what projects will be sprinting, just keep an eye
>> on the sprints page on the PyCon web site at
>> https://us.pycon.org/2016/community/sprints/ Be sure to check back, as
>> groups are being added all the time.
>>
>> If you haven't sprinted before, or if you just need to brush up on
>> sprinting tools and techniques, there will again be an 'Intro to Sprinting'
>> session the evening of June 1, lead by Shauna Gordon-McKeon and other
>> members of Python community. To grab a free ticket for this session, just
>> visit
>> https://www.eventbrite.com/e/introduction-to-open-source-the-pycon-sprints-tickets-22435151141
>> .
>>
>> *Please note that conference registration is sold out, but you do not
>> need a conference registration to come to the Sprints.
>>
>> _______________________________________________
>> PSF-Community mailing list
>> PSF-Community at python.org
>> https://mail.python.org/mailman/listinfo/psf-community
>>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/camillamon%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160506/653d0558/attachment.html>

From brett at python.org  Fri May  6 17:38:22 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 06 May 2016 21:38:22 +0000
Subject: [Python-Dev] Anyone want to lead the sprints at PyCon US 2016?
In-Reply-To: <CAHh6JByvUwZ6HhTAGo19Ng0Mc=R-3-nwJb744ppLEWRPfP5qwQ@mail.gmail.com>
References: <CADx+GQPKnvWWB1e_kL6Hqh7skuDiVRBMK8szs5QGS4NGt-rnJg@mail.gmail.com>
 <CAP1=2W4sDeTTZWUxkoieD+bJprXFK=5NfBkLUg=vopgNFPSMYg@mail.gmail.com>
 <CAP1=2W4Sb94TPeViPwTmOEkCNikjm4+BCVM32E4wF-j54-tyFw@mail.gmail.com>
 <CAHh6JByvUwZ6HhTAGo19Ng0Mc=R-3-nwJb744ppLEWRPfP5qwQ@mail.gmail.com>
Message-ID: <CAP1=2W5Us-zM9-yHAs4AtFHjTXsnDS1c_B428vbk=6MEMCwk0Q@mail.gmail.com>

On Fri, 6 May 2016 at 14:14 Camilla <camillamon at gmail.com> wrote:

> I was thinking about holding a Patch Review Party/Sprint, which would
> provide people unfamiliar with the Python dev process a way to contribute
> to the project and get familiar with running tests, applying patches and so
> forth. I have a list of easy-ish patches that I wanted to take a look at
> and I could expand that and use those as a starting for people who don't
> have any particular bug tracker issues in mind.
> I'm not a patch review guru by any means, though. Also not sure if this is
> a good idea or if this is just late night caffeine talking.
>

I have absolutely no problem if you want to pitch this idea to new
contributors who show up at the sprints!

-Brett


>
>
> 2016-04-28 20:07 GMT+01:00 Brett Cannon <brett at python.org>:
>
>> No one stepped forward to lead the sprints this year, so I will put
>> myself as the sprint leader and lean on everyone else who appears to help.
>> :)
>>
>>
>> On Tue, 5 Apr 2016 at 09:36 Brett Cannon <brett at python.org> wrote:
>>
>>> The call has started to go out for sprint groups to list themselves
>>> online. Anyone want to specifically lead the core sprint this year? If no
>>> one specifically does then I will sign us up and do my usual thing of
>>> pointing people at the devguide and encourage people to ask questions but
>>> not do a lot of hand-holding (I'm expecting to be busy either working on
>>> GitHub migration stuff or doing other things that I have been neglecting
>>> due to my GitHub migration work).
>>>
>>> ---------- Forwarded message ---------
>>> From: Ewa Jodlowska <ewa at python.org>
>>> Date: Mon, 4 Apr 2016 at 07:14
>>> Subject: [PSF-Community] Sprinting at PyCon US 2016
>>> To: <psf-community at python.org>
>>>
>>>
>>> Are you coming to PyCon US? Have you thought about sprinting?
>>>
>>> The coding Sprints are the hidden gem of PyCon, up to 4 days (June 2-5)
>>> of coding with many Python projects and their maintainers. And if you're
>>> coming to PyCon, taking part in the Sprints is easy!
>>>
>>> You don?t need to change your registration* to join the Sprints. There?s
>>> no additional registration fee, and you even get lunch. You do need to
>>> cover the additional lodging and other meals, but that?s it. If you?ve
>>> booked a room through the PyCon registration system, you'll need to contact
>>> the registration team at pycon2016 at cteusa.com as soon as possible to
>>> request the extra nights. The sprinting itself (along with lunch every day)
>>> is free, so your only expenses are your room and other meals.
>>>
>>> If you're interested in what projects will be sprinting, just keep an
>>> eye on the sprints page on the PyCon web site at
>>> https://us.pycon.org/2016/community/sprints/ Be sure to check back, as
>>> groups are being added all the time.
>>>
>>> If you haven't sprinted before, or if you just need to brush up on
>>> sprinting tools and techniques, there will again be an 'Intro to Sprinting'
>>> session the evening of June 1, lead by Shauna Gordon-McKeon and other
>>> members of Python community. To grab a free ticket for this session, just
>>> visit
>>> https://www.eventbrite.com/e/introduction-to-open-source-the-pycon-sprints-tickets-22435151141
>>> .
>>>
>>> *Please note that conference registration is sold out, but you do not
>>> need a conference registration to come to the Sprints.
>>>
>>> _______________________________________________
>>> PSF-Community mailing list
>>> PSF-Community at python.org
>>> https://mail.python.org/mailman/listinfo/psf-community
>>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/camillamon%40gmail.com
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160506/2b2f3fdf/attachment-0001.html>

From willingc at willingconsulting.com  Fri May  6 17:49:24 2016
From: willingc at willingconsulting.com (Carol Willing)
Date: Fri, 06 May 2016 14:49:24 -0700
Subject: [Python-Dev] Anyone want to lead the sprints at PyCon US 2016?
In-Reply-To: <CAP1=2W5Us-zM9-yHAs4AtFHjTXsnDS1c_B428vbk=6MEMCwk0Q@mail.gmail.com>
References: <CADx+GQPKnvWWB1e_kL6Hqh7skuDiVRBMK8szs5QGS4NGt-rnJg@mail.gmail.com>
 <CAP1=2W4sDeTTZWUxkoieD+bJprXFK=5NfBkLUg=vopgNFPSMYg@mail.gmail.com>
 <CAP1=2W4Sb94TPeViPwTmOEkCNikjm4+BCVM32E4wF-j54-tyFw@mail.gmail.com>
 <CAHh6JByvUwZ6HhTAGo19Ng0Mc=R-3-nwJb744ppLEWRPfP5qwQ@mail.gmail.com>
 <CAP1=2W5Us-zM9-yHAs4AtFHjTXsnDS1c_B428vbk=6MEMCwk0Q@mail.gmail.com>
Message-ID: <BF9C14B2-0DB0-4C1D-9473-B129A13C35B6@willingconsulting.com>

On 6 May 2016, at 14:38, Brett Cannon wrote:

> On Fri, 6 May 2016 at 14:14 Camilla <camillamon at gmail.com> wrote:
>
>> I was thinking about holding a Patch Review Party/Sprint, which would
>> provide people unfamiliar with the Python dev process a way to 
>> contribute
>> to the project and get familiar with running tests, applying patches 
>> and so
>> forth. I have a list of easy-ish patches that I wanted to take a look 
>> at
>> and I could expand that and use those as a starting for people who 
>> don't
>> have any particular bug tracker issues in mind.
>> I'm not a patch review guru by any means, though. Also not sure if 
>> this is
>> a good idea or if this is just late night caffeine talking.
>>
>
> I have absolutely no problem if you want to pitch this idea to new
> contributors who show up at the sprints!
>
> -Brett

Camilla,

I would be happy to support your effort. I find it a wonderful idea!

Carol

Carol Willing
Research Software Engineer, Project Jupyter @ Cal Poly
Director, Python Software Foundation

>>
>>
>> 2016-04-28 20:07 GMT+01:00 Brett Cannon <brett at python.org>:
>>
>>> No one stepped forward to lead the sprints this year, so I will put
>>> myself as the sprint leader and lean on everyone else who appears to 
>>> help.
>>> :)
>>>
>>>
>>> On Tue, 5 Apr 2016 at 09:36 Brett Cannon <brett at python.org> wrote:
>>>
>>>> The call has started to go out for sprint groups to list themselves
>>>> online. Anyone want to specifically lead the core sprint this year? 
>>>> If no
>>>> one specifically does then I will sign us up and do my usual thing 
>>>> of
>>>> pointing people at the devguide and encourage people to ask 
>>>> questions but
>>>> not do a lot of hand-holding (I'm expecting to be busy either 
>>>> working on
>>>> GitHub migration stuff or doing other things that I have been 
>>>> neglecting
>>>> due to my GitHub migration work).
>>>>
>>>> ---------- Forwarded message ---------
>>>> From: Ewa Jodlowska <ewa at python.org>
>>>> Date: Mon, 4 Apr 2016 at 07:14
>>>> Subject: [PSF-Community] Sprinting at PyCon US 2016
>>>> To: <psf-community at python.org>
>>>>
>>>>
>>>> Are you coming to PyCon US? Have you thought about sprinting?
>>>>
>>>> The coding Sprints are the hidden gem of PyCon, up to 4 days (June 
>>>> 2-5)
>>>> of coding with many Python projects and their maintainers. And if 
>>>> you're
>>>> coming to PyCon, taking part in the Sprints is easy!
>>>>
>>>> You don?t need to change your registration* to join the Sprints. 
>>>> There?s
>>>> no additional registration fee, and you even get lunch. You do need 
>>>> to
>>>> cover the additional lodging and other meals, but that?s it. If 
>>>> you?ve
>>>> booked a room through the PyCon registration system, you'll need to 
>>>> contact
>>>> the registration team at pycon2016 at cteusa.com as soon as possible 
>>>> to
>>>> request the extra nights. The sprinting itself (along with lunch 
>>>> every day)
>>>> is free, so your only expenses are your room and other meals.
>>>>
>>>> If you're interested in what projects will be sprinting, just keep 
>>>> an
>>>> eye on the sprints page on the PyCon web site at
>>>> https://us.pycon.org/2016/community/sprints/ Be sure to check back, 
>>>> as
>>>> groups are being added all the time.
>>>>
>>>> If you haven't sprinted before, or if you just need to brush up on
>>>> sprinting tools and techniques, there will again be an 'Intro to 
>>>> Sprinting'
>>>> session the evening of June 1, lead by Shauna Gordon-McKeon and 
>>>> other
>>>> members of Python community. To grab a free ticket for this 
>>>> session, just
>>>> visit
>>>> https://www.eventbrite.com/e/introduction-to-open-source-the-pycon-sprints-tickets-22435151141
>>>> .
>>>>
>>>> *Please note that conference registration is sold out, but you do 
>>>> not
>>>> need a conference registration to come to the Sprints.
>>>>
>>>> _______________________________________________
>>>> PSF-Community mailing list
>>>> PSF-Community at python.org
>>>> https://mail.python.org/mailman/listinfo/psf-community
>>>>
>>>
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/camillamon%40gmail.com
>>>
>>>
>>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/willingc%40willingconsulting.com



From victor.stinner at gmail.com  Fri May  6 17:58:52 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 6 May 2016 23:58:52 +0200
Subject: [Python-Dev] Wordcode: new regular bytecode using 16-bit units
In-Reply-To: <1462472902.2822020.599321113.046AABAF@webmail.messagingengine.com>
References: <1462472902.2822020.599321113.046AABAF@webmail.messagingengine.com>
Message-ID: <CAMpsgwYGV2bee_hyjc6hsTnL6C+9zW2iNX+EVZOBmPhUpXv54w@mail.gmail.com>

Oh nice. Did you see my recent "bytecode" project?
http://bytecode.readthedocs.io/

Victor
Le 5 mai 2016 8:30 PM, <zreed at fastmail.com> a ?crit :

> Here is something I wrote because I was also unsatisfied with byteplay's
> API: https://github.com/zachariahreed/byteasm. Maybe it's useful in a
> discussion of "minimum viable" api for bytecode manipulation.
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160506/ddc49a9b/attachment.html>

From storchaka at gmail.com  Sat May  7 18:39:13 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 8 May 2016 01:39:13 +0300
Subject: [Python-Dev] Return type of alternative constructors
Message-ID: <nglqqi$83q$1@ger.gmane.org>

Some types have alternative constructors -- class methods used to create 
an instance of the class. For example: int.from_bytes(), 
float.fromhex(), dict.fromkeys(), Decimal.from_float().

But what should return these methods for subclasses? Should they return 
an instance of base class or an instance of subclass? Almost all 
alternative constructors return an instance of subclass (exceptions are 
new in 3.6 bytes.fromhex() and bytearray.fromhex() that return bare 
bytes and bytearray). But there is a problem, because this allows to 
break invariants provided by the main constructor.

For example, there are only two instances of the bool class: False and 
True. But with the from_bytes() method inherited from int you can create 
new boolean values!

    >>> Confusion = bool.from_bytes(b'\2', 'big')
    >>> isinstance(Confusion, bool)
    True
    >>> Confusion == True
    False
    >>> bool(Confusion)
    True
    >>> Confusion
    False
    >>> not Confusion
    False

bool is just the most impressive example, the same problem exists with 
IntEnum and other enums derived from float, Decimal, datetime. [1]

The simplest solution is to return an instance of base class. But this 
can breaks a code, and for this case we should be use static method 
(like str.maketrans), not class method.

Should alternative constructor call __new__ and __init__ methods? Thay 
can change signature in derived class. Should it complain if __new__ or 
__init__ were overridden?

[1] http://bugs.python.org/issue23640


From ethan at stoneleaf.us  Sat May  7 19:06:51 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 07 May 2016 16:06:51 -0700
Subject: [Python-Dev] Tracker Etiquette
Message-ID: <572E750B.80706@stoneleaf.us>

I just reviewed an issue on the tracker, and found that many, if not 
all, of my comments made at the time I completely disagree with (both 
now, and before -- I'm going to claim sleep deprivation).

Obviously I should post a new comment explaining my thoughts about the 
issue, but what should I do with the old comments?  Is it acceptable to 
delete them?

--
~Ethan~

From brett at python.org  Sat May  7 19:10:43 2016
From: brett at python.org (Brett Cannon)
Date: Sat, 07 May 2016 23:10:43 +0000
Subject: [Python-Dev] Tracker Etiquette
In-Reply-To: <572E750B.80706@stoneleaf.us>
References: <572E750B.80706@stoneleaf.us>
Message-ID: <CAP1=2W7unZ_JURXReSMg8Q1-nY5zj7BUgp449axEA-p41JKngw@mail.gmail.com>

I think if you leave a comment that to deleted your own comments and why
you did it then it's fine.

On Sat, May 7, 2016, 16:07 Ethan Furman <ethan at stoneleaf.us> wrote:

> I just reviewed an issue on the tracker, and found that many, if not
> all, of my comments made at the time I completely disagree with (both
> now, and before -- I'm going to claim sleep deprivation).
>
> Obviously I should post a new comment explaining my thoughts about the
> issue, but what should I do with the old comments?  Is it acceptable to
> delete them?
>
> --
> ~Ethan~
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160507/74bf2c52/attachment.html>

From ethan at stoneleaf.us  Sat May  7 19:14:12 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 07 May 2016 16:14:12 -0700
Subject: [Python-Dev] Return type of alternative constructors
In-Reply-To: <nglqqi$83q$1@ger.gmane.org>
References: <nglqqi$83q$1@ger.gmane.org>
Message-ID: <572E76C4.3020907@stoneleaf.us>

On 05/07/2016 03:39 PM, Serhiy Storchaka wrote:
> Some types have alternative constructors -- class methods used to create
> an instance of the class. For example: int.from_bytes(),
> float.fromhex(), dict.fromkeys(), Decimal.from_float().
>
> But what should return these methods for subclasses? Should they return
> an instance of base class or an instance of subclass? Almost all
> alternative constructors return an instance of subclass (exceptions are
> new in 3.6 bytes.fromhex() and bytearray.fromhex() that return bare
> bytes and bytearray). But there is a problem, because this allows to
> break invariants provided by the main constructor.

Please ignore my comments in that issue.  I actually prefer that class 
constructors go through the subclass' __new__ and __init__.  Overriding 
parent class methods for the sole purpose of getting the subclass's type 
is quite irritating.

--
~Ethan~

From python at mrabarnett.plus.com  Sat May  7 19:17:19 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Sun, 8 May 2016 00:17:19 +0100
Subject: [Python-Dev] Tracker Etiquette
In-Reply-To: <572E750B.80706@stoneleaf.us>
References: <572E750B.80706@stoneleaf.us>
Message-ID: <8d44f36f-e18d-5540-e6af-c13392bd7e44@mrabarnett.plus.com>

On 2016-05-08 00:06, Ethan Furman wrote:
> I just reviewed an issue on the tracker, and found that many, if not
> all, of my comments made at the time I completely disagree with (both
> now, and before -- I'm going to claim sleep deprivation).
>
> Obviously I should post a new comment explaining my thoughts about the
> issue, but what should I do with the old comments?  Is it acceptable to
> delete them?
>
I think you shouldn't delete them. It would be better just to say that 
you've changed your mind and explain why.


From senthil at uthcode.com  Sat May  7 19:23:32 2016
From: senthil at uthcode.com (Senthil Kumaran)
Date: Sat, 7 May 2016 16:23:32 -0700
Subject: [Python-Dev] Tracker Etiquette
In-Reply-To: <8d44f36f-e18d-5540-e6af-c13392bd7e44@mrabarnett.plus.com>
References: <572E750B.80706@stoneleaf.us>
 <8d44f36f-e18d-5540-e6af-c13392bd7e44@mrabarnett.plus.com>
Message-ID: <CAPOVWOTTzYpo77QxQmL0ZeOHmTqgo709ToR-Q3FnMc08S1t1HQ@mail.gmail.com>

On Sat, May 7, 2016 at 4:17 PM, MRAB <python at mrabarnett.plus.com> wrote:

> I think you shouldn't delete them. It would be better just to say that
> you've changed your mind and explain why.
>

I support this. Please leave your new comments correcting previous one and
support your current stance. I think, it is alright to make revisions in
comments.

-- 
Senthil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160507/b7f344d0/attachment.html>

From vadmium+py at gmail.com  Sat May  7 19:57:05 2016
From: vadmium+py at gmail.com (Martin Panter)
Date: Sat, 7 May 2016 23:57:05 +0000
Subject: [Python-Dev] Tracker Etiquette
In-Reply-To: <CAPOVWOTTzYpo77QxQmL0ZeOHmTqgo709ToR-Q3FnMc08S1t1HQ@mail.gmail.com>
References: <572E750B.80706@stoneleaf.us>
 <8d44f36f-e18d-5540-e6af-c13392bd7e44@mrabarnett.plus.com>
 <CAPOVWOTTzYpo77QxQmL0ZeOHmTqgo709ToR-Q3FnMc08S1t1HQ@mail.gmail.com>
Message-ID: <CA+eR4cFBBBd3_aLT6nOs=61osF0d4Y+eNZcj91j6tX5jR2CB-A@mail.gmail.com>

On 7 May 2016 at 23:23, Senthil Kumaran <senthil at uthcode.com> wrote:
>
> On Sat, May 7, 2016 at 4:17 PM, MRAB <python at mrabarnett.plus.com> wrote:
>>
>> I think you shouldn't delete them. It would be better just to say that
>> you've changed your mind and explain why.
>
>
> I support this. Please leave your new comments correcting previous one and
> support your current stance. I think, it is alright to make revisions in
> comments.

Taking a middle ground between this and what Brett said, I would say
it is okay to delete stuff if it is fresh and hasn?t been replied to
(but maybe leave a message if it is not obvious why you delete it). On
the other hand, avoid deleting stuff if it would harm the overall
thread.

From chris.barker at noaa.gov  Sun May  8 01:59:16 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Sat, 7 May 2016 22:59:16 -0700
Subject: [Python-Dev] Wrong OSX platform in sysconfig.py causing
 installation problems
In-Reply-To: <F36B85F7-3553-4BC7-8847-6189B56736E4@gmail.com>
References: <loom.20160506T111745-702@post.gmane.org>
 <905F2C35-855E-4733-9023-4B30EFF228B2@python.org>
 <F36B85F7-3553-4BC7-8847-6189B56736E4@gmail.com>
Message-ID: <CALGmxE+hWRzeqhFwjYVSP1jBnSFJtbQjyvw_rYeVyT0tevZ8HA@mail.gmail.com>

On Fri, May 6, 2016 at 10:43 AM, Joseph Lee Nunn III <josephnunn at gmail.com>
wrote:

> Then I have found the right list to post on, for my problem is a bug in
> the Python 3.5.1 distribution.  Albeit one which will only manifest in
> certain situations.
>
> The presence of numerous `-isysroot /Developer/SDKs/MacOSX10.6.sdk`
> directives in the following file
>
>
> /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/_sysconfigdata.py
>

As I understand it, that is all there so that when extensions are built,
they are linked against that SDK -- so you can build extensions that can
then be run on any system 10.6 and greater.


> Will cause errors for people who have kept around the old /Developer
> directory tree, and old SDKs like 10.6, for building backwards compatible
> old code.


what are the errors -- wouldn't the old SDK be found, and used, and that's
exactly what's intended?


>   Newer versions of the SDKs are now kept inside the Xcode application
> itself.  And then they will only have a problem when trying to install a
> package with C extensions,


well, sure. of course this is an issue for building C extensions.


> which in my case was Persistence, a dependency of ZODB.  And even more so,
> ONLY when trying to pip install said package into a virtual environment
> created by 3.5?s pyvenv command, if installing the package into the system
> wide Python there will be no error.
>

hmm, As I understand it the building process should be exactly the same
whether you are on a virtualenv or not. but if so -- maybe the "solution"
is to build a wheel outside virtualenv, and then install that wheel in the
virtualenv. But I agree, it should work, of course.


> You see the -isysroot flag has the unusual behavior where instead of
> reporting an error when it cannot find the newly specified system root it
> simply has no effect.  So if you don?t have the SDK specified in the
> location specified then the flag does nothing, as I assume is the case for
> 99% of people.
>

Indeed, that's been working fine for me for a while. However it's this
behavior that is actually the bug -- extensions are supposed to be build
against the 10.6 SDK -- and that's not what's happening in this case


> I understand the need to make Python 3.5 backwards compatible, but doing
> it this way would appear to have the side effect of breaking the ability to
> install packages using C extensions in virtual environments, or at least it
> did for me.
>

It does sound like you've hit a bug, but I actually think you've
misidentified the bug -- I'm trying to understand why it's not working in a
virtualenv -- THAT's the bug.

And I have no idea what to do about it -- thanks Apple for moving things --
that's a great way to keep backward compatibility!

Ned: I'm trying to explore this a bit now (on a 10.9 system), so I followed
the instructions in the Python developers guide, and did:

xcode-select --install

And was a bit surprised that It did a substantial download -- I thought I
already had the XCode command line tools -- but whatever. After running
that, I found I had the 10.9 and 10.10 SDKs (I think that installed the
10.10 one, I previously had only 10.9). And they are buried deep in
XCode.app ....

OK, but how would I go about getting the 10.6 SDK??? This SO post:

http://stackoverflow.com/questions/11989007/where-to-get-macos-sdk-10-6-for-xcode

indicates that it's a serious pain in the $%^% -- thank Apple! But, as far
as I can tell, if you install an old XCode, you will get the older SDKs,
and installed into the old location.

(this is why I kept a 10.6 system around for a long time -- to build python
extensions...)

But what should we do? IF you are building extensions only for yourself,
then I think whatever the heck SDK you have will work fine -- but how to
build for distribution?

.... Trying to decide if I want to go through the hassle of installing an
old XCode to see if i can replicate the bug the OP is reporting....

-CHB


> https://docs.python.org/devguide/setup.html#build-dependencies
>




> >
> > Installing Xcode by itself is not sufficient.  The sysroot and flag
> values you note in _sysconfigdata.py are normal and there for compatibility
> with installing on older versions of OS X; they should not cause a problem
> on newer releases.  For what it's worth, I was able to install ZODB with
> the current python.org 3.5.1 on OS X 10.11.4 using either venv or
> virtualenv.
> >
> > Good luck!
> >
> > --Ned
> >
> > P.S. Mailing list info is available here: https://mail.python.org.  The
> bug tracker is here: https://bugs.python.org
> >
> > --
> >  Ned Deily
> >  nad at python.org -- []
> >
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160507/c3bd4bf8/attachment.html>

From chris.barker at noaa.gov  Sun May  8 02:52:48 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Sat, 7 May 2016 23:52:48 -0700
Subject: [Python-Dev] Wrong OSX platform in sysconfig.py causing
 installation problems
In-Reply-To: <CALGmxE+hWRzeqhFwjYVSP1jBnSFJtbQjyvw_rYeVyT0tevZ8HA@mail.gmail.com>
References: <loom.20160506T111745-702@post.gmane.org>
 <905F2C35-855E-4733-9023-4B30EFF228B2@python.org>
 <F36B85F7-3553-4BC7-8847-6189B56736E4@gmail.com>
 <CALGmxE+hWRzeqhFwjYVSP1jBnSFJtbQjyvw_rYeVyT0tevZ8HA@mail.gmail.com>
Message-ID: <CALGmxEJaHqP5Ymd6SOzJgR4O+X3kBB9DJ_d8e_pu0+4ezNx0cw@mail.gmail.com>

On Sat, May 7, 2016 at 10:59 PM, Chris Barker <chris.barker at noaa.gov> wrote:

>
> .... Trying to decide if I want to go through the hassle of installing an
> old XCode to see if i can replicate the bug the OP is reporting....
>

OK, I found this gitHub repo with all the old SDKs:

https://github.com/phracker/MacOSX-SDKs

I put a copy of the 10.6 SDk in it's old location:

/Developer/SDKs/MacOSX10.6.sdk/

/before I did that, I did a pip install of zodb in a virtualenv, and all
seemed to go fine. THe extensions rae linked against :

$ otool -L cPersistence.cpython-35m-darwin.so
cPersistence.cpython-35m-darwin.so:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1197.1.1)

where that comes from -- who knows??

then I tried again, after putting the sdk in place:

[note: serious pain the $%^# to get pip to stop using it's cache! ]

and everything seemed to build fine, and at least one simple test of zodb
works fine. And this is how the extensions are linked:

$  otool -L cPersistence.cpython-35m-darwin.so
cPersistence.cpython-35m-darwin.so:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 125.2.11)

which is the same as above, but the "current version" is much older -- so
does that mean it used the 10.6 SDK????

Anyway, I can't reproduce the OP's bug.....

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160507/4536669f/attachment-0001.html>

From guido at python.org  Sun May  8 03:03:45 2016
From: guido at python.org (Guido van Rossum)
Date: Sun, 8 May 2016 00:03:45 -0700
Subject: [Python-Dev] Return type of alternative constructors
In-Reply-To: <572E76C4.3020907@stoneleaf.us>
References: <nglqqi$83q$1@ger.gmane.org> <572E76C4.3020907@stoneleaf.us>
Message-ID: <CAP7+vJJK8i=KcV8dHJmfF9tYbowP3sL_W2WA6Vn59wwg+qutsA@mail.gmail.com>

IMO bool is a special case because it's meant to be a final class, and the
implementation of int (which is in C and so can violate most rules) doesn't
respect that.

But in general I think the only reasonable approach is that a construction
class method should return an instance of the subclass; these class methods
have a signature that's constrained by their signature in the base class.

OTOH operators like __add__ cannot be expected to return an instance of the
subclass, because these typically construct an instance using
__new__/__init__, whose signatures are *not* constrained by the base class.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160508/52474383/attachment.html>

From steve at pearwood.info  Sun May  8 06:56:34 2016
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 8 May 2016 20:56:34 +1000
Subject: [Python-Dev] Tracker Etiquette
In-Reply-To: <572E750B.80706@stoneleaf.us>
References: <572E750B.80706@stoneleaf.us>
Message-ID: <20160508105633.GW12028@ando.pearwood.info>

On Sat, May 07, 2016 at 04:06:51PM -0700, Ethan Furman wrote:
> I just reviewed an issue on the tracker, and found that many, if not 
> all, of my comments made at the time I completely disagree with (both 
> now, and before -- I'm going to claim sleep deprivation).
> 
> Obviously I should post a new comment explaining my thoughts about the 
> issue, but what should I do with the old comments?  Is it acceptable to 
> delete them?

My preference would be not to delete them. Certainly don't delete them 
if others have explicitly replied to them. Even if you no longer agree 
with the conclusion, there may be something of value in the comments.

-- 
Steve

From ncoghlan at gmail.com  Sun May  8 07:49:20 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 8 May 2016 21:49:20 +1000
Subject: [Python-Dev] Return type of alternative constructors
In-Reply-To: <nglqqi$83q$1@ger.gmane.org>
References: <nglqqi$83q$1@ger.gmane.org>
Message-ID: <CADiSq7fJDFE6-LkH=bT+8CV9NpFxX9REPBxV0+GFAqFYJ0PrMw@mail.gmail.com>

On 8 May 2016 at 08:39, Serhiy Storchaka <storchaka at gmail.com> wrote:
> Should alternative constructor call __new__ and __init__ methods? Thay can
> change signature in derived class.

I think this is typically the way to go (although, depending on the
specific type, the unpickling related methods may be a more
appropriate way for the alternate constructor to populate the instance
state)

> Should it complain if __new__ or __init__ were overridden?

If there are alternate constructors that depend on either the
signature of __new__/__init__, unpickling support, or some other
mechanism for creating new instances, this should be mentioned in the
class documentation as a constraint on subclasses - if subclasses
don't want to meet the constraint, they'll need to override the
affected alternate constructors.

Cheers,
Nick.

P.S. The potential complexity of that is one of the reasons the design
philosophy of "prefer composition to inheritance" has emerged -
subclassing is a powerful tool, but it does mean you often end up
needing to care about more interactions between the subclass and the
base class than you really wanted to.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun May  8 07:58:02 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 8 May 2016 21:58:02 +1000
Subject: [Python-Dev] Tracker Etiquette
In-Reply-To: <20160508105633.GW12028@ando.pearwood.info>
References: <572E750B.80706@stoneleaf.us>
 <20160508105633.GW12028@ando.pearwood.info>
Message-ID: <CADiSq7d7CaDg=uB2fhfxD1-m3D-031Kn=diX4H2TLBFJjfRNwg@mail.gmail.com>

On 8 May 2016 at 20:56, Steven D'Aprano <steve at pearwood.info> wrote:
> On Sat, May 07, 2016 at 04:06:51PM -0700, Ethan Furman wrote:
>> I just reviewed an issue on the tracker, and found that many, if not
>> all, of my comments made at the time I completely disagree with (both
>> now, and before -- I'm going to claim sleep deprivation).
>>
>> Obviously I should post a new comment explaining my thoughts about the
>> issue, but what should I do with the old comments?  Is it acceptable to
>> delete them?
>
> My preference would be not to delete them. Certainly don't delete them
> if others have explicitly replied to them. Even if you no longer agree
> with the conclusion, there may be something of value in the comments.

I generally only delete mine if I've posted to the wrong issue (which
happens sometimes if I have multiple issues open and lose track of
which tab is which).

For other discussions, I figure realising I'd made a mistake on my own
isn't that different from someone else replying to point out the error
- I'm sure there a few tracker comments floating around from
Nick-of-several-hours-later suggesting that
Nick-of-several-hours-earlier probably shouldn't be commenting on
tracker issues when he's in dire need of sleep ;)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ethan at stoneleaf.us  Sun May  8 11:51:22 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 08 May 2016 08:51:22 -0700
Subject: [Python-Dev] Tracker Etiquette
In-Reply-To: <572E750B.80706@stoneleaf.us>
References: <572E750B.80706@stoneleaf.us>
Message-ID: <572F607A.4070904@stoneleaf.us>

On 05/07/2016 04:06 PM, Ethan Furman wrote:

> I just reviewed an issue on the tracker, and found that many, if not
> all, of my comments made at the time I completely disagree with (both
> now, and before -- I'm going to claim sleep deprivation).
>
> Obviously I should post a new comment explaining my thoughts about the
> issue, but what should I do with the old comments?  Is it acceptable to
> delete them?

Thank you all for your comments.  Old ones kept, new one posted.

--
~Ethan~




From ethan at stoneleaf.us  Sun May  8 16:10:05 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 08 May 2016 13:10:05 -0700
Subject: [Python-Dev] Minor change to Enum -- should it go into 3.5.2?
Message-ID: <572F9D1D.8090309@stoneleaf.us>

Currently, the Enum creation process ignores __dunder__ attributes, and 
blocks all _sunder_ attributes.

Because of this, the enum34 backport used __order__ instead of _order_ 
to provide a mechanism for ordering the enum members (which I never 
really liked).

However, I've been working on my aenum [1] package, which uses several 
other _sunder_ attributes (for python2 compatibility) so I enabled 
_order_ instead and promote that spelling in the docs.

Unlike the other _sunder_ attributes, _order_ has no meaningful affect 
in Python 3 so I'd like to change the stdlib Enum to allow it (and 
either ignore completely, or check it is the same as the definition order).

My question is:  Should I put this change in 3.5.2?

- Yes means 3.5.2+ will work with _order_, 3.4, 3.5.0, and 3.5.1
   will not;

- No means 3.4 and all of 3.5 will not.

--
~Ethan~


[1] https://pypi.python.org/pypi/aenum

From brett at python.org  Sun May  8 16:41:04 2016
From: brett at python.org (Brett Cannon)
Date: Sun, 08 May 2016 20:41:04 +0000
Subject: [Python-Dev] devinabox has moved to GitHub
Message-ID: <CAP1=2W7-+fNbeo57vLnMd4OptSEwvKLg+HgED96bPoxOyWNgYQ@mail.gmail.com>

https://github.com/python/devinabox

The single issue for devinabox has moved to its own issue tracker, so
there's no need to worry about those issues cluttering b.p.o in the future.
I have made the Python core team I created on GitHub last week have write
privileges and Nick and I as admins on the repository. I have also turned
on the CLA bot for the repository (FYI
https://github.com/python/the-knights-who-say-ni has that bot's code).

I've asked Georg, Antoine, and Benjamin to tell me what I need to do to
shut off -- either by making it read-only or just deleting --
hg.python.org/devinabox.

Now that the migration has seriously begun, the next repos will be the peps
and devguide (slightly more complicated thanks to needing to update
commands for building their online versions). There's also the benchmarks
repo, but that might not get migrated if we start from scratch (see the
speed@ ML about that).

As for cpython, I've been asked to talk about it at the language summit
where I will start a conversation about what the minimal feature set will
need to be to migrate. Once we have  settled on what has to be in place to
migrate the cpython repo then we can start working on that TODO list.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160508/526272d8/attachment.html>

From gvanrossum at gmail.com  Sun May  8 18:29:20 2016
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Sun, 8 May 2016 15:29:20 -0700
Subject: [Python-Dev] Minor change to Enum -- should it go into 3.5.2?
In-Reply-To: <572F9D1D.8090309@stoneleaf.us>
References: <572F9D1D.8090309@stoneleaf.us>
Message-ID: <CAP7+vJJf0_rkzSThykpaNOn6F0b+2tE4=SwyShaYzkRKCvDSxw@mail.gmail.com>

If enum were provisional it would be okay, but since it isn't, I think this
change can't go into 3.5.2. Think if this: could any code that works in
3.5.1 be broken by the change?

--Guido (mobile)
On May 8, 2016 1:11 PM, "Ethan Furman" <ethan at stoneleaf.us> wrote:

> Currently, the Enum creation process ignores __dunder__ attributes, and
> blocks all _sunder_ attributes.
>
> Because of this, the enum34 backport used __order__ instead of _order_ to
> provide a mechanism for ordering the enum members (which I never really
> liked).
>
> However, I've been working on my aenum [1] package, which uses several
> other _sunder_ attributes (for python2 compatibility) so I enabled _order_
> instead and promote that spelling in the docs.
>
> Unlike the other _sunder_ attributes, _order_ has no meaningful affect in
> Python 3 so I'd like to change the stdlib Enum to allow it (and either
> ignore completely, or check it is the same as the definition order).
>
> My question is:  Should I put this change in 3.5.2?
>
> - Yes means 3.5.2+ will work with _order_, 3.4, 3.5.0, and 3.5.1
>   will not;
>
> - No means 3.4 and all of 3.5 will not.
>
> --
> ~Ethan~
>
>
> [1] https://pypi.python.org/pypi/aenum
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160508/1d2c16a8/attachment.html>

From ethan at stoneleaf.us  Sun May  8 18:43:38 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 08 May 2016 15:43:38 -0700
Subject: [Python-Dev] Minor change to Enum -- should it go into 3.5.2?
In-Reply-To: <CAP7+vJJf0_rkzSThykpaNOn6F0b+2tE4=SwyShaYzkRKCvDSxw@mail.gmail.com>
References: <572F9D1D.8090309@stoneleaf.us>
 <CAP7+vJJf0_rkzSThykpaNOn6F0b+2tE4=SwyShaYzkRKCvDSxw@mail.gmail.com>
Message-ID: <572FC11A.9060104@stoneleaf.us>

On 05/08/2016 03:29 PM, Guido van Rossum wrote:

> If enum were provisional it would be okay, but since it isn't, I think
> this change can't go into 3.5.2. Think if this: could any code that
> works in 3.5.1 be broken by the change?

No, but with the change code that works in 3.5.2 could break in 3.5.1 or 
3.5.0.

It's a 2/3 compatibility issue with enum34 and aenum which support 
_order_, and Python3.4+ which does not.  The work-around is to use 
__order__ instead (or use enum34 or aenum instead ;) .

Either way, it's only similarity to a bug is I should have named it 
_order_ in the beginning, and put the compatibility shim into the stdlib 
version at the same time.

--
~Ethan~

From guido at python.org  Sun May  8 18:50:10 2016
From: guido at python.org (Guido van Rossum)
Date: Sun, 8 May 2016 15:50:10 -0700
Subject: [Python-Dev] Return type of alternative constructors
In-Reply-To: <CADiSq7fJDFE6-LkH=bT+8CV9NpFxX9REPBxV0+GFAqFYJ0PrMw@mail.gmail.com>
References: <nglqqi$83q$1@ger.gmane.org>
 <CADiSq7fJDFE6-LkH=bT+8CV9NpFxX9REPBxV0+GFAqFYJ0PrMw@mail.gmail.com>
Message-ID: <CAP7+vJ+RospP6xRRoROKETcmvxMe+5rRfQ_uZs-aWpKynKsGkw@mail.gmail.com>

On Sun, May 8, 2016 at 4:49 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 8 May 2016 at 08:39, Serhiy Storchaka <storchaka at gmail.com> wrote:
> > Should alternative constructor call __new__ and __init__ methods? Thay
> can
> > change signature in derived class.
>
> I think this is typically the way to go (although, depending on the
> specific type, the unpickling related methods may be a more
> appropriate way for the alternate constructor to populate the instance
> state)
>

Honestly, either of these sounds like fragile, even though I really want
the alternative constructor to return an instance of the subclass (else why
invoke it through the subclass).


> > Should it complain if __new__ or __init__ were overridden?
>
> If there are alternate constructors that depend on either the
> signature of __new__/__init__, unpickling support, or some other
> mechanism for creating new instances, this should be mentioned in the
> class documentation as a constraint on subclasses - if subclasses
> don't want to meet the constraint, they'll need to override the
> affected alternate constructors.
>

Putting this constraint in the docs sounds fragile too. :-(


> Cheers,
> Nick.
>
> P.S. The potential complexity of that is one of the reasons the design
> philosophy of "prefer composition to inheritance" has emerged -
> subclassing is a powerful tool, but it does mean you often end up
> needing to care about more interactions between the subclass and the
> base class than you really wanted to.
>

Indeed!

We could also consider this a general weakness of the "alternative
constructors are class methods" pattern. If instead these alternative
constructors were folded into the main constructor (e.g. via special
keyword args) it would be altogether clearer what a subclass should do.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160508/ad7d89f3/attachment.html>

From guido at python.org  Sun May  8 18:52:25 2016
From: guido at python.org (Guido van Rossum)
Date: Sun, 8 May 2016 15:52:25 -0700
Subject: [Python-Dev] Minor change to Enum -- should it go into 3.5.2?
In-Reply-To: <572FC11A.9060104@stoneleaf.us>
References: <572F9D1D.8090309@stoneleaf.us>
 <CAP7+vJJf0_rkzSThykpaNOn6F0b+2tE4=SwyShaYzkRKCvDSxw@mail.gmail.com>
 <572FC11A.9060104@stoneleaf.us>
Message-ID: <CAP7+vJJacRqQGO5OxN-dO1q5rSLOhSY9=fXpk6RxmohFq4xsaA@mail.gmail.com>

On Sun, May 8, 2016 at 3:43 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 05/08/2016 03:29 PM, Guido van Rossum wrote:
>
> If enum were provisional it would be okay, but since it isn't, I think
>> this change can't go into 3.5.2. Think if this: could any code that
>> works in 3.5.1 be broken by the change?
>>
>
> No, but with the change code that works in 3.5.2 could break in 3.5.1 or
> 3.5.0.
>

That's bad too (and it's one reason why we're generally strict about the
"no new features in bugfix releases" rule.


> It's a 2/3 compatibility issue with enum34 and aenum which support
> _order_, and Python3.4+ which does not.  The work-around is to use
> __order__ instead (or use enum34 or aenum instead ;) .
>
> Either way, it's only similarity to a bug is I should have named it
> _order_ in the beginning, and put the compatibility shim into the stdlib
> version at the same time.
>

I think it's a case of water under the bridge and learning to live with
your mistakes.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160508/0d96265d/attachment.html>

From ethan at stoneleaf.us  Sun May  8 19:39:13 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 08 May 2016 16:39:13 -0700
Subject: [Python-Dev] Minor change to Enum -- should it go into 3.5.2?
In-Reply-To: <CAP7+vJJacRqQGO5OxN-dO1q5rSLOhSY9=fXpk6RxmohFq4xsaA@mail.gmail.com>
References: <572F9D1D.8090309@stoneleaf.us>
 <CAP7+vJJf0_rkzSThykpaNOn6F0b+2tE4=SwyShaYzkRKCvDSxw@mail.gmail.com>
 <572FC11A.9060104@stoneleaf.us>
 <CAP7+vJJacRqQGO5OxN-dO1q5rSLOhSY9=fXpk6RxmohFq4xsaA@mail.gmail.com>
Message-ID: <572FCE21.9020309@stoneleaf.us>

On 05/08/2016 03:52 PM, Guido van Rossum wrote:

> I think it's a case of water under the bridge and learning to live with
> your mistakes.

Better than dying from them.

:)

--
~Ethan~



From greg.ewing at canterbury.ac.nz  Sun May  8 20:55:09 2016
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 09 May 2016 12:55:09 +1200
Subject: [Python-Dev] Return type of alternative constructors
In-Reply-To: <CAP7+vJ+RospP6xRRoROKETcmvxMe+5rRfQ_uZs-aWpKynKsGkw@mail.gmail.com>
References: <nglqqi$83q$1@ger.gmane.org>
 <CADiSq7fJDFE6-LkH=bT+8CV9NpFxX9REPBxV0+GFAqFYJ0PrMw@mail.gmail.com>
 <CAP7+vJ+RospP6xRRoROKETcmvxMe+5rRfQ_uZs-aWpKynKsGkw@mail.gmail.com>
Message-ID: <572FDFED.1000502@canterbury.ac.nz>

Guido van Rossum wrote:
> We could also consider this a general weakness of the "alternative 
> constructors are class methods" pattern. If instead these alternative 
> constructors were folded into the main constructor (e.g. via special 
> keyword args) it would be altogether clearer what a subclass should do.

A useful guideline might be that class methods can be provided
as sugar for alternative constructors, but they should all
funnel through the main constructor.

There's a convention like this in the Objective-C world where
one of an object's "init" methods is supposed to be documented
as the "designated initialiser", so that there is just one
thing that subclasses need to override.

-- 
Greg



From josephnunn at gmail.com  Sun May  8 21:49:11 2016
From: josephnunn at gmail.com (Joseph Lee Nunn III)
Date: Sun, 8 May 2016 18:49:11 -0700
Subject: [Python-Dev] Wrong OSX platform in sysconfig.py causing
 installation problems
In-Reply-To: <CALGmxEJaHqP5Ymd6SOzJgR4O+X3kBB9DJ_d8e_pu0+4ezNx0cw@mail.gmail.com>
References: <loom.20160506T111745-702@post.gmane.org>
 <905F2C35-855E-4733-9023-4B30EFF228B2@python.org>
 <F36B85F7-3553-4BC7-8847-6189B56736E4@gmail.com>
 <CALGmxE+hWRzeqhFwjYVSP1jBnSFJtbQjyvw_rYeVyT0tevZ8HA@mail.gmail.com>
 <CALGmxEJaHqP5Ymd6SOzJgR4O+X3kBB9DJ_d8e_pu0+4ezNx0cw@mail.gmail.com>
Message-ID: <EDA2D2CD-6571-4F7E-BBC4-A1D27444B570@gmail.com>

Thank you for looking into this for me, but since I fixed my problem by simply deleting the old SDKs I?ve moved on.

However let me clarify exactly what was going on one more time, just in case you guys decide to pursue further.

I had the old 10.6 SDK in /Developer/SDKs, it could have been ?special? in that I did alot of weird things for a former project and I just don?t recall.

For the sake of explanation though lets say it wasn?t, what then?

I installed OSX Python 3.5.1, created a virtual environment using the pyvenv command from that newly installed Python 3.5.1, and sourced its bin/activate.
Then when running pip install ZODB, the compiler would be passed the -isysroot flag for the 10.6 SDK, which would change its include paths and it would find no includes or libs (if you sym link the includes into the include directory of the virtual env you will get farther and see that the linker does not find libs either).  By suppling adding -v to CFLAGS you can have the compiler output the include paths and compiler call when running pip install.  That is how I noticed the alternate system root supplied by the isysroot flag.  Grepping around found me the _sysconfigdata.py file where that stuff was specified, it took a bit to realize that file came with the distribution and wasn?t generated on my system.

When not in a virtual environment, ZODB would install just fine into the system python (2.7).  

Not sure why that is, or what would make the virtual environment different, but there you go.

Joseph Nunn
josephnunn at gmail.com
joseph.nunn at uci.edu


> On May 07 , 2016, at 11:52 PM, Chris Barker <chris.barker at noaa.gov> wrote:
> 
> On Sat, May 7, 2016 at 10:59 PM, Chris Barker <chris.barker at noaa.gov <mailto:chris.barker at noaa.gov>> wrote:
> 
> .... Trying to decide if I want to go through the hassle of installing an old XCode to see if i can replicate the bug the OP is reporting....
> 
> OK, I found this gitHub repo with all the old SDKs:
> 
> https://github.com/phracker/MacOSX-SDKs <https://github.com/phracker/MacOSX-SDKs>
> 
> I put a copy of the 10.6 SDk in it's old location:
> 
> /Developer/SDKs/MacOSX10.6.sdk/
> 
> /before I did that, I did a pip install of zodb in a virtualenv, and all seemed to go fine. THe extensions rae linked against :
> 
> $ otool -L cPersistence.cpython-35m-darwin.so <http://cpersistence.cpython-35m-darwin.so/> 
> cPersistence.cpython-35m-darwin.so <http://cpersistence.cpython-35m-darwin.so/>:
>     /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
> 
> where that comes from -- who knows?? 
> 
> then I tried again, after putting the sdk in place:
> 
> [note: serious pain the $%^# to get pip to stop using it's cache! ]
> 
> and everything seemed to build fine, and at least one simple test of zodb works fine. And this is how the extensions are linked:
> 
> $  otool -L cPersistence.cpython-35m-darwin.so <http://cpersistence.cpython-35m-darwin.so/>
> cPersistence.cpython-35m-darwin.so <http://cpersistence.cpython-35m-darwin.so/>:
>     /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.11)
> 
> which is the same as above, but the "current version" is much older -- so does that mean it used the 10.6 SDK????
> 
> Anyway, I can't reproduce the OP's bug.....
> 
> -CHB
> 
> 
> 
> -- 
> 
> Christopher Barker, Ph.D.
> Oceanographer
> 
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
> 
> Chris.Barker at noaa.gov <mailto:Chris.Barker at noaa.gov>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160508/c3193a4d/attachment-0001.html>

From ncoghlan at gmail.com  Sun May  8 22:15:59 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 9 May 2016 12:15:59 +1000
Subject: [Python-Dev] Minor change to Enum -- should it go into 3.5.2?
In-Reply-To: <572FC11A.9060104@stoneleaf.us>
References: <572F9D1D.8090309@stoneleaf.us>
 <CAP7+vJJf0_rkzSThykpaNOn6F0b+2tE4=SwyShaYzkRKCvDSxw@mail.gmail.com>
 <572FC11A.9060104@stoneleaf.us>
Message-ID: <CADiSq7exkfO-XuLQyP7OR+qhcTmjFDYHAYRBrkZZ28uUzXkKbQ@mail.gmail.com>

On 9 May 2016 at 08:43, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 05/08/2016 03:29 PM, Guido van Rossum wrote:
>
>> If enum were provisional it would be okay, but since it isn't, I think
>> this change can't go into 3.5.2. Think if this: could any code that
>> works in 3.5.1 be broken by the change?
>
>
> No, but with the change code that works in 3.5.2 could break in 3.5.1 or
> 3.5.0.
>
> It's a 2/3 compatibility issue with enum34 and aenum which support _order_,
> and Python3.4+ which does not.  The work-around is to use __order__ instead
> (or use enum34 or aenum instead ;) .

Needing to use a PyPI alternative to a stdlib module for increased
cross-version consistency is a pretty common experience these days, so
I think that's OK - end users can choose for themselves between the
stability of the stdlib version and the reduced update latency of the
PyPI version.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From gvanrossum at gmail.com  Sun May  8 22:27:40 2016
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Sun, 8 May 2016 19:27:40 -0700
Subject: [Python-Dev] Minor change to Enum -- should it go into 3.5.2?
In-Reply-To: <CADiSq7exkfO-XuLQyP7OR+qhcTmjFDYHAYRBrkZZ28uUzXkKbQ@mail.gmail.com>
References: <572F9D1D.8090309@stoneleaf.us>
 <CAP7+vJJf0_rkzSThykpaNOn6F0b+2tE4=SwyShaYzkRKCvDSxw@mail.gmail.com>
 <572FC11A.9060104@stoneleaf.us>
 <CADiSq7exkfO-XuLQyP7OR+qhcTmjFDYHAYRBrkZZ28uUzXkKbQ@mail.gmail.com>
Message-ID: <CAP7+vJKWGqD7sqcUQDPbQjcb8SBZc23cpfPAQxrAm1AOveV4cw@mail.gmail.com>

I wonder if we should add something for this?

--Guido (mobile)
On May 8, 2016 7:18 PM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:

> On 9 May 2016 at 08:43, Ethan Furman <ethan at stoneleaf.us> wrote:
> > On 05/08/2016 03:29 PM, Guido van Rossum wrote:
> >
> >> If enum were provisional it would be okay, but since it isn't, I think
> >> this change can't go into 3.5.2. Think if this: could any code that
> >> works in 3.5.1 be broken by the change?
> >
> >
> > No, but with the change code that works in 3.5.2 could break in 3.5.1 or
> > 3.5.0.
> >
> > It's a 2/3 compatibility issue with enum34 and aenum which support
> _order_,
> > and Python3.4+ which does not.  The work-around is to use __order__
> instead
> > (or use enum34 or aenum instead ;) .
>
> Needing to use a PyPI alternative to a stdlib module for increased
> cross-version consistency is a pretty common experience these days, so
> I think that's OK - end users can choose for themselves between the
> stability of the stdlib version and the reduced update latency of the
> PyPI version.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160508/8e4aacd7/attachment.html>

From ethan at stoneleaf.us  Sun May  8 22:39:51 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 08 May 2016 19:39:51 -0700
Subject: [Python-Dev] Minor change to Enum -- should it go into 3.5.2?
In-Reply-To: <CADiSq7exkfO-XuLQyP7OR+qhcTmjFDYHAYRBrkZZ28uUzXkKbQ@mail.gmail.com>
References: <572F9D1D.8090309@stoneleaf.us>	<CAP7+vJJf0_rkzSThykpaNOn6F0b+2tE4=SwyShaYzkRKCvDSxw@mail.gmail.com>	<572FC11A.9060104@stoneleaf.us>
 <CADiSq7exkfO-XuLQyP7OR+qhcTmjFDYHAYRBrkZZ28uUzXkKbQ@mail.gmail.com>
Message-ID: <572FF877.7090308@stoneleaf.us>

On 05/08/2016 07:15 PM, Nick Coghlan wrote:
> On 9 May 2016 at 08:43, Ethan Furman wrote:

>> It's a 2/3 compatibility issue with enum34 and aenum which support _order_,
>> and Python3.4+ which does not.  The work-around is to use __order__ instead
>> (or use enum34 or aenum instead ;) .
>
> Needing to use a PyPI alternative to a stdlib module for increased
> cross-version consistency is a pretty common experience these days, so
> I think that's OK - end users can choose for themselves between the
> stability of the stdlib version and the reduced update latency of the
> PyPI version.

Are you saying I shouldn't bother updating the 3.6 Enum to ignore _order_?

--
~Ethan~


From ncoghlan at gmail.com  Sun May  8 22:52:30 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 9 May 2016 12:52:30 +1000
Subject: [Python-Dev] Return type of alternative constructors
In-Reply-To: <CAP7+vJ+RospP6xRRoROKETcmvxMe+5rRfQ_uZs-aWpKynKsGkw@mail.gmail.com>
References: <nglqqi$83q$1@ger.gmane.org>
 <CADiSq7fJDFE6-LkH=bT+8CV9NpFxX9REPBxV0+GFAqFYJ0PrMw@mail.gmail.com>
 <CAP7+vJ+RospP6xRRoROKETcmvxMe+5rRfQ_uZs-aWpKynKsGkw@mail.gmail.com>
Message-ID: <CADiSq7cXi-i0VOwdQD4dpSqDvs8h3yWmyq2ByYGFSYtyjrUrZw@mail.gmail.com>

On 9 May 2016 at 08:50, Guido van Rossum <guido at python.org> wrote:
> On Sun, May 8, 2016 at 4:49 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> P.S. The potential complexity of that is one of the reasons the design
>> philosophy of "prefer composition to inheritance" has emerged -
>> subclassing is a powerful tool, but it does mean you often end up
>> needing to care about more interactions between the subclass and the
>> base class than you really wanted to.
>
> Indeed!
>
> We could also consider this a general weakness of the "alternative
> constructors are class methods" pattern. If instead these alternative
> constructors were folded into the main constructor (e.g. via special keyword
> args) it would be altogether clearer what a subclass should do.

Unfortunately, even that approach gets tricky when the inheritance
relationship crosses the boundary between components with independent
release cycles.

In my experience, this timeline is the main one that causes the pain:

* Base class is released in Component A (e.g. CPython)
* Subclass is released in Component B (e.g. PyPI module)
* Component A releases a new base class construction feature

Question: does the new construction feature work with the existing
subclass in Component B if you combine it with the new version of
Component A?

When alternate constructors can be implemented as class methods that
work by creating a default instance and using existing public API
methods to mutate it, then the answer to that question is "yes", since
the default constructor hasn't changed, and the new convenience
constructor isn't relying on any other new features.

The answer is also "yes" for existing subclasses that only add new
behaviour without adding any new state, and hence just use the base
class __new__ and __init__ without overriding either of them.

It's when the existing subclasses overrides __new__ or __init__ and
one or both of the following is true that things can get tricky:

- you're working with an immutable type
- the API implementing the post-creation mutation is a new one

In both of those cases, the new construction feature of the base class
probably won't work right without updates to the affected subclass to
support the new capability (whether that's supporting a new parameter
in __new__ and __init__, or adding their own implementation of the new
alternate constructor).

I'm genuinely unsure that's a solvable problem in the general case -
it seems to be an inherent consequence of the coupling between
subclasses and base classes during instance construction, akin to the
challenges with subclass compatibility of the unpickling APIs when a
base class adds new state.

However, from a pragmatic perspective, the following approach seems to
work reasonably well:

* assume subclasses don't change the signature of __new__ or __init__
* note the assumptions about the default constructor signature in the
alternate constructor docs to let implementors of subclasses that
change the signature know they'll need to explicitly test
compatibility and perhaps provide their own implementation of the
alternate constructor

You *do* still end up with some cases where a subclass needs to be
upgraded before a new base class feature works properly for that
particular subclass, but subclasses that *don't* change the
constructor signature "just work".

Cheers,
Nick.

P.S. It occurs to me that a sufficiently sophisticated typechecker
might be able to look at all of the calls to "cls(*args, **kwds)" in
class methods and "type(self)(*args, **kwds)" in instance methods, and
use those to define a set of type constraints for the expected
constructor signatures in subclassses, even if the current code base
never actually invokes those code paths.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Mon May  9 06:21:06 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 9 May 2016 20:21:06 +1000
Subject: [Python-Dev] Minor change to Enum -- should it go into 3.5.2?
In-Reply-To: <572FF877.7090308@stoneleaf.us>
References: <572F9D1D.8090309@stoneleaf.us>
 <CAP7+vJJf0_rkzSThykpaNOn6F0b+2tE4=SwyShaYzkRKCvDSxw@mail.gmail.com>
 <572FC11A.9060104@stoneleaf.us>
 <CADiSq7exkfO-XuLQyP7OR+qhcTmjFDYHAYRBrkZZ28uUzXkKbQ@mail.gmail.com>
 <572FF877.7090308@stoneleaf.us>
Message-ID: <CADiSq7ep=WQ1hWnsNu0M0myGxfBcNDk=wCGEgkDn_nmcd1iKjg@mail.gmail.com>

On 9 May 2016 at 12:39, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 05/08/2016 07:15 PM, Nick Coghlan wrote:
>> Needing to use a PyPI alternative to a stdlib module for increased
>> cross-version consistency is a pretty common experience these days, so
>> I think that's OK - end users can choose for themselves between the
>> stability of the stdlib version and the reduced update latency of the
>> PyPI version.
>
> Are you saying I shouldn't bother updating the 3.6 Enum to ignore _order_?

I don't have an opinion on the change itself (although keeping PyPI
and 3.6-to-be aligned seems reasonable), just that it's OK to suggest
folks use a backport module even on earlier 3.x versions if they want
the behaviour of the latest (or even the next upcoming) stdlib
release, rather than the baseline behaviour of their current runtime.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From guido at python.org  Mon May  9 12:30:06 2016
From: guido at python.org (Guido van Rossum)
Date: Mon, 9 May 2016 09:30:06 -0700
Subject: [Python-Dev] Return type of alternative constructors
In-Reply-To: <CADiSq7cXi-i0VOwdQD4dpSqDvs8h3yWmyq2ByYGFSYtyjrUrZw@mail.gmail.com>
References: <nglqqi$83q$1@ger.gmane.org>
 <CADiSq7fJDFE6-LkH=bT+8CV9NpFxX9REPBxV0+GFAqFYJ0PrMw@mail.gmail.com>
 <CAP7+vJ+RospP6xRRoROKETcmvxMe+5rRfQ_uZs-aWpKynKsGkw@mail.gmail.com>
 <CADiSq7cXi-i0VOwdQD4dpSqDvs8h3yWmyq2ByYGFSYtyjrUrZw@mail.gmail.com>
Message-ID: <CAP7+vJJnw=wUd7oUC7N4ae3wMpjmP6tM5Wpcf23iyE31WZ5UzA@mail.gmail.com>

On Sun, May 8, 2016 at 7:52 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 9 May 2016 at 08:50, Guido van Rossum <guido at python.org> wrote:
> > On Sun, May 8, 2016 at 4:49 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >> P.S. The potential complexity of that is one of the reasons the design
> >> philosophy of "prefer composition to inheritance" has emerged -
> >> subclassing is a powerful tool, but it does mean you often end up
> >> needing to care about more interactions between the subclass and the
> >> base class than you really wanted to.
> >
> > Indeed!
> >
> > We could also consider this a general weakness of the "alternative
> > constructors are class methods" pattern. If instead these alternative
> > constructors were folded into the main constructor (e.g. via special
> keyword
> > args) it would be altogether clearer what a subclass should do.
>
> Unfortunately, even that approach gets tricky when the inheritance
> relationship crosses the boundary between components with independent
> release cycles.
>
> In my experience, this timeline is the main one that causes the pain:
>
> * Base class is released in Component A (e.g. CPython)
> * Subclass is released in Component B (e.g. PyPI module)
> * Component A releases a new base class construction feature
>
> Question: does the new construction feature work with the existing
> subclass in Component B if you combine it with the new version of
> Component A?
>
> When alternate constructors can be implemented as class methods that
> work by creating a default instance and using existing public API
> methods to mutate it, then the answer to that question is "yes", since
> the default constructor hasn't changed, and the new convenience
> constructor isn't relying on any other new features.
>
> The answer is also "yes" for existing subclasses that only add new
> behaviour without adding any new state, and hence just use the base
> class __new__ and __init__ without overriding either of them.
>
> It's when the existing subclasses overrides __new__ or __init__ and
> one or both of the following is true that things can get tricky:
>
> - you're working with an immutable type
> - the API implementing the post-creation mutation is a new one
>
> In both of those cases, the new construction feature of the base class
> probably won't work right without updates to the affected subclass to
> support the new capability (whether that's supporting a new parameter
> in __new__ and __init__, or adding their own implementation of the new
> alternate constructor).
>

OTOH it's not the end of the world -- until B is updated, you can't use the
new construction feature with subclass B, you have to use the old way of
constructing instances of B. Presumably that old way is still supported,
otherwise the change to A has just broken all of B, regardless of the new
construction feature. Which is possible, but it's a choice that A's author
has to make after careful deliberation.

Or maybe the "class construction" machinery in A is so prominent that it
really is part of the interface between A and any of its subclasses, and
then that API had better be documented. Just saying "you can subclass this"
won't be sufficient.


> I'm genuinely unsure that's a solvable problem in the general case -
> it seems to be an inherent consequence of the coupling between
> subclasses and base classes during instance construction, akin to the
> challenges with subclass compatibility of the unpickling APIs when a
> base class adds new state.
>

Yup. My summary of it is that versioning sucks.


> However, from a pragmatic perspective, the following approach seems to
> work reasonably well:
>
> * assume subclasses don't change the signature of __new__ or __init__
>

I still find that a distasteful choice, because *in general* there is no
requirement like that and there are good reasons why subclasses might have
a different __init__/__new__ signature. (For example dict and defaultdict.)


> * note the assumptions about the default constructor signature in the
> alternate constructor docs to let implementors of subclasses that
> change the signature know they'll need to explicitly test
> compatibility and perhaps provide their own implementation of the
> alternate constructor
>

Yup, you end up having to design the API for subclasses carefully and then
document it precisely.

This is what people too often forget when they complain e.g. "why can't I
subclass EventLoop more easily" -- we don't want to have a public API for
that, so we discourage it, but people mistakenly believe that anything
that's a class should be subclassable.


> You *do* still end up with some cases where a subclass needs to be
> upgraded before a new base class feature works properly for that
> particular subclass, but subclasses that *don't* change the
> constructor signature "just work".
>

The key is that there's an API requirement and that you have to design and
document that API with future evolution in mind. If you don't do and let
people write subclasses that just happen to work, you're in a lot of pain.
The interface between a base class and a subclass just is very complex so
must designers of base classes get this wrong initially.


> Cheers,
> Nick.
>
> P.S. It occurs to me that a sufficiently sophisticated typechecker
> might be able to look at all of the calls to "cls(*args, **kwds)" in
> class methods and "type(self)(*args, **kwds)" in instance methods, and
> use those to define a set of type constraints for the expected
> constructor signatures in subclassses, even if the current code base
> never actually invokes those code paths.
>

Could you restate that as a concrete code example? (Examples of the
problems with "construction features" would also be helpful, probably --
abstract descriptions of problems often lead me astray.)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160509/cea598ca/attachment.html>

From chris.barker at noaa.gov  Mon May  9 14:03:31 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 9 May 2016 11:03:31 -0700
Subject: [Python-Dev] Wrong OSX platform in sysconfig.py causing
 installation problems
In-Reply-To: <EDA2D2CD-6571-4F7E-BBC4-A1D27444B570@gmail.com>
References: <loom.20160506T111745-702@post.gmane.org>
 <905F2C35-855E-4733-9023-4B30EFF228B2@python.org>
 <F36B85F7-3553-4BC7-8847-6189B56736E4@gmail.com>
 <CALGmxE+hWRzeqhFwjYVSP1jBnSFJtbQjyvw_rYeVyT0tevZ8HA@mail.gmail.com>
 <CALGmxEJaHqP5Ymd6SOzJgR4O+X3kBB9DJ_d8e_pu0+4ezNx0cw@mail.gmail.com>
 <EDA2D2CD-6571-4F7E-BBC4-A1D27444B570@gmail.com>
Message-ID: <CALGmxEKOpKC2L8RzZ7uiwWKYOmT32NVgsi=mOFXw1871s54T6w@mail.gmail.com>

Thanks Joseph,

I say this isn't worth pursuing until someone else reports the issue, as I
couldn't replicate it with a "clean" system.

I had the old 10.6 SDK in /Developer/SDKs, it could have been ?special? in
> that I did alot of weird things for a former project and I just don?t
> recall.
>

I suspect your 10.6 SDK was cnfigured oddly, or ????

For the sake of explanation though lets say it wasn?t, what then?
>
> I installed OSX Python 3.5.1, created a virtual environment using the
> pyvenv command from that newly installed Python 3.5.1, and sourced its
> bin/activate.
> Then when running pip install ZODB, the compiler would be passed the
> -isysroot flag for the 10.6 SDK, which would change its include paths and
> it would find no includes or libs
>

this is the odd part -- shouldn't the new sysroot (i.e. the SDK) have all
the includes and libs  needed?

Not sure why that is, or what would make the virtual environment different,
> but there you go.
>

yeah, me neither ....

Anyway, unless someone else sees this, I think we're done.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160509/ff85031c/attachment.html>

From josephnunn at gmail.com  Mon May  9 15:08:10 2016
From: josephnunn at gmail.com (Joseph Lee Nunn III)
Date: Mon, 9 May 2016 12:08:10 -0700
Subject: [Python-Dev] Wrong OSX platform in sysconfig.py causing
 installation problems
In-Reply-To: <CALGmxEKOpKC2L8RzZ7uiwWKYOmT32NVgsi=mOFXw1871s54T6w@mail.gmail.com>
References: <loom.20160506T111745-702@post.gmane.org>
 <905F2C35-855E-4733-9023-4B30EFF228B2@python.org>
 <F36B85F7-3553-4BC7-8847-6189B56736E4@gmail.com>
 <CALGmxE+hWRzeqhFwjYVSP1jBnSFJtbQjyvw_rYeVyT0tevZ8HA@mail.gmail.com>
 <CALGmxEJaHqP5Ymd6SOzJgR4O+X3kBB9DJ_d8e_pu0+4ezNx0cw@mail.gmail.com>
 <EDA2D2CD-6571-4F7E-BBC4-A1D27444B570@gmail.com>
 <CALGmxEKOpKC2L8RzZ7uiwWKYOmT32NVgsi=mOFXw1871s54T6w@mail.gmail.com>
Message-ID: <66EF7E49-D14C-42CD-B68A-10AC63566213@gmail.com>


> On May 09 , 2016, at 11:03 AM, Chris Barker <chris.barker at noaa.gov> wrote:
> 
> I installed OSX Python 3.5.1, created a virtual environment using the pyvenv command from that newly installed Python 3.5.1, and sourced its bin/activate.
> Then when running pip install ZODB, the compiler would be passed the -isysroot flag for the 10.6 SDK, which would change its include paths and it would find no includes or libs 
> 
> this is the odd part -- shouldn't the new sysroot (i.e. the SDK) have all the includes and libs  needed?
> 

That is the weird part, if installing into the system python the new sysroot worked just fine.  It was only when trying to install into a virtual environment that it failed.

What the difference is I just don?t know.

Thanks again for you efforts!

Joseph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160509/8cbefd81/attachment.html>

From lukasz at langa.pl  Mon May  9 22:13:48 2016
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Mon, 9 May 2016 19:13:48 -0700
Subject: [Python-Dev] PyGC_Collect ignores state of `enabled`
Message-ID: <DDA40D7F-F070-42A0-AFFF-B03828894A68@langa.pl>

Is this deliberate? Could we change it for 3.6 so PyGC_Collect does take `enabled` into account?

Context: we?re experimenting with disabling the gc entirely for some workloads because it doesn?t fare too well with Copy-on-Write on Linux. What ends up happening is that disabling the gc actually drops memory usage significantly if you?re using hundreds of the same processes on one box :-)

However, because of PyGC_Collect() called in Py_Finalize(), during interpreter shutdown the collection is done anyway, Linux does CoW and the memory usage spikes. Which is ironic on process shutdown.

--
Lukasz Langa | Facebook
Production Engineer | The Ministry of Silly Walks
(+1) 650-681-7811

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160509/ad72912f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160509/ad72912f/attachment.sig>

From benjamin at python.org  Tue May 10 01:55:51 2016
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 09 May 2016 22:55:51 -0700
Subject: [Python-Dev] PyGC_Collect ignores state of `enabled`
In-Reply-To: <DDA40D7F-F070-42A0-AFFF-B03828894A68@langa.pl>
References: <DDA40D7F-F070-42A0-AFFF-B03828894A68@langa.pl>
Message-ID: <1462859751.2026529.603130785.5D4338A1@webmail.messagingengine.com>

Adding PyGC_CollectIfEnabled() and calling it in Py_Finalize is probably
fine. I don't think the contract of PyGC_Collect itself (or gc.collect()
for that matter) should be changed. You might want to disable GC but
invoke it yourself.

On Mon, May 9, 2016, at 19:13, ?ukasz Langa wrote:
> Is this deliberate? Could we change it for 3.6 so PyGC_Collect does take
> `enabled` into account?
> 
> Context: we?re experimenting with disabling the gc entirely for some
> workloads because it doesn?t fare too well with Copy-on-Write on Linux.
> What ends up happening is that disabling the gc actually drops memory
> usage significantly if you?re using hundreds of the same processes on one
> box :-)
> 
> However, because of PyGC_Collect() called in Py_Finalize(), during
> interpreter shutdown the collection is done anyway, Linux does CoW and
> the memory usage spikes. Which is ironic on process shutdown.
> 
> --
> Lukasz Langa | Facebook
> Production Engineer | The Ministry of Silly Walks
> (+1) 650-681-7811
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
> Email had 1 attachment:
> + signature.asc
>   1k (application/pgp-signature)

From vadmium+py at gmail.com  Tue May 10 04:17:58 2016
From: vadmium+py at gmail.com (Martin Panter)
Date: Tue, 10 May 2016 08:17:58 +0000
Subject: [Python-Dev] ctypes compatibility with 2.3
Message-ID: <CA+eR4cF8--D_eSCo1+oNd=ZKQ46QFyLmF4bYqxni9CMs7H9wDQ@mail.gmail.com>

I am working on <https://bugs.python.org/issue22636>, to fix shell
injection problems with ctypes.util.find_library(). The proposal for
Python 3 is to change os.popen(shell-script) calls to use
subprocess.Popen().

However the Python 2.7 version of the module has a comment which says
?This file should be kept compatible with Python 2.3, see PEP 291.?
Looking at <https://www.python.org/dev/peps/pep-0291/>, it is not
clear why we have to maintain this compatibility. My best guess is
that there may be an external ctypes package that people want(ed) to
keep compatible with 2.3, and also keep synchronized with 2.7.

I would like to lift this restriction to at least 2.4, because that is
when the ?subprocess? module was added. I notice that there is already
code that relies on the list.sort(key=...) feature, which was added in
2.4. Ideally I would prefer to drop the restriction and only require
2.7 compatibility. Would either of these options be a problem?

If it is a problem, I think it is still possible to avoid the shell by
passing a sequence of program arguments to os.popen(). But I prefer
not to do this, because the function is marked as deprecated, and the
code would be substantially different to Python 3.

-Martin

From ncoghlan at gmail.com  Tue May 10 09:21:57 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 10 May 2016 23:21:57 +1000
Subject: [Python-Dev] Return type of alternative constructors
In-Reply-To: <CAP7+vJJnw=wUd7oUC7N4ae3wMpjmP6tM5Wpcf23iyE31WZ5UzA@mail.gmail.com>
References: <nglqqi$83q$1@ger.gmane.org>
 <CADiSq7fJDFE6-LkH=bT+8CV9NpFxX9REPBxV0+GFAqFYJ0PrMw@mail.gmail.com>
 <CAP7+vJ+RospP6xRRoROKETcmvxMe+5rRfQ_uZs-aWpKynKsGkw@mail.gmail.com>
 <CADiSq7cXi-i0VOwdQD4dpSqDvs8h3yWmyq2ByYGFSYtyjrUrZw@mail.gmail.com>
 <CAP7+vJJnw=wUd7oUC7N4ae3wMpjmP6tM5Wpcf23iyE31WZ5UzA@mail.gmail.com>
Message-ID: <CADiSq7cBmvJu-+Amv4ztohRoZqFXLmb=upYsT2PVbwtwGNRt3Q@mail.gmail.com>

On 10 May 2016 at 02:30, Guido van Rossum <guido at python.org> wrote:
> On Sun, May 8, 2016 at 7:52 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> P.S. It occurs to me that a sufficiently sophisticated typechecker
>> might be able to look at all of the calls to "cls(*args, **kwds)" in
>> class methods and "type(self)(*args, **kwds)" in instance methods, and
>> use those to define a set of type constraints for the expected
>> constructor signatures in subclassses, even if the current code base
>> never actually invokes those code paths.
>
> Could you restate that as a concrete code example? (Examples of the problems
> with "construction features" would also be helpful, probably -- abstract
> descriptions of problems often lead me astray.)

Rectangle/Square is a classic example of the constructor signature
changing, so I'll try to use that to illustrate the point with a
"displaced_copy" alternate constructor:

    class Rectangle:
        def __init__(self, top_left_point, width, height):
            self.top_left_point = top_left_point
            self.width = width
            self.height = height

        @classmethod
        def displaced_copy(cls, other_rectangle, offset):
            """Create a new instance from an existing one"""
            return cls(other.top_left_point + offset, other.width, other.height)

    class Square:
        def __init__(self, top_left_point, side_length):
            super().__init__(top_left_point, side_length, side_length)

At this point, a typechecker *could* have enough info to know that
"Square.displaced_copy(some_rectangle, offset)" is necessarily going
to fail, even if nothing in the application actually *calls*
Square.displaced_copy.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From brett at python.org  Tue May 10 13:39:44 2016
From: brett at python.org (Brett Cannon)
Date: Tue, 10 May 2016 17:39:44 +0000
Subject: [Python-Dev] ctypes compatibility with 2.3
In-Reply-To: <CA+eR4cF8--D_eSCo1+oNd=ZKQ46QFyLmF4bYqxni9CMs7H9wDQ@mail.gmail.com>
References: <CA+eR4cF8--D_eSCo1+oNd=ZKQ46QFyLmF4bYqxni9CMs7H9wDQ@mail.gmail.com>
Message-ID: <CAP1=2W4YStjMo3r2918k8C4=F7DK99UFW21BdpyHb7pOQWTZWA@mail.gmail.com>

On Tue, 10 May 2016 at 01:18 Martin Panter <vadmium+py at gmail.com> wrote:

> I am working on <https://bugs.python.org/issue22636>, to fix shell
> injection problems with ctypes.util.find_library(). The proposal for
> Python 3 is to change os.popen(shell-script) calls to use
> subprocess.Popen().
>
> However the Python 2.7 version of the module has a comment which says
> ?This file should be kept compatible with Python 2.3, see PEP 291.?
> Looking at <https://www.python.org/dev/peps/pep-0291/>, it is not
> clear why we have to maintain this compatibility. My best guess is
> that there may be an external ctypes package that people want(ed) to
> keep compatible with 2.3, and also keep synchronized with 2.7.
>

That's correct and the maintainer is/was Thomas Heller who I have cc'ed to
see if he's okay with lifting the restriction.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160510/3695f414/attachment.html>

From guido at python.org  Tue May 10 14:02:13 2016
From: guido at python.org (Guido van Rossum)
Date: Tue, 10 May 2016 11:02:13 -0700
Subject: [Python-Dev] Return type of alternative constructors
In-Reply-To: <CADiSq7cBmvJu-+Amv4ztohRoZqFXLmb=upYsT2PVbwtwGNRt3Q@mail.gmail.com>
References: <nglqqi$83q$1@ger.gmane.org>
 <CADiSq7fJDFE6-LkH=bT+8CV9NpFxX9REPBxV0+GFAqFYJ0PrMw@mail.gmail.com>
 <CAP7+vJ+RospP6xRRoROKETcmvxMe+5rRfQ_uZs-aWpKynKsGkw@mail.gmail.com>
 <CADiSq7cXi-i0VOwdQD4dpSqDvs8h3yWmyq2ByYGFSYtyjrUrZw@mail.gmail.com>
 <CAP7+vJJnw=wUd7oUC7N4ae3wMpjmP6tM5Wpcf23iyE31WZ5UzA@mail.gmail.com>
 <CADiSq7cBmvJu-+Amv4ztohRoZqFXLmb=upYsT2PVbwtwGNRt3Q@mail.gmail.com>
Message-ID: <CAP7+vJJX_KT290+-BMtyE1ZA-WbUD7NQSwXX0+pVtL6jrf3ZNQ@mail.gmail.com>

On Tue, May 10, 2016 at 6:21 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 10 May 2016 at 02:30, Guido van Rossum <guido at python.org> wrote:
> > On Sun, May 8, 2016 at 7:52 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >> P.S. It occurs to me that a sufficiently sophisticated typechecker
> >> might be able to look at all of the calls to "cls(*args, **kwds)" in
> >> class methods and "type(self)(*args, **kwds)" in instance methods, and
> >> use those to define a set of type constraints for the expected
> >> constructor signatures in subclassses, even if the current code base
> >> never actually invokes those code paths.
> >
> > Could you restate that as a concrete code example? (Examples of the
> problems
> > with "construction features" would also be helpful, probably -- abstract
> > descriptions of problems often lead me astray.)
>
> Rectangle/Square is a classic example of the constructor signature
> changing, so I'll try to use that to illustrate the point with a
> "displaced_copy" alternate constructor:
>
>     class Rectangle:
>         def __init__(self, top_left_point, width, height):
>             self.top_left_point = top_left_point
>             self.width = width
>             self.height = height
>
>         @classmethod
>         def displaced_copy(cls, other_rectangle, offset):
>             """Create a new instance from an existing one"""
>             return cls(other.top_left_point + offset, other.width,
> other.height)
>

(But why is it a class method? I guess the example could also use an
instance method and it would still have the same properties relevant for
this discussion.)


>     class Square:
>         def __init__(self, top_left_point, side_length):
>             super().__init__(top_left_point, side_length, side_length)
>
> At this point, a typechecker *could* have enough info to know that
> "Square.displaced_copy(some_rectangle, offset)" is necessarily going
> to fail, even if nothing in the application actually *calls*
> Square.displaced_copy.
>

The question remains of course whether the type checker should flag Square
to be an invalid subclass or merely as not implementing displaced_copy().

Anyway, at this point I believe we're just violently agreeing, so no need
for another response. Though Serhiy may be unhappy with the lack of
guidance he's received...

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160510/0c27b9bd/attachment.html>

From theller at ctypes.org  Wed May 11 07:35:23 2016
From: theller at ctypes.org (Thomas Heller)
Date: Wed, 11 May 2016 13:35:23 +0200
Subject: [Python-Dev] ctypes compatibility with 2.3
In-Reply-To: <CAP1=2W4YStjMo3r2918k8C4=F7DK99UFW21BdpyHb7pOQWTZWA@mail.gmail.com>
References: <CA+eR4cF8--D_eSCo1+oNd=ZKQ46QFyLmF4bYqxni9CMs7H9wDQ@mail.gmail.com>
 <CAP1=2W4YStjMo3r2918k8C4=F7DK99UFW21BdpyHb7pOQWTZWA@mail.gmail.com>
Message-ID: <813cec64-397a-2ca1-1f53-6bed0886c80b@ctypes.org>

Am 10.05.2016 um 19:39 schrieb Brett Cannon:
>
>
> On Tue, 10 May 2016 at 01:18 Martin Panter <vadmium+py at gmail.com
> <mailto:vadmium%2Bpy at gmail.com>> wrote:
>
>     I am working on <https://bugs.python.org/issue22636>, to fix shell
>     injection problems with ctypes.util.find_library(). The proposal for
>     Python 3 is to change os.popen(shell-script) calls to use
>     subprocess.Popen().
>
>     However the Python 2.7 version of the module has a comment which says
>     ?This file should be kept compatible with Python 2.3, see PEP 291.?
>     Looking at <https://www.python.org/dev/peps/pep-0291/>, it is not
>     clear why we have to maintain this compatibility. My best guess is
>     that there may be an external ctypes package that people want(ed) to
>     keep compatible with 2.3, and also keep synchronized with 2.7.
>
>
> That's correct and the maintainer is/was Thomas Heller who I have cc'ed
> to see if he's okay with lifting the restriction.

For me it is totally ok to lift this restriction.

Thomas


From brett at python.org  Wed May 11 12:04:27 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 11 May 2016 16:04:27 +0000
Subject: [Python-Dev] ctypes compatibility with 2.3
In-Reply-To: <813cec64-397a-2ca1-1f53-6bed0886c80b@ctypes.org>
References: <CA+eR4cF8--D_eSCo1+oNd=ZKQ46QFyLmF4bYqxni9CMs7H9wDQ@mail.gmail.com>
 <CAP1=2W4YStjMo3r2918k8C4=F7DK99UFW21BdpyHb7pOQWTZWA@mail.gmail.com>
 <813cec64-397a-2ca1-1f53-6bed0886c80b@ctypes.org>
Message-ID: <CAP1=2W6Qq96BiAEOKLi-x4qvJX88S1-Eo5C1fr+yvC0-7P0O7A@mail.gmail.com>

On Wed, 11 May 2016 at 04:35 Thomas Heller <theller at ctypes.org> wrote:

> Am 10.05.2016 um 19:39 schrieb Brett Cannon:
> >
> >
> > On Tue, 10 May 2016 at 01:18 Martin Panter <vadmium+py at gmail.com
> > <mailto:vadmium%2Bpy at gmail.com>> wrote:
> >
> >     I am working on <https://bugs.python.org/issue22636>, to fix shell
> >     injection problems with ctypes.util.find_library(). The proposal for
> >     Python 3 is to change os.popen(shell-script) calls to use
> >     subprocess.Popen().
> >
> >     However the Python 2.7 version of the module has a comment which says
> >     ?This file should be kept compatible with Python 2.3, see PEP 291.?
> >     Looking at <https://www.python.org/dev/peps/pep-0291/>, it is not
> >     clear why we have to maintain this compatibility. My best guess is
> >     that there may be an external ctypes package that people want(ed) to
> >     keep compatible with 2.3, and also keep synchronized with 2.7.
> >
> >
> > That's correct and the maintainer is/was Thomas Heller who I have cc'ed
> > to see if he's okay with lifting the restriction.
>
> For me it is totally ok to lift this restriction.
>

Great! I'll also update PEP 291.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160511/344fbb67/attachment.html>

From theller at ctypes.org  Wed May 11 12:07:30 2016
From: theller at ctypes.org (Thomas Heller)
Date: Wed, 11 May 2016 18:07:30 +0200
Subject: [Python-Dev] ctypes compatibility with 2.3
In-Reply-To: <CAP1=2W6Qq96BiAEOKLi-x4qvJX88S1-Eo5C1fr+yvC0-7P0O7A@mail.gmail.com>
References: <CA+eR4cF8--D_eSCo1+oNd=ZKQ46QFyLmF4bYqxni9CMs7H9wDQ@mail.gmail.com>
 <CAP1=2W4YStjMo3r2918k8C4=F7DK99UFW21BdpyHb7pOQWTZWA@mail.gmail.com>
 <813cec64-397a-2ca1-1f53-6bed0886c80b@ctypes.org>
 <CAP1=2W6Qq96BiAEOKLi-x4qvJX88S1-Eo5C1fr+yvC0-7P0O7A@mail.gmail.com>
Message-ID: <3283bb48-531a-0795-babd-df203115df58@ctypes.org>

Am 11.05.2016 um 18:04 schrieb Brett Cannon:
>
>
> On Wed, 11 May 2016 at 04:35 Thomas Heller <theller at ctypes.org
> <mailto:theller at ctypes.org>> wrote:
>
>     Am 10.05.2016 um 19:39 schrieb Brett Cannon:
>     >
>     >
>     > On Tue, 10 May 2016 at 01:18 Martin Panter <vadmium+py at gmail.com
>     <mailto:vadmium%2Bpy at gmail.com>
>     > <mailto:vadmium%2Bpy at gmail.com <mailto:vadmium%252Bpy at gmail.com>>>
>     wrote:
>     >
>     >     I am working on <https://bugs.python.org/issue22636>, to fix shell
>     >     injection problems with ctypes.util.find_library(). The
>     proposal for
>     >     Python 3 is to change os.popen(shell-script) calls to use
>     >     subprocess.Popen().
>     >
>     >     However the Python 2.7 version of the module has a comment
>     which says
>     >     ?This file should be kept compatible with Python 2.3, see PEP
>     291.?
>     >     Looking at <https://www.python.org/dev/peps/pep-0291/>, it is not
>     >     clear why we have to maintain this compatibility. My best guess is
>     >     that there may be an external ctypes package that people
>     want(ed) to
>     >     keep compatible with 2.3, and also keep synchronized with 2.7.
>     >
>     >
>     > That's correct and the maintainer is/was Thomas Heller who I have
>     cc'ed
>     > to see if he's okay with lifting the restriction.
>
>     For me it is totally ok to lift this restriction.
>
>
> Great! I'll also update PEP 291.

Cool.  While you're at it, the compatibility restriction for 
modulefinder could also be lifted.

From meadori at gmail.com  Wed May 11 12:19:09 2016
From: meadori at gmail.com (Meador Inge)
Date: Wed, 11 May 2016 11:19:09 -0500
Subject: [Python-Dev] ctypes compatibility with 2.3
In-Reply-To: <3283bb48-531a-0795-babd-df203115df58@ctypes.org>
References: <CA+eR4cF8--D_eSCo1+oNd=ZKQ46QFyLmF4bYqxni9CMs7H9wDQ@mail.gmail.com>
 <CAP1=2W4YStjMo3r2918k8C4=F7DK99UFW21BdpyHb7pOQWTZWA@mail.gmail.com>
 <813cec64-397a-2ca1-1f53-6bed0886c80b@ctypes.org>
 <CAP1=2W6Qq96BiAEOKLi-x4qvJX88S1-Eo5C1fr+yvC0-7P0O7A@mail.gmail.com>
 <3283bb48-531a-0795-babd-df203115df58@ctypes.org>
Message-ID: <CAK1QoorVOvdbW2ffZtsbLvPqKepsy4inPPqa-m=CK0pC-NZLHA@mail.gmail.com>

On Wed, May 11, 2016 at 11:07 AM, Thomas Heller <theller at ctypes.org> wrote:

Cool.  While you're at it, the compatibility restriction for modulefinder
> could also be lifted.


+1

The question of modulefinder actually came up recently*:

http://bugs.python.org/issue26881

-- Meador

* Posting here for reference.  Thomas already knows this as he is on the
issue26881 watch list :-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160511/49983333/attachment.html>

From brett at python.org  Wed May 11 12:28:47 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 11 May 2016 16:28:47 +0000
Subject: [Python-Dev] ctypes compatibility with 2.3
In-Reply-To: <3283bb48-531a-0795-babd-df203115df58@ctypes.org>
References: <CA+eR4cF8--D_eSCo1+oNd=ZKQ46QFyLmF4bYqxni9CMs7H9wDQ@mail.gmail.com>
 <CAP1=2W4YStjMo3r2918k8C4=F7DK99UFW21BdpyHb7pOQWTZWA@mail.gmail.com>
 <813cec64-397a-2ca1-1f53-6bed0886c80b@ctypes.org>
 <CAP1=2W6Qq96BiAEOKLi-x4qvJX88S1-Eo5C1fr+yvC0-7P0O7A@mail.gmail.com>
 <3283bb48-531a-0795-babd-df203115df58@ctypes.org>
Message-ID: <CAP1=2W6dE8bRKe9q+SToX-+cT1qF90Og7f565_=o4vP3nLYsXQ@mail.gmail.com>

On Wed, 11 May 2016 at 09:07 Thomas Heller <theller at ctypes.org> wrote:

> Am 11.05.2016 um 18:04 schrieb Brett Cannon:
> >
> >
> > On Wed, 11 May 2016 at 04:35 Thomas Heller <theller at ctypes.org
> > <mailto:theller at ctypes.org>> wrote:
> >
> >     Am 10.05.2016 um 19:39 schrieb Brett Cannon:
> >     >
> >     >
> >     > On Tue, 10 May 2016 at 01:18 Martin Panter <vadmium+py at gmail.com
> >     <mailto:vadmium%2Bpy at gmail.com>
> >     > <mailto:vadmium%2Bpy at gmail.com <mailto:vadmium%252Bpy at gmail.com>>>
> >     wrote:
> >     >
> >     >     I am working on <https://bugs.python.org/issue22636>, to fix
> shell
> >     >     injection problems with ctypes.util.find_library(). The
> >     proposal for
> >     >     Python 3 is to change os.popen(shell-script) calls to use
> >     >     subprocess.Popen().
> >     >
> >     >     However the Python 2.7 version of the module has a comment
> >     which says
> >     >     ?This file should be kept compatible with Python 2.3, see PEP
> >     291.?
> >     >     Looking at <https://www.python.org/dev/peps/pep-0291/>, it is
> not
> >     >     clear why we have to maintain this compatibility. My best
> guess is
> >     >     that there may be an external ctypes package that people
> >     want(ed) to
> >     >     keep compatible with 2.3, and also keep synchronized with 2.7.
> >     >
> >     >
> >     > That's correct and the maintainer is/was Thomas Heller who I have
> >     cc'ed
> >     > to see if he's okay with lifting the restriction.
> >
> >     For me it is totally ok to lift this restriction.
> >
> >
> > Great! I'll also update PEP 291.
>
> Cool.  While you're at it, the compatibility restriction for
> modulefinder could also be lifted.
>

Will do.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160511/f9115b42/attachment.html>

From brett at python.org  Wed May 11 12:34:10 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 11 May 2016 16:34:10 +0000
Subject: [Python-Dev] PEP 515: Underscores in Numeric Literals (revision
 3)
In-Reply-To: <CAP7+vJ+F=d9ycxZhaWkoYB6SYXf82T=Lf4QxWubwNy_FKJUACw@mail.gmail.com>
References: <n9mqlh$oeb$1@ger.gmane.org>
 <CAP1=2W4P7skRpewpgxXBgHHicngMU3mJ2V1s3NVXgQL9PiCJmw@mail.gmail.com>
 <CAP7+vJLUumJiOzExqdmffJG_ywpFA9vXAWUH2Bpzvpiwcwp_RQ@mail.gmail.com>
 <ncisha$auc$1@ger.gmane.org>
 <CADiSq7etYj9Wy4++kP0z_5=-x6Pm0E9=6VASh+nYfJk8bM6PfQ@mail.gmail.com>
 <CAP7+vJ+e60gid=R8gjhmyiiNkCMC0Db4zt-Bct_qKUN29Vbr_Q@mail.gmail.com>
 <loom.20160319T181628-51@post.gmane.org>
 <CAP7+vJ+PewZ=ojQU0LzsZh0n0payOgzdOzfx6zQ73Xh9OYt5aQ@mail.gmail.com>
 <loom.20160319T191115-310@post.gmane.org>
 <CAP7+vJ+F=d9ycxZhaWkoYB6SYXf82T=Lf4QxWubwNy_FKJUACw@mail.gmail.com>
Message-ID: <CAP1=2W4SM-_SBUo5YGdwyvq2N62+FC3TEgLnmYMW=2VDkV6+2A@mail.gmail.com>

Is there anything holding up PEP 515 at this point in terms of acceptance
or implementation?

On Sat, 19 Mar 2016 at 11:56 Guido van Rossum <guido at python.org> wrote:

> All that sounds fine!
>
> On Sat, Mar 19, 2016 at 11:28 AM, Stefan Krah <stefan at bytereef.org> wrote:
> > Guido van Rossum <guido <at> python.org> writes:
> >> So should the preprocessing step just be s.replace('_', ''), or should
> >> it reject underscores that don't follow the rules from the PEP
> >> (perhaps augmented so they follow the spirit of the PEP and the letter
> >> of the IBM spec)?
> >>
> >> Honestly I think it's also fine if specifying this exactly is left out
> >> of the PEP, and handled by whoever adds this to Decimal. Having a PEP
> >> to work from for the language spec and core builtins (int(), float()
> >> complex()) is more important.
> >
> > I'd keep it simple for Decimal: Remove left and right whitespace (we're
> > already doing this), then remove underscores from the remaining string
> > (which must not contain any further whitespace), then use the IBM
> grammar.
> >
> >
> > We could add a clause to the PEP that only those strings that follow
> > the spirit of the PEP are guaranteed to be accepted in the future.
> >
> >
> > One reason for keeping it simple is that I would not like to slow down
> > string conversion, but thinking about two grammars is also a problem --
> > part of the string conversion in libmpdec is modeled in ACL2, which
> > would be invalidated or at least complicated with two grammars.
> >
> >
> >
> > Stefan Krah
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160511/a27adc64/attachment.html>

From brett at python.org  Wed May 11 12:43:24 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 11 May 2016 16:43:24 +0000
Subject: [Python-Dev] file system path protocol PEP
Message-ID: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>

**deep, calming breath**

Here is the PEP for __fspath__(). The draft lives at
https://github.com/brettcannon/path-pep so feel free to send me PRs for
spelling mistakes, grammatical errors, etc.

-----

PEP: NNN
Title: Adding a file system path protocol
Version: $Revision$
Last-Modified: $Date$
Author: Brett Cannon <brett at python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 11-May-2016
Post-History: 11-May-2016


Abstract
========

This PEP proposes a protocol for classes which represent a file system
path to be able to provide a ``str`` or ``bytes`` representation.
Changes to Python's standard library are also proposed to utilize this
protocol where appropriate to facilitate the use of path objects where
historically only ``str`` and/or ``bytes`` file system paths are
accepted. The goal is to allow users to use the representation of a
file system path that's easiest for them now as they migrate towards
using path objects in the future.


Rationale
=========

Historically in Python, file system paths have been represented as
strings or bytes. This choice of representation has stemmed from C's
own decision to represent file system paths as
``const char *`` [#libc-open]_. While that is a totally serviceable
format to use for file system paths, it's not necessarily optimal. At
issue is the fact that while all file system paths can be represented
as strings or bytes, not all strings or bytes represent a file system
path. This can lead to issues where any e.g. string duck-types to a
file system path whether it actually represents a path or not.

To help elevate the representation of file system paths from their
representation as strings and bytes to a more appropriate object
representation, the pathlib module [#pathlib]_ was provisionally
introduced in Python 3.4 through PEP 428. While considered by some as
an improvement over strings and bytes for file system paths, it has
suffered from a lack of adoption. Typically the key issue listed
for the low adoption rate has been the lack of support in the standard
library. This lack of support required users of pathlib to manually
convert path objects to strings by calling ``str(path)`` which many
found error-prone.

One issue in converting path objects to strings comes from
the fact that only generic way to get a string representation of the
path was to pass the object to ``str()``. This can pose a
problem when done blindly as nearly all Python objects have some
string representation whether they are a path or not, e.g.
``str(None)`` will give a result that
``builtins.open()`` [#builtins-open]_ will happily use to create a new
file.

Exacerbating this whole situation is the
``DirEntry`` object [#os-direntry]_. While path objects have a
representation that can be extracted using ``str()``, ``DirEntry``
objects expose a ``path`` attribute instead. Having no common
interface between path objects, ``DirEntry``, and any other
third-party path library had become an issue. A solution that allowed
any path-representing object to declare that is was a path and a way
to extract a low-level representation that all path objects could
support was desired.

This PEP then proposes to introduce a new protocol to be followed by
objects which represent file system paths. Providing a protocol allows
for clear signalling of what objects represent file system paths as
well as a way to extract a lower-level representation that can be used
with older APIs which only support strings or bytes.

Discussions regarding path objects that led to this PEP can be found
in multiple threads on the python-ideas mailing list archive
[#python-ideas-archive]_ for the months of March and April 2016 and on
the python-dev mailing list archives [#python-dev-archive]_ during
April 2016.


Proposal
========

This proposal is split into two parts. One part is the proposal of a
protocol for objects to declare and provide support for exposing a
file system path representation. The other part is changes to Python's
standard library to support the new protocol. These changes will also
have the pathlib module drop its provisional status.


Protocol
--------

The following abstract base class defines the protocol for an object
to be considered a path object::

    import abc
    import typing as t


    class PathLike(abc.ABC):

        """Abstract base class for implementing the file system path
protocol."""

        @abc.abstractmethod
        def __fspath__(self) -> t.Union[str, bytes]:
            """Return the file system path representation of the object."""
            raise NotImplementedError


Objects representing file system paths will implement the
``__fspath__()`` method which will return the ``str`` or ``bytes``
representation of the path. The ``str`` representation is the
preferred low-level path representation as it is human-readable and
what people historically represent paths as.


Standard library changes
------------------------

It is expected that most APIs in Python's standard library that
currently accept a file system path will be updated appropriately to
accept path objects (whether that requires code or simply an update
to documentation will vary). The modules mentioned below, though,
deserve specific details as they have either fundamental changes that
empower the ability to use path objects, or entail additions/removal
of APIs.


builtins
''''''''

``open()`` [#builtins-open]_ will be updated to accept path objects as
well as continue to accept ``str`` and ``bytes``.


os
'''

The ``fspath()`` function will be added with the following semantics
(although this is still an open issue:
`Should os.fspath() return bytes?`_)::

    import typing as t


    def fspath(path: t.Union[PathLike, str]) -> str:
        """Return the string representation of the path.

        If a string is passed in then it is returned unchanged.
        """
        if hasattr(path, '__fspath__'):
            path = path.__fspath__()
        if not isinstance(path, str):
            type_name = type(path).__name__
            raise TypeError("expected a str or path object, not " +
type_name)
        return path

The ``os.fsencode()`` [#os-fsencode]_ and
``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept
path objects. As both functions coerce their arguments to
``bytes`` and ``str``, respectively, they will be updated to call
``__fspath__()`` if present to convert the path object to a ``str`` or
``bytes`` representation, and then perform their appropriate
coercion operations as if the return value from ``__fspath__()`` had
been the original argument to the coercion function in question.

The addition of ``os.fspath()``, the updates to
``os.fsencode()``/``os.fsdecode()``, and the current semantics of
``pathlib.PurePath`` provide the semantics necessary to
get the path representation one prefers. For a path object,
``pathlib.PurePath``/``Path`` can be used. If ``str`` is desired and
a ``bytes`` object is considered an error, then ``os.fspath()`` can be
used. If a ``str`` is desired and the encoding of ``bytes`` should be
assumed to be the default file system encoding, then ``os.fsdecode()``
should be used. Finally, if a ``bytes`` representation is desired and
any strings should be encoded using the default file system encoding
then ``os.fsencode()`` is used.

This PEP recommends using path objects when possible and falling back
to string paths as necessary. Therefore, no function is provided for
the case of wanting a bytes representation but without any automatic
encoding to help discourage the use of multiple bytes encodings on a
single file system. If it is necessary to deal with an existing file
system directory with entries in a non-default encoding, this can be
done with low-level functions using ``str`` and the PEP 383
``surrogateescape`` error handler, or by using ``bytes`` directly.

Another way to view this is as a hierarchy of file system path
representations (highest- to lowest-level): path -> str -> bytes. The
functions and classes under discussion can all accept objects on the
same level of the hierarchy, but they vary in whether they promote or
demote objects to another level. The ``pathlib.PurePath`` class can
promote a ``str`` to a path object. The ``os.fspath()`` function can
demote a path object to a string, but only if ``__fspath__()`` returns
a string. The ``os.fsdecode()`` function will demote a path object to
a string or promote a ``bytes`` object to a ``str``. The
``os.fsencode()`` function will demote a path or string object to
``bytes``. There is no function that provides a way to demote a path
object directly to ``bytes`` and not allow demoting strings.

The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()``
method. It will return the same value as currently found on the
``path`` attribute of ``DirEntry`` instances.


os.path
'''''''

The various path-manipulation functions of ``os.path`` [#os-path]_
will be updated to accept path objects. For polymorphic functions that
accept both bytes and strings, they will be updated to simply use
code very much similar to
``path.__fspath__() if  hasattr(path, '__fspath__') else path``. This
will allow for their pre-existing type-checking code to continue to
function.

During the discussions leading up to this PEP it was suggested that
``os.path`` not be updated using an "explicit is better than implicit"
argument. The thinking was that since ``__fspath__()`` is polymorphic
itself it may be better to have code working with ``os.path`` extract
the path representation from path objects explicitly. There is also
the consideration that adding support this deep into the low-level OS
APIs will lead to code magically supporting path objects without
requiring any documentation updated, leading to potential complaints
when it doesn't work, unbeknownst to the project author.

But it is the view of the authors that "practicality beats purity" in
this instance. To help facilitate the transition to supporting path
objects, it is better to make the transition as easy as possible than
to worry about unexpected/undocumented duck typing support for
projects.


pathlib
'''''''

The ``PathLike`` ABC as discussed in the Protocol_ section will be
added to the pathlib module [#pathlib]_. The constructor for
``pathlib.PurePath`` and ``pathlib.Path`` will be updated to accept
path objects. Both ``PurePath`` and ``Path`` will continue to not
accept ``bytes`` path representations, and so if ``__fspath__()``
returns ``bytes`` it will raise an exception.

The ``path`` attribute will be removed as this PEP makes its
redundant (it has not been included in any released version of Python
and so is not a backwards-compatibility concern).


C API
'''''

The C API will gain an equivalent function to ``os.fspath()`` that
also allows bytes objects through::

    /*
        Return the file system path of the object.

        If the object is str or bytes, then allow it to pass through with
        an incremented refcount. All other types raise a TypeError.
    */
    PyObject *
    PyOS_RawFSPath(PyObject *path)
    {
        if (PyObject_HasAttrString(path, "__fspath__")) {
            path = PyObject_CallMethodObjArgs(path, "__fspath__", NULL);
            if (path == NULL) {
                return NULL;
            }
        }
        else {
            Py_INCREF(path);
        }

        if (!PyUnicode_Check(path) && !PyBytes_Check(path)) {
            Py_DECREF(path);
            return PyErr_Format(PyExc_TypeError,
                                "expected a string, bytes, or path object,
not %S",
                                path->ob_type);
        }

        return path;
}


Backwards compatibility
=======================

There are no explicit backwards-compatibility concerns. Unless an
object incidentally already defines a ``__fspath__()`` method there is
not reason to expect pre-existing code to break or expect to have
their semantics implicitly changed.

Libraries wishing to support path objects and a version of Python
prior to Python 3.6 can use the idiom of
``path.__fspath__() if hasattr(path, '__fspath__') else path``.


Open Issues
===========

Should os.fspath() return bytes?
--------------------------------

Some have argued that ``os.fspath()`` should be configurable so that
the user can specify what types are acceptable (e.g. an argument to
say that bytes are acceptable instead of strings, or both types).
Others have suggested that ``os.fspath()`` match the proposed
semantics of ``PyOS_RawFSPath()``. Both camps argue that use of
``os.fspath()`` will only be for a transitionary period while more
libraries gain acceptance of path objects, and so being more flexible
in what ``os.fspath()`` works with will help with the transition. The
opponents to this -- which support the currently proposed sematnics --
worry that being so flexible with accepting bytes will lead to people
not properly considering the ramifications of working with bytes,
especially if bytes are transparently appearing in their code due to
``os.fspath()``.


The name and location of the protocol's ABC
-------------------------------------------

The name of the ABC being proposed to represent the protocol has not
been discussed very much, not which module it should exist in.
Names other than ``PathLike`` which are viable are ``PathABC``
and ``FSPathABC``. The name can't be ``Path`` if the ABC is put into
the pathlib module.


Type hint for path-like objects
-------------------------------

Creating a proper type hint for  APIs that accept path objects as well
as strings and bytes will probably be needed. It could be as simple
as defining ``typing.Path``/``typing.FSPath`` to correspond to the ABC
and then having
``typing.PathLike = typing.Union[typing.Path, str, bytes]``. The type
hint could also potentially be made to be generic to accept the
specific low-level representation, e.g. ``typing.PathLike[str]``.

In the end the type hinting solution should be properly discussed
with the right type hinting experts if this is the best approach.


Rejected Ideas
==============

Other names for the protocol's function
---------------------------------------

Various names were proposed during discussions leading to this PEP,
including ``__path__``, ``__pathname__``, and ``__fspathname__``. In
the end people seemed to gravitate towards ``__fspath__`` for being
unambiguous without unnecessarily long.


Separate str/bytes methods
--------------------------

At one point it was suggested that ``__fspath__()`` only return
strings and another method named ``__fspathb__()`` be introduced to
return bytes. The thinking that by making ``__fspath__()`` not be
polymorphic it could make dealing with the potential string or bytes
representations easier. But the general consensus was that returning
bytes will more than likely be rare and that the various functions in
the os module are the better abstraction to be promoting over direct
calls to ``__fspath__()``.


Providing a path attribute
--------------------------

To help deal with the issue of ``pathlib.PurePath`` not inheriting
from ``str``, originally it was proposed to introduce a ``path``
attribute to mirror what ``os.DirEntry`` provides. In the end, though,
it was determined that a protocol would provide the same result while
not directly exposing an API that most people will never need to
interact with directly.


Have ``__fspath__()`` only return strings
------------------------------------------

Much of the discussion that led to this PEP revolved around whether
``__fspath__()`` should be polymorphic and return ``bytes`` as well as
``str`` instead of only ``str``. The general sentiment for this view
was that ``bytes`` are difficult to work with due to their
inherent lack of information about their encoding, and PEP 383 makes
it possible to represent all file system paths using ``str`` with the
``surrogateescape`` handler. Thus it would be better to forcibly
promote the use of ``str`` as the low-level path representation for
high-level path objects.

In the end it was decided that using ``bytes`` to represent paths is
simply not going to go away and thus they should be supported to some
degree. For those not wanting the hassle of working with ``bytes``,
``os.fspath()`` is provided.


A generic string encoding mechanism
-----------------------------------

At one point there was discussion of developing a generic mechanism to
extract a string representation of an object that had semantic meaning
(``__str__()`` does not necessarily return anything of semantic
significance beyond what may be helpful for debugging). In the end it
was deemed to lack a motivating need beyond the one this PEP is
trying to solve in a specific fashion.


Acknowledgements
================

Thanks to everyone who participated in the various discussions related
to this PEP that spanned both python-ideas and python-dev. Special
thanks to Koos Zevenhoven and Stephen Turnbull for direct feedback on
early drafts of this PEP.


References
==========

.. [#python-ideas-archive] The python-ideas mailing list archive
   (https://mail.python.org/pipermail/python-ideas/)

.. [#python-dev-archive] The python-dev mailing list archive
   (https://mail.python.org/pipermail/python-dev/)

.. [#libc-open] ``open()`` documention for the C standard library
   (
http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html
)

.. [#pathlib] The ``pathlib`` module
   (https://docs.python.org/3/library/pathlib.html#module-pathlib)

.. [#builtins-open] The ``builtins.open()`` function
   (https://docs.python.org/3/library/functions.html#open)

.. [#os-fsencode] The ``os.fsencode()`` function
   (https://docs.python.org/3/library/os.html#os.fsencode)

.. [#os-fsdecode] The ``os.fsdecode()`` function
   (https://docs.python.org/3/library/os.html#os.fsdecode)

.. [#os-direntry] The ``os.DirEntry`` class
   (https://docs.python.org/3/library/os.html#os.DirEntry)

.. [#os-path] The ``os.path`` module
   (https://docs.python.org/3/library/os.path.html#module-os.path)


Copyright
=========

This document has been placed in the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160511/0a957ff0/attachment-0001.html>

From guido at python.org  Wed May 11 12:46:52 2016
From: guido at python.org (Guido van Rossum)
Date: Wed, 11 May 2016 09:46:52 -0700
Subject: [Python-Dev] PEP 515: Underscores in Numeric Literals (revision
 3)
In-Reply-To: <CAP1=2W4SM-_SBUo5YGdwyvq2N62+FC3TEgLnmYMW=2VDkV6+2A@mail.gmail.com>
References: <n9mqlh$oeb$1@ger.gmane.org>
 <CAP1=2W4P7skRpewpgxXBgHHicngMU3mJ2V1s3NVXgQL9PiCJmw@mail.gmail.com>
 <CAP7+vJLUumJiOzExqdmffJG_ywpFA9vXAWUH2Bpzvpiwcwp_RQ@mail.gmail.com>
 <ncisha$auc$1@ger.gmane.org>
 <CADiSq7etYj9Wy4++kP0z_5=-x6Pm0E9=6VASh+nYfJk8bM6PfQ@mail.gmail.com>
 <CAP7+vJ+e60gid=R8gjhmyiiNkCMC0Db4zt-Bct_qKUN29Vbr_Q@mail.gmail.com>
 <loom.20160319T181628-51@post.gmane.org>
 <CAP7+vJ+PewZ=ojQU0LzsZh0n0payOgzdOzfx6zQ73Xh9OYt5aQ@mail.gmail.com>
 <loom.20160319T191115-310@post.gmane.org>
 <CAP7+vJ+F=d9ycxZhaWkoYB6SYXf82T=Lf4QxWubwNy_FKJUACw@mail.gmail.com>
 <CAP1=2W4SM-_SBUo5YGdwyvq2N62+FC3TEgLnmYMW=2VDkV6+2A@mail.gmail.com>
Message-ID: <CAP7+vJ+BicSaZiizSpjE8q2g=OGHMLgOre1tLwo3o5DN2crZ3g@mail.gmail.com>

If the authors are happy I'll accept it right away.

(I vaguely recall there's another PEP that's ready for pronouncement -- but
which one?)

On Wed, May 11, 2016 at 9:34 AM, Brett Cannon <brett at python.org> wrote:

> Is there anything holding up PEP 515 at this point in terms of acceptance
> or implementation?
>
> On Sat, 19 Mar 2016 at 11:56 Guido van Rossum <guido at python.org> wrote:
>
>> All that sounds fine!
>>
>> On Sat, Mar 19, 2016 at 11:28 AM, Stefan Krah <stefan at bytereef.org>
>> wrote:
>> > Guido van Rossum <guido <at> python.org> writes:
>> >> So should the preprocessing step just be s.replace('_', ''), or should
>> >> it reject underscores that don't follow the rules from the PEP
>> >> (perhaps augmented so they follow the spirit of the PEP and the letter
>> >> of the IBM spec)?
>> >>
>> >> Honestly I think it's also fine if specifying this exactly is left out
>> >> of the PEP, and handled by whoever adds this to Decimal. Having a PEP
>> >> to work from for the language spec and core builtins (int(), float()
>> >> complex()) is more important.
>> >
>> > I'd keep it simple for Decimal: Remove left and right whitespace (we're
>> > already doing this), then remove underscores from the remaining string
>> > (which must not contain any further whitespace), then use the IBM
>> grammar.
>> >
>> >
>> > We could add a clause to the PEP that only those strings that follow
>> > the spirit of the PEP are guaranteed to be accepted in the future.
>> >
>> >
>> > One reason for keeping it simple is that I would not like to slow down
>> > string conversion, but thinking about two grammars is also a problem --
>> > part of the string conversion in libmpdec is modeled in ACL2, which
>> > would be invalidated or at least complicated with two grammars.
>> >
>> >
>> >
>> > Stefan Krah
>> >
>> > _______________________________________________
>> > Python-Dev mailing list
>> > Python-Dev at python.org
>> > https://mail.python.org/mailman/listinfo/python-dev
>> > Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160511/a922ea1b/attachment.html>

From brett at python.org  Wed May 11 13:28:51 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 11 May 2016 17:28:51 +0000
Subject: [Python-Dev] PEP 515: Underscores in Numeric Literals (revision
 3)
In-Reply-To: <CAP7+vJ+BicSaZiizSpjE8q2g=OGHMLgOre1tLwo3o5DN2crZ3g@mail.gmail.com>
References: <n9mqlh$oeb$1@ger.gmane.org>
 <CAP1=2W4P7skRpewpgxXBgHHicngMU3mJ2V1s3NVXgQL9PiCJmw@mail.gmail.com>
 <CAP7+vJLUumJiOzExqdmffJG_ywpFA9vXAWUH2Bpzvpiwcwp_RQ@mail.gmail.com>
 <ncisha$auc$1@ger.gmane.org>
 <CADiSq7etYj9Wy4++kP0z_5=-x6Pm0E9=6VASh+nYfJk8bM6PfQ@mail.gmail.com>
 <CAP7+vJ+e60gid=R8gjhmyiiNkCMC0Db4zt-Bct_qKUN29Vbr_Q@mail.gmail.com>
 <loom.20160319T181628-51@post.gmane.org>
 <CAP7+vJ+PewZ=ojQU0LzsZh0n0payOgzdOzfx6zQ73Xh9OYt5aQ@mail.gmail.com>
 <loom.20160319T191115-310@post.gmane.org>
 <CAP7+vJ+F=d9ycxZhaWkoYB6SYXf82T=Lf4QxWubwNy_FKJUACw@mail.gmail.com>
 <CAP1=2W4SM-_SBUo5YGdwyvq2N62+FC3TEgLnmYMW=2VDkV6+2A@mail.gmail.com>
 <CAP7+vJ+BicSaZiizSpjE8q2g=OGHMLgOre1tLwo3o5DN2crZ3g@mail.gmail.com>
Message-ID: <CAP1=2W7EZSyyT-tT-NbUyKaYSvBamzwk8Vh-gY6nYee14LZZyw@mail.gmail.com>

On Wed, 11 May 2016 at 09:47 Guido van Rossum <guido at python.org> wrote:

> If the authors are happy I'll accept it right away.
>
> (I vaguely recall there's another PEP that's ready for pronouncement --
> but which one?)
>

PEP 509 is the only one I can think of.

-Brett


>
> On Wed, May 11, 2016 at 9:34 AM, Brett Cannon <brett at python.org> wrote:
>
>> Is there anything holding up PEP 515 at this point in terms of acceptance
>> or implementation?
>>
>> On Sat, 19 Mar 2016 at 11:56 Guido van Rossum <guido at python.org> wrote:
>>
>>> All that sounds fine!
>>>
>>> On Sat, Mar 19, 2016 at 11:28 AM, Stefan Krah <stefan at bytereef.org>
>>> wrote:
>>> > Guido van Rossum <guido <at> python.org> writes:
>>> >> So should the preprocessing step just be s.replace('_', ''), or should
>>> >> it reject underscores that don't follow the rules from the PEP
>>> >> (perhaps augmented so they follow the spirit of the PEP and the letter
>>> >> of the IBM spec)?
>>> >>
>>> >> Honestly I think it's also fine if specifying this exactly is left out
>>> >> of the PEP, and handled by whoever adds this to Decimal. Having a PEP
>>> >> to work from for the language spec and core builtins (int(), float()
>>> >> complex()) is more important.
>>> >
>>> > I'd keep it simple for Decimal: Remove left and right whitespace (we're
>>> > already doing this), then remove underscores from the remaining string
>>> > (which must not contain any further whitespace), then use the IBM
>>> grammar.
>>> >
>>> >
>>> > We could add a clause to the PEP that only those strings that follow
>>> > the spirit of the PEP are guaranteed to be accepted in the future.
>>> >
>>> >
>>> > One reason for keeping it simple is that I would not like to slow down
>>> > string conversion, but thinking about two grammars is also a problem --
>>> > part of the string conversion in libmpdec is modeled in ACL2, which
>>> > would be invalidated or at least complicated with two grammars.
>>> >
>>> >
>>> >
>>> > Stefan Krah
>>> >
>>> > _______________________________________________
>>> > Python-Dev mailing list
>>> > Python-Dev at python.org
>>> > https://mail.python.org/mailman/listinfo/python-dev
>>> > Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>>
>>>
>>>
>>> --
>>> --Guido van Rossum (python.org/~guido)
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160511/6d1d6de9/attachment.html>

From k7hoven at gmail.com  Wed May 11 13:36:46 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Wed, 11 May 2016 20:36:46 +0300
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
Message-ID: <CAMiohoh7oBo7FOJaru2_HMLUt0XkWSm_2E1LddYJ8wWPHQKtww@mail.gmail.com>

**another deep, calming breath**

On Wed, May 11, 2016 at 7:43 PM, Brett Cannon <brett at python.org> wrote:
> Open Issues
> ===========
>
> Should os.fspath() return bytes?
> --------------------------------
>

In most cases, it of course should not. The section (or the title) do
not represent my view on the topic, but bytes paths are not going away
any time soon, so this requires considerations.

Below is a copy paste of my comment in the discussion on my pull
request ( https://github.com/brettcannon/path-pep/pull/2 ). It is
about whether os.fspath should look like

def fspath(path, *, type_constraint=str): ...

This was already discussed before, receiving positive reactions. It
would be an extension to what is in the draft Brett just posted. Note
that the return type, however, does not depend on `type_constraint`.
It fully depends on `path`. The constraint only determines when an
exception is raised instead of returning a value.  When using
str-based paths normally, one would do

os.fspath(path)

Calling it with

os.fspath(path, type_constraint=(str, bytes))

would turn off rejection of bytes paths (and be consistent with the
rest of `os` and `os.path` where functions accept both types). But the
default value for the keyword argument would be a good reminder that
str-based paths should be used by default. This str-constraint is also
present in the current drafted version, but it is not optional or
visible in the signature.

(See the diffs of my pull request for an example implementation of this.)

So below is the copy-paste from the pull request discussion that I promised:

"""
Nobody wants to promote using `bytes` for paths without a proper
reason. However, `os.fspath(path, type_contraint=str)` is already a
compromise from the current `os.*` convention (of symmetrically
supporting both `str` and `bytes`) towards enforcing `str`-based
paths.

As you know, the reason for using `os.fspath` is to switch to a
lower-level representation of paths as strings. When you use it, you
are already deciding that you are lower-level and want to examine or
manipulate path strings manually. I think the `type_constraint=str`
keyword-only argument (with the default) is a good way to remind the
'user' that `str` is the way to go unless you know what you are doing
and, to `bytes` users, that `os.fspath` (by default) rejects bytes.

Looking at the discussions on python-dev, one notices that the active
people in the discussions were mostly in favor of exposing the
bytes-supporting version in one way or the other. Some were even
against bytes-rejecting versions.

Most people (Python programmers), especially in the long term, should
not be using `os.fspath` a lot. If they want to print a path, they
should simply do so: `print(path_obj)`, or `print(f"The path is:
{path_obj}")`. However, there will always be people that for whatever
reason want to convert a pathlib object into a string, and without
even consulting the docs, they may write `str(path_obj)`. And that's
not the end of the world. It's not any worse than `str(integer)` for
instance. After all, if the main point is to convert something into a
generic string, the function should probably be called `str`, not
`fspath`. But if the idea is to convert any path object into something
that `os` and similar code internally understand, then `os.fspath`
sounds fine to me. So I would not go as far as 'never use
`str(path)`'. What we don't want is things like `open(str(path),
'w')`. I know you agree, because you wrote that yourself. But once
this protocol is available, I see no reason why people would
voluntarily wrap their path objects in `str(...)`, so I don't see that
as a problem in the future (thanks to this PEP). So, especially in the
long term, I don't expect `fspath` to be an everyday tool for people.
Even in shorter term, people don't usually need it, because the stdlib
will already support pathlib objects.

Since it is not an everyday tool, I think it's ok to have that extra
keyword-only argument. Even if it were an everyday tool, I can see the
keyword-only argument in the signature as useful thing in a
documentation sense.

[As a side note, I could imagine someone convincing me that `fspath`
should *always* accept both `str`- and `bytes`-based paths, because
that would simplify it and align it with the rest of `os`. This would
not be any worse than the status quo in terms of early failures. That
would, however, still not detect accidental use of bytes paths (when
forgotten to decode from the encoding it was received in).]
"""

-- Koos

From guido at python.org  Wed May 11 13:49:10 2016
From: guido at python.org (Guido van Rossum)
Date: Wed, 11 May 2016 10:49:10 -0700
Subject: [Python-Dev] PEP 515: Underscores in Numeric Literals (revision
 3)
In-Reply-To: <CAP1=2W7EZSyyT-tT-NbUyKaYSvBamzwk8Vh-gY6nYee14LZZyw@mail.gmail.com>
References: <n9mqlh$oeb$1@ger.gmane.org>
 <CAP1=2W4P7skRpewpgxXBgHHicngMU3mJ2V1s3NVXgQL9PiCJmw@mail.gmail.com>
 <CAP7+vJLUumJiOzExqdmffJG_ywpFA9vXAWUH2Bpzvpiwcwp_RQ@mail.gmail.com>
 <ncisha$auc$1@ger.gmane.org>
 <CADiSq7etYj9Wy4++kP0z_5=-x6Pm0E9=6VASh+nYfJk8bM6PfQ@mail.gmail.com>
 <CAP7+vJ+e60gid=R8gjhmyiiNkCMC0Db4zt-Bct_qKUN29Vbr_Q@mail.gmail.com>
 <loom.20160319T181628-51@post.gmane.org>
 <CAP7+vJ+PewZ=ojQU0LzsZh0n0payOgzdOzfx6zQ73Xh9OYt5aQ@mail.gmail.com>
 <loom.20160319T191115-310@post.gmane.org>
 <CAP7+vJ+F=d9ycxZhaWkoYB6SYXf82T=Lf4QxWubwNy_FKJUACw@mail.gmail.com>
 <CAP1=2W4SM-_SBUo5YGdwyvq2N62+FC3TEgLnmYMW=2VDkV6+2A@mail.gmail.com>
 <CAP7+vJ+BicSaZiizSpjE8q2g=OGHMLgOre1tLwo3o5DN2crZ3g@mail.gmail.com>
 <CAP1=2W7EZSyyT-tT-NbUyKaYSvBamzwk8Vh-gY6nYee14LZZyw@mail.gmail.com>
Message-ID: <CAP7+vJKLzX4d4R4=Z5nMLaHK1iQ4N8bddbd2dWnUm9zw0K7jVQ@mail.gmail.com>

On Wed, May 11, 2016 at 10:28 AM, Brett Cannon <brett at python.org> wrote:

>
>
> On Wed, 11 May 2016 at 09:47 Guido van Rossum <guido at python.org> wrote:
>
>> If the authors are happy I'll accept it right away.
>>
>> (I vaguely recall there's another PEP that's ready for pronouncement --
>> but which one?)
>>
>
> PEP 509 is the only one I can think of.
>

That's in limbo pending conclusive proof (through benchmarks) that at least
one of Yury's patches that depends on it makes a big enough difference.

Which IIUC itself is in limbo pending the wordcode changes (we're doing
that right?).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160511/a70a4561/attachment.html>

From dimaqq at gmail.com  Wed May 11 13:56:07 2016
From: dimaqq at gmail.com (Dima Tisnek)
Date: Wed, 11 May 2016 19:56:07 +0200
Subject: [Python-Dev] Slow downloads from python.org
Message-ID: <CAGGBzXJ80M=Coq_zig6w_rBqzX82JWch-sBUBm9CH+Fx6Vok7A@mail.gmail.com>

Sorry, this is probably wrong place to ask, but is it only me?
I can't get more than 40KB/s downloading from python.org

From brett at python.org  Wed May 11 14:16:27 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 11 May 2016 18:16:27 +0000
Subject: [Python-Dev] PEP 515: Underscores in Numeric Literals (revision
 3)
In-Reply-To: <CAP7+vJKLzX4d4R4=Z5nMLaHK1iQ4N8bddbd2dWnUm9zw0K7jVQ@mail.gmail.com>
References: <n9mqlh$oeb$1@ger.gmane.org>
 <CAP1=2W4P7skRpewpgxXBgHHicngMU3mJ2V1s3NVXgQL9PiCJmw@mail.gmail.com>
 <CAP7+vJLUumJiOzExqdmffJG_ywpFA9vXAWUH2Bpzvpiwcwp_RQ@mail.gmail.com>
 <ncisha$auc$1@ger.gmane.org>
 <CADiSq7etYj9Wy4++kP0z_5=-x6Pm0E9=6VASh+nYfJk8bM6PfQ@mail.gmail.com>
 <CAP7+vJ+e60gid=R8gjhmyiiNkCMC0Db4zt-Bct_qKUN29Vbr_Q@mail.gmail.com>
 <loom.20160319T181628-51@post.gmane.org>
 <CAP7+vJ+PewZ=ojQU0LzsZh0n0payOgzdOzfx6zQ73Xh9OYt5aQ@mail.gmail.com>
 <loom.20160319T191115-310@post.gmane.org>
 <CAP7+vJ+F=d9ycxZhaWkoYB6SYXf82T=Lf4QxWubwNy_FKJUACw@mail.gmail.com>
 <CAP1=2W4SM-_SBUo5YGdwyvq2N62+FC3TEgLnmYMW=2VDkV6+2A@mail.gmail.com>
 <CAP7+vJ+BicSaZiizSpjE8q2g=OGHMLgOre1tLwo3o5DN2crZ3g@mail.gmail.com>
 <CAP1=2W7EZSyyT-tT-NbUyKaYSvBamzwk8Vh-gY6nYee14LZZyw@mail.gmail.com>
 <CAP7+vJKLzX4d4R4=Z5nMLaHK1iQ4N8bddbd2dWnUm9zw0K7jVQ@mail.gmail.com>
Message-ID: <CAP1=2W6d7Zcb=XT859+erCUCwOXfVsr_by-zk9PM1t4-eX-tkw@mail.gmail.com>

On Wed, 11 May 2016 at 10:49 Guido van Rossum <guido at python.org> wrote:

> On Wed, May 11, 2016 at 10:28 AM, Brett Cannon <brett at python.org> wrote:
>
>>
>>
>> On Wed, 11 May 2016 at 09:47 Guido van Rossum <guido at python.org> wrote:
>>
>>> If the authors are happy I'll accept it right away.
>>>
>>> (I vaguely recall there's another PEP that's ready for pronouncement --
>>> but which one?)
>>>
>>
>> PEP 509 is the only one I can think of.
>>
>
> That's in limbo pending conclusive proof (through benchmarks) that at
> least one of Yury's patches that depends on it makes a big enough
> difference.
>
> Which IIUC itself is in limbo pending the wordcode changes (we're doing
> that right?).
>

Yes. Last I checked the author of the patch was waiting on a review from
Serhiy.

-Brett


>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160511/7c7f45de/attachment.html>

From brett at python.org  Wed May 11 14:17:29 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 11 May 2016 18:17:29 +0000
Subject: [Python-Dev] Slow downloads from python.org
In-Reply-To: <CAGGBzXJ80M=Coq_zig6w_rBqzX82JWch-sBUBm9CH+Fx6Vok7A@mail.gmail.com>
References: <CAGGBzXJ80M=Coq_zig6w_rBqzX82JWch-sBUBm9CH+Fx6Vok7A@mail.gmail.com>
Message-ID: <CAP1=2W5hW0jus3oGBcROXuGVePTGMr_8q5f2b7a-Ma5YZDT6AA@mail.gmail.com>

On Wed, 11 May 2016 at 10:56 Dima Tisnek <dimaqq at gmail.com> wrote:

> Sorry, this is probably wrong place to ask, but is it only me?
> I can't get more than 40KB/s downloading from python.org


It's just you or the problem has passed; just downloaded much faster than
40KB/s.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160511/e811bab3/attachment-0001.html>

From brett at python.org  Wed May 11 16:04:04 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 11 May 2016 20:04:04 +0000
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
Message-ID: <CAP1=2W5Duha22mcxnxCLuMuLh9yPPPdKyY+yWOFY+Yw6mUex1A@mail.gmail.com>

A quick comment about sending me fixes. While I do appreciate them, sending
them as a pull request is much easier for me as (a) I don't have to hunt
the changes down in the text, and (b) you will see the fixes others have
done already to the PEP and I then don't have to figure out what changes
have not already been fixed. And honestly, reading the PEP in its rendered
format on GitHub is easier IMO than in the text format unless you have
something specific to respond to (and even if you do, you can copy and
paste the relevant bits into an email reply).

On Wed, 11 May 2016 at 09:43 Brett Cannon <brett at python.org> wrote:

> **deep, calming breath**
>
> Here is the PEP for __fspath__(). The draft lives at
> https://github.com/brettcannon/path-pep so feel free to send me PRs for
> spelling mistakes, grammatical errors, etc.
>
> -----
>
> PEP: NNN
> Title: Adding a file system path protocol
> Version: $Revision$
> Last-Modified: $Date$
> Author: Brett Cannon <brett at python.org>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 11-May-2016
> Post-History: 11-May-2016
>
>
> Abstract
> ========
>
> This PEP proposes a protocol for classes which represent a file system
> path to be able to provide a ``str`` or ``bytes`` representation.
> Changes to Python's standard library are also proposed to utilize this
> protocol where appropriate to facilitate the use of path objects where
> historically only ``str`` and/or ``bytes`` file system paths are
> accepted. The goal is to allow users to use the representation of a
> file system path that's easiest for them now as they migrate towards
> using path objects in the future.
>
>
> Rationale
> =========
>
> Historically in Python, file system paths have been represented as
> strings or bytes. This choice of representation has stemmed from C's
> own decision to represent file system paths as
> ``const char *`` [#libc-open]_. While that is a totally serviceable
> format to use for file system paths, it's not necessarily optimal. At
> issue is the fact that while all file system paths can be represented
> as strings or bytes, not all strings or bytes represent a file system
> path. This can lead to issues where any e.g. string duck-types to a
> file system path whether it actually represents a path or not.
>
> To help elevate the representation of file system paths from their
> representation as strings and bytes to a more appropriate object
> representation, the pathlib module [#pathlib]_ was provisionally
> introduced in Python 3.4 through PEP 428. While considered by some as
> an improvement over strings and bytes for file system paths, it has
> suffered from a lack of adoption. Typically the key issue listed
> for the low adoption rate has been the lack of support in the standard
> library. This lack of support required users of pathlib to manually
> convert path objects to strings by calling ``str(path)`` which many
> found error-prone.
>
> One issue in converting path objects to strings comes from
> the fact that only generic way to get a string representation of the
> path was to pass the object to ``str()``. This can pose a
> problem when done blindly as nearly all Python objects have some
> string representation whether they are a path or not, e.g.
> ``str(None)`` will give a result that
> ``builtins.open()`` [#builtins-open]_ will happily use to create a new
> file.
>
> Exacerbating this whole situation is the
> ``DirEntry`` object [#os-direntry]_. While path objects have a
> representation that can be extracted using ``str()``, ``DirEntry``
> objects expose a ``path`` attribute instead. Having no common
> interface between path objects, ``DirEntry``, and any other
> third-party path library had become an issue. A solution that allowed
> any path-representing object to declare that is was a path and a way
> to extract a low-level representation that all path objects could
> support was desired.
>
> This PEP then proposes to introduce a new protocol to be followed by
> objects which represent file system paths. Providing a protocol allows
> for clear signalling of what objects represent file system paths as
> well as a way to extract a lower-level representation that can be used
> with older APIs which only support strings or bytes.
>
> Discussions regarding path objects that led to this PEP can be found
> in multiple threads on the python-ideas mailing list archive
> [#python-ideas-archive]_ for the months of March and April 2016 and on
> the python-dev mailing list archives [#python-dev-archive]_ during
> April 2016.
>
>
> Proposal
> ========
>
> This proposal is split into two parts. One part is the proposal of a
> protocol for objects to declare and provide support for exposing a
> file system path representation. The other part is changes to Python's
> standard library to support the new protocol. These changes will also
> have the pathlib module drop its provisional status.
>
>
> Protocol
> --------
>
> The following abstract base class defines the protocol for an object
> to be considered a path object::
>
>     import abc
>     import typing as t
>
>
>     class PathLike(abc.ABC):
>
>         """Abstract base class for implementing the file system path
> protocol."""
>
>         @abc.abstractmethod
>         def __fspath__(self) -> t.Union[str, bytes]:
>             """Return the file system path representation of the object."""
>             raise NotImplementedError
>
>
> Objects representing file system paths will implement the
> ``__fspath__()`` method which will return the ``str`` or ``bytes``
> representation of the path. The ``str`` representation is the
> preferred low-level path representation as it is human-readable and
> what people historically represent paths as.
>
>
> Standard library changes
> ------------------------
>
> It is expected that most APIs in Python's standard library that
> currently accept a file system path will be updated appropriately to
> accept path objects (whether that requires code or simply an update
> to documentation will vary). The modules mentioned below, though,
> deserve specific details as they have either fundamental changes that
> empower the ability to use path objects, or entail additions/removal
> of APIs.
>
>
> builtins
> ''''''''
>
> ``open()`` [#builtins-open]_ will be updated to accept path objects as
> well as continue to accept ``str`` and ``bytes``.
>
>
> os
> '''
>
> The ``fspath()`` function will be added with the following semantics
> (although this is still an open issue:
> `Should os.fspath() return bytes?`_)::
>
>     import typing as t
>
>
>     def fspath(path: t.Union[PathLike, str]) -> str:
>         """Return the string representation of the path.
>
>         If a string is passed in then it is returned unchanged.
>         """
>         if hasattr(path, '__fspath__'):
>             path = path.__fspath__()
>         if not isinstance(path, str):
>             type_name = type(path).__name__
>             raise TypeError("expected a str or path object, not " +
> type_name)
>         return path
>
> The ``os.fsencode()`` [#os-fsencode]_ and
> ``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept
> path objects. As both functions coerce their arguments to
> ``bytes`` and ``str``, respectively, they will be updated to call
> ``__fspath__()`` if present to convert the path object to a ``str`` or
> ``bytes`` representation, and then perform their appropriate
> coercion operations as if the return value from ``__fspath__()`` had
> been the original argument to the coercion function in question.
>
> The addition of ``os.fspath()``, the updates to
> ``os.fsencode()``/``os.fsdecode()``, and the current semantics of
> ``pathlib.PurePath`` provide the semantics necessary to
> get the path representation one prefers. For a path object,
> ``pathlib.PurePath``/``Path`` can be used. If ``str`` is desired and
> a ``bytes`` object is considered an error, then ``os.fspath()`` can be
> used. If a ``str`` is desired and the encoding of ``bytes`` should be
> assumed to be the default file system encoding, then ``os.fsdecode()``
> should be used. Finally, if a ``bytes`` representation is desired and
> any strings should be encoded using the default file system encoding
> then ``os.fsencode()`` is used.
>
> This PEP recommends using path objects when possible and falling back
> to string paths as necessary. Therefore, no function is provided for
> the case of wanting a bytes representation but without any automatic
> encoding to help discourage the use of multiple bytes encodings on a
> single file system. If it is necessary to deal with an existing file
> system directory with entries in a non-default encoding, this can be
> done with low-level functions using ``str`` and the PEP 383
> ``surrogateescape`` error handler, or by using ``bytes`` directly.
>
> Another way to view this is as a hierarchy of file system path
> representations (highest- to lowest-level): path -> str -> bytes. The
> functions and classes under discussion can all accept objects on the
> same level of the hierarchy, but they vary in whether they promote or
> demote objects to another level. The ``pathlib.PurePath`` class can
> promote a ``str`` to a path object. The ``os.fspath()`` function can
> demote a path object to a string, but only if ``__fspath__()`` returns
> a string. The ``os.fsdecode()`` function will demote a path object to
> a string or promote a ``bytes`` object to a ``str``. The
> ``os.fsencode()`` function will demote a path or string object to
> ``bytes``. There is no function that provides a way to demote a path
> object directly to ``bytes`` and not allow demoting strings.
>
> The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()``
> method. It will return the same value as currently found on the
> ``path`` attribute of ``DirEntry`` instances.
>
>
> os.path
> '''''''
>
> The various path-manipulation functions of ``os.path`` [#os-path]_
> will be updated to accept path objects. For polymorphic functions that
> accept both bytes and strings, they will be updated to simply use
> code very much similar to
> ``path.__fspath__() if  hasattr(path, '__fspath__') else path``. This
> will allow for their pre-existing type-checking code to continue to
> function.
>
> During the discussions leading up to this PEP it was suggested that
> ``os.path`` not be updated using an "explicit is better than implicit"
> argument. The thinking was that since ``__fspath__()`` is polymorphic
> itself it may be better to have code working with ``os.path`` extract
> the path representation from path objects explicitly. There is also
> the consideration that adding support this deep into the low-level OS
> APIs will lead to code magically supporting path objects without
> requiring any documentation updated, leading to potential complaints
> when it doesn't work, unbeknownst to the project author.
>
> But it is the view of the authors that "practicality beats purity" in
> this instance. To help facilitate the transition to supporting path
> objects, it is better to make the transition as easy as possible than
> to worry about unexpected/undocumented duck typing support for
> projects.
>
>
> pathlib
> '''''''
>
> The ``PathLike`` ABC as discussed in the Protocol_ section will be
> added to the pathlib module [#pathlib]_. The constructor for
> ``pathlib.PurePath`` and ``pathlib.Path`` will be updated to accept
> path objects. Both ``PurePath`` and ``Path`` will continue to not
> accept ``bytes`` path representations, and so if ``__fspath__()``
> returns ``bytes`` it will raise an exception.
>
> The ``path`` attribute will be removed as this PEP makes its
> redundant (it has not been included in any released version of Python
> and so is not a backwards-compatibility concern).
>
>
> C API
> '''''
>
> The C API will gain an equivalent function to ``os.fspath()`` that
> also allows bytes objects through::
>
>     /*
>         Return the file system path of the object.
>
>         If the object is str or bytes, then allow it to pass through with
>         an incremented refcount. All other types raise a TypeError.
>     */
>     PyObject *
>     PyOS_RawFSPath(PyObject *path)
>     {
>         if (PyObject_HasAttrString(path, "__fspath__")) {
>             path = PyObject_CallMethodObjArgs(path, "__fspath__", NULL);
>             if (path == NULL) {
>                 return NULL;
>             }
>         }
>         else {
>             Py_INCREF(path);
>         }
>
>         if (!PyUnicode_Check(path) && !PyBytes_Check(path)) {
>             Py_DECREF(path);
>             return PyErr_Format(PyExc_TypeError,
>                                 "expected a string, bytes, or path object,
> not %S",
>                                 path->ob_type);
>         }
>
>         return path;
> }
>
>
> Backwards compatibility
> =======================
>
> There are no explicit backwards-compatibility concerns. Unless an
> object incidentally already defines a ``__fspath__()`` method there is
> not reason to expect pre-existing code to break or expect to have
> their semantics implicitly changed.
>
> Libraries wishing to support path objects and a version of Python
> prior to Python 3.6 can use the idiom of
> ``path.__fspath__() if hasattr(path, '__fspath__') else path``.
>
>
> Open Issues
> ===========
>
> Should os.fspath() return bytes?
> --------------------------------
>
> Some have argued that ``os.fspath()`` should be configurable so that
> the user can specify what types are acceptable (e.g. an argument to
> say that bytes are acceptable instead of strings, or both types).
> Others have suggested that ``os.fspath()`` match the proposed
> semantics of ``PyOS_RawFSPath()``. Both camps argue that use of
> ``os.fspath()`` will only be for a transitionary period while more
> libraries gain acceptance of path objects, and so being more flexible
> in what ``os.fspath()`` works with will help with the transition. The
> opponents to this -- which support the currently proposed sematnics --
> worry that being so flexible with accepting bytes will lead to people
> not properly considering the ramifications of working with bytes,
> especially if bytes are transparently appearing in their code due to
> ``os.fspath()``.
>
>
> The name and location of the protocol's ABC
> -------------------------------------------
>
> The name of the ABC being proposed to represent the protocol has not
> been discussed very much, not which module it should exist in.
> Names other than ``PathLike`` which are viable are ``PathABC``
> and ``FSPathABC``. The name can't be ``Path`` if the ABC is put into
> the pathlib module.
>
>
> Type hint for path-like objects
> -------------------------------
>
> Creating a proper type hint for  APIs that accept path objects as well
> as strings and bytes will probably be needed. It could be as simple
> as defining ``typing.Path``/``typing.FSPath`` to correspond to the ABC
> and then having
> ``typing.PathLike = typing.Union[typing.Path, str, bytes]``. The type
> hint could also potentially be made to be generic to accept the
> specific low-level representation, e.g. ``typing.PathLike[str]``.
>
> In the end the type hinting solution should be properly discussed
> with the right type hinting experts if this is the best approach.
>
>
> Rejected Ideas
> ==============
>
> Other names for the protocol's function
> ---------------------------------------
>
> Various names were proposed during discussions leading to this PEP,
> including ``__path__``, ``__pathname__``, and ``__fspathname__``. In
> the end people seemed to gravitate towards ``__fspath__`` for being
> unambiguous without unnecessarily long.
>
>
> Separate str/bytes methods
> --------------------------
>
> At one point it was suggested that ``__fspath__()`` only return
> strings and another method named ``__fspathb__()`` be introduced to
> return bytes. The thinking that by making ``__fspath__()`` not be
> polymorphic it could make dealing with the potential string or bytes
> representations easier. But the general consensus was that returning
> bytes will more than likely be rare and that the various functions in
> the os module are the better abstraction to be promoting over direct
> calls to ``__fspath__()``.
>
>
> Providing a path attribute
> --------------------------
>
> To help deal with the issue of ``pathlib.PurePath`` not inheriting
> from ``str``, originally it was proposed to introduce a ``path``
> attribute to mirror what ``os.DirEntry`` provides. In the end, though,
> it was determined that a protocol would provide the same result while
> not directly exposing an API that most people will never need to
> interact with directly.
>
>
> Have ``__fspath__()`` only return strings
> ------------------------------------------
>
> Much of the discussion that led to this PEP revolved around whether
> ``__fspath__()`` should be polymorphic and return ``bytes`` as well as
> ``str`` instead of only ``str``. The general sentiment for this view
> was that ``bytes`` are difficult to work with due to their
> inherent lack of information about their encoding, and PEP 383 makes
> it possible to represent all file system paths using ``str`` with the
> ``surrogateescape`` handler. Thus it would be better to forcibly
> promote the use of ``str`` as the low-level path representation for
> high-level path objects.
>
> In the end it was decided that using ``bytes`` to represent paths is
> simply not going to go away and thus they should be supported to some
> degree. For those not wanting the hassle of working with ``bytes``,
> ``os.fspath()`` is provided.
>
>
> A generic string encoding mechanism
> -----------------------------------
>
> At one point there was discussion of developing a generic mechanism to
> extract a string representation of an object that had semantic meaning
> (``__str__()`` does not necessarily return anything of semantic
> significance beyond what may be helpful for debugging). In the end it
> was deemed to lack a motivating need beyond the one this PEP is
> trying to solve in a specific fashion.
>
>
> Acknowledgements
> ================
>
> Thanks to everyone who participated in the various discussions related
> to this PEP that spanned both python-ideas and python-dev. Special
> thanks to Koos Zevenhoven and Stephen Turnbull for direct feedback on
> early drafts of this PEP.
>
>
> References
> ==========
>
> .. [#python-ideas-archive] The python-ideas mailing list archive
>    (https://mail.python.org/pipermail/python-ideas/)
>
> .. [#python-dev-archive] The python-dev mailing list archive
>    (https://mail.python.org/pipermail/python-dev/)
>
> .. [#libc-open] ``open()`` documention for the C standard library
>    (
> http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html
> )
>
> .. [#pathlib] The ``pathlib`` module
>    (https://docs.python.org/3/library/pathlib.html#module-pathlib)
>
> .. [#builtins-open] The ``builtins.open()`` function
>    (https://docs.python.org/3/library/functions.html#open)
>
> .. [#os-fsencode] The ``os.fsencode()`` function
>    (https://docs.python.org/3/library/os.html#os.fsencode)
>
> .. [#os-fsdecode] The ``os.fsdecode()`` function
>    (https://docs.python.org/3/library/os.html#os.fsdecode)
>
> .. [#os-direntry] The ``os.DirEntry`` class
>    (https://docs.python.org/3/library/os.html#os.DirEntry)
>
> .. [#os-path] The ``os.path`` module
>    (https://docs.python.org/3/library/os.path.html#module-os.path)
>
>
> Copyright
> =========
>
> This document has been placed in the public domain.
>
>
> ..
>    Local Variables:
>    mode: indented-text
>    indent-tabs-mode: nil
>    sentence-end-double-space: t
>    fill-column: 70
>    coding: utf-8
>    End:
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160511/d63c7084/attachment.html>

From storchaka at gmail.com  Wed May 11 16:44:36 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 11 May 2016 23:44:36 +0300
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
Message-ID: <nh05jk$58e$1@ger.gmane.org>

On 11.05.16 19:43, Brett Cannon wrote:
> os.path
> '''''''
>
> The various path-manipulation functions of ``os.path`` [#os-path]_
> will be updated to accept path objects. For polymorphic functions that
> accept both bytes and strings, they will be updated to simply use
> code very much similar to
> ``path.__fspath__() if  hasattr(path, '__fspath__') else path``. This
> will allow for their pre-existing type-checking code to continue to
> function.

I afraid that this will hit a performance. Some os.path functions are 
used in tight loops, they are hard optimized, and adding support of path 
protocol can have visible negative effect.

I suggest first implement other changes and then look whether it is 
worth to add support of path protocol in os.path functions.



From ethan at stoneleaf.us  Wed May 11 16:51:31 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 11 May 2016 13:51:31 -0700
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <nh05jk$58e$1@ger.gmane.org>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <nh05jk$58e$1@ger.gmane.org>
Message-ID: <57339B53.3010506@stoneleaf.us>

On 05/11/2016 01:44 PM, Serhiy Storchaka wrote:

>> os.path
>> '''''''
>>
>> The various path-manipulation functions of ``os.path`` [#os-path]_
>> will be updated to accept path objects. For polymorphic functions that
>> accept both bytes and strings, they will be updated to simply use
>> code very much similar to
>> ``path.__fspath__() if  hasattr(path, '__fspath__') else path``. This
>> will allow for their pre-existing type-checking code to continue to
>> function.
>
> I afraid that this will hit a performance. Some os.path functions are
> used in tight loops, they are hard optimized, and adding support of path
> protocol can have visible negative effect.

Do you have an example of os.path functions being used in a tight loop?

--
~Ethan~


From k7hoven at gmail.com  Wed May 11 16:57:37 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Wed, 11 May 2016 23:57:37 +0300
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP1=2W5Duha22mcxnxCLuMuLh9yPPPdKyY+yWOFY+Yw6mUex1A@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <CAP1=2W5Duha22mcxnxCLuMuLh9yPPPdKyY+yWOFY+Yw6mUex1A@mail.gmail.com>
Message-ID: <CAMiohog-urpaiLcki2UprqBvHsVmZYPgqr7qt9WMEPz2==jUWA@mail.gmail.com>

On Wed, May 11, 2016 at 11:04 PM, Brett Cannon <brett at python.org> wrote:
> A quick comment about sending me fixes. While I do appreciate them, sending
> them as a pull request is much easier for me as (a) I don't have to hunt the
> changes down in the text, and (b) you will see the fixes others have done
> already to the PEP and I then don't have to figure out what changes have not
> already been fixed. And honestly, reading the PEP in its rendered format on
> GitHub is easier IMO than in the text format unless you have something
> specific to respond to (and even if you do, you can copy and paste the
> relevant bits into an email reply).
>

Personally, I find it more important to settle the open issues first,
so I will be looking for typos at a later stage. Besides, more typos
may be added in the process. I previously left some typos uncorrected
in attempt to keep the diffs of my commits cleaner. By the way, to see
what additions and wordings I have suggested, the right way is to look
at my PR(s).  I might send more PRs to clarify open issues etc., but
this will largely be based on my previous commits.

-- Koos

From ethan at stoneleaf.us  Wed May 11 17:15:37 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 11 May 2016 14:15:37 -0700
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <57339B53.3010506@stoneleaf.us>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <nh05jk$58e$1@ger.gmane.org> <57339B53.3010506@stoneleaf.us>
Message-ID: <5733A0F9.7070704@stoneleaf.us>

On 05/11/2016 01:51 PM, Ethan Furman wrote:
> On 05/11/2016 01:44 PM, Serhiy Storchaka wrote:

>>> os.path
>>> '''''''
>>>
>>> The various path-manipulation functions of ``os.path`` [#os-path]_
>>> will be updated to accept path objects. For polymorphic functions that
>>> accept both bytes and strings, they will be updated to simply use
>>> code very much similar to
>>> ``path.__fspath__() if  hasattr(path, '__fspath__') else path``. This
>>> will allow for their pre-existing type-checking code to continue to
>>> function.
>>
>> I afraid that this will hit a performance. Some os.path functions are
>> used in tight loops, they are hard optimized, and adding support of path
>> protocol can have visible negative effect.
>
> Do you have an example of os.path functions being used in a tight loop?

Also, the C code for fspath can check types first and take the fast path 
if bytes/str are passed in, only falling back to the __fspath__ protocol 
if something else was passed in -- which should make any performance 
hits negligible.

--
~Ethan~


From k7hoven at gmail.com  Wed May 11 17:22:22 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Thu, 12 May 2016 00:22:22 +0300
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <5733A0F9.7070704@stoneleaf.us>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <nh05jk$58e$1@ger.gmane.org> <57339B53.3010506@stoneleaf.us>
 <5733A0F9.7070704@stoneleaf.us>
Message-ID: <CAMiohoisLYFBv_gF=ivpgV4=TSCiQdjuh=5LGS_Wn+yo5G4OMg@mail.gmail.com>

On Thu, May 12, 2016 at 12:15 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 05/11/2016 01:51 PM, Ethan Furman wrote:
>>
>> On 05/11/2016 01:44 PM, Serhiy Storchaka wrote:
>
>
>>>> os.path
>>>> '''''''
>>>>
>>>> The various path-manipulation functions of ``os.path`` [#os-path]_
>>>> will be updated to accept path objects. For polymorphic functions that
>>>> accept both bytes and strings, they will be updated to simply use
>>>> code very much similar to
>>>> ``path.__fspath__() if  hasattr(path, '__fspath__') else path``. This
>>>> will allow for their pre-existing type-checking code to continue to
>>>> function.
>>>
>>>
>>> I afraid that this will hit a performance. Some os.path functions are
>>> used in tight loops, they are hard optimized, and adding support of path
>>> protocol can have visible negative effect.
>>
>>
>> Do you have an example of os.path functions being used in a tight loop?
>

I'd be interested in this too.

>
> Also, the C code for fspath can check types first and take the fast path if
> bytes/str are passed in, only falling back to the __fspath__ protocol if
> something else was passed in -- which should make any performance hits
> negligible.

My suggestion for the *python version* already does this too (again,
see my PR). This is a win-win, as it also improves error messages (as
suggested by Nick in the earlier discussions).

-- Koos

>
>
> --
> ~Ethan~
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com

From Nikolaus at rath.org  Wed May 11 17:28:31 2016
From: Nikolaus at rath.org (Nikolaus Rath)
Date: Wed, 11 May 2016 14:28:31 -0700
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 (Brett Cannon's message of "Wed, 11 May 2016 16:43:24 +0000")
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
Message-ID: <87shxo1dz4.fsf@thinkpad.rath.org>

On May 11 2016, Brett Cannon <brett at python.org> wrote:
> This PEP proposes a protocol for classes which represent a file system
> path to be able to provide a ``str`` or ``bytes`` representation.
[...]

As I said before, to me this seems like a lot of effort for a very
specific use-case. So let me put forward two hypothetical scenarios to
better understand your position:

- A new module for URL handling is added to the standard library (or
  urllib is suitably extended). There is a proposal to add a new
  protocol that allows classes to provide a ``str`` or ``bytes``
  representation of URLs.

- A new (third-party) library for natural language processing arises
  that exposes a specific class for representing audio data. Existing
  language processing code just uses bytes objects. To ease transition
  and interoperability, it is proposed to add a new protocol for classes
  that represend audio data to provide a bytes representation.

Do you think you would you be in favor of adding these protocols to
the stdlib/languange reference as well? If not, what's the crucial
difference to file system paths?


Thanks,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             ?Time flies like an arrow, fruit flies like a Banana.?

From k7hoven at gmail.com  Wed May 11 17:42:08 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Thu, 12 May 2016 00:42:08 +0300
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <87shxo1dz4.fsf@thinkpad.rath.org>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <87shxo1dz4.fsf@thinkpad.rath.org>
Message-ID: <CAMiohoj+jdX9E94DOP98EW2=07WRwaXyf7OU+A+r8Ni2nEKMbw@mail.gmail.com>

On Thu, May 12, 2016 at 12:28 AM, Nikolaus Rath <Nikolaus at rath.org> wrote:
> On May 11 2016, Brett Cannon <brett at python.org> wrote:
>> This PEP proposes a protocol for classes which represent a file system
>> path to be able to provide a ``str`` or ``bytes`` representation.
> [...]
>
> As I said before, to me this seems like a lot of effort for a very
> specific use-case. So let me put forward two hypothetical scenarios to
> better understand your position:

I think you are touching important points.

> - A new module for URL handling is added to the standard library (or
>   urllib is suitably extended). There is a proposal to add a new
>   protocol that allows classes to provide a ``str`` or ``bytes``
>   representation of URLs.

This reminds me of the thread I recently started on python-ideas about
extending the concept of paths to URLs. I don't know if you are
referring to something like that or not.

Anyway, it would be important to know whether the str or bytes
representation is to be used a file system path or an URL, so that
would need to be a separate protocol. But everyone would have the
experience from these discussions, so hopefully less discussion then
:). (By the way, this is one reason to have 'fs' in the name of the
__fspath__ method, although I have not mentioned it before.)

>
> - A new (third-party) library for natural language processing arises
>   that exposes a specific class for representing audio data. Existing
>   language processing code just uses bytes objects. To ease transition
>   and interoperability, it is proposed to add a new protocol for classes
>   that represend audio data to provide a bytes representation.
>
> Do you think you would you be in favor of adding these protocols to
> the stdlib/languange reference as well? If not, what's the crucial
> difference to file system paths?
>

File system paths are very fundamental, and will probably be used in
context of your natural language example too.

-- Koos

>
> Thanks,
> -Nikolaus

From ethan at stoneleaf.us  Wed May 11 17:52:54 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 11 May 2016 14:52:54 -0700
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <87shxo1dz4.fsf@thinkpad.rath.org>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <87shxo1dz4.fsf@thinkpad.rath.org>
Message-ID: <5733A9B6.1000707@stoneleaf.us>

On 05/11/2016 02:28 PM, Nikolaus Rath wrote:
> On May 11 2016, Brett Cannon wrote:

>> This PEP proposes a protocol for classes which represent a file system
>> path to be able to provide a ``str`` or ``bytes`` representation.
> [...]
>
> As I said before, to me this seems like a lot of effort for a very
> specific use-case. So let me put forward two hypothetical scenarios to
> better understand your position:
>
> - A new module for URL handling is added to the standard library (or
>    urllib is suitably extended). There is a proposal to add a new
>    protocol that allows classes to provide a ``str`` or ``bytes``
>    representation of URLs.
>
> - A new (third-party) library for natural language processing arises
>    that exposes a specific class for representing audio data. Existing
>    language processing code just uses bytes objects. To ease transition
>    and interoperability, it is proposed to add a new protocol for classes
>    that represend audio data to provide a bytes representation.
>
> Do you think you would you be in favor of adding these protocols to
> the stdlib/languange reference as well? If not, what's the crucial
> difference to file system paths?

I think a crucial reason for this work is to unify the stdlib: we 
currently have four (?) different things that can be or represent a 
file-system path:

- str
- bytes
- DirEntry
- Path

Half of those objects don't work well with the rest of the standard library.

As for your second example, the protocol already exists: it's called 
__bytes__.  ;)

--
~Ethan~

From brett at python.org  Wed May 11 17:57:59 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 11 May 2016 21:57:59 +0000
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <87shxo1dz4.fsf@thinkpad.rath.org>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <87shxo1dz4.fsf@thinkpad.rath.org>
Message-ID: <CAP1=2W7doTALgDcZ8zixVqf_CyyXeqs_U_0AQvkDSq9Lsu=cMw@mail.gmail.com>

On Wed, 11 May 2016 at 14:29 Nikolaus Rath <Nikolaus at rath.org> wrote:

> On May 11 2016, Brett Cannon <brett at python.org> wrote:
> > This PEP proposes a protocol for classes which represent a file system
> > path to be able to provide a ``str`` or ``bytes`` representation.
> [...]
>
> As I said before, to me this seems like a lot of effort for a very
> specific use-case. So let me put forward two hypothetical scenarios to
> better understand your position:
>
> - A new module for URL handling is added to the standard library (or
>   urllib is suitably extended). There is a proposal to add a new
>   protocol that allows classes to provide a ``str`` or ``bytes``
>   representation of URLs.
>
> - A new (third-party) library for natural language processing arises
>   that exposes a specific class for representing audio data. Existing
>   language processing code just uses bytes objects. To ease transition
>   and interoperability, it is proposed to add a new protocol for classes
>   that represend audio data to provide a bytes representation.
>
> Do you think you would you be in favor of adding these protocols to
> the stdlib/languange reference as well?


Maybe for URLs, not for audio data (at least not in the stdlib; community
can do what they want).


> If not, what's the crucial
> difference to file system paths?
>

Nearly everyone uses file system paths on a regular basis, less so than
URLs but still a good amount of people. Very few people work with audio
data.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160511/a7c8fc19/attachment.html>

From brett at python.org  Wed May 11 18:13:52 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 11 May 2016 22:13:52 +0000
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <nh05jk$58e$1@ger.gmane.org>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <nh05jk$58e$1@ger.gmane.org>
Message-ID: <CAP1=2W6gFRRArm1j5QX47pDJD-Z_ng85CD79goJYJFzEXQdaUg@mail.gmail.com>

On Wed, 11 May 2016 at 13:45 Serhiy Storchaka <storchaka at gmail.com> wrote:

> On 11.05.16 19:43, Brett Cannon wrote:
> > os.path
> > '''''''
> >
> > The various path-manipulation functions of ``os.path`` [#os-path]_
> > will be updated to accept path objects. For polymorphic functions that
> > accept both bytes and strings, they will be updated to simply use
> > code very much similar to
> > ``path.__fspath__() if  hasattr(path, '__fspath__') else path``. This
> > will allow for their pre-existing type-checking code to continue to
> > function.
>
> I afraid that this will hit a performance. Some os.path functions are
> used in tight loops, they are hard optimized, and adding support of path
> protocol can have visible negative effect.
>

As others have asked, what specific examples do you have that os.path is
used in a tight loop w/o any I/O that would overwhelm the performance?


>
> I suggest first implement other changes and then look whether it is
> worth to add support of path protocol in os.path functions.
>

I see this whole discussion breaking down into a few groups which changes
what gets done upfront and what might be done farther down the line:

   1. Maximum acceptance: do whatever we can to make all representation of
   paths just work, which means making all places working with a path in the
   stdlib accept path objects, str, and bytes.
   2. Safely use path objects: __fspath__() is there to signal an object is
   a file system path and to get back a lower-level representation so people
   stop calling str() on everything, providing some interface signaling that
   someone doesn't misuse an object as a path and only changing path
   consumptions APIs -- e.g. open() -- and not path manipulation APIs -- e.g.
   os.path -- in the stdlib.
   3. It ain't worth it: those that would rather just skip all of this and
   drop pathlib from the stdlib.

Ethan and Koos are in group #1 and I'm personally in group #2 but I tried
to compromise somewhat and find a middle ground in the PEP with the level
of changes in the stdlib but being more restrictive with os.fspath(). If I
were doing a pure group #2 PEP I would drop os.path changes and make
os.fspath() do what Ethan and Koos have suggested and simply pass through
without checks whatever path.__fspath__() returned if the argument wasn't
str or bytes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160511/4aa3733c/attachment.html>

From ethan at stoneleaf.us  Wed May 11 19:05:32 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 11 May 2016 16:05:32 -0700
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP1=2W6gFRRArm1j5QX47pDJD-Z_ng85CD79goJYJFzEXQdaUg@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <nh05jk$58e$1@ger.gmane.org>
 <CAP1=2W6gFRRArm1j5QX47pDJD-Z_ng85CD79goJYJFzEXQdaUg@mail.gmail.com>
Message-ID: <5733BABC.3040903@stoneleaf.us>

On 05/11/2016 03:13 PM, Brett Cannon wrote:

> If [...] I would drop os.path changes and make os.fspath() do what
 > Ethan and Koos have suggested and simply pass through without checks
> whatever path.__fspath__() returned if the argument wasn't str or bytes.

Not to derail the conversation too much, as I know we're all getting 
burned out on the topic, but that last bit is not accurate: my druthers 
are to have __fspath__ be able to return str /or/ bytes, and if anything 
else comes from the object in question an exception must be raised. 
Maybe a word got lost between your thoughts and your fingers -- happens 
to me all the time.  :)

--
~Ethan~

From k7hoven at gmail.com  Wed May 11 19:08:03 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Thu, 12 May 2016 02:08:03 +0300
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP1=2W6gFRRArm1j5QX47pDJD-Z_ng85CD79goJYJFzEXQdaUg@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <nh05jk$58e$1@ger.gmane.org>
 <CAP1=2W6gFRRArm1j5QX47pDJD-Z_ng85CD79goJYJFzEXQdaUg@mail.gmail.com>
Message-ID: <CAMiohoiMwid0d9S_Hb_q2rE_gu7pywYM-v49_0C_gp0HgxsSLg@mail.gmail.com>

On Thu, May 12, 2016 at 1:13 AM, Brett Cannon <brett at python.org> wrote:
>
>
> On Wed, 11 May 2016 at 13:45 Serhiy Storchaka <storchaka at gmail.com> wrote:
>>
>> On 11.05.16 19:43, Brett Cannon wrote:
>> > os.path
>> > '''''''
>> >
>> > The various path-manipulation functions of ``os.path`` [#os-path]_
>> > will be updated to accept path objects. For polymorphic functions that
>> > accept both bytes and strings, they will be updated to simply use
>> > code very much similar to
>> > ``path.__fspath__() if  hasattr(path, '__fspath__') else path``. This
>> > will allow for their pre-existing type-checking code to continue to
>> > function.
>>
>> I afraid that this will hit a performance. Some os.path functions are
>> used in tight loops, they are hard optimized, and adding support of path
>> protocol can have visible negative effect.
>
>
> As others have asked, what specific examples do you have that os.path is
> used in a tight loop w/o any I/O that would overwhelm the performance?
>
>>
>>
>> I suggest first implement other changes and then look whether it is
>> worth to add support of path protocol in os.path functions.
>
>
> I see this whole discussion breaking down into a few groups which changes
> what gets done upfront and what might be done farther down the line:
>
> 1. Maximum acceptance: do whatever we can to make all representation of paths
> just work, which means making all places working with a path in the stdlib
> accept path objects, str, and bytes.

Since you are putting me in this camp, there is at least one thing you
are wrong about. I don't want all places that work with a path to
accept bytes. Only those that already do so, including os/os.path. And
yes, I think the stdlib should show a good example in accepting path
types (especially those provided in the stdlib itself).

Whether Ethan is fully in camp 1, I don't know. Not that I think he
would be any closer to the other camps, though.

> 2. Safely use path objects: __fspath__() is there to signal an object is a file
> system path and to get back a lower-level representation so people stop
> calling str() on everything, providing some interface signaling that someone
> doesn't misuse an object as a path and only changing path consumptions APIs
> -- e.g. open() -- and not path manipulation APIs -- e.g. os.path -- in the
> stdlib.
>
> 3. It ain't worth it: those that would rather just skip all of this and drop
> pathlib from the stdlib.
>
> Ethan and Koos are in group #1 and I'm personally in group #2 but I tried to
> compromise somewhat and find a middle ground in the PEP with the level of
> changes in the stdlib but being more restrictive with os.fspath(). If I were
> doing a pure group #2 PEP I would drop os.path changes and make os.fspath()
> do what Ethan and Koos have suggested and simply pass through without checks
> whatever path.__fspath__() returned if the argument wasn't str or bytes.
>

Related to this, based on the earlier discussions, I had the
impression that you were largely in the same camp as me. In fact, I
thought you had politely left some things out of the PEP draft so I
could fill them in. It turned out I was wrong about that, because you
didn't merge them.

-- Koos

> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
>

From k7hoven at gmail.com  Wed May 11 19:49:30 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Thu, 12 May 2016 02:49:30 +0300
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <5733BABC.3040903@stoneleaf.us>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <nh05jk$58e$1@ger.gmane.org>
 <CAP1=2W6gFRRArm1j5QX47pDJD-Z_ng85CD79goJYJFzEXQdaUg@mail.gmail.com>
 <5733BABC.3040903@stoneleaf.us>
Message-ID: <CAMiohojBKWxds_ppyZPf6acjkHYctyENKzJo5JcU=ukueVcENA@mail.gmail.com>

On Thu, May 12, 2016 at 2:05 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 05/11/2016 03:13 PM, Brett Cannon wrote:
>
>> If [...] I would drop os.path changes and make os.fspath() do what
>
>> Ethan and Koos have suggested and simply pass through without checks
>>
>> whatever path.__fspath__() returned if the argument wasn't str or bytes.
>
>
> Not to derail the conversation too much, as I know we're all getting burned
> out on the topic, but that last bit is not accurate: my druthers are to have
> __fspath__ be able to return str /or/ bytes, and if anything else comes from
> the object in question an exception must be raised. Maybe a word got lost
> between your thoughts and your fingers -- happens to me all the time.  :)

Yes. This would also be equivalent to my fspath(path,
type_constraint=(str,bytes)). And if the compromise I mentioned about
the rejecting (by default or optionally) is lifted, the keyword
argument would not be needed.  I might be ok with throwing away the
isinstance check on the return value of __fspath__() if it has
significant impact on performance in realistic cases (with DirEntry
most likely, I suppose), but I doubt it.

-- Koos

>
> --
> ~Ethan~
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com

From k7hoven at gmail.com  Wed May 11 19:53:41 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Thu, 12 May 2016 02:53:41 +0300
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAMiohojBKWxds_ppyZPf6acjkHYctyENKzJo5JcU=ukueVcENA@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <nh05jk$58e$1@ger.gmane.org>
 <CAP1=2W6gFRRArm1j5QX47pDJD-Z_ng85CD79goJYJFzEXQdaUg@mail.gmail.com>
 <5733BABC.3040903@stoneleaf.us>
 <CAMiohojBKWxds_ppyZPf6acjkHYctyENKzJo5JcU=ukueVcENA@mail.gmail.com>
Message-ID: <CAMiohoi51FhrVWBBXh86ZEma-jUDa=7GGeoUjv1Dfr-tyBWuVw@mail.gmail.com>

On Thu, May 12, 2016 at 2:49 AM, Koos Zevenhoven <k7hoven at gmail.com> wrote:
> On Thu, May 12, 2016 at 2:05 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
>> On 05/11/2016 03:13 PM, Brett Cannon wrote:
>>
>>> If [...] I would drop os.path changes and make os.fspath() do what
>>
>>> Ethan and Koos have suggested and simply pass through without checks
>>>
>>> whatever path.__fspath__() returned if the argument wasn't str or bytes.
>>
>>
>> Not to derail the conversation too much, as I know we're all getting burned
>> out on the topic, but that last bit is not accurate: my druthers are to have
>> __fspath__ be able to return str /or/ bytes, and if anything else comes from
>> the object in question an exception must be raised. Maybe a word got lost
>> between your thoughts and your fingers -- happens to me all the time.  :)
>
> Yes. This would also be equivalent to my fspath(path,
> type_constraint=(str,bytes)). And if the compromise I mentioned about
> the rejecting (by default or optionally) is lifted, the keyword

the rejecting -> rejecting bytes

Good night.

-- Koos


> argument would not be needed.  I might be ok with throwing away the
> isinstance check on the return value of __fspath__() if it has
> significant impact on performance in realistic cases (with DirEntry
> most likely, I suppose), but I doubt it.
>
> -- Koos
>
>>
>> --
>> ~Ethan~
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com

From k7hoven at gmail.com  Wed May 11 20:00:42 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Thu, 12 May 2016 03:00:42 +0300
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAMiohoi51FhrVWBBXh86ZEma-jUDa=7GGeoUjv1Dfr-tyBWuVw@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <nh05jk$58e$1@ger.gmane.org>
 <CAP1=2W6gFRRArm1j5QX47pDJD-Z_ng85CD79goJYJFzEXQdaUg@mail.gmail.com>
 <5733BABC.3040903@stoneleaf.us>
 <CAMiohojBKWxds_ppyZPf6acjkHYctyENKzJo5JcU=ukueVcENA@mail.gmail.com>
 <CAMiohoi51FhrVWBBXh86ZEma-jUDa=7GGeoUjv1Dfr-tyBWuVw@mail.gmail.com>
Message-ID: <CAMiohojSkuPfNBBbHnhKv5SB_AZmnih=FH16kC5+tZLJeXoYwQ@mail.gmail.com>

On Thu, May 12, 2016 at 2:53 AM, Koos Zevenhoven <k7hoven at gmail.com> wrote:
> On Thu, May 12, 2016 at 2:49 AM, Koos Zevenhoven <k7hoven at gmail.com> wrote:
>> On Thu, May 12, 2016 at 2:05 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
>>> On 05/11/2016 03:13 PM, Brett Cannon wrote:
>>>
>>>> If [...] I would drop os.path changes and make os.fspath() do what
>>>
>>>> Ethan and Koos have suggested and simply pass through without checks
>>>>
>>>> whatever path.__fspath__() returned if the argument wasn't str or bytes.
>>>
>>>
>>> Not to derail the conversation too much, as I know we're all getting burned
>>> out on the topic, but that last bit is not accurate: my druthers are to have
>>> __fspath__ be able to return str /or/ bytes, and if anything else comes from
>>> the object in question an exception must be raised. Maybe a word got lost
>>> between your thoughts and your fingers -- happens to me all the time.  :)
>>
>> Yes. This would also be equivalent to my fspath(path,
>> type_constraint=(str,bytes)). And if the compromise I mentioned about
>> the rejecting (by default or optionally) is lifted, the keyword
>
> the rejecting -> rejecting bytes
>
>> argument would not be needed.  I might be ok with throwing away the
>> isinstance check on the return value of __fspath__() if it has
>> significant impact on performance in realistic cases (with DirEntry
>> most likely, I suppose), but I doubt it.
>>
>> -- Koos
>>

I will send a pull request about this tomorrow.

-- Koos

>>>
>>> --
>>> ~Ethan~
>>>
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com

From arthur.darcet+list at m4x.org  Wed May 11 20:13:32 2016
From: arthur.darcet+list at m4x.org (Arthur Darcet)
Date: Thu, 12 May 2016 02:13:32 +0200
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <57339B53.3010506@stoneleaf.us>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <nh05jk$58e$1@ger.gmane.org> <57339B53.3010506@stoneleaf.us>
Message-ID: <CAOwXji6A2iJu16JSJQ+DY0EWtoL3pwXmLLKNkx_KiH83_Hw0cw@mail.gmail.com>

On 11 May 2016 at 22:51, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 05/11/2016 01:44 PM, Serhiy Storchaka wrote:
>
> os.path
>>> '''''''
>>>
>>> The various path-manipulation functions of ``os.path`` [#os-path]_
>>> will be updated to accept path objects. For polymorphic functions that
>>> accept both bytes and strings, they will be updated to simply use
>>> code very much similar to
>>> ``path.__fspath__() if  hasattr(path, '__fspath__') else path``. This
>>> will allow for their pre-existing type-checking code to continue to
>>> function.
>>>
>>
>> I afraid that this will hit a performance. Some os.path functions are
>> used in tight loops, they are hard optimized, and adding support of path
>> protocol can have visible negative effect.
>>
>
> Do you have an example of os.path functions being used in a tight loop?
>

os.path.getmtime could be used in a tight loop, to sync directories with a
lot of files for instance.

% python3 -m timeit -s "import os.path; p = 'out'" "hasattr(p,
'__fspath__'), os.path.getmtime(p)"
100000 loops, best of 3: 2.67 usec per loop
% python3 -m timeit -s "import os.path; p = 'out'" "isinstance(p, (str,
bytes)), os.path.getmtime(p)"
100000 loops, best of 3: 2.45 usec per loop
% python3 -m timeit -s "import os.path; p = 'out'" "os.path.getmtime(p)"
100000 loops, best of 3: 2.02 usec per loop

a 25% markup is a lot imo.

a isinstance check prior to the hasattr might be a way to mitigate this a
bit (but it doesn't help much)
Granted, this example could be optimised by calling os.stat directly, which
is not in os.path, but still, worth considering
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160512/843b47d9/attachment.html>

From ethan at stoneleaf.us  Wed May 11 20:40:22 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 11 May 2016 17:40:22 -0700
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAOwXji6A2iJu16JSJQ+DY0EWtoL3pwXmLLKNkx_KiH83_Hw0cw@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <nh05jk$58e$1@ger.gmane.org> <57339B53.3010506@stoneleaf.us>
 <CAOwXji6A2iJu16JSJQ+DY0EWtoL3pwXmLLKNkx_KiH83_Hw0cw@mail.gmail.com>
Message-ID: <5733D0F6.9030007@stoneleaf.us>

On 05/11/2016 05:13 PM, Arthur Darcet wrote:

> os.path.getmtime could be used in a tight loop, to sync directories with
> a lot of files for instance.
>
> % python3 -m timeit -s "import os.path; p = 'out'" "hasattr(p,
> '__fspath__'), os.path.getmtime(p)"
> 100000 loops, best of 3: 2.67 usec per loop
> % python3 -m timeit -s "import os.path; p = 'out'" "isinstance(p, (str,
> bytes)), os.path.getmtime(p)"
> 100000 loops, best of 3: 2.45 usec per loop
> % python3 -m timeit -s "import os.path; p = 'out'" "os.path.getmtime(p)"
> 100000 loops, best of 3: 2.02 usec per loop
>
> a 25% markup is a lot imo.

I don't think those results are very informative, since 
os.path.getmtime() accepts both str and bytes, so it must already have 
the str/bytes check.

--
~Ethan~


From brett at python.org  Wed May 11 21:01:25 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 12 May 2016 01:01:25 +0000
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAMiohoiMwid0d9S_Hb_q2rE_gu7pywYM-v49_0C_gp0HgxsSLg@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <nh05jk$58e$1@ger.gmane.org>
 <CAP1=2W6gFRRArm1j5QX47pDJD-Z_ng85CD79goJYJFzEXQdaUg@mail.gmail.com>
 <CAMiohoiMwid0d9S_Hb_q2rE_gu7pywYM-v49_0C_gp0HgxsSLg@mail.gmail.com>
Message-ID: <CAP1=2W6LT55B=7i6dMadsq=1_LqRwH+Oy18Zgk2o0E3c1Rau7Q@mail.gmail.com>

On Wed, 11 May 2016 at 16:08 Koos Zevenhoven <k7hoven at gmail.com> wrote:

> On Thu, May 12, 2016 at 1:13 AM, Brett Cannon <brett at python.org> wrote:
> >
> >
> > On Wed, 11 May 2016 at 13:45 Serhiy Storchaka <storchaka at gmail.com>
> wrote:
> >>
> >> On 11.05.16 19:43, Brett Cannon wrote:
> >> > os.path
> >> > '''''''
> >> >
> >> > The various path-manipulation functions of ``os.path`` [#os-path]_
> >> > will be updated to accept path objects. For polymorphic functions that
> >> > accept both bytes and strings, they will be updated to simply use
> >> > code very much similar to
> >> > ``path.__fspath__() if  hasattr(path, '__fspath__') else path``. This
> >> > will allow for their pre-existing type-checking code to continue to
> >> > function.
> >>
> >> I afraid that this will hit a performance. Some os.path functions are
> >> used in tight loops, they are hard optimized, and adding support of path
> >> protocol can have visible negative effect.
> >
> >
> > As others have asked, what specific examples do you have that os.path is
> > used in a tight loop w/o any I/O that would overwhelm the performance?
> >
> >>
> >>
> >> I suggest first implement other changes and then look whether it is
> >> worth to add support of path protocol in os.path functions.
> >
> >
> > I see this whole discussion breaking down into a few groups which changes
> > what gets done upfront and what might be done farther down the line:
> >
> > 1. Maximum acceptance: do whatever we can to make all representation of
> paths
> > just work, which means making all places working with a path in the
> stdlib
> > accept path objects, str, and bytes.
>
> Since you are putting me in this camp, there is at least one thing you
> are wrong about. I don't want all places that work with a path to
> accept bytes. Only those that already do so, including os/os.path. And
> yes, I think the stdlib should show a good example in accepting path
> types (especially those provided in the stdlib itself).
>

That's actually what I meant. I'm not advocating widening the APIs that
accept bytes at all.

-Brett


>
> Whether Ethan is fully in camp 1, I don't know. Not that I think he
> would be any closer to the other camps, though.
>
> > 2. Safely use path objects: __fspath__() is there to signal an object is
> a file
> > system path and to get back a lower-level representation so people stop
> > calling str() on everything, providing some interface signaling that
> someone
> > doesn't misuse an object as a path and only changing path consumptions
> APIs
> > -- e.g. open() -- and not path manipulation APIs -- e.g. os.path -- in
> the
> > stdlib.
> >
> > 3. It ain't worth it: those that would rather just skip all of this and
> drop
> > pathlib from the stdlib.
> >
> > Ethan and Koos are in group #1 and I'm personally in group #2 but I
> tried to
> > compromise somewhat and find a middle ground in the PEP with the level of
> > changes in the stdlib but being more restrictive with os.fspath(). If I
> were
> > doing a pure group #2 PEP I would drop os.path changes and make
> os.fspath()
> > do what Ethan and Koos have suggested and simply pass through without
> checks
> > whatever path.__fspath__() returned if the argument wasn't str or bytes.
> >
>
> Related to this, based on the earlier discussions, I had the
> impression that you were largely in the same camp as me. In fact, I
> thought you had politely left some things out of the PEP draft so I
> could fill them in. It turned out I was wrong about that, because you
> didn't merge them.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160512/ae9e58c3/attachment.html>

From brett at python.org  Wed May 11 21:03:49 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 12 May 2016 01:03:49 +0000
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP1=2W6gFRRArm1j5QX47pDJD-Z_ng85CD79goJYJFzEXQdaUg@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <nh05jk$58e$1@ger.gmane.org>
 <CAP1=2W6gFRRArm1j5QX47pDJD-Z_ng85CD79goJYJFzEXQdaUg@mail.gmail.com>
Message-ID: <CAP1=2W4PUhYK6YTc=6uzc_3BsKpP-9dx-nFe0tJ-oTR+SoYLCw@mail.gmail.com>

On Wed, 11 May 2016 at 15:13 Brett Cannon <brett at python.org> wrote:

> On Wed, 11 May 2016 at 13:45 Serhiy Storchaka <storchaka at gmail.com> wrote:
>
>> On 11.05.16 19:43, Brett Cannon wrote:
>> > os.path
>> > '''''''
>> >
>> > The various path-manipulation functions of ``os.path`` [#os-path]_
>> > will be updated to accept path objects. For polymorphic functions that
>> > accept both bytes and strings, they will be updated to simply use
>> > code very much similar to
>> > ``path.__fspath__() if  hasattr(path, '__fspath__') else path``. This
>> > will allow for their pre-existing type-checking code to continue to
>> > function.
>>
>> I afraid that this will hit a performance. Some os.path functions are
>> used in tight loops, they are hard optimized, and adding support of path
>> protocol can have visible negative effect.
>>
>
> As others have asked, what specific examples do you have that os.path is
> used in a tight loop w/o any I/O that would overwhelm the performance?
>
>
>>
>> I suggest first implement other changes and then look whether it is
>> worth to add support of path protocol in os.path functions.
>>
>
> I see this whole discussion breaking down into a few groups which changes
> what gets done upfront and what might be done farther down the line:
>
>    1. Maximum acceptance: do whatever we can to make all representation
>    of paths just work, which means making all places working with a path in
>    the stdlib accept path objects, str, and bytes.
>    2. Safely use path objects: __fspath__() is there to signal an object
>    is a file system path and to get back a lower-level representation so
>    people stop calling str() on everything, providing some interface signaling
>    that someone doesn't misuse an object as a path and only changing path
>    consumptions APIs -- e.g. open() -- and not path manipulation APIs -- e.g.
>    os.path -- in the stdlib.
>    3. It ain't worth it: those that would rather just skip all of this
>    and drop pathlib from the stdlib.
>
> Ethan and Koos are in group #1 and I'm personally in group #2 but I tried
> to compromise somewhat and find a middle ground in the PEP with the level
> of changes in the stdlib but being more restrictive with os.fspath(). If I
> were doing a pure group #2 PEP I would drop os.path changes and make
> os.fspath() do what Ethan and Koos have suggested and simply pass through
> without checks whatever path.__fspath__() returned if the argument wasn't
> str or bytes.
>

I should mention there's also a side group who really wanted to either
minimize bytes usage in all of this or not have a polymorphic os.fspath(),
hence why that function is defined the way it is around only returning
strings in the PEP.

IOW the PEP is already an attempt at compromise and if we can't find a
consensus somehow I'll update the PEP to directly reflect my personal
preferences and let others write their own competing PEPs if they prefer
(but I think it's in everyone's best interests if that's a last resort).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160512/89a1ce61/attachment-0001.html>

From brett at python.org  Wed May 11 21:05:29 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 12 May 2016 01:05:29 +0000
Subject: [Python-Dev] ctypes compatibility with 2.3
In-Reply-To: <CAP1=2W6dE8bRKe9q+SToX-+cT1qF90Og7f565_=o4vP3nLYsXQ@mail.gmail.com>
References: <CA+eR4cF8--D_eSCo1+oNd=ZKQ46QFyLmF4bYqxni9CMs7H9wDQ@mail.gmail.com>
 <CAP1=2W4YStjMo3r2918k8C4=F7DK99UFW21BdpyHb7pOQWTZWA@mail.gmail.com>
 <813cec64-397a-2ca1-1f53-6bed0886c80b@ctypes.org>
 <CAP1=2W6Qq96BiAEOKLi-x4qvJX88S1-Eo5C1fr+yvC0-7P0O7A@mail.gmail.com>
 <3283bb48-531a-0795-babd-df203115df58@ctypes.org>
 <CAP1=2W6dE8bRKe9q+SToX-+cT1qF90Og7f565_=o4vP3nLYsXQ@mail.gmail.com>
Message-ID: <CAP1=2W7LkhJxCH0Hf9=AZ8rg6USg_1PiAujE=k+g-Z9EiVh75A@mail.gmail.com>

On Wed, 11 May 2016 at 09:28 Brett Cannon <brett at python.org> wrote:

> On Wed, 11 May 2016 at 09:07 Thomas Heller <theller at ctypes.org> wrote:
>
>> Am 11.05.2016 um 18:04 schrieb Brett Cannon:
>> >
>> >
>> > On Wed, 11 May 2016 at 04:35 Thomas Heller <theller at ctypes.org
>> > <mailto:theller at ctypes.org>> wrote:
>> >
>> >     Am 10.05.2016 um 19:39 schrieb Brett Cannon:
>> >     >
>> >     >
>> >     > On Tue, 10 May 2016 at 01:18 Martin Panter <vadmium+py at gmail.com
>> >     <mailto:vadmium%2Bpy at gmail.com>
>> >     > <mailto:vadmium%2Bpy at gmail.com <mailto:vadmium%252Bpy at gmail.com
>> >>>
>> >     wrote:
>> >     >
>> >     >     I am working on <https://bugs.python.org/issue22636>, to fix
>> shell
>> >     >     injection problems with ctypes.util.find_library(). The
>> >     proposal for
>> >     >     Python 3 is to change os.popen(shell-script) calls to use
>> >     >     subprocess.Popen().
>> >     >
>> >     >     However the Python 2.7 version of the module has a comment
>> >     which says
>> >     >     ?This file should be kept compatible with Python 2.3, see PEP
>> >     291.?
>> >     >     Looking at <https://www.python.org/dev/peps/pep-0291/>, it
>> is not
>> >     >     clear why we have to maintain this compatibility. My best
>> guess is
>> >     >     that there may be an external ctypes package that people
>> >     want(ed) to
>> >     >     keep compatible with 2.3, and also keep synchronized with 2.7.
>> >     >
>> >     >
>> >     > That's correct and the maintainer is/was Thomas Heller who I have
>> >     cc'ed
>> >     > to see if he's okay with lifting the restriction.
>> >
>> >     For me it is totally ok to lift this restriction.
>> >
>> >
>> > Great! I'll also update PEP 291.
>>
>> Cool.  While you're at it, the compatibility restriction for
>> modulefinder could also be lifted.
>>
>
> Will do.
>

PEP 291 no longer lists any restrictions on ctypes or modulefinder.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160512/43c83f46/attachment.html>

From vadmium+py at gmail.com  Wed May 11 23:50:53 2016
From: vadmium+py at gmail.com (Martin Panter)
Date: Thu, 12 May 2016 03:50:53 +0000
Subject: [Python-Dev] ctypes compatibility with 2.3
In-Reply-To: <CAP1=2W7LkhJxCH0Hf9=AZ8rg6USg_1PiAujE=k+g-Z9EiVh75A@mail.gmail.com>
References: <CA+eR4cF8--D_eSCo1+oNd=ZKQ46QFyLmF4bYqxni9CMs7H9wDQ@mail.gmail.com>
 <CAP1=2W4YStjMo3r2918k8C4=F7DK99UFW21BdpyHb7pOQWTZWA@mail.gmail.com>
 <813cec64-397a-2ca1-1f53-6bed0886c80b@ctypes.org>
 <CAP1=2W6Qq96BiAEOKLi-x4qvJX88S1-Eo5C1fr+yvC0-7P0O7A@mail.gmail.com>
 <3283bb48-531a-0795-babd-df203115df58@ctypes.org>
 <CAP1=2W6dE8bRKe9q+SToX-+cT1qF90Og7f565_=o4vP3nLYsXQ@mail.gmail.com>
 <CAP1=2W7LkhJxCH0Hf9=AZ8rg6USg_1PiAujE=k+g-Z9EiVh75A@mail.gmail.com>
Message-ID: <CA+eR4cErC2H9nH8h0BZAU5HPaTCRdEVALCfXySwfHVOudsmDvg@mail.gmail.com>

On 12 May 2016 at 01:05, Brett Cannon <brett at python.org> wrote:
>
>
> On Wed, 11 May 2016 at 09:28 Brett Cannon <brett at python.org> wrote:
>>
>> On Wed, 11 May 2016 at 09:07 Thomas Heller <theller at ctypes.org> wrote:
>>>
>>> Am 11.05.2016 um 18:04 schrieb Brett Cannon:
>>> >
>>> >
>>> > On Wed, 11 May 2016 at 04:35 Thomas Heller <theller at ctypes.org
>>> > <mailto:theller at ctypes.org>> wrote:
>>> >     For me it is totally ok to lift this restriction.
>>> >
>>> >
>>> > Great! I'll also update PEP 291.
>>>
>>> Cool.  While you're at it, the compatibility restriction for
>>> modulefinder could also be lifted.
>>
>>
>> Will do.
>
>
> PEP 291 no longer lists any restrictions on ctypes or modulefinder.

Thanks everyone for your responses. I will look at removing the
notices in the code when I get a chance. That would probably involve
reverting

https://hg.python.org/cpython/rev/381a72ab5fb8
And also the modulefinder.py comment

There are also these commits that could be backported
https://hg.python.org/cpython/rev/0980034adaa7 (ctypes)
https://hg.python.org/cpython/diff/627db59031be/Lib/modulefinder.py
but it might be safer just to leave the compatibility code there,
perhaps with a clarifying comment.

From storchaka at gmail.com  Thu May 12 00:32:21 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 12 May 2016 07:32:21 +0300
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <57339B53.3010506@stoneleaf.us>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <nh05jk$58e$1@ger.gmane.org> <57339B53.3010506@stoneleaf.us>
Message-ID: <nh110l$kep$1@ger.gmane.org>

On 11.05.16 23:51, Ethan Furman wrote:
> On 05/11/2016 01:44 PM, Serhiy Storchaka wrote:
>> I afraid that this will hit a performance. Some os.path functions are
>> used in tight loops, they are hard optimized, and adding support of path
>> protocol can have visible negative effect.
>
> Do you have an example of os.path functions being used in a tight loop?

posixpath.realpath(), os.walk(), glob.glob() calls split() and join() 
for every path component. dirname() and basename() are also often 
called. I doesn't count functions like islink() and isfile() since they 
just pass the argument to underlying stat function and don't need 
conversion.



From storchaka at gmail.com  Thu May 12 00:55:59 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 12 May 2016 07:55:59 +0300
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP1=2W6gFRRArm1j5QX47pDJD-Z_ng85CD79goJYJFzEXQdaUg@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <nh05jk$58e$1@ger.gmane.org>
 <CAP1=2W6gFRRArm1j5QX47pDJD-Z_ng85CD79goJYJFzEXQdaUg@mail.gmail.com>
Message-ID: <nh12d0$7df$1@ger.gmane.org>

On 12.05.16 01:13, Brett Cannon wrote:
>
>
> On Wed, 11 May 2016 at 13:45 Serhiy Storchaka <storchaka at gmail.com
> <mailto:storchaka at gmail.com>> wrote:
>
>     On 11.05.16 19:43, Brett Cannon wrote:
>      > os.path
>      > '''''''
>      >
>      > The various path-manipulation functions of ``os.path`` [#os-path]_
>      > will be updated to accept path objects. For polymorphic functions
>     that
>      > accept both bytes and strings, they will be updated to simply use
>      > code very much similar to
>      > ``path.__fspath__() if  hasattr(path, '__fspath__') else path``. This
>      > will allow for their pre-existing type-checking code to continue to
>      > function.
>
>     I afraid that this will hit a performance. Some os.path functions are
>     used in tight loops, they are hard optimized, and adding support of path
>     protocol can have visible negative effect.
>
>
> As others have asked, what specific examples do you have that os.path is
> used in a tight loop w/o any I/O that would overwhelm the performance?

Most examples does some I/O (like os.lstat()): posixpath.realpath(), 
os.walk(), glob.glob(). But for example os.walk() was significantly 
boosted with using os.scandir(), it would be sad to make it slower 
again. os.path is used in number of files, sometimes in loops, sometimes 
indirectly. It is hard to find all examples.

Such functions as glob.glob() calls split() and join() for every 
component, but they also use string or bytes operations with paths. So 
they need to convert argument to str or bytes before start iteration, 
and always call os.path functions only with str or bytes. Additional 
conversion in every os.path function is redundant. I suppose most other 
high-level functions that manipulates paths in a loop also should 
convert arguments once at the start and don't need the support of path 
protocol in os.path functions.

> I see this whole discussion breaking down into a few groups which
> changes what gets done upfront and what might be done farther down the line:
>
>  1. Maximum acceptance: do whatever we can to make all representation of
>     paths just work, which means making all places working with a path
>     in the stdlib accept path objects, str, and bytes.
>  2. Safely use path objects: __fspath__() is there to signal an object
>     is a file system path and to get back a lower-level representation
>     so people stop calling str() on everything, providing some interface
>     signaling that someone doesn't misuse an object as a path and only
>     changing path consumptions APIs -- e.g. open() -- and not path
>     manipulation APIs -- e.g. os.path -- in the stdlib.
>  3. It ain't worth it: those that would rather just skip all of this and
>     drop pathlib from the stdlib.
>
> Ethan and Koos are in group #1 and I'm personally in group #2 but I
> tried to compromise somewhat and find a middle ground in the PEP with
> the level of changes in the stdlib but being more restrictive with
> os.fspath(). If I were doing a pure group #2 PEP I would drop os.path
> changes and make os.fspath() do what Ethan and Koos have suggested and
> simply pass through without checks whatever path.__fspath__() returned
> if the argument wasn't str or bytes.

I'm for adding conversions in C implemented path consuming APIs and may 
be in high-level path manipulation functions like os.walk(), but left 
low-level API of os.path, fnmatch and glob unchanged.


From g.brandl at gmx.net  Thu May 12 01:30:49 2016
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 12 May 2016 07:30:49 +0200
Subject: [Python-Dev] PEP 515: Underscores in Numeric Literals (revision
 3)
In-Reply-To: <CAP7+vJ+BicSaZiizSpjE8q2g=OGHMLgOre1tLwo3o5DN2crZ3g@mail.gmail.com>
References: <n9mqlh$oeb$1@ger.gmane.org>
 <CAP1=2W4P7skRpewpgxXBgHHicngMU3mJ2V1s3NVXgQL9PiCJmw@mail.gmail.com>
 <CAP7+vJLUumJiOzExqdmffJG_ywpFA9vXAWUH2Bpzvpiwcwp_RQ@mail.gmail.com>
 <ncisha$auc$1@ger.gmane.org>
 <CADiSq7etYj9Wy4++kP0z_5=-x6Pm0E9=6VASh+nYfJk8bM6PfQ@mail.gmail.com>
 <CAP7+vJ+e60gid=R8gjhmyiiNkCMC0Db4zt-Bct_qKUN29Vbr_Q@mail.gmail.com>
 <loom.20160319T181628-51@post.gmane.org>
 <CAP7+vJ+PewZ=ojQU0LzsZh0n0payOgzdOzfx6zQ73Xh9OYt5aQ@mail.gmail.com>
 <loom.20160319T191115-310@post.gmane.org>
 <CAP7+vJ+F=d9ycxZhaWkoYB6SYXf82T=Lf4QxWubwNy_FKJUACw@mail.gmail.com>
 <CAP1=2W4SM-_SBUo5YGdwyvq2N62+FC3TEgLnmYMW=2VDkV6+2A@mail.gmail.com>
 <CAP7+vJ+BicSaZiizSpjE8q2g=OGHMLgOre1tLwo3o5DN2crZ3g@mail.gmail.com>
Message-ID: <nh14ea$5aa$1@ger.gmane.org>

I'm happy with the latest version.

Georg

On 05/11/2016 06:46 PM, Guido van Rossum wrote:
> If the authors are happy I'll accept it right away.
> 
> (I vaguely recall there's another PEP that's ready for pronouncement -- but
> which one?)
> 
> On Wed, May 11, 2016 at 9:34 AM, Brett Cannon <brett at python.org
> <mailto:brett at python.org>> wrote:
> 
>     Is there anything holding up PEP 515 at this point in terms of acceptance or
>     implementation?
> 
>     On Sat, 19 Mar 2016 at 11:56 Guido van Rossum <guido at python.org
>     <mailto:guido at python.org>> wrote:
> 
>         All that sounds fine!
> 
>         On Sat, Mar 19, 2016 at 11:28 AM, Stefan Krah <stefan at bytereef.org
>         <mailto:stefan at bytereef.org>> wrote:
>         > Guido van Rossum <guido <at> python.org <http://python.org>> writes:
>         >> So should the preprocessing step just be s.replace('_', ''), or should
>         >> it reject underscores that don't follow the rules from the PEP
>         >> (perhaps augmented so they follow the spirit of the PEP and the letter
>         >> of the IBM spec)?
>         >>
>         >> Honestly I think it's also fine if specifying this exactly is left out
>         >> of the PEP, and handled by whoever adds this to Decimal. Having a PEP
>         >> to work from for the language spec and core builtins (int(), float()
>         >> complex()) is more important.
>         >
>         > I'd keep it simple for Decimal: Remove left and right whitespace (we're
>         > already doing this), then remove underscores from the remaining string
>         > (which must not contain any further whitespace), then use the IBM grammar.
>         >
>         >
>         > We could add a clause to the PEP that only those strings that follow
>         > the spirit of the PEP are guaranteed to be accepted in the future.
>         >
>         >
>         > One reason for keeping it simple is that I would not like to slow down
>         > string conversion, but thinking about two grammars is also a problem --
>         > part of the string conversion in libmpdec is modeled in ACL2, which
>         > would be invalidated or at least complicated with two grammars.
>         >
>         >
>         >
>         > Stefan Krah
>         >
>         > _______________________________________________
>         > Python-Dev mailing list
>         > Python-Dev at python.org <mailto:Python-Dev at python.org>
>         > https://mail.python.org/mailman/listinfo/python-dev
>         > Unsubscribe:
>         https://mail.python.org/mailman/options/python-dev/guido%40python.org
> 
> 
> 
>         --
>         --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
>         _______________________________________________
>         Python-Dev mailing list
>         Python-Dev at python.org <mailto:Python-Dev at python.org>
>         https://mail.python.org/mailman/listinfo/python-dev
>         Unsubscribe:
>         https://mail.python.org/mailman/options/python-dev/brett%40python.org
> 
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
> 
> 



From ethan at stoneleaf.us  Thu May 12 03:54:14 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 12 May 2016 00:54:14 -0700
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <nh12d0$7df$1@ger.gmane.org>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <nh05jk$58e$1@ger.gmane.org>
 <CAP1=2W6gFRRArm1j5QX47pDJD-Z_ng85CD79goJYJFzEXQdaUg@mail.gmail.com>
 <nh12d0$7df$1@ger.gmane.org>
Message-ID: <573436A6.3000509@stoneleaf.us>

On 05/11/2016 09:55 PM, Serhiy Storchaka wrote:

> [...]  But for example os.walk() was significantly
> boosted with using os.scandir(), it would be sad to make it slower
> again.

scandir's speed improvement is due to not not throwing away data the OS 
was already giving us.

> os.path is used in number of files, sometimes in loops, sometimes
> indirectly. It is hard to find all examples.

Currently, any of these functions that already take a string have to do 
a couple pointer comparisons to make sure they have a string; any of 
these functions that take both a string and a bytes have to do a couple 
pointer comparisons to make sure they have a string or a bytes;  the 
only difference if this PEP is accepted is the fall-back path when those 
first checks fail.

As an example: if os.walk is called with a Path, it converts the Path to 
a string (once!) and then uses that string to generate more strings and 
return strings.  When os.walk calls os.path.join or os.path.split it 
will be with strings (or bytes), not with the original Path object.

> Such functions as glob.glob() calls split() and join() for every
> component, but they also use string or bytes operations with paths. So
> they need to convert argument to str or bytes before start iteration,
> and always call os.path functions only with str or bytes.

Exactly.

> Additional
> conversion in every os.path function is redundant.

And won't happen, since the fast-path checks will confirm that the 
argument is a string or bytes object.

> I suppose most other
> high-level functions that manipulates paths in a loop also should
> convert arguments once at the start and don't need the support of path
> protocol in os.path functions.

Human's can call the os.path functions; therefore, the os.path functions 
need to support the __fspath__ protocol.

> I'm for adding conversions in C implemented path consuming APIs and may
> be in high-level path manipulation functions like os.walk(), but left
> low-level API of os.path, fnmatch and glob unchanged.

So instead of the easy to remember "Path doesn't work with the rest of 
the standard library" we'll have "Path works with some APIs, but not 
others -- guess you better look it up" ?  That is not an improvement.

--
~Ethan~

From srkunze at mail.de  Thu May 12 04:04:42 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Thu, 12 May 2016 10:04:42 +0200
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP1=2W6gFRRArm1j5QX47pDJD-Z_ng85CD79goJYJFzEXQdaUg@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <nh05jk$58e$1@ger.gmane.org>
 <CAP1=2W6gFRRArm1j5QX47pDJD-Z_ng85CD79goJYJFzEXQdaUg@mail.gmail.com>
Message-ID: <5734391A.702@mail.de>

On 12.05.2016 00:13, Brett Cannon wrote:
> I see this whole discussion breaking down into a few groups which 
> changes what gets done upfront and what might be done farther down the 
> line:
>
>  1. Maximum acceptance: do whatever we can to make all representation
>     of paths just work, which means making all places working with a
>     path in the stdlib accept path objects, str, and bytes.
>  2. Safely use path objects: __fspath__() is there to signal an object
>     is a file system path and to get back a lower-level representation
>     so people stop calling str() on everything, providing some
>     interface signaling that someone doesn't misuse an object as a
>     path and only changing path consumptions APIs -- e.g. open() --
>     and not path manipulation APIs -- e.g. os.path -- in the stdlib.
>  3. It ain't worth it: those that would rather just skip all of this
>     and drop pathlib from the stdlib.
>

Sorry for being picky here. I think the last group needs to be split up:

3. It ain't worth it: those that would rather just skip all of this
4. drop pathlib from the stdlib.

I put myself into camp 3, mostly because I don't consider the "wallet 
garden problem" a problem at all and I realized that our past issues 
with pathlib resulted from missing features in pathlib not in the rest 
of the stdlib.


Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160512/877fa176/attachment.html>

From storchaka at gmail.com  Thu May 12 04:14:38 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 12 May 2016 11:14:38 +0300
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <573436A6.3000509@stoneleaf.us>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <nh05jk$58e$1@ger.gmane.org>
 <CAP1=2W6gFRRArm1j5QX47pDJD-Z_ng85CD79goJYJFzEXQdaUg@mail.gmail.com>
 <nh12d0$7df$1@ger.gmane.org> <573436A6.3000509@stoneleaf.us>
Message-ID: <nh1e1f$ncp$1@ger.gmane.org>

On 12.05.16 10:54, Ethan Furman wrote:
> Currently, any of these functions that already take a string have to do
> a couple pointer comparisons to make sure they have a string; any of
> these functions that take both a string and a bytes have to do a couple
> pointer comparisons to make sure they have a string or a bytes;  the
> only difference if this PEP is accepted is the fall-back path when those
> first checks fail.

This is cheap in C, but os.path functions are implemented in Python. 
They have to make at least one function call (os.fspath(), hasattr() or 
isinstance()), not counting a bytecode for retrieving arguments, 
resolving attributes, comparing, jumps. Currently os.path functions use 
tricks to avoid overheads

Yet one problem is that currently many os,path functions work with 
duck-typed strings (e.g. UserString). Using os.fspath() likely limit 
supported types to str, bytes and types that support the path protocol.



From srkunze at mail.de  Thu May 12 04:31:18 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Thu, 12 May 2016 10:31:18 +0200
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
Message-ID: <57343F56.6090609@mail.de>

Thanks Brett for your hard work. My comments below:

On 11.05.2016 18:43, Brett Cannon wrote:
> Rationale
> =========
>
> Historically in Python, file system paths have been represented as
> strings or bytes. This choice of representation has stemmed from C's
> own decision to represent file system paths as
> ``const char *`` [#libc-open]_. While that is a totally serviceable
> format to use for file system paths, it's not necessarily optimal. At
> issue is the fact that while all file system paths can be represented
> as strings or bytes, not all strings or bytes represent a file system
> path.

I can remember this argument being made during the discussion. I am not 
sure if that 100% correct as soon as we talk about PurePaths.

> This can lead to issues where any e.g. string duck-types to a
> file system path whether it actually represents a path or not.
>
> To help elevate the representation of file system paths from their
> representation as strings and bytes to a more appropriate object
> representation, the pathlib module [#pathlib]_ was provisionally
> introduced in Python 3.4 through PEP 428. While considered by some as
> an improvement over strings and bytes for file system paths, it has
> suffered from a lack of adoption. Typically the key issue listed
> for the low adoption rate has been the lack of support in the standard
> library. This lack of support required users of pathlib to manually
> convert path objects to strings by calling ``str(path)`` which many
> found error-prone.
>
> One issue in converting path objects to strings comes from
> the fact that only generic way to get a string representation of the
> path was to pass the object to ``str()``. This can pose a
> problem when done blindly as nearly all Python objects have some
> string representation whether they are a path or not, e.g.
> ``str(None)`` will give a result that
> ``builtins.open()`` [#builtins-open]_ will happily use to create a new
> file.
>
> Exacerbating this whole situation is the
> ``DirEntry`` object [#os-direntry]_. While path objects have a
> representation that can be extracted using ``str()``, ``DirEntry``
> objects expose a ``path`` attribute instead. Having no common
> interface between path objects, ``DirEntry``, and any other
> third-party path library had become an issue. A solution that allowed
> any path-representing object to declare that is was a path and a way
> to extract a low-level representation that all path objects could
> support was desired.

I think the "Rationale" section ignores the fact the Path also supports 
the .path attribute now. Which indeed defines a common interface between 
path objects.

>
> [...]
>
> Proposal
> ========
>
> This proposal is split into two parts. One part is the proposal of a
> protocol for objects to declare and provide support for exposing a
> file system path representation.

https://docs.python.org/3/whatsnew/changelog.html says:

"Add ?path? attribute to pathlib.Path objects, returning the same as 
str(), to make it more similar to DirEntry. Library code can now write 
getattr(p, ?path?, p) to get the path as a string from a Path, a 
DirEntry, or a plain string. This is essentially a small one-off protocol."

So, in order to promote the "small one-off protocol" to a more broader 
protocol, this PEP proposes a simple rename of .path to .__fspath__, is 
that correct?


The only issue I see with it is that it requires another function 
(os.fspath) to extract the "low-level representation". .path seems far 
easier to me.

> The other part is changes to Python's
> standard library to support the new protocol.

I think this could be another PEP unrelated to the first part.

> These changes will also have the pathlib module drop its provisional 
> status.

Not sure if that should be part of the PEP, maybe yes.

> [...]

The remainder of the PEP unfolds as a flawless implication of the 
rationale and the proposed idea.

Unfortunately, I don't have anything to contribute to the open issues. 
All solutions have their pros and cons and everything that could be said 
has been said. I think you need to decide.

Sven


From k7hoven at gmail.com  Thu May 12 05:52:44 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Thu, 12 May 2016 12:52:44 +0300
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <nh1e1f$ncp$1@ger.gmane.org>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <nh05jk$58e$1@ger.gmane.org>
 <CAP1=2W6gFRRArm1j5QX47pDJD-Z_ng85CD79goJYJFzEXQdaUg@mail.gmail.com>
 <nh12d0$7df$1@ger.gmane.org> <573436A6.3000509@stoneleaf.us>
 <nh1e1f$ncp$1@ger.gmane.org>
Message-ID: <CAMiohois6FpZ=ReEXeEJg7a_1FBgVzyr_rJPtfCC=TF-17ob_w@mail.gmail.com>

On Thu, May 12, 2016 at 11:14 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
>
> This is cheap in C, but os.path functions are implemented in Python. They
> have to make at least one function call (os.fspath(), hasattr() or
> isinstance()), not counting a bytecode for retrieving arguments, resolving
> attributes, comparing, jumps. Currently os.path functions use tricks to
> avoid overheads
>

I suppose a C-implemented version of fspath *called from python* might
be the fastest option at least in some cases. After all, a function
call (isinstance or hasattr) is likely anyway, unless of course `try:
path.__fspath__` is used.

> Yet one problem is that currently many os,path functions work with
> duck-typed strings (e.g. UserString). Using os.fspath() likely limit
> supported types to str, bytes and types that support the path protocol.
>

Something like

path = path.__fspath__() if hasattr(path, '__fspath__') else path

as currently in the PEP, would not have this problem. However, I
wonder whether such duck string paths actually exist (although it does
remind me of my earlier experiments for solving the pathlib
compatibility problem ;-).


-- Koos

P.S: I think it's great that you are concerned about the performance,
and I find it important. However, this does feel like premature
optimization to me at this point. We should first decide what
functions should support the protocol, and after that, find the
performance concerns and fix them. I have the feeling that the cases
where this would be a performance bottleneck are quite rare, but if
they are not, they may well be fixable.

From k7hoven at gmail.com  Thu May 12 06:12:58 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Thu, 12 May 2016 13:12:58 +0300
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <57343F56.6090609@mail.de>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de>
Message-ID: <CAMiohojnwS32ny0Kis0dtxMBZ8_BP_6im_aUcZDBSxZYKBgQhw@mail.gmail.com>

On Thu, May 12, 2016 at 11:31 AM, Sven R. Kunze <srkunze at mail.de> wrote:
> On 11.05.2016 18:43, Brett Cannon wrote:
>>
>> Rationale
>> =========
>>
>> Historically in Python, file system paths have been represented as
>> strings or bytes. This choice of representation has stemmed from C's
>> own decision to represent file system paths as
>> ``const char *`` [#libc-open]_. While that is a totally serviceable
>> format to use for file system paths, it's not necessarily optimal. At
>> issue is the fact that while all file system paths can be represented
>> as strings or bytes, not all strings or bytes represent a file system
>> path.
>
>
> I can remember this argument being made during the discussion. I am not sure
> if that 100% correct as soon as we talk about PurePaths.
>

I had suggested an alternative wording for this (see my commit on the
work on Rationale).


>> Proposal
>> ========
>>
>> This proposal is split into two parts. One part is the proposal of a
>> protocol for objects to declare and provide support for exposing a
>> file system path representation.
>
>
> https://docs.python.org/3/whatsnew/changelog.html says:
>
> "Add ?path? attribute to pathlib.Path objects, returning the same as str(),
> to make it more similar to DirEntry. Library code can now write getattr(p,
> ?path?, p) to get the path as a string from a Path, a DirEntry, or a plain
> string. This is essentially a small one-off protocol."
>
> So, in order to promote the "small one-off protocol" to a more broader
> protocol, this PEP proposes a simple rename of .path to .__fspath__, is that
> correct?
>

Well, I have brought this up previously several times. Indeed I see
this as a further development of that duck-typing compatiblity
approach.  However, while the .path attribute is prior art, it has not
been in a release yet.

> Unfortunately, I don't have anything to contribute to the open issues. All
> solutions have their pros and cons and everything that could be said has
> been said. I think you need to decide.
>

Surprising enough, there are new things being said all the time. But
luckily there seem to be signs of convergence.

-- Koos

From ncoghlan at gmail.com  Thu May 12 08:04:02 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 12 May 2016 22:04:02 +1000
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
Message-ID: <CADiSq7cFSyGWH=idtk0u5NCA5A1SMyJ=gOTMO6+G320oGVqwdg@mail.gmail.com>

On 12 May 2016 at 02:43, Brett Cannon <brett at python.org> wrote:
> **deep, calming breath**
>
> Here is the PEP for __fspath__(). The draft lives at
> https://github.com/brettcannon/path-pep so feel free to send me PRs for
> spelling mistakes, grammatical errors, etc.

Thanks for putting this together :)

> C API
> '''''
>
> The C API will gain an equivalent function to ``os.fspath()`` that
> also allows bytes objects through::
>
>     /*
>         Return the file system path of the object.
>
>         If the object is str or bytes, then allow it to pass through with
>         an incremented refcount. All other types raise a TypeError.
>     */
>     PyObject *
>     PyOS_RawFSPath(PyObject *path)
>     {
>         if (PyObject_HasAttrString(path, "__fspath__")) {
>             path = PyObject_CallMethodObjArgs(path, "__fspath__", NULL);
>             if (path == NULL) {
>                 return NULL;
>             }
>         }
>         else {
>             Py_INCREF(path);
>         }
>
>         if (!PyUnicode_Check(path) && !PyBytes_Check(path)) {
>             Py_DECREF(path);
>             return PyErr_Format(PyExc_TypeError,
>                                 "expected a string, bytes, or path object,
> not %S",
>                                 path->ob_type);
>         }
>
>         return path;
> }

I'd still like to see this exposed to Python code as os._raw_fspath()
(with the leading underscore just meaning "this probably isn't the API
you want" rather than indicating a private or unstable API), and then
fspath() defined as a wrapper around it which disallows bytes as
output.

However, I don't have a specific use case, and it would be
straightforward to add later, so the overall PEP gets a +1 from me.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From k7hoven at gmail.com  Thu May 12 08:40:21 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Thu, 12 May 2016 15:40:21 +0300
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CADiSq7cFSyGWH=idtk0u5NCA5A1SMyJ=gOTMO6+G320oGVqwdg@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <CADiSq7cFSyGWH=idtk0u5NCA5A1SMyJ=gOTMO6+G320oGVqwdg@mail.gmail.com>
Message-ID: <CAMiohojVHWPoU5axuA29UQHL1pRakFbzDoeGYn==c0CedpNDyw@mail.gmail.com>

On Thu, May 12, 2016 at 3:04 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> I'd still like to see this exposed to Python code as os._raw_fspath()
> (with the leading underscore just meaning "this probably isn't the API
> you want" rather than indicating a private or unstable API), and then
> fspath() defined as a wrapper around it which disallows bytes as
> output.
>

I don't remember (should probably check) if you previously proposed
implementing exactly that in C, but I indeed agree with what you write
above, except that I don't like the "_raw_" prefix in the name. I
would be willing to call that simply fspath though, since as mentioned
before in this thread (I think by Brett and me), the reasons for
rejecting bytes in fspath are really quite minor.

From ncoghlan at gmail.com  Thu May 12 09:20:25 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 12 May 2016 23:20:25 +1000
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAMiohojVHWPoU5axuA29UQHL1pRakFbzDoeGYn==c0CedpNDyw@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <CADiSq7cFSyGWH=idtk0u5NCA5A1SMyJ=gOTMO6+G320oGVqwdg@mail.gmail.com>
 <CAMiohojVHWPoU5axuA29UQHL1pRakFbzDoeGYn==c0CedpNDyw@mail.gmail.com>
Message-ID: <CADiSq7caoy9+Xhb9CQ4Rnv0pSwqje2+4jGGXnaDHpXGH71+JBw@mail.gmail.com>

On 12 May 2016 at 22:40, Koos Zevenhoven <k7hoven at gmail.com> wrote:
> On Thu, May 12, 2016 at 3:04 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> I'd still like to see this exposed to Python code as os._raw_fspath()
>> (with the leading underscore just meaning "this probably isn't the API
>> you want" rather than indicating a private or unstable API), and then
>> fspath() defined as a wrapper around it which disallows bytes as
>> output.
>
> I don't remember (should probably check) if you previously proposed
> implementing exactly that in C, but I indeed agree with what you write
> above, except that I don't like the "_raw_" prefix in the name. I
> would be willing to call that simply fspath though, since as mentioned
> before in this thread (I think by Brett and me), the reasons for
> rejecting bytes in fspath are really quite minor.

It's not unusual for me to encounter "POSIX oughtta be enough for
anyone" folks that are not yet entirely convinced that
bytes-are-not-text, so I'm actually in favour of making the default
Python-level API str-only as a healthy nudge away from the
"text-is-just-bytes-with-an-encoding!" school of thought.

However, in terms of the three groups Brett articulated (maximum
flexibility, encouraging cross-platform correctness, and forgetting
the whole idea), I'm in both camps 1 & 2 - I work with POSIX enough
that I'm entirely on board with the notion that if you're specifically
modelling *POSIX* paths, then bytes-with-an-assumed-encoding is
frequently a good enough representation, but also deal with other
environments (like Windows, the JVM and the CLR) enough to know that
that particular representation of filesystem paths breaks down the
moment you expand your scope of interest beyond *nix platforms.

Hence the suggestion of having os.fspath() be the group 2
guaranteed-to-only-emit-str API, with os._raw_fspath() as the lower
level "I know I'm potentially being POSIX-centric here, but that's OK
for my use case" group 1 interface.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From arigo at tunes.org  Thu May 12 09:39:11 2016
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 12 May 2016 15:39:11 +0200
Subject: [Python-Dev] PyGC_Collect ignores state of `enabled`
In-Reply-To: <DDA40D7F-F070-42A0-AFFF-B03828894A68@langa.pl>
References: <DDA40D7F-F070-42A0-AFFF-B03828894A68@langa.pl>
Message-ID: <CAMSv6X1Y2ZSn53KpCHEBAZr_VtkpDXbqXE2_UB8NT8TrzGxpLA@mail.gmail.com>

Hi Lukasz,

On 10 May 2016 at 04:13, ?ukasz Langa <lukasz at langa.pl> wrote:
> However, because of PyGC_Collect() called in Py_Finalize(), during
> interpreter shutdown the collection is done anyway, Linux does CoW and the
> memory usage spikes. Which is ironic on process shutdown.

Try to call os._exit() to avoid doing all this work on shutdown (after
you have checked that it is indeed not doing anything interesting).


A bient?t,

Armin.

From k7hoven at gmail.com  Thu May 12 09:41:26 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Thu, 12 May 2016 16:41:26 +0300
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CADiSq7caoy9+Xhb9CQ4Rnv0pSwqje2+4jGGXnaDHpXGH71+JBw@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <CADiSq7cFSyGWH=idtk0u5NCA5A1SMyJ=gOTMO6+G320oGVqwdg@mail.gmail.com>
 <CAMiohojVHWPoU5axuA29UQHL1pRakFbzDoeGYn==c0CedpNDyw@mail.gmail.com>
 <CADiSq7caoy9+Xhb9CQ4Rnv0pSwqje2+4jGGXnaDHpXGH71+JBw@mail.gmail.com>
Message-ID: <CAMiohoiGnzt+c+QWnqP3HKqw6VJ+nN=Fh5uFV=MwNmn0MoPwFA@mail.gmail.com>

On Thu, May 12, 2016 at 4:20 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> It's not unusual for me to encounter "POSIX oughtta be enough for
> anyone" folks that are not yet entirely convinced that
> bytes-are-not-text, so I'm actually in favour of making the default
> Python-level API str-only as a healthy nudge away from the
> "text-is-just-bytes-with-an-encoding!" school of thought.
>

This was also how I convinced myself about the default str constraint.
However, I'm afraid it would be a weak weapon against using bytes
paths, since the people using bytes paths would not be likely to call
it, regardless of whether it supports bytes or not.

The nice thing is that pathlib is str-only and *that* will push people
away from bytes paths.

> However, in terms of the three groups Brett articulated (maximum
> flexibility, encouraging cross-platform correctness, and forgetting
> the whole idea), I'm in both camps 1 & 2 - I work with POSIX enough
> that I'm entirely on board with the notion that if you're specifically
> modelling *POSIX* paths, then bytes-with-an-assumed-encoding is
> frequently a good enough representation, but also deal with other
> environments (like Windows, the JVM and the CLR) enough to know that
> that particular representation of filesystem paths breaks down the
> moment you expand your scope of interest beyond *nix platforms.
>

I also agree with parts about Brett's "camp 2".

-- Koos

From dimaqq at gmail.com  Thu May 12 10:45:34 2016
From: dimaqq at gmail.com (Dima Tisnek)
Date: Thu, 12 May 2016 16:45:34 +0200
Subject: [Python-Dev] Slow downloads from python.org
In-Reply-To: <CAP1=2W5hW0jus3oGBcROXuGVePTGMr_8q5f2b7a-Ma5YZDT6AA@mail.gmail.com>
References: <CAGGBzXJ80M=Coq_zig6w_rBqzX82JWch-sBUBm9CH+Fx6Vok7A@mail.gmail.com>
 <CAP1=2W5hW0jus3oGBcROXuGVePTGMr_8q5f2b7a-Ma5YZDT6AA@mail.gmail.com>
Message-ID: <CAGGBzXLCcApWv-QOJ2ihviVqRzqw0uXXNXYf5dDGbj3GkkEDgA@mail.gmail.com>

Gone now, must've been transient.
I've no idea if it was python end, my end (tail?) or something
slithery inbetween.

On 11 May 2016 at 20:17, Brett Cannon <brett at python.org> wrote:
>
>
> On Wed, 11 May 2016 at 10:56 Dima Tisnek <dimaqq at gmail.com> wrote:
>>
>> Sorry, this is probably wrong place to ask, but is it only me?
>> I can't get more than 40KB/s downloading from python.org
>
>
> It's just you or the problem has passed; just downloaded much faster than
> 40KB/s.

From gvanrossum at gmail.com  Thu May 12 11:11:37 2016
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Thu, 12 May 2016 08:11:37 -0700
Subject: [Python-Dev] PEP 515: Underscores in Numeric Literals (revision
 3)
In-Reply-To: <CAP7+vJ+qAjuVrsrigS-vjouTCS5s4J6d2aDLjmX04MHenMev6g@mail.gmail.com>
References: <n9mqlh$oeb$1@ger.gmane.org>
 <CAP1=2W4P7skRpewpgxXBgHHicngMU3mJ2V1s3NVXgQL9PiCJmw@mail.gmail.com>
 <CAP7+vJLUumJiOzExqdmffJG_ywpFA9vXAWUH2Bpzvpiwcwp_RQ@mail.gmail.com>
 <ncisha$auc$1@ger.gmane.org>
 <CADiSq7etYj9Wy4++kP0z_5=-x6Pm0E9=6VASh+nYfJk8bM6PfQ@mail.gmail.com>
 <CAP7+vJ+e60gid=R8gjhmyiiNkCMC0Db4zt-Bct_qKUN29Vbr_Q@mail.gmail.com>
 <loom.20160319T181628-51@post.gmane.org>
 <CAP7+vJ+PewZ=ojQU0LzsZh0n0payOgzdOzfx6zQ73Xh9OYt5aQ@mail.gmail.com>
 <loom.20160319T191115-310@post.gmane.org>
 <CAP7+vJ+F=d9ycxZhaWkoYB6SYXf82T=Lf4QxWubwNy_FKJUACw@mail.gmail.com>
 <CAP1=2W4SM-_SBUo5YGdwyvq2N62+FC3TEgLnmYMW=2VDkV6+2A@mail.gmail.com>
 <CAP7+vJ+BicSaZiizSpjE8q2g=OGHMLgOre1tLwo3o5DN2crZ3g@mail.gmail.com>
 <nh14ea$5aa$1@ger.gmane.org>
 <CAP7+vJ+qAjuVrsrigS-vjouTCS5s4J6d2aDLjmX04MHenMev6g@mail.gmail.com>
Message-ID: <CAP7+vJLd5wBRD1YwGjQuM6kTpnai9YdiLAvcHSM3yZ1vdr55Hw@mail.gmail.com>

OK, then PEP 515 is now officially accepted! Congratulations. Start the
implementation work!

--Guido (mobile)
On May 11, 2016 10:33 PM, "Georg Brandl" <g.brandl at gmx.net> wrote:

I'm happy with the latest version.

Georg

On 05/11/2016 06:46 PM, Guido van Rossum wrote:
> If the authors are happy I'll accept it right away.
>
> (I vaguely recall there's another PEP that's ready for pronouncement --
but
> which one?)
>
> On Wed, May 11, 2016 at 9:34 AM, Brett Cannon <brett at python.org
> <mailto:brett at python.org>> wrote:
>
>     Is there anything holding up PEP 515 at this point in terms of
acceptance or
>     implementation?
>
>     On Sat, 19 Mar 2016 at 11:56 Guido van Rossum <guido at python.org
>     <mailto:guido at python.org>> wrote:
>
>         All that sounds fine!
>
>         On Sat, Mar 19, 2016 at 11:28 AM, Stefan Krah <stefan at bytereef.org
>         <mailto:stefan at bytereef.org>> wrote:
>         > Guido van Rossum <guido <at> python.org <http://python.org>>
writes:
>         >> So should the preprocessing step just be s.replace('_', ''),
or should
>         >> it reject underscores that don't follow the rules from the PEP
>         >> (perhaps augmented so they follow the spirit of the PEP and
the letter
>         >> of the IBM spec)?
>         >>
>         >> Honestly I think it's also fine if specifying this exactly is
left out
>         >> of the PEP, and handled by whoever adds this to Decimal.
Having a PEP
>         >> to work from for the language spec and core builtins (int(),
float()
>         >> complex()) is more important.
>         >
>         > I'd keep it simple for Decimal: Remove left and right
whitespace (we're
>         > already doing this), then remove underscores from the remaining
string
>         > (which must not contain any further whitespace), then use the
IBM grammar.
>         >
>         >
>         > We could add a clause to the PEP that only those strings that
follow
>         > the spirit of the PEP are guaranteed to be accepted in the
future.
>         >
>         >
>         > One reason for keeping it simple is that I would not like to
slow down
>         > string conversion, but thinking about two grammars is also a
problem --
>         > part of the string conversion in libmpdec is modeled in ACL2,
which
>         > would be invalidated or at least complicated with two grammars.
>         >
>         >
>         >
>         > Stefan Krah
>         >
>         > _______________________________________________
>         > Python-Dev mailing list
>         > Python-Dev at python.org <mailto:Python-Dev at python.org>
>         > https://mail.python.org/mailman/listinfo/python-dev
>         > Unsubscribe:
>
https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
>         --
>         --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
>         _______________________________________________
>         Python-Dev mailing list
>         Python-Dev at python.org <mailto:Python-Dev at python.org>
>         https://mail.python.org/mailman/listinfo/python-dev
>         Unsubscribe:
>
https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
>
>
>
> --
> --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
>
>


_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/guido%40python.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160512/9c7a6739/attachment-0001.html>

From ethan at stoneleaf.us  Thu May 12 11:42:36 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 12 May 2016 08:42:36 -0700
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <57343F56.6090609@mail.de>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de>
Message-ID: <5734A46C.40306@stoneleaf.us>

On 05/12/2016 01:31 AM, Sven R. Kunze wrote:

> I think the "Rationale" section ignores the fact the Path also supports
> the .path attribute now. Which indeed defines a common interface between
> path objects.

The version of Python that has Path.path has not been released yet.  And 
even so, .path is not a "common interface" as neither str nor bytes have 
it, and they also are used as path objects.

And even given all that, for smoother interoperability with the rest of 
the stdlib, or at least the os.* portion, those functions would still 
need to be upgraded to check for .path on the incoming arguments -- at 
which point we may as well make a protocol to properly support file 
system paths instead of relying on the rather generic attribute name of 
'path'.

--
~Ethan~

From srkunze at mail.de  Thu May 12 12:13:08 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Thu, 12 May 2016 18:13:08 +0200
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP1=2W7doTALgDcZ8zixVqf_CyyXeqs_U_0AQvkDSq9Lsu=cMw@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <87shxo1dz4.fsf@thinkpad.rath.org>
 <CAP1=2W7doTALgDcZ8zixVqf_CyyXeqs_U_0AQvkDSq9Lsu=cMw@mail.gmail.com>
Message-ID: <5734AB94.4010807@mail.de>

On 11.05.2016 23:57, Brett Cannon wrote:
> On Wed, 11 May 2016 at 14:29 Nikolaus Rath <Nikolaus at rath.org 
> <mailto:Nikolaus at rath.org>> wrote:
>
>     On May 11 2016, Brett Cannon <brett at python.org
>     <mailto:brett at python.org>> wrote:
>     > This PEP proposes a protocol for classes which represent a file
>     system
>     > path to be able to provide a ``str`` or ``bytes`` representation.
>     [...]
>
>     As I said before, to me this seems like a lot of effort for a very
>     specific use-case.
>

Exactly. Especially when considering what else can be done to improve 
the situation considerably.

>     So let me put forward two hypothetical scenarios to
>     better understand your position:
>
>     - A new module for URL handling is added to the standard library (or
>       urllib is suitably extended). There is a proposal to add a new
>       protocol that allows classes to provide a ``str`` or ``bytes``
>       representation of URLs.
>
>     - A new (third-party) library for natural language processing arises
>       that exposes a specific class for representing audio data. Existing
>       language processing code just uses bytes objects. To ease transition
>       and interoperability, it is proposed to add a new protocol for
>     classes
>       that represend audio data to provide a bytes representation.
>

You can even add the timedelta-to-seconds protocol that somebody thought 
would be good idea:

https://mail.python.org/pipermail/python-dev/2016-April/144018.html
https://mail.python.org/pipermail/python-ideas/2016-May/040226.html

The generalization is straight-forward and a result of this discussion. 
If it works and is a good idea for pathlib, then there's absolutely no 
reason not to do this for the datetime lib and other rich-object libs. 
Same goes the other way round. Question still is: is it a good idea?

Maybe, it will become a successful pattern. Maybe not.

>     Do you think you would you be in favor of adding these protocols to
>     the stdlib/languange reference as well?
>
>
> Maybe for URLs, not for audio data (at least not in the stdlib; 
> community can do what they want).
>
>     If not, what's the crucial
>     difference to file system paths?
>
>
> Nearly everyone uses file system paths on a regular basis, less so 
> than URLs but still a good amount of people. Very few people work with 
> audio data.

Amount of usage should be taken into account of course. However, 
question remains if that suffices as a justification for the effort.


Best,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160512/0e9bffaf/attachment.html>

From guido at python.org  Thu May 12 12:24:04 2016
From: guido at python.org (Guido van Rossum)
Date: Thu, 12 May 2016 09:24:04 -0700
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <5734A46C.40306@stoneleaf.us>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
Message-ID: <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>

I am glad this is finally happening. There's quite a bit of noise in the
thread which I have to ignore. The two issues that I want to respond to are
speed and whether os.fspath() can return bytes.

- Speed: We should trust our ability to optimize the implementations where
necessary. First the API issues need to be settled.

- Bytes: I strongly believe that os.fspath() should be a thin wrapper
around the __fspath__ protocol, like next() wraps the .__next__ protocol.
It should not get into bytes vs. string politics. If your app really needs
strings, call os.fsdecode(). So this is my version (unoptimized):

def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]:
    if isinstance(p, (str, bytes)):
        return p
    try:
        return p.__fspath__
    except AttributeError:
        raise TypeError(...)

Other than that I think the PEP is already in fine shape.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160512/dd232fdc/attachment.html>

From srkunze at mail.de  Thu May 12 12:26:14 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Thu, 12 May 2016 18:26:14 +0200
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <5734A46C.40306@stoneleaf.us>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
Message-ID: <5734AEA6.5010308@mail.de>

On 12.05.2016 17:42, Ethan Furman wrote:
> On 05/12/2016 01:31 AM, Sven R. Kunze wrote:
>
>> I think the "Rationale" section ignores the fact the Path also supports
>> the .path attribute now. Which indeed defines a common interface between
>> path objects.
>
> The version of Python that has Path.path has not been released yet.  
> And even so, .path is not a "common interface" as neither str nor 
> bytes have it, and they also are used as path objects.

str and bytes will receive the __fspath__ attribute when this PEP is 
accepted?

> And even given all that, for smoother interoperability with the rest 
> of the stdlib, or at least the os.* portion, those functions would 
> still need to be upgraded to check for .path on the incoming arguments 
> -- at which point we may as well make a protocol to properly support 
> file system paths instead of relying on the rather generic attribute 
> name of 'path'.

Just so, if you accept changing os.* as a necessary solution.

If not, keeping .path would suffice and would be much simpler.


Best,
Sven

From ethan at stoneleaf.us  Thu May 12 12:56:39 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 12 May 2016 09:56:39 -0700
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <5734AEA6.5010308@mail.de>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <5734AEA6.5010308@mail.de>
Message-ID: <5734B5C7.4060206@stoneleaf.us>

On 05/12/2016 09:26 AM, Sven R. Kunze wrote:
> On 12.05.2016 17:42, Ethan Furman wrote:
>> On 05/12/2016 01:31 AM, Sven R. Kunze wrote:

>>> I think the "Rationale" section ignores the fact the Path also supports
>>> the .path attribute now. Which indeed defines a common interface between
>>> path objects.
>>
>> The version of Python that has Path.path has not been released yet.
>> And even so, .path is not a "common interface" as neither str nor
>> bytes have it, and they also are used as path objects.
>
> str and bytes will receive the __fspath__ attribute when this PEP is
> accepted?

No, they won't.  The __fspath__ protocol will reduce the rich path 
object down to a str/bytes object.

One could argue that a .path attribute is similar, but consider:  if you 
are handed a random object with a .path attribute, how certain can you 
be that it represents a file system path?  Contrariwise, how certain can 
you be of an object that has __fspath__?

At any rate, we seem to be down to the details of os.fspath() so I don't 
see any reason to discuss .path any further.

--
~Ethan~

From brett at python.org  Thu May 12 13:18:45 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 12 May 2016 17:18:45 +0000
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
Message-ID: <CAP1=2W6Qyis87YJ8LQf+P8QiY9Knvzbi+41Kz3LcKov=QEqP=Q@mail.gmail.com>

On Thu, 12 May 2016 at 09:25 Guido van Rossum <guido at python.org> wrote:

> I am glad this is finally happening. There's quite a bit of noise in the
> thread which I have to ignore.
>

Don't worry, I'm not ignoring it on your behalf. :)


> The two issues that I want to respond to are speed and whether os.fspath()
> can return bytes.
>
> - Speed: We should trust our ability to optimize the implementations where
> necessary. First the API issues need to be settled.
>

Added a note to the PEP to say perf isn't a worry for os.path.


>
> - Bytes: I strongly believe that os.fspath() should be a thin wrapper
> around the __fspath__ protocol, like next() wraps the .__next__ protocol.
> It should not get into bytes vs. string politics. If your app really needs
> strings, call os.fsdecode(). So this is my version (unoptimized):
>
> def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]:
>     if isinstance(p, (str, bytes)):
>         return p
>     try:
>         return p.__fspath__
>     except AttributeError:
>         raise TypeError(...)
>

> Other than that I think the PEP is already in fine shape.
>

Just to double-check, did you mean for __fspath__ to only be an attribute
in your example, or did you leave off the `()` by accident? As of right now
the PEP is proposing a method for the protocol to follow common practice of
using methods and in case the representation is not always pre-computed and
thus not necessarily giving the wrong impression that the attribute access
is cheap. But admittedly an attribute was previously proposed and there
wasn't a terribly strong argument against it beyond "we historically
haven't done it that way", so I'm open to swapping to an attribute if
that's your preference.

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160512/8020bf59/attachment.html>

From srkunze at mail.de  Thu May 12 13:21:45 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Thu, 12 May 2016 19:21:45 +0200
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <5734B5C7.4060206@stoneleaf.us>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <5734AEA6.5010308@mail.de> <5734B5C7.4060206@stoneleaf.us>
Message-ID: <5734BBA9.2050807@mail.de>

On 12.05.2016 18:56, Ethan Furman wrote:
> On 05/12/2016 09:26 AM, Sven R. Kunze wrote:
>> str and bytes will receive the __fspath__ attribute when this PEP is
>> accepted?
>
> No, they won't.  The __fspath__ protocol will reduce the rich path 
> object down to a str/bytes object.

Would this make the implementation of os.fspath simpler?

After all, str and bytes are to some extend path-like objects.


Best,
Sven

From ethan at stoneleaf.us  Thu May 12 13:27:21 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 12 May 2016 10:27:21 -0700
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <5734BBA9.2050807@mail.de>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <5734AEA6.5010308@mail.de> <5734B5C7.4060206@stoneleaf.us>
 <5734BBA9.2050807@mail.de>
Message-ID: <5734BCF9.6060305@stoneleaf.us>

On 05/12/2016 10:21 AM, Sven R. Kunze wrote:
> On 12.05.2016 18:56, Ethan Furman wrote:
>> On 05/12/2016 09:26 AM, Sven R. Kunze wrote:

>>> str and bytes will receive the __fspath__ attribute when this PEP is
>>> accepted?
>>
>> No, they won't.  The __fspath__ protocol will reduce the rich path
>> object down to a str/bytes object.
>
> Would this make the implementation of os.fspath simpler?

Maybe, but a bad idea for two reasons:

1) Reducing a str to the exact same str is silly; and, more importantly
2) not every str/bytes is a path

--
~Ethan~

From sjoerdjob at sjoerdjob.com  Thu May 12 13:22:25 2016
From: sjoerdjob at sjoerdjob.com (Sjoerd Job Postmus)
Date: Thu, 12 May 2016 19:22:25 +0200
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
Message-ID: <20160512172225.GA16846@sjoerdjob.com>

I would like to make just 1 comment regarding the question of accepting
(or not) bytes as output of `os.fspath`.

The whole point of adding `os.fspath` is to make it easier to use Path
objects. This is in an effort to gain greater adoption of pathlib in
libraries. Now, this is an excellent idea.

However, if it were to reject bytes, that would mean that when libraries
start to use pathlib, it would suddenly become harder for people that
actually need bytes-support to use pathlib.

Now, the claim 'if you need bytes, you should not be using pathlib` is a
reasonable one. But what if I need bytes *and* a specific library (say,
image handling, or a web framework, or ...). It's not up to me if that
library uses pathlib or plain old os.path.join.

Is using surrogate-escapes enough for this case? I myself am not sure,
(and also not affected), but it sounds to me that rejecting bytes is a
wrong approach if there is no proper workaround (assuming the use-case
of pathlib is somewhere deep in library code).


From k7hoven at gmail.com  Thu May 12 14:11:18 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Thu, 12 May 2016 21:11:18 +0300
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
Message-ID: <CAMiohohcu2g5aeakHyThVV4D1rb2mZDxLKU82q1CHzWqGzYdzg@mail.gmail.com>

On Thu, May 12, 2016 at 7:24 PM, Guido van Rossum <guido at python.org> wrote:
> I am glad this is finally happening. There's quite a bit of noise in the
> thread which I have to ignore. The two issues that I want to respond to are
> speed and whether os.fspath() can return bytes.
>
> - Speed: We should trust our ability to optimize the implementations where
> necessary. First the API issues need to be settled.
>
> - Bytes: I strongly believe that os.fspath() should be a thin wrapper around
> the __fspath__ protocol, like next() wraps the .__next__ protocol. It should
> not get into bytes vs. string politics. If your app really needs strings,
> call os.fsdecode(). So this is my version (unoptimized):
>

:)

Thank you for this. I can breathe now.

Some questions remain:

> def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]:
>     if isinstance(p, (str, bytes)):
>         return p
>     try:
>         return p.__fspath__
>     except AttributeError:
>         raise TypeError(...)
>

(I know Brett already posted this question, but somehow it did not
show up in my mailbox before I had written this. I'm (re)posting
because there is some stuff here that is not in Brett's email )

You might be suggesting that __fspath__ should be an attribute, not a
method, or did you mean something like:

def fspath(p):
    if isinstance(p, (str, bytes)):
        return p
    try:
        p.__fspath__
    except AttributeError:
        raise TypeError(...)
    return p.__fspath__()

IMO, either is fine, I suppose. As you know, it's mostly a question of
whether __fspath__ will be a property or a method (on PurePath for
instance). But if you meant the former, that would change also the ABC
and the protocol description.

-- Koos

From k7hoven at gmail.com  Thu May 12 14:32:35 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Thu, 12 May 2016 21:32:35 +0300
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <20160512172225.GA16846@sjoerdjob.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <20160512172225.GA16846@sjoerdjob.com>
Message-ID: <CAMiohoi166qUW8n_R2BqrXxqxYLNkfF_2QLs4euX9i2pfYOmbQ@mail.gmail.com>

On Thu, May 12, 2016 at 8:22 PM, Sjoerd Job Postmus
<sjoerdjob at sjoerdjob.com> wrote:
> I would like to make just 1 comment regarding the question of accepting
> (or not) bytes as output of `os.fspath`.
>
> The whole point of adding `os.fspath` is to make it easier to use Path
> objects. This is in an effort to gain greater adoption of pathlib in
> libraries. Now, this is an excellent idea.
>
> However, if it were to reject bytes, that would mean that when libraries
> start to use pathlib, it would suddenly become harder for people that
> actually need bytes-support to use pathlib.
>
> Now, the claim 'if you need bytes, you should not be using pathlib` is a
> reasonable one. But what if I need bytes *and* a specific library (say,
> image handling, or a web framework, or ...). It's not up to me if that
> library uses pathlib or plain old os.path.join.
>
> Is using surrogate-escapes enough for this case? I myself am not sure,
> (and also not affected), but it sounds to me that rejecting bytes is a
> wrong approach if there is no proper workaround (assuming the use-case
> of pathlib is somewhere deep in library code).
>

This is out of the scope of this PEP and probably a very insignificant
issue (luckily, this is not the pathlib PEP). Surrogates will probably
work and if not, on can "blaim" broken filenames ;).

-- Koos

From guido at python.org  Thu May 12 14:35:40 2016
From: guido at python.org (Guido van Rossum)
Date: Thu, 12 May 2016 11:35:40 -0700
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP1=2W6Qyis87YJ8LQf+P8QiY9Knvzbi+41Kz3LcKov=QEqP=Q@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
 <CAP1=2W6Qyis87YJ8LQf+P8QiY9Knvzbi+41Kz3LcKov=QEqP=Q@mail.gmail.com>
Message-ID: <CAP7+vJL8Mxu0wdCVq8xdJy3m-R4YpVBhOgUrFnqCeie7e=0XvA@mail.gmail.com>

On Thu, May 12, 2016 at 10:18 AM, Brett Cannon <brett at python.org> wrote:

>
> On Thu, 12 May 2016 at 09:25 Guido van Rossum <guido at python.org> wrote:
>
>> def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]:
>>     if isinstance(p, (str, bytes)):
>>         return p
>>     try:
>>         return p.__fspath__
>>     except AttributeError:
>>         raise TypeError(...)
>>
>
>> Other than that I think the PEP is already in fine shape.
>>
>
> - Bytes: I strongly believe that os.fspath() should be a thin wrapper
> around the __fspath__ protocol, like next() wraps the .__next__ protocol.
> It should not get into bytes vs. string politics. If your app really needs
> strings, call os.fsdecode(). So this is my version (unoptimized):
>
> Just to double-check, did you mean for __fspath__ to only be an attribute
> in your example, or did you leave off the `()` by accident? As of right now
> the PEP is proposing a method for the protocol to follow common practice of
> using methods and in case the representation is not always pre-computed and
> thus not necessarily giving the wrong impression that the attribute access
> is cheap. But admittedly an attribute was previously proposed and there
> wasn't a terribly strong argument against it beyond "we historically
> haven't done it that way", so I'm open to swapping to an attribute if
> that's your preference.
>
>>
Whoops. Didn't mean to change that! Yes, __fspath__ should remain a method.
You can breathe again. :-)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160512/62f6c160/attachment.html>

From brett at python.org  Thu May 12 14:51:55 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 12 May 2016 18:51:55 +0000
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP7+vJL8Mxu0wdCVq8xdJy3m-R4YpVBhOgUrFnqCeie7e=0XvA@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
 <CAP1=2W6Qyis87YJ8LQf+P8QiY9Knvzbi+41Kz3LcKov=QEqP=Q@mail.gmail.com>
 <CAP7+vJL8Mxu0wdCVq8xdJy3m-R4YpVBhOgUrFnqCeie7e=0XvA@mail.gmail.com>
Message-ID: <CAP1=2W6wDy9iNeron7ZkVJ-VqNbd=-iGzKbtUi+8U9OiN_uiSA@mail.gmail.com>

On Thu, 12 May 2016 at 11:36 Guido van Rossum <guido at python.org> wrote:

> On Thu, May 12, 2016 at 10:18 AM, Brett Cannon <brett at python.org> wrote:
>
>> On Thu, 12 May 2016 at 09:25 Guido van Rossum <guido at python.org> wrote:
>>
> def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]:
>>>     if isinstance(p, (str, bytes)):
>>>         return p
>>>     try:
>>>         return p.__fspath__
>>>     except AttributeError:
>>>         raise TypeError(...)
>>>
>>
>>> Other than that I think the PEP is already in fine shape.
>>>
>>
>> - Bytes: I strongly believe that os.fspath() should be a thin wrapper
>> around the __fspath__ protocol, like next() wraps the .__next__ protocol.
>> It should not get into bytes vs. string politics. If your app really needs
>> strings, call os.fsdecode(). So this is my version (unoptimized):
>>
>> Just to double-check, did you mean for __fspath__ to only be an attribute
>> in your example, or did you leave off the `()` by accident? As of right now
>> the PEP is proposing a method for the protocol to follow common practice of
>> using methods and in case the representation is not always pre-computed and
>> thus not necessarily giving the wrong impression that the attribute access
>> is cheap. But admittedly an attribute was previously proposed and there
>> wasn't a terribly strong argument against it beyond "we historically
>> haven't done it that way", so I'm open to swapping to an attribute if
>> that's your preference.
>>
>>>
> Whoops. Didn't mean to change that! Yes, __fspath__ should remain a
> method. You can breathe again. :-)
>

That's a mechanical change so not exactly the most stressful aspect of this
PEP. :) I'll add the attribute angle to the Rejected Ideas, though.

Anyway, with your strong preference of how to tweak os.fspath() what
specifically would you like to see discussed at this point? Assuming the
os.fspath() -> bytes discussion is dealt with, the only open issues listed
in the PEP are the naming and placement of the ABC and how to do type hints
for all of this (thanks to the dichotomy of path objects using the protocol
and path-like objects which is the union of path object, str, and bytes and
the joy of trying to name all of this well).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160512/2a1f589f/attachment.html>

From guido at python.org  Thu May 12 15:02:55 2016
From: guido at python.org (Guido van Rossum)
Date: Thu, 12 May 2016 12:02:55 -0700
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP1=2W6wDy9iNeron7ZkVJ-VqNbd=-iGzKbtUi+8U9OiN_uiSA@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
 <CAP1=2W6Qyis87YJ8LQf+P8QiY9Knvzbi+41Kz3LcKov=QEqP=Q@mail.gmail.com>
 <CAP7+vJL8Mxu0wdCVq8xdJy3m-R4YpVBhOgUrFnqCeie7e=0XvA@mail.gmail.com>
 <CAP1=2W6wDy9iNeron7ZkVJ-VqNbd=-iGzKbtUi+8U9OiN_uiSA@mail.gmail.com>
Message-ID: <CAP7+vJLBx2OOz4RaYYrLfVLAPQ3SO8GhxvsAJ09bb6Vyz2UjKQ@mail.gmail.com>

On Thu, May 12, 2016 at 11:51 AM, Brett Cannon <brett at python.org> wrote:

>
> Anyway, with your strong preference of how to tweak os.fspath() what
> specifically would you like to see discussed at this point?
>

Preferably nothing. :-) There's been too much discussion already.


> Assuming the os.fspath() -> bytes discussion is dealt with, the only open
> issues listed in the PEP are the naming and placement of the ABC and how to
> do type hints for all of this (thanks to the dichotomy of path objects
> using the protocol and path-like objects which is the union of path object,
> str, and bytes and the joy of trying to name all of this well).
>

Name and placement: I think it belongs in os, and os.PathLike sounds like a
fine name. (I'm surprised that os.DirEntry doesn't exist.)

Typing: do you want it to be a generic class? If not, the types can be left
out of the stdlib and only put in the stub (though you can show them in the
PEP of course).

If you want it to be generic we have more work to do. I'm out of time now
but we can discuss that after 3pm today.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160512/a09e56f7/attachment.html>

From brett at python.org  Thu May 12 15:18:12 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 12 May 2016 19:18:12 +0000
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP7+vJLBx2OOz4RaYYrLfVLAPQ3SO8GhxvsAJ09bb6Vyz2UjKQ@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
 <CAP1=2W6Qyis87YJ8LQf+P8QiY9Knvzbi+41Kz3LcKov=QEqP=Q@mail.gmail.com>
 <CAP7+vJL8Mxu0wdCVq8xdJy3m-R4YpVBhOgUrFnqCeie7e=0XvA@mail.gmail.com>
 <CAP1=2W6wDy9iNeron7ZkVJ-VqNbd=-iGzKbtUi+8U9OiN_uiSA@mail.gmail.com>
 <CAP7+vJLBx2OOz4RaYYrLfVLAPQ3SO8GhxvsAJ09bb6Vyz2UjKQ@mail.gmail.com>
Message-ID: <CAP1=2W6ktHRdHOuZ0g3rQ01t8EkJwxuBSrru5PRKd1kCNk0CXw@mail.gmail.com>

On Thu, 12 May 2016 at 12:03 Guido van Rossum <guido at python.org> wrote:

> On Thu, May 12, 2016 at 11:51 AM, Brett Cannon <brett at python.org> wrote:
>
>>
>> Anyway, with your strong preference of how to tweak os.fspath() what
>> specifically would you like to see discussed at this point?
>>
>
> Preferably nothing. :-) There's been too much discussion already.
>

Works for me. :) I'll update the PEP with the new semantics for os.fspath()
and send out the updated version later today.


>
>
>> Assuming the os.fspath() -> bytes discussion is dealt with, the only open
>> issues listed in the PEP are the naming and placement of the ABC and how to
>> do type hints for all of this (thanks to the dichotomy of path objects
>> using the protocol and path-like objects which is the union of path object,
>> str, and bytes and the joy of trying to name all of this well).
>>
>
> Name and placement: I think it belongs in os, and os.PathLike sounds like
> a fine name. (I'm surprised that os.DirEntry doesn't exist.)
>

SGTM. And os.DirEntry doesn't exist simply because it's posix.DirEntry
since it's implemented entirely in C. We could add an alias but since it
isn't constructed from scratch I don't think it's worth it.


>
> Typing: do you want it to be a generic class? If not, the types can be
> left out of the stdlib and only put in the stub (though you can show them
> in the PEP of course).
>

If we aren't going to restrict what os.fspath() returns then I don't see
any need to make the type generic (I mean technically a generic version
might be nice for e.g. the constructor of pathlib only taking strings, but
it's probably overkill).

I guess my real question is whether we want to create typing.PathLike to
match os.PathLike? And it sounds like you don't want to bother with a
potential second type that corresponds to Union[str, bytes, PathLike].


>
> If you want it to be generic we have more work to do. I'm out of time now
> but we can discuss that after 3pm today.
>

Sure thing.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160512/c2cc359b/attachment.html>

From guido at python.org  Thu May 12 15:20:53 2016
From: guido at python.org (Guido van Rossum)
Date: Thu, 12 May 2016 12:20:53 -0700
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP1=2W6ktHRdHOuZ0g3rQ01t8EkJwxuBSrru5PRKd1kCNk0CXw@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
 <CAP1=2W6Qyis87YJ8LQf+P8QiY9Knvzbi+41Kz3LcKov=QEqP=Q@mail.gmail.com>
 <CAP7+vJL8Mxu0wdCVq8xdJy3m-R4YpVBhOgUrFnqCeie7e=0XvA@mail.gmail.com>
 <CAP1=2W6wDy9iNeron7ZkVJ-VqNbd=-iGzKbtUi+8U9OiN_uiSA@mail.gmail.com>
 <CAP7+vJLBx2OOz4RaYYrLfVLAPQ3SO8GhxvsAJ09bb6Vyz2UjKQ@mail.gmail.com>
 <CAP1=2W6ktHRdHOuZ0g3rQ01t8EkJwxuBSrru5PRKd1kCNk0CXw@mail.gmail.com>
Message-ID: <CAP7+vJLyU_AA664H1a4smHaHc=Y6CabMg1geGF00KTRnuOAQog@mail.gmail.com>

There's no need for typing.PathLike.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160512/c7f6c28d/attachment.html>

From ethan at stoneleaf.us  Thu May 12 15:30:45 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 12 May 2016 12:30:45 -0700
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <20160512172225.GA16846@sjoerdjob.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <20160512172225.GA16846@sjoerdjob.com>
Message-ID: <5734D9E5.6060701@stoneleaf.us>

On 05/12/2016 10:22 AM, Sjoerd Job Postmus wrote:

> However, if it were to reject bytes, that would mean that when libraries
> start to use pathlib, it would suddenly become harder for people that
> actually need bytes-support to use pathlib.

pathlib is not about bytes support.  While bytes are necessary in this 
digital world we live in, most things that look like text should be 
text, and that includes most paths.
>
> Now, the claim 'if you need bytes, you should not be using pathlib` is a
> reasonable one. But what if I need bytes *and* a specific library (say,
> image handling, or a web framework, or ...). It's not up to me if that
> library uses pathlib or plain old os.path.join.

If you need bytes support for your paths, there's at least one [1] that 
has that support.

--
~Ethan~


[1] Yeah, it's mine [2].  ;)  I haven't checked if the other third-party 
libs do or not.

[2] <shameless plug> https://pypi.python.org/pypi/antipathy </shameless 
plug>

From brett at python.org  Thu May 12 16:53:12 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 12 May 2016 20:53:12 +0000
Subject: [Python-Dev] File system path PEP, part 2
Message-ID: <CAP1=2W6Bf3faNg7UdVgGB2ZXWF2V3nMx2jL_EiGHUnizeCtisA@mail.gmail.com>

Second draft that takes Guido's comments into consideration. The biggest
change is os.fspath() now returns whatever path.__fspath__() returns
instead of restricting it to only str.

Minor changes:
- Renamed the C function to PyOS_FSPath()
- Added an Implementation section with a TODO list
- Bunch of things added to the Rejected Ideas section

----------
PEP: NNN
Title: Adding a file system path protocol
Version: $Revision$
Last-Modified: $Date$
Author: Brett Cannon <brett at python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 11-May-2016
Post-History: 11-May-2016,
              12-May-2016


Abstract
========

This PEP proposes a protocol for classes which represent a file system
path to be able to provide a ``str`` or ``bytes`` representation.
Changes to Python's standard library are also proposed to utilize this
protocol where appropriate to facilitate the use of path objects where
historically only ``str`` and/or ``bytes`` file system paths are
accepted. The goal is to facilitate the migration of users towards
rich path objects while providing an easy way to work with code
expecting ``str`` or ``bytes``.


Rationale
=========

Historically in Python, file system paths have been represented as
strings or bytes. This choice of representation has stemmed from C's
own decision to represent file system paths as
``const char *`` [#libc-open]_. While that is a totally serviceable
format to use for file system paths, it's not necessarily optimal. At
issue is the fact that while all file system paths can be represented
as strings or bytes, not all strings or bytes represent a file system
path. This can lead to issues where any e.g. string duck-types to a
file system path whether it actually represents a path or not.

To help elevate the representation of file system paths from their
representation as strings and bytes to a richer object representation,
the pathlib module [#pathlib]_ was provisionally introduced in
Python 3.4 through PEP 428. While considered by some as an improvement
over strings and bytes for file system paths, it has suffered from a
lack of adoption. Typically the key issue listed for the low adoption
rate has been the lack of support in the standard library. This lack
of support required users of pathlib to manually convert path objects
to strings by calling ``str(path)`` which many found error-prone.

One issue in converting path objects to strings comes from
the fact that the only generic way to get a string representation of
the path was to pass the object to ``str()``. This can pose a
problem when done blindly as nearly all Python objects have some
string representation whether they are a path or not, e.g.
``str(None)`` will give a result that
``builtins.open()`` [#builtins-open]_ will happily use to create a new
file.

Exacerbating this whole situation is the
``DirEntry`` object [#os-direntry]_. While path objects have a
representation that can be extracted using ``str()``, ``DirEntry``
objects expose a ``path`` attribute instead. Having no common
interface between path objects, ``DirEntry``, and any other
third-party path library has become an issue. A solution that allows
any path-representing object to declare that it is a path and a way
to extract a low-level representation that all path objects could
support is desired.

This PEP then proposes to introduce a new protocol to be followed by
objects which represent file system paths. Providing a protocol allows
for explicit signaling of what objects represent file system paths as
well as a way to extract a lower-level representation that can be used
with older APIs which only support strings or bytes.

Discussions regarding path objects that led to this PEP can be found
in multiple threads on the python-ideas mailing list archive
[#python-ideas-archive]_ for the months of March and April 2016 and on
the python-dev mailing list archives [#python-dev-archive]_ during
April 2016.


Proposal
========

This proposal is split into two parts. One part is the proposal of a
protocol for objects to declare and provide support for exposing a
file system path representation. The other part deals with changes to
Python's standard library to support the new protocol. These changes
will also lead to the pathlib module dropping its provisional status.

Protocol
--------

The following abstract base class defines the protocol for an object
to be considered a path object::

    import abc
    import typing as t


    class PathLike(abc.ABC):

        """Abstract base class for implementing the file system path
protocol."""

        @abc.abstractmethod
        def __fspath__(self) -> t.Union[str, bytes]:
            """Return the file system path representation of the object."""
            raise NotImplementedError


Objects representing file system paths will implement the
``__fspath__()`` method which will return the ``str`` or ``bytes``
representation of the path. The ``str`` representation is the
preferred low-level path representation as it is human-readable and
what people historically represent paths as.


Standard library changes
------------------------

It is expected that most APIs in Python's standard library that
currently accept a file system path will be updated appropriately to
accept path objects (whether that requires code or simply an update
to documentation will vary). The modules mentioned below, though,
deserve specific details as they have either fundamental changes that
empower the ability to use path objects, or entail additions/removal
of APIs.


builtins
''''''''

``open()`` [#builtins-open]_ will be updated to accept path objects as
well as continue to accept ``str`` and ``bytes``.


os
'''

The ``fspath()`` function will be added with the following semantics::

    import typing as t


    def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str, bytes]:
        """Return the string representation of the path.

        If str or bytes is passed in, it is returned unchanged.
        """
        if isinstance(path, (str, bytes)):
            return path

        try:
            return path.__fspath__()
        except AttributeError:
            if hasattr(path, '__fspath__'):
                raise

            raise TypeError("expected str, bytes or path object, not "
                            + type(path).__name__)

The ``os.fsencode()`` [#os-fsencode]_ and
``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept
path objects. As both functions coerce their arguments to
``bytes`` and ``str``, respectively, they will be updated to call
``__fspath__()`` if present to convert the path object to a ``str`` or
``bytes`` representation, and then perform their appropriate
coercion operations as if the return value from ``__fspath__()`` had
been the original argument to the coercion function in question.

The addition of ``os.fspath()``, the updates to
``os.fsencode()``/``os.fsdecode()``, and the current semantics of
``pathlib.PurePath`` provide the semantics necessary to
get the path representation one prefers. For a path object,
``pathlib.PurePath``/``Path`` can be used. To obtain the ``str`` or
``bytes`` representation without any coersion, then ``os.fspath()``
can be used. If a ``str`` is desired and the encoding of ``bytes``
should be assumed to be the default file system encoding, then
``os.fsdecode()`` should be used. If a ``bytes`` representation is
desired and any strings should be encoded using the default file
system encoding, then ``os.fsencode()`` is used. This PEP recommends
using path objects when possible and falling back to string paths as
necessary and using ``bytes`` as a last resort.

Another way to view this is as a hierarchy of file system path
representations (highest- to lowest-level): path ? str ? bytes. The
functions and classes under discussion can all accept objects on the
same level of the hierarchy, but they vary in whether they promote or
demote objects to another level. The ``pathlib.PurePath`` class can
promote a ``str`` to a path object. The ``os.fspath()`` function can
demote a path object to a ``str`` or ``bytes`` instance, depending
on what ``__fspath__()`` returns.
The ``os.fsdecode()`` function will demote a path object to
a string or promote a ``bytes`` object to a ``str``. The
``os.fsencode()`` function will demote a path or string object to
``bytes``. There is no function that provides a way to demote a path
object directly to ``bytes`` while bypassing string demotion.

The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()``
method. It will return the same value as currently found on the
``path`` attribute of ``DirEntry`` instances.

The Protocol_ ABC will be added to the ``os`` module under the name
``os.PathLike``.


os.path
'''''''

The various path-manipulation functions of ``os.path`` [#os-path]_
will be updated to accept path objects. For polymorphic functions that
accept both bytes and strings, they will be updated to simply use
``os.fspath()``.

During the discussions leading up to this PEP it was suggested that
``os.path`` not be updated using an "explicit is better than implicit"
argument. The thinking was that since ``__fspath__()`` is polymorphic
itself it may be better to have code working with ``os.path`` extract
the path representation from path objects explicitly. There is also
the consideration that adding support this deep into the low-level OS
APIs will lead to code magically supporting path objects without
requiring any documentation updated, leading to potential complaints
when it doesn't work, unbeknownst to the project author.

But it is the view of this PEP that "practicality beats purity" in
this instance. To help facilitate the transition to supporting path
objects, it is better to make the transition as easy as possible than
to worry about unexpected/undocumented duck typing support for
path objects by projects.

There has also been the suggestion that ``os.path`` functions could be
used in a tight loop and the overhead of checking or calling
``__fspath__()`` would be too costly. In this scenario only
path-consuming APIs would be directly updated and path-manipulating
APIs like the ones in ``os.path`` would go unmodified. This would
require library authors to update their code to support path objects
if they performed any path manipulations, but if the library code
passed the path straight through then the library wouldn't need to be
updated. It is the view of this PEP and Guido, though, that this is an
unnecessary worry and that performance will still be acceptable.


pathlib
'''''''

The constructor for ``pathlib.PurePath`` and ``pathlib.Path`` will be
updated to accept ``PathLike`` objects. Both ``PurePath`` and ``Path``
will continue to not accept ``bytes`` path representations, and so if
``__fspath__()`` returns ``bytes`` it will raise an exception.

The ``path`` attribute will be removed as this PEP makes it
redundant (it has not been included in any released version of Python
and so is not a backwards-compatibility concern).


C API
'''''

The C API will gain an equivalent function to ``os.fspath()``::

    /*
        Return the file system path of the object.

        If the object is str or bytes, then allow it to pass through with
        an incremented refcount. If the object defines __fspath__(), then
        return the result of that method. All other types raise a TypeError.
    */
    PyObject *
    PyOS_FSPath(PyObject *path)
    {
        if (PyUnicode_Check(path) || PyBytes_Check(path)) {
            Py_INCREF(path);
            return path;
        }

        if (PyObject_HasAttrString(path, "__fspath__")) {
            return PyObject_CallMethodObjArgs(path, "__fspath__", NULL);
        }

        return PyErr_Format(PyExc_TypeError,
                            "expected a str, bytes, or path object, not %S",
                                path->ob_type);
    }


Backwards compatibility
=======================

There are no explicit backwards-compatibility concerns. Unless an
object incidentally already defines a ``__fspath__()`` method there is
no reason to expect the pre-existing code to break or expect to have
its semantics implicitly changed.

Libraries wishing to support path objects and a version of Python
prior to Python 3.6 and the existence of ``os.fspath()`` can use the
idiom of
``path.__fspath__() if hasattr(path, "__fspath__") else path``.


Implementation
==============

This is the task list for what this PEP proposes:

#. Remove the ``path`` attribute from pathlib
#. Remove the provisional status of pathlib
#. Add ``os.PathLike``
#. Add ``os.fspath()``
#. Add ``PyOS_FSPath()``
#. Update ``os.fsencode()``
#. Update ``os.fsdecode()``
#. Update ``pathlib.PurePath`` and ``pathlib.Path``
#. Update ``builtins.open()``
#. Update ``os.DirEntry``
#. Update ``os.path``


Rejected Ideas
==============

Other names for the protocol's method
-------------------------------------

Various names were proposed during discussions leading to this PEP,
including ``__path__``, ``__pathname__``, and ``__fspathname__``. In
the end people seemed to gravitate towards ``__fspath__`` for being
unambiguous without being unnecessarily long.


Separate str/bytes methods
--------------------------

At one point it was suggested that ``__fspath__()`` only return
strings and another method named ``__fspathb__()`` be introduced to
return bytes. The thinking is that by making ``__fspath__()`` not be
polymorphic it could make dealing with the potential string or bytes
representations easier. But the general consensus was that returning
bytes will more than likely be rare and that the various functions in
the os module are the better abstraction to promote over direct
calls to ``__fspath__()``.


Providing a ``path`` attribute
------------------------------

To help deal with the issue of ``pathlib.PurePath`` not inheriting
from ``str``, originally it was proposed to introduce a ``path``
attribute to mirror what ``os.DirEntry`` provides. In the end,
though, it was determined that a protocol would provide the same
result while not directly exposing an API that most people will never
need to interact with directly.


Have ``__fspath__()`` only return strings
------------------------------------------

Much of the discussion that led to this PEP revolved around whether
``__fspath__()`` should be polymorphic and return ``bytes`` as well as
``str`` or only return ``str``. The general sentiment for this view
was that ``bytes`` are difficult to work with due to their
inherent lack of information about their encoding and PEP 383 makes
it possible to represent all file system paths using ``str`` with the
``surrogateescape`` handler. Thus, it would be better to forcibly
promote the use of ``str`` as the low-level path representation for
high-level path objects.

In the end, it was decided that using ``bytes`` to represent paths is
simply not going to go away and thus they should be supported to some
degree. The hope is that people will gravitate towards path objects
like pathlib and that will move people away from operating directly
with ``bytes``.


A generic string encoding mechanism
-----------------------------------

At one point there was a discussion of developing a generic mechanism
to extract a string representation of an object that had semantic
meaning (``__str__()`` does not necessarily return anything of
semantic significance beyond what may be helpful for debugging). In
the end, it was deemed to lack a motivating need beyond the one this
PEP is trying to solve in a specific fashion.


Have __fspath__ be an attribute
-------------------------------

It was briefly considered to have ``__fspath__`` be an attribute
instead of a method. This was rejected for two reasons. One,
historically protocols have been implemented as "magic methods" and
not "magic methods and attributes". Two, there is no guarantee that
the lower-level representation of a path object will be pre-computed,
potentially misleading users that there was no expensive computation
behind the scenes in case the attribute was implemented as a property.

This also indirectly ties into the idea of introducing a ``path``
attribute to accomplish the same thing. This idea has an added issue,
though, of accidentally having any object with a ``path`` attribute
meet the protocol's duck typing. Introducing a new magic method for
the protocol helpfully avoids any accidental opting into the protocol.


Provide specific type hinting support
-------------------------------------

There was some consideration to provdinga generic ``typing.PathLike``
class which would allow for e.g. ``typing.PathLike[str]`` to specify
a type hint for a path object which returned a string representation.
While potentially beneficial, the usefulness was deemed too small to
bother adding the type hint class.

This also removed any desire to have a class in the ``typing`` module
which represented the union of all acceptable path-representing types
as that can be represented with
``typing.Union[str, bytes, os.PathLike]`` easily enough and the hope
is users will slowly gravitate to path objects only.


Acknowledgements
================

Thanks to everyone who participated in the various discussions related
to this PEP that spanned both python-ideas and python-dev. Special
thanks to Stephen Turnbull for direct feedback on early drafts of this
PEP. More special thanks to Koos Zevenhoven and Ethan Furman for not
only feedback on early drafts of this PEP but also helping to drive
the overall discussion on this topic across the two mailing lists.


References
==========

.. [#python-ideas-archive] The python-ideas mailing list archive
   (https://mail.python.org/pipermail/python-ideas/)

.. [#python-dev-archive] The python-dev mailing list archive
   (https://mail.python.org/pipermail/python-dev/)

.. [#libc-open] ``open()`` documention for the C standard library
   (
http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html
)

.. [#pathlib] The ``pathlib`` module
   (https://docs.python.org/3/library/pathlib.html#module-pathlib)

.. [#builtins-open] The ``builtins.open()`` function
   (https://docs.python.org/3/library/functions.html#open)

.. [#os-fsencode] The ``os.fsencode()`` function
   (https://docs.python.org/3/library/os.html#os.fsencode)

.. [#os-fsdecode] The ``os.fsdecode()`` function
   (https://docs.python.org/3/library/os.html#os.fsdecode)

.. [#os-direntry] The ``os.DirEntry`` class
   (https://docs.python.org/3/library/os.html#os.DirEntry)

.. [#os-path] The ``os.path`` module
   (https://docs.python.org/3/library/os.path.html#module-os.path)


Copyright
=========

This document has been placed in the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160512/5c2030db/attachment-0001.html>

From sjoerdjob at sjoerdjob.com  Thu May 12 16:59:29 2016
From: sjoerdjob at sjoerdjob.com (Sjoerd Job Postmus)
Date: Thu, 12 May 2016 22:59:29 +0200
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <5734D9E5.6060701@stoneleaf.us>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <20160512172225.GA16846@sjoerdjob.com> <5734D9E5.6060701@stoneleaf.us>
Message-ID: <E7D6329B-B0AC-463C-B935-58A9C74144AC@sjoerdjob.com>



> On 12 May 2016, at 21:30, Ethan Furman <ethan at stoneleaf.us> wrote:
> 
> If you need bytes support for your paths, there's at least one [1] that has that support.

So if I would need bytes support, I should submit a pull request to <arbitrary but awesome library> which replaces usage of the stdlib pathlib with another variant, upon which they will decline the pull request because it introduces another "useless" dependency.

Good to know.

From brett at python.org  Thu May 12 17:34:04 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 12 May 2016 21:34:04 +0000
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <E7D6329B-B0AC-463C-B935-58A9C74144AC@sjoerdjob.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <20160512172225.GA16846@sjoerdjob.com> <5734D9E5.6060701@stoneleaf.us>
 <E7D6329B-B0AC-463C-B935-58A9C74144AC@sjoerdjob.com>
Message-ID: <CAP1=2W5GMb-E-C8WD2rZpfRPjdwFEr9-Tzd3PhoXK95fw-be5w@mail.gmail.com>

On Thu, 12 May 2016 at 14:04 Sjoerd Job Postmus <sjoerdjob at sjoerdjob.com>
wrote:

>
>
> > On 12 May 2016, at 21:30, Ethan Furman <ethan at stoneleaf.us> wrote:
> >
> > If you need bytes support for your paths, there's at least one [1] that
> has that support.
>
> So if I would need bytes support, I should submit a pull request to
> <arbitrary but awesome library> which replaces usage of the stdlib pathlib
> with another variant, upon which they will decline the pull request because
> it introduces another "useless" dependency.
>

No, what you should do is ask them to create the pathlib instance lazily
and only when duck typing shows they weren't given a compatible object.
Then you could pass in some bytes-based solution like Ethan's and not worry
about pathlib's refusal to work with bytes. Or you simply ask them to work
with os.path after calling os.fspath() and be very careful to not use any
strings with the functions.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160512/ae3370ee/attachment.html>

From gvanrossum at gmail.com  Thu May 12 19:58:53 2016
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Thu, 12 May 2016 16:58:53 -0700
Subject: [Python-Dev] PEP 515: Underscores in Numeric Literals (revision
 3)
In-Reply-To: <CAP7+vJLd5wBRD1YwGjQuM6kTpnai9YdiLAvcHSM3yZ1vdr55Hw@mail.gmail.com>
References: <n9mqlh$oeb$1@ger.gmane.org>
 <CAP1=2W4P7skRpewpgxXBgHHicngMU3mJ2V1s3NVXgQL9PiCJmw@mail.gmail.com>
 <CAP7+vJLUumJiOzExqdmffJG_ywpFA9vXAWUH2Bpzvpiwcwp_RQ@mail.gmail.com>
 <ncisha$auc$1@ger.gmane.org>
 <CADiSq7etYj9Wy4++kP0z_5=-x6Pm0E9=6VASh+nYfJk8bM6PfQ@mail.gmail.com>
 <CAP7+vJ+e60gid=R8gjhmyiiNkCMC0Db4zt-Bct_qKUN29Vbr_Q@mail.gmail.com>
 <loom.20160319T181628-51@post.gmane.org>
 <CAP7+vJ+PewZ=ojQU0LzsZh0n0payOgzdOzfx6zQ73Xh9OYt5aQ@mail.gmail.com>
 <loom.20160319T191115-310@post.gmane.org>
 <CAP7+vJ+F=d9ycxZhaWkoYB6SYXf82T=Lf4QxWubwNy_FKJUACw@mail.gmail.com>
 <CAP1=2W4SM-_SBUo5YGdwyvq2N62+FC3TEgLnmYMW=2VDkV6+2A@mail.gmail.com>
 <CAP7+vJ+BicSaZiizSpjE8q2g=OGHMLgOre1tLwo3o5DN2crZ3g@mail.gmail.com>
 <nh14ea$5aa$1@ger.gmane.org>
 <CAP7+vJ+qAjuVrsrigS-vjouTCS5s4J6d2aDLjmX04MHenMev6g@mail.gmail.com>
 <CAP7+vJLd5wBRD1YwGjQuM6kTpnai9YdiLAvcHSM3yZ1vdr55Hw@mail.gmail.com>
Message-ID: <CAP7+vJK2u4-r0yNY_gH-CW7kzzk4522cLB2f6Hi50pdE1UAm+w@mail.gmail.com>

Is anyone going to mark the PEP as accepted?

On Thu, May 12, 2016 at 8:11 AM, Guido van Rossum <gvanrossum at gmail.com>
wrote:

> OK, then PEP 515 is now officially accepted! Congratulations. Start the
> implementation work!
>
> --Guido (mobile)
> On May 11, 2016 10:33 PM, "Georg Brandl" <g.brandl at gmx.net> wrote:
>
> I'm happy with the latest version.
>
> Georg
>
> On 05/11/2016 06:46 PM, Guido van Rossum wrote:
> > If the authors are happy I'll accept it right away.
> >
> > (I vaguely recall there's another PEP that's ready for pronouncement --
> but
> > which one?)
> >
> > On Wed, May 11, 2016 at 9:34 AM, Brett Cannon <brett at python.org
> > <mailto:brett at python.org>> wrote:
> >
> >     Is there anything holding up PEP 515 at this point in terms of
> acceptance or
> >     implementation?
> >
> >     On Sat, 19 Mar 2016 at 11:56 Guido van Rossum <guido at python.org
> >     <mailto:guido at python.org>> wrote:
> >
> >         All that sounds fine!
> >
> >         On Sat, Mar 19, 2016 at 11:28 AM, Stefan Krah <
> stefan at bytereef.org
> >         <mailto:stefan at bytereef.org>> wrote:
> >         > Guido van Rossum <guido <at> python.org <http://python.org>>
> writes:
> >         >> So should the preprocessing step just be s.replace('_', ''),
> or should
> >         >> it reject underscores that don't follow the rules from the PEP
> >         >> (perhaps augmented so they follow the spirit of the PEP and
> the letter
> >         >> of the IBM spec)?
> >         >>
> >         >> Honestly I think it's also fine if specifying this exactly is
> left out
> >         >> of the PEP, and handled by whoever adds this to Decimal.
> Having a PEP
> >         >> to work from for the language spec and core builtins (int(),
> float()
> >         >> complex()) is more important.
> >         >
> >         > I'd keep it simple for Decimal: Remove left and right
> whitespace (we're
> >         > already doing this), then remove underscores from the
> remaining string
> >         > (which must not contain any further whitespace), then use the
> IBM grammar.
> >         >
> >         >
> >         > We could add a clause to the PEP that only those strings that
> follow
> >         > the spirit of the PEP are guaranteed to be accepted in the
> future.
> >         >
> >         >
> >         > One reason for keeping it simple is that I would not like to
> slow down
> >         > string conversion, but thinking about two grammars is also a
> problem --
> >         > part of the string conversion in libmpdec is modeled in ACL2,
> which
> >         > would be invalidated or at least complicated with two grammars.
> >         >
> >         >
> >         >
> >         > Stefan Krah
> >         >
> >         > _______________________________________________
> >         > Python-Dev mailing list
> >         > Python-Dev at python.org <mailto:Python-Dev at python.org>
> >         > https://mail.python.org/mailman/listinfo/python-dev
> >         > Unsubscribe:
> >
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
> >
> >
> >
> >         --
> >         --Guido van Rossum (python.org/~guido <http://python.org/~guido
> >)
> >         _______________________________________________
> >         Python-Dev mailing list
> >         Python-Dev at python.org <mailto:Python-Dev at python.org>
> >         https://mail.python.org/mailman/listinfo/python-dev
> >         Unsubscribe:
> >
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
> >
> >
> >
> >
> > --
> > --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
> >
> >
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160512/d00ca922/attachment.html>

From brett at python.org  Thu May 12 20:25:51 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 13 May 2016 00:25:51 +0000
Subject: [Python-Dev] PEP 515: Underscores in Numeric Literals (revision
 3)
In-Reply-To: <CAP7+vJK2u4-r0yNY_gH-CW7kzzk4522cLB2f6Hi50pdE1UAm+w@mail.gmail.com>
References: <n9mqlh$oeb$1@ger.gmane.org>
 <CAP1=2W4P7skRpewpgxXBgHHicngMU3mJ2V1s3NVXgQL9PiCJmw@mail.gmail.com>
 <CAP7+vJLUumJiOzExqdmffJG_ywpFA9vXAWUH2Bpzvpiwcwp_RQ@mail.gmail.com>
 <ncisha$auc$1@ger.gmane.org>
 <CADiSq7etYj9Wy4++kP0z_5=-x6Pm0E9=6VASh+nYfJk8bM6PfQ@mail.gmail.com>
 <CAP7+vJ+e60gid=R8gjhmyiiNkCMC0Db4zt-Bct_qKUN29Vbr_Q@mail.gmail.com>
 <loom.20160319T181628-51@post.gmane.org>
 <CAP7+vJ+PewZ=ojQU0LzsZh0n0payOgzdOzfx6zQ73Xh9OYt5aQ@mail.gmail.com>
 <loom.20160319T191115-310@post.gmane.org>
 <CAP7+vJ+F=d9ycxZhaWkoYB6SYXf82T=Lf4QxWubwNy_FKJUACw@mail.gmail.com>
 <CAP1=2W4SM-_SBUo5YGdwyvq2N62+FC3TEgLnmYMW=2VDkV6+2A@mail.gmail.com>
 <CAP7+vJ+BicSaZiizSpjE8q2g=OGHMLgOre1tLwo3o5DN2crZ3g@mail.gmail.com>
 <nh14ea$5aa$1@ger.gmane.org>
 <CAP7+vJ+qAjuVrsrigS-vjouTCS5s4J6d2aDLjmX04MHenMev6g@mail.gmail.com>
 <CAP7+vJLd5wBRD1YwGjQuM6kTpnai9YdiLAvcHSM3yZ1vdr55Hw@mail.gmail.com>
 <CAP7+vJK2u4-r0yNY_gH-CW7kzzk4522cLB2f6Hi50pdE1UAm+w@mail.gmail.com>
Message-ID: <CAP1=2W5z-khXCm0prK35MbDjSeEwkR__ZUiD3-AM0BekM+4S=A@mail.gmail.com>

Marked as accepted in https://hg.python.org/peps/rev/a52066565cc2

On Thu, 12 May 2016 at 16:59 Guido van Rossum <gvanrossum at gmail.com> wrote:

> Is anyone going to mark the PEP as accepted?
>
> On Thu, May 12, 2016 at 8:11 AM, Guido van Rossum <gvanrossum at gmail.com>
> wrote:
>
>> OK, then PEP 515 is now officially accepted! Congratulations. Start the
>> implementation work!
>>
>> --Guido (mobile)
>> On May 11, 2016 10:33 PM, "Georg Brandl" <g.brandl at gmx.net> wrote:
>>
>> I'm happy with the latest version.
>>
>> Georg
>>
>> On 05/11/2016 06:46 PM, Guido van Rossum wrote:
>> > If the authors are happy I'll accept it right away.
>> >
>> > (I vaguely recall there's another PEP that's ready for pronouncement --
>> but
>> > which one?)
>> >
>> > On Wed, May 11, 2016 at 9:34 AM, Brett Cannon <brett at python.org
>> > <mailto:brett at python.org>> wrote:
>> >
>> >     Is there anything holding up PEP 515 at this point in terms of
>> acceptance or
>> >     implementation?
>> >
>> >     On Sat, 19 Mar 2016 at 11:56 Guido van Rossum <guido at python.org
>> >     <mailto:guido at python.org>> wrote:
>> >
>> >         All that sounds fine!
>> >
>> >         On Sat, Mar 19, 2016 at 11:28 AM, Stefan Krah <
>> stefan at bytereef.org
>> >         <mailto:stefan at bytereef.org>> wrote:
>> >         > Guido van Rossum <guido <at> python.org <http://python.org>>
>> writes:
>> >         >> So should the preprocessing step just be s.replace('_', ''),
>> or should
>> >         >> it reject underscores that don't follow the rules from the
>> PEP
>> >         >> (perhaps augmented so they follow the spirit of the PEP and
>> the letter
>> >         >> of the IBM spec)?
>> >         >>
>> >         >> Honestly I think it's also fine if specifying this exactly
>> is left out
>> >         >> of the PEP, and handled by whoever adds this to Decimal.
>> Having a PEP
>> >         >> to work from for the language spec and core builtins (int(),
>> float()
>> >         >> complex()) is more important.
>> >         >
>> >         > I'd keep it simple for Decimal: Remove left and right
>> whitespace (we're
>> >         > already doing this), then remove underscores from the
>> remaining string
>> >         > (which must not contain any further whitespace), then use the
>> IBM grammar.
>> >         >
>> >         >
>> >         > We could add a clause to the PEP that only those strings that
>> follow
>> >         > the spirit of the PEP are guaranteed to be accepted in the
>> future.
>> >         >
>> >         >
>> >         > One reason for keeping it simple is that I would not like to
>> slow down
>> >         > string conversion, but thinking about two grammars is also a
>> problem --
>> >         > part of the string conversion in libmpdec is modeled in ACL2,
>> which
>> >         > would be invalidated or at least complicated with two
>> grammars.
>> >         >
>> >         >
>> >         >
>> >         > Stefan Krah
>> >         >
>> >         > _______________________________________________
>> >         > Python-Dev mailing list
>> >         > Python-Dev at python.org <mailto:Python-Dev at python.org>
>> >         > https://mail.python.org/mailman/listinfo/python-dev
>> >         > Unsubscribe:
>> >
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>> >
>> >
>> >
>> >         --
>> >         --Guido van Rossum (python.org/~guido <http://python.org/~guido
>> >)
>> >         _______________________________________________
>> >         Python-Dev mailing list
>> >         Python-Dev at python.org <mailto:Python-Dev at python.org>
>> >         https://mail.python.org/mailman/listinfo/python-dev
>> >         Unsubscribe:
>> >
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>> >
>> >
>> >
>> >
>> > --
>> > --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
>> >
>> >
>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160513/1edd222e/attachment.html>

From ncoghlan at gmail.com  Fri May 13 00:04:24 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 13 May 2016 14:04:24 +1000
Subject: [Python-Dev] File system path PEP, part 2
In-Reply-To: <CAP1=2W6Bf3faNg7UdVgGB2ZXWF2V3nMx2jL_EiGHUnizeCtisA@mail.gmail.com>
References: <CAP1=2W6Bf3faNg7UdVgGB2ZXWF2V3nMx2jL_EiGHUnizeCtisA@mail.gmail.com>
Message-ID: <CADiSq7fBKT62U1ofiJdOKDCEx2_JW2GEidqpzZwGWiC3KDkY7Q@mail.gmail.com>

On 13 May 2016 at 06:53, Brett Cannon <brett at python.org> wrote:
> Second draft that takes Guido's comments into consideration. The biggest
> change is os.fspath() now returns whatever path.__fspath__() returns instead
> of restricting it to only str.
>
> Minor changes:
> - Renamed the C function to PyOS_FSPath()
> - Added an Implementation section with a TODO list
> - Bunch of things added to the Rejected Ideas section

+1 for this version from me, as it means we have:

- os.fsencode(obj) as the coerce-to-bytes API
- os.fspath(obj) as the str/bytes hybrid API
- os.fsdecode(obj) as the coerce-to-str API
- os.fspath(pathlib.PurePath(obj)) as the error-on-bytes API

That more strongly nudges people towards "use pathlib if you want to
ensure cross-platform friendly path handling", which is an outcome I'm
fine with.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ethan at stoneleaf.us  Fri May 13 02:07:10 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 12 May 2016 23:07:10 -0700
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <E7D6329B-B0AC-463C-B935-58A9C74144AC@sjoerdjob.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <20160512172225.GA16846@sjoerdjob.com> <5734D9E5.6060701@stoneleaf.us>
 <E7D6329B-B0AC-463C-B935-58A9C74144AC@sjoerdjob.com>
Message-ID: <57356F0E.2030407@stoneleaf.us>

On 05/12/2016 01:59 PM, Sjoerd Job Postmus wrote:
>> On 12 May 2016, at 21:30, Ethan Furman <ethan at stoneleaf.us> wrote:
>>
>> If you need bytes support for your paths, there's at least one [1] that has that support.
>
> So if I would need bytes support, I should submit a pull request to <arbitrary but awesome
 > library> which replaces usage of the stdlib pathlib with another 
variant, upon which they
 > will decline the pull request because it introduces another "useless" 
dependency.

My apologies, I thought you were talking about your own code.

As far as <arbitrary but awesome library> -- it wouldn't be awesome to 
me if I couldn't pass in the data types I need.

--
~Ethan~


From srkunze at mail.de  Fri May 13 03:53:29 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Fri, 13 May 2016 09:53:29 +0200
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <5734BCF9.6060305@stoneleaf.us>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <5734AEA6.5010308@mail.de> <5734B5C7.4060206@stoneleaf.us>
 <5734BBA9.2050807@mail.de> <5734BCF9.6060305@stoneleaf.us>
Message-ID: <573587F9.7040201@mail.de>

On 12.05.2016 19:27, Ethan Furman wrote:
> Maybe, but a bad idea for two reasons:
>
> 1) Reducing a str to the exact same str is silly; and, more importantly

Finding something silly is no technical argument.

Best,
Sven

From srkunze at mail.de  Fri May 13 03:58:13 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Fri, 13 May 2016 09:58:13 +0200
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
Message-ID: <57358915.7040606@mail.de>

On 12.05.2016 18:24, Guido van Rossum wrote:
> def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]:
>     if isinstance(p, (str, bytes)):
>         return p
>     try:
>         return p.__fspath__
>     except AttributeError:
>         raise TypeError(...)

@Brett
Would you think it makes sense to swap the str/bytes check and the 
__fspath__ check?


I just thought of a class subclassing str/bytes and defines __fspath__. 
Its __fspath__ method would be ignored currently.


Best,
Sven

From k7hoven at gmail.com  Fri May 13 04:36:04 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Fri, 13 May 2016 11:36:04 +0300
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <57358915.7040606@mail.de>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
 <57358915.7040606@mail.de>
Message-ID: <CAMiohoijCDKBuMCioY48UkU=WYy+DJqJyCL6mDBFx1gRXAv+0A@mail.gmail.com>

This has just been discussed very recently in this thread (and earlier
too). It may make sense, but it's not among our current worries.
Besides, we already added the new fspath semantics to the PEP.

While I hope Brett is asleep in his time zone, I'm guessing he will
agree (just saying this because you write "@Brett").

-- Koos


On Fri, May 13, 2016 at 10:58 AM, Sven R. Kunze <srkunze at mail.de> wrote:
> On 12.05.2016 18:24, Guido van Rossum wrote:
>>
>> def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]:
>>     if isinstance(p, (str, bytes)):
>>         return p
>>     try:
>>         return p.__fspath__
>>     except AttributeError:
>>         raise TypeError(...)
>
>
> @Brett
> Would you think it makes sense to swap the str/bytes check and the
> __fspath__ check?
>
>
> I just thought of a class subclassing str/bytes and defines __fspath__. Its
> __fspath__ method would be ignored currently.
>
>
> Best,
> Sven
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com

From srkunze at mail.de  Fri May 13 05:24:57 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Fri, 13 May 2016 11:24:57 +0200
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAMiohoijCDKBuMCioY48UkU=WYy+DJqJyCL6mDBFx1gRXAv+0A@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
 <57358915.7040606@mail.de>
 <CAMiohoijCDKBuMCioY48UkU=WYy+DJqJyCL6mDBFx1gRXAv+0A@mail.gmail.com>
Message-ID: <57359D69.6040108@mail.de>

On 13.05.2016 10:36, Koos Zevenhoven wrote:
> This has just been discussed very recently in this thread (and earlier
> too).

Could you point me to that? It seems I missed that part. I only found 
posts related to performance degradation.

However, the proposed semantics will change if the checks are swapped. 
So, my actual question is:

Is that an intended API inconsistency or a known bug supposed to be 
resolved later?

> It may make sense, but it's not among our current worries.

It might not be yours but mine. ;) That's why I was asking.

> Besides, we already added the new fspath semantics to the PEP.
>
> While I hope Brett is asleep in his time zone, I'm guessing he will
> agree (just saying this because you write "@Brett").
>
> -- Koos
>
>
> On Fri, May 13, 2016 at 10:58 AM, Sven R. Kunze <srkunze at mail.de> wrote:
>> On 12.05.2016 18:24, Guido van Rossum wrote:
>>> def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]:
>>>      if isinstance(p, (str, bytes)):
>>>          return p
>>>      try:
>>>          return p.__fspath__
>>>      except AttributeError:
>>>          raise TypeError(...)
>>
>> @Brett
>> Would you think it makes sense to swap the str/bytes check and the
>> __fspath__ check?
>>
>>
>> I just thought of a class subclassing str/bytes and defines __fspath__. Its
>> __fspath__ method would be ignored currently.
>>
>>
>> Best,
>> Sven
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com


From larry at hastings.org  Fri May 13 05:46:12 2016
From: larry at hastings.org (Larry Hastings)
Date: Fri, 13 May 2016 11:46:12 +0200
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <5734A46C.40306@stoneleaf.us>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
Message-ID: <5735A264.2020103@hastings.org>



On 05/12/2016 05:42 PM, Ethan Furman wrote:
> And even given all that, for smoother interoperability with the rest 
> of the stdlib, or at least the os.* portion, those functions would 
> still need to be upgraded to check for .path on the incoming arguments 
> -- at which point we may as well make a protocol to properly support 
> file system paths instead of relying on the rather generic attribute 
> name of 'path'.

FWIW, this shouldn't be very difficult.  Paths are already complicated 
enough that the parsing has been factored out into a PyArg_Parse 
"converter" called path_converter.  path_converter isn't used everywhere 
in posixmodule yet, but it would probably make sense to convert 
everything to use it anyway.  And then we would only need to implement 
Path object support once.  It's remotely possible that converting 
functions to use path_converter will create slight incompatibilities 
(though I don't know how, this is just FUD really) so 3.6 would be a 
good opportunity to make this change.

Optimistically,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160513/b6de2e1d/attachment.html>

From k7hoven at gmail.com  Fri May 13 05:48:33 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Fri, 13 May 2016 12:48:33 +0300
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <57359D69.6040108@mail.de>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
 <57358915.7040606@mail.de>
 <CAMiohoijCDKBuMCioY48UkU=WYy+DJqJyCL6mDBFx1gRXAv+0A@mail.gmail.com>
 <57359D69.6040108@mail.de>
Message-ID: <CAMiohojq6DDLmWbDv1TiB26CVoUFNw5mNSaiPB+w2zeOgcqOUg@mail.gmail.com>

On Fri, May 13, 2016 at 12:24 PM, Sven R. Kunze <srkunze at mail.de> wrote:
> On 13.05.2016 10:36, Koos Zevenhoven wrote:
>>
>> This has just been discussed very recently in this thread (and earlier
>> too).
>
>
> Could you point me to that? It seems I missed that part. I only found posts
> related to performance degradation.
>

This issue is coupled with the future optimization questions.

> However, the proposed semantics will change if the checks are swapped. So,
> my actual question is:
>
> Is that an intended API inconsistency or a known bug supposed to be resolved
> later?
>

Taking into account the description (and the drafted type hint), which
the documentation will probably reflect, the semantic effects of that
are very minor or nonexistent.

I do think the documentation of the protocol should say that str or
bytes subclasses should not implement __fspath__.

So no API inconsistency there.

-- Koos

From steve at pearwood.info  Fri May 13 06:19:40 2016
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 13 May 2016 20:19:40 +1000
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <20160512172225.GA16846@sjoerdjob.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <20160512172225.GA16846@sjoerdjob.com>
Message-ID: <20160513101939.GG12028@ando.pearwood.info>

On Thu, May 12, 2016 at 07:22:25PM +0200, Sjoerd Job Postmus wrote:

> The whole point of adding `os.fspath` is to make it easier to use Path
> objects. This is in an effort to gain greater adoption of pathlib in
> libraries. Now, this is an excellent idea.
> 
> However, if it were to reject bytes, that would mean that when libraries
> start to use pathlib, it would suddenly become harder for people that
> actually need bytes-support to use pathlib.

I'm not sure how it would be *harder*. Currently, if you need bytes 
paths, you can't use pathlib. That won't change, so it will be exactly 
as hard as it is now.


> Now, the claim 'if you need bytes, you should not be using pathlib` is a
> reasonable one. But what if I need bytes *and* a specific library (say,
> image handling, or a web framework, or ...). It's not up to me if that
> library uses pathlib or plain old os.path.join.

I feel this is Not Our Problem. Surely the stdlib cannot be held 
responsible for all the poor decisions made by third-party libraries? If 
a library hard-codes "\\" as their directory separator, because everyone 
uses Windows, you'd simply say "this library is not very good for my 
use-case". Likewise if it rejects bytes paths, then:

- perhaps that library is not fit for your purpose;

- perhaps you can use it, but with the limitation that you can no 
  longer support all paths, but only the ones which can be 
  accessed via a string path;

- perhaps you can build a compatibility layer that lets you use 
  both bytes and the library.

I don't doubt that all three options are unpleasant, but I don't see how 
they are the fault of this PEP. This PEP supports Path-Like objects that 
return bytes paths. Its up to library authors to support bytes paths, or 
not, as they see fit.



-- 
Steve

From rosuav at gmail.com  Fri May 13 06:54:05 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 13 May 2016 20:54:05 +1000
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <20160513101939.GG12028@ando.pearwood.info>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <20160512172225.GA16846@sjoerdjob.com>
 <20160513101939.GG12028@ando.pearwood.info>
Message-ID: <CAPTjJmqszcTwh2MDigY1JoeP6cyWvAVpyKvY0v09z-JirEmHwg@mail.gmail.com>

On Fri, May 13, 2016 at 8:19 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> I feel this is Not Our Problem. Surely the stdlib cannot be held
> responsible for all the poor decisions made by third-party libraries? If
> a library hard-codes "\\" as their directory separator, because everyone
> uses Windows, you'd simply say "this library is not very good for my
> use-case". Likewise if it rejects bytes paths, then:
>
> - perhaps that library is not fit for your purpose;
>
> - perhaps you can use it, but with the limitation that you can no
>   longer support all paths, but only the ones which can be
>   accessed via a string path;
>
> - perhaps you can build a compatibility layer that lets you use
>   both bytes and the library.

AFAICT, the compatibility layer would simply decode the bytes using
surrogateescape handling, which should round-trip anything.

Or am I wrong here somewhere?

ChrisA

From steve at pearwood.info  Fri May 13 07:00:14 2016
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 13 May 2016 21:00:14 +1000
Subject: [Python-Dev] File system path PEP, part 2
In-Reply-To: <CAP1=2W6Bf3faNg7UdVgGB2ZXWF2V3nMx2jL_EiGHUnizeCtisA@mail.gmail.com>
References: <CAP1=2W6Bf3faNg7UdVgGB2ZXWF2V3nMx2jL_EiGHUnizeCtisA@mail.gmail.com>
Message-ID: <20160513110012.GH12028@ando.pearwood.info>

On Thu, May 12, 2016 at 08:53:12PM +0000, Brett Cannon wrote:

> Second draft that takes Guido's comments into consideration. The biggest
> change is os.fspath() now returns whatever path.__fspath__() returns
> instead of restricting it to only str.

Counter suggestion:

- __fspath__() method may return either bytes or str (no change 
  from the PEP as it stands now);

- but os.fspath() will only return str;

- and os.fspathb() will only return bytes;

- there is no os function that returns "str or bytes, I don't 
  care which". (If you really need that, call __fspath__ directly.)

Note that this differs from the already rejected suggestion that there 
should be two dunder methods, __fspath__() and __fspathb__().

Why? 

(1) Normally, the caller knows whether they want str or bytes. (That's 
been my experience, you may disagree.) If so, and they call os.fspath() 
expecting a str, they won't be surprised by it returning bytes. And visa 
versa for when you expect a bytes path.

(2) This behaviour will match that of os.{environ[b],getcwd[b],getenv[b]}.

Cons:

(3) Polymorphic code that truly doesn't care whether it gets bytes or 
str will have a slightly less convenient way of getting it, namely by 
calling __fspath__() itself, instead of os.fspath().




A few other comments below:


> builtins
> ''''''''
> 
> ``open()`` [#builtins-open]_ will be updated to accept path objects as
> well as continue to accept ``str`` and ``bytes``.

I think it is a bit confusing to refer to "path objects", as that seems 
like you are referring only to pathlib.Path objects. It took me far too 
long to realise that here you mean generic path-like objects that obey 
the __fspath__ protocol rather than a specific concrete class.

Since the ABC is called "PathLike", I suggest we refer to "path-like 
objects" rather than "path objects", both in the PEP and in the Python 
docs for this protocol.




>     def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str, bytes]:
>         """Return the string representation of the path.
> 
>         If str or bytes is passed in, it is returned unchanged.
>         """

I've already suggested a change to this, above, but independent of that, 
a minor technical query:

>         try:
>             return path.__fspath__()

Would I be right in saying that in practice this will actually end up 
being type(path).__fspath__() to match the behaviour of all(?) other 
dunder methods?



-- 
Steve

From rosuav at gmail.com  Fri May 13 07:28:46 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 13 May 2016 21:28:46 +1000
Subject: [Python-Dev] File system path PEP, part 2
In-Reply-To: <20160513110012.GH12028@ando.pearwood.info>
References: <CAP1=2W6Bf3faNg7UdVgGB2ZXWF2V3nMx2jL_EiGHUnizeCtisA@mail.gmail.com>
 <20160513110012.GH12028@ando.pearwood.info>
Message-ID: <CAPTjJmoyo2XjuisA96+D6Mhs1BAWAp6eP+QNMKWOhJaRk3h98A@mail.gmail.com>

On Fri, May 13, 2016 at 9:00 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> Cons:
> (3) Polymorphic code that truly doesn't care whether it gets bytes or
> str will have a slightly less convenient way of getting it, namely by
> calling __fspath__() itself, instead of os.fspath().

I don't like this; it goes against the general principle that dunders
are for defining, not calling. Generally, a given dunder method has
approximately one call site, eg __reduce__ in pickle.py, and everyone
else defines it. (You might call super's dunder in the definition of
your own, but that's still defining it, not calling it.) Having an
official statement that it's appropriate to call a dunder confuses
this. So this isn't a "slightly less convenient way", it's a bad way
(IMO), and this is a very strong con.

ChrisA

From ncoghlan at gmail.com  Fri May 13 07:56:34 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 13 May 2016 21:56:34 +1000
Subject: [Python-Dev] File system path PEP, part 2
In-Reply-To: <20160513110012.GH12028@ando.pearwood.info>
References: <CAP1=2W6Bf3faNg7UdVgGB2ZXWF2V3nMx2jL_EiGHUnizeCtisA@mail.gmail.com>
 <20160513110012.GH12028@ando.pearwood.info>
Message-ID: <CADiSq7eA6PcFPF7zT_p=mxJBcHJR2bw1zh2GzsraTyuni=uJWw@mail.gmail.com>

On 13 May 2016 at 21:00, Steven D'Aprano <steve at pearwood.info> wrote:
> On Thu, May 12, 2016 at 08:53:12PM +0000, Brett Cannon wrote:
>
>> Second draft that takes Guido's comments into consideration. The biggest
>> change is os.fspath() now returns whatever path.__fspath__() returns
>> instead of restricting it to only str.
>
> Counter suggestion:
>
> - __fspath__() method may return either bytes or str (no change
>   from the PEP as it stands now);
>
> - but os.fspath() will only return str;
>
> - and os.fspathb() will only return bytes;

We don't have any use cases for a bytes-only API - only str-or-bytes
(in the polymorphic low level functions) and str-only (in newer high
level APIs like pathlib).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From k7hoven at gmail.com  Fri May 13 09:51:44 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Fri, 13 May 2016 16:51:44 +0300
Subject: [Python-Dev] File system path PEP, part 2
In-Reply-To: <CAP1=2W6Bf3faNg7UdVgGB2ZXWF2V3nMx2jL_EiGHUnizeCtisA@mail.gmail.com>
References: <CAP1=2W6Bf3faNg7UdVgGB2ZXWF2V3nMx2jL_EiGHUnizeCtisA@mail.gmail.com>
Message-ID: <CAMiohohJue5GdjpMbA_Zq2=afKVLPp9RQna_ovYWfagR=7hhQg@mail.gmail.com>

Thanks Brett!

Now one thing is that, despite your suggestion, I had not added myself
as an author in my big pull request. Originally, it was because I
simply forgot to copy and paste it when I split my edits into separate
commits ;-). Sorry about that (not sure if you care though, and I've
been defending the PEP regardless).

Anyway, especially now that my main worry regarding the open questions
has been resolved, I would be more than happy to have my name on it.
So Brett, could you add me as author? (Koos Zevenhoven and
k7hoven at gmail.com will be fine)

It looks like this is finally happening :)

-- Koos

On Thu, May 12, 2016 at 11:53 PM, Brett Cannon <brett at python.org> wrote:
> Second draft that takes Guido's comments into consideration. The biggest
> change is os.fspath() now returns whatever path.__fspath__() returns instead
> of restricting it to only str.
>
> Minor changes:
> - Renamed the C function to PyOS_FSPath()
> - Added an Implementation section with a TODO list
> - Bunch of things added to the Rejected Ideas section
>

From contrebasse at gmail.com  Fri May 13 10:23:09 2016
From: contrebasse at gmail.com (Joseph Martinot-Lagarde)
Date: Fri, 13 May 2016 14:23:09 +0000 (UTC)
Subject: [Python-Dev] File system path PEP, part 2
References: <CAP1=2W6Bf3faNg7UdVgGB2ZXWF2V3nMx2jL_EiGHUnizeCtisA@mail.gmail.com>
 <20160513110012.GH12028@ando.pearwood.info>
Message-ID: <loom.20160513T161950-222@post.gmane.org>

> - there is no os function that returns "str or bytes, I don't 
>   care which". (If you really need that, call __fspath__ directly.)

os.fspath() in the PEP works when given str or bytes directly, but those
don't have a __fspath__ method, so directly calling the dunder method is not
equivalent to calling os.fspath(). The whole point of having os.fspath() is
to pass str or paths (or bytes) and get an appropriate object for lew-level
functions.


From sjoerdjob at sjoerdjob.com  Fri May 13 02:22:12 2016
From: sjoerdjob at sjoerdjob.com (Sjoerd Job Postmus)
Date: Fri, 13 May 2016 08:22:12 +0200
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <57356F0E.2030407@stoneleaf.us>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <20160512172225.GA16846@sjoerdjob.com> <5734D9E5.6060701@stoneleaf.us>
 <E7D6329B-B0AC-463C-B935-58A9C74144AC@sjoerdjob.com>
 <57356F0E.2030407@stoneleaf.us>
Message-ID: <6484E0A4-BF91-4643-8224-4E69D8AE354B@sjoerdjob.com>



> On 13 May 2016, at 08:07, Ethan Furman <ethan at stoneleaf.us> wrote:
> 
> On 05/12/2016 01:59 PM, Sjoerd Job Postmus wrote:
>>> On 12 May 2016, at 21:30, Ethan Furman <ethan at stoneleaf.us> wrote:
>>> 
>>> If you need bytes support for your paths, there's at least one [1] that has that support.
>> 
>> So if I would need bytes support, I should submit a pull request to <arbitrary but awesome
> > library> which replaces usage of the stdlib pathlib with another variant, upon which they
> > will decline the pull request because it introduces another "useless" dependency.
> 
> My apologies, I thought you were talking about your own code.
> 
> As far as <arbitrary but awesome library> -- it wouldn't be awesome to me if I couldn't pass in the data types I need.
> 
> --
> ~Ethan~

I just hope you are right. Thank you (and Brett) for clearing things up for me.

My apologies if I came off as (a bit) harsh.

From random832 at fastmail.com  Fri May 13 11:25:26 2016
From: random832 at fastmail.com (Random832)
Date: Fri, 13 May 2016 11:25:26 -0400
Subject: [Python-Dev] File system path PEP, part 2
In-Reply-To: <20160513110012.GH12028@ando.pearwood.info>
References: <CAP1=2W6Bf3faNg7UdVgGB2ZXWF2V3nMx2jL_EiGHUnizeCtisA@mail.gmail.com>
 <20160513110012.GH12028@ando.pearwood.info>
Message-ID: <1463153126.2155923.607016537.1E02B685@webmail.messagingengine.com>

On Fri, May 13, 2016, at 07:00, Steven D'Aprano wrote:
> - but os.fspath() will only return str;
> 
> - and os.fspathb() will only return bytes;

And raise an exception if __fspath__ returns the other, I suppose.
What's the use case for these functions? When would I call them rather
than fsdecode and fsencode? (assuming the latter will support the path
protocol - I've got more objections if they're not going to)

Also, what happens if you pass a string to os.fspath? Statements like
"str will not implement __fspath__" have thus far been light on details
of what functions will accept "str or
path-like-object-whose-__fspath__returns-str" and which ones will only
accept the latter (and what, if any, will continue to only accept the
former). If no-one's supposed to directly call dunder methods, and
os.fspath accepts str, then what difference does it make whether this is
implemented by having a special case within os.fspath or by calling
str.__fspath__?

From brett at python.org  Fri May 13 11:29:29 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 13 May 2016 15:29:29 +0000
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <57359D69.6040108@mail.de>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
 <57358915.7040606@mail.de>
 <CAMiohoijCDKBuMCioY48UkU=WYy+DJqJyCL6mDBFx1gRXAv+0A@mail.gmail.com>
 <57359D69.6040108@mail.de>
Message-ID: <CAP1=2W62hAtWN96aG0UO0nm0UXY1Tm3MrNX_UefR0eb=EW4BSw@mail.gmail.com>

On Fri, 13 May 2016 at 02:25 Sven R. Kunze <srkunze at mail.de> wrote:

> On 13.05.2016 10:36, Koos Zevenhoven wrote:
> > This has just been discussed very recently in this thread (and earlier
> > too).
>
> Could you point me to that? It seems I missed that part. I only found
> posts related to performance degradation.
>
> However, the proposed semantics will change if the checks are swapped.
> So, my actual question is:
>
> Is that an intended API inconsistency or a known bug supposed to be
> resolved later?
>

Purposeful change. It was what I had in my head after I posted my "groups"
breakdown email and what Guido suggested as well independently. This helps
alleviate any perf worries as type checks in C are pointer checks that are
cheap to make compared to attribute lookup. And even if we do get the
community to move on to path objects the check is still cheap enough to not
worry. And finally, this is simply how we have almost always coded this
kind of special-casing at the C level throughout Python and the stdlib, so
it now is more inline with standard coding practices than the original
proposal in the PEP.


>
> > It may make sense, but it's not among our current worries.
>
> It might not be yours but mine. ;) That's why I was asking.
>

Yes, I was very much asleep. :)

-Brett


>
> > Besides, we already added the new fspath semantics to the PEP.
> >
> > While I hope Brett is asleep in his time zone, I'm guessing he will
> > agree (just saying this because you write "@Brett").
> >
> > -- Koos
> >
> >
> > On Fri, May 13, 2016 at 10:58 AM, Sven R. Kunze <srkunze at mail.de> wrote:
> >> On 12.05.2016 18:24, Guido van Rossum wrote:
> >>> def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]:
> >>>      if isinstance(p, (str, bytes)):
> >>>          return p
> >>>      try:
> >>>          return p.__fspath__
> >>>      except AttributeError:
> >>>          raise TypeError(...)
> >>
> >> @Brett
> >> Would you think it makes sense to swap the str/bytes check and the
> >> __fspath__ check?
> >>
> >>
> >> I just thought of a class subclassing str/bytes and defines __fspath__.
> Its
> >> __fspath__ method would be ignored currently.
> >>
> >>
> >> Best,
> >> Sven
> >>
> >> _______________________________________________
> >> Python-Dev mailing list
> >> Python-Dev at python.org
> >> https://mail.python.org/mailman/listinfo/python-dev
> >> Unsubscribe:
> >> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160513/322c3965/attachment.html>

From brett at python.org  Fri May 13 11:43:29 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 13 May 2016 15:43:29 +0000
Subject: [Python-Dev] File system path PEP, part 2
In-Reply-To: <20160513110012.GH12028@ando.pearwood.info>
References: <CAP1=2W6Bf3faNg7UdVgGB2ZXWF2V3nMx2jL_EiGHUnizeCtisA@mail.gmail.com>
 <20160513110012.GH12028@ando.pearwood.info>
Message-ID: <CAP1=2W5VLshb9XK9km3u0jGMKZ5Gs=b0NyrG4n9fb2pV6COTpA@mail.gmail.com>

On Fri, 13 May 2016 at 04:00 Steven D'Aprano <steve at pearwood.info> wrote:

> On Thu, May 12, 2016 at 08:53:12PM +0000, Brett Cannon wrote:
>
> > Second draft that takes Guido's comments into consideration. The biggest
> > change is os.fspath() now returns whatever path.__fspath__() returns
> > instead of restricting it to only str.
>
> Counter suggestion:
>
> - __fspath__() method may return either bytes or str (no change
>   from the PEP as it stands now);
>
> - but os.fspath() will only return str;
>
> - and os.fspathb() will only return bytes;
>
> - there is no os function that returns "str or bytes, I don't
>   care which". (If you really need that, call __fspath__ directly.)
>
> Note that this differs from the already rejected suggestion that there
> should be two dunder methods, __fspath__() and __fspathb__().
>
> Why?
>
> (1) Normally, the caller knows whether they want str or bytes. (That's
> been my experience, you may disagree.) If so, and they call os.fspath()
> expecting a str, they won't be surprised by it returning bytes. And visa
> versa for when you expect a bytes path.
>
> (2) This behaviour will match that of os.{environ[b],getcwd[b],getenv[b]}.
>
> Cons:
>
> (3) Polymorphic code that truly doesn't care whether it gets bytes or
> str will have a slightly less convenient way of getting it, namely by
> calling __fspath__() itself, instead of os.fspath().
>

I prefer what's in the PEP. I get where you coming from, Steven, but I
don't think it will be common enough to worry about. Think of os.fspath()
like next() where it truly is a very minor convenience function that
happens to special-case str and bytes.


>
>
>
>
> A few other comments below:
>
>
> > builtins
> > ''''''''
> >
> > ``open()`` [#builtins-open]_ will be updated to accept path objects as
> > well as continue to accept ``str`` and ``bytes``.
>
> I think it is a bit confusing to refer to "path objects", as that seems
> like you are referring only to pathlib.Path objects. It took me far too
> long to realise that here you mean generic path-like objects that obey
> the __fspath__ protocol rather than a specific concrete class.
>
> Since the ABC is called "PathLike", I suggest we refer to "path-like
> objects" rather than "path objects", both in the PEP and in the Python
> docs for this protocol.
>

I went back and forth with this in my head while writing the PEP. The
problem with making "path-like" mean "objects implementing the PathLike
ABC" becomes how do you refer to an argument of a function that accepts
anything os.fspath() does (i.e. PathLike, str, and bytes)?


>
>
>
>
> >     def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str,
> bytes]:
> >         """Return the string representation of the path.
> >
> >         If str or bytes is passed in, it is returned unchanged.
> >         """
>
> I've already suggested a change to this, above, but independent of that,
> a minor technical query:
>
> >         try:
> >             return path.__fspath__()
>
> Would I be right in saying that in practice this will actually end up
> being type(path).__fspath__() to match the behaviour of all(?) other
> dunder methods?
>

I wasn't planning on it because for most types the accessing of the method
directly off of the type for magic methods is because of some special
struct field at the C level that we're pulling from. Since we're not
planning to have an equivalent struct field I don't see any need to do the
extra work of avoiding the instance participating in method lookup.
Obviously if people disagree for some reason then please let me know (maybe
for perf by avoiding the overhead of checking for the method on the
instance?).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160513/475e23e3/attachment.html>

From srkunze at mail.de  Fri May 13 12:06:17 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Fri, 13 May 2016 18:06:17 +0200
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAMiohojq6DDLmWbDv1TiB26CVoUFNw5mNSaiPB+w2zeOgcqOUg@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
 <57358915.7040606@mail.de>
 <CAMiohoijCDKBuMCioY48UkU=WYy+DJqJyCL6mDBFx1gRXAv+0A@mail.gmail.com>
 <57359D69.6040108@mail.de>
 <CAMiohojq6DDLmWbDv1TiB26CVoUFNw5mNSaiPB+w2zeOgcqOUg@mail.gmail.com>
Message-ID: <5735FB79.1010201@mail.de>

On 13.05.2016 11:48, Koos Zevenhoven wrote:
> This issue is coupled with the future optimization questions.

AFAIC coupling API design to optimization is called premature optimization.

>> However, the proposed semantics will change if the checks are swapped. So,
>> my actual question is:
>>
>> Is that an intended API inconsistency or a known bug supposed to be resolved
>> later?
>>
> Taking into account the description (and the drafted type hint), which
> the documentation will probably reflect, the semantic effects of that
> are very minor or nonexistent.

 From your perspective. As far as I remember, one goal of this proposal 
was to avoid wallet gardens. During the lengthy discussion on 
python-ideas people brought up that some third-party libs indeed 
subclass from str. They are currently locked out.

> I do think the documentation of the protocol should say that str or
> bytes subclasses should not implement __fspath__.

Indeed. Just one minor note here: str or bytes subclasses *can* 
implement __fspath__ and currently it will be *ignored*. Maybe that 
changes in the future. So, that's the reason it should not be implemented.

> So no API inconsistency there.

API consistency is not defined by docs-matching-implementation but by 
implementation-matching-expectations.


Best,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160513/e93219dc/attachment.html>

From status at bugs.python.org  Fri May 13 12:08:41 2016
From: status at bugs.python.org (Python tracker)
Date: Fri, 13 May 2016 18:08:41 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20160513160841.22A8F56AB1@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2016-05-06 - 2016-05-13)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5501 ( -3)
  closed 33270 (+47)
  total  38771 (+44)

Open issues with patches: 2383 


Issues opened (28)
==================

#22570: Better stdlib support for Path objects
http://bugs.python.org/issue22570  reopened by gvanrossum

#26857: gethostbyname_r() is broken on android
http://bugs.python.org/issue26857  reopened by xdegaye

#26971: python v3.5.1: sys.paths not respecting DESTDIRS and DESTSHARE
http://bugs.python.org/issue26971  opened by yaro-yaro

#26972: mistakes in docstrings in the import machinery
http://bugs.python.org/issue26972  opened by Oren Milman

#26974: Crash in Decimal.from_float
http://bugs.python.org/issue26974  opened by serhiy.storchaka

#26975: Decimal.from_float works incorrectly for non-binary floats
http://bugs.python.org/issue26975  opened by serhiy.storchaka

#26978: Implement pathlib.Path.link (Using os.link)
http://bugs.python.org/issue26978  opened by cool-RR

#26979: The danger of PyType_FromSpec()
http://bugs.python.org/issue26979  opened by serhiy.storchaka

#26980: The path argument of asyncio.BaseEventLoop.create_unix_connect
http://bugs.python.org/issue26980  opened by texttheater

#26981: add compatibility shim for enum34 backport
http://bugs.python.org/issue26981  opened by ethan.furman

#26982: Clarify forward annotations in PEP 484
http://bugs.python.org/issue26982  opened by James.Tatum

#26983: float() can return not exact float instance
http://bugs.python.org/issue26983  opened by serhiy.storchaka

#26984: int() can return not exact int instance
http://bugs.python.org/issue26984  opened by serhiy.storchaka

#26985: Information about CodeType in inspect documentation is outdate
http://bugs.python.org/issue26985  opened by xiang.zhang

#26988: Add AutoNumberedEnum to stdlib
http://bugs.python.org/issue26988  opened by John Hagen

#26990: file.tell affect decoding
http://bugs.python.org/issue26990  opened by mfmain

#26991: Possible reference leak in MAKE_FUNCTION
http://bugs.python.org/issue26991  opened by xiang.zhang

#26992: 64-bit Python 2.7.11 hangs in 64-bit Windows 10 - CMD and Git 
http://bugs.python.org/issue26992  opened by sam.taylor.bs

#26993: Copy idlelib *.py files with new names
http://bugs.python.org/issue26993  opened by terry.reedy

#26995: Add tests for parsing float and object arguments
http://bugs.python.org/issue26995  opened by serhiy.storchaka

#26997: Docs for pdb should note that __future__ magic doesn't work
http://bugs.python.org/issue26997  opened by Martin Jones

#27002: Support different modes in posixpath.realpath()
http://bugs.python.org/issue27002  opened by serhiy.storchaka

#27004: Handle script shbang options
http://bugs.python.org/issue27004  opened by opoplawski

#27006: C implementation of Decimal.from_float() bypasses __new__ and 
http://bugs.python.org/issue27006  opened by serhiy.storchaka

#27007: Alternate constructors bytes.fromhex() and bytearray.fromhex()
http://bugs.python.org/issue27007  opened by serhiy.storchaka

#27010: email library could "recover" from bad mime boundary like (som
http://bugs.python.org/issue27010  opened by Fedele Mantuano

#27012: Rename the posix module to _os
http://bugs.python.org/issue27012  opened by brett.cannon

#27014: maximum recursion depth when using typing options
http://bugs.python.org/issue27014  opened by Roy Shmueli



Most recent 15 issues with no replies (15)
==========================================

#27014: maximum recursion depth when using typing options
http://bugs.python.org/issue27014

#27006: C implementation of Decimal.from_float() bypasses __new__ and 
http://bugs.python.org/issue27006

#27002: Support different modes in posixpath.realpath()
http://bugs.python.org/issue27002

#26997: Docs for pdb should note that __future__ magic doesn't work
http://bugs.python.org/issue26997

#26995: Add tests for parsing float and object arguments
http://bugs.python.org/issue26995

#26990: file.tell affect decoding
http://bugs.python.org/issue26990

#26982: Clarify forward annotations in PEP 484
http://bugs.python.org/issue26982

#26980: The path argument of asyncio.BaseEventLoop.create_unix_connect
http://bugs.python.org/issue26980

#26979: The danger of PyType_FromSpec()
http://bugs.python.org/issue26979

#26978: Implement pathlib.Path.link (Using os.link)
http://bugs.python.org/issue26978

#26964: Incorrect documentation for `-u`-flag
http://bugs.python.org/issue26964

#26955: Implement equivalent to `pip.locations.distutils_scheme` in di
http://bugs.python.org/issue26955

#26954: Add Guido's rejection notice to the "with" FAQ
http://bugs.python.org/issue26954

#26942: android: test_ctypes crashes on armv7
http://bugs.python.org/issue26942

#26937: android: test_tarfile fails
http://bugs.python.org/issue26937



Most recent 15 issues waiting for review (15)
=============================================

#27007: Alternate constructors bytes.fromhex() and bytearray.fromhex()
http://bugs.python.org/issue27007

#27002: Support different modes in posixpath.realpath()
http://bugs.python.org/issue27002

#26995: Add tests for parsing float and object arguments
http://bugs.python.org/issue26995

#26991: Possible reference leak in MAKE_FUNCTION
http://bugs.python.org/issue26991

#26985: Information about CodeType in inspect documentation is outdate
http://bugs.python.org/issue26985

#26983: float() can return not exact float instance
http://bugs.python.org/issue26983

#26982: Clarify forward annotations in PEP 484
http://bugs.python.org/issue26982

#26974: Crash in Decimal.from_float
http://bugs.python.org/issue26974

#26972: mistakes in docstrings in the import machinery
http://bugs.python.org/issue26972

#26970: Replace OpenSSL's CPRNG with system entropy source
http://bugs.python.org/issue26970

#26964: Incorrect documentation for `-u`-flag
http://bugs.python.org/issue26964

#26959: pickle: respect dispatch for functions again
http://bugs.python.org/issue26959

#26948: Simplify PyImport_ImportModuleLevelObject: avoid temporary tup
http://bugs.python.org/issue26948

#26939: android: test_functools hangs on armv7
http://bugs.python.org/issue26939

#26935: android: test_os fails
http://bugs.python.org/issue26935



Top 10 most discussed issues (10)
=================================

#26969: ascynio should provide a policy to address pass-loop-everywher
http://bugs.python.org/issue26969  36 msgs

#26439: ctypes.util.find_library fails when ldconfig/glibc not availab
http://bugs.python.org/issue26439  12 msgs

#26993: Copy idlelib *.py files with new names
http://bugs.python.org/issue26993   9 msgs

#26906: format(object.__reduce__) fails intermittently
http://bugs.python.org/issue26906   8 msgs

#6422: timeit called from within Python should allow autoranging
http://bugs.python.org/issue6422   7 msgs

#18531: Undocumented different between METH_KEYWORDS and **kws
http://bugs.python.org/issue18531   7 msgs

#26632: __all__ decorator
http://bugs.python.org/issue26632   7 msgs

#26983: float() can return not exact float instance
http://bugs.python.org/issue26983   7 msgs

#26039: More flexibility in zipfile write interface
http://bugs.python.org/issue26039   6 msgs

#26369: unicode.decode and str.encode are unnecessarily confusing for 
http://bugs.python.org/issue26369   6 msgs



Issues closed (45)
==================

#5062: Rlcompleter.Completer does not use __dir__ magic method
http://bugs.python.org/issue5062  closed by martin.panter

#11643: Use |version| instead of X.Y in the doc
http://bugs.python.org/issue11643  closed by serhiy.storchaka

#15721: PEP 3121, 384 Refactoring applied to tkinter module
http://bugs.python.org/issue15721  closed by serhiy.storchaka

#17765: weakref.ref ignores a 'callback' keyword argument
http://bugs.python.org/issue17765  closed by serhiy.storchaka

#18864: Implementation for PEP 451 (importlib.machinery.ModuleSpec)
http://bugs.python.org/issue18864  closed by eric.snow

#19444: mmap.mmap() allocates a file descriptor that isn't CLOEXEC
http://bugs.python.org/issue19444  closed by martin.panter

#21012: Figure out how to best leverage pip in devinabox
http://bugs.python.org/issue21012  closed by brett.cannon

#22143: rlcompleter.Completer has duplicate matches
http://bugs.python.org/issue22143  closed by martin.panter

#22274: subprocess.Popen(stderr=STDOUT) fails to redirect subprocess s
http://bugs.python.org/issue22274  closed by martin.panter

#23213: subprocess communicate() hangs when stderr isn't closed
http://bugs.python.org/issue23213  closed by martin.panter

#23640: int.from_bytes() is broken for subclasses
http://bugs.python.org/issue23640  closed by serhiy.storchaka

#23815: Segmentation fault when create _tkinter objects
http://bugs.python.org/issue23815  closed by serhiy.storchaka

#23921: Standardize documentation whitespace, formatting
http://bugs.python.org/issue23921  closed by serhiy.storchaka

#24550: maxint on 64 bit platforms breaks os.read
http://bugs.python.org/issue24550  closed by serhiy.storchaka

#25745: Reference leaks in test_curses
http://bugs.python.org/issue25745  closed by serhiy.storchaka

#26352: getpass incorrectly displays password prompt on stderr on fall
http://bugs.python.org/issue26352  closed by serhiy.storchaka

#26512: Vocabulary: Using "integral" in library/stdtypes.html
http://bugs.python.org/issue26512  closed by martin.panter

#26708: Constify C string pointers in the posix module
http://bugs.python.org/issue26708  closed by serhiy.storchaka

#26736: Use HTTPS protocol in links
http://bugs.python.org/issue26736  closed by serhiy.storchaka

#26881: modulefinder should reuse the dis module
http://bugs.python.org/issue26881  closed by serhiy.storchaka

#26889: Improve Doc/library/xmlrpc.client.rst
http://bugs.python.org/issue26889  closed by serhiy.storchaka

#26892: debuglevel not honored in urllib
http://bugs.python.org/issue26892  closed by orsenthil

#26911: lib2to3/tests/pytree_idempotency.py has broken imports
http://bugs.python.org/issue26911  closed by terry.reedy

#26912: Broken imports in test/test_email/torture_test.py, test_asian_
http://bugs.python.org/issue26912  closed by terry.reedy

#26924: android: test_concurrent_futures fails
http://bugs.python.org/issue26924  closed by berker.peksag

#26925: android: test_multiprocessing_main_handling fails
http://bugs.python.org/issue26925  closed by xdegaye

#26960: urllib hangs when closing connection
http://bugs.python.org/issue26960  closed by python-dev

#26973: Dict View binops permit non-set types
http://bugs.python.org/issue26973  closed by r.david.murray

#26976: pathlib equivalent for os.path.realpath()
http://bugs.python.org/issue26976  closed by serhiy.storchaka

#26977: Unnecessary line in statistics.pvariance
http://bugs.python.org/issue26977  closed by steven.daprano

#26986: Enhance PyFunction_New documentation
http://bugs.python.org/issue26986  closed by python-dev

#26987: Comment/implementation disagreement in Thread._bootstrap_inner
http://bugs.python.org/issue26987  closed by python-dev

#26989: error in installation of ez_setup.py
http://bugs.python.org/issue26989  closed by benjamin.peterson

#26994: unexpected behavior for booleans in argparse
http://bugs.python.org/issue26994  closed by r.david.murray

#26996: Add secrets module as per PEP 506.
http://bugs.python.org/issue26996  closed by steven.daprano

#26998: Unable to import "site" in an embedded application
http://bugs.python.org/issue26998  closed by steve.dower

#26999: Add child separator keyword to logging.basicConfig and use it 
http://bugs.python.org/issue26999  closed by vinay.sajip

#27000: improve document of filter
http://bugs.python.org/issue27000  closed by r.david.murray

#27001: Python Interface X-Plane 10.45
http://bugs.python.org/issue27001  closed by r.david.murray

#27003: Python 3.5.1 fails at HTTPSTest with SSL CERT error
http://bugs.python.org/issue27003  closed by martin.panter

#27005: Faster float.fromhex()
http://bugs.python.org/issue27005  closed by serhiy.storchaka

#27008: error message of ord is not intuitive
http://bugs.python.org/issue27008  closed by r.david.murray

#27009: multiprocessing Connection data length limited to max(int32)
http://bugs.python.org/issue27009  closed by serhiy.storchaka

#27011: Identation error
http://bugs.python.org/issue27011  closed by python-dev

#27013: IDLE completion for dict keys
http://bugs.python.org/issue27013  closed by terry.reedy

From guido at python.org  Fri May 13 12:21:09 2016
From: guido at python.org (Guido van Rossum)
Date: Fri, 13 May 2016 09:21:09 -0700
Subject: [Python-Dev] File system path PEP, part 2
In-Reply-To: <20160513110012.GH12028@ando.pearwood.info>
References: <CAP1=2W6Bf3faNg7UdVgGB2ZXWF2V3nMx2jL_EiGHUnizeCtisA@mail.gmail.com>
 <20160513110012.GH12028@ando.pearwood.info>
Message-ID: <CAP7+vJKYi6qrQN6um0cfPT5hxgNQBr5ie=aVFWq3Po4ex_xTKA@mail.gmail.com>

On Fri, May 13, 2016 at 4:00 AM, Steven D'Aprano <steve at pearwood.info>
wrote:

> On Thu, May 12, 2016 at 08:53:12PM +0000, Brett Cannon wrote:
>
> > Second draft that takes Guido's comments into consideration. The biggest
> > change is os.fspath() now returns whatever path.__fspath__() returns
> > instead of restricting it to only str.
>
> Counter suggestion:
>
> - __fspath__() method may return either bytes or str (no change
>   from the PEP as it stands now);
>
> - but os.fspath() will only return str;
>
> - and os.fspathb() will only return bytes;
>
> - there is no os function that returns "str or bytes, I don't
>   care which". (If you really need that, call __fspath__ directly.)
>
> Note that this differs from the already rejected suggestion that there
> should be two dunder methods, __fspath__() and __fspathb__().
>
> Why?
>
> (1) Normally, the caller knows whether they want str or bytes. (That's
> been my experience, you may disagree.) If so, and they call os.fspath()
> expecting a str, they won't be surprised by it returning bytes. And visa
> versa for when you expect a bytes path.
>
> (2) This behaviour will match that of os.{environ[b],getcwd[b],getenv[b]}.
>
> Cons:
>
> (3) Polymorphic code that truly doesn't care whether it gets bytes or
> str will have a slightly less convenient way of getting it, namely by
> calling __fspath__() itself, instead of os.fspath().
>

More cons:

- It would be confusing that there'd be no direct corresponding between
os.fspath(x) and x.__fspath__(), unlike for most other dunders (next(x) ->
x.__next__(), iter(x) -> x.__iter__(), and so on).

- Your examples like os.getcwd() have no input to determine whether to
return bytes or str, so for those we use an alternate name to request
bytes. However for most os methods the same method handles both, e.g.
os.listdir(b'.') will return a list of bytes objects while os.listdir('.')
will return a list of strings. os.fspath() firmly falls in the latter camp:
it takes an input whose __fspath__() method will return either str or
bytes, so that's all the guidance it needs.

Really, if you want bytes, you should use os.fsencode(); if you want
strings, use os.fsencode(); if you want to be polymorphic, use os.fspath()
and check the type it returns. I find the case where you'd explicitly want
to exclude bytes unusual; Nick already proposed pathlib.Path(os.fspath(x))
for that.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160513/5160f1d6/attachment.html>

From k7hoven at gmail.com  Fri May 13 12:24:54 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Fri, 13 May 2016 19:24:54 +0300
Subject: [Python-Dev] File system path PEP, part 2
In-Reply-To: <20160513110012.GH12028@ando.pearwood.info>
References: <CAP1=2W6Bf3faNg7UdVgGB2ZXWF2V3nMx2jL_EiGHUnizeCtisA@mail.gmail.com>
 <20160513110012.GH12028@ando.pearwood.info>
Message-ID: <CAMiohoi+dWRXqv3SH6vs3Z8Bk93oi-p=giZrMBXFxRFYFCgD=g@mail.gmail.com>

On Fri, May 13, 2016 at 2:00 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> Counter suggestion:
>
> - __fspath__() method may return either bytes or str (no change
>   from the PEP as it stands now);
>
> - but os.fspath() will only return str;
>
> - and os.fspathb() will only return bytes;
>
> - there is no os function that returns "str or bytes, I don't
>   care which". (If you really need that, call __fspath__ directly.)
>
> Note that this differs from the already rejected suggestion that there
> should be two dunder methods, __fspath__() and __fspathb__().
>
> Why?
>
> (1) Normally, the caller knows whether they want str or bytes. (That's
> been my experience, you may disagree.) If so, and they call os.fspath()
> expecting a str, they won't be surprised by it returning bytes. And visa
> versa for when you expect a bytes path.
>
> (2) This behaviour will match that of os.{environ[b],getcwd[b],getenv[b]}.

I would think these have the b suffix because there is no good way to
infer which type should be returned. In things like os.path.join or
os.path.dirname you pass in the object(s) that determine the return
type.

In os.fspath, you pass in an object, whose type (str/bytes) or
"underlying path string type" (as returned by __fspath__()) determines
the return type of fspath. I think this is well in line with os.path
functions.


-- Koos

>
> Cons:
>
> (3) Polymorphic code that truly doesn't care whether it gets bytes or
> str will have a slightly less convenient way of getting it, namely by
> calling __fspath__() itself, instead of os.fspath().
>

From larry at hastings.org  Fri May 13 12:33:10 2016
From: larry at hastings.org (Larry Hastings)
Date: Fri, 13 May 2016 18:33:10 +0200
Subject: [Python-Dev] File system path PEP, part 2
In-Reply-To: <CAP7+vJKYi6qrQN6um0cfPT5hxgNQBr5ie=aVFWq3Po4ex_xTKA@mail.gmail.com>
References: <CAP1=2W6Bf3faNg7UdVgGB2ZXWF2V3nMx2jL_EiGHUnizeCtisA@mail.gmail.com>
 <20160513110012.GH12028@ando.pearwood.info>
 <CAP7+vJKYi6qrQN6um0cfPT5hxgNQBr5ie=aVFWq3Po4ex_xTKA@mail.gmail.com>
Message-ID: <573601C6.8080804@hastings.org>



On 05/13/2016 06:21 PM, Guido van Rossum wrote:
> Really, if you want bytes, you should use os.fsencode(); if you want 
> strings, use os.fsencode(); if you want to be polymorphic, use 
> os.fspath() and check the type it returns.

Am I severely misunderstanding the API, or did you mean "if you want 
strings, use os.fsdecode()" here?


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160513/e64a22d4/attachment.html>

From srkunze at mail.de  Fri May 13 12:33:47 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Fri, 13 May 2016 18:33:47 +0200
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAP1=2W62hAtWN96aG0UO0nm0UXY1Tm3MrNX_UefR0eb=EW4BSw@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
 <57358915.7040606@mail.de>
 <CAMiohoijCDKBuMCioY48UkU=WYy+DJqJyCL6mDBFx1gRXAv+0A@mail.gmail.com>
 <57359D69.6040108@mail.de>
 <CAP1=2W62hAtWN96aG0UO0nm0UXY1Tm3MrNX_UefR0eb=EW4BSw@mail.gmail.com>
Message-ID: <573601EB.50904@mail.de>

On 13.05.2016 17:29, Brett Cannon wrote:
> Purposeful change. It was what I had in my head after I posted my 
> "groups" breakdown email and what Guido suggested as well 
> independently. This helps alleviate any perf worries as type checks in 
> C are pointer checks that are cheap to make compared to attribute 
> lookup. And even if we do get the community to move on to path objects 
> the check is still cheap enough to not worry. And finally, this is 
> simply how we have almost always coded this kind of special-casing at 
> the C level throughout Python and the stdlib, so it now is more inline 
> with standard coding practices than the original proposal in the PEP.

Maybe, we have a misunderstanding here. Does "PyUnicode_Check(path)" 
respect subclass hierarchies such as isinstance(path, str) does? Could 
not find the definition of it in the source.

You said it's just a pointer comparison. To me that implies it could not 
respect subclasses as isinstance does. If that is the case, the C 
implementation differs from the Python version in semantics. I'm sorry 
if I misunderstood the whole implementation here.


Best,
Sven

From ethan at stoneleaf.us  Fri May 13 12:34:36 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 13 May 2016 09:34:36 -0700
Subject: [Python-Dev] File system path PEP, part 2
In-Reply-To: <CAP1=2W5VLshb9XK9km3u0jGMKZ5Gs=b0NyrG4n9fb2pV6COTpA@mail.gmail.com>
References: <CAP1=2W6Bf3faNg7UdVgGB2ZXWF2V3nMx2jL_EiGHUnizeCtisA@mail.gmail.com>
 <20160513110012.GH12028@ando.pearwood.info>
 <CAP1=2W5VLshb9XK9km3u0jGMKZ5Gs=b0NyrG4n9fb2pV6COTpA@mail.gmail.com>
Message-ID: <5736021C.1070300@stoneleaf.us>

On 05/13/2016 08:43 AM, Brett Cannon wrote:

>> a minor technical query:
>>
>>         try:
>>             return path.__fspath__()
>>
>> Would I be right in saying that in practice this will actually end up
>> being type(path).__fspath__() to match the behaviour of all(?) other
>> dunder methods?
>
> I wasn't planning on it because for most types the accessing of the
> method directly off of the type for magic methods is because of some
> special struct field at the C level that we're pulling from. Since we're
> not planning to have an equivalent struct field I don't see any need to
> do the extra work of avoiding the instance participating in method
> lookup. Obviously if people disagree for some reason then please let me
> know (maybe for perf by avoiding the overhead of checking for the method
> on the instance?).

I would say use `type(x).__fspath__`.  I'm not aware of any other 
__dunder__ method that doesn't access the attribute from the type 
instead of the instance, and I see no point in making this one different.

I know there's a line in the Zen about foolish conistencies, but I 
suspect there's a corollary about foolish inconsistencies.  ;)

--
~Ethan~


From rosuav at gmail.com  Fri May 13 12:43:19 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 14 May 2016 02:43:19 +1000
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <573601EB.50904@mail.de>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
 <57358915.7040606@mail.de>
 <CAMiohoijCDKBuMCioY48UkU=WYy+DJqJyCL6mDBFx1gRXAv+0A@mail.gmail.com>
 <57359D69.6040108@mail.de>
 <CAP1=2W62hAtWN96aG0UO0nm0UXY1Tm3MrNX_UefR0eb=EW4BSw@mail.gmail.com>
 <573601EB.50904@mail.de>
Message-ID: <CAPTjJmrQtZckThXR-r5eRARmCZHJWgQSW_CF9+LmSUV1T2eRoQ@mail.gmail.com>

On Sat, May 14, 2016 at 2:33 AM, Sven R. Kunze <srkunze at mail.de> wrote:
> On 13.05.2016 17:29, Brett Cannon wrote:
>>
>> Purposeful change. It was what I had in my head after I posted my "groups"
>> breakdown email and what Guido suggested as well independently. This helps
>> alleviate any perf worries as type checks in C are pointer checks that are
>> cheap to make compared to attribute lookup. And even if we do get the
>> community to move on to path objects the check is still cheap enough to not
>> worry. And finally, this is simply how we have almost always coded this kind
>> of special-casing at the C level throughout Python and the stdlib, so it now
>> is more inline with standard coding practices than the original proposal in
>> the PEP.
>
>
> Maybe, we have a misunderstanding here. Does "PyUnicode_Check(path)" respect
> subclass hierarchies such as isinstance(path, str) does? Could not find the
> definition of it in the source.
>
> You said it's just a pointer comparison. To me that implies it could not
> respect subclasses as isinstance does. If that is the case, the C
> implementation differs from the Python version in semantics. I'm sorry if I
> misunderstood the whole implementation here.

https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_Check

https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_CheckExact

"Check" accepts subclasses; "CheckExact" doesn't (it's like "type(x)
is str"). The question is, which one SHOULD be being done? What should
this do:

class TmpPath(str):
    def __fspath__(self):
        return "/tmp/"+self
x = TmpPath("foo/bar")
open(x, "w")

Does that respect __fspath__, or respect the fact that it's a string?

ChrisA

From rosuav at gmail.com  Fri May 13 12:45:40 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 14 May 2016 02:45:40 +1000
Subject: [Python-Dev] File system path PEP, part 2
In-Reply-To: <5736021C.1070300@stoneleaf.us>
References: <CAP1=2W6Bf3faNg7UdVgGB2ZXWF2V3nMx2jL_EiGHUnizeCtisA@mail.gmail.com>
 <20160513110012.GH12028@ando.pearwood.info>
 <CAP1=2W5VLshb9XK9km3u0jGMKZ5Gs=b0NyrG4n9fb2pV6COTpA@mail.gmail.com>
 <5736021C.1070300@stoneleaf.us>
Message-ID: <CAPTjJmrU_pY5pOnJ=pUDvaBYQRufgTdeq0N6M1q+niPADF9ufw@mail.gmail.com>

On Sat, May 14, 2016 at 2:34 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> I would say use `type(x).__fspath__`.  I'm not aware of any other __dunder__
> method that doesn't access the attribute from the type instead of the
> instance, and I see no point in making this one different.
>

__reduce__ / __reduce_ex__ in pickle.py is accessed with a
straight-forward getattr() call. It's the ones that are called from
deep within the interpreter core (eg __iter__) that are always looked
up on the type.

ChrisA

From p.f.moore at gmail.com  Fri May 13 12:52:20 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 13 May 2016 17:52:20 +0100
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <5735FB79.1010201@mail.de>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
 <57358915.7040606@mail.de>
 <CAMiohoijCDKBuMCioY48UkU=WYy+DJqJyCL6mDBFx1gRXAv+0A@mail.gmail.com>
 <57359D69.6040108@mail.de>
 <CAMiohojq6DDLmWbDv1TiB26CVoUFNw5mNSaiPB+w2zeOgcqOUg@mail.gmail.com>
 <5735FB79.1010201@mail.de>
Message-ID: <CACac1F8=1UBmHpaOuGjx0AUX61VCwZXRj5v8v_-kSmoWB11dpw@mail.gmail.com>

On 13 May 2016 at 17:06, Sven R. Kunze <srkunze at mail.de> wrote:
> As far as I remember, one goal of this proposal was to avoid wallet gardens.
> During the lengthy discussion on python-ideas people brought up that some
> third-party libs indeed subclass from str. They are currently locked out.

Hardly. Libraries that subclass from str will provide their string
representation precisely by being (a subclass of) a string. So they
don't need the __fspath__ protocol, and can continue to work exactly
as now.

This PEP merely provides the same capability for libraries that prefer
*not* to subclass str.
Paul

From ethan at stoneleaf.us  Fri May 13 12:57:19 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 13 May 2016 09:57:19 -0700
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <5735FB79.1010201@mail.de>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
 <57358915.7040606@mail.de>
 <CAMiohoijCDKBuMCioY48UkU=WYy+DJqJyCL6mDBFx1gRXAv+0A@mail.gmail.com>
 <57359D69.6040108@mail.de>
 <CAMiohojq6DDLmWbDv1TiB26CVoUFNw5mNSaiPB+w2zeOgcqOUg@mail.gmail.com>
 <5735FB79.1010201@mail.de>
Message-ID: <5736076F.6010905@stoneleaf.us>

On 05/13/2016 09:06 AM, Sven R. Kunze wrote:
> On 13.05.2016 11:48, Koos Zevenhoven wrote:
 >> Sven R Kunze had previously queried:

>>> However, the proposed semantics will change if the checks are swapped. So,
>>> my actual question is:
>>>
>>> Is that an intended API inconsistency or a known bug supposed to be resolved
>>> later?
>>
>> Taking into account the description (and the drafted type hint), which
>> the documentation will probably reflect, the semantic effects of that
>> are very minor or nonexistent.
>
> From your perspective. As far as I remember, one goal of this proposal
> was to avoid wallet gardens. During the lengthy discussion on
> python-ideas people brought up that some third-party libs indeed
> subclass from str. They are currently locked out.

Two points:

1) What is a wallet garden?

2) If a third-party path lib subclasses either str or bytes, and the 
os.* and other stdlib functions accept str or bytes, how are they locked 
out by not having an __fspath__?

Assuming, of course, that those functions use isinstance and not type 
(if do use type a bug should be filed against them.

> Indeed. Just one minor note here: str or bytes subclasses *can*
> implement __fspath__ and currently it will be *ignored*.

Sure.  And int subclasses can implement __index__, which will likewise 
be ignored.

>> So no API inconsistency there.
>
> API consistency is not defined by docs-matching-implementation but by
> implementation-matching-expectations.

Well, probably both.  ;)

--
~Ethan~


From k7hoven at gmail.com  Fri May 13 13:05:38 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Fri, 13 May 2016 20:05:38 +0300
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <5735FB79.1010201@mail.de>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
 <57358915.7040606@mail.de>
 <CAMiohoijCDKBuMCioY48UkU=WYy+DJqJyCL6mDBFx1gRXAv+0A@mail.gmail.com>
 <57359D69.6040108@mail.de>
 <CAMiohojq6DDLmWbDv1TiB26CVoUFNw5mNSaiPB+w2zeOgcqOUg@mail.gmail.com>
 <5735FB79.1010201@mail.de>
Message-ID: <CAMiohojezx_dpHr3huDoTVBHw6B=1appt780MZ_gbQwoPVzx5g@mail.gmail.com>

On Fri, May 13, 2016 at 7:06 PM, Sven R. Kunze <srkunze at mail.de> wrote:
> On 13.05.2016 11:48, Koos Zevenhoven wrote:
>>
>> This issue is coupled with the future optimization questions.
>>
>
> AFAIC coupling API design to optimization is called premature optimization.
>

I suppose so, but currently I consider the API not to make guarantees
about what happens if you pass in something other than specified as
valid input. I'm not sure what level of precision is usual at this
point, but I think this is sufficient for now.

I'm sure we will pay attention to this (as we have been doing, both
while optimizing prematurely and for other reasons).

-- Koos

From guido at python.org  Fri May 13 13:38:35 2016
From: guido at python.org (Guido van Rossum)
Date: Fri, 13 May 2016 10:38:35 -0700
Subject: [Python-Dev] File system path PEP, part 2
In-Reply-To: <573601C6.8080804@hastings.org>
References: <CAP1=2W6Bf3faNg7UdVgGB2ZXWF2V3nMx2jL_EiGHUnizeCtisA@mail.gmail.com>
 <20160513110012.GH12028@ando.pearwood.info>
 <CAP7+vJKYi6qrQN6um0cfPT5hxgNQBr5ie=aVFWq3Po4ex_xTKA@mail.gmail.com>
 <573601C6.8080804@hastings.org>
Message-ID: <CAP7+vJLe-+WQF9RYGWg2ikML6qE=Re6Ub6KYunQMXa41KdeMzw@mail.gmail.com>

On Fri, May 13, 2016 at 9:33 AM, Larry Hastings <larry at hastings.org> wrote:

>
>
> On 05/13/2016 06:21 PM, Guido van Rossum wrote:
>
> Really, if you want bytes, you should use os.fsencode(); if you want
> strings, use os.fsencode(); if you want to be polymorphic, use os.fspath()
> and check the type it returns.
>
>
> Am I severely misunderstanding the API, or did you mean "if you want
> strings, use os.fsdecode()" here?
>

encode, decode -- poh-tay-toe, poh-tah-toe. :-)

Another slip of the fingers, I did mean os.fsdecode() when you want strings.

encode, schmencode...

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160513/3bb15205/attachment.html>

From guido at python.org  Fri May 13 13:39:23 2016
From: guido at python.org (Guido van Rossum)
Date: Fri, 13 May 2016 10:39:23 -0700
Subject: [Python-Dev] File system path PEP, part 2
In-Reply-To: <5736021C.1070300@stoneleaf.us>
References: <CAP1=2W6Bf3faNg7UdVgGB2ZXWF2V3nMx2jL_EiGHUnizeCtisA@mail.gmail.com>
 <20160513110012.GH12028@ando.pearwood.info>
 <CAP1=2W5VLshb9XK9km3u0jGMKZ5Gs=b0NyrG4n9fb2pV6COTpA@mail.gmail.com>
 <5736021C.1070300@stoneleaf.us>
Message-ID: <CAP7+vJLta1OmFAQL3iQOKvNa8sR+N459CPNZzJK7x4R3k52Srw@mail.gmail.com>

On Fri, May 13, 2016 at 9:34 AM, Ethan Furman <ethan at stoneleaf.us> wrote:

> I would say use `type(x).__fspath__`.  I'm not aware of any other
> __dunder__ method that doesn't access the attribute from the type instead
> of the instance, and I see no point in making this one different.
>

Agreed.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160513/f479b74e/attachment.html>

From p.f.moore at gmail.com  Fri May 13 13:42:23 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 13 May 2016 18:42:23 +0100
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <5736076F.6010905@stoneleaf.us>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
 <57358915.7040606@mail.de>
 <CAMiohoijCDKBuMCioY48UkU=WYy+DJqJyCL6mDBFx1gRXAv+0A@mail.gmail.com>
 <57359D69.6040108@mail.de>
 <CAMiohojq6DDLmWbDv1TiB26CVoUFNw5mNSaiPB+w2zeOgcqOUg@mail.gmail.com>
 <5735FB79.1010201@mail.de> <5736076F.6010905@stoneleaf.us>
Message-ID: <CACac1F_dtX+K+kE=Z5AWGY8h3p2E=x4_fx0KoQ7eUhMuMddJVw@mail.gmail.com>

On 13 May 2016 at 17:57, Ethan Furman <ethan at stoneleaf.us> wrote:
> 1) What is a wallet garden?

I assumed he meant "walled garden" (for people who don't know the
phrase, it refers to an area or capability that's only made accessible
to "favoured" clients - think of something like Apple's App Store,
where you have to be approved to publish there).

Paul

From steve at pearwood.info  Fri May 13 13:52:26 2016
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 14 May 2016 03:52:26 +1000
Subject: [Python-Dev] File system path PEP, part 2
In-Reply-To: <CAP1=2W5VLshb9XK9km3u0jGMKZ5Gs=b0NyrG4n9fb2pV6COTpA@mail.gmail.com>
References: <CAP1=2W6Bf3faNg7UdVgGB2ZXWF2V3nMx2jL_EiGHUnizeCtisA@mail.gmail.com>
 <20160513110012.GH12028@ando.pearwood.info>
 <CAP1=2W5VLshb9XK9km3u0jGMKZ5Gs=b0NyrG4n9fb2pV6COTpA@mail.gmail.com>
Message-ID: <20160513175226.GI12028@ando.pearwood.info>

On Fri, May 13, 2016 at 03:43:29PM +0000, Brett Cannon wrote:
> On Fri, 13 May 2016 at 04:00 Steven D'Aprano <steve at pearwood.info> wrote:

[...]
> > - but os.fspath() will only return str;
> > - and os.fspathb() will only return bytes;

> I prefer what's in the PEP. I get where you coming from, Steven, but I
> don't think it will be common enough to worry about. Think of os.fspath()
> like next() where it truly is a very minor convenience function that
> happens to special-case str and bytes.

Okay, I'm satisfied by the various arguments against this idea.


[...]
> > I think it is a bit confusing to refer to "path objects", as that seems
> > like you are referring only to pathlib.Path objects. It took me far too
> > long to realise that here you mean generic path-like objects that obey
> > the __fspath__ protocol rather than a specific concrete class.
> >
> > Since the ABC is called "PathLike", I suggest we refer to "path-like
> > objects" rather than "path objects", both in the PEP and in the Python
> > docs for this protocol.
> >
> 
> I went back and forth with this in my head while writing the PEP. The
> problem with making "path-like" mean "objects implementing the PathLike
> ABC" becomes how do you refer to an argument of a function that accepts
> anything os.fspath() does (i.e. PathLike, str, and bytes)?

On further reflection, I think the right language is to use "path-like" 
for the union of Pathlike, str and bytes. That will, I think, cover the 
majority of cases: most functions which want to work on a file system 
path should accept all three. When you want to specify only an object 
which implements the PathLike ABC, that's called a (virtual) instance of 
PathLike.

I see this as analogous to "iterable". An iterable is anything which can 
be iterated over. Sometimes that's an iterator. Sometimes its not an 
iterator. We don't have a special term for "iterable which is not an 
iterator", mostly because its rare to care about the distinction, but on 
those rare cases that we do care, we can describe it explicitly.

In this case, "path-like" would be equivalent to iterable: anything that 
can be used as a file system path. Some path-like objects are actual 
pathlib.Path objects, some are some other PathLike object, and some are 
strings or bytes.


[...]
> > >         try:
> > >             return path.__fspath__()
> >
> > Would I be right in saying that in practice this will actually end up
> > being type(path).__fspath__() to match the behaviour of all(?) other
> > dunder methods?
> >
> 
> I wasn't planning on it because for most types the accessing of the method
> directly off of the type for magic methods is because of some special
> struct field at the C level that we're pulling from. Since we're not
> planning to have an equivalent struct field I don't see any need to do the
> extra work of avoiding the instance participating in method lookup.
> Obviously if people disagree for some reason then please let me know (maybe
> for perf by avoiding the overhead of checking for the method on the
> instance?).

The reasons I would disagree are:

(1) It took me a long time to learn the rule that dunder methods are 
always called from the class, not the instance, and now I have to learn 
that there are exceptions? Grrrr argggh.

(2) If we ever do change to a C struct field, the behaviour will change. 
Maybe it's better to emulate the same behaviour from the start?

(3) If there's a performance speed up, that's a bonus!



-- 
Steven

From brett at python.org  Fri May 13 13:56:35 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 13 May 2016 17:56:35 +0000
Subject: [Python-Dev] File system path PEP, part 2
In-Reply-To: <20160513175226.GI12028@ando.pearwood.info>
References: <CAP1=2W6Bf3faNg7UdVgGB2ZXWF2V3nMx2jL_EiGHUnizeCtisA@mail.gmail.com>
 <20160513110012.GH12028@ando.pearwood.info>
 <CAP1=2W5VLshb9XK9km3u0jGMKZ5Gs=b0NyrG4n9fb2pV6COTpA@mail.gmail.com>
 <20160513175226.GI12028@ando.pearwood.info>
Message-ID: <CAP1=2W7hjXLV+tHw4iwGzCP9eJEjaKPW81RvwgcMEcO3b3VcrQ@mail.gmail.com>

On Fri, 13 May 2016 at 10:53 Steven D'Aprano <steve at pearwood.info> wrote:

> On Fri, May 13, 2016 at 03:43:29PM +0000, Brett Cannon wrote:
> > On Fri, 13 May 2016 at 04:00 Steven D'Aprano <steve at pearwood.info>
> wrote:
>
> [...]
> > > - but os.fspath() will only return str;
> > > - and os.fspathb() will only return bytes;
>
> > I prefer what's in the PEP. I get where you coming from, Steven, but I
> > don't think it will be common enough to worry about. Think of os.fspath()
> > like next() where it truly is a very minor convenience function that
> > happens to special-case str and bytes.
>
> Okay, I'm satisfied by the various arguments against this idea.
>

I'll add a note in the Rejected Ideas section about this.


>
>
> [...]
> > > I think it is a bit confusing to refer to "path objects", as that seems
> > > like you are referring only to pathlib.Path objects. It took me far too
> > > long to realise that here you mean generic path-like objects that obey
> > > the __fspath__ protocol rather than a specific concrete class.
> > >
> > > Since the ABC is called "PathLike", I suggest we refer to "path-like
> > > objects" rather than "path objects", both in the PEP and in the Python
> > > docs for this protocol.
> > >
> >
> > I went back and forth with this in my head while writing the PEP. The
> > problem with making "path-like" mean "objects implementing the PathLike
> > ABC" becomes how do you refer to an argument of a function that accepts
> > anything os.fspath() does (i.e. PathLike, str, and bytes)?
>
> On further reflection, I think the right language is to use "path-like"
> for the union of Pathlike, str and bytes. That will, I think, cover the
> majority of cases: most functions which want to work on a file system
> path should accept all three. When you want to specify only an object
> which implements the PathLike ABC, that's called a (virtual) instance of
> PathLike.
>
> I see this as analogous to "iterable". An iterable is anything which can
> be iterated over. Sometimes that's an iterator. Sometimes its not an
> iterator. We don't have a special term for "iterable which is not an
> iterator", mostly because its rare to care about the distinction, but on
> those rare cases that we do care, we can describe it explicitly.
>
> In this case, "path-like" would be equivalent to iterable: anything that
> can be used as a file system path. Some path-like objects are actual
> pathlib.Path objects, some are some other PathLike object, and some are
> strings or bytes.
>

That was my general thinking. I'll add a TODO to add an entry in the
glossary for "path-like".


>
>
> [...]
> > > >         try:
> > > >             return path.__fspath__()
> > >
> > > Would I be right in saying that in practice this will actually end up
> > > being type(path).__fspath__() to match the behaviour of all(?) other
> > > dunder methods?
> > >
> >
> > I wasn't planning on it because for most types the accessing of the
> method
> > directly off of the type for magic methods is because of some special
> > struct field at the C level that we're pulling from. Since we're not
> > planning to have an equivalent struct field I don't see any need to do
> the
> > extra work of avoiding the instance participating in method lookup.
> > Obviously if people disagree for some reason then please let me know
> (maybe
> > for perf by avoiding the overhead of checking for the method on the
> > instance?).
>
> The reasons I would disagree are:
>
> (1) It took me a long time to learn the rule that dunder methods are
> always called from the class, not the instance, and now I have to learn
> that there are exceptions? Grrrr argggh.
>
> (2) If we ever do change to a C struct field, the behaviour will change.
> Maybe it's better to emulate the same behaviour from the start?
>
> (3) If there's a performance speed up, that's a bonus!
>

And Guido agrees with this as well so I'll be updating the PEP shortly
after I remember how to do the equivalent from C. :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160513/fc780568/attachment-0001.html>

From brett at python.org  Fri May 13 14:37:33 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 13 May 2016 18:37:33 +0000
Subject: [Python-Dev] File system path PEP, 3rd draft
Message-ID: <CAP1=2W6W+WO3-NjiSdCnqKVW1PBRzeEiW8gZGw1GSk1PLsufxA@mail.gmail.com>

Biggest changes since the second draft:

   1. Resolve __fspath__() from the type, not the instance (for Guido)
   2. Updated the TypeError messages to say "os.PathLike object" instead of
   "path object" (implicitly for Steven)
   3. TODO item to define "path-like" in the glossary (for Steven)
   4. Various more things added to Rejected Ideas
   5. Added Koos as a co-author (for Koos :)

----------
PEP: NNN
Title: Adding a file system path protocol
Version: $Revision$
Last-Modified: $Date$
Author: Brett Cannon <brett at python.org>,
        Koos Zevenhoven <k7hoven at gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 11-May-2016
Post-History: 11-May-2016,
              12-May-2016,
              13-May-2016


Abstract
========

This PEP proposes a protocol for classes which represent a file system
path to be able to provide a ``str`` or ``bytes`` representation.
Changes to Python's standard library are also proposed to utilize this
protocol where appropriate to facilitate the use of path objects where
historically only ``str`` and/or ``bytes`` file system paths are
accepted. The goal is to facilitate the migration of users towards
rich path objects while providing an easy way to work with code
expecting ``str`` or ``bytes``.


Rationale
=========

Historically in Python, file system paths have been represented as
strings or bytes. This choice of representation has stemmed from C's
own decision to represent file system paths as
``const char *`` [#libc-open]_. While that is a totally serviceable
format to use for file system paths, it's not necessarily optimal. At
issue is the fact that while all file system paths can be represented
as strings or bytes, not all strings or bytes represent a file system
path. This can lead to issues where any e.g. string duck-types to a
file system path whether it actually represents a path or not.

To help elevate the representation of file system paths from their
representation as strings and bytes to a richer object representation,
the pathlib module [#pathlib]_ was provisionally introduced in
Python 3.4 through PEP 428. While considered by some as an improvement
over strings and bytes for file system paths, it has suffered from a
lack of adoption. Typically the key issue listed for the low adoption
rate has been the lack of support in the standard library. This lack
of support required users of pathlib to manually convert path objects
to strings by calling ``str(path)`` which many found error-prone.

One issue in converting path objects to strings comes from
the fact that the only generic way to get a string representation of
the path was to pass the object to ``str()``. This can pose a
problem when done blindly as nearly all Python objects have some
string representation whether they are a path or not, e.g.
``str(None)`` will give a result that
``builtins.open()`` [#builtins-open]_ will happily use to create a new
file.

Exacerbating this whole situation is the
``DirEntry`` object [#os-direntry]_. While path objects have a
representation that can be extracted using ``str()``, ``DirEntry``
objects expose a ``path`` attribute instead. Having no common
interface between path objects, ``DirEntry``, and any other
third-party path library has become an issue. A solution that allows
any path-representing object to declare that it is a path and a way
to extract a low-level representation that all path objects could
support is desired.

This PEP then proposes to introduce a new protocol to be followed by
objects which represent file system paths. Providing a protocol allows
for explicit signaling of what objects represent file system paths as
well as a way to extract a lower-level representation that can be used
with older APIs which only support strings or bytes.

Discussions regarding path objects that led to this PEP can be found
in multiple threads on the python-ideas mailing list archive
[#python-ideas-archive]_ for the months of March and April 2016 and on
the python-dev mailing list archives [#python-dev-archive]_ during
April 2016.


Proposal
========

This proposal is split into two parts. One part is the proposal of a
protocol for objects to declare and provide support for exposing a
file system path representation. The other part deals with changes to
Python's standard library to support the new protocol. These changes
will also lead to the pathlib module dropping its provisional status.

Protocol
--------

The following abstract base class defines the protocol for an object
to be considered a path object::

    import abc
    import typing as t


    class PathLike(abc.ABC):

        """Abstract base class for implementing the file system path
protocol."""

        @abc.abstractmethod
        def __fspath__(self) -> t.Union[str, bytes]:
            """Return the file system path representation of the object."""
            raise NotImplementedError


Objects representing file system paths will implement the
``__fspath__()`` method which will return the ``str`` or ``bytes``
representation of the path. The ``str`` representation is the
preferred low-level path representation as it is human-readable and
what people historically represent paths as.


Standard library changes
------------------------

It is expected that most APIs in Python's standard library that
currently accept a file system path will be updated appropriately to
accept path objects (whether that requires code or simply an update
to documentation will vary). The modules mentioned below, though,
deserve specific details as they have either fundamental changes that
empower the ability to use path objects, or entail additions/removal
of APIs.


builtins
''''''''

``open()`` [#builtins-open]_ will be updated to accept path objects as
well as continue to accept ``str`` and ``bytes``.


os
'''

The ``fspath()`` function will be added with the following semantics::

    import typing as t


    def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str, bytes]:
        """Return the string representation of the path.

        If str or bytes is passed in, it is returned unchanged.
        """
        if isinstance(path, (str, bytes)):
            return path

        # Work from the object's type to match method resolution of other
magic
        # methods.
        path_type = type(path)
        try:
            return path_type.__fspath__(path)
        except AttributeError:
            if hasattr(path_type, '__fspath__'):
                raise

            raise TypeError("expected str, bytes or os.PathLike object, not
"
                            + path_type.__name__)

The ``os.fsencode()`` [#os-fsencode]_ and
``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept
path objects. As both functions coerce their arguments to
``bytes`` and ``str``, respectively, they will be updated to call
``__fspath__()`` if present to convert the path object to a ``str`` or
``bytes`` representation, and then perform their appropriate
coercion operations as if the return value from ``__fspath__()`` had
been the original argument to the coercion function in question.

The addition of ``os.fspath()``, the updates to
``os.fsencode()``/``os.fsdecode()``, and the current semantics of
``pathlib.PurePath`` provide the semantics necessary to
get the path representation one prefers. For a path object,
``pathlib.PurePath``/``Path`` can be used. To obtain the ``str`` or
``bytes`` representation without any coersion, then ``os.fspath()``
can be used. If a ``str`` is desired and the encoding of ``bytes``
should be assumed to be the default file system encoding, then
``os.fsdecode()`` should be used. If a ``bytes`` representation is
desired and any strings should be encoded using the default file
system encoding, then ``os.fsencode()`` is used. This PEP recommends
using path objects when possible and falling back to string paths as
necessary and using ``bytes`` as a last resort.

Another way to view this is as a hierarchy of file system path
representations (highest- to lowest-level): path ? str ? bytes. The
functions and classes under discussion can all accept objects on the
same level of the hierarchy, but they vary in whether they promote or
demote objects to another level. The ``pathlib.PurePath`` class can
promote a ``str`` to a path object. The ``os.fspath()`` function can
demote a path object to a ``str`` or ``bytes`` instance, depending
on what ``__fspath__()`` returns.
The ``os.fsdecode()`` function will demote a path object to
a string or promote a ``bytes`` object to a ``str``. The
``os.fsencode()`` function will demote a path or string object to
``bytes``. There is no function that provides a way to demote a path
object directly to ``bytes`` while bypassing string demotion.

The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()``
method. It will return the same value as currently found on the
``path`` attribute of ``DirEntry`` instances.

The Protocol_ ABC will be added to the ``os`` module under the name
``os.PathLike``.


os.path
'''''''

The various path-manipulation functions of ``os.path`` [#os-path]_
will be updated to accept path objects. For polymorphic functions that
accept both bytes and strings, they will be updated to simply use
``os.fspath()``.

During the discussions leading up to this PEP it was suggested that
``os.path`` not be updated using an "explicit is better than implicit"
argument. The thinking was that since ``__fspath__()`` is polymorphic
itself it may be better to have code working with ``os.path`` extract
the path representation from path objects explicitly. There is also
the consideration that adding support this deep into the low-level OS
APIs will lead to code magically supporting path objects without
requiring any documentation updated, leading to potential complaints
when it doesn't work, unbeknownst to the project author.

But it is the view of this PEP that "practicality beats purity" in
this instance. To help facilitate the transition to supporting path
objects, it is better to make the transition as easy as possible than
to worry about unexpected/undocumented duck typing support for
path objects by projects.

There has also been the suggestion that ``os.path`` functions could be
used in a tight loop and the overhead of checking or calling
``__fspath__()`` would be too costly. In this scenario only
path-consuming APIs would be directly updated and path-manipulating
APIs like the ones in ``os.path`` would go unmodified. This would
require library authors to update their code to support path objects
if they performed any path manipulations, but if the library code
passed the path straight through then the library wouldn't need to be
updated. It is the view of this PEP and Guido, though, that this is an
unnecessary worry and that performance will still be acceptable.


pathlib
'''''''

The constructor for ``pathlib.PurePath`` and ``pathlib.Path`` will be
updated to accept ``PathLike`` objects. Both ``PurePath`` and ``Path``
will continue to not accept ``bytes`` path representations, and so if
``__fspath__()`` returns ``bytes`` it will raise an exception.

The ``path`` attribute will be removed as this PEP makes it
redundant (it has not been included in any released version of Python
and so is not a backwards-compatibility concern).


C API
'''''

The C API will gain an equivalent function to ``os.fspath()``::

    /*
        Return the file system path of the object.

        If the object is str or bytes, then allow it to pass through with
        an incremented refcount. If the object defines __fspath__(), then
        return the result of that method. All other types raise a TypeError.
    */
    PyObject *
    PyOS_FSPath(PyObject *path)
    {
        if (PyUnicode_Check(path) || PyBytes_Check(path)) {
            Py_INCREF(path);
            return path;
        }

        if (PyObject_HasAttrString(path->ob_type, "__fspath__")) {
            return PyObject_CallMethodObjArgs(path->ob_type, "__fspath__",
path,
                                            NULL);
        }

        return PyErr_Format(PyExc_TypeError,
                            "expected a str, bytes, or os.PathLike object,
not %S",
                            path->ob_type);
    }



Backwards compatibility
=======================

There are no explicit backwards-compatibility concerns. Unless an
object incidentally already defines a ``__fspath__()`` method there is
no reason to expect the pre-existing code to break or expect to have
its semantics implicitly changed.

Libraries wishing to support path objects and a version of Python
prior to Python 3.6 and the existence of ``os.fspath()`` can use the
idiom of
``path.__fspath__() if hasattr(path, "__fspath__") else path``.


Implementation
==============

This is the task list for what this PEP proposes:

#. Remove the ``path`` attribute from pathlib
#. Remove the provisional status of pathlib
#. Add ``os.PathLike``
#. Add ``os.fspath()``
#. Add ``PyOS_FSPath()``
#. Update ``os.fsencode()``
#. Update ``os.fsdecode()``
#. Update ``pathlib.PurePath`` and ``pathlib.Path``
#. Update ``builtins.open()``
#. Update ``os.DirEntry``
#. Update ``os.path``
#. Add a glossary entry for "path-like"


Rejected Ideas
==============

Other names for the protocol's method
-------------------------------------

Various names were proposed during discussions leading to this PEP,
including ``__path__``, ``__pathname__``, and ``__fspathname__``. In
the end people seemed to gravitate towards ``__fspath__`` for being
unambiguous without being unnecessarily long.


Separate str/bytes methods
--------------------------

At one point it was suggested that ``__fspath__()`` only return
strings and another method named ``__fspathb__()`` be introduced to
return bytes. The thinking is that by making ``__fspath__()`` not be
polymorphic it could make dealing with the potential string or bytes
representations easier. But the general consensus was that returning
bytes will more than likely be rare and that the various functions in
the os module are the better abstraction to promote over direct
calls to ``__fspath__()``.


Providing a ``path`` attribute
------------------------------

To help deal with the issue of ``pathlib.PurePath`` not inheriting
from ``str``, originally it was proposed to introduce a ``path``
attribute to mirror what ``os.DirEntry`` provides. In the end,
though, it was determined that a protocol would provide the same
result while not directly exposing an API that most people will never
need to interact with directly.


Have ``__fspath__()`` only return strings
------------------------------------------

Much of the discussion that led to this PEP revolved around whether
``__fspath__()`` should be polymorphic and return ``bytes`` as well as
``str`` or only return ``str``. The general sentiment for this view
was that ``bytes`` are difficult to work with due to their
inherent lack of information about their encoding and PEP 383 makes
it possible to represent all file system paths using ``str`` with the
``surrogateescape`` handler. Thus, it would be better to forcibly
promote the use of ``str`` as the low-level path representation for
high-level path objects.

In the end, it was decided that using ``bytes`` to represent paths is
simply not going to go away and thus they should be supported to some
degree. The hope is that people will gravitate towards path objects
like pathlib and that will move people away from operating directly
with ``bytes``.


A generic string encoding mechanism
-----------------------------------

At one point there was a discussion of developing a generic mechanism
to extract a string representation of an object that had semantic
meaning (``__str__()`` does not necessarily return anything of
semantic significance beyond what may be helpful for debugging). In
the end, it was deemed to lack a motivating need beyond the one this
PEP is trying to solve in a specific fashion.


Have __fspath__ be an attribute
-------------------------------

It was briefly considered to have ``__fspath__`` be an attribute
instead of a method. This was rejected for two reasons. One,
historically protocols have been implemented as "magic methods" and
not "magic methods and attributes". Two, there is no guarantee that
the lower-level representation of a path object will be pre-computed,
potentially misleading users that there was no expensive computation
behind the scenes in case the attribute was implemented as a property.

This also indirectly ties into the idea of introducing a ``path``
attribute to accomplish the same thing. This idea has an added issue,
though, of accidentally having any object with a ``path`` attribute
meet the protocol's duck typing. Introducing a new magic method for
the protocol helpfully avoids any accidental opting into the protocol.


Provide specific type hinting support
-------------------------------------

There was some consideration to provdinga generic ``typing.PathLike``
class which would allow for e.g. ``typing.PathLike[str]`` to specify
a type hint for a path object which returned a string representation.
While potentially beneficial, the usefulness was deemed too small to
bother adding the type hint class.

This also removed any desire to have a class in the ``typing`` module
which represented the union of all acceptable path-representing types
as that can be represented with
``typing.Union[str, bytes, os.PathLike]`` easily enough and the hope
is users will slowly gravitate to path objects only.


Provide ``os.fspathb()``
------------------------

It was suggested that to mirror the structure of e.g.
``os.getcwd()``/``os.getcwdb()``, that ``os.fspath()`` only return
``str`` and that another function named ``os.fspathb()`` be
introduced that only returned ``bytes``. This was rejected as the
purposes of the ``*b()`` functions are tied to querying the file
system where there is a need to get the raw bytes back. As this PEP
does not work directly with data on a file system (but which *may*
be), the view was taken this distinction is unnecessary. It's also
believed that the need for only bytes will not be common enough to
need to support in such a specific manner as ``os.fsencode()`` will
provide similar functionality.


Call ``__fspath__()`` off of the instance
-----------------------------------------

An earlier draft of this PEP had ``os.fspath()`` calling
``path.__fspath__()`` instead of ``type(path).__fspath__(path)``. The
changed to be consistent with how other magic methods in Python are
resolved.


Acknowledgements
================

Thanks to everyone who participated in the various discussions related
to this PEP that spanned both python-ideas and python-dev. Special
thanks to Stephen Turnbull for direct feedback on early drafts of this
PEP. More special thanks to Koos Zevenhoven and Ethan Furman for not
only feedback on early drafts of this PEP but also helping to drive
the overall discussion on this topic across the two mailing lists.


References
==========

.. [#python-ideas-archive] The python-ideas mailing list archive
   (https://mail.python.org/pipermail/python-ideas/)

.. [#python-dev-archive] The python-dev mailing list archive
   (https://mail.python.org/pipermail/python-dev/)

.. [#libc-open] ``open()`` documention for the C standard library
   (
http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html
)

.. [#pathlib] The ``pathlib`` module
   (https://docs.python.org/3/library/pathlib.html#module-pathlib)

.. [#builtins-open] The ``builtins.open()`` function
   (https://docs.python.org/3/library/functions.html#open)

.. [#os-fsencode] The ``os.fsencode()`` function
   (https://docs.python.org/3/library/os.html#os.fsencode)

.. [#os-fsdecode] The ``os.fsdecode()`` function
   (https://docs.python.org/3/library/os.html#os.fsdecode)

.. [#os-direntry] The ``os.DirEntry`` class
   (https://docs.python.org/3/library/os.html#os.DirEntry)

.. [#os-path] The ``os.path`` module
   (https://docs.python.org/3/library/os.path.html#module-os.path)


Copyright
=========

This document has been placed in the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160513/98a068c9/attachment.html>

From barry at python.org  Fri May 13 15:08:10 2016
From: barry at python.org (Barry Warsaw)
Date: Fri, 13 May 2016 15:08:10 -0400
Subject: [Python-Dev] File system path PEP, 3rd draft
In-Reply-To: <CAP1=2W6W+WO3-NjiSdCnqKVW1PBRzeEiW8gZGw1GSk1PLsufxA@mail.gmail.com>
References: <CAP1=2W6W+WO3-NjiSdCnqKVW1PBRzeEiW8gZGw1GSk1PLsufxA@mail.gmail.com>
Message-ID: <20160513150810.2ae51e79.barry@wooz.org>

On May 13, 2016, at 06:37 PM, Brett Cannon wrote:

>PEP: NNN
>Title: Adding a file system path protocol
>Version: $Revision$
>Last-Modified: $Date$
>Author: Brett Cannon <brett at python.org>,
>        Koos Zevenhoven <k7hoven at gmail.com>
>Status: Draft
>Type: Standards Track
>Content-Type: text/x-rst
>Created: 11-May-2016
>Post-History: 11-May-2016,
>              12-May-2016,
>              13-May-2016

I think this is far enough along that you should self-assign a PEP number and
check it into the official repo.

Cheers,
-Barry

From brett at python.org  Fri May 13 15:24:17 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 13 May 2016 19:24:17 +0000
Subject: [Python-Dev] File system path PEP, 3rd draft
In-Reply-To: <20160513150810.2ae51e79.barry@wooz.org>
References: <CAP1=2W6W+WO3-NjiSdCnqKVW1PBRzeEiW8gZGw1GSk1PLsufxA@mail.gmail.com>
 <20160513150810.2ae51e79.barry@wooz.org>
Message-ID: <CAP1=2W7NqCR7jxn71E6eKm_EXcqBrbVvFaOPQ+PkeGdpjzJFcA@mail.gmail.com>

On Fri, 13 May 2016 at 12:08 Barry Warsaw <barry at python.org> wrote:

> On May 13, 2016, at 06:37 PM, Brett Cannon wrote:
>
> >PEP: NNN
> >Title: Adding a file system path protocol
> >Version: $Revision$
> >Last-Modified: $Date$
> >Author: Brett Cannon <brett at python.org>,
> >        Koos Zevenhoven <k7hoven at gmail.com>
> >Status: Draft
> >Type: Standards Track
> >Content-Type: text/x-rst
> >Created: 11-May-2016
> >Post-History: 11-May-2016,
> >              12-May-2016,
> >              13-May-2016
>
> I think this is far enough along that you should self-assign a PEP number
> and
> check it into the official repo.
>

Paraphrasing *someone* from Twitter yesterday, I'm just going to work on
GitHub until I absolutely have to check into hg, m'kay? ;)

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160513/01faf911/attachment-0001.html>

From k7hoven at gmail.com  Fri May 13 15:26:24 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Fri, 13 May 2016 22:26:24 +0300
Subject: [Python-Dev] File system path PEP, part 2
In-Reply-To: <20160513175226.GI12028@ando.pearwood.info>
References: <CAP1=2W6Bf3faNg7UdVgGB2ZXWF2V3nMx2jL_EiGHUnizeCtisA@mail.gmail.com>
 <20160513110012.GH12028@ando.pearwood.info>
 <CAP1=2W5VLshb9XK9km3u0jGMKZ5Gs=b0NyrG4n9fb2pV6COTpA@mail.gmail.com>
 <20160513175226.GI12028@ando.pearwood.info>
Message-ID: <CAMiohogeEOsfs997N7dvGumCgXEn3mQ+ryXgjgpC1-FDpxoX8Q@mail.gmail.com>

On Fri, May 13, 2016 at 8:52 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Fri, May 13, 2016 at 03:43:29PM +0000, Brett Cannon wrote:
>> On Fri, 13 May 2016 at 04:00 Steven D'Aprano <steve at pearwood.info> wrote:
>
> [...]
>> > I think it is a bit confusing to refer to "path objects", as that seems
>> > like you are referring only to pathlib.Path objects. It took me far too
>> > long to realise that here you mean generic path-like objects that obey
>> > the __fspath__ protocol rather than a specific concrete class.
>> >

This terminology is indeed a bit difficult, not least because there
are 6 different path classes in pathlib. A couple of months ago, I
decided to start to call these pathlib objects and path objects,
because I did not know what else to call them without doubling the
length.

>> > Since the ABC is called "PathLike", I suggest we refer to "path-like
>> > objects" rather than "path objects", both in the PEP and in the Python
>> > docs for this protocol.
>>
>> I went back and forth with this in my head while writing the PEP. The
>> problem with making "path-like" mean "objects implementing the PathLike
>> ABC" becomes how do you refer to an argument of a function that accepts
>> anything os.fspath() does (i.e. PathLike, str, and bytes)?
>
> On further reflection, I think the right language is to use "path-like"
> for the union of Pathlike, str and bytes. That will, I think, cover the
> majority of cases: most functions which want to work on a file system
> path should accept all three. When you want to specify only an object
> which implements the PathLike ABC, that's called a (virtual) instance of
> PathLike.
>

As I've told Brett before, I think exactly this reasoning would be a
good enough reason not to call the ABC PathLike.

[...]
>> > Would I be right in saying that in practice this will actually end up
>> > being type(path).__fspath__() to match the behaviour of all(?) other
>> > dunder methods?
>>
>> I wasn't planning on it because for most types the accessing of the method
>> directly off of the type for magic methods is because of some special
>> struct field at the C level that we're pulling from. Since we're not
>> planning to have an equivalent struct field I don't see any need to do the
>> extra work of avoiding the instance participating in method lookup.
>> Obviously if people disagree for some reason then please let me know (maybe
>> for perf by avoiding the overhead of checking for the method on the
>> instance?).
>
> The reasons I would disagree are:
>
> (1) It took me a long time to learn the rule that dunder methods are
> always called from the class, not the instance, and now I have to learn
> that there are exceptions? Grrrr argggh.

Who knows, maybe the exception made it take a longer time..

IIRC the docs say that dunder methods are *not guaranteed* to be
called on the instance, because they may be called on the class.

> (2) If we ever do change to a C struct field, the behaviour will change.
> Maybe it's better to emulate the same behaviour from the start?
>
> (3) If there's a performance speed up, that's a bonus!
>
>
>
> --
> Steven
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com

From guido at python.org  Fri May 13 15:31:04 2016
From: guido at python.org (Guido van Rossum)
Date: Fri, 13 May 2016 12:31:04 -0700
Subject: [Python-Dev] Adding Type[C] to PEP 484
Message-ID: <CAP7+vJL+mpL=N31idm0SGWxAeChJ3eDWiedbYL0m93Ucs3PR6g@mail.gmail.com>

I've got another proposed update to PEP 484 ready. There was previously
some discussion on python-ideas (Bcc'ed for closure) and quite a bit on the
typing issue tracker: https://github.com/python/typing/issues/107

I've worked all that into a patch for PEP 484:
https://github.com/python/typing/pull/218

Once we're okay with that I will work on getting the implementation ready
in time for Python 3.5.2.

Here's the added text:

Meta-types
----------

A meta-type can be used to indicate a value that is itself a class
object that is a subclass of a given class.  It is spelled as
``Type[C]`` where ``C`` is a class.  To clarify: while ``C`` (when
used as an annotation) refers to instances of class ``C``, ``Type[C]``
refers to *subclasses* of ``C``.  (This is a similar distinction as
between ``object`` and ``type``.)

For example, suppose we have the following classes::

  class User: ...  # Abstract base for User classes
  class BasicUser(User): ...
  class ProUser(User): ...
  class TeamUser(User): ...

And suppose we have a function that creates an instance of one of
these classes if you pass it a class object::

  def new_user(user_class):
      user = user_class()
      # (Here we could write the user object to a database)
      return user

Without ``Type[]`` the best we could do to annotate ``new_user()``
would be::

  def new_user(user_class: type) -> User:
      ...

However using ``Type[]`` and a type variable with an upper bound we
can do much better::

  U = TypeVar('U', bound=User)
  def new_user(user_class: Type[U]) -> U:
      ...

Now when we call ``new_user()`` with a specific subclass of ``User`` a
type checker will infer the correct type of the result::

  joe = new_user(BasicUser)  # Inferred type is BasicUser

At runtime the value corresponding to ``Type[C]`` must be an actual
class object that's a subtype of ``C``, not a special form.  IOW, in
the above example calling e.g. ``new_user(Union[BasicUser, ProUser])``
is not allowed.

There are some concerns with this feature: for example when
``new_user()`` calls ``user_class()`` this implies that all subclasses
of ``User`` must support this in their constructor signature.  However
this is not unique to ``Type[]``: class methods have similar concerns.
A type checker ought to flag violations of such assumptions, but by
default constructor calls that match the constructor signature in the
indicated base class (``User`` in the example above) should be
allowed.  A program containing a complex or extensible class hierarchy
might also handle this by using a factory class method.

Plain ``Type`` without brackets is equivalent to using ``type`` (the
root of Python's metaclass hierarchy).  This equivalence also
motivates the name, ``Type``, as opposed to alternatives like
``Class`` or ``SubType``, which were proposed while this feature was
under discussion; this is similar to the relationship between
e.g. ``List`` and ``list``.


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160513/74558d17/attachment.html>

From brett at python.org  Fri May 13 15:53:03 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 13 May 2016 19:53:03 +0000
Subject: [Python-Dev] Adding Type[C] to PEP 484
In-Reply-To: <CAP7+vJL+mpL=N31idm0SGWxAeChJ3eDWiedbYL0m93Ucs3PR6g@mail.gmail.com>
References: <CAP7+vJL+mpL=N31idm0SGWxAeChJ3eDWiedbYL0m93Ucs3PR6g@mail.gmail.com>
Message-ID: <CAP1=2W4+gS3Cz7pUkSDcQEF49zJLtc4Zak8YX2iSrPT9U-OFAQ@mail.gmail.com>

On Fri, 13 May 2016 at 12:38 Guido van Rossum <guido at python.org> wrote:

> I've got another proposed update to PEP 484 ready. There was previously
> some discussion on python-ideas (Bcc'ed for closure) and quite a bit on the
> typing issue tracker: https://github.com/python/typing/issues/107
>
> I've worked all that into a patch for PEP 484:
> https://github.com/python/typing/pull/218
>
> Once we're okay with that I will work on getting the implementation ready
> in time for Python 3.5.2.
>

+1 from me.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160513/bcbeeb2c/attachment.html>

From guido at python.org  Fri May 13 15:55:40 2016
From: guido at python.org (Guido van Rossum)
Date: Fri, 13 May 2016 12:55:40 -0700
Subject: [Python-Dev] File system path PEP, 3rd draft
In-Reply-To: <CAP1=2W7NqCR7jxn71E6eKm_EXcqBrbVvFaOPQ+PkeGdpjzJFcA@mail.gmail.com>
References: <CAP1=2W6W+WO3-NjiSdCnqKVW1PBRzeEiW8gZGw1GSk1PLsufxA@mail.gmail.com>
 <20160513150810.2ae51e79.barry@wooz.org>
 <CAP1=2W7NqCR7jxn71E6eKm_EXcqBrbVvFaOPQ+PkeGdpjzJFcA@mail.gmail.com>
Message-ID: <CAP7+vJLFPEtVe9Xs=+RpmaU0BYsncj1um6R+=Eg5Ggv8TPN7kw@mail.gmail.com>

Can't wait until the peps repo is on GitHub!

On Fri, May 13, 2016 at 12:24 PM, Brett Cannon <brett at python.org> wrote:

>
>
> On Fri, 13 May 2016 at 12:08 Barry Warsaw <barry at python.org> wrote:
>
>> On May 13, 2016, at 06:37 PM, Brett Cannon wrote:
>>
>> >PEP: NNN
>> >Title: Adding a file system path protocol
>> >Version: $Revision$
>> >Last-Modified: $Date$
>> >Author: Brett Cannon <brett at python.org>,
>> >        Koos Zevenhoven <k7hoven at gmail.com>
>> >Status: Draft
>> >Type: Standards Track
>> >Content-Type: text/x-rst
>> >Created: 11-May-2016
>> >Post-History: 11-May-2016,
>> >              12-May-2016,
>> >              13-May-2016
>>
>> I think this is far enough along that you should self-assign a PEP number
>> and
>> check it into the official repo.
>>
>
> Paraphrasing *someone* from Twitter yesterday, I'm just going to work on
> GitHub until I absolutely have to check into hg, m'kay? ;)
>
> -Brett
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160513/d969a3fd/attachment.html>

From brett at python.org  Fri May 13 16:31:33 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 13 May 2016 20:31:33 +0000
Subject: [Python-Dev] quick GitHub transition update (was: File system path
 PEP, 3rd draft)
In-Reply-To: <CAP7+vJLFPEtVe9Xs=+RpmaU0BYsncj1um6R+=Eg5Ggv8TPN7kw@mail.gmail.com>
References: <CAP1=2W6W+WO3-NjiSdCnqKVW1PBRzeEiW8gZGw1GSk1PLsufxA@mail.gmail.com>
 <20160513150810.2ae51e79.barry@wooz.org>
 <CAP1=2W7NqCR7jxn71E6eKm_EXcqBrbVvFaOPQ+PkeGdpjzJFcA@mail.gmail.com>
 <CAP7+vJLFPEtVe9Xs=+RpmaU0BYsncj1um6R+=Eg5Ggv8TPN7kw@mail.gmail.com>
Message-ID: <CAP1=2W6ANmSYqQuTdJvj6WxSv1OdL4tKxtro_-HFLyNhpHi76Q@mail.gmail.com>

On Fri, 13 May 2016 at 12:56 Guido van Rossum <guido at python.org> wrote:

> Can't wait until the peps repo is on GitHub!
>

With devinabox now moved over to GitHub, the peps repo is next on the list
to move (and the devguide after that). I don't know if I'm going to be able
to make the migration happen before PyCon US, but my goal is to have it
either done before the language summit or have every little step lined up
by the end of the sprints and then finished up the week following.

I got one of the coveted 20 minute timeslots at the language summit to give
an update on everything and soliciting feedback on what needs to be in
place to migrate the cpython repo so I can have a clear idea of what work
is left to do.

If people want to prep for this discussion (which will be reported back
here so everyone can participate in the end), think about what you
*absolutely* must have in place to make working with GitHub for the cpython
repo work for you. I don't want any emails on this now (I've got 3 PEPs at
various stages plus travel between now and PyCon at this point so I'm busy
enough as it is, so any emails discussing this now *will be ignored* as I
don't have the mental bandwidth for that discussion prior to PyCon US).
Please just reflect on the required parts of your workflow over the next 2
weeks to know how much work you are going to ask of *me* as minimum
requirements to make the GitHub migration of the cpython repo happen (i.e.
I don't care about "nice to have", only about "without this I flat-out
won't contribute to Python on GitHub").


> On Fri, May 13, 2016 at 12:24 PM, Brett Cannon <brett at python.org> wrote:
>
>>
>>
>> On Fri, 13 May 2016 at 12:08 Barry Warsaw <barry at python.org> wrote:
>>
>>> On May 13, 2016, at 06:37 PM, Brett Cannon wrote:
>>>
>>> >PEP: NNN
>>> >Title: Adding a file system path protocol
>>> >Version: $Revision$
>>> >Last-Modified: $Date$
>>> >Author: Brett Cannon <brett at python.org>,
>>> >        Koos Zevenhoven <k7hoven at gmail.com>
>>> >Status: Draft
>>> >Type: Standards Track
>>> >Content-Type: text/x-rst
>>> >Created: 11-May-2016
>>> >Post-History: 11-May-2016,
>>> >              12-May-2016,
>>> >              13-May-2016
>>>
>>> I think this is far enough along that you should self-assign a PEP
>>> number and
>>> check it into the official repo.
>>>
>>
>> Paraphrasing *someone* from Twitter yesterday, I'm just going to work on
>> GitHub until I absolutely have to check into hg, m'kay? ;)
>>
>> -Brett
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>>
> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160513/621daf52/attachment.html>

From pjenvey at underboss.org  Fri May 13 17:30:10 2016
From: pjenvey at underboss.org (Philip Jenvey)
Date: Fri, 13 May 2016 14:30:10 -0700
Subject: [Python-Dev] File system path PEP, 3rd draft
In-Reply-To: <CAP1=2W6W+WO3-NjiSdCnqKVW1PBRzeEiW8gZGw1GSk1PLsufxA@mail.gmail.com>
References: <CAP1=2W6W+WO3-NjiSdCnqKVW1PBRzeEiW8gZGw1GSk1PLsufxA@mail.gmail.com>
Message-ID: <F74121BD-0DF5-486E-AEDB-BBCA35ED79E8@underboss.org>


> On May 13, 2016, at 11:37 AM, Brett Cannon <brett at python.org> wrote:
> 
> Biggest changes since the second draft:
> Resolve __fspath__() from the type, not the instance (for Guido)

>         if (PyObject_HasAttrString(path->ob_type, "__fspath__")) {
>             return PyObject_CallMethodObjArgs(path->ob_type, "__fspath__", path,
>                                             NULL);

_PyObject_LookupSpecial would be preferable then.

--
Philip Jenvey
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160513/c9ccda1b/attachment.html>

From brett at python.org  Fri May 13 17:48:04 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 13 May 2016 21:48:04 +0000
Subject: [Python-Dev] File system path PEP, 3rd draft
In-Reply-To: <F74121BD-0DF5-486E-AEDB-BBCA35ED79E8@underboss.org>
References: <CAP1=2W6W+WO3-NjiSdCnqKVW1PBRzeEiW8gZGw1GSk1PLsufxA@mail.gmail.com>
 <F74121BD-0DF5-486E-AEDB-BBCA35ED79E8@underboss.org>
Message-ID: <CAP1=2W5iq8ZpY5gxJG_wTPbFq-KiG--==ugYs+B3JpM1XLGVYw@mail.gmail.com>

On Fri, 13 May 2016 at 14:30 Philip Jenvey <pjenvey at underboss.org> wrote:

>
> On May 13, 2016, at 11:37 AM, Brett Cannon <brett at python.org> wrote:
>
> Biggest changes since the second draft:
>
>    1. Resolve __fspath__() from the type, not the instance (for Guido)
>
>
>         if (PyObject_HasAttrString(path->ob_type, "__fspath__")) {
>             return PyObject_CallMethodObjArgs(path->ob_type, "__fspath__",
> path,
>                                             NULL);
>
>
> _PyObject_LookupSpecial would be preferable then.
>

Yes it would be. :) I'll add it to the PEP.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160513/059ce4d4/attachment-0001.html>

From k7hoven at gmail.com  Fri May 13 18:49:24 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Sat, 14 May 2016 01:49:24 +0300
Subject: [Python-Dev] File system path PEP, 3rd draft
In-Reply-To: <CAP1=2W5iq8ZpY5gxJG_wTPbFq-KiG--==ugYs+B3JpM1XLGVYw@mail.gmail.com>
References: <CAP1=2W6W+WO3-NjiSdCnqKVW1PBRzeEiW8gZGw1GSk1PLsufxA@mail.gmail.com>
 <F74121BD-0DF5-486E-AEDB-BBCA35ED79E8@underboss.org>
 <CAP1=2W5iq8ZpY5gxJG_wTPbFq-KiG--==ugYs+B3JpM1XLGVYw@mail.gmail.com>
Message-ID: <CAMiohojmdcb-Zo5b8L+RKNzah85HnPmNy7CM6eyEHu=z55nA=g@mail.gmail.com>

FYI, I recently sent a couple of emails in my earlier type hinting
thread on -ideas. What I wrote now is about the path ABC regarding
type hints.

-- Koos

On Sat, May 14, 2016 at 12:48 AM, Brett Cannon <brett at python.org> wrote:
>
>
> On Fri, 13 May 2016 at 14:30 Philip Jenvey <pjenvey at underboss.org> wrote:
>>
>>
>> On May 13, 2016, at 11:37 AM, Brett Cannon <brett at python.org> wrote:
>>
>> Biggest changes since the second draft:
>>
>> Resolve __fspath__() from the type, not the instance (for Guido)
>>
>>
>>         if (PyObject_HasAttrString(path->ob_type, "__fspath__")) {
>>             return PyObject_CallMethodObjArgs(path->ob_type, "__fspath__",
>> path,
>>                                             NULL);
>>
>>
>> _PyObject_LookupSpecial would be preferable then.
>
>
> Yes it would be. :) I'll add it to the PEP.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
>

From nad at python.org  Fri May 13 19:21:40 2016
From: nad at python.org (Ned Deily)
Date: Fri, 13 May 2016 19:21:40 -0400
Subject: [Python-Dev] First 3.6.0 alpha release coming up: 2016-05-16 12:00
 UTC
Message-ID: <9B21FE29-226F-4DC6-AA8E-4651CDCD1BDB@python.org>

Hi everyone!

Thank you for all of your work so far on our next feature release cycle, Python 3.6.  Code checkins for 3.6 began 12 months ago, after the feature code cutoff and beta phase for Python 3.5.0.  We are now about to do 3.6.0 alpha 1, our first of 4 alpha and 4 beta releases planned for the next 6 months.

As a reminder, alpha releases are intended to make it easier for the wider community to test the current state of new features and bug fixes for an upcoming Python release as a whole and for us to test the release process.  The start of the alpha phase does *not* mean feature development is complete.  During the alpha phase, features may be added, modified, or deleted up until the start of the beta phase.  Alpha users beware!

3.6.0 beta 1, currently scheduled for 2016-09-07, marks feature code cutoff, at which time new feature development is expected to be complete and no additional features are to be added to 3.6.  At the same time, feature development will begin for the following release cycle, 3.7.  During the 3.6.0 beta phase, which is planned to end on 2016-12-04 with the first final release candidate, the focus will be on bug fixes and regressions and preparing for the final release.  The goal is for the first release candidate to be the same as the final release: that is, no changes will be accepted for 3.6.0 after release candidate 1 without discussion, review, and release manager approval.  Please plan accordingly!

Although I've been involved behind the scenes in most releases over the past several years, 3.6.0a1 will be my first as a release manager so no doubt there will be bumps along the way.  Many thanks to my fellow release managers, particularly Benjamin, Georg, Larry, and Barry, for their help and support. Your understanding, help, and feedback will be greatly appreciated!

For this first alpha release at least, the release team (primarily Steve Dower and myself) are going to try a slightly different timetable for producing the release.  We are planning for the code cutoff for alpha 1 to be on 2016-05-16 around 12:00 UTC; that allows for code checkins to continue through Sunday almost anywhere on the planet.  We will then go off and manufacture the release bits and make them available via the python.org website as soon as possible thereafter, assuming no major glitches, that should be no later than 24 hours thereafter.

In the remaining hours until the code cutoff, please get your commits in as soon as possible and please pay attention to the buildbots for possible regressions.

Looking ahead, the next alpha release, 3.6.0a2, will follow in about a month on 2016-06-12, which is a week after the end of the development sprints at US PyCon 2016 in Portland, Oregon.  Hope to see you there!
 
To summarize:

2016-05-16 ~12:00 UTC: code snapshot for 3.6.0 alpha 1

now to 2016-09-07: Alpha phase (unrestricted feature development)

2016-09-07: 3.6.0 feature code freeze, 3.7.0 feature development begins

2016-09-07 to 2016-12-04: 3.6.0 beta phase (bug and regression fixes, no new features)

2016-12-04 3.6.0 release candidate 1 (3.6.0 code freeze)

2016-12-16 3.6.0 release (3.6.0rc1 plus, if necessary, any dire emergency fixes)


Let me know if you have questions or comments.  And thanks again for all of your work towards making Python even better!


--Ned D
  AKA The Other Ned

https://www.python.org/dev/peps/pep-0494/
https://docs.python.org/devguide/devcycle.html#stages
https://en.wikipedia.org/wiki/Software_release_life_cycle#Alpha
https://docs.python.org/devguide/buildbots.html

--
  Ned Deily
  nad at python.org -- []
  Freenode IRC: ned_deily, #python-dev


From greg.ewing at canterbury.ac.nz  Fri May 13 21:02:55 2016
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 14 May 2016 13:02:55 +1200
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CACac1F_dtX+K+kE=Z5AWGY8h3p2E=x4_fx0KoQ7eUhMuMddJVw@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
 <57358915.7040606@mail.de>
 <CAMiohoijCDKBuMCioY48UkU=WYy+DJqJyCL6mDBFx1gRXAv+0A@mail.gmail.com>
 <57359D69.6040108@mail.de>
 <CAMiohojq6DDLmWbDv1TiB26CVoUFNw5mNSaiPB+w2zeOgcqOUg@mail.gmail.com>
 <5735FB79.1010201@mail.de> <5736076F.6010905@stoneleaf.us>
 <CACac1F_dtX+K+kE=Z5AWGY8h3p2E=x4_fx0KoQ7eUhMuMddJVw@mail.gmail.com>
Message-ID: <5736793F.5030401@canterbury.ac.nz>

Paul Moore wrote:
> On 13 May 2016 at 17:57, Ethan Furman <ethan at stoneleaf.us> wrote:
> 
>>1) What is a wallet garden?
> 
> I assumed he meant "walled garden"

Works either way -- you'd want a wall around your wallet
garden to stop people stealing your wallets.

-- 
Greg

From stephen at xemacs.org  Sat May 14 02:56:30 2016
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 14 May 2016 15:56:30 +0900
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAPTjJmqszcTwh2MDigY1JoeP6cyWvAVpyKvY0v09z-JirEmHwg@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <20160512172225.GA16846@sjoerdjob.com>
 <20160513101939.GG12028@ando.pearwood.info>
 <CAPTjJmqszcTwh2MDigY1JoeP6cyWvAVpyKvY0v09z-JirEmHwg@mail.gmail.com>
Message-ID: <22326.52254.70620.535515@turnbull.sk.tsukuba.ac.jp>

Chris Angelico writes:

 > AFAICT, the compatibility layer would simply decode the bytes using
 > surrogateescape handling, which should round-trip anything.

By design.  See PEP 383.  Or rather, the OP should; he has not done
his homework and is confused by his own FUD.  This whole subthread is
really python-list territory.

Since a lot of people I respect seem uncertain about the facts, for
the record, let's lay out the (putative) issues remaining for
post-PEP-383 Python vs. str-y path objects.

(0) "Can't work with some POSIX (bytes) paths" is closed by PEP
    383, forget it.  os.fsdecode(bytespath) as soon as you get one,
    os.fsencode(strpath) just before you need one, done.  Surrogates
    embedded in strpath may need special handling depending on the
    application (see (1)).

(1) str.encode(errors='strict') (the default) will blow up on embedded
    surrogates.  Yes, but that's a *good* thing if you're mixing str
    derived from filesystem paths with other text.  There's no way to
    avoid it.  If you're just passing it back to open(), it Just
    Works, done.

(2) You're using bytes as text a la 2.x for "efficiency's" sake, and
    you're worried that you might pass a str-y Path deep into bytes
    territory and it will explode there.

    I don't think there is any sympathy left for that use case on
    Python dev channels.  Define a clear boundary with well-defined
    entry and exit gates, and convert there.  Then you can get some
    sleep.  (How-to example: your "compatibility layer".)

(3) You're worried about inefficiency of decoding/encoding to the same
    or trivially changed bytes (ie, you didn't need pathlib in the
    first place, but you got it anyway) -- this especially matters for
    2.7, but is significant for 3.x too, if you're using a bunch of
    paths in a tight loop.

    I don't have sympathy for that use case, but Brett and Guido do,
    and Brett's PEP handles it by making __fspath__ polymorphic in the
    usual os.path-y way, with Guido's modification.

    This is always a tradeoff.  If you know your JPEGs all have
    extension '.JPG' and

        png_path = jpeg_path[:-4] + b'.png'

    is readable enough for you, use that, not pathlib or Antipathy,
    and you get your efficiency.  (Doing jpeg_path.rindex(b'.') is left
    as an exercise for the reader.  Part (i): Is it really worth it?)

    If you want the readability of a rich path library and the
    efficiency of bytes, you *may* have the option of using Ethan's
    Antipathy (or whatever).

    If you can't use Antipathy, use bytes methods directly, or accept
    that it isn't *that* inefficient and use pathlib.  At this point,
    I think this subcase is just FUD, no real examples were presented
    where the efficiency hit of encoding/decoding gets in the way of
    getting work done using pathlib.

    If you need to stick to stdlib for some reason (eg, to use a
    higher-level library that uses pathlib), live with the
    "compatibility layer"'s inefficiency.  Decoding and encoding are
    actually rather low-cost operations at path lengths (PATHMAX=256
    was common, not so long ago!).  Most high-level libraries will
    impose a lot more overhead elsewhere, and calling into pathlib by
    itself will add a certain amount of overhead as well.

(4) Lack of transparency/readability for "simple" operations.  If
    Antipathy is something you can use, I agree it's plausible that
    avoiding a few os.fsdecode and os.fsencode calls would look nicer,
    but this is really a style question.

    My take: I think of paths as human-readable, so presenting them as
    str (not bytes) is important to me, important enough that I
    advocate that position to other developers.  If you do the
    conversion at the boundary between a bytes-y module and pathlib
    ("compatibility layer") I don't see how it affects readability of
    the path manipulation code, while data marshaling at boundaries is
    a expected fact of software development.  YMMV.

(0) is thus a non-issue.  (1) is not something that can be addressed
by general principles, let alone language design.  (2)-(4) are all
real issues regardless of how I feel they should be resolved :-), but
they're all design trade-offs, not things that can completely block
you from getting some kinds of work done in your own style (eg, the
situation str-minded people were in before PEP 383).

Python 3 is an example of how language design can help alleviate
issues like (2), by discouraging that use case in various ways.
Brett's PEP is an example of how language design can help alleviate
issues like (3) and (4).  In particular, it helps us to interface
pathlib to open() and friends in a very natural, readable way, without
explicit conversions that should be unnecessary by the nature of the
operation and its arguments.  By contrast, the conversion of bytes to
str is important to do explicitly because they are different
representations of the same thing, and it's important that readers be
notified of that change of representation.

 > Or am I wrong here somewhere?

Well, considering the length of this irrelevant-to-the-PEP subthread,
arguably you are feeding a successful troll.  I hope that having
posted the above, in the future there will be *one*, *short* reply to
such questions:

		    Not a problem.  Read PEP 383.

and the thread will end there.

Steve

From srkunze at mail.de  Sat May 14 05:36:08 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Sat, 14 May 2016 11:36:08 +0200
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAPTjJmrQtZckThXR-r5eRARmCZHJWgQSW_CF9+LmSUV1T2eRoQ@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
 <57358915.7040606@mail.de>
 <CAMiohoijCDKBuMCioY48UkU=WYy+DJqJyCL6mDBFx1gRXAv+0A@mail.gmail.com>
 <57359D69.6040108@mail.de>
 <CAP1=2W62hAtWN96aG0UO0nm0UXY1Tm3MrNX_UefR0eb=EW4BSw@mail.gmail.com>
 <573601EB.50904@mail.de>
 <CAPTjJmrQtZckThXR-r5eRARmCZHJWgQSW_CF9+LmSUV1T2eRoQ@mail.gmail.com>
Message-ID: <5736F188.90705@mail.de>

On 13.05.2016 18:43, Chris Angelico wrote:
> https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_Check
>
> https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_CheckExact

Thanks for pointing me at this.
I searched via github and found usages only: 
https://github.com/python/cpython/search?utf8=%E2%9C%93&q=PyUnicode_Check

> "Check" accepts subclasses; "CheckExact" doesn't (it's like "type(x)
> is str"). The question is, which one SHOULD be being done? What should
> this do:
>
> class TmpPath(str):
>      def __fspath__(self):
>          return "/tmp/"+self
> x = TmpPath("foo/bar")
> open(x, "w")
>
> Does that respect __fspath__, or respect the fact that it's a string?

Fine example. Thinking naively, I would say, when somebody made an 
effort to write __fspath__, it should be respected. Maybe, that's just me.

Not sure if that can be a source of errors, when somebody adds str as 
baseclass later because the team needs str functionality. At least I 
would expect __fspath__ to still work .

Best,
Sven

From ncoghlan at gmail.com  Sat May 14 08:24:04 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 14 May 2016 22:24:04 +1000
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <5736F188.90705@mail.de>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
 <57358915.7040606@mail.de>
 <CAMiohoijCDKBuMCioY48UkU=WYy+DJqJyCL6mDBFx1gRXAv+0A@mail.gmail.com>
 <57359D69.6040108@mail.de>
 <CAP1=2W62hAtWN96aG0UO0nm0UXY1Tm3MrNX_UefR0eb=EW4BSw@mail.gmail.com>
 <573601EB.50904@mail.de>
 <CAPTjJmrQtZckThXR-r5eRARmCZHJWgQSW_CF9+LmSUV1T2eRoQ@mail.gmail.com>
 <5736F188.90705@mail.de>
Message-ID: <CADiSq7ef0n5_BXb5voH4WvJv7wP-ygkRVF5guFBN2r1HKuEsgg@mail.gmail.com>

On 14 May 2016 at 19:36, Sven R. Kunze <srkunze at mail.de> wrote:
> Fine example. Thinking naively, I would say, when somebody made an effort to
> write __fspath__, it should be respected. Maybe, that's just me.

Over time, we've generally moved towards treating subclasses as if
they were the base class where protocols are concerned, and
encouraging the use of composition if people don't actually want that
"is-a" behaviour.

For example, compare the int protocol to the last major type protocol
we added, index:

>>> from operator import index
>>> class MyInt(int):
...     def __int__(self):
...         return 42
...     def __index__(self):
...         return 42
...
>>> x = MyInt(5)
>>> x
5
>>> int(x)
42
>>> index(x)
5

In the case of retroactively added protocols like operator.index() and
os.fspath(), where inheritance from the relevant builtin type was
previously the only option, retaining that behaviour is also the only
way to avoid a performance regression for those subclasses.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From pludemann at google.com  Sat May 14 14:18:59 2016
From: pludemann at google.com (Peter Ludemann)
Date: Sat, 14 May 2016 11:18:59 -0700
Subject: [Python-Dev] Adding Type[C] to PEP 484
In-Reply-To: <CAP1=2W4+gS3Cz7pUkSDcQEF49zJLtc4Zak8YX2iSrPT9U-OFAQ@mail.gmail.com>
References: <CAP7+vJL+mpL=N31idm0SGWxAeChJ3eDWiedbYL0m93Ucs3PR6g@mail.gmail.com>
 <CAP1=2W4+gS3Cz7pUkSDcQEF49zJLtc4Zak8YX2iSrPT9U-OFAQ@mail.gmail.com>
Message-ID: <CACsRUKLyWjCFWP6UtNZf7+NsJ0JdHSVFMiqk2ZmS7d7qj7XmSw@mail.gmail.com>

Is Type[C,D] allowed? Or should multiple types be restricted to TypeVar?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160514/5796166f/attachment.html>

From gvanrossum at gmail.com  Sat May 14 14:30:00 2016
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Sat, 14 May 2016 11:30:00 -0700
Subject: [Python-Dev] Adding Type[C] to PEP 484
In-Reply-To: <CACsRUKLyWjCFWP6UtNZf7+NsJ0JdHSVFMiqk2ZmS7d7qj7XmSw@mail.gmail.com>
References: <CAP7+vJL+mpL=N31idm0SGWxAeChJ3eDWiedbYL0m93Ucs3PR6g@mail.gmail.com>
 <CAP1=2W4+gS3Cz7pUkSDcQEF49zJLtc4Zak8YX2iSrPT9U-OFAQ@mail.gmail.com>
 <CACsRUKLyWjCFWP6UtNZf7+NsJ0JdHSVFMiqk2ZmS7d7qj7XmSw@mail.gmail.com>
Message-ID: <CAP7+vJLYPR1c+ZkB3Z9U57X8vktVn7czA6p4Xi6cd_CjnBDofg@mail.gmail.com>

What would Type[C, D] mean?

--Guido (mobile)
On May 14, 2016 11:21 AM, "Peter Ludemann via Python-Dev" <
python-dev at python.org> wrote:

> Is Type[C,D] allowed? Or should multiple types be restricted to TypeVar?
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160514/1c9a990d/attachment.html>

From pludemann at google.com  Sat May 14 14:47:15 2016
From: pludemann at google.com (Peter Ludemann)
Date: Sat, 14 May 2016 11:47:15 -0700
Subject: [Python-Dev] Adding Type[C] to PEP 484
In-Reply-To: <CAP7+vJLYPR1c+ZkB3Z9U57X8vktVn7czA6p4Xi6cd_CjnBDofg@mail.gmail.com>
References: <CAP7+vJL+mpL=N31idm0SGWxAeChJ3eDWiedbYL0m93Ucs3PR6g@mail.gmail.com>
 <CAP1=2W4+gS3Cz7pUkSDcQEF49zJLtc4Zak8YX2iSrPT9U-OFAQ@mail.gmail.com>
 <CACsRUKLyWjCFWP6UtNZf7+NsJ0JdHSVFMiqk2ZmS7d7qj7XmSw@mail.gmail.com>
 <CAP7+vJLYPR1c+ZkB3Z9U57X8vktVn7czA6p4Xi6cd_CjnBDofg@mail.gmail.com>
Message-ID: <CACsRUKLNXnsUCNeC4g0j2yneqSNhNf4NVsyhRJgTegVg3srmZg@mail.gmail.com>

I think that Type[C,D] would mean Union[Type[C], Type[D]], but I'm not sure
... I should probably talk to a typing expert about this.
(Sorry for thinking out loud; but if we decide that Type[C,D] doesn't make
sense, we need to prohibit it)

I suppose I'm asking: do we allow new_user(Type[BasicUser, ProUser])?

Here's a trivial example that off the top of my head makes sense:

BasicOrProUser = TypeVar('BasicOrProUser', BasicUser, ProUser)
def new_user(name: str, user_factory: Type[BasicOrProuser]) ->
BasicOrProUser:
    return user_factory(name)

Or must the Type[...] item have been defined with a TypeVar(...,
bound=...), in which case multiple types aren't allowed with Type[...]?


On 14 May 2016 at 11:30, Guido van Rossum <gvanrossum at gmail.com> wrote:

> What would Type[C, D] mean?
>
> --Guido (mobile)
> On May 14, 2016 11:21 AM, "Peter Ludemann via Python-Dev" <
> python-dev at python.org> wrote:
>
>> Is Type[C,D] allowed? Or should multiple types be restricted to TypeVar?
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160514/83adc33d/attachment.html>

From gvanrossum at gmail.com  Sat May 14 16:28:46 2016
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Sat, 14 May 2016 13:28:46 -0700
Subject: [Python-Dev] Adding Type[C] to PEP 484
In-Reply-To: <CACsRUKLNXnsUCNeC4g0j2yneqSNhNf4NVsyhRJgTegVg3srmZg@mail.gmail.com>
References: <CAP7+vJL+mpL=N31idm0SGWxAeChJ3eDWiedbYL0m93Ucs3PR6g@mail.gmail.com>
 <CAP1=2W4+gS3Cz7pUkSDcQEF49zJLtc4Zak8YX2iSrPT9U-OFAQ@mail.gmail.com>
 <CACsRUKLyWjCFWP6UtNZf7+NsJ0JdHSVFMiqk2ZmS7d7qj7XmSw@mail.gmail.com>
 <CAP7+vJLYPR1c+ZkB3Z9U57X8vktVn7czA6p4Xi6cd_CjnBDofg@mail.gmail.com>
 <CACsRUKLNXnsUCNeC4g0j2yneqSNhNf4NVsyhRJgTegVg3srmZg@mail.gmail.com>
Message-ID: <CAP7+vJJ=_w7Grc_RiG18TQtKhDFYicqxULx4pnfX2UiX6MdPdQ@mail.gmail.com>

On Sat, May 14, 2016 at 11:47 AM, Peter Ludemann <pludemann at google.com>
wrote:

> I think that Type[C,D] would mean Union[Type[C], Type[D]], but I'm not
> sure ... I should probably talk to a typing expert about this.
> (Sorry for thinking out loud; but if we decide that Type[C,D] doesn't make
> sense, we need to prohibit it)
>

It sounds iffy enough to prohibit.


> I suppose I'm asking: do we allow new_user(Type[BasicUser, ProUser])?
>

That looks like a call, but Type[] is something to use in the definition.


> Here's a trivial example that off the top of my head makes sense:
>
> BasicOrProUser = TypeVar('BasicOrProUser', BasicUser, ProUser)
> def new_user(name: str, user_factory: Type[BasicOrProuser]) ->
> BasicOrProUser:
>     return user_factory(name)
>

That looks reasonable and only has one parameter to Type[].


> Or must the Type[...] item have been defined with a TypeVar(...,
> bound=...), in which case multiple types aren't allowed with Type[...]?
>

You can use whatever you want. If you want to write Type[int] go right
ahead. Even Type[Union[int, str]] should be allowed. There is *usually* a
good reason to use a type variable though -- without one the return type
would be the base class. (But then again maybe that could be what you want,
or you might not even have a return type, e.g. when the thing just gets
thrown into the database.)

--Guido


>
>
> On 14 May 2016 at 11:30, Guido van Rossum <gvanrossum at gmail.com> wrote:
>
>> What would Type[C, D] mean?
>>
>> --Guido (mobile)
>> On May 14, 2016 11:21 AM, "Peter Ludemann via Python-Dev" <
>> python-dev at python.org> wrote:
>>
>>> Is Type[C,D] allowed? Or should multiple types be restricted to TypeVar?
>>>
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>>
>>>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160514/4c7488c8/attachment.html>

From cesare.di.mauro at gmail.com  Sun May 15 03:23:15 2016
From: cesare.di.mauro at gmail.com (Cesare Di Mauro)
Date: Sun, 15 May 2016 09:23:15 +0200
Subject: [Python-Dev] Speeding up CPython 5-10%
In-Reply-To: <CAMpsgwZXxKDzUAQ+_hApdHPQEwnEQLUyQz764VvMW5EXywd=NA@mail.gmail.com>
References: <56A90B97.7090001@gmail.com>
 <CAMpsgwZXxKDzUAQ+_hApdHPQEwnEQLUyQz764VvMW5EXywd=NA@mail.gmail.com>
Message-ID: <CAP7v7k7eHEKFqbVX2-7RudS0u4oqhrVTuC25FE_dJo8PXQaimA@mail.gmail.com>

2016-02-02 10:28 GMT+01:00 Victor Stinner <victor.stinner at gmail.com>:

> 2016-01-27 19:25 GMT+01:00 Yury Selivanov <yselivanov.ml at gmail.com>:
> > tl;dr The summary is that I have a patch that improves CPython
> performance
> > up to 5-10% on macro benchmarks.  Benchmarks results on Macbook Pro/Mac
> OS
> > X, desktop CPU/Linux, server CPU/Linux are available at [1].  There are
> no
> > slowdowns that I could reproduce consistently.
>
> That's really impressive, great job Yury :-) Getting non-negligible
> speedup on large macrobenchmarks became really hard in CPython.
> CPython is already well optimized in all corners.


It's long time since I took a look at CPython (3.2), but if it didn't
changed a lot then there might be some corner cases still waiting to be
optimized. ;-)

Just one thing that comes to my mind: is the stack depth calculation
routine changed? It was suboptimal, and calculating a better number
decreases stack allocation, and increases the frame usage.


> It looks like the
> overall Python performance still depends heavily on the performance of
> dictionary and attribute lookups. Even if it was well known, I didn't
> expect up to 10% speedup on *macro* benchmarks.
>

True, but it might be mitigated in some ways, at least for built-in types.
There are ideas about that, but they are a bit complicated to implement.

The problem is with functions like len, which IMO should become attribute
lookups ('foo'.len) or method executions ('foo'.len()). Then it'll be
easier to accelerate their execution, with one of the above ideas.

However such kind of changes belong to Guido, which defines the language
structure/philosophy. IMO something like len should be part of the
attributes exposed by an object: it's more "object-oriented". Whereas other
things like open, file, sum, etc., are "general facilities".

Regards,
Cesare
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160515/857ee8ff/attachment.html>

From k7hoven at gmail.com  Sun May 15 07:57:39 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Sun, 15 May 2016 14:57:39 +0300
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <CAPTjJmrQtZckThXR-r5eRARmCZHJWgQSW_CF9+LmSUV1T2eRoQ@mail.gmail.com>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
 <57358915.7040606@mail.de>
 <CAMiohoijCDKBuMCioY48UkU=WYy+DJqJyCL6mDBFx1gRXAv+0A@mail.gmail.com>
 <57359D69.6040108@mail.de>
 <CAP1=2W62hAtWN96aG0UO0nm0UXY1Tm3MrNX_UefR0eb=EW4BSw@mail.gmail.com>
 <573601EB.50904@mail.de>
 <CAPTjJmrQtZckThXR-r5eRARmCZHJWgQSW_CF9+LmSUV1T2eRoQ@mail.gmail.com>
Message-ID: <CAMiohoiWdb-OTLdiMyOTgcmTk2F0dKjgCecx9Y_EZqr5EkXeGw@mail.gmail.com>

On Fri, May 13, 2016 at 7:43 PM, Chris Angelico <rosuav at gmail.com> wrote:
[...]
> "Check" accepts subclasses; "CheckExact" doesn't (it's like "type(x)
> is str"). The question is, which one SHOULD be being done?

Indeed it should do "Check", so that path libraries that do inherit
from str will still work (see also below).

> What should this do:
>
> class TmpPath(str):
>     def __fspath__(self):
>         return "/tmp/"+self
> x = TmpPath("foo/bar")
> open(x, "w")
>
> Does that respect __fspath__, or respect the fact that it's a string?

It should not respect __fspath__ because that would mean the behavior
is different in older versions of Python(!). This also means that such
code should never be written (except for that example you wrote ;-).
So indeed, composition would be the way to go.

So yes, I still think str subclasses should not implement __fspath__,
and that, at the very least, the docs should say that :).

-- Koos


> ChrisA
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com

From cesare.di.mauro at gmail.com  Sun May 15 08:12:46 2016
From: cesare.di.mauro at gmail.com (Cesare Di Mauro)
Date: Sun, 15 May 2016 14:12:46 +0200
Subject: [Python-Dev] Wordcode v2
In-Reply-To: <CAA1B2Jxs0Vj0wgynRwDoFpXbVT+FmBE+CVKnEL1y77iyQL-_GQ@mail.gmail.com>
References: <CAA1B2Jxs0Vj0wgynRwDoFpXbVT+FmBE+CVKnEL1y77iyQL-_GQ@mail.gmail.com>
Message-ID: <CAP7v7k7-YPVfdrZejcK=WEEYy1dxMuXLz0R3sxW-yrOFcHZrmg@mail.gmail.com>

2016-02-15 1:20 GMT+01:00 Demur Rumed <gunkmute at gmail.com>:

> Saw recent discussion:
> https://mail.python.org/pipermail/python-dev/2016-February/143013.html
>
> I remember trying WPython; it was fast. Unfortunately it feels it came at
> the wrong time when development was invested in getting py3k out the door.
>

Not only that. IMO the primary problem was related to the fact the "patch"
was too big to be reviewed. Unfortunately it was my first attempt, and
having worked alone I introduced too much optimizations and (heavy) changes
to the code. An incremental approach should have worked better, albeit I
believe that such drastic move from the consolidated bytecodes to the new
wordcodes would have produced strong resistance anyway.


> It also had a lot of other ideas like *_INT instructions which allowed
> having oparg to be a constant int rather than needing to LOAD_CONST one.
>

This, specifically, was an experiment that I made with WPython 1.1, which I
recommend to do not follow. There are other, more general, ways to speedup
the execution when dealing with integers.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160515/9a07d1e8/attachment.html>

From cesare.di.mauro at gmail.com  Sun May 15 08:27:52 2016
From: cesare.di.mauro at gmail.com (Cesare Di Mauro)
Date: Sun, 15 May 2016 14:27:52 +0200
Subject: [Python-Dev] Wordcode v2
In-Reply-To: <E5868ACB-414D-4E1B-AD70-B3657905C555@yahoo.com>
References: <CAA1B2Jxs0Vj0wgynRwDoFpXbVT+FmBE+CVKnEL1y77iyQL-_GQ@mail.gmail.com>
 <CAP7+vJL=yg9pEDcZBqZ5xrKshomc6BJiLgOT8Ah=WF=C2goXRw@mail.gmail.com>
 <E5868ACB-414D-4E1B-AD70-B3657905C555@yahoo.com>
Message-ID: <CAP7v7k7fpjxJoJ7tz7qa5RtEKXCnCv=rDdN0jVe9e8Si65nnmA@mail.gmail.com>

2016-02-15 8:14 GMT+01:00 Andrew Barnert via Python-Dev <
python-dev at python.org>:

> Despite the name (and inspiration), my fork has very little to do with
> WPython. I'm just focused on simpler (hopefully = faster) fetch code; he
> started with that, but ended up going the exact opposite direction,
> accepting more complicated (and much slower) fetch code as a reasonable
> cost for drastically reducing the number of instructions. (If you double
> the 30% fetch-and-parse overhead per instruction, but cut the number of
> instructions to 40%, the net is a huge win.)


I don't know why you consider slower the WPython's code that fetches the
more complicated instructions. On the contrary, I've structured such
"superinstructions" in order to simplify their decoding. Arguments are
decoded as they are needed in a specific moment, in order to reduce or
completely avoid the usage of temporary variables to keep such values.

Can you provide some example about your claim?

Regarding the WPython goal, it wasn't only about introducing simpler
instructions. As I've written also in my presentation, it's an hybrid VM:
stack and register-based. I've introduced a new instruction format for the
existing CPython's instructions, which are now easier to fetch, decode,
execute, and provide a better density too (for the most common case:
arguments with a maximum of 255 as value/index).
However I've also added superinstructions to better pack more "useful
work", which provides more code density and they are the primary
responsible for improving the execution speed.

Regards,
Cesare
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160515/51ba5171/attachment-0001.html>

From cesare.di.mauro at gmail.com  Sun May 15 08:32:02 2016
From: cesare.di.mauro at gmail.com (Cesare Di Mauro)
Date: Sun, 15 May 2016 14:32:02 +0200
Subject: [Python-Dev] Wordcode v2
In-Reply-To: <loom.20160217T120242-244@post.gmane.org>
References: <CAA1B2Jxs0Vj0wgynRwDoFpXbVT+FmBE+CVKnEL1y77iyQL-_GQ@mail.gmail.com>
 <loom.20160217T120242-244@post.gmane.org>
Message-ID: <CAP7v7k6mfvyP8WhJPSF5tY8+vLQv5Z1me3aw+Fey2O42RYBZBg@mail.gmail.com>

2016-02-17 12:04 GMT+01:00 Antoine Pitrou <antoine at python.org>:

> Demur Rumed <gunkmute <at> gmail.com> writes:
> > I've personally benchmarked this fork with positive results.
>
> I'm skeptical of claims like this. What did you benchmark exactly, and with
> which results?
> I don't think changing the opcode encoding per se will bring any large
> benefit...
>
> Regards
>
> Antoine.


With WPython I've introduced several optimizations which improved a lot the
execution speed (+25% with PyStone, at the time, compared to CPython 2.6),
but most of the benefits came from the new opcode format.

Regards,
Cesare
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160515/3271b5b1/attachment.html>

From cesare.di.mauro at gmail.com  Sun May 15 09:10:40 2016
From: cesare.di.mauro at gmail.com (Cesare Di Mauro)
Date: Sun, 15 May 2016 15:10:40 +0200
Subject: [Python-Dev] Wordcode: new regular bytecode using 16-bit units
In-Reply-To: <CAMpsgwaaDW2ZcPWXqnzTNz19CN-vKw7=fUj==32HbdEh4+oEuA@mail.gmail.com>
References: <CAMpsgwaaDW2ZcPWXqnzTNz19CN-vKw7=fUj==32HbdEh4+oEuA@mail.gmail.com>
Message-ID: <CAP7v7k5baaQ9Ss_n=Fh8Yzkp+fnDeehAfcopdGBm_TYP9Tc_tw@mail.gmail.com>

2016-04-13 18:24 GMT+02:00 Victor Stinner <victor.stinner at gmail.com>:

> Demur Rumed proposes a different change to use a regular bytecode
> using 16-bit units: an instruction has always one 8-bit argument, it's
> zero if the instruction doesn't have an argument:
>
>    http://bugs.python.org/issue26647
>
> According to benchmarks, it looks faster:
>
>   http://bugs.python.org/issue26647#msg263339
>
> IMHO it's a nice enhancement: it makes the code simpler. The most
> interesting change is made in Python/ceval.c:
>
> -        if (HAS_ARG(opcode))
> -            oparg = NEXTARG();
> +        oparg = NEXTARG();
>
> This code is the very hot loop evaluating Python bytecode. I expect
> that removing a conditional branch here can reduce the CPU branch
> misprediction.
>

Correct. The old bytecode format wasn't so much predictable for the CPU.

>
> Right now, ceval.c still fetchs opcode and then oparg with two 8-bit
> instructions. Later, we can discuss if it would be possible to ensure
> that the bytecode is always aligned to 16-bit in memory to fetch the
> two bytes using a uint16_t* pointer.
>
> Maybe we can overallocate 1 byte in codeobject.c and align manually
> the memory block if needed. Or ceval.c should maybe copy the code if
> it's not aligned?
>
> Raymond Hettinger proposes something like that, but it looks like
> there are concerns about non-aligned memory accesses:
>
>    http://bugs.python.org/issue25823
>
> The cost of non-aligned memory accesses depends on the CPU
> architecture, but it can raise a SIGBUS on some arch (MIPS and
> SPARC?).
>
> Victor
>

It should not be a problem, since every PyObject is allocated with PyAlloc
(however I don't remember if it's the correct name) which AFAIK guarantees
a base 8 bytes alignment.

So, it's safe to use an unsigned int for keeping/referencing a word at the
time.

The only problem with such approach is related to the processor endianess,
but it can be solved with proper macros (like I did with WPython).

Regards,
Cesare
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160515/1d3f2b03/attachment.html>

From cesare.di.mauro at gmail.com  Sun May 15 09:25:27 2016
From: cesare.di.mauro at gmail.com (Cesare Di Mauro)
Date: Sun, 15 May 2016 15:25:27 +0200
Subject: [Python-Dev] Wordcode: new regular bytecode using 16-bit units
In-Reply-To: <CAMpsgwbvNpoi969FWDyNkxr1X3qeN-1yuc4nkRiq7RjU9rDvZA@mail.gmail.com>
References: <CAMpsgwaaDW2ZcPWXqnzTNz19CN-vKw7=fUj==32HbdEh4+oEuA@mail.gmail.com>
 <030201d195c7$ca9de130$5fd9a390$@gmail.com>
 <CAMpsgwbvNpoi969FWDyNkxr1X3qeN-1yuc4nkRiq7RjU9rDvZA@mail.gmail.com>
Message-ID: <CAP7v7k5RbQ+RGZJD_k7jKxsbEHu5R-qETa9sRjBJzF2gL7vrkQ@mail.gmail.com>

2016-04-13 23:23 GMT+02:00 Victor Stinner <victor.stinner at gmail.com>:

> Hopefully, I don't expect 32-bit parameters in the wild, only 24-bit
> parameter for function with annotation.
>

I never found 32-bit parameters, and not even 24-bit ones. I think that
their usage is as rare as all planets alignment. ;-)

That's why with in WPython I supported only 8, 16, and 32-bit parameters
(which are 6 bytes long).

Regards,
Cesare
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160515/efe6e590/attachment.html>

From cesare.di.mauro at gmail.com  Sun May 15 10:00:09 2016
From: cesare.di.mauro at gmail.com (Cesare Di Mauro)
Date: Sun, 15 May 2016 16:00:09 +0200
Subject: [Python-Dev] Speeding up CPython 5-10%
In-Reply-To: <56AF8DB9.30900@gmail.com>
References: <56A90B97.7090001@gmail.com>
 <20160130042835.GJ4619@ando.pearwood.info>
 <56AF8DB9.30900@gmail.com>
Message-ID: <CAP7v7k6mh0-G7v60K2xMWTatk0Xb1cLtz3DfYRUweGqBEoM4BQ@mail.gmail.com>

2016-02-01 17:54 GMT+01:00 Yury Selivanov <yselivanov.ml at gmail.com>:

> Thanks for bringing this up!
>
> IIRC wpython was about using "fat" bytecodes, i.e. using 64bits per
> bytecode instead of 8.


No, it used 16, 32, and 48-bit per opcode (1, 2, or 3 16-bit words).


> That allows to minimize the number of bytecodes, thus having some
> performance increase.  TBH, I don't think it was "significantly faster".
>

Please, take a look at the benchmarks, or compile it and check yourself. ;-)

If I were to do some big refactoring of the ceval loop, I'd probably
> consider implementing a register VM.  While register VMs are a bit faster
> than stack VMs (up to 20-30%), they would also allow us to apply more
> optimizations, and even bolt on a simple JIT compiler.
>
> Yury


WPython was an hybrid-VM: it supported both a stack-based and a
register-based approach.

I think that it's needed, since the nature of Python, because you can have
operations with intermixed operands: constants, locals, globals, names.
It's quite difficult to handle all possible cases with a register-based VM.

Regards,
Cesare
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160515/96b81183/attachment.html>

From stephen at xemacs.org  Sun May 15 13:21:13 2016
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Mon, 16 May 2016 02:21:13 +0900
Subject: [Python-Dev] file system path protocol PEP
In-Reply-To: <5736793F.5030401@canterbury.ac.nz>
References: <CAP1=2W7AxEr8sxdDJ-qpVCBaVEnYsQRSe886P_DsBKp+ZxuGSQ@mail.gmail.com>
 <57343F56.6090609@mail.de> <5734A46C.40306@stoneleaf.us>
 <CAP7+vJKbunhHEgony6dXAQAHaOuiB_6yTm+nksgo4V_aai826Q@mail.gmail.com>
 <57358915.7040606@mail.de>
 <CAMiohoijCDKBuMCioY48UkU=WYy+DJqJyCL6mDBFx1gRXAv+0A@mail.gmail.com>
 <57359D69.6040108@mail.de>
 <CAMiohojq6DDLmWbDv1TiB26CVoUFNw5mNSaiPB+w2zeOgcqOUg@mail.gmail.com>
 <5735FB79.1010201@mail.de> <5736076F.6010905@stoneleaf.us>
 <CACac1F_dtX+K+kE=Z5AWGY8h3p2E=x4_fx0KoQ7eUhMuMddJVw@mail.gmail.com>
 <5736793F.5030401@canterbury.ac.nz>
Message-ID: <22328.45065.716113.338675@turnbull.sk.tsukuba.ac.jp>

Greg Ewing writes:
 > Paul Moore wrote:
 > > On 13 May 2016 at 17:57, Ethan Furman <ethan at stoneleaf.us> wrote:
 > > 
 > >>1) What is a wallet garden?
 > > 
 > > I assumed he meant "walled garden"
 > 
 > Works either way -- you'd want a wall around your wallet
 > garden to stop people stealing your wallets.

Actually, I expect it's a wall intended to keep your clients' wallets
in a garden where you can harvest them at your convenience.



From pludemann at google.com  Sun May 15 13:24:08 2016
From: pludemann at google.com (Peter Ludemann)
Date: Sun, 15 May 2016 10:24:08 -0700
Subject: [Python-Dev] Adding Type[C] to PEP 484
In-Reply-To: <CAP7+vJJ=_w7Grc_RiG18TQtKhDFYicqxULx4pnfX2UiX6MdPdQ@mail.gmail.com>
References: <CAP7+vJL+mpL=N31idm0SGWxAeChJ3eDWiedbYL0m93Ucs3PR6g@mail.gmail.com>
 <CAP1=2W4+gS3Cz7pUkSDcQEF49zJLtc4Zak8YX2iSrPT9U-OFAQ@mail.gmail.com>
 <CACsRUKLyWjCFWP6UtNZf7+NsJ0JdHSVFMiqk2ZmS7d7qj7XmSw@mail.gmail.com>
 <CAP7+vJLYPR1c+ZkB3Z9U57X8vktVn7czA6p4Xi6cd_CjnBDofg@mail.gmail.com>
 <CACsRUKLNXnsUCNeC4g0j2yneqSNhNf4NVsyhRJgTegVg3srmZg@mail.gmail.com>
 <CAP7+vJJ=_w7Grc_RiG18TQtKhDFYicqxULx4pnfX2UiX6MdPdQ@mail.gmail.com>
Message-ID: <CACsRUKJRP0g_g3jtFEmTJmX2V-f74aH0mtdTByw4V-4odUmWrA@mail.gmail.com>

On 14 May 2016 at 13:28, Guido van Rossum <gvanrossum at gmail.com> wrote:

> On Sat, May 14, 2016 at 11:47 AM, Peter Ludemann <pludemann at google.com>
> wrote:
>
>> I think that Type[C,D] would mean Union[Type[C], Type[D]], but I'm not
>> sure ... I should probably talk to a typing expert about this.
>> (Sorry for thinking out loud; but if we decide that Type[C,D] doesn't
>> make sense, we need to prohibit it)
>>
>
> It sounds iffy enough to prohibit.
>
>
>> I suppose I'm asking: do we allow new_user(Type[BasicUser, ProUser])?
>>
>
> That looks like a call, but Type[] is something to use in the definition.
>
>
>> Here's a trivial example that off the top of my head makes sense:
>>
>> BasicOrProUser = TypeVar('BasicOrProUser', BasicUser, ProUser)
>> def new_user(name: str, user_factory: Type[BasicOrProuser]) ->
>> BasicOrProUser:
>>     return user_factory(name)
>>
>
> That looks reasonable and only has one parameter to Type[].
>

So, should we allow this (by analogy to TypeVar allowing multiple types):

?  ?
def new_user(name: str, user_factory: Type[BasicUser, ProUser]) ->
Union[BasicUser, ProUser]:

?  ?
  return user_factory(name)

?
?... ?
?or should it be:

?
def new_user(name: str,
?              ?
user_factory: Type[
?
Union[
BasicUser, ProUser
?]?
]) -> Union[BasicUser, ProUser]:
      return user_factory(name)
?
?

Although both of these lose the information that the output type is related
to the input type; and hopefully the verbosity would encourage people to
use TypeVar.

(From these examples, my preference is for Type to be limited to a single
choice and anything more complicated should be done with TypeVar, Union,
etc.)



>
>
>> Or must the Type[...] item have been defined with a TypeVar(...,
>> bound=...), in which case multiple types aren't allowed with Type[...]?
>>
>
> You can use whatever you want. If you want to write Type[int] go right
> ahead. Even Type[Union[int, str]] should be allowed. There is *usually* a
> good reason to use a type variable though -- without one the return type
> would be the base class. (But then again maybe that could be what you want,
> or you might not even have a return type, e.g. when the thing just gets
> thrown into the database.)
>
> --Guido
>
>
>>
>>
>> On 14 May 2016 at 11:30, Guido van Rossum <gvanrossum at gmail.com> wrote:
>>
>>> What would Type[C, D] mean?
>>>
>>> --Guido (mobile)
>>> On May 14, 2016 11:21 AM, "Peter Ludemann via Python-Dev" <
>>> python-dev at python.org> wrote:
>>>
>>>> Is Type[C,D] allowed? Or should multiple types be restricted to
>>>> TypeVar?
>>>>
>>>> _______________________________________________
>>>> Python-Dev mailing list
>>>> Python-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/python-dev
>>>> Unsubscribe:
>>>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>>>
>>>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160515/020e532f/attachment.html>

From gvanrossum at gmail.com  Sun May 15 13:30:05 2016
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Sun, 15 May 2016 10:30:05 -0700
Subject: [Python-Dev] Adding Type[C] to PEP 484
In-Reply-To: <CACsRUKJRP0g_g3jtFEmTJmX2V-f74aH0mtdTByw4V-4odUmWrA@mail.gmail.com>
References: <CAP7+vJL+mpL=N31idm0SGWxAeChJ3eDWiedbYL0m93Ucs3PR6g@mail.gmail.com>
 <CAP1=2W4+gS3Cz7pUkSDcQEF49zJLtc4Zak8YX2iSrPT9U-OFAQ@mail.gmail.com>
 <CACsRUKLyWjCFWP6UtNZf7+NsJ0JdHSVFMiqk2ZmS7d7qj7XmSw@mail.gmail.com>
 <CAP7+vJLYPR1c+ZkB3Z9U57X8vktVn7czA6p4Xi6cd_CjnBDofg@mail.gmail.com>
 <CACsRUKLNXnsUCNeC4g0j2yneqSNhNf4NVsyhRJgTegVg3srmZg@mail.gmail.com>
 <CAP7+vJJ=_w7Grc_RiG18TQtKhDFYicqxULx4pnfX2UiX6MdPdQ@mail.gmail.com>
 <CACsRUKJRP0g_g3jtFEmTJmX2V-f74aH0mtdTByw4V-4odUmWrA@mail.gmail.com>
Message-ID: <CAP7+vJJVnEHYLfFfmRFxYRQbcZE4RLSwT6-JMLdtfkRyXDyNeA@mail.gmail.com>

Right. I still have no good intuition for what Type[BasicUser, ProUser]
would mean so I think you should be required to use the Union, which is
clear. Type[] should only allow one parameter.

Most real uses of Type[] use a type variable anyway.

I really recommend that you read through
https://github.com/python/typing/issues/107 even though it's long and
meanders a bit; it comes to the right conclusion in the end.

On Sun, May 15, 2016 at 10:24 AM, Peter Ludemann <pludemann at google.com>
wrote:

>
>
> On 14 May 2016 at 13:28, Guido van Rossum <gvanrossum at gmail.com> wrote:
>
>> On Sat, May 14, 2016 at 11:47 AM, Peter Ludemann <pludemann at google.com>
>> wrote:
>>
>>> I think that Type[C,D] would mean Union[Type[C], Type[D]], but I'm not
>>> sure ... I should probably talk to a typing expert about this.
>>> (Sorry for thinking out loud; but if we decide that Type[C,D] doesn't
>>> make sense, we need to prohibit it)
>>>
>>
>> It sounds iffy enough to prohibit.
>>
>>
>>> I suppose I'm asking: do we allow new_user(Type[BasicUser, ProUser])?
>>>
>>
>> That looks like a call, but Type[] is something to use in the definition.
>>
>>
>>> Here's a trivial example that off the top of my head makes sense:
>>>
>>> BasicOrProUser = TypeVar('BasicOrProUser', BasicUser, ProUser)
>>> def new_user(name: str, user_factory: Type[BasicOrProuser]) ->
>>> BasicOrProUser:
>>>     return user_factory(name)
>>>
>>
>> That looks reasonable and only has one parameter to Type[].
>>
>
> So, should we allow this (by analogy to TypeVar allowing multiple types):
>
> ?  ?
> def new_user(name: str, user_factory: Type[BasicUser, ProUser]) ->
> Union[BasicUser, ProUser]:
>
> ?  ?
>   return user_factory(name)
>
> ?
> ?... ?
> ?or should it be:
>
> ?
> def new_user(name: str,
> ?              ?
> user_factory: Type[
> ?
> Union[
> BasicUser, ProUser
> ?]?
> ]) -> Union[BasicUser, ProUser]:
>       return user_factory(name)
> ?
> ?
>
> Although both of these lose the information that the output type is
> related to the input type; and hopefully the verbosity would encourage
> people to use TypeVar.
>
> (From these examples, my preference is for Type to be limited to a single
> choice and anything more complicated should be done with TypeVar, Union,
> etc.)
>
>
>
>>
>>
>>> Or must the Type[...] item have been defined with a TypeVar(...,
>>> bound=...), in which case multiple types aren't allowed with Type[...]?
>>>
>>
>> You can use whatever you want. If you want to write Type[int] go right
>> ahead. Even Type[Union[int, str]] should be allowed. There is *usually* a
>> good reason to use a type variable though -- without one the return type
>> would be the base class. (But then again maybe that could be what you want,
>> or you might not even have a return type, e.g. when the thing just gets
>> thrown into the database.)
>>
>> --Guido
>>
>>
>>>
>>>
>>> On 14 May 2016 at 11:30, Guido van Rossum <gvanrossum at gmail.com> wrote:
>>>
>>>> What would Type[C, D] mean?
>>>>
>>>> --Guido (mobile)
>>>> On May 14, 2016 11:21 AM, "Peter Ludemann via Python-Dev" <
>>>> python-dev at python.org> wrote:
>>>>
>>>>> Is Type[C,D] allowed? Or should multiple types be restricted to
>>>>> TypeVar?
>>>>>
>>>>> _______________________________________________
>>>>> Python-Dev mailing list
>>>>> Python-Dev at python.org
>>>>> https://mail.python.org/mailman/listinfo/python-dev
>>>>> Unsubscribe:
>>>>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>>>>
>>>>>
>>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160515/9c8c75ea/attachment.html>

From nas at arctrix.com  Mon May 16 00:48:50 2016
From: nas at arctrix.com (Neil Schemenauer)
Date: Mon, 16 May 2016 04:48:50 +0000 (UTC)
Subject: [Python-Dev] PyGC_Collect ignores state of `enabled`
References: <DDA40D7F-F070-42A0-AFFF-B03828894A68@langa.pl>
Message-ID: <nhbjfi$u93$1@ger.gmane.org>

Hi,

I intended for gc.collect() to actually collect cycles even if the
auto-GC was disabled.  Having Py_Finalize() run GC even when it has
been disabled seems wrong to me.  Originally, cyclic GC was supposed
to be optional.  Back then, most programs did not leak cycles.  I
would vote for Py_Finalize() checking the 'enabled' flag and not
calling PyGC_Collect if false.

Regards,

  Neil


From meadori at gmail.com  Mon May 16 11:55:18 2016
From: meadori at gmail.com (Meador Inge)
Date: Mon, 16 May 2016 10:55:18 -0500
Subject: [Python-Dev] Speeding up CPython 5-10%
In-Reply-To: <CAP7v7k7eHEKFqbVX2-7RudS0u4oqhrVTuC25FE_dJo8PXQaimA@mail.gmail.com>
References: <56A90B97.7090001@gmail.com>
 <CAMpsgwZXxKDzUAQ+_hApdHPQEwnEQLUyQz764VvMW5EXywd=NA@mail.gmail.com>
 <CAP7v7k7eHEKFqbVX2-7RudS0u4oqhrVTuC25FE_dJo8PXQaimA@mail.gmail.com>
Message-ID: <CAK1QoopawhOE7dUkUYjHj=SSsE74NhgkH6atM4yYJSRBmHbszA@mail.gmail.com>

On Sun, May 15, 2016 at 2:23 AM, Cesare Di Mauro <cesare.di.mauro at gmail.com>
wrote:


> Just one thing that comes to my mind: is the stack depth calculation
> routine changed? It was suboptimal, and calculating a better number
> decreases stack allocation, and increases the frame usage.
>

This is still a problem and came up again recently:

http://bugs.python.org/issue26549

-- Meador
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160516/c8ff0b60/attachment.html>

From brett at python.org  Mon May 16 16:00:58 2016
From: brett at python.org (Brett Cannon)
Date: Mon, 16 May 2016 20:00:58 +0000
Subject: [Python-Dev] File system path PEP, 3rd draft
In-Reply-To: <CAP1=2W6W+WO3-NjiSdCnqKVW1PBRzeEiW8gZGw1GSk1PLsufxA@mail.gmail.com>
References: <CAP1=2W6W+WO3-NjiSdCnqKVW1PBRzeEiW8gZGw1GSk1PLsufxA@mail.gmail.com>
Message-ID: <CAP1=2W5nKxqjBUeAqK8GK9yKrLXrzDEXVz9eUPQF07Gh2yui4w@mail.gmail.com>

Recent discussions have been about type hints which are orthogonal to the
PEP, so things have seemed to have reached a steady state.

Was there anything else that needed clarification, Guido, or are you ready
to pronounce? Or did you want to wait until the language summit? Or did you
want to assign a BDFL delegate?

On Fri, 13 May 2016 at 11:37 Brett Cannon <brett at python.org> wrote:

> Biggest changes since the second draft:
>
>    1. Resolve __fspath__() from the type, not the instance (for Guido)
>    2. Updated the TypeError messages to say "os.PathLike object" instead
>    of "path object" (implicitly for Steven)
>    3. TODO item to define "path-like" in the glossary (for Steven)
>    4. Various more things added to Rejected Ideas
>    5. Added Koos as a co-author (for Koos :)
>
> ----------
> PEP: NNN
> Title: Adding a file system path protocol
> Version: $Revision$
> Last-Modified: $Date$
> Author: Brett Cannon <brett at python.org>,
>         Koos Zevenhoven <k7hoven at gmail.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 11-May-2016
> Post-History: 11-May-2016,
>               12-May-2016,
>               13-May-2016
>
>
> Abstract
> ========
>
> This PEP proposes a protocol for classes which represent a file system
> path to be able to provide a ``str`` or ``bytes`` representation.
> Changes to Python's standard library are also proposed to utilize this
> protocol where appropriate to facilitate the use of path objects where
> historically only ``str`` and/or ``bytes`` file system paths are
> accepted. The goal is to facilitate the migration of users towards
> rich path objects while providing an easy way to work with code
> expecting ``str`` or ``bytes``.
>
>
> Rationale
> =========
>
> Historically in Python, file system paths have been represented as
> strings or bytes. This choice of representation has stemmed from C's
> own decision to represent file system paths as
> ``const char *`` [#libc-open]_. While that is a totally serviceable
> format to use for file system paths, it's not necessarily optimal. At
> issue is the fact that while all file system paths can be represented
> as strings or bytes, not all strings or bytes represent a file system
> path. This can lead to issues where any e.g. string duck-types to a
> file system path whether it actually represents a path or not.
>
> To help elevate the representation of file system paths from their
> representation as strings and bytes to a richer object representation,
> the pathlib module [#pathlib]_ was provisionally introduced in
> Python 3.4 through PEP 428. While considered by some as an improvement
> over strings and bytes for file system paths, it has suffered from a
> lack of adoption. Typically the key issue listed for the low adoption
> rate has been the lack of support in the standard library. This lack
> of support required users of pathlib to manually convert path objects
> to strings by calling ``str(path)`` which many found error-prone.
>
> One issue in converting path objects to strings comes from
> the fact that the only generic way to get a string representation of
> the path was to pass the object to ``str()``. This can pose a
> problem when done blindly as nearly all Python objects have some
> string representation whether they are a path or not, e.g.
> ``str(None)`` will give a result that
> ``builtins.open()`` [#builtins-open]_ will happily use to create a new
> file.
>
> Exacerbating this whole situation is the
> ``DirEntry`` object [#os-direntry]_. While path objects have a
> representation that can be extracted using ``str()``, ``DirEntry``
> objects expose a ``path`` attribute instead. Having no common
> interface between path objects, ``DirEntry``, and any other
> third-party path library has become an issue. A solution that allows
> any path-representing object to declare that it is a path and a way
> to extract a low-level representation that all path objects could
> support is desired.
>
> This PEP then proposes to introduce a new protocol to be followed by
> objects which represent file system paths. Providing a protocol allows
> for explicit signaling of what objects represent file system paths as
> well as a way to extract a lower-level representation that can be used
> with older APIs which only support strings or bytes.
>
> Discussions regarding path objects that led to this PEP can be found
> in multiple threads on the python-ideas mailing list archive
> [#python-ideas-archive]_ for the months of March and April 2016 and on
> the python-dev mailing list archives [#python-dev-archive]_ during
> April 2016.
>
>
> Proposal
> ========
>
> This proposal is split into two parts. One part is the proposal of a
> protocol for objects to declare and provide support for exposing a
> file system path representation. The other part deals with changes to
> Python's standard library to support the new protocol. These changes
> will also lead to the pathlib module dropping its provisional status.
>
> Protocol
> --------
>
> The following abstract base class defines the protocol for an object
> to be considered a path object::
>
>     import abc
>     import typing as t
>
>
>     class PathLike(abc.ABC):
>
>         """Abstract base class for implementing the file system path
> protocol."""
>
>         @abc.abstractmethod
>         def __fspath__(self) -> t.Union[str, bytes]:
>             """Return the file system path representation of the object."""
>             raise NotImplementedError
>
>
> Objects representing file system paths will implement the
> ``__fspath__()`` method which will return the ``str`` or ``bytes``
> representation of the path. The ``str`` representation is the
> preferred low-level path representation as it is human-readable and
> what people historically represent paths as.
>
>
> Standard library changes
> ------------------------
>
> It is expected that most APIs in Python's standard library that
> currently accept a file system path will be updated appropriately to
> accept path objects (whether that requires code or simply an update
> to documentation will vary). The modules mentioned below, though,
> deserve specific details as they have either fundamental changes that
> empower the ability to use path objects, or entail additions/removal
> of APIs.
>
>
> builtins
> ''''''''
>
> ``open()`` [#builtins-open]_ will be updated to accept path objects as
> well as continue to accept ``str`` and ``bytes``.
>
>
> os
> '''
>
> The ``fspath()`` function will be added with the following semantics::
>
>     import typing as t
>
>
>     def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str, bytes]:
>         """Return the string representation of the path.
>
>         If str or bytes is passed in, it is returned unchanged.
>         """
>         if isinstance(path, (str, bytes)):
>             return path
>
>         # Work from the object's type to match method resolution of other
> magic
>         # methods.
>         path_type = type(path)
>         try:
>             return path_type.__fspath__(path)
>         except AttributeError:
>             if hasattr(path_type, '__fspath__'):
>                 raise
>
>             raise TypeError("expected str, bytes or os.PathLike object,
> not "
>                             + path_type.__name__)
>
> The ``os.fsencode()`` [#os-fsencode]_ and
> ``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept
> path objects. As both functions coerce their arguments to
> ``bytes`` and ``str``, respectively, they will be updated to call
> ``__fspath__()`` if present to convert the path object to a ``str`` or
> ``bytes`` representation, and then perform their appropriate
> coercion operations as if the return value from ``__fspath__()`` had
> been the original argument to the coercion function in question.
>
> The addition of ``os.fspath()``, the updates to
> ``os.fsencode()``/``os.fsdecode()``, and the current semantics of
> ``pathlib.PurePath`` provide the semantics necessary to
> get the path representation one prefers. For a path object,
> ``pathlib.PurePath``/``Path`` can be used. To obtain the ``str`` or
> ``bytes`` representation without any coersion, then ``os.fspath()``
> can be used. If a ``str`` is desired and the encoding of ``bytes``
> should be assumed to be the default file system encoding, then
> ``os.fsdecode()`` should be used. If a ``bytes`` representation is
> desired and any strings should be encoded using the default file
> system encoding, then ``os.fsencode()`` is used. This PEP recommends
> using path objects when possible and falling back to string paths as
> necessary and using ``bytes`` as a last resort.
>
> Another way to view this is as a hierarchy of file system path
> representations (highest- to lowest-level): path ? str ? bytes. The
> functions and classes under discussion can all accept objects on the
> same level of the hierarchy, but they vary in whether they promote or
> demote objects to another level. The ``pathlib.PurePath`` class can
> promote a ``str`` to a path object. The ``os.fspath()`` function can
> demote a path object to a ``str`` or ``bytes`` instance, depending
> on what ``__fspath__()`` returns.
> The ``os.fsdecode()`` function will demote a path object to
> a string or promote a ``bytes`` object to a ``str``. The
> ``os.fsencode()`` function will demote a path or string object to
> ``bytes``. There is no function that provides a way to demote a path
> object directly to ``bytes`` while bypassing string demotion.
>
> The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()``
> method. It will return the same value as currently found on the
> ``path`` attribute of ``DirEntry`` instances.
>
> The Protocol_ ABC will be added to the ``os`` module under the name
> ``os.PathLike``.
>
>
> os.path
> '''''''
>
> The various path-manipulation functions of ``os.path`` [#os-path]_
> will be updated to accept path objects. For polymorphic functions that
> accept both bytes and strings, they will be updated to simply use
> ``os.fspath()``.
>
> During the discussions leading up to this PEP it was suggested that
> ``os.path`` not be updated using an "explicit is better than implicit"
> argument. The thinking was that since ``__fspath__()`` is polymorphic
> itself it may be better to have code working with ``os.path`` extract
> the path representation from path objects explicitly. There is also
> the consideration that adding support this deep into the low-level OS
> APIs will lead to code magically supporting path objects without
> requiring any documentation updated, leading to potential complaints
> when it doesn't work, unbeknownst to the project author.
>
> But it is the view of this PEP that "practicality beats purity" in
> this instance. To help facilitate the transition to supporting path
> objects, it is better to make the transition as easy as possible than
> to worry about unexpected/undocumented duck typing support for
> path objects by projects.
>
> There has also been the suggestion that ``os.path`` functions could be
> used in a tight loop and the overhead of checking or calling
> ``__fspath__()`` would be too costly. In this scenario only
> path-consuming APIs would be directly updated and path-manipulating
> APIs like the ones in ``os.path`` would go unmodified. This would
> require library authors to update their code to support path objects
> if they performed any path manipulations, but if the library code
> passed the path straight through then the library wouldn't need to be
> updated. It is the view of this PEP and Guido, though, that this is an
> unnecessary worry and that performance will still be acceptable.
>
>
> pathlib
> '''''''
>
> The constructor for ``pathlib.PurePath`` and ``pathlib.Path`` will be
> updated to accept ``PathLike`` objects. Both ``PurePath`` and ``Path``
> will continue to not accept ``bytes`` path representations, and so if
> ``__fspath__()`` returns ``bytes`` it will raise an exception.
>
> The ``path`` attribute will be removed as this PEP makes it
> redundant (it has not been included in any released version of Python
> and so is not a backwards-compatibility concern).
>
>
> C API
> '''''
>
> The C API will gain an equivalent function to ``os.fspath()``::
>
>     /*
>         Return the file system path of the object.
>
>         If the object is str or bytes, then allow it to pass through with
>         an incremented refcount. If the object defines __fspath__(), then
>         return the result of that method. All other types raise a
> TypeError.
>     */
>     PyObject *
>     PyOS_FSPath(PyObject *path)
>     {
>         if (PyUnicode_Check(path) || PyBytes_Check(path)) {
>             Py_INCREF(path);
>             return path;
>         }
>
>         if (PyObject_HasAttrString(path->ob_type, "__fspath__")) {
>             return PyObject_CallMethodObjArgs(path->ob_type, "__fspath__",
> path,
>                                             NULL);
>         }
>
>         return PyErr_Format(PyExc_TypeError,
>                             "expected a str, bytes, or os.PathLike object,
> not %S",
>                             path->ob_type);
>     }
>
>
>
> Backwards compatibility
> =======================
>
> There are no explicit backwards-compatibility concerns. Unless an
> object incidentally already defines a ``__fspath__()`` method there is
> no reason to expect the pre-existing code to break or expect to have
> its semantics implicitly changed.
>
> Libraries wishing to support path objects and a version of Python
> prior to Python 3.6 and the existence of ``os.fspath()`` can use the
> idiom of
> ``path.__fspath__() if hasattr(path, "__fspath__") else path``.
>
>
> Implementation
> ==============
>
> This is the task list for what this PEP proposes:
>
> #. Remove the ``path`` attribute from pathlib
> #. Remove the provisional status of pathlib
> #. Add ``os.PathLike``
> #. Add ``os.fspath()``
> #. Add ``PyOS_FSPath()``
> #. Update ``os.fsencode()``
> #. Update ``os.fsdecode()``
> #. Update ``pathlib.PurePath`` and ``pathlib.Path``
> #. Update ``builtins.open()``
> #. Update ``os.DirEntry``
> #. Update ``os.path``
> #. Add a glossary entry for "path-like"
>
>
> Rejected Ideas
> ==============
>
> Other names for the protocol's method
> -------------------------------------
>
> Various names were proposed during discussions leading to this PEP,
> including ``__path__``, ``__pathname__``, and ``__fspathname__``. In
> the end people seemed to gravitate towards ``__fspath__`` for being
> unambiguous without being unnecessarily long.
>
>
> Separate str/bytes methods
> --------------------------
>
> At one point it was suggested that ``__fspath__()`` only return
> strings and another method named ``__fspathb__()`` be introduced to
> return bytes. The thinking is that by making ``__fspath__()`` not be
> polymorphic it could make dealing with the potential string or bytes
> representations easier. But the general consensus was that returning
> bytes will more than likely be rare and that the various functions in
> the os module are the better abstraction to promote over direct
> calls to ``__fspath__()``.
>
>
> Providing a ``path`` attribute
> ------------------------------
>
> To help deal with the issue of ``pathlib.PurePath`` not inheriting
> from ``str``, originally it was proposed to introduce a ``path``
> attribute to mirror what ``os.DirEntry`` provides. In the end,
> though, it was determined that a protocol would provide the same
> result while not directly exposing an API that most people will never
> need to interact with directly.
>
>
> Have ``__fspath__()`` only return strings
> ------------------------------------------
>
> Much of the discussion that led to this PEP revolved around whether
> ``__fspath__()`` should be polymorphic and return ``bytes`` as well as
> ``str`` or only return ``str``. The general sentiment for this view
> was that ``bytes`` are difficult to work with due to their
> inherent lack of information about their encoding and PEP 383 makes
> it possible to represent all file system paths using ``str`` with the
> ``surrogateescape`` handler. Thus, it would be better to forcibly
> promote the use of ``str`` as the low-level path representation for
> high-level path objects.
>
> In the end, it was decided that using ``bytes`` to represent paths is
> simply not going to go away and thus they should be supported to some
> degree. The hope is that people will gravitate towards path objects
> like pathlib and that will move people away from operating directly
> with ``bytes``.
>
>
> A generic string encoding mechanism
> -----------------------------------
>
> At one point there was a discussion of developing a generic mechanism
> to extract a string representation of an object that had semantic
> meaning (``__str__()`` does not necessarily return anything of
> semantic significance beyond what may be helpful for debugging). In
> the end, it was deemed to lack a motivating need beyond the one this
> PEP is trying to solve in a specific fashion.
>
>
> Have __fspath__ be an attribute
> -------------------------------
>
> It was briefly considered to have ``__fspath__`` be an attribute
> instead of a method. This was rejected for two reasons. One,
> historically protocols have been implemented as "magic methods" and
> not "magic methods and attributes". Two, there is no guarantee that
> the lower-level representation of a path object will be pre-computed,
> potentially misleading users that there was no expensive computation
> behind the scenes in case the attribute was implemented as a property.
>
> This also indirectly ties into the idea of introducing a ``path``
> attribute to accomplish the same thing. This idea has an added issue,
> though, of accidentally having any object with a ``path`` attribute
> meet the protocol's duck typing. Introducing a new magic method for
> the protocol helpfully avoids any accidental opting into the protocol.
>
>
> Provide specific type hinting support
> -------------------------------------
>
> There was some consideration to provdinga generic ``typing.PathLike``
> class which would allow for e.g. ``typing.PathLike[str]`` to specify
> a type hint for a path object which returned a string representation.
> While potentially beneficial, the usefulness was deemed too small to
> bother adding the type hint class.
>
> This also removed any desire to have a class in the ``typing`` module
> which represented the union of all acceptable path-representing types
> as that can be represented with
> ``typing.Union[str, bytes, os.PathLike]`` easily enough and the hope
> is users will slowly gravitate to path objects only.
>
>
> Provide ``os.fspathb()``
> ------------------------
>
> It was suggested that to mirror the structure of e.g.
> ``os.getcwd()``/``os.getcwdb()``, that ``os.fspath()`` only return
> ``str`` and that another function named ``os.fspathb()`` be
> introduced that only returned ``bytes``. This was rejected as the
> purposes of the ``*b()`` functions are tied to querying the file
> system where there is a need to get the raw bytes back. As this PEP
> does not work directly with data on a file system (but which *may*
> be), the view was taken this distinction is unnecessary. It's also
> believed that the need for only bytes will not be common enough to
> need to support in such a specific manner as ``os.fsencode()`` will
> provide similar functionality.
>
>
> Call ``__fspath__()`` off of the instance
> -----------------------------------------
>
> An earlier draft of this PEP had ``os.fspath()`` calling
> ``path.__fspath__()`` instead of ``type(path).__fspath__(path)``. The
> changed to be consistent with how other magic methods in Python are
> resolved.
>
>
> Acknowledgements
> ================
>
> Thanks to everyone who participated in the various discussions related
> to this PEP that spanned both python-ideas and python-dev. Special
> thanks to Stephen Turnbull for direct feedback on early drafts of this
> PEP. More special thanks to Koos Zevenhoven and Ethan Furman for not
> only feedback on early drafts of this PEP but also helping to drive
> the overall discussion on this topic across the two mailing lists.
>
>
> References
> ==========
>
> .. [#python-ideas-archive] The python-ideas mailing list archive
>    (https://mail.python.org/pipermail/python-ideas/)
>
> .. [#python-dev-archive] The python-dev mailing list archive
>    (https://mail.python.org/pipermail/python-dev/)
>
> .. [#libc-open] ``open()`` documention for the C standard library
>    (
> http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html
> )
>
> .. [#pathlib] The ``pathlib`` module
>    (https://docs.python.org/3/library/pathlib.html#module-pathlib)
>
> .. [#builtins-open] The ``builtins.open()`` function
>    (https://docs.python.org/3/library/functions.html#open)
>
> .. [#os-fsencode] The ``os.fsencode()`` function
>    (https://docs.python.org/3/library/os.html#os.fsencode)
>
> .. [#os-fsdecode] The ``os.fsdecode()`` function
>    (https://docs.python.org/3/library/os.html#os.fsdecode)
>
> .. [#os-direntry] The ``os.DirEntry`` class
>    (https://docs.python.org/3/library/os.html#os.DirEntry)
>
> .. [#os-path] The ``os.path`` module
>    (https://docs.python.org/3/library/os.path.html#module-os.path)
>
>
> Copyright
> =========
>
> This document has been placed in the public domain.
>
>
> ..
>    Local Variables:
>    mode: indented-text
>    indent-tabs-mode: nil
>    sentence-end-double-space: t
>    fill-column: 70
>    coding: utf-8
>    End:
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160516/98e1e93f/attachment.html>

From guido at python.org  Mon May 16 16:11:48 2016
From: guido at python.org (Guido van Rossum)
Date: Mon, 16 May 2016 13:11:48 -0700
Subject: [Python-Dev] File system path PEP, 3rd draft
In-Reply-To: <CAP1=2W5nKxqjBUeAqK8GK9yKrLXrzDEXVz9eUPQF07Gh2yui4w@mail.gmail.com>
References: <CAP1=2W6W+WO3-NjiSdCnqKVW1PBRzeEiW8gZGw1GSk1PLsufxA@mail.gmail.com>
 <CAP1=2W5nKxqjBUeAqK8GK9yKrLXrzDEXVz9eUPQF07Gh2yui4w@mail.gmail.com>
Message-ID: <CAP7+vJJ-m0f-GEknLwH2WOcKvQtXba8aVb7o3m0AWBts9e07Pg@mail.gmail.com>

Once you assign yourself a PEP number I'll do one more pass and then I
expect to accept it -- the draft looks good to me!

On Mon, May 16, 2016 at 1:00 PM, Brett Cannon <brett at python.org> wrote:

> Recent discussions have been about type hints which are orthogonal to the
> PEP, so things have seemed to have reached a steady state.
>
> Was there anything else that needed clarification, Guido, or are you ready
> to pronounce? Or did you want to wait until the language summit? Or did you
> want to assign a BDFL delegate?
>
>
> On Fri, 13 May 2016 at 11:37 Brett Cannon <brett at python.org> wrote:
>
>> Biggest changes since the second draft:
>>
>>    1. Resolve __fspath__() from the type, not the instance (for Guido)
>>    2. Updated the TypeError messages to say "os.PathLike object" instead
>>    of "path object" (implicitly for Steven)
>>    3. TODO item to define "path-like" in the glossary (for Steven)
>>    4. Various more things added to Rejected Ideas
>>    5. Added Koos as a co-author (for Koos :)
>>
>> ----------
>> PEP: NNN
>> Title: Adding a file system path protocol
>> Version: $Revision$
>> Last-Modified: $Date$
>> Author: Brett Cannon <brett at python.org>,
>>         Koos Zevenhoven <k7hoven at gmail.com>
>> Status: Draft
>> Type: Standards Track
>> Content-Type: text/x-rst
>> Created: 11-May-2016
>> Post-History: 11-May-2016,
>>               12-May-2016,
>>               13-May-2016
>>
>>
>> Abstract
>> ========
>>
>> This PEP proposes a protocol for classes which represent a file system
>> path to be able to provide a ``str`` or ``bytes`` representation.
>> Changes to Python's standard library are also proposed to utilize this
>> protocol where appropriate to facilitate the use of path objects where
>> historically only ``str`` and/or ``bytes`` file system paths are
>> accepted. The goal is to facilitate the migration of users towards
>> rich path objects while providing an easy way to work with code
>> expecting ``str`` or ``bytes``.
>>
>>
>> Rationale
>> =========
>>
>> Historically in Python, file system paths have been represented as
>> strings or bytes. This choice of representation has stemmed from C's
>> own decision to represent file system paths as
>> ``const char *`` [#libc-open]_. While that is a totally serviceable
>> format to use for file system paths, it's not necessarily optimal. At
>> issue is the fact that while all file system paths can be represented
>> as strings or bytes, not all strings or bytes represent a file system
>> path. This can lead to issues where any e.g. string duck-types to a
>> file system path whether it actually represents a path or not.
>>
>> To help elevate the representation of file system paths from their
>> representation as strings and bytes to a richer object representation,
>> the pathlib module [#pathlib]_ was provisionally introduced in
>> Python 3.4 through PEP 428. While considered by some as an improvement
>> over strings and bytes for file system paths, it has suffered from a
>> lack of adoption. Typically the key issue listed for the low adoption
>> rate has been the lack of support in the standard library. This lack
>> of support required users of pathlib to manually convert path objects
>> to strings by calling ``str(path)`` which many found error-prone.
>>
>> One issue in converting path objects to strings comes from
>> the fact that the only generic way to get a string representation of
>> the path was to pass the object to ``str()``. This can pose a
>> problem when done blindly as nearly all Python objects have some
>> string representation whether they are a path or not, e.g.
>> ``str(None)`` will give a result that
>> ``builtins.open()`` [#builtins-open]_ will happily use to create a new
>> file.
>>
>> Exacerbating this whole situation is the
>> ``DirEntry`` object [#os-direntry]_. While path objects have a
>> representation that can be extracted using ``str()``, ``DirEntry``
>> objects expose a ``path`` attribute instead. Having no common
>> interface between path objects, ``DirEntry``, and any other
>> third-party path library has become an issue. A solution that allows
>> any path-representing object to declare that it is a path and a way
>> to extract a low-level representation that all path objects could
>> support is desired.
>>
>> This PEP then proposes to introduce a new protocol to be followed by
>> objects which represent file system paths. Providing a protocol allows
>> for explicit signaling of what objects represent file system paths as
>> well as a way to extract a lower-level representation that can be used
>> with older APIs which only support strings or bytes.
>>
>> Discussions regarding path objects that led to this PEP can be found
>> in multiple threads on the python-ideas mailing list archive
>> [#python-ideas-archive]_ for the months of March and April 2016 and on
>> the python-dev mailing list archives [#python-dev-archive]_ during
>> April 2016.
>>
>>
>> Proposal
>> ========
>>
>> This proposal is split into two parts. One part is the proposal of a
>> protocol for objects to declare and provide support for exposing a
>> file system path representation. The other part deals with changes to
>> Python's standard library to support the new protocol. These changes
>> will also lead to the pathlib module dropping its provisional status.
>>
>> Protocol
>> --------
>>
>> The following abstract base class defines the protocol for an object
>> to be considered a path object::
>>
>>     import abc
>>     import typing as t
>>
>>
>>     class PathLike(abc.ABC):
>>
>>         """Abstract base class for implementing the file system path
>> protocol."""
>>
>>         @abc.abstractmethod
>>         def __fspath__(self) -> t.Union[str, bytes]:
>>             """Return the file system path representation of the
>> object."""
>>             raise NotImplementedError
>>
>>
>> Objects representing file system paths will implement the
>> ``__fspath__()`` method which will return the ``str`` or ``bytes``
>> representation of the path. The ``str`` representation is the
>> preferred low-level path representation as it is human-readable and
>> what people historically represent paths as.
>>
>>
>> Standard library changes
>> ------------------------
>>
>> It is expected that most APIs in Python's standard library that
>> currently accept a file system path will be updated appropriately to
>> accept path objects (whether that requires code or simply an update
>> to documentation will vary). The modules mentioned below, though,
>> deserve specific details as they have either fundamental changes that
>> empower the ability to use path objects, or entail additions/removal
>> of APIs.
>>
>>
>> builtins
>> ''''''''
>>
>> ``open()`` [#builtins-open]_ will be updated to accept path objects as
>> well as continue to accept ``str`` and ``bytes``.
>>
>>
>> os
>> '''
>>
>> The ``fspath()`` function will be added with the following semantics::
>>
>>     import typing as t
>>
>>
>>     def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str,
>> bytes]:
>>         """Return the string representation of the path.
>>
>>         If str or bytes is passed in, it is returned unchanged.
>>         """
>>         if isinstance(path, (str, bytes)):
>>             return path
>>
>>         # Work from the object's type to match method resolution of other
>> magic
>>         # methods.
>>         path_type = type(path)
>>         try:
>>             return path_type.__fspath__(path)
>>         except AttributeError:
>>             if hasattr(path_type, '__fspath__'):
>>                 raise
>>
>>             raise TypeError("expected str, bytes or os.PathLike object,
>> not "
>>                             + path_type.__name__)
>>
>> The ``os.fsencode()`` [#os-fsencode]_ and
>> ``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept
>> path objects. As both functions coerce their arguments to
>> ``bytes`` and ``str``, respectively, they will be updated to call
>> ``__fspath__()`` if present to convert the path object to a ``str`` or
>> ``bytes`` representation, and then perform their appropriate
>> coercion operations as if the return value from ``__fspath__()`` had
>> been the original argument to the coercion function in question.
>>
>> The addition of ``os.fspath()``, the updates to
>> ``os.fsencode()``/``os.fsdecode()``, and the current semantics of
>> ``pathlib.PurePath`` provide the semantics necessary to
>> get the path representation one prefers. For a path object,
>> ``pathlib.PurePath``/``Path`` can be used. To obtain the ``str`` or
>> ``bytes`` representation without any coersion, then ``os.fspath()``
>> can be used. If a ``str`` is desired and the encoding of ``bytes``
>> should be assumed to be the default file system encoding, then
>> ``os.fsdecode()`` should be used. If a ``bytes`` representation is
>> desired and any strings should be encoded using the default file
>> system encoding, then ``os.fsencode()`` is used. This PEP recommends
>> using path objects when possible and falling back to string paths as
>> necessary and using ``bytes`` as a last resort.
>>
>> Another way to view this is as a hierarchy of file system path
>> representations (highest- to lowest-level): path ? str ? bytes. The
>> functions and classes under discussion can all accept objects on the
>> same level of the hierarchy, but they vary in whether they promote or
>> demote objects to another level. The ``pathlib.PurePath`` class can
>> promote a ``str`` to a path object. The ``os.fspath()`` function can
>> demote a path object to a ``str`` or ``bytes`` instance, depending
>> on what ``__fspath__()`` returns.
>> The ``os.fsdecode()`` function will demote a path object to
>> a string or promote a ``bytes`` object to a ``str``. The
>> ``os.fsencode()`` function will demote a path or string object to
>> ``bytes``. There is no function that provides a way to demote a path
>> object directly to ``bytes`` while bypassing string demotion.
>>
>> The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()``
>> method. It will return the same value as currently found on the
>> ``path`` attribute of ``DirEntry`` instances.
>>
>> The Protocol_ ABC will be added to the ``os`` module under the name
>> ``os.PathLike``.
>>
>>
>> os.path
>> '''''''
>>
>> The various path-manipulation functions of ``os.path`` [#os-path]_
>> will be updated to accept path objects. For polymorphic functions that
>> accept both bytes and strings, they will be updated to simply use
>> ``os.fspath()``.
>>
>> During the discussions leading up to this PEP it was suggested that
>> ``os.path`` not be updated using an "explicit is better than implicit"
>> argument. The thinking was that since ``__fspath__()`` is polymorphic
>> itself it may be better to have code working with ``os.path`` extract
>> the path representation from path objects explicitly. There is also
>> the consideration that adding support this deep into the low-level OS
>> APIs will lead to code magically supporting path objects without
>> requiring any documentation updated, leading to potential complaints
>> when it doesn't work, unbeknownst to the project author.
>>
>> But it is the view of this PEP that "practicality beats purity" in
>> this instance. To help facilitate the transition to supporting path
>> objects, it is better to make the transition as easy as possible than
>> to worry about unexpected/undocumented duck typing support for
>> path objects by projects.
>>
>> There has also been the suggestion that ``os.path`` functions could be
>> used in a tight loop and the overhead of checking or calling
>> ``__fspath__()`` would be too costly. In this scenario only
>> path-consuming APIs would be directly updated and path-manipulating
>> APIs like the ones in ``os.path`` would go unmodified. This would
>> require library authors to update their code to support path objects
>> if they performed any path manipulations, but if the library code
>> passed the path straight through then the library wouldn't need to be
>> updated. It is the view of this PEP and Guido, though, that this is an
>> unnecessary worry and that performance will still be acceptable.
>>
>>
>> pathlib
>> '''''''
>>
>> The constructor for ``pathlib.PurePath`` and ``pathlib.Path`` will be
>> updated to accept ``PathLike`` objects. Both ``PurePath`` and ``Path``
>> will continue to not accept ``bytes`` path representations, and so if
>> ``__fspath__()`` returns ``bytes`` it will raise an exception.
>>
>> The ``path`` attribute will be removed as this PEP makes it
>> redundant (it has not been included in any released version of Python
>> and so is not a backwards-compatibility concern).
>>
>>
>> C API
>> '''''
>>
>> The C API will gain an equivalent function to ``os.fspath()``::
>>
>>     /*
>>         Return the file system path of the object.
>>
>>         If the object is str or bytes, then allow it to pass through with
>>         an incremented refcount. If the object defines __fspath__(), then
>>         return the result of that method. All other types raise a
>> TypeError.
>>     */
>>     PyObject *
>>     PyOS_FSPath(PyObject *path)
>>     {
>>         if (PyUnicode_Check(path) || PyBytes_Check(path)) {
>>             Py_INCREF(path);
>>             return path;
>>         }
>>
>>         if (PyObject_HasAttrString(path->ob_type, "__fspath__")) {
>>             return PyObject_CallMethodObjArgs(path->ob_type,
>> "__fspath__", path,
>>                                             NULL);
>>         }
>>
>>         return PyErr_Format(PyExc_TypeError,
>>                             "expected a str, bytes, or os.PathLike
>> object, not %S",
>>                             path->ob_type);
>>     }
>>
>>
>>
>> Backwards compatibility
>> =======================
>>
>> There are no explicit backwards-compatibility concerns. Unless an
>> object incidentally already defines a ``__fspath__()`` method there is
>> no reason to expect the pre-existing code to break or expect to have
>> its semantics implicitly changed.
>>
>> Libraries wishing to support path objects and a version of Python
>> prior to Python 3.6 and the existence of ``os.fspath()`` can use the
>> idiom of
>> ``path.__fspath__() if hasattr(path, "__fspath__") else path``.
>>
>>
>> Implementation
>> ==============
>>
>> This is the task list for what this PEP proposes:
>>
>> #. Remove the ``path`` attribute from pathlib
>> #. Remove the provisional status of pathlib
>> #. Add ``os.PathLike``
>> #. Add ``os.fspath()``
>> #. Add ``PyOS_FSPath()``
>> #. Update ``os.fsencode()``
>> #. Update ``os.fsdecode()``
>> #. Update ``pathlib.PurePath`` and ``pathlib.Path``
>> #. Update ``builtins.open()``
>> #. Update ``os.DirEntry``
>> #. Update ``os.path``
>> #. Add a glossary entry for "path-like"
>>
>>
>> Rejected Ideas
>> ==============
>>
>> Other names for the protocol's method
>> -------------------------------------
>>
>> Various names were proposed during discussions leading to this PEP,
>> including ``__path__``, ``__pathname__``, and ``__fspathname__``. In
>> the end people seemed to gravitate towards ``__fspath__`` for being
>> unambiguous without being unnecessarily long.
>>
>>
>> Separate str/bytes methods
>> --------------------------
>>
>> At one point it was suggested that ``__fspath__()`` only return
>> strings and another method named ``__fspathb__()`` be introduced to
>> return bytes. The thinking is that by making ``__fspath__()`` not be
>> polymorphic it could make dealing with the potential string or bytes
>> representations easier. But the general consensus was that returning
>> bytes will more than likely be rare and that the various functions in
>> the os module are the better abstraction to promote over direct
>> calls to ``__fspath__()``.
>>
>>
>> Providing a ``path`` attribute
>> ------------------------------
>>
>> To help deal with the issue of ``pathlib.PurePath`` not inheriting
>> from ``str``, originally it was proposed to introduce a ``path``
>> attribute to mirror what ``os.DirEntry`` provides. In the end,
>> though, it was determined that a protocol would provide the same
>> result while not directly exposing an API that most people will never
>> need to interact with directly.
>>
>>
>> Have ``__fspath__()`` only return strings
>> ------------------------------------------
>>
>> Much of the discussion that led to this PEP revolved around whether
>> ``__fspath__()`` should be polymorphic and return ``bytes`` as well as
>> ``str`` or only return ``str``. The general sentiment for this view
>> was that ``bytes`` are difficult to work with due to their
>> inherent lack of information about their encoding and PEP 383 makes
>> it possible to represent all file system paths using ``str`` with the
>> ``surrogateescape`` handler. Thus, it would be better to forcibly
>> promote the use of ``str`` as the low-level path representation for
>> high-level path objects.
>>
>> In the end, it was decided that using ``bytes`` to represent paths is
>> simply not going to go away and thus they should be supported to some
>> degree. The hope is that people will gravitate towards path objects
>> like pathlib and that will move people away from operating directly
>> with ``bytes``.
>>
>>
>> A generic string encoding mechanism
>> -----------------------------------
>>
>> At one point there was a discussion of developing a generic mechanism
>> to extract a string representation of an object that had semantic
>> meaning (``__str__()`` does not necessarily return anything of
>> semantic significance beyond what may be helpful for debugging). In
>> the end, it was deemed to lack a motivating need beyond the one this
>> PEP is trying to solve in a specific fashion.
>>
>>
>> Have __fspath__ be an attribute
>> -------------------------------
>>
>> It was briefly considered to have ``__fspath__`` be an attribute
>> instead of a method. This was rejected for two reasons. One,
>> historically protocols have been implemented as "magic methods" and
>> not "magic methods and attributes". Two, there is no guarantee that
>> the lower-level representation of a path object will be pre-computed,
>> potentially misleading users that there was no expensive computation
>> behind the scenes in case the attribute was implemented as a property.
>>
>> This also indirectly ties into the idea of introducing a ``path``
>> attribute to accomplish the same thing. This idea has an added issue,
>> though, of accidentally having any object with a ``path`` attribute
>> meet the protocol's duck typing. Introducing a new magic method for
>> the protocol helpfully avoids any accidental opting into the protocol.
>>
>>
>> Provide specific type hinting support
>> -------------------------------------
>>
>> There was some consideration to provdinga generic ``typing.PathLike``
>> class which would allow for e.g. ``typing.PathLike[str]`` to specify
>> a type hint for a path object which returned a string representation.
>> While potentially beneficial, the usefulness was deemed too small to
>> bother adding the type hint class.
>>
>> This also removed any desire to have a class in the ``typing`` module
>> which represented the union of all acceptable path-representing types
>> as that can be represented with
>> ``typing.Union[str, bytes, os.PathLike]`` easily enough and the hope
>> is users will slowly gravitate to path objects only.
>>
>>
>> Provide ``os.fspathb()``
>> ------------------------
>>
>> It was suggested that to mirror the structure of e.g.
>> ``os.getcwd()``/``os.getcwdb()``, that ``os.fspath()`` only return
>> ``str`` and that another function named ``os.fspathb()`` be
>> introduced that only returned ``bytes``. This was rejected as the
>> purposes of the ``*b()`` functions are tied to querying the file
>> system where there is a need to get the raw bytes back. As this PEP
>> does not work directly with data on a file system (but which *may*
>> be), the view was taken this distinction is unnecessary. It's also
>> believed that the need for only bytes will not be common enough to
>> need to support in such a specific manner as ``os.fsencode()`` will
>> provide similar functionality.
>>
>>
>> Call ``__fspath__()`` off of the instance
>> -----------------------------------------
>>
>> An earlier draft of this PEP had ``os.fspath()`` calling
>> ``path.__fspath__()`` instead of ``type(path).__fspath__(path)``. The
>> changed to be consistent with how other magic methods in Python are
>> resolved.
>>
>>
>> Acknowledgements
>> ================
>>
>> Thanks to everyone who participated in the various discussions related
>> to this PEP that spanned both python-ideas and python-dev. Special
>> thanks to Stephen Turnbull for direct feedback on early drafts of this
>> PEP. More special thanks to Koos Zevenhoven and Ethan Furman for not
>> only feedback on early drafts of this PEP but also helping to drive
>> the overall discussion on this topic across the two mailing lists.
>>
>>
>> References
>> ==========
>>
>> .. [#python-ideas-archive] The python-ideas mailing list archive
>>    (https://mail.python.org/pipermail/python-ideas/)
>>
>> .. [#python-dev-archive] The python-dev mailing list archive
>>    (https://mail.python.org/pipermail/python-dev/)
>>
>> .. [#libc-open] ``open()`` documention for the C standard library
>>    (
>> http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html
>> )
>>
>> .. [#pathlib] The ``pathlib`` module
>>    (https://docs.python.org/3/library/pathlib.html#module-pathlib)
>>
>> .. [#builtins-open] The ``builtins.open()`` function
>>    (https://docs.python.org/3/library/functions.html#open)
>>
>> .. [#os-fsencode] The ``os.fsencode()`` function
>>    (https://docs.python.org/3/library/os.html#os.fsencode)
>>
>> .. [#os-fsdecode] The ``os.fsdecode()`` function
>>    (https://docs.python.org/3/library/os.html#os.fsdecode)
>>
>> .. [#os-direntry] The ``os.DirEntry`` class
>>    (https://docs.python.org/3/library/os.html#os.DirEntry)
>>
>> .. [#os-path] The ``os.path`` module
>>    (https://docs.python.org/3/library/os.path.html#module-os.path)
>>
>>
>> Copyright
>> =========
>>
>> This document has been placed in the public domain.
>>
>>
>> ..
>>    Local Variables:
>>    mode: indented-text
>>    indent-tabs-mode: nil
>>    sentence-end-double-space: t
>>    fill-column: 70
>>    coding: utf-8
>>    End:
>>
>>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160516/adc56474/attachment-0001.html>

From brett at python.org  Mon May 16 16:26:25 2016
From: brett at python.org (Brett Cannon)
Date: Mon, 16 May 2016 20:26:25 +0000
Subject: [Python-Dev] File system path PEP, 3rd draft
In-Reply-To: <CAP7+vJJ-m0f-GEknLwH2WOcKvQtXba8aVb7o3m0AWBts9e07Pg@mail.gmail.com>
References: <CAP1=2W6W+WO3-NjiSdCnqKVW1PBRzeEiW8gZGw1GSk1PLsufxA@mail.gmail.com>
 <CAP1=2W5nKxqjBUeAqK8GK9yKrLXrzDEXVz9eUPQF07Gh2yui4w@mail.gmail.com>
 <CAP7+vJJ-m0f-GEknLwH2WOcKvQtXba8aVb7o3m0AWBts9e07Pg@mail.gmail.com>
Message-ID: <CAP1=2W5FHAZuVOXp6pBFQ35YH_58oTMJ1L-KAGdTHNZdb=PzkQ@mail.gmail.com>

On Mon, 16 May 2016 at 13:12 Guido van Rossum <guido at python.org> wrote:

> Once you assign yourself a PEP number I'll do one more pass and then I
> expect to accept it -- the draft looks good to me!
>

Done: https://hg.python.org/peps/rev/b41cb718054a


>
> On Mon, May 16, 2016 at 1:00 PM, Brett Cannon <brett at python.org> wrote:
>
>> Recent discussions have been about type hints which are orthogonal to the
>> PEP, so things have seemed to have reached a steady state.
>>
>> Was there anything else that needed clarification, Guido, or are you
>> ready to pronounce? Or did you want to wait until the language summit? Or
>> did you want to assign a BDFL delegate?
>>
>>
>> On Fri, 13 May 2016 at 11:37 Brett Cannon <brett at python.org> wrote:
>>
>>> Biggest changes since the second draft:
>>>
>>>    1. Resolve __fspath__() from the type, not the instance (for Guido)
>>>    2. Updated the TypeError messages to say "os.PathLike object"
>>>    instead of "path object" (implicitly for Steven)
>>>    3. TODO item to define "path-like" in the glossary (for Steven)
>>>    4. Various more things added to Rejected Ideas
>>>    5. Added Koos as a co-author (for Koos :)
>>>
>>> ----------
>>> PEP: NNN
>>> Title: Adding a file system path protocol
>>> Version: $Revision$
>>> Last-Modified: $Date$
>>> Author: Brett Cannon <brett at python.org>,
>>>         Koos Zevenhoven <k7hoven at gmail.com>
>>> Status: Draft
>>> Type: Standards Track
>>> Content-Type: text/x-rst
>>> Created: 11-May-2016
>>> Post-History: 11-May-2016,
>>>               12-May-2016,
>>>               13-May-2016
>>>
>>>
>>> Abstract
>>> ========
>>>
>>> This PEP proposes a protocol for classes which represent a file system
>>> path to be able to provide a ``str`` or ``bytes`` representation.
>>> Changes to Python's standard library are also proposed to utilize this
>>> protocol where appropriate to facilitate the use of path objects where
>>> historically only ``str`` and/or ``bytes`` file system paths are
>>> accepted. The goal is to facilitate the migration of users towards
>>> rich path objects while providing an easy way to work with code
>>> expecting ``str`` or ``bytes``.
>>>
>>>
>>> Rationale
>>> =========
>>>
>>> Historically in Python, file system paths have been represented as
>>> strings or bytes. This choice of representation has stemmed from C's
>>> own decision to represent file system paths as
>>> ``const char *`` [#libc-open]_. While that is a totally serviceable
>>> format to use for file system paths, it's not necessarily optimal. At
>>> issue is the fact that while all file system paths can be represented
>>> as strings or bytes, not all strings or bytes represent a file system
>>> path. This can lead to issues where any e.g. string duck-types to a
>>> file system path whether it actually represents a path or not.
>>>
>>> To help elevate the representation of file system paths from their
>>> representation as strings and bytes to a richer object representation,
>>> the pathlib module [#pathlib]_ was provisionally introduced in
>>> Python 3.4 through PEP 428. While considered by some as an improvement
>>> over strings and bytes for file system paths, it has suffered from a
>>> lack of adoption. Typically the key issue listed for the low adoption
>>> rate has been the lack of support in the standard library. This lack
>>> of support required users of pathlib to manually convert path objects
>>> to strings by calling ``str(path)`` which many found error-prone.
>>>
>>> One issue in converting path objects to strings comes from
>>> the fact that the only generic way to get a string representation of
>>> the path was to pass the object to ``str()``. This can pose a
>>> problem when done blindly as nearly all Python objects have some
>>> string representation whether they are a path or not, e.g.
>>> ``str(None)`` will give a result that
>>> ``builtins.open()`` [#builtins-open]_ will happily use to create a new
>>> file.
>>>
>>> Exacerbating this whole situation is the
>>> ``DirEntry`` object [#os-direntry]_. While path objects have a
>>> representation that can be extracted using ``str()``, ``DirEntry``
>>> objects expose a ``path`` attribute instead. Having no common
>>> interface between path objects, ``DirEntry``, and any other
>>> third-party path library has become an issue. A solution that allows
>>> any path-representing object to declare that it is a path and a way
>>> to extract a low-level representation that all path objects could
>>> support is desired.
>>>
>>> This PEP then proposes to introduce a new protocol to be followed by
>>> objects which represent file system paths. Providing a protocol allows
>>> for explicit signaling of what objects represent file system paths as
>>> well as a way to extract a lower-level representation that can be used
>>> with older APIs which only support strings or bytes.
>>>
>>> Discussions regarding path objects that led to this PEP can be found
>>> in multiple threads on the python-ideas mailing list archive
>>> [#python-ideas-archive]_ for the months of March and April 2016 and on
>>> the python-dev mailing list archives [#python-dev-archive]_ during
>>> April 2016.
>>>
>>>
>>> Proposal
>>> ========
>>>
>>> This proposal is split into two parts. One part is the proposal of a
>>> protocol for objects to declare and provide support for exposing a
>>> file system path representation. The other part deals with changes to
>>> Python's standard library to support the new protocol. These changes
>>> will also lead to the pathlib module dropping its provisional status.
>>>
>>> Protocol
>>> --------
>>>
>>> The following abstract base class defines the protocol for an object
>>> to be considered a path object::
>>>
>>>     import abc
>>>     import typing as t
>>>
>>>
>>>     class PathLike(abc.ABC):
>>>
>>>         """Abstract base class for implementing the file system path
>>> protocol."""
>>>
>>>         @abc.abstractmethod
>>>         def __fspath__(self) -> t.Union[str, bytes]:
>>>             """Return the file system path representation of the
>>> object."""
>>>             raise NotImplementedError
>>>
>>>
>>> Objects representing file system paths will implement the
>>> ``__fspath__()`` method which will return the ``str`` or ``bytes``
>>> representation of the path. The ``str`` representation is the
>>> preferred low-level path representation as it is human-readable and
>>> what people historically represent paths as.
>>>
>>>
>>> Standard library changes
>>> ------------------------
>>>
>>> It is expected that most APIs in Python's standard library that
>>> currently accept a file system path will be updated appropriately to
>>> accept path objects (whether that requires code or simply an update
>>> to documentation will vary). The modules mentioned below, though,
>>> deserve specific details as they have either fundamental changes that
>>> empower the ability to use path objects, or entail additions/removal
>>> of APIs.
>>>
>>>
>>> builtins
>>> ''''''''
>>>
>>> ``open()`` [#builtins-open]_ will be updated to accept path objects as
>>> well as continue to accept ``str`` and ``bytes``.
>>>
>>>
>>> os
>>> '''
>>>
>>> The ``fspath()`` function will be added with the following semantics::
>>>
>>>     import typing as t
>>>
>>>
>>>     def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str,
>>> bytes]:
>>>         """Return the string representation of the path.
>>>
>>>         If str or bytes is passed in, it is returned unchanged.
>>>         """
>>>         if isinstance(path, (str, bytes)):
>>>             return path
>>>
>>>         # Work from the object's type to match method resolution of
>>> other magic
>>>         # methods.
>>>         path_type = type(path)
>>>         try:
>>>             return path_type.__fspath__(path)
>>>         except AttributeError:
>>>             if hasattr(path_type, '__fspath__'):
>>>                 raise
>>>
>>>             raise TypeError("expected str, bytes or os.PathLike object,
>>> not "
>>>                             + path_type.__name__)
>>>
>>> The ``os.fsencode()`` [#os-fsencode]_ and
>>> ``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept
>>> path objects. As both functions coerce their arguments to
>>> ``bytes`` and ``str``, respectively, they will be updated to call
>>> ``__fspath__()`` if present to convert the path object to a ``str`` or
>>> ``bytes`` representation, and then perform their appropriate
>>> coercion operations as if the return value from ``__fspath__()`` had
>>> been the original argument to the coercion function in question.
>>>
>>> The addition of ``os.fspath()``, the updates to
>>> ``os.fsencode()``/``os.fsdecode()``, and the current semantics of
>>> ``pathlib.PurePath`` provide the semantics necessary to
>>> get the path representation one prefers. For a path object,
>>> ``pathlib.PurePath``/``Path`` can be used. To obtain the ``str`` or
>>> ``bytes`` representation without any coersion, then ``os.fspath()``
>>> can be used. If a ``str`` is desired and the encoding of ``bytes``
>>> should be assumed to be the default file system encoding, then
>>> ``os.fsdecode()`` should be used. If a ``bytes`` representation is
>>> desired and any strings should be encoded using the default file
>>> system encoding, then ``os.fsencode()`` is used. This PEP recommends
>>> using path objects when possible and falling back to string paths as
>>> necessary and using ``bytes`` as a last resort.
>>>
>>> Another way to view this is as a hierarchy of file system path
>>> representations (highest- to lowest-level): path ? str ? bytes. The
>>> functions and classes under discussion can all accept objects on the
>>> same level of the hierarchy, but they vary in whether they promote or
>>> demote objects to another level. The ``pathlib.PurePath`` class can
>>> promote a ``str`` to a path object. The ``os.fspath()`` function can
>>> demote a path object to a ``str`` or ``bytes`` instance, depending
>>> on what ``__fspath__()`` returns.
>>> The ``os.fsdecode()`` function will demote a path object to
>>> a string or promote a ``bytes`` object to a ``str``. The
>>> ``os.fsencode()`` function will demote a path or string object to
>>> ``bytes``. There is no function that provides a way to demote a path
>>> object directly to ``bytes`` while bypassing string demotion.
>>>
>>> The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()``
>>> method. It will return the same value as currently found on the
>>> ``path`` attribute of ``DirEntry`` instances.
>>>
>>> The Protocol_ ABC will be added to the ``os`` module under the name
>>> ``os.PathLike``.
>>>
>>>
>>> os.path
>>> '''''''
>>>
>>> The various path-manipulation functions of ``os.path`` [#os-path]_
>>> will be updated to accept path objects. For polymorphic functions that
>>> accept both bytes and strings, they will be updated to simply use
>>> ``os.fspath()``.
>>>
>>> During the discussions leading up to this PEP it was suggested that
>>> ``os.path`` not be updated using an "explicit is better than implicit"
>>> argument. The thinking was that since ``__fspath__()`` is polymorphic
>>> itself it may be better to have code working with ``os.path`` extract
>>> the path representation from path objects explicitly. There is also
>>> the consideration that adding support this deep into the low-level OS
>>> APIs will lead to code magically supporting path objects without
>>> requiring any documentation updated, leading to potential complaints
>>> when it doesn't work, unbeknownst to the project author.
>>>
>>> But it is the view of this PEP that "practicality beats purity" in
>>> this instance. To help facilitate the transition to supporting path
>>> objects, it is better to make the transition as easy as possible than
>>> to worry about unexpected/undocumented duck typing support for
>>> path objects by projects.
>>>
>>> There has also been the suggestion that ``os.path`` functions could be
>>> used in a tight loop and the overhead of checking or calling
>>> ``__fspath__()`` would be too costly. In this scenario only
>>> path-consuming APIs would be directly updated and path-manipulating
>>> APIs like the ones in ``os.path`` would go unmodified. This would
>>> require library authors to update their code to support path objects
>>> if they performed any path manipulations, but if the library code
>>> passed the path straight through then the library wouldn't need to be
>>> updated. It is the view of this PEP and Guido, though, that this is an
>>> unnecessary worry and that performance will still be acceptable.
>>>
>>>
>>> pathlib
>>> '''''''
>>>
>>> The constructor for ``pathlib.PurePath`` and ``pathlib.Path`` will be
>>> updated to accept ``PathLike`` objects. Both ``PurePath`` and ``Path``
>>> will continue to not accept ``bytes`` path representations, and so if
>>> ``__fspath__()`` returns ``bytes`` it will raise an exception.
>>>
>>> The ``path`` attribute will be removed as this PEP makes it
>>> redundant (it has not been included in any released version of Python
>>> and so is not a backwards-compatibility concern).
>>>
>>>
>>> C API
>>> '''''
>>>
>>> The C API will gain an equivalent function to ``os.fspath()``::
>>>
>>>     /*
>>>         Return the file system path of the object.
>>>
>>>         If the object is str or bytes, then allow it to pass through with
>>>         an incremented refcount. If the object defines __fspath__(), then
>>>         return the result of that method. All other types raise a
>>> TypeError.
>>>     */
>>>     PyObject *
>>>     PyOS_FSPath(PyObject *path)
>>>     {
>>>         if (PyUnicode_Check(path) || PyBytes_Check(path)) {
>>>             Py_INCREF(path);
>>>             return path;
>>>         }
>>>
>>>         if (PyObject_HasAttrString(path->ob_type, "__fspath__")) {
>>>             return PyObject_CallMethodObjArgs(path->ob_type,
>>> "__fspath__", path,
>>>                                             NULL);
>>>         }
>>>
>>>         return PyErr_Format(PyExc_TypeError,
>>>                             "expected a str, bytes, or os.PathLike
>>> object, not %S",
>>>                             path->ob_type);
>>>     }
>>>
>>>
>>>
>>> Backwards compatibility
>>> =======================
>>>
>>> There are no explicit backwards-compatibility concerns. Unless an
>>> object incidentally already defines a ``__fspath__()`` method there is
>>> no reason to expect the pre-existing code to break or expect to have
>>> its semantics implicitly changed.
>>>
>>> Libraries wishing to support path objects and a version of Python
>>> prior to Python 3.6 and the existence of ``os.fspath()`` can use the
>>> idiom of
>>> ``path.__fspath__() if hasattr(path, "__fspath__") else path``.
>>>
>>>
>>> Implementation
>>> ==============
>>>
>>> This is the task list for what this PEP proposes:
>>>
>>> #. Remove the ``path`` attribute from pathlib
>>> #. Remove the provisional status of pathlib
>>> #. Add ``os.PathLike``
>>> #. Add ``os.fspath()``
>>> #. Add ``PyOS_FSPath()``
>>> #. Update ``os.fsencode()``
>>> #. Update ``os.fsdecode()``
>>> #. Update ``pathlib.PurePath`` and ``pathlib.Path``
>>> #. Update ``builtins.open()``
>>> #. Update ``os.DirEntry``
>>> #. Update ``os.path``
>>> #. Add a glossary entry for "path-like"
>>>
>>>
>>> Rejected Ideas
>>> ==============
>>>
>>> Other names for the protocol's method
>>> -------------------------------------
>>>
>>> Various names were proposed during discussions leading to this PEP,
>>> including ``__path__``, ``__pathname__``, and ``__fspathname__``. In
>>> the end people seemed to gravitate towards ``__fspath__`` for being
>>> unambiguous without being unnecessarily long.
>>>
>>>
>>> Separate str/bytes methods
>>> --------------------------
>>>
>>> At one point it was suggested that ``__fspath__()`` only return
>>> strings and another method named ``__fspathb__()`` be introduced to
>>> return bytes. The thinking is that by making ``__fspath__()`` not be
>>> polymorphic it could make dealing with the potential string or bytes
>>> representations easier. But the general consensus was that returning
>>> bytes will more than likely be rare and that the various functions in
>>> the os module are the better abstraction to promote over direct
>>> calls to ``__fspath__()``.
>>>
>>>
>>> Providing a ``path`` attribute
>>> ------------------------------
>>>
>>> To help deal with the issue of ``pathlib.PurePath`` not inheriting
>>> from ``str``, originally it was proposed to introduce a ``path``
>>> attribute to mirror what ``os.DirEntry`` provides. In the end,
>>> though, it was determined that a protocol would provide the same
>>> result while not directly exposing an API that most people will never
>>> need to interact with directly.
>>>
>>>
>>> Have ``__fspath__()`` only return strings
>>> ------------------------------------------
>>>
>>> Much of the discussion that led to this PEP revolved around whether
>>> ``__fspath__()`` should be polymorphic and return ``bytes`` as well as
>>> ``str`` or only return ``str``. The general sentiment for this view
>>> was that ``bytes`` are difficult to work with due to their
>>> inherent lack of information about their encoding and PEP 383 makes
>>> it possible to represent all file system paths using ``str`` with the
>>> ``surrogateescape`` handler. Thus, it would be better to forcibly
>>> promote the use of ``str`` as the low-level path representation for
>>> high-level path objects.
>>>
>>> In the end, it was decided that using ``bytes`` to represent paths is
>>> simply not going to go away and thus they should be supported to some
>>> degree. The hope is that people will gravitate towards path objects
>>> like pathlib and that will move people away from operating directly
>>> with ``bytes``.
>>>
>>>
>>> A generic string encoding mechanism
>>> -----------------------------------
>>>
>>> At one point there was a discussion of developing a generic mechanism
>>> to extract a string representation of an object that had semantic
>>> meaning (``__str__()`` does not necessarily return anything of
>>> semantic significance beyond what may be helpful for debugging). In
>>> the end, it was deemed to lack a motivating need beyond the one this
>>> PEP is trying to solve in a specific fashion.
>>>
>>>
>>> Have __fspath__ be an attribute
>>> -------------------------------
>>>
>>> It was briefly considered to have ``__fspath__`` be an attribute
>>> instead of a method. This was rejected for two reasons. One,
>>> historically protocols have been implemented as "magic methods" and
>>> not "magic methods and attributes". Two, there is no guarantee that
>>> the lower-level representation of a path object will be pre-computed,
>>> potentially misleading users that there was no expensive computation
>>> behind the scenes in case the attribute was implemented as a property.
>>>
>>> This also indirectly ties into the idea of introducing a ``path``
>>> attribute to accomplish the same thing. This idea has an added issue,
>>> though, of accidentally having any object with a ``path`` attribute
>>> meet the protocol's duck typing. Introducing a new magic method for
>>> the protocol helpfully avoids any accidental opting into the protocol.
>>>
>>>
>>> Provide specific type hinting support
>>> -------------------------------------
>>>
>>> There was some consideration to provdinga generic ``typing.PathLike``
>>> class which would allow for e.g. ``typing.PathLike[str]`` to specify
>>> a type hint for a path object which returned a string representation.
>>> While potentially beneficial, the usefulness was deemed too small to
>>> bother adding the type hint class.
>>>
>>> This also removed any desire to have a class in the ``typing`` module
>>> which represented the union of all acceptable path-representing types
>>> as that can be represented with
>>> ``typing.Union[str, bytes, os.PathLike]`` easily enough and the hope
>>> is users will slowly gravitate to path objects only.
>>>
>>>
>>> Provide ``os.fspathb()``
>>> ------------------------
>>>
>>> It was suggested that to mirror the structure of e.g.
>>> ``os.getcwd()``/``os.getcwdb()``, that ``os.fspath()`` only return
>>> ``str`` and that another function named ``os.fspathb()`` be
>>> introduced that only returned ``bytes``. This was rejected as the
>>> purposes of the ``*b()`` functions are tied to querying the file
>>> system where there is a need to get the raw bytes back. As this PEP
>>> does not work directly with data on a file system (but which *may*
>>> be), the view was taken this distinction is unnecessary. It's also
>>> believed that the need for only bytes will not be common enough to
>>> need to support in such a specific manner as ``os.fsencode()`` will
>>> provide similar functionality.
>>>
>>>
>>> Call ``__fspath__()`` off of the instance
>>> -----------------------------------------
>>>
>>> An earlier draft of this PEP had ``os.fspath()`` calling
>>> ``path.__fspath__()`` instead of ``type(path).__fspath__(path)``. The
>>> changed to be consistent with how other magic methods in Python are
>>> resolved.
>>>
>>>
>>> Acknowledgements
>>> ================
>>>
>>> Thanks to everyone who participated in the various discussions related
>>> to this PEP that spanned both python-ideas and python-dev. Special
>>> thanks to Stephen Turnbull for direct feedback on early drafts of this
>>> PEP. More special thanks to Koos Zevenhoven and Ethan Furman for not
>>> only feedback on early drafts of this PEP but also helping to drive
>>> the overall discussion on this topic across the two mailing lists.
>>>
>>>
>>> References
>>> ==========
>>>
>>> .. [#python-ideas-archive] The python-ideas mailing list archive
>>>    (https://mail.python.org/pipermail/python-ideas/)
>>>
>>> .. [#python-dev-archive] The python-dev mailing list archive
>>>    (https://mail.python.org/pipermail/python-dev/)
>>>
>>> .. [#libc-open] ``open()`` documention for the C standard library
>>>    (
>>> http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html
>>> )
>>>
>>> .. [#pathlib] The ``pathlib`` module
>>>    (https://docs.python.org/3/library/pathlib.html#module-pathlib)
>>>
>>> .. [#builtins-open] The ``builtins.open()`` function
>>>    (https://docs.python.org/3/library/functions.html#open)
>>>
>>> .. [#os-fsencode] The ``os.fsencode()`` function
>>>    (https://docs.python.org/3/library/os.html#os.fsencode)
>>>
>>> .. [#os-fsdecode] The ``os.fsdecode()`` function
>>>    (https://docs.python.org/3/library/os.html#os.fsdecode)
>>>
>>> .. [#os-direntry] The ``os.DirEntry`` class
>>>    (https://docs.python.org/3/library/os.html#os.DirEntry)
>>>
>>> .. [#os-path] The ``os.path`` module
>>>    (https://docs.python.org/3/library/os.path.html#module-os.path)
>>>
>>>
>>> Copyright
>>> =========
>>>
>>> This document has been placed in the public domain.
>>>
>>>
>>> ..
>>>    Local Variables:
>>>    mode: indented-text
>>>    indent-tabs-mode: nil
>>>    sentence-end-double-space: t
>>>    fill-column: 70
>>>    coding: utf-8
>>>    End:
>>>
>>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160516/88951465/attachment-0001.html>

From guido at python.org  Mon May 16 17:02:42 2016
From: guido at python.org (Guido van Rossum)
Date: Mon, 16 May 2016 14:02:42 -0700
Subject: [Python-Dev] File system path PEP, 3rd draft
In-Reply-To: <CAP1=2W5FHAZuVOXp6pBFQ35YH_58oTMJ1L-KAGdTHNZdb=PzkQ@mail.gmail.com>
References: <CAP1=2W6W+WO3-NjiSdCnqKVW1PBRzeEiW8gZGw1GSk1PLsufxA@mail.gmail.com>
 <CAP1=2W5nKxqjBUeAqK8GK9yKrLXrzDEXVz9eUPQF07Gh2yui4w@mail.gmail.com>
 <CAP7+vJJ-m0f-GEknLwH2WOcKvQtXba8aVb7o3m0AWBts9e07Pg@mail.gmail.com>
 <CAP1=2W5FHAZuVOXp6pBFQ35YH_58oTMJ1L-KAGdTHNZdb=PzkQ@mail.gmail.com>
Message-ID: <CAP7+vJL-Pi+xYt7VzVdRPjLQxcO6K5bm-2DSY2Qanf5d9U9wSA@mail.gmail.com>

For those following along on the list, it's PEP 519:
https://www.python.org/dev/peps/pep-0519/

I've read it once more and found nothing worth bickering about, so I am
hereby approving PEP 519. Thanks Brett and Koos for getting this one over
the finish line, and congrats! It's been quite an effort (I personally did
not have the patience to keep up with the thread...).



On Mon, May 16, 2016 at 1:26 PM, Brett Cannon <brett at python.org> wrote:

>
>
> On Mon, 16 May 2016 at 13:12 Guido van Rossum <guido at python.org> wrote:
>
>> Once you assign yourself a PEP number I'll do one more pass and then I
>> expect to accept it -- the draft looks good to me!
>>
>
> Done: https://hg.python.org/peps/rev/b41cb718054a
>
>
>>
>> On Mon, May 16, 2016 at 1:00 PM, Brett Cannon <brett at python.org> wrote:
>>
>>> Recent discussions have been about type hints which are orthogonal to
>>> the PEP, so things have seemed to have reached a steady state.
>>>
>>> Was there anything else that needed clarification, Guido, or are you
>>> ready to pronounce? Or did you want to wait until the language summit? Or
>>> did you want to assign a BDFL delegate?
>>>
>>>
>>> On Fri, 13 May 2016 at 11:37 Brett Cannon <brett at python.org> wrote:
>>>
>>>> Biggest changes since the second draft:
>>>>
>>>>    1. Resolve __fspath__() from the type, not the instance (for Guido)
>>>>    2. Updated the TypeError messages to say "os.PathLike object"
>>>>    instead of "path object" (implicitly for Steven)
>>>>    3. TODO item to define "path-like" in the glossary (for Steven)
>>>>    4. Various more things added to Rejected Ideas
>>>>    5. Added Koos as a co-author (for Koos :)
>>>>
>>>> ----------
>>>> PEP: NNN
>>>> Title: Adding a file system path protocol
>>>> Version: $Revision$
>>>> Last-Modified: $Date$
>>>> Author: Brett Cannon <brett at python.org>,
>>>>         Koos Zevenhoven <k7hoven at gmail.com>
>>>> Status: Draft
>>>> Type: Standards Track
>>>> Content-Type: text/x-rst
>>>> Created: 11-May-2016
>>>> Post-History: 11-May-2016,
>>>>               12-May-2016,
>>>>               13-May-2016
>>>>
>>>>
>>>> Abstract
>>>> ========
>>>>
>>>> This PEP proposes a protocol for classes which represent a file system
>>>> path to be able to provide a ``str`` or ``bytes`` representation.
>>>> Changes to Python's standard library are also proposed to utilize this
>>>> protocol where appropriate to facilitate the use of path objects where
>>>> historically only ``str`` and/or ``bytes`` file system paths are
>>>> accepted. The goal is to facilitate the migration of users towards
>>>> rich path objects while providing an easy way to work with code
>>>> expecting ``str`` or ``bytes``.
>>>>
>>>>
>>>> Rationale
>>>> =========
>>>>
>>>> Historically in Python, file system paths have been represented as
>>>> strings or bytes. This choice of representation has stemmed from C's
>>>> own decision to represent file system paths as
>>>> ``const char *`` [#libc-open]_. While that is a totally serviceable
>>>> format to use for file system paths, it's not necessarily optimal. At
>>>> issue is the fact that while all file system paths can be represented
>>>> as strings or bytes, not all strings or bytes represent a file system
>>>> path. This can lead to issues where any e.g. string duck-types to a
>>>> file system path whether it actually represents a path or not.
>>>>
>>>> To help elevate the representation of file system paths from their
>>>> representation as strings and bytes to a richer object representation,
>>>> the pathlib module [#pathlib]_ was provisionally introduced in
>>>> Python 3.4 through PEP 428. While considered by some as an improvement
>>>> over strings and bytes for file system paths, it has suffered from a
>>>> lack of adoption. Typically the key issue listed for the low adoption
>>>> rate has been the lack of support in the standard library. This lack
>>>> of support required users of pathlib to manually convert path objects
>>>> to strings by calling ``str(path)`` which many found error-prone.
>>>>
>>>> One issue in converting path objects to strings comes from
>>>> the fact that the only generic way to get a string representation of
>>>> the path was to pass the object to ``str()``. This can pose a
>>>> problem when done blindly as nearly all Python objects have some
>>>> string representation whether they are a path or not, e.g.
>>>> ``str(None)`` will give a result that
>>>> ``builtins.open()`` [#builtins-open]_ will happily use to create a new
>>>> file.
>>>>
>>>> Exacerbating this whole situation is the
>>>> ``DirEntry`` object [#os-direntry]_. While path objects have a
>>>> representation that can be extracted using ``str()``, ``DirEntry``
>>>> objects expose a ``path`` attribute instead. Having no common
>>>> interface between path objects, ``DirEntry``, and any other
>>>> third-party path library has become an issue. A solution that allows
>>>> any path-representing object to declare that it is a path and a way
>>>> to extract a low-level representation that all path objects could
>>>> support is desired.
>>>>
>>>> This PEP then proposes to introduce a new protocol to be followed by
>>>> objects which represent file system paths. Providing a protocol allows
>>>> for explicit signaling of what objects represent file system paths as
>>>> well as a way to extract a lower-level representation that can be used
>>>> with older APIs which only support strings or bytes.
>>>>
>>>> Discussions regarding path objects that led to this PEP can be found
>>>> in multiple threads on the python-ideas mailing list archive
>>>> [#python-ideas-archive]_ for the months of March and April 2016 and on
>>>> the python-dev mailing list archives [#python-dev-archive]_ during
>>>> April 2016.
>>>>
>>>>
>>>> Proposal
>>>> ========
>>>>
>>>> This proposal is split into two parts. One part is the proposal of a
>>>> protocol for objects to declare and provide support for exposing a
>>>> file system path representation. The other part deals with changes to
>>>> Python's standard library to support the new protocol. These changes
>>>> will also lead to the pathlib module dropping its provisional status.
>>>>
>>>> Protocol
>>>> --------
>>>>
>>>> The following abstract base class defines the protocol for an object
>>>> to be considered a path object::
>>>>
>>>>     import abc
>>>>     import typing as t
>>>>
>>>>
>>>>     class PathLike(abc.ABC):
>>>>
>>>>         """Abstract base class for implementing the file system path
>>>> protocol."""
>>>>
>>>>         @abc.abstractmethod
>>>>         def __fspath__(self) -> t.Union[str, bytes]:
>>>>             """Return the file system path representation of the
>>>> object."""
>>>>             raise NotImplementedError
>>>>
>>>>
>>>> Objects representing file system paths will implement the
>>>> ``__fspath__()`` method which will return the ``str`` or ``bytes``
>>>> representation of the path. The ``str`` representation is the
>>>> preferred low-level path representation as it is human-readable and
>>>> what people historically represent paths as.
>>>>
>>>>
>>>> Standard library changes
>>>> ------------------------
>>>>
>>>> It is expected that most APIs in Python's standard library that
>>>> currently accept a file system path will be updated appropriately to
>>>> accept path objects (whether that requires code or simply an update
>>>> to documentation will vary). The modules mentioned below, though,
>>>> deserve specific details as they have either fundamental changes that
>>>> empower the ability to use path objects, or entail additions/removal
>>>> of APIs.
>>>>
>>>>
>>>> builtins
>>>> ''''''''
>>>>
>>>> ``open()`` [#builtins-open]_ will be updated to accept path objects as
>>>> well as continue to accept ``str`` and ``bytes``.
>>>>
>>>>
>>>> os
>>>> '''
>>>>
>>>> The ``fspath()`` function will be added with the following semantics::
>>>>
>>>>     import typing as t
>>>>
>>>>
>>>>     def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str,
>>>> bytes]:
>>>>         """Return the string representation of the path.
>>>>
>>>>         If str or bytes is passed in, it is returned unchanged.
>>>>         """
>>>>         if isinstance(path, (str, bytes)):
>>>>             return path
>>>>
>>>>         # Work from the object's type to match method resolution of
>>>> other magic
>>>>         # methods.
>>>>         path_type = type(path)
>>>>         try:
>>>>             return path_type.__fspath__(path)
>>>>         except AttributeError:
>>>>             if hasattr(path_type, '__fspath__'):
>>>>                 raise
>>>>
>>>>             raise TypeError("expected str, bytes or os.PathLike object,
>>>> not "
>>>>                             + path_type.__name__)
>>>>
>>>> The ``os.fsencode()`` [#os-fsencode]_ and
>>>> ``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept
>>>> path objects. As both functions coerce their arguments to
>>>> ``bytes`` and ``str``, respectively, they will be updated to call
>>>> ``__fspath__()`` if present to convert the path object to a ``str`` or
>>>> ``bytes`` representation, and then perform their appropriate
>>>> coercion operations as if the return value from ``__fspath__()`` had
>>>> been the original argument to the coercion function in question.
>>>>
>>>> The addition of ``os.fspath()``, the updates to
>>>> ``os.fsencode()``/``os.fsdecode()``, and the current semantics of
>>>> ``pathlib.PurePath`` provide the semantics necessary to
>>>> get the path representation one prefers. For a path object,
>>>> ``pathlib.PurePath``/``Path`` can be used. To obtain the ``str`` or
>>>> ``bytes`` representation without any coersion, then ``os.fspath()``
>>>> can be used. If a ``str`` is desired and the encoding of ``bytes``
>>>> should be assumed to be the default file system encoding, then
>>>> ``os.fsdecode()`` should be used. If a ``bytes`` representation is
>>>> desired and any strings should be encoded using the default file
>>>> system encoding, then ``os.fsencode()`` is used. This PEP recommends
>>>> using path objects when possible and falling back to string paths as
>>>> necessary and using ``bytes`` as a last resort.
>>>>
>>>> Another way to view this is as a hierarchy of file system path
>>>> representations (highest- to lowest-level): path ? str ? bytes. The
>>>> functions and classes under discussion can all accept objects on the
>>>> same level of the hierarchy, but they vary in whether they promote or
>>>> demote objects to another level. The ``pathlib.PurePath`` class can
>>>> promote a ``str`` to a path object. The ``os.fspath()`` function can
>>>> demote a path object to a ``str`` or ``bytes`` instance, depending
>>>> on what ``__fspath__()`` returns.
>>>> The ``os.fsdecode()`` function will demote a path object to
>>>> a string or promote a ``bytes`` object to a ``str``. The
>>>> ``os.fsencode()`` function will demote a path or string object to
>>>> ``bytes``. There is no function that provides a way to demote a path
>>>> object directly to ``bytes`` while bypassing string demotion.
>>>>
>>>> The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()``
>>>> method. It will return the same value as currently found on the
>>>> ``path`` attribute of ``DirEntry`` instances.
>>>>
>>>> The Protocol_ ABC will be added to the ``os`` module under the name
>>>> ``os.PathLike``.
>>>>
>>>>
>>>> os.path
>>>> '''''''
>>>>
>>>> The various path-manipulation functions of ``os.path`` [#os-path]_
>>>> will be updated to accept path objects. For polymorphic functions that
>>>> accept both bytes and strings, they will be updated to simply use
>>>> ``os.fspath()``.
>>>>
>>>> During the discussions leading up to this PEP it was suggested that
>>>> ``os.path`` not be updated using an "explicit is better than implicit"
>>>> argument. The thinking was that since ``__fspath__()`` is polymorphic
>>>> itself it may be better to have code working with ``os.path`` extract
>>>> the path representation from path objects explicitly. There is also
>>>> the consideration that adding support this deep into the low-level OS
>>>> APIs will lead to code magically supporting path objects without
>>>> requiring any documentation updated, leading to potential complaints
>>>> when it doesn't work, unbeknownst to the project author.
>>>>
>>>> But it is the view of this PEP that "practicality beats purity" in
>>>> this instance. To help facilitate the transition to supporting path
>>>> objects, it is better to make the transition as easy as possible than
>>>> to worry about unexpected/undocumented duck typing support for
>>>> path objects by projects.
>>>>
>>>> There has also been the suggestion that ``os.path`` functions could be
>>>> used in a tight loop and the overhead of checking or calling
>>>> ``__fspath__()`` would be too costly. In this scenario only
>>>> path-consuming APIs would be directly updated and path-manipulating
>>>> APIs like the ones in ``os.path`` would go unmodified. This would
>>>> require library authors to update their code to support path objects
>>>> if they performed any path manipulations, but if the library code
>>>> passed the path straight through then the library wouldn't need to be
>>>> updated. It is the view of this PEP and Guido, though, that this is an
>>>> unnecessary worry and that performance will still be acceptable.
>>>>
>>>>
>>>> pathlib
>>>> '''''''
>>>>
>>>> The constructor for ``pathlib.PurePath`` and ``pathlib.Path`` will be
>>>> updated to accept ``PathLike`` objects. Both ``PurePath`` and ``Path``
>>>> will continue to not accept ``bytes`` path representations, and so if
>>>> ``__fspath__()`` returns ``bytes`` it will raise an exception.
>>>>
>>>> The ``path`` attribute will be removed as this PEP makes it
>>>> redundant (it has not been included in any released version of Python
>>>> and so is not a backwards-compatibility concern).
>>>>
>>>>
>>>> C API
>>>> '''''
>>>>
>>>> The C API will gain an equivalent function to ``os.fspath()``::
>>>>
>>>>     /*
>>>>         Return the file system path of the object.
>>>>
>>>>         If the object is str or bytes, then allow it to pass through
>>>> with
>>>>         an incremented refcount. If the object defines __fspath__(),
>>>> then
>>>>         return the result of that method. All other types raise a
>>>> TypeError.
>>>>     */
>>>>     PyObject *
>>>>     PyOS_FSPath(PyObject *path)
>>>>     {
>>>>         if (PyUnicode_Check(path) || PyBytes_Check(path)) {
>>>>             Py_INCREF(path);
>>>>             return path;
>>>>         }
>>>>
>>>>         if (PyObject_HasAttrString(path->ob_type, "__fspath__")) {
>>>>             return PyObject_CallMethodObjArgs(path->ob_type,
>>>> "__fspath__", path,
>>>>                                             NULL);
>>>>         }
>>>>
>>>>         return PyErr_Format(PyExc_TypeError,
>>>>                             "expected a str, bytes, or os.PathLike
>>>> object, not %S",
>>>>                             path->ob_type);
>>>>     }
>>>>
>>>>
>>>>
>>>> Backwards compatibility
>>>> =======================
>>>>
>>>> There are no explicit backwards-compatibility concerns. Unless an
>>>> object incidentally already defines a ``__fspath__()`` method there is
>>>> no reason to expect the pre-existing code to break or expect to have
>>>> its semantics implicitly changed.
>>>>
>>>> Libraries wishing to support path objects and a version of Python
>>>> prior to Python 3.6 and the existence of ``os.fspath()`` can use the
>>>> idiom of
>>>> ``path.__fspath__() if hasattr(path, "__fspath__") else path``.
>>>>
>>>>
>>>> Implementation
>>>> ==============
>>>>
>>>> This is the task list for what this PEP proposes:
>>>>
>>>> #. Remove the ``path`` attribute from pathlib
>>>> #. Remove the provisional status of pathlib
>>>> #. Add ``os.PathLike``
>>>> #. Add ``os.fspath()``
>>>> #. Add ``PyOS_FSPath()``
>>>> #. Update ``os.fsencode()``
>>>> #. Update ``os.fsdecode()``
>>>> #. Update ``pathlib.PurePath`` and ``pathlib.Path``
>>>> #. Update ``builtins.open()``
>>>> #. Update ``os.DirEntry``
>>>> #. Update ``os.path``
>>>> #. Add a glossary entry for "path-like"
>>>>
>>>>
>>>> Rejected Ideas
>>>> ==============
>>>>
>>>> Other names for the protocol's method
>>>> -------------------------------------
>>>>
>>>> Various names were proposed during discussions leading to this PEP,
>>>> including ``__path__``, ``__pathname__``, and ``__fspathname__``. In
>>>> the end people seemed to gravitate towards ``__fspath__`` for being
>>>> unambiguous without being unnecessarily long.
>>>>
>>>>
>>>> Separate str/bytes methods
>>>> --------------------------
>>>>
>>>> At one point it was suggested that ``__fspath__()`` only return
>>>> strings and another method named ``__fspathb__()`` be introduced to
>>>> return bytes. The thinking is that by making ``__fspath__()`` not be
>>>> polymorphic it could make dealing with the potential string or bytes
>>>> representations easier. But the general consensus was that returning
>>>> bytes will more than likely be rare and that the various functions in
>>>> the os module are the better abstraction to promote over direct
>>>> calls to ``__fspath__()``.
>>>>
>>>>
>>>> Providing a ``path`` attribute
>>>> ------------------------------
>>>>
>>>> To help deal with the issue of ``pathlib.PurePath`` not inheriting
>>>> from ``str``, originally it was proposed to introduce a ``path``
>>>> attribute to mirror what ``os.DirEntry`` provides. In the end,
>>>> though, it was determined that a protocol would provide the same
>>>> result while not directly exposing an API that most people will never
>>>> need to interact with directly.
>>>>
>>>>
>>>> Have ``__fspath__()`` only return strings
>>>> ------------------------------------------
>>>>
>>>> Much of the discussion that led to this PEP revolved around whether
>>>> ``__fspath__()`` should be polymorphic and return ``bytes`` as well as
>>>> ``str`` or only return ``str``. The general sentiment for this view
>>>> was that ``bytes`` are difficult to work with due to their
>>>> inherent lack of information about their encoding and PEP 383 makes
>>>> it possible to represent all file system paths using ``str`` with the
>>>> ``surrogateescape`` handler. Thus, it would be better to forcibly
>>>> promote the use of ``str`` as the low-level path representation for
>>>> high-level path objects.
>>>>
>>>> In the end, it was decided that using ``bytes`` to represent paths is
>>>> simply not going to go away and thus they should be supported to some
>>>> degree. The hope is that people will gravitate towards path objects
>>>> like pathlib and that will move people away from operating directly
>>>> with ``bytes``.
>>>>
>>>>
>>>> A generic string encoding mechanism
>>>> -----------------------------------
>>>>
>>>> At one point there was a discussion of developing a generic mechanism
>>>> to extract a string representation of an object that had semantic
>>>> meaning (``__str__()`` does not necessarily return anything of
>>>> semantic significance beyond what may be helpful for debugging). In
>>>> the end, it was deemed to lack a motivating need beyond the one this
>>>> PEP is trying to solve in a specific fashion.
>>>>
>>>>
>>>> Have __fspath__ be an attribute
>>>> -------------------------------
>>>>
>>>> It was briefly considered to have ``__fspath__`` be an attribute
>>>> instead of a method. This was rejected for two reasons. One,
>>>> historically protocols have been implemented as "magic methods" and
>>>> not "magic methods and attributes". Two, there is no guarantee that
>>>> the lower-level representation of a path object will be pre-computed,
>>>> potentially misleading users that there was no expensive computation
>>>> behind the scenes in case the attribute was implemented as a property.
>>>>
>>>> This also indirectly ties into the idea of introducing a ``path``
>>>> attribute to accomplish the same thing. This idea has an added issue,
>>>> though, of accidentally having any object with a ``path`` attribute
>>>> meet the protocol's duck typing. Introducing a new magic method for
>>>> the protocol helpfully avoids any accidental opting into the protocol.
>>>>
>>>>
>>>> Provide specific type hinting support
>>>> -------------------------------------
>>>>
>>>> There was some consideration to provdinga generic ``typing.PathLike``
>>>> class which would allow for e.g. ``typing.PathLike[str]`` to specify
>>>> a type hint for a path object which returned a string representation.
>>>> While potentially beneficial, the usefulness was deemed too small to
>>>> bother adding the type hint class.
>>>>
>>>> This also removed any desire to have a class in the ``typing`` module
>>>> which represented the union of all acceptable path-representing types
>>>> as that can be represented with
>>>> ``typing.Union[str, bytes, os.PathLike]`` easily enough and the hope
>>>> is users will slowly gravitate to path objects only.
>>>>
>>>>
>>>> Provide ``os.fspathb()``
>>>> ------------------------
>>>>
>>>> It was suggested that to mirror the structure of e.g.
>>>> ``os.getcwd()``/``os.getcwdb()``, that ``os.fspath()`` only return
>>>> ``str`` and that another function named ``os.fspathb()`` be
>>>> introduced that only returned ``bytes``. This was rejected as the
>>>> purposes of the ``*b()`` functions are tied to querying the file
>>>> system where there is a need to get the raw bytes back. As this PEP
>>>> does not work directly with data on a file system (but which *may*
>>>> be), the view was taken this distinction is unnecessary. It's also
>>>> believed that the need for only bytes will not be common enough to
>>>> need to support in such a specific manner as ``os.fsencode()`` will
>>>> provide similar functionality.
>>>>
>>>>
>>>> Call ``__fspath__()`` off of the instance
>>>> -----------------------------------------
>>>>
>>>> An earlier draft of this PEP had ``os.fspath()`` calling
>>>> ``path.__fspath__()`` instead of ``type(path).__fspath__(path)``. The
>>>> changed to be consistent with how other magic methods in Python are
>>>> resolved.
>>>>
>>>>
>>>> Acknowledgements
>>>> ================
>>>>
>>>> Thanks to everyone who participated in the various discussions related
>>>> to this PEP that spanned both python-ideas and python-dev. Special
>>>> thanks to Stephen Turnbull for direct feedback on early drafts of this
>>>> PEP. More special thanks to Koos Zevenhoven and Ethan Furman for not
>>>> only feedback on early drafts of this PEP but also helping to drive
>>>> the overall discussion on this topic across the two mailing lists.
>>>>
>>>>
>>>> References
>>>> ==========
>>>>
>>>> .. [#python-ideas-archive] The python-ideas mailing list archive
>>>>    (https://mail.python.org/pipermail/python-ideas/)
>>>>
>>>> .. [#python-dev-archive] The python-dev mailing list archive
>>>>    (https://mail.python.org/pipermail/python-dev/)
>>>>
>>>> .. [#libc-open] ``open()`` documention for the C standard library
>>>>    (
>>>> http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html
>>>> )
>>>>
>>>> .. [#pathlib] The ``pathlib`` module
>>>>    (https://docs.python.org/3/library/pathlib.html#module-pathlib)
>>>>
>>>> .. [#builtins-open] The ``builtins.open()`` function
>>>>    (https://docs.python.org/3/library/functions.html#open)
>>>>
>>>> .. [#os-fsencode] The ``os.fsencode()`` function
>>>>    (https://docs.python.org/3/library/os.html#os.fsencode)
>>>>
>>>> .. [#os-fsdecode] The ``os.fsdecode()`` function
>>>>    (https://docs.python.org/3/library/os.html#os.fsdecode)
>>>>
>>>> .. [#os-direntry] The ``os.DirEntry`` class
>>>>    (https://docs.python.org/3/library/os.html#os.DirEntry)
>>>>
>>>> .. [#os-path] The ``os.path`` module
>>>>    (https://docs.python.org/3/library/os.path.html#module-os.path)
>>>>
>>>>
>>>> Copyright
>>>> =========
>>>>
>>>> This document has been placed in the public domain.
>>>>
>>>>
>>>> ..
>>>>    Local Variables:
>>>>    mode: indented-text
>>>>    indent-tabs-mode: nil
>>>>    sentence-end-double-space: t
>>>>    fill-column: 70
>>>>    coding: utf-8
>>>>    End:
>>>>
>>>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160516/4882e394/attachment-0001.html>

From brett at python.org  Mon May 16 17:32:48 2016
From: brett at python.org (Brett Cannon)
Date: Mon, 16 May 2016 21:32:48 +0000
Subject: [Python-Dev] File system path PEP, 3rd draft
In-Reply-To: <CAP7+vJL-Pi+xYt7VzVdRPjLQxcO6K5bm-2DSY2Qanf5d9U9wSA@mail.gmail.com>
References: <CAP1=2W6W+WO3-NjiSdCnqKVW1PBRzeEiW8gZGw1GSk1PLsufxA@mail.gmail.com>
 <CAP1=2W5nKxqjBUeAqK8GK9yKrLXrzDEXVz9eUPQF07Gh2yui4w@mail.gmail.com>
 <CAP7+vJJ-m0f-GEknLwH2WOcKvQtXba8aVb7o3m0AWBts9e07Pg@mail.gmail.com>
 <CAP1=2W5FHAZuVOXp6pBFQ35YH_58oTMJ1L-KAGdTHNZdb=PzkQ@mail.gmail.com>
 <CAP7+vJL-Pi+xYt7VzVdRPjLQxcO6K5bm-2DSY2Qanf5d9U9wSA@mail.gmail.com>
Message-ID: <CAP1=2W7tD6e8Xk57_bHhjqdjgA6bwawVYJWanO9wcSz_HxERQg@mail.gmail.com>

On Mon, 16 May 2016 at 14:03 Guido van Rossum <guido at python.org> wrote:

> For those following along on the list, it's PEP 519:
> https://www.python.org/dev/peps/pep-0519/
>
> I've read it once more and found nothing worth bickering about, so I am
> hereby approving PEP 519. Thanks Brett and Koos for getting this one over
> the finish line, and congrats! It's been quite an effort (I personally did
> not have the patience to keep up with the thread...).
>

PEP 519 is now marked as accepted! Two PEPs accepted in one day (Nick
approved 518 earlier)!

If pathlib doesn't get updated in 3.4, 3.5, and 3.6 before PyCon it will be
the first thing I do at the sprints so that it isn't an issue for the 3.5.2
release.

-Brett


>
>
>
> On Mon, May 16, 2016 at 1:26 PM, Brett Cannon <brett at python.org> wrote:
>
>>
>>
>> On Mon, 16 May 2016 at 13:12 Guido van Rossum <guido at python.org> wrote:
>>
>>> Once you assign yourself a PEP number I'll do one more pass and then I
>>> expect to accept it -- the draft looks good to me!
>>>
>>
>> Done: https://hg.python.org/peps/rev/b41cb718054a
>>
>>
>>>
>>> On Mon, May 16, 2016 at 1:00 PM, Brett Cannon <brett at python.org> wrote:
>>>
>>>> Recent discussions have been about type hints which are orthogonal to
>>>> the PEP, so things have seemed to have reached a steady state.
>>>>
>>>> Was there anything else that needed clarification, Guido, or are you
>>>> ready to pronounce? Or did you want to wait until the language summit? Or
>>>> did you want to assign a BDFL delegate?
>>>>
>>>>
>>>> On Fri, 13 May 2016 at 11:37 Brett Cannon <brett at python.org> wrote:
>>>>
>>>>> Biggest changes since the second draft:
>>>>>
>>>>>    1. Resolve __fspath__() from the type, not the instance (for Guido)
>>>>>    2. Updated the TypeError messages to say "os.PathLike object"
>>>>>    instead of "path object" (implicitly for Steven)
>>>>>    3. TODO item to define "path-like" in the glossary (for Steven)
>>>>>    4. Various more things added to Rejected Ideas
>>>>>    5. Added Koos as a co-author (for Koos :)
>>>>>
>>>>> ----------
>>>>> PEP: NNN
>>>>> Title: Adding a file system path protocol
>>>>> Version: $Revision$
>>>>> Last-Modified: $Date$
>>>>> Author: Brett Cannon <brett at python.org>,
>>>>>         Koos Zevenhoven <k7hoven at gmail.com>
>>>>> Status: Draft
>>>>> Type: Standards Track
>>>>> Content-Type: text/x-rst
>>>>> Created: 11-May-2016
>>>>> Post-History: 11-May-2016,
>>>>>               12-May-2016,
>>>>>               13-May-2016
>>>>>
>>>>>
>>>>> Abstract
>>>>> ========
>>>>>
>>>>> This PEP proposes a protocol for classes which represent a file system
>>>>> path to be able to provide a ``str`` or ``bytes`` representation.
>>>>> Changes to Python's standard library are also proposed to utilize this
>>>>> protocol where appropriate to facilitate the use of path objects where
>>>>> historically only ``str`` and/or ``bytes`` file system paths are
>>>>> accepted. The goal is to facilitate the migration of users towards
>>>>> rich path objects while providing an easy way to work with code
>>>>> expecting ``str`` or ``bytes``.
>>>>>
>>>>>
>>>>> Rationale
>>>>> =========
>>>>>
>>>>> Historically in Python, file system paths have been represented as
>>>>> strings or bytes. This choice of representation has stemmed from C's
>>>>> own decision to represent file system paths as
>>>>> ``const char *`` [#libc-open]_. While that is a totally serviceable
>>>>> format to use for file system paths, it's not necessarily optimal. At
>>>>> issue is the fact that while all file system paths can be represented
>>>>> as strings or bytes, not all strings or bytes represent a file system
>>>>> path. This can lead to issues where any e.g. string duck-types to a
>>>>> file system path whether it actually represents a path or not.
>>>>>
>>>>> To help elevate the representation of file system paths from their
>>>>> representation as strings and bytes to a richer object representation,
>>>>> the pathlib module [#pathlib]_ was provisionally introduced in
>>>>> Python 3.4 through PEP 428. While considered by some as an improvement
>>>>> over strings and bytes for file system paths, it has suffered from a
>>>>> lack of adoption. Typically the key issue listed for the low adoption
>>>>> rate has been the lack of support in the standard library. This lack
>>>>> of support required users of pathlib to manually convert path objects
>>>>> to strings by calling ``str(path)`` which many found error-prone.
>>>>>
>>>>> One issue in converting path objects to strings comes from
>>>>> the fact that the only generic way to get a string representation of
>>>>> the path was to pass the object to ``str()``. This can pose a
>>>>> problem when done blindly as nearly all Python objects have some
>>>>> string representation whether they are a path or not, e.g.
>>>>> ``str(None)`` will give a result that
>>>>> ``builtins.open()`` [#builtins-open]_ will happily use to create a new
>>>>> file.
>>>>>
>>>>> Exacerbating this whole situation is the
>>>>> ``DirEntry`` object [#os-direntry]_. While path objects have a
>>>>> representation that can be extracted using ``str()``, ``DirEntry``
>>>>> objects expose a ``path`` attribute instead. Having no common
>>>>> interface between path objects, ``DirEntry``, and any other
>>>>> third-party path library has become an issue. A solution that allows
>>>>> any path-representing object to declare that it is a path and a way
>>>>> to extract a low-level representation that all path objects could
>>>>> support is desired.
>>>>>
>>>>> This PEP then proposes to introduce a new protocol to be followed by
>>>>> objects which represent file system paths. Providing a protocol allows
>>>>> for explicit signaling of what objects represent file system paths as
>>>>> well as a way to extract a lower-level representation that can be used
>>>>> with older APIs which only support strings or bytes.
>>>>>
>>>>> Discussions regarding path objects that led to this PEP can be found
>>>>> in multiple threads on the python-ideas mailing list archive
>>>>> [#python-ideas-archive]_ for the months of March and April 2016 and on
>>>>> the python-dev mailing list archives [#python-dev-archive]_ during
>>>>> April 2016.
>>>>>
>>>>>
>>>>> Proposal
>>>>> ========
>>>>>
>>>>> This proposal is split into two parts. One part is the proposal of a
>>>>> protocol for objects to declare and provide support for exposing a
>>>>> file system path representation. The other part deals with changes to
>>>>> Python's standard library to support the new protocol. These changes
>>>>> will also lead to the pathlib module dropping its provisional status.
>>>>>
>>>>> Protocol
>>>>> --------
>>>>>
>>>>> The following abstract base class defines the protocol for an object
>>>>> to be considered a path object::
>>>>>
>>>>>     import abc
>>>>>     import typing as t
>>>>>
>>>>>
>>>>>     class PathLike(abc.ABC):
>>>>>
>>>>>         """Abstract base class for implementing the file system path
>>>>> protocol."""
>>>>>
>>>>>         @abc.abstractmethod
>>>>>         def __fspath__(self) -> t.Union[str, bytes]:
>>>>>             """Return the file system path representation of the
>>>>> object."""
>>>>>             raise NotImplementedError
>>>>>
>>>>>
>>>>> Objects representing file system paths will implement the
>>>>> ``__fspath__()`` method which will return the ``str`` or ``bytes``
>>>>> representation of the path. The ``str`` representation is the
>>>>> preferred low-level path representation as it is human-readable and
>>>>> what people historically represent paths as.
>>>>>
>>>>>
>>>>> Standard library changes
>>>>> ------------------------
>>>>>
>>>>> It is expected that most APIs in Python's standard library that
>>>>> currently accept a file system path will be updated appropriately to
>>>>> accept path objects (whether that requires code or simply an update
>>>>> to documentation will vary). The modules mentioned below, though,
>>>>> deserve specific details as they have either fundamental changes that
>>>>> empower the ability to use path objects, or entail additions/removal
>>>>> of APIs.
>>>>>
>>>>>
>>>>> builtins
>>>>> ''''''''
>>>>>
>>>>> ``open()`` [#builtins-open]_ will be updated to accept path objects as
>>>>> well as continue to accept ``str`` and ``bytes``.
>>>>>
>>>>>
>>>>> os
>>>>> '''
>>>>>
>>>>> The ``fspath()`` function will be added with the following semantics::
>>>>>
>>>>>     import typing as t
>>>>>
>>>>>
>>>>>     def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str,
>>>>> bytes]:
>>>>>         """Return the string representation of the path.
>>>>>
>>>>>         If str or bytes is passed in, it is returned unchanged.
>>>>>         """
>>>>>         if isinstance(path, (str, bytes)):
>>>>>             return path
>>>>>
>>>>>         # Work from the object's type to match method resolution of
>>>>> other magic
>>>>>         # methods.
>>>>>         path_type = type(path)
>>>>>         try:
>>>>>             return path_type.__fspath__(path)
>>>>>         except AttributeError:
>>>>>             if hasattr(path_type, '__fspath__'):
>>>>>                 raise
>>>>>
>>>>>             raise TypeError("expected str, bytes or os.PathLike
>>>>> object, not "
>>>>>                             + path_type.__name__)
>>>>>
>>>>> The ``os.fsencode()`` [#os-fsencode]_ and
>>>>> ``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept
>>>>> path objects. As both functions coerce their arguments to
>>>>> ``bytes`` and ``str``, respectively, they will be updated to call
>>>>> ``__fspath__()`` if present to convert the path object to a ``str`` or
>>>>> ``bytes`` representation, and then perform their appropriate
>>>>> coercion operations as if the return value from ``__fspath__()`` had
>>>>> been the original argument to the coercion function in question.
>>>>>
>>>>> The addition of ``os.fspath()``, the updates to
>>>>> ``os.fsencode()``/``os.fsdecode()``, and the current semantics of
>>>>> ``pathlib.PurePath`` provide the semantics necessary to
>>>>> get the path representation one prefers. For a path object,
>>>>> ``pathlib.PurePath``/``Path`` can be used. To obtain the ``str`` or
>>>>> ``bytes`` representation without any coersion, then ``os.fspath()``
>>>>> can be used. If a ``str`` is desired and the encoding of ``bytes``
>>>>> should be assumed to be the default file system encoding, then
>>>>> ``os.fsdecode()`` should be used. If a ``bytes`` representation is
>>>>> desired and any strings should be encoded using the default file
>>>>> system encoding, then ``os.fsencode()`` is used. This PEP recommends
>>>>> using path objects when possible and falling back to string paths as
>>>>> necessary and using ``bytes`` as a last resort.
>>>>>
>>>>> Another way to view this is as a hierarchy of file system path
>>>>> representations (highest- to lowest-level): path ? str ? bytes. The
>>>>> functions and classes under discussion can all accept objects on the
>>>>> same level of the hierarchy, but they vary in whether they promote or
>>>>> demote objects to another level. The ``pathlib.PurePath`` class can
>>>>> promote a ``str`` to a path object. The ``os.fspath()`` function can
>>>>> demote a path object to a ``str`` or ``bytes`` instance, depending
>>>>> on what ``__fspath__()`` returns.
>>>>> The ``os.fsdecode()`` function will demote a path object to
>>>>> a string or promote a ``bytes`` object to a ``str``. The
>>>>> ``os.fsencode()`` function will demote a path or string object to
>>>>> ``bytes``. There is no function that provides a way to demote a path
>>>>> object directly to ``bytes`` while bypassing string demotion.
>>>>>
>>>>> The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()``
>>>>> method. It will return the same value as currently found on the
>>>>> ``path`` attribute of ``DirEntry`` instances.
>>>>>
>>>>> The Protocol_ ABC will be added to the ``os`` module under the name
>>>>> ``os.PathLike``.
>>>>>
>>>>>
>>>>> os.path
>>>>> '''''''
>>>>>
>>>>> The various path-manipulation functions of ``os.path`` [#os-path]_
>>>>> will be updated to accept path objects. For polymorphic functions that
>>>>> accept both bytes and strings, they will be updated to simply use
>>>>> ``os.fspath()``.
>>>>>
>>>>> During the discussions leading up to this PEP it was suggested that
>>>>> ``os.path`` not be updated using an "explicit is better than implicit"
>>>>> argument. The thinking was that since ``__fspath__()`` is polymorphic
>>>>> itself it may be better to have code working with ``os.path`` extract
>>>>> the path representation from path objects explicitly. There is also
>>>>> the consideration that adding support this deep into the low-level OS
>>>>> APIs will lead to code magically supporting path objects without
>>>>> requiring any documentation updated, leading to potential complaints
>>>>> when it doesn't work, unbeknownst to the project author.
>>>>>
>>>>> But it is the view of this PEP that "practicality beats purity" in
>>>>> this instance. To help facilitate the transition to supporting path
>>>>> objects, it is better to make the transition as easy as possible than
>>>>> to worry about unexpected/undocumented duck typing support for
>>>>> path objects by projects.
>>>>>
>>>>> There has also been the suggestion that ``os.path`` functions could be
>>>>> used in a tight loop and the overhead of checking or calling
>>>>> ``__fspath__()`` would be too costly. In this scenario only
>>>>> path-consuming APIs would be directly updated and path-manipulating
>>>>> APIs like the ones in ``os.path`` would go unmodified. This would
>>>>> require library authors to update their code to support path objects
>>>>> if they performed any path manipulations, but if the library code
>>>>> passed the path straight through then the library wouldn't need to be
>>>>> updated. It is the view of this PEP and Guido, though, that this is an
>>>>> unnecessary worry and that performance will still be acceptable.
>>>>>
>>>>>
>>>>> pathlib
>>>>> '''''''
>>>>>
>>>>> The constructor for ``pathlib.PurePath`` and ``pathlib.Path`` will be
>>>>> updated to accept ``PathLike`` objects. Both ``PurePath`` and ``Path``
>>>>> will continue to not accept ``bytes`` path representations, and so if
>>>>> ``__fspath__()`` returns ``bytes`` it will raise an exception.
>>>>>
>>>>> The ``path`` attribute will be removed as this PEP makes it
>>>>> redundant (it has not been included in any released version of Python
>>>>> and so is not a backwards-compatibility concern).
>>>>>
>>>>>
>>>>> C API
>>>>> '''''
>>>>>
>>>>> The C API will gain an equivalent function to ``os.fspath()``::
>>>>>
>>>>>     /*
>>>>>         Return the file system path of the object.
>>>>>
>>>>>         If the object is str or bytes, then allow it to pass through
>>>>> with
>>>>>         an incremented refcount. If the object defines __fspath__(),
>>>>> then
>>>>>         return the result of that method. All other types raise a
>>>>> TypeError.
>>>>>     */
>>>>>     PyObject *
>>>>>     PyOS_FSPath(PyObject *path)
>>>>>     {
>>>>>         if (PyUnicode_Check(path) || PyBytes_Check(path)) {
>>>>>             Py_INCREF(path);
>>>>>             return path;
>>>>>         }
>>>>>
>>>>>         if (PyObject_HasAttrString(path->ob_type, "__fspath__")) {
>>>>>             return PyObject_CallMethodObjArgs(path->ob_type,
>>>>> "__fspath__", path,
>>>>>                                             NULL);
>>>>>         }
>>>>>
>>>>>         return PyErr_Format(PyExc_TypeError,
>>>>>                             "expected a str, bytes, or os.PathLike
>>>>> object, not %S",
>>>>>                             path->ob_type);
>>>>>     }
>>>>>
>>>>>
>>>>>
>>>>> Backwards compatibility
>>>>> =======================
>>>>>
>>>>> There are no explicit backwards-compatibility concerns. Unless an
>>>>> object incidentally already defines a ``__fspath__()`` method there is
>>>>> no reason to expect the pre-existing code to break or expect to have
>>>>> its semantics implicitly changed.
>>>>>
>>>>> Libraries wishing to support path objects and a version of Python
>>>>> prior to Python 3.6 and the existence of ``os.fspath()`` can use the
>>>>> idiom of
>>>>> ``path.__fspath__() if hasattr(path, "__fspath__") else path``.
>>>>>
>>>>>
>>>>> Implementation
>>>>> ==============
>>>>>
>>>>> This is the task list for what this PEP proposes:
>>>>>
>>>>> #. Remove the ``path`` attribute from pathlib
>>>>> #. Remove the provisional status of pathlib
>>>>> #. Add ``os.PathLike``
>>>>> #. Add ``os.fspath()``
>>>>> #. Add ``PyOS_FSPath()``
>>>>> #. Update ``os.fsencode()``
>>>>> #. Update ``os.fsdecode()``
>>>>> #. Update ``pathlib.PurePath`` and ``pathlib.Path``
>>>>> #. Update ``builtins.open()``
>>>>> #. Update ``os.DirEntry``
>>>>> #. Update ``os.path``
>>>>> #. Add a glossary entry for "path-like"
>>>>>
>>>>>
>>>>> Rejected Ideas
>>>>> ==============
>>>>>
>>>>> Other names for the protocol's method
>>>>> -------------------------------------
>>>>>
>>>>> Various names were proposed during discussions leading to this PEP,
>>>>> including ``__path__``, ``__pathname__``, and ``__fspathname__``. In
>>>>> the end people seemed to gravitate towards ``__fspath__`` for being
>>>>> unambiguous without being unnecessarily long.
>>>>>
>>>>>
>>>>> Separate str/bytes methods
>>>>> --------------------------
>>>>>
>>>>> At one point it was suggested that ``__fspath__()`` only return
>>>>> strings and another method named ``__fspathb__()`` be introduced to
>>>>> return bytes. The thinking is that by making ``__fspath__()`` not be
>>>>> polymorphic it could make dealing with the potential string or bytes
>>>>> representations easier. But the general consensus was that returning
>>>>> bytes will more than likely be rare and that the various functions in
>>>>> the os module are the better abstraction to promote over direct
>>>>> calls to ``__fspath__()``.
>>>>>
>>>>>
>>>>> Providing a ``path`` attribute
>>>>> ------------------------------
>>>>>
>>>>> To help deal with the issue of ``pathlib.PurePath`` not inheriting
>>>>> from ``str``, originally it was proposed to introduce a ``path``
>>>>> attribute to mirror what ``os.DirEntry`` provides. In the end,
>>>>> though, it was determined that a protocol would provide the same
>>>>> result while not directly exposing an API that most people will never
>>>>> need to interact with directly.
>>>>>
>>>>>
>>>>> Have ``__fspath__()`` only return strings
>>>>> ------------------------------------------
>>>>>
>>>>> Much of the discussion that led to this PEP revolved around whether
>>>>> ``__fspath__()`` should be polymorphic and return ``bytes`` as well as
>>>>> ``str`` or only return ``str``. The general sentiment for this view
>>>>> was that ``bytes`` are difficult to work with due to their
>>>>> inherent lack of information about their encoding and PEP 383 makes
>>>>> it possible to represent all file system paths using ``str`` with the
>>>>> ``surrogateescape`` handler. Thus, it would be better to forcibly
>>>>> promote the use of ``str`` as the low-level path representation for
>>>>> high-level path objects.
>>>>>
>>>>> In the end, it was decided that using ``bytes`` to represent paths is
>>>>> simply not going to go away and thus they should be supported to some
>>>>> degree. The hope is that people will gravitate towards path objects
>>>>> like pathlib and that will move people away from operating directly
>>>>> with ``bytes``.
>>>>>
>>>>>
>>>>> A generic string encoding mechanism
>>>>> -----------------------------------
>>>>>
>>>>> At one point there was a discussion of developing a generic mechanism
>>>>> to extract a string representation of an object that had semantic
>>>>> meaning (``__str__()`` does not necessarily return anything of
>>>>> semantic significance beyond what may be helpful for debugging). In
>>>>> the end, it was deemed to lack a motivating need beyond the one this
>>>>> PEP is trying to solve in a specific fashion.
>>>>>
>>>>>
>>>>> Have __fspath__ be an attribute
>>>>> -------------------------------
>>>>>
>>>>> It was briefly considered to have ``__fspath__`` be an attribute
>>>>> instead of a method. This was rejected for two reasons. One,
>>>>> historically protocols have been implemented as "magic methods" and
>>>>> not "magic methods and attributes". Two, there is no guarantee that
>>>>> the lower-level representation of a path object will be pre-computed,
>>>>> potentially misleading users that there was no expensive computation
>>>>> behind the scenes in case the attribute was implemented as a property.
>>>>>
>>>>> This also indirectly ties into the idea of introducing a ``path``
>>>>> attribute to accomplish the same thing. This idea has an added issue,
>>>>> though, of accidentally having any object with a ``path`` attribute
>>>>> meet the protocol's duck typing. Introducing a new magic method for
>>>>> the protocol helpfully avoids any accidental opting into the protocol.
>>>>>
>>>>>
>>>>> Provide specific type hinting support
>>>>> -------------------------------------
>>>>>
>>>>> There was some consideration to provdinga generic ``typing.PathLike``
>>>>> class which would allow for e.g. ``typing.PathLike[str]`` to specify
>>>>> a type hint for a path object which returned a string representation.
>>>>> While potentially beneficial, the usefulness was deemed too small to
>>>>> bother adding the type hint class.
>>>>>
>>>>> This also removed any desire to have a class in the ``typing`` module
>>>>> which represented the union of all acceptable path-representing types
>>>>> as that can be represented with
>>>>> ``typing.Union[str, bytes, os.PathLike]`` easily enough and the hope
>>>>> is users will slowly gravitate to path objects only.
>>>>>
>>>>>
>>>>> Provide ``os.fspathb()``
>>>>> ------------------------
>>>>>
>>>>> It was suggested that to mirror the structure of e.g.
>>>>> ``os.getcwd()``/``os.getcwdb()``, that ``os.fspath()`` only return
>>>>> ``str`` and that another function named ``os.fspathb()`` be
>>>>> introduced that only returned ``bytes``. This was rejected as the
>>>>> purposes of the ``*b()`` functions are tied to querying the file
>>>>> system where there is a need to get the raw bytes back. As this PEP
>>>>> does not work directly with data on a file system (but which *may*
>>>>> be), the view was taken this distinction is unnecessary. It's also
>>>>> believed that the need for only bytes will not be common enough to
>>>>> need to support in such a specific manner as ``os.fsencode()`` will
>>>>> provide similar functionality.
>>>>>
>>>>>
>>>>> Call ``__fspath__()`` off of the instance
>>>>> -----------------------------------------
>>>>>
>>>>> An earlier draft of this PEP had ``os.fspath()`` calling
>>>>> ``path.__fspath__()`` instead of ``type(path).__fspath__(path)``. The
>>>>> changed to be consistent with how other magic methods in Python are
>>>>> resolved.
>>>>>
>>>>>
>>>>> Acknowledgements
>>>>> ================
>>>>>
>>>>> Thanks to everyone who participated in the various discussions related
>>>>> to this PEP that spanned both python-ideas and python-dev. Special
>>>>> thanks to Stephen Turnbull for direct feedback on early drafts of this
>>>>> PEP. More special thanks to Koos Zevenhoven and Ethan Furman for not
>>>>> only feedback on early drafts of this PEP but also helping to drive
>>>>> the overall discussion on this topic across the two mailing lists.
>>>>>
>>>>>
>>>>> References
>>>>> ==========
>>>>>
>>>>> .. [#python-ideas-archive] The python-ideas mailing list archive
>>>>>    (https://mail.python.org/pipermail/python-ideas/)
>>>>>
>>>>> .. [#python-dev-archive] The python-dev mailing list archive
>>>>>    (https://mail.python.org/pipermail/python-dev/)
>>>>>
>>>>> .. [#libc-open] ``open()`` documention for the C standard library
>>>>>    (
>>>>> http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html
>>>>> )
>>>>>
>>>>> .. [#pathlib] The ``pathlib`` module
>>>>>    (https://docs.python.org/3/library/pathlib.html#module-pathlib)
>>>>>
>>>>> .. [#builtins-open] The ``builtins.open()`` function
>>>>>    (https://docs.python.org/3/library/functions.html#open)
>>>>>
>>>>> .. [#os-fsencode] The ``os.fsencode()`` function
>>>>>    (https://docs.python.org/3/library/os.html#os.fsencode)
>>>>>
>>>>> .. [#os-fsdecode] The ``os.fsdecode()`` function
>>>>>    (https://docs.python.org/3/library/os.html#os.fsdecode)
>>>>>
>>>>> .. [#os-direntry] The ``os.DirEntry`` class
>>>>>    (https://docs.python.org/3/library/os.html#os.DirEntry)
>>>>>
>>>>> .. [#os-path] The ``os.path`` module
>>>>>    (https://docs.python.org/3/library/os.path.html#module-os.path)
>>>>>
>>>>>
>>>>> Copyright
>>>>> =========
>>>>>
>>>>> This document has been placed in the public domain.
>>>>>
>>>>>
>>>>> ..
>>>>>    Local Variables:
>>>>>    mode: indented-text
>>>>>    indent-tabs-mode: nil
>>>>>    sentence-end-double-space: t
>>>>>    fill-column: 70
>>>>>    coding: utf-8
>>>>>    End:
>>>>>
>>>>>
>>>
>>>
>>> --
>>> --Guido van Rossum (python.org/~guido)
>>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160516/7a6231d2/attachment-0001.html>

From ethan at stoneleaf.us  Mon May 16 20:01:26 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 16 May 2016 17:01:26 -0700
Subject: [Python-Dev] File system path PEP, 3rd draft
In-Reply-To: <CAP1=2W7tD6e8Xk57_bHhjqdjgA6bwawVYJWanO9wcSz_HxERQg@mail.gmail.com>
References: <CAP1=2W6W+WO3-NjiSdCnqKVW1PBRzeEiW8gZGw1GSk1PLsufxA@mail.gmail.com>
 <CAP1=2W5nKxqjBUeAqK8GK9yKrLXrzDEXVz9eUPQF07Gh2yui4w@mail.gmail.com>
 <CAP7+vJJ-m0f-GEknLwH2WOcKvQtXba8aVb7o3m0AWBts9e07Pg@mail.gmail.com>
 <CAP1=2W5FHAZuVOXp6pBFQ35YH_58oTMJ1L-KAGdTHNZdb=PzkQ@mail.gmail.com>
 <CAP7+vJL-Pi+xYt7VzVdRPjLQxcO6K5bm-2DSY2Qanf5d9U9wSA@mail.gmail.com>
 <CAP1=2W7tD6e8Xk57_bHhjqdjgA6bwawVYJWanO9wcSz_HxERQg@mail.gmail.com>
Message-ID: <573A5F56.3030702@stoneleaf.us>

On 05/16/2016 02:32 PM, Brett Cannon wrote:

> PEP 519 is now marked as accepted! Two PEPs accepted in one day (Nick
> approved 518 earlier)!

Excellent!  :)

> If pathlib doesn't get updated in 3.4, 3.5, and 3.6 before PyCon it will
> be the first thing I do at the sprints so that it isn't an issue for the
> 3.5.2 release.

Isn't 3.4 now in security-fix mode?

--
~Ethan~


From brett at python.org  Mon May 16 20:57:42 2016
From: brett at python.org (Brett Cannon)
Date: Tue, 17 May 2016 00:57:42 +0000
Subject: [Python-Dev] File system path PEP, 3rd draft
In-Reply-To: <573A5F56.3030702@stoneleaf.us>
References: <CAP1=2W6W+WO3-NjiSdCnqKVW1PBRzeEiW8gZGw1GSk1PLsufxA@mail.gmail.com>
 <CAP1=2W5nKxqjBUeAqK8GK9yKrLXrzDEXVz9eUPQF07Gh2yui4w@mail.gmail.com>
 <CAP7+vJJ-m0f-GEknLwH2WOcKvQtXba8aVb7o3m0AWBts9e07Pg@mail.gmail.com>
 <CAP1=2W5FHAZuVOXp6pBFQ35YH_58oTMJ1L-KAGdTHNZdb=PzkQ@mail.gmail.com>
 <CAP7+vJL-Pi+xYt7VzVdRPjLQxcO6K5bm-2DSY2Qanf5d9U9wSA@mail.gmail.com>
 <CAP1=2W7tD6e8Xk57_bHhjqdjgA6bwawVYJWanO9wcSz_HxERQg@mail.gmail.com>
 <573A5F56.3030702@stoneleaf.us>
Message-ID: <CAP1=2W7920Lkd_6Ej_+sTdt0Trke9z5tc3OnBjfc6zxAXxL_wQ@mail.gmail.com>

On Mon, May 16, 2016, 19:02 Ethan Furman <ethan at stoneleaf.us> wrote:

> On 05/16/2016 02:32 PM, Brett Cannon wrote:
>
> > PEP 519 is now marked as accepted! Two PEPs accepted in one day (Nick
> > approved 518 earlier)!
>
> Excellent!  :)
>
> > If pathlib doesn't get updated in 3.4, 3.5, and 3.6 before PyCon it will
> > be the first thing I do at the sprints so that it isn't an issue for the
> > 3.5.2 release.
>
> Isn't 3.4 now in security-fix mode?
>

Yes, but because of pathlib's provisional status, Guido backported the path
attribute from 3.4 forward.

-brett


> --
> ~Ethan~
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160517/0ab56269/attachment.html>

From cesare.di.mauro at gmail.com  Tue May 17 01:12:22 2016
From: cesare.di.mauro at gmail.com (Cesare Di Mauro)
Date: Tue, 17 May 2016 07:12:22 +0200
Subject: [Python-Dev] Speeding up CPython 5-10%
In-Reply-To: <CAK1QoopawhOE7dUkUYjHj=SSsE74NhgkH6atM4yYJSRBmHbszA@mail.gmail.com>
References: <56A90B97.7090001@gmail.com>
 <CAMpsgwZXxKDzUAQ+_hApdHPQEwnEQLUyQz764VvMW5EXywd=NA@mail.gmail.com>
 <CAP7v7k7eHEKFqbVX2-7RudS0u4oqhrVTuC25FE_dJo8PXQaimA@mail.gmail.com>
 <CAK1QoopawhOE7dUkUYjHj=SSsE74NhgkH6atM4yYJSRBmHbszA@mail.gmail.com>
Message-ID: <CAP7v7k7RgOBsRbuQiBsurQ9DR6o3y3c7iXOkNtEwARbBqUmv7w@mail.gmail.com>

2016-05-16 17:55 GMT+02:00 Meador Inge <meadori at gmail.com>:

> On Sun, May 15, 2016 at 2:23 AM, Cesare Di Mauro <
> cesare.di.mauro at gmail.com> wrote:
>
>
>> Just one thing that comes to my mind: is the stack depth calculation
>> routine changed? It was suboptimal, and calculating a better number
>> decreases stack allocation, and increases the frame usage.
>>
>
> This is still a problem and came up again recently:
>
> http://bugs.python.org/issue26549
>
> -- Meador
>

I saw the last two comments of the issues: this is what I was talking about
(in particular the issue opened by Armin applies).

However there's another case where the situation is even worse.

Let me show a small reproducer:

def test(self):
    for i in range(self.count):
        with self: pass

The stack size reported by Python 2.7.11:
>>> test.__code__.co_stacksize
6

Adding another with statement:
>>> test.__code__.co_stacksize
7

But unfortunately with Python 3.5.1 the problematic is much worse:

>>> test.__code__.co_stacksize
10

>>> test.__code__.co_stacksize
17

Here the situation is exacerbated by the fact that the WITH_CLEANUP
instruction of Python 2.x was split into two (WITH_CLEANUP_START and
WITH_CLEANUP_FINISH) in some Python 3 release.

I don't know why two different instructions were introduced, but IMO it's
better to have one instruction which handles all code finalization of the
with statement, at least in this case. If there are other scenarios where
two different instructions are needed, then ad-hoc instructions like those
can be used.

Regards,
Cesare
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160517/5c5a1f5b/attachment.html>

From larry at hastings.org  Tue May 17 01:28:41 2016
From: larry at hastings.org (Larry Hastings)
Date: Tue, 17 May 2016 07:28:41 +0200
Subject: [Python-Dev] File system path PEP, 3rd draft
In-Reply-To: <CAP1=2W7920Lkd_6Ej_+sTdt0Trke9z5tc3OnBjfc6zxAXxL_wQ@mail.gmail.com>
References: <CAP1=2W6W+WO3-NjiSdCnqKVW1PBRzeEiW8gZGw1GSk1PLsufxA@mail.gmail.com>
 <CAP1=2W5nKxqjBUeAqK8GK9yKrLXrzDEXVz9eUPQF07Gh2yui4w@mail.gmail.com>
 <CAP7+vJJ-m0f-GEknLwH2WOcKvQtXba8aVb7o3m0AWBts9e07Pg@mail.gmail.com>
 <CAP1=2W5FHAZuVOXp6pBFQ35YH_58oTMJ1L-KAGdTHNZdb=PzkQ@mail.gmail.com>
 <CAP7+vJL-Pi+xYt7VzVdRPjLQxcO6K5bm-2DSY2Qanf5d9U9wSA@mail.gmail.com>
 <CAP1=2W7tD6e8Xk57_bHhjqdjgA6bwawVYJWanO9wcSz_HxERQg@mail.gmail.com>
 <573A5F56.3030702@stoneleaf.us>
 <CAP1=2W7920Lkd_6Ej_+sTdt0Trke9z5tc3OnBjfc6zxAXxL_wQ@mail.gmail.com>
Message-ID: <573AAC09.3040206@hastings.org>



On 05/17/2016 02:57 AM, Brett Cannon wrote:
> On Mon, May 16, 2016, 19:02 Ethan Furman <ethan at stoneleaf.us 
> <mailto:ethan at stoneleaf.us>> wrote:
>
>     Isn't 3.4 now in security-fix mode?
>
>
> Yes, but because of pathlib's provisional status, Guido backported the 
> path attribute from 3.4 forward.

I think this is one reason why Raymond Hettinger says "Python is Guido's 
language, he just lets us use it."


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160517/95a3e504/attachment.html>

From jan.jockusch at perfact.de  Tue May 17 08:35:58 2016
From: jan.jockusch at perfact.de (=?UTF-8?Q?Dr._J=c3=a1n_Jockusch?=)
Date: Tue, 17 May 2016 14:35:58 +0200
Subject: [Python-Dev] Python2.7.11+ as in Debian testing and Ubuntu 16.04
 LTS crashes with segfault
Message-ID: <573B102E.5090805@perfact.de>

Hello Python Developers,

I would like to draw your attention to a segmentation fault bug which
concerns the packaged versions of Python 2.7.11 as seen in Ubuntu's
latest 16.04 release and in Debian testing.


I have reported this bug in Ubuntu's tracker here:

https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/1580577


The report contains a script which can reproduce the error for both
distributions' python2.7 packages.

The plain python.org version 2.7.11 does not exhibit this problem.



Matthias Klose, Debian maintainer for the package, has asked me to
report this problem upstream, because otherwise we will invariably run
into this regression in 2.7.12.

I'd like to help in tracking down this bug, please advise.


Thank you,

- J?n Jockusch


From zreed at fastmail.com  Tue May 17 02:25:59 2016
From: zreed at fastmail.com (zreed at fastmail.com)
Date: Tue, 17 May 2016 01:25:59 -0500
Subject: [Python-Dev] Speeding up CPython 5-10%
Message-ID: <1463466359.1523864.609956433.01DA4AAC@webmail.messagingengine.com>

In the project https://github.com/zachariahreed/byteasm I mentioned on
the list earlier this month, I have a pass that to computes stack usage
for a given sequence of bytecodes. It seems to be a fair bit more
agressive than cpython. Maybe it's more generally useful. It's pure
python rather than C though.

From brett at python.org  Tue May 17 11:42:01 2016
From: brett at python.org (Brett Cannon)
Date: Tue, 17 May 2016 15:42:01 +0000
Subject: [Python-Dev] Python2.7.11+ as in Debian testing and Ubuntu
 16.04 LTS crashes with segfault
In-Reply-To: <573B102E.5090805@perfact.de>
References: <573B102E.5090805@perfact.de>
Message-ID: <CAP1=2W71ebnYX6gK15cmmvQK89Mh1dVLFZG2CWEJqM2mibb8Zw@mail.gmail.com>

On Tue, 17 May 2016 at 08:15 Dr. J?n Jockusch <jan.jockusch at perfact.de>
wrote:

> Hello Python Developers,
>
> I would like to draw your attention to a segmentation fault bug which
> concerns the packaged versions of Python 2.7.11 as seen in Ubuntu's
> latest 16.04 release and in Debian testing.
>
>
> I have reported this bug in Ubuntu's tracker here:
>
> https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/1580577
>
>
> The report contains a script which can reproduce the error for both
> distributions' python2.7 packages.
>
> The plain python.org version 2.7.11 does not exhibit this problem.
>
>
>
> Matthias Klose, Debian maintainer for the package, has asked me to
> report this problem upstream, because otherwise we will invariably run
> into this regression in 2.7.12.
>
> I'd like to help in tracking down this bug, please advise.
>

Please file the bug on bugs.python.org so we don't lose track of this.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160517/f4c0f27e/attachment.html>

From nad at python.org  Tue May 17 17:15:19 2016
From: nad at python.org (Ned Deily)
Date: Tue, 17 May 2016 17:15:19 -0400
Subject: [Python-Dev] Python 3.6.0a1 is now available
Message-ID: <28E1448B-F604-48EB-B4E4-0AE2528C8383@python.org>

On behalf of the Python development community and the Python 3.6 release
team, I'm happy to announce the availability of Python 3.6.0a1.
3.6.0a1 is the first of four planned alpha releases of Python 3.6,
the next major release of Python.  During the alpha phase, Python 3.6
remains under heavy development: additional features will be added
and existing features may be modified or deleted.  Please keep in mind
that this is a preview release and its use is not recommended for
production environments.

You can find Python 3.6.0a1 here:

https://www.python.org/downloads/release/python-360a1/ 

The next release of Python 3.6 will be 3.6.0a2, currently scheduled for
2016-06-13.

Enjoy!

--Ned

--
  Ned Deily
  nad at python.org -- []


From dholth at gmail.com  Wed May 18 09:20:41 2016
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 18 May 2016 13:20:41 +0000
Subject: [Python-Dev] Adding a threadlocal to the Python interpreter
Message-ID: <CAG8k2+4SiXjdfC5LE+fwvUqR+HzKJbj8XtkHny1h8cHGdZRM2Q@mail.gmail.com>

I would like to take another stab at adding a threadlocal "str(bytes)
raises an exception" to the Python interpreter, but I had a very hard time
understanding both how to add a threadlocal value to either the interpreter
state or the threadlocal dict that is part of that state, and then how to
access the same value from both Python and CPython code. The structs were
there but it was just hard to understand. Can someone explain it to me?

Thanks,

Daniel Holth
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160518/79c7c7ec/attachment.html>

From cesare.di.mauro at gmail.com  Wed May 18 12:35:30 2016
From: cesare.di.mauro at gmail.com (Cesare Di Mauro)
Date: Wed, 18 May 2016 18:35:30 +0200
Subject: [Python-Dev] Speeding up CPython 5-10%
In-Reply-To: <1463466359.1523864.609956433.01DA4AAC@webmail.messagingengine.com>
References: <1463466359.1523864.609956433.01DA4AAC@webmail.messagingengine.com>
Message-ID: <CAP7v7k5dExS4bTWi_3qG8CvKQen6-fPBVc6q=QAfN6ZjeL5p9w@mail.gmail.com>

2016-05-17 8:25 GMT+02:00 <zreed at fastmail.com>:

> In the project https://github.com/zachariahreed/byteasm I mentioned on
> the list earlier this month, I have a pass that to computes stack usage
> for a given sequence of bytecodes. It seems to be a fair bit more
> agressive than cpython. Maybe it's more generally useful. It's pure
> python rather than C though.
>

IMO it's too big, resource hungry, and slower, even if you convert it in C.

If you take a look at the current stackdepth_walk function which CPython
uses, it's much smaller (not even 50 lines in simple C code) and quite
efficient.

Currently the problem is that it doesn't return the maximum depth of the
tree, but it updates the intermediate/current maximum, and *then* it uses
it for the subsequent calculations. So, the depth artificially grows, like
in the reported cases.

It doesn't require a complete rewrite, but spending some time for
fine-tuning it.

Regards
Cesare
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160518/8e9b2b16/attachment.html>

From zreed at fastmail.com  Wed May 18 12:50:09 2016
From: zreed at fastmail.com (zreed at fastmail.com)
Date: Wed, 18 May 2016 11:50:09 -0500
Subject: [Python-Dev] Speeding up CPython 5-10%
In-Reply-To: <CAP7v7k5dExS4bTWi_3qG8CvKQen6-fPBVc6q=QAfN6ZjeL5p9w@mail.gmail.com>
References: <1463466359.1523864.609956433.01DA4AAC@webmail.messagingengine.com>
 <CAP7v7k5dExS4bTWi_3qG8CvKQen6-fPBVc6q=QAfN6ZjeL5p9w@mail.gmail.com>
Message-ID: <1463590209.997754.611701105.71BF1B08@webmail.messagingengine.com>

Your criticisms may very well be true. IIRC though, I wrote that pass
because what was available was not general enough. The?stackdepth_walk
function made assumptions that, while true of code generated by the
current cpython frontend, were not universally true. If a goal is to
move this calculation after any bytecode optimization, something along
these lines seems like it will eventually be necessary.
 
Anyway, just offering things already written. If you don't feel it's
useful, no worries.
 
 
On Wed, May 18, 2016, at 11:35 AM, Cesare Di Mauro wrote:
> 2016-05-17 8:25 GMT+02:00 <zreed at fastmail.com>:
>> In the project https://github.com/zachariahreed/byteasm I
>> mentioned on
>>  the list earlier this month, I have a pass that to computes
>>  stack usage
>>  for a given sequence of bytecodes. It seems to be a fair bit more
>>  agressive than cpython. Maybe it's more generally useful. It's pure
>>  python rather than C though.
>>
> IMO it's too big, resource hungry, and slower, even if you convert
> it in C.
>
> If you take a look at the current stackdepth_walk function which
> CPython uses, it's much smaller (not even 50 lines in simple C code)
> and quite efficient.
>
> Currently the problem is that it doesn't return the maximum depth of
> the tree, but it updates the intermediate/current maximum, and *then*
> it uses it for the subsequent calculations. So, the depth artificially
> grows, like in the reported cases.
>
> It doesn't require a complete rewrite, but spending some time for fine-
> tuning it.
>
> Regards
> Cesare
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160518/db63084f/attachment.html>

From cesare.di.mauro at gmail.com  Wed May 18 14:06:08 2016
From: cesare.di.mauro at gmail.com (Cesare Di Mauro)
Date: Wed, 18 May 2016 20:06:08 +0200
Subject: [Python-Dev] Speeding up CPython 5-10%
In-Reply-To: <1463590209.997754.611701105.71BF1B08@webmail.messagingengine.com>
References: <1463466359.1523864.609956433.01DA4AAC@webmail.messagingengine.com>
 <CAP7v7k5dExS4bTWi_3qG8CvKQen6-fPBVc6q=QAfN6ZjeL5p9w@mail.gmail.com>
 <1463590209.997754.611701105.71BF1B08@webmail.messagingengine.com>
Message-ID: <CAP7v7k7JQO5+FGrGS9CVtBVveRs5ArPmjpevxZmv9XfubWvzAw@mail.gmail.com>

If you feel like I've attacked you, I apologize: it wasn't my intention.
Please, don't get it personal: I only reported my honest opinion, albeit
after a re-read it looks too rude, and I'm sorry for that.

Regarding the post-bytecode optimization issues, they are mainly
represented by the constant folding code, which is still in the peephole
stage. Once it's moved to the proper place (ASDL/AST), then such kind of
issues with the stack calculations disappear, whereas the remaining ones
can be addressed by a fix of the current stackdepth_walk function.

And just to be clear, I've nothing against your code. I simply think that,
due to my experience, it doesn't fit in CPython.

Regards
Cesare

2016-05-18 18:50 GMT+02:00 <zreed at fastmail.com>:

> Your criticisms may very well be true. IIRC though, I wrote that pass
> because what was available was not general enough. The stackdepth_walk
> function made assumptions that, while true of code generated by the current
> cpython frontend, were not universally true. If a goal is to move this
> calculation after any bytecode optimization, something along these lines
> seems like it will eventually be necessary.
>
> Anyway, just offering things already written. If you don't feel it's
> useful, no worries.
>
>
> On Wed, May 18, 2016, at 11:35 AM, Cesare Di Mauro wrote:
>
> 2016-05-17 8:25 GMT+02:00 <zreed at fastmail.com>:
>
> In the project https://github.com/zachariahreed/byteasm I mentioned on
> the list earlier this month, I have a pass that to computes stack usage
> for a given sequence of bytecodes. It seems to be a fair bit more
> agressive than cpython. Maybe it's more generally useful. It's pure
> python rather than C though.
>
>
> IMO it's too big, resource hungry, and slower, even if you convert it in C.
>
> If you take a look at the current stackdepth_walk function which CPython
> uses, it's much smaller (not even 50 lines in simple C code) and quite
> efficient.
>
> Currently the problem is that it doesn't return the maximum depth of the
> tree, but it updates the intermediate/current maximum, and *then* it uses
> it for the subsequent calculations. So, the depth artificially grows, like
> in the reported cases.
>
> It doesn't require a complete rewrite, but spending some time for
> fine-tuning it.
>
> Regards
> Cesare
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160518/7c8f011c/attachment.html>

From nas at arctrix.com  Wed May 18 14:52:40 2016
From: nas at arctrix.com (Neil Schemenauer)
Date: Wed, 18 May 2016 18:52:40 +0000 (UTC)
Subject: [Python-Dev] PyGC_Collect ignores state of `enabled`
References: <DDA40D7F-F070-42A0-AFFF-B03828894A68@langa.pl>
 <1462859751.2026529.603130785.5D4338A1@webmail.messagingengine.com>
Message-ID: <nhidlo$b8u$1@ger.gmane.org>

Benjamin Peterson <benjamin at python.org> wrote:
> Adding PyGC_CollectIfEnabled() and calling it in Py_Finalize is probably
> fine. I don't think the contract of PyGC_Collect itself (or gc.collect()
> for that matter) should be changed. You might want to disable GC but
> invoke it yourself.

Yes, that sounds okay to me.

I poked around at the calls to PyGC_Collect() and
_PyGC_CollectNoFail().  The cyclic garbage collector gets invoked at
least three times during shutdown.  Once by Py_FinalizeEx() and two
times by PyImport_Cleanup().  That seems a bit exessively expensive
to me.  The collection time can be significant for programs with a
lot of "container" objects in memory.

The whole finalize/shutdown logic of the CPython interpreter could
badly use some improvement.  Currently it is a set of ugly hacks
piled on top of each other.  Now that we have PEP 3121,

    Extension Module Initialization and Finalization
    https://www.python.org/dev/peps/pep-3121/

we should be able to cleanup this mess.  PyImport_Cleanup() is the
main area of trouble.  I don't think we should not be clearing
sys.modules and we should certainly not be clearing module dicts.

If there is some whippersnapper out there who wants to get their
hands dirty with Python internals, fixing PyImport_Cleanup() would
be a juicy project.

  Neil


From ethan at stoneleaf.us  Wed May 18 15:04:23 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 18 May 2016 12:04:23 -0700
Subject: [Python-Dev] PyGC_Collect ignores state of `enabled`
In-Reply-To: <nhidlo$b8u$1@ger.gmane.org>
References: <DDA40D7F-F070-42A0-AFFF-B03828894A68@langa.pl>
 <1462859751.2026529.603130785.5D4338A1@webmail.messagingengine.com>
 <nhidlo$b8u$1@ger.gmane.org>
Message-ID: <573CBCB7.7000202@stoneleaf.us>

On 05/18/2016 11:52 AM, Neil Schemenauer wrote:
> Benjamin Peterson wrote:

>> Adding PyGC_CollectIfEnabled() and calling it in Py_Finalize is probably
>> fine. I don't think the contract of PyGC_Collect itself (or gc.collect()
>> for that matter) should be changed. You might want to disable GC but
>> invoke it yourself.
>
> Yes, that sounds okay to me.
>
> I poked around at the calls to PyGC_Collect() and
> _PyGC_CollectNoFail().  The cyclic garbage collector gets invoked at
> least three times during shutdown.  Once by Py_FinalizeEx() and two
> times by PyImport_Cleanup().  That seems a bit exessively expensive
> to me.  The collection time can be significant for programs with a
> lot of "container" objects in memory.
>
> The whole finalize/shutdown logic of the CPython interpreter could
> badly use some improvement.  Currently it is a set of ugly hacks
> piled on top of each other.  Now that we have PEP 3121,
>
>      Extension Module Initialization and Finalization
>      https://www.python.org/dev/peps/pep-3121/
>
> we should be able to cleanup this mess.  PyImport_Cleanup() is the
> main area of trouble.  I don't think we should not be clearing
> sys.modules and we should certainly not be clearing module dicts.
>
> If there is some whippersnapper out there who wants to get their
> hands dirty with Python internals, fixing PyImport_Cleanup() would
> be a juicy project.

Is there an issue filed for it?

--
~Ethan~


From zreed at fastmail.com  Wed May 18 16:55:03 2016
From: zreed at fastmail.com (zreed at fastmail.com)
Date: Wed, 18 May 2016 15:55:03 -0500
Subject: [Python-Dev] Speeding up CPython 5-10%
In-Reply-To: <CAP7v7k7JQO5+FGrGS9CVtBVveRs5ArPmjpevxZmv9XfubWvzAw@mail.gmail.com>
References: <1463466359.1523864.609956433.01DA4AAC@webmail.messagingengine.com>
 <CAP7v7k5dExS4bTWi_3qG8CvKQen6-fPBVc6q=QAfN6ZjeL5p9w@mail.gmail.com>
 <1463590209.997754.611701105.71BF1B08@webmail.messagingengine.com>
 <CAP7v7k7JQO5+FGrGS9CVtBVveRs5ArPmjpevxZmv9XfubWvzAw@mail.gmail.com>
Message-ID: <1463604903.1810445.611934721.2D819177@webmail.messagingengine.com>

No problem, I did not think you were attacking me or find your
response rude.
 
 
On Wed, May 18, 2016, at 01:06 PM, Cesare Di Mauro wrote:
> If you feel like I've attacked you, I apologize: it wasn't my
> intention. Please, don't get it personal: I only reported my honest
> opinion, albeit after a re-read it looks too rude, and I'm sorry
> for that.
>
> Regarding the post-bytecode optimization issues, they are mainly
> represented by the constant folding code, which is still in the
> peephole stage. Once it's moved to the proper place (ASDL/AST), then
> such kind of issues with the stack calculations disappear, whereas the
> remaining ones can be addressed by a fix of the current
> stackdepth_walk function.
>
> And just to be clear, I've nothing against your code. I simply think
> that, due to my experience, it doesn't fit in CPython.
>
> Regards
> Cesare
>
> 2016-05-18 18:50 GMT+02:00 <zreed at fastmail.com>:
>> __
>> Your criticisms may very well be true. IIRC though, I wrote that pass
>> because what was available was not general enough. The
>> stackdepth_walk function made assumptions that, while true of code
>> generated by the current cpython frontend, were not universally true.
>> If a goal is to move this calculation after any bytecode
>> optimization, something along these lines seems like it will
>> eventually be necessary.
>>
>> Anyway, just offering things already written. If you don't feel it's
>> useful, no worries.
>>
>>
>> On Wed, May 18, 2016, at 11:35 AM, Cesare Di Mauro wrote:
>>> 2016-05-17 8:25 GMT+02:00 <zreed at fastmail.com>:
>>>> In the project https://github.com/zachariahreed/byteasm I
>>>> mentioned on
>>>> the list earlier this month, I have a pass that to computes stack
>>>> usage
>>>> for a given sequence of bytecodes. It seems to be a fair bit more
>>>> agressive than cpython. Maybe it's more generally useful. It's pure
>>>> python rather than C though.
>>>>
>>> IMO it's too big, resource hungry, and slower, even if you convert
>>> it in C.
>>>
>>> If you take a look at the current stackdepth_walk function which
>>> CPython uses, it's much smaller (not even 50 lines in simple C code)
>>> and quite efficient.
>>>
>>> Currently the problem is that it doesn't return the maximum depth of
>>> the tree, but it updates the intermediate/current maximum, and
>>> *then* it uses it for the subsequent calculations. So, the depth
>>> artificially grows, like in the reported cases.
>>>
>>> It doesn't require a complete rewrite, but spending some time for
>>> fine-tuning it.
>>>
>>> Regards
>>> Cesare
>>
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160518/02bef6b8/attachment.html>

From christian at python.org  Wed May 18 17:36:42 2016
From: christian at python.org (Christian Heimes)
Date: Wed, 18 May 2016 23:36:42 +0200
Subject: [Python-Dev] Adding a threadlocal to the Python interpreter
In-Reply-To: <CAG8k2+4SiXjdfC5LE+fwvUqR+HzKJbj8XtkHny1h8cHGdZRM2Q@mail.gmail.com>
References: <CAG8k2+4SiXjdfC5LE+fwvUqR+HzKJbj8XtkHny1h8cHGdZRM2Q@mail.gmail.com>
Message-ID: <e6e5321d-d25e-365f-ba60-f17da2e7abd4@python.org>

On 2016-05-18 15:20, Daniel Holth wrote:
> I would like to take another stab at adding a threadlocal "str(bytes)
> raises an exception" to the Python interpreter, but I had a very hard
> time understanding both how to add a threadlocal value to either the
> interpreter state or the threadlocal dict that is part of that state,
> and then how to access the same value from both Python and CPython code.
> The structs were there but it was just hard to understand. Can someone
> explain it to me?

Python has a two important states related to threads. The
PyInterpreterState contains the state of an interpreter instance: sys
module, loaded modules and a couple of additional settings. Usually
there is just one interpreter state in a Python process. Additional
interpreter states are used to implement subinterpreters.

Each C thread, that wants to run Python code, must have a PyThreadState.
The thread state contains a reference to a PyInterpreterState. Each
PyThreadState has a PyObject *dict member. You can stick Python objects
into the dict. The interpreter cleans up the dict when it reaps a thread.

How performance critical is your code? Does the interpreter have to
check the value of the thread local frequently? In that case you should
add a new member to typedef struct _ts PyThreadState in pystate.h right
before /* XXX signal handlers should also be here */.

Otherwise you can simply use PyThreadState_GetDict(). It returns a
Python dict object that is local to the current thread. You can simply
use a fixed key like in Modules/_decimal/_decimal.c.

Christian

From gvanrossum at gmail.com  Wed May 18 18:39:02 2016
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Wed, 18 May 2016 15:39:02 -0700
Subject: [Python-Dev] Adding Type[C] to PEP 484
In-Reply-To: <CAP7+vJJVnEHYLfFfmRFxYRQbcZE4RLSwT6-JMLdtfkRyXDyNeA@mail.gmail.com>
References: <CAP7+vJL+mpL=N31idm0SGWxAeChJ3eDWiedbYL0m93Ucs3PR6g@mail.gmail.com>
 <CAP1=2W4+gS3Cz7pUkSDcQEF49zJLtc4Zak8YX2iSrPT9U-OFAQ@mail.gmail.com>
 <CACsRUKLyWjCFWP6UtNZf7+NsJ0JdHSVFMiqk2ZmS7d7qj7XmSw@mail.gmail.com>
 <CAP7+vJLYPR1c+ZkB3Z9U57X8vktVn7czA6p4Xi6cd_CjnBDofg@mail.gmail.com>
 <CACsRUKLNXnsUCNeC4g0j2yneqSNhNf4NVsyhRJgTegVg3srmZg@mail.gmail.com>
 <CAP7+vJJ=_w7Grc_RiG18TQtKhDFYicqxULx4pnfX2UiX6MdPdQ@mail.gmail.com>
 <CACsRUKJRP0g_g3jtFEmTJmX2V-f74aH0mtdTByw4V-4odUmWrA@mail.gmail.com>
 <CAP7+vJJVnEHYLfFfmRFxYRQbcZE4RLSwT6-JMLdtfkRyXDyNeA@mail.gmail.com>
Message-ID: <CAP7+vJK3jOkXX2ChkyLb5dkgH_OSFDE_UAk6814=haBVXgsD1A@mail.gmail.com>

FYI, a few people gave useful feedback on my draft text, and I've now
pushed it to PEP 484. All new text is one section:
https://www.python.org/dev/peps/pep-0484/#the-type-of-class-objects

Next I'm going to implement it. Subscribe to this issue if you want to
follow along:
https://github.com/python/typing/issues/107

-- 
--Guido van Rossum (python.org/~guido)

From ncoghlan at gmail.com  Wed May 18 22:11:51 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 19 May 2016 12:11:51 +1000
Subject: [Python-Dev] PyGC_Collect ignores state of `enabled`
In-Reply-To: <573CBCB7.7000202@stoneleaf.us>
References: <DDA40D7F-F070-42A0-AFFF-B03828894A68@langa.pl>
 <1462859751.2026529.603130785.5D4338A1@webmail.messagingengine.com>
 <nhidlo$b8u$1@ger.gmane.org> <573CBCB7.7000202@stoneleaf.us>
Message-ID: <CADiSq7cP-kUYvAkmT4LmoKt5YY-wfC_zc=LZmjj-y5e3ni7SEQ@mail.gmail.com>

On 19 May 2016 at 05:04, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 05/18/2016 11:52 AM, Neil Schemenauer wrote:
>> The whole finalize/shutdown logic of the CPython interpreter could
>> badly use some improvement.  Currently it is a set of ugly hacks
>> piled on top of each other.  Now that we have PEP 3121,
>>
>>      Extension Module Initialization and Finalization
>>      https://www.python.org/dev/peps/pep-3121/
>>
>> we should be able to cleanup this mess.

PEP 3121 is insufficient, since a lot of extension modules can't (or
at least haven't) adopted it in practice.
https://www.python.org/dev/peps/pep-0489/ has some more background on
that (since it was the first step towards tackling the problem in a
different way that extension module authors may be more likely to
actually adopt)

>> PyImport_Cleanup() is the
>> main area of trouble.  I don't think we should not be clearing
>> sys.modules and we should certainly not be clearing module dicts.
>>
>> If there is some whippersnapper out there who wants to get their
>> hands dirty with Python internals, fixing PyImport_Cleanup() would
>> be a juicy project.
>
> Is there an issue filed for it?

It isn't really any one issue, since PyImport_Cleanup aims to tolerate
misbehaving modules across multiple Py_Initialize/Finalize cycles
within a single process, and hence tries as hard as it can to forcibly
break reference cycles and clean up resource allocations.

Switching it over to the suggested PyGC_CollectIfEnabled() API should
be fine though - it will just need to be documented that calling
Py_Initialize again in the same process is unsupported if you have the
GC disabled during a previous call to Py_Finalize.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed May 18 22:30:27 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 19 May 2016 12:30:27 +1000
Subject: [Python-Dev] Adding a threadlocal to the Python interpreter
In-Reply-To: <CAG8k2+4SiXjdfC5LE+fwvUqR+HzKJbj8XtkHny1h8cHGdZRM2Q@mail.gmail.com>
References: <CAG8k2+4SiXjdfC5LE+fwvUqR+HzKJbj8XtkHny1h8cHGdZRM2Q@mail.gmail.com>
Message-ID: <CADiSq7cUm4THGu4tfoRneiZVB6ozWtkA2nsGPUvWAEhEMB2H+g@mail.gmail.com>

On 18 May 2016 at 23:20, Daniel Holth <dholth at gmail.com> wrote:
> I would like to take another stab at adding a threadlocal "str(bytes) raises
> an exception" to the Python interpreter, but I had a very hard time
> understanding both how to add a threadlocal value to either the interpreter
> state or the threadlocal dict that is part of that state, and then how to
> access the same value from both Python and CPython code. The structs were
> there but it was just hard to understand. Can someone explain it to me?

Christian covered the C aspects of the API, while the general purpose
Python aspects live in the threading module.

However, the Python level thread-local API doesn't provide direct
access to the thread state dict. Instead, it provides access to
subdicts stored under per-object keys in that dict, keyed as
"thread.local.<id>":

* Key definition in local_new:
https://hg.python.org/cpython/file/tip/Modules/_threadmodule.c#l705
* Subdict creation in _ldict:
https://hg.python.org/cpython/file/tip/Modules/_threadmodule.c#l810

Getting access to state stored that way from C is *possible*, but
significantly less convenient than access the thread state directly.

What that means is that any time we want to expose thread local state
to both C and Python code, it will generally be the responsibility of
the C code to both manage the key in the thread state dict (or the
field in the thread state struct), and also to provide a Python API
for access that state.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From christian at python.org  Thu May 19 06:34:22 2016
From: christian at python.org (Christian Heimes)
Date: Thu, 19 May 2016 12:34:22 +0200
Subject: [Python-Dev] Adding a threadlocal to the Python interpreter
In-Reply-To: <CADiSq7cUm4THGu4tfoRneiZVB6ozWtkA2nsGPUvWAEhEMB2H+g@mail.gmail.com>
References: <CAG8k2+4SiXjdfC5LE+fwvUqR+HzKJbj8XtkHny1h8cHGdZRM2Q@mail.gmail.com>
 <CADiSq7cUm4THGu4tfoRneiZVB6ozWtkA2nsGPUvWAEhEMB2H+g@mail.gmail.com>
Message-ID: <e402047b-9b1e-474f-3e85-c8cd5cca4622@python.org>

On 2016-05-19 04:30, Nick Coghlan wrote:
> On 18 May 2016 at 23:20, Daniel Holth <dholth at gmail.com> wrote:
>> I would like to take another stab at adding a threadlocal "str(bytes) raises
>> an exception" to the Python interpreter, but I had a very hard time
>> understanding both how to add a threadlocal value to either the interpreter
>> state or the threadlocal dict that is part of that state, and then how to
>> access the same value from both Python and CPython code. The structs were
>> there but it was just hard to understand. Can someone explain it to me?
> 
> Christian covered the C aspects of the API, while the general purpose
> Python aspects live in the threading module.
> 
> However, the Python level thread-local API doesn't provide direct
> access to the thread state dict. Instead, it provides access to
> subdicts stored under per-object keys in that dict, keyed as
> "thread.local.<id>":

In case you wonder about subdicts, they are required to provide multiple
thread local objects. Each thread local instance has its own key in each
thread state dict. The approach enables thread local to have independent
storage objects.

Christian

From srkunze at mail.de  Fri May 20 06:56:24 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Fri, 20 May 2016 12:56:24 +0200
Subject: [Python-Dev] Adding Type[C] to PEP 484
In-Reply-To: <CAP7+vJJVnEHYLfFfmRFxYRQbcZE4RLSwT6-JMLdtfkRyXDyNeA@mail.gmail.com>
References: <CAP7+vJL+mpL=N31idm0SGWxAeChJ3eDWiedbYL0m93Ucs3PR6g@mail.gmail.com>
 <CAP1=2W4+gS3Cz7pUkSDcQEF49zJLtc4Zak8YX2iSrPT9U-OFAQ@mail.gmail.com>
 <CACsRUKLyWjCFWP6UtNZf7+NsJ0JdHSVFMiqk2ZmS7d7qj7XmSw@mail.gmail.com>
 <CAP7+vJLYPR1c+ZkB3Z9U57X8vktVn7czA6p4Xi6cd_CjnBDofg@mail.gmail.com>
 <CACsRUKLNXnsUCNeC4g0j2yneqSNhNf4NVsyhRJgTegVg3srmZg@mail.gmail.com>
 <CAP7+vJJ=_w7Grc_RiG18TQtKhDFYicqxULx4pnfX2UiX6MdPdQ@mail.gmail.com>
 <CACsRUKJRP0g_g3jtFEmTJmX2V-f74aH0mtdTByw4V-4odUmWrA@mail.gmail.com>
 <CAP7+vJJVnEHYLfFfmRFxYRQbcZE4RLSwT6-JMLdtfkRyXDyNeA@mail.gmail.com>
Message-ID: <573EED58.3040508@mail.de>

On 15.05.2016 19:30, Guido van Rossum wrote:
> Right. I still have no good intuition for what Type[BasicUser, 
> ProUser] would mean so I think you should be required to use the 
> Union, which is clear. Type[] should only allow one parameter.

Type[A, B] reminds me of isinstance(obj, (A, B)).


Sven

From victor.stinner at gmail.com  Fri May 20 07:53:04 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 20 May 2016 13:53:04 +0200
Subject: [Python-Dev] Buildbots now always run tests in subprocesses
Message-ID: <CAMpsgwZ1xzVuzM3zhJk6iG6Fg5J2ya-3GdRTuv+z=0USFnNbFw@mail.gmail.com>

Hi,

I modified regrtest to not ignore -j1 anymore (issue #25285). I also
modified Tools/buildbot/test.bat to add the -j1 option (-j1 was
already used on UNIX on "make buildbottest").

All buildbots will now run each test file in a fresh subprocess. It
helps to restrict all kinds of side effects of unit tests to a single
file. The drawback is that buildbots will be less efficient to detect
side effects of unit tests.


By the way, I also changed regrtest output in "multiprocess mode"
(when -jN is used) to add "passed" to test names:
---
$ ./python -m test -j1
...
Run tests in parallel using 1 child processes
0:00:00 [  1/401] test_grammar passed
0:00:00 [  2/401] test_opcodes passed
0:00:01 [  3/401] test_dict passed
---

When tests are run sequentially, the output is different:
---
$ ./python -u -m test
...
Run tests sequentially
0:00:00 [  1/401] test_grammar
0:00:00 [  2/401] test_opcodes
0:00:00 [  3/401] test_dict
...
---

In multiprocess mode, regrtest displays the name of the test which
just completed. In sequential mode, it displays the name of the test
which is currently running.

Maybe the output of sequential mode can also be enhanced, but I made
the minimum change to avoid breaking regrtest. Some parts of the code
rely on the regrtest output format. I added many unit tests to
test_regrtest, but not everything is tested yet.

Victor

From status at bugs.python.org  Fri May 20 12:08:42 2016
From: status at bugs.python.org (Python tracker)
Date: Fri, 20 May 2016 18:08:42 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20160520160842.313B856A9C@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2016-05-13 - 2016-05-20)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5504 ( +3)
  closed 33322 (+52)
  total  38826 (+55)

Open issues with patches: 2381 


Issues opened (33)
==================

#26741: subprocess.Popen should emit a ResourceWarning in destructor i
http://bugs.python.org/issue26741  reopened by haypo

#27015: subprocess.CalledProcessError's repr changes based on kwargs, 
http://bugs.python.org/issue27015  opened by Taywee

#27016: BlockingIOError not raised inside function.
http://bugs.python.org/issue27016  opened by pcunningham80 at gmail.com

#27019: Reduce marshal stack depth for 2.7 on Windows debug build
http://bugs.python.org/issue27019  opened by db3l

#27020: os.writev() does not accept generators (as buffers argument)
http://bugs.python.org/issue27020  opened by mmarkk

#27021: It is not documented that os.writev() suffer from SC_IOV_MAX
http://bugs.python.org/issue27021  opened by mmarkk

#27022: expose sendmmsg() syscall  in sockets API
http://bugs.python.org/issue27022  opened by mmarkk

#27024: IDLE shutdown glitch when started by import
http://bugs.python.org/issue27024  opened by terry.reedy

#27025: More human readable generated widget names
http://bugs.python.org/issue27025  opened by serhiy.storchaka

#27026: async/await keywords are missing from reference docs
http://bugs.python.org/issue27026  opened by hongminhee

#27027: add is_android in test.support to detect Android platform
http://bugs.python.org/issue27027  opened by xdegaye

#27029: Remove support of deprecated mode 'U' in zipfile
http://bugs.python.org/issue27029  opened by serhiy.storchaka

#27030: Remove deprecated re features
http://bugs.python.org/issue27030  opened by serhiy.storchaka

#27032: Remove deprecated html.parser.HTMLParser.unescape()
http://bugs.python.org/issue27032  opened by serhiy.storchaka

#27033: Change the decode_data default in smtpd to False
http://bugs.python.org/issue27033  opened by serhiy.storchaka

#27035: Cannot set exit code in atexit callback
http://bugs.python.org/issue27035  opened by Melebius

#27036: Mark up of references in different form
http://bugs.python.org/issue27036  opened by serhiy.storchaka

#27037: Universal newline support for zipFile.ZipExtFile.read() is not
http://bugs.python.org/issue27037  opened by David Pitchford

#27038: Make os.DirEntry exist
http://bugs.python.org/issue27038  opened by brett.cannon

#27043: Describe what ???inspect.cleandoc??? does to synopsis line.
http://bugs.python.org/issue27043  opened by bignose

#27048: distutils._msvccompiler._get_vc_env() fails with UnicodeDecode
http://bugs.python.org/issue27048  opened by ebarry

#27050: Demote run() below the high level APIs in subprocess docs
http://bugs.python.org/issue27050  opened by ncoghlan

#27051: Create PIP gui
http://bugs.python.org/issue27051  opened by upendra-k14

#27052: Python2.7.11+ as in Debian testing and Ubuntu 16.04 LTS crashe
http://bugs.python.org/issue27052  opened by J??n Jockusch

#27054: Python installation problem: No module named 'encodings'
http://bugs.python.org/issue27054  opened by lauer at wpi.edu

#27060: Documentation of assertItemsEqual in unittest is VERY misleadi
http://bugs.python.org/issue27060  opened by vitaly

#27062: `inspect` doesn't have `__all__`
http://bugs.python.org/issue27062  opened by leewz

#27063: Some unittest loader tests are silently skipped by mistake
http://bugs.python.org/issue27063  opened by serhiy.storchaka

#27064: Make py.exe default to Python 3 when used interactively
http://bugs.python.org/issue27064  opened by paul.moore

#27066: SystemError if custom opener returns -1
http://bugs.python.org/issue27066  opened by barry

#27067: Improve curses tests
http://bugs.python.org/issue27067  opened by serhiy.storchaka

#27068: Add a detach() method to subprocess.Popen
http://bugs.python.org/issue27068  opened by haypo

#27069: webbrowser creates zombi processes in the background mode
http://bugs.python.org/issue27069  opened by haypo



Most recent 15 issues with no replies (15)
==========================================

#27067: Improve curses tests
http://bugs.python.org/issue27067

#27063: Some unittest loader tests are silently skipped by mistake
http://bugs.python.org/issue27063

#27060: Documentation of assertItemsEqual in unittest is VERY misleadi
http://bugs.python.org/issue27060

#27051: Create PIP gui
http://bugs.python.org/issue27051

#27043: Describe what ???inspect.cleandoc??? does to synopsis line.
http://bugs.python.org/issue27043

#27038: Make os.DirEntry exist
http://bugs.python.org/issue27038

#27032: Remove deprecated html.parser.HTMLParser.unescape()
http://bugs.python.org/issue27032

#27030: Remove deprecated re features
http://bugs.python.org/issue27030

#27029: Remove support of deprecated mode 'U' in zipfile
http://bugs.python.org/issue27029

#27021: It is not documented that os.writev() suffer from SC_IOV_MAX
http://bugs.python.org/issue27021

#27019: Reduce marshal stack depth for 2.7 on Windows debug build
http://bugs.python.org/issue27019

#27006: C implementation of Decimal.from_float() bypasses __new__ and 
http://bugs.python.org/issue27006

#27002: Support different modes in posixpath.realpath()
http://bugs.python.org/issue27002

#26997: Docs for pdb should note that __future__ magic doesn't work
http://bugs.python.org/issue26997

#26990: file.tell affect decoding
http://bugs.python.org/issue26990



Most recent 15 issues waiting for review (15)
=============================================

#27067: Improve curses tests
http://bugs.python.org/issue27067

#27066: SystemError if custom opener returns -1
http://bugs.python.org/issue27066

#27064: Make py.exe default to Python 3 when used interactively
http://bugs.python.org/issue27064

#27063: Some unittest loader tests are silently skipped by mistake
http://bugs.python.org/issue27063

#27048: distutils._msvccompiler._get_vc_env() fails with UnicodeDecode
http://bugs.python.org/issue27048

#27036: Mark up of references in different form
http://bugs.python.org/issue27036

#27033: Change the decode_data default in smtpd to False
http://bugs.python.org/issue27033

#27032: Remove deprecated html.parser.HTMLParser.unescape()
http://bugs.python.org/issue27032

#27030: Remove deprecated re features
http://bugs.python.org/issue27030

#27029: Remove support of deprecated mode 'U' in zipfile
http://bugs.python.org/issue27029

#27027: add is_android in test.support to detect Android platform
http://bugs.python.org/issue27027

#27026: async/await keywords are missing from reference docs
http://bugs.python.org/issue27026

#27025: More human readable generated widget names
http://bugs.python.org/issue27025

#27024: IDLE shutdown glitch when started by import
http://bugs.python.org/issue27024

#27019: Reduce marshal stack depth for 2.7 on Windows debug build
http://bugs.python.org/issue27019



Top 10 most discussed issues (10)
=================================

#26870: Unexpected call to readline's add_history in call_readline
http://bugs.python.org/issue26870  11 msgs

#27048: distutils._msvccompiler._get_vc_env() fails with UnicodeDecode
http://bugs.python.org/issue27048  11 msgs

#20371: datetime.datetime.replace bypasses a subclass's __new__
http://bugs.python.org/issue20371   9 msgs

#27036: Mark up of references in different form
http://bugs.python.org/issue27036   8 msgs

#27054: Python installation problem: No module named 'encodings'
http://bugs.python.org/issue27054   8 msgs

#26168: Py_BuildValue may leak 'N' arguments on PyTuple_New failure
http://bugs.python.org/issue26168   6 msgs

#26331: Tokenizer: allow underscores for grouping in numeric literals
http://bugs.python.org/issue26331   6 msgs

#26369: unicode.decode and str.encode are unnecessarily confusing for 
http://bugs.python.org/issue26369   6 msgs

#26814: [WIP] Add a new _PyObject_FastCall() function which avoids the
http://bugs.python.org/issue26814   6 msgs

#26839: Python 3.5 running on Linux kernel 3.17+ can block at startup 
http://bugs.python.org/issue26839   6 msgs



Issues closed (48)
==================

#3213: "pydoc -p" should listen to [::] if IPv6 is supported
http://bugs.python.org/issue3213  closed by martin.panter

#3948: readline steals sigwinch
http://bugs.python.org/issue3948  closed by martin.panter

#11980: zipfile.ZipFile.write should accept fp as argument
http://bugs.python.org/issue11980  closed by serhiy.storchaka

#14132: Redirect is not working correctly in urllib2
http://bugs.python.org/issue14132  closed by martin.panter

#18410: Idle: test SearchDialog.py
http://bugs.python.org/issue18410  closed by terry.reedy

#20640: Idle: test configHelpSourceEdit
http://bugs.python.org/issue20640  closed by terry.reedy

#20792: Idle: test PathBrowser more
http://bugs.python.org/issue20792  closed by terry.reedy

#21676: IDLE - Test Replace Dialog
http://bugs.python.org/issue21676  closed by terry.reedy

#21703: IDLE: Test UndoDelegator
http://bugs.python.org/issue21703  closed by terry.reedy

#21939: IDLE - Test Percolator
http://bugs.python.org/issue21939  closed by terry.reedy

#23275: Can assign [] = (), but not () = []
http://bugs.python.org/issue23275  closed by berker.peksag

#23958: compile warnings in libffi
http://bugs.python.org/issue23958  closed by berker.peksag

#23977: Enhancing IDLE's test_delegator.py unit test
http://bugs.python.org/issue23977  closed by terry.reedy

#25285: regrtest: run tests in subprocesses with -j1 on buildbots
http://bugs.python.org/issue25285  closed by haypo

#25650: Mismatching documentation <=> behaviour for typing.Any
http://bugs.python.org/issue25650  closed by berker.peksag

#25747: test_idle failure in leaks searching mode
http://bugs.python.org/issue25747  closed by terry.reedy

#26039: More flexibility in zipfile write interface
http://bugs.python.org/issue26039  closed by takluyver

#26807: mock_open()().readline() fails at EOF
http://bugs.python.org/issue26807  closed by rbcollins

#26848: asyncio.subprocess's communicate() method mishandles empty inp
http://bugs.python.org/issue26848  closed by berker.peksag

#26850: PyMem_RawMalloc(): update also sys.getallocatedblocks() in deb
http://bugs.python.org/issue26850  closed by haypo

#26938: android: test_concurrent_futures hangs on armv7
http://bugs.python.org/issue26938  closed by xdegaye

#26948: Simplify PyImport_ImportModuleLevelObject: avoid temporary tup
http://bugs.python.org/issue26948  closed by haypo

#26991: Possible reference leak in MAKE_FUNCTION
http://bugs.python.org/issue26991  closed by python-dev

#26995: Add tests for parsing float and object arguments
http://bugs.python.org/issue26995  closed by serhiy.storchaka

#27014: maximum recursion depth when using typing options
http://bugs.python.org/issue27014  closed by gvanrossum

#27017: Python3.5.1: type().startswith()
http://bugs.python.org/issue27017  closed by SilentGhost

#27018: Incorrect documentation of select module
http://bugs.python.org/issue27018  closed by orsenthil

#27023: Documentation of tempfile.gettempdir() does not mention it onl
http://bugs.python.org/issue27023  closed by r.david.murray

#27028: configparser BasicInterpolation issue with %()
http://bugs.python.org/issue27028  closed by r.david.murray

#27031: Remove interface to obsolete Tk commands
http://bugs.python.org/issue27031  closed by serhiy.storchaka

#27034: Remove deprecated asynchat.fifo
http://bugs.python.org/issue27034  closed by serhiy.storchaka

#27039: bytearray.remove cannot remove bytes with value greater than 1
http://bugs.python.org/issue27039  closed by serhiy.storchaka

#27040: asyncio: Add loop.get_exception_handler method
http://bugs.python.org/issue27040  closed by yselivanov

#27041: asyncio: Add loop.create_future method
http://bugs.python.org/issue27041  closed by yselivanov

#27042: Incorrect grammar for function definitions
http://bugs.python.org/issue27042  closed by python-dev

#27044: Stop test_idle memory leaks
http://bugs.python.org/issue27044  closed by terry.reedy

#27045: Forward slashes in Windows paths
http://bugs.python.org/issue27045  closed by ebarry

#27046: Emoji support in Pie Chart legend with matplotlib
http://bugs.python.org/issue27046  closed by berker.peksag

#27047: O(1) deque indexing
http://bugs.python.org/issue27047  closed by rhettinger

#27049: Typo in "The Python Language Reference" 7.11 -- "individiual"
http://bugs.python.org/issue27049  closed by ned.deily

#27053: Python Windows x86-64 embeddable zip file missing module encod
http://bugs.python.org/issue27053  closed by steve.dower

#27055: python fails to run zipapp when zip contains comments
http://bugs.python.org/issue27055  closed by serhiy.storchaka

#27056: pickle: constant propagation in _Unpickler_Read()
http://bugs.python.org/issue27056  closed by haypo

#27057: os.set_inheritable(): fall back to fcntl() if ioctl() fails wi
http://bugs.python.org/issue27057  closed by haypo

#27058: Tkinter's canvas' dashed lines have incorrect segment lengths
http://bugs.python.org/issue27058  closed by kms70847

#27059: find_spec and find_loader ignore package/path argument
http://bugs.python.org/issue27059  closed by brett.cannon

#27061: 2.7.11 OS X pkg doesn't put proper version in the receipts
http://bugs.python.org/issue27061  closed by ned.deily

#27065: robotparser user agent considered hostile by mod_security rule
http://bugs.python.org/issue27065  closed by berker.peksag

From brett at python.org  Fri May 20 12:27:26 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 20 May 2016 16:27:26 +0000
Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after a
 period?
Message-ID: <CAP1=2W4P0+-pmYaNUYhmbt2HdYk6gT+7Xiq92Lv2ZFrGsOAcWQ@mail.gmail.com>

Is there a specific reason for calling out two spaces in comments after a
period(I realize it's probably for consistency, but I sure don't ever think
about this when I write comment)? Otherwise who actually still writes using
two spaces after punctuation? :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160520/6f9b6559/attachment.html>

From guido at python.org  Fri May 20 12:35:21 2016
From: guido at python.org (Guido van Rossum)
Date: Fri, 20 May 2016 09:35:21 -0700
Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after
 a period?
In-Reply-To: <CAP1=2W4P0+-pmYaNUYhmbt2HdYk6gT+7Xiq92Lv2ZFrGsOAcWQ@mail.gmail.com>
References: <CAP1=2W4P0+-pmYaNUYhmbt2HdYk6gT+7Xiq92Lv2ZFrGsOAcWQ@mail.gmail.com>
Message-ID: <CAP7+vJLxZyF8ALUSX=c6Tcc9zi29DmbaUuQ__XKzCDn-YF=Jjw@mail.gmail.com>

In Emacs, the paragraph reflow commands still try to keep a period
followed by only one space together on a line, under the assumption
that it might be something like `Dr. Brett Cannon`. A reasonable HTML
generator should ignore the second space. So all in all I still like
the two-space rule -- not so much for readability but because of
tooling.

On Fri, May 20, 2016 at 9:27 AM, Brett Cannon <brett at python.org> wrote:
> Is there a specific reason for calling out two spaces in comments after a
> period(I realize it's probably for consistency, but I sure don't ever think
> about this when I write comment)? Otherwise who actually still writes using
> two spaces after punctuation? :)
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)

From mafagafogigante at gmail.com  Fri May 20 12:37:05 2016
From: mafagafogigante at gmail.com (Bernardo Sulzbach)
Date: Fri, 20 May 2016 13:37:05 -0300
Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after
 a period?
In-Reply-To: <CAP1=2W4P0+-pmYaNUYhmbt2HdYk6gT+7Xiq92Lv2ZFrGsOAcWQ@mail.gmail.com>
References: <CAP1=2W4P0+-pmYaNUYhmbt2HdYk6gT+7Xiq92Lv2ZFrGsOAcWQ@mail.gmail.com>
Message-ID: <573F3D31.1020806@gmail.com>

On 05/20/2016 01:27 PM, Brett Cannon wrote:
> Is there a specific reason for calling out two spaces in comments after a
> period(I realize it's probably for consistency, but I sure don't ever think
> about this when I write comment)? Otherwise who actually still writes using
> two spaces after punctuation? :)
>

I've also asked myself this question. It may have made more sense back 
then, but nowadays I only get double spaces after a period if Ms. Vim 
decides to use them there for me, making me do substitution magic to 
"clean" it later on.

There may even be a way to disable this in Vim (without changing the 
sources and recompiling), but I never looked after it.

For modern monospaced typefaces and screen resolutions one space after a 
period is as readable (arguably even better looking) than two.

From facundobatista at gmail.com  Fri May 20 12:40:18 2016
From: facundobatista at gmail.com (Facundo Batista)
Date: Fri, 20 May 2016 13:40:18 -0300
Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after
 a period?
In-Reply-To: <CAP1=2W4P0+-pmYaNUYhmbt2HdYk6gT+7Xiq92Lv2ZFrGsOAcWQ@mail.gmail.com>
References: <CAP1=2W4P0+-pmYaNUYhmbt2HdYk6gT+7Xiq92Lv2ZFrGsOAcWQ@mail.gmail.com>
Message-ID: <CAM09pzSZ0Jq8XrpiZ2Q9TRRLXQhkGW1Veop9042ooXMP0E+Xxg@mail.gmail.com>

On Fri, May 20, 2016 at 1:27 PM, Brett Cannon <brett at python.org> wrote:

> Is there a specific reason for calling out two spaces in comments after a
> period(I realize it's probably for consistency, but I sure don't ever think
> about this when I write comment)? Otherwise who actually still writes using
> two spaces after punctuation? :)

I couldn't find it in the reference but AFAIK is something that is
also done in ReStructured Text.

Regards,

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/
Twitter: @facundobatista

From brett at python.org  Fri May 20 12:43:09 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 20 May 2016 16:43:09 +0000
Subject: [Python-Dev] Removing the provisional label from pathlib
Message-ID: <CAP1=2W7pV0joOcoA4Z1RgsrWjgJFjb_JOzJ83pk6cdRjej9_Eg@mail.gmail.com>

Three questions:

   1. Should pathlib gain __fspath__() all the way back to 3.4?
   2. Should pathlib's constructor support __fspath__() all the way back to
   3.4? (separate question as os.fspath() will only be in 3.6; and if we
   backport I'm not looking forward to making Typeshed happy w/o os.PathLike
   being available only in 3.6 :/)
   3. Should the docs from 3.4 and forward reflect the removal of the
   provisional status? (I assume yes, but wanted to double-check)

And a quick thanks to Guido for removing `path` from pathlib for me
already. :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160520/4ba9934a/attachment.html>

From ethan at stoneleaf.us  Fri May 20 12:45:04 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 20 May 2016 09:45:04 -0700
Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after
 a period?
In-Reply-To: <CAP1=2W4P0+-pmYaNUYhmbt2HdYk6gT+7Xiq92Lv2ZFrGsOAcWQ@mail.gmail.com>
References: <CAP1=2W4P0+-pmYaNUYhmbt2HdYk6gT+7Xiq92Lv2ZFrGsOAcWQ@mail.gmail.com>
Message-ID: <573F3F10.4000408@stoneleaf.us>

On 05/20/2016 09:27 AM, Brett Cannon wrote:

> Is there a specific reason for calling out two spaces in comments after
> a period(I realize it's probably for consistency, but I sure don't ever
> think about this when I write comment)? Otherwise who actually still
> writes using two spaces after punctuation? :)

I do!!  :)

The period is a full-stop punctuation mark, but visually obscure [1]. 
The two spaces reinforces its full-stop nature [2].

white-space-matters-ly yrs,

--
~Ethan~


[1] I'm pretty sure "obscure" is not the word I want, but it's  close.

[2] No, I'm not happy with the decision to drop to one space after a
     period in modern grammar [3].

[3] At least, at the colleges my daughter has attended.

From guido at python.org  Fri May 20 12:46:09 2016
From: guido at python.org (Guido van Rossum)
Date: Fri, 20 May 2016 09:46:09 -0700
Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after
 a period?
In-Reply-To: <CAM09pzSZ0Jq8XrpiZ2Q9TRRLXQhkGW1Veop9042ooXMP0E+Xxg@mail.gmail.com>
References: <CAP1=2W4P0+-pmYaNUYhmbt2HdYk6gT+7Xiq92Lv2ZFrGsOAcWQ@mail.gmail.com>
 <CAM09pzSZ0Jq8XrpiZ2Q9TRRLXQhkGW1Veop9042ooXMP0E+Xxg@mail.gmail.com>
Message-ID: <CAP7+vJJRTVB=RqOe7VQvaoWomGnQEhXhCoXjBPYVTEsDMjEakQ@mail.gmail.com>

ALso, in case anyone reading this didn't realize, this is a classic
internet debate that has been going on for decades (just not on
python-dev). Just Google it.

On Fri, May 20, 2016 at 9:40 AM, Facundo Batista
<facundobatista at gmail.com> wrote:
> On Fri, May 20, 2016 at 1:27 PM, Brett Cannon <brett at python.org> wrote:
>
>> Is there a specific reason for calling out two spaces in comments after a
>> period(I realize it's probably for consistency, but I sure don't ever think
>> about this when I write comment)? Otherwise who actually still writes using
>> two spaces after punctuation? :)
>
> I couldn't find it in the reference but AFAIK is something that is
> also done in ReStructured Text.
>
> Regards,
>
> --
> .    Facundo
>
> Blog: http://www.taniquetil.com.ar/plog/
> PyAr: http://www.python.org/ar/
> Twitter: @facundobatista
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From guido at python.org  Fri May 20 12:56:00 2016
From: guido at python.org (Guido van Rossum)
Date: Fri, 20 May 2016 09:56:00 -0700
Subject: [Python-Dev] Removing the provisional label from pathlib
In-Reply-To: <CAP1=2W7pV0joOcoA4Z1RgsrWjgJFjb_JOzJ83pk6cdRjej9_Eg@mail.gmail.com>
References: <CAP1=2W7pV0joOcoA4Z1RgsrWjgJFjb_JOzJ83pk6cdRjej9_Eg@mail.gmail.com>
Message-ID: <CAP7+vJJu8vpEVDGqmLxdGvmZeHw0s0XSb2MPCYRL1AyRAQLJFQ@mail.gmail.com>

Let's start in 3.6 with all this. I added path to 3.4 because I didn't
realize it was in security-mode only. I've now undone all my work
there. Let's not disturb it again, not even its docs.

I don't think there's an "upstream" repo for pathlib (like there still
is for asyncio) and I don't think there's much of a point in
supporting __fspath__ in pathlib if there's no os.fspath(). It would
only encourage hackery in apps that want to play with __fspath__.

On Fri, May 20, 2016 at 9:43 AM, Brett Cannon <brett at python.org> wrote:
> Three questions:
>
> Should pathlib gain __fspath__() all the way back to 3.4?
> Should pathlib's constructor support __fspath__() all the way back to 3.4?
> (separate question as os.fspath() will only be in 3.6; and if we backport
> I'm not looking forward to making Typeshed happy w/o os.PathLike being
> available only in 3.6 :/)
> Should the docs from 3.4 and forward reflect the removal of the provisional
> status? (I assume yes, but wanted to double-check)
>
> And a quick thanks to Guido for removing `path` from pathlib for me already.
> :)
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)

From nas at arctrix.com  Fri May 20 13:00:19 2016
From: nas at arctrix.com (Neil Schemenauer)
Date: Fri, 20 May 2016 17:00:19 +0000 (UTC)
Subject: [Python-Dev] PyGC_Collect ignores state of `enabled`
References: <DDA40D7F-F070-42A0-AFFF-B03828894A68@langa.pl>
 <1462859751.2026529.603130785.5D4338A1@webmail.messagingengine.com>
 <nhidlo$b8u$1@ger.gmane.org> <573CBCB7.7000202@stoneleaf.us>
 <CADiSq7cP-kUYvAkmT4LmoKt5YY-wfC_zc=LZmjj-y5e3ni7SEQ@mail.gmail.com>
Message-ID: <nhnfr2$jqr$1@ger.gmane.org>

Nick Coghlan <ncoghlan at gmail.com> wrote:
> PEP 3121 is insufficient, since a lot of extension modules can't (or
> at least haven't) adopted it in practice.
> https://www.python.org/dev/peps/pep-0489/ has some more background on
> that (since it was the first step towards tackling the problem in a
> different way that extension module authors may be more likely to
> actually adopt)

My idea is that if we can cleanup the built-in extension modules
then maybe that would be enough to stop doing a lot of the
finalization hacks (e.g. module dict clearing).  If 3rd party
extension modules are not fixed, then some finalizers that used to
be called might no longer get called.


From mafagafogigante at gmail.com  Fri May 20 13:30:08 2016
From: mafagafogigante at gmail.com (Bernardo Sulzbach)
Date: Fri, 20 May 2016 14:30:08 -0300
Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after
 a period?
In-Reply-To: <573F3F10.4000408@stoneleaf.us>
References: <CAP1=2W4P0+-pmYaNUYhmbt2HdYk6gT+7Xiq92Lv2ZFrGsOAcWQ@mail.gmail.com>
 <573F3F10.4000408@stoneleaf.us>
Message-ID: <573F49A0.8090404@gmail.com>

> On 05/20/2016 09:27 AM, Brett Cannon wrote:
>
> The period is a full-stop punctuation mark, but visually obscure [1].

It is small. This does not make it hard to notice, however. Not to 
mention that it will usually be followed by a capitalized letter, which 
further highlights it. Commas usually use more pixels or ink and the 
tiny size of the period makes it quite distinguishable to me.

This email-based "tennis" match could go on for quite a while as Guido 
pointed out and there are several good points for both sides. This is 
not a critic to the thread, which regarded PEP 7/8 and seems perfectly 
fine. This is also not a critic to Cannon's view on this.

Who is right in the end? ~One space defenders~ Nobody. There is no 
single right solution to this. As there is no single right decision 
regarding tabs or spaces~, except for spaces, always use spaces~. 
Consistency is what really matters (assuming we agree that one space is 
enough to make it visible).

From brett at python.org  Fri May 20 14:40:12 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 20 May 2016 18:40:12 +0000
Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after
 a period?
In-Reply-To: <CAP7+vJJRTVB=RqOe7VQvaoWomGnQEhXhCoXjBPYVTEsDMjEakQ@mail.gmail.com>
References: <CAP1=2W4P0+-pmYaNUYhmbt2HdYk6gT+7Xiq92Lv2ZFrGsOAcWQ@mail.gmail.com>
 <CAM09pzSZ0Jq8XrpiZ2Q9TRRLXQhkGW1Veop9042ooXMP0E+Xxg@mail.gmail.com>
 <CAP7+vJJRTVB=RqOe7VQvaoWomGnQEhXhCoXjBPYVTEsDMjEakQ@mail.gmail.com>
Message-ID: <CAP1=2W7uxzCEH1=EE7J_y9mFE2xAfS5Oar15AQ9DpKUkdpak0g@mail.gmail.com>

On Fri, 20 May 2016 at 09:46 Guido van Rossum <guido at python.org> wrote:

> ALso, in case anyone reading this didn't realize, this is a classic
> internet debate that has been going on for decades (just not on
> python-dev). Just Google it.
>

Yes, I don't mean to suggest we should change the two space specification
to one-space, just why specify at all to that granularity of formatting?

-Brett


>
> On Fri, May 20, 2016 at 9:40 AM, Facundo Batista
> <facundobatista at gmail.com> wrote:
> > On Fri, May 20, 2016 at 1:27 PM, Brett Cannon <brett at python.org> wrote:
> >
> >> Is there a specific reason for calling out two spaces in comments after
> a
> >> period(I realize it's probably for consistency, but I sure don't ever
> think
> >> about this when I write comment)? Otherwise who actually still writes
> using
> >> two spaces after punctuation? :)
> >
> > I couldn't find it in the reference but AFAIK is something that is
> > also done in ReStructured Text.
> >
> > Regards,
> >
> > --
> > .    Facundo
> >
> > Blog: http://www.taniquetil.com.ar/plog/
> > PyAr: http://www.python.org/ar/
> > Twitter: @facundobatista
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160520/ce1334c6/attachment.html>

From guido at python.org  Fri May 20 14:41:09 2016
From: guido at python.org (Guido van Rossum)
Date: Fri, 20 May 2016 11:41:09 -0700
Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after
 a period?
In-Reply-To: <CAP1=2W7uxzCEH1=EE7J_y9mFE2xAfS5Oar15AQ9DpKUkdpak0g@mail.gmail.com>
References: <CAP1=2W4P0+-pmYaNUYhmbt2HdYk6gT+7Xiq92Lv2ZFrGsOAcWQ@mail.gmail.com>
 <CAM09pzSZ0Jq8XrpiZ2Q9TRRLXQhkGW1Veop9042ooXMP0E+Xxg@mail.gmail.com>
 <CAP7+vJJRTVB=RqOe7VQvaoWomGnQEhXhCoXjBPYVTEsDMjEakQ@mail.gmail.com>
 <CAP1=2W7uxzCEH1=EE7J_y9mFE2xAfS5Oar15AQ9DpKUkdpak0g@mail.gmail.com>
Message-ID: <CAP7+vJKFNoR99NmA7ACrw7NZ+hnh7+yN-1BzuR6HFkt2UgkazA@mail.gmail.com>

Because Emacs. :-) I am going to mute this thread now.

On Fri, May 20, 2016 at 11:40 AM, Brett Cannon <brett at python.org> wrote:
>
>
> On Fri, 20 May 2016 at 09:46 Guido van Rossum <guido at python.org> wrote:
>>
>> ALso, in case anyone reading this didn't realize, this is a classic
>> internet debate that has been going on for decades (just not on
>> python-dev). Just Google it.
>
>
> Yes, I don't mean to suggest we should change the two space specification to
> one-space, just why specify at all to that granularity of formatting?
>
> -Brett
>
>>
>>
>> On Fri, May 20, 2016 at 9:40 AM, Facundo Batista
>> <facundobatista at gmail.com> wrote:
>> > On Fri, May 20, 2016 at 1:27 PM, Brett Cannon <brett at python.org> wrote:
>> >
>> >> Is there a specific reason for calling out two spaces in comments after
>> >> a
>> >> period(I realize it's probably for consistency, but I sure don't ever
>> >> think
>> >> about this when I write comment)? Otherwise who actually still writes
>> >> using
>> >> two spaces after punctuation? :)
>> >
>> > I couldn't find it in the reference but AFAIK is something that is
>> > also done in ReStructured Text.
>> >
>> > Regards,
>> >
>> > --
>> > .    Facundo
>> >
>> > Blog: http://www.taniquetil.com.ar/plog/
>> > PyAr: http://www.python.org/ar/
>> > Twitter: @facundobatista
>> > _______________________________________________
>> > Python-Dev mailing list
>> > Python-Dev at python.org
>> > https://mail.python.org/mailman/listinfo/python-dev
>> > Unsubscribe:
>> > https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)



-- 
--Guido van Rossum (python.org/~guido)

From brett at python.org  Fri May 20 14:42:06 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 20 May 2016 18:42:06 +0000
Subject: [Python-Dev] Removing the provisional label from pathlib
In-Reply-To: <CAP7+vJJu8vpEVDGqmLxdGvmZeHw0s0XSb2MPCYRL1AyRAQLJFQ@mail.gmail.com>
References: <CAP1=2W7pV0joOcoA4Z1RgsrWjgJFjb_JOzJ83pk6cdRjej9_Eg@mail.gmail.com>
 <CAP7+vJJu8vpEVDGqmLxdGvmZeHw0s0XSb2MPCYRL1AyRAQLJFQ@mail.gmail.com>
Message-ID: <CAP1=2W73_-g9B+NttA_GpWkyFaNnpovu2dxpaxi3DoXyiiO16Q@mail.gmail.com>

On Fri, 20 May 2016 at 09:56 Guido van Rossum <guido at python.org> wrote:

> Let's start in 3.6 with all this. I added path to 3.4 because I didn't
> realize it was in security-mode only. I've now undone all my work
> there. Let's not disturb it again, not even its docs.
>
> I don't think there's an "upstream" repo for pathlib (like there still
> is for asyncio) and I don't think there's much of a point in
> supporting __fspath__ in pathlib if there's no os.fspath(). It would
> only encourage hackery in apps that want to play with __fspath__.
>

WFM. I'll let 3.4 and 3.5 just stay as they are and make all PEP 519
changes a 3.6 thing.

I'll update the implementation task list shortly.

-Brett


>
> On Fri, May 20, 2016 at 9:43 AM, Brett Cannon <brett at python.org> wrote:
> > Three questions:
> >
> > Should pathlib gain __fspath__() all the way back to 3.4?
> > Should pathlib's constructor support __fspath__() all the way back to
> 3.4?
> > (separate question as os.fspath() will only be in 3.6; and if we backport
> > I'm not looking forward to making Typeshed happy w/o os.PathLike being
> > available only in 3.6 :/)
> > Should the docs from 3.4 and forward reflect the removal of the
> provisional
> > status? (I assume yes, but wanted to double-check)
> >
> > And a quick thanks to Guido for removing `path` from pathlib for me
> already.
> > :)
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/guido%40python.org
> >
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160520/ecb6e755/attachment.html>

From victor.stinner at gmail.com  Fri May 20 15:15:18 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 20 May 2016 21:15:18 +0200
Subject: [Python-Dev] Removing the provisional label from pathlib
In-Reply-To: <CAP7+vJJu8vpEVDGqmLxdGvmZeHw0s0XSb2MPCYRL1AyRAQLJFQ@mail.gmail.com>
References: <CAP1=2W7pV0joOcoA4Z1RgsrWjgJFjb_JOzJ83pk6cdRjej9_Eg@mail.gmail.com>
 <CAP7+vJJu8vpEVDGqmLxdGvmZeHw0s0XSb2MPCYRL1AyRAQLJFQ@mail.gmail.com>
Message-ID: <CAMpsgwZLNRz=fWGaOnBFVfFff5tLarMGtXnQ7SKtp-7P8AhtVA@mail.gmail.com>

2016-05-20 18:56 GMT+02:00 Guido van Rossum <guido at python.org>:
> Let's start in 3.6 with all this. I added path to 3.4 because I didn't
> realize it was in security-mode only.

I also had to ask the question to myself about branches, that's why I
wrote this table ;-)
https://docs.python.org/devguide/#status-of-python-branches

By the way, is it still up to date? Python 3.2 end-of-line is
documented as 2016-02-20, so its status should be end-of-life, no?

Georg Brandl scheduled a 3.2.7 release at the end of February 2016,
but I don't see it on python.org and I don't recall its announcement.
https://mail.python.org/pipermail/python-dev/2016-February/143300.html

Georg: any update on the Python 3.2 last release?

Victor

From pjenvey at underboss.org  Fri May 20 15:21:15 2016
From: pjenvey at underboss.org (Philip Jenvey)
Date: Fri, 20 May 2016 12:21:15 -0700
Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after
 a period?
In-Reply-To: <CAP7+vJJRTVB=RqOe7VQvaoWomGnQEhXhCoXjBPYVTEsDMjEakQ@mail.gmail.com>
References: <CAP1=2W4P0+-pmYaNUYhmbt2HdYk6gT+7Xiq92Lv2ZFrGsOAcWQ@mail.gmail.com>
 <CAM09pzSZ0Jq8XrpiZ2Q9TRRLXQhkGW1Veop9042ooXMP0E+Xxg@mail.gmail.com>
 <CAP7+vJJRTVB=RqOe7VQvaoWomGnQEhXhCoXjBPYVTEsDMjEakQ@mail.gmail.com>
Message-ID: <00F1D709-27D0-48C4-8CDC-DAC7D70CAD83@underboss.org>

So is line breaks before or after operators <ducks>.

--
Philip Jenvey

> On May 20, 2016, at 9:46 AM, Guido van Rossum <guido at python.org> wrote:
> 
> ALso, in case anyone reading this didn't realize, this is a classic
> internet debate that has been going on for decades (just not on
> python-dev). Just Google it.
> 
> On Fri, May 20, 2016 at 9:40 AM, Facundo Batista
> <facundobatista at gmail.com> wrote:
>> On Fri, May 20, 2016 at 1:27 PM, Brett Cannon <brett at python.org> wrote:
>> 
>>> Is there a specific reason for calling out two spaces in comments after a
>>> period(I realize it's probably for consistency, but I sure don't ever think
>>> about this when I write comment)? Otherwise who actually still writes using
>>> two spaces after punctuation? :)
>> 
>> I couldn't find it in the reference but AFAIK is something that is
>> also done in ReStructured Text.
>> 
>> Regards,
>> 
>> --
>> .    Facundo
>> 
>> Blog: http://www.taniquetil.com.ar/plog/
>> PyAr: http://www.python.org/ar/
>> Twitter: @facundobatista
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/pjenvey%40underboss.org


From g.brandl at gmx.net  Fri May 20 15:18:37 2016
From: g.brandl at gmx.net (Georg Brandl)
Date: Fri, 20 May 2016 21:18:37 +0200
Subject: [Python-Dev] Removing the provisional label from pathlib
In-Reply-To: <CAMpsgwZLNRz=fWGaOnBFVfFff5tLarMGtXnQ7SKtp-7P8AhtVA@mail.gmail.com>
References: <CAP1=2W7pV0joOcoA4Z1RgsrWjgJFjb_JOzJ83pk6cdRjej9_Eg@mail.gmail.com>
 <CAP7+vJJu8vpEVDGqmLxdGvmZeHw0s0XSb2MPCYRL1AyRAQLJFQ@mail.gmail.com>
 <CAMpsgwZLNRz=fWGaOnBFVfFff5tLarMGtXnQ7SKtp-7P8AhtVA@mail.gmail.com>
Message-ID: <cdd9cdf3-ad18-a8b4-9054-b027fc7acfc2@gmx.net>

On 05/20/2016 09:15 PM, Victor Stinner wrote:
> 2016-05-20 18:56 GMT+02:00 Guido van Rossum <guido at python.org>:
>> Let's start in 3.6 with all this. I added path to 3.4 because I didn't
>> realize it was in security-mode only.
> 
> I also had to ask the question to myself about branches, that's why I
> wrote this table ;-)
> https://docs.python.org/devguide/#status-of-python-branches
> 
> By the way, is it still up to date? Python 3.2 end-of-line is
> documented as 2016-02-20, so its status should be end-of-life, no?
> 
> Georg Brandl scheduled a 3.2.7 release at the end of February 2016,
> but I don't see it on python.org and I don't recall its announcement.
> https://mail.python.org/pipermail/python-dev/2016-February/143300.html
> 
> Georg: any update on the Python 3.2 last release?

Still waiting for some last security update. I sent another mail to
python-dev that announced the delay, I think.

Georg

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160520/8fec1d8c/attachment.sig>

From stephen at xemacs.org  Sat May 21 03:44:20 2016
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 21 May 2016 16:44:20 +0900
Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after
 a period?
In-Reply-To: <573F49A0.8090404@gmail.com>
References: <CAP1=2W4P0+-pmYaNUYhmbt2HdYk6gT+7Xiq92Lv2ZFrGsOAcWQ@mail.gmail.com>
 <573F3F10.4000408@stoneleaf.us> <573F49A0.8090404@gmail.com>
Message-ID: <22336.4564.834078.187823@turnbull.sk.tsukuba.ac.jp>

Bernardo Sulzbach writes:
 > > On 05/20/2016 09:27 AM, Brett Cannon wrote:
 > >
 > > The period is a full-stop punctuation mark, but visually obscure [1].
 > 
 > It is small. This does not make it hard to notice, however.

That depends on fonts, and on the ability of the user to write
correctly as well.  "Syntax should not look like grit on Tim's screen"
applies to natural language as well.  I read a lot of English-as-a-
second-language text, and it really does help to have the extra space,
because the period often appears where English grammar suggests it
doesn't belong.  The extra effort to use the second space helps to
emphasize the author's intent, and even subjectively make the period
"more visible".

But I agree with Guido.  No Python contributors write poorly, so the
real issues are consistency with past practice (a minor consideration
in the case of comments) and tooling (since both Emacs and vim seem to
come out of the box believing in this rule, I guess it's widespread
enough to care).

As for Brett's concern about whether this should be specified at this
level, I think tooling again is an answer.  Emacs reflows whole
paragraphs.  That means that text that is off-screen may be changed
according to this rule, and that is an annoyance when you only notice
it while preparing a patch for submission.  I think a detail of style
that triggers unwanted (but plausible[1]) behavior in a widely-used
tool is worth standardizing.  Sometimes, of course, you standardize
*despite* the tool, but then at least the tool-users know that a
decision has been taken to impose the burden on them, vs. imposing the
burden on people who prefer the other style.

Footnotes: 
[1]  If the behavior is implausible, fix the tool, of course!


From stephen at xemacs.org  Sat May 21 03:49:11 2016
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 21 May 2016 16:49:11 +0900
Subject: [Python-Dev] Adding Type[C] to PEP 484
In-Reply-To: <573EED58.3040508@mail.de>
References: <CAP7+vJL+mpL=N31idm0SGWxAeChJ3eDWiedbYL0m93Ucs3PR6g@mail.gmail.com>
 <CAP1=2W4+gS3Cz7pUkSDcQEF49zJLtc4Zak8YX2iSrPT9U-OFAQ@mail.gmail.com>
 <CACsRUKLyWjCFWP6UtNZf7+NsJ0JdHSVFMiqk2ZmS7d7qj7XmSw@mail.gmail.com>
 <CAP7+vJLYPR1c+ZkB3Z9U57X8vktVn7czA6p4Xi6cd_CjnBDofg@mail.gmail.com>
 <CACsRUKLNXnsUCNeC4g0j2yneqSNhNf4NVsyhRJgTegVg3srmZg@mail.gmail.com>
 <CAP7+vJJ=_w7Grc_RiG18TQtKhDFYicqxULx4pnfX2UiX6MdPdQ@mail.gmail.com>
 <CACsRUKJRP0g_g3jtFEmTJmX2V-f74aH0mtdTByw4V-4odUmWrA@mail.gmail.com>
 <CAP7+vJJVnEHYLfFfmRFxYRQbcZE4RLSwT6-JMLdtfkRyXDyNeA@mail.gmail.com>
 <573EED58.3040508@mail.de>
Message-ID: <22336.4855.445392.833399@turnbull.sk.tsukuba.ac.jp>

Sven R. Kunze writes:

 > Type[A, B] reminds me of isinstance(obj, (A, B)).

Sure, but Guido already invoked EIBTI.

For one (obscure) alternative, Type[A, B] reminded me of Fun(A,B) (the
functors from category A to category B), and in Python I would tend to
map that to a function type (taking argument from A, returning value
from B).  Another (probably more plausible to Pythonistas) would be a
type that *mixes* A and B, ie, one might be reminded of

    class C(A, B):
        pass

    isinstance(obj, C)

or even (continuing the example)

    class E(B, A, D):        # Order of A, B irrelevant!  Yikes!
        pass

    isinstance(obj, (C, E))

Regards,

From pludemann at google.com  Sun May 22 11:15:12 2016
From: pludemann at google.com (Peter Ludemann)
Date: Sun, 22 May 2016 08:15:12 -0700
Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after
 a period?
In-Reply-To: <22336.4564.834078.187823@turnbull.sk.tsukuba.ac.jp>
References: <CAP1=2W4P0+-pmYaNUYhmbt2HdYk6gT+7Xiq92Lv2ZFrGsOAcWQ@mail.gmail.com>
 <573F3F10.4000408@stoneleaf.us> <573F49A0.8090404@gmail.com>
 <22336.4564.834078.187823@turnbull.sk.tsukuba.ac.jp>
Message-ID: <CACsRUKLViNskDT=_LpWKYnRJ8wd+32X67x7kc5j=uM1V1C0nKw@mail.gmail.com>

I just tried a re-flow in emacs (M-q or fill-paragraph) and it didn't
change my single space to two (it also changed 3 spaces after a period to
2; and it preserved my hanging indents). I have no idea what vim does.
(OTOH, emacs annoyingly adds a blank line to my """ docstrings, which I
remove manually.)

I'm not going to stoke the fires of debate (especially as I've worked with
typographers over the years and know that the typographical sins of
programmers go far beyond the double-space-after-period), except to note
that if you read the thousands of words that have been written about
spacing after periods, they're usually talking about *proportional* fonts;
and we still tend to use monospaced fonts with programming tools.


On 21 May 2016 at 00:44, Stephen J. Turnbull <stephen at xemacs.org> wrote:

> Bernardo Sulzbach writes:
>  > > On 05/20/2016 09:27 AM, Brett Cannon wrote:
>  > >
>  > > The period is a full-stop punctuation mark, but visually obscure [1].
>  >
>  > It is small. This does not make it hard to notice, however.
>
> That depends on fonts, and on the ability of the user to write
> correctly as well.  "Syntax should not look like grit on Tim's screen"
> applies to natural language as well.  I read a lot of English-as-a-
> second-language text, and it really does help to have the extra space,
> because the period often appears where English grammar suggests it
> doesn't belong.  The extra effort to use the second space helps to
> emphasize the author's intent, and even subjectively make the period
> "more visible".
>
> But I agree with Guido.  No Python contributors write poorly, so the
> real issues are consistency with past practice (a minor consideration
> in the case of comments) and tooling (since both Emacs and vim seem to
> come out of the box believing in this rule, I guess it's widespread
> enough to care).
>
> As for Brett's concern about whether this should be specified at this
> level, I think tooling again is an answer.  Emacs reflows whole
> paragraphs.  That means that text that is off-screen may be changed
> according to this rule, and that is an annoyance when you only notice
> it while preparing a patch for submission.  I think a detail of style
> that triggers unwanted (but plausible[1]) behavior in a widely-used
> tool is worth standardizing.  Sometimes, of course, you standardize
> *despite* the tool, but then at least the tool-users know that a
> decision has been taken to impose the burden on them, vs. imposing the
> burden on people who prefer the other style.
>
> Footnotes:
> [1]  If the behavior is implausible, fix the tool, of course!
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/pludemann%40google.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160522/2d4505e8/attachment.html>

From random832 at fastmail.com  Sun May 22 14:01:44 2016
From: random832 at fastmail.com (Random832)
Date: Sun, 22 May 2016 14:01:44 -0400
Subject: [Python-Dev] Why does PEP 7/8 explicitly suggest 2 spaces after
 a period?
In-Reply-To: <CACsRUKLViNskDT=_LpWKYnRJ8wd+32X67x7kc5j=uM1V1C0nKw@mail.gmail.com>
References: <CAP1=2W4P0+-pmYaNUYhmbt2HdYk6gT+7Xiq92Lv2ZFrGsOAcWQ@mail.gmail.com>
 <573F3F10.4000408@stoneleaf.us> <573F49A0.8090404@gmail.com>
 <22336.4564.834078.187823@turnbull.sk.tsukuba.ac.jp>
 <CACsRUKLViNskDT=_LpWKYnRJ8wd+32X67x7kc5j=uM1V1C0nKw@mail.gmail.com>
Message-ID: <1463940104.1833604.615256001.4820654B@webmail.messagingengine.com>

On Sun, May 22, 2016, at 11:15, Peter Ludemann via Python-Dev wrote:
> I just tried a re-flow in emacs (M-q or fill-paragraph) and it didn't
> change my single space to two (it also changed 3 spaces after a period
> to 2; and it preserved my hanging indents).

It won't change a single space to two, but it will add two spaces after
a sentence which was previously at the end of a line... and it *won't*
break a line after a dot followed by a single space (Vim isn't clever
enough for this second bit), preferring to break before the preceding
word instead.

A dot followed by a single space is *permitted* (hence why you observed
it didn't change it to two), it is simply assumed to represent something
other than the end of a sentence, such as an initial in a name.

From chris.barker at noaa.gov  Mon May 23 12:55:04 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 23 May 2016 09:55:04 -0700
Subject: [Python-Dev] Removing the provisional label from pathlib
In-Reply-To: <CAP1=2W73_-g9B+NttA_GpWkyFaNnpovu2dxpaxi3DoXyiiO16Q@mail.gmail.com>
References: <CAP1=2W7pV0joOcoA4Z1RgsrWjgJFjb_JOzJ83pk6cdRjej9_Eg@mail.gmail.com>
 <CAP7+vJJu8vpEVDGqmLxdGvmZeHw0s0XSb2MPCYRL1AyRAQLJFQ@mail.gmail.com>
 <CAP1=2W73_-g9B+NttA_GpWkyFaNnpovu2dxpaxi3DoXyiiO16Q@mail.gmail.com>
Message-ID: <CALGmxEKg+kg63nBwLD0Jkh9d_TGzN3B=NW_Px6VG+Aauge5h+g@mail.gmail.com>

On Fri, May 20, 2016 at 11:42 AM, Brett Cannon <brett at python.org> wrote:

>
> WFM. I'll let 3.4 and 3.5 just stay as they are and make all PEP 519
> changes a 3.6 thing.
>

I'd like to see it in 3.5 if we could do that -- that'll be available for
operational code a lot sooner than 3.6, and it's still in active
maintenance mode, yes?

-CHB




> I'll update the implementation task list shortly.
>
> -Brett
>
>
>>
>> On Fri, May 20, 2016 at 9:43 AM, Brett Cannon <brett at python.org> wrote:
>> > Three questions:
>> >
>> > Should pathlib gain __fspath__() all the way back to 3.4?
>> > Should pathlib's constructor support __fspath__() all the way back to
>> 3.4?
>> > (separate question as os.fspath() will only be in 3.6; and if we
>> backport
>> > I'm not looking forward to making Typeshed happy w/o os.PathLike being
>> > available only in 3.6 :/)
>> > Should the docs from 3.4 and forward reflect the removal of the
>> provisional
>> > status? (I assume yes, but wanted to double-check)
>> >
>> > And a quick thanks to Guido for removing `path` from pathlib for me
>> already.
>> > :)
>> >
>> > _______________________________________________
>> > Python-Dev mailing list
>> > Python-Dev at python.org
>> > https://mail.python.org/mailman/listinfo/python-dev
>> > Unsubscribe:
>> > https://mail.python.org/mailman/options/python-dev/guido%40python.org
>> >
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160523/f7aa2401/attachment.html>

From brett at python.org  Mon May 23 13:13:51 2016
From: brett at python.org (Brett Cannon)
Date: Mon, 23 May 2016 17:13:51 +0000
Subject: [Python-Dev] Removing the provisional label from pathlib
In-Reply-To: <CALGmxEKg+kg63nBwLD0Jkh9d_TGzN3B=NW_Px6VG+Aauge5h+g@mail.gmail.com>
References: <CAP1=2W7pV0joOcoA4Z1RgsrWjgJFjb_JOzJ83pk6cdRjej9_Eg@mail.gmail.com>
 <CAP7+vJJu8vpEVDGqmLxdGvmZeHw0s0XSb2MPCYRL1AyRAQLJFQ@mail.gmail.com>
 <CAP1=2W73_-g9B+NttA_GpWkyFaNnpovu2dxpaxi3DoXyiiO16Q@mail.gmail.com>
 <CALGmxEKg+kg63nBwLD0Jkh9d_TGzN3B=NW_Px6VG+Aauge5h+g@mail.gmail.com>
Message-ID: <CAP1=2W6hPfMjzjwVK3MFK1rbGpb3=kQHAnBw27PrHs6fBa0WOw@mail.gmail.com>

On Mon, May 23, 2016, 09:55 Chris Barker <chris.barker at noaa.gov> wrote:

> On Fri, May 20, 2016 at 11:42 AM, Brett Cannon <brett at python.org> wrote:
>
>>
>> WFM. I'll let 3.4 and 3.5 just stay as they are and make all PEP 519
>> changes a 3.6 thing.
>>
>
> I'd like to see it in 3.5 if we could do that -- that'll be available for
> operational code a lot sooner than 3.6, and it's still in active
> maintenance mode, yes?
>

3.5 is still getting bugfixes:
https://docs.python.org/devguide/#status-of-python-branches

As for backporting __fspath__() for pathlib, you can easily write your own
subclass that adds it. And since the stdlib won't be updated in 3.5 for
consumption of fspath you're only missing out on production which your
subclass can take care of.

-brett



> -CHB
>
>
>
>
>> I'll update the implementation task list shortly.
>>
>> -Brett
>>
>>
>>>
>>> On Fri, May 20, 2016 at 9:43 AM, Brett Cannon <brett at python.org> wrote:
>>> > Three questions:
>>> >
>>> > Should pathlib gain __fspath__() all the way back to 3.4?
>>> > Should pathlib's constructor support __fspath__() all the way back to
>>> 3.4?
>>> > (separate question as os.fspath() will only be in 3.6; and if we
>>> backport
>>> > I'm not looking forward to making Typeshed happy w/o os.PathLike being
>>> > available only in 3.6 :/)
>>> > Should the docs from 3.4 and forward reflect the removal of the
>>> provisional
>>> > status? (I assume yes, but wanted to double-check)
>>> >
>>> > And a quick thanks to Guido for removing `path` from pathlib for me
>>> already.
>>> > :)
>>> >
>>> > _______________________________________________
>>> > Python-Dev mailing list
>>> > Python-Dev at python.org
>>> > https://mail.python.org/mailman/listinfo/python-dev
>>> > Unsubscribe:
>>> > https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>> >
>>>
>>>
>>>
>>> --
>>> --Guido van Rossum (python.org/~guido)
>>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>>
> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
>>
>>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160523/7d3c178d/attachment.html>

From chris.barker at noaa.gov  Mon May 23 15:38:53 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 23 May 2016 12:38:53 -0700
Subject: [Python-Dev] Removing the provisional label from pathlib
In-Reply-To: <CAP1=2W6hPfMjzjwVK3MFK1rbGpb3=kQHAnBw27PrHs6fBa0WOw@mail.gmail.com>
References: <CAP1=2W7pV0joOcoA4Z1RgsrWjgJFjb_JOzJ83pk6cdRjej9_Eg@mail.gmail.com>
 <CAP7+vJJu8vpEVDGqmLxdGvmZeHw0s0XSb2MPCYRL1AyRAQLJFQ@mail.gmail.com>
 <CAP1=2W73_-g9B+NttA_GpWkyFaNnpovu2dxpaxi3DoXyiiO16Q@mail.gmail.com>
 <CALGmxEKg+kg63nBwLD0Jkh9d_TGzN3B=NW_Px6VG+Aauge5h+g@mail.gmail.com>
 <CAP1=2W6hPfMjzjwVK3MFK1rbGpb3=kQHAnBw27PrHs6fBa0WOw@mail.gmail.com>
Message-ID: <CALGmxELA0D9PcrqasMRJ_9M_hPuv6VcWzYoBjnHV40sEZwidAg@mail.gmail.com>

On Mon, May 23, 2016 at 10:13 AM, Brett Cannon <brett at python.org> wrote:

> 3.5 is still getting bugfixes:
> https://docs.python.org/devguide/#status-of-python-branches
>
> As for backporting __fspath__() for pathlib, you can easily write your own
> subclass that adds it. And since the stdlib won't be updated in 3.5 for
> consumption of fspath
>

OK -- when I said "it", I meant the whole shebang -- i.e. the stdlib stuff
too. But fair enough, we can't be backporting everything, and I that would
be touching a lot of the lib.

-CHB



> you're only missing out on production which your subclass can take care of.
>
> -brett
>
>
>
>> -CHB
>>
>>
>>
>>
>>> I'll update the implementation task list shortly.
>>>
>>> -Brett
>>>
>>>
>>>>
>>>> On Fri, May 20, 2016 at 9:43 AM, Brett Cannon <brett at python.org> wrote:
>>>> > Three questions:
>>>> >
>>>> > Should pathlib gain __fspath__() all the way back to 3.4?
>>>> > Should pathlib's constructor support __fspath__() all the way back to
>>>> 3.4?
>>>> > (separate question as os.fspath() will only be in 3.6; and if we
>>>> backport
>>>> > I'm not looking forward to making Typeshed happy w/o os.PathLike being
>>>> > available only in 3.6 :/)
>>>> > Should the docs from 3.4 and forward reflect the removal of the
>>>> provisional
>>>> > status? (I assume yes, but wanted to double-check)
>>>> >
>>>> > And a quick thanks to Guido for removing `path` from pathlib for me
>>>> already.
>>>> > :)
>>>> >
>>>> > _______________________________________________
>>>> > Python-Dev mailing list
>>>> > Python-Dev at python.org
>>>> > https://mail.python.org/mailman/listinfo/python-dev
>>>> > Unsubscribe:
>>>> > https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> --Guido van Rossum (python.org/~guido)
>>>>
>>>
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>>
>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
>>>
>>>
>>
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR&R            (206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115       (206) 526-6317   main reception
>>
>> Chris.Barker at noaa.gov
>>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160523/f855f0c7/attachment.html>

From chris.barker at noaa.gov  Mon May 23 21:36:06 2016
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Mon, 23 May 2016 18:36:06 -0700
Subject: [Python-Dev] Python 3.6.0a1 is now available
In-Reply-To: <28E1448B-F604-48EB-B4E4-0AE2528C8383@python.org>
References: <28E1448B-F604-48EB-B4E4-0AE2528C8383@python.org>
Message-ID: <-7150763876064620774@unknownmsgid>

Ned,
It looks like you're still building for OS-X v. 10.6.

I can't find an official statement, but this:
http://www.computerworld.com/article/2950580/operating-systems/the-end-is-near-for-os-x-mountain-lion-support.html

Indicates that 10.8 is falling off Apple's support.

Maybe it's time for a newer baseline.

-Chris

> On May 17, 2016, at 2:18 PM, Ned Deily <nad at python.org> wrote:
>
> On behalf of the Python development community and the Python 3.6 release
> team, I'm happy to announce the availability of Python 3.6.0a1.
> 3.6.0a1 is the first of four planned alpha releases of Python 3.6,
> the next major release of Python.  During the alpha phase, Python 3.6
> remains under heavy development: additional features will be added
> and existing features may be modified or deleted.  Please keep in mind
> that this is a preview release and its use is not recommended for
> production environments.
>
> You can find Python 3.6.0a1 here:
>
> https://www.python.org/downloads/release/python-360a1/
>
> The next release of Python 3.6 will be 3.6.0a2, currently scheduled for
> 2016-06-13.
>
> Enjoy!
>
> --Ned
>
> --
>  Ned Deily
>  nad at python.org -- []
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov

From k7hoven at gmail.com  Tue May 24 07:03:34 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Tue, 24 May 2016 14:03:34 +0300
Subject: [Python-Dev] Removing the provisional label from pathlib
In-Reply-To: <CALGmxELA0D9PcrqasMRJ_9M_hPuv6VcWzYoBjnHV40sEZwidAg@mail.gmail.com>
References: <CAP1=2W7pV0joOcoA4Z1RgsrWjgJFjb_JOzJ83pk6cdRjej9_Eg@mail.gmail.com>
 <CAP7+vJJu8vpEVDGqmLxdGvmZeHw0s0XSb2MPCYRL1AyRAQLJFQ@mail.gmail.com>
 <CAP1=2W73_-g9B+NttA_GpWkyFaNnpovu2dxpaxi3DoXyiiO16Q@mail.gmail.com>
 <CALGmxEKg+kg63nBwLD0Jkh9d_TGzN3B=NW_Px6VG+Aauge5h+g@mail.gmail.com>
 <CAP1=2W6hPfMjzjwVK3MFK1rbGpb3=kQHAnBw27PrHs6fBa0WOw@mail.gmail.com>
 <CALGmxELA0D9PcrqasMRJ_9M_hPuv6VcWzYoBjnHV40sEZwidAg@mail.gmail.com>
Message-ID: <CAMiohoie6O04NDm-UOby+tKxdT7zuB4gEu1WTAX86iTFO8-HRg@mail.gmail.com>

On Mon, May 23, 2016 at 10:38 PM, Chris Barker <chris.barker at noaa.gov> wrote:
> On Mon, May 23, 2016 at 10:13 AM, Brett Cannon <brett at python.org> wrote:
>>
>> 3.5 is still getting bugfixes:
>> https://docs.python.org/devguide/#status-of-python-branches
>>
>> As for backporting __fspath__() for pathlib, you can easily write your own
>> subclass that adds it. And since the stdlib won't be updated in 3.5 for
>> consumption of fspath
>
>
> OK -- when I said "it", I meant the whole shebang -- i.e. the stdlib stuff
> too. But fair enough, we can't be backporting everything, and I that would
> be touching a lot of the lib.
>
> -CHB

I guess we might consider adding __fspath__ in maintenance releases,
and make open() support it? That would cover a significant share of
use cases, although it might be weird if code written for 3.5.2
doesn't run on 3.5.1... So maybe just 3.6-> ?

I'm not quite as busy as last week, so I might consider working on the
stdlib changes if I find the time. Or is someone already working on
this?

-- Koos

From barry at python.org  Tue May 24 09:56:51 2016
From: barry at python.org (Barry Warsaw)
Date: Tue, 24 May 2016 09:56:51 -0400
Subject: [Python-Dev] Removing the provisional label from pathlib
In-Reply-To: <CAMiohoie6O04NDm-UOby+tKxdT7zuB4gEu1WTAX86iTFO8-HRg@mail.gmail.com>
References: <CAP1=2W7pV0joOcoA4Z1RgsrWjgJFjb_JOzJ83pk6cdRjej9_Eg@mail.gmail.com>
 <CAP7+vJJu8vpEVDGqmLxdGvmZeHw0s0XSb2MPCYRL1AyRAQLJFQ@mail.gmail.com>
 <CAP1=2W73_-g9B+NttA_GpWkyFaNnpovu2dxpaxi3DoXyiiO16Q@mail.gmail.com>
 <CALGmxEKg+kg63nBwLD0Jkh9d_TGzN3B=NW_Px6VG+Aauge5h+g@mail.gmail.com>
 <CAP1=2W6hPfMjzjwVK3MFK1rbGpb3=kQHAnBw27PrHs6fBa0WOw@mail.gmail.com>
 <CALGmxELA0D9PcrqasMRJ_9M_hPuv6VcWzYoBjnHV40sEZwidAg@mail.gmail.com>
 <CAMiohoie6O04NDm-UOby+tKxdT7zuB4gEu1WTAX86iTFO8-HRg@mail.gmail.com>
Message-ID: <20160524095651.014e37c4.barry@wooz.org>

On May 24, 2016, at 02:03 PM, Koos Zevenhoven wrote:

>I guess we might consider adding __fspath__ in maintenance releases,
>and make open() support it? That would cover a significant share of
>use cases, although it might be weird if code written for 3.5.2
>doesn't run on 3.5.1...

Please, no.  We learned that lesson in Python 2.2.1 with True/False.

Cheers,
-Barry

From k7hoven at gmail.com  Tue May 24 10:11:46 2016
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Tue, 24 May 2016 17:11:46 +0300
Subject: [Python-Dev] Removing the provisional label from pathlib
In-Reply-To: <20160524095651.014e37c4.barry@wooz.org>
References: <CAP1=2W7pV0joOcoA4Z1RgsrWjgJFjb_JOzJ83pk6cdRjej9_Eg@mail.gmail.com>
 <CAP7+vJJu8vpEVDGqmLxdGvmZeHw0s0XSb2MPCYRL1AyRAQLJFQ@mail.gmail.com>
 <CAP1=2W73_-g9B+NttA_GpWkyFaNnpovu2dxpaxi3DoXyiiO16Q@mail.gmail.com>
 <CALGmxEKg+kg63nBwLD0Jkh9d_TGzN3B=NW_Px6VG+Aauge5h+g@mail.gmail.com>
 <CAP1=2W6hPfMjzjwVK3MFK1rbGpb3=kQHAnBw27PrHs6fBa0WOw@mail.gmail.com>
 <CALGmxELA0D9PcrqasMRJ_9M_hPuv6VcWzYoBjnHV40sEZwidAg@mail.gmail.com>
 <CAMiohoie6O04NDm-UOby+tKxdT7zuB4gEu1WTAX86iTFO8-HRg@mail.gmail.com>
 <20160524095651.014e37c4.barry@wooz.org>
Message-ID: <CAMiohogjJvjAs+X4t5LQCNv2LU5pDcBzHBPOpJvJ8uDYRP_AYg@mail.gmail.com>

On Tue, May 24, 2016 at 4:56 PM, Barry Warsaw <barry at python.org> wrote:
> On May 24, 2016, at 02:03 PM, Koos Zevenhoven wrote:
>
>>I guess we might consider adding __fspath__ in maintenance releases,
>>and make open() support it? That would cover a significant share of
>>use cases, although it might be weird if code written for 3.5.2
>>doesn't run on 3.5.1...
>
> Please, no.  We learned that lesson in Python 2.2.1 with True/False.

What happened? True was included in 2.2.1 but not False?-). Anyway, I
guess you are probably right, and "3.6->" is the way to go. Besides,
Guido already wrote that in the first response.

-- Koos

> Cheers,
> -Barry
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com

From p.f.moore at gmail.com  Tue May 24 10:49:34 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 24 May 2016 15:49:34 +0100
Subject: [Python-Dev] Removing the provisional label from pathlib
In-Reply-To: <CAMiohogjJvjAs+X4t5LQCNv2LU5pDcBzHBPOpJvJ8uDYRP_AYg@mail.gmail.com>
References: <CAP1=2W7pV0joOcoA4Z1RgsrWjgJFjb_JOzJ83pk6cdRjej9_Eg@mail.gmail.com>
 <CAP7+vJJu8vpEVDGqmLxdGvmZeHw0s0XSb2MPCYRL1AyRAQLJFQ@mail.gmail.com>
 <CAP1=2W73_-g9B+NttA_GpWkyFaNnpovu2dxpaxi3DoXyiiO16Q@mail.gmail.com>
 <CALGmxEKg+kg63nBwLD0Jkh9d_TGzN3B=NW_Px6VG+Aauge5h+g@mail.gmail.com>
 <CAP1=2W6hPfMjzjwVK3MFK1rbGpb3=kQHAnBw27PrHs6fBa0WOw@mail.gmail.com>
 <CALGmxELA0D9PcrqasMRJ_9M_hPuv6VcWzYoBjnHV40sEZwidAg@mail.gmail.com>
 <CAMiohoie6O04NDm-UOby+tKxdT7zuB4gEu1WTAX86iTFO8-HRg@mail.gmail.com>
 <20160524095651.014e37c4.barry@wooz.org>
 <CAMiohogjJvjAs+X4t5LQCNv2LU5pDcBzHBPOpJvJ8uDYRP_AYg@mail.gmail.com>
Message-ID: <CACac1F8oLH-5FjjhLhS6CpP-TJ3x1fkCDpAC9hsw3gAhcwxCEQ@mail.gmail.com>

On 24 May 2016 at 15:11, Koos Zevenhoven <k7hoven at gmail.com> wrote:
>> Please, no.  We learned that lesson in Python 2.2.1 with True/False.
>
> What happened? True was included in 2.2.1 but not False?-). Anyway, I
> guess you are probably right, and "3.6->" is the way to go. Besides,
> Guido already wrote that in the first response.

The history is at
http://python-history.blogspot.co.uk/2013/11/the-history-of-bool-true-and-false.html
but basically we added new *constants* for True/False in a point
release. Some people used them, and their code only worked on 2.2.1
and above but not 2.2.0. Others were using variables called True and
False, and the 2.2.1 change broke their code (you could no longer
assign to those names).

So essentially it ended up that people couldn't say "works on 2.2" and
had to distinguish between 2.2.0 and 2.2.1+.

Python's strict backward compatibility rules basically stem from the
grief caused by that change.

Paul

From brett at snarky.ca  Tue May 24 12:57:18 2016
From: brett at snarky.ca (Brett Cannon)
Date: Tue, 24 May 2016 16:57:18 +0000
Subject: [Python-Dev] [Python-checkins] cpython: Issue #26647: Python
 interpreter now uses 16-bit wordcode instead of bytecode.
In-Reply-To: <20160524061542.28801.84595.29E5EEA7@psf.io>
References: <20160524061542.28801.84595.29E5EEA7@psf.io>
Message-ID: <CAP1=2W5Q5SKSQf6TJ=+P0NpGFfKDnnJJi0PBjPER_a9g7kmJUg@mail.gmail.com>

This probably deserves a "What's New" entry since it does break
bytecode-inspecting code.

On Mon, 23 May 2016 at 23:15 serhiy.storchaka <python-checkins at python.org>
wrote:

> https://hg.python.org/cpython/rev/3a57eafd8401
> changeset:   101486:3a57eafd8401
> user:        Serhiy Storchaka <storchaka at gmail.com>
> date:        Tue May 24 09:15:14 2016 +0300
> summary:
>   Issue #26647: Python interpreter now uses 16-bit wordcode instead of
> bytecode.
> Patch by Demur Rumed.
>
> files:
>   Doc/library/dis.rst                  |     9 +-
>   Lib/ctypes/test/test_values.py       |     6 +-
>   Lib/dis.py                           |    35 +-
>   Lib/importlib/_bootstrap_external.py |     3 +-
>   Lib/test/test_dis.py                 |   512 +-
>   Misc/NEWS                            |     3 +
>   Objects/frameobject.c                |    12 +-
>   Objects/genobject.c                  |     4 +-
>   PC/launcher.c                        |     2 +-
>   PCbuild/pythoncore.vcxproj           |     1 +
>   PCbuild/pythoncore.vcxproj.filters   |     3 +
>   Python/ceval.c                       |   133 +-
>   Python/compile.c                     |    81 +-
>   Python/frozen.c                      |    20 +-
>   Python/importlib.h                   |  3719 +++++-----
>   Python/importlib_external.h          |  4612 ++++++-------
>   Python/peephole.c                    |   578 +-
>   Python/wordcode_helpers.h            |    38 +
>   18 files changed, 4748 insertions(+), 5023 deletions(-)
>
>
> diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst
> --- a/Doc/library/dis.rst
> +++ b/Doc/library/dis.rst
> @@ -31,9 +31,9 @@
>
>     >>> dis.dis(myfunc)
>       2           0 LOAD_GLOBAL              0 (len)
> -                 3 LOAD_FAST                0 (alist)
> -                 6 CALL_FUNCTION            1
> -                 9 RETURN_VALUE
> +                 2 LOAD_FAST                0 (alist)
> +                 4 CALL_FUNCTION            1
> +                 6 RETURN_VALUE
>
>  (The "2" is a line number).
>
> @@ -682,8 +682,7 @@
>     .. XXX explain the WHY stuff!
>
>
> -All of the following opcodes expect arguments.  An argument is two bytes,
> with
> -the more significant byte last.
> +All of the following opcodes use their arguments.
>
>  .. opcode:: STORE_NAME (namei)
>
> diff --git a/Lib/ctypes/test/test_values.py
> b/Lib/ctypes/test/test_values.py
> --- a/Lib/ctypes/test/test_values.py
> +++ b/Lib/ctypes/test/test_values.py
> @@ -79,9 +79,9 @@
>                  continue
>              items.append((entry.name.decode("ascii"), entry.size))
>
> -        expected = [("__hello__", 161),
> -                    ("__phello__", -161),
> -                    ("__phello__.spam", 161),
> +        expected = [("__hello__", 139),
> +                    ("__phello__", -139),
> +                    ("__phello__.spam", 139),
>                      ]
>          self.assertEqual(items, expected, "PyImport_FrozenModules example
> "
>              "in Doc/library/ctypes.rst may be out of date")
> diff --git a/Lib/dis.py b/Lib/dis.py
> --- a/Lib/dis.py
> +++ b/Lib/dis.py
> @@ -285,7 +285,6 @@
>      """
>      labels = findlabels(code)
>      starts_line = None
> -    free = None
>      for offset, op, arg in _unpack_opargs(code):
>          if linestarts is not None:
>              starts_line = linestarts.get(offset, None)
> @@ -296,7 +295,7 @@
>          argrepr = ''
>          if arg is not None:
>              #  Set argval to the dereferenced value of the argument when
> -            #  availabe, and argrepr to the string representation of
> argval.
> +            #  available, and argrepr to the string representation of
> argval.
>              #    _disassemble_bytes needs the string repr of the
>              #    raw name index for LOAD_GLOBAL, LOAD_CONST, etc.
>              argval = arg
> @@ -305,7 +304,7 @@
>              elif op in hasname:
>                  argval, argrepr = _get_name_info(arg, names)
>              elif op in hasjrel:
> -                argval = offset + 3 + arg
> +                argval = offset + 2 + arg
>                  argrepr = "to " + repr(argval)
>              elif op in haslocal:
>                  argval, argrepr = _get_name_info(arg, varnames)
> @@ -352,23 +351,15 @@
>  disco = disassemble                     # XXX For backwards compatibility
>
>  def _unpack_opargs(code):
> -    # enumerate() is not an option, since we sometimes process
> -    # multiple elements on a single pass through the loop
>      extended_arg = 0
> -    n = len(code)
> -    i = 0
> -    while i < n:
> +    for i in range(0, len(code), 2):
>          op = code[i]
> -        offset = i
> -        i = i+1
> -        arg = None
>          if op >= HAVE_ARGUMENT:
> -            arg = code[i] + code[i+1]*256 + extended_arg
> -            extended_arg = 0
> -            i = i+2
> -            if op == EXTENDED_ARG:
> -                extended_arg = arg*65536
> -        yield (offset, op, arg)
> +            arg = code[i+1] | extended_arg
> +            extended_arg = (arg << 8) if op == EXTENDED_ARG else 0
> +        else:
> +            arg = None
> +        yield (i, op, arg)
>
>  def findlabels(code):
>      """Detect all offsets in a byte code which are jump targets.
> @@ -379,14 +370,14 @@
>      labels = []
>      for offset, op, arg in _unpack_opargs(code):
>          if arg is not None:
> -            label = -1
>              if op in hasjrel:
> -                label = offset + 3 + arg
> +                label = offset + 2 + arg
>              elif op in hasjabs:
>                  label = arg
> -            if label >= 0:
> -                if label not in labels:
> -                    labels.append(label)
> +            else:
> +                continue
> +            if label not in labels:
> +                labels.append(label)
>      return labels
>
>  def findlinestarts(code):
> diff --git a/Lib/importlib/_bootstrap_external.py
> b/Lib/importlib/_bootstrap_external.py
> --- a/Lib/importlib/_bootstrap_external.py
> +++ b/Lib/importlib/_bootstrap_external.py
> @@ -225,6 +225,7 @@
>  #     Python 3.5b2  3350 (add GET_YIELD_FROM_ITER opcode #24400)
>  #     Python 3.6a0  3360 (add FORMAT_VALUE opcode #25483
>  #     Python 3.6a0  3361 (lineno delta of code.co_lnotab becomes signed)
> +#     Python 3.6a0  3370 (16 bit wordcode)
>  #
>  # MAGIC must change whenever the bytecode emitted by the compiler may no
>  # longer be understood by older implementations of the eval loop (usually
> @@ -233,7 +234,7 @@
>  # Whenever MAGIC_NUMBER is changed, the ranges in the magic_values array
>  # in PC/launcher.c must also be updated.
>
> -MAGIC_NUMBER = (3361).to_bytes(2, 'little') + b'\r\n'
> +MAGIC_NUMBER = (3370).to_bytes(2, 'little') + b'\r\n'
>  _RAW_MAGIC_NUMBER = int.from_bytes(MAGIC_NUMBER, 'little')  # For import.c
>
>  _PYCACHE = '__pycache__'
> diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py
> --- a/Lib/test/test_dis.py
> +++ b/Lib/test/test_dis.py
> @@ -40,41 +40,41 @@
>
>  dis_c_instance_method = """\
>  %3d           0 LOAD_FAST                1 (x)
> -              3 LOAD_CONST               1 (1)
> -              6 COMPARE_OP               2 (==)
> -              9 LOAD_FAST                0 (self)
> -             12 STORE_ATTR               0 (x)
> -             15 LOAD_CONST               0 (None)
> -             18 RETURN_VALUE
> +              2 LOAD_CONST               1 (1)
> +              4 COMPARE_OP               2 (==)
> +              6 LOAD_FAST                0 (self)
> +              8 STORE_ATTR               0 (x)
> +             10 LOAD_CONST               0 (None)
> +             12 RETURN_VALUE
>  """ % (_C.__init__.__code__.co_firstlineno + 1,)
>
>  dis_c_instance_method_bytes = """\
>            0 LOAD_FAST                1 (1)
> -          3 LOAD_CONST               1 (1)
> -          6 COMPARE_OP               2 (==)
> -          9 LOAD_FAST                0 (0)
> -         12 STORE_ATTR               0 (0)
> -         15 LOAD_CONST               0 (0)
> -         18 RETURN_VALUE
> +          2 LOAD_CONST               1 (1)
> +          4 COMPARE_OP               2 (==)
> +          6 LOAD_FAST                0 (0)
> +          8 STORE_ATTR               0 (0)
> +         10 LOAD_CONST               0 (0)
> +         12 RETURN_VALUE
>  """
>
>  dis_c_class_method = """\
>  %3d           0 LOAD_FAST                1 (x)
> -              3 LOAD_CONST               1 (1)
> -              6 COMPARE_OP               2 (==)
> -              9 LOAD_FAST                0 (cls)
> -             12 STORE_ATTR               0 (x)
> -             15 LOAD_CONST               0 (None)
> -             18 RETURN_VALUE
> +              2 LOAD_CONST               1 (1)
> +              4 COMPARE_OP               2 (==)
> +              6 LOAD_FAST                0 (cls)
> +              8 STORE_ATTR               0 (x)
> +             10 LOAD_CONST               0 (None)
> +             12 RETURN_VALUE
>  """ % (_C.cm.__code__.co_firstlineno + 2,)
>
>  dis_c_static_method = """\
>  %3d           0 LOAD_FAST                0 (x)
> -              3 LOAD_CONST               1 (1)
> -              6 COMPARE_OP               2 (==)
> -              9 STORE_FAST               0 (x)
> -             12 LOAD_CONST               0 (None)
> -             15 RETURN_VALUE
> +              2 LOAD_CONST               1 (1)
> +              4 COMPARE_OP               2 (==)
> +              6 STORE_FAST               0 (x)
> +              8 LOAD_CONST               0 (None)
> +             10 RETURN_VALUE
>  """ % (_C.sm.__code__.co_firstlineno + 2,)
>
>  # Class disassembling info has an extra newline at end.
> @@ -95,23 +95,23 @@
>
>  dis_f = """\
>  %3d           0 LOAD_GLOBAL              0 (print)
> -              3 LOAD_FAST                0 (a)
> -              6 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
> -              9 POP_TOP
> +              2 LOAD_FAST                0 (a)
> +              4 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
> +              6 POP_TOP
>
> -%3d          10 LOAD_CONST               1 (1)
> -             13 RETURN_VALUE
> +%3d           8 LOAD_CONST               1 (1)
> +             10 RETURN_VALUE
>  """ % (_f.__code__.co_firstlineno + 1,
>         _f.__code__.co_firstlineno + 2)
>
>
>  dis_f_co_code = """\
>            0 LOAD_GLOBAL              0 (0)
> -          3 LOAD_FAST                0 (0)
> -          6 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
> -          9 POP_TOP
> -         10 LOAD_CONST               1 (1)
> -         13 RETURN_VALUE
> +          2 LOAD_FAST                0 (0)
> +          4 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
> +          6 POP_TOP
> +          8 LOAD_CONST               1 (1)
> +         10 RETURN_VALUE
>  """
>
>
> @@ -121,20 +121,20 @@
>          pass
>
>  dis_bug708901 = """\
> -%3d           0 SETUP_LOOP              23 (to 26)
> -              3 LOAD_GLOBAL              0 (range)
> -              6 LOAD_CONST               1 (1)
> +%3d           0 SETUP_LOOP              18 (to 20)
> +              2 LOAD_GLOBAL              0 (range)
> +              4 LOAD_CONST               1 (1)
>
> -%3d           9 LOAD_CONST               2 (10)
> -             12 CALL_FUNCTION            2 (2 positional, 0 keyword pair)
> -             15 GET_ITER
> -        >>   16 FOR_ITER                 6 (to 25)
> -             19 STORE_FAST               0 (res)
> +%3d           6 LOAD_CONST               2 (10)
> +              8 CALL_FUNCTION            2 (2 positional, 0 keyword pair)
> +             10 GET_ITER
> +        >>   12 FOR_ITER                 4 (to 18)
> +             14 STORE_FAST               0 (res)
>
> -%3d          22 JUMP_ABSOLUTE           16
> -        >>   25 POP_BLOCK
> -        >>   26 LOAD_CONST               0 (None)
> -             29 RETURN_VALUE
> +%3d          16 JUMP_ABSOLUTE           12
> +        >>   18 POP_BLOCK
> +        >>   20 LOAD_CONST               0 (None)
> +             22 RETURN_VALUE
>  """ % (bug708901.__code__.co_firstlineno + 1,
>         bug708901.__code__.co_firstlineno + 2,
>         bug708901.__code__.co_firstlineno + 3)
> @@ -147,22 +147,22 @@
>
>  dis_bug1333982 = """\
>  %3d           0 LOAD_CONST               1 (0)
> -              3 POP_JUMP_IF_TRUE        35
> -              6 LOAD_GLOBAL              0 (AssertionError)
> -              9 LOAD_CONST               2 (<code object <listcomp> at
> 0x..., file "%s", line %d>)
> -             12 LOAD_CONST               3
> ('bug1333982.<locals>.<listcomp>')
> -             15 MAKE_FUNCTION            0
> -             18 LOAD_FAST                0 (x)
> -             21 GET_ITER
> +              2 POP_JUMP_IF_TRUE        26
> +              4 LOAD_GLOBAL              0 (AssertionError)
> +              6 LOAD_CONST               2 (<code object <listcomp> at
> 0x..., file "%s", line %d>)
> +              8 LOAD_CONST               3
> ('bug1333982.<locals>.<listcomp>')
> +             10 MAKE_FUNCTION            0
> +             12 LOAD_FAST                0 (x)
> +             14 GET_ITER
> +             16 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
> +
> +%3d          18 LOAD_CONST               4 (1)
> +             20 BINARY_ADD
>               22 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
> +             24 RAISE_VARARGS            1
>
> -%3d          25 LOAD_CONST               4 (1)
> -             28 BINARY_ADD
> -             29 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
> -             32 RAISE_VARARGS            1
> -
> -%3d     >>   35 LOAD_CONST               0 (None)
> -             38 RETURN_VALUE
> +%3d     >>   26 LOAD_CONST               0 (None)
> +             28 RETURN_VALUE
>  """ % (bug1333982.__code__.co_firstlineno + 1,
>         __file__,
>         bug1333982.__code__.co_firstlineno + 1,
> @@ -171,19 +171,19 @@
>
>  _BIG_LINENO_FORMAT = """\
>  %3d           0 LOAD_GLOBAL              0 (spam)
> -              3 POP_TOP
> +              2 POP_TOP
>                4 LOAD_CONST               0 (None)
> -              7 RETURN_VALUE
> +              6 RETURN_VALUE
>  """
>
>  dis_module_expected_results = """\
>  Disassembly of f:
>    4           0 LOAD_CONST               0 (None)
> -              3 RETURN_VALUE
> +              2 RETURN_VALUE
>
>  Disassembly of g:
>    5           0 LOAD_CONST               0 (None)
> -              3 RETURN_VALUE
> +              2 RETURN_VALUE
>
>  """
>
> @@ -191,20 +191,20 @@
>
>  dis_expr_str = """\
>    1           0 LOAD_NAME                0 (x)
> -              3 LOAD_CONST               0 (1)
> -              6 BINARY_ADD
> -              7 RETURN_VALUE
> +              2 LOAD_CONST               0 (1)
> +              4 BINARY_ADD
> +              6 RETURN_VALUE
>  """
>
>  simple_stmt_str = "x = x + 1"
>
>  dis_simple_stmt_str = """\
>    1           0 LOAD_NAME                0 (x)
> -              3 LOAD_CONST               0 (1)
> -              6 BINARY_ADD
> -              7 STORE_NAME               0 (x)
> -             10 LOAD_CONST               1 (None)
> -             13 RETURN_VALUE
> +              2 LOAD_CONST               0 (1)
> +              4 BINARY_ADD
> +              6 STORE_NAME               0 (x)
> +              8 LOAD_CONST               1 (None)
> +             10 RETURN_VALUE
>  """
>
>  compound_stmt_str = """\
> @@ -215,54 +215,54 @@
>
>  dis_compound_stmt_str = """\
>    1           0 LOAD_CONST               0 (0)
> -              3 STORE_NAME               0 (x)
> +              2 STORE_NAME               0 (x)
>
> -  2           6 SETUP_LOOP              14 (to 23)
> +  2           4 SETUP_LOOP              12 (to 18)
>
> -  3     >>    9 LOAD_NAME                0 (x)
> -             12 LOAD_CONST               1 (1)
> -             15 INPLACE_ADD
> -             16 STORE_NAME               0 (x)
> -             19 JUMP_ABSOLUTE            9
> -             22 POP_BLOCK
> -        >>   23 LOAD_CONST               2 (None)
> -             26 RETURN_VALUE
> +  3     >>    6 LOAD_NAME                0 (x)
> +              8 LOAD_CONST               1 (1)
> +             10 INPLACE_ADD
> +             12 STORE_NAME               0 (x)
> +             14 JUMP_ABSOLUTE            6
> +             16 POP_BLOCK
> +        >>   18 LOAD_CONST               2 (None)
> +             20 RETURN_VALUE
>  """
>
>  dis_traceback = """\
> -%3d           0 SETUP_EXCEPT            12 (to 15)
> +%3d           0 SETUP_EXCEPT            12 (to 14)
>
> -%3d           3 LOAD_CONST               1 (1)
> -              6 LOAD_CONST               2 (0)
> -    -->       9 BINARY_TRUE_DIVIDE
> -             10 POP_TOP
> -             11 POP_BLOCK
> -             12 JUMP_FORWARD            46 (to 61)
> +%3d           2 LOAD_CONST               1 (1)
> +              4 LOAD_CONST               2 (0)
> +    -->       6 BINARY_TRUE_DIVIDE
> +              8 POP_TOP
> +             10 POP_BLOCK
> +             12 JUMP_FORWARD            40 (to 54)
>
> -%3d     >>   15 DUP_TOP
> +%3d     >>   14 DUP_TOP
>               16 LOAD_GLOBAL              0 (Exception)
> -             19 COMPARE_OP              10 (exception match)
> -             22 POP_JUMP_IF_FALSE       60
> -             25 POP_TOP
> -             26 STORE_FAST               0 (e)
> -             29 POP_TOP
> -             30 SETUP_FINALLY           14 (to 47)
> +             18 COMPARE_OP              10 (exception match)
> +             20 POP_JUMP_IF_FALSE       52
> +             22 POP_TOP
> +             24 STORE_FAST               0 (e)
> +             26 POP_TOP
> +             28 SETUP_FINALLY           12 (to 42)
>
> -%3d          33 LOAD_FAST                0 (e)
> -             36 LOAD_ATTR                1 (__traceback__)
> -             39 STORE_FAST               1 (tb)
> -             42 POP_BLOCK
> -             43 POP_EXCEPT
> -             44 LOAD_CONST               0 (None)
> -        >>   47 LOAD_CONST               0 (None)
> -             50 STORE_FAST               0 (e)
> -             53 DELETE_FAST              0 (e)
> -             56 END_FINALLY
> -             57 JUMP_FORWARD             1 (to 61)
> -        >>   60 END_FINALLY
> +%3d          30 LOAD_FAST                0 (e)
> +             32 LOAD_ATTR                1 (__traceback__)
> +             34 STORE_FAST               1 (tb)
> +             36 POP_BLOCK
> +             38 POP_EXCEPT
> +             40 LOAD_CONST               0 (None)
> +        >>   42 LOAD_CONST               0 (None)
> +             44 STORE_FAST               0 (e)
> +             46 DELETE_FAST              0 (e)
> +             48 END_FINALLY
> +             50 JUMP_FORWARD             2 (to 54)
> +        >>   52 END_FINALLY
>
> -%3d     >>   61 LOAD_FAST                1 (tb)
> -             64 RETURN_VALUE
> +%3d     >>   54 LOAD_FAST                1 (tb)
> +             56 RETURN_VALUE
>  """ % (TRACEBACK_CODE.co_firstlineno + 1,
>         TRACEBACK_CODE.co_firstlineno + 2,
>         TRACEBACK_CODE.co_firstlineno + 3,
> @@ -650,170 +650,170 @@
>  Instruction = dis.Instruction
>  expected_opinfo_outer = [
>    Instruction(opname='LOAD_CONST', opcode=100, arg=1, argval=3,
> argrepr='3', offset=0, starts_line=2, is_jump_target=False),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=4,
> argrepr='4', offset=3, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_CLOSURE', opcode=135, arg=0, argval='a',
> argrepr='a', offset=6, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_CLOSURE', opcode=135, arg=1, argval='b',
> argrepr='b', offset=9, starts_line=None, is_jump_target=False),
> -  Instruction(opname='BUILD_TUPLE', opcode=102, arg=2, argval=2,
> argrepr='', offset=12, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=3,
> argval=code_object_f, argrepr=repr(code_object_f), offset=15,
> starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=4,
> argval='outer.<locals>.f', argrepr="'outer.<locals>.f'", offset=18,
> starts_line=None, is_jump_target=False),
> -  Instruction(opname='MAKE_CLOSURE', opcode=134, arg=2, argval=2,
> argrepr='', offset=21, starts_line=None, is_jump_target=False),
> -  Instruction(opname='STORE_FAST', opcode=125, arg=2, argval='f',
> argrepr='f', offset=24, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_GLOBAL', opcode=116, arg=0, argval='print',
> argrepr='print', offset=27, starts_line=7, is_jump_target=False),
> -  Instruction(opname='LOAD_DEREF', opcode=136, arg=0, argval='a',
> argrepr='a', offset=30, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_DEREF', opcode=136, arg=1, argval='b',
> argrepr='b', offset=33, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=5, argval='',
> argrepr="''", offset=36, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=6, argval=1,
> argrepr='1', offset=39, starts_line=None, is_jump_target=False),
> -  Instruction(opname='BUILD_LIST', opcode=103, arg=0, argval=0,
> argrepr='', offset=42, starts_line=None, is_jump_target=False),
> -  Instruction(opname='BUILD_MAP', opcode=105, arg=0, argval=0,
> argrepr='', offset=45, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=7, argval='Hello
> world!', argrepr="'Hello world!'", offset=48, starts_line=None,
> is_jump_target=False),
> -  Instruction(opname='CALL_FUNCTION', opcode=131, arg=7, argval=7,
> argrepr='7 positional, 0 keyword pair', offset=51, starts_line=None,
> is_jump_target=False),
> -  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=54, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_FAST', opcode=124, arg=2, argval='f',
> argrepr='f', offset=55, starts_line=8, is_jump_target=False),
> -  Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None,
> argrepr='', offset=58, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=4,
> argrepr='4', offset=2, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_CLOSURE', opcode=135, arg=0, argval='a',
> argrepr='a', offset=4, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_CLOSURE', opcode=135, arg=1, argval='b',
> argrepr='b', offset=6, starts_line=None, is_jump_target=False),
> +  Instruction(opname='BUILD_TUPLE', opcode=102, arg=2, argval=2,
> argrepr='', offset=8, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=3,
> argval=code_object_f, argrepr=repr(code_object_f), offset=10,
> starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=4,
> argval='outer.<locals>.f', argrepr="'outer.<locals>.f'", offset=12,
> starts_line=None, is_jump_target=False),
> +  Instruction(opname='MAKE_CLOSURE', opcode=134, arg=2, argval=2,
> argrepr='', offset=14, starts_line=None, is_jump_target=False),
> +  Instruction(opname='STORE_FAST', opcode=125, arg=2, argval='f',
> argrepr='f', offset=16, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_GLOBAL', opcode=116, arg=0, argval='print',
> argrepr='print', offset=18, starts_line=7, is_jump_target=False),
> +  Instruction(opname='LOAD_DEREF', opcode=136, arg=0, argval='a',
> argrepr='a', offset=20, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_DEREF', opcode=136, arg=1, argval='b',
> argrepr='b', offset=22, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=5, argval='',
> argrepr="''", offset=24, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=6, argval=1,
> argrepr='1', offset=26, starts_line=None, is_jump_target=False),
> +  Instruction(opname='BUILD_LIST', opcode=103, arg=0, argval=0,
> argrepr='', offset=28, starts_line=None, is_jump_target=False),
> +  Instruction(opname='BUILD_MAP', opcode=105, arg=0, argval=0,
> argrepr='', offset=30, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=7, argval='Hello
> world!', argrepr="'Hello world!'", offset=32, starts_line=None,
> is_jump_target=False),
> +  Instruction(opname='CALL_FUNCTION', opcode=131, arg=7, argval=7,
> argrepr='7 positional, 0 keyword pair', offset=34, starts_line=None,
> is_jump_target=False),
> +  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=36, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_FAST', opcode=124, arg=2, argval='f',
> argrepr='f', offset=38, starts_line=8, is_jump_target=False),
> +  Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None,
> argrepr='', offset=40, starts_line=None, is_jump_target=False),
>  ]
>
>  expected_opinfo_f = [
>    Instruction(opname='LOAD_CONST', opcode=100, arg=1, argval=5,
> argrepr='5', offset=0, starts_line=3, is_jump_target=False),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=6,
> argrepr='6', offset=3, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_CLOSURE', opcode=135, arg=2, argval='a',
> argrepr='a', offset=6, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_CLOSURE', opcode=135, arg=3, argval='b',
> argrepr='b', offset=9, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_CLOSURE', opcode=135, arg=0, argval='c',
> argrepr='c', offset=12, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_CLOSURE', opcode=135, arg=1, argval='d',
> argrepr='d', offset=15, starts_line=None, is_jump_target=False),
> -  Instruction(opname='BUILD_TUPLE', opcode=102, arg=4, argval=4,
> argrepr='', offset=18, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=3,
> argval=code_object_inner, argrepr=repr(code_object_inner), offset=21,
> starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=4,
> argval='outer.<locals>.f.<locals>.inner',
> argrepr="'outer.<locals>.f.<locals>.inner'", offset=24, starts_line=None,
> is_jump_target=False),
> -  Instruction(opname='MAKE_CLOSURE', opcode=134, arg=2, argval=2,
> argrepr='', offset=27, starts_line=None, is_jump_target=False),
> -  Instruction(opname='STORE_FAST', opcode=125, arg=2, argval='inner',
> argrepr='inner', offset=30, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_GLOBAL', opcode=116, arg=0, argval='print',
> argrepr='print', offset=33, starts_line=5, is_jump_target=False),
> -  Instruction(opname='LOAD_DEREF', opcode=136, arg=2, argval='a',
> argrepr='a', offset=36, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_DEREF', opcode=136, arg=3, argval='b',
> argrepr='b', offset=39, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_DEREF', opcode=136, arg=0, argval='c',
> argrepr='c', offset=42, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_DEREF', opcode=136, arg=1, argval='d',
> argrepr='d', offset=45, starts_line=None, is_jump_target=False),
> -  Instruction(opname='CALL_FUNCTION', opcode=131, arg=4, argval=4,
> argrepr='4 positional, 0 keyword pair', offset=48, starts_line=None,
> is_jump_target=False),
> -  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=51, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_FAST', opcode=124, arg=2, argval='inner',
> argrepr='inner', offset=52, starts_line=6, is_jump_target=False),
> -  Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None,
> argrepr='', offset=55, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=6,
> argrepr='6', offset=2, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_CLOSURE', opcode=135, arg=2, argval='a',
> argrepr='a', offset=4, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_CLOSURE', opcode=135, arg=3, argval='b',
> argrepr='b', offset=6, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_CLOSURE', opcode=135, arg=0, argval='c',
> argrepr='c', offset=8, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_CLOSURE', opcode=135, arg=1, argval='d',
> argrepr='d', offset=10, starts_line=None, is_jump_target=False),
> +  Instruction(opname='BUILD_TUPLE', opcode=102, arg=4, argval=4,
> argrepr='', offset=12, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=3,
> argval=code_object_inner, argrepr=repr(code_object_inner), offset=14,
> starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=4,
> argval='outer.<locals>.f.<locals>.inner',
> argrepr="'outer.<locals>.f.<locals>.inner'", offset=16, starts_line=None,
> is_jump_target=False),
> +  Instruction(opname='MAKE_CLOSURE', opcode=134, arg=2, argval=2,
> argrepr='', offset=18, starts_line=None, is_jump_target=False),
> +  Instruction(opname='STORE_FAST', opcode=125, arg=2, argval='inner',
> argrepr='inner', offset=20, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_GLOBAL', opcode=116, arg=0, argval='print',
> argrepr='print', offset=22, starts_line=5, is_jump_target=False),
> +  Instruction(opname='LOAD_DEREF', opcode=136, arg=2, argval='a',
> argrepr='a', offset=24, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_DEREF', opcode=136, arg=3, argval='b',
> argrepr='b', offset=26, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_DEREF', opcode=136, arg=0, argval='c',
> argrepr='c', offset=28, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_DEREF', opcode=136, arg=1, argval='d',
> argrepr='d', offset=30, starts_line=None, is_jump_target=False),
> +  Instruction(opname='CALL_FUNCTION', opcode=131, arg=4, argval=4,
> argrepr='4 positional, 0 keyword pair', offset=32, starts_line=None,
> is_jump_target=False),
> +  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=34, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_FAST', opcode=124, arg=2, argval='inner',
> argrepr='inner', offset=36, starts_line=6, is_jump_target=False),
> +  Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None,
> argrepr='', offset=38, starts_line=None, is_jump_target=False),
>  ]
>
>  expected_opinfo_inner = [
>    Instruction(opname='LOAD_GLOBAL', opcode=116, arg=0, argval='print',
> argrepr='print', offset=0, starts_line=4, is_jump_target=False),
> -  Instruction(opname='LOAD_DEREF', opcode=136, arg=0, argval='a',
> argrepr='a', offset=3, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_DEREF', opcode=136, arg=1, argval='b',
> argrepr='b', offset=6, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_DEREF', opcode=136, arg=2, argval='c',
> argrepr='c', offset=9, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_DEREF', opcode=136, arg=3, argval='d',
> argrepr='d', offset=12, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='e',
> argrepr='e', offset=15, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_FAST', opcode=124, arg=1, argval='f',
> argrepr='f', offset=18, starts_line=None, is_jump_target=False),
> -  Instruction(opname='CALL_FUNCTION', opcode=131, arg=6, argval=6,
> argrepr='6 positional, 0 keyword pair', offset=21, starts_line=None,
> is_jump_target=False),
> -  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=24, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None,
> argrepr='None', offset=25, starts_line=None, is_jump_target=False),
> -  Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None,
> argrepr='', offset=28, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_DEREF', opcode=136, arg=0, argval='a',
> argrepr='a', offset=2, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_DEREF', opcode=136, arg=1, argval='b',
> argrepr='b', offset=4, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_DEREF', opcode=136, arg=2, argval='c',
> argrepr='c', offset=6, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_DEREF', opcode=136, arg=3, argval='d',
> argrepr='d', offset=8, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='e',
> argrepr='e', offset=10, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_FAST', opcode=124, arg=1, argval='f',
> argrepr='f', offset=12, starts_line=None, is_jump_target=False),
> +  Instruction(opname='CALL_FUNCTION', opcode=131, arg=6, argval=6,
> argrepr='6 positional, 0 keyword pair', offset=14, starts_line=None,
> is_jump_target=False),
> +  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=16, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None,
> argrepr='None', offset=18, starts_line=None, is_jump_target=False),
> +  Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None,
> argrepr='', offset=20, starts_line=None, is_jump_target=False),
>  ]
>
>  expected_opinfo_jumpy = [
> -  Instruction(opname='SETUP_LOOP', opcode=120, arg=68, argval=71,
> argrepr='to 71', offset=0, starts_line=3, is_jump_target=False),
> -  Instruction(opname='LOAD_GLOBAL', opcode=116, arg=0, argval='range',
> argrepr='range', offset=3, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=1, argval=10,
> argrepr='10', offset=6, starts_line=None, is_jump_target=False),
> -  Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1,
> argrepr='1 positional, 0 keyword pair', offset=9, starts_line=None,
> is_jump_target=False),
> -  Instruction(opname='GET_ITER', opcode=68, arg=None, argval=None,
> argrepr='', offset=12, starts_line=None, is_jump_target=False),
> -  Instruction(opname='FOR_ITER', opcode=93, arg=44, argval=60,
> argrepr='to 60', offset=13, starts_line=None, is_jump_target=True),
> -  Instruction(opname='STORE_FAST', opcode=125, arg=0, argval='i',
> argrepr='i', offset=16, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print',
> argrepr='print', offset=19, starts_line=4, is_jump_target=False),
> -  Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i',
> argrepr='i', offset=22, starts_line=None, is_jump_target=False),
> -  Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1,
> argrepr='1 positional, 0 keyword pair', offset=25, starts_line=None,
> is_jump_target=False),
> -  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=28, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i',
> argrepr='i', offset=29, starts_line=5, is_jump_target=False),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=4,
> argrepr='4', offset=32, starts_line=None, is_jump_target=False),
> -  Instruction(opname='COMPARE_OP', opcode=107, arg=0, argval='<',
> argrepr='<', offset=35, starts_line=None, is_jump_target=False),
> -  Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=44, argval=44,
> argrepr='', offset=38, starts_line=None, is_jump_target=False),
> -  Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=13, argval=13,
> argrepr='', offset=41, starts_line=6, is_jump_target=False),
> -  Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i',
> argrepr='i', offset=44, starts_line=7, is_jump_target=True),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=3, argval=6,
> argrepr='6', offset=47, starts_line=None, is_jump_target=False),
> -  Instruction(opname='COMPARE_OP', opcode=107, arg=4, argval='>',
> argrepr='>', offset=50, starts_line=None, is_jump_target=False),
> -  Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=13, argval=13,
> argrepr='', offset=53, starts_line=None, is_jump_target=False),
> -  Instruction(opname='BREAK_LOOP', opcode=80, arg=None, argval=None,
> argrepr='', offset=56, starts_line=8, is_jump_target=False),
> -  Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=13, argval=13,
> argrepr='', offset=57, starts_line=None, is_jump_target=False),
> -  Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None,
> argrepr='', offset=60, starts_line=None, is_jump_target=True),
> -  Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print',
> argrepr='print', offset=61, starts_line=10, is_jump_target=False),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=4, argval='I can haz
> else clause?', argrepr="'I can haz else clause?'", offset=64,
> starts_line=None, is_jump_target=False),
> -  Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1,
> argrepr='1 positional, 0 keyword pair', offset=67, starts_line=None,
> is_jump_target=False),
> -  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=70, starts_line=None, is_jump_target=False),
> -  Instruction(opname='SETUP_LOOP', opcode=120, arg=68, argval=142,
> argrepr='to 142', offset=71, starts_line=11, is_jump_target=True),
> -  Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i',
> argrepr='i', offset=74, starts_line=None, is_jump_target=True),
> -  Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=131,
> argval=131, argrepr='', offset=77, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print',
> argrepr='print', offset=80, starts_line=12, is_jump_target=False),
> -  Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i',
> argrepr='i', offset=83, starts_line=None, is_jump_target=False),
> -  Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1,
> argrepr='1 positional, 0 keyword pair', offset=86, starts_line=None,
> is_jump_target=False),
> -  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=89, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i',
> argrepr='i', offset=90, starts_line=13, is_jump_target=False),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=5, argval=1,
> argrepr='1', offset=93, starts_line=None, is_jump_target=False),
> -  Instruction(opname='INPLACE_SUBTRACT', opcode=56, arg=None,
> argval=None, argrepr='', offset=96, starts_line=None, is_jump_target=False),
> -  Instruction(opname='STORE_FAST', opcode=125, arg=0, argval='i',
> argrepr='i', offset=97, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i',
> argrepr='i', offset=100, starts_line=14, is_jump_target=False),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=3, argval=6,
> argrepr='6', offset=103, starts_line=None, is_jump_target=False),
> -  Instruction(opname='COMPARE_OP', opcode=107, arg=4, argval='>',
> argrepr='>', offset=106, starts_line=None, is_jump_target=False),
> -  Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=115,
> argval=115, argrepr='', offset=109, starts_line=None, is_jump_target=False),
> -  Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=74, argval=74,
> argrepr='', offset=112, starts_line=15, is_jump_target=False),
> -  Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i',
> argrepr='i', offset=115, starts_line=16, is_jump_target=True),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=4,
> argrepr='4', offset=118, starts_line=None, is_jump_target=False),
> -  Instruction(opname='COMPARE_OP', opcode=107, arg=0, argval='<',
> argrepr='<', offset=121, starts_line=None, is_jump_target=False),
> -  Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=74, argval=74,
> argrepr='', offset=124, starts_line=None, is_jump_target=False),
> -  Instruction(opname='BREAK_LOOP', opcode=80, arg=None, argval=None,
> argrepr='', offset=127, starts_line=17, is_jump_target=False),
> -  Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=74, argval=74,
> argrepr='', offset=128, starts_line=None, is_jump_target=False),
> -  Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None,
> argrepr='', offset=131, starts_line=None, is_jump_target=True),
> -  Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print',
> argrepr='print', offset=132, starts_line=19, is_jump_target=False),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=6, argval='Who let
> lolcatz into this test suite?', argrepr="'Who let lolcatz into this test
> suite?'", offset=135, starts_line=None, is_jump_target=False),
> -  Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1,
> argrepr='1 positional, 0 keyword pair', offset=138, starts_line=None,
> is_jump_target=False),
> -  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=141, starts_line=None, is_jump_target=False),
> -  Instruction(opname='SETUP_FINALLY', opcode=122, arg=73, argval=218,
> argrepr='to 218', offset=142, starts_line=20, is_jump_target=True),
> -  Instruction(opname='SETUP_EXCEPT', opcode=121, arg=12, argval=160,
> argrepr='to 160', offset=145, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=5, argval=1,
> argrepr='1', offset=148, starts_line=21, is_jump_target=False),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=7, argval=0,
> argrepr='0', offset=151, starts_line=None, is_jump_target=False),
> -  Instruction(opname='BINARY_TRUE_DIVIDE', opcode=27, arg=None,
> argval=None, argrepr='', offset=154, starts_line=None,
> is_jump_target=False),
> -  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=155, starts_line=None, is_jump_target=False),
> -  Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None,
> argrepr='', offset=156, starts_line=None, is_jump_target=False),
> -  Instruction(opname='JUMP_FORWARD', opcode=110, arg=28, argval=188,
> argrepr='to 188', offset=157, starts_line=None, is_jump_target=False),
> -  Instruction(opname='DUP_TOP', opcode=4, arg=None, argval=None,
> argrepr='', offset=160, starts_line=22, is_jump_target=True),
> -  Instruction(opname='LOAD_GLOBAL', opcode=116, arg=2,
> argval='ZeroDivisionError', argrepr='ZeroDivisionError', offset=161,
> starts_line=None, is_jump_target=False),
> -  Instruction(opname='COMPARE_OP', opcode=107, arg=10, argval='exception
> match', argrepr='exception match', offset=164, starts_line=None,
> is_jump_target=False),
> -  Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=187,
> argval=187, argrepr='', offset=167, starts_line=None, is_jump_target=False),
> -  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=170, starts_line=None, is_jump_target=False),
> -  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=171, starts_line=None, is_jump_target=False),
> -  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=172, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print',
> argrepr='print', offset=173, starts_line=23, is_jump_target=False),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=8, argval='Here we go,
> here we go, here we go...', argrepr="'Here we go, here we go, here we
> go...'", offset=176, starts_line=None, is_jump_target=False),
> -  Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1,
> argrepr='1 positional, 0 keyword pair', offset=179, starts_line=None,
> is_jump_target=False),
> -  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=182, starts_line=None, is_jump_target=False),
> -  Instruction(opname='POP_EXCEPT', opcode=89, arg=None, argval=None,
> argrepr='', offset=183, starts_line=None, is_jump_target=False),
> -  Instruction(opname='JUMP_FORWARD', opcode=110, arg=27, argval=214,
> argrepr='to 214', offset=184, starts_line=None, is_jump_target=False),
> -  Instruction(opname='END_FINALLY', opcode=88, arg=None, argval=None,
> argrepr='', offset=187, starts_line=None, is_jump_target=True),
> -  Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i',
> argrepr='i', offset=188, starts_line=25, is_jump_target=True),
> -  Instruction(opname='SETUP_WITH', opcode=143, arg=17, argval=211,
> argrepr='to 211', offset=191, starts_line=None, is_jump_target=False),
> -  Instruction(opname='STORE_FAST', opcode=125, arg=1, argval='dodgy',
> argrepr='dodgy', offset=194, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print',
> argrepr='print', offset=197, starts_line=26, is_jump_target=False),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=9, argval='Never reach
> this', argrepr="'Never reach this'", offset=200, starts_line=None,
> is_jump_target=False),
> -  Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1,
> argrepr='1 positional, 0 keyword pair', offset=203, starts_line=None,
> is_jump_target=False),
> -  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=206, starts_line=None, is_jump_target=False),
> -  Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None,
> argrepr='', offset=207, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None,
> argrepr='None', offset=208, starts_line=None, is_jump_target=False),
> -  Instruction(opname='WITH_CLEANUP_START', opcode=81, arg=None,
> argval=None, argrepr='', offset=211, starts_line=None, is_jump_target=True),
> -  Instruction(opname='WITH_CLEANUP_FINISH', opcode=82, arg=None,
> argval=None, argrepr='', offset=212, starts_line=None,
> is_jump_target=False),
> -  Instruction(opname='END_FINALLY', opcode=88, arg=None, argval=None,
> argrepr='', offset=213, starts_line=None, is_jump_target=False),
> -  Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None,
> argrepr='', offset=214, starts_line=None, is_jump_target=True),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None,
> argrepr='None', offset=215, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print',
> argrepr='print', offset=218, starts_line=28, is_jump_target=True),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=10, argval="OK, now
> we're done", argrepr='"OK, now we\'re done"', offset=221, starts_line=None,
> is_jump_target=False),
> -  Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1,
> argrepr='1 positional, 0 keyword pair', offset=224, starts_line=None,
> is_jump_target=False),
> -  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=227, starts_line=None, is_jump_target=False),
> -  Instruction(opname='END_FINALLY', opcode=88, arg=None, argval=None,
> argrepr='', offset=228, starts_line=None, is_jump_target=False),
> -  Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None,
> argrepr='None', offset=229, starts_line=None, is_jump_target=False),
> -  Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None,
> argrepr='', offset=232, starts_line=None, is_jump_target=False),
> +  Instruction(opname='SETUP_LOOP', opcode=120, arg=52, argval=54,
> argrepr='to 54', offset=0, starts_line=3, is_jump_target=False),
> +  Instruction(opname='LOAD_GLOBAL', opcode=116, arg=0, argval='range',
> argrepr='range', offset=2, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=1, argval=10,
> argrepr='10', offset=4, starts_line=None, is_jump_target=False),
> +  Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1,
> argrepr='1 positional, 0 keyword pair', offset=6, starts_line=None,
> is_jump_target=False),
> +  Instruction(opname='GET_ITER', opcode=68, arg=None, argval=None,
> argrepr='', offset=8, starts_line=None, is_jump_target=False),
> +  Instruction(opname='FOR_ITER', opcode=93, arg=32, argval=44,
> argrepr='to 44', offset=10, starts_line=None, is_jump_target=True),
> +  Instruction(opname='STORE_FAST', opcode=125, arg=0, argval='i',
> argrepr='i', offset=12, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print',
> argrepr='print', offset=14, starts_line=4, is_jump_target=False),
> +  Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i',
> argrepr='i', offset=16, starts_line=None, is_jump_target=False),
> +  Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1,
> argrepr='1 positional, 0 keyword pair', offset=18, starts_line=None,
> is_jump_target=False),
> +  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=20, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i',
> argrepr='i', offset=22, starts_line=5, is_jump_target=False),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=4,
> argrepr='4', offset=24, starts_line=None, is_jump_target=False),
> +  Instruction(opname='COMPARE_OP', opcode=107, arg=0, argval='<',
> argrepr='<', offset=26, starts_line=None, is_jump_target=False),
> +  Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=32, argval=32,
> argrepr='', offset=28, starts_line=None, is_jump_target=False),
> +  Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=10, argval=10,
> argrepr='', offset=30, starts_line=6, is_jump_target=False),
> +  Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i',
> argrepr='i', offset=32, starts_line=7, is_jump_target=True),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=3, argval=6,
> argrepr='6', offset=34, starts_line=None, is_jump_target=False),
> +  Instruction(opname='COMPARE_OP', opcode=107, arg=4, argval='>',
> argrepr='>', offset=36, starts_line=None, is_jump_target=False),
> +  Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=10, argval=10,
> argrepr='', offset=38, starts_line=None, is_jump_target=False),
> +  Instruction(opname='BREAK_LOOP', opcode=80, arg=None, argval=None,
> argrepr='', offset=40, starts_line=8, is_jump_target=False),
> +  Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=10, argval=10,
> argrepr='', offset=42, starts_line=None, is_jump_target=False),
> +  Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None,
> argrepr='', offset=44, starts_line=None, is_jump_target=True),
> +  Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print',
> argrepr='print', offset=46, starts_line=10, is_jump_target=False),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=4, argval='I can haz
> else clause?', argrepr="'I can haz else clause?'", offset=48,
> starts_line=None, is_jump_target=False),
> +  Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1,
> argrepr='1 positional, 0 keyword pair', offset=50, starts_line=None,
> is_jump_target=False),
> +  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=52, starts_line=None, is_jump_target=False),
> +  Instruction(opname='SETUP_LOOP', opcode=120, arg=52, argval=108,
> argrepr='to 108', offset=54, starts_line=11, is_jump_target=True),
> +  Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i',
> argrepr='i', offset=56, starts_line=None, is_jump_target=True),
> +  Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=98, argval=98,
> argrepr='', offset=58, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print',
> argrepr='print', offset=60, starts_line=12, is_jump_target=False),
> +  Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i',
> argrepr='i', offset=62, starts_line=None, is_jump_target=False),
> +  Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1,
> argrepr='1 positional, 0 keyword pair', offset=64, starts_line=None,
> is_jump_target=False),
> +  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=66, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i',
> argrepr='i', offset=68, starts_line=13, is_jump_target=False),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=5, argval=1,
> argrepr='1', offset=70, starts_line=None, is_jump_target=False),
> +  Instruction(opname='INPLACE_SUBTRACT', opcode=56, arg=None,
> argval=None, argrepr='', offset=72, starts_line=None, is_jump_target=False),
> +  Instruction(opname='STORE_FAST', opcode=125, arg=0, argval='i',
> argrepr='i', offset=74, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i',
> argrepr='i', offset=76, starts_line=14, is_jump_target=False),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=3, argval=6,
> argrepr='6', offset=78, starts_line=None, is_jump_target=False),
> +  Instruction(opname='COMPARE_OP', opcode=107, arg=4, argval='>',
> argrepr='>', offset=80, starts_line=None, is_jump_target=False),
> +  Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=86, argval=86,
> argrepr='', offset=82, starts_line=None, is_jump_target=False),
> +  Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=56, argval=56,
> argrepr='', offset=84, starts_line=15, is_jump_target=False),
> +  Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i',
> argrepr='i', offset=86, starts_line=16, is_jump_target=True),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=4,
> argrepr='4', offset=88, starts_line=None, is_jump_target=False),
> +  Instruction(opname='COMPARE_OP', opcode=107, arg=0, argval='<',
> argrepr='<', offset=90, starts_line=None, is_jump_target=False),
> +  Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=56, argval=56,
> argrepr='', offset=92, starts_line=None, is_jump_target=False),
> +  Instruction(opname='BREAK_LOOP', opcode=80, arg=None, argval=None,
> argrepr='', offset=94, starts_line=17, is_jump_target=False),
> +  Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=56, argval=56,
> argrepr='', offset=96, starts_line=None, is_jump_target=False),
> +  Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None,
> argrepr='', offset=98, starts_line=None, is_jump_target=True),
> +  Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print',
> argrepr='print', offset=100, starts_line=19, is_jump_target=False),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=6, argval='Who let
> lolcatz into this test suite?', argrepr="'Who let lolcatz into this test
> suite?'", offset=102, starts_line=None, is_jump_target=False),
> +  Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1,
> argrepr='1 positional, 0 keyword pair', offset=104, starts_line=None,
> is_jump_target=False),
> +  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=106, starts_line=None, is_jump_target=False),
> +  Instruction(opname='SETUP_FINALLY', opcode=122, arg=70, argval=180,
> argrepr='to 180', offset=108, starts_line=20, is_jump_target=True),
> +  Instruction(opname='SETUP_EXCEPT', opcode=121, arg=12, argval=124,
> argrepr='to 124', offset=110, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=5, argval=1,
> argrepr='1', offset=112, starts_line=21, is_jump_target=False),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=7, argval=0,
> argrepr='0', offset=114, starts_line=None, is_jump_target=False),
> +  Instruction(opname='BINARY_TRUE_DIVIDE', opcode=27, arg=None,
> argval=None, argrepr='', offset=116, starts_line=None,
> is_jump_target=False),
> +  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=118, starts_line=None, is_jump_target=False),
> +  Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None,
> argrepr='', offset=120, starts_line=None, is_jump_target=False),
> +  Instruction(opname='JUMP_FORWARD', opcode=110, arg=28, argval=152,
> argrepr='to 152', offset=122, starts_line=None, is_jump_target=False),
> +  Instruction(opname='DUP_TOP', opcode=4, arg=None, argval=None,
> argrepr='', offset=124, starts_line=22, is_jump_target=True),
> +  Instruction(opname='LOAD_GLOBAL', opcode=116, arg=2,
> argval='ZeroDivisionError', argrepr='ZeroDivisionError', offset=126,
> starts_line=None, is_jump_target=False),
> +  Instruction(opname='COMPARE_OP', opcode=107, arg=10, argval='exception
> match', argrepr='exception match', offset=128, starts_line=None,
> is_jump_target=False),
> +  Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=150,
> argval=150, argrepr='', offset=130, starts_line=None, is_jump_target=False),
> +  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=132, starts_line=None, is_jump_target=False),
> +  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=134, starts_line=None, is_jump_target=False),
> +  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=136, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print',
> argrepr='print', offset=138, starts_line=23, is_jump_target=False),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=8, argval='Here we go,
> here we go, here we go...', argrepr="'Here we go, here we go, here we
> go...'", offset=140, starts_line=None, is_jump_target=False),
> +  Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1,
> argrepr='1 positional, 0 keyword pair', offset=142, starts_line=None,
> is_jump_target=False),
> +  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=144, starts_line=None, is_jump_target=False),
> +  Instruction(opname='POP_EXCEPT', opcode=89, arg=None, argval=None,
> argrepr='', offset=146, starts_line=None, is_jump_target=False),
> +  Instruction(opname='JUMP_FORWARD', opcode=110, arg=26, argval=176,
> argrepr='to 176', offset=148, starts_line=None, is_jump_target=False),
> +  Instruction(opname='END_FINALLY', opcode=88, arg=None, argval=None,
> argrepr='', offset=150, starts_line=None, is_jump_target=True),
> +  Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i',
> argrepr='i', offset=152, starts_line=25, is_jump_target=True),
> +  Instruction(opname='SETUP_WITH', opcode=143, arg=14, argval=170,
> argrepr='to 170', offset=154, starts_line=None, is_jump_target=False),
> +  Instruction(opname='STORE_FAST', opcode=125, arg=1, argval='dodgy',
> argrepr='dodgy', offset=156, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print',
> argrepr='print', offset=158, starts_line=26, is_jump_target=False),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=9, argval='Never reach
> this', argrepr="'Never reach this'", offset=160, starts_line=None,
> is_jump_target=False),
> +  Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1,
> argrepr='1 positional, 0 keyword pair', offset=162, starts_line=None,
> is_jump_target=False),
> +  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=164, starts_line=None, is_jump_target=False),
> +  Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None,
> argrepr='', offset=166, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None,
> argrepr='None', offset=168, starts_line=None, is_jump_target=False),
> +  Instruction(opname='WITH_CLEANUP_START', opcode=81, arg=None,
> argval=None, argrepr='', offset=170, starts_line=None, is_jump_target=True),
> +  Instruction(opname='WITH_CLEANUP_FINISH', opcode=82, arg=None,
> argval=None, argrepr='', offset=172, starts_line=None,
> is_jump_target=False),
> +  Instruction(opname='END_FINALLY', opcode=88, arg=None, argval=None,
> argrepr='', offset=174, starts_line=None, is_jump_target=False),
> +  Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None,
> argrepr='', offset=176, starts_line=None, is_jump_target=True),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None,
> argrepr='None', offset=178, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print',
> argrepr='print', offset=180, starts_line=28, is_jump_target=True),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=10, argval="OK, now
> we're done", argrepr='"OK, now we\'re done"', offset=182, starts_line=None,
> is_jump_target=False),
> +  Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1,
> argrepr='1 positional, 0 keyword pair', offset=184, starts_line=None,
> is_jump_target=False),
> +  Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None,
> argrepr='', offset=186, starts_line=None, is_jump_target=False),
> +  Instruction(opname='END_FINALLY', opcode=88, arg=None, argval=None,
> argrepr='', offset=188, starts_line=None, is_jump_target=False),
> +  Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None,
> argrepr='None', offset=190, starts_line=None, is_jump_target=False),
> +  Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None,
> argrepr='', offset=192, starts_line=None, is_jump_target=False),
>  ]
>
>  # One last piece of inspect fodder to check the default line number
> handling
>  def simple(): pass
>  expected_opinfo_simple = [
>    Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None,
> argrepr='None', offset=0, starts_line=simple.__code__.co_firstlineno,
> is_jump_target=False),
> -  Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None,
> argrepr='', offset=3, starts_line=None, is_jump_target=False)
> +  Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None,
> argrepr='', offset=2, starts_line=None, is_jump_target=False)
>  ]
>
>
> diff --git a/Misc/NEWS b/Misc/NEWS
> --- a/Misc/NEWS
> +++ b/Misc/NEWS
> @@ -10,6 +10,9 @@
>  Core and Builtins
>  -----------------
>
> +- Issue #26647: Python interpreter now uses 16-bit wordcode instead of
> bytecode.
> +  Patch by Demur Rumed.
> +
>  - Issue #23275: Allow assigning to an empty target list in round brackets:
>    () = iterable.
>
> diff --git a/Objects/frameobject.c b/Objects/frameobject.c
> --- a/Objects/frameobject.c
> +++ b/Objects/frameobject.c
> @@ -189,7 +189,7 @@
>      memset(blockstack, '\0', sizeof(blockstack));
>      memset(in_finally, '\0', sizeof(in_finally));
>      blockstack_top = 0;
> -    for (addr = 0; addr < code_len; addr++) {
> +    for (addr = 0; addr < code_len; addr += 2) {
>          unsigned char op = code[addr];
>          switch (op) {
>          case SETUP_LOOP:
> @@ -251,10 +251,6 @@
>                  }
>              }
>          }
> -
> -        if (op >= HAVE_ARGUMENT) {
> -            addr += 2;
> -        }
>      }
>
>      /* Verify that the blockstack tracking code didn't get lost. */
> @@ -277,7 +273,7 @@
>       * can tell whether the jump goes into any blocks without coming out
>       * again - in that case we raise an exception below. */
>      delta_iblock = 0;
> -    for (addr = min_addr; addr < max_addr; addr++) {
> +    for (addr = min_addr; addr < max_addr; addr += 2) {
>          unsigned char op = code[addr];
>          switch (op) {
>          case SETUP_LOOP:
> @@ -294,10 +290,6 @@
>          }
>
>          min_delta_iblock = Py_MIN(min_delta_iblock, delta_iblock);
> -
> -        if (op >= HAVE_ARGUMENT) {
> -            addr += 2;
> -        }
>      }
>
>      /* Derive the absolute iblock values from the deltas. */
> diff --git a/Objects/genobject.c b/Objects/genobject.c
> --- a/Objects/genobject.c
> +++ b/Objects/genobject.c
> @@ -277,7 +277,7 @@
>          PyObject *bytecode = f->f_code->co_code;
>          unsigned char *code = (unsigned char
> *)PyBytes_AS_STRING(bytecode);
>
> -        if (code[f->f_lasti + 1] != YIELD_FROM)
> +        if (code[f->f_lasti + 2] != YIELD_FROM)
>              return NULL;
>          yf = f->f_stacktop[-1];
>          Py_INCREF(yf);
> @@ -376,7 +376,7 @@
>              assert(ret == yf);
>              Py_DECREF(ret);
>              /* Termination repetition of YIELD_FROM */
> -            gen->gi_frame->f_lasti++;
> +            gen->gi_frame->f_lasti += 2;
>              if (_PyGen_FetchStopIterationValue(&val) == 0) {
>                  ret = gen_send_ex(gen, val, 0, 0);
>                  Py_DECREF(val);
> diff --git a/PC/launcher.c b/PC/launcher.c
> --- a/PC/launcher.c
> +++ b/PC/launcher.c
> @@ -1089,7 +1089,7 @@
>      { 3190, 3230, L"3.3" },
>      { 3250, 3310, L"3.4" },
>      { 3320, 3350, L"3.5" },
> -    { 3360, 3361, L"3.6" },
> +    { 3360, 3370, L"3.6" },
>      { 0 }
>  };
>
> diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj
> --- a/PCbuild/pythoncore.vcxproj
> +++ b/PCbuild/pythoncore.vcxproj
> @@ -209,6 +209,7 @@
>      <ClInclude Include="..\Python\condvar.h" />
>      <ClInclude Include="..\Python\importdl.h" />
>      <ClInclude Include="..\Python\thread_nt.h" />
> +    <ClInclude Include="..\Python\wordcode_helpers.h" />
>    </ItemGroup>
>    <ItemGroup>
>      <ClCompile Include="..\Modules\_bisectmodule.c" />
> diff --git a/PCbuild/pythoncore.vcxproj.filters
> b/PCbuild/pythoncore.vcxproj.filters
> --- a/PCbuild/pythoncore.vcxproj.filters
> +++ b/PCbuild/pythoncore.vcxproj.filters
> @@ -420,6 +420,9 @@
>      <ClInclude Include="..\Python\thread_nt.h">
>        <Filter>Python</Filter>
>      </ClInclude>
> +    <ClInclude Include="..\Python\wordcode_helpers.h">
> +      <Filter>Python</Filter>
> +    </ClInclude>
>      <ClInclude Include="..\Python\condvar.h">
>        <Filter>Python</Filter>
>      </ClInclude>
> diff --git a/Python/ceval.c b/Python/ceval.c
> --- a/Python/ceval.c
> +++ b/Python/ceval.c
> @@ -886,24 +886,10 @@
>  /* Import the static jump table */
>  #include "opcode_targets.h"
>
> -/* This macro is used when several opcodes defer to the same
> implementation
> -   (e.g. SETUP_LOOP, SETUP_FINALLY) */
> -#define TARGET_WITH_IMPL(op, impl) \
> -    TARGET_##op: \
> -        opcode = op; \
> -        if (HAS_ARG(op)) \
> -            oparg = NEXTARG(); \
> -    case op: \
> -        goto impl; \
> -
>  #define TARGET(op) \
>      TARGET_##op: \
> -        opcode = op; \
> -        if (HAS_ARG(op)) \
> -            oparg = NEXTARG(); \
>      case op:
>
> -
>  #define DISPATCH() \
>      { \
>          if (!_Py_atomic_load_relaxed(&eval_breaker)) {      \
> @@ -917,7 +903,9 @@
>      { \
>          if (!lltrace && !_Py_TracingPossible) { \
>              f->f_lasti = INSTR_OFFSET(); \
> -            goto *opcode_targets[*next_instr++]; \
> +            opcode = NEXTOP(); \
> +            oparg = NEXTARG(); \
> +            goto *opcode_targets[opcode]; \
>          } \
>          goto fast_next_opcode; \
>      }
> @@ -926,7 +914,9 @@
>      { \
>          if (!_Py_TracingPossible) { \
>              f->f_lasti = INSTR_OFFSET(); \
> -            goto *opcode_targets[*next_instr++]; \
> +            opcode = NEXTOP(); \
> +            oparg = NEXTARG(); \
> +            goto *opcode_targets[opcode]; \
>          } \
>          goto fast_next_opcode; \
>      }
> @@ -935,10 +925,7 @@
>  #else
>  #define TARGET(op) \
>      case op:
> -#define TARGET_WITH_IMPL(op, impl) \
> -    /* silence compiler warnings about `impl` unused */ \
> -    if (0) goto impl; \
> -    case op:
> +
>  #define DISPATCH() continue
>  #define FAST_DISPATCH() goto fast_next_opcod
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160524/04eb66b8/attachment-0001.html>

From tjreedy at udel.edu  Tue May 24 13:06:00 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 24 May 2016 13:06:00 -0400
Subject: [Python-Dev] Removing the provisional label from pathlib
In-Reply-To: <CACac1F8oLH-5FjjhLhS6CpP-TJ3x1fkCDpAC9hsw3gAhcwxCEQ@mail.gmail.com>
References: <CAP1=2W7pV0joOcoA4Z1RgsrWjgJFjb_JOzJ83pk6cdRjej9_Eg@mail.gmail.com>
 <CAP7+vJJu8vpEVDGqmLxdGvmZeHw0s0XSb2MPCYRL1AyRAQLJFQ@mail.gmail.com>
 <CAP1=2W73_-g9B+NttA_GpWkyFaNnpovu2dxpaxi3DoXyiiO16Q@mail.gmail.com>
 <CALGmxEKg+kg63nBwLD0Jkh9d_TGzN3B=NW_Px6VG+Aauge5h+g@mail.gmail.com>
 <CAP1=2W6hPfMjzjwVK3MFK1rbGpb3=kQHAnBw27PrHs6fBa0WOw@mail.gmail.com>
 <CALGmxELA0D9PcrqasMRJ_9M_hPuv6VcWzYoBjnHV40sEZwidAg@mail.gmail.com>
 <CAMiohoie6O04NDm-UOby+tKxdT7zuB4gEu1WTAX86iTFO8-HRg@mail.gmail.com>
 <20160524095651.014e37c4.barry@wooz.org>
 <CAMiohogjJvjAs+X4t5LQCNv2LU5pDcBzHBPOpJvJ8uDYRP_AYg@mail.gmail.com>
 <CACac1F8oLH-5FjjhLhS6CpP-TJ3x1fkCDpAC9hsw3gAhcwxCEQ@mail.gmail.com>
Message-ID: <ni21ls$d7q$1@ger.gmane.org>

On 5/24/2016 10:49 AM, Paul Moore wrote:
> On 24 May 2016 at 15:11, Koos Zevenhoven <k7hoven at gmail.com> wrote:
>>> Please, no.  We learned that lesson in Python 2.2.1 with True/False.
>>
>> What happened? True was included in 2.2.1 but not False?-). Anyway, I
>> guess you are probably right, and "3.6->" is the way to go. Besides,
>> Guido already wrote that in the first response.

He did so because our policy rightly forbids changing os.

> The history is at
> http://python-history.blogspot.co.uk/2013/11/the-history-of-bool-true-and-false.html
> but basically we added new *constants* for True/False in a point
> release. Some people used them, and their code only worked on 2.2.1
> and above but not 2.2.0. Others were using variables called True and
> False, and the 2.2.1 change broke their code (you could no longer
> assign to those names).
>
> So essentially it ended up that people couldn't say "works on 2.2" and
> had to distinguish between 2.2.0 and 2.2.1+.
>
> Python's strict backward compatibility rules basically stem from the
> grief caused by that change.

The policy was reinforced by its violation in changing the behavior of a 
StringIO method in 3.1.2 or something like that.  For people affected 
(like me), the change resulted in a subtle bug that only manifested later.

-- 
Terry Jan Reedy


From christian at python.org  Wed May 25 06:29:12 2016
From: christian at python.org (Christian Heimes)
Date: Wed, 25 May 2016 12:29:12 +0200
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
Message-ID: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>

Hi everybody,

I have three hashing-related patches for Python 3.6 that are waiting for
review. Altogether the three patches add ten new hash algorithms to the
hashlib module: SHA3 (224, 256, 384, 512), SHAKE (SHA3 XOF 128, 256),
BLAKE2 (blake2b, blake2s) and truncated SHA512 (224, 256).


SHA-3 / SHAKE: https://bugs.python.org/issue16113
BLAKE2: https://bugs.python.org/issue26798
SHA512/224 / SHA512/256: https://bugs.python.org/issue26834


I like to push the patches during the sprints at PyCon. Please assist
with reviews.

Regards,
Christian

From leewangzhong+python at gmail.com  Wed May 25 13:23:42 2016
From: leewangzhong+python at gmail.com (Franklin? Lee)
Date: Wed, 25 May 2016 13:23:42 -0400
Subject: [Python-Dev] devinabox has moved to GitHub
In-Reply-To: <CAP1=2W7-+fNbeo57vLnMd4OptSEwvKLg+HgED96bPoxOyWNgYQ@mail.gmail.com>
References: <CAP1=2W7-+fNbeo57vLnMd4OptSEwvKLg+HgED96bPoxOyWNgYQ@mail.gmail.com>
Message-ID: <CAB_e7izU_=kGeQEEx-AHiiqiyqBc2qW_8XXLX7MjKP47k29RqQ@mail.gmail.com>

Should these notes come with version requirements/minimums?

"OS X users should be told to download XCode from the Apple App Store
ahead of time."
"If new contributors think they may be doing C development, suggest
the use of LLVM + clang as this provides better error reporting than
gcc."
"For Windows users, ask them to download and install Visual Studio
Community edition ahead of time."

On Sun, May 8, 2016 at 4:41 PM, Brett Cannon <brett at python.org> wrote:
> https://github.com/python/devinabox
>
> The single issue for devinabox has moved to its own issue tracker, so
> there's no need to worry about those issues cluttering b.p.o in the future.
> I have made the Python core team I created on GitHub last week have write
> privileges and Nick and I as admins on the repository. I have also turned on
> the CLA bot for the repository (FYI
> https://github.com/python/the-knights-who-say-ni has that bot's code).
>
> I've asked Georg, Antoine, and Benjamin to tell me what I need to do to shut
> off -- either by making it read-only or just deleting --
> hg.python.org/devinabox.
>
> Now that the migration has seriously begun, the next repos will be the peps
> and devguide (slightly more complicated thanks to needing to update commands
> for building their online versions). There's also the benchmarks repo, but
> that might not get migrated if we start from scratch (see the speed@ ML
> about that).
>
> As for cpython, I've been asked to talk about it at the language summit
> where I will start a conversation about what the minimal feature set will
> need to be to migrate. Once we have  settled on what has to be in place to
> migrate the cpython repo then we can start working on that TODO list.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/leewangzhong%2Bpython%40gmail.com
>

From brett at python.org  Wed May 25 13:37:52 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 25 May 2016 17:37:52 +0000
Subject: [Python-Dev] devinabox has moved to GitHub
In-Reply-To: <CAB_e7izU_=kGeQEEx-AHiiqiyqBc2qW_8XXLX7MjKP47k29RqQ@mail.gmail.com>
References: <CAP1=2W7-+fNbeo57vLnMd4OptSEwvKLg+HgED96bPoxOyWNgYQ@mail.gmail.com>
 <CAB_e7izU_=kGeQEEx-AHiiqiyqBc2qW_8XXLX7MjKP47k29RqQ@mail.gmail.com>
Message-ID: <CAP1=2W7pCA7EzF_Tg1PJAg3s-auG-_QkyKVB8k2U4XQqG5Ba=A@mail.gmail.com>

On Wed, 25 May 2016 at 10:24 Franklin? Lee <leewangzhong+python at gmail.com>
wrote:

> Should these notes come with version requirements/minimums?
>
> "OS X users should be told to download XCode from the Apple App Store
> ahead of time."
> "If new contributors think they may be doing C development, suggest
> the use of LLVM + clang as this provides better error reporting than
> gcc."
> "For Windows users, ask them to download and install Visual Studio
> Community edition ahead of time."
>

If you want to submit a PR to say "the latest" that's fine, but I don't
know if any of those tools really promote downloading older versions such
that one has to worry about it.


>
> On Sun, May 8, 2016 at 4:41 PM, Brett Cannon <brett at python.org> wrote:
> > https://github.com/python/devinabox
> >
> > The single issue for devinabox has moved to its own issue tracker, so
> > there's no need to worry about those issues cluttering b.p.o in the
> future.
> > I have made the Python core team I created on GitHub last week have write
> > privileges and Nick and I as admins on the repository. I have also
> turned on
> > the CLA bot for the repository (FYI
> > https://github.com/python/the-knights-who-say-ni has that bot's code).
> >
> > I've asked Georg, Antoine, and Benjamin to tell me what I need to do to
> shut
> > off -- either by making it read-only or just deleting --
> > hg.python.org/devinabox.
> >
> > Now that the migration has seriously begun, the next repos will be the
> peps
> > and devguide (slightly more complicated thanks to needing to update
> commands
> > for building their online versions). There's also the benchmarks repo,
> but
> > that might not get migrated if we start from scratch (see the speed@ ML
> > about that).
> >
> > As for cpython, I've been asked to talk about it at the language summit
> > where I will start a conversation about what the minimal feature set will
> > need to be to migrate. Once we have  settled on what has to be in place
> to
> > migrate the cpython repo then we can start working on that TODO list.
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> >
> https://mail.python.org/mailman/options/python-dev/leewangzhong%2Bpython%40gmail.com
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160525/1065841d/attachment.html>

From leewangzhong+python at gmail.com  Wed May 25 14:54:01 2016
From: leewangzhong+python at gmail.com (Franklin? Lee)
Date: Wed, 25 May 2016 14:54:01 -0400
Subject: [Python-Dev] devinabox has moved to GitHub
In-Reply-To: <CAP1=2W7pCA7EzF_Tg1PJAg3s-auG-_QkyKVB8k2U4XQqG5Ba=A@mail.gmail.com>
References: <CAP1=2W7-+fNbeo57vLnMd4OptSEwvKLg+HgED96bPoxOyWNgYQ@mail.gmail.com>
 <CAB_e7izU_=kGeQEEx-AHiiqiyqBc2qW_8XXLX7MjKP47k29RqQ@mail.gmail.com>
 <CAP1=2W7pCA7EzF_Tg1PJAg3s-auG-_QkyKVB8k2U4XQqG5Ba=A@mail.gmail.com>
Message-ID: <CAB_e7ixVb-abURq1KKPnpn=hceSZiGcPwm+M+DP+=du+gtkW4w@mail.gmail.com>

It's just that I don't know whether any of them require particular
versions. If you say the latest is fine, then okay.

On Wed, May 25, 2016 at 1:37 PM, Brett Cannon <brett at python.org> wrote:
>
>
> On Wed, 25 May 2016 at 10:24 Franklin? Lee <leewangzhong+python at gmail.com>
> wrote:
>>
>> Should these notes come with version requirements/minimums?
>>
>> "OS X users should be told to download XCode from the Apple App Store
>> ahead of time."
>> "If new contributors think they may be doing C development, suggest
>> the use of LLVM + clang as this provides better error reporting than
>> gcc."
>> "For Windows users, ask them to download and install Visual Studio
>> Community edition ahead of time."
>
>
> If you want to submit a PR to say "the latest" that's fine, but I don't know
> if any of those tools really promote downloading older versions such that
> one has to worry about it.
>
>>
>>
>> On Sun, May 8, 2016 at 4:41 PM, Brett Cannon <brett at python.org> wrote:
>> > https://github.com/python/devinabox
>> >
>> > The single issue for devinabox has moved to its own issue tracker, so
>> > there's no need to worry about those issues cluttering b.p.o in the
>> > future.
>> > I have made the Python core team I created on GitHub last week have
>> > write
>> > privileges and Nick and I as admins on the repository. I have also
>> > turned on
>> > the CLA bot for the repository (FYI
>> > https://github.com/python/the-knights-who-say-ni has that bot's code).
>> >
>> > I've asked Georg, Antoine, and Benjamin to tell me what I need to do to
>> > shut
>> > off -- either by making it read-only or just deleting --
>> > hg.python.org/devinabox.
>> >
>> > Now that the migration has seriously begun, the next repos will be the
>> > peps
>> > and devguide (slightly more complicated thanks to needing to update
>> > commands
>> > for building their online versions). There's also the benchmarks repo,
>> > but
>> > that might not get migrated if we start from scratch (see the speed@ ML
>> > about that).
>> >
>> > As for cpython, I've been asked to talk about it at the language summit
>> > where I will start a conversation about what the minimal feature set
>> > will
>> > need to be to migrate. Once we have  settled on what has to be in place
>> > to
>> > migrate the cpython repo then we can start working on that TODO list.
>> >
>> > _______________________________________________
>> > Python-Dev mailing list
>> > Python-Dev at python.org
>> > https://mail.python.org/mailman/listinfo/python-dev
>> > Unsubscribe:
>> >
>> > https://mail.python.org/mailman/options/python-dev/leewangzhong%2Bpython%40gmail.com
>> >

From brett at snarky.ca  Wed May 25 13:45:55 2016
From: brett at snarky.ca (Brett Cannon)
Date: Wed, 25 May 2016 17:45:55 +0000
Subject: [Python-Dev] [Python-checkins] peps: Short subsection on
 annotating coroutines (Ivan L, #225).
In-Reply-To: <20160524192004.14798.31017.1CC1E33A@psf.io>
References: <20160524192004.14798.31017.1CC1E33A@psf.io>
Message-ID: <CAP1=2W66XeP=d0ktrgAAdYNYeX6oogu7xDim51L83YOUtU5s7A@mail.gmail.com>

On Tue, 24 May 2016 at 12:20 guido.van.rossum <python-checkins at python.org>
wrote:

> https://hg.python.org/peps/rev/50c3f5aefbb7
> changeset:   6341:50c3f5aefbb7
> user:        Guido van Rossum <guido at python.org>
> date:        Tue May 24 12:18:54 2016 -0700
> summary:
>   Short subsection on annotating coroutines (Ivan L, #225).
>
> files:
>   pep-0484.txt |  34 ++++++++++++++++++++++++++++++++++
>   1 files changed, 34 insertions(+), 0 deletions(-)
>
>
> diff --git a/pep-0484.txt b/pep-0484.txt
> --- a/pep-0484.txt
> +++ b/pep-0484.txt
> @@ -1015,6 +1015,40 @@
>  ellipsis, i.e. the above example is literally what you would write.
>
>
> +Annotating generator functions and coroutines
> +---------------------------------------------
> +
> +The return type of generator functions can be annotated by
> +the generic type ``Generator[yield_type, send_type,
> +return_type]`` provided by ``typing.py`` module::
> +
> +  def echo_round() -> Generator[int, float, str]:
> +      res = yield
> +      while res:
> +          res = yield round(res)
> +      return 'OK'
> +
> +Coroutines introduced in PEP 492 are annotated with the same syntax as
> +ordinary functions. However, the return type annotation corresponds to the
> +type of ``await`` expression, not to the coroutine type::
> +
> +  async def spam(ignored: int) -> str:
> +      return 'spam'
> +
> +  async def foo():
> +      bar = await spam(42) # type: str
>

If the coroutine has multiple await expressions that return different types
then do you simply use a Union to express that?

-Brett



> +
> +The ``typing.py`` module also provides generic ABCs ``Awaitable``,
> +``AsyncIterable``, and ``AsyncIterator`` for situations where more precise
> +types cannot be specified::
> +
> +  def op() -> typing.Awaitable[str]:
> +      if cond:
> +          return spam(42)
> +      else:
> +          return asyncio.Future(...)
> +
> +
>  Compatibility with other uses of function annotations
>  =====================================================
>
>
> --
> Repository URL: https://hg.python.org/peps
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> https://mail.python.org/mailman/listinfo/python-checkins
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160525/dff70b60/attachment.html>

From chris.barker at noaa.gov  Wed May 25 15:29:48 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 25 May 2016 12:29:48 -0700
Subject: [Python-Dev] runtime dlls on Windows
Message-ID: <CALGmxEKA2JepnHACMiwJdzVhgiRokeDfxnNv+_qBKWdNCM4g6w@mail.gmail.com>

Hi folks,

The standard build of Py3.5 for Windows is built with VS2015 (correct??)
And it includes the runtime dlls it needs.

However, we've found that wxPython wheels for win32 (not sure about win64)
also need:

MSVCP140.DLL

So: wxPython could include that of course, But it looks like it's getting
included with the Matplotlib wheels already (so folks with Matplotlib can
run wx....). I'm just guessing, but this looks like the standard run time
for C++ with that compiler.

Python itself doesn't use C++, of course, but maybe we should include that
dll with Python anyway -- that way folks can build wheels of packages with
C++ extensions  in the normal way, and those wheels will "just work", and
we don't have to have every individual package ship the same dll.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160525/614a94b2/attachment.html>

From rymg19 at gmail.com  Wed May 25 15:37:48 2016
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Wed, 25 May 2016 14:37:48 -0500
Subject: [Python-Dev] runtime dlls on Windows
In-Reply-To: <CALGmxEKA2JepnHACMiwJdzVhgiRokeDfxnNv+_qBKWdNCM4g6w@mail.gmail.com>
References: <CALGmxEKA2JepnHACMiwJdzVhgiRokeDfxnNv+_qBKWdNCM4g6w@mail.gmail.com>
Message-ID: <CAO41-mPLvUtUQy9ELdDwus5r=ic7A=oZxsPf6PORg2Nu+Wrb6w@mail.gmail.com>

Wouldn't downloading the Microsoft C++ Runtime 2015 also work? Many recent
computers already have it pre-installed.

--
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
On May 25, 2016 2:31 PM, "Chris Barker" <chris.barker at noaa.gov> wrote:

> Hi folks,
>
> The standard build of Py3.5 for Windows is built with VS2015 (correct??)
> And it includes the runtime dlls it needs.
>
> However, we've found that wxPython wheels for win32 (not sure about win64)
> also need:
>
> MSVCP140.DLL
>
> So: wxPython could include that of course, But it looks like it's getting
> included with the Matplotlib wheels already (so folks with Matplotlib can
> run wx....). I'm just guessing, but this looks like the standard run time
> for C++ with that compiler.
>
> Python itself doesn't use C++, of course, but maybe we should include that
> dll with Python anyway -- that way folks can build wheels of packages with
> C++ extensions  in the normal way, and those wheels will "just work", and
> we don't have to have every individual package ship the same dll.
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160525/7d739d17/attachment.html>

From chris.barker at noaa.gov  Wed May 25 16:26:54 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 25 May 2016 13:26:54 -0700
Subject: [Python-Dev] runtime dlls on Windows
In-Reply-To: <CAO41-mPLvUtUQy9ELdDwus5r=ic7A=oZxsPf6PORg2Nu+Wrb6w@mail.gmail.com>
References: <CALGmxEKA2JepnHACMiwJdzVhgiRokeDfxnNv+_qBKWdNCM4g6w@mail.gmail.com>
 <CAO41-mPLvUtUQy9ELdDwus5r=ic7A=oZxsPf6PORg2Nu+Wrb6w@mail.gmail.com>
Message-ID: <CALGmxE+mHLz0-=7yVx6vgKHbCmbbo7Ern+=0baa+ZZ7xHLyY=g@mail.gmail.com>

On Wed, May 25, 2016 at 12:37 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:

> Wouldn't downloading the Microsoft C++ Runtime 2015 also work?
>
I'm sure it would -- I know that installing the entire MSVC2015 Community
Edition does...

but the point here is that end users should be able to:

pip install something

and if there is a binary wheel for something, it should work without them
having to install something else. (why MS doesn't ship ALL their runtimes
with eh OS is beyond me...)

As it stands now, there are two options:

1) users need to install something else themselves: the runtime, that
particular dll, the compiler...
2) every package maintainer that uses C++ needs to ship that dll with the
binary wheels.

If we put the dll into the python binary, then it would "just work"

-CHB





> Many recent computers already have it pre-installed.
>
> --
> Ryan
> [ERROR]: Your autotools build scripts are 200 lines longer than your
> program. Something?s wrong.
> http://kirbyfan64.github.io/
> On May 25, 2016 2:31 PM, "Chris Barker" <chris.barker at noaa.gov> wrote:
>
>> Hi folks,
>>
>> The standard build of Py3.5 for Windows is built with VS2015 (correct??)
>> And it includes the runtime dlls it needs.
>>
>> However, we've found that wxPython wheels for win32 (not sure about
>> win64) also need:
>>
>> MSVCP140.DLL
>>
>> So: wxPython could include that of course, But it looks like it's getting
>> included with the Matplotlib wheels already (so folks with Matplotlib can
>> run wx....). I'm just guessing, but this looks like the standard run time
>> for C++ with that compiler.
>>
>> Python itself doesn't use C++, of course, but maybe we should include
>> that dll with Python anyway -- that way folks can build wheels of packages
>> with C++ extensions  in the normal way, and those wheels will "just work",
>> and we don't have to have every individual package ship the same dll.
>>
>> -CHB
>>
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR&R            (206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115       (206) 526-6317   main reception
>>
>> Chris.Barker at noaa.gov
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
>>
>>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160525/6a4218e6/attachment-0001.html>

From rosuav at gmail.com  Wed May 25 16:40:19 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 26 May 2016 06:40:19 +1000
Subject: [Python-Dev] runtime dlls on Windows
In-Reply-To: <CALGmxE+mHLz0-=7yVx6vgKHbCmbbo7Ern+=0baa+ZZ7xHLyY=g@mail.gmail.com>
References: <CALGmxEKA2JepnHACMiwJdzVhgiRokeDfxnNv+_qBKWdNCM4g6w@mail.gmail.com>
 <CAO41-mPLvUtUQy9ELdDwus5r=ic7A=oZxsPf6PORg2Nu+Wrb6w@mail.gmail.com>
 <CALGmxE+mHLz0-=7yVx6vgKHbCmbbo7Ern+=0baa+ZZ7xHLyY=g@mail.gmail.com>
Message-ID: <CAPTjJmpqTCSkwwonvDumCPpqT3aYcSOiyPt463=O2PwFUnvF_A@mail.gmail.com>

On Thu, May 26, 2016 at 6:26 AM, Chris Barker <chris.barker at noaa.gov> wrote:
> the point here is that end users should be able to:
>
> pip install something
>
> and if there is a binary wheel for something, it should work without them
> having to install something else. (why MS doesn't ship ALL their runtimes
> with eh OS is beyond me...)

Agreed with that, but meh.

> As it stands now, there are two options:
>
> 1) users need to install something else themselves: the runtime, that
> particular dll, the compiler...
> 2) every package maintainer that uses C++ needs to ship that dll with the
> binary wheels.
>
> If we put the dll into the python binary, then it would "just work"

Counting your conclusion as option 3, you're offering the status quo
and two competing solutions to it. I agree that the status quo is
unideal; a binary wheel should either include or be able to fetch
everything necessary for a supported platform. But why should CPython
package a runtime that it doesn't use? Which is more common - someone
uses two C++ modules, or someone uses none of them?

I don't know how hard it is for the wheels to ship the DLL ("hard"
here including any licensing or versioning issues, as well as the
actual effort involved), but from a purely logical standpoint, it
seems the most sane option. So I'm in favour of your option 2.

ChrisA

From mafagafogigante at gmail.com  Wed May 25 17:05:52 2016
From: mafagafogigante at gmail.com (Bernardo Sulzbach)
Date: Wed, 25 May 2016 18:05:52 -0300
Subject: [Python-Dev] runtime dlls on Windows
In-Reply-To: <CAO41-mPLvUtUQy9ELdDwus5r=ic7A=oZxsPf6PORg2Nu+Wrb6w@mail.gmail.com>
References: <CALGmxEKA2JepnHACMiwJdzVhgiRokeDfxnNv+_qBKWdNCM4g6w@mail.gmail.com>
 <CAO41-mPLvUtUQy9ELdDwus5r=ic7A=oZxsPf6PORg2Nu+Wrb6w@mail.gmail.com>
Message-ID: <4c3c0dcb-1050-86ec-a229-3a4fbe79ec41@gmail.com>

On 05/25/2016 04:37 PM, Ryan Gonzalez wrote:
> Wouldn't downloading the Microsoft C++ Runtime 2015 also work? Many recent
> computers already have it pre-installed.

Even though the download seems to be only 14 MB (I don't have a Windows 
machine, so I cannot assure that just the file on MS website is enough), 
it is an inconvenience. Additionally, I don't know how clear it will be 
for the average Python user on Windows that a C++ runtime is missing.

From njs at pobox.com  Wed May 25 17:52:58 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 25 May 2016 14:52:58 -0700
Subject: [Python-Dev] runtime dlls on Windows
In-Reply-To: <CALGmxEKA2JepnHACMiwJdzVhgiRokeDfxnNv+_qBKWdNCM4g6w@mail.gmail.com>
References: <CALGmxEKA2JepnHACMiwJdzVhgiRokeDfxnNv+_qBKWdNCM4g6w@mail.gmail.com>
Message-ID: <CAPJVwB=VTguAhHiGRXdPs4-PQD=Mfgs=98OtLNYaGLnDokeFmA@mail.gmail.com>

On Wed, May 25, 2016 at 12:29 PM, Chris Barker <chris.barker at noaa.gov> wrote:
>
> Hi folks,
>
> The standard build of Py3.5 for Windows is built with VS2015 (correct??) And it includes the runtime dlls it needs.
>
> However, we've found that wxPython wheels for win32 (not sure about win64) also need:
>
> MSVCP140.DLL
>
> So: wxPython could include that of course, But it looks like it's getting included with the Matplotlib wheels already (so folks with Matplotlib can run wx....). I'm just guessing, but this looks like the standard run time for C++ with that compiler.
>
> Python itself doesn't use C++, of course, but maybe we should include that dll with Python anyway -- that way folks can build wheels of packages with C++ extensions  in the normal way, and those wheels will "just work", and we don't have to have every individual package ship the same dll.

The other challenge with this proposal is that Python 3.5.0 and 3.5.1
have already shipped without this .dll. So the most we could hope for
is for it to be included in 3.5.2, and then wxPython and Matplotlib
would either have to continue shipping it anyway, or else accept that
their wheels actually require 3.5.2+ and will be broken if installed
onto 3.5.0 or 3.5.1. And unfortunately there's no way in wheel
metadata to express this kind of requirement -- wheels assume that all
3.5.x are both forwards and backwards compatible. So it would just be
a weird unexplained breakage that some users would see.

An alternative approach would be to stick MSVCP140.DLL into a tiny
shim wheel and upload that to PyPI, and then wxPython and matplotlib's
windows wheels could declare a dependency on this msvcp410 wheel.

Basically this is the idea of my pynativelib proposal, though in this
case you would only be using a small piece of the full proposal:
   https://mail.python.org/pipermail/wheel-builders/2016-April/000090.html

(I really need to get back to that... if anyone wants to help sprint
on this at PyCon let me know :-))

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From steve.dower at python.org  Wed May 25 17:20:55 2016
From: steve.dower at python.org (Steve Dower)
Date: Wed, 25 May 2016 14:20:55 -0700
Subject: [Python-Dev] runtime dlls on Windows
In-Reply-To: <CALGmxEKA2JepnHACMiwJdzVhgiRokeDfxnNv+_qBKWdNCM4g6w@mail.gmail.com>
References: <CALGmxEKA2JepnHACMiwJdzVhgiRokeDfxnNv+_qBKWdNCM4g6w@mail.gmail.com>
Message-ID: <43d238d5-c5a5-057f-b509-dcadfb34bc60@python.org>

On 25May2016 1229, Chris Barker wrote:
> Hi folks,
>
> The standard build of Py3.5 for Windows is built with VS2015 (correct??)
> And it includes the runtime dlls it needs.
>
> However, we've found that wxPython wheels for win32 (not sure about
> win64) also need:
>
> MSVCP140.DLL

There are two different versions of this DLL for each architecture (same 
name).

> So: wxPython could include that of course, But it looks like it's
> getting included with the Matplotlib wheels already (so folks with
> Matplotlib can run wx....). I'm just guessing, but this looks like the
> standard run time for C++ with that compiler.
>
> Python itself doesn't use C++, of course, but maybe we should include
> that dll with Python anyway -- that way folks can build wheels of
> packages with C++ extensions  in the normal way, and those wheels will
> "just work", and we don't have to have every individual package ship the
> same dll.

Unfortunately, it won't "just work". More often than not, it will break 
in weird and very difficult to diagnose ways, as the version of 
msvcp140.dll changes on a regular basis (there are at least four 
different version of it "in the wild" since VS 2015 released, not 
counting the preview releases before mid-last year).

Importantly, it will break forward compatibility. We are already on the 
hook to keep shipping vcruntime140.dll for all 3.5 (and probably 3.6) 
releases, and if binary packages are built with later versions of VS 
then they'll need to include a (hypothetical) vcruntime150.dll (which 
distutils will already do, because I added that functionality). Other 
than that, every dependency of Python 3.5+ is forwards-compatible with 
new operating systems and compilers.

There's also a slippery slope argument that would legitimately justify 
shipping all sorts of "common" dependencies with the core product. I 
chose to draw the line at "needed by the core Python product", rather 
than "created by Microsoft" or "used by many packages". (Hence 
vcruntime140.dll is included, despite not having forwards-compatibility, 
and I tried real hard to avoid including that one too.)

Finally, unless we demand users are administrators for every install, we 
will quickly have many Python installs using versions of msvcp140.dll 
(or any other dependency) with security vulnerabilities, with no way to 
update them other than a full Python release. Installing the regular 
runtime (which is patched automatically) or as part of a package (which 
can be independently updated) is far more feasible. I really don't want 
to be doing security updates for 15 years just to keep an unused DLL secure.

Hopefully that helps explain the position somewhat. I'm happy to go into 
more detail on any of these issues if anyone would like (potentially 
just linking to where I've previously discussed it either here, on 
bugs.p.o or on my blog).

Cheers,
Steve


From ncoghlan at gmail.com  Thu May 26 01:34:55 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 26 May 2016 15:34:55 +1000
Subject: [Python-Dev] devinabox has moved to GitHub
In-Reply-To: <CAB_e7ixVb-abURq1KKPnpn=hceSZiGcPwm+M+DP+=du+gtkW4w@mail.gmail.com>
References: <CAP1=2W7-+fNbeo57vLnMd4OptSEwvKLg+HgED96bPoxOyWNgYQ@mail.gmail.com>
 <CAB_e7izU_=kGeQEEx-AHiiqiyqBc2qW_8XXLX7MjKP47k29RqQ@mail.gmail.com>
 <CAP1=2W7pCA7EzF_Tg1PJAg3s-auG-_QkyKVB8k2U4XQqG5Ba=A@mail.gmail.com>
 <CAB_e7ixVb-abURq1KKPnpn=hceSZiGcPwm+M+DP+=du+gtkW4w@mail.gmail.com>
Message-ID: <CADiSq7dxnjqmfXUFi8U8H6=x+n5CKB-q113kFg+4+1xgbPQMeg@mail.gmail.com>

On 26 May 2016 at 04:54, Franklin? Lee <leewangzhong+python at gmail.com> wrote:
> It's just that I don't know whether any of them require particular
> versions. If you say the latest is fine, then okay.

For working on CPython trunk, the latest is fine. Things only have the
potential to get trickier when building extensions for older Python
versions (since you need to make sure the binaries are compatible),
but that's not a topic devinabox concerns itself with.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From Artyom.Skrobov at arm.com  Thu May 26 06:19:05 2016
From: Artyom.Skrobov at arm.com (Artyom Skrobov)
Date: Thu, 26 May 2016 10:19:05 +0000
Subject: [Python-Dev] Python parser performance optimizations
Message-ID: <DB6PR0801MB1400C8D31C17EFC547E8544287410@DB6PR0801MB1400.eurprd08.prod.outlook.com>

Hello,

Back in March, I've posted a patch at http://bugs.python.org/issue26526 -- "In parsermodule.c, replace over 2KLOC of hand-crafted validation code, with a DFA".

The motivation for this patch was to enable a memory footprint optimization, discussed at http://bugs.python.org/issue26415
My proposed optimization reduces the memory footprint by up to 30% on the standard benchmarks, and by 200% on a degenerate case which sparked the discussion.
The run time stays unaffected by this optimization.

Python Developer's Guide says: "If you don't get a response within a few days after pinging the issue, then you can try emailing python-dev at python.org<mailto:python-dev at python.org> asking for someone to review your patch."

So, here I am.
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160526/19f1d423/attachment.html>

From chris.barker at noaa.gov  Thu May 26 12:42:56 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 26 May 2016 09:42:56 -0700
Subject: [Python-Dev] runtime dlls on Windows
In-Reply-To: <CAPTjJmpqTCSkwwonvDumCPpqT3aYcSOiyPt463=O2PwFUnvF_A@mail.gmail.com>
References: <CALGmxEKA2JepnHACMiwJdzVhgiRokeDfxnNv+_qBKWdNCM4g6w@mail.gmail.com>
 <CAO41-mPLvUtUQy9ELdDwus5r=ic7A=oZxsPf6PORg2Nu+Wrb6w@mail.gmail.com>
 <CALGmxE+mHLz0-=7yVx6vgKHbCmbbo7Ern+=0baa+ZZ7xHLyY=g@mail.gmail.com>
 <CAPTjJmpqTCSkwwonvDumCPpqT3aYcSOiyPt463=O2PwFUnvF_A@mail.gmail.com>
Message-ID: <CALGmxELf2vRdPs6dX-S5oBDaKpFry+FqnmGOcav3QDvNZi1BzQ@mail.gmail.com>

On Wed, May 25, 2016 at 1:40 PM, Chris Angelico <rosuav at gmail.com> wrote:

> But why should CPython
> package a runtime that it doesn't use?


Because it IS part of the "standard runtime environment" that cPython is
providing. The cPython Windows builds are built with a particular compiler
that expects a particular runtime. C++ is part of that runtime, even if the
cPython code itself doesn't use that part. I'm sure there are features in
the C runtime DLL that cPython doesn't happen to use, either. The fact that
it's a separate dll is kind of an implementation detail.


> Which is more common - someone
> uses two C++ modules, or someone uses none of them?
>

Irrelevant -- which is the bigger problem for users? a lot of folks getting
an extra dll as part of a large install that they  aren't using, or folks
getting cryptic error messages and/or every package builder of C++ packages
having to identify and solve the problem?

BUT -- Steve Dower seems to have identified that the wonders of dll hell
never cease, and this isn't possible anyway. Oh well.


I don't know how hard it is for the wheels to ship the DLL ("hard"
> here including any licensing or versioning issues, as well as the
> actual effort involved),


It's not all that hard. but one more step that has to be done. I wonder if
setuptools / wheel could automate this somehow? -- hmm.

On Wed, May 25, 2016 at 2:05 PM, Bernardo Sulzbach <
mafagafogigante at gmail.com> wrote:

> I don't know how clear it will be for the average Python user on Windows
> that a C++ runtime is missing.


That's the key problem -- the error message is NOT helpful -- it makes it
look like the extension dll is missing, when it's actually that the
extension dll can't link to the runtime dll -- but the message doesn't say
what dll it's trying to link to. sigh. You need to be a pretty
sophisticated user to figure it out -- or use Google :-)

On Wed, May 25, 2016 at 2:52 PM, Nathaniel Smith <njs at pobox.com> wrote:

> The other challenge with this proposal is that Python 3.5.0 and 3.5.1
> have already shipped without this .dll.


yeah, maybe only 3.6 then, if possible at all.


>  wxPython and Matplotlib
> would either have to continue shipping it anyway, or else accept that
> their wheels actually require 3.5.2+ and will be broken if installed
> onto 3.5.0 or 3.5.1.


(unless the user has that dll ;-) )

And to be clear, this is going to effect any package that used C++ -- those
two are two I happen to be familiar with. There can't be a small number of
such packages.

(hmm -- GDAL, anything Boost-based, anything else that uses AGG, .....)


> An alternative approach would be to stick MSVCP140.DLL into a tiny
> shim wheel and upload that to PyPI, and then wxPython and matplotlib's
> windows wheels could declare a dependency on this msvcp410 wheel.
>
> Basically this is the idea of my pynativelib proposal,


Yes -- if pynativelib becomes standard practice, then yes, that would be a
good way to handle this issue -- jsut ike any other lib multiple packages
might need.

(I really need to get back to that... if anyone wants to help sprint
> on this at PyCon let me know :-))
>

I might -- haven't decided what to sprint on yet -- let's talk. (let's talk
anyway)

On Wed, May 25, 2016 at 2:20 PM, Steve Dower <steve.dower at python.org> wrote:

> MSVCP140.DLL
>>
>
> There are two different versions of this DLL for each architecture (same
> name).


got to love Windows! Does the OS do the right thing if they are both
installed? ouch!


> Unfortunately, it won't "just work". More often than not, it will break in
>> weird and very difficult to diagnose ways, as the version of msvcp140.dll
>> changes on a regular basis (there are at least four different version of it
>> "in the wild" since VS 2015 released, not counting the preview releases
>> before mid-last year).
>>
>
oh God. Well, that will  pretty much kill it. So essentially every single
app or lib written in C++ has to ship this dll? And IIUC, folks that have
the compiler installed have SOME version on the system paths, and who knows
what version other arbitrary stuff is linked to? How can this possibly be
considered "Enterprise" software? Wow!

Importantly, it will break forward compatibility. We are already on the
> hook to keep shipping vcruntime140.dll for all 3.5 (and probably 3.6)
> releases, and if binary packages are built with later versions of VS then
> they'll need to include a (hypothetical) vcruntime150.dll (which distutils
> will already do, because I added that functionality).


wow. So could that be extended to do the same thing with MSVCP140.DLL? then
each package would be shipping it -- but it would happen automagically,
rather than everyone having to figure out that they have to do it, and how
to do it.


> There's also a slippery slope argument that would legitimately justify
> shipping all sorts of "common" dependencies with the core product. I chose
> to draw the line at "needed by the core Python product", rather than
> "created by Microsoft" or "used by many packages". (Hence vcruntime140.dll
> is included, despite not having forwards-compatibility, and I tried real
> hard to avoid including that one too.)
>

yeah, a line has to be drawn, and "created by Microsoft" wouldn't make any
sense either. but "standard runtime expected by the compiler" would -- but,
as you say above, you have may have to ship a new version with your
extension anyway -- so it seems adding the feature for distutils to do that
for C++ extensions would both solve the problem and not ship anything extra
with the core install.


> Finally, unless we demand users are administrators for every install, we
> will quickly have many Python installs using versions of msvcp140.dll (or
> any other dependency) with security vulnerabilities, with no way to update
> them other than a full Python release. Installing the regular runtime
> (which is patched automatically) or as part of a package (which can be
> independently updated) is far more feasible. I really don't want to be
> doing security updates for 15 years just to keep an unused DLL secure.
>

Though we have to do that for msvcp140.dll anyway, yes? god, what a pain!
(and why MS should ship the damn things themselves!)

Hopefully that helps explain the position somewhat. I'm happy to go into
> more detail on any of these issues if anyone would like (potentially just
> linking to where I've previously discussed it either here, on bugs.p.o or
> on my blog).


Thanks Steve. Will  you be at PyCon? if Nathaniel and I look at this during
the sprints, maybe you could coach us a bit.

Anyway, proposal withdrawn.

Replaced with possible proposal to have distutils include the C++ runtime
dll like it does now for the C one.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160526/6c32d718/attachment.html>

From rosuav at gmail.com  Thu May 26 12:53:27 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 27 May 2016 02:53:27 +1000
Subject: [Python-Dev] runtime dlls on Windows
In-Reply-To: <CALGmxELf2vRdPs6dX-S5oBDaKpFry+FqnmGOcav3QDvNZi1BzQ@mail.gmail.com>
References: <CALGmxEKA2JepnHACMiwJdzVhgiRokeDfxnNv+_qBKWdNCM4g6w@mail.gmail.com>
 <CAO41-mPLvUtUQy9ELdDwus5r=ic7A=oZxsPf6PORg2Nu+Wrb6w@mail.gmail.com>
 <CALGmxE+mHLz0-=7yVx6vgKHbCmbbo7Ern+=0baa+ZZ7xHLyY=g@mail.gmail.com>
 <CAPTjJmpqTCSkwwonvDumCPpqT3aYcSOiyPt463=O2PwFUnvF_A@mail.gmail.com>
 <CALGmxELf2vRdPs6dX-S5oBDaKpFry+FqnmGOcav3QDvNZi1BzQ@mail.gmail.com>
Message-ID: <CAPTjJmr4cRvJS5xZMLVffAL5kVkkmfcRMkBGwTeEL5gr-q+oaA@mail.gmail.com>

On Fri, May 27, 2016 at 2:42 AM, Chris Barker <chris.barker at noaa.gov> wrote:
> On Wed, May 25, 2016 at 1:40 PM, Chris Angelico <rosuav at gmail.com> wrote:
>>
>> But why should CPython
>> package a runtime that it doesn't use?
>
>
> Because it IS part of the "standard runtime environment" that cPython is
> providing. The cPython Windows builds are built with a particular compiler
> that expects a particular runtime. C++ is part of that runtime, even if the
> cPython code itself doesn't use that part. I'm sure there are features in
> the C runtime DLL that cPython doesn't happen to use, either. The fact that
> it's a separate dll is kind of an implementation detail.

I'm not sure that it is an implementation detail; it's indicative that
the compiler/runtime designers intended for the C runtime to be able
to be distributed without the C++ runtime. You wouldn't expect to have
to ship a Fortran runtime and a Cobol runtime, even if MSVC happened
to package up compilers for all of the above, would you?

>> Which is more common - someone
>> uses two C++ modules, or someone uses none of them?
>
>
> Irrelevant -- which is the bigger problem for users? a lot of folks getting
> an extra dll as part of a large install that they  aren't using, or folks
> getting cryptic error messages and/or every package builder of C++ packages
> having to identify and solve the problem?

I agree with you that end users getting cryptic error messages is a
problem. But the choice of solutions is:

1) CPython ships the C++ runtime that it doesn't use, in case someone
pip-installs something that needs it
2) Package authors who choose to use C++ must ship the C++ runtime.

Even apart from the DLL Hell that Steve mentioned, the second option
appears more sane. Package authors get the freedom to choose any
runtime dependencies they want, and then they package them or make it
possible to obtain them.

> BUT -- Steve Dower seems to have identified that the wonders of dll hell
> never cease, and this isn't possible anyway. Oh well.
>

I'm not entirely grasping what's happening here. There are multiple
versions of msvcp140.dll floating around out there; what happens if
one app requires one of them and another requires another? Can you
simply say "take the newer one"? Is there a way to instantly recognize
that there's a newer one available, and ignore the older one?
Versioning is a *pain*.

ChrisA

From chris.barker at noaa.gov  Thu May 26 13:12:53 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 26 May 2016 10:12:53 -0700
Subject: [Python-Dev] runtime dlls on Windows
In-Reply-To: <CAPTjJmr4cRvJS5xZMLVffAL5kVkkmfcRMkBGwTeEL5gr-q+oaA@mail.gmail.com>
References: <CALGmxEKA2JepnHACMiwJdzVhgiRokeDfxnNv+_qBKWdNCM4g6w@mail.gmail.com>
 <CAO41-mPLvUtUQy9ELdDwus5r=ic7A=oZxsPf6PORg2Nu+Wrb6w@mail.gmail.com>
 <CALGmxE+mHLz0-=7yVx6vgKHbCmbbo7Ern+=0baa+ZZ7xHLyY=g@mail.gmail.com>
 <CAPTjJmpqTCSkwwonvDumCPpqT3aYcSOiyPt463=O2PwFUnvF_A@mail.gmail.com>
 <CALGmxELf2vRdPs6dX-S5oBDaKpFry+FqnmGOcav3QDvNZi1BzQ@mail.gmail.com>
 <CAPTjJmr4cRvJS5xZMLVffAL5kVkkmfcRMkBGwTeEL5gr-q+oaA@mail.gmail.com>
Message-ID: <CALGmxEJSvjCFFmGBGbPiLBC=2nJ=NqjJdX8d-tMf8RN70dm14A@mail.gmail.com>

On Thu, May 26, 2016 at 9:53 AM, Chris Angelico <rosuav at gmail.com> wrote:

> > BUT -- Steve Dower seems to have identified that the wonders of dll hell
> > never cease, and this isn't possible anyway. Oh well.
>
> I'm not entirely grasping what's happening here.


me neither :-(


> There are multiple
> versions of msvcp140.dll floating around out there; what happens if
> one app requires one of them and another requires another?


That's pretty much the dll hell that has plagued Windows FOREVER! why they
don't do some versioning is totally beyond me, though 8.3 filenames have
not been a restriction for a long time.


> Can you simply say "take the newer one"?


I *think* that standard Windows policy/practice is that newer versions of a
dll with the same name should always be backward compatible -- so something
built against an older version should work with the newer one -- so you can
ship a dll with a security patch, and it will all work. I presume that MS
itself is careful about this with their runtimes.


> Is there a way to instantly recognize
> that there's a newer one available, and ignore the older one?
>

I don't think so -- the linker will pick the first one it finds on the
search path :-(

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160526/14e0f73a/attachment.html>

From brett at python.org  Thu May 26 14:02:12 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 26 May 2016 18:02:12 +0000
Subject: [Python-Dev] runtime dlls on Windows
In-Reply-To: <CALGmxELf2vRdPs6dX-S5oBDaKpFry+FqnmGOcav3QDvNZi1BzQ@mail.gmail.com>
References: <CALGmxEKA2JepnHACMiwJdzVhgiRokeDfxnNv+_qBKWdNCM4g6w@mail.gmail.com>
 <CAO41-mPLvUtUQy9ELdDwus5r=ic7A=oZxsPf6PORg2Nu+Wrb6w@mail.gmail.com>
 <CALGmxE+mHLz0-=7yVx6vgKHbCmbbo7Ern+=0baa+ZZ7xHLyY=g@mail.gmail.com>
 <CAPTjJmpqTCSkwwonvDumCPpqT3aYcSOiyPt463=O2PwFUnvF_A@mail.gmail.com>
 <CALGmxELf2vRdPs6dX-S5oBDaKpFry+FqnmGOcav3QDvNZi1BzQ@mail.gmail.com>
Message-ID: <CAP1=2W4V6DvPCedyU7W_v1Yt+iu3BaFE1KDNc5dBqvzdjFFR5w@mail.gmail.com>

On Thu, 26 May 2016 at 09:44 Chris Barker <chris.barker at noaa.gov> wrote:

> [SNIP]
> Thanks Steve. Will  you be at PyCon? if Nathaniel and I look at this
> during the sprints, maybe you could coach us a bit.
>

Steve will be at PyCon but I don't think he will be around for the sprints
as he has to catch a flight out for PyCon Taiwan.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160526/1ceeb477/attachment.html>

From steve.dower at python.org  Thu May 26 15:13:10 2016
From: steve.dower at python.org (Steve Dower)
Date: Thu, 26 May 2016 12:13:10 -0700
Subject: [Python-Dev] runtime dlls on Windows
In-Reply-To: <CALGmxELf2vRdPs6dX-S5oBDaKpFry+FqnmGOcav3QDvNZi1BzQ@mail.gmail.com>
References: <CALGmxEKA2JepnHACMiwJdzVhgiRokeDfxnNv+_qBKWdNCM4g6w@mail.gmail.com>
 <CAO41-mPLvUtUQy9ELdDwus5r=ic7A=oZxsPf6PORg2Nu+Wrb6w@mail.gmail.com>
 <CALGmxE+mHLz0-=7yVx6vgKHbCmbbo7Ern+=0baa+ZZ7xHLyY=g@mail.gmail.com>
 <CAPTjJmpqTCSkwwonvDumCPpqT3aYcSOiyPt463=O2PwFUnvF_A@mail.gmail.com>
 <CALGmxELf2vRdPs6dX-S5oBDaKpFry+FqnmGOcav3QDvNZi1BzQ@mail.gmail.com>
Message-ID: <1feb49e3-f2e0-11c0-640f-ca4ef75bf8bc@python.org>

On 26May2016 0942, Chris Barker wrote:
>     An alternative approach would be to stick MSVCP140.DLL into a tiny
>     shim wheel and upload that to PyPI, and then wxPython and matplotlib's
>     windows wheels could declare a dependency on this msvcp410 wheel.
>
>     Basically this is the idea of my pynativelib proposal,
>
>
> Yes -- if pynativelib becomes standard practice, then yes, that would be
> a good way to handle this issue -- jsut ike any other lib multiple
> packages might need.
>
>     (I really need to get back to that... if anyone wants to help sprint
>     on this at PyCon let me know :-))
>
>
> I might -- haven't decided what to sprint on yet -- let's talk. (let's
> talk anyway)

This seems like a good solution. I won't be around for sprints, but we 
can chat about it before then.

>
>         MSVCP140.DLL
>
>
>     There are two different versions of this DLL for each architecture
>     (same name).
>
>
> got to love Windows! Does the OS do the right thing if they are both
> installed? ouch!

If they're installed properly into System32/SysWOW64 (using the official 
installer), then yes. If you simply drop them into your Python install 
directory, then no, unless you drop the right one - it has to match the 
python.exe architecture.

>
>         Unfortunately, it won't "just work". More often than not, it
>         will break in weird and very difficult to diagnose ways, as the
>         version of msvcp140.dll changes on a regular basis (there are at
>         least four different version of it "in the wild" since VS 2015
>         released, not counting the preview releases before mid-last year).
>
>
> oh God. Well, that will  pretty much kill it. So essentially every
> single app or lib written in C++ has to ship this dll? And IIUC, folks
> that have the compiler installed have SOME version on the system paths,
> and who knows what version other arbitrary stuff is linked to? How can
> this possibly be considered "Enterprise" software? Wow!

To make it worse, you can only load it once per process. So the first 
extension to load their private copy wins.

The DLL is designed to be installed using the official installer, 
statically linked, or embedded as part of an entire application. It's 
not meant to be injected into someone else's application (which is 
essentially what Python extension modules are doing).

>     There's also a slippery slope argument that would legitimately
>     justify shipping all sorts of "common" dependencies with the core
>     product. I chose to draw the line at "needed by the core Python
>     product", rather than "created by Microsoft" or "used by many
>     packages". (Hence vcruntime140.dll is included, despite not having
>     forwards-compatibility, and I tried real hard to avoid including
>     that one too.)
>
>
> yeah, a line has to be drawn, and "created by Microsoft" wouldn't make
> any sense either. but "standard runtime expected by the compiler" would
> -- but, as you say above, you have may have to ship a new version with
> your extension anyway -- so it seems adding the feature for distutils to
> do that for C++ extensions would both solve the problem and not ship
> anything extra with the core install.

It would be possible. It might be simpler to update the default compiler 
settings to statically link the C++ runtime while still dynamically 
linking the C runtime (and any project is already capable of doing this 
on their own - I'm 99% sure it works).

>     Finally, unless we demand users are administrators for every
>     install, we will quickly have many Python installs using versions of
>     msvcp140.dll (or any other dependency) with security
>     vulnerabilities, with no way to update them other than a full Python
>     release. Installing the regular runtime (which is patched
>     automatically) or as part of a package (which can be independently
>     updated) is far more feasible. I really don't want to be doing
>     security updates for 15 years just to keep an unused DLL secure.
>
>
> Though we have to do that for msvcp140.dll anyway, yes? god, what a
> pain! (and why MS should ship the damn things themselves!)

Maybe, but it's also the same reason why MS doesn't want to ship it 
themselves either. In fact, all the reasons why we don't want to include 
it in Python equally apply to Windows - until it's a direct dependency 
of the product itself, it has a really high bar to be included (SQLite 
is included in Windows because it is a dependency, not just as a 
convenience to developers).

> Anyway, proposal withdrawn.
>
> Replaced with possible proposal to have distutils include the C++
> runtime dll like it does now for the C one.

The big difference here is that msvcp140.dll contains significant 
functionality while vcruntime140.dll is a very small amount of shared 
state and initialization. Changes/additions/fixes to msvcp140.dll are 
far more likely as a result.

I don't recall whether pip extensions are a thing yet, but if they are 
this would be a great candidate for that (trigger the runtime installer 
if necessary). Otherwise, having a package that only includes this DLL 
would probably be fine too, though it really only defers the problem to 
the next DLL you depend on.

Cheers,
Steve



From breamoreboy at yahoo.co.uk  Thu May 26 17:01:51 2016
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Thu, 26 May 2016 22:01:51 +0100
Subject: [Python-Dev] runtime dlls on Windows
In-Reply-To: <CAP1=2W4V6DvPCedyU7W_v1Yt+iu3BaFE1KDNc5dBqvzdjFFR5w@mail.gmail.com>
References: <CALGmxEKA2JepnHACMiwJdzVhgiRokeDfxnNv+_qBKWdNCM4g6w@mail.gmail.com>
 <CAO41-mPLvUtUQy9ELdDwus5r=ic7A=oZxsPf6PORg2Nu+Wrb6w@mail.gmail.com>
 <CALGmxE+mHLz0-=7yVx6vgKHbCmbbo7Ern+=0baa+ZZ7xHLyY=g@mail.gmail.com>
 <CAPTjJmpqTCSkwwonvDumCPpqT3aYcSOiyPt463=O2PwFUnvF_A@mail.gmail.com>
 <CALGmxELf2vRdPs6dX-S5oBDaKpFry+FqnmGOcav3QDvNZi1BzQ@mail.gmail.com>
 <CAP1=2W4V6DvPCedyU7W_v1Yt+iu3BaFE1KDNc5dBqvzdjFFR5w@mail.gmail.com>
Message-ID: <ni7o83$q3b$1@ger.gmane.org>

On 26/05/2016 19:02, Brett Cannon wrote:
>
>
> On Thu, 26 May 2016 at 09:44 Chris Barker <chris.barker at noaa.gov
> <mailto:chris.barker at noaa.gov>> wrote:
>
>     [SNIP]
>     Thanks Steve. Will  you be at PyCon? if Nathaniel and I look at this
>     during the sprints, maybe you could coach us a bit.
>
> Steve will be at PyCon but I don't think he will be around for the
> sprints as he has to catch a flight out for PyCon Taiwan.
>

Steve writes such an excellent blog that when and if he has time I'm 
certain that he'll put something together.  With people such as him 
following on from Martin Loewis, Tim Golden, Mark "I've forgotten more 
about Python on Windows than you'll ever know" Hammond and Andy "ditto" 
Robinson I can't say that I'll lose too much sleep over it.  As I suffer 
insomnia on top of chronic fatigue syndrome, that is quite a statement 
for me to make.

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


From sebastian at realpath.org  Thu May 26 17:53:12 2016
From: sebastian at realpath.org (Sebastian Krause)
Date: Thu, 26 May 2016 23:53:12 +0200
Subject: [Python-Dev] runtime dlls on Windows
In-Reply-To: <CAPTjJmr4cRvJS5xZMLVffAL5kVkkmfcRMkBGwTeEL5gr-q+oaA@mail.gmail.com>
 (Chris Angelico's message of "Fri, 27 May 2016 02:53:27 +1000")
References: <CALGmxEKA2JepnHACMiwJdzVhgiRokeDfxnNv+_qBKWdNCM4g6w@mail.gmail.com>
 <CAO41-mPLvUtUQy9ELdDwus5r=ic7A=oZxsPf6PORg2Nu+Wrb6w@mail.gmail.com>
 <CALGmxE+mHLz0-=7yVx6vgKHbCmbbo7Ern+=0baa+ZZ7xHLyY=g@mail.gmail.com>
 <CAPTjJmpqTCSkwwonvDumCPpqT3aYcSOiyPt463=O2PwFUnvF_A@mail.gmail.com>
 <CALGmxELf2vRdPs6dX-S5oBDaKpFry+FqnmGOcav3QDvNZi1BzQ@mail.gmail.com>
 <CAPTjJmr4cRvJS5xZMLVffAL5kVkkmfcRMkBGwTeEL5gr-q+oaA@mail.gmail.com>
Message-ID: <m2bn3szdtj.fsf@news.realpath.org>

Chris Angelico <rosuav at gmail.com> wrote:
>> BUT -- Steve Dower seems to have identified that the wonders of dll hell
>> never cease, and this isn't possible anyway. Oh well.
>
> I'm not entirely grasping what's happening here. There are multiple
> versions of msvcp140.dll floating around out there; what happens if
> one app requires one of them and another requires another? Can you
> simply say "take the newer one"? Is there a way to instantly recognize
> that there's a newer one available, and ignore the older one?
> Versioning is a *pain*.

Windows executables can and should ask for specific versions of the
DLLs in the manifest (an XML file embedded in the EXE). Details
here: https://en.wikipedia.org/wiki/Side-by-side_assembly

Sebastian

From steve.dower at python.org  Thu May 26 18:18:35 2016
From: steve.dower at python.org (Steve Dower)
Date: Thu, 26 May 2016 15:18:35 -0700
Subject: [Python-Dev] runtime dlls on Windows
In-Reply-To: <m2bn3szdtj.fsf@news.realpath.org>
References: <CALGmxEKA2JepnHACMiwJdzVhgiRokeDfxnNv+_qBKWdNCM4g6w@mail.gmail.com>
 <CAO41-mPLvUtUQy9ELdDwus5r=ic7A=oZxsPf6PORg2Nu+Wrb6w@mail.gmail.com>
 <CALGmxE+mHLz0-=7yVx6vgKHbCmbbo7Ern+=0baa+ZZ7xHLyY=g@mail.gmail.com>
 <CAPTjJmpqTCSkwwonvDumCPpqT3aYcSOiyPt463=O2PwFUnvF_A@mail.gmail.com>
 <CALGmxELf2vRdPs6dX-S5oBDaKpFry+FqnmGOcav3QDvNZi1BzQ@mail.gmail.com>
 <CAPTjJmr4cRvJS5xZMLVffAL5kVkkmfcRMkBGwTeEL5gr-q+oaA@mail.gmail.com>
 <m2bn3szdtj.fsf@news.realpath.org>
Message-ID: <a9f5bf21-8469-c5b3-348f-f3758a8dc6c4@python.org>

On 26May2016 1453, Sebastian Krause wrote:
> Chris Angelico <rosuav at gmail.com> wrote:
>>> BUT -- Steve Dower seems to have identified that the wonders of dll hell
>>> never cease, and this isn't possible anyway. Oh well.
>>
>> I'm not entirely grasping what's happening here. There are multiple
>> versions of msvcp140.dll floating around out there; what happens if
>> one app requires one of them and another requires another? Can you
>> simply say "take the newer one"? Is there a way to instantly recognize
>> that there's a newer one available, and ignore the older one?
>> Versioning is a *pain*.
>
> Windows executables can and should ask for specific versions of the
> DLLs in the manifest (an XML file embedded in the EXE). Details
> here: https://en.wikipedia.org/wiki/Side-by-side_assembly

This has been deprecated. It sounded like a great idea at the time (~8 
years ago) but caused more problems than it solved. Python 2.7 and 3.2 
(IIRC) were the last versions to use this feature.

The way to get the newest version is to run the installer, which will 
upgrade if it has a newer version or leave the old one alone.

Cheers,
Steve

From v+python at g.nevcal.com  Thu May 26 19:01:37 2016
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Thu, 26 May 2016 16:01:37 -0700
Subject: [Python-Dev] runtime dlls on Windows
In-Reply-To: <a9f5bf21-8469-c5b3-348f-f3758a8dc6c4@python.org>
References: <CALGmxEKA2JepnHACMiwJdzVhgiRokeDfxnNv+_qBKWdNCM4g6w@mail.gmail.com>
 <CAO41-mPLvUtUQy9ELdDwus5r=ic7A=oZxsPf6PORg2Nu+Wrb6w@mail.gmail.com>
 <CALGmxE+mHLz0-=7yVx6vgKHbCmbbo7Ern+=0baa+ZZ7xHLyY=g@mail.gmail.com>
 <CAPTjJmpqTCSkwwonvDumCPpqT3aYcSOiyPt463=O2PwFUnvF_A@mail.gmail.com>
 <CALGmxELf2vRdPs6dX-S5oBDaKpFry+FqnmGOcav3QDvNZi1BzQ@mail.gmail.com>
 <CAPTjJmr4cRvJS5xZMLVffAL5kVkkmfcRMkBGwTeEL5gr-q+oaA@mail.gmail.com>
 <m2bn3szdtj.fsf@news.realpath.org>
 <a9f5bf21-8469-c5b3-348f-f3758a8dc6c4@python.org>
Message-ID: <b8aaef15-76de-3c09-18a1-6b50d9b1b382@g.nevcal.com>

On 5/26/2016 3:18 PM, Steve Dower wrote:
> This has been deprecated. It sounded like a great idea at the time (~8 
> years ago) but caused more problems than it solved.

Somehow I missed the announcement of the deprecation.

When I first heard of Windows manifests, though, my first reaction was 
that it sounded like a solution that was more complex than the problem 
it was trying to solve.

Glad to hear they've been deprecated.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160526/a18b4508/attachment.html>

From steve.dower at python.org  Thu May 26 19:42:07 2016
From: steve.dower at python.org (Steve Dower)
Date: Thu, 26 May 2016 16:42:07 -0700
Subject: [Python-Dev] runtime dlls on Windows
In-Reply-To: <b8aaef15-76de-3c09-18a1-6b50d9b1b382@g.nevcal.com>
References: <CALGmxEKA2JepnHACMiwJdzVhgiRokeDfxnNv+_qBKWdNCM4g6w@mail.gmail.com>
 <CAO41-mPLvUtUQy9ELdDwus5r=ic7A=oZxsPf6PORg2Nu+Wrb6w@mail.gmail.com>
 <CALGmxE+mHLz0-=7yVx6vgKHbCmbbo7Ern+=0baa+ZZ7xHLyY=g@mail.gmail.com>
 <CAPTjJmpqTCSkwwonvDumCPpqT3aYcSOiyPt463=O2PwFUnvF_A@mail.gmail.com>
 <CALGmxELf2vRdPs6dX-S5oBDaKpFry+FqnmGOcav3QDvNZi1BzQ@mail.gmail.com>
 <CAPTjJmr4cRvJS5xZMLVffAL5kVkkmfcRMkBGwTeEL5gr-q+oaA@mail.gmail.com>
 <m2bn3szdtj.fsf@news.realpath.org>
 <a9f5bf21-8469-c5b3-348f-f3758a8dc6c4@python.org>
 <b8aaef15-76de-3c09-18a1-6b50d9b1b382@g.nevcal.com>
Message-ID: <6e86988e-ca93-22a7-1bdf-3cde408e8c2c@python.org>

On 26May2016 1601, Glenn Linderman wrote:
> On 5/26/2016 3:18 PM, Steve Dower wrote:
>> This has been deprecated. It sounded like a great idea at the time (~8
>> years ago) but caused more problems than it solved.
>
> Somehow I missed the announcement of the deprecation.

The feature itself probably hasn't been officially deprecated, but using 
it for the C and C++ runtime libraries has not been required for a long 
time now. (I need to be more careful that I don't accidentally announce 
things that may not be true :) )

> When I first heard of Windows manifests, though, my first reaction was
> that it sounded like a solution that was more complex than the problem
> it was trying to solve.

Indeed.

Cheers,
Steve

From rosuav at gmail.com  Thu May 26 19:58:56 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 27 May 2016 09:58:56 +1000
Subject: [Python-Dev] runtime dlls on Windows
In-Reply-To: <1feb49e3-f2e0-11c0-640f-ca4ef75bf8bc@python.org>
References: <CALGmxEKA2JepnHACMiwJdzVhgiRokeDfxnNv+_qBKWdNCM4g6w@mail.gmail.com>
 <CAO41-mPLvUtUQy9ELdDwus5r=ic7A=oZxsPf6PORg2Nu+Wrb6w@mail.gmail.com>
 <CALGmxE+mHLz0-=7yVx6vgKHbCmbbo7Ern+=0baa+ZZ7xHLyY=g@mail.gmail.com>
 <CAPTjJmpqTCSkwwonvDumCPpqT3aYcSOiyPt463=O2PwFUnvF_A@mail.gmail.com>
 <CALGmxELf2vRdPs6dX-S5oBDaKpFry+FqnmGOcav3QDvNZi1BzQ@mail.gmail.com>
 <1feb49e3-f2e0-11c0-640f-ca4ef75bf8bc@python.org>
Message-ID: <CAPTjJmq70_b2fktbrUnAhwjZt_hH04vT8KEzwVhxp+sRweOLBg@mail.gmail.com>

On Fri, May 27, 2016 at 5:13 AM, Steve Dower <steve.dower at python.org> wrote:
> If they're installed properly into System32/SysWOW64 (using the official
> installer), then yes. If you simply drop them into your Python install
> directory, then no, unless you drop the right one - it has to match the
> python.exe architecture.
>

What if the Python installation isn't being run as admin? Or is that
deprecated too?

ChrisA

From dholth at gmail.com  Thu May 26 22:41:38 2016
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 27 May 2016 02:41:38 +0000
Subject: [Python-Dev] Adding a threadlocal to the Python interpreter
In-Reply-To: <e402047b-9b1e-474f-3e85-c8cd5cca4622@python.org>
References: <CAG8k2+4SiXjdfC5LE+fwvUqR+HzKJbj8XtkHny1h8cHGdZRM2Q@mail.gmail.com>
 <CADiSq7cUm4THGu4tfoRneiZVB6ozWtkA2nsGPUvWAEhEMB2H+g@mail.gmail.com>
 <e402047b-9b1e-474f-3e85-c8cd5cca4622@python.org>
Message-ID: <CAG8k2+77wcFzUTf+V4aA6zxmo7M3rXhvXGtJj-sSV_UE2o5Dsw@mail.gmail.com>

Thank you. I started my patch http://bugs.python.org/issue27134 to allow
Python code to set a flag causing the evil str(b'bytes') to raise an
exception. I wasn't sure exactly which module to put it in, so it's in
_string. Please let me know the best place to put the feature and what I
should polish to get it in.

Thanks,

Daniel Holth

On Thu, May 19, 2016 at 6:34 AM Christian Heimes <christian at python.org>
wrote:

> On 2016-05-19 04:30, Nick Coghlan wrote:
> > On 18 May 2016 at 23:20, Daniel Holth <dholth at gmail.com> wrote:
> >> I would like to take another stab at adding a threadlocal "str(bytes)
> raises
> >> an exception" to the Python interpreter, but I had a very hard time
> >> understanding both how to add a threadlocal value to either the
> interpreter
> >> state or the threadlocal dict that is part of that state, and then how
> to
> >> access the same value from both Python and CPython code. The structs
> were
> >> there but it was just hard to understand. Can someone explain it to me?
> >
> > Christian covered the C aspects of the API, while the general purpose
> > Python aspects live in the threading module.
> >
> > However, the Python level thread-local API doesn't provide direct
> > access to the thread state dict. Instead, it provides access to
> > subdicts stored under per-object keys in that dict, keyed as
> > "thread.local.<id>":
>
> In case you wonder about subdicts, they are required to provide multiple
> thread local objects. Each thread local instance has its own key in each
> thread state dict. The approach enables thread local to have independent
> storage objects.
>
> Christian
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160527/de3f854f/attachment.html>

From steve.dower at python.org  Thu May 26 23:45:43 2016
From: steve.dower at python.org (Steve Dower)
Date: Thu, 26 May 2016 20:45:43 -0700
Subject: [Python-Dev] runtime dlls on Windows
In-Reply-To: <CAPTjJmq70_b2fktbrUnAhwjZt_hH04vT8KEzwVhxp+sRweOLBg@mail.gmail.com>
References: <CALGmxEKA2JepnHACMiwJdzVhgiRokeDfxnNv+_qBKWdNCM4g6w@mail.gmail.com>
 <CAO41-mPLvUtUQy9ELdDwus5r=ic7A=oZxsPf6PORg2Nu+Wrb6w@mail.gmail.com>
 <CALGmxE+mHLz0-=7yVx6vgKHbCmbbo7Ern+=0baa+ZZ7xHLyY=g@mail.gmail.com>
 <CAPTjJmpqTCSkwwonvDumCPpqT3aYcSOiyPt463=O2PwFUnvF_A@mail.gmail.com>
 <CALGmxELf2vRdPs6dX-S5oBDaKpFry+FqnmGOcav3QDvNZi1BzQ@mail.gmail.com>
 <1feb49e3-f2e0-11c0-640f-ca4ef75bf8bc@python.org>
 <CAPTjJmq70_b2fktbrUnAhwjZt_hH04vT8KEzwVhxp+sRweOLBg@mail.gmail.com>
Message-ID: <E1b68ik-0002i7-O8@se2-syd.hostedmail.net.au>

Another reason why we don't want to install this DLL with Python :) Per-user install is properly supported in 3.5 for I believe the first time ever.

But yes, installing them properly requires admin privileges. An unfortunately messy situation, one that I'm happy to not have to resolve for Python's incredibly diverse user base.

Top-posted from my Windows Phone

-----Original Message-----
From: "Chris Angelico" <rosuav at gmail.com>
Sent: ?5/?26/?2016 17:04
Cc: "python-dev" <python-dev at python.org>
Subject: Re: [Python-Dev] runtime dlls on Windows

On Fri, May 27, 2016 at 5:13 AM, Steve Dower <steve.dower at python.org> wrote:
> If they're installed properly into System32/SysWOW64 (using the official
> installer), then yes. If you simply drop them into your Python install
> directory, then no, unless you drop the right one - it has to match the
> python.exe architecture.
>

What if the Python installation isn't being run as admin? Or is that
deprecated too?

ChrisA
_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160526/55bc2cf9/attachment.html>

From raymond.hettinger at gmail.com  Fri May 27 00:54:33 2016
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Thu, 26 May 2016 21:54:33 -0700
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
Message-ID: <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>


> On May 25, 2016, at 3:29 AM, Christian Heimes <christian at python.org> wrote:
> 
> I have three hashing-related patches for Python 3.6 that are waiting for
> review. Altogether the three patches add ten new hash algorithms to the
> hashlib module: SHA3 (224, 256, 384, 512), SHAKE (SHA3 XOF 128, 256),
> BLAKE2 (blake2b, blake2s) and truncated SHA512 (224, 256).

Do we really need ten?  I don't think the standard library is the place to offer all variants of hashing.  And we should avoid getting in a cycle of "this was just released by NIST" and "nobody uses that one anymore".  Is any one of them an emergent best practice (i.e. starting to be commonly used in network protocols because it is better, faster, stronger, etc)?

Your last message on https://bugs.python.org/issue16113 suggests that these aren't essential and that there is room for debate about whether some of them are standard-library worthy (i.e. we will have them around forever).


Raymond

From donald at stufft.io  Fri May 27 06:01:40 2016
From: donald at stufft.io (Donald Stufft)
Date: Fri, 27 May 2016 06:01:40 -0400
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
Message-ID: <ADD38FA0-5F5E-4EBE-992F-0A882B00F483@stufft.io>


> On May 27, 2016, at 12:54 AM, Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
> 
> 
>> On May 25, 2016, at 3:29 AM, Christian Heimes <christian at python.org> wrote:
>> 
>> I have three hashing-related patches for Python 3.6 that are waiting for
>> review. Altogether the three patches add ten new hash algorithms to the
>> hashlib module: SHA3 (224, 256, 384, 512), SHAKE (SHA3 XOF 128, 256),
>> BLAKE2 (blake2b, blake2s) and truncated SHA512 (224, 256).
> 
> Do we really need ten?  I don't think the standard library is the place to offer all variants of hashing.  And we should avoid getting in a cycle of "this was just released by NIST" and "nobody uses that one anymore".  Is any one of them an emergent best practice (i.e. starting to be commonly used in network protocols because it is better, faster, stronger, etc)?
> 
> Your last message on https://bugs.python.org/issue16113 suggests that these aren't essential and that there is room for debate about whether some of them are standard-library worthy (i.e. we will have them around forever).
> 


I think that adding sha3 here is a net positive. While there isn?t a huge amount of things using it today, that?s largely because it?s fairly new? It?s a NIST standard so it won?t be long until things are using it. It would be surprising to me to be able to use sha1 and sha2 from the standard library, but not sha3.

SHAKE is really just SHA3 with some additional tweaks to the parameters. I think if you?re adding SHA3 it?s pretty easy to also add these, though I don?t think that it?s as important as adding SHA3 itself.

BLAKE2 is an interesting one, because while SHA3 is a NIST standard (so it?s going to gain adoption because of that), BLAKE2 is at least as strong as SHA3 but is better in many ways, particularly in speed? it?s actually faster than MD5 while being as secure as SHA3. This one I think is a good one to have in the standard library as well because it is all around a really great hash and a lot of things are starting to be built on top of it. In particularly I?d like to use this in PyPI and pip- but I can?t unless it?s in the standard library.

?
Donald Stufft




From victor.stinner at gmail.com  Fri May 27 06:44:00 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 27 May 2016 12:44:00 +0200
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <ADD38FA0-5F5E-4EBE-992F-0A882B00F483@stufft.io>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <ADD38FA0-5F5E-4EBE-992F-0A882B00F483@stufft.io>
Message-ID: <CAMpsgwb3Bc9u702ExNPer8KQQ+wx8JSQ0LcABiB1UKOUA-Hmhw@mail.gmail.com>

Le 27 mai 2016 12:05 PM, "Donald Stufft" <donald at stufft.io> a ?crit :
> BLAKE2 is an interesting one, because while SHA3 is a NIST standard (so
it?s going to gain adoption because of that), BLAKE2 is at least as strong
as SHA3 but is better in many ways, particularly in speed? it?s actually
faster than MD5 while being as secure as SHA3.

BLAKE2 was part of the SHA3 competition and it was in finalists. The SHA3
competition is interesting because each algorithm is deeply tested and
analyzed by many teams all around the world. Obvious vulnerabilities are
quickly found.

The advantage of putting SHA3 and BLAKE2 in the stdlib is that they have a
different design. I don't expect that two designs have the same
vulnerabilities, but I'm not ax expert :-)

SHA3 (Keccak) is based on a new sponge construction:
https://en.m.wikipedia.org/wiki/SHA-3

BLAKE is based on ChaCha:
https://en.m.wikipedia.org/wiki/BLAKE_(hash_function)
https://en.m.wikipedia.org/wiki/Salsa20#ChaCha_variant

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160527/f7488d82/attachment.html>

From mal at egenix.com  Fri May 27 06:54:49 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 27 May 2016 12:54:49 +0200
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
Message-ID: <57482779.6030100@egenix.com>

On 27.05.2016 06:54, Raymond Hettinger wrote:
> 
>> On May 25, 2016, at 3:29 AM, Christian Heimes <christian at python.org> wrote:
>>
>> I have three hashing-related patches for Python 3.6 that are waiting for
>> review. Altogether the three patches add ten new hash algorithms to the
>> hashlib module: SHA3 (224, 256, 384, 512), SHAKE (SHA3 XOF 128, 256),
>> BLAKE2 (blake2b, blake2s) and truncated SHA512 (224, 256).
> 
> Do we really need ten?  I don't think the standard library is the place to offer all variants of hashing.  And we should avoid getting in a cycle of "this was just released by NIST" and "nobody uses that one anymore".  Is any one of them an emergent best practice (i.e. starting to be commonly used in network protocols because it is better, faster, stronger, etc)?
> 
> Your last message on https://bugs.python.org/issue16113 suggests that these aren't essential and that there is room for debate about whether some of them are standard-library worthy (i.e. we will have them around forever).

I can understand your eagerness to get this landed, since it's
been 4 years since work started, but I think we should wait with
the addition until OpenSSL has them:

https://github.com/openssl/openssl/issues/439

The current patch is 1.2MB for SHA-3 - that's pretty heavy for just
a few hash functions, which aren't in any wide spread use yet and
probably won't be for quite a few years ahead.

IMO, relying on OpenSSL is a better strategy than providing
(and maintaining) our own compatibility versions. Until OpenSSL
has them, people can use Bj?rn's package:

https://github.com/bjornedstrom/python-sha3

Perhaps you could join forces with Bj?rn to create a standard
SHA-3 standalone package on PyPI based on your two variants
which we could recommend to people in the docs ?!

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, May 27 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From donald at stufft.io  Fri May 27 07:03:46 2016
From: donald at stufft.io (Donald Stufft)
Date: Fri, 27 May 2016 07:03:46 -0400
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <57482779.6030100@egenix.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com>
Message-ID: <2ABB301C-544B-4BE3-9A7B-0B1EEA392144@stufft.io>


> On May 27, 2016, at 6:54 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> 
> IMO, relying on OpenSSL is a better strategy than providing
> (and maintaining) our own compatibility versions. Until OpenSSL
> has them, people can use Bj?rn's package:

Even now, hashlib doesn?t rely on OpenSSL if I recall, I mean it will
use it if OpenSSL is available but otherwise it has internal implementations
too.

?
Donald Stufft




From mal at egenix.com  Fri May 27 07:13:05 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 27 May 2016 13:13:05 +0200
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <2ABB301C-544B-4BE3-9A7B-0B1EEA392144@stufft.io>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com>
 <2ABB301C-544B-4BE3-9A7B-0B1EEA392144@stufft.io>
Message-ID: <57482BC1.6000200@egenix.com>

On 27.05.2016 13:03, Donald Stufft wrote:
> 
>> On May 27, 2016, at 6:54 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>>
>> IMO, relying on OpenSSL is a better strategy than providing
>> (and maintaining) our own compatibility versions. Until OpenSSL
>> has them, people can use Bj?rn's package:
> 
> Even now, hashlib doesn?t rely on OpenSSL if I recall, I mean it will
> use it if OpenSSL is available but otherwise it has internal implementations
> too.

I know, but still don't think that's a good idea. It makes
sense in case you don't want to carry around OpenSSL all the
time, but how often does that happen nowadays ?

BTW: If I recall correctly, those hash implementations predate
the deeper support for OpenSSL we now have in Python.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, May 27 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From dholth at gmail.com  Fri May 27 10:31:45 2016
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 27 May 2016 14:31:45 +0000
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <57482BC1.6000200@egenix.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com> <57482779.6030100@egenix.com>
 <2ABB301C-544B-4BE3-9A7B-0B1EEA392144@stufft.io> <57482BC1.6000200@egenix.com>
Message-ID: <CAG8k2+755mtq-xvM7HNep-_sjU+1BT=_H+uZZGgAhpqtxYAiKg@mail.gmail.com>

OpenSSL sucks. Python would only have to bundle a reference implementation
of the new hash algorithm(s), and unlike TLS suites they tend to just work.

BLAKE2 is important, since it removes the last objection to replacing MD5 -
speed - that has made it hard for cryptography fans to convince MD5 users
to upgrade.

On Fri, May 27, 2016 at 7:13 AM M.-A. Lemburg <mal at egenix.com> wrote:

> On 27.05.2016 13:03, Donald Stufft wrote:
> >
> >> On May 27, 2016, at 6:54 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> >>
> >> IMO, relying on OpenSSL is a better strategy than providing
> >> (and maintaining) our own compatibility versions. Until OpenSSL
> >> has them, people can use Bj?rn's package:
> >
> > Even now, hashlib doesn?t rely on OpenSSL if I recall, I mean it will
> > use it if OpenSSL is available but otherwise it has internal
> implementations
> > too.
>
> I know, but still don't think that's a good idea. It makes
> sense in case you don't want to carry around OpenSSL all the
> time, but how often does that happen nowadays ?
>
> BTW: If I recall correctly, those hash implementations predate
> the deeper support for OpenSSL we now have in Python.
>
> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Experts (#1, May 27 2016)
> >>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
> >>> Python Database Interfaces ...           http://products.egenix.com/
> >>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
> ________________________________________________________________________
>
> ::: We implement business ideas - efficiently in both time and costs :::
>
>    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>            Registered at Amtsgericht Duesseldorf: HRB 46611
>                http://www.egenix.com/company/contact/
>                       http://www.malemburg.com/
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160527/b5f87cab/attachment.html>

From vitaly.murashev at gmail.com  Fri May 27 05:49:08 2016
From: vitaly.murashev at gmail.com (Vitaly Murashev)
Date: Fri, 27 May 2016 12:49:08 +0300
Subject: [Python-Dev] embedding - PyImport_AppendInittab() after
 Py_Initialize()
Message-ID: <CALPW6hy=RT+k+LSm8+HQ_3iv_amnttriwEo94S9eg4kPNAWX3g@mail.gmail.com>

Could anyone please clarify whether it is correct in python3.5 to call
PyImport_AppendInittab() after Py_Initialize()

I found this case among the tests for boost.python
https://github.com/boostorg/python/blob/develop/test/exec.cpp

And this test doesn't work for me with python3.5.An error looks like:

exec.cpp(137): Python Error detected in function 'void __cdecl
check_pyerr(bool)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: 'embedded_hello' is not a built-in module
1 error detected.

After debugging I found out that root cause is in
importlib/_bootstrap.py which do the following:

        if fullname not in sys.builtin_module_names:
            raise ImportError('{!r} is not a built-in
module'.format(fullname), name=fullname)

but sys.builtin_module_names is the constant which initialized once in
Py_Initialize()

---
To summarize:
Is it a bug in python3.5 or original test inside boost.python should be
fixed ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160527/552c4b8b/attachment.html>

From mafagafogigante at gmail.com  Fri May 27 11:37:48 2016
From: mafagafogigante at gmail.com (Bernardo Sulzbach)
Date: Fri, 27 May 2016 12:37:48 -0300
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <CAG8k2+755mtq-xvM7HNep-_sjU+1BT=_H+uZZGgAhpqtxYAiKg@mail.gmail.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com>
 <2ABB301C-544B-4BE3-9A7B-0B1EEA392144@stufft.io>
 <57482BC1.6000200@egenix.com>
 <CAG8k2+755mtq-xvM7HNep-_sjU+1BT=_H+uZZGgAhpqtxYAiKg@mail.gmail.com>
Message-ID: <d72f46f9-e745-97cb-4969-0f8cb04f2fab@gmail.com>

On 05/27/2016 11:31 AM, Daniel Holth wrote:
 >
> BLAKE2 is important, since it removes the last objection to replacing MD5 -
> speed - that has made it hard for cryptography fans to convince MD5 users
> to upgrade.
>

I have had to stick to MD5 for performance reasons (2 seconds in MD5 or 
9.6 seconds in SHA256, IIRC) in scenarios that did not require an SHA*. 
Having BLAKE2 around wouldn't be a necessity, but if it shipped with 
newer versions of Python eventually there would be a commit switching 
the underlying hash function.

From chris.barker at noaa.gov  Fri May 27 11:44:45 2016
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Fri, 27 May 2016 08:44:45 -0700
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <57482779.6030100@egenix.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com> <57482779.6030100@egenix.com>
Message-ID: <7067504675934721722@unknownmsgid>

>> , which aren't in any wide spread use yet and
> probably won't be for quite a few years ahead.

Anything added to the stdlib now will be in py3.6+, yes?

Which won't be in widespread use for quite a few years yet, either.

So if ( and that's a big if) it's possible to anticipate what will be
in widespread use in a couple years, getting it in now would be a good
thing.

-CHB



>
> IMO, relying on OpenSSL is a better strategy than providing
> (and maintaining) our own compatibility versions. Until OpenSSL
> has them, people can use Bj?rn's package:
>
> https://github.com/bjornedstrom/python-sha3
>
> Perhaps you could join forces with Bj?rn to create a standard
> SHA-3 standalone package on PyPI based on your two variants
> which we could recommend to people in the docs ?!
>
> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Experts (#1, May 27 2016)
>>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>>> Python Database Interfaces ...           http://products.egenix.com/
>>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
> ________________________________________________________________________
>
> ::: We implement business ideas - efficiently in both time and costs :::
>
>   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>           Registered at Amtsgericht Duesseldorf: HRB 46611
>               http://www.egenix.com/company/contact/
>                      http://www.malemburg.com/
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov

From status at bugs.python.org  Fri May 27 12:08:42 2016
From: status at bugs.python.org (Python tracker)
Date: Fri, 27 May 2016 18:08:42 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20160527160842.70BFF56A1E@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2016-05-20 - 2016-05-27)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5529 (+25)
  closed 33364 (+42)
  total  38893 (+67)

Open issues with patches: 2400 


Issues opened (46)
==================

#16858: tarfile silently hides errors
http://bugs.python.org/issue16858  reopened by mmarkk

#26915: Test identity first in membership operation of ItemsView, Valu
http://bugs.python.org/issue26915  reopened by xiang.zhang

#27067: Improve curses tests
http://bugs.python.org/issue27067  reopened by martin.panter

#27072: random.getrandbits is limited to 2**31-1 bits on 64-bit Window
http://bugs.python.org/issue27072  opened by Steven.Barker

#27073: redundant checks in long_add and long_sub
http://bugs.python.org/issue27073  opened by Oren Milman

#27074: Confusing text about __all__ in __init__.py in tutorial
http://bugs.python.org/issue27074  opened by ztane

#27078: Make f'' strings faster than .format: BUILD_STRING opcode?
http://bugs.python.org/issue27078  opened by ztane

#27079: Bugs in curses.ascii predicates
http://bugs.python.org/issue27079  opened by serhiy.storchaka

#27080: Implement the formatting part of PEP 515, '_' in numeric liter
http://bugs.python.org/issue27080  opened by eric.smith

#27081: Multiprocessing is not robust against sys.stderr changes invol
http://bugs.python.org/issue27081  opened by ppperry

#27083: PYTHONCASEOK is ignored on Windows
http://bugs.python.org/issue27083  opened by eryksun

#27084: Add dir_fd and follow_symlinks kwargs to os.listdir and os.sca
http://bugs.python.org/issue27084  opened by abacabadabacaba

#27085: Make it possible to select return type for os.listdir
http://bugs.python.org/issue27085  opened by abacabadabacaba

#27086: Add closefd argument to os.listdir
http://bugs.python.org/issue27086  opened by abacabadabacaba

#27088: doc: select: epoll.poll: incorrect timeout units, missing maxe
http://bugs.python.org/issue27088  opened by fordsfords

#27095: Simplify MAKE_FUNCTION
http://bugs.python.org/issue27095  opened by Demur Rumed

#27099: IDLE: turn builting extensions into regular modules
http://bugs.python.org/issue27099  opened by terry.reedy

#27100: Attempting to use class with both __enter__ & __exit__ undefin
http://bugs.python.org/issue27100  opened by ellingtonjp

#27101: Compilation of python (modules) for foreign target platform pr
http://bugs.python.org/issue27101  opened by complement

#27103: regrtest: capture stdout (-W) option is incompatible with refl
http://bugs.python.org/issue27103  opened by haypo

#27105: cgi.__all__ is incomplete
http://bugs.python.org/issue27105  opened by Unit03

#27106: configparser.__all__ is incomplete
http://bugs.python.org/issue27106  opened by Unit03

#27107: mailbox.__all__ list is incomplete
http://bugs.python.org/issue27107  opened by Unit03

#27108: mimetypes.__all__ list is incomplete
http://bugs.python.org/issue27108  opened by Unit03

#27109: plistlib.__all__ list is incomplete
http://bugs.python.org/issue27109  opened by Unit03

#27110: smtpd.__all__ list is incomplete
http://bugs.python.org/issue27110  opened by Unit03

#27111: redundant variables in long_add and long_sub
http://bugs.python.org/issue27111  opened by Oren Milman

#27112: tokenize.__all__ list is incomplete
http://bugs.python.org/issue27112  opened by Unit03

#27113: sqlite3 connect parameter "check_same_thread" not documented
http://bugs.python.org/issue27113  opened by Dave Sawyer

#27115: IDLE/tkinter: in simpledialog, <Return> != [OK] click
http://bugs.python.org/issue27115  opened by terry.reedy

#27117: turtledemo does not work with IDLE's new dark theme.
http://bugs.python.org/issue27117  opened by terry.reedy

#27119: `compile` doesn't compile into an AST object as specified
http://bugs.python.org/issue27119  opened by leewz

#27121: imghdr does not support jpg files with Lavc bytes
http://bugs.python.org/issue27121  opened by Ren?? L??we Jacobsen

#27122: Hang with contextlib.ExitStack and subprocess.Popen (regressio
http://bugs.python.org/issue27122  opened by Valentin David

#27123: Allow `install_headers` command to follow specific directory s
http://bugs.python.org/issue27123  opened by sylvain.corlay

#27124: binascii.a2b_hex raises binascii.Error and ValueError, not Typ
http://bugs.python.org/issue27124  opened by Lennart Grahl

#27125: Typo in Python 2 multiprocessing documentation
http://bugs.python.org/issue27125  opened by phx

#27126: Apple-supplied libsqlite3 on OS X is not fork safe;  can cause
http://bugs.python.org/issue27126  opened by evan.jones at bluecore.com

#27127: Never have GET_ITER not followed by FOR_ITER
http://bugs.python.org/issue27127  opened by Demur Rumed

#27128: Add _PyObject_FastCall()
http://bugs.python.org/issue27128  opened by haypo

#27129: Wordcode, part 2
http://bugs.python.org/issue27129  opened by serhiy.storchaka

#27130: zlib: OverflowError while trying to compress 2^32 bytes or mor
http://bugs.python.org/issue27130  opened by Klamann

#27131: Unit test random shuffle
http://bugs.python.org/issue27131  opened by jonathan.kross

#27132: New assert method that checks an error message for a list of s
http://bugs.python.org/issue27132  opened by maciej.szulik

#27133: python 3.5.1 will not compile because libffi module uses wrong
http://bugs.python.org/issue27133  opened by blastwave

#27136: sock_connect fails for bluetooth (and probably others)
http://bugs.python.org/issue27136  opened by pyptr2



Most recent 15 issues with no replies (15)
==========================================

#27123: Allow `install_headers` command to follow specific directory s
http://bugs.python.org/issue27123

#27121: imghdr does not support jpg files with Lavc bytes
http://bugs.python.org/issue27121

#27117: turtledemo does not work with IDLE's new dark theme.
http://bugs.python.org/issue27117

#27115: IDLE/tkinter: in simpledialog, <Return> != [OK] click
http://bugs.python.org/issue27115

#27111: redundant variables in long_add and long_sub
http://bugs.python.org/issue27111

#27109: plistlib.__all__ list is incomplete
http://bugs.python.org/issue27109

#27103: regrtest: capture stdout (-W) option is incompatible with refl
http://bugs.python.org/issue27103

#27088: doc: select: epoll.poll: incorrect timeout units, missing maxe
http://bugs.python.org/issue27088

#27080: Implement the formatting part of PEP 515, '_' in numeric liter
http://bugs.python.org/issue27080

#27079: Bugs in curses.ascii predicates
http://bugs.python.org/issue27079

#27074: Confusing text about __all__ in __init__.py in tutorial
http://bugs.python.org/issue27074

#27038: Make os.DirEntry exist
http://bugs.python.org/issue27038

#27030: Remove deprecated re features
http://bugs.python.org/issue27030

#27021: It is not documented that os.writev() suffer from SC_IOV_MAX
http://bugs.python.org/issue27021

#27019: Reduce marshal stack depth for 2.7 on Windows debug build
http://bugs.python.org/issue27019



Most recent 15 issues waiting for review (15)
=============================================

#27136: sock_connect fails for bluetooth (and probably others)
http://bugs.python.org/issue27136

#27131: Unit test random shuffle
http://bugs.python.org/issue27131

#27129: Wordcode, part 2
http://bugs.python.org/issue27129

#27128: Add _PyObject_FastCall()
http://bugs.python.org/issue27128

#27127: Never have GET_ITER not followed by FOR_ITER
http://bugs.python.org/issue27127

#27125: Typo in Python 2 multiprocessing documentation
http://bugs.python.org/issue27125

#27124: binascii.a2b_hex raises binascii.Error and ValueError, not Typ
http://bugs.python.org/issue27124

#27113: sqlite3 connect parameter "check_same_thread" not documented
http://bugs.python.org/issue27113

#27112: tokenize.__all__ list is incomplete
http://bugs.python.org/issue27112

#27111: redundant variables in long_add and long_sub
http://bugs.python.org/issue27111

#27110: smtpd.__all__ list is incomplete
http://bugs.python.org/issue27110

#27109: plistlib.__all__ list is incomplete
http://bugs.python.org/issue27109

#27108: mimetypes.__all__ list is incomplete
http://bugs.python.org/issue27108

#27107: mailbox.__all__ list is incomplete
http://bugs.python.org/issue27107

#27106: configparser.__all__ is incomplete
http://bugs.python.org/issue27106



Top 10 most discussed issues (10)
=================================

#26647: ceval: use Wordcode, 16-bit bytecode
http://bugs.python.org/issue26647  23 msgs

#27067: Improve curses tests
http://bugs.python.org/issue27067  13 msgs

#26632: __all__ decorator
http://bugs.python.org/issue26632   9 msgs

#27051: Create PIP gui
http://bugs.python.org/issue27051   9 msgs

#24225: Idlelib: changing file names
http://bugs.python.org/issue24225   8 msgs

#27122: Hang with contextlib.ExitStack and subprocess.Popen (regressio
http://bugs.python.org/issue27122   8 msgs

#27085: Make it possible to select return type for os.listdir
http://bugs.python.org/issue27085   6 msgs

#18373: let code force str(bytes) to raise an exception
http://bugs.python.org/issue18373   5 msgs

#27032: Remove deprecated html.parser.HTMLParser.unescape()
http://bugs.python.org/issue27032   5 msgs

#27072: random.getrandbits is limited to 2**31-1 bits on 64-bit Window
http://bugs.python.org/issue27072   5 msgs



Issues closed (41)
==================

#5187: distutils upload should prompt for the user/password too
http://bugs.python.org/issue5187  closed by berker.peksag

#5784: raw deflate format and zlib module
http://bugs.python.org/issue5784  closed by martin.panter

#7159: Urllib2 authentication memory.
http://bugs.python.org/issue7159  closed by terry.reedy

#13615: setup.py register fails with -r argument
http://bugs.python.org/issue13615  closed by berker.peksag

#16509: sqlite3 docs do not explain check_same_thread
http://bugs.python.org/issue16509  closed by berker.peksag

#18383: test_warnings modifies warnings.filters when running with "-W 
http://bugs.python.org/issue18383  closed by martin.panter

#23026: Winreg module doesn't support REG_QWORD, small DWORD doc updat
http://bugs.python.org/issue23026  closed by python-dev

#25823: Speed-up oparg decoding on little-endian machines
http://bugs.python.org/issue25823  closed by serhiy.storchaka

#26168: Py_BuildValue may leak 'N' arguments on PyTuple_New failure
http://bugs.python.org/issue26168  closed by serhiy.storchaka

#26306: Can't create abstract tuple
http://bugs.python.org/issue26306  closed by r.david.murray

#26742: imports in test_warnings changes warnings.filters
http://bugs.python.org/issue26742  closed by martin.panter

#26816: Make concurrent.futures.Executor an abc
http://bugs.python.org/issue26816  closed by xiang.zhang

#26877: tarfile use wrong code when read from fileobj
http://bugs.python.org/issue26877  closed by mmarkk

#27054: Python installation problem: No module named 'encodings'
http://bugs.python.org/issue27054  closed by steve.dower

#27060: Documentation of assertItemsEqual in unittest is VERY misleadi
http://bugs.python.org/issue27060  closed by terry.reedy

#27063: Some unittest loader tests are silently skipped by mistake
http://bugs.python.org/issue27063  closed by serhiy.storchaka

#27064: Make py.exe default to Python 3 when used interactively
http://bugs.python.org/issue27064  closed by paul.moore

#27070: Add ability to freeze (seal) mutable objects
http://bugs.python.org/issue27070  closed by eric.smith

#27071: unittest.TestCase.assertCountEqual is a very misleading name
http://bugs.python.org/issue27071  closed by r.david.murray

#27075: Link to StreamReader/Writer from codecs.getreader/writer
http://bugs.python.org/issue27075  closed by berker.peksag

#27076: Doc and comment spelling fixes
http://bugs.python.org/issue27076  closed by martin.panter

#27077: test_port_parameter_types fails with BytesWarning
http://bugs.python.org/issue27077  closed by berker.peksag

#27082: IDLE seriously degrades during and after printing large single
http://bugs.python.org/issue27082  closed by terry.reedy

#27087: unable to use socket send and sendall   due to type error
http://bugs.python.org/issue27087  closed by eric.smith

#27089: I think this is a small bug in urlparse.py
http://bugs.python.org/issue27089  closed by berker.peksag

#27090: Python 3 import error after installation
http://bugs.python.org/issue27090  closed by zach.ware

#27091: Python IDLE doesn't like ????
http://bugs.python.org/issue27091  closed by eryksun

#27092: ord() raises TypeError on string/bytes input
http://bugs.python.org/issue27092  closed by cwr

#27093: Silence warning in cjkcodecs.h
http://bugs.python.org/issue27093  closed by python-dev

#27094: Floating Point Printing Error with 2+.8553 or 3+.8553
http://bugs.python.org/issue27094  closed by r.david.murray

#27096: Ability to get random bytes from random.Random (as with os.ura
http://bugs.python.org/issue27096  closed by rhettinger

#27097: ceval: Wordcode follow up, explicit unsigned short read
http://bugs.python.org/issue27097  closed by serhiy.storchaka

#27098: Pickle of memoryview not raising error
http://bugs.python.org/issue27098  closed by serhiy.storchaka

#27102: Spam
http://bugs.python.org/issue27102  closed by zach.ware

#27104: enable-framework
http://bugs.python.org/issue27104  closed by alexandra.beebe at gmail.com

#27114: SSLContext._load_windows_store_certs fails with PermissionErro
http://bugs.python.org/issue27114  closed by steve.dower

#27116: Documentation for regular expression alphanumeric matching are
http://bugs.python.org/issue27116  closed by zach.ware

#27118: Several Py_XDE/INCREFs in typeobject.c are not necessary
http://bugs.python.org/issue27118  closed by serhiy.storchaka

#27120: xmllib unable to parse in UTF8 format
http://bugs.python.org/issue27120  closed by serhiy.storchaka

#27134: allow str(bytes) raises an exception to be controlled programm
http://bugs.python.org/issue27134  closed by dholth

#27135: nested list produced with multiplication is linked to the same
http://bugs.python.org/issue27135  closed by benjamin.peterson

From mal at egenix.com  Fri May 27 12:35:17 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 27 May 2016 18:35:17 +0200
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <7067504675934721722@unknownmsgid>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com> <7067504675934721722@unknownmsgid>
Message-ID: <57487745.2080900@egenix.com>



On 27.05.2016 17:44, Chris Barker - NOAA Federal wrote:
>>> , which aren't in any wide spread use yet and
>> probably won't be for quite a few years ahead.
> 
> Anything added to the stdlib now will be in py3.6+, yes?
> 
> Which won't be in widespread use for quite a few years yet, either.
> 
> So if ( and that's a big if) it's possible to anticipate what will be
> in widespread use in a couple years, getting it in now would be a good
> thing.

You cut away the important part of what I said:
"The current patch is 1.2MB for SHA-3 - that's pretty heavy for
just a few hash functions, ..."

If people want to use the hashes earlier, this is already possible
via a separate package, so we're not delaying their use.

It is clear that SHA-3 will get more traction in coming years (*),
but I'm pretty sure that OpenSSL will have good implementations by
the time people will actively start using the new hash algorithm
and then hashlib will automatically make that available (hashlib
uses the OpenSSL EVP abstraction, so will be able to use any
new algorithms added to OpenSSL).

However, if we add the reference implementation now, we'd then be
left with 1.2MB unnecessary code in the stdlib.

The question is not so much: is SHA-3 useful or not, it's
whether we want to maintain this forever going forward or
not.

(*) People are just now starting to move from SHA-1 to SHA-2
and SHA-2 was standardized in 2001. Python received SHA-2 support
in 2006. So there's plenty of time to decide :-)

> -CHB
> 
> 
> 
>>
>> IMO, relying on OpenSSL is a better strategy than providing
>> (and maintaining) our own compatibility versions. Until OpenSSL
>> has them, people can use Bj?rn's package:
>>
>> https://github.com/bjornedstrom/python-sha3
>>
>> Perhaps you could join forces with Bj?rn to create a standard
>> SHA-3 standalone package on PyPI based on your two variants
>> which we could recommend to people in the docs ?!
>>
>> --
>> Marc-Andre Lemburg
>> eGenix.com
>>
>> Professional Python Services directly from the Experts (#1, May 27 2016)
>>>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>>>> Python Database Interfaces ...           http://products.egenix.com/
>>>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
>> ________________________________________________________________________
>>
>> ::: We implement business ideas - efficiently in both time and costs :::
>>
>>   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>>    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>>           Registered at Amtsgericht Duesseldorf: HRB 46611
>>               http://www.egenix.com/company/contact/
>>                      http://www.malemburg.com/
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, May 27 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From chris.barker at noaa.gov  Fri May 27 12:41:11 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 27 May 2016 09:41:11 -0700
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <57487745.2080900@egenix.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com> <57482779.6030100@egenix.com>
 <7067504675934721722@unknownmsgid> <57487745.2080900@egenix.com>
Message-ID: <CALGmxEJYwU7pSL--u9RJo4XFuev32wG4mSo6WgLrRZTFr0tyTQ@mail.gmail.com>

On Fri, May 27, 2016 at 9:35 AM, M.-A. Lemburg <mal at egenix.com> wrote:

> > So if ( and that's a big if) it's possible to anticipate what will be
> > in widespread use in a couple years, getting it in now would be a good
> > thing.
>
> You cut away the important part of what I said:
> "The current patch is 1.2MB for SHA-3 - that's pretty heavy for
> just a few hash functions, ..."
>
> If people want to use the hashes earlier, this is already possible
> via a separate package, so we're not delaying their use.
>

That's true for ANY addition to the stdlib -- it could always be made
available in a third party lib. (unless you want to use it in another part
of the stdlib...)


> It is clear that SHA-3 will get more traction in coming years (*),
> but I'm pretty sure that OpenSSL will have good implementations by
> the time people will actively start using the new hash algorithm
> and then hashlib will automatically make that available (hashlib
> uses the OpenSSL EVP abstraction, so will be able to use any
> new algorithms added to OpenSSL).
>
> However, if we add the reference implementation now, we'd then be
> left with 1.2MB unnecessary code in the stdlib.
>

I'm probably showing my ignorance here, but couldn't we swap in the OpenSSL
implementation when that becomes available?

-CHB


(*) People are just now starting to move from SHA-1 to SHA-2
> and SHA-2 was standardized in 2001. Python received SHA-2 support
> in 2006. So there's plenty of time to decide :-)


can't deny the history, nor the inertia -- but that doesn't make it a good
thing...


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160527/fca2d73c/attachment.html>

From mal at egenix.com  Fri May 27 12:56:21 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 27 May 2016 18:56:21 +0200
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <CALGmxEJYwU7pSL--u9RJo4XFuev32wG4mSo6WgLrRZTFr0tyTQ@mail.gmail.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com> <7067504675934721722@unknownmsgid>
 <57487745.2080900@egenix.com>
 <CALGmxEJYwU7pSL--u9RJo4XFuev32wG4mSo6WgLrRZTFr0tyTQ@mail.gmail.com>
Message-ID: <57487C35.9050102@egenix.com>

On 27.05.2016 18:41, Chris Barker wrote:
> On Fri, May 27, 2016 at 9:35 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> 
>>> So if ( and that's a big if) it's possible to anticipate what will be
>>> in widespread use in a couple years, getting it in now would be a good
>>> thing.
>>
>> You cut away the important part of what I said:
>> "The current patch is 1.2MB for SHA-3 - that's pretty heavy for
>> just a few hash functions, ..."
>>
>> If people want to use the hashes earlier, this is already possible
>> via a separate package, so we're not delaying their use.
>>
> 
> That's true for ANY addition to the stdlib -- it could always be made
> available in a third party lib. (unless you want to use it in another part
> of the stdlib...)

Well, any addition for which someone already wrote a package,
but yes...

>> It is clear that SHA-3 will get more traction in coming years (*),
>> but I'm pretty sure that OpenSSL will have good implementations by
>> the time people will actively start using the new hash algorithm
>> and then hashlib will automatically make that available (hashlib
>> uses the OpenSSL EVP abstraction, so will be able to use any
>> new algorithms added to OpenSSL).
>>
>> However, if we add the reference implementation now, we'd then be
>> left with 1.2MB unnecessary code in the stdlib.
>>
> 
> I'm probably showing my ignorance here, but couldn't we swap in the OpenSSL
> implementation when that becomes available?

We could, but only if we don't expose separate interfaces
for the hashes and not add them to hashlib.

hashlib.algorithms
hashlib.algorithms_guaranteed

> -CHB
> 
> 
> (*) People are just now starting to move from SHA-1 to SHA-2
>> and SHA-2 was standardized in 2001. Python received SHA-2 support
>> in 2006. So there's plenty of time to decide :-)
> 
> 
> can't deny the history, nor the inertia -- but that doesn't make it a good
> thing...
> 
> 
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/mal%40egenix.com
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, May 27 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From levkivskyi at gmail.com  Fri May 27 14:58:26 2016
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Fri, 27 May 2016 20:58:26 +0200
Subject: [Python-Dev] Adding NewType() to PEP 484
Message-ID: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>

Hi,

It has been proposed to enhance the typing module with a NewType function
that allows to define simple unique types with almost zero runtime overhead.

The PR containing actual implementation and PEP 484 update is here:
https://github.com/python/typing/pull/226

Review comments are very welcome.

Best regards,
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160527/5a6d297a/attachment.html>

From brett at python.org  Fri May 27 15:50:40 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 27 May 2016 19:50:40 +0000
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
Message-ID: <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>

Patches to Python's stdlib should go through bugs.python.org so it isn't
lost in email.

On Fri, May 27, 2016, 12:00 Ivan Levkivskyi <levkivskyi at gmail.com> wrote:

> Hi,
>
> It has been proposed to enhance the typing module with a NewType function
> that allows to define simple unique types with almost zero runtime
> overhead.
>
> The PR containing actual implementation and PEP 484 update is here:
> https://github.com/python/typing/pull/226
>
> Review comments are very welcome.
>
> Best regards,
> Ivan
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160527/8579a38e/attachment.html>

From gvanrossum at gmail.com  Fri May 27 15:54:33 2016
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Fri, 27 May 2016 12:54:33 -0700
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
Message-ID: <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>

Sorry, this is more meant to be the start of a discussion about the
proposed feature. And typing.py has its own upstream repo (like asyncio).

--Guido (mobile)
On May 27, 2016 12:52 PM, "Brett Cannon" <brett at python.org> wrote:

> Patches to Python's stdlib should go through bugs.python.org so it isn't
> lost in email.
>
> On Fri, May 27, 2016, 12:00 Ivan Levkivskyi <levkivskyi at gmail.com> wrote:
>
>> Hi,
>>
>> It has been proposed to enhance the typing module with a NewType function
>> that allows to define simple unique types with almost zero runtime
>> overhead.
>>
>> The PR containing actual implementation and PEP 484 update is here:
>> https://github.com/python/typing/pull/226
>>
>> Review comments are very welcome.
>>
>> Best regards,
>> Ivan
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160527/06dfdf05/attachment-0001.html>

From victor.stinner at gmail.com  Fri May 27 16:03:36 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 27 May 2016 22:03:36 +0200
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <57482779.6030100@egenix.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com>
Message-ID: <CAMpsgwbKvDYBJMmmw42o6n8bY7DWi_huRuXZTBtGH939eGy_Ww@mail.gmail.com>

Le vendredi 27 mai 2016, M.-A. Lemburg <mal at egenix.com> a ?crit :
>
> The curent patch is 1.2MB for SHA-3 - that's pretty heavy for just
> a few hash functions, which aren't in any wide spread use yet and
> probably won't be for quite a few years ahead.


Oh wow, it's so fat? Why is it so big? Can't we use a lighter version?

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160527/890e0fc9/attachment.html>

From rymg19 at gmail.com  Fri May 27 16:58:23 2016
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Fri, 27 May 2016 15:58:23 -0500
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <CAMpsgwbKvDYBJMmmw42o6n8bY7DWi_huRuXZTBtGH939eGy_Ww@mail.gmail.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com>
 <CAMpsgwbKvDYBJMmmw42o6n8bY7DWi_huRuXZTBtGH939eGy_Ww@mail.gmail.com>
Message-ID: <CAO41-mNBLSnZpJBFKqsuW4v4bi4vwVYhzvhbrBsMTEZ0fpkpuA@mail.gmail.com>

On May 27, 2016 3:04 PM, "Victor Stinner" <victor.stinner at gmail.com> wrote:
>
> Le vendredi 27 mai 2016, M.-A. Lemburg <mal at egenix.com> a ?crit :
>>
>> The curent patch is 1.2MB for SHA-3 - that's pretty heavy for just
>>
>> a few hash functions, which aren't in any wide spread use yet and
>> probably won't be for quite a few years ahead.
>
>
> Oh wow, it's so fat? Why is it so big? Can't we use a lighter version?
>

The stark majority of the patch is Lib/test/vectors/sha3_224.txt, which
seems to be (as the file path implies) just test data. A whopping >1k LOC
of really long hashes.

> Victor
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
>

--
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160527/abb3f86a/attachment.html>

From mal at egenix.com  Fri May 27 17:41:26 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 27 May 2016 23:41:26 +0200
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <CAO41-mNBLSnZpJBFKqsuW4v4bi4vwVYhzvhbrBsMTEZ0fpkpuA@mail.gmail.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com>
 <CAMpsgwbKvDYBJMmmw42o6n8bY7DWi_huRuXZTBtGH939eGy_Ww@mail.gmail.com>
 <CAO41-mNBLSnZpJBFKqsuW4v4bi4vwVYhzvhbrBsMTEZ0fpkpuA@mail.gmail.com>
Message-ID: <5748BF06.50606@egenix.com>

On 27.05.2016 22:58, Ryan Gonzalez wrote:
> On May 27, 2016 3:04 PM, "Victor Stinner" <victor.stinner at gmail.com> wrote:
>>
>> Le vendredi 27 mai 2016, M.-A. Lemburg <mal at egenix.com> a ?crit :
>>>
>>> The current patch is 1.2MB for SHA-3 - that's pretty heavy for just
>>> a few hash functions, which aren't in any wide spread use yet and
>>> probably won't be for quite a few years ahead.
>>
>>
>> Oh wow, it's so fat? Why is it so big? Can't we use a lighter version?
>>
> 
> The stark majority of the patch is Lib/test/vectors/sha3_224.txt, which
> seems to be (as the file path implies) just test data. A whopping >1k LOC
> of really long hashes.

Right. There's about 1MB test data in the patch, but even
without that data, the patch adds more than 6400 lines of code.

If we add this now, there should at least be an exit strategy
to remove the code again, when OpenSSL ships with the same
code, IMO.

Aside: BLAKE2 has already landed in OpenSSL 1.1.0:

https://github.com/openssl/openssl/tree/master/crypto/blake2

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, May 27 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From donald at stufft.io  Fri May 27 17:46:27 2016
From: donald at stufft.io (Donald Stufft)
Date: Fri, 27 May 2016 17:46:27 -0400
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <5748BF06.50606@egenix.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com>
 <CAMpsgwbKvDYBJMmmw42o6n8bY7DWi_huRuXZTBtGH939eGy_Ww@mail.gmail.com>
 <CAO41-mNBLSnZpJBFKqsuW4v4bi4vwVYhzvhbrBsMTEZ0fpkpuA@mail.gmail.com>
 <5748BF06.50606@egenix.com>
Message-ID: <074FDA7C-1079-403A-B567-D1709A8E3F8C@stufft.io>


> On May 27, 2016, at 5:41 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> 
> If we add this now, there should at least be an exit strategy
> to remove the code again, when OpenSSL ships with the same
> code, IMO.


I think it is a clear win to have the fallback implementations in cases where people either don?t have OpenSSL or don?t have a new enough OpenSSL for those implementations. Not having the fallback just makes it more difficult for people to rely on those hash functions.

?
Donald Stufft




From mal at egenix.com  Fri May 27 18:08:38 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Sat, 28 May 2016 00:08:38 +0200
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <074FDA7C-1079-403A-B567-D1709A8E3F8C@stufft.io>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com>
 <CAMpsgwbKvDYBJMmmw42o6n8bY7DWi_huRuXZTBtGH939eGy_Ww@mail.gmail.com>
 <CAO41-mNBLSnZpJBFKqsuW4v4bi4vwVYhzvhbrBsMTEZ0fpkpuA@mail.gmail.com>
 <5748BF06.50606@egenix.com> <074FDA7C-1079-403A-B567-D1709A8E3F8C@stufft.io>
Message-ID: <5748C566.1070904@egenix.com>

On 27.05.2016 23:46, Donald Stufft wrote:
> 
>> On May 27, 2016, at 5:41 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>>
>> If we add this now, there should at least be an exit strategy
>> to remove the code again, when OpenSSL ships with the same
>> code, IMO.
> 
> I think it is a clear win to have the fallback implementations in cases where people either don?t have OpenSSL or don?t have a new enough OpenSSL for those implementations. Not having the fallback just makes it more difficult for people to rely on those hash functions.

This will only be needed once the stdlib itself starts requiring
support for some of these hashes and for that we could add
a pure Python implementation, eg.

https://github.com/coruus/py-keccak

In all other cases, you can simply add the support via a
package such as Bj?rn's or Christian's.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, May 27 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From njs at pobox.com  Fri May 27 18:52:00 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 27 May 2016 15:52:00 -0700
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <5748C566.1070904@egenix.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com>
 <CAMpsgwbKvDYBJMmmw42o6n8bY7DWi_huRuXZTBtGH939eGy_Ww@mail.gmail.com>
 <CAO41-mNBLSnZpJBFKqsuW4v4bi4vwVYhzvhbrBsMTEZ0fpkpuA@mail.gmail.com>
 <5748BF06.50606@egenix.com>
 <074FDA7C-1079-403A-B567-D1709A8E3F8C@stufft.io>
 <5748C566.1070904@egenix.com>
Message-ID: <CAPJVwB=y3+=yJTAjTwU6xBzNQ4SEsi1J=H+Wv5QM1iSDmE1H4Q@mail.gmail.com>

On Fri, May 27, 2016 at 3:08 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> On 27.05.2016 23:46, Donald Stufft wrote:
>>
>>> On May 27, 2016, at 5:41 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>>>
>>> If we add this now, there should at least be an exit strategy
>>> to remove the code again, when OpenSSL ships with the same
>>> code, IMO.
>>
>> I think it is a clear win to have the fallback implementations in cases where people either don?t have OpenSSL or don?t have a new enough OpenSSL for those implementations. Not having the fallback just makes it more difficult for people to rely on those hash functions.
>
> This will only be needed once the stdlib itself starts requiring
> support for some of these hashes and for that we could add
> a pure Python implementation, eg.
>
> https://github.com/coruus/py-keccak
>
> In all other cases, you can simply add the support via a
> package such as Bj?rn's or Christian's.

SHA-3 and BLAKE are extremely widely accepted standards, our users
will expect them, and they're significant improvements over all the
current hashes in the algorithms_guaranteed list. If we demote them to
second-class support (by making them only available in some builds, or
using a slow pure Python implementation), then we'll be encouraging
users to use inferior hashes. We shouldn't do this without a very good
reason, and I don't see anything very convincing here... by all means
drop the megabyte of test data, but why does it matter how many lines
of code the algorithm is? No python developer will ever have to look
at it -- hash code by its nature is *very* low maintenance (it either
computes the right function or it doesn't, and the right answer never
changes). And in unlikely case where some terrible unexpected bug is
discovered then the only maintenance needed will be to delete the
current impl and drop-in whatever the new fixed one is.

So +1 to adding SHA-3 and BLAKE to algorithms_guaranteed.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From gvanrossum at gmail.com  Fri May 27 19:01:11 2016
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Fri, 27 May 2016 16:01:11 -0700
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
Message-ID: <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>

Also -- the most important thing. :-) What to call these things? We're
pretty much settled on the semantics and how to create them (A =
NewType('A', int)) but what should we call types like A when we're
talking about them? "New types" sounds awkward.

On Fri, May 27, 2016 at 12:54 PM, Guido van Rossum <gvanrossum at gmail.com> wrote:
> Sorry, this is more meant to be the start of a discussion about the proposed
> feature. And typing.py has its own upstream repo (like asyncio).
>
> --Guido (mobile)
>
> On May 27, 2016 12:52 PM, "Brett Cannon" <brett at python.org> wrote:
>>
>> Patches to Python's stdlib should go through bugs.python.org so it isn't
>> lost in email.
>>
>>
>> On Fri, May 27, 2016, 12:00 Ivan Levkivskyi <levkivskyi at gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> It has been proposed to enhance the typing module with a NewType function
>>> that allows to define simple unique types with almost zero runtime
>>> overhead.
>>>
>>> The PR containing actual implementation and PEP 484 update is here:
>>> https://github.com/python/typing/pull/226
>>>
>>> Review comments are very welcome.
>>>
>>> Best regards,
>>> Ivan
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>



-- 
--Guido van Rossum (python.org/~guido)

From mafagafogigante at gmail.com  Fri May 27 19:19:15 2016
From: mafagafogigante at gmail.com (Bernardo Sulzbach)
Date: Fri, 27 May 2016 20:19:15 -0300
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <CAPJVwB=y3+=yJTAjTwU6xBzNQ4SEsi1J=H+Wv5QM1iSDmE1H4Q@mail.gmail.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com>
 <CAMpsgwbKvDYBJMmmw42o6n8bY7DWi_huRuXZTBtGH939eGy_Ww@mail.gmail.com>
 <CAO41-mNBLSnZpJBFKqsuW4v4bi4vwVYhzvhbrBsMTEZ0fpkpuA@mail.gmail.com>
 <5748BF06.50606@egenix.com> <074FDA7C-1079-403A-B567-D1709A8E3F8C@stufft.io>
 <5748C566.1070904@egenix.com>
 <CAPJVwB=y3+=yJTAjTwU6xBzNQ4SEsi1J=H+Wv5QM1iSDmE1H4Q@mail.gmail.com>
Message-ID: <11e202b5-c129-b587-6594-45470591ebdf@gmail.com>

On 05/27/2016 07:52 PM, Nathaniel Smith wrote:
> If we demote them to second-class support (by making them only
 > available in some builds, or using a slow pure Python implementation),
 > then we'll be encouraging users to use inferior hashes. We shouldn't
 > do this without a very good reason.

I agree. And I really think we shouldn't even ship pure Python 
implementations of these hashing algorithms. I am fairly confident that 
these algorithms would be prohibitively slow if written in pure Python.

From steve at pearwood.info  Fri May 27 20:58:31 2016
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 28 May 2016 10:58:31 +1000
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
Message-ID: <20160528005826.GJ12028@ando.pearwood.info>

On Fri, May 27, 2016 at 04:01:11PM -0700, Guido van Rossum wrote:
> Also -- the most important thing. :-) What to call these things? We're
> pretty much settled on the semantics and how to create them (A =
> NewType('A', int)) but what should we call types like A when we're
> talking about them? "New types" sounds awkward.

TypeAlias? Because A is an alias for int?


-- 
Steve

From greg.ewing at canterbury.ac.nz  Fri May 27 21:51:46 2016
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 28 May 2016 13:51:46 +1200
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
Message-ID: <5748F9B2.5000604@canterbury.ac.nz>

Guido van Rossum wrote:
> Also -- the most important thing. :-) What to call these things? We're
> pretty much settled on the semantics and how to create them (A =
> NewType('A', int)) but what should we call types like A when we're
> talking about them? "New types" sounds awkward.

Fake types? Virtual types? Pseudo-types?

-- 
Greg

From greg.ewing at canterbury.ac.nz  Fri May 27 21:59:02 2016
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 28 May 2016 13:59:02 +1200
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <20160528005826.GJ12028@ando.pearwood.info>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
 <20160528005826.GJ12028@ando.pearwood.info>
Message-ID: <5748FB66.7090107@canterbury.ac.nz>

Steven D'Aprano wrote:

> TypeAlias? Because A is an alias for int?

That suggests it's just another name for the same type,
but it's not. It's a distinct type as far as the static
type checker is concerned.

-- 
Greg

From guido at python.org  Sat May 28 00:26:29 2016
From: guido at python.org (Guido van Rossum)
Date: Fri, 27 May 2016 21:26:29 -0700
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <5748FB66.7090107@canterbury.ac.nz>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
 <20160528005826.GJ12028@ando.pearwood.info>
 <5748FB66.7090107@canterbury.ac.nz>
Message-ID: <CAP7+vJ+CAVuXaRgA=UgKjurYPA3ndzR=+Tp4MeRXEyD6h4MoeQ@mail.gmail.com>

On Fri, May 27, 2016 at 6:59 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Steven D'Aprano wrote:
>
>> TypeAlias? Because A is an alias for int?
>
>
> That suggests it's just another name for the same type,
> but it's not. It's a distinct type as far as the static
> type checker is concerned.

We discussed this over dinner at PyCon, some ideas we came up with:

- Dependent types, harking back to a similar concept in Ada
(https://en.wikibooks.org/wiki/Ada_Programming/Type_System#Derived_types)
which in that language is also spelled with "new".

- New type

- Distinguished Type

- Distinguished Subtype

- Distinguished Type Alias

- Distinguished Alias

- BoatyMcBoatType

The nice thing about "distinguished" is that it's a relatively rare
word so it is easy to remember or look up.

Personally I'm still in favor of Derived type (but I'm more into
ancient programming languages than most folks here). I could also live
with Distinguished Type.

The problem with Alias is that these types are *not* aliases (at least
not to the type checker, which is where we need the terminology -- so
we can talk about what the type checker does to these).

> Fake types? Virtual types? Pseudo-types?

I'm not keen on any of these.

-- 
--Guido van Rossum (python.org/~guido)

From nd at perlig.de  Sat May 28 07:26:57 2016
From: nd at perlig.de (=?iso-8859-1?q?Andr=E9_Malo?=)
Date: Sat, 28 May 2016 13:26:57 +0200
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <20160528005826.GJ12028@ando.pearwood.info>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
 <20160528005826.GJ12028@ando.pearwood.info>
Message-ID: <201605281326.57929@news.perlig.de>

* Steven D'Aprano wrote:

> On Fri, May 27, 2016 at 04:01:11PM -0700, Guido van Rossum wrote:
> > Also -- the most important thing. :-) What to call these things? We're
> > pretty much settled on the semantics and how to create them (A =
> > NewType('A', int)) but what should we call types like A when we're
> > talking about them? "New types" sounds awkward.
>
> TypeAlias? Because A is an alias for int?

I like the view C takes on this: typedef.

Cheers,
-- 
Wer sein Wissen nicht teilen will, besitzt wahrscheinlich zu wenig davon.
  -- Unbekannt

From steve at pearwood.info  Sat May 28 08:24:45 2016
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 28 May 2016 22:24:45 +1000
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <CAP7+vJ+CAVuXaRgA=UgKjurYPA3ndzR=+Tp4MeRXEyD6h4MoeQ@mail.gmail.com>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
 <20160528005826.GJ12028@ando.pearwood.info>
 <5748FB66.7090107@canterbury.ac.nz>
 <CAP7+vJ+CAVuXaRgA=UgKjurYPA3ndzR=+Tp4MeRXEyD6h4MoeQ@mail.gmail.com>
Message-ID: <20160528122445.GK12028@ando.pearwood.info>

On Fri, May 27, 2016 at 09:26:29PM -0700, Guido van Rossum wrote:

> We discussed this over dinner at PyCon, some ideas we came up with:
> 
> - Dependent types, harking back to a similar concept in Ada
> (https://en.wikibooks.org/wiki/Ada_Programming/Type_System#Derived_types)
> which in that language is also spelled with "new".

I started to explain this to my non-programmer wife, I got as far as 
explaining types, and that we need a name for this thing, and she 
stopped me and said 

"Please don't tell me this is leading to TypyMcTypeCheck."

[...]
> - BoatyMcBoatType

> The nice thing about "distinguished" is that it's a relatively rare
> word so it is easy to remember or look up.

I would have thought that being rare, it would be *harder* to remember.


> Personally I'm still in favor of Derived type (but I'm more into
> ancient programming languages than most folks here). I could also live
> with Distinguished Type.

I think Derived Type is the nicest of the options. It accurately 
describes what it is: a type derived from another. And its shorter and 
easy to both say and write than "Distinguished type" (which sounds like 
"distinguished gentlemen" -- is it wearing a monocle and a top hat?).

"Distinguished" is too vague for my tastes, it might as well be 
"flibblegubble type". *All* types are distinguished, the type checker 
has to distinguish int from float from list from str, so to call 
NewType("userid", int) a "distinguished type" is only to call it a type.



-- 
Steve

From guido at python.org  Sat May 28 10:35:27 2016
From: guido at python.org (Guido van Rossum)
Date: Sat, 28 May 2016 07:35:27 -0700
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <20160528122445.GK12028@ando.pearwood.info>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
 <20160528005826.GJ12028@ando.pearwood.info>
 <5748FB66.7090107@canterbury.ac.nz>
 <CAP7+vJ+CAVuXaRgA=UgKjurYPA3ndzR=+Tp4MeRXEyD6h4MoeQ@mail.gmail.com>
 <20160528122445.GK12028@ando.pearwood.info>
Message-ID: <CAP7+vJLhR0whVf-Oiwz=9fcSgZgwp8zhZOqZzYq50=VFC_8Yfg@mail.gmail.com>

Just to add to the list of options, Twitter also came up with

- invention

- DomainType

- TypedAlias

But seriously I think we should just decide between Derived Type and
Distinguished Type [Alias].

The latter comes from the idea that when you write e.g.

UserId = int

then UserId is a type alias (that's existing PEP 484 terminology) and
the type checker doesn't distinguish it from int -- you can use it in
places where you logically expect a UserId but to the type checker
those variables have the type int.

There is even a neat potential "origin story" that would explain why
we'd call it Distinguished Type Alias.

The story is about gradually converting a large code base to being
consistent: initially you make UserId a regular type alias and you
start putting it in your code incrementally, making sure it has no
mypy errors as you go (but this just treats it as an int). After days,
when you think you are done, you change UserId to a distinguished type
alias and then mypy will point out the places where you've missed
something or you're doing something questionable with user IDs.

And yes, in the wider context of subclassing, Derived Type is probably
confusing because a subclass is also called a derived class.


On Sat, May 28, 2016 at 5:24 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Fri, May 27, 2016 at 09:26:29PM -0700, Guido van Rossum wrote:
>
>> We discussed this over dinner at PyCon, some ideas we came up with:
>>
>> - Dependent types, harking back to a similar concept in Ada
>> (https://en.wikibooks.org/wiki/Ada_Programming/Type_System#Derived_types)
>> which in that language is also spelled with "new".
>
> I started to explain this to my non-programmer wife, I got as far as
> explaining types, and that we need a name for this thing, and she
> stopped me and said
>
> "Please don't tell me this is leading to TypyMcTypeCheck."
>
> [...]
>> - BoatyMcBoatType
>
>> The nice thing about "distinguished" is that it's a relatively rare
>> word so it is easy to remember or look up.
>
> I would have thought that being rare, it would be *harder* to remember.
>
>
>> Personally I'm still in favor of Derived type (but I'm more into
>> ancient programming languages than most folks here). I could also live
>> with Distinguished Type.
>
> I think Derived Type is the nicest of the options. It accurately
> describes what it is: a type derived from another. And its shorter and
> easy to both say and write than "Distinguished type" (which sounds like
> "distinguished gentlemen" -- is it wearing a monocle and a top hat?).
>
> "Distinguished" is too vague for my tastes, it might as well be
> "flibblegubble type". *All* types are distinguished, the type checker
> has to distinguish int from float from list from str, so to call
> NewType("userid", int) a "distinguished type" is only to call it a type.
>
>
>
> --
> Steve
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From steve.dower at python.org  Sat May 28 11:19:35 2016
From: steve.dower at python.org (Steve Dower)
Date: Sat, 28 May 2016 08:19:35 -0700
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <CAP7+vJLhR0whVf-Oiwz=9fcSgZgwp8zhZOqZzYq50=VFC_8Yfg@mail.gmail.com>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
 <20160528005826.GJ12028@ando.pearwood.info>
 <5748FB66.7090107@canterbury.ac.nz>
 <CAP7+vJ+CAVuXaRgA=UgKjurYPA3ndzR=+Tp4MeRXEyD6h4MoeQ@mail.gmail.com>
 <20160528122445.GK12028@ando.pearwood.info>
 <CAP7+vJLhR0whVf-Oiwz=9fcSgZgwp8zhZOqZzYq50=VFC_8Yfg@mail.gmail.com>
Message-ID: <E1b6g1q-0002er-KN@se2-syd.hostedmail.net.au>

Did anyone suggest "distinct type alias"?

Regardless of what name, I'm fairly sure people will call it whatever the function to create it is called. So if the function is typings.distinguish_type(...), then distinguished will stick.

Top-posted from my Windows Phone

-----Original Message-----
From: "Guido van Rossum" <guido at python.org>
Sent: ?5/?28/?2016 7:38
To: "Steven D'Aprano" <steve at pearwood.info>
Cc: "Python-Dev" <python-dev at python.org>
Subject: Re: [Python-Dev] Adding NewType() to PEP 484

Just to add to the list of options, Twitter also came up with

- invention

- DomainType

- TypedAlias

But seriously I think we should just decide between Derived Type and
Distinguished Type [Alias].

The latter comes from the idea that when you write e.g.

UserId = int

then UserId is a type alias (that's existing PEP 484 terminology) and
the type checker doesn't distinguish it from int -- you can use it in
places where you logically expect a UserId but to the type checker
those variables have the type int.

There is even a neat potential "origin story" that would explain why
we'd call it Distinguished Type Alias.

The story is about gradually converting a large code base to being
consistent: initially you make UserId a regular type alias and you
start putting it in your code incrementally, making sure it has no
mypy errors as you go (but this just treats it as an int). After days,
when you think you are done, you change UserId to a distinguished type
alias and then mypy will point out the places where you've missed
something or you're doing something questionable with user IDs.

And yes, in the wider context of subclassing, Derived Type is probably
confusing because a subclass is also called a derived class.


On Sat, May 28, 2016 at 5:24 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Fri, May 27, 2016 at 09:26:29PM -0700, Guido van Rossum wrote:
>
>> We discussed this over dinner at PyCon, some ideas we came up with:
>>
>> - Dependent types, harking back to a similar concept in Ada
>> (https://en.wikibooks.org/wiki/Ada_Programming/Type_System#Derived_types)
>> which in that language is also spelled with "new".
>
> I started to explain this to my non-programmer wife, I got as far as
> explaining types, and that we need a name for this thing, and she
> stopped me and said
>
> "Please don't tell me this is leading to TypyMcTypeCheck."
>
> [...]
>> - BoatyMcBoatType
>
>> The nice thing about "distinguished" is that it's a relatively rare
>> word so it is easy to remember or look up.
>
> I would have thought that being rare, it would be *harder* to remember.
>
>
>> Personally I'm still in favor of Derived type (but I'm more into
>> ancient programming languages than most folks here). I could also live
>> with Distinguished Type.
>
> I think Derived Type is the nicest of the options. It accurately
> describes what it is: a type derived from another. And its shorter and
> easy to both say and write than "Distinguished type" (which sounds like
> "distinguished gentlemen" -- is it wearing a monocle and a top hat?).
>
> "Distinguished" is too vague for my tastes, it might as well be
> "flibblegubble type". *All* types are distinguished, the type checker
> has to distinguish int from float from list from str, so to call
> NewType("userid", int) a "distinguished type" is only to call it a type.
>
>
>
> --
> Steve
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160528/62243ba0/attachment.html>

From levkivskyi at gmail.com  Sat May 28 05:01:03 2016
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Sat, 28 May 2016 11:01:03 +0200
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <CAP7+vJ+CAVuXaRgA=UgKjurYPA3ndzR=+Tp4MeRXEyD6h4MoeQ@mail.gmail.com>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
 <20160528005826.GJ12028@ando.pearwood.info>
 <5748FB66.7090107@canterbury.ac.nz>
 <CAP7+vJ+CAVuXaRgA=UgKjurYPA3ndzR=+Tp4MeRXEyD6h4MoeQ@mail.gmail.com>
Message-ID: <CAOMjWknTjS46RXEqFA5FyJSdcCdnhE=B73JviB1wbidQuB-jvg@mail.gmail.com>

On 28 May 2016 at 06:26, Guido van Rossum <guido at python.org> wrote:

> Personally I'm still in favor of Derived type (but I'm more into
> ancient programming languages than most folks here). I could also live
> with Distinguished Type.

I think both "derived" and "distinguished" are OK, but I am more in favour
of "distinguished".
The reason  is sometimes people might talk about derived types while
meaning normal derived classes.
Also, indeed "distinguished" is a rare word, so that it will not cause
confusions and easy to look up.

--
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160528/801d601b/attachment.html>

From levkivskyi at gmail.com  Sat May 28 06:46:55 2016
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Sat, 28 May 2016 12:46:55 +0200
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <CAOMjWknTjS46RXEqFA5FyJSdcCdnhE=B73JviB1wbidQuB-jvg@mail.gmail.com>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
 <20160528005826.GJ12028@ando.pearwood.info>
 <5748FB66.7090107@canterbury.ac.nz>
 <CAP7+vJ+CAVuXaRgA=UgKjurYPA3ndzR=+Tp4MeRXEyD6h4MoeQ@mail.gmail.com>
 <CAOMjWknTjS46RXEqFA5FyJSdcCdnhE=B73JviB1wbidQuB-jvg@mail.gmail.com>
Message-ID: <CAOMjWkn5S6XAYiOzQgMP_WtDovfwS0JqsU5A1vucQmPCk0gL=w@mail.gmail.com>

There is another interesting idea from Twitter: call them type wrappers.

--
Ivan

On 28 May 2016 at 11:01, Ivan Levkivskyi <levkivskyi at gmail.com> wrote:
>
> On 28 May 2016 at 06:26, Guido van Rossum <guido at python.org> wrote:
>
> > Personally I'm still in favor of Derived type (but I'm more into
> > ancient programming languages than most folks here). I could also live
> > with Distinguished Type.
>
> I think both "derived" and "distinguished" are OK, but I am more in
favour of "distinguished".
> The reason  is sometimes people might talk about derived types while
meaning normal derived classes.
> Also, indeed "distinguished" is a rare word, so that it will not cause
confusions and easy to look up.
>
> --
> Ivan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160528/0cc02b03/attachment.html>

From guido at python.org  Sat May 28 13:11:54 2016
From: guido at python.org (Guido van Rossum)
Date: Sat, 28 May 2016 10:11:54 -0700
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <E1b6g1q-0002er-KN@se2-syd.hostedmail.net.au>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
 <20160528005826.GJ12028@ando.pearwood.info>
 <5748FB66.7090107@canterbury.ac.nz>
 <CAP7+vJ+CAVuXaRgA=UgKjurYPA3ndzR=+Tp4MeRXEyD6h4MoeQ@mail.gmail.com>
 <20160528122445.GK12028@ando.pearwood.info>
 <CAP7+vJLhR0whVf-Oiwz=9fcSgZgwp8zhZOqZzYq50=VFC_8Yfg@mail.gmail.com>
 <E1b6g1q-0002er-KN@se2-syd.hostedmail.net.au>
Message-ID: <CAP7+vJ+zA6S4Fb-vkmGcoB=_MCtC2YmH32AXt+qoSxC6Cwx=gA@mail.gmail.com>

Ohhhhh, another D-word! I really like distinct.

On Sat, May 28, 2016 at 8:19 AM, Steve Dower <steve.dower at python.org> wrote:
> Did anyone suggest "distinct type alias"?
>
> Regardless of what name, I'm fairly sure people will call it whatever the
> function to create it is called. So if the function is
> typings.distinguish_type(...), then distinguished will stick.
>
> Top-posted from my Windows Phone
> ________________________________
> From: Guido van Rossum
> Sent: ?5/?28/?2016 7:38
> To: Steven D'Aprano
> Cc: Python-Dev
> Subject: Re: [Python-Dev] Adding NewType() to PEP 484
>
> Just to add to the list of options, Twitter also came up with
>
> - invention
>
> - DomainType
>
> - TypedAlias
>
> But seriously I think we should just decide between Derived Type and
> Distinguished Type [Alias].
>
> The latter comes from the idea that when you write e.g.
>
> UserId = int
>
> then UserId is a type alias (that's existing PEP 484 terminology) and
> the type checker doesn't distinguish it from int -- you can use it in
> places where you logically expect a UserId but to the type checker
> those variables have the type int.
>
> There is even a neat potential "origin story" that would explain why
> we'd call it Distinguished Type Alias.
>
> The story is about gradually converting a large code base to being
> consistent: initially you make UserId a regular type alias and you
> start putting it in your code incrementally, making sure it has no
> mypy errors as you go (but this just treats it as an int). After days,
> when you think you are done, you change UserId to a distinguished type
> alias and then mypy will point out the places where you've missed
> something or you're doing something questionable with user IDs.
>
> And yes, in the wider context of subclassing, Derived Type is probably
> confusing because a subclass is also called a derived class.
>
>
> On Sat, May 28, 2016 at 5:24 AM, Steven D'Aprano <steve at pearwood.info>
> wrote:
>> On Fri, May 27, 2016 at 09:26:29PM -0700, Guido van Rossum wrote:
>>
>>> We discussed this over dinner at PyCon, some ideas we came up with:
>>>
>>> - Dependent types, harking back to a similar concept in Ada
>>> (https://en.wikibooks.org/wiki/Ada_Programming/Type_System#Derived_types)
>>> which in that language is also spelled with "new".
>>
>> I started to explain this to my non-programmer wife, I got as far as
>> explaining types, and that we need a name for this thing, and she
>> stopped me and said
>>
>> "Please don't tell me this is leading to TypyMcTypeCheck."
>>
>> [...]
>>> - BoatyMcBoatType
>>
>>> The nice thing about "distinguished" is that it's a relatively rare
>>> word so it is easy to remember or look up.
>>
>> I would have thought that being rare, it would be *harder* to remember.
>>
>>
>>> Personally I'm still in favor of Derived type (but I'm more into
>>> ancient programming languages than most folks here). I could also live
>>> with Distinguished Type.
>>
>> I think Derived Type is the nicest of the options. It accurately
>> describes what it is: a type derived from another. And its shorter and
>> easy to both say and write than "Distinguished type" (which sounds like
>> "distinguished gentlemen" -- is it wearing a monocle and a top hat?).
>>
>> "Distinguished" is too vague for my tastes, it might as well be
>> "flibblegubble type". *All* types are distinguished, the type checker
>> has to distinguish int from float from list from str, so to call
>> NewType("userid", int) a "distinguished type" is only to call it a type.
>>
>>
>>
>> --
>> Steve
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From levkivskyi at gmail.com  Sat May 28 13:17:33 2016
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Sat, 28 May 2016 19:17:33 +0200
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <CAP7+vJ+zA6S4Fb-vkmGcoB=_MCtC2YmH32AXt+qoSxC6Cwx=gA@mail.gmail.com>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
 <20160528005826.GJ12028@ando.pearwood.info>
 <5748FB66.7090107@canterbury.ac.nz>
 <CAP7+vJ+CAVuXaRgA=UgKjurYPA3ndzR=+Tp4MeRXEyD6h4MoeQ@mail.gmail.com>
 <20160528122445.GK12028@ando.pearwood.info>
 <CAP7+vJLhR0whVf-Oiwz=9fcSgZgwp8zhZOqZzYq50=VFC_8Yfg@mail.gmail.com>
 <E1b6g1q-0002er-KN@se2-syd.hostedmail.net.au>
 <CAP7+vJ+zA6S4Fb-vkmGcoB=_MCtC2YmH32AXt+qoSxC6Cwx=gA@mail.gmail.com>
Message-ID: <CAOMjWkmkjodtZgSqzbUDyE1HzMq4Qo9E25j6Qr17cCQVcig0wQ@mail.gmail.com>

My final vote goes to "distinct type alias".
But how should we call the function? NewType? Or should we change it to
DistinctAlias or something?

--
Ivan
28 ??? 2016 19:14 "Guido van Rossum" <guido at python.org> ????:

> Ohhhhh, another D-word! I really like distinct.
>
> On Sat, May 28, 2016 at 8:19 AM, Steve Dower <steve.dower at python.org>
> wrote:
> > Did anyone suggest "distinct type alias"?
> >
> > Regardless of what name, I'm fairly sure people will call it whatever the
> > function to create it is called. So if the function is
> > typings.distinguish_type(...), then distinguished will stick.
> >
> > Top-posted from my Windows Phone
> > ________________________________
> > From: Guido van Rossum
> > Sent: ?5/?28/?2016 7:38
> > To: Steven D'Aprano
> > Cc: Python-Dev
> > Subject: Re: [Python-Dev] Adding NewType() to PEP 484
> >
> > Just to add to the list of options, Twitter also came up with
> >
> > - invention
> >
> > - DomainType
> >
> > - TypedAlias
> >
> > But seriously I think we should just decide between Derived Type and
> > Distinguished Type [Alias].
> >
> > The latter comes from the idea that when you write e.g.
> >
> > UserId = int
> >
> > then UserId is a type alias (that's existing PEP 484 terminology) and
> > the type checker doesn't distinguish it from int -- you can use it in
> > places where you logically expect a UserId but to the type checker
> > those variables have the type int.
> >
> > There is even a neat potential "origin story" that would explain why
> > we'd call it Distinguished Type Alias.
> >
> > The story is about gradually converting a large code base to being
> > consistent: initially you make UserId a regular type alias and you
> > start putting it in your code incrementally, making sure it has no
> > mypy errors as you go (but this just treats it as an int). After days,
> > when you think you are done, you change UserId to a distinguished type
> > alias and then mypy will point out the places where you've missed
> > something or you're doing something questionable with user IDs.
> >
> > And yes, in the wider context of subclassing, Derived Type is probably
> > confusing because a subclass is also called a derived class.
> >
> >
> > On Sat, May 28, 2016 at 5:24 AM, Steven D'Aprano <steve at pearwood.info>
> > wrote:
> >> On Fri, May 27, 2016 at 09:26:29PM -0700, Guido van Rossum wrote:
> >>
> >>> We discussed this over dinner at PyCon, some ideas we came up with:
> >>>
> >>> - Dependent types, harking back to a similar concept in Ada
> >>> (
> https://en.wikibooks.org/wiki/Ada_Programming/Type_System#Derived_types)
> >>> which in that language is also spelled with "new".
> >>
> >> I started to explain this to my non-programmer wife, I got as far as
> >> explaining types, and that we need a name for this thing, and she
> >> stopped me and said
> >>
> >> "Please don't tell me this is leading to TypyMcTypeCheck."
> >>
> >> [...]
> >>> - BoatyMcBoatType
> >>
> >>> The nice thing about "distinguished" is that it's a relatively rare
> >>> word so it is easy to remember or look up.
> >>
> >> I would have thought that being rare, it would be *harder* to remember.
> >>
> >>
> >>> Personally I'm still in favor of Derived type (but I'm more into
> >>> ancient programming languages than most folks here). I could also live
> >>> with Distinguished Type.
> >>
> >> I think Derived Type is the nicest of the options. It accurately
> >> describes what it is: a type derived from another. And its shorter and
> >> easy to both say and write than "Distinguished type" (which sounds like
> >> "distinguished gentlemen" -- is it wearing a monocle and a top hat?).
> >>
> >> "Distinguished" is too vague for my tastes, it might as well be
> >> "flibblegubble type". *All* types are distinguished, the type checker
> >> has to distinguish int from float from list from str, so to call
> >> NewType("userid", int) a "distinguished type" is only to call it a type.
> >>
> >>
> >>
> >> --
> >> Steve
> >> _______________________________________________
> >> Python-Dev mailing list
> >> Python-Dev at python.org
> >> https://mail.python.org/mailman/listinfo/python-dev
> >> Unsubscribe:
> >> https://mail.python.org/mailman/options/python-dev/guido%40python.org
> >
> >
> >
> > --
> > --Guido van Rossum (python.org/~guido)
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> >
> https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/levkivskyi%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160528/21c84ba5/attachment.html>

From pludemann at google.com  Sat May 28 13:43:22 2016
From: pludemann at google.com (Peter Ludemann)
Date: Sat, 28 May 2016 10:43:22 -0700
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <CAOMjWkmkjodtZgSqzbUDyE1HzMq4Qo9E25j6Qr17cCQVcig0wQ@mail.gmail.com>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
 <20160528005826.GJ12028@ando.pearwood.info>
 <5748FB66.7090107@canterbury.ac.nz>
 <CAP7+vJ+CAVuXaRgA=UgKjurYPA3ndzR=+Tp4MeRXEyD6h4MoeQ@mail.gmail.com>
 <20160528122445.GK12028@ando.pearwood.info>
 <CAP7+vJLhR0whVf-Oiwz=9fcSgZgwp8zhZOqZzYq50=VFC_8Yfg@mail.gmail.com>
 <E1b6g1q-0002er-KN@se2-syd.hostedmail.net.au>
 <CAP7+vJ+zA6S4Fb-vkmGcoB=_MCtC2YmH32AXt+qoSxC6Cwx=gA@mail.gmail.com>
 <CAOMjWkmkjodtZgSqzbUDyE1HzMq4Qo9E25j6Qr17cCQVcig0wQ@mail.gmail.com>
Message-ID: <CACsRUKK8DKhMcMA_cO1WP6_haRjQRDTNQnFxJ=JpKGDvpLnDxA@mail.gmail.com>

I'm surprised that nobody has suggested UniqueType. ;)


On 28 May 2016 at 10:17, Ivan Levkivskyi <levkivskyi at gmail.com> wrote:

> My final vote goes to "distinct type alias".
> But how should we call the function? NewType? Or should we change it to
> DistinctAlias or something?
>
> --
> Ivan
> 28 ??? 2016 19:14 "Guido van Rossum" <guido at python.org> ????:
>
>> Ohhhhh, another D-word! I really like distinct.
>>
>> On Sat, May 28, 2016 at 8:19 AM, Steve Dower <steve.dower at python.org>
>> wrote:
>> > Did anyone suggest "distinct type alias"?
>> >
>> > Regardless of what name, I'm fairly sure people will call it whatever
>> the
>> > function to create it is called. So if the function is
>> > typings.distinguish_type(...), then distinguished will stick.
>> >
>> > Top-posted from my Windows Phone
>> > ________________________________
>> > From: Guido van Rossum
>> > Sent: ?5/?28/?2016 7:38
>> > To: Steven D'Aprano
>> > Cc: Python-Dev
>> > Subject: Re: [Python-Dev] Adding NewType() to PEP 484
>> >
>> > Just to add to the list of options, Twitter also came up with
>> >
>> > - invention
>> >
>> > - DomainType
>> >
>> > - TypedAlias
>> >
>> > But seriously I think we should just decide between Derived Type and
>> > Distinguished Type [Alias].
>> >
>> > The latter comes from the idea that when you write e.g.
>> >
>> > UserId = int
>> >
>> > then UserId is a type alias (that's existing PEP 484 terminology) and
>> > the type checker doesn't distinguish it from int -- you can use it in
>> > places where you logically expect a UserId but to the type checker
>> > those variables have the type int.
>> >
>> > There is even a neat potential "origin story" that would explain why
>> > we'd call it Distinguished Type Alias.
>> >
>> > The story is about gradually converting a large code base to being
>> > consistent: initially you make UserId a regular type alias and you
>> > start putting it in your code incrementally, making sure it has no
>> > mypy errors as you go (but this just treats it as an int). After days,
>> > when you think you are done, you change UserId to a distinguished type
>> > alias and then mypy will point out the places where you've missed
>> > something or you're doing something questionable with user IDs.
>> >
>> > And yes, in the wider context of subclassing, Derived Type is probably
>> > confusing because a subclass is also called a derived class.
>> >
>> >
>> > On Sat, May 28, 2016 at 5:24 AM, Steven D'Aprano <steve at pearwood.info>
>> > wrote:
>> >> On Fri, May 27, 2016 at 09:26:29PM -0700, Guido van Rossum wrote:
>> >>
>> >>> We discussed this over dinner at PyCon, some ideas we came up with:
>> >>>
>> >>> - Dependent types, harking back to a similar concept in Ada
>> >>> (
>> https://en.wikibooks.org/wiki/Ada_Programming/Type_System#Derived_types)
>> >>> which in that language is also spelled with "new".
>> >>
>> >> I started to explain this to my non-programmer wife, I got as far as
>> >> explaining types, and that we need a name for this thing, and she
>> >> stopped me and said
>> >>
>> >> "Please don't tell me this is leading to TypyMcTypeCheck."
>> >>
>> >> [...]
>> >>> - BoatyMcBoatType
>> >>
>> >>> The nice thing about "distinguished" is that it's a relatively rare
>> >>> word so it is easy to remember or look up.
>> >>
>> >> I would have thought that being rare, it would be *harder* to remember.
>> >>
>> >>
>> >>> Personally I'm still in favor of Derived type (but I'm more into
>> >>> ancient programming languages than most folks here). I could also live
>> >>> with Distinguished Type.
>> >>
>> >> I think Derived Type is the nicest of the options. It accurately
>> >> describes what it is: a type derived from another. And its shorter and
>> >> easy to both say and write than "Distinguished type" (which sounds like
>> >> "distinguished gentlemen" -- is it wearing a monocle and a top hat?).
>> >>
>> >> "Distinguished" is too vague for my tastes, it might as well be
>> >> "flibblegubble type". *All* types are distinguished, the type checker
>> >> has to distinguish int from float from list from str, so to call
>> >> NewType("userid", int) a "distinguished type" is only to call it a
>> type.
>> >>
>> >>
>> >>
>> >> --
>> >> Steve
>> >> _______________________________________________
>> >> Python-Dev mailing list
>> >> Python-Dev at python.org
>> >> https://mail.python.org/mailman/listinfo/python-dev
>> >> Unsubscribe:
>> >> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>> >
>> >
>> >
>> > --
>> > --Guido van Rossum (python.org/~guido)
>> > _______________________________________________
>> > Python-Dev mailing list
>> > Python-Dev at python.org
>> > https://mail.python.org/mailman/listinfo/python-dev
>> > Unsubscribe:
>> >
>> https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/levkivskyi%40gmail.com
>>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/pludemann%40google.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160528/1acc7d3c/attachment.html>

From stephen at xemacs.org  Sat May 28 14:46:10 2016
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sun, 29 May 2016 03:46:10 +0900
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <CAP7+vJLhR0whVf-Oiwz=9fcSgZgwp8zhZOqZzYq50=VFC_8Yfg@mail.gmail.com>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
 <20160528005826.GJ12028@ando.pearwood.info>
 <5748FB66.7090107@canterbury.ac.nz>
 <CAP7+vJ+CAVuXaRgA=UgKjurYPA3ndzR=+Tp4MeRXEyD6h4MoeQ@mail.gmail.com>
 <20160528122445.GK12028@ando.pearwood.info>
 <CAP7+vJLhR0whVf-Oiwz=9fcSgZgwp8zhZOqZzYq50=VFC_8Yfg@mail.gmail.com>
Message-ID: <22345.59250.962744.337024@turnbull.sk.tsukuba.ac.jp>

Guido van Rossum writes:

 > But seriously I think we should just decide between Derived Type and
 > Distinguished Type [Alias].

I like "typedef", but I'm pretty sure it wouldn't carry the same
connotations to people who aren't C programming oldtimers.

I dislike "derived" because that fits stuff like Union[int, str] or
List[int] *better* than NewType("A", int).

Bottom line: +1 for distinguished.

I may have a little trouble remembering that NewType creates
Distinguished Types (do they get a medal from the Queen, by the way?),
but if you call something a "distinguished type" I will have zero
trouble remembering that it's a new type created out of thin air by
distinguishing it from an existing type.


From mafagafogigante at gmail.com  Sat May 28 14:49:09 2016
From: mafagafogigante at gmail.com (Bernardo Sulzbach)
Date: Sat, 28 May 2016 15:49:09 -0300
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <E1b6g1q-0002er-KN@se2-syd.hostedmail.net.au>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
 <20160528005826.GJ12028@ando.pearwood.info>
 <5748FB66.7090107@canterbury.ac.nz>
 <CAP7+vJ+CAVuXaRgA=UgKjurYPA3ndzR=+Tp4MeRXEyD6h4MoeQ@mail.gmail.com>
 <20160528122445.GK12028@ando.pearwood.info>
 <CAP7+vJLhR0whVf-Oiwz=9fcSgZgwp8zhZOqZzYq50=VFC_8Yfg@mail.gmail.com>
 <E1b6g1q-0002er-KN@se2-syd.hostedmail.net.au>
Message-ID: <281d0886-9202-851e-ac8b-8da5de1ad3f4@gmail.com>

On 05/28/2016 12:19 PM, Steve Dower wrote:
> Did anyone suggest "distinct type alias"?
>

I would just like to mention that "distinguished" seems to be more often 
associated with notability and excellence than "distinct", which is 
usually more neutral towards the quality of what it describes.

Unless we want to say that these types are nobler than their 
counterparts, "distinguished" seems worse than "distinct".


From christian at python.org  Sat May 28 16:44:07 2016
From: christian at python.org (Christian Heimes)
Date: Sat, 28 May 2016 13:44:07 -0700
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <CAMpsgwb3Bc9u702ExNPer8KQQ+wx8JSQ0LcABiB1UKOUA-Hmhw@mail.gmail.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <ADD38FA0-5F5E-4EBE-992F-0A882B00F483@stufft.io>
 <CAMpsgwb3Bc9u702ExNPer8KQQ+wx8JSQ0LcABiB1UKOUA-Hmhw@mail.gmail.com>
Message-ID: <5913a7dc-5c67-6b30-1e34-71a729aa09bd@python.org>

On 2016-05-27 03:44, Victor Stinner wrote:
> Le 27 mai 2016 12:05 PM, "Donald Stufft" <donald at stufft.io
> <mailto:donald at stufft.io>> a ?crit :
>> BLAKE2 is an interesting one, because while SHA3 is a NIST standard
> (so it?s going to gain adoption because of that), BLAKE2 is at least as
> strong as SHA3 but is better in many ways, particularly in speed? it?s
> actually faster than MD5 while being as secure as SHA3.
> 
> BLAKE2 was part of the SHA3 competition and it was in finalists. The
> SHA3 competition is interesting because each algorithm is deeply tested
> and analyzed by many teams all around the world. Obvious vulnerabilities
> are quickly found.

Thanks Victor,

minor correction, BLAKE was a finalist in the SHA3 competition, not
BLAKE2. BLAKE2 is an improved version of BLAKE2 with additional features.

Christian


From christian at python.org  Sat May 28 16:56:48 2016
From: christian at python.org (Christian Heimes)
Date: Sat, 28 May 2016 13:56:48 -0700
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <57482779.6030100@egenix.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com>
Message-ID: <6c634ad5-5004-86a9-6338-8233c0c02b65@python.org>

On 2016-05-27 03:54, M.-A. Lemburg wrote:
> On 27.05.2016 06:54, Raymond Hettinger wrote:
>>
>>> On May 25, 2016, at 3:29 AM, Christian Heimes <christian at python.org> wrote:
>>>
>>> I have three hashing-related patches for Python 3.6 that are waiting for
>>> review. Altogether the three patches add ten new hash algorithms to the
>>> hashlib module: SHA3 (224, 256, 384, 512), SHAKE (SHA3 XOF 128, 256),
>>> BLAKE2 (blake2b, blake2s) and truncated SHA512 (224, 256).
>>
>> Do we really need ten?  I don't think the standard library is the place to offer all variants of hashing.  And we should avoid getting in a cycle of "this was just released by NIST" and "nobody uses that one anymore".  Is any one of them an emergent best practice (i.e. starting to be commonly used in network protocols because it is better, faster, stronger, etc)?
>>
>> Your last message on https://bugs.python.org/issue16113 suggests that these aren't essential and that there is room for debate about whether some of them are standard-library worthy (i.e. we will have them around forever).
> 
> I can understand your eagerness to get this landed, since it's
> been 4 years since work started, but I think we should wait with
> the addition until OpenSSL has them:
> 
> https://github.com/openssl/openssl/issues/439
> 
> The current patch is 1.2MB for SHA-3 - that's pretty heavy for just
> a few hash functions, which aren't in any wide spread use yet and
> probably won't be for quite a few years ahead.

About 1 MB of the 1.2 MB are test vectors for SHA3. Strictly speaking
the test vectors are not required.

> IMO, relying on OpenSSL is a better strategy than providing
> (and maintaining) our own compatibility versions. Until OpenSSL
> has them, people can use Bj?rn's package:
> 
> https://github.com/bjornedstrom/python-sha3
> 
> Perhaps you could join forces with Bj?rn to create a standard
> SHA-3 standalone package on PyPI based on your two variants
> which we could recommend to people in the docs ?!

I have been maintaining my own SHA3 module for couple of years. A month
ago I moved my code to github and ported it to the new Keccak Code
Package. The standalone package uses the same code as my patch but also
provides the old Keccak hashes and works on Python 2.7.

https://github.com/tiran/pysha3
https://pypi.python.org/pypi/pysha3

From christian at python.org  Sat May 28 17:01:35 2016
From: christian at python.org (Christian Heimes)
Date: Sat, 28 May 2016 14:01:35 -0700
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <CALGmxEJYwU7pSL--u9RJo4XFuev32wG4mSo6WgLrRZTFr0tyTQ@mail.gmail.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com> <7067504675934721722@unknownmsgid>
 <57487745.2080900@egenix.com>
 <CALGmxEJYwU7pSL--u9RJo4XFuev32wG4mSo6WgLrRZTFr0tyTQ@mail.gmail.com>
Message-ID: <f82b0a1a-ccf2-2779-f88d-a9893a0b2c92@python.org>

On 2016-05-27 09:41, Chris Barker wrote:
> I'm probably showing my ignorance here, but couldn't we swap in the
> OpenSSL implementation when that becomes available?

No, not any time soon. As soon as we guarantee SHA3 support we have to
keep our own implementation for a couple of additional releases. We can
drop our own SHA3 code as soon as all supported OpenSSL versions have SHA3.

For example when OpenSSL 1.2.0 is going to have SHA3 support, we must
wait until OpenSSL 1.1 and 1.0.2 are no longer supported by OpenSSL.

Christian



From donald at stufft.io  Sat May 28 17:04:55 2016
From: donald at stufft.io (Donald Stufft)
Date: Sat, 28 May 2016 17:04:55 -0400
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <f82b0a1a-ccf2-2779-f88d-a9893a0b2c92@python.org>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com> <7067504675934721722@unknownmsgid>
 <57487745.2080900@egenix.com>
 <CALGmxEJYwU7pSL--u9RJo4XFuev32wG4mSo6WgLrRZTFr0tyTQ@mail.gmail.com>
 <f82b0a1a-ccf2-2779-f88d-a9893a0b2c92@python.org>
Message-ID: <7F576C05-72CC-48D6-82A7-6EF0A94F9B92@stufft.io>


> On May 28, 2016, at 5:01 PM, Christian Heimes <christian at python.org> wrote:
> 
> No, not any time soon. As soon as we guarantee SHA3 support we have to
> keep our own implementation for a couple of additional releases. We can
> drop our own SHA3 code as soon as all supported OpenSSL versions have SHA3.


It still will be needed for as long as it?s possible to build Python without
OpenSSL.

?
Donald Stufft




From brett at python.org  Sat May 28 17:06:49 2016
From: brett at python.org (Brett Cannon)
Date: Sat, 28 May 2016 21:06:49 +0000
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <6c634ad5-5004-86a9-6338-8233c0c02b65@python.org>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com> <57482779.6030100@egenix.com>
 <6c634ad5-5004-86a9-6338-8233c0c02b65@python.org>
Message-ID: <CAP1=2W4dZSQiCV2BwvpgBVDNbfUf9frzUKzWs6Am+-thRru8Eg@mail.gmail.com>

On Sat, May 28, 2016, 13:58 Christian Heimes <christian at python.org> wrote:

> On 2016-05-27 03:54, M.-A. Lemburg wrote:
> > On 27.05.2016 06:54, Raymond Hettinger wrote:
> >>
> >>> On May 25, 2016, at 3:29 AM, Christian Heimes <christian at python.org>
> wrote:
> >>>
> >>> I have three hashing-related patches for Python 3.6 that are waiting
> for
> >>> review. Altogether the three patches add ten new hash algorithms to the
> >>> hashlib module: SHA3 (224, 256, 384, 512), SHAKE (SHA3 XOF 128, 256),
> >>> BLAKE2 (blake2b, blake2s) and truncated SHA512 (224, 256).
> >>
> >> Do we really need ten?  I don't think the standard library is the place
> to offer all variants of hashing.  And we should avoid getting in a cycle
> of "this was just released by NIST" and "nobody uses that one anymore".  Is
> any one of them an emergent best practice (i.e. starting to be commonly
> used in network protocols because it is better, faster, stronger, etc)?
> >>
> >> Your last message on https://bugs.python.org/issue16113 suggests that
> these aren't essential and that there is room for debate about whether some
> of them are standard-library worthy (i.e. we will have them around forever).
> >
> > I can understand your eagerness to get this landed, since it's
> > been 4 years since work started, but I think we should wait with
> > the addition until OpenSSL has them:
> >
> > https://github.com/openssl/openssl/issues/439
> >
> > The current patch is 1.2MB for SHA-3 - that's pretty heavy for just
> > a few hash functions, which aren't in any wide spread use yet and
> > probably won't be for quite a few years ahead.
>
> About 1 MB of the 1.2 MB are test vectors for SHA3. Strictly speaking
> the test vectors are not required.
>

We can always make the test vector file an external download like we do for
some of the codec tests.

-brett



> > IMO, relying on OpenSSL is a better strategy than providing
> > (and maintaining) our own compatibility versions. Until OpenSSL
> > has them, people can use Bj?rn's package:
> >
> > https://github.com/bjornedstrom/python-sha3
> >
> > Perhaps you could join forces with Bj?rn to create a standard
> > SHA-3 standalone package on PyPI based on your two variants
> > which we could recommend to people in the docs ?!
>
> I have been maintaining my own SHA3 module for couple of years. A month
> ago I moved my code to github and ported it to the new Keccak Code
> Package. The standalone package uses the same code as my patch but also
> provides the old Keccak hashes and works on Python 2.7.
>
> https://github.com/tiran/pysha3
> https://pypi.python.org/pypi/pysha3
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160528/8b608382/attachment.html>

From guido at python.org  Sat May 28 17:06:43 2016
From: guido at python.org (Guido van Rossum)
Date: Sat, 28 May 2016 14:06:43 -0700
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <f82b0a1a-ccf2-2779-f88d-a9893a0b2c92@python.org>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com> <57482779.6030100@egenix.com>
 <7067504675934721722@unknownmsgid> <57487745.2080900@egenix.com>
 <CALGmxEJYwU7pSL--u9RJo4XFuev32wG4mSo6WgLrRZTFr0tyTQ@mail.gmail.com>
 <f82b0a1a-ccf2-2779-f88d-a9893a0b2c92@python.org>
Message-ID: <CAP7+vJ+TdT3-63m2CVQ83kG3N4QZp+YStSDoYQCT0uCy9425Yg@mail.gmail.com>

But you could choose which implementation to use at compile time based
on the autoconf output, right?

On Sat, May 28, 2016 at 2:01 PM, Christian Heimes <christian at python.org> wrote:
> On 2016-05-27 09:41, Chris Barker wrote:
>> I'm probably showing my ignorance here, but couldn't we swap in the
>> OpenSSL implementation when that becomes available?
>
> No, not any time soon. As soon as we guarantee SHA3 support we have to
> keep our own implementation for a couple of additional releases. We can
> drop our own SHA3 code as soon as all supported OpenSSL versions have SHA3.
>
> For example when OpenSSL 1.2.0 is going to have SHA3 support, we must
> wait until OpenSSL 1.1 and 1.0.2 are no longer supported by OpenSSL.
>
> Christian
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From christian at python.org  Sat May 28 17:13:28 2016
From: christian at python.org (Christian Heimes)
Date: Sat, 28 May 2016 14:13:28 -0700
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <5748BF06.50606@egenix.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com>
 <CAMpsgwbKvDYBJMmmw42o6n8bY7DWi_huRuXZTBtGH939eGy_Ww@mail.gmail.com>
 <CAO41-mNBLSnZpJBFKqsuW4v4bi4vwVYhzvhbrBsMTEZ0fpkpuA@mail.gmail.com>
 <5748BF06.50606@egenix.com>
Message-ID: <d0eaf723-455d-74e3-b248-bc6c5f507b35@python.org>

On 2016-05-27 14:41, M.-A. Lemburg wrote:
> On 27.05.2016 22:58, Ryan Gonzalez wrote:
>> On May 27, 2016 3:04 PM, "Victor Stinner" <victor.stinner at gmail.com> wrote:
>>>
>>> Le vendredi 27 mai 2016, M.-A. Lemburg <mal at egenix.com> a ?crit :
>>>>
>>>> The current patch is 1.2MB for SHA-3 - that's pretty heavy for just
>>>> a few hash functions, which aren't in any wide spread use yet and
>>>> probably won't be for quite a few years ahead.
>>>
>>>
>>> Oh wow, it's so fat? Why is it so big? Can't we use a lighter version?
>>>
>>
>> The stark majority of the patch is Lib/test/vectors/sha3_224.txt, which
>> seems to be (as the file path implies) just test data. A whopping >1k LOC
>> of really long hashes.
> 
> Right. There's about 1MB test data in the patch, but even
> without that data, the patch adds more than 6400 lines of code.

The KeccakCodePackage is rather large. I already removed all unnecessary
files and modified some files so more code is shared between 32 and
64bit optimized variants. Please keep in mind that the KCP contains
multiple implementations with different optimizations for CPU
architectures. I already removed the ARM NEON optimization.
I also don't get your obsession with lines of code. The gzip and expat
are far bigger than the KeccakCodePackage.


> If we add this now, there should at least be an exit strategy
> to remove the code again, when OpenSSL ships with the same
> code, IMO.
> 
> Aside: BLAKE2 has already landed in OpenSSL 1.1.0:
> 
> https://github.com/openssl/openssl/tree/master/crypto/blake2

Except BLAKE2 in OpenSSL is severely castrated and tailored towards a
very limited use case. The implementation does not support any of the
useful advanced features like keyed hashing (MAC), salt,
personalization, tree hashing and variable hash length.


From donald at stufft.io  Sat May 28 17:18:52 2016
From: donald at stufft.io (Donald Stufft)
Date: Sat, 28 May 2016 17:18:52 -0400
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <CAP7+vJ+TdT3-63m2CVQ83kG3N4QZp+YStSDoYQCT0uCy9425Yg@mail.gmail.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com> <7067504675934721722@unknownmsgid>
 <57487745.2080900@egenix.com>
 <CALGmxEJYwU7pSL--u9RJo4XFuev32wG4mSo6WgLrRZTFr0tyTQ@mail.gmail.com>
 <f82b0a1a-ccf2-2779-f88d-a9893a0b2c92@python.org>
 <CAP7+vJ+TdT3-63m2CVQ83kG3N4QZp+YStSDoYQCT0uCy9425Yg@mail.gmail.com>
Message-ID: <27853AD8-D31F-464F-A3F1-4B94907492AB@stufft.io>


> On May 28, 2016, at 5:06 PM, Guido van Rossum <guido at python.org> wrote:
> 
> But you could choose which implementation to use at compile time based
> on the autoconf output, right?

I think we should follow what hashlib already does. If we want to change the way it works that's fine but these hashes shouldn't be special. They should work the way that all the other standard hashes in hashlib work. 

From christian at python.org  Sat May 28 17:30:37 2016
From: christian at python.org (Christian Heimes)
Date: Sat, 28 May 2016 14:30:37 -0700
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <CAP7+vJ+TdT3-63m2CVQ83kG3N4QZp+YStSDoYQCT0uCy9425Yg@mail.gmail.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com> <7067504675934721722@unknownmsgid>
 <57487745.2080900@egenix.com>
 <CALGmxEJYwU7pSL--u9RJo4XFuev32wG4mSo6WgLrRZTFr0tyTQ@mail.gmail.com>
 <f82b0a1a-ccf2-2779-f88d-a9893a0b2c92@python.org>
 <CAP7+vJ+TdT3-63m2CVQ83kG3N4QZp+YStSDoYQCT0uCy9425Yg@mail.gmail.com>
Message-ID: <71750130-8286-cb64-63e0-6d869145c512@python.org>

On 2016-05-28 14:06, Guido van Rossum wrote:
> But you could choose which implementation to use at compile time based
> on the autoconf output, right?

We compile all modules and then let hashlib decide which implementation
is used. hashlib prefers OpenSSL but falls back to our builtin modules.
For MD5, SHA1 and SHA2 OpenSSL's implementation has better performance
(up to twice the speed).


From christian at python.org  Sat May 28 17:32:40 2016
From: christian at python.org (Christian Heimes)
Date: Sat, 28 May 2016 14:32:40 -0700
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <CAP1=2W4dZSQiCV2BwvpgBVDNbfUf9frzUKzWs6Am+-thRru8Eg@mail.gmail.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com>
 <6c634ad5-5004-86a9-6338-8233c0c02b65@python.org>
 <CAP1=2W4dZSQiCV2BwvpgBVDNbfUf9frzUKzWs6Am+-thRru8Eg@mail.gmail.com>
Message-ID: <a3d34fcd-89cf-f193-c861-011b55859a37@python.org>

On 2016-05-28 14:06, Brett Cannon wrote:
> We can always make the test vector file an external download like we do
> for some of the codec tests.

That is actually a great idea! :)

Thanks Brett


From christian at python.org  Sat May 28 17:56:00 2016
From: christian at python.org (Christian Heimes)
Date: Sat, 28 May 2016 14:56:00 -0700
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <CAPJVwB=y3+=yJTAjTwU6xBzNQ4SEsi1J=H+Wv5QM1iSDmE1H4Q@mail.gmail.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com>
 <CAMpsgwbKvDYBJMmmw42o6n8bY7DWi_huRuXZTBtGH939eGy_Ww@mail.gmail.com>
 <CAO41-mNBLSnZpJBFKqsuW4v4bi4vwVYhzvhbrBsMTEZ0fpkpuA@mail.gmail.com>
 <5748BF06.50606@egenix.com> <074FDA7C-1079-403A-B567-D1709A8E3F8C@stufft.io>
 <5748C566.1070904@egenix.com>
 <CAPJVwB=y3+=yJTAjTwU6xBzNQ4SEsi1J=H+Wv5QM1iSDmE1H4Q@mail.gmail.com>
Message-ID: <ab5de545-2c62-7669-571a-335eca36e6b7@python.org>

On 2016-05-27 15:52, Nathaniel Smith wrote:
> On Fri, May 27, 2016 at 3:08 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> On 27.05.2016 23:46, Donald Stufft wrote:
>>>
>>>> On May 27, 2016, at 5:41 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>>>>
>>>> If we add this now, there should at least be an exit strategy
>>>> to remove the code again, when OpenSSL ships with the same
>>>> code, IMO.
>>>
>>> I think it is a clear win to have the fallback implementations in cases where people either don?t have OpenSSL or don?t have a new enough OpenSSL for those implementations. Not having the fallback just makes it more difficult for people to rely on those hash functions.
>>
>> This will only be needed once the stdlib itself starts requiring
>> support for some of these hashes and for that we could add
>> a pure Python implementation, eg.
>>
>> https://github.com/coruus/py-keccak
>>
>> In all other cases, you can simply add the support via a
>> package such as Bj?rn's or Christian's.
> 
> SHA-3 and BLAKE are extremely widely accepted standards, our users
> will expect them, and they're significant improvements over all the
> current hashes in the algorithms_guaranteed list. If we demote them to
> second-class support (by making them only available in some builds, or
> using a slow pure Python implementation), then we'll be encouraging
> users to use inferior hashes. We shouldn't do this without a very good
> reason, and I don't see anything very convincing here... by all means
> drop the megabyte of test data, but why does it matter how many lines
> of code the algorithm is? No python developer will ever have to look
> at it -- hash code by its nature is *very* low maintenance (it either
> computes the right function or it doesn't, and the right answer never
> changes). And in unlikely case where some terrible unexpected bug is
> discovered then the only maintenance needed will be to delete the
> current impl and drop-in whatever the new fixed one is.
> 
> So +1 to adding SHA-3 and BLAKE to algorithms_guaranteed.

Thanks Nathaniel,

my patches don't add SHA3 and BLAKE2 to algorithms_guaranteed because
Python still supports C89 platforms without a 64 bit integer type.
Theoretically 64bit ints are not required except for BLAKE2b. Since
Trent's snakebite.org is dead I don't have access to these old platforms
any more.

Christian

From chris.jerdonek at gmail.com  Sat May 28 18:59:35 2016
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Sat, 28 May 2016 15:59:35 -0700
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <CAP7+vJ+CAVuXaRgA=UgKjurYPA3ndzR=+Tp4MeRXEyD6h4MoeQ@mail.gmail.com>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
 <20160528005826.GJ12028@ando.pearwood.info>
 <5748FB66.7090107@canterbury.ac.nz>
 <CAP7+vJ+CAVuXaRgA=UgKjurYPA3ndzR=+Tp4MeRXEyD6h4MoeQ@mail.gmail.com>
Message-ID: <CAOTb1wfHjG=nYxUCK8Ng+4YuCHn5+K-URR+J+Revk2UNmzhovw@mail.gmail.com>

On Fri, May 27, 2016 at 9:26 PM, Guido van Rossum <guido at python.org> wrote:
> We discussed this over dinner at PyCon, some ideas we came up with:
>
> - Dependent types, harking back to a similar concept in Ada
> (https://en.wikibooks.org/wiki/Ada_Programming/Type_System#Derived_types)
> which in that language is also spelled with "new".
>
> - New type
>
> - Distinguished Type
>
> - Distinguished Subtype
>
> - Distinguished Type Alias
>
> - Distinguished Alias
>
> - BoatyMcBoatType

Some more suggestions:

- Cloned Type (or Type Clone)

- Copied Type (or Type Copy)

- Named Type

- Renamed Type

- Twin Type

- Wrapped Type

- Doppelganger Type (not serious)

--Chris

From cs at zip.com.au  Sat May 28 19:24:23 2016
From: cs at zip.com.au (cs at zip.com.au)
Date: Sun, 29 May 2016 09:24:23 +1000
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <E1b6g1q-0002er-KN@se2-syd.hostedmail.net.au>
References: <E1b6g1q-0002er-KN@se2-syd.hostedmail.net.au>
Message-ID: <20160528232423.GA78519@cskk.homeip.net>

On 28May2016 08:19, Steve Dower <steve.dower at python.org> wrote:
>Did anyone suggest "distinct type alias"?
>
>Regardless of what name, I'm fairly sure people will call it whatever the 
>function to create it is called. So if the function is 
>typings.distinguish_type(...), then distinguished will stick.

Just casting an opinion in support of Greg Ewing's remark: I don't think we 
should use the word "alias", regardless of what else is used.

In <5748FB66.7090107 at canterbury.ac.nz>, Greg said:

  Steven D'Aprano wrote:
  > TypeAlias? Because A is an alias for int?
  That suggests it's just another name for the same type, but it's not. It's a 
  distinct type as far as the static type checker is concerned

and I agree entirely.

Cheers,
Cameron Simpson <cs at zip.com.au>

From levkivskyi at gmail.com  Sun May 29 01:07:17 2016
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Sun, 29 May 2016 07:07:17 +0200
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <CAOTb1wfHjG=nYxUCK8Ng+4YuCHn5+K-URR+J+Revk2UNmzhovw@mail.gmail.com>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
 <20160528005826.GJ12028@ando.pearwood.info>
 <5748FB66.7090107@canterbury.ac.nz>
 <CAP7+vJ+CAVuXaRgA=UgKjurYPA3ndzR=+Tp4MeRXEyD6h4MoeQ@mail.gmail.com>
 <CAOTb1wfHjG=nYxUCK8Ng+4YuCHn5+K-URR+J+Revk2UNmzhovw@mail.gmail.com>
Message-ID: <CAOMjWknDBKRAR0S2+TyN3TVeptEW0RyT2UXQ2wqJhtRMPdTC4w@mail.gmail.com>

Type clone sounds good. But I am still inclined more towards "distinct".

--
Ivan
29 ??? 2016 01:04 "Chris Jerdonek" <chris.jerdonek at gmail.com> ????:

> On Fri, May 27, 2016 at 9:26 PM, Guido van Rossum <guido at python.org>
> wrote:
> > We discussed this over dinner at PyCon, some ideas we came up with:
> >
> > - Dependent types, harking back to a similar concept in Ada
> > (https://en.wikibooks.org/wiki/Ada_Programming/Type_System#Derived_types
> )
> > which in that language is also spelled with "new".
> >
> > - New type
> >
> > - Distinguished Type
> >
> > - Distinguished Subtype
> >
> > - Distinguished Type Alias
> >
> > - Distinguished Alias
> >
> > - BoatyMcBoatType
>
> Some more suggestions:
>
> - Cloned Type (or Type Clone)
>
> - Copied Type (or Type Copy)
>
> - Named Type
>
> - Renamed Type
>
> - Twin Type
>
> - Wrapped Type
>
> - Doppelganger Type (not serious)
>
> --Chris
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/levkivskyi%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160529/491c9805/attachment.html>

From victor.stinner at gmail.com  Sun May 29 02:51:23 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 29 May 2016 08:51:23 +0200
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <ab5de545-2c62-7669-571a-335eca36e6b7@python.org>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com>
 <CAMpsgwbKvDYBJMmmw42o6n8bY7DWi_huRuXZTBtGH939eGy_Ww@mail.gmail.com>
 <CAO41-mNBLSnZpJBFKqsuW4v4bi4vwVYhzvhbrBsMTEZ0fpkpuA@mail.gmail.com>
 <5748BF06.50606@egenix.com>
 <074FDA7C-1079-403A-B567-D1709A8E3F8C@stufft.io>
 <5748C566.1070904@egenix.com>
 <CAPJVwB=y3+=yJTAjTwU6xBzNQ4SEsi1J=H+Wv5QM1iSDmE1H4Q@mail.gmail.com>
 <ab5de545-2c62-7669-571a-335eca36e6b7@python.org>
Message-ID: <CAMpsgwZJ47osOt597YedkAqXisu3b_4giTBXSMq6GH+_yLf2aw@mail.gmail.com>

Python 3.5 requires a 64 bit signed integer to build. Search for _PyTime
type in pytime.h ;-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160529/a76484e0/attachment.html>

From mal at egenix.com  Sun May 29 10:08:24 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Sun, 29 May 2016 16:08:24 +0200
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <d0eaf723-455d-74e3-b248-bc6c5f507b35@python.org>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com>
 <CAMpsgwbKvDYBJMmmw42o6n8bY7DWi_huRuXZTBtGH939eGy_Ww@mail.gmail.com>
 <CAO41-mNBLSnZpJBFKqsuW4v4bi4vwVYhzvhbrBsMTEZ0fpkpuA@mail.gmail.com>
 <5748BF06.50606@egenix.com> <d0eaf723-455d-74e3-b248-bc6c5f507b35@python.org>
Message-ID: <574AF7D8.30406@egenix.com>

On 28.05.2016 23:13, Christian Heimes wrote:
> On 2016-05-27 14:41, M.-A. Lemburg wrote:
>> On 27.05.2016 22:58, Ryan Gonzalez wrote:
>>> On May 27, 2016 3:04 PM, "Victor Stinner" <victor.stinner at gmail.com> wrote:
>>>>
>>>> Le vendredi 27 mai 2016, M.-A. Lemburg <mal at egenix.com> a ?crit :
>>>>>
>>>>> The current patch is 1.2MB for SHA-3 - that's pretty heavy for just
>>>>> a few hash functions, which aren't in any wide spread use yet and
>>>>> probably won't be for quite a few years ahead.
>>>>
>>>>
>>>> Oh wow, it's so fat? Why is it so big? Can't we use a lighter version?
>>>>
>>>
>>> The stark majority of the patch is Lib/test/vectors/sha3_224.txt, which
>>> seems to be (as the file path implies) just test data. A whopping >1k LOC
>>> of really long hashes.
>>
>> Right. There's about 1MB test data in the patch, but even
>> without that data, the patch adds more than 6400 lines of code.
> 
> The KeccakCodePackage is rather large. I already removed all unnecessary
> files and modified some files so more code is shared between 32 and
> 64bit optimized variants. Please keep in mind that the KCP contains
> multiple implementations with different optimizations for CPU
> architectures. I already removed the ARM NEON optimization.
> I also don't get your obsession with lines of code. The gzip and expat
> are far bigger than the KeccakCodePackage.

For a small piece of code, it's fine to have a copy in the
stdlib, but for larger chunks such as this one, I think we
ought to consider alternative options, since I don't think
it's good to have to carry around this baggage forever.

OpenSSL will eventually have good enough support for what
most Python users will need from these new hash functions.
That's why I think it's better to have a discussion of whether
we need to full package in the stdlib or better only provide
limited support built into the stdlib and refer people to
PyPI packages for things that you don't need every day.

Regarding the stories for zlib and expat, I only remember
that expat was essentially unmaintained when we added it
and the existing version at the time had known bugs (but
could be wrong).

For zlib, I have no clue as to why we have a copy in the stdlib.
That lib is available on all systems nowadays. Perhaps it wasn't
when we added it; don't remember. If so, it's a good example of
why adding copies to the stdlib is not such a good idea :-)

>> If we add this now, there should at least be an exit strategy
>> to remove the code again, when OpenSSL ships with the same
>> code, IMO.
>>
>> Aside: BLAKE2 has already landed in OpenSSL 1.1.0:
>>
>> https://github.com/openssl/openssl/tree/master/crypto/blake2
> 
> Except BLAKE2 in OpenSSL is severely castrated and tailored towards a
> very limited use case. The implementation does not support any of the
> useful advanced features like keyed hashing (MAC), salt,
> personalization, tree hashing and variable hash length.

I bet that the use cases they put into OpenSSL is what
most people will eventually use, so essentially the same
reasoning we use for putting stuff into the stdlib.

Besides, the code just landed in OpenSSL. It's likely they'll
continue to optimize it and possibly also add the variants
they left out initially.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, May 29 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From jcronin at egh.com  Sun May 29 12:57:01 2016
From: jcronin at egh.com (jon)
Date: Sun, 29 May 2016 12:57:01 -0400
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <20160528232423.GA78519@cskk.homeip.net>
References: <E1b6g1q-0002er-KN@se2-syd.hostedmail.net.au>
 <20160528232423.GA78519@cskk.homeip.net>
Message-ID: <1E0AD890-997B-4142-B1F2-E97EC0A91EAC@egh.com>


The aspects we want to capture in a name or adjective for these types are:

a) The types have identical implementations or definitions.
b) They are distinct types.

I think ?Distinguished Type? or?Cloned Type? best captures these qualities.

I think the following also capture the quality, but feel a bit pejorative.

"Segregated Type"
"Arbitrary Type"

Jonathan


From tjreedy at udel.edu  Sun May 29 13:54:39 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 29 May 2016 13:54:39 -0400
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <1E0AD890-997B-4142-B1F2-E97EC0A91EAC@egh.com>
References: <E1b6g1q-0002er-KN@se2-syd.hostedmail.net.au>
 <20160528232423.GA78519@cskk.homeip.net>
 <1E0AD890-997B-4142-B1F2-E97EC0A91EAC@egh.com>
Message-ID: <nifad1$eio$1@ger.gmane.org>

On 5/29/2016 12:57 PM, jon wrote:
>
> The aspects we want to capture in a name or adjective for these types are:
>
> a) The types have identical implementations or definitions.
> b) They are distinct types.
>
> I think ?Distinguished Type? or?Cloned Type? best captures these qualities.

'Cloned Type' conveys the meaning above, "Distinguished Type' does not, 
unless one knows what it us supposed to mean.  One meaning is 'distinct 
because superior (or inferior)'.

> I think the following also capture the quality, but feel a bit pejorative.
>
> "Segregated Type"
> "Arbitrary Type"

Things are segregated because they are thought to be different. 
'Arbitrary' means that it could be anything ;-).


-- 
Terry Jan Reedy



From jelle.zijlstra at gmail.com  Sun May 29 15:07:57 2016
From: jelle.zijlstra at gmail.com (Jelle Zijlstra)
Date: Sun, 29 May 2016 12:07:57 -0700
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
Message-ID: <CAFp3-p9EEdbg_75d+1jq2C7_sptAczhKYP97393Jk+0eAkFoEQ@mail.gmail.com>

2016-05-27 16:01 GMT-07:00 Guido van Rossum <gvanrossum at gmail.com>:

> Also -- the most important thing. :-) What to call these things? We're
> pretty much settled on the semantics and how to create them (A =
> NewType('A', int)) but what should we call types like A when we're
> talking about them? "New types" sounds awkward.
>

For what it's worth, Haskell uses the term "newtype" for a very similar
concept (https://wiki.haskell.org/Newtype), so maybe Python should follow
suit in the interest of not creating new confusing terminology.

"Dependent type" (proposed by a few people) already means something else (
https://en.wikipedia.org/wiki/Dependent_type), so it doesn't seem like a
good choice here.

>
> On Fri, May 27, 2016 at 12:54 PM, Guido van Rossum <gvanrossum at gmail.com>
> wrote:
> > Sorry, this is more meant to be the start of a discussion about the
> proposed
> > feature. And typing.py has its own upstream repo (like asyncio).
> >
> > --Guido (mobile)
> >
> > On May 27, 2016 12:52 PM, "Brett Cannon" <brett at python.org> wrote:
> >>
> >> Patches to Python's stdlib should go through bugs.python.org so it
> isn't
> >> lost in email.
> >>
> >>
> >> On Fri, May 27, 2016, 12:00 Ivan Levkivskyi <levkivskyi at gmail.com>
> wrote:
> >>>
> >>> Hi,
> >>>
> >>> It has been proposed to enhance the typing module with a NewType
> function
> >>> that allows to define simple unique types with almost zero runtime
> >>> overhead.
> >>>
> >>> The PR containing actual implementation and PEP 484 update is here:
> >>> https://github.com/python/typing/pull/226
> >>>
> >>> Review comments are very welcome.
> >>>
> >>> Best regards,
> >>> Ivan
> >>> _______________________________________________
> >>> Python-Dev mailing list
> >>> Python-Dev at python.org
> >>> https://mail.python.org/mailman/listinfo/python-dev
> >>> Unsubscribe:
> >>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
> >>
> >>
> >> _______________________________________________
> >> Python-Dev mailing list
> >> Python-Dev at python.org
> >> https://mail.python.org/mailman/listinfo/python-dev
> >> Unsubscribe:
> >> https://mail.python.org/mailman/options/python-dev/guido%40python.org
> >>
> >
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/jelle.zijlstra%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160529/cb63a38d/attachment.html>

From christian at python.org  Sun May 29 16:21:02 2016
From: christian at python.org (Christian Heimes)
Date: Sun, 29 May 2016 13:21:02 -0700
Subject: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2,
 truncated SHA512
In-Reply-To: <CAMpsgwZJ47osOt597YedkAqXisu3b_4giTBXSMq6GH+_yLf2aw@mail.gmail.com>
References: <52d03a08-8d5e-9751-405d-aeeca740d832@python.org>
 <B1A1E9FF-3E8E-48F9-AF31-B3F2F8854105@gmail.com>
 <57482779.6030100@egenix.com>
 <CAMpsgwbKvDYBJMmmw42o6n8bY7DWi_huRuXZTBtGH939eGy_Ww@mail.gmail.com>
 <CAO41-mNBLSnZpJBFKqsuW4v4bi4vwVYhzvhbrBsMTEZ0fpkpuA@mail.gmail.com>
 <5748BF06.50606@egenix.com> <074FDA7C-1079-403A-B567-D1709A8E3F8C@stufft.io>
 <5748C566.1070904@egenix.com>
 <CAPJVwB=y3+=yJTAjTwU6xBzNQ4SEsi1J=H+Wv5QM1iSDmE1H4Q@mail.gmail.com>
 <ab5de545-2c62-7669-571a-335eca36e6b7@python.org>
 <CAMpsgwZJ47osOt597YedkAqXisu3b_4giTBXSMq6GH+_yLf2aw@mail.gmail.com>
Message-ID: <44005240-8c05-f012-bb87-79d1d0fc18cf@python.org>

On 2016-05-28 23:51, Victor Stinner wrote:
> Python 3.5 requires a 64 bit signed integer to build. Search for _PyTime
> type in pytime.h ;-)

Awesome! Thanks :)


From oscar.j.benjamin at gmail.com  Sun May 29 19:08:42 2016
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Mon, 30 May 2016 00:08:42 +0100
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
Message-ID: <CAHVvXxSPug8i6Q7Rw6k2LgcwRrOTt4CO61LEgXMLB7pEZmk0LA@mail.gmail.com>

On 28 May 2016 00:03, "Guido van Rossum" <gvanrossum at gmail.com> wrote:
>
> Also -- the most important thing. :-) What to call these things? We're
> pretty much settled on the semantics and how to create them (A =
> NewType('A', int)) but what should we call types like A when we're
> talking about them? "New types" sounds awkward.

How about "nominal type" since the type is distinct only in name?

--
Oscar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160530/152afffb/attachment.html>

From gvanrossum at gmail.com  Sun May 29 19:53:02 2016
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Sun, 29 May 2016 16:53:02 -0700
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <CAFp3-p9EEdbg_75d+1jq2C7_sptAczhKYP97393Jk+0eAkFoEQ@mail.gmail.com>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
 <CAFp3-p9EEdbg_75d+1jq2C7_sptAczhKYP97393Jk+0eAkFoEQ@mail.gmail.com>
Message-ID: <CAP7+vJKVMzRfZhJBFjdhQJAcB+YpYrFDPCxwNJGAK9joY0eHXQ@mail.gmail.com>

On Sun, May 29, 2016 at 12:07 PM, Jelle Zijlstra
<jelle.zijlstra at gmail.com> wrote:
>
>
> 2016-05-27 16:01 GMT-07:00 Guido van Rossum <gvanrossum at gmail.com>:
>>
>> Also -- the most important thing. :-) What to call these things? We're
>> pretty much settled on the semantics and how to create them (A =
>> NewType('A', int)) but what should we call types like A when we're
>> talking about them? "New types" sounds awkward.
>
>
> For what it's worth, Haskell uses the term "newtype" for a very similar
> concept (https://wiki.haskell.org/Newtype), so maybe Python should follow
> suit in the interest of not creating new confusing terminology.

Yeah, I started out rejecting this because I don't like such neologisms.

> "Dependent type" (proposed by a few people) already means something else
> (https://en.wikipedia.org/wiki/Dependent_type), so it doesn't seem like a
> good choice here.

I apologize, when I brought that up I meant to say "derived type" (as
should have been clear from the link I provided
(https://en.wikibooks.org/wiki/Ada_Programming/Type_System#Derived_types).
I agree dependent types would be a terrible name here. :-)

I am currently in favor of Distinct Type [Alias].

-- 
--Guido van Rossum (python.org/~guido)

From gvanrossum at gmail.com  Sun May 29 19:55:50 2016
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Sun, 29 May 2016 16:55:50 -0700
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <CAHVvXxSPug8i6Q7Rw6k2LgcwRrOTt4CO61LEgXMLB7pEZmk0LA@mail.gmail.com>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
 <CAHVvXxSPug8i6Q7Rw6k2LgcwRrOTt4CO61LEgXMLB7pEZmk0LA@mail.gmail.com>
Message-ID: <CAP7+vJJk6A+8UiHd42eMDhHzvK_eZ7Tq2-4tHvo=FVQoUwudKw@mail.gmail.com>

On Sun, May 29, 2016 at 4:08 PM, Oscar Benjamin
<oscar.j.benjamin at gmail.com> wrote:
>
> On 28 May 2016 00:03, "Guido van Rossum" <gvanrossum at gmail.com> wrote:
>>
>> Also -- the most important thing. :-) What to call these things? We're
>> pretty much settled on the semantics and how to create them (A =
>> NewType('A', int)) but what should we call types like A when we're
>> talking about them? "New types" sounds awkward.
>
> How about "nominal type" since the type is distinct only in name?

I believe that would be confusing because nominal type systems are a
different idea (https://en.wikipedia.org/wiki/Nominal_type_system).

-- 
--Guido van Rossum (python.org/~guido)

From fred at fdrake.net  Sun May 29 20:12:04 2016
From: fred at fdrake.net (Fred Drake)
Date: Sun, 29 May 2016 17:12:04 -0700
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <CAP7+vJKVMzRfZhJBFjdhQJAcB+YpYrFDPCxwNJGAK9joY0eHXQ@mail.gmail.com>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
 <CAFp3-p9EEdbg_75d+1jq2C7_sptAczhKYP97393Jk+0eAkFoEQ@mail.gmail.com>
 <CAP7+vJKVMzRfZhJBFjdhQJAcB+YpYrFDPCxwNJGAK9joY0eHXQ@mail.gmail.com>
Message-ID: <CAFT4OTEi1K8eJ3XWKK9pByqJ1FeyUgSQg2JLHnUXeJ+B3k8dJg@mail.gmail.com>

On Sun, May 29, 2016 at 4:53 PM, Guido van Rossum <gvanrossum at gmail.com> wrote:
> I am currently in favor of Distinct Type [Alias].

I actually like distinguished type better:

A = typing.distinguish("A", int)


  -Fred

-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein

From steve at pearwood.info  Sun May 29 21:53:59 2016
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 30 May 2016 11:53:59 +1000
Subject: [Python-Dev] Python parser performance optimizations
In-Reply-To: <DB6PR0801MB1400C8D31C17EFC547E8544287410@DB6PR0801MB1400.eurprd08.prod.outlook.com>
References: <DB6PR0801MB1400C8D31C17EFC547E8544287410@DB6PR0801MB1400.eurprd08.prod.outlook.com>
Message-ID: <20160530015359.GL12028@ando.pearwood.info>

On Thu, May 26, 2016 at 10:19:05AM +0000, Artyom Skrobov wrote:
[...]
> The motivation for this patch was to enable a memory footprint 
> optimization, discussed at http://bugs.python.org/issue26415 My 
> proposed optimization reduces the memory footprint by up to 30% on the 
> standard benchmarks, and by 200% on a degenerate case which sparked 
> the discussion. The run time stays unaffected by this optimization.

That can't be right. How can you reduce memory usage by more than one 
hundred percent? That would mean you have saved more memory than was 
originally used and are now using a negative amount of memory.


-- 
Steve

From mafagafogigante at gmail.com  Mon May 30 05:50:46 2016
From: mafagafogigante at gmail.com (Bernardo Sulzbach)
Date: Mon, 30 May 2016 06:50:46 -0300
Subject: [Python-Dev] Python parser performance optimizations
In-Reply-To: <20160530015359.GL12028@ando.pearwood.info>
References: <DB6PR0801MB1400C8D31C17EFC547E8544287410@DB6PR0801MB1400.eurprd08.prod.outlook.com>
 <20160530015359.GL12028@ando.pearwood.info>
Message-ID: <29194dc2-496c-6a8b-1fc7-47637ef15258@gmail.com>

On 05/29/2016 10:53 PM, Steven D'Aprano wrote:
> On Thu, May 26, 2016 at 10:19:05AM +0000, Artyom Skrobov wrote:
> [...]
>> The motivation for this patch was to enable a memory footprint
>> optimization, discussed at http://bugs.python.org/issue26415 My
>> proposed optimization reduces the memory footprint by up to 30% on the
>> standard benchmarks, and by 200% on a degenerate case which sparked
>> the discussion. The run time stays unaffected by this optimization.
>
> That can't be right. How can you reduce memory usage by more than one
> hundred percent? That would mean you have saved more memory than was
> originally used and are now using a negative amount of memory.
>

It is not. It would be nice to have the values that were used to 
calculate these percentages.

From greg.ewing at canterbury.ac.nz  Mon May 30 08:45:46 2016
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 31 May 2016 00:45:46 +1200
Subject: [Python-Dev] Python parser performance optimizations
In-Reply-To: <20160530015359.GL12028@ando.pearwood.info>
References: <DB6PR0801MB1400C8D31C17EFC547E8544287410@DB6PR0801MB1400.eurprd08.prod.outlook.com>
 <20160530015359.GL12028@ando.pearwood.info>
Message-ID: <574C35FA.9060103@canterbury.ac.nz>

Steven D'Aprano wrote:
> That can't be right. How can you reduce memory usage by more than one 
> hundred percent? That would mean you have saved more memory than was 
> originally used and are now using a negative amount of memory.

It emails an order for more RAM to Amazon, who send out
a robot drone to install it in your computer.

-- 
Greg

From v+python at g.nevcal.com  Mon May 30 10:15:02 2016
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Mon, 30 May 2016 07:15:02 -0700
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <1E0AD890-997B-4142-B1F2-E97EC0A91EAC@egh.com>
References: <E1b6g1q-0002er-KN@se2-syd.hostedmail.net.au>
 <20160528232423.GA78519@cskk.homeip.net>
 <1E0AD890-997B-4142-B1F2-E97EC0A91EAC@egh.com>
Message-ID: <c6460e34-5207-c981-37cf-beb5989d6a60@g.nevcal.com>

On 5/29/2016 9:57 AM, jon wrote:
> The aspects we want to capture in a name or adjective for these types are:
>
> a) The types have identical implementations or definitions.
> b) They are distinct types.
>
> I think ?Distinguished Type? or?Cloned Type? best captures these qualities.
>
> I think the following also capture the quality, but feel a bit pejorative.
>
> "Segregated Type"
> "Arbitrary Type"

"Exclusive Type"

"Separate Type"

Glenn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160530/ebffbf95/attachment-0001.html>

From gvanrossum at gmail.com  Mon May 30 12:31:29 2016
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Mon, 30 May 2016 09:31:29 -0700
Subject: [Python-Dev] Python parser performance optimizations
In-Reply-To: <574C35FA.9060103@canterbury.ac.nz>
References: <DB6PR0801MB1400C8D31C17EFC547E8544287410@DB6PR0801MB1400.eurprd08.prod.outlook.com>
 <20160530015359.GL12028@ando.pearwood.info>
 <574C35FA.9060103@canterbury.ac.nz>
Message-ID: <CAP7+vJJO_197LFSgU25Bkv191Lfy2=6wa4zN+nOhUz4r_wmvbA@mail.gmail.com>

I know we're all just having fun, but that's probably a rather stressful
welcome to the list. Maybe we can tone down the humor a bit and instead
review the OP's patches?

--Guido (mobile)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160530/3e761635/attachment.html>

From brett at python.org  Mon May 30 17:59:29 2016
From: brett at python.org (Brett Cannon)
Date: Mon, 30 May 2016 21:59:29 +0000
Subject: [Python-Dev] embedding - PyImport_AppendInittab() after
 Py_Initialize()
In-Reply-To: <CALPW6hy=RT+k+LSm8+HQ_3iv_amnttriwEo94S9eg4kPNAWX3g@mail.gmail.com>
References: <CALPW6hy=RT+k+LSm8+HQ_3iv_amnttriwEo94S9eg4kPNAWX3g@mail.gmail.com>
Message-ID: <CAP1=2W57UP_CgE1ypjkgOktzZBMFXs5bL4rZOtVVAgPSEpoXRg@mail.gmail.com>

On Fri, 27 May 2016 at 07:50 Vitaly Murashev <vitaly.murashev at gmail.com>
wrote:

> Could anyone please clarify whether it is correct in python3.5 to call
> PyImport_AppendInittab() after Py_Initialize()
>

It should be called before as PyImport_AppendInittab() is shorthand for
PyImport_ExtendInittab() and it says to call prior to Py_Initialize():
https://github.com/python/cpython/blob/f680b517e2701e9a3859afb62628a46eccdce17c/Python/import.c#L2153
 .

-Brett


>
> I found this case among the tests for boost.python
> https://github.com/boostorg/python/blob/develop/test/exec.cpp
>
> And this test doesn't work for me with python3.5.An error looks like:
>
> exec.cpp(137): Python Error detected in function 'void __cdecl
> check_pyerr(bool)'
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
> ImportError: 'embedded_hello' is not a built-in module
> 1 error detected.
>
> After debugging I found out that root cause is in
> importlib/_bootstrap.py which do the following:
>
>         if fullname not in sys.builtin_module_names:
>             raise ImportError('{!r} is not a built-in
> module'.format(fullname), name=fullname)
>
> but sys.builtin_module_names is the constant which initialized once in
> Py_Initialize()
>
> ---
> To summarize:
> Is it a bug in python3.5 or original test inside boost.python should be
> fixed ?
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160530/54b50cd9/attachment.html>

From rndblnch at gmail.com  Tue May 31 15:55:53 2016
From: rndblnch at gmail.com (rndblnch)
Date: Tue, 31 May 2016 19:55:53 +0000 (UTC)
Subject: [Python-Dev] Adding NewType() to PEP 484
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
Message-ID: <loom.20160531T214828-498@post.gmane.org>

Guido van Rossum <gvanrossum <at> gmail.com> writes:

> 
> Also -- the most important thing.  What to call these things? We're
> pretty much settled on the semantics and how to create them (A =
> NewType('A', int)) but what should we call types like A when we're
> talking about them? "New types" sounds awkward.

back in high school, i was introduced to c programming with the "disciplined
C" preprocessor [0].
it made the distinction between information type and representation type
(e.g. between the semantic and the implementation).
those new types where created using typedefs and were named 'parallel types'
below is the relevant part of the dcc presentation:

"""
a major innovation of Disciplined C is the notion of "parallel type", that
allows a distinction between information type and representation type. The
following:

typedef int Tindex, Tval;
typedef Tindex Trow, Tcol;

creates four distinct types, but which all accept the same operations and
the same constants as the "representation" type ('int' here). Tindex, Tval,
Trow and Tcol are examples of "information" types, because they convey an
idea of the semantics of the corresponding objects. For example, they may be
put to use in a checkers playing program: Tval will name 'int's that
represent values of checkers, Trow and Tcol, 'int's that represent row and
column indexes, Tindex, generic type for indexes.
Tindex, Tval, Trow and Tcol are called parallel types; in fact, a type T1 is
said to be parallel to a type T2 iff both are defined through a chain of
typedefs starting from the same 'baseType', with no intervening qualifier
nor modifier (pointer/array/function decla- rator, see grammar in Appendix
A). In other words, T and T2 must be strict synonyms of baseType.
"""

renaud

0. Disciplined C

ACM SIGPLAN Notices Homepage archive
Volume 30 Issue 12, Dec. 1995
Pages 43 - 50
http://dl.acm.org/citation.cfm?id=219726.219747
http://www.digiater.nl/openvms/freeware/v50/dcc/dcc-v2_7d/dccarticle.ps


From v+python at g.nevcal.com  Tue May 31 18:02:48 2016
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Tue, 31 May 2016 15:02:48 -0700
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <loom.20160531T214828-498@post.gmane.org>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
 <loom.20160531T214828-498@post.gmane.org>
Message-ID: <38cb015b-2d32-b9a7-d5b7-eef312eb4fa7@g.nevcal.com>

On 5/31/2016 12:55 PM, rndblnch wrote:
> Guido van Rossum <gvanrossum <at> gmail.com> writes:
>
>> Also -- the most important thing.  What to call these things? We're
>> pretty much settled on the semantics and how to create them (A =
>> NewType('A', int)) but what should we call types like A when we're
>> talking about them? "New types" sounds awkward.
> back in high school, i was introduced to c programming with the "disciplined
> C" preprocessor [0].
> it made the distinction between information type and representation type
> (e.g. between the semantic and the implementation).
> those new types where created using typedefs and were named 'parallel types'
> below is the relevant part of the dcc presentation:

Interesting! Prior art. And parallel type isn't a bad name...

>
> """
> a major innovation of Disciplined C is the notion of "parallel type", that
> allows a distinction between information type and representation type. The
> following:
>
> typedef int Tindex, Tval;
> typedef Tindex Trow, Tcol;
>
> creates four distinct types, but which all accept the same operations and
> the same constants as the "representation" type ('int' here). Tindex, Tval,
> Trow and Tcol are examples of "information" types, because they convey an
> idea of the semantics of the corresponding objects. For example, they may be
> put to use in a checkers playing program: Tval will name 'int's that
> represent values of checkers, Trow and Tcol, 'int's that represent row and
> column indexes, Tindex, generic type for indexes.
> Tindex, Tval, Trow and Tcol are called parallel types; in fact, a type T1 is
> said to be parallel to a type T2 iff both are defined through a chain of
> typedefs starting from the same 'baseType', with no intervening qualifier
> nor modifier (pointer/array/function decla- rator, see grammar in Appendix
> A). In other words, T and T2 must be strict synonyms of baseType.
> """
>
> renaud
>
> 0. Disciplined C
>
> ACM SIGPLAN Notices Homepage archive
> Volume 30 Issue 12, Dec. 1995
> Pages 43 - 50
> http://dl.acm.org/citation.cfm?id=219726.219747
> http://www.digiater.nl/openvms/freeware/v50/dcc/dcc-v2_7d/dccarticle.ps
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/v%2Bpython%40g.nevcal.com
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160531/4c478111/attachment.html>

From ncoghlan at gmail.com  Tue May 31 19:58:50 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 1 Jun 2016 09:58:50 +1000
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <38cb015b-2d32-b9a7-d5b7-eef312eb4fa7@g.nevcal.com>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
 <loom.20160531T214828-498@post.gmane.org>
 <38cb015b-2d32-b9a7-d5b7-eef312eb4fa7@g.nevcal.com>
Message-ID: <CADiSq7fnQEZVxkKJ5T-=rU7CPtsE4c7SuyDH+1SnW8KZg-FnOA@mail.gmail.com>

On 31 May 2016 3:12 pm, "Glenn Linderman" <v+python at g.nevcal.com> wrote:
>
> On 5/31/2016 12:55 PM, rndblnch wrote:
>>
>> Guido van Rossum <gvanrossum <at> gmail.com> writes:
>>
>>>
>>> Also -- the most important thing.  What to call these things? We're
>>> pretty much settled on the semantics and how to create them (A =
>>> NewType('A', int)) but what should we call types like A when we're
>>> talking about them? "New types" sounds awkward.
>>
>> back in high school, i was introduced to c programming with the
"disciplined
>> C" preprocessor [0].
>> it made the distinction between information type and representation type
>> (e.g. between the semantic and the implementation).
>> those new types where created using typedefs and were named 'parallel
types'
>> below is the relevant part of the dcc presentation:
>
>
> Interesting! Prior art. And parallel type isn't a bad name...

If I heard "parallel type", I'd assume it had something to do with parallel
processing.

Of the options suggested so far, DistinctType seems the most promising to
me.

Cheers,
Nick.

>
>>
>>
>> """
>> a major innovation of Disciplined C is the notion of "parallel type",
that
>> allows a distinction between information type and representation type.
The
>> following:
>>
>> typedef int Tindex, Tval;
>> typedef Tindex Trow, Tcol;
>>
>> creates four distinct types, but which all accept the same operations and
>> the same constants as the "representation" type ('int' here). Tindex,
Tval,
>> Trow and Tcol are examples of "information" types, because they convey an
>> idea of the semantics of the corresponding objects. For example, they
may be
>> put to use in a checkers playing program: Tval will name 'int's that
>> represent values of checkers, Trow and Tcol, 'int's that represent row
and
>> column indexes, Tindex, generic type for indexes.
>> Tindex, Tval, Trow and Tcol are called parallel types; in fact, a type
T1 is
>> said to be parallel to a type T2 iff both are defined through a chain of
>> typedefs starting from the same 'baseType', with no intervening qualifier
>> nor modifier (pointer/array/function decla- rator, see grammar in
Appendix
>> A). In other words, T and T2 must be strict synonyms of baseType.
>> """
>>
>> renaud
>>
>> 0. Disciplined C
>>
>> ACM SIGPLAN Notices Homepage archive
>> Volume 30 Issue 12, Dec. 1995
>> Pages 43 - 50
>> http://dl.acm.org/citation.cfm?id=219726.219747
>> http://www.digiater.nl/openvms/freeware/v50/dcc/dcc-v2_7d/dccarticle.ps
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/v%2Bpython%40g.nevcal.com
>>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160601/6468b935/attachment.html>

From v+python at g.nevcal.com  Tue May 31 20:46:38 2016
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Tue, 31 May 2016 17:46:38 -0700
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <CADiSq7fnQEZVxkKJ5T-=rU7CPtsE4c7SuyDH+1SnW8KZg-FnOA@mail.gmail.com>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
 <loom.20160531T214828-498@post.gmane.org>
 <38cb015b-2d32-b9a7-d5b7-eef312eb4fa7@g.nevcal.com>
 <CADiSq7fnQEZVxkKJ5T-=rU7CPtsE4c7SuyDH+1SnW8KZg-FnOA@mail.gmail.com>
Message-ID: <ef472cb4-ecf1-5a7a-d1b4-7cc8410e94e1@g.nevcal.com>

On 5/31/2016 4:58 PM, Nick Coghlan wrote:
>
>
> On 31 May 2016 3:12 pm, "Glenn Linderman" <v+python at g.nevcal.com 
> <mailto:v%2Bpython at g.nevcal.com>> wrote:
> >
> > On 5/31/2016 12:55 PM, rndblnch wrote:
> >>
> >> Guido van Rossum <gvanrossum <at> gmail.com <http://gmail.com>> writes:
> >>
> >>>
> >>> Also -- the most important thing.  What to call these things? We're
> >>> pretty much settled on the semantics and how to create them (A =
> >>> NewType('A', int)) but what should we call types like A when we're
> >>> talking about them? "New types" sounds awkward.
> >>
> >> back in high school, i was introduced to c programming with the 
> "disciplined
> >> C" preprocessor [0].
> >> it made the distinction between information type and representation 
> type
> >> (e.g. between the semantic and the implementation).
> >> those new types where created using typedefs and were named 
> 'parallel types'
> >> below is the relevant part of the dcc presentation:
> >
> >
> > Interesting! Prior art. And parallel type isn't a bad name...
>
> If I heard "parallel type", I'd assume it had something to do with 
> parallel processing.
>

Google seems to think so also... "Disciplined C" probably predates 
parallel processing, but adding disciplined C preprocessor to the search 
does find a document about the preprocessor in the same sense used by 
@rndblnch. And the types do have "parallel" behavior... every behavior 
of one is found identically in the other. So it makes sense, but it 
would be hard to find the prior art without extra information, as the 
term is now overwhelmed by "Parallel Type-checking" for parallel processing.

Still, some mention should probably be made of "parallel type in the 
'Disciplined C preprocessor' " in the "prior art" category of PEP 484.

 From reading about Disciplined C, would seem that this feature could be 
used as a foundation to add "units" support to variables, validated only 
by a type checker, and thus having no runtime overhead, although it 
would take some more features from Disciplined C, regarding the 
definition of legal combinations of types and operations.

> Of the options suggested so far, DistinctType seems the most promising 
> to me.
>
> Cheers,
> Nick.
>
> >
> >>
> >>
> >> """
> >> a major innovation of Disciplined C is the notion of "parallel 
> type", that
> >> allows a distinction between information type and representation 
> type. The
> >> following:
> >>
> >> typedef int Tindex, Tval;
> >> typedef Tindex Trow, Tcol;
> >>
> >> creates four distinct types, but which all accept the same 
> operations and
> >> the same constants as the "representation" type ('int' here). 
> Tindex, Tval,
> >> Trow and Tcol are examples of "information" types, because they 
> convey an
> >> idea of the semantics of the corresponding objects. For example, 
> they may be
> >> put to use in a checkers playing program: Tval will name 'int's that
> >> represent values of checkers, Trow and Tcol, 'int's that represent 
> row and
> >> column indexes, Tindex, generic type for indexes.
> >> Tindex, Tval, Trow and Tcol are called parallel types; in fact, a 
> type T1 is
> >> said to be parallel to a type T2 iff both are defined through a 
> chain of
> >> typedefs starting from the same 'baseType', with no intervening 
> qualifier
> >> nor modifier (pointer/array/function decla- rator, see grammar in 
> Appendix
> >> A). In other words, T and T2 must be strict synonyms of baseType.
> >> """
> >>
> >> renaud
> >>
> >> 0. Disciplined C
> >>
> >> ACM SIGPLAN Notices Homepage archive
> >> Volume 30 Issue 12, Dec. 1995
> >> Pages 43 - 50
> >> http://dl.acm.org/citation.cfm?id=219726.219747
> >> http://www.digiater.nl/openvms/freeware/v50/dcc/dcc-v2_7d/dccarticle.ps
> >>
> >> _______________________________________________
> >> Python-Dev mailing list
> >> Python-Dev at python.org <mailto:Python-Dev at python.org>
> >> https://mail.python.org/mailman/listinfo/python-dev
> >> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/v%2Bpython%40g.nevcal.com
> >>
> >
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org <mailto:Python-Dev at python.org>
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
> >
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160531/52f0b7a5/attachment.html>

From mafagafogigante at gmail.com  Tue May 31 21:32:48 2016
From: mafagafogigante at gmail.com (Bernardo Sulzbach)
Date: Tue, 31 May 2016 22:32:48 -0300
Subject: [Python-Dev] Adding NewType() to PEP 484
In-Reply-To: <CADiSq7fnQEZVxkKJ5T-=rU7CPtsE4c7SuyDH+1SnW8KZg-FnOA@mail.gmail.com>
References: <CAOMjWkk=gyj+Zp-F1PygoqqA9ZOoc=1cWw-G5J9scGFFEND4Jw@mail.gmail.com>
 <CAP1=2W7jNH6kS9Of5=VSov_314HoZkjZhbtMtzv-Y4aiM3jaNA@mail.gmail.com>
 <CAP7+vJLxiGg-=TdumxLCyrZbUzp2fdEL8TAhVCx9it0H8PZHMg@mail.gmail.com>
 <CAP7+vJJBsY34wx+peUdxLDUmLVMpE-+ws=uSZbYvdUzAaCp=NA@mail.gmail.com>
 <loom.20160531T214828-498@post.gmane.org>
 <38cb015b-2d32-b9a7-d5b7-eef312eb4fa7@g.nevcal.com>
 <CADiSq7fnQEZVxkKJ5T-=rU7CPtsE4c7SuyDH+1SnW8KZg-FnOA@mail.gmail.com>
Message-ID: <614d2422-ce06-9b0c-672d-3102bbd90ddc@gmail.com>

On 05/31/2016 08:58 PM, Nick Coghlan wrote:
> On 31 May 2016 3:12 pm, "Glenn Linderman" <v+python at g.nevcal.com> wrote:
>>
>> On 5/31/2016 12:55 PM, rndblnch wrote:
>>>
>>> Guido van Rossum <gvanrossum <at> gmail.com> writes:
>>>
>>>>
>>>> Also -- the most important thing.  What to call these things? We're
>>>> pretty much settled on the semantics and how to create them (A =
>>>> NewType('A', int)) but what should we call types like A when we're
>>>> talking about them? "New types" sounds awkward.
>>>
>>> back in high school, i was introduced to c programming with the
> "disciplined
>>> C" preprocessor [0].
>>> it made the distinction between information type and representation type
>>> (e.g. between the semantic and the implementation).
>>> those new types where created using typedefs and were named 'parallel
> types'
>>> below is the relevant part of the dcc presentation:
>>
>>
>> Interesting! Prior art. And parallel type isn't a bad name...
>
> If I heard "parallel type", I'd assume it had something to do with parallel
> processing.
>

This also happens to me every time I read it.