From steve at pearwood.info  Sun Nov  1 01:06:30 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 1 Nov 2015 17:06:30 +1100
Subject: [Python-Dev] If you shadow a module in the standard library
 that IDLE depends on, bad things happen
In-Reply-To: <n0ugob$mvt$1@ger.gmane.org>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org>
Message-ID: <20151101060629.GE10946@ando.pearwood.info>

CC'ing Python-Ideas. Follow-ups to Python-Ideas please.

On Thu, Oct 29, 2015 at 09:22:15PM -0400, Terry Reedy wrote:

> Leaving IDLE aside, the reason '' is added to sys.path is so that people 
> can import their own modules.  This is very useful.  Shadowing is the 
> result of putting it at the front.  I have long thought this a dubious 
> choice.  If '' were instead appended, people could still import modules 
> that did not duplicate stdlib names.  Anyone who wanted shadowing could 
> move '' to the front.  But then shadowing would be intentional, not an 
> accident.

Terry is right. Shadowing should be possible, and it should require a 
deliberate decision on the part of the programmer.

Consider the shell, say, bash or similar. My understanding is that the 
shell PATH deliberately excludes the current directory because of the 
possibility of malicious software shadowing usual commands in /bin etc. 
If you want to run an executable in the current directory, you have to 
explicitly provide the path to it: ./myscript rather than just myscript.

Now Python isn't exactly the shell, and so I'm not proposing that Python 
does the same thing. But surely we can agree on the following?

- Shadowing explicitly installed packages, including the stdlib, is 
  *occasionally* useful.

- But when shadowing occurs, it is *nearly always* accidental.

- Such accidental shadowing often causes problems.

- And further more, debugging shadowing problems is sometimes tricky 
  even for experienced coders, and almost impossible for beginners.

  (It's not until you've been burned once or thrice by shadowing that 
  you recognise the symptoms, at which point it is then usually easy to 
  debug.)

- Hence, we should put the onus on those who want to shadow installed 
  packages) to do so *explicitly*, or at least make it easier to avoid 
  accidental shadowing.


I propose the following two changes:


(1) Beginning with Python 3.6, the default is that the current directory 
is put at the end of sys.path rather than the beginning. Instead of:

    >>> print(sys.path)
    ['', '/this', '/that', '/another']

we will have this instead:

    >>> print(sys.path)
    ['/this', '/that', '/another', '']

Those who don't shadow installed packages won't notice any 
difference.

Scripts which deliberately or unintentionally shadow installed packages 
will break from this change. I don't have a problem with this. You can't 
fix harmful behaviour without breaking code that depends on that harmful 
behaviour. Additionally, I expect that those who rely on the current 
behaviour will be in a small minority, much fewer than those who will be 
bitten by accidental shadowing into the indefinite future. And if you 
want the old behaviour back, it is easy to do so, by changing the path 
before doing your imports:

    import sys
    if sys.path[-1] == "":  sys.path = [""] + sys.path[:-1]

or equivalent.

I do not belive that it is onerous for those who want shadowing to have 
to take steps to do so explicitly. That can be added to your scripts on 
a case-by-case basis, or your PYTHONSTARTUP file, by modifying your 
site.py, or (I think) by putting the code into the sitecustomize or 
usercustomize modules.

(2) IDLE doesn't need to wait for Python 3.6 to make this change. I 
believe that IDLE is permitted to make backwards incompatible changes in 
minor releases, so there is no reason why it can't change the path 
effective immediately.

That's a simpler fix than scanning the entire path, raising warnings 
(which beginners won't understand and will either ignore or panic over) 
or other complex solutions. It may not prevent *every* shadowing 
incident, but it will improve the situation immeasurably.


Thoughts?



-- 
Steve

From storchaka at gmail.com  Sun Nov  1 01:29:32 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 1 Nov 2015 08:29:32 +0200
Subject: [Python-Dev] Unable to submit a patch to the tracker
Message-ID: <n14bgd$7ov$1@ger.gmane.org>

I'm unable to submit any file to any issue, neither via web-form nor via 
e-mail. Checked with different browsers from different computers. 
Meta-tracker doesn't work too.

http://psf.upfronthosting.co.za/roundup/meta/issue575


From lac at openend.se  Sun Nov  1 02:40:36 2015
From: lac at openend.se (Laura Creighton)
Date: Sun, 01 Nov 2015 08:40:36 +0100
Subject: [Python-Dev] Unable to submit a patch to the tracker
In-Reply-To: <n14bgd$7ov$1@ger.gmane.org>
References: <n14bgd$7ov$1@ger.gmane.org>
Message-ID: <201511010740.tA17easo024848@fido.openend.se>

In a message of Sun, 01 Nov 2015 08:29:32 +0200, Serhiy Storchaka writes:
>I'm unable to submit any file to any issue, neither via web-form nor via 
>e-mail. Checked with different browsers from different computers. 
>Meta-tracker doesn't work too.
>
>http://psf.upfronthosting.co.za/roundup/meta/issue575

It's working for me.

Laura

From rosuav at gmail.com  Sun Nov  1 05:30:40 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 1 Nov 2015 21:30:40 +1100
Subject: [Python-Dev] Python 3.5.1 plans
Message-ID: <CAPTjJmqR8vzxfU7CBUXVZM4R9c2k-16bQXTFNQ+gB-biMwso7A@mail.gmail.com>

PEP 478 [1] doesn't currently have any info on a planned 3.5.1 release
(and actually, it has 3.5.0 Final listed as a future release). About
when is it likely to happen? The one thing I'm hanging out for is an
installer patch on Windows that detects XP and immediately aborts with
a convenient error; if the number of emails to python-list is
indicative, there are a lot of people out there getting confused.

ChrisA

[1] https://www.python.org/dev/peps/pep-0478/

From phil at riverbankcomputing.com  Sun Nov  1 05:55:43 2015
From: phil at riverbankcomputing.com (Phil Thompson)
Date: Sun, 1 Nov 2015 10:55:43 +0000
Subject: [Python-Dev] Python 3.5.1 plans
In-Reply-To: <CAPTjJmqR8vzxfU7CBUXVZM4R9c2k-16bQXTFNQ+gB-biMwso7A@mail.gmail.com>
References: <CAPTjJmqR8vzxfU7CBUXVZM4R9c2k-16bQXTFNQ+gB-biMwso7A@mail.gmail.com>
Message-ID: <B78990F4-4073-4318-9C6D-6A2376F38A28@riverbankcomputing.com>

On 1 Nov 2015, at 10:30 a.m., Chris Angelico <rosuav at gmail.com> wrote:
> 
> PEP 478 [1] doesn't currently have any info on a planned 3.5.1 release
> (and actually, it has 3.5.0 Final listed as a future release). About
> when is it likely to happen? The one thing I'm hanging out for is an
> installer patch on Windows that detects XP and immediately aborts with
> a convenient error; if the number of emails to python-list is
> indicative, there are a lot of people out there getting confused.

That doesn't need a new version of Python, just a new installer.

Phil

From rosuav at gmail.com  Sun Nov  1 06:01:54 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 1 Nov 2015 22:01:54 +1100
Subject: [Python-Dev] Python 3.5.1 plans
In-Reply-To: <B78990F4-4073-4318-9C6D-6A2376F38A28@riverbankcomputing.com>
References: <CAPTjJmqR8vzxfU7CBUXVZM4R9c2k-16bQXTFNQ+gB-biMwso7A@mail.gmail.com>
 <B78990F4-4073-4318-9C6D-6A2376F38A28@riverbankcomputing.com>
Message-ID: <CAPTjJmqv=OXU3R+hyYhjzeU+qkxTGfnQxUnDpQZ2D7zpxEgMcA@mail.gmail.com>

On Sun, Nov 1, 2015 at 9:55 PM, Phil Thompson
<phil at riverbankcomputing.com> wrote:
> On 1 Nov 2015, at 10:30 a.m., Chris Angelico <rosuav at gmail.com> wrote:
>>
>> PEP 478 [1] doesn't currently have any info on a planned 3.5.1 release
>> (and actually, it has 3.5.0 Final listed as a future release). About
>> when is it likely to happen? The one thing I'm hanging out for is an
>> installer patch on Windows that detects XP and immediately aborts with
>> a convenient error; if the number of emails to python-list is
>> indicative, there are a lot of people out there getting confused.
>
> That doesn't need a new version of Python, just a new installer.

True, it doesn't, but it needs that installer to have as much checking
as a new point release will get. My understanding was that it was
going to be done as part of 3.5.1.

ChrisA

From steve.dower at python.org  Sun Nov  1 08:52:40 2015
From: steve.dower at python.org (Steve Dower)
Date: Sun, 1 Nov 2015 05:52:40 -0800
Subject: [Python-Dev] Python 3.5.1 plans
In-Reply-To: <CAPTjJmqv=OXU3R+hyYhjzeU+qkxTGfnQxUnDpQZ2D7zpxEgMcA@mail.gmail.com>
References: <CAPTjJmqR8vzxfU7CBUXVZM4R9c2k-16bQXTFNQ+gB-biMwso7A@mail.gmail.com>
 <B78990F4-4073-4318-9C6D-6A2376F38A28@riverbankcomputing.com>
 <CAPTjJmqv=OXU3R+hyYhjzeU+qkxTGfnQxUnDpQZ2D7zpxEgMcA@mail.gmail.com>
Message-ID: <E1Zst4s-0002B1-Qi@se2-syd.hostedmail.net.au>

The installer and the contained contents are currently tied together, making it fairly difficult to mix and match versions.

When 3.5.1 happens is up to Larry, but I'm feeling like the initial rush of bug reports has died down and we should be considering the remaining open ones in the context of blocking that release. (The XP notification, as well as the more confusing Vista and Win7 w/o service packs notifications, have been added already. Python-list will probably keep asking though, just in case it can get something for free...)

Cheers,
Steve

Top-posted from my Windows Phone

-----Original Message-----
From: "Chris Angelico" <rosuav at gmail.com>
Sent: ?11/?1/?2015 3:03
To: "Phil Thompson" <phil at riverbankcomputing.com>
Cc: "python-dev" <python-dev at python.org>
Subject: Re: [Python-Dev] Python 3.5.1 plans

On Sun, Nov 1, 2015 at 9:55 PM, Phil Thompson
<phil at riverbankcomputing.com> wrote:
> On 1 Nov 2015, at 10:30 a.m., Chris Angelico <rosuav at gmail.com> wrote:
>>
>> PEP 478 [1] doesn't currently have any info on a planned 3.5.1 release
>> (and actually, it has 3.5.0 Final listed as a future release). About
>> when is it likely to happen? The one thing I'm hanging out for is an
>> installer patch on Windows that detects XP and immediately aborts with
>> a convenient error; if the number of emails to python-list is
>> indicative, there are a lot of people out there getting confused.
>
> That doesn't need a new version of Python, just a new installer.

True, it doesn't, but it needs that installer to have as much checking
as a new point release will get. My understanding was that it was
going to be done as part of 3.5.1.

ChrisA
_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151101/949dee36/attachment-0001.html>

From lac at openend.se  Sun Nov  1 09:21:12 2015
From: lac at openend.se (Laura Creighton)
Date: Sun, 01 Nov 2015 15:21:12 +0100
Subject: [Python-Dev] Python 3.5.1 plans
In-Reply-To: <E1Zst4s-0002B1-Qi@se2-syd.hostedmail.net.au>
References: <CAPTjJmqR8vzxfU7CBUXVZM4R9c2k-16bQXTFNQ+gB-biMwso7A@mail.gmail.com>
 <B78990F4-4073-4318-9C6D-6A2376F38A28@riverbankcomputing.com>
 <CAPTjJmqv=OXU3R+hyYhjzeU+qkxTGfnQxUnDpQZ2D7zpxEgMcA@mail.gmail.com>
 <E1Zst4s-0002B1-Qi@se2-syd.hostedmail.net.au>
Message-ID: <201511011421.tA1ELCTq021979@fido.openend.se>

In a message of Sun, 01 Nov 2015 05:52:40 -0800, Steve Dower writes:
>The installer and the contained contents are currently tied together, making it fairly difficult to mix and match versions.
>
>When 3.5.1 happens is up to Larry, but I'm feeling like the initial rush of bug reports has died down and we should be considering the remaining open ones in the context of blocking that release. (The XP notification, as well as the more confusing Vista and Win7 w/o service packs notifications, have been added already. Python-list will probably keep asking though, just in case it can get something for free...)
>
>Cheers,
>Steve

Webmaster here.
The initial rush of bug reports I see has not gone down.  It's been in
steady increase.  This is not a data-blip due to last weekend's whole
outage of mail.python.org.

Getting the 'XP -- you are out of luck' message 

and the api-ms-win-crt-runtime-l1-1-0-dll is Microsofts Universal 
CRT.  You don't have one.  You need to install it. Get it here:
http://www.microsoft.com/en-us/download/details.aspx?id=48234

message out of a failed install -- as soon as possible -- would
have a serious positive impact around here.

Most of the replies I make are of that sort.  I have 2 templates to
make it quick to do, but still, be nice if I could save my time for 
others.  And I don't report the number of people I talk to with this
problem to the bug tracker.  This could possibly have mislead you.

It is not like the:
The 3.5 Windows installer fails with "The TARGETDIR variable must be provided when invoking this installer" bug
https://bugs.python.org/issue25144

where I report when I hear about them if I think there is a good chance
that having me report it will cause the person who submitted the bug
to me to go track it -- because maybe asking him or her to give you
more information about their system will help you resolve it.

For the first 2 sorts, well, I won't spam you with the knowledge
that windows xp users are complaining to webmaster a whole lot about
3.5 not working.  

But from my end -- sooner python.org serves up the improved installer,
the better.

Even if you made a release-candidate-right-now to do nothing but that.

Laura

From storchaka at gmail.com  Sun Nov  1 10:50:51 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 1 Nov 2015 17:50:51 +0200
Subject: [Python-Dev] Unable to submit a patch to the tracker
In-Reply-To: <n14bgd$7ov$1@ger.gmane.org>
References: <n14bgd$7ov$1@ger.gmane.org>
Message-ID: <n15ccr$e7j$1@ger.gmane.org>

On 01.11.15 08:29, Serhiy Storchaka wrote:
> I'm unable to submit any file to any issue, neither via web-form nor via
> e-mail. Checked with different browsers from different computers.
> Meta-tracker doesn't work too.
>
> http://psf.upfronthosting.co.za/roundup/meta/issue575

Sorry for the noise. The cause was in my mistake.


From steve.dower at python.org  Sun Nov  1 11:23:48 2015
From: steve.dower at python.org (Steve Dower)
Date: Sun, 1 Nov 2015 08:23:48 -0800
Subject: [Python-Dev] Python 3.5.1 plans
In-Reply-To: <201511011421.tA1ELCTq021979@fido.openend.se>
References: <CAPTjJmqR8vzxfU7CBUXVZM4R9c2k-16bQXTFNQ+gB-biMwso7A@mail.gmail.com>
 <B78990F4-4073-4318-9C6D-6A2376F38A28@riverbankcomputing.com>
 <CAPTjJmqv=OXU3R+hyYhjzeU+qkxTGfnQxUnDpQZ2D7zpxEgMcA@mail.gmail.com>
 <E1Zst4s-0002B1-Qi@se2-syd.hostedmail.net.au>
 <201511011421.tA1ELCTq021979@fido.openend.se>
Message-ID: <E1ZsvR7-0005l6-Ms@se2-syd.hostedmail.net.au>

"The initial rush of bug reports I see has not gone down.  It's been in steady increase."

To clarify, I meant unique bugs. When my job is to fix them, more reports of the same bug are not important to me, especially once they've been fixed but not released.

I do understand that someone has to deal with every report, new or not (for my work project, that's me), and you are appreciated for it especially since it's not your role. You should probably just have one template redirecting people to python-list though.

Top-posted from my Windows Phone

-----Original Message-----
From: "Laura Creighton" <lac at openend.se>
Sent: ?11/?1/?2015 6:22
To: "Steve Dower" <steve.dower at python.org>
Cc: "Chris Angelico" <rosuav at gmail.com>; "Phil Thompson" <phil at riverbankcomputing.com>; "python-dev" <python-dev at python.org>; "lac at openend.se" <lac at openend.se>
Subject: Re: [Python-Dev] Python 3.5.1 plans

In a message of Sun, 01 Nov 2015 05:52:40 -0800, Steve Dower writes:
>The installer and the contained contents are currently tied together, making it fairly difficult to mix and match versions.
>
>When 3.5.1 happens is up to Larry, but I'm feeling like the initial rush of bug reports has died down and we should be considering the remaining open ones in the context of blocking that release. (The XP notification, as well as the more confusing Vista and Win7 w/o service packs notifications, have been added already. Python-list will probably keep asking though, just in case it can get something for free...)
>
>Cheers,
>Steve

Webmaster here.
The initial rush of bug reports I see has not gone down.  It's been in
steady increase.  This is not a data-blip due to last weekend's whole
outage of mail.python.org.

Getting the 'XP -- you are out of luck' message 

and the api-ms-win-crt-runtime-l1-1-0-dll is Microsofts Universal 
CRT.  You don't have one.  You need to install it. Get it here:
http://www.microsoft.com/en-us/download/details.aspx?id=48234

message out of a failed install -- as soon as possible -- would
have a serious positive impact around here.

Most of the replies I make are of that sort.  I have 2 templates to
make it quick to do, but still, be nice if I could save my time for 
others.  And I don't report the number of people I talk to with this
problem to the bug tracker.  This could possibly have mislead you.

It is not like the:
The 3.5 Windows installer fails with "The TARGETDIR variable must be provided when invoking this installer" bug
https://bugs.python.org/issue25144

where I report when I hear about them if I think there is a good chance
that having me report it will cause the person who submitted the bug
to me to go track it -- because maybe asking him or her to give you
more information about their system will help you resolve it.

For the first 2 sorts, well, I won't spam you with the knowledge
that windows xp users are complaining to webmaster a whole lot about
3.5 not working.  

But from my end -- sooner python.org serves up the improved installer,
the better.

Even if you made a release-candidate-right-now to do nothing but that.

Laura
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151101/11b6260a/attachment.html>

From lac at openend.se  Sun Nov  1 12:10:30 2015
From: lac at openend.se (Laura Creighton)
Date: Sun, 01 Nov 2015 18:10:30 +0100
Subject: [Python-Dev] Python 3.5.1 plans
In-Reply-To: <E1ZsvR7-0005l6-Ms@se2-syd.hostedmail.net.au>
References: <CAPTjJmqR8vzxfU7CBUXVZM4R9c2k-16bQXTFNQ+gB-biMwso7A@mail.gmail.com>
 <B78990F4-4073-4318-9C6D-6A2376F38A28@riverbankcomputing.com>
 <CAPTjJmqv=OXU3R+hyYhjzeU+qkxTGfnQxUnDpQZ2D7zpxEgMcA@mail.gmail.com>
 <E1Zst4s-0002B1-Qi@se2-syd.hostedmail.net.au>
 <201511011421.tA1ELCTq021979@fido.openend.se>
 <E1ZsvR7-0005l6-Ms@se2-syd.hostedmail.net.au>
Message-ID: <201511011710.tA1HAULq002365@fido.openend.se>

In a message of Sun, 01 Nov 2015 08:23:48 -0800, Steve Dower writes:
>"The initial rush of bug reports I see has not gone down.  It's been in steady increase."
>
>To clarify, I meant unique bugs. When my job is to fix them, more reports of the same bug are not important to me, especially once they've been fixed but not released.
>
>I do understand that someone has to deal with every report, new or not (for my work project, that's me), and you are appreciated for it especially since it's not your role. You should probably just have one template redirecting people to python-list though.
>
>Top-posted from my Windows Phone

sending them to python-list is not a good idea.
people get bored at the same old bugs and by now
bored python-listers will reply something sarcastic
at people with the same old bugs.

Especialy if they post there not saying what windows version
they have.

There is only so much --'be kinder, it may be the 35th time for us,
it is still that person's first bug report with the same old same old'
I can do.

The chief reason I am subscribed to python-list is so that newbies
are not greeted with such sarcasm alone.

A better experience from the download page should not wait for
Larry's other concerns.  If the new one is ready to go, and tells
XP users to get a new OS and universal-crt missers to go get one here


Put that on python.org as soon as possible.
even if you need to bump the python 3.5 numbering.
you get 3.5.1 for this, and this alone and everybody else gets to
wait for 3.5.2 and when Larry feels this is a good idea.

Laura

From larry at hastings.org  Sun Nov  1 13:51:05 2015
From: larry at hastings.org (Larry Hastings)
Date: Sun, 1 Nov 2015 10:51:05 -0800
Subject: [Python-Dev] Python 3.5.1 plans
In-Reply-To: <201511011710.tA1HAULq002365@fido.openend.se>
References: <CAPTjJmqR8vzxfU7CBUXVZM4R9c2k-16bQXTFNQ+gB-biMwso7A@mail.gmail.com>
 <B78990F4-4073-4318-9C6D-6A2376F38A28@riverbankcomputing.com>
 <CAPTjJmqv=OXU3R+hyYhjzeU+qkxTGfnQxUnDpQZ2D7zpxEgMcA@mail.gmail.com>
 <E1Zst4s-0002B1-Qi@se2-syd.hostedmail.net.au>
 <201511011421.tA1ELCTq021979@fido.openend.se>
 <E1ZsvR7-0005l6-Ms@se2-syd.hostedmail.net.au>
 <201511011710.tA1HAULq002365@fido.openend.se>
Message-ID: <56365F19.1060008@hastings.org>



On 11/01/2015 09:10 AM, Laura Creighton wrote:
> Put that on python.org as soon as possible.
> even if you need to bump the python 3.5 numbering.
> you get 3.5.1 for this, and this alone and everybody else gets to
> wait for 3.5.2 and when Larry feels this is a good idea.

Well, let me say this.  I haven't been added to any apocalyptic 3.5 bugs 
recently.  If Steve says he's not seeing novel bugs, and Laura says 
she's seeing a steady trickle of people reporting the same bugs that 
have already been fixed, then it seems like a good time to cut 3.5.1.

3.5 does have a single issue on the tracker marked as a "release blocker":

    smtplib.py AUTH LOGIN code messed up sending login and password data
    since 3.5

    http://bugs.python.org/issue25446

R. David Murray set it to "release blocker", and I trust his judgment, 
so 3.5.1 can't go out until this is fixed.  I'll engage with him on the 
issue to see about when we can get a fix.

If I didn't have any release blockers, I'd schedule 3.5.1 for sometime 
around three weeks from now.  And FYI I want to release 3.4.4 about two 
weeks after that.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151101/e29532f2/attachment.html>

From lac at openend.se  Sun Nov  1 16:28:52 2015
From: lac at openend.se (Laura Creighton)
Date: Sun, 01 Nov 2015 22:28:52 +0100
Subject: [Python-Dev] Python 3.5.1 plans
In-Reply-To: <56365F19.1060008@hastings.org>
References: <CAPTjJmqR8vzxfU7CBUXVZM4R9c2k-16bQXTFNQ+gB-biMwso7A@mail.gmail.com>
 <B78990F4-4073-4318-9C6D-6A2376F38A28@riverbankcomputing.com>
 <CAPTjJmqv=OXU3R+hyYhjzeU+qkxTGfnQxUnDpQZ2D7zpxEgMcA@mail.gmail.com>
 <E1Zst4s-0002B1-Qi@se2-syd.hostedmail.net.au>
 <201511011421.tA1ELCTq021979@fido.openend.se>
 <E1ZsvR7-0005l6-Ms@se2-syd.hostedmail.net.au>
 <201511011710.tA1HAULq002365@fido.openend.se> <56365F19.1060008@hastings.org>
Message-ID: <201511012128.tA1LSqD0021706@fido.openend.se>

In a message of Sun, 01 Nov 2015 10:51:05 -0800, Larry Hastings writes:
>
>
>On 11/01/2015 09:10 AM, Laura Creighton wrote:
>> Put that on python.org as soon as possible.
>> even if you need to bump the python 3.5 numbering.
>> you get 3.5.1 for this, and this alone and everybody else gets to
>> wait for 3.5.2 and when Larry feels this is a good idea.
>
>Well, let me say this.  I haven't been added to any apocalyptic 3.5 bugs 
>recently.  If Steve says he's not seeing novel bugs, and Laura says 
>she's seeing a steady trickle of people reporting the same bugs that 
>have already been fixed, then it seems like a good time to cut 3.5.1.
>
>3.5 does have a single issue on the tracker marked as a "release blocker":
>
>    smtplib.py AUTH LOGIN code messed up sending login and password data
>    since 3.5
>
>    http://bugs.python.org/issue25446
>
>R. David Murray set it to "release blocker", and I trust his judgment, 
>so 3.5.1 can't go out until this is fixed.  I'll engage with him on the 
>issue to see about when we can get a fix.
>
>If I didn't have any release blockers, I'd schedule 3.5.1 for sometime 
>around three weeks from now.  And FYI I want to release 3.4.4 about two 
>weeks after that.
>
>
>//arry/

Sounds wonderful to me.  Thank you Larry.

Laura


From rosuav at gmail.com  Sun Nov  1 16:46:43 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 2 Nov 2015 08:46:43 +1100
Subject: [Python-Dev] Python 3.5.1 plans
In-Reply-To: <56365F19.1060008@hastings.org>
References: <CAPTjJmqR8vzxfU7CBUXVZM4R9c2k-16bQXTFNQ+gB-biMwso7A@mail.gmail.com>
 <B78990F4-4073-4318-9C6D-6A2376F38A28@riverbankcomputing.com>
 <CAPTjJmqv=OXU3R+hyYhjzeU+qkxTGfnQxUnDpQZ2D7zpxEgMcA@mail.gmail.com>
 <E1Zst4s-0002B1-Qi@se2-syd.hostedmail.net.au>
 <201511011421.tA1ELCTq021979@fido.openend.se>
 <E1ZsvR7-0005l6-Ms@se2-syd.hostedmail.net.au>
 <201511011710.tA1HAULq002365@fido.openend.se>
 <56365F19.1060008@hastings.org>
Message-ID: <CAPTjJmq+nqBgoR4niM1CrDvCHR2tTORofO=zPxbWcCZ8UxRY6w@mail.gmail.com>

On Mon, Nov 2, 2015 at 5:51 AM, Larry Hastings <larry at hastings.org> wrote:
> If I didn't have any release blockers, I'd schedule 3.5.1 for sometime
> around three weeks from now.  And FYI I want to release 3.4.4 about two
> weeks after that.
>

Thanks Larry! That'd be awesome. If there's testing needed on that
ticket, or anything, I'll help out.

ChrisA

From meadori at gmail.com  Sun Nov  1 17:00:49 2015
From: meadori at gmail.com (Meador Inge)
Date: Sun, 1 Nov 2015 16:00:49 -0600
Subject: [Python-Dev] Generated Bytecode ...
In-Reply-To: <CAMpsgwYpA2rr5RM6KHa1atM5f0--t72Rfz_3YLvzqvBRng=+hA@mail.gmail.com>
References: <AE9467DB-9895-4D36-AB08-FB4E2360904E@wirtel.be>
 <CAP1=2W7c=74MmMzcRocn+iaBLJfT9yHRwf+LXFUmJDoORF7ZJA@mail.gmail.com>
 <CAMpsgwYpA2rr5RM6KHa1atM5f0--t72Rfz_3YLvzqvBRng=+hA@mail.gmail.com>
Message-ID: <CAK1QoorU0VF9GXFZzZ0xOGpVAkkTQVtAt37N=vpZvFKKSosSdw@mail.gmail.com>

On Fri, Oct 23, 2015 at 3:23 AM, Victor Stinner
<victor.stinner at gmail.com> wrote:

> Note: I propose "noopt" because we already have "optimization level 0"
> which still uses optimizations, it's the default mode. It's different
> than gcc -O0 which really disables all optimizations. I already prefix
> the "noopt" suffix for .pyc files than "opt--1" proposed by Brett in
> http://bugs.python.org/issue2506

Strictly speaking, this isn't completely true.  Most (all?) C/C++
compilers do some level of optimization when they are "disabled".
GCC/Clang, for example, both will do some form of constant folding
with -O0.  The compilers are just *much* less aggressive with the
primary goal of making the debugging experience much more enjoyable.

The situation here sounds similar -- we want an option to provide a
better debugging and tracing experience, which seems like a very
reasonable idea to me.

-- Meador

From benjamin at python.org  Sun Nov  1 17:42:56 2015
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 01 Nov 2015 14:42:56 -0800
Subject: [Python-Dev] 2.7.11 plans
Message-ID: <1446417776.2942029.426137689.4B30F1D4@webmail.messagingengine.com>

Happy November, everyone. It?s nearly time for the next semi-annual
instalment of the 2.7 series. I?m planning to release a 2.7.11 release
candidate on November 21st and 2.7.11 final on December 5.

More than half-done releasing 2.7-ly yours,
Benjamin

From rymg19 at gmail.com  Sun Nov  1 18:26:27 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Sun, 1 Nov 2015 17:26:27 -0600
Subject: [Python-Dev] 2.7.11 plans
In-Reply-To: <1446417776.2942029.426137689.4B30F1D4@webmail.messagingengine.com>
References: <1446417776.2942029.426137689.4B30F1D4@webmail.messagingengine.com>
Message-ID: <CAO41-mOg=1d-rgs9ZTv4ZY1yrPnjoZXByQBP0g3h30Wtycgb-w@mail.gmail.com>

On Sun, Nov 1, 2015 at 4:42 PM, Benjamin Peterson <benjamin at python.org>
wrote:

> Happy November, everyone. It?s nearly time for the next semi-annual
> instalment of the 2.7 series. I?m planning to release a 2.7.11 release
> candidate on November 21st and 2.7.11 final on December 5.
>
> More than half-done releasing 2.7-ly yours,
>

This made me laugh!



> Benjamin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
>



-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151101/d8619249/attachment.html>

From victor.stinner at gmail.com  Wed Nov  4 03:50:33 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 4 Nov 2015 09:50:33 +0100
Subject: [Python-Dev] Second milestone of FAT Python
Message-ID: <CAMpsgwagj1Jz=Zue_hNYLtsUg7oJEKvo0hB73A3w-XxH1iJi8Q@mail.gmail.com>

Hi,

I'm writing a new "FAT Python" project to try to implement optimizations in
CPython (inlining, constant folding, move invariants out of loops, etc.)
using a "static" optimizer (not a JIT). For the background, see the thread
on python-ideas:
https://mail.python.org/pipermail/python-ideas/2015-October/036908.html

See also the documentation:
https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst
https://hg.python.org/sandbox/fatpython/file/tip/ASTOPTIMIZER.rst

I implemented the most basic optimization to test my code: replace calls to
builtin functions (with constant arguments) with the result. For example,
len("abc") is replaced with 3. I reached the second milestone: it's now
possible to run the full Python test suite with these optimizations
enabled. It confirms that the optimizations don't break the Python semantic.

Example:
---
>>> def func():
...     return len("abc")
...
>>> import dis
>>> dis.dis(func)
  2           0 LOAD_GLOBAL              0 (len)
              3 LOAD_CONST               1 ('abc')
              6 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
              9 RETURN_VALUE

>>> len(func.get_specialized())
1
>>> specialized=func.get_specialized()[0]
>>> dis.dis(specialized['code'])
  2           0 LOAD_CONST               1 (3)
              3 RETURN_VALUE
>>> len(specialized['guards'])
2

>>> func()
3

>>> len=lambda obj: "mock"
>>> func()
'mock'
>>> func.get_specialized()
[]
---

The function func() has specialized bytecode which returns directly 3
instead of calling len("abc"). The specialized bytecode has two guards
dictionary keys: builtins.__dict__['len'] and globals()['len']. If one of
these keys is modified, the specialized bytecode is simply removed (when
the function is called) and the original bytecode is executed.


You cannot expect any speedup at this milestone, it's just to validate the
implementation. You can only get speedup if you implement *manually*
optimizations. See for example posixpath.isabs() which inlines manually the
call to the _get_sep() function. More optimizations will be implemented in
the third milestone. I don't know yet if I will be able to implement
constant folding, function inlining and/or moving invariants out of loops.


Download, compile and test FAT Python with:

    hg clone http://hg.python.org/sandbox/fatpython
    ./configure && make && ./python -m test test_astoptimizer test_fat


Currently, only 24 functions are specialized in the standard library.
Calling a builtin function with constant arguments in not common (it was
expected, it's only the first step for my optimizer). But 161 functions are
specialized in tests.


To be honest, I had to modify some tests to make them pass in FAT mode. But
most changes are related to the .pyc filename, or to the exact size in
bytes of dictionary objects.

FAT Python is still experimental. Currently, the main bug is that the AST
optimizer can optimize a call to a function which is not the expected
builtin function. I already started to implement code to understand
namespaces (detect global and local variables), but it's not enough yet to
detect when a builtin is overriden. See TODO.rst for known bugs and
limitations.

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151104/557b1384/attachment.html>

From fijall at gmail.com  Wed Nov  4 04:07:41 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 4 Nov 2015 09:07:41 +0000
Subject: [Python-Dev] Second milestone of FAT Python
In-Reply-To: <CAMpsgwagj1Jz=Zue_hNYLtsUg7oJEKvo0hB73A3w-XxH1iJi8Q@mail.gmail.com>
References: <CAMpsgwagj1Jz=Zue_hNYLtsUg7oJEKvo0hB73A3w-XxH1iJi8Q@mail.gmail.com>
Message-ID: <CAK5idxTgj9RMLgoeaaZUq6DM38TpJ-2GVka8DD2aBrj=oG1kPg@mail.gmail.com>

How do you check that someone did not e.g. bind something different to "len"?

On Wed, Nov 4, 2015 at 8:50 AM, Victor Stinner <victor.stinner at gmail.com> wrote:
> Hi,
>
> I'm writing a new "FAT Python" project to try to implement optimizations in
> CPython (inlining, constant folding, move invariants out of loops, etc.)
> using a "static" optimizer (not a JIT). For the background, see the thread
> on python-ideas:
> https://mail.python.org/pipermail/python-ideas/2015-October/036908.html
>
> See also the documentation:
> https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst
> https://hg.python.org/sandbox/fatpython/file/tip/ASTOPTIMIZER.rst
>
> I implemented the most basic optimization to test my code: replace calls to
> builtin functions (with constant arguments) with the result. For example,
> len("abc") is replaced with 3. I reached the second milestone: it's now
> possible to run the full Python test suite with these optimizations enabled.
> It confirms that the optimizations don't break the Python semantic.
>
> Example:
> ---
>>>> def func():
> ...     return len("abc")
> ...
>>>> import dis
>>>> dis.dis(func)
>   2           0 LOAD_GLOBAL              0 (len)
>               3 LOAD_CONST               1 ('abc')
>               6 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
>               9 RETURN_VALUE
>
>>>> len(func.get_specialized())
> 1
>>>> specialized=func.get_specialized()[0]
>>>> dis.dis(specialized['code'])
>   2           0 LOAD_CONST               1 (3)
>               3 RETURN_VALUE
>>>> len(specialized['guards'])
> 2
>
>>>> func()
> 3
>
>>>> len=lambda obj: "mock"
>>>> func()
> 'mock'
>>>> func.get_specialized()
> []
> ---
>
> The function func() has specialized bytecode which returns directly 3
> instead of calling len("abc"). The specialized bytecode has two guards
> dictionary keys: builtins.__dict__['len'] and globals()['len']. If one of
> these keys is modified, the specialized bytecode is simply removed (when the
> function is called) and the original bytecode is executed.
>
>
> You cannot expect any speedup at this milestone, it's just to validate the
> implementation. You can only get speedup if you implement *manually*
> optimizations. See for example posixpath.isabs() which inlines manually the
> call to the _get_sep() function. More optimizations will be implemented in
> the third milestone. I don't know yet if I will be able to implement
> constant folding, function inlining and/or moving invariants out of loops.
>
>
> Download, compile and test FAT Python with:
>
>     hg clone http://hg.python.org/sandbox/fatpython
>     ./configure && make && ./python -m test test_astoptimizer test_fat
>
>
> Currently, only 24 functions are specialized in the standard library.
> Calling a builtin function with constant arguments in not common (it was
> expected, it's only the first step for my optimizer). But 161 functions are
> specialized in tests.
>
>
> To be honest, I had to modify some tests to make them pass in FAT mode. But
> most changes are related to the .pyc filename, or to the exact size in bytes
> of dictionary objects.
>
> FAT Python is still experimental. Currently, the main bug is that the AST
> optimizer can optimize a call to a function which is not the expected
> builtin function. I already started to implement code to understand
> namespaces (detect global and local variables), but it's not enough yet to
> detect when a builtin is overriden. See TODO.rst for known bugs and
> limitations.
>
> Victor
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>

From fijall at gmail.com  Wed Nov  4 04:08:17 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 4 Nov 2015 09:08:17 +0000
Subject: [Python-Dev] Second milestone of FAT Python
In-Reply-To: <CAK5idxTgj9RMLgoeaaZUq6DM38TpJ-2GVka8DD2aBrj=oG1kPg@mail.gmail.com>
References: <CAMpsgwagj1Jz=Zue_hNYLtsUg7oJEKvo0hB73A3w-XxH1iJi8Q@mail.gmail.com>
 <CAK5idxTgj9RMLgoeaaZUq6DM38TpJ-2GVka8DD2aBrj=oG1kPg@mail.gmail.com>
Message-ID: <CAK5idxR+UMN98m0ipsr1ejGbk_VJ62pHf7DXJETr8mkwc6CMuQ@mail.gmail.com>

Uh, sorry, misread your full mail, scratch that

On Wed, Nov 4, 2015 at 9:07 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> How do you check that someone did not e.g. bind something different to "len"?
>
> On Wed, Nov 4, 2015 at 8:50 AM, Victor Stinner <victor.stinner at gmail.com> wrote:
>> Hi,
>>
>> I'm writing a new "FAT Python" project to try to implement optimizations in
>> CPython (inlining, constant folding, move invariants out of loops, etc.)
>> using a "static" optimizer (not a JIT). For the background, see the thread
>> on python-ideas:
>> https://mail.python.org/pipermail/python-ideas/2015-October/036908.html
>>
>> See also the documentation:
>> https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst
>> https://hg.python.org/sandbox/fatpython/file/tip/ASTOPTIMIZER.rst
>>
>> I implemented the most basic optimization to test my code: replace calls to
>> builtin functions (with constant arguments) with the result. For example,
>> len("abc") is replaced with 3. I reached the second milestone: it's now
>> possible to run the full Python test suite with these optimizations enabled.
>> It confirms that the optimizations don't break the Python semantic.
>>
>> Example:
>> ---
>>>>> def func():
>> ...     return len("abc")
>> ...
>>>>> import dis
>>>>> dis.dis(func)
>>   2           0 LOAD_GLOBAL              0 (len)
>>               3 LOAD_CONST               1 ('abc')
>>               6 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
>>               9 RETURN_VALUE
>>
>>>>> len(func.get_specialized())
>> 1
>>>>> specialized=func.get_specialized()[0]
>>>>> dis.dis(specialized['code'])
>>   2           0 LOAD_CONST               1 (3)
>>               3 RETURN_VALUE
>>>>> len(specialized['guards'])
>> 2
>>
>>>>> func()
>> 3
>>
>>>>> len=lambda obj: "mock"
>>>>> func()
>> 'mock'
>>>>> func.get_specialized()
>> []
>> ---
>>
>> The function func() has specialized bytecode which returns directly 3
>> instead of calling len("abc"). The specialized bytecode has two guards
>> dictionary keys: builtins.__dict__['len'] and globals()['len']. If one of
>> these keys is modified, the specialized bytecode is simply removed (when the
>> function is called) and the original bytecode is executed.
>>
>>
>> You cannot expect any speedup at this milestone, it's just to validate the
>> implementation. You can only get speedup if you implement *manually*
>> optimizations. See for example posixpath.isabs() which inlines manually the
>> call to the _get_sep() function. More optimizations will be implemented in
>> the third milestone. I don't know yet if I will be able to implement
>> constant folding, function inlining and/or moving invariants out of loops.
>>
>>
>> Download, compile and test FAT Python with:
>>
>>     hg clone http://hg.python.org/sandbox/fatpython
>>     ./configure && make && ./python -m test test_astoptimizer test_fat
>>
>>
>> Currently, only 24 functions are specialized in the standard library.
>> Calling a builtin function with constant arguments in not common (it was
>> expected, it's only the first step for my optimizer). But 161 functions are
>> specialized in tests.
>>
>>
>> To be honest, I had to modify some tests to make them pass in FAT mode. But
>> most changes are related to the .pyc filename, or to the exact size in bytes
>> of dictionary objects.
>>
>> FAT Python is still experimental. Currently, the main bug is that the AST
>> optimizer can optimize a call to a function which is not the expected
>> builtin function. I already started to implement code to understand
>> namespaces (detect global and local variables), but it's not enough yet to
>> detect when a builtin is overriden. See TODO.rst for known bugs and
>> limitations.
>>
>> Victor
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>>

From steve at pearwood.info  Wed Nov  4 05:16:15 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 4 Nov 2015 21:16:15 +1100
Subject: [Python-Dev] Second milestone of FAT Python
In-Reply-To: <CAMpsgwagj1Jz=Zue_hNYLtsUg7oJEKvo0hB73A3w-XxH1iJi8Q@mail.gmail.com>
References: <CAMpsgwagj1Jz=Zue_hNYLtsUg7oJEKvo0hB73A3w-XxH1iJi8Q@mail.gmail.com>
Message-ID: <20151104101615.GO10946@ando.pearwood.info>

On Wed, Nov 04, 2015 at 09:50:33AM +0100, Victor Stinner wrote:
> Hi,
> 
> I'm writing a new "FAT Python" project to try to implement optimizations in
> CPython (inlining, constant folding, move invariants out of loops, etc.)
> using a "static" optimizer (not a JIT). For the background, see the thread
> on python-ideas:
> https://mail.python.org/pipermail/python-ideas/2015-October/036908.html

Nice work Victor, and thank you for your efforts!



-- 
Steve

From srkunze at mail.de  Wed Nov  4 15:14:37 2015
From: srkunze at mail.de (Sven R. Kunze)
Date: Wed, 4 Nov 2015 21:14:37 +0100
Subject: [Python-Dev] Second milestone of FAT Python
In-Reply-To: <CAMpsgwagj1Jz=Zue_hNYLtsUg7oJEKvo0hB73A3w-XxH1iJi8Q@mail.gmail.com>
References: <CAMpsgwagj1Jz=Zue_hNYLtsUg7oJEKvo0hB73A3w-XxH1iJi8Q@mail.gmail.com>
Message-ID: <563A672D.6080803@mail.de>

Hi Victor,

great to hear. I think everybody here appreciates your efforts.

Do you think there will be any change of merging this back into CPython?

Best,
Sven

On 04.11.2015 09:50, Victor Stinner wrote:
> Hi,
>
> I'm writing a new "FAT Python" project to try to implement 
> optimizations in CPython (inlining, constant folding, move invariants 
> out of loops, etc.) using a "static" optimizer (not a JIT). For the 
> background, see the thread on python-ideas:
> https://mail.python.org/pipermail/python-ideas/2015-October/036908.html
>
> See also the documentation:
> https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst
> https://hg.python.org/sandbox/fatpython/file/tip/ASTOPTIMIZER.rst
>
> I implemented the most basic optimization to test my code: replace 
> calls to builtin functions (with constant arguments) with the result. 
> For example, len("abc") is replaced with 3. I reached the second 
> milestone: it's now possible to run the full Python test suite with 
> these optimizations enabled. It confirms that the optimizations don't 
> break the Python semantic.
>
> Example:
> ---
> >>> def func():
> ...     return len("abc")
> ...
> >>> import dis
> >>> dis.dis(func)
>   2           0 LOAD_GLOBAL              0 (len)
>               3 LOAD_CONST               1 ('abc')
>               6 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
>               9 RETURN_VALUE
>
> >>> len(func.get_specialized())
> 1
> >>> specialized=func.get_specialized()[0]
> >>> dis.dis(specialized['code'])
>   2           0 LOAD_CONST               1 (3)
>               3 RETURN_VALUE
> >>> len(specialized['guards'])
> 2
>
> >>> func()
> 3
>
> >>> len=lambda obj: "mock"
> >>> func()
> 'mock'
> >>> func.get_specialized()
> []
> ---
>
> The function func() has specialized bytecode which returns directly 3 
> instead of calling len("abc"). The specialized bytecode has two guards 
> dictionary keys: builtins.__dict__['len'] and globals()['len']. If one 
> of these keys is modified, the specialized bytecode is simply removed 
> (when the function is called) and the original bytecode is executed.
>
>
> You cannot expect any speedup at this milestone, it's just to validate 
> the implementation. You can only get speedup if you implement 
> *manually* optimizations. See for example posixpath.isabs() which 
> inlines manually the call to the _get_sep() function. More 
> optimizations will be implemented in the third milestone. I don't know 
> yet if I will be able to implement constant folding, function inlining 
> and/or moving invariants out of loops.
>
>
> Download, compile and test FAT Python with:
>
>     hg clone http://hg.python.org/sandbox/fatpython
>     ./configure && make && ./python -m test test_astoptimizer test_fat
>
>
> Currently, only 24 functions are specialized in the standard library. 
> Calling a builtin function with constant arguments in not common (it 
> was expected, it's only the first step for my optimizer). But 161 
> functions are specialized in tests.
>
>
> To be honest, I had to modify some tests to make them pass in FAT 
> mode. But most changes are related to the .pyc filename, or to the 
> exact size in bytes of dictionary objects.
>
> FAT Python is still experimental. Currently, the main bug is that the 
> AST optimizer can optimize a call to a function which is not the 
> expected builtin function. I already started to implement code to 
> understand namespaces (detect global and local variables), but it's 
> not enough yet to detect when a builtin is overriden. See TODO.rst for 
> known bugs and limitations.
>
> Victor
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151104/3210f0ec/attachment.html>

From srkunze at mail.de  Wed Nov  4 15:34:24 2015
From: srkunze at mail.de (Sven R. Kunze)
Date: Wed, 4 Nov 2015 21:34:24 +0100
Subject: [Python-Dev] Second milestone of FAT Python
In-Reply-To: <563A672D.6080803@mail.de>
References: <CAMpsgwagj1Jz=Zue_hNYLtsUg7oJEKvo0hB73A3w-XxH1iJi8Q@mail.gmail.com>
 <563A672D.6080803@mail.de>
Message-ID: <563A6BD0.805@mail.de>

typo: "chance" instead of "change"

On 04.11.2015 21:14, Sven R. Kunze wrote:
> Hi Victor,
>
> great to hear. I think everybody here appreciates your efforts.
>
> Do you think there will be any change of merging this back into CPython?
>
> Best,
> Sven
>
> On 04.11.2015 09:50, Victor Stinner wrote:
>> Hi,
>>
>> I'm writing a new "FAT Python" project to try to implement 
>> optimizations in CPython (inlining, constant folding, move invariants 
>> out of loops, etc.) using a "static" optimizer (not a JIT). For the 
>> background, see the thread on python-ideas:
>> https://mail.python.org/pipermail/python-ideas/2015-October/036908.html
>>
>> See also the documentation:
>> https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst
>> https://hg.python.org/sandbox/fatpython/file/tip/ASTOPTIMIZER.rst
>>
>> I implemented the most basic optimization to test my code: replace 
>> calls to builtin functions (with constant arguments) with the result. 
>> For example, len("abc") is replaced with 3. I reached the second 
>> milestone: it's now possible to run the full Python test suite with 
>> these optimizations enabled. It confirms that the optimizations don't 
>> break the Python semantic.
>>
>> Example:
>> ---
>> >>> def func():
>> ...     return len("abc")
>> ...
>> >>> import dis
>> >>> dis.dis(func)
>>   2           0 LOAD_GLOBAL              0 (len)
>>               3 LOAD_CONST               1 ('abc')
>>               6 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
>>               9 RETURN_VALUE
>>
>> >>> len(func.get_specialized())
>> 1
>> >>> specialized=func.get_specialized()[0]
>> >>> dis.dis(specialized['code'])
>>   2           0 LOAD_CONST               1 (3)
>>               3 RETURN_VALUE
>> >>> len(specialized['guards'])
>> 2
>>
>> >>> func()
>> 3
>>
>> >>> len=lambda obj: "mock"
>> >>> func()
>> 'mock'
>> >>> func.get_specialized()
>> []
>> ---
>>
>> The function func() has specialized bytecode which returns directly 3 
>> instead of calling len("abc"). The specialized bytecode has two 
>> guards dictionary keys: builtins.__dict__['len'] and 
>> globals()['len']. If one of these keys is modified, the specialized 
>> bytecode is simply removed (when the function is called) and the 
>> original bytecode is executed.
>>
>>
>> You cannot expect any speedup at this milestone, it's just to 
>> validate the implementation. You can only get speedup if you 
>> implement *manually* optimizations. See for example posixpath.isabs() 
>> which inlines manually the call to the _get_sep() function. More 
>> optimizations will be implemented in the third milestone. I don't 
>> know yet if I will be able to implement constant folding, function 
>> inlining and/or moving invariants out of loops.
>>
>>
>> Download, compile and test FAT Python with:
>>
>>     hg clone http://hg.python.org/sandbox/fatpython
>>     ./configure && make && ./python -m test test_astoptimizer test_fat
>>
>>
>> Currently, only 24 functions are specialized in the standard library. 
>> Calling a builtin function with constant arguments in not common (it 
>> was expected, it's only the first step for my optimizer). But 161 
>> functions are specialized in tests.
>>
>>
>> To be honest, I had to modify some tests to make them pass in FAT 
>> mode. But most changes are related to the .pyc filename, or to the 
>> exact size in bytes of dictionary objects.
>>
>> FAT Python is still experimental. Currently, the main bug is that the 
>> AST optimizer can optimize a call to a function which is not the 
>> expected builtin function. I already started to implement code to 
>> understand namespaces (detect global and local variables), but it's 
>> not enough yet to detect when a builtin is overriden. See TODO.rst 
>> for known bugs and limitations.
>>
>> Victor
>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151104/4ab4c0c6/attachment.html>

From tjreedy at udel.edu  Wed Nov  4 17:20:49 2015
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 4 Nov 2015 17:20:49 -0500
Subject: [Python-Dev] Second milestone of FAT Python
In-Reply-To: <CAMpsgwagj1Jz=Zue_hNYLtsUg7oJEKvo0hB73A3w-XxH1iJi8Q@mail.gmail.com>
References: <CAMpsgwagj1Jz=Zue_hNYLtsUg7oJEKvo0hB73A3w-XxH1iJi8Q@mail.gmail.com>
Message-ID: <n1e0c3$jj4$1@ger.gmane.org>

On 11/4/2015 3:50 AM, Victor Stinner wrote:
> Hi,
>
> I'm writing a new "FAT Python" project to try to implement optimizations
> in CPython (inlining, constant folding, move invariants out of loops,
> etc.) using a "static" optimizer (not a JIT). For the background, see
> the thread on python-ideas:
> https://mail.python.org/pipermail/python-ideas/2015-October/036908.html
>
> See also the documentation:
> https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst
> https://hg.python.org/sandbox/fatpython/file/tip/ASTOPTIMIZER.rst
>
> I implemented the most basic optimization to test my code: replace calls
> to builtin functions (with constant arguments) with the result. For
> example, len("abc") is replaced with 3. I reached the second milestone:
> it's now possible to run the full Python test suite with these
> optimizations enabled. It confirms that the optimizations don't break
> the Python semantic.

Is the test suite complete enough to say this? (see below)

> Example:
> ---
>  >>> def func():
> ...     return len("abc")
> ...
>  >>> import dis
>  >>> dis.dis(func)
>    2           0 LOAD_GLOBAL              0 (len)
>                3 LOAD_CONST               1 ('abc')
>                6 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
>                9 RETURN_VALUE
>
>  >>> len(func.get_specialized())
> 1
>  >>> specialized=func.get_specialized()[0]
>  >>> dis.dis(specialized['code'])
>    2           0 LOAD_CONST               1 (3)
>                3 RETURN_VALUE
>  >>> len(specialized['guards'])
> 2
>
>  >>> func()
> 3
>
>  >>> len=lambda obj: "mock"
>  >>> func()
> 'mock'

In particular, does the test suite have tests like this, to verify that 
over-riding builtins works?

>  >>> func.get_specialized()
> []
> ---

-- 
Terry Jan Reedy


From victor.stinner at gmail.com  Wed Nov  4 17:23:34 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 4 Nov 2015 23:23:34 +0100
Subject: [Python-Dev] Second milestone of FAT Python
In-Reply-To: <563A672D.6080803@mail.de>
References: <CAMpsgwagj1Jz=Zue_hNYLtsUg7oJEKvo0hB73A3w-XxH1iJi8Q@mail.gmail.com>
 <563A672D.6080803@mail.de>
Message-ID: <CAMpsgwZBt09=tNb1ZepTz84kijKVT77KYykOTuRSETYvh0D69A@mail.gmail.com>

2015-11-04 21:14 GMT+01:00 Sven R. Kunze <srkunze at mail.de>:
> Do you think there will be any change of merging this back into CPython?

If I understood correctly the first feedback on python-ideas, there
were mostly positive, so I believe that it would be possible to merge
the FAT mode into CPython. The changes required to implement the FAT
mode are restricted to a few files and are small. But I would prefer
to keep the whole "FAT mode" optional, to keep exactly the same
performances than before for the "standard" mode.

I will write a PEP for that, but first I have to ensure that FAT
Python works as expected (it's really faster), and that it's generic
enough to be used by projects like Numba, Cython, pythran, etc. (but
not JIT-based interpreters like PyPy or pyston which already have
their own efficient guards.)

Victor

From mike at selik.org  Wed Nov  4 22:59:05 2015
From: mike at selik.org (Michael Selik)
Date: Thu, 05 Nov 2015 03:59:05 +0000
Subject: [Python-Dev] Rationale behind lazy map/filter
Message-ID: <CADqi795c91au8_WZ0uVaoG6uvLx4jsRaPHc6RiKoZhbg9H1mbQ@mail.gmail.com>

> I'm not suggesting restarting at the top (I've elsewhere suggested that
> many such methods would be better as an *iterable* that can be restarted
> at the top by calling iter() multiple times, but that's not the same
> thing). I'm suggesting raising an exception other than StopIteration, so
> that this situation can be detected. If you are writing code that tries
> to resume iterating after the iterator has been exhausted, I have to
> ask: why?

The most obvious case for me would be tailing a file. Loop over the lines
in the file, sleep, then do it again. There are many tasks analogous to
that scenario -- anything querying a shared resource.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151105/8cd90771/attachment.html>

From larry at hastings.org  Wed Nov  4 23:06:32 2015
From: larry at hastings.org (Larry Hastings)
Date: Wed, 4 Nov 2015 20:06:32 -0800
Subject: [Python-Dev] Python 3.5.1 release schedule
Message-ID: <563AD5C8.7040600@hastings.org>


At the request of the platform experts, 3.5.1 is now scheduled to happen 
simultaneously with 2.7.11.  That means:

Saturday November 21, 2015
     tag 3.5.1rc1

Sunday November 22, 2015
     release 3.5.1rc1

Saturday December 5, 2015
     tag 3.5.1 final

Sunday December 6, 2015
     release 3.5.1 final

I'll update the release schedule soon,


//arry/


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151104/2a7e5818/attachment.html>

From victor.stinner at gmail.com  Thu Nov  5 05:33:56 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 5 Nov 2015 11:33:56 +0100
Subject: [Python-Dev] PYC filename and optimization
Message-ID: <CAMpsgwba0OQ6KVyZ9Rw2_gdXHvYVkHcZP8fRqcpVPgWJMefQ_Q@mail.gmail.com>

Hi,

While working on the FAT Python project to try to optimize CPython, I
have an issue with the cached .pyc files. I would like to have one
.pyc without optimization when the FAT mode is disabled (default) and
one .pyc with optimization when the FAT mode is enabled (-F command
line option). Currently, I'm using this filenames:

- Lib/__pycache__/os.cpython-36.pyc: default mode
- Lib/__pycache__/os.cpython-36.fat-0.pyc: FAT mode

With -O:

- Lib/__pycache__/os.cpython-36.opt-1.pyc: default mode
- Lib/__pycache__/os.cpython-36.fat-1.pyc: FAT mode


With -OO:

- Lib/__pycache__/os.cpython-36.opt-2.pyc: default mode
- Lib/__pycache__/os.cpython-36.fat-2.pyc: FAT mode

Is it the "correct" way to name .pyc files? I had to modify unit tests
which expected an exact filename.

An alternative would be to add a new dot:

- Lib/__pycache__/os.cpython-36.fat.pyc
- Lib/__pycache__/os.cpython-36.fat.opt-1.pyc
- Lib/__pycache__/os.cpython-36.fat.opt-2.pyc

Or should I add a flag to the cache tag ("cpython-36")?

- Lib/__pycache__/os.cpython-36f.pyc
- Lib/__pycache__/os.cpython-36f.opt-1.pyc
- Lib/__pycache__/os.cpython-36f.opt-2.pyc

CPython adds the "d" suffix to the cache tag in debug build, but in my
case the flag would be added dynamically at Python startup. The FAT
mode is enabled by the -F command line option, not by a compilation
flag.

Victor

From victor.stinner at gmail.com  Thu Nov  5 09:33:40 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 5 Nov 2015 15:33:40 +0100
Subject: [Python-Dev] Second milestone of FAT Python
In-Reply-To: <n1e0c3$jj4$1@ger.gmane.org>
References: <CAMpsgwagj1Jz=Zue_hNYLtsUg7oJEKvo0hB73A3w-XxH1iJi8Q@mail.gmail.com>
 <n1e0c3$jj4$1@ger.gmane.org>
Message-ID: <CAMpsgwZb_qOLiRm9kd8LBu6_NKHWCpXqJ0VDX_AJdZhzPbEETQ@mail.gmail.com>

Hi,

2015-11-04 23:20 GMT+01:00 Terry Reedy <tjreedy at udel.edu>:
> On 11/4/2015 3:50 AM, Victor Stinner wrote:
>> (...) I reached the second milestone:
>> it's now possible to run the full Python test suite with these
>> optimizations enabled. It confirms that the optimizations don't break
>> the Python semantic.
>
> Is the test suite complete enough to say this? (see below)
> (...)
> In particular, does the test suite have tests like this, to verify that
> over-riding builtins works?

I modified the AST optimizer to not emit guards (remove all guards),
tests still pass. So: no, the optimized code doesn't override builtin
functions.

But I have dedicated unit tests for that in test_fat ;-)

Victor

From barry at python.org  Thu Nov  5 10:40:40 2015
From: barry at python.org (Barry Warsaw)
Date: Thu, 5 Nov 2015 09:40:40 -0600
Subject: [Python-Dev] PYC filename and optimization
In-Reply-To: <CAMpsgwba0OQ6KVyZ9Rw2_gdXHvYVkHcZP8fRqcpVPgWJMefQ_Q@mail.gmail.com>
References: <CAMpsgwba0OQ6KVyZ9Rw2_gdXHvYVkHcZP8fRqcpVPgWJMefQ_Q@mail.gmail.com>
Message-ID: <20151105094040.6062a7b9@anarchist>

On Nov 05, 2015, at 11:33 AM, Victor Stinner wrote:

>- Lib/__pycache__/os.cpython-36.pyc: default mode
>- Lib/__pycache__/os.cpython-36.fat-0.pyc: FAT mode
>
>With -O:
>
>- Lib/__pycache__/os.cpython-36.opt-1.pyc: default mode
>- Lib/__pycache__/os.cpython-36.fat-1.pyc: FAT mode
>
>
>With -OO:
>
>- Lib/__pycache__/os.cpython-36.opt-2.pyc: default mode
>- Lib/__pycache__/os.cpython-36.fat-2.pyc: FAT mode
>
>Is it the "correct" way to name .pyc files? I had to modify unit tests
>which expected an exact filename.

I think this is the logical extension of PEP 488 for your use case.

Cheers,
-Barry

From brett at python.org  Thu Nov  5 13:10:41 2015
From: brett at python.org (Brett Cannon)
Date: Thu, 05 Nov 2015 18:10:41 +0000
Subject: [Python-Dev] PYC filename and optimization
In-Reply-To: <20151105094040.6062a7b9@anarchist>
References: <CAMpsgwba0OQ6KVyZ9Rw2_gdXHvYVkHcZP8fRqcpVPgWJMefQ_Q@mail.gmail.com>
 <20151105094040.6062a7b9@anarchist>
Message-ID: <CAP1=2W7B1h2C7NTAxs-wR77fP42zoKjcWHCtrjALEjB8-VfLOg@mail.gmail.com>

On Thu, 5 Nov 2015 at 07:41 Barry Warsaw <barry at python.org> wrote:

> On Nov 05, 2015, at 11:33 AM, Victor Stinner wrote:
>
> >- Lib/__pycache__/os.cpython-36.pyc: default mode
> >- Lib/__pycache__/os.cpython-36.fat-0.pyc: FAT mode
> >
> >With -O:
> >
> >- Lib/__pycache__/os.cpython-36.opt-1.pyc: default mode
> >- Lib/__pycache__/os.cpython-36.fat-1.pyc: FAT mode
> >
> >
> >With -OO:
> >
> >- Lib/__pycache__/os.cpython-36.opt-2.pyc: default mode
> >- Lib/__pycache__/os.cpython-36.fat-2.pyc: FAT mode
> >
> >Is it the "correct" way to name .pyc files? I had to modify unit tests
> >which expected an exact filename.
>
> I think this is the logical extension of PEP 488 for your use case.
>

It will require tweaking importlib.util.cache_from_source() to no longer
restriction to alphanumeric-only for the optimization level, but that can
be tweaked to use `opt-` when given an alphanumeric and then to blindly use
another value otherwise.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151105/a08c286d/attachment.html>

From rdmurray at bitdance.com  Thu Nov  5 13:24:07 2015
From: rdmurray at bitdance.com (R. David Murray)
Date: Thu, 05 Nov 2015 13:24:07 -0500
Subject: [Python-Dev] Rationale behind lazy map/filter
In-Reply-To: <CADqi795c91au8_WZ0uVaoG6uvLx4jsRaPHc6RiKoZhbg9H1mbQ@mail.gmail.com>
References: <CADqi795c91au8_WZ0uVaoG6uvLx4jsRaPHc6RiKoZhbg9H1mbQ@mail.gmail.com>
Message-ID: <20151105182408.47A59B1408B@webabinitio.net>

On Thu, 05 Nov 2015 03:59:05 +0000, Michael Selik <mike at selik.org> wrote:
> > I'm not suggesting restarting at the top (I've elsewhere suggested that
> > many such methods would be better as an *iterable* that can be restarted
> > at the top by calling iter() multiple times, but that's not the same
> > thing). I'm suggesting raising an exception other than StopIteration, so
> > that this situation can be detected. If you are writing code that tries
> > to resume iterating after the iterator has been exhausted, I have to
> > ask: why?
> 
> The most obvious case for me would be tailing a file. Loop over the lines
> in the file, sleep, then do it again. There are many tasks analogous to
> that scenario -- anything querying a shared resource.

The 'file' iterator actually breaks the rules of iterators: it does
*not* continue to raise StopIteration once it has been exhausted, if
more input becomes available.  Given that it is one of the most commonly
used iterators (and I would not be surprised if other special-purpose
iterators copied its design), this pattern does seem like a blocker for
the proposal.

--David

From status at bugs.python.org  Fri Nov  6 12:08:33 2015
From: status at bugs.python.org (Python tracker)
Date: Fri,  6 Nov 2015 18:08:33 +0100 (CET)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20151106170833.6216C56764@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2015-10-30 - 2015-11-06)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5218 (+14)
  closed 32108 (+35)
  total  37326 (+49)

Open issues with patches: 2301 


Issues opened (33)
==================

#25520: unittest load_tests protocol not working as documented
http://bugs.python.org/issue25520  opened by stefan

#25521: optparse module does not emit DeprecationWarning
http://bugs.python.org/issue25521  opened by John Hagen

#25522: IDLE: warn if save-as name matches stdlib name
http://bugs.python.org/issue25522  opened by terry.reedy

#25524: Add PyMemoryView_FromObjectWithFlags()
http://bugs.python.org/issue25524  opened by skrah

#25527: Invalid (... and confusing) warning raised by 2to3 regarding r
http://bugs.python.org/issue25527  opened by Antony.Lee

#25528: Attempt to further increase test coverage of calendar module
http://bugs.python.org/issue25528  opened by Rohit Mediratta

#25529: Provide access to the validated certificate chain in ssl modul
http://bugs.python.org/issue25529  opened by Lukasa

#25530: ssl: OP_NO_SSLv3 should always be set unless a user specifical
http://bugs.python.org/issue25530  opened by alex

#25531: greenlet header file is missing inside virtualenv
http://bugs.python.org/issue25531  opened by kracekumar

#25532: infinite loop when running inspect.unwrap over unittest.mock.c
http://bugs.python.org/issue25532  opened by cjw296

#25533: Make pkgutil.iter_modules() yield built-in modules
http://bugs.python.org/issue25533  opened by martin.panter

#25534: SimpleHTTPServer throwed an exception due to negtive st_mtime 
http://bugs.python.org/issue25534  opened by Sean.Wang

#25538: Traceback from __exit__ method is misleading
http://bugs.python.org/issue25538  opened by r.david.murray

#25541: Wrong usage of sockaddr_un struct for abstract namespace unix 
http://bugs.python.org/issue25541  opened by soutys

#25543: locale.atof keep '.' even if not part of the localeconv
http://bugs.python.org/issue25543  opened by ced

#25544: cleanup temporary files in distutils.has_function
http://bugs.python.org/issue25544  opened by minrk

#25545: email parsing docs need to be clear that only ASCII strings ar
http://bugs.python.org/issue25545  opened by tanzer at swing.co.at

#25546: python 3.5 installation problem; Error 0x80240017: Failed to e
http://bugs.python.org/issue25546  opened by rfrank1234

#25547: Improve repr for files to show whether the file is open or clo
http://bugs.python.org/issue25547  opened by rhettinger

#25548: Show the address in the repr for class objects
http://bugs.python.org/issue25548  opened by rhettinger

#25550: RecursionError in re with '(' * 500
http://bugs.python.org/issue25550  opened by The Compiler

#25551: Event's test_reset_internal_locks too fragile
http://bugs.python.org/issue25551  opened by nirs

#25553: smtpd strips final carraige return from received message body
http://bugs.python.org/issue25553  opened by Deli Zhang

#25555: Fix parser and AST: fill lineno and col_offset when compiling 
http://bugs.python.org/issue25555  opened by haypo

#25557: Optimize LOAD_NAME bytecode
http://bugs.python.org/issue25557  opened by haypo

#25558: Use static asserts in C code
http://bugs.python.org/issue25558  opened by serhiy.storchaka

#25559: signal.siginterrupt description has typo
http://bugs.python.org/issue25559  opened by Tom Meagher

#25560: Unhandled warning in test_unicode_file
http://bugs.python.org/issue25560  opened by ppperry

#25562: Python 2 & 3 don't allow the user to disable ctypes SEH in win
http://bugs.python.org/issue25562  opened by tzickel

#25563: Windows 10 IDLE / Tkinter startup problem
http://bugs.python.org/issue25563  opened by veensl

#25564: IDLE behaves differently than python on `del __builtins__`
http://bugs.python.org/issue25564  opened by ppperry

#25566: asyncio reference cycles after ConnectionResetError
http://bugs.python.org/issue25566  opened by joente

#25567: shlex.quote doesn't work on bytestrings
http://bugs.python.org/issue25567  opened by Jonas Thiem



Most recent 15 issues with no replies (15)
==========================================

#25566: asyncio reference cycles after ConnectionResetError
http://bugs.python.org/issue25566

#25560: Unhandled warning in test_unicode_file
http://bugs.python.org/issue25560

#25551: Event's test_reset_internal_locks too fragile
http://bugs.python.org/issue25551

#25544: cleanup temporary files in distutils.has_function
http://bugs.python.org/issue25544

#25529: Provide access to the validated certificate chain in ssl modul
http://bugs.python.org/issue25529

#25528: Attempt to further increase test coverage of calendar module
http://bugs.python.org/issue25528

#25527: Invalid (... and confusing) warning raised by 2to3 regarding r
http://bugs.python.org/issue25527

#25524: Add PyMemoryView_FromObjectWithFlags()
http://bugs.python.org/issue25524

#25520: unittest load_tests protocol not working as documented
http://bugs.python.org/issue25520

#25518: Investigate implementation of PyOS_CheckStack on OSX
http://bugs.python.org/issue25518

#25511: multiprocessing pool blocks SIGTERM from being handled
http://bugs.python.org/issue25511

#25497: Rewrite test_robotparser
http://bugs.python.org/issue25497

#25491: ftplib.sendcmd only accepts string
http://bugs.python.org/issue25491

#25486: Resurrect inspect.getargspec() in 3.6
http://bugs.python.org/issue25486

#25479: Increase unit test coverage for abc.py
http://bugs.python.org/issue25479



Most recent 15 issues waiting for review (15)
=============================================

#25558: Use static asserts in C code
http://bugs.python.org/issue25558

#25557: Optimize LOAD_NAME bytecode
http://bugs.python.org/issue25557

#25555: Fix parser and AST: fill lineno and col_offset when compiling 
http://bugs.python.org/issue25555

#25551: Event's test_reset_internal_locks too fragile
http://bugs.python.org/issue25551

#25544: cleanup temporary files in distutils.has_function
http://bugs.python.org/issue25544

#25533: Make pkgutil.iter_modules() yield built-in modules
http://bugs.python.org/issue25533

#25530: ssl: OP_NO_SSLv3 should always be set unless a user specifical
http://bugs.python.org/issue25530

#25528: Attempt to further increase test coverage of calendar module
http://bugs.python.org/issue25528

#25521: optparse module does not emit DeprecationWarning
http://bugs.python.org/issue25521

#25516: threading.Condition._is_owned() is wrong when using threading.
http://bugs.python.org/issue25516

#25498: Python 3.4.3 core dump with simple sample code
http://bugs.python.org/issue25498

#25497: Rewrite test_robotparser
http://bugs.python.org/issue25497

#25495: binascii documentation incorrect
http://bugs.python.org/issue25495

#25489: sys.exit() caught in exception handler
http://bugs.python.org/issue25489

#25485: Add a context manager to telnetlib.Telnet
http://bugs.python.org/issue25485



Top 10 most discussed issues (10)
=================================

#25521: optparse module does not emit DeprecationWarning
http://bugs.python.org/issue25521  18 msgs

#25495: binascii documentation incorrect
http://bugs.python.org/issue25495  15 msgs

#25531: greenlet header file is missing inside virtualenv
http://bugs.python.org/issue25531  13 msgs

#25558: Use static asserts in C code
http://bugs.python.org/issue25558  12 msgs

#25545: email parsing docs need to be clear that only ASCII strings ar
http://bugs.python.org/issue25545   9 msgs

#25498: Python 3.4.3 core dump with simple sample code
http://bugs.python.org/issue25498   7 msgs

#23496: Steps for Android Native Build of Python 3.4.2
http://bugs.python.org/issue23496   6 msgs

#24726: OrderedDict has strange behaviour when dict.__setitem__ is use
http://bugs.python.org/issue24726   6 msgs

#25532: infinite loop when running inspect.unwrap over unittest.mock.c
http://bugs.python.org/issue25532   6 msgs

#18010: pydoc search chokes on import errors
http://bugs.python.org/issue18010   5 msgs



Issues closed (33)
==================

#5550: [urllib.request]: Comparison of HTTP headers should be insensi
http://bugs.python.org/issue5550  closed by martin.panter

#12300: Document pydoc.help
http://bugs.python.org/issue12300  closed by terry.reedy

#18973: Use argparse in the calendar module
http://bugs.python.org/issue18973  closed by serhiy.storchaka

#24803: PyNumber_Long Buffer Over-read.patch
http://bugs.python.org/issue24803  closed by serhiy.storchaka

#25395: SIGSEGV using json.tool: highly nested OrderedDict
http://bugs.python.org/issue25395  closed by serhiy.storchaka

#25426: Deprecate the regex_compile benchmark
http://bugs.python.org/issue25426  closed by brett.cannon

#25443: Add a count of how many benchmarks are left to run
http://bugs.python.org/issue25443  closed by brett.cannon

#25449: Test OrderedDict subclass
http://bugs.python.org/issue25449  closed by serhiy.storchaka

#25450: Python 3.5 starts in C:\Windows\system32 as current directory
http://bugs.python.org/issue25450  closed by steve.dower

#25454: operator.methodcaller should accept additional arguments durin
http://bugs.python.org/issue25454  closed by rhettinger

#25474: Weird behavior when setting f_trace in a context manager
http://bugs.python.org/issue25474  closed by Fred Gansevles

#25483: Improve f-string implementation: FORMAT_VALUE opcode
http://bugs.python.org/issue25483  closed by eric.smith

#25487: imp module DeprecationWarning in test suite
http://bugs.python.org/issue25487  closed by brett.cannon

#25493: warnings.warn: wrong stacklevel causes import of local file "s
http://bugs.python.org/issue25493  closed by brett.cannon

#25505: undefined name 'window' in Tools/scripts/fixdiv.py
http://bugs.python.org/issue25505  closed by terry.reedy

#25510: fileinput.FileInput.readline() always returns str object at th
http://bugs.python.org/issue25510  closed by serhiy.storchaka

#25519: Minor difflib documentation bug
http://bugs.python.org/issue25519  closed by terry.reedy

#25523: Correct "a" article to "an" article
http://bugs.python.org/issue25523  closed by martin.panter

#25525: Deallocation scheme for memoryview is unsafe
http://bugs.python.org/issue25525  closed by skrah

#25526: (python2.7.10)ImportError: No module named _ssl
http://bugs.python.org/issue25526  closed by zach.ware

#25535: collections.Counter methods return Counter objects instead of 
http://bugs.python.org/issue25535  closed by rhettinger

#25536: use sys.platform instead of os.name in asyncio docs consistent
http://bugs.python.org/issue25536  closed by gvanrossum

#25537: Call `isinstance` instead of `issubclass` during exception han
http://bugs.python.org/issue25537  closed by r.david.murray

#25539: python3 fail on parsing http header
http://bugs.python.org/issue25539  closed by ?????????

#25540: virtual conf file encoding error on windows
http://bugs.python.org/issue25540  closed by serhiy.storchaka

#25542: tuple unpacking on assignment should be lazy for unpacked gene
http://bugs.python.org/issue25542  closed by Mario Wenzel

#25549: call sum on list of timedelta throws TypeError
http://bugs.python.org/issue25549  closed by r.david.murray

#25552: python turtle page does not run
http://bugs.python.org/issue25552  closed by r.david.murray

#25554: memory leak (reference cycles) using re
http://bugs.python.org/issue25554  closed by serhiy.storchaka

#25556: Fixes for PyObject_GetItem()
http://bugs.python.org/issue25556  closed by haypo

#25561: unstable result of time.monotonic
http://bugs.python.org/issue25561  closed by eric.smith

#25565: subprocess.Popen creates inheritable file descriptors on Windo
http://bugs.python.org/issue25565  closed by eryksun

#25568: typing backport: AttributeError: 'module' object has no attrib
http://bugs.python.org/issue25568  closed by nicolas33

From mark at markroseman.com  Fri Nov  6 12:08:24 2015
From: mark at markroseman.com (Mark Roseman)
Date: Fri, 6 Nov 2015 09:08:24 -0800
Subject: [Python-Dev] modernizing IDLE
Message-ID: <A135E83B-E078-482C-B062-D4C07BC6A30C@markroseman.com>

Just wanted to share some documentation I?ve put together (and a snapshot of the code) on the effort to improve the appearance and behaviour of IDLE:

	http://www.tkdocs.com/tutorial/idle.html <http://www.tkdocs.com/tutorial/idle.html>

(There?s also currently a post on Hacker News about this).

While Terry can better speak to plans and timelines about getting these changes integrated in, it seems a good time to get some other people to have a closer look and share any feedback.

Mark

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151106/6d283242/attachment.html>

From cmkleffner at gmail.com  Sat Nov  7 10:55:33 2015
From: cmkleffner at gmail.com (Carl Kleffner)
Date: Sat, 7 Nov 2015 16:55:33 +0100
Subject: [Python-Dev] modernizing IDLE
In-Reply-To: <A135E83B-E078-482C-B062-D4C07BC6A30C@markroseman.com>
References: <A135E83B-E078-482C-B062-D4C07BC6A30C@markroseman.com>
Message-ID: <CAGGsPMy8aSzw4LOYP7bN1h8dsHbGFugeUhTt8H_j_EkpV2=qGw@mail.gmail.com>

there is also http://idlex.sourceforge.net/about.html to mention.

2015-11-06 18:08 GMT+01:00 Mark Roseman <mark at markroseman.com>:

> Just wanted to share some documentation I?ve put together (and a snapshot
> of the code) on the effort to improve the appearance and behaviour of IDLE:
>
> http://www.tkdocs.com/tutorial/idle.html
>
> (There?s also currently a post on Hacker News about this).
>
> While Terry can better speak to plans and timelines about getting these
> changes integrated in, it seems a good time to get some other people to
> have a closer look and share any feedback.
>
> Mark
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/cmkleffner%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151107/ac2e1528/attachment.html>

From pmiscml at gmail.com  Sat Nov  7 18:00:59 2015
From: pmiscml at gmail.com (Paul Sokolovsky)
Date: Sun, 8 Nov 2015 02:00:59 +0300
Subject: [Python-Dev] [ANN] MicroPython 1.5
Message-ID: <20151108020059.54a36344@x230>

Hello,

MicroPython is a lean and efficient Python implementation for
microcontrollers, embedded, and mobile systems (which also runs just as
fine on desktops of course).

https://github.com/micropython/micropython

Recent 1.5 release is an important milestone for the project, major
changes including:

1. Advanced REPL support with smart auto-indent and auto-completion for
comfortable interactive typing pf Python code, easily switchable to
paste mode to copy pre-made code snippets.

2. Support for client SSL connections.

3. upip, MicroPython builtin package manager, is now fully standalone
thanks to SSL support.

4. There's new, elaborated API to access hardware features ("machine"
module).

5. There're now 2 well-supported microcontroller boards for
MicroPython, and dozen(s) community-supported ones.

6. MicroPython was selected as one of the languages supported for BBC
micro:bit initiative:
http://ntoll.org/article/story-micropython-on-microbit

7. There's work on native efficient JNI bridge, inspired by Kivy's
PyJNIus module, included into growing Android port.


More detailed changelog for this version is at
https://github.com/micropython/micropython/releases/tag/v1.5

MicroPython supports growing subset of Python3 standard library
(including simplified asyncio package) - not included by default with
an executable, but easily installable per-module using builtin "upip"
package manager:
https://github.com/micropython/micropython-lib


-- 
Best regards,
 Paul                          mailto:pmiscml at gmail.com

From ncoghlan at gmail.com  Tue Nov 10 07:45:04 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 10 Nov 2015 22:45:04 +1000
Subject: [Python-Dev] modernizing IDLE
In-Reply-To: <A135E83B-E078-482C-B062-D4C07BC6A30C@markroseman.com>
References: <A135E83B-E078-482C-B062-D4C07BC6A30C@markroseman.com>
Message-ID: <CADiSq7dHV-zH9gmgKaF0OCrsJarrEki9_iN_JdGFXb7s8mxWPA@mail.gmail.com>

On 7 November 2015 at 03:08, Mark Roseman <mark at markroseman.com> wrote:
> Just wanted to share some documentation I?ve put together (and a snapshot of
> the code) on the effort to improve the appearance and behaviour of IDLE:
>
> http://www.tkdocs.com/tutorial/idle.html
>
> (There?s also currently a post on Hacker News about this).
>
> While Terry can better speak to plans and timelines about getting these
> changes integrated in, it seems a good time to get some other people to have
> a closer look and share any feedback.

Very cool! Thanks to all involved for these modernisation efforts :)

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From a.r.kaija at gmail.com  Tue Nov 10 02:19:24 2015
From: a.r.kaija at gmail.com (Alec Kaija)
Date: Tue, 10 Nov 2015 02:19:24 -0500
Subject: [Python-Dev] This isn't a question
Message-ID: <CAL-r3eHdNBbTvBnH78RYWd4wcS6JjHgA+Pt00O-FQ8tawhnAWA@mail.gmail.com>

Sorry to bother anyone, but I just wanted to thank the Python community for
being awesome and for all of the ways you have all inspired/supported me
(mostly without even knowing it).

Thanks!


a
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151110/df4e98f4/attachment.html>

From brett at python.org  Tue Nov 10 13:09:37 2015
From: brett at python.org (Brett Cannon)
Date: Tue, 10 Nov 2015 18:09:37 +0000
Subject: [Python-Dev] This isn't a question
In-Reply-To: <CAL-r3eHdNBbTvBnH78RYWd4wcS6JjHgA+Pt00O-FQ8tawhnAWA@mail.gmail.com>
References: <CAL-r3eHdNBbTvBnH78RYWd4wcS6JjHgA+Pt00O-FQ8tawhnAWA@mail.gmail.com>
Message-ID: <CAP1=2W6u3co+fqyW31mZHS-ix28uZEoh9oTV2MsnvsXpeh+2nQ@mail.gmail.com>

You're quite welcome! And thanks for taking the time to say "thanks". :)

On Tue, 10 Nov 2015, 11:35 Alec Kaija <a.r.kaija at gmail.com> wrote:

> Sorry to bother anyone, but I just wanted to thank the Python community
> for being awesome and for all of the ways you have all inspired/supported
> me (mostly without even knowing it).
>
> Thanks!
>
>
> a
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151110/8d2ee1c2/attachment.html>

From vgr255 at live.ca  Tue Nov 10 17:00:04 2015
From: vgr255 at live.ca (Emanuel Barry)
Date: Tue, 10 Nov 2015 17:00:04 -0500
Subject: [Python-Dev] This isn't a question
In-Reply-To: <CAP1=2W6u3co+fqyW31mZHS-ix28uZEoh9oTV2MsnvsXpeh+2nQ@mail.gmail.com>
References: <CAL-r3eHdNBbTvBnH78RYWd4wcS6JjHgA+Pt00O-FQ8tawhnAWA@mail.gmail.com>,
 <CAP1=2W6u3co+fqyW31mZHS-ix28uZEoh9oTV2MsnvsXpeh+2nQ@mail.gmail.com>
Message-ID: <BLU172-W25C1C1E7ABBC709F2184D291140@phx.gbl>

While I'm a part of the community, that wasn't always the case and it's very large, so I'll take this opportunity to come in and say a big thank you as well!

From: brett at python.org
Date: Tue, 10 Nov 2015 18:09:37 +0000
To: a.r.kaija at gmail.com; python-dev at python.org
Subject: Re: [Python-Dev] This isn't a question

You're quite welcome! And thanks for taking the time to say "thanks". :)

On Tue, 10 Nov 2015, 11:35 Alec Kaija <a.r.kaija at gmail.com> wrote:
Sorry to bother anyone, but I just wanted to thank the Python community for being awesome and for all of the ways you have all inspired/supported me (mostly without even knowing it).
Thanks!

a
_______________________________________________

Python-Dev mailing list

Python-Dev at python.org

https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org



_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/vgr255%40live.ca 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151110/c39fcb79/attachment.html>

From motoom at xs4all.nl  Tue Nov 10 18:02:59 2015
From: motoom at xs4all.nl (Michiel Overtoom)
Date: Wed, 11 Nov 2015 00:02:59 +0100
Subject: [Python-Dev] modernizing IDLE
In-Reply-To: <A135E83B-E078-482C-B062-D4C07BC6A30C@markroseman.com>
References: <A135E83B-E078-482C-B062-D4C07BC6A30C@markroseman.com>
Message-ID: <25544A86-7073-47F6-9189-F734D5007B01@xs4all.nl>


> On 06 Nov 2015, at 18:08, Mark Roseman <mark at markroseman.com> wrote:
> 
> (There?s also currently a post on Hacker News about this).

You have a link for that HN item? I've looked at the first five pages but couldn't find it.

Greetings,


From rymg19 at gmail.com  Tue Nov 10 18:23:15 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Tue, 10 Nov 2015 17:23:15 -0600
Subject: [Python-Dev] modernizing IDLE
In-Reply-To: <25544A86-7073-47F6-9189-F734D5007B01@xs4all.nl>
References: <A135E83B-E078-482C-B062-D4C07BC6A30C@markroseman.com>
 <25544A86-7073-47F6-9189-F734D5007B01@xs4all.nl>
Message-ID: <CAO41-mM2kTxLB0chwrpoTKVFRxyyLoOZeAOq_zUvw=czgox4OQ@mail.gmail.com>

On Tue, Nov 10, 2015 at 5:02 PM, Michiel Overtoom <motoom at xs4all.nl> wrote:

>
> > On 06 Nov 2015, at 18:08, Mark Roseman <mark at markroseman.com> wrote:
> >
> > (There?s also currently a post on Hacker News about this).
>
> You have a link for that HN item? I've looked at the first five pages but
> couldn't find it.
>
>
https://news.ycombinator.com/item?id=10519785

I used the big "Search" box at the bottom of the main Hacker News page. ;)


> Greetings,
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
>



-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151110/5b246968/attachment.html>

From ncoghlan at gmail.com  Tue Nov 10 19:06:57 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 11 Nov 2015 10:06:57 +1000
Subject: [Python-Dev] This isn't a question
In-Reply-To: <CAP1=2W6u3co+fqyW31mZHS-ix28uZEoh9oTV2MsnvsXpeh+2nQ@mail.gmail.com>
References: <CAL-r3eHdNBbTvBnH78RYWd4wcS6JjHgA+Pt00O-FQ8tawhnAWA@mail.gmail.com>
 <CAP1=2W6u3co+fqyW31mZHS-ix28uZEoh9oTV2MsnvsXpeh+2nQ@mail.gmail.com>
Message-ID: <CADiSq7dd_ZZTXmsv5dg-3SFpf_to_cg4Fn=R0B+3=K_LMp=pLg@mail.gmail.com>

On 11 November 2015 at 04:09, Brett Cannon <brett at python.org> wrote:
> You're quite welcome! And thanks for taking the time to say "thanks". :)

Indeed! Folks saying "Thank you" has a surprisingly big impact, since
our natural tendency is to spend time dwelling on all the things that
still need improvement :)

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Nov 10 19:47:53 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 11 Nov 2015 10:47:53 +1000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
Message-ID: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>

Hi folks,

I have a confession to make - I dropped the ball on the HTTPS
verification backport proposals in PEP 493, and let the upstream and
downstream approval processes get out of sequence.

As a result, the RHEL 7.2 beta released back in September incorporates
the HTTPS verification feature backport based on the current PEP 493
draft, even though that hasn't formally been pronounced as an Active
recommendation by python-dev yet.

Accordingly, I'm belatedly submitting it for pronouncement now:
https://www.python.org/dev/peps/pep-0493/

There's currently no BDFL-Delegate assigned, so if Guido doesn't want
to handle it, we'll need to address that question first.

Our last discussion back in July seemed to show that folks either
didn't care about the question (because they're using unmodified
upstream versions so the PEP didn't affect them), or else thought the
approach described in the PEP was reasonable, so I'm hoping the
consequences of my mistake won't be too severe.

Regards,
Nick.

P.S. I'm aware that this looks like presenting a fait accompli at a
point where it's too late to realistically say "No", but the truth is
that preparation for the Python in Education miniconf at PyCon
Australia ramped up immediately after the July discussion, and then I
personally got confused as to the scope of what was being included in
7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493
being deferred to a later release, but it's actually the whole package
of 466+476+493). That's my fault for trying to keep track of too many
things at once (and thus failing at some of them), not anyone else's.

================================

PEP: 493
Title: HTTPS verification recommendations for Python 2.7 redistributors
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan <ncoghlan at gmail.com>,
        Robert Kuska <rkuska at redhat.com>,
        Marc-Andr? Lemburg <mal at lemburg.com>
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 10-May-2015
Post-History: 06-Jul-2015


Abstract
========

PEP 476 updated Python's default handling of HTTPS certificates to be
appropriate for communication over the public internet. The Python 2.7 long
term maintenance series was judged to be in scope for this change, with the
new behaviour introduced in the Python 2.7.9 maintenance release.

This PEP provides recommendations to downstream redistributors wishing to
provide a smoother migration experience when helping their users to manage
this change in Python's default behaviour.


Rationale
=========

PEP 476 changed Python's default behaviour to better match the needs and
expectations of developers operating over the public internet, a category
which appears to include most new Python developers. It is the position of
the authors of this PEP that this was a correct decision.

However, it is also the case that this change *does* cause problems for
infrastructure administrators operating private intranets that rely on
self-signed certificates, or otherwise encounter problems with the new default
certificate verification settings.

The long term answer for such environments is to update their internal
certificate management to at least match the standards set by the public
internet, but in the meantime, it is desirable to offer these administrators
a way to continue receiving maintenance updates to the Python 2.7 series,
without having to gate that on upgrades to their certificate management
infrastructure.

PEP 476 did attempt to address this question, by covering how to revert the
new settings process wide by monkeypatching the ``ssl`` module to restore the
old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed
to allow system administrators to disable the feature by default in their
Standard Operating Environment definition has been determined to be
insufficient in at least some cases. The specific case of interest to the
authors of this PEP is the one where a Linux distributor aims to provide
their users with a
`smoother migration path
<https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__
than the standard one provided by consuming upstream CPython 2.7 releases
directly, but other potential challenges have also been pointed out with
updating embedded Python runtimes and other user level installations of Python.

Rather than allowing a plethora of mutually incompatibile migration techniques
to bloom, this PEP proposes two alternative approaches that redistributors
may take when addressing these problems. Redistributors may choose to implement
one, both, or neither of these approaches based on their assessment of the
needs of their particular userbase.

These designs are being proposed as a recommendation for redistributors, rather
than as new upstream features, as they are needed purely to support legacy
environments migrating from older versions of Python 2.7. Neither approach
is being proposed as an upstream Python 2.7 feature, nor as a feature in any
version of Python 3 (whether published directly by the Python Software
Foundation or by a redistributor).


Requirements for capability detection
=====================================

As these recommendations are intended to cover backports to earlier Python
versions, the Python version number cannot be used as a reliable means for
detecting them. Instead, the recommendations are defined to allow the presence
or absence of the feature to be determined using the following technique::

    python -c "import ssl; ssl._relevant_attribute"

This will fail with `AttributeError` (and hence a non-zero return code) if the
relevant capability is not available.

The marker attributes are prefixed with an underscore to indicate the
implementation dependent nature of these capabilities - not all Python
distributions will offer them, only those that are providing a multi-stage
migration process from the legacy HTTPS handling to the new default behaviour.


Recommendation for an environment variable based security downgrade
===================================================================

Some redistributors may wish to provide a per-application option to disable
certificate verification in selected applications that run on or embed CPython
without needing to modify the application itself.

In these cases, a configuration mechanism is needed that provides:

* an opt-out model that allows certificate verification to be selectively
  turned off for particular applications after upgrading to a version of
  Python that verifies certificates by default
* the ability for all users to configure this setting on a per-application
  basis, rather than on a per-system, or per-Python-installation basis

This approach may be used for any redistributor provided version of Python 2.7,
including those that advertise themselves as providing Python 2.7.9 or later.


Required marker attribute
-------------------------

The required marker attribute on the ``ssl`` module when implementing this
recommendation is::

    _https_verify_envvar = 'PYTHONHTTPSVERIFY'

This not only makes it straightforward to detect the presence (or absence) of
the capability, it also makes it possible to programmatically determine the
relevant environment variable name.


Recommended modifications to the Python standard library
--------------------------------------------------------

The recommended approach to providing a per-application configuration setting
for HTTPS certificate verification that doesn't require modifications to the
application itself is to:

* modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment
  variable when the module is first imported into a Python process
* set the ``ssl._create_default_https_context`` function to be an alias for
  ``ssl._create_unverified_context`` if this environment variable is present
  and set to ``'0'``
* otherwise, set the ``ssl._create_default_https_context`` function to be an
  alias for ``ssl.create_default_context`` as usual


Example implementation
----------------------

::

    _https_verify_envvar = 'PYTHONHTTPSVERIFY'

    def _get_https_context_factory():
        config_setting = os.environ.get(_https_verify_envvar)
        if config_setting == '0':
            return _create_unverified_context
        return create_default_context

    _create_default_https_context = _get_https_context_factory()


Security Considerations
-----------------------

Relative to an unmodified version of CPython 2.7.9 or later, this approach
does introduce a new downgrade attack against the default security settings
that potentially allows a sufficiently determined attacker to revert Python
to the vulnerable configuration used in CPython 2.7.8 and earlier releases.
However, such an attack requires the ability to modify the execution
environment of a Python process prior to the import of the ``ssl`` module,
and any attacker with such access would already be able to modify the
behaviour of the underlying OpenSSL implementation.


Recommendation for backporting to earlier Python versions
=========================================================

Some redistributors, most notably Linux distributions, may choose to backport
the PEP 476 HTTPS verification changes to modified Python versions based on
earlier Python 2 maintenance releases. In these cases, a configuration
mechanism is needed that provides:

* an opt-in model that allows the decision to enable HTTPS certificate
  verification to be made independently of the decision to upgrade to the
  Python version where the feature was first backported
* the ability for system administrators to set the default behaviour of Python
  applications and scripts run directly in the system Python installation
* the ability for the redistributor to consider changing the default behaviour
  of *new* installations at some point in the future without impacting existing
  installations that have been explicitly configured to skip verifying HTTPS
  certificates by default

This approach should not be used for any Python installation that advertises
itself as providing Python 2.7.9 or later, as most Python users will have the
reasonable expectation that all such environments will validate HTTPS
certificates by default.


Required marker attribute
-------------------------

The required marker attribute on the ``ssl`` module when implementing this
recommendation is::

    _cert_verification_config = '<path to configuration file>'

This not only makes it straightforward to detect the presence (or absence) of
the capability, it also makes it possible to programmatically determine the
relevant configuration file name.


Recommended modifications to the Python standard library
--------------------------------------------------------

The recommended approach to backporting the PEP 476 modifications to an earlier
point release is to implement the following changes relative to the default
PEP 476 behaviour implemented in Python 2.7.9+:

* modify the ``ssl`` module to read a system wide configuration file when the
  module is first imported into a Python process
* define a platform default behaviour (either verifying or not verifying HTTPS
  certificates) to be used if this configuration file is not present
* support selection between the following three modes of operation:

  * ensure HTTPS certificate verification is enabled
  * ensure HTTPS certificate verification is disabled
  * delegate the decision to the redistributor providing this Python version

* set the ``ssl._create_default_https_context`` function to be an alias for
  either ``ssl.create_default_context`` or ``ssl._create_unverified_context``
  based on the given configuration setting.


Recommended file location
-------------------------

This approach is currently only defined for \*nix system Python installations.

The recommended configuration file name is
``/etc/python/cert-verification.cfg``.

The ``.cfg`` filename extension is recommended for consistency with the
``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library.


Recommended file format
-----------------------

The configuration file should use a ConfigParser ini-style format with a
single section named ``[https]`` containing one required setting ``verify``.

Permitted values for ``verify`` are:

* ``enable``: ensure HTTPS certificate verification is enabled by default
* ``disable``: ensure HTTPS certificate verification is disabled by default
* ``platform_default``: delegate the decision to the redistributor providing
  this particular Python version

If the ``[https]`` section or the ``verify`` setting are missing, or if the
``verify`` setting is set to an unknown value, it should be treated as if the
configuration file is not present.


Example implementation
----------------------

::

    _cert_verification_config = '/etc/python/cert-verification.cfg'

    def _get_https_context_factory():
        # Check for a system-wide override of the default behaviour
        context_factories = {
            'enable': create_default_context,
            'disable': _create_unverified_context,
            'platform_default': _create_unverified_context, # For now :)
        }
        import ConfigParser
        config = ConfigParser.RawConfigParser()
        config.read(_cert_verification_config)
        try:
            verify_mode = config.get('https', 'verify')
        except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
            verify_mode = 'platform_default'
        default_factory = context_factories.get('platform_default')
        return context_factories.get(verify_mode, default_factory)

    _create_default_https_context = _get_https_context_factory()


Security Considerations
-----------------------

The specific recommendations for the backporting case are designed to work for
privileged, security sensitive processes, even those being run in the following
locked down configuration:

* run from a locked down administrator controlled directory rather than a normal
  user directory (preventing ``sys.path[0]`` based privilege escalation attacks)
* run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based
  privilege escalation attacks)
* run using the ``-s`` switch (preventing user site directory based privilege
  escalation attacks)
* run using the ``-S`` switch (preventing ``sitecustomize`` based privilege
  escalation attacks)

The intent is that the *only* reason HTTPS verification should be getting
turned off system wide when using this approach is because:

* an end user is running a redistributor provided version of CPython rather
  than running upstream CPython directly
* that redistributor has decided to provide a smoother migration path to
  verifying HTTPS certificates by default than that being provided by the
  upstream project
* either the redistributor or the local infrastructure administrator has
  determined that it is appropriate to override the default upstream behaviour
  (at least for the time being)

Using an administrator controlled configuration file rather than an environment
variable has the essential feature of providing a smoother migration path, even
for applications being run with the ``-E`` switch.


Combining the recommendations
=============================

If a redistributor chooses to implement both recommendations, then the
environment variable should take precedence over the system-wide configuration
setting. This allows the setting to be changed for a given user, virtual
environment or application, regardless of the system-wide default behaviour.

In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and
set to anything *other* than ``'0'``, then HTTPS certificate verification
should be enabled.

Example implementation
----------------------

::

    _https_verify_envvar = 'PYTHONHTTPSVERIFY'
    _cert_verification_config = '/etc/python/cert-verification.cfg'

    def _get_https_context_factory():
        # Check for am environmental override of the default behaviour
        config_setting = os.environ.get(_https_verify_envvar)
        if config_setting is not None:
            if config_setting == '0':
                return _create_unverified_context
            return create_default_context

        # Check for a system-wide override of the default behaviour
        context_factories = {
            'enable': create_default_context,
            'disable': _create_unverified_context,
            'platform_default': _create_unverified_context, # For now :)
        }
        import ConfigParser
        config = ConfigParser.RawConfigParser()
        config.read(_cert_verification_config)
        try:
            verify_mode = config.get('https', 'verify')
        except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
            verify_mode = 'platform_default'
        default_factory = context_factories.get('platform_default')
        return context_factories.get(verify_mode, default_factory)

    _create_default_https_context = _get_https_context_factory()


Copyright
=========

This document has been placed into the public domain.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From motoom at xs4all.nl  Tue Nov 10 21:07:28 2015
From: motoom at xs4all.nl (Michiel Overtoom)
Date: Wed, 11 Nov 2015 03:07:28 +0100
Subject: [Python-Dev] modernizing IDLE
In-Reply-To: <CAO41-mM2kTxLB0chwrpoTKVFRxyyLoOZeAOq_zUvw=czgox4OQ@mail.gmail.com>
References: <A135E83B-E078-482C-B062-D4C07BC6A30C@markroseman.com>
 <25544A86-7073-47F6-9189-F734D5007B01@xs4all.nl>
 <CAO41-mM2kTxLB0chwrpoTKVFRxyyLoOZeAOq_zUvw=czgox4OQ@mail.gmail.com>
Message-ID: <CA51A0E9-D7F4-4819-8B56-0C7D3C545478@xs4all.nl>


> On 11 Nov 2015, at 00:23, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> 
> https://news.ycombinator.com/item?id=10519785

Thanks for the URL.

I consider myself a reasonably seasoned IDE user, having used Visual Studio, emacs with SLIME, Borland C++Builder, Xcode, Delphi, Eclipse, LightTable and PyCharm, and probably I forgot some.

However, I still use IDLE regularly on Windows and Unix systems (not on OSX). I don't care much about advanced IDE features. Basically I want a programmers' text editor with minimal syntax highlighting and a shortcut key like F9 or CMD+R to run my program. When developing webapps, I rely on the 'save' function of the editor and having the web framework reload the sources automatically while in developer mode.  For debugging, I use Python's 'pdb' or the more fancier 'pudb' where appropriate.

I really like to see IDLE stay as part of the Python installation. And I'm sure, with all the attention it gets nowadays, that it'll stay a minimal but usable basic IDE for developing Python programs.

Greetings,

From projetmbc at gmail.com  Wed Nov 11 10:13:03 2015
From: projetmbc at gmail.com (Christophe Bal)
Date: Wed, 11 Nov 2015 16:13:03 +0100
Subject: [Python-Dev] Translate Python language
Message-ID: <CAAb4jG=RkFVkJRhzhGYX2u0bzuUceYvew=OB7sQ4HjweKoaEqw@mail.gmail.com>

Hello.

I'm a french teacher and I would like to use Python with young child but
I've a big problem. All the keyword are in english. So I would like to know
if there is a way to hack the AST tools such as to use french keywords and
then translate a list of french keywords to their english regular version.

Where should I start ? My idea is not to build a new language but simply
translate words using an hacked version of the AST tools.

Hoping to be clear and I do not pollute the message in this list.

*Christophe BAL*
*Enseignant de math?matiques en Lyc?e **et d?veloppeur Python amateur*
*---*
*French math teacher in a "Lyc?e" **and **Python **amateur developer*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151111/baaf4344/attachment-0001.html>

From p.f.moore at gmail.com  Wed Nov 11 11:17:33 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 11 Nov 2015 16:17:33 +0000
Subject: [Python-Dev] Translate Python language
In-Reply-To: <CAAb4jG=RkFVkJRhzhGYX2u0bzuUceYvew=OB7sQ4HjweKoaEqw@mail.gmail.com>
References: <CAAb4jG=RkFVkJRhzhGYX2u0bzuUceYvew=OB7sQ4HjweKoaEqw@mail.gmail.com>
Message-ID: <CACac1F9GR+Tt+WRrbMYJn=DCGD0GaGqY+1BH0hw=m2EtzY1_xA@mail.gmail.com>

On 11 November 2015 at 15:13, Christophe Bal <projetmbc at gmail.com> wrote:
> Hello.
>
> I'm a french teacher and I would like to use Python with young child but
> I've a big problem. All the keyword are in english. So I would like to know
> if there is a way to hack the AST tools such as to use french keywords and
> then translate a list of french keywords to their english regular version.
>
> Where should I start ? My idea is not to build a new language but simply
> translate words using an hacked version of the AST tools.
>
> Hoping to be clear and I do not pollute the message in this list.

Do you really just mean keywords? Most of the English words used in
Python (for example "open") aren't actually keywords but are names of
functions defined in the builtins and/or standard library. Those are
not exposed as distinct objects in the AST, but are simply name
lookups.

If you do mean just keywords, then you could probably relatively
simply hack the Python interpreter code to use different names for the
keywords, and build your own translated copy of Python. I don't know
the details, though. But in practice this wouldn't really help your
student, as there would still be many English words they would
encounter and need to understand (at least in the context of the code
they are writing).

My understanding is that most non-English speaking teachers tend to
accept that their students simply need to treat the keywords and
function names used in typical programming languages (Python included)
as things they just have to learn. But I suspect most students are
typically older than your young child.

Sorry I can't be more help.
Paul

From donald at stufft.io  Wed Nov 11 11:20:04 2015
From: donald at stufft.io (Donald Stufft)
Date: Wed, 11 Nov 2015 11:20:04 -0500
Subject: [Python-Dev] Translate Python language
In-Reply-To: <CACac1F9GR+Tt+WRrbMYJn=DCGD0GaGqY+1BH0hw=m2EtzY1_xA@mail.gmail.com>
References: <CAAb4jG=RkFVkJRhzhGYX2u0bzuUceYvew=OB7sQ4HjweKoaEqw@mail.gmail.com>
 <CACac1F9GR+Tt+WRrbMYJn=DCGD0GaGqY+1BH0hw=m2EtzY1_xA@mail.gmail.com>
Message-ID: <etPan.56436ab4.4f9d23e3.136c2@Draupnir.local>

On November 11, 2015 at 11:19:07 AM, Paul Moore (p.f.moore at gmail.com) wrote:
> On 11 November 2015 at 15:13, Christophe Bal wrote:
> > Hello.
> >
> > I'm a french teacher and I would like to use Python with young child but
> > I've a big problem. All the keyword are in english. So I would like to know
> > if there is a way to hack the AST tools such as to use french keywords and
> > then translate a list of french keywords to their english regular version.
> >
> > Where should I start ? My idea is not to build a new language but simply
> > translate words using an hacked version of the AST tools.
> >
> > Hoping to be clear and I do not pollute the message in this list.
> 
> Do you really just mean keywords? Most of the English words used in
> Python (for example "open") aren't actually keywords but are names of
> functions defined in the builtins and/or standard library. Those are
> not exposed as distinct objects in the AST, but are simply name
> lookups.
> 

You might be able to use an encoding to do this.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



From brett at python.org  Wed Nov 11 14:31:52 2015
From: brett at python.org (Brett Cannon)
Date: Wed, 11 Nov 2015 19:31:52 +0000
Subject: [Python-Dev] Translate Python language
In-Reply-To: <CAAb4jG=RkFVkJRhzhGYX2u0bzuUceYvew=OB7sQ4HjweKoaEqw@mail.gmail.com>
References: <CAAb4jG=RkFVkJRhzhGYX2u0bzuUceYvew=OB7sQ4HjweKoaEqw@mail.gmail.com>
Message-ID: <CAP1=2W7Tgk+UT9jeptxrD1VTcFzwQUhDB1r1v4TLpmTxKQXGnQ@mail.gmail.com>

On Wed, 11 Nov 2015, 10:55 Christophe Bal <projetmbc at gmail.com> wrote:

Hello.

I'm a french teacher and I would like to use Python with young child but
I've a big problem. All the keyword are in english. So I would like to know
if there is a way to hack the AST tools such as to use french keywords and
then translate a list of french keywords to their english regular version.

Where should I start ? My idea is not to build a new language but simply
translate words using an hacked version of the AST tools.

You actually need a new grammar for the parser as the AST comes from that.
Then you may be able to build the AST.

-brett


Hoping to be clear and I do not pollute the message in this list.

*Christophe BAL*

*Enseignant** de **math?matiques* *en** Lyc?e **et **d?veloppeur** Python
amateur*

*---*

*French math teacher in a "Lyc?e" **and **Python **amateur developer*

_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https <https://mail.python.org/mailman/listinfo/python-dev>://
<https://mail.python.org/mailman/listinfo/python-dev>mail.python.org
<https://mail.python.org/mailman/listinfo/python-dev>/mailman/
<https://mail.python.org/mailman/listinfo/python-dev>listinfo
<https://mail.python.org/mailman/listinfo/python-dev>/
<https://mail.python.org/mailman/listinfo/python-dev>python-dev
<https://mail.python.org/mailman/listinfo/python-dev>
Unsubscribe: https
<https://mail.python.org/mailman/options/python-dev/brett%40python.org>://
<https://mail.python.org/mailman/options/python-dev/brett%40python.org>
mail.python.org
<https://mail.python.org/mailman/options/python-dev/brett%40python.org>
/mailman/options/
<https://mail.python.org/mailman/options/python-dev/brett%40python.org>
python-dev
<https://mail.python.org/mailman/options/python-dev/brett%40python.org>/
<https://mail.python.org/mailman/options/python-dev/brett%40python.org>brett
<https://mail.python.org/mailman/options/python-dev/brett%40python.org>
%40python.org
<https://mail.python.org/mailman/options/python-dev/brett%40python.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151111/42b0e5aa/attachment.html>

From nirsof at gmail.com  Wed Nov 11 14:47:11 2015
From: nirsof at gmail.com (Nir Soffer)
Date: Wed, 11 Nov 2015 21:47:11 +0200
Subject: [Python-Dev] Translate Python language
In-Reply-To: <CAAb4jG=RkFVkJRhzhGYX2u0bzuUceYvew=OB7sQ4HjweKoaEqw@mail.gmail.com>
References: <CAAb4jG=RkFVkJRhzhGYX2u0bzuUceYvew=OB7sQ4HjweKoaEqw@mail.gmail.com>
Message-ID: <CAMr-obtn6JpLva=iM6+n84E1-n3S9z+U+eVmae-6O+ehetocdA@mail.gmail.com>

On Wed, Nov 11, 2015 at 5:13 PM, Christophe Bal <projetmbc at gmail.com> wrote:

> Hello.
>
> I'm a french teacher and I would like to use Python with young child but
> I've a big problem. All the keyword are in english. So I would like to know
> if there is a way to hack the AST tools such as to use french keywords and
> then translate a list of french keywords to their english regular version.
>
> Where should I start ? My idea is not to build a new language but simply
> translate words using an hacked version of the AST tools.
>
> Hoping to be clear and I do not pollute the message in this list.
>

I did this in 2006 for Hebrew - we had Hebrew Python dialect, with
Hebrew turtle module and interactive shell.

You can try to reuse this:
https://github.com/nirs/hpy

Nir


>
>
> *Christophe BAL*
> *Enseignant de math?matiques en Lyc?e **et d?veloppeur Python amateur*
> *---*
> *French math teacher in a "Lyc?e" **and **Python **amateur developer*
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/nirsof%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151111/7a1ffd81/attachment.html>

From mal at egenix.com  Wed Nov 11 14:55:23 2015
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 11 Nov 2015 20:55:23 +0100
Subject: [Python-Dev] Translate Python language
In-Reply-To: <etPan.56436ab4.4f9d23e3.136c2@Draupnir.local>
References: <CAAb4jG=RkFVkJRhzhGYX2u0bzuUceYvew=OB7sQ4HjweKoaEqw@mail.gmail.com>
 <CACac1F9GR+Tt+WRrbMYJn=DCGD0GaGqY+1BH0hw=m2EtzY1_xA@mail.gmail.com>
 <etPan.56436ab4.4f9d23e3.136c2@Draupnir.local>
Message-ID: <56439D2B.5060707@egenix.com>

On 11.11.2015 17:20, Donald Stufft wrote:
> On November 11, 2015 at 11:19:07 AM, Paul Moore (p.f.moore at gmail.com) wrote:
>> On 11 November 2015 at 15:13, Christophe Bal wrote:
>>> Hello.
>>>
>>> I'm a french teacher and I would like to use Python with young child but
>>> I've a big problem. All the keyword are in english. So I would like to know
>>> if there is a way to hack the AST tools such as to use french keywords and
>>> then translate a list of french keywords to their english regular version.
>>>
>>> Where should I start ? My idea is not to build a new language but simply
>>> translate words using an hacked version of the AST tools.
>>>
>>> Hoping to be clear and I do not pollute the message in this list.
>>
>> Do you really just mean keywords? Most of the English words used in
>> Python (for example "open") aren't actually keywords but are names of
>> functions defined in the builtins and/or standard library. Those are
>> not exposed as distinct objects in the AST, but are simply name
>> lookups.
>>
> 
> You might be able to use an encoding to do this.

Or use Andrew Dalke's fun project LOLPython as basis:

http://www.dalkescientific.com/writings/diary/archive/2007/06/01/lolpython.html

It uses a parser which then translates the source code
into regular Python source code, which it then runs.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 11 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________
2015-10-23: Released mxODBC Connect 2.1.5 ...     http://egenix.com/go85

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From benjamin at python.org  Thu Nov 12 01:50:05 2015
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 11 Nov 2015 22:50:05 -0800
Subject: [Python-Dev] [Python-checkins] cpython (2.7): Backport
 early-out 91259f061cfb to reduce the cost of bb1a2944bcb6
In-Reply-To: <20151007031207.18390.20414@psf.io>
References: <20151007031207.18390.20414@psf.io>
Message-ID: <1447311005.2650788.436807905.05A186FE@webmail.messagingengine.com>



On Tue, Oct 6, 2015, at 19:12, raymond.hettinger wrote:
> https://hg.python.org/cpython/rev/37aee118e1a3
> changeset:   98578:37aee118e1a3
> branch:      2.7
> parent:      98572:60c44a09c5fc
> user:        Raymond Hettinger <python at rcn.com>
> date:        Tue Oct 06 23:12:02 2015 -0400
> summary:
>   Backport early-out 91259f061cfb to reduce the cost of bb1a2944bcb6
> 
> files:
>   Modules/_collectionsmodule.c |  6 +++++-
>   1 files changed, 5 insertions(+), 1 deletions(-)
> 
> 
> diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c
> --- a/Modules/_collectionsmodule.c
> +++ b/Modules/_collectionsmodule.c
> @@ -651,6 +651,9 @@
>      Py_ssize_t n;
>      PyObject *item;
>  
> +    if (Py_SIZE(deque) == 0)
> +        return;
> +

dequeue is not varsized in Python 2.7, so using Py_SIZE() is incorrect.

From python at rcn.com  Thu Nov 12 10:20:29 2015
From: python at rcn.com (Raymond Hettinger)
Date: Thu, 12 Nov 2015 07:20:29 -0800
Subject: [Python-Dev] [Python-checkins] cpython (2.7): Backport
 early-out 91259f061cfb to reduce the cost of bb1a2944bcb6
In-Reply-To: <1447311005.2650788.436807905.05A186FE@webmail.messagingengine.com>
References: <20151007031207.18390.20414@psf.io>
 <1447311005.2650788.436807905.05A186FE@webmail.messagingengine.com>
Message-ID: <305A353F-82F7-4A6C-A9E5-235808CBA966@rcn.com>


> On Nov 11, 2015, at 10:50 PM, Benjamin Peterson <benjamin at python.org> wrote:
> 
>> +    if (Py_SIZE(deque) == 0)
>> +        return;
>> +
> 
> dequeue is not varsized in Python 2.7, so using Py_SIZE() is incorrect.

Fixed in a2a518b6ded4.

-    if (Py_SIZE(deque) == 0)
+    if (deque->len == 0)


Raymond

From benjamin at python.org  Thu Nov 12 15:13:10 2015
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 12 Nov 2015 12:13:10 -0800
Subject: [Python-Dev] [Python-checkins] cpython (2.7): Backport
 early-out 91259f061cfb to reduce the cost of bb1a2944bcb6
In-Reply-To: <305A353F-82F7-4A6C-A9E5-235808CBA966@rcn.com>
References: <20151007031207.18390.20414@psf.io>
 <1447311005.2650788.436807905.05A186FE@webmail.messagingengine.com>
 <305A353F-82F7-4A6C-A9E5-235808CBA966@rcn.com>
Message-ID: <1447359190.2841964.438218129.0466D7F0@webmail.messagingengine.com>

There's two instances in that file.

On Thu, Nov 12, 2015, at 07:20, Raymond Hettinger wrote:
> 
> > On Nov 11, 2015, at 10:50 PM, Benjamin Peterson <benjamin at python.org> wrote:
> > 
> >> +    if (Py_SIZE(deque) == 0)
> >> +        return;
> >> +
> > 
> > dequeue is not varsized in Python 2.7, so using Py_SIZE() is incorrect.
> 
> Fixed in a2a518b6ded4.
> 
> -    if (Py_SIZE(deque) == 0)
> +    if (deque->len == 0)
> 
> 
> Raymond

From ethan at stoneleaf.us  Fri Nov 13 10:12:32 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 13 Nov 2015 07:12:32 -0800
Subject: [Python-Dev] doc tests failing
Message-ID: <5645FDE0.9040304@stoneleaf.us>

What am I doing wrong?

I have tried:

--------------------------------------------------------
hg update 3.5  # and hg update default
make distclean && ./configure --with-pydebug && make -j2
cd Doc
make doctest
--------------------------------------------------------

and in both cases I get page after page of errors.  I have tried 
installing python-sphinx and python3-sphinx; I have tried adding 
PYTHON=../python and PYTHON=python3 to the `make doctest` line -- all to 
no avail.

Here's a random sample of the errors:

**********************************************************************
File "library/shlex.rst", line ?, in default
Failed example:
     remote_command
Exception raised:
     Traceback (most recent call last):
       File "/usr/lib/python2.7/doctest.py", line 1315, in __run
         compileflags, 1) in test.globs
       File "<doctest default[1]>", line 1, in <module>
         remote_command
     NameError: name 'remote_command' is not defined

**********************************************************************
File "howto/sorting.rst", line ?, in default
Failed example:
     sorted([5, 2, 4, 1, 3], cmp=numeric_compare)
Exception raised:
     Traceback (most recent call last):
       File "/usr/lib/python2.7/doctest.py", line 1315, in __run
         compileflags, 1) in test.globs
       File "<doctest default[1]>", line 1, in <module>
         sorted([5, 2, 4, 1, 3], cmp=numeric_compare)
     NameError: name 'numeric_compare' is not defined

**********************************************************************
File "library/ipaddress.rst", line ?, in default
Failed example:
     n2 = ip_network('192.0.2.1/32')
Exception raised:
     Traceback (most recent call last):
       File "/usr/lib/python2.7/doctest.py", line 1315, in __run
         compileflags, 1) in test.globs
       File "<doctest default[1]>", line 1, in <module>
         n2 = ip_network('192.0.2.1/32')
     NameError: name 'ip_network' is not defined

--
~Ethan~

From g.brandl at gmx.net  Fri Nov 13 10:40:16 2015
From: g.brandl at gmx.net (Georg Brandl)
Date: Fri, 13 Nov 2015 16:40:16 +0100
Subject: [Python-Dev] doc tests failing
In-Reply-To: <5645FDE0.9040304@stoneleaf.us>
References: <5645FDE0.9040304@stoneleaf.us>
Message-ID: <n25090$ib8$1@ger.gmane.org>

On 11/13/2015 04:12 PM, Ethan Furman wrote:
> What am I doing wrong?

Running "make doctest" in the first place. The doctests in the core docs
were not written to support that, and also never updated to do so.

Georg

> I have tried:
> 
> --------------------------------------------------------
> hg update 3.5  # and hg update default
> make distclean && ./configure --with-pydebug && make -j2
> cd Doc
> make doctest
> --------------------------------------------------------
> 
> and in both cases I get page after page of errors.  I have tried 
> installing python-sphinx and python3-sphinx; I have tried adding 
> PYTHON=../python and PYTHON=python3 to the `make doctest` line -- all to 
> no avail.
> 
> Here's a random sample of the errors:
> 
> **********************************************************************
> File "library/shlex.rst", line ?, in default
> Failed example:
>      remote_command
> Exception raised:
>      Traceback (most recent call last):
>        File "/usr/lib/python2.7/doctest.py", line 1315, in __run
>          compileflags, 1) in test.globs
>        File "<doctest default[1]>", line 1, in <module>
>          remote_command
>      NameError: name 'remote_command' is not defined
> 
> **********************************************************************
> File "howto/sorting.rst", line ?, in default
> Failed example:
>      sorted([5, 2, 4, 1, 3], cmp=numeric_compare)
> Exception raised:
>      Traceback (most recent call last):
>        File "/usr/lib/python2.7/doctest.py", line 1315, in __run
>          compileflags, 1) in test.globs
>        File "<doctest default[1]>", line 1, in <module>
>          sorted([5, 2, 4, 1, 3], cmp=numeric_compare)
>      NameError: name 'numeric_compare' is not defined
> 
> **********************************************************************
> File "library/ipaddress.rst", line ?, in default
> Failed example:
>      n2 = ip_network('192.0.2.1/32')
> Exception raised:
>      Traceback (most recent call last):
>        File "/usr/lib/python2.7/doctest.py", line 1315, in __run
>          compileflags, 1) in test.globs
>        File "<doctest default[1]>", line 1, in <module>
>          n2 = ip_network('192.0.2.1/32')
>      NameError: name 'ip_network' is not defined
> 
> --
> ~Ethan~
> 



From zachary.ware+pydev at gmail.com  Fri Nov 13 10:42:41 2015
From: zachary.ware+pydev at gmail.com (Zachary Ware)
Date: Fri, 13 Nov 2015 09:42:41 -0600
Subject: [Python-Dev] doc tests failing
In-Reply-To: <5645FDE0.9040304@stoneleaf.us>
References: <5645FDE0.9040304@stoneleaf.us>
Message-ID: <CAKJDb-M9qtjcde3EPYQNFVHStQ5EwnMTc4ksqnFydvSvbTjEUg@mail.gmail.com>

On Fri, Nov 13, 2015 at 9:12 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> What am I doing wrong?

Expecting it to work :)

`make doctest` is a Sphinx feature, not specific to our docs.  Ideally
someday it should work (and I'll add it to the Docs buildbot), but
nobody has tried to make it work yet.  Making it work might be a good
large-ish project for newer contributors.

-- 
Zach

From rdmurray at bitdance.com  Fri Nov 13 10:44:26 2015
From: rdmurray at bitdance.com (R. David Murray)
Date: Fri, 13 Nov 2015 10:44:26 -0500
Subject: [Python-Dev] doc tests failing
In-Reply-To: <5645FDE0.9040304@stoneleaf.us>
References: <5645FDE0.9040304@stoneleaf.us>
Message-ID: <20151113154427.07A63B18088@webabinitio.net>

We don't have clean doctests for the docs.  Patches welcome.

At one point I had made the turtle doctests pass (it draws a bunch of
stuff on the screen) because otherwise we don't have very many turtle
tests, but I haven't checked it in a couple years.

Hmm.  We could list making the doc doctests pass as an activity for
beginners in the devguide.

On Fri, 13 Nov 2015 07:12:32 -0800, Ethan Furman <ethan at stoneleaf.us> wrote:
> What am I doing wrong?
> 
> I have tried:
> 
> --------------------------------------------------------
> hg update 3.5  # and hg update default
> make distclean && ./configure --with-pydebug && make -j2
> cd Doc
> make doctest
> --------------------------------------------------------
> 
> and in both cases I get page after page of errors.  I have tried 
> installing python-sphinx and python3-sphinx; I have tried adding 
> PYTHON=../python and PYTHON=python3 to the `make doctest` line -- all to 
> no avail.
> 
> Here's a random sample of the errors:
> 
> **********************************************************************
> File "library/shlex.rst", line ?, in default
> Failed example:
>      remote_command
> Exception raised:
>      Traceback (most recent call last):
>        File "/usr/lib/python2.7/doctest.py", line 1315, in __run
>          compileflags, 1) in test.globs
>        File "<doctest default[1]>", line 1, in <module>
>          remote_command
>      NameError: name 'remote_command' is not defined
> 
> **********************************************************************
> File "howto/sorting.rst", line ?, in default
> Failed example:
>      sorted([5, 2, 4, 1, 3], cmp=numeric_compare)
> Exception raised:
>      Traceback (most recent call last):
>        File "/usr/lib/python2.7/doctest.py", line 1315, in __run
>          compileflags, 1) in test.globs
>        File "<doctest default[1]>", line 1, in <module>
>          sorted([5, 2, 4, 1, 3], cmp=numeric_compare)
>      NameError: name 'numeric_compare' is not defined
> 
> **********************************************************************
> File "library/ipaddress.rst", line ?, in default
> Failed example:
>      n2 = ip_network('192.0.2.1/32')
> Exception raised:
>      Traceback (most recent call last):
>        File "/usr/lib/python2.7/doctest.py", line 1315, in __run
>          compileflags, 1) in test.globs
>        File "<doctest default[1]>", line 1, in <module>
>          n2 = ip_network('192.0.2.1/32')
>      NameError: name 'ip_network' is not defined
> 
> --
> ~Ethan~
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com

From status at bugs.python.org  Fri Nov 13 12:08:33 2015
From: status at bugs.python.org (Python tracker)
Date: Fri, 13 Nov 2015 18:08:33 +0100 (CET)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20151113170833.E47ED56881@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2015-11-06 - 2015-11-13)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5232 (+14)
  closed 32143 (+35)
  total  37375 (+49)

Open issues with patches: 2307 


Issues opened (36)
==================

#7759: mhlib fails on Btrfs filesystem (test_mhlib failure)
http://bugs.python.org/issue7759  reopened by serhiy.storchaka

#25570: urllib.request > Request.add_header("abcd","efgh") fails with 
http://bugs.python.org/issue25570  opened by crickert

#25571: Improve the lltrace feature with the Py_Debug mode
http://bugs.python.org/issue25571  opened by matrixise

#25572: _ssl doesn't build on OSX 10.11
http://bugs.python.org/issue25572  opened by matrixise

#25573: FrameSummary repr() does not support previously working uses o
http://bugs.python.org/issue25573  opened by zopieux

#25574: 2.7 incorrectly documents objects hash as equal to id
http://bugs.python.org/issue25574  opened by wim.glenn

#25576: Remove ???Content-Type: application/x-www-form-urlencoded; cha
http://bugs.python.org/issue25576  opened by martin.panter

#25578: Memory leak in SSLSocket.getpeercert() with 0-length AIA exten
http://bugs.python.org/issue25578  opened by alex

#25580: async and await missing from token list
http://bugs.python.org/issue25580  opened by SilentGhost

#25582: Fixed memory leaks in test_ctypes
http://bugs.python.org/issue25582  opened by serhiy.storchaka

#25583: os.makedirs with exist_ok=True raises PermissionError on Windo
http://bugs.python.org/issue25583  opened by plachotich

#25585: Bad path leads to: ImportError: DLL load failed: %1 is not a v
http://bugs.python.org/issue25585  opened by lac

#25586: socket.sendall broken when a socket has a timeout
http://bugs.python.org/issue25586  opened by jstasiak

#25588: Run test suite from IDLE idlelib.run subprocess
http://bugs.python.org/issue25588  opened by terry.reedy

#25590: tab-completition on instances repeatedly accesses attribute/de
http://bugs.python.org/issue25590  opened by ezio.melotti

#25591: improve test coverage for the imaplib
http://bugs.python.org/issue25591  opened by maciej.szulik

#25592: distutils docs: data_files always uses sys.prefix
http://bugs.python.org/issue25592  opened by jdemeyer

#25593: _sock_connect_cb can be called twice resulting in InvalidState
http://bugs.python.org/issue25593  opened by thehesiod

#25594: enum instance attribute access possible
http://bugs.python.org/issue25594  opened by SilentGhost

#25596: regular files handled as directories in the glob module
http://bugs.python.org/issue25596  opened by xdegaye

#25597: unittest.mock does not wrap dict objects correctly
http://bugs.python.org/issue25597  opened by Darragh Bailey

#25599: asyncio.iscoroutinefunction returns unexpected results when pr
http://bugs.python.org/issue25599  opened by luhn

#25601: test_cpickle failure on the ware-gentoo-x86 buildbot
http://bugs.python.org/issue25601  opened by serhiy.storchaka

#25602: Add support for EVFILT_USER kqueue filter
http://bugs.python.org/issue25602  opened by jceel

#25604: [Minor] bug in integer true division algorithm
http://bugs.python.org/issue25604  opened by mark.dickinson

#25606: asyncio doc: 'socket' transport extra info is not mandatory
http://bugs.python.org/issue25606  opened by haypo

#25608: ascynio readexactly() should raise ValueError if passed length
http://bugs.python.org/issue25608  opened by mmarkk

#25609: Add a ContextManager ABC and type
http://bugs.python.org/issue25609  opened by brett.cannon

#25610: Add typing.Awaitable
http://bugs.python.org/issue25610  opened by brett.cannon

#25611: test_deque failure on Gentoo and OpenIndiana buildbots on 2.7
http://bugs.python.org/issue25611  opened by serhiy.storchaka

#25612: nested try..excepts don't work correctly for generators
http://bugs.python.org/issue25612  opened by yselivanov

#25613: fix ssl tests with sslv3 disabled
http://bugs.python.org/issue25613  opened by doko

#25614: Lib/code.py: InteractiveConsole.raw_input writes prompt to std
http://bugs.python.org/issue25614  opened by Yclept.Nemo

#25615: Document unsorted behaviour of glob.glob
http://bugs.python.org/issue25615  opened by Dave Jones

#25616: Extract OrderedDict tests into separate file
http://bugs.python.org/issue25616  opened by serhiy.storchaka

#25617: Installing local installation of Python
http://bugs.python.org/issue25617  opened by koustavpal88



Most recent 15 issues with no replies (15)
==========================================

#25617: Installing local installation of Python
http://bugs.python.org/issue25617

#25616: Extract OrderedDict tests into separate file
http://bugs.python.org/issue25616

#25615: Document unsorted behaviour of glob.glob
http://bugs.python.org/issue25615

#25614: Lib/code.py: InteractiveConsole.raw_input writes prompt to std
http://bugs.python.org/issue25614

#25611: test_deque failure on Gentoo and OpenIndiana buildbots on 2.7
http://bugs.python.org/issue25611

#25610: Add typing.Awaitable
http://bugs.python.org/issue25610

#25602: Add support for EVFILT_USER kqueue filter
http://bugs.python.org/issue25602

#25592: distutils docs: data_files always uses sys.prefix
http://bugs.python.org/issue25592

#25591: improve test coverage for the imaplib
http://bugs.python.org/issue25591

#25588: Run test suite from IDLE idlelib.run subprocess
http://bugs.python.org/issue25588

#25578: Memory leak in SSLSocket.getpeercert() with 0-length AIA exten
http://bugs.python.org/issue25578

#25574: 2.7 incorrectly documents objects hash as equal to id
http://bugs.python.org/issue25574

#25566: asyncio reference cycles after ConnectionResetError
http://bugs.python.org/issue25566

#25560: Unhandled warning in test_unicode_file
http://bugs.python.org/issue25560

#25551: Event's test_reset_internal_locks too fragile
http://bugs.python.org/issue25551



Most recent 15 issues waiting for review (15)
=============================================

#25615: Document unsorted behaviour of glob.glob
http://bugs.python.org/issue25615

#25612: nested try..excepts don't work correctly for generators
http://bugs.python.org/issue25612

#25602: Add support for EVFILT_USER kqueue filter
http://bugs.python.org/issue25602

#25596: regular files handled as directories in the glob module
http://bugs.python.org/issue25596

#25594: enum instance attribute access possible
http://bugs.python.org/issue25594

#25593: _sock_connect_cb can be called twice resulting in InvalidState
http://bugs.python.org/issue25593

#25592: distutils docs: data_files always uses sys.prefix
http://bugs.python.org/issue25592

#25591: improve test coverage for the imaplib
http://bugs.python.org/issue25591

#25590: tab-completition on instances repeatedly accesses attribute/de
http://bugs.python.org/issue25590

#25586: socket.sendall broken when a socket has a timeout
http://bugs.python.org/issue25586

#25583: os.makedirs with exist_ok=True raises PermissionError on Windo
http://bugs.python.org/issue25583

#25582: Fixed memory leaks in test_ctypes
http://bugs.python.org/issue25582

#25580: async and await missing from token list
http://bugs.python.org/issue25580

#25578: Memory leak in SSLSocket.getpeercert() with 0-length AIA exten
http://bugs.python.org/issue25578

#25576: Remove ???Content-Type: application/x-www-form-urlencoded; cha
http://bugs.python.org/issue25576



Top 10 most discussed issues (10)
=================================

#25593: _sock_connect_cb can be called twice resulting in InvalidState
http://bugs.python.org/issue25593  27 msgs

#25570: urllib.request > Request.add_header("abcd","efgh") fails with 
http://bugs.python.org/issue25570  12 msgs

#25583: os.makedirs with exist_ok=True raises PermissionError on Windo
http://bugs.python.org/issue25583  12 msgs

#25586: socket.sendall broken when a socket has a timeout
http://bugs.python.org/issue25586  12 msgs

#25585: Bad path leads to: ImportError: DLL load failed: %1 is not a v
http://bugs.python.org/issue25585   8 msgs

#7759: mhlib fails on Btrfs filesystem (test_mhlib failure)
http://bugs.python.org/issue7759   7 msgs

#24999: ICC compiler: ICC treats denormal floating point numbers as 0.
http://bugs.python.org/issue24999   7 msgs

#25590: tab-completition on instances repeatedly accesses attribute/de
http://bugs.python.org/issue25590   7 msgs

#23883: __all__ lists are incomplete
http://bugs.python.org/issue23883   6 msgs

#25179: PEP 498 f-strings need to be documented
http://bugs.python.org/issue25179   6 msgs



Issues closed (34)
==================

#6598: calling email.utils.make_msgid frequently has a non-trivial pr
http://bugs.python.org/issue6598  closed by serhiy.storchaka

#7267: format method: c presentation type broken in 2.7
http://bugs.python.org/issue7267  closed by haypo

#14350: Strange Exception from copying an iterator
http://bugs.python.org/issue14350  closed by serhiy.storchaka

#18010: pydoc search chokes on import errors
http://bugs.python.org/issue18010  closed by martin.panter

#21935: Implement AUTH command in smtpd.
http://bugs.python.org/issue21935  closed by r.david.murray

#24306: Backport py.exe to 3.4
http://bugs.python.org/issue24306  closed by steve.dower

#24661: CGIHTTPServer: premature unescaping of query string
http://bugs.python.org/issue24661  closed by martin.panter

#24802: int and float constructing from non NUL-terminated buffer
http://bugs.python.org/issue24802  closed by martin.panter

#24805: Python installer having problem in installing Python for all u
http://bugs.python.org/issue24805  closed by steve.dower

#24848: Warts in UTF-7 error handling
http://bugs.python.org/issue24848  closed by serhiy.storchaka

#25064: Adjust tempfile documentation for bytes filename support
http://bugs.python.org/issue25064  closed by martin.panter

#25263: test_tkinter fails randomly on the buildbots "AMD64 Windows10"
http://bugs.python.org/issue25263  closed by serhiy.storchaka

#25313: IDLE: gracefully handle themes (or keysets, or ...) not presen
http://bugs.python.org/issue25313  closed by terry.reedy

#25446: smtplib.py AUTH LOGIN code messed up sending login and passwor
http://bugs.python.org/issue25446  closed by r.david.murray

#25462: Avoid repeated hash calculation in C implementation of Ordered
http://bugs.python.org/issue25462  closed by serhiy.storchaka

#25465: Pickle uses O(n) memory overhead
http://bugs.python.org/issue25465  closed by serhiy.storchaka

#25530: ssl: OP_NO_SSLv3 should always be set unless a user specifical
http://bugs.python.org/issue25530  closed by python-dev

#25555: Fix parser and AST: fill lineno and col_offset when compiling 
http://bugs.python.org/issue25555  closed by haypo

#25558: Use static asserts in C code
http://bugs.python.org/issue25558  closed by serhiy.storchaka

#25569: Memory leak in SSLSocket.getpeercert()
http://bugs.python.org/issue25569  closed by python-dev

#25575: idle not working
http://bugs.python.org/issue25575  closed by zach.ware

#25577: os.path.dirname leave left quote and remove right one
http://bugs.python.org/issue25577  closed by zach.ware

#25579: def is not a keyword with tokenize.py
http://bugs.python.org/issue25579  closed by SilentGhost

#25581: Python -U raises error during site import
http://bugs.python.org/issue25581  closed by r.david.murray

#25584: a recursive glob pattern fails to list files in the current di
http://bugs.python.org/issue25584  closed by serhiy.storchaka

#25587: Useless addition in PyTuple_New
http://bugs.python.org/issue25587  closed by haypo

#25589: test_ascii_formatd fails on Mac when built with Intel compiler
http://bugs.python.org/issue25589  closed by christopher.hogan

#25595: test_deleted_cwd in test_importlib is failed on AIX
http://bugs.python.org/issue25595  closed by serhiy.storchaka

#25598: Fix memory_hex (#9951) for non-contiguous buffers
http://bugs.python.org/issue25598  closed by skrah

#25600: argparse, argument_default=argparse.SUPPRESS seems to have no 
http://bugs.python.org/issue25600  closed by r.david.murray

#25603: spelling mistake - 26.1 typing
http://bugs.python.org/issue25603  closed by python-dev

#25605: fcntl doc: document exceptions raised on error for ioctl() and
http://bugs.python.org/issue25605  closed by haypo

#25607: test_distutils provokes unexpected output in test_shutil
http://bugs.python.org/issue25607  closed by serhiy.storchaka

#892902: problem with pickling newstyle class instances
http://bugs.python.org/issue892902  closed by serhiy.storchaka

From ethan at stoneleaf.us  Fri Nov 13 13:01:40 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 13 Nov 2015 10:01:40 -0800
Subject: [Python-Dev] doc tests failing
In-Reply-To: <5645FDE0.9040304@stoneleaf.us>
References: <5645FDE0.9040304@stoneleaf.us>
Message-ID: <56462584.1030100@stoneleaf.us>

On 11/13/2015 07:12 AM, Ethan Furman wrote:
 > What am I doing wrong?

On 11/13/2015 07:40 AM, Georg Brandl wrote:
 > Running "make doctest" in the first place. The doctests in the core
 > docs were not written to support that, and also never updated to do
 > so.

On 11/13/2015 07:42 AM, Zachary Ware wrote:
 > Expecting it to work :)

On 11/13/2015 07:44 AM, R. David Murray wrote:
 > We don't have clean doctests for the docs.  Patches welcome.

Ah, okay, that explains it.  :)  Which means I had tested 'enum.rst' 
manually back when I wrote it, and had forgotten.

Thank you for the reminder!

--
~Ethan~

From andrew.svetlov at gmail.com  Fri Nov 13 17:06:23 2015
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Sat, 14 Nov 2015 00:06:23 +0200
Subject: [Python-Dev] Rietveld is broken
Message-ID: <CAL3CFcXfY8+fYsW8bFhwGs=fzvRmWM2ZdimBXRFR8qbXtQHF=A@mail.gmail.com>

When I try to press 'retry' on http://bugs.python.org/review/25074/
page and send a message server responds with 500 error.
Going through the page for viewing diff for concrete file
http://bugs.python.org/review/25074/diff/15535/Lib/test/test_asyncio/test_base_events.py
with adding a comment works well.

The behavior is very annoying -- I was forced to write missed message twice.

-- 
Thanks,
Andrew Svetlov

From damien.p.george at gmail.com  Fri Nov 13 17:32:07 2015
From: damien.p.george at gmail.com (Damien George)
Date: Fri, 13 Nov 2015 22:32:07 +0000
Subject: [Python-Dev] Clarification of PEP 394 for scripts that run under
 Python 2 and 3
Message-ID: <CANw+MznRKh-AyF4ObSf_0GaepZgtpAAdqJBw3eFCqc8PgxQzyw@mail.gmail.com>

Hi python-dev,

We have a Python script that runs correctly under Python 2.6, 2.7 and 3.3+.
It is executed on a *nix system using the "python" executable (ie not
python2 or python3 specifically). This works just fine for systems that
have Python 2 installed, or 2 and 3, or just 3 and symlink "python" to
"python3" (eg Arch Linux).

But it fails for systems that have only Python 3 and do not create a
"python" symlink, ie only "python3" exists as an executable.

We thought that PEP 394 would come to the rescue here but it seems to be
unclear on this point. In particular it says:

- 4th point of the abstract: "so python should be used in the shebang line
only for scripts that are source compatible with both Python 2 and 3"

- 6th point of the recommendation section: "One exception to this is
scripts that are deliberately written to be source compatible with both
Python 2.x and 3.x. Such scripts may continue to use python on their
shebang line without affecting their portability"

- 8th point in the migration notes: "If these conventions are adhered to,
it will become the case that the python command is only executed in an
interactive manner as a user convenience, or to run scripts that are source
compatible with both Python 2 and Python 3."

Well, that's pretty clear to me: one can expect the "python" executable to
be available to run scripts that are compatible with versions 2.x and 3.x.

The confusion comes because there are systems that install Python 3 without
creating a "python" symlink (hence breaking the above).  And Guido said
that "'python' should always be the same as 'python2'" (see
https://mail.python.org/pipermail/python-dev/2014-September/136389.html).
Further, Nick Coghlan seemed to agree that "when there's only python3
installed, there should be no /usr/bin/python" (see
https://mail.python.org/pipermail/python-dev/2014-September/136527.html).

My questions are:

1. What is the true intent of PEP 394 when only Python 3 is installed?  Is
"python" available or not to run scripts compatible with 2.x and 3.x?

2. Is it possible to write a shebang line that supports all variations of
Python installations on *nix machines?

3. If the answer to 2 is no, then what is the recommended way to support
all Python installations with one standalone script?

Thanks!

Regards,
Damien.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151113/86278b4e/attachment.html>

From barry at python.org  Fri Nov 13 17:57:27 2015
From: barry at python.org (Barry Warsaw)
Date: Fri, 13 Nov 2015 17:57:27 -0500
Subject: [Python-Dev] Clarification of PEP 394 for scripts that run
 under Python 2 and 3
In-Reply-To: <CANw+MznRKh-AyF4ObSf_0GaepZgtpAAdqJBw3eFCqc8PgxQzyw@mail.gmail.com>
References: <CANw+MznRKh-AyF4ObSf_0GaepZgtpAAdqJBw3eFCqc8PgxQzyw@mail.gmail.com>
Message-ID: <20151113175727.338f192c@limelight.wooz.org>

On Nov 13, 2015, at 10:32 PM, Damien George wrote:

>1. What is the true intent of PEP 394 when only Python 3 is installed?  Is
>"python" available or not to run scripts compatible with 2.x and 3.x?
>
>2. Is it possible to write a shebang line that supports all variations of
>Python installations on *nix machines?
>
>3. If the answer to 2 is no, then what is the recommended way to support
>all Python installations with one standalone script?

It's important to remember that PEP 394 is an informational PEP, still under
active (if dormant) discussion.  It is also just a recommendation, and can't
force any downstream redistributors to do one thing or another.  Still, the
intent is to provide a set of guidelines for the majority of *nix distributors
to (eventually) adopt.

I'll also point you to the recently created linux-sig where this topic is
being discussed.

https://mail.python.org/pipermail/linux-sig/2015-October/000000.html

As you've noticed, Arch took a particular approach, but speaking as part of
the Debian/Ubuntu community, don't expect that ecosystem to go down the same
path.  It's highly unlikely /usr/bin/python will ever point to anything other
than Python 2, even when only Python 3 is installed by default.  That might
change once Python 2.7 is actually EOL'd, and that won't happen for quite a
long time.  PEP 373 currently says there will be bug fix releases until 2020,
and I'd expect security-only releases for some time after that.

In general, the discussions on linux-sig and elsewhere is coalescing around a
launcher-type approach, where you'd put something like /usr/bin/py in your
shebang if your script really is bilingual.  Then `py` can try to figure out
what's available on your system, what your preference is, etc. and then
execute your script using that version-specific binary.  I would expect the
launcher to eventually be provided by the upstream Python development
community.

AFAIK, there have been a few stabs at writing such a launcher, but nothing
that's far enough to be usable.

So for now, you don't have great options.

Cheers,
-Barry

From storchaka at gmail.com  Sat Nov 14 14:19:37 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 14 Nov 2015 21:19:37 +0200
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
Message-ID: <n281g9$hbq$1@ger.gmane.org>

For now UTF-16 and UTF-32 source encodings are not supported. There is a 
comment in Parser/tokenizer.c:

     /* Disable support for UTF-16 BOMs until a decision
        is made whether this needs to be supported.  */

Can we make a decision whether this support will be added in foreseeable 
future (say in near 10 years), or no?

Removing commented out and related code will help to refactor the 
tokenizer, and that can help to fix some existing bugs (e.g. issue14811, 
issue18961, issue20115 and may be others). Current tokenizing code is 
too tangled.

If the support of UTF-16 and UTF-32 is planned, I'll take this to 
attention during refactoring. But in many places besides the tokenizer 
the ASCII compatible encoding of source files is expected.


From victor.stinner at gmail.com  Sat Nov 14 17:56:43 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sat, 14 Nov 2015 23:56:43 +0100
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <n281g9$hbq$1@ger.gmane.org>
References: <n281g9$hbq$1@ger.gmane.org>
Message-ID: <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>

These encodings are rarely used. I don't think that any text editor use
them. Editors use ascii, latin1, utf8 and... all locale encoding. But I
don't know any OS using UTF-16 as a locale encoding. UTF-32 wastes disk
space.

Ok, even if it exists, Python already accepts a very wide range of
encoding. It is not worth to make the parser much more complex just to
support encodings which are also never used (for .py files).

Victor
Le 14 nov. 2015 20:20, "Serhiy Storchaka" <storchaka at gmail.com> a ?crit :

> For now UTF-16 and UTF-32 source encodings are not supported. There is a
> comment in Parser/tokenizer.c:
>
>     /* Disable support for UTF-16 BOMs until a decision
>        is made whether this needs to be supported.  */
>
> Can we make a decision whether this support will be added in foreseeable
> future (say in near 10 years), or no?
>
> Removing commented out and related code will help to refactor the
> tokenizer, and that can help to fix some existing bugs (e.g. issue14811,
> issue18961, issue20115 and may be others). Current tokenizing code is too
> tangled.
>
> If the support of UTF-16 and UTF-32 is planned, I'll take this to
> attention during refactoring. But in many places besides the tokenizer the
> ASCII compatible encoding of source files is expected.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151114/ac0a9077/attachment.html>

From benjamin at python.org  Sat Nov 14 17:57:58 2015
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 14 Nov 2015 14:57:58 -0800
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <n281g9$hbq$1@ger.gmane.org>
References: <n281g9$hbq$1@ger.gmane.org>
Message-ID: <1447541878.551398.439918241.1E1CFA11@webmail.messagingengine.com>

I agree that supporting UTF-16 doesn't seem terribly useful. Also, thank
you for giving the tokenizer some love!

On Sat, Nov 14, 2015, at 11:19, Serhiy Storchaka wrote:
> For now UTF-16 and UTF-32 source encodings are not supported. There is a 
> comment in Parser/tokenizer.c:
> 
>      /* Disable support for UTF-16 BOMs until a decision
>         is made whether this needs to be supported.  */
> 
> Can we make a decision whether this support will be added in foreseeable 
> future (say in near 10 years), or no?
> 
> Removing commented out and related code will help to refactor the 
> tokenizer, and that can help to fix some existing bugs (e.g. issue14811, 
> issue18961, issue20115 and may be others). Current tokenizing code is 
> too tangled.
> 
> If the support of UTF-16 and UTF-32 is planned, I'll take this to 
> attention during refactoring. But in many places besides the tokenizer 
> the ASCII compatible encoding of source files is expected.
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org

From storchaka at gmail.com  Sat Nov 14 18:21:54 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 15 Nov 2015 01:21:54 +0200
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
Message-ID: <n28fmj$hb$1@ger.gmane.org>

On 15.11.15 00:56, Victor Stinner wrote:
> These encodings are rarely used. I don't think that any text editor use
> them. Editors use ascii, latin1, utf8 and... all locale encoding. But I
> don't know any OS using UTF-16 as a locale encoding. UTF-32 wastes disk
> space.

AFAIK the standard Windows editor Notepad uses UTF-16. And I often 
encountered Windows resource files in UTF-16. UTF-16 was more popular 
than UTF-8 on Windows some time. If this horse is dead I'll throw it away.



From v+python at g.nevcal.com  Sat Nov 14 18:39:32 2015
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Sat, 14 Nov 2015 15:39:32 -0800
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <n28fmj$hb$1@ger.gmane.org>
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
 <n28fmj$hb$1@ger.gmane.org>
Message-ID: <5647C634.5030700@g.nevcal.com>

On 11/14/2015 3:21 PM, Serhiy Storchaka wrote:
> On 15.11.15 00:56, Victor Stinner wrote:
>> These encodings are rarely used. I don't think that any text editor use
>> them. Editors use ascii, latin1, utf8 and... all locale encoding. But I
>> don't know any OS using UTF-16 as a locale encoding. UTF-32 wastes disk
>> space.
>
> AFAIK the standard Windows editor Notepad uses UTF-16. And I often 
> encountered Windows resource files in UTF-16. UTF-16 was more popular 
> than UTF-8 on Windows some time. If this horse is dead I'll throw it away.

Just use UTF-8, ignoring an optional leading BOM. If someone wants to 
use something else, they can write a "preprocessor" to convert it to 
UTF-8 for use by Python.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151114/7f745ea8/attachment.html>

From random832 at fastmail.com  Sat Nov 14 18:48:04 2015
From: random832 at fastmail.com (Random832)
Date: Sat, 14 Nov 2015 18:48:04 -0500
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
Message-ID: <m2twoop2h7.fsf@fastmail.com>

Victor Stinner <victor.stinner at gmail.com> writes:
> These encodings are rarely used. I don't think that any text editor
> use them.

MS Windows' Notepad can be made to use UTF-16.


From steve.dower at python.org  Sat Nov 14 20:06:47 2015
From: steve.dower at python.org (Steve Dower)
Date: Sat, 14 Nov 2015 17:06:47 -0800
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
Message-ID: <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>

The native encoding on Windows has been UTF-16 since Windows NT. Obviously we've survived without Python tokenization support for a long time, but every API uses it.

I've hit a few cases where it would have been handy for Python to be able to detect it, though nothing I couldn't work around. Saying it is rarely used is rather exposing your own unawareness though - it could arguably be the most commonly used encoding (depending on how you define "used").

Cheers,
Steve

Top-posted from my Windows Phone

-----Original Message-----
From: "Victor Stinner" <victor.stinner at gmail.com>
Sent: ?11/?14/?2015 14:58
To: "Serhiy Storchaka" <storchaka at gmail.com>
Cc: "python-dev at python.org" <python-dev at python.org>
Subject: Re: [Python-Dev] Support of UTF-16 and UTF-32 source encodings

These encodings are rarely used. I don't think that any text editor use them. Editors use ascii, latin1, utf8 and... all locale encoding. But I don't know any OS using UTF-16 as a locale encoding. UTF-32 wastes disk space.
Ok, even if it exists, Python already accepts a very wide range of encoding. It is not worth to make the parser much more complex just to support encodings which are also never used (for .py files).
Victor

Le 14 nov. 2015 20:20, "Serhiy Storchaka" <storchaka at gmail.com> a ?crit :

For now UTF-16 and UTF-32 source encodings are not supported. There is a comment in Parser/tokenizer.c:

    /* Disable support for UTF-16 BOMs until a decision
       is made whether this needs to be supported.  */

Can we make a decision whether this support will be added in foreseeable future (say in near 10 years), or no?

Removing commented out and related code will help to refactor the tokenizer, and that can help to fix some existing bugs (e.g. issue14811, issue18961, issue20115 and may be others). Current tokenizing code is too tangled.

If the support of UTF-16 and UTF-32 is planned, I'll take this to attention during refactoring. But in many places besides the tokenizer the ASCII compatible encoding of source files is expected.

_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151114/4a88d2dc/attachment.html>

From rosuav at gmail.com  Sat Nov 14 20:15:28 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 15 Nov 2015 12:15:28 +1100
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
 <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
Message-ID: <CAPTjJmpc8CRCqTxEt49E5CYicXwkDrKCXreNJ1GVN-HFDB9HBA@mail.gmail.com>

On Sun, Nov 15, 2015 at 12:06 PM, Steve Dower <steve.dower at python.org> wrote:
> The native encoding on Windows has been UTF-16 since Windows NT. Obviously
> we've survived without Python tokenization support for a long time, but
> every API uses it.
>
> I've hit a few cases where it would have been handy for Python to be able to
> detect it, though nothing I couldn't work around. Saying it is rarely used
> is rather exposing your own unawareness though - it could arguably be the
> most commonly used encoding (depending on how you define "used").

What matters here is: How likely is it that an arbitrary Python script
(or, say, "arbitrary text file") is encoded UTF-16 rather than
something ASCII-compatible? I think even Notepad defaults to UTF-8 for
files, now. The fact that it's sending text to the GUI subsystem in
UTF-16 is immaterial here.

Can the py.exe launcher handle a UTF-16 shebang? (I'm pretty sure Unix
program loaders won't.) That alone might be a reason for strongly
encouraging ASCII-compat encodings.

ChrisA

From v+python at g.nevcal.com  Sat Nov 14 20:24:05 2015
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Sat, 14 Nov 2015 17:24:05 -0800
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <CAPTjJmpc8CRCqTxEt49E5CYicXwkDrKCXreNJ1GVN-HFDB9HBA@mail.gmail.com>
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
 <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
 <CAPTjJmpc8CRCqTxEt49E5CYicXwkDrKCXreNJ1GVN-HFDB9HBA@mail.gmail.com>
Message-ID: <5647DEB5.2000905@g.nevcal.com>

On 11/14/2015 5:15 PM, Chris Angelico wrote:
> Can the py.exe launcher handle a UTF-16 shebang? (I'm pretty sure Unix
> program loaders won't.) That alone might be a reason for strongly
> encouraging ASCII-compat encodings.

That raises an interesting question about if py.exe can handle a leading 
UTF-8 BOM.  I have my emacs-on-Windows configured to store UTF-8 without 
BOM, but Notepad would put a BOM when saving UTF-8, last I checked.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151114/85e557ae/attachment.html>

From random832 at fastmail.com  Sat Nov 14 20:25:57 2015
From: random832 at fastmail.com (Random832)
Date: Sat, 14 Nov 2015 20:25:57 -0500
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
 <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
 <CAPTjJmpc8CRCqTxEt49E5CYicXwkDrKCXreNJ1GVN-HFDB9HBA@mail.gmail.com>
Message-ID: <m2io54oxy2.fsf@fastmail.com>

Chris Angelico <rosuav at gmail.com> writes:
> Can the py.exe launcher handle a UTF-16 shebang? (I'm pretty sure Unix
> program loaders won't.)

A lot of them can't handle UTF-8 with a BOM, either.

> That alone might be a reason for strongly encouraging ASCII-compat
> encodings.

A "python" or "python3" or "env" executable in any particular location
such as /usr/bin isn't technically guaranteed, either, it's just relied
on as a "works 99% of the time" thing. There's a reasonable case to be
made that transforming files in such a way as they get launched by
python (which may require an encoding change to an ASCII-compatible
encoding, or a wrapper script, or the python -x hack) is the
responsibility of platform-specific installer code.


From v+python at g.nevcal.com  Sat Nov 14 20:27:06 2015
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Sat, 14 Nov 2015 17:27:06 -0800
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <CAPTjJmpc8CRCqTxEt49E5CYicXwkDrKCXreNJ1GVN-HFDB9HBA@mail.gmail.com>
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
 <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
 <CAPTjJmpc8CRCqTxEt49E5CYicXwkDrKCXreNJ1GVN-HFDB9HBA@mail.gmail.com>
Message-ID: <5647DF6A.7000900@g.nevcal.com>

On 11/14/2015 5:15 PM, Chris Angelico wrote:
> I think even Notepad defaults to UTF-8 for
> files, now.

Just installed Windows 10 on a new machine, and upgraded to the latest 
Windows 10 release, 1511.

Notepad defaults to ANSI encoding, as I think it always has.  UTF-8 is 
an option, and it does seem to try to notice the original encoding of 
the file, when editing old files, but when creating a new one.... ANSI.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151114/2508c1d6/attachment-0001.html>

From rosuav at gmail.com  Sat Nov 14 20:37:54 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 15 Nov 2015 12:37:54 +1100
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <5647DF6A.7000900@g.nevcal.com>
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
 <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
 <CAPTjJmpc8CRCqTxEt49E5CYicXwkDrKCXreNJ1GVN-HFDB9HBA@mail.gmail.com>
 <5647DF6A.7000900@g.nevcal.com>
Message-ID: <CAPTjJmqz3oJphPKsQEQYH3OJJP-Cj9DqdPTZ+8HycuOTQ-BQJQ@mail.gmail.com>

On Sun, Nov 15, 2015 at 12:27 PM, Glenn Linderman <v+python at g.nevcal.com> wrote:
> Notepad defaults to ANSI encoding, as I think it always has.  UTF-8 is an
> option, and it does seem to try to notice the original encoding of the file,
> when editing old files, but when creating a new one.... ANSI.

Thanks. Is "ANSI" always an eight-bit ASCII-compatible encoding?

ChrisA

From v+python at g.nevcal.com  Sat Nov 14 20:47:48 2015
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Sat, 14 Nov 2015 17:47:48 -0800
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <CAPTjJmqz3oJphPKsQEQYH3OJJP-Cj9DqdPTZ+8HycuOTQ-BQJQ@mail.gmail.com>
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
 <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
 <CAPTjJmpc8CRCqTxEt49E5CYicXwkDrKCXreNJ1GVN-HFDB9HBA@mail.gmail.com>
 <5647DF6A.7000900@g.nevcal.com>
 <CAPTjJmqz3oJphPKsQEQYH3OJJP-Cj9DqdPTZ+8HycuOTQ-BQJQ@mail.gmail.com>
Message-ID: <5647E444.3080701@g.nevcal.com>

On 11/14/2015 5:37 PM, Chris Angelico wrote:
> On Sun, Nov 15, 2015 at 12:27 PM, Glenn Linderman <v+python at g.nevcal.com> wrote:
>> Notepad defaults to ANSI encoding, as I think it always has.  UTF-8 is an
>> option, and it does seem to try to notice the original encoding of the file,
>> when editing old files, but when creating a new one.... ANSI.
> Thanks. Is "ANSI" always an eight-bit ASCII-compatible encoding?

I wouldn't trust an answer to this question that didn't come from 
someone that used Windows with Chinese, Japanese, or Korean, as their 
default language for the install. So I don't have a trustworthy answer 
to give.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151114/761110be/attachment.html>

From rosuav at gmail.com  Sat Nov 14 20:59:32 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 15 Nov 2015 12:59:32 +1100
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <5647E444.3080701@g.nevcal.com>
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
 <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
 <CAPTjJmpc8CRCqTxEt49E5CYicXwkDrKCXreNJ1GVN-HFDB9HBA@mail.gmail.com>
 <5647DF6A.7000900@g.nevcal.com>
 <CAPTjJmqz3oJphPKsQEQYH3OJJP-Cj9DqdPTZ+8HycuOTQ-BQJQ@mail.gmail.com>
 <5647E444.3080701@g.nevcal.com>
Message-ID: <CAPTjJmp0LhsRyfkDxrJJrByn1KV7VLm0M2Mdz3pmScAiJJ0CSg@mail.gmail.com>

On Sun, Nov 15, 2015 at 12:47 PM, Glenn Linderman <v+python at g.nevcal.com> wrote:
> On 11/14/2015 5:37 PM, Chris Angelico wrote:
>
> On Sun, Nov 15, 2015 at 12:27 PM, Glenn Linderman <v+python at g.nevcal.com>
> wrote:
>
> Notepad defaults to ANSI encoding, as I think it always has.  UTF-8 is an
> option, and it does seem to try to notice the original encoding of the file,
> when editing old files, but when creating a new one.... ANSI.
>
> Thanks. Is "ANSI" always an eight-bit ASCII-compatible encoding?
>
>
> I wouldn't trust an answer to this question that didn't come from someone
> that used Windows with Chinese, Japanese, or Korean, as their default
> language for the install. So I don't have a trustworthy answer to give.
>

Heh, yeah. But I'd trust an answer from Steve Dower :)

ChrisA

From random832 at fastmail.com  Sat Nov 14 21:10:02 2015
From: random832 at fastmail.com (Random832)
Date: Sat, 14 Nov 2015 21:10:02 -0500
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
 <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
 <CAPTjJmpc8CRCqTxEt49E5CYicXwkDrKCXreNJ1GVN-HFDB9HBA@mail.gmail.com>
 <5647DF6A.7000900@g.nevcal.com>
 <CAPTjJmqz3oJphPKsQEQYH3OJJP-Cj9DqdPTZ+8HycuOTQ-BQJQ@mail.gmail.com>
 <5647E444.3080701@g.nevcal.com>
Message-ID: <m2bnawovwl.fsf@fastmail.com>

Glenn Linderman <v+python at g.nevcal.com> writes:
> On 11/14/2015 5:37 PM, Chris Angelico wrote:
> > Thanks. Is "ANSI" always an eight-bit ASCII-compatible encoding?
>
> I wouldn't trust an answer to this question that didn't come from
> someone that used Windows with Chinese, Japanese, or Korean, as their
> default language for the install. So I don't have a trustworthy answer
> to give.

AFAIK (I haven't actually used it as a default language, but I do know
some details of their encodings) There are two main "issues" with the
windows CJK encodings regarding ASCII compatibility:

- There is a different symbol (a currency symbol) at 0x5c. Sort of. Most
  unicode translations of it will treat it as a backslash, and users do
  expect it to work for things like \n, path separators, etc, but it
  displays as ? or ?.

- Dual-byte characters can have ASCII bytes as their *second* byte.


From steve at pearwood.info  Sat Nov 14 21:19:44 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 15 Nov 2015 13:19:44 +1100
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <n281g9$hbq$1@ger.gmane.org>
References: <n281g9$hbq$1@ger.gmane.org>
Message-ID: <20151115021941.GE2038@ando.pearwood.info>

On Sat, Nov 14, 2015 at 09:19:37PM +0200, Serhiy Storchaka wrote:

> If the support of UTF-16 and UTF-32 is planned, I'll take this to 
> attention during refactoring. But in many places besides the tokenizer 
> the ASCII compatible encoding of source files is expected.

Perhaps another way of looking at this:

Is it feasible to drop support for arbitrary encodings and just require 
UTF-8 (with or without a pseudo-BOM)?



-- 
Steve

From eryksun at gmail.com  Sat Nov 14 21:57:51 2015
From: eryksun at gmail.com (eryksun)
Date: Sat, 14 Nov 2015 20:57:51 -0600
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
 <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
Message-ID: <CACL+1avcArqf7OR9pyeSys11JiVn_RsVgiQrQ8zi=WCNR2TZag@mail.gmail.com>

On Sat, Nov 14, 2015 at 7:06 PM, Steve Dower <steve.dower at python.org> wrote:
> The native encoding on Windows has been UTF-16 since Windows NT. Obviously
> we've survived without Python tokenization support for a long time, but
> every API uses it.

Windows 2000 was the first version to have broad support for UTF-16.
Windows NT (1993) was released before UTF-16, so its Unicode support
is limited to UCS-2.

(Note that console windows still restrict each character cell to a
single WCHAR character. So a non-BMP character encoded as a UTF-16
surrogate pair always appears as two box glyphs. Of course you can
copy and paste from the console to a UTF-16 aware window just fine.)

> I've hit a few cases where it would have been handy for Python to be able to
> detect it, though nothing I couldn't work around.

Can you elaborate some example cases? I can see using UTF-16 for the
REPL in the Windows console, but a hypothetical WinConIO class could
simply transcode to and from UTF-8. Drekin's win-unicode-console
package works like this.

From eryksun at gmail.com  Sat Nov 14 21:59:01 2015
From: eryksun at gmail.com (eryksun)
Date: Sat, 14 Nov 2015 20:59:01 -0600
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <CAPTjJmpc8CRCqTxEt49E5CYicXwkDrKCXreNJ1GVN-HFDB9HBA@mail.gmail.com>
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
 <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
 <CAPTjJmpc8CRCqTxEt49E5CYicXwkDrKCXreNJ1GVN-HFDB9HBA@mail.gmail.com>
Message-ID: <CACL+1atM9==46jfuJDge9TGq_i=A-Guz9cKP8S5mNwEughtSXw@mail.gmail.com>

On Sat, Nov 14, 2015 at 7:15 PM, Chris Angelico <rosuav at gmail.com> wrote:
> Can the py.exe launcher handle a UTF-16 shebang? (I'm pretty sure Unix
> program loaders won't.) That alone might be a reason for strongly
> encouraging ASCII-compat encodings.

The launcher supports shebangs encoded as UTF-8 (default), UTF-16
(LE/BE), and UTF-32 (LE/BE):

https://hg.python.org/cpython/file/v3.5.0/PC/launcher.c#l1138

From stephen at xemacs.org  Sun Nov 15 02:23:50 2015
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sun, 15 Nov 2015 16:23:50 +0900
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
 <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
Message-ID: <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp>

Steve Dower writes:

 > Saying [UTF-16] is rarely used is rather exposing your own
 > unawareness though - it could arguably be the most commonly used
 > encoding (depending on how you define "used").

Because we're discussing the storage of .py files, the relevant
definition is the one used by the Unicode Standard, of course: a
text/plain stream intended to be manipulated by any conformant Unicode
processor that claims to handle text/plain.  File formats with in-band
formatting codes and allowing embedded non-text content like Word, or
operating system or stdlib APIs, don't count.  Nor have I seen UTF-16
used in email or HTML since the unregretted days of Win2k betas[1]
(but I don't frequent Windows- or Java-oriented sites, so I have to
admit my experience is limited in a possibly relevant way).

In Japan my impression is that modern versions of Windows have
Memopad[sic] configured to emit UTF-8-with-signature by default for
new files, and if not, the abomination known as Shift JIS (I'm not
sure if that is a user or OEM option, though).  Never a widechar
encoding (after all, the whole point of Shift JIS was to use an 8-bit
encoding for the katakana syllabary to save space or bandwidth).

I think if anyone wants to use UTF-16 or UTF-32 for exchange of Python
programs, they probably already know how to convert them to UTF-8.  As
somebody already suggested, this can be delegated to the py.exe
launcher, if necessary, AFAICS.

I don't see any good reason for allowing non-ASCII-compatible
encodings in the reference CPython interpreter.

However, having mentioned Windows and Java, I have to wonder about
IronPython and Jython, respectively.  Having never lived in either of
those environments, I don't know what text encoding their users might
prefer (or even occasionally encounter) in Python program source.

Steve

Footnotes: 
[1]  The version of Outlook Express shipped with them would emit
"HTML" mail with ASCII tags and UTF-8-encoded text (even if it was
encodable in pure ASCII).  No, it wasn't spam, either, so it probably
really was Outlook Express as it claimed to be in one of the headers.


From random832 at fastmail.com  Sun Nov 15 06:51:12 2015
From: random832 at fastmail.com (Random832)
Date: Sun, 15 Nov 2015 06:51:12 -0500
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
 <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
 <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp>
Message-ID: <m24mgnfplb.fsf@fastmail.com>

"Stephen J. Turnbull" <stephen at xemacs.org> writes:
> I don't see any good reason for allowing non-ASCII-compatible
> encodings in the reference CPython interpreter.

There might be a case for having the tokenizer not care about encodings
at all and just operate on a stream of unicode characters provided by a
different layer.


From p.f.moore at gmail.com  Sun Nov 15 07:56:18 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 15 Nov 2015 12:56:18 +0000
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp>
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
 <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
 <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp>
Message-ID: <CACac1F8STZYwwUbV_bNLmfMZarS2uidsSttbhFL1CzJi229nNA@mail.gmail.com>

On 15 November 2015 at 07:23, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> I don't see any good reason for allowing non-ASCII-compatible
> encodings in the reference CPython interpreter.

>From PEP 263:

       Any encoding which allows processing the first two lines in the
       way indicated above is allowed as source code encoding, this
       includes ASCII compatible encodings as well as certain
       multi-byte encodings such as Shift_JIS. It does not include
       encodings which use two or more bytes for all characters like
       e.g. UTF-16. The reason for this is to keep the encoding
       detection algorithm in the tokenizer simple.

So this pretty much confirms that double-byte encodings are not valid
for Python source files.

Paul

From lac at openend.se  Sun Nov 15 09:43:58 2015
From: lac at openend.se (Laura Creighton)
Date: Sun, 15 Nov 2015 15:43:58 +0100
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <CACac1F8STZYwwUbV_bNLmfMZarS2uidsSttbhFL1CzJi229nNA@mail.gmail.com>
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
 <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
 <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp>
 <CACac1F8STZYwwUbV_bNLmfMZarS2uidsSttbhFL1CzJi229nNA@mail.gmail.com>
Message-ID: <201511151443.tAFEhwfi029674@fido.openend.se>

In a message of Sun, 15 Nov 2015 12:56:18 +0000, Paul Moore writes:
>On 15 November 2015 at 07:23, Stephen J. Turnbull <stephen at xemacs.org> wrote:
>> I don't see any good reason for allowing non-ASCII-compatible
>> encodings in the reference CPython interpreter.
>
>>From PEP 263:
>
>       Any encoding which allows processing the first two lines in the
>       way indicated above is allowed as source code encoding, this
>       includes ASCII compatible encodings as well as certain
>       multi-byte encodings such as Shift_JIS. It does not include
>       encodings which use two or more bytes for all characters like
>       e.g. UTF-16. The reason for this is to keep the encoding
>       detection algorithm in the tokenizer simple.
>
>So this pretty much confirms that double-byte encodings are not valid
>for Python source files.
>
>Paul

Steve Turnbull, who lives in Japan, and speaks and writes Japanese
is saying that "he cannot see any reason for allowing non-ASCII
compatible encodings in Cpython".

This makes me wonder.

Is this along the lines of 'even in Japan we do not want such
things' or along the lines of 'when in Japan we want such things
we want to so brutally do so much more, so keep the reference
implementation simple, and don't try to help us with this 
seems-like-a-good-idea-but-isnt-in-practice' ideas like this one,
or
....

Laura

From ja.py at farowl.co.uk  Sun Nov 15 10:17:49 2015
From: ja.py at farowl.co.uk (Jeff Allen)
Date: Sun, 15 Nov 2015 15:17:49 +0000
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp>
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
 <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
 <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp>
Message-ID: <5648A21D.5050707@farowl.co.uk>

I'm approaching this from the premise that we would like to avoid 
needless surprises for users not versed in text encoding. I did a simple 
experiment with notepad on Windows 7 as if a na?ve user. If I write the 
one-line program:

print("Hello world.") # by Jeff

It runs, no surprise.

We may legitimately encounter Unicode in string literals and comments. 
If I write:

print("j't'kif Ana?s!") # par Herv?

and try to save it, notepad tells me this file "contains characters in 
Unicode format which will be lost if you save this as an ANSI encoded 
text file." To keep the Unicode information I should cancel and choose a 
Unicode option. In the "Save as" dialogue the default encoding is ANSI. 
The second option "Unicode" is clearly right as the warning said 
"Unicode" 3 times and I don't know what big-endian or UTF-8 mean. Good 
that worked. Closed and opened it looks exactly as I typed it.

But the bytes I actually wrote on disk consist of a BOM and UTF-16-LE. 
And running it I get:
   File "bonjour.py", line 1
SyntaxError: Non-UTF-8 code starting with '\xff' in file bonjour.py on 
line 1, but no encoding declared; see 
http://python.org/dev/peps/pep-0263/ for details

If I take the hint here and save as UTF-8, then it works, including 
printing the accent. Inspection of the bytes shows it starts with a 
UTF-8 BOM.

In Jython I get the same results (choking on UTF-16), but saved as 
UTF-8, it works. I just have to make sure that's a Unicode constant if I 
want it to print correctly, as we're at 2.7. Jython has a checkered past 
with encodings, but tries to do exactly the same as CPython 2.7.x.

Now, a fact I haven't mentioned is that my machine was localised to 
simplified Chinese (to diagnose some bug) during this test. If I 
re-localise to my usual English (UK), I do not get the guidance from 
notepad: instead it quietly saves as Latin-1 (cp1252), perhaps because 
I'm in Western Europe. Python baulks at this, at the first accented 
character. If I save from notepad as Unicode or UTF-8 the results are as 
before, including the BOM.

In some circumstances, then, the natural result of using notepad and not 
sticking to ASCII may be UTF-16-LE with a BOM, or Latin-1 depending on 
localisation, it seems. The Python error message provides a clue what a 
user should do, but they would need some background, a helpful teacher, 
or the Internet to sort it out.

Jeff Allen

On 15/11/2015 07:23, Stephen J. Turnbull wrote:
> Steve Dower writes:
>
>   > Saying [UTF-16] is rarely used is rather exposing your own
>   > unawareness though - it could arguably be the most commonly used
>   > encoding (depending on how you define "used").
>
> Because we're discussing the storage of .py files, the relevant
> definition is the one used by the Unicode Standard, of course: a
> text/plain stream intended to be manipulated by any conformant Unicode
> processor that claims to handle text/plain.  File formats with in-band
> formatting codes and allowing embedded non-text content like Word, or
> operating system or stdlib APIs, don't count.  Nor have I seen UTF-16
> used in email or HTML since the unregretted days of Win2k betas[1]
> (but I don't frequent Windows- or Java-oriented sites, so I have to
> admit my experience is limited in a possibly relevant way).
>
> In Japan my impression is that modern versions of Windows have
> Memopad[sic] configured to emit UTF-8-with-signature by default for
> new files, and if not, the abomination known as Shift JIS (I'm not
> sure if that is a user or OEM option, though).  Never a widechar
> encoding (after all, the whole point of Shift JIS was to use an 8-bit
> encoding for the katakana syllabary to save space or bandwidth).
>
> I think if anyone wants to use UTF-16 or UTF-32 for exchange of Python
> programs, they probably already know how to convert them to UTF-8.  As
> somebody already suggested, this can be delegated to the py.exe
> launcher, if necessary, AFAICS.
>
> I don't see any good reason for allowing non-ASCII-compatible
> encodings in the reference CPython interpreter.
>
> However, having mentioned Windows and Java, I have to wonder about
> IronPython and Jython, respectively.  Having never lived in either of
> those environments, I don't know what text encoding their users might
> prefer (or even occasionally encounter) in Python program source.
>
> Steve
>
> Footnotes:
> [1]  The version of Outlook Express shipped with them would emit
> "HTML" mail with ASCII tags and UTF-8-encoded text (even if it was
> encodable in pure ASCII).  No, it wasn't spam, either, so it probably
> really was Outlook Express as it claimed to be in one of the headers.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:https://mail.python.org/mailman/options/python-dev/ja.py%40farowl.co.uk
>


From stephen at xemacs.org  Sun Nov 15 11:40:09 2015
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Mon, 16 Nov 2015 01:40:09 +0900
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <201511151443.tAFEhwfi029674@fido.openend.se>
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
 <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
 <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp>
 <CACac1F8STZYwwUbV_bNLmfMZarS2uidsSttbhFL1CzJi229nNA@mail.gmail.com>
 <201511151443.tAFEhwfi029674@fido.openend.se>
Message-ID: <22088.46441.171953.459003@turnbull.sk.tsukuba.ac.jp>

Laura Creighton writes:

 > Steve Turnbull, who lives in Japan, and speaks and writes Japanese
 > is saying that "he cannot see any reason for allowing non-ASCII
 > compatible encodings in Cpython".
 > 
 > This makes me wonder.
 > 
 > Is this along the lines of 'even in Japan we do not want such
 > things' or along the lines of 'when in Japan we want such things
 > we want to so brutally do so much more, so keep the reference
 > implementation simple, and don't try to help us with this 
 > seems-like-a-good-idea-but-isnt-in-practice' ideas like this one,
 > or
 > ....

I'm saying that to my knowledge Japan is the most complicated place
there is when it comes to encodings, and even so, nobody here seems to
be using UTF-16 as the encoding for program sources (or any other
text/* media).  Of course as Steve Dower pointed out it's in heavy use
as an internal text encoding, in OS APIs, in some languages' stdlib
APIs (ie, Java and I suppose .NET), and I guess in single-application
file formats (Word), but the programs that use those APIs are written
in ASCII compatible-encodings (and Shift JIS and Big5).  The Japanese
don't need or want UTF-16 in text files, etc.

Besides that, I can also say that PEP 263 didn't legislate the use of
ASCII-compatible encodings.  For one thing, Shift JIS and Big5 aren't
100% compatible because they uses 0x20-0x7f in multibyte characters.
They're just close enough to ASCII compatible to mostly "just work",
at least on Microsoft OSes provided by OEMs in the relevant countries.

What PEP 263 did do was to specify that non-ASCII-compatible encodings
are not supported by the PEP 263 mechanism for declaring the encoding
of a Python source program.  That's because it looks for a "magic
number" which is the ASCII-encoded form of "coding:" in the first two
lines.  It doesn't rule out alternative mechanisms for encoding
detection (specifically, use of the UTF-16 "BOM" signature); it just
doesn't propose implementing them.

IIRC nobody has ever asked for them, but I think the idea is absurd
so I have to admit I may have seen a request and forgot it instantly.

Bottom line: as long as Python (or the launcher) is able to transcode
the source to the internal Unicode format (UTF-8 in Python 2, and
widechar or PEP 393 in Python 3) before actually beginning parsing,
any on-disk encoding is OK.  But I just don't see a use case for
UTF-16.  If I'm wrong, I think that this feature should be added to
launchers, not CPython, because it forces the decoder to know what
formats other than ASCII are implemented and to try heuristics to
guess, rather than just obeying the coding cookie.


From stephen at xemacs.org  Sun Nov 15 11:42:12 2015
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Mon, 16 Nov 2015 01:42:12 +0900
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <m24mgnfplb.fsf@fastmail.com>
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
 <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
 <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp>
 <m24mgnfplb.fsf@fastmail.com>
Message-ID: <22088.46564.623670.58229@turnbull.sk.tsukuba.ac.jp>

Random832 writes:
 > "Stephen J. Turnbull" <stephen at xemacs.org> writes:
 > > I don't see any good reason for allowing non-ASCII-compatible
 > > encodings in the reference CPython interpreter.
 > 
 > There might be a case for having the tokenizer not care about encodings
 > at all and just operate on a stream of unicode characters provided by a
 > different layer.

That's exactly what the PEP 263 implementation does in Python 2 (with
the caveat that Python 2 doesn't know anything about Unicode, it's a
UTF-8 stream and the non-ASCII characters are treated as bytes of
unknown semantics, so they can't be used in syntax).  I don't know
about Python 3, I haven't looked at the decoding of source programs.
But I would assume it implements PEP 263 still, except that since str
is now either widechars or PEP 393 encoding (ie, flexible widechars)
that encoding is now used instead of UTF-8.

I'm sure that there are plenty of ASCII-isms in the tokenizer in the
sense that it assumes the ASCII *character* (not byte) repertoire.
But I'm not sure why Serhiy thinks that the tokenizer cares about the
representation on-disk.  But as I say, I haven't looked at the code so
he might be right.

Steve


From guido at python.org  Sun Nov 15 12:34:50 2015
From: guido at python.org (Guido van Rossum)
Date: Sun, 15 Nov 2015 09:34:50 -0800
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <22088.46564.623670.58229@turnbull.sk.tsukuba.ac.jp>
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
 <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
 <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp>
 <m24mgnfplb.fsf@fastmail.com>
 <22088.46564.623670.58229@turnbull.sk.tsukuba.ac.jp>
Message-ID: <CAP7+vJJOu+pZ84vCnoGXnAvkV296S1qbNCkn+E_wJ=U4onwPWg@mail.gmail.com>

Let me just unilaterally end this discussion. It's fine to disregard
the future possibility of using UTF-16 or -32 for Python source code.
Serhiy can happily rip out any comments or dead code dealing with that
possibility.

-- 
--Guido van Rossum (python.org/~guido)

From p.f.moore at gmail.com  Sun Nov 15 14:26:59 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 15 Nov 2015 19:26:59 +0000
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <22088.46441.171953.459003@turnbull.sk.tsukuba.ac.jp>
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
 <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
 <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp>
 <CACac1F8STZYwwUbV_bNLmfMZarS2uidsSttbhFL1CzJi229nNA@mail.gmail.com>
 <201511151443.tAFEhwfi029674@fido.openend.se>
 <22088.46441.171953.459003@turnbull.sk.tsukuba.ac.jp>
Message-ID: <CACac1F8FTC2wf6kv_gdt2tQY37jjuePbygkaAxOktKp4veTDKQ@mail.gmail.com>

On 15 November 2015 at 16:40, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> What PEP 263 did do was to specify that non-ASCII-compatible encodings
> are not supported by the PEP 263 mechanism for declaring the encoding
> of a Python source program.  That's because it looks for a "magic
> number" which is the ASCII-encoded form of "coding:" in the first two
> lines.  It doesn't rule out alternative mechanisms for encoding
> detection (specifically, use of the UTF-16 "BOM" signature); it just
> doesn't propose implementing them.

That was my initial thought. But combine this with the statement from
the language docs that the default encoding when there is no PEP 263
encoding specification is UTF-8 (or ASCII in Python 2) and there's no
valid way that I can see that a UTF-16 encoding could be valid (short
of a formal language change).

Anyway, Guido has spoken, so I'll leave it there.

Paul

From raymond.hettinger at gmail.com  Sun Nov 15 14:57:39 2015
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sun, 15 Nov 2015 11:57:39 -0800
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <CAP7+vJJOu+pZ84vCnoGXnAvkV296S1qbNCkn+E_wJ=U4onwPWg@mail.gmail.com>
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
 <E1Zxlmo-000090-W9@se2-syd.hostedmail.net.au>
 <22088.13062.191972.33951@turnbull.sk.tsukuba.ac.jp>
 <m24mgnfplb.fsf@fastmail.com>
 <22088.46564.623670.58229@turnbull.sk.tsukuba.ac.jp>
 <CAP7+vJJOu+pZ84vCnoGXnAvkV296S1qbNCkn+E_wJ=U4onwPWg@mail.gmail.com>
Message-ID: <34F3CFE0-1F32-4BD2-A5DD-E77D2F6CC76B@gmail.com>


> On Nov 15, 2015, at 9:34 AM, Guido van Rossum <guido at python.org> wrote:
> 
> Let me just unilaterally end this discussion. It's fine to disregard
> the future possibility of using UTF-16 or -32 for Python source code.
> Serhiy can happily rip out any comments or dead code dealing with that
> possibility.

Thank you.


Raymond

From mal at egenix.com  Sun Nov 15 15:38:45 2015
From: mal at egenix.com (M.-A. Lemburg)
Date: Sun, 15 Nov 2015 21:38:45 +0100
Subject: [Python-Dev] Support of UTF-16 and UTF-32 source encodings
In-Reply-To: <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
References: <n281g9$hbq$1@ger.gmane.org>
 <CAMpsgwZN-hTcx85jVJDfN+YnbWewajnupYmmLH_E7SW_aD_UFg@mail.gmail.com>
Message-ID: <5648ED55.4020906@egenix.com>

On 14.11.2015 23:56, Victor Stinner wrote:
> These encodings are rarely used. I don't think that any text editor use
> them. Editors use ascii, latin1, utf8 and... all locale encoding. But I
> don't know any OS using UTF-16 as a locale encoding. UTF-32 wastes disk
> space.

UTF-16 is used a lot for Windows text files, e.g. Unicode
CSV files (the save as "Unicode text file" option writes
UTF-16).

However, nowadays, all text editors also support UTF-8 and
many of these recognize the UTF-8 BOM as identifier to detect
Unicode text files.

> Ok, even if it exists, Python already accepts a very wide range of
> encoding. It is not worth to make the parser much more complex just to
> support encodings which are also never used (for .py files).

Agreed. In Python 2 we decided to only allow ASCII super-sets
for Python source files, which out ruled multi-byte encodings
such as UTF-16 and -32. I don't think we need to make the parser
more complex just to support them. UTF-8 works fine as Python
source code encoding.

> Victor
> Le 14 nov. 2015 20:20, "Serhiy Storchaka" <storchaka at gmail.com> a ?crit :
> 
>> For now UTF-16 and UTF-32 source encodings are not supported. There is a
>> comment in Parser/tokenizer.c:
>>
>>     /* Disable support for UTF-16 BOMs until a decision
>>        is made whether this needs to be supported.  */
>>
>> Can we make a decision whether this support will be added in foreseeable
>> future (say in near 10 years), or no?
>>
>> Removing commented out and related code will help to refactor the
>> tokenizer, and that can help to fix some existing bugs (e.g. issue14811,
>> issue18961, issue20115 and may be others). Current tokenizing code is too
>> tangled.
>>
>> If the support of UTF-16 and UTF-32 is planned, I'll take this to
>> attention during refactoring. But in many places besides the tokenizer the
>> ASCII compatible encoding of source files is expected.
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>>
> 
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/mal%40egenix.com
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 15 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________
2015-10-23: Released mxODBC Connect 2.1.5 ...     http://egenix.com/go85

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From ncoghlan at gmail.com  Sun Nov 15 18:48:27 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 16 Nov 2015 09:48:27 +1000
Subject: [Python-Dev] Clarification of PEP 394 for scripts that run
 under Python 2 and 3
In-Reply-To: <20151113175727.338f192c@limelight.wooz.org>
References: <CANw+MznRKh-AyF4ObSf_0GaepZgtpAAdqJBw3eFCqc8PgxQzyw@mail.gmail.com>
 <20151113175727.338f192c@limelight.wooz.org>
Message-ID: <CADiSq7c1YNLbW-DuZG+Ws_NpswoBKp_+nsHmNSxrPag3X4nWqw@mail.gmail.com>

On 14 November 2015 at 08:57, Barry Warsaw <barry at python.org> wrote:
> On Nov 13, 2015, at 10:32 PM, Damien George wrote:
>
>>1. What is the true intent of PEP 394 when only Python 3 is installed?  Is
>>"python" available or not to run scripts compatible with 2.x and 3.x?
>>
>>2. Is it possible to write a shebang line that supports all variations of
>>Python installations on *nix machines?
>>
>>3. If the answer to 2 is no, then what is the recommended way to support
>>all Python installations with one standalone script?
>
> It's important to remember that PEP 394 is an informational PEP, still under
> active (if dormant) discussion.  It is also just a recommendation, and can't
> force any downstream redistributors to do one thing or another.  Still, the
> intent is to provide a set of guidelines for the majority of *nix distributors
> to (eventually) adopt.
>
> I'll also point you to the recently created linux-sig where this topic is
> being discussed.
>
> https://mail.python.org/pipermail/linux-sig/2015-October/000000.html
>
> As you've noticed, Arch took a particular approach, but speaking as part of
> the Debian/Ubuntu community, don't expect that ecosystem to go down the same
> path.  It's highly unlikely /usr/bin/python will ever point to anything other
> than Python 2, even when only Python 3 is installed by default.  That might
> change once Python 2.7 is actually EOL'd, and that won't happen for quite a
> long time.  PEP 373 currently says there will be bug fix releases until 2020,
> and I'd expect security-only releases for some time after that.

This is the same situation for the Fedora/RHEL/CentOS ecosystem - for
the time being, we think failure -> "yum install /usr/bin/python" is
the cleanest user experience we can readily offer for Python 3 only
systems. However, we don't think it's an acceptable long term answer,
and something *else* needs to be done with /usr/bin/python than just
making it a symlink to Python 3 (since the latter will lead to cryptic
error messages for Python 2 only scripts).

> In general, the discussions on linux-sig and elsewhere is coalescing around a
> launcher-type approach, where you'd put something like /usr/bin/py in your
> shebang if your script really is bilingual.  Then `py` can try to figure out
> what's available on your system, what your preference is, etc. and then
> execute your script using that version-specific binary.  I would expect the
> launcher to eventually be provided by the upstream Python development
> community.

A couple of other links folks may find worth reading for background:

* LWN summary of the discussion at this year's language summit:
https://lwn.net/Articles/640296/
* Geoffrey Thomas's launcher concept:
https://ldpreload.com/blog/usr-bin-python-23

This thread prompted me to do an update pass over
https://wiki.python.org/moin/Python3LinuxDistroPortingStatus, adding
these background links to the Python symlink section, and updating the
Fedora porting status link to refer to the recently published porting
DB: http://portingdb-encukou.rhcloud.com/

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From brett at python.org  Mon Nov 16 15:18:02 2015
From: brett at python.org (Brett Cannon)
Date: Mon, 16 Nov 2015 20:18:02 +0000
Subject: [Python-Dev] Benchmark results across all major Python
 implementations
Message-ID: <CAP1=2W52Dq8EfFC=eA5GU73s+mRD10j=K+GLk4HTWYyLK2DZ1w@mail.gmail.com>

I gave the opening keynote at PyCon CA and then gave the same talk at
PyData NYC on the various interpreters of Python (Jupyter notebook of my
presentation can be found at bit.ly/pycon-ca-keynote; no video yet). I
figured people here might find the benchmark numbers interesting so I'm
sharing the link here.

I'm still hoping someday speed.python.org becomes a thing so I never have
to spend so much time benchmarking so may Python implementations ever again
and this sort of thing is just part of what we do to keep the
implementation ecosystem healthy.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151116/f15fa256/attachment.html>

From fijall at gmail.com  Mon Nov 16 15:23:49 2015
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 16 Nov 2015 21:23:49 +0100
Subject: [Python-Dev] Benchmark results across all major Python
 implementations
In-Reply-To: <CAP1=2W52Dq8EfFC=eA5GU73s+mRD10j=K+GLk4HTWYyLK2DZ1w@mail.gmail.com>
References: <CAP1=2W52Dq8EfFC=eA5GU73s+mRD10j=K+GLk4HTWYyLK2DZ1w@mail.gmail.com>
Message-ID: <CAK5idxQV19fDYZjbf4trEi9ttDH8KnySL0Ehq566A3HkBu7x5Q@mail.gmail.com>

Hi Brett

Any thoughts on improving the benchmark set (I think all of
{cpython,pypy,pyston} introduced new benchmarks to the set).
"speed.python.org" becoming a thing is generally stopped on "noone
cares enough to set it up".

Cheers,
fijal


On Mon, Nov 16, 2015 at 9:18 PM, Brett Cannon <brett at python.org> wrote:
> I gave the opening keynote at PyCon CA and then gave the same talk at PyData
> NYC on the various interpreters of Python (Jupyter notebook of my
> presentation can be found at bit.ly/pycon-ca-keynote; no video yet). I
> figured people here might find the benchmark numbers interesting so I'm
> sharing the link here.
>
> I'm still hoping someday speed.python.org becomes a thing so I never have to
> spend so much time benchmarking so may Python implementations ever again and
> this sort of thing is just part of what we do to keep the implementation
> ecosystem healthy.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>

From rdmurray at bitdance.com  Mon Nov 16 15:47:49 2015
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 16 Nov 2015 15:47:49 -0500
Subject: [Python-Dev] Benchmark results across all major Python
 implementations
In-Reply-To: <CAK5idxQV19fDYZjbf4trEi9ttDH8KnySL0Ehq566A3HkBu7x5Q@mail.gmail.com>
References: <CAP1=2W52Dq8EfFC=eA5GU73s+mRD10j=K+GLk4HTWYyLK2DZ1w@mail.gmail.com>
 <CAK5idxQV19fDYZjbf4trEi9ttDH8KnySL0Ehq566A3HkBu7x5Q@mail.gmail.com>
Message-ID: <20151116204749.DBFE9B14091@webabinitio.net>

On Mon, 16 Nov 2015 21:23:49 +0100, Maciej Fijalkowski <fijall at gmail.com> wrote:
> Any thoughts on improving the benchmark set (I think all of
> {cpython,pypy,pyston} introduced new benchmarks to the set).
> "speed.python.org" becoming a thing is generally stopped on "noone
> cares enough to set it up".

Actually, with some help from Intel, it is getting there.  You can see
the 'benchmarks' entry in the buildbot console:

    http://buildbot.python.org/all/console

but it isn't quite working yet.  We are also waiting on a review of the
salt state:

    https://github.com/python/psf-salt/pull/74

(All work done by Zach Ware.)

--David

From brett at python.org  Mon Nov 16 15:51:35 2015
From: brett at python.org (Brett Cannon)
Date: Mon, 16 Nov 2015 20:51:35 +0000
Subject: [Python-Dev] Benchmark results across all major Python
 implementations
In-Reply-To: <CAK5idxQV19fDYZjbf4trEi9ttDH8KnySL0Ehq566A3HkBu7x5Q@mail.gmail.com>
References: <CAP1=2W52Dq8EfFC=eA5GU73s+mRD10j=K+GLk4HTWYyLK2DZ1w@mail.gmail.com>
 <CAK5idxQV19fDYZjbf4trEi9ttDH8KnySL0Ehq566A3HkBu7x5Q@mail.gmail.com>
Message-ID: <CAP1=2W4bTdR0yB7Ltq6+L9uoS=CsKhj4s03vi0rdAhBovb53Bg@mail.gmail.com>

On Mon, 16 Nov 2015 at 12:24 Maciej Fijalkowski <fijall at gmail.com> wrote:

> Hi Brett
>
> Any thoughts on improving the benchmark set (I think all of
> {cpython,pypy,pyston} introduced new benchmarks to the set).
>

We should probably start a mailing list and finally hash out a common set
of benchmarks that we all agree are reasonable for measuring performance. I
think we all generally agree that high-level benchmarks are good and
micro-benchmarks aren't that important for cross-implementation comparisons
(obviously they have a usefulness when trying to work on a specific feature
set, but that should be considered specific to an implementation and not to
some globally accepted set of benchmarks). So we should identify benchmarks
which somewhat represent real world workloads and try to have a balanced
representation that doesn't lean one way or another (e.g., not all string
manipulation or scientific computing, but both should obviously be
represented).


> "speed.python.org" becoming a thing is generally stopped on "noone
> cares enough to set it up".
>

Oh, I know. I didn't say this could be considered wishful thinking since I
know I have enough on my plate to prevent me from making it happen.

-Brett


>
> Cheers,
> fijal
>
>
> On Mon, Nov 16, 2015 at 9:18 PM, Brett Cannon <brett at python.org> wrote:
> > I gave the opening keynote at PyCon CA and then gave the same talk at
> PyData
> > NYC on the various interpreters of Python (Jupyter notebook of my
> > presentation can be found at bit.ly/pycon-ca-keynote; no video yet). I
> > figured people here might find the benchmark numbers interesting so I'm
> > sharing the link here.
> >
> > I'm still hoping someday speed.python.org becomes a thing so I never
> have to
> > spend so much time benchmarking so may Python implementations ever again
> and
> > this sort of thing is just part of what we do to keep the implementation
> > ecosystem healthy.
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151116/90112786/attachment.html>

From brian at python.org  Mon Nov 16 16:38:05 2015
From: brian at python.org (Brian Curtin)
Date: Mon, 16 Nov 2015 16:38:05 -0500
Subject: [Python-Dev] Benchmark results across all major Python
 implementations
In-Reply-To: <CAP1=2W4bTdR0yB7Ltq6+L9uoS=CsKhj4s03vi0rdAhBovb53Bg@mail.gmail.com>
References: <CAP1=2W52Dq8EfFC=eA5GU73s+mRD10j=K+GLk4HTWYyLK2DZ1w@mail.gmail.com>
 <CAK5idxQV19fDYZjbf4trEi9ttDH8KnySL0Ehq566A3HkBu7x5Q@mail.gmail.com>
 <CAP1=2W4bTdR0yB7Ltq6+L9uoS=CsKhj4s03vi0rdAhBovb53Bg@mail.gmail.com>
Message-ID: <CAD+XWwoGJBRjgB1G1ZSD=4GRr5+bG1QZxRa=MGySc0GoaV1qKg@mail.gmail.com>

On Monday, November 16, 2015, Brett Cannon <brett at python.org
<javascript:_e(%7B%7D,'cvml','brett at python.org');>> wrote:

>
>
> On Mon, 16 Nov 2015 at 12:24 Maciej Fijalkowski <fijall at gmail.com> wrote:
>
>> Hi Brett
>>
>> Any thoughts on improving the benchmark set (I think all of
>> {cpython,pypy,pyston} introduced new benchmarks to the set).
>>
>
> We should probably start a mailing list
>

There is/was a speed at python.org list.


> "speed.python.org" becoming a thing is generally stopped on "noone
>> cares enough to set it up".
>>
>
> Oh, I know. I didn't say this could be considered wishful thinking since I
> know I have enough on my plate to prevent me from making it happen.
>

There was a grant given years ago to improve some of this stuff but I don't
believe the work ever saw the light of day.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151116/41b656c5/attachment.html>

From brian at python.org  Mon Nov 16 17:16:25 2015
From: brian at python.org (Brian Curtin)
Date: Mon, 16 Nov 2015 17:16:25 -0500
Subject: [Python-Dev] Benchmark results across all major Python
 implementations
In-Reply-To: <CAP1=2W4bTdR0yB7Ltq6+L9uoS=CsKhj4s03vi0rdAhBovb53Bg@mail.gmail.com>
References: <CAP1=2W52Dq8EfFC=eA5GU73s+mRD10j=K+GLk4HTWYyLK2DZ1w@mail.gmail.com>
 <CAK5idxQV19fDYZjbf4trEi9ttDH8KnySL0Ehq566A3HkBu7x5Q@mail.gmail.com>
 <CAP1=2W4bTdR0yB7Ltq6+L9uoS=CsKhj4s03vi0rdAhBovb53Bg@mail.gmail.com>
Message-ID: <CAD+XWwoQuN5xdbsLdCop3_23d+Lr9o9XF4eLH2whQ4AV_u0CvQ@mail.gmail.com>

On Monday, November 16, 2015, Brett Cannon <brett at python.org
<javascript:_e(%7B%7D,'cvml','brett at python.org');>> wrote:

>
>
> On Mon, 16 Nov 2015 at 12:24 Maciej Fijalkowski <fijall at gmail.com> wrote:
>
>> Hi Brett
>>
>> Any thoughts on improving the benchmark set (I think all of
>> {cpython,pypy,pyston} introduced new benchmarks to the set).
>>
>
> We should probably start a mailing list
>

There is/was a speed at python.org list.


> "speed.python.org" becoming a thing is generally stopped on "noone
>> cares enough to set it up".
>>
>
> Oh, I know. I didn't say this could be considered wishful thinking since I
> know I have enough on my plate to prevent me from making it happen.
>

There was a grant given years ago to improve some of this stuff but I don't
believe the work ever saw the light of day.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151116/d6a2bebe/attachment.html>

From zachary.ware+pydev at gmail.com  Mon Nov 16 17:23:51 2015
From: zachary.ware+pydev at gmail.com (Zachary Ware)
Date: Mon, 16 Nov 2015 16:23:51 -0600
Subject: [Python-Dev] Benchmark results across all major Python
 implementations
In-Reply-To: <CAD+XWwoGJBRjgB1G1ZSD=4GRr5+bG1QZxRa=MGySc0GoaV1qKg@mail.gmail.com>
References: <CAP1=2W52Dq8EfFC=eA5GU73s+mRD10j=K+GLk4HTWYyLK2DZ1w@mail.gmail.com>
 <CAK5idxQV19fDYZjbf4trEi9ttDH8KnySL0Ehq566A3HkBu7x5Q@mail.gmail.com>
 <CAP1=2W4bTdR0yB7Ltq6+L9uoS=CsKhj4s03vi0rdAhBovb53Bg@mail.gmail.com>
 <CAD+XWwoGJBRjgB1G1ZSD=4GRr5+bG1QZxRa=MGySc0GoaV1qKg@mail.gmail.com>
Message-ID: <CAKJDb-METwvW2CLEL1_YYcw03eeqRromp-vuoEE-T3ka=vO4Uw@mail.gmail.com>

On Mon, Nov 16, 2015 at 3:38 PM, Brian Curtin <brian at python.org> wrote:
> On Monday, November 16, 2015, Brett Cannon <brett at python.org> wrote:

>>>
>>> Hi Brett
>>>
>>> Any thoughts on improving the benchmark set (I think all of
>>> {cpython,pypy,pyston} introduced new benchmarks to the set).
>>
>>
>> We should probably start a mailing list
>
>
> There is/was a speed at python.org list.

Is, but seems to be heavily moderated without active moderation.  I
sent an offer to speed-owner this morning to help with moderation, but
as yet no response.  I know I have a couple of messages waiting in the
queue :)

On Mon, 16 Nov 2015 at 12:24 Maciej Fijalkowski <fijall at gmail.com> wrote:
> "speed.python.org" becoming a thing is generally stopped on "noone
> cares enough to set it up".

Setup is done in my dev environment, it's now blocking on people more
qualified than me to review and merge, then final tweaking of the
buildbot setup.  For those interested:

The part in most need of review is the changes to the PSF Salt
configuration to set up and run Codespeed on speed.python.org.  The
changes can be found in PR #74 on the psf-salt Github repo[0].

The Codespeed instance is housed in a Github repo owned by me[1]
currently.  There's one small patch to the codespeed code (which was
merged upstream this morning), the rest of the changes in my fork are
adapting a copy of the sample_project to be our own instance.  I've
been told that the grant proposal from long ago expected the use of
codespeed2 instead of codespeed.  I chose codespeed over codespeed2
because it appeared to be easier to get set up and running (which may
or may not have actually been true), but also because I've not seen
codespeed2 in action anywhere.

The final piece that could use review is the changes to our benchmark
repository, currently available in a sandbox repo on hg.python.org[2].
My original plan had been to use PyPy's benchmark suite, but as you
can tell from the logs of the existing buildslave, cpython doesn't run
that suite especially well, and the cpython suite has the advantage of
working with Python 2 and 3 out of the box.

Please have a look, give it a try if you can, and let me know what
needs improvement!

[0] https://github.com/python/psf-salt/pull/74
[1] https://github.com/zware/codespeed
[2] https://hg.python.org/sandbox/zware-benchmarks

-- 
Zach

From mail at timgolden.me.uk  Mon Nov 16 17:34:02 2015
From: mail at timgolden.me.uk (Tim Golden)
Date: Mon, 16 Nov 2015 22:34:02 +0000
Subject: [Python-Dev] Benchmark results across all major Python
 implementations
In-Reply-To: <CAKJDb-METwvW2CLEL1_YYcw03eeqRromp-vuoEE-T3ka=vO4Uw@mail.gmail.com>
References: <CAP1=2W52Dq8EfFC=eA5GU73s+mRD10j=K+GLk4HTWYyLK2DZ1w@mail.gmail.com>
 <CAK5idxQV19fDYZjbf4trEi9ttDH8KnySL0Ehq566A3HkBu7x5Q@mail.gmail.com>
 <CAP1=2W4bTdR0yB7Ltq6+L9uoS=CsKhj4s03vi0rdAhBovb53Bg@mail.gmail.com>
 <CAD+XWwoGJBRjgB1G1ZSD=4GRr5+bG1QZxRa=MGySc0GoaV1qKg@mail.gmail.com>
 <CAKJDb-METwvW2CLEL1_YYcw03eeqRromp-vuoEE-T3ka=vO4Uw@mail.gmail.com>
Message-ID: <564A59DA.2070704@timgolden.me.uk>

On 16/11/2015 22:23, Zachary Ware wrote:
> On Mon, Nov 16, 2015 at 3:38 PM, Brian Curtin <brian at python.org> wrote:
>> On Monday, November 16, 2015, Brett Cannon <brett at python.org> wrote:
>
>>>>
>>>> Hi Brett
>>>>
>>>> Any thoughts on improving the benchmark set (I think all of
>>>> {cpython,pypy,pyston} introduced new benchmarks to the set).
>>>
>>>
>>> We should probably start a mailing list
>>
>>
>> There is/was a speed at python.org list.
>
> Is, but seems to be heavily moderated without active moderation.  I
> sent an offer to speed-owner this morning to help with moderation, but
> as yet no response.  I know I have a couple of messages waiting in the
> queue :)

Just to help things along, I've added myself as list owner and released 
a bunch of messages.

TJG

From jim.baker at python.org  Mon Nov 16 17:59:54 2015
From: jim.baker at python.org (Jim Baker)
Date: Mon, 16 Nov 2015 15:59:54 -0700
Subject: [Python-Dev] Benchmark results across all major Python
 implementations
In-Reply-To: <CAP1=2W52Dq8EfFC=eA5GU73s+mRD10j=K+GLk4HTWYyLK2DZ1w@mail.gmail.com>
References: <CAP1=2W52Dq8EfFC=eA5GU73s+mRD10j=K+GLk4HTWYyLK2DZ1w@mail.gmail.com>
Message-ID: <CAOhO=aNggmjCpQMQ+u1TyU3qvRTwMC2CCMJwR560va0ZR8wffQ@mail.gmail.com>

Brett,

Very cool, I'm glad to see that Jython's performance was competitive under
most of these benchmarks. I would also be interested in joining the
proposed mailing list.

re elementtree - I assume the benchmarking is usually done with
cElementTree. However Jython currently lacks a Java equivalent, so
importing cElementTree just uses the pure Python version. Hence the
significant performance difference of approx. 40x for etree_parse and 16x
for etree_iterparse.

- Jim

On Mon, Nov 16, 2015 at 1:18 PM, Brett Cannon <brett at python.org> wrote:

> I gave the opening keynote at PyCon CA and then gave the same talk at
> PyData NYC on the various interpreters of Python (Jupyter notebook of my
> presentation can be found at bit.ly/pycon-ca-keynote; no video yet). I
> figured people here might find the benchmark numbers interesting so I'm
> sharing the link here.
>
> I'm still hoping someday speed.python.org becomes a thing so I never have
> to spend so much time benchmarking so may Python implementations ever again
> and this sort of thing is just part of what we do to keep the
> implementation ecosystem healthy.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/jbaker%40zyasoft.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151116/c4350f97/attachment.html>

From guido at python.org  Mon Nov 16 18:07:24 2015
From: guido at python.org (Guido van Rossum)
Date: Mon, 16 Nov 2015 15:07:24 -0800
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
Message-ID: <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>

So I dropped the ball on this too -- I was going to either have a look
or tell you quickly to find a BDFL-delegate, but ended up doing
neither. Skimming it now I really don't think I'm the right person to
review this. Maybe you could ask Alex Gaynor?

On Tue, Nov 10, 2015 at 4:47 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Hi folks,
>
> I have a confession to make - I dropped the ball on the HTTPS
> verification backport proposals in PEP 493, and let the upstream and
> downstream approval processes get out of sequence.
>
> As a result, the RHEL 7.2 beta released back in September incorporates
> the HTTPS verification feature backport based on the current PEP 493
> draft, even though that hasn't formally been pronounced as an Active
> recommendation by python-dev yet.
>
> Accordingly, I'm belatedly submitting it for pronouncement now:
> https://www.python.org/dev/peps/pep-0493/
>
> There's currently no BDFL-Delegate assigned, so if Guido doesn't want
> to handle it, we'll need to address that question first.
>
> Our last discussion back in July seemed to show that folks either
> didn't care about the question (because they're using unmodified
> upstream versions so the PEP didn't affect them), or else thought the
> approach described in the PEP was reasonable, so I'm hoping the
> consequences of my mistake won't be too severe.
>
> Regards,
> Nick.
>
> P.S. I'm aware that this looks like presenting a fait accompli at a
> point where it's too late to realistically say "No", but the truth is
> that preparation for the Python in Education miniconf at PyCon
> Australia ramped up immediately after the July discussion, and then I
> personally got confused as to the scope of what was being included in
> 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493
> being deferred to a later release, but it's actually the whole package
> of 466+476+493). That's my fault for trying to keep track of too many
> things at once (and thus failing at some of them), not anyone else's.
>
> ================================
>
> PEP: 493
> Title: HTTPS verification recommendations for Python 2.7 redistributors
> Version: $Revision$
> Last-Modified: $Date$
> Author: Nick Coghlan <ncoghlan at gmail.com>,
>         Robert Kuska <rkuska at redhat.com>,
>         Marc-Andr? Lemburg <mal at lemburg.com>
> Status: Draft
> Type: Informational
> Content-Type: text/x-rst
> Created: 10-May-2015
> Post-History: 06-Jul-2015
>
>
> Abstract
> ========
>
> PEP 476 updated Python's default handling of HTTPS certificates to be
> appropriate for communication over the public internet. The Python 2.7 long
> term maintenance series was judged to be in scope for this change, with the
> new behaviour introduced in the Python 2.7.9 maintenance release.
>
> This PEP provides recommendations to downstream redistributors wishing to
> provide a smoother migration experience when helping their users to manage
> this change in Python's default behaviour.
>
>
> Rationale
> =========
>
> PEP 476 changed Python's default behaviour to better match the needs and
> expectations of developers operating over the public internet, a category
> which appears to include most new Python developers. It is the position of
> the authors of this PEP that this was a correct decision.
>
> However, it is also the case that this change *does* cause problems for
> infrastructure administrators operating private intranets that rely on
> self-signed certificates, or otherwise encounter problems with the new default
> certificate verification settings.
>
> The long term answer for such environments is to update their internal
> certificate management to at least match the standards set by the public
> internet, but in the meantime, it is desirable to offer these administrators
> a way to continue receiving maintenance updates to the Python 2.7 series,
> without having to gate that on upgrades to their certificate management
> infrastructure.
>
> PEP 476 did attempt to address this question, by covering how to revert the
> new settings process wide by monkeypatching the ``ssl`` module to restore the
> old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed
> to allow system administrators to disable the feature by default in their
> Standard Operating Environment definition has been determined to be
> insufficient in at least some cases. The specific case of interest to the
> authors of this PEP is the one where a Linux distributor aims to provide
> their users with a
> `smoother migration path
> <https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__
> than the standard one provided by consuming upstream CPython 2.7 releases
> directly, but other potential challenges have also been pointed out with
> updating embedded Python runtimes and other user level installations of Python.
>
> Rather than allowing a plethora of mutually incompatibile migration techniques
> to bloom, this PEP proposes two alternative approaches that redistributors
> may take when addressing these problems. Redistributors may choose to implement
> one, both, or neither of these approaches based on their assessment of the
> needs of their particular userbase.
>
> These designs are being proposed as a recommendation for redistributors, rather
> than as new upstream features, as they are needed purely to support legacy
> environments migrating from older versions of Python 2.7. Neither approach
> is being proposed as an upstream Python 2.7 feature, nor as a feature in any
> version of Python 3 (whether published directly by the Python Software
> Foundation or by a redistributor).
>
>
> Requirements for capability detection
> =====================================
>
> As these recommendations are intended to cover backports to earlier Python
> versions, the Python version number cannot be used as a reliable means for
> detecting them. Instead, the recommendations are defined to allow the presence
> or absence of the feature to be determined using the following technique::
>
>     python -c "import ssl; ssl._relevant_attribute"
>
> This will fail with `AttributeError` (and hence a non-zero return code) if the
> relevant capability is not available.
>
> The marker attributes are prefixed with an underscore to indicate the
> implementation dependent nature of these capabilities - not all Python
> distributions will offer them, only those that are providing a multi-stage
> migration process from the legacy HTTPS handling to the new default behaviour.
>
>
> Recommendation for an environment variable based security downgrade
> ===================================================================
>
> Some redistributors may wish to provide a per-application option to disable
> certificate verification in selected applications that run on or embed CPython
> without needing to modify the application itself.
>
> In these cases, a configuration mechanism is needed that provides:
>
> * an opt-out model that allows certificate verification to be selectively
>   turned off for particular applications after upgrading to a version of
>   Python that verifies certificates by default
> * the ability for all users to configure this setting on a per-application
>   basis, rather than on a per-system, or per-Python-installation basis
>
> This approach may be used for any redistributor provided version of Python 2.7,
> including those that advertise themselves as providing Python 2.7.9 or later.
>
>
> Required marker attribute
> -------------------------
>
> The required marker attribute on the ``ssl`` module when implementing this
> recommendation is::
>
>     _https_verify_envvar = 'PYTHONHTTPSVERIFY'
>
> This not only makes it straightforward to detect the presence (or absence) of
> the capability, it also makes it possible to programmatically determine the
> relevant environment variable name.
>
>
> Recommended modifications to the Python standard library
> --------------------------------------------------------
>
> The recommended approach to providing a per-application configuration setting
> for HTTPS certificate verification that doesn't require modifications to the
> application itself is to:
>
> * modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment
>   variable when the module is first imported into a Python process
> * set the ``ssl._create_default_https_context`` function to be an alias for
>   ``ssl._create_unverified_context`` if this environment variable is present
>   and set to ``'0'``
> * otherwise, set the ``ssl._create_default_https_context`` function to be an
>   alias for ``ssl.create_default_context`` as usual
>
>
> Example implementation
> ----------------------
>
> ::
>
>     _https_verify_envvar = 'PYTHONHTTPSVERIFY'
>
>     def _get_https_context_factory():
>         config_setting = os.environ.get(_https_verify_envvar)
>         if config_setting == '0':
>             return _create_unverified_context
>         return create_default_context
>
>     _create_default_https_context = _get_https_context_factory()
>
>
> Security Considerations
> -----------------------
>
> Relative to an unmodified version of CPython 2.7.9 or later, this approach
> does introduce a new downgrade attack against the default security settings
> that potentially allows a sufficiently determined attacker to revert Python
> to the vulnerable configuration used in CPython 2.7.8 and earlier releases.
> However, such an attack requires the ability to modify the execution
> environment of a Python process prior to the import of the ``ssl`` module,
> and any attacker with such access would already be able to modify the
> behaviour of the underlying OpenSSL implementation.
>
>
> Recommendation for backporting to earlier Python versions
> =========================================================
>
> Some redistributors, most notably Linux distributions, may choose to backport
> the PEP 476 HTTPS verification changes to modified Python versions based on
> earlier Python 2 maintenance releases. In these cases, a configuration
> mechanism is needed that provides:
>
> * an opt-in model that allows the decision to enable HTTPS certificate
>   verification to be made independently of the decision to upgrade to the
>   Python version where the feature was first backported
> * the ability for system administrators to set the default behaviour of Python
>   applications and scripts run directly in the system Python installation
> * the ability for the redistributor to consider changing the default behaviour
>   of *new* installations at some point in the future without impacting existing
>   installations that have been explicitly configured to skip verifying HTTPS
>   certificates by default
>
> This approach should not be used for any Python installation that advertises
> itself as providing Python 2.7.9 or later, as most Python users will have the
> reasonable expectation that all such environments will validate HTTPS
> certificates by default.
>
>
> Required marker attribute
> -------------------------
>
> The required marker attribute on the ``ssl`` module when implementing this
> recommendation is::
>
>     _cert_verification_config = '<path to configuration file>'
>
> This not only makes it straightforward to detect the presence (or absence) of
> the capability, it also makes it possible to programmatically determine the
> relevant configuration file name.
>
>
> Recommended modifications to the Python standard library
> --------------------------------------------------------
>
> The recommended approach to backporting the PEP 476 modifications to an earlier
> point release is to implement the following changes relative to the default
> PEP 476 behaviour implemented in Python 2.7.9+:
>
> * modify the ``ssl`` module to read a system wide configuration file when the
>   module is first imported into a Python process
> * define a platform default behaviour (either verifying or not verifying HTTPS
>   certificates) to be used if this configuration file is not present
> * support selection between the following three modes of operation:
>
>   * ensure HTTPS certificate verification is enabled
>   * ensure HTTPS certificate verification is disabled
>   * delegate the decision to the redistributor providing this Python version
>
> * set the ``ssl._create_default_https_context`` function to be an alias for
>   either ``ssl.create_default_context`` or ``ssl._create_unverified_context``
>   based on the given configuration setting.
>
>
> Recommended file location
> -------------------------
>
> This approach is currently only defined for \*nix system Python installations.
>
> The recommended configuration file name is
> ``/etc/python/cert-verification.cfg``.
>
> The ``.cfg`` filename extension is recommended for consistency with the
> ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library.
>
>
> Recommended file format
> -----------------------
>
> The configuration file should use a ConfigParser ini-style format with a
> single section named ``[https]`` containing one required setting ``verify``.
>
> Permitted values for ``verify`` are:
>
> * ``enable``: ensure HTTPS certificate verification is enabled by default
> * ``disable``: ensure HTTPS certificate verification is disabled by default
> * ``platform_default``: delegate the decision to the redistributor providing
>   this particular Python version
>
> If the ``[https]`` section or the ``verify`` setting are missing, or if the
> ``verify`` setting is set to an unknown value, it should be treated as if the
> configuration file is not present.
>
>
> Example implementation
> ----------------------
>
> ::
>
>     _cert_verification_config = '/etc/python/cert-verification.cfg'
>
>     def _get_https_context_factory():
>         # Check for a system-wide override of the default behaviour
>         context_factories = {
>             'enable': create_default_context,
>             'disable': _create_unverified_context,
>             'platform_default': _create_unverified_context, # For now :)
>         }
>         import ConfigParser
>         config = ConfigParser.RawConfigParser()
>         config.read(_cert_verification_config)
>         try:
>             verify_mode = config.get('https', 'verify')
>         except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
>             verify_mode = 'platform_default'
>         default_factory = context_factories.get('platform_default')
>         return context_factories.get(verify_mode, default_factory)
>
>     _create_default_https_context = _get_https_context_factory()
>
>
> Security Considerations
> -----------------------
>
> The specific recommendations for the backporting case are designed to work for
> privileged, security sensitive processes, even those being run in the following
> locked down configuration:
>
> * run from a locked down administrator controlled directory rather than a normal
>   user directory (preventing ``sys.path[0]`` based privilege escalation attacks)
> * run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based
>   privilege escalation attacks)
> * run using the ``-s`` switch (preventing user site directory based privilege
>   escalation attacks)
> * run using the ``-S`` switch (preventing ``sitecustomize`` based privilege
>   escalation attacks)
>
> The intent is that the *only* reason HTTPS verification should be getting
> turned off system wide when using this approach is because:
>
> * an end user is running a redistributor provided version of CPython rather
>   than running upstream CPython directly
> * that redistributor has decided to provide a smoother migration path to
>   verifying HTTPS certificates by default than that being provided by the
>   upstream project
> * either the redistributor or the local infrastructure administrator has
>   determined that it is appropriate to override the default upstream behaviour
>   (at least for the time being)
>
> Using an administrator controlled configuration file rather than an environment
> variable has the essential feature of providing a smoother migration path, even
> for applications being run with the ``-E`` switch.
>
>
> Combining the recommendations
> =============================
>
> If a redistributor chooses to implement both recommendations, then the
> environment variable should take precedence over the system-wide configuration
> setting. This allows the setting to be changed for a given user, virtual
> environment or application, regardless of the system-wide default behaviour.
>
> In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and
> set to anything *other* than ``'0'``, then HTTPS certificate verification
> should be enabled.
>
> Example implementation
> ----------------------
>
> ::
>
>     _https_verify_envvar = 'PYTHONHTTPSVERIFY'
>     _cert_verification_config = '/etc/python/cert-verification.cfg'
>
>     def _get_https_context_factory():
>         # Check for am environmental override of the default behaviour
>         config_setting = os.environ.get(_https_verify_envvar)
>         if config_setting is not None:
>             if config_setting == '0':
>                 return _create_unverified_context
>             return create_default_context
>
>         # Check for a system-wide override of the default behaviour
>         context_factories = {
>             'enable': create_default_context,
>             'disable': _create_unverified_context,
>             'platform_default': _create_unverified_context, # For now :)
>         }
>         import ConfigParser
>         config = ConfigParser.RawConfigParser()
>         config.read(_cert_verification_config)
>         try:
>             verify_mode = config.get('https', 'verify')
>         except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
>             verify_mode = 'platform_default'
>         default_factory = context_factories.get('platform_default')
>         return context_factories.get(verify_mode, default_factory)
>
>     _create_default_https_context = _get_https_context_factory()
>
>
> Copyright
> =========
>
> This document has been placed into the public domain.
>
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From david.c.stewart at intel.com  Mon Nov 16 18:37:06 2015
From: david.c.stewart at intel.com (Stewart, David C)
Date: Mon, 16 Nov 2015 23:37:06 +0000
Subject: [Python-Dev] Benchmark results across all major Python
 implementations
In-Reply-To: <CAP1=2W52Dq8EfFC=eA5GU73s+mRD10j=K+GLk4HTWYyLK2DZ1w@mail.gmail.com>
References: <CAP1=2W52Dq8EfFC=eA5GU73s+mRD10j=K+GLk4HTWYyLK2DZ1w@mail.gmail.com>
Message-ID: <CABA9BEA-27EE-4F9B-8AD0-5C9B1243E93A@intel.com>

Last June we started publishing a daily performance report of the latest Python tip against the previous day's run and some established synch point. We mail these to the community to act as a "canary in the coal mine." I wrote about it at https://01.org/lp/blog/0-day-challenge-what-pulse-internet

You can see our manager-style dashboard of a couple of key workloads at http://languagesperformance.intel.com/
(I have this running constantly on a dedicated screen in my office).

Would love to get better workloads if we can.

Dave

From: Python-Dev on behalf of Brett Cannon
Date: Monday, November 16, 2015 at 12:18 PM
To: python-dev
Subject: [Python-Dev] Benchmark results across all major Python implementations

I gave the opening keynote at PyCon CA and then gave the same talk at PyData NYC on the various interpreters of Python (Jupyter notebook of my presentation can be found at bit.ly/pycon-ca-keynote<http://bit.ly/pycon-ca-keynote>; no video yet). I figured people here might find the benchmark numbers interesting so I'm sharing the link here.

I'm still hoping someday speed.python.org<http://speed.python.org> becomes a thing so I never have to spend so much time benchmarking so may Python implementations ever again and this sort of thing is just part of what we do to keep the implementation ecosystem healthy.

From ncoghlan at gmail.com  Mon Nov 16 18:54:45 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 17 Nov 2015 09:54:45 +1000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
Message-ID: <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>

On 17 November 2015 at 09:07, Guido van Rossum <guido at python.org> wrote:
> So I dropped the ball on this too -- I was going to either have a look
> or tell you quickly to find a BDFL-delegate, but ended up doing
> neither. Skimming it now I really don't think I'm the right person to
> review this. Maybe you could ask Alex Gaynor?

If Alex is interested, that could definitely work. I've also asked
Christian Heimes if he might be interested in handling it, as he
worked on several of the SSL module changes that were backported, and
he also works on FreeIPA [1] now, which will potentially be affected
by the changes to RHEL & CentOS 7.

Regards,
Nick.

[1] FreeIPA is an open source Identity Management & Authentication
system: http://www.freeipa.org/

>
> On Tue, Nov 10, 2015 at 4:47 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Hi folks,
>>
>> I have a confession to make - I dropped the ball on the HTTPS
>> verification backport proposals in PEP 493, and let the upstream and
>> downstream approval processes get out of sequence.
>>
>> As a result, the RHEL 7.2 beta released back in September incorporates
>> the HTTPS verification feature backport based on the current PEP 493
>> draft, even though that hasn't formally been pronounced as an Active
>> recommendation by python-dev yet.
>>
>> Accordingly, I'm belatedly submitting it for pronouncement now:
>> https://www.python.org/dev/peps/pep-0493/
>>
>> There's currently no BDFL-Delegate assigned, so if Guido doesn't want
>> to handle it, we'll need to address that question first.
>>
>> Our last discussion back in July seemed to show that folks either
>> didn't care about the question (because they're using unmodified
>> upstream versions so the PEP didn't affect them), or else thought the
>> approach described in the PEP was reasonable, so I'm hoping the
>> consequences of my mistake won't be too severe.
>>
>> Regards,
>> Nick.
>>
>> P.S. I'm aware that this looks like presenting a fait accompli at a
>> point where it's too late to realistically say "No", but the truth is
>> that preparation for the Python in Education miniconf at PyCon
>> Australia ramped up immediately after the July discussion, and then I
>> personally got confused as to the scope of what was being included in
>> 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493
>> being deferred to a later release, but it's actually the whole package
>> of 466+476+493). That's my fault for trying to keep track of too many
>> things at once (and thus failing at some of them), not anyone else's.
>>
>> ================================
>>
>> PEP: 493
>> Title: HTTPS verification recommendations for Python 2.7 redistributors
>> Version: $Revision$
>> Last-Modified: $Date$
>> Author: Nick Coghlan <ncoghlan at gmail.com>,
>>         Robert Kuska <rkuska at redhat.com>,
>>         Marc-Andr? Lemburg <mal at lemburg.com>
>> Status: Draft
>> Type: Informational
>> Content-Type: text/x-rst
>> Created: 10-May-2015
>> Post-History: 06-Jul-2015
>>
>>
>> Abstract
>> ========
>>
>> PEP 476 updated Python's default handling of HTTPS certificates to be
>> appropriate for communication over the public internet. The Python 2.7 long
>> term maintenance series was judged to be in scope for this change, with the
>> new behaviour introduced in the Python 2.7.9 maintenance release.
>>
>> This PEP provides recommendations to downstream redistributors wishing to
>> provide a smoother migration experience when helping their users to manage
>> this change in Python's default behaviour.
>>
>>
>> Rationale
>> =========
>>
>> PEP 476 changed Python's default behaviour to better match the needs and
>> expectations of developers operating over the public internet, a category
>> which appears to include most new Python developers. It is the position of
>> the authors of this PEP that this was a correct decision.
>>
>> However, it is also the case that this change *does* cause problems for
>> infrastructure administrators operating private intranets that rely on
>> self-signed certificates, or otherwise encounter problems with the new default
>> certificate verification settings.
>>
>> The long term answer for such environments is to update their internal
>> certificate management to at least match the standards set by the public
>> internet, but in the meantime, it is desirable to offer these administrators
>> a way to continue receiving maintenance updates to the Python 2.7 series,
>> without having to gate that on upgrades to their certificate management
>> infrastructure.
>>
>> PEP 476 did attempt to address this question, by covering how to revert the
>> new settings process wide by monkeypatching the ``ssl`` module to restore the
>> old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed
>> to allow system administrators to disable the feature by default in their
>> Standard Operating Environment definition has been determined to be
>> insufficient in at least some cases. The specific case of interest to the
>> authors of this PEP is the one where a Linux distributor aims to provide
>> their users with a
>> `smoother migration path
>> <https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__
>> than the standard one provided by consuming upstream CPython 2.7 releases
>> directly, but other potential challenges have also been pointed out with
>> updating embedded Python runtimes and other user level installations of Python.
>>
>> Rather than allowing a plethora of mutually incompatibile migration techniques
>> to bloom, this PEP proposes two alternative approaches that redistributors
>> may take when addressing these problems. Redistributors may choose to implement
>> one, both, or neither of these approaches based on their assessment of the
>> needs of their particular userbase.
>>
>> These designs are being proposed as a recommendation for redistributors, rather
>> than as new upstream features, as they are needed purely to support legacy
>> environments migrating from older versions of Python 2.7. Neither approach
>> is being proposed as an upstream Python 2.7 feature, nor as a feature in any
>> version of Python 3 (whether published directly by the Python Software
>> Foundation or by a redistributor).
>>
>>
>> Requirements for capability detection
>> =====================================
>>
>> As these recommendations are intended to cover backports to earlier Python
>> versions, the Python version number cannot be used as a reliable means for
>> detecting them. Instead, the recommendations are defined to allow the presence
>> or absence of the feature to be determined using the following technique::
>>
>>     python -c "import ssl; ssl._relevant_attribute"
>>
>> This will fail with `AttributeError` (and hence a non-zero return code) if the
>> relevant capability is not available.
>>
>> The marker attributes are prefixed with an underscore to indicate the
>> implementation dependent nature of these capabilities - not all Python
>> distributions will offer them, only those that are providing a multi-stage
>> migration process from the legacy HTTPS handling to the new default behaviour.
>>
>>
>> Recommendation for an environment variable based security downgrade
>> ===================================================================
>>
>> Some redistributors may wish to provide a per-application option to disable
>> certificate verification in selected applications that run on or embed CPython
>> without needing to modify the application itself.
>>
>> In these cases, a configuration mechanism is needed that provides:
>>
>> * an opt-out model that allows certificate verification to be selectively
>>   turned off for particular applications after upgrading to a version of
>>   Python that verifies certificates by default
>> * the ability for all users to configure this setting on a per-application
>>   basis, rather than on a per-system, or per-Python-installation basis
>>
>> This approach may be used for any redistributor provided version of Python 2.7,
>> including those that advertise themselves as providing Python 2.7.9 or later.
>>
>>
>> Required marker attribute
>> -------------------------
>>
>> The required marker attribute on the ``ssl`` module when implementing this
>> recommendation is::
>>
>>     _https_verify_envvar = 'PYTHONHTTPSVERIFY'
>>
>> This not only makes it straightforward to detect the presence (or absence) of
>> the capability, it also makes it possible to programmatically determine the
>> relevant environment variable name.
>>
>>
>> Recommended modifications to the Python standard library
>> --------------------------------------------------------
>>
>> The recommended approach to providing a per-application configuration setting
>> for HTTPS certificate verification that doesn't require modifications to the
>> application itself is to:
>>
>> * modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment
>>   variable when the module is first imported into a Python process
>> * set the ``ssl._create_default_https_context`` function to be an alias for
>>   ``ssl._create_unverified_context`` if this environment variable is present
>>   and set to ``'0'``
>> * otherwise, set the ``ssl._create_default_https_context`` function to be an
>>   alias for ``ssl.create_default_context`` as usual
>>
>>
>> Example implementation
>> ----------------------
>>
>> ::
>>
>>     _https_verify_envvar = 'PYTHONHTTPSVERIFY'
>>
>>     def _get_https_context_factory():
>>         config_setting = os.environ.get(_https_verify_envvar)
>>         if config_setting == '0':
>>             return _create_unverified_context
>>         return create_default_context
>>
>>     _create_default_https_context = _get_https_context_factory()
>>
>>
>> Security Considerations
>> -----------------------
>>
>> Relative to an unmodified version of CPython 2.7.9 or later, this approach
>> does introduce a new downgrade attack against the default security settings
>> that potentially allows a sufficiently determined attacker to revert Python
>> to the vulnerable configuration used in CPython 2.7.8 and earlier releases.
>> However, such an attack requires the ability to modify the execution
>> environment of a Python process prior to the import of the ``ssl`` module,
>> and any attacker with such access would already be able to modify the
>> behaviour of the underlying OpenSSL implementation.
>>
>>
>> Recommendation for backporting to earlier Python versions
>> =========================================================
>>
>> Some redistributors, most notably Linux distributions, may choose to backport
>> the PEP 476 HTTPS verification changes to modified Python versions based on
>> earlier Python 2 maintenance releases. In these cases, a configuration
>> mechanism is needed that provides:
>>
>> * an opt-in model that allows the decision to enable HTTPS certificate
>>   verification to be made independently of the decision to upgrade to the
>>   Python version where the feature was first backported
>> * the ability for system administrators to set the default behaviour of Python
>>   applications and scripts run directly in the system Python installation
>> * the ability for the redistributor to consider changing the default behaviour
>>   of *new* installations at some point in the future without impacting existing
>>   installations that have been explicitly configured to skip verifying HTTPS
>>   certificates by default
>>
>> This approach should not be used for any Python installation that advertises
>> itself as providing Python 2.7.9 or later, as most Python users will have the
>> reasonable expectation that all such environments will validate HTTPS
>> certificates by default.
>>
>>
>> Required marker attribute
>> -------------------------
>>
>> The required marker attribute on the ``ssl`` module when implementing this
>> recommendation is::
>>
>>     _cert_verification_config = '<path to configuration file>'
>>
>> This not only makes it straightforward to detect the presence (or absence) of
>> the capability, it also makes it possible to programmatically determine the
>> relevant configuration file name.
>>
>>
>> Recommended modifications to the Python standard library
>> --------------------------------------------------------
>>
>> The recommended approach to backporting the PEP 476 modifications to an earlier
>> point release is to implement the following changes relative to the default
>> PEP 476 behaviour implemented in Python 2.7.9+:
>>
>> * modify the ``ssl`` module to read a system wide configuration file when the
>>   module is first imported into a Python process
>> * define a platform default behaviour (either verifying or not verifying HTTPS
>>   certificates) to be used if this configuration file is not present
>> * support selection between the following three modes of operation:
>>
>>   * ensure HTTPS certificate verification is enabled
>>   * ensure HTTPS certificate verification is disabled
>>   * delegate the decision to the redistributor providing this Python version
>>
>> * set the ``ssl._create_default_https_context`` function to be an alias for
>>   either ``ssl.create_default_context`` or ``ssl._create_unverified_context``
>>   based on the given configuration setting.
>>
>>
>> Recommended file location
>> -------------------------
>>
>> This approach is currently only defined for \*nix system Python installations.
>>
>> The recommended configuration file name is
>> ``/etc/python/cert-verification.cfg``.
>>
>> The ``.cfg`` filename extension is recommended for consistency with the
>> ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library.
>>
>>
>> Recommended file format
>> -----------------------
>>
>> The configuration file should use a ConfigParser ini-style format with a
>> single section named ``[https]`` containing one required setting ``verify``.
>>
>> Permitted values for ``verify`` are:
>>
>> * ``enable``: ensure HTTPS certificate verification is enabled by default
>> * ``disable``: ensure HTTPS certificate verification is disabled by default
>> * ``platform_default``: delegate the decision to the redistributor providing
>>   this particular Python version
>>
>> If the ``[https]`` section or the ``verify`` setting are missing, or if the
>> ``verify`` setting is set to an unknown value, it should be treated as if the
>> configuration file is not present.
>>
>>
>> Example implementation
>> ----------------------
>>
>> ::
>>
>>     _cert_verification_config = '/etc/python/cert-verification.cfg'
>>
>>     def _get_https_context_factory():
>>         # Check for a system-wide override of the default behaviour
>>         context_factories = {
>>             'enable': create_default_context,
>>             'disable': _create_unverified_context,
>>             'platform_default': _create_unverified_context, # For now :)
>>         }
>>         import ConfigParser
>>         config = ConfigParser.RawConfigParser()
>>         config.read(_cert_verification_config)
>>         try:
>>             verify_mode = config.get('https', 'verify')
>>         except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
>>             verify_mode = 'platform_default'
>>         default_factory = context_factories.get('platform_default')
>>         return context_factories.get(verify_mode, default_factory)
>>
>>     _create_default_https_context = _get_https_context_factory()
>>
>>
>> Security Considerations
>> -----------------------
>>
>> The specific recommendations for the backporting case are designed to work for
>> privileged, security sensitive processes, even those being run in the following
>> locked down configuration:
>>
>> * run from a locked down administrator controlled directory rather than a normal
>>   user directory (preventing ``sys.path[0]`` based privilege escalation attacks)
>> * run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based
>>   privilege escalation attacks)
>> * run using the ``-s`` switch (preventing user site directory based privilege
>>   escalation attacks)
>> * run using the ``-S`` switch (preventing ``sitecustomize`` based privilege
>>   escalation attacks)
>>
>> The intent is that the *only* reason HTTPS verification should be getting
>> turned off system wide when using this approach is because:
>>
>> * an end user is running a redistributor provided version of CPython rather
>>   than running upstream CPython directly
>> * that redistributor has decided to provide a smoother migration path to
>>   verifying HTTPS certificates by default than that being provided by the
>>   upstream project
>> * either the redistributor or the local infrastructure administrator has
>>   determined that it is appropriate to override the default upstream behaviour
>>   (at least for the time being)
>>
>> Using an administrator controlled configuration file rather than an environment
>> variable has the essential feature of providing a smoother migration path, even
>> for applications being run with the ``-E`` switch.
>>
>>
>> Combining the recommendations
>> =============================
>>
>> If a redistributor chooses to implement both recommendations, then the
>> environment variable should take precedence over the system-wide configuration
>> setting. This allows the setting to be changed for a given user, virtual
>> environment or application, regardless of the system-wide default behaviour.
>>
>> In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and
>> set to anything *other* than ``'0'``, then HTTPS certificate verification
>> should be enabled.
>>
>> Example implementation
>> ----------------------
>>
>> ::
>>
>>     _https_verify_envvar = 'PYTHONHTTPSVERIFY'
>>     _cert_verification_config = '/etc/python/cert-verification.cfg'
>>
>>     def _get_https_context_factory():
>>         # Check for am environmental override of the default behaviour
>>         config_setting = os.environ.get(_https_verify_envvar)
>>         if config_setting is not None:
>>             if config_setting == '0':
>>                 return _create_unverified_context
>>             return create_default_context
>>
>>         # Check for a system-wide override of the default behaviour
>>         context_factories = {
>>             'enable': create_default_context,
>>             'disable': _create_unverified_context,
>>             'platform_default': _create_unverified_context, # For now :)
>>         }
>>         import ConfigParser
>>         config = ConfigParser.RawConfigParser()
>>         config.read(_cert_verification_config)
>>         try:
>>             verify_mode = config.get('https', 'verify')
>>         except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
>>             verify_mode = 'platform_default'
>>         default_factory = context_factories.get('platform_default')
>>         return context_factories.get(verify_mode, default_factory)
>>
>>     _create_default_https_context = _get_https_context_factory()
>>
>>
>> Copyright
>> =========
>>
>> This document has been placed into the public domain.
>>
>>
>> --
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
> --
> --Guido van Rossum (python.org/~guido)



-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From guido at python.org  Mon Nov 16 19:00:46 2015
From: guido at python.org (Guido van Rossum)
Date: Mon, 16 Nov 2015 16:00:46 -0800
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
Message-ID: <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>

Hm, making Christian the BDFL-delegate would mean two out of three
authors *and* the BDFL-delegate all working for Red Hat, which clearly
has a stake (and IIUC has already committed to this approach ahead of
PEP approval). SO then it would look like this is just rubber-stamping
Red Hat's internal decision process (if it's a process -- sounds more
like an accident :-).

So, Alex, do you want to approve this PEP?

On Mon, Nov 16, 2015 at 3:54 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 17 November 2015 at 09:07, Guido van Rossum <guido at python.org> wrote:
>> So I dropped the ball on this too -- I was going to either have a look
>> or tell you quickly to find a BDFL-delegate, but ended up doing
>> neither. Skimming it now I really don't think I'm the right person to
>> review this. Maybe you could ask Alex Gaynor?
>
> If Alex is interested, that could definitely work. I've also asked
> Christian Heimes if he might be interested in handling it, as he
> worked on several of the SSL module changes that were backported, and
> he also works on FreeIPA [1] now, which will potentially be affected
> by the changes to RHEL & CentOS 7.
>
> Regards,
> Nick.
>
> [1] FreeIPA is an open source Identity Management & Authentication
> system: http://www.freeipa.org/
>
>>
>> On Tue, Nov 10, 2015 at 4:47 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> Hi folks,
>>>
>>> I have a confession to make - I dropped the ball on the HTTPS
>>> verification backport proposals in PEP 493, and let the upstream and
>>> downstream approval processes get out of sequence.
>>>
>>> As a result, the RHEL 7.2 beta released back in September incorporates
>>> the HTTPS verification feature backport based on the current PEP 493
>>> draft, even though that hasn't formally been pronounced as an Active
>>> recommendation by python-dev yet.
>>>
>>> Accordingly, I'm belatedly submitting it for pronouncement now:
>>> https://www.python.org/dev/peps/pep-0493/
>>>
>>> There's currently no BDFL-Delegate assigned, so if Guido doesn't want
>>> to handle it, we'll need to address that question first.
>>>
>>> Our last discussion back in July seemed to show that folks either
>>> didn't care about the question (because they're using unmodified
>>> upstream versions so the PEP didn't affect them), or else thought the
>>> approach described in the PEP was reasonable, so I'm hoping the
>>> consequences of my mistake won't be too severe.
>>>
>>> Regards,
>>> Nick.
>>>
>>> P.S. I'm aware that this looks like presenting a fait accompli at a
>>> point where it's too late to realistically say "No", but the truth is
>>> that preparation for the Python in Education miniconf at PyCon
>>> Australia ramped up immediately after the July discussion, and then I
>>> personally got confused as to the scope of what was being included in
>>> 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493
>>> being deferred to a later release, but it's actually the whole package
>>> of 466+476+493). That's my fault for trying to keep track of too many
>>> things at once (and thus failing at some of them), not anyone else's.
>>>
>>> ================================
>>>
>>> PEP: 493
>>> Title: HTTPS verification recommendations for Python 2.7 redistributors
>>> Version: $Revision$
>>> Last-Modified: $Date$
>>> Author: Nick Coghlan <ncoghlan at gmail.com>,
>>>         Robert Kuska <rkuska at redhat.com>,
>>>         Marc-Andr? Lemburg <mal at lemburg.com>
>>> Status: Draft
>>> Type: Informational
>>> Content-Type: text/x-rst
>>> Created: 10-May-2015
>>> Post-History: 06-Jul-2015
>>>
>>>
>>> Abstract
>>> ========
>>>
>>> PEP 476 updated Python's default handling of HTTPS certificates to be
>>> appropriate for communication over the public internet. The Python 2.7 long
>>> term maintenance series was judged to be in scope for this change, with the
>>> new behaviour introduced in the Python 2.7.9 maintenance release.
>>>
>>> This PEP provides recommendations to downstream redistributors wishing to
>>> provide a smoother migration experience when helping their users to manage
>>> this change in Python's default behaviour.
>>>
>>>
>>> Rationale
>>> =========
>>>
>>> PEP 476 changed Python's default behaviour to better match the needs and
>>> expectations of developers operating over the public internet, a category
>>> which appears to include most new Python developers. It is the position of
>>> the authors of this PEP that this was a correct decision.
>>>
>>> However, it is also the case that this change *does* cause problems for
>>> infrastructure administrators operating private intranets that rely on
>>> self-signed certificates, or otherwise encounter problems with the new default
>>> certificate verification settings.
>>>
>>> The long term answer for such environments is to update their internal
>>> certificate management to at least match the standards set by the public
>>> internet, but in the meantime, it is desirable to offer these administrators
>>> a way to continue receiving maintenance updates to the Python 2.7 series,
>>> without having to gate that on upgrades to their certificate management
>>> infrastructure.
>>>
>>> PEP 476 did attempt to address this question, by covering how to revert the
>>> new settings process wide by monkeypatching the ``ssl`` module to restore the
>>> old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed
>>> to allow system administrators to disable the feature by default in their
>>> Standard Operating Environment definition has been determined to be
>>> insufficient in at least some cases. The specific case of interest to the
>>> authors of this PEP is the one where a Linux distributor aims to provide
>>> their users with a
>>> `smoother migration path
>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__
>>> than the standard one provided by consuming upstream CPython 2.7 releases
>>> directly, but other potential challenges have also been pointed out with
>>> updating embedded Python runtimes and other user level installations of Python.
>>>
>>> Rather than allowing a plethora of mutually incompatibile migration techniques
>>> to bloom, this PEP proposes two alternative approaches that redistributors
>>> may take when addressing these problems. Redistributors may choose to implement
>>> one, both, or neither of these approaches based on their assessment of the
>>> needs of their particular userbase.
>>>
>>> These designs are being proposed as a recommendation for redistributors, rather
>>> than as new upstream features, as they are needed purely to support legacy
>>> environments migrating from older versions of Python 2.7. Neither approach
>>> is being proposed as an upstream Python 2.7 feature, nor as a feature in any
>>> version of Python 3 (whether published directly by the Python Software
>>> Foundation or by a redistributor).
>>>
>>>
>>> Requirements for capability detection
>>> =====================================
>>>
>>> As these recommendations are intended to cover backports to earlier Python
>>> versions, the Python version number cannot be used as a reliable means for
>>> detecting them. Instead, the recommendations are defined to allow the presence
>>> or absence of the feature to be determined using the following technique::
>>>
>>>     python -c "import ssl; ssl._relevant_attribute"
>>>
>>> This will fail with `AttributeError` (and hence a non-zero return code) if the
>>> relevant capability is not available.
>>>
>>> The marker attributes are prefixed with an underscore to indicate the
>>> implementation dependent nature of these capabilities - not all Python
>>> distributions will offer them, only those that are providing a multi-stage
>>> migration process from the legacy HTTPS handling to the new default behaviour.
>>>
>>>
>>> Recommendation for an environment variable based security downgrade
>>> ===================================================================
>>>
>>> Some redistributors may wish to provide a per-application option to disable
>>> certificate verification in selected applications that run on or embed CPython
>>> without needing to modify the application itself.
>>>
>>> In these cases, a configuration mechanism is needed that provides:
>>>
>>> * an opt-out model that allows certificate verification to be selectively
>>>   turned off for particular applications after upgrading to a version of
>>>   Python that verifies certificates by default
>>> * the ability for all users to configure this setting on a per-application
>>>   basis, rather than on a per-system, or per-Python-installation basis
>>>
>>> This approach may be used for any redistributor provided version of Python 2.7,
>>> including those that advertise themselves as providing Python 2.7.9 or later.
>>>
>>>
>>> Required marker attribute
>>> -------------------------
>>>
>>> The required marker attribute on the ``ssl`` module when implementing this
>>> recommendation is::
>>>
>>>     _https_verify_envvar = 'PYTHONHTTPSVERIFY'
>>>
>>> This not only makes it straightforward to detect the presence (or absence) of
>>> the capability, it also makes it possible to programmatically determine the
>>> relevant environment variable name.
>>>
>>>
>>> Recommended modifications to the Python standard library
>>> --------------------------------------------------------
>>>
>>> The recommended approach to providing a per-application configuration setting
>>> for HTTPS certificate verification that doesn't require modifications to the
>>> application itself is to:
>>>
>>> * modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment
>>>   variable when the module is first imported into a Python process
>>> * set the ``ssl._create_default_https_context`` function to be an alias for
>>>   ``ssl._create_unverified_context`` if this environment variable is present
>>>   and set to ``'0'``
>>> * otherwise, set the ``ssl._create_default_https_context`` function to be an
>>>   alias for ``ssl.create_default_context`` as usual
>>>
>>>
>>> Example implementation
>>> ----------------------
>>>
>>> ::
>>>
>>>     _https_verify_envvar = 'PYTHONHTTPSVERIFY'
>>>
>>>     def _get_https_context_factory():
>>>         config_setting = os.environ.get(_https_verify_envvar)
>>>         if config_setting == '0':
>>>             return _create_unverified_context
>>>         return create_default_context
>>>
>>>     _create_default_https_context = _get_https_context_factory()
>>>
>>>
>>> Security Considerations
>>> -----------------------
>>>
>>> Relative to an unmodified version of CPython 2.7.9 or later, this approach
>>> does introduce a new downgrade attack against the default security settings
>>> that potentially allows a sufficiently determined attacker to revert Python
>>> to the vulnerable configuration used in CPython 2.7.8 and earlier releases.
>>> However, such an attack requires the ability to modify the execution
>>> environment of a Python process prior to the import of the ``ssl`` module,
>>> and any attacker with such access would already be able to modify the
>>> behaviour of the underlying OpenSSL implementation.
>>>
>>>
>>> Recommendation for backporting to earlier Python versions
>>> =========================================================
>>>
>>> Some redistributors, most notably Linux distributions, may choose to backport
>>> the PEP 476 HTTPS verification changes to modified Python versions based on
>>> earlier Python 2 maintenance releases. In these cases, a configuration
>>> mechanism is needed that provides:
>>>
>>> * an opt-in model that allows the decision to enable HTTPS certificate
>>>   verification to be made independently of the decision to upgrade to the
>>>   Python version where the feature was first backported
>>> * the ability for system administrators to set the default behaviour of Python
>>>   applications and scripts run directly in the system Python installation
>>> * the ability for the redistributor to consider changing the default behaviour
>>>   of *new* installations at some point in the future without impacting existing
>>>   installations that have been explicitly configured to skip verifying HTTPS
>>>   certificates by default
>>>
>>> This approach should not be used for any Python installation that advertises
>>> itself as providing Python 2.7.9 or later, as most Python users will have the
>>> reasonable expectation that all such environments will validate HTTPS
>>> certificates by default.
>>>
>>>
>>> Required marker attribute
>>> -------------------------
>>>
>>> The required marker attribute on the ``ssl`` module when implementing this
>>> recommendation is::
>>>
>>>     _cert_verification_config = '<path to configuration file>'
>>>
>>> This not only makes it straightforward to detect the presence (or absence) of
>>> the capability, it also makes it possible to programmatically determine the
>>> relevant configuration file name.
>>>
>>>
>>> Recommended modifications to the Python standard library
>>> --------------------------------------------------------
>>>
>>> The recommended approach to backporting the PEP 476 modifications to an earlier
>>> point release is to implement the following changes relative to the default
>>> PEP 476 behaviour implemented in Python 2.7.9+:
>>>
>>> * modify the ``ssl`` module to read a system wide configuration file when the
>>>   module is first imported into a Python process
>>> * define a platform default behaviour (either verifying or not verifying HTTPS
>>>   certificates) to be used if this configuration file is not present
>>> * support selection between the following three modes of operation:
>>>
>>>   * ensure HTTPS certificate verification is enabled
>>>   * ensure HTTPS certificate verification is disabled
>>>   * delegate the decision to the redistributor providing this Python version
>>>
>>> * set the ``ssl._create_default_https_context`` function to be an alias for
>>>   either ``ssl.create_default_context`` or ``ssl._create_unverified_context``
>>>   based on the given configuration setting.
>>>
>>>
>>> Recommended file location
>>> -------------------------
>>>
>>> This approach is currently only defined for \*nix system Python installations.
>>>
>>> The recommended configuration file name is
>>> ``/etc/python/cert-verification.cfg``.
>>>
>>> The ``.cfg`` filename extension is recommended for consistency with the
>>> ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library.
>>>
>>>
>>> Recommended file format
>>> -----------------------
>>>
>>> The configuration file should use a ConfigParser ini-style format with a
>>> single section named ``[https]`` containing one required setting ``verify``.
>>>
>>> Permitted values for ``verify`` are:
>>>
>>> * ``enable``: ensure HTTPS certificate verification is enabled by default
>>> * ``disable``: ensure HTTPS certificate verification is disabled by default
>>> * ``platform_default``: delegate the decision to the redistributor providing
>>>   this particular Python version
>>>
>>> If the ``[https]`` section or the ``verify`` setting are missing, or if the
>>> ``verify`` setting is set to an unknown value, it should be treated as if the
>>> configuration file is not present.
>>>
>>>
>>> Example implementation
>>> ----------------------
>>>
>>> ::
>>>
>>>     _cert_verification_config = '/etc/python/cert-verification.cfg'
>>>
>>>     def _get_https_context_factory():
>>>         # Check for a system-wide override of the default behaviour
>>>         context_factories = {
>>>             'enable': create_default_context,
>>>             'disable': _create_unverified_context,
>>>             'platform_default': _create_unverified_context, # For now :)
>>>         }
>>>         import ConfigParser
>>>         config = ConfigParser.RawConfigParser()
>>>         config.read(_cert_verification_config)
>>>         try:
>>>             verify_mode = config.get('https', 'verify')
>>>         except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
>>>             verify_mode = 'platform_default'
>>>         default_factory = context_factories.get('platform_default')
>>>         return context_factories.get(verify_mode, default_factory)
>>>
>>>     _create_default_https_context = _get_https_context_factory()
>>>
>>>
>>> Security Considerations
>>> -----------------------
>>>
>>> The specific recommendations for the backporting case are designed to work for
>>> privileged, security sensitive processes, even those being run in the following
>>> locked down configuration:
>>>
>>> * run from a locked down administrator controlled directory rather than a normal
>>>   user directory (preventing ``sys.path[0]`` based privilege escalation attacks)
>>> * run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based
>>>   privilege escalation attacks)
>>> * run using the ``-s`` switch (preventing user site directory based privilege
>>>   escalation attacks)
>>> * run using the ``-S`` switch (preventing ``sitecustomize`` based privilege
>>>   escalation attacks)
>>>
>>> The intent is that the *only* reason HTTPS verification should be getting
>>> turned off system wide when using this approach is because:
>>>
>>> * an end user is running a redistributor provided version of CPython rather
>>>   than running upstream CPython directly
>>> * that redistributor has decided to provide a smoother migration path to
>>>   verifying HTTPS certificates by default than that being provided by the
>>>   upstream project
>>> * either the redistributor or the local infrastructure administrator has
>>>   determined that it is appropriate to override the default upstream behaviour
>>>   (at least for the time being)
>>>
>>> Using an administrator controlled configuration file rather than an environment
>>> variable has the essential feature of providing a smoother migration path, even
>>> for applications being run with the ``-E`` switch.
>>>
>>>
>>> Combining the recommendations
>>> =============================
>>>
>>> If a redistributor chooses to implement both recommendations, then the
>>> environment variable should take precedence over the system-wide configuration
>>> setting. This allows the setting to be changed for a given user, virtual
>>> environment or application, regardless of the system-wide default behaviour.
>>>
>>> In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and
>>> set to anything *other* than ``'0'``, then HTTPS certificate verification
>>> should be enabled.
>>>
>>> Example implementation
>>> ----------------------
>>>
>>> ::
>>>
>>>     _https_verify_envvar = 'PYTHONHTTPSVERIFY'
>>>     _cert_verification_config = '/etc/python/cert-verification.cfg'
>>>
>>>     def _get_https_context_factory():
>>>         # Check for am environmental override of the default behaviour
>>>         config_setting = os.environ.get(_https_verify_envvar)
>>>         if config_setting is not None:
>>>             if config_setting == '0':
>>>                 return _create_unverified_context
>>>             return create_default_context
>>>
>>>         # Check for a system-wide override of the default behaviour
>>>         context_factories = {
>>>             'enable': create_default_context,
>>>             'disable': _create_unverified_context,
>>>             'platform_default': _create_unverified_context, # For now :)
>>>         }
>>>         import ConfigParser
>>>         config = ConfigParser.RawConfigParser()
>>>         config.read(_cert_verification_config)
>>>         try:
>>>             verify_mode = config.get('https', 'verify')
>>>         except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
>>>             verify_mode = 'platform_default'
>>>         default_factory = context_factories.get('platform_default')
>>>         return context_factories.get(verify_mode, default_factory)
>>>
>>>     _create_default_https_context = _get_https_context_factory()
>>>
>>>
>>> Copyright
>>> =========
>>>
>>> This document has been placed into the public domain.
>>>
>>>
>>> --
>>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>
>
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia



-- 
--Guido van Rossum (python.org/~guido)

From storchaka at gmail.com  Mon Nov 16 20:53:32 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 17 Nov 2015 03:53:32 +0200
Subject: [Python-Dev] Reading Python source file
Message-ID: <n2e1as$9f7$1@ger.gmane.org>

I'm working on rewriting Python tokenizer (in particular the part that 
reads and decodes Python source file). The code is complicated. For now 
there are such cases:

* Reading from the string in memory.
* Interactive reading from the file.
* Reading from the file:
   - Raw reading ignoring encoding in parser generator.
   - Raw reading UTF-8 encoded file.
   - Reading and recoding to UTF-8.

The file is read by the line. It makes hard to check correctness of the 
first line if the encoding is specified in the second line. And it makes 
very hard problems with null bytes and with desynchronizing buffered C 
and Python files. All this problems can be easily solved if read all 
Python source file in memory and then parse it as string. This would 
allow to drop a large complex and buggy part of code.

Are there disadvantages in this solution? As for memory consumption, the 
source text itself will consume only small part of the memory consumed 
by AST tree and other structures. As for performance, reading and 
decoding all file can be faster then by the line.

[1] http://bugs.python.org/issue25643


From guido at python.org  Mon Nov 16 22:00:39 2015
From: guido at python.org (Guido van Rossum)
Date: Mon, 16 Nov 2015 19:00:39 -0800
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <n2e1as$9f7$1@ger.gmane.org>
References: <n2e1as$9f7$1@ger.gmane.org>
Message-ID: <CAP7+vJ+N1KZYkH_LtefqORZNWNwTzpi9FpGJ-aCYcFgpsbSmzw@mail.gmail.com>

If you free the memory used for the source buffer before starting code
generation you should be good.

On Mon, Nov 16, 2015 at 5:53 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> I'm working on rewriting Python tokenizer (in particular the part that reads
> and decodes Python source file). The code is complicated. For now there are
> such cases:
>
> * Reading from the string in memory.
> * Interactive reading from the file.
> * Reading from the file:
>   - Raw reading ignoring encoding in parser generator.
>   - Raw reading UTF-8 encoded file.
>   - Reading and recoding to UTF-8.
>
> The file is read by the line. It makes hard to check correctness of the
> first line if the encoding is specified in the second line. And it makes
> very hard problems with null bytes and with desynchronizing buffered C and
> Python files. All this problems can be easily solved if read all Python
> source file in memory and then parse it as string. This would allow to drop
> a large complex and buggy part of code.
>
> Are there disadvantages in this solution? As for memory consumption, the
> source text itself will consume only small part of the memory consumed by
> AST tree and other structures. As for performance, reading and decoding all
> file can be faster then by the line.
>
> [1] http://bugs.python.org/issue25643
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From python at mrabarnett.plus.com  Mon Nov 16 22:05:33 2015
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 17 Nov 2015 03:05:33 +0000
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <n2e1as$9f7$1@ger.gmane.org>
References: <n2e1as$9f7$1@ger.gmane.org>
Message-ID: <564A997D.2080706@mrabarnett.plus.com>

On 2015-11-17 01:53, Serhiy Storchaka wrote:
> I'm working on rewriting Python tokenizer (in particular the part that
> reads and decodes Python source file). The code is complicated. For now
> there are such cases:
>
> * Reading from the string in memory.
> * Interactive reading from the file.
> * Reading from the file:
>     - Raw reading ignoring encoding in parser generator.
>     - Raw reading UTF-8 encoded file.
>     - Reading and recoding to UTF-8.
>
> The file is read by the line. It makes hard to check correctness of the
> first line if the encoding is specified in the second line. And it makes
> very hard problems with null bytes and with desynchronizing buffered C
> and Python files. All this problems can be easily solved if read all
> Python source file in memory and then parse it as string. This would
> allow to drop a large complex and buggy part of code.
>
> Are there disadvantages in this solution? As for memory consumption, the
> source text itself will consume only small part of the memory consumed
> by AST tree and other structures. As for performance, reading and
> decoding all file can be faster then by the line.
>
> [1] http://bugs.python.org/issue25643
>
As I understand it, *nix expects the shebang to be b'#!', which means 
that the
first line should be ASCII-compatible (it's possible that the UTF-8 BOM 
might
be present). This kind of suggests that encodings like UTF-16 would cause a
problem on such systems.

The encoding line also needs to be ASCII-compatible.

I believe that the recent thread "Support of UTF-16 and UTF-32 source
encodings" also concluded that UTF-16 and UTF-32 shouldn't be supported.

This means that you could treat the first 2 lines as though they were some
kind of extended ASCII (Latin-1?), the line ending being '\n' or '\r' or
'\r\n'.

Once you'd identify the encoding, you could decode everything (including the
shebang line) using that encoding.

(What should happen if the encoding line then decoded differently, i.e.
encoding_line.decode(encoding) != encoding_line.decode('latin-1')?)


From random832 at fastmail.com  Mon Nov 16 23:09:51 2015
From: random832 at fastmail.com (Random832)
Date: Mon, 16 Nov 2015 23:09:51 -0500
Subject: [Python-Dev] Reading Python source file
References: <n2e1as$9f7$1@ger.gmane.org> <564A997D.2080706@mrabarnett.plus.com>
Message-ID: <m2h9klnu5s.fsf@fastmail.com>

MRAB <python at mrabarnett.plus.com> writes:
> As I understand it, *nix expects the shebang to be b'#!',
> which means that the first line should be ASCII-compatible
> (it's possible that the UTF-8 BOM might be present).

The UTF-8 BOM interferes with it on Mac OSX and Linux, at
least.


From brett at snarky.ca  Mon Nov 16 13:49:03 2015
From: brett at snarky.ca (Brett Cannon)
Date: Mon, 16 Nov 2015 18:49:03 +0000
Subject: [Python-Dev] [Python-checkins] Daily reference leaks
 (97e2a6810f7f): sum=10
In-Reply-To: <20151116084238.37478.4533@psf.io>
References: <20151116084238.37478.4533@psf.io>
Message-ID: <CAP1=2W7qA3CdJC1rrveu7wae5VEYcttr3XKpCPwZDp=CpXSEfw@mail.gmail.com>

Just an FYI there seems to be a consistent, minor refcount leak found by
test_capi that has been there for what seems like a couple of weeks.

On Mon, 16 Nov 2015 at 00:42 <solipsis at pitrou.net> wrote:

> results for 97e2a6810f7f on branch "default"
> --------------------------------------------
>
> test_asyncio leaked [0, 0, 3] memory blocks, sum=3
> test_capi leaked [1, 1, 1] references, sum=3
> test_functools leaked [0, 2, 2] memory blocks, sum=4
>
>
> Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R',
> '3:3:/home/psf-users/antoine/refleaks/reflogBLsY2a', '--timeout', '7200']
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> https://mail.python.org/mailman/listinfo/python-checkins
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151116/07d473a6/attachment.html>

From mal at egenix.com  Tue Nov 17 04:59:06 2015
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 17 Nov 2015 10:59:06 +0100
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <n2e1as$9f7$1@ger.gmane.org>
References: <n2e1as$9f7$1@ger.gmane.org>
Message-ID: <564AFA6A.8010505@egenix.com>

On 17.11.2015 02:53, Serhiy Storchaka wrote:
> I'm working on rewriting Python tokenizer (in particular the part that reads and decodes Python
> source file). The code is complicated. For now there are such cases:
> 
> * Reading from the string in memory.
> * Interactive reading from the file.
> * Reading from the file:
>   - Raw reading ignoring encoding in parser generator.
>   - Raw reading UTF-8 encoded file.
>   - Reading and recoding to UTF-8.
> 
> The file is read by the line. It makes hard to check correctness of the first line if the encoding
> is specified in the second line. And it makes very hard problems with null bytes and with
> desynchronizing buffered C and Python files. All this problems can be easily solved if read all
> Python source file in memory and then parse it as string. This would allow to drop a large complex
> and buggy part of code.
> 
> Are there disadvantages in this solution? As for memory consumption, the source text itself will
> consume only small part of the memory consumed by AST tree and other structures. As for performance,
> reading and decoding all file can be faster then by the line.

A problem with this approach is that you can no
longer fail early and detect indentation errors et al. while
parsing the data (which may well come from a pipe).

Another related problem is that you have to wait for the full
input data before you can start compiling the code.

I don't think these situations are all that common, though,
so reading in the full source code before compiling it
sounds like a reasonable approach.

We use the same simplification in eGenix PyRun's emulation of
the Python command line interface and it has so far not
caused any problems.

> [1] http://bugs.python.org/issue25643

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 17 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________
2015-10-23: Released mxODBC Connect 2.1.5 ...     http://egenix.com/go85

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From victor.stinner at gmail.com  Tue Nov 17 05:24:07 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 17 Nov 2015 11:24:07 +0100
Subject: [Python-Dev] [Python-checkins] Daily reference leaks
 (97e2a6810f7f): sum=10
In-Reply-To: <CAP1=2W7qA3CdJC1rrveu7wae5VEYcttr3XKpCPwZDp=CpXSEfw@mail.gmail.com>
References: <20151116084238.37478.4533@psf.io>
 <CAP1=2W7qA3CdJC1rrveu7wae5VEYcttr3XKpCPwZDp=CpXSEfw@mail.gmail.com>
Message-ID: <CAMpsgwYm0ZV4wSNy8bPH6YZJWj7fhEWzbObDhK0npW=4QawK+A@mail.gmail.com>

Hi,

I opened the http://bugs.python.org/issue25645

It looks like a simple regression in _pickle. I wrote a patch.

Victor

2015-11-16 19:49 GMT+01:00 Brett Cannon <brett at snarky.ca>:
> Just an FYI there seems to be a consistent, minor refcount leak found by
> test_capi that has been there for what seems like a couple of weeks.
>
> On Mon, 16 Nov 2015 at 00:42 <solipsis at pitrou.net> wrote:
>>
>> results for 97e2a6810f7f on branch "default"
>> --------------------------------------------
>>
>> test_asyncio leaked [0, 0, 3] memory blocks, sum=3
>> test_capi leaked [1, 1, 1] references, sum=3
>> test_functools leaked [0, 2, 2] memory blocks, sum=4
>>
>>
>> Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R',
>> '3:3:/home/psf-users/antoine/refleaks/reflogBLsY2a', '--timeout', '7200']
>> _______________________________________________
>> Python-checkins mailing list
>> Python-checkins at python.org
>> https://mail.python.org/mailman/listinfo/python-checkins
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>

From victor.stinner at gmail.com  Tue Nov 17 05:33:30 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 17 Nov 2015 11:33:30 +0100
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
Message-ID: <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>

2015-11-17 1:00 GMT+01:00 Guido van Rossum <guido at python.org>:
> Hm, making Christian the BDFL-delegate would mean two out of three
> authors *and* the BDFL-delegate all working for Red Hat, which clearly
> has a stake (and IIUC has already committed to this approach ahead of
> PEP approval). SO then it would look like this is just rubber-stamping
> Red Hat's internal decision process (if it's a process -- sounds more
> like an accident :-).

Can we try to get a vote from maintainers of the Python2/3 packages of
other Linux distributions? Debian, Ubuntu, OpenSUSE, etc.?

Victor

From ncoghlan at gmail.com  Tue Nov 17 08:44:50 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 17 Nov 2015 23:44:50 +1000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
Message-ID: <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>

On 17 November 2015 at 20:33, Victor Stinner <victor.stinner at gmail.com> wrote:
> 2015-11-17 1:00 GMT+01:00 Guido van Rossum <guido at python.org>:
>> Hm, making Christian the BDFL-delegate would mean two out of three
>> authors *and* the BDFL-delegate all working for Red Hat, which clearly
>> has a stake (and IIUC has already committed to this approach ahead of
>> PEP approval). SO then it would look like this is just rubber-stamping
>> Red Hat's internal decision process (if it's a process -- sounds more
>> like an accident :-).
>
> Can we try to get a vote from maintainers of the Python2/3 packages of
> other Linux distributions? Debian, Ubuntu, OpenSUSE, etc.?

I know Oracle were interested based on a discussion between them and a
member of Red Hat's product security team about it on oss-security,
but their devs never followed up on it upstream (even after an
explicit suggestion that they do so), so I'm interpreting that as
willingness to go along with whatever happens in RHEL.

For Debian, Ubuntu and SUSE, their original determinations for the
relevant CVE were "too intrusive to backport", so folks currently need
to upgrade to newer versions of those distros to get the improved
default behaviour:

* http://people.canonical.com/~ubuntu-security/cve/2014/CVE-2014-9365.html
* https://security-tracker.debian.org/tracker/CVE-2014-9365
* https://www.suse.com/security/cve/CVE-2014-9365.html

If having an opt-in backwards-compatible-by-default approach available
(albeit as a PEP 466+476+493 patch set in the RHEL/CentOS system
Python 2.7.5 package) prompts other distro security teams to
reconsider those initial assessments, that would be a nice outcome,
but it isn't my own main priority (so Guido makes a good point in
favouring finding a non-Red-Hatter willing to act as BDFL-Delegate)

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From guido at python.org  Tue Nov 17 10:22:37 2015
From: guido at python.org (Guido van Rossum)
Date: Tue, 17 Nov 2015 07:22:37 -0800
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <564AFA6A.8010505@egenix.com>
References: <n2e1as$9f7$1@ger.gmane.org> <564AFA6A.8010505@egenix.com>
Message-ID: <CAP7+vJKDyPmO9-0WxtFxoGSGvtPeUBm2sBzo7G=aS5g4e3eM4w@mail.gmail.com>

On Tue, Nov 17, 2015 at 1:59 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> On 17.11.2015 02:53, Serhiy Storchaka wrote:
>> I'm working on rewriting Python tokenizer (in particular the part that reads and decodes Python
>> source file). The code is complicated. For now there are such cases:
>>
>> * Reading from the string in memory.
>> * Interactive reading from the file.
>> * Reading from the file:
>>   - Raw reading ignoring encoding in parser generator.
>>   - Raw reading UTF-8 encoded file.
>>   - Reading and recoding to UTF-8.
>>
>> The file is read by the line. It makes hard to check correctness of the first line if the encoding
>> is specified in the second line. And it makes very hard problems with null bytes and with
>> desynchronizing buffered C and Python files. All this problems can be easily solved if read all
>> Python source file in memory and then parse it as string. This would allow to drop a large complex
>> and buggy part of code.
>>
>> Are there disadvantages in this solution? As for memory consumption, the source text itself will
>> consume only small part of the memory consumed by AST tree and other structures. As for performance,
>> reading and decoding all file can be faster then by the line.
>
> A problem with this approach is that you can no
> longer fail early and detect indentation errors et al. while
> parsing the data (which may well come from a pipe).

Oh, this use case I had forgotten about. I don't know how common or
important it is though.

But more important is the interactive REPL, which parses your input
fully each time you hit ENTER.

> Another related problem is that you have to wait for the full
> input data before you can start compiling the code.

That's always the case -- we don't start compiling before we have the
full parse tree.

> I don't think these situations are all that common, though,
> so reading in the full source code before compiling it
> sounds like a reasonable approach.
>
> We use the same simplification in eGenix PyRun's emulation of
> the Python command line interface and it has so far not
> caused any problems.

Curious how you do it? I'd actually be quite disappointed if the
amount of parsing done by the standard REPL went down.

>> [1] http://bugs.python.org/issue25643

-- 
--Guido van Rossum (python.org/~guido)

From storchaka at gmail.com  Tue Nov 17 10:40:37 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 17 Nov 2015 17:40:37 +0200
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <CAP7+vJ+N1KZYkH_LtefqORZNWNwTzpi9FpGJ-aCYcFgpsbSmzw@mail.gmail.com>
References: <n2e1as$9f7$1@ger.gmane.org>
 <CAP7+vJ+N1KZYkH_LtefqORZNWNwTzpi9FpGJ-aCYcFgpsbSmzw@mail.gmail.com>
Message-ID: <n2fhpl$q6k$1@ger.gmane.org>

On 17.11.15 05:00, Guido van Rossum wrote:
> If you free the memory used for the source buffer before starting code
> generation you should be good.

Thank you. The buffer is freed just after the end of generating AST.

> On Mon, Nov 16, 2015 at 5:53 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
>> I'm working on rewriting Python tokenizer (in particular the part that reads
>> and decodes Python source file). The code is complicated. For now there are
>> such cases:
>>
>> * Reading from the string in memory.
>> * Interactive reading from the file.
>> * Reading from the file:
>>    - Raw reading ignoring encoding in parser generator.
>>    - Raw reading UTF-8 encoded file.
>>    - Reading and recoding to UTF-8.
>>
>> The file is read by the line. It makes hard to check correctness of the
>> first line if the encoding is specified in the second line. And it makes
>> very hard problems with null bytes and with desynchronizing buffered C and
>> Python files. All this problems can be easily solved if read all Python
>> source file in memory and then parse it as string. This would allow to drop
>> a large complex and buggy part of code.
>>
>> Are there disadvantages in this solution? As for memory consumption, the
>> source text itself will consume only small part of the memory consumed by
>> AST tree and other structures. As for performance, reading and decoding all
>> file can be faster then by the line.
>>
>> [1] http://bugs.python.org/issue25643
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>



From guido at python.org  Tue Nov 17 11:06:01 2015
From: guido at python.org (Guido van Rossum)
Date: Tue, 17 Nov 2015 08:06:01 -0800
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <n2fhpl$q6k$1@ger.gmane.org>
References: <n2e1as$9f7$1@ger.gmane.org>
 <CAP7+vJ+N1KZYkH_LtefqORZNWNwTzpi9FpGJ-aCYcFgpsbSmzw@mail.gmail.com>
 <n2fhpl$q6k$1@ger.gmane.org>
Message-ID: <CAP7+vJLWbiymvB16LoCbAe8jZHhdMga6ei5WSyJmN8qz1uiPmA@mail.gmail.com>

OK, but what are you going to do about the interactive REPL?

On Tue, Nov 17, 2015 at 7:40 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> On 17.11.15 05:00, Guido van Rossum wrote:
>>
>> If you free the memory used for the source buffer before starting code
>> generation you should be good.
>
>
> Thank you. The buffer is freed just after the end of generating AST.
>
>
>> On Mon, Nov 16, 2015 at 5:53 PM, Serhiy Storchaka <storchaka at gmail.com>
>> wrote:
>>>
>>> I'm working on rewriting Python tokenizer (in particular the part that
>>> reads
>>> and decodes Python source file). The code is complicated. For now there
>>> are
>>> such cases:
>>>
>>> * Reading from the string in memory.
>>> * Interactive reading from the file.
>>> * Reading from the file:
>>>    - Raw reading ignoring encoding in parser generator.
>>>    - Raw reading UTF-8 encoded file.
>>>    - Reading and recoding to UTF-8.
>>>
>>> The file is read by the line. It makes hard to check correctness of the
>>> first line if the encoding is specified in the second line. And it makes
>>> very hard problems with null bytes and with desynchronizing buffered C
>>> and
>>> Python files. All this problems can be easily solved if read all Python
>>> source file in memory and then parse it as string. This would allow to
>>> drop
>>> a large complex and buggy part of code.
>>>
>>> Are there disadvantages in this solution? As for memory consumption, the
>>> source text itself will consume only small part of the memory consumed by
>>> AST tree and other structures. As for performance, reading and decoding
>>> all
>>> file can be faster then by the line.
>>>
>>> [1] http://bugs.python.org/issue25643
>>>
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>>
>>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From storchaka at gmail.com  Tue Nov 17 11:06:17 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 17 Nov 2015 18:06:17 +0200
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <564A997D.2080706@mrabarnett.plus.com>
References: <n2e1as$9f7$1@ger.gmane.org> <564A997D.2080706@mrabarnett.plus.com>
Message-ID: <n2fj9q$k2f$1@ger.gmane.org>

On 17.11.15 05:05, MRAB wrote:
> As I understand it, *nix expects the shebang to be b'#!', which means
> that the
> first line should be ASCII-compatible (it's possible that the UTF-8 BOM
> might
> be present). This kind of suggests that encodings like UTF-16 would cause a
> problem on such systems.
>
> The encoding line also needs to be ASCII-compatible.
>
> I believe that the recent thread "Support of UTF-16 and UTF-32 source
> encodings" also concluded that UTF-16 and UTF-32 shouldn't be supported.
>
> This means that you could treat the first 2 lines as though they were some
> kind of extended ASCII (Latin-1?), the line ending being '\n' or '\r' or
> '\r\n'.
>
> Once you'd identify the encoding, you could decode everything (including
> the
> shebang line) using that encoding.

Yes, that is what I were going to implement (and already halfway here). 
My question is whether it is worth to complicate the code further to 
preserve reading by the line. In any case after reading the first line 
that doesn't contain neither coding cookie, nor non-comment tokens, we 
need to wait the second line.

> (What should happen if the encoding line then decoded differently, i.e.
> encoding_line.decode(encoding) != encoding_line.decode('latin-1')?)

The parser should got the line decoded with specified encoding.


From storchaka at gmail.com  Tue Nov 17 11:20:30 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 17 Nov 2015 18:20:30 +0200
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <564AFA6A.8010505@egenix.com>
References: <n2e1as$9f7$1@ger.gmane.org> <564AFA6A.8010505@egenix.com>
Message-ID: <n2fk53$305$1@ger.gmane.org>

On 17.11.15 11:59, M.-A. Lemburg wrote:
> I don't think these situations are all that common, though,
> so reading in the full source code before compiling it
> sounds like a reasonable approach.
>
> We use the same simplification in eGenix PyRun's emulation of
> the Python command line interface and it has so far not
> caused any problems.

Current implementation of import system went the same way. As a result 
importing the script as a module and running it with command line can 
have different behaviours in corner cases.



From storchaka at gmail.com  Tue Nov 17 11:26:15 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 17 Nov 2015 18:26:15 +0200
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <CAP7+vJKDyPmO9-0WxtFxoGSGvtPeUBm2sBzo7G=aS5g4e3eM4w@mail.gmail.com>
References: <n2e1as$9f7$1@ger.gmane.org> <564AFA6A.8010505@egenix.com>
 <CAP7+vJKDyPmO9-0WxtFxoGSGvtPeUBm2sBzo7G=aS5g4e3eM4w@mail.gmail.com>
Message-ID: <n2fkf8$90o$1@ger.gmane.org>

On 17.11.15 17:22, Guido van Rossum wrote:
> But more important is the interactive REPL, which parses your input
> fully each time you hit ENTER.

Interactive REPL runs different code. It is simpler that the code for 
reading from file, because it have no care about BOM or coding cookie.



From storchaka at gmail.com  Tue Nov 17 11:27:49 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 17 Nov 2015 18:27:49 +0200
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <CAP7+vJLWbiymvB16LoCbAe8jZHhdMga6ei5WSyJmN8qz1uiPmA@mail.gmail.com>
References: <n2e1as$9f7$1@ger.gmane.org>
 <CAP7+vJ+N1KZYkH_LtefqORZNWNwTzpi9FpGJ-aCYcFgpsbSmzw@mail.gmail.com>
 <n2fhpl$q6k$1@ger.gmane.org>
 <CAP7+vJLWbiymvB16LoCbAe8jZHhdMga6ei5WSyJmN8qz1uiPmA@mail.gmail.com>
Message-ID: <n2fki5$90o$2@ger.gmane.org>

On 17.11.15 18:06, Guido van Rossum wrote:
> OK, but what are you going to do about the interactive REPL?

Nothing (except some simplification). This is a separate branch of the code.



From guido at python.org  Tue Nov 17 11:50:53 2015
From: guido at python.org (Guido van Rossum)
Date: Tue, 17 Nov 2015 08:50:53 -0800
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <n2fk53$305$1@ger.gmane.org>
References: <n2e1as$9f7$1@ger.gmane.org> <564AFA6A.8010505@egenix.com>
 <n2fk53$305$1@ger.gmane.org>
Message-ID: <CAP7+vJ+voOGQBwS-o4xUbD_J3b4sF4PzRJYmA5U3nhA+etTPFQ@mail.gmail.com>

On Tue, Nov 17, 2015 at 8:20 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> On 17.11.15 11:59, M.-A. Lemburg wrote:
>>
>> I don't think these situations are all that common, though,
>> so reading in the full source code before compiling it
>> sounds like a reasonable approach.
>>
>> We use the same simplification in eGenix PyRun's emulation of
>> the Python command line interface and it has so far not
>> caused any problems.
>
> Current implementation of import system went the same way. As a result
> importing the script as a module and running it with command line can have
> different behaviours in corner cases.

I'm confused. *Of course* these two behaviors differ, since Python
uses a different __name__. Not sure how this relates to the REPL.

-- 
--Guido van Rossum (python.org/~guido)

From guido at python.org  Tue Nov 17 11:51:15 2015
From: guido at python.org (Guido van Rossum)
Date: Tue, 17 Nov 2015 08:51:15 -0800
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <n2fki5$90o$2@ger.gmane.org>
References: <n2e1as$9f7$1@ger.gmane.org>
 <CAP7+vJ+N1KZYkH_LtefqORZNWNwTzpi9FpGJ-aCYcFgpsbSmzw@mail.gmail.com>
 <n2fhpl$q6k$1@ger.gmane.org>
 <CAP7+vJLWbiymvB16LoCbAe8jZHhdMga6ei5WSyJmN8qz1uiPmA@mail.gmail.com>
 <n2fki5$90o$2@ger.gmane.org>
Message-ID: <CAP7+vJK=nZGs5CWYJGXZj1KsZOegwtussZpr0QLk6+UTAbF5Dg@mail.gmail.com>

Oh, cool! Sorry for the disturbance.

On Tue, Nov 17, 2015 at 8:27 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> On 17.11.15 18:06, Guido van Rossum wrote:
>>
>> OK, but what are you going to do about the interactive REPL?
>
>
> Nothing (except some simplification). This is a separate branch of the code.
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From rdmurray at bitdance.com  Tue Nov 17 13:40:32 2015
From: rdmurray at bitdance.com (R. David Murray)
Date: Tue, 17 Nov 2015 13:40:32 -0500
Subject: [Python-Dev] Benchmark results across all major Python
 implementations
In-Reply-To: <CABA9BEA-27EE-4F9B-8AD0-5C9B1243E93A@intel.com>
References: <CAP1=2W52Dq8EfFC=eA5GU73s+mRD10j=K+GLk4HTWYyLK2DZ1w@mail.gmail.com>
 <CABA9BEA-27EE-4F9B-8AD0-5C9B1243E93A@intel.com>
Message-ID: <20151117184034.1C19FB14089@webabinitio.net>

On Mon, 16 Nov 2015 23:37:06 +0000, "Stewart, David C" <david.c.stewart at intel.com> wrote:
> Last June we started publishing a daily performance report of the latest Python tip against the previous day's run and some established synch point. We mail these to the community to act as a "canary in the coal mine." I wrote about it at https://01.org/lp/blog/0-day-challenge-what-pulse-internet
> 
> You can see our manager-style dashboard of a couple of key workloads at http://languagesperformance.intel.com/
> (I have this running constantly on a dedicated screen in my office).

Just took a look at this.  Pretty cool.  The web page is a bit confusing,
though.  It doesn't give any clue as to what is being measured by the
numbers presented...it isn't obvious whether those downward sloping
lines represent progress or regression.  Also, the intro talks about
historical data, but other than the older dates[*] in the graph there's
no access to it.  Do you have plans to provide access to the raw data?
It also doesn't show all of the test shown in the example email in your
blog post or the emails to python-checkins...do you plan to make those
graphs available in the future as well?

Also, in the emails, what is the PGO column percentage relative to?

I suppose that for this to have maximum effect someone would have to
specifically be paying attention to performance and figuring out why
every (real) regression happened.  I don't suppose we have anyone in the
community currently who is taking on that role, though we certainly do
have people who are *interested* in Python performance :)

--David

[*] Personally I'd find it easier to read those dates in MM-DD form,
but I suppose that's a US quirk, since in the US when using slashes
the month comes first...

From david.c.stewart at intel.com  Tue Nov 17 14:22:48 2015
From: david.c.stewart at intel.com (Stewart, David C)
Date: Tue, 17 Nov 2015 19:22:48 +0000
Subject: [Python-Dev] Benchmark results across all major Python
 implementations
In-Reply-To: <20151117184034.1C19FB14089@webabinitio.net>
References: <CAP1=2W52Dq8EfFC=eA5GU73s+mRD10j=K+GLk4HTWYyLK2DZ1w@mail.gmail.com>
 <CABA9BEA-27EE-4F9B-8AD0-5C9B1243E93A@intel.com>
 <20151117184034.1C19FB14089@webabinitio.net>
Message-ID: <D5C4B4DB-F0AB-4097-A37E-0D04AA10FE04@intel.com>

+Stefan (owner of the 0-day lab)




On 11/17/15, 10:40 AM, "Python-Dev on behalf of R. David Murray" <python-dev-bounces+david.c.stewart=intel.com at python.org on behalf of rdmurray at bitdance.com> wrote:

>On Mon, 16 Nov 2015 23:37:06 +0000, "Stewart, David C" <david.c.stewart at intel.com> wrote:
>> Last June we started publishing a daily performance report of the latest Python tip against the previous day's run and some established synch point. We mail these to the community to act as a "canary in the coal mine." I wrote about it at https://01.org/lp/blog/0-day-challenge-what-pulse-internet
>> 
>> You can see our manager-style dashboard of a couple of key workloads at http://languagesperformance.intel.com/
>> (I have this running constantly on a dedicated screen in my office).
>
>Just took a look at this.  Pretty cool.  The web page is a bit confusing,
>though.  It doesn't give any clue as to what is being measured by the
>numbers presented...it isn't obvious whether those downward sloping
>lines represent progress or regression.  Also, the intro talks about
>historical data, but other than the older dates[*] in the graph there's
>no access to it.  Do you have plans to provide access to the raw data?
>It also doesn't show all of the test shown in the example email in your
>blog post or the emails to python-checkins...do you plan to make those
>graphs available in the future as well?

The data on this website has been normalized so "up" is "good" so far as the slope of the line. The daily email has a lot more detail about the hardware and software configuration and the versions being compared. We run workloads multiple times and visually show the relative standard distribution on the graph.

No plans to show the raw data.

I think showing multiple workloads graphically sounds useful, we should look into that.

>
>Also, in the emails, what is the PGO column percentage relative to?

It's the performance boost on the current rev from just using PGO. Another way to think about it is, this is the performance that you leave on the table by *not* building Cpython with PGO. For example, from last night's run, we would see an 18.54% boost in django_v2 by building Python using PGO.

Note: PGO is not the default way to build Python because it is relatively slow to compile it that way. (I think it should be the default). 

Here are the instructions for using it (thanks to Peter Wang for the instructions):

hg clone https://hg.python.org/cpython cpython
cd cpython
hg update 2.7
./configure
make profile-opt



>
>I suppose that for this to have maximum effect someone would have to
>specifically be paying attention to performance and figuring out why
>every (real) regression happened.  I don't suppose we have anyone in the
>community currently who is taking on that role, though we certainly do
>have people who are *interested* in Python performance :)

We're trying to fill that role as much as we can. When there is a significant (and unexplained) regression that we see, I usually ask our engineers to bisect it to identify the offending patch and root-cause it.

>
>--David
>
>[*] Personally I'd find it easier to read those dates in MM-DD form,
>but I suppose that's a US quirk, since in the US when using slashes
>the month comes first...

You and me both. As you surmised, the site was developed by our friends in Europe. :-)

>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>https://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe: https://mail.python.org/mailman/options/python-dev/david.c.stewart%40intel.com

From ncoghlan at gmail.com  Tue Nov 17 21:31:43 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 18 Nov 2015 12:31:43 +1000
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <CAP7+vJ+voOGQBwS-o4xUbD_J3b4sF4PzRJYmA5U3nhA+etTPFQ@mail.gmail.com>
References: <n2e1as$9f7$1@ger.gmane.org> <564AFA6A.8010505@egenix.com>
 <n2fk53$305$1@ger.gmane.org>
 <CAP7+vJ+voOGQBwS-o4xUbD_J3b4sF4PzRJYmA5U3nhA+etTPFQ@mail.gmail.com>
Message-ID: <CADiSq7dqfdsvJQtXh7qD7xrTv6KaabH4HhYdLg3Mg+==C_kfDQ@mail.gmail.com>

On 18 November 2015 at 02:50, Guido van Rossum <guido at python.org> wrote:
> On Tue, Nov 17, 2015 at 8:20 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
>> On 17.11.15 11:59, M.-A. Lemburg wrote:
>>>
>>> I don't think these situations are all that common, though,
>>> so reading in the full source code before compiling it
>>> sounds like a reasonable approach.
>>>
>>> We use the same simplification in eGenix PyRun's emulation of
>>> the Python command line interface and it has so far not
>>> caused any problems.
>>
>> Current implementation of import system went the same way. As a result
>> importing the script as a module and running it with command line can have
>> different behaviours in corner cases.
>
> I'm confused. *Of course* these two behaviors differ, since Python
> uses a different __name__. Not sure how this relates to the REPL.

I think Serhiy was referring to the fact that importlib already reads
the entire file before compiling it - since the import abstraction
doesn't require modules to be files on disk, it uses the get_code()
API on the module loader internally, which is typically implemented by
calling get_source() and then passing the result to compile().

That behaviour is then inherited at the command line by both the -m
switch and the support for executing directories and zip archives.
When we consider that the "-c" switch also executes an in-memory
string, direct script execution is currently the odd one out in *not*
reading the entire source file into memory first, so Serhiy's proposed
simplification of the implementation makes sense to me.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From guido at python.org  Tue Nov 17 21:46:49 2015
From: guido at python.org (Guido van Rossum)
Date: Tue, 17 Nov 2015 18:46:49 -0800
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <CADiSq7dqfdsvJQtXh7qD7xrTv6KaabH4HhYdLg3Mg+==C_kfDQ@mail.gmail.com>
References: <n2e1as$9f7$1@ger.gmane.org> <564AFA6A.8010505@egenix.com>
 <n2fk53$305$1@ger.gmane.org>
 <CAP7+vJ+voOGQBwS-o4xUbD_J3b4sF4PzRJYmA5U3nhA+etTPFQ@mail.gmail.com>
 <CADiSq7dqfdsvJQtXh7qD7xrTv6KaabH4HhYdLg3Mg+==C_kfDQ@mail.gmail.com>
Message-ID: <CAP7+vJKqiz=etRYQvFGDY70s0PVxyeM32vKO=CvsvLpC75M8jg@mail.gmail.com>

Aha, so the only code path that's being replaced is the code that
reads the script file when invoking "python FILE" or "python <FILE" or
"cat FILE | python" (but not "python" reading from a tty device)? That
makes the whole endeavor even more attractive!

On Tue, Nov 17, 2015 at 6:31 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 18 November 2015 at 02:50, Guido van Rossum <guido at python.org> wrote:
>> On Tue, Nov 17, 2015 at 8:20 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
>>> On 17.11.15 11:59, M.-A. Lemburg wrote:
>>>>
>>>> I don't think these situations are all that common, though,
>>>> so reading in the full source code before compiling it
>>>> sounds like a reasonable approach.
>>>>
>>>> We use the same simplification in eGenix PyRun's emulation of
>>>> the Python command line interface and it has so far not
>>>> caused any problems.
>>>
>>> Current implementation of import system went the same way. As a result
>>> importing the script as a module and running it with command line can have
>>> different behaviours in corner cases.
>>
>> I'm confused. *Of course* these two behaviors differ, since Python
>> uses a different __name__. Not sure how this relates to the REPL.
>
> I think Serhiy was referring to the fact that importlib already reads
> the entire file before compiling it - since the import abstraction
> doesn't require modules to be files on disk, it uses the get_code()
> API on the module loader internally, which is typically implemented by
> calling get_source() and then passing the result to compile().
>
> That behaviour is then inherited at the command line by both the -m
> switch and the support for executing directories and zip archives.
> When we consider that the "-c" switch also executes an in-memory
> string, direct script execution is currently the odd one out in *not*
> reading the entire source file into memory first, so Serhiy's proposed
> simplification of the implementation makes sense to me.
>
> Regards,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia



-- 
--Guido van Rossum (python.org/~guido)

From stefan.a.popa at intel.com  Wed Nov 18 01:10:30 2015
From: stefan.a.popa at intel.com (Popa, Stefan A)
Date: Wed, 18 Nov 2015 06:10:30 +0000
Subject: [Python-Dev] Benchmark results across all major Python
 implementations
In-Reply-To: <D5C4B4DB-F0AB-4097-A37E-0D04AA10FE04@intel.com>
References: <CAP1=2W52Dq8EfFC=eA5GU73s+mRD10j=K+GLk4HTWYyLK2DZ1w@mail.gmail.com>
 <CABA9BEA-27EE-4F9B-8AD0-5C9B1243E93A@intel.com>
 <20151117184034.1C19FB14089@webabinitio.net>,
 <D5C4B4DB-F0AB-4097-A37E-0D04AA10FE04@intel.com>
Message-ID: <63E314D4-979E-44B8-9E3D-BE624D476E86@intel.com>

Hi Python community,

Thank you for your feedback! We will look into this and come up with an e-mail format proposal in the following days.

Best regards,

--
Stefan A. POPA
Software Engineering Manager
System Technologies and Optimization Division
Software Services Group, Intel Romania

> On 17 Nov 2015, at 21:22, Stewart, David C <david.c.stewart at intel.com> wrote:
> 
> +Stefan (owner of the 0-day lab)
> 
> 
> 
> 
>> On 11/17/15, 10:40 AM, "Python-Dev on behalf of R. David Murray" <python-dev-bounces+david.c.stewart=intel.com at python.org on behalf of rdmurray at bitdance.com> wrote:
>> 
>>> On Mon, 16 Nov 2015 23:37:06 +0000, "Stewart, David C" <david.c.stewart at intel.com> wrote:
>>> Last June we started publishing a daily performance report of the latest Python tip against the previous day's run and some established synch point. We mail these to the community to act as a "canary in the coal mine." I wrote about it at https://01.org/lp/blog/0-day-challenge-what-pulse-internet
>>> 
>>> You can see our manager-style dashboard of a couple of key workloads at http://languagesperformance.intel.com/
>>> (I have this running constantly on a dedicated screen in my office).
>> 
>> Just took a look at this.  Pretty cool.  The web page is a bit confusing,
>> though.  It doesn't give any clue as to what is being measured by the
>> numbers presented...it isn't obvious whether those downward sloping
>> lines represent progress or regression.  Also, the intro talks about
>> historical data, but other than the older dates[*] in the graph there's
>> no access to it.  Do you have plans to provide access to the raw data?
>> It also doesn't show all of the test shown in the example email in your
>> blog post or the emails to python-checkins...do you plan to make those
>> graphs available in the future as well?
> 
> The data on this website has been normalized so "up" is "good" so far as the slope of the line. The daily email has a lot more detail about the hardware and software configuration and the versions being compared. We run workloads multiple times and visually show the relative standard distribution on the graph.
> 
> No plans to show the raw data.
> 
> I think showing multiple workloads graphically sounds useful, we should look into that.
> 
>> 
>> Also, in the emails, what is the PGO column percentage relative to?
> 
> It's the performance boost on the current rev from just using PGO. Another way to think about it is, this is the performance that you leave on the table by *not* building Cpython with PGO. For example, from last night's run, we would see an 18.54% boost in django_v2 by building Python using PGO.
> 
> Note: PGO is not the default way to build Python because it is relatively slow to compile it that way. (I think it should be the default). 
> 
> Here are the instructions for using it (thanks to Peter Wang for the instructions):
> 
> hg clone https://hg.python.org/cpython cpython
> cd cpython
> hg update 2.7
> ./configure
> make profile-opt
> 
> 
> 
>> 
>> I suppose that for this to have maximum effect someone would have to
>> specifically be paying attention to performance and figuring out why
>> every (real) regression happened.  I don't suppose we have anyone in the
>> community currently who is taking on that role, though we certainly do
>> have people who are *interested* in Python performance :)
> 
> We're trying to fill that role as much as we can. When there is a significant (and unexplained) regression that we see, I usually ask our engineers to bisect it to identify the offending patch and root-cause it.
> 
>> 
>> --David
>> 
>> [*] Personally I'd find it easier to read those dates in MM-DD form,
>> but I suppose that's a US quirk, since in the US when using slashes
>> the month comes first...
> 
> You and me both. As you surmised, the site was developed by our friends in Europe. :-)
> 
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/david.c.stewart%40intel.com

From stephen at xemacs.org  Wed Nov 18 02:11:59 2015
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 18 Nov 2015 16:11:59 +0900
Subject: [Python-Dev] Benchmark results across all major Python
 implementations
In-Reply-To: <D5C4B4DB-F0AB-4097-A37E-0D04AA10FE04@intel.com>
References: <CAP1=2W52Dq8EfFC=eA5GU73s+mRD10j=K+GLk4HTWYyLK2DZ1w@mail.gmail.com>
 <CABA9BEA-27EE-4F9B-8AD0-5C9B1243E93A@intel.com>
 <20151117184034.1C19FB14089@webabinitio.net>
 <D5C4B4DB-F0AB-4097-A37E-0D04AA10FE04@intel.com>
Message-ID: <22092.9407.502230.151306@turnbull.sk.tsukuba.ac.jp>

Stewart, David C writes:

 > Note: PGO is not the default way to build Python because it is
 > relatively slow to compile it that way. (I think it should be the
 > default).

+1

Slow-build-fast-run should be the default if you're sure the
optimization works.  Only developers are likely to run a given build
few enough times to save seconds, and most people are like to turn to
some other task as soon as they type "make".

It's a slightly different use case, but in XEmacs we have a
--quick-build configure option which means that the "usual targets"
don't rebuild a bunch of auxiliary targets (mostly documentation and
development infrastructure such as xref caches).  Never heard a
complaint about that either from the developers (who learned to use
--quick-build easily enough) or the beta testers (who do remark on
long build times, but only once a week or so for most of them).


From hrvoje.niksic at avl.com  Wed Nov 18 07:15:16 2015
From: hrvoje.niksic at avl.com (Hrvoje Niksic)
Date: Wed, 18 Nov 2015 13:15:16 +0100
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <CADiSq7dqfdsvJQtXh7qD7xrTv6KaabH4HhYdLg3Mg+==C_kfDQ@mail.gmail.com>
References: <n2e1as$9f7$1@ger.gmane.org> <564AFA6A.8010505@egenix.com>
 <n2fk53$305$1@ger.gmane.org>
 <CAP7+vJ+voOGQBwS-o4xUbD_J3b4sF4PzRJYmA5U3nhA+etTPFQ@mail.gmail.com>
 <CADiSq7dqfdsvJQtXh7qD7xrTv6KaabH4HhYdLg3Mg+==C_kfDQ@mail.gmail.com>
Message-ID: <564C6BD4.6030205@avl.com>

On 11/18/2015 03:31 AM, Nick Coghlan wrote:
> That behaviour is then inherited at the command line by both the -m
> switch and the support for executing directories and zip archives.
> When we consider that the "-c" switch also executes an in-memory
> string, direct script execution is currently the odd one out in *not*
> reading the entire source file into memory first, so Serhiy's proposed
> simplification of the implementation makes sense to me.

Reading the whole script in memory will incur an overhead when executing 
scripts that contain (potentially large) data embedded after the end of 
script source.

The technique of reading data from sys.argv[0] is probably obsolete now 
that Python supports executing zipped archives, but it is popular in 
shell scripting and might still be used for self-extracting scripts that 
must support older Python versions. This feature doesn't affect imports 
and -c which are not expected to contain non-Python data.

Hrvoje


From guido at python.org  Wed Nov 18 10:48:41 2015
From: guido at python.org (Guido van Rossum)
Date: Wed, 18 Nov 2015 07:48:41 -0800
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <564C6BD4.6030205@avl.com>
References: <n2e1as$9f7$1@ger.gmane.org> <564AFA6A.8010505@egenix.com>
 <n2fk53$305$1@ger.gmane.org>
 <CAP7+vJ+voOGQBwS-o4xUbD_J3b4sF4PzRJYmA5U3nhA+etTPFQ@mail.gmail.com>
 <CADiSq7dqfdsvJQtXh7qD7xrTv6KaabH4HhYdLg3Mg+==C_kfDQ@mail.gmail.com>
 <564C6BD4.6030205@avl.com>
Message-ID: <CAP7+vJLG2KFVKd5Yuv_kSW2Th9RuA-r0GiqupiiTuozf3qNK9A@mail.gmail.com>

On Wed, Nov 18, 2015 at 4:15 AM, Hrvoje Niksic <hrvoje.niksic at avl.com> wrote:
> On 11/18/2015 03:31 AM, Nick Coghlan wrote:
>>
>> That behaviour is then inherited at the command line by both the -m
>> switch and the support for executing directories and zip archives.
>> When we consider that the "-c" switch also executes an in-memory
>> string, direct script execution is currently the odd one out in *not*
>> reading the entire source file into memory first, so Serhiy's proposed
>> simplification of the implementation makes sense to me.
>
>
> Reading the whole script in memory will incur an overhead when executing
> scripts that contain (potentially large) data embedded after the end of
> script source.
>
> The technique of reading data from sys.argv[0] is probably obsolete now that
> Python supports executing zipped archives, but it is popular in shell
> scripting and might still be used for self-extracting scripts that must
> support older Python versions. This feature doesn't affect imports and -c
> which are not expected to contain non-Python data.

That trick doesn't work unless the data looks like Python comments or
data (e.g. a docstring). Python has always insisted on being able to
parse until EOF. The only extreme case would be a small script
followed by e.g. 4 GB of comments (where the old parser would indeed
be more efficient). But unless you can point me to an occurrence of
this in the wild I'm going to speculate that you just made this up
based on the shell analogy (which isn't perfect).

-- 
--Guido van Rossum (python.org/~guido)

From hrvoje.niksic at avl.com  Wed Nov 18 10:57:31 2015
From: hrvoje.niksic at avl.com (Hrvoje Niksic)
Date: Wed, 18 Nov 2015 16:57:31 +0100
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <CAP7+vJLG2KFVKd5Yuv_kSW2Th9RuA-r0GiqupiiTuozf3qNK9A@mail.gmail.com>
References: <n2e1as$9f7$1@ger.gmane.org> <564AFA6A.8010505@egenix.com>
 <n2fk53$305$1@ger.gmane.org>
 <CAP7+vJ+voOGQBwS-o4xUbD_J3b4sF4PzRJYmA5U3nhA+etTPFQ@mail.gmail.com>
 <CADiSq7dqfdsvJQtXh7qD7xrTv6KaabH4HhYdLg3Mg+==C_kfDQ@mail.gmail.com>
 <564C6BD4.6030205@avl.com>
 <CAP7+vJLG2KFVKd5Yuv_kSW2Th9RuA-r0GiqupiiTuozf3qNK9A@mail.gmail.com>
Message-ID: <564C9FEB.6030604@avl.com>

On 11/18/2015 04:48 PM, Guido van Rossum wrote:
> That trick doesn't work unless the data looks like Python comments or
> data (e.g. a docstring). Python has always insisted on being able to
> parse until EOF. The only extreme case would be a small script
> followed by e.g. 4 GB of comments (where the old parser would indeed
> be more efficient). But unless you can point me to an occurrence of
> this in the wild I'm going to speculate that you just made this up
> based on the shell analogy (which isn't perfect).

If this never really worked in Python, feel free to drop the issue. I 
may be misremembering the language in which scripts I saw using this 
techniques years ago were written - most likely sh or Perl.

Hrvoje


From rdmurray at bitdance.com  Wed Nov 18 11:24:03 2015
From: rdmurray at bitdance.com (R. David Murray)
Date: Wed, 18 Nov 2015 11:24:03 -0500
Subject: [Python-Dev] Benchmark results across all major Python
 implementations
In-Reply-To: <63E314D4-979E-44B8-9E3D-BE624D476E86@intel.com>
References: <CAP1=2W52Dq8EfFC=eA5GU73s+mRD10j=K+GLk4HTWYyLK2DZ1w@mail.gmail.com>
 <CABA9BEA-27EE-4F9B-8AD0-5C9B1243E93A@intel.com>
 <20151117184034.1C19FB14089@webabinitio.net>,
 <D5C4B4DB-F0AB-4097-A37E-0D04AA10FE04@intel.com>
 <63E314D4-979E-44B8-9E3D-BE624D476E86@intel.com>
Message-ID: <20151118162404.417A5B14089@webabinitio.net>

On 17 Nov 2015, at 21:22, Stewart, David C <david.c.stewart at intel.com> wrote:
> On 11/17/15, 10:40 AM, "Python-Dev on behalf of R. David Murray" <python-dev-bounces+david.c.stewart=intel.com at python.org on behalf of rdmurray at bitdance.com> wrote:
>>
>> I suppose that for this to have maximum effect someone would have to
>> specifically be paying attention to performance and figuring out why
>> every (real) regression happened.  I don't suppose we have anyone in the
>> community currently who is taking on that role, though we certainly do
>> have people who are *interested* in Python performance :)
> 
> We're trying to fill that role as much as we can. When there is a
> significant (and unexplained) regression that we see, I usually ask
> our engineers to bisect it to identify the offending patch and
> root-cause it.

That's great news.

--David

From rymg19 at gmail.com  Wed Nov 18 11:50:48 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Wed, 18 Nov 2015 10:50:48 -0600
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <CAP7+vJLG2KFVKd5Yuv_kSW2Th9RuA-r0GiqupiiTuozf3qNK9A@mail.gmail.com>
References: <n2e1as$9f7$1@ger.gmane.org> <564AFA6A.8010505@egenix.com>
 <n2fk53$305$1@ger.gmane.org>
 <CAP7+vJ+voOGQBwS-o4xUbD_J3b4sF4PzRJYmA5U3nhA+etTPFQ@mail.gmail.com>
 <CADiSq7dqfdsvJQtXh7qD7xrTv6KaabH4HhYdLg3Mg+==C_kfDQ@mail.gmail.com>
 <564C6BD4.6030205@avl.com>
 <CAP7+vJLG2KFVKd5Yuv_kSW2Th9RuA-r0GiqupiiTuozf3qNK9A@mail.gmail.com>
Message-ID: <9A6B0389-9F66-4A1B-8D96-A3E57D42DAB4@gmail.com>

Well, not quite the same thing, but https://github.com/kirbyfan64/pfbuild/blob/master/pfbuild embeds the compressed version of 16k LOC. Would it be affected negatively in any way be this?

Since all the data is on one line, I'd think the old (current) parser would end up reading in the whole line anyway.

On November 18, 2015 9:48:41 AM CST, Guido van Rossum <guido at python.org> wrote:
>On Wed, Nov 18, 2015 at 4:15 AM, Hrvoje Niksic <hrvoje.niksic at avl.com>
>wrote:
>> On 11/18/2015 03:31 AM, Nick Coghlan wrote:
>>>
>>> That behaviour is then inherited at the command line by both the -m
>>> switch and the support for executing directories and zip archives.
>>> When we consider that the "-c" switch also executes an in-memory
>>> string, direct script execution is currently the odd one out in
>*not*
>>> reading the entire source file into memory first, so Serhiy's
>proposed
>>> simplification of the implementation makes sense to me.
>>
>>
>> Reading the whole script in memory will incur an overhead when
>executing
>> scripts that contain (potentially large) data embedded after the end
>of
>> script source.
>>
>> The technique of reading data from sys.argv[0] is probably obsolete
>now that
>> Python supports executing zipped archives, but it is popular in shell
>> scripting and might still be used for self-extracting scripts that
>must
>> support older Python versions. This feature doesn't affect imports
>and -c
>> which are not expected to contain non-Python data.
>
>That trick doesn't work unless the data looks like Python comments or
>data (e.g. a docstring). Python has always insisted on being able to
>parse until EOF. The only extreme case would be a small script
>followed by e.g. 4 GB of comments (where the old parser would indeed
>be more efficient). But unless you can point me to an occurrence of
>this in the wild I'm going to speculate that you just made this up
>based on the shell analogy (which isn't perfect).
>
>-- 
>--Guido van Rossum (python.org/~guido)
>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>https://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe:
>https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com

-- 
Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151118/f6bbaf1a/attachment.html>

From ncoghlan at gmail.com  Wed Nov 18 21:30:15 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 19 Nov 2015 12:30:15 +1000
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <9A6B0389-9F66-4A1B-8D96-A3E57D42DAB4@gmail.com>
References: <n2e1as$9f7$1@ger.gmane.org> <564AFA6A.8010505@egenix.com>
 <n2fk53$305$1@ger.gmane.org>
 <CAP7+vJ+voOGQBwS-o4xUbD_J3b4sF4PzRJYmA5U3nhA+etTPFQ@mail.gmail.com>
 <CADiSq7dqfdsvJQtXh7qD7xrTv6KaabH4HhYdLg3Mg+==C_kfDQ@mail.gmail.com>
 <564C6BD4.6030205@avl.com>
 <CAP7+vJLG2KFVKd5Yuv_kSW2Th9RuA-r0GiqupiiTuozf3qNK9A@mail.gmail.com>
 <9A6B0389-9F66-4A1B-8D96-A3E57D42DAB4@gmail.com>
Message-ID: <CADiSq7cHU8h=6dkXemPFTBLGZAb23J1V08hjwC_yrYA04QtsgA@mail.gmail.com>

On 19 November 2015 at 02:50, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> Well, not quite the same thing, but
> https://github.com/kirbyfan64/pfbuild/blob/master/pfbuild embeds the
> compressed version of 16k LOC. Would it be affected negatively in any way be
> this?
>
> Since all the data is on one line, I'd think the old (current) parser would
> end up reading in the whole line anyway.

Right. The other main "embedded binary blob" case I'm familiar with is
get-pip.py, which embeds a base85 encoded copy of pip as a zip archive
in a DATA variable, and there aren't any appending tricks there either
- it's a normal variable, and the "if __name__ == '__main__':" block
is the final two lines of the file, so Python already has to read the
whole thing before it starts the process of unpacking it and executing
it.

Two things worth keeping in mind here is that if a script is embedding
enough data for reading the whole thing into memory it to pose a
problem:

* compiling and executing it is likely to pose an even bigger problem
than reading it in
* downloading it in the first place is also likely to be annoying

We couldn't make a change like this in a maintenance release, but for
a new feature release, the consistency gain is definitely worth it.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From vadmium+py at gmail.com  Wed Nov 18 21:28:40 2015
From: vadmium+py at gmail.com (Martin Panter)
Date: Thu, 19 Nov 2015 02:28:40 +0000
Subject: [Python-Dev] [Python-checkins] cpython (3.4): asyncio.docs: Fix
 versionadded
In-Reply-To: <20151118174516.21855.43706@psf.io>
References: <20151118174516.21855.43706@psf.io>
Message-ID: <CA+eR4cFVqz0UHhjd1mRtLN67rPCsqtmSC8-zou_n8ZPNvFGEiw@mail.gmail.com>

On 18 November 2015 at 17:45, yury.selivanov <python-checkins at python.org> wrote:
> https://hg.python.org/cpython/rev/b34c42e46e7b
> changeset:   99204:b34c42e46e7b
> branch:      3.4
> parent:      99201:89d66f912671
> user:        Yury Selivanov <yselivanov at sprymix.com>
> date:        Wed Nov 18 12:44:31 2015 -0500
> summary:
>   asyncio.docs: Fix versionadded
>
> files:
>   Doc/library/asyncio-task.rst |  2 +-
>   1 files changed, 1 insertions(+), 1 deletions(-)
>
>
> diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst
> --- a/Doc/library/asyncio-task.rst
> +++ b/Doc/library/asyncio-task.rst
> @@ -699,4 +699,4 @@
>        Unlike the functions above, :func:`run_coroutine_threadsafe` requires the
>        *loop* argument to be passed explicitely.
>
> -   .. versionadded:: 3.4.4
> +   .. versionadded:: 3.4.4, 3.5.1

This doesn?t get marked up properly; have a look at
<https://docs.python.org/3.4/library/asyncio-task.html#asyncio.run_coroutine_threadsafe>.

From ben at groovie.org  Wed Nov 18 22:53:14 2015
From: ben at groovie.org (Ben Bangert)
Date: Wed, 18 Nov 2015 19:53:14 -0800
Subject: [Python-Dev] Python stdlib ssl.SSLContext is missing mode setting
 ability
Message-ID: <CAOUmX516-1PxF5wm6CSCpkfu2RD0cLJxGE8K13eGhXCLZPwu1w@mail.gmail.com>

In Python 2 and 3, the ssl module's SSLContext object has a way to set
SSL options, but not to set SSL modes.

The set_mode command and some of the available modes:
https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_set_mode.html

The most critical mode is SSL_MODE_RELEASE_BUFFERS, which can drop the
SSL overhead *per connection* from around 25kb to ~7kb. The pyopenssl
library allows the setting of SSLContext modes, it seems very odd that
the Python 2/3 ssl modules do not. Though I could understand that
perhaps not all SSL libraries Python might build against would have
this mode thing available.

(BoringSSL sets this mode by default its considered such an obvious win)

If there is some way to set this I happened to miss, apologies, I only
went looking through the docs for it, not the code.

On a side-note, in my testing, Python 3.4 had about 20kb/connection of
overhead for using SSL, but Python 3.5 jumped to 30kb/connection of
SSL overhead. These numbers for SSL overhead are far too high for any
reasonable use of Python+SSL on highly concurrent systems. Test repo
for testing SSL overhead here:
https://github.com/bbangert/ssl-ram-testing/

Cheers,
Ben

From cory at lukasa.co.uk  Thu Nov 19 03:14:17 2015
From: cory at lukasa.co.uk (Cory Benfield)
Date: Thu, 19 Nov 2015 08:14:17 +0000
Subject: [Python-Dev] Python stdlib ssl.SSLContext is missing mode
 setting ability
In-Reply-To: <CAOUmX516-1PxF5wm6CSCpkfu2RD0cLJxGE8K13eGhXCLZPwu1w@mail.gmail.com>
References: <CAOUmX516-1PxF5wm6CSCpkfu2RD0cLJxGE8K13eGhXCLZPwu1w@mail.gmail.com>
Message-ID: <2E57C2E0-C03C-4057-8996-1D4BD691532D@lukasa.co.uk>


> On 19 Nov 2015, at 03:53, Ben Bangert <ben at groovie.org> wrote:
> 
> In Python 2 and 3, the ssl module's SSLContext object has a way to set
> SSL options, but not to set SSL modes.
> 
> The set_mode command and some of the available modes:
> https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_set_mode.html
> 
> The most critical mode is SSL_MODE_RELEASE_BUFFERS, which can drop the
> SSL overhead *per connection* from around 25kb to ~7kb. The pyopenssl
> library allows the setting of SSLContext modes, it seems very odd that
> the Python 2/3 ssl modules do not. Though I could understand that
> perhaps not all SSL libraries Python might build against would have
> this mode thing available.
> 

Ben,

Do we need the ability to set arbitrary modes? Most of the modes mentioned in the OpenSSL documentation are things we actively don?t want the user to set because stuff will randomly break. With that in mind, and with the fact that SSL_MODE_RELEASE_BUFFERS is so obviously better than the standard, should we just instead have the ssl module automatically set SSL_MODE_RELEASE_BUFFERS unconditionally?

If so, I?m happy to submit a bug/patch to get that to happen.

Cory

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151119/4996b1d1/attachment.sig>

From p.f.moore at gmail.com  Thu Nov 19 04:28:43 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 19 Nov 2015 09:28:43 +0000
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <564C9FEB.6030604@avl.com>
References: <n2e1as$9f7$1@ger.gmane.org> <564AFA6A.8010505@egenix.com>
 <n2fk53$305$1@ger.gmane.org>
 <CAP7+vJ+voOGQBwS-o4xUbD_J3b4sF4PzRJYmA5U3nhA+etTPFQ@mail.gmail.com>
 <CADiSq7dqfdsvJQtXh7qD7xrTv6KaabH4HhYdLg3Mg+==C_kfDQ@mail.gmail.com>
 <564C6BD4.6030205@avl.com>
 <CAP7+vJLG2KFVKd5Yuv_kSW2Th9RuA-r0GiqupiiTuozf3qNK9A@mail.gmail.com>
 <564C9FEB.6030604@avl.com>
Message-ID: <CACac1F8K38YzAEf_wngTdhNeqP01p6owq7idzj_rAg6QYW365w@mail.gmail.com>

On 18 November 2015 at 15:57, Hrvoje Niksic <hrvoje.niksic at avl.com> wrote:
> If this never really worked in Python, feel free to drop the issue. I may be
> misremembering the language in which scripts I saw using this techniques
> years ago were written - most likely sh or Perl.

It was Perl. In the past I've tried to emulate this in Python and it's
not been very successful, so I'd say it's not something to worry about
here.
Paul

From mal at egenix.com  Thu Nov 19 04:47:24 2015
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 19 Nov 2015 10:47:24 +0100
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <CAP7+vJKDyPmO9-0WxtFxoGSGvtPeUBm2sBzo7G=aS5g4e3eM4w@mail.gmail.com>
References: <n2e1as$9f7$1@ger.gmane.org> <564AFA6A.8010505@egenix.com>
 <CAP7+vJKDyPmO9-0WxtFxoGSGvtPeUBm2sBzo7G=aS5g4e3eM4w@mail.gmail.com>
Message-ID: <564D9AAC.1090409@egenix.com>

On 17.11.2015 16:22, Guido van Rossum wrote:
> On Tue, Nov 17, 2015 at 1:59 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>>> [moving from read source line by line to reading all in one go]
>> We use the same simplification in eGenix PyRun's emulation of
>> the Python command line interface and it has so far not
>> caused any problems.
> 
> Curious how you do it? I'd actually be quite disappointed if the
> amount of parsing done by the standard REPL went down.

Oh, that's easy:

        elif sys.argv[0] == '-' and not (pyrun_as_string or pyrun_as_module):
            # Read the script from stdin
            pyrun_as_string = True
            pyrun_script = sys.stdin.read()

and then, later on:

        # Run the script
        try:
            pyrun_execute_script(pyrun_script, mode)
        except Exception as reason:
            if pyrun_interactive:
                import traceback
                traceback.print_exc()
                pyrun_prompt(banner='')
            else:
                raise
        else:
            # Enter interactive mode, in case wanted
            if pyrun_interactive:
                pyrun_prompt()

The REPL is not affected by this, since we use the standard
code.interact() for the prompt. This reads the entry line
by line, joins the lines and tries to compile the entry every
time it receives a new line until it succeeds or fails.

Serhiy's proposed change should not affect this mode of
operation.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 19 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________
2015-10-23: Released mxODBC Connect 2.1.5 ...     http://egenix.com/go85

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From mal at egenix.com  Thu Nov 19 04:58:31 2015
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 19 Nov 2015 10:58:31 +0100
Subject: [Python-Dev] Python stdlib ssl.SSLContext is missing mode
 setting ability
In-Reply-To: <2E57C2E0-C03C-4057-8996-1D4BD691532D@lukasa.co.uk>
References: <CAOUmX516-1PxF5wm6CSCpkfu2RD0cLJxGE8K13eGhXCLZPwu1w@mail.gmail.com>
 <2E57C2E0-C03C-4057-8996-1D4BD691532D@lukasa.co.uk>
Message-ID: <564D9D47.1060500@egenix.com>

On 19.11.2015 09:14, Cory Benfield wrote:
> 
>> On 19 Nov 2015, at 03:53, Ben Bangert <ben at groovie.org> wrote:
>>
>> In Python 2 and 3, the ssl module's SSLContext object has a way to set
>> SSL options, but not to set SSL modes.
>>
>> The set_mode command and some of the available modes:
>> https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_set_mode.html
>>
>> The most critical mode is SSL_MODE_RELEASE_BUFFERS, which can drop the
>> SSL overhead *per connection* from around 25kb to ~7kb. The pyopenssl
>> library allows the setting of SSLContext modes, it seems very odd that
>> the Python 2/3 ssl modules do not. Though I could understand that
>> perhaps not all SSL libraries Python might build against would have
>> this mode thing available.
>>
> 
> Ben,
> 
> Do we need the ability to set arbitrary modes? Most of the modes mentioned in the OpenSSL documentation are things we actively don?t want the user to set because stuff will randomly break. With that in mind, and with the fact that SSL_MODE_RELEASE_BUFFERS is so obviously better than the standard, should we just instead have the ssl module automatically set SSL_MODE_RELEASE_BUFFERS unconditionally?
> 
> If so, I?m happy to submit a bug/patch to get that to happen.

The mode should only be enabled for OpenSSL versions which are
not affected by this vulnerability:

https://www.rapid7.com/db/vulnerabilities/http-openssl-cve-2014-0198

Other than that it seems like a good way forward. Plenty other
projects have had this enabled per default for years:

http://www.dovecot.org/list/dovecot/2011-October/131381.html
https://svn.boost.org/trac/boost/changeset/71706
https://community.openvpn.net/openvpn/ticket/157

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 19 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________
2015-10-23: Released mxODBC Connect 2.1.5 ...     http://egenix.com/go85

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From storchaka at gmail.com  Thu Nov 19 06:51:39 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 19 Nov 2015 13:51:39 +0200
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <CAP7+vJ+voOGQBwS-o4xUbD_J3b4sF4PzRJYmA5U3nhA+etTPFQ@mail.gmail.com>
References: <n2e1as$9f7$1@ger.gmane.org> <564AFA6A.8010505@egenix.com>
 <n2fk53$305$1@ger.gmane.org>
 <CAP7+vJ+voOGQBwS-o4xUbD_J3b4sF4PzRJYmA5U3nhA+etTPFQ@mail.gmail.com>
Message-ID: <n2kd4c$dmq$1@ger.gmane.org>

On 17.11.15 18:50, Guido van Rossum wrote:
> On Tue, Nov 17, 2015 at 8:20 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
>> Current implementation of import system went the same way. As a result
>> importing the script as a module and running it with command line can have
>> different behaviours in corner cases.
>
> I'm confused. *Of course* these two behaviors differ, since Python
> uses a different __name__. Not sure how this relates to the REPL.

Sorry for confusing. I meant parser level. File parser has a few bugs, 
that can cause that the source will be differently interpreted with file 
and string parsers. For example attached script produces different 
output, "?" if executed as a script, and "??" if imported as a module.

And there is a question about the null byte. Now compile(), exec(), 
eval() raises an exception if the script contains the null byte. 
Formerly they accepted it, but the null byte ended the script. The 
behavior of file parser is more weird. The null byte makes parser to 
ignore the end of script including the newline byte [1]. E.g. 
"#\0\nprint('a')" is interpreted as "#print('a')". This is different 
from PyPy (and may be other implementations) that interprets the null 
byte just as ordinal character.

The question is wherever we should support the null byte in Python sources.

[1] http://bugs.python.org/issue20115
-------------- next part --------------
A non-text attachment was scrubbed...
Name: long_first_coding_line.py
Type: text/x-python
Size: 8223 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151119/4616fb00/attachment.py>

From rosuav at gmail.com  Thu Nov 19 07:10:18 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 19 Nov 2015 23:10:18 +1100
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <n2kd4c$dmq$1@ger.gmane.org>
References: <n2e1as$9f7$1@ger.gmane.org> <564AFA6A.8010505@egenix.com>
 <n2fk53$305$1@ger.gmane.org>
 <CAP7+vJ+voOGQBwS-o4xUbD_J3b4sF4PzRJYmA5U3nhA+etTPFQ@mail.gmail.com>
 <n2kd4c$dmq$1@ger.gmane.org>
Message-ID: <CAPTjJmo2CBnFihtegT0HpbUQKF_BuUKnUpPSS6V8=LC9njEtjg@mail.gmail.com>

On Thu, Nov 19, 2015 at 10:51 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> http://bugs.python.org/issue20115

Interestingly, the file linked in the last comment on that issue [1]
ties in with another part of this thread, regarding binary blobs in
Python scripts. It uses open(sys.argv[0],'rb') to find itself, and has
the binary data embedded as a comment. (Though the particular
technique strikes me as fragile; to prevent newlines from messing
things up, \n becomes #% and \r becomes #$, but I don't see any
protection against those sequences occurring naturally in the blob.
Given that it's a bz2 archive, I would expect any two-byte sequence to
be capable of occurring.) To be quite honest, I wouldn't mind if
Python objected to this kind of code. If I were writing it myself, I'd
use a triple-quoted string containing some kind of textualized version
- either the repr of the string, or some kind of base 64 or base 85
encoding. Binary data embedded literally will prevent non-ASCII file
encodings.

ChrisA

[1] http://ftp.waf.io/pub/release/waf-1.7.16

From ncoghlan at gmail.com  Thu Nov 19 08:35:01 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 19 Nov 2015 23:35:01 +1000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
Message-ID: <CADiSq7cejYf9NbYr28af=DH7fgtih6SZviGP_-3i+2uArFNbcQ@mail.gmail.com>

On 11 November 2015 at 10:47, Nick Coghlan <ncoghlan at gmail.com> wrote:

> Our last discussion back in July seemed to show that folks either
> didn't care about the question (because they're using unmodified
> upstream versions so the PEP didn't affect them), or else thought the
> approach described in the PEP was reasonable, so I'm hoping the
> consequences of my mistake won't be too severe.
>

RHEL 7.2 is out today, together with details of what we've now committed to
supporting for certification verification configuration in the RHEL 7
system Python: https://access.redhat.com/articles/2039753

That article also describes the PEP 476 status across the different Python
builds Red Hat supports, which I've summarised below.

Versions with PEP 476 implemented:

* Python 3.4 SCL: cert verification always enabled by default
* system Python in RHEL 7.2+: still off by default due to the compatibility
break, but with PEP 493's file based configuration setting to enable it by
default

Versions without PEP 476 implemented:

* Python 2.7 SCL
* Python 3.3 SCL
* system Python in RHEL 7.1 and earlier
* system Python in all versions of RHEL 6

I assume that status of the Python 2.7 SCL will change at some point, but
don't have an ETA or any technical details to share at this point.

The system Python versions in RHEL 5 and earlier didn't include the ssl
module at all (since that was only added to the standard library in Python
2.6), so they're not affected by the concerns raised in PEP 476 in the
first place.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151119/a055b026/attachment.html>

From ben at groovie.org  Thu Nov 19 10:26:28 2015
From: ben at groovie.org (Ben Bangert)
Date: Thu, 19 Nov 2015 07:26:28 -0800
Subject: [Python-Dev] Python stdlib ssl.SSLContext is missing mode
 setting ability
In-Reply-To: <564D9D47.1060500@egenix.com>
References: <CAOUmX516-1PxF5wm6CSCpkfu2RD0cLJxGE8K13eGhXCLZPwu1w@mail.gmail.com>
 <2E57C2E0-C03C-4057-8996-1D4BD691532D@lukasa.co.uk>
 <564D9D47.1060500@egenix.com>
Message-ID: <CAOUmX52VryYVs-Q-Jr1ifi2Yd8pQQ1xfvrs56mfPJBnuVPW8qA@mail.gmail.com>

On Thu, Nov 19, 2015 at 1:58 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> On 19.11.2015 09:14, Cory Benfield wrote:
>>
>>> On 19 Nov 2015, at 03:53, Ben Bangert <ben at groovie.org> wrote:
>>>
>>> In Python 2 and 3, the ssl module's SSLContext object has a way to set
>>> SSL options, but not to set SSL modes.
>>>
>>> The set_mode command and some of the available modes:
>>> https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_set_mode.html
>>>
>>> The most critical mode is SSL_MODE_RELEASE_BUFFERS, which can drop the
>>> SSL overhead *per connection* from around 25kb to ~7kb. The pyopenssl
>>> library allows the setting of SSLContext modes, it seems very odd that
>>> the Python 2/3 ssl modules do not. Though I could understand that
>>> perhaps not all SSL libraries Python might build against would have
>>> this mode thing available.
>>>
>>
>> Ben,
>>
>> Do we need the ability to set arbitrary modes? Most of the modes mentioned in the OpenSSL documentation are things we actively don?t want the user to set because stuff will randomly break. With that in mind, and with the fact that SSL_MODE_RELEASE_BUFFERS is so obviously better than the standard, should we just instead have the ssl module automatically set SSL_MODE_RELEASE_BUFFERS unconditionally?
>>
>> If so, I?m happy to submit a bug/patch to get that to happen.
>
> The mode should only be enabled for OpenSSL versions which are
> not affected by this vulnerability:
>
> https://www.rapid7.com/db/vulnerabilities/http-openssl-cve-2014-0198
>
> Other than that it seems like a good way forward. Plenty other
> projects have had this enabled per default for years:

I can't think of any other mode to set, setting this with the
condition cited for that vulnerability looks like a good idea.

Cheers,
Ben

From guido at python.org  Thu Nov 19 12:41:48 2015
From: guido at python.org (Guido van Rossum)
Date: Thu, 19 Nov 2015 09:41:48 -0800
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <564D9AAC.1090409@egenix.com>
References: <n2e1as$9f7$1@ger.gmane.org> <564AFA6A.8010505@egenix.com>
 <CAP7+vJKDyPmO9-0WxtFxoGSGvtPeUBm2sBzo7G=aS5g4e3eM4w@mail.gmail.com>
 <564D9AAC.1090409@egenix.com>
Message-ID: <CAP7+vJ+XWFFwbsqk9pkEOPrqgPKXFd-XXaz1gXiD9kMaYtf3Tw@mail.gmail.com>

On Thu, Nov 19, 2015 at 1:47 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> On 17.11.2015 16:22, Guido van Rossum wrote:
>> On Tue, Nov 17, 2015 at 1:59 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>>>> [moving from read source line by line to reading all in one go]
>>> We use the same simplification in eGenix PyRun's emulation of
>>> the Python command line interface and it has so far not
>>> caused any problems.
>>
>> Curious how you do it? I'd actually be quite disappointed if the
>> amount of parsing done by the standard REPL went down.
>
> Oh, that's easy:
>
>         elif sys.argv[0] == '-' and not (pyrun_as_string or pyrun_as_module):
>             # Read the script from stdin
>             pyrun_as_string = True
>             pyrun_script = sys.stdin.read()
>
> and then, later on:
>
>         # Run the script
>         try:
>             pyrun_execute_script(pyrun_script, mode)
>         except Exception as reason:
>             if pyrun_interactive:
>                 import traceback
>                 traceback.print_exc()
>                 pyrun_prompt(banner='')
>             else:
>                 raise
>         else:
>             # Enter interactive mode, in case wanted
>             if pyrun_interactive:
>                 pyrun_prompt()

Yes, this makes sense.

> The REPL is not affected by this, since we use the standard
> code.interact() for the prompt. This reads the entry line
> by line, joins the lines and tries to compile the entry every
> time it receives a new line until it succeeds or fails.

Heh, I wrote code.interact() as a poor-man's substitute for what the
"real" REPL (implemented in C) does. :-) It usually ends up doing the
same thing, but I'm sure there are edge cases where the "real" REPL is
better. It doesn't re-parse after each line is read, it actually keeps
the parser state and adds new tokens read from the tty. There is even
a special grammar root symbol ('single') for this mode.

> Serhiy's proposed change should not affect this mode of
> operation.

I sure hope not.

Though there is actually one case that IIRC doesn't work today: if
sys.stdin is a stream that doesn't wrap a file descriptor. Would be
nice to make that work. (Pretty esoteric use case though.)

-- 
--Guido van Rossum (python.org/~guido)

From cory at lukasa.co.uk  Thu Nov 19 13:17:41 2015
From: cory at lukasa.co.uk (Cory Benfield)
Date: Thu, 19 Nov 2015 18:17:41 +0000
Subject: [Python-Dev] Python stdlib ssl.SSLContext is missing mode
 setting ability
In-Reply-To: <CAOUmX52VryYVs-Q-Jr1ifi2Yd8pQQ1xfvrs56mfPJBnuVPW8qA@mail.gmail.com>
References: <CAOUmX516-1PxF5wm6CSCpkfu2RD0cLJxGE8K13eGhXCLZPwu1w@mail.gmail.com>
 <2E57C2E0-C03C-4057-8996-1D4BD691532D@lukasa.co.uk>
 <564D9D47.1060500@egenix.com>
 <CAOUmX52VryYVs-Q-Jr1ifi2Yd8pQQ1xfvrs56mfPJBnuVPW8qA@mail.gmail.com>
Message-ID: <53DD9408-F0DF-42D4-809B-19F29E83D870@lukasa.co.uk>


> On 19 Nov 2015, at 15:26, Ben Bangert <ben at groovie.org> wrote:
> 
> I can't think of any other mode to set, setting this with the
> condition cited for that vulnerability looks like a good idea.
> 
> Cheers,
> Ben

Ok, we?re agreed. The work can be tracked under Issue 25672: https://bugs.python.org/issue25672

Cory
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151119/8c44c134/attachment.sig>

From guido at python.org  Thu Nov 19 13:27:18 2015
From: guido at python.org (Guido van Rossum)
Date: Thu, 19 Nov 2015 10:27:18 -0800
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <n2kd4c$dmq$1@ger.gmane.org>
References: <n2e1as$9f7$1@ger.gmane.org> <564AFA6A.8010505@egenix.com>
 <n2fk53$305$1@ger.gmane.org>
 <CAP7+vJ+voOGQBwS-o4xUbD_J3b4sF4PzRJYmA5U3nhA+etTPFQ@mail.gmail.com>
 <n2kd4c$dmq$1@ger.gmane.org>
Message-ID: <CAP7+vJ+33U1WW25XuyM3nVc96iamC9Cu4-W7ZqR1r_vGTyeZGw@mail.gmail.com>

On Thu, Nov 19, 2015 at 3:51 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> On 17.11.15 18:50, Guido van Rossum wrote:
>>
>> On Tue, Nov 17, 2015 at 8:20 AM, Serhiy Storchaka <storchaka at gmail.com>
>> wrote:
>>>
>>> Current implementation of import system went the same way. As a result
>>> importing the script as a module and running it with command line can
>>> have
>>> different behaviours in corner cases.
>>
>>
>> I'm confused. *Of course* these two behaviors differ, since Python
>> uses a different __name__. Not sure how this relates to the REPL.
>
> Sorry for confusing. I meant parser level. File parser has a few bugs, that
> can cause that the source will be differently interpreted with file and
> string parsers. For example attached script produces different output, "?"
> if executed as a script, and "??" if imported as a module.

I see. Well, I trust you in this area (it's been too long since I
wrote the original code for all that :-).

> And there is a question about the null byte. Now compile(), exec(), eval()
> raises an exception if the script contains the null byte. Formerly they
> accepted it, but the null byte ended the script.

I like erroring out better.

> The behavior of file parser
> is more weird. The null byte makes parser to ignore the end of script
> including the newline byte [1]. E.g. "#\0\nprint('a')" is interpreted as
> "#print('a')". This is different from PyPy (and may be other
> implementations) that interprets the null byte just as ordinal character.

Yeah, this is just poorly written old code that didn't expect anyone
to care about null bytes. IMO here too the null byte should be an
error.

> The question is wherever we should support the null byte in Python sources.
>
> [1] http://bugs.python.org/issue20115

Good luck, and thanks for working on this!

-- 
--Guido van Rossum (python.org/~guido)

From guido at python.org  Thu Nov 19 13:28:02 2015
From: guido at python.org (Guido van Rossum)
Date: Thu, 19 Nov 2015 10:28:02 -0800
Subject: [Python-Dev] Reading Python source file
In-Reply-To: <CAPTjJmo2CBnFihtegT0HpbUQKF_BuUKnUpPSS6V8=LC9njEtjg@mail.gmail.com>
References: <n2e1as$9f7$1@ger.gmane.org> <564AFA6A.8010505@egenix.com>
 <n2fk53$305$1@ger.gmane.org>
 <CAP7+vJ+voOGQBwS-o4xUbD_J3b4sF4PzRJYmA5U3nhA+etTPFQ@mail.gmail.com>
 <n2kd4c$dmq$1@ger.gmane.org>
 <CAPTjJmo2CBnFihtegT0HpbUQKF_BuUKnUpPSS6V8=LC9njEtjg@mail.gmail.com>
Message-ID: <CAP7+vJJa8fmyui80+Pm_jYthOvu2Uy+4svWwQ2ozO1A8b63Qvg@mail.gmail.com>

Yeah, let's kill this undefined behavior.

On Thu, Nov 19, 2015 at 4:10 AM, Chris Angelico <rosuav at gmail.com> wrote:
> On Thu, Nov 19, 2015 at 10:51 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
>> http://bugs.python.org/issue20115
>
> Interestingly, the file linked in the last comment on that issue [1]
> ties in with another part of this thread, regarding binary blobs in
> Python scripts. It uses open(sys.argv[0],'rb') to find itself, and has
> the binary data embedded as a comment. (Though the particular
> technique strikes me as fragile; to prevent newlines from messing
> things up, \n becomes #% and \r becomes #$, but I don't see any
> protection against those sequences occurring naturally in the blob.
> Given that it's a bz2 archive, I would expect any two-byte sequence to
> be capable of occurring.) To be quite honest, I wouldn't mind if
> Python objected to this kind of code. If I were writing it myself, I'd
> use a triple-quoted string containing some kind of textualized version
> - either the repr of the string, or some kind of base 64 or base 85
> encoding. Binary data embedded literally will prevent non-ASCII file
> encodings.
>
> ChrisA
>
> [1] http://ftp.waf.io/pub/release/waf-1.7.16
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From drekin at gmail.com  Thu Nov 19 16:50:10 2015
From: drekin at gmail.com (=?UTF-8?B?QWRhbSBCYXJ0b8Wh?=)
Date: Thu, 19 Nov 2015 22:50:10 +0100
Subject: [Python-Dev] Improving the reading part of REPL
Message-ID: <CACvLUa=CEbVh=b9QM8+S+faAid_K-NFpsHD02mspwcfHUjUwjw@mail.gmail.com>

It seems that there will be some refactoring of the tokenizer code.
Regarding this, I'd like to recall my proposal on readline hooks. It would
be nice if char* based PyOS_Readline API was replaced by a Python str based
hook customizable by Python code. I propose to add function
sys.readlinehook accepting optional prompt and returning a line read
interactively from a user. There would also be sys.__readlinehook__
containing the original value of sys.readlinehook (similarly to
sys.(__)displayhook(__), sys.(__)excepthook(__) and
sys.(__)std(in/out/err)(__)).

Currently, the input is read from C stdin even if sys.stdin is changed (see
http://bugs.python.org/issue17620). This complicates fixing
http://bugs.python.org/issue1602 ? the standard sys.std* streams are not
capable of communicating in Unicode with Windows console, and replacing the
streams with custom ones is not enough ? one has also to install a custom
readline hook, which is currently complicated. And even after installing a
custom readine hook one finds out that Python tokenizer cannot handle
UTF-16, so he has to wrap the custom stream objects just to let their
encoding attribute have a different value, because readlinehook currently
returns char* rather than a Python string. For more details see the
documentation of my package: https://github.com/Drekin/win-unicode-console.

The pyreadline package also sets up a custom readline so it would benefit
if doing so would be easier. Moreover, the two consumers of PyOS_Readline
API ? the input function and the tokenizer ? assume a different encoding of
the bytes returned by the readlinehook. Effectively, one assumes
sys.stdout.encoding and the other sys.stdin.encoding, so if these two are
different, there is no way to implement a correct readline hook.

If sys.readlinehook was added, the builting input function would be just a
thin wrapper over sys.readlinehook removing the newline character and
turning no input into EOFError. I thing that the best default value for
sys.readlinehook on Windows would be stdio_readline ? just write the prompt
to sys.stdout and read a line from sys.stdin. On Linux, the default
implementation would call GNU readline if it is available and sys.stdin and
sys.stdout are standard TTYs (the check present in the current
implementation of the input function), and it would call stdio_readline
otherwise.

Regards, Adam Barto?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151119/bc99be19/attachment.html>

From drekin at gmail.com  Fri Nov 20 10:23:49 2015
From: drekin at gmail.com (=?UTF-8?B?QWRhbSBCYXJ0b8Wh?=)
Date: Fri, 20 Nov 2015 16:23:49 +0100
Subject: [Python-Dev] Improving the reading part of REPL
In-Reply-To: <CACvLUa=CEbVh=b9QM8+S+faAid_K-NFpsHD02mspwcfHUjUwjw@mail.gmail.com>
References: <CACvLUa=CEbVh=b9QM8+S+faAid_K-NFpsHD02mspwcfHUjUwjw@mail.gmail.com>
Message-ID: <CACvLUam93oVeM_zJgZ9NNJmZp59XQ7303u7d_UML+nAxuWHF5A@mail.gmail.com>

Another issue with the current implementation is
http://bugs.python.org/issue24829. Even if I fix my Python environment by
win_unicode_console so >>> "?" really results in "?" rather than "?", the
feature vanishes when I try to redirect stdout.

On Thu, Nov 19, 2015 at 10:50 PM, Adam Barto? <drekin at gmail.com> wrote:

> It seems that there will be some refactoring of the tokenizer code.
> Regarding this, I'd like to recall my proposal on readline hooks. It would
> be nice if char* based PyOS_Readline API was replaced by a Python str based
> hook customizable by Python code. I propose to add function
> sys.readlinehook accepting optional prompt and returning a line read
> interactively from a user. There would also be sys.__readlinehook__
> containing the original value of sys.readlinehook (similarly to
> sys.(__)displayhook(__), sys.(__)excepthook(__) and
> sys.(__)std(in/out/err)(__)).
>
> Currently, the input is read from C stdin even if sys.stdin is changed
> (see http://bugs.python.org/issue17620). This complicates fixing
> http://bugs.python.org/issue1602 ? the standard sys.std* streams are not
> capable of communicating in Unicode with Windows console, and replacing the
> streams with custom ones is not enough ? one has also to install a custom
> readline hook, which is currently complicated. And even after installing a
> custom readine hook one finds out that Python tokenizer cannot handle
> UTF-16, so he has to wrap the custom stream objects just to let their
> encoding attribute have a different value, because readlinehook currently
> returns char* rather than a Python string. For more details see the
> documentation of my package: https://github.com/Drekin/win-unicode-console
> .
>
> The pyreadline package also sets up a custom readline so it would benefit
> if doing so would be easier. Moreover, the two consumers of PyOS_Readline
> API ? the input function and the tokenizer ? assume a different encoding of
> the bytes returned by the readlinehook. Effectively, one assumes
> sys.stdout.encoding and the other sys.stdin.encoding, so if these two are
> different, there is no way to implement a correct readline hook.
>
> If sys.readlinehook was added, the builting input function would be just a
> thin wrapper over sys.readlinehook removing the newline character and
> turning no input into EOFError. I thing that the best default value for
> sys.readlinehook on Windows would be stdio_readline ? just write the prompt
> to sys.stdout and read a line from sys.stdin. On Linux, the default
> implementation would call GNU readline if it is available and sys.stdin and
> sys.stdout are standard TTYs (the check present in the current
> implementation of the input function), and it would call stdio_readline
> otherwise.
>
> Regards, Adam Barto?
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151120/b845fb47/attachment.html>

From yselivanov.ml at gmail.com  Fri Nov 20 11:30:40 2015
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Fri, 20 Nov 2015 11:30:40 -0500
Subject: [Python-Dev] nested try..except bug in generators
Message-ID: <564F4AB0.5060409@gmail.com>

Hi,

For anyone interested in tickling the low-level CPython
internals (in this case eval loop and generators
implementations), here's an interesting bug:
http://bugs.python.org/issue25612.

I have a patch that fixes it, but I'm not sure that it's
correct, and I desperately need someone to review it.

While debugging the issue, I found another oddity:

     class MainError(Exception): pass
     class SubError(Exception): pass

     def main():
         try:
             raise MainError()
         except MainError:
             yield

     coro = main()
     coro.send(None)
     coro.throw(SubError())

On the last line of the above snippet, SubError will
propagate.  However, it won't have it's __context__ set
to MainError.  Martin Panter suggests that it's not a bug,
and after giving this some thought, I now don't think it's
a bug too -- SubError was instantiated outside of MainError
from a code location where there was no error.  Thoughts?
Here's a separate issue for the __context__ bug/feature:
http://bugs.python.org/issue25683

Thanks,
Yury

From status at bugs.python.org  Fri Nov 20 12:08:26 2015
From: status at bugs.python.org (Python tracker)
Date: Fri, 20 Nov 2015 18:08:26 +0100 (CET)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20151120170826.D92B5567EC@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2015-11-13 - 2015-11-20)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5260 (+28)
  closed 32181 (+38)
  total  37441 (+66)

Open issues with patches: 2311 


Issues opened (51)
==================

#25589: test_ascii_formatd fails on Mac when built with Intel compiler
http://bugs.python.org/issue25589  reopened by christopher.hogan

#25622: Enable ctypes.test.test_values.Win_ValuesTestCase for non-Wind
http://bugs.python.org/issue25622  opened by martin.panter

#25623: Keep the link to Python implementation of OrderedDict
http://bugs.python.org/issue25623  opened by serhiy.storchaka

#25624: shutil.make_archive makes invalid directory entries
http://bugs.python.org/issue25624  opened by gumblex

#25625: "chdir" Contex manager for pathlib
http://bugs.python.org/issue25625  opened by J??chym Barv??nek

#25626: Gzip fails for file over 2**32 bytes
http://bugs.python.org/issue25626  opened by Ben Cipollini

#25627: distutils : file "bdist_rpm.py" does not quote filenames when 
http://bugs.python.org/issue25627  opened by TheRegRunner

#25628: Make namedtuple "verbose" and "rename" parameters into keyword
http://bugs.python.org/issue25628  opened by rhettinger

#25631: Segmentation fault with invalid Unicode command-line arguments
http://bugs.python.org/issue25631  opened by itaibn

#25632: Document BUILD_*_UNPACK opcodes
http://bugs.python.org/issue25632  opened by Antony.Lee

#25633: The documentation for urllib.request should mention http.clien
http://bugs.python.org/issue25633  opened by jon orebro

#25635: urllib2 cannot fully read FTP file
http://bugs.python.org/issue25635  opened by keakon

#25636: Let launcher autoselect best 32-bit Python on 64-bit system
http://bugs.python.org/issue25636  opened by Christian.Ullrich

#25637: Move non-collections-related ABCs out of collections.abc
http://bugs.python.org/issue25637  opened by brett.cannon

#25638: Verify the etree_parse and etree_iterparse benchmarks are work
http://bugs.python.org/issue25638  opened by brett.cannon

#25639: open 'PhysicalDriveN' on windows fails (since python 3.5) with
http://bugs.python.org/issue25639  opened by rokozh

#25640: EmailMessage should throw an error if policy is set to compat3
http://bugs.python.org/issue25640  opened by Matthieu Pepin

#25641: urllib/request.py/getproxies_environment() throws "dictionary 
http://bugs.python.org/issue25641  opened by alphalfalfa

#25642: Setting maxsize breaks asyncio.JoinableQueue/Queue
http://bugs.python.org/issue25642  opened by Chris Seto

#25643: Python tokenizer rewriting
http://bugs.python.org/issue25643  opened by serhiy.storchaka

#25644: Unable to open IDLE on Windows 10 with Python 2.7.10
http://bugs.python.org/issue25644  opened by jatanig

#25646: Distutils and Windows SDK 7.1
http://bugs.python.org/issue25646  opened by JGoutin

#25647: Return of asyncio.coroutine from asyncio.coroutine doesn't wor
http://bugs.python.org/issue25647  opened by vrutsky

#25649: Come up with a good way to handle module aliasing
http://bugs.python.org/issue25649  opened by brett.cannon

#25650: Mismatching documentation <=> behaviour for typing.Any
http://bugs.python.org/issue25650  opened by ebarry

#25651: Confusing output for TestCase.subTest(0)
http://bugs.python.org/issue25651  opened by ezio.melotti

#25652: collections.UserString.__rmod__() raises NameError
http://bugs.python.org/issue25652  opened by Jonathan Goble

#25653: ctypes+callbacks+fork+selinux = crash
http://bugs.python.org/issue25653  opened by arigo

#25654: test_multiprocessing_spawn ResourceWarning with -Werror
http://bugs.python.org/issue25654  opened by martin.panter

#25655: Python errors related to failures loading DLL's lack informati
http://bugs.python.org/issue25655  opened by never-eat-yellow-snow

#25656: multiprocessing.dummy: pool.map hangs on empty list
http://bugs.python.org/issue25656  opened by tomer70

#25657: virtualenv's activate does not update LD_LIBRARY_PATH
http://bugs.python.org/issue25657  opened by serge-sans-paille

#25658: PyThread assumes pthread_key_t is an integer, which is against
http://bugs.python.org/issue25658  opened by EdSchouten

#25659: ctypes.Array.from_buffer segmentation fault when trying to cre
http://bugs.python.org/issue25659  opened by spearsem at gmail.com

#25660: tabs don't work correctly in python repl
http://bugs.python.org/issue25660  opened by yselivanov

#25661: tokenize.untokenize does not maintain the order of tabbed inde
http://bugs.python.org/issue25661  opened by Blake Hilliard

#25663: Make rlcompleter avoid duplicate global names
http://bugs.python.org/issue25663  opened by martin.panter

#25664: Unexpected UnicodeDecodeError in logging module
http://bugs.python.org/issue25664  opened by zephor

#25666: Python unexpectedly ignores a signal after fork
http://bugs.python.org/issue25666  opened by jon orebro

#25667: Supply dual-stack (IPv4/IPv6) socket bind routine
http://bugs.python.org/issue25667  opened by jason.coombs

#25668: Deadlock in logging caused by a possible race condition with "
http://bugs.python.org/issue25668  opened by fviard

#25669: unittest.assertEqual() on un-equal types inheriting from colle
http://bugs.python.org/issue25669  opened by Nathan Herring

#25671: Fix venv activate.fish to maintain $status
http://bugs.python.org/issue25671  opened by Lucretiel

#25672: set SSL_MODE_RELEASE_BUFFERS
http://bugs.python.org/issue25672  opened by Lukasa

#25674: test_ssl (test_algorithms) failures on bolen-ubuntu slaves: sh
http://bugs.python.org/issue25674  opened by db3l

#25675: doc for BaseEventLoop.run_in_executor() says its a coroutine, 
http://bugs.python.org/issue25675  opened by ealfie

#25677: Syntax error caret confused by indentation
http://bugs.python.org/issue25677  opened by Michael Layzell

#25678: int() from a buffer reads past the buffer boundaries
http://bugs.python.org/issue25678  opened by svenberkvens

#25680: Selector.select() hangs when there is nothing to select
http://bugs.python.org/issue25680  opened by Aleksey Kladov

#25682: __package__ not set to None under pdb in Python 3
http://bugs.python.org/issue25682  opened by Keith Prussing

#25683: __context__ for yields inside except clause
http://bugs.python.org/issue25683  opened by yselivanov



Most recent 15 issues with no replies (15)
==========================================

#25683: __context__ for yields inside except clause
http://bugs.python.org/issue25683

#25682: __package__ not set to None under pdb in Python 3
http://bugs.python.org/issue25682

#25675: doc for BaseEventLoop.run_in_executor() says its a coroutine, 
http://bugs.python.org/issue25675

#25671: Fix venv activate.fish to maintain $status
http://bugs.python.org/issue25671

#25664: Unexpected UnicodeDecodeError in logging module
http://bugs.python.org/issue25664

#25661: tokenize.untokenize does not maintain the order of tabbed inde
http://bugs.python.org/issue25661

#25658: PyThread assumes pthread_key_t is an integer, which is against
http://bugs.python.org/issue25658

#25657: virtualenv's activate does not update LD_LIBRARY_PATH
http://bugs.python.org/issue25657

#25653: ctypes+callbacks+fork+selinux = crash
http://bugs.python.org/issue25653

#25651: Confusing output for TestCase.subTest(0)
http://bugs.python.org/issue25651

#25650: Mismatching documentation <=> behaviour for typing.Any
http://bugs.python.org/issue25650

#25647: Return of asyncio.coroutine from asyncio.coroutine doesn't wor
http://bugs.python.org/issue25647

#25644: Unable to open IDLE on Windows 10 with Python 2.7.10
http://bugs.python.org/issue25644

#25643: Python tokenizer rewriting
http://bugs.python.org/issue25643

#25636: Let launcher autoselect best 32-bit Python on 64-bit system
http://bugs.python.org/issue25636



Most recent 15 issues waiting for review (15)
=============================================

#25678: int() from a buffer reads past the buffer boundaries
http://bugs.python.org/issue25678

#25677: Syntax error caret confused by indentation
http://bugs.python.org/issue25677

#25672: set SSL_MODE_RELEASE_BUFFERS
http://bugs.python.org/issue25672

#25671: Fix venv activate.fish to maintain $status
http://bugs.python.org/issue25671

#25663: Make rlcompleter avoid duplicate global names
http://bugs.python.org/issue25663

#25660: tabs don't work correctly in python repl
http://bugs.python.org/issue25660

#25654: test_multiprocessing_spawn ResourceWarning with -Werror
http://bugs.python.org/issue25654

#25647: Return of asyncio.coroutine from asyncio.coroutine doesn't wor
http://bugs.python.org/issue25647

#25643: Python tokenizer rewriting
http://bugs.python.org/issue25643

#25628: Make namedtuple "verbose" and "rename" parameters into keyword
http://bugs.python.org/issue25628

#25627: distutils : file "bdist_rpm.py" does not quote filenames when 
http://bugs.python.org/issue25627

#25626: Gzip fails for file over 2**32 bytes
http://bugs.python.org/issue25626

#25624: shutil.make_archive makes invalid directory entries
http://bugs.python.org/issue25624

#25623: Keep the link to Python implementation of OrderedDict
http://bugs.python.org/issue25623

#25622: Enable ctypes.test.test_values.Win_ValuesTestCase for non-Wind
http://bugs.python.org/issue25622



Top 10 most discussed issues (10)
=================================

#25626: Gzip fails for file over 2**32 bytes
http://bugs.python.org/issue25626  13 msgs

#25612: nested try..excepts don't work correctly for generators
http://bugs.python.org/issue25612  12 msgs

#25660: tabs don't work correctly in python repl
http://bugs.python.org/issue25660   8 msgs

#25654: test_multiprocessing_spawn ResourceWarning with -Werror
http://bugs.python.org/issue25654   7 msgs

#25672: set SSL_MODE_RELEASE_BUFFERS
http://bugs.python.org/issue25672   7 msgs

#23883: __all__ lists are incomplete
http://bugs.python.org/issue23883   6 msgs

#25638: Verify the etree_parse and etree_iterparse benchmarks are work
http://bugs.python.org/issue25638   6 msgs

#25646: Distutils and Windows SDK 7.1
http://bugs.python.org/issue25646   6 msgs

#25628: Make namedtuple "verbose" and "rename" parameters into keyword
http://bugs.python.org/issue25628   5 msgs

#25640: EmailMessage should throw an error if policy is set to compat3
http://bugs.python.org/issue25640   5 msgs



Issues closed (36)
==================

#6973: subprocess.Popen.send_signal doesn't check whether the process
http://bugs.python.org/issue6973  closed by gregory.p.smith

#8823: urllib2 does not catch httplib.BadStatusLine
http://bugs.python.org/issue8823  closed by martin.panter

#17131: subprocess.Popen.terminate can raise exception on Posix
http://bugs.python.org/issue17131  closed by gregory.p.smith

#20220: test_datetime, test_tarfile, test_strptime time zone failures
http://bugs.python.org/issue20220  closed by martin.panter

#20387: tokenize/untokenize roundtrip fails with tabs
http://bugs.python.org/issue20387  closed by jason.coombs

#21779: test_multiprocessing_spawn fails when ran with -Werror
http://bugs.python.org/issue21779  closed by serhiy.storchaka

#23564: Patch fixing sanity check for ordered fd sequence in _posixsub
http://bugs.python.org/issue23564  closed by gregory.p.smith

#24172: Errors in resource.getpagesize module documentation
http://bugs.python.org/issue24172  closed by martin.panter

#24623: Parser: broken line numbers for triple-quoted strings
http://bugs.python.org/issue24623  closed by serhiy.storchaka

#25017: htmllib deprecated: Which library to use? Missing sane default
http://bugs.python.org/issue25017  closed by martin.panter

#25168: test_datetime.test_strptime() random failures on "s390x SLES" 
http://bugs.python.org/issue25168  closed by martin.panter

#25388: tokenizer crash/misbehavior -- heap use-after-free
http://bugs.python.org/issue25388  closed by serhiy.storchaka

#25472: Typing: Specialized subclasses of generics cannot be unpickled
http://bugs.python.org/issue25472  closed by gvanrossum

#25498: Python 3.4.3 regression in ctypes.frombuffer()
http://bugs.python.org/issue25498  closed by martin.panter

#25557: Optimize LOAD_NAME bytecode
http://bugs.python.org/issue25557  closed by haypo

#25578: Memory leak in SSLSocket.getpeercert() with 0-length AIA exten
http://bugs.python.org/issue25578  closed by python-dev

#25583: os.makedirs with exist_ok=True raises PermissionError on Windo
http://bugs.python.org/issue25583  closed by martin.panter

#25590: tab-completition on instances repeatedly accesses attribute/de
http://bugs.python.org/issue25590  closed by martin.panter

#25611: test_deque failure on Gentoo and OpenIndiana buildbots on 2.7
http://bugs.python.org/issue25611  closed by rhettinger

#25615: Document unsorted behaviour of glob.glob
http://bugs.python.org/issue25615  closed by martin.panter

#25618: gdk attribute error
http://bugs.python.org/issue25618  closed by r.david.murray

#25619: Exception AttributeError: "'NoneType'.... thrown on exit
http://bugs.python.org/issue25619  closed by r.david.murray

#25620: Bytes Issue
http://bugs.python.org/issue25620  closed by SilentGhost

#25621: ast.literal_eval() fails on docstrings
http://bugs.python.org/issue25621  closed by r.david.murray

#25629: Move set fill/used updates out of set_insert_clean()
http://bugs.python.org/issue25629  closed by rhettinger

#25630: SIGSEGV in os.putenv()
http://bugs.python.org/issue25630  closed by python-dev

#25634: Add a dedicated subclass for attribute missing errors
http://bugs.python.org/issue25634  closed by r.david.murray

#25645: Reference leak in test_capi, on "import _pickle" in a subinter
http://bugs.python.org/issue25645  closed by python-dev

#25648: asyncio.coroutine fails if asyncio debug mode is enabled and w
http://bugs.python.org/issue25648  closed by gvanrossum

#25662: _tkinter.TclError: bad event type or keysym "Alt"
http://bugs.python.org/issue25662  closed by ronixx

#25665: typing.NamedTuple instances are not picklable.
http://bugs.python.org/issue25665  closed by gvanrossum

#25670: ast.NodeTransformer: remove duplicate getattr()
http://bugs.python.org/issue25670  closed by haypo

#25673: unittest assertLessEqual not working properly with lists
http://bugs.python.org/issue25673  closed by ezio.melotti

#25676: Spam
http://bugs.python.org/issue25676  closed by zach.ware

#25679: Fix typo in Doc/reference/executionmodel.rst
http://bugs.python.org/issue25679  closed by r.david.murray

#25681: Assignment of one element in nested list changes multiple elem
http://bugs.python.org/issue25681  closed by haypo

From benjamin at python.org  Sat Nov 21 21:48:27 2015
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 21 Nov 2015 18:48:27 -0800
Subject: [Python-Dev] [RELASE] Python 2.7.11 release candidate 1
Message-ID: <1448160507.1985354.446437097.5E584CD6@webmail.messagingengine.com>

Greetings,
It is my duty and honor to announce the immediate availability of Python
2.7.11 release candidate 1. Python 2.7.11 will be another bug fix
release for the 2.7.x series.

Downloads are at:

    https://www.python.org/downloads/release/python-2711rc1/

Please test the candidate and report bugs to

   https://bugs.python.org

If no serious problems are found, 2.7.11 final will be released in two
weeks.

Regards,
Benjamin Peterson
(on behalf of Python 2.7.11's contributors)

From larry at hastings.org  Sun Nov 22 03:12:35 2015
From: larry at hastings.org (Larry Hastings)
Date: Sun, 22 Nov 2015 00:12:35 -0800
Subject: [Python-Dev] Release schedule for 3.4.4
Message-ID: <565178F3.5070701@hastings.org>



Sorry for the short notice of this schedule, but... here goes:

    Sat Dec 05 - tag 3.4.4rc1
    Sun Dec 06 - release 3.4.4rc1
    Sat Dec 19 - tag 3.4.4 final
    Sun Dec 20 - release 3.4.4 final


The reason for the short notice: we may have to change personnel for 
this release, and we need to work around some holiday vacation schedules 
too.  However, note that we have two weeks for the RC release.  Because 
of the short notice, I plan to be a little more flexible about changes 
between RC1 and final.  It also means we can pretty easily add a second 
RC if we need to.

Unless something amazing happens, 3.4.4 will be the final release of 3.4 
with binary installers.  3.4 will also move into "security fixes only" 
mode at that time.


Best wishes, and again my apologies for the short notice,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151122/6b13a91b/attachment.html>

From tritium-list at sdamon.com  Mon Nov 23 01:21:43 2015
From: tritium-list at sdamon.com (Alexander Walters)
Date: Mon, 23 Nov 2015 01:21:43 -0500
Subject: [Python-Dev] collections.Counter __add__ implementation quirk
Message-ID: <5652B077.9000606@sdamon.com>

collections.Counter.__add__ as a bit of a quirk.

Counters allow for negative numbers.  You can subtract from a counter 
into the negative no problem.  However, if you have a counter with a 
negative value and add it to another counter, and if that value, after 
addition, would still be negative... that value is not included in the 
resulting Counter object.  This is kind of weird, to the point of 
thinking I had a bug in other code for several hours until I went and 
checked how Counters are implemented.

Is there any particular reason counters drop negative values when you 
add them together?  I definitely expected them to act like ints do when 
you add negatives, and had to subclass it to get what I think is the 
obvious behavior.

From larry at hastings.org  Mon Nov 23 02:16:44 2015
From: larry at hastings.org (Larry Hastings)
Date: Sun, 22 Nov 2015 23:16:44 -0800
Subject: [Python-Dev] [RELEASED] Python 3.5.1rc1 is now available
Message-ID: <5652BD5C.6040107@hastings.org>


On behalf of the Python development community and the Python 3.5 release 
team, I'm pleased to announce the availability of Python 3.5.1rc1.  
Python 3.5.1 will be the first update for Python 3.5.  Python 3.5 is the 
newest version of the Python language, and it contains many exciting new 
features and optimizations.


You can see what's changed in Python 3.5.1rc1 (as compared to 3.5.0) here:

    https://docs.python.org/3.5/whatsnew/changelog.html


And you can download Python 3.5.1 here:

    https://www.python.org/downloads/release/python-351rc1/

Windows and Mac users: please read the important platform-specific 
"Notes on this release" section near the end of that page.



We hope you enjoy Python 3.5.1!


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151122/b2c70690/attachment.html>

From larry at hastings.org  Mon Nov 23 02:47:33 2015
From: larry at hastings.org (Larry Hastings)
Date: Sun, 22 Nov 2015 23:47:33 -0800
Subject: [Python-Dev] Reminder: checkins into 3.5 after rc1 won't
 automatically go into final
Message-ID: <5652C495.2080804@hastings.org>



3.5.1rc1 is tagged, merged, and pushed back into the central repo. At 
this point I don't plan to merge further changes from hg.python.org into 
3.5.1 final.  If you have an important bug fix that didn't make it into 
3.5.1rc1 and *has* to go into 3.5.1 final, add me to the issue and we'll 
talk about it.  If it happens I'll have to manually cherry-pick the change.

I've added a 3.5.2rc1 section to Misc/NEWS; please add new entries there.

Happy holidays,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151122/eb4148aa/attachment.html>

From vlastimil.brom at gmail.com  Mon Nov 23 13:43:15 2015
From: vlastimil.brom at gmail.com (Vlastimil Brom)
Date: Mon, 23 Nov 2015 19:43:15 +0100
Subject: [Python-Dev] collections.Counter __add__ implementation quirk
In-Reply-To: <5652B077.9000606@sdamon.com>
References: <5652B077.9000606@sdamon.com>
Message-ID: <CAHzaPEMveK8FeHAxExCAnbbhDahodr67o+99PqN6ajw99c+xDA@mail.gmail.com>

2015-11-23 7:21 GMT+01:00 Alexander Walters <tritium-list at sdamon.com>:
> collections.Counter.__add__ as a bit of a quirk.
>
> Counters allow for negative numbers.  You can subtract from a counter into
> the negative no problem.  However, if you have a counter with a negative
> value and add it to another counter, and if that value, after addition,
> would still be negative... that value is not included in the resulting
> Counter object.  This is kind of weird, to the point of thinking I had a bug
> in other code for several hours until I went and checked how Counters are
> implemented.
>
> Is there any particular reason counters drop negative values when you add
> them together?  I definitely expected them to act like ints do when you add
> negatives, and had to subclass it to get what I think is the obvious
> behavior.
> _______________________________________________
> Python-Dev mailing list
...
Hi,
this is probably more appropriate for the general python list rathere
then this developers' maillist, however, as I asked a similar question
some time ago, I got some detailed explanations for the the current
design decissions from the original developer; cf.:
https://mail.python.org/pipermail/python-list/2010-March/570618.html

(I didn't check possible changes in Counter since that version (3.1 at
that time).)

hth,
  vbr

From christian at python.org  Mon Nov 23 14:35:06 2015
From: christian at python.org (Christian Heimes)
Date: Mon, 23 Nov 2015 20:35:06 +0100
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
Message-ID: <56536A6A.9030902@python.org>

On 2015-11-17 01:00, Guido van Rossum wrote:
> Hm, making Christian the BDFL-delegate would mean two out of three
> authors *and* the BDFL-delegate all working for Red Hat, which clearly
> has a stake (and IIUC has already committed to this approach ahead of
> PEP approval). SO then it would look like this is just rubber-stamping
> Red Hat's internal decision process (if it's a process -- sounds more
> like an accident :-).
> 
> So, Alex, do you want to approve this PEP?

I haven't read this thread until now. Independently from your objection
I have raised the same concern with Nick today. I'd be willing to BDFL
the PEP but I'd rather have somebody outside of Red Hat. Alex is a great
choice.


In the same mail I sent Nick a quick review of the latest PEP version in
private.


1) The example implementation of the function doesn't check the
sys.flags.ignore_environment. Internally CPython has specialized getenv
function that ignores env vars with PYTHON prefix when the flag is set.
PYTHON* env vars aren't removed from os.environ. Modules have to check
the flag.


2) The PEP is rather Linux-centric. What's the recommended path to the
config file on other platforms like BDS (/usr/local/etc/ is preferred
for additional dependencies on FreeBSD), OSX and Windows?


3) What's the interaction between the location of the config file and
virtual envs? Would it make sense to search for the file in a venv's
etc/ first and then dispatch to global /etc/? That way venvs can
influence the setting, too.


4) It makes sense to make the cert-verification.cfg file future-proof
and reserve it for other cert-related configuration in the future. For
example it could be used to define new contexts, set protocols, ciphers
or hashes for cert pinning. It should be enough to say that CPython
reserves the right to add more sections and keys later.


5) I'm not particular fond of the section name [https]. For one It is
ambiguous because it doesn't distinguish between server certs and client
certs. It's also not correct. The default context is used for other
protocols like imap, smtp etc. over TLS.

Christian

From christian at python.org  Mon Nov 23 14:35:06 2015
From: christian at python.org (Christian Heimes)
Date: Mon, 23 Nov 2015 20:35:06 +0100
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
Message-ID: <56536A6A.9030902@python.org>

On 2015-11-17 01:00, Guido van Rossum wrote:
> Hm, making Christian the BDFL-delegate would mean two out of three
> authors *and* the BDFL-delegate all working for Red Hat, which clearly
> has a stake (and IIUC has already committed to this approach ahead of
> PEP approval). SO then it would look like this is just rubber-stamping
> Red Hat's internal decision process (if it's a process -- sounds more
> like an accident :-).
> 
> So, Alex, do you want to approve this PEP?

I haven't read this thread until now. Independently from your objection
I have raised the same concern with Nick today. I'd be willing to BDFL
the PEP but I'd rather have somebody outside of Red Hat. Alex is a great
choice.


In the same mail I sent Nick a quick review of the latest PEP version in
private.


1) The example implementation of the function doesn't check the
sys.flags.ignore_environment. Internally CPython has specialized getenv
function that ignores env vars with PYTHON prefix when the flag is set.
PYTHON* env vars aren't removed from os.environ. Modules have to check
the flag.


2) The PEP is rather Linux-centric. What's the recommended path to the
config file on other platforms like BDS (/usr/local/etc/ is preferred
for additional dependencies on FreeBSD), OSX and Windows?


3) What's the interaction between the location of the config file and
virtual envs? Would it make sense to search for the file in a venv's
etc/ first and then dispatch to global /etc/? That way venvs can
influence the setting, too.


4) It makes sense to make the cert-verification.cfg file future-proof
and reserve it for other cert-related configuration in the future. For
example it could be used to define new contexts, set protocols, ciphers
or hashes for cert pinning. It should be enough to say that CPython
reserves the right to add more sections and keys later.


5) I'm not particular fond of the section name [https]. For one It is
ambiguous because it doesn't distinguish between server certs and client
certs. It's also not correct. The default context is used for other
protocols like imap, smtp etc. over TLS.

Christian


From alex.gaynor at gmail.com  Mon Nov 23 14:38:36 2015
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Mon, 23 Nov 2015 14:38:36 -0500
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <56536A6A.9030902@python.org>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <56536A6A.9030902@python.org>
Message-ID: <CAFRnB2Whft-YwDRnX2NOnBSvm_ygCg3RLNr_+D0y942B7ZUMWQ@mail.gmail.com>

Hi all,

While I appreciate the vote of confidence from everyone, I'm not interested
in being the BDFL-delegate for this. I don't think it's a good idea, and
I'm not willing to put further time into.

If he's interested, Donald Stufft would make a good choice for delegate.

Really do appreciate everyone's confidence.

Cheers,
Alex

On Mon, Nov 23, 2015 at 2:35 PM, Christian Heimes <christian at python.org>
wrote:

> On 2015-11-17 01:00, Guido van Rossum wrote:
> > Hm, making Christian the BDFL-delegate would mean two out of three
> > authors *and* the BDFL-delegate all working for Red Hat, which clearly
> > has a stake (and IIUC has already committed to this approach ahead of
> > PEP approval). SO then it would look like this is just rubber-stamping
> > Red Hat's internal decision process (if it's a process -- sounds more
> > like an accident :-).
> >
> > So, Alex, do you want to approve this PEP?
>
> I haven't read this thread until now. Independently from your objection
> I have raised the same concern with Nick today. I'd be willing to BDFL
> the PEP but I'd rather have somebody outside of Red Hat. Alex is a great
> choice.
>
>
> In the same mail I sent Nick a quick review of the latest PEP version in
> private.
>
>
> 1) The example implementation of the function doesn't check the
> sys.flags.ignore_environment. Internally CPython has specialized getenv
> function that ignores env vars with PYTHON prefix when the flag is set.
> PYTHON* env vars aren't removed from os.environ. Modules have to check
> the flag.
>
>
> 2) The PEP is rather Linux-centric. What's the recommended path to the
> config file on other platforms like BDS (/usr/local/etc/ is preferred
> for additional dependencies on FreeBSD), OSX and Windows?
>
>
> 3) What's the interaction between the location of the config file and
> virtual envs? Would it make sense to search for the file in a venv's
> etc/ first and then dispatch to global /etc/? That way venvs can
> influence the setting, too.
>
>
> 4) It makes sense to make the cert-verification.cfg file future-proof
> and reserve it for other cert-related configuration in the future. For
> example it could be used to define new contexts, set protocols, ciphers
> or hashes for cert pinning. It should be enough to say that CPython
> reserves the right to add more sections and keys later.
>
>
> 5) I'm not particular fond of the section name [https]. For one It is
> ambiguous because it doesn't distinguish between server certs and client
> certs. It's also not correct. The default context is used for other
> protocols like imap, smtp etc. over TLS.
>
> Christian
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151123/42948481/attachment.html>

From wes.turner at gmail.com  Mon Nov 23 15:47:52 2015
From: wes.turner at gmail.com (Wes Turner)
Date: Mon, 23 Nov 2015 14:47:52 -0600
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
Message-ID: <CACfEFw_67sGz=GBPvET1uJzgciXkxWitoAP7YceAcZbSsEnryA@mail.gmail.com>

1. Does this affect easy_install?
2. If/because this affects easy_install,
  should the guidance / suggested package installation tool be [pip];
because pip install_requires backports.ssl_match_hostname

https://pypi.python.org/pypi/backports.ssl_match_hostname
On Nov 10, 2015 6:48 PM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:

> Hi folks,
>
> I have a confession to make - I dropped the ball on the HTTPS
> verification backport proposals in PEP 493, and let the upstream and
> downstream approval processes get out of sequence.
>
> As a result, the RHEL 7.2 beta released back in September incorporates
> the HTTPS verification feature backport based on the current PEP 493
> draft, even though that hasn't formally been pronounced as an Active
> recommendation by python-dev yet.
>
> Accordingly, I'm belatedly submitting it for pronouncement now:
> https://www.python.org/dev/peps/pep-0493/
>
> There's currently no BDFL-Delegate assigned, so if Guido doesn't want
> to handle it, we'll need to address that question first.
>
> Our last discussion back in July seemed to show that folks either
> didn't care about the question (because they're using unmodified
> upstream versions so the PEP didn't affect them), or else thought the
> approach described in the PEP was reasonable, so I'm hoping the
> consequences of my mistake won't be too severe.
>
> Regards,
> Nick.
>
> P.S. I'm aware that this looks like presenting a fait accompli at a
> point where it's too late to realistically say "No", but the truth is
> that preparation for the Python in Education miniconf at PyCon
> Australia ramped up immediately after the July discussion, and then I
> personally got confused as to the scope of what was being included in
> 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493
> being deferred to a later release, but it's actually the whole package
> of 466+476+493). That's my fault for trying to keep track of too many
> things at once (and thus failing at some of them), not anyone else's.
>
> ================================
>
> PEP: 493
> Title: HTTPS verification recommendations for Python 2.7 redistributors
> Version: $Revision$
> Last-Modified: $Date$
> Author: Nick Coghlan <ncoghlan at gmail.com>,
>         Robert Kuska <rkuska at redhat.com>,
>         Marc-Andr? Lemburg <mal at lemburg.com>
> Status: Draft
> Type: Informational
> Content-Type: text/x-rst
> Created: 10-May-2015
> Post-History: 06-Jul-2015
>
>
> Abstract
> ========
>
> PEP 476 updated Python's default handling of HTTPS certificates to be
> appropriate for communication over the public internet. The Python 2.7 long
> term maintenance series was judged to be in scope for this change, with the
> new behaviour introduced in the Python 2.7.9 maintenance release.
>
> This PEP provides recommendations to downstream redistributors wishing to
> provide a smoother migration experience when helping their users to manage
> this change in Python's default behaviour.
>
>
> Rationale
> =========
>
> PEP 476 changed Python's default behaviour to better match the needs and
> expectations of developers operating over the public internet, a category
> which appears to include most new Python developers. It is the position of
> the authors of this PEP that this was a correct decision.
>
> However, it is also the case that this change *does* cause problems for
> infrastructure administrators operating private intranets that rely on
> self-signed certificates, or otherwise encounter problems with the new
> default
> certificate verification settings.
>
> The long term answer for such environments is to update their internal
> certificate management to at least match the standards set by the public
> internet, but in the meantime, it is desirable to offer these
> administrators
> a way to continue receiving maintenance updates to the Python 2.7 series,
> without having to gate that on upgrades to their certificate management
> infrastructure.
>
> PEP 476 did attempt to address this question, by covering how to revert the
> new settings process wide by monkeypatching the ``ssl`` module to restore
> the
> old behaviour. Unfortunately, the ``sitecustomize.py`` based technique
> proposed
> to allow system administrators to disable the feature by default in their
> Standard Operating Environment definition has been determined to be
> insufficient in at least some cases. The specific case of interest to the
> authors of this PEP is the one where a Linux distributor aims to provide
> their users with a
> `smoother migration path
> <https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__
> than the standard one provided by consuming upstream CPython 2.7 releases
> directly, but other potential challenges have also been pointed out with
> updating embedded Python runtimes and other user level installations of
> Python.
>
> Rather than allowing a plethora of mutually incompatibile migration
> techniques
> to bloom, this PEP proposes two alternative approaches that redistributors
> may take when addressing these problems. Redistributors may choose to
> implement
> one, both, or neither of these approaches based on their assessment of the
> needs of their particular userbase.
>
> These designs are being proposed as a recommendation for redistributors,
> rather
> than as new upstream features, as they are needed purely to support legacy
> environments migrating from older versions of Python 2.7. Neither approach
> is being proposed as an upstream Python 2.7 feature, nor as a feature in
> any
> version of Python 3 (whether published directly by the Python Software
> Foundation or by a redistributor).
>
>
> Requirements for capability detection
> =====================================
>
> As these recommendations are intended to cover backports to earlier Python
> versions, the Python version number cannot be used as a reliable means for
> detecting them. Instead, the recommendations are defined to allow the
> presence
> or absence of the feature to be determined using the following technique::
>
>     python -c "import ssl; ssl._relevant_attribute"
>
> This will fail with `AttributeError` (and hence a non-zero return code) if
> the
> relevant capability is not available.
>
> The marker attributes are prefixed with an underscore to indicate the
> implementation dependent nature of these capabilities - not all Python
> distributions will offer them, only those that are providing a multi-stage
> migration process from the legacy HTTPS handling to the new default
> behaviour.
>
>
> Recommendation for an environment variable based security downgrade
> ===================================================================
>
> Some redistributors may wish to provide a per-application option to disable
> certificate verification in selected applications that run on or embed
> CPython
> without needing to modify the application itself.
>
> In these cases, a configuration mechanism is needed that provides:
>
> * an opt-out model that allows certificate verification to be selectively
>   turned off for particular applications after upgrading to a version of
>   Python that verifies certificates by default
> * the ability for all users to configure this setting on a per-application
>   basis, rather than on a per-system, or per-Python-installation basis
>
> This approach may be used for any redistributor provided version of Python
> 2.7,
> including those that advertise themselves as providing Python 2.7.9 or
> later.
>
>
> Required marker attribute
> -------------------------
>
> The required marker attribute on the ``ssl`` module when implementing this
> recommendation is::
>
>     _https_verify_envvar = 'PYTHONHTTPSVERIFY'
>
> This not only makes it straightforward to detect the presence (or absence)
> of
> the capability, it also makes it possible to programmatically determine the
> relevant environment variable name.
>
>
> Recommended modifications to the Python standard library
> --------------------------------------------------------
>
> The recommended approach to providing a per-application configuration
> setting
> for HTTPS certificate verification that doesn't require modifications to
> the
> application itself is to:
>
> * modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment
>   variable when the module is first imported into a Python process
> * set the ``ssl._create_default_https_context`` function to be an alias for
>   ``ssl._create_unverified_context`` if this environment variable is
> present
>   and set to ``'0'``
> * otherwise, set the ``ssl._create_default_https_context`` function to be
> an
>   alias for ``ssl.create_default_context`` as usual
>
>
> Example implementation
> ----------------------
>
> ::
>
>     _https_verify_envvar = 'PYTHONHTTPSVERIFY'
>
>     def _get_https_context_factory():
>         config_setting = os.environ.get(_https_verify_envvar)
>         if config_setting == '0':
>             return _create_unverified_context
>         return create_default_context
>
>     _create_default_https_context = _get_https_context_factory()
>
>
> Security Considerations
> -----------------------
>
> Relative to an unmodified version of CPython 2.7.9 or later, this approach
> does introduce a new downgrade attack against the default security settings
> that potentially allows a sufficiently determined attacker to revert Python
> to the vulnerable configuration used in CPython 2.7.8 and earlier releases.
> However, such an attack requires the ability to modify the execution
> environment of a Python process prior to the import of the ``ssl`` module,
> and any attacker with such access would already be able to modify the
> behaviour of the underlying OpenSSL implementation.
>
>
> Recommendation for backporting to earlier Python versions
> =========================================================
>
> Some redistributors, most notably Linux distributions, may choose to
> backport
> the PEP 476 HTTPS verification changes to modified Python versions based on
> earlier Python 2 maintenance releases. In these cases, a configuration
> mechanism is needed that provides:
>
> * an opt-in model that allows the decision to enable HTTPS certificate
>   verification to be made independently of the decision to upgrade to the
>   Python version where the feature was first backported
> * the ability for system administrators to set the default behaviour of
> Python
>   applications and scripts run directly in the system Python installation
> * the ability for the redistributor to consider changing the default
> behaviour
>   of *new* installations at some point in the future without impacting
> existing
>   installations that have been explicitly configured to skip verifying
> HTTPS
>   certificates by default
>
> This approach should not be used for any Python installation that
> advertises
> itself as providing Python 2.7.9 or later, as most Python users will have
> the
> reasonable expectation that all such environments will validate HTTPS
> certificates by default.
>
>
> Required marker attribute
> -------------------------
>
> The required marker attribute on the ``ssl`` module when implementing this
> recommendation is::
>
>     _cert_verification_config = '<path to configuration file>'
>
> This not only makes it straightforward to detect the presence (or absence)
> of
> the capability, it also makes it possible to programmatically determine the
> relevant configuration file name.
>
>
> Recommended modifications to the Python standard library
> --------------------------------------------------------
>
> The recommended approach to backporting the PEP 476 modifications to an
> earlier
> point release is to implement the following changes relative to the default
> PEP 476 behaviour implemented in Python 2.7.9+:
>
> * modify the ``ssl`` module to read a system wide configuration file when
> the
>   module is first imported into a Python process
> * define a platform default behaviour (either verifying or not verifying
> HTTPS
>   certificates) to be used if this configuration file is not present
> * support selection between the following three modes of operation:
>
>   * ensure HTTPS certificate verification is enabled
>   * ensure HTTPS certificate verification is disabled
>   * delegate the decision to the redistributor providing this Python
> version
>
> * set the ``ssl._create_default_https_context`` function to be an alias for
>   either ``ssl.create_default_context`` or
> ``ssl._create_unverified_context``
>   based on the given configuration setting.
>
>
> Recommended file location
> -------------------------
>
> This approach is currently only defined for \*nix system Python
> installations.
>
> The recommended configuration file name is
> ``/etc/python/cert-verification.cfg``.
>
> The ``.cfg`` filename extension is recommended for consistency with the
> ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library.
>
>
> Recommended file format
> -----------------------
>
> The configuration file should use a ConfigParser ini-style format with a
> single section named ``[https]`` containing one required setting
> ``verify``.
>
> Permitted values for ``verify`` are:
>
> * ``enable``: ensure HTTPS certificate verification is enabled by default
> * ``disable``: ensure HTTPS certificate verification is disabled by default
> * ``platform_default``: delegate the decision to the redistributor
> providing
>   this particular Python version
>
> If the ``[https]`` section or the ``verify`` setting are missing, or if the
> ``verify`` setting is set to an unknown value, it should be treated as if
> the
> configuration file is not present.
>
>
> Example implementation
> ----------------------
>
> ::
>
>     _cert_verification_config = '/etc/python/cert-verification.cfg'
>
>     def _get_https_context_factory():
>         # Check for a system-wide override of the default behaviour
>         context_factories = {
>             'enable': create_default_context,
>             'disable': _create_unverified_context,
>             'platform_default': _create_unverified_context, # For now :)
>         }
>         import ConfigParser
>         config = ConfigParser.RawConfigParser()
>         config.read(_cert_verification_config)
>         try:
>             verify_mode = config.get('https', 'verify')
>         except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
>             verify_mode = 'platform_default'
>         default_factory = context_factories.get('platform_default')
>         return context_factories.get(verify_mode, default_factory)
>
>     _create_default_https_context = _get_https_context_factory()
>
>
> Security Considerations
> -----------------------
>
> The specific recommendations for the backporting case are designed to work
> for
> privileged, security sensitive processes, even those being run in the
> following
> locked down configuration:
>
> * run from a locked down administrator controlled directory rather than a
> normal
>   user directory (preventing ``sys.path[0]`` based privilege escalation
> attacks)
> * run using the ``-E`` switch (preventing ``PYTHON*`` environment variable
> based
>   privilege escalation attacks)
> * run using the ``-s`` switch (preventing user site directory based
> privilege
>   escalation attacks)
> * run using the ``-S`` switch (preventing ``sitecustomize`` based privilege
>   escalation attacks)
>
> The intent is that the *only* reason HTTPS verification should be getting
> turned off system wide when using this approach is because:
>
> * an end user is running a redistributor provided version of CPython rather
>   than running upstream CPython directly
> * that redistributor has decided to provide a smoother migration path to
>   verifying HTTPS certificates by default than that being provided by the
>   upstream project
> * either the redistributor or the local infrastructure administrator has
>   determined that it is appropriate to override the default upstream
> behaviour
>   (at least for the time being)
>
> Using an administrator controlled configuration file rather than an
> environment
> variable has the essential feature of providing a smoother migration path,
> even
> for applications being run with the ``-E`` switch.
>
>
> Combining the recommendations
> =============================
>
> If a redistributor chooses to implement both recommendations, then the
> environment variable should take precedence over the system-wide
> configuration
> setting. This allows the setting to be changed for a given user, virtual
> environment or application, regardless of the system-wide default
> behaviour.
>
> In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is
> defined, and
> set to anything *other* than ``'0'``, then HTTPS certificate verification
> should be enabled.
>
> Example implementation
> ----------------------
>
> ::
>
>     _https_verify_envvar = 'PYTHONHTTPSVERIFY'
>     _cert_verification_config = '/etc/python/cert-verification.cfg'
>
>     def _get_https_context_factory():
>         # Check for am environmental override of the default behaviour
>         config_setting = os.environ.get(_https_verify_envvar)
>         if config_setting is not None:
>             if config_setting == '0':
>                 return _create_unverified_context
>             return create_default_context
>
>         # Check for a system-wide override of the default behaviour
>         context_factories = {
>             'enable': create_default_context,
>             'disable': _create_unverified_context,
>             'platform_default': _create_unverified_context, # For now :)
>         }
>         import ConfigParser
>         config = ConfigParser.RawConfigParser()
>         config.read(_cert_verification_config)
>         try:
>             verify_mode = config.get('https', 'verify')
>         except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
>             verify_mode = 'platform_default'
>         default_factory = context_factories.get('platform_default')
>         return context_factories.get(verify_mode, default_factory)
>
>     _create_default_https_context = _get_https_context_factory()
>
>
> Copyright
> =========
>
> This document has been placed into the public domain.
>
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151123/c7da2713/attachment-0001.html>

From wes.turner at gmail.com  Mon Nov 23 15:51:01 2015
From: wes.turner at gmail.com (Wes Turner)
Date: Mon, 23 Nov 2015 14:51:01 -0600
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CACfEFw_67sGz=GBPvET1uJzgciXkxWitoAP7YceAcZbSsEnryA@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CACfEFw_67sGz=GBPvET1uJzgciXkxWitoAP7YceAcZbSsEnryA@mail.gmail.com>
Message-ID: <CACfEFw9XR4O_fVLyoj6p2cpR-nSteN6K+UB77-EASOkdCwuv-w@mail.gmail.com>

... Just had this discussion in regards to easy_install, Ubuntu 14.04 LTS,
and the ReadTheDocs Docker images (as well as: ~why should I have to
wget/curl get-pip.py)
https://github.com/rtfd/readthedocs-docker-images/pull/3
On Nov 23, 2015 2:47 PM, "Wes Turner" <wes.turner at gmail.com> wrote:

> 1. Does this affect easy_install?
> 2. If/because this affects easy_install,
>   should the guidance / suggested package installation tool be [pip];
> because pip install_requires backports.ssl_match_hostname
>
> https://pypi.python.org/pypi/backports.ssl_match_hostname
> On Nov 10, 2015 6:48 PM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
>
>> Hi folks,
>>
>> I have a confession to make - I dropped the ball on the HTTPS
>> verification backport proposals in PEP 493, and let the upstream and
>> downstream approval processes get out of sequence.
>>
>> As a result, the RHEL 7.2 beta released back in September incorporates
>> the HTTPS verification feature backport based on the current PEP 493
>> draft, even though that hasn't formally been pronounced as an Active
>> recommendation by python-dev yet.
>>
>> Accordingly, I'm belatedly submitting it for pronouncement now:
>> https://www.python.org/dev/peps/pep-0493/
>>
>> There's currently no BDFL-Delegate assigned, so if Guido doesn't want
>> to handle it, we'll need to address that question first.
>>
>> Our last discussion back in July seemed to show that folks either
>> didn't care about the question (because they're using unmodified
>> upstream versions so the PEP didn't affect them), or else thought the
>> approach described in the PEP was reasonable, so I'm hoping the
>> consequences of my mistake won't be too severe.
>>
>> Regards,
>> Nick.
>>
>> P.S. I'm aware that this looks like presenting a fait accompli at a
>> point where it's too late to realistically say "No", but the truth is
>> that preparation for the Python in Education miniconf at PyCon
>> Australia ramped up immediately after the July discussion, and then I
>> personally got confused as to the scope of what was being included in
>> 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493
>> being deferred to a later release, but it's actually the whole package
>> of 466+476+493). That's my fault for trying to keep track of too many
>> things at once (and thus failing at some of them), not anyone else's.
>>
>> ================================
>>
>> PEP: 493
>> Title: HTTPS verification recommendations for Python 2.7 redistributors
>> Version: $Revision$
>> Last-Modified: $Date$
>> Author: Nick Coghlan <ncoghlan at gmail.com>,
>>         Robert Kuska <rkuska at redhat.com>,
>>         Marc-Andr? Lemburg <mal at lemburg.com>
>> Status: Draft
>> Type: Informational
>> Content-Type: text/x-rst
>> Created: 10-May-2015
>> Post-History: 06-Jul-2015
>>
>>
>> Abstract
>> ========
>>
>> PEP 476 updated Python's default handling of HTTPS certificates to be
>> appropriate for communication over the public internet. The Python 2.7
>> long
>> term maintenance series was judged to be in scope for this change, with
>> the
>> new behaviour introduced in the Python 2.7.9 maintenance release.
>>
>> This PEP provides recommendations to downstream redistributors wishing to
>> provide a smoother migration experience when helping their users to manage
>> this change in Python's default behaviour.
>>
>>
>> Rationale
>> =========
>>
>> PEP 476 changed Python's default behaviour to better match the needs and
>> expectations of developers operating over the public internet, a category
>> which appears to include most new Python developers. It is the position of
>> the authors of this PEP that this was a correct decision.
>>
>> However, it is also the case that this change *does* cause problems for
>> infrastructure administrators operating private intranets that rely on
>> self-signed certificates, or otherwise encounter problems with the new
>> default
>> certificate verification settings.
>>
>> The long term answer for such environments is to update their internal
>> certificate management to at least match the standards set by the public
>> internet, but in the meantime, it is desirable to offer these
>> administrators
>> a way to continue receiving maintenance updates to the Python 2.7 series,
>> without having to gate that on upgrades to their certificate management
>> infrastructure.
>>
>> PEP 476 did attempt to address this question, by covering how to revert
>> the
>> new settings process wide by monkeypatching the ``ssl`` module to restore
>> the
>> old behaviour. Unfortunately, the ``sitecustomize.py`` based technique
>> proposed
>> to allow system administrators to disable the feature by default in their
>> Standard Operating Environment definition has been determined to be
>> insufficient in at least some cases. The specific case of interest to the
>> authors of this PEP is the one where a Linux distributor aims to provide
>> their users with a
>> `smoother migration path
>> <https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__
>> than the standard one provided by consuming upstream CPython 2.7 releases
>> directly, but other potential challenges have also been pointed out with
>> updating embedded Python runtimes and other user level installations of
>> Python.
>>
>> Rather than allowing a plethora of mutually incompatibile migration
>> techniques
>> to bloom, this PEP proposes two alternative approaches that redistributors
>> may take when addressing these problems. Redistributors may choose to
>> implement
>> one, both, or neither of these approaches based on their assessment of the
>> needs of their particular userbase.
>>
>> These designs are being proposed as a recommendation for redistributors,
>> rather
>> than as new upstream features, as they are needed purely to support legacy
>> environments migrating from older versions of Python 2.7. Neither approach
>> is being proposed as an upstream Python 2.7 feature, nor as a feature in
>> any
>> version of Python 3 (whether published directly by the Python Software
>> Foundation or by a redistributor).
>>
>>
>> Requirements for capability detection
>> =====================================
>>
>> As these recommendations are intended to cover backports to earlier Python
>> versions, the Python version number cannot be used as a reliable means for
>> detecting them. Instead, the recommendations are defined to allow the
>> presence
>> or absence of the feature to be determined using the following technique::
>>
>>     python -c "import ssl; ssl._relevant_attribute"
>>
>> This will fail with `AttributeError` (and hence a non-zero return code)
>> if the
>> relevant capability is not available.
>>
>> The marker attributes are prefixed with an underscore to indicate the
>> implementation dependent nature of these capabilities - not all Python
>> distributions will offer them, only those that are providing a multi-stage
>> migration process from the legacy HTTPS handling to the new default
>> behaviour.
>>
>>
>> Recommendation for an environment variable based security downgrade
>> ===================================================================
>>
>> Some redistributors may wish to provide a per-application option to
>> disable
>> certificate verification in selected applications that run on or embed
>> CPython
>> without needing to modify the application itself.
>>
>> In these cases, a configuration mechanism is needed that provides:
>>
>> * an opt-out model that allows certificate verification to be selectively
>>   turned off for particular applications after upgrading to a version of
>>   Python that verifies certificates by default
>> * the ability for all users to configure this setting on a per-application
>>   basis, rather than on a per-system, or per-Python-installation basis
>>
>> This approach may be used for any redistributor provided version of
>> Python 2.7,
>> including those that advertise themselves as providing Python 2.7.9 or
>> later.
>>
>>
>> Required marker attribute
>> -------------------------
>>
>> The required marker attribute on the ``ssl`` module when implementing this
>> recommendation is::
>>
>>     _https_verify_envvar = 'PYTHONHTTPSVERIFY'
>>
>> This not only makes it straightforward to detect the presence (or
>> absence) of
>> the capability, it also makes it possible to programmatically determine
>> the
>> relevant environment variable name.
>>
>>
>> Recommended modifications to the Python standard library
>> --------------------------------------------------------
>>
>> The recommended approach to providing a per-application configuration
>> setting
>> for HTTPS certificate verification that doesn't require modifications to
>> the
>> application itself is to:
>>
>> * modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment
>>   variable when the module is first imported into a Python process
>> * set the ``ssl._create_default_https_context`` function to be an alias
>> for
>>   ``ssl._create_unverified_context`` if this environment variable is
>> present
>>   and set to ``'0'``
>> * otherwise, set the ``ssl._create_default_https_context`` function to be
>> an
>>   alias for ``ssl.create_default_context`` as usual
>>
>>
>> Example implementation
>> ----------------------
>>
>> ::
>>
>>     _https_verify_envvar = 'PYTHONHTTPSVERIFY'
>>
>>     def _get_https_context_factory():
>>         config_setting = os.environ.get(_https_verify_envvar)
>>         if config_setting == '0':
>>             return _create_unverified_context
>>         return create_default_context
>>
>>     _create_default_https_context = _get_https_context_factory()
>>
>>
>> Security Considerations
>> -----------------------
>>
>> Relative to an unmodified version of CPython 2.7.9 or later, this approach
>> does introduce a new downgrade attack against the default security
>> settings
>> that potentially allows a sufficiently determined attacker to revert
>> Python
>> to the vulnerable configuration used in CPython 2.7.8 and earlier
>> releases.
>> However, such an attack requires the ability to modify the execution
>> environment of a Python process prior to the import of the ``ssl`` module,
>> and any attacker with such access would already be able to modify the
>> behaviour of the underlying OpenSSL implementation.
>>
>>
>> Recommendation for backporting to earlier Python versions
>> =========================================================
>>
>> Some redistributors, most notably Linux distributions, may choose to
>> backport
>> the PEP 476 HTTPS verification changes to modified Python versions based
>> on
>> earlier Python 2 maintenance releases. In these cases, a configuration
>> mechanism is needed that provides:
>>
>> * an opt-in model that allows the decision to enable HTTPS certificate
>>   verification to be made independently of the decision to upgrade to the
>>   Python version where the feature was first backported
>> * the ability for system administrators to set the default behaviour of
>> Python
>>   applications and scripts run directly in the system Python installation
>> * the ability for the redistributor to consider changing the default
>> behaviour
>>   of *new* installations at some point in the future without impacting
>> existing
>>   installations that have been explicitly configured to skip verifying
>> HTTPS
>>   certificates by default
>>
>> This approach should not be used for any Python installation that
>> advertises
>> itself as providing Python 2.7.9 or later, as most Python users will have
>> the
>> reasonable expectation that all such environments will validate HTTPS
>> certificates by default.
>>
>>
>> Required marker attribute
>> -------------------------
>>
>> The required marker attribute on the ``ssl`` module when implementing this
>> recommendation is::
>>
>>     _cert_verification_config = '<path to configuration file>'
>>
>> This not only makes it straightforward to detect the presence (or
>> absence) of
>> the capability, it also makes it possible to programmatically determine
>> the
>> relevant configuration file name.
>>
>>
>> Recommended modifications to the Python standard library
>> --------------------------------------------------------
>>
>> The recommended approach to backporting the PEP 476 modifications to an
>> earlier
>> point release is to implement the following changes relative to the
>> default
>> PEP 476 behaviour implemented in Python 2.7.9+:
>>
>> * modify the ``ssl`` module to read a system wide configuration file when
>> the
>>   module is first imported into a Python process
>> * define a platform default behaviour (either verifying or not verifying
>> HTTPS
>>   certificates) to be used if this configuration file is not present
>> * support selection between the following three modes of operation:
>>
>>   * ensure HTTPS certificate verification is enabled
>>   * ensure HTTPS certificate verification is disabled
>>   * delegate the decision to the redistributor providing this Python
>> version
>>
>> * set the ``ssl._create_default_https_context`` function to be an alias
>> for
>>   either ``ssl.create_default_context`` or
>> ``ssl._create_unverified_context``
>>   based on the given configuration setting.
>>
>>
>> Recommended file location
>> -------------------------
>>
>> This approach is currently only defined for \*nix system Python
>> installations.
>>
>> The recommended configuration file name is
>> ``/etc/python/cert-verification.cfg``.
>>
>> The ``.cfg`` filename extension is recommended for consistency with the
>> ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library.
>>
>>
>> Recommended file format
>> -----------------------
>>
>> The configuration file should use a ConfigParser ini-style format with a
>> single section named ``[https]`` containing one required setting
>> ``verify``.
>>
>> Permitted values for ``verify`` are:
>>
>> * ``enable``: ensure HTTPS certificate verification is enabled by default
>> * ``disable``: ensure HTTPS certificate verification is disabled by
>> default
>> * ``platform_default``: delegate the decision to the redistributor
>> providing
>>   this particular Python version
>>
>> If the ``[https]`` section or the ``verify`` setting are missing, or if
>> the
>> ``verify`` setting is set to an unknown value, it should be treated as if
>> the
>> configuration file is not present.
>>
>>
>> Example implementation
>> ----------------------
>>
>> ::
>>
>>     _cert_verification_config = '/etc/python/cert-verification.cfg'
>>
>>     def _get_https_context_factory():
>>         # Check for a system-wide override of the default behaviour
>>         context_factories = {
>>             'enable': create_default_context,
>>             'disable': _create_unverified_context,
>>             'platform_default': _create_unverified_context, # For now :)
>>         }
>>         import ConfigParser
>>         config = ConfigParser.RawConfigParser()
>>         config.read(_cert_verification_config)
>>         try:
>>             verify_mode = config.get('https', 'verify')
>>         except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
>>             verify_mode = 'platform_default'
>>         default_factory = context_factories.get('platform_default')
>>         return context_factories.get(verify_mode, default_factory)
>>
>>     _create_default_https_context = _get_https_context_factory()
>>
>>
>> Security Considerations
>> -----------------------
>>
>> The specific recommendations for the backporting case are designed to
>> work for
>> privileged, security sensitive processes, even those being run in the
>> following
>> locked down configuration:
>>
>> * run from a locked down administrator controlled directory rather than a
>> normal
>>   user directory (preventing ``sys.path[0]`` based privilege escalation
>> attacks)
>> * run using the ``-E`` switch (preventing ``PYTHON*`` environment
>> variable based
>>   privilege escalation attacks)
>> * run using the ``-s`` switch (preventing user site directory based
>> privilege
>>   escalation attacks)
>> * run using the ``-S`` switch (preventing ``sitecustomize`` based
>> privilege
>>   escalation attacks)
>>
>> The intent is that the *only* reason HTTPS verification should be getting
>> turned off system wide when using this approach is because:
>>
>> * an end user is running a redistributor provided version of CPython
>> rather
>>   than running upstream CPython directly
>> * that redistributor has decided to provide a smoother migration path to
>>   verifying HTTPS certificates by default than that being provided by the
>>   upstream project
>> * either the redistributor or the local infrastructure administrator has
>>   determined that it is appropriate to override the default upstream
>> behaviour
>>   (at least for the time being)
>>
>> Using an administrator controlled configuration file rather than an
>> environment
>> variable has the essential feature of providing a smoother migration
>> path, even
>> for applications being run with the ``-E`` switch.
>>
>>
>> Combining the recommendations
>> =============================
>>
>> If a redistributor chooses to implement both recommendations, then the
>> environment variable should take precedence over the system-wide
>> configuration
>> setting. This allows the setting to be changed for a given user, virtual
>> environment or application, regardless of the system-wide default
>> behaviour.
>>
>> In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is
>> defined, and
>> set to anything *other* than ``'0'``, then HTTPS certificate verification
>> should be enabled.
>>
>> Example implementation
>> ----------------------
>>
>> ::
>>
>>     _https_verify_envvar = 'PYTHONHTTPSVERIFY'
>>     _cert_verification_config = '/etc/python/cert-verification.cfg'
>>
>>     def _get_https_context_factory():
>>         # Check for am environmental override of the default behaviour
>>         config_setting = os.environ.get(_https_verify_envvar)
>>         if config_setting is not None:
>>             if config_setting == '0':
>>                 return _create_unverified_context
>>             return create_default_context
>>
>>         # Check for a system-wide override of the default behaviour
>>         context_factories = {
>>             'enable': create_default_context,
>>             'disable': _create_unverified_context,
>>             'platform_default': _create_unverified_context, # For now :)
>>         }
>>         import ConfigParser
>>         config = ConfigParser.RawConfigParser()
>>         config.read(_cert_verification_config)
>>         try:
>>             verify_mode = config.get('https', 'verify')
>>         except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
>>             verify_mode = 'platform_default'
>>         default_factory = context_factories.get('platform_default')
>>         return context_factories.get(verify_mode, default_factory)
>>
>>     _create_default_https_context = _get_https_context_factory()
>>
>>
>> Copyright
>> =========
>>
>> This document has been placed into the public domain.
>>
>>
>> --
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151123/d9a50539/attachment.html>

From ncoghlan at gmail.com  Mon Nov 23 18:47:04 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 24 Nov 2015 09:47:04 +1000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <56536A6A.9030902@python.org>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <56536A6A.9030902@python.org>
Message-ID: <CADiSq7ftft6hny35U0sfuq1cvdB=9arT4m56WMiVgy0+=rdJBQ@mail.gmail.com>

Updated version of the PEP posted: https://hg.python.org/peps/rev/8decac213ebf

On 24 November 2015 at 05:35, Christian Heimes <christian at python.org> wrote:
> 1) The example implementation of the function doesn't check the
> sys.flags.ignore_environment. Internally CPython has specialized getenv
> function that ignores env vars with PYTHON prefix when the flag is set.
> PYTHON* env vars aren't removed from os.environ. Modules have to check
> the flag.

I guarded the relevant sections in the examples with "if not
sys.flags.ignore_environment:"

> 2) The PEP is rather Linux-centric. What's the recommended path to the
> config file on other platforms like BDS (/usr/local/etc/ is preferred
> for additional dependencies on FreeBSD), OSX and Windows?

The environment variable section should be generic, and the
configuration file section is largely specific to vendors offering
long term commercial support options for Linux distros.

There was already a qualification in the "Recommended file location"
section limiting the scope to *nix systems, so rather than trying to
cover non-Linux systems, I've instead updated that qualification to
limit the recommendation even further to specifically Linux systems.
If *BSD folks pipe up saying they'd like to be included, then adding
another filename recommendation wouldn't be difficult.

> 3) What's the interaction between the location of the config file and
> virtual envs? Would it make sense to search for the file in a venv's
> etc/ first and then dispatch to global /etc/? That way venvs can
> influence the setting, too.

This is a system level configuration setting to preserve backwards
compatible behaviour in the system Python, so scoping it at the
interpreter level was a deliberate design decision. However, I added a
new section to both recommendations describing the lack of interaction
with virtual environments.

> 4) It makes sense to make the cert-verification.cfg file future-proof
> and reserve it for other cert-related configuration in the future. For
> example it could be used to define new contexts, set protocols, ciphers
> or hashes for cert pinning. It should be enough to say that CPython
> reserves the right to add more sections and keys later.

For future releases, I think a different filename makes sense, as we
don't really want a global "turn off HTTPS certificate verification"
flag. Perhaps something like "/etc/python/network-security.cfg", for
example.

> 5) I'm not particular fond of the section name [https]. For one It is
> ambiguous because it doesn't distinguish between server certs and client
> certs.

I added a note to clarify that the section name comes from the "https"
URL schema passed to the relevant client APIs.

> It's also not correct. The default context is used for other
> protocols like imap, smtp etc. over TLS.

That changed in PEP 476 - there's a separate private context for HTTPS
standard library clients now, which allows HTTPS to verify hostnames
by default, while other clients are still permissive. This is noted in
PEP 476: https://www.python.org/dev/peps/pep-0476/#other-protocols

The reason for this is that browsers have provided a pretty good
forcing function in getting folks to use properly signed certificates,
at least on the public internet, but self-signed and improperly signed
certificates are still significantly more common for other protocols.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Mon Nov 23 18:56:16 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 24 Nov 2015 09:56:16 +1000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CACfEFw_67sGz=GBPvET1uJzgciXkxWitoAP7YceAcZbSsEnryA@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CACfEFw_67sGz=GBPvET1uJzgciXkxWitoAP7YceAcZbSsEnryA@mail.gmail.com>
Message-ID: <CADiSq7dMG5HEc2gUbv0pAU-5pVYWGiaR67A0xTyqdXpR7SWniA@mail.gmail.com>

On 24 November 2015 at 06:47, Wes Turner <wes.turner at gmail.com> wrote:
> 1. Does this affect easy_install?

easy_install has validated certificates since distribute was merged
back into the project as part of setuptools 0.7 [1], and aside from
one issue with HTTPS tunnelling [2], the certificate verification code
has been stable since setuptools 1.3 [3].

> 2. If/because this affects easy_install,
>   should the guidance / suggested package installation tool be [pip];
> because pip install_requires backports.ssl_match_hostname

setuptools/easy_install uses backports.ssl_match_hostname if it's
available, and otherwise has its own implementation.

Cheers,
Nick.

[1] https://pythonhosted.org/setuptools/history.html#id159
[2] https://pythonhosted.org/setuptools/history.html#id80
[3] https://pythonhosted.org/setuptools/history.html#id123

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Mon Nov 23 19:18:50 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 24 Nov 2015 10:18:50 +1000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <56536A6A.9030902@python.org>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <56536A6A.9030902@python.org>
Message-ID: <CADiSq7cifEZSYNHGZH4bFTpNqXEJnAk_xro83Xq-5yBpOoFktw@mail.gmail.com>

On 24 November 2015 at 05:35, Christian Heimes <christian at python.org> wrote:
> On 2015-11-17 01:00, Guido van Rossum wrote:
>> Hm, making Christian the BDFL-delegate would mean two out of three
>> authors *and* the BDFL-delegate all working for Red Hat, which clearly
>> has a stake (and IIUC has already committed to this approach ahead of
>> PEP approval). SO then it would look like this is just rubber-stamping
>> Red Hat's internal decision process (if it's a process -- sounds more
>> like an accident :-).
>>
>> So, Alex, do you want to approve this PEP?
>
> I haven't read this thread until now. Independently from your objection
> I have raised the same concern with Nick today. I'd be willing to BDFL
> the PEP but I'd rather have somebody outside of Red Hat.

Likewise, but the intersection between "wants to get PEP 476 into the
hands of as many system operators as possible as soon as possible",
"is a CPython core developer", and "doesn't work for Red Hat" is
looking to be a rather select group :)

Since we already know Red Hat are OK with the draft recommendations,
and I missed the RHEL 7.2 release date anyway, perhaps Barry or
Matthias might be interested in tilting at the Ubuntu 14.04 LTS stable
release update windmill? I know there was previously a decision from
Ubuntu Security not to backport PEPs 466 & 476 to 2.7.5 due to the
stability risks [1], but the configuration file based approach
recommended in PEP 493 is backwards compatible by default, with the
decision to opt in to the improved settings after upgrading current
systems being made by system administrators rather than the distro
vendor. With around 3 1/2 years still to run on 14.04's support
lifecycle, that has the potential to reach quite a few systems that
otherwise wouldn't benefit from the change until well after Ubuntu
16.04 is released next year.

Regards,
Nick.

[1] http://people.canonical.com/~ubuntu-security/cve/2014/CVE-2014-9365.html

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From wes.turner at gmail.com  Mon Nov 23 20:17:41 2015
From: wes.turner at gmail.com (Wes Turner)
Date: Mon, 23 Nov 2015 19:17:41 -0600
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CADiSq7dMG5HEc2gUbv0pAU-5pVYWGiaR67A0xTyqdXpR7SWniA@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CACfEFw_67sGz=GBPvET1uJzgciXkxWitoAP7YceAcZbSsEnryA@mail.gmail.com>
 <CADiSq7dMG5HEc2gUbv0pAU-5pVYWGiaR67A0xTyqdXpR7SWniA@mail.gmail.com>
Message-ID: <CACfEFw-My0xAaSvE5pMd6Ld6QEckztTvEiiLwngZveSUc1jG9g@mail.gmail.com>

On Mon, Nov 23, 2015 at 5:56 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 24 November 2015 at 06:47, Wes Turner <wes.turner at gmail.com> wrote:
> > 1. Does this affect easy_install?
>
> easy_install has validated certificates since distribute was merged
> back into the project as part of setuptools 0.7 [1], and aside from
> one issue with HTTPS tunnelling [2], the certificate verification code
> has been stable since setuptools 1.3 [3].
>

Got it; thanks.

* https://bitbucket.org/pypa/setuptools/src/tip/setuptools/ssl_support.py


>
> > 2. If/because this affects easy_install,
> >   should the guidance / suggested package installation tool be [pip];
> > because pip install_requires backports.ssl_match_hostname
>
> setuptools/easy_install uses backports.ssl_match_hostname if it's
> available, and otherwise has its own implementation.
>

* setuptools 0.7 [src] -- 2013-06-?? --
https://bitbucket.org/pypa/setuptools/src/7f2c08e9/?at=0.7
* setuptools 0.7 [distro] -- 201X-XX-XX [see: whohas, ]

Setuptools latest is now at version 18.5 (2015-11-02).






>
> Cheers,
> Nick.
>
> [1] https://pythonhosted.org/setuptools/history.html#id159
> [2] https://pythonhosted.org/setuptools/history.html#id80
> [3] https://pythonhosted.org/setuptools/history.html#id123
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151123/bb091c4b/attachment.html>

From barry at python.org  Mon Nov 23 20:59:06 2015
From: barry at python.org (Barry Warsaw)
Date: Mon, 23 Nov 2015 20:59:06 -0500
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CADiSq7cifEZSYNHGZH4bFTpNqXEJnAk_xro83Xq-5yBpOoFktw@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <56536A6A.9030902@python.org>
 <CADiSq7cifEZSYNHGZH4bFTpNqXEJnAk_xro83Xq-5yBpOoFktw@mail.gmail.com>
Message-ID: <20151123205906.02cd57ec@limelight.wooz.org>

On Nov 24, 2015, at 10:18 AM, Nick Coghlan wrote:

>Since we already know Red Hat are OK with the draft recommendations,
>and I missed the RHEL 7.2 release date anyway, perhaps Barry or
>Matthias might be interested in tilting at the Ubuntu 14.04 LTS stable
>release update windmill? I know there was previously a decision from
>Ubuntu Security not to backport PEPs 466 & 476 to 2.7.5 due to the
>stability risks [1], but the configuration file based approach
>recommended in PEP 493 is backwards compatible by default

Right, but this isn't a patch we'd particularly want to carry ourselves.
Maybe if it were available upstream, tried and tested, it could be considered
for backporting, but it still wouldn't be zero cost.  We'd have to also handle
migration paths to newer Ubuntu releases, which probably means removing the
config file on future upgrades.  There's also the possibility of implementing
different defaults on new installs of 14.04 versus upgrades to 14.04.  And
even if a system administrator enabled it for one particular application, it
could break other applications on the same machine, so it just punts a
difficult decision down the line.

We're also not seeing much (any?) demand from our users, and the initial
attempt at turning this on by default *did* get a strong negative reaction
because of the compatibility break.

I'm concerned about accepting PEP 493 making a strong recommendation to
downstreams.  Yes, in an ideal world we all want security by default, but I
think the backward compatibility concerns of the PEP are understated,
especially as they relate to a maintenance release of a stable long term
support version of the OS.  I don't want PEP 493 to be a cudgel that people
beat us up with instead of having an honest discussion of the difficult
trade-offs involved.

Having said all that, I think informing people of the issue, and letting any
future reconsideration be demand driven is the right approach for now.

$0.02-ly y'rs,
-Barry

From barry at python.org  Mon Nov 23 21:05:57 2015
From: barry at python.org (Barry Warsaw)
Date: Mon, 23 Nov 2015 21:05:57 -0500
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
Message-ID: <20151123210557.57b7e732@limelight.wooz.org>

On Nov 17, 2015, at 11:44 PM, Nick Coghlan wrote:

>For Debian, Ubuntu and SUSE, their original determinations for the
>relevant CVE were "too intrusive to backport", so folks currently need
>to upgrade to newer versions of those distros to get the improved
>default behaviour:

This is an example of my problem with the tone of PEP 493 (sorry Nick, nothing
personal!).  "Improved default behavior"... for whom?  It's not improved for
the folks whose applications are broken by changing the default.

Cheers,
-Barry

From ncoghlan at gmail.com  Mon Nov 23 22:46:55 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 24 Nov 2015 13:46:55 +1000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <20151123210557.57b7e732@limelight.wooz.org>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
Message-ID: <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>

On 24 November 2015 at 12:05, Barry Warsaw <barry at python.org> wrote:
> On Nov 17, 2015, at 11:44 PM, Nick Coghlan wrote:
>
>>For Debian, Ubuntu and SUSE, their original determinations for the
>>relevant CVE were "too intrusive to backport", so folks currently need
>>to upgrade to newer versions of those distros to get the improved
>>default behaviour:
>
> This is an example of my problem with the tone of PEP 493 (sorry Nick, nothing
> personal!).  "Improved default behavior"... for whom?  It's not improved for
> the folks whose applications are broken by changing the default.

I think there are three relevant categories here:

1. Folks who assume that "https" means the same thing in Python that
it means in web browsers, and are currently experiencing a silent
security failure
2. Folks who already know it doesn't, and are relying on that to keep
their infrastructure working
3. Folks currently in group 2 who would like to improve their
infrastructure to default to verifying certificates

The upstream change in PEP 476 was driven by the interests of folks in
group 1. This makes a lot of sense, given that a popular way of
introducing folks to programming now is by writing programs that
interact with web APIs over HTTPS. Rebasing is fine for reaching
desktop audiences, so this group should already have been covered by
the updated python.org binaries for Windows and Mac OS X, new binary
releases from the cross-platform redistributors, and updates to
non-LTS Linux distros.

For folks in group 2, the main goal is "Don't break their stuff as a
side effect of updating a supported version", which is why PEP 493
recommends leaving certificate verification disabled by default if
backporting the PEP 476 changes.

The target audience for PEP 493 is then the folks in group 3:
infrastructure operators that *want* to turn failures to validate
certificates in Python applications into a noisy failure, but *don't*
want to have to switch to a new major version of their entire base
operating system to do it. With the PEP, opting in to PEP 476 for
Python 2 applications and services is just a configuration file
setting, readily managed with any configuration management tool. If
just the configuration file approach is supported (as is the case for
RHEL 7.2), then that decision needs to be made on a system-wide basis,
and you need to use a chroot (or a full Linux container) in order to
opt out (or in) for a single application. If the environment variable
downgrade is also supported, then the environments for individual
applications can be tweaked, even while the overall environment is
upgraded to complain about missing security checks by default.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Mon Nov 23 23:12:56 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 24 Nov 2015 14:12:56 +1000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <20151123205906.02cd57ec@limelight.wooz.org>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <56536A6A.9030902@python.org>
 <CADiSq7cifEZSYNHGZH4bFTpNqXEJnAk_xro83Xq-5yBpOoFktw@mail.gmail.com>
 <20151123205906.02cd57ec@limelight.wooz.org>
Message-ID: <CADiSq7eewfMuWg7RM3KFkJfLCk5NOEfRtcwTTrA2W=9ofp2jfg@mail.gmail.com>

On 24 November 2015 at 11:59, Barry Warsaw <barry at python.org> wrote:
> On Nov 24, 2015, at 10:18 AM, Nick Coghlan wrote:
>
>>Since we already know Red Hat are OK with the draft recommendations,
>>and I missed the RHEL 7.2 release date anyway, perhaps Barry or
>>Matthias might be interested in tilting at the Ubuntu 14.04 LTS stable
>>release update windmill? I know there was previously a decision from
>>Ubuntu Security not to backport PEPs 466 & 476 to 2.7.5 due to the
>>stability risks [1], but the configuration file based approach
>>recommended in PEP 493 is backwards compatible by default
>
> Right, but this isn't a patch we'd particularly want to carry ourselves.
> Maybe if it were available upstream, tried and tested, it could be considered
> for backporting, but it still wouldn't be zero cost.

If anyone wants to look through the patches implementing the backport,
they're available through the CentOS git repositories:
https://git.centos.org/commit/rpms!python.git/refs!heads!c7

> We'd have to also handle
> migration paths to newer Ubuntu releases, which probably means removing the
> config file on future upgrades.

You'll need to figure out those migration paths anyway, as the
certificate verification compatibility break will exist between 14.04
and 16.04 regardless of what happens with PEP 493.

> There's also the possibility of implementing
> different defaults on new installs of 14.04 versus upgrades to 14.04.

For those aspects, we left the defaults in RHEL 7.2 as implementing
the 2.7.5 behaviour (i.e. no cert verification), but used the
"platform default" mechanism in PEP 493 to ensure we had the ability
to change our mind if circumstances change.

For the immediately foreseeable future though, it will be up to
operations teams to decide how they want the Python 2.7 standard
library to behave with respect to HTTPS client certificate
verification on RHEL & CentOS 7.

> And
> even if a system administrator enabled it for one particular application, it
> could break other applications on the same machine, so it just punts a
> difficult decision down the line.
>
> We're also not seeing much (any?) demand from our users, and the initial
> attempt at turning this on by default *did* get a strong negative reaction
> because of the compatibility break.

My own main concern is providing an additional layer of defence
against certificate verification related security bugs in network
services and clients written in Python like OpenStack, Ansible,
FreeIPA, Pulp, etc, which is why I'm happy with the approach of giving
system operators the tools they need to make their own decision about
whether or not they want that additional layer of protection on a
given host. Folks building Docker images can also easily opt in or out
by including a suitable configuration file.

>From a change management perspective, there's also a very big
difference between "We made this decision for you, and you'll need to
update your systems accordingly in order to upgrade", and "We made it
possible for you to make your own decision about this new
configuration setting, based on what you think is best for your
particular situation".

There's still some risk in the latter case (since you may accidentally
break things), but the upstream releases flushed out a lot of those
problems for this particular update (e.g. the gevent compatibility
issue that affected the upstream PEP 466 backport).

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From raymond.hettinger at gmail.com  Tue Nov 24 01:01:10 2015
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 23 Nov 2015 22:01:10 -0800
Subject: [Python-Dev] collections.Counter __add__ implementation quirk
In-Reply-To: <CAHzaPEMveK8FeHAxExCAnbbhDahodr67o+99PqN6ajw99c+xDA@mail.gmail.com>
References: <5652B077.9000606@sdamon.com>
 <CAHzaPEMveK8FeHAxExCAnbbhDahodr67o+99PqN6ajw99c+xDA@mail.gmail.com>
Message-ID: <B079CF34-DBE4-4B3E-A87A-63CE420405D5@gmail.com>


> On Nov 23, 2015, at 10:43 AM, Vlastimil Brom <vlastimil.brom at gmail.com> wrote:
> 
>> Is there any particular reason counters drop negative values when you add
>> them together?  I definitely expected them to act like ints do when you add
>> negatives, and had to subclass it to get what I think is the obvious
>> behavior.
>> _______________________________________________
>> Python-Dev mailing list
> ...
> Hi,
> this is probably more appropriate for the general python list rathere
> then this developers' maillist, however, as I asked a similar question
> some time ago, I got some detailed explanations for the the current
> design decissions from the original developer; cf.:
> https://mail.python.org/pipermail/python-list/2010-March/570618.html
> 
> (I didn't check possible changes in Counter since that version (3.1 at
> that time).)

In Python3.2, Counter grew a subtract() method:

>>> c = Counter(a=4, b=2, c=0, d=-2)
>>> d = Counter(a=1, b=2, c=3, d=4)
>>> c.subtract(d)
>>> c
Counter({'a': 3, 'b': 0, 'c': -3, 'd': -6})

The update() method has been around since the beginning:

>>> from collections import Counter
>>> c = Counter(a=4, b=2, c=0, d=-2)
>>> d = Counter(a=1, b=-5, c=-2, d=6)
>>> c.update(d)
>>> d
Counter({'d': 6, 'a': 1, 'c': -2, 'b': -5})


So, you have two ways of doing counter math:

1. Normal integer arithmetic using update() and subtract() does straight addition and subtraction, either starting with or ending-up with negative values.

2. Saturating arithmetic using the operators: + - & | excludes non-positive results.  This supports bag-like behavior (c.f. smalltalk) and multiset operations (https://en.wikipedia.org/wiki/Multiset).


Raymond







From p.f.moore at gmail.com  Tue Nov 24 05:12:10 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 24 Nov 2015 10:12:10 +0000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
Message-ID: <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>

On 24 November 2015 at 03:46, Nick Coghlan <ncoghlan at gmail.com> wrote:
> I think there are three relevant categories here:
>
> 1. Folks who assume that "https" means the same thing in Python that
> it means in web browsers, and are currently experiencing a silent
> security failure
> 2. Folks who already know it doesn't, and are relying on that to keep
> their infrastructure working
> 3. Folks currently in group 2 who would like to improve their
> infrastructure to default to verifying certificates

I'm not directly interested in this PEP (any code I write that is
relevant will run on Windows) but there's another group:

4. People who use https because it's the published API, but are only
looking at internal systems (where the threat window is minimised) and
the internal infrastructure is out of their control but insecure.
Browser users and other apps have to go through pain to work
(accepting self-signed certs, repeatedly responding "Yes" to security
warnings). Python code written for internal use just works at the
moment. Maybe it "shouldn't", in some abstract sense, but it *does*
and that is important to users.

This group may know that they are taking risks, and may want to
improve the infrastructure, but they *can't*. And this change breaks
their apps (possibly in ways they can't fix easily - not all client
environments offer ways to add trust, there's a reason tools like wget
have "ignore certificate checks" flags even though everyone knows they
are a bad idea).

The push for "secure by default" is important and needed, but needs to
be handled in a more pragmatic way, recognising that Python is used in
situations like the above, where people know they aren't doing the
right thing, but have no better options. Shutting down their least bad
option on a point of principle isn't helpful.

Paul

From christian at python.org  Tue Nov 24 07:53:48 2015
From: christian at python.org (Christian Heimes)
Date: Tue, 24 Nov 2015 13:53:48 +0100
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CADiSq7cifEZSYNHGZH4bFTpNqXEJnAk_xro83Xq-5yBpOoFktw@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <56536A6A.9030902@python.org>
 <CADiSq7cifEZSYNHGZH4bFTpNqXEJnAk_xro83Xq-5yBpOoFktw@mail.gmail.com>
Message-ID: <56545DDC.7000407@python.org>

On 2015-11-24 01:18, Nick Coghlan wrote:
> On 24 November 2015 at 05:35, Christian Heimes <christian at python.org> wrote:
>> On 2015-11-17 01:00, Guido van Rossum wrote:
>>> Hm, making Christian the BDFL-delegate would mean two out of three
>>> authors *and* the BDFL-delegate all working for Red Hat, which clearly
>>> has a stake (and IIUC has already committed to this approach ahead of
>>> PEP approval). SO then it would look like this is just rubber-stamping
>>> Red Hat's internal decision process (if it's a process -- sounds more
>>> like an accident :-).
>>>
>>> So, Alex, do you want to approve this PEP?
>>
>> I haven't read this thread until now. Independently from your objection
>> I have raised the same concern with Nick today. I'd be willing to BDFL
>> the PEP but I'd rather have somebody outside of Red Hat.
> 
> Likewise, but the intersection between "wants to get PEP 476 into the
> hands of as many system operators as possible as soon as possible",
> "is a CPython core developer", and "doesn't work for Red Hat" is
> looking to be a rather select group :)

Right, with Antoine and Alex out of scope and you, Victor and me working
for Red Hat, the air is getting thin. Benjamin is familiar with the ssl
module. Or we can follow Alex's advice and ask somebody from the PyCA
group (Donald, Paul, lvh) or requests (Cory) to get some outside
perspective.


> Since we already know Red Hat are OK with the draft recommendations,
> and I missed the RHEL 7.2 release date anyway, perhaps Barry or
> Matthias might be interested in tilting at the Ubuntu 14.04 LTS stable
> release update windmill? I know there was previously a decision from
> Ubuntu Security not to backport PEPs 466 & 476 to 2.7.5 due to the
> stability risks [1], but the configuration file based approach
> recommended in PEP 493 is backwards compatible by default, with the
> decision to opt in to the improved settings after upgrading current
> systems being made by system administrators rather than the distro
> vendor. With around 3 1/2 years still to run on 14.04's support
> lifecycle, that has the potential to reach quite a few systems that
> otherwise wouldn't benefit from the change until well after Ubuntu
> 16.04 is released next year.

Yes, that makes a lot of sense.

Christian

From donald at stufft.io  Tue Nov 24 07:59:37 2015
From: donald at stufft.io (Donald Stufft)
Date: Tue, 24 Nov 2015 07:59:37 -0500
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <56545DDC.7000407@python.org>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <56536A6A.9030902@python.org>
 <CADiSq7cifEZSYNHGZH4bFTpNqXEJnAk_xro83Xq-5yBpOoFktw@mail.gmail.com>
 <56545DDC.7000407@python.org>
Message-ID: <905691F9-40EB-4745-9DA0-BDA769FB9541@stufft.io>


> On Nov 24, 2015, at 7:53 AM, Christian Heimes <christian at python.org> wrote:
> 
> On 2015-11-24 01:18, Nick Coghlan wrote:
>> On 24 November 2015 at 05:35, Christian Heimes <christian at python.org> wrote:
>>> On 2015-11-17 01:00, Guido van Rossum wrote:
>>>> Hm, making Christian the BDFL-delegate would mean two out of three
>>>> authors *and* the BDFL-delegate all working for Red Hat, which clearly
>>>> has a stake (and IIUC has already committed to this approach ahead of
>>>> PEP approval). SO then it would look like this is just rubber-stamping
>>>> Red Hat's internal decision process (if it's a process -- sounds more
>>>> like an accident :-).
>>>> 
>>>> So, Alex, do you want to approve this PEP?
>>> 
>>> I haven't read this thread until now. Independently from your objection
>>> I have raised the same concern with Nick today. I'd be willing to BDFL
>>> the PEP but I'd rather have somebody outside of Red Hat.
>> 
>> Likewise, but the intersection between "wants to get PEP 476 into the
>> hands of as many system operators as possible as soon as possible",
>> "is a CPython core developer", and "doesn't work for Red Hat" is
>> looking to be a rather select group :)
> 
> Right, with Antoine and Alex out of scope and you, Victor and me working
> for Red Hat, the air is getting thin. Benjamin is familiar with the ssl
> module. Or we can follow Alex's advice and ask somebody from the PyCA
> group (Donald, Paul, lvh) or requests (Cory) to get some outside
> perspective.
> 

Under normal circumstances I'd probably be willing to do it even though I have
doubts about the value of it. However, I'm less than two weeks away from
closing on a house and will be moving into it after that, so my time is very
limited right now.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151124/db91e3b5/attachment.sig>

From ncoghlan at gmail.com  Tue Nov 24 08:20:16 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 24 Nov 2015 23:20:16 +1000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
Message-ID: <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>

On 24 Nov 2015 8:12 pm, "Paul Moore" <p.f.moore at gmail.com> wrote:
>
> On 24 November 2015 at 03:46, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > I think there are three relevant categories here:
> >
> > 1. Folks who assume that "https" means the same thing in Python that
> > it means in web browsers, and are currently experiencing a silent
> > security failure
> > 2. Folks who already know it doesn't, and are relying on that to keep
> > their infrastructure working
> > 3. Folks currently in group 2 who would like to improve their
> > infrastructure to default to verifying certificates
>
> I'm not directly interested in this PEP (any code I write that is
> relevant will run on Windows) but there's another group:
>
> 4. People who use https because it's the published API, but are only
> looking at internal systems (where the threat window is minimised) and
> the internal infrastructure is out of their control but insecure.
> Browser users and other apps have to go through pain to work
> (accepting self-signed certs, repeatedly responding "Yes" to security
> warnings). Python code written for internal use just works at the
> moment. Maybe it "shouldn't", in some abstract sense, but it *does*
> and that is important to users.
>
> This group may know that they are taking risks, and may want to
> improve the infrastructure, but they *can't*. And this change breaks
> their apps (possibly in ways they can't fix easily - not all client
> environments offer ways to add trust, there's a reason tools like wget
> have "ignore certificate checks" flags even though everyone knows they
> are a bad idea).

I believe you're referring mainly to the original PEP 476 change there. In
the context of PEP 493, this is another group that would potentially
benefit from the suggested "security downgrade" environment variable (if
any redistributors decide to implement that - RHEL doesn't as yet), since
it would provide a way to restore the old behaviour without changing their
client code or monkeypatching the SSL module as described in PEP 476.

Regards,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151124/044e5125/attachment.html>

From p.f.moore at gmail.com  Tue Nov 24 09:05:53 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 24 Nov 2015 14:05:53 +0000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
Message-ID: <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>

On 24 November 2015 at 13:20, Nick Coghlan <ncoghlan at gmail.com> wrote:
> I believe you're referring mainly to the original PEP 476 change there. In
> the context of PEP 493, this is another group that would potentially benefit
> from the suggested "security downgrade" environment variable (if any
> redistributors decide to implement that - RHEL doesn't as yet), since it
> would provide a way to restore the old behaviour without changing their
> client code or monkeypatching the SSL module as described in PEP 476.

I'm actually referring to the fact that your classification didn't
seem to include people who have no control over their infrastructure
(except in class 1 which implies ignorance rather than
powerlessness...). PEP 493 is of benefit to such people, so there's
now downside in explicitly noting this.

My concern is that *because* people consistently forget about the
class of people who have to put up with bad infrastructure but can't
do anything about it, we risk promoting a sense of "security as the
enemy" - which is the direct opposite of what we're trying to do.

I have no interest or opinion regarding this PEP itself, but I would
like to see "people who have to put up with whatever infrastructure
they are dumped with, and use Python to ease that burden" recognised
as an important class of user. They are very under-represented in
discussions, as it's usually big business closed source and similar
environments that are in that situation.

Simply adding "people who have no control over their broken
infrastructure" with a note that this PEP helps them, would be
sufficient here (and actually helps the case for the PEP, so why not?
;-))

Apologies, this is a bit of a hobby horse of mine, I'll pipe down now.

Paul.

From lac at openend.se  Tue Nov 24 09:27:15 2015
From: lac at openend.se (Laura Creighton)
Date: Tue, 24 Nov 2015 15:27:15 +0100
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
Message-ID: <201511241427.tAOERFJC028474@fido.openend.se>

In a message of Tue, 24 Nov 2015 14:05:53 +0000, Paul Moore writes:
>Simply adding "people who have no control over their broken
>infrastructure" with a note that this PEP helps them, would be
>sufficient here (and actually helps the case for the PEP, so why not?
>;-))

But does it help them?  Or does it increase the power of those who
hand out certificates and who are intensely security conscious over
those who would like to get some work done this afternoon?

Laura

From ncoghlan at gmail.com  Tue Nov 24 09:59:30 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 25 Nov 2015 00:59:30 +1000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <201511241427.tAOERFJC028474@fido.openend.se>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
 <201511241427.tAOERFJC028474@fido.openend.se>
Message-ID: <CADiSq7dvpY3zJT=ZO4CWfu=243FQLZOL87_bZmaOAM3n=WJQTg@mail.gmail.com>

On 25 November 2015 at 00:27, Laura Creighton <lac at openend.se> wrote:
> In a message of Tue, 24 Nov 2015 14:05:53 +0000, Paul Moore writes:
>>Simply adding "people who have no control over their broken
>>infrastructure" with a note that this PEP helps them, would be
>>sufficient here (and actually helps the case for the PEP, so why not?
>>;-))
>
> But does it help them?  Or does it increase the power of those who
> hand out certificates and who are intensely security conscious over
> those who would like to get some work done this afternoon?

In situations where IT are still the "Department of No", rather than
focusing on facilitating folks in getting their work done, I think the
most likely outcome of the configuration file recommendation in PEP
493 is preservation of the status quo: admins simply won't change the
config setting, even if they deploy a version of Linux that adopts the
approach suggested in the PEP. If they do enable full certificate
verification (or upgrade the environments they provide to a version of
Python that has it enabled by default) without doing appropriate
compatibility testing first, then they're going to hit the
compatibility problems Paul is talking about.

The aspect of the PEP that has the potential to help in the case of
poor infrastructure management is providing the ability to globally
turn off certificate verification on a per-process basis. It's the
networking equivalent of monkeypatching - you know there are risks
with doing it, but also judge the near term benefits to outweigh those
longer term risks in your particular situation.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From mal at egenix.com  Tue Nov 24 10:04:08 2015
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 24 Nov 2015 16:04:08 +0100
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <201511241427.tAOERFJC028474@fido.openend.se>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
 <201511241427.tAOERFJC028474@fido.openend.se>
Message-ID: <56547C68.8060506@egenix.com>

I think the PEP is a good step forward to compromise between
the crypto purists (use whatever technologies makes us more
secure even if it breaks things) and those who cannot upgrade
their Python 2.7 because of the PEP 476 change, since it causes their
applications to fail (e.g. because the embedded devices they want to
interface to only support self signed certs).

I would still find having built-in support for the recommendations
in the Python stdlib a better approach, but PEP 493 is good enough
in at least solving real problems people are having.

PS: Would be great to have a PyPI package which implements these
recommendations so that you can simply add it as dependency - and
then please for Python 3 as well, since people with embedded
devices will want to be able to use Python 3 as well ;-)


On 24.11.2015 15:27, Laura Creighton wrote:
> In a message of Tue, 24 Nov 2015 14:05:53 +0000, Paul Moore writes:
>> Simply adding "people who have no control over their broken
>> infrastructure" with a note that this PEP helps them, would be
>> sufficient here (and actually helps the case for the PEP, so why not?
>> ;-))
> 
> But does it help them?  Or does it increase the power of those who
> hand out certificates and who are intensely security conscious over
> those who would like to get some work done this afternoon?
> 
> Laura
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/mal%40egenix.com
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 24 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From a.badger at gmail.com  Tue Nov 24 10:19:22 2015
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Tue, 24 Nov 2015 07:19:22 -0800
Subject: [Python-Dev] Fwd: Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CABVPEKoEdU2eKXXBmw+im6MsbXLBB8ff9RCejAie+Hw39OyyDw@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
 <201511241427.tAOERFJC028474@fido.openend.se>
 <CABVPEKoEdU2eKXXBmw+im6MsbXLBB8ff9RCejAie+Hw39OyyDw@mail.gmail.com>
Message-ID: <CABVPEKpwtweLG5Gi=OedSc+=f111e0F7Hz9jYYSxqCGBUAWuyw@mail.gmail.com>

On Nov 24, 2015 6:28 AM, "Laura Creighton" <lac at openend.se> wrote:
>
> In a message of Tue, 24 Nov 2015 14:05:53 +0000, Paul Moore writes:
> >Simply adding "people who have no control over their broken
> >infrastructure" with a note that this PEP helps them, would be
> >sufficient here (and actually helps the case for the PEP, so why not?
> >;-))
>
> But does it help them?  Or does it increase the power of those who
> hand out certificates and who are intensely security conscious over
> those who would like to get some work done this afternoon?
>
My reading is that it will help more people but lockdown environments can
still trump their users if they wish.

If a distribution wishes to give users of older python versions the option
of verifying certificates then they will need to backport changes
authorized by previous peps.  By themselves, those changes would make it so
environment owners and application authors are in complete control.  If an
application is coded to do cert verification and the remote end has
certificates that aren't recognized as valid on the client end then the
user would have to change the client application code to be able to use it
in their environment (or figure out how to get the ca for the remote end
into their local certificate store... in extreme cases, this might be
impossible - the ca cert has been lost or belongs to another company).

This pep tells distributions how they might give the client end a bit more
power when they backport.  The settings file allows the client to toggle
verification site wide.  The environment variable allows clients to toggle
it per application invocation.  Both of these situations are better for a
client than having the backport and nothing else.  Both of these can be
shut down by an environment owner with sufficient authority to limit what's
running on the client (not sure the scope of the environment owner's powers
here so I thought I should acknowledge this factor).

So basically: backporting other peps (to increase security) will subtract
power from the clients.  This pep specifies several facilities the
backporters can implement to give some of that power back to the clients.

-Toshio

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151124/19230645/attachment.html>

From christian at python.org  Tue Nov 24 10:30:07 2015
From: christian at python.org (Christian Heimes)
Date: Tue, 24 Nov 2015 16:30:07 +0100
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CADiSq7ftft6hny35U0sfuq1cvdB=9arT4m56WMiVgy0+=rdJBQ@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <56536A6A.9030902@python.org>
 <CADiSq7ftft6hny35U0sfuq1cvdB=9arT4m56WMiVgy0+=rdJBQ@mail.gmail.com>
Message-ID: <5654827F.9010900@python.org>

On 2015-11-24 00:47, Nick Coghlan wrote:
> Updated version of the PEP posted: https://hg.python.org/peps/rev/8decac213ebf
> 
> On 24 November 2015 at 05:35, Christian Heimes <christian at python.org> wrote:
>> 1) The example implementation of the function doesn't check the
>> sys.flags.ignore_environment. Internally CPython has specialized getenv
>> function that ignores env vars with PYTHON prefix when the flag is set.
>> PYTHON* env vars aren't removed from os.environ. Modules have to check
>> the flag.
> 
> I guarded the relevant sections in the examples with "if not
> sys.flags.ignore_environment:"

Thanks!


>> 2) The PEP is rather Linux-centric. What's the recommended path to the
>> config file on other platforms like BDS (/usr/local/etc/ is preferred
>> for additional dependencies on FreeBSD), OSX and Windows?
> 
> The environment variable section should be generic, and the
> configuration file section is largely specific to vendors offering
> long term commercial support options for Linux distros.
> 
> There was already a qualification in the "Recommended file location"
> section limiting the scope to *nix systems, so rather than trying to
> cover non-Linux systems, I've instead updated that qualification to
> limit the recommendation even further to specifically Linux systems.
> If *BSD folks pipe up saying they'd like to be included, then adding
> another filename recommendation wouldn't be difficult.
>> 3) What's the interaction between the location of the config file and
>> virtual envs? Would it make sense to search for the file in a venv's
>> etc/ first and then dispatch to global /etc/? That way venvs can
>> influence the setting, too.
> 
> This is a system level configuration setting to preserve backwards
> compatible behaviour in the system Python, so scoping it at the
> interpreter level was a deliberate design decision. However, I added a
> new section to both recommendations describing the lack of interaction
> with virtual environments.

I mentioned 2) and 3) because I suspect that some people will want to
use the new setting to configure applications. You know how people
think. They get a new shiny tool and then they abuse it as a hammer to
drive in all their problems. :)


>> 4) It makes sense to make the cert-verification.cfg file future-proof
>> and reserve it for other cert-related configuration in the future. For
>> example it could be used to define new contexts, set protocols, ciphers
>> or hashes for cert pinning. It should be enough to say that CPython
>> reserves the right to add more sections and keys later.
> 
> For future releases, I think a different filename makes sense, as we
> don't really want a global "turn off HTTPS certificate verification"
> flag. Perhaps something like "/etc/python/network-security.cfg", for
> example.

Are you planning to remove the "disable verification flag" in the future?


>> 5) I'm not particular fond of the section name [https]. For one It is
>> ambiguous because it doesn't distinguish between server certs and client
>> certs.
> 
> I added a note to clarify that the section name comes from the "https"
> URL schema passed to the relevant client APIs.
> 
>> It's also not correct. The default context is used for other
>> protocols like imap, smtp etc. over TLS.
> 
> That changed in PEP 476 - there's a separate private context for HTTPS
> standard library clients now, which allows HTTPS to verify hostnames
> by default, while other clients are still permissive. This is noted in
> PEP 476: https://www.python.org/dev/peps/pep-0476/#other-protocols
> 
> The reason for this is that browsers have provided a pretty good
> forcing function in getting folks to use properly signed certificates,
> at least on the public internet, but self-signed and improperly signed
> certificates are still significantly more common for other protocols.

I'm sorry, I forgot about PEP 476 and the separate context function for
HTTPS. Somehow I expected the setting to influence all TLS/SSL
connection, not just HTTPS. I should have read the topic line.

In the light of the new information the setting name makes sense.

Christian


From p.f.moore at gmail.com  Tue Nov 24 10:34:53 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 24 Nov 2015 15:34:53 +0000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <201511241427.tAOERFJC028474@fido.openend.se>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
 <201511241427.tAOERFJC028474@fido.openend.se>
Message-ID: <CACac1F9QUvu3_c3v5EJ5vhhFBWU=LLPyShcwmFJ7U66_g9OSUA@mail.gmail.com>

On 24 November 2015 at 14:27, Laura Creighton <lac at openend.se> wrote:
> In a message of Tue, 24 Nov 2015 14:05:53 +0000, Paul Moore writes:
>>Simply adding "people who have no control over their broken
>>infrastructure" with a note that this PEP helps them, would be
>>sufficient here (and actually helps the case for the PEP, so why not?
>>;-))
>
> But does it help them?  Or does it increase the power of those who
> hand out certificates and who are intensely security conscious over
> those who would like to get some work done this afternoon?

My reading is that if fully implemented (and Nick has already
confirmed that Red Hat didn't do this) it would add an environment
variable that would allow the user to (in effect) say "I can't fix my
security infrastructure, so just leave me alone and let me take the
risk".

So in theory this PEP would give back some of the ability to ignore
the problem that previous PEPs took away. (And by "ignore the
problem", here I mean "just try to get some work done in spite of a
security and infrastructure group that don't understand how to
implement security and infrastructure, dammit!")

Like it or not, in many organisations, security and development are a
huge "us and them" battle. For me, it's important that Python doesn't
take sides in that battle, while still offering education to anyone
willing to listen. (All I've learned about security is as a result of
working with Python - sadly, that knowledge has not made my job one
iota easier, it's just increased my stress levels :-()

Paul

From mike at selik.org  Tue Nov 24 03:19:36 2015
From: mike at selik.org (Michael Selik)
Date: Tue, 24 Nov 2015 08:19:36 +0000
Subject: [Python-Dev] collections.Counter __add__ implementation quirk
In-Reply-To: <B079CF34-DBE4-4B3E-A87A-63CE420405D5@gmail.com>
References: <5652B077.9000606@sdamon.com>
 <CAHzaPEMveK8FeHAxExCAnbbhDahodr67o+99PqN6ajw99c+xDA@mail.gmail.com>
 <B079CF34-DBE4-4B3E-A87A-63CE420405D5@gmail.com>
Message-ID: <CADqi794B2gVd7c0OAQDMQJ7A_dZj9kD+EgY3JPzVbVFgruWWag@mail.gmail.com>

Raymond,
I think you made a typographical error in your Counter.update example.

    >>> from collections import Counter
    >>> c = Counter(a=4, b=2, c=0, d=-2)
    >>> d = Counter(a=1, b=-5, c=-2, d=6)
    >>> c.update(d)
    >>> c
    Counter({'a': 5, 'd': 4, 'c': -2, 'b': -3})

Pair programming ;-)


On Tue, Nov 24, 2015 at 1:02 AM Raymond Hettinger <
raymond.hettinger at gmail.com> wrote:

>
> > On Nov 23, 2015, at 10:43 AM, Vlastimil Brom <vlastimil.brom at gmail.com>
> wrote:
> >
> >> Is there any particular reason counters drop negative values when you
> add
> >> them together?  I definitely expected them to act like ints do when you
> add
> >> negatives, and had to subclass it to get what I think is the obvious
> >> behavior.
> >> _______________________________________________
> >> Python-Dev mailing list
> > ...
> > Hi,
> > this is probably more appropriate for the general python list rathere
> > then this developers' maillist, however, as I asked a similar question
> > some time ago, I got some detailed explanations for the the current
> > design decissions from the original developer; cf.:
> > https://mail.python.org/pipermail/python-list/2010-March/570618.html
> >
> > (I didn't check possible changes in Counter since that version (3.1 at
> > that time).)
>
> In Python3.2, Counter grew a subtract() method:
>
> >>> c = Counter(a=4, b=2, c=0, d=-2)
> >>> d = Counter(a=1, b=2, c=3, d=4)
> >>> c.subtract(d)
> >>> c
> Counter({'a': 3, 'b': 0, 'c': -3, 'd': -6})
>
> The update() method has been around since the beginning:
>
> >>> from collections import Counter
> >>> c = Counter(a=4, b=2, c=0, d=-2)
> >>> d = Counter(a=1, b=-5, c=-2, d=6)
> >>> c.update(d)
> >>> d
> Counter({'d': 6, 'a': 1, 'c': -2, 'b': -5})
>
>
> So, you have two ways of doing counter math:
>
> 1. Normal integer arithmetic using update() and subtract() does straight
> addition and subtraction, either starting with or ending-up with negative
> values.
>
> 2. Saturating arithmetic using the operators: + - & | excludes
> non-positive results.  This supports bag-like behavior (c.f. smalltalk) and
> multiset operations (https://en.wikipedia.org/wiki/Multiset).
>
>
> Raymond
>
>
>
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mike%40selik.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151124/e199e1cb/attachment.html>

From a.badger at gmail.com  Tue Nov 24 12:16:30 2015
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Tue, 24 Nov 2015 09:16:30 -0800
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <20151123205906.02cd57ec@limelight.wooz.org>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <56536A6A.9030902@python.org>
 <CADiSq7cifEZSYNHGZH4bFTpNqXEJnAk_xro83Xq-5yBpOoFktw@mail.gmail.com>
 <20151123205906.02cd57ec@limelight.wooz.org>
Message-ID: <CABVPEKq4Ea9aP-aNVF9XFoQXX9=UvXo7beHg9Lq8_JT43X-oxA@mail.gmail.com>

On Mon, Nov 23, 2015 at 5:59 PM, Barry Warsaw <barry at python.org> wrote:

> I'm concerned about accepting PEP 493 making a strong recommendation to
> downstreams.  Yes, in an ideal world we all want security by default, but I
> think the backward compatibility concerns of the PEP are understated,
> especially as they relate to a maintenance release of a stable long term
> support version of the OS.  I don't want PEP 493 to be a cudgel that people
> beat us up with instead of having an honest discussion of the difficult
> trade-offs involved.
>
It sounds like the implementation sections of the PEP are acceptable
but that the PEP's general tone seems to assume that distributors are
champing at the bit to backport and that the recommendations here make
it so that backporting is a no-brainer -- which does not seem to
reflect the real-world?

I think the tone could be changed to address that as it doesn't seem
like forcing distros to backport is a real goal of the PEP.  The main
purposes of the PEP seem to be:

* Enumerate several ways that distributors can backport these 2.7.9
features to older releases
* Allow programmers to detect the presence of the features from their code
* Give end-users the ability to choose between backwards compatibility
and enhanced security

Here's some ideas for changing the tone:

  Abstract
  ========

  PEP 476 updated Python's default handling of HTTPS certificates to be
  appropriate for communication over the public internet. The Python 2.7 long
  term maintenance series was judged to be in scope for this change, with the
  new behaviour introduced in the Python 2.7.9 maintenance release.

+ Change to "PEP 476 updated Python's default handling of HTTPS
certificates to validate that the certs belonged to the server".  This
way we're saying what the change is rather than making a value
judgement of whether people who don't choose to backport are
"appropriate" or not.  Appropriate-ness is probably best left as an
argument in the text of PEP 476.

  This PEP provides recommendations to downstream redistributors wishing to
  provide a smoother migration experience when helping their users to manage
  this change in Python's default behaviour.

+ Change to "downstream redistributors wishing to backport the
enhancements in a way that allows users to choose between backwards
compatible behaviour or more secure certificate handling."  As barry
noted, this PEP doesn't change the amount of work needed to migrate.
It does, however, give users some choice in when they are going to
perform that work.  Additionally, this isn't simply about distributors
who want to make the transition smoother... (there's no downstreams
that want to make it "more painful" are there? ;-)  It's really about
making backporting of the enhancements less painful for users.

  Rationale
  =========

  PEP 476 changed Python's default behaviour to better match the needs and
  expectations of developers operating over the public internet, a category
  which appears to include most new Python developers. It is the position of
  the authors of this PEP that this was a correct decision.

  However, it is also the case that this change *does* cause problems for
  infrastructure administrators operating private intranets that rely on
  self-signed certificates, or otherwise encounter problems with the new default
  certificate verification settings.

+ per barry's mesage, it would be good to either devote a paragraph to
the backwards compatibility implications here or link to
https://www.python.org/dev/peps/pep-0476/#backwards-compatibility

  The long term answer for such environments is to update their internal
  certificate management to at least match the standards set by the public
  internet, but in the meantime, it is desirable to offer these administrators
  a way to continue receiving maintenance updates to the Python 2.7 series,
  without having to gate that on upgrades to their certificate management
  infrastructure.

+ The wording here seems to reflect a different scope than merely
backporting by distros.  Perhaps we should change it to: "[...]set by
the public internet.  Distributions may wish to help these sites
transition by backporting the PEP 476 changes to earlier versions of
python in a way that does not require the administrators to upgrade
their certificate management infrastructure immediately.  This would
allow sites to choose to use the distribution suppiied python in a
backwards compatible fashion until their certificate management
infrastructure was updated and then toggle their site to utilize the
more secure features provided by PEP 476."

[...]

  These designs are being proposed as a recommendation for
redistributors, rather
  than as new upstream features, as they are needed purely to support legacy
  environments migrating from older versions of Python 2.7. Neither approach
  is being proposed as an upstream Python 2.7 feature, nor as a feature in any
  version of Python 3 (whether published directly by the Python Software
  Foundation or by a redistributor).

+ This paragraph seems a little out of place here as it is not about
why this PEP is useful so much as who will implement it.  It also
seems to tie into Abstract closely as it reinforces and expands on
where this is going to make a difference.  Perhaps move it to Abstract
or create a "Scope" section that can follow abstract and contain this
information.  wording can be changed a bit to show that it isn't
expected that all redistributors are expected to follow this but only
those who are already backporting: "These recommendations are being
proposed to redistributors who wish to backport PEP 476 features
rather than as a new upstream feature as they are needed purely to
support migrating legacy environments from older versions of python
2.7"


-Toshio

From p.f.moore at gmail.com  Tue Nov 24 13:08:58 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 24 Nov 2015 18:08:58 +0000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CABVPEKq4Ea9aP-aNVF9XFoQXX9=UvXo7beHg9Lq8_JT43X-oxA@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <56536A6A.9030902@python.org>
 <CADiSq7cifEZSYNHGZH4bFTpNqXEJnAk_xro83Xq-5yBpOoFktw@mail.gmail.com>
 <20151123205906.02cd57ec@limelight.wooz.org>
 <CABVPEKq4Ea9aP-aNVF9XFoQXX9=UvXo7beHg9Lq8_JT43X-oxA@mail.gmail.com>
Message-ID: <CACac1F_N5fZS0Vj_CMpWj1AR15fFaBVwyn4+A7QCFYYuL3WJDQ@mail.gmail.com>

On 24 November 2015 at 17:16, Toshio Kuratomi <a.badger at gmail.com> wrote:
> The long term answer for such environments is to update their internal
>   certificate management to at least match the standards set by the public
>   internet, but in the meantime, it is desirable to offer these administrators
>   a way to continue receiving maintenance updates to the Python 2.7 series,
>   without having to gate that on upgrades to their certificate management
>   infrastructure.
>
> + The wording here seems to reflect a different scope than merely
> backporting by distros.  Perhaps we should change it to: "[...]set by
> the public internet.  Distributions may wish to help these sites
> transition by backporting the PEP 476 changes to earlier versions of
> python in a way that does not require the administrators to upgrade
> their certificate management infrastructure immediately.  This would
> allow sites to choose to use the distribution suppiied python in a
> backwards compatible fashion until their certificate management
> infrastructure was updated and then toggle their site to utilize the
> more secure features provided by PEP 476."

I'm not actually sure that it's the place of this PEP to even comment
on what the long term answer for such environments should be (or
indeed, any answer, long term or not). I've argued the position that
in some organisations it feels like security don't actually understand
the issues of carefully balancing secure operation against flexible
development practices, but conversely it's certainly true that in many
organisations, they *have* weighed the various arguments and made an
informed decision on how to set up their internal network. It's
entirely possible that self-signed certificates are entirely the right
decision for their circumstances. Why would a Python PEP be qualified
to comment on that decision?

In my opinion, we should take all of the value judgements out of this
paragraph, and just state the facts. How about:

"""
In order to provide additional flexibility to allow infrastructure
administrators to provide the appropriate solution for their
environment, this PEP offers a way for administrators to upgrade to
later versions of the Python 2.7 series without being forced to update
their existing security certificate management infrastructure as a
prerequisite.
"""

Paul

From a.badger at gmail.com  Tue Nov 24 13:37:09 2015
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Tue, 24 Nov 2015 10:37:09 -0800
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CACac1F_N5fZS0Vj_CMpWj1AR15fFaBVwyn4+A7QCFYYuL3WJDQ@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <56536A6A.9030902@python.org>
 <CADiSq7cifEZSYNHGZH4bFTpNqXEJnAk_xro83Xq-5yBpOoFktw@mail.gmail.com>
 <20151123205906.02cd57ec@limelight.wooz.org>
 <CABVPEKq4Ea9aP-aNVF9XFoQXX9=UvXo7beHg9Lq8_JT43X-oxA@mail.gmail.com>
 <CACac1F_N5fZS0Vj_CMpWj1AR15fFaBVwyn4+A7QCFYYuL3WJDQ@mail.gmail.com>
Message-ID: <CABVPEKp7hPnyLx209Ypet4rsNFC5a39Zy7BFNrq2Y-riutp=Lw@mail.gmail.com>

On Tue, Nov 24, 2015 at 10:08 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> I'm not actually sure that it's the place of this PEP to even comment
> on what the long term answer for such environments should be (or
> indeed, any answer, long term or not). I've argued the position that
> in some organisations it feels like security don't actually understand
> the issues of carefully balancing secure operation against flexible
> development practices,

I agree with this.

> but conversely it's certainly true that in many
> organisations, they *have* weighed the various arguments and made an
> informed decision on how to set up their internal network. It's
> entirely possible that self-signed certificates are entirely the right
> decision for their circumstances. Why would a Python PEP be qualified
> to comment on that decision?

I don't quite agree with this but it's probably moot in the face of
the previous and certain cornercases.  Self-signed certs work just
fine with the backports to python-2.7.9+ but you have to add the ca to
the clients.  An organization that has weighed the arguments and made
an informed decision to use self-signed certs should either do this
(to prevent MITM) or they should switch to using http instead of https
(because MITM isn't a feasible attack here).  The cornercases come
into play because you don't always control all of the devices and
services on your network.  The site could evaluate and decide that
MITM isn't a threat to their switch's configuration interface but that
interface might be served over https using a certificate signed by
their network vendor who doesn't give out their ca certificate (simply
stated: your security team knows what they're doing but your vendor's
does not).

> In my opinion, we should take all of the value judgements out of this
> paragraph, and just state the facts. How about:
>
> """
> In order to provide additional flexibility to allow infrastructure
> administrators to provide the appropriate solution for their
> environment, this PEP offers a way for administrators to upgrade to
> later versions of the Python 2.7 series without being forced to update
> their existing security certificate management infrastructure as a
> prerequisite.
> """

Two notes:

* python-2.7.9+ doesn't give you flexibility in this regard so
organizations do have to update their certificate management
infrastructure.  The cornercase described above becomes something that
has to be addressed at the code level.  Environments that are simply
misconfigured have to be fixed.  So in that regard, a value judgement
does seem appropriate here.  the judgement is "Listen guys, this PEP
advises redistributors on how they might provide a migration path for
you but it *does not bandaid the problem indefinitely*.  So long term,
you have to change your practices or you'll be out in the cold when
your redistributor upgrades to python-2.7.9+"

* Your proposed text actually removes the fix that I was adding --
this version of the paragraph implies that if your environment is
compatible with the redistributors' python-2.7.8 (or less) then it
will also be compatible with the redistributors' python-2.7.9+.  That
is not true.  Whether or not we take out any value judgement as to the
user's present environment this paragraph needs to be fixed to make it
clear that this only affects redistributor's packages which have
backported pep 476 to python-2.7.8 or older.  Once the redistributor
updates to a newer python sites which relied on this crutch will
break.

-Toshio

From p.f.moore at gmail.com  Tue Nov 24 13:56:47 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 24 Nov 2015 18:56:47 +0000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CABVPEKp7hPnyLx209Ypet4rsNFC5a39Zy7BFNrq2Y-riutp=Lw@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <56536A6A.9030902@python.org>
 <CADiSq7cifEZSYNHGZH4bFTpNqXEJnAk_xro83Xq-5yBpOoFktw@mail.gmail.com>
 <20151123205906.02cd57ec@limelight.wooz.org>
 <CABVPEKq4Ea9aP-aNVF9XFoQXX9=UvXo7beHg9Lq8_JT43X-oxA@mail.gmail.com>
 <CACac1F_N5fZS0Vj_CMpWj1AR15fFaBVwyn4+A7QCFYYuL3WJDQ@mail.gmail.com>
 <CABVPEKp7hPnyLx209Ypet4rsNFC5a39Zy7BFNrq2Y-riutp=Lw@mail.gmail.com>
Message-ID: <CACac1F8JFYvFSenC1SMpVGRSMQjPOaK59fkqW4yYSgJu6-Au_w@mail.gmail.com>

On 24 November 2015 at 18:37, Toshio Kuratomi <a.badger at gmail.com> wrote:
> I don't quite agree with this but it's probably moot in the face of
> the previous and certain cornercases.  Self-signed certs work just
> fine with the backports to python-2.7.9+ but you have to add the ca to
> the clients.  An organization that has weighed the arguments and made
> an informed decision to use self-signed certs should either do this
> (to prevent MITM) or they should switch to using http instead of https
> (because MITM isn't a feasible attack here).

There's an assumption (which is untrue for the big organisations I'm
familiar with) that security are involved with the full stack -
typically in my experience "corporate security" set rules that are
filtered down through many layers, and development teams have no
influence on those decisions - conversely, the people setting security
policies aren't aware of the details of development day to day needs.

It's easy to say that users just click on the "allow" button in their
browser if they want to use self-signed certificates, but not have any
viable solution for automation code that doesn't use a browser - and
such automation code is "not corporately supported", so policy won't
get updated to address the disconnect.

> The cornercases come
> into play because you don't always control all of the devices and
> services on your network.  The site could evaluate and decide that
> MITM isn't a threat to their switch's configuration interface but that
> interface might be served over https using a certificate signed by
> their network vendor who doesn't give out their ca certificate (simply
> stated: your security team knows what they're doing but your vendor's
> does not).

This sounds like a similar situation to what I described above. I'm
not sure I'd see these as corner cases, though - they are pretty much
day to day business in my experience :-(

>> In my opinion, we should take all of the value judgements out of this
>> paragraph, and just state the facts. How about:
>>
>> """
>> In order to provide additional flexibility to allow infrastructure
>> administrators to provide the appropriate solution for their
>> environment, this PEP offers a way for administrators to upgrade to
>> later versions of the Python 2.7 series without being forced to update
>> their existing security certificate management infrastructure as a
>> prerequisite.
>> """
>
> Two notes:
>
> * python-2.7.9+ doesn't give you flexibility in this regard so
> organizations do have to update their certificate management
> infrastructure.  The cornercase described above becomes something that
> has to be addressed at the code level.  Environments that are simply
> misconfigured have to be fixed.  So in that regard, a value judgement
> does seem appropriate here.  the judgement is "Listen guys, this PEP
> advises redistributors on how they might provide a migration path for
> you but it *does not bandaid the problem indefinitely*.  So long term,
> you have to change your practices or you'll be out in the cold when
> your redistributor upgrades to python-2.7.9+"

Hmm, maybe I misread the PEP (I only skimmed it - as I say, Linux is
of limited interest to me). I thought that the environment variable
gave developers a "get out" clause. Maybe it's not what we want them
to do (for some value of "we") but isn't that the point of the PEP?

Admittedly if distributions don't *implement* that part of the PEP
(and I understand Red Hat haven't) then people are still stuck. But
"this PEP offers a way" is not incompatible with "your vendor didn't
implement the PEP so you're still stuck, sorry"...

> * Your proposed text actually removes the fix that I was adding --
> this version of the paragraph implies that if your environment is
> compatible with the redistributors' python-2.7.8 (or less) then it
> will also be compatible with the redistributors' python-2.7.9+.  That
> is not true.  Whether or not we take out any value judgement as to the
> user's present environment this paragraph needs to be fixed to make it
> clear that this only affects redistributor's packages which have
> backported pep 476 to python-2.7.8 or older.  Once the redistributor
> updates to a newer python sites which relied on this crutch will
> break.

Sorry for that. Certainly getting the facts right is crucial, and it
looks like my suggestion didn't do that. But hopefully someone can fix
it up (if people think it's a good way to go).

Paul

From a.badger at gmail.com  Tue Nov 24 14:40:50 2015
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Tue, 24 Nov 2015 11:40:50 -0800
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CACac1F8JFYvFSenC1SMpVGRSMQjPOaK59fkqW4yYSgJu6-Au_w@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <56536A6A.9030902@python.org>
 <CADiSq7cifEZSYNHGZH4bFTpNqXEJnAk_xro83Xq-5yBpOoFktw@mail.gmail.com>
 <20151123205906.02cd57ec@limelight.wooz.org>
 <CABVPEKq4Ea9aP-aNVF9XFoQXX9=UvXo7beHg9Lq8_JT43X-oxA@mail.gmail.com>
 <CACac1F_N5fZS0Vj_CMpWj1AR15fFaBVwyn4+A7QCFYYuL3WJDQ@mail.gmail.com>
 <CABVPEKp7hPnyLx209Ypet4rsNFC5a39Zy7BFNrq2Y-riutp=Lw@mail.gmail.com>
 <CACac1F8JFYvFSenC1SMpVGRSMQjPOaK59fkqW4yYSgJu6-Au_w@mail.gmail.com>
Message-ID: <CABVPEKqt5oi=5dKHnGJtOSc5zkORLwOdcv+BLxp14GmF1ubYgA@mail.gmail.com>

On Tue, Nov 24, 2015 at 10:56 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 24 November 2015 at 18:37, Toshio Kuratomi <a.badger at gmail.com> wrote:

>> The cornercases come
>> into play because you don't always control all of the devices and
>> services on your network.  The site could evaluate and decide that
>> MITM isn't a threat to their switch's configuration interface but that
>> interface might be served over https using a certificate signed by
>> their network vendor who doesn't give out their ca certificate (simply
>> stated: your security team knows what they're doing but your vendor's
>> does not).
>
> This sounds like a similar situation to what I described above. I'm
> not sure I'd see these as corner cases, though - they are pretty much
> day to day business in my experience :-(
>
It sounds like you're coming from a Windows background and I'm coming
from a Linux background which might be a small disconnect here -- we
do seem to be in agreement that what's "right to do" isn't always easy
or possible for the client to accomplish so I think we should probably
leave it at that.

>>> In my opinion, we should take all of the value judgements out of this
>>> paragraph, and just state the facts. How about:
>>>
>>> """
>>> In order to provide additional flexibility to allow infrastructure
>>> administrators to provide the appropriate solution for their
>>> environment, this PEP offers a way for administrators to upgrade to
>>> later versions of the Python 2.7 series without being forced to update
>>> their existing security certificate management infrastructure as a
>>> prerequisite.
>>> """
>>
>> Two notes:
>>
>> * python-2.7.9+ doesn't give you flexibility in this regard so
>> organizations do have to update their certificate management
>> infrastructure.  The cornercase described above becomes something that
>> has to be addressed at the code level.  Environments that are simply
>> misconfigured have to be fixed.  So in that regard, a value judgement
>> does seem appropriate here.  the judgement is "Listen guys, this PEP
>> advises redistributors on how they might provide a migration path for
>> you but it *does not bandaid the problem indefinitely*.  So long term,
>> you have to change your practices or you'll be out in the cold when
>> your redistributor upgrades to python-2.7.9+"
>
> Hmm, maybe I misread the PEP (I only skimmed it - as I say, Linux is
> of limited interest to me). I thought that the environment variable
> gave developers a "get out" clause. Maybe it's not what we want them
> to do (for some value of "we") but isn't that the point of the PEP?
>
> Admittedly if distributions don't *implement* that part of the PEP
> (and I understand Red Hat haven't) then people are still stuck. But
> "this PEP offers a way" is not incompatible with "your vendor didn't
> implement the PEP so you're still stuck, sorry"...
>

Yeah, I think you are correct in your understanding of what actual
changes to the python distrribution are being proposed for
redistributors in the PEP.  Reading through the PEP again, I'm not
sure if I'm correct in thinking that this only applies to
backporting... it may be that the environment section of the PEP
applies to any python-2 while the config file section only applies to
backporting.  Nick, could you clarify?

The PEP is clear that it doesn't apply to python-3 or cross-distro.
So that means that sites still can't rely on this long-term (but their
long term would extend to the lifetime of their vendor supporting
python2 rather than when their vendor updated to 2.7.9+) and also that
developers can't depend on this if they're developing portable code.

>> * Your proposed text actually removes the fix that I was adding --
>> this version of the paragraph implies that if your environment is
>> compatible with the redistributors' python-2.7.8 (or less) then it
>> will also be compatible with the redistributors' python-2.7.9+.  That
>> is not true.  Whether or not we take out any value judgement as to the
>> user's present environment this paragraph needs to be fixed to make it
>> clear that this only affects redistributor's packages which have
>> backported pep 476 to python-2.7.8 or older.  Once the redistributor
>> updates to a newer python sites which relied on this crutch will
>> break.
>
> Sorry for that. Certainly getting the facts right is crucial, and it
> looks like my suggestion didn't do that. But hopefully someone can fix
> it up (if people think it's a good way to go).

Could be that I'm wrong -- will wait for Nick to clarify before I
think about what could be done to make this wording better.

-Toshio

From brett at snarky.ca  Tue Nov 24 14:31:48 2015
From: brett at snarky.ca (Brett Cannon)
Date: Tue, 24 Nov 2015 19:31:48 +0000
Subject: [Python-Dev] [Python-checkins] Daily reference leaks
 (e5e507a357a6): sum=103
In-Reply-To: <20151124084506.81870.97482@psf.io>
References: <20151124084506.81870.97482@psf.io>
Message-ID: <CAP1=2W6D=WRzR0+V2K=R_VK381PEB+oAcaW4DgzoeA-4hUrJ1A@mail.gmail.com>

Someone just added a leak to pickle.

On Tue, 24 Nov 2015 at 01:45 <solipsis at pitrou.net> wrote:

> results for e5e507a357a6 on branch "default"
> --------------------------------------------
>
> test_asyncio leaked [3, 0, 0] memory blocks, sum=3
> test_functools leaked [0, 2, 2] memory blocks, sum=4
> test_pickle leaked [20, 20, 20] references, sum=60
> test_pickle leaked [12, 12, 12] memory blocks, sum=36
>
>
> Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R',
> '3:3:/home/psf-users/antoine/refleaks/reflogUP9efi', '--timeout', '7200']
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> https://mail.python.org/mailman/listinfo/python-checkins
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151124/1eb05b7a/attachment.html>

From storchaka at gmail.com  Tue Nov 24 16:21:21 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 24 Nov 2015 23:21:21 +0200
Subject: [Python-Dev] Deleting with setting C API functions
Message-ID: <n32kch$eie$1@ger.gmane.org>

Slots like PyTypeObject.tp_setattr, PySequenceMethods.sq_ass_item, 
PyMappingMethods.mp_ass_subscript are used not only for setting 
attribute/item value, but also for deleting attribute/item (if value is 
NULL). This is not documented and should be. [1]  Correspondingly public 
API functions like PyObject_SetAttr, PyObject_SetItem, 
PySequence_SetItem, PySequence_SetSlice, PyMapping_SetItemString can be 
used for deleting. But all these functions have special counterparts for 
deleting: PyObject_DelAttr etc.

The question is wherever deleting ability of Set-functions is 
intentional, should we document this or deprecate and then delete?

[1] http://bugs.python.org/issue25701


From storchaka at gmail.com  Tue Nov 24 16:22:22 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 24 Nov 2015 23:22:22 +0200
Subject: [Python-Dev] [Python-checkins] Daily reference leaks
 (e5e507a357a6): sum=103
In-Reply-To: <CAP1=2W6D=WRzR0+V2K=R_VK381PEB+oAcaW4DgzoeA-4hUrJ1A@mail.gmail.com>
References: <20151124084506.81870.97482@psf.io>
 <CAP1=2W6D=WRzR0+V2K=R_VK381PEB+oAcaW4DgzoeA-4hUrJ1A@mail.gmail.com>
Message-ID: <n32ked$eie$2@ger.gmane.org>

On 24.11.15 21:31, Brett Cannon wrote:
> Someone just added a leak to pickle.

It always was here. I just added tests that expose it.

http://bugs.python.org/issue25725



From guido at python.org  Tue Nov 24 16:33:09 2015
From: guido at python.org (Guido van Rossum)
Date: Tue, 24 Nov 2015 13:33:09 -0800
Subject: [Python-Dev] Deleting with setting C API functions
In-Reply-To: <n32kch$eie$1@ger.gmane.org>
References: <n32kch$eie$1@ger.gmane.org>
Message-ID: <CAP7+vJJidsPNU0avVDNHZOq=mqzvmBVHcoTWNQjwAZa2Bzz6nw@mail.gmail.com>

Ooooh, that's probably really old code. I guess for the slots the
reasoning is to save on slots. For the public functions, alas it will
be hard to know if anyone is depending on it, even if it's
undocumented. Perhaps add a deprecation warning to these if the value
is NULL for one release cycle?

On Tue, Nov 24, 2015 at 1:21 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> Slots like PyTypeObject.tp_setattr, PySequenceMethods.sq_ass_item,
> PyMappingMethods.mp_ass_subscript are used not only for setting
> attribute/item value, but also for deleting attribute/item (if value is
> NULL). This is not documented and should be. [1]  Correspondingly public API
> functions like PyObject_SetAttr, PyObject_SetItem, PySequence_SetItem,
> PySequence_SetSlice, PyMapping_SetItemString can be used for deleting. But
> all these functions have special counterparts for deleting: PyObject_DelAttr
> etc.
>
> The question is wherever deleting ability of Set-functions is intentional,
> should we document this or deprecate and then delete?
>
> [1] http://bugs.python.org/issue25701
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From ncoghlan at gmail.com  Tue Nov 24 20:07:02 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 25 Nov 2015 11:07:02 +1000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CABVPEKqt5oi=5dKHnGJtOSc5zkORLwOdcv+BLxp14GmF1ubYgA@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <56536A6A.9030902@python.org>
 <CADiSq7cifEZSYNHGZH4bFTpNqXEJnAk_xro83Xq-5yBpOoFktw@mail.gmail.com>
 <20151123205906.02cd57ec@limelight.wooz.org>
 <CABVPEKq4Ea9aP-aNVF9XFoQXX9=UvXo7beHg9Lq8_JT43X-oxA@mail.gmail.com>
 <CACac1F_N5fZS0Vj_CMpWj1AR15fFaBVwyn4+A7QCFYYuL3WJDQ@mail.gmail.com>
 <CABVPEKp7hPnyLx209Ypet4rsNFC5a39Zy7BFNrq2Y-riutp=Lw@mail.gmail.com>
 <CACac1F8JFYvFSenC1SMpVGRSMQjPOaK59fkqW4yYSgJu6-Au_w@mail.gmail.com>
 <CABVPEKqt5oi=5dKHnGJtOSc5zkORLwOdcv+BLxp14GmF1ubYgA@mail.gmail.com>
Message-ID: <CADiSq7eFV_MtvRcXezza5tEhU+KH4S2qjctXhAYMhZ7=nKExzw@mail.gmail.com>

On 25 Nov 2015 05:42, "Toshio Kuratomi" <a.badger at gmail.com> wrote:

> >
>
> Yeah, I think you are correct in your understanding of what actual
> changes to the python distrribution are being proposed for
> redistributors in the PEP.  Reading through the PEP again, I'm not
> sure if I'm correct in thinking that this only applies to
> backporting... it may be that the environment section of the PEP
> applies to any python-2 while the config file section only applies to
> backporting.  Nick, could you clarify?

Yep, the environment variable section is the part MAL persuaded me was a
good idea during the original discussion of this PEP, based on the
interests of eGenix's customers. It's a cross-platform suggestion for
distributors wanting to provide a smoother upgrade path, applicable to both
backports and rebasing on 2.7.9+. (I originally didn't like it, but became
amenable once it was pointed out that anyone with the ability to set
process environment variables can already alter OpenSSL's settings to use a
different CA cert file or directory)

The config file recommendation is specific to Linux distro backports, with
the aim of saying "IF you backport PEP 476 to an older Python 2.7 point
release, THEN you should do it like this for cross-distro consistency". As
the current discussion shows, I'm *personally* an advocate for backporting,
but I don't think the PEP itself should reflect that - the suggestions to
adjust the tone are good ones, so I'll update the text accordingly.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151125/921c65bb/attachment.html>

From ncoghlan at gmail.com  Wed Nov 25 01:39:27 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 25 Nov 2015 16:39:27 +1000
Subject: [Python-Dev] Deleting with setting C API functions
In-Reply-To: <CAP7+vJJidsPNU0avVDNHZOq=mqzvmBVHcoTWNQjwAZa2Bzz6nw@mail.gmail.com>
References: <n32kch$eie$1@ger.gmane.org>
 <CAP7+vJJidsPNU0avVDNHZOq=mqzvmBVHcoTWNQjwAZa2Bzz6nw@mail.gmail.com>
Message-ID: <CADiSq7cxGtzJMXCmZgbcTVRy3R8biS-2rf-Szmc6udYRz+uB_g@mail.gmail.com>

On 25 November 2015 at 07:33, Guido van Rossum <guido at python.org> wrote:
> Ooooh, that's probably really old code. I guess for the slots the
> reasoning is to save on slots. For the public functions, alas it will
> be hard to know if anyone is depending on it, even if it's
> undocumented. Perhaps add a deprecation warning to these if the value
> is NULL for one release cycle?

I did a quick scan for "PyObject_SetAttr", and it turns out
PyObject_DelAttr is only a convenience macro for calling
PyObject_SetAttr with NULL as the value argument. bltinmodule.c and
ceval.c also both include direct calls to PyObject_SetAttr with
"(PyObject *)NULL" as the value argument.

Investigating some of the uses that passed a variable as the value
argument, one case is the weakref proxy implementation, which uses
PyObject_SetAttr on the underlying object in its implementation of the
setattr slot in the proxy.

So it looks to me like replicating the NULL-handling behaviour of the
slots in the public Set* APIs was intentional, and it's just the
documentation of that detail that was missed (since most folks
presumably use the Del* convenience APIs instead).

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From cory at lukasa.co.uk  Wed Nov 25 04:12:29 2015
From: cory at lukasa.co.uk (Cory Benfield)
Date: Wed, 25 Nov 2015 09:12:29 +0000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <56545DDC.7000407@python.org>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <56536A6A.9030902@python.org>
 <CADiSq7cifEZSYNHGZH4bFTpNqXEJnAk_xro83Xq-5yBpOoFktw@mail.gmail.com>
 <56545DDC.7000407@python.org>
Message-ID: <AD542A2C-8277-45A0-87ED-C1974E28E4E7@lukasa.co.uk>


> On 24 Nov 2015, at 12:53, Christian Heimes <christian at python.org> wrote:
> Right, with Antoine and Alex out of scope and you, Victor and me working
> for Red Hat, the air is getting thin. Benjamin is familiar with the ssl
> module. Or we can follow Alex's advice and ask somebody from the PyCA
> group (Donald, Paul, lvh) or requests (Cory) to get some outside
> perspective.

I?m not a CPython core developer, so I think that excludes me.

For what it?s worth, I think that this is a good PEP, and I?m strongly in favour of it being approved in some form.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151125/5b50a36e/attachment.sig>

From barry at python.org  Wed Nov 25 14:57:14 2015
From: barry at python.org (Barry Warsaw)
Date: Wed, 25 Nov 2015 14:57:14 -0500
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <56547C68.8060506@egenix.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
 <201511241427.tAOERFJC028474@fido.openend.se>
 <56547C68.8060506@egenix.com>
Message-ID: <20151125145714.2a02651d@limelight.wooz.org>

There's a lot to process in this thread, but as I see it, the issue breaks
down to these questions:

* How should PEP 493 be implemented?

* What should the default be?

* How should PEP 493 be worded to express the right tone to redistributors?

Let me take on the implementation details here.

On Nov 24, 2015, at 04:04 PM, M.-A. Lemburg wrote:

>I would still find having built-in support for the recommendations
>in the Python stdlib a better approach

As would I.

Currently the PEP contains some sample code for reading a configuration file
and an environment variable, and assigning the default behavior based on these
values.

The first improvement to the stdlib I would have made would have been to add
a convenience function to ssl to enable and disable the verification.
Something like this:

    def ssl.verify_certificates(enable)

Then at the simplest level, an application that wants to enable higher
security could do this early in its main entry point:

    import ssl
    if hasattr(ssl, 'verify_certificates'):
        ssl.verify_certificates(True)

or

    ...
        ssl.verify_certificates(False)

and that would work regardless of the platform default, and it would work
properly on older versions of Python 2.7 that didn't have the backport.  This
seems so straightforward, I'm wondering why PEP 493 doesn't consider it.

I would like to see the sample code in PEP 493 implemented in the stdlib.  It
would then be a matter of cherry picking that should a distro decide to
backport it.  It means one less non-trivial delta from upstream which has to
be managed separately.  The only difference to the combined approach I'd make
is to handle the case where the _cert_verification_config file doesn't exist.

(If you included the convenience method above, then the sample code could be
simplified down to a default for the `enable` parameter and a call to
ssl.verify_certificates().)

Is it too late to include both of those suggestions in PEP 493 and Python
2.7.11?

Cheers,
-Barry

From robertc at robertcollins.net  Wed Nov 25 15:17:02 2015
From: robertc at robertcollins.net (Robert Collins)
Date: Thu, 26 Nov 2015 09:17:02 +1300
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <20151125145714.2a02651d@limelight.wooz.org>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
 <201511241427.tAOERFJC028474@fido.openend.se>
 <56547C68.8060506@egenix.com>
 <20151125145714.2a02651d@limelight.wooz.org>
Message-ID: <CAJ3HoZ0DKXRZ2DYQAm+m=xsYgbf+akC3JTfgFXhe9ZpP2dn2Bw@mail.gmail.com>

On 26 November 2015 at 08:57, Barry Warsaw <barry at python.org> wrote:
> There's a lot to process in this thread, but as I see it, the issue breaks
> down to these questions:
>
> * How should PEP 493 be implemented?
>
> * What should the default be?
>
> * How should PEP 493 be worded to express the right tone to redistributors?
>
> Let me take on the implementation details here.
>
> On Nov 24, 2015, at 04:04 PM, M.-A. Lemburg wrote:
>
>>I would still find having built-in support for the recommendations
>>in the Python stdlib a better approach
>
> As would I.

For what its worth: a PEP telling distributors to patch the standard
library is really distasteful to me.

We've spent a long time trying to build close relations such that when
something doesn't work distributors can share their needs with us and
we can make Python out of the box be a good fit. This seems to fly in
the exact opposite direction: we're explicitly making it so that
Python builds on these vendor's platforms will not be the same as you
get by checking out the Python source code.

Ugh.

-Rob



-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud

From rdmurray at bitdance.com  Wed Nov 25 15:39:54 2015
From: rdmurray at bitdance.com (R. David Murray)
Date: Wed, 25 Nov 2015 15:39:54 -0500
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CAJ3HoZ0DKXRZ2DYQAm+m=xsYgbf+akC3JTfgFXhe9ZpP2dn2Bw@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
 <201511241427.tAOERFJC028474@fido.openend.se> <56547C68.8060506@egenix.com>
 <20151125145714.2a02651d@limelight.wooz.org>
 <CAJ3HoZ0DKXRZ2DYQAm+m=xsYgbf+akC3JTfgFXhe9ZpP2dn2Bw@mail.gmail.com>
Message-ID: <20151125203955.D1791B1408E@webabinitio.net>

On Thu, 26 Nov 2015 09:17:02 +1300, Robert Collins <robertc at robertcollins.net> wrote:
> On 26 November 2015 at 08:57, Barry Warsaw <barry at python.org> wrote:
> > There's a lot to process in this thread, but as I see it, the issue breaks
> > down to these questions:
> >
> > * How should PEP 493 be implemented?
> >
> > * What should the default be?
> >
> > * How should PEP 493 be worded to express the right tone to redistributors?
> >
> > Let me take on the implementation details here.
> >
> > On Nov 24, 2015, at 04:04 PM, M.-A. Lemburg wrote:
> >
> >>I would still find having built-in support for the recommendations
> >>in the Python stdlib a better approach
> >
> > As would I.
> 
> For what its worth: a PEP telling distributors to patch the standard
> library is really distasteful to me.
> 
> We've spent a long time trying to build close relations such that when
> something doesn't work distributors can share their needs with us and
> we can make Python out of the box be a good fit. This seems to fly in
> the exact opposite direction: we're explicitly making it so that
> Python builds on these vendor's platforms will not be the same as you
> get by checking out the Python source code.

I think we should include the environment variable support in CPython
and be done with it (nuke the PEP otherwise).  Which is what I've
thought from the beginning :)

--David

From sad2project at gmail.com  Wed Nov 25 15:31:09 2015
From: sad2project at gmail.com (Jacob Zimmerman)
Date: Wed, 25 Nov 2015 14:31:09 -0600
Subject: [Python-Dev] Help with a book
Message-ID: <CAMis+RMi6_By1Krnk_dGFaXfT0fpSRifm597yuQ2n_TccN-LHw@mail.gmail.com>

Hello Python Developers!

I'm writing a book about descriptors, and I'm hoping to make it a
comprehensive guide, covering just about everything there is to know about
descriptors, including many tips to get past certain knowledge barriers and
how to avoid certain pitfalls. I'm hoping for it to become the definitive
guide for those who want to really understand descriptors. Raymond's
Descriptor HowTo Guide is great, but focused. I want to expand on what that
guide teaches.

Anyway, I am currently reading through my initial rough draft and making
corrections for a revised draft. Once I'm done with that, I'd like a few
volunteers to read through it. First, to find spelling, grammatical, or
consistency mistakes, though those should be few in number, but also to
give input as to whether something seems too repetitive, if some more
information should be given in a spot, and if I'm even blatantly wrong
about anything (hopefully not).

Anyone who helps will be able to receive a free ebook of the final copy as
well as mention in the Special Thanks.

I appreciate any help anyone can give to assist in making this book as
great as it can be.

Thanks,
Jacob Zimmerman

Twitter: @jacobz_20
Blog: http://programmingideaswithjake.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151125/453ff131/attachment.html>

From barry at python.org  Wed Nov 25 15:57:13 2015
From: barry at python.org (Barry Warsaw)
Date: Wed, 25 Nov 2015 15:57:13 -0500
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CABVPEKq4Ea9aP-aNVF9XFoQXX9=UvXo7beHg9Lq8_JT43X-oxA@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <56536A6A.9030902@python.org>
 <CADiSq7cifEZSYNHGZH4bFTpNqXEJnAk_xro83Xq-5yBpOoFktw@mail.gmail.com>
 <20151123205906.02cd57ec@limelight.wooz.org>
 <CABVPEKq4Ea9aP-aNVF9XFoQXX9=UvXo7beHg9Lq8_JT43X-oxA@mail.gmail.com>
Message-ID: <20151125155713.42936cff@limelight.wooz.org>

On Nov 24, 2015, at 09:16 AM, Toshio Kuratomi wrote:

>It sounds like the implementation sections of the PEP are acceptable

Well, see my follow up to MAL's email.  From Rob's and RDM's responses, it
seems we're not alone. :)

I like all your other proposed changes, although I'll note the preference to
minimize the unique-to-downstream changes.

Cheers,
-Barry

From lac at openend.se  Wed Nov 25 15:58:44 2015
From: lac at openend.se (Laura Creighton)
Date: Wed, 25 Nov 2015 21:58:44 +0100
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <20151125203955.D1791B1408E@webabinitio.net>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
 <201511241427.tAOERFJC028474@fido.openend.se> <56547C68.8060506@egenix.com>
 <20151125145714.2a02651d@limelight.wooz.org>
 <CAJ3HoZ0DKXRZ2DYQAm+m=xsYgbf+akC3JTfgFXhe9ZpP2dn2Bw@mail.gmail.com>
 <20151125203955.D1791B1408E@webabinitio.net>
Message-ID: <201511252058.tAPKwiiY025031@fido.openend.se>

In a message of Wed, 25 Nov 2015 15:39:54 -0500, "R. David Murray" writes:
>I think we should include the environment variable support in CPython
>and be done with it (nuke the PEP otherwise).  Which is what I've
>thought from the beginning :)
>
>--David

I like this idea too.

Laura

From barry at python.org  Wed Nov 25 16:04:35 2015
From: barry at python.org (Barry Warsaw)
Date: Wed, 25 Nov 2015 16:04:35 -0500
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CADiSq7eewfMuWg7RM3KFkJfLCk5NOEfRtcwTTrA2W=9ofp2jfg@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <56536A6A.9030902@python.org>
 <CADiSq7cifEZSYNHGZH4bFTpNqXEJnAk_xro83Xq-5yBpOoFktw@mail.gmail.com>
 <20151123205906.02cd57ec@limelight.wooz.org>
 <CADiSq7eewfMuWg7RM3KFkJfLCk5NOEfRtcwTTrA2W=9ofp2jfg@mail.gmail.com>
Message-ID: <20151125160435.2de0791c@limelight.wooz.org>

On Nov 24, 2015, at 02:12 PM, Nick Coghlan wrote:

>On 24 November 2015 at 11:59, Barry Warsaw <barry at python.org> wrote:

>> We'd have to also handle
>> migration paths to newer Ubuntu releases, which probably means removing the
>> config file on future upgrades.
>
>You'll need to figure out those migration paths anyway, as the certificate
>verification compatibility break will exist between 14.04 and 16.04
>regardless of what happens with PEP 493.

Sure, but that's a very different situation.  If someone upgraded from 14.04.3
to 14.04.4, they definitely wouldn't expect it to break working code.  If it
did, it would be a pretty serious regression.  Upgrading to LTS maintenance
releases should be a no-brainer.

Upgrading from one LTS to a newer one, you *know* things are going to change,
so you'll review the release notes and do some very careful analysis of your
mission critical code.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151125/1945a3dc/attachment.sig>

From barry at python.org  Wed Nov 25 16:13:32 2015
From: barry at python.org (Barry Warsaw)
Date: Wed, 25 Nov 2015 16:13:32 -0500
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <20151125203955.D1791B1408E@webabinitio.net>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
 <201511241427.tAOERFJC028474@fido.openend.se>
 <56547C68.8060506@egenix.com>
 <20151125145714.2a02651d@limelight.wooz.org>
 <CAJ3HoZ0DKXRZ2DYQAm+m=xsYgbf+akC3JTfgFXhe9ZpP2dn2Bw@mail.gmail.com>
 <20151125203955.D1791B1408E@webabinitio.net>
Message-ID: <20151125161332.43b0dd5b@limelight.wooz.org>

On Nov 25, 2015, at 03:39 PM, R. David Murray wrote:

>I think we should include the environment variable support in CPython
>and be done with it (nuke the PEP otherwise).  Which is what I've
>thought from the beginning :)

+1.  I'd like to see my proposed convenience function added too.  I'm -0 on
the config file though since while I think it might be useful, I'm not sure
we'd actually install that on Ubuntu.

Cheers,
-Barry

From vgr255 at live.ca  Wed Nov 25 17:48:00 2015
From: vgr255 at live.ca (Emanuel Barry)
Date: Wed, 25 Nov 2015 17:48:00 -0500
Subject: [Python-Dev] Help with a book
In-Reply-To: <CAMis+RMi6_By1Krnk_dGFaXfT0fpSRifm597yuQ2n_TccN-LHw@mail.gmail.com>
References: <CAMis+RMi6_By1Krnk_dGFaXfT0fpSRifm597yuQ2n_TccN-LHw@mail.gmail.com>
Message-ID: <BLU172-W17BC54CFB9D4F6FBBF0B1F91050@phx.gbl>

As someone who plays around a *lot* with descriptors, I would be interested in reviewing it. I may not have the same knowledge as a core developer, but I can still help.

Date: Wed, 25 Nov 2015 14:31:09 -0600
From: sad2project at gmail.com
To: python-dev at python.org
Subject: [Python-Dev] Help with a book

Hello Python Developers!
I'm writing a book about descriptors, and I'm hoping to make it a comprehensive guide, covering just about everything there is to know about descriptors, including many tips to get past certain knowledge barriers and how to avoid certain pitfalls. I'm hoping for it to become the definitive guide for those who want to really understand descriptors. Raymond's Descriptor HowTo Guide is great, but focused. I want to expand on what that guide teaches.
Anyway, I am currently reading through my initial rough draft and making corrections for a revised draft. Once I'm done with that, I'd like a few volunteers to read through it. First, to find spelling, grammatical, or consistency mistakes, though those should be few in number, but also to give input as to whether something seems too repetitive, if some more information should be given in a spot, and if I'm even blatantly wrong about anything (hopefully not).

Anyone who helps will be able to receive a free ebook of the final copy as well as mention in the Special Thanks.
I appreciate any help anyone can give to assist in making this book as great as it can be.
Thanks,Jacob Zimmerman
Twitter: @jacobz_20Blog: http://programmingideaswithjake.wordpress.com

 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151125/dec60c4c/attachment.html>

From rosuav at gmail.com  Wed Nov 25 18:55:05 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 26 Nov 2015 10:55:05 +1100
Subject: [Python-Dev] Help with a book
In-Reply-To: <CAMis+RMi6_By1Krnk_dGFaXfT0fpSRifm597yuQ2n_TccN-LHw@mail.gmail.com>
References: <CAMis+RMi6_By1Krnk_dGFaXfT0fpSRifm597yuQ2n_TccN-LHw@mail.gmail.com>
Message-ID: <CAPTjJmqJ89pbuTc9GJ2SWtVA8B3dAV48H6KpENgpw7tm5dUa9A@mail.gmail.com>

On Thu, Nov 26, 2015 at 7:31 AM, Jacob Zimmerman <sad2project at gmail.com> wrote:
> Anyway, I am currently reading through my initial rough draft and making
> corrections for a revised draft. Once I'm done with that, I'd like a few
> volunteers to read through it. First, to find spelling, grammatical, or
> consistency mistakes, though those should be few in number, but also to give
> input as to whether something seems too repetitive, if some more information
> should be given in a spot, and if I'm even blatantly wrong about anything
> (hopefully not).

I'd be happy to help out. Wordsmithing is my life, and I've used
descriptors some (not as much as Emanuel will have, though!).

ChrisA

From ncoghlan at gmail.com  Wed Nov 25 23:13:14 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 26 Nov 2015 14:13:14 +1000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <20151125145714.2a02651d@limelight.wooz.org>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
 <201511241427.tAOERFJC028474@fido.openend.se>
 <56547C68.8060506@egenix.com>
 <20151125145714.2a02651d@limelight.wooz.org>
Message-ID: <CADiSq7eAFAK71F7oC7E=MH=MB+mCQtVBLsM_5kVEPNFbzRXhqA@mail.gmail.com>

On 26 November 2015 at 05:57, Barry Warsaw <barry at python.org> wrote:
> There's a lot to process in this thread, but as I see it, the issue breaks
> down to these questions:
>
> * How should PEP 493 be implemented?
>
> * What should the default be?
>
> * How should PEP 493 be worded to express the right tone to redistributors?
>
> Let me take on the implementation details here.
>
> On Nov 24, 2015, at 04:04 PM, M.-A. Lemburg wrote:
>
>>I would still find having built-in support for the recommendations
>>in the Python stdlib a better approach
>
> As would I.
>
> Currently the PEP contains some sample code for reading a configuration file
> and an environment variable, and assigning the default behavior based on these
> values.
>
> The first improvement to the stdlib I would have made would have been to add
> a convenience function to ssl to enable and disable the verification.
> Something like this:
>
>     def ssl.verify_certificates(enable)
>
> Then at the simplest level, an application that wants to enable higher
> security could do this early in its main entry point:
>
>     import ssl
>     if hasattr(ssl, 'verify_certificates'):
>         ssl.verify_certificates(True)
>
> or
>
>     ...
>         ssl.verify_certificates(False)
>
> and that would work regardless of the platform default, and it would work
> properly on older versions of Python 2.7 that didn't have the backport.  This
> seems so straightforward, I'm wondering why PEP 493 doesn't consider it.

PEP 476 rejected providing a public indefinitely maintained API for
this, so PEP 493 is specifically about helping commercial
redistributors offer a smoother transition plan to customers without
affecting the public Python level API, and without encouraging a
plethora of mutually incompatible transition schemes.

> I would like to see the sample code in PEP 493 implemented in the stdlib.  It
> would then be a matter of cherry picking that should a distro decide to
> backport it.  It means one less non-trivial delta from upstream which has to
> be managed separately.  The only difference to the combined approach I'd make
> is to handle the case where the _cert_verification_config file doesn't exist.
>
> (If you included the convenience method above, then the sample code could be
> simplified down to a default for the `enable` parameter and a call to
> ssl.verify_certificates().)
>
> Is it too late to include both of those suggestions in PEP 493 and Python
> 2.7.11?

PEP 493 isn't about attempting to rehash the PEP 476 discussions in
search of a different conclusion, so this would need to be a different
PEP, preferably one that targets Python 3.6 first and covers more than
just HTTPS.  Such an API could then be considered as a backport
candidate for 2.7.12. (2.7.11rc1 has already been published, so there
isn't any scope for updates there). I actually think that's a nicer
approach, but also think the recommendations in PEP 493 are much
simpler to implement and test, while still meeting the bar of "good
enough to solve the problem at hand".

If anyone does decide to write that PEP, then the fact that SELinux
offers "setenforce 0" and AppArmor can be turned off with a kernel
boot option can be used as examples of platform developers letting end
users make their own security decisions, even if the platform provider
recommends leaving the more opinionated default behaviour enabled.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Thu Nov 26 00:06:00 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 26 Nov 2015 15:06:00 +1000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CAJ3HoZ0DKXRZ2DYQAm+m=xsYgbf+akC3JTfgFXhe9ZpP2dn2Bw@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
 <201511241427.tAOERFJC028474@fido.openend.se>
 <56547C68.8060506@egenix.com>
 <20151125145714.2a02651d@limelight.wooz.org>
 <CAJ3HoZ0DKXRZ2DYQAm+m=xsYgbf+akC3JTfgFXhe9ZpP2dn2Bw@mail.gmail.com>
Message-ID: <CADiSq7fReF2uAdzxxLMrwLTeHJLJxQNfMUupP5dnYtf6LU6BEw@mail.gmail.com>

On 26 November 2015 at 06:17, Robert Collins <robertc at robertcollins.net> wrote:
> On 26 November 2015 at 08:57, Barry Warsaw <barry at python.org> wrote:
>> There's a lot to process in this thread, but as I see it, the issue breaks
>> down to these questions:
>>
>> * How should PEP 493 be implemented?
>>
>> * What should the default be?
>>
>> * How should PEP 493 be worded to express the right tone to redistributors?
>>
>> Let me take on the implementation details here.
>>
>> On Nov 24, 2015, at 04:04 PM, M.-A. Lemburg wrote:
>>
>>>I would still find having built-in support for the recommendations
>>>in the Python stdlib a better approach
>>
>> As would I.
>
> For what its worth: a PEP telling distributors to patch the standard
> library is really distasteful to me.

I'm not a big fan of it either, but it's the way sustainable
commercial open source distribution works in practice: customers want
selective upgrades, so customers get selective upgrades; customers
want greater platform integration, so customers get greater platform
integration; etc, etc. Large Python shops like Google will typically
maintain their own custom Python variants with their own patches
applied that are suitable and sufficient for their use cases, but may
not be suitable for pushing upstream (e.g. they may be Linux specific
patches, without support for other platforms).

That's why I've focused on providing guidance to redistributors to
keep their patches within reasonably consistent bounds for PEPs like
bundling pip, backporting the network security enhancements to Python
2.7, and changing the default behaviour for HTTPS verification.
Different redistributors have different customer bases, which also
differ from the audience for upstream python.org releases, so saying
"don't patch Python" is denying the reality of the obligations
commercial vendors have to their customers, while "when you patch
Python, please ensure you abide by these guidelines" is something
redistributors can realistically do.

> We've spent a long time trying to build close relations such that when
> something doesn't work distributors can share their needs with us and
> we can make Python out of the box be a good fit. This seems to fly in
> the exact opposite direction: we're explicitly making it so that
> Python builds on these vendor's platforms will not be the same as you
> get by checking out the Python source code.

This has always been true for the RHEL system Python - it gets
backported changes, just like the kernel does (albeit nowhere near as
many). That's one of the main things long term support customers are
paying for - developers continuing to care about a code base after the
volunteers working on the project upstream have (entirely reasonably)
shifted their attentions to newer versions. We try to keep the patches
to a minimum, and avoid having to carry large patches indefinitely by
only backporting changes that have already been accepted in later
versions, but "never apply patches to upstream projects as a
downstream redistributor" isn't a realistic goal.

In this particular case, the migration problems were already raised in
the PEP 476 discussions, and the decision was made to *not* provide a
public API specifically for globally turning off HTTPS certificate
verification, since that would either:

1. Need to be added to Python 3 and maintained indefinitely; or
2. Be implemented in Python 2.7 only, and thus create another barrier
to 2->3 migration

A HTTPS specific flag wasn't considered a suitable long term API
design for Python 3, while a multi-protocol configuration design that
also handled setting the defaults for other protocols like secure
SMTP, IMAP, POP3, SFTP, etc seemed generally desirable (but also
doesn't exist anywhere, and poses signficant challenges in designing
and implementing a good test matrix).

The only concrete things that have changed since PEP 476 was approved is that:

* we figured out the monkeypatching based migration recommendations in
that PEP didn't actually work for the redistributor case
* Robert Kuska came up with a way to do a backwards compatible
backport of PEP 476 to RHEL 7.2 that doesn't raise any new barriers to
migrating from Python 2 to Python 3

PEP 493 originally only described the latter idea - I added MAL's
design for the environment variable based security downgrade based on
the original PEP 493 discussions in July. Splitting the latter out to
its own PEP that proposes it as a forwards compatible configuration
setting akin to "curl --insecure" and "wget --no-check-certificate"
for Python 2.7.12, 3.4.4, 3.5.2 and 3.6 upstream would be fine by me,
since that would also significantly clarify the fact that the two
ideas are entirely orthogonal to each other (except for the need to
define a relative priority when you combine the two).

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From barry at python.org  Thu Nov 26 11:48:43 2015
From: barry at python.org (Barry Warsaw)
Date: Thu, 26 Nov 2015 11:48:43 -0500
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CADiSq7eAFAK71F7oC7E=MH=MB+mCQtVBLsM_5kVEPNFbzRXhqA@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CAP7+vJJQ8WTXF2iUKuwavGLTJKgx61rj=Le_7abnyUs7yCQ=8g@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
 <201511241427.tAOERFJC028474@fido.openend.se>
 <56547C68.8060506@egenix.com>
 <20151125145714.2a02651d@limelight.wooz.org>
 <CADiSq7eAFAK71F7oC7E=MH=MB+mCQtVBLsM_5kVEPNFbzRXhqA@mail.gmail.com>
Message-ID: <20151126114843.3db83d5f@anarchist.wooz.org>

On Nov 26, 2015, at 02:13 PM, Nick Coghlan wrote:

>PEP 476 rejected providing a public indefinitely maintained API for this, so
>PEP 493 is specifically about helping commercial redistributors offer a
>smoother transition plan to customers without affecting the public Python
>level API, and without encouraging a plethora of mutually incompatible
>transition schemes.

Of course, the API would only have to be support for the life of 2.7; it would
never go in 3.x so the burden is minimal.

>PEP 493 isn't about attempting to rehash the PEP 476 discussions in
>search of a different conclusion, so this would need to be a different
>PEP, preferably one that targets Python 3.6 first and covers more than
>just HTTPS.

That seems like overkill.  PEP 493 is specifically about Python 2.7 and
providing ways for downstreams to facilitate more choice for end-users and
end-administrators.  Although I think it could safely sneak in after rc1, that
would be for the RM to decide.  Even if it were deferred to 2.7.12, it would
still provide a better, more consistent experience if implemented upstream.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151126/f9c5486f/attachment.sig>

From barry at python.org  Thu Nov 26 12:15:10 2015
From: barry at python.org (Barry Warsaw)
Date: Thu, 26 Nov 2015 12:15:10 -0500
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CADiSq7fReF2uAdzxxLMrwLTeHJLJxQNfMUupP5dnYtf6LU6BEw@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
 <201511241427.tAOERFJC028474@fido.openend.se>
 <56547C68.8060506@egenix.com>
 <20151125145714.2a02651d@limelight.wooz.org>
 <CAJ3HoZ0DKXRZ2DYQAm+m=xsYgbf+akC3JTfgFXhe9ZpP2dn2Bw@mail.gmail.com>
 <CADiSq7fReF2uAdzxxLMrwLTeHJLJxQNfMUupP5dnYtf6LU6BEw@mail.gmail.com>
Message-ID: <20151126121510.32c9a500@anarchist.wooz.org>

On Nov 26, 2015, at 03:06 PM, Nick Coghlan wrote:

>I'm not a big fan of it either, but it's the way sustainable
>commercial open source distribution works in practice:

While it's inevitable that redistributors have to deviate from upstream, in
Debian and Ubuntu, we really try to keep that at a minimum.  Debian, of course
is an all volunteer organization so our "customers" are just normal users who
I think want an experience as close to installing from source as possible,
consistent with the principles and policies of the Debian project.  Debian at
least tries to document the handful of deviations from upstream.

Likewise in Ubuntu, we try to keep deviations from Debian at a minimum, and
document them when we must deviate.  Ubuntu is a community driven distro so
while Canonical itself has customers, it's much more likely that feedback
about the Python stack comes from ordinary users.  Again, my personal goal is
to make Python on Ubuntu a pleasant and comfortable environment, as close to
installing from source as possible, consistent with the principles and
policies of the project.

All this to say that IMHO, solutions to problems should go as far upstream as
possible.  Guidelines certainly are useful (e.g. PEP 394) to maintain
consistency across downstream providers where the decisions are primarily in
the sphere of influence of the downstreams, but IMHO there's still more that
upstream can do to promote PEP 493.

>Different redistributors have different customer bases, which also
>differ from the audience for upstream python.org releases, so saying
>"don't patch Python" is denying the reality of the obligations
>commercial vendors have to their customers, while "when you patch
>Python, please ensure you abide by these guidelines" is something
>redistributors can realistically do.

ISTM that the same forces are in play regardless of whether the change is in
code or in an informational PEP.  Best to get consensus where possible, and
manifest those decisions in code, but if competing goals are the outcome of a
code change or informational PEP, downstream consumers will still make what
they judge to be the best decision in the interest of their users, balanced
against their own competing constraints.

>In this particular case, the migration problems were already raised in
>the PEP 476 discussions, and the decision was made to *not* provide a
>public API specifically for globally turning off HTTPS certificate
>verification, since that would either:
>
>1. Need to be added to Python 3 and maintained indefinitely; or
>2. Be implemented in Python 2.7 only, and thus create another barrier
>to 2->3 migration

A minor one, for sure.  It wouldn't significantly bother me if the API were
underscore-prefixed as a big honkin' clue about the guarantees being made.

>* Robert Kuska came up with a way to do a backwards compatible
>backport of PEP 476 to RHEL 7.2 that doesn't raise any new barriers to
>migrating from Python 2 to Python 3

I'm not sure I agree with that.  If you have to set an environment variable to
get your application to work in Python 2.7, it's *still* a migration issue
when you move to Python 3.  You're just moving the change out of the code and
into the environment, where it's arguably more difficult to re-discover down
the road.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151126/8ffa6410/attachment.sig>

From ncoghlan at gmail.com  Thu Nov 26 19:52:27 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 27 Nov 2015 10:52:27 +1000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <20151126121510.32c9a500@anarchist.wooz.org>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
 <201511241427.tAOERFJC028474@fido.openend.se>
 <56547C68.8060506@egenix.com>
 <20151125145714.2a02651d@limelight.wooz.org>
 <CAJ3HoZ0DKXRZ2DYQAm+m=xsYgbf+akC3JTfgFXhe9ZpP2dn2Bw@mail.gmail.com>
 <CADiSq7fReF2uAdzxxLMrwLTeHJLJxQNfMUupP5dnYtf6LU6BEw@mail.gmail.com>
 <20151126121510.32c9a500@anarchist.wooz.org>
Message-ID: <CADiSq7fqG5wLMu-QjRneogKF8cmfTxc0cJeAwj8irZQ25D51gg@mail.gmail.com>

On 27 November 2015 at 03:15, Barry Warsaw <barry at python.org> wrote:
> On Nov 26, 2015, at 03:06 PM, Nick Coghlan wrote:
>
>>I'm not a big fan of it either, but it's the way sustainable
>>commercial open source distribution works in practice:
>
> While it's inevitable that redistributors have to deviate from upstream, in
> Debian and Ubuntu, we really try to keep that at a minimum.  Debian, of course
> is an all volunteer organization so our "customers" are just normal users who
> I think want an experience as close to installing from source as possible,
> consistent with the principles and policies of the Debian project.  Debian at
> least tries to document the handful of deviations from upstream.

Yeah, we need to be better about that - while
https://access.redhat.com/articles/2039753 covers the details for the
certificate verification changes, there's currently no easy way to get
a description of the changes backported to the RHEL/CentOS system
Python without trawling through old Errata announcements :(

> Likewise in Ubuntu, we try to keep deviations from Debian at a minimum, and
> document them when we must deviate.  Ubuntu is a community driven distro so
> while Canonical itself has customers, it's much more likely that feedback
> about the Python stack comes from ordinary users.  Again, my personal goal is
> to make Python on Ubuntu a pleasant and comfortable environment, as close to
> installing from source as possible, consistent with the principles and
> policies of the project.

I'd strongly agree with that description for Fedora and
softwarecollections.org, but for the RHEL/CentOS system Python I think
the situation is slightly different: there, the goal is to meet the
long term support commitments involved in being a base RHEL package.
As the nominal base version of the package (2.7.5 in the case of RHEL
7) doesn't change, there is naturally going to be increasing
divergence from the nominal version.

> All this to say that IMHO, solutions to problems should go as far upstream as
> possible.  Guidelines certainly are useful (e.g. PEP 394) to maintain
> consistency across downstream providers where the decisions are primarily in
> the sphere of influence of the downstreams, but IMHO there's still more that
> upstream can do to promote PEP 493.

I tried to go down the "upstream first" path with a properly supported
"off switch" in PEP 476, and didn't succeed (hence the monkeypatch
compromise). It sounds like several folks would like to see us revisit
that decision, though.

>>Different redistributors have different customer bases, which also
>>differ from the audience for upstream python.org releases, so saying
>>"don't patch Python" is denying the reality of the obligations
>>commercial vendors have to their customers, while "when you patch
>>Python, please ensure you abide by these guidelines" is something
>>redistributors can realistically do.
>
> ISTM that the same forces are in play regardless of whether the change is in
> code or in an informational PEP.  Best to get consensus where possible, and
> manifest those decisions in code, but if competing goals are the outcome of a
> code change or informational PEP, downstream consumers will still make what
> they judge to be the best decision in the interest of their users, balanced
> against their own competing constraints.
>
>>In this particular case, the migration problems were already raised in
>>the PEP 476 discussions, and the decision was made to *not* provide a
>>public API specifically for globally turning off HTTPS certificate
>>verification, since that would either:
>>
>>1. Need to be added to Python 3 and maintained indefinitely; or
>>2. Be implemented in Python 2.7 only, and thus create another barrier
>>to 2->3 migration
>
> A minor one, for sure.  It wouldn't significantly bother me if the API were
> underscore-prefixed as a big honkin' clue about the guarantees being made.

OK, you've persuaded me to recast the PEP from an Informational one to
a Standards track one. The section about backporting to versions prior
to 2.7.9 won't change much (except in tone), but the ability to opt
out on a process-wide basis will be pitched as a language level
feature.

>>* Robert Kuska came up with a way to do a backwards compatible
>>backport of PEP 476 to RHEL 7.2 that doesn't raise any new barriers to
>>migrating from Python 2 to Python 3
>
> I'm not sure I agree with that.  If you have to set an environment variable to
> get your application to work in Python 2.7, it's *still* a migration issue
> when you move to Python 3.  You're just moving the change out of the code and
> into the environment, where it's arguably more difficult to re-discover down
> the road.

By that I meant that setting the environment variable won't *break*
anything in Python 3, whereas if we add a new Python level API and
people call it without a hasattr() guard, then even if their
infrastructure has been upgraded to use verifiable certificates, the
client applications will break on Python 3.

However, with an underscore prefix and an admonition to use a
hasattr() check, I agree a 2.7.x-only Python level API isn't really
any more of a barrier to migration than only offering the environment
variable.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ldeller at iress.com.au  Thu Nov 26 21:06:53 2015
From: ldeller at iress.com.au (Luke Deller)
Date: Fri, 27 Nov 2015 02:06:53 +0000
Subject: [Python-Dev] crash bug in weakref_richcompare?
Message-ID: <d6e014cb9bc44d3093e1f157407dcfe7@AU01-EXCH1.devel.iress.com.au>

Hi

I have come across some dubious code in Objects/weakrefobject.c which looks like a bug to me, but wanted to run it past others.

This was discovered from looking at crash dumps from a multithreaded python app (using Python 2.7.9, but the same weakref code exists in 3.5 and hg tip).

The code that worries me is at the end of the "weakref_richcompare" function:

return PyObject_RichCompare(PyWeakref_GET_OBJECT(self),
                            PyWeakref_GET_OBJECT(other), op);

At this point the code has established that the referents are still alive, and it is trying to compare the referents.  However it has not acquired a strong reference to the referents, so I think it is possible for one of them to be deleted half way through this comparison.  This can lead to a crash, because PyObject_RichCompare assumes that the PyObject*'s it was passed will remain usable for the duration of the call.

The crashes I have seen involve data corruption consistent with one of these PyObject's being deleted and the memory reused for something else, eg:

00 python27!try_3way_compare+0x15 [objects\object.c @ 712]
01 python27!try_3way_to_rich_compare+0xb [objects\object.c @ 901]
02 python27!do_richcmp+0x2c [objects\object.c @ 935]
03 python27!PyObject_RichCompare+0x99 [objects\object.c @ 982]
04 python27!weakref_richcompare+0x6a [objects\weakrefobject.c @ 212]

(In this example v->ob_type was 0x5f637865 which is ASCII "exc_", not a valid pointer at all)

Other places in weakrefobject.c seem to have a similar weakness, eg in weakref_hash and weakref_repr.

I have not been able to produce a small test case to exhibit this crash, but from this inspection of the code it looks like a bug - am I understanding this correctly?

Thanks,
Luke.
**********************************************************************************************
Important Note
This email (including any attachments) contains information which is confidential and may be subject to legal privilege.  If you are not the intended recipient you must not use, distribute or copy this email.  If you have received this email in error please notify the
sender immediately and delete this email. Any views expressed in this email are not necessarily the views of IRESS Limited.

It is the duty of the recipient to virus scan and otherwise test the information provided before loading onto any computer system.
IRESS Limited does not warrant that the information is free of a virus or any other defect or error.
**********************************************************************************************


From a.badger at gmail.com  Thu Nov 26 23:02:58 2015
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Thu, 26 Nov 2015 20:02:58 -0800
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CADiSq7fqG5wLMu-QjRneogKF8cmfTxc0cJeAwj8irZQ25D51gg@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
 <201511241427.tAOERFJC028474@fido.openend.se>
 <56547C68.8060506@egenix.com>
 <20151125145714.2a02651d@limelight.wooz.org>
 <CAJ3HoZ0DKXRZ2DYQAm+m=xsYgbf+akC3JTfgFXhe9ZpP2dn2Bw@mail.gmail.com>
 <CADiSq7fReF2uAdzxxLMrwLTeHJLJxQNfMUupP5dnYtf6LU6BEw@mail.gmail.com>
 <20151126121510.32c9a500@anarchist.wooz.org>
 <CADiSq7fqG5wLMu-QjRneogKF8cmfTxc0cJeAwj8irZQ25D51gg@mail.gmail.com>
Message-ID: <CABVPEKp2YYctLRzOLgCMi2Vdgs7PG6gMPhdvRM4sKUThwzz-zg@mail.gmail.com>

On Nov 26, 2015 4:53 PM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
>
> On 27 November 2015 at 03:15, Barry Warsaw <barry at python.org> wrote:

>
> > Likewise in Ubuntu, we try to keep deviations from Debian at a minimum,
and
> > document them when we must deviate.  Ubuntu is a community driven
distro so
> > while Canonical itself has customers, it's much more likely that
feedback
> > about the Python stack comes from ordinary users.  Again, my personal
goal is
> > to make Python on Ubuntu a pleasant and comfortable environment, as
close to
> > installing from source as possible, consistent with the principles and
> > policies of the project.
>
> I'd strongly agree with that description for Fedora and
> softwarecollections.org, but for the RHEL/CentOS system Python I think
> the situation is slightly different: there, the goal is to meet the
> long term support commitments involved in being a base RHEL package.
> As the nominal base version of the package (2.7.5 in the case of RHEL
> 7) doesn't change, there is naturally going to be increasing
> divergence from the nominal version.

I think the goal in rhel/centos is similar, actually.  The maintenance
burden for non upstream changes has been acknowledged as a problem to be
avoided by rhel maintainers before.  The caveat for those distributions is
that they accumulate more *backports*.

However, backports are easier to maintain than non upstream changes.  The
test of the upstream community helps to find and fix bugs in the code; the
downstream maintainer just needs to stay aware of whether fixes are going
into the code they've backported.

> I tried to go down the "upstream first" path with a properly supported
> "off switch" in PEP 476, and didn't succeed (hence the monkeypatch
> compromise). It sounds like several folks would like to see us revisit
> that decision, though.
>
That's the rub.  If there's now enough support to push this upstream I
think everyone downstream will be happier.  If it turns out there's still
enough resistance to keep it from upstream then I suppose you cross that
bridge if it becomes necessary.

-Toshio
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151126/2758d689/attachment.html>

From ncoghlan at gmail.com  Fri Nov 27 01:04:30 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 27 Nov 2015 16:04:30 +1000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CADiSq7fqG5wLMu-QjRneogKF8cmfTxc0cJeAwj8irZQ25D51gg@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
 <201511241427.tAOERFJC028474@fido.openend.se>
 <56547C68.8060506@egenix.com>
 <20151125145714.2a02651d@limelight.wooz.org>
 <CAJ3HoZ0DKXRZ2DYQAm+m=xsYgbf+akC3JTfgFXhe9ZpP2dn2Bw@mail.gmail.com>
 <CADiSq7fReF2uAdzxxLMrwLTeHJLJxQNfMUupP5dnYtf6LU6BEw@mail.gmail.com>
 <20151126121510.32c9a500@anarchist.wooz.org>
 <CADiSq7fqG5wLMu-QjRneogKF8cmfTxc0cJeAwj8irZQ25D51gg@mail.gmail.com>
Message-ID: <CADiSq7cxCOmsaWEpGPMqjhKtQL2BQowxG092A1bmvC8bAtFrcQ@mail.gmail.com>

On 27 November 2015 at 10:52, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 27 November 2015 at 03:15, Barry Warsaw <barry at python.org> wrote:
>> On Nov 26, 2015, at 03:06 PM, Nick Coghlan wrote:
>>>In this particular case, the migration problems were already raised in
>>>the PEP 476 discussions, and the decision was made to *not* provide a
>>>public API specifically for globally turning off HTTPS certificate
>>>verification, since that would either:
>>>
>>>1. Need to be added to Python 3 and maintained indefinitely; or
>>>2. Be implemented in Python 2.7 only, and thus create another barrier
>>>to 2->3 migration
>>
>> A minor one, for sure.  It wouldn't significantly bother me if the API were
>> underscore-prefixed as a big honkin' clue about the guarantees being made.
>
> OK, you've persuaded me to recast the PEP from an Informational one to
> a Standards track one. The section about backporting to versions prior
> to 2.7.9 won't change much (except in tone), but the ability to opt
> out on a process-wide basis will be pitched as a language level
> feature.

New draft pushed: https://hg.python.org/peps/rev/f602a47ea795

This is a significant rewrite that switches the PEP to a Standards
Track PEP proposing two new features for 2.7.12+: an
"ssl._verify_https_certificates()" configuration function, and a
"PYTHONHTTPSVERIFY" environment variable (although writing them
together like that makes me wonder if the latter should now be
"PYTHONVERIFYHTTPS" instead).

There are then 3 backport recommendation sections:

* one for backporting the ability to enable HTTPS verification by
default for the entire Python installation
* one for backporting the new features described in this PEP
* one for combining both backports in a single implementation

I tried to trim some of the resulting duplication between the
sections, but I suspect the overall text is still longer than it needs
to be. I've also added links describing the origin of the
configuration file based backport in RHEL development, and where to
find the source code for the patches in Fedora.

Cheers,
Nick.

===================================
PEP: 493
Title: HTTPS verification migration tools for Python 2.7
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan <ncoghlan at gmail.com>,
        Robert Kuska <rkuska at redhat.com>,
        Marc-Andr? Lemburg <mal at lemburg.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 10-May-2015
Python-Version: 2.7.12
Post-History: 06-Jul-2015, 11-Nov-2015, 24-Nov-2015


Abstract
========

PEP 476 updated Python's default handling of HTTPS certificates in client
modules to align with certificate handling in web browsers, by validating
that the certificates received belonged to the server the client was attempting
to contact. The Python 2.7 long term maintenance series was judged to be in
scope for this change, with the new behaviour introduced in the Python 2.7.9
maintenance release.

This has created a non-trivial barrier to adoption for affected Python 2.7
maintenance releases, so this PEP proposes additional Python 2.7 specific
features that allow system administrators and other users to more easily
decouple the decision to verify server certificates in HTTPS client modules
from the decision to update to newer Python 2.7 maintenance releases.


Rationale
=========

PEP 476 changed Python's default behaviour to align with expectations
established by web browsers in regards to the semantics of HTTPS URLs:
starting with Python 2.7.9 and 3.4.3, HTTPS clients in the standard library
validate server certificates by default.

However, it is also the case that this change *does* cause problems for
infrastructure administrators operating private intranets that rely on
self-signed certificates, or otherwise encounter problems with the new default
certificate verification settings.

To manage these kinds of situations, web browsers provide users with "click
through" warnings that allow the user to add the server's certificate to the
browser's certificate store. Network client tools like ``curl`` and ``wget``
offer options to switch off certificate checking entirely (by way of
``curl --insecure`` and ``wget --no-check-certificate``, respectively).

At a different layer of the technology stack, Linux security modules like
`SELinux` and `AppArmor`, while enabled by default by distribution vendors,
offer relatively straightforward mechanisms for turning them off.

At the moment, no such convenient mechanisms exist to disable Python's
default certificate checking for a whole process.

PEP 476 did attempt to address this question, by covering how to revert to the
old settings process wide by monkeypatching the ``ssl`` module to restore the
old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed
to allow system administrators to disable the feature by default in their
Standard Operating Environment definition has been determined to be
insufficient in at least some cases. The specific case that led to the
initial creation of this PEP is the one where a Linux distributor aims to
provide their users with a
`smoother migration path
<https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__
than the standard one provided by consuming upstream CPython 2.7 releases
directly, but other potential challenges have also been pointed out with
updating embedded Python runtimes and other user level installations of Python.

Rather than allowing a plethora of mutually incompatibile migration techniques
to bloom, this PEP proposes an additional feature to be added to Python 2.7.12
to make it easier to revert a process to the past behaviour of skipping
certificate validation in HTTPS client modules. It also provides additional
recommendations to redistributors backporting these features to versions of
Python prior to Python 2.7.9.

These designs are being proposed purely as tools for helping to manage the
transition to the new default certificate handling behaviour in the context
of Python 2.7. They are not being proposed as new features for Python 3, as
it is expected that the vast majority of client applications affected by this
problem without the ability to update the application itself will be Python 2
applications.

It would likely be desirable for a future version of Python 3 to allow default
certificate handling for secure protocols to be configurable on a per-protocol
basis, but that question is beyond the scope of this PEP.

Alternatives
------------

In the absence of clear upstream guidance and recommendations, commercial
redistributors will still make their own design decisions in the interests of
their customers. The main approaches available are:

* Continuing to rebase on new Python 2.7.x releases, while providing no
  additional assistance beyond the mechanisms defined in PEP 476 in migrating
  from unchecked to checked hostnames in standard library HTTPS clients
* Gating availability of the changes in default handling of HTTPS connections
  on upgrading from Python 2 to Python 3
* For Linux distribution vendors, gating availability of the changes in default
  handling of HTTPS connections on upgrading to a new operating system version
* Implementing one or both of the backport suggestions described in this PEP,
  regardless of the formal status of the PEP


Requirements for capability detection
=====================================

As the proposals in this PEP aim to facilitate backports to earlier Python
versions, the Python version number cannot be used as a reliable means for
detecting them. Instead, they are designed to allow the presence
or absence of the feature to be determined using the following technique::

    python -c "import ssl; ssl._relevant_attribute"

This will fail with `AttributeError` (and hence a non-zero return code) if the
relevant capability is not available.

The marker attributes are prefixed with an underscore to indicate the
implementation dependent nature of these capabilities - not all Python
distributions will offer them, only those that are providing a multi-stage
migration process from the original Python 2.7 HTTPS handling to the new
default behaviour.

Feature: Configuration API
==========================

This change is proposed for inclusion in CPython 2.7.12 and later CPython 2.7.x
releases. It consists of a new ``ssl._verify_https_certificates()`` to specify
the default handling of HTTPS certificates in standard library client libraries.

It is not proposed to forward port this change to Python 3, so Python 3
applications that need to support skipping certificate verification will still
need to define their own suitable security context.

Feature detection
-----------------

The marker attribute on the ``ssl`` module related to this feature is the
``ssl._verify_https_certificates`` function itself.

Specification
-------------

The ``ssl._verify_https_certificates`` function will work as follows::

    def _verify_https_certificates(enable=True):
        """Verify server HTTPS certificates by default?"""
        global _create_default_https_context
        if enable:
            _create_default_https_context = create_default_context
        else:
            _create_default_https_context = _create_unverified_context

If called without arguments, or with ``enable`` set to a true value, then
standard library client modules will subsequently verify HTTPS
certificates by default, otherwise they will skip verification.

If called with ``enable`` set to a false value, then standard library client
modules will subsequently skip verifying HTTPS certificates by default.

Security Considerations
-----------------------

The inclusion of this feature will allow security sensitive applications to
include the following forward-compatible snippet in their code::

    if hasattr(ssl, "_verify_https_certificates"):
        ssl._verify_https_certificates()

Some developers may also choose to opt out of certificate checking using
``ssl._verify_https_certificates(enable=False)``. This doesn't introduce any
major new security concerns, as monkeypatching the affected internal APIs was
already possible.


Feature: environment based configuration
========================================

This change is proposed for inclusion in CPython 2.7.12 and later CPython 2.7.x
releases. It consists of a new ``PYTHONHTTPSVERIFY`` environment variable that
allows the default verification to be disabled without modifying the
application source code (which may not even be available in cases of
bytecode-only application distribution)

It is not proposed to forward port this change to Python 3, so Python 3
applications that need to support skipping certificate verification will still
need to define their own suitable security context.

Feature detection
-----------------

The marker attribute on the ``ssl`` module related to this feature is:

* the ``ssl._https_verify_envvar`` attribute, giving the name of environment
  variable affecting the default behaviour

This not only makes it straightforward to detect the presence (or absence) of
the capability, it also makes it possible to programmatically determine the
relevant environment variable name.

Specification
-------------

Rather than always defaulting to the use of ``ssl.create_default_context``,
the ``ssl`` module will be modified to:

* read the ``PYTHONHTTPSVERIFY`` environment variable when the module is first
  imported into a Python process
* set the ``ssl._create_default_https_context`` function to be an alias for
  ``ssl._create_unverified_context`` if this environment variable is present
  and set to ``'0'``
* otherwise, set the ``ssl._create_default_https_context`` function to be an
  alias for ``ssl.create_default_context`` as usual

Example implementation
----------------------

::

    _https_verify_envvar = 'PYTHONHTTPSVERIFY'

    def _get_https_context_factory():
        if not sys.flags.ignore_environment:
            config_setting = os.environ.get(_https_verify_envvar)
            if config_setting == '0':
                return _create_unverified_context
        return create_default_context

    _create_default_https_context = _get_https_context_factory()

Security Considerations
-----------------------

Relative to the behaviour in Python 3.4.3+ and Python 2.7.9->2.7.11, this
approach does introduce a new downgrade attack against the default security
settings that potentially allows a sufficiently determined attacker to revert
Python to the default behaviour used in CPython 2.7.8 and earlier releases.
However, such an attack requires the ability to modify the execution
environment of a Python process prior to the import of the ``ssl`` module,
and any attacker with such access would already be able to modify the
behaviour of the underlying OpenSSL implementation.

Interaction with Python virtual environments
--------------------------------------------

The default setting is read directly from the process environment, and hence
works the same way regardless of whether or not the interpreter is being run
inside an activated Python virtual environment.


Backporting PEP 476 to earlier Python versions
==============================================

Some redistributors, most notably Linux distributions, may choose to backport
the PEP 476 HTTPS verification changes to modified Python versions based on
earlier Python 2 maintenance releases. In these cases, a configuration
mechanism is needed that provides:

* an opt-in model that allows the decision to enable HTTPS certificate
  verification to be made independently of the decision to upgrade to the
  Python version where the feature was first backported
* the ability for system administrators to set the default behaviour of Python
  applications and scripts run directly in the system Python installation
* the ability for the redistributor to consider changing the default behaviour
  of *new* installations at some point in the future without impacting existing
  installations that have been explicitly configured to skip verifying HTTPS
  certificates by default

As it only affects backports to earlier releases of Python 2.7, this change is
not proposed for inclusion in upstream CPython, but rather is offered as
guidance to redistributors to reduce the likelihood of multiple mutually
incompatible approaches to backporting being adopted.

This approach SHOULD NOT be used for any Python installation that advertises
itself as providing Python 2.7.9 or later, as most Python users will have the
reasonable expectation that all such environments will validate HTTPS
certificates by default.


Feature detection
-----------------

The marker attribute on the ``ssl`` module related to this feature is::

    _cert_verification_config = '<path to configuration file>'

This not only makes it straightforward to detect the presence (or absence) of
the capability, it also makes it possible to programmatically determine the
relevant configuration file name.


Recommended modifications to the Python standard library
--------------------------------------------------------

The recommended approach to backporting the PEP 476 modifications to an earlier
point release is to implement the following changes relative to the default
PEP 476 behaviour implemented in Python 2.7.9+:

* modify the ``ssl`` module to read a system wide configuration file when the
  module is first imported into a Python process
* define a platform default behaviour (either verifying or not verifying HTTPS
  certificates) to be used if this configuration file is not present
* support selection between the following three modes of operation:

  * ensure HTTPS certificate verification is enabled
  * ensure HTTPS certificate verification is disabled
  * delegate the decision to the redistributor providing this Python version

* set the ``ssl._create_default_https_context`` function to be an alias for
  either ``ssl.create_default_context`` or ``ssl._create_unverified_context``
  based on the given configuration setting.


Recommended file location
-------------------------

As the PEP authors are not aware of any vendors providing long-term support
releases targeting Windows, Mac OS X or \*BSD systems, this approach is
currently only specifically defined for Linux system Python installations.

The recommended configuration file name on Linux systems is
``/etc/python/cert-verification.cfg``.

The ``.cfg`` filename extension is recommended for consistency with the
``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library.


Recommended file format
-----------------------

The configuration file should use a ConfigParser ini-style format with a
single section named ``[https]`` containing one required setting ``verify``.

The suggested section name is taken from the "https" URL schema passed to
affected client APIs.

Permitted values for ``verify`` are:

* ``enable``: ensure HTTPS certificate verification is enabled by default
* ``disable``: ensure HTTPS certificate verification is disabled by default
* ``platform_default``: delegate the decision to the redistributor providing
  this particular Python version

If the ``[https]`` section or the ``verify`` setting are missing, or if the
``verify`` setting is set to an unknown value, it should be treated as if the
configuration file is not present.


Example implementation
----------------------

::

    _cert_verification_config = '/etc/python/cert-verification.cfg'

    def _get_https_context_factory():
        # Check for a system-wide override of the default behaviour
        context_factories = {
            'enable': create_default_context,
            'disable': _create_unverified_context,
            'platform_default': _create_unverified_context, # For now :)
        }
        import ConfigParser
        config = ConfigParser.RawConfigParser()
        config.read(_cert_verification_config)
        try:
            verify_mode = config.get('https', 'verify')
        except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
            verify_mode = 'platform_default'
        default_factory = context_factories.get('platform_default')
        return context_factories.get(verify_mode, default_factory)

    _create_default_https_context = _get_https_context_factory()


Security Considerations
-----------------------

The specific recommendations for this backporting case are designed to work for
privileged, security sensitive processes, even those being run in the following
locked down configuration:

* run from a locked down administrator controlled directory rather than a normal
  user directory (preventing ``sys.path[0]`` based privilege escalation attacks)
* run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based
  privilege escalation attacks)
* run using the ``-s`` switch (preventing user site directory based privilege
  escalation attacks)
* run using the ``-S`` switch (preventing ``sitecustomize`` based privilege
  escalation attacks)

The intent is that the *only* reason HTTPS verification should be getting
turned off system wide when using this approach is because:

* an end user is running a redistributor provided version of CPython rather
  than running upstream CPython directly
* that redistributor has decided to provide a smoother migration path to
  verifying HTTPS certificates by default than that being provided by the
  upstream project
* either the redistributor or the local infrastructure administrator has
  determined that it is appropriate to override the default upstream behaviour
  (at least for the time being)

Using an administrator controlled configuration file rather than an environment
variable has the essential feature of providing a smoother migration path, even
for applications being run with the ``-E`` switch.

Interaction with Python virtual environments
--------------------------------------------

This setting is scoped by the interpreter installation and affects all Python
processes using that interpreter, regardless of whether or not the interpreter
is being run inside an activated Python virtual environment.

Origins of this recommendation
------------------------------

This recommendation is based on the backporting approach adopted for Red Hat
Enterprise Linux 7.2, as published in the original July 2015 draft of this PEP
and described in detail in `this KnowledgeBase article
<https://access.redhat.com/articles/2039753>`__. Red Hat's patches implementing
this backport for Python 2.7.5 can be found in the `CentOS git repository
<https://git.centos.org/commit/rpms!python.git/refs!heads!c7>`__.


Backporting this PEP to earlier Python versions
===============================================

The configuration file based backport described above is designed to cover
backporting the PEP 476 changes to default certificate handling without the
additional configuration mechanisms defined in this PEP.

If this PEP is accepted, then an additional backporting option becomes
available, which is to backport the per-process configuration mechanisms
defined in this PEP, without backporting the ability to change the
default behaviour of the overall Python installation.

Such a backport would differ from the mechanism proposed in this PEP solely in
the default behaviour when ``PYTHONHTTPSVERIFY`` was not set at all: it would
continue to default to skipping certificate validation.

In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and
set to anything *other* than ``'0'``, then HTTPS certificate verification
should be enabled.


Feature detection
-----------------

There's no specific attribute indicating that this situation applies. Rather,
it is indicated by the ``ssl._verify_https_certificates`` and
``ssl._https_verify_envvar`` attributes being present in a Python version that
is nominally older than Python 2.7.9.

Specification
-------------

Implementing this backport involves backporting the changes in PEP 466, 476 and
this PEP, with the following change to the handling of the
``PYTHONHTTPSVERIFY`` environment variable in the ``ssl`` module:

* read the ``PYTHONHTTPSVERIFY`` environment variable when the module is first
  imported into a Python process
* set the ``ssl._create_default_https_context`` function to be an alias for
  ``ssl.create_default_context`` if this environment variable is present
  and set to any value other than ``'0'``
* otherwise, set the ``ssl._create_default_https_context`` function to be an
  alias for ``ssl._create_unverified_context``

Example implementation
----------------------

::

    _https_verify_envvar = 'PYTHONHTTPSVERIFY'

    def _get_https_context_factory():
        if not sys.flags.ignore_environment:
            config_setting = os.environ.get(_https_verify_envvar)
            if config_setting != '0':
                return create_default_context
        return _create_unverified_context

    _create_default_https_context = _get_https_context_factory()

    def _disable_https_default_verification():
        """Skip verification of HTTPS certificates by default"""
        global _create_default_https_context
        _create_default_https_context = _create_unverified_context

Security Considerations
-----------------------

This change would be a strict security upgrade for any Python version that
currently defaults to skipping certificate validation in standard library
HTTPS clients. The technical trade-offs to be taken into account relate largely
to the magnitude of the PEP 466 backport also required rather than to anything
security related.

Interaction with Python virtual environments
--------------------------------------------

The default setting is read directly from the process environment, and hence
works the same way regardless of whether or not the interpreter is being run
inside an activated Python virtual environment.


Recommendation for combined feature backports
=============================================

If a redistributor chooses to backport the environment variable based
configuration setting from this PEP to a modified Python version that also
implements the configuration file based PEP 476 , then the environment
variable should take precedence over the system-wide configuration setting.
This allows the setting to be changed for a given user or application,
regardless of the installation-wide default behaviour.

Example implementation
----------------------

::

    _https_verify_envvar = 'PYTHONHTTPSVERIFY'
    _cert_verification_config = '/etc/python/cert-verification.cfg'

    def _get_https_context_factory():
        # Check for an environmental override of the default behaviour
        if not sys.flags.ignore_environment:
            config_setting = os.environ.get(_https_verify_envvar)
            if config_setting is not None:
                if config_setting == '0':
                    return _create_unverified_context
                return create_default_context

        # Check for a system-wide override of the default behaviour
        context_factories = {
            'enable': create_default_context,
            'disable': _create_unverified_context,
            'platform_default': _create_unverified_context, # For now :)
        }
        import ConfigParser
        config = ConfigParser.RawConfigParser()
        config.read(_cert_verification_config)
        try:
            verify_mode = config.get('https', 'verify')
        except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
            verify_mode = 'platform_default'
        default_factory = context_factories.get('platform_default')
        return context_factories.get(verify_mode, default_factory)

    _create_default_https_context = _get_https_context_factory()



-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Fri Nov 27 02:47:33 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 27 Nov 2015 17:47:33 +1000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <22104.2412.671436.247736@turnbull.sk.tsukuba.ac.jp>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
 <201511241427.tAOERFJC028474@fido.openend.se>
 <56547C68.8060506@egenix.com>
 <20151125145714.2a02651d@limelight.wooz.org>
 <CAJ3HoZ0DKXRZ2DYQAm+m=xsYgbf+akC3JTfgFXhe9ZpP2dn2Bw@mail.gmail.com>
 <CADiSq7fReF2uAdzxxLMrwLTeHJLJxQNfMUupP5dnYtf6LU6BEw@mail.gmail.com>
 <20151126121510.32c9a500@anarchist.wooz.org>
 <CADiSq7fqG5wLMu-QjRneogKF8cmfTxc0cJeAwj8irZQ25D51gg@mail.gmail.com>
 <CADiSq7cxCOmsaWEpGPMqjhKtQL2BQowxG092A1bmvC8bAtFrcQ@mail.gmail.com>
 <22104.2412.671436.247736@turnbull.sk.tsukuba.ac.jp>
Message-ID: <CADiSq7djsBOm_oySrW3F+haBc=zCn2T34YMnb557Lk0hrXoivg@mail.gmail.com>

On 27 November 2015 at 17:42, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> Nick Coghlan writes:
>
>  > This is a significant rewrite that switches the PEP to a Standards
>  > Track PEP proposing two new features for 2.7.12+: an
>  > "ssl._verify_https_certificates()" configuration function, and a
>  > "PYTHONHTTPSVERIFY" environment variable (although writing them
>  > together like that makes me wonder if the latter should now be
>  > "PYTHONVERIFYHTTPS" instead).
>
> +1 on the same order of "words" in the function name and the
> evnironment variable.  I tend to prefer
> "ssl._https_verify_certificates" since these aren't "HTTPS
> certificates".

Ah, yes, that would make sense, so I've just pushed an update that
switches the function name order to match the environment variable.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From stephen at xemacs.org  Fri Nov 27 02:42:36 2015
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 27 Nov 2015 16:42:36 +0900
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CADiSq7cxCOmsaWEpGPMqjhKtQL2BQowxG092A1bmvC8bAtFrcQ@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
 <201511241427.tAOERFJC028474@fido.openend.se>
 <56547C68.8060506@egenix.com>
 <20151125145714.2a02651d@limelight.wooz.org>
 <CAJ3HoZ0DKXRZ2DYQAm+m=xsYgbf+akC3JTfgFXhe9ZpP2dn2Bw@mail.gmail.com>
 <CADiSq7fReF2uAdzxxLMrwLTeHJLJxQNfMUupP5dnYtf6LU6BEw@mail.gmail.com>
 <20151126121510.32c9a500@anarchist.wooz.org>
 <CADiSq7fqG5wLMu-QjRneogKF8cmfTxc0cJeAwj8irZQ25D51gg@mail.gmail.com>
 <CADiSq7cxCOmsaWEpGPMqjhKtQL2BQowxG092A1bmvC8bAtFrcQ@mail.gmail.com>
Message-ID: <22104.2412.671436.247736@turnbull.sk.tsukuba.ac.jp>

Nick Coghlan writes:

 > This is a significant rewrite that switches the PEP to a Standards
 > Track PEP proposing two new features for 2.7.12+: an
 > "ssl._verify_https_certificates()" configuration function, and a
 > "PYTHONHTTPSVERIFY" environment variable (although writing them
 > together like that makes me wonder if the latter should now be
 > "PYTHONVERIFYHTTPS" instead).

+1 on the same order of "words" in the function name and the
evnironment variable.  I tend to prefer
"ssl._https_verify_certificates" since these aren't "HTTPS
certificates".


From cory at lukasa.co.uk  Fri Nov 27 03:47:17 2015
From: cory at lukasa.co.uk (Cory Benfield)
Date: Fri, 27 Nov 2015 08:47:17 +0000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CADiSq7cxCOmsaWEpGPMqjhKtQL2BQowxG092A1bmvC8bAtFrcQ@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
 <201511241427.tAOERFJC028474@fido.openend.se> <56547C68.8060506@egenix.com>
 <20151125145714.2a02651d@limelight.wooz.org>
 <CAJ3HoZ0DKXRZ2DYQAm+m=xsYgbf+akC3JTfgFXhe9ZpP2dn2Bw@mail.gmail.com>
 <CADiSq7fReF2uAdzxxLMrwLTeHJLJxQNfMUupP5dnYtf6LU6BEw@mail.gmail.com>
 <20151126121510.32c9a500@anarchist.wooz.org>
 <CADiSq7fqG5wLMu-QjRneogKF8cmfTxc0cJeAwj8irZQ25D51gg@mail.gmail.com>
 <CADiSq7cxCOmsaWEpGPMqjhKtQL2BQowxG092A1bmvC8bAtFrcQ@mail.gmail.com>
Message-ID: <27C511D5-0393-4473-8CAA-272298A2BAE1@lukasa.co.uk>


> On 27 Nov 2015, at 06:04, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> Feature: Configuration API
> ==========================
> 
> This change is proposed for inclusion in CPython 2.7.12 and later CPython 2.7.x
> releases. It consists of a new ``ssl._verify_https_certificates()`` to specify
> the default handling of HTTPS certificates in standard library client libraries.
> 
> It is not proposed to forward port this change to Python 3, so Python 3
> applications that need to support skipping certificate verification will still
> need to define their own suitable security context.
> 
> Feature detection
> -----------------
> 
> The marker attribute on the ``ssl`` module related to this feature is the
> ``ssl._verify_https_certificates`` function itself.
> 
> Specification
> -------------
> 
> The ``ssl._verify_https_certificates`` function will work as follows::
> 
>    def _verify_https_certificates(enable=True):
>        """Verify server HTTPS certificates by default?"""
>        global _create_default_https_context
>        if enable:
>            _create_default_https_context = create_default_context
>        else:
>            _create_default_https_context = _create_unverified_context
> 
> If called without arguments, or with ``enable`` set to a true value, then
> standard library client modules will subsequently verify HTTPS
> certificates by default, otherwise they will skip verification.

Perhaps I missed this, Nick, but what happens if multiple third party libraries apply updates to call this function in incompatible ways? For example, if you depend on libfoo which calls ssl._verify_https_certificates(False) and libbar which calls ssl._verify_https_certificates(True)? Is it?last import wins?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151127/2b7da067/attachment.sig>

From ncoghlan at gmail.com  Fri Nov 27 06:38:11 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 27 Nov 2015 21:38:11 +1000
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <27C511D5-0393-4473-8CAA-272298A2BAE1@lukasa.co.uk>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
 <201511241427.tAOERFJC028474@fido.openend.se>
 <56547C68.8060506@egenix.com>
 <20151125145714.2a02651d@limelight.wooz.org>
 <CAJ3HoZ0DKXRZ2DYQAm+m=xsYgbf+akC3JTfgFXhe9ZpP2dn2Bw@mail.gmail.com>
 <CADiSq7fReF2uAdzxxLMrwLTeHJLJxQNfMUupP5dnYtf6LU6BEw@mail.gmail.com>
 <20151126121510.32c9a500@anarchist.wooz.org>
 <CADiSq7fqG5wLMu-QjRneogKF8cmfTxc0cJeAwj8irZQ25D51gg@mail.gmail.com>
 <CADiSq7cxCOmsaWEpGPMqjhKtQL2BQowxG092A1bmvC8bAtFrcQ@mail.gmail.com>
 <27C511D5-0393-4473-8CAA-272298A2BAE1@lukasa.co.uk>
Message-ID: <CADiSq7dwuVFsPmdpFgrb7yfp+ZjuKG3j6=nYAqK9rLiRxSYNCg@mail.gmail.com>

On 27 November 2015 at 18:47, Cory Benfield <cory at lukasa.co.uk> wrote:
> Perhaps I missed this, Nick, but what happens if multiple third party libraries apply updates to call this function in incompatible ways? For example, if you depend on libfoo which calls ssl._verify_https_certificates(False) and libbar which calls ssl._verify_https_certificates(True)? Is it?last import wins?

Last import wins, but libaries shouldn't be mutating process global
state as a side effect of import - like the sys module, the ssl module
configuration should only be modified (directly or indirectly) from
__main__.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From wjun77 at gmail.com  Fri Nov 27 03:40:12 2015
From: wjun77 at gmail.com (=?UTF-8?B?546L54+6?=)
Date: Fri, 27 Nov 2015 16:40:12 +0800
Subject: [Python-Dev] The AttributeError/__getattr__ mechanism
Message-ID: <CAPM967Txs4Vr35zY-Rk2dGhCZuEqjfLjfYdvE47gqMYe-Qst0w@mail.gmail.com>

Hello everyone:

I'm suggesting a modification to the AttributeError/__getattr__ mechanism,
see issue25634:
http://bugs.python.org/issue25634

I used __getattr__ sometimes, and *descriptor* especially *property* is so
widely used. I wonder whether someone
had encountered the same problem with me.

However, this is a complicated problem including performance issues, and
backward compatibility.

Thanks for your attention,
Jun Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151127/a7759101/attachment.html>

From victor.stinner at gmail.com  Fri Nov 27 11:31:24 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 27 Nov 2015 17:31:24 +0100
Subject: [Python-Dev] The AttributeError/__getattr__ mechanism
In-Reply-To: <CAPM967Txs4Vr35zY-Rk2dGhCZuEqjfLjfYdvE47gqMYe-Qst0w@mail.gmail.com>
References: <CAPM967Txs4Vr35zY-Rk2dGhCZuEqjfLjfYdvE47gqMYe-Qst0w@mail.gmail.com>
Message-ID: <CAMpsgwaboBFf64xN1erWtZAkHOk6npeK39ET-8Wy8u6Qqyd-pQ@mail.gmail.com>

Hi,

As R. David Murray wrote on the issue, the python-ideas mailing list
is better suited to start such discussion ;-)

Victor

2015-11-27 9:40 GMT+01:00 ?? <wjun77 at gmail.com>:
> Hello everyone:
>
> I'm suggesting a modification to the AttributeError/__getattr__ mechanism,
> see issue25634:
> http://bugs.python.org/issue25634
>
> I used __getattr__ sometimes, and descriptor especially property is so
> widely used. I wonder whether someone
> had encountered the same problem with me.
>
> However, this is a complicated problem including performance issues, and
> backward compatibility.
>
> Thanks for your attention,
> Jun Wang
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>

From status at bugs.python.org  Fri Nov 27 12:07:11 2015
From: status at bugs.python.org (Python tracker)
Date: Fri, 27 Nov 2015 18:07:11 +0100 (CET)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20151127170711.79B8256755@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2015-11-20 - 2015-11-27)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5263 ( +3)
  closed 32238 (+57)
  total  37501 (+60)

Open issues with patches: 2312 


Issues opened (41)
==================

#20923: Explain ConfigParser 'valid section name' and .SECTCRE
http://bugs.python.org/issue20923  reopened by terry.reedy

#25684: ttk.OptionMenu radiobuttons aren't unique between two instance
http://bugs.python.org/issue25684  opened by Bryan.Oakley

#25685: Inefficiency with SocketHandler - may send log record message 
http://bugs.python.org/issue25685  opened by Marvin Greenberg

#25687: Error during test case and tearDown
http://bugs.python.org/issue25687  opened by bennoleslie

#25690: Replacement for unittest.mock.mock_open
http://bugs.python.org/issue25690  opened by Niv Ben-David

#25694: test.libregrtest not installed
http://bugs.python.org/issue25694  opened by Arfrever

#25696: "make -j9 install" fails because bininstall target requires th
http://bugs.python.org/issue25696  opened by haypo

#25698: test regressions introduced with the fix for #22995
http://bugs.python.org/issue25698  opened by doko

#25699: Easier way to specify reduced globals for doctest
http://bugs.python.org/issue25699  opened by John.Mark.Vandenberg

#25701: Document that tp_setattro and tp_setattr are used for deleting
http://bugs.python.org/issue25701  opened by serhiy.storchaka

#25702: Link Time Optimizations support for GCC and CLANG
http://bugs.python.org/issue25702  opened by alecsandru.patrascu

#25704: Update the devguide to 3.5
http://bugs.python.org/issue25704  opened by matrixise

#25707: Add the close method for ElementTree.iterparse() object
http://bugs.python.org/issue25707  opened by serhiy.storchaka

#25708: runpy hides traceback for some exceptions
http://bugs.python.org/issue25708  opened by sleepycal

#25709: Problem with string concatenation and utf-8 cache.
http://bugs.python.org/issue25709  opened by ??rp??d K??sa

#25710: zipimport is not PEP 3147 or PEP 488 compliant
http://bugs.python.org/issue25710  opened by byrnes

#25711: Rewrite zipimport from scratch
http://bugs.python.org/issue25711  opened by brett.cannon

#25713: Setuptools included with 64-bit Windows installer is outdated
http://bugs.python.org/issue25713  opened by James.Paget

#25714: Consider isinstance(..., numbers.Integral) instead of isinstan
http://bugs.python.org/issue25714  opened by posita

#25715: Python 3.5.1 installer shows wrong upgrade path
http://bugs.python.org/issue25715  opened by steve.dower

#25716: typeobject.c call_method & call_maybe can leak references on '
http://bugs.python.org/issue25716  opened by Myron Walker

#25717: tempfile.TemporaryFile fails when dir option set to directory 
http://bugs.python.org/issue25717  opened by Hans Lawrenz

#25718: itertools.accumulate __reduce__/__setstate__ bug
http://bugs.python.org/issue25718  opened by arigo

#25719: Deprecate spitfire benchmark
http://bugs.python.org/issue25719  opened by florin.papa

#25720: Fix curses module compilation with ncurses6
http://bugs.python.org/issue25720  opened by donmez

#25723: ConfigParser should never write broken configurations
http://bugs.python.org/issue25723  opened by spaceone

#25724: SSLv3 test failure on Ubuntu 16.04 LTS
http://bugs.python.org/issue25724  opened by mdeslaur

#25726: sys.setprofile / sys.getprofile asymetry
http://bugs.python.org/issue25726  opened by stefan

#25728: email parser ignores inner multipart boundary when outer messa
http://bugs.python.org/issue25728  opened by forest

#25729: update pure python datetime.timedelta creation
http://bugs.python.org/issue25729  opened by bdkearns

#25730: invisible sidebar content with code snippets
http://bugs.python.org/issue25730  opened by xwhhsprings

#25731: Assigning and deleting __new__ attr on the class does not allo
http://bugs.python.org/issue25731  opened by Stanis??aw Skonieczny (Uosiu)

#25732: functools.total_ordering does not correctly implement not equa
http://bugs.python.org/issue25732  opened by David Seddon

#25733: Idle does not handle non-standard compile errors
http://bugs.python.org/issue25733  opened by ppperry

#25735: math.factorial doc should mention integer return type
http://bugs.python.org/issue25735  opened by John.Yeung

#25738: http.server doesn't handle RESET CONTENT status correctly
http://bugs.python.org/issue25738  opened by spaceone

#25739: Add PAYLOAD_TOO_LARGE / URI_TOO_LONG (new name in RFC 7231)
http://bugs.python.org/issue25739  opened by spaceone

#25740: multiple issues in http.client
http://bugs.python.org/issue25740  opened by spaceone

#25741: Usual Installation Directory
http://bugs.python.org/issue25741  opened by firatozgul

#25742: locale.setlocale does not work with unicode strings
http://bugs.python.org/issue25742  opened by tierlieb

#25743: Clarify exactly what \w matches in UNICODE mode
http://bugs.python.org/issue25743  opened by zwol



Most recent 15 issues with no replies (15)
==========================================

#25742: locale.setlocale does not work with unicode strings
http://bugs.python.org/issue25742

#25735: math.factorial doc should mention integer return type
http://bugs.python.org/issue25735

#25733: Idle does not handle non-standard compile errors
http://bugs.python.org/issue25733

#25726: sys.setprofile / sys.getprofile asymetry
http://bugs.python.org/issue25726

#25724: SSLv3 test failure on Ubuntu 16.04 LTS
http://bugs.python.org/issue25724

#25720: Fix curses module compilation with ncurses6
http://bugs.python.org/issue25720

#25713: Setuptools included with 64-bit Windows installer is outdated
http://bugs.python.org/issue25713

#25708: runpy hides traceback for some exceptions
http://bugs.python.org/issue25708

#25687: Error during test case and tearDown
http://bugs.python.org/issue25687

#25685: Inefficiency with SocketHandler - may send log record message 
http://bugs.python.org/issue25685

#25682: __package__ not set to None under pdb in Python 3
http://bugs.python.org/issue25682

#25675: doc for BaseEventLoop.run_in_executor() says its a coroutine, 
http://bugs.python.org/issue25675

#25661: tokenize.untokenize does not maintain the order of tabbed inde
http://bugs.python.org/issue25661

#25658: PyThread assumes pthread_key_t is an integer, which is against
http://bugs.python.org/issue25658

#25653: ctypes+callbacks+fork+selinux = crash
http://bugs.python.org/issue25653



Most recent 15 issues waiting for review (15)
=============================================

#25740: multiple issues in http.client
http://bugs.python.org/issue25740

#25739: Add PAYLOAD_TOO_LARGE / URI_TOO_LONG (new name in RFC 7231)
http://bugs.python.org/issue25739

#25738: http.server doesn't handle RESET CONTENT status correctly
http://bugs.python.org/issue25738

#25729: update pure python datetime.timedelta creation
http://bugs.python.org/issue25729

#25724: SSLv3 test failure on Ubuntu 16.04 LTS
http://bugs.python.org/issue25724

#25720: Fix curses module compilation with ncurses6
http://bugs.python.org/issue25720

#25719: Deprecate spitfire benchmark
http://bugs.python.org/issue25719

#25718: itertools.accumulate __reduce__/__setstate__ bug
http://bugs.python.org/issue25718

#25717: tempfile.TemporaryFile fails when dir option set to directory 
http://bugs.python.org/issue25717

#25716: typeobject.c call_method & call_maybe can leak references on '
http://bugs.python.org/issue25716

#25715: Python 3.5.1 installer shows wrong upgrade path
http://bugs.python.org/issue25715

#25709: Problem with string concatenation and utf-8 cache.
http://bugs.python.org/issue25709

#25704: Update the devguide to 3.5
http://bugs.python.org/issue25704

#25702: Link Time Optimizations support for GCC and CLANG
http://bugs.python.org/issue25702

#25701: Document that tp_setattro and tp_setattr are used for deleting
http://bugs.python.org/issue25701



Top 10 most discussed issues (10)
=================================

#25709: Problem with string concatenation and utf-8 cache.
http://bugs.python.org/issue25709  24 msgs

#25717: tempfile.TemporaryFile fails when dir option set to directory 
http://bugs.python.org/issue25717  13 msgs

#25723: ConfigParser should never write broken configurations
http://bugs.python.org/issue25723  11 msgs

#20923: Explain ConfigParser 'valid section name' and .SECTCRE
http://bugs.python.org/issue20923  10 msgs

#25660: tabs don't work correctly in python repl
http://bugs.python.org/issue25660  10 msgs

#25730: invisible sidebar content with code snippets
http://bugs.python.org/issue25730   7 msgs

#25677: Syntax error caret confused by indentation
http://bugs.python.org/issue25677   6 msgs

#25694: test.libregrtest not installed
http://bugs.python.org/issue25694   6 msgs

#25729: update pure python datetime.timedelta creation
http://bugs.python.org/issue25729   6 msgs

#15348: IDLE - shell becomes unresponsive if debugger windows is close
http://bugs.python.org/issue15348   5 msgs



Issues closed (51)
==================

#4744: asynchat documentation needs to be more precise
http://bugs.python.org/issue4744  closed by r.david.murray

#7990: xml.etree.cElementTree lacks full dir() on Element
http://bugs.python.org/issue7990  closed by serhiy.storchaka

#10131: deepcopying an xml.dom.minidom.Document generates an invalid X
http://bugs.python.org/issue10131  closed by serhiy.storchaka

#10942: xml.etree.ElementTree.tostring returns type bytes, expected ty
http://bugs.python.org/issue10942  closed by serhiy.storchaka

#13275: Recommend xml.etree for XML processing
http://bugs.python.org/issue13275  closed by serhiy.storchaka

#14044: IncompleteRead error with urllib2 or urllib.request -- fine wi
http://bugs.python.org/issue14044  closed by martin.panter

#14988: _elementtree: Raise ImportError when importing of pyexpat fail
http://bugs.python.org/issue14988  closed by serhiy.storchaka

#19687: Fixes for elementtree integer overflow
http://bugs.python.org/issue19687  closed by serhiy.storchaka

#21228: Missing enumeration of HTTPResponse Objects methods of urllib.
http://bugs.python.org/issue21228  closed by martin.panter

#22989: HTTPResponse.msg not as documented
http://bugs.python.org/issue22989  closed by martin.panter

#23914: pickle fails with SystemError
http://bugs.python.org/issue23914  closed by serhiy.storchaka

#24097: Use after free in PyObject_GetState
http://bugs.python.org/issue24097  closed by serhiy.storchaka

#24455: IDLE debugger causes crash if not quitted properly before next
http://bugs.python.org/issue24455  closed by terry.reedy

#24731: Incorrect assert in str_subtype_new
http://bugs.python.org/issue24731  closed by serhiy.storchaka

#24858: python3 -m test -ugui -v test_tk gives 3 failures under Debian
http://bugs.python.org/issue24858  closed by serhiy.storchaka

#25508: LogRecord attributes are not tuple, when logging only dict
http://bugs.python.org/issue25508  closed by python-dev

#25576: Remove ???Content-Type: application/x-www-form-urlencoded; cha
http://bugs.python.org/issue25576  closed by martin.panter

#25594: enum instance attribute access possible
http://bugs.python.org/issue25594  closed by python-dev

#25616: Extract OrderedDict tests into separate file
http://bugs.python.org/issue25616  closed by serhiy.storchaka

#25622: Enable ctypes.test.test_values.Win_ValuesTestCase for non-Wind
http://bugs.python.org/issue25622  closed by martin.panter

#25623: Keep the link to Python implementation of OrderedDict
http://bugs.python.org/issue25623  closed by serhiy.storchaka

#25624: shutil.make_archive makes invalid directory entries
http://bugs.python.org/issue25624  closed by serhiy.storchaka

#25625: "chdir" Contex manager for pathlib
http://bugs.python.org/issue25625  closed by serhiy.storchaka

#25626: Gzip fails for file over 2**32 bytes
http://bugs.python.org/issue25626  closed by martin.panter

#25646: Distutils and Windows SDK 7.1
http://bugs.python.org/issue25646  closed by JGoutin

#25656: multiprocessing.dummy: pool.map hangs on empty list
http://bugs.python.org/issue25656  closed by ebarry

#25657: virtualenv's activate does not update LD_LIBRARY_PATH
http://bugs.python.org/issue25657  closed by serge-sans-paille

#25663: Make rlcompleter avoid duplicate global names
http://bugs.python.org/issue25663  closed by martin.panter

#25668: Deadlock in logging caused by a possible race condition with "
http://bugs.python.org/issue25668  closed by vinay.sajip

#25669: unittest.assertEqual() on un-equal types inheriting from colle
http://bugs.python.org/issue25669  closed by michael.foord

#25678: int() from a buffer reads past the buffer boundaries
http://bugs.python.org/issue25678  closed by serhiy.storchaka

#25686: Don't use distutils in test_shutil
http://bugs.python.org/issue25686  closed by serhiy.storchaka

#25688: File leak in ElementTree.iterparse()
http://bugs.python.org/issue25688  closed by serhiy.storchaka

#25689: ftplib docs language fix
http://bugs.python.org/issue25689  closed by martin.panter

#25691: Crash on deleting Element attribute
http://bugs.python.org/issue25691  closed by serhiy.storchaka

#25692: input function outputs prompt to stderr
http://bugs.python.org/issue25692  closed by martin.panter

#25693: test_asyncio prints some messages
http://bugs.python.org/issue25693  closed by martin.panter

#25695: test___all__ and test_support alter execution environment
http://bugs.python.org/issue25695  closed by martin.panter

#25697: Fix rough alphabetical order in Misc/ACKS
http://bugs.python.org/issue25697  closed by serhiy.storchaka

#25700: namedtuple documentation
http://bugs.python.org/issue25700  closed by Laur Joost

#25703: test_sys is failed if standard stream is redirected in 2.7
http://bugs.python.org/issue25703  closed by serhiy.storchaka

#25705: Fix regex_compile benchmark crash
http://bugs.python.org/issue25705  closed by zach.ware

#25706: problems in library/base64.rst
http://bugs.python.org/issue25706  closed by serhiy.storchaka

#25712: Dead link in the PEP-3147
http://bugs.python.org/issue25712  closed by haypo

#25721: Fix pybench always_display error
http://bugs.python.org/issue25721  closed by haypo

#25722: Lib/ssl.py breaks certificate validation for wildcard domains,
http://bugs.python.org/issue25722  closed by christian.heimes

#25725: Memory leak on unpickling bogus data
http://bugs.python.org/issue25725  closed by serhiy.storchaka

#25727: os.startfile implementation for other OS'es besides Windows
http://bugs.python.org/issue25727  closed by r.david.murray

#25734: fnmatch regular expression can be improved
http://bugs.python.org/issue25734  closed by r.david.murray

#25736: smtplib can't send 8bit encoded utf-8 message
http://bugs.python.org/issue25736  closed by r.david.murray

#25737: array is not a Sequence
http://bugs.python.org/issue25737  closed by eryksun

From lac at openend.se  Fri Nov 27 12:45:18 2015
From: lac at openend.se (Laura Creighton)
Date: Fri, 27 Nov 2015 18:45:18 +0100
Subject: [Python-Dev] ACTIVITY SUMMARY (2015-11-20 - 2015-11-27)
Message-ID: <201511271745.tARHjIl3001173@fido.openend.se>

My mailer just barfed trying to read that summary.
The problem is that the mail comes out with the encoding:
Content-Type: text/plain; charset="us-ascii"
 but then wants to print 'http://bugs.python.org/issue25709  opened 
 by ??rp??d K??sa'

Can we change the encoding to utf-8 ?

Laura



From ethan at stoneleaf.us  Sat Nov 28 10:37:31 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 28 Nov 2015 07:37:31 -0800
Subject: [Python-Dev] Help with a book
In-Reply-To: <CAMis+RMi6_By1Krnk_dGFaXfT0fpSRifm597yuQ2n_TccN-LHw@mail.gmail.com>
References: <CAMis+RMi6_By1Krnk_dGFaXfT0fpSRifm597yuQ2n_TccN-LHw@mail.gmail.com>
Message-ID: <5659CA3B.20509@stoneleaf.us>

On 11/25/2015 12:31 PM, Jacob Zimmerman wrote:

> I appreciate any help anyone can give to assist in making this book as
> great as it can be.

I'm a pretty good editor, and have used descriptors quite a bit.  I'd be 
happy to help.

--
~Ethan~


From ethan at stoneleaf.us  Sat Nov 28 10:35:33 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 28 Nov 2015 07:35:33 -0800
Subject: [Python-Dev] crash bug in weakref_richcompare?
In-Reply-To: <d6e014cb9bc44d3093e1f157407dcfe7@AU01-EXCH1.devel.iress.com.au>
References: <d6e014cb9bc44d3093e1f157407dcfe7@AU01-EXCH1.devel.iress.com.au>
Message-ID: <5659C9C5.3090204@stoneleaf.us>

On 11/26/2015 06:06 PM, Luke Deller wrote:

> I have come across some dubious code in Objects/weakrefobject.c which looks like a bug to me, but wanted to run it past others.
>
> This was discovered from looking at crash dumps from a multithreaded python app (using Python 2.7.9, but the same weakref code exists in 3.5 and hg tip).
>
> The code that worries me is at the end of the "weakref_richcompare" function:
>
> return PyObject_RichCompare(PyWeakref_GET_OBJECT(self),
>                              PyWeakref_GET_OBJECT(other), op);
>
> At this point the code has established that the referents are still alive, and it is trying to compare the referents.  However it has not acquired a strong reference to the referents, so I think it is possible for one of them to be deleted half way through this comparison.  This can lead to a crash, because PyObject_RichCompare assumes that the PyObject*'s it was passed will remain usable for the duration of the call.
>
> The crashes I have seen involve data corruption consistent with one of these PyObject's being deleted and the memory reused for something else, eg:
>
> 00 python27!try_3way_compare+0x15 [objects\object.c @ 712]
> 01 python27!try_3way_to_rich_compare+0xb [objects\object.c @ 901]
> 02 python27!do_richcmp+0x2c [objects\object.c @ 935]
> 03 python27!PyObject_RichCompare+0x99 [objects\object.c @ 982]
> 04 python27!weakref_richcompare+0x6a [objects\weakrefobject.c @ 212]
>
> (In this example v->ob_type was 0x5f637865 which is ASCII "exc_", not a valid pointer at all)
>
> Other places in weakrefobject.c seem to have a similar weakness, eg in weakref_hash and weakref_repr.
>
> I have not been able to produce a small test case to exhibit this crash, but from this inspection of the code it looks like a bug - am I understanding this correctly?

Luke, go ahead and open an issue on the bug tracker [1] for this; email 
threads are too easily lost.

--
~Ethan~


[1] bugs.python.org


From 15834119171 at 163.com  Sat Nov 28 20:34:43 2015
From: 15834119171 at 163.com (maochongzhao)
Date: Sun, 29 Nov 2015 09:34:43 +0800
Subject: [Python-Dev] Can not pass configuration on mac os
Message-ID: <B3AAAE7A-31FF-43F7-BB01-1FC3C372E290@163.com>

Hi,

I am new here wishing to contribute to python. 

I tried to go through the tutorial for seting up Cpython, while I cannot pass the configuration. My system is Mac OS and I have installed the command tool of Xcode and hg. But on terminal it said that no such file or directory when I type ?./configure --with-pydebug?.  What was wrong with my action? How can I fix the issue?

This may a little question, but thanks for your help and tutoring!


Sincerely, 

Chongzhao Mao
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151129/0e1e4122/attachment.html>

From brett at python.org  Sat Nov 28 20:58:16 2015
From: brett at python.org (Brett Cannon)
Date: Sun, 29 Nov 2015 01:58:16 +0000
Subject: [Python-Dev] Can not pass configuration on mac os
In-Reply-To: <B3AAAE7A-31FF-43F7-BB01-1FC3C372E290@163.com>
References: <B3AAAE7A-31FF-43F7-BB01-1FC3C372E290@163.com>
Message-ID: <CAP1=2W6x=ERXz8DEGk9kDT-dYU7CXU-zdf7UVqQ555nk_eq4xQ@mail.gmail.com>

The best to get help is the tutor mailing list:
https://mail.python.org/mailman/listinfo/tutor . After that you can get
help contributing by reading https://docs.python.org/devguide/ .

On Sat, 28 Nov 2015, 18:53 maochongzhao <15834119171 at 163.com> wrote:

> Hi,
>
> I am new here wishing to contribute to python.
>
> I tried to go through the tutorial for seting up Cpython, while I cannot
> pass the configuration. My system is Mac OS and I have installed the
> command tool of Xcode and hg. But on terminal it said that no such file or
> directory when I type ?./configure --with-pydebug?.  What was wrong with
> my action? How can I fix the issue?
>
> This may a little question, but thanks for your help and tutoring!
>
>
> Sincerely,
>
> Chongzhao Mao
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151129/ee058603/attachment.html>

From stephen at xemacs.org  Fri Nov 27 02:30:10 2015
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 27 Nov 2015 16:30:10 +0900
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CADiSq7cxCOmsaWEpGPMqjhKtQL2BQowxG092A1bmvC8bAtFrcQ@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CADiSq7eKapqb2s4bk3Bs5SzZkFxo+Uo-2v_H5PqAC86ZF6Q2+w@mail.gmail.com>
 <CAP7+vJLBGk37pRA+Tpwi4Rw8SiQGUY0866_btJmAoWavEcjcEA@mail.gmail.com>
 <CAMpsgwa4p-z1Kj8p-o4_hM0vJSaQTL7eckzZeX3ZHUK0GYwK6A@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
 <201511241427.tAOERFJC028474@fido.openend.se>
 <56547C68.8060506@egenix.com>
 <20151125145714.2a02651d@limelight.wooz.org>
 <CAJ3HoZ0DKXRZ2DYQAm+m=xsYgbf+akC3JTfgFXhe9ZpP2dn2Bw@mail.gmail.com>
 <CADiSq7fReF2uAdzxxLMrwLTeHJLJxQNfMUupP5dnYtf6LU6BEw@mail.gmail.com>
 <20151126121510.32c9a500@anarchist.wooz.org>
 <CADiSq7fqG5wLMu-QjRneogKF8cmfTxc0cJeAwj8irZQ25D51gg@mail.gmail.com>
 <CADiSq7cxCOmsaWEpGPMqjhKtQL2BQowxG092A1bmvC8bAtFrcQ@mail.gmail.com>
Message-ID: <m2bnafrjb1.fsf@xemacs.org>

Nick Coghlan writes:

 > This is a significant rewrite that switches the PEP to a Standards
 > Track PEP proposing two new features for 2.7.12+: an
 > "ssl._verify_https_certificates()" configuration function, and a
 > "PYTHONHTTPSVERIFY" environment variable (although writing them
 > together like that makes me wonder if the latter should now be
 > "PYTHONVERIFYHTTPS" instead).

+1 on the same order of "words" in the function name and the
evnironment variable.  I tend to prefer
"ssl._https_verify_certificates" since these aren't "HTTPS
certificates".


From fabiofz at gmail.com  Mon Nov 30 06:02:12 2015
From: fabiofz at gmail.com (Fabio Zadrozny)
Date: Mon, 30 Nov 2015 09:02:12 -0200
Subject: [Python-Dev] Avoiding CPython performance regressions
Message-ID: <CANXBEFr8FDqVEiqoFbPoudtHq2V=PTot9z023O3o2O4uknY=Ww@mail.gmail.com>

Hi python-dev,

I've seen that on and off CPython had attempts to measure benchmarks over
time to avoid performance regressions (i.e.: https://speed.python.org), but
had nothing concrete so far, so, I ended up creating a hosted service for
that (https://www.speedtin.com) and I'd like to help in setting up a
structure to run the benchmarks from https://hg.python.org/benchmarks/ and
properly upload them to SpeedTin (if CPython devs are Ok with that) -- note
that I don't really have server to run the benchmarks, only to host the
data (but https://speed.python.org seems to indicate that such a server is
available...).

There's a sample report at:
https://www.speedtin.com/reports/1_CPython27x_Performance_Over_Time/ (it
has real data from running using the PyPy benchmarks as I only discovered
about the benchmarks from https://hg.python.org/benchmarks/ later on --
also, it doesn't seem to support Python 3 right now, so, it's probably not
that useful for the current Python dev, but it does have some nice insight
on CPython 2.7.x performance over time).

Later on, the idea is being able to compare across different Python
implementations which use the same benchmark set... (although that needs
other implementations to also post to the data to SpeedTin).

Note that uploading the data to SpeedTin should be pretty straightforward
(by using https://github.com/fabioz/pyspeedtin, so, the main issue would be
setting up o machine to run the benchmarks).

Best Regards,

Fabio
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151130/5ddee35e/attachment.html>

From rdmurray at bitdance.com  Mon Nov 30 08:52:39 2015
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 30 Nov 2015 08:52:39 -0500
Subject: [Python-Dev] Avoiding CPython performance regressions
In-Reply-To: <CANXBEFr8FDqVEiqoFbPoudtHq2V=PTot9z023O3o2O4uknY=Ww@mail.gmail.com>
References: <CANXBEFr8FDqVEiqoFbPoudtHq2V=PTot9z023O3o2O4uknY=Ww@mail.gmail.com>
Message-ID: <20151130135240.3E341251104@webabinitio.net>

On Mon, 30 Nov 2015 09:02:12 -0200, Fabio Zadrozny <fabiofz at gmail.com> wrote:
> Note that uploading the data to SpeedTin should be pretty straightforward
> (by using https://github.com/fabioz/pyspeedtin, so, the main issue would be
> setting up o machine to run the benchmarks).

Thanks, but Zach almost has this working using codespeed (he's still
waiting on a review from infrastructure, I think).  The server was not in
fact running; a large part of what Zach did was to get that server set up.
I don't know what it would take to export the data to another consumer,
but if you want to work on that I'm guessing there would be no objection.
And I'm sure there would be no objection if you want to get involved
in maintaining the benchmark server!

There's also an Intel project posted about here recently that checks
individual benchmarks for performance regressions and posts the results
to python-checkins.

--David

From sad2project at gmail.com  Mon Nov 30 12:23:51 2015
From: sad2project at gmail.com (Jacob Zimmerman)
Date: Mon, 30 Nov 2015 11:23:51 -0600
Subject: [Python-Dev] Help with a book
In-Reply-To: <CAMis+RMi6_By1Krnk_dGFaXfT0fpSRifm597yuQ2n_TccN-LHw@mail.gmail.com>
References: <CAMis+RMi6_By1Krnk_dGFaXfT0fpSRifm597yuQ2n_TccN-LHw@mail.gmail.com>
Message-ID: <CAMis+RMHW5Domwx8vbKX+50BGc_7WqFv7VsRbENoWZ9mRevdbQ@mail.gmail.com>

Thanks to everyone who has offered to look at and edit my book. I have
enough help now, so I won't be taking anymore requests.

On Wed, Nov 25, 2015 at 2:31 PM, Jacob Zimmerman <sad2project at gmail.com>
wrote:

> Hello Python Developers!
>
> I'm writing a book about descriptors, and I'm hoping to make it a
> comprehensive guide, covering just about everything there is to know about
> descriptors, including many tips to get past certain knowledge barriers and
> how to avoid certain pitfalls. I'm hoping for it to become the definitive
> guide for those who want to really understand descriptors. Raymond's
> Descriptor HowTo Guide is great, but focused. I want to expand on what that
> guide teaches.
>
> Anyway, I am currently reading through my initial rough draft and making
> corrections for a revised draft. Once I'm done with that, I'd like a few
> volunteers to read through it. First, to find spelling, grammatical, or
> consistency mistakes, though those should be few in number, but also to
> give input as to whether something seems too repetitive, if some more
> information should be given in a spot, and if I'm even blatantly wrong
> about anything (hopefully not).
>
> Anyone who helps will be able to receive a free ebook of the final copy as
> well as mention in the Special Thanks.
>
> I appreciate any help anyone can give to assist in making this book as
> great as it can be.
>
> Thanks,
> Jacob Zimmerman
>
> Twitter: @jacobz_20
> Blog: http://programmingideaswithjake.wordpress.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151130/940f1e88/attachment.html>

From david.c.stewart at intel.com  Mon Nov 30 12:33:05 2015
From: david.c.stewart at intel.com (Stewart, David C)
Date: Mon, 30 Nov 2015 17:33:05 +0000
Subject: [Python-Dev] Avoiding CPython performance regressions
In-Reply-To: <20151130135240.3E341251104@webabinitio.net>
References: <CANXBEFr8FDqVEiqoFbPoudtHq2V=PTot9z023O3o2O4uknY=Ww@mail.gmail.com>
 <20151130135240.3E341251104@webabinitio.net>
Message-ID: <5A37912B-B69D-452D-847E-EACBF8E5F3E4@intel.com>






On 11/30/15, 5:52 AM, "Python-Dev on behalf of R. David Murray" <python-dev-bounces+david.c.stewart=intel.com at python.org on behalf of rdmurray at bitdance.com> wrote:

>
>There's also an Intel project posted about here recently that checks
>individual benchmarks for performance regressions and posts the results
>to python-checkins.

The description of the project is at https://01.org/lp - Python results are indeed sent daily to python-checkins. (No results for Nov 30 and Dec 1 due to Romania National Day holiday!)

There is also a graphic dashboard at http://languagesperformance.intel.com/

Dave

From barry at python.org  Mon Nov 30 15:32:30 2015
From: barry at python.org (Barry Warsaw)
Date: Mon, 30 Nov 2015 15:32:30 -0500
Subject: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS
 verification backport guidance)
In-Reply-To: <CADiSq7cxCOmsaWEpGPMqjhKtQL2BQowxG092A1bmvC8bAtFrcQ@mail.gmail.com>
References: <CADiSq7e2NeAMJQpCgUQcAC6d1FrCuMxZL4ggC8cQWKTsGZSjUA@mail.gmail.com>
 <CADiSq7dQnjUZvA5gvfGSnSTv6fCke7VH_wi7GW2Z-PeXryQYCg@mail.gmail.com>
 <20151123210557.57b7e732@limelight.wooz.org>
 <CADiSq7et1yuyxDFw65dhZf0FK1YtTJ0Mn882Rsy2DoNGXois3A@mail.gmail.com>
 <CACac1F_e2MWWF1wHbSk4pXVoAHYDwbRbW__OfHtje76f-Gmhjw@mail.gmail.com>
 <CADiSq7dBpPpO8oeADKPbp9+D_SW1w+sBbHunbgvzVDBi4ffxuw@mail.gmail.com>
 <CACac1F-1K9Vs04H6uZf4USuFu0fsLYTYgjJBwTkO=1tztn2g+A@mail.gmail.com>
 <201511241427.tAOERFJC028474@fido.openend.se>
 <56547C68.8060506@egenix.com>
 <20151125145714.2a02651d@limelight.wooz.org>
 <CAJ3HoZ0DKXRZ2DYQAm+m=xsYgbf+akC3JTfgFXhe9ZpP2dn2Bw@mail.gmail.com>
 <CADiSq7fReF2uAdzxxLMrwLTeHJLJxQNfMUupP5dnYtf6LU6BEw@mail.gmail.com>
 <20151126121510.32c9a500@anarchist.wooz.org>
 <CADiSq7fqG5wLMu-QjRneogKF8cmfTxc0cJeAwj8irZQ25D51gg@mail.gmail.com>
 <CADiSq7cxCOmsaWEpGPMqjhKtQL2BQowxG092A1bmvC8bAtFrcQ@mail.gmail.com>
Message-ID: <20151130153230.2117981c@anarchist.wooz.org>

On Nov 27, 2015, at 04:04 PM, Nick Coghlan wrote:

>New draft pushed: https://hg.python.org/peps/rev/f602a47ea795
>
>This is a significant rewrite that switches the PEP to a Standards Track PEP
>proposing two new features for 2.7.12+: an "ssl._verify_https_certificates()"
>configuration function, and a "PYTHONHTTPSVERIFY" environment variable
>(although writing them together like that makes me wonder if the latter
>should now be "PYTHONVERIFYHTTPS" instead).

Thanks for this, and +1 on Stephen's suggested name change (which you've
already pushed).

Two comments: the PEP still describes the configuration file implementation.
Is this slated for 2.7.12 also?  If not, should it just be dropped from the
PEP?

I'd mildly prefer no default value for `enable` in
_https_verify_certificates().  I'd have preferred a keyword-only argument, but
of course this is Python 2.  Instead, I'd like to force passing True or False
(and document using `enable=True` or `enable=False`) and not rely on a default
argument.  But I'm only +0 on that detail.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151130/32329a14/attachment.sig>