PEP476: Enabling certificate validation by default
Hi all, I've just updated the PEP to reflect the API suggestions from Nick, and the fact that the necessary changes to urllib were landed. I think this is ready for pronouncement, Guido? Cheers, Alex
The PEP doesn't specify any of the API changes for Python 2.7. I feel it is necessary for the PEP to show a few typical code snippets using urllib in Python 2.7 and how one would modify these in order to disable the cert checking. There are also a few typos; especially this paragraph puzzled me: This will be acheived by adding a new ``ssl._create_default_https_context`` function, which is the same as ``ssl.create_default``. ``http.client`` can then replace it's usage of ``ssl._create_stdlib_context`` with the new ``ssl._create_default_https_context``. (1) spelling: it's achieved, not achieved (2) method name: it's ssl.create_default_context, not ssl.create_default (3) There's not enough whitespace (in the rendered HTML on legacy.python.org) before http.client -- I kept reading it as "... which is the same as ssl.create_default.http.client ..." (4) There's no mention of the Python 2 equivalent of http.client. Finally, it's kind of non-obvious in the PEP that this affects Python 2.7.X (I guess the one after the next) as well as 3.4 and 3.5. On Fri, Sep 19, 2014 at 9:53 AM, Alex Gaynor <alex.gaynor@gmail.com> wrote:
Hi all,
I've just updated the PEP to reflect the API suggestions from Nick, and the fact that the necessary changes to urllib were landed.
I think this is ready for pronouncement, Guido?
Cheers, Alex
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
Pushed a new version which I believe adresses all of these. I added an example of opting-out with urllib.urlopen, let me know if there's any other APIs you think I should show an example with. On Fri, Sep 19, 2014 at 3:06 PM, Guido van Rossum <guido@python.org> wrote:
The PEP doesn't specify any of the API changes for Python 2.7. I feel it is necessary for the PEP to show a few typical code snippets using urllib in Python 2.7 and how one would modify these in order to disable the cert checking.
There are also a few typos; especially this paragraph puzzled me:
This will be acheived by adding a new ``ssl._create_default_https_context`` function, which is the same as ``ssl.create_default``. ``http.client`` can then replace it's usage of ``ssl._create_stdlib_context`` with the new ``ssl._create_default_https_context``.
(1) spelling: it's achieved, not achieved
(2) method name: it's ssl.create_default_context, not ssl.create_default
(3) There's not enough whitespace (in the rendered HTML on legacy.python.org) before http.client -- I kept reading it as "... which is the same as ssl.create_default.http.client ..."
(4) There's no mention of the Python 2 equivalent of http.client.
Finally, it's kind of non-obvious in the PEP that this affects Python 2.7.X (I guess the one after the next) as well as 3.4 and 3.5.
On Fri, Sep 19, 2014 at 9:53 AM, Alex Gaynor <alex.gaynor@gmail.com> wrote:
Hi all,
I've just updated the PEP to reflect the API suggestions from Nick, and the fact that the necessary changes to urllib were landed.
I think this is ready for pronouncement, Guido?
Cheers, Alex
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084
On 20 September 2014 08:34, Alex Gaynor <alex.gaynor@gmail.com> wrote:
Pushed a new version which I believe adresses all of these. I added an example of opting-out with urllib.urlopen, let me know if there's any other APIs you think I should show an example with.
It would be worth explicitly stating the process global monkeypatching hack: import ssl ssl._create_default_https_context = ssl._create_unverified_context Adding that hack to sitecustomize allows corporate sysadmins that can update their standard operating environment more easily than they can fix invalid certificate infrastructure to work around the problem on behalf of their users. It also helps out users that will be able to deal with such broken infrastructure without updating each and every one of their scripts. It's deliberately ugly because it's a genuinely bad idea that folks should want to avoid using, but as a matter of practical reality, corporate IT departments are chronically understaffed, and often fully committed to fighting the crisis du jour, without sufficient time being available for regular infrastructure maintenance tasks. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
+1 on Nick's suggestion. (Might also mention that this is the reason why both functions should exist and have compatible signatures.) Also please, please, please add explicit mention of Python 2.7, 3.4 and 3.5 in the Abstract (for example in the 3rd paragraph of the abstract). On Fri, Sep 19, 2014 at 3:52 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 20 September 2014 08:34, Alex Gaynor <alex.gaynor@gmail.com> wrote:
Pushed a new version which I believe adresses all of these. I added an example of opting-out with urllib.urlopen, let me know if there's any other APIs you think I should show an example with.
It would be worth explicitly stating the process global monkeypatching hack:
import ssl ssl._create_default_https_context = ssl._create_unverified_context
Adding that hack to sitecustomize allows corporate sysadmins that can update their standard operating environment more easily than they can fix invalid certificate infrastructure to work around the problem on behalf of their users. It also helps out users that will be able to deal with such broken infrastructure without updating each and every one of their scripts.
It's deliberately ugly because it's a genuinely bad idea that folks should want to avoid using, but as a matter of practical reality, corporate IT departments are chronically understaffed, and often fully committed to fighting the crisis du jour, without sufficient time being available for regular infrastructure maintenance tasks.
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-- --Guido van Rossum (python.org/~guido)
Done and done. Alex On Fri, Sep 19, 2014 at 4:13 PM, Guido van Rossum <guido@python.org> wrote:
+1 on Nick's suggestion. (Might also mention that this is the reason why both functions should exist and have compatible signatures.)
Also please, please, please add explicit mention of Python 2.7, 3.4 and 3.5 in the Abstract (for example in the 3rd paragraph of the abstract).
On Fri, Sep 19, 2014 at 3:52 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 20 September 2014 08:34, Alex Gaynor <alex.gaynor@gmail.com> wrote:
Pushed a new version which I believe adresses all of these. I added an example of opting-out with urllib.urlopen, let me know if there's any other APIs you think I should show an example with.
It would be worth explicitly stating the process global monkeypatching hack:
import ssl ssl._create_default_https_context = ssl._create_unverified_context
Adding that hack to sitecustomize allows corporate sysadmins that can update their standard operating environment more easily than they can fix invalid certificate infrastructure to work around the problem on behalf of their users. It also helps out users that will be able to deal with such broken infrastructure without updating each and every one of their scripts.
It's deliberately ugly because it's a genuinely bad idea that folks should want to avoid using, but as a matter of practical reality, corporate IT departments are chronically understaffed, and often fully committed to fighting the crisis du jour, without sufficient time being available for regular infrastructure maintenance tasks.
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-- --Guido van Rossum (python.org/~guido)
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084
Nice. I just realized the release candidate for 3.4.2 is really close (RC1 Monday, final Oct 6, see PEP 429). What's your schedule for 3.4? I see no date for 2.7.9 yet (but that could just be that PEP 373 hasn't been updated). What about the Apple and Microsoft issues Christian pointed out? Regarding the approval process, I want to get this into 2.7 and 3.4, but I want it done right, and I'm not convinced that the implementation is sufficiently worked out. I don't want you to feel rushed, and I don't want you to feel that you can't start coding until the PEP is approved, but I also feel that I want to see more working code and some beta testing before it goes live. Perhaps I should just approve the PEP but separately get to approve the code? (Others will have to review it for correctness -- but I want to understand and review the API.) On Sat, Sep 20, 2014 at 8:54 AM, Alex Gaynor <alex.gaynor@gmail.com> wrote:
Done and done.
Alex
On Fri, Sep 19, 2014 at 4:13 PM, Guido van Rossum <guido@python.org> wrote:
+1 on Nick's suggestion. (Might also mention that this is the reason why both functions should exist and have compatible signatures.)
Also please, please, please add explicit mention of Python 2.7, 3.4 and 3.5 in the Abstract (for example in the 3rd paragraph of the abstract).
On Fri, Sep 19, 2014 at 3:52 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 20 September 2014 08:34, Alex Gaynor <alex.gaynor@gmail.com> wrote:
Pushed a new version which I believe adresses all of these. I added an example of opting-out with urllib.urlopen, let me know if there's any other APIs you think I should show an example with.
It would be worth explicitly stating the process global monkeypatching hack:
import ssl ssl._create_default_https_context = ssl._create_unverified_context
Adding that hack to sitecustomize allows corporate sysadmins that can update their standard operating environment more easily than they can fix invalid certificate infrastructure to work around the problem on behalf of their users. It also helps out users that will be able to deal with such broken infrastructure without updating each and every one of their scripts.
It's deliberately ugly because it's a genuinely bad idea that folks should want to avoid using, but as a matter of practical reality, corporate IT departments are chronically understaffed, and often fully committed to fighting the crisis du jour, without sufficient time being available for regular infrastructure maintenance tasks.
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-- --Guido van Rossum (python.org/~guido)
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084
-- --Guido van Rossum (python.org/~guido)
That sounds reasonable to me -- at this point I don't expect this to make it into 3.4.2; Nick has some working code on the ticket: http://bugs.python.org/issue22417 it's mostly missing documentation. Alex On Sat, Sep 20, 2014 at 9:46 AM, Guido van Rossum <guido@python.org> wrote:
Nice. I just realized the release candidate for 3.4.2 is really close (RC1 Monday, final Oct 6, see PEP 429). What's your schedule for 3.4? I see no date for 2.7.9 yet (but that could just be that PEP 373 hasn't been updated). What about the Apple and Microsoft issues Christian pointed out?
Regarding the approval process, I want to get this into 2.7 and 3.4, but I want it done right, and I'm not convinced that the implementation is sufficiently worked out. I don't want you to feel rushed, and I don't want you to feel that you can't start coding until the PEP is approved, but I also feel that I want to see more working code and some beta testing before it goes live. Perhaps I should just approve the PEP but separately get to approve the code? (Others will have to review it for correctness -- but I want to understand and review the API.)
On Sat, Sep 20, 2014 at 8:54 AM, Alex Gaynor <alex.gaynor@gmail.com> wrote:
Done and done.
Alex
On Fri, Sep 19, 2014 at 4:13 PM, Guido van Rossum <guido@python.org> wrote:
+1 on Nick's suggestion. (Might also mention that this is the reason why both functions should exist and have compatible signatures.)
Also please, please, please add explicit mention of Python 2.7, 3.4 and 3.5 in the Abstract (for example in the 3rd paragraph of the abstract).
On Fri, Sep 19, 2014 at 3:52 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 20 September 2014 08:34, Alex Gaynor <alex.gaynor@gmail.com> wrote:
Pushed a new version which I believe adresses all of these. I added an example of opting-out with urllib.urlopen, let me know if there's any other APIs you think I should show an example with.
It would be worth explicitly stating the process global monkeypatching hack:
import ssl ssl._create_default_https_context = ssl._create_unverified_context
Adding that hack to sitecustomize allows corporate sysadmins that can update their standard operating environment more easily than they can fix invalid certificate infrastructure to work around the problem on behalf of their users. It also helps out users that will be able to deal with such broken infrastructure without updating each and every one of their scripts.
It's deliberately ugly because it's a genuinely bad idea that folks should want to avoid using, but as a matter of practical reality, corporate IT departments are chronically understaffed, and often fully committed to fighting the crisis du jour, without sufficient time being available for regular infrastructure maintenance tasks.
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-- --Guido van Rossum (python.org/~guido)
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084
-- --Guido van Rossum (python.org/~guido)
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084
On 21 September 2014 03:05, Alex Gaynor <alex.gaynor@gmail.com> wrote:
That sounds reasonable to me -- at this point I don't expect this to make it into 3.4.2; Nick has some working code on the ticket: http://bugs.python.org/issue22417 it's mostly missing documentation.
I also think it's more sensible to target 2.7.9 & 3.4.3 for this change, especially given the remaining rough edges in custom trust database configuration on WIndows and Mac OS X that Christian pointed out in http://bugs.python.org/issue22449 I don't believe Benjamin has picked a specific date for 2.7.9 yet, but the regular maintenance release cadence (ignoring security releases) would put it some time in November, which should be sufficient time to get the remaining issues ironed out for 3.5 under the normal development process, and then included under the banner of PEP 476 for backporting to the maintenance branches. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Sounds good. Maybe we should put the specifically targeted releases in PEP 476? Nick, do Christian's issues need to be mentioned in the PEP or should we just keep those in the corresponding tracker items? On Sat, Sep 20, 2014 at 3:05 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 21 September 2014 03:05, Alex Gaynor <alex.gaynor@gmail.com> wrote:
That sounds reasonable to me -- at this point I don't expect this to make it into 3.4.2; Nick has some working code on the ticket: http://bugs.python.org/issue22417 it's mostly missing documentation.
I also think it's more sensible to target 2.7.9 & 3.4.3 for this change, especially given the remaining rough edges in custom trust database configuration on WIndows and Mac OS X that Christian pointed out in http://bugs.python.org/issue22449
I don't believe Benjamin has picked a specific date for 2.7.9 yet, but the regular maintenance release cadence (ignoring security releases) would put it some time in November, which should be sufficient time to get the remaining issues ironed out for 3.5 under the normal development process, and then included under the banner of PEP 476 for backporting to the maintenance branches.
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-- --Guido van Rossum (python.org/~guido)
On 21 September 2014 08:22, Guido van Rossum <guido@python.org> wrote:
Sounds good. Maybe we should put the specifically targeted releases in PEP 476?
Nick, do Christian's issues need to be mentioned in the PEP or should we just keep those in the corresponding tracker items?
They should be mentioned in the PEP, as they will impact the way the proposed change interacts with the platform trust database - I didn't realise the differences on Windows and Mac OS X myself until Christian mentioned them. To be completely independent of the system trust database in a reliable, cross-platform way, folks will need to use a custom SSL context that doesn't enable the system trust store, rather than relying on the OpenSSL config options - the latter will reliably *add* certificates, but they won't reliably ignore the default ones provided by the system. We may also need some clarification from Ned regarding the status of OpenSSL and the potential impact switching from dynamic linking to static linking of OpenSSL may have in terms of the "OPENSSL_X509_TEA_DISABLE" setting. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
OK, I'll hold off a bit on approving the PEP, but my intention is to approve it. Go Alex go! On Sat, Sep 20, 2014 at 4:03 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 21 September 2014 08:22, Guido van Rossum <guido@python.org> wrote:
Sounds good. Maybe we should put the specifically targeted releases in PEP 476?
Nick, do Christian's issues need to be mentioned in the PEP or should we just keep those in the corresponding tracker items?
They should be mentioned in the PEP, as they will impact the way the proposed change interacts with the platform trust database - I didn't realise the differences on Windows and Mac OS X myself until Christian mentioned them.
To be completely independent of the system trust database in a reliable, cross-platform way, folks will need to use a custom SSL context that doesn't enable the system trust store, rather than relying on the OpenSSL config options - the latter will reliably *add* certificates, but they won't reliably ignore the default ones provided by the system.
We may also need some clarification from Ned regarding the status of OpenSSL and the potential impact switching from dynamic linking to static linking of OpenSSL may have in terms of the "OPENSSL_X509_TEA_DISABLE" setting.
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-- --Guido van Rossum (python.org/~guido)
I see no reason to hold up this PEP's approval any longer, so I hereby approve PEP 476. It looks like a fair amount of work is still needed to backport this to Python 2.7 (and a smaller amount for 3.4) but I trust that this will all happen before the next releases of these two. Congrats Alex! On Fri, Oct 3, 2014 at 2:57 PM, Alex Gaynor <alex.gaynor@gmail.com> wrote:
Guido van Rossum <guido <at> python.org> writes:
OK, I'll hold off a bit on approving the PEP, but my intention is to
approve
it. Go Alex go!
A patch for the environmental variable overrides on Windows has landed; thanks Benjamin!
Alex
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
On 13 Oct 2014 08:58, "Guido van Rossum" <guido@python.org> wrote:
I see no reason to hold up this PEP's approval any longer, so I hereby
approve PEP 476. It looks like a fair amount of work is still needed to backport this to Python 2.7 (and a smaller amount for 3.4) but I trust that this will all happen before the next releases of these two. Congrats Alex! Huzzah! Thanks to everyone involved in getting this one through to acceptance. Cheers, Nick.
On Fri, Oct 3, 2014 at 2:57 PM, Alex Gaynor <alex.gaynor@gmail.com> wrote:
Guido van Rossum <guido <at> python.org> writes:
OK, I'll hold off a bit on approving the PEP, but my intention is to
it. Go Alex go!
A patch for the environmental variable overrides on Windows has landed;
approve thanks
Benjamin!
Alex
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
On 21.09.2014 01:03, Nick Coghlan wrote:
We may also need some clarification from Ned regarding the status of OpenSSL and the potential impact switching from dynamic linking to static linking of OpenSSL may have in terms of the "OPENSSL_X509_TEA_DISABLE" setting.
You may want to ask Hynek, too. He initially discovered the issue and made me aware how Apple is tying keychain into OpenSSL. You may find the code in: http://opensource.apple.com/source/OpenSSL098/OpenSSL098-35.1/src/crypto/x50... http://opensource.apple.com/source/OpenSSL098/OpenSSL098-35.1/src/crypto/x50... The TEA (Trust Evaluation Agent) switch is global and affects all SSL context instances. There is non canonical way to set the TEA flag for a single context or socket. /* -1: not set * 0: set to false * 1: set to true */ static tea_enabled = -1; void X509_TEA_set_state(int change) { tea_enabled = (change) ? 1 : 0; } int X509_TEA_is_enabled() { if (tea_enabled < 0) tea_enabled = (NULL == getenv(X509_TEA_ENV_DISABLE)); return tea_enabled != 0; }
On 09/20/2014 11:05 PM, Nick Coghlan wrote:
On 21 September 2014 03:05, Alex Gaynor <alex.gaynor@gmail.com> wrote:
That sounds reasonable to me -- at this point I don't expect this to make it into 3.4.2; Nick has some working code on the ticket: http://bugs.python.org/issue22417 it's mostly missing documentation. I also think it's more sensible to target 2.7.9 & 3.4.3 for this change, especially given the remaining rough edges in custom trust database configuration on WIndows and Mac OS X that Christian pointed out in http://bugs.python.org/issue22449
FWIW, I tagged 3.4.2rc1 about 24 hours ago, so it certainly didn't make it into that. I could live with it going into 3.4.2 final if the community was convinced that was the right thing to do. But I'm more than happy about the conservative turn the plans have taken. If this isn't going in until 3.4.3, does that mean we need 3.4.3 on any sort of particular schedule? //arry/
On 19.09.2014 18:53, Alex Gaynor wrote:
Hi all,
I've just updated the PEP to reflect the API suggestions from Nick, and the fact that the necessary changes to urllib were landed.
I think this is ready for pronouncement, Guido?
There is still the issue with SSL_CERT_DIR and SSL_CERT_FILE on Windows and Apple's OpenSSL builds on OSX. I've opened a bug report http://bugs.python.org/issue22449 tl;dr On Windows SSL_CERT_DIR and SSL_CERT_FILE are simply ignored by SSLContext.load_verify_locations. On OSX Apple's Trust Evaluation Agent adds certs behind the scene.
participants (5)
-
Alex Gaynor -
Christian Heimes -
Guido van Rossum -
Larry Hastings -
Nick Coghlan