Clarification of PEP 476 "opting out" section
Hi folks, This is just a note to highlight the fact that I tweaked the "Opting out" section in PEP 476 based on various discussions I've had over the past few months: https://hg.python.org/peps/rev/dfd96ee9d6a8 The notable changes: * the example monkeypatching code handles AttributeError when looking up "ssl._create_unverified_context", in order to accommodate older versions of Python that don't have PEP 476 implemented * new paragraph making it clearer that while the intended use case for the monkeypatching trick is as a workaround to handle environments where you *know* HTTPS certificate verification won't work properly (including explicit references to sitecustomize.py and Standard Operating Environments for Python), there's also a secondary use case in allowing applications to provide a system administrator controlled setting to globally disable certificate verification (hence the change to the example code) * new paragraph making it explicit that even though we've improved Python's default behaviour, particularly security sensitive applications should still provide their own context rather than relying on the defaults Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 30.04.2015 02:33, Nick Coghlan wrote:
Can we please make the monkeypatch a regular part of Python's site.py which can enabled via an environment variable, say export PYTHONHTTPSVERIFY=0. See http://bugs.python.org/issue23857 for the discussion. Esp. for Python 2.7.9 the default verification from PEP 476 is causing problems for admins who want to upgrade their Python installation without breaking applications using Python. They need an easy and official non-hackish way to opt-out from the PEP 476 default on a per application basis. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 30 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Thu, 30 Apr 2015 09:59:34 +0200 "M.-A. Lemburg" <mal@egenix.com> wrote:
-1 (already explained in the bug below).
See http://bugs.python.org/issue23857 for the discussion.
Regards Antoine.
On 30 Apr 2015 5:59 pm, "M.-A. Lemburg" <mal@egenix.com> wrote:
My current recommendation to the Red Hat Python team is to put together a draft PEP 394 style informational PEP to define a documented configuration file based approach that redistributors may choose to implement in order to provide a smoother transition path for their end users: https://bugzilla.redhat.com/show_bug.cgi?id=1173041#c8 The problem I identified with reusing the environment variable based approach that we used with hash randomisation is that "-E" turns the upstream default behaviour back on. That was acceptable for hash randomisation, but would be too limiting here (as the appropriate value for the HTTPS certificate verification configuration setting relates primarily to the general state of SSL/TLS certificate management in the environment where Python is being deployed, moreso than to the specific networked applications that are being run). I actually do think it would be good to have such a feature as a native part of Python 2.7 in order to provide a nicer "revert to the pre-PEP-476 behaviour" experience for Python 2.7 users (leaving the "there's no easy way to turn HTTPS certificate verification off globally" state of affairs to Python 3), but I don't currently have the time available to push for that against the "end users can't be trusted not to turn certificate verification off when they should be fixing their certificate management instead" perspective. Cheers, Nick.
On 07.05.2015 04:30, Nick Coghlan wrote:
We're currently working on a new release of eGenix PyRun and this will include Python 2.7.9. We do want to add such an env switch to disable the cert verification, so would like to know whether we can use PYTHONHTTPSVERIFY for this or not. We mainly need this to reenable simple use of self-signed certificates which 2.7.9 disables. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 08 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On 8 May 2015 6:52 pm, "M.-A. Lemburg" <mal@egenix.com> wrote:
That's a slightly misleading quotation of my post, as I'm opposed to the use of an environment variable for this, due to the fact that using the "-E" switch will then revert to the upstream default behaviour of verifying certificates, rather defeating the point of introducing the legacy infrastructure compatibility feature in the first place. A new informational PEP akin to PEP 394 that defines a config file location & contents for downstream redistributors that need a smoother transition plan for PEP 476 will let us handle this in a consistent way across redistributors that's also compatible with runtime use of the -E switch. Cheers, Nick.
On 08.05.2015 11:36, Nick Coghlan wrote:
Oh, sorry. I read your email implying that you are fine with the env var approach. I don't really see the issue with -E, though. It's well possible to internally set PYTHONHTTPSVERIFY=0 when -E is used to regain backwards compatibility per default for Python 2.7. Regarding the config file approach and letting distributions set their own defaults: I don't think it's a good idea to have one distribution default to verifying HTTPS certs via a global config file and another distribution do the opposite. Python itself should define the defaults to be used, not the distributions. The Python Linux distribution is too complex already due to the many different ways Python is installed on the systems. Not having to deal with this complexity was the main motivation for us to create eGenix PyRun, since it works the same on all Linux platforms and doesn't use any of the system wide installed Python interpreters, settings or packages (unless you tell it to).
Regardless of whether a global config file is a good idea or not, I don't think we can wait with 2.7.10 until a whole new PEP process has run through.
-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 08 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
Using an environment variable to disable a security feature sounds like an extremely bad idea. Environment variables are hidden state. Generally you don't know up front which values they will have when running an executable, and people don't think about inspecting them. This opens the door to mistakes (or even attacks) where certificate validation is disabled without the user knowing. Regards Antoine. On Fri, 08 May 2015 12:13:52 +0200 "M.-A. Lemburg" <mal@egenix.com> wrote:
On 8 May 2015 9:52 pm, "Antoine Pitrou" <solipsis@pitrou.net> wrote:
Yes, that's also a consideration. A config file lets us bring the full battery of Linux security tools (most obviously file system permissions, but also other systems like SELinux & AppArmor) to bear on controlling who (and what) has permission to change the default security settings. Cheers, Nick.
use of an environment variable for this, due to the fact that using
verification, the the transition 2015)
On 8 May 2015 8:14 pm, "M.-A. Lemburg" <mal@egenix.com> wrote:
verifying
As a result of the discussions around both PEP 466 and 476, I'm now firmly of the view that it's correct for the upstream Python community to assume the public internet as its standard operating environment when it comes to network security settings, and for those of us working for commercial redistributors to subsequently bear the cost of adapting from that upstream assumption to the different assumptions that may apply in the context of organisational intranets. That's also why I've come around to the view that an informational PEP with recommendations for redistributors, rather than an actual change to Python 2.7, is the right answer (at least initially) when it comes to providing a smoother transition plan for PEP 476 - the folks saying "it's a bad idea to make this easy to turn off" are *right* from the perspective of operating over the public internet, or with well designed internal SSL/TLS certificate management, it's just also a *necessary* idea (in my view) to help CIOs and other infrastructure leads responsibly and effectively manage the wide range of risks associated with internal infrastructure upgrades. Regards, Nick.
location 2015)
On 09.05.2015 02:29, Nick Coghlan wrote:
I don't agree. We've broken the contract that people had with Python 2.7 by introducing a major breakage in a patch level release very far down the line in 2.7.9, without providing an easy and official way to opt-out that does not involve hacking your installation. IMO, we should not fall for the somewhat arrogant view that we know better than all the Python users out there when it comes to running secure systems. By providing a way to intentionally switch off the new default, we do make people aware of the risks and that's good enough, while still maintaining the contract people rightly expect of patch level releases of Python. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 09 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Sun, May 10, 2015 at 4:13 AM, M.-A. Lemburg <mal@egenix.com> wrote:
Just as long as it's the sysadmin, and NOT some random attacker over the internet, who has the power to downgrade security. Environment variables can be attacked in various ways. ChrisA
On 10 May 2015 at 11:44, Chris Angelico <rosuav@gmail.com> wrote:
They can, and the bash fun was very good evidence of that. OTOH if someones environment is at risk, PATH and PYTHONPATH are already very effective attack vectors. -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud
On 10 May 2015 at 13:04, Robert Collins <robertc@robertcollins.net> wrote:
Right, which is why -E is an important existing hardening technique for protecting privileged services against local attackers. Isolated mode in Python 3.4+ is easier to use, but you can get a functional equivalent in Python 2 using: * running from a directory under /usr (Program Files on Windows) rather than your home directory (to protect against sys.path[0] based attacks) * running with -E (to protect against PYTHON* environment variable attacks) * running with -S (to protect against site.py and sitecustomize.py based attacks) * running with -s (to protect against hostile packages in the user site directory) That's how I came to the conclusion that adding a new environment variable to turn off a network security hardening feature isn't a good idea: * it significantly increases the attack surface area if you're *not* using -E when running a privileged service * it doesn't work at all if you *are* using -E when running a privileged service That was OK when we were dealing with the hash randomisation problem, mostly because the consequence of that vulnerability was "denial of service", and the question of whether or not hash randomisation caused problems came up on an application-by-application basis, rather than being related to the way an entire network environment was managed. The question becomes very different when the failure mode we're talking about is transparent interception of nominally confidential communication. Instead, we want a configuration file stored in a protected directory, such that for an attacker to modify it they *already* need to have achieved a local privilege escalation, in which case, they can just attack the system certificate store directly, rather than messing about downgrading Python's default HTTPS verification settings. In my case, I don't actually need the *feature itself* in upstream CPython, but I *would* like to have upstream CPython's blessing of the design as a recommendation to redistributors that need a capability like this to meet the needs of their end users. I've been talking about "someone" putting together a PEP to that effect, so given this discussion, I'll go ahead and do that myself, with Robert Kuska listed as co-author (since he came up with the general design I'm advocating for). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 10.05.2015 05:04, Robert Collins wrote:
If an attacker has access to the process environment, you're doomed anyway, so that's not really an argument for or against using environment variables :-) You'd just need to create a file os.py and point PYTHONPATH at it. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 11 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On 11 May 2015 at 18:04, M.-A. Lemburg <mal@egenix.com> wrote:
The core issue lies in managing the "user" vs "administrator" permissions split. Even for self-administered systems, it's recommended practice to run *without* administrative permissions normally, and only elevate when you need them. One of the things you're watching out for in such cases is that it shouldn't be possible for an attacker to make a change to the user environment, and have that propagate to have an effect on a process running with administrative access. One of the recommended hardening measures against that kind of attack vector is to *turn off* Python's environment variable processing when launching Python processes with administrative access. We didn't care about that in the hash randomisation case, as the compatibility concern there applied on a per application basis, and caring about hash order was technically a bug in its own right. By contrast, in the situation we're worried about for certificate verification compatibility, the issue is environmental: certificate management in many private intranets isn't yet to the same standard as that on the public internet, so administrators may have a valid reason for defaulting Python back to the old behaviour, and redistributors may feel obliged to provide an opt-in period prior to switching the default behaviour to opt-out. Having the new setting be ignored in Python processes run under a hardened configuration means that an environment variable based solution won't have the desired effect in providing that smoother migration path to the more hardened configuration. I've now written a draft "recommendations to redistributors" PEP for Robert's configuration file based design: https://www.python.org/dev/peps/pep-0493/ (exact file names & config setting names TBD) I wouldn't be opposed to seeing that as an upstream Python 2.7.x feature, but agreement amongst redistributors on using a file-based approach is the main outcome I'm looking for. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 11.05.2015 11:13, Nick Coghlan wrote:
The env var would not be read at Python startup time, only when loading the ssl module, so the -E switch would not have the effect of disabling it - unlike the hash seed logic, which is run (and has to be run) at Python startup time.
The Fastly cache seems to be having problems again. I only get: 503 Backend is unhealthy - Details: cache-fra1225-FRA 1431335851 2631441948
Can't we have both ? I don't think that we can wait for a whole PEP process to run through to fix this regression in 2.7.9. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 11 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On 11 May 2015 at 19:22, M.-A. Lemburg <mal@egenix.com> wrote:
I'd strongly advise against it, as we're deliberately increasing the attack surface area here by providing a potential path to carry out a downgrade attack on the HTTPS certificate verification by forcing it back to the old behaviour. The main existing environment variable based attack vector would be to manage to run a process with administrative privileges and SSL_CERT_DIR and/or SSL_CERT_FILE pointing to a certificate written to a user or world-writable directory by the attacker. Providing a "don't verify HTTPS" flag at the interpreter level would let an attacker skip the first step of writing the certificate file to disk somewhere, making a system compromise harder to detect. (An especially paranoid SSL implementation would disallow reading certs from locations that allow write access to non-administrative users, but I don't believe OpenSSL is that paranoid) By contrast, the configuration file shouldn't provide a new attack vector (or simplify any existing attack vector), as if you have the permissions needed to modify the config file, you likely also have the permissions needed to modify the system certificate store, and the latter is a *far* more interesting attack vector than a downgrade attack solely against Python. Thus the environment variable based off switch is neither necessary (as an administrator controlled configuration file can do the job), nor sufficient (it can't handle the -E switch), *and* it represents an increase in the attack surface area relative to a Python implementation without the capability.
I don't think that we can wait for a whole PEP process to run through
Matrix multiplication made it through the PEP process in 8 days. If we do this as a redistributor recommendation rather than attempting to get it into upstream Python 2.7, we could potentially propose you take on the role of BDFL-Delegate and mark it as Accepted as soon as the two of us agree on a common approach. The reason I think that's a reasonable way forward is because we already know there are folks opposed to making the change upstream. If the PEP just provides recommendations for redistributors that *do* decide to provide a "global off switch" to revert to the old behaviour, then the perspective of the folks opposed to the feature is respected by the fact that this is a feature some redistributors *may* choose to add to provide a smoother migration path to more secure default HTTPS handling, rather than something upstream provides by default. I assume the Debian, Ubuntu and Suse folks won't care, as they have all already decided against backporting the change to their long term support releases where the compatibility break would pose a problem (and I can certainly sympathise with that perspective given the dependency on backporting the PEP 466 SSL changes first, and the work involved in seeking consensus on a smoother migration path from the old default to the new one). It would be nice to hear from ActiveState, Enthought & Continuum Analytics as well, but if they don't chime in here, I don't see any particular need to go chasing them explicitly.
to fix this regression in 2.7.9.
We made the decision when PEP 476 was accepted that this change turned a silent security failure into a noisy one, rather than being a regression in its own right. PEP 493 isn't about disagreeing with that decision, it's about providing a smoother upgrade path in contexts where letting the security failure remain silent is deemed to be preferred in the near term. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I don't really agree that the decision to disable TLS is an environment one, it's really a per application decision. This is why I was against having some sort of global off switch for all of Python because just because one application needs it turned off doesn't mean you want it turned off for another Python application. You might have some script that is interacting with a custom internal server which doesn’t have a valid TLS certificate but then you also have pip* installed which is reaching out to PyPI and downloading code from the internet. You might want to disable TLS verification for the first but you almost certainly don't want it to disable TLS verification for the second one. In this regard I think that environment variables are somewhat better because they are far easier to set per application instead of in a way that affects every python program. Per application is the *right* scope for this setting, especially in a system where people may or may not realize what is written in Python and what isn't. I think it's absolutely wrong to give people a footgun in the terms of a switch that turns off all of Python's TLS verification when for many applications the fact they use Python is simply an implementation detail. That being said, since it's not being included in Python core and it's only some patch that some downstream's are going to apply I also don't really care that much because it's not going to effect me and if it turns out to be a bad idea and a footgun like I think it is, then the blame can rest on those downstreams and not us :) I'm also not a fan of the environment variable either really for a lot of the reasons you've outlined here. * Ignoring the fact that pip has (via requests/urllib3) worked around this deficiency in Python and isn't going to be affected by this configuration switch at all. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 11 May 2015 at 20:23, Donald Stufft <donald@stufft.io> wrote:
The scenario I'm interested in is the one where it *was* off globally (i.e. you were already running Python 2.7.8 or earlier) and you want to manage a global rollout of a new Python version that supports being configured to verify HTTPS certificates by default, while making the decision on whether or not to enable HTTPS certificate verification on a server-by-server basis, rather than having that decision be coupled directly to the rollout of the updated version of Python. I agree that the desired end state is where Python 3 is, and where upstream Python 2.7.9+ is, this is solely about how to facilitate folks getting from point A to point B without an intervening window of "I broke the world and now my boss is yelling at me about it" :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 11.05.2015 12:47, Nick Coghlan wrote:
I guess we're approaching this from different perspectives :-) I'm mostly interested in having a switch that can be set on a per application basis, not globally. I think the global default is fine and I'm just looking for a way to have admins disable it on a case-by-case basis for those applications which have problems with the new default. Hence the env var approach - the admin would simply edit the application's startup shell script, add the env var and that's it. For pip et al. which don't use the ssl module, the admins can simply continue using older versions for those applications - ones which don't implement the extra verification. In many cases, this is not necessary, since production environments typically don't use PyPI at all: they use a local directory with the needed distribution files, which is both more secure and reliable.
-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 11 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
Oh, another issue that I forgot to mention-- A fair number of people had no idea that Python wasn't validating TLS before 2.7.9/3.4.3 however as part of the processing of changing that in 2.7.9 a lot of people became aware that Python's before 2.7.9 didn't validate but that Python 2.7.9+ does. I worry that if Redhat (or anyone) ships a Python 2.7.9 that doesn't verify by default then they are going to be shipping something which defies the expectations of those users who were relying on the fact that Python 2.7.9+ was supposed to be secure by default now. You're (understandibly) focusing on "I already have my thing running on Python 2.7.8 and I want to yum update and get 2.7.9 and have things not visibly break", however there is the other use case of "I'm setting up a new environment, and I installed RHEL and got 2.7.9, I remembered reading in LWN that 2.7.9 verifies now so I must be safe". If you *do* provide such a switch, defaulting it to verify and having people where that breaks go in and turn it off is probably a safer mechanism since the cases where 2.7.9 verification breaks things for people is a visible change where the case that someone expects 2.7.9 to verify and it doesn't isn't a visible change and is easily missed unless they go out of their way to try and test it against a server with an invalid certificate. Either way, if there is some sort of global off switch, having that off switch set to off should raise some kind of warning (like urllib3 does if you use the unverified HTTPS methods). To be clear, I don't mean that using the built in ssl module APIs to disable verification should raise a warning, I mean the hypothetical "make my Python insecurely access HTTPS" configuration file (or environment variable) that is being proposed. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 11 May 2015 10:16 pm, "Robert Kuska" <rkuska@redhat.com> wrote: that
As Robert noted, it would be a matter of updating to a 2.7.5 with more patches backported, rather than rebasing to a newer upstream version. I can make the "do not change the default behaviour relative to the corresponding upstream version" guidance explicit in the PEP, though. Cheers, Nick.
On 11.05.2015 12:15, Nick Coghlan wrote:
Correct. OpenSSL will happily read the cert files from anywhere you point it to.
Whether or not -E will have an effect on the env var depends on the implementation. At the moment, -E only has an effect on the C runtime, while the stdlib happily reads from os.environ without taking the flag into account. As proposed, the PYTHONHTTPSVERIFY would only affect the ssl module and only be checked when loading this module, i.e. not at Python startup time.
I think the approach to only consider a subset of redistributors as viable targets for such a switch is a bit too narrow. You are leaving out all the parties which use custom Python installations to run their applications, e.g. the Plone and Zope community, the ZenOSS community, the many Windows applications built on Python, etc.
The change wasn't regression. The missing downgrade path is a regression. Some other comments on PEP 493: * I don't think we really want to add the overhead of having to parse an INI file every time Python starts up. Please remember that we just parsing of the sysconfig data not long ago because we wanted to avoid this startup time. * I don't see why the attack surface of using an INI file somewhere in the system should be smaller than e.g. using sitecustomize.py * If done right, we'd also need a switch to ignore this global config file and recommend using it to reduce the attack surface (for the same reason you explain in the PEP) * I don't think a global switch is the right way forward. Many applications on properly configured systems will work fine with the new default. The downgrade option is only needed for those cases, where they don't and you don't have a good way to fix the application. * Most applications use some kind of virtualenv Python environment to run the code. These are typically isolated from the system Python installation and so wouldn't want to use a system wide global INI neither. * The -S switch completely disables importing site.py. That's not really a viable solution in the age of pip - your local installation wouldn't find the installed packages anymore, since these are installed in site-packages/ which again, is set up by site.py. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 11 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On 12 May 2015 at 04:49, M.-A. Lemburg <mal@egenix.com> wrote:
I had an off-list discussion with Christian Heimes about that in relation to the OpenSSL flags, and he pointed out the reason -E specifically needs to be a command line switch is because that is the only way to affect how environment variables are processed during interpreter startup. Once an application is up and running, further environment variable sanitisation can be handled at an application level by whitelisting entries in os.environ and deleting everything else.
Right, the same is true for the configuration file proposal.
No, they already have a solution: monkeypatch (or just plain patch) the SSL module. That's an upstream supported technique, which is why it's documented in the PEP. The problem we (as in Red Hat) ran into was that that technique *doesn't work* for the case of backporting PEP 476 to Python 2.7.5 as an opt-in feature.
It's a shame we don't have "-X" options in Python 2, as that would be a nice hard-to-attack option (although it wouldn't play well with subprocesses)
Compared to the overhead of reading from the system cert database, reading a config file at ssl module import time should be trivial.
You can put sitecustomize.py in a user directory, and if there's no system wide sitecustomize, Python will read it automatically (unless user site directories are turned off).
No, the recommendation there would be to upgrade to a newer version of Python that doesn't offer this downgrade capability. It's a proposal for a transition smoothing technique, not a permanent capability (I did suggest the latter at one point, but the discussion on the issue tracker persuaded me that was a bad idea, with the increased attack surface being a key part of that change of heart).
And techniques like chroots and containers let you do that selectively. The key thing I'm after is an agreed technique for backporting to earlier 2.7.x releases that allows PEP 476 to be provided as an opt-in capability, rather than gating it on folks upgrading to 2.7.9, which isn't going to happen for *years* in a great many environments (Ubuntu 14.04 LTS, for example, doesn't go end of life until 2019 and ships 2.7.6 + non-intrusive security patches, while RHEL 7 doesn't go end of life until 2024 and ships 2.7.5 + compatible backports that address customer problems). While other Linux vendors have currently decided to leave the HTTPS problem unfixed in their long term support releases (due to the risk of causing service failures in customer environments), I'm hoping they may revisit those decisions if there's a specific technique already agreed with upstream for backporting the capability in a way that makes it an opt-in feature that customers can switch on independently of the inclusion of the feature backport in the system Python.
Having a separate configuration setting that controls verification in virtual environments is a good idea. That could also provide the per-application opt-out capability that you're after.
Yes, getting an administrative application to the point where -S can be used means getting it to a point where it has *no* Python dependencies outside the standard library. It can certainly be done, but often won't be worth the hassle. As a result, using -s to turn off the user site directory and -E to turn off PYTHONPATH processing are the more common sys.path related hardening techniques in Python 2.7. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 12.05.2015 05:03, Nick Coghlan wrote:
Again, this is not a proper solution for your friendly sys admin. They won't go in and patch Python, but rather consider it broken, if it doesn't provide a configuration option and simply stay with 2.7.8. We've had that discussion already, so I won't go into details again.
And neither does it work for people deploying Windows apps built on Python (where the code is usually hidden away in a ZIP archive or even compiled into a DLL), or people deploying Plone or ZenOSS. It's not only Red Hat's customers that are affected :-)
The cert database is only read when importing the ssl module, not with each Python startup, so that makes a big difference. The INI file would have to parsed at ssl module load time as well to work around this.
In a user based installation (which most applications shipping their own Python installation are), you can always do this provided you can gain the application user permissions. For a system installed Python, you'd always have to run Python with -E -S -s to avoid this, but then you'd also loose the system wide site-packages dir, so I'm not sure whether that's really an option.
You mean the config file approach would only exist for Python 2.7 ?
Nick, I lost you there. Are you suggesting that sys admins who want to upgrade to Python 2.7.9 and need the downgrade option should put the whole application into a container to work around a design deficiency in the downgrade option ?
Again, you're only seeing Red Hat's customers as target for this. Not everyone in this world runs on Red Hat, even if you'd probably like that :-) OS distributions can easily patch their system Python as needed. The point here is that sys admins should not have to patch Python to make things work again, in case an application is not prepared for the certificate verification - which is rather likely, since the pre-Python 2.7.9 doesn't even provide the necessary APIs to pass certificate locations to urllib or urllib2.
The env var would already enable virtualenv to set this up automatically, without any local config file. To cover all the different cases, you will inevitably have to provide a system wide config file, a user one and a local one, just like we have for distutils: https://docs.python.org/2/install/index.html#distutils-configuration-files
Overall, I find the config file approach a too big a hammer to solve this simple problem. If there were more use cases for a Python config file, the added overhead could pay off, but then we should put more thought into optimizing the config file load time and probably end up using a Python module as config file (which provides these optimizations for free). In the end, we'd be introducing another sitecustomize.py, usercustomize.py and perhaps localcustomize.py - only with fixed locations rather than importing them via sys.path. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 12 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On 12 May 2015 at 17:57, M.-A. Lemburg <mal@egenix.com> wrote:
You've persuaded me that we should describe *both* configuration mechanisms in the recommendations PEP (one for a cross-platform environment variable based approach that also works for embedded Python instances and user level distributions, one for a configuration file based approach that only covers *nix system Python installations), and leave the decision to redistributors as to which approach makes the most sense for their particular audience. Neither goes into upstream CPython 2.7, and neither goes into any version of Python 3.
The problem of managing this security issue on behalf of customers without risking breaking their world is in no way simple - hence why most redistributors lobbed it into the "too hard" basket instead, and why we've been discussing possible UX improvements with upstream on-and-off for a couple of months now :)
This is why the proposal is for an SSL specific configuration file, loaded only when the SSL module is imported.
It won't come to that, as Linux system package managers don't support any of these - that's what containers are for. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Of course, if the application is shipping it’s own Python then it has to actually do something to update to 2.7.9 and it can add it’s own option to disable TLS verification. I personally think that the application providing that option is the *right* way and all these other things are, at best, just temporary shims until the applications do that. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 12.05.2015 12:04, Donald Stufft wrote:
I still believe that requiring to monkeypatch Python is a very poor approach in terms of quality software design. We can do better and we should. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 12 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On 12 May 2015 at 21:09, Donald Stufft <donald@stufft.io> wrote:
If you control the app you don't need to do that. All relevant api accept the context parameter. The shims are only useful when you don't control the app. So an app shipping their own python doesn't fall under that.
I think the "bundled Python" scenario MAL is interested in is this one: 1. An application with a bundled CPython runtime is using the verification defaults 2. Upgraded the bundled Python to 2.7.9 3. Didn't provide new configuration settings to disable certificate verification 4. Is being upgraded in an environment where verifying certificates makes the app unusable for environmental reasons related to certificate management The PyRun single-file Python interpreter has a similar need, where some apps than ran fine under 2.7.8 will need a way to disable cert verification in 2.7.9+ on a per-application basis, *without* modifying the applications. Both of those make sense to me as cases where the environment variable based security downgrade approach is the "least bad" answer available, which is why I eventually agreed it should be one of the recommendations in the PEP. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 12 May 2015 at 21:17, Nick Coghlan <ncoghlan@gmail.com> wrote:
It occurs to me that the subtitle of PEP 493 could be "All software is terrible, but it's often a system administrator's job to make it run anyway" :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 12.05.2015 13:19, Nick Coghlan wrote:
+1 :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 12 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
Why is without modifying the app a reasonable goal? If Python is bundled with the app then you have direct control over when that upgrade happens, so you can delay the upgrade to 2.7.9 until your application which is bundling Python has the relevant switches. This is distinctly different from a situation like downstream distributors where the version of Python being provided is being provided by a group different than the person providing the application. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 12 May 2015 at 21:21, Donald Stufft <donald@stufft.io> wrote:
Because of the way redistribution works. MAL was right that I was thinking specifically in terms of the Linux distributor case, where we're the OS vendor, so we need a way to offer "off by default, opt-in on a per-server basis". Once I got past that perspective, I was able to figure out where he was coming from as someone that offers explicit support for the "redistribution for bundling" use case. When apps "bundle Python", what's usually happening is that they'll just bundle whatever version is used on the build server that does the bundling. If the app developer's testing all uses valid HTTPS certificates (or simply doesn't test HTTPS at all), they won't see any problems with the 2.7.9 upgrade, and hence will ship that along to their customers, where it may break if that customer's environment turns out to be relying on the lack of certificate verification in 2.7.8 and earlier. If that scenario happens with unmodified upstream 2.7.9, the redistributor has no workaround they can pass along to app developers to in turn pass on to customers - the app developer simply has to tell their customers to downgrade back to the previous release, and then issue an updated version with a configuration setting to disable HTTPS verification as fast as they can. Customers tend to get rather grouchy about things like that :) By contrast, if the redistributor for the bundled version of Python has injected PYTHONHTTPSVERIFY support, then the app developers can at least pass along "set PYTHONHTTPSVERIFY=0 in the environment" to their customers as an interim workaround until they get a release out the door with a proper configuration setting to control whether or not the app verifies certificates (assuming they don't decide the environment variable is a good enough workaround). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Honestly, this reads like "If the person bundling 2.7.9 with their app doesn't bother to pay attention to what 2.7.9 means then things might break", but that's hardly special to TLS, there are lots of things that change in a release that may end up breaking in certain cases. Looking at Marc-Andre's latest email though, it appears we're using bundling in a different context? I'm thinking of things like PyInstaller or such where you're distributing Python + your own App, but this appears to just be some third party tool is installing Python and an App? Ultimately, as long as it doesn't end up in upstream CPython the PEP can recommend any approach and I'm OK with it in the sense that it won't affect me. Though the PEP should be clear it's for 2.7 only and not 3.x or 2.8 if that ever gets reversed and we end up with a 2.8. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 12 May 2015 at 21:45, Donald Stufft <donald@stufft.io> wrote:
Cert verification is a special case, as cert management in intranets tends to rely on two things: * browser users just click through SSL security warnings * automated internal tools don't check certs at all Organisations with a specific interest in network security may have their systems in a happier state, but I wouldn't bet on them being a substantial minority of organisations, let alone a signficant majority. It's hard to overstate how big a mindset shift "getting intranet network security right isn't optional" represents in our industry, and it's going to take years for that attitude change to filter out through the later parts of the technology adoption curve.
I'm personally talking about both, as for a lot of folks bundling Python, the actual bundling process is handled by a third party tool, and a lot of application vendors aren't going to think about what happens where their app is run in an environment with bad certificate management.
Yep, I just pushed an update (https://hg.python.org/peps/rev/b395246d0af7) that adds MAL's environment variable based solution as an alternative recommendation, and that update includes this phrase: =================== These designs are being proposed as a recommendation for redistributors, rather than as new upstream features, as they are needed purely to support legacy environments migrating from older versions of Python 2.7. Neither approach is being proposed as an upstream Python 2.7 feature, nor as a feature in any version of Python 3 (whether published directly by the Python Software Foundation or by a redistributor). ===================
or 2.8 if that ever gets reversed and we end up with a 2.8.
There won't be a 2.8, so mentioning that these downstream modifications wouldn't be used on a release that is never going to happen would just confuse people :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 12.05.2015 13:21, Donald Stufft wrote:
Take a Plone Intranet as example: The unified installer downloads and installs Python 2.7 for you. As of Plone 4.3.3 the version is Python 2.7.6. Now say you are a sys admin and your Intranet users are affected by some bug in 2.7.6 which is fixed in 2.7.9. The natural approach would be to upgrade the bundled Python to 2.7.9. Because it's an Intranet and Plone is used to aggregate information from other systems which use self-signed certificates, you don't want to risk breaking your Plone installation and need a way to disable the cert checks. The best way to do this is by configuring the bundled Python to disable the checks, since you would not want to mess with the Plone application itself. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 12 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On 12 May 2015 at 20:56, M.-A. Lemburg <mal@egenix.com> wrote:
It's a deliberate design choice to actively discourage people from doing it - your "Ewww" reaction to monkeypatching is exactly the one we want. There's no technical reason for people to be bothered by it, since it's a documented and supported technique covered by the relevant PEP - it just so happens that the configuration being done is to switch a function alias between two different functions. Both of the recommended options I'm putting in the PEP (essentially the Red Hat design and the eGenix design, since we cover two different use cases) still adopt that same basic implementation model, they just provide ways for redistributors to move the configuration inside the SSL module itself if they decide it is in their users' interests for them to do so. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (8)
-
Antoine Pitrou
-
Chris Angelico
-
Donald Stufft
-
Ethan Furman
-
M.-A. Lemburg
-
Nick Coghlan
-
Robert Collins
-
Robert Kuska