From davidc at enthought.com Wed Mar 1 09:59:35 2017 From: davidc at enthought.com (David Cournapeau) Date: Wed, 1 Mar 2017 14:59:35 +0000 Subject: [Security-sig] Patching ssl.py to workaround ssl lack of relocability Message-ID: Hi there, This is hopefully the right mailing for this question. We are shipping our own python interpreter in our product, and following some discussions on https://mail.python.org/pipermail/python-dev/2017-January/147282.html, we understand shipping the certificate from certifi in our python is the best approach on Linux/OS X. Unfortunately, ssl hardcodes at compilation time the default location of certificate. I could workaround this at the python level by patching ssl.SSLContext.load_default_certs to look as follows: def load_default_certs(...): .... if sys.platform == "win32": ... else: prefix = os.path.normpath(sys.prefix) default_cert = os.path.join(prefix, "ssl", "cert.pem") if os.path.isfile(default_cert): self.load_verify_locations(default_cert) else: self.set_default_verify_paths() While this seems to work, my lack of knowledge in all things related to security and ssl in particular makes me worry to patch anything in there. Is this a sane approach ? If not, is there a better way ? Thank you, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Wed Mar 1 10:51:57 2017 From: christian at python.org (Christian Heimes) Date: Wed, 1 Mar 2017 16:51:57 +0100 Subject: [Security-sig] Patching ssl.py to workaround ssl lack of relocability In-Reply-To: References: Message-ID: <2e0607a5-a75b-cbe1-8e29-0b4e057b1526@python.org> On 2017-03-01 15:59, David Cournapeau wrote: > Hi there, > > This is hopefully the right mailing for this question. We are shipping > our own python interpreter in our product, and following some > discussions on > https://mail.python.org/pipermail/python-dev/2017-January/147282.html, > we understand shipping the certificate from certifi in our python is the > best approach on Linux/OS X. Yeah, it's the easiest option for you. I don't necessarily agree it's the best option. > Unfortunately, ssl hardcodes at compilation time the default location of > certificate. I could workaround this at the python level by patching > ssl.SSLContext.load_default_certs to look as follows: Small correction: The location of the certs is hard-coded in OpenSSL. The ssl module simply uses OpenSSL's defaults on non-Windows platforms. > def load_default_certs(...): > .... > > if sys.platform == "win32": > ... > else: > prefix = os.path.normpath(sys.prefix) > default_cert = os.path.join(prefix, "ssl", "cert.pem") > if os.path.isfile(default_cert): > self.load_verify_locations(default_cert) > else: > self.set_default_verify_paths() > > While this seems to work, my lack of knowledge in all things related to > security and ssl in particular makes me worry to patch anything in > there. Is this a sane approach ? If not, is there a better way ? You can override the default verify paths already. There is no need to monkey patch load_default_certs(). You have two options: 1) Create your own custom SSLContext with a custom trust anchor, e.g. ctx = ssl.create_default_context(cafile='...') and pass the context along. 2) Set SSL_CERT_FILE env var to override OpenSSL's default setting, see https://docs.python.org/3/library/ssl.html#ssl.get_default_verify_paths The second option may work for you. You can set the env var at any time before you create a new SSLContext object. Christian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: OpenPGP digital signature URL: From davidc at enthought.com Wed Mar 1 11:44:40 2017 From: davidc at enthought.com (David Cournapeau) Date: Wed, 1 Mar 2017 16:44:40 +0000 Subject: [Security-sig] Patching ssl.py to workaround ssl lack of relocability In-Reply-To: <2e0607a5-a75b-cbe1-8e29-0b4e057b1526@python.org> References: <2e0607a5-a75b-cbe1-8e29-0b4e057b1526@python.org> Message-ID: On Wed, Mar 1, 2017 at 3:51 PM, Christian Heimes wrote: > On 2017-03-01 15:59, David Cournapeau wrote: > > Hi there, > > > > This is hopefully the right mailing for this question. We are shipping > > our own python interpreter in our product, and following some > > discussions on > > https://mail.python.org/pipermail/python-dev/2017-January/147282.html, > > we understand shipping the certificate from certifi in our python is the > > best approach on Linux/OS X. > > Yeah, it's the easiest option for you. I don't necessarily agree it's > the best option. > We are certainly interested in a better option ! Our constraints are: 1. we cannot control where python actually get installed 2. on Linux, we cannot afford to have distribution-specific builds (we use something similar to the manylinux setup) > > Unfortunately, ssl hardcodes at compilation time the default location of > > certificate. I could workaround this at the python level by patching > > ssl.SSLContext.load_default_certs to look as follows: > > Small correction: The location of the certs is hard-coded in OpenSSL. > The ssl module simply uses OpenSSL's defaults on non-Windows platforms. > > > def load_default_certs(...): > > .... > > > > if sys.platform == "win32": > > ... > > else: > > prefix = os.path.normpath(sys.prefix) > > default_cert = os.path.join(prefix, "ssl", "cert.pem") > > if os.path.isfile(default_cert): > > self.load_verify_locations(default_cert) > > else: > > self.set_default_verify_paths() > > > > While this seems to work, my lack of knowledge in all things related to > > security and ssl in particular makes me worry to patch anything in > > there. Is this a sane approach ? If not, is there a better way ? > > You can override the default verify paths already. There is no need to > monkey patch load_default_certs(). You have two options: > > 1) Create your own custom SSLContext with a custom trust anchor, e.g. > ctx = ssl.create_default_context(cafile='...') and pass the context along. > > 2) Set SSL_CERT_FILE env var to override OpenSSL's default setting, see > https://docs.python.org/3/library/ssl.html#ssl.get_default_verify_paths > > The second option may work for you. You can set the env var at any time > before you create a new SSLContext object. > > Just to clarify: I am aware that for code we write/distribute, there are better ways (we tend to always use requests for http(s) handling). But as we are distributing python as a distribution, the goal is to make things work by default for 3rd party users, without compromising security, so 1. is not applicable. 2. is not a good solution in our opinion because it may cause trouble when people use programs linked against openssl through subprocess. David Christian > > > _______________________________________________ > Security-SIG mailing list > Security-SIG at python.org > https://mail.python.org/mailman/listinfo/security-sig > > -- blog: http://cournape.wordpress.com code: https://github.com/cournape twitter: @cournape -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Mar 2 02:06:01 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 2 Mar 2017 17:06:01 +1000 Subject: [Security-sig] Patching ssl.py to workaround ssl lack of relocability In-Reply-To: References: <2e0607a5-a75b-cbe1-8e29-0b4e057b1526@python.org> Message-ID: On 2 March 2017 at 02:44, David Cournapeau wrote: > Just to clarify: I am aware that for code we write/distribute, there are > better ways (we tend to always use requests for http(s) handling). > Christian may have missed the domain on your email address, so calling out the redistribution context explicitly: https://www.enthought.com/products/epd/ And yes, since the Linux distros can't agree on a reliable way for third party applications to find the system certificate store without distro-specific patches, I'd agree that bundling certifi and patching your Python builds is your best currently available option for getting good "out of the box" behaviour. Donald tried to make location autodetection work for pip, but the distros unfortunately not only can't agree on how the default certs should be located, they also don't make sure the other potential locations reliable give a detectable error :( Given your context of use though, the one potential incompatibility you're going to have to watch out for is losing access to any custom CA certificates that are installed into the system trust stores (since certifi won't have any knowledge of those). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidc at enthought.com Thu Mar 2 04:41:42 2017 From: davidc at enthought.com (David Cournapeau) Date: Thu, 2 Mar 2017 09:41:42 +0000 Subject: [Security-sig] Patching ssl.py to workaround ssl lack of relocability In-Reply-To: References: <2e0607a5-a75b-cbe1-8e29-0b4e057b1526@python.org> Message-ID: On Thu, Mar 2, 2017 at 7:06 AM, Nick Coghlan wrote: > > > Given your context of use though, the one potential incompatibility you're > going to have to watch out for is losing access to any custom CA > certificates that are installed into the system trust stores (since certifi > won't have any knowledge of those). > > That's indeed a cons of this approach, at least on Linux. My understanding is that the PEP worked on by Cory Benfield would improve this. We will use this hack for now, keeping a look on the progress being made David -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Mar 9 12:37:50 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 9 Mar 2017 18:37:50 +0100 Subject: [Security-sig] Archives (.tar or .zip) with absolute paths Message-ID: Hi, I noticed that "python3 -m tarfile -x archive.tar" uses absolute paths by default, whereas the UNIX tar command doesn't by default. The UNIX tar command requires to add explicitly --absolute-paths (-P) option. The tarfile and zipfile modules (maybe also some others, I didn't check) contain warnings absolute paths and paths containing "..". Why not ignoring "/" at start of filenames *by default*? By backward compatibility? I suggest to add a boolean absolute_path option to tarfile and zipfile and disable it by default in the CLI. The question is what should be the default value for the Python API. I suggest to use absolute_path=False by default for safety. Example to create such archive. See that tar also removes "/" by default and requires to pass explicitly -P: $ cd $HOME # /home/haypo $ echo TEST > test $ tar -cf test.tar /home/haypo/test tar: Removing leading `/' from member names $ rm -f test.tar $ tar -P -cf test.tar /home/haypo/test $ rm -f test Extracting such archive using tar is safe *by default*: $ mkdir z $ cd z $ tar -xf ~/test.tar tar: Removing leading `/' from member names $ find . ./home ./home/haypo ./home/haypo/test Extracting such archive using Python is unsafe: $ python3 -m tarfile -e ~/test.tar $ cat ~/test TEST $ pwd /home/haypo/z Python creates files outside the current directory which is unsafe, wheras tar doesn't. Victor From wes.turner at gmail.com Thu Mar 9 14:33:44 2017 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 9 Mar 2017 13:33:44 -0600 Subject: [Security-sig] Archives (.tar or .zip) with absolute paths In-Reply-To: References: Message-ID: CWE-22: Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal') https://cwe.mitre.org/top25/#CWE-22 http://cwe.mitre.org/data/definitions/22.html - [ ] BUG,SEC: -P/--absolute-names *13* *CWE-22 : Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal')* CWE CATEGORY 21: Pathname Traversal and Equivalence Errors https://cwe.mitre.org/data/definitions/21.html https://cwe.mitre.org/data/definitions/21.html#Relationships ... CWE 59: "Improper Link Resolution Before File Access ('Link Following')" https://cwe.mitre.org/data/definitions/59.html - [ ] ? BUG,SEC: symlinks? Docs: https://docs.python.org/3/library/tarfile.html - https://github.com/python/cpython/blob/2.6/Doc/library/tarfile.rst - https://github.com/python/cpython/blob/3.6/Doc/library/tarfile.rst - https://github.com/python/cpython/blob/master/Doc/library/tarfile.rst Src: https://github.com/python/cpython/blob/master/Lib/tarfile.py - https://github.com/python/cpython/blob/2.6/Lib/tarfile.py - https://github.com/python/cpython/blob/3.6/Lib/tarfile.py https://www.python.org/news/security/#reporting-security-issues-in-python - https://docs.python.org/devguide/documenting.html#security-considerations-and-other-concerns - https://cpython-devguide.readthedocs.io/documenting.html#security-considerations-and-other-concerns - [o] email to: security at python.org - [ ] Create an issue: https://bugs.python.org/ - [ ] Create a pull request: https://docs.python.org/devguide/pullrequest.html - [ ] DOC: - [ ] BUG,ENH,SEC: `python -m tarfile -P/--absolute-names` - [ ] BUG,ENH,SEC: `python -m tarfile -h/--dereference --hard-dereference` - [ ] bandit test(s): https://github.com/openstack/bandit#writing-tests - [ ] Python API - [ ] - [ ] python -m tarfile - [ ] someone to lead on this because I am actually looking for a job ... https://python-security.readthedocs.io/ - Src: https://github.com/haypo/python-security/blob/master/index.rst - [ ] https://github.com/ebranca/owasp-pysec/wiki - [ ] https://github.com/ebranca/owasp-pysec/wiki/Security-Concerns-in-modules-and-functions On Thu, Mar 9, 2017 at 11:37 AM, Victor Stinner wrote: > Hi, > > I noticed that "python3 -m tarfile -x archive.tar" uses absolute paths > by default, whereas the UNIX tar command doesn't by default. The UNIX > tar command requires to add explicitly --absolute-paths (-P) option. > > The tarfile and zipfile modules (maybe also some others, I didn't > check) contain warnings absolute paths and paths containing "..". > > Why not ignoring "/" at start of filenames *by default*? By backward > compatibility? > > I suggest to add a boolean absolute_path option to tarfile and zipfile > and disable it by default in the CLI. The question is what should be > the default value for the Python API. I suggest to use > absolute_path=False by default for safety. > > > Example to create such archive. See that tar also removes "/" by > default and requires to pass explicitly -P: > > $ cd $HOME > # /home/haypo > $ echo TEST > test > $ tar -cf test.tar /home/haypo/test > tar: Removing leading `/' from member names > > $ rm -f test.tar > $ tar -P -cf test.tar /home/haypo/test > $ rm -f test > > > Extracting such archive using tar is safe *by default*: > > $ mkdir z > $ cd z > $ tar -xf ~/test.tar > tar: Removing leading `/' from member names > $ find > . > ./home > ./home/haypo > ./home/haypo/test > > > Extracting such archive using Python is unsafe: > > $ python3 -m tarfile -e ~/test.tar > $ cat ~/test > TEST > $ pwd > /home/haypo/z > > Python creates files outside the current directory which is unsafe, > wheras tar doesn't. > > Victor > _______________________________________________ > Security-SIG mailing list > Security-SIG at python.org > https://mail.python.org/mailman/listinfo/security-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Mar 9 16:44:35 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 9 Mar 2017 22:44:35 +0100 Subject: [Security-sig] Archives (.tar or .zip) with absolute paths In-Reply-To: References: Message-ID: Hi, I'm sorry Wes, but I don't understand your long list of urls :-( Can you elaborate? I'm asking if there is a reason for allowing absolute paths by default. Maybe backward compatibility? 2017-03-09 20:33 GMT+01:00 Wes Turner : > Docs: https://docs.python.org/3/library/tarfile.html I didn't write a private email to security@ because as you pointed, the issue is known and *documented* in Python since 10 years. > https://python-security.readthedocs.io/ I wrote this doc :-) I just added notes about tarfile and zipfile. Victor From njs at pobox.com Thu Mar 9 17:40:17 2017 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 9 Mar 2017 14:40:17 -0800 Subject: [Security-sig] Archives (.tar or .zip) with absolute paths In-Reply-To: References: Message-ID: On Thu, Mar 9, 2017 at 9:37 AM, Victor Stinner wrote: > Hi, > > I noticed that "python3 -m tarfile -x archive.tar" uses absolute paths > by default, whereas the UNIX tar command doesn't by default. The UNIX > tar command requires to add explicitly --absolute-paths (-P) option. > > The tarfile and zipfile modules (maybe also some others, I didn't > check) contain warnings absolute paths and paths containing "..". > > Why not ignoring "/" at start of filenames *by default*? By backward > compatibility? > > I suggest to add a boolean absolute_path option to tarfile and zipfile > and disable it by default in the CLI. The question is what should be > the default value for the Python API. I suggest to use > absolute_path=False by default for safety. This sounds like the right default to me. Technically there's some backwards compatibility risk, but tarfiles and zipfiles with absolute paths are really rare and this behavior is really dangerous (unpacking a file gives the person who created the file the ability to overwrite arbitrary files on your computer), so it's unlikely that much will break and what does break was likely a huge security hole in the first place. -n -- Nathaniel J. Smith -- https://vorpus.org From victor.stinner at gmail.com Thu Mar 9 19:06:49 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 10 Mar 2017 01:06:49 +0100 Subject: [Security-sig] HTML page of Python security vulnerabilities In-Reply-To: References: Message-ID: Hi, Minor update on http://python-security.readthedocs.io/vulnerabilities.html : I enhanced render_doc.py script to download issue title, author and date from bugs.python.org. It allows to remove more lines from vulnerabilities.yaml, so each YAML entry is now shorter and human mistakes are less likely! Note: Sadly, it seems like Roundup XML-RPC API requires to pass a user + password in the URL to get the author of the first message of an issue, whereas this information is public if you look at the HTML page. Victor 2017-02-22 1:11 GMT+01:00 Victor Stinner : > I completed my list: the 30 CVE are now listed on my page! Well, > except of two special cases: > > * CVE-2016-1494: vulnerability in the 3rd party module "python-rsa" > * CVE-2015-5652: sys.path on Windows -- not fixed > > See also my notes on sys.path: > http://python-security.readthedocs.io/#misc > > > The last major vulnerability not documented yet is cookielib which has > a long story. I don't know yet how to summarize it as individual > "vulnerabilities". > > https://hackerone.com/reports/26647 > > https://bugs.python.org/issue16611 > #16611: BaseCookie now parses 'secure' and 'httponly' flags. > https://bugs.python.org/issue22796 > Regression in Python 3.2 cookie parsing > https://bugs.python.org/issue25228 > Support for httponly/secure cookies reintroduced lax parsing behavior > https://code.djangoproject.com/ticket/26158 > cookie parsing fails with python 3.x if request contains unnamed cookie > > Victor From wes.turner at gmail.com Thu Mar 9 19:52:42 2017 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 9 Mar 2017 18:52:42 -0600 Subject: [Security-sig] Archives (.tar or .zip) with absolute paths In-Reply-To: References: Message-ID: On Thursday, March 9, 2017, Victor Stinner wrote: > Hi, > > I'm sorry Wes, but I don't understand your long list of urls :-( Can > you elaborate? I thought that's what I was doing? > > I'm asking if there is a reason for allowing absolute paths by > default. Maybe backward compatibility? I think secure by default would be good here. > > > 2017-03-09 20:33 GMT+01:00 Wes Turner > >: > > Docs: https://docs.python.org/3/library/tarfile.html > > I didn't write a private email to security@ because as you pointed, > the issue is known and *documented* in Python since 10 years. Doesn't mean it's not broken > > > > https://python-security.readthedocs.io/ > > I wrote this doc :-) I just added notes about tarfile and zipfile. The [ ] wiki links could also be useful > > Victor > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Mar 9 20:03:53 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 10 Mar 2017 02:03:53 +0100 Subject: [Security-sig] Archives (.tar or .zip) with absolute paths In-Reply-To: References: Message-ID: Le 10 mars 2017 1:52 AM, "Wes Turner" a ?crit : > https://python-security.readthedocs.io/ > > I wrote this doc :-) I just added notes about tarfile and zipfile. The [ ] wiki links could also be useful My long term plan is to not maintain this doc alone. Would you mind to propose a pull request directly? Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Thu Mar 9 22:59:34 2017 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 9 Mar 2017 21:59:34 -0600 Subject: [Security-sig] Archives (.tar or .zip) with absolute paths In-Reply-To: References: Message-ID: On Thu, Mar 9, 2017 at 7:03 PM, Victor Stinner wrote: > Le 10 mars 2017 1:52 AM, "Wes Turner" a ?crit : > > > https://python-security.readthedocs.io/ >> >> I wrote this doc :-) I just added notes about tarfile and zipfile. > > > The [ ] wiki links could also be useful > > > My long term plan is to not maintain this doc alone. Would you mind to > propose a pull request directly? > - [ ] Add (nested?) links to the given wiki pages to python-security.rtfd.org - [ ] Would this (see: thread) info be useful for the docs or the devguide? - [ ] Would this (or a similar) checklist be useful for the security guidelines? > Victor > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Fri Mar 10 08:25:57 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 10 Mar 2017 14:25:57 +0100 Subject: [Security-sig] New report of Python vulnerabilities Message-ID: Hi, I made major enhancements in my tool to generate reports on Python vulnerabilities. * A timeline is now automatically generated using the different dates: initial report date, disclosure date, commits, Python releases, etc. * CVE details are now downloaded automatically, so it's not more filled manually in the YAML file. * Each vulnerability has now its own page Good example showing everything altogether, the Sweet32 attack: http://python-security.readthedocs.io/vuln/cve-2016-2183_sweet32_attack_des_3des.html The next step is to make sure that all maintained branches got a fix! I should enhance the tool to compute the list of vulnerable Python versions. Victor From victor.stinner at gmail.com Fri Mar 10 11:16:03 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 10 Mar 2017 17:16:03 +0100 Subject: [Security-sig] Archives (.tar or .zip) with absolute paths In-Reply-To: References: Message-ID: I opened two public bug reports: tarfile: http://bugs.python.org/issue29788 zipfile: http://bugs.python.org/issue29789 It's unclear to me if it's ok or not to backport the new absolute_path option to stable Python versions, to fix the vulnerability? Victor From victor.stinner at gmail.com Thu Mar 23 08:32:57 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 23 Mar 2017 13:32:57 +0100 Subject: [Security-sig] Python Vulnerabilities: Vulnerable Python versions added Message-ID: Hi, I just added the list of vulnerable Python versions to my report: http://python-security.readthedocs.io/vulnerabilities.html So I checked the status of backports, and I identified one last vulnerability not fixed in Python 3.4 yet: HTTP directory traversal on Windows. I proposed a cherry-pick: https://github.com/python/cpython/pull/782 Python 3.2 and 3.3 lack a lot of fixes, but the last release was in 2014. Fixes were backported in the meanwhile, but no new security version was released since that time. gettext: FIXME 3.2 3.3 fixed: no release yet Sweet32 3.4 fixed: no release yet HTTPoxy attack: 3.2 and 3.3 FIXME 3.2 3.3 fixed: no release yet smtplib TLS striping FIXME 3.2 3.3 fixed: no release yet HTTP directory traversal FIXME 3.2 FIXME 3.3 FIXME 3.4 => https://github.com/python/cpython/pull/782 Expat 2.1.1 FIXME 3.2 FIXME 3.3 zipimporter overflow FIXME 3.2 3.3 fixed, no release yet HTTP Header injection FIXME 3.2 FIXME 3.3 Validate TLS certificate 3.2 and 3.3 vulnerable: no plan to backport the feature SSL: NULL byte 3.3 fixed: no release yet match_hostname IDNA FIXME 3.2 xmlrpc gzip decode 3.2 fixed in 2014: no release yet 3.3 fixed: no release yet Victor