We cannot fix all issues: let's close XML security issues (not fix them)
Hi, The Python bug tracker is full of bugs, and sadly we don't have enough people to take care of all of them. There are 3 open bugs about security issues in XML and I simply propose to close it: https://bugs.python.org/issue17318 https://bugs.python.org/issue17239 https://bugs.python.org/issue24238 The XML documentation already starts with a red warning explaining the security limitations of the Python implementation and points to defusedxml and defusedexpat which are existing and working counter-measures: https://docs.python.org/dev/library/xml.html Note: Christian Heimes, author of these 2 packages, told me that these modules may not work on Python 3.7, he didn't have time to maintain them recently. Maybe someone might want to help him? I suggest to close the 3 Python bugs without doing anything. Are you ok with that? Keeping the issue open for 3 years doesn't help anyone, and there is already a security warning in all supported version (I checked 2.7 and 3.4). It seems like XML is getting less popular because of JSON becoming more popular (even if JSON obviously comes with its own set of security issues...). It seems like less core developers care about XML (today than 3 years ago). We should just accept that core developers have limited availability and that documenting security issues is an *acceptable* trade-off. I don't see any value of keeping these 3 issues open. Victor
On Thu, 6 Sep 2018 16:18:33 +0200 Victor Stinner <vstinner@redhat.com> wrote:
It seems like XML is getting less popular because of JSON becoming more popular (even if JSON obviously comes with its own set of security issues...). It seems like less core developers care about XML (today than 3 years ago).
We should just accept that core developers have limited availability and that documenting security issues is an *acceptable* trade-off. I don't see any value of keeping these 3 issues open.
If we consider fixing these issues to be desirable, then the issues should be kept open. Closing issues because no-one is working on them sounds a bit silly to me. Regards Antoine.
Le jeu. 6 sept. 2018 à 16:33, Antoine Pitrou <solipsis@pitrou.net> a écrit :
If we consider fixing these issues to be desirable, then the issues should be kept open. Closing issues because no-one is working on them sounds a bit silly to me.
I forgot to mention that closing these issues is my reply to Larry's call to fix 3 security issues: https://mail.python.org/pipermail/python-committers/2018-August/006031.html Larry wrote "If they're really all wontfix, maybe we should mark them as wontfix, thus giving 3.4 a sendoff worthy of its heroic stature." For these XML issues, the security vulnerabilities can also been seen as XML features. Loading an external DTD is part of the XML specification, as well as entity expansion. I'm also dubious about PyYAML which allows to run arbitrary Python code in a configuration *by default*. But well, it seems like nobody stepped in to change the default. Victor
Le 06/09/2018 à 16:40, Victor Stinner a écrit :
Le jeu. 6 sept. 2018 à 16:33, Antoine Pitrou <solipsis@pitrou.net> a écrit :
If we consider fixing these issues to be desirable, then the issues should be kept open. Closing issues because no-one is working on them sounds a bit silly to me.
I forgot to mention that closing these issues is my reply to Larry's call to fix 3 security issues:
https://mail.python.org/pipermail/python-committers/2018-August/006031.html
Larry wrote "If they're really all wontfix, maybe we should mark them as wontfix, thus giving 3.4 a sendoff worthy of its heroic stature."
"wontfix" on 3.4 doesn't mean we won't fix them later, e.g. in 3.8.
For these XML issues, the security vulnerabilities can also been seen as XML features. Loading an external DTD is part of the XML specification, as well as entity expansion.
That doesn't mean there shouldn't be any hard limits to expansion depth or breadth. Function calls are a Python feature, yet we limit the amount of recursion allowed. Regards Antoine.
Are you volunteer to fix the XML modules? Victor Le jeu. 6 sept. 2018 à 16:50, Antoine Pitrou <antoine@python.org> a écrit :
Le 06/09/2018 à 16:40, Victor Stinner a écrit :
Le jeu. 6 sept. 2018 à 16:33, Antoine Pitrou <solipsis@pitrou.net> a écrit :
If we consider fixing these issues to be desirable, then the issues should be kept open. Closing issues because no-one is working on them sounds a bit silly to me.
I forgot to mention that closing these issues is my reply to Larry's call to fix 3 security issues:
https://mail.python.org/pipermail/python-committers/2018-August/006031.html
Larry wrote "If they're really all wontfix, maybe we should mark them as wontfix, thus giving 3.4 a sendoff worthy of its heroic stature."
"wontfix" on 3.4 doesn't mean we won't fix them later, e.g. in 3.8.
For these XML issues, the security vulnerabilities can also been seen as XML features. Loading an external DTD is part of the XML specification, as well as entity expansion.
That doesn't mean there shouldn't be any hard limits to expansion depth or breadth.
Function calls are a Python feature, yet we limit the amount of recursion allowed.
Regards
Antoine. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
Le 06/09/2018 à 16:58, Victor Stinner a écrit :
Are you volunteer to fix the XML modules?
No. That doesn't mean nobody else will be. Regards Antoine.
Victor Le jeu. 6 sept. 2018 à 16:50, Antoine Pitrou <antoine@python.org> a écrit :
Le 06/09/2018 à 16:40, Victor Stinner a écrit :
Le jeu. 6 sept. 2018 à 16:33, Antoine Pitrou <solipsis@pitrou.net> a écrit :
If we consider fixing these issues to be desirable, then the issues should be kept open. Closing issues because no-one is working on them sounds a bit silly to me.
I forgot to mention that closing these issues is my reply to Larry's call to fix 3 security issues:
https://mail.python.org/pipermail/python-committers/2018-August/006031.html
Larry wrote "If they're really all wontfix, maybe we should mark them as wontfix, thus giving 3.4 a sendoff worthy of its heroic stature."
"wontfix" on 3.4 doesn't mean we won't fix them later, e.g. in 3.8.
For these XML issues, the security vulnerabilities can also been seen as XML features. Loading an external DTD is part of the XML specification, as well as entity expansion.
That doesn't mean there shouldn't be any hard limits to expansion depth or breadth.
Function calls are a Python feature, yet we limit the amount of recursion allowed.
Regards
Antoine. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
On 06Sep2018 0758, Victor Stinner wrote:
Are you volunteer to fix the XML modules?
If Christian is not able to keep maintaining the defused* packages, then I may take a look at this next week at the sprints. The built-in XML packages actually don't meet Microsoft's internal security requirements, so I have some business motivation to do it. Hopefully it doesn't turn me into the sole XML maintainer... Ultimately, however, I think we're looking at technically incompatible design changes, which is why simply dropping in a "fix" for 3.4 would not work whereas adding new options (with more secure defaults) may work for 3.8. So I'm agreed with nearly everyone else - bugs should stay open as long as we're interested in taking a fix, even if they've already been open for a long time. Our issue tracker is a backlog, not a plan, so there is no penalty for something sitting in there for a long time. Cheers, Steve
Le jeu. 6 sept. 2018 à 21:10, Steve Dower <steve.dower@python.org> a écrit :
If Christian is not able to keep maintaining the defused* packages, then I may take a look at this next week at the sprints. The built-in XML packages actually don't meet Microsoft's internal security requirements, so I have some business motivation to do it.
Great! The best would be to be able to merge defuse* features into the stdlib. Maybe not change the default, but add an option to enable security counter-measures. Victor
Thank you Victor. XML support in Python is critical and desired for many sectors like banking or telecoms, and code base based on XML is still on rise in such world. That's why keeping such bugs open is important, as it is not impossible that someone (banks, telecoms, google camps, government grants) would simply fund small project aiming at fixing those bugs in XML. We never know. -------- Beginning of forwarded message -------- 07.09.2018, 09:03, "Victor Stinner" <vstinner@redhat.com>: Le jeu. 6 sept. 2018 à 21:10, Steve Dower <steve.dower@python.org> a écrit :
If Christian is not able to keep maintaining the defused* packages, then I may take a look at this next week at the sprints. The built-in XML packages actually don't meet Microsoft's internal security requirements, so I have some business motivation to do it.
Great! The best would be to be able to merge defuse* features into the stdlib. Maybe not change the default, but add an option to enable security counter-measures. Victor _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/pms.coder%40yandex.ru -------- End of forwarded message --------
Le ven. 7 sept. 2018 à 17:02, PMS PMS <pms.coder@yandex.com> a écrit :
XML support in Python is critical and desired for many sectors like banking or telecoms, and code base based on XML is still on rise in such world.
Would it be possible to send money to the PSF? I'm sure that the PSF will be able to find you a developer able to quickly fix these XML issues! Victor
@VictorStinner snif, que dire? il me semble que cet issue ait pris une nouvelle dimension @appinv Abdur-Rahmaan Janhangeer https://github.com/Abdur-rahmaanJ Mauritius
On 2018-09-07 17:46, Victor Stinner wrote:
Le ven. 7 sept. 2018 à 17:02, PMS PMS <pms.coder@yandex.com> a écrit :
XML support in Python is critical and desired for many sectors like banking or telecoms, and code base based on XML is still on rise in such world.
Would it be possible to send money to the PSF? I'm sure that the PSF will be able to find you a developer able to quickly fix these XML issues!
Feel free to send the money directly to me. After all I found the bugs, documented them, and fixed them in defusedxml.
FWIW I'm with Antoine here -- XML is still important and I'd like us to go the extra mile here, not just give up because the issues have been inactive for a long time. We can't control what PyYAML does, but for the stdlib XML code, the buck stops here, and we should do the responsible thing. On Thu, Sep 6, 2018 at 7:49 AM Antoine Pitrou <antoine@python.org> wrote:
Le 06/09/2018 à 16:40, Victor Stinner a écrit :
Le jeu. 6 sept. 2018 à 16:33, Antoine Pitrou <solipsis@pitrou.net> a écrit :
If we consider fixing these issues to be desirable, then the issues should be kept open. Closing issues because no-one is working on them sounds a bit silly to me.
I forgot to mention that closing these issues is my reply to Larry's call to fix 3 security issues:
https://mail.python.org/pipermail/python-committers/2018-August/006031.html
Larry wrote "If they're really all wontfix, maybe we should mark them as wontfix, thus giving 3.4 a sendoff worthy of its heroic stature."
"wontfix" on 3.4 doesn't mean we won't fix them later, e.g. in 3.8.
For these XML issues, the security vulnerabilities can also been seen as XML features. Loading an external DTD is part of the XML specification, as well as entity expansion.
That doesn't mean there shouldn't be any hard limits to expansion depth or breadth.
Function calls are a Python feature, yet we limit the amount of recursion allowed.
Regards
Antoine. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
Thought: what if there's a label on the bug tracker meaning roughly "we're probably not going to fix this anytime soon, but we won't mind someone stepping up"? On Thu, Sep 6, 2018, 10:04 AM Guido van Rossum <guido@python.org> wrote:
FWIW I'm with Antoine here -- XML is still important and I'd like us to go the extra mile here, not just give up because the issues have been inactive for a long time. We can't control what PyYAML does, but for the stdlib XML code, the buck stops here, and we should do the responsible thing.
On Thu, Sep 6, 2018 at 7:49 AM Antoine Pitrou <antoine@python.org> wrote:
Le 06/09/2018 à 16:40, Victor Stinner a écrit :
Le jeu. 6 sept. 2018 à 16:33, Antoine Pitrou <solipsis@pitrou.net> a écrit :
If we consider fixing these issues to be desirable, then the issues should be kept open. Closing issues because no-one is working on them sounds a bit silly to me.
I forgot to mention that closing these issues is my reply to Larry's call to fix 3 security issues:
https://mail.python.org/pipermail/python-committers/2018-August/006031.html
Larry wrote "If they're really all wontfix, maybe we should mark them as wontfix, thus giving 3.4 a sendoff worthy of its heroic stature."
"wontfix" on 3.4 doesn't mean we won't fix them later, e.g. in 3.8.
For these XML issues, the security vulnerabilities can also been seen as XML features. Loading an external DTD is part of the XML specification, as well as entity expansion.
That doesn't mean there shouldn't be any hard limits to expansion depth or breadth.
Function calls are a Python feature, yet we limit the amount of recursion allowed.
Regards
Antoine. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
-- Ryan (ライアン) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else https://refi64.com/
On 09/06/2018 11:05 AM, Ryan Gonzalez wrote:
Thought: what if there's a label on the bug tracker meaning roughly "we're probably not going to fix this anytime soon, but we won't mind someone stepping up"?
"help-wanted" Tres. -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com
On 9/6/2018 11:05 AM, Ryan Gonzalez wrote:
Thought: what if there's a label on the bug tracker meaning roughly "we're probably not going to fix this anytime soon, but we won't mind someone stepping up"?
Not needed. Good patches are always welcome. And if there is no current PR or other information indicating otherwise, a fix 'soon' is usually unlikely. But what we mostly need is not more patches, but more reviews. Anyone can act like a core dev up to the point of actually pushing the green merge button. -- Terry Jan Reedy
On 2018-09-06 17:03, Guido van Rossum wrote:
FWIW I'm with Antoine here -- XML is still important and I'd like us to go the extra mile here, not just give up because the issues have been inactive for a long time. We can't control what PyYAML does, but for the stdlib XML code, the buck stops here, and we should do the responsible thing.
Back in the days, I didn't push hard for the necessary fixes, because all fixes were breaking changes. After all I'd have to disable some features that people may have relied upon. The XML security stuff was my first major security topic for Python, even before SipHash24. I was more concerned not to break people's software than to keep the majority of users safe. I have changed my opinion over the last six, seven years. By the way I couldn't fix some problems in Python and our expat wrapper either. The expat parser was missing features to properly implement security measurements. I need to check if expat has been improved over the years. The topic is on the agenda for the core dev sprint. Christian
Le ven. 7 sept. 2018 à 10:23, Christian Heimes <christian@python.org> a écrit :
Back in the days, I didn't push hard for the necessary fixes, because all fixes were breaking changes. After all I'd have to disable some features that people may have relied upon. The XML security stuff was my first major security topic for Python, even before SipHash24. I was more concerned not to break people's software than to keep the majority of users safe. I have changed my opinion over the last six, seven years.
I understood that Python 2.7.9 which required a valid TLS certificate annoyed many customers. So I don't think that it would be a good idea to enforce XML security in a minor Python release. But would it make sense to make XML stricter in Python 3.8 and add an option to opt-out? Or do we need a cycle of 1.5 year (Python 3.8) with a warning, and change the default in the next cycle?
The topic is on the agenda for the core dev sprint.
Great :-) Thanks are moving on. Victor
* Victor Stinner <vstinner@redhat.com>, 2018-09-06, 16:40:
I'm also dubious about PyYAML which allows to run arbitrary Python code in a configuration *by default*. But well, it seems like nobody stepped in to change the default.
PyYAML maintainers intend to change the default soon: https://github.com/yaml/pyyaml/issues/207 -- Jakub Wilk
no time? i have seen them countless of time on this list e.g. no ... don't implement this in the workflow as my volunteer time will be lost etc etc etc. i guess a call for more core contributors will be nice. for myself i have some translations ahead (finally getting the chance to read the docs from cover to cover), but i guess actually core-contributing will be a nice experience. the problem with getting contributors is that the docs need to be more readable, more tutos need to be written (less people are contributors / pyramid effect -> less guides written). the devs are doing a nice job guiding etc but the first step must be made easier. lack of time for an exceedingly popular project, for a very open system as python hints to a bottleneck somehere, not that there are no interests, but that they get blocked. Abdur-Rahmaan Janhangeer https://github.com/Abdur-rahmaanJ Mauritius
On 06/09/2018 07.18, Victor Stinner wrote:
Hi,
The Python bug tracker is full of bugs, and sadly we don't have enough people to take care of all of them. There are 3 open bugs about security issues in XML and I simply propose to close it:
https://bugs.python.org/issue17318 https://bugs.python.org/issue17239 https://bugs.python.org/issue24238
The XML documentation already starts with a red warning explaining the security limitations of the Python implementation and points to defusedxml and defusedexpat which are existing and working counter-measures:
https://docs.python.org/dev/library/xml.html
Note: Christian Heimes, author of these 2 packages, told me that these modules may not work on Python 3.7, he didn't have time to maintain them recently. Maybe someone might want to help him?
I suggest to close the 3 Python bugs without doing anything. Are you ok with that? Keeping the issue open for 3 years doesn't help anyone, and there is already a security warning in all supported version (I checked 2.7 and 3.4).
It seems like XML is getting less popular because of JSON becoming more popular (even if JSON obviously comes with its own set of security issues...). It seems like less core developers care about XML (today than 3 years ago).
We should just accept that core developers have limited availability and that documenting security issues is an *acceptable* trade-off. I don't see any value of keeping these 3 issues open.
Hi, during the Python core developer sprint, Steve Dower forced ^H^H^H^H^H^H convinced me into looking into the XML security bugs again. I come with fixes for all issues. However all security fixes require a change of behavior. I strongly believe that the change doesn't affect the majority of users in a negative way. For entity expansion attacks (billion laughs, quadratic blowup), the issue cannot be fixed in a libexpat callback. I decided that it's better to fix the issue in expat directly. libxml2 added limits for entity expansion many years ago, too. I created a patch for libexpat to limit nesting depths, entity length and ratio between XML data and expansion, https://github.com/libexpat/libexpat/pull/220 . The PR is a proof of concept. For the external entity and DTD bug in SAX and pulldom parser, I changed the default setting in PR https://github.com/python/cpython/pull/9217 . When accepted, the parsers no longer load and embed files from local directories or network locations. Regards, Christian
participants (13)
-
Abdur-Rahmaan Janhangeer
-
Antoine Pitrou
-
Antoine Pitrou
-
Christian Heimes
-
Guido van Rossum
-
Jakub Wilk
-
PMS PMS
-
Ryan Gonzalez
-
Simon Cross
-
Steve Dower
-
Terry Reedy
-
Tres Seaver
-
Victor Stinner