[Python-Dev] XML DoS vulnerabilities and exploits in Python

[Christian Heimes]

Hello,

in August 2012 I found a DoS vulnerability in expat and XML libraries in
Python's standard library. Since then I have found several more issues.
I have been working on fixes ever since.

The README of https://pypi.python.org/pypi/defusedxml contains detailed
explanations of my research and all issues


Blog post:
http://blog.python.org/2013/02/announcing-defusedxml-fixes-for-xml.html

Hotfixes:
https://pypi.python.org/pypi/defusedxml
https://pypi.python.org/pypi/defusedexpat


Repositories:
https://bitbucket.org/tiran/defusedxml
https://bitbucket.org/tiran/defusedexpat
https://bitbucket.org/tiran/expat

CVE (work in progress)

CVE-2013-1664
    Unrestricted entity expansion induces DoS vulnerabilities in
    Python XML libraries (XML bomb)
CVE-2013-1665
    External entity expansion in Python XML libraries
    inflicts potential security flaws and DoS vulnerabilities


Regards,
Christian



Extract from the documentation:

Synopsis
--------

The results of an attack on a vulnerable XML library can be fairly
dramatic. With just a few hundred Bytes of XML data an attacker can
occupy several Gigabytes of memory within seconds. An attacker can also
keep CPUs busy for a long time with a small to medium size request.
Under some circumstances it is even possible to access local files on
your server, to circumvent a firewall, or to abuse services to rebound
attacks to third parties.

The attacks use and abuse less common features of XML and its parsers.
The majority of developers are unacquainted with features such as
processing instructions and entity expansions that XML inherited from
SGML. At best they know about <!DOCTYPE> from experience with HTML but
they are not aware that a document type definition (DTD) can generate an
HTTP request or load a file from the file system.

None of the issues is new. They have been known for a long time. Billion
laughs was first reported in 2003. Nevertheless some XML libraries and
applications are still vulnerable and even heavy users of XML are
surprised by these features. It's hard to say whom to blame for the
situation. It's too short sighted to shift all blame on XML parsers and
XML libraries for using insecure default settings. After all they
properly implement XML specifications. Application developers must not
rely that a library is always configured for security and potential
harmful data by default.


Attack vectors
==============

billion laughs / exponential entity expansion
---------------------------------------------

The Billion Laughs attack -- also known as exponential entity expansion
-- uses multiple levels of nested entities. The original example uses 9
levels of 10 expansions in each level to expand the string lol to a
string of 3 * 10 9 bytes, hence the name "billion laughs". The resulting
string occupies 3 GB (2.79 GiB) of memory; intermediate strings require
additional memory. Because most parsers don't cache the intermediate
step for every expansion it is repeated over and over again. It
increases the CPU load even more.

An XML document of just a few hundred bytes can disrupt all services on
a machine within seconds.

Example XML:

<!DOCTYPE xmlbomb [
<!ENTITY a "1234567890" >
<!ENTITY b "&a;&a;&a;&a;&a;&a;&a;&a;">
<!ENTITY c "&b;&b;&b;&b;&b;&b;&b;&b;">
<!ENTITY d "&c;&c;&c;&c;&c;&c;&c;&c;">
]>
<bomb>&d;</bomb>

quadratic blowup entity expansion
---------------------------------

A quadratic blowup attack is similar to a Billion Laughs attack; it
abuses entity expansion, too. Instead of nested entities it repeats one
large entity with a couple of ten thousand chars over and over again.
The attack isn't as efficient as the exponential case but it avoids
triggering countermeasures of parsers against heavily nested entities.
Some parsers limit the depth and breadth of a single entity but not the
total amount of expanded text throughout an entire XML document.

A medium-sized XML document with a couple of hundred kilobytes can
require a couple of hundred MB to several GB of memory. When the attack
is combined with some level of nested expansion an attacker is able to
achieve a higher ratio of success.

<!DOCTYPE bomb [
<!ENTITY a "xxxxxxx... a couple of ten thousand chars">
]>
<bomb>&a;&a;&a;... repeat</bomb>


external entity expansion (remote)
----------------------------------

Entity declarations can contain more than just text for replacement.
They can also point to external resources by public identifiers or
system identifiers. System identifiers are standard URIs. When the URI
is a URL (e.g. a http:// locator) some parsers download the resource
from the remote location and embed them into the XML document verbatim.

Simple example of a parsed external entity:

<!DOCTYPE external [
<!ENTITY ee SYSTEM "http://www.python.org/some.xml">
]>
<root>ⅇ</root>

The case of parsed external entities works only for valid XML content.
The XML standard also supports unparsed external entities with a NData
declaration.

External entity expansion opens the door to plenty of exploits. An
attacker can abuse a vulnerable XML library and application to rebound
and forward network requests with the IP address of the server. It
highly depends on the parser and the application what kind of exploit is
possible. For example:

* An attacker can circumvent firewalls and gain access to restricted
resources as all the requests are made from an internal and trustworthy
IP address, not from the outside.
* An attacker can abuse a service to attack, spy on or DoS your servers
but also third party services. The attack is disguised with the IP
address of the server and the attacker is able to utilize the high
bandwidth of a big machine.
* An attacker can exhaust additional resources on the machine, e.g. with
requests to a service that doesn't respond or responds with very large
files.
* An attacker may gain knowledge, when, how often and from which IP
address a XML document is accessed.
    An attacker could send mail from inside your network if the URL
handler supports smtp:// URIs.


external entity expansion (local file)
--------------------------------------

External entities with references to local files are a sub-case of
external entity expansion. It's listed as an extra attack because it
deserves extra attention. Some XML libraries such as lxml disable
network access by default but still allow entity expansion with local
file access by default. Local files are either referenced with a file://
URL or by a file path (either relative or absolute).

An attacker may be able to access and download all files that can be
read by the application process. This may include critical configuration
files, too.

<!DOCTYPE external [
<!ENTITY ee SYSTEM "file:///PATH/TO/simple.xml">
]>
<root>ⅇ</root>

DTD retrieval
-------------

This case is similar to external entity expansion, too. Some XML
libraries like Python's xml.dom.pulldown retrieve document type
definitions from remote or local locations. Several attack scenarios
from the external entity case apply to this issue as well.

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
    <head/>
    <body>text</body>
</html>