Remove embedded expat library?
Hi, Python embeds a copy of the expat library which already got two major security vulnerabilities: "CVE-2016-0718: expat bug #537" http://python-security.readthedocs.io/vuln/cve-2016-0718_expat_bug_537.html "Issue #26556: Expat 2.1.1" http://python-security.readthedocs.io/vuln/issue_26556_expat_2.1.1.html Would it be possible to maintain this dependency on an external repository which would be easier to maintain? Like http://svn.python.org/projects/external/ used to build Python on Windows. I expect that all Linux distributions build Python using --with-system-expat. It may become the default? What about macOS and other operating systems? By the way, Zachary Ware is working on converting this repository to Git. I don't know his progress: - https://github.com/python/cpython-bin-deps - https://github.com/python/cpython-source-deps Victor
On Jun 9, 2017, at 08:43, Victor Stinner
Python embeds a copy of the expat library which already got two major security vulnerabilities:
"CVE-2016-0718: expat bug #537" http://python-security.readthedocs.io/vuln/cve-2016-0718_expat_bug_537.html
"Issue #26556: Expat 2.1.1" http://python-security.readthedocs.io/vuln/issue_26556_expat_2.1.1.html
Would it be possible to maintain this dependency on an external repository which would be easier to maintain? Like http://svn.python.org/projects/external/ used to build Python on Windows.
Considering how infrequently there have been releases of upstream expat over the past decade or so, even if we had caught every release, I don't see that the benefit to maintaining our embedded copy as a separate repo rather than have to apply the upstreams copies a few time (to each Python branch) is worth the disruptive effort in changing the current Python build process (assuming we don't change the default, see below). Now, as to whether we should continue to provide our own copy of expat, that's a different question. Are there differences between our bundled expat and upstream?
I expect that all Linux distributions build Python using --with-system-expat. It may become the default? What about macOS and other operating systems?
The current default is --with-system-expat=no so, unless builders of Python take explicit action, the bundled version of expat is used. Using the bundled version is also currently the case for the python.org macOS installer, no idea what other distributors do. Apple supplies a version of expat with macOS so we presumably we could use the system version for the installer. Presumably (Zach?) we would need to continue to supply a version of expat for Windows builds. But do we need to for others? If it were only Windows, *then* perhaps it might make sense to make all the changes to move expat out of cpython into the common repo for third-party Windows libs.
By the way, Zachary Ware is working on converting this repository to Git. I don't know his progress: - https://github.com/python/cpython-bin-deps - https://github.com/python/cpython-source-deps
It appears that the project itself is moving from Sourceforge to Github: https://github.com/libexpat/libexpat -- Ned Deily nad@python.org -- []
On Fri, Jun 9, 2017 at 11:03 AM, Ned Deily
On Jun 9, 2017, at 08:43, Victor Stinner
wrote: I expect that all Linux distributions build Python using --with-system-expat. It may become the default? What about macOS and other operating systems?
The current default is --with-system-expat=no so, unless builders of Python take explicit action, the bundled version of expat is used. Using the bundled version is also currently the case for the python.org macOS installer, no idea what other distributors do. Apple supplies a version of expat with macOS so we presumably we could use the system version for the installer. Presumably (Zach?) we would need to continue to supply a version of expat for Windows builds. But do we need to for others? If it were only Windows, *then* perhaps it might make sense to make all the changes to move expat out of cpython into the common repo for third-party Windows libs.
Yes, we would need to continue providing a version for Windows. It would be a relatively small change to move it to the externals repository. I would be fine with switching to `--with-system-expat=yes` by default and building from externals on Windows in 3.7, and removing the bundled expat in 3.8.
By the way, Zachary Ware is working on converting this repository to Git. I don't know his progress: - https://github.com/python/cpython-bin-deps - https://github.com/python/cpython-source-deps
PR 1783 (https://github.com/python/cpython/pull/1783); needs another review from Steve to make sure I haven't made a complete mess of things, then it should be ready to go. Anyone else on Windows (Terry Reedy?) who can test it and provide feedback, please do! -- Zach
On 6/9/2017 12:40 PM, Zachary Ware wrote:
PR 1783 (https://github.com/python/cpython/pull/1783); needs another review from Steve to make sure I haven't made a complete mess of things, then it should be ready to go. Anyone else on Windows (Terry Reedy?) who can test it and provide feedback, please do!
Something like the following condensed log? f:\dev>git clone https://github.com/python/cpython tem f:\dev>cd tem f:\dev\tem>git remote add upstream https://github.com/python/cpython f:\dev\tem>git pr 1783 f:\dev\tem>pcbuild\build.bat -e Fetching external libraries... Fetching bzip2-1.0.6... Fetching openssl-1.0.2k... Fetching sqlite-3.14.2.0... Fetching tcl-core-8.6.6.0... Fetching tk-8.6.6.0... Fetching tix-8.4.3.6... Fetching xz-5.2.2... Fetching external binaries... Fetching nasm-2.11.06... ... # Appeared to finish with usual warnings, but no errors. f:\dev\tem>python -m test -j14 ... 367 tests OK. 38 tests skipped: test_crypt test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue test_nis test_openpty test_ossaudiodev test_pipes test_poll test_posix test_pty test_pwd test_readline test_resource test_smtpnet test_socketserver test_spwd test_syslog test_threadsignals test_timeout test_tix test_tk test_ttk_guionly test_urllib2net test_urllibnet test_wait3 test_wait4 test_winsound test_xmlrpc_net test_zipfile64 Total duration: 2 min 15 sec Tests result: SUCCESS ----- Duration aside, my working clone master, without the patch, gives the same result. (The usual debug build takes twice as long.) In this respect, the patch seems ready to go. If you have any questions about the compile or test logs, I will try to keep them around at least a few hours. -- Terry Jan Reedy
On 9 Jun 2017, at 18:03, Ned Deily
wrote: On Jun 9, 2017, at 08:43, Victor Stinner
wrote: I expect that all Linux distributions build Python using --with-system-expat. It may become the default? What about macOS and other operating systems?
The current default is --with-system-expat=no so, unless builders of Python take explicit action, the bundled version of expat is used. Using the bundled version is also currently the case for the python.org macOS installer, no idea what other distributors do. Apple supplies a version of expat with macOS so we presumably we could use the system version for the installer. Presumably (Zach?) we would need to continue to supply a version of expat for Windows builds. But do we need to for others? If it were only Windows, *then* perhaps it might make sense to make all the changes to move expat out of cpython into the common repo for third-party Windows libs.
I don’t think it would be a good idea to rely on the system provided libexpat on macOS, as Apple is not exactly fast w.r.t. upgrading their external dependencies and could easily stop updating libraries when the no longer need them (see for example the mess w.r.t. OpenSSL). Ronald
Le 11 juin 2017 09:38, "Ronald Oussoren"
On 11 Jun 2017, at 12:10, Victor Stinner
wrote: Le 11 juin 2017 09:38, "Ronald Oussoren"
mailto:ronaldoussoren@mac.com> a écrit : I don’t think it would be a good idea to rely on the system provided libexpat on macOS, as Apple is not exactly fast w.r.t. upgrading their external dependencies and could easily stop updating libraries when the no longer need them (see for example the mess w.r.t. OpenSSL). Ok, but can't we download expat instead of keeping an old copy in our repisitory?
Sure. The script that creates the installer already downloads a number of libraries, adding one more shouldn’t be a problem.
Having a copy is useful when we modify it. I don't that it is the case here.
I don’t know why expat was included in the CPython tree and if those reasons are still valid. I therefore have no opinion on keeping it, other than that expat shouldn’t be kept in the CPython tree unless there’s a good reason for doing so. BTW. Removing 3th-party libraries from the source tree doesn’t fully isolate us from security issues in those libraries due to shipping the libraries in binary installers on Windows and macOS. The removal does make maintenance easier (no need to guess whether or not there are local patches). Ronald
Victor _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com
participants (5)
-
Ned Deily
-
Ronald Oussoren
-
Terry Reedy
-
Victor Stinner
-
Zachary Ware