OS X 10.9 Mavericks -> 2.7.6/3.3.3 updates needed
On Tuesday, Apple released OS X 10.9 (a.k.a. Mavericks). There has already been a lot of interest in it, in part because Apple has made it available for free and has made it easy for users with 10.8, 10.7, and (most) 10.6 systems to upgrade directly to 10.9. Unfortunately, there are issues with our current maintenance releases (3.3.2 and 2.7.5) on OS X 10.9 that call for new maintenance releases as soon as possible. One is critical in that it causes the interpreter to crash when running in interactive mode (http://bugs.python.org/issue18458). The problem was due to a long-standing compatibility issue in libedit's readline compatibility layer that upstream has finally fixed and Apple has now shipped in 10.9. Because the python.org installers dynamically link to libedit, the original workaround in readline.c for the original design flaw in history indexing now causes a segfault on 10.9 when the user types in the second command interactively. Not good. Ronald devised a fix that allows readline.so at runtime to detect and work with either version of libedit so that we continue to have binary compatibility across multiple OS X releases. That fix is already out in the 3.4.0 alphas and backported to the 3.3 and 2.7 branches, awaiting release there. Just in the last 12 hours, there have been at least four duplicates of the issue reported by users. I've updated the original issue to explicitly mention 10.9, now that it is no longer under NDA, and to provide a downloadable script for inexperienced users to workaround the problem by "removing" readline.so. Presumably, as word gets out, there will be fewer duplicate issues opened but the impact will remain. The only other major 10.9-related issue that I'm aware of at the moment is one with the OS X native Tk (of course!) that can affect IDLE and other Tkinter applications. See http://bugs.python.org/issue19373 for details. The workaround for it is to run Tk in 32-bit mode. I've "engaged" the Tk project in the problem. Since it can affect all users of Tk, and not just Python ones, I'm hopeful that there will be a fix soon. Note that, for 3.4.0, we are now packaging a private copy of Tcl/Tk with the 10.6+ OS X installer to avoid the chronic problems caused by Apple not updating the buggy system Tcl/Tk and forcing users to download and install a third-party Tcl/Tk just to be able to use IDLE. With the concurrence of the release managers, I am in the process of backporting that packaging support into 3.3 and 2.7. That means any Tk fix will likely require an update to those private copies. Because of the criticality of the readline issue, we really need to get 2.7.6 and 3.3.3 out as soon as practical. Besides the Tk packaging, there are a few other 10.9 build-related issues that should go into these releases. I expect to complete testing across all releases and have everything checked-in sometime this weekend. I don't think we should wait for the Issue19373 Tk fix if it isn't available when everything else is. If necessary, we can later re-release the OS X installers with just the updated Tcl/Tk shared libraries. I don't know where any other potential 2.7.6 or 3.3.3 issues stand at this point. But I'd like Benjamin and Georg to propose an aggressive schedule so we can get these fixes out there. -- Ned Deily, nad@acm.org
Am 24.10.2013 11:11, schrieb Ned Deily:
I don't know where any other potential 2.7.6 or 3.3.3 issues stand at this point. But I'd like Benjamin and Georg to propose an aggressive schedule so we can get these fixes out there.
We've basically agreed to do rc releases this weekend. I don't have time the weekend of Nov 2, so 3.3.3 final would be scheduled for Nov 9. There are a few "unlimited read" issues still open; I will have a look at porting their patches tomorrow. Georg
Am 24.10.2013 11:26, schrieb Georg Brandl:
Am 24.10.2013 11:11, schrieb Ned Deily:
I don't know where any other potential 2.7.6 or 3.3.3 issues stand at this point. But I'd like Benjamin and Georg to propose an aggressive schedule so we can get these fixes out there.
We've basically agreed to do rc releases this weekend. I don't have time the weekend of Nov 2, so 3.3.3 final would be scheduled for Nov 9. There are a few "unlimited read" issues still open; I will have a look at porting their patches tomorrow.
There seems to be a problem with the security fix "Re-seed OpenSSL's PRNG after fork": http://bugs.python.org/issue18747 http://bugs.python.org/issue19227 Perhaps it's best to disable the pthread_atfork() handler for the upcoming releases and replace it with security note in the ssl module, os.fork() and release notes. <big_friendy_letters> If you are using fork() and the ssl module in the same application then you must re-seed the PRNG with ssl.RAND_add() every now and then. </big_friendy_letters> Christian
2013/10/24 Christian Heimes <christian@python.org>:
There seems to be a problem with the security fix "Re-seed OpenSSL's PRNG after fork":
http://bugs.python.org/issue18747 http://bugs.python.org/issue19227
Yes, Charles Francois warned us that adding a pthread_atfork() hook to add entropy may add a deadlock. Such change should *not* be added to stable releases, even if it is supposed to improve the security.
Perhaps it's best to disable the pthread_atfork() handler for the upcoming releases and replace it with security note in the ssl module, os.fork() and release notes.
IMO the best place to fix the bug is in OpenSSL directly: RAND_bytes() function of OpenSSL can detect a fork using getpid() and add more entropy (in the child or maybe in the parent process). OpenSSL has access to entropy sources and knows all mutexes, whereas Python can only guess the list of mutexes (there are probably many more private mutexes). OpenSSL may use pthread_atfork() internally. http://www.openwall.com/lists/oss-security/2013/04/12/3 "I believe it is wrong to fix this in PostgreSQL. Rather, this is a bug in the OpenSSL fork protection code. It should either install a fork hook, or reseed the PRNG from /dev/urandom if a PID change is detected." I wrote a patch for OpenSSL long time ago (when working on my Hasard project, PRNG library), but I don't remember if I proposed it upstream: https://bitbucket.org/haypo/hasard/src/tip/patches/openssl_rand_fork.patch
<big_friendy_letters> If you are using fork() and the ssl module in the same application then you must re-seed the PRNG with ssl.RAND_add() every now and then. </big_friendy_letters>
You should explain that the issue only occurs if you fork more than 2^16 times and two child processes get the same pid. I wouldn't say that it is unlikely, but just that there is no reason to scare users if they are not vulnerable. The issue can occur if a process spawn a lot of processes. Reminder: you are supposed to only call exec() after fork(), nothing else :-) (Only signal-safe functions are officially supported between exec() and fork()). Victor
Am 24.10.2013 13:36, schrieb Victor Stinner:
IMO the best place to fix the bug is in OpenSSL directly: RAND_bytes() function of OpenSSL can detect a fork using getpid() and add more entropy (in the child or maybe in the parent process). OpenSSL has access to entropy sources and knows all mutexes, whereas Python can only guess the list of mutexes (there are probably many more private mutexes). OpenSSL may use pthread_atfork() internally.
That's not going to happen anytime soon (if ever). It doesn't solve old installation, too. You are forgetting that ssl.RAND_bytes() and RAND_pseudo_bytes() are our least concern. The problem also affects the TLS/SSL handshake and any internal OpenSSL code that needs a CPRNG.
Reminder: you are supposed to only call exec() after fork(), nothing else :-) (Only signal-safe functions are officially supported between exec() and fork()).
How are forking web servers suppose to work if you can't use read() or write()? :)
Kind of on a tangent (and I suspect I know what the answer to my question will be), but here at work they have asked people who use Macs not to upgrade to Mavericks until Apple fixes one or more networking problems which make it basically unusable, especially when connecting to our VPN. I realize y'all will be busy trying to fix these various problems as they relate to Python, but might it not be better to simply tell people to hold off on the Python installers until Apple gets their act together? Otherwise, I fear you'll just be trying to hit a moving target. Skip
2013/10/24 Georg Brandl <g.brandl@gmx.net>:
Am 24.10.2013 11:11, schrieb Ned Deily:
I don't know where any other potential 2.7.6 or 3.3.3 issues stand at this point. But I'd like Benjamin and Georg to propose an aggressive schedule so we can get these fixes out there.
We've basically agreed to do rc releases this weekend. I don't have time the weekend of Nov 2, so 3.3.3 final would be scheduled for Nov 9. There are a few "unlimited read" issues still open; I will have a look at porting their patches tomorrow.
What Georg said, but I'll probably be around Nov 2. -- Regards, Benjamin
On Oct 24, 2013, at 02:11 AM, Ned Deily wrote:
I don't know where any other potential 2.7.6 or 3.3.3 issues stand at this point. But I'd like Benjamin and Georg to propose an aggressive schedule so we can get these fixes out there.
Does this problem affect 2.6? 2.6.9 final is scheduled for Monday, so if there's something we need to get in before then, please let me know asap. Cheers, -Barry
In article <20131024094436.230220bf@anarchist>, Barry Warsaw <barry@python.org> wrote:
I don't know where any other potential 2.7.6 or 3.3.3 issues stand at this point. But I'd like Benjamin and Georg to propose an aggressive schedule so we can get these fixes out there. Does this problem affect 2.6? 2.6.9 final is scheduled for Monday, so if
On Oct 24, 2013, at 02:11 AM, Ned Deily wrote: there's something we need to get in before then, please let me know asap.
Yes, this problem also affects 2.6. There are some mitigating factors. The support for libedit on OS X is only enabled when building for an OS X 10.5 or later ABI because in earlier releases, the readline emulation of libedit was judged too buggy. In 2.6 as is also the case in 2.7, when building from source, ./configure defaults to using a 10.4 ABI unless certain universal build options are selected or the user explicitly sets MACOSX_DEPLOYMENT_TARGET=10.5 or higher when running configure. With the default 10.4 setting, readline.so fails to build so there is no crash - no readline features, either. Also, if one supplies a version of GNU readline (which Apple does not ship) as many people do, there is also no crash. The 2.7 change of Issue18458 (http://hg.python.org/cpython/rev/1e03fd72e116) depends on some previous 2.7-only changes in Modules/readline.c so it does not apply cleanly to 2.6. However, there aren't *that* many other earlier changes to 2.7 readline that are not in 2.6 and, for the most part, they have to do with fixing memory leaks, including some introduced by using newer versions of GNU readline, and I didn't see any new features. Copying the current 2.7 tip of readline.c over to the current tip of 2.6 builds cleanly on 10.9. test_readline passes and the simple history scrolling seems to work without crashing. I'd recommend either doing nothing or backporting everything. -- Ned Deily, nad@acm.org
On Oct 24, 2013, at 01:12 PM, Ned Deily wrote:
Yes, this problem also affects 2.6. There are some mitigating factors. The support for libedit on OS X is only enabled when building for an OS X 10.5 or later ABI because in earlier releases, the readline emulation of libedit was judged too buggy. In 2.6 as is also the case in 2.7, when building from source, ./configure defaults to using a 10.4 ABI unless certain universal build options are selected or the user explicitly sets MACOSX_DEPLOYMENT_TARGET=10.5 or higher when running configure. With the default 10.4 setting, readline.so fails to build so there is no crash - no readline features, either. Also, if one supplies a version of GNU readline (which Apple does not ship) as many people do, there is also no crash.
The 2.7 change of Issue18458 (http://hg.python.org/cpython/rev/1e03fd72e116) depends on some previous 2.7-only changes in Modules/readline.c so it does not apply cleanly to 2.6. However, there aren't *that* many other earlier changes to 2.7 readline that are not in 2.6 and, for the most part, they have to do with fixing memory leaks, including some introduced by using newer versions of GNU readline, and I didn't see any new features. Copying the current 2.7 tip of readline.c over to the current tip of 2.6 builds cleanly on 10.9. test_readline passes and the simple history scrolling seems to work without crashing. I'd recommend either doing nothing or backporting everything.
Thanks for the background Ned. Given that 2.6.9 will be the last release of the 2.6 series, we won't have much of a chance of going back to fix things if we/I mess it up. Our options then are to do another release candidate, which frankly will probably get about as much testing as the first one (i.e. close to zero, afaict) and push the final release out a few weeks. Or, as you say we can just do nothing. Maybe that means 2.6 will be broken on OS X 10.9 unless someone crafts and applies patches themselves, but I think that's acceptable given how old 2.6 is (EOL) and how new 10.9 is. If 10.9 had come out two weeks from now, "do nothing" would have been the obvious answer. :) So unless I hear strong objections before next Monday, I am not going to fix this in 2.6.9. Cheers, -Barry
On 25 Oct 2013 06:42, "Barry Warsaw" <barry@python.org> wrote:
On Oct 24, 2013, at 01:12 PM, Ned Deily wrote:
Yes, this problem also affects 2.6. There are some mitigating factors.
support for libedit on OS X is only enabled when building for an OS X 10.5 or later ABI because in earlier releases, the readline emulation of libedit was judged too buggy. In 2.6 as is also the case in 2.7, when building from source, ./configure defaults to using a 10.4 ABI unless certain universal build options are selected or the user explicitly sets MACOSX_DEPLOYMENT_TARGET=10.5 or higher when running configure. With
The the
default 10.4 setting, readline.so fails to build so there is no crash - no readline features, either. Also, if one supplies a version of GNU readline (which Apple does not ship) as many people do, there is also no crash.
The 2.7 change of Issue18458 ( http://hg.python.org/cpython/rev/1e03fd72e116) depends on some previous 2.7-only changes in Modules/readline.c so it does not apply cleanly to 2.6. However, there aren't *that* many other earlier changes to 2.7 readline that are not in 2.6 and, for the most part, they have to do with fixing memory leaks, including some introduced by using newer versions of GNU readline, and I didn't see any new features. Copying the current 2.7 tip of readline.c over to the current tip of 2.6 builds cleanly on 10.9. test_readline passes and the simple history scrolling seems to work without crashing. I'd recommend either doing nothing or backporting everything.
Thanks for the background Ned.
Given that 2.6.9 will be the last release of the 2.6 series, we won't have much of a chance of going back to fix things if we/I mess it up.
Our options then are to do another release candidate, which frankly will probably get about as much testing as the first one (i.e. close to zero, afaict) and push the final release out a few weeks. Or, as you say we can just do nothing. Maybe that means 2.6 will be broken on OS X 10.9 unless someone crafts and applies patches themselves, but I think that's acceptable given how old 2.6 is (EOL) and how new 10.9 is.
If 10.9 had come out two weeks from now, "do nothing" would have been the obvious answer. :) So unless I hear strong objections before next Monday, I am not going to fix this in 2.6.9.
Since the default build settings work, that sounds reasonable. Perhaps include a note somewhere that targeting a more recent ABI may involve copying the 2.7 readline.c? Cheers, Nick.
Cheers, -Barry _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
On Oct 25, 2013, at 07:54 AM, Nick Coghlan wrote:
Since the default build settings work, that sounds reasonable. Perhaps include a note somewhere that targeting a more recent ABI may involve copying the 2.7 readline.c?
I'll probably add issue 18458 to the announcement and release page. That way we can still use the tracker to manage patches for 2.6, even though they won't be officially applied. -Barry
participants (8)
-
Barry Warsaw -
Benjamin Peterson -
Christian Heimes -
Georg Brandl -
Ned Deily -
Nick Coghlan -
Skip Montanaro -
Victor Stinner