<br><br><div class="gmail_quote">On Wed, Sep 9, 2009 at 20:19, Gregory P. Smith <span dir="ltr"><<a href="mailto:greg@krypto.org">greg@krypto.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div><div></div><div class="h5">On Wed, Sep 9, 2009 at 9:07 AM, Thomas Wouters<<a href="mailto:thomas@python.org">thomas@python.org</a>> wrote:<br>
><br>
><br>
> On Sat, Jul 25, 2009 at 19:25, Gregory P. Smith <<a href="mailto:greg@krypto.org">greg@krypto.org</a>> wrote:<br>
>><br>
>> On Thu, Jul 23, 2009 at 4:28 PM, Thomas Wouters<<a href="mailto:thomas@python.org">thomas@python.org</a>> wrote:<br>
>> ><br>
>> > So attached (and at <a href="http://codereview.appspot.com/96125/show" target="_blank">http://codereview.appspot.com/96125/show</a> ) is a<br>
>> > preliminary fix, correcting the problem with os.fork(), os.forkpty() and<br>
>> > os.fork1(). This doesn't expose a general API for C code to use, for two<br>
>> > reasons: it's not easy, and I need this fix more than I need the API<br>
>> > change<br>
>> > :-) (I actually need this fix myself for Python 2.4, but it applies<br>
>> > fairly<br>
>> > cleanly.)<br>
>><br>
>> This looks good to me.<br>
><br>
> Anyone else want to take a look at this before I check it in? I updated the<br>
> patch (in Rietveld) to contain some documentation about the hazards of<br>
> mixing fork and threads, which is the best we can do at the moment, at least<br>
> without seriously overhauling the threading APIs (which, granted, is not<br>
> that bad an idea, considering the mess they're in.) I've now thoroughly<br>
> tested the patch, and for most platforms it's strictly better. On AIX it<br>
> *may* behave differently (possibly 'incorrectly' for specific cases) if<br>
> something other than os.fork() calls the C fork() and calls<br>
> PyOS_AfterFork(), since on AIX it used to nuke the thread lock. *I* think<br>
> the new behaviour (not nuking the lock) is the correct thing to do, but<br>
> since most places that release the import lock don't bother to check if the<br>
> lock was even held, the old behaviour may have been succesfully masking the<br>
> bug on AIX systems.<br>
> Perhaps for the backport to 2.6 (which I think is in order, and also in<br>
> accordance with the guidelines) I should leave the AIX workaround in? Anyone<br>
> think it should not be removed from 3.x/2.7?<br>
><br>
>><br>
>> Your idea of making this an API called a 'fork lock' or something<br>
>> sounds good (though I think it needs a better name. PyBeforeFork &<br>
>> PyAfterFork?). The subprocess module, for example, disables garbage<br>
>> collection before forking and restores it afterwards to avoid<br>
>> <a href="http://bugs.python.org/issue1336" target="_blank">http://bugs.python.org/issue1336</a>. That type of thing could also be<br>
>> done in such a function.<br>
><br>
> Unfortunately it's rather hard to make those functions work correctly with<br>
> the current API -- we can't provide functions you can just use as arguments<br>
> to pthread_atfork because the global interpreter lock is not re-entrant and<br>
> we have no way of testing whether the current thread holds the GIL. I also<br>
> get the creepy-crawlies when I look at the various thread_* implementations<br>
> and see the horribly unsafe things they do (and also, for instance, the<br>
> PendingCall stuff in ceval.c :S) Unfortunately there's no good way to fix<br>
> these things without breaking API compatibility, let alone ABI<br>
> compatibility.<br>
<br>
</div></div>Take a look at <a href="http://code.google.com/p/python-atfork/" target="_blank">http://code.google.com/p/python-atfork/</a> which I created<br>
to address the general fork+threading with locks held causing<br>
deadlocks issue with many standard library modules. Currently it only<br>
patches the logging module but I intend to support others. I want to<br>
get an atfork mechanism into 2.7/3.2 along with every lock in the<br>
standard library making proper use of it.<br>
<br>
See also <a href="http://bugs.python.org/issue6721" target="_blank">http://bugs.python.org/issue6721</a><br>
<br>
I make no attempt to deal with C-level locks, only those acquired from<br>
python. It doesn't use pthread_atfork but instead models its behavior<br>
after that and wraps os.fork and os.forkpty so that they call the<br>
registered atfork methods as appropriate.</blockquote><div><br></div><div>Well, sure, the *Python code* side of the problem is fixable, thanks to Python being so awesome ;-P I'm strictly thinking of applications embedding Python (or even extending it and calling into code that doesn't consider Python.) Your python-atfork project looks like a terribly good idea, but it won't fix the embedding/extending issues (nor is it intended to, right?)</div>
</div><br>-- <br>Thomas Wouters <<a href="mailto:thomas@python.org">thomas@python.org</a>><br><br>Hi! I'm a .signature virus! copy me into your .signature file to help me spread!<br>