[Python-Dev] writelines() not thread-safe
Thu, 9 Mar 2000 21:13:51 -0500
[Guido van Rossum]
> Christian Tismer just did an exhaustive search for thread unsafe use
> of Python operations, and found two weaknesses. One is
> posix.listdir(), which I had already found; the other is
> file.writelines(). Here's a program that demonstrates the bug;
> basically, while writelines is walking down the list, another thread
> could truncate the list, causing PyList_GetItem() to fail or a string
> object to be deallocated while writelines is using it. On my SOlaris
> 7 system it typically crashes in the first or second iteration.
> It's easy to fix: just don't use release the interpreter lock (get rid
> of Py_BEGIN_ALLOW_THREADS c.s.). This would however prevent other
> threads from doing any work while this thread may be blocked for I/O.
> An alternative solution is to put Py_BEGIN_ALLOW_THREADS and
> Py_END_ALLOW_THREADS just around the fwrite() call. This is safe, but
> would require a lot of lock operations and would probably slow things
> down too much.
1: Before releasing the lock, make a shallow copy of the list.
1.5: As in #1, but iteratively peeling off "the next N" values, for some N
balancing the number of lock operations against the memory burden (I don't
care about the speed of a shallow copy here ...).
2. Pull the same trick list.sort() uses: make the list object immutable for
the duration (I know you think that's a hack, and it is <wink>, but it costs
virtually nothing and would raise an approriate error when they attempted
the insane mutation).
I actually like #2 best now, but won't in the future, because
file_writelines() should really accept an argument of any sequence type.
This makes 1.5 a better long-term hack.
although-adding-1.5-to-1.6-is-confusing<wink>-ly y'rs - tim