Is python is fully thread safe
dave at pythonapocrypha.com
Mon Jul 26 17:25:24 CEST 2004
Donn Cave wrote:
> Quoth Franois Pinard <pinard at iro.umontreal.ca>:
> | Well, bugs occurring out of lack of thread-safety might, in many
> | circumstances, hide for long before popping out, and hide again for long
> | afterwards. They are not easy to reproduce. "Testing reveals presence
> | of bugs, not their absence", as a proverb, is especially true in the
> | context of thread-safety.
> | Unless I'm missing a point (which is very possible), it does not seem
> | to me that Python documentation is especially collaborative at teaching
> | where threads are safe or not to use, so I'm left with the unpleasant
> | feeling that my threaded Python applications should (ideally) never be
> | allowed to go in real production. It's OK if I'm the only one to suffer
> | from unexpected problems.
> | I would much like if Python documentation had more definite and
> | definitive answers about thread-safety of Python, everywhere.
> Well, I too am skeptical about multithreaded programs in production -
> written in Python or anything else. That is not to say it can't work,
> but concurrency is just fraught with potential for trouble. I don't
> imagine that's going to stop people from using them, but they really
> need to be able to think somewhat analytically about the issues.
> I think you understand that it's a little more complex than "Hot dog,
> the documentation says Python is thread safe!"
> Ironically, Python is actually a little ahead here because it doesn't
> strictly speaking support concurrency. That makes Python internals
> effectively the same as thread safe, so thread safety issues would be
> confined to application level issues and external functions that run
> after releasing the interpreter lock. I've had no particular trouble
> in my programs using a multithreaded API where others reportedly did
> have trouble, and for all I know it may be just because I was using
> Python and they weren't.
This is quite true - automatic cleanup of resources on thread
termination combined with the fact that multithreaded Python code won't
corrupt the interpreter internals makes Python a fairly safe platform
for multi-threading - you certainly end up with a bigger safety net
than, say, C++.
As an aside, I think the documentation wrt to multi-threaded safety
could be improved, but I'm not sure how exactly to improve it because
"thread safety" doesn't always have a clear definition. For example, two
threads can safely access and even modify the same Python object (such
as a dictionary) without explicit locking and the interpreter will not
crash, so at that low-level you could say it is thread safe. But at the
semantic level of your particular application, a function that modifies
the same dictionary by multiple threads may or may not need explicit
locking around the dictionary modification (depending on what is being
done), so at that level the access may not be considered thread safe.
As to what the Python documentation needs to say (and, in the end,
perhaps it already does this well enough), I'd think it needs to make a
few things clear:
1) Accessing the same Python object from multiple threads from Python
code won't corrupt the internals of the interpreter.
2) Modules/functions that are just plain not thread-safe should be
marked as such (e.g. the fpectl module).
Beyond that, can you think of anything else? I doubt that the work
required to document the low-level thread safety of each and every
function is really worth it (not to mention the effort required to keep
it up to date over time) - partly because of #1, above, partly because
in decent multi-threaded programs there aren't too many shared objects
and the points at which they are shared are fairly well-defined, and
partly because when in doubt, it's fairly simple to add a mutex around a
More information about the Python-list