New submission from STINNER Victor <vstinner@redhat.com>: Many developers only discover that a Python function/module is not thread safe because they have a bug in production... Some examples: * bpo-7672: ssl * bpo-8865: select.poll is not thread safe * bpo-539175, bpo-21216: socket.gethostbyname() * bpo-7980: time.strptime() * bpo-6647: warnings.catch_warnings() * bpo-11077, bpo-33479: Tkinter * bpo-1336, bpo-19809: subprocess on Python 2 * bpo-15329: deque * bpo-35275: os.umask() Hopefully, sometimes it was possible to fix it: * bpo-3139: bytearray, buffer protocol * bpo-28969: @functools.lru_cache * bpo-21291: subprocess.Popen.wait() In the asyncio documentation, I explicitly documented that, by design, most classes are not thread-safe. For example, asyncio.Lock() is *NOT* thread-safe: https://docs.python.org/dev/library/asyncio-sync.html#asyncio.Lock Maybe we should start to use a standard way to describe "thread safety". See "POSIX Safety Concepts" of the GNU libc: https://www.gnu.org/software/libc/manual/html_node/POSIX-Safety-Concepts.htm... Example with setlocale, "MT-Unsafe": https://www.gnu.org/software/libc/manual/html_node/Setting-the-Locale.html -- My own (incomplete) list of "process-wide states": https://vstinner.readthedocs.io/threads.html#process-wide ---------- assignee: docs@python components: Documentation messages: 330093 nosy: docs@python, vstinner priority: normal severity: normal status: open title: Document thread safety versions: Python 3.8 _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35276> _______________________________________
STINNER Victor <vstinner@redhat.com> added the comment: "Proces-wide" vs "thread-safe" is one thing. Another interesting property is "reentrant", but maybe that's too wide? Somehow related, iterating on a container and modify it may or may not work depending on the container type and the kind of modifications. --
bpo-8865: select.poll is not thread safe
I checked select documentation, it doesn't mention "thread" anywhere :-( https://docs.python.org/dev/library/select.html#poll-objects ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35276> _______________________________________
Raymond Hettinger <raymond.hettinger@gmail.com> added the comment: The general rule in Python is that nothing can be considered atomic unless specifically documented as atomic (such as the queue module or lru_cache which have internal locks). The only safe action is to put locks around all accesses to shared resources. We should have a FAQ entry to that effect. It should also note that "thread-safe" means different things to different people. ---------- nosy: +rhettinger _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35276> _______________________________________
Raymond Hettinger <raymond.hettinger@gmail.com> added the comment: One idea is that you could author a threading HOWTO document that covers atomicity, locks, queues, reentrancy, etc. This is a thorny topic and it would be nice to have the principles and techniques collected in one place. Ideally, it would include examples of what to expect in various situations. For example, the pure python OrderedDict can be put in an inconsistent state if two threads make updates without a mutex; however, the containers implemented in C can never be broken even if they don't guarantee atomicity (i.e. a dict update making a pure python callback to __hash__ will never result in a broken dict). ISTM that the docs have presumed that people using threading know what they're doing; however, we know that isn't always true ;-) ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35276> _______________________________________
STINNER Victor <vstinner@redhat.com> added the comment: I don't think that it would make sense to annotate *each* function in the documentation, that would require too much work. I suggest to start to annotate functions which are known to have "side effects" or are not safe whereas users can be surprised to read that they are unsafe in the context of multithreading. For example, document that os.chdir() is "process-wide" (I'm not sure how to document it). I think that everybody agrees that it's the case, and IMHO it's an useful information. At least, one user has been surprised to read that os.umask() is process-wide: "I would be great, if the python standard library would provide correspondig thread safe method." https://bugs.python.org/issue35275
One idea is that you could author a threading HOWTO document that covers atomicity, locks, queues, reentrancy, etc. This is a thorny topic and it would be nice to have the principles and techniques collected in one place.
Yes. About the definition of atomicity, thread-safe, process-wide, etc. There is an existing example: contextlib describes "Reentrant context managers". https://docs.python.org/dev/library/contextlib.html#reentrant-cms
Ideally, it would include examples of what to expect in various situations. For example, the pure python OrderedDict can be put in an inconsistent state if two threads make updates without a mutex; however, the containers implemented in C can never be broken even if they don't guarantee atomicity (i.e. a dict update making a pure python callback to __hash__ will never result in a broken dict).
This is where Python becomes obscure. Depending on the implementation, a method can be atomic or not :-/ Maybe atomicity should be documented as a "CPython implementation detail"!?
ISTM that the docs have presumed that people using threading know what they're doing; however, we know that isn't always true ;-)
I don't understand threads and I have no idea which method are thread-safe or not. Each time, I should look at the doc, read the implementation, etc. :-) ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35276> _______________________________________
STINNER Victor <vstinner@redhat.com> added the comment: Aha, another strange beast: in Python 3, locale.localeconv() now changes temporarily the LC_CTYPE locale is some cases, and this change affects other threads. That's not something expected. ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35276> _______________________________________
STINNER Victor <vstinner@redhat.com> added the comment: About system-wide: see also PR 11190, "time.monotonic() is now always available and always system-wide." ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35276> _______________________________________
Change by Sergey Fedoseev <fedoseev.sergey@gmail.com>: ---------- nosy: +sir-sigurd _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35276> _______________________________________
participants (3)
-
Raymond Hettinger -
Sergey Fedoseev
-
STINNER Victor