Mailman 3 December 2018 - docs

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted
by STINNER Victor Dec. 4, 2018

Dec. 4, 2018

STINNER Victor <vstinner(a)redhat.com> added the comment: multiprocessing should help the developer to detect when the API is misused. For example, emit a ResourceWarning if a pool is not released explicitly. It might help to detect such bugs: * bpo-33676 * bpo-35413 ---------- _______________________________________ Python tracker <report(a)bugs.python.org> <https://bugs.python.org/issue34172> _______________________________________

1 0

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted
by Pablo Galindo Salgado Dec. 4, 2018

Dec. 4, 2018

Pablo Galindo Salgado <pablogsal(a)gmail.com> added the comment: > that there is code in the stdlib that holdes reference between child and parent Just to clarify: is not that is just code in the stdlib that keeps a reference between child and parent. The examples I have given are the exact same situation that we have here: the iterator object associated with another needs to keep its parent alive to work correctly. ---------- _______________________________________ Python tracker <report(a)bugs.python.org> <https://bugs.python.org/issue34172> _______________________________________

1 0

[issue35395] Typo in asyncio eventloop documentation
by Naglis Dec. 4, 2018

Dec. 4, 2018

New submission from Naglis <naglis(a)mailbox.org>: loop.add_writer and loop.add_signal_handler have *callback* in their signatures, but in their documentation regarding functools.partial usage the function is referred to as *func*. ---------- assignee: docs@python components: Documentation messages: 330989 nosy: docs@python, naglis priority: normal severity: normal status: open title: Typo in asyncio eventloop documentation _______________________________________ Python tracker <report(a)bugs.python.org> <https://bugs.python.org/issue35395> _______________________________________

3 7

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted
by tzickel Dec. 4, 2018

Dec. 4, 2018

tzickel <icebreak(a)yahoo.com> added the comment: Reverting the code will cause another class of problems, like the reason I fixed it. Programs written such as the example that Pablo gave (and what I've seen) will quietly leak child processes, file descriptors (for the pipes) and memory to a variety degree might not be detected, or in the end detected in a big error or crash. Also in some ResourceWarnings (if not all), the resources are closed in the end (like in sockets), here without this code patch you cannot implicitly reclaim the resources (because there is a Thread involved here), which I think is a high bar for the user to think about. You can also enable multiprocessing's debug logging to see how the code behaves with and without the fix: https://stackoverflow.com/a/1353037 I also agree with Pablo that there is code in the stdlib that holdes reference between child and parent. There is also code that has circular reference (for example Python 2's OrderedDict) and that is ok as well (not that this is the situation here). ---------- _______________________________________ Python tracker <report(a)bugs.python.org> <https://bugs.python.org/issue34172> _______________________________________

1 0

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted
by Pablo Galindo Salgado Dec. 3, 2018

Dec. 3, 2018

Pablo Galindo Salgado <pablogsal(a)gmail.com> added the comment: > I disagree that a child should keep its parent alive. But this is normal across the standard library. For example, here is how a deque iterator keeps the deque alive: >>> x = deque([1,2,3]) >>> deque_iter = iter(x) >>> deque_weakref = weakref.ref(x) >>> del x >>> gc.collect() >>> gc.get_referrers(deque_weakref()) [<_collections._deque_iterator object at 0x0000024447ED6EA8>] Here, the deque iterator is the *only* reference to the deque. When we destroy it, the deque is destroyed: >>> del deque_iter >>> gc.collect() >>> deque_weakref() None ---------- _______________________________________ Python tracker <report(a)bugs.python.org> <https://bugs.python.org/issue34172> _______________________________________

1 0

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted
by STINNER Victor Dec. 3, 2018

Dec. 3, 2018

STINNER Victor <vstinner(a)redhat.com> added the comment: > I found the weird code in the example in several projects. I have corrected it to use the pool as a context manager or to call close(), but this means that users are doing this and it used to work and not it does not: technically is a regression. That's why I'm asking for a revert :-) I suggest to revert this change immediately from 2.7, 3.6 and 3.7, and later see what can be done for master. Even if we design carefully an API, there will be always someone to misuse it :-) I would prefer to stop promoting such bad code and find a transition to more correct code. I disagree that a child should keep its parent alive. I would be ok to use a *weak reference* from the child to the parent to detect when the parent goes away, and maybe trigger an action in that case. For example, raise an exception or log a warning. ---------- _______________________________________ Python tracker <report(a)bugs.python.org> <https://bugs.python.org/issue34172> _______________________________________

1 0

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted
by Pablo Galindo Salgado Dec. 3, 2018

Dec. 3, 2018

Pablo Galindo Salgado <pablogsal(a)gmail.com> added the comment: > I'm not comfortable with the fix. I cannot explain why but I feel like adding a strong dependency from a child to its parent is going to lead to more bugs, not less. It sounds like a recipe for reference cycles. Maybe I'm just plain wrong. The pool child objects (imap iterators, async results...etc) need to keep a reference to the parent because if not, the caller is in charge of doing that and that may lead to bugs. Is the same scenario as if I get a dictionary iterator and then I destroy my reference to the dictionary: if the iterator does not keep a reference to the parent (the dictionary) it will not be possible to be used in the future. Indeed, we can see that this is what happens: I'm not comfortable with the fix. I cannot explain why but I feel like adding a strong dependency from a child to its parent is going to lead to more bugs, not less. It sounds like a recipe for reference cycles. Maybe I'm just plain wrong. >>> x = {1:2} >>> y = iter(x) >>> gc.get_referrers(x) [<dict_keyiterator object at 0x0000024447D6D598>, {'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, 'y': <dict_keyiterator object at 0x0000024447D6D598>, 'gc': <module 'gc' (built-in)>, 'x': {1: 2}} ] We can see that the dict_keyiterator refers to the dictionary, keeping it alive. Here we have the same situation: if we do not keep the pool alive, the iterator will hang when iterating because the jobs won't get finished. >At this point, I would like that someone explains to me what the problem is. https://github.com/python/cpython/pull/10852 is a solution, ok, but what is the problem? What does the code hangs, whereas >previously it was fine? Is the example code really correct? Do we want to support such usage? The code hangs because the pool was not being destroyed before due to the reference cycle between the pool and an internal object (a Thread). Now it hangs because the worker thread is destroyed with the pool as no references are kept to it and the job that the iterator is waiting on is never finished. >I understand that msg330864 rely on black magic to expect that it's going to be fine. The lifetime of the pool is implicit and it sounds like a bad design. I don't want to endorse that. I found the weird code in the example in several projects. I have corrected it to use the pool as a context manager or to call close(), but this means that users are doing this and it used to work and not it does not: technically is a regression. ---------- _______________________________________ Python tracker <report(a)bugs.python.org> <https://bugs.python.org/issue34172> _______________________________________

1 0

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted
by Pablo Galindo Salgado Dec. 3, 2018

Dec. 3, 2018

Change by Pablo Galindo Salgado <pablogsal(a)gmail.com>: ---------- Removed message: https://bugs.python.org/msg330995 _______________________________________ Python tracker <report(a)bugs.python.org> <https://bugs.python.org/issue34172> _______________________________________

1 0

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted
by Pablo Galindo Salgado Dec. 3, 2018

Dec. 3, 2018

Pablo Galindo Salgado <pablogsal(a)gmail.com> added the comment: > I'm not comfortable with the fix. I cannot explain why but I feel like adding a strong dependency from a child to its parent is going to lead to more bugs, not less. It sounds like a recipe for reference cycles. Maybe I'm just plain wrong. The pool child objects (imap iterators, async results...etc) need to keep a reference to the parent because if not, the caller is in charge of doing that and that may lead to bugs. Is the same scenario as if I get a dictionary iterator and then I destroy my reference to the dictionary: if the iterator does not keep a reference to the parent (the dictionary) it will not be possible to be used in the future. Indeed, we can see that this is what happens: >>> x = {1:2} >>> y = iter(x) >>> gc.get_referrents(x) Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: module 'gc' has no attribute 'get_referrents' >>> gc.get_referrers(x) [<dict_keyiterator object at 0x0000024447D6D598>, {'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, 'y': <dict_keyiterator object at 0x0000024447D6D598>, 'gc': <module 'gc' (built-in)>, 'x': {1: 2}} ] We can see that the dict_keyiterator refers to the dictionary, keeping it alive. Here we have the same situation: if we do not keep the pool alive, the iterator will hang when iterating because the jobs won't get finished. >At this point, I would like that someone explains to me what the problem is. https://github.com/python/cpython/pull/10852 is a solution, ok, but what is the problem? What does the code hangs, whereas >previously it was fine? Is the example code really correct? Do we want to support such usage? The code hangs because the pool was not being destroyed before due to the reference cycle between the pool and an internal object (a Thread). Now it hangs because the worker thread is destroyed with the pool as no references are kept to it and the job that the iterator is waiting on is never finished. >I understand that msg330864 rely on black magic to expect that it's going to be fine. The lifetime of the pool is implicit and it sounds like a bad design. I don't want to endorse that. I found the weird code in the example in several projects. I have corrected it to use the pool as a context manager or to call close(), but this means that users are doing this and it used to work and not it does not: technically is a regression. ---------- _______________________________________ Python tracker <report(a)bugs.python.org> <https://bugs.python.org/issue34172> _______________________________________

1 0

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted
by Pablo Galindo Salgado Dec. 3, 2018

Dec. 3, 2018

Change by Pablo Galindo Salgado <pablogsal(a)gmail.com>: ---------- Removed message: https://bugs.python.org/msg330994 _______________________________________ Python tracker <report(a)bugs.python.org> <https://bugs.python.org/issue34172> _______________________________________

1 0