Mailman 3 December 2018 - docs

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted
by Pablo Galindo Salgado Dec. 3, 2018

Dec. 3, 2018

Pablo Galindo Salgado <pablogsal(a)gmail.com> added the comment: > I'm not comfortable with the fix. I cannot explain why but I feel like adding a strong dependency from a child to its parent is going to lead to more bugs, not less. It sounds like a recipe for reference cycles. Maybe I'm just plain wrong. The pool child objects (imap iterators, async results...etc) need to keep a reference to the parent because if not, the caller is in charge of doing that and that may lead to bugs. Is the same scenario as if I get a dictionary iterator and then I destroy my reference to the dictionary: if the iterator does not keep a reference to the parent (the dictionary) it will not be possible to be used in the future. Indeed, we can see that this is what happens: I'm not comfortable with the fix. I cannot explain why but I feel like adding a strong dependency from a child to its parent is going to lead to more bugs, not less. It sounds like a recipe for reference cycles. Maybe I'm just plain wrong. >>> x = {1:2} >>> y = iter(x) >>> gc.get_referrents(x) Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: module 'gc' has no attribute 'get_referrents' >>> gc.get_referrers(x) [<dict_keyiterator object at 0x0000024447D6D598>, {'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, 'y': <dict_keyiterator object at 0x0000024447D6D598>, 'gc': <module 'gc' (built-in)>, 'x': {1: 2}} ] We can see that the dict_keyiterator refers to the dictionary, keeping it alive. Here we have the same situation: if we do not keep the pool alive, the iterator will hang when iterating because the jobs won't get finished. >At this point, I would like that someone explains to me what the problem is. https://github.com/python/cpython/pull/10852 is a solution, ok, but what is the problem? What does the code hangs, whereas >previously it was fine? Is the example code really correct? Do we want to support such usage? The code hangs because the pool was not being destroyed before due to the reference cycle between the pool and an internal object (a Thread). Now it hangs because the worker thread is destroyed with the pool as no references are kept to it and the job that the iterator is waiting on is never finished. >I understand that msg330864 rely on black magic to expect that it's going to be fine. The lifetime of the pool is implicit and it sounds like a bad design. I don't want to endorse that. I found the weird code in the example in several projects. I have corrected it to use the pool as a context manager or to call close(), but this means that users are doing this and it used to work and not it does not: technically is a regression. ---------- _______________________________________ Python tracker <report(a)bugs.python.org> <https://bugs.python.org/issue34172> _______________________________________

1 0

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted
by STINNER Victor Dec. 3, 2018

Dec. 3, 2018

STINNER Victor <vstinner(a)redhat.com> added the comment: tzickel: > A. The documentation explicitly says: "When the pool object is garbage collected terminate() will be called immediately." (Happened till a code refactor 9 years ago introduced this bug). It is a *very bad* practice to rely on __del__. Don't do that. That's why we introduced ResourceWarning. tzickel: > https://github.com/python/cpython/blob/master/Lib/multiprocessing/util.py#L… (Till the end of the file almost) Is this API *incompatible* with pool.close()? Explicitly release resources? Pablo: > Víctor, I have a PR fixing this in: (...) The fix is really simple, so I think we should fix it and not revert the change on this case. I'm not comfortable with the fix. I cannot explain why but I feel like adding a strong dependency from a child to its parent is going to lead to more bugs, not less. It sounds like a recipe for reference cycles. Maybe I'm just plain wrong. At this point, I would like that someone explains me what the problem is. https://github.com/python/cpython/pull/10852 is a solution, ok, but what is the problem? What does the code hangs, whereas previously it was fine? Is the example code really correct? Do we want to support such usage? I understand that msg330864 rely on black magic to expect that it's going to be fine. The lifetime of the pool is implicit and it sounds like a bad design. I don't want to endorse that. ---------- _______________________________________ Python tracker <report(a)bugs.python.org> <https://bugs.python.org/issue34172> _______________________________________

1 0

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted
by tzickel Dec. 3, 2018

Dec. 3, 2018

tzickel <icebreak(a)yahoo.com> added the comment: The previous posts here touch all this subjects: A. The documentation explicitly says: "When the pool object is garbage collected terminate() will be called immediately." (Happened till a code refactor 9 years ago introduced this bug). B. Large amount of code was developed for this technique: https://github.com/python/cpython/blob/master/Lib/multiprocessing/util.py#L… (Till the end of the file almost) C. The reason I opened this bug was because I was called to see why a long running process crashes after a while, and found out it leaked tons of subprocesses / pool._cache memory. D. The quoted code, will currently simply leak each invocation lots of subprocesses... I too, think we should push for the said fix. ---------- _______________________________________ Python tracker <report(a)bugs.python.org> <https://bugs.python.org/issue34172> _______________________________________

1 0

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted
by Pablo Galindo Salgado Dec. 3, 2018

Dec. 3, 2018

Pablo Galindo Salgado <pablogsal(a)gmail.com> added the comment: Víctor, I have a PR fixing this in: issue35378 Even if is not correct to not call close or join on the Pool, this behaviour was working before while now it hangs. The fix is really simple, si I think we should fix it and not revert the change on this case. ---------- _______________________________________ Python tracker <report(a)bugs.python.org> <https://bugs.python.org/issue34172> _______________________________________

1 0

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted
by STINNER Victor Dec. 3, 2018

Dec. 3, 2018

STINNER Victor <vstinner(a)redhat.com> added the comment: """ def the_test(): print("Begin") for x in multiprocessing.Pool().imap(int, ["4", "3"]): print(x) print("End") """ Side-note: Is it correct to use a pool without calling terminate() nor close()? Should we start to emit a ResourceWarning if a pool is not closed explicitly, as we did for files, sockets, asyncio event loops and subprocess.Popen objects? ---------- _______________________________________ Python tracker <report(a)bugs.python.org> <https://bugs.python.org/issue34172> _______________________________________

1 0

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted
by STINNER Victor Dec. 3, 2018

Dec. 3, 2018

STINNER Victor <vstinner(a)redhat.com> added the comment: I suggest to revert this change, since it caused a regression: "multiprocessing.Pool.imap hangs in MacOs after applying this commit: (...)" ---------- nosy: +benjamin.peterson, ned.deily, vstinner priority: normal -> release blocker resolution: fixed -> status: closed -> open _______________________________________ Python tracker <report(a)bugs.python.org> <https://bugs.python.org/issue34172> _______________________________________

1 0

[issue22021] shutil.make_archive() root_dir do not work
by Lysandros Nikolaou Dec. 2, 2018

Dec. 2, 2018

Lysandros Nikolaou <lisandrosnik(a)gmail.com> added the comment: Ping. ---------- _______________________________________ Python tracker <report(a)bugs.python.org> <https://bugs.python.org/issue22021> _______________________________________

1 0

Learning to code
by michaeltsayre＠gmail.com Dec. 2, 2018

Dec. 2, 2018

Greetings, I'm not sure if this is the proper channel to communicate this; so I hope someone doesn't mind taking the time to read this, forward to whomever might need to see this, and of course possibly reply. My cause for reaching out is because I'm left wondering if this (https://docs.python.org/2/library/index.html) is or isn't the "bible" or "rulebook" on python usage and if it is; then would it be correct to consider this source as a technical "textbook" of sorts? As silly as that question might sound; the reason I ask is because I'm presently enrolled in a Data Analysis and Visualization Coding Bootcamp that is partnered with the University of Texas - Austin and I don't recall being formally introduced to this document. In fact, I stumbled across this while conducting a number of Google searches; trying to locate documentation for the various functions I've been exposed to at this point. Assuming that I am correct in referring to this document as a python "textbook"; then I am left with questioning the instructional reliance I've seen with Googling solutions to issues that resemble a chat forum where one user files an issue and multiple users respond with solutions/advice. If it's possible; I'd prefer to learn off of the official documentation before attempting to interpret solutions from people in forums. Thank you for your time. I look forward to a response! Very Respectfully, Michael Sayre Cell: (682)712-9741 https://www.linkedin.com/in/michaelsayre1/ "We have become a nation of employees. We are dependent upon others for our means of livelihood, and most of our people have become completely dependent upon wages. If they lose their jobs they lose every resource except for the relief supplied by the various forms of social security. Such dependence of the mass of the people upon others for all of their income is something new in the world. For our generation, the substance of life is in another man's hands." F. Tannenbaum, A Philosophy of Labor (1951)

1 0

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted
by Pablo Galindo Salgado Dec. 2, 2018

Dec. 2, 2018

Pablo Galindo Salgado <pablogsal(a)gmail.com> added the comment: As this issue was somehow still present previous to this commit, I am going to track this problem in a new issue: issue35378 ---------- status: open -> closed _______________________________________ Python tracker <report(a)bugs.python.org> <https://bugs.python.org/issue34172> _______________________________________

1 0

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted
by tzickel Dec. 2, 2018

Dec. 2, 2018

tzickel <icebreak(a)yahoo.com> added the comment: here is something quick I did to check if it works (it works) but I'm not fluent in multiprocessing code, so If i'm missing something or doing something wrong feel free to tell me: https://github.com/tzickel/cpython/commit/ec63a43706f3bf615ab7ed30fb095607f… ---------- _______________________________________ Python tracker <report(a)bugs.python.org> <https://bugs.python.org/issue34172> _______________________________________

1 0