Mailman 3 July 2016 - docs

[issue27422] Deadlock when mixing threading and multiprocessing
by R. David Murray 04 Jul '16

04 Jul '16

R. David Murray added the comment: To clarify the GIL issue (for davin, I guess? :): if the library you are using to interface with the FORTRAN code drops the GIL before calling the FORTRAN, then you *can* take advantage of multiple cores. It is only the python code (and some of the code interacting with the python objects) that is limited to executing on one core at a time. (As far as I know it isn't restricted to be the *same* core unless you set CPU affinity somehow, and I have no idea if it improves performance to use CPU affinity or not). ---------- _______________________________________ Python tracker <report(a)bugs.python.org> <http://bugs.python.org/issue27422> _______________________________________

1 0

[issue27422] Deadlock when mixing threading and multiprocessing
by Martin Ritter 04 Jul '16

04 Jul '16

Martin Ritter added the comment: Dear Davin, Thanks for the input, I was perfectly aware that the "solution" I proposed is not realistic. But the feedback that multiprocessing is using threads internally is useful as I can quickly abandon the idea to do something like the check I proposed in our code base without spending time on it. I was aware of the Gil, I just did not anticipate that big a problem when mixing threads and processes with rather simple python code. My bad, sorry for the noise. Cheers, Martin ---------- _______________________________________ Python tracker <report(a)bugs.python.org> <http://bugs.python.org/issue27422> _______________________________________

1 0

[issue27422] Deadlock when mixing threading and multiprocessing
by Davin Potts 04 Jul '16

04 Jul '16

Davin Potts added the comment: While I believe I understand the motivation behind the suggestion to detect when the code is doing something potentially dangerous, I'll point out a few things: * any time you ask for a layer of convenience, you must choose something to sacrifice to get it (usually performance is sacrificed) and this sacrifice will affect all code (including non-problematic code) * behind the scenes multiprocessing itself is employing multiple threads in the creation and coordination between processes -- "checking to see if there are multiple threads active on process creation" is therefore a more complicated request than it maybe first appears * Regarding "python makes it very easy to mix these two", I'd say it's nearly as easy to mix the two in C code -- the common pattern across different languages is to learn the pros+cons+gotchyas of working with processes and threads I too come from the world of scientific software and the mixing of Fortran, C/C++, and Python (yay science and yay Fortran) so I'll make another point (apologies if you already knew this): There's a lot of computationally intensive code in scientific code/applications and being able to perform those computations in parallel is a wonderful thing. I am unsure if the tests you're trying to speed up exercise compute-intensive functions but let's assume they do. For reasons not described here, using the CPython implementation, there is a constraint on the use of threads that restricts them to all run on a single core of your multi-core cpu (and on only one cpu if you have an SMP system). Hence spinning up threads to perform compute intensive tasks will likely result in no better throughput (no speedup) because they're all fighting over the same maxed-out core. To spread out onto and take advantage of multiple cores (and multiple cpus on an SMP system) you will want switch to creating processes (as you say you now have). I'd make the distinction that you are likely much more interested in 'parallel computing' than 'concurrent execution'. Since you're already using multiprocessing you might also simply use `multiprocessing.Pool`. ---------- _______________________________________ Python tracker <report(a)bugs.python.org> <http://bugs.python.org/issue27422> _______________________________________

1 0

[issue27422] Deadlock when mixing threading and multiprocessing
by Martin Ritter 04 Jul '16

04 Jul '16

Martin Ritter added the comment: I agree that this is error prone and can not be fixed reliably on the python side. However, python makes it very easy to mix these two, a user might not even notice it if a function he calls uses fork and thus just use a ThreadPoolExecutor() because it's the simplest thing to do. What could be an nice solution in my opinion if the multiprocessing module could check if there are already multiple threads active on process creation and issue a warning if so. This warning could of course be optional but would make this issue more obvious. In my case we have a large C++ code base which still includes a lot of Fortran 77 code with common blocks all over the place (yay science). Everything is interfaced in python so to make sure that I do not have any side effects I run the some of the functions in a fork using multiprocessing.Process(). And in this case I just wanted to run some testing in parallel. I now switched to a ProcessPoolExecutor which works fine but for me. ---------- _______________________________________ Python tracker <report(a)bugs.python.org> <http://bugs.python.org/issue27422> _______________________________________

1 0

python cam style docs on mac
by Manikandan Bakthavatchalam 03 Jul '16

03 Jul '16

Hi, First, thanks for this wonderful language. I am using python on my mac OSX El Capitan. Where can I find the offline help files on my mac? In windows, I see people using the offline chm. This is very useful while programming without having to browse the internet for docs each time. regards, Manikandan

2 1

How best to submit a change?
by Jim Belton 03 Jul '16

03 Jul '16

Hi all: I'm more than happy to hack some restructured text. How best to submit a patch to a doc? I'm not going to take the time to improve unless I know how to get my change included. I'd be hacking the 2.7 branch, since that's the version I seem to be working in. Hopefully, 3 will start getting more traction in real world dev.ops. environments soon. Thanks, Jim.

2 1

[issue27429] xml.sax.saxutils.escape doesn't escape multiple characters safely
by Xiang Zhang 03 Jul '16

03 Jul '16

Xiang Zhang added the comment: Put a not in escape's doc. ---------- Added file: http://bugs.python.org/file43617/sax_escape_doc.patch _______________________________________ Python tracker <report(a)bugs.python.org> <http://bugs.python.org/issue27429> _______________________________________

1 0

[issue27429] xml.sax.saxutils.escape doesn't escape multiple characters safely
by Serhiy Storchaka 03 Jul '16

03 Jul '16

Changes by Serhiy Storchaka <storchaka+cpython(a)gmail.com>: ---------- assignee: -> docs@python components: +Documentation nosy: +docs@python type: behavior -> enhancement _______________________________________ Python tracker <report(a)bugs.python.org> <http://bugs.python.org/issue27429> _______________________________________

1 0

[issue27422] Deadlock when mixing threading and multiprocessing
by Raymond Hettinger 02 Jul '16

02 Jul '16

Raymond Hettinger added the comment: FWIW, this isn't even a Python specific behavior. It is just how threads, locks, and processes work (or in this case don't work). The code is doing what it is told to do which happens to not be what you want (i.e. a user bug rather than a Python bug). I think a FAQ entry would be a reasonable place to mention this (it comes up more often than one would hope). ---------- assignee: -> docs@python components: +Documentation nosy: +docs@python resolution: -> not a bug _______________________________________ Python tracker <report(a)bugs.python.org> <http://bugs.python.org/issue27422> _______________________________________

1 0

Usage error in the documentation for the “del” statement
by David Wheelock 01 Jul '16

01 Jul '16

Usage error in the documentation for the “del” statement here: https://docs.python.org/2/reference/simple_stmts.html#del In the following statement: “Deletion is recursively defined very similar…” “similar” should be changed to “similarly” David Wheelock, GISP Application Specialist david.wheelock(a)ag.ok.gov<mailto:david.wheelock@ag.ok.gov> Oklahoma Department of Agriculture, Food and Forestry 2800 North Lincoln Boulevard Oklahoma City, OK 73105-4298 Phone: (405) 522-5477 FAX: (405) 522-3590 http://ag.ok.gov<http://ag.ok.gov/> "We are usually convinced more easily by reasons we have found ourselves, than by those which have occurred to others." Pascal

1 0