[Python-ideas] Concurrency Modules
Sven R. Kunze
srkunze at mail.de
Wed Jul 29 08:29:15 CEST 2015
On 28.07.2015 22:15, Ludovic Gasc wrote:
> This discussion is pretty interesting to try to list when each
> architecture is the most efficient, based on the need.
> However, just a small precision: multiprocess/multiworker isn't
> antinomic with AsyncIO: You can have an event loop in each process to
> try to combine the "best" of two "worlds".
> As usual in IT, it isn't a silver bullet that will care the cancer,
> however, at least to my understanding, it should be useful for some
> business needs like server daemons.
I think that should be clear for everybody using any of these modules.
But you are right to point it out explicitly.
> It isn't a crazy new idea, this design pattern is implemented since a
> long time ago at least in Nginx: http://www.aosabook.org/en/nginx.html
> If you are interested in to use this design pattern to build a HTTP
> server only, you can use easily aiohttp.web+gunicorn:
> If you want to use any AsyncIO server protocol (aiohttp.web,
> panoramisk, asyncssh, irc3d), you can use API-Hour: http://www.api-hour.io
> And if you want to implement by yourself this design pattern, be my
> guest, if a Python peon like me has implemented API-Hour, everybody on
> this mailing-list can do that.
> For communication between workers, I use Redis, however, you have
> plenty of solutions to do that.
> As usual, before to select a communication mechanism you should
> benchmark based on your use cases: some results should surprise you.
I hope not to disappoint you. I actually strive not to do that manually
for each tiny bit of program (assuming there are many place in the code
base where a project could benefit from concurrency). Personally, I use
benchmarks for optimizing problematic code.
But if Python would be able to do that without choosing the right and
correctly configured approach (to be determined by benchmarks) that
would be awesome. As usual, that needs time to evolve.
I found that benchmark resulted improvements do not last forever,
unfortunately, and that most of the time nobody is able to keep track of
everything. So, as soon as something changes, you need to start anew.
That is not acceptable for me.
Btw. that is also a reason why a I said recently (another topic on this
list), 'if Python could optimize that without my attention that would be
great'. The simplest solution and therefore the easiest to comprehend
for all team members is the way to go. If that is not efficient enough
that is actually a Python issue. Readability counts most. And
fortunately, most of the cases that attitude works perfectly with Python. :)
> Have a nice week.
> PS: Thank you everybody for EuroPython, it was amazing ;-)
> Ludovic Gasc (GMLudo)
> 2015-07-26 23:26 GMT+02:00 Sven R. Kunze <srkunze at mail.de
> <mailto:srkunze at mail.de>>:
> Next update:
> Improving Performance by Running Independent Tasks Concurrently -
> A Survey
> | processes | threads
> | coroutines
> purpose | cpu-bound tasks | cpu- & i/o-bound
> tasks | i/o-bound tasks
> | | |
> managed by | os scheduler | os scheduler +
> interpreter | customizable event loop
> controllable | no |
> no | yes
> | | |
> parallelism | yes | depends (cf.
> GIL) | no
> switching | at any time | after any
> bytecode | at user-defined points
> shared state | no | yes | yes
> | | |
> startup impact | biggest/medium* |
> medium | smallest
> cpu impact** | biggest |
> medium | smallest
> memory impact | biggest | medium | smallest
> | | |
> pool module | multiprocessing.Pool |
> multiprocessing.dummy.Pool | asyncio.BaseEventLoop
> solo module | multiprocessing.Process |
> threading.Thread | ---
> biggest - if spawn (fork+exec) and always on Windows
> medium - if fork alone
> due to context switching
> On 26.07.2015 14:18, Paul Moore wrote:
>> Just as a note - even given the various provisos and "it's not that
>> simple" comments that have been made, I found this table extremely
>> useful. Like any such high-level summary, I expect to have to take it
>> with a pinch of salt, but I don't see that as an issue - anyone who
>> doesn't fully appreciate that there are subtleties, probably wouldn't
>> read a longer explanation anyway.
>> So many thanks for taking the time to put this together (and for
>> continuing to improve it).
> You are welcome. :)
>> +1 on something like this ending up in the Python docs somewhere.
> Not sure how the process for this is but I think the Python gurus
> will find a way.
> Python-ideas mailing list
> Python-ideas at python.org <mailto:Python-ideas at python.org>
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Python-ideas