<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">2015-07-29 8:29 GMT+02:00 Sven R. Kunze <span dir="ltr"><<a href="mailto:srkunze@mail.de" target="_blank">srkunze@mail.de</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Thanks Ludovic.<span><br>
<br>
<div>On 28.07.2015 22:15, Ludovic Gasc
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hello,</div>
<div><br>
</div>
<div>This discussion is pretty interesting to try to list when
each architecture is the most efficient, based on the need.</div>
<div><br>
</div>
<div>However, just a small precision: multiprocess/multiworker
isn't antinomic with AsyncIO: You can have an event loop in
each process to try to combine the "best" of two "worlds".</div>
<div>As usual in IT, it isn't a silver bullet that will care the
cancer, however, at least to my understanding, it should be
useful for some business needs like server daemons.</div>
</div>
</blockquote>
<br></span>
I think that should be clear for everybody using any of these
modules. But you are right to point it out explicitly.</div></blockquote><div><br></div><div>Based on my discussions at EuroPython and PyCON-US, it's certainly clear for the middle-class management of Python community, however, not really from the typical Python end-dev: Several persons tried to troll me that multiprocessing is more efficient than AsyncIO.<br></div><div><br></div><div>To me, it was a opportunity to transform the negative troll attempt to a positive exchange about efficiency and understand before to troll ;-)</div><div>More seriously, I've the feeling that it isn't very clear for everybody, especially for the new comers.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><span><blockquote type="cite"><div dir="ltr"><div>It isn't a crazy new idea, this design pattern is
implemented since a long time ago at least in Nginx: <a href="http://www.aosabook.org/en/nginx.html" target="_blank">http://www.aosabook.org/en/nginx.html</a><br></div>
<div><br>
</div>
<div>If you are interested in to use this design pattern to
build a HTTP server only, you can use easily
aiohttp.web+gunicorn: <a href="http://aiohttp.readthedocs.org/en/stable/gunicorn.html" target="_blank">http://aiohttp.readthedocs.org/en/stable/gunicorn.html</a></div>
<div>If you want to use any AsyncIO server protocol
(aiohttp.web, panoramisk, asyncssh, irc3d), you can use
API-Hour: <a href="http://www.api-hour.io" target="_blank">http://www.api-hour.io</a></div>
<div><br>
</div>
<div>And if you want to implement by yourself this design
pattern, be my guest, if a Python peon like me has implemented
API-Hour, everybody on this mailing-list can do that.</div>
<div><br>
</div>
<div>For communication between workers, I use Redis, however,
you have plenty of solutions to do that.</div>
<div>As usual, before to select a communication mechanism you
should benchmark based on your use cases: some results should
surprise you.</div>
<div><br>
</div>
</div>
</blockquote>
<br></span>
I hope not to disappoint you. </div></blockquote><div><br></div><div>Don't worry for that, don't hesitate to "hit", I have a very strong shield to avoid disappointments ;-)</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000">I actually strive not to do that
manually for each tiny bit of program</div></blockquote><div><br></div><div>You're right, micro-benchmarks isn't a good approach to decide macro architecture of application.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"> (assuming there are many place
in the code base where a project could benefit from concurrency).</div></blockquote><div><br></div><div>As usual, depends on your architecture/need.</div><div>If you do a lot of network than CPU usage, the waiting time of network should play for more concurrency.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000">
Personally, I use benchmarks for optimizing problematic code.<br>
<br>
But if Python would be able to do that without choosing the right
and correctly configured approach (to be determined by benchmarks)
that would be awesome. As usual, that needs time to evolve.<br></div></blockquote><div><br></div><div>It should technically possible, however, I don't believe too much in implicit hidden optimizations to the end-dev: It's very complicated to hide the magic, few people have the skills to implement that, and the day you have an issue, you're almost alone.</div><div>See PyPy: certainly one day they will provide a good solution for that, however, it isn't trivial to implement, see the time they need.</div><div><br></div><div>With the time, I believe more and more to educate developers who help them to understand the big picture and use explicitly optimizations: The learning curve is more important, however, at the end, you have more autonomous developers who will resolve more problems and less afraid to break the standard frame to innovate.</div><div><br></div><div>I don't have scientific proof of that, it's only a feeling.</div><div>However, again both approaches aren't antinomic: Each time we have an automagic optimization like computed gotos without side effects, I will use that.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000">I found that benchmark resulted improvements do not last forever,
unfortunately, and that most of the time nobody is able to keep
track of everything. So, as soon as something changes, you need to
start anew. That is not acceptable for me.<br></div></blockquote><div><br></div><div>I'm fully agree with you: Until it works, don't break for the pleasure.<br>Moreover, instead of to trash your full stack for efficiency reasons (For example, drop all your Python code to migrate to Go) where you need to relearn everything, you should maybe first find a solution in your actual stack.</div><div>At least to me, it was very less complicated to migrate to Python 3, multiworker pattern and AsyncIO than to migrate to Go/NodeJS/Erlang/...</div><div>Moreover, with a niche language, it's more complicated to find developers and harder to spot impostors:</div><div>Some people use alternative languages not really used only to try to convince others who are good developers.</div><div>Another solution is also to add more servers to handle load, but it isn't always the solution with the smallest TCO, don't forget to count sysadmin costs+complexity to debug when you have an issue on your production.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000">Btw. that is also a reason why a I said recently (another topic on
this list), 'if Python could optimize that without my attention that
would be great'. The simplest solution and therefore the easiest to
comprehend for all team members is the way to go.</div></blockquote><div><br></div><div>Again, I'm strongly agree with you, however, with the age of Python and the big size of performance community we have (PyPy, Numba, Cython, Pyston...) I believe that less and less automagic solutions without side effects will be find. Not impossible, but harder and harder (I secretly hope that somebody will prove me I was wrong ;-) )</div><div>Maybe to "steal" some optimizations from others languages ?</div><div>I don't have the technical level to help for that, I'm more a business logic dev than a low level dev.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"> If that is not
efficient enough that is actually a Python issue. Readability counts
most. And fortunately, most of the cases that attitude works
perfectly with Python. :)</div></blockquote><div><br></div><div>Again and again, I'm agree with you: the combo size of community (big toolbox and a lot of developers) + readability to be newcomer friendly is clearly a big win-win, at least to me.</div><div>The only issue I had it was efficiency: with the success of our company, we couldn't be stopped by the programming language/framework to build quickly efficient daemons, it's why I've dropped quickly PHP and Ruby in the past.</div><div>Now, with our new stack, based on the trusted predictions of our fortune-telling telephony service department, we could survive a long time before to replace some Python parts with C or other.</div><div><br></div><div>Have a nice week-end.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><div><div><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>Have a nice week.</div>
<div><br>
</div>
<div>PS: Thank you everybody for EuroPython, it was amazing ;-)</div>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div>
<div dir="ltr">
<div>--<br>
<div style="font-size:small">
<div>Ludovic Gasc (GMLudo)</div>
</div>
<div style="font-size:small"><a href="http://www.gmludo.eu/" target="_blank">http://www.gmludo.eu/</a></div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">2015-07-26 23:26 GMT+02:00 Sven R.
Kunze <span dir="ltr"><<a href="mailto:srkunze@mail.de" target="_blank">srkunze@mail.de</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <font face="monospace"><font face="sans-serif">Next update:</font><br>
<br>
<br>
Improving Performance by Running Independent Tasks
Concurrently - A Survey<br>
<br>
<br>
| </font><font face="monospace"><font face="monospace">processes</font> | </font><font face="monospace"><font face="monospace">threads</font>
| </font><font face="monospace"><font face="monospace">coroutines</font> <br>
---------------+-------------------------+----------------------------+-------------------------<br>
</font><font face="monospace"><font face="monospace"><font face="monospace">purpose | cpu-bound
tasks | cpu- & i/o-bound tasks |
i/o-bound tasks <br>
</font> |
| |
<br>
</font><span>managed by | os
scheduler | os scheduler + interpreter | </span></font><font face="monospace"><font face="monospace">customizable </font>event
loop <br>
controllable | no |
no | yes <br>
</font><font face="monospace"><span><font face="monospace">
|
|
| <br>
</font>parallelism | yes |
depends (cf. GIL) | no <br>
</span> switching | at any time | after
any bytecode | at user-defined points <br>
<span> shared state | no
| yes | yes
<br>
</span></font><font face="monospace">
| |
| <br>
startup impact | biggest/medium* |
medium | smallest <br>
cpu impact** | biggest |
medium | smallest <br>
<span> memory impact | biggest
| medium | smallest
<br>
</span></font><font face="monospace"><font face="monospace"><font face="monospace">
|
|
| <br>
</font><font face="monospace"><font face="monospace">pool
</font>module | multiprocessing.Pool |
multiprocessing.dummy.Pool | asyncio.BaseEventLoop
<br>
</font><font face="monospace"><font face="monospace">solo
</font>module | multiprocessing.Process |
threading.Thread | ---
<br>
</font></font><br>
<br>
*<br>
biggest - if spawn (fork+exec) and always on Windows<br>
medium - if fork alone<br>
<br>
**<br>
due to context switching</font><span><font face="monospace"><br>
</font><br>
<br>
<div>On 26.07.2015 14:18, Paul Moore wrote:<br>
</div>
<blockquote type="cite">
<pre>Just as a note - even given the various provisos and "it's not that
simple" comments that have been made, I found this table extremely
useful. Like any such high-level summary, I expect to have to take it
with a pinch of salt, but I don't see that as an issue - anyone who
doesn't fully appreciate that there are subtleties, probably wouldn't
read a longer explanation anyway.
So many thanks for taking the time to put this together (and for
continuing to improve it).</pre>
</blockquote>
</span> You are welcome. :)<span><br>
<blockquote type="cite">
<pre>+1 on something like this ending up in the Python docs somewhere.
</pre>
</blockquote>
</span> Not sure how the process for this is but I think
the Python gurus will find a way.<br>
</div>
<br>
_______________________________________________<br>
Python-ideas mailing list<br>
<a href="mailto:Python-ideas@python.org" target="_blank">Python-ideas@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-ideas" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/python-ideas</a><br>
Code of Conduct: <a href="http://python.org/psf/codeofconduct/" rel="noreferrer" target="_blank">http://python.org/psf/codeofconduct/</a><br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br></div></div>