[pypy-dev] Pypy Benchmark of Tornado.
Jolitz Ben - bjolit
Ben.Jolitz at acxiom.com
Fri Sep 5 18:47:22 CEST 2014
I don’t have specific suggestions on Mongo, but I can share what I’ve learned in a few months of using PyPy and Tornado.
You want to make use of CFFI in PyPy to accelerate operations that would usually be slow in Python, namely encryption and database drivers. But always test first to see if you really need to go to C.
/Any/ CPython C-Extensions will torpedo performance. Anything blocking (like the majority of DB drivers) will similarly destroy Tornado performance.
Code that is overly dynamic also does lousy on PyPy. If your driver has a ton of paths or makes idiot use of threading.Lock, expect to have an uphill struggle in optimization.
When in doubt, ask yourself if the algorithm is appropriate.
If you can’t make the Python driver performant and there exists a C-API for it, then it is trivial to wrap it with CFFI.
If it doesn’t support nonblocking operations, you can find alternatives. For example in MySQLdb, others have found you can add the Connection._fd to the IOLoop and use it to do a send_query, read_query. If you still can’t find an alternative to a blocking call, you can still mimic nonblocking IO by using pthreads, a work queue and a callback pthread. It’s not perfect and theres a lot you can optimize, but it can easily allow you to delegate long running operations to C.
Another thing I learned was to avoid generating Python-side c callback pointers frequently. If C is going to callback into Python with your result to a unique Request, you’re going to need to tag it appropriately. I prefer to use dictionaries and the callback attribute handed to me by gen.Task and pass a unique-enough key to C to callback in with.
As with everything in optimization, profile your code first. If you’re losing speed heavily somewhere else, then the above will only serve to distract you.
Cheers,
Ben
From: Phyo Arkar <phyo.arkarlwin at gmail.com<mailto:phyo.arkarlwin at gmail.com>>
Date: Thursday, September 4, 2014 at 1:55 PM
To: Ben Jolitz <Ben.Jolitz at acxiom.com<mailto:Ben.Jolitz at acxiom.com>>
Cc: pypy-dev <pypy-dev at python.org<mailto:pypy-dev at python.org>>
Subject: Re: [pypy-dev] Pypy Benchmark of Tornado.
Thanks alot Ben,
Ok , as PyPy is choice of quora and they also use tornado , i might keep testing on larget projects.
How about mongodb performance on PyPy , i heard its slower due to no C Extension (no CFFI) for pypy.
Your suggestion will be very appreciated.
On Thu, Sep 4, 2014 at 4:56 AM, Jolitz Ben - bjolit <Ben.Jolitz at acxiom.com<mailto:Ben.Jolitz at acxiom.com>> wrote:
I use Tornado and have found PyPy can yield a 30-50% performance increase for a moderately complex project.
Ben
From: Phyo Arkar <phyo.arkarlwin at gmail.com<mailto:phyo.arkarlwin at gmail.com>>
Date: Wednesday, September 3, 2014 at 8:21 AM
To: pypy-dev <pypy-dev at python.org<mailto:pypy-dev at python.org>>
Subject: Re: [pypy-dev] Pypy Benchmark of Tornado.
I expect pypy to be faster in those cases but select io is not cpu intensive thing to do so no real benefit using pypy here i guess.
On Wed, Sep 3, 2014 at 4:43 PM, Phyo Arkar <phyo.arkarlwin at gmail.com<mailto:phyo.arkarlwin at gmail.com>> wrote:
>
> It just return json document with a few thousand characters (1053 bytes)
> $siege -c 400 -t 20s -r 2000 http://localhost:9999/js
>
> Python 2.7.7:
>
> Lifting the server siege... done.
>
> Transactions: 14478 hits
> Availability: 100.00 %
> Elapsed time: 19.10 secs
> Data transferred: 14.54 MB
> Response time: 0.01 secs
> Transaction rate: 758.01 trans/sec
> Throughput: 0.76 MB/sec
> Concurrency: 8.91
> Successful transactions: 14478
> Failed transactions: 0
> Longest transaction: 1.08 seconds
> Shortest transaction: 0.00
>
> pypy-2.3.1 stable:
>
> Transactions: 15149 hits
> Availability: 100.00 %
> Elapsed time: 19.63 secs
> Data transferred: 15.21 MB
> Response time: 0.02 secs
> Transaction rate: 771.73 trans/sec
> Throughput: 0.77 MB/sec
> Concurrency: 11.92
> Successful transactions: 15149
> Failed transactions: 0
> Longest transaction: 1.09 seconds
> Shortest transaction: 0.00
>
>
>
> pypy--c-jit-73283-912dd9df99a8-linux64 (latest nightly build)
>
> Lifting the server siege... done.
>
> Transactions: 14361 hits
> Availability: 100.00 %
> Elapsed time: 19.13 secs
> Data transferred: 14.42 MB
> Response time: 0.03 secs
> Transaction rate: 750.71 trans/sec
> Throughput: 0.75 MB/sec
> Concurrency: 21.53
> Successful transactions: 14361
> Failed transactions: 0
> Longest transaction: 3.03 seconds
> Shortest transaction: 0.00
>
>
>
>
> Pypy Nightly have some request Randomly get to 3.0 Seconds , normally those requests (in Cpython) are only ~0.001 to 0.002 sec.
>
It just return json document with a few thousand characters (1053 bytes)
$siege -c 400 -t 20s -r 2000 http://localhost:9999/js
Python 2.7.7:
Lifting the server siege... done.
Transactions: 14478 hits
Availability: 100.00 %
Elapsed time: 19.10 secs
Data transferred: 14.54 MB
Response time: 0.01 secs
Transaction rate: 758.01 trans/sec
Throughput: 0.76 MB/sec
Concurrency: 8.91
Successful transactions: 14478
Failed transactions: 0
Longest transaction: 1.08 seconds
Shortest transaction: 0.00
pypy-2.3.1 stable:
Transactions: 15149 hits
Availability: 100.00 %
Elapsed time: 19.63 secs
Data transferred: 15.21 MB
Response time: 0.02 secs
Transaction rate: 771.73 trans/sec
Throughput: 0.77 MB/sec
Concurrency: 11.92
Successful transactions: 15149
Failed transactions: 0
Longest transaction: 1.09 seconds
Shortest transaction: 0.00
pypy--c-jit-73283-912dd9df99a8-linux64 (latest nightly build)
Lifting the server siege... done.
Transactions: 14361 hits
Availability: 100.00 %
Elapsed time: 19.13 secs
Data transferred: 14.42 MB
Response time: 0.03 secs
Transaction rate: 750.71 trans/sec
Throughput: 0.75 MB/sec
Concurrency: 21.53
Successful transactions: 14361
Failed transactions: 0
Longest transaction: 3.03 seconds
Shortest transaction: 0.00
Pypy Nightly have some request Randomly get to 3.0 Seconds , normally those requests (in Cpython) are only ~0.001 to 0.002 sec.
***************************************************************************
The information contained in this communication is confidential, is
intended only for the use of the recipient named above, and may be legally
privileged.
If the reader of this message is not the intended recipient, you are
hereby notified that any dissemination, distribution or copying of this
communication is strictly prohibited.
If you have received this communication in error, please resend this
communication to the sender and delete the original message or any copy
of it from your computer system.
Thank You.
****************************************************************************
_______________________________________________
pypy-dev mailing list
pypy-dev at python.org<mailto:pypy-dev at python.org>
https://mail.python.org/mailman/listinfo/pypy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140905/2d9df91c/attachment-0001.html>
More information about the pypy-dev
mailing list