[pypy-dev] Benchmarking PyPy performance on real-world Django app

[Leonardo Santagada]

No one complains about it, but Solr does do a warmup phase while
bringing the server up before starting to serve real requests.

On Fri, Oct 7, 2011 at 2:15 PM, Michael Foord <fuzzyman at gmail.com> wrote:
>
>
> On 7 October 2011 18:08, Maciej Fijalkowski <fijall at gmail.com> wrote:
>>
>> On Fri, Oct 7, 2011 at 7:04 PM, Michael Foord <fuzzyman at gmail.com> wrote:
>> >
>> >
>> > On 7 October 2011 17:50, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> >>
>> >> On Fri, Oct 7, 2011 at 1:33 PM, Igor Katson <igor.katson at gmail.com>
>> >> wrote:
>> >> > When I first started benchmarking one of my Django sites,
>> >> > http://trip-travel.ru/ (using postgres driver pypq),
>> >> > I was disappointed by the results. PyPy was visually much slower than
>> >> > cPython (I just looked at how the site loads in the browser)
>> >> >
>> >> > But today I got some incredible results, I finally made PyPy to work
>> >> > faster
>> >> > than cPython, and found out that it got faster after loading the page
>> >> > several hundred times with "ab" or "siege"
>> >> >
>> >> > Here a the results, of mean response time by querying the home page
>> >> > with
>> >> > apache's "ab" (cPython 2.7.2, Django 1.3, PyPy 1.6.0), served with
>> >> > cherrypy
>> >> > wsgi server:
>> >> >
>> >> > After 10 requests (excluding the first request):
>> >> > cPython - 163.529 ms
>> >> > PyPy - 460.879 ms
>> >> >
>> >> > 50 request more:
>> >> > cPython - 168.539
>> >> > PyPy - 249.850
>> >> >
>> >> > 100 requests more:
>> >> > cPython - 166.278 ms
>> >> > PyPy - 131.104
>> >> >
>> >> > 100 requests more:
>> >> > cPython - 165.820
>> >> > PyPy - 115.446
>> >> >
>> >> > 300 requests more:
>> >> > cPython - 165.543
>> >> > PyPy - 107.636
>> >> >
>> >> > 300 requests more:
>> >> > cPython - 166.425
>> >> > PyPy - 103.065
>> >>
>> >> Thanks for doing the benchmarks :)
>> >>
>> >> >
>> >> > As we can see, the JIT needs much time to warm up, but when it does,
>> >> > the
>> >> > result is pretty noticeable.
>> >> > By the way, with psycopg2, the site responds for 155.766 ms in
>> >> > average
>> >> > (only
>> >> > 10 ms faster), so using PyPy with Django makes much sense for me.
>> >> >
>> >> > As for now, pypy cannot run with uWSGI, which I use in production,
>> >> > but
>> >> > maybe
>> >> > i'll switch to PyPy for production deployments if "PyPy + PyPQ + Some
>> >> > pure
>> >> > python WSGI server" suite will outperform (uWSGI + cPython +
>> >> > psycopg2).
>> >> > Though, the need to load the page 500 times after each server reload
>> >> > is
>> >> > not
>> >> > comfortable.
>> >>
>> >> I've heard people using gunicorn. Maybe this is a good try? Loading
>> >> the pages is indeed annoying, but you need to load it a couple times
>> >> for results not to be noticably slower :) We kind of know that the JIT
>> >> warmup time is high, but it's partly a thing to fix and partly an
>> >> inherent property of the JIT.
>> >>
>> >
>> >
>> > FWIW I shared this internally at Canonical, and whilst people were
>> > impressed
>> > there was some concern that having substantially worse performance for
>> > the
>> > first few hundred requests would a) be a showstopper and b) screw up
>> > metrics. The technique of deliberate warming immediately after restart
>> > is
>> > interesting, but it would be hard to hit all the code paths.
>> >
>> > I realise this is inherent in the way the jit works - but I thought it
>> > was a
>> > response worth sharing. I also assured them that pre-warm-up performance
>> > would continue to improve as pypy improves. ;-)
>> >
>> > All the best,
>> >
>> > Michael Foord
>> >
>>
>> It's also true for anything based on JVM and I've *never* seen anyone
>> complain about it. The whole concept of a JIT is just "new" to python
>> world. Seriously, would actually anyone notice if first 50 requests
>> after restart take extra 200ms??? I'm not sure, my computer and my
>> internet connection both have hiccups of over 200ms. Note that many
>> code paths are common, which means that speed up is also shared.
>
> I think you have a point and I've added your response to the internal
> discussion.
>
> Michael
>
> --
>
> http://www.voidspace.org.uk/
>
>
> May you do good and not evil
> May you find forgiveness for yourself and forgive others
> May you share freely, never taking more than you give.
> -- the sqlite blessing http://www.sqlite.org/different.html
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
>



-- 
Leonardo Santagada