[pypy-dev] Playing with PyPy and Django

[Omer Katz]

I have some Django benchmarks that I was too lazy to contribute yet.
I will do so soon.

2015-02-09 8:41 GMT+02:00 Maciej Fijalkowski <fijall at gmail.com>:

> The answer is - yes maybe, but there is no such thing as "django
> site". Most of the time the bottlenecks are somewhere else (as you can
> see, the admin does not help you) and is often something stupid (e.g.
> like here people defining classes at runtime for no good reason) that
> has to be looked on a case to case basis. Ideally we would not care
> and just optimize everything away anyway, but it's difficult and the
> sort of Python that PyPy improves is growing but still not covering
> everything we would want.
>
> Cheers,
> fijal
>
>
> On Mon, Feb 9, 2015 at 1:02 AM, Tin Tvrtković <tinchester at gmail.com>
> wrote:
> > Wow, that's what I call customer support! :) Can confirm PyPy is better
> than
> > CPython on the admin now, post warmup.
> >
> > You're right, it doesn't help my site very much, but that's my fault for
> > preparing the wrong benchmark fixture. :)
> >
> > On the other hand, it doesn't really make sense for you PyPy devs to help
> > each and every user with their Django site. Would it be worth it for
> PyPy to
> > add a few benchmarks involving Django (and say, Flask) to the regular
> > benchmark suite? I see there's a "django" benchmark but it only involves
> the
> > templating subsystem.
> >
> > For example, one benchmark might be serving the admin page; another might
> > involve a dummy site with a custom app, 3-4 models from sqlite, and 1-2
> > templates? And the same for Flask with Flask-SQLAlchemy. Another option
> > would be using a complex, existing free software Django app (Review
> Board,
> > Seafile, ...)
> >
> > This would probably need to be worked out further (i.e. which versions to
> > use, when to bump them, etc) but there's probably a large number of users
> > out there using Python for exactly these kinds of things.
> >
> > In any case, thanks for looking into this, it sure does feel nice when
> > upstream devs engage with users. :)
> >
> >
> > On 08.02.2015 23:09, Maciej Fijalkowski wrote:
> >>
> >> It got merged into django, PyPy (2, didn't test the 3) is now faster
> >> than cpython on django admin. It likely does not help your cause
> >> though so you need to come up with a better benchmark :-)
> >>
> >> On Sun, Feb 8, 2015 at 8:30 PM, Maciej Fijalkowski <fijall at gmail.com>
> >> wrote:
> >>>
> >>> Hey Tin.
> >>>
> >>> We are in the process of two (trivial) pull requests to django that
> >>> drop the rendering time from 25ms -> 8ms for this case. I'm not a
> >>> farseer, however I suspect something like this can be done with your
> >>> site too (depending what it really does).
> >>>
> >>> On Sun, Feb 8, 2015 at 8:23 PM, Tin Tvrtković <tinchester at gmail.com>
> >>> wrote:
> >>>>
> >>>> Hello,
> >>>>
> >>>> thanks to everyone for their input (especially Maciej).
> >>>>
> >>>> Since I've ripped out all my code from the test project, it's not a
> >>>> Python 3
> >>>> project anymore, so I did try PyPy 2 with it too. It's faster, yes;
> the
> >>>> last
> >>>> test looked something like:
> >>>>
> >>>> PyPy 2         20.206 [ms] (mean)
> >>>> PyPy 3         27.588 [ms] (mean)
> >>>> CPython 3.4    10.865 [ms] (mean)
> >>>>
> >>>> (tried both PyPy 2.5, downloaded directly, and 2.4 from the Ubuntu
> PPA -
> >>>> about the same.)
> >>>>
> >>>> I don't know, maybe the admin page isn't a good test candidate since
> it
> >>>> might do more introspective/reflective/meta stuff. I've tried doing
> this
> >>>> same test (10k warmup requests, 100 benchmark requests) on the main
> page
> >>>> of
> >>>> the site I'm working on; the results are:
> >>>>
> >>>> PyPy 3         15.863 [ms] (mean)
> >>>> CPython 3.48.668 [ms] (mean)
> >>>>
> >>>> and I can be certain there's nothing much going on there. Just some
> >>>> permission checks, grabbing some stuff from the database, checking
> some
> >>>> query parameters and a template render.
> >>>>
> >>>> I suppose if folks have CPU intensive stuff in their view functions
> >>>> (nested
> >>>> for-loops etc) the optimizations PyPy can do to that far outweigh the
> >>>> non-optimized Django parts; my sites generally don't though.
> >>>>
> >>>>
> >>>> On 08.02.2015 18:44, Maciej Fijalkowski wrote:
> >>>>>
> >>>>> then it's used in the wrong ways in say django admin, look at
> >>>>> invocations to lazy there (this is how it showed up in my profile)
> >>>>>
> >>>>> On Sun, Feb 8, 2015 at 5:08 PM, Alex Gaynor <alex.gaynor at gmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> FWIW, I've definitely seen and worked on Django sites that got major
> >>>>>> improvements out of PyPy -- both the templates and ORM are both sped
> >>>>>> up
> >>>>>> substantially by it. I think we should try to fix issues as we see
> >>>>>> them,
> >>>>>> before writing it off.
> >>>>>>
> >>>>>> FWIW: lazy does not create a new class per call of a function, it's
> >>>>>> only
> >>>>>> one
> >>>>>> class per decorated function.
> >>>>>>
> >>>>>> Alex
> >>>>>>
> >>>>>> On Sun, Feb 8, 2015 at 6:59 AM, Maciej Fijalkowski <
> fijall at gmail.com>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> I don't know :-( not sure if fixing those issues one by one is the
> >>>>>>> way
> >>>>>>> to go, it's not like *this* particular code is a problem, it's the
> >>>>>>> culture that breeds those sort of things
> >>>>>>>
> >>>>>>> On Sun, Feb 8, 2015 at 2:09 PM, Omer Katz <omer.drow at gmail.com>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Isn't there anything we can do about it?
> >>>>>>>> We can open issues at the Django issue tracker if some code slows
> >>>>>>>> down
> >>>>>>>> execution in PyPy.
> >>>>>>>>
> >>>>>>>> 2015-02-08 12:17 GMT+02:00 Maciej Fijalkowski <fijall at gmail.com>:
> >>>>>>>>>
> >>>>>>>>> Hi Tin
> >>>>>>>>>
> >>>>>>>>> I have a bit sad news for you, but essentially from what I looked
> >>>>>>>>> at
> >>>>>>>>> profling, the answer seems to be "don't use django, it has not
> been
> >>>>>>>>> written with performance in mind". For example here (which
> features
> >>>>>>>>> in
> >>>>>>>>> django admin quite prominently, some stuff calls lazy() at each
> >>>>>>>>> call):
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> https://github.com/django/django/blob/master/django/utils/functional.py#L48
> >>>>>>>>>
> >>>>>>>>> What does it, per call that pypy does not like:
> >>>>>>>>>
> >>>>>>>>> * redefines classes
> >>>>>>>>>
> >>>>>>>>> * walks the class __mro__ and defines tons of new methods
> >>>>>>>>>
> >>>>>>>>> * proxies everything through a not so efficietn mechanisms
> >>>>>>>>>
> >>>>>>>>> Those things quite defy the purpose of the JIT - you're making
> tons
> >>>>>>>>> of
> >>>>>>>>> very dynamic things that pypy generally assumes is "static
> enough",
> >>>>>>>>> if
> >>>>>>>>> you wanted to do the same thing in C++, you would need to call
> gcc
> >>>>>>>>> with each request, that would not be so much fun.
> >>>>>>>>>
> >>>>>>>>> If you really care about performance, consider swapping your web
> >>>>>>>>> framework to something more performance-oriented where JIT
> >>>>>>>>> compilation
> >>>>>>>>> can help.
> >>>>>>>>>
> >>>>>>>>> As a side note, we (me and my company, not to be confused with
> PyPy
> >>>>>>>>> open source project) do offer looking at proprietary code for
> >>>>>>>>> benchmarking purposes as a commercial service, look at
> >>>>>>>>> baroquesoftware.com
> >>>>>>>>>
> >>>>>>>>> Cheers,
> >>>>>>>>> fijal
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Sat, Feb 7, 2015 at 1:12 AM, Tin Tvrtković
> >>>>>>>>> <tinchester at gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hello, PyPy folks!
> >>>>>>>>>>
> >>>>>>>>>> While trying to speed up one of my Django sites, I noticed a new
> >>>>>>>>>> version
> >>>>>>>>>> of
> >>>>>>>>>> PyPy
> >>>>>>>>>> had just been released. So I grabbed a fresh download of PyPy 3
> >>>>>>>>>> (since
> >>>>>>>>>> this
> >>>>>>>>>> is
> >>>>>>>>>> a Python 3 codebase) and tried taking it out for a spin.
> >>>>>>>>>>
> >>>>>>>>>> However, as far as I can see, whatever I try PyPy is
> consistently
> >>>>>>>>>> slower
> >>>>>>>>>> than
> >>>>>>>>>> CPython for this.
> >>>>>>>>>>
> >>>>>>>>>> Since this is a proprietary site, I've basically ripped out all
> >>>>>>>>>> the
> >>>>>>>>>> code
> >>>>>>>>>> except
> >>>>>>>>>> my settings.py and my requirements; and am benchmarking the
> Django
> >>>>>>>>>> admin
> >>>>>>>>>> index.
> >>>>>>>>>> The results are about the same.
> >>>>>>>>>>
> >>>>>>>>>> I've set up a small repo that can be used to reproduce the
> >>>>>>>>>> environment:
> >>>>>>>>>> https://github.com/Tinche/PyPy-Django-Playground. There's
> >>>>>>>>>> additional
> >>>>>>>>>> info in
> >>>>>>>>>> the README there.
> >>>>>>>>>>
> >>>>>>>>>> These tests have been carried out on Ubuntu Trusty, 64-bit.
> >>>>>>>>>> CPython 3
> >>>>>>>>>> is
> >>>>>>>>>> the
> >>>>>>>>>> system Python, 3.4. PyPy has been downloaded from the official
> >>>>>>>>>> site
> >>>>>>>>>> and
> >>>>>>>>>> unzipped.
> >>>>>>>>>>
> >>>>>>>>>> So what I basically do is set up an admin session, and use the
> >>>>>>>>>> Django
> >>>>>>>>>> main
> >>>>>>>>>> admin
> >>>>>>>>>> page. 200 warmup requests, then 100 benchmarked requests, look
> at
> >>>>>>>>>> the
> >>>>>>>>>> mean
> >>>>>>>>>> request time.
> >>>>>>>>>>
> >>>>>>>>>> Some results:
> >>>>>>>>>>
> >>>>>>>>>> Django's runserver, DEBUG mode:
> >>>>>>>>>>
> >>>>>>>>>> PyPy3            485.389 [ms]
> >>>>>>>>>> CPython 3.4      105.777 [ms]
> >>>>>>>>>>
> >>>>>>>>>> Django's runserver, no debug:
> >>>>>>>>>>
> >>>>>>>>>> PyPy3             44.661 [ms]
> >>>>>>>>>> CPython 3.4       18.697 [ms]
> >>>>>>>>>>
> >>>>>>>>>> Gunicorn, 1 worker, no debug:
> >>>>>>>>>>
> >>>>>>>>>> PyPy3             28.615 [ms]
> >>>>>>>>>> CPython 3.4       13.532 [ms]
> >>>>>>>>>>
> >>>>>>>>>> I don't exactly claim to be an expert on benchmarking, but
> >>>>>>>>>> assuming
> >>>>>>>>>> my
> >>>>>>>>>> site
> >>>>>>>>>> is similar to the Django admin, CPython's gonna be giving me
> >>>>>>>>>> better
> >>>>>>>>>> performance.
> >>>>>>>>>> Also the debug runserver performance is kinda worrying. Nobody's
> >>>>>>>>>> going
> >>>>>>>>>> to be
> >>>>>>>>>> running this in production, but it makes development a little
> >>>>>>>>>> slower
> >>>>>>>>>> and
> >>>>>>>>>> more
> >>>>>>>>>> annoying than it should be.
> >>>>>>>>>>
> >>>>>>>>>> Is there anything to make PyPy more competitive in these kinds
> of
> >>>>>>>>>> scenarios?
> >>>>>>>>>>
> >>>>>>>>>> Kind regards,
> >>>>>>>>>> Tin
> >>>>>>>>>> _______________________________________________
> >>>>>>>>>> pypy-dev mailing list
> >>>>>>>>>> pypy-dev at python.org
> >>>>>>>>>> https://mail.python.org/mailman/listinfo/pypy-dev
> >>>>>>>>>
> >>>>>>>>> _______________________________________________
> >>>>>>>>> pypy-dev mailing list
> >>>>>>>>> pypy-dev at python.org
> >>>>>>>>> https://mail.python.org/mailman/listinfo/pypy-dev
> >>>>>>>>
> >>>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> pypy-dev mailing list
> >>>>>>> pypy-dev at python.org
> >>>>>>> https://mail.python.org/mailman/listinfo/pypy-dev
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> "I disapprove of what you say, but I will defend to the death your
> >>>>>> right
> >>>>>> to
> >>>>>> say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> >>>>>> "The people's good is the highest law." -- Cicero
> >>>>>> GPG Key fingerprint: 125F 5C67 DFE9 4084
> >>>>
> >>>>
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20150209/3d803c16/attachment-0001.html>