Slowdown on latest 3.5 nightly?
I’ve been working on porting PyPI/Warehouse to using PyPy3, using the latest PyPy3 release (6.0.0) my tests on Warehouse take ~200 seconds to run, which is roughly within the realm of what I expected. I noticed when I tried to use the latest PyPy3 3.5 nightly (pypy-c-jit-95315-0a5cd8d36e99-linux64.tar.bz2) that my tests were randomly “hanging” (or taking so long to complete a single test that I got bored and filled it). It wasn’t every time, (IOW, the test it happened to would switch around each run). I thought maybe it was a problem with the functional tests, and ran only the unit tests, and it got ~19% of the way through and happened again (and in general the tests were way way slower). I went back one nightly release, to pypy-c-jit-95312-3bb86a88e1d3-linux64.tar.bz2 and tried with that, and that works as expected, with a similar runtime to PyPy3-6.0.0 and without any hangs.
On 16/11/18 4:50 pm, Donald Stufft wrote:
That is very strange, the only difference between those two commits is changing a spaace.bytes_w() call to space.|charbuf_w() in PyUnicode_FromEncodedObject, which should have no effect (and some testing changes). Could you try a latest nightly ||(|e1b0f8e6c29c from 2018-11-20 or after) http://buildbot.pypy.org/nightly/py3.5 ? |I refactored that code to call |spaace.bytes_w()|| if possible before other checks. If you have extra time, maybe you could even try the unicode-utf8-py3 nightly, http://buildbot.pypy.org/nightly/unicode-utf8-py3 which is a WIP to use utf8 internally everywhere without converting back and forth to unicode. It would be nice to know if that is any faster (or even works). Matti ||
Hi Matti, On 20/11/2018, Matti Picus <matti.picus@gmail.com> wrote:
That's even assuming the slow-down is in any way related to unicodes. I think we might as well assume that the slowdown is not related to any change in the code and due to other random effects. If we want to know more precisely what is going on, the first step would be to get the real code that is unexpectedly slower on one pypy than on the other, and try to run it ourselves---e.g. inside gdb, to start with. Armin
It’s in a branch on Github https://github.com/pypa/warehouse/pull/5057 <https://github.com/pypa/warehouse/pull/5057> which runs everything in docker, ``make build`` then ``make serve``, and then in another terminal ``make tests`` runs the tests. That may make it. A lot harder to get a gdb going though :( It’s easy to switch what Nightly is being used though, that’s just a line in the Dockerfile to pick what download from PyPy to use. I’m going to go ahead and test those other nightlies later today though.
On 16/11/18 4:50 pm, Donald Stufft wrote:
That is very strange, the only difference between those two commits is changing a spaace.bytes_w() call to space.|charbuf_w() in PyUnicode_FromEncodedObject, which should have no effect (and some testing changes). Could you try a latest nightly ||(|e1b0f8e6c29c from 2018-11-20 or after) http://buildbot.pypy.org/nightly/py3.5 ? |I refactored that code to call |spaace.bytes_w()|| if possible before other checks. If you have extra time, maybe you could even try the unicode-utf8-py3 nightly, http://buildbot.pypy.org/nightly/unicode-utf8-py3 which is a WIP to use utf8 internally everywhere without converting back and forth to unicode. It would be nice to know if that is any faster (or even works). Matti ||
Hi Matti, On 20/11/2018, Matti Picus <matti.picus@gmail.com> wrote:
That's even assuming the slow-down is in any way related to unicodes. I think we might as well assume that the slowdown is not related to any change in the code and due to other random effects. If we want to know more precisely what is going on, the first step would be to get the real code that is unexpectedly slower on one pypy than on the other, and try to run it ourselves---e.g. inside gdb, to start with. Armin
It’s in a branch on Github https://github.com/pypa/warehouse/pull/5057 <https://github.com/pypa/warehouse/pull/5057> which runs everything in docker, ``make build`` then ``make serve``, and then in another terminal ``make tests`` runs the tests. That may make it. A lot harder to get a gdb going though :( It’s easy to switch what Nightly is being used though, that’s just a line in the Dockerfile to pick what download from PyPy to use. I’m going to go ahead and test those other nightlies later today though.
participants (3)
-
Armin Rigo -
Donald Stufft -
Matti Picus