release seems ready
It seems we have a release, version ad5a4e55fa8e. Is there a reason to wait? buildbots http://buildbot.pypy.org/summary?branch=release-5.x release notice http://doc.pypy.org/en/latest/release-5.0.0.html Hopefully we can release 5.1 once s360-x lands on default Matti
Hi Matti, On 8 March 2016 at 15:15, matti picus <matti.picus@gmail.com> wrote:
+1. Go ahead as far as I'm concerned. About the release notice: "As a result, lxml with its cython compiled component passes all tests on PyPy" is not clear until the next official lxml is released. The current lxml 3.5.0 still contains a partially buggy workaround that tries to make it work on previous versions of cpyext. The trunk version at https://github.com/lxml/lxml has got this code removed, and that's the version that works. I'll make the ppc releases once the other releases are out. A bientôt, Armin
Did the lxml project indicate they will provide a new release soon that incorporates these fixes? I tried to build the latest development code from source, but run into many issues (lxml build server down, source package missing the pre-generated C code etc. etc.), and customer company policy wouldn't allow using a development version in production anyway. The lxml 3.5.0 does not install with pypy-5.0.0 (it used to with pypy-4.0.1, though it was too buggy to be useful), and the lxml-cffi no longer installs. On 08-03-16 15:26, Armin Rigo wrote:
There is a new lxml release as of today, unfortunately there is an issue: https://bitbucket.org/pypy/pypy/issues/2260/pypy-500-dumps-core-with-lxml-36... On 16-03-16 18:07, Armin Rigo wrote:
Hi, On 17 March 2016 at 16:13, Pim van der Eijk (Lists) <lists@sonnenglanz.net> wrote:
There is a new lxml release as of today, unfortunately there is an issue: https://bitbucket.org/pypy/pypy/issues/2260/pypy-500-dumps-core-with-lxml-36...
Yes, it's what we get when both sides (lxml and pypy) are half-hearted about supporting the other. The lxml tests seem to pass, but that may be because they are small. Many bigger and longer-running processes seem to crash like that. I'm investigating. A bientôt, Armin.
Hi again, On 17 March 2016 at 17:27, Armin Rigo <arigo@tunes.org> wrote:
Fixed in 0173cdbbbacc, which then seems to work with lxml even on these larger examples. I'd love some more testing before we do the 5.0.1 bugfix release. Please try with a version of PyPy on the "release-5.x" branch recent enough to contain a09a60a9c381; an Ubuntu precompiled version is here: http://buildbot.pypy.org/nightly/release-5.x/pypy-c-jit-83125-a09a60a9c381-l... A bientôt, Armin.
Hi, I did some tests and there are no crashes. However, compared to CPython 2.7.10 there are some serious issues: - For my test programs (the script in the issue on BitBucket is derived from one of them), PyPy is much slower. script A: 256 seconds in PyPy versus 78 seconds in CPython script B: 9.73 seconds in PyPy versus 2.6 in Cpython - Memory use continues to grow up to over 80% at which time where my laptop starts swapping, whereas with CPython usage is never more than 4%. - Perhaps caused by the above, there are occasional freezes of several seconds in which nothing seems to happen, although CPU usage is still 100%. Kind Regards, Pim On 18-03-16 12:13, Armin Rigo wrote:
Hi Pim, On 18 March 2016 at 13:25, Pim van der Eijk (Lists) <lists@sonnenglanz.net> wrote:
- For my test programs (the script in the issue on BitBucket is derived from one of them), PyPy is much slower.
If you're comparing the speed of scripts that have a large amount of crossings of the cpyext layer (i.e. crossings between Python code and CPython C extension code), then yes, it's expected to be much slower. The speed improved a lot recently, which means it is now *much slower* instead of *very, very much slower*. It makes no sense, now or in the future, to use PyPy in the hope to speed up a script that does _only_ lxml stuff with almost no Python code running in-between.
- Memory use continues to grow up to over 80% at which time where my laptop starts swapping, whereas with CPython usage is never more than 4%.
This is more annoying. Can you give us a way to reproduce this? Armin
On 18-03-16 14:57, Armin Rigo wrote:
It already happens with the script I attached to the original issue, which you already have: https://bitbucket.org/pypy/pypy/issues/2260/pypy-500-dumps-core-with-lxml-36...
Hi Pim, On 18 March 2016 at 15:08, Pim van der Eijk (Lists) <lists@sonnenglanz.net> wrote:
Ok, partially reproduced. With CPython it grows continously too, but only up to 1.2GB and then it finishes. With PyPy it grows faster up to 22GB. If I add some "gc.collect()" executed every few seconds, then PyPy only grows up to 1.7GB. I added "add_memory_pressure=True" to some chosen mallocs inside cpyext, and it seems to be enough to fix the problem. Now PyPy grows up to 1.7GB even without any gc.collect(). Yay! (changeset 9137853fd0ec, grafted to release-5.x too) A bientôt, Armin.
Hi Armin, On 18-03-16 18:52, Armin Rigo wrote:
I retested and confirm that the library works and memory use is now like CPython, which is great. It is still slower than CPython, for reasons you explained before, but that is because my test script heavily uses of lxml. In larger applications where lxml processing is a smaller part of the overall functionality, the PyPy speed-up of regular Python code could well compensate for this. Many thanks, Pim
A bientôt,
Armin.
Hi, On 8 March 2016 at 15:42, Maciej Fijalkowski <fijall@gmail.com> wrote:
btw, should we mention packages.pypy.org?
I would do so but only under two conditions: * it reports a post-cpyext-fixes result: which packages run or don't run now, ideally on the current "release 5.0" branch, but at least after the merge of the cpyext-gc-support-2 branch * we quickly review and fix the few manual comments, notably lxml's (we no longer recommend lxml-cffi). A bientôt, Armin
Hi Matti, On 8 March 2016 at 15:15, matti picus <matti.picus@gmail.com> wrote:
+1. Go ahead as far as I'm concerned. About the release notice: "As a result, lxml with its cython compiled component passes all tests on PyPy" is not clear until the next official lxml is released. The current lxml 3.5.0 still contains a partially buggy workaround that tries to make it work on previous versions of cpyext. The trunk version at https://github.com/lxml/lxml has got this code removed, and that's the version that works. I'll make the ppc releases once the other releases are out. A bientôt, Armin
Did the lxml project indicate they will provide a new release soon that incorporates these fixes? I tried to build the latest development code from source, but run into many issues (lxml build server down, source package missing the pre-generated C code etc. etc.), and customer company policy wouldn't allow using a development version in production anyway. The lxml 3.5.0 does not install with pypy-5.0.0 (it used to with pypy-4.0.1, though it was too buggy to be useful), and the lxml-cffi no longer installs. On 08-03-16 15:26, Armin Rigo wrote:
There is a new lxml release as of today, unfortunately there is an issue: https://bitbucket.org/pypy/pypy/issues/2260/pypy-500-dumps-core-with-lxml-36... On 16-03-16 18:07, Armin Rigo wrote:
Hi, On 17 March 2016 at 16:13, Pim van der Eijk (Lists) <lists@sonnenglanz.net> wrote:
There is a new lxml release as of today, unfortunately there is an issue: https://bitbucket.org/pypy/pypy/issues/2260/pypy-500-dumps-core-with-lxml-36...
Yes, it's what we get when both sides (lxml and pypy) are half-hearted about supporting the other. The lxml tests seem to pass, but that may be because they are small. Many bigger and longer-running processes seem to crash like that. I'm investigating. A bientôt, Armin.
Hi again, On 17 March 2016 at 17:27, Armin Rigo <arigo@tunes.org> wrote:
Fixed in 0173cdbbbacc, which then seems to work with lxml even on these larger examples. I'd love some more testing before we do the 5.0.1 bugfix release. Please try with a version of PyPy on the "release-5.x" branch recent enough to contain a09a60a9c381; an Ubuntu precompiled version is here: http://buildbot.pypy.org/nightly/release-5.x/pypy-c-jit-83125-a09a60a9c381-l... A bientôt, Armin.
Hi, I did some tests and there are no crashes. However, compared to CPython 2.7.10 there are some serious issues: - For my test programs (the script in the issue on BitBucket is derived from one of them), PyPy is much slower. script A: 256 seconds in PyPy versus 78 seconds in CPython script B: 9.73 seconds in PyPy versus 2.6 in Cpython - Memory use continues to grow up to over 80% at which time where my laptop starts swapping, whereas with CPython usage is never more than 4%. - Perhaps caused by the above, there are occasional freezes of several seconds in which nothing seems to happen, although CPU usage is still 100%. Kind Regards, Pim On 18-03-16 12:13, Armin Rigo wrote:
Hi Pim, On 18 March 2016 at 13:25, Pim van der Eijk (Lists) <lists@sonnenglanz.net> wrote:
- For my test programs (the script in the issue on BitBucket is derived from one of them), PyPy is much slower.
If you're comparing the speed of scripts that have a large amount of crossings of the cpyext layer (i.e. crossings between Python code and CPython C extension code), then yes, it's expected to be much slower. The speed improved a lot recently, which means it is now *much slower* instead of *very, very much slower*. It makes no sense, now or in the future, to use PyPy in the hope to speed up a script that does _only_ lxml stuff with almost no Python code running in-between.
- Memory use continues to grow up to over 80% at which time where my laptop starts swapping, whereas with CPython usage is never more than 4%.
This is more annoying. Can you give us a way to reproduce this? Armin
On 18-03-16 14:57, Armin Rigo wrote:
It already happens with the script I attached to the original issue, which you already have: https://bitbucket.org/pypy/pypy/issues/2260/pypy-500-dumps-core-with-lxml-36...
Hi Pim, On 18 March 2016 at 15:08, Pim van der Eijk (Lists) <lists@sonnenglanz.net> wrote:
Ok, partially reproduced. With CPython it grows continously too, but only up to 1.2GB and then it finishes. With PyPy it grows faster up to 22GB. If I add some "gc.collect()" executed every few seconds, then PyPy only grows up to 1.7GB. I added "add_memory_pressure=True" to some chosen mallocs inside cpyext, and it seems to be enough to fix the problem. Now PyPy grows up to 1.7GB even without any gc.collect(). Yay! (changeset 9137853fd0ec, grafted to release-5.x too) A bientôt, Armin.
Hi Armin, On 18-03-16 18:52, Armin Rigo wrote:
I retested and confirm that the library works and memory use is now like CPython, which is great. It is still slower than CPython, for reasons you explained before, but that is because my test script heavily uses of lxml. In larger applications where lxml processing is a smaller part of the overall functionality, the PyPy speed-up of regular Python code could well compensate for this. Many thanks, Pim
A bientôt,
Armin.
Hi, On 8 March 2016 at 15:42, Maciej Fijalkowski <fijall@gmail.com> wrote:
btw, should we mention packages.pypy.org?
I would do so but only under two conditions: * it reports a post-cpyext-fixes result: which packages run or don't run now, ideally on the current "release 5.0" branch, but at least after the merge of the cpyext-gc-support-2 branch * we quickly review and fix the few manual comments, notably lxml's (we no longer recommend lxml-cffi). A bientôt, Armin
participants (5)
-
Armin Rigo -
Maciej Fijalkowski -
matti picus -
Phyo Arkar -
Pim van der Eijk (Lists)