Hello all, thank you for your work on pypy.
I'm a pypy newbie and thought to try it on a program I use a lot and where I appreciate fast response times (especially when running on a webhost). I keep my bibliography notes in interconnected XML-based mindmaps (Freeplane). `fe.py` parses and walks those XML files and generates output (bibtex, YAML, wikipedia, or HTML that highlights a queried search) [1].
[1]: https://github.com/reagle/thunderdell/blob/master/fe.py
Running it with pypy is slower:
```
time python3 fe.py -cq Giddens
python3 fe.py -cq Giddens 1.46s user 0.16s system 97% cpu 1.649 total
time pypy3 fe.py -cq Giddens
pypy3 fe.py -cq Giddens 2.81s user 0.26s system 93% cpu 3.292 total ```
I tried to use the pypy profiler but it would seemingly lockup (and vmprof.com seems down to boot). I've attached a cProfile. As you might expect, it spends a lot of time parsing XML, doing the regex search on nodes, and parsing citation strings.
Any idea why pypy is slower?
Hi Joseph
My first intuition would be to run it for a bit longer (can you run it in a loop couple times and see if it speeds up?) 2s might not be enough for JIT to kick in on something as complicated
On Wed, Feb 13, 2019 at 3:11 PM Joseph Reagle joseph.2011@reagle.org wrote:
Hello all, thank you for your work on pypy.
I'm a pypy newbie and thought to try it on a program I use a lot and where I appreciate fast response times (especially when running on a webhost). I keep my bibliography notes in interconnected XML-based mindmaps (Freeplane). `fe.py` parses and walks those XML files and generates output (bibtex, YAML, wikipedia, or HTML that highlights a queried search) [1].
Running it with pypy is slower:
> time python3 fe.py -cq Giddens python3 fe.py -cq Giddens 1.46s user 0.16s system 97% cpu 1.649 total > time pypy3 fe.py -cq Giddens pypy3 fe.py -cq Giddens 2.81s user 0.26s system 93% cpu 3.292 totalI tried to use the pypy profiler but it would seemingly lockup (and vmprof.com seems down to boot). I've attached a cProfile. As you might expect, it spends a lot of time parsing XML, doing the regex search on nodes, and parsing citation strings.
Any idea why pypy is slower?
pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
On 2/13/19 9:38 AM, Maciej Fijalkowski wrote:
My first intuition would be to run it for a bit longer (can you run it in a loop couple times and see if it speeds up?) 2s might not be enough for JIT to kick in on something as complicated
It's a single use utility where each run processes about a 100 XML files, doing things like string regex and munging thousands of times.
Is it possible for pypy to remember optimizations across instantiations?
On Wed, Feb 13, 2019 at 3:57 PM Joseph Reagle joseph.2011@reagle.org wrote:
On 2/13/19 9:38 AM, Maciej Fijalkowski wrote:
My first intuition would be to run it for a bit longer (can you run it in a loop couple times and see if it speeds up?) 2s might not be enough for JIT to kick in on something as complicated
It's a single use utility where each run processes about a 100 XML files, doing things like string regex and munging thousands of times.
Is it possible for pypy to remember optimizations across instantiations?
It is not possible.
Here is the explanation: http://doc.pypy.org/en/latest/faq.html#couldn-t-the-jit-dump-and-reload-alre...
Best, Maciej Fijalkowski
Hi Joseph,
On Wed, 13 Feb 2019 at 16:19, Maciej Fijalkowski fijall@gmail.com wrote:
On Wed, Feb 13, 2019 at 3:57 PM Joseph Reagle joseph.2011@reagle.org wrote:
Is it possible for pypy to remember optimizations across instantiations?
It is not possible.
A more constructive answer: in some cases, you can change the overall approach. The problem is likely not that it takes 2s instead of 1s to run the program, but that this difference is multiplied many times because you run the same program many times on different input data. In that case, you may try to convert the single-use script into a local "server" that is only started once. Then you change your ``fe.py`` script to connect to it and "download" the result locally. The point is that the server runs in a single process that remains alive.
A bientôt,
Armin.
Hi,
you can run it as a daemon/server(for example a little flask app). This optimization also works for cpython apps if you want to avoid the startup/import time.
Can the work be split up per xml file easily? Then perhaps multiprocessing will work nicely for you.
Do you need to process all the files each time? Or can you avoid work?
cheers,
On Wed, Feb 13, 2019 at 3:57 PM Joseph Reagle joseph.2011@reagle.org wrote:
On 2/13/19 9:38 AM, Maciej Fijalkowski wrote:
My first intuition would be to run it for a bit longer (can you run it in a loop couple times and see if it speeds up?) 2s might not be enough for JIT to kick in on something as complicated
It's a single use utility where each run processes about a 100 XML files, doing things like string regex and munging thousands of times.
Is it possible for pypy to remember optimizations across instantiations? _______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
On 2/13/19 10:42 AM, René Dudfield wrote:
you can run it as a daemon/server(for example a little flask app). This optimization also works for cpython apps if you want to avoid the startup/import time.
That would be a big change for a uncertain improvement, so I'm not willing to go there yet. Performance is okay, but I want to see if I can improve it further as a stand-alone.
Can the work be split up per xml file easily? Then perhaps multiprocessing will work nicely for you.
Do you need to process all the files each time? Or can you avoid work?
I've tried multiprocessing too, but it is slower. Parsing the XML can be done in parallel but I suspect the overhead of multi-processing was the drag. I've also thought about caching the XML parse trees, but serializing and reloading pickles of unchanged parse trees seems slower than just parsing the XML anew. Using lru_cache on text processing functions (e.g., removing accents) didn't help either.
I haven't been able to find good examples of people using multiprocessing or pypy for XML processing, perhaps this is why.
Thank you all for the suggestions!
I wonder how nuitka might do?
http://nuitka.net/pages/overview.html
m
On Wed, Feb 13, 2019 at 10:58:32AM -0500, Joseph Reagle wrote:
On 2/13/19 10:42 AM, Ren� Dudfield wrote:
you can run it as a daemon/server(for example a little flask app). This optimization also works for cpython apps if you want to avoid the startup/import time.
That would be a big change for a uncertain improvement, so I'm not willing to go there yet. Performance is okay, but I want to see if I can improve it further as a stand-alone.
Can the work be split up per xml file easily? Then perhaps multiprocessing will work nicely for you.
Do you need to process all the files each time? Or can you avoid work?
I've tried multiprocessing too, but it is slower. Parsing the XML can be done in parallel but I suspect the overhead of multi-processing was the drag. I've also thought about caching the XML parse trees, but serializing and reloading pickles of unchanged parse trees seems slower than just parsing the XML anew. Using lru_cache on text processing functions (e.g., removing accents) didn't help either.
I haven't been able to find good examples of people using multiprocessing or pypy for XML processing, perhaps this is why.
Thank you all for the suggestions!
pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
On 2/13/19 3:15 PM, Matt Billenstein wrote:
Hi Matt, thanks for the suggestion. For my program, Nuitka's performance is very close to Cpython's.
Hi Joseph,
wow, interesting project! Would you be up to sharing some example input that you are trying this on? I might be up to looking into why the program is slower than on CPython, but I'd need a way that I can actually run it with realistic input.
Cheers,
Carl Friedrich
On 13/02/2019 16:58, Joseph Reagle wrote:
On 2/13/19 10:42 AM, René Dudfield wrote:
you can run it as a daemon/server(for example a little flask app). This optimization also works for cpython apps if you want to avoid the startup/import time.
That would be a big change for a uncertain improvement, so I'm not willing to go there yet. Performance is okay, but I want to see if I can improve it further as a stand-alone.
Can the work be split up per xml file easily? Then perhaps multiprocessing will work nicely for you.
Do you need to process all the files each time? Or can you avoid work?
I've tried multiprocessing too, but it is slower. Parsing the XML can be done in parallel but I suspect the overhead of multi-processing was the drag. I've also thought about caching the XML parse trees, but serializing and reloading pickles of unchanged parse trees seems slower than just parsing the XML anew. Using lru_cache on text processing functions (e.g., removing accents) didn't help either.
I haven't been able to find good examples of people using multiprocessing or pypy for XML processing, perhaps this is why.
Thank you all for the suggestions!
pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
On 2/13/19 4:03 PM, Carl Friedrich Bolz-Tereick wrote:
wow, interesting project! Would you be up to sharing some example input that you are trying this on? I might be up to looking into why the program is slower than on CPython, but I'd need a way that I can actually run it with realistic input.
Thanks Carl. I don't have a nice package, but the repo is:
https://github.com/reagle/thunderdell/
The dependencies should be easy, `fe.py` imports web_little, which will require requests. Everything else should be stdlib.
Below I show how to grab the realistic input and run fe.py on it yielding a 1.6MB YAML file.
``` ╭─reagle@hom ~/tmp ╰─➤ wget http://reagle.org/joseph/2005/ethno/field-notes.zip --2019-02-13 16:40:13-- http://reagle.org/joseph/2005/ethno/field-notes.zip ... 2019-02-13 16:40:13 (4.92 MB/s) - ‘field-notes.zip’ saved [1987518/1987518]
╭─reagle@hom ~/tmp ╰─➤ unzip field-notes.zip Archive: field-notes.zip inflating: field-notes-2008-cat.mm inflating: field-notes-2009-cat.mm inflating: field-notes-2010-cat.mm inflating: field-notes-2011-cat.mm inflating: field-notes-2012-cat.mm inflating: field-notes-2013-cat.mm inflating: field-notes-2014-cat.mm inflating: field-notes-2015-cat.mm inflating: field-notes-2016-cat.mm inflating: field-notes-2017-cat.mm inflating: field-notes-2017.mm inflating: field-notes-2018.mm inflating: field-notes.mm ╭─reagle@hom ~/tmp ╰─➤ ~/bin/fe/fe.py -i field-notes.mm -c -o ╭─reagle@hom ~/tmp ╰─➤ head field-notes.yaml --- references: - id: ACLU2018acg type: webpage author: - family: "ACLU" container-title: "American Civil Liberties Union" custom2: "field-notes-2018.mm" issued: year: 2018 ```
参加者 (6)
-
Armin Rigo -
Carl Friedrich Bolz-Tereick -
Joseph Reagle -
Maciej Fijalkowski -
Matt Billenstein -
René Dudfield