time python3 fe.py -cq Giddens
time pypy3 fe.py -cq Giddens
Hello all, thank you for your work on pypy. I'm a pypy newbie and thought to try it on a program I use a lot and where I appreciate fast response times (especially when running on a webhost). I keep my bibliography notes in interconnected XML-based mindmaps (Freeplane). `fe.py` parses and walks those XML files and generates output (bibtex, YAML, wikipedia, or HTML that highlights a queried search) [1]. [1]: https://github.com/reagle/thunderdell/blob/master/fe.py Running it with pypy is slower: ``` python3 fe.py -cq Giddens 1.46s user 0.16s system 97% cpu 1.649 total pypy3 fe.py -cq Giddens 2.81s user 0.26s system 93% cpu 3.292 total ``` I tried to use the pypy profiler but it would seemingly lockup (and vmprof.com seems down to boot). I've attached a cProfile. As you might expect, it spends a lot of time parsing XML, doing the regex search on nodes, and parsing citation strings. Any idea why pypy is slower?
Hi Joseph My first intuition would be to run it for a bit longer (can you run it in a loop couple times and see if it speeds up?) 2s might not be enough for JIT to kick in on something as complicated On Wed, Feb 13, 2019 at 3:11 PM Joseph Reagle <joseph.2011@reagle.org> wrote:
Hello all, thank you for your work on pypy.
I'm a pypy newbie and thought to try it on a program I use a lot and where I appreciate fast response times (especially when running on a webhost). I keep my bibliography notes in interconnected XML-based mindmaps (Freeplane). `fe.py` parses and walks those XML files and generates output (bibtex, YAML, wikipedia, or HTML that highlights a queried search) [1].
[1]: https://github.com/reagle/thunderdell/blob/master/fe.py
Running it with pypy is slower:
time python3 fe.py -cq Giddens
time pypy3 fe.py -cq Giddens
``` python3 fe.py -cq Giddens 1.46s user 0.16s system 97% cpu 1.649 total pypy3 fe.py -cq Giddens 2.81s user 0.26s system 93% cpu 3.292 total ```
I tried to use the pypy profiler but it would seemingly lockup (and vmprof.com seems down to boot). I've attached a cProfile. As you might expect, it spends a lot of time parsing XML, doing the regex search on nodes, and parsing citation strings.
Any idea why pypy is slower?
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
On 2/13/19 9:38 AM, Maciej Fijalkowski wrote:
My first intuition would be to run it for a bit longer (can you run it in a loop couple times and see if it speeds up?) 2s might not be enough for JIT to kick in on something as complicated
It's a single use utility where each run processes about a 100 XML files, doing things like string regex and munging thousands of times. Is it possible for pypy to remember optimizations across instantiations?
On Wed, Feb 13, 2019 at 3:57 PM Joseph Reagle <joseph.2011@reagle.org> wrote:
On 2/13/19 9:38 AM, Maciej Fijalkowski wrote:
My first intuition would be to run it for a bit longer (can you run it in a loop couple times and see if it speeds up?) 2s might not be enough for JIT to kick in on something as complicated
It's a single use utility where each run processes about a 100 XML files, doing things like string regex and munging thousands of times.
Is it possible for pypy to remember optimizations across instantiations?
It is not possible. Here is the explanation: http://doc.pypy.org/en/latest/faq.html#couldn-t-the-jit-dump-and-reload-alre... Best, Maciej Fijalkowski
Hi Joseph, On Wed, 13 Feb 2019 at 16:19, Maciej Fijalkowski <fijall@gmail.com> wrote:
On Wed, Feb 13, 2019 at 3:57 PM Joseph Reagle <joseph.2011@reagle.org> wrote:
Is it possible for pypy to remember optimizations across instantiations?
It is not possible.
A more constructive answer: in some cases, you can change the overall approach. The problem is likely not that it takes 2s instead of 1s to run the program, but that this difference is multiplied many times because you run the same program many times on different input data. In that case, you may try to convert the single-use script into a local "server" that is only started once. Then you change your ``fe.py`` script to connect to it and "download" the result locally. The point is that the server runs in a single process that remains alive. A bientôt, Armin.
Hi, you can run it as a daemon/server(for example a little flask app). This optimization also works for cpython apps if you want to avoid the startup/import time. Can the work be split up per xml file easily? Then perhaps multiprocessing will work nicely for you. Do you need to process all the files each time? Or can you avoid work? cheers, On Wed, Feb 13, 2019 at 3:57 PM Joseph Reagle <joseph.2011@reagle.org> wrote:
On 2/13/19 9:38 AM, Maciej Fijalkowski wrote:
My first intuition would be to run it for a bit longer (can you run it in a loop couple times and see if it speeds up?) 2s might not be enough for JIT to kick in on something as complicated
It's a single use utility where each run processes about a 100 XML files, doing things like string regex and munging thousands of times.
Is it possible for pypy to remember optimizations across instantiations? _______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
On 2/13/19 10:42 AM, René Dudfield wrote:
you can run it as a daemon/server(for example a little flask app). This optimization also works for cpython apps if you want to avoid the startup/import time.
That would be a big change for a uncertain improvement, so I'm not willing to go there yet. Performance is okay, but I want to see if I can improve it further as a stand-alone.
Can the work be split up per xml file easily? Then perhaps multiprocessing will work nicely for you.
Do you need to process all the files each time? Or can you avoid work?
I've tried multiprocessing too, but it is slower. Parsing the XML can be done in parallel but I suspect the overhead of multi-processing was the drag. I've also thought about caching the XML parse trees, but serializing and reloading pickles of unchanged parse trees seems slower than just parsing the XML anew. Using lru_cache on text processing functions (e.g., removing accents) didn't help either. I haven't been able to find good examples of people using multiprocessing or pypy for XML processing, perhaps this is why. Thank you all for the suggestions!
I wonder how nuitka might do? http://nuitka.net/pages/overview.html m On Wed, Feb 13, 2019 at 10:58:32AM -0500, Joseph Reagle wrote:
On 2/13/19 10:42 AM, Ren� Dudfield wrote:
you can run it as a daemon/server(for example a little flask app). This optimization also works for cpython apps if you want to avoid the startup/import time.
That would be a big change for a uncertain improvement, so I'm not willing to go there yet. Performance is okay, but I want to see if I can improve it further as a stand-alone.
Can the work be split up per xml file easily? Then perhaps multiprocessing will work nicely for you.
Do you need to process all the files each time? Or can you avoid work?
I've tried multiprocessing too, but it is slower. Parsing the XML can be done in parallel but I suspect the overhead of multi-processing was the drag. I've also thought about caching the XML parse trees, but serializing and reloading pickles of unchanged parse trees seems slower than just parsing the XML anew. Using lru_cache on text processing functions (e.g., removing accents) didn't help either.
I haven't been able to find good examples of people using multiprocessing or pypy for XML processing, perhaps this is why.
Thank you all for the suggestions!
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
-- Matt Billenstein matt@vazor.com http://www.vazor.com/
On 2/13/19 3:15 PM, Matt Billenstein wrote:
Hi Matt, thanks for the suggestion. For my program, Nuitka's performance is very close to Cpython's.
Hi Joseph, wow, interesting project! Would you be up to sharing some example input that you are trying this on? I might be up to looking into why the program is slower than on CPython, but I'd need a way that I can actually run it with realistic input. Cheers, Carl Friedrich On 13/02/2019 16:58, Joseph Reagle wrote:
On 2/13/19 10:42 AM, René Dudfield wrote:
you can run it as a daemon/server(for example a little flask app). This optimization also works for cpython apps if you want to avoid the startup/import time.
That would be a big change for a uncertain improvement, so I'm not willing to go there yet. Performance is okay, but I want to see if I can improve it further as a stand-alone.
Can the work be split up per xml file easily? Then perhaps multiprocessing will work nicely for you.
Do you need to process all the files each time? Or can you avoid work?
I've tried multiprocessing too, but it is slower. Parsing the XML can be done in parallel but I suspect the overhead of multi-processing was the drag. I've also thought about caching the XML parse trees, but serializing and reloading pickles of unchanged parse trees seems slower than just parsing the XML anew. Using lru_cache on text processing functions (e.g., removing accents) didn't help either.
I haven't been able to find good examples of people using multiprocessing or pypy for XML processing, perhaps this is why.
Thank you all for the suggestions!
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
On 2/13/19 4:03 PM, Carl Friedrich Bolz-Tereick wrote:
wow, interesting project! Would you be up to sharing some example input that you are trying this on? I might be up to looking into why the program is slower than on CPython, but I'd need a way that I can actually run it with realistic input.
Thanks Carl. I don't have a nice package, but the repo is: https://github.com/reagle/thunderdell/ The dependencies should be easy, `fe.py` imports web_little, which will require requests. Everything else should be stdlib. Below I show how to grab the realistic input and run fe.py on it yielding a 1.6MB YAML file. ``` ╭─reagle@hom ~/tmp ╰─➤ wget http://reagle.org/joseph/2005/ethno/field-notes.zip --2019-02-13 16:40:13-- http://reagle.org/joseph/2005/ethno/field-notes.zip ... 2019-02-13 16:40:13 (4.92 MB/s) - ‘field-notes.zip’ saved [1987518/1987518] ╭─reagle@hom ~/tmp ╰─➤ unzip field-notes.zip Archive: field-notes.zip inflating: field-notes-2008-cat.mm inflating: field-notes-2009-cat.mm inflating: field-notes-2010-cat.mm inflating: field-notes-2011-cat.mm inflating: field-notes-2012-cat.mm inflating: field-notes-2013-cat.mm inflating: field-notes-2014-cat.mm inflating: field-notes-2015-cat.mm inflating: field-notes-2016-cat.mm inflating: field-notes-2017-cat.mm inflating: field-notes-2017.mm inflating: field-notes-2018.mm inflating: field-notes.mm ╭─reagle@hom ~/tmp ╰─➤ ~/bin/fe/fe.py -i field-notes.mm -c -o ╭─reagle@hom ~/tmp ╰─➤ head field-notes.yaml --- references: - id: ACLU2018acg type: webpage author: - family: "ACLU" container-title: "American Civil Liberties Union" custom2: "field-notes-2018.mm" issued: year: 2018 ```
participants (6)
-
Armin Rigo -
Carl Friedrich Bolz-Tereick -
Joseph Reagle -
Maciej Fijalkowski -
Matt Billenstein -
René Dudfield