time python3 fe.py -cq Giddens
time pypy3 fe.py -cq Giddens
Hello all, thank you for your work on pypy. I'm a pypy newbie and thought to try it on a program I use a lot and where I appreciate fast response times (especially when running on a webhost). I keep my bibliography notes in interconnected XML-based mindmaps (Freeplane). `fe.py` parses and walks those XML files and generates output (bibtex, YAML, wikipedia, or HTML that highlights a queried search) [1]. [1]: https://github.com/reagle/thunderdell/blob/master/fe.py Running it with pypy is slower: ``` python3 fe.py -cq Giddens 1.46s user 0.16s system 97% cpu 1.649 total pypy3 fe.py -cq Giddens 2.81s user 0.26s system 93% cpu 3.292 total ``` I tried to use the pypy profiler but it would seemingly lockup (and vmprof.com seems down to boot). I've attached a cProfile. As you might expect, it spends a lot of time parsing XML, doing the regex search on nodes, and parsing citation strings. Any idea why pypy is slower?
Hi Joseph My first intuition would be to run it for a bit longer (can you run it in a loop couple times and see if it speeds up?) 2s might not be enough for JIT to kick in on something as complicated On Wed, Feb 13, 2019 at 3:11 PM Joseph Reagle <joseph.2011@reagle.org> wrote:
On Wed, Feb 13, 2019 at 3:57 PM Joseph Reagle <joseph.2011@reagle.org> wrote:
It is not possible. Here is the explanation: http://doc.pypy.org/en/latest/faq.html#couldn-t-the-jit-dump-and-reload-alre... Best, Maciej Fijalkowski
Hi Joseph, On Wed, 13 Feb 2019 at 16:19, Maciej Fijalkowski <fijall@gmail.com> wrote:
A more constructive answer: in some cases, you can change the overall approach. The problem is likely not that it takes 2s instead of 1s to run the program, but that this difference is multiplied many times because you run the same program many times on different input data. In that case, you may try to convert the single-use script into a local "server" that is only started once. Then you change your ``fe.py`` script to connect to it and "download" the result locally. The point is that the server runs in a single process that remains alive. A bientôt, Armin.
Hi, you can run it as a daemon/server(for example a little flask app). This optimization also works for cpython apps if you want to avoid the startup/import time. Can the work be split up per xml file easily? Then perhaps multiprocessing will work nicely for you. Do you need to process all the files each time? Or can you avoid work? cheers, On Wed, Feb 13, 2019 at 3:57 PM Joseph Reagle <joseph.2011@reagle.org> wrote:
On 2/13/19 10:42 AM, René Dudfield wrote:
That would be a big change for a uncertain improvement, so I'm not willing to go there yet. Performance is okay, but I want to see if I can improve it further as a stand-alone.
I've tried multiprocessing too, but it is slower. Parsing the XML can be done in parallel but I suspect the overhead of multi-processing was the drag. I've also thought about caching the XML parse trees, but serializing and reloading pickles of unchanged parse trees seems slower than just parsing the XML anew. Using lru_cache on text processing functions (e.g., removing accents) didn't help either. I haven't been able to find good examples of people using multiprocessing or pypy for XML processing, perhaps this is why. Thank you all for the suggestions!
I wonder how nuitka might do? http://nuitka.net/pages/overview.html m On Wed, Feb 13, 2019 at 10:58:32AM -0500, Joseph Reagle wrote:
-- Matt Billenstein matt@vazor.com http://www.vazor.com/
On 2/13/19 3:15 PM, Matt Billenstein wrote:
Hi Matt, thanks for the suggestion. For my program, Nuitka's performance is very close to Cpython's.
Hi Joseph, wow, interesting project! Would you be up to sharing some example input that you are trying this on? I might be up to looking into why the program is slower than on CPython, but I'd need a way that I can actually run it with realistic input. Cheers, Carl Friedrich On 13/02/2019 16:58, Joseph Reagle wrote:
On 2/13/19 4:03 PM, Carl Friedrich Bolz-Tereick wrote:
Thanks Carl. I don't have a nice package, but the repo is: https://github.com/reagle/thunderdell/ The dependencies should be easy, `fe.py` imports web_little, which will require requests. Everything else should be stdlib. Below I show how to grab the realistic input and run fe.py on it yielding a 1.6MB YAML file. ``` ╭─reagle@hom ~/tmp ╰─➤ wget http://reagle.org/joseph/2005/ethno/field-notes.zip --2019-02-13 16:40:13-- http://reagle.org/joseph/2005/ethno/field-notes.zip ... 2019-02-13 16:40:13 (4.92 MB/s) - ‘field-notes.zip’ saved [1987518/1987518] ╭─reagle@hom ~/tmp ╰─➤ unzip field-notes.zip Archive: field-notes.zip inflating: field-notes-2008-cat.mm inflating: field-notes-2009-cat.mm inflating: field-notes-2010-cat.mm inflating: field-notes-2011-cat.mm inflating: field-notes-2012-cat.mm inflating: field-notes-2013-cat.mm inflating: field-notes-2014-cat.mm inflating: field-notes-2015-cat.mm inflating: field-notes-2016-cat.mm inflating: field-notes-2017-cat.mm inflating: field-notes-2017.mm inflating: field-notes-2018.mm inflating: field-notes.mm ╭─reagle@hom ~/tmp ╰─➤ ~/bin/fe/fe.py -i field-notes.mm -c -o ╭─reagle@hom ~/tmp ╰─➤ head field-notes.yaml --- references: - id: ACLU2018acg type: webpage author: - family: "ACLU" container-title: "American Civil Liberties Union" custom2: "field-notes-2018.mm" issued: year: 2018 ```
Hi Joseph My first intuition would be to run it for a bit longer (can you run it in a loop couple times and see if it speeds up?) 2s might not be enough for JIT to kick in on something as complicated On Wed, Feb 13, 2019 at 3:11 PM Joseph Reagle <joseph.2011@reagle.org> wrote:
On Wed, Feb 13, 2019 at 3:57 PM Joseph Reagle <joseph.2011@reagle.org> wrote:
It is not possible. Here is the explanation: http://doc.pypy.org/en/latest/faq.html#couldn-t-the-jit-dump-and-reload-alre... Best, Maciej Fijalkowski
Hi Joseph, On Wed, 13 Feb 2019 at 16:19, Maciej Fijalkowski <fijall@gmail.com> wrote:
A more constructive answer: in some cases, you can change the overall approach. The problem is likely not that it takes 2s instead of 1s to run the program, but that this difference is multiplied many times because you run the same program many times on different input data. In that case, you may try to convert the single-use script into a local "server" that is only started once. Then you change your ``fe.py`` script to connect to it and "download" the result locally. The point is that the server runs in a single process that remains alive. A bientôt, Armin.
Hi, you can run it as a daemon/server(for example a little flask app). This optimization also works for cpython apps if you want to avoid the startup/import time. Can the work be split up per xml file easily? Then perhaps multiprocessing will work nicely for you. Do you need to process all the files each time? Or can you avoid work? cheers, On Wed, Feb 13, 2019 at 3:57 PM Joseph Reagle <joseph.2011@reagle.org> wrote:
On 2/13/19 10:42 AM, René Dudfield wrote:
That would be a big change for a uncertain improvement, so I'm not willing to go there yet. Performance is okay, but I want to see if I can improve it further as a stand-alone.
I've tried multiprocessing too, but it is slower. Parsing the XML can be done in parallel but I suspect the overhead of multi-processing was the drag. I've also thought about caching the XML parse trees, but serializing and reloading pickles of unchanged parse trees seems slower than just parsing the XML anew. Using lru_cache on text processing functions (e.g., removing accents) didn't help either. I haven't been able to find good examples of people using multiprocessing or pypy for XML processing, perhaps this is why. Thank you all for the suggestions!
I wonder how nuitka might do? http://nuitka.net/pages/overview.html m On Wed, Feb 13, 2019 at 10:58:32AM -0500, Joseph Reagle wrote:
-- Matt Billenstein matt@vazor.com http://www.vazor.com/
On 2/13/19 3:15 PM, Matt Billenstein wrote:
Hi Matt, thanks for the suggestion. For my program, Nuitka's performance is very close to Cpython's.
Hi Joseph, wow, interesting project! Would you be up to sharing some example input that you are trying this on? I might be up to looking into why the program is slower than on CPython, but I'd need a way that I can actually run it with realistic input. Cheers, Carl Friedrich On 13/02/2019 16:58, Joseph Reagle wrote:
On 2/13/19 4:03 PM, Carl Friedrich Bolz-Tereick wrote:
Thanks Carl. I don't have a nice package, but the repo is: https://github.com/reagle/thunderdell/ The dependencies should be easy, `fe.py` imports web_little, which will require requests. Everything else should be stdlib. Below I show how to grab the realistic input and run fe.py on it yielding a 1.6MB YAML file. ``` ╭─reagle@hom ~/tmp ╰─➤ wget http://reagle.org/joseph/2005/ethno/field-notes.zip --2019-02-13 16:40:13-- http://reagle.org/joseph/2005/ethno/field-notes.zip ... 2019-02-13 16:40:13 (4.92 MB/s) - ‘field-notes.zip’ saved [1987518/1987518] ╭─reagle@hom ~/tmp ╰─➤ unzip field-notes.zip Archive: field-notes.zip inflating: field-notes-2008-cat.mm inflating: field-notes-2009-cat.mm inflating: field-notes-2010-cat.mm inflating: field-notes-2011-cat.mm inflating: field-notes-2012-cat.mm inflating: field-notes-2013-cat.mm inflating: field-notes-2014-cat.mm inflating: field-notes-2015-cat.mm inflating: field-notes-2016-cat.mm inflating: field-notes-2017-cat.mm inflating: field-notes-2017.mm inflating: field-notes-2018.mm inflating: field-notes.mm ╭─reagle@hom ~/tmp ╰─➤ ~/bin/fe/fe.py -i field-notes.mm -c -o ╭─reagle@hom ~/tmp ╰─➤ head field-notes.yaml --- references: - id: ACLU2018acg type: webpage author: - family: "ACLU" container-title: "American Civil Liberties Union" custom2: "field-notes-2018.mm" issued: year: 2018 ```
participants (6)
-
Armin Rigo -
Carl Friedrich Bolz-Tereick -
Joseph Reagle -
Maciej Fijalkowski -
Matt Billenstein -
René Dudfield