[pypy-dev] Paper on the HOPE JIT for Python

[Yury V. Zaytsev]

Hi Armin,

On Sun, 2015-06-21 at 14:50 +0200, Armin Rigo wrote:
> 
> There is not much we can actively do.  Of course running a loop only
> 100 times gives a bad idea about the performance of any real JIT.

Just to clarify, from what I understood, they basically ran

    timeit.repeat(stmt, repeat=100, number=1) [1]

where `stmt` is each of their benchmarks.

[1]: https://github.com/cosmo-ethz/hope/blob/master/benchmarks/native_cpp_gen.ipynb

The benchmarks themselves are partly taken from the Julia website (fib,
quicksort, pisum), and partly original (polynomial, simplify, pairwise
distance and point spread function).

Some benchmarks contain nested inner loops (e.g. pisum has an outer loop
that runs 500 times and an inner loop that runs 10000 times), and some
do not (e.g. fib measures function call overhead, yay!).

Unlike for the rest of the tools, the raw data for PyPy is not
available, so it's unclear to me whether they allowed the JIT to warm up
in all cases, which is what I alluded to in my previous email.

By the way, upon closer inspection, it seems that they have actually
tried some of the optimizations that I've mentioned for the Cython
implementation of the "point spread function", and it ran faster than
their reference implementation in C++, which is why, I guess, they chose
to not to include it in the table ;-), and used the unoptimized version
instead... 

> (Did they include the time spent calling gcc/llvm, for some of the
> other "JITs"?)

No, they didn't, but what they did was to call each function once with
the right parameters at the beginning of the benchmarking session in
order to allow all of the tools to compile the code (and infer types,
when applicable).

> We can however point people to your mail, if they come and complain to
> us. It gives a nice summary of a point of view much closer to our own.

I wonder if I should let the authors know of this thread; on one hand,
they might learn something from it, but, on the other hand, detailed
discussion of the flaws of this paper is probably offtopic for this
list.

On the third hand ^__^, however, I would be uncomfortable with offlist
discussions, as they surely will take a lot of my private time, but
likely have very little yield for the world :-/

> That said, I'm afraid the scientific community will continue to
> produce "JITs" as long as there is no well-accepted alternative to
> CPython. I think that fighting every attempt or paper we disagree with
> is pointless.  I'd rather get PyPy to progress to a point where it be
> can used as a drop-in replacement of CPython in this domain too. (I'm
> not in numpy at all myself, so it's more a would-be-nice wish here.)

Yes, I agree with you on that, but personally, I don't see much of a
problem with folks out there playing with source to source translation
and type inference; to my mind that's perfectly alright.

My grief with the re-inventors of "JITs" is that as a rule they use
confusing terminology, conduct flawed benchmarks and publish their
reports in specialist journals, which are not able to properly screen
such articles... oh well.

-- 
Sincerely yours,
Yury V. Zaytsev