[pypy-dev] crosstwine linker

[Damien Diederen]

Hi,

I'm the author of the CrossTwine Linker framework.  I hope most of you
will be at EuroSciPy so that we get to meet in-person; in the meantime,
I'm going to try to cover some of the questions raised in this thread:

Antonio Cuni <anto.cuni at gmail.com> writes:
> Laura Creighton wrote:
>> Anybody heard of this?
>> http://crosstwine.com/linker/index.html
>> 
>> Damien Diederen is giving a talk about speeding up python at EuroSciPy,
>> and since this is his company, it should be about this.

It is.  I'm going to talk about some of the ideas, inner workings, and
methodologies underlying the project.  I will also discuss some of the
challenges of the current Python environment, and how we overcome
them—the ones we have figured out, that is.

[ Feel free to suggest topics for me to cover if you are interested in a
  particular aspect! ]

> I skimmed over the website and read the whitepaper few weeks ago.
> Honestly, I don't understand if and how they manage to speedup python.
> In fact, they show results for only 4 benchmarks: one is a version of
> bpnn (which we also have, somewhere) modified to be "static enough" to
> be compiled by shedskin; the other three are really microbenchmarks.

You are perfectly right: alpha 1 focused on microbenchmarks (one has to
start somewhere, right? :)  Cutting new releases, updating the website
and publishing a not-so-lame set of benchmarks are very high on my TODO
list—but unfortunately not quite at the top yet.

> Since he doesn't show any other benchmark, I wonder if his product
> gives any speedup at all on real programs.

Real programs are very much on the radar screen now that the framework
is getting more mature.  We are seeing interesting progress on the
development versions, even if there is still a *lot* of work to do.  On
the positive side, the growing set of components at our disposal seems
to “easily” accommodate more and more specialization scenarios.

>>> Maciej Fijalkowski wrote:
>>>> As usual with such products, we don't know if they implemented 100%
>>>> python or 99%.  Can they run existing apps?

The Python demo is a derivative of the default interpreter, and we were
very careful to avoid breakage while integrating pieces of the “bolt-on”
JIT and utilities.  We are actually taking quite a speed hit to
accommodate some of the corner cases, and hope to suppress some of that
overhead by being a bit “smarter” in the future.

So we should be able to run 100% of existing apps, and differences in
behaviour are to be considered bugs.

Maciej Fijalkowski <fijall at gmail.com> writes:
> Ok, so I did my homework and performed a couple of checks:
>
> * it seems to mostly work on a couple of examples that I checked
>
> * I was a bit comparing aples to oranges (python 2.6 vs xtpython 3.1),
>   but it basically gives a bit of speedup (10%-2x) over loops which do stuff.

We have a 2.6 version in the labs, to be included in the next (alpha 2)
release.  This will make comparisions more interesting, and allow us to
focus more on real-world scenarios.

> * checking it on almost anything but stuff written by hand makes no sense
>   since it's 3.1

Tell me about it!  :)

[ Starting with 3.x was an “accident.”  I am a relative newcomer to the
  Python world, and it looked like everybody was about to jump to 3.0
  when we got started.  Oh well… ]

> * result binary is 64bit only.

Yes, and while we could definitely develop a 32 bit version or other
architecture backends, these are not currently part of the roadmap.
We're firmly in x86-64 land for the time being, except if somebody
steers us away with a particular use case.

Cheers,
Damien

-- 
http://crosstwine.com

"Strong Opinions, Weakly Held"
                 -- Bob Johansen