[pypy-dev] jitlog 2.0

[Richard Plangger]

Hello,

I have spent a lot of my free time to improve our jitviewer. Finally I'm
happy to present some results and I want to get some feedback (bad or good).

I run a test server (at http://217.172.172.207:8000) so without
installing anything you can just have a quick look. Here [1] is
richards.py uploaded to that server.

Here some more information:

What has changed?
=================

We already host vmprof.com for profile data. This is only part of the
information that is interesting. I think it would be very helpful to see
what the JIT compiler currently produces in a more 'global view'. The
combination of profile data available at vmprof.com and the log output
of pypy can really help to understand what is going on inside the VM.

The new log output is easier to process and allows to fully rebuild the
information in jit viewer (with no dependency to the pypy source code).
I have added a svg chart that displays the trace tree. It can be
examined further which is really something. I think that is a much nicer
high level view of our tracing JIT compiler than in the old jit viewer.

Technically
-----------

I have mostly rewritten the log format. It writes a binary file and
currently encodes pretty much every information the PYPYLOG does.
It is a custom binary format similar to the one in vmprof. It is
intended to reduce the size of jitlog.

We discussed that it would be a rather good idea to use a standardized
binary protocol instead of creating our own. I agree with that, but
there are no real options without significant effort. E.g. capnproto
would have required me to write the encoder on the rpython side and
adjust the javascript capnproto library that it is able to run in the
browser (nodejs only).

I decided to stick with the custom very simple binary format and
extended it that it is able to version the jitlog. We can support
several versions, ideally never dropping support for any official
version of the jitlog.

I extended vmprof-python [2] to read that binary file format and upload
it along with the normal vmprof data. It has a pretty printer which you
can use to get information on the console. It's state is very minimal
and my current idea is to extend a very minimal api to be able to query
the jitlog. I have something in mind like:

$ python -m vmprof.prettyprint /tmp/abcdef.jitlog --query
'trace.get_func_name() == "schedule"'
> 1x trace matched your query
> trace started in the function schedule in path/to/richards.py:501
> label(...)
> ...
> jump(...)

A sample command to upload jitlog data would be:

python -m vmprof --jitlog --web --web-url http://217.172.172.207:8000
<path to script>/richards.py

Here are some things I'm aware of and want to change:

* No compression of JSON. That would greatly speed up loading!
* The live ranges are not connected when assembler is shown (style fix)
* The entry count (of a trace) displayed in the trace view is always 0,
the data is present but not serialized when invoked with -m vmprof --jitlog

What's next
===========

I hope someone could take a look at the new-jit-log branch so we can
discuss if it.

I also want to achieve:

* Feature parity with the old jitviewer (what is missing?)
* Be able to jump from vmprof profile data to jitlog and back
* Show the jumps in the trace view
* Merge to default :)
* Upload the source code to give a better pictures of what has been
produced from which source line
* Display merge point information in the resoperation list
* Does it work on IE?

Cheers,
Richard

[1] http://217.172.172.207:8000/#/92a1ff97c87d187eee3f60a0a088e669/traces
[2] https://github.com/vmprof/vmprof-python

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20160429/d93fee3c/attachment.sig>