-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 06/09/13 17:14, Guido van Rossum wrote:

I've heard good things about DTRACE but never used it myself.

Do I understand correctly that you have to build a separate Python executable with it turned on?

It is a patch you apply on stock Python and you get a new configure option: "--with-dtrace". If you compile Python with that flag, you get a python interpreter with dtrace probes on it.

I noticed one odd thing in the patch: apparently the dtrace module only exists to have a flag that tells whether it is enabled. Can't that flag be added to the sys module?

My plan is to (in the future) use that module to create new probes on the fly, like http://chrisa.github.io/blog/2011/12/04/libusdt-runtime-dtrace-providers/, and to export functions to communicate data to dtrace scripts, if running.

That is, this module is currently a shell I plan to fill.

- -- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 06/09/13 17:29, Charles-François Natali wrote:

IMO, that's a large, intrusive patch, which distracts the reader from the main code and logic.

Yes, the patch is intrusive. It must be, to get its goals. Could be improved, nevertheless. Help and suggestions welcome.

I want to write a probe for the GIL, but I didn't because I know adding the cost of an extra single machine code branch would be anathema here :-) (lets do baby steps), but I would love to be able to watch with detail all interaction between the GIL, threads, and OS scheduling on my Solaris :). That is very valuable information to have, for instance, to guide future improvements of the GIL. How can you get that kind of information with any other tool?

IMO it's not worth it (personally strace/gdb/valgrind are more than enough for me, and we''re about to gain memory tracing with Victor's tracemalloc).

The main value of DTrace is systemwide observability. You can see something "strange" at kernel level and trace it to a particular line of code in a random Python script. There is no other tool that can do that. You have complete transversal observability of ALL the code running in your computer, kernel or usermode, clean reports with threads, etc.

Valgrind doesn't work on Solaris and *BSD support is unclean.

You can run a dtrace script in a long running python process if you need it, after launching it, at runtime, and when the dtrace script is done, the python program keeps running without any penalty. Just poking around, for instance. I do it constantly for, for instance, profiling covering both Python as any C code/libraries called from it.

Maybe the biggest objection would be that most python-devs are running Linux, and you don't have dtrace support on linux unless you are running Oracle distribution. But world is larger than linux, and there are some efforts to port DTrace to Linux itself. DTrace is available on Solaris and derivatives, MacOS X and FreeBSD.

- -- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz

I think there are a couple of issues.

- In order for the patch to be acceptable, we'd need someone to take responsibility for owning and maintaining it for at least several years. Jesus, are you willing to commit to this?

- I think it's not all that important whether any core developer would want to use this for themselves; it's enough if it's useful for others to have this, and it's clear that at least *some* users really like this. Different people like different tools, and that's fine.

- There are good reasons (e.g. standardizing a set of probes) for having a somewhat supported version around.

- Since this *has* to be done in the form of a patch plus a new configure option, an externally-maintained patch is awkward at least, and likely to be continually out of date -- this is not something that you can just as easily put on PyPI as the recommended 3rd party solution.

- So I'm personally inclined to say "yes, Jesus, if you are willing to maintain this code (while it is in the core) for several more years, please invest more of your time, and, assuming you can satisfy the code reviewers' feedback, we want to accept this in the core, hopefully in 3.4 if you get it in an acceptable state before 3.4 beta 1 is released."

On Fri, Sep 6, 2013 at 10:12 AM, Charles-François Natali cf.natali@gmail.com wrote:

...

The main value of DTrace is systemwide observability. You can see something "strange" at kernel level and trace it to a particular line of code in a random Python script. There is no other tool that can do that. You have complete transversal observability of ALL the code running in your computer, kernel or usermode, clean reports with threads, etc.

Don't get me wrong, I'm not saying DTrace is useless. I'm just saying that, as far as I'm concerned, I've never had any trouble debugging/tunning a Python script with non-intrusive tools (strace, gdb, valgrind, and oprofile for profiling). Of course, this includes analysing bug reports.

...

Maybe the biggest objection would be that most python-devs are running Linux, and you don't have dtrace support on linux unless you are running Oracle distribution. But world is larger than linux, and there are some efforts to port DTrace to Linux itself. DTrace is available on Solaris and derivatives, MacOS X and FreeBSD.

That's true, I might have a different opinion if I used Solaris. But that's not the case, so te me, the cognitive overhead incurred by this large patch isn't worth it.

So I'm -1, but that's a personal opinion :-)

cf _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 06/09/13 17:41, Antoine Pitrou wrote:

You should start by addressing review comments: http://bugs.python.org/issue13405#msg151751

Antoine, my first step now is to poke Python-DEV about this subject. If the consensus is "DON'T" I will probably maintain this patch by myself. I saw big "BUTs" with this work in the bug tracker, so I rather prefer to ask for actual interest before investing in iron out patch implementation details.

If devs actually want it, then next step is to improve current work to make it acceptable in mainline. There are quite a few ugly corners I would love to file out, beside your (really valuable) feedback.

That is how I see it.

Right now, I agree with Charles-François: your patch is too intrusive.

It is intrusive. Yes. I think it must be, by its own nature. Probably room for improvement and code transparency. But... are Python-DEVs interested in the project?. That is the point :)

- -- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz

On 2013-09-06, at 19:05 , Antoine Pitrou wrote:

On Fri, 06 Sep 2013 18:14:26 +0200 Jesus Cea jcea@jcea.es wrote:

...

...

Right now, I agree with Charles-François: your patch is too intrusive.

It is intrusive. Yes. I think it must be, by its own nature. Probably room for improvement and code transparency. But... are Python-DEVs interested in the project?. That is the point :)

Well, I'm not *personally* interested in anything that only addresses Solaris, OS X and the like :)

For what it's worth, there's also a linux port and oracle's distro has dtrace support.

And, no, it doesn't have to be *that* intrusive. Take a look at Dave Malcolm's systemtap patch, which IIRC takes a much more sensible approach.

Is there a possibility of compatibility there, using the same placeholders for a --with-dtrace and --with-systemtap build?

Jesus seems to instrument more points than Dave, but the extra points could just be defined to nothing in the systemtap implementation.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 06/09/13 20:33, Antoine Pitrou wrote:

On Fri, 06 Sep 2013 18:14:26 +0200 Jesus Cea jcea@jcea.es wrote:

...

It is intrusive. Yes. I think it must be, by its own nature. Probably room for improvement and code transparency. But... are Python-DEVs interested in the project?. That is the point :)

As a concrete data point: - here are Dave's modifications to ceval.c for systemtap: http://bugs.python.org/review/14776/diff/5177/Python/ceval.c - here are your modifications to ceval.c for dtrace: http://bugs.python.org/review/13405/diff/6151/Python/ceval.c

Unfair, because that code is not doing the same thing.

Most of the extra complexity is there to deal with DTRACE ability to provide meaningful stackframes, with Python code instead of CPython evaluation loop. This is kind of magical.

Look at this:

"" [root@stargate-host /]# ps -lAf|grep -i correo 0 S correo 1371 1 0 40 20 ? 277063 ? Jul 29 ? 90:43 /usr/local/bin/python /export/home/ 0 S root 20397 20373 0 40 20 ? 1294 ? 05:18:16 pts/12 0:00 grep -i correo

[root@stargate-host /]# dtrace -n 'python1371:::function-entry {jstack();}'

5 3164 PyEval_EvalFrameEx:function-entry libpython2.7.so.1.0`PyEval_EvalFrameEx+0x89a [ /usr/local/lib/python2.7/logging/__init__.py:1332 (isEnabledFor) ] libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231 [ /usr/local/lib/python2.7/logging/__init__.py:1202 (log) ] libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0`PyEval_EvalFrameEx+0x59ae [ /home/correo/durus/connection.py:483 (shrink) ] libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231 [ /home/correo/durus/connection.py:225 (shrink_cache) ] libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231 [ /home/correo/durus/connection.py:303 (commit2) ] libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231 [ /home/correo/durus/connection.py:261 (commit) ] libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231 [ /export/home/correo/lmtp.py:191 (_monitor) ] libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0`PyEval_EvalFrameEx+0x59ae [ /export/home/correo/lmtp.py:125 (process_vacation2) ] libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0`PyEval_EvalFrameEx+0x59ae [ /export/home/correo/lmtp.py:138 (process_vacation) ] libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0`function_call+0x192 libpython2.7.so.1.0`PyObject_Call+0x5c libpython2.7.so.1.0`PyEval_EvalFrameEx+0x2b94 [ /usr/local/lib/python2.7/threading.py: ] libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231 libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231 libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0`function_call+0xa4 libpython2.7.so.1.0`PyObject_Call+0x5c libpython2.7.so.1.0`instancemethod_call+0xa1 libpython2.7.so.1.0`PyObject_Call+0x5c libpython2.7.so.1.0`PyEval_CallObjectWithKeywords+0x58 libpython2.7.so.1.0`t_bootstrap+0x52 libc.so.1`_thr_setup+0x4e libc.so.1`_lwp_start

^C """

Lets say I want to know about what codepaths are hitting the OS "sync" features, so I do this:

""" [root@stargate-host /]# dtrace -n 'syscall::fdsync:entry/pid==1371/{jstack();}' dtrace: description 'syscall::fdsync:entry' matched 1 probe

CPU ID FUNCTION:NAME 3 58344 fdsync:entry libc.so.1`__fdsync+0x15 libdb-6.0.so`__os_fsync+0x91 libdb-6.0.so`__log_flush_int+0x685 libdb-6.0.so`__log_flush+0x6a libdb-6.0.so`__log_flush_pp+0x121 _pybsddb.so`DBEnv_log_flush+0x39 libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6992 [ /export/home/correo/durus-berkeleydbstorage/berkeleydb_storage.py:918 (flush) ] libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231 [ /export/home/correo/lmtp.py:1005 (lmtp2) ] libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231 [ /export/home/correo/lmtp.py:763 (lmtp) ] libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0`function_call+0x192 libpython2.7.so.1.0`PyObject_Call+0x5c libpython2.7.so.1.0`PyEval_EvalFrameEx+0x2b94 [ /export/home/correo/lmtp.py:214 (_ignore_socket_errors) ] libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0`function_call+0x192 libpython2.7.so.1.0`PyObject_Call+0x5c libpython2.7.so.1.0`PyEval_EvalFrameEx+0x2b94 [ /usr/local/lib/python2.7/threading.py:504 (run) ] libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231 [ /usr/local/lib/python2.7/threading.py:551 (__bootstrap_inner) ] libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231 [ /usr/local/lib/python2.7/threading.py:524 (__bootstrap) ] libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0`function_call+0xa4 libpython2.7.so.1.0`PyObject_Call+0x5c libpython2.7.so.1.0`instancemethod_call+0xa1 libpython2.7.so.1.0`PyObject_Call+0x5c libpython2.7.so.1.0`PyEval_CallObjectWithKeywords+0x58 libpython2.7.so.1.0`t_bootstrap+0x52 libc.so.1`_thr_setup+0x4e libc.so.1`_lwp_start """

I could even aggregate stacktraces and print them by frequency, cost, whatever.

Be aware that to provide this information I am not even using the probes I have integrated in Python. The program is in production and I don't need to stop it to make those measurements. They are 100% "live", not interrupting the service and without any kind of support in that Python program.

I can even plug to TCP Dtrace providers in kernel and know what exact line of Python code is sending those bytes that are being delayed because the Nagle algorithm https://en.wikipedia.org/wiki/Nagle%27s_algorithm. I actually digged this five years ago. what I hunt it was! :-)

PS: And it is supporting Unicode :-), another big complexity in the code.

- -- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz

On Sat, 7 Sep 2013 08:57:07 +0200 Xavier Morel python-dev@masklinn.net wrote:

On 2013-09-07, at 05:40 , Jesus Cea wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 06/09/13 20:33, Antoine Pitrou wrote:

...

On Fri, 06 Sep 2013 18:14:26 +0200 Jesus Cea jcea@jcea.es wrote:

...

It is intrusive. Yes. I think it must be, by its own nature. Probably room for improvement and code transparency. But... are Python-DEVs interested in the project?. That is the point :)

As a concrete data point: - here are Dave's modifications to ceval.c for systemtap: http://bugs.python.org/review/14776/diff/5177/Python/ceval.c - here are your modifications to ceval.c for dtrace: http://bugs.python.org/review/13405/diff/6151/Python/ceval.c

Unfair, because that code is not doing the same thing.

Most of the extra complexity is there to deal with DTRACE ability to provide meaningful stackframes, with Python code instead of CPython evaluation loop. This is kind of magical.

Antoine will correct me if I'm wrong, but I believe his point is less about the complexity of dtrace provision and more about how much of it lives in ceval.c: the systemtap provision also takes quite a bit of code, but almost all of that code is extracted into a separate file and only a pair of calls live in ceval.c

Yes, that's exactly my point. There can be an arbitrarily complex ceval-dtrace.h for all I care :-)

Thanks for the examples, by the way.

Regards

Antoine.