[Python-Dev] PEP 578: Python Runtime Audit Hooks

[Steve Dower]

On 29Mar2019 0334, Christian Heimes wrote:
> On 28/03/2019 23.35, Steve Dower wrote:
>> Audit Hook
>> ----------
>>
>> In order to observe actions taken by the runtime (on behalf of the
>> caller), an API is required to raise messages from within certain
>> operations. These operations are typically deep within the Python
>> runtime or standard library, such as dynamic code compilation, module
>> imports, DNS resolution, or use of certain modules such as ``ctypes``.
>>
>> The following new C APIs allow embedders and CPython implementors to
>> send and receive audit hook messages::
>>
>>     # Add an auditing hook
>>     typedef int (*hook_func)(const char *event, PyObject *args,
>>                              void *userData);
>>     int PySys_AddAuditHook(hook_func hook, void *userData);
>>
>>     # Raise an event with all auditing hooks
>>     int PySys_Audit(const char *event, PyObject *args);
>>
>>     # Internal API used during Py_Finalize() - not publicly accessible
>>     void _Py_ClearAuditHooks(void);
>>
>> The new Python APIs for receiving and raising audit hooks are::
>>
>>     # Add an auditing hook
>>     sys.addaudithook(hook: Callable[[str, tuple]])
>>
>>     # Raise an event with all auditing hooks
>>     sys.audit(str, *args)
>>
>>
>> Hooks are added by calling ``PySys_AddAuditHook()`` from C at any time,
>> including before ``Py_Initialize()``, or by calling
>> ``sys.addaudithook()`` from Python code. Hooks cannot be removed or
>> replaced.
> 
> Hi Steve,
> 
> I wonder if the hooks could be replaced by a more efficient mechanism.
> These days, Linux, macOS, and most recently Windows [1] support dtrace
> probes. DTrace is a very powerful and efficient mechanism to trace
> user-space processes from Kernel space. At least we should consider to
> add DTrace probes to the auditing framework.
> 
> [1]
> https://techcommunity.microsoft.com/t5/Windows-Kernel-Internals/DTrace-on-Windows/ba-p/362902

Calling into those frameworks will still require as much work as these 
hooks do, and also make it very difficult to do things like auditing 
unit tests or using pure-Python code when you're not concerned about 
initialization (e.g. in a long-running web server).

So I'm inclined to say that if you want those probes, you can enable 
them by adding a hook that calls into them? A similar argument is made 
for using ETW on Windows (which will work on versions of Windows that 
have been released, unlike DTrace) (and yes, this is a real argument 
I've already had over this proposal ;) ), so I really think leaving it 
open-ended and Python-specific is the best approach.

(Reading further down the link you provided, it seems DTrace in Windows 
will only be enabled for essentially-administrators. So that rules it 
out as a substitute for this proposal in my opinion.)

Cheers,
Steve