[Python-ideas] A resources tracker ?

[Nick Coghlan]

On 19 Mar 2014 06:09, "Tarek Ziadé" <tarek at ziade.org> wrote:
>
> Hey
>
> ResourceWarning instances that are created in some classes' __del__
> function, like FileIO
> are a great tool to track down a program bad behavior.
>
> In network programming, that's even more important to avoid crashes, or
> huge leaks.
>
> But some of them are very hard to fix because we don't get much context,
> we just
> get warnings at the end of the program execution, when gc.collect is
called.
>
> sys:1: ResourceWarning: unclosed file <_io.FileIO name=18 mode='wb'>
> sys:1: ResourceWarning: unclosed file <_io.FileIO name=17 mode='rb'>
>
> Here I just know that somewhere, 2 file descriptors where not closed.
>
> What I'd like to be able to do is to track down the origin of those
> warnings.
>
> Since __del__ is called asynchronously, it's impossible to track it
> right now (or I don't know how)
>
> What we need is a way to keep track of any resource allocation *when it
> happens*.
>
> Here's an idea: let's add three private functions in Python's io:
>
> def __allocate_resource(fd) => records the file descriptor that was
> allocated, along with the current traceback.
> def __free_resource(fd) => removes the fd from the list.
> def __is_resource_allocated(fd) => tell if the resource is in the list.
>
> These three functions, plugged in somewhere in io's classes, could be
> used in conjunction with ResourceWarning:
> when __del__ is called, if the resource was not freed - we'd be able to
> know where it was created.
>
> Of course these functions are just a brain dump - I have no idea how io
> internals work. But unless I missed it,
> something like I've just described is missing in Python.

You should be able to experiment with something based on tracemalloc
(although it may require patching the io implementation or else installing
a GC callback that looks for particular types).

Cheers,
Nick.

>
> Cheers
> Tarek
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140319/e746ce73/attachment.html>