Hi Barry, It's not just about leaks. You might want to know if certain objects are occupying a lot of memory by themselves. Then you can optimize the memory usage of these objects. Another possibility is to do binary instrumentation and see how the user code is interacting with objects. If we can't tell which objects are created by the interpreter internals, then interpreter accesses and user accesses would be mixed together. It's likely that some accesses would be connected of course, but I don't think this should be outright labeled as useless. Also, I'm not saying "we must implement this because it's so useful." My original intention is to understand: (1) is the differentiation being done at all? (2) if it's not being done, why? (3) does it make sense to implement it? So far I think I've got the answers to 1 & 2--it's not being done because people don't find it useful. The answer to 3 is most likely "no" due to the costs, but it would be nice if someone could weigh in on this part. Maybe there's some workaround. Thanks On Mon, Jul 20, 2020 at 5:16 PM Barry <barry@barrys-emacs.org> wrote:
On 20 Jul 2020, at 18:51, Wenjun Huang <wenjunhuang@umass.edu> wrote:
Hi Guido,
It's great to hear from you directly :)
Sorry for not mentioning this earlier. The use case here is profiling. Specifically, Scalene: https://github.com/emeryberger/scalene.
At the moment, Scalene does stack inspection to decide if an allocation is from the user. If there are hooks helping with the differentiation, it's arguably more efficient. Additionally, users might want to know about memory allocated explicitly by their objects instead of by the interpreter internals.
In my day job and open source projects I care about memory leaks and fixing them.
If my code creates a list you call that user allocated? If memory increases as part of python book keeping that is counted as something else?
Why is that interesting?
When looking for memory leaks I have not cared about that distinction. Should I be caring? Am I missing something?
Barry
If the interpreter doesn't have this functionality already, it would probably be difficult to add, as the existing allocation API is called throughout the interpreter code base.
Thanks
On Sun, Jul 19, 2020 at 10:39 PM Guido van Rossum <guido@python.org> wrote:
What purpose do you have in mind for making this distinction? Even if it could be done easily (which I doubt), why would this be useful?
On Sun, Jul 19, 2020 at 19:01 <wenjunhuang@umass.edu> wrote:
Hello,
The API provided by PEP 445 makes it possible to intercept allocation requests through hooks, but it seems that both user allocations and interpreter allocations are sent to the hooks.
Here, user allocations refer to those that are triggered explicitly by the code (e.g. memory allocations to hold the integer created by x = 1), and interpreter allocations refer to everything else (e.g. memory allocations for internal states).
I've poked around a bit in the interpreter source code, and I think such differentiations aren't being done at all, so all allocations are directed to the same set of API. If that's indeed the case, why is the interpreter implemented this way? Would it make sense to implement the differentiation?
Thanks _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/HJQW2M... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile)
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/H33JNM... Code of Conduct: http://python.org/psf/codeofconduct/