Proposal: -X importcache to supplement -X importtime for loaded modules
All, I'm writing to propose an adjacent interpreter flag to `-X importtime`: `-X importcache` (open to alternative naming). While `-X importtime` is incredibly useful for analyzing module import times, by design, it doesn't log anything if an imported module has already been loaded. `-X importcache` would provide additional output for every module that's already been loaded: ```
import uuid import time: cached | cached | _io import time: cached | cached | _io import time: cached | cached | os import time: cached | cached | sys import time: cached | cached | enum import time: cached | cached | _io import time: cached | cached | _io import time: cached | cached | collections import time: cached | cached | os import time: cached | cached | re import time: cached | cached | sys import time: cached | cached | functools import time: cached | cached | itertools import time: 151 | 151 | _wmi import time: 18290 | 18440 | platform import time: 372 | 372 | _uuid import time: 10955 | 29766 | uuid
In codebases with convoluted/poorly managed import graphs (and
consequently, workloads that suffer from long import times), the ability to
record all paths to an expensive dependency--not just the
first-imported--can help expedite refactoring (and help scale
identification of this type of issue). More generally, this flag would
provide a more efficient path to tracking runtime dependencies.
As a proof of concept, I was able to hack this functionality into `-X
importtime` by adding a couple lines to `import_ensure_initialized` in
`Python/import.c` (hence the output above). A separate flag is probably
desirable to preserve backwards compatibility.
Looking forward to your feedback,
Noah
Personally, I am -1 on the name - I think '-X importtrace' or similar would be better I think. I am +100 on the functionality - Unless you put in log messages, tracking imports is complex - so something like this I can see being very useful. I know I don't make the decisions, but it seems if this really is only a couple of lines, and all the test cases pass this (or something very similar) should be a considered for 3.12. Thank you Noah, ------ Original Message ------ From: "Noah Kim" <noahbkim@gmail.com> To: python-ideas@python.org Sent: Saturday, 11 Feb, 23 At 00:07 Subject: [Python-ideas] Proposal: -X importcache to supplement -X importtime for loaded modules All, I'm writing to propose an adjacent interpreter flag to `-X importtime`: `-X importcache` (open to alternative naming). While `-X importtime` is incredibly useful for analyzing module import times, by design, it doesn't log anything if an imported module has already been loaded. `-X importcache` would provide additional output for every module that's already been loaded: ```
import uuid import time: cached | cached | _io import time: cached | cached | _io import time: cached | cached | os import time: cached | cached | sys import time: cached | cached | enum import time: cached | cached | _io import time: cached | cached | _io import time: cached | cached | collections import time: cached | cached | os import time: cached | cached | re import time: cached | cached | sys import time: cached | cached | functools import time: cached | cached | itertools import time: 151 | 151 | _wmi import time: 18290 | 18440 | platform import time: 372 | 372 | _uuid import time: 10955 | 29766 | uuid
In codebases with convoluted/poorly managed import graphs (and
consequently, workloads that suffer from long import times), the ability
to record all paths to an expensive dependency--not just the
first-imported--can help expedite refactoring (and help scale
identification of this type of issue). More generally, this flag would
provide a more efficient path to tracking runtime dependencies.
As a proof of concept, I was able to hack this functionality into `-X
importtime` by adding a couple lines to `import_ensure_initialized` in
`Python/import.c` (hence the output above). A separate flag is probably
desirable to preserve backwards compatibility.
Looking forward to your feedback,
Noah
_______________________________________________
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-leave@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
<https://mail.python.org/mailman3/lists/python-ideas.python.org/>
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/GEISYQ5BXWGKT33RWF77EOSOMMMFUBUS/
<https://mail.python.org/archives/list/python-ideas@python.org/message/GEISYQ5BXWGKT33RWF77EOSOMMMFUBUS/>
Code of Conduct: http://python.org/psf/codeofconduct/
<http://python.org/psf/codeofconduct/>
Try raising on https://discuss.python.org/ where you are more likely to have a python core dev comment on this idea. Barry
On 14 Feb 2023, at 15:12, anthony.flury via Python-ideas <python-ideas@python.org> wrote:
Personally, I am -1 on the name - I think '-X importtrace' or similar would be better I think.
I am +100 on the functionality - Unless you put in log messages, tracking imports is complex - so something like this I can see being very useful.
I know I don't make the decisions, but it seems if this really is only a couple of lines, and all the test cases pass this (or something very similar) should be a considered for 3.12.
Thank you Noah,
------ Original Message ------ From: "Noah Kim" <noahbkim@gmail.com> To: python-ideas@python.org Sent: Saturday, 11 Feb, 23 At 00:07 Subject: [Python-ideas] Proposal: -X importcache to supplement -X importtime for loaded modules
All,
I'm writing to propose an adjacent interpreter flag to `-X importtime`: `-X importcache` (open to alternative naming).
While `-X importtime` is incredibly useful for analyzing module import times, by design, it doesn't log anything if an imported module has already been loaded. `-X importcache` would provide additional output for every module that's already been loaded:
```
import uuid import time: cached | cached | _io import time: cached | cached | _io import time: cached | cached | os import time: cached | cached | sys import time: cached | cached | enum import time: cached | cached | _io import time: cached | cached | _io import time: cached | cached | collections import time: cached | cached | os import time: cached | cached | re import time: cached | cached | sys import time: cached | cached | functools import time: cached | cached | itertools import time: 151 | 151 | _wmi import time: 18290 | 18440 | platform import time: 372 | 372 | _uuid import time: 10955 | 29766 | uuid
In codebases with convoluted/poorly managed import graphs (and consequently, workloads that suffer from long import times), the ability to record all paths to an expensive dependency--not just the first-imported--can help expedite refactoring (and help scale identification of this type of issue). More generally, this flag would provide a more efficient path to tracking runtime dependencies. As a proof of concept, I was able to hack this functionality into `-X importtime` by adding a couple lines to `import_ensure_initialized` in `Python/import.c` (hence the output above). A separate flag is probably desirable to preserve backwards compatibility. Looking forward to your feedback, Noah _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/GEISYQ5BXWGKT33RWF77EOSOMMMFUBUS/ Code of Conduct: http://python.org/psf/codeofconduct/ _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/ENFJY2RP73IYPUSRVGHBPNELS5JVQLAA/ Code of Conduct: http://python.org/psf/codeofconduct/
`-X importtime=2` This is expert-only tool. So no need for verbose name. On Sat, Feb 11, 2023 at 10:01 PM Noah Kim <noahbkim@gmail.com> wrote:
All,
I'm writing to propose an adjacent interpreter flag to `-X importtime`: `-X importcache` (open to alternative naming).
While `-X importtime` is incredibly useful for analyzing module import times, by design, it doesn't log anything if an imported module has already been loaded. `-X importcache` would provide additional output for every module that's already been loaded:
```
import uuid import time: cached | cached | _io import time: cached | cached | _io import time: cached | cached | os import time: cached | cached | sys import time: cached | cached | enum import time: cached | cached | _io import time: cached | cached | _io import time: cached | cached | collections import time: cached | cached | os import time: cached | cached | re import time: cached | cached | sys import time: cached | cached | functools import time: cached | cached | itertools import time: 151 | 151 | _wmi import time: 18290 | 18440 | platform import time: 372 | 372 | _uuid import time: 10955 | 29766 | uuid
In codebases with convoluted/poorly managed import graphs (and consequently, workloads that suffer from long import times), the ability to record all paths to an expensive dependency--not just the first-imported--can help expedite refactoring (and help scale identification of this type of issue). More generally, this flag would provide a more efficient path to tracking runtime dependencies. As a proof of concept, I was able to hack this functionality into `-X importtime` by adding a couple lines to `import_ensure_initialized` in `Python/import.c` (hence the output above). A separate flag is probably desirable to preserve backwards compatibility. Looking forward to your feedback, Noah _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/GEISYQ5BXWGKT33RWF77EOSOMMMFUBUS/ Code of Conduct: http://python.org/psf/codeofconduct/
-- Inada Naoki <songofacandy@gmail.com>
I really like this approach!
On Feb 14, 2023, at 9:04 PM, Inada Naoki <songofacandy@gmail.com> wrote:
`-X importtime=2`
This is expert-only tool. So no need for verbose name.
On Sat, Feb 11, 2023 at 10:01 PM Noah Kim <noahbkim@gmail.com> wrote:
All,
I'm writing to propose an adjacent interpreter flag to `-X importtime`: `-X importcache` (open to alternative naming).
While `-X importtime` is incredibly useful for analyzing module import times, by design, it doesn't log anything if an imported module has already been loaded. `-X importcache` would provide additional output for every module that's already been loaded:
```
import uuid import time: cached | cached | _io import time: cached | cached | _io import time: cached | cached | os import time: cached | cached | sys import time: cached | cached | enum import time: cached | cached | _io import time: cached | cached | _io import time: cached | cached | collections import time: cached | cached | os import time: cached | cached | re import time: cached | cached | sys import time: cached | cached | functools import time: cached | cached | itertools import time: 151 | 151 | _wmi import time: 18290 | 18440 | platform import time: 372 | 372 | _uuid import time: 10955 | 29766 | uuid
In codebases with convoluted/poorly managed import graphs (and consequently, workloads that suffer from long import times), the ability to record all paths to an expensive dependency--not just the first-imported--can help expedite refactoring (and help scale identification of this type of issue). More generally, this flag would provide a more efficient path to tracking runtime dependencies. As a proof of concept, I was able to hack this functionality into `-X importtime` by adding a couple lines to `import_ensure_initialized` in `Python/import.c` (hence the output above). A separate flag is probably desirable to preserve backwards compatibility. Looking forward to your feedback, Noah _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/GEISYQ5BXWGKT33RWF77EOSOMMMFUBUS/ Code of Conduct: http://python.org/psf/codeofconduct/-- Inada Naoki <songofacandy@gmail.com>
Hi all, I've reposted this to the python.org discussion board with a followup on `-X importtime=2`. I think it'll be more accessible if we move the conversation there. Here's a link: https://discuss.python.org/t/x-importtrace-to-supplement-x-importtime-for-lo... On Tue, Feb 14, 2023 at 9:45 PM Noah Kim <noahbkim@gmail.com> wrote:
I really like this approach!
On Feb 14, 2023, at 9:04 PM, Inada Naoki <songofacandy@gmail.com> wrote:
`-X importtime=2`
This is expert-only tool. So no need for verbose name.
On Sat, Feb 11, 2023 at 10:01 PM Noah Kim <noahbkim@gmail.com> wrote:
All,
I'm writing to propose an adjacent interpreter flag to `-X importtime`: `-X importcache` (open to alternative naming).
While `-X importtime` is incredibly useful for analyzing module import times, by design, it doesn't log anything if an imported module has already been loaded. `-X importcache` would provide additional output for every module that's already been loaded:
```
import uuid import time: cached | cached | _io import time: cached | cached | _io import time: cached | cached | os import time: cached | cached | sys import time: cached | cached | enum import time: cached | cached | _io import time: cached | cached | _io import time: cached | cached | collections import time: cached | cached | os import time: cached | cached | re import time: cached | cached | sys import time: cached | cached | functools import time: cached | cached | itertools import time: 151 | 151 | _wmi import time: 18290 | 18440 | platform import time: 372 | 372 | _uuid import time: 10955 | 29766 | uuid
In codebases with convoluted/poorly managed import graphs (and consequently, workloads that suffer from long import times), the ability to record all paths to an expensive dependency--not just the first-imported--can help expedite refactoring (and help scale identification of this type of issue). More generally, this flag would provide a more efficient path to tracking runtime dependencies. As a proof of concept, I was able to hack this functionality into `-X importtime` by adding a couple lines to `import_ensure_initialized` in `Python/import.c` (hence the output above). A separate flag is probably desirable to preserve backwards compatibility. Looking forward to your feedback, Noah _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/GEISYQ5BXWGKT33RWF77EOSOMMMFUBUS/ Code of Conduct: http://python.org/psf/codeofconduct/-- Inada Naoki <songofacandy@gmail.com>
On Wed, 15 Feb 2023 at 02:06, Inada Naoki <songofacandy@gmail.com> wrote:
`-X importtime=2`
This is expert-only tool. So no need for verbose name.
+1 that re-using existing parameter names is both concise and also convenient. However: do we have an opportunity to make the parameter _value_ both concise and also memorable? `-X importtime` # existing behaviour (backwards-compatible) `-X importtime=initial` # existing behaviour (memorable) `-X importtime=all` # record timing for every import that happens, including cases when the module is retrieved from cache Regards, James
participants (5)
-
anthony.flury -
Barry -
Inada Naoki -
James Addison -
Noah Kim