Re: [Python-Dev] PEP 3147: PYC Repository Directories
On Jan 31, 2010, at 01:06 PM, Ron Adam wrote:
With a single cache directory, we could have an option to force writing bytecode to a desired location. That might be useful on it's own for creating runtime bytecode only installations for installers.
One important reason for wanting to keep the bytecode cache files colocated with the source files is that I want to be able to continue to manipulate $PYTHONPATH to control how Python finds its modules. With a single system-wide cache directory that won't be easy. E.g. $PYTHONPATH might be hacked to find the source file you expect, but how would that interact with how Python finds its cache files? I'm strongly in favor of keeping the cache files as close to the source they were generated from as possible. -Barry
Barry Warsaw wrote:
On Jan 31, 2010, at 01:06 PM, Ron Adam wrote:
With a single cache directory, we could have an option to force writing bytecode to a desired location. That might be useful on it's own for creating runtime bytecode only installations for installers.
One important reason for wanting to keep the bytecode cache files colocated with the source files is that I want to be able to continue to manipulate $PYTHONPATH to control how Python finds its modules. With a single system-wide cache directory that won't be easy. E.g. $PYTHONPATH might be hacked to find the source file you expect, but how would that interact with how Python finds its cache files? I'm strongly in favor of keeping the cache files as close to the source they were generated from as possible.
Yes, I agree, after thinking about it, it does seems like it may be more complex than I first thought. I think the folder-per-folder option sounds like the best default option at this time. It reduces folder clutter for the python developer and may loosen the link between source files and byte code files just enough that it will be easier to experiment with more flexible modes later. It seems to me that in the long run, (probably no time soon), it might be nice to even do away with on disk byte code altogether unless it's explicitly asked for. As computers get faster, the time it takes to compile byte code may become a smaller and smaller percent of the total run time. That is unless the size of python programs increase at the same rate or faster. To tell the truth in most cases I hardly notice the extra time the first run takes compared to later runs with the precompiled byte code. Yes it may be a few seconds at start up, but after that it's usually not a big part of the execution time. Hmmm, I wonder if there's a threshold in file size where it really doesn't make a significant difference? Regards, Ron
Ron Adam wrote:
To tell the truth in most cases I hardly notice the extra time the first run takes compared to later runs with the precompiled byte code. Yes it may be a few seconds at start up, but after that it's usually not a big part of the execution time. Hmmm, I wonder if there's a threshold in file size where it really doesn't make a significant difference?
It's relative to runtime for the application itself (long-running applications aren't going to notice as much of a percentage effect on runtime) as well as to how many Python files are actually imported at startup (only importing a limited number of modules, importing primarily extension modules or effective use of a lazy module loading mechanism will all drastically reduce the proportional impact of precompiled bytecode) We struggle enough with startup time that doing anything that makes it slower is rather undesirable though. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On 2/8/2010 7:54 AM, Nick Coghlan wrote:
Ron Adam wrote:
To tell the truth in most cases I hardly notice the extra time the first run takes compared to later runs with the precompiled byte code. Yes it may be a few seconds at start up, but after that it's usually not a big part of the execution time. Hmmm, I wonder if there's a threshold in file size where it really doesn't make a significant difference?
It's relative to runtime for the application itself (long-running applications aren't going to notice as much of a percentage effect on runtime) as well as to how many Python files are actually imported at startup (only importing a limited number of modules, importing primarily extension modules or effective use of a lazy module loading mechanism will all drastically reduce the proportional impact of precompiled bytecode)
We struggle enough with startup time that doing anything that makes it slower is rather undesirable though.
Definitely. I have even wondered whether it would be possible to cache not just the bytecode for initializing a module, but also the initialized module itself (perhaps minus the name bindings for other imported modules). Terry Jan Reedy
Terry Reedy wrote:
Definitely. I have even wondered whether it would be possible to cache not just the bytecode for initializing a module, but also the initialized module itself (perhaps minus the name bindings for other imported modules).
Not easily, since running the module may have other side effects that can't be cached. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
participants (4)
-
Barry Warsaw -
Nick Coghlan -
Ron Adam -
Terry Reedy