[Python-Dev] Proto-PEP regarding writing bytecode files

[Brett Cannon]

[Skip Montanaro]

> Folks,
>
> Here's a first stab at a PEP about controlling generation of bytecode
> files.  Feedback appreciated.
>

Now I don't need to worry as much about summarizing this thread.  =)

<snip>
> Issues
> ======
>
> - When looking for a bytecode file should the directory holding the
>   source file be considered as well, or just the location implied by
>   PYCROOT?  If so, which should be searched first?  It seems to me
>   that if a module lives in /usr/local/lib/python2.3/mod.py and was
>   installed by root without PYCROOT set, you'd want to use the
>   bytecode file there if it was up-to-date without ever considering
>   os.environ["PYCROOT"] + "/usr/local/lib/python2.3/".  Only if you
>   need to write out a bytecode file would anything turn up there.
>

In other words you are wondering about the situation of where root
installs Python (and thus generates .pyo files in the install directory)
but PYCROOT is set later on.  Seems reasonable to check the holding
directory, just don't know how often this situation will come up.  But if
you are going to all the trouble of having a separate place to keep your
byte-compiled files, shouldn't you keep *all* of the byte-compiled files
you want to use together?  Since you are storing them it is a one-time
compile and thus not that big of a deal.

> - Operation on multi-root file systems (e.g., Windows).  On Windows
>   each drive is fairly independent.  If PYCROOT is set to C:\TEMP and
>   a module is located in D:\PYTHON22\mod.py, where should the bytecode
>   file be written?  I think a scheme similar to what Cygwin uses
>   (treat drive letters more-or-less as directory names) would work in
>   practice, but I have no direct experience to draw on.  The above
>   might cause C:\TEMP\D\PYTHON22\mod.pyc to be written.
>

Using the drive letter as a directory names seems like a good solution.

>   What if PYCROOT doesn't include a drive letter?  Perhaps the current
>   drive at startup should be assumed.
>

Probably should.

> - Interpretation of a module's __file__ attribute.  I believe the
>   __file__ attribute of a module should reflect the true location of
>   the bytecode file.  If people want to locate a module's source code,
>   they should use imp.find_module(module).
>
> - Security - What if root has PYCROOT set?  Yes, this can present a
>   security risk, but so can many things the root user does.  The root
>   user should probably not set PYCROOT except during installation.
>   Still, perhaps this problem can be minimized.  When running as root
>   the interpreter should check to see if PYCROOT refers to a
>   world-writable directory.  If so, it could raise an exception or
>   warning and reset PYCROOT to the empty string.  Or, see the next
>   item.
>
> - More security - What if PYCROOT refers to a general directory (say,
>   /tmp)?  In this case, perhaps loading of a preexisting bytecode file
>   should occur only if the file is owned by the current user or root.
>   (Does this matter on Windows?)
>

To comment on both security issues: if someone is worrying about security
they should just turn byte-compiling off completely.

> - Runtime control - should there be a variable in sys (say,
>   sys.pycroot) which takes on the value of PYCROOT (or an empty string
>   or None) and which is modifiable on-the-fly?  Should sys.pycroot be
>   initialized from PYCROOT and then PYCROOT ignored (that is, what if
>   they differ)?
>

I think this is YAGNI (hey, I am using the abbreviations I learned from
the list; I'm learning, I'm learning!  =).  Why the heck would your needs
for compiling bytecode files change while running a program?  Just turn
off the compiling if you think you are not going to need it.

> - Should there be a command-line flag for the interpreter instead of
>   or in addition to an environment variable?  This seems like it would
>   be less flexible.  During Python installation, the user frequently
>   doesn't have ready access to the interpreter command line.  Using an
>   environment variable makes it easier to control behavior.
>

I think it should be one or the other but not both.  If the features of
having PYCROOT is worth it than there is no need to deal with a
command-line option *unless* having a different setting for different
interpreter invocations under the same user is considered more important.
In that case go with the command-line and ditch the environment variable.

> - Should PYCROOT be interpreted differently during installation than
>   at runtime?  I have no idea.  (Maybe it's just a stupid thought, but
>   the thought occurred to me, so I thought I'd mention it.)
>

No.  Should be the same.

<snip>

-Brett