[Python-Dev] Re: Proto-PEP regarding writing bytecode files

[Skip Montanaro]

    Paul> The patch as posted suited me because I could just set the C flag
    Paul> or argv entry from my own main code before I initialized Python;
    Paul> or I could set the command line flag when executing with a shell
    Paul> script or as an alias.

I see no particular reason a command-line flag can't be provided, but I
think for a small extra effort significantly more control can be provided
than to simply do or don't generate bytecode files.  A flag works for you,
but for other people (e.g., running an application in which all the .py
files live in a read-only zip file), it makes some sense to provde them with
a way to cache the byte compilation step.

    Paul> I'm unclear about how this works if I have one environment
    Paul> variable but am running different installations of
    Paul> Python.

The PYCROOT (now PYTHONBYTECODEBASE) environment variable refers to an
alternate root directory for writing bytecode files.  Have a look at the
examples in the latest version of PEP 304 and see if they don't answer your
questions.  Briefly, if PYTHONBYTECODEBASE is set to /tmp and urllib is
found in /usr/lib/python2.3/urllib.py, given the right circumstances the
bytecode would be written to /tmp/usr/lib/python2.3/urllib.pyc, not
/tmp/urllib.pyc.

    Paul> Bottom line: I find any messing with my path to be suicidal.

Which is why I won't mess with your path. ;-) Again, consider that you have
/usr/lib/python2.3 in sys.path and PYTHONBYTECODEBASE is set to /tmp.  When
the importer looks for the urllib module, it will first look for urllib.pyc
in /usr/lib/python2.3 (because that's where urllib.py exists).  If it's not
found there (or if what's there is out-of-date), a second check will be made
in /tmp/usr/lib/python2.3.  If a bytecode file needs to be written, it will
be written to /tmp/usr/lib/python2.3/urllib.pyc.  This should solve your
problem if you set PYTHONBYTECODEBASE to the empty string (meaning don't
write bytecode files at all) or if each of your 384 parallel processes sets
it to a unique directory, perhaps something like:

    #!/bin/sh
    ...
    export PYTHONBYTECODEBASE=/tmp/`domainname --fqdn`/$$
    mkdir -p $PYTHONBYTECODEBASE
    python ...
    rm -rf $PYTHONBYTECODEBASE

The post-run cleanup won't be necessary if you can avoid making the
directory pid-specific.

Skip