[Python-Dev] PEP 3147, __pycache__ directories and umask

[Steven D'Aprano]

On Wed, 24 Mar 2010 12:35:43 am Ben Finney wrote:
> Matthias Klose <doko at ubuntu.com> writes:
> > On 23.03.2010 02:28, Ben Finney wrote:
> > > Perhaps also of note is that the FHS recommends systems use
> > > ‘/var/cache/foo/’ for cached data from applications:
> > >
> > >      /var/cache : Application cache data
> > >
> > >      Purpose
> > >
> > >      /var/cache is intended for cached data from applications.
> > > Such data is locally generated as a result of time-consuming I/O
> > > or calculation. The application must be able to regenerate or
> > > restore the data. Unlike /var/spool, the cached files can be
> > > deleted without data loss. The data must remain valid between
> > > invocations of the application and rebooting the system.
> > >
> > >     
> > > <URL:http://www.debian.org/doc/packaging-manuals/fhs/fhs-2.3.html
> > >#VARCACHEAPPLICATIONCACHEDATA>
> > >
> > > This would suggest that Python could start using
> > > ‘/var/cache/python/’ for its cached bytecode tree on systems that
> > > implement the FHS.
> >
> > it reads *data*, not code.
>
> So what? There's no implication that data-which-happens-to-be-code is
> unsuitable for storage in ‘/var/cache/foo/’. Easily-regenerated
> Python byte code for caching meets the description quite well,
> AFAICT.


While I strongly approve of the concept of a central cache directory for 
many things, I don't think that .pyc files fit the bill.

Since there is no privileged python user that runs all Python code, and 
since any unprivileged user needs to be able to write .pyc files, the 
cache directory needs to be world-writable. But since the data being 
stored is code, that opens up a fairly nasty security hole: user fred 
could overwrite the cached .pyc files used by user barney and cause 
barney to run any arbitrary code fred likes.

The alternative would be to have individual caches for every user. Apart 
from being wasteful of disk space ("but who cares, bigger disks are 
cheap") that just complicates everything. You would need:

/var/cache/python/<user>/

which would essentially make it impossible to ship pre-compiled .pyc 
files, since the packaging system couldn't predict what usernames (note 
plural) to store them under.

It is not true that one can necessarily delete the cached files without 
data loss. Python still supports .pyc-only packages, and AFAIK there 
are no plans to stop that, so deleting the .pyc file may delete the 
module.


-- 
Steven D'Aprano