[Python-Dev] PEP 3147: PYC Repository Directories

[Guido van Rossum]

On Sat, Feb 6, 2010 at 3:28 PM, Barry Warsaw <barry at python.org> wrote:
> On Feb 01, 2010, at 02:04 PM, Paul Du Bois wrote:
>
>>It's an interesting challenge to write the file in such a way that
>>it's safe for a reader and writer to co-exist. Like Brett, I
>>considered an append-only scheme, but one needs to handle the case
>>where the bytecode for a particular magic number changes. At some
>>point you'd need to sweep garbage from the file. All solutions seem
>>unnecessarily complex, and unnecessary since in practice the case
>>should not come up.
>
> I don't think that part's difficult.  The byte code's only going to change if
> the source file has changed, and in that case, /all/ the byte code in the "fat
> pyc" file will be invalidated, so the whole thing can be deleted by the first
> writer.  I'd worked that out in the original fat pyc version of the PEP.

I'm sorry, but I'm totally against fat bytecode files. They make
things harder for all tools. The beauty of the existing bytecode
format is that it's totally trivial: magic number, source mtime,
unmarshalled code object. You can't beat the beauty of that.

For the traditional "skinny" bytecode files, I believe that the
existing algorithm which writes zeros in the place of the magic number
first, writes the rest of the file, and then goes back to write the
correct magic number, is correct with a single writer and multiple
readers (assuming the readers ignore the file if its magic number is
invalid). The creat(O_EXCL) option ensures that there won't be
multiple writers. No rename() is necessary; POSIX rename() may be
atomic, but it's a directory modification which makes it potentially
slow.

-- 
--Guido van Rossum (python.org/~guido)