[Python-ideas] disabling .pyc and .pyo files

Brett Cannon brett at python.org
Wed Dec 16 20:20:42 CET 2009


On Tue, Dec 15, 2009 at 14:37, Kevin Watters <kevin.watters at gmail.com>wrote:

> For what it's worth, I've got an entirely different use case than the ones
> I've seen in this thread so far.
>
> I'd like Python to read .pyo files, but not search for .py or .pyc files.
> This is because we ship a py2exe app in it's "exploded" form, where there is
> an .exe and a lib/ folder full of .pyos.  Purely as an optimization, it'd be
> nice to not have Python stat for .py and then .pyc for every new import.
>
> I remember glancing at Python/import.c and thinking that this could easily
> be accomplished by allowing the user to customize _PyImport_StandardFiletab
> at runtime--in fact there is already an PyImport_AppendInittab; it's just
> not exposed to Python. With a function like imp.set_inittab, I could get
> what I want with something like
>
>    imp.set_inittab(['.pyo', 'rb', imp.PY_COMPILED])
>
> And then of course to read just .py files, you could do
>
>    imp.set_inittab([".py", "U", PY_SOURCE])
>
>
The problem with this is I could easily see it leading to tons of people
using custom file extensions which seems to just be asking for trouble.
Restricting that ability to only people who recompile the interpreter has
kept that in check.

As for avoiding the extra stat calls, your best bet is to either compile
your own version of CPython or use a custom importer (I will be giving a
talk on that at PyCon).

-Brett



> - Kevin
>
> Kristján Valur Jónsson wrote:
>
>> Hello there.
>>
>> We have a large project involving multiple perforce branches of hundreds
>> of .py files each.
>>
>> Although we employ our own import mechanism for the bulk of these files,
>> we do use the regular import mechanism for an essential core of them.
>>
>>
>> Repeatedly we run into trouble because of stray .pyo (and/or .pyc) files.
>>  This can happen for a variety of reasons, but most often it occurs when .py
>> files are being removed, or moved in the hierarchy.  The problem is that the
>> application will happily load and import an orphaned .pyo file, even though
>> the .py file has gone or moved.
>>
>>
>> I looked at the import code and I found that it is trivial to block the
>> reading and writing of .pyo files.  I am about to implement that patch for
>> our purposes, thus forcing recompilation of the .py files on each run if so
>> specified.   This will ensure that the application will execute only the
>> code represented by the checked-out .py files.  But it occurred to me that
>> this functionality might be of interest to other people than just us.  I can
>> imagine, for example, that buildbots running the python regression testsuite
>> might be running into problems with stray .pyo files from time to time.
>>
>>
>> Do you think that such a command line option would be useful for Python at
>> large?
>>
>>
>> Cheers,
>>
>> Kristján
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20091216/568f6d7d/attachment.html>


More information about the Python-ideas mailing list