<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Aug 14, 2016 at 9:35 PM, Xavier Combelle <span dir="ltr"><<a href="mailto:xavier.combelle@gmail.com" target="_blank">xavier.combelle@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On 15/08/2016 02:45, Wes Turner wrote:<br>
><br>
> You can add a `make clean` build step:<br>
><br>
>   pyclean:<br>
>       find . -name '*.pyc' -delete<br>
><br>
> You can delete all .pyc files<br>
><br>
> - $ find . -name '*.pyc' -delete<br>
> - <a href="http://manpages.ubuntu.com/manpages/precise/man1/pyclean.1.html" rel="noreferrer" target="_blank">http://manpages.ubuntu.com/<wbr>manpages/precise/man1/pyclean.<wbr>1.html</a><br>
> #.pyc, .pyo<br>
><br>
> You can rebuild all .pyc files (for a given directory):<br>
><br>
> - $ python -m compileall -h<br>
> - <a href="https://docs.python.org/2/library/compileall.html" rel="noreferrer" target="_blank">https://docs.python.org/2/<wbr>library/compileall.html</a><br>
> - <a href="https://docs.python.org/3/library/compileall.html" rel="noreferrer" target="_blank">https://docs.python.org/3/<wbr>library/compileall.html</a><br>
><br>
><br>
><br>
> You can, instead of building .pyc, build .pyo<br>
><br>
> - <a href="https://docs.python.org/2/using/cmdline.html#envvar-PYTHONOPTIMIZE" rel="noreferrer" target="_blank">https://docs.python.org/2/<wbr>using/cmdline.html#envvar-<wbr>PYTHONOPTIMIZE</a><br>
> - <a href="https://docs.python.org/2/using/cmdline.html#cmdoption-O" rel="noreferrer" target="_blank">https://docs.python.org/2/<wbr>using/cmdline.html#cmdoption-O</a><br>
><br>
> You can not write .pyc or .pyo w/ PYTHONDONTWRITEBYTECODE / -B<br>
><br>
> - <a href="https://docs.python.org/2/using/cmdline.html#envvar-PYTHONDONTWRITEBYTECODE" rel="noreferrer" target="_blank">https://docs.python.org/2/<wbr>using/cmdline.html#envvar-<wbr>PYTHONDONTWRITEBYTECODE</a><br>
> - <a href="https://docs.python.org/2/using/cmdline.html#cmdoption-B" rel="noreferrer" target="_blank">https://docs.python.org/2/<wbr>using/cmdline.html#cmdoption-B</a><br>
> - If the files exist though,<br>
>   - <a href="https://docs.python.org/3/reference/import.html" rel="noreferrer" target="_blank">https://docs.python.org/3/<wbr>reference/import.html</a><br>
><br>
> You can build a PEX (which rebuilds .pyc files) and test/deploy that:<br>
><br>
> - <a href="https://github.com/pantsbuild/pex#integrating-pex-into-your-workflow" rel="noreferrer" target="_blank">https://github.com/pantsbuild/<wbr>pex#integrating-pex-into-your-<wbr>workflow</a><br>
> - <a href="https://pantsbuild.github.io/python-readme.html#more-about-python-tests" rel="noreferrer" target="_blank">https://pantsbuild.github.io/<wbr>python-readme.html#more-about-<wbr>python-tests</a><br>
><br>
> How .pyc files currently work:<br>
><br>
> - <a href="http://nedbatchelder.com/blog/200804/the_structure_of_pyc_files.html" rel="noreferrer" target="_blank">http://nedbatchelder.com/blog/<wbr>200804/the_structure_of_pyc_<wbr>files.html</a><br>
> - <a href="https://www.python.org/dev/peps/pep-3147/#flow-chart" rel="noreferrer" target="_blank">https://www.python.org/dev/<wbr>peps/pep-3147/#flow-chart</a> (*.pyc -><br>
> ./__pycache__)<br>
> - <a href="http://raulcd.com/how-python-caches-compiled-bytecode.html" rel="noreferrer" target="_blank">http://raulcd.com/how-python-<wbr>caches-compiled-bytecode.html</a><br>
><br>
> You could add a hash of the .py source file in the header of the<br>
> .pyc/.pyo object (as proposed)<br>
><br>
> - The overhead of this hashing would be a significant performance<br>
> regression<br>
> - Instead, today, the build step can just pyclean or build a<br>
> .zip/.WHL/.PEX which is expected to be a fresh build<br>
><br>
</div></div>The problem is not the option of you have to prevent the problem, the<br>
simplest way being<br>
to delete the .pyc file, It is easy to do once you spot it. The problem<br>
is that it randomly happen in<br>
normal workflow.<br></blockquote><div><br></div><div>IIUC, the timestamp in the .pyc header is designed to prevent this ocurrence?</div><div> </div><div>Reasons that the modification timestamp comparison could be off:</div><div><br></div><div>- Time change</div><div>  - Daylight savings time</div><div>  - NTP drift adjustment? </div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
To have an idea of the overhead of the whole hashing procedure I run the<br>
following script<br>
<br>
import sys<br>
<br>
from time import time<br>
from zlib import adler32 as h<br>
t2 =time()<br>
import decimal<br>
print(decimal.__file__)<br>
c1 = time()-t2<br>
t1=time()<br>
r=h(open(decimal.__file__,'rb'<wbr>).read())<br>
c2= time()-t1<br>
print(c2,c1,c2/c1)<br>
<br>
decimal was chosen because it was the biggest file of the standard library.<br>
on 20 runs, the overhead was always between 1% and 1.5%<br>
So yes the overhead on the import process is measurable but very small.<br>
By consequence, I would not call it significant.<br>
Moreover the import process is only a part (and not the biggest one) of<br>
a whole.<br></blockquote><div><br></div><div>I agree that 1 to 1.5% is not significant.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
At the difference of my first mail I now consider only a non<br>
cryptographic hash/checksum<br>
as the only aim is to prevent accidental unmatch between .pyc and .py file.<br>
<div class="HOEnZb"><div class="h5"><br>
______________________________<wbr>_________________<br>
Python-ideas mailing list<br>
<a href="mailto:Python-ideas@python.org">Python-ideas@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-ideas" rel="noreferrer" target="_blank">https://mail.python.org/<wbr>mailman/listinfo/python-ideas</a><br>
Code of Conduct: <a href="http://python.org/psf/codeofconduct/" rel="noreferrer" target="_blank">http://python.org/psf/<wbr>codeofconduct/</a><br>
</div></div></blockquote></div><br></div></div>