<div><span style="color: rgb(160, 160, 168); ">On Tuesday, April 17, 2012 at 6:50 AM, Tarek Ziadé wrote:</span></div>
<blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;">
<span><div><div><div>On 4/17/12 11:57 AM, <a href="mailto:martin@v.loewis.de">martin@v.loewis.de</a> wrote:</div><blockquote type="cite"><div><blockquote type="cite"><div>by calculating the grand hash of each file hash.</div></blockquote><div><br></div><div>In this case, the checksum would not be a reliable indication that the</div><div>files are actually up-to-date. For example, a mirror may keep updating</div><div>files into the wrong location (not the location that is then used to</div><div>serve the files), so that the files being served are from a stale copy.</div><div>This is not theoretical - it actually happened in my mirror setup at one</div><div>time.</div></div></blockquote><div>So you were updating a directory but serving another directory ?</div><div><br></div><div>But then updating the right last-modified page people were seeing ?</div><div><br></div><div>In that case, updating the checksum would have revealed you were on the </div><div>wrong set of files.</div><div><br></div><div>Unless you script was updating everything on a stale copy that was not </div><div>published ?</div><div><br></div><div><br></div><blockquote type="cite"><div><blockquote type="cite"><div><blockquote type="cite"><div>That could take a few hours per change.</div></blockquote><div>why that ? you don't calculate the checksum of a file your already </div><div>have twice.</div><div><br></div><div>Even if you do, it's very fast to call md5.</div><div><br></div><div>try it:</div><div><br></div><div>$ find mirror | xargs md5</div><div><br></div><div>this takes a few seconds at most on the whole mirror</div></div></blockquote><div><br></div><div>I tried it, and on my mirror, it took 27 minutes and 7 seconds.</div><div>So not exactly hours, but not "a few seconds" either.</div></div></blockquote><div>oops sorry I ran it on the wrong directory, it's true that it takes more </div><div>time !</div><div><br></div><div>So on my centos 5 VM - which is quite slow and doing many other stuff </div><div>like running Jenkins jobs, running the "md5deep" program like this : </div><div><a href="http://tarek.pastebin.mozilla.org/1574557">http://tarek.pastebin.mozilla.org/1574557</a></div><div><br></div><div>It took 15minutes and 1 second. It can be optimized of course, since </div><div>most directories are done quickly and everything is in /source. That </div><div>time can be divided by 2 at least with the proper load balancing between </div><div>a few md5 runners.</div><div><br></div><div>But that just to be run *once*. You would not compute it on every mirror </div><div>update but keep all md5 values somewhere.</div><div><br></div><div>So, recalculating the grand hash on every mirror update should takes a </div><div>few seconds because it would just consist of calculating the hash for </div><div>the new files, then</div><div>calculating the grand hash -- a loop that updates a md5 hash with 20k </div><div>hashes takes less than a second if I don't count the file reading.</div><div><br></div><div>(see <a href="http://tarek.pastebin.mozilla.org/1574574">http://tarek.pastebin.mozilla.org/1574574</a>)</div><div><br></div><div>I am not sure why we're having this discussion since it's implementation </div><div>details, but it's fun :)</div><div><br></div><div>If there's interest I can write a multiprocess-based script that keeps a </div><div>md5 database up-to-date</div></div></div></span></blockquote><div>I'd be interested ;) Although i'd prefer sha256 personally. </div><blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;"><span><div><div><div><br></div><div>Cheers</div><div>Tarek</div><div><br></div><blockquote type="cite"><div><div><br></div><div>Regards,</div><div>Martin</div></div></blockquote><div><br></div><div>_______________________________________________</div><div>Catalog-SIG mailing list</div><div><a href="mailto:Catalog-SIG@python.org">Catalog-SIG@python.org</a></div><div><a href="http://mail.python.org/mailman/listinfo/catalog-sig">http://mail.python.org/mailman/listinfo/catalog-sig</a></div></div></div></span>
</blockquote>
<div>
<br>
</div>