<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
On 4/17/12 12:54 PM, Donald Stufft wrote:
<blockquote cite="mid:6375BB58D5494D61BB0F2599C75B9F93@gmail.com"
type="cite"><br>
<blockquote type="cite"
style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;"><span>
<div>
<div>
<div><br>
</div>
<div>If there's interest I can write a multiprocess-based
script that keeps a </div>
<div>md5 database up-to-date</div>
</div>
</div>
</span></blockquote>
<div>I'd be interested ;) Although i'd prefer sha256 personally. </div>
<blockquote type="cite" style="border-left-style: solid;
border-width: 1px; margin-left: 0px; padding-left: 10px;"><span>
<div>
<div>
<div><br>
</div>
</div>
</div>
</span></blockquote>
</blockquote>
well, this is not really for security and I don't think a collision
can happen that often with md5 :D<br>
<br>
here's a raw script: <a class="moz-txt-link-freetext" href="http://tarek.pastebin.mozilla.org/1575563">http://tarek.pastebin.mozilla.org/1575563</a><br>
<br>
The grand digest is done like a derived secret: I loop on the hash
and do <br>
<br>
grand hash = hash(n & n+1) for n in hashes<br>
<br>
I've run it against the 111,196 files I currently have in my mirror<br>
<br>
- First *full* run from scratch - 15m32s (not sure why I don't have
better here, maybe Python's md5 is slower than md5deep)<br>
<br>
- Second *full* run, md5 database filled - 2m33s - it scans the
mirror, and adds missing md5s + build the grand digest.<br>
<br>
- Just the digest, against a synced MD5 DB - 1m1s (I just commented
the first part that builds/updates the md5 db)<br>
<br>
In a real mirror, once the first full run is done, the md5 db would
be updated continuously everytime a new file is added<br>
in the mirror, so the only extra load is recalculating the digest
again.<br>
<br>
So it would take around a minute each time, not a few seconds as I
said previously. But that seems ok if a mirror is updated<br>
for example every 5 minutes, 4 minutes can be spent to sync the
files, and 1 minute to do the checksum I guess<br>
<br>
<br>
<br>
<br>
<blockquote cite="mid:6375BB58D5494D61BB0F2599C75B9F93@gmail.com"
type="cite">
<blockquote type="cite"
style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;"><span>
<div>
<div>
<div>Cheers</div>
<div>Tarek</div>
<div><br>
</div>
<blockquote type="cite">
<div>
<div><br>
</div>
<div>Regards,</div>
<div>Martin</div>
</div>
</blockquote>
<div><br>
</div>
<div>_______________________________________________</div>
<div>Catalog-SIG mailing list</div>
<div><a moz-do-not-send="true"
href="mailto:Catalog-SIG@python.org">Catalog-SIG@python.org</a></div>
<div><a moz-do-not-send="true"
href="http://mail.python.org/mailman/listinfo/catalog-sig">http://mail.python.org/mailman/listinfo/catalog-sig</a></div>
</div>
</div>
</span> </blockquote>
<div> <br>
</div>
</blockquote>
<br>
</body>
</html>