[Distutils] [Catalog-sig] distribute D.C. sprint tasks

"Martin v. Löwis" martin at v.loewis.de
Tue Oct 14 00:16:56 CEST 2008


> Maybe we could use one subfolder per alphabet letter,

Would that simplify anything?

PyPI uses one directory per letter to reduce the number of files in a
single directory, in case ext3 doesn't deal with large directories well.
For the stats, the "large directories" argument wouldn't count.

OTOH, if you do have separate pages per letter, the master server would
still need to download all individual files. Having them split into
chunks just increases the load, rather than reducing it.


> You would need to specify a timestamp for each single download though,
> to make sure PyPI
> knows which hits to count, depending on the last date it checked the
> mirror.

No. It would just compute the grand total from scratch each time.


Regards,
Martin


More information about the Distutils-SIG mailing list