[Distutils] [tuf] Re: Automation for creating, updating and destroying a TUF-secured PyPI mirror

[Trishank Karthik Kuppusamy]

On 04/09/2013 11:52 PM, Trishank Karthik Kuppusamy wrote:
> I have finished generating the /simple metadata and they are about 52MB
> --- not too far off from my estimate of 59MB. Remember: this is the
> worst-case size for simple metadata.

Okay, so we have finished generating the TUF metadata for a complete (if
not the latest) set of PyPI packages.

Summary of the largest metadata, assuming the worst case of a key per
package on PyPI:

release.txt: 11MB
/simple metadata: 52MB
/packages metadata: 96MB

All in all, the metadata sums to about 159MB. With the data being 45GB,
that works out to the metadata size being 0.35% of the data size.

Remember: this is the worst case for the metadata, where every PyPI
package has its own key, and there is a role for every possible target
subdirectory. The metadata is also uncompressed JSON.

As we have said before, we think we can do better (e.g, by reusing keys
for multiple packages), and we are working on it.

Simultaneously, we are testing a TUF-enabled version of pip against a
TUF-secured PyPI mirror.