<div><span style="color: rgb(160, 160, 168); ">On Thursday, February 28, 2013 at 5:55 AM, M.-A. Lemburg wrote:</span></div>
<blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;">
<span><div><div><div>I think we all agree that scanning arbitrary HTML pages</div><div>for download links is not a good idea and we need to</div><div>transition away from this towards a more reliable system.</div><div><br></div><div>Here's an approach that would work to start the transition</div><div>while not breaking old tools (sketching here to describe the</div><div>basic idea):</div><div><br></div><div>Limiting scans to download_url</div><div>------------------------------</div><div><br></div><div>Installers and similar tools preferably no longer scan the all</div><div>links on the /simple/ index, but instead only look at</div><div>the download links (which can be defined in the package</div><div>meta data) for packages that don't host files on PyPI.</div><div><br></div><div>Going only one level deep</div><div>-------------------------</div><div><br></div><div>If the download links point to a meta-file named</div><div>"<packagename>-<version>-downloads.html#<sha256-hashvalue>",</div><div>the installers download that file, check whether the</div><div>hash value matches and if it does, scan the file in</div><div>the same way they would parse the /simple/ index page of</div><div>the package - think of the downloads.html file as a symlink</div><div>to extend the search to an external location, but in a</div><div>predefined and safe way.</div><div><br></div><div>Comments</div><div>--------</div><div><br></div><div>* The creation of the downloads.html file is left to the</div><div> package owner (we could have a tool to easily create it).</div><div><br></div><div>* Since the file would use the same format as the PyPI</div><div> /simple/ index directory listing, installers would be</div><div> able to verify the embedded hash values (and later</div><div> GPG signatures) just as they do for files hosted directly</div><div> on PyPI.</div><div><br></div><div>* The URL of the downloads.html file, together with the</div><div> hash fragment, would be placed into the setup.py</div><div> download_url variable. This is supported by all recent</div><div> and not so recent Python versions.</div><div><br></div><div>* No changes to older Python versions of distutils are</div><div> necessary to make this work, since the download_url</div><div> field is a free form field.</div><div><br></div><div>* No changes to existing distutils meta data formats are</div><div> necessary, since the download_url field has always</div><div> been meant for download URLs.</div><div><br></div><div>* Installers would not need to learn about a new meta</div><div> data format, because they already know how to parse</div><div> PyPI style index listings.</div><div><br></div><div>* Installers would prefer the above approach for downloads,</div><div> and warn users if they have to revert back to the old</div><div> method of scanning all links.</div><div><br></div><div>* Installers could impose extra security requirements,</div><div> such as only following HTTPS links and verifying</div><div> all certificates.</div><div><br></div><div>* In a later phase of the transition we could have</div><div> PyPI cache the referenced distribution files locally</div><div> to improve reliability. This would turn the push</div><div> strategy for uploading files to PyPI into a pull</div><div> strategy for those packages and make things a lot</div><div> easier to handle for package maintainers.</div><div><br></div></div></div></span></blockquote><div>I don't have time to respond to the rest right now, but this isn't doable</div><div>I don't think. The purpose of that legalese you pointed out is to make</div><div>it possible for PyPI to serve those files legally. We don't know if those</div><div>files are something PyPI is allowed to distribute so PyPI can't cache them.</div><blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;"><span><div><div><div></div><div>What do you think ?</div><div><br></div><div>-- </div><div>Marc-Andre Lemburg</div><div><a href="http://eGenix.com">eGenix.com</a></div><div><br></div><div>Professional Python Services directly from the Source (#1, Feb 28 2013)</div><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><div><div>Python Projects, Consulting and Support ... <a href="http://www.egenix.com">http://www.egenix.com</a>/</div><div>mxODBC.Zope/Plone.Database.Adapter ... <a href="http://zope.egenix.com">http://zope.egenix.com</a>/</div><div>mxODBC, mxDateTime, mxTextTools ... <a href="http://python.egenix.com">http://python.egenix.com</a>/</div></div></blockquote></blockquote></blockquote><div>________________________________________________________________________</div><div><br></div><div>::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::</div><div><br></div><div> <a href="http://eGenix.com">eGenix.com</a> Software, Skills and Services GmbH Pastor-Loeh-Str.48</div><div> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg</div><div> Registered at Amtsgericht Duesseldorf: HRB 46611</div><div> <a href="http://www.egenix.com/company/contact/">http://www.egenix.com/company/contact/</a></div><div>_______________________________________________</div><div>Catalog-SIG mailing list</div><div><a href="mailto:Catalog-SIG@python.org">Catalog-SIG@python.org</a></div><div><a href="http://mail.python.org/mailman/listinfo/catalog-sig">http://mail.python.org/mailman/listinfo/catalog-sig</a></div></div></div></span>
</blockquote>
<div>
<br>
</div>