Hashes are also available via PyPI's JSON API <https://warehouse.readthedocs.io/api-reference/json/>. On Tue, Feb 12, 2019 at 7:15 AM Tzu-ping Chung <uranusjr@gmail.com> wrote:

I believe you’re looking for the PEP 503 simple API <https://www.python.org/dev/peps/pep-0503/>. This is what pip uses to find the hashes (among other things) as well. The hash value is included as a fragment in the URL.

TP

On 12/2/2019, at 23:03, Eric Peterson <epeterson@interactivebrokers.com> wrote:

Hi all,

When in the "Download files" section of a project on PyPI, next to each download there is a convenient "SHA256" link that will copy the SHA-256 fingerprint for that file to the clipboard. I am wondering if there is a programmatic way to access the SHA-256 for a file (besides just scraping the web page)? Ideally there would be some way to construct a URL based on the name of the file that, when called, would return the fingerprint.

Thanks, Eric -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/FLNOE...

-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/EY2SR...

On Tue, 12 Feb 2019 at 16:28, Eric Peterson <epeterson@interactivebrokers.com> wrote:

Brilliant, that's exactly what I was looking for—both the simple API and json API look very useful. Thanks!

Just a quick note, the simple API is required for every index server to support, whereas the JSON API is not (yet?) standardised and may not be supported anywhere other than PyPI (I don't know about devpi, for example). This may not matter for your use case, but is useful to know more generally. Paul

Most likely (someone more familiar with Warehouse could answer this) Warehouse will select sha256 whenever it is available, so the simple API may be just as good for you. But it's something to consider.

The simple API will always have the sha256 digest, for every file. <https://github.com/pypa/warehouse/blob/04667bc20d5a8ab25062e64c5bd51eaf150ba532/warehouse/templates/legacy/api/simple/detail.html#L22> On Tue, Feb 12, 2019 at 10:04 AM Alex Becker <alcubecker@gmail.com> wrote:

Also note that the simple API only includes a single hash for each file, and may use md5 hashes instead of sha256 (technically it may use any of the hash algorithms guaranteed by hashlib, but I've only seen those two). The JSON API will give you *all* the hashes warehouse has for the file, which may be more useful.

Most likely (someone more familiar with Warehouse could answer this) Warehouse will select sha256 whenever it is available, so the simple API may be just as good for you. But it's something to consider.

Best,

Alex Becker

On Tue, Feb 12, 2019 at 9:58 AM Paul Moore <p.f.moore@gmail.com> wrote:

...

On Tue, 12 Feb 2019 at 16:28, Eric Peterson <epeterson@interactivebrokers.com> wrote:

...

Brilliant, that's exactly what I was looking for—both the simple API

and json API look very useful. Thanks!

Just a quick note, the simple API is required for every index server to support, whereas the JSON API is not (yet?) standardised and may not be supported anywhere other than PyPI (I don't know about devpi, for example). This may not matter for your use case, but is useful to know more generally.

Paul -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/ZOU33...

On Tuesday, February 12, 2019, Alex Becker <alcubecker@gmail.com> wrote:

Also note that the simple API only includes a single hash for each file, and may use md5 hashes instead of sha256 (technically it may use any of the hash algorithms guaranteed by hashlib, but I've only seen those two). The JSON API will give you *all* the hashes warehouse has for the file, which may be more useful.

MD5 is no longer suitable for verifying package integrity. https://en.wikipedia.org/wiki/MD5#Security

The security of the MD5 hash function is severely compromised. A

collision attack exists that can find collisions within seconds on a computer with a 2.6 GHz Pentium 4 processor (complexity of 224.1).[18] Further, there is also a chosen-prefix collision attack that can produce a collision for two inputs with specified prefixes within hours, using off-the-shelf computing hardware (complexity 239).[19]

Most likely (someone more familiar with Warehouse could answer this) Warehouse will select sha256 whenever it is available, so the simple API may be just as good for you. But it's something to consider.

https://github.com/pypa/warehouse/blob/master/warehouse/legacy/api/simple.py https://github.com/pypa/warehouse/blob/master/tests/unit/legacy/api/test_sim... https://github.com/pypa/warehouse/blob/master/warehouse/packaging/models.py File has a .md5_digest, .sha256_digest, and .blake2_256_digest https://github.com/pypa/warehouse/search?q=md5_digest doesn't show selection of a hash with precedence; so IDK where that functionality is?

Best,

Alex Becker

On Tue, Feb 12, 2019 at 9:58 AM Paul Moore <p.f.moore@gmail.com> wrote:

...

On Tue, 12 Feb 2019 at 16:28, Eric Peterson <epeterson@interactivebrokers.com> wrote:

...

Brilliant, that's exactly what I was looking for—both the simple API

and json API look very useful. Thanks!

Just a quick note, the simple API is required for every index server to support, whereas the JSON API is not (yet?) standardised and may not be supported anywhere other than PyPI (I don't know about devpi, for example). This may not matter for your use case, but is useful to know more generally.

Paul -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@ python.org/message/ZOU33JCVN32DWHRU5MJYGOV52BE5JIR3/

On 2019-02-12 13:37:20 -0500 (-0500), Wes Turner wrote:

MD5 is no longer suitable for verifying package integrity.

https://en.wikipedia.org/wiki/MD5#Security

...

The security of the MD5 hash function is severely compromised. A collision attack exists [...] there is also a chosen-prefix collision attack [...]

The difference between collision (or chosen-prefix collision) and preimage (or second preimage) attacks is still very relevant. With MD5 you can't trust that someone who provided you with an input and a hash of that input hasn't carefully crafted that input so that there is also a second input which results in the same hash. Or in package terms, you can't trust that the package you've received wasn't part of a contrived scheme on the part of someone you've already decided to trust. You can still rest assured (for now anyway) that the package you receive is the same one the person or system providing the MD5 checksum intended for you to receive. But because trying to explain this nuance to people is considerably harder than just saying "MD5 bad" it's simply not worth trying to have the discussion most of the time, and so easier instead to replace it with a more modern alternative and move on with your life. -- Jeremy Stanley

On Tuesday, February 12, 2019, Wes Turner <wes.turner@gmail.com> wrote:

On Tuesday, February 12, 2019, Jeremy Stanley <fungi@yuggoth.org> wrote:

...

On 2019-02-12 13:37:20 -0500 (-0500), Wes Turner wrote:

...

MD5 is no longer suitable for verifying package integrity.

https://en.wikipedia.org/wiki/MD5#Security

...

The security of the MD5 hash function is severely compromised. A collision attack exists [...] there is also a chosen-prefix collision attack [...]

The difference between collision (or chosen-prefix collision) and preimage (or second preimage) attacks is still very relevant. With MD5 you can't trust that someone who provided you with an input and a hash of that input hasn't carefully crafted that input so that there is also a second input which results in the same hash. Or in package terms, you can't trust that the package you've received wasn't part of a contrived scheme on the part of someone you've already decided to trust. You can still rest assured (for now anyway) that the package you receive is the same one the person or system providing the MD5 checksum intended for you to receive.

It is possible to find a nonce value that causes an arbitrary package to have the same MD5 hash as the actual package.

e.g. browsers MUST NOT rely upon MD5 for x.509 certificate SSL/TLS/HTTPS fingerprints for exactly this reason.

...

But because trying to explain this nuance to people is considerably harder than just saying "MD5 bad" it's simply not worth trying to have the discussion most of the time, and so easier instead to replace it with a more modern alternative and move on with your life. -- Jeremy Stanley

On Tuesday, February 12, 2019, Jeremy Stanley <fungi@yuggoth.org> wrote:

On 2019-02-12 17:02:25 -0500 (-0500), Wes Turner wrote:

...

On Tuesday, February 12, 2019, Wes Turner <wes.turner@gmail.com> wrote: [...]

...

It is possible to find a nonce value that causes an arbitrary package to have the same MD5 hash as the actual package.

e.g. browsers MUST NOT rely upon MD5 for x.509 certificate SSL/TLS/HTTPS fingerprints for exactly this reason. [...]

I fear we're verging far into armchair crypto here, but you're either making buzzword soup or have a severely flawed understanding of the algorithms involved. There is no nonce in an IETF RFC 1321 (colloquially "MD5 checksum") implementation, so please at least attempt to frame your assertions using terms found in the canonical literature.

Creating a malicious package which computes to the same MD5 checksum as an existing package of your choice would require that the second preimage resistance of the MD5 algorithm is broken, or that you got (time complexity 2^128) "lucky." Uses of MD5 elsewhere which mix in attacker-controlled inputs to generate the reference output are another story entirely, but as with the any of the information security field the actual risk depends on your threat model.

I'm not about to recommend MD5 to anyone these days, don't get me wrong. There are (at least marginally, again depending on your threat model) better alternatives which require no additional effort if you're designing a system from scratch. But let's not mischaracterize the qualities of any algorithm, as it makes it difficult for someone who does understand the differences to take us seriously.

All it has to be is an archive containing a setup.py. "MD5 considered harmful today: Creating a rogue CA certificate" (2008) https://www.win.tue.nl/hashclash/rogue-ca/

-- Jeremy Stanley

On Tuesday, February 12, 2019, Jeremy Stanley <fungi@yuggoth.org> wrote:

On 2019-02-12 18:42:29 -0500 (-0500), Wes Turner wrote: [...]

...

All it has to be is an archive containing a setup.py.

"MD5 considered harmful today: Creating a rogue CA certificate" (2008) https://www.win.tue.nl/hashclash/rogue-ca/

You keep trotting out PKI examples as if they have anything whatsoever to do with checksumming, but I'm quickly getting the distinct impression you don't actually know the difference so I'll stop now as we've gone well off-topic for this list. -- Jeremy Stanley

you hash the file. the hash is compared against a list. if the hash matches, it's considered valid. In 2008, they were able to generate a file that has the same MD5 hash as one in a list of considered-good hashes, which is also a valid x.509 cert. How is that at all different from generating an archive with a setup.py that has the same hash as something listed on PyPI? Trotting ... "Westminster Dad Show" https://youtu.be/2S2gQjTURvU ... Now you've suggested that I'm FUD'ing: is there a difference between finding an x.509 cert hash and a .tgz/.zip with a setup.py or setup.pyc hash? Maybe there's something fundamental that I've misunderstood? (So sorry to interrupt)

TUF should be handled via a grant from Facebook this year once Ernest and I get this underway: https://pyfound.blogspot.com/2018/12/upcoming-pypi-improvements-for-2019.htm... <https://pyfound.blogspot.com/2018/12/upcoming-pypi-improvements-for-2019.html> We will take all the help we can get, but we'll have Project management and some funds! Cooper

On Feb 12, 2019, at 9:42 AM, Wes Turner <wes.turner@gmail.com> wrote:

... The Update Framework (TUF) is in part derived from Thandy (the tor updater). There's an automotive derivative of TUF called Uptane. https://theupdateframework.github.io/ <https://theupdateframework.github.io/>

"Roadmap update for TUF support" https://github.com/pypa/warehouse/issues/5247 <https://github.com/pypa/warehouse/issues/5247>

"TUF deployment roadmap for PyPI" https://github.com/theupdateframework/tuf/issues/816# <https://github.com/theupdateframework/tuf/issues/816#>

SHA-256 is not sufficient. GPG was removed because insufficient. Does TUF need funding, person-hours, new code, or code-review?

On Tuesday, February 12, 2019, Trishank Kuppusamy < trishank.kuppusamy@datadoghq.com> wrote:

On Tue, Feb 12, 2019 at 5:32 PM Cooper Ry Lees <me@cooperlees.com> wrote:

...

TUF should be handled via a grant from Facebook this year once Ernest and I get this underway: https://pyfound.blogspot.com/2018/12/upcoming-pypi- improvements-for-2019.html

We will take all the help we can get, but we'll have Project management and some funds!

Happy to hear, and happy to pitch in when the time comes.

Same. Good to hear