API CHANGE - Migrating from MD5 to SHA2, Take 2
Starting a new thread with more explicit details at Richard’s request. Essentially the tl;dr here is that we'll switch to using sha2 (specifically sha256). Simple API ---------- Drop the #md5= from the PyPI hosted tarballs and replace it with #sha256, the ~60 or so externally hosted files which are using #md5 links will be fetched (one time) verified, and have their #md5= hash replaced with a computed #sha256= hash. Impact: - pip: Will work with no issues, pip has supported sha256 since 1.2, and < 1.2 will install without a hash just fine. - setuptools: Will work with no issues, setuptools has supported sha256 since 0.9 and < 0.9 will install without a hash just fine. - distribute: Doesn't support sha256, will intall without a hash just fine. - buildout: Uses setuptools/distribute to do the downloads I believe. - z3c.pypimirror: Appears to use MD5 hashes, but appears it won't error out if they do not exist. JSON / XMLRPC API ----------------- Keep the md5_sum field, add an additional sha256_sum, suggest that applications switch to using sha256 for verification. Impact: - bandersnatch: bandersnatch will continue to use the md5_sum field from the JSON (and previously XMLRPC) and should be updated to using sha256 in the future. Web UI ------ Simply replace any use of MD5 with SHA256, no clients are expected to access anything here so this should be perfectly fine. Other Clients ------------- - pep381client: Doesn't do anything special with the hash, will continue to work. - devpi: ??? Unsure, I don't follow the code which fetches from PyPI so I can't determine where it gets the md5sum from and what it will do if it doesn't exist. It does have some handling of md5 though. List of clients to look at taken from http://d.stufft.io/image/402r1s442m2r, which is generated by looking at what is downloading the files from PyPI. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
+1 thanks for the detail On 14 November 2014 13:21, Donald Stufft <donald@stufft.io> wrote:
Starting a new thread with more explicit details at Richard’s request. Essentially the tl;dr here is that we'll switch to using sha2 (specifically sha256).
Simple API ----------
Drop the #md5= from the PyPI hosted tarballs and replace it with #sha256, the ~60 or so externally hosted files which are using #md5 links will be fetched (one time) verified, and have their #md5= hash replaced with a computed #sha256= hash.
Impact: - pip: Will work with no issues, pip has supported sha256 since 1.2, and < 1.2 will install without a hash just fine. - setuptools: Will work with no issues, setuptools has supported sha256 since 0.9 and < 0.9 will install without a hash just fine. - distribute: Doesn't support sha256, will intall without a hash just fine. - buildout: Uses setuptools/distribute to do the downloads I believe. - z3c.pypimirror: Appears to use MD5 hashes, but appears it won't error out if they do not exist.
JSON / XMLRPC API -----------------
Keep the md5_sum field, add an additional sha256_sum, suggest that applications switch to using sha256 for verification.
Impact: - bandersnatch: bandersnatch will continue to use the md5_sum field from the JSON (and previously XMLRPC) and should be updated to using sha256 in the future.
Web UI ------
Simply replace any use of MD5 with SHA256, no clients are expected to access anything here so this should be perfectly fine.
Other Clients -------------
- pep381client: Doesn't do anything special with the hash, will continue to work. - devpi: ??? Unsure, I don't follow the code which fetches from PyPI so I can't determine where it gets the md5sum from and what it will do if it doesn't exist. It does have some handling of md5 though.
List of clients to look at taken from http://d.stufft.io/image/402r1s442m2r, which is generated by looking at what is downloading the files from PyPI.
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Hi Donald, thanks for the detail and the pre-announcement! I am all for the change but indeed need to check how devpi code is affected (pretty sure it is) and how to accomodate the change. Will see to do so next week and get back to this thread. best, holger On Thu, Nov 13, 2014 at 21:21 -0500, Donald Stufft wrote:
Starting a new thread with more explicit details at Richard’s request. Essentially the tl;dr here is that we'll switch to using sha2 (specifically sha256).
Simple API ----------
Drop the #md5= from the PyPI hosted tarballs and replace it with #sha256, the ~60 or so externally hosted files which are using #md5 links will be fetched (one time) verified, and have their #md5= hash replaced with a computed #sha256= hash.
Impact: - pip: Will work with no issues, pip has supported sha256 since 1.2, and < 1.2 will install without a hash just fine. - setuptools: Will work with no issues, setuptools has supported sha256 since 0.9 and < 0.9 will install without a hash just fine. - distribute: Doesn't support sha256, will intall without a hash just fine. - buildout: Uses setuptools/distribute to do the downloads I believe. - z3c.pypimirror: Appears to use MD5 hashes, but appears it won't error out if they do not exist.
JSON / XMLRPC API -----------------
Keep the md5_sum field, add an additional sha256_sum, suggest that applications switch to using sha256 for verification.
Impact: - bandersnatch: bandersnatch will continue to use the md5_sum field from the JSON (and previously XMLRPC) and should be updated to using sha256 in the future.
Web UI ------
Simply replace any use of MD5 with SHA256, no clients are expected to access anything here so this should be perfectly fine.
Other Clients -------------
- pep381client: Doesn't do anything special with the hash, will continue to work. - devpi: ??? Unsure, I don't follow the code which fetches from PyPI so I can't determine where it gets the md5sum from and what it will do if it doesn't exist. It does have some handling of md5 though.
List of clients to look at taken from http://d.stufft.io/image/402r1s442m2r, which is generated by looking at what is downloading the files from PyPI.
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Nov 13, 2014, at 9:21 PM, Donald Stufft <donald@stufft.io> wrote:
Starting a new thread with more explicit details at Richard’s request. Essentially the tl;dr here is that we'll switch to using sha2 (specifically sha256).
Ping? Are we OK to make this change? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Hi Donald, On Sat, Nov 29, 2014 at 19:43 -0500, Donald Stufft wrote:
On Nov 13, 2014, at 9:21 PM, Donald Stufft <donald@stufft.io> wrote:
Starting a new thread with more explicit details at Richard’s request. Essentially the tl;dr here is that we'll switch to using sha2 (specifically sha256).
Ping?
Are we OK to make this change?
sorry i didn't get back earlier. Before the minor release of devpi-server last week i tried for two hours to change devpi-server to accomodate your planned pypi.python.org checksum changes. I found the change cannot easily be done without changes to the underlying database schema and thus needs a major new release of devpi-server because an export/import cycle is needed. When doing that i also want to do some internal cleanup related to name normalization (and also relating to recent pypi.python.org changes) but i need a week or two i guess to do that. However i now think that if you do the pypi.python.org checksum change it shouldn't directly break devpi-server but it would remove checksum checking. I'd rather like to have a new major devpi-server release out when you do the change. Is it ok for you to wait a bit still? best, holger
On Dec 1, 2014, at 4:25 AM, holger krekel <holger@merlinux.eu> wrote:
Hi Donald,
On Sat, Nov 29, 2014 at 19:43 -0500, Donald Stufft wrote:
On Nov 13, 2014, at 9:21 PM, Donald Stufft <donald@stufft.io> wrote:
Starting a new thread with more explicit details at Richard’s request. Essentially the tl;dr here is that we'll switch to using sha2 (specifically sha256).
Ping?
Are we OK to make this change?
sorry i didn't get back earlier. Before the minor release of devpi-server last week i tried for two hours to change devpi-server to accomodate your planned pypi.python.org checksum changes.
I found the change cannot easily be done without changes to the underlying database schema and thus needs a major new release of devpi-server because an export/import cycle is needed. When doing that i also want to do some internal cleanup related to name normalization (and also relating to recent pypi.python.org changes) but i need a week or two i guess to do that. However i now think that if you do the pypi.python.org checksum change it shouldn't directly break devpi-server but it would remove checksum checking. I'd rather like to have a new major devpi-server release out when you do the change. Is it ok for you to wait a bit still?
best, holger
Yes, we can wait a bit. I was just going over my TODO list and making sure things weren’t getting lost in the shuffle. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Mon, Dec 1, 2014 at 12:35 PM, Donald Stufft <donald@stufft.io> wrote:
On Dec 1, 2014, at 4:25 AM, holger krekel <holger@merlinux.eu> wrote:
Hi Donald,
On Sat, Nov 29, 2014 at 19:43 -0500, Donald Stufft wrote:
On Nov 13, 2014, at 9:21 PM, Donald Stufft <donald@stufft.io> wrote:
Starting a new thread with more explicit details at Richard’s request. Essentially the tl;dr here is that we'll switch to using sha2 (specifically sha256).
Ping?
Are we OK to make this change?
sorry i didn't get back earlier. Before the minor release of devpi-server last week i tried for two hours to change devpi-server to accomodate your planned pypi.python.org checksum changes.
I found the change cannot easily be done without changes to the underlying database schema and thus needs a major new release of devpi-server because an export/import cycle is needed. When doing that i also want to do some internal cleanup related to name normalization (and also relating to recent pypi.python.org changes) but i need a week or two i guess to do that. However i now think that if you do the pypi.python.org checksum change it shouldn't directly break devpi-server but it would remove checksum checking. I'd rather like to have a new major devpi-server release out when you do the change. Is it ok for you to wait a bit still?
best, holger
Yes, we can wait a bit. I was just going over my TODO list and making sure things weren’t getting lost in the shuffle.
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Holger, Is there anyway people on this list can help with the updates to devpi so that we can get this out sooner? Cheers, Ian
On Mon, Dec 01, 2014 at 12:45 -0600, Ian Cordasco wrote:
On Mon, Dec 1, 2014 at 12:35 PM, Donald Stufft <donald@stufft.io> wrote:
On Dec 1, 2014, at 4:25 AM, holger krekel <holger@merlinux.eu> wrote:
Hi Donald,
On Sat, Nov 29, 2014 at 19:43 -0500, Donald Stufft wrote:
On Nov 13, 2014, at 9:21 PM, Donald Stufft <donald@stufft.io> wrote:
Starting a new thread with more explicit details at Richard’s request. Essentially the tl;dr here is that we'll switch to using sha2 (specifically sha256).
Ping?
Are we OK to make this change?
sorry i didn't get back earlier. Before the minor release of devpi-server last week i tried for two hours to change devpi-server to accomodate your planned pypi.python.org checksum changes.
I found the change cannot easily be done without changes to the underlying database schema and thus needs a major new release of devpi-server because an export/import cycle is needed. When doing that i also want to do some internal cleanup related to name normalization (and also relating to recent pypi.python.org changes) but i need a week or two i guess to do that. However i now think that if you do the pypi.python.org checksum change it shouldn't directly break devpi-server but it would remove checksum checking. I'd rather like to have a new major devpi-server release out when you do the change. Is it ok for you to wait a bit still?
best, holger
Yes, we can wait a bit. I was just going over my TODO list and making sure things weren’t getting lost in the shuffle.
Holger,
Is there anyway people on this list can help with the updates to devpi so that we can get this out sooner?
Looking at devpi/server/devpi_server/extpypi.py and devpi/server/devpi_server/model.py mainly and changing most places where "md5" is found in the source and adapting related tests. Is there a specific reason you are in a hurry if i may ask? best, holger
On Mon, Dec 1, 2014 at 3:23 PM, holger krekel <holger@merlinux.eu> wrote:
On Mon, Dec 01, 2014 at 12:45 -0600, Ian Cordasco wrote:
On Mon, Dec 1, 2014 at 12:35 PM, Donald Stufft <donald@stufft.io> wrote:
On Dec 1, 2014, at 4:25 AM, holger krekel <holger@merlinux.eu> wrote:
Hi Donald,
On Sat, Nov 29, 2014 at 19:43 -0500, Donald Stufft wrote:
On Nov 13, 2014, at 9:21 PM, Donald Stufft <donald@stufft.io> wrote:
Starting a new thread with more explicit details at Richard’s request. Essentially the tl;dr here is that we'll switch to using sha2 (specifically sha256).
Ping?
Are we OK to make this change?
sorry i didn't get back earlier. Before the minor release of devpi-server last week i tried for two hours to change devpi-server to accomodate your planned pypi.python.org checksum changes.
I found the change cannot easily be done without changes to the underlying database schema and thus needs a major new release of devpi-server because an export/import cycle is needed. When doing that i also want to do some internal cleanup related to name normalization (and also relating to recent pypi.python.org changes) but i need a week or two i guess to do that. However i now think that if you do the pypi.python.org checksum change it shouldn't directly break devpi-server but it would remove checksum checking. I'd rather like to have a new major devpi-server release out when you do the change. Is it ok for you to wait a bit still?
best, holger
Yes, we can wait a bit. I was just going over my TODO list and making sure things weren’t getting lost in the shuffle.
Holger,
Is there anyway people on this list can help with the updates to devpi so that we can get this out sooner?
Looking at devpi/server/devpi_server/extpypi.py and devpi/server/devpi_server/model.py mainly and changing most places where "md5" is found in the source and adapting related tests.
Is there a specific reason you are in a hurry if i may ask?
best, holger
No real hurry. I just like helping out when there's an opening and this thread has been around for a short while already. Given the topic is related to the security of PyPI and its users, I'd like to help move that forward if possible. That's all. (It's mostly me being selfish.)
On Mon, Dec 01, 2014 at 15:29 -0600, Ian Cordasco wrote:
On Mon, Dec 1, 2014 at 3:23 PM, holger krekel <holger@merlinux.eu> wrote:
On Mon, Dec 01, 2014 at 12:45 -0600, Ian Cordasco wrote:
On Mon, Dec 1, 2014 at 12:35 PM, Donald Stufft <donald@stufft.io> wrote:
On Dec 1, 2014, at 4:25 AM, holger krekel <holger@merlinux.eu> wrote:
Hi Donald,
On Sat, Nov 29, 2014 at 19:43 -0500, Donald Stufft wrote:
> On Nov 13, 2014, at 9:21 PM, Donald Stufft <donald@stufft.io> wrote: > > Starting a new thread with more explicit details at Richard’s request. > Essentially the tl;dr here is that we'll switch to using sha2 (specifically > sha256).
Ping?
Are we OK to make this change?
sorry i didn't get back earlier. Before the minor release of devpi-server last week i tried for two hours to change devpi-server to accomodate your planned pypi.python.org checksum changes.
I found the change cannot easily be done without changes to the underlying database schema and thus needs a major new release of devpi-server because an export/import cycle is needed. When doing that i also want to do some internal cleanup related to name normalization (and also relating to recent pypi.python.org changes) but i need a week or two i guess to do that. However i now think that if you do the pypi.python.org checksum change it shouldn't directly break devpi-server but it would remove checksum checking. I'd rather like to have a new major devpi-server release out when you do the change. Is it ok for you to wait a bit still?
best, holger
Yes, we can wait a bit. I was just going over my TODO list and making sure things weren’t getting lost in the shuffle.
Holger,
Is there anyway people on this list can help with the updates to devpi so that we can get this out sooner?
Looking at devpi/server/devpi_server/extpypi.py and devpi/server/devpi_server/model.py mainly and changing most places where "md5" is found in the source and adapting related tests.
Is there a specific reason you are in a hurry if i may ask?
best, holger
No real hurry. I just like helping out when there's an opening and this thread has been around for a short while already. Given the topic is related to the security of PyPI and its users, I'd like to help move that forward if possible. That's all. (It's mostly me being selfish.)
Quite an empathic form of selfishness. If you want to check things out and have questions please feel free to ask maybe privately. holger
Holger, has this happened yet?
On Dec 1, 2014, at 4:23 PM, holger krekel <holger@merlinux.eu> wrote:
On Mon, Dec 01, 2014 at 12:45 -0600, Ian Cordasco wrote:
On Mon, Dec 1, 2014 at 12:35 PM, Donald Stufft <donald@stufft.io> wrote:
On Dec 1, 2014, at 4:25 AM, holger krekel <holger@merlinux.eu> wrote:
Hi Donald,
On Sat, Nov 29, 2014 at 19:43 -0500, Donald Stufft wrote:
On Nov 13, 2014, at 9:21 PM, Donald Stufft <donald@stufft.io> wrote:
Starting a new thread with more explicit details at Richard’s request. Essentially the tl;dr here is that we'll switch to using sha2 (specifically sha256).
Ping?
Are we OK to make this change?
sorry i didn't get back earlier. Before the minor release of devpi-server last week i tried for two hours to change devpi-server to accomodate your planned pypi.python.org checksum changes.
I found the change cannot easily be done without changes to the underlying database schema and thus needs a major new release of devpi-server because an export/import cycle is needed. When doing that i also want to do some internal cleanup related to name normalization (and also relating to recent pypi.python.org changes) but i need a week or two i guess to do that. However i now think that if you do the pypi.python.org checksum change it shouldn't directly break devpi-server but it would remove checksum checking. I'd rather like to have a new major devpi-server release out when you do the change. Is it ok for you to wait a bit still?
best, holger
Yes, we can wait a bit. I was just going over my TODO list and making sure things weren’t getting lost in the shuffle.
Holger,
Is there anyway people on this list can help with the updates to devpi so that we can get this out sooner?
Looking at devpi/server/devpi_server/extpypi.py and devpi/server/devpi_server/model.py mainly and changing most places where "md5" is found in the source and adapting related tests.
Is there a specific reason you are in a hurry if i may ask?
best, holger
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
participants (4)
-
Donald Stufft -
holger krekel -
Ian Cordasco -
Richard Jones