Migrating Hashes from MD5 to SHA256
PyPI has historically used MD5 in order to verify the downloads. However MD5 is severely broken and is generally regarded as something that should be migrated away from ASAP. From speaking with a number of cryptographers they've more or less said that the major reason they believe that MD5 hasn't had a published pre-image attack is just because it's so broken that most researchers have moved on to newer hashes. Since versions 1.2 pip has supported md5, sha1, and any of the sha2 family. Additionally it has only supported SSL verification since 1.3. This means there is no version of pip which both verifies SSL and only allows MD5. Since version 0.9 setuptools has supported md5, sha1, and any of the sha2 family and it has only supported SSL verification since 0.7. I propose we switch PyPI from using MD5 to using SHA256. There is no security lost from using a hash that pip prior to version 1.2 doesn't understand as it didn't verify SSL so an attacker could simply modify the hashes if they wanted. Additionally there is no security list from setuptools versions earlier than 0.7. However setuptools versions 0.7-0.8 will lose their hashes. I believe this is a ok thing to happen because the uptake on 0.7-0.8 is fairly low. Most people will use the setuptools bundled with virtualenv which has only ever bundled 0.6 or 0.9. So essentially moving from MD5 to SHA256 will only negatively affect the security of a handful of users while positively impacting the security of everyone else. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Fri, Jul 26, 2013 at 12:25 PM, Donald Stufft <donald@stufft.io> wrote:
Additionally there is no security list from setuptools versions earlier than 0.7.
Not true, actually. Setuptools 0.6 dev releases supported SSL verification since mid-May, but don't support any hashes besides MD5. Anybody who updated their setuptools between then and the release of 0.7 would have that version. Unfortunately, it's hard to tell how many people that is, though I could try and dig through my server logs to find out. There's also another issue with jumping to SHA256: Python prior to 2.5 didn't support it. Which brings up another point: the setuptools 0.6 series is the only setuptools available for Python 2.3. That's one of the reasons it's still available for download. If you want SSL verification on 2.3, it's the only thing available. (Meanwhile, a lot of people are still downloading 0.6c11; probably I should package up an 0.6c12 so those folks pick it up instead of 0.6c11.) Anyway, this is all somewhat moot since the hashes only matter when the download is hosted somewhere besides PyPI, since SSL verification is available for the PyPI part. Even so, I'd suggest that moving to SHA1 might be a good intermediate step: it's available on Python 2.3, so I could backport the relevant support to the 0.6 branch. (IIUC, Python 2.3 is still the default version for many Linux distros that have not reached end-of-life support.)
PJ Eby <pje <at> telecommunity.com> writes:
There's also another issue with jumping to SHA256: Python prior to 2.5 didn't support it.
I'm not sure this is a particularly relevant concern. Python's prior to 2.5 are no longer supported by the people who wrote them, or almost any major packages (SQLAlchemy, Django, pip, virtualenv, the list goes on and on). I'm not sure why we should expend any effort to support them either. Alex
On Jul 26, 2013, at 2:33 PM, PJ Eby <pje@telecommunity.com> wrote:
On Fri, Jul 26, 2013 at 12:25 PM, Donald Stufft <donald@stufft.io> wrote:
Additionally there is no security list from setuptools versions earlier than 0.7.
Not true, actually. Setuptools 0.6 dev releases supported SSL verification since mid-May, but don't support any hashes besides MD5. Anybody who updated their setuptools between then and the release of 0.7 would have that version. Unfortunately, it's hard to tell how many people that is, though I could try and dig through my server logs to find out.
There's also another issue with jumping to SHA256: Python prior to 2.5 didn't support it.
Which brings up another point: the setuptools 0.6 series is the only setuptools available for Python 2.3. That's one of the reasons it's still available for download. If you want SSL verification on 2.3, it's the only thing available. (Meanwhile, a lot of people are still downloading 0.6c11; probably I should package up an 0.6c12 so those folks pick it up instead of 0.6c11.)
Anyway, this is all somewhat moot since the hashes only matter when the download is hosted somewhere besides PyPI, since SSL verification is available for the PyPI part. Even so, I'd suggest that moving to SHA1 might be a good intermediate step: it's available on Python 2.3, so I could backport the relevant support to the 0.6 branch. (IIUC, Python 2.3 is still the default version for many Linux distros that have not reached end-of-life support.)
I don't have a Python 2.3 available to attempt to test. To be honest I've never even used Python 2.3. Does the hashlib backport I added to setuptools 0.9 for Python 2.4 work on 2.3? It's a pure python implementation of hashlib. Sha1 is better but it already has weaknesses so if at all possible it would be much preferred and significantly better to switch to sha256. Setuptools doesn't appear to include the python version in it's user agent so I can't get any sort of information about Python 2.3 usage. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Fri, Jul 26, 2013 at 3:14 PM, Donald Stufft <donald@stufft.io> wrote:
Does the hashlib backport I added to setuptools 0.9 for Python 2.4 work on 2.3? It's a pure python implementation of hashlib.
Ah, didn't know about that! I can't imagine what problems there would be; not much changed in 2.4 that can't be emulated in 2.3. Anyway, I'll have a look. Thanks!
I don't have a Python 2.3 available to attempt to test. To be honest I've never even used Python 2.3.
Heh. Noob. ;-) (j/k) 2.3 is basically 2.4 minus decorators, generator expressions, various builtins and stdlib features. Unless you used set types, reversed(), or various other odds and ends, I should be able to backport it.
[stuff about RHEL support]
If there's a 2.4 hashlib backport, that addresses my concerns just fine. If I need to, I'll backport it to setuptools 0.6. Thanks.
On Jul 26, 2013, at 4:59 PM, PJ Eby <pje@telecommunity.com> wrote:
On Fri, Jul 26, 2013 at 3:14 PM, Donald Stufft <donald@stufft.io> wrote:
Does the hashlib backport I added to setuptools 0.9 for Python 2.4 work on 2.3? It's a pure python implementation of hashlib.
Ah, didn't know about that! I can't imagine what problems there would be; not much changed in 2.4 that can't be emulated in 2.3.
Anyway, I'll have a look. Thanks!
Here's the relevant commits in the new setuptools stuff: https://bitbucket.org/pypa/setuptools/commits/330b628f38c9380c95a818e65fd568... https://bitbucket.org/pypa/setuptools/commits/b1d4e48beebdcc3cf7cb06fae4c400... https://bitbucket.org/pypa/setuptools/commits/12dd4b89148a225856a060cbee1137... The implementations are taken from PyPy so they are made to work on Python 2.7, but they worked just fine on 2.4 after removing a single b"".
I don't have a Python 2.3 available to attempt to test. To be honest I've never even used Python 2.3.
Heh. Noob. ;-) (j/k)
:) I was still in high school when the last 2.3 was released :/
2.3 is basically 2.4 minus decorators, generator expressions, various builtins and stdlib features. Unless you used set types, reversed(), or various other odds and ends, I should be able to backport it.
Great!
[stuff about RHEL support]
If there's a 2.4 hashlib backport, that addresses my concerns just fine. If I need to, I'll backport it to setuptools 0.6. Thanks.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 27 July 2013 06:59, PJ Eby <pje@telecommunity.com> wrote:
On Fri, Jul 26, 2013 at 3:14 PM, Donald Stufft <donald@stufft.io> wrote:
Does the hashlib backport I added to setuptools 0.9 for Python 2.4 work on 2.3? It's a pure python implementation of hashlib.
Ah, didn't know about that! I can't imagine what problems there would be; not much changed in 2.4 that can't be emulated in 2.3.
Anyway, I'll have a look. Thanks!
FWIW, I expect the intersection of "running RHEL 4" and "downloading software directly from PyPI" to be a vanishingly small subset of humanity - anybody conservative enough to be running a version of RHEL that old is going to have *very* strict rules about how software gets onto their production servers and are also likely to be using something far more recent to talk to the outside world. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jul 26, 2013, at 2:33 PM, PJ Eby <pje@telecommunity.com> wrote:
Anyway, this is all somewhat moot since the hashes only matter when the download is hosted somewhere besides PyPI, since SSL verification is available for the PyPI part. Even so, I'd suggest that moving to SHA1 might be a good intermediate step: it's available on Python 2.3, so I could backport the relevant support to the 0.6 branch. (IIUC, Python 2.3 is still the default version for many Linux distros that have not reached end-of-life support.)
I think RHEL is the one that will support things the longest. As far as I can tell Python 2.3 was default in RHEL4 and RHEL5 used Python 2.4 ELS support for RHEL4 ends Feb of 2015, so that's roughly a year and a half till not even Red Hat supports Python 2.3 anymore that I can tell? It also appears that support for new installations ended roughly a year and a half ago. Many (most?) projects no longer support Python 2.3 on PyPI, and I seriously doubt that there is a significant number of people who both want the stability provided by RHEL and is willing to continue using a release that is this close to being EOL'd while simultaneously wanting to download things from PyPI. CPython hasn't supported Python 2.3 in years. Basically If RHEL customers want the security updates they should bother Red Hat for them, that's part of why they pay for RHEL instead of going with a free system. I don't think it's appropriate to allow a handful of people who might still be on a version of python first released 10 years ago and last released 8 years ago to negatively impact everyone else. Note: I don't use RHEL so my understanding of it's life cycle is from reading their support page. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Am 26.07.2013 18:25, schrieb Donald Stufft:
PyPI has historically used MD5 in order to verify the downloads. However MD5 is severely broken and is generally regarded as something that should be migrated away from ASAP. From speaking with a number of cryptographers they've more or less said that the major reason they believe that MD5 hasn't had a published pre-image attack is just because it's so broken that most researchers have moved on to newer hashes.
Since versions 1.2 pip has supported md5, sha1, and any of the sha2 family. Additionally it has only supported SSL verification since 1.3. This means there is no version of pip which both verifies SSL and only allows MD5.
Since version 0.9 setuptools has supported md5, sha1, and any of the sha2 family and it has only supported SSL verification since 0.7.
I propose we switch PyPI from using MD5 to using SHA256. There is no security lost from using a hash that pip prior to version 1.2 doesn't understand as it didn't verify SSL so an attacker could simply modify the hashes if they wanted. Additionally there is no security list from setuptools versions earlier than 0.7.
A couple of months ago I suggested a schema that includes MD5, SHA-2 and file size: file.tar.gz#MD5=1234&SHA-256=abcd&filesize=5023 That should work for old versions of setuptool and can easily be supported in new versions of pip and setuptools. A new hash sum scheme must include the possibility to add multiple and new hash algorithms. A download tool shall check the hash sum for all supported algorithms, too. I also like to see the file size in the scheme. It's useful to know the file size in preparation of the download. The file size validation mitigates some attack possibilities. Christian -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iQIcBAEBCAAGBQJR8s0HAAoJEMeIxMHUVQ1FO90P+gM6Lx3WZA94Tg8bEco6ckLI m00Yt+Gwn1HgnF+wSAaxIsThy2C/yltTPqfJiwZGErlzt0tAIdFZhYMbkhzO1//Z N6i1O+seH5eqMSgUd7K1mgiRIAsKpXH6SEt/U3VzPNm/qVvIV0FFIUTEIx9xXpkD HYKbup7dkcBkIkhreUpIG4TGEK22/Vcs+G4NjR8UllcqRS4iWrkiKzXuLfnto++t 9fnYfz0uxh2nG3doFGr2gzypLtctrRzAqy28AtlbgGEKaK5E2/hoGrRE8VIBZg+f SEWKLctTLoOcXHVTaxoAcp+3XzwKZPpGoJzjyLtPFDrH55kZFLA2a25vB51xteLA A7Kz60eccHe7Io76incJiL+RmorcpUTTp6FRoTCdqDUW2rSmTM1z8tUtenJNAQYG UnuyRrRbTeQ1JlImdakqXA1X5/qsYLy7kcaf4Xb9SXxIdlEk//0o3tiB4B92oIgF If5yx65KoKPUCg1pXA/ZawTuH/d1aJQWOjz0eP7Wn+GnEnHxoKIwYMP65xVyNCXU 0afS5lRs7gxtOKlXofWoXO1u7H7EHJQzFbbgdJkSl65mz+hOVMu1w7RQwPb7LzeO 16gnUtvIpXFaab/NCM4UmXuHWLx07jWB4ZJ45zsXuyXa3m4kdt9aMS0oVaSYgA/a Zq84rJiWc17eItR9iyU5 =ZRue -----END PGP SIGNATURE-----
On Jul 26, 2013, at 3:24 PM, Christian Heimes <christian@python.org> wrote:
A couple of months ago I suggested a schema that includes MD5, SHA-2 and file size:
file.tar.gz#MD5=1234&SHA-256=abcd&filesize=5023
That should work for old versions of setuptool and can easily be supported in new versions of pip and setuptools.
It won't work for old versions, it explicitly includes the end of line terminator and the #.
A new hash sum scheme must include the possibility to add multiple and new hash algorithms. A download tool shall check the hash sum for all supported algorithms, too. I also like to see the file size in the scheme. It's useful to know the file size in preparation of the download. The file size validation mitigates some attack possibilities.
Right now that would break too much. I agree this is where we need to get too but It'll likely need to wait for the new API in Warehouse.
Christian
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Fri, Jul 26, 2013 at 12:25:36PM -0400, Donald Stufft wrote:
PyPI has historically used MD5 in order to verify the downloads. However MD5 is severely broken and is generally regarded as something that should be migrated away from ASAP. From speaking with a number of cryptographers they've more or less said that the major reason they believe that MD5 hasn't had a published pre-image attack is just because it's so broken that most researchers have moved on to newer hashes.
Who said that? That contradicts my beliefs. Thanks! Regards, Zooko
On Jul 26, 2013, at 8:55 PM, zooko <zooko@zooko.com> wrote:
On Fri, Jul 26, 2013 at 12:25:36PM -0400, Donald Stufft wrote:
PyPI has historically used MD5 in order to verify the downloads. However MD5 is severely broken and is generally regarded as something that should be migrated away from ASAP. From speaking with a number of cryptographers they've more or less said that the major reason they believe that MD5 hasn't had a published pre-image attack is just because it's so broken that most researchers have moved on to newer hashes.
Who said that? That contradicts my beliefs.
It's possible I misunderstood the exact implications of what they were saying. I am not a cryptographer and it was a month or two ago we spoke. It was stressed to me that PyPI should be moving off of MD5. I do believe however that we don't know for sure if MD5 is going to be have a practical pre-image attack tomorrow, or if it will last another 10 years. Given that all security systems are not infallible and are generally designed so that you have margins of security so there is time to migrate. The safety margins on MD5 have long since gone so by continuing to use it we are ignoring prudence (especially at a fairly ideal time where we are at a transitioning from unverified HTTPS/HTTP to HTTPS so we do not need to regard backwards compatibility as highly). As far as I am aware these attacks tend to come all of a sudden and without warning. I would much rather have already migrated to something that still has it's safety margins than be caught with our proverbial pants down and need to scramble *if* an attack is discovered. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Donald Stufft <donald <at> stufft.io> writes:
Most people will use the setuptools bundled with virtualenv which has only ever bundled 0.6 or 0.9.
Why do you think that? According to PyPI: - virtualenv: 4664 downloads in the last day 34512 downloads in the last week - setuptools: 23901 downloads in the last day 197351 downloads in the last week Certainly setuptools use seems much more widespread than virtualenv use. Regards Antoine.
On Jul 27, 2013, at 7:12 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Donald Stufft <donald <at> stufft.io> writes:
Most people will use the setuptools bundled with virtualenv which has only ever bundled 0.6 or 0.9.
Why do you think that? According to PyPI: - virtualenv: 4664 downloads in the last day 34512 downloads in the last week - setuptools: 23901 downloads in the last day 197351 downloads in the last week
Certainly setuptools use seems much more widespread than virtualenv use.
Regards
Antoine.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Comparing download numbers isn't generally useful. People almost never install virtualenv more than once per machine. There are a myriad of reasons why they might directly install setuptools. It's impossible to know for sure how it'll be gotten but my gut is that most people use whatever is default inside of virtualenv because that's how almost everyone i've seen who uses virtualenv does it unless they have special needs. However the 0.9 line have almost as many downloads alone as the 0.7 and 0.8 lines combined. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Donald Stufft <donald <at> stufft.io> writes:
On Jul 27, 2013, at 7:12 AM, Antoine Pitrou <solipsis <at> pitrou.net> wrote:
Donald Stufft <donald <at> stufft.io> writes:
Most people will use the setuptools bundled with virtualenv which has only ever bundled 0.6 or 0.9.
Why do you think that? According to PyPI: - virtualenv: 4664 downloads in the last day 34512 downloads in the last week - setuptools: 23901 downloads in the last day 197351 downloads in the last week
Certainly setuptools use seems much more widespread than virtualenv use.
Regards
Antoine.
Comparing download numbers isn't generally useful. People almost never install virtualenv more than once per machine. There are a myriad of reasons why they might directly install setuptools.
If your assertion were true ("Most people will use the setuptools bundled with virtualenv"), then the setuptools download numbers would be minuscule. The actual numbers show it to be untrue. Whether or not they are directly comparable isn't important: the orders are magnitude are sufficiently eloquent.
It's impossible to know for sure how it'll be gotten but my gut is that most people use whatever is default inside of virtualenv because that's how almost everyone i've seen who uses virtualenv does it unless they have special needs.
Perhaps you don't realize that many people don't use virtualenv at all, so they simply cannot use virtualenv's setuptools, either. Which is perfectly compatible with those download numbers, unlike your original assertion. Regards Antoine.
On Jul 27, 2013, at 1:25 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
If your assertion were true ("Most people will use the setuptools bundled with virtualenv"), then the setuptools download numbers would be minuscule. The actual numbers show it to be untrue. Whether or not they are directly comparable isn't important: the orders are magnitude are sufficiently eloquent.
It's impossible to know for sure how it'll be gotten but my gut is that most people use whatever is default inside of virtualenv because that's how almost everyone i've seen who uses virtualenv does it unless they have special needs.
Perhaps you don't realize that many people don't use virtualenv at all, so they simply cannot use virtualenv's setuptools, either. Which is perfectly compatible with those download numbers, unlike your original assertion.
I don't think any claim can be made about the relative use between the two tools by looking at the download counts because their typical use is generally very different. But sure you're right whatever does that make you feel better? Are you trying to claim we shouldn't move to a stronger hash? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Donald Stufft <donald <at> stufft.io> writes:
I don't think any claim can be made about the relative use between the two tools by looking at the download counts because their typical use is generally very different.
I'll try to phrase it more clearly then: I am not *comparing* their relative use. I am simply pointing out that an extremely large number of people install setuptools separately. Whether or not they also use virtualenv is completely irrelevent (but, of course, chances are they don't: otherwise, as you say, they'd use the bundled versions).
But sure you're right whatever does that make you feel better?
Now, please calm down...
Are you trying to claim we shouldn't move to a stronger hash?
No, I'm just saying the possibility of regressions isn't as small as you think based on a misinterpretation of how people actually get setuptools installed (many of them get it directly from PyPI). But, yes, we should of course move to something better than md5, and ideally make the format flexible enough to avoid further breakage when switching hashes again. Regards Antoine.
Ok so given that: - There's a readably available solution for Python 2.4+ with the likelihood being that most users are either using it or using an older version which doesn't support SSL. - The number of folks likely to be on Python 2.3 and wanting to install things from PyPI is likely to be very small. - There's possibly a future solution for Python 2.3 - The safety margins for MD5 are gone and cryptographers heavily suggest moving away from it. - A revised scheme will break backwards compatibility with the versions of the tooling that do support a stronger hash. I'm going to go ahead and make this change unless someone comes out and contests moving PyPI to SHA256. I'll give it a bit to make sure no one does have an issue with the move. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 28 July 2013 20:55, Donald Stufft <donald@stufft.io> wrote:
Ok so given that:
- There's a readably available solution for Python 2.4+ with the likelihood being that most users are either using it or using an older version which doesn't support SSL. - The number of folks likely to be on Python 2.3 and wanting to install things from PyPI is likely to be very small. - There's possibly a future solution for Python 2.3 - The safety margins for MD5 are gone and cryptographers heavily suggest moving away from it. - A revised scheme will break backwards compatibility with the versions of the tooling that do support a stronger hash.
I'm going to go ahead and make this change unless someone comes out and contests moving PyPI to SHA256. I'll give it a bit to make sure no one does have an issue with the move.
+1, this sounds like a good way forward for the existing PyPI interfaces. We can do something better once the focus shifts from "make the status quo not broken" to making the next generation interfaces a reality (PEP 426 et al). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Hi Nick, Donald, all, On Sun, Jul 28, 2013 at 22:23 +1000, Nick Coghlan wrote:
On 28 July 2013 20:55, Donald Stufft <donald@stufft.io> wrote:
Ok so given that:
- There's a readably available solution for Python 2.4+ with the likelihood being that most users are either using it or using an older version which doesn't support SSL. - The number of folks likely to be on Python 2.3 and wanting to install things from PyPI is likely to be very small. - There's possibly a future solution for Python 2.3 - The safety margins for MD5 are gone and cryptographers heavily suggest moving away from it.
Please detail the actual attack scenario wrt PyPI/installer processes.
- A revised scheme will break backwards compatibility with the versions of the tooling that do support a stronger hash.
I'm going to go ahead and make this change unless someone comes out and contests moving PyPI to SHA256. I'll give it a bit to make sure no one does have an issue with the move.
Actually, i strongly object further backward-incompatible changes. Please (generally) find a way to introduce improvements without breaking existing installation processes at the same time. For example, in this case pip/easy_install could indicate to PYPI what kind of hashes it accepts (through a header or query param or whatever) and PyPI could serve it but we'd default to MD5 for now if nothing else was requested. Please also consider the PEP438 vetted registration of externals+hashses in this context. Once things and tools are working nicely we can switch to serving a non-MD5 hash as default after a sufficient grace period.
+1, this sounds like a good way forward for the existing PyPI interfaces.
We can do something better once the focus shifts from "make the status quo not broken" to making the next generation interfaces a reality (PEP 426 et al).
Just switching the hashes is likely to break things. Do you want to take a bet on that? Just switching to SSL broke things. Just switching pip-1.4 broke things. Just switching to newer setuptools broke things. For me the fall-out of all the well-intentioned changes has been frustrating lately. For example, one of my projects documented how to generate a supervisor-controled deployment and suddenly "pip install supervisor" does not work anymore because pip-1.4 doesn't consider any existing supervisor distributions as non-dev. You can argue that's supervisor's fault but in the end it doesn't matter: What used to work and was documented is now broken, users complaining or simply turning away, with a re-inforced view of "python packaging sucks" or "this tool sucks, the docs are wrong". There are >1000 times more people using python packaging tools than there are ones following the recent distutils-sig/Donald/Nick verdicts or tool release notes. And heck, even for me being quite involved in all this stuff, having written a related PEP, it's really hard to see what's coming and to prevent breakage for my tools's/projects's users. Therefore, to be honest, at this point i am almost afraid of what is released/deployed next in PyPI/Installer land. I'd rather like to get into a "can't wait to get the next release" mode. best, holger
On 29 July 2013 19:38, holger krekel <holger@merlinux.eu> wrote:
Hi Nick, Donald, all,
On Sun, Jul 28, 2013 at 22:23 +1000, Nick Coghlan wrote:
On 28 July 2013 20:55, Donald Stufft <donald@stufft.io> wrote:
Ok so given that:
- There's a readably available solution for Python 2.4+ with the likelihood being that most users are either using it or using an older version which doesn't support SSL. - The number of folks likely to be on Python 2.3 and wanting to install things from PyPI is likely to be very small. - There's possibly a future solution for Python 2.3 - The safety margins for MD5 are gone and cryptographers heavily suggest moving away from it.
Please detail the actual attack scenario wrt PyPI/installer processes.
- A revised scheme will break backwards compatibility with the versions of the tooling that do support a stronger hash.
I'm going to go ahead and make this change unless someone comes out and contests moving PyPI to SHA256. I'll give it a bit to make sure no one does have an issue with the move.
Actually, i strongly object further backward-incompatible changes.
Please (generally) find a way to introduce improvements without breaking existing installation processes at the same time.
For example, in this case pip/easy_install could indicate to PYPI what kind of hashes it accepts (through a header or query param or whatever) and PyPI could serve it but we'd default to MD5 for now if nothing else was requested. Please also consider the PEP438 vetted registration of externals+hashses in this context. Once things and tools are working nicely we can switch to serving a non-MD5 hash as default after a sufficient grace period.
Having the improved hashes be opt-in (by the client) strikes me as a reasonable request. Yes, this means nothing will actually happen until easy_install/pip are updated to request those improved hashes and those versions see significant uptake, but as Holger says, we need to ensure we put sufficient effort into smoothing out the roller coaster ride that has been the recent experience of packaging system users. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jul 29, 2013, at 7:58 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Actually, i strongly object further backward-incompatible changes.
Please (generally) find a way to introduce improvements without breaking existing installation processes at the same time.
For example, in this case pip/easy_install could indicate to PYPI what kind of hashes it accepts (through a header or query param or whatever) and PyPI could serve it but we'd default to MD5 for now if nothing else was requested. Please also consider the PEP438 vetted registration of externals+hashses in this context. Once things and tools are working nicely we can switch to serving a non-MD5 hash as default after a sufficient grace period.
Having the improved hashes be opt-in (by the client) strikes me as a reasonable request.
Yes, this means nothing will actually happen until easy_install/pip are updated to request those improved hashes and those versions see significant uptake, but as Holger says, we need to ensure we put sufficient effort into smoothing out the roller coaster ride that has been the recent experience of packaging system users.
There's basically zero way for this to fail closed in any of the current installers. The failure mode is unverified packages not uninstallable packages. I am not aware of a single installer that mandates the use of a hash. Crate.io has never used md5 hashes and has always used sha256 and I've never received a single report of an installer being unable to install because of it, which is exactly what I expect. Indicating via Header or query param pretty much destroys the effectiveness of the CDN's cache in order to fix a problem with a theoretical (as far as I am aware) installer that requires a md5 hash (and thus has never worked for any of the externally hosted packages. Additionally it doesn't account for external urls which need to be registered *with* a hash. As far as available attacks, *today* an author could upload a package that has been created so as to have a sister package with malicious code that has the same hash allowing them to have a malicious package they can substitute at will without the hashes changing at all. In the future it's possible that a pre-image attack on MD5 will be found and then we'll be dealing with this problem then when we've lost all verification on external urls instead of now when we have time to get external urls to switch. So by all means I will not migrate us if that's what you want. Old versions of the installation clients stick around far to long for the opt in mechanism to be much use. The point of switching was to cover the existing clients as well to narrow the gap until a new API is developed. Hopefully no one is relying on these hashes to prevent an author from maliciously injecting a sister package and hopefully the strength of MD5 holds and no new research is found that blows it's pre-image attack residence to pieces. As far as not breaking things goes backwards compatibility has been an important concern however progress forward *requires* breakage. It is required because there is a vast array of available ways to have your package and/or hosting configured many of them horrible practices which need to be killed. Killing them requires breaking backwards compatibility. You cite SSL, yes SSL has caused a number of errors for people mostly related to older versions of OpenSSL being unable to use a SSL certificate but downloading code you're going to execute over plaintext isn't just bad, it's downright negligent on the part of the toolchain. So that was a required breakage. You also mention the pip 1.4 *not* installing pre-releases by default. Yes that broke a handful of packages Supervisor and pytz being the major ones that I've seen anyone complain about. It was also known ahead of time that this was a backwards incompatible change (and it was noted as such in the release notes). It wasn't a surprising outcome. The pip developers "drew a line in the sand" to quote Paul Moore and I expect pip 1.5 where PEP438 becomes default to break even more packages from people who just haven't bothered to change their practices until it's forced on them. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Mon, Jul 29, 2013 at 10:30 -0400, Donald Stufft wrote:
On Jul 29, 2013, at 7:58 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Actually, i strongly object further backward-incompatible changes.
Please (generally) find a way to introduce improvements without breaking existing installation processes at the same time.
For example, in this case pip/easy_install could indicate to PYPI what kind of hashes it accepts (through a header or query param or whatever) and PyPI could serve it but we'd default to MD5 for now if nothing else was requested. Please also consider the PEP438 vetted registration of externals+hashses in this context. Once things and tools are working nicely we can switch to serving a non-MD5 hash as default after a sufficient grace period.
Having the improved hashes be opt-in (by the client) strikes me as a reasonable request.
Yes, this means nothing will actually happen until easy_install/pip are updated to request those improved hashes and those versions see significant uptake, but as Holger says, we need to ensure we put sufficient effort into smoothing out the roller coaster ride that has been the recent experience of packaging system users.
There's basically zero way for this to fail closed in any of the current installers. The failure mode is unverified packages not uninstallable packages. I am not aware of a single installer that mandates the use of a hash. Crate.io has never used md5 hashes and has always used sha256 and I've never received a single report of an installer being unable to install because of it, which is exactly what I expect.
So you think the worst case for forcing SHA256 hashes is that installers who don't yet support sha256 hashes would just ignore it (and thus wouldn't do hash verification)?
Indicating via Header or query param pretty much destroys the effectiveness of the CDN's cache in order to fix a problem with a theoretical (as far as I am aware) installer that requires a md5 hash (and thus has never worked for any of the externally hosted packages. Additionally it doesn't account for external urls which need to be registered *with* a hash.
Currently there is no hash-type enforcement on registered externals, is there?
As far as available attacks, *today* an author could upload a package that has been created so as to have a sister package with malicious code that has the same hash allowing them to have a malicious package they can substitute at will without the hashes changing at all. In the future it's possible that a pre-image attack on MD5 will be found and then we'll be dealing with this problem then when we've lost all verification on external urls instead of now when we have time to get external urls to switch.
So the attack is a malicious author or someone else modifying an external release file (either directly on the server or via MITM) while maintaining the pre-registered MD5 hash, right? I am currently merely trying to understand more exactly what you are worried about. best, holger
So by all means I will not migrate us if that's what you want. Old versions of the installation clients stick around far to long for the opt in mechanism to be much use. The point of switching was to cover the existing clients as well to narrow the gap until a new API is developed.
Hopefully no one is relying on these hashes to prevent an author from maliciously injecting a sister package and hopefully the strength of MD5 holds and no new research is found that blows it's pre-image attack residence to pieces.
As far as not breaking things goes backwards compatibility has been an important concern however progress forward *requires* breakage. It is required because there is a vast array of available ways to have your package and/or hosting configured many of them horrible practices which need to be killed. Killing them requires breaking backwards compatibility. You cite SSL, yes SSL has caused a number of errors for people mostly related to older versions of OpenSSL being unable to use a SSL certificate but downloading code you're going to execute over plaintext isn't just bad, it's downright negligent on the part of the toolchain. So that was a required breakage.
You also mention the pip 1.4 *not* installing pre-releases by default. Yes that broke a handful of packages Supervisor and pytz being the major ones that I've seen anyone complain about. It was also known ahead of time that this was a backwards incompatible change (and it was noted as such in the release notes). It wasn't a surprising outcome. The pip developers "drew a line in the sand" to quote Paul Moore and I expect pip 1.5 where PEP438 becomes default to break even more packages from people who just haven't bothered to change their practices until it's forced on them.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Jul 29, 2013, at 1:28 PM, holger krekel <holger@merlinux.eu> wrote:
On Mon, Jul 29, 2013 at 10:30 -0400, Donald Stufft wrote:
On Jul 29, 2013, at 7:58 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Actually, i strongly object further backward-incompatible changes.
Please (generally) find a way to introduce improvements without breaking existing installation processes at the same time.
For example, in this case pip/easy_install could indicate to PYPI what kind of hashes it accepts (through a header or query param or whatever) and PyPI could serve it but we'd default to MD5 for now if nothing else was requested. Please also consider the PEP438 vetted registration of externals+hashses in this context. Once things and tools are working nicely we can switch to serving a non-MD5 hash as default after a sufficient grace period.
Having the improved hashes be opt-in (by the client) strikes me as a reasonable request.
Yes, this means nothing will actually happen until easy_install/pip are updated to request those improved hashes and those versions see significant uptake, but as Holger says, we need to ensure we put sufficient effort into smoothing out the roller coaster ride that has been the recent experience of packaging system users.
There's basically zero way for this to fail closed in any of the current installers. The failure mode is unverified packages not uninstallable packages. I am not aware of a single installer that mandates the use of a hash. Crate.io has never used md5 hashes and has always used sha256 and I've never received a single report of an installer being unable to install because of it, which is exactly what I expect.
So you think the worst case for forcing SHA256 hashes is that installers who don't yet support sha256 hashes would just ignore it (and thus wouldn't do hash verification)?
Yes. I've been using sha256 on simple.crate.io for over a year and zero people have ever stated it didn't work for them. This also fits in with my knowledge of how setuptools and pip works. I know zc.buildout less well but to my knowledge they simple allow setuptools to handle the downloading.
Indicating via Header or query param pretty much destroys the effectiveness of the CDN's cache in order to fix a problem with a theoretical (as far as I am aware) installer that requires a md5 hash (and thus has never worked for any of the externally hosted packages. Additionally it doesn't account for external urls which need to be registered *with* a hash.
Currently there is no hash-type enforcement on registered externals, is there?
Registered externals must register with a md5 hash, scraped links and download urls etc do not require it because they are indirectly added. There is no verification by PyPI that the given hash matches the package at the end of the url.
As far as available attacks, *today* an author could upload a package that has been created so as to have a sister package with malicious code that has the same hash allowing them to have a malicious package they can substitute at will without the hashes changing at all. In the future it's possible that a pre-image attack on MD5 will be found and then we'll be dealing with this problem then when we've lost all verification on external urls instead of now when we have time to get external urls to switch.
So the attack is a malicious author or someone else modifying an external release file (either directly on the server or via MITM) while maintaining the pre-registered MD5 hash, right?
I am currently merely trying to understand more exactly what you are worried about.
For any hash function there are two major types of attacks you worry about. The first is a collision attack which is the ability to generate two arbitrary inputs that hash to the same thing. The second is a pre-image attack (either first or second pre-image) which essentially means given an already existing input generate another input that hashes to the same thing. So basically the difference between the two attacks are wether you have a hash you're trying to match or if you just need two inputs that hash to the same thing. MD5 is currently broken for collision resistance. This means that an author can generate two packages that hash to the same thing. Once package might be benign and one might be malicious. Given those two packages people using the md5 hashes will not be able to differentiate between the benign and the malicous package. MD5 is currently *not* broken for pre image resistance. This means that as of right now someone can not take an already existing package on PyPI and generate a second package that hashes to the same thing (besides via brute forcing). So right now, collision attacks possible == yes, pre image attacks possible == no. However designing secure systems is a practice of building in safety margins. If someone, for instance, can break 5 rounds of a function you use 15 rounds. With cryptographic hashes collision attacks are easier than pre-image attacks, so if you have two functions, one that has a collision attack and one that doesn't you can generally assume that the one without a collision attack is stronger and has a longer shelf life. So the problem with MD5 (ignoring for a second the fact that a collision attack can be bad on it's own) is that there are no more safety nets. If it gets broken for a pre-image then there's not likely to be any warning (we've already *had* the warning). It will just be broken and we will be scrambling to update things then (and hopefully nobody gets attacked in the meantime). And I do say *if* because as zooko pointed out, it's not a guarantee that MD5 will ever lose it's pre-image resistance (which just means that brute forcing is the quickest way to generate a hash).
best, holger
So by all means I will not migrate us if that's what you want. Old versions of the installation clients stick around far to long for the opt in mechanism to be much use. The point of switching was to cover the existing clients as well to narrow the gap until a new API is developed.
Hopefully no one is relying on these hashes to prevent an author from maliciously injecting a sister package and hopefully the strength of MD5 holds and no new research is found that blows it's pre-image attack residence to pieces.
As far as not breaking things goes backwards compatibility has been an important concern however progress forward *requires* breakage. It is required because there is a vast array of available ways to have your package and/or hosting configured many of them horrible practices which need to be killed. Killing them requires breaking backwards compatibility. You cite SSL, yes SSL has caused a number of errors for people mostly related to older versions of OpenSSL being unable to use a SSL certificate but downloading code you're going to execute over plaintext isn't just bad, it's downright negligent on the part of the toolchain. So that was a required breakage.
You also mention the pip 1.4 *not* installing pre-releases by default. Yes that broke a handful of packages Supervisor and pytz being the major ones that I've seen anyone complain about. It was also known ahead of time that this was a backwards incompatible change (and it was noted as such in the release notes). It wasn't a surprising outcome. The pip developers "drew a line in the sand" to quote Paul Moore and I expect pip 1.5 where PEP438 becomes default to break even more packages from people who just haven't bothered to change their practices until it's forced on them.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Mon, Jul 29, 2013 at 14:30 -0400, Donald Stufft wrote:
On Jul 29, 2013, at 1:28 PM, holger krekel <holger@merlinux.eu> wrote:
On Mon, Jul 29, 2013 at 10:30 -0400, Donald Stufft wrote:
On Jul 29, 2013, at 7:58 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Actually, i strongly object further backward-incompatible changes.
Please (generally) find a way to introduce improvements without breaking existing installation processes at the same time.
For example, in this case pip/easy_install could indicate to PYPI what kind of hashes it accepts (through a header or query param or whatever) and PyPI could serve it but we'd default to MD5 for now if nothing else was requested. Please also consider the PEP438 vetted registration of externals+hashses in this context. Once things and tools are working nicely we can switch to serving a non-MD5 hash as default after a sufficient grace period.
Having the improved hashes be opt-in (by the client) strikes me as a reasonable request.
Yes, this means nothing will actually happen until easy_install/pip are updated to request those improved hashes and those versions see significant uptake, but as Holger says, we need to ensure we put sufficient effort into smoothing out the roller coaster ride that has been the recent experience of packaging system users.
There's basically zero way for this to fail closed in any of the current installers. The failure mode is unverified packages not uninstallable packages. I am not aware of a single installer that mandates the use of a hash. Crate.io has never used md5 hashes and has always used sha256 and I've never received a single report of an installer being unable to install because of it, which is exactly what I expect.
So you think the worst case for forcing SHA256 hashes is that installers who don't yet support sha256 hashes would just ignore it (and thus wouldn't do hash verification)?
Yes. I've been using sha256 on simple.crate.io for over a year and zero people have ever stated it didn't work for them. This also fits in with my knowledge of how setuptools and pip works. I know zc.buildout less well but to my knowledge they simple allow setuptools to handle the downloading.
Sounds good. Maybe "secondary" tools could get problems, though. I know for sure that devpi-server might stumble but i can fix that. Also i remember there were tools that memorized MD5 hashes in requirements files etc.
Indicating via Header or query param pretty much destroys the effectiveness of the CDN's cache in order to fix a problem with a theoretical (as far as I am aware) installer that requires a md5 hash (and thus has never worked for any of the externally hosted packages. Additionally it doesn't account for external urls which need to be registered *with* a hash.
Currently there is no hash-type enforcement on registered externals, is there?
Registered externals must register with a md5 hash, scraped links and download urls etc do not require it because they are indirectly added. There is no verification by PyPI that the given hash matches the package at the end of the url.
Hum, can we allow submitting multiple hashes? Are there tools already that help with registering externals?
As far as available attacks, *today* an author could upload a package that has been created so as to have a sister package with malicious code that has the same hash allowing them to have a malicious package they can substitute at will without the hashes changing at all. In the future it's possible that a pre-image attack on MD5 will be found and then we'll be dealing with this problem then when we've lost all verification on external urls instead of now when we have time to get external urls to switch.
So the attack is a malicious author or someone else modifying an external release file (either directly on the server or via MITM) while maintaining the pre-registered MD5 hash, right?
I am currently merely trying to understand more exactly what you are worried about.
(...)
MD5 is currently broken for collision resistance. This means that an author can generate two packages that hash to the same thing. Once package might be benign and one might be malicious. Given those two packages people using the md5 hashes will not be able to differentiate between the benign and the malicous package.
I think we should not pretend that PyPI has (by itself) any safety belts against malicious authors. There are numerous ways for malicious authors to do evil if they choose to. The potential ability to fake a package using a collision attack merely adds another way. Do you know, btw, if TUF is going to help with any of what we are discussing here? (I am again a bit lost as to the roadmap wrt to TUF - is there something?) cheers, holger
MD5 is currently *not* broken for pre image resistance. This means that as of right now someone can not take an already existing package on PyPI and generate a second package that hashes to the same thing (besides via brute forcing).
So right now, collision attacks possible == yes, pre image attacks possible == no.
However designing secure systems is a practice of building in safety margins. If someone, for instance, can break 5 rounds of a function you use 15 rounds. With cryptographic hashes collision attacks are easier than pre-image attacks, so if you have two functions, one that has a collision attack and one that doesn't you can generally assume that the one without a collision attack is stronger and has a longer shelf life.
So the problem with MD5 (ignoring for a second the fact that a collision attack can be bad on it's own) is that there are no more safety nets. If it gets broken for a pre-image then there's not likely to be any warning (we've already *had* the warning). It will just be broken and we will be scrambling to update things then (and hopefully nobody gets attacked in the meantime).
And I do say *if* because as zooko pointed out, it's not a guarantee that MD5 will ever lose it's pre-image resistance (which just means that brute forcing is the quickest way to generate a hash).
best, holger
So by all means I will not migrate us if that's what you want. Old versions of the installation clients stick around far to long for the opt in mechanism to be much use. The point of switching was to cover the existing clients as well to narrow the gap until a new API is developed.
Hopefully no one is relying on these hashes to prevent an author from maliciously injecting a sister package and hopefully the strength of MD5 holds and no new research is found that blows it's pre-image attack residence to pieces.
As far as not breaking things goes backwards compatibility has been an important concern however progress forward *requires* breakage. It is required because there is a vast array of available ways to have your package and/or hosting configured many of them horrible practices which need to be killed. Killing them requires breaking backwards compatibility. You cite SSL, yes SSL has caused a number of errors for people mostly related to older versions of OpenSSL being unable to use a SSL certificate but downloading code you're going to execute over plaintext isn't just bad, it's downright negligent on the part of the toolchain. So that was a required breakage.
You also mention the pip 1.4 *not* installing pre-releases by default. Yes that broke a handful of packages Supervisor and pytz being the major ones that I've seen anyone complain about. It was also known ahead of time that this was a backwards incompatible change (and it was noted as such in the release notes). It wasn't a surprising outcome. The pip developers "drew a line in the sand" to quote Paul Moore and I expect pip 1.5 where PEP438 becomes default to break even more packages from people who just haven't bothered to change their practices until it's forced on them.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Jul 30, 2013, at 3:56 AM, holger krekel <holger@merlinux.eu> wrote:
Yes. I've been using sha256 on simple.crate.io for over a year and zero people have ever stated it didn't work for them. This also fits in with my knowledge of how setuptools and pip works. I know zc.buildout less well but to my knowledge they simple allow setuptools to handle the downloading.
Sounds good. Maybe "secondary" tools could get problems, though. I know for sure that devpi-server might stumble but i can fix that. Also i remember there were tools that memorized MD5 hashes in requirements files etc.
The change was nixed but it wasn't about removing the ability for pip and such to use MD5s. Merely what PyPI serves.
Registered externals must register with a md5 hash, scraped links and download urls etc do not require it because they are indirectly added. There is no verification by PyPI that the given hash matches the package at the end of the url.
Hum, can we allow submitting multiple hashes? Are there tools already that help with registering externals?
Not easily and in a backwards compatible way.
MD5 is currently broken for collision resistance. This means that an author can generate two packages that hash to the same thing. Once package might be benign and one might be malicious. Given those two packages people using the md5 hashes will not be able to differentiate between the benign and the malicous package.
I think we should not pretend that PyPI has (by itself) any safety belts against malicious authors. There are numerous ways for malicious authors to do evil if they choose to. The potential ability to fake a package using a collision attack merely adds another way.
Correct, which is why I tried to be very specific about the types of attacks :)
Do you know, btw, if TUF is going to help with any of what we are discussing here? (I am again a bit lost as to the roadmap wrt to TUF - is there something?)
TUF would provide protections against a pre-image attack yes. However it has it's own problems and is still likely a ways out (if we use it). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Jul 29, 2013, at 5:38 AM, holger krekel <holger@merlinux.eu> wrote:
There are >1000 times more people using python packaging tools than there are ones following the recent distutils-sig/Donald/Nick verdicts or tool release notes. And heck, even for me being quite involved in all this stuff, having written a related PEP, it's really hard to see what's coming and to prevent breakage for my tools's/projects's users. Therefore, to be honest, at this point i am almost afraid of what is released/deployed next in PyPI/Installer land. I'd rather like to get into a "can't wait to get the next release" mode.
I'd also like to point out this. I think you got bitten by one particular backward incompatibility and are "Once bitten twice shy". I had a number people asking me near every day how long until pip 1.4 was coming out because they were excited for the new changes including the improved pre-release handling. I've personally gotten or seen more complaints over the naming of a variable in the config file then I have over any changes we've made. The runner up to that is the fallout from switching to requiring verified SSL. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/29/2013 10:51 AM, Donald Stufft wrote:
I've personally gotten or seen more complaints over the naming of a variable in the config file then I have over any changes we've made. The runner up to that is the fallout from switching to requiring verified SSL.
The past few months have generated a *lot* of teeth-gnashing / hair-pulling, especially among "downstream" developers (those unlikely to be reading this SIG): - - HTTPS-only PyPY - - Distribute / setuptools merge, e.g. cratering folks who use a distro-managed 'python-distribute' package. - - Pip's new backward-imcompatible "final releases by default". I think we are going to be in a much better place for all that, but let's not deny the pain involved for *everybody* in getting there. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlH2n+EACgkQ+gerLs4ltQ6megCeN8V8IN4OT6rWfZg1tw1GtUaO 2jwAoLRXzLyjjMbiLrcfqLG0Ge1NQnMq =7ufm -----END PGP SIGNATURE-----
On 29 July 2013 18:01, Tres Seaver <tseaver@palladion.com> wrote:
I think we are going to be in a much better place for all that, but let's not deny the pain involved for *everybody* in getting there.
Agreed. I think the goal is valid, and the approach is fine. But we need to do a better job in managing people's expectations. I'd like to see a roadmap of the various changes planned, as well as some sort of explanation of how each of the changes contributes towards the end goal. Personally, none of the changes have detrimentally affected me, so my opinion is largely theoretical. But even I am getting a little frustrated by the constant claims that "what we have now is insecure and broken, and must be fixed ASAP". The reality is that everything's more or less OK - there's a risk, certainly, and it could be severe, but many, many people are routinely using PyPI all the time without issues. And telling them that they are wrong to do so, or that they are being extremely naive over security, isn't helping. Paul
On Mon, Jul 29, 2013 at 10:18 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 29 July 2013 18:01, Tres Seaver <tseaver@palladion.com> wrote:
I think we are going to be in a much better place for all that, but let's not deny the pain involved for *everybody* in getting there.
Agreed. I think the goal is valid, and the approach is fine. But we need to do a better job in managing people's expectations. I'd like to see a roadmap of the various changes planned, as well as some sort of explanation of how each of the changes contributes towards the end goal.
Nick said he's planning a roadmap on the new User Guide (in August I think after vacation), so that's something.
On Jul 29, 2013, at 1:18 PM, Paul Moore <p.f.moore@gmail.com> wrote:
But even I am getting a little frustrated by the constant claims that "what we have now is insecure and broken, and must be fixed ASAP". The reality is that everything's more or less OK - there's a risk, certainly, and it could be severe, but many, many people are routinely using PyPI all the time without issues. And telling them that they are wrong to do so, or that they are being extremely naive over security, isn't helping.
This shows a fundamental misunderstanding of how security issues present themselves. Of course things just work for people because security issues are not like regular bugs. They don't negatively affect you until someone attempts to use them to attack you. Keep your front door unlocked on your house and your valuables will remain inside _until_ someone decides to try and rob you. If you wait until people are affected by a security vulnerability then the horse has already fled the pasture and you're just attempting to close the gate after the fact. I'm pushing hard on doing what we can to secure the infrastructure because this shit matters. Everything is more or less OK, only because no one has decided that people installing from PyPI are not a valuable enough target to go after. Prior to this push that was basically the only thing prevent someone from attacking people, that they had never decided to bother too. We are better, it's somewhat harder now, but in many areas that's still the only thing keeping people safe. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Mon, Jul 29, 2013 at 2:15 PM, Donald Stufft <donald@stufft.io> wrote:
On Jul 29, 2013, at 1:18 PM, Paul Moore <p.f.moore@gmail.com> wrote:
But even I am getting a little frustrated by the constant claims that "what we have now is insecure and broken, and must be fixed ASAP". The reality is that everything's more or less OK - there's a risk, certainly, and it could be severe, but many, many people are routinely using PyPI all the time without issues. And telling them that they are wrong to do so, or that they are being extremely naive over security, isn't helping.
This shows a fundamental misunderstanding of how security issues present themselves. Of course things just work for people because security issues are not like regular bugs. They don't negatively affect you until someone attempts to use them to attack you. Keep your front door unlocked on your house and your valuables will remain inside _until_ someone decides to try and rob you. If you wait until people are affected by a security vulnerability then the horse has already fled the pasture and you're just attempting to close the gate after the fact.
I'm pushing hard on doing what we can to secure the infrastructure because this shit matters. Everything is more or less OK, only because no one has decided that people installing from PyPI are not a valuable enough target to go after. Prior to this push that was basically the only thing prevent someone from attacking people, that they had never decided to bother too. We are better, it's somewhat harder now, but in many areas that's still the only thing keeping people safe.
Well said. Security is a pain, but I'm really glad and appreciate that you and others are paying attention to it. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton
Folks: dstufft already correctly explained that relying on MD5 allows for "doppelganger" packages -- two (or more) packages which are engineered at birth to have the same hash as each other. It isn't clear to me that this can be used for evil, but it isn't obvious that it *can't* be used for evil, either. So it would certainly be helpful to upgrade the hash function so that we don't have to think about that anymore, but in my opinion it is not an emergency. I'd like to push back on the other risk, that someone might figure out how to make MD5 second-pre-images. I don't think this is a risk that we need to urgently address, and I've written a short note explaining why. This note is incomplete, badly edited, has not been peer-reviewed, and is not ready for publication, but I thought it might help folks evaluate how urgent it is to upgrade from MD5, so here it is. Regards, Zooko
On Jul 29, 2013, at 2:57 PM, zooko <zooko@zooko.com> wrote:
I'd like to push back on the other risk, that someone might figure out how to make MD5 second-pre-images. I don't think this is a risk that we need to urgently address, and I've written a short note explaining why. This note is incomplete, badly edited, has not been peer-reviewed, and is not ready for publication, but I thought it might help folks evaluate how urgent it is to upgrade from MD5, so here it is.
I don't think it's urgent to fix it, but I think it's a good security hardening effort with very little downside and very little chance of regression. However, as I said if Holger, or anyone else, has a concern about the affects of adding this bit of security hardening to give us a safety net again then I simply won't do it in the simple API. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Jul 29, 2013, at 3:14 PM, Donald Stufft <donald@stufft.io> wrote:
On Jul 29, 2013, at 2:57 PM, zooko <zooko@zooko.com> wrote:
I'd like to push back on the other risk, that someone might figure out how to make MD5 second-pre-images. I don't think this is a risk that we need to urgently address, and I've written a short note explaining why. This note is incomplete, badly edited, has not been peer-reviewed, and is not ready for publication, but I thought it might help folks evaluate how urgent it is to upgrade from MD5, so here it is.
I don't think it's urgent to fix it, but I think it's a good security hardening effort with very little downside and very little chance of regression. However, as I said if Holger, or anyone else, has a concern about the affects of adding this bit of security hardening to give us a safety net again then I simply won't do it in the simple API.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Somewhat relevant to the question at hand: http://valerieaurora.org/hash.html (Yes it lists sha-2 as weakened, which it is. However sha-3 isn't widespread enough for us :( ) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Mon, Jul 29, 2013 at 04:33:11PM -0400, Donald Stufft wrote:
Somewhat relevant to the question at hand: http://valerieaurora.org/hash.html
Heh heh. That page is cited in my note. My note is kind of a response to that page, showing that the history of pre-image attacks is completely different than the history of collision attacks.
(Yes it lists sha-2 as weakened, which it is. However sha-3 isn't widespread enough for us :( )
There's no reason to worry about SHA-2. In my opinion, there's no particular reason to think that it will be made vulnerable to collisions within the next decade! By the way, I'm a co-author of a secure hash function -- BLAKE2: https://blake2.net/ The intent of BLAKE2 is to be as secure as SHA-3 but as fast as MD5. Not only is it as fast as MD5, but it also has an optional parallel mode that can go 4 or 8 times as fast as MD5 by using 4 or 8 CPU cores! It is currently being adopted for uses like data deduplication, archiving, and distributed filesystems, where the data can be large (terabytes or more), and the performance of the hash function is a bottleneck. I don't think Python packaging has such needs, and BLAKE2 is not a standard like SHA-2 and SHA-3, so I'm not pushing to add support for it. Regards, Zooko
On 29 Jul, 2013, at 22:33, Donald Stufft <donald@stufft.io> wrote:
On Jul 29, 2013, at 3:14 PM, Donald Stufft <donald@stufft.io> wrote:
On Jul 29, 2013, at 2:57 PM, zooko <zooko@zooko.com> wrote:
I'd like to push back on the other risk, that someone might figure out how to make MD5 second-pre-images. I don't think this is a risk that we need to urgently address, and I've written a short note explaining why. This note is incomplete, badly edited, has not been peer-reviewed, and is not ready for publication, but I thought it might help folks evaluate how urgent it is to upgrade from MD5, so here it is.
I don't think it's urgent to fix it, but I think it's a good security hardening effort with very little downside and very little chance of regression. However, as I said if Holger, or anyone else, has a concern about the affects of adding this bit of security hardening to give us a safety net again then I simply won't do it in the simple API.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Somewhat relevant to the question at hand: http://valerieaurora.org/hash.html
(Yes it lists sha-2 as weakened, which it is. However sha-3 isn't widespread enough for us :( )
That SHA-3 isn't widespread yet is not a surprise, AFAIK it isn't even a standard yet :-). According to <http://csrc.nist.gov/groups/ST/hash/sha-3/timeline_fips.html> the standard will be finalized in Q2 2014. BTW. I agree that the MD5 checksums on PyPI will have to go some time, and it would be nice if the replacement scheme had a way to use multiple hashes to make it easier to switch to a hash in future. I know to little of the setuptools and pip implementations to have anything useful to add to the discussion about the timing for this. Ronald
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On 30 Jul 2013 05:15, "Donald Stufft" <donald@stufft.io> wrote:
On Jul 29, 2013, at 2:57 PM, zooko <zooko@zooko.com> wrote:
I'd like to push back on the other risk, that someone might figure out
make MD5 second-pre-images. I don't think this is a risk that we need to urgently address, and I've written a short note explaining why. This note is incomplete, badly edited, has not been peer-reviewed, and is not ready for publication, but I thought it might help folks evaluate how urgent it is to upgrade from MD5, so here it is.
I don't think it's urgent to fix it, but I think it's a good security hardening effort with very little downside and very little chance of regression. However, as I said if Holger, or anyone else, has a concern about the affects of adding
how to this
bit of security hardening to give us a safety net again then I simply won't do it in the simple API.
I'm thinking that may be the way to go - treat verified SSL as our final stop-gap for the simple API and focus on hardening the next generation APIs. This is more for social reasons than strictly technical ones. I think you're right this particular change is unlikely to break anything, but there are also enough genuinely essential changes needed that we should avoid unnecessary flux in other areas. In this case, I think the need for a pre-image attack that still produces a working download and an old installer that isn't using verified SSL but can check SHA256 hashes reduces the attack window to a point where I'm prepared to live with the use of MD5 as a known risk. Cheers, Nick.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Paul Moore <p.f.moore <at> gmail.com> writes:
Personally, none of the changes have detrimentally affected me, so my opinion is largely theoretical. But even I am getting a little frustrated by the constant claims that "what we have now is insecure and broken, and must be fixed ASAP".
FWIW, +1. You may be paranoid, but not everyone has to be (or suffer the consequences of it). Security issues should be fixed without breaking things in a hassle (which is the policy we followed e.g. for the ssl module, or hash randomization). The whole python.org infrastructure is built on an OS kernel written by someone who thinks security issues are normal bugs. AFAIK there is no plan to switch to OpenBSD. Regards Antoine.
On Jul 29, 2013, at 10:41 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Paul Moore <p.f.moore <at> gmail.com> writes:
Personally, none of the changes have detrimentally affected me, so my opinion is largely theoretical. But even I am getting a little frustrated by the constant claims that "what we have now is insecure and broken, and must be fixed ASAP".
FWIW, +1. You may be paranoid, but not everyone has to be (or suffer the consequences of it). Security issues should be fixed without breaking things in a hassle (which is the policy we followed e.g. for the ssl module, or hash randomization).
You missed a key word "… when possible". If there is a problem we will fix it, when we can do that in a way that minimizes breakages we will do that. Its all just about cost-benefit, and when you are talking about "executing code downloaded from the internet" it becomes quite easy to see benefits outweighing costs even with pretty major UX changes. Not something we do lightly, but status quo does not win here, sorry.
The whole python.org infrastructure is built on an OS kernel written by someone who thinks security issues are normal bugs. AFAIK there is no plan to switch to OpenBSD.
This is news to me, we specifically run Ubuntu LTS because Canonical's security response team has a proven track record of handling issues. If you mean that Linus doesn't handle security issues well, then it is fortunate indeed that we don't actually use his software. --Noah
On Jul 30, 2013, at 2:02 AM, Noah Kantrowitz <noah@coderanger.net> wrote:
On Jul 29, 2013, at 10:41 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Paul Moore <p.f.moore <at> gmail.com> writes:
Personally, none of the changes have detrimentally affected me, so my opinion is largely theoretical. But even I am getting a little frustrated by the constant claims that "what we have now is insecure and broken, and must be fixed ASAP".
FWIW, +1. You may be paranoid, but not everyone has to be (or suffer the consequences of it). Security issues should be fixed without breaking things in a hassle (which is the policy we followed e.g. for the ssl module, or hash randomization).
You missed a key word "… when possible". If there is a problem we will fix it, when we can do that in a way that minimizes breakages we will do that. Its all just about cost-benefit, and when you are talking about "executing code downloaded from the internet" it becomes quite easy to see benefits outweighing costs even with pretty major UX changes. Not something we do lightly, but status quo does not win here, sorry.
Basically said it better than I could :) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Noah Kantrowitz <noah <at> coderanger.net> writes:
The whole python.org infrastructure is built on an OS kernel written by someone who thinks security issues are normal bugs. AFAIK there is no plan to switch to OpenBSD.
This is news to me, we specifically run Ubuntu LTS because Canonical's security response team has a proven track record of handling issues. If you mean that Linus doesn't handle security issues well, then it is fortunate indeed that we don't actually use his software.
Did you already forget what the discussion is about? Security/bugfix Ubuntu LTS updates don't break compatibility for the sake of hardening things, which is the whole point. (As for the idea that using "Canonical's kernel" amounts to not using "Linus' software", that's a rather unorthodox notion of authorship. It's very likely Canonical doesn't change more than 1% LOC in the kernel, so you're still bound to Linus' decisions for at least 99% of the code - and even probably for the remaining 1%, since Canonical's version won't be massively divergent.) Regards Antoine.
On Jul 29, 2013, at 11:19 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Noah Kantrowitz <noah <at> coderanger.net> writes:
The whole python.org infrastructure is built on an OS kernel written by someone who thinks security issues are normal bugs. AFAIK there is no plan to switch to OpenBSD.
This is news to me, we specifically run Ubuntu LTS because Canonical's security response team has a proven track record of handling issues. If you mean that Linus doesn't handle security issues well, then it is fortunate indeed that we don't actually use his software.
Did you already forget what the discussion is about? Security/bugfix Ubuntu LTS updates don't break compatibility for the sake of hardening things, which is the whole point.
Again, speaking as the guy that has to clean up the mess when they do break compat, I promise you they do. Same deal, they only break compat when keeping compat would present a threat to users, which is quite often the case with security bugs. They are fortunately a bit further ahead of us on the long tail of finding problems, so this is far less frequent than it was in years past. We will get there too, but like I said, status quo is not a defense here, just strap in and hang on. --Noah
On Jul 30, 2013, at 2:19 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Noah Kantrowitz <noah <at> coderanger.net> writes:
The whole python.org infrastructure is built on an OS kernel written by someone who thinks security issues are normal bugs. AFAIK there is no plan to switch to OpenBSD.
This is news to me, we specifically run Ubuntu LTS because Canonical's security response team has a proven track record of handling issues. If you mean that Linus doesn't handle security issues well, then it is fortunate indeed that we don't actually use his software.
Did you already forget what the discussion is about? Security/bugfix Ubuntu LTS updates don't break compatibility for the sake of hardening things, which is the whole point.
Well for one PyPI doesn't have releases so there is no LTS or not LTS, it's just what's being served so there's no simple place to break backwards compatibility. As far as forgetting what's being discussed here then it sounds like you've apparently missed the fact I already conceded the change to MD5 and further more this thread was explicitly split off from the MD5 request because, as far as I can tell, Holger wanted to discuss the broader topic of compatibility in general and not just specific to this particular issue.
(As for the idea that using "Canonical's kernel" amounts to not using "Linus' software", that's a rather unorthodox notion of authorship. It's very likely Canonical doesn't change more than 1% LOC in the kernel, so you're still bound to Linus' decisions for at least 99% of the code - and even probably for the remaining 1%, since Canonical's version won't be massively divergent.)
Regards
Antoine.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Jul 30, 2013, at 1:41 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Paul Moore <p.f.moore <at> gmail.com> writes:
Personally, none of the changes have detrimentally affected me, so my opinion is largely theoretical. But even I am getting a little frustrated by the constant claims that "what we have now is insecure and broken, and must be fixed ASAP".
FWIW, +1. You may be paranoid, but not everyone has to be (or suffer the consequences of it). Security issues should be fixed without breaking things in a hassle (which is the policy we followed e.g. for the ssl module, or hash randomization).
People are generally not paranoid until they've been successfully attacked. I *will* advocate and push for breaking things where security is concerned because regardless of if you care or not, a lot of people *do* care and the nature of the beast is that you're only as strong as the weakest link. This particular change wasn't an immediate vulnerability that I felt was urgent, hence why I've backed off on it when people were concerned about the backwards compat implications. I will not back off when it comes to issues that *do* have an immediate or near term issue, regardless of if some people don't care or not.
The whole python.org infrastructure is built on an OS kernel written by someone who thinks security issues are normal bugs. AFAIK there is no plan to switch to OpenBSD.
So classifying bugs as security vs "normal" is supposed to make it easier on people. The thought is that creating new releases and applying updates is a time consuming process and often times requires things such as reboots or service restarts so by dividing issues up into security vs not security the amount of disruption can be minimized for only "important" updates. There's actually pretty strong evidence that shows the process of classifying bugs as security bugs is a harmful process and that all updates should be treated the same because it's often times not immediately obvious what the security implications are, even to security experts[1]. I'm sure your dig at the OS is supposed to be some sort of masterstroke about how we're not being as secure as possible anyways however I would contest that OpenBSD is actually more secure. It's major claim to fame is that they haven't had a vulnerably in the OpenBSD base system in "a heck of a long time". The problem is the OpenBSD base system is terribly small and that claim cannot be made once you include their packages. Further more at the last I checked OpenBSD does not provide (although this may have changed) and abilities to do MAC which means you're relying entirely on an attackers ability to *not* get in versus providing fail safes to contain an attack once it's happened. Infrastructure is not using MAC currently but I would love to get us to that point as well. [1] citeseerx.ist.psu.edu/viewdoc/download;jsessionid=7B6E224144709E99B7FAEBFC497621A1?doi=10.1.1.148.9757&rep=rep1&type=pdf ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Donald Stufft <donald <at> stufft.io> writes:
I *will* advocate and push for breaking things where security is concerned
because
regardless of if you care or not, a lot of people *do* care and the nature of the beast is that you're only as strong as the weakest link.
There's actually pretty strong evidence that shows the process of classifying bugs as security bugs is a harmful
That's nice, but you're not alone here, so whatever you want to "push for" needn't always happen. process and that
all updates should be treated the same because it's often times not immediately obvious what the security implications are, even to security experts[1].
Doesn't it contradict your own stance on the subject? ("This shows a fundamental misunderstanding of how security issues present themselves. Of course things just work for people because security issues are not like regular bugs" - which is a flawed argument btw. Many bugs have random or rare occurrences - not just security issues)
I'm sure your dig at the OS is supposed to be some sort of masterstroke about how we're not being as secure as possible anyways however I would contest that OpenBSD is actually more secure.
WTF are you talking about? No it's not. I'm simply pointing out that, for some strange reason, you decided to trust an OS whose author has very different views on how to fix security issues than you have. Regards Antoine.
On Jul 30, 2013, at 2:28 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Donald Stufft <donald <at> stufft.io> writes:
I *will* advocate and push for breaking things where security is concerned
because
regardless of if you care or not, a lot of people *do* care and the nature of the beast is that you're only as strong as the weakest link.
That's nice, but you're not alone here, so whatever you want to "push for" needn't always happen.
I have zero qualms about releasing a full disclosure along with working exploits into the wild for a security vulnerability that people block me on. If I'm unable to rectify the problem I will make sure that everyone *knows* about the problem.
There's actually pretty strong evidence that shows the process of classifying bugs as security bugs is a harmful process and that all updates should be treated the same because it's often times not immediately obvious what the security implications are, even to security experts[1].
Doesn't it contradict your own stance on the subject?
("This shows a fundamental misunderstanding of how security issues present themselves. Of course things just work for people because security issues are not like regular bugs" - which is a flawed argument btw. Many bugs have random or rare occurrences - not just security issues)
No? How you treat a bug and how they present themselves are not the same thing? Even a random occurrence will break for some percentage of people using the software some percentage of the time. If it didn't then it's unlikely anyone would notice it. Security vulnerabilities typically won't break until someone actively tries to break them.
I'm sure your dig at the OS is supposed to be some sort of masterstroke about how we're not being as secure as possible anyways however I would contest that OpenBSD is actually more secure.
WTF are you talking about? No it's not. I'm simply pointing out that, for some strange reason, you decided to trust an OS whose author has very different views on how to fix security issues than you have.
Well the Kernel isn't the OS, it's part of the OS and we run an OS whose authors actually care a whole lot about security.
Regards
Antoine.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Donald Stufft <donald <at> stufft.io> writes:
I have zero qualms about releasing a full disclosure along with working
into the wild for a security vulnerability that people block me on. If I'm unable to rectify the problem I will make sure that everyone *knows* about the
exploits problem. I don't know what I'm supposed to infer from such a statement, except that I probably don't want to trust you. You might think that "publish[ing] working exploits into the wild" is some kind of heroic, altruistic act, but I think few people would agree.
Even a random occurrence will break for some percentage of people using the software some percentage of the time. If it didn't then it's unlikely anyone would notice it. Security vulnerabilities typically won't break until someone actively tries to break them.
You're mistaken. Bugs can sometimes be fixed preemptively, even before they're noticed in the wild (by means of perusing the code and noticing an issue, for example). Which also includes, of course, security issues (which often get fixed before they ever get exploited). Regards Antoine.
On Jul 30, 2013, at 12:01 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Donald Stufft <donald <at> stufft.io> writes:
I have zero qualms about releasing a full disclosure along with working
into the wild for a security vulnerability that people block me on. If I'm unable to rectify the problem I will make sure that everyone *knows* about the
exploits problem.
I don't know what I'm supposed to infer from such a statement, except that I probably don't want to trust you. You might think that "publish[ing] working exploits into the wild" is some kind of heroic, altruistic act, but I think few people would agree.
No, this is the standard for security researchers. If the vendor ignores the reported exploit for long enough, they go public and try to make sure users understand the risks and how to mitigate them in the time it takes the vendor to fix it. --Noah
On Jul 30, 2013, at 3:01 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
I don't know what I'm supposed to infer from such a statement, except that I probably don't want to trust you. You might think that "publish[ing] working exploits into the wild" is some kind of heroic, altruistic act, but I think few people would agree.
Full Disclosure is a common practice amongst security professionals when the upstream project is unwilling to rectify the problem. So yes I do think the practice of Full Disclosure is an altruistic act and often times the only thing that gets people who don't care to pull their head out of the sand and actually care. If you don't believe my words on it here's an essay by Bruce Schneier one of the foremost experts on security and a well respected and well trusted member of the security community. https://www.schneier.com/essay-146.html ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Donald Stufft <donald <at> stufft.io> writes:
On Jul 30, 2013, at 3:01 AM, Antoine Pitrou <solipsis <at> pitrou.net> wrote:
I don't know what I'm supposed to infer from such a statement, except that
Iprobably don't want to trust you. You might think that "publish[ing] workingexploits into the wild" is some kind of heroic, altruistic act, but I think fewpeople would agree.
Full Disclosure is a common practice amongst security professionals whenthe upstream project is unwilling to rectify the problem. So yes I do
think
the practice of Full Disclosure is an altruistic act and often times the only thing that gets people who don't care to pull their head out of the sand and actually care.
You don't happen to be a random security professional, you are actually part of that upstream project and you have access to non-public (possibly confidential) data about its infrastructure, which gives you responsibilities towards your peers. I don't think I would be the only one to be angry if an infrastructure member starting publishing working exploits for unfixed vulnerabilities in the pdo infrastructure. It is a completely irresponsible way to act when you are part of a project or community. Regards Antoine.
On Jul 30, 2013, at 5:57 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Donald Stufft <donald <at> stufft.io> writes:
On Jul 30, 2013, at 3:01 AM, Antoine Pitrou <solipsis <at> pitrou.net> wrote:
I don't know what I'm supposed to infer from such a statement, except that
Iprobably don't want to trust you. You might think that "publish[ing] workingexploits into the wild" is some kind of heroic, altruistic act, but I think fewpeople would agree.
Full Disclosure is a common practice amongst security professionals whenthe upstream project is unwilling to rectify the problem. So yes I do
think
the practice of Full Disclosure is an altruistic act and often times the only thing that gets people who don't care to pull their head out of the sand and actually care.
You don't happen to be a random security professional, you are actually part of that upstream project and you have access to non-public (possibly confidential) data about its infrastructure, which gives you responsibilities towards your peers.
I don't think I would be the only one to be angry if an infrastructure member starting publishing working exploits for unfixed vulnerabilities in the pdo infrastructure. It is a completely irresponsible way to act when you are part of a project or community.
I don't really care if you'd be angry. The point of Full Disclosure (and it's cousin Responsible Disclosure) is to A) Inform everyone involved that they are taking a huge risk by using a particular thing and B) Provide incentive to people to fix their shit. If others are preventing fixes from landing then both reasons still apply wether the reporter is involved with the project or not. If I can find a vulnerability then so can someone else. Someone who won't inform people and will use it to maliciously attack people. If you feel I'd be overstepping my bounds then complain to my superiors, Richard/Nick on the packaging side of things and Noah on the Infrastructure team side of things. I'll continue to do what I feel best serves the community for as long as I have the ability to do so. Which I believe is work on improving these issues, fight and advocate for the important ones, accept defeat on less important ones, and, if necessary, issue a Full Disclosure. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
You don't happen to be a random security professional, you are actually part of that upstream project and you have access to non-public (possibly confidential) data about its infrastructure, which gives you responsibilities towards your peers.
I don't think I would be the only one to be angry if an infrastructure member starting publishing working exploits for unfixed vulnerabilities in the pdo infrastructure. It is a completely irresponsible way to act when you are
Donald Stufft <donald <at> stufft.io> writes: part
of a project or community.
I don't really care if you'd be angry.
Great to hear. This mindset is typical of many "security specialists": you're ready to tell everyone to go f*** themselves (I don't know how to voice this differently) if you think you have a higher mission to denounce some vulnerability.
The point of Full Disclosure (and it's cousin Responsible Disclosure) is to A) Inform everyone involved that they are taking a huge risk by using a particular thing and B) Provide incentive to people to fix their shit.
This does not necessarily involve publishing working exploits. By giving out code that can immediately attack and bring down the pdo infrastructure, you would be doing something else than merely "informing the public". (neither Bruce Schneier nor Wikipedia states that Full Disclosure implies publishing working exploits, btw. I suppose it means there is at the minimum some contention, rather than consensus, over the issue.)
If I can find a vulnerability then so can someone else.
You may (and probably do) have domain knowledge and inside knowledge that others don't.
If you feel I'd be overstepping my bounds then complain to my superiors, Richard/Nick on the packaging side of things and Noah on the Infrastructure team side of things.
"Superiors"? Are you making things up, or do you have an org chart to back that up? :-) (regardless, I would be surprised if any of those ordered *you* to publish an exploit, rather than take the responsibility of doing it themselves - or, rather, not doing it at all) Regards Antoine.
On Jul 30, 2013, at 7:40 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
You don't happen to be a random security professional, you are actually part of that upstream project and you have access to non-public (possibly confidential) data about its infrastructure, which gives you responsibilities towards your peers.
I don't think I would be the only one to be angry if an infrastructure member starting publishing working exploits for unfixed vulnerabilities in the pdo infrastructure. It is a completely irresponsible way to act when you are
Donald Stufft <donald <at> stufft.io> writes: part
of a project or community.
I don't really care if you'd be angry.
Great to hear. This mindset is typical of many "security specialists": you're ready to tell everyone to go f*** themselves (I don't know how to voice this differently) if you think you have a higher mission to denounce some vulnerability.
Keeping an issue secret doesn't make people more secure, it only prevents them from making an informed decision about the things they use.
The point of Full Disclosure (and it's cousin Responsible Disclosure) is to A) Inform everyone involved that they are taking a huge risk by using a particular thing and B) Provide incentive to people to fix their shit.
This does not necessarily involve publishing working exploits. By giving out code that can immediately attack and bring down the pdo infrastructure, you would be doing something else than merely "informing the public".
(neither Bruce Schneier nor Wikipedia states that Full Disclosure implies publishing working exploits, btw. I suppose it means there is at the minimum some contention, rather than consensus, over the issue.)
Partial disclosure is typically publishing enough information to allow people to make an informed choice but (hopefully) not enough information to allow others to attack it. Full Disclosure implies laying out the full details of the vulnerability. It doesn't necessarily mean wrapping it up in ready to run code but they almost always provide enough information so that anyone with a cursory understanding of programming (or the area it affects depending on how hard it is to replicate) can implement the attack and reproduce it. The problem with Partial Disclosure is that often times vendors/upstream/whatever will simply call the problem theoretical and still attempt to not fix things.
If I can find a vulnerability then so can someone else.
You may (and probably do) have domain knowledge and inside knowledge that others don't.
There's not really any inside knowledge. Everything is OSS. Domain knowledge? Sure, but I really hope we aren't relying on people not knowing enough about how the tooling works to exploit it.
If you feel I'd be overstepping my bounds then complain to my superiors, Richard/Nick on the packaging side of things and Noah on the Infrastructure team side of things.
"Superiors"? Are you making things up, or do you have an org chart to back that up? :-) (regardless, I would be surprised if any of those ordered *you* to publish an exploit, rather than take the responsibility of doing it themselves - or, rather, not doing it at all)
Nick is the BDFL delegate for packaging tooling and Richard is the same for PyPI and I generally consider my involvement as a PyPI admin to be under him. Noah is in charge of the Infra team of which I am a member.
Regards
Antoine.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 30 Jul 2013 21:41, "Antoine Pitrou" <solipsis@pitrou.net> wrote:
Donald Stufft <donald <at> stufft.io> writes:
You don't happen to be a random security professional, you are
of that upstream project and you have access to non-public (possibly confidential) data about its infrastructure, which gives you responsibilities towards your peers.
I don't think I would be the only one to be angry if an infrastructure member starting publishing working exploits for unfixed vulnerabilities in
infrastructure. It is a completely irresponsible way to act when you are part of a project or community.
I don't really care if you'd be angry.
Great to hear. This mindset is typical of many "security specialists": you're ready to tell everyone to go f*** themselves (I don't know how to voice this differently) if you think you have a higher mission to denounce some vulnerability.
The point of Full Disclosure (and it's cousin Responsible Disclosure) is to A) Inform everyone involved that they are taking a huge risk by using a particular thing and B) Provide incentive to
fix their shit.
This does not necessarily involve publishing working exploits. By giving out code that can immediately attack and bring down the pdo infrastructure, you would be doing something else than merely "informing the public".
(neither Bruce Schneier nor Wikipedia states that Full Disclosure implies publishing working exploits, btw. I suppose it means there is at the minimum some contention, rather than consensus, over the issue.)
If I can find a vulnerability then so can someone else.
You may (and probably do) have domain knowledge and inside knowledge that others don't.
If you feel I'd be overstepping my bounds then complain to my superiors, Richard/Nick on
packaging side of things and Noah on the Infrastructure team side of
actually part the pdo people to the things.
"Superiors"? Are you making things up, or do you have an org chart to
back that
up? :-)
(regardless, I would be surprised if any of those ordered *you* to
Effectively he does, yes. Richard is responsible for approving PyPI API changes (including PyPI specific PEPs), I'm BDFL delegate for other packaging PEPs and Noah has final say over the operation of the python.orginfrastructure. One or more of us are the ones that need to say yes on potentially controversial changes, so the responsibility for any mistakes ultimately lies with us, rather than Donald (and I'm greatly appreciative of the huge amount of work he is putting into improving the PyPI security story). publish an
exploit, rather than take the responsibility of doing it themselves - or, rather, not doing it at all)
If Donald informed us of a vulnerability and we refused to allow him (or anyone else) to take the necessary steps to close it, then he would be *completely* justified in publishing full details of the vulnerability, up to and including working exploit code. It won't come to that though, because we're taking this seriously and closing security holes as quickly as is feasible while still ensuring a reasonable level of backwards compatibility :) Cheers, Nick.
Regards
Antoine.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On Jul 30, 2013, at 8:06 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
If Donald informed us of a vulnerability and we refused to allow him (or anyone else) to take the necessary steps to close it, then he would be *completely* justified in publishing full details of the vulnerability, up to and including working exploit code.
It won't come to that though, because we're taking this seriously and closing security holes as quickly as is feasible while still ensuring a reasonable level of backwards compatibility :)
This basically. Maybe I'm not being clear because I have a headache and I'm reading too much into things because I'm sensitive to being shutdown on efforts to fix these things*. I don't expect with Nick, Richard, and Noah to ever need to do a Full Disclosure. I was only trying to be clear about what I consider my escalation path to be if a current, or near future vulnerability is forced to remain open. * I started trying to push for this ~2 years ago and got repeatedly shut down, for one reason or another. Which lead to to create Crate.io. It's only been relatively recently that I've been given permission to actually fix things. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Tue, Jul 30, 2013 at 8:43 AM, Donald Stufft <donald@stufft.io> wrote:
I have zero qualms about releasing a full disclosure along with working exploits into the wild for a security vulnerability that people block me on.
This is a moot point, as nobody is going to block anyone. The discussion is silly. //Lennart
On Jul 30, 2013, at 7:49 AM, Lennart Regebro <regebro@gmail.com> wrote:
On Tue, Jul 30, 2013 at 8:43 AM, Donald Stufft <donald@stufft.io> wrote:
I have zero qualms about releasing a full disclosure along with working exploits into the wild for a security vulnerability that people block me on.
This is a moot point, as nobody is going to block anyone. The discussion is silly.
//Lennart
*Shrug* If someone says people who don't care shouldn't have to suffer the consequences i assume that means they are advocating for no breakages in API (else the hypothetical person would be suffering). If you can't break fundamentally insecure things then that's a blockade on fixing the system. If that's not what the comment was proposing then I have no idea what it means. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 30 July 2013 07:08, Donald Stufft <donald@stufft.io> wrote:
On Jul 30, 2013, at 1:41 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Paul Moore <p.f.moore <at> gmail.com> writes:
Personally, none of the changes have detrimentally affected me, so my opinion is largely theoretical. But even I am getting a little
frustrated
by the constant claims that "what we have now is insecure and broken, and must be fixed ASAP".
FWIW, +1. You may be paranoid, but not everyone has to be (or suffer the consequences of it). Security issues should be fixed without breaking things in a hassle (which is the policy we followed e.g. for the ssl module, or hash randomization).
People are generally not paranoid until they've been successfully attacked. I *will* advocate and push for breaking things where security is concerned because regardless of if you care or not, a lot of people *do* care and the nature of the beast is that you're only as strong as the weakest link. This particular change wasn't an immediate vulnerability that I felt was urgent, hence why I've backed off on it when people were concerned about the backwards compat implications. I will not back off when it comes to issues that *do* have an immediate or near term issue, regardless of if some people don't care or not.
And in case it's not obvious, I think this is important. We need to have this sort of debate, certainly, but it won't happen without someone advocating (and implementing!) the changes, so many thanks for being that person and putting up with the flak. Paul
On 30 July 2013 17:33, Paul Moore <p.f.moore@gmail.com> wrote:
And in case it's not obvious, I think this is important. We need to have this sort of debate, certainly, but it won't happen without someone advocating (and implementing!) the changes, so many thanks for being that person and putting up with the flak.
Part of the idea of having myself and Richard in the approval chain for packaging ecosystem and PyPI changes is to encourage people to get angry at *us* for saying "Yes, let's do that" when things break, rather than getting mad at the people actually volunteering to do the work needed to improve the Python community's software distribution infrastructure. That's the plan, anyway, even if it isn't always all that effective in practice :( Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jul 30, 2013, at 4:10 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 30 July 2013 17:33, Paul Moore <p.f.moore@gmail.com> wrote:
And in case it's not obvious, I think this is important. We need to have this sort of debate, certainly, but it won't happen without someone advocating (and implementing!) the changes, so many thanks for being that person and putting up with the flak.
Part of the idea of having myself and Richard in the approval chain for packaging ecosystem and PyPI changes is to encourage people to get angry at *us* for saying "Yes, let's do that" when things break, rather than getting mad at the people actually volunteering to do the work needed to improve the Python community's software distribution infrastructure.
That's the plan, anyway, even if it isn't always all that effective in practice :(
Heh, I'm pretty good at getting yelled at :) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Tue, Jul 30, 2013 at 4:14 AM, Donald Stufft <donald@stufft.io> wrote:
Heh, I'm pretty good at getting yelled at :)
Nick is also pretty good at making people feel like he both knows and *cares* about their breakage, and isn't just dismissing their concerns as trivial or unimportant. Breakage isn't trivial or unimportant to the person who's yelling, so this is an important community-maintenance skill. It builds trust, and reduces the total amount of yelling.
On Jul 30, 2013, at 1:13 PM, PJ Eby <pje@telecommunity.com> wrote:
On Tue, Jul 30, 2013 at 4:14 AM, Donald Stufft <donald@stufft.io> wrote:
Heh, I'm pretty good at getting yelled at :)
Nick is also pretty good at making people feel like he both knows and *cares* about their breakage, and isn't just dismissing their concerns as trivial or unimportant. Breakage isn't trivial or unimportant to the person who's yelling, so this is an important community-maintenance skill. It builds trust, and reduces the total amount of yelling.
*shrug*, If I didn't care I would have made this change as soon as Nick said it was ok. Instead I declared I was going to and waited to make sure nobody else had any concerns. And once Holger said he did I said ok I won't do it. Maybe my mannerisms give the impression I don't but that's actually pretty far from the truth. For this particular change I originally created the pip commit that allowed it, and then again I created the setuptools commit, backporting hashlib into setuptools to support Python 2.4. I put a decent amount of effort into trying to make sure that nothing broke but in the end there were still concerns :) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Donald Stufft <donald <at> stufft.io> writes:
*shrug*, If I didn't care I would have made this change as soon as Nick said it was ok. Instead I declared I was going to and waited to make sure nobody else had any concerns. And once Holger said he did I said ok I won't do it. Maybe my mannerisms give the impression I don't but that's actually pretty far from the truth. For this particular change I
originally
created the pip commit that allowed it, and then again I created the setuptools commit, backporting hashlib into setuptools to support Python 2.4. I put a decent amount of effort into trying to make sure that nothing broke but in the end there were still concerns :)
You're right. I've been a bit harsh, but you're definitely doing some very useful things here. Regards Antoine.
On Tue, Jul 30, 2013 at 2:04 PM, Donald Stufft <donald@stufft.io> wrote:
On Jul 30, 2013, at 1:13 PM, PJ Eby <pje@telecommunity.com> wrote:
On Tue, Jul 30, 2013 at 4:14 AM, Donald Stufft <donald@stufft.io> wrote:
Heh, I'm pretty good at getting yelled at :)
Nick is also pretty good at making people feel like he both knows and *cares* about their breakage, and isn't just dismissing their concerns as trivial or unimportant. Breakage isn't trivial or unimportant to the person who's yelling, so this is an important community-maintenance skill. It builds trust, and reduces the total amount of yelling.
*shrug*, If I didn't care I would have made this change as soon as Nick said it was ok. Instead I declared I was going to and waited to make sure nobody else had any concerns. And once Holger said he did I said ok I won't do it. Maybe my mannerisms give the impression I don't but that's actually pretty far from the truth.
I did say "feel like". ;-) Nick usually gives more of an impression that he's thought about concerns raised before rejecting them; your responses often sound like, "Who cares about that?" Asking for suggestions, for example, would be good. Nick also rarely seems irritated by people's concerns or problems, whereas you sometimes seem in a big hurry to fix something today or this week that's been broken for years, without giving folks a while to get used to the idea. Often your proposals seem less like proposals than, "I've decided to do this, so deal with it". I'm not saying all this because I want to complain or yell at you; I'm saying it because I think you do care enough to know how you're coming across, at least to me. Our discussions have gotten heated in the past because my impression of your reaction to the concerns I raise often seems like, "I don't care about supporting [group of people affected], so neither should you." Perhaps the issue is just one of confusion. When I raise an issue about, say, Python 2.3 users (who are still downloading setuptools 0.6 releases, and presumably also using them), it's not because I expect *you* to change your plans to support them, but because I need to know how *I* can, if the issue arises. So I don't actually expect you to care about Python 2.3 users (again, as an example), but I do expect you to care about *me* supporting them. In the most recent situation, you did in fact point me to your awesome hashlib port, so I do know you *do* care to at least that extent. But the rhetoric that you sent both before and after the helpful bit seemed on the inflammatory side, as though I were crazy to be thinking of Python 2.3. Whether or not this is true ;-) -- it's not especially *helpful* to the discussion. If I may offer a suggestion, asking questions in response to objections is generally more useful than immediately arguing the relevance of the objection. First, it tells the objector that you're interested in what they have to say, and second, it may well help the objector understand that there isn't actually any real problem, and gives them an easier path to backing down and/or compromising, whereas a frontal assault tends to focus people on responding to you instead of reconsidering their objection. On the hashlib issue, for example, it actually occurred to me later that it's completely a non-issue because the actual hash scenario I was most concerned about *can't actually happen*. (MD5 hashes in code or dependency_links, used e.g. by setuptools itself to secure its own downloads. Changing PyPI won't affect these, duh.) It might've occurred to me sooner, though, if you'd actually asked what scenario I was worried about, instead of arguing about the relevance. This isn't to say that you're responsible for what I do or don't figure out; my point is simply that asking questions and inviting suggestions in response to people's objections will generally get you more thoughtful responses and more trust, and resolve issues sooner, with less arguing.
On Jul 30, 2013, at 4:01 PM, PJ Eby <pje@telecommunity.com> wrote:
On Tue, Jul 30, 2013 at 2:04 PM, Donald Stufft <donald@stufft.io> wrote:
On Jul 30, 2013, at 1:13 PM, PJ Eby <pje@telecommunity.com> wrote:
On Tue, Jul 30, 2013 at 4:14 AM, Donald Stufft <donald@stufft.io> wrote:
Heh, I'm pretty good at getting yelled at :)
Nick is also pretty good at making people feel like he both knows and *cares* about their breakage, and isn't just dismissing their concerns as trivial or unimportant. Breakage isn't trivial or unimportant to the person who's yelling, so this is an important community-maintenance skill. It builds trust, and reduces the total amount of yelling.
*shrug*, If I didn't care I would have made this change as soon as Nick said it was ok. Instead I declared I was going to and waited to make sure nobody else had any concerns. And once Holger said he did I said ok I won't do it. Maybe my mannerisms give the impression I don't but that's actually pretty far from the truth.
I did say "feel like". ;-)
Nick usually gives more of an impression that he's thought about concerns raised before rejecting them; your responses often sound like, "Who cares about that?" Asking for suggestions, for example, would be good. Nick also rarely seems irritated by people's concerns or problems, whereas you sometimes seem in a big hurry to fix something today or this week that's been broken for years, without giving folks a while to get used to the idea. Often your proposals seem less like proposals than, "I've decided to do this, so deal with it".
I'm not saying all this because I want to complain or yell at you; I'm saying it because I think you do care enough to know how you're coming across, at least to me. Our discussions have gotten heated in the past because my impression of your reaction to the concerns I raise often seems like, "I don't care about supporting [group of people affected], so neither should you."
Perhaps the issue is just one of confusion. When I raise an issue about, say, Python 2.3 users (who are still downloading setuptools 0.6 releases, and presumably also using them), it's not because I expect *you* to change your plans to support them, but because I need to know how *I* can, if the issue arises. So I don't actually expect you to care about Python 2.3 users (again, as an example), but I do expect you to care about *me* supporting them.
In the most recent situation, you did in fact point me to your awesome hashlib port, so I do know you *do* care to at least that extent. But the rhetoric that you sent both before and after the helpful bit seemed on the inflammatory side, as though I were crazy to be thinking of Python 2.3. Whether or not this is true ;-) -- it's not especially *helpful* to the discussion.
If I may offer a suggestion, asking questions in response to objections is generally more useful than immediately arguing the relevance of the objection. First, it tells the objector that you're interested in what they have to say, and second, it may well help the objector understand that there isn't actually any real problem, and gives them an easier path to backing down and/or compromising, whereas a frontal assault tends to focus people on responding to you instead of reconsidering their objection.
On the hashlib issue, for example, it actually occurred to me later that it's completely a non-issue because the actual hash scenario I was most concerned about *can't actually happen*. (MD5 hashes in code or dependency_links, used e.g. by setuptools itself to secure its own downloads. Changing PyPI won't affect these, duh.) It might've occurred to me sooner, though, if you'd actually asked what scenario I was worried about, instead of arguing about the relevance.
This isn't to say that you're responsible for what I do or don't figure out; my point is simply that asking questions and inviting suggestions in response to people's objections will generally get you more thoughtful responses and more trust, and resolve issues sooner, with less arguing.
Hrm. So I hear what you're saying and part of the problem is likely due to the history of where I tried to get a change through and then felt like all I was getting was stop energy and people wanting to keep the status quo which ultimately ended up preventing changes has lead me to view distutils-sig in more of an adversarial light then is probably appropriate for the distutils-sig of 2013 (versus the distutils-sig of 2011/2012). This is probably reflected in my tone and likely has others, such as yourself, respond similarly, pushing us further down that path. My thought process has become "Ok here's something that needs to happen, now how do I get distutils-sig not to prevent it from happening". This was again reflected in the Python 2.3 discussion because my immediate reaction and impression was that you were attempting to block the move from MD5 due to Python 2.3, which I felt 2.3 wasn't worth blocking enhancements to PyPI. The "snark" in my statements primarily came from that feeling of someone was trying to "shut down" an enhancement. I'd have to look to see when I first starting trying to advocate for change on catalog-sig but it was sometime in 2011 because I had finally gotten frustrated and went to "take my ball and go home" and start a competitor to PyPI with Crate.io on Dec 25th 2011. So there was a good 1-2 years there where the various mailing lists *were* an adversary towards getting fixes put into place. Now that I am able to get changes put into place I'm trying to go back and solve the various issues that I've found and I do have a feeling of urgency because I get concerned that this willingness to move forward and fix things is temporary and I want to fit in as many fixes as I can as quickly as I can before my window of opportunity closes. I've gone so far as begun sleeping less (not that i ever slept anything that one could call a normal sleep schedule) in order to buy more hours to work on these issues to get them fixed faster. So I do get frustrated and/or irritated and a lot of that comes from the history there because prior to Nick/Richard becoming BDFL Delegates there was never anyone willing to say "Yes, we can do that". I'm also generally a person who gets angry quickly (but gets un-angry quickly as well) which leads to it being harder for me not to snark or go on the offensive. There is a reason my personal domain is https://caremad.io/ ;) As far as how to fix it I don't have a particularly magic answer. I will try to be more mindful of my tone and that distutils-sig is likely not my adversary anymore as well as try to ask questions instead of arguing the relevance immediately. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 30 July 2013 21:58, Donald Stufft <donald@stufft.io> wrote:
As far as how to fix it I don't have a particularly magic answer. I will try to be more mindful of my tone and that distutils-sig is likely not my adversary anymore as well as try to ask questions instead of arguing the relevance immediately.
Thank you - both for the thoughtful response and the explanation of what is going on in your thinking. I certainly tend to think about possible issues rather than benefits, maybe more than I should as I don't personally have any particular problems with a rapid pace of change (except in some specific areas where I'm in so much of a minority that I don't expect my concerns to be a big driver in the grand scheme of things). I think I'm doing this to make sure people haven't missed potential issues I'm aware of, but I can easily imagine that this comes across as negative "stop energy". I'll try to stick to issues where I have genuine concerns, and not theoretical ones in future. And yes, it does feel like distutils-sig of 2013 is a much nicer place than it used to be. Thanks to everyone for working to keep that the case. Paul
On Jul 30, 2013, at 5:17 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 30 July 2013 21:58, Donald Stufft <donald@stufft.io> wrote: As far as how to fix it I don't have a particularly magic answer. I will try to be more mindful of my tone and that distutils-sig is likely not my adversary anymore as well as try to ask questions instead of arguing the relevance immediately.
Thank you - both for the thoughtful response and the explanation of what is going on in your thinking.
I certainly tend to think about possible issues rather than benefits, maybe more than I should as I don't personally have any particular problems with a rapid pace of change (except in some specific areas where I'm in so much of a minority that I don't expect my concerns to be a big driver in the grand scheme of things). I think I'm doing this to make sure people haven't missed potential issues I'm aware of, but I can easily imagine that this comes across as negative "stop energy". I'll try to stick to issues where I have genuine concerns, and not theoretical ones in future.
I don't actually mind even theoretical problems nor do I want people to feel like they need to coddle me because of my history with distutils-sig. That history and how It affects me is *my* problem. I think as long as we, including myself, approach theoretical problems as "Let's figure out if this theoretical problem is actually a problem, and if it is do we care about it" and not "here's a bunch of possible problems, we can't do this" then there will be no issues. It's true that theoretical problems can make me feel more like someones applying stop energy than concrete problems, but that's not a problem of the person bringing up that problem and more just it triggering old feelings from a time where I couldn't even get a SSL certificate trusted by most major browsers by default deployed ;). Not allowing those feelings to poison current efforts is on me.
And yes, it does feel like distutils-sig of 2013 is a much nicer place than it used to be. Thanks to everyone for working to keep that the case.
Paul
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Donald Stufft <donald <at> stufft.io> writes:
to PyPI. The "snark" in my statements primarily came from that feeling of someone was trying to "shut down" an enhancement.
To be fair, the tone you've taken (which can rub people up the wrong way) hasn't just been over long-standing security issues. I've also felt uncomfortable with some of your responses relating to the PEP 426 discussion around the JSON schema for requirements, where there is not the same level of urgency, nor valid reason to believe that you're being thwarted somehow. If someone's approach comes off as unduly dismissive (even if not meant that way), it makes it harder to engage in constructive discussion. I recognise that it's hard communicating with people remotely, many of whom one might never have met. It's easy to miss nuance and misconstrue turns of phrase. It's best to tread gingerly, else too much time is spent smoothing ruffled feathers at the expense of getting real work done. Regards, Vinay Sajip
On Jul 30, 2013, at 6:19 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Donald Stufft <donald <at> stufft.io> writes:
to PyPI. The "snark" in my statements primarily came from that feeling of someone was trying to "shut down" an enhancement.
To be fair, the tone you've taken (which can rub people up the wrong way) hasn't just been over long-standing security issues. I've also felt uncomfortable with some of your responses relating to the PEP 426 discussion around the JSON schema for requirements, where there is not the same level of urgency, nor valid reason to believe that you're being thwarted somehow.
I never meant to claim it was simply over security issues. I'm certainly more "energetic" about advocating for security issues and they frame the backdrop for a lot of my feelings with distutils-sig as an adversary rather than an ally . The simple fact is that every disagreement, or even simple suggestion, is framed against that feeling leading towards the Me vs Them tone.
If someone's approach comes off as unduly dismissive (even if not meant that way), it makes it harder to engage in constructive discussion. I recognise that it's hard communicating with people remotely, many of whom one might never have met. It's easy to miss nuance and misconstrue turns of phrase. It's best to tread gingerly, else too much time is spent smoothing ruffled feathers at the expense of getting real work done.
Regards,
Vinay Sajip
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Tue, Jul 30, 2013 at 4:58 PM, Donald Stufft <donald@stufft.io> wrote:
Hrm.
So I hear what you're saying and part of the problem is likely due to the history of where I tried to get a change through and then felt like all I was getting was stop energy and people wanting to keep the status quo which ultimately ended up preventing changes has lead me to view distutils-sig in more of an adversarial light then is probably appropriate for the distutils-sig of 2013 (versus the distutils-sig of 2011/2012). This is probably reflected in my tone and likely has others, such as yourself, respond similarly, pushing us further down that path. My thought process has become "Ok here's something that needs to happen, now how do I get distutils-sig not to prevent it from happening".
Thanks for the thoughtful response. I appreciate it. I also want to just throw in one extra piece of information for you and anybody else reading this: 99% of "stop energy" doesn't happen because people actively want to prevent progress or frustrate other people. It simply happens when people notice a problem but don't have as much personal stake in your goal as they do in not experiencing the problem they will experience (or perceive they will), from the proposed change. When you look at it from this perspective, it's easier to understand that the way to fix this is with more engagement on their part, which can only be gotten by engagement on your part. When I first proposed WSGI, the initial reaction of Web-SIG was pretty negative. "Stop energy" if you will. Things only moved forward once I was able to channel the energy of objections into offering solutions. It's helpful to remember that asking, "okay, so how you would recommend I do it?" *doesn't* obligate you to actually follow all of the recommendations you get. (Especially since some of them will be mutually contradictory!) Anyway, I guess what I'm saying is that people lacking enthusiasm for your goals is not really them trying to stop you. In fact, objections are a positive thing: it means you got somebody's attention. The next step is to leverage that attention into actually getting help, or at least more constructive input. ;-) It's true that some individuals will never provide really helpful input. In the WSGI effort, there were people whose advice I never took, because their goals were directed entirely opposite to where I wanted to go. But I remained engaged until it was mutually clear (or at least I thought it was) that our goals were different, and didn't try to persuade them to go in the same direction. Such attempts at persuasion are pretty much a waste of time, and a big energy drain. Consensus-building is something that you do with people who have at least some goals in common, so it's best to focus on finding out what you *do* agree on.
This was again reflected in the Python 2.3 discussion because my immediate reaction and impression was that you were attempting to block the move from MD5 due to Python 2.3, which I felt 2.3 wasn't worth blocking enhancements to PyPI. The "snark" in my statements primarily came from that feeling of someone was trying to "shut down" an enhancement.
Right. In such a case, a question you could ask is, "Do you agree in general that we should move to a better hash at some point in the future?", because then the disagreement can be narrowed down to timeframe, migration or deprecation process, etc. The truth is, I had no intention of "blocking the move", I had concerns I wanted addressed about the impact, timing and process. (Actually, I originally just noticed a couple of errors in what you'd laid out as the current state of things, and wanted to make sure they were included in the assessment.) The point is, if somebody doesn't have *any* common ground with you, it's unlikely they're even talking to you. At the very least, if they're talking with you about PyPI, they must care about PyPI, even if they care about different things than you, or with different relative priorities. ;-)
As far as how to fix it I don't have a particularly magic answer. I will try to be more mindful of my tone and that distutils-sig is likely not my adversary anymore as well as try to ask questions instead of arguing the relevance immediately.
Again, thank you. And hopefully, remember that probably nobody was intentionally being your adversary before, either. As the old adage says, the best way to destroy your enemies is to make friends with them. ;-) And we do that by focusing on common ground, and inviting participation. (This is again not to say that I've been 100% Mr. Wonderful myself; I know I haven't. But the community's best consensus-building happens when somebody is doing the tough work of engaging with all parties. Sometimes this doesn't happen, alas; back when I was developing setuptools there just weren't enough people interested in the problems available on Distutils-SIG to build any sort of consensus on the solutions, so I *had* to go run with the ball myself. I'm really happy that there is now BOTH a quorum of interested parties who understand the problems, AND a few leaders able to drive the consensus-building and actual development. If only we had done this ten years ago, setuptools might not have been necessary. Well, actually, that's probably not true: without *something* existing first, that quorum of people who understand the problem wouldn't have existed back then. But you know what I mean.)
On Aug 2, 2013, at 4:28 PM, PJ Eby <pje@telecommunity.com> wrote:
I also want to just throw in one extra piece of information for you and anybody else reading this: 99% of "stop energy" doesn't happen because people actively want to prevent progress or frustrate other people. It simply happens when people notice a problem but don't have as much personal stake in your goal as they do in not experiencing the problem they will experience (or perceive they will), from the proposed change.
One of the biggest things I viewed as stop energy was how impossible it seemed to get a SSL certificate that was trusted in most major browsers to be used on PyPI. There was never any reason given that we couldn't move to one, just simply "we already have a certificate" without addressing the reasons I was trying to get a different one. Beyond that it was other random proposals that it felt like people put in the minimal amount of effort in order to find a reason to say no to it. Then if I addressed that particular concern, they'd again seem to put in the minimal amount of effort to find a reason to say no. It felt like the goal posts were just constantly being moved to slightly further away each time. That sort of behavior is mostly what I mean by "stop energy". It's entirely frustrating and it feels like it's entire purpose is to dissuade someone from changing the status quo, wether that was the intention or not. Everything else you said has been noted :) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Hi Donald, On Tue, Jul 30, 2013 at 14:04 -0400, Donald Stufft wrote:
On Jul 30, 2013, at 1:13 PM, PJ Eby <pje@telecommunity.com> wrote:
On Tue, Jul 30, 2013 at 4:14 AM, Donald Stufft <donald@stufft.io> wrote:
Heh, I'm pretty good at getting yelled at :)
Nick is also pretty good at making people feel like he both knows and *cares* about their breakage, and isn't just dismissing their concerns as trivial or unimportant. Breakage isn't trivial or unimportant to the person who's yelling, so this is an important community-maintenance skill. It builds trust, and reduces the total amount of yelling.
*shrug*, If I didn't care I would have made this change as soon as Nick said it was ok. Instead I declared I was going to and waited to make sure nobody else had any concerns. And once Holger said he did I said ok I won't do it. Maybe my mannerisms give the impression I don't but that's actually pretty far from the truth. For this particular change I originally created the pip commit that allowed it, and then again I created the setuptools commit, backporting hashlib into setuptools to support Python 2.4. I put a decent amount of effort into trying to make sure that nothing broke but in the end there were still concerns :)
For the record, i am all for putting generic hash support into the installers and maybe prepare for an eventual change to make PyPI serve sha256 hashes. However, to me it's not clear if such a move may become obsolete through the potential advent of TUF. My original objection reason was tied to generally pushing for more focus on backward-compatibility. I am grateful that several people including you, Nick and Jannis acknowledged the point. best, holger
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On Jul 31, 2013, at 1:38 AM, holger krekel <holger@merlinux.eu> wrote:
Hi Donald,
On Tue, Jul 30, 2013 at 14:04 -0400, Donald Stufft wrote:
On Jul 30, 2013, at 1:13 PM, PJ Eby <pje@telecommunity.com> wrote:
On Tue, Jul 30, 2013 at 4:14 AM, Donald Stufft <donald@stufft.io> wrote:
Heh, I'm pretty good at getting yelled at :)
Nick is also pretty good at making people feel like he both knows and *cares* about their breakage, and isn't just dismissing their concerns as trivial or unimportant. Breakage isn't trivial or unimportant to the person who's yelling, so this is an important community-maintenance skill. It builds trust, and reduces the total amount of yelling.
*shrug*, If I didn't care I would have made this change as soon as Nick said it was ok. Instead I declared I was going to and waited to make sure nobody else had any concerns. And once Holger said he did I said ok I won't do it. Maybe my mannerisms give the impression I don't but that's actually pretty far from the truth. For this particular change I originally created the pip commit that allowed it, and then again I created the setuptools commit, backporting hashlib into setuptools to support Python 2.4. I put a decent amount of effort into trying to make sure that nothing broke but in the end there were still concerns :)
For the record, i am all for putting generic hash support into the installers and maybe prepare for an eventual change to make PyPI serve sha256 hashes. However, to me it's not clear if such a move may become obsolete through the potential advent of TUF.
pip has had generic hash support since 1.2, setuptools since 0.9, and since I believe zc.buildout just lets setuptools do the downloads so it'd be when using 0.9 for it. The idea was this was an easy win security hardening. Something like TUF would more or less obsolete it but TUF isn't on the immediate radar.
My original objection reason was tied to generally pushing for more focus on backward-compatibility. I am grateful that several people including you, Nick and Jannis acknowledged the point.
best, holger
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Jul 29, 2013, at 1:01 PM, Tres Seaver <tseaver@palladion.com> wrote:
Signed PGP part On 07/29/2013 10:51 AM, Donald Stufft wrote:
I've personally gotten or seen more complaints over the naming of a variable in the config file then I have over any changes we've made. The runner up to that is the fallout from switching to requiring verified SSL.
The past few months have generated a *lot* of teeth-gnashing / hair-pulling, especially among "downstream" developers (those unlikely to be reading this SIG):
- - HTTPS-only PyPY
I've seen very few people complain about this. It's probably the largest issue in either of the projects I'm involved with.
- - Distribute / setuptools merge, e.g. cratering folks who use a distro-managed 'python-distribute' package.
This is the biggest issue. I wasn't involved and it could have been handled better sure.
- - Pip's new backward-imcompatible "final releases by default".
Again this negatively affects very few packages and it positively affects a number of people who were bit by the fact that pip does this. SQLAlchemy was one of the ones that were offhand. People don't realize the tools *didn't* do this and often times caused pain inadvertently for the down stream developers.
I think we are going to be in a much better place for all that, but let's not deny the pain involved for *everybody* in getting there.
Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
- - Distribute / setuptools merge, e.g. cratering folks who use a distro-managed 'python-distribute' package.
This is the biggest issue. I wasn't involved and it could have been handled better sure.
what issue are we talking about exactly? I'm aware of the "Import setuptools" problem during upgrades from distribute to setuptools: https://github.com/pypa/pip/issues/1064 pip-1.4 does prevent that now (released last week) but you specifically mentioned the linux distro name, "python-distribute"? I'm guessing maybe you're concerned with the scenario of how upgrading some project (that depends on setuptools/distribute) forces an unintended upgrade of distribute on a system where distribute is system-managed. Is there a link to a tracker issue for that somewhere? 3 solutions to ponder: 1) unreleasing the distribute-0.7.3 wrapper, and making any upgrades from distribute to setuptools manual. 2) changing pip's -U logic to not recursively upgrade satisfied dependencies, but that's a *big* change (it's a long story, but there are tentative plans in 1.5 for changing it) 3) in a patch release, include a special case for distribute to *not* upgrade if satisfied (when distribute is a dependency in the requirement set). Marcus
- - Distribute / setuptools merge, e.g. cratering folks who use a distro-managed 'python-distribute' package.
This is the biggest issue. I wasn't involved and it could have been handled better sure.
what issue are we talking about exactly?
I'm aware of the "Import setuptools" problem during upgrades from distribute to setuptools: https://github.com/pypa/pip/issues/1064 pip-1.4 does prevent that now (released last week)
but you specifically mentioned the linux distro name, "python-distribute"? I'm guessing maybe you're concerned with the scenario of how upgrading some project (that depends on setuptools/distribute) forces an unintended upgrade of distribute on a system where distribute is system-managed. Is there a link to a tracker issue for that somewhere?
but on 2nd thought, this is not a new problem. this would always been happening, just not involving a transition to setuptools. so I imagine, your concern was just the "Import" error?
On Mon, Jul 29, 2013 at 8:17 PM, Donald Stufft <donald@stufft.io> wrote:
- - Distribute / setuptools merge, e.g. cratering folks who use a distro-managed 'python-distribute' package.
This is the biggest issue. I wasn't involved and it could have been handled better sure.
I'm not convinced. The use-cases and ways these things can be installed in conjunction with things like system packages, virtualenv and buildout in various configurations and various python versions are mindboggling. Some of them were not initially tested, probably mostly because nobody realized the finer details of what would happen in these sometimes fairly obscure cases. The problems have, afaik, now been fixed, and this was done reasonably quickly IMO. OK, of course it *could* have been handled better, but I don't think it was handled badly. The current situation is a huge mess. Untangling it *will* be tricky. //Lennart
Donald Stufft <donald <at> stufft.io> writes:
I'm going to go ahead and make this change unless someone comes out and contests moving PyPI to SHA256. I'll give it a bit to make sure no one does have an issue with the move.
Your proposal is a little light on specification, unless I've missed it. For example: * How exactly will download URLs change? One would assume they'd have a fragment of 'sha256=...', where they currently have 'md5=...', but can you confirm this? * PyPI's XML-RPC API provides MD5 hashes in result dictionaries using a key 'md5_digest'. How will these result dictionaries change under your proposal? * PyPI's web interface has actions such as 'show_md5', will these stop working? (By actions, I mean query strings such as ':action=show_md5'.) Will new actions be added? I'm not familiar with the change process for PyPI - what is the workflow? For example, are patches posted for review? Regards, Vinay Sajip
On Jul 28, 2013, at 8:31 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Donald Stufft <donald <at> stufft.io> writes:
I'm going to go ahead and make this change unless someone comes out and contests moving PyPI to SHA256. I'll give it a bit to make sure no one does have an issue with the move.
Your proposal is a little light on specification, unless I've missed it. For example:
* How exactly will download URLs change? One would assume they'd have a fragment of 'sha256=...', where they currently have 'md5=...', but can you confirm this?
Yes they will change to have #sha256=…. instead of #md5=...
* PyPI's XML-RPC API provides MD5 hashes in result dictionaries using a key 'md5_digest'. How will these result dictionaries change under your proposal?
Here we are a little more flexible. I can leave the md5_digest key there and simply add a sha256_digest key.
* PyPI's web interface has actions such as 'show_md5', will these stop working? (By actions, I mean query strings such as ':action=show_md5'.) Will new actions be added?
Again more flexible. I can simply add a show_sha256 action.
I'm not familiar with the change process for PyPI - what is the workflow? For example, are patches posted for review?
Typically it's left up to us. We often just work and deploy changes without any review process but we can (and I have) get reviews before hand. The biggest problem with Reviews is PyPI is a very messy codebase with very few people who understand it so the pool of developers qualified to review the code is very small. On the warehouse side of things I don't develop directly on master everything comes through pull requests and while there's no formal review process A number of folks have been checking my PR's and making comments as they deemed fit.
Regards,
Vinay Sajip
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
From: Donald Stufft <donald@stufft.io>
Yes they will change to have #sha256=…. instead of #md5=... [snip] Here we are a little more flexible. I can leave the md5_digest key there and simply add a sha256_digest key. [snip] Again more flexible. I can simply add a show_sha256 action. [snip] Typically it's left up to us. We often just work and deploy changes without [snip]
Thanks for the update, it'll help me to update distlib when the time comes. Regards, Vinay Sajip
On Jul 28, 2013, at 3:13 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
From: Donald Stufft <donald@stufft.io>
Yes they will change to have #sha256=…. instead of #md5=... [snip] Here we are a little more flexible. I can leave the md5_digest key there and simply add a sha256_digest key. [snip] Again more flexible. I can simply add a show_sha256 action. [snip] Typically it's left up to us. We often just work and deploy changes without [snip]
Thanks for the update, it'll help me to update distlib when the time comes.
Regards,
Vinay Sajip
I sugggest distlib take the same approach as pip and setuptools. Allow any #hashname=hash so that it will function with whatever hash people use. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
participants (16)
-
Alex Gaynor -
Antoine Pitrou -
Christian Heimes -
Donald Stufft -
holger krekel -
Jim Fulton -
Lennart Regebro -
Marcus Smith -
Nick Coghlan -
Noah Kantrowitz -
Paul Moore -
PJ Eby -
Ronald Oussoren -
Tres Seaver -
Vinay Sajip -
zooko