Status report on PyPI+pip+TUF
Hello Nick and the PyPI community, This is a brief status report on the integration of PyPI and pip with TUF. (A quick reminder: TUF is a general "plug-n-play" update framework designed to introduce usable security to community software repositories such as PyPI. If you think of PyPI as HTTP, then TUF is like adding SSL, and more, to HTTP. More information may be found at [https://www.updateframework.com/].) Firstly, thanks to the generous funding of the National Science Foundation, we are pleased to introduce the addition of a full-time developer, Vladimir Diaz, to our team. Vladimir has been instrumental to the development of TUF, and we are excited to have him join us full-time. (Now we do not just have one PhD student who works on TUF when he is not busy working on other projects!) We are also happy to have a few interns --- Zane Fisher, Tian Tian, John Ward, and Yuyu Zheng --- on board for the summer. Since the security attacks on the Python wiki infrastructure earlier this year, we have been closely following Distutils-SIG to see what we could do to help secure PyPI. We use Python heavily in all of our projects, and would love to help in any way we can. Here is what we have done: ========================== 1. At PyCon 2013, we showed that pip needs very little modification to work with a TUF-enabled PyPI mirror. 2. Soon after (during the spring break), we wrote automation to build a TUF-secured PyPI mirror (which is indistinguishable from any other PyPI mirror except that it has signed metadata about all of the files on PyPI). 3. At the same time, thanks to efforts of Konstantin Andrianov, we also wrote a lot of unit and integration tests to show the attacks that are possible without TUF and impossible with TUF. 4. After that, we started investigating the most efficient way to build TUF metadata for PyPI. We found that requiring a separate key for every package on PyPI may sound like a good idea, but besides generating too much metadata, this scheme also makes key management difficult. Here is what we are doing now: ============================== We are designing a usable key management scheme, coupled with efficient generation and download of metadata, which we think should make for a smooth integration of PyPI with TUF. We are actively working on this and think that we are almost there. As a conservative estimate, we do not believe that this should take longer than two weeks. Here is what we are going to do next: ===================================== In about a month, we will present to you a demonstration of a PyPI mirror and a pip client which are robust against entire classes of security attacks. We welcome you then to try our demo, be really critical of it and tell us what you think about what we could do better. Our goal with TUF is to provide a framework that works with as many software community repositories as possible and that secures as many users as possible. More details on our development are available at our mailing list: https://groups.google.com/forum/#!forum/theupdateframework We hope this gives you a good idea of the current status of integrating TUF with PyPI and pip. Let us know if you have questions. Thanks, The TUF team
Hi Trishank, thanks for the high level overview. Do you have a current web page with more detailed technical info with respect to PyPI/TUF? best, holger On Wed, Jul 31, 2013 at 07:27 -0400, Trishank Karthik Kuppusamy wrote:
Hello Nick and the PyPI community,
This is a brief status report on the integration of PyPI and pip with TUF.
(A quick reminder: TUF is a general "plug-n-play" update framework designed to introduce usable security to community software repositories such as PyPI. If you think of PyPI as HTTP, then TUF is like adding SSL, and more, to HTTP. More information may be found at [https://www.updateframework.com/].)
Firstly, thanks to the generous funding of the National Science Foundation, we are pleased to introduce the addition of a full-time developer, Vladimir Diaz, to our team. Vladimir has been instrumental to the development of TUF, and we are excited to have him join us full-time. (Now we do not just have one PhD student who works on TUF when he is not busy working on other projects!) We are also happy to have a few interns --- Zane Fisher, Tian Tian, John Ward, and Yuyu Zheng --- on board for the summer.
Since the security attacks on the Python wiki infrastructure earlier this year, we have been closely following Distutils-SIG to see what we could do to help secure PyPI. We use Python heavily in all of our projects, and would love to help in any way we can.
Here is what we have done: ==========================
1. At PyCon 2013, we showed that pip needs very little modification to work with a TUF-enabled PyPI mirror.
2. Soon after (during the spring break), we wrote automation to build a TUF-secured PyPI mirror (which is indistinguishable from any other PyPI mirror except that it has signed metadata about all of the files on PyPI).
3. At the same time, thanks to efforts of Konstantin Andrianov, we also wrote a lot of unit and integration tests to show the attacks that are possible without TUF and impossible with TUF.
4. After that, we started investigating the most efficient way to build TUF metadata for PyPI. We found that requiring a separate key for every package on PyPI may sound like a good idea, but besides generating too much metadata, this scheme also makes key management difficult.
Here is what we are doing now: ==============================
We are designing a usable key management scheme, coupled with efficient generation and download of metadata, which we think should make for a smooth integration of PyPI with TUF. We are actively working on this and think that we are almost there. As a conservative estimate, we do not believe that this should take longer than two weeks.
Here is what we are going to do next: =====================================
In about a month, we will present to you a demonstration of a PyPI mirror and a pip client which are robust against entire classes of security attacks. We welcome you then to try our demo, be really critical of it and tell us what you think about what we could do better. Our goal with TUF is to provide a framework that works with as many software community repositories as possible and that secures as many users as possible.
More details on our development are available at our mailing list: https://groups.google.com/forum/#!forum/theupdateframework
We hope this gives you a good idea of the current status of integrating TUF with PyPI and pip. Let us know if you have questions.
Thanks, The TUF team
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Hello Holger, On 07/31/2013 08:13 AM, holger krekel wrote:
thanks for the high level overview. Do you have a current web page with more detailed technical info with respect to PyPI/TUF?
Good question! I think it is a good idea to put up a "PyPI+pip+TUF current status" page on our web site, but in the meantime, here are a few links which should point you in the right direction: 1. pip+TUF: we use the interposition technique [https://github.com/theupdateframework/tuf/tree/master/tuf/interposition] to minimally modify pip [https://github.com/theupdateframework/pip/compare/tuf] to talk to a TUF-secured PyPI mirror. 2. PyPI+TUF: we use automation to build a testbed for investigating different key management and metadata schemes to secure PyPI [https://github.com/theupdateframework/pypi.updateframework.com]. (Note: at the time of writing, the automation is slightly out-of-date with our work-in-progress.) 3. These two links should give you a good picture, but they will not give you a complete one. We will formally write about what we mean with our upcoming key management as well as metadata generation and download scheme. Let me start a document and get back to you on that. Thanks, Trishank
Hi Trishank, On Wed, Jul 31, 2013 at 10:02 -0400, Trishank Karthik Kuppusamy wrote:
Hello Holger,
On 07/31/2013 08:13 AM, holger krekel wrote:
thanks for the high level overview. Do you have a current web page with more detailed technical info with respect to PyPI/TUF?
Good question! I think it is a good idea to put up a "PyPI+pip+TUF current status" page on our web site, but in the meantime, here are a few links which should point you in the right direction:
1. pip+TUF: we use the interposition technique [https://github.com/theupdateframework/tuf/tree/master/tuf/interposition] to minimally modify pip [https://github.com/theupdateframework/pip/compare/tuf] to talk to a TUF-secured PyPI mirror.
2. PyPI+TUF: we use automation to build a testbed for investigating different key management and metadata schemes to secure PyPI [https://github.com/theupdateframework/pypi.updateframework.com]. (Note: at the time of writing, the automation is slightly out-of-date with our work-in-progress.)
3. These two links should give you a good picture, but they will not give you a complete one. We will formally write about what we mean with our upcoming key management as well as metadata generation and download scheme. Let me start a document and get back to you on that.
thanks for the links. They contain code instructions but i am not sure i get the overall picture yet. Do you have a whitepaper or overview describing the approach wrt to PyPI? If i understand the code correctly, you are implementing key signing, verification and revocation through calling openssl library functions. Have you considered just invoking or interfacing with "gpg"? On a minor note, for creating a pypi mirror it's better to use bandersnatch instead of pep381 (i am refering to this here: https://github.com/theupdateframework/pip/wiki/PyPI-over-TUF#mirror-pypi ) Lastly, maybe the advertisement that "TUF is like the 'S' in HTTPS" is not really a good advertisement given the several currently discussed problems with HTTPS, the most recent one being the BREACH attack: http://arstechnica.com/security/2013/08/gone-in-30-seconds-new-attack-plucks... :) cheers, holger
Thanks, Trishank
On 08/01/2013 05:02 PM, holger krekel wrote:
thanks for the links. They contain code instructions but i am not sure i get the overall picture yet. Do you have a whitepaper or overview describing the approach wrt to PyPI?
We do, but it is not up-to-date with our latest thoughts. We will rectify this soon enough: https://docs.google.com/document/d/1sHMhgrGXNCvBZdmjVJzuoN5uMaUAUDWBmn3jo7vx...
If i understand the code correctly, you are implementing key signing, verification and revocation through calling openssl library functions. Have you considered just invoking or interfacing with "gpg"?
Yes, that is an option we could decide to implement, along with other cryptography libraries. I think we chose to start with interfacing with OpenSSL because it is generic, time-tested to be secure and available on many platforms. TUF does not need to exclusively depend on either OpenSSL, GPG or anything else: we can extend it to use what is available.
On a minor note, for creating a pypi mirror it's better to use bandersnatch instead of pep381 (i am refering to this here: https://github.com/theupdateframework/pip/wiki/PyPI-over-TUF#mirror-pypi )
Thanks for the tip. Indeed, we do use bandersnatch [https://github.com/theupdateframework/pypi.updateframework.com/blob/master/s...]. That wiki entry points to an old set of instructions that we will remove soon.
Lastly, maybe the advertisement that "TUF is like the 'S' in HTTPS" is not really a good advertisement given the several currently discussed problems with HTTPS, the most recent one being the BREACH attack: http://arstechnica.com/security/2013/08/gone-in-30-seconds-new-attack-plucks...
I see what you are saying, but I do not think that it follows that TUF works like SSL :) Perhaps we can think of a better metaphor, but the idea we wanted to convey is that TUF is like a plug-in you simply drop into your software update system, and voilà, you get security for relatively little work. Let us know if you have more questions. In the meantime, we are busy designing our key management scheme for PyPI+TUF (which I think would highly interest you), so please bear with us while we hammer that out over this week.
participants (2)
-
holger krekel -
Trishank Karthik Kuppusamy