Hi,
I just read this on reddit[0], a thread asking if PyPI packages are audited and somebody pointed the `python-nation`[1] which is a harmful and useless module, installing itself and sending the `/etc/passwd` content to external endpoint.
The app receiving the data is hosted at http://python-nation.herokuapp.com
and as the PSF mission [2] says
The mission of the Python Software Foundation is to promote, protect, and advance the Python programming language
I wonder if there are some workgroup at PSF to handle this? and not only the specific case of `python-nation` which should be deleted and the user banned maybe, But also to handle the audit of other packages?
[0] https://www.reddit.com/r/Python/comments/697da2/does_ pypi_review_code_thats_uploaded/ [1] https://www.reddit.com/r/Python/comments/697da2/does_ pypi_review_code_thats_uploaded/dh4uyf8/ [2] https://www.python.org/psf/mission/
Cheers,
Interesting detail, the mentioned package https://pypi.python.org/pypi/python-nation/1.0.1 was created and uploaded by Jacob Kaplan Moss, so I guess this is intended to be a POC, to show PyPI vulnerabilities or some Infosec experiment.
On Thu, May 4, 2017 at 8:41 PM, Bruno Rocha rochacbruno@gmail.com wrote:
Hi,
I just read this on reddit[0], a thread asking if PyPI packages are audited and somebody pointed the `python-nation`[1] which is a harmful and useless module, installing itself and sending the `/etc/passwd` content to external endpoint.
The app receiving the data is hosted at http://python-nation.herokuapp.com
and as the PSF mission [2] says
The mission of the Python Software Foundation is to promote, protect, and advance the Python programming language
I wonder if there are some workgroup at PSF to handle this? and not only the specific case of `python-nation` which should be deleted and the user banned maybe, But also to handle the audit of other packages?
[0] https://www.reddit.com/r/Python/comments/697da2/does_pyp i_review_code_thats_uploaded/ [1] https://www.reddit.com/r/Python/comments/697da2/does_pyp i_review_code_thats_uploaded/dh4uyf8/ [2] https://www.python.org/psf/mission/
Cheers,
--
*Bruno Rocha - @rochacbruno http://twitter.com/rochacbruno* http://brunorocha.org
That is a great observation Bruno!
-Jackie
On Thu, May 4, 2017 at 8:08 PM, Bruno Rocha rochacbruno@gmail.com wrote:
Interesting detail, the mentioned package https://pypi.python. org/pypi/python-nation/1.0.1 was created and uploaded by Jacob Kaplan Moss, so I guess this is intended to be a POC, to show PyPI vulnerabilities or some Infosec experiment.
On Thu, May 4, 2017 at 8:41 PM, Bruno Rocha rochacbruno@gmail.com wrote:
Hi,
I just read this on reddit[0], a thread asking if PyPI packages are audited and somebody pointed the `python-nation`[1] which is a harmful and useless module, installing itself and sending the `/etc/passwd` content to external endpoint.
The app receiving the data is hosted at http://python-nation.heroku app.com
and as the PSF mission [2] says
The mission of the Python Software Foundation is to promote, protect, and advance the Python programming language
I wonder if there are some workgroup at PSF to handle this? and not only the specific case of `python-nation` which should be deleted and the user banned maybe, But also to handle the audit of other packages?
[0] https://www.reddit.com/r/Python/comments/697da2/does_pyp i_review_code_thats_uploaded/ [1] https://www.reddit.com/r/Python/comments/697da2/does_pyp i_review_code_thats_uploaded/dh4uyf8/ [2] https://www.python.org/psf/mission/
Cheers,
--
*Bruno Rocha - @rochacbruno http://twitter.com/rochacbruno* http://brunorocha.org
--
*Bruno Rocha - @rochacbruno http://twitter.com/rochacbruno* http://brunorocha.org
PSF-Community mailing list PSF-Community@python.org https://mail.python.org/mailman/listinfo/psf-community
I'm not sure what effective package review would look like here. Perhaps we could establish an entity to screen packages on an opt-in basis, but I don't know if we have the resources/people for this. Automated code screening could and probably would miss the python nation example due to the unorthodox use of compressed instructions. Does anyone have any ideas?
-Ryan Birmingham
On 4 May 2017 at 20:08, Bruno Rocha rochacbruno@gmail.com wrote:
Interesting detail, the mentioned package https://pypi.python. org/pypi/python-nation/1.0.1 was created and uploaded by Jacob Kaplan Moss, so I guess this is intended to be a POC, to show PyPI vulnerabilities or some Infosec experiment.
On Thu, May 4, 2017 at 8:41 PM, Bruno Rocha rochacbruno@gmail.com wrote:
Hi,
I just read this on reddit[0], a thread asking if PyPI packages are audited and somebody pointed the `python-nation`[1] which is a harmful and useless module, installing itself and sending the `/etc/passwd` content to external endpoint.
The app receiving the data is hosted at http://python-nation.heroku app.com
and as the PSF mission [2] says
The mission of the Python Software Foundation is to promote, protect, and advance the Python programming language
I wonder if there are some workgroup at PSF to handle this? and not only the specific case of `python-nation` which should be deleted and the user banned maybe, But also to handle the audit of other packages?
[0] https://www.reddit.com/r/Python/comments/697da2/does_pyp i_review_code_thats_uploaded/ [1] https://www.reddit.com/r/Python/comments/697da2/does_pyp i_review_code_thats_uploaded/dh4uyf8/ [2] https://www.python.org/psf/mission/
Cheers,
--
*Bruno Rocha - @rochacbruno http://twitter.com/rochacbruno* http://brunorocha.org
--
*Bruno Rocha - @rochacbruno http://twitter.com/rochacbruno* http://brunorocha.org
PSF-Community mailing list PSF-Community@python.org https://mail.python.org/mailman/listinfo/psf-community
This is not a solvable problem. IMNSHO We should never attempt to implement pre screening of packages.
It is a good post-package-upload task for someone to try and do as a research project.
Automated code scanning can only find already known things and similar signatures (at which point it can have false positives) and we aren't just talking about obfuscated source code. PyPI hosts binary wheels made using unreproduceable build processes on untrusted machines created from unverifiable inputs. Scanning services such as Google's https://www.virustotal.com/en/about/ exist but I'm not sure that'd be of much value to PyPI.
-gps
On Thu, May 4, 2017 at 7:28 PM Ryan Birmingham rainventions@gmail.com wrote:
I'm not sure what effective package review would look like here. Perhaps we could establish an entity to screen packages on an opt-in basis, but I don't know if we have the resources/people for this. Automated code screening could and probably would miss the python nation example due to the unorthodox use of compressed instructions. Does anyone have any ideas?
-Ryan Birmingham
On 4 May 2017 at 20:08, Bruno Rocha rochacbruno@gmail.com wrote:
Interesting detail, the mentioned package https://pypi.python.org/pypi/python-nation/1.0.1 was created and uploaded by Jacob Kaplan Moss, so I guess this is intended to be a POC, to show PyPI vulnerabilities or some Infosec experiment.
On Thu, May 4, 2017 at 8:41 PM, Bruno Rocha rochacbruno@gmail.com wrote:
Hi,
I just read this on reddit[0], a thread asking if PyPI packages are audited and somebody pointed the `python-nation`[1] which is a harmful and useless module, installing itself and sending the `/etc/passwd` content to external endpoint.
The app receiving the data is hosted at http://python-nation.herokuapp.com
and as the PSF mission [2] says
The mission of the Python Software Foundation is to promote, protect, and advance the Python programming language
I wonder if there are some workgroup at PSF to handle this? and not only the specific case of `python-nation` which should be deleted and the user banned maybe, But also to handle the audit of other packages?
[0] https://www.reddit.com/r/Python/comments/697da2/does_pypi_review_code_thats_... [1] https://www.reddit.com/r/Python/comments/697da2/does_pypi_review_code_thats_... [2] https://www.python.org/psf/mission/
Cheers,
--
*Bruno Rocha - @rochacbruno http://twitter.com/rochacbruno* http://brunorocha.org
--
*Bruno Rocha - @rochacbruno http://twitter.com/rochacbruno* http://brunorocha.org
PSF-Community mailing list PSF-Community@python.org https://mail.python.org/mailman/listinfo/psf-community
PSF-Community mailing list PSF-Community@python.org https://mail.python.org/mailman/listinfo/psf-community
On 5 May 2017 at 14:10, Gregory P. Smith greg@krypto.org wrote:
This is not a solvable problem. IMNSHO We should never attempt to implement pre screening of packages.
It is a good post-package-upload task for someone to try and do as a research project.
Automated code scanning can only find already known things and similar signatures (at which point it can have false positives) and we aren't just talking about obfuscated source code. PyPI hosts binary wheels made using unreproduceable build processes on untrusted machines created from unverifiable inputs. Scanning services such as Google's https://www.virustotal.com/en/about/ exist but I'm not sure that'd be of much value to PyPI.
Red Hat's approach to this (https://github.com/fabric8-analytics/) relies heavily on "popularity within your cohort" as a proxy for safety. It's far from being a perfect approach (since there's still a risk of the "bystander effect" coming into play, where everyone assumes everyone else is handling the security audits), but it at least gives people a heads up when they're doing something relatively unusual and hence may want to take more care and treat their potential dependencies with a bit more suspicion.
Cheers, Nick.
P.S. Full disclosure: until I switched teams a few months ago, working on fabric8-analytics (and its precursor projects) was my day job at Red Hat. As far as I'm aware, the current version still doesn't take the raw PyPI Big Query download data into account, but it does track component usage across public GitHub repositories - the benefit of focusing on the latter is that it gives you co-occurence information (i.e. "component X is often used in combination with component Y"), rather than the raw popularity metrics offered by the download numbers (which can also be heavily skewed by artifact caches, and the lack thereof, in automated build and test pipelines).
On May 4, 2017, at 4:41 PM, Bruno Rocha rochacbruno@gmail.com wrote:
Hi,
I just read this on reddit[0], a thread asking if PyPI packages are audited and somebody pointed the `python-nation`[1] which is a harmful and useless module, installing itself and sending the `/etc/passwd` content to external endpoint.
The app receiving the data is hosted at http://python-nation.herokuapp.com
and as the PSF mission [2] says
The mission of the Python Software Foundation is to promote, protect, and advance the Python programming language
I wonder if there are some workgroup at PSF to handle this? and not only the specific case of `python-nation` which should be deleted and the user banned maybe, But also to handle the audit of other packages?
[0] https://www.reddit.com/r/Python/comments/697da2/does_pypi_review_code_thats_... [1] https://www.reddit.com/r/Python/comments/697da2/does_pypi_review_code_thats_... [2] https://www.python.org/psf/mission/
Specifically re: the vector of running code at install time, wheels can help with this though I don't think there is a good way to tell pip to ignore non-wheel builds. But even then, the whole point is that you're downloading code from the internet :) If you want to discuss this further I recommend the distutils-sig mailing list.
--Noah
On 5 May 2017 at 09:41, Bruno Rocha rochacbruno@gmail.com wrote:
Hi,
I just read this on reddit[0], a thread asking if PyPI packages are audited and somebody pointed the `python-nation`[1] which is a harmful and useless module, installing itself and sending the `/etc/passwd` content to external endpoint.
The app receiving the data is hosted at http://python-nation.herokuapp.com
This is something that Jacob Kaplan-Moss wrote for a PyCon Australia security lightning talk a few years back: https://www.youtube.com/watch?list=PLs4CJRBY5F1KDIN6pv6daYWN_RnFOYvt0&fe...
That talk was prompted by a similar social engineering exercise carried out in the Ruby community: http://blog.honeybadger.io/stop-using-rubygemsorg-in-production/
and as the PSF mission [2] says
The mission of the Python Software Foundation is to promote, protect, and advance the Python programming language
I wonder if there are some workgroup at PSF to handle this? and not only the specific case of `python-nation` which should be deleted and the user banned maybe,
python-nation does not violate PyPI's Terms of Service. However, it does provide a useful reminder to end users that mistakenly view PyPI as a restricted app store rather than as an open publication platform akin to the web itself that "pip install <arbitrary-component>" is essentially no safer than "curl <arbitrary-script-url> | sh" (although it does offer greater assurances that if you pin your dependencies to particular versions, future downloads will either get you the same thing, or else fail outright).
But also to handle the audit of other packages?
When people and organisations want security audits of open source software, they either have to do them themselves, pay someone else to do them on their behalf, or else rely on one of the volunteer-driven collaborative software auditing projects more commonly known as "community Linux distributions" (accepting the couple of orders of magnitude reduction in available components that comes from that last choice).
Most large organisations will end up relying on some combination of the three (e.g. it's not uncommon for a RHEL-hosted application to include commercially audited packages from Red Hat and certified partners, community audited packages from EPEL, IUS, Fedora COPR, and/or softwarecollection.org, and team audited packages directly from PyPI, and we see the same kinds of layered architectures showing up regardless of which distro or platform people target).
Cheers, Nick.
-
Bruno Rocha -
Gregory P. Smith -
Jacqueline Kazil -
Nick Coghlan -
Noah Kantrowitz -
Ryan Birmingham