Hi. I recently found myself installing a node.js package, and in the process noticed that (sometime recently?) it started automatically warning about known vulnerabilities during installation of package.jsons (see https://docs.npmjs.com/cli/audit). At work, we run safety (https://pypi.org/project/safety/) on all our projects (which has both free and paid versions). It's great. I know there's a ton of wonderful work happening at the minute to improve underlying scaffolding + specification to enable tools other than setuptools + pip to thrive, so maybe this is the wrong moment, but I figured I'd ask anyways :) -- what are opinions on running a similar thing during pip install? -J
Would something like this require: - a pip extension/plugin/post-install hook API - a post-install hook that discloses all installed packages and versions (from pypi.org, mirrors, local directory) in exchange for checking and online security DB - a way to specify a key to e.g. pyup GItHub and GitLab offer similar functionality: https://github.blog/2018-07-12-security-vulnerability-alerts-for-python/ https://help.github.com/articles/about-security-alerts-for-vulnerable-depend... https://docs.gitlab.com/ee/user/project/merge_requests/dependency_scanning.h... https://gitlab.com/gitlab-org/security-products/dependency-scanning#supporte... https://pyup.io https://github.com/pyupio/safety-db
pipenv check relies on safety and Safety-DB to check for known vulnerabilities in locked components
On Monday, February 11, 2019, Julian Berman <julian@grayvines.com> wrote:
Hi.
I recently found myself installing a node.js package, and in the process noticed that (sometime recently?) it started automatically warning about known vulnerabilities during installation of package.jsons (see https://docs.npmjs.com/cli/audit).
At work, we run safety (https://pypi.org/project/safety/) on all our projects (which has both free and paid versions). It's great.
I know there's a ton of wonderful work happening at the minute to improve underlying scaffolding + specification to enable tools other than setuptools + pip to thrive, so maybe this is the wrong moment, but I figured I'd ask anyways :) -- what are opinions on running a similar thing during pip install?
-J
One way to avoid disclosing user environments to a third party is to build this into PyPI instead. The API could generate the warning for pip to display. This only covers packages on PyPI, of course, but trying to audit local and self-hosted packages is is a fools errand anyway IMO since there is no practical way for any tool to reliably know what *actually* is installed. -- Tzu-ping Chung (@uranusjr) uranusjr@gmail.com Sent from my iPhone
On 12 Feb 2019, at 11:34, Wes Turner <wes.turner@gmail.com> wrote:
Would something like this require:
- a pip extension/plugin/post-install hook API - a post-install hook that discloses all installed packages and versions (from pypi.org, mirrors, local directory) in exchange for checking and online security DB - a way to specify a key to e.g. pyup
GItHub and GitLab offer similar functionality:
https://github.blog/2018-07-12-security-vulnerability-alerts-for-python/ https://help.github.com/articles/about-security-alerts-for-vulnerable-depend...
https://docs.gitlab.com/ee/user/project/merge_requests/dependency_scanning.h... https://gitlab.com/gitlab-org/security-products/dependency-scanning#supporte...
https://github.com/pyupio/safety-db
pipenv check relies on safety and Safety-DB to check for known vulnerabilities in locked components
On Monday, February 11, 2019, Julian Berman <julian@grayvines.com> wrote: Hi.
I recently found myself installing a node.js package, and in the process noticed that (sometime recently?) it started automatically warning about known vulnerabilities during installation of package.jsons (see https://docs.npmjs.com/cli/audit).
At work, we run safety (https://pypi.org/project/safety/) on all our projects (which has both free and paid versions). It's great.
I know there's a ton of wonderful work happening at the minute to improve underlying scaffolding + specification to enable tools other than setuptools + pip to thrive, so maybe this is the wrong moment, but I figured I'd ask anyways :) -- what are opinions on running a similar thing during pip install?
-J -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/GSTL4...
Note In order to enable this functionality while maintaining its permissive copyright license, pipenv embeds an API client key for the backend Safety API operated by pyup.io rather than including a full copy of the CC-BY-NC-SA licensed Safety-DB database. This embedded client key is shared across all pipenv check users, and hence will be subject to API access
Good call. I didn't realize that that's how safety works. Is this the same data that pipenv/safety retrieves from pyup? https://github.com/pyupio/safety-db/blob/master/data/insecure_full.json https://pipenv.readthedocs.io/en/latest/advanced/#detection-of-security-vuln... : throttling based on overall usage rather than individual client usage.
You can also use your own safety API key by setting the environment
variable PIPENV_PYUP_API_KEY. https://github.com/pypa/pipenv/blob/master/pipenv/patched/safety/cli.py vulns = safety.check(packages=packages, key=key, db_mirror=db, cached=cache, ignore_ids=ignore) https://github.com/pypa/pipenv/blob/master/tasks/vendoring/__init__.py def update_safety On Monday, February 11, 2019, Tzu-ping Chung <uranusjr@gmail.com> wrote:
One way to avoid disclosing user environments to a third party is to build this into PyPI instead. The API could generate the warning for pip to display.
This only covers packages on PyPI, of course, but trying to audit local and self-hosted packages is is a fools errand anyway IMO since there is no practical way for any tool to reliably know what *actually* is installed.
-- Tzu-ping Chung (@uranusjr) uranusjr@gmail.com Sent from my iPhone
On 12 Feb 2019, at 11:34, Wes Turner <wes.turner@gmail.com> wrote:
Would something like this require:
- a pip extension/plugin/post-install hook API - a post-install hook that discloses all installed packages and versions (from pypi.org, mirrors, local directory) in exchange for checking and online security DB - a way to specify a key to e.g. pyup
GItHub and GitLab offer similar functionality:
https://github.blog/2018-07-12-security-vulnerability-alerts-for-python/ https://help.github.com/articles/about-security-alerts-for-vulnerable- dependencies/
https://docs.gitlab.com/ee/user/project/merge_requests/ dependency_scanning.html https://gitlab.com/gitlab-org/security-products/dependency- scanning#supported-languages-and-package-managers
https://github.com/pyupio/safety-db
pipenv check relies on safety and Safety-DB to check for known vulnerabilities in locked components
On Monday, February 11, 2019, Julian Berman <julian@grayvines.com> wrote:
Hi.
I recently found myself installing a node.js package, and in the process noticed that (sometime recently?) it started automatically warning about known vulnerabilities during installation of package.jsons (see https://docs.npmjs.com/cli/audit).
At work, we run safety (https://pypi.org/project/safety/) on all our projects (which has both free and paid versions). It's great.
I know there's a ton of wonderful work happening at the minute to improve underlying scaffolding + specification to enable tools other than setuptools + pip to thrive, so maybe this is the wrong moment, but I figured I'd ask anyways :) -- what are opinions on running a similar thing during pip install?
-J
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@ python.org/message/GSTL47B4CREYHKOS5I47WOPQURBKTOAY/
https://github.com/pypa/pipenv/blob/master/pipenv/patched/safety/safety.py : def write_to_cache(db_name, data): # cache is in: ~/safety/cache.json # ... def fetch_database(...) On Tuesday, February 12, 2019, Wes Turner <wes.turner@gmail.com> wrote:
Good call. I didn't realize that that's how safety works.
Is this the same data that pipenv/safety retrieves from pyup? https://github.com/pyupio/safety-db/blob/master/data/insecure_full.json
https://pipenv.readthedocs.io/en/latest/advanced/#detection- of-security-vulnerabilities :
Note In order to enable this functionality while maintaining its permissive copyright license, pipenv embeds an API client key for the backend Safety API operated by pyup.io rather than including a full copy of the CC-BY-NC-SA licensed Safety-DB database. This embedded client key is shared across all pipenv check users, and hence will be subject to API access throttling based on overall usage rather than individual client usage.
You can also use your own safety API key by setting the environment variable PIPENV_PYUP_API_KEY.
https://github.com/pypa/pipenv/blob/master/pipenv/patched/safety/cli.py vulns = safety.check(packages=packages, key=key, db_mirror=db, cached=cache, ignore_ids=ignore)
https://github.com/pypa/pipenv/blob/master/tasks/vendoring/__init__.py def update_safety
On Monday, February 11, 2019, Tzu-ping Chung <uranusjr@gmail.com> wrote:
One way to avoid disclosing user environments to a third party is to build this into PyPI instead. The API could generate the warning for pip to display.
This only covers packages on PyPI, of course, but trying to audit local and self-hosted packages is is a fools errand anyway IMO since there is no practical way for any tool to reliably know what *actually* is installed.
-- Tzu-ping Chung (@uranusjr) uranusjr@gmail.com Sent from my iPhone
On 12 Feb 2019, at 11:34, Wes Turner <wes.turner@gmail.com> wrote:
Would something like this require:
- a pip extension/plugin/post-install hook API - a post-install hook that discloses all installed packages and versions (from pypi.org, mirrors, local directory) in exchange for checking and online security DB - a way to specify a key to e.g. pyup
GItHub and GitLab offer similar functionality:
https://github.blog/2018-07-12-security-vulnerability-alerts-for-python/ https://help.github.com/articles/about-security-alerts- for-vulnerable-dependencies/
https://docs.gitlab.com/ee/user/project/merge_requests/depen dency_scanning.html https://gitlab.com/gitlab-org/security-products/dependency-s canning#supported-languages-and-package-managers
https://github.com/pyupio/safety-db
pipenv check relies on safety and Safety-DB to check for known vulnerabilities in locked components
On Monday, February 11, 2019, Julian Berman <julian@grayvines.com> wrote:
Hi.
I recently found myself installing a node.js package, and in the process noticed that (sometime recently?) it started automatically warning about known vulnerabilities during installation of package.jsons (see https://docs.npmjs.com/cli/audit).
At work, we run safety (https://pypi.org/project/safety/) on all our projects (which has both free and paid versions). It's great.
I know there's a ton of wonderful work happening at the minute to improve underlying scaffolding + specification to enable tools other than setuptools + pip to thrive, so maybe this is the wrong moment, but I figured I'd ask anyways :) -- what are opinions on running a similar thing during pip install?
-J
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archiv es/list/distutils-sig@python.org/message/GSTL47B4CREYHKOS5I 47WOPQURBKTOAY/
One way to avoid disclosing user environments to a third party is to build this into PyPI instead
Warehouse: - [ ] clarify pyup/safety-db license terms re: PyPI and attribution and noncommercial - [ ] packaging.tasks.?: retrieve pyup JSON - [ ] packaging.views.release_detail: add the data to the package release view json - [ ] packaging.tasks.?: invalidate cached release JSON when pyup json changes https://github.com/pypa/warehouse/blob/master/warehouse/packaging/views.py https://github.com/pypa/warehouse/blob/master/warehouse/packaging/tasks.py Pip - [ ] display release vuln info as retrieved from release_detail JSON - should this prompt or fail? PyPA - is there a PyPI release JSON spec that should or must be updated first? (So that other package indexes and tools can also implement this JSON key and insecure_full.json-style vuln info) On Tuesday, February 12, 2019, Wes Turner <wes.turner@gmail.com> wrote:
https://github.com/pypa/pipenv/blob/master/pipenv/patched/safety/safety.py :
def write_to_cache(db_name, data): # cache is in: ~/safety/cache.json # ... def fetch_database(...)
On Tuesday, February 12, 2019, Wes Turner <wes.turner@gmail.com> wrote:
Good call. I didn't realize that that's how safety works.
Is this the same data that pipenv/safety retrieves from pyup? https://github.com/pyupio/safety-db/blob/master/data/insecure_full.json
https://pipenv.readthedocs.io/en/latest/advanced/#detection- of-security-vulnerabilities :
Note In order to enable this functionality while maintaining its permissive copyright license, pipenv embeds an API client key for the backend Safety API operated by pyup.io rather than including a full copy of the CC-BY-NC-SA licensed Safety-DB database. This embedded client key is shared across all pipenv check users, and hence will be subject to API access throttling based on overall usage rather than individual client usage.
You can also use your own safety API key by setting the environment variable PIPENV_PYUP_API_KEY.
https://github.com/pypa/pipenv/blob/master/pipenv/patched/safety/cli.py vulns = safety.check(packages=packages, key=key, db_mirror=db, cached=cache, ignore_ids=ignore)
https://github.com/pypa/pipenv/blob/master/tasks/vendoring/__init__.py def update_safety
On Monday, February 11, 2019, Tzu-ping Chung <uranusjr@gmail.com> wrote:
One way to avoid disclosing user environments to a third party is to build this into PyPI instead. The API could generate the warning for pip to display.
This only covers packages on PyPI, of course, but trying to audit local and self-hosted packages is is a fools errand anyway IMO since there is no practical way for any tool to reliably know what *actually* is installed.
-- Tzu-ping Chung (@uranusjr) uranusjr@gmail.com Sent from my iPhone
On 12 Feb 2019, at 11:34, Wes Turner <wes.turner@gmail.com> wrote:
Would something like this require:
- a pip extension/plugin/post-install hook API - a post-install hook that discloses all installed packages and versions (from pypi.org, mirrors, local directory) in exchange for checking and online security DB - a way to specify a key to e.g. pyup
GItHub and GitLab offer similar functionality:
https://github.blog/2018-07-12-security-vulnerability-alerts-for-python/ https://help.github.com/articles/about-security-alerts-for -vulnerable-dependencies/
https://docs.gitlab.com/ee/user/project/merge_requests/depen dency_scanning.html https://gitlab.com/gitlab-org/security-products/dependency-s canning#supported-languages-and-package-managers
https://github.com/pyupio/safety-db
pipenv check relies on safety and Safety-DB to check for known vulnerabilities in locked components
On Monday, February 11, 2019, Julian Berman <julian@grayvines.com> wrote:
Hi.
I recently found myself installing a node.js package, and in the process noticed that (sometime recently?) it started automatically warning about known vulnerabilities during installation of package.jsons (see https://docs.npmjs.com/cli/audit).
At work, we run safety (https://pypi.org/project/safety/) on all our projects (which has both free and paid versions). It's great.
I know there's a ton of wonderful work happening at the minute to improve underlying scaffolding + specification to enable tools other than setuptools + pip to thrive, so maybe this is the wrong moment, but I figured I'd ask anyways :) -- what are opinions on running a similar thing during pip install?
-J
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archiv es/list/distutils-sig@python.org/message/GSTL47B4CREYHKOS5I4 7WOPQURBKTOAY/
On Tue, Feb 12, 2019 at 5:24 AM Tzu-ping Chung <uranusjr@gmail.com> wrote:
One way to avoid disclosing user environments to a third party is to build this into PyPI instead. The API could generate the warning for pip to display.
How large are these kinds of databases? Would it be a conceivable thought end users and/or CI infrastructures of organisations keep and update their local copies and thus only disclose the fact they're using such a database? -- Joni Orponen
PyUp’s dataset is public, and the insecure_full document posted earlier in thread is 344 kb, so yeah, it is totally possible. https://github.com/pyupio/safety-db/blob/master/data/insecure_full.json <https://github.com/pyupio/safety-db/blob/master/data/insecure_full.json>
On 12/2, 2019, at 17:05, Joni Orponen <j.orponen@4teamwork.ch> wrote:
On Tue, Feb 12, 2019 at 5:24 AM Tzu-ping Chung <uranusjr@gmail.com <mailto:uranusjr@gmail.com>> wrote: One way to avoid disclosing user environments to a third party is to build this into PyPI instead. The API could generate the warning for pip to display.
How large are these kinds of databases? Would it be a conceivable thought end users and/or CI infrastructures of organisations keep and update their local copies and thus only disclose the fact they're using such a database?
-- Joni Orponen -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/ERBNV...
participants (4)
-
Joni Orponen -
Julian Berman -
Tzu-ping Chung -
Wes Turner