December PyPA CI/CD Bug-o-Rama
Hi PyPA Committers, This is an idea I've been discussing with Seth Larson (PSF SDIR, but who is also a PyPA member and therefore should be on this list). The TL;DR is this: I did an org-wide scan of PyPA using zizmor[1] a few days ago, and there are a couple of thousands of CI/CD findings that would ideally be addressed. Most of these aren't critical (in the sense that they don't pose an immediate risk to the ecosystem), but many are the kinds of latent risks/code injection weaknesses that make campaigns like the one we're currently seeing in npm land easier (with "Shai-Hulud"[2]). Basically, what I would like to do is dedicate some time (and snipe some of Seth's time) for fixing as many of these findings as possible in the month of December. Seth and I would basically do this in the clear: we'd work together and submit public PRs, which other PyPA members (respective owners of projects) would then review and merge. I'm starting this as a thread because this is a nontrivial undertaking, and it's going to involve a lot of public activity/review cycles. So, before I did it, I wanted to gather feedback, namely: 1. Does anybody think we should generally *not* do this? 2. Does anybody prefer we do *not* attempt CI/CD patches on their specific project(s)? If so, we're happy to avoid those projects; we don't want to add additional noise or maintenance burden to anybody who would find it stressful (especially given that it's the holiday season). 3. Finally, would anybody else like to help? I'm interested to hear peoples' thoughts, including frank opinions about whether this kind of mass initiative makes sense -- my approach over the last year has been to send fixes to projects on a one-off basis, but I think a "Bug-o-Rama" session could be really productive for clearing things out in bulk. As a last note, I've started this on-thread because it has a security-sensitive nature -- not one so critical that it needs an embargo or anything like that, but I figured it made sense to solicit thoughts/consensus here first rather than on DPO. However, I'm happy to move to DPO if there's consensus that that's a better venue, given that the work itself will be public as it happens anyways. Best, William (P.S. This is entirely an independent idea -- I'm not representing my employer or any interests other than my own desire to improve the security of the PyPA's projects.) [1]: https://docs.zizmor.sh/ [2]: https://www.wiz.io/blog/shai-hulud-2-0-ongoing-supply-chain-attack
I am supportive! (I just added Zizmor to pypi/linehaul-cloud-function <https://github.com/pypi/linehaul-cloud-function/pull/240> yesterday and thought it would be good to audit that org, so maybe we bring it in scope as well?) For anyone unfamiliar with the tool, in my experience I've found that it is a great addition to prevent common mistakes in GitHub workflows (beyond what CodeQL currently offers), generally not fussy, has a very low false positive rate and meaningful/actionable messages when it does find something. I expect we will see high-quality PRs from this effort. On Mon, Nov 24, 2025 at 7:38 PM William Woodruff <william@yossarian.net> wrote:
Hi PyPA Committers,
This is an idea I've been discussing with Seth Larson (PSF SDIR, but who is also a PyPA member and therefore should be on this list).
The TL;DR is this: I did an org-wide scan of PyPA using zizmor[1] a few days ago, and there are a couple of thousands of CI/CD findings that would ideally be addressed. Most of these aren't critical (in the sense that they don't pose an immediate risk to the ecosystem), but many are the kinds of latent risks/code injection weaknesses that make campaigns like the one we're currently seeing in npm land easier (with "Shai-Hulud"[2]).
Basically, what I would like to do is dedicate some time (and snipe some of Seth's time) for fixing as many of these findings as possible in the month of December. Seth and I would basically do this in the clear: we'd work together and submit public PRs, which other PyPA members (respective owners of projects) would then review and merge.
I'm starting this as a thread because this is a nontrivial undertaking, and it's going to involve a lot of public activity/review cycles. So, before I did it, I wanted to gather feedback, namely:
1. Does anybody think we should generally *not* do this? 2. Does anybody prefer we do *not* attempt CI/CD patches on their specific project(s)? If so, we're happy to avoid those projects; we don't want to add additional noise or maintenance burden to anybody who would find it stressful (especially given that it's the holiday season). 3. Finally, would anybody else like to help?
I'm interested to hear peoples' thoughts, including frank opinions about whether this kind of mass initiative makes sense -- my approach over the last year has been to send fixes to projects on a one-off basis, but I think a "Bug-o-Rama" session could be really productive for clearing things out in bulk.
As a last note, I've started this on-thread because it has a security-sensitive nature -- not one so critical that it needs an embargo or anything like that, but I figured it made sense to solicit thoughts/consensus here first rather than on DPO. However, I'm happy to move to DPO if there's consensus that that's a better venue, given that the work itself will be public as it happens anyways.
Best, William
(P.S. This is entirely an independent idea -- I'm not representing my employer or any interests other than my own desire to improve the security of the PyPA's projects.)
[2]: https://www.wiz.io/blog/shai-hulud-2-0-ongoing-supply-chain-attack _______________________________________________ PyPA-Committers mailing list -- pypa-committers@python.org To unsubscribe send an email to pypa-committers-leave@python.org https://mail.python.org/mailman3//lists/pypa-committers.python.org Member address: di@python.org
No major concerns from me. I'd be happy to review any patches for pip or other repositories owned by the pip core team in the PyPA GitHub organisation. The only thing I'd suggest is that you should be clear whenever you're opening a PR on a new repository that the PR's purpose is explained (lest we annoy a fellow maintainer). I don't mean that every PR needs a wall of text, but cross-linking PRs you've already raised on other PyPA repositories for context could be helpful. Thanks for taking this on! *Richard Si *(he/him) GitHub: @ichard26 On Tue, 25 Nov 2025 at 09:22, Dustin Ingram <di@python.org> wrote:
I am supportive! (I just added Zizmor to pypi/linehaul-cloud-function <https://github.com/pypi/linehaul-cloud-function/pull/240> yesterday and thought it would be good to audit that org, so maybe we bring it in scope as well?)
For anyone unfamiliar with the tool, in my experience I've found that it is a great addition to prevent common mistakes in GitHub workflows (beyond what CodeQL currently offers), generally not fussy, has a very low false positive rate and meaningful/actionable messages when it does find something. I expect we will see high-quality PRs from this effort.
On Mon, Nov 24, 2025 at 7:38 PM William Woodruff <william@yossarian.net> wrote:
Hi PyPA Committers,
This is an idea I've been discussing with Seth Larson (PSF SDIR, but who is also a PyPA member and therefore should be on this list).
The TL;DR is this: I did an org-wide scan of PyPA using zizmor[1] a few days ago, and there are a couple of thousands of CI/CD findings that would ideally be addressed. Most of these aren't critical (in the sense that they don't pose an immediate risk to the ecosystem), but many are the kinds of latent risks/code injection weaknesses that make campaigns like the one we're currently seeing in npm land easier (with "Shai-Hulud"[2]).
Basically, what I would like to do is dedicate some time (and snipe some of Seth's time) for fixing as many of these findings as possible in the month of December. Seth and I would basically do this in the clear: we'd work together and submit public PRs, which other PyPA members (respective owners of projects) would then review and merge.
I'm starting this as a thread because this is a nontrivial undertaking, and it's going to involve a lot of public activity/review cycles. So, before I did it, I wanted to gather feedback, namely:
1. Does anybody think we should generally *not* do this? 2. Does anybody prefer we do *not* attempt CI/CD patches on their specific project(s)? If so, we're happy to avoid those projects; we don't want to add additional noise or maintenance burden to anybody who would find it stressful (especially given that it's the holiday season). 3. Finally, would anybody else like to help?
I'm interested to hear peoples' thoughts, including frank opinions about whether this kind of mass initiative makes sense -- my approach over the last year has been to send fixes to projects on a one-off basis, but I think a "Bug-o-Rama" session could be really productive for clearing things out in bulk.
As a last note, I've started this on-thread because it has a security-sensitive nature -- not one so critical that it needs an embargo or anything like that, but I figured it made sense to solicit thoughts/consensus here first rather than on DPO. However, I'm happy to move to DPO if there's consensus that that's a better venue, given that the work itself will be public as it happens anyways.
Best, William
(P.S. This is entirely an independent idea -- I'm not representing my employer or any interests other than my own desire to improve the security of the PyPA's projects.)
[2]: https://www.wiz.io/blog/shai-hulud-2-0-ongoing-supply-chain-attack _______________________________________________ PyPA-Committers mailing list -- pypa-committers@python.org To unsubscribe send an email to pypa-committers-leave@python.org https://mail.python.org/mailman3//lists/pypa-committers.python.org Member address: di@python.org
_______________________________________________ PyPA-Committers mailing list -- pypa-committers@python.org To unsubscribe send an email to pypa-committers-leave@python.org https://mail.python.org/mailman3//lists/pypa-committers.python.org Member address: sichard26@gmail.com
Hi Richard, Thank you! Yeah, I completely agree about making the purpose (and relative importance, since not all of the PRs that stem from this work will be equally important/critical) clear -- I think the worst possible outcome here would be annoying/burning out a bunch of maintainers due to high PR volume + lack of visibility, so we'll take pains to make sure everything is as clearly documented and cross-referenced as possible. Separately, on a logistical note: Seth noted that he won't have time for this in December (he's too smart to get sniped into doing extra work by me), and I also realized that this is going to require more planning to do right than I want to commit in December. So I'm hoping to start this more formally in January or so, rather than try to force an artificial timeline :-) Best, William On Thu, Nov 27, 2025 at 8:38 PM Richard Si <sichard26@gmail.com> wrote:
No major concerns from me. I'd be happy to review any patches for pip or other repositories owned by the pip core team in the PyPA GitHub organisation. The only thing I'd suggest is that you should be clear whenever you're opening a PR on a new repository that the PR's purpose is explained (lest we annoy a fellow maintainer). I don't mean that every PR needs a wall of text, but cross-linking PRs you've already raised on other PyPA repositories for context could be helpful.
Thanks for taking this on!
Richard Si (he/him) GitHub: @ichard26
On Tue, 25 Nov 2025 at 09:22, Dustin Ingram <di@python.org> wrote:
I am supportive! (I just added Zizmor to pypi/linehaul-cloud-function yesterday and thought it would be good to audit that org, so maybe we bring it in scope as well?)
For anyone unfamiliar with the tool, in my experience I've found that it is a great addition to prevent common mistakes in GitHub workflows (beyond what CodeQL currently offers), generally not fussy, has a very low false positive rate and meaningful/actionable messages when it does find something. I expect we will see high-quality PRs from this effort.
On Mon, Nov 24, 2025 at 7:38 PM William Woodruff <william@yossarian.net> wrote:
Hi PyPA Committers,
This is an idea I've been discussing with Seth Larson (PSF SDIR, but who is also a PyPA member and therefore should be on this list).
The TL;DR is this: I did an org-wide scan of PyPA using zizmor[1] a few days ago, and there are a couple of thousands of CI/CD findings that would ideally be addressed. Most of these aren't critical (in the sense that they don't pose an immediate risk to the ecosystem), but many are the kinds of latent risks/code injection weaknesses that make campaigns like the one we're currently seeing in npm land easier (with "Shai-Hulud"[2]).
Basically, what I would like to do is dedicate some time (and snipe some of Seth's time) for fixing as many of these findings as possible in the month of December. Seth and I would basically do this in the clear: we'd work together and submit public PRs, which other PyPA members (respective owners of projects) would then review and merge.
I'm starting this as a thread because this is a nontrivial undertaking, and it's going to involve a lot of public activity/review cycles. So, before I did it, I wanted to gather feedback, namely:
1. Does anybody think we should generally *not* do this? 2. Does anybody prefer we do *not* attempt CI/CD patches on their specific project(s)? If so, we're happy to avoid those projects; we don't want to add additional noise or maintenance burden to anybody who would find it stressful (especially given that it's the holiday season). 3. Finally, would anybody else like to help?
I'm interested to hear peoples' thoughts, including frank opinions about whether this kind of mass initiative makes sense -- my approach over the last year has been to send fixes to projects on a one-off basis, but I think a "Bug-o-Rama" session could be really productive for clearing things out in bulk.
As a last note, I've started this on-thread because it has a security-sensitive nature -- not one so critical that it needs an embargo or anything like that, but I figured it made sense to solicit thoughts/consensus here first rather than on DPO. However, I'm happy to move to DPO if there's consensus that that's a better venue, given that the work itself will be public as it happens anyways.
Best, William
(P.S. This is entirely an independent idea -- I'm not representing my employer or any interests other than my own desire to improve the security of the PyPA's projects.)
[2]: https://www.wiz.io/blog/shai-hulud-2-0-ongoing-supply-chain-attack _______________________________________________ PyPA-Committers mailing list -- pypa-committers@python.org To unsubscribe send an email to pypa-committers-leave@python.org https://mail.python.org/mailman3//lists/pypa-committers.python.org Member address: di@python.org
_______________________________________________ PyPA-Committers mailing list -- pypa-committers@python.org To unsubscribe send an email to pypa-committers-leave@python.org https://mail.python.org/mailman3//lists/pypa-committers.python.org Member address: sichard26@gmail.com
participants (3)
-
Dustin Ingram -
Richard Si -
William Woodruff