[Twisted-Python] twistedchecker now uses pylint >= 2.4.4, can we use type annotations in Twisted now?
Hi, Last week I did some work and updated twistedchecker so that it uses pylint >= 2.4.4. Before that, it was using a really old version of pylint. twistedchecker is run over the Twisted code as part of CI, and reports various style issues. Now that twistedchecker is using a newer pylint, it should be compatible with newer Python language features. Now that Twisted supports Python 3.5+ (with Python 2.7 support being dropped), is it OK to start using newer features of the Python language in the Twisted codebase such as type hints ( https://docs.python.org/3/library/typing.html )? I don't have experience with that feature, but it looks like it could be useful, when combined with a tool like mypy. Does anyone else have opinions on type hints and mypy? -- Craig
On Apr 21, 2020, at 8:57 PM, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
Hi,
Last week I did some work and updated twistedchecker so that it uses pylint >= 2.4.4. Before that, it was using a really old version of pylint.
twistedchecker is run over the Twisted code as part of CI, and reports various style issues. Now that twistedchecker is using a newer pylint, it should be compatible with newer Python language features.
Now that Twisted supports Python 3.5+ (with Python 2.7 support being dropped), is it OK to start using newer features of the Python language in the Twisted codebase such as type hints ( https://docs.python.org/3/library/typing.html <https://docs.python.org/3/library/typing.html> )?
I don't have experience with that feature, but it looks like it could be useful, when combined with a tool like mypy. Does anyone else have opinions on type hints and mypy?
We use them at work, and on some other Twisted projects (Klein) and they're absolutely awesome. Even with type comments, mypy is a huge upgrade to how one writes and maintains Python; with annotations, it's a major upgrade to the language. The first step here, however, is to set up the CI infrastructure (tox, etc) to run mypy so that we can ensure that as we start writing type hints, we don't accidentally get any of them wrong and back ourselves into any corners. Mypy can catch a surprising number of bugs with just the implicit type-checking it does on values that come from the standard library. In fact, if we do `mypy src/twisted` right now, and exclude the things that would be fixed by adding in https://github.com/Shoobx/mypy-zope <https://github.com/Shoobx/mypy-zope> ("method must have at least one argument" zope.interface definition errors) and fixing up some simple type hints (has no attribute "skip") we still have almost a thousand type errors that we should figure out a way to start correcting or systematically skipping if they're false positives. I guarantee you there's at least one real bug in there though. As you know, I tend to be pretty cautious about sweeping changes to the code that might make it harder to maintain on older versions - type hints are an exception where I think it's absolutely worthwhile to go All In early on. But CI infrastructure for this stuff is a must-have and it might be tricky to get set up initially. -glyph
On Wed, Apr 22, 2020 at 12:28 AM Glyph <glyph@twistedmatrix.com> wrote:
On Apr 21, 2020, at 8:57 PM, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
Does anyone else have opinions on type hints and mypy?
We use them at work, and on some other Twisted projects (Klein) and they're absolutely awesome. Even with type comments, mypy is a huge upgrade to how one writes and maintains Python; with annotations, it's a major upgrade to the language.
The first step here, however, is to set up the CI infrastructure (tox, etc) to run mypy so that we can ensure that as we start writing type hints, we don't accidentally get any of them wrong and back ourselves into any corners. Mypy can catch a surprising number of bugs with just the implicit type-checking it does on values that come from the standard library. In fact, if we do `mypy src/twisted` right now, and exclude the things that would be fixed by adding in https://github.com/Shoobx/mypy-zope ("method must have at least one argument" zope.interface definition errors) and fixing up some simple type hints (has no attribute "skip") we still have almost a thousand type errors that we should figure out a way to start correcting or systematically skipping if they're false positives. I guarantee you there's at least one real bug in there though.
As you know, I tend to be pretty cautious about sweeping changes to the code that might make it harder to maintain on older versions - type hints are an exception where I think it's absolutely worthwhile to go All In early on. But CI infrastructure for this stuff is a must-have and it might be tricky to get set up initially.
Earlier this week, I merged to trunk a new tox rule for running mypy with mypy-zope. You can use it by running the following command in the top-level Twisted directory: *tox -e mypy* mypy reports a lot of errors, but I think we can take several passes through the Twisted codebase to clean those up. Once we clean up the errors, we can integrate the tox rule with Travis or Azure CI or CircleCI (whatever works best). I submitted a few PR's to start cleaning up these errors: - https://github.com/twisted/twisted/pull/1264 (has no attribute "skip") - https://github.com/twisted/twisted/pull/1261 (Too few arguments to "makeTestCaseClasses") -- Craig
On May 7, 2020, at 1:56 PM, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
On Wed, Apr 22, 2020 at 12:28 AM Glyph <glyph@twistedmatrix.com <mailto:glyph@twistedmatrix.com>> wrote:
On Apr 21, 2020, at 8:57 PM, Craig Rodrigues <rodrigc@crodrigues.org <mailto:rodrigc@crodrigues.org>> wrote:
Does anyone else have opinions on type hints and mypy?
We use them at work, and on some other Twisted projects (Klein) and they're absolutely awesome. Even with type comments, mypy is a huge upgrade to how one writes and maintains Python; with annotations, it's a major upgrade to the language.
The first step here, however, is to set up the CI infrastructure (tox, etc) to run mypy so that we can ensure that as we start writing type hints, we don't accidentally get any of them wrong and back ourselves into any corners. Mypy can catch a surprising number of bugs with just the implicit type-checking it does on values that come from the standard library. In fact, if we do `mypy src/twisted` right now, and exclude the things that would be fixed by adding in https://github.com/Shoobx/mypy-zope <https://github.com/Shoobx/mypy-zope> ("method must have at least one argument" zope.interface definition errors) and fixing up some simple type hints (has no attribute "skip") we still have almost a thousand type errors that we should figure out a way to start correcting or systematically skipping if they're false positives. I guarantee you there's at least one real bug in there though.
As you know, I tend to be pretty cautious about sweeping changes to the code that might make it harder to maintain on older versions - type hints are an exception where I think it's absolutely worthwhile to go All In early on. But CI infrastructure for this stuff is a must-have and it might be tricky to get set up initially.
Earlier this week, I merged to trunk a new tox rule for running mypy with mypy-zope. You can use it by running the following command in the top-level Twisted directory:
tox -e mypy
mypy reports a lot of errors, but I think we can take several passes through the Twisted codebase to clean those up.
Once we clean up the errors, we can integrate the tox rule with Travis or Azure CI or CircleCI (whatever works best).
I submitted a few PR's to start cleaning up these errors: https://github.com/twisted/twisted/pull/1264 <https://github.com/twisted/twisted/pull/1264> (has no attribute "skip") https://github.com/twisted/twisted/pull/1261 <https://github.com/twisted/twisted/pull/1261> (Too few arguments to "makeTestCaseClasses")
Thank you for doing this work, Craig! I strongly suspect that mypy is going to spot more than a few real bugs in our codebase, and I can think of at least one enthusiastic user (me) who wants Twisted to start shipping stubs for our users to use! This also reminded me to file https://twistedmatrix.com/trac/ticket/9816#ticket <https://twistedmatrix.com/trac/ticket/9816#ticket> because i couldn't find it filed already. -glyph
I have merged a few PR's to trunk which eliminate hundreds of errors encountered with: *tox -e mypy* I think we can take several passes with more PR's to whack away all these mypy errors, and turn on mypy as part of the default CI for Twisted. I have seen a few errors like: src/twisted/words/protocols/jabber/sasl_mechanisms.py:47:1: error: 'Anonymous' is missing following 'ISASLMechanism' interface members: getResponse. [misc] class Anonymous(object): src/twisted/words/protocols/jabber/sasl_mechanisms.py:61:1: error: 'Plain' is missing following 'ISASLMechanism' interface members: getResponse. [misc] class Plain(object): src/twisted/internet/_dumbwin32proc.py:110:1: error: 'Process' is missing following 'twisted.internet.interfaces.ITransport' interface members: getHost, getPeer. [misc] class Process(_pollingfile._PollingTimer, BaseProcess): src/twisted/internet/process.py:959:1: error: 'PTYProcess' is missing following 'twisted.internet.interfaces.ITransport' interface members: getHost, getPeer. [misc] class PTYProcess(abstract.FileDescriptor, _BaseProcess): src/twisted/internet/process.py:959:1: error: 'PTYProcess' is missing following 'IProcessTransport' interface members: closeChildFD, writeToChild. [misc] class PTYProcess(abstract.FileDescriptor, _BaseProcess): src/twisted/internet/base.py:504:1: error: 'ReactorBase' is missing following 'IReactorCore' interface members: run. [misc] class ReactorBase(PluggableResolverMixin) For a class to properly implement a Zope interface, is it mandatory that it implement every method in that interface? If we modify the classes with mypy errors to properly implement these methods (even with no-ops) is that the correct way to go? -- Craig
On Sun, Jun 7, 2020 at 7:22 PM Craig Rodrigues <rodrigc@crodrigues.org> wrote:
I have merged a few PR's to trunk which eliminate hundreds of errors encountered with:
*tox -e mypy* I think we can take several passes with more PR's to whack away all these mypy errors, and turn on mypy as part of the default CI for Twisted.
I have seen a few errors like:
src/twisted/words/protocols/jabber/sasl_mechanisms.py:47:1: error: 'Anonymous' is missing following 'ISASLMechanism' interface members: getResponse. [misc] class Anonymous(object): src/twisted/words/protocols/jabber/sasl_mechanisms.py:61:1: error: 'Plain' is missing following 'ISASLMechanism' interface members: getResponse. [misc] class Plain(object): src/twisted/internet/_dumbwin32proc.py:110:1: error: 'Process' is missing following 'twisted.internet.interfaces.ITransport' interface members: getHost, getPeer. [misc] class Process(_pollingfile._PollingTimer, BaseProcess): src/twisted/internet/process.py:959:1: error: 'PTYProcess' is missing following 'twisted.internet.interfaces.ITransport' interface members: getHost, getPeer. [misc] class PTYProcess(abstract.FileDescriptor, _BaseProcess): src/twisted/internet/process.py:959:1: error: 'PTYProcess' is missing following 'IProcessTransport' interface members: closeChildFD, writeToChild. [misc] class PTYProcess(abstract.FileDescriptor, _BaseProcess): src/twisted/internet/base.py:504:1: error: 'ReactorBase' is missing following 'IReactorCore' interface members: run. [misc] class ReactorBase(PluggableResolverMixin)
For a class to properly implement a Zope interface, is it mandatory that it implement every method in that interface?
Yes.
If we modify the classes with mypy errors to properly implement these methods (even with no-ops) is that the correct way to go?
Who does this serve? I would say no, this is not correct. If a type declares it implements an interface and it cannot provide useful implementations of every method/attribute, then it made a mistake in its declaration or the interface has the wrong methods/attributes. Jean-Paul
On Jun 7, 2020, at 4:43 PM, Jean-Paul Calderone <exarkun@twistedmatrix.com> wrote:
On Sun, Jun 7, 2020 at 7:22 PM Craig Rodrigues <rodrigc@crodrigues.org <mailto:rodrigc@crodrigues.org>> wrote: I have merged a few PR's to trunk which eliminate hundreds of errors encountered with:
tox -e mypy
I think we can take several passes with more PR's to whack away all these mypy errors, and turn on mypy as part of the default CI for Twisted.
I have seen a few errors like:
src/twisted/words/protocols/jabber/sasl_mechanisms.py:47:1: error: 'Anonymous' is missing following 'ISASLMechanism' interface members: getResponse. [misc] class Anonymous(object): src/twisted/words/protocols/jabber/sasl_mechanisms.py:61:1: error: 'Plain' is missing following 'ISASLMechanism' interface members: getResponse. [misc] class Plain(object): src/twisted/internet/_dumbwin32proc.py:110:1: error: 'Process' is missing following 'twisted.internet.interfaces.ITransport' interface members: getHost, getPeer. [misc] class Process(_pollingfile._PollingTimer, BaseProcess): src/twisted/internet/process.py:959:1: error: 'PTYProcess' is missing following 'twisted.internet.interfaces.ITransport' interface members: getHost, getPeer. [misc] class PTYProcess(abstract.FileDescriptor, _BaseProcess): src/twisted/internet/process.py:959:1: error: 'PTYProcess' is missing following 'IProcessTransport' interface members: closeChildFD, writeToChild. [misc] class PTYProcess(abstract.FileDescriptor, _BaseProcess): src/twisted/internet/base.py:504:1: error: 'ReactorBase' is missing following 'IReactorCore' interface members: run. [misc] class ReactorBase(PluggableResolverMixin)
For a class to properly implement a Zope interface, is it mandatory that it implement every method in that interface?
Yes.
If we modify the classes with mypy errors to properly implement these methods (even with no-ops) is that the correct way to go?
Who does this serve? I would say no, this is not correct. If a type declares it implements an interface and it cannot provide useful implementations of every method/attribute, then it made a mistake in its declaration or the interface has the wrong methods/attributes.
First of all, it's totally awesome that we're catching these problems with mypy! Thank you to everyone who contributed to get this set up. I think that different errors probably indicate different problems. Without a lot of motivated consumers of this interface information, it's easy to slip up, and we have slipped up. And without accurate interface information it's hard to be a discerning consumer! So there's a bit of a chicken-and-egg problem here, and mypy will help us resolve that paradox, and let people really depend on these. To opine on the ones listed above specifically, in case this can be more readily generalized: Anonymous & Plain missing getResponse is probably something they can get away with due to the specifics of how SASL works with those specific mechanisms. They should still provide an implementation, although it would be fine if it was one that simply raised an explanatory exception explaining why they don't expect to get called in their specific cases. PTYProcess and Process's missing attributes are just bugs. I thought they were even already-filed bugs, but I was remembering this one: https://twistedmatrix.com/trac/ticket/4585. There was also https://twistedmatrix.com/trac/ticket/6606 . As I said: easy to screw this up without any way to consistently check; we've screwed it up in other ways in the past. We should implement all of these methods. ReactorBase is another depressing artifact of our inheritance-obsessed initial design, which I hold out hope that we may recover from before my 50th birthday. The idea here being that subclasses of ReactorBase should be implementing those interfaces, so it'll go ahead and declare them for you. Now, ReactorBase probably just shouldn't be public in the first place, but this specific attribute of its declaration is simply wrong; it doesn't implement those interfaces and it should not say that it does. -glyph
Thanks to Adi and Wilfredo who have reviewed my PR's. Currently in trunk, I have eliminated hundreds of mypy errors through a combination of trivial fixes to code, adding type annotations, and adding special comments to turn off mypy errors in a few places. I have one more pending PR ( https://github.com/twisted/twisted/pull/1290 ) to turn on a mypy build in an Azure pipeline. In this run of *tox -e mypy* there are* 437* errors from mypy: https://dev.azure.com/twistedmatrix/twisted/_build/results?buildId=1904&view=logs&j=d78da089-4bf6-58fe-28a5-46635fd5b8d1&t=45ce7910-c28c-5e9a-4ee7-e041d314a5fc I think we can gradually whack these down to zero mypy errors, and turn mypy on as part of default CI for Twisted. -- Craig On Wed, Apr 22, 2020 at 12:28 AM Glyph <glyph@twistedmatrix.com> wrote:
The first step here, however, is to set up the CI infrastructure (tox, etc) to run mypy so that we can ensure that as we start writing type hints, we don't accidentally get any of them wrong and back ourselves into any corners. Mypy can catch a surprising number of bugs with just the implicit type-checking it does on values that come from the standard library. In fact, if we do `mypy src/twisted` right now, and exclude the things that would be fixed by adding in https://github.com/Shoobx/mypy-zope ("method must have at least one argument" zope.interface definition errors) and fixing up some simple type hints (has no attribute "skip") we still have almost a thousand type errors that we should figure out a way to start correcting or systematically skipping if they're false positives. I guarantee you there's at least one real bug in there though.
As you know, I tend to be pretty cautious about sweeping changes to the code that might make it harder to maintain on older versions - type hints are an exception where I think it's absolutely worthwhile to go All In early on. But CI infrastructure for this stuff is a must-have and it might be tricky to get set up initially.
-glyph
Awesome, thanks for doing this work Craig, Adi, and Wilfredo! -g
On Jun 15, 2020, at 8:32 PM, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
Thanks to Adi and Wilfredo who have reviewed my PR's. Currently in trunk, I have eliminated hundreds of mypy errors through a combination of trivial fixes to code, adding type annotations, and adding special comments to turn off mypy errors in a few places.
I have one more pending PR ( https://github.com/twisted/twisted/pull/1290 <https://github.com/twisted/twisted/pull/1290> ) to turn on a mypy build in an Azure pipeline. In this run of tox -e mypy there are 437 errors from mypy:
https://dev.azure.com/twistedmatrix/twisted/_build/results?buildId=1904&view=logs&j=d78da089-4bf6-58fe-28a5-46635fd5b8d1&t=45ce7910-c28c-5e9a-4ee7-e041d314a5fc <https://dev.azure.com/twistedmatrix/twisted/_build/results?buildId=1904&view=logs&j=d78da089-4bf6-58fe-28a5-46635fd5b8d1&t=45ce7910-c28c-5e9a-4ee7-e041d314a5fc>
I think we can gradually whack these down to zero mypy errors, and turn mypy on as part of default CI for Twisted.
-- Craig
On Wed, Apr 22, 2020 at 12:28 AM Glyph <glyph@twistedmatrix.com <mailto:glyph@twistedmatrix.com>> wrote:
The first step here, however, is to set up the CI infrastructure (tox, etc) to run mypy so that we can ensure that as we start writing type hints, we don't accidentally get any of them wrong and back ourselves into any corners. Mypy can catch a surprising number of bugs with just the implicit type-checking it does on values that come from the standard library. In fact, if we do `mypy src/twisted` right now, and exclude the things that would be fixed by adding in https://github.com/Shoobx/mypy-zope <https://github.com/Shoobx/mypy-zope> ("method must have at least one argument" zope.interface definition errors) and fixing up some simple type hints (has no attribute "skip") we still have almost a thousand type errors that we should figure out a way to start correcting or systematically skipping if they're false positives. I guarantee you there's at least one real bug in there though.
As you know, I tend to be pretty cautious about sweeping changes to the code that might make it harder to maintain on older versions - type hints are an exception where I think it's absolutely worthwhile to go All In early on. But CI infrastructure for this stuff is a must-have and it might be tricky to get set up initially.
-glyph _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
I have merged some more fixes for mypy to Twisted trunk branch. In trunk, you can run mypy with: *tox -e mypy* Currently this results in *171* errors, which is way down from >1000 errors a month ago. In addition, if you look at any new PR's there is a *Mypy Ubuntu* job running on Azure pipeline, which runs mypy. Right now errors from this job are ignored and does not block the PR. However, if we can get the mypy errors down to zero, we can make mypy status a blocker for the PR. -- Craig
Hi Craig, On Tue, 23 Jun 2020 at 00:36, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
I have merged some more fixes for mypy to Twisted trunk branch.
In trunk, you can run mypy with:
*tox -e mypy*
Currently this results in *171* errors, which is way down from >1000 errors a month ago.
In addition, if you look at any new PR's there is a *Mypy Ubuntu* job running on Azure pipeline, which runs mypy. Right now errors from this job are ignored and does not block the PR. However, if we can get the mypy errors down to zero, we can make mypy status a blocker for the PR.
Thanks for working on this. Looking forward to have a real green mypy build. A general question: Why Twisted used Azure Devops and not GitHub actions? With Github Actions you can add a workload / pipeline with only GitHub access. There is no need for extra Azure permissions. I guess that with Github actions, if we run all the tests on Linux/Windows/MacOS we will very soon run out of time :) But maybe we can have the linters and other quick tests there. --- Adi Roiban
On Jun 23, 2020, at 5:34 AM, Adi Roiban <adi@roiban.ro> wrote:
Hi Craig,
On Tue, 23 Jun 2020 at 00:36, Craig Rodrigues <rodrigc@crodrigues.org <mailto:rodrigc@crodrigues.org>> wrote: I have merged some more fixes for mypy to Twisted trunk branch.
In trunk, you can run mypy with:
tox -e mypy
Currently this results in 171 errors, which is way down from >1000 errors a month ago.
In addition, if you look at any new PR's there is a Mypy Ubuntu job running on Azure pipeline, which runs mypy. Right now errors from this job are ignored and does not block the PR. However, if we can get the mypy errors down to zero, we can make mypy status a blocker for the PR.
Thanks for working on this.
Looking forward to have a real green mypy build.
A general question: Why Twisted used Azure Devops and not GitHub actions?
Azure Pipelines gave us substantially more parallel capacity than is available via Github Actions, which means we can make build statuses appear much sooner. Plus they support more platforms. For what it's worth, this build should just be "mypy", not "mypy ubuntu"; there should never be any differences between mypy on different platforms since it's statically analyzing the code. If there are, that sounds like it's mypy's problem, not ours, so it shouldn't be on our CI :). -glyph
On Wed, Jun 24, 2020 at 12:44 AM Glyph <glyph@twistedmatrix.com> wrote:
On Jun 23, 2020, at 5:34 AM, Adi Roiban <adi@roiban.ro> wrote:
Hi Craig,
On Tue, 23 Jun 2020 at 00:36, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
I have merged some more fixes for mypy to Twisted trunk branch.
In trunk, you can run mypy with:
*tox -e mypy*
Currently this results in *171* errors, which is way down from >1000 errors a month ago.
In addition, if you look at any new PR's there is a *Mypy Ubuntu* job running on Azure pipeline, which runs mypy. Right now errors from this job are ignored and does not block the PR. However, if we can get the mypy errors down to zero, we can make mypy status a blocker for the PR.
Thanks for working on this.
Looking forward to have a real green mypy build.
A general question: Why Twisted used Azure Devops and not GitHub actions?
Azure Pipelines gave us substantially more parallel capacity than is available via Github Actions, which means we can make build statuses appear much sooner. Plus they support more platforms.
Does Twisted have a special deal with Azure Pipelines? Or is the use of past-tense in this sentence intentional? :) Or are the docs for the respective platforms wrong/misleading? https://docs.microsoft.com/en-us/azure/devops/pipelines/licensing/concurrent... says free-tier public projects get 10 parallel jobs. https://help.github.com/en/actions/getting-started-with-github-actions/about... says free tier projects get 20 parallel jobs. (Of course this says nothing about the number of supported platforms.) Jean-Paul
On Jun 24, 2020, at 5:48 AM, Jean-Paul Calderone <exarkun@twistedmatrix.com> wrote:
On Wed, Jun 24, 2020 at 12:44 AM Glyph <glyph@twistedmatrix.com <mailto:glyph@twistedmatrix.com>> wrote:
On Jun 23, 2020, at 5:34 AM, Adi Roiban <adi@roiban.ro <mailto:adi@roiban.ro>> wrote:
Hi Craig,
On Tue, 23 Jun 2020 at 00:36, Craig Rodrigues <rodrigc@crodrigues.org <mailto:rodrigc@crodrigues.org>> wrote: I have merged some more fixes for mypy to Twisted trunk branch.
In trunk, you can run mypy with:
tox -e mypy
Currently this results in 171 errors, which is way down from >1000 errors a month ago.
In addition, if you look at any new PR's there is a Mypy Ubuntu job running on Azure pipeline, which runs mypy. Right now errors from this job are ignored and does not block the PR. However, if we can get the mypy errors down to zero, we can make mypy status a blocker for the PR.
Thanks for working on this.
Looking forward to have a real green mypy build.
A general question: Why Twisted used Azure Devops and not GitHub actions?
Azure Pipelines gave us substantially more parallel capacity than is available via Github Actions, which means we can make build statuses appear much sooner. Plus they support more platforms.
Does Twisted have a special deal with Azure Pipelines?
Yes. They set it up for us and boosted our capacity at PyCon 2019.
Or is the use of past-tense in this sentence intentional? :)
Also yes :). I am not aware of any changes to the default capacity of either offering since then and now, so I wanted to be clear that this was a decision that happened in the past.
Or are the docs for the respective platforms wrong/misleading?
I'm not sure what our total capacity is, I think it's 30, but I don't know how to check. I'm sure these are right though.
https://docs.microsoft.com/en-us/azure/devops/pipelines/licensing/concurrent... <https://docs.microsoft.com/en-us/azure/devops/pipelines/licensing/concurrent...> says free-tier public projects get 10 parallel jobs.
https://help.github.com/en/actions/getting-started-with-github-actions/about... <https://help.github.com/en/actions/getting-started-with-github-actions/about...> says free tier projects get 20 parallel jobs.
(Of course this says nothing about the number of supported platforms.)
Jean-Paul _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Wed, 24 Jun 2020 at 13:48, Jean-Paul Calderone <exarkun@twistedmatrix.com> wrote:
On Wed, Jun 24, 2020 at 12:44 AM Glyph <glyph@twistedmatrix.com> wrote:
On Jun 23, 2020, at 5:34 AM, Adi Roiban <adi@roiban.ro> wrote:
Hi Craig,
On Tue, 23 Jun 2020 at 00:36, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
I have merged some more fixes for mypy to Twisted trunk branch.
In trunk, you can run mypy with:
*tox -e mypy*
Currently this results in *171* errors, which is way down from >1000 errors a month ago.
In addition, if you look at any new PR's there is a *Mypy Ubuntu* job running on Azure pipeline, which runs mypy. Right now errors from this job are ignored and does not block the PR. However, if we can get the mypy errors down to zero, we can make mypy status a blocker for the PR.
Thanks for working on this.
Looking forward to have a real green mypy build.
A general question: Why Twisted used Azure Devops and not GitHub actions?
Azure Pipelines gave us substantially more parallel capacity than is available via Github Actions, which means we can make build statuses appear much sooner. Plus they support more platforms.
Does Twisted have a special deal with Azure Pipelines? Or is the use of past-tense in this sentence intentional? :) Or are the docs for the respective platforms wrong/misleading?
https://docs.microsoft.com/en-us/azure/devops/pipelines/licensing/concurrent... says free-tier public projects get 10 parallel jobs.
https://help.github.com/en/actions/getting-started-with-github-actions/about... says free tier projects get 20 parallel jobs.
(Of course this says nothing about the number of supported platforms.)
My understanding is that GitHub actions are free for public repositories. My suggestion is to use Azure Pipelines and Travis for the main trial tests and use Circle-CI or GitHub Actions for the other tests. GitHub Actions has a nice integration with a GitHub PR and you can check the results without having to navigate to a different page. And with GitHub actions you can add any new workflow without extra permission to Azure Devop. With GitHub actions for free and available on LInux/Windows/macOS , I am not sure if keeping Circle-Ci makes sense. -- Adi Roiban
On Thursday, 25 June 2020 14:18:04 CEST Adi Roiban wrote:
On Wed, 24 Jun 2020 at 13:48, Jean-Paul Calderone <exarkun@twistedmatrix.com> wrote:
On Wed, Jun 24, 2020 at 12:44 AM Glyph <glyph@twistedmatrix.com> wrote:
On Jun 23, 2020, at 5:34 AM, Adi Roiban <adi@roiban.ro> wrote:
Hi Craig,
On Tue, 23 Jun 2020 at 00:36, Craig Rodrigues <rodrigc@crodrigues.org>>> wrote:
I have merged some more fixes for mypy to Twisted trunk branch.
In trunk, you can run mypy with:
*tox -e mypy*
Currently this results in *171* errors, which is way down from
1000 errors a month ago.
In addition, if you look at any new PR's there is a *Mypy Ubuntu* job running on Azure pipeline, which runs mypy. Right now errors from this job are ignored and does not block the PR. However, if we can get the mypy errors down to zero, we can make mypy status a blocker for the PR.>> Thanks for working on this.
Looking forward to have a real green mypy build.
A general question: Why Twisted used Azure Devops and not GitHub actions?
Azure Pipelines gave us substantially more parallel capacity than is available via Github Actions, which means we can make build statuses appear much sooner. Plus they support more platforms.
Does Twisted have a special deal with Azure Pipelines? Or is the use of past-tense in this sentence intentional? :) Or are the docs for the respective platforms wrong/misleading?
https://docs.microsoft.com/en-us/azure/devops/pipelines/licensing/co ncurrent-jobs?view=azure-devops says free-tier public projects get 10 parallel jobs.
https://help.github.com/en/actions/getting-started-with-github-actio ns/about-github-actions#usage-limits says free tier projects get 20 parallel jobs.
(Of course this says nothing about the number of supported platforms.) My understanding is that GitHub actions are free for public repositories.
My suggestion is to use Azure Pipelines and Travis for the main trial tests and use Circle-CI or GitHub Actions for the other tests. GitHub Actions has a nice integration with a GitHub PR and you can check the results without having to navigate to a different page. And with GitHub actions you can add any new workflow without extra permission to Azure Devop.
With GitHub actions for free and available on LInux/Windows/macOS , I am not sure if keeping Circle-Ci makes sense.
One problem with the Circle CI runs for Twisted that I ran into recently is that it won't let me view the results of runs unless I grant it a bunch of permissions it doesn't need, including read/write access to all my repositories. Since that is an unnecessary security risk, I refused, but that does mean that I can't view some of the CI results, which is a pain when one of those checks fails. Bye, Maarten
On 2020-06-24 00:43, Glyph wrote:
On Jun 23, 2020, at 5:34 AM, Adi Roiban <adi@roiban.ro> wrote:
Hi Craig,
On Tue, 23 Jun 2020 at 00:36, Craig Rodrigues <rodrigc@crodrigues.org> wrote: I have merged some more fixes for mypy to Twisted trunk branch.
In trunk, you can run mypy with:
tox -e mypy
Currently this results in 171 errors, which is way down from >1000 errors a month ago.
In addition, if you look at any new PR's there is a Mypy Ubuntu job running on Azure pipeline, which runs mypy. Right now errors from this job are ignored and does not block the PR. However, if we can get the mypy errors down to zero, we can make mypy status a blocker for the PR.
Thanks for working on this.
Looking forward to have a real green mypy build.
A general question: Why Twisted used Azure Devops and not GitHub actions?
Azure Pipelines gave us substantially more parallel capacity than is available via Github Actions, which means we can make build statuses appear much sooner. Plus they support more platforms.
Just noticed we don't actually link to Azure from the readme. Presently we have 14 builds there so within the tense pickiness :] we don't get any benefit (yet). I'm curious though, what additional platforms does Azure get us?
On Jun 24, 2020, at 12:45 PM, Kyle Altendorf <sda@fstab.net> wrote:
On 2020-06-24 00:43, Glyph wrote:
On Jun 23, 2020, at 5:34 AM, Adi Roiban <adi@roiban.ro> wrote:
Hi Craig,
On Tue, 23 Jun 2020 at 00:36, Craig Rodrigues <rodrigc@crodrigues.org> wrote: I have merged some more fixes for mypy to Twisted trunk branch.
In trunk, you can run mypy with:
tox -e mypy
Currently this results in 171 errors, which is way down from >1000 errors a month ago.
In addition, if you look at any new PR's there is a Mypy Ubuntu job running on Azure pipeline, which runs mypy. Right now errors from this job are ignored and does not block the PR. However, if we can get the mypy errors down to zero, we can make mypy status a blocker for the PR.
Thanks for working on this.
Looking forward to have a real green mypy build.
A general question: Why Twisted used Azure Devops and not GitHub actions? Azure Pipelines gave us substantially more parallel capacity than is available via Github Actions, which means we can make build statuses appear much sooner. Plus they support more platforms.
Just noticed we don't actually link to Azure from the readme. Presently we have 14 builds there so within the tense pickiness :] we don't get any benefit (yet).
If 3 PRs are building at the same time, we see a benefit.
I'm curious though, what additional platforms does Azure get us?
Oh, never mind! Last I was investigating this Actions was Linux-only; looks like they've really improved a lot! -g
On 25/6/20 5:50 am, Glyph wrote:
On Jun 24, 2020, at 12:45 PM, Kyle Altendorf <sda@fstab.net> wrote:
On 2020-06-24 00:43, Glyph wrote:
On Jun 23, 2020, at 5:34 AM, Adi Roiban <adi@roiban.ro> wrote:
Hi Craig,
On Tue, 23 Jun 2020 at 00:36, Craig Rodrigues <rodrigc@crodrigues.org> wrote: I have merged some more fixes for mypy to Twisted trunk branch.
In trunk, you can run mypy with:
tox -e mypy
Currently this results in 171 errors, which is way down from >1000 errors a month ago.
In addition, if you look at any new PR's there is a Mypy Ubuntu job running on Azure pipeline, which runs mypy. Right now errors from this job are ignored and does not block the PR. However, if we can get the mypy errors down to zero, we can make mypy status a blocker for the PR.
Thanks for working on this.
Looking forward to have a real green mypy build.
A general question: Why Twisted used Azure Devops and not GitHub actions? Azure Pipelines gave us substantially more parallel capacity than is available via Github Actions, which means we can make build statuses appear much sooner. Plus they support more platforms.
Just noticed we don't actually link to Azure from the readme. Presently we have 14 builds there so within the tense pickiness :] we don't get any benefit (yet).
If 3 PRs are building at the same time, we see a benefit.
I'm curious though, what additional platforms does Azure get us?
Oh, never mind! Last I was investigating this Actions was Linux-only; looks like they've really improved a lot!
-g
IIRC, they share infrastructure now, due to the GitHub-Microsoft thing. - Amber
participants (7)
-
Adi Roiban -
Amber Brown (hawkowl) -
Craig Rodrigues -
Glyph -
Jean-Paul Calderone -
Kyle Altendorf -
Maarten ter Huurne