Mailman 3 March 2023 - Twisted

review needed: better type hints for FilePath, Deferred
by Glyph 21 Mar '23

21 Mar '23

I've got two PRs up right now which pretty significantly enhance Twisted's type annotations in two core areas: FilePath and Deferred. In FilePath's case, things were previously entirely un-annotated. With the PR I have up, FilePath is now generic, i.e. FilePath[str] or FilePath[bytes] so you can clearly see what the type of `.path` (and friends) are and get proper type checking. I'm writing some FilePath intensive code in an application right now and this is really handy and has caught several minor bugs for me already. (And because I've gotta make CI happy, this includes ZipPath…) https://github.com/twisted/twisted/pull/11823 #11822 filepath annotations by glyph · Pull Request #11823 · twisted/twisted github.com For Deferreds, there are just a bunch of places where type-checking silently fails; for example, addCallback and addErrback don't check their arguments at all. Perhaps more saliently to modern code, Deferred.fromFuture() wasn't specializing its result, so you'd lose all type-checking across an asyncio boundary. And maybeDeferred() just returned a nonsense result of Deferred[Deferred] so it was effectively unusable. https://github.com/twisted/twisted/pull/11770 #11772 Correct deferred annotations by glyph · Pull Request #11770 · twisted/twisted github.com Both are passing tests, have full coverage, and have had at least a little bit of feedback, so they should be relatively easy reviews. If you've got some application code that you would like to type-check against these to see if they spot any bugs for you, that would be great, but if we I could get these approved, this type of comprehensive type-safety upgrade would make a nice tentpole for our first 2023 release. -g

2 2

posix_spawn would really speed up spawnProcess if we used it
by Glyph 16 Mar '23

16 Mar '23

I am writing some code that was spawning a lot of processes with spawnProcess, and it was going a bit slower than I expected; it was taking about 51 seconds on my M1 Max mac. So I hopped over to my local copy of Twisted and applied this patch: 1 file changed, 9 insertions(+) src/twisted/internet/process.py | 9 +++++++++ modified src/twisted/internet/process.py @@ -367,6 +367,15 @@ class _BaseProcess(BaseProcess): @type environment: L{dict}. @param kwargs: keyword arguments to L{_setupChild} method. """ + from os import posix_spawn, POSIX_SPAWN_DUP2, POSIX_SPAWN_CLOSE + fdmap = kwargs.get("fdmap", {}) + actions = [ + (POSIX_SPAWN_DUP2, childFD, parentFD) + for (parentFD, childFD) in fdmap.items() + ] + self.pid = posix_spawn(executable, args, environment, file_actions=actions) + return + collectorEnabled = gc.isenabled() gc.disable() try: This is obviously not a complete reimplementation and might have some slight performance advantage due to its incompleteness, but the task in question then took 16 seconds; i.e. a 319% speedup. Anyone want to do this implementation for real? Or want to commit to a review, if I did it? -g

2 3

Weird "bugs" in EPollReactor? You may be using the reactor from a thread by mistake
by Chris Withers 15 Mar '23

15 Mar '23

Hi All, So, this was "fun"... Eventually tracked down a bug in Twisted based application that uses Autobahn for websockets, some AMP stuff and a good amount of other Twisted stuff, but that has to call a bunch of blocking code, some of it written by users of the app, so we want to insulate them from thinking about Deferreds or anything async. We saw weird KeyErrors, ENOENT, "Reactor was unclean" during shutdown in trial-based test suites and generally head-in-sanded it or grumbled about CI instability due to the environment in which the tests were running. This went on for a couple of years... no kidding... Just recently, finally spotted the problem... a bunch of handler code was being deferToThread'ed so that it didn't accidentally block the reactor, BUT it was also calling sendMessage [1]... Big "d'oh" when that penny finally dropped... Moving that code to using blockingCallFromThread magically fixed all the "reactor bugs" and "CI environment instability", so wanted to write something up here for future self or google hunters... More over on Stack Overflow: https://stackoverflow.com/a/75674510/216229 cheers, Chris [1] https://autobahn.readthedocs.io/en/latest/reference/autobahn.websocket.html…

1 0

treq digest_auth
by jjaherne＠gmail.com 13 Mar '23

13 Mar '23

I'm not sure tis is the right p[lace to ask this question but here goes. I see that digest_auth was added to treq last November/December. I can't see any way to get a copy of the code so I can run some tests to see if it fixes a problem I have. Anyone point me in the right direction.

2 2