Mailman 3 July 2019 - Twisted

[Twisted-Python] Need some enlightenment on using web client properly, or maybe nudge a bug to get fixed
by Jarosław Fedewicz July 11, 2019

July 11, 2019

I have written a simple service which takes data from network, massages it until it's useful enough, and sends the results out periodically via HTTP to an API. It all works for a while, then I get an error like this approximately 40 minutes into the service's uptime: ResponseNeverReceived: [<twisted.python.failure.Failure OpenSSL.SSL.ZeroReturnError: >] Then a couple more like this: ResponseNeverReceived: [<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.>] Then it ends with TimeoutError: User timeout caused connection failure. Then every request results in the same TimeoutError. I don't know if using HTTPS important in this case. Restarting the whole service, of course, makes the problem go for a while. The other side is the Slack API, so I rather assume it's not very much to blame, it can be demonstrated to work rather reliably, all its criticisms notwithstanding. I cannot yet tell if this bug is a function of uptime, or the number of requests made. I have tried to work around the problem by discarding the agent object, and using an HTTPConnectionPool with persistent=False, but it didn't help at all. I think it made the problem worse because the framework seems to refer to some objects the Agent creates, and the process becomes a CPU hogs in a couple hours (with the TimeoutErrors still happening all the time). The closest I've got on the internets which describes a similar problem, apart from people complaining on StackOverflow about precisely this to happen when they are using Scrapy, is this blog post from almost a decade ago: http://www.chris-wong.net/twisted-web-framework-user-timeout-caused-connect… . There could be a small chance I'm holding it wrong(tm), but maybe there exists a ticket, just worded differently, which could help me get to the bottom of it. -- Yaroslav Fedevych IT Philosopher

4 4

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com July 6, 2019

July 6, 2019

1 0

[Twisted-Python] IDelayedCall.reset does not work with AsyncIOReactor
by notafile July 5, 2019

July 5, 2019

Hello, I thought I'd follow the commenters recommendations and crosspost on the mailing list after all. When switching to AsyncIOReactor a while back, we discovered that calling .reset() on a delayed call was just a noop when using that reactor. We've since worked around it, but it would be very neat to see this fixed in twisted directly. Exarkun on IRC mentioned that this was a very obvious mistake, but I'm personally not really familiar enough with twisted internals to write a patch for it. The relevant ticket is https://twistedmatrix.com/trac/ticket/9611. - nota

1 0

[Twisted-Python] Getting nice logging outputs in py.test
by Thomas Westfeld July 5, 2019

July 5, 2019

Dear all, I am having an issue with logging of my Twisted application. I am using the new twisted.logger framework. The output produced by twist or twistd when runnning my plugin is nice. However when running a test suite using py.test I only get this as an output to stdout: {"log_io": "\"127.0.0.1\" - - [04/Jul/2019:13:12:04 +0000] \"GET /index HTTP/1.1\" 401 - \"-\" \"python-requests/2.22.0\"", "log_logger": {"unpersistable": true}, "log_level": {"name": "info", "__class_uuid__": "02e59486-f24d-46ad-8224-3acdf2a5732a"}, "log_namespace": "twisted.python.log", "log_source": null, "log_format": "{log_io}", "log_time": 1562245925.0769854, "log_flattened": {"log_io!s:": "\"127.0.0.1\" - - [04/Jul/2019:13:12:04 +0000] \"GET /index HTTP/1.1\" 401 - \"-\" \"python-requests/2.22.0\"", "log_io!:": "\"127.0.0.1\" - - [04/Jul/2019:13:12:04 +0000] \"GET /index HTTP/1.1\" 401 - \"-\" \"python-requests/2.22.0\""}} I tried to initialize a log observer like this, to get it better formatted: globalLogBeginner.beginLoggingTo([textFileLogObserver(sys.stdout)]) But it does not work. I did this in a conftest.py file, but it did not produce the desired input. Any idea how I can produce a nice log output to stdout? Thanks in advance, Thomas

4 8