hi there, folks:
I'd really like to release 0.7.0 but I would like it to be at least a
little bit tested before I do so. Could those of you with CVS trees check
everything out and see if it performs as advertised? Deeper bugs than
that will have to wait for the next release, but I'd at least like to know
if it works for someone other than me.
______ __ __ _____ _ _
| ____ | \_/ |_____] |_____|
|_____| |_____ | | | |
@ t w i s t e d m a t r i x . c o m
I'm investigating installing the Evennia MU* server on Windows. We use
Twisted and will be requiring Python3.7 in our next release. I need to make
easy-to-use install instructions since a lot of Windows users use our
I tested with a Windows7 64bit VM and installed everything from scratch to
emulate what a non-dev Windows user would see. I don't have Windows10 so
can't compare to the install experience there (but Windows7 64bit is still
relevant, having something like 24% of the active Windows user-base
according to Steam).
Using pip to install Evennia, at the Twisted requirement install step I run
into an error telling me that I need "Microsoft Visual C++ build tools"
from the URL https://visualstudio.microsoft.com/downloads. The first issue
is that there does not appear to be any build-tools package named like that
on that page or sub-page (at least not what I could find after digging
around). I tried to install a few similarly-named packages, like "Visual
Studio Build Tools", but had no luck getting past the Twisted install
point. So that recommendation-string should likely be updated.
The thing is though, while I could probably personally figure out how to
set it up eventually, our Windows users are likely the least tech-savvy of
our users. Requiring them to set up a compiler environment (despite us
telling them that Python code does not need compilation) a bit too much.
It seems Twisted has distributed binary Windows wheels in the past, would
it be possible to get them again? Or should I recommend some other,
specific install procedure for our Windows users?
Griatch, Evennia dev
I am experiencing an error when trying to use the quick usage example posted on the ldaptor github page https://github.com/twisted/ldaptor . I have included the starttls command to promote the connection to an encrypted one, but I am experiencing an deprecation warning:
DeprecationWarning: LDAPStartTLSInvalidResponseName.__str__ method is deprecated and will not be used for getting bytes representation in the future releases, use LDAPStartTLSInvalidResponseName.toWire instead
This puzzles me as I cannot track this error down. The usual strategy to introduce debugging print statements or start the debugger like "import pdb; pdb.set_trace()" does not work. I would appreciate help here how to debug this deferred code.
Besides this warning I am receiving no output whatsoever. Also print statements in the example method do not print anything, which is strange to me.
To debug it on a networking level, I have wiresharked the ldap communication and it seems fine (see below):
Transmission Control Protocol, Src Port: 53964, Dst Port: 389, Seq: 1, Ack: 1, Len: 31
Lightweight Directory Access Protocol
protocolOp: extendedReq (23)
requestName: 126.96.36.199.4.1.1466.20037 (LDAP_START_TLS_OID)
[Response In: 2587]
Transmission Control Protocol, Src Port: 389, Dst Port: 53964, Seq: 1, Ack: 32, Len: 46
Lightweight Directory Access Protocol
protocolOp: extendedResp (24)
resultCode: success (0)
responseName: 188.8.131.52.4.1.1466.20037 (LDAP_START_TLS_OID)
[Response To: 2581]
[Time: 0.245555000 seconds]
I have posted the example code with the starttls command in this gist:
Thanks for your help!
I just migrated some quite basic twisted code from Python 2.7 to Python 3. Everything worked quite nicely, the main issue was the correct handling of strings vs. bytes.
There was a mean error I made when using the Twisted Agent http client: I specified the method as a Unicode string, instead of using Agent.request(b'GET',b'http://whatever.com'). The mean thing is that the code does not produce any error. It connects to the server via TCP but then stops.
It is given correctly in the examples of the docs and also in the source code, but wouldn't it make things easier to make a type check in the request method if the method parameter is of type Unicode and raise an error?
Quick intro - I am new to Python, and come from Java background (with some
exposure to NIO). I am part of a small team setup for testing of some 5G
telecom network software that have micro-service decomposition and the
components communicate with each other using REST interface over HTTP/2.0
>From my reading so far, I've come to realize that they don't use the
edge-HTTP/2.0 approach i.e. delegate HTTP/2.0 termination to nginx as proxy
then talk HTTP/1.1 internally, but each component is a HTTP/2.0 endpoint
(client and server as they switch roles).
My understanding is that Twisted framework supports HTTP/2.0 via hyper-h2.
>From a very quick read of the Twisted docs (and some stackoverflow Q&A) I
get the impression that it is possible to use Flask as the RESTful
web-service framework using HTTP/2.0 for the server-end, but perhaps there
is no Twisted framework provided HTTP/2.0 client at similar level of
abstraction, and one perhaps needs to use hyper-h2 at more nuts-n-bolts
level. I hope I am completely wrong on the latter part.
If someone could shed some clarity on the above, and if possible point to
some relevant tutorials and samples that may help me get started would be
After filing https://github.com/tornadoweb/tornado/issues/2636 <https://github.com/tornadoweb/tornado/issues/2636> recently, I was reminded that Twisted should support asyncio seamlessly, and currently we have some quite-visible seams.
There are three major use-cases for asyncio integration:
I've got a large Twisted application. I run the reactor at startup. I find a cool asyncio lib. I want to use it. What do I do?
I've got a large asyncio application. I run the main loop at startup. I find a cool twisted lib. I want to use it. What do I do?
I'm noodling around in an environment, like a Jupyter notebook, which already happens to have an event loop but I don't really know which one it is. How do I have the fewest number of steps?
What happens today in each case?
Case 1: If I've got a large Twisted application, I either start off by doing Deferred.fromFuture or ensureDeferred. These give me Deferreds that never fire, because no asyncio loop is running, but they don't yell at me; they just hang. Now I have to switch my event loop over to be an AsyncioSelectorReactor. Except wait, my application is a GTK+ application, and the only GTK+ main loop I can get that implements the asyncio APIs is unmaintained! So I either give up my custom Twisted reactor or I give up my asyncio functionality or I run it all in a thread. None of these are great experiences.
Case 2: if I've got an asyncio application, I have a similar problem; I need a reactor, but one isn't running, so all my Deferred.asFuture(get_event_loop())s just hang. So I need to run AsyncioSelectorReactor, but now I need to know a bunch of obscure Twisted trivia to boostrap all of this properly:
Case 3: oh no, how do I even know which one I need to run?
In all of these cases I need to know a bunch of really inane trivia:
I need to call startRunning() on the reactor, but just once, when it gets set up, or threadpool invocations (such as name resolution) will hang. Unless I'm lucky enough to actually control the whole process startup, in which case I can replace my event loop's .run_forever() with Twisted's .run().
I need to know whether I need subprocess support, and from whom. If I need it in Twisted, I need to do startRunning(installSignalHandlers=True); if I need it in asyncio, I need to do installSignalHandlers=False. I don't think there's a way to have both.
I'd better hope that this is not on Windows, because Twisted is going to use its POSIX socket I/O implementation no matter what.
Then, once it's up and running, rather than just 'await'-ing Deferreds, I always need to await someDeferred().asFuture(loop=get_event_loop()). From a practical perspective, get_event_loop() is the only correct value that I'd ever want to pass here, but for some reason the library makes me pass it every time.
I propose a series of changes that would make this seamless from either side.
Make Deferreds awaitable by asyncio by just calling `get_event_loop` and lying about what loop they're connected to. My reading of this is that https://github.com/python/cpython/blob/c5c6cdada3d41148bdeeacfe7528327b481c… <https://github.com/python/cpython/blob/c5c6cdada3d41148bdeeacfe7528327b481c…> will totally let us do that, if we just have a get_loop method (or, gross, _loop property) on Deferred or whatever's returned from Deferred.__await__. It would be fine if this raised a warning or something, as long as it worked in the 80% case so people could get some initial success and a pointer as to how to succeed, and not just hitting a wall with "task got bad yield".
Make the reactor automagical.
Phase one of this change would be: when you do 'from twisted.internet import reactor', if there's already an asyncio loop installed and running, automatically select the asyncio integration. This only helps you if you're in a context like a Jupyter notebook where you're not doing it at the module level, but that's still interesting.
Make 'twisted.internet.reactor' into a dynamic proxy object which forwards reactor calls to whichever the running reactor is at the moment of the method call (connectTCP, callLater, etc). This can move the reactor selection to whenever the "first touch" on the reactor is, rather than whenever it's imported. (This also fixes a ton of of annoying import-order stuff in Twisted itself, as a bonus.)
Automatically call startRunning() as necessary if another loop is in charge.
Fix the subprocess integration:
As a simple first step, for UNIX, at least participate in the asyncio get_child_watcher() / set_child_watcher() protocol, so that at least someone trying to coordinate a Twisted loop and an asyncio loop can intentionally select which one gets child process termination notifications, and possibly even multiplex this.
Fix subprocesses along with any platform-specific socket quirks, by doing the next step...
Actually use the asyncio APIs in the "asyncio side down" integration, i.e. AsyncioSelectorReactor. Presently we implement everything in terms of the add_reader and add_writer APIs, which is both very low-level and also fairly UNIX-specific. We should instead be using loop.create_connection, loop.create_server, loop.subprocess_exec, loop.getaddrinfo, etc, and translating between asyncio protocol/transport APIs and our own.
Implement the "twisted side down" integration with asyncio; i.e. instead of implementing the twisted APIs in terms of asyncio's interfaces, implement the asyncio APIs in terms of Twisted's interfaces, so we can use existing custom reactors, GUI loop integration, etc.
This is all quite a bit of work but I think it would massively improve the experience of a novice trying to adopt Twisted in a modern Python stack.
In particular, I think that in addition to being a good example of the general problem domain, Jupyter is quite specifically incredibly strategic, and having the ability to just grab Treq and start doing massively parallel I/O to ingest data, then just 'await' on it, would be a very powerful demonstration of Twisted's capabilities.
Let me know what you all think!
On some busy instances of the app I'm working with/on, we see websocket
connections being dropped due to not ping/ponging in time, I'm fairly
sure this is as a result of some user-written callbacks blocking the
reactor for long periods of time (seconds...).
What's the best way to debug a twisted application to find out what's
blocking the reactor?