hi there, folks:
I'd really like to release 0.7.0 but I would like it to be at least a
little bit tested before I do so. Could those of you with CVS trees check
everything out and see if it performs as advertised? Deeper bugs than
that will have to wait for the next release, but I'd at least like to know
if it works for someone other than me.
______ __ __ _____ _ _
| ____ | \_/ |_____] |_____|
|_____| |_____ | | | |
@ t w i s t e d m a t r i x . c o m
I've uploaded a prerelease of Nevow 0.14.3 to PyPI. This contains a fix
for some Athena client-side timeout behaviour (or more specifically, the
lack of one) and removes use of FilePath.getmtime.
If you are using Athena, please a) let me know! and b) give this prerelease
a try to see if anything breaks. As the de facto Nevow release manager, and
one of its last users, I'd like to avoid going overboard on release
engineering if it isn't necessary, but I'd also like to avoid making life
worse for any other remaining users, so it would be useful to hear from any
of you out there still using Nevow.
I use pytest to write integration test automation. If the tests are stripped
down to their core, they are basically using different network connections and
protocols to drive multiple applications. Tests use the following Python
I use pytest fixtures heavily. Each of the aforementioned libraries is used
such that each connection (say SSH) is provided to tests through that fixture.
This makes writing tests quite simple as a single object can be used to
maintain a network connection that can be reused across multiple tests in the
same session. In case the connection needs to be re-established, the class has
methods to hide that detail from the tests and things just work.
Sample code is presented below:
# In conftest.py
from path_to_lib.ssh import SSH
# In path_to_lib/ssh.py
def __init__(self, host, port, username, pem):
self.host = host
self.port = port
self.username = username
self.pem = pem
self.client = paramiko.SSHClient()
self.transport = None
def exec_cmd(self, cmd):
_, stdout, stderr = self.client.exec_command(
_, stdout, stderr = self.client.exec_command(
return stdout, stderr
# In path_to_tests/test_foo.py
cmd = "hostname"
stdo, stde = ssh_fixture.exec_cmd(cmd)
# In path_to_tests/test_blah.py
cmd = "ls /"
stdo, stde = ssh_fixture.exec_cmd(cmd)
The above sample code is essentially how all other libraries (say for Mongo)
are used in my tests. Each type of protocol is presented to tests through a
fixture and tests only need to call certain methods without having to know
the gory details behind them.
This method provides very readable code. It also makes writing tests much
In addition to using these libraries sometimes I need to use the socket library
to talk to proprietary services using TCP and/or UDP that don't have open
source libraries available.
The problem is that I have a *lot* of these integration tests that can run for
a long time (anywhere from 2 hours to 12 hours based on the test set selected)
(I need to test a *lot* of services and components). This is causing some
issues when I get my hands dirty with the socket library.
Instead of writing a custom and badly implemented Twisted alternative I thought
I could use Twisted to replace a bunch of these libraries. Of course, I can't
replace all of them at once today but I'd much rather start on the path.
As I started getting familiar with the official and community docs I realized
that most of them deal with writing servers (and sometimes clients). I can't
find any docs on how people in my situation use Twisted.
>From what I understand -- forgive my beginner-level grasp of the subject
matter -- once the reactor starts no more listenTCP() or connectTCP()
(for example) can be added. And until the reactor stops code execution does not
progress beyond ``reactor.run()``. For example, in the code below, _Done_ is
not printed until there's a ``reactor.stop()`` somewhere above it.
My questions for the community:
* How can I use Twisted that fits the way my tests are written?
* If I have to rewrite my tests to fit the Twisted model, do I start the
reactor every time for each test and stop it at the end?
* Are there any known issues with start/stop reactor multiple times in a
single, long-running process?
* Are there any good examples of using Twisted in integration tests that I
have failed to find?
Thanks in advance.
Hello Twisted Project Members (i.e.: committers, people with repo:write access),
Whenever you get a code review from a non-member, it's important to always ensure that they remove the 'review' keyword. This is not really important in and of itself, but it is very important for other reasons, which I will attempt to explain, since I think I may not have properly laid this out sufficient detail before.
The goal of allowing non-project-members (i.e.: external contributors) to review changes in Twisted is to provide a pipeline for folks to become developers of the project; to allow people to gain experience with reviewing, so they can proceed to being full reviewers, without requiring that they already be project members before they can practice. In other words, it is a process to train reviewers so that we can have higher code-review throughput in general in the future, not a shortcut to get changes reviewed more quickly.
It is therefore the responsibility of the developer accepting the code review to ensure that the reviewer is following directions and has given sufficient attention to the review, and to provide feedback on those reviews to ensure that the new contributors are learning how to look for the things that help us maintain Twisted's quality.
Sometimes, the only code review you need is a "thumbs up, looks good", because everything really is fine, the fix is simple, and there's nothing much the reviewer could say. Ideally, all changes are small, and obvious, and reviewers can just give that review regularly.
However, such a review is always suspicious, because it's very easy for a reviewer, especially an external reviewer with little understanding of the project, to just rubber-stamp a fix they want to see merged quickly without giving it any real analysis.
As such, it's very important that when accepting a very short review that you use whatever clues you have at your disposal that the reviewer has actually reviewed the code in some detail.
One potential proxy for this is that our process requires some small peculiarities, such as attaching and removing the 'review' keyword in Trac. If a contributor has not even read the instructions carefully enough to know that they need to remove this keyword, it is virtually impossible for them to have dedicated the attention and care required by https://twistedmatrix.com/trac/wiki/ReviewProcess#Howtobeagoodreviewer <https://twistedmatrix.com/trac/wiki/ReviewProcess#Howtobeagoodreviewer> to do a full review and consider everything that cannot be automatically verified, such as impact to Twisted's public interface, the legibility of documentation, duplication of features, et cetera.
If the contributor has demonstrated their attention to detail in some other way, by giving a thorough review and writing some analysis of the change's implications (even if it's not actionable, just something like "I read over the narrative documentation to ensure it didn't need any changes and it looks like we're good") then it might be acceptable to gently point out the small process mistake of not updating trac. But if the change is just a Github comment saying "LGTM" with nothing else associated, it is important not to accept the review and to explain that the reviewer should re-read the directions explaining how to give a review.
As you might have guessed, I have seen "LGTM" changes landing on trunk recently whose reviews looked somewhat suspicious to me. I, and a few other long-time Twisted developers, have complained about this before, but in re-reading those complaints in preparation for writing this message, it occurred to me that I may not have spelled out exactly what I was concerned with and why, so, in the spirit if blameless post-mortems, I don't want to call out anyone specifically.
Pascal's apology applies; I haven't had much time to edit.
Thanks for reading,
: Iron-clad process rigidity is not the goal here. Sometimes there are legitimate reasons to need to move things along a little faster than a full thorough review requires. For example, if there's a pressing security issue, or all the CI infrastructure is failing due to a bug in one of our dependencies. There are also cases where the important thing that needs to be done in a review is a highly specialized analysis, like; for example, if we need a GPS expert to verify that some change to twisted.positioning is correct, and beyond the domain knowledge the code change is otherwise trivial. In these cases it might be fine to accept a "LGTM", if the question posed in the review is highly specific, but in that case, it's important to leave a comment explaining what exactly has transpired: "I'm removing the review keyword myself because I needed a domain expert to review this, and XXX YYY was helpful enough to have a look but is not familiar with our process." or whatever the case may be.
In general, putting more thorough comments on tickets is always a good thing. However clear that you as a Twisted project insider think a thing is, chances are it is about 1/4 that clear to anyone trying to understand project direction from the outside. I'm certainly guilty of under-documenting myself :-).
: we are not preserving Trac purely as a shibboleth to check on people installing the directions; software development is hard enough without extra roadblocks thrown in the way. There are just too many tools connected to the 'review' keyword right now that are hard to connect to Github automatically. Whatever our process, however, it's important to verify that reviewers are following it carefully.
: I realize that there are too many bits of identifying information here to be meaningfully anonymous, so some folks will immediately know the specific ticket that precipitated this, but don't be a jerk and dig up someone's name from the trac timeline because you can; the goal here is not (and should not) be to throw anyone under the bus, but to improve communication around project policy.
I am experiencing an issue with my twistd plugin. It is in a project_plugin.py
file in the twisted/plugins directory of my project.
Calling "twistd --help" works fine up to twistd version 16.3.2 but in 16.4.0
the plugin is not recognized anymore.
This is how it looks like:
from zope.interface import implementer
from twisted.application.service import IServiceMaker
from twisted.application import internet
from twisted.plugin import IPlugin
from twisted.python import usage
from myfile import MyFactory
optParameters = [["port", "p", 8000, "The port number to listen on."],
["my-hostname", "l", "", "the hostname of other
["my-port", "i", 9000,"the port of the other
tapname = "myserver"
description = ""
options = Options
def makeService(self, options):
serviceMaker = MyServiceMaker()
Any help is appreciated.
I'm getting this report under pypy2 5.8.0 running Twisted trunk from github:
Traceback (most recent call last):
File "/opt/twisted/src/twisted/test/test_plugin.py", line 566, in
File "/opt/twisted/src/twisted/trial/_synctest.py", line 492, in assertIn
% (containee, container))
twisted.trial.unittest.FailTest: 'one' not in ['dev', 'app']
Traceback (most recent call last):
File "/opt/twisted/src/twisted/plugin.py", line 171, in getCache
provider = pluginModule.load()
File "/opt/twisted/src/twisted/python/modules.py", line 392, in load
File "/opt/twisted/src/twisted/python/reflect.py", line 312, in namedAny
obj = getattr(obj, n)
exceptions.AttributeError: 'module' object has no attribute 'stale'
I'm not sure yet if it works under pypy3.
My initial tracings seem to suggest this is somehow due to __import__
under pypy not finding the stale.pyc where no stale.py exists ... but
I have to admit that at this point if this is actually expected, all I
know is the test passes under python 2.7.13 and if I comment out the
mypath.remove() on line 562...
Willing to do more work to get to the bottom of this, just thought I'd
put this out there to see if I could get any clues from the more
experienced before I dive in. There seem to be some old tickets that
could vaguely relate...
Am I correct in the fact that Twisted doesn't have Travis-CI builds on PyPy?
Maybe there's a reason for this I'm missing...
Thanks in advance for any info or clues,
Daniel Sutcliffe <dansut(a)gmail.com>