I am plannig to upgrade twisted's buildbot in the near future. In
preperation for that, I am going to take down the production buildbot
on Tuesday, June 26, around 1600 UTC.
During that time, the result of any builds done will be lost. Thus, I'd
ask that people avoid committing to trunk during that time.
I will send a reminder before I take down the buildbot both here, and on
#twisted and #twisted-dev.
hi there, folks:
I'd really like to release 0.7.0 but I would like it to be at least a
little bit tested before I do so. Could those of you with CVS trees check
everything out and see if it performs as advertised? Deeper bugs than
that will have to wait for the next release, but I'd at least like to know
if it works for someone other than me.
______ __ __ _____ _ _
| ____ | \_/ |_____] |_____|
|_____| |_____ | | | |
@ t w i s t e d m a t r i x . c o m
Last week I did some work and updated twistedchecker
so that it uses pylint >= 2.4.4. Before that, it was using a really old
twistedchecker is run over the Twisted code as part of CI, and reports
various style issues. Now that twistedchecker is using a newer pylint,
it should be compatible with newer Python language features.
Now that Twisted supports Python 3.5+ (with Python 2.7 support being
is it OK to start using newer features of the Python language in the
such as type hints ( https://docs.python.org/3/library/typing.html )?
I don't have experience with that feature, but it looks like it could be
when combined with a tool like mypy. Does anyone else have opinions on
type hints and mypy?
I'm looking at a fix for bug <https://twistedmatrix.com/trac/ticket/9804>
(Cannot load a PEM certificate with Unicode in subject). The underlying
problem is that the DistinguishedName class can't handle non-ascii AVAs.
The fix I've made simply avoids creating DistinguishedName instances when
it isn't necessary, but that leaves the question of what to do with the
class. I think that the best thing to do is to deprecate the class
entirely and replace it with simpler API.
Reasons I think that the DN class is broken:
- The values in a certificate are conceptually text-strings, not
byte strings; they may be in ASCII, UTF8, UTF16, or several
other encodings. However
- DN represents these textual values as `bytes` instead of `str`
- DN can't handle non-ASCII-representable values at all, even if
the user never tries to access that value
- It can only handle a subset of the attribute-assertions found in
a PKIX DN; there's no escape hatch for others (e.g. OID keys or
- It can't represent the full structure of a DN (specific ordering,
multiple-value RDNs, AVAs whos values aren't textual, etc.) ---
these are not common in the PKIX world but they are valid
What I propose as an alternative:
- Replace APIs that take `DistinguishedName` classes with ones that take
`Union[OpenSSL.crypto.X509Name, dict]` where the `dict` format is parsed
with the same convenience semantics as DistinguishedName, except that
values are `str`
- Replace APIs that return `DistinguishedName` with ones that return
OpenSSL.crypto.X509Name, which is already fairly convenient to use
(e.g. it has attributes for retrieving/setting commonName
and so on without dealing with the full complexity of X.500 names)
- Deprecate `DistinguishedName` and the APIs that use it for eventual removal
- Expose a convenience function for the dict -> X509Name transform
Any objections? Thoughts on how I should go about doing this? Should I
do it as part of this Trac ticket or split it out?
The only downside I can think of is that this exposes the
OpenSSL.crypto.X509Name type as part of Twisted's API. I don't think
this is a huge reduction in flexibility --- Twisted's API already somewhat
assumes that TLS is implemented using OpenSSL, and only users whose needs
are *already* not well met by DistinguishedName will care if that `Union`
type changes in the future.
Wim Lewis / wiml(a)hhhh.org
> On May 14, 2020, at 5:23 PM, Wilfredo Sánchez Vega <email@example.com(mailto:firstname.lastname@example.org)> wrote:
> On May 7, 2020, at 12:48 AM, Glyph <email@example.com(mailto:firstname.lastname@example.org)> wrote:
> > If you want to include it in Twisted itself, your best bet is to actually develop it within twisted, as a series of small contributions, rather than as one gigantic one-shot one. Contributions over, say, 400 lines, take exponentially longer to review.
> > Developing it within Twisted will make things go slower; you'll need to get everything code reviewed, you'll need to support multiple versions of Python (no py2 any more, but py3.5 is still pretty old), you'll have to have full test coverage from the get-go. But doing these things from the start is much easier than trying to retrofit them.
> > I actually think that this would be a pretty good fit for Twisted, in the same way that it's been a benefit to have Conch maintained alongside the rest of Twisted. I can see you're developing things very much in line with Twisted's architecture (using cred for authentication, a realm interface, etc) and you've voiced this interest, so it would be great to have you along!
> I think it's great to get an SMB implementation in the Twisted org, but why would we even consider adding something like this to the main Twisted project?
Just practically speaking, we've tried both approaches, and (very broadly generalizing) this one works and the other one doesn't.
Separate projects that have been adopted from outside after they gained some traction are doing okay (i.e. Treq and Klein, some of the most popular applications for Twisted). But others (txacme) are suffering from a lack of maintenance and still others (ldaptor) just have a largely non-overlapping community and hardly benefit at all from org membership. And as you've noted, even the ones doing their best often want for reviews for far longer than Twisted's own latency.
> The main repo is large, and including Twisted in your project brings along far more functionality than you are likely to use, and packages are a fine way to get the functionality you want, so what's the logic for including a new thing in Twisted proper?
Abstractly you make a plausible case. Concretely we tried to start splitting things up on the axes of protocol support and it just made a giant mess and created lots of overhead for the project. Modern infrastructure makes this slightly easier (last time we tried to start this process we didn't have a Github or a Travis), but not much easier. And even modern, better infrastructure leads to an avalanche of tedium as we copy CI configuration back and forth endlessly, replicating lessons that we learn about travis, and azure, and codecov, and (ad nauseam)
Even in the abstract case though there’s a question of what twisted “is” and on what axes it should be split up. I think splitting up by protocol was largely a mistake, but splitting some of the really low level core stuff out (towncrier, automat, and hopefully filepath and deferred) has been pretty successful and worthwhile. Particularly in a post-asyncio world, the core event loop isn’t super interesting and the broad protocol support is the appeal.
So even in the abstract case, there’s a case to be made that twisted is a bunch of implementations and interfaces for wire protocols that involve navigating hierarchies of things (DNS, HTTP, IMAP, SFTP) and SMB fits right in there.
> You cite Conch above, where I'd actually argue that Conch should be moved to a separate repo. Am I crazy?
Not crazy, there's a valid case to be made, but Conch has received a tremendous amount of routine maintenance keeping it up to date with Twisted and Python API evolution by members of the project who would almost certainly otherwise not bother, just to keep the test suite going and the project as a whole integrated. This allows it to remain viable for the much smaller number of SSH-specific contributions that come along.
> That said, a counter argument is that projects in the Twisted org suffer greatly from poor participation and strict development rules, resulting in glacial progress. It's not unusual for me to have a PR take 6 months to a year to get a review (I have two right now that have been waiting since November), and perhaps such PRs would get some attention in the main repo. But I think that's an unfortunate way to address that problem.
This is just a natural consequence of affordances provided by the tooling we presently have. Personally I look at the global queue of pull requests across all these projects and it's kind of exhausting. I don't think many other people do. Case in point: how many ldaptor or txacme reviews have you done since November? This is not to throw any shade; I’m not saying you should have, but it illustrates that right now, separate repos engender a really unfortunate level of community fragmentation. I agree that this is not really an ideal situation, but addressing it requires building a degree of tooling that we just don’t have the resources to do. Our previous attempt to do so was … bad; it kind of illustrated that we don’t have the relevant expertise to build these kinds of tools and mores either, and we just got kind of lucky with the somewhat random pile of jank we have landed on with the current twistedmatrix.com process.
Anyway if someone wanted to do the work to separate out the concerns of version control, pip installability, issue tracking, and integrated regression testing for a whole suite of related python packages in an org, I would be extremely excited, but the benefits of just having Twisted be, say, 5% bigger overall to foster development of a whole new popular and intimidating protocol seem clear given the state of the world today. If those tools do show up later to make this tractable, we can easily split it out later! Along with conch, and mail, and names, for that matter…