Mailman 3 September 2020 - Twisted

[Twisted-Python] Conch SFTP Questions
by Robert DiFalco Sept. 24, 2020

Sept. 24, 2020

Hey folks, I've cobbled together an SFTP client based on bits and pieces I've found around the web. The issue is that it appears to be almost one shot. I will need to send many files (the number not known ahead of time). It's not clear to me when the connection is closed or how many factories I'm creating. All the code I've grabbed looks like it's creating a new factory for every SFTP file I send. Here's some of the code I have. It's fairly straight forward in that it creates a directory if it doesn't exist and then writes a file. @attr.s(frozen=True) class FileInfo(object): """ Class that tells SFTP details about the file to send. """ directory = attr.ib(converter=str) # type: str name = attr.ib(converter=str) # type: str data = attr.ib() # type: str chunk_size = attr.ib(converter=int, default=CHUNK_SIZE) # type: int def to_path(self): """ Turns the folder and file name into a file path. """ return self.directory + "/" + self.name @attr.s(frozen=True) class SFTPClientOptions(object): """ Client options for sending SFTP files. :param host: the host of the SFTP server :param port: the port ofo the SFTP server :param fingerprint: the expected fingerprint of the host :param user: the user to login as :param identity: the identity file, optional and like the "-i" command line option :param password: an optional password """ host = attr.ib(converter=str) # type: str port = attr.ib(converter=int) # type: int fingerprint = attr.ib(converter=str) # type: str user = attr.ib(converter=str) # type: str identity = attr.ib(converter=optional(str), default=None) # type: Optional[str] password = attr.ib(converter=optional(str), default=None) # type: Optional[str] @inlineCallbacks def sftp_send(client_options, file_info): # type: (SFTPClientOptions, FileInfo)->Deferred """ Primary function to send an file over SFTP. You can send a password, identity, or both. :param client_options: the client connection options :param file_info: contains the file info to write :return: A deferred that signals "OK" if successful. """ options = ClientOptions() options["host"] = client_options.host options["port"] = client_options.port options["password"] = client_options.password options["fingerprint"] = client_options.fingerprint if client_options.identity: options.identitys = [client_options.identity] conn = SFTPConnection() auth = SFTPUserAuthClient(client_options.user, options, conn) yield connect(client_options.host, client_options.port, options, _verify_host_key, auth) sftpClient = yield conn.getSftpClientDeferred() yield _send_file(sftpClient, file_info) returnValue("OK") def _verify_host_key(transport, host, pubKey, fingerprint): """ Verify a host's key. Based on what is specified in options. @param host: Due to a bug in L{SSHClientTransport.verifyHostKey}, this is always the dotted-quad IP address of the host being connected to. @type host: L{str} @param transport: the client transport which is attempting to connect to the given host. @type transport: L{SSHClientTransport} @param fingerprint: the fingerprint of the given public key, in xx:xx:xx:... format. @param pubKey: The public key of the server being connected to. @type pubKey: L{str} @return: a L{Deferred} which is success or error """ expected = transport.factory.options.get("fingerprint", "no_fingerprint") if fingerprint == expected: return succeed(1) log.error( "SSH Host Key fingerprint of ({fp}) does not match the expected value of ({expected}).", fp=fingerprint, expected=expected) return fail(ConchError("Host fingerprint is unexpected.")) class SFTPSession(SSHChannel): """ Creates an SFTP session. """ name = "session" @inlineCallbacks def channelOpen(self, whatever): """ Called when the channel is opened. "whatever" is any data that the other side sent us when opening the channel. @type whatever: L{bytes} """ yield self.conn.sendRequest(self, "subsystem", NS("sftp"), wantReply=True) client = FileTransferClient() client.makeConnection(self) self.dataReceived = client.dataReceived self.conn.notifyClientIsReady(client) class SFTPConnection(SSHConnection): def __init__(self): """ Adds a deferred here so client can add a callback when the SFTP client is ready. """ SSHConnection.__init__(self) self._sftpClient = Deferred() def serviceStarted(self): """ Opens an SFTP session when the SSH connection has been started. """ self.openChannel(SFTPSession()) def notifyClientIsReady(self, client): """ Trigger callbacks associated with our SFTP client deferred. It's ready! """ self._sftpClient.callback(client) def getSftpClientDeferred(self): return self._sftpClient class SFTPUserAuthClient(SSHUserAuthClient): """ Twisted Conch doesn't have a way of getting a password. By default it gets it from stdin. This allows it to be retrieved from options instead. """ def getPassword(self, prompt = None): """ Get the password from the client options, is specified. """ if "password" in self.options: return succeed(self.options["password"]) return SSHUserAuthClient.getPassword(self, prompt) @inlineCallbacks def _send_file(client, file_info): # type: (FileTransferClient, FileInfo) -> Deferred """ Creates a directory if required and then creates the file. :param client: the SFTP client to use :param file_info: contains file name, directory, and data """ try: yield client.makeDirectory(file_info.directory, {}) except SFTPError as e: # In testing on various system, either a 4 or an 11 will indicate the directory # already exist. We are fine with that and want to continue if it does. If we misinterpreted # error code here we are probably still ok since we will just get the more systemic error # again on the next call to openFile. if e.code != 4 and e.code != 11: raise e f = yield client.openFile(file_info.to_path(), FXF_WRITE | FXF_CREAT | FXF_TRUNC, {}) try: yield _write_chunks(f, file_info.data, file_info.chunk_size) finally: yield f.close() @inlineCallbacks def _write_chunks(f, data, chunk_size): # type: (ClientFile, str, int) -> Deferred """ Convenience function to write data in chunks :param f: the file to write to :param data: the data to write :param chunk_size: the chunk size """ for offset in range(0, len(data), chunk_size): chunk = data[offset: offset + chunk_size] yield f.writeChunk(offset, chunk) It gets called like this: return sftp.sftp_send( client_options=SFTPClientOptions( host=self.options.host, port=self.options.port, user=self.options.user, fingerprint=self.options.fingerprint, identity=getattr(self.options, "identity", None), password=self._getPassword()), file_info=sftp.FileInfo( directory=self.options.directory, name=fileName, data=data, chunk_size=getattr(self.options, "chunkSize", sftp.CHUNK_SIZE))) But I supposed I'd like to see something more like this: sftpClient = self.getSftpClient( client_options=SFTPClientOptions( host=self.options.host, port=self.options.port, user=self.options.user, fingerprint=self.options.fingerprint, identity=getattr(self.options, "identity", None), password=self._getPassword())) return sftpClient.send( file_info=sftp.FileInfo( directory=self.options.directory, name=fileName, data=data, chunk_size=getattr(self.options, "chunkSize", sftp.CHUNK_SIZE))) Where sftpClient reuses the existing SSH connection if it is active (rather than logging in each time). But maybe the sftp service doesn't multiplex so I have to create a new SSHClientFactory every time I want to send a distinct file? Sorry for all the questions, new to twisted and a bit confused. Thanks! Robert

3 4

[Twisted-Python] Only run Azure, Travis, or CircleCI if GitHub actions pass
by Craig Rodrigues Sept. 22, 2020

Sept. 22, 2020

Adi, Is it possible to configure the CI so that Azure, Travis and CircleCI only runs if GitHub actions pass? For example here: https://github.com/twisted/twisted/runs/1137175269 *Azure Pipelines* and *Travis* should only run if the GitHub actions pass. -- Craig

3 3

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com Sept. 20, 2020

Sept. 20, 2020

1 0

[Twisted-Python] Broken Javascript on https://twistedmatrix.com ?
by Craig Rodrigues Sept. 19, 2020

Sept. 19, 2020

If I go to https://twistedmatrix.com and click on one of the tabs in the *Code Examples* section, such as *Publish/Subscribe*, the tab does not switch. This looks like a Javascript problem. Does anyone with knowledge of Javascript have an idea what the problem is? -- Craig

4 7

[Twisted-Python] Black enabled in trunk
by Tom Most Sept. 16, 2020

Sept. 16, 2020

Hi all, Craig and I have been working to land a switch to The *Black* Coding Style <https://black.readthedocs.io/en/stable/the_black_code_style.html>. It's in! The tree has been reformatted <https://github.com/twisted/twisted/commit/bc96c774be6e307e8e3e4d39780d37b04…>. Some notable changes: * 3-2-1 blank lines are replaced by PEP 8 style 2-1-1. * String syntax uses "double quotes". Run this command to exclude the formatting from git blame <https://github.com/twisted/twisted/blob/trunk/.git-blame-ignore-revs>: git config blame.ignoreRevsFile .git-blame-ignore-revs There are two new Tox testenvs. To check style: tox -e black To adjust the formatting: tox -e black-reformat The formatting is checked by a new GitHub Actions lint built. There is some follow-up work planned: * Many PRs have conflicts (more on this below). * Various wiki pages need updates. I've done a few, but there are probably more. * The coding standard should be updated <https://twistedmatrix.com/trac/ticket/9957#ticket>. * The PR template needs an update <https://twistedmatrix.com/trac/ticket/9956#ticket>. * A few examples need reformatting <https://tm.tl/#9952> (they weren't formatted because they fail to lint). Inevitably, changes like this cause conflicts. For small PRs it's easiest to merge forward and then run tox -e black-reformat. For larger ones it can help to apply formatting before merge. To do this: * Run black on the files your branch changes (be sure to use Black 20.8b1, not an older version) * Commit the result, like `git commit -am "Fade to black"` * Add that commit to .git-blame-ignore-revs to avoid polluting git blame. * Merge forward. You can ignore formatting while performing the merge. * Then run `tox -e black-reformat` to ensure you are in sync. Commit any changes it generates. A big thanks to Hawkowl for kicking this all off and Craig for quick review turnaround. May we all be mildly displeased with the formatting, yet never have to review it again! ---Tom

4 6

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com Sept. 13, 2020

Sept. 13, 2020

1 0

[Twisted-Python] Pycodestyle disabled in trunk, time to enable black?
by Craig Rodrigues Sept. 12, 2020

Sept. 12, 2020

In this PR: https://github.com/twisted/twisted/pull/1365 pycodestyle was disabled in trunk in order to make it easier to merge changes and not have pycodestyle error out, due to the code not meeting existing Twisted style guidelines. So this means that the old pycodestyle is not enforced in trunk. In this PR: https://github.com/twisted/twisted/pull/1356 a `.git-blame-ignore-revs <https://github.com/twisted/twisted/pull/1356/commits/294aaf3eeaba41d64a8064…>` file was added to the tree. I guess in future, if the entire tree is reformatted, then we can add that git SHA for the PR which reformats the tree, so that that revision will be ignored during a `git blame`. There is this pending (over 1 year) PR to add black to CI: https://github.com/twisted/twisted/pull/1134 There are some comments in there related to documentation changes. Can the non-documentation changes be pulled out of PR 1134 and merged to trunk now? I guess whenever black is merged to trunk, then a follow-up PR to update `.git-blame-ignore-devs` would be required? -- Craig

2 1

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com Sept. 6, 2020

Sept. 6, 2020

1 0

[Twisted-Python] Block TLS 1.0 and TLS 1.1 support on windows
by John Aherne Sept. 1, 2020

Sept. 1, 2020

I'm using twisted 20.3 and python3.6.8 and Windows 10 I'm using endpoint_description with a tac file to start up a server. But I need to disable tls 1.0 and 1.1. I was hoping to find a parameter I could pass in to make the system only recognise 1.2 and 1.3. But could not find anything that would do that. I thought sslmethod would be what I wanted but that is limited to : Must be one of: "SSLv23_METHOD", "SSLv2_METHOD", "SSLv3_METHOD", "TLSv1_METHOD". If I choose TLSv1_METHOD, TLS1.0 and 1.1 are still enabled and QUALYS complains and downgrades the rating to B In the end I found _defaultMinimumTLSVersion in _sslverify.py. I set this to TLSVersion.TLSv1_2 and that seemed to do the trick. But I don't think I should be doing that. I think I've missed some obvious place where I can pass in a value to change this. Anyone know where I should be looking. Thanks for any info -- *John Aherne* *www.rocs.co.uk <http://www.rocs.co.uk>* 020 7223 7567

3 6