Twisted May 2022

twisted@python.org

6 participants
11 discussions

Preparing the migration from Trac tickets to GitHub Issues
by Adi Roiban 20 Jun '22

20 Jun '22

Hi, In preparation for the Trac ticket migration to GitHub Issues a staging/testing repository was created You can find the list of migrated issues here https://github.com/twisted/twisted-trac-migration/issues Send any feedback at the following ticket and my colleague Dan will try to address all the comments: https://github.com/twisted/twisted-trac-migration/issues/1 A bit of hard work is left to find the mapping between Trac users and GitHub users Handling at: https://github.com/twisted/twisted-trac-migration/issues/2 -------------- Once the migration starts, Trac will be in read-only mode with a message on the Trac wiki start page. The plan is to keep it only in read mode for a few more days until redirections are implemented. We still need to see how to put the GitHub twisted/twisted repo in a protected mode during the migration. One option to temporarily rename the main branch as "DONT-CREATE-A-PR-YET" and have a new PR template that will explain that people should wait 1 day for the migration to be finalized. Suggestions are welcomed for how to do the migration. Making the repo private is not an option. The followers/starts will be lost, and repo owners can still create PRs. ---------------- There is a list of issues with ID from 1 to 1732 So Trac ID 1 was migrated as https://github.com/twisted/twisted-trac-migration/issues/11323 Trac tickets with IDs 1733 or greater have a direct mapping. So Trac with ID 1733 is https://github.com/twisted/twisted-trac-migration/issues/1733 ---------- The migration will take about 1 day. This is slow as GitHub API is throttling the requests. The high-level plan targeting 4-5 June for going live: * Gather feedback in the next days and implement the required changes. * Next Monday, 30rd of May, re-run a full import for testing * By next Wednesday, the 1st of June, the testing import should be ready * Review the full migration and see if this is ready for going live. * Wait for feedback until Friday 3 of June. * If there are no complaints, the final / PROD migration will be triggered over the 4-5 June weekend. ----------------- Below are some examples. # Reopened, enhancement, link to another ticket https://github.com/twisted/twisted-trac-migration/issues/3621 # Reopened, attachment https://github.com/twisted/twisted-trac-migration/issues/4258 # a lot of changes and attachments https://github.com/twisted/twisted-trac-migration/issues/4536 # Enhancement, second comment replies to different ticket https://github.com/twisted/twisted-trac-migration/issues/6887 # Reopened -> new https://github.com/twisted/twisted-trac-migration/issues/9300 # Reopened -> closed, milestone, code example https://github.com/twisted/twisted-trac-migration/issues/9335 # With assigned team member, enhancement, new milestone, fixed https://github.com/twisted/twisted-trac-migration/issues/10027 Thanks -- Adi Roiban

4 10

Warning about ISessionSetEnv interface
by Peter Westlake 25 May '22

25 May '22

I'm setting up a manhole with twisted.conch.manhole, and it works, but gives this warning: Can't handle environment variables for SSH avatar <twisted.conch.manhole_ssh.TerminalUser object at 0x7fef33174f60>: <twisted.conch.manhole_ssh.TerminalSession object at 0x7fef33174fd0> does not provide ISessionSetEnv interface. It should be decorated with @implementer(ISession, ISessionSetEnv) to support env variables. I've looked at the release notes about #9315 <https://twistedmatrix.com/trac/ticket/9315>, but can't see where in my code to add @implementer. Here's the code: def create_shell_server(inputs): import pwd # The manhole service from twisted.conch import manhole, manhole_ssh from twisted.conch.checkers import IAuthorizedKeysDB, SSHPublicKeyChecker, readAuthorizedKeyFile from twisted.conch.ssh import keys from twisted.cred import portal from twisted.python.filepath import FilePath from zope.interface import implementer # pylint: disable=no-name-in-module from twisted.application.internet import TCPServer # pylint: enable=no-name-in-module @implementer(IAuthorizedKeysDB) class RunbenchAuthorizedKeysFiles(object): """Object that provides SSH public keys. There is a central file readable only by services that run as users who have permissions to read it, and users are allowed to provide their own file for use during development. The files have fixed names and are not overrideable by inputs or the environment, because that would make it harder to see which files a service was actually using. """ def getAuthorizedKeys(self, username): try: passwd = pwd.getpwnam(bytes_to_native_str(username)) except KeyError: return () def read_keys(filepaths): for fp in filepaths: if fp.exists(): try: with fp.open() as f: for key in readAuthorizedKeyFile(f, keys.Key.fromString): yield key except (IOError, OSError): pass return read_keys([ FilePath(passwd.pw_dir).child('.ssh').child(b'runbench_manhole.pub'), FilePath(rb_site.runbench.keydir).child(b'manhole_authkeys') ]) checker = SSHPublicKeyChecker(RunbenchAuthorizedKeysFiles()) import gc from pprint import pprint namespace = {'sys': sys, 'time': time, 'datetime': datetime, 'gc': gc, 'os': os, 'reactor': reactor, 'runbench': runbench, 'execute': execute, 'pprint': pprint, 'inputs': inputs} class PF(ServerProtocol): # pylint: disable=E0202 def protocolFactory(self, *a, **kw): return manhole.Manhole(namespace) realm = manhole_ssh.TerminalRealm() realm.chainedProtocolFactory = PF mh_portal = portal.Portal(realm) mh_portal.registerChecker(checker) cf = manhole_ssh.ConchFactory(mh_portal) public, private = get_rsa_keys(os.path.join(os.path.dirname(__file__), 'rb_host_rsa')) cf.publicKeys = {b'ssh-rsa': public} cf.privateKeys = {b'ssh-rsa': private} manhole_service = TCPServer(0, cf) manhole_service.startService() print('Started manhole service on port', manhole_service._port._realPortNumber) What do I need to change to make this work? Peter.

4 7

Feedback for Trac to GiHub Issues migration
by Adi Roiban 14 May '22

14 May '22

Hi, I/We plan to migrate the existing Trac Ticket database to twisted/twisted GitHub Issues. Here is a preview of a migrated Trac Ticket https://github.com/chevah/trac-migration-staging/issues/71 Any feedback or help is appreciated. Things to check: * Note that comment date and time is the same as from Trac... but author is not. * Author is included with an avatar in the comment * The attachments are lists * Trac wiki format is not fully converted ... I think it's good enough. If anyone wants to help, please contribute with the code to convert from Trac wiki to GitHub flawored Markdown. * The ticket metadata is kept in the initial comment, in a format that is not human-readable but should make it easy to use the GitHub Issues search to do a match. The GitHub search is very relaxed in terms of hyphens and this is why 2 underscores are used. * Any comments or concerns about using the Trac data without approval from the original Trac ticket authors. * Comment if the table from the first comment should include other data. * Comment/feedback about to migrate, use, or implementing the CC field functionality. Things that will be different for the final migration. * All the comments will be made by an "import-robot" account. The current "danuker" account is Dan's account while he is helping with the development, and trying to keep it simple. Unknown users will have the "GitHub Ghost" avatar and not the current robot avatar. * The initial comment will have the Ticket creation data. Now it has the "Last changed" date. ----------------- The hard work is done by Dan and you can leave your feedback on this PR https://github.com/chevah/trac-to-github/pull/23 This that still needs to be done: * See how to map from Ticket ID to GitHub Issue ID. * Associate each issue with a milestone . This should be a direct mapping from Trac to GitHub * Decide colors for the labels :) * Handle the label for closed tickets - decide what convention to use to decide the close resolution for a ticket ... most probably we will use labels. There are still many things left to be done. We will not do the migration directly to twisted/twisted. After this initial round of feedback, the plan is to do the migration in the testing/staging repo, using the exact code for production. The staging repo will be similar to the current twisted/twisted repo, with already existing PRs. We can then wait for some time for feedback and if all is ok, we can do the migration in PROD. Thanks! -- Adi Roiban

1 0

(no subject)
by Adi Roiban 11 May '22

11 May '22

Hi, With the latest release of cryptography 37.0 the Twisted tests suite fails. I have create a ticket here https://twistedmatrix.com/trac/ticket/10337 As part of another PR [1] and in order to have the CI checks pass, the cryptography dependency was defined as cryptography >= 2.6, < 37.0 So for now, Twisted is limited to cryptography up to 36.0.2 My hope is that we can find the volunteers to look into this issue and have Twisted working with latest cryptography and latest OpenSSL It might be the case that Twisted is OK and we only need to update the assertions in the test code. I guess that the main reason for the failure is OpenSSL 3.0.0. For Twisted CI we used the upstream cryptography wheels, and they are distributed with an embedded OpenSSL If you have time, try to run your Twisted TLS based apps together with cryptography 37.0 upstream wheels and see if you observe any failures. If you have custom wheels that link to OpenSSL 1.1.1, it would also help to test cryptography 37.0 and OpenSSL 1.1.1 Feel free to add any feedback here over email or over IRC / Gittter or over Trac. Thanks [1] https://github.com/twisted/twisted/pull/1724 -- Adi Roiban

2 2

Re: pydoctor: @type or Pythons type annotations?
by Glyph 08 May '22

08 May '22

Personally I'd say that @type is effectively deprecated by the fact that pydoctor supports annotations now. (In fact I kind of hope Pydoctor eventually grows support for Annotated so you can eventually do def query(name: Doc[str, "The name of the thing to query"]) -> Doc[str, "The value associated with C{name}"]:, and avoid the possibility of misspelling parameter names, and get all the documentation inline.) > On May 8, 2022, at 6:15 AM, c.buhtz@posteo.jp(mailto:c.buhtz@posteo.jp) wrote: > Hello, > > I assume this is again a more religious question but I would be glad to hear your opinions and experiences about it. > > Let's assume we are using Python 3.9 or higher and epytext as documentation format with pydocotor. Whould recommand to use type-annotations or @type to give type informations to the pydoctor generated documentation? > _______________________________________________ > Twisted mailing list -- twisted@python.org(mailto:twisted@python.org) > To unsubscribe send an email to twisted-leave@python.org(mailto:twisted-leave@python.org) > https://mail.python.org/mailman3/lists/twisted.python.org/ > Message archived at https://mail.python.org/archives/list/twisted@python.org/message/LRTDFNOSNT… > Code of Conduct: https://twisted.org/conduct

1 0

pydoctor: @type or Pythons type annotations?
by c.buhtz＠posteo.jp 08 May '22

08 May '22

Hello, I assume this is again a more religious question but I would be glad to hear your opinions and experiences about it. Let's assume we are using Python 3.9 or higher and epytext as documentation format with pydocotor. Whould recommand to use type-annotations or @type to give type informations to the pydoctor generated documentation?

2 1

Links to external types
by c.buhtz＠posteo.jp 08 May '22

08 May '22

Hello, in your great epytext demo [1] I can see that types are generated as hypertext. Great! ;) But I don't get it how you did it. 1. Maybe it is a good idea to add the pydoctor call you used to generate the demos to the documentation. Or did I missed it somewhere? 2. The demo show Python types only. Is it possible to use hypertext for 3rd party-types also (e.g. pandas, numpy, ...)? [1] -- <https://pydoctor.readthedocs.io/en/latest/docformat/epytext/epytext_demo.de…>

2 7

pydoctor: Python code-blocks in Google-style doc-strings
by c.buhtz＠posteo.jp 08 May '22

08 May '22

Looking into the Google style demo and the original Google style guide it is unclear for me how to create code blocks in doc strings. As an example (from [1]) def read_and_validate_csv(file_path: pathlib.Path, specs_and_rules: dict, encoding: str = 'utf-8') -> pandas.DataFrame: """Read a CSV file with respect to specifications about format and rules about valid values. Hints: Do not use objects of type type (e.g. str instead of "str") when specificing the column type. Examples: .. code-block:: python specs_and_rules = { 'ColumnA': 'str', 'ColumnB': ('str', 'no answer'), 'ColumnC': None, 'ColumnD': ( 'Int16', -9, { 'len': [1, 2, (4-8)], 'val': [0, 1, (3-9)] } } } Args: file_path: The csv file to read from. specs_and_rules: A column name indexed dictionary. encoding (optional, default='utf-8'): Optional encoding type used for reading the csv file. Returns: The resulting data frame. """ There is something unclear for me in your example_generator() [2] function. It is unclear for me how pydoctor decide what is simple text and what is code. [1] -- <https://codeberg.org/buhtz/buhtzology/src/commit/e7177a8e17c627f5c93befbc02…> [2] -- <https://github.com/twisted/pydoctor/blob/6e2febe63a8ed0f2e316009f34721dc177…>

1 0

pydoctor: exclude py-files?
by c.buhtz＠posteo.jp 08 May '22

08 May '22

Hello, is there a way to exclude py files from a pydoctor run? The file I am talking about is in the projects folder but it is nowhere imported. But it is touchted by pydoctor.

1 0

pydoctor: Problems with sub-modules and namespaces
by c.buhtz＠posteo.jp 07 May '22

07 May '22

Hello, I have a packaged named "buhtzology" [1]. So you can do "import buhtzology" and then "buhtzology.generate_filepath()". But on the level of the filesystem the "def generate_filepath" is not in "buhtzology/__init__.py" but "buhtzology/_buhtzology.py". Everything in "_buhtzology.py" is imported into the namespace of "buhtzology" when importing "buhtzology". This is the relevant part of the "__init__.py" file. from ._buhtzology import * The problem is that "generate_filepath()" doesn't appear as "buhtzology.generate_filepath()" in the pydoctor generated html files but as "buhtzology._buhtzology.generate_filepath()". You ask why I do it like this? It is because buhtzology has some packages that should load by default (e.g. _buhtzology) and some that don't (buhtzology.bandas). I also tried to manipulate the __all__ variable which works in Doxygen in that case but no in pydoctor. [1] -- <https://codeberg.org/buhtz/buhtzology/src/branch/develop>

2 2

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

Twisted May 2022