Mailman 3 Pylint transform plugin and transforming imported module - code-quality

4 Oct 2021

      Hi all,

I'm trying to write a transform plugin to better work with a library 
that I maintain. For background my library extends a third-party library 
and uses some dynamic typing to add new features and wrappers on top of 
the third-party library. Because it uses dynamically created classes 
pylint complains about having missing members (E1101) and unexpected 
keyword arguments in the constructor (E1123).

The class that is being dynamically generated only gets generated once 
when the module is imported and provides a proxy/wrapper around an 
existing class from the third-party library. The wrapper mirrors the 
member functions from the original class, as well as overriding and 
adding some new member functions. So the structure is something like:

     tpmodule.py

        # third-party library

        class Control:
             ....

    mymodule.py

        # my library

        import tpmodule

        def _make_wrapper(towrap):
              .... # dynamically create new class

        Control = _make_wrapper(tpmodule.Control)

     user.py

         # user code to run pylint on

         import mymodule

         ctrl = Control(...)
         ctrl.original_func(...)
         ctrl.new_func(...)

My plan with the transform plugin is to use 
'astroid.MANAGER.register_transform()' to detect the 'astroid.Module' 
calls and turn 'mymodule.py' into something more palatable for pylint; 
in particular to replace 'mymodule.Control' with an 'astroid.ClassDef' 
that has the correct initializer and member functions. Unfortunately, 
things are not working the way I would like and I'm still getting the 
missing member and bad constructor arguments errors, even though, as far 
as I can see, I am returning the appropriately modified module.

However, there are many things that I don't understand:

1) Relating to the above example, pylint seems to be treating the 'ctrl' 
variable in 'user.py' 'ctrl' as being of type 'tpmodule.Control' so 
doesn't complain about things that are common to 'tpmodule.Control' and 
'mymodule.Control'. But I cannot understand how pylint could possibly 
detect the relationship between 'tpmodule.Control' and 
'mymodule.Control' without getting into the guts of the 
'mymodule._make_wrapper()' function.

2) It is not clear to me how pylint treats imports from external 
libraries? In my example tpmodule.py and mymodule.py are not things that 
I want pylint to report on. But presumably it still needs to follow some 
of the chain of dependencies to work out the signature of what is 
exposed by the module so I understand why it would parse 'mymodule.py', 
but I don't understand why it would parse 'tpmodule.py'.

In fact, another curious thing is that I created a cut down example to 
highlight the problem and while it has some of the same errors it also 
behaves slightly differently. In the cut-down example pylint DOESN'T 
actually parse 'tpmodule.py' so 'tpmodule.Control' doesn't trigger the 
'register_transform()' the way it does in the real system. Even though 
this makes more sense to me than what is happening in the actual code, 
it is unfortunate because I wanted to dynamically construct the 
signature of 'tpmodule.Control' so was relying on pylint parsing 
'tpmodule.py'.

Note for clarification, in the real system 'mymodule.py' and 
'tpmodule.py' are more complex and both use functions from their 
respective internal modules. 'tpmodule.py' is actually written in C++ 
and is using CFFI to generate the Python API, so maybe that can explain 
some difference in the behaviour to the pylint expert?

Anyway, I'm sorry for the long winded questions. However, I'm hoping 
someone has someone insight or can point me to some docs or a specific 
example that solves a similar issue. I can also post a zip file of my 
cut down example if it is appropriate to do so. It consists of four 
short files ('mymodule.py', 'tpmodule.py', 'user.py', and 'plugin.py').

Regards,

Dave

To be clear I'm not using pylint on the library itself, instead I want 
pylint to work nicely with users of the library; although at some point 
I will try to configure pylint to help me with refactoring some of the 
code as well.

My library extends an existing library with some new feature. and does 
some dynamic typing.

I'm new to the group and

Pylint transform plugin and transforming imported module

David Rajaratnam

Pierre Sassoulas

David Rajaratnam

tags

participants (2)