In the latest Kwant manual describing the details about kwant.physics.modes, I find a sentence "This function uses the most stable and efficient algorithm for calculating the mode decompositionthat the Kwant authors are aware about. Its details are to be published." Do anyone know what the algorithm is? I look forward some references if possible.
That is an excellent question.
There is a review of numerical quantum transport currently in preparation by the main Kwant authors, which will include explicit details of the algorithm implemented by Kwant. Unfortunately the review is not yet in a state where it can be shared, but rest assured that as soon as it is we will inform the Kwant community via the mailing list.
In the meantime there are a few places where you can get some more information:
This is a tutorial I gave at the university of Maryland about
how Kwant works. It gives a high-level overview of the scattering
problem and the lead-modes problem, but doesn't go into details
This is a talk I gave at Delft university that dives deep into the details. It contains code snippets from the Kwant source code, and links back to the relevant places in the source code itself.
The Kwant source code itself! This is of course the canonical place to find out how Kwant works internally. The start of the modes-related code is in 'kwant/physics/leads.py' around line 990, but it is quite dense (and that's putting it mildly!).
Hopefully that will be enough to get you started; let us know how you get on.