[Mailman-Developers] Listing Lists Faster in 2.0?

[Roberto Ullfig]

"Barry A. Warsaw" wrote:
> 
> >>>>> "RU" == Roberto Ullfig <rullfig@uchicago.edu> writes:
> 
>     RU> So, in 1.0rc2, displaying the list of lists for 529 lists
>     RU> requires 529**2 = 279841 system stat calls and takes over one
>     RU> and a half minutes on our Ultra-2 2x296 processor system! Is
>     RU> this because of Python, Mailman, or both? Has this been
>     RU> "fixed" in 2.0? You really should only need to make one stat
>     RU> call per list.
> 
> Uh, it's because of Mailman :)
> 
> I implemented a list_lists scripts which does on the command line what
> listinfo.py does in HTML (see attached).  Here's what truss -c gives
> me:
> 
> -------------------- snip snip --------------------
> Portal - [no description available]
> Postal - [no description available]
>  Stage - Staging new Mailman releases
>   Test - [no description available]
> syscall      seconds   calls  errors
> _exit            .00       1
> read             .00     102
> write            .00       8
> open             .11     607    474
> close            .01     143
> time             .00       3
> brk              .03     227
> stat             .03     201    157
> getpid           .00      10
> fstat            .00      66
> ioctl            .02      63     61
> execve           .00      10      8
> umask            .00       2
> fcntl            .00       7
> readlink         .00       2      2
> sigprocmask      .00       2
> sigaction        .00      50
> sigpending       .00       1
> mmap             .00      42
> mprotect         .00      10
> munmap           .00      11
> uname            .00       4
> sysconfig        .00       1
> lwp_create       .00       6
> lwp_continue     .00       2
> lwp_self         .00       3
> llseek           .00     114
> door             .00       5
> lwp_schedctl     .01       5
> getdents64       .01      15
> fstat64          .00      67
> open64           .00       7
>                 ----     ---    ---
> sys totals:      .22    1797    702
> usr time:        .51
> elapsed:        1.19
> -------------------- snip snip --------------------
> 
> Getting the list of list names, requires at least a listdir() and an
> exists() for every directory found there.
> 
> Nothing about this will change for 2.0.
> 
> -Barry

Thanks for the script.

Now this is the truss output for the listinfo that is called by
driver:

syscall      seconds   calls  errors
_exit            .00       1
read             .21    1979
write            .15    1638
open             .12    1233    579
close            .06    1189
time             .00       1
brk              .43    5026
stat           25.58  285877    174
fstat            .00      63
ioctl            .01     591    589
execve           .00       1
umask            .00       2
fcntl            .02     535
readlink         .00       3      2
sigaction        .00      48
mmap             .00      32
munmap           .00       8
llseek           .05     643
getdents64       .85   10165
fstat64          .01    1123
open64           .03     535
                ----     ---    ---
sys totals:    27.52  310693   1344
usr time:      54.01
elapsed:      173.76

I can understand a listdir and an exists for each directory;
that should come out to 2 * n stat calls right (~1000 for us).
What I'm saying is that we are seeing n ** 2 stat calls (that's
squared) or 285877 of them. The above truss is from running the
driver manually after setting PYTHONPATH as suggested by Dan
(thanks Dan):

setenv PYTHONPATH /opt/http/opt/pkgs/mailman/Mailman
python /opt/http//opt/pkgs/mailman/scripts/driver listinfo

I've also trussed the running process and gotten similar
results; I see it stat'ing every directory once for every
directory stat'ed or n-squared stats.

Note that this is with 1.0rc2; still waiting for 2.0.

--
Roberto Ullfig : rullfig@uchicago.edu
Systems Administrator
Networking Services and Information Technologies
University of Chicago