[Mailman-Developers] Listing Lists Faster in 2.0?

[Roberto Ullfig]

Roberto Ullfig wrote:
> 
> "Barry A. Warsaw" wrote:
> >
> > >>>>> "RU" == Roberto Ullfig <rullfig@uchicago.edu> writes:
> >
> >     RU> So, in 1.0rc2, displaying the list of lists for 529 lists
> >     RU> requires 529**2 = 279841 system stat calls and takes over one
> >     RU> and a half minutes on our Ultra-2 2x296 processor system! Is
> >     RU> this because of Python, Mailman, or both? Has this been
> >     RU> "fixed" in 2.0? You really should only need to make one stat
> >     RU> call per list.
> >
> > Uh, it's because of Mailman :)
> >
> > I implemented a list_lists scripts which does on the command line what
> > listinfo.py does in HTML (see attached).  Here's what truss -c gives
> > me:
> >
> > -------------------- snip snip --------------------
> > Portal - [no description available]
> > Postal - [no description available]
> >  Stage - Staging new Mailman releases
> >   Test - [no description available]
> > syscall      seconds   calls  errors
> > _exit            .00       1
> > read             .00     102
> > write            .00       8
> > open             .11     607    474
> > close            .01     143
> > time             .00       3
> > brk              .03     227
> > stat             .03     201    157
> > getpid           .00      10
> > fstat            .00      66
> > ioctl            .02      63     61
> > execve           .00      10      8
> > umask            .00       2
> > fcntl            .00       7
> > readlink         .00       2      2
> > sigprocmask      .00       2
> > sigaction        .00      50
> > sigpending       .00       1
> > mmap             .00      42
> > mprotect         .00      10
> > munmap           .00      11
> > uname            .00       4
> > sysconfig        .00       1
> > lwp_create       .00       6
> > lwp_continue     .00       2
> > lwp_self         .00       3
> > llseek           .00     114
> > door             .00       5
> > lwp_schedctl     .01       5
> > getdents64       .01      15
> > fstat64          .00      67
> > open64           .00       7
> >                 ----     ---    ---
> > sys totals:      .22    1797    702
> > usr time:        .51
> > elapsed:        1.19
> > -------------------- snip snip --------------------
> >
> > Getting the list of list names, requires at least a listdir() and an
> > exists() for every directory found there.
> >
> > Nothing about this will change for 2.0.
> >
> > -Barry
> 
> Thanks for the script.
> 
> Now this is the truss output for the listinfo that is called by
> driver:
> 
> syscall      seconds   calls  errors
> _exit            .00       1
> read             .21    1979
> write            .15    1638
> open             .12    1233    579
> close            .06    1189
> time             .00       1
> brk              .43    5026
> stat           25.58  285877    174
> fstat            .00      63
> ioctl            .01     591    589
> execve           .00       1
> umask            .00       2
> fcntl            .02     535
> readlink         .00       3      2
> sigaction        .00      48
> mmap             .00      32
> munmap           .00       8
> llseek           .05     643
> getdents64       .85   10165
> fstat64          .01    1123
> open64           .03     535
>                 ----     ---    ---
> sys totals:    27.52  310693   1344
> usr time:      54.01
> elapsed:      173.76

And this is the truss from using the script you sent:

syscall      seconds   calls  errors
_exit            .00       1
read             .29    1963
write            .03     534
open             .13    1132    485
close            .00    1182
time             .00       1
brk              .41    4944
stat           28.32  285849    148
fstat            .00      61
ioctl            .02     586    584
execve           .01       3      1
fcntl            .00     535
readlink         .00       3      2
sigaction        .01      48
mmap             .00      40
munmap           .01      10
llseek           .05     632
getdents64       .92   10165
fstat64          .02    1118
open64           .06     535
                ----     ---    ---
sys totals:    30.28  309342   1220
usr time:      56.08
elapsed:      182.84

All those stat calls just don't seem right to me. Using python
1.5.2 if that matters.

--
Roberto Ullfig : rullfig@uchicago.edu
Systems Administrator
Networking Services and Information Technologies
University of Chicago