Yes, that's the problem. Is it not possible to have finer permission (instead of boolean permission: commit or not commit)? Eg. give commit access but only for a file or a directory? It looks like Tarek Ziade is now allowed to commit, but only on distutils. I like such permission because nobody knows the whole Python project, it's too huge for a single brain ;-)
I like them, too - that's why I'm generally not opposed to handing out such privileges fairly generously. In our experience, you don't need to enforce such a restriction technically - the social enforcement (you lose access if you are changing things you were not supposed to change) is sufficient.
In fact, the Python repository also hosts Stackless Python, so technically, Python committers can commit to stackless also, and vice versa - but nobody does, of course.
There are many other people who commit only to specific files, or only specific kinds of changes; see Misc/developers.txt for a (possibly incomplete) list.