[Python-Dev] os.getgroups() on MacOS X Was: red buildbots on 2.7

[Ronald Oussoren]

On 23 Jun, 2010, at 16:48, Alexander Belopolsky wrote:

> On Wed, Jun 23, 2010 at 2:08 AM, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
> ..
>>> 
>>>> * [Ronald's proposal] results in posix.getgroups not reflecting results of posix.setgroups
>>>> 
>>> 
>>> This effectively substitutes getgrouplist called on the current user
>>> for getgroups.  In 3.x, I believe the correct action will be to
>>> provide direct access to getgrouplist which is while not POSIX (yet?),
>>> is widely available.
>> 
>> I don't mind adding getgrouplist, but that issue is seperator from this one. BTW. Appearently getgrouplist is posix
>> (<http://refspecs.freestandards.org/LSB_3.1.1/LSB-Core-generic/LSB-Core-generic/libc.html>), although this isn't a
>> requirement for being added to the posix module.
>> 
> 
> (The link you provided leads to "Linux Standard Base Core
> Specification," which is different from POSIX, but the distinction is
> not relevant for our discussion.)

I know, but the page claims getgrouplist is in SUS.  I've since looked at what claims to be a copy of SUS: http://www.unix.org/single_unix_specification/ and that does not contain getgrouplist. 

> 
>> 
>> It is still my opinion that the second option is preferable for better compatibility with system tools, even if the patch
>> is more complicated and the library function we use can be considered to be broken.
> 
> Let me try to formulate what the disagreement is.  There are two
> different group lists that can be associated with a running process:
> 1) The list of current supplementary group IDs maintained by the
> system for each process and stored in per-process system tables; and
> 2) The list of the groups that include the uid under which the process
> is running as a member.
> 
> The first list is returned by a system call getgroups and the second
> can be obtained using system database access functions as follows:
> 
> pw = getpwuid(getuid())
> getgrouplist(pw->pw_name, ..)
> 
> The first list can be modified by privileged processes using setgroups
> system call, while the second changes when system databases change.
> 
> The problem that _DARWIN_C_SOURCE introduces is that it replaces
> system getgroups with a database query effectively making the true
> process' list of supplementary group IDs inaccessible to programs.
> See source code at
> <http://www.opensource.apple.com/source/Libc/Libc-594.1.4/sys/getgroups.c>.
> 
> The problem is complicated by the fact that OSX true getgroups call
> appears to truncate the list of groups to NGROUPS_MAX=16.  Note,
> however that it is not clear whether the system call truncates the
> list or the underlying process tables are limited to 16 entries and
> additional groups are ignored when the process is created.
> 
> In my view, getgroups and getgrouplist are two fundamentally different
> operations and both should be provided by the os module.  Redefining
> os.getgroups to invoke getgrouplist instead of system getgroups on one
> particular platform to work around that platform's system call
> limitation is not right.

But we don't redefine os.getgroups to call getgrouplist, it is the system library that
seems to implement getgroups(3) using getgrouplist(3).  I agree that that is odd at best,
but it is IMHO functioning as designed by Apple (that is, Apple choose the pick
the current behavior, they didn't accidently break this).

The previous paragraph is nitpicky, but this is IMO an important distinction.


I've done some more experimentation:

*  compat(5) lies: not setting _DARWIN_C_SOURCE is not the same as settings _DARWIN_C_SOURCE when the deployment target is 10.5, with _DARWIN_C_SOURCE getgroups it translated to the symbol "_getgroups$DARWIN_EXTSN" in the object file, without it is "_getgroups".

* the id(1) command uses the version of getgroups that does not reflect setgroups. Given this script:
import os

os.system("id")
os.setgroups([1])
os.system("id")

Running it gives an unexpected output:

# /usr/bin/python doit.py
uid=0(root) gid=0(wheel) groups=0(wheel),204(_developer),100(_lpoperator),98(_lpadmin),80(admin),61(localaccounts),29(certusers),20(staff),12(everyone),9(procmod),8(procview),5(operator),4(tty),3(sys),2(kmem),1(daemon),401(com.apple.access_screensharing)
uid=0(root) gid=0(wheel) groups=0(wheel),204(_developer),100(_lpoperator),98(_lpadmin),80(admin),61(localaccounts),29(certusers),20(staff),12(everyone),9(procmod),8(procview),5(operator),4(tty),3(sys),2(kmem),1(daemon),401(com.apple.access_screensharing)

* when I add a group in the Accounts panel in System Preferences and add my account to it the id(1) command immediately reflects the change (as expected given the previous result)

* adding a non-administrator account to a newly created group does not affect filesystem access for existing process (that is, if I created a file that's only readable for the new group and the test user couldn't read that file until I logged out and in again), which means the Account panel doesn't magically alter kernel state for running processes.

* Setting or unsetting _DARWIN_C_SOURCE doesn't affect the contents of pyconfig.h beyond that setting:

$ diff pyconfig.h-DARWIN_C_SOURCE pyconfig.h-NO_DARWIN_SOURCE 
1124c1124
< #define _DARWIN_C_SOURCE 1
---
> /* #undef _DARWIN_C_SOURCE */

"pyconfig.h-DARWIN_C_SOURCE" is generated by the current configure script, the other one is generated by a configure script that was patched to not yet _DARWIN_C_SOURCE (by removing "AC_DEFINE(_DARWIN_C_SOURCE, 1, [Define on Darwin to activate all library features])" from configure.in and regenerating configure).  Both were generated using "configure MACOSX_DEPLOYMENT_TARGET=10.5".

* setgroups(3) cannot set more than 16 groups, that is "setgroups(17, gidset)" will always return EINVAL (this is on OSX 10.6.4). I've verified this using a C program that directly calls the right APIs.   

I'm busy with projects for the rest of the week and won't be able to do anything python-dev related until Sunday.

Ronald



-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3567 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100623/90441445/attachment-0001.bin>