
I have temporary access to a Mac OSX box. I find the following problems: - test_re crashes unless I do ulimit -s 2000 -- haven't tried other values, but the default of 512 (KB) is insufficient. I know the README file explains this, but now that I've experienced this myself, I wonder if we shouldn't hack main() to increase the stack size to 2 MB, inside an #ifdef darwin or something like that. - test_socket fails with errno 3, 'Unknown server error' on a socket.gethostbyaddr() call. Does anybody know what that is about? - test_locale fails with a complaint about "1,024" vs. "1024". Since this is probably a libc bug (though wouldn't this also occur on other BSD systems?) I can live with it. - test_largefile takes a *very* long time. Perhaps it actually creates a truly large file (the Mac OSX filesystem is case insensitive, so I suppose it may be so different that it doesn't support files with holes in them). Maybe the test should disable itself (or part of itself) unless a specific resource is requested? --Guido van Rossum (home page: http://www.python.org/~guido/)

On donderdag, nov 28, 2002, at 05:16 Europe/Amsterdam, Guido van Rossum wrote:
Do you mean that on other systems it does *not* create these gigantic files??!? I've always wondered what the use was... Hmm, if this test test support for files with holes, how come that Windows then doesn't have the same problem as the Mac, it also doesn't support holes in files, or does it? -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -

Jack Jansen <Jack.Jansen@oratrix.com> writes:
Do you mean that on other systems it does *not* create these gigantic files??!?
Certainly not. Being able to seek to some large file offset, then write one byte, without consuming huge amounts of disk space, has been a long-standing Unix feature.
Depends on the file system; NTFS surely does. However, on Windows, the test is not run unless the largefile resource is given to regrtest.py. Regards, Martin

Jack Jansen:
Martin v. Löwis: # Depends on the file system; NTFS surely does. # # However, on Windows, the test is not run unless the largefile resource # is given to regrtest.py. NTFS has only supported sparse files since Windows 2000 and then only if you set the sparse file attribute. http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base /sparse_files.asp Neil

RTSL: # On Windows this test comsumes large resources; It takes a long time to build # the >2GB file and takes >2GB of disk space therefore the resource must be # enabled to run this test. If not, nothing after this line stanza will be # executed. if sys.platform[:3] == 'win': test_support.requires( 'largefile', 'test requires %s bytes and a long time to run' % str(size)) --Guido van Rossum (home page: http://www.python.org/~guido/)

On donderdag, nov 28, 2002, at 15:02 Europe/Amsterdam, Guido van Rossum wrote:
Ok, I'll add mac and darwin to the platform tests, then. -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -

[Jack Jansen]
No, it's testing support for large files period. If the OS supports large files, and you've got enough disk space, it should pass regardless of whether the filesystem sticks holes in the middle.
how come that Windows then doesn't have the same problem as the Mac, it also doesn't support holes in files, or does it?
Holes are irrelevant to correct functioning. On Win2K the test passes and does indeed take a very long time. On Win9X the test passes but takes very little time. The difference seems to be that Win2K actually fills a file with gigabytes of NUL characters, but on Win9X you effectively get a window onto whatever happened to be sitting on disk in the blocks allocated for the file. But in neither case are there holes. Here on Win98SE:
You could very well find personal info in there! Win2K prevents that.

"GvR" == Guido van Rossum <guido@python.org> writes:
GvR> - test_re crashes unless I do ulimit -s 2000 -- haven't tried GvR> other values, but the default of 512 (KB) is insufficient. I GvR> know the README file explains this, but now that I've GvR> experienced this myself, I wonder if we shouldn't hack main() GvR> to increase the stack size to 2 MB, inside an #ifdef darwin GvR> or something like that. +1, but I'll defer to Jack. We discussed this a while back (for Py2.2.2 IIRC), which is when I updated the README bit. FWIW, Pipermail (in Mailman) suffers the same fate and contains the following bit of code. -Barry -------------------- snip snip -------------------- # MacOSX has a default stack size that is too small for deeply recursive # regular expressions. We see this as crashes in the Python test suite when # running test_re.py and test_sre.py. The fix is to set the stack limit to # 2048; the general recommendation is to do in the shell before running the # test suite. But that's inconvenient for a daemon like the qrunner. # # AFAIK, this problem only affects the archiver, so we're adding this work # around to this file (it'll get imported by the bundled pipermail or by the # bin/arch script. We also only do this on darwin, a.k.a. MacOSX. if sys.platform == 'darwin': try: import resource except ImportError: pass else: soft, hard = resource.getrlimit(resource.RLIMIT_STACK) newsoft = min(hard, max(soft, 1024*2048)) resource.setrlimit(resource.RLIMIT_STACK, (newsoft, hard))

On Monday, Dec 2, 2002, at 05:39 Europe/Amsterdam, Barry A. Warsaw wrote:
I'm +0 on this. The reason I'm not +1 is that 2MB is just as arbitrary as 1MB, and it's only picked because it'll make the current set of tests pass. And the current set of tests doesn't pass on MacOSX because another equally arbitrary number in test_re happens to create a stack >= 1MB but <= 2MB. I would be +1 on just changing the test to fit in a 1MB stack. That is, unless there are subtleties in the test that I missed, and there really is a reason for the specific choice of the number 50000. -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -

[Jack] (BTW all your mail comes poorly folded like shown here. What's up with that?)
I vaguely recall that the number 50000 is somewhat historically significant, but I don't remember why. But it works on all other platforms I know. Do you know if setting the stack limit actually allocates that much memory for the stack in the process in Mac OSX, or does it only reserve VM space (like on traditional Unix)? --Guido van Rossum (home page: http://www.python.org/~guido/)

On Monday, Dec 2, 2002, at 11:30 Europe/Amsterdam, Guido van Rossum wrote:
Except all other platforms with small stacks (MacOS9, WinCE, probably more).
I don't know, but I can't imagine it does anything more than reserve VM space. -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -

Guido:
Strictly speaking, in traditional Unix it doesn't do either, it just sets a limit on how big the stack can grow. Actually allocating stack pages is done on demand. I expect this is what MacOSX does. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

[Barry]
Thanks! I've added your code to regrtest.py. I'm reluctant to do this in PyMain(), because I don't know whether this adds to Python's resource usage; the re tests are pretty extreme. I'm also considering disabling test_largefile on Mac OSX unless a -u option is given. I'm still seeing errors in test_socket; today (on a different network) I got this: test test_socket failed -- Traceback (most recent call last): File "/Users/guido/python/src/Lib/test/test_socket.py", line 223, in testHostnameRes ip = socket.gethostbyname(hostname) gaierror: (7, 'No address associated with nodename') --Guido van Rossum (home page: http://www.python.org/~guido/)

"GvR" == Guido van Rossum <guido@python.org> writes:
GvR> [Barry] >> +1, but I'll defer to Jack. We discussed this a while back >> (for Py2.2.2 IIRC), which is when I updated the README bit. >> FWIW, Pipermail (in Mailman) suffers the same fate and contains >> the following bit of code. GvR> Thanks! I've added your code to regrtest.py. I'm reluctant GvR> to do this in PyMain(), because I don't know whether this GvR> adds to Python's resource usage; the re tests are pretty GvR> extreme. Cool, works for me. GvR> I'm also considering disabling test_largefile on Mac OSX GvR> unless a -u option is given. GvR> I'm still seeing errors in test_socket; today (on a different GvR> network) I got this: GvR> test test_socket failed -- Traceback (most recent call last): GvR> File "/Users/guido/python/src/Lib/test/test_socket.py", GvR> line 223, in testHostnameRes ip = GvR> socket.gethostbyname(hostname) GvR> gaierror: (7, 'No address associated with nodename') I'll be booting up OSX tonight, so I'll try doing a cvsup and running the tests locally. -Barry

Guido> I'm still seeing errors in test_socket; today (on a different Guido> network) I got this: Guido> test test_socket failed -- Traceback (most recent call last): Guido> File "/Users/guido/python/src/Lib/test/test_socket.py", line 223, in testHostnameRes Guido> ip = socket.gethostbyname(hostname) Guido> gaierror: (7, 'No address associated with nodename') What does the hostname(1) command return? By default I think it generates some oddball thing like "Montanaro.local.". Apple calls it a Rendevous name. (I've no idea what that is.) The fix for me was recommended by another group: get a dyndns.org account and change the HOSTNAME line in /etc/hostconfig from HOSTNAME=-AUTOMATIC- to HOSTNAME=montanaro.dyndns.org Skip

Thanks, Skip, but I'm more interested in fixing test_socket so it doesn 't rely on this. Maybe testHostnameRes() should be disabled when gethostbyaddr() gives an error? It makes unwarranted assumptions IMO about how DNS is set up. --Guido van Rossum (home page: http://www.python.org/~guido/)

"GvR" == Guido van Rossum <guido@python.org> writes:
GvR> Thanks, Skip, but I'm more interested in fixing test_socket GvR> so it doesn 't rely on this. Maybe testHostnameRes() should GvR> be disabled when gethostbyaddr() gives an error? It makes GvR> unwarranted assumptions IMO about how DNS is set up. BTW, test_socket.py passes just fine for me. hostname(1) returns "freewill.local." (yes, trailing dot). The difference may be that I used Netinfo Manager to add hostname entries for all my local machines, including the MacOSX machine I ran the tests on. My Netinfo-fu is pretty meager so I don't know if I did it completely right, but it seems to work for me. -Barry

That fixed it for me too. I still think the test should be skipped rather than fail when gethostbyname() raises an error. --Guido van Rossum (home page: http://www.python.org/~guido/)

On donderdag, nov 28, 2002, at 05:16 Europe/Amsterdam, Guido van Rossum wrote:
Do you mean that on other systems it does *not* create these gigantic files??!? I've always wondered what the use was... Hmm, if this test test support for files with holes, how come that Windows then doesn't have the same problem as the Mac, it also doesn't support holes in files, or does it? -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -

Jack Jansen <Jack.Jansen@oratrix.com> writes:
Do you mean that on other systems it does *not* create these gigantic files??!?
Certainly not. Being able to seek to some large file offset, then write one byte, without consuming huge amounts of disk space, has been a long-standing Unix feature.
Depends on the file system; NTFS surely does. However, on Windows, the test is not run unless the largefile resource is given to regrtest.py. Regards, Martin

Jack Jansen:
Martin v. Löwis: # Depends on the file system; NTFS surely does. # # However, on Windows, the test is not run unless the largefile resource # is given to regrtest.py. NTFS has only supported sparse files since Windows 2000 and then only if you set the sparse file attribute. http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base /sparse_files.asp Neil

RTSL: # On Windows this test comsumes large resources; It takes a long time to build # the >2GB file and takes >2GB of disk space therefore the resource must be # enabled to run this test. If not, nothing after this line stanza will be # executed. if sys.platform[:3] == 'win': test_support.requires( 'largefile', 'test requires %s bytes and a long time to run' % str(size)) --Guido van Rossum (home page: http://www.python.org/~guido/)

On donderdag, nov 28, 2002, at 15:02 Europe/Amsterdam, Guido van Rossum wrote:
Ok, I'll add mac and darwin to the platform tests, then. -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -

[Jack Jansen]
No, it's testing support for large files period. If the OS supports large files, and you've got enough disk space, it should pass regardless of whether the filesystem sticks holes in the middle.
how come that Windows then doesn't have the same problem as the Mac, it also doesn't support holes in files, or does it?
Holes are irrelevant to correct functioning. On Win2K the test passes and does indeed take a very long time. On Win9X the test passes but takes very little time. The difference seems to be that Win2K actually fills a file with gigabytes of NUL characters, but on Win9X you effectively get a window onto whatever happened to be sitting on disk in the blocks allocated for the file. But in neither case are there holes. Here on Win98SE:
You could very well find personal info in there! Win2K prevents that.

"GvR" == Guido van Rossum <guido@python.org> writes:
GvR> - test_re crashes unless I do ulimit -s 2000 -- haven't tried GvR> other values, but the default of 512 (KB) is insufficient. I GvR> know the README file explains this, but now that I've GvR> experienced this myself, I wonder if we shouldn't hack main() GvR> to increase the stack size to 2 MB, inside an #ifdef darwin GvR> or something like that. +1, but I'll defer to Jack. We discussed this a while back (for Py2.2.2 IIRC), which is when I updated the README bit. FWIW, Pipermail (in Mailman) suffers the same fate and contains the following bit of code. -Barry -------------------- snip snip -------------------- # MacOSX has a default stack size that is too small for deeply recursive # regular expressions. We see this as crashes in the Python test suite when # running test_re.py and test_sre.py. The fix is to set the stack limit to # 2048; the general recommendation is to do in the shell before running the # test suite. But that's inconvenient for a daemon like the qrunner. # # AFAIK, this problem only affects the archiver, so we're adding this work # around to this file (it'll get imported by the bundled pipermail or by the # bin/arch script. We also only do this on darwin, a.k.a. MacOSX. if sys.platform == 'darwin': try: import resource except ImportError: pass else: soft, hard = resource.getrlimit(resource.RLIMIT_STACK) newsoft = min(hard, max(soft, 1024*2048)) resource.setrlimit(resource.RLIMIT_STACK, (newsoft, hard))

On Monday, Dec 2, 2002, at 05:39 Europe/Amsterdam, Barry A. Warsaw wrote:
I'm +0 on this. The reason I'm not +1 is that 2MB is just as arbitrary as 1MB, and it's only picked because it'll make the current set of tests pass. And the current set of tests doesn't pass on MacOSX because another equally arbitrary number in test_re happens to create a stack >= 1MB but <= 2MB. I would be +1 on just changing the test to fit in a 1MB stack. That is, unless there are subtleties in the test that I missed, and there really is a reason for the specific choice of the number 50000. -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -

[Jack] (BTW all your mail comes poorly folded like shown here. What's up with that?)
I vaguely recall that the number 50000 is somewhat historically significant, but I don't remember why. But it works on all other platforms I know. Do you know if setting the stack limit actually allocates that much memory for the stack in the process in Mac OSX, or does it only reserve VM space (like on traditional Unix)? --Guido van Rossum (home page: http://www.python.org/~guido/)

On Monday, Dec 2, 2002, at 11:30 Europe/Amsterdam, Guido van Rossum wrote:
Except all other platforms with small stacks (MacOS9, WinCE, probably more).
I don't know, but I can't imagine it does anything more than reserve VM space. -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -

Guido:
Strictly speaking, in traditional Unix it doesn't do either, it just sets a limit on how big the stack can grow. Actually allocating stack pages is done on demand. I expect this is what MacOSX does. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

[Barry]
Thanks! I've added your code to regrtest.py. I'm reluctant to do this in PyMain(), because I don't know whether this adds to Python's resource usage; the re tests are pretty extreme. I'm also considering disabling test_largefile on Mac OSX unless a -u option is given. I'm still seeing errors in test_socket; today (on a different network) I got this: test test_socket failed -- Traceback (most recent call last): File "/Users/guido/python/src/Lib/test/test_socket.py", line 223, in testHostnameRes ip = socket.gethostbyname(hostname) gaierror: (7, 'No address associated with nodename') --Guido van Rossum (home page: http://www.python.org/~guido/)

"GvR" == Guido van Rossum <guido@python.org> writes:
GvR> [Barry] >> +1, but I'll defer to Jack. We discussed this a while back >> (for Py2.2.2 IIRC), which is when I updated the README bit. >> FWIW, Pipermail (in Mailman) suffers the same fate and contains >> the following bit of code. GvR> Thanks! I've added your code to regrtest.py. I'm reluctant GvR> to do this in PyMain(), because I don't know whether this GvR> adds to Python's resource usage; the re tests are pretty GvR> extreme. Cool, works for me. GvR> I'm also considering disabling test_largefile on Mac OSX GvR> unless a -u option is given. GvR> I'm still seeing errors in test_socket; today (on a different GvR> network) I got this: GvR> test test_socket failed -- Traceback (most recent call last): GvR> File "/Users/guido/python/src/Lib/test/test_socket.py", GvR> line 223, in testHostnameRes ip = GvR> socket.gethostbyname(hostname) GvR> gaierror: (7, 'No address associated with nodename') I'll be booting up OSX tonight, so I'll try doing a cvsup and running the tests locally. -Barry

Guido> I'm still seeing errors in test_socket; today (on a different Guido> network) I got this: Guido> test test_socket failed -- Traceback (most recent call last): Guido> File "/Users/guido/python/src/Lib/test/test_socket.py", line 223, in testHostnameRes Guido> ip = socket.gethostbyname(hostname) Guido> gaierror: (7, 'No address associated with nodename') What does the hostname(1) command return? By default I think it generates some oddball thing like "Montanaro.local.". Apple calls it a Rendevous name. (I've no idea what that is.) The fix for me was recommended by another group: get a dyndns.org account and change the HOSTNAME line in /etc/hostconfig from HOSTNAME=-AUTOMATIC- to HOSTNAME=montanaro.dyndns.org Skip

Thanks, Skip, but I'm more interested in fixing test_socket so it doesn 't rely on this. Maybe testHostnameRes() should be disabled when gethostbyaddr() gives an error? It makes unwarranted assumptions IMO about how DNS is set up. --Guido van Rossum (home page: http://www.python.org/~guido/)

"GvR" == Guido van Rossum <guido@python.org> writes:
GvR> Thanks, Skip, but I'm more interested in fixing test_socket GvR> so it doesn 't rely on this. Maybe testHostnameRes() should GvR> be disabled when gethostbyaddr() gives an error? It makes GvR> unwarranted assumptions IMO about how DNS is set up. BTW, test_socket.py passes just fine for me. hostname(1) returns "freewill.local." (yes, trailing dot). The difference may be that I used Netinfo Manager to add hostname entries for all my local machines, including the MacOSX machine I ran the tests on. My Netinfo-fu is pretty meager so I don't know if I did it completely right, but it seems to work for me. -Barry

That fixed it for me too. I still think the test should be skipped rather than fail when gethostbyname() raises an error. --Guido van Rossum (home page: http://www.python.org/~guido/)
participants (10)
-
barry@python.org
-
Brett Cannon
-
Greg Ewing
-
Guido van Rossum
-
Jack Jansen
-
Jack Jansen
-
martin@v.loewis.de
-
Neil Hodgson
-
Skip Montanaro
-
Tim Peters