[Patches] test largefile support (test_largefile.py)

Trent Mick trentm@activestate.com
Thu, 1 Jun 2000 18:48:39 -0700


Discussion:

This patch adds a test for largefiles (creating, seeking, telling, etc.).
The test skips if there is no largefile support.

There is one further problem. The test basically involves creating a file
greater than 2GB and playing with it. On UN*X systems with sparse files this
is no problem. On Win64 (which I have heard *can* do sparse files, but not in
Python yet), however, >2GB space and a *long* time is required to run the
test. I don't think it is reasonable to turn this on by default... so here is
what I did.

I extended regrtest.py to accept the --have-resources switch. This sets
test_support.use_large_resources, which is checked in test_largefile.py.
By default 'use_large_resources' is false. On Win64, then, by default the
largefile test is skipped but can be run via the --have-resources switch to
regrtest.py or by running the test directly. This seems to me the Right
Thing.

The affected files are:
 Lib/test/regrtest.py
 Lib/test/test_support.py
 Lib/test/test_largefile.py (new)
 Lib/test/output/test_largefile (new)



Legal:

I confirm that, to the best of my knowledge and belief, this
contribution is free of any claims of third parties under
copyright, patent or other rights or interests ("claims").  To
the extent that I have any such claims, I hereby grant to CNRI a
nonexclusive, irrevocable, royalty-free, worldwide license to
reproduce, distribute, perform and/or display publicly, prepare
derivative versions, and otherwise use this contribution as part
of the Python software and its related documentation, or any
derivative versions thereof, at no cost to CNRI or its licensed
users, and to authorize others to do so.

I acknowledge that CNRI may, at its sole discretion, decide
whether or not to incorporate this contribution in the Python
software and its related documentation.  I further grant CNRI
permission to use my name and other identifying information
provided to CNRI by me for use in connection with the Python
software and its related documentation.


Patch:

*** /home/trentm/main/contrib/python/dist/src/Lib/test/regrtest.py	Thu Jun  1 00:13:38 2000
--- /home/trentm/main/Apps/Perlium/Python/dist/src/Lib/test/regrtest.py	Thu Jun  1 16:11:30 2000
***************
*** 13,18 ****
--- 13,19 ----
  -g: generate -- write the output file for a test instead of comparing it
  -x: exclude  -- arguments are tests to *exclude*
  -s: single   -- run only a single test (see below)
+ --have-resources   -- run tests that require large resources (time/space)
  
  If non-option arguments are present, they are names for tests to run,
  unless -x is given, in which case they are names for tests not to run.
***************
*** 56,62 ****
      """
      
      try:
!         opts, args = getopt.getopt(sys.argv[1:], 'vgqxs')
      except getopt.error, msg:
          print msg
          print __doc__
--- 57,63 ----
      """
      
      try:
!         opts, args = getopt.getopt(sys.argv[1:], 'vgqxs', ['have-resources'])
      except getopt.error, msg:
          print msg
          print __doc__
***************
*** 66,77 ****
--- 67,80 ----
      generate = 0
      exclude = 0
      single = 0
+     use_large_resources = 0
      for o, a in opts:
          if o == '-v': verbose = verbose+1
          if o == '-q': quiet = 1; verbose = 0
          if o == '-g': generate = 1
          if o == '-x': exclude = 1
          if o == '-s': single = 1
+         if o == '--have-resources': use_large_resources = 1
      if generate and verbose:
          print "-g and -v don't go together!"
          return 2
***************
*** 105,123 ****
      if single:
          tests = tests[:1]
      test_support.verbose = verbose      # Tell tests to be moderately quiet
      save_modules = sys.modules.keys()
      for test in tests:
          if not quiet:
              print test
!         ok = runtest(test, generate, verbose, testdir)
          if ok > 0:
              good.append(test)
          elif ok == 0:
              bad.append(test)
          else:
-             if not quiet:
-                 print "test", test,
-                 print "skipped -- an optional feature could not be imported"
              skipped.append(test)
          # Unload the newly imported modules (best effort finalization)
          for module in sys.modules.keys():
--- 108,124 ----
      if single:
          tests = tests[:1]
      test_support.verbose = verbose      # Tell tests to be moderately quiet
+     test_support.use_large_resources = use_large_resources
      save_modules = sys.modules.keys()
      for test in tests:
          if not quiet:
              print test
!         ok = runtest(test, generate, verbose, quiet, testdir)
          if ok > 0:
              good.append(test)
          elif ok == 0:
              bad.append(test)
          else:
              skipped.append(test)
          # Unload the newly imported modules (best effort finalization)
          for module in sys.modules.keys():
***************
*** 178,189 ****
      tests.sort()
      return stdtests + tests
  
! def runtest(test, generate, verbose, testdir = None):
      """Run a single test.
      test -- the name of the test
      generate -- if true, generate output, instead of running the test
      and comparing it to a previously created output file
      verbose -- if true, print more messages
      testdir -- test directory
      """
      test_support.unload(test)
--- 179,191 ----
      tests.sort()
      return stdtests + tests
  
! def runtest(test, generate, verbose, quiet, testdir = None):
      """Run a single test.
      test -- the name of the test
      generate -- if true, generate output, instead of running the test
      and comparing it to a previously created output file
      verbose -- if true, print more messages
+     quiet -- if true, don't print 'skipped' messages (probably redundant)
      testdir -- test directory
      """
      test_support.unload(test)
***************
*** 210,218 ****
--- 212,225 ----
          finally:
              sys.stdout = save_stdout
      except ImportError, msg:
+         if not quiet:
+             print "test", test,
+             print "skipped -- an optional feature could not be imported"
          return -1
      except KeyboardInterrupt, v:
          raise KeyboardInterrupt, v, sys.exc_info()[2]
+     except test_support.TestSkipped, msg:
+         print "test", test, "skipped --", msg
      except test_support.TestFailed, msg:
          print "test", test, "failed --", msg
          return 0
*** /home/trentm/main/contrib/python/dist/src/Lib/test/test_support.py	Thu Jun  1 00:13:38 2000
--- /home/trentm/main/Apps/Perlium/Python/dist/src/Lib/test/test_support.py	Thu Jun  1 15:52:04 2000
***************
*** 1,8 ****
--- 1,10 ----
  # Python test set -- supporting definitions.
  
  TestFailed = 'test_support -- test failed'	# Exception
+ TestSkipped = 'test_support -- test skipped'	# Exception
  
  verbose = 1				# Flag set to 0 by regrtest.py
+ use_large_resources = 1 # Flag set to 0 by regrtest.py
  
  def unload(name):
  	import sys
*** /home/trentm/main/contrib/python/dist/src/Lib/test/test_largefile.py	Thu Jun  1 18:30:28 2000
--- /home/trentm/main/Apps/Perlium/Python/dist/src/Lib/test/test_largefile.py	Thu Jun  1 17:25:23 2000
***************
*** 0 ****
--- 1,129 ----
+ #!python
+ 
+ #----------------------------------------------------------------------
+ # test largefile support on system where this makes sense
+ #
+ #XXX how to only run this when support is there
+ #XXX how to only optionally run this, it will take along time
+ #----------------------------------------------------------------------
+ 
+ import test_support
+ import os, struct, stat, sys
+ 
+ 
+ # only run if the current system support large files
+ f = open(test_support.TESTFN, 'w')
+ try:
+ 	# 2**31 == 2147483648
+ 	f.seek(2147483649L)
+ except OverflowError:
+ 	raise test_support.TestSkipped, "platform does not have largefile support"
+ else:
+ 	f.close()
+ 
+ 
+ # create >2GB file (2GB = 2147483648 bytes)
+ size = 2500000000L
+ name = test_support.TESTFN
+ 
+ 
+ # on Windows this test comsumes large resources:
+ #  it takes a long time to build the >2GB file and takes >2GB of disk space
+ # therefore test_support.use_large_resources must be defined to run this test
+ if sys.platform[:3] == 'win' and not test_support.use_large_resources:
+ 	raise test_support.TestSkipped, \
+ 		"test requires %s bytes and a long time to run" % str(size)
+ 
+ 
+ 
+ def expect(got_this, expect_this):
+ 	if test_support.verbose:
+ 		print '%s =?= %s ...' % (`got_this`, `expect_this`),
+ 	if got_this != expect_this:
+ 		if test_support.verbose:
+ 			print 'no'
+ 		raise test_support.TestFailed, 'got %s, but expected %s' %\
+ 			(str(got_this), str(expect_this))
+ 	else:
+ 		if test_support.verbose:
+ 			print 'yes'
+ 
+ 
+ # test that each file function works as expected for a large (i.e. >2GB, do
+ # we have to check >4GB) files
+ 
+ if test_support.verbose:
+ 	print 'create large file via seek (may be sparse file) ...'
+ f = open(name, 'w')
+ f.seek(size)
+ f.write('a')
+ f.flush()
+ expect(os.fstat(f.fileno())[stat.ST_SIZE], size+1)
+ if test_support.verbose:
+ 	print 'check file size with os.fstat'
+ f.close()
+ if test_support.verbose:
+ 	print 'check file size with os.stat'
+ expect(os.stat(name)[stat.ST_SIZE], size+1)
+ 
+ if test_support.verbose:
+ 	print 'play around with seek() and read() with the built largefile'
+ f = open(name, 'r')
+ expect(f.tell(), 0)
+ expect(f.read(1), '\000')
+ expect(f.tell(), 1)
+ f.seek(0)
+ expect(f.tell(), 0)
+ f.seek(0, 0)
+ expect(f.tell(), 0)
+ f.seek(42)
+ expect(f.tell(), 42)
+ f.seek(42, 0)
+ expect(f.tell(), 42)
+ f.seek(42, 1)
+ expect(f.tell(), 84)
+ f.seek(0, 1)
+ expect(f.tell(), 84)
+ f.seek(0, 2) # seek from the end
+ expect(f.tell(), size + 1 + 0)
+ f.seek(-10, 2)
+ expect(f.tell(), size + 1 - 10)
+ f.seek(-size-1, 2)
+ expect(f.tell(), 0)
+ f.seek(size)
+ expect(f.tell(), size)
+ expect(f.read(1), 'a') # the 'a' that was written at the end of the file above
+ f.close()
+ 
+ if test_support.verbose:
+ 	print 'play around with os.lseek() with the built largefile'
+ f = open(name, 'r')
+ expect(os.lseek(f.fileno(), 0, 0), 0)
+ expect(os.lseek(f.fileno(), 42, 0), 42)
+ expect(os.lseek(f.fileno(), 42, 1), 84)
+ expect(os.lseek(f.fileno(), 0, 1), 84)
+ expect(os.lseek(f.fileno(), 0, 2), size+1+0)
+ expect(os.lseek(f.fileno(), -10, 2), size+1-10)
+ expect(os.lseek(f.fileno(), -size-1, 2), 0)
+ expect(os.lseek(f.fileno(), size, 0), size)
+ expect(f.read(1), 'a') # the 'a' that was written at the end of the file above
+ f.close()
+ 
+ 
+ # XXX add tests for truncate if it exists
+ # XXX has truncate ever worked on Windows? specifically on WinNT I get:
+ #     "IOError: [Errno 13] Permission denied"
+ ##try:
+ ##	newsize = size - 10
+ ##	f.seek(newsize)
+ ##	f.truncate()
+ ##	expect(f.tell(), newsize)
+ ##	newsize = newsize - 1
+ ##	f.seek(0)
+ ##	f.truncate(newsize)
+ ##	expect(f.tell(), newsize)
+ ##except AttributeError:
+ ##	pass
+ 
+ os.unlink(name)
+ 
*** /home/trentm/main/contrib/python/dist/src/Lib/test/output/test_largefile	Thu Jun  1 18:30:29 2000
--- /home/trentm/main/Apps/Perlium/Python/dist/src/Lib/test/output/test_largefile	Thu Jun  1 17:20:53 2000
***************
*** 0 ****
--- 1 ----
+ test_largefile

-- 
Trent Mick
trentm@activestate.com