[Python-bugs-list] [ python-Bugs-451890 ] Building with Large File Support fails

[noreply@sourceforge.net]

Bugs item #451890, was opened at 2001-08-16 18:00
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=451890&group_id=5470

Category: Build
Group: Python 2.2
Status: Open
Resolution: None
Priority: 5
Submitted By: Gerhard Häring (ghaering)
Assigned to: Guido van Rossum (gvanrossum)
Summary: Building with Large File Support fails

Initial Comment:
(At least) on Linux, building 2.2-HEAD fails when 
building with Large File Support. In 
Objects/fileobject.c function _portable_ftell line 
262.


----------------------------------------------------------------------

>Comment By: Gerhard Häring (ghaering)
Date: 2001-09-08 12:12

Message:
Logged In: YES 
user_id=163326

Guido, I can build the current CVS now with LFS, too (Linux
2.4, glibc 2.2). I saw you did a lot in the configure
script, but I still had to give options to the make command
(grabbed them from Sean's latest source RPMs).

This worked for me:
./configure
make OPT="-g -O3 -D_FILE_OFFSET_BITS=64
-DHAVE_LARGEFILE_SUPPORT" CFLAGS="-g -O3
-D_FILE_OFFSET_BITS=64 -DHAVE_LARGEFILE_SUPPORT" 

Shouldn't the feature define HAVE_LARGEFILE_SUPPORT be
automatically added to pyconfig.h?

It would perhaps be a good idea add the info on how to build
with LFS to the build instructions.

Thanks,
Gerhard


----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-09-05 11:36

Message:
Logged In: YES 
user_id=6380

Gerhard, can you try the current CVS? I've done a few things
to try and fix this. I can now build just fine on a pretty
recent Linux 2.4 kernel.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2001-09-03 02:23

Message:
Logged In: YES 
user_id=21627

To fix the bug at hand (building fails), the following
strategy might be sufficient:
- produce an autoconf test that checks whether fpos_t is
integral, and "large"; define this by default for MSVC
- use this test in portable_fseek/portable_ftell.

I also wonder why the order in which APIs are tried is
different in fseek and ftell (fseek tries fseeko first,
ftell tries ftello only second).

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2001-08-20 13:19

Message:
Logged In: YES 
user_id=31435

By itself, adding opaque getpos/setpos sounds pretty easy 
(BTW, f{get,set}pos are std in C99).

Returning a usable 64-bit integer remains a x-platform 
mess.  The C99 rationale sez f{get,set}pos are the intended 
way to work with large files, but they provide no way to 
break the abstraction (Jeremy & I both looked in vain -- 
there is no defined way to extract the stream position from 
an fpos_t object, neither to do arithmetic on one).

On Windows, f{get,set}pos are (currently) the only way to 
get a 64-bit stream position from the MS C library (and MS 
doesn't (currently) mix that in with a state encoding; the 
Win32 API has other ways to deal with this).

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-08-20 06:21

Message:
Logged In: YES 
user_id=6380

OK, so we need to add separate getpos() and setpos() methods
that return an opaque wrapper for an fpos_t. That sounds
like serious work, plus it will require changing Python apps
that use seek and tell.

So I think we shold *also* continue to search for a way to
use a 64-bit seek offset for Python's seek() and tell()
methods -- I'm presuming this is hidden *somewhere* in the
fpos_t still, since the underlying OS certainly uses
lseek64(). If there's no way to extract it out of the
fpos_t, I propose to call lseek64() directly (after using a
fflush()) on the file descriptor.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2001-08-19 22:24

Message:
Logged In: YES 
user_id=31435

Noting that C99 *requires* fpos_t values to hold all the 
info in an mbstate_t, in addition to stream position info.  
So we have to expect others to follow glibc in this, and 
eventually everyone.  fpos_t cannot resolve to an array 
type, but anything else is fair (in particular it need not 
map to an integral type -- and probably won't anymore).

We have to give up belief that fpos_t is a number, because 
it's not.  We can believe that ftell returns a number, 
because it does <wink> -- but ftell isn't suitable for 
large file support.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2001-08-17 06:13

Message:
Logged In: YES 
user_id=21627

This started in glibc 2.2, I believe, so it would appear in
Redhat 7, SuSE 7, etc.
To see the problem, you have to ./configure with
CFLAGS="-D_FILE_OFFSET_BITS=64" OPT="-O2 $(CFLAGS)"; see
pyconfig.h.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-08-17 03:55

Message:
Logged In: YES 
user_id=6380

Whoa.  Interesting. Which Linux version is this?

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2001-08-17 00:21

Message:
Logged In: YES 
user_id=21627

This fails because in glibc, fpos_t contains an mb_state 
field, so that on restoring the file position, the 
multibyte encoding state of the file can be restored.

I see two solutions here:
- Python could give up the guarantee that the ftell result 
is a number, and return an object that embeds the fpos_t.
- Python could give up that guarantee that ftell/fseek 
works in all cases, and only use ftell(o), which should 
always return a number (atleast in Posix). If that 
approach is taken, an additional fgetpos/fsetpos call may 
be appropriate.


----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=451890&group_id=5470