[Pythonmac-SIG] Re: [Python-Dev] Import hook to do end-of-line
conversion?
Chris Barker
chrishbarker@home.net
Tue, 22 May 2001 16:21:01 -0700
This is a multi-part message in MIME format.
--------------B9643430766B782E71A5BE98
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Just van Rossum wrote:
> Agreed. I'll try to write one, once I'm feeling better: having the flu doesn't
> seem to help focussing on actual content...
>
> Just
Just (or anyone else)
Have you made any progress on this PEP? I'd like to see it happen, so if
you havn't done it, I'll try to find the time to make a start on it
myself.
I have written a simple class that impliments a line-ending-neutral text
file class. I wrote it because I have a need for it, and I thought it
would be a reasonable prototype for any syntax and methods we might want
to use in an actual implimentation. I doubt anyone would find the
methods I used particularly clean or elegant (or fast) but it's the
first thing I've come up with, and it seems to work.
I've enclosed the module with this email. If that doesn't work, let me
know and I'll put it on a website.
-Chris
--
Christopher Barker,
Ph.D.
ChrisHBarker@home.net --- --- ---
http://members.home.net/barkerlohmann ---@@ -----@@ -----@@
------@@@ ------@@@ ------@@@
Oil Spill Modeling ------ @ ------ @ ------ @
Water Resources Engineering ------- --------- --------
Coastal and Fluvial Hydrodynamics --------------------------------------
------------------------------------------------------------------------
--------------B9643430766B782E71A5BE98
Content-Type: text/plain; charset=us-ascii;
name="TextFile.py"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="TextFile.py"
#!/usr/bin/env python
"""
TextFile.py : a module that provides a UniversalTextFile class, and a
replacement for the native python "open" command that provides an
interface to that class.
It would usually be used as:
from TextFile import open
then you can use the new open just like the old one (with some added flags and arguments)
or
import TextFile
file = TextFile.open(filename,flags,[bufsize], [LineEndingType], [LineBufferSize])
"""
import os
## Re-map the open function
_OrigOpen = open
def open(filename,flags = "",bufsize = -1, LineEndingType = "", LineBufferSize = ""):
"""
A new open function, that returns a regular python file object for
the old calls, and returns a new nifty universal text file when
required.
This works just like the regular open command, except that a new
flag and a new parameter has been added.
Call:
file = open(filename,flags = "",bufsize = -1, LineEndingType = ""):
- filename is the name of the file to be opened
- flags is a string of one letter flags, the same as the standard open
command, plus a "t" for universal text file.
- - "b" means binary file, this returns the standard binary file object
- - "t" means universal text file
- - "r" for read only
- - "w" for write. If there is both "w" and "t" than the user can
specify a line ending type to be used with the LineEndingType
parameter.
- - "a" means append to existing file
- bufsize specifies the buffer size to be used by the system. Same
as the regular open function
- LineEndingType is used only for writing (and appending) files, to specify a
non-native line ending to be written.
- - The options are: "native", "DOS", "Posix", "Unix", "Mac", or the
characters themselves( "\r\n", etc. ). "native" will result in
using the standard file object, which uses whatever is native
for the system that python is running on.
- LineBufferSize is the size of the buffer used to read data in
a readline() operation. The default is currently set to 200
characters. If you will be reading files with many lines over 200
characters long, you should set this number to the largest expected
line length.
"""
if "t" in flags: # this is a universal text file
if ("w" in flags or "a" in flags) and LineEndingType == "native":
return _OrigOpen(filename,flags.replace("t",""), bufsize)
return UniversalTextFile(filename,flags,LineEndingType,LineBufferSize)
else: # this is a regular old file
return _OrigOpen(filename,flags,bufsize)
class UniversalTextFile:
"""
A class that acts just like a python file object, but has a mode
that allows the reading of arbitrary formated text files, i.e. with
either Unix, DOS or Mac line endings. [\n , \r\n, or \r]
To keep it truly universal, it checks for each of these line ending
possibilities at every line, so it should work on a file with mixed
endings as well.
"""
def __init__(self,filename,flags = "",LineEndingType = "native",LineBufferSize = ""):
self._file = _OrigOpen(filename,flags.replace("t","")+"b")
LineEndingType = LineEndingType.lower()
if LineEndingType == "native":
self.LineSep = os.linesep()
elif LineEndingType == "dos":
self.LineSep = "\r\n"
elif LineEndingType == "posix" or LineEndingType == "unix" :
self.LineSep = "\n"
elif LineEndingType == "mac":
self.LineSep = "\r"
else:
self.LineSep = LineEndingType
## some attributes
self.closed = 0
self.mode = flags
self.softspace = 0
if LineBufferSize:
self._BufferSize = LineBufferSize
else:
self._BufferSize = 100
def readline(self):
start_pos = self._file.tell()
##print "Current file posistion is:", start_pos
line = ""
TotalBytes = 0
Buffer = self._file.read(self._BufferSize)
while Buffer:
##print "Buffer = ",repr(Buffer)
newline_pos = Buffer.find("\n")
return_pos = Buffer.find("\r")
if return_pos == newline_pos-1 and return_pos >= 0: # we have a DOS line
line = Buffer[:return_pos]+ "\n"
TotalBytes = newline_pos+1
break
elif ((return_pos < newline_pos) or newline_pos < 0 ) and return_pos >=0: # we have a Mac line
line = Buffer[:return_pos]+ "\n"
TotalBytes = return_pos+1
break
elif newline_pos >= 0: # we have a Posix line
line = Buffer[:newline_pos]+ "\n"
TotalBytes = newline_pos+1
break
else: # we need a larger buffer
NewBuffer = self._file.read(self._BufferSize)
if NewBuffer:
Buffer = Buffer + NewBuffer
else: # we are at the end of the file, without a line ending.
self._file.seek(start_pos + len(Buffer))
return Buffer
self._file.seek(start_pos + TotalBytes)
return line
def readlines(self,sizehint = None):
"""
readlines acts like the regular readlines, except that it
understands any of the standard text file line endings ("\r\n",
"\n", "\r").
If sizehint is used, it will read a a mximum of that many
bytes. It will not round up, as the regular readline does. This
means that if your buffer size is less thatn the length of the
next line, you won't get anything.
"""
if sizehint:
Data = self._file.read(sizehint)
else:
Data = self._file.read()
if len(Data) == sizehint:
#print "The buffer is full"
FullBuffer = 1
else:
FullBuffer = 0
Data = Data.replace("\r\n","\n").replace("\r","\n")
Lines = [line + "\n" for line in Data.split('\n')]
#print Lines
## If the last line is only a linefeed it is an extra line
if Lines[-1] == "\n":
del Lines[-1]
## if it isn't then the last line didn't have a linefeed, so we need to remove the one we put on.
else:
## or it's the end of the buffer
if FullBuffer:
#print "the file is at:",self._file.tell()
#print "the last line has length:",len(Lines[-1])
self._file.seek(-(len(Lines[-1])-1),1) # reset the file position
del(Lines[-1])
else:
Lines[-1] = Lines[-1][:-1]
return Lines
def readnumlines(self,NumLines = 1):
"""
readnumlines is an extension to the standard file object. It
returns a list containing the number of lines that are
requested. I have found this to be very usefull, and allows me to avoid the many loops like:
lines = []
for i in range(N):
lines.append(file.readline())
Also, If I ever get around to writing this in C, it will provide a speed improvement.
"""
Lines = []
while len(Lines) < NumLines:
Lines.append(self.readline())
return Lines
def read(self,size = None):
"""
read acts like the regular read, except that it tranlates any of
the standard text file line endings ("\r\n", "\n", "\r") into a
"\n"
If size is used, it will read a maximum of that many bytes,
before translation. This means that if the line endings have
more than one character, the size returned will be smaller. This
could gbe patched, but it didn't seem worth it. If you want that
much control, use a binary file.
"""
if size:
Data = self._file.read(size)
else:
Data = self._file.read()
return Data.replace("\r\n","\n").replace("\r","\n")
def write(self,string):
"""
write is just like the regular one, except that it uses the line
separator specified when the file was opened for writing or
appending.
"""
self._file.write(string.replace("\n",self.LineSep))
def writelines(self,list):
for line in list:
self.write(line)
# The rest of the standard file methods mapped
def close(self):
self._file.close()
self.closed = 1
def flush(self):
self._file.flush()
def fileno(self):
return self._file.fileno()
def seek(self,offset,whence = 0):
self._file.seek(offset,whence)
def tell(self):
return self._file.tell()
--------------B9643430766B782E71A5BE98--