Reading \n unescaped from a file
MRAB
python at mrabarnett.plus.com
Wed Sep 2 18:31:35 EDT 2015
On 2015-09-02 03:03, Rob Hills wrote:
> Hi,
>
> I am developing code (Python 3.4) that transforms text data from one
> format to another.
>
> As part of the process, I had a set of hard-coded str.replace(...)
> functions that I used to clean up the incoming text into the desired
> output format, something like this:
>
> dataIn = dataIn.replace('\r', '\\n') # Tidy up linefeeds
> dataIn = dataIn.replace('<','<') # Tidy up < character
> dataIn = dataIn.replace('>','>') # Tidy up < character
> dataIn = dataIn.replace('o','o') # No idea why but lots of these: convert to 'o' character
> dataIn = dataIn.replace('f','f') # .. and these: convert to 'f' character
> dataIn = dataIn.replace('e','e') # .. 'e'
> dataIn = dataIn.replace('O','O') # .. 'O'
>
The problem with this approach is that the order of the replacements
matters. For example, changing '<' to '<' and then '&' to '&'
can give a different result to changing '&' to '&' and then '<'
to '<'. If you started with the string '<', then the first order
would go '<' => '<' => '<', whereas the second order
would go '<' => '<' => '<'.
> These statements transform my data correctly, but the list of statements
> grows as I test the data so I thought it made sense to store the
> replacement mappings in a file, read them into a dict and loop through
> that to do the cleaning up, like this:
>
> with open(fileName, 'r+t', encoding='utf-8') as mapFile:
> for line in mapFile:
> line = line.strip()
> try:
> if (line) and not line.startswith('#'):
> line = line.split('#')[:1][0].strip() # trim any trailing comments
> name, value = line.split('=')
> name = name.strip()
> self.filterMap[name]=value.strip()
> except:
> self.logger.error('exception occurred parsing line [{0}] in file [{1}]'.format(line, fileName))
> raise
>
> Elsewhere, I use the following code to do the actual cleaning up:
>
> def filter(self, dataIn):
> if dataIn:
> for token, replacement in self.filterMap.items():
> dataIn = dataIn.replace(token, replacement)
> return dataIn
>
>
> My mapping file contents look like this:
>
> \r = \\n
> “ = "
> < = <
> > = >
> ' = '
> F = F
> o = o
> f = f
> e = e
> O = O
>
> This all works "as advertised" */except/* for the '\r' => '\\n'
> replacement. Debugging the code, I see that my '\r' character is
> "escaped" to '\\r' and the '\\n' to '\\\\n' when they are read in from
> the file.
>
> I've been googling hard and reading the Python docs, trying to get my
> head around character encoding, but I just can't figure out how to get
> these bits of code to do what I want.
>
> It seems to me that I need to either:
>
> * change the way I represent '\r' and '\\n' in my mapping file; or
> * transform them somehow when I read them in
>
> However, I haven't figured out how to do either of these.
>
Try ast.literal_eval, although you'd need to make it look like a string
literal first:
>>> import ast
>>> line = r'\r = \\n'
>>> print(line)
\r = \\n
>>> old, sep, new = line.partition(' = ')
>>> print(old)
\r
>>> print(new)
\\n
>>> ast.literal_eval('"%s"' % old)
'\r'
>>> ast.literal_eval('"%s"' % new)
'\\n'
>>>
I wouldn't put the &#...; forms into the mappings file (except for the
' one) because they can all be recognised and done in code
('F' is chr(int('070')), for example).
More information about the Python-list
mailing list