String multi-replace

Steven D'Aprano steve-REMOVE-THIS at
Thu Nov 18 06:10:03 CET 2010

On Wed, 17 Nov 2010 20:21:06 -0800, Sorin Schwimmer wrote:

> Hi All,
> I have to eliminate diacritics in a fairly large file.

What's "fairly large"? Large to you is probably not large to your 
computer. Anything less than a few dozen megabytes is small enough to be 
read entirely into memory.

> Inspired by, I came up with
> the following code:

If all you are doing is replacing single characters, then there's no need 
for the 80lb sledgehammer of regular expressions when all you need is a 
delicate tack hammer. Instead of this:

* read the file as bytes
* search for pairs of bytes like chr(195)+chr(130) using a regex
* replace them with single bytes like 'A'

do this:

* read the file as a Unicode 
* search for characters like Â
* replace them with single characters like A using unicode.translate()

(or str.translate() in Python 3.x)

The only gotcha is that you need to know (or guess) the encoding to read 
the file correctly.


More information about the Python-list mailing list