ASCII delimited files
D'Arcy J.M. Cain
darcy at vex.net
Thu Nov 11 18:05:16 CET 1999
D'Arcy J.M. Cain <darcy at vex.net> wrote:
> I couldn't find this on the Python or Starship sites but you can get it
> from http://www.deja.com/getdoc.xp?AN=448769706. One small problem with
> it is that it doesn't handle commas within quoted strings properly. I
> am going to look at this today for another project and I will post it
> here when I have something.
And here it is. Note that Christian Tismer wrote the original (at least
he posted it) but didn't leave his name in the script itself. I didn't
want to add it in since he may have had his reasons but I wanted to
acknowledge that he wrote it.
One thing I didn't change was the replacement of newlines with commas
although I'm not sure that that is the best thing to replace it with.
In fact, why replace it at all? I may change this for myself.
Oh, if anyone has a suggestion for a more efficient way to handle
embedded commas I sure would be interested.
#! /usr/bin/env python
# Modified from the original by D'Arcy J.M. Cain <darcy at druid.net>
# CT990226 V0.1
breaking tab delimited or CSV text into a list of tuples.
Original version: SMC databook plugin, readxls.py
Generalized to be usable for any delimiter but \n.
def split_delimited(s, delimiter=None) :
"""split a delimited text file string into a list of tuples.
Quotes are obeyed, enclosed newlines are expanded to tab,
double quotes go to quotes"""
# 980426 finding line delimiter dynamically
probe = s[:10000]
eol = findlinedelimiter(probe)
# 990226 guessing field delimiter from '\t,' if not supplied
if not delimiter:
candidates = [
(string.count(probe, '\t'), '\t'),
(string.count(probe, ','), ','),
delimiter = candidates[-1][-1]
# the trick to make this handy is to use \0 as a placeholder
# Kind of ugly but it works for embedded commas - DJMC
inquote = 0
for i in range(len(s)):
if s[i] == '"': inquote = (inquote == 0)
if inquote == 0 and s[i] == delimiter:
s = s[:i] + "\0" + s[i + 1:]
parts = string.split(s, '"')
limits = (0, len(parts)-1)
for i in range(len(parts)) :
# may as well strip the spaces now - DJMC
part = string.strip(parts[i])
if i%2 :
part = string.replace(part, eol, delimiter)
if not part and i not in limits: part = '"'
parts[i] = part
# merge it back
txt = string.join(parts, "")
parts = string.split(txt, eol)
# now break by \0
for i in range(len(parts)) :
fields = string.split(parts[i], "\0")
parts[i] = tuple(fields)
def findlinedelimiter(txt) :
provide some kb of text to this function. It will determine
the best delimiter and therefore guess the system
mac = "\x0D"
unix = "\x0A"
dos = mac+unix
oses = [dos, unix, mac]
# find the one which gives the most lines.
# in doubt, the longest delimiter wins.
lis = 
while txt and txt[-1] in dos: txt = txt[:-1] # CT970904
for delim in oses:
lis.append((len(string.split(txt, delim)), delim))
for l in split_delimited('''1, 2,3, "vier", "quo""te", "embedded, comma", "this
is with a newline", here
D'Arcy J.M. Cain <darcy at caingang.com> | Democracy is three wolves
http://www.druid.net/darcy/ | and a sheep voting on
+1 416 425 1212 (DoD#0082) (eNTP) | what's for dinner.
More information about the Python-list