write csv to object and read into pandas

[Oscar Benjamin]

On 15 October 2015 at 09:16, Peter Otten <__peter__ at web.de> wrote:
>
> def preprocess(filename):
>     with open(filename) as f:
>         for row in csv.reader(f):
>             # do stuff
>             yield row
>
> rows = preprocess("pandas.csv")

Take the with statement outside of the generator and do something like:

def preprocess(lines):
    for row in csv.reader(lines):
        # do stuff
        yield row

with open("pandas.csv") as pandasfile:
    rows = preprocess(pandasfile)
    df = pandas... # etc.

This ensures that the file will be closed if an exception occurs
outside the generator or if the generator is not fully consumed.

> df = pandas.DataFrame.from_records(rows, columns=next(rows))

next(rows) can raise StopIteration which can be problematic so perhaps
something like:

try:
    fieldnames = next(rows)
except StopIteration:
    raise ValueError
df = pandas... (columns=fieldnames) # etc

--
Oscar