[Tutor] CSV file processing...

[Dave Kuhlman]

On Thu, Mar 20, 2008 at 03:16:05PM -0600, Spencer Parker wrote:
> I am trying to read a CSV file and the get that information into a MySQL
> database.  I am able to do this, but I have a small problem.  I have a piece
> of software that runs and each iteration is one like.  It only runs once
> right now; there is only one line + the headers.  I use the csv module to
> kill the headers and import the one line.  The problem is...I need to have
> it split the csv file at some point. I need to first 20 items taken off and
> then have the next 7 + the first 20 imported into the database...then have
> it do this for the next 7 + the first 20...so on and so forth until hits the
> end of the line.
> 
> Any ideas?

Consider the following bit of code:

    In [6]: f = open('csv_report.csv', 'r')
    In [7]: r = csv.reader(f)
    In [8]: lines = [line for line in r]
    In [9]: lines
    Out[9]:
    [['Name', 'Description', 'Rating'],
     ['Lemon', 'Bright yellow and tart', '5'],
     ['Eggplant', 'Purple and shiny', '6'],
     ['Tangerine', 'Succulent', '8']]

Once you get a list of lists (a list of rows containing lists of
fields), you can process and skip whichever rows you wish.

Are the items you want to skip/process the rows or the fields.  If
it's the fields, then you might consider flattening your list of
lists.  If you need to do that, then the following might help:

    In [23]: fields = []
    In [24]: for line in lines:
       ....:     for field in line:
       ....:         fields.append(field)
       ....:
    In [25]: fields
    Out[25]:
    ['Name',
     'Description',
     'Rating',
     'Lemon',
     'Bright yellow and tart',
     '5',
     'Eggplant',
     'Purple and shiny',
     '6',
     'Tangerine',
     'Succulent',
     '8']

Then process *that* list.

Does someone know a slicker way to flatten a list?  I could not
find anything helpful in itertools.

Or, is your input so large that you do not want to read it in all at
once.  It would have to be quite large before that would be a
problem.

- Dave


-- 
Dave Kuhlman
http://www.rexx.com/~dkuhlman