FW: Reading a remove csv file
Terry Reedy
tjreedy at udel.edu
Thu Nov 2 18:57:12 EDT 2017
On 11/2/2017 9:18 AM, ROGER GRAYDON CHRISTMAN wrote:
> I have a partial answer to my own question:
> This seems to work for me:
>
> ---
> link = urllib.request.urlopen(urlpath)
> data = link.read().decode('utf-8').split('\n')
>
> reader = csv.DictReader(data)
> for row in reader:
> ---
>
> I think here my concern is that now 'data' is now a variable
> in my program's memory (all of the data),
> instead of streamlike. I suppose I did read
> somewhere about setting a stream option.
csv.reader and csv.DictReader are transformation iterators whose first
argument must be an iterable of string lines. Given an iterator of
bytes lines, you just need to interpose a bytes to string iterator --
something like (untested)
def strgen(bytesource, encoding):
for line in bytesource:
yield line.decode(encoding)
with urllib.request.urlopen(urlpath) as link:
lines = strgen(link, 'utf-8')
reader = csv.DictReader(lines) # plus any other args
for row in reader:
process(row)
Iterators are intended to be chained together like this.
--
Terry Jan Reedy
More information about the Python-list
mailing list