Context manager for csv module 'reader' and 'DictReader'?
Hi all, the product of Sunday morning idle curiosity... I’ve been using the csv module a lot, and I’m wondering if there would be value in adding a standard mechanism for opening a CSV file (correctly) using a context manager? So, instead of with open(filename, newline=“”) as fp: r = csv.DictReader(fp) for row in r: … support something like with csv.DictReader.open(filename) as r: for row in r: … ? And something similar for ‘csv.reader’? I’m not wedded to the details here. The two main reasons I think this might be a positive addition are - * you wouldn’t have to know or remember the right way to open a CSV file (newline=“”). * it elides very common code. but perhaps there are things I’m missing here? As a side note, I think ‘csv.reader’ could usefully be renamed to something else (maybe just Reader?), since it’s kind of out of sync with the CamelCase used in ‘DictReader’. But maybe that’s just an attempt at foolish consistency :). best, —titus
This would only be helpful when the CSV is on the disk but csv.reader() takes a file object so that it can used with anything like a socket for example. json.load() does the same thing. It seems to me that it is better to keep a generic interface that are composable. Cheers, Rémi
There has been discussion about adding loading from a “path like” to the JSON lib. See this list about ten months ago and a recent thread on discuss. There resistance, but it ended with the idea that maybe there should be a PEP for a common interface for all “file” readers — eg JSON, CSV, etc.. And that interface could be supported by third party libs. That interface *maybe* would include a single step load from a path-like functionality. It seems to me that it is better to keep a generic interface that are
composable.
No one is suggesting removing the load-from-a-file-like interface. I have no idea why the fact that some people sometimes need a more flexible interface (maybe most people, even) somehow means that we shouldn’t make things easy and obvious for a common use case. -CHB
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Most Pandas read methods take either a path-like argument or a file-like argument, and figure out which it is by introspection when called. Actually, most of them even accept a URL-like argument as well I don't think this is a terrible approach. It doesn't make things quite as explicit as the standard library generally does. But it's convenient, and there's no real ambiguity. On Sun, Sep 5, 2021, 1:19 PM Christopher Barker <pythonchb@gmail.com> wrote:
On Sun, Sep 5, 2021 at 10:32 AM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:
The folks in favor of adding this to json are split between overloading load() (my preference) and adding a new, explicit method: loadf() or something like that. Either way, it's useful. I honestly don't get the objection. MY takle is that mosto f the core devs are doing "systsms programming" rather than "scripting" -- which means that they a: Want to be more explicit and robust Need more flexibiilty around various file-like objects Don't mind a bit more expertise required. (e.g. specifying the encoding correctly) Don't mind a couple extra lines of code. So why add a function to make it simple and easy to read a file fro a path-like? I really wish the core devs would remember how useful Python is as a scripting language, and how many people use it that way. Pandas is a good example, I lot of folks use it in a scripting context, so it provided features that make it easy to do so. -CHB On Sun, Sep 5, 2021, 1:19 PM Christopher Barker <pythonchb@gmail.com> wrote:
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On 6/09/21 3:07 am, C. Titus Brown via Python-ideas wrote:
You can do this now: from contextlib import closing with closing(csv.DictReader.open(filename)) as r: ... IMO this is preferable than going around adding context manager methods to everything that has open-like functionality. -- Greg
On Mon, 6 Sept 2021 at 01:13, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
What version of Python are you using?
IMO this is preferable than going around adding context manager methods to everything that has open-like functionality.
I disagree. It would be better if resource acquisition (e.g. opening a file) always took place in an __enter__ method so that it could always be under control of a with statement. Having closing as a separate function negates that because there has to be a separate function that acquires the resource before the closing function is called and hence before __enter__ is called. -- Oscar
Thanks for your comments, everyone! Right, I’m struggling to figure out Greg's example :). Maybe Greg missed that DictReader.open didn’t exist and was in fact what I was asking for! (contextlib.closing is great, thank you for that!) Anyway, just to re-up the original points and add a few - * opening a CSV file with the right newline setting and then applying csv.reader and csv.DictReader is super common. * newline=“” has important ramifications for Windows functionality, as we <ahem> recently discovered when we tried to extend sourmash with windows compat. * yes it’s very easy to write my own utility function to do this, and in fact I have done so …repeatedly. :) * no one is proposing to remove any functionality. * I like Chris Barker’s comment, “”” it ended with the idea that maybe there should be a PEP for a common interface for all “file” readers — eg JSON, CSV, etc.. And that interface could be supported by third party libs. That interface *maybe* would include a single step load from a path-like functionality. “”” I’m +0 on David Mertz’s suggestion, “”" Most Pandas read methods take either a path-like argument or a file-like argument, and figure out which it is by introspection when called. Actually, most of them even accept a URL-like argument as well I don't think this is a terrible approach. It doesn't make things quite as explicit as the standard library generally does. But it's convenient, and there's no real ambiguity. “”” mostly because when we’ve done this in our own packages, I've struggled to figure out the best method to figure out if something is file-like. For example, it looks like ‘csv.DictReader’ will take any iterable, which means passing in a string is problematic; perhaps we could be looking for read or readline instead? Codifying that in some standard way could be nice. best, —titus
On 7/09/21 5:46 am, C. Titus Brown via Python-ideas wrote:
Maybe Greg missed that DictReader.open didn’t exist and was in fact what I was asking for!
Sorry about that, I thought you were asking for context manager methods to be added to an existing file-like object. If csv.DictReader had a close() method you could use closing() on it, but it seems it doesn't, so closing() doesn't really help here. -- Greg
I think the point here is not the context manager, but rather, having the reader open the file itself, rather than taking an already open file-like object. And if it’s going to do that, it should provide. Context manager. Personally, while we are at it, I’d like to see a “read all” method, analogous to file.readlines(). For scripting use cases, reading a standard file format should be a one liner. -CHB On Sun, Sep 5, 2021 at 5:31 PM Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On 2021-09-06 10:50, Christopher Barker wrote:
I agree, and I think having such capability is very useful. I'm always annoyed by things like json.load that require me to open the file separately. There isn't much genuine ambiguity here: I think it's fine to treat strings and Path instances as meaning "this is a filename, open it" and if it's not specifically one of those, then treat it as a file-like object and try to read it. Alternatively, as mentioned on this thread, there's no reason that all these functions that require you to open the file yourself couldn't just grow a new method that's explicitly documented to accept a filename instead of an open file. But I don't see how requiring the user to open the file first on their own gains anything. In my experience 90% of the time that's just more cumbersome and I would prefer the library to handle the entire file operation internally (like pandas does). -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
This would only be helpful when the CSV is on the disk but csv.reader() takes a file object so that it can used with anything like a socket for example. json.load() does the same thing. It seems to me that it is better to keep a generic interface that are composable. Cheers, Rémi
There has been discussion about adding loading from a “path like” to the JSON lib. See this list about ten months ago and a recent thread on discuss. There resistance, but it ended with the idea that maybe there should be a PEP for a common interface for all “file” readers — eg JSON, CSV, etc.. And that interface could be supported by third party libs. That interface *maybe* would include a single step load from a path-like functionality. It seems to me that it is better to keep a generic interface that are
composable.
No one is suggesting removing the load-from-a-file-like interface. I have no idea why the fact that some people sometimes need a more flexible interface (maybe most people, even) somehow means that we shouldn’t make things easy and obvious for a common use case. -CHB
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Most Pandas read methods take either a path-like argument or a file-like argument, and figure out which it is by introspection when called. Actually, most of them even accept a URL-like argument as well I don't think this is a terrible approach. It doesn't make things quite as explicit as the standard library generally does. But it's convenient, and there's no real ambiguity. On Sun, Sep 5, 2021, 1:19 PM Christopher Barker <pythonchb@gmail.com> wrote:
On Sun, Sep 5, 2021 at 10:32 AM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:
The folks in favor of adding this to json are split between overloading load() (my preference) and adding a new, explicit method: loadf() or something like that. Either way, it's useful. I honestly don't get the objection. MY takle is that mosto f the core devs are doing "systsms programming" rather than "scripting" -- which means that they a: Want to be more explicit and robust Need more flexibiilty around various file-like objects Don't mind a bit more expertise required. (e.g. specifying the encoding correctly) Don't mind a couple extra lines of code. So why add a function to make it simple and easy to read a file fro a path-like? I really wish the core devs would remember how useful Python is as a scripting language, and how many people use it that way. Pandas is a good example, I lot of folks use it in a scripting context, so it provided features that make it easy to do so. -CHB On Sun, Sep 5, 2021, 1:19 PM Christopher Barker <pythonchb@gmail.com> wrote:
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On 6/09/21 3:07 am, C. Titus Brown via Python-ideas wrote:
You can do this now: from contextlib import closing with closing(csv.DictReader.open(filename)) as r: ... IMO this is preferable than going around adding context manager methods to everything that has open-like functionality. -- Greg
On Mon, 6 Sept 2021 at 01:13, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
What version of Python are you using?
IMO this is preferable than going around adding context manager methods to everything that has open-like functionality.
I disagree. It would be better if resource acquisition (e.g. opening a file) always took place in an __enter__ method so that it could always be under control of a with statement. Having closing as a separate function negates that because there has to be a separate function that acquires the resource before the closing function is called and hence before __enter__ is called. -- Oscar
Thanks for your comments, everyone! Right, I’m struggling to figure out Greg's example :). Maybe Greg missed that DictReader.open didn’t exist and was in fact what I was asking for! (contextlib.closing is great, thank you for that!) Anyway, just to re-up the original points and add a few - * opening a CSV file with the right newline setting and then applying csv.reader and csv.DictReader is super common. * newline=“” has important ramifications for Windows functionality, as we <ahem> recently discovered when we tried to extend sourmash with windows compat. * yes it’s very easy to write my own utility function to do this, and in fact I have done so …repeatedly. :) * no one is proposing to remove any functionality. * I like Chris Barker’s comment, “”” it ended with the idea that maybe there should be a PEP for a common interface for all “file” readers — eg JSON, CSV, etc.. And that interface could be supported by third party libs. That interface *maybe* would include a single step load from a path-like functionality. “”” I’m +0 on David Mertz’s suggestion, “”" Most Pandas read methods take either a path-like argument or a file-like argument, and figure out which it is by introspection when called. Actually, most of them even accept a URL-like argument as well I don't think this is a terrible approach. It doesn't make things quite as explicit as the standard library generally does. But it's convenient, and there's no real ambiguity. “”” mostly because when we’ve done this in our own packages, I've struggled to figure out the best method to figure out if something is file-like. For example, it looks like ‘csv.DictReader’ will take any iterable, which means passing in a string is problematic; perhaps we could be looking for read or readline instead? Codifying that in some standard way could be nice. best, —titus
On 7/09/21 5:46 am, C. Titus Brown via Python-ideas wrote:
Maybe Greg missed that DictReader.open didn’t exist and was in fact what I was asking for!
Sorry about that, I thought you were asking for context manager methods to be added to an existing file-like object. If csv.DictReader had a close() method you could use closing() on it, but it seems it doesn't, so closing() doesn't really help here. -- Greg
I think the point here is not the context manager, but rather, having the reader open the file itself, rather than taking an already open file-like object. And if it’s going to do that, it should provide. Context manager. Personally, while we are at it, I’d like to see a “read all” method, analogous to file.readlines(). For scripting use cases, reading a standard file format should be a one liner. -CHB On Sun, Sep 5, 2021 at 5:31 PM Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On 2021-09-06 10:50, Christopher Barker wrote:
I agree, and I think having such capability is very useful. I'm always annoyed by things like json.load that require me to open the file separately. There isn't much genuine ambiguity here: I think it's fine to treat strings and Path instances as meaning "this is a filename, open it" and if it's not specifically one of those, then treat it as a file-like object and try to read it. Alternatively, as mentioned on this thread, there's no reason that all these functions that require you to open the file yourself couldn't just grow a new method that's explicitly documented to accept a filename instead of an open file. But I don't see how requiring the user to open the file first on their own gains anything. In my experience 90% of the time that's just more cumbersome and I would prefer the library to handle the entire file operation internally (like pandas does). -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
participants (8)
-
Brendan Barnwell -
C. Titus Brown -
Christopher Barker -
David Mertz, Ph.D. -
Greg Ewing -
Oscar Benjamin -
Rémi Lapeyre -
Thomas Grainger