CSV Dictwriter - Handling escape characters
Hi, I would like if the DictWriter would be able to write escape characters (\n and \r and their combinations such as \n\n\n and \r\n) as literal characters. At the moment, DictWriter writes \n:s as new lines (shown in notepad or Excel) and I would like it to just write "\n" characters in my csv file. I asked a question about this on Stack Overflow but have had no good answers yet. My question also has an example of the data and its form. https://stackoverflow.com/questions/60073809/python-how-to-write-literal-n-a...
On 12/02/2020 08:15, allu.matikainen@hotmail.com wrote:
Hi, I would like if the DictWriter would be able to write escape characters (\n and \r and their combinations such as \n\n\n and \r\n) as literal characters. At the moment, DictWriter writes \n:s as new lines (shown in notepad or Excel) and I would like it to just write "\n" characters in my csv file.
I asked a question about this on Stack Overflow but have had no good answers yet. My question also has an example of the data and its form. https://stackoverflow.com/questions/60073809/python-how-to-write-literal-n-a...
I think my advice would be a variation on "Don't do that!" If you can massage your data into the right format before you try writing it, that would be preferable to having DictWriter mangle your strings in what is essentially an arbitrary way. -- Rhodri James *-* Kynesim Ltd
On Feb 12, 2020, at 06:11, allu.matikainen@hotmail.com wrote:
I would like if the DictWriter would be able to write escape characters (\n and \r and their combinations such as \n\n\n and \r\n) as literal characters.
I don’t think you’re asking to write escape sequences, but how to write actual newline and carriage return characters by escaping them. Thats what the strings in your example post text have, at least. In a string literal, "a\nb" is the three characters a, newline, b; the string literal "a\\nb" is the four character a, backslash, n, b In your comments, you seem to have a lot of confusion about the difference between Python string literals, JSON string encodings, and the underlying strings, so it’s hard to be sure, but I’m about 90% sure that your actual strings have newlines, not backslash-escaped newlines. But regardless of what you actually have, there’s no reason the csv module should be changed to help you with this. It’s meant to write data in the same language used by Excel (or some other known CSV dialect). There is no backslash escaping of control characters in Excel—as is obvious from the fact that Excel shows a \n as literally a backslash and an n rather than a newline within the cell. If there’s some other CSV dialect that does use backslash escaping that you want to support, that would be different—but you’re trying to use Excel, so that can’t be the issue. The csv module doesn’t help you do web percent encoding or rot13 encryption or reversing every other string because those are meaningless to Excel and other CSV dialects, and the same is true here. But that’s fine. You can already arbitrarily transform strings in Python, before passing them to the csv module. For example, instead of `c.writerow(row)` you can do `c.writerow(map(transformer, row))`, and you’re done. That transformer could be an existing escaping function that means exactly what you want, or you could write it yourself as a one-liner (`return value.replace('\r', '\\r').replace('\n', '\\n')`). Also, even if you disagree and think the csv module does need to change, you need to explain what that change is. Would you add a new Dialect attribute? What would it be? Would it share the same escapechar used for escaping quotes, or have a different attribute, or be hardcoded to backslash? And so on.
I asked a question about this on Stack Overflow but have had no good answers yet.
You got comments telling you to use str.replace to modify the strings as you read them or as you write them. Which is the right answer. Your reply was that you don’t have permission to modify the text. So you’re asking for is a way to modify the text without modifying the text, which is obviously impossible. If that’s true, then even if Python 3.9 added your requested feature and you waited until it came out before continuing your project, you still wouldn’t be able to use it, because if you don’t have permission to replace newlines with \n escapes then you don’t have permission to ask the csv module to do it either. So no wonder you haven’t gotten any good answers. If you asked how to encode some text in UTF-16 and then said you don’t have permission to encode the text, you’d get the same result—either no answers, or bad answers from people who don’t know anything but are desperate for points so they just guess wildly at something that might be kind of similar to what you want.
Hello Andrew, thank you for your answer. "Your reply was that you don’t have permission to modify the text." by this I meant that the data is transferred to me as it is and I can't modify the text in the source (because someone else is the database admin) but it is possible for me to modify the text that has been transferred to me. I'm sorry, this was my mistake in writing my question in an unclear way. And in the end, I ended up transforming the strings, using str.replace as you and the comments in Stack Overflow suggested. It worked great! It is also a good point that I should explain in detail why a module should be changed, I will definitely think about that later first if I run into another problem. I'm still a bit surprised at the negative tone in your reply. I can tell you I am just a junior developer who started coding full time last summer. The best way to keep up the positive attitude of a junior is to answer their questions in a positive and constructive way. From your answer I got the impression that just because I wrote my question in an unclear way, you think I'm stupid and asking stupid questions (there are no stupid questions in the world and everyone has their own pace of learning new things). Even though I had written my question unclearly, I would have liked a more positive and constructive reply, because I am willing to learn new things and learn from my mistakes, and it is not right for anyone to get verbally "punished" for their (human!) mistakes. We could also have continued our conversation in a constructive way and learn that I just wrote an incorrect question! (example: you could also have asked "I did not understand, did you mean this by your question?" --> I say "No I'm sorry, I wrote my question in an unclear way, I actually meant...") Moreover, how could I know which is the best and right answer if I don't ask first? It is not for granted that a junior knows exactly what he or she should do / what is the most efficient way / what are the capabilities and restrictions of the csv module, and that is why we are asking questions. Yes, I know it is frustrating to answer "stupid" questions because seniors have been dealing with the issues several more years than juniors and these kind of things are for granted for seniors. But juniors (and all humans) learn slowly, they need time to repeat their successes and most importantly, mistakes, to learn new skills. Making mistakes is not stupid, it is human, and it is needed for learning new things. (N.B. I have also seen this kind of negative reply culture before in the whole Stack Overflow community, which makes me a bit sad. Is it because many programmers are engineers? I have noticed this kind of behavior in the engineering student / tech community too: always looking for answers for every problem, always thinking their own solution is the best one, no matter what the situation, often forgetting the manners or that we all are humans, after all, and some are just learning new things out of curiosity. I'm an engineer/former tech student myself.) I hope you will think about what I wrote in the third paragraph. You will have much more success if you shape your answers in a positive and constructive way and leave negative things behind. I have observed this in every learning situation I have been in and the best results have been in the positive communities where I get positive and constructive feedback. Kind regards - Aleksi ________________________________ From: Andrew Barnert <abarnert@yahoo.com> Sent: Wednesday, February 12, 2020 7:41 PM To: allu.matikainen@hotmail.com <allu.matikainen@hotmail.com> Cc: python-ideas@python.org <python-ideas@python.org> Subject: Re: [Python-ideas] CSV Dictwriter - Handling escape characters On Feb 12, 2020, at 06:11, allu.matikainen@hotmail.com wrote:
I would like if the DictWriter would be able to write escape characters (\n and \r and their combinations such as \n\n\n and \r\n) as literal characters.
I don’t think you’re asking to write escape sequences, but how to write actual newline and carriage return characters by escaping them. Thats what the strings in your example post text have, at least. In a string literal, "a\nb" is the three characters a, newline, b; the string literal "a\\nb" is the four character a, backslash, n, b In your comments, you seem to have a lot of confusion about the difference between Python string literals, JSON string encodings, and the underlying strings, so it’s hard to be sure, but I’m about 90% sure that your actual strings have newlines, not backslash-escaped newlines. But regardless of what you actually have, there’s no reason the csv module should be changed to help you with this. It’s meant to write data in the same language used by Excel (or some other known CSV dialect). There is no backslash escaping of control characters in Excel—as is obvious from the fact that Excel shows a \n as literally a backslash and an n rather than a newline within the cell. If there’s some other CSV dialect that does use backslash escaping that you want to support, that would be different—but you’re trying to use Excel, so that can’t be the issue. The csv module doesn’t help you do web percent encoding or rot13 encryption or reversing every other string because those are meaningless to Excel and other CSV dialects, and the same is true here. But that’s fine. You can already arbitrarily transform strings in Python, before passing them to the csv module. For example, instead of `c.writerow(row)` you can do `c.writerow(map(transformer, row))`, and you’re done. That transformer could be an existing escaping function that means exactly what you want, or you could write it yourself as a one-liner (`return value.replace('\r', '\\r').replace('\n<file://\\r').replace('\n>', '\\n')`). Also, even if you disagree and think the csv module does need to change, you need to explain what that change is. Would you add a new Dialect attribute? What would it be? Would it share the same escapechar used for escaping quotes, or have a different attribute, or be hardcoded to backslash? And so on.
I asked a question about this on Stack Overflow but have had no good answers yet.
You got comments telling you to use str.replace to modify the strings as you read them or as you write them. Which is the right answer. Your reply was that you don’t have permission to modify the text. So you’re asking for is a way to modify the text without modifying the text, which is obviously impossible. If that’s true, then even if Python 3.9 added your requested feature and you waited until it came out before continuing your project, you still wouldn’t be able to use it, because if you don’t have permission to replace newlines with \n escapes then you don’t have permission to ask the csv module to do it either. So no wonder you haven’t gotten any good answers. If you asked how to encode some text in UTF-16 and then said you don’t have permission to encode the text, you’d get the same result—either no answers, or bad answers from people who don’t know anything but are desperate for points so they just guess wildly at something that might be kind of similar to what you want.
On Fri, Mar 6, 2020 at 6:22 PM Aleksi Matikainen <allu.matikainen@hotmail.com> wrote:
And in the end, I ended up transforming the strings, using str.replace as you and the comments in Stack Overflow suggested. It worked great! It is also a good point that I should explain in detail why a module should be changed, I will definitely think about that later first if I run into another problem.
In other words, what Andrew suggested was, in fact, the solution you went with.
I'm still a bit surprised at the negative tone in your reply. [chomp lots of passive-aggressive chuff] (N.B. I have also seen this kind of negative reply culture before in the whole Stack Overflow community, which makes me a bit sad. Is it because many programmers are engineers? I have noticed this kind of behavior in the engineering student / tech community too: always looking for answers for every problem, always thinking their own solution is the best one, no matter what the situation, often forgetting the manners or that we all are humans, after all, and some are just learning new things out of curiosity. I'm an engineer/former tech student myself.)
By "negative tone", do you mean that you were given advice that involved something other than what you specifically asked for help with? You had a problem, and your proposed solution was to *change a programming language* to suit your purposes. Instead, you were given an extremely workable method - which you ended up using - for solving the problem by writing code. Which is exactly what programmers do. http://www.catb.org/esr/faqs/smart-questions.html#goal When you start out by requesting a specific step, rather than your ultimate goal, you have to be prepared for people to offer solutions that achieve the goal by some other means. (Sometimes those solutions won't actually work. But you can always respond saying "yes, that would work, except that X Y and Z".)
I hope you will think about what I wrote in the third paragraph. You will have much more success if you shape your answers in a positive and constructive way and leave negative things behind. I have observed this in every learning situation I have been in and the best results have been in the positive communities where I get positive and constructive feedback.
Define "success". What would Andrew succeed at better? Remember, he isn't being paid to solve your problems in your preferred way. What he offered you was a perfectly valid solution to your problem, and ultimately, he isn't paid to do that either, so "success" isn't really the best term to use even for that. But let's suppose that success is defined as "getting a response from the OP saying that the problem was solved". If that's the case, then this thread is a success for both Andrew and Rhodri, since both posts were making the same recommendation (although Rhodri didn't go into much detail). Perhaps, before you jump to the conclusion that people are being "negative", you should reconsider how you are judging those responses. Focus less on whether someone gave you the precise solution you hoped they'd magic up, and more on whether your actual problem was solved. ChrisA
EVERYONE, please don’t follow up on the meta discussion. What’s said is said. But this type of discussion never leads to anything useful, and often causes more bad blood. (Chris, next time I think it would be better if you sat on your hands. And ditto for everyone else who thinks to respond to an issue about tone.) On Thu, Mar 5, 2020 at 23:44 Chris Angelico <rosuav@gmail.com> wrote:
On Fri, Mar 6, 2020 at 6:22 PM Aleksi Matikainen <allu.matikainen@hotmail.com> wrote:
And in the end, I ended up transforming the strings, using str.replace as you and the comments in Stack Overflow suggested. It worked great! It is also a good point that I should explain in detail why a module should be changed, I will definitely think about that later first if I run into another problem.
In other words, what Andrew suggested was, in fact, the solution you went with.
I'm still a bit surprised at the negative tone in your reply. [chomp lots of passive-aggressive chuff] (N.B. I have also seen this kind of negative reply culture before in the whole Stack Overflow community, which makes me a bit sad. Is it because many programmers are engineers? I have noticed this kind of behavior in the engineering student / tech community too: always looking for answers for every problem, always thinking their own solution is the best one, no matter what the situation, often forgetting the manners or that we all are humans, after all, and some are just learning new things out of curiosity. I'm an engineer/former tech student myself.)
By "negative tone", do you mean that you were given advice that involved something other than what you specifically asked for help with? You had a problem, and your proposed solution was to *change a programming language* to suit your purposes. Instead, you were given an extremely workable method - which you ended up using - for solving the problem by writing code. Which is exactly what programmers do.
http://www.catb.org/esr/faqs/smart-questions.html#goal
When you start out by requesting a specific step, rather than your ultimate goal, you have to be prepared for people to offer solutions that achieve the goal by some other means. (Sometimes those solutions won't actually work. But you can always respond saying "yes, that would work, except that X Y and Z".)
I hope you will think about what I wrote in the third paragraph. You will have much more success if you shape your answers in a positive and constructive way and leave negative things behind. I have observed this in every learning situation I have been in and the best results have been in the positive communities where I get positive and constructive feedback.
Define "success". What would Andrew succeed at better? Remember, he isn't being paid to solve your problems in your preferred way. What he offered you was a perfectly valid solution to your problem, and ultimately, he isn't paid to do that either, so "success" isn't really the best term to use even for that. But let's suppose that success is defined as "getting a response from the OP saying that the problem was solved". If that's the case, then this thread is a success for both Andrew and Rhodri, since both posts were making the same recommendation (although Rhodri didn't go into much detail).
Perhaps, before you jump to the conclusion that people are being "negative", you should reconsider how you are judging those responses. Focus less on whether someone gave you the precise solution you hoped they'd magic up, and more on whether your actual problem was solved.
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/5XJVTI... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile)
participants (6)
-
Aleksi Matikainen -
allu.matikainen＠hotmail.com
-
Andrew Barnert -
Chris Angelico -
Guido van Rossum -
Rhodri James