Problem with np.savetxt
I am embarrassed to be asking this question, but I have exhausted Google at this point . I have a number of identically formatted text files from which I want to extract data, as an example (hopefully, putting these in as quotes will persevere the format):
The problem lies in tfe np.savetxt line, on execution I get:
The data is in the data file, but the only entry in '13-7', the saved file, is the label. Obviously, the error is in the format argument. Help will be much appreciated. Thanks in advance. -- Stephen P. Molnar, Ph.D. www.molecular-modeling.net 614.312.7528 (c) Skype: smolnar1
On Tue, Oct 8, 2019 at 3:17 PM Stephen P. Molnar <s.molnar@sbcglobal.net> wrote:
Hi, One problem is the format: the error is telling you that you have strings in your array (compare the `'<U12'` dtype and the output of your `print(data)` call with strings inside), whereas %16.9f can only be used to format floats (f for float). You would first have to convert your array of strings to an array numbers. I don't usually use genfromtxt so I'm not sure how you can make it return floats for you in the first place, but I suspect `dtype=None` in the call to genfromtxt might be responsible. In any case making it return numbers should be the easier case. The second problem is that you should make sure you mean `[data]` in the call to savetxt. As it is now this would give you a 2d array of shape (1, 20), and the output would correspondingly contain a single row of 20 values (hence the 20 instances of '%16.9f' in the error message). In case you meant to print one value per row in a single column, you should drop the brackets around `data`: np.savetxt('13-7', data, fmt='%16.9f', header='13-7') And just a personal note, but I'd find an output file named '13-7' to be a bit surprising. Perhaps some extension or prefix would help organize these files? Regards, András
PS. if you just want to specify the width of the fields you wouldn't have to convert anything, because you can specify the size and justification of a %s format. But arguably having float data as floats is more natural anyway. On Tue, Oct 8, 2019 at 3:42 PM Andras Deak <deak.andris@gmail.com> wrote:
I am slowly and not quickly stumbling forward, but at this point my degree of mental entropy (confusion) is monumental. This works:
which produces:
But, if I run this bash script:
where DeltaGTable_V_sl.py is:
I get:
So, it would appear that the log file labels are in the workspace, but '${d}.log' is not being recognized as fname by genfromtxt. Although i have googled every combination of terms I can think of I am obviously missing something. As I have potentially hundreds of files to process, I would appreciate pointers towards a solution to the problem. Thanks in advance. On 10/08/2019 10:49 AM, Stephen P. Molnar wrote:
-- Stephen P. Molnar, Ph.D. www.molecular-modeling.net 614.312.7528 (c) Skype: smolnar1
You're trying to read a file with a name of literally `${d}.log`, which is unlikely to be the name of your file. `${}` is bash syntax, not python syntax. This has drifted out of numpy territory and into "how to coordinate between bash and python" territory - I'd perhaps recommend you ask this to a wider python audience on StackOverflow, where you'll get a faster response. Eric On Thu, 10 Oct 2019 at 15:11, Stephen P. Molnar <s.molnar@sbcglobal.net> wrote:
-------- Forwarded Message -------- Subject: Re: [Numpy-discussion] Problem with np.savetxt Date: Thu, 10 Oct 2019 10:10:58 -0400 From: Stephen P. Molnar <s.molnar@sbcglobal.net> To: numpy-discussion@python.org I am slowly and not quickly stumbling forward, but at this point my degree of mental entropy (confusion) is monumental. This works:
which produces:
But, if I run this bash script:
where DeltaGTable_V_sl.py is:
I get:
So, it would appear that the log file labels are in the workspace, but '${d}.log' is not being recognized as fname by genfromtxt. Although i have googled every combination of terms I can think of I am obviously missing something. As I have potentially hundreds of files to process, I would appreciate pointers towards a solution to the problem. Thanks in advance. On 10/08/2019 10:49 AM, Stephen P. Molnar wrote:
-- Stephen P. Molnar, Ph.D. www.molecular-modeling.net 614.312.7528 (c) Skype: smolnar1
Le mardi 08 octobre 2019, Stephen P. Molnar a écrit :
Hi, Note that your data array is made of strings and not floats. The default value of the dtype argument is float, which you override by None. Remove the 'dtype=None' part to correctly load data You then have no problem to save your data with the format you want. Fabrice PS : be aware that [data] is a 2D row array, that will end up inlined with command np.savetxt('13-7', [data], fmt='%15.9f', header='13-7') Remove the bracket for a one-per-line formatted output np.savetxt('13-7', data, fmt='%15.9f', header='13-7')
Thanks for the replies. All is now well! I'm thankful that this list is so very patient with ROF's (retired old fools) struggling to learn a new programmng language. On 10/08/2019 10:42 AM, Fabrice Silva wrote:
-- Stephen P. Molnar, Ph.D. www.molecular-modeling.net 614.312.7528 (c) Skype: smolnar1
On Tue, Oct 8, 2019 at 3:17 PM Stephen P. Molnar <s.molnar@sbcglobal.net> wrote:
Hi, One problem is the format: the error is telling you that you have strings in your array (compare the `'<U12'` dtype and the output of your `print(data)` call with strings inside), whereas %16.9f can only be used to format floats (f for float). You would first have to convert your array of strings to an array numbers. I don't usually use genfromtxt so I'm not sure how you can make it return floats for you in the first place, but I suspect `dtype=None` in the call to genfromtxt might be responsible. In any case making it return numbers should be the easier case. The second problem is that you should make sure you mean `[data]` in the call to savetxt. As it is now this would give you a 2d array of shape (1, 20), and the output would correspondingly contain a single row of 20 values (hence the 20 instances of '%16.9f' in the error message). In case you meant to print one value per row in a single column, you should drop the brackets around `data`: np.savetxt('13-7', data, fmt='%16.9f', header='13-7') And just a personal note, but I'd find an output file named '13-7' to be a bit surprising. Perhaps some extension or prefix would help organize these files? Regards, András
PS. if you just want to specify the width of the fields you wouldn't have to convert anything, because you can specify the size and justification of a %s format. But arguably having float data as floats is more natural anyway. On Tue, Oct 8, 2019 at 3:42 PM Andras Deak <deak.andris@gmail.com> wrote:
I am slowly and not quickly stumbling forward, but at this point my degree of mental entropy (confusion) is monumental. This works:
which produces:
But, if I run this bash script:
where DeltaGTable_V_sl.py is:
I get:
So, it would appear that the log file labels are in the workspace, but '${d}.log' is not being recognized as fname by genfromtxt. Although i have googled every combination of terms I can think of I am obviously missing something. As I have potentially hundreds of files to process, I would appreciate pointers towards a solution to the problem. Thanks in advance. On 10/08/2019 10:49 AM, Stephen P. Molnar wrote:
-- Stephen P. Molnar, Ph.D. www.molecular-modeling.net 614.312.7528 (c) Skype: smolnar1
You're trying to read a file with a name of literally `${d}.log`, which is unlikely to be the name of your file. `${}` is bash syntax, not python syntax. This has drifted out of numpy territory and into "how to coordinate between bash and python" territory - I'd perhaps recommend you ask this to a wider python audience on StackOverflow, where you'll get a faster response. Eric On Thu, 10 Oct 2019 at 15:11, Stephen P. Molnar <s.molnar@sbcglobal.net> wrote:
-------- Forwarded Message -------- Subject: Re: [Numpy-discussion] Problem with np.savetxt Date: Thu, 10 Oct 2019 10:10:58 -0400 From: Stephen P. Molnar <s.molnar@sbcglobal.net> To: numpy-discussion@python.org I am slowly and not quickly stumbling forward, but at this point my degree of mental entropy (confusion) is monumental. This works:
which produces:
But, if I run this bash script:
where DeltaGTable_V_sl.py is:
I get:
So, it would appear that the log file labels are in the workspace, but '${d}.log' is not being recognized as fname by genfromtxt. Although i have googled every combination of terms I can think of I am obviously missing something. As I have potentially hundreds of files to process, I would appreciate pointers towards a solution to the problem. Thanks in advance. On 10/08/2019 10:49 AM, Stephen P. Molnar wrote:
-- Stephen P. Molnar, Ph.D. www.molecular-modeling.net 614.312.7528 (c) Skype: smolnar1
Le mardi 08 octobre 2019, Stephen P. Molnar a écrit :
Hi, Note that your data array is made of strings and not floats. The default value of the dtype argument is float, which you override by None. Remove the 'dtype=None' part to correctly load data You then have no problem to save your data with the format you want. Fabrice PS : be aware that [data] is a 2D row array, that will end up inlined with command np.savetxt('13-7', [data], fmt='%15.9f', header='13-7') Remove the bracket for a one-per-line formatted output np.savetxt('13-7', data, fmt='%15.9f', header='13-7')
Thanks for the replies. All is now well! I'm thankful that this list is so very patient with ROF's (retired old fools) struggling to learn a new programmng language. On 10/08/2019 10:42 AM, Fabrice Silva wrote:
-- Stephen P. Molnar, Ph.D. www.molecular-modeling.net 614.312.7528 (c) Skype: smolnar1
participants (4)
-
Andras Deak -
Eric Wieser -
Fabrice Silva -
Stephen P. Molnar