<div dir="ltr">You're trying to read a file with a name of literally `${d}.log`, which is unlikely to be the name of your file. `${}` is bash syntax, not python syntax.<div><br></div><div>This has drifted out of numpy territory and into "how to coordinate between bash and python" territory - I'd perhaps recommend you ask this to a wider python audience on StackOverflow, where you'll get a faster response.</div><div><br></div><div>Eric</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 10 Oct 2019 at 15:11, Stephen P. Molnar <<a href="mailto:s.molnar@sbcglobal.net">s.molnar@sbcglobal.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I am slowly and not quickly stumbling forward, but at this point my <br>
degree of mental entropy (confusion) is monumental.<br>
<br>
This works:<br>
<br>
> import numpy as np<br>
><br>
> print('${d}')<br>
><br>
> data = np.genfromtxt("14-7.log", usecols=(1), skip_header=27, <br>
> skip_footer=1, encoding=None)<br>
><br>
> print(data)<br>
><br>
> np.savetxt('14-7.dG', data, fmt='%12.9f', header='14-7')<br>
> print(data)<br>
<br>
which produces:<br>
<br>
> runfile('/home/comp/Apps/Python/PsoVina/DeltaGTable_V_s.py', <br>
> wdir='/home/comp/Apps/Python/PsoVina', current_namespace=True)<br>
> ${d}<br>
> [-9.96090267 -8.97950478 -8.94261136 -8.91552301 -8.73650883 -8.66338714<br>
> -8.41073971 -8.38914635 -8.29679891 -8.16845411 -8.12799082 -8.12710377<br>
> -7.97909074 -7.94187268 -7.90076621 -7.88148523 -7.83782648 -7.8159095<br>
> -7.72254029 -7.72034674]<br>
> [-9.96090267 -8.97950478 -8.94261136 -8.91552301 -8.73650883 -8.66338714<br>
> -8.41073971 -8.38914635 -8.29679891 -8.16845411 -8.12799082 -8.12710377<br>
> -7.97909074 -7.94187268 -7.90076621 -7.88148523 -7.83782648 -7.8159095<br>
> -7.72254029 -7.72034674]<br>
Note; the print statements are for a quick check o the output, which is:<br>
<br>
> # 14-7<br>
> -9.960902669<br>
> -8.979504781<br>
> -8.942611364<br>
> -8.915523010<br>
> -8.736508831<br>
> -8.663387139<br>
> -8.410739711<br>
> -8.389146347<br>
> -8.296798909<br>
> -8.168454106<br>
> -8.127990818<br>
> -8.127103774<br>
> -7.979090739<br>
> -7.941872682<br>
> -7.900766215<br>
> -7.881485228<br>
> -7.837826485<br>
> -7.815909505<br>
> -7.722540286<br>
> -7.720346742<br>
Also, this bash script works:<br>
<br>
> #!/bin/bash<br>
><br>
> # Run.dG.list_1<br>
><br>
> while IFS= read -r d<br>
> do<br>
> echo "${d}.log"<br>
><br>
> done <ligand.list<br>
which returns the three log file names:<br>
<br>
> 14-7.log<br>
> 15-7.log<br>
> 18-7.log<br>
> C-VX3.log<br>
<br>
<br>
But, if I run this bash script:<br>
<br>
> #!/bin/bash<br>
><br>
> # Run.dG.list_1<br>
><br>
> while IFS= read -r d<br>
> do<br>
> echo "${d}.log"<br>
> python3 DeltaGTable_V_sl.py<br>
><br>
><br>
> done <ligand.list<br>
><br>
where DeltaGTable_V_sl.py is:<br>
<br>
> import numpy as np<br>
><br>
> print('${d}')<br>
><br>
> data = np.genfromtxt('${d}.log', usecols=(1), skip_header=27, <br>
> skip_footer=1, encoding=None)<br>
> print(data)<br>
><br>
> np.savetxt('${d}.dG', data, fmt='%12.9f', header='${d}')<br>
> print(data.dG)<br>
<br>
I get:<br>
<br>
> (base) comp@AbNormal:~/Apps/Python/PsoVina$ sh ./<a href="http://Run.dG.list_1.sh" rel="noreferrer" target="_blank">Run.dG.list_1.sh</a><br>
> 14-7.log<br>
> python3: can't open file 'DeltaGTable_V_sl.py': [Errno 2] No such file <br>
> or directory<br>
> 15-7.log<br>
> python3: can't open file 'DeltaGTable_V_sl.py': [Errno 2] No such file <br>
> or directory<br>
> 18-7.log<br>
> python3: can't open file 'DeltaGTable_V_sl.py': [Errno 2] No such file <br>
> or directory<br>
> C-VX3.log<br>
> python3: can't open file 'DeltaGTable_V_sl.py': [Errno 2] No such file <br>
> or directory<br>
<br>
So, it would appear that the log file labels are in the workspace, but <br>
'${d}.log' is not being recognized as fname by genfromtxt. Although i <br>
have googled every combination of terms I can think of I am obviously <br>
missing something.<br>
<br>
As I have potentially hundreds of files to process, I would appreciate <br>
pointers towards a solution to the problem.<br>
<br>
Thanks in advance.<br>
<br>
On 10/08/2019 10:49 AM, Stephen P. Molnar wrote:<br>
> Many thanks or your kind replies.<br>
><br>
> I really appreciate your suggestions.<br>
><br>
> On 10/08/2019 09:44 AM, Andras Deak wrote:<br>
>> PS. if you just want to specify the width of the fields you wouldn't<br>
>> have to convert anything, because you can specify the size and<br>
>> justification of a %s format. But arguably having float data as floats<br>
>> is more natural anyway.<br>
>><br>
>> On Tue, Oct 8, 2019 at 3:42 PM Andras Deak <<a href="mailto:deak.andris@gmail.com" target="_blank">deak.andris@gmail.com</a>> <br>
>> wrote:<br>
>>> On Tue, Oct 8, 2019 at 3:17 PM Stephen P. Molnar <br>
>>> <<a href="mailto:s.molnar@sbcglobal.net" target="_blank">s.molnar@sbcglobal.net</a>> wrote:<br>
>>>> I am embarrassed to be asking this question, but I have exhausted <br>
>>>> Google<br>
>>>> at this point .<br>
>>>><br>
>>>> I have a number of identically formatted text files from which I <br>
>>>> want to<br>
>>>> extract data, as an example (hopefully, putting these in as quotes <br>
>>>> will<br>
>>>> persevere the format):<br>
>>>><br>
>>>>> ======================================================================= <br>
>>>>><br>
>>>>> PSOVina version 2.0<br>
>>>>> Giotto H. K. Tai & Shirley W. I. Siu<br>
>>>>><br>
>>>>> Computational Biology and Bioinformatics Lab<br>
>>>>> University of Macau<br>
>>>>><br>
>>>>> Visit <a href="http://cbbio.cis.umac.mo" rel="noreferrer" target="_blank">http://cbbio.cis.umac.mo</a> for more information.<br>
>>>>><br>
>>>>> PSOVina was developed based on the framework of AutoDock Vina.<br>
>>>>><br>
>>>>> For more information about Vina, please visit <br>
>>>>> <a href="http://vina.scripps.edu" rel="noreferrer" target="_blank">http://vina.scripps.edu</a>.<br>
>>>>><br>
>>>>> ======================================================================= <br>
>>>>><br>
>>>>><br>
>>>>> Output will be 13-7_out.pdbqt<br>
>>>>> Reading input ... done.<br>
>>>>> Setting up the scoring function ... done.<br>
>>>>> Analyzing the binding site ... done.<br>
>>>>> Using random seed: 1828390527<br>
>>>>> Performing search ... done.<br>
>>>>><br>
>>>>> Refining results ... done.<br>
>>>>><br>
>>>>> mode | affinity | dist from best mode<br>
>>>>> | (kcal/mol) | rmsd l.b.| rmsd u.b.<br>
>>>>> -----+------------+----------+----------<br>
>>>>> 1 -8.862004149 0.000 0.000<br>
>>>>> 2 -8.403522829 2.992 6.553<br>
>>>>> 3 -8.401384636 2.707 5.220<br>
>>>>> 4 -7.886402037 4.907 6.862<br>
>>>>> 5 -7.845519031 3.233 5.915<br>
>>>>> 6 -7.837434227 3.954 5.641<br>
>>>>> 7 -7.834584887 3.188 7.294<br>
>>>>> 8 -7.694395765 3.746 7.553<br>
>>>>> 9 -7.691211177 3.536 5.745<br>
>>>>> 10 -7.670759445 3.698 7.587<br>
>>>>> 11 -7.661882758 4.882 7.044<br>
>>>>> 12 -7.636280303 2.347 3.284<br>
>>>>> 13 -7.635788052 3.511 6.250<br>
>>>>> 14 -7.611175249 2.427 3.449<br>
>>>>> 15 -7.586368357 2.142 2.864<br>
>>>>> 16 -7.531307666 2.976 4.980<br>
>>>>> 17 -7.520501084 3.085 5.775<br>
>>>>> 18 -7.512906514 4.220 7.672<br>
>>>>> 19 -7.307403528 3.240 4.354<br>
>>>>> 20 -7.256063348 3.694 7.252<br>
>>>>> Writing output ... done.<br>
>>>> At this point, my python script consists of only the following:<br>
>>>><br>
>>>>> #!/usr/bin/env python3<br>
>>>>> # -*- coding: utf-8 -*-<br>
>>>>> """<br>
>>>>><br>
>>>>> Created on Tue Sep 24 07:51:11 2019<br>
>>>>><br>
>>>>> """<br>
>>>>> import numpy as np<br>
>>>>><br>
>>>>> data = []<br>
>>>>><br>
>>>>> data = np.genfromtxt("13-7.log", usecols=(1), dtype=None,<br>
>>>>> skip_header=27, skip_footer=1, encoding=None)<br>
>>>>><br>
>>>>> print(data)<br>
>>>>><br>
>>>>> np.savetxt('13-7', [data], fmt='%15.9f', header='13-7')<br>
>>>> The problem lies in tfe np.savetxt line, on execution I get:<br>
>>>><br>
>>>>> runfile('/home/comp/Apps/Models/1-PhosphorusLigands/CombinedLigands/MOL/Docking/VINA20/SmallSet/DeltaGTable_V_s.py', <br>
>>>>><br>
>>>>> wdir='/home/comp/Apps/Models/1-PhosphorusLigands/CombinedLigands/MOL/Docking/VINA20/SmallSet', <br>
>>>>><br>
>>>>> current_namespace=True)<br>
>>>>> ['-8.839713733' '-8.743377250' '-8.151051167' '-8.090452911'<br>
>>>>> '-7.967494477' '-7.854890056' '-7.757417879' '-7.741557490'<br>
>>>>> '-7.643885488' '-7.611595767' '-7.507605524' '-7.413920814'<br>
>>>>> '-7.389408331' '-7.384446364' '-7.374206276' '-7.368808179'<br>
>>>>> '-7.346641418' '-7.325037898' '-7.309614787' '-7.113209147']<br>
>>>>> Traceback (most recent call last):<br>
>>>>><br>
>>>>> File<br>
>>>>> "/home/comp/Apps/Models/1-PhosphorusLigands/CombinedLigands/MOL/Docking/VINA20/SmallSet/DeltaGTable_V_s.py", <br>
>>>>><br>
>>>>> line 16, in <module><br>
>>>>> np.savetxt('13-7', [data], fmt='%16.9f', header='13-7')<br>
>>>>><br>
>>>>> File "<__array_function__ internals>", line 6, in savetxt<br>
>>>>><br>
>>>>> File<br>
>>>>> "/home/comp/Apps/Miniconda3/lib/python3.7/site-packages/numpy/lib/npyio.py", <br>
>>>>><br>
>>>>> line 1438, in savetxt<br>
>>>>> % (str(X.dtype), format))<br>
>>>>><br>
>>>>> TypeError: Mismatch between array dtype ('<U12') and format specifier<br>
>>>>> ('%16.9f %16.9f %16.9f %16.9f %16.9f %16.9f %16.9f %16.9f %16.9f<br>
>>>>> %16.9f %16.9f %16.9f %16.9f %16.9f %16.9f %16.9f %16.9f %16.9f %16.9f<br>
>>>>> %16.9f')<br>
>>>> The data is in the data file, but the only entry in '13-7', the saved<br>
>>>> file, is the label. Obviously, the error is in the format argument.<br>
>>> Hi,<br>
>>><br>
>>> One problem is the format: the error is telling you that you have<br>
>>> strings in your array (compare the `'<U12'` dtype and the output of<br>
>>> your `print(data)` call with strings inside), whereas %16.9f can only<br>
>>> be used to format floats (f for float). You would first have to<br>
>>> convert your array of strings to an array numbers. I don't usually use<br>
>>> genfromtxt so I'm not sure how you can make it return floats for you<br>
>>> in the first place, but I suspect `dtype=None` in the call to<br>
>>> genfromtxt might be responsible. In any case making it return numbers<br>
>>> should be the easier case.<br>
>>> The second problem is that you should make sure you mean `[data]` in<br>
>>> the call to savetxt. As it is now this would give you a 2d array of<br>
>>> shape (1, 20), and the output would correspondingly contain a single<br>
>>> row of 20 values (hence the 20 instances of '%16.9f' in the error<br>
>>> message). In case you meant to print one value per row in a single<br>
>>> column, you should drop the brackets around `data`:<br>
>>> np.savetxt('13-7', data, fmt='%16.9f', header='13-7')<br>
>>><br>
>>> And just a personal note, but I'd find an output file named '13-7' to<br>
>>> be a bit surprising. Perhaps some extension or prefix would help<br>
>>> organize these files?<br>
>>> Regards,<br>
>>><br>
>>> Andr??s<br>
>>><br>
>>>> Help will be much appreciated.<br>
>>>><br>
>>>> Thanks in advance.<br>
>>>><br>
>>>> -- <br>
>>>> Stephen P. Molnar, Ph.D.<br>
>>>> <a href="http://www.molecular-modeling.net" rel="noreferrer" target="_blank">www.molecular-modeling.net</a><br>
>>>> 614.312.7528 (c)<br>
>>>> Skype: smolnar1<br>
>>>><br>
>>>> _______________________________________________<br>
>>>> NumPy-Discussion mailing list<br>
>>>> <a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a><br>
>>>> <a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a><br>
>> _______________________________________________<br>
>> NumPy-Discussion mailing list<br>
>> <a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a><br>
>> <a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a><br>
><br>
<br>
-- <br>
Stephen P. Molnar, Ph.D.<br>
<a href="http://www.molecular-modeling.net" rel="noreferrer" target="_blank">www.molecular-modeling.net</a><br>
614.312.7528 (c)<br>
Skype: smolnar1<br>
<br>
_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@python.org" target="_blank">NumPy-Discussion@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/numpy-discussion" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/numpy-discussion</a><br>
</blockquote></div>