<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /></head><body style='font-size: 10pt; font-family: Verdana,Geneva,sans-serif'>
<p>Dear All</p>
<div>
<p><br /></p>
<p>First of all thanks for the answers and the information’s (I’ll ding into it) and let me trying to add comments on what I want to :</p>
<ol>
<li>My asci file mainly contains data (float and int) in a single column</li>
<li>(it is not always the case but I can easily manage it – as well I saw I can use ‘spli’ instruction if necessary)</li>
<li>Comments/texts indicates the beginning of a bloc immediately followed by the number of sub-blocs</li>
<li>So I need to read/record all the values in order to build a matrix before working on it (using Numpy & vectorization)
<ul>
<li>The columns 2 and 3 have been added for further treatments</li>
<li>The ‘0’ values will be specifically treated afterward</li>
</ul>
</li>
</ol>
<p><br /></p>
<p>Numpy won’t be a problem I guess (I did some basic tests and I’m quite confident) on how to proceed, but I’m really blocked on data records … I trying to find a way to efficiently read and record data in a matrix:</p>
<ul>
<li>avoiding dynamic memory allocation (here using ‘append’ in python meaning, not np),</li>
<li>dealing with huge asci file: the latest file I get contains more than <span style="text-decoration: underline; color: #0000ff;"><strong>60 million of lines</strong></span></li>
</ul>
<p><br /></p>
<p>Please find in attachment an extract of the input format (‘example_of_input’), and the matrix I’m trying to create and manage with Numpy</p>
<p><br /></p>
<p>Thanks again for your time</p>
<p>Paul</p>
<p><br /></p>
<p>#######################################</p>
<p>##BEGIN <span style="color: #ff0000;"><em>-> line number x in the original file</em></span></p>
<p>42 <em><span style="color: #ff0000;">-> indicates the number of sub-blocs</span></em></p>
<p>1 <em><span style="color: #ff0000;">-> number of the 1rst sub-bloc</span></em></p>
<p>6 <em><span style="color: #ff0000;">-> gives how many value belong to the sub bloc</span></em></p>
<p>12</p>
<p>47</p>
<p>2</p>
<p>46</p>
<p>3</p>
<p>51</p>
<p>….</p>
<p>13 <span style="color: #ff0000;"><em> -> another type of sub-bloc with 25 values</em></span></p>
<p>25</p>
<p>15</p>
<p>88</p>
<p>21</p>
<p>42</p>
<p>22</p>
<p>76</p>
<p>19</p>
<p>89</p>
<p>0</p>
<p>18</p>
<p>80</p>
<p>23</p>
<p>38</p>
<p>24</p>
<p>73</p>
<p>20</p>
<p>81</p>
<p>0</p>
<p>90</p>
<p>0</p>
<p>41</p>
<p>0</p>
<p>39</p>
<p>0</p>
<p>77</p>
<p>…</p>
<p>42 <em><span style="color: #ff0000;">-> another type of sub-bloc with 2 values</span></em></p>
<p>2</p>
<p>115</p>
<p>109</p>
<p><br /></p>
<p> #######################################</p>
<p><span style="text-decoration: underline;"><strong>The matrix result</strong></span></p>
<p>1 0 0 6 12 47 2 46 3 51 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0</p>
<p>2 0 0 6 3 50 11 70 12 51 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0</p>
<p>3 0 0 8 11 50 3 49 4 54 5 57 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0</p>
<p>4 0 0 8 12 70 11 66 9 65 10 68 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0</p>
<p>5 0 0 8 2 47 12 68 10 44 1 43 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0</p>
<p>6 0 0 8 5 56 6 58 7 61 11 57 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0</p>
<p>7 0 0 8 11 61 7 60 8 63 9 66 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0</p>
<p>8 0 0 19 12 47 2 46 3 51 0 13 97 14 92 15 96 0 72 0 48 0 52 0 0 0 0 0 0</p>
<p>9 0 0 19 13 97 14 92 15 96 0 16 86 17 82 18 85 0 95 0 91 0 90 0 0 0 0 0 0</p>
<p>10 0 0 19 3 50 11 70 12 51 0 15 89 19 94 13 96 0 52 0 71 0 72 0 0 0 0 0 0</p>
<p>11 0 0 19 15 89 19 94 13 96 0 18 81 20 84 16 85 0 90 0 77 0 95 0 0 0 0 0 0</p>
<p>12 0 0 25 3 49 4 54 5 57 11 50 0 15 88 21 42 22 76 19 89 0 52 0 53 0 55 0 71</p>
<p>13 0 0 25 15 88 21 42 22 76 19 89 0 18 80 23 38 24 73 20 81 0 90 0 41 0 39 0 77</p>
<p>14 0 0 25 11 66 9 65 10 68 12 70 0 19 78 25 99 26 98 13 94 0 71 0 67 0 69 0 72</p>
<p>….</p>
<p><br /></p>
<p>#######################################</p>
<p><span style="text-decoration: underline;"><strong>An example of the code I started to write</strong></span></p>
<p># -*- coding: utf-8 -*-</p>
<p> import time, sys, os, re</p>
<p>import itertools</p>
<p>import numpy as np</p>
<p><br /></p>
<p>PATH = str(os.path.abspath(''))</p>
<p><br /></p>
<p>input_file_name ='/example_of_input.txt'</p>
<p><br /></p>
<p><br /></p>
<p><br /></p>
<p>## check if the file exists, then if it's empty or not</p>
<p>if (os.path.isfile(PATH + input_file_name)):</p>
<p> if (os.stat(PATH + input_file_name).st_size > 0):</p>
<p> </p>
<p> ## go through the file in order to find specific sentences</p>
<p> ## specific blocks will be defined afterward </p>
<p> Block_position = []; j=0;</p>
<p> with open(PATH + input_file_name, "r") as data:</p>
<p> for line in data:</p>
<p> if '##BEGIN' in line:</p>
<p> Block_position.append(j)</p>
<p> j=j+1</p>
<p> </p>
<p> </p>
<p> ## just to tests to get all the values</p>
<p># i = 0</p>
<p># data = np.zeros( (505), dtype=np.int )</p>
<p># with open(PATH + input_file_name, "r") as f:</p>
<p># for i in range (0,505):</p>
<p># data[i] = int(f.read(Block_position[0]+1+i))</p>
<p># print ("i = ", i)</p>
<p> </p>
<p> </p>
<p># for line in itertools.islice(f,Block_position[0],516):</p>
<p># data[i]=f.read(0+i)</p>
<p># i=i+1</p>
<p> </p>
<p><br /></p>
<p><br /></p>
<p><br /></p>
<p> else:</p>
<p> print "The file %s is empty : post-processing cannot be performed !!!\n" % input_file_name</p>
<p><br /></p>
<p> </p>
<p>else:</p>
<p> print "Error : the file %s does not exist: post-processing stops !!!\n" % input_file_name</p>
<p> </p>
</div>
</body></html>