Parallel(?) programming with python
Louis Krupp
lkrupp at invalid.pssw.com.invalid
Mon Aug 8 14:02:19 EDT 2022
On 8/8/2022 4:47 AM, Andreas Croci wrote:
> tI would like to write a program, that reads from the network a fixed
> amount of bytes and appends them to a list. This should happen once a
> second.
>
> Another part of the program should take the list, as it has been
> filled so far, every 6 hours or so, and do some computations on the
> data (a FFT).
>
> Every so often (say once a week) the list should be saved to a file,
> shorthened in the front by so many items, and filled further with the
> data coming fom the network. After the first saving of the whole list,
> only the new part (the data that have come since the last saving)
> should be appended to the file. A timestamp is in the data, so it's
> easy to say what is new and what was already there.
>
> I'm not sure how to do this properly: can I write a part of a program
> that keeps doing its job (appending data to the list once every
> second) while another part computes something on the data of the same
> list, ignoring the new data being written?
>
> Basically the question boils down to wether it is possible to have
> parts of a program (could be functions) that keep doing their job
> while other parts do something else on the same data, and what is the
> best way to do this.
You might be able to do what you need by making the file system work for
you:
Use numbered files, something like DATA/0001, DATA/0002, etc.
Start by initializing a file number variable to 1 and creating an empty
file, DATA/0001. The current time will be your start time.
In an infinite loop, just as in Stefan's example:
Read from the network and append to the current data file. This
shouldn't take long unless the file is on a remote system.
If six hours have gone by (compare the current time to the start time),
close the current date file, create a thread (see Stefan's example) to
call your FFT with the name of the current file, increment the file
number, and open a new empty data file.
If you want to, you can consolidate files every week or so. The Python
library has functions that will let you get a list files in a directory.
If you're on a Linux or UNIX system, you can use shell commands to
append, copy or rename files.
Have fun.
Louis
More information about the Python-list
mailing list