multiprocessing & more

[Adam Skutt]

On Feb 13, 12:34 pm, Andrea Crotti <andrea.crott... at gmail.com> wrote:
>
> First of all, does it make sense to use multiprocessing and a short
> value as boolean to check if the simulation is over or not?
>

Maybe, but without knowing exactly what you're doing it's difficult to
say if any other approach would be superior.  Plus, most of the other
approaches I can think of would require code modifications or platform-
specific assumptions.

>
> As last thing to know when to start to analyze the data I thought about this
>
>         while len(listdir(res_dir)) < len(network):
>             sleep(0.2)
>
> which in theory it should be correct, when there are enough files as
> the number of nodes in the network everything should be written.  BUT
> once every 5 times I get an error, telling me one file doens't exists.
>
> That means that for listdir the file is already there but trying to
> access to it gives error, how is that possible?

File I/O is inherently a concurrent, unsynchronized activity.  A
directory listing can become stale at any time, even while the
directory listing is being built (e.g., imagine running ls or dir in a
directory where rm or del is currently executing).  When a directory
is being modified while you're listing it, the contents of the listing
essentially become "undefined": you may get entries for files that no
longer exist, and you may not get entries for that do exist.  A
directory listing may also return duplicate entries; this is what I
expect is happening to you.

The right thing to do is actually check to see if all the files you
want exist, if you can.  If not, you'll have to keep waiting until
you've opened all the files you expect to open.

Adam