Thread safety issue (I think) with defaultdict

[Steve D'Aprano]

On Sat, 4 Nov 2017 05:12 am, Israel Brewster wrote:

[...]
>> People generally understand how to move data around, and the mistakes are
>> usually pretty obvious when they happen.
> 
> I think the existence of this thread indicates otherwise :-) This mistake
> was far from obvious, and clearly I didn't understand properly how to move
> data around *between processes*. Unless you are just saying I am ignorant or
> something? :-)

Yes, you were ignorant -- you didn't even realise that you were using
processes, you thought you were using threaded code when it was actually
multiprocessing code. No wonder you got it wrong.

Of course you have a good excuse: the multiprocessing is hidden deep inside
not just the library you were using, but the library *it* was using.

(I don't know how obvious the documentation of the libraries make this --
maybe they're to blame, for not being clear enough -- or maybe you were
simply ignorant about the tools you were using.)

You can't judge multiprocessing code on the basis of bugs caused by assuming
that it was threading code, writing in a threading style with shared data. If
you misuse your tools, that's not the tool's fault.

If anything, we can say that the ultimate error was that you decided to write
in a threaded style without actually using threads: the error was your
(dangerous?) choice to write non-deterministic code using shared data.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.