Is there a more efficient threading lock?
Thomas Passin
list1 at tompassin.net
Sat Feb 25 11:48:37 EST 2023
On 2/25/2023 10:52 AM, Skip Montanaro wrote:
> I have a multi-threaded program which calls out to a non-thread-safe
> library (not mine) in a couple places. I guard against multiple
> threads executing code there using threading.Lock. The code is
> straightforward:
>
> from threading import Lock
>
> # Something in textblob and/or nltk doesn't play nice with no-gil, so just
> # serialize all blobby accesses.
> BLOB_LOCK = Lock()
>
> def get_terms(text):
> with BLOB_LOCK:
> phrases = TextBlob(text, np_extractor=EXTRACTOR).noun_phrases
> for phrase in phrases:
> yield phrase
>
> When I monitor the application using py-spy, that with statement is
> consuming huge amounts of CPU. Does threading.Lock.acquire() sleep
> anywhere? I didn't see anything obvious poking around in the C code
> which implements this stuff. I'm no expert though, so could easily
> have missed something.
I'm no expert on locks, but you don't usually want to keep a lock while
some long-running computation goes on. You want the computation to be
done by a separate thread, put its results somewhere, and then notify
the choreographing thread that the result is ready.
This link may be helpful -
https://anandology.com/blog/using-iterators-and-generators/
More information about the Python-list
mailing list