strptime issue in multi-threaded application

[Christian Heimes]

Joe Holloway schrieb:
> We recently uplifted our web application to run on Python 2.6.2.
> We've noticed on a couple occasions that calls into time.strptime have
> failed with this exception:
> 
> ImportError: Failed to import _strptime because the import lockis
> [sic] held by another thread.

The error message is my fault. The cause of the mistake is obvious:

PyErr_Format(PyExc_ImportError,
            "Failed to import %.200s because the import lock"
            "is held by another thread.",
            name);

> I poked around the source code enough to realize that this is
> apparently due to time.strptime using PyImport_ImportModuleNoBlock
> which potentially raises an ImportError rather than waiting for the
> "import lock" to be released [1].  This appears to have been
> introduced as a workaround for other thread safety concerns [2].

Without PyImport_ImportModuleNoBlock() your application would block forever.

> Does this indicate that strptime and any other library function that
> uses the non-blocking import call in this fashion are not thread safe?
> Is there an idiomatic way of dealing with this error in
> multi-threaded applications?

It's not a matter of thread safety per se. I've added the function to
work around a dead lock situation. Python allows you to import a module
in a subthread but the entire import system is protected by a lock.

> Like I mentioned, it's only happened on a couple occasions because the
> right conditions have to be in place, but something doesn't seem right
> about it.  I thought I'd ask on the mailing list before going so far
> as to open a ticket, but feel free to direct me there if that's the
> appropriate place for this.

I have an idea what might happen in your application. Is an import
triggering the start of a thread? You can get around the issue by
decoupling imports from thread startups. Your application should import
all modules before it starts its threaded components.

For now you can decrease the severity of your issue by placing "import
_strptime" next to "import time" somewhere in your code. Once it a
module is loaded PyImport_ImportModuleNoBlock() will not fail to import
it a second time.

Christian