Thank you Steve for your comments, but I am still interested in knowing how Mailman developers would debug issues with Hyperkitty periodic tasks.
If no logging is possible for those tasks, I'd consider this as an issue to fix. But I'm not a Python developer and looking at the application from the infrastructure side. So I'm not sure if it's possible or not and if is it an issue or not. I appreciate any clarification you can provide.
Danil
On Fri, Sep 29, 2023 at 6:30 PM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Danil Smirnov writes:
The issue was initially noticed by a very high load of CPU when running periodic tasks.
I may have seen this but no longer have access to the system where it happened, and there were other issues so I have no confidence that the high loads were due to the problem described below. This was during the initial import of a very large number (~5000) of archives, and running mailmanweb update_index for the whole thing. The problem was that with qcluster running, the whole archive indexing and the periodic indexing stepped on each other's locks.
We solved this by telling people archive search would be available randomly over the next few days (ie, when the archiving process completed that list's archive ;-) and stopping the periodic process (just renaming the index task in qcluster's queue does this) before running the index everything task..
I suggest doing this to get your archives close to up to date so the periodic indexing task doesn't get overwhelmed and fail to complete with its interval. I have no confidence this will solve your problem completely, but the indexing task may get in the way of solving it.
I also recommend getting in touch with the Django community as they're more likely to know things like how to set logging verbosity in these low-level components.
Steve