We are happy to see that you plan to address the performance issues related to the indexing of devpi-web.
Your proposal regarding splitting up the search indexing sounds good. Am I correct in assuming that this will mean that we will get a replica that, from the API point of view, catches up as quickly as one without devpi-web and, afterwards, will start returning reasonable search results step by step? This would help us with disaster recovery and, even more, with the export-import cycle for the major version upgrade.
The second problem you mention in your mail, the reindexing of whole projects, is
however the one more interesting for us in day to day operations. We have some projects
with a significant number of versions and documentation of significant size. For these
projects, uploading a version will mean that the replica process will be stuck indexing
for around 10 minutes. We are currently mitigating this issue by running additional
--requests-only replica process. However, if the load induced could by
uploading a new version could be reduced conceptually this would of course be the better
There might be two limitations that a solution for this second problem could have that would be non-issues for us:
It could be constrained to non-volatile indices. As for these versions can't change this might allow for some optimization.
Indexing could be running in a separate process. This would avoid indexing hogging the global lock of the interpreter running the replication process. Additional code complexity might be an issue in this case, though.
In any case, we are looking forward to any improvements in this area.
Kind Regards, Matthias
Dr. Matthias Bach Senior Software Engineer
firstname.lastname@example.org T +49 721 383117 6244
Blue Yonder GmbH Ohiostraße 8 76149 Karlsruhe
Registergericht Mannheim, HRB 704547 · USt-IdNr. DE 277 091 535 · Geschäftsführer: Uwe Weiss (CEO), Jochen Bossert
Diese E-Mail enthaelt vertrauliche oder geschuetzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtuemlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet.
This e-mail may contain confidential and/or privileged information. If you are not the recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.