Hi Holger, doing a quick 2.4/2.5 bugfix release sounds good. If possible this should also address the regressions https://bitbucket.org/hpk42/devpi/issues/286/ and https://bitbucket.org/hpk42/devpi/issues/288. We did some performance benchmarks on our end. With the recent optimizations and changes we are much better of. Waiting for a full blown 3.0 release for the simple-page optimization as proposed by you is ok for us. Cheers, Stephan ________________________________________ From: holger krekel <hol...@merlinux.eu> Sent: Wednesday, November 18, 2015 5:18 PM To: Erb, Stephan Cc: holger krekel; devpi-dev Subject: Re: [devpi-dev] Re: simple-page serving for private indexes optimized Hi Stephan, On Mon, Nov 16, 2015 at 08:45 +0000, Erb, Stephan wrote:
Hi Holger,
I really like the idea of the simplelinks cache. It seems to strike a nice balance between effect and implementation effort. I have made a few comments on your commit diff (https://bitbucket.org/hpk42/devpi/commits/54ae2eb00d97602330af86ba9892367932...).
What would it take to make this work without a full export import cycle? A normal version update takes a couple of minutes for us, whereas an export-import cycle is more like a multi-day project.
we could probably avoid an import/export cycle by regenerating the "simplelinks" cache for private indexes if it is not there. But it might get complicated wrt to replicas. Also it seems that we have a somewhat bad behavioural regression in 2.4.0 which leads to many more write-changes on simple-page requests on mirrors than neccessary (pre-2.4.0 the behaviour was better). I am currently preparing a new release on https://bitbucket.org/hpk42/devpi/branch/less-commits to remedy this issue. This fix will not require an export/import cycle but it would make sense to do one anyway to reduce the number of commits/changes a replica needs to pull. So the plan is: release the changes contained in less-commits soon (as 2.4.1 or 2.5.0) and head for a 3.0 which could, apart from the simple-page optimization, incorporate the long-discussed "projectname" normalization which definitely requires an export/import cycle, and possibly the "mirror" work which Florian brought up recently. holger
Best Regards, Stephan ________________________________________ From: holger krekel <hol...@merlinux.eu> Sent: Thursday, November 12, 2015 12:26 PM To: devpi-dev; Erb, Stephan Subject: simple-page serving for private indexes optimized
Hi Stephan, Florian, all,
while the last release speed up simple-page serving it helped more for the root/pypi case than the private indexes. I just did some benchmarking for private indexes with roughly 200 release file entries. I got around 34 requests per second on my machine and the attached proflog-2.4.0 profile. Most of the time is spent in "get_linkstore_perstage" and "get_versiondata_perstage" because we store each version under its own key, resulting in hundreds of db-accesses and dependent processing.
So I introduced a per-project db-persisted "simplelinks" cache similar to the root/pypi one and got 133 requests per second, roughly 4 times better, also see "proflog-simplecache" profile attached. There are still some easy improvements that can be made on top. The diff is quite simple but it would currently require an export-import:
https://bitbucket.org/hpk42/devpi/branch/simplecache#diff
It's likely possible to avoid the export/import if neccessary. If you'd like to repeat the measurements you can import this devpi-server state:
http://merlinux.eu/~hpk/exp1.tar.gz
either from devpi-server-2.4.0 or from the branch and after server-startup run:
ab -H "User-Agent: pip/6.0" -n 100 http://localhost:3141/hpk/dev/+simple/dddttt
The "dddttt" project also has releases on pypi.python.org.
Maybe after merging the simplecache branch we could also cleanup how we internally treat projectnames: always use normalized names for keys in the DB so that we can simply compute them without any db-access (currently we need to access the database to get from a name to the "real projectname", e.g. to "Django" from "django"). It's a long standing issue to simplify this and we would anyway require an export-import cycle for it.
best, holger
-- about me: http://holgerkrekel.net/about-me/ contracting: http://merlinux.eu
-- You received this message because you are subscribed to the Google Groups "devpi-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to devpi-dev+...@googlegroups.com. To post to this group, send email to devp...@googlegroups.com. Visit this group at http://groups.google.com/group/devpi-dev. For more options, visit https://groups.google.com/d/optout.
-- about me: http://holgerkrekel.net/about-me/ contracting: http://merlinux.eu