Re: [devpi-dev] Re: simple-page serving for private indexes optimized

Hi Stephan, On Mon, Nov 16, 2015 at 08:45 +0000, Erb, Stephan wrote:

Hi Holger,

I really like the idea of the simplelinks cache. It seems to strike a nice balance between effect and implementation effort. I have made a few comments on your commit diff (https://bitbucket.org/hpk42/devpi/commits/54ae2eb00d97602330af86ba9892367932...).

What would it take to make this work without a full export import cycle? A normal version update takes a couple of minutes for us, whereas an export-import cycle is more like a multi-day project.

we could probably avoid an import/export cycle by regenerating the "simplelinks" cache for private indexes if it is not there. But it might get complicated wrt to replicas. Also it seems that we have a somewhat bad behavioural regression in 2.4.0 which leads to many more write-changes on simple-page requests on mirrors than neccessary (pre-2.4.0 the behaviour was better). I am currently preparing a new release on https://bitbucket.org/hpk42/devpi/branch/less-commits to remedy this issue. This fix will not require an export/import cycle but it would make sense to do one anyway to reduce the number of commits/changes a replica needs to pull. So the plan is: release the changes contained in less-commits soon (as 2.4.1 or 2.5.0) and head for a 3.0 which could, apart from the simple-page optimization, incorporate the long-discussed "projectname" normalization which definitely requires an export/import cycle, and possibly the "mirror" work which Florian brought up recently. holger

Best Regards, Stephan ________________________________________ From: holger krekel Sent: Thursday, November 12, 2015 12:26 PM To: devpi-dev; Erb, Stephan Subject: simple-page serving for private indexes optimized

Hi Stephan, Florian, all,

while the last release speed up simple-page serving it helped more for the root/pypi case than the private indexes. I just did some benchmarking for private indexes with roughly 200 release file entries. I got around 34 requests per second on my machine and the attached proflog-2.4.0 profile. Most of the time is spent in "get_linkstore_perstage" and "get_versiondata_perstage" because we store each version under its own key, resulting in hundreds of db-accesses and dependent processing.

So I introduced a per-project db-persisted "simplelinks" cache similar to the root/pypi one and got 133 requests per second, roughly 4 times better, also see "proflog-simplecache" profile attached. There are still some easy improvements that can be made on top. The diff is quite simple but it would currently require an export-import:

https://bitbucket.org/hpk42/devpi/branch/simplecache#diff

It's likely possible to avoid the export/import if neccessary. If you'd like to repeat the measurements you can import this devpi-server state:

http://merlinux.eu/~hpk/exp1.tar.gz

either from devpi-server-2.4.0 or from the branch and after server-startup run:

ab -H "User-Agent: pip/6.0" -n 100 http://localhost:3141/hpk/dev/+simple/dddttt

The "dddttt" project also has releases on pypi.python.org.

Maybe after merging the simplecache branch we could also cleanup how we internally treat projectnames: always use normalized names for keys in the DB so that we can simply compute them without any db-access (currently we need to access the database to get from a name to the "real projectname", e.g. to "Django" from "django"). It's a long standing issue to simplify this and we would anyway require an export-import cycle for it.

best, holger

-- about me: http://holgerkrekel.net/about-me/ contracting: http://merlinux.eu

-- You received this message because you are subscribed to the Google Groups "devpi-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to devpi-dev+...@googlegroups.com. To post to this group, send email to devp...@googlegroups.com. Visit this group at http://groups.google.com/group/devpi-dev. For more options, visit https://groups.google.com/d/optout.

-- about me: http://holgerkrekel.net/about-me/ contracting: http://merlinux.eu