Mailman 3 October 2015 - devpi-dev

devpi web themes
by Andrzej Ostrowski 15 Apr '16

15 Apr '16

Hi, I'm playing around with devpi themes. My goal is to create a nice theme for my company. I'd like to add extra information to some pages, like the index page or the main page. I added a snippet of code which produces an "index inheritance diagram" on each index page display. Because the company I work for is not that small, these diagrams can get strange. A sample diagram: index inheritance diagram <https://lh3.googleusercontent.com/--9yr5WSohOc/VZ0BL6xN3YI/AAAAAAAAAAs/8wZq…>. Each box is an index, I intentionally removed all user and index names, except for root/pypi. It's not my indices, please don't judge me for the design :) I'm concerned about performance here. First of all, currently I generate those on display action, which doesn't seem the way to go. Could devpi provide a hook so that I'm able to regenerate a diagram on an index update event? Additionally, I'd like to add more information to an index page and the root pages. So the question is: Could functions in devpi_web.views expose more data in get_index() and root()? Potentially other functions? I know I can always get the data with python snippets in the templates, but it all takes precious time. For instance, in my company, some indices are regular user indices, some of them are project indices. I'd like to sort the list of indices on the root page: display the project ones first, and user indices below. The way I would see this is the custom_data property of an index. The bottom line: I think people could easily create fancier themes if the devpi web views functions exposed more data to the templates, even if that data is not used in the default theme. It would be great to get the full indexconfig json structure for each index in both get_index() and root() functions from devpi_web.views module. Now I'm not sure how much more data others might want to get there. I hope these things make sense. Any thoughts?

4 6

improving concurrency, reliability of devpi-server
by holger krekel 07 Nov '15

07 Nov '15

Hi Florian, all, there are at least three issues that somewhat interelate and share the common topic of service reliability, concurrency and interactions with remote pypi.python.org or devpi masters: https://bitbucket.org/hpk42/devpi/issues/267/intermittent-assertionerror-in… multiple devpi-server processes write to the same (networked shared) file system resulting in failed transaction handling. devpi-server was not designed for it. https://bitbucket.org/hpk42/devpi/issues/274/recurring-consistency-issues-w… under high load database/transaction handling issues arise. (although it's unclear what the precise scenario is, how to replicate) https://bitbucket.org/hpk42/devpi/issues/208/pip-gets-timeout-on-large-pack… trying to install an uncached package that originates from pypi.python.org can fail if devpi-server cannot download the package fast enough. Starting with the last issue, we probably need to re-introduce a way to stream remote files instead of first retrieving it in full and only then starting a client response . This should take into account that there could be two threads (or even two processes) which try to retrieve the same file. This means that we start a response as soon as we got a http return code and them forward-stream the content. The first two issues could be mitigated by introducing a better read/write transaction separation. background: GET-ting simple pages or release files can cause write transactions in a devpi-server process because we may need to retrive & cache information from pypi.python.org or a devpi-server master. Currently, during the processing of the GET request we at some point promote a READ-transaction into a WRITE-transaction through a call to keyfs.restart_as_write_transaction() and persist what we have. This all happens before the response to the client is returned. "Restarting as write" is somewhat brittle because something might have changed since we started our long-running request. Strategy and notes on how to mitigate all three issues: - release files: cache and stream chunks of what we remotely receive, all within a READ-transaction and all within RAM. This should ideally be done in such a way that if multiple threads stream the same file, only one remote http request is done for fetching the file. Otherwise we end up retrieving large files multiple times unneccessarily. After the http response to the client is complete we (try to) write it to sqlite/filesystem so that subsequent requests can work from the local filesystem. Here we need to be careful and consider that we might have multiple writers/streamers. If we discover that where we want to write someone else already has we can simply forget about it. - simple pages: first retrieve the remote simple page in RAM, process it, serve the full pyramid response and then (try to) cache it after the response is completed. Here we probably don't need to care if multiple threads are trying to retrieve the same simple page because simple pages are not big. - we cache things in RAM because even for large files it shouldn't matter given that servers typically have multiple gigabytes of RAM. And we can avoid synchronization issues wrt to the file system (see also the first issues where multiple processes write to the file system). - we always finish response to the client before we attempt to do a write transaction. The write transaction part should be implemented in a separate function to make it clear what kind of state we can rely on and what we must re-check. (currently we do the READ->Write switch in the middle of a view function). - we also need to review how exactly we open the sqlite DB for writing and if multiple processes correctly serialize on their write attempts, particularly in the multi-process case. - care must be taken wrt to waitress and nginx configuration and their buffering, see for example: http://www.4byte.cn/question/68410/pyramid-stream-response-body.html any feedback or thoughts welcome. holger -- about me: http://holgerkrekel.net/about-me/ contracting: http://merlinux.eu

2 6

Can devpi proxy multiple remote respositories?
by Sam Reynolds 12 Oct '15

12 Oct '15

Our corp standard requires that all in-house build products (notably including .whl files) must be housed in Artifactory. We can resolve them from there via Artifactory's PyPi facade (read-only). But we don't want to have to check in external packages (from public PyPi) into Artifactory--and it makes no sense to do so. Can an index in devpi have, as a base, both public/pipi and an additional external, read-only repo, and serve as proxy for both?

2 2

devpi fails when package if found but not correct version
by Jens Andersen 09 Oct '15

09 Oct '15

Hi, I meet a bad issue yesterday: - We have a devpi server running with some local packages uploaded, but with normal fall back to the pypi index. - Referring packages not found on the devpi server is correctly fetched from pypi. - As part of building a local wheel using 'pip wheel' and pushing it to devpi, I also pushed all the depending wheels generated to devpi. - Now, when I try 'pip install maven' I properly get the version stored on my devpi server (e.g. version 3.4) - HOWEVER, if I do 'pip install maven==3.5', devpi reports no such version and fails as it only knows of version 3.4 - it does NOT fall back to search the pypi index... Is this by design? Shouldn't devpi also fall back to searching the pypi index in this case (package is found, but no matching version)? Comments anyone?

2 3

devpi-dev October 2015