Hi Richard, On Tue, Apr 29, 2014 at 15:46 +1000, Richard Jones wrote:

I think that's a great plan, Holger.

Also, I've proposed a talk about devpi for PyCon AU in August so if you can get the reworking out of the way by then, that'd be great... ;-)

our master is already using Pyramid now thanks to Florian's work, so we are getting started and I am confident we have most of the below ready and released by August :) cheers, holger

Richard

On 28 April 2014 22:42, holger krekel <hol...@merlinux.eu> wrote:

...

Hi there, (CC Donald who develops warehouse/next-pypi),

There are two upcoming major new features coming up for devpi and devpi-server, partially funded by an undisclosed company:

- a near-real-time mirroring facility that allows to have one devpi-server instance mirror all changes from another devpi-server instance. One major use case is to have a failover server that you can switch in if the main server crashed. Another use case is to allow for faster nearby mirrors in geo-distributed settings.

- a new web/search interface that is being developed currently by Florian Schulze. It currently uses "whoosh" as a search backend and is generally based on the Pyramid web framework.

One bigger architectural question is how the web interface and the devpi-server core communicate and relate to each other, also in context of the new mirroring functionality.

One possibility (A) is to run the web interface in a separate process (or even on a separate server) and have it access the json core API via http. This allows for de-coupling and the search/web interface would be free to implement its own backend and data handling. It also means you would need to run two processes to get all benefits. And we would need to add new externally visible APIs to minimize the number of network requests between the components (give me all metadata of private projects and of all accessed PyPI ones etc.)

The other possibility (B) is run both the core and web/search components in one process. This allows direct access of internal data structures without the need to define external APIs and without the implied network communication overheads.

If we can run the web interface within a mirroring devpi-server process we still achieve a decoupled deployment of the more complex web interface and the core devpi-server interface. In this setting, the devpi-server core package will only implement the json/pip/setuptools and the mirroring APIs. The devpi-web package would additionally implement a web interface by depending on the devpi-server package and modifying/enriching the http endpoints.

After a longer discussion with Florian this morning i therefore tend to go for this possibility B and to go for migrating also the core devpi-server to use Pyramid instead of bottle, resulting in more unified server code. Also Pyramid is certainly a rich and very well tested system for building web services. It particularly allows for explicit http request passing without relying on thread locals.

So much for the current thoughts and plans. If you have any thoughts/comments, please shoot.

best, holger

-- You received this message because you are subscribed to the Google Groups "devpi-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to devpi-dev+...@googlegroups.com. To post to this group, send email to devp...@googlegroups.com. Visit this group at http://groups.google.com/group/devpi-dev. For more options, visit https://groups.google.com/d/optout.