Hi there, (CC Donald who develops warehouse/next-pypi),
There are two upcoming major new features coming up for devpi and
devpi-server, partially funded by an undisclosed company:
- a near-real-time mirroring facility that allows to have one
devpi-server instance mirror all changes from another devpi-server
instance. One major use case is to have a failover server that
you can switch in if the main server crashed. Another use case
is to allow for faster nearby mirrors in geo-distributed settings.
- a new web/search interface that is being developed currently
by Florian Schulze. It currently uses "whoosh" as a search backend
and is generally based on the Pyramid web framework.
One bigger architectural question is how the web interface and
the devpi-server core communicate and relate to each other,
also in context of the new mirroring functionality.
One possibility (A) is to run the web interface in a separate process
(or even on a separate server) and have it access the json core API
via http. This allows for de-coupling and the search/web interface
would be free to implement its own backend and data handling. It
also means you would need to run two processes to get all benefits.
And we would need to add new externally visible APIs to minimize
the number of network requests between the components (give me
all metadata of private projects and of all accessed PyPI ones etc.)
The other possibility (B) is run both the core and web/search components
in one process. This allows direct access of internal data structures
without the need to define external APIs and without the implied
network communication overheads.
If we can run the web interface within a mirroring devpi-server process
we still achieve a decoupled deployment of the more complex web
interface and the core devpi-server interface. In this setting,
the devpi-server core package will only implement the json/pip/setuptools
and the mirroring APIs. The devpi-web package would additionally implement
a web interface by depending on the devpi-server package and
modifying/enriching the http endpoints.
After a longer discussion with Florian this morning i therefore
tend to go for this possibility B and to go for migrating also the core
devpi-server to use Pyramid instead of bottle, resulting in more unified
server code. Also Pyramid is certainly a rich and very well tested system
for building web services. It particularly allows for explicit http
request passing without relying on thread locals.
So much for the current thoughts and plans. If you have any
thoughts/comments, please shoot.
best,
holger
