several people asked me to convert the devpi repository from
hg/mercurial to git. Although i am mostly fine with hg/mercurial
myself i'd like to use git in some more depth. Therefore i plan
to convert the repository to git but continue to host it at bitbucket
for the time being. Probably going to happen sometime this week.
I have a central devpi-server instance deployed on a vps on the cloud,
which I'd like to mirror locally on my laptop.
Is it currently possible ? I would appreciate if someone could point me the
right docs. I already searched without any success.
Cheers, and thanks for the nice work done so far !
Hi there, (CC Donald who develops warehouse/next-pypi),
There are two upcoming major new features coming up for devpi and
devpi-server, partially funded by an undisclosed company:
- a near-real-time mirroring facility that allows to have one
devpi-server instance mirror all changes from another devpi-server
instance. One major use case is to have a failover server that
you can switch in if the main server crashed. Another use case
is to allow for faster nearby mirrors in geo-distributed settings.
- a new web/search interface that is being developed currently
by Florian Schulze. It currently uses "whoosh" as a search backend
and is generally based on the Pyramid web framework.
One bigger architectural question is how the web interface and
the devpi-server core communicate and relate to each other,
also in context of the new mirroring functionality.
One possibility (A) is to run the web interface in a separate process
(or even on a separate server) and have it access the json core API
via http. This allows for de-coupling and the search/web interface
would be free to implement its own backend and data handling. It
also means you would need to run two processes to get all benefits.
And we would need to add new externally visible APIs to minimize
the number of network requests between the components (give me
all metadata of private projects and of all accessed PyPI ones etc.)
The other possibility (B) is run both the core and web/search components
in one process. This allows direct access of internal data structures
without the need to define external APIs and without the implied
network communication overheads.
If we can run the web interface within a mirroring devpi-server process
we still achieve a decoupled deployment of the more complex web
interface and the core devpi-server interface. In this setting,
the devpi-server core package will only implement the json/pip/setuptools
and the mirroring APIs. The devpi-web package would additionally implement
a web interface by depending on the devpi-server package and
modifying/enriching the http endpoints.
After a longer discussion with Florian this morning i therefore
tend to go for this possibility B and to go for migrating also the core
devpi-server to use Pyramid instead of bottle, resulting in more unified
server code. Also Pyramid is certainly a rich and very well tested system
for building web services. It particularly allows for explicit http
request passing without relying on thread locals.
So much for the current thoughts and plans. If you have any
thoughts/comments, please shoot.
finally i got around to do 1.2.2 releases of the devpi pypi server tools
with a few fixes and improvements. See http://doc.devpi.net for docs
and instructions and below for the changelog.
- fix issue78: create less directories for pypi package files by
splitting the md5 part into two. Avoids TooManyLinks errors in
- fix --stop on windows. Thanks to Christian Ullrich for the PR.
- fix issue79: interoperate with pip-1.5 by interpreting accept
header as "*/*" as html_preferred. Thanks Richard Jones.
- use latest virtualenv-1.11.2 when bootstrapping on jenkins
- fix issue89: adapt for bottle changes in 0.12.1. Thanks
- depend on pkginfo>1.2b1 for wheel metadata reading support,
remove twine dependency. Thanks Tres Seaver.
- new: also write buildout configuration file with --set-cfg.
Thanks Christian Ullrich for the PR.