Mailman 3 May 2014 - devpi-dev

hg->git conversion of devpi
by holger krekel 01 Sep '14

01 Sep '14

Hi all, several people asked me to convert the devpi repository from hg/mercurial to git. Although i am mostly fine with hg/mercurial myself i'd like to use git in some more depth. Therefore i plan to convert the repository to git but continue to host it at bitbucket for the time being. Probably going to happen sometime this week. best, holger

3 3

Index inheritance across devpi-server instances
by Cristiano Cortezia 15 Jul '14

15 Jul '14

Hello, I have a central devpi-server instance deployed on a vps on the cloud, which I'd like to mirror locally on my laptop. Is it currently possible ? I would appreciate if someone could point me the right docs. I already searched without any success. Cheers, and thanks for the nice work done so far !

2 3

Plain setup.py uploads
by Eric Smith 17 Jun '14

17 Jun '14

I'm trying to follow the directions for doing uploads to devpi using setup.py rather than 'devpi upload' found here: http://doc.devpi.net/1.2.1/userman/devpi_misc.html#using-plain-setup-py-for… Things are well enough configured that 'devpi upload' works, but trying a command like 'python setup.py sdist --formats=zip upload --repository=myrepo' results in: Upload failed (405): Method Not Allowed Spying on the HTTP traffic, I can see that the request is going to the expected URL and includes an Authorization header with the credentials (encoded for basic auth) from my .pypirc. Is this known to be working in 1.2.1? Any suggestions for tracking down the problem? Thanks, Eric

5 10

replication protocol draft (RFC)
by holger krekel 16 May '14

16 May '14

Hi devpi interested parties, Florian and me are progressing a lot on enhancing and refactoring devpi-server, resulting in a new web interface and search capabilities. This will eventually be part a new devpi-server release, either 1.3 or maybe directly going for 2.0 :) One issue I have been thinking about in some more depth about the replication-protocol that devpi is to grow. Please find below the current state of thoughts and plans. It includes considerations so that we can all run a lightweight local replica server on our laptops. Replication is an intricate topic and i'd be very happy if you give it a read and feedback. best, holger server Replication protocol (DRAFT) ==================================== The devpi-server replication protocol aims to: - support high-availability: when the master goes down, a replica node can be taken as the new master. - support faster installs: a geographically closer replica will answer queries much faster than the more remote master node. The devpi-server replication protocol is designed with the following user stories in mind: - User wants to connect to a geographically close devpi-serverÂ instance (real-time "replica") and use it instead of the geographically remote master devpi-server instance. The real-time replica serves the same indices,Â packages and documentation that the master provides. - user uploads to the replica server aÂ package, documentation, test result or modifies an index and expects the change to be visible at the master and other replicas. The replica will proxy such operations to theÂ master and notify the user of a successful change only after the operation propagated back to the replica. A user's authenticationÂ against a mirror is handled by relaying it to the master as well. - A server administrator can start a devpi-server in replicatingÂ mode by providing a master URL. The node will start with replicatingÂ the master information. During the initial replica/master synchronization the replica will not answer network requests. After the initial sync the replica maintains a connection to the master in order to get notified immediatelyÂ of any changes. we maintain the following functional properties and considerations: - When a package orÂ index on the master changes the replica receives the changes within a minute so that the user can access them locally. - High availability is only required for installation activities. This implies that we: 1. could handle Authentication by the master only. If the master is down, users will not be able to authenticate until the master is back online. This also implies that authentication should occur by proxy (relay) via replicas. 2. Release file upload is also done by proxying it to the master. Performance/robustness is not required for upload operations as the primary goal behind mirrors is to provide a fast and reliable mechanism to deploy test systems. Replication Protocol outline ------------------------------------------ For replicating changes from master to replica: - Every server-state change is logged to a **changelog** where each entry is associated to exactly one serial. - A replica maintains its own serial (starting at ``0``) and asks the master for the changelog entries since that serial. For each such changelog entry it changes the replica state and then increments its own serial. So the replica uses an RPC-API like this:: changelog_since_serial(since_serial) -> list of change entries We have the following change entry types: - ``RESOURCE create/modify/delete`` where resource can be a user, an index config, a project config, or a versionconfig. resource changes can be be fully transported with each changelog entry because they rarely exceed 1K (except for versionconfigs which contain release-registration metadata, itself containing the long description which can be a few Ks). - ``release archive upload``: an upload of a release file related to a specific USER/INDEX/PROJECT/VERSION. - ``documentation archive upload``: an upload of a documentation zip file related to a specific USER/INDEX/PROJECT/VERSION. - ``test resultlog upload``: an upload of a test result file related to a specific MD5/archive - ``[pypimirror] pypi project metadata``: metadata about a pypi project - ``[pypimirror] release archive``: a release file mirrored from pypi.python.org or an external site, related to a particular project/version of a pypi project. The change entries marked with ``[pypimirror]`` are a bit special because server-state changes are triggered by just accessing projects on the ``/root/pypi`` mirror. Including pypi-changes in the replication protocol will increase replication traffic considerably, see also the discussion about `laptop replication`_. Note that all non-upload changelog entries will carry the full information of the change while upload changelog entries will only contain a checksummed reference to allow replicas to obtain the respective file. This distinction is done to minimize the size of changelog entries and the bandwidth used by the replica protocol. Also, the changelog entry information will be encoded using JSON. Release archives, documentation zips and test results are rather binary files that are more efficient for a replica to download in binary mode. .. _`http relaying`: HTTP relaying of replica server to master ----------------------------------------------------------- devpi-server in replica mode serves the same API and endpoints as the master server but it will internally relay change-operations (see change entry types above). In general, such state-changing requests will be relayed to the master which should in its success code tell what serial this change refers to. The replica server can then return a success code to its client after that serial has been synchronized successfully. Other replicas may or may not have synchronized the same change. .. _`laptop replication`: Laptop replication (frequent disconnects) ------------------------------------------------ The primary user story for replication is maintaining a per-organisation multi-server install of devpi-server. In principle, a local per-laptop replica can use the same replication mechanisms, however. As laptops are often disconnected or sleeping, the replication pattern will look different: it will be more common to have relatively large lists of change entries to process. In the future, we can think about ways to minimize replication traffic by: a) subscribing to changes based on a filter (only changes related to a user, certain indices, etc.) b) only retrieving archive or documentation files into the replica if they are actually accessed on the replica side. Handling concurrency within the replica server ------------------------------------------------- Both master and replica servers can handle multiple concurrent requests. HTTP requests are run in threads (or greenlets) and we thus need to insure that the backend layer is thread safe and provides means for manipulating state in an atomic way. One particular complication is `http relaying`_ of state changes posted to the replica. Here is how that happens: - a replica receives a state-changing http request from a client - to handle this request, it trigger a respective state change on the master and waits for success response which includes the ``SERIAL`` of that change. - Before returning to the client, the replica needs to wait for the concurrently running replication protocol to reach at least the serial ``SERIAL``. After this event occurs, it can return with a success code to the client. - if the wait times out we need to return a 408/timeout or 504/GATEWAYTIMEOUT response code. In this case, it will not be clear if the state change on the server succeeded. Note that this sequence could be interrupted at any point in time because of a partial network disconnect or a failure between the three parties (replica, master, client). This may make it hard for the client to know the exact result of the original state-changing operation. To remedy this, we consider implementing a per-server (and maybe also per-index) view on "recent changes", and also detailing the "local" serials and "remote serials" as well as the replica/master connection status. Transactional master state changes / SQL ------------------------------------------------------- Every change on the devpi-server master side needs to happen atomically and needs to be associated with a unique incrementing serial number. ``devpi-server-1.2`` uses a simple internal filesystem storage and a custom ``keyfs`` module to manage this storage. While we could go for implementing some transaction mechanism on top we rather aim to switch to an SQL database and use its inbuilt ACID guarantees. We'll probably use the simple ``sqlite`` database support and `sqlalchemy <https://pypi.python.org/pypi/sqlalchemy>`_ to perform SQL operations. Binary or larger files will remain stored outside.

1 0

Re: [devpi-dev] Permanent installation: gendeploy and devpi-server options
by Jonathan Roy 09 May '14

09 May '14

I tried to fix the command in the the supervisor config file in : /home/jroy/.virtualenvs/devpi-server/etc/supervisord.conf But it is still listeming only to localhost, I doesn't seem to honor the --host 0.0.0.0 parameter. devpi-ctl status using supervisor config: /home/jroy/.virtualenvs/devpi-server/etc/supervisord.conf Should be this configuration file! 2014-05-09 11:20 GMT-04:00 Florian Schulze <ma...(a)florian-schulze.net>: > > > On 8 May 2014, at 22:02, Jonathan Roy wrote: > > If I manually run the server executing the >> command: /home/jroy/.virtualenvs/devpi-server/bin/devpi-server --port >> 3141 >> --host 0.0.0.0 --serverdir /home/jroy/.virtualenvs/devpi-server/data >> It is listening to 0.0.0.0 but not if it is start with the supervisor >> config: /home/jroy/.virtualenvs/devpi-server/bin/devpi-ctl start all >> >> Is there something I can do or this is a bug? >> > > I don't know if there is a bug in gendeploy, but what you can do is > editing supervisord.cfg to fix the command line options there. I'm not > exactly sure where it is located when using gendeploy, you might have to > use "find . -name supervisord.cfg". > > Regards, > Florian Schulze >

3 2

Re: [devpi-dev] new devpi features / architcture / Pyramid migration
by holger krekel 01 May '14

01 May '14

Hi Richard, On Tue, Apr 29, 2014 at 15:46 +1000, Richard Jones wrote: > I think that's a great plan, Holger. > > Also, I've proposed a talk about devpi for PyCon AU in August so if > you can get the reworking out of the way by then, that'd be great... > ;-) our master is already using Pyramid now thanks to Florian's work, so we are getting started and I am confident we have most of the below ready and released by August :) cheers, holger > > Richard > > > On 28 April 2014 22:42, holger krekel <hol...(a)merlinux.eu> wrote: > > Hi there, (CC Donald who develops warehouse/next-pypi), > > > > There are two upcoming major new features coming up for devpi and > > devpi-server, partially funded by an undisclosed company: > > > > - a near-real-time mirroring facility that allows to have one > > devpi-server instance mirror all changes from another devpi-server > > instance. One major use case is to have a failover server that > > you can switch in if the main server crashed. Another use case > > is to allow for faster nearby mirrors in geo-distributed settings. > > > > - a new web/search interface that is being developed currently > > by Florian Schulze. It currently uses "whoosh" as a search backend > > and is generally based on the Pyramid web framework. > > > > One bigger architectural question is how the web interface and > > the devpi-server core communicate and relate to each other, > > also in context of the new mirroring functionality. > > > > One possibility (A) is to run the web interface in a separate process > > (or even on a separate server) and have it access the json core API > > via http. This allows for de-coupling and the search/web interface > > would be free to implement its own backend and data handling. It > > also means you would need to run two processes to get all benefits. > > And we would need to add new externally visible APIs to minimize > > the number of network requests between the components (give me > > all metadata of private projects and of all accessed PyPI ones etc.) > > > > The other possibility (B) is run both the core and web/search components > > in one process. This allows direct access of internal data structures > > without the need to define external APIs and without the implied > > network communication overheads. > > > > If we can run the web interface within a mirroring devpi-server process > > we still achieve a decoupled deployment of the more complex web > > interface and the core devpi-server interface. In this setting, > > the devpi-server core package will only implement the json/pip/setuptools > > and the mirroring APIs. The devpi-web package would additionally implement > > a web interface by depending on the devpi-server package and > > modifying/enriching the http endpoints. > > > > After a longer discussion with Florian this morning i therefore > > tend to go for this possibility B and to go for migrating also the core > > devpi-server to use Pyramid instead of bottle, resulting in more unified > > server code. Also Pyramid is certainly a rich and very well tested system > > for building web services. It particularly allows for explicit http > > request passing without relying on thread locals. > > > > So much for the current thoughts and plans. If you have any > > thoughts/comments, please shoot. > > > > best, > > holger > > > > -- > > You received this message because you are subscribed to the Google Groups "devpi-dev" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to devpi-dev+...(a)googlegroups.com. > > To post to this group, send email to devp...(a)googlegroups.com. > > Visit this group at http://groups.google.com/group/devpi-dev. > > For more options, visit https://groups.google.com/d/optout. >

1 0

devpi-dev May 2014