I haven't looked into yas3fs. Currently devpi by default uses sqlite. Everything that applies to sqlite in regard to file system locking and concurrent read/writes applies to devpi as well. In addition it writes the files to the file system directly. If yas3fs can't promise the necessary locking, then devpi will not work. There is the possibility to write your own storage backend. We have devpi-postgresql for example. But you already need some kind of global identifier, in this case the S3 storage for yas3fs, so I'd say it would overall be easier to have a central devpi instance where the cloud instances upload the packages to. You could even create the authentication token which is valid for 10 hours in advance when setting up the cloud instance, or you could add a custom authentication plugin if necessary. Regards, Florian Schulze On 11 May 2016, at 23:32, Russ Ferriday wrote:

Hi devpi experts,

I've been trying to use yas3fs under devpi. Why? Because I've been beguiled by Codeship's new Docker CI/CD and the option to run up test instances automatically, test a github commit, and register package updates with devpi on success. All of this needs to run on cloud services, not on a VPN setting, so having transient devpi services connected to a global data file makes some sense.

I've made a devpi service that puts S3 bucket under it <https://github.com/topiaruss/docker-devpi>self. this is based on some work by bubenkoff <https://github.com/topiaruss/docker-devpi/commit/223dc282d9a254f2ba17ed00b7a...> .

In my codeship configuration, I run the devpi service just to pull/push packages via devpi to my S3 bucket, then the container is discarded, leaving the packages in S3 for future tests and deployment.

It partially works, at the moment, so that after I upload a package I can list it. The next instantiation of the image connects to the bucket, but never lists existing packages. I see the bucket filling with packages, but devpi list does not list them.

I found that I needed to use the recheck flag, or sometimes only a few of the devpi files would be written. I'm guessing that the process closed before all the locally cached files were purged through yas3fs.

yas3fs --recheck-s3 s3://$S3_PATH /mnt

I'm going to play more with the yas3fs settings.

Has anyone played with this? Had success? Know any immediate gotchas?

--r.

-- You received this message because you are subscribed to the Google Groups "devpi-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to devpi-dev+...@googlegroups.com. To post to this group, send email to devp...@googlegroups.com. Visit this group at https://groups.google.com/group/devpi-dev. For more options, visit https://groups.google.com/d/optout.