* It is [currently [#PEP426JSONLD)] necessary to run setup.py with each given destination platform, because parameters are expanded within the scope of setup.py.
  * Because of this, client side dependency resolution (with a given platform) is currently the only viable option for something like this


* Build: Docker, Tox (Dox) to build package(s)
  * Each assembly of packages is / could be a package with a setup.py (and/or a requirements.txt)
     * And tests:
       * http://conda.pydata.org/docs/building/meta-yaml.html#test-section
* Release: DevPi
   * http://doc.devpi.net/latest/
* conda env environment.yml YAML: http://conda.pydata.org/docs/using/envs.html
  * [x] conda packages
  * [x] pip packages
  * [ ] system packages (configuration management)

And then, really, Is there a stored version of this instance of a named Docker image?
#reproducibility #linkedreproducibility

On Sat, Nov 7, 2015 at 8:37 AM, Thomas Güttler <guettliml@thomas-guettler.de> wrote:
I wrote down a tought about Serverside Dependency Resolution and Virtualenv Build Server

What do you think?

Latest version: https://github.com/guettli/virtualenv-build-server


Rough roadmap how a server to build virtualenvs for the python programming language could be implemented.

Highlevel goal

Make creating new virtual envionments for the Python programming language easy and fast.

Input: fuzzy requirements like this: django>=1.8, requests=>2.7

Output: virtualenv with packages installed.

Two APIs

 #. Resolve fuzzy requirements to a fixed set of packages with exactly pinned versions.
 #. Read fixed set of packages. Build virtualenv according to given platform.



#. Client sends list of fuzzy requirements to server:

   * I need: django>=1.8, requests=>2.7, ...

#. Server solves the fuzzy requirements to a fixed set of requirememnts: django==1.8.2, requests==2.8.1, ...

#. Client reads the fixed set of requirements.

#. Optional: Client sends fixed set of requirements to the server. Telling him the plattform

   * My platform: sys.version==2.7.6 and sys.platform=linux2

#. Server builds a virtualenv according to the fixed set of requirements.

#. Server sends the environment to the client

#. Client unpacks the data and has a usable virtualenv



* There is only one round-trip from client to server. If the dependencies get resolved on the client the client would need to download the available version information.
* Caching: If the server gets input parameters (fuzzy requirements and platform information) which he has seen before, he can return the cached result from the previous request.

Possible Implementations

Both APIs could be implementated by a webservice/Rest interface passing json or yaml.


Implementation Strategie "PostgreSQL"

Since the API is de-coupled from the internals the implementation could be exchanged without the need for changes at the client side.

I suggest using the PostgreSQL und resolving the dependcy graph using SQL (WITH RECURSIVE).

The package and version data gets stored in PostgreSQL via ORM (Django or SQLAlchemy).

The version numbers need to be normalized to ascii to allow fast comparision.

Related: https://www.python.org/dev/peps/pep-0440/

Implementation Strategie "Node.js"

I like python, but I am not married with it. Why not use a different tools that is already working? Maybe the node package manager: https://www.npmjs.com/

Are virtualenv relocatable? AFAIK they are not.

General Thoughts

 * Ignore Updates. Focus on creating new virtualenvs. The server can do caching and that's why I prefer creating virtualenvs which never get updated. They get created and removed (immutable).

I won't implement it

This idea is in the public domain. If you are young and brave or old and wise: Go ahead, try to implement it. Please communicate early and often. Ask on mailing-lists or me for feedback. Good luck :-)

I love feedback

Please tell me want you like or dislike:

 * typos and spelling stuff (I am not a native speaker)
 * alternative implementation strategies.
 * existing software which does this (even if implemented in a different programming language).
 * ...

Distutils-SIG maillist  -  Distutils-SIG@python.org