[Distutils] Serverside Dependency Resolution and Virtualenv Build Server

[Thomas Güttler]

I wrote down a tought about Serverside Dependency Resolution and Virtualenv Build Server


What do you think?

Latest version: https://github.com/guettli/virtualenv-build-server

virtualenv-build-server
#######################

Rough roadmap how a server to build virtualenvs for the python programming language could be implemented.

Highlevel goal
--------------

Make creating new virtual envionments for the Python programming language easy and fast. 

Input: fuzzy requirements like this: django>=1.8, requests=>2.7

Output: virtualenv with packages installed.

Two APIs
------------

 #. Resolve fuzzy requirements to a fixed set of packages with exactly pinned versions.
 #. Read fixed set of packages. Build virtualenv according to given platform.


Steps
-----

Steps:

#. Client sends list of fuzzy requirements to server: 

   * I need: django>=1.8, requests=>2.7, ...

 
#. Server solves the fuzzy requirements to a fixed set of requirememnts: django==1.8.2, requests==2.8.1, ...

#. Client reads the fixed set of requirements.

#. Optional: Client sends fixed set of requirements to the server. Telling him the plattform

   * My platform: sys.version==2.7.6 and sys.platform=linux2

#. Server builds a virtualenv according to the fixed set of requirements.

#. Server sends the environment to the client

#. Client unpacks the data and has a usable virtualenv

Benefits
--------

Speed: 

* There is only one round-trip from client to server. If the dependencies get resolved on the client the client would need to download the available version information.
* Caching: If the server gets input parameters (fuzzy requirements and platform information) which he has seen before, he can return the cached result from the previous request.

Possible Implementations
------------------------

APIs
====
Both APIs could be implementated by a webservice/Rest interface passing json or yaml.

Serverside
==========


Implementation Strategie "PostgreSQL"
.....................................

Since the API is de-coupled from the internals the implementation could be exchanged without the need for changes at the client side.

I suggest using the PostgreSQL und resolving the dependcy graph using SQL (WITH RECURSIVE).

The package and version data gets stored in PostgreSQL via ORM (Django or SQLAlchemy).

The version numbers need to be normalized to ascii to allow fast comparision.

Related: https://www.python.org/dev/peps/pep-0440/

Implementation Strategie "Node.js"
..................................

I like python, but I am not married with it. Why not use a different tools that is already working? Maybe the node package manager: https://www.npmjs.com/

Questions
---------
Are virtualenv relocatable? AFAIK they are not.

General Thoughts
----------------

 * Ignore Updates. Focus on creating new virtualenvs. The server can do caching and that's why I prefer creating virtualenvs which never get updated. They get created and removed (immutable).
 

I won't implement it
--------------------

This idea is in the public domain. If you are young and brave or old and wise: Go ahead, try to implement it. Please communicate early and often. Ask on mailing-lists or me for feedback. Good luck :-)

I love feedback
---------------

Please tell me want you like or dislike:

 * typos and spelling stuff (I am not a native speaker)
 * alternative implementation strategies.
 * existing software which does this (even if implemented in a different programming language).
 * ...
 
-- 
http://www.thomas-guettler.de/