[Distutils] Warehouse Migration Plan

[Donald Stufft]

On Jul 31, 2013, at 8:11 AM, holger krekel <holger at merlinux.eu> wrote:

> On Wed, Jul 31, 2013 at 00:15 -0400, Donald Stufft wrote:
>> So, in the spirit of not treating distutils-sig like an adversary, here's
>> the main thing I've been working on lately with regards to PyPI. None
>> of this is set in stone but this is the general gist of the plan for moving
>> things to be developed in a modern framework, as well as cleaning
>> up the code and getting repeatable deployments.
> 
> Is warehouse a re-implementation or did it start from the current code base?

Reimplementation.

> 
>> Warehouse Migration Plan
>> ------------------------------------
>> 
>> Warehouse is currently primarily besides modeling for user accounts. It
>> will be deployed alongside pypi-legacy at next.pypi.python.org in the near
>> future. Initially it will have zero user facing value.

Just an update here, the name was changed to preview.pypi.python.org to
follow suite with preview.python.org and to prevent any sort of confusion
with last.pypi.python.org from the mirroring protocol.

>> 
>> As time goes on the database tables will be migrated from being "owned"
>> by pypi-legacy to being "owned" by Warehouse. This primarily means that
>> the schema definition and migration of those tables will be controlled by
>> Warehouse. As tables are migrated to Warehouse ownership the PyPI code
>> will be updated to reflect any changes in schema (without modification to
>> what end users see).
> 
> Am a bit skeptical if sharing databases is a good approach.
> Certainly has potential for disrupting pypi.python.org and making
> it harder for next.

Sharing the database is has two major purposes, first it allows Warehouse to
be phased in having people test it with real live data while PyPI itself is largely
unaffected. Second it prevents the need to have a big major switch point which
will largely be a one way switch with no easy way to revert it.

Essentially it's designed to strangle pypi-legacy slowly over time in smaller
bite sized chunks instead of needing to do everything at once.

> 
>> Once all tables that are going to be kept have been migrated, we will have
>> a shared database that is accessible from both pypi-legacy and Warehouse.
>> From this point Warehouse will begin evolving "legacy" views such as the
>> simple and other pieces of API. The UX itself will continue to be ignored and
>> focus will be on getting feature parity for automated tooling.
>> 
>> Changes in behavior on these legacy views should be minimal and
>> discussed on distutils-sig.
> 
> Having a doc/spec of those interactions would indeed help and contribute
> to "not defined by implementation" as you state below.
> 
>> Once the legacy views are finished people will be encouraged to test their
>> real world workloads against those reimplemented legacy APIs. As changes
>> in behaviors, missing functionality, or bugs are found they will be rectified or
>> declared unsupported.
>> 
>> During this time work on the UI of Warehouse will begin focusing on maintaing
>> feature parity but with no promises of no changes to the url structure or UX.
>> 
>> Once Warehouse gains feature parity with PyPI and has gotten enough testing
>> against it's APIs then pypi-legacy will be retired and Warehouse will move from
>> next.pypi.python.org to pypi.python.org. From there it will evolve on it's own without
>> needing to keep pypi-legacy in sync.
>> 
>> Specification & Acceptance Testing
>> ------------------------------------------------
>> 
>> I do not want a packaging index server to be specified by implementation, so as
>> the legacy API is ported over to Warehouse a specification will be drafted. This
>> spec will represent the "promise" that PyPI makes about the API it presents to be
>> consumed by machines. During the migration any behavior not codified inside of
>> the spec is considered implementation defined behavior and backwards compatibility
>> for that behavior will not be promised.
>> 
>> In addition to a defined specification A repository of acceptance tests will be developed.
>> These tests will be part of the test framework for any future changes to Warehouse
>> but will be maintained separately alongside the specification. They will also allow
>> any alternative implementation (such as DevPI) to test themselves against the spec.
> 
> I'd be happy to discuss if we can collaborate or even merge some of our 
> efforts.

Sure.

> 
> cheers,
> holger
> 
>> 
>> -----------------
>> Donald Stufft
>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>> 
> 
> 
> 
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> http://mail.python.org/mailman/listinfo/distutils-sig
> 


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130731/141fb352/attachment.pgp>