So, in the spirit of not treating distutils-sig like an adversary, here's the main thing I've been working on lately with regards to PyPI. None of this is set in stone but this is the general gist of the plan for moving things to be developed in a modern framework, as well as cleaning up the code and getting repeatable deployments. Warehouse Migration Plan ------------------------------------ Warehouse is currently primarily besides modeling for user accounts. It will be deployed alongside pypi-legacy at next.pypi.python.org in the near future. Initially it will have zero user facing value. As time goes on the database tables will be migrated from being "owned" by pypi-legacy to being "owned" by Warehouse. This primarily means that the schema definition and migration of those tables will be controlled by Warehouse. As tables are migrated to Warehouse ownership the PyPI code will be updated to reflect any changes in schema (without modification to what end users see). Once all tables that are going to be kept have been migrated, we will have a shared database that is accessible from both pypi-legacy and Warehouse. From this point Warehouse will begin evolving "legacy" views such as the simple and other pieces of API. The UX itself will continue to be ignored and focus will be on getting feature parity for automated tooling. Changes in behavior on these legacy views should be minimal and discussed on distutils-sig. Once the legacy views are finished people will be encouraged to test their real world workloads against those reimplemented legacy APIs. As changes in behaviors, missing functionality, or bugs are found they will be rectified or declared unsupported. During this time work on the UI of Warehouse will begin focusing on maintaing feature parity but with no promises of no changes to the url structure or UX. Once Warehouse gains feature parity with PyPI and has gotten enough testing against it's APIs then pypi-legacy will be retired and Warehouse will move from next.pypi.python.org to pypi.python.org. From there it will evolve on it's own without needing to keep pypi-legacy in sync. Specification & Acceptance Testing ------------------------------------------------ I do not want a packaging index server to be specified by implementation, so as the legacy API is ported over to Warehouse a specification will be drafted. This spec will represent the "promise" that PyPI makes about the API it presents to be consumed by machines. During the migration any behavior not codified inside of the spec is considered implementation defined behavior and backwards compatibility for that behavior will not be promised. In addition to a defined specification A repository of acceptance tests will be developed. These tests will be part of the test framework for any future changes to Warehouse but will be maintained separately alongside the specification. They will also allow any alternative implementation (such as DevPI) to test themselves against the spec. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 31 July 2013 05:15, Donald Stufft <donald@stufft.io> wrote: [...] +1. This sounds like a good plan I do not want a packaging index server to be specified by implementation,
so as the legacy API is ported over to Warehouse a specification will be drafted. This spec will represent the "promise" that PyPI makes about the API it presents to be consumed by machines. During the migration any behavior not codified inside of the spec is considered implementation defined behavior and backwards compatibility for that behavior will not be promised.
And this is something I wholeheartedly support - having a properly specified PyPI API will not only help protect against future changes, it will also ensure that people writing their own index servers have a well-defined spec to work to. Paul
On Wed, Jul 31, 2013 at 00:15 -0400, Donald Stufft wrote:
So, in the spirit of not treating distutils-sig like an adversary, here's the main thing I've been working on lately with regards to PyPI. None of this is set in stone but this is the general gist of the plan for moving things to be developed in a modern framework, as well as cleaning up the code and getting repeatable deployments.
Is warehouse a re-implementation or did it start from the current code base?
Warehouse Migration Plan ------------------------------------
Warehouse is currently primarily besides modeling for user accounts. It will be deployed alongside pypi-legacy at next.pypi.python.org in the near future. Initially it will have zero user facing value.
As time goes on the database tables will be migrated from being "owned" by pypi-legacy to being "owned" by Warehouse. This primarily means that the schema definition and migration of those tables will be controlled by Warehouse. As tables are migrated to Warehouse ownership the PyPI code will be updated to reflect any changes in schema (without modification to what end users see).
Am a bit skeptical if sharing databases is a good approach. Certainly has potential for disrupting pypi.python.org and making it harder for next.
Once all tables that are going to be kept have been migrated, we will have a shared database that is accessible from both pypi-legacy and Warehouse. From this point Warehouse will begin evolving "legacy" views such as the simple and other pieces of API. The UX itself will continue to be ignored and focus will be on getting feature parity for automated tooling.
Changes in behavior on these legacy views should be minimal and discussed on distutils-sig.
Having a doc/spec of those interactions would indeed help and contribute to "not defined by implementation" as you state below.
Once the legacy views are finished people will be encouraged to test their real world workloads against those reimplemented legacy APIs. As changes in behaviors, missing functionality, or bugs are found they will be rectified or declared unsupported.
During this time work on the UI of Warehouse will begin focusing on maintaing feature parity but with no promises of no changes to the url structure or UX.
Once Warehouse gains feature parity with PyPI and has gotten enough testing against it's APIs then pypi-legacy will be retired and Warehouse will move from next.pypi.python.org to pypi.python.org. From there it will evolve on it's own without needing to keep pypi-legacy in sync.
Specification & Acceptance Testing ------------------------------------------------
I do not want a packaging index server to be specified by implementation, so as the legacy API is ported over to Warehouse a specification will be drafted. This spec will represent the "promise" that PyPI makes about the API it presents to be consumed by machines. During the migration any behavior not codified inside of the spec is considered implementation defined behavior and backwards compatibility for that behavior will not be promised.
In addition to a defined specification A repository of acceptance tests will be developed. These tests will be part of the test framework for any future changes to Warehouse but will be maintained separately alongside the specification. They will also allow any alternative implementation (such as DevPI) to test themselves against the spec.
I'd be happy to discuss if we can collaborate or even merge some of our efforts. cheers, holger
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On Jul 31, 2013, at 8:11 AM, holger krekel <holger@merlinux.eu> wrote:
On Wed, Jul 31, 2013 at 00:15 -0400, Donald Stufft wrote:
So, in the spirit of not treating distutils-sig like an adversary, here's the main thing I've been working on lately with regards to PyPI. None of this is set in stone but this is the general gist of the plan for moving things to be developed in a modern framework, as well as cleaning up the code and getting repeatable deployments.
Is warehouse a re-implementation or did it start from the current code base?
Reimplementation.
Warehouse Migration Plan ------------------------------------
Warehouse is currently primarily besides modeling for user accounts. It will be deployed alongside pypi-legacy at next.pypi.python.org in the near future. Initially it will have zero user facing value.
Just an update here, the name was changed to preview.pypi.python.org to follow suite with preview.python.org and to prevent any sort of confusion with last.pypi.python.org from the mirroring protocol.
As time goes on the database tables will be migrated from being "owned" by pypi-legacy to being "owned" by Warehouse. This primarily means that the schema definition and migration of those tables will be controlled by Warehouse. As tables are migrated to Warehouse ownership the PyPI code will be updated to reflect any changes in schema (without modification to what end users see).
Am a bit skeptical if sharing databases is a good approach. Certainly has potential for disrupting pypi.python.org and making it harder for next.
Sharing the database is has two major purposes, first it allows Warehouse to be phased in having people test it with real live data while PyPI itself is largely unaffected. Second it prevents the need to have a big major switch point which will largely be a one way switch with no easy way to revert it. Essentially it's designed to strangle pypi-legacy slowly over time in smaller bite sized chunks instead of needing to do everything at once.
Once all tables that are going to be kept have been migrated, we will have a shared database that is accessible from both pypi-legacy and Warehouse. From this point Warehouse will begin evolving "legacy" views such as the simple and other pieces of API. The UX itself will continue to be ignored and focus will be on getting feature parity for automated tooling.
Changes in behavior on these legacy views should be minimal and discussed on distutils-sig.
Having a doc/spec of those interactions would indeed help and contribute to "not defined by implementation" as you state below.
Once the legacy views are finished people will be encouraged to test their real world workloads against those reimplemented legacy APIs. As changes in behaviors, missing functionality, or bugs are found they will be rectified or declared unsupported.
During this time work on the UI of Warehouse will begin focusing on maintaing feature parity but with no promises of no changes to the url structure or UX.
Once Warehouse gains feature parity with PyPI and has gotten enough testing against it's APIs then pypi-legacy will be retired and Warehouse will move from next.pypi.python.org to pypi.python.org. From there it will evolve on it's own without needing to keep pypi-legacy in sync.
Specification & Acceptance Testing ------------------------------------------------
I do not want a packaging index server to be specified by implementation, so as the legacy API is ported over to Warehouse a specification will be drafted. This spec will represent the "promise" that PyPI makes about the API it presents to be consumed by machines. During the migration any behavior not codified inside of the spec is considered implementation defined behavior and backwards compatibility for that behavior will not be promised.
In addition to a defined specification A repository of acceptance tests will be developed. These tests will be part of the test framework for any future changes to Warehouse but will be maintained separately alongside the specification. They will also allow any alternative implementation (such as DevPI) to test themselves against the spec.
I'd be happy to discuss if we can collaborate or even merge some of our efforts.
Sure.
cheers, holger
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
participants (3)
-
Donald Stufft -
holger krekel -
Paul Moore