On Dec 15, 2016, at 8:18 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 16 December 2016 at 07:14, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Dec 15, 2016, at 6:39 AM, Donald Stufft <donald@stufft.io> wrote:
Theoretically we could allow people to not just select packages, but also package specifiers for their “curated package set”, so instead of saying “requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we really wanted to get slick we could even provide a requirements.txt file format, and have people able to install the entire set by doing something like:


Can't people already do this by publishing a package that just depends on their whole 'package set'?

Technically, sure, but it adds a lot of overhead. The closest equivalent right now would be maintaining a source control repo somewhere with various requirements files in it.

However, at an ecosystem level, that doesn't have the same user experience impact. The idea of building this into PyPI itself would be to *reshape the learning curve of how people learn about dependency management as they're introduced to Python*.

Going back to the CurseGaming example, I actually use the free version of their client to manage the Warcraft addons on my gaming PC. The basic usage model is really simple and (not coincidentally) very similar to the way the Add-on manager works in Firefox and other GUI apps with integrated plugin managers:

- you have an "Installed" tab for the addons you have installed
- when you start the client, it checks for updates for all your installed addons and the out of date ones gain an "Update" button
- there's a separate tab where you can search all the available addons and install new ones

I've never used any of Curse's other clients (like the Minecraft or Kerbal Space Program ones), but I assume they operate in a similar way.

The paid tier of the Curse Client, and the account sync feature of Firefox, then offer the ability to synchronize your installed addons across machines. (There are also a lot of similarities between this model and the way mobile app stores work)

A comparable UX for Python/PyPI/pip would focus less on the library-and-application development cases (where the presence of source control is assumed), and more on the ad hoc scripting and learning-to-program use cases, where you're typically more interested in "--user" installations and the question of which parts of the Python ecosystem are just an import away than you are in reproducability and maintainability.

The ecosystem level learning curve then becomes:

- did you know you can back up your list of user installed packages to PyPI?
- did you know you can use PyPI to sync your user installs between systems?
- did you know you can use PyPI to categorise your user installs and share them with others?
- OK, now it's time to start learning about version control, virtual environments and automated testing

It wouldn't necessarily make sense to bake this *directly* into Warehouse, and the Mozilla folks responsible for Firefox Sync could no doubt offer real word guidance on the infrastructure and support needed to operate a service like that at scale, but the core concept of allowing package management to be introduced independently of both version control and virtual environments sounds potentially valuable to me.

Yeah, I think that this focus on curating packages on PyPI is reminiscent about the old yarn about looking for lost keys under the streetlight because it's dark everywhere else.  We're all familiar with web services and data formats, so we want to somehow have a data format or a web service be the answer to this problem.  But I don't believe that's where the problem is.

("this problem" being "let's make it easy and fun to a) bootstrap a common Python experimentation environment across multiple machines and b) _know that you have to do that_")

At the beginning of your story you mentioned the GUI client - that is the missing piece ;).  I've been saying for years that we need a Python.app that lets you easily bootstrap all this stuff: walk you through installing C dev tools if your packages need them, present a GUI search interface to finding packages, present a normal "file->open" dialog for selecting a location for a new virtualenv, automatically pop open a terminal, launching a Jupyter notebook whose kernel is pointed at said environment...

User-curated package sets strikes me as the _lowest_ priority feature out of all of those, if we are ordering by priority to deliver a good user experience.  I know "steam curators" have been brought up before - but we're talking about adding curators (one of my least favorite features of Steam, for what it's worth) before we've added "install game" ;-).

Users might even figure out this sort of stuff for themselves if they are given a discoverable API for things like search and installation of packages.

-glyph