Re: [Distutils] Maintaining a curated set of Python packages
On Dec 15, 2016, at 9:35 AM, Steve Dower <steve.dower@python.org> wrote:
The "curated package sets" on PyPI idea sounds a bit like Steam's curator lists, which I like to think of as Twitter for game reviews. You can follow a curator to see their comments on particular games, and the most popular curators have their comments appear on the actual listings too.
Might be interesting to see how something like that worked for PyPI, though the initial investment is pretty high. (It doesn't solve the coherent bundle problem either, just the discovery of good libraries problem.)
Theoretically we could allow people to not just select packages, but also package specifiers for their “curated package set”, so instead of saying “requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we really wanted to get slick we could even provide a requirements.txt file format, and have people able to install the entire set by doing something like: $ pip install -r https://pypi.org/sets/dstufft/my-cool-set/requirements.txt — Donald Stufft
On Thursday, December 15, 2016, Donald Stufft <donald@stufft.io> wrote:
On Dec 15, 2016, at 9:35 AM, Steve Dower <steve.dower@python.org <javascript:_e(%7B%7D,'cvml','steve.dower@python.org');>> wrote:
The "curated package sets" on PyPI idea sounds a bit like Steam's curator lists, which I like to think of as Twitter for game reviews. You can follow a curator to see their comments on particular games, and the most popular curators have their comments appear on the actual listings too.
Might be interesting to see how something like that worked for PyPI, though the initial investment is pretty high. (It doesn't solve the coherent bundle problem either, just the discovery of good libraries problem.)
Theoretically we could allow people to not just select packages, but also package specifiers for their “curated package set”, so instead of saying “requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we really wanted to get slick we could even provide a requirements.txt file format, and have people able to install the entire set by doing something like:
$ pip install -r https://pypi.org/sets/dstufft/ my-cool-set/requirements.txt
With version control? $ pip install -r https://pypi.org/sets/dstufft/ my-cool-set/abcd123/requirements.txt <https://pypi.org/sets/dstufft/my-cool-set/requirements.txt> $ pip install -r https://pypi.org/sets/dstufft/ my-cool-set/v0.0.1/requirements.txt <https://pypi.org/sets/dstufft/my-cool-set/requirements.txt> This would be a graph. JSONLD? #PEP426JSONLD: - https://www.google.com/search?q=pep426jsonld - https://github.com/pypa/interoperability-peps/issues/31 With JSONLD, we could merge SoftwarePackage metadata with SoftwarePackageCollection metadata (just throwing some types out there). A http://schema.org/SoftwareApplication is a http://schema.org/CreativeWork . http://schema.org/softwareVersion
— Donald Stufft
On 16 December 2016 at 00:39, Donald Stufft <donald@stufft.io> wrote:
Theoretically we could allow people to not just select packages, but also package specifiers for their “curated package set”, so instead of saying “requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we really wanted to get slick we could even provide a requirements.txt file format, and have people able to install the entire set by doing something like:
$ pip install -r https://pypi.org/sets/dstufft/my-cool-set/requirements.txt
CurseGaming provide addon managers for a variety of game addons (Warcraft, Minecraft, etc), and the ability to define "AddOn Packs" is one of the ways they make it useful to have an account on the site even if you don't publish any addons of your own. Even if you don't make them public, you can still use them to sync your addon sets between different machines. In the context of Python, where I can see this kind of thing being potentially useful is for folks to manage package sets that aren't necessarily coupled to any specific project, but match the way they *personally* work. - "These are the packages I like to have installed to --user" - "These are the packages I use to start a CLI app" - "These are the packages I use to start a web app" - etc... It also provides a way for people to vote on projects that's a little more meaningful than stars - projects that appear in a lot of personal stack definitions are likely to be generally valuable (the closest equivalent to that today is mining code repositories like GitHub for requirements.txt files and seeing what people are using that way). So yeah, if folks interested in this were to add it to Warehouse (and hence pypi.org), I think it would definitely be a valuable enhancement to the overall ecosystem. "What needs to be implemented in order to be able to shut down the legacy service at pypi.python.org?" is the *PSF's* focus, but that doesn't mean it needs to be everyone's focus. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thursday, December 15, 2016, Wes Turner <wes.turner@gmail.com> wrote:
On Thursday, December 15, 2016, Donald Stufft <donald@stufft.io <javascript:_e(%7B%7D,'cvml','donald@stufft.io');>> wrote:
On Dec 15, 2016, at 9:35 AM, Steve Dower <steve.dower@python.org> wrote:
The "curated package sets" on PyPI idea sounds a bit like Steam's curator lists, which I like to think of as Twitter for game reviews. You can follow a curator to see their comments on particular games, and the most popular curators have their comments appear on the actual listings too.
Might be interesting to see how something like that worked for PyPI, though the initial investment is pretty high. (It doesn't solve the coherent bundle problem either, just the discovery of good libraries problem.)
Theoretically we could allow people to not just select packages, but also package specifiers for their “curated package set”, so instead of saying “requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we really wanted to get slick we could even provide a requirements.txt file format, and have people able to install the entire set by doing something like:
$ pip install -r https://pypi.org/sets/dstufft/ my-cool-set/requirements.txt
With version control?
$ pip install -r https://pypi.org/sets/dstufft/my-cool-set/abcd123/ requirements.txt <https://pypi.org/sets/dstufft/my-cool-set/requirements.txt>
$ pip install -r https://pypi.org/sets/dstufft/my-cool-set/v0.0.1/ requirements.txt <https://pypi.org/sets/dstufft/my-cool-set/requirements.txt>
This would be a graph. JSONLD? #PEP426JSONLD: - https://www.google.com/search?q=pep426jsonld - https://github.com/pypa/interoperability-peps/issues/31
With JSONLD, we could merge SoftwarePackage metadata with SoftwarePackageCollection metadata (just throwing some types out there).
A http://schema.org/SoftwareApplication is a http://schema.org/ CreativeWork .
- https://pypi.org/project/<name> as the canonical project http://schema.org/url - There's almost certainly a transform of TOML to JSONLD (" TOMLLD ") - There is a standardized transform of JSONLD to RDF - YAMLLD is a stricter subset of YAML (because just OrderedDicts
— Donald Stufft
Theoretically we could allow people to not just select packages, but also package specifiers for their “curated package set”, so instead of saying “requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we really wanted to get slick we could even provide a requirements.txt file format, and have people able to install the entire set by doing something like: $ pip install -r https://pypi.org/sets/dstufft/ my-cool-set/requirements.txt
What you want to be able to do is install a set of (abstract) packages based on a curated set of (concrete) packages. Doing that you just need a file with your concrete dependencies, and now we're back at pipfile and https://github.com/pypa/pipfile/issues/10#issuecomment-262229620 On Thu, Dec 15, 2016 at 4:10 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 16 December 2016 at 00:39, Donald Stufft <donald@stufft.io> wrote:
Theoretically we could allow people to not just select packages, but also package specifiers for their “curated package set”, so instead of saying “requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we really wanted to get slick we could even provide a requirements.txt file format, and have people able to install the entire set by doing something like:
$ pip install -r https://pypi.org/sets/dstufft/my-cool-set/requirements.txt
CurseGaming provide addon managers for a variety of game addons (Warcraft, Minecraft, etc), and the ability to define "AddOn Packs" is one of the ways they make it useful to have an account on the site even if you don't publish any addons of your own. Even if you don't make them public, you can still use them to sync your addon sets between different machines.
In the context of Python, where I can see this kind of thing being potentially useful is for folks to manage package sets that aren't necessarily coupled to any specific project, but match the way they *personally* work.
- "These are the packages I like to have installed to --user" - "These are the packages I use to start a CLI app" - "These are the packages I use to start a web app" - etc...
It also provides a way for people to vote on projects that's a little more meaningful than stars - projects that appear in a lot of personal stack definitions are likely to be generally valuable (the closest equivalent to that today is mining code repositories like GitHub for requirements.txt files and seeing what people are using that way).
So yeah, if folks interested in this were to add it to Warehouse (and hence pypi.org), I think it would definitely be a valuable enhancement to the overall ecosystem. "What needs to be implemented in order to be able to shut down the legacy service at pypi.python.org?" is the *PSF's* focus, but that doesn't mean it needs to be everyone's focus.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 16 December 2016 at 00:57, Wes Turner <wes.turner@gmail.com> wrote:
This would be a graph. JSONLD? #PEP426JSONLD: - https://www.google.com/search?q=pep426jsonld - https://github.com/pypa/interoperability-peps/issues/31
With JSONLD, we could merge SoftwarePackage metadata with SoftwarePackageCollection metadata (just throwing some types out there).
Wes, JSON-LD is a metasystem used for descriptive analytics across mixed datasets, which *isn't a problem we have*. We have full authority over the data formats we care about, and the user needs that matter to distutils-sig are: - publishers of Python packages - consumers of Python packages - maintainers of the toolchain It would *absolutely* make sense for Semantic Web folks to get involved in the libraries.io project (either directly, or by building a separate service backed by the libraries.io data set) and seek to produce a global set of semantically linked data that spans not only dependencies within language ecosystems, but also dependencies between them. It *doesn't* make sense for every single language ecosystem to come up with its own unique spin on how to incorporate software packages into semantic web models, nor does it make sense to try to warp the Python packaging user experience to better meet the needs of taxonomists of knowledge. This answer hasn't changed the last dozen times you've brought up JSON-LD. It isn't *going* to change. So please stop bringing it up. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thursday, December 15, 2016, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 16 December 2016 at 00:39, Donald Stufft <donald@stufft.io <javascript:;>> wrote:
Theoretically we could allow people to not just select packages, but also package specifiers for their “curated package set”, so instead of saying “requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we really wanted to get slick we could even provide a requirements.txt file format, and have people able to install the entire set by doing something like:
$ pip install -r https://pypi.org/sets/dstufft/my-cool-set/requirements.txt
CurseGaming provide addon managers for a variety of game addons (Warcraft, Minecraft, etc), and the ability to define "AddOn Packs" is one of the ways they make it useful to have an account on the site even if you don't publish any addons of your own. Even if you don't make them public, you can still use them to sync your addon sets between different machines.
In the context of Python, where I can see this kind of thing being potentially useful is for folks to manage package sets that aren't necessarily coupled to any specific project, but match the way they *personally* work.
- "These are the packages I like to have installed to --user" - "These are the packages I use to start a CLI app" - "These are the packages I use to start a web app" - etc...
Does a requirements.txt in a {git,} repo solve for this already? A Collection contains (hasPart) CreativeWorks - https://schema.org/Collection - https://schema.org/hasPart RDFa and JSONLD representations do parse as ordered lists. SoftwarePackageCollection SoftwareApplicationCollection
It also provides a way for people to vote on projects that's a little more meaningful than stars - projects that appear in a lot of personal stack definitions are likely to be generally valuable (the closest equivalent to that today is mining code repositories like GitHub for requirements.txt files and seeing what people are using that way).
https://schema.org/InteractionCounter > https://schema.org/UserLikes D: CreativeWork - https://schema.org/interactionCount is now - https://schema.org/interactionStatistic (These are write-heavy features: they would change the database load of Warehouse)
So yeah, if folks interested in this were to add it to Warehouse (and hence pypi.org), I think it would definitely be a valuable enhancement to the overall ecosystem. "What needs to be implemented in order to be able to shut down the legacy service at pypi.python.org?" is the *PSF's* focus, but that doesn't mean it needs to be everyone's focus.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com <javascript:;> | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org <javascript:;> https://mail.python.org/mailman/listinfo/distutils-sig
On Thursday, December 15, 2016, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 16 December 2016 at 00:57, Wes Turner <wes.turner@gmail.com <javascript:;>> wrote:
This would be a graph. JSONLD? #PEP426JSONLD: - https://www.google.com/search?q=pep426jsonld - https://github.com/pypa/interoperability-peps/issues/31
With JSONLD, we could merge SoftwarePackage metadata with SoftwarePackageCollection metadata (just throwing some types out there).
Wes, JSON-LD is a metasystem used for descriptive analytics across mixed datasets, which *isn't a problem we have*. We have full authority over the data formats we care about, and the user needs that matter to distutils-sig are:
- publishers of Python packages - consumers of Python packages - maintainers of the toolchain
It would *absolutely* make sense for Semantic Web folks to get involved in the libraries.io project (either directly, or by building a separate service backed by the libraries.io data set) and seek to produce a global set of semantically linked data that spans not only dependencies within language ecosystems, but also dependencies between them. It *doesn't* make sense for every single language ecosystem to come up with its own unique spin on how to incorporate software packages into semantic web models, nor does it make sense to try to warp the Python packaging user experience to better meet the needs of taxonomists of knowledge.
This answer hasn't changed the last dozen times you've brought up JSON-LD. It isn't *going* to change. So please stop bringing it up.
No, the problem is the same; and solving it (joining user-specific package metadata with central repository metadata on a common URI) with web standards is the best approach.
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com <javascript:;> | Brisbane, Australia
On Thursday, December 15, 2016, Wes Turner <wes.turner@gmail.com> wrote:
On Thursday, December 15, 2016, Nick Coghlan <ncoghlan@gmail.com <javascript:_e(%7B%7D,'cvml','ncoghlan@gmail.com');>> wrote:
On 16 December 2016 at 00:39, Donald Stufft <donald@stufft.io> wrote:
Theoretically we could allow people to not just select packages, but also package specifiers for their “curated package set”, so instead of saying “requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we really wanted to get slick we could even provide a requirements.txt file format, and have people able to install the entire set by doing something like:
$ pip install -r https://pypi.org/sets/dstufft/my-cool-set/requirements.txt
CurseGaming provide addon managers for a variety of game addons (Warcraft, Minecraft, etc), and the ability to define "AddOn Packs" is one of the ways they make it useful to have an account on the site even if you don't publish any addons of your own. Even if you don't make them public, you can still use them to sync your addon sets between different machines.
In the context of Python, where I can see this kind of thing being potentially useful is for folks to manage package sets that aren't necessarily coupled to any specific project, but match the way they *personally* work.
- "These are the packages I like to have installed to --user" - "These are the packages I use to start a CLI app" - "These are the packages I use to start a web app" - etc...
Does a requirements.txt in a {git,} repo solve for this already?
Could you just generate a README.rst for a long_description from requirements.txt (or requirements.in, or pipfile), store that in a {git,} repository, and use the existing pypi release versioning and upload machinery? Or would it be more useful to surface the other graph edges on project pages? - These additional queries would be less burdensome as AJAX requests to JSON REST API views. Project pages could continue to load without waiting for these additional cached edge bundles (and interactionStatistic(s)) to load: - "This SoftwarePackage is in the following n SoftwarePackageCollections, with the following comments and reviews" - "This SoftwarePackage has received n UserLikes (through Warehouse)"
A Collection contains (hasPart) CreativeWorks
- https://schema.org/Collection - https://schema.org/hasPart
RDFa and JSONLD representations do parse as ordered lists.
SoftwarePackageCollection SoftwareApplicationCollection
It also provides a way for people to vote on projects that's a little more meaningful than stars - projects that appear in a lot of personal stack definitions are likely to be generally valuable (the closest equivalent to that today is mining code repositories like GitHub for requirements.txt files and seeing what people are using that way).
https://schema.org/InteractionCounter > https://schema.org/UserLikes
D: CreativeWork - https://schema.org/interactionCount is now - https://schema.org/interactionStatistic
(These are write-heavy features: they would change the database load of Warehouse)
So yeah, if folks interested in this were to add it to Warehouse (and hence pypi.org), I think it would definitely be a valuable enhancement to the overall ecosystem. "What needs to be implemented in order to be able to shut down the legacy service at pypi.python.org?" is the *PSF's* focus, but that doesn't mean it needs to be everyone's focus.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 16 December 2016 at 01:38, Wes Turner <wes.turner@gmail.com> wrote:
On Thursday, December 15, 2016, Nick Coghlan <ncoghlan@gmail.com> wrote:
This answer hasn't changed the last dozen times you've brought up JSON-LD. It isn't *going* to change. So please stop bringing it up.
No, the problem is the same; and solving it (joining user-specific package metadata with central repository metadata on a common URI) with web standards is the best approach.
Then do what Donald did with crate.io: go develop your own PyPI competitor that uses your new and better approach to prove your point. As things stand, you're just generating noise and link spam on the list, and asking politely for you to stop it doesn't appear to be working Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 15 December 2016 at 15:58, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 16 December 2016 at 01:38, Wes Turner <wes.turner@gmail.com> wrote:
On Thursday, December 15, 2016, Nick Coghlan <ncoghlan@gmail.com> wrote:
This answer hasn't changed the last dozen times you've brought up JSON-LD. It isn't *going* to change. So please stop bringing it up.
No, the problem is the same; and solving it (joining user-specific package metadata with central repository metadata on a common URI) with web standards is the best approach.
Then do what Donald did with crate.io: go develop your own PyPI competitor that uses your new and better approach to prove your point.
As things stand, you're just generating noise and link spam on the list, and asking politely for you to stop it doesn't appear to be working
Just to add my POV, I also find your posts unhelpful, Wes. There's not enough information for me to evaluate what you say, and you offer no actual solutions to what's being discussed. As Nick says, if you can demonstrate what you're suggesting via a prototype implementation, that might help. But endless posts of links to standards documents that I have neither the time nor the inclination to read are not useful. Paul
On Thursday, December 15, 2016, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Thursday, December 15, 2016, Nick Coghlan <ncoghlan@gmail.com <javascript:;>> wrote:
This answer hasn't changed the last dozen times you've brought up JSON-LD. It isn't *going* to change. So please stop bringing it up.
No, the problem is the same; and solving it (joining user-specific
On 16 December 2016 at 01:38, Wes Turner <wes.turner@gmail.com <javascript:;>> wrote: package
metadata with central repository metadata on a common URI) with web standards is the best approach.
Then do what Donald did with crate.io: go develop your own PyPI competitor that uses your new and better approach to prove your point.
As things stand, you're just generating noise and link spam on the list, and asking politely for you to stop it doesn't appear to be working
I've intentionally included the URLs which support my position; none of which do I have any commercial interest in. This is not link spam, this is me explaining your problem to you in order to save time and money. There are your: - existing solutions that work today - types (web standard types) - use cases - server load engineering challenges (web apps must cache) Have a good day.
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com <javascript:;> | Brisbane, Australia
On Thursday, December 15, 2016, Paul Moore <p.f.moore@gmail.com> wrote:
On 16 December 2016 at 01:38, Wes Turner <wes.turner@gmail.com <javascript:;>> wrote:
On Thursday, December 15, 2016, Nick Coghlan <ncoghlan@gmail.com <javascript:;>> wrote:
This answer hasn't changed the last dozen times you've brought up JSON-LD. It isn't *going* to change. So please stop bringing it up.
No, the problem is the same; and solving it (joining user-specific
On 15 December 2016 at 15:58, Nick Coghlan <ncoghlan@gmail.com <javascript:;>> wrote: package
metadata with central repository metadata on a common URI) with web standards is the best approach.
Then do what Donald did with crate.io: go develop your own PyPI competitor that uses your new and better approach to prove your point.
As things stand, you're just generating noise and link spam on the list, and asking politely for you to stop it doesn't appear to be working
Just to add my POV, I also find your posts unhelpful, Wes. There's not enough information for me to evaluate what you say, and you offer no actual solutions to what's being discussed.
I could quote myself suggesting solutions in this thread, if you like?
As Nick says, if you can demonstrate what you're suggesting via a prototype implementation, that might help. But endless posts of links to standards documents that I have neither the time nor the inclination to read are not useful.
I'm suggesting that the solution here is to create version-controlled collections of resources with metadata. With schema.org types, those are Collection s of CreativeWork s. (Or, e.g. SoftwareApplicationCollection s of SoftwareApplication s) With Django, one would create a ListView generic view for the given resource type; and then define e.g. an api.py for DRF (Django REST Framework), and an index.py for Haystack (Elastic search). A Pyramid implementation would be similar; with additional tests and code just like Donald. [Links omitted because you can now copy/paste keywords into the search engines yourself; because you're unable to scroll past citations to find the content you're looking for.] DWBP and BP-LD would be useful reading for Warehouse API developers seeking to add additional functionality to support the aforementioned use cases with maximally-compatible linked open data. Again, Have a good day.
Paul
On 15 December 2016 at 19:13, Wes Turner <wes.turner@gmail.com> wrote:
Just to add my POV, I also find your posts unhelpful, Wes. There's not enough information for me to evaluate what you say, and you offer no actual solutions to what's being discussed.
I could quote myself suggesting solutions in this thread, if you like?
You offer lots of pointers to information. But that's different. Anyway, let's just agree to differ - I can skip your mails if they aren't helpful to me, and you don't need to bother about the fact that you're not getting your points across to me. Who knows, one day I might find the time to look into JSON-LD, at which point I may or may not understand why you think it's such a useful tool for solving all these problems (in spite of the fact that no-one else seems to think the same...) Paul
On Dec 15, 2016, at 6:39 AM, Donald Stufft <donald@stufft.io> wrote:
On Dec 15, 2016, at 9:35 AM, Steve Dower <steve.dower@python.org <mailto:steve.dower@python.org>> wrote:
The "curated package sets" on PyPI idea sounds a bit like Steam's curator lists, which I like to think of as Twitter for game reviews. You can follow a curator to see their comments on particular games, and the most popular curators have their comments appear on the actual listings too.
Might be interesting to see how something like that worked for PyPI, though the initial investment is pretty high. (It doesn't solve the coherent bundle problem either, just the discovery of good libraries problem.)
Theoretically we could allow people to not just select packages, but also package specifiers for their “curated package set”, so instead of saying “requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we really wanted to get slick we could even provide a requirements.txt file format, and have people able to install the entire set by doing something like:
$ pip install -r https://pypi.org/sets/dstufft/my-cool-set/requirements.txt <https://pypi.org/sets/dstufft/my-cool-set/requirements.txt>
Can't people already do this by publishing a package that just depends on their whole 'package set'? -glyph
On 16 December 2016 at 05:50, Paul Moore <p.f.moore@gmail.com> wrote:
On 15 December 2016 at 19:13, Wes Turner <wes.turner@gmail.com> wrote:
Just to add my POV, I also find your posts unhelpful, Wes. There's not enough information for me to evaluate what you say, and you offer no actual solutions to what's being discussed.
I could quote myself suggesting solutions in this thread, if you like?
You offer lots of pointers to information. But that's different.
Exactly. There are *lots* of information processing standards out there, and lots of things we *could* provide natively that simply aren't worth the hassle since folks that care can provide them as "after market addons" for the audiences that considers them relevant. For example, a few things that can matter to different audiences are: - SPDX (Software Package Data Exchange) identifiers for licenses - CPE (Common Product Enumeration) and SWID (Software Identification) tags for published software - DOI (Digital Object Identifier) tags for citation purposes - Common Criteria certification for software supply chains I don't push for these upstream in distutils-sig not because I don't think they're important in general, but because I *don't think they're a priority for distutils-sig*. If you're teaching Python to school students, or teaching engineers and scientists how to better analyse their own data, or building a web service for yourself or your employer, these kinds of things simply don't matter. The end users that care about them are well-positioned to tackle them on their own (or pay other organisations to do it for them), and because they span arbitrary publishing communities anyway, it doesn't really matter all that much if any given publishing community participates directly in the process (the only real beneficiaries are the intermediaries that actively blur the distinctions between the cooperative communities and the recalcitrant ones).
Anyway, let's just agree to differ - I can skip your mails if they aren't helpful to me, and you don't need to bother about the fact that you're not getting your points across to me.
I consider it fairly important that we have a reasonably common understanding of the target userbase for direct consumption of PyPI data, and what we expect to be supplied as third party services. It's also important that we have a shared understanding of how to constructively frame proposals for change. For the former, the Semantic Web, and folks that care about Semantic Web concepts like "Linked Data" in the abstract sense are not part of our primary audience. We don't go out of our way to make their lives difficult, but "it makes semantic analysis easier" also isn't a compelling rationale for change. For the latter, some variants of constructive proposals look like: - "this kind of user has this kind of problem and this proposed solution will help mitigate it this way (and, by the way, here's an existing standard we can use)" - "this feature exists in <third party tool or service>, it's really valuable to users for <these reasons>, how about we offer it by default?" - "I wrote <thing> for myself, and I think it would also help others for <these reasons>, can you help me make it more widely known and available?" They don't look like "Here's a bunch of technologies and organisations that exist on the internet that may in some way potentially be relevant to the management of a software distribution network", and nor does it look like "This data modeling standard exists, so we should use it, even though it doesn't actually simplify our lives or our users' lives in any way, and in fact makes them more complicated".
Who knows, one day I might find the time to look into JSON-LD, at which point I may or may not understand why you think it's such a useful tool for solving all these problems (in spite of the fact that no-one else seems to think the same...)
I *have* looked at JSON-LD (based primarily on Wes's original suggestions), both from the perspective of the Python packaging ecosystem specifically, as well as my day job working on software supply chain management. My verdict was that for managing a dependency graph implementation, it ends up in the category of technologies that qualify as "interesting, but not helpful". In many ways, it's the urllib2 of data linking - just as urllib2 gives you a URL handling framework which you can configure to handle HTTP rather than just providing a HTTP-specific interface the way requests does [1], JSON-LD gives you a data linking framework, which you can then use to define links between your data, rather than just linking the data directly in a domain-appropriate fashion. Using a framework for the sake of using a framework rather than out of a genuine engineering need doesn't tend to lead to good software systems. Wes seems to think that my perspective on this is born out of ignorance, so repeatedly bringing it up may make me change my point of view. However, our problems haven't changed, and the nature and purpose of JSON-LD haven't changed, so it really won't - the one thing that will change my mind is demonstrated popularity and utility of a service that integrates raw PyPI data with JSON-LD and schema.org. Hence my suggestions (made with varying degrees of politeness) to go build a dependency analysis service that extracts the dependency trees from libraries.io, maps them to schema.org concepts in a way that makes sense, and then demonstrate what that makes possible that can't be done with the libraries.io data directly. Neither Wes nor anyone else needs anyone's permission to go do that, and it will be far more enjoyable for all concerned than the status quo where Wes is refusing to take "No, we've already looked at JSON-LD, and we believe it adds needless complexity for no benefit that we care about" for an answer by continuing to post about it *here* rather than either venting his frustrations about our collective lack of interest somewhere else, or else channeling that frustration into building the system he wishes existed. Cheers, Nick. [1] http://www.curiousefficiency.org/posts/2016/08/what-problem-does-it-solve.ht... -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 16 December 2016 at 07:14, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Dec 15, 2016, at 6:39 AM, Donald Stufft <donald@stufft.io> wrote: Theoretically we could allow people to not just select packages, but also package specifiers for their “curated package set”, so instead of saying “requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we really wanted to get slick we could even provide a requirements.txt file format, and have people able to install the entire set by doing something like:
$ pip install -r https://pypi.org/sets/dstufft/ my-cool-set/requirements.txt
Can't people already do this by publishing a package that just depends on their whole 'package set'?
Technically, sure, but it adds a lot of overhead. The closest equivalent right now would be maintaining a source control repo somewhere with various requirements files in it. However, at an ecosystem level, that doesn't have the same user experience impact. The idea of building this into PyPI itself would be to *reshape the learning curve of how people learn about dependency management as they're introduced to Python*. Going back to the CurseGaming example, I actually use the free version of their client to manage the Warcraft addons on my gaming PC. The basic usage model is really simple and (not coincidentally) very similar to the way the Add-on manager works in Firefox and other GUI apps with integrated plugin managers: - you have an "Installed" tab for the addons you have installed - when you start the client, it checks for updates for all your installed addons and the out of date ones gain an "Update" button - there's a separate tab where you can search all the available addons and install new ones I've never used any of Curse's other clients (like the Minecraft or Kerbal Space Program ones), but I assume they operate in a similar way. The paid tier of the Curse Client, and the account sync feature of Firefox, then offer the ability to synchronize your installed addons across machines. (There are also a lot of similarities between this model and the way mobile app stores work) A comparable UX for Python/PyPI/pip would focus less on the library-and-application development cases (where the presence of source control is assumed), and more on the ad hoc scripting and learning-to-program use cases, where you're typically more interested in "--user" installations and the question of which parts of the Python ecosystem are just an import away than you are in reproducability and maintainability. The ecosystem level learning curve then becomes: - did you know you can back up your list of user installed packages to PyPI? - did you know you can use PyPI to sync your user installs between systems? - did you know you can use PyPI to categorise your user installs and share them with others? - OK, now it's time to start learning about version control, virtual environments and automated testing It wouldn't necessarily make sense to bake this *directly* into Warehouse, and the Mozilla folks responsible for Firefox Sync could no doubt offer real word guidance on the infrastructure and support needed to operate a service like that at scale, but the core concept of allowing package management to be introduced independently of both version control and virtual environments sounds potentially valuable to me. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Dec 15, 2016, at 8:18 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 16 December 2016 at 07:14, Glyph Lefkowitz <glyph@twistedmatrix.com <mailto:glyph@twistedmatrix.com>> wrote:
On Dec 15, 2016, at 6:39 AM, Donald Stufft <donald@stufft.io <mailto:donald@stufft.io>> wrote: Theoretically we could allow people to not just select packages, but also package specifiers for their “curated package set”, so instead of saying “requests”, you could say “requests~=2.12” or “requests==2.12.2”. If we really wanted to get slick we could even provide a requirements.txt file format, and have people able to install the entire set by doing something like:
$ pip install -r https://pypi.org/sets/dstufft/my-cool-set/requirements.txt <https://pypi.org/sets/dstufft/my-cool-set/requirements.txt> Can't people already do this by publishing a package that just depends on their whole 'package set'?
Technically, sure, but it adds a lot of overhead. The closest equivalent right now would be maintaining a source control repo somewhere with various requirements files in it.
However, at an ecosystem level, that doesn't have the same user experience impact. The idea of building this into PyPI itself would be to *reshape the learning curve of how people learn about dependency management as they're introduced to Python*.
Going back to the CurseGaming example, I actually use the free version of their client to manage the Warcraft addons on my gaming PC. The basic usage model is really simple and (not coincidentally) very similar to the way the Add-on manager works in Firefox and other GUI apps with integrated plugin managers:
- you have an "Installed" tab for the addons you have installed - when you start the client, it checks for updates for all your installed addons and the out of date ones gain an "Update" button - there's a separate tab where you can search all the available addons and install new ones
I've never used any of Curse's other clients (like the Minecraft or Kerbal Space Program ones), but I assume they operate in a similar way.
The paid tier of the Curse Client, and the account sync feature of Firefox, then offer the ability to synchronize your installed addons across machines. (There are also a lot of similarities between this model and the way mobile app stores work)
A comparable UX for Python/PyPI/pip would focus less on the library-and-application development cases (where the presence of source control is assumed), and more on the ad hoc scripting and learning-to-program use cases, where you're typically more interested in "--user" installations and the question of which parts of the Python ecosystem are just an import away than you are in reproducability and maintainability.
The ecosystem level learning curve then becomes:
- did you know you can back up your list of user installed packages to PyPI? - did you know you can use PyPI to sync your user installs between systems? - did you know you can use PyPI to categorise your user installs and share them with others? - OK, now it's time to start learning about version control, virtual environments and automated testing
It wouldn't necessarily make sense to bake this *directly* into Warehouse, and the Mozilla folks responsible for Firefox Sync could no doubt offer real word guidance on the infrastructure and support needed to operate a service like that at scale, but the core concept of allowing package management to be introduced independently of both version control and virtual environments sounds potentially valuable to me.
Yeah, I think that this focus on curating packages on PyPI is reminiscent about the old yarn about looking for lost keys under the streetlight because it's dark everywhere else. We're all familiar with web services and data formats, so we want to somehow have a data format or a web service be the answer to this problem. But I don't believe that's where the problem is. ("this problem" being "let's make it easy and fun to a) bootstrap a common Python experimentation environment across multiple machines and b) _know that you have to do that_") At the beginning of your story you mentioned the GUI client - that is the missing piece ;). I've been saying for years that we need a Python.app that lets you easily bootstrap all this stuff: walk you through installing C dev tools if your packages need them, present a GUI search interface to finding packages, present a normal "file->open" dialog for selecting a location for a new virtualenv, automatically pop open a terminal, launching a Jupyter notebook whose kernel is pointed at said environment... User-curated package sets strikes me as the _lowest_ priority feature out of all of those, if we are ordering by priority to deliver a good user experience. I know "steam curators" have been brought up before - but we're talking about adding curators (one of my least favorite features of Steam, for what it's worth) before we've added "install game" ;-). Users might even figure out this sort of stuff for themselves if they are given a discoverable API for things like search and installation of packages. -glyph
On Dec 15, 2016, at 11:29 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
User-curated package sets strikes me as the _lowest_ priority feature out of all of those
I don’t think anyone in the PyPA is planning on working on this currently. It was a possible idea that was spawned from this thread. However the nature of volunteer OSS is that volunteer time is not fungible and if someone feels particularly enthused about this idea they are free to pursue it. — Donald Stufft
On Dec 15, 2016, at 8:33 PM, Donald Stufft <donald@stufft.io> wrote:
On Dec 15, 2016, at 11:29 PM, Glyph Lefkowitz <glyph@twistedmatrix.com <mailto:glyph@twistedmatrix.com>> wrote:
User-curated package sets strikes me as the _lowest_ priority feature out of all of those
I don’t think anyone in the PyPA is planning on working on this currently. It was a possible idea that was spawned from this thread. However the nature of volunteer OSS is that volunteer time is not fungible and if someone feels particularly enthused about this idea they are free to pursue it.
I did very consciously choose my words there: "strikes me as", not "is" ;-) -glyph
On Thursday, December 15, 2016, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 16 December 2016 at 05:50, Paul Moore <p.f.moore@gmail.com <javascript:;>> wrote:
On 15 December 2016 at 19:13, Wes Turner <wes.turner@gmail.com <javascript:;>> wrote:
Just to add my POV, I also find your posts unhelpful, Wes. There's not enough information for me to evaluate what you say, and you offer no actual solutions to what's being discussed.
I could quote myself suggesting solutions in this thread, if you like?
You offer lots of pointers to information. But that's different.
Exactly. There are *lots* of information processing standards out there, and lots of things we *could* provide natively that simply aren't worth the hassle since folks that care can provide them as "after market addons" for the audiences that considers them relevant.
For example, a few things that can matter to different audiences are:
- SPDX (Software Package Data Exchange) identifiers for licenses - CPE (Common Product Enumeration) and SWID (Software Identification) tags for published software - DOI (Digital Object Identifier) tags for citation purposes - Common Criteria certification for software supply chains
These are called properties with RDFS. It takes very little effort to add additional properties. If the unqualified attribute is not listed in a JSONLD @context, it can still be added by specifying a URI
I don't push for these upstream in distutils-sig not because I don't think they're important in general, but because I *don't think they're a priority for distutils-sig*. If you're teaching Python to school students, or teaching engineers and scientists how to better analyse their own data, or building a web service for yourself or your employer, these kinds of things simply don't matter.
#31 lists a number of advantages. OTOMH, CVE security reports could be linked to the project/package URI (and thus displayed along with the project detail page)
The end users that care about them are well-positioned to tackle them on their own (or pay other organisations to do it for them), and because they span arbitrary publishing communities anyway, it doesn't really matter all that much if any given publishing community participates directly in the process (the only real beneficiaries are the intermediaries that actively blur the distinctions between the cooperative communities and the recalcitrant ones).
Linked Data minimizes
Anyway, let's just agree to differ - I can skip your mails if they aren't helpful to me, and you don't need to bother about the fact that you're not getting your points across to me.
I consider it fairly important that we have a reasonably common understanding of the target userbase for direct consumption of PyPI data, and what we expect to be supplied as third party services. It's also important that we have a shared understanding of how to constructively frame proposals for change.
When I can afford the time, I'll again take a look at fixing the metadata specification once and for all by (1) defining an @context for the existing metadata, and (2) producing an additional pydist.jsonld TODO metadata document (because the releases are currently keyed by version), and (3) adding the model attribute and view to Warehouse.
For the former, the Semantic Web, and folks that care about Semantic Web concepts like "Linked Data" in the abstract sense are not part of our primary audience. We don't go out of our way to make their lives difficult, but "it makes semantic analysis easier" also isn't a compelling rationale for change.
Unfortunately, you types are not well-versed in the problems that Linked Data solves: it's all your data in your schema in your database; and URIs are far less useful than RAM-local references (pointers). See: BP-LD
For the latter, some variants of constructive proposals look like:
- "this kind of user has this kind of problem and this proposed solution will help mitigate it this way (and, by the way, here's an existing standard we can use)" - "this feature exists in <third party tool or service>, it's really valuable to users for <these reasons>, how about we offer it by default?" - "I wrote <thing> for myself, and I think it would also help others for <these reasons>, can you help me make it more widely known and available?"
One could stuff additional metadata in # comments of a requirements.txt, but that would be an ad-hoc parsing scheme with a SPOF tool dependency.
They don't look like "Here's a bunch of technologies and organisations that exist on the internet that may in some way potentially be relevant to the management of a software distribution network", and nor does it look like "This data modeling standard exists, so we should use it, even though it doesn't actually simplify our lives or our users' lives in any way, and in fact makes them more complicated".
Those badges we all (!) add to our README.rst long_descriptions point to third-party services with lots of potentially structured linked data that is very relevant to curating a collection of resources: test coverage, build stats, discoverable documentation which could be searched en-masse, security vulnerability reports, downstream packages; but they're unfortunately just <a href> links; whereas they could be <a href property="URI"> edges that other tools could make use of.
Who knows, one day I might find the time to look into JSON-LD, at which point I may or may not understand why you think it's such a useful tool for solving all these problems (in spite of the fact that no-one else seems to think the same...)
It would be logically fallacious of me to, without an understanding of a web standard graph representation format, suggest that it's not sufficient (or ideally-suited) for these very use cases. #31 TODO somewhat laboriously lists ROI; Though I haven't yet had the time for an impact study.
I *have* looked at JSON-LD (based primarily on Wes's original suggestions), both from the perspective of the Python packaging ecosystem specifically, as well as my day job working on software supply chain management.
I recognize your expertise and your preference for given Linux distributions. I can tell you that, while many of the linked data examples describe social graph applications regarding Bob and Alice, there are very many domains where Linked Data is worth learning: medicine (research, clinical), open government data (where tool-dependence is a no-no and a lost opportunity). When you have data in lots of different datasets, it really starts to make sense to: - use URIs as keys - use URIs as column names - recognize that you're just reimplementing graph semantics which are already well-solved (RDF, RDFS, OWL, and now JSONLD because JS)
My verdict was that for managing a dependency graph implementation, it ends up in the category of technologies that qualify as "interesting, but not helpful". In many ways, it's the urllib2 of data linking - just as urllib2 gives you a URL handling framework which you can configure to handle HTTP rather than just providing a HTTP-specific interface the way requests does [1], JSON-LD gives you a data linking framework, which you can then use to define links between your data, rather than just linking the data directly in a domain-appropriate fashion. Using a framework for the sake of using a framework rather than out of a genuine engineering need doesn't tend to lead to good software systems.
Interesting analogy. urllib, urlparse, urllib2, urllib3, requests; and now we have cert hostname checking. SemWeb standards are also layered. There were other standards for triples in JSON that do still exist, but none could map an existing JSON document to RDF with such flexibility. Followed by a truism.
Wes seems to think that my perspective on this is born out of ignorance, so repeatedly bringing it up may make me change my point of view. However, our problems haven't changed, and the nature and purpose of JSON-LD haven't changed, so it really won't - the one thing that will change my mind is demonstrated popularity and utility of a service that integrates raw PyPI data with JSON-LD and schema.org.
Hence my suggestions (made with varying degrees of politeness) to go build a dependency analysis service that extracts the dependency trees from libraries.io, maps them to schema.org concepts in a way that makes sense, and then demonstrate what that makes possible that can't be done with the libraries.io data directly.
A chicken-and-egg problem, ironically. There are many proprietary solutions for aggregating software quality information; all of which must write parsers and JOIN logic for each packaging ecosystem's ad-hoc partial graph implementations. When the key of a thing is a URI, other datasets which reference the same URI just magically join together: The justifying metadata for a package in a curated collection could just join with the actual package metadata (and the aforementioned datasources)
Neither Wes nor anyone else needs anyone's permission to go do that, and it will be far more enjoyable for all concerned than the status quo where Wes is refusing to take "No, we've already looked at JSON-LD, and we believe it adds needless complexity for no benefit that we care about" for an answer by continuing to post about it *here* rather than either venting his frustrations about our collective lack of interest somewhere else, or else channeling that frustration into building the system he wishes existed.
If you've never written an @context for an existing JSON schema, I question both your assessment of complexity and your experience with sharing graph data with myriad applications; But that's irrelevant, Because here all I think I need is a table of dataset-local autoincrement IDs and some columns And ALTER TABLE migrations, And then someone else can write a parser for the schema I expose with my JSON REST API, So that I can JOIN this data with other useful datasets (In order to share a versioned Collection of CreativeWorks which already have URIs; Because I'm unsatisfied with requirements.txt Because it's line-based, And I can't just add additional attributes, And there's no key because indexes and editable URLs stuffed with checksums and eggwheel names, Oh and the JSON metadata specification is fixed and doesn't support URI attribute names So I can't just add additional attributes or values from a controlled vocabulary as-needed, Unless it's ReStructuredText-come-HTML (now with pypi:readme) Is that in PEP form? Someone should really put together an industry-group to produce some UML here, So our tools can talk, And JOIN on URIs With versions, platforms, and custom URI schemes, In order to curate a collection of packages, As a team, With group permissions, according to structured criteria (and comments!) Because "ld-signatures" A #LinkedMetaAnalyses (#LinkedReproducibility) application would similarly support curation of resources with URIs and defined criteria in order to elicit redundant expert evaluations of CreativeWorks (likely with JSONLD). In light of the reception here, that may be a better use of resources. I've said my piece, I'll leave you guys to find a solution to this use case which minimizes re-work and maximizes data integration potential. [Pip checksum docs, JSONLD] https://github.com/pypa/interoperability-peps/issues/31#issuecomment-1609701...
Cheers, Nick.
[1] http://www.curiousefficiency.org/posts/2016/08/what- problem-does-it-solve.html
-- Nick Coghlan | ncoghlan@gmail.com <javascript:;> | Brisbane, Australia
On 16 December 2016 at 14:29, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Dec 15, 2016, at 8:18 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
At the beginning of your story you mentioned the GUI client - *that* is the missing piece ;). I've been saying for years that we need a Python.app that lets you easily bootstrap all this stuff: walk you through installing C dev tools if your packages need them, present a GUI search interface to finding packages, present a normal "file->open" dialog for selecting a location for a new virtualenv, automatically pop open a terminal, launching a Jupyter notebook whose kernel is pointed at said environment...
It isn't really, as we started looking at this for IDLE, and the entire current UX is just fundamentally beginner hostile: - virtual environments are hard - requirements files are hard - knowing what packages are trustworthy and worth your time is hard - limiting students to a set of "known safe" packages is hard - components that assume command line use are hard They're especially hard if the only way to distribute a fix is to release an entire new edition of CPython rather than having IDLE talk to a (preferably configurable) backend cloud service for updated instructions. So there's a reason so many learning and even full development environments are moving online - they let the service provider deal with all the hassles of providing an appropriately configured environment, while the students can focus on learning how to code, and the developers can focus on defining their application logic. However, the reason I brought up the Curse and Firefox GUI examples was to emphasise the problems they hide from the default rich client experience: - their default focus is on managing one environment per device - they both may require environment restarts for changes to take effect - they both reference an at least somewhat moderated back end (by Curse in the Curse client case, by Mozilla in the Firefox case) - they both incorporate popularity metrics and addon ratings into the client experience Mobile app store clients also share those four characteristics (where "number of installations" and "star ratings" are critically important to search rankings, but gaming the latter is mitigated by hiding the "Write a review" feature if you haven't actually installed the app anywhere)
User-curated package sets strikes me as the _lowest_ priority feature out of all of those, if we are ordering by priority to deliver a good user experience. I know "steam curators" have been brought up before - but we're talking about adding curators (one of my least favorite features of Steam, for what it's worth) before we've added "install game" ;-).
In many educational contexts, adding "install game" without support for institutional curators of some kind is a complete non-starter (even if those curators are a collaborative community like a Linux distribution, there's still more accountability than software publishing sites like PyPI tend to provide).
Users might even figure out this sort of stuff for themselves if they are given a discoverable API for things like search and installation of packages.
That sounds a bit like agreement that we're still missing some of the backend pieces needed to make a beginner-friendly client really viable :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Dec 15, 2016, at 9:23 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 16 December 2016 at 14:29, Glyph Lefkowitz <glyph@twistedmatrix.com <mailto:glyph@twistedmatrix.com>> wrote:
On Dec 15, 2016, at 8:18 PM, Nick Coghlan <ncoghlan@gmail.com <mailto:ncoghlan@gmail.com>> wrote:
At the beginning of your story you mentioned the GUI client - that is the missing piece ;). I've been saying for years that we need a Python.app that lets you easily bootstrap all this stuff: walk you through installing C dev tools if your packages need them, present a GUI search interface to finding packages, present a normal "file->open" dialog for selecting a location for a new virtualenv, automatically pop open a terminal, launching a Jupyter notebook whose kernel is pointed at said environment...
It isn't really, as we started looking at this for IDLE, and the entire current UX is just fundamentally beginner hostile:
- virtual environments are hard - requirements files are hard - knowing what packages are trustworthy and worth your time is hard - limiting students to a set of "known safe" packages is hard - components that assume command line use are hard
They're especially hard if the only way to distribute a fix is to release an entire new edition of CPython rather than having IDLE talk to a (preferably configurable) backend cloud service for updated instructions.
So there's a reason so many learning and even full development environments are moving online - they let the service provider deal with all the hassles of providing an appropriately configured environment, while the students can focus on learning how to code, and the developers can focus on defining their application logic.
None of what you're saying is wrong here, so I don't want to disagree. But, I think this is just one perspective; i.e. moving to a cloud environment is one approach to providing a more circumscribed environment, but embracing endpoint sandboxing is another. For example, learning how to use Xcode is a fundamentally different (and easier!) sort of experience than learning the traditional UNIX development pipeline, due in large part to the fact that it provides a unified, discoverable interface. This is despite the fact that Xcode projects are actually substantially more complex than their UNIX-y equivalents, due to the high levels of coupling and complexity in the way that you have to interface with certain system services (signing with entitlements, bundle metadata, etc). You still have to retrieve many resources from the cloud - simulators, documentation, SDKs - but the UI tells you that you need those things, and straightforwardly automates the process of getting them. Everything else that goes into a development project is not "environment setup", but a part of the Xcode project itself. Similarly, version control (a git repository) is nearly implicitly a part of the project. It's tricky to even create one without a VCS backing it any more. Anyhow, Xcode is far from perfect - many of the places it touches the UNIX pipeline are extremely sharp edges you can easily impale yourself on (and don't get me started about codesigning) - but it nevertheless points at a different potential direction. For example; why expose the concept of a "virtual environment" directly at all? "New Project" could just create a requirements.txt and a setup.py for you, alongside a git repo and a virtualenv for that project. Or, the UI could be geared towards setting up a tox.ini rather than a virtualenv, and run everything through tox so it's in an isolated environment with defined requirements. This is a best practice anyway so why not make it easier to start early? This might all be way too much work, but I think it's important to remember it's possible.
However, the reason I brought up the Curse and Firefox GUI examples was to emphasise the problems they hide from the default rich client experience:
- their default focus is on managing one environment per device
In the analogous Python tool, one could replace "per device" with "per project" - and perhaps have a "default project" so something useful could happen even before you've decided what you're doing...
- they both may require environment restarts for changes to take effect
... one could just put a little blinking red light on any jupyter windows whose kernels need to be restarted :) ...
- they both reference an at least somewhat moderated back end (by Curse in the Curse client case, by Mozilla in the Firefox case) - they both incorporate popularity metrics and addon ratings into the client experience
I thought this thread was already interminable, I look forward to reading the never-ending rest of it now that you've raised the grim spectre of the PyPI user-ratings feature from the dead :).
Mobile app store clients also share those four characteristics (where "number of installations" and "star ratings" are critically important to search rankings, but gaming the latter is mitigated by hiding the "Write a review" feature if you haven't actually installed the app anywhere)
User-curated package sets strikes me as the _lowest_ priority feature out of all of those, if we are ordering by priority to deliver a good user experience. I know "steam curators" have been brought up before - but we're talking about adding curators (one of my least favorite features of Steam, for what it's worth) before we've added "install game" ;-).
In many educational contexts, adding "install game" without support for institutional curators of some kind is a complete non-starter (even if those curators are a collaborative community like a Linux distribution, there's still more accountability than software publishing sites like PyPI tend to provide).
I initially wanted to disagree when I read this, but I'm not actually sure what educational contexts you're talking about, and why "accountability" is important?
Users might even figure out this sort of stuff for themselves if they are given a discoverable API for things like search and installation of packages.
That sounds a bit like agreement that we're still missing some of the backend pieces needed to make a beginner-friendly client really viable :)
Oops. I meant to say "UI". "beginner" is a direction, and not a fixed position; many people more "beginner" than the current audience could be well-served by a discoverable initial project-creation and REPL UI. While I don't doubt that some backend pieces might help (although I still don't see how the one being discussed would), I also think that it would be very hard to say that the back-end is a limiting factor in UX improvement for the Python onboarding process; the front end could move quite a bit up the value chain without touching any of the various backends it would need to interact with. But of course, if I really wanted to make this point, I'd just write it; dstufft is certainly right that volunteer time is not fungible. If I'm lucky, I'll have the time to do that at some point, since my efforts to convince someone else that this is the high-value target have been failing for some years now ;). -glyph
On 16 December 2016 at 20:57, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
Anyhow, Xcode is far from perfect - many of the places it touches the UNIX pipeline are extremely sharp edges you can easily impale yourself on (and don't get me started about codesigning) - but it nevertheless points at a different potential direction. For example; why expose the concept of a "virtual environment" directly at all? "New Project" could just create a requirements.txt and a setup.py for you, alongside a git repo and a virtualenv for that project. Or, the UI could be geared towards setting up a tox.ini rather than a virtualenv, and run everything through tox so it's in an isolated environment with defined requirements. This is a best practice anyway so why not make it easier to start early?
This might all be way too much work, but I think it's important to remember it's possible.
Yeah, I think we agree more than we disagree here. The main thing is that one of the key ways newcomer-friendly environments make themselves more approachable is to *constrain choice*. XCode usability benefits from being Apple-centric. Ditto for Visual Studio and MS. Linux and Python, by contrast, were both born out of a DIY culture where folks being free to choose their own tools was initially perceived solely as a highly desirable feature, rather than as a potential barrier to entry for newcomers. That means there's an argument to be made that something like YHat's Rodeo [1] might be a better starting point for data analytics in Python than jumping straight to Jupyter Notebook, and it's also why the Mu editor [2] exists as a dedicated tool for folks learning Python by way of the micro:bit project. [1] http://rodeo.yhat.com/docs/ [2] http://codewith.mu/
However, the reason I brought up the Curse and Firefox GUI examples was to emphasise the problems they hide from the default rich client experience:
- their default focus is on managing one environment per device
In the analogous Python tool, one could replace "per device" with "per project" - and perhaps have a "default project" so something useful could happen even before you've decided what you're doing...
But we've immediately bumped the complexity level up in doing so, and it's a level of complexity that many people initially spending all of their development time on a single project may not need. I thought this thread was already interminable, I look forward to reading
the never-ending rest of it now that you've raised the grim spectre of the PyPI user-ratings feature from the dead :).
All the arguments against integrating user ratings into a service that's focused on lowering barriers to publication still hold, so I'm really just noting that that decision to create a friendlier publishing environment *does* introduce some additional constraints elsewhere in the distribution pipeline.
User-curated package sets strikes me as the _lowest_ priority feature out
of all of those, if we are ordering by priority to deliver a good user experience. I know "steam curators" have been brought up before - but we're talking about adding curators (one of my least favorite features of Steam, for what it's worth) before we've added "install game" ;-).
In many educational contexts, adding "install game" without support for institutional curators of some kind is a complete non-starter (even if those curators are a collaborative community like a Linux distribution, there's still more accountability than software publishing sites like PyPI tend to provide).
I initially wanted to disagree when I read this, but I'm not actually sure what educational contexts you're talking about, and why "accountability" is important?
Schools, mainly. Lots of administrators are still scared of the internet, so one of the attractions of things like Raspberry Pi is that the software updates come from Debian rather than directly from the software publishers. Sometimes you can get away with "What the bureaucracy doesn't know won't hurt it", but it's more convenient when teachers don't have to do that.
"beginner" is a direction, and not a fixed position; many people more "beginner" than the current audience could be well-served by a discoverable initial project-creation and REPL UI. While I don't doubt that some backend pieces might help (although I still don't see how the one being discussed would), I also think that it would be very hard to say that the back-end is a *limiting factor* in UX improvement for the Python onboarding process; the front end could move quite a bit up the value chain without touching any of the various backends it would need to interact with.
But of course, if I really wanted to make this point, I'd just write it; dstufft is certainly right that volunteer time is not fungible. If I'm lucky, I'll have the time to do that at some point, since my efforts to convince someone else that this is the high-value target have been failing for some years now ;).
I'll confess that one of my ulterior motives for encouraging computing teachers to engage more directly with the upstream Python community is that I kinda hope we'll eventually run into one that either decides none of the current editors are good enough and creates their own, or else decides that "create the introductory editor that you wish you had when you started learning to program" might make a good collaborative student project :) Alternatively, I've recently started using Visual Studio Code as my editor for work [1], and it seems likely that would be hackable enough for someone to create a plugin that bootstrapped a complete Python toolchain such that the bootstrapping flow became: 1. Install VSCode 2. Install the "New to Python" plugin 3. Run the plugin's "Setup Python Environment" command Cheers, Nick. [1] Oh the irony that the first cross-platform editor I've tried that I actually think looks nice and find pleasant to use on Fedora was released by Microsoft :) -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I'm also a visual studio code fan. It is the first editor I've tried that feels lightweight like Vim but has the power of many plugins. That, and the text rendering is excellent. https://pypi.python.org/pypi/Stallion is a lovely GUI package manager. One possibility to consider is that virtualenv itself is a bad idea. Why should the Python interpreter executable, rather than the program being run, determine the set of packages that is available for import? It is confusing and inconvenient to have to deal with environments at all. Yes, even if you are using a helper. Maybe there can be a better way to manage dependencies that is not completely disjoint from setup.py. On Fri, Dec 16, 2016 at 8:07 AM Nick Coghlan <ncoghlan@gmail.com> wrote:
On 16 December 2016 at 20:57, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
Anyhow, Xcode is far from perfect - many of the places it touches the UNIX pipeline are extremely sharp edges you can easily impale yourself on (and don't get me started about codesigning) - but it nevertheless points at a different potential direction. For example; why expose the concept of a "virtual environment" directly at all? "New Project" could just create a requirements.txt and a setup.py for you, alongside a git repo and a virtualenv for that project. Or, the UI could be geared towards setting up a tox.ini rather than a virtualenv, and run everything through tox so it's in an isolated environment with defined requirements. This is a best practice anyway so why not make it easier to start early?
This might all be way too much work, but I think it's important to remember it's possible.
Yeah, I think we agree more than we disagree here. The main thing is that one of the key ways newcomer-friendly environments make themselves more approachable is to *constrain choice*.
XCode usability benefits from being Apple-centric. Ditto for Visual Studio and MS.
Linux and Python, by contrast, were both born out of a DIY culture where folks being free to choose their own tools was initially perceived solely as a highly desirable feature, rather than as a potential barrier to entry for newcomers.
That means there's an argument to be made that something like YHat's Rodeo [1] might be a better starting point for data analytics in Python than jumping straight to Jupyter Notebook, and it's also why the Mu editor [2] exists as a dedicated tool for folks learning Python by way of the micro:bit project.
[1] http://rodeo.yhat.com/docs/ [2] http://codewith.mu/
However, the reason I brought up the Curse and Firefox GUI examples was to emphasise the problems they hide from the default rich client experience:
- their default focus is on managing one environment per device
In the analogous Python tool, one could replace "per device" with "per project" - and perhaps have a "default project" so something useful could happen even before you've decided what you're doing...
But we've immediately bumped the complexity level up in doing so, and it's a level of complexity that many people initially spending all of their development time on a single project may not need.
I thought this thread was already interminable, I look forward to reading the never-ending rest of it now that you've raised the grim spectre of the PyPI user-ratings feature from the dead :).
All the arguments against integrating user ratings into a service that's focused on lowering barriers to publication still hold, so I'm really just noting that that decision to create a friendlier publishing environment *does* introduce some additional constraints elsewhere in the distribution pipeline.
User-curated package sets strikes me as the _lowest_ priority feature out of all of those, if we are ordering by priority to deliver a good user experience. I know "steam curators" have been brought up before - but we're talking about adding curators (one of my least favorite features of Steam, for what it's worth) before we've added "install game" ;-).
In many educational contexts, adding "install game" without support for institutional curators of some kind is a complete non-starter (even if those curators are a collaborative community like a Linux distribution, there's still more accountability than software publishing sites like PyPI tend to provide).
I initially wanted to disagree when I read this, but I'm not actually sure what educational contexts you're talking about, and why "accountability" is important?
Schools, mainly. Lots of administrators are still scared of the internet, so one of the attractions of things like Raspberry Pi is that the software updates come from Debian rather than directly from the software publishers.
Sometimes you can get away with "What the bureaucracy doesn't know won't hurt it", but it's more convenient when teachers don't have to do that.
"beginner" is a direction, and not a fixed position; many people more "beginner" than the current audience could be well-served by a discoverable initial project-creation and REPL UI. While I don't doubt that some backend pieces might help (although I still don't see how the one being discussed would), I also think that it would be very hard to say that the back-end is a *limiting factor* in UX improvement for the Python onboarding process; the front end could move quite a bit up the value chain without touching any of the various backends it would need to interact with.
But of course, if I really wanted to make this point, I'd just write it; dstufft is certainly right that volunteer time is not fungible. If I'm lucky, I'll have the time to do that at some point, since my efforts to convince someone else that this is the high-value target have been failing for some years now ;).
I'll confess that one of my ulterior motives for encouraging computing teachers to engage more directly with the upstream Python community is that I kinda hope we'll eventually run into one that either decides none of the current editors are good enough and creates their own, or else decides that "create the introductory editor that you wish you had when you started learning to program" might make a good collaborative student project :)
Alternatively, I've recently started using Visual Studio Code as my editor for work [1], and it seems likely that would be hackable enough for someone to create a plugin that bootstrapped a complete Python toolchain such that the bootstrapping flow became:
1. Install VSCode 2. Install the "New to Python" plugin 3. Run the plugin's "Setup Python Environment" command
Cheers, Nick.
[1] Oh the irony that the first cross-platform editor I've tried that I actually think looks nice and find pleasant to use on Fedora was released by Microsoft :)
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
"Alternatively, I've recently started using Visual Studio Code as my editor for work ..." FWIW, the long-term Python story in VSCode is currently (largely) one of my responsibilities, and that bootstrapping flow is exactly one of the pieces I really want to put in. Unfortunately, nobody has let me have any engineers to implement it yet :( (mostly for big-company-politics reasons, rather than hiring trouble) Top-posted from my Windows Phone -----Original Message----- From: "Nick Coghlan" <ncoghlan@gmail.com> Sent: 12/16/2016 5:08 To: "Glyph Lefkowitz" <glyph@twistedmatrix.com> Cc: "Barry Warsaw" <barry@python.org>; "DistUtils mailing list" <Distutils-Sig@python.org> Subject: Re: [Distutils] Maintaining a curated set of Python packages On 16 December 2016 at 20:57, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote: Anyhow, Xcode is far from perfect - many of the places it touches the UNIX pipeline are extremely sharp edges you can easily impale yourself on (and don't get me started about codesigning) - but it nevertheless points at a different potential direction. For example; why expose the concept of a "virtual environment" directly at all? "New Project" could just create a requirements.txt and a setup.py for you, alongside a git repo and a virtualenv for that project. Or, the UI could be geared towards setting up a tox.ini rather than a virtualenv, and run everything through tox so it's in an isolated environment with defined requirements. This is a best practice anyway so why not make it easier to start early? This might all be way too much work, but I think it's important to remember it's possible. Yeah, I think we agree more than we disagree here. The main thing is that one of the key ways newcomer-friendly environments make themselves more approachable is to *constrain choice*. XCode usability benefits from being Apple-centric. Ditto for Visual Studio and MS. Linux and Python, by contrast, were both born out of a DIY culture where folks being free to choose their own tools was initially perceived solely as a highly desirable feature, rather than as a potential barrier to entry for newcomers. That means there's an argument to be made that something like YHat's Rodeo [1] might be a better starting point for data analytics in Python than jumping straight to Jupyter Notebook, and it's also why the Mu editor [2] exists as a dedicated tool for folks learning Python by way of the micro:bit project. [1] http://rodeo.yhat.com/docs/ [2] http://codewith.mu/ However, the reason I brought up the Curse and Firefox GUI examples was to emphasise the problems they hide from the default rich client experience: - their default focus is on managing one environment per device In the analogous Python tool, one could replace "per device" with "per project" - and perhaps have a "default project" so something useful could happen even before you've decided what you're doing... But we've immediately bumped the complexity level up in doing so, and it's a level of complexity that many people initially spending all of their development time on a single project may not need. I thought this thread was already interminable, I look forward to reading the never-ending rest of it now that you've raised the grim spectre of the PyPI user-ratings feature from the dead :). All the arguments against integrating user ratings into a service that's focused on lowering barriers to publication still hold, so I'm really just noting that that decision to create a friendlier publishing environment *does* introduce some additional constraints elsewhere in the distribution pipeline. User-curated package sets strikes me as the _lowest_ priority feature out of all of those, if we are ordering by priority to deliver a good user experience. I know "steam curators" have been brought up before - but we're talking about adding curators (one of my least favorite features of Steam, for what it's worth) before we've added "install game" ;-). In many educational contexts, adding "install game" without support for institutional curators of some kind is a complete non-starter (even if those curators are a collaborative community like a Linux distribution, there's still more accountability than software publishing sites like PyPI tend to provide). I initially wanted to disagree when I read this, but I'm not actually sure what educational contexts you're talking about, and why "accountability" is important? Schools, mainly. Lots of administrators are still scared of the internet, so one of the attractions of things like Raspberry Pi is that the software updates come from Debian rather than directly from the software publishers. Sometimes you can get away with "What the bureaucracy doesn't know won't hurt it", but it's more convenient when teachers don't have to do that. "beginner" is a direction, and not a fixed position; many people more "beginner" than the current audience could be well-served by a discoverable initial project-creation and REPL UI. While I don't doubt that some backend pieces might help (although I still don't see how the one being discussed would), I also think that it would be very hard to say that the back-end is a limiting factor in UX improvement for the Python onboarding process; the front end could move quite a bit up the value chain without touching any of the various backends it would need to interact with. But of course, if I really wanted to make this point, I'd just write it; dstufft is certainly right that volunteer time is not fungible. If I'm lucky, I'll have the time to do that at some point, since my efforts to convince someone else that this is the high-value target have been failing for some years now ;). I'll confess that one of my ulterior motives for encouraging computing teachers to engage more directly with the upstream Python community is that I kinda hope we'll eventually run into one that either decides none of the current editors are good enough and creates their own, or else decides that "create the introductory editor that you wish you had when you started learning to program" might make a good collaborative student project :) Alternatively, I've recently started using Visual Studio Code as my editor for work [1], and it seems likely that would be hackable enough for someone to create a plugin that bootstrapped a complete Python toolchain such that the bootstrapping flow became: 1. Install VSCode 2. Install the "New to Python" plugin 3. Run the plugin's "Setup Python Environment" command Cheers, Nick. [1] Oh the irony that the first cross-platform editor I've tried that I actually think looks nice and find pleasant to use on Fedora was released by Microsoft :) -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
If people are serious about trying to prototype this stuff then the easiest way might be coming up with shell scripts that do the prompting if it's faster to iterate that way than doing a full-blown GUI. Now that WIndows 10 has WSL/Bash it means for the first time all 3 major OSs have a common shell people can work from. You could even go as far as making the shell scripts be Cookiecutter templates such that people can experiment with things being included/left out (e.g. an instructor wants to require Python 3.5, no git, and have people work from a virtual environment and so they generate the shell script everyone is told to run to get things going/verify the student's system is set up properly). On Fri, 16 Dec 2016 at 05:52 Daniel Holth <dholth@gmail.com> wrote:
I'm also a visual studio code fan. It is the first editor I've tried that feels lightweight like Vim but has the power of many plugins. That, and the text rendering is excellent.
https://pypi.python.org/pypi/Stallion is a lovely GUI package manager.
One possibility to consider is that virtualenv itself is a bad idea. Why should the Python interpreter executable, rather than the program being run, determine the set of packages that is available for import? It is confusing and inconvenient to have to deal with environments at all. Yes, even if you are using a helper. Maybe there can be a better way to manage dependencies that is not completely disjoint from setup.py.
That just sounds like node_modules/ and I personally don't want to go down that route. If you view the interpreter as another component of an app then the disconnect doesn't seem so nutty (at least in my head; at that point it's just another /usr/local to me). -Brett
On Fri, Dec 16, 2016 at 8:07 AM Nick Coghlan <ncoghlan@gmail.com> wrote:
On 16 December 2016 at 20:57, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
Anyhow, Xcode is far from perfect - many of the places it touches the UNIX pipeline are extremely sharp edges you can easily impale yourself on (and don't get me started about codesigning) - but it nevertheless points at a different potential direction. For example; why expose the concept of a "virtual environment" directly at all? "New Project" could just create a requirements.txt and a setup.py for you, alongside a git repo and a virtualenv for that project. Or, the UI could be geared towards setting up a tox.ini rather than a virtualenv, and run everything through tox so it's in an isolated environment with defined requirements. This is a best practice anyway so why not make it easier to start early?
This might all be way too much work, but I think it's important to remember it's possible.
Yeah, I think we agree more than we disagree here. The main thing is that one of the key ways newcomer-friendly environments make themselves more approachable is to *constrain choice*.
XCode usability benefits from being Apple-centric. Ditto for Visual Studio and MS.
Linux and Python, by contrast, were both born out of a DIY culture where folks being free to choose their own tools was initially perceived solely as a highly desirable feature, rather than as a potential barrier to entry for newcomers.
That means there's an argument to be made that something like YHat's Rodeo [1] might be a better starting point for data analytics in Python than jumping straight to Jupyter Notebook, and it's also why the Mu editor [2] exists as a dedicated tool for folks learning Python by way of the micro:bit project.
[1] http://rodeo.yhat.com/docs/ [2] http://codewith.mu/
However, the reason I brought up the Curse and Firefox GUI examples was to emphasise the problems they hide from the default rich client experience:
- their default focus is on managing one environment per device
In the analogous Python tool, one could replace "per device" with "per project" - and perhaps have a "default project" so something useful could happen even before you've decided what you're doing...
But we've immediately bumped the complexity level up in doing so, and it's a level of complexity that many people initially spending all of their development time on a single project may not need.
I thought this thread was already interminable, I look forward to reading the never-ending rest of it now that you've raised the grim spectre of the PyPI user-ratings feature from the dead :).
All the arguments against integrating user ratings into a service that's focused on lowering barriers to publication still hold, so I'm really just noting that that decision to create a friendlier publishing environment *does* introduce some additional constraints elsewhere in the distribution pipeline.
User-curated package sets strikes me as the _lowest_ priority feature out of all of those, if we are ordering by priority to deliver a good user experience. I know "steam curators" have been brought up before - but we're talking about adding curators (one of my least favorite features of Steam, for what it's worth) before we've added "install game" ;-).
In many educational contexts, adding "install game" without support for institutional curators of some kind is a complete non-starter (even if those curators are a collaborative community like a Linux distribution, there's still more accountability than software publishing sites like PyPI tend to provide).
I initially wanted to disagree when I read this, but I'm not actually sure what educational contexts you're talking about, and why "accountability" is important?
Schools, mainly. Lots of administrators are still scared of the internet, so one of the attractions of things like Raspberry Pi is that the software updates come from Debian rather than directly from the software publishers.
Sometimes you can get away with "What the bureaucracy doesn't know won't hurt it", but it's more convenient when teachers don't have to do that.
"beginner" is a direction, and not a fixed position; many people more "beginner" than the current audience could be well-served by a discoverable initial project-creation and REPL UI. While I don't doubt that some backend pieces might help (although I still don't see how the one being discussed would), I also think that it would be very hard to say that the back-end is a *limiting factor* in UX improvement for the Python onboarding process; the front end could move quite a bit up the value chain without touching any of the various backends it would need to interact with.
But of course, if I really wanted to make this point, I'd just write it; dstufft is certainly right that volunteer time is not fungible. If I'm lucky, I'll have the time to do that at some point, since my efforts to convince someone else that this is the high-value target have been failing for some years now ;).
I'll confess that one of my ulterior motives for encouraging computing teachers to engage more directly with the upstream Python community is that I kinda hope we'll eventually run into one that either decides none of the current editors are good enough and creates their own, or else decides that "create the introductory editor that you wish you had when you started learning to program" might make a good collaborative student project :)
Alternatively, I've recently started using Visual Studio Code as my editor for work [1], and it seems likely that would be hackable enough for someone to create a plugin that bootstrapped a complete Python toolchain such that the bootstrapping flow became:
1. Install VSCode 2. Install the "New to Python" plugin 3. Run the plugin's "Setup Python Environment" command
Cheers, Nick.
[1] Oh the irony that the first cross-platform editor I've tried that I actually think looks nice and find pleasant to use on Fedora was released by Microsoft :)
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Dec 16, 2016, at 5:51 AM, Daniel Holth <dholth@gmail.com> wrote:
I'm also a visual studio code fan. It is the first editor I've tried that feels lightweight like Vim but has the power of many plugins. That, and the text rendering is excellent.
https://pypi.python.org/pypi/Stallion <https://pypi.python.org/pypi/Stallion> is a lovely GUI package manager.
There's a lot to like here - no disrespect to the Stallion team - but it's worth remembering this lesson from Havoc Pennington: https://blog.ometer.com/2016/01/26/the-dangerous-ui-team/ <https://blog.ometer.com/2016/01/26/the-dangerous-ui-team/> The major reason UI is important for this use-case - onboarding of new people to Python programming - is to give them discoverability on terms they're already familiar with. That means that the first "UI" element has to be a cross-platform UI bundle. Stallion is still a thing you have to install (and from what I can see, a thing you have to install into a virtualenv?)
One possibility to consider is that virtualenv itself is a bad idea. Why should the Python interpreter executable, rather than the program being run, determine the set of packages that is available for import? It is confusing and inconvenient to have to deal with environments at all. Yes, even if you are using a helper. Maybe there can be a better way to manage dependencies that is not completely disjoint from setup.py.
I can see why you'd say that, but I disagree. I think the name "virtualenv" is really confusing, but the general idea of "it's the interpreter and not the app" is a very powerful concept because you can run a REPL (or a notebook, or a debugger, or a doc generator, or any other dev tool) in the same context as your application code, without actually loading or executing any specific thing from your application code. Virtualenv also lets you easily control which Python version or interpreter (hello, pypy!) is being used in each context. -glyph
On Dec 16, 2016, at 5:07 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 16 December 2016 at 20:57, Glyph Lefkowitz <glyph@twistedmatrix.com <mailto:glyph@twistedmatrix.com>> wrote:
Anyhow, Xcode is far from perfect - many of the places it touches the UNIX pipeline are extremely sharp edges you can easily impale yourself on (and don't get me started about codesigning) - but it nevertheless points at a different potential direction. For example; why expose the concept of a "virtual environment" directly at all? "New Project" could just create a requirements.txt and a setup.py for you, alongside a git repo and a virtualenv for that project. Or, the UI could be geared towards setting up a tox.ini rather than a virtualenv, and run everything through tox so it's in an isolated environment with defined requirements. This is a best practice anyway so why not make it easier to start early?
This might all be way too much work, but I think it's important to remember it's possible.
Yeah, I think we agree more than we disagree here.
Quite. But the devil's in the details :).
The main thing is that one of the key ways newcomer-friendly environments make themselves more approachable is to *constrain choice*.
I think framing this as "constraint" is a little misleading. In a sense it is a constraint, but a better way to think of it is: provide a reasonable default. Right now, the "default UI" that most users get is a bare bash prompt where commands like 'pip install' fail with an error for no reason they can discern. They can still choose to inject a different tool at any point in the process (after all, we're talking about frontends which create existing concepts like virtualenvs and package installations) if they so choose; they just get a default that does something - anything - useful.
XCode usability benefits from being Apple-centric. Ditto for Visual Studio and MS.
Linux and Python, by contrast, were both born out of a DIY culture where folks being free to choose their own tools was initially perceived solely as a highly desirable feature, rather than as a potential barrier to entry for newcomers.
That means there's an argument to be made that something like YHat's Rodeo [1] might be a better starting point for data analytics in Python than jumping straight to Jupyter Notebook, and it's also why the Mu editor [2] exists as a dedicated tool for folks learning Python by way of the micro:bit project.
[1] http://rodeo.yhat.com/docs/ <http://rodeo.yhat.com/docs/> [2] http://codewith.mu/ <http://codewith.mu/>
Minor point - nobody should use Mu yet, at least not on the mac: https://github.com/mu-editor/mu/issues/206 <https://github.com/mu-editor/mu/issues/206> :) More significantly, I think any text editor will do just fine (as long as it's not Emacs or Vim) - I've had great success with http://www.barebones.com/products/textwrangler/ <http://www.barebones.com/products/textwrangler/>, and even Notepad will do in a pinch. There are already pretty good integration points where editors can be told to open specific files. One of my frustrations with the educational ecosystem is the focus on the (quite difficult) problem of providing students with a fully integrated text editing / script running / debugging environment, rather than figuring out how to orchestrate and launch the quite powerful and sophisticated tools we already have.
However, the reason I brought up the Curse and Firefox GUI examples was to emphasise the problems they hide from the default rich client experience:
- their default focus is on managing one environment per device
In the analogous Python tool, one could replace "per device" with "per project" - and perhaps have a "default project" so something useful could happen even before you've decided what you're doing...
But we've immediately bumped the complexity level up in doing so, and it's a level of complexity that many people initially spending all of their development time on a single project may not need.
I think we're underestimating potential programming students. The idea of managing multiple documents is likely something they're familiar with from word processing apps. If not, then fine - we can start them off with a default document.
I thought this thread was already interminable, I look forward to reading the never-ending rest of it now that you've raised the grim spectre of the PyPI user-ratings feature from the dead :).
All the arguments against integrating user ratings into a service that's focused on lowering barriers to publication still hold, so I'm really just noting that that decision to create a friendlier publishing environment *does* introduce some additional constraints elsewhere in the distribution pipeline.
User-curated package sets strikes me as the _lowest_ priority feature out of all of those, if we are ordering by priority to deliver a good user experience. I know "steam curators" have been brought up before - but we're talking about adding curators (one of my least favorite features of Steam, for what it's worth) before we've added "install game" ;-).
In many educational contexts, adding "install game" without support for institutional curators of some kind is a complete non-starter (even if those curators are a collaborative community like a Linux distribution, there's still more accountability than software publishing sites like PyPI tend to provide).
I initially wanted to disagree when I read this, but I'm not actually sure what educational contexts you're talking about, and why "accountability" is important?
Schools, mainly. Lots of administrators are still scared of the internet, so one of the attractions of things like Raspberry Pi is that the software updates come from Debian rather than directly from the software publishers.
Sometimes you can get away with "What the bureaucracy doesn't know won't hurt it", but it's more convenient when teachers don't have to do that.
I think it's very good that we have people focused on institutional concerns. My focus is somewhere else - peer learning. I want students to be able to take what they've learned in class and easily apply it in a personally useful context. And frankly, I think this is useful to a huge audience of people who aren't exactly "students" - people who already sort of know how computers work, who spend all day long sitting in Excel and writing macros (because the UI to do so is discoverable!) and are keen to learn something better to actually do tasks that they have in their job, not just to learn to code.
"beginner" is a direction, and not a fixed position; many people more "beginner" than the current audience could be well-served by a discoverable initial project-creation and REPL UI. While I don't doubt that some backend pieces might help (although I still don't see how the one being discussed would), I also think that it would be very hard to say that the back-end is a limiting factor in UX improvement for the Python onboarding process; the front end could move quite a bit up the value chain without touching any of the various backends it would need to interact with.
But of course, if I really wanted to make this point, I'd just write it; dstufft is certainly right that volunteer time is not fungible. If I'm lucky, I'll have the time to do that at some point, since my efforts to convince someone else that this is the high-value target have been failing for some years now ;).
I'll confess that one of my ulterior motives for encouraging computing teachers to engage more directly with the upstream Python community is that I kinda hope we'll eventually run into one that either decides none of the current editors are good enough and creates their own, or else decides that "create the introductory editor that you wish you had when you started learning to program" might make a good collaborative student project :)
I think we're going to remain stuck in our current quagmire until we let go of the idea of writing a text editor :). An all-singing all-dancing integrated development environment would be great. But editing text is a problem that's already solvable with a huge plethora of tools, most of them free. The tricky part where everybody gets stuck is making 'pip' or 'python' in a command-line window do anything useful at all.
Alternatively, I've recently started using Visual Studio Code as my editor for work [1], and it seems likely that would be hackable enough for someone to create a plugin that bootstrapped a complete Python toolchain such that the bootstrapping flow became:
1. Install VSCode 2. Install the "New to Python" plugin 3. Run the plugin's "Setup Python Environment" command
Cheers, Nick.
[1] Oh the irony that the first cross-platform editor I've tried that I actually think looks nice and find pleasant to use on Fedora was released by Microsoft :)
Turnabout is fair play - maybe 5 years from now we'll all be lamenting the terrible state of Python on UNIX and wishing it was as well-supported as Windows :). -g
On Fri, Dec 16, 2016 at 3:40 PM Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Dec 16, 2016, at 5:51 AM, Daniel Holth <dholth@gmail.com> wrote:
I'm also a visual studio code fan. It is the first editor I've tried that feels lightweight like Vim but has the power of many plugins. That, and the text rendering is excellent.
https://pypi.python.org/pypi/Stallion is a lovely GUI package manager.
There's a lot to like here - no disrespect to the Stallion team - but it's worth remembering this lesson from Havoc Pennington: https://blog.ometer.com/2016/01/26/the-dangerous-ui-team/
The *major* reason UI is important for this use-case - onboarding of new people to Python programming - is to give them *discoverability* on terms they're already familiar with. That means that the first "UI" element has to be a cross-platform UI bundle. Stallion is still a thing you have to install (and from what I can see, a thing you have to install into a virtualenv?)
One possibility to consider is that virtualenv itself is a bad idea. Why should the Python interpreter executable, rather than the program being run, determine the set of packages that is available for import? It is confusing and inconvenient to have to deal with environments at all. Yes, even if you are using a helper. Maybe there can be a better way to manage dependencies that is not completely disjoint from setup.py.
I can see why you'd say that, but I disagree. I think the *name* "virtualenv" is really confusing, but the general idea of "it's the interpreter and not the app" is a very powerful concept because you can run a REPL (or a notebook, or a debugger, or a doc generator, or any other dev tool) in the same *context* as your application code, without actually loading or executing any specific thing from your application code. Virtualenv also lets you easily control which Python version or interpreter (hello, pypy!) is being used in each context.
My point is really that virtualenv both causes and solves problems. There's jaraco's thing I was trying to remember https://pypi.org/project/rwt/ which is a creative alternative to virtualenv for some situations. I wish there was a smoother path between virtualenv develop-against-pypi-libraries and end-user application deployment. Stallion has always looked cool but it hasn't been updated in a few years, I admit I never used it "for real". Don't know what it would it take to make a useful GUI package manager. Interactive dependency conflict resolution graphs? Able to install into remote environments? Maybe the command line version will always be easier to use.
Glyph Lefkowitz wrote:
"New Project" could just create a requirements.txt and a setup.py for you, alongside a git repo and a virtualenv for that project. Or, the UI could be geared towards setting up a tox.ini rather than a virtualenv, and run everything through tox so it's in an isolated environment with defined requirements.
I'd be very interested in something like this. I'm not a big fan of IDEs generally, but one feature I do appreciate greatly is having a one-button "build" process that creates a distributable app bundled with everything it needs, and be assured it will work on someone else's machine. That's currently rather difficult to do with Python in any but the simplest cases, even for a single platform. Cross-platform is even worse. +1 on providing some tools to make it easier. -- Greg
On 17 December 2016 at 06:40, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Dec 16, 2016, at 5:51 AM, Daniel Holth <dholth@gmail.com> wrote:
One possibility to consider is that virtualenv itself is a bad idea. Why should the Python interpreter executable, rather than the program being run, determine the set of packages that is available for import? It is confusing and inconvenient to have to deal with environments at all. Yes, even if you are using a helper. Maybe there can be a better way to manage dependencies that is not completely disjoint from setup.py.
I can see why you'd say that, but I disagree. I think the *name* "virtualenv" is really confusing, but the general idea of "it's the interpreter and not the app" is a very powerful concept because you can run a REPL (or a notebook, or a debugger, or a doc generator, or any other dev tool) in the same *context* as your application code, without actually loading or executing any specific thing from your application code. Virtualenv also lets you easily control which Python version or interpreter (hello, pypy!) is being used in each context.
I'll also note that VSCode's Python plugin will find virtual environments that are located inside the project directory by default. That approach of "the virtualenv is inside the project directory" is probably a decent pattern to recommend as a default, since it aligns with the way a lot of IDEs (including VSCode itself) already work. When you use that model, rather than being something you have to think about explicitly, the "Python virtual environment" just becomes an implementation detail of how the IDE manages your application dependencies, with the added bonus that *if you want to*, you can re-use that environment independently of both the application *and* the IDE. And while I personally prefer to keep the notion of "project" and "environment" more explicitly separate (i.e. I have an M:N mapping between a collection of virtualenvs centrally managed with vex and the various projects in my devel, fedoradevel and rhdevel folders, hence [1]), I believe that level of complexity in a local dev setup isn't really normal even for experienced programmers, let alone folks that are still early in the learning process. Cheers, Nick. [1] https://github.com/DonJayamanne/pythonVSCode/issues/569 -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Dec 15, 2016 at 8:29 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
At the beginning of your story you mentioned the GUI client - *that* is the missing piece ;). I've been saying for years that we need a Python.app that lets you easily bootstrap all this stuff: walk you through installing C dev tools if your packages need them, present a GUI search interface to finding packages, present a normal "file->open" dialog for selecting a location for a new virtualenv, automatically pop open a terminal, launching a Jupyter notebook whose kernel is pointed at said environment...
Anaconda provides something like this -- personally, I;m a command lien geek, so have no idea how much or if it's any good. But might be worth a look if you're interested. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Fri, Dec 16, 2016 at 5:51 AM, Daniel Holth <dholth@gmail.com> wrote:
One possibility to consider is that virtualenv itself is a bad idea. Why should the Python interpreter executable, rather than the program being run, determine the set of packages that is available for import?
well, way back when, som eof us suggestted that pyton have pacakge version mangement built in to import: import this_package>=2.1 or whatever. At that time, the pyGTK and wxPython projects had done a role-your-own version of this. wxPython's was: import wxversion wxversion.select('2.3') import wx kind a kludgy, but it worked. However, Guido, among others was pretty adamant that this was NOT python's responsibility. Then, along came setuptools that kinda-sorta provided something like that, and then virtualenv -- and the rest is history. I now use conda, which provides environments that manage python itself, other C libs, etc, and it works pretty well. And there are utilities that let you run a script in a given environment: https://github.com/pelson/conda-execute (and maybe others) So that does kinda pass the responsibility to the app. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 22 December 2016 at 09:08, Chris Barker <chris.barker@noaa.gov> wrote:
And there are utilities that let you run a script in a given environment:
https://github.com/pelson/conda-execute
(and maybe others)
https://pypi.python.org/pypi/pipsi (pip Script Installer) creates a dedicated venv for the module and its dependencies, and then adds symlinks from ~/.local/bin to any scripts installed into the venv's bin directory. As Armin notes in the README, it's a really nice way to handle utilities that happen to be written in Python and published via PyPI, without having them impact any other aspect of your system. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Dec 22, 2016, at 11:15 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 22 December 2016 at 09:08, Chris Barker <chris.barker@noaa.gov <mailto:chris.barker@noaa.gov>> wrote: And there are utilities that let you run a script in a given environment:
https://github.com/pelson/conda-execute <https://github.com/pelson/conda-execute>
(and maybe others)
https://pypi.python.org/pypi/pipsi <https://pypi.python.org/pypi/pipsi> (pip Script Installer) creates a dedicated venv for the module and its dependencies, and then adds symlinks from ~/.local/bin to any scripts installed into the venv's bin directory. As Armin notes in the README, it's a really nice way to handle utilities that happen to be written in Python and published via PyPI, without having them impact any other aspect of your system.
I just wanted to echo that this is a great tool, and it teaches really good habits (i.e. don't install your general-purpose python tools into project-specific virtual environments). -glyph
participants (11)
-
Brett Cannon -
Chris Barker -
Daniel Holth -
Donald Stufft -
Freddy Rietdijk -
Glyph Lefkowitz -
Greg Ewing -
Nick Coghlan -
Paul Moore -
Steve Dower -
Wes Turner