stdlib crowdsourcing
The problem with stdlib - it is all damn subjective. There is no process to add functions and modules if you're not well-behaved and skilled in public debates and don't have really a lot of time to be a champion of your module/function. In other words - it is hard (if not impossible for 80% of Python Earth population). So, many people and projects decide to opt-out. Take a look at Twisted - a lot of useful stuff, but not in Python stdlib. So.. Provide a way for people to opt-out from core stuff, but still allow to share the changes and update code if necessary. This will require: - a local stdlib Python path convention - snippet normalization function and AST hash dumper - web site with stats - source code crawler How it works: 1. Every project maintains its own stdlib directory with functions that they feel are good to have in standard library 2. Functions are placed so that they are imported as if from standard library, but this time with stdlib prefix 3. The license for this directory is public domain to remove all legal barriers (credits are welcome, but optional) 4. Crawler (probably PyPI) scans this stdlib dir, finds functions, normalizes them, calculates hash and submits to web site 4.1 Normalization is required to find the shared function copy/pasted across different projects with different indentation level, docstrings, parameters/variable names etc. 4.2 Hash is calculated upon AST. There are at least three hashes for each entry: 4.2.1 Full hash - all docstrings and variable names are preserved, whitespace normalized 4.2.2 Stripped hash - docstrings are stripped, variable names are normalized 4.2.3 Signature hash - a mark placed in a comment above function name, either calculated from function signature or generated randomly, used for manual tracking of copy/paste e.g. pd:ac546df6b8340a92 5. Web site maintains usage and popularity staff, accepts votes on inclusion of snippets User stories: 1. "I want to find if there is a better/updated version of my function available" 1.1 I enter hash into web site search form 1.2 Site gives me a link to my snippet 1.3 I can see what people proposed to replace this function with 1.4 I can choose the function with most votes 1.5 I can flag the functions I may find irrelevant or 1.5 I can tag the functions that divert in different direction than I need to filter them 2. "I want to reuse code snippets without additional dependencies on 3rd party projects" 1.1 Just place them into my own stdlib directory 3. "I want to update code snippets when there is an update for them" 1.1 I run scanner, it extracts signature hashes, stripped hashes and looks if web-site version of signature matches normalized hash 4. "I want to see what people want to include in the next Python version" 1.1 A call for proposals is made 1.2 People place wannabe's into their stdlib dirs 1.3 Crawl generates new functions on a web site 1.4 Functions are categorized 1.5 Optionally included / declined with a short one-liner reason - why 1.6 Optionally provided with more detailed info why --- feature creep cut --- 5. "I want to see what functions are popular in other languages" 1.1 A separate crawler for Ruby, PHP etc. stdlib converts their AST into compatible format where possible 1.2 Submit to site stats 6. "I want to download the function in Ruby format" 1.1 AST converter tries to do the job automatically where possible 1.2 If it fails - you are encouraged to fix the converter rules or write the replacement for this signature manually Just an idea. -- anatoly t.
Once again, you're completely ignoring all existing knowledge and expertise on open collaboration and trying to reinvent the world. It's *not going to happen*. The standard library is just the curated core, and *yes*, it's damn hard to get anything added to it (deliberately so). There's a place where anyone can post anything they want, and see if others find it useful: PyPI. The standard library provides tools to upload to PyPI, and, as of 3.3, will even include tools to download and install from it. If you don't like our ecosystem (it's hard to tell whether or not you do: everything you post is about how utterly awful and unusable everything is, yet you're still here years later). If you think the PyPI UI is awful or inadequate, follow the example of crate.io or pythonpackage.com and *create your own*. There's far more to the Python universe than just core development, stop trying to shoehorn everything into a place where it doesn't belong. Finally-giving-up'ly, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, May 29, 2012 at 2:02 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
If you don't like our ecosystem (it's hard to tell whether or not you do: everything you post is about how utterly awful and unusable everything is, yet you're still here years later).
I understand the discouragement with regard to repeating yourself over and over again. But, let's keep the discussion friendly here, okay? This is Python-ideas: crazy proposals are fine. We can simply ignore those and move on.
Am 29.05.2012 09:00, schrieb Alexandre Vassalotti:
On Tue, May 29, 2012 at 2:02 AM, Nick Coghlan <ncoghlan@gmail.com <mailto:ncoghlan@gmail.com>> wrote:
If you don't like our ecosystem (it's hard to tell whether or not you do: everything you post is about how utterly awful and unusable everything is, yet you're still here years later).
I understand the discouragement with regard to repeating yourself over and over again. But, let's keep the discussion friendly here, okay? This is Python-ideas: crazy proposals are fine. We can simply ignore those and move on.
I don't see what's unfriendly about that paragraph: it's a quite accurate matter-of-fact statement... Georg
On Tue, May 29, 2012 at 9:02 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Once again, you're completely ignoring all existing knowledge and expertise on open collaboration and trying to reinvent the world. It's *not going to happen*.
It's too boring to live in a world of existing knowledge and expertise, and yes, I am not aware of any open collaboration stuff expertise. Any reading recommendations with concentrated knowledge that can fit my brain?
The standard library is just the curated core, and *yes*, it's damn hard to get anything added to it (deliberately so). There's a place where anyone can post anything they want, and see if others find it useful: PyPI.
The major drawbacks of remote packages in general is that it bring back project compilation from the old days. The biggest Python advantage at all times was "copy and run" ability. The drawbacks of PyPI for this proposal are: 1. every function you need will require a separate upload to PyPI 2. you can't upload function with the same stdlib name, but slightly different implementation as it is used in different projects 3. you can't find functions that people recommend to be included into stdlib 4. it is hard (impossible) to gather feedback on the quality of these proposals
The standard library provides tools to upload to PyPI, and, as of 3.3, will even include tools to download and install from it.
I am glad 3.3 is giving virtualenv and bootstrap stuff. It would really rock, if the new feature won't be settled in stone right after release and will gain a few UX iterations with allowed break-ability. As for PyPI, the major drawback of it is security - DNS attack for a couple of minutes, and one of your automatically deployed nodes is trojan ready. I remember PyPI password are stored in clear-text on developer's machine, but I don't remember if anyone turned off HTTP basic authorization on PyPI to protect passwords travelling to PyPI with every upload from intercepting. It would be an interesting exercise to sniff PyPI passwords over WiFi during next conference (i.e. https://ep2012.europython.eu/) and match those to the developer's accounts on *.python.org ;)
If you don't like our ecosystem (it's hard to tell whether or not you do: everything you post is about how utterly awful and unusable everything is, yet you're still here years later).
You're absolutely right - I like the Python ecosystem, otherwise I wouldn't stick there. It is like a vintage car - awesome, nice looking, and there is even this new twisted pyusion engine inside, but.. well - it's not for youngsters.
If you think the PyPI UI is awful or inadequate, follow the example of crate.io or pythonpackage.com and *create your own*. There's far more to the Python universe than just core development, stop trying to shoehorn everything into a place where it doesn't belong.
I have absolutely no idea how aforementioned post touches PyPI UI. Speaking about PyPI enhancements and ecosystem, instead of reinventing bicycles I'd rather patch existing one. The only problem is that patches are not accepted. https://bitbucket.org/loewis/pypi/pull-request/1/fix-imports-add-logging-to-...
On Fri, Jun 1, 2012 at 11:08 AM, anatoly techtonik <techtonik@gmail.com> wrote:
On Tue, May 29, 2012 at 9:02 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Once again, you're completely ignoring all existing knowledge and expertise on open collaboration and trying to reinvent the world. It's *not going to happen*.
It's too boring to live in a world of existing knowledge and expertise,
Frankly, this one fragment is enough to stop me reading further. Who wants to learn from the vast and broad experience when you could simply randomize the rules of reality through ignorance and stubbornness? I sound fickle, because I am.
and yes, I am not aware of any open collaboration stuff expertise. Any reading recommendations with concentrated knowledge that can fit my brain?
The standard library is just the curated core, and *yes*, it's damn hard to get anything added to it (deliberately so). There's a place where anyone can post anything they want, and see if others find it useful: PyPI.
The major drawbacks of remote packages in general is that it bring back project compilation from the old days. The biggest Python advantage at all times was "copy and run" ability.
The drawbacks of PyPI for this proposal are: 1. every function you need will require a separate upload to PyPI 2. you can't upload function with the same stdlib name, but slightly different implementation as it is used in different projects 3. you can't find functions that people recommend to be included into stdlib 4. it is hard (impossible) to gather feedback on the quality of these proposals
The standard library provides tools to upload to PyPI, and, as of 3.3, will even include tools to download and install from it.
I am glad 3.3 is giving virtualenv and bootstrap stuff. It would really rock, if the new feature won't be settled in stone right after release and will gain a few UX iterations with allowed break-ability.
As for PyPI, the major drawback of it is security - DNS attack for a couple of minutes, and one of your automatically deployed nodes is trojan ready. I remember PyPI password are stored in clear-text on developer's machine, but I don't remember if anyone turned off HTTP basic authorization on PyPI to protect passwords travelling to PyPI with every upload from intercepting. It would be an interesting exercise to sniff PyPI passwords over WiFi during next conference (i.e. https://ep2012.europython.eu/) and match those to the developer's accounts on *.python.org ;)
If you don't like our ecosystem (it's hard to tell whether or not you do: everything you post is about how utterly awful and unusable everything is, yet you're still here years later).
You're absolutely right - I like the Python ecosystem, otherwise I wouldn't stick there. It is like a vintage car - awesome, nice looking, and there is even this new twisted pyusion engine inside, but.. well - it's not for youngsters.
If you think the PyPI UI is awful or inadequate, follow the example of crate.io or pythonpackage.com and *create your own*. There's far more to the Python universe than just core development, stop trying to shoehorn everything into a place where it doesn't belong.
I have absolutely no idea how aforementioned post touches PyPI UI. Speaking about PyPI enhancements and ecosystem, instead of reinventing bicycles I'd rather patch existing one. The only problem is that patches are not accepted. https://bitbucket.org/loewis/pypi/pull-request/1/fix-imports-add-logging-to-... _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy
On Sat, Jun 2, 2012 at 8:24 PM, Calvin Spealman <ironfroggy@gmail.com> wrote:
On Fri, Jun 1, 2012 at 11:08 AM, anatoly techtonik <techtonik@gmail.com> wrote:
On Tue, May 29, 2012 at 9:02 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Once again, you're completely ignoring all existing knowledge and expertise on open collaboration and trying to reinvent the world. It's *not going to happen*.
It's too boring to live in a world of existing knowledge and expertise,
Frankly, this one fragment is enough to stop me reading further. Who wants to learn from the vast and broad experience when you could simply randomize the rules of reality through ignorance and stubbornness?
If everybody would think like this, the world will never learn about anti-patterns, and the software craftmanship collapsed in astonishing agony some years ago. If it doesn't make it clear - it is not randomizing - it is putting beliefs to the test asking for the current status.
I sound fickle, because I am.
It doesn't matter how do you sound, what matters is that you spoiled the fun to discuss the technical part no matter how long ago it was invented. If you have a lot of people who ask the same question - create a FAQ. That's not a vast and broad experience - that's just a time proven practice from usenet times. Common guys, what's wrong with you? It is just an idea, not a proposal or scientific paper. And I am not a scientist - I just want to discuss the idea, and I am not sending mails to python-dev anymore, because you asked to. I've spent some time trying to make the idea interesting. It is fine If you know a scientific paper about the matter, can explain it in a few words and send a link for more details. But the replies like "you're stubborn and ignorant, and nobody should help you" doesn't make you a better person. I am criticizing, because I lack time, motivation and fantasy to write stuff about good and bright sides in my life that I just don't see. I write because I see bad things that can be better, and I am still open to discuss if it is real or not. -- anatoly t.
I just want to discuss the idea,
Great! You even got a perfectly good answer already: *not going to happen*! Hey, this seems to be working: you raise a point, the community discusses it and after careful deliberation comes up with an answer! This is how things are supposed to be working, aren't they?
But the replies like "you're stubborn and ignorant, and nobody should help you" doesn't make you a better person.
Hey, hey, hey, you are overlooking the other answers you got! You think they came from thin air? People read your post, thought about it, considered the pros and cons and then put in the time to answer it, write an email and hit the Send button. Now, move on, nothing to be seen here, chop, chop, carry on! Cheers, Daniel -- Psss, psss, put it down! - http://www.cafepress.com/putitdown
On Tue, 12 Jun 2012 12:04:58 +0300 anatoly techtonik <techtonik@gmail.com> wrote:
On Sat, Jun 2, 2012 at 8:24 PM, Calvin Spealman <ironfroggy@gmail.com> wrote:
On Fri, Jun 1, 2012 at 11:08 AM, anatoly techtonik <techtonik@gmail.com> wrote:
On Tue, May 29, 2012 at 9:02 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Once again, you're completely ignoring all existing knowledge and expertise on open collaboration and trying to reinvent the world. It's *not going to happen*. It's too boring to live in a world of existing knowledge and expertise, Frankly, this one fragment is enough to stop me reading further. Who wants to learn from the vast and broad experience when you could simply randomize the rules of reality through ignorance and stubbornness? If everybody would think like this, the world will never learn about anti-patterns, and the software craftmanship collapsed in astonishing agony some years ago. If it doesn't make it clear - it is not randomizing - it is putting beliefs to the test asking for the current status.
Ah, I think I see Anatoly's problem here. It's an impedance mismatch. He wants to discuss language/platform/environment ideas. This is valuable work, and he does have some interesting ideas. It definitely has a place in the world. It's just that this isn't that place. Python has a set of objectives for the language that have been around long enough to qualify as "traditions". As such, it's not a good place to experiment with arbitrary changes to things, because you keep running afoul of the traditions.
Common guys, what's wrong with you? It is just an idea, not a proposal or scientific paper.
Yes, but it's an idea that ignores the traditions of the environment you're proposing it for. If you're serious about discussing ideas about changing Python, you need to do the groundwork of understanding those traditions, and try and make sure your ideas don't collide with them. It doesn't matter whether or not they're good ideas, if they clash with the traditions, they aren't going to happen. You need to figure that out yourself, and not ask us to do it for you. If, on the other hand, you want to talk about language/platform/environment design ideas without that restriction, then you need a different forum. Just because you happen to be working in Python doesn't mean that a Python forum is appropriate for them, any more than discussing (say) drone control programs would be appropriate in a Python forum just because I happen to be writing it in Python. If you're somewhere in between the two, maybe a PyPy forum would be more appropriate? I dunno. I'm sorry I can't really recommend a good forum for you. The last time I was seriously interested in such things, Python hadn't been released yet. <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
I write because I see bad things that can be better, and I am still open to discuss if it is real or not.
There's nothing wrong with that in its place. But python-ideas is a place for ideas where the poster is pretty sure it's real *and* has a concrete proposal (the "idea" in python-ideas) to make it better *and* has the will to follow up themselves if nobody else grabs the ball. There's some room for blue-sky ideas (lacking concrete proposals or personal commitment), but if all you ever offer is blue-sky ideas that get no uptake, you're just wasting time, yours as well as everybody else's.
On Tue, May 29, 2012 at 8:05 AM, anatoly techtonik <techtonik@gmail.com>wrote:
The problem with stdlib - it is all damn subjective. There is no process to add functions and modules if you're not well-behaved and skilled in public debates and don't have really a lot of time to be a champion of your module/function. In other words - it is hard (if not impossible for 80% of Python Earth population). So, many people and projects decide to opt-out. Take a look at Twisted - a lot of useful stuff, but not in Python stdlib. So..
Provide a way for people to opt-out from core stuff, but still allow to share the changes and update code if necessary.
This will require: - a local stdlib Python path convention - snippet normalization function and AST hash dumper - web site with stats - source code crawler
How it works: 1. Every project maintains its own stdlib directory with functions that they feel are good to have in standard library 2. Functions are placed so that they are imported as if from standard library, but this time with stdlib prefix 3. The license for this directory is public domain to remove all legal barriers (credits are welcome, but optional) 4. Crawler (probably PyPI) scans this stdlib dir, finds functions, normalizes them, calculates hash and submits to web site 4.1 Normalization is required to find the shared function copy/pasted across different projects with different indentation level, docstrings, parameters/variable names etc. 4.2 Hash is calculated upon AST. There are at least three hashes for each entry: 4.2.1 Full hash - all docstrings and variable names are preserved, whitespace normalized 4.2.2 Stripped hash - docstrings are stripped, variable names are normalized 4.2.3 Signature hash - a mark placed in a comment above function name, either calculated from function signature or generated randomly, used for manual tracking of copy/paste e.g. pd:ac546df6b8340a92 5. Web site maintains usage and popularity staff, accepts votes on inclusion of snippets
User stories: 1. "I want to find if there is a better/updated version of my function available" 1.1 I enter hash into web site search form 1.2 Site gives me a link to my snippet 1.3 I can see what people proposed to replace this function with 1.4 I can choose the function with most votes 1.5 I can flag the functions I may find irrelevant or 1.5 I can tag the functions that divert in different direction than I need to filter them
2. "I want to reuse code snippets without additional dependencies on 3rd party projects" 1.1 Just place them into my own stdlib directory
3. "I want to update code snippets when there is an update for them" 1.1 I run scanner, it extracts signature hashes, stripped hashes and looks if web-site version of signature matches normalized hash
4. "I want to see what people want to include in the next Python version" 1.1 A call for proposals is made 1.2 People place wannabe's into their stdlib dirs 1.3 Crawl generates new functions on a web site 1.4 Functions are categorized 1.5 Optionally included / declined with a short one-liner reason - why 1.6 Optionally provided with more detailed info why
--- feature creep cut --- 5. "I want to see what functions are popular in other languages" 1.1 A separate crawler for Ruby, PHP etc. stdlib converts their AST into compatible format where possible 1.2 Submit to site stats
6. "I want to download the function in Ruby format" 1.1 AST converter tries to do the job automatically where possible 1.2 If it fails - you are encouraged to fix the converter rules or write the replacement for this signature manually
Just an idea. -- anatoly t.
I think having a separate site "anatloy's std-lib" which somehow implemented an easy install of the top 10-100 most useful/popular/selected packages on pypi could be nice. I considered making such a bundle myself a while ago. I don't think it really needs to be python.org sanctioned. Yuval PS I like how candid the replies you got were, and indeed getting a reply is better than the sound of crickets. Though some of these replies carried the scent of excrement poredom - the author's need to import niceness.
participants (9)
-
Alexandre Vassalotti -
anatoly techtonik -
Calvin Spealman -
Daniel Fetchinson -
Georg Brandl -
Mike Meyer -
Nick Coghlan -
Stephen J. Turnbull -
Yuval Greenfield