Hi all, I just discovered this alias and I thought I'd post a few ideas. I wouldn't call myself a Python master yet, but it's an amazing language and my biggest wish is that it was more widely used in the industry. Here are a few suggestions: 1. Decrease the bug count. I recently noticed that there are about 5,400 active bugs in http://bugs.python.org/. That surprised me because I almost never see anyone complain about bugs in Python (compared to the number who complain about bugs in LibreOffice, graphics drivers, Gnome / KDE, etc.) There are a lot of people on this list, and if the brainpower can be applied to practical, known, existing problems, it is a great way to improve Python while also considering more exotic ideas. I can also suggest making a pretty graph of the bug count and putting it on the front page of python.org for greater visibility. 2. Python is somewhat popular on servers, and there is a lot of potential for more. WordPress is easy to use and powerful, but lots of people don't want to program in PHP. Or Javascript, Java, Ruby, etc. Codebases like Whoosh full-text search (https://bitbucket.org/mchaput/whoosh/wiki/Home) are important, but have minimal dev resources as most people are using Lucene / ElasticSearch. The common choice is between 1.3M lines of Java: https://www.openhub.net/p/elasticsearch, containing 1100 todos and 1000 references to "deprecated", or 41K lines of Python written mostly by one person. Hadoop is another big Java project (1.9M lines), and there is even an ecosystem around it. Python interoperates with Hadoop, but it should be possible to build a radically simpler framework that provides the same functionality using Python-native functionality and without all the baggage. Hadoop has several interesting sister projects: a distributed database, scalable machine learning, a high-level data flow language, a coordination service, etc. I'm sure you'd build something smaller, cleaner, faster in many cases, more reliable, etc. 3. It was a sad mistake that Google picked Java over Python for Android. However, there is now a great program called Kivy which allows people to write apps for IOS or Android with one codebase, but it could also use more resources, as for example it doesn't fully support Python 3.x yet. There are 10s of thousands of bugs in the popular Python libraries and I would fix those before proposing more language changes. 4. I enjoy reading about the Python performance improvements, but it is mostly a perception problem with all the existing workarounds. Gnome wrote version 3 of their shell in Javascript because they didn't think Python would be fast enough. Lots of people write Node because it's compiled and "fast". I suggest taking some of the effort working on performance, and spend it on evangelizing to other programmers that Python / Cython / PyPy, etc. are already good enough! There are a lot of programmers out there who would be happier if they could work in Python. 5. It would be great to get Python in the web browsers as an alternative to Javascript. There are a number of projects which convert Python to Javascript, but this would be more direct. LibreOffice ships with a Python interpreter, why can't Firefox and Webkit? ;-) Obviously there are interoperability issues, but it would be great to just side-step all the complexity of Javascript (Here is a server-oriented article, but it gives a flavor: http://notes.ericjiang.com/posts/751) This might sound like a crazy idea, but the engineering problems aren't that hard. 6. In a few cases, there are two many codebases providing the same functionality, and none of them are really doing the job. For example, the de-facto MySQL Python interop library (https://pypi.python.org/pypi/MySQL-python) only supports Python 2.x and appears to be abandoned. There are several other libraries out there with different features, performance, compatibility, etc. and it's kind of a minefield for what should be a basic scenario. It takes leadership to jump in and figure out how to fix it and make one codebase good enough that the others can switch over. 7. Focus more on evolving the libraries rather than the language. I've recently discovered Toolz, which has a more complete set of functional language methods. I think some of them should be included in the official versions. A lot of people don't think Python is good enough for functional programming and this would help. These new routines add complexity, but a newbie doesn't need to write in a functional way, so it obeys the "only pay for what you use" rule. There are a number of under-staffed libraries and frameworks. I see people complain about the YAML parsing library being unmaintained, the default HTTP functionality being difficult and limiting, poor SOAP support, etc. There are a million ways to improve the Python ecosystem without making any language changes. You don't have a big rich company who can pay for thousands of full-time developers working on libraries, but the bug reports are a great way to prioritize. 8. I've yet to find a nice simple free IDE with debugging for Python. I use Atom, but it has primitive debugging. I tried PyCharm but it's very complicated (and not free, and Java). I use Jupyter sometimes also but I'd prefer a rich client app with watch windows, etc. 9. It would be interesting to re-imagine the spreadsheet with a more native Python interface. Pandas and matplotlib are great, but it would be cool to have it in LibreOffice Calc that supports drag and drop, copy and paste, can read and write ODS, etc. (Also, LibreOffice Base is basically unmaintained. I think if 10 Python programmers passionate about databases and GUIs showed up, it could re-invigorate this dead codebase.) 10. Being simple to learn and powerful is very hard. Fortunately, you can break compatibility every 10 years. My only suggestion is to get rid of the __self__ somehow ;-) Regards, -Keith http://keithcu.com/
On Fri, 8 Apr 2016 at 14:16 Keith Curtis <keithcu@gmail.com> wrote:
That bug count is misleading because it includes not only legitimate bugs but also enhancement proposals.
Two things. One, there is actually a good amount of performance work potentially landing in Python 3.6. Two, we have been saying Python is fast enough for most things for decades at this point, so this is an old issue that's never going away.
Once https://github.com/WebAssembly lands in the browser then it won't be as much of an issue. There are also various transpilers of Python to JavaScript already. I also tried back in 2006 but Mozilla rejected the idea so this has been tried before. :) -Brett
On 9 April 2016 at 07:15, Keith Curtis <keithcu@gmail.com> wrote:
Folks in institutional environments can most readily contribute to this by obtaining Python from commercial redistributors (rather than using the free community versions and then expecting free ongoing support from volunteers), and then funneling bug reports through their vendor rather than filing them directly with upstream (since donating bug fixes to commercial organisations for free isn't something most people consider a fun hobby).
This is about business dynamics and institutional supply chain management, not software, so wishing won't make it so. However, given the business challenges facing the vendors behind the Java projects you mention here, Python-based alternatives are also going to be a tough sell to potential investors. (That said, you may also be interested in the Apache Spark project, where Python, Java, Scala and R are the top tier analytics development languages)
People work on things in their own time because they find them enjoyable (or otherwise inherently rewarding), not because they're interested in facilitating increased corporate adoption. That said, yes, while it's high profile, contributing to CPython is one of the *least* effective ways of helping to improve the overall Python ecosystem in the near term, as it can literally take years to roll out major changes (as Python 2.6 is still the de facto baseline version, and there are many situations where folks are still using Python 2.4 and earlier). The best cases are those where we can define new APIs, protocols and idioms in ways that can also be adopted in earlier versions of the language by way of third party libraries and standard library backports.
There's nothing stopping anyone interested in this area from working on any kind of evangelisation they want to. However, what harm does it cause us personally if people decide to use other programming languages? Python tautologically fits the brains of Pythonistas, but that's nowhere near being the same thing as it being the right language for everyone for every purpose. It's also the case that any developer with only one language currently in their toolbox (even when that language is Python) is a developer with lots of learning opportunities ahead of them: http://www.curiousefficiency.org/posts/2015/10/languages-to-improve-your-pyt...
This misunderstands the nature of the relationship between JavaScript, CSS and the HTML Domain Object Model: these are technologies that have co-evolved for describing and dynamically updating user interfaces that interact with remote services over a network, and they're *really* good at it. Replicating that ecosystem in other program languages would technically be possible, but there's little incentive to do so given the work on WebAssembly and ongoing improvements in transpilers.
This is why people pay open source redistributors to ensure they have access to commercially supported components, rather than trusting that whatever they happened to find on the internet will continue to be maintained by anonymous benefactors.
"Some of them should be included" is not an actionable proposal. Anyone is free to propose specific additions to functools, and make the case for why those particular ones should be included in the standard library rather than continuing to be accessed via version independent 3rd party libraries like Toolz. (Advance warning: "I want a purely functional solution to a problem that can already be readily solved with procedural code" generally isn't accepted as a compelling justification) A simpler possibility might be to review the Functional Programming HOWTO at https://docs.python.org/3/howto/functional.html and consider ways that that might be updated to reference third part libraries, as well as potentially made more discoverable via the functools and itertools reference documentation.
What makes you think anyone here has the authority to tell anyone else how to spend their time? Folks work on community open source projects based on their personal interests and their commercial interests. While you're right that lots of things could stand to be improved, the best people to complain to about underinvestment (or misdirected investment) are commercial redistributors selling supported versions of CPython and other community projects in the Python ecosystem, rather than the already overcommitted volunteers contributing their own time for their own reasons.
Not the right list for that question.
Also not the right list.
Methods are just functions, so this is never going to happen. (When folks understand why, they've generally made a decent step forward in appreciating the differences between a procedural-first usage model for a language, vs an objects-first one) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Hi again all, Thanks for the replies. I don't have a studied opinion of what new methods should be added to functools, I mostly brought it up because it won't create an incompatible change and things like that should be much easier to do. Functools is one of the few remaining places in the core runtime where it seems what is provided is meant to be a sample or flavor, and not used on a daily basis without group-by, merge-sort, frequencies, pipe, take / drop, etc. I agree that more companies paying for Python support would be a great thing, but most often companies buy support at a different level of granularity: for example, RHEL where they get it for an entire OS, or hiring Django people to build and maintain a website. I wouldn't count on commercial redistributors of Python as ever being a big source of resources to fix the thousands of random bugs. The vast majority of people use the standard runtime. I realize that volunteers work on their own time and choice of tasks, but you've got quite a large community, and by periodically reporting on the bug count and having a goal to resolve old ones eventually, you will even find problem areas no one is talking about. Sometimes the issue isn't resources, but grit, focus, pride, etc. If you never weighed yourself, at looked at yourself in the mirror, you'd get heavier than you realized. You are mostly volunteers, but you can try to produce software as good as what paid professionals can do. Perhaps the bug list should be broken up into two categories: future optional or incompatible features, and code that people can easily agree are worth doing now. That way important holes don't get ignored for years. People should be able to trust the standard runtime. It didn't seem like WebAssembly would enable access to Numpy. It seems doubtful it will be much adopted if it can't access the rich Python runtime that might be installed. Python is much cleaner than some of the alternatives, so you should consider adoption as a humanitarian effort! Regards, -Keith
Hi Keith, On Sat, Apr 09, 2016 at 08:28:52PM -0400, Keith Curtis wrote: [...]
Python is much cleaner than some of the alternatives, so you should consider adoption as a humanitarian effort!
I keep seeing you telling us that "you should do this, you should do that, you you you..." but I don't see *you* volunteering to work on any of the bugs. Or do you see yourself more in a supervisory role? -- Steve
On 10 April 2016 at 10:28, Keith Curtis <keithcu@gmail.com> wrote:
For context, I work for Red Hat's developer experience team and occasionally put the graph at http://bugs.python.org/issue?@template=stats in front of our Python maintenance team and ask "This is a trend that worries me, what are we currently doing about it?". I'd encourage anyone using Python in a commercial or large organisational context that are getting their Python runtimes via a commercially supported Linux distro, Platform-as-a-Service provider, or one of the cross-platform CPython redistributors to take that graph, show it to their vendor and say: "These upstream CPython metrics worry us, what are you doing about them on our behalf?". (If you don't know how to ask your vendor that kind of question since another team handles the supplier relationship, then find that team and ask *them* what they're doing to address your concerns) For folks in those kinds of contexts running directly off the binaries or source code published by the Python Software Foundation, then my recommendation is a bit different: * option 1 is to advocate for switching to a commercial redistributor, and asking prospective vendors how they're supporting ongoing CPython maintenance as part of the vendor evaluation * option 2 is to advocate for becoming a PSF Sponsor Member and then advocate for a technical fellowship program aimed at general maintenance and project health monitoring for CPython (keeping in mind that "How do we manage it fairly and sustainably?" would actually be a bigger challenge for such a fellowship than funding it) Whether advocating for option 1 or option 2 makes more sense will vary by organisation, as it depends greatly on whether or not the organisation would prefer to work with a traditional commercial supplier, or engage directly with a charitable foundation that operates in the wider public interest. There's also option 3, which is to hire existing core developers and give them time to work on general upstream CPython maintenance, but there are actually relatively few organisations for which that strategy makes commercial sense (and they're largely going to be operating system companies, public cloud infrastructure companies, or commercial Python redistributors).
If that were an accurate assessment, I wouldn't need to write articles like http://www.curiousefficiency.org/posts/2015/04/stop-supporting-python26.html and the manylinux cross-distro Linux ABI wouldn't be based on the 9 year old CentOS 5 userspace. (We also wouldn't be needing to pursue redistributor-friendly proposals like PEP 493 to help propagate the network security enhancements in the 2.7.x series) Direct downloads from python.org are certainly a major factor on Windows (especially in educational contexts), but even there, many of the largest commercial adopters are using ActiveState's distribution (see http://www.activestate.com/customers ) or one of the analytical distributions from Enthought or Continuum Analytics.
So, you're suggesting we deliberately set out to make our volunteers feel bad in an effort to guilt them into donating more free work to corporations and other large organisations? When I phrase it in that deliberately negative way, I hope it becomes obvious why we *don't* do this. We do send out the weekly tracker metrics emails, and offers the statistics graphs on the tracker itself if people want to use them as parts of their own business cases for increased investment (whether direct or indirect) in upstream sustaining engineering for CPython.
You seem to be operating under some significant misapprehensions here. As far as I am aware, most of the core developers *are* paid professionals (even the folks that didn't start out that way) - we're just typically not paid specifically to work on CPython. Instead, we work on CPython to make sure it's fit for *our* purposes, or because we find it to be an enjoyable way to spend our free time. Engaging effectively with that environment means pursuing a "co-contributor" experience - we each make CPython better for our own purposes, and by doing so, end up making it better for everyone. The experience you seem to be seeking in this thread is that of a pure software consumer, and that's the role downstream redistributors fulfil in an open source ecosystem: they provide (usually for a fee) a traditional customer experience, with the redistributor taking care of the community engagement side of things.
Important holes don't get ignored for years - they get addressed, since people are motivated to invest the time to ensure they're fixed. That said, we do experience the traditional open source "community gap" where problems that are only important to folks that aren't yet personally adept at effective community engagement can languish indefinitely (hence my emphasis above on organisations considering the idea of outsourcing their community engagement efforts, rather than assuming that it's easy, or that someone else will take care of it for them without any particular personal motivation).
Project Jupyter already addressed that aspect by separating out the platform independent UI service for code entry from the language kernel backend for code evaluation. There's certainly a lot that could be done in terms of making Jupyter language kernels installable as web browser add-ons running native code, but that would be a question for the Project Jupyter folks, rather than here.
Python is much cleaner than some of the alternatives, so you should consider adoption as a humanitarian effort!
As tempting as it is to believe that "things that matter to software developers" dictate where the world moves, it's important to keep in mind that it's estimated we make up barely 0.25% of the world's population [1] - this means that for everyone that would consider themselves a "software developer", there's around 399 people who don't. Accordingly, if your motivation is "How do we help empower people to control the technology in their lives, rather than having it control them?" (which is a genuinely humanitarian effort), then I'd recommend getting involved with education-focused organisations like the Raspberry Pi Foundation, Software Carpentry and Django Girls, or otherwise helping to build bridges between the software development community and professional educators (see [2] for an example of the latter). When it comes to technology choices *within* that 0.25%, it's worth remembering that one of the key design principles of Python is *trusting developers to make their own decisions* (hence things like allowing monkeypatching as being incredibly useful in cases like testing, even while actively advising against it as a regular programming practice). That trust extends to trusting them to choose languages that are appropriate for them and their use cases. If that leads to them choosing Python, cool. If it leads to them choosing something else, that's cool, too - there are so many decent open source programming languages and runtimes out there these days that software developers are truly spoiled for choice, so providing a solid foundation for exploring those alternatives is at least as important a goal as reducing the incentives for people that already favour Python to need to explore them. Cheers, Nick. [1] http://www.infoq.com/news/2014/01/IDC-software-developers [2] https://2016.pycon-au.org/programme/python_in_education_seminar -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I feel like this demonstrates a confusion about how the web ecosystem works. Code that executes inside a web browser’s Javascript runtime does not have arbitrary access to the system. That’s not an arbitrary limitation, it’s vital, because code shipped over the web is inherently untrusted: it can be intercepted, manipulated, edited, and attacked. Each time the web application developers of the world want a new interface to system hardware they have to specify up a whole set of APIs for the browser Javascript to obtain that access in a way that is controlled and enabled by the user. Browser developers will allow code executing in the browser to access binaries on the host system *over their dead bodies*. Allowing that represents a terrifying security vulnerability. At no point will any sane browser developer allow that. As it turns out, of course, WebAssembly *would* enable access to NumPy because NumPy could simply be compiled to WebAssembly as well, and distributed along with the Python interpreter and all the other code you’d have to ship. The TL;DR here is: no web browser will ever allow access to a Python runtime (or any other runtime) that is installed. However, WebAssembly does not have the limitation you’ve suggested. Cory
Hi again, I mean this merely as food for thought. Thank you for reading. I personally find Python very stable, but the bug counts have been heading in the wrong direction: http://bugs.python.org/issue?@template=stats I recently discovered those charts, and they have valuable data that can be turned into action. My distributor of Python is Arch. I presume it is very close to what you guys release. That's just my example, but from what I've seen, few pay for a Python runtime. Given your license, I wouldn't expect a lot to. If someone waits for the wrong train, they won't get there. I don't suggest guilt-tripping volunteers into fixing bugs. If you've got to dig a ditch, you can try to enjoy the sun. Getting the bug count under control is an admirable goal. The list is an opportunity to focus on the known problems real people care about most, and to try to deal with old issues before taking on new ones. I don't recommend people fix bugs to help "corporations and large organizations". You won't be very motivated by that anti-capitalist mindset. People should fix bugs because it helps real Python users, and gets rid of barriers. If people steadily remove roadblocks, things will flourish. I was teasing about you guys not being paid professionals, however, it appears that very few of you have the goal of getting the official bug count down. That is a big difference between amateurs and professionals. If few people feel ownership of the official Python, it can be bad. Maybe the PSF could hire more with that mindset. I don't know the solutions, but I am grateful to be able to send you a few words. As for WebAssembly, I don't know if Numpy can "simply" be re-compiled for it. It seems like the sort of workitem that could take 2-3 years, and never be actually fully compatible or as fast. Python can run in sandboxes as well: https://wiki.python.org/moin/SandboxedPython. People who care about code being intercepted and manipulated should use SSL or sign it. People who write their own code to run on their own machine would likely prefer to be able to just directly reference the Python runtime they've already setup. I wonder whether the sandbox can be used outside of Web Assembly so that code distribution and security are not so intermingled. I did see someone write that WebAssembly is the "dawn of a new era", but I wonder whether it is mostly a bunch of Javascript people trying to solve their own problems rather than those who care about making Python work well on it. Regards, -Keith
I don’t think you’ve understood the problem here: you seem to be saying that we can solve the lack of trust issue by “rubbing some crypto on it”. But that doesn’t solve the problem at all. Let’s take this apart. “People who care about code being intercepted and manipulated”: if that code runs directly on the user’s machine then *everyone* should care. Put another way: it doesn’t matter what the *author* of the code cares about, it matters what the user cares about, and all users care about executing safe code! Otherwise, I can insert whatever Python code I like and run arbitrary code with the permissions of the browser. This opens the entire machine up to attack: an attacker can consume their CPU resources, transition into FS access, and basically do all kinds of wacky things. But wrapping this in SSL or signing the code doesn’t solve the problem at all. I opened a new browser window, turned my ad blocker off, and then loaded www.wired.com. The following domains loaded and executed Javascript code (this isn’t a complete list either, I gave up and got bored): - ads.rubiconproject.com - 5be16.v.fwmrm.net - optimized-by.rubiconproject.com - cdn.optimizely.com - rtb.adgrx.com - player.cnevids.com - dy48bnzanqw0v.cloudfront.net - cdn-akamai.mookie1.com - cdnjs.cloudflare.com - tpc.googlesyndication.com - securepubads.g.doubleclick.net - c.amazon-adsystem.com - s.update.rubiconproject.com - b.scorecardresearch.com - aax.amazon-adsystem.com - static.chartbeat.com - segment-data.zqtk.net - wired.disqus.com - animate.adobe.com - aam.wired.com - pagead2.googlesyndication.com - condenast.demdex.net - www.googletagservices.com - h5.adgear.com - dpm.demdex.net - a.adgear.com - i.yldbt.com - use.typekit.net - assets.adobedtm.com - z.moatads.com The point is that I, the user, did not consent to *any* of those. This is the way the web platform works: users don’t get asked who gets to execute code on their machine. Most users would not download a random binary to their machine from one of those domains and execute it. Wrapping that code in SSL or signing it prevents man-in-the-middle attacks on the code, but doesn’t in any sense prevent those actors listed there from doing terrible stuff! It is well known that Sony wrote an actual rootkit that they used as DRM and distributed it via CDs. If you believe that one of the above sites wouldn’t do something equally malicious with full access to my machine, you’re living in a fantasy world. This means that any code that those domains are allowed to execute needs to run in an absurdly restricted context: because neither I nor any other user trusts arbitrary domains to run arbitrary code! A Python sandbox that allows access to any code not distributed via the web browser is not restrictive enough. That Python tells you too much about the machine on which it is running. This is doubly bad if that Python is capable of calling into native code extensions distributed outside the browser (such as NumPy), because sandboxing code like that requires running a complete virtual machine. Either distributing that code would be a nightmare, or you’d be forcing users to run a complete x86 virtual machine in order to keep them safe from the arbitrary code that these actors are delivering.
People who write their own code to run on their own machine can write Python directly. They don’t need a web browser. Hell, they can bundle Flask and provide a localhost website that runs Python code. Those users are currently served just fine.
Python will work just fine on it if you don’t add the bizarre requirement to be able to access the user’s machine from inside the browser sandbox. Python on the web is a laudable goal, and should be pursued, but the idea of a Python-on-the-web as powerful as Python-on-the-machine is not. No-one wants their website Javascript to be able to call into a Node.js installed on the user’s machine, because they know that’s absurd. We shouldn’t want Python to be able to do it either. Anyway, at this point I think we’re about as off-topic as we can get for this list, so I’m stepping back out of this conversation now. Feel free to follow-up off-list. Cory
On 11 April 2016 at 07:24, Keith Curtis <keithcu@gmail.com> wrote:
How? Are you offer to pay someone to bring them down? Are you offering to work on it yourself? Are you offering to help with the current workflow improvement efforts that are designed to make more efficient use of the limited pool of available contributor time? It's very easy to say "Hey, these metrics are going in the wrong direction". It's very hard to do something productive about it, rather than just complaining on the internet.
Then you'd be incorrect (they may pay for it as part of something else, like a Red Hat subscription, or an Azure/AWS/GCE/Heroku/OpenShift account, but they're paying for it). It's true that *open source and free software communities* are very bad at ensuring the development processes for the software we use are sustainable, but that seems to be because we're bad at supply chain management in general and even more confused than most by the difference between "duplicating an already existing piece of software can readily be made zero cost" and "ensuring a piece of software remains useful as the world around it changes requires ongoing investments of time and money".
But is that a fun way for volunteers to prioritise their time, as compared to working on things that interest them personally, or that they otherwise find inherently rewarding? Many of the oldest issues remain open because they're rare, easily worked around, hard to fix, only arguably a bug, or some combination of the above. Even reviewing them to see if they're still valid can be time consuming. A lot of projects deal with that problem by automatically closing old issues that haven't been touched in a while, which seems to be a text book case of gaming the system - once you treat "number of open issues" as an important metric, you've created an incentive to "default to closing", even if the underlying problem hasn't been addressed.
What anti-capitalist mindset? "If someone wants me to fix bugs for their reasons rather than mine, they can pay me" is a quintessentiallity capitalist attitude. I only add the "corporations and large organisations" qualifier because they're more likely to be able to afford to pay someone, and less likely to have volunteers willing to work on their problems for altruistic reasons.
People should fix bugs because it helps real Python users, and gets rid of barriers.
This is still guilt tripping volunteers - neither "you're not volunteering enough" nor "you're volunteering for the wrong things" are acceptable messages to send to folks that are already contributing (and folks that send that message regardless of its inappropriateness are a major factor in open source contributors burning out and quitting community contributions entirely, so it has the exact opposite effect of the intended one). By contrast, it's perfectly reasonable to let folks that have *spare* time they're willing to give to the community know that there are plenty of opportunities to contribute and provide info regarding some of the many areas where assistance would be helpful, as long as it's accompanied by the reminder that a sensible and sustainable personal priority order is "health, relationships, paid work, volunteer work". Even putting that question of healthy priorities aside, it's also the case that the vast majority of Pythonistas *aren't* affected by the edge cases that aren't handled correctly, or they're using third party libraries that already work around those issues. The latter is especially true for folks working across multiple Python versions.
As noted above, commercial projects tend to be far more ruthless about culling old "We're not going to invest resources in addressing this" issues. We prefer to leave them open in case they catch someone's interest (and because politely explaining to someone why you closed their issue report can itself be quite time consuming). However, I'll also reiterate my original point: if you want a customer experience, pay someone.
If few people feel ownership of the official Python, it can be bad. Maybe the PSF could hire more with that mindset.
There's a big difference between "not offering a customer experience for free" and not feeling ownership. Now, you may *want* a customer experience for free, but that doesn't make it a reasonable expectation.
This isn't a case where articulating the problem is at all helpful - we're well aware of the problem already (hence the metrics). The PSF is also aware of the concern, but it's a longer term challenge compared to other areas (such as the Python Package Index), so it's not something we're interested in driving from the Board level at this point in time. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Apr 11, 2016 at 6:51 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Maybe this is where someone like Keith can contribute? Go through a lot of old issues and inevitably there'll be some that you can reproduce with the version of Python that was current then, but can't repro with today's Python, and they can be closed as fixed. Doesn't take any knowledge of C, and maybe not even of Python (if there's a good enough test case there). ChrisA
On 11 April 2016 at 18:56, Chris Angelico <rosuav@gmail.com> wrote:
+1 As David Murray has pointed out on occasion, browsing through the tracker "oldest first" can also be interesting in terms of reading the discussions and seeing what leads to issues remaining open for a long time. Separating out "Open Enhancements" as a subcategory of "Open Issues" is also a longstanding "nice-to-have" for the metrics collection - the script for the weekly data collection is at https://hg.python.org/tracker/python-dev/file/tip/scripts/roundup-summary , while https://hg.python.org/tracker/python-dev/file/tip/scripts/issuestats.py pulls the time series created by those notifications and turns it into a chart. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 4/11/2016 4:56 AM, Chris Angelico wrote:
Or: look through old enhancement requests. There are many that were posted before python-ideas existed. Today, they would likely be posted here first, and if no support (as happens more often than not), never posted. Or is posted first to bugs, told to go to python-ideas for discussion. Anyone could repost old ideas to python-list now and post the result of discussion back to the tracker. If the discussion suggests rejection, then a coredev can close the issue. (I certainly would.) Closure is not erasure, so the idea is there permanently anyway. -- Terry Jan Reedy
Executive summary: Keep your shirts on, Python-Dev is scaling. Nick Coghlan writes:
Wrong question, in my opinion. "Why do you think so?" is what I'd ask. To me, only the last graph, which shows the closed and total growing at the same rate (more or less), and therefore suggests that Python development is on a stable path vis-a-vis bugginess, is particularly interesting. Growth in number of open issues in that situation is arguably a *good* thing. Why might the open issues be growing, even though the fraction of open issues (= 1 - the fraction of closed issues) is constant? (1) Users are reporting lots of issues. How would that happen? Python is getting lots of users, and they aren't going away because Python is "too buggy". Hard to see that as a bad thing. (2) Python has a growing amount of code to be buggy, and users are exercising the new code and reporting issues they encounter, and aren't going away because it's too buggy. Hard to see that as a bad thing. (3) The reported issues are duplicates reported because nobody's fixing an important subset. That's bad, but is it real? Well, can't disprove that just looking at the numbers, but (a) we'd notice the dupes (people are looking at the issue tracker, because issues are getting closed) and (b) the users would go away, but (1) and (2) say they aren't. The same "back of the envelope" analysis applies to the "issues with patches," I think. Not to be Pollyanna about it; there are problems with Python's issue management (and Nick knows them better than most). And perhaps there is an opportunity to leverage that stability, and improve the open to total ratio while maintaining user and feature growth. But AFAICS, those graphs don't really tell us anything we can act on (except to reassure us that the rumors that Python is about to be consumed by termites are unfounded).
On 11 April 2016 at 22:01, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Yeah, I have a biased perspective here as I have a vested interest in folks taking greater advantage of any support contracts they already have (since Red Hat's subscriptions are what ultimately pay my salary), and my pre-Red-Hat employment was with a large scale system integrator, so I'm now inclined to look for supply chain based solutions to capacity problems whenever there are credible commercial incentives to be found. Since keeping high touch customers out of community channels also helps reduce the overall support burden for volunteers, I'll generally advocate for that approach whenever the questions of increased development capacity or changes in focus come up :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Apr 11, 2016 at 12:24 AM, Keith Curtis <keithcu@gmail.com> wrote:
While I understand your point, it's more complicated than that. I spent some of my weekends to close/commit old issues on bugs.python.org and unfortunately I can say that issue triaging is one of the most unrewarding (you will probably get a "oh so it took three years to commit a two lines patch?" or "wow it took only two years to notice my problem" response) and time consuming (you will spend at least 30 minutes to understand what the issue is about) tasks in open source development. --Berker
On Fri, 8 Apr 2016 at 14:16 Keith Curtis <keithcu@gmail.com> wrote:
That bug count is misleading because it includes not only legitimate bugs but also enhancement proposals.
Two things. One, there is actually a good amount of performance work potentially landing in Python 3.6. Two, we have been saying Python is fast enough for most things for decades at this point, so this is an old issue that's never going away.
Once https://github.com/WebAssembly lands in the browser then it won't be as much of an issue. There are also various transpilers of Python to JavaScript already. I also tried back in 2006 but Mozilla rejected the idea so this has been tried before. :) -Brett
On 9 April 2016 at 07:15, Keith Curtis <keithcu@gmail.com> wrote:
Folks in institutional environments can most readily contribute to this by obtaining Python from commercial redistributors (rather than using the free community versions and then expecting free ongoing support from volunteers), and then funneling bug reports through their vendor rather than filing them directly with upstream (since donating bug fixes to commercial organisations for free isn't something most people consider a fun hobby).
This is about business dynamics and institutional supply chain management, not software, so wishing won't make it so. However, given the business challenges facing the vendors behind the Java projects you mention here, Python-based alternatives are also going to be a tough sell to potential investors. (That said, you may also be interested in the Apache Spark project, where Python, Java, Scala and R are the top tier analytics development languages)
People work on things in their own time because they find them enjoyable (or otherwise inherently rewarding), not because they're interested in facilitating increased corporate adoption. That said, yes, while it's high profile, contributing to CPython is one of the *least* effective ways of helping to improve the overall Python ecosystem in the near term, as it can literally take years to roll out major changes (as Python 2.6 is still the de facto baseline version, and there are many situations where folks are still using Python 2.4 and earlier). The best cases are those where we can define new APIs, protocols and idioms in ways that can also be adopted in earlier versions of the language by way of third party libraries and standard library backports.
There's nothing stopping anyone interested in this area from working on any kind of evangelisation they want to. However, what harm does it cause us personally if people decide to use other programming languages? Python tautologically fits the brains of Pythonistas, but that's nowhere near being the same thing as it being the right language for everyone for every purpose. It's also the case that any developer with only one language currently in their toolbox (even when that language is Python) is a developer with lots of learning opportunities ahead of them: http://www.curiousefficiency.org/posts/2015/10/languages-to-improve-your-pyt...
This misunderstands the nature of the relationship between JavaScript, CSS and the HTML Domain Object Model: these are technologies that have co-evolved for describing and dynamically updating user interfaces that interact with remote services over a network, and they're *really* good at it. Replicating that ecosystem in other program languages would technically be possible, but there's little incentive to do so given the work on WebAssembly and ongoing improvements in transpilers.
This is why people pay open source redistributors to ensure they have access to commercially supported components, rather than trusting that whatever they happened to find on the internet will continue to be maintained by anonymous benefactors.
"Some of them should be included" is not an actionable proposal. Anyone is free to propose specific additions to functools, and make the case for why those particular ones should be included in the standard library rather than continuing to be accessed via version independent 3rd party libraries like Toolz. (Advance warning: "I want a purely functional solution to a problem that can already be readily solved with procedural code" generally isn't accepted as a compelling justification) A simpler possibility might be to review the Functional Programming HOWTO at https://docs.python.org/3/howto/functional.html and consider ways that that might be updated to reference third part libraries, as well as potentially made more discoverable via the functools and itertools reference documentation.
What makes you think anyone here has the authority to tell anyone else how to spend their time? Folks work on community open source projects based on their personal interests and their commercial interests. While you're right that lots of things could stand to be improved, the best people to complain to about underinvestment (or misdirected investment) are commercial redistributors selling supported versions of CPython and other community projects in the Python ecosystem, rather than the already overcommitted volunteers contributing their own time for their own reasons.
Not the right list for that question.
Also not the right list.
Methods are just functions, so this is never going to happen. (When folks understand why, they've generally made a decent step forward in appreciating the differences between a procedural-first usage model for a language, vs an objects-first one) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Hi again all, Thanks for the replies. I don't have a studied opinion of what new methods should be added to functools, I mostly brought it up because it won't create an incompatible change and things like that should be much easier to do. Functools is one of the few remaining places in the core runtime where it seems what is provided is meant to be a sample or flavor, and not used on a daily basis without group-by, merge-sort, frequencies, pipe, take / drop, etc. I agree that more companies paying for Python support would be a great thing, but most often companies buy support at a different level of granularity: for example, RHEL where they get it for an entire OS, or hiring Django people to build and maintain a website. I wouldn't count on commercial redistributors of Python as ever being a big source of resources to fix the thousands of random bugs. The vast majority of people use the standard runtime. I realize that volunteers work on their own time and choice of tasks, but you've got quite a large community, and by periodically reporting on the bug count and having a goal to resolve old ones eventually, you will even find problem areas no one is talking about. Sometimes the issue isn't resources, but grit, focus, pride, etc. If you never weighed yourself, at looked at yourself in the mirror, you'd get heavier than you realized. You are mostly volunteers, but you can try to produce software as good as what paid professionals can do. Perhaps the bug list should be broken up into two categories: future optional or incompatible features, and code that people can easily agree are worth doing now. That way important holes don't get ignored for years. People should be able to trust the standard runtime. It didn't seem like WebAssembly would enable access to Numpy. It seems doubtful it will be much adopted if it can't access the rich Python runtime that might be installed. Python is much cleaner than some of the alternatives, so you should consider adoption as a humanitarian effort! Regards, -Keith
Hi Keith, On Sat, Apr 09, 2016 at 08:28:52PM -0400, Keith Curtis wrote: [...]
Python is much cleaner than some of the alternatives, so you should consider adoption as a humanitarian effort!
I keep seeing you telling us that "you should do this, you should do that, you you you..." but I don't see *you* volunteering to work on any of the bugs. Or do you see yourself more in a supervisory role? -- Steve
On 10 April 2016 at 10:28, Keith Curtis <keithcu@gmail.com> wrote:
For context, I work for Red Hat's developer experience team and occasionally put the graph at http://bugs.python.org/issue?@template=stats in front of our Python maintenance team and ask "This is a trend that worries me, what are we currently doing about it?". I'd encourage anyone using Python in a commercial or large organisational context that are getting their Python runtimes via a commercially supported Linux distro, Platform-as-a-Service provider, or one of the cross-platform CPython redistributors to take that graph, show it to their vendor and say: "These upstream CPython metrics worry us, what are you doing about them on our behalf?". (If you don't know how to ask your vendor that kind of question since another team handles the supplier relationship, then find that team and ask *them* what they're doing to address your concerns) For folks in those kinds of contexts running directly off the binaries or source code published by the Python Software Foundation, then my recommendation is a bit different: * option 1 is to advocate for switching to a commercial redistributor, and asking prospective vendors how they're supporting ongoing CPython maintenance as part of the vendor evaluation * option 2 is to advocate for becoming a PSF Sponsor Member and then advocate for a technical fellowship program aimed at general maintenance and project health monitoring for CPython (keeping in mind that "How do we manage it fairly and sustainably?" would actually be a bigger challenge for such a fellowship than funding it) Whether advocating for option 1 or option 2 makes more sense will vary by organisation, as it depends greatly on whether or not the organisation would prefer to work with a traditional commercial supplier, or engage directly with a charitable foundation that operates in the wider public interest. There's also option 3, which is to hire existing core developers and give them time to work on general upstream CPython maintenance, but there are actually relatively few organisations for which that strategy makes commercial sense (and they're largely going to be operating system companies, public cloud infrastructure companies, or commercial Python redistributors).
If that were an accurate assessment, I wouldn't need to write articles like http://www.curiousefficiency.org/posts/2015/04/stop-supporting-python26.html and the manylinux cross-distro Linux ABI wouldn't be based on the 9 year old CentOS 5 userspace. (We also wouldn't be needing to pursue redistributor-friendly proposals like PEP 493 to help propagate the network security enhancements in the 2.7.x series) Direct downloads from python.org are certainly a major factor on Windows (especially in educational contexts), but even there, many of the largest commercial adopters are using ActiveState's distribution (see http://www.activestate.com/customers ) or one of the analytical distributions from Enthought or Continuum Analytics.
So, you're suggesting we deliberately set out to make our volunteers feel bad in an effort to guilt them into donating more free work to corporations and other large organisations? When I phrase it in that deliberately negative way, I hope it becomes obvious why we *don't* do this. We do send out the weekly tracker metrics emails, and offers the statistics graphs on the tracker itself if people want to use them as parts of their own business cases for increased investment (whether direct or indirect) in upstream sustaining engineering for CPython.
You seem to be operating under some significant misapprehensions here. As far as I am aware, most of the core developers *are* paid professionals (even the folks that didn't start out that way) - we're just typically not paid specifically to work on CPython. Instead, we work on CPython to make sure it's fit for *our* purposes, or because we find it to be an enjoyable way to spend our free time. Engaging effectively with that environment means pursuing a "co-contributor" experience - we each make CPython better for our own purposes, and by doing so, end up making it better for everyone. The experience you seem to be seeking in this thread is that of a pure software consumer, and that's the role downstream redistributors fulfil in an open source ecosystem: they provide (usually for a fee) a traditional customer experience, with the redistributor taking care of the community engagement side of things.
Important holes don't get ignored for years - they get addressed, since people are motivated to invest the time to ensure they're fixed. That said, we do experience the traditional open source "community gap" where problems that are only important to folks that aren't yet personally adept at effective community engagement can languish indefinitely (hence my emphasis above on organisations considering the idea of outsourcing their community engagement efforts, rather than assuming that it's easy, or that someone else will take care of it for them without any particular personal motivation).
Project Jupyter already addressed that aspect by separating out the platform independent UI service for code entry from the language kernel backend for code evaluation. There's certainly a lot that could be done in terms of making Jupyter language kernels installable as web browser add-ons running native code, but that would be a question for the Project Jupyter folks, rather than here.
Python is much cleaner than some of the alternatives, so you should consider adoption as a humanitarian effort!
As tempting as it is to believe that "things that matter to software developers" dictate where the world moves, it's important to keep in mind that it's estimated we make up barely 0.25% of the world's population [1] - this means that for everyone that would consider themselves a "software developer", there's around 399 people who don't. Accordingly, if your motivation is "How do we help empower people to control the technology in their lives, rather than having it control them?" (which is a genuinely humanitarian effort), then I'd recommend getting involved with education-focused organisations like the Raspberry Pi Foundation, Software Carpentry and Django Girls, or otherwise helping to build bridges between the software development community and professional educators (see [2] for an example of the latter). When it comes to technology choices *within* that 0.25%, it's worth remembering that one of the key design principles of Python is *trusting developers to make their own decisions* (hence things like allowing monkeypatching as being incredibly useful in cases like testing, even while actively advising against it as a regular programming practice). That trust extends to trusting them to choose languages that are appropriate for them and their use cases. If that leads to them choosing Python, cool. If it leads to them choosing something else, that's cool, too - there are so many decent open source programming languages and runtimes out there these days that software developers are truly spoiled for choice, so providing a solid foundation for exploring those alternatives is at least as important a goal as reducing the incentives for people that already favour Python to need to explore them. Cheers, Nick. [1] http://www.infoq.com/news/2014/01/IDC-software-developers [2] https://2016.pycon-au.org/programme/python_in_education_seminar -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I feel like this demonstrates a confusion about how the web ecosystem works. Code that executes inside a web browser’s Javascript runtime does not have arbitrary access to the system. That’s not an arbitrary limitation, it’s vital, because code shipped over the web is inherently untrusted: it can be intercepted, manipulated, edited, and attacked. Each time the web application developers of the world want a new interface to system hardware they have to specify up a whole set of APIs for the browser Javascript to obtain that access in a way that is controlled and enabled by the user. Browser developers will allow code executing in the browser to access binaries on the host system *over their dead bodies*. Allowing that represents a terrifying security vulnerability. At no point will any sane browser developer allow that. As it turns out, of course, WebAssembly *would* enable access to NumPy because NumPy could simply be compiled to WebAssembly as well, and distributed along with the Python interpreter and all the other code you’d have to ship. The TL;DR here is: no web browser will ever allow access to a Python runtime (or any other runtime) that is installed. However, WebAssembly does not have the limitation you’ve suggested. Cory
Hi again, I mean this merely as food for thought. Thank you for reading. I personally find Python very stable, but the bug counts have been heading in the wrong direction: http://bugs.python.org/issue?@template=stats I recently discovered those charts, and they have valuable data that can be turned into action. My distributor of Python is Arch. I presume it is very close to what you guys release. That's just my example, but from what I've seen, few pay for a Python runtime. Given your license, I wouldn't expect a lot to. If someone waits for the wrong train, they won't get there. I don't suggest guilt-tripping volunteers into fixing bugs. If you've got to dig a ditch, you can try to enjoy the sun. Getting the bug count under control is an admirable goal. The list is an opportunity to focus on the known problems real people care about most, and to try to deal with old issues before taking on new ones. I don't recommend people fix bugs to help "corporations and large organizations". You won't be very motivated by that anti-capitalist mindset. People should fix bugs because it helps real Python users, and gets rid of barriers. If people steadily remove roadblocks, things will flourish. I was teasing about you guys not being paid professionals, however, it appears that very few of you have the goal of getting the official bug count down. That is a big difference between amateurs and professionals. If few people feel ownership of the official Python, it can be bad. Maybe the PSF could hire more with that mindset. I don't know the solutions, but I am grateful to be able to send you a few words. As for WebAssembly, I don't know if Numpy can "simply" be re-compiled for it. It seems like the sort of workitem that could take 2-3 years, and never be actually fully compatible or as fast. Python can run in sandboxes as well: https://wiki.python.org/moin/SandboxedPython. People who care about code being intercepted and manipulated should use SSL or sign it. People who write their own code to run on their own machine would likely prefer to be able to just directly reference the Python runtime they've already setup. I wonder whether the sandbox can be used outside of Web Assembly so that code distribution and security are not so intermingled. I did see someone write that WebAssembly is the "dawn of a new era", but I wonder whether it is mostly a bunch of Javascript people trying to solve their own problems rather than those who care about making Python work well on it. Regards, -Keith
I don’t think you’ve understood the problem here: you seem to be saying that we can solve the lack of trust issue by “rubbing some crypto on it”. But that doesn’t solve the problem at all. Let’s take this apart. “People who care about code being intercepted and manipulated”: if that code runs directly on the user’s machine then *everyone* should care. Put another way: it doesn’t matter what the *author* of the code cares about, it matters what the user cares about, and all users care about executing safe code! Otherwise, I can insert whatever Python code I like and run arbitrary code with the permissions of the browser. This opens the entire machine up to attack: an attacker can consume their CPU resources, transition into FS access, and basically do all kinds of wacky things. But wrapping this in SSL or signing the code doesn’t solve the problem at all. I opened a new browser window, turned my ad blocker off, and then loaded www.wired.com. The following domains loaded and executed Javascript code (this isn’t a complete list either, I gave up and got bored): - ads.rubiconproject.com - 5be16.v.fwmrm.net - optimized-by.rubiconproject.com - cdn.optimizely.com - rtb.adgrx.com - player.cnevids.com - dy48bnzanqw0v.cloudfront.net - cdn-akamai.mookie1.com - cdnjs.cloudflare.com - tpc.googlesyndication.com - securepubads.g.doubleclick.net - c.amazon-adsystem.com - s.update.rubiconproject.com - b.scorecardresearch.com - aax.amazon-adsystem.com - static.chartbeat.com - segment-data.zqtk.net - wired.disqus.com - animate.adobe.com - aam.wired.com - pagead2.googlesyndication.com - condenast.demdex.net - www.googletagservices.com - h5.adgear.com - dpm.demdex.net - a.adgear.com - i.yldbt.com - use.typekit.net - assets.adobedtm.com - z.moatads.com The point is that I, the user, did not consent to *any* of those. This is the way the web platform works: users don’t get asked who gets to execute code on their machine. Most users would not download a random binary to their machine from one of those domains and execute it. Wrapping that code in SSL or signing it prevents man-in-the-middle attacks on the code, but doesn’t in any sense prevent those actors listed there from doing terrible stuff! It is well known that Sony wrote an actual rootkit that they used as DRM and distributed it via CDs. If you believe that one of the above sites wouldn’t do something equally malicious with full access to my machine, you’re living in a fantasy world. This means that any code that those domains are allowed to execute needs to run in an absurdly restricted context: because neither I nor any other user trusts arbitrary domains to run arbitrary code! A Python sandbox that allows access to any code not distributed via the web browser is not restrictive enough. That Python tells you too much about the machine on which it is running. This is doubly bad if that Python is capable of calling into native code extensions distributed outside the browser (such as NumPy), because sandboxing code like that requires running a complete virtual machine. Either distributing that code would be a nightmare, or you’d be forcing users to run a complete x86 virtual machine in order to keep them safe from the arbitrary code that these actors are delivering.
People who write their own code to run on their own machine can write Python directly. They don’t need a web browser. Hell, they can bundle Flask and provide a localhost website that runs Python code. Those users are currently served just fine.
Python will work just fine on it if you don’t add the bizarre requirement to be able to access the user’s machine from inside the browser sandbox. Python on the web is a laudable goal, and should be pursued, but the idea of a Python-on-the-web as powerful as Python-on-the-machine is not. No-one wants their website Javascript to be able to call into a Node.js installed on the user’s machine, because they know that’s absurd. We shouldn’t want Python to be able to do it either. Anyway, at this point I think we’re about as off-topic as we can get for this list, so I’m stepping back out of this conversation now. Feel free to follow-up off-list. Cory
On 11 April 2016 at 07:24, Keith Curtis <keithcu@gmail.com> wrote:
How? Are you offer to pay someone to bring them down? Are you offering to work on it yourself? Are you offering to help with the current workflow improvement efforts that are designed to make more efficient use of the limited pool of available contributor time? It's very easy to say "Hey, these metrics are going in the wrong direction". It's very hard to do something productive about it, rather than just complaining on the internet.
Then you'd be incorrect (they may pay for it as part of something else, like a Red Hat subscription, or an Azure/AWS/GCE/Heroku/OpenShift account, but they're paying for it). It's true that *open source and free software communities* are very bad at ensuring the development processes for the software we use are sustainable, but that seems to be because we're bad at supply chain management in general and even more confused than most by the difference between "duplicating an already existing piece of software can readily be made zero cost" and "ensuring a piece of software remains useful as the world around it changes requires ongoing investments of time and money".
But is that a fun way for volunteers to prioritise their time, as compared to working on things that interest them personally, or that they otherwise find inherently rewarding? Many of the oldest issues remain open because they're rare, easily worked around, hard to fix, only arguably a bug, or some combination of the above. Even reviewing them to see if they're still valid can be time consuming. A lot of projects deal with that problem by automatically closing old issues that haven't been touched in a while, which seems to be a text book case of gaming the system - once you treat "number of open issues" as an important metric, you've created an incentive to "default to closing", even if the underlying problem hasn't been addressed.
What anti-capitalist mindset? "If someone wants me to fix bugs for their reasons rather than mine, they can pay me" is a quintessentiallity capitalist attitude. I only add the "corporations and large organisations" qualifier because they're more likely to be able to afford to pay someone, and less likely to have volunteers willing to work on their problems for altruistic reasons.
People should fix bugs because it helps real Python users, and gets rid of barriers.
This is still guilt tripping volunteers - neither "you're not volunteering enough" nor "you're volunteering for the wrong things" are acceptable messages to send to folks that are already contributing (and folks that send that message regardless of its inappropriateness are a major factor in open source contributors burning out and quitting community contributions entirely, so it has the exact opposite effect of the intended one). By contrast, it's perfectly reasonable to let folks that have *spare* time they're willing to give to the community know that there are plenty of opportunities to contribute and provide info regarding some of the many areas where assistance would be helpful, as long as it's accompanied by the reminder that a sensible and sustainable personal priority order is "health, relationships, paid work, volunteer work". Even putting that question of healthy priorities aside, it's also the case that the vast majority of Pythonistas *aren't* affected by the edge cases that aren't handled correctly, or they're using third party libraries that already work around those issues. The latter is especially true for folks working across multiple Python versions.
As noted above, commercial projects tend to be far more ruthless about culling old "We're not going to invest resources in addressing this" issues. We prefer to leave them open in case they catch someone's interest (and because politely explaining to someone why you closed their issue report can itself be quite time consuming). However, I'll also reiterate my original point: if you want a customer experience, pay someone.
If few people feel ownership of the official Python, it can be bad. Maybe the PSF could hire more with that mindset.
There's a big difference between "not offering a customer experience for free" and not feeling ownership. Now, you may *want* a customer experience for free, but that doesn't make it a reasonable expectation.
This isn't a case where articulating the problem is at all helpful - we're well aware of the problem already (hence the metrics). The PSF is also aware of the concern, but it's a longer term challenge compared to other areas (such as the Python Package Index), so it's not something we're interested in driving from the Board level at this point in time. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Apr 11, 2016 at 6:51 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Maybe this is where someone like Keith can contribute? Go through a lot of old issues and inevitably there'll be some that you can reproduce with the version of Python that was current then, but can't repro with today's Python, and they can be closed as fixed. Doesn't take any knowledge of C, and maybe not even of Python (if there's a good enough test case there). ChrisA
On 11 April 2016 at 18:56, Chris Angelico <rosuav@gmail.com> wrote:
+1 As David Murray has pointed out on occasion, browsing through the tracker "oldest first" can also be interesting in terms of reading the discussions and seeing what leads to issues remaining open for a long time. Separating out "Open Enhancements" as a subcategory of "Open Issues" is also a longstanding "nice-to-have" for the metrics collection - the script for the weekly data collection is at https://hg.python.org/tracker/python-dev/file/tip/scripts/roundup-summary , while https://hg.python.org/tracker/python-dev/file/tip/scripts/issuestats.py pulls the time series created by those notifications and turns it into a chart. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 4/11/2016 4:56 AM, Chris Angelico wrote:
Or: look through old enhancement requests. There are many that were posted before python-ideas existed. Today, they would likely be posted here first, and if no support (as happens more often than not), never posted. Or is posted first to bugs, told to go to python-ideas for discussion. Anyone could repost old ideas to python-list now and post the result of discussion back to the tracker. If the discussion suggests rejection, then a coredev can close the issue. (I certainly would.) Closure is not erasure, so the idea is there permanently anyway. -- Terry Jan Reedy
Executive summary: Keep your shirts on, Python-Dev is scaling. Nick Coghlan writes:
Wrong question, in my opinion. "Why do you think so?" is what I'd ask. To me, only the last graph, which shows the closed and total growing at the same rate (more or less), and therefore suggests that Python development is on a stable path vis-a-vis bugginess, is particularly interesting. Growth in number of open issues in that situation is arguably a *good* thing. Why might the open issues be growing, even though the fraction of open issues (= 1 - the fraction of closed issues) is constant? (1) Users are reporting lots of issues. How would that happen? Python is getting lots of users, and they aren't going away because Python is "too buggy". Hard to see that as a bad thing. (2) Python has a growing amount of code to be buggy, and users are exercising the new code and reporting issues they encounter, and aren't going away because it's too buggy. Hard to see that as a bad thing. (3) The reported issues are duplicates reported because nobody's fixing an important subset. That's bad, but is it real? Well, can't disprove that just looking at the numbers, but (a) we'd notice the dupes (people are looking at the issue tracker, because issues are getting closed) and (b) the users would go away, but (1) and (2) say they aren't. The same "back of the envelope" analysis applies to the "issues with patches," I think. Not to be Pollyanna about it; there are problems with Python's issue management (and Nick knows them better than most). And perhaps there is an opportunity to leverage that stability, and improve the open to total ratio while maintaining user and feature growth. But AFAICS, those graphs don't really tell us anything we can act on (except to reassure us that the rumors that Python is about to be consumed by termites are unfounded).
On 11 April 2016 at 22:01, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Yeah, I have a biased perspective here as I have a vested interest in folks taking greater advantage of any support contracts they already have (since Red Hat's subscriptions are what ultimately pay my salary), and my pre-Red-Hat employment was with a large scale system integrator, so I'm now inclined to look for supply chain based solutions to capacity problems whenever there are credible commercial incentives to be found. Since keeping high touch customers out of community channels also helps reduce the overall support burden for volunteers, I'll generally advocate for that approach whenever the questions of increased development capacity or changes in focus come up :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Apr 11, 2016 at 12:24 AM, Keith Curtis <keithcu@gmail.com> wrote:
While I understand your point, it's more complicated than that. I spent some of my weekends to close/commit old issues on bugs.python.org and unfortunately I can say that issue triaging is one of the most unrewarding (you will probably get a "oh so it took three years to commit a two lines patch?" or "wow it took only two years to notice my problem" response) and time consuming (you will spend at least 30 minutes to understand what the issue is about) tasks in open source development. --Berker
participants (9)
-
Berker Peksağ -
Brett Cannon -
Chris Angelico -
Cory Benfield -
Keith Curtis -
Nick Coghlan -
Stephen J. Turnbull -
Steven D'Aprano -
Terry Reedy