From mike at selik.org Mon Feb 1 01:00:35 2016 From: mike at selik.org (Michael Selik) Date: Mon, 01 Feb 2016 06:00:35 +0000 Subject: [Python-ideas] A bit meta In-Reply-To: <85bn81eesp.fsf@benfinney.id.au> References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <56AB94A2.20901@stoneleaf.us> <20160130041001.GI4619@ando.pearwood.info> <2067CA09-F205-41A2-A0E0-17065DB1BE00@gmail.com> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> <85bn81eesp.fsf@benfinney.id.au> Message-ID: For what it's worth, my interest in starting this thread was, as a new participant to the list: - already overwhelmed by the frequency of emails - archives are difficult to search for relevant previous conversations - it's hard to observe agreement, but easy to observe disagreement - it's hard to observe the opinions of decision-makers - the last response in a conversation can feel like the consensus, creating an incentive to "get the last word" by repeating oneself, by writing in a style that discourages response, or by speaking on a different aspect of the topic without direct response (as I suppose I am doing here) The reason I latched onto Sjoerd's question of whether he had the right to say "+1" was to highlight the issue of observing agreement, especially the agreement of folks who can change the language. I'm not interested in a voting system for the purpose of having a vote, but so I can see who has been convinced of what, what clout they have, and whether another message is useful. On Sun, Jan 31, 2016, 2:21 PM Ben Finney wrote: > Brett Cannon writes: > > > For instance, people have said they don't want to set up another > > account. > > The complaint expressed (by me, at least; perhaps others agree) was not > against setting up an account. As you point out, PSF mailing lists > already require creating accounts. It's against being required to > maintain a trusted relationship with some non-PSF-accountable entity, in > order to participate in some aspect of Python community. > > I agree with others that a Discourse instance entirely controlled by PSF > would avoid that problem. > > -- > \ ?Consider the daffodil. And while you're doing that, I'll be | > `\ over here, looking through your stuff.? ?Jack Handey | > _o__) | > Ben Finney > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Feb 1 01:05:20 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 1 Feb 2016 17:05:20 +1100 Subject: [Python-ideas] A bit meta In-Reply-To: References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <56AB94A2.20901@stoneleaf.us> <20160130041001.GI4619@ando.pearwood.info> <2067CA09-F205-41A2-A0E0-17065DB1BE00@gmail.com> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> <85bn81eesp.fsf@benfinney.id.au> Message-ID: On Mon, Feb 1, 2016 at 5:00 PM, Michael Selik wrote: > For what it's worth, my interest in starting this thread was, as a new > participant to the list: > > - already overwhelmed by the frequency of emails > - archives are difficult to search for relevant previous conversations > - it's hard to observe agreement, but easy to observe disagreement > - it's hard to observe the opinions of decision-makers > - the last response in a conversation can feel like the consensus, creating > an incentive to "get the last word" by repeating oneself, by writing in a > style that discourages response, or by speaking on a different aspect of the > topic without direct response (as I suppose I am doing here) > > The reason I latched onto Sjoerd's question of whether he had the right to > say "+1" was to highlight the issue of observing agreement, especially the > agreement of folks who can change the language. I'm not interested in a > voting system for the purpose of having a vote, but so I can see who has > been convinced of what, what clout they have, and whether another message is > useful. It's worth noting that important decisions don't usually get made on -ideas alone. If something's controversial and needs a lot of what you're saying is hard to observe, it probably merits a PEP - the current state of consensus or disagreement can be detailed in a stand-alone editable document, rather than depending on mail archives. ChrisA From abarnert at yahoo.com Mon Feb 1 01:08:03 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 31 Jan 2016 22:08:03 -0800 Subject: [Python-ideas] A bit meta In-Reply-To: References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <56AB94A2.20901@stoneleaf.us> <20160130041001.GI4619@ando.pearwood.info> <2067CA09-F205-41A2-A0E0-17065DB1BE00@gmail.com> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> <85bn81eesp.fsf@benfinney.id.au> Message-ID: On Jan 31, 2016, at 13:53, Nicholas Chammas wrote: > > I don?t think older generations of developers are intrinsically any more tolerant of bad UX than the younger generations are. They hate bad UX too, and they had to figure out their own solutions to make things better ? their email filters, their clients, their homegrown scripts, etc. ? when nothing better was available, and eventually settled into a flow that worked for them. And, from a later message: > I *have* been arguing that a modern web-based forum solves common discussion issues in a way that mailing lists cannot match. A modern forum can definitely be much better out-of-the-box than a mailing list and traditional mail tools. But a suite of mail tools configured over many years for one user's idiosyncratic needs can outdo anything general-purpose. And that's not even considering the fact that the oldster has adapted to his mail tools, just as much as he's adapted them to his needs, and would have to adapt again to anything new. Compare the case with text editors. For me, Emacs is much better than some new editor like Atom. But for a novice, I'd definitely suggest Atom over Emacs (even Aquamacs). The difference here is that I can use Emacs while you use Atom, and we won't even notice we're using different tools; if I want to use email while you use Discourse, it seems unavoidable that one of us is going to be participating as a second-class citizen. You're effectively forcing me (and, more importantly, some core devs) to change. You're not going to convince Stefan that he's wrong for preferring mailing lists, because he's _not_ wrong. You may, however, convince him that the overall benefit (helping bring in new blood, providing PSF-owned permalinks, etc.) is worth the cost to him. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Mon Feb 1 03:12:23 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 1 Feb 2016 09:12:23 +0100 Subject: [Python-ideas] A bit meta In-Reply-To: References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <56AB94A2.20901@stoneleaf.us> <20160130041001.GI4619@ando.pearwood.info> <2067CA09-F205-41A2-A0E0-17065DB1BE00@gmail.com> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> <85bn81eesp.fsf@benfinney.id.au> Message-ID: <56AF1367.6010206@egenix.com> On 31.01.2016 22:11, Nicholas Chammas wrote: >> If both Discourse and Mailman can live side-by-side, with >> Discourse being the ?web interface? to the Mailman list,I think we?d get >> the best of both worlds. > > Funny you ask that, since I wondered about exactly the same thing when I > looked into using Discourse for an Apache project. The Apache Software > Foundation has a strict policy about ASF-owned mailing lists being the > place of discussion, so the only way Discourse would have been able to play > a role was as an interface to an existing, ASF-owned mailing list. > > Here is the discussion I started about this > > on Discourse Meta around a year ago. > > In short, I think the answer that came out from that discussion is (quoting > > Jeff Atwood; emphasis his): > > """ > This really depends on the culture of the mailing list. Discourse has > fairly robust email support (for notifications, and if configured, for > replies and email-in to start new topics), but it is still fundamentally > web-centric in the way that it views the world. There will be clashes for > people who are 100% email-centric. > > Do you have support from the ?powers that be? at said mailing lists to make > such a change? Are they asking for such a change? We are very open to > working with a partner on migrating mailing lists and further enhancing the > mailing list support in Discourse, but it very much requires solid support > from the *leadership* and a significant part of the *community*. > > There?s a lot of friction involved in changes for groups! > """ Jeff's reply doesn't sound overly optimistic. If the only way to get Discourse working for python-ideas (or any other PSF mailing list) is to *switch* to it, I'm firmly -1 on that approach. Forums are nice for things like Stack Overflow which are focusing more on questions and answer, with just a single linear thread being active going from the question to the answer. For discussions, which often branch in multiple sub-threads and don't necessarily start with a clear questions and final answers, I find mailing lists much more practical and closer to real life discussions in groups. Mailing list discussion "features" like being able to overhear something in another thread and the jumping in to participate should not be underestimated either. I know that other tools have grown bridges between the UI client world and serial line communication protocols, e.g. Slack and IRC, which works reasonably well. If we could make that happen, I'd be +1 on giving Discourse a try in order to invite new input from people who prefer the forum style UI approach. PS: I've added some extra quoting chars to your reply. HTML emails don't work well for mailing lists - better use plain text to start with, so that the context is not lost when an email client or archiver converts messages to plain text :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 01 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From antoine at python.org Mon Feb 1 06:46:23 2016 From: antoine at python.org (Antoine Pitrou) Date: Mon, 1 Feb 2016 11:46:23 +0000 (UTC) Subject: [Python-ideas] A bit meta References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <20160129080349.GD4619@ando.pearwood.info> <56AB94A2.20901@stoneleaf.us> <20160130041001.GI4619@ando.pearwood.info> <2067CA09-F205-41A2-A0E0-17065DB1BE00@gmail.com> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> Message-ID: Stefan Krah writes: > > It does not sound interesting at all -- Python development is increasingly > turning into a circus, with fewer and fewer people actually writing code. An interesting metric is to compute how much the set of posters on this "meta" thread intersects the set of the most active developers: $ hg churn -c -d "2015 to 2016" storchaka at gmail.com 1142 ************************************* victor.stinner at gmail.com 803 ************************** benjamin at python.org 582 ******************* yselivanov at sprymix.com 441 ************** tjreedy at udel.edu 394 ************* steve.dower at microsoft.com 302 ********** berker.peksag at gmail.com 285 ********* python at rcn.com 265 ********* zachary.ware at gmail.com 236 ******** vadmium+py at gmail.com 220 ******* larry at hastings.org 158 ***** nad at acm.org 121 **** rdmurray at bitdance.com 113 **** rbtcollins at hp.com 106 *** [...] (of course this is not entirely fair, since some people like Donald are quite active in other related areas) Regards Antoine. From donald at stufft.io Mon Feb 1 07:08:27 2016 From: donald at stufft.io (Donald Stufft) Date: Mon, 1 Feb 2016 07:08:27 -0500 Subject: [Python-ideas] A bit meta In-Reply-To: References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <20160129080349.GD4619@ando.pearwood.info> <56AB94A2.20901@stoneleaf.us> <20160130041001.GI4619@ando.pearwood.info> <2067CA09-F205-41A2-A0E0-17065DB1BE00@gmail.com> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> Message-ID: > On Feb 1, 2016, at 6:46 AM, Antoine Pitrou wrote: > > Stefan Krah writes: >> >> It does not sound interesting at all -- Python development is increasingly >> turning into a circus, with fewer and fewer people actually writing code. > > An interesting metric is to compute how much the set of posters on this > "meta" thread intersects the set of the most active developers: > Arguably changes like this are not primarily to benefit the people who are already contributing, those people have already decided that the current tooling isn't enough of a deterence to not contribute, but either allow new people to more easily contribute (in this case, via discussion) or people who are small time contributors to contribute more often. Of course, you don't want to ignore the people who are already contributing or make a decision solely focused on trying to attract new contributors (or increase contributions from small time contributors) since you run the risk of alienating the folks currently doing most of the work and then not getting any new contributions anyways. More to the topic at hand, I think the mailing list interface is kind of crummy and relies on people hand tailoring a bunch of filters to make it in any way reasonable. I have no idea if discourse or mailman3 is any better or if they are just differently bad, though on the packaging side of things I'd love to at least experiment with discourse since I think the *idea* of it is very nice, but no idea how it works in practice. There was a post earlier on about someone's experience with using discourse entirely in the "pretend it's a mailing list" mode and there were some pretty decent flaws, which is kind of disheartening. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From rustompmody at gmail.com Mon Feb 1 08:22:11 2016 From: rustompmody at gmail.com (Rustom Mody) Date: Mon, 1 Feb 2016 05:22:11 -0800 (PST) Subject: [Python-ideas] A bit meta In-Reply-To: References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <56AB94A2.20901@stoneleaf.us> <20160130041001.GI4619@ando.pearwood.info> <2067CA09-F205-41A2-A0E0-17065DB1BE00@gmail.com> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> <85bn81eesp.fsf@benfinney.id.au> Message-ID: <506f4995-f5ca-4431-abe9-ed83f33c0406@googlegroups.com> On Monday, February 1, 2016 at 11:38:51 AM UTC+5:30, Andrew Barnert via Python-ideas wrote: > > On Jan 31, 2016, at 13:53, Nicholas Chammas > wrote: > > I don?t think older generations of developers are intrinsically any more > tolerant of bad UX than the younger generations are. They hate bad UX too, > and they had to figure out their own solutions to make things better ? > their email filters, their clients, their homegrown scripts, etc. ? when > nothing better was available, and eventually settled into a flow that > worked for them. > > > And, from a later message: > > I *have* been arguing that a modern web-based forum solves common > discussion issues in a way that mailing lists cannot match. > > > A modern forum can definitely be much better out-of-the-box than a mailing > list and traditional mail tools. But a suite of mail tools configured over > many years for one user's idiosyncratic needs can outdo anything > general-purpose. And that's not even considering the fact that the oldster > has adapted to his mail tools, just as much as he's adapted them to his > needs, and would have to adapt again to anything new. > > Compare the case with text editors. For me, Emacs is much better than some > new editor like Atom. But for a novice, I'd definitely suggest Atom over > Emacs (even Aquamacs). The difference here is that I can use Emacs while > you use Atom, and we won't even notice we're using different tools; if I > want to use email while you use Discourse, it seems unavoidable that one of > us is going to be participating as a second-class citizen. You're > effectively forcing me (and, more importantly, some core devs) to change. > > [Since you mentioned emacs...] Things can get bitrotten and worse simply by passage of time Here's my mail to emacs list about python mode(s) that seems to be getting brokener and brokener by the day http://lists.gnu.org/archive/html/help-gnu-emacs/2016-01/msg00372.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Mon Feb 1 08:25:50 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 1 Feb 2016 14:25:50 +0100 Subject: [Python-ideas] A bit meta In-Reply-To: <20160131133618.2350a79e@subdivisions.wooz.org> References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <20160130161726.2fce1648@anarchist.wooz.org> <20160131124036.6e0bd451@subdivisions.wooz.org> <56AE50EE.4080809@egenix.com> <20160131133618.2350a79e@subdivisions.wooz.org> Message-ID: <56AF5CDE.9010101@egenix.com> On 31.01.2016 19:36, Barry Warsaw wrote: > On Jan 31, 2016, at 07:22 PM, M.-A. Lemburg wrote: > >> Would it be possible to provide an integration of Mailman with >> Discourse ? > > Possible, I don't know, but on the wish list, yes! None of the core Mailman > developers have time for this, but we would gladly help and work with anybody > who wanted to look into this. > >> If both Discourse and Mailman can live side-by-side, with >> Discourse being the "web interface" to the Mailman list, >> I think we'd get the best of both worlds. > > Definitely. Also note that we'd like to build NNTP and IMAP support into > Mailman, again though it's lack of resources. > > If anybody wants to work on these areas, please contact us over in > mailman-developers at python.org Would this be something the PSF could help kickstart with a development grant ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 01 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From barry at python.org Mon Feb 1 09:56:05 2016 From: barry at python.org (Barry Warsaw) Date: Mon, 1 Feb 2016 09:56:05 -0500 Subject: [Python-ideas] A bit meta In-Reply-To: <56AF5CDE.9010101@egenix.com> References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <20160130161726.2fce1648@anarchist.wooz.org> <20160131124036.6e0bd451@subdivisions.wooz.org> <56AE50EE.4080809@egenix.com> <20160131133618.2350a79e@subdivisions.wooz.org> <56AF5CDE.9010101@egenix.com> Message-ID: <20160201095605.27e24f67@subdivisions.wooz.org> On Feb 01, 2016, at 02:25 PM, M.-A. Lemburg wrote: >Would this be something the PSF could help kickstart with >a development grant ? Indeed, it probably could be. Happy to discuss further, but let's take it off python-ideas. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From nicholas.chammas at gmail.com Mon Feb 1 12:47:07 2016 From: nicholas.chammas at gmail.com (Nicholas Chammas) Date: Mon, 01 Feb 2016 17:47:07 +0000 Subject: [Python-ideas] A bit meta In-Reply-To: References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <56AB94A2.20901@stoneleaf.us> <20160130041001.GI4619@ando.pearwood.info> <2067CA09-F205-41A2-A0E0-17065DB1BE00@gmail.com> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> <85bn81eesp.fsf@benfinney.id.au> Message-ID: On Mon, Feb 1, 2016 at 1:08 AM Andrew Barnert wrote: > if I want to use email while you use Discourse, it seems unavoidable that one of us is going to be participating as a second-class citizen. You're effectively forcing me (and, more importantly, some core devs) to change. Yes, at some level there has to be a compromise between the needs of the established developers, and the needs of new contributors, as Brett explained in an earlier email. As a newbie, I'm not in a position to understand how to strike that balance; all I can do is argue for the side I understand best. > You're not going to convince Stefan that he's wrong for preferring mailing lists, because he's _not_ wrong. I haven't said and wouldn't say that anyone is "wrong" for using tools that work for them. That's preposterous. This isn't a case of "You're *gonna* use Discourse and *you're gonna like it*!" Come on, is that really the message you got out of my writing? :) I've been making the case that Discourse can benefit newcomers and hopefully also the overall community by solving common problems better than other tools do. The focus is on what works better in the general case for most people. That's not to say that someone can't have figured out a better solution for themselves. The question is whether we want to expect everyone to arrive at the same solution, or give them something better out of the box. Nick -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Feb 1 13:02:11 2016 From: brett at python.org (Brett Cannon) Date: Mon, 01 Feb 2016 18:02:11 +0000 Subject: [Python-ideas] A bit meta In-Reply-To: <56AF1367.6010206@egenix.com> References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <56AB94A2.20901@stoneleaf.us> <20160130041001.GI4619@ando.pearwood.info> <2067CA09-F205-41A2-A0E0-17065DB1BE00@gmail.com> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> <85bn81eesp.fsf@benfinney.id.au> <56AF1367.6010206@egenix.com> Message-ID: On Mon, 1 Feb 2016 at 00:12 M.-A. Lemburg wrote: > On 31.01.2016 22:11, Nicholas Chammas wrote: > >> If both Discourse and Mailman can live side-by-side, with > >> Discourse being the ?web interface? to the Mailman list,I think we?d get > >> the best of both worlds. > > > > Funny you ask that, since I wondered about exactly the same thing when I > > looked into using Discourse for an Apache project. The Apache Software > > Foundation has a strict policy about ASF-owned mailing lists being the > > place of discussion, so the only way Discourse would have been able to > play > > a role was as an interface to an existing, ASF-owned mailing list. > > > > Here is the discussion I started about this > > < > https://meta.discourse.org/t/discourse-as-a-front-end-for-existing-asf-mailing-lists/23167/2 > > > > on Discourse Meta around a year ago. > > > > In short, I think the answer that came out from that discussion is > (quoting > > < > https://meta.discourse.org/t/discourse-as-a-front-end-for-existing-asf-mailing-lists/23167/2?u=nicholaschammas > > > > Jeff Atwood; emphasis his): > > > > """ > > This really depends on the culture of the mailing list. Discourse has > > fairly robust email support (for notifications, and if configured, for > > replies and email-in to start new topics), but it is still fundamentally > > web-centric in the way that it views the world. There will be clashes for > > people who are 100% email-centric. > > > > Do you have support from the ?powers that be? at said mailing lists to > make > > such a change? Are they asking for such a change? We are very open to > > working with a partner on migrating mailing lists and further enhancing > the > > mailing list support in Discourse, but it very much requires solid > support > > from the *leadership* and a significant part of the *community*. > > > > There?s a lot of friction involved in changes for groups! > > """ > > Jeff's reply doesn't sound overly optimistic. > > If the only way to get Discourse working for python-ideas > (or any other PSF mailing list) is to *switch* to it, I'm > firmly -1 on that approach. > > Forums are nice for things like Stack Overflow which are focusing > more on questions and answer, with just a single linear > thread being active going from the question to the answer. > > For discussions, which often branch in multiple sub-threads and > don't necessarily start with a clear questions and final answers, > I find mailing lists much more practical and closer to real life > discussions in groups. Mailing list discussion "features" like > being able to overhear something in another thread and the jumping in > to participate should not be underestimated either. > I don't quite understand how any of that is exclusive to a mailing list? If a forum has thread topics which are clearly marked with new content since the last time you visited then how is that any different then a threaded email client that tells you have new mail on that thread? > > I know that other tools have grown bridges between the UI client > world and serial line communication protocols, e.g. Slack and IRC, > which works reasonably well. If we could make that happen, > I'd be +1 on giving Discourse a try in order to invite new > input from people who prefer the forum style UI approach. > I guess what we need is someone who is going to want to stay on the email-based side of things to look at the feature set of Discourse and let us know whether its feature set is adequate, or if not what is falls short of. And I have not heard what HyperKitty offers either. > > PS: I've added some extra quoting chars to your reply. HTML > emails don't work well for mailing lists - better use plain > text to start with, so that the context is not lost when > an email client or archiver converts messages to plain text :-) > That assumes you can even do that, e.g., I use Google Inbox and there is no plain text option. I'm afraid this is an example of the OSS community trying to swim against the stream where the rest of the world has moved on and it is slowly making it harder to get new people to participate in OSS. -Brett > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts (#1, Feb 01 2016) > >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ > >>> Python Database Interfaces ... http://products.egenix.com/ > >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and costs ::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > http://www.malemburg.com/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Mon Feb 1 13:49:23 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Mon, 1 Feb 2016 19:49:23 +0100 Subject: [Python-ideas] A bit meta In-Reply-To: References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <20160130041001.GI4619@ando.pearwood.info> <2067CA09-F205-41A2-A0E0-17065DB1BE00@gmail.com> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> <85bn81eesp.fsf@benfinney.id.au> <56AF1367.6010206@egenix.com> Message-ID: <56AFA8B3.70302@mail.de> 1) I like the current development of Python to be more community-friendly and visible! Big thanks to Brett (github). +1 Another step will definitely involve a more modern discussion structure. 2) Just give it a try and don't discuss so much about it. Too much bikeshedding. As Brett said, it's social. So, everybody is right of course. ;) 3) Nobody says its mandatory. Have several channels is perfectly fine. 4) Learning new things is great. Some guys here might think email is the best thing since sliced bread. No! It is not. Not saying Discourse is better, but it has definitely more visibility and a more modern UI (well, it actually HAS an UI). Everything else remains to be seen. If you want fresh blood, visibility counts. Best, Sven From mal at egenix.com Mon Feb 1 14:23:21 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 1 Feb 2016 20:23:21 +0100 Subject: [Python-ideas] A bit meta In-Reply-To: References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <20160130041001.GI4619@ando.pearwood.info> <2067CA09-F205-41A2-A0E0-17065DB1BE00@gmail.com> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> <85bn81eesp.fsf@benfinney.id.au> <56AF1367.6010206@egenix.com> Message-ID: <56AFB0A9.2010107@egenix.com> On 01.02.2016 19:02, Brett Cannon wrote: > On Mon, 1 Feb 2016 at 00:12 M.-A. Lemburg wrote: >> Jeff's reply doesn't sound overly optimistic. >> >> If the only way to get Discourse working for python-ideas >> (or any other PSF mailing list) is to *switch* to it, I'm >> firmly -1 on that approach. >> >> Forums are nice for things like Stack Overflow which are focusing >> more on questions and answer, with just a single linear >> thread being active going from the question to the answer. >> >> For discussions, which often branch in multiple sub-threads and >> don't necessarily start with a clear questions and final answers, >> I find mailing lists much more practical and closer to real life >> discussions in groups. Mailing list discussion "features" like >> being able to overhear something in another thread and the jumping in >> to participate should not be underestimated either. >> > > I don't quite understand how any of that is exclusive to a mailing list? If > a forum has thread topics which are clearly marked with new content since > the last time you visited then how is that any different then a threaded > email client that tells you have new mail on that thread? I was referring to the standard feature of email clients to display the threads in a tree which shows the subthreads. I haven't seen such a feature in Discourse. Some forums and gmane use indentation to a similar effect. The way Discourse handles replies to in-thread postings is also a bit strange, since the replies to messages are shown both under the parent message and in the chronological order (i.e. twice): https://community.lsst.org/t/understanding-and-using-discourses-flat-threading/150 See e.g. https://community.lsst.org/t/preparing-the-dm-update-for-scientists-session/72/14 for an example. Forking off new topics inside a thread breaks the threading as well. In email you simply change the subject line. >> I know that other tools have grown bridges between the UI client >> world and serial line communication protocols, e.g. Slack and IRC, >> which works reasonably well. If we could make that happen, >> I'd be +1 on giving Discourse a try in order to invite new >> input from people who prefer the forum style UI approach. >> > > I guess what we need is someone who is going to want to stay on the > email-based side of things to look at the feature set of Discourse and let > us know whether its feature set is adequate, or if not what is falls short > of. I had already posted a few links to people summarizing their experience. > And I have not heard what HyperKitty offers either. > >> PS: I've added some extra quoting chars to your reply. HTML >> emails don't work well for mailing lists - better use plain >> text to start with, so that the context is not lost when >> an email client or archiver converts messages to plain text :-) >> > > That assumes you can even do that, e.g., I use Google Inbox and there is no > plain text option. I'm afraid this is an example of the OSS community > trying to swim against the stream where the rest of the world has moved on > and it is slowly making it harder to get new people to participate in OSS. Gmail sends both plain text and HTML, so this is not much of an issue. Mailman will pick the plain text for archiving. The formatting problems only occur when a mail client only sends HTML which then has to be converted to plain text by Mailman or other mail clients. Features such as indentation or highlighting in different colors are often lost in this conversion. Anyway, this whole discussion is way off-topic for the mailing list and I haven't really seen a compelling argument for switching away from mailing lists completely yet, though several who would like to see both the mailing list and a Discourse like client for people who prefer web UI and forum style discussions. I'd suggest to postpone any decision until better tools are available to make this happen. Email is not going to go away, but I certainly have seen a lot of other communication tools come and go - and I bet there's something to learn in that :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 01 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From brett at python.org Mon Feb 1 15:05:40 2016 From: brett at python.org (Brett Cannon) Date: Mon, 01 Feb 2016 20:05:40 +0000 Subject: [Python-ideas] A bit meta In-Reply-To: <56AFB0A9.2010107@egenix.com> References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <20160130041001.GI4619@ando.pearwood.info> <2067CA09-F205-41A2-A0E0-17065DB1BE00@gmail.com> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> <85bn81eesp.fsf@benfinney.id.au> <56AF1367.6010206@egenix.com> <56AFB0A9.2010107@egenix.com> Message-ID: On Mon, 1 Feb 2016 at 11:23 M.-A. Lemburg wrote: > On 01.02.2016 19:02, Brett Cannon wrote: > > On Mon, 1 Feb 2016 at 00:12 M.-A. Lemburg wrote: > >> Jeff's reply doesn't sound overly optimistic. > >> > >> If the only way to get Discourse working for python-ideas > >> (or any other PSF mailing list) is to *switch* to it, I'm > >> firmly -1 on that approach. > >> > >> Forums are nice for things like Stack Overflow which are focusing > >> more on questions and answer, with just a single linear > >> thread being active going from the question to the answer. > >> > >> For discussions, which often branch in multiple sub-threads and > >> don't necessarily start with a clear questions and final answers, > >> I find mailing lists much more practical and closer to real life > >> discussions in groups. Mailing list discussion "features" like > >> being able to overhear something in another thread and the jumping in > >> to participate should not be underestimated either. > >> > > > > I don't quite understand how any of that is exclusive to a mailing list? > If > > a forum has thread topics which are clearly marked with new content since > > the last time you visited then how is that any different then a threaded > > email client that tells you have new mail on that thread? > > I was referring to the standard feature of email clients to > display the threads in a tree which shows the subthreads. > I haven't seen such a feature in Discourse. Some forums and gmane > use indentation to a similar effect. > I don't think that is as standard as you think. I know for a fact that Gmail and Inbox don't have a tree view, and based on searches I don't think Yahoo or Hotmail/Outlook support it either. I also know for a fact that Outlook (web or desktop) don't support a tree view. All of them only support a conversation view where emails that part of the same conversation end up in a serialized thread of email (so you can think of it as a tree view of depth 2: folder and then conversation). The only place I have come across a tree view for email is in some desktop email clients and terminal-based ones which I'm willing to bet are not the way the majority of people read their email these days. > > The way Discourse handles replies to in-thread postings is > also a bit strange, since the replies to messages are shown > both under the parent message and in the chronological > order (i.e. twice): > > > https://community.lsst.org/t/understanding-and-using-discourses-flat-threading/150 > > See e.g. > > https://community.lsst.org/t/preparing-the-dm-update-for-scientists-session/72/14 > for an example. > > Forking off new topics inside a thread breaks the threading > as well. In email you simply change the subject line. > > >> I know that other tools have grown bridges between the UI client > >> world and serial line communication protocols, e.g. Slack and IRC, > >> which works reasonably well. If we could make that happen, > >> I'd be +1 on giving Discourse a try in order to invite new > >> input from people who prefer the forum style UI approach. > >> > > > > I guess what we need is someone who is going to want to stay on the > > email-based side of things to look at the feature set of Discourse and > let > > us know whether its feature set is adequate, or if not what is falls > short > > of. > > I had already posted a few links to people summarizing their > experience. > Yes, but what I'm talking about is someone who knows how the Python community has historically worked to write their own summary instead of asking all of us to read a bunch of separate summaries. Consider it a literature review or a summary of summaries. :) > > > And I have not heard what HyperKitty offers either. > > > >> PS: I've added some extra quoting chars to your reply. HTML > >> emails don't work well for mailing lists - better use plain > >> text to start with, so that the context is not lost when > >> an email client or archiver converts messages to plain text :-) > >> > > > > That assumes you can even do that, e.g., I use Google Inbox and there is > no > > plain text option. I'm afraid this is an example of the OSS community > > trying to swim against the stream where the rest of the world has moved > on > > and it is slowly making it harder to get new people to participate in > OSS. > > Gmail sends both plain text and HTML, so this is not much of > an issue. Mailman will pick the plain text for archiving. > The formatting problems only occur when a mail client only sends > HTML which then has to be converted to plain text by Mailman > or other mail clients. Features such as indentation or highlighting > in different colors are often lost in this conversion. > > Anyway, this whole discussion is way off-topic for the mailing > list I disagree with that. As I said in my reply to Nick Chammas, we occasionally need to re-evaluate our choice in tooling to make sure we are not ignoring improvements made in the wider world that we are ignoring. And this becomes especially important if the way we manage our communication becomes an impediment to people newcomers because the ramp-up costs are too high for what becomes a "unique" requirement we impose on others. > and I haven't really seen a compelling argument for > switching away from mailing lists completely yet, though > several who would like to see both the mailing list and > a Discourse like client for people who prefer web UI > and forum style discussions. > This whole discussion kicked off when the point was brought up that having some way to infer reputation would be useful. If it was obvious who on this list consistently posts in an off-topic fashion, then new people would know that they should probably ignore those people's emails until they are more familiar with the mailing lists. We have persistently had this problem where long-time subscribers know who to mute and thus spare themselves a lot of time and anguish with poor actors, while new people have no idea and then engage people in conversations that go nowhere. > > I'd suggest to postpone any decision until better tools are > available to make this happen. Email is not going to go away, > but I certainly have seen a lot of other communication tools > come and go - and I bet there's something to learn in that :-) As I have said before, this switch won't happen until someone takes up an evaluation to properly show what we would gain or lose from a switch to something like Discourse or HyperKitty. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Mon Feb 1 16:45:02 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 1 Feb 2016 22:45:02 +0100 Subject: [Python-ideas] A bit meta In-Reply-To: References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <2067CA09-F205-41A2-A0E0-17065DB1BE00@gmail.com> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> <85bn81eesp.fsf@benfinney.id.au> <56AF1367.6010206@egenix.com> <56AFB0A9.2010107@egenix.com> Message-ID: <56AFD1DE.3010402@egenix.com> On 01.02.2016 21:05, Brett Cannon wrote: > On Mon, 1 Feb 2016 at 11:23 M.-A. Lemburg wrote: >> Anyway, this whole discussion is way off-topic for the mailing >> list > > > I disagree with that. As I said in my reply to Nick Chammas, we > occasionally need to re-evaluate our choice in tooling to make sure we are > not ignoring improvements made in the wider world that we are ignoring. And > this becomes especially important if the way we manage our communication > becomes an impediment to people newcomers because the ramp-up costs are too > high for what becomes a "unique" requirement we impose on others. Brett, I understand where you are coming from, but I don't follow the ramp-up costs argument. What we should be after is better signal-to-noise ratio. Changing ramp-up costs doesn't necessarily correlate with this. >> and I haven't really seen a compelling argument for >> switching away from mailing lists completely yet, though >> several who would like to see both the mailing list and >> a Discourse like client for people who prefer web UI >> and forum style discussions. >> > > This whole discussion kicked off when the point was brought up that having > some way to infer reputation would be useful. If it was obvious who on this > list consistently posts in an off-topic fashion, then new people would know > that they should probably ignore those people's emails until they are more > familiar with the mailing lists. We have persistently had this problem > where long-time subscribers know who to mute and thus spare themselves a > lot of time and anguish with poor actors, while new people have no idea and > then engage people in conversations that go nowhere. This is something good moderators should know how to handle. I don't think technology is good at such social interactions. Such voting features often result in shy people who are really knowledgeable to stay away from discussions simply because they are afraid to get poor scores. >> I'd suggest to postpone any decision until better tools are >> available to make this happen. Email is not going to go away, >> but I certainly have seen a lot of other communication tools >> come and go - and I bet there's something to learn in that :-) > > As I have said before, this switch won't happen until someone takes up an > evaluation to properly show what we would gain or lose from a switch to > something like Discourse or HyperKitty. Fair enough. We can continue the discussion when that happens :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 01 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From kesavarapu.siva at gmail.com Wed Feb 3 03:34:53 2016 From: kesavarapu.siva at gmail.com (shiva prasanth) Date: Wed, 3 Feb 2016 14:04:53 +0530 Subject: [Python-ideas] List Comprehensions Message-ID: python input and output In [2]: z=[ x for x in range(10)] In [3]: z Out[3]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] In [4]: z=[x for x in range(10),100] # should be [ 0 , 1, 2, 3, 4, 5, 6, 7, 8, 9, 100] # but output is not as expected # following is the output # other wise it should show error In [5]: z Out[5]: [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], 100] -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritium-list at sdamon.com Wed Feb 3 04:01:02 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Wed, 03 Feb 2016 04:01:02 -0500 Subject: [Python-ideas] List Comprehensions In-Reply-To: References: Message-ID: <56B1C1CE.3040708@sdamon.com> That is working exactly as expected. You iterate over a tuple of a list (the range), and an int (100). On 2/3/2016 03:34, shiva prasanth wrote: > python input and output > > In [2]: z=[ x for x in range(10)] > > In [3]: z > Out[3]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] > > In [4]: z=[x for x in range(10),100] # should be [ 0 , 1, 2, 3, 4, 5, > 6, 7, 8, 9, 100] > > > # but output is not as expected > # following is the output > # other wise it should show error > > In [5]: z > Out[5]: [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], 100] > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Feb 3 04:05:30 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 3 Feb 2016 20:05:30 +1100 Subject: [Python-ideas] List Comprehensions In-Reply-To: <56B1C1CE.3040708@sdamon.com> References: <56B1C1CE.3040708@sdamon.com> Message-ID: On Wed, Feb 3, 2016 at 8:01 PM, Alexander Walters wrote: > That is working exactly as expected. You iterate over a tuple of a list > (the range), and an int (100). I have to say, though, I was a little confused at first, because I'm used to range objects not being lists. And trying it showed that the syntax has been tightened up somewhere: >>> [x for x in range(10),100] File "", line 1 [x for x in range(10),100] ^ SyntaxError: invalid syntax But in Python 2.7, it behaves as the OP describes. I wonder, does python-ideas need a bit more clarification somewhere that this is (almost) exclusively for the discussion of Python 3, as that's where new ideas can actually be implemented? ChrisA From tritium-list at sdamon.com Wed Feb 3 04:10:54 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Wed, 03 Feb 2016 04:10:54 -0500 Subject: [Python-ideas] List Comprehensions In-Reply-To: References: <56B1C1CE.3040708@sdamon.com> Message-ID: <56B1C41E.6040207@sdamon.com> On 2/3/2016 04:05, Chris Angelico wrote: > I have to say, though, I was a little confused at first, because I'm > used to range objects not being lists. And trying it showed that the > syntax has been tightened up somewhere: I don't know when that was done either, but it is apparently the creation of a tuple without the parens isn't permitted inside listcomps anymore. It probably makes them easier to parse. *shrug* From stephen at xemacs.org Wed Feb 3 05:10:03 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 3 Feb 2016 19:10:03 +0900 Subject: [Python-ideas] List Comprehensions In-Reply-To: References: <56B1C1CE.3040708@sdamon.com> Message-ID: <22193.53755.267634.331413@turnbull.sk.tsukuba.ac.jp> Chris Angelico writes: > I wonder, does python-ideas need a bit more clarification somewhere > that this is (almost) exclusively for the discussion of Python 3, > as that's where new ideas can actually be implemented? It's not that simple because sometimes ideas for Python 3 are intended to making porting or transition from Python 2 simpler. True, we hope that phase of Python 3 development is over by now, but you never know. %-Formatting for bytes was approved in March 2014 according to the PEP (and released in Python 3.5 last September). From rosuav at gmail.com Wed Feb 3 05:18:27 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 3 Feb 2016 21:18:27 +1100 Subject: [Python-ideas] List Comprehensions In-Reply-To: <22193.53755.267634.331413@turnbull.sk.tsukuba.ac.jp> References: <56B1C1CE.3040708@sdamon.com> <22193.53755.267634.331413@turnbull.sk.tsukuba.ac.jp> Message-ID: On Wed, Feb 3, 2016 at 9:10 PM, Stephen J. Turnbull wrote: > Chris Angelico writes: > > > I wonder, does python-ideas need a bit more clarification somewhere > > that this is (almost) exclusively for the discussion of Python 3, > > as that's where new ideas can actually be implemented? > > It's not that simple because sometimes ideas for Python 3 are intended > to making porting or transition from Python 2 simpler. True, we hope > that phase of Python 3 development is over by now, but you never know. > %-Formatting for bytes was approved in March 2014 according to the PEP > (and released in Python 3.5 last September). > Yes, that's true - but it's still a new feature for Python 3.5, not a change to Python 2. Same with reinstating the u"..." prefix for Unicode strings - clearly and solely for Py2 compat, but not a change to Py2. Anyone who's proposing changes to the language needs to at least be aware of Python 3, and have tried the code in question under it. (Yes, there are very rare exceptions, where changes DO happen in Py2 - hash randomization comes to mind - but if I were explaining the list's purpose to someone, I'd ignore those rarities in the interests of simplicity.) ChrisA From tritium-list at sdamon.com Wed Feb 3 05:22:57 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Wed, 03 Feb 2016 05:22:57 -0500 Subject: [Python-ideas] List Comprehensions In-Reply-To: References: <56B1C1CE.3040708@sdamon.com> <22193.53755.267634.331413@turnbull.sk.tsukuba.ac.jp> Message-ID: <56B1D501.1000408@sdamon.com> On 2/3/2016 05:18, Chris Angelico wrote: > Yes, that's true - but it's still a new feature for Python 3.5, not a > change to Python 2. Same with reinstating the u"..." prefix for > Unicode strings - clearly and solely for Py2 compat, but not a change > to Py2. Anyone who's proposing changes to the language needs to at > least be aware of Python 3, and have tried the code in question under > it. (Yes, there are very rare exceptions, where changes DO happen in > Py2 - hash randomization comes to mind - but if I were explaining the > list's purpose to someone, I'd ignore those rarities in the interests > of simplicity.) It's also not that simple, since 2.7 got put into a weird state that it will receive critical feature updates for at least 4 more years. We kinda have to be open to hearing about them on -ideas. (I do not foresee that happening a lot, but it is possible for say... a security related feature that needs to be added to 2.7 to be suggested here - like the tls and hmac changes) From tritium-list at sdamon.com Wed Feb 3 05:26:24 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Wed, 03 Feb 2016 05:26:24 -0500 Subject: [Python-ideas] List Comprehensions In-Reply-To: References: <56B1C1CE.3040708@sdamon.com> <22193.53755.267634.331413@turnbull.sk.tsukuba.ac.jp> Message-ID: <56B1D5D0.1050000@sdamon.com> On 2/3/2016 05:18, Chris Angelico wrote: > On Wed, Feb 3, 2016 at 9:10 PM, Stephen J. Turnbull wrote: >> Chris Angelico writes: >> >> > I wonder, does python-ideas need a bit more clarification somewhere >> > that this is (almost) exclusively for the discussion of Python 3, >> > as that's where new ideas can actually be implemented? >> >> It's not that simple because sometimes ideas for Python 3 are intended >> to making porting or transition from Python 2 simpler. True, we hope >> that phase of Python 3 development is over by now, but you never know. >> %-Formatting for bytes was approved in March 2014 according to the PEP >> (and released in Python 3.5 last September). >> > Yes, that's true - but it's still a new feature for Python 3.5, not a > change to Python 2. Same with reinstating the u"..." prefix for > Unicode strings - clearly and solely for Py2 compat, but not a change > to Py2. Anyone who's proposing changes to the language needs to at > least be aware of Python 3, and have tried the code in question under > it. (Yes, there are very rare exceptions, where changes DO happen in > Py2 - hash randomization comes to mind - but if I were explaining the > list's purpose to someone, I'd ignore those rarities in the interests > of simplicity.) > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ And of course my eyes missed about HALF the last paragraph there. This is the problem with being on a list during a bout of insomnia. From rosuav at gmail.com Wed Feb 3 05:27:16 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 3 Feb 2016 21:27:16 +1100 Subject: [Python-ideas] List Comprehensions In-Reply-To: <56B1D501.1000408@sdamon.com> References: <56B1C1CE.3040708@sdamon.com> <22193.53755.267634.331413@turnbull.sk.tsukuba.ac.jp> <56B1D501.1000408@sdamon.com> Message-ID: On Wed, Feb 3, 2016 at 9:22 PM, Alexander Walters wrote: > > > On 2/3/2016 05:18, Chris Angelico wrote: >> >> Yes, that's true - but it's still a new feature for Python 3.5, not a >> change to Python 2. Same with reinstating the u"..." prefix for Unicode >> strings - clearly and solely for Py2 compat, but not a change to Py2. Anyone >> who's proposing changes to the language needs to at least be aware of Python >> 3, and have tried the code in question under it. (Yes, there are very rare >> exceptions, where changes DO happen in Py2 - hash randomization comes to >> mind - but if I were explaining the list's purpose to someone, I'd ignore >> those rarities in the interests of simplicity.) > > It's also not that simple, since 2.7 got put into a weird state that it will > receive critical feature updates for at least 4 more years. We kinda have to > be open to hearing about them on -ideas. (I do not foresee that happening a > lot, but it is possible for say... a security related feature that needs to > be added to 2.7 to be suggested here - like the tls and hmac changes) Those usually also apply to Python 3, and the cases where they don't are *incredibly* rare. A lot rarer than the query that ought to have gone to python-list in which someone assumes Python 2 (usually without even saying anything about version) and asks for some new feature that already exists in Py3, or uses example code that doesn't work under Py3. But I'm just spouting nonsense here. People won't read descriptions anyway, so what difference does it make? :) ChrisA From steve at pearwood.info Wed Feb 3 06:04:03 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 3 Feb 2016 22:04:03 +1100 Subject: [Python-ideas] List Comprehensions In-Reply-To: References: Message-ID: <20160203110403.GK31806@ando.pearwood.info> On Wed, Feb 03, 2016 at 02:04:53PM +0530, shiva prasanth wrote: > In [4]: z=[x for x in range(10),100] # should be [ 0 , 1, 2, 3, 4, 5, 6, > 7, 8, 9, 100] > > # but output is not as expected > # following is the output > # other wise it should show error > In [5]: z > Out[5]: [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], 100] Your suggestion is not how iteration works in Python: py> for x in range(10), 100: ... print x ... [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] 100 To get the result you want, you need to do this: # Python 2 py> [x for x in range(10) + [100]] [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 100] # Python 3 py> from itertools import chain py> [x for x in chain(range(10), [100])] [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 100] I've sometimes thought that Python should have a iterator concatenation operator (perhaps &) which we could use: [x for x in range(5) & [100] & "spam"] => returns [0, 1, 2, 3, 4, 100, 's', 'p', 'a', 'm'] -- Steve From rosuav at gmail.com Wed Feb 3 06:14:34 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 3 Feb 2016 22:14:34 +1100 Subject: [Python-ideas] List Comprehensions In-Reply-To: <20160203110403.GK31806@ando.pearwood.info> References: <20160203110403.GK31806@ando.pearwood.info> Message-ID: On Wed, Feb 3, 2016 at 10:04 PM, Steven D'Aprano wrote: > I've sometimes thought that Python should have a iterator concatenation > operator (perhaps &) which we could use: > > [x for x in range(5) & [100] & "spam"] > => returns [0, 1, 2, 3, 4, 100, 's', 'p', 'a', 'm'] Might be problematic for generic iterables, as your three examples are; but if you explicitly request an iterator, it's not hard to make it support + or & for chaining: from itertools import chain _iter = iter class iter: def __init__(self, *args): self.iter = _iter(*args) def __add__(self, other): return type(self)(chain(self.iter, _iter(other))) __and__ = __add__ def __iter__(self): return self def __next__(self): return next(self.iter) print(list(iter(range(5)) & [100] & "spam")) Is that good enough? ChrisA From tjreedy at udel.edu Wed Feb 3 16:16:48 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 3 Feb 2016 16:16:48 -0500 Subject: [Python-ideas] Prevent importing yourself? In-Reply-To: <20160130065851.GB27229@sjoerdjob.com> References: <56ABEACA.8000707@nedbatchelder.com> <20160129234226.GA23485@sjoerdjob.com> <86B59981-4CA8-4E01-89D5-FA05884BC22C@yahoo.com> <20160130065851.GB27229@sjoerdjob.com> Message-ID: On 1/30/2016 1:58 AM, Sjoerd Job Postmus wrote: > Yes, indeed. That's what I was thinking of. I decided to write up a quick hack that added the filename to the exception string. > > sjoerdjob$ ../python mod_a.py > Traceback (most recent call last): > File "mod_a.py", line 4, in > print(parse(JSON_DATA)) > File "/home/sjoerdjob/dev/cpython/tmp/mod_b.py", line 4, in parse > return json.loads(blob) > AttributeError: module 'json' (loaded from /home/sjoerdjob/dev/cpython/tmp/json.py) has no attribute 'loads' Would it help is we added an AttributeError subclass, ModuleAttributeError and raise that instead module_getattro? > Here's the patch, in case anyone is interested. > > diff --git a/Objects/moduleobject.c b/Objects/moduleobject.c > index 24c5f4c..5cc144a 100644 > --- a/Objects/moduleobject.c > +++ b/Objects/moduleobject.c > @@ -654,17 +654,25 @@ module_repr(PyModuleObject *m) > static PyObject* > module_getattro(PyModuleObject *m, PyObject *name) > { > - PyObject *attr, *mod_name; > + PyObject *attr, *mod_name, *mod_file; > attr = PyObject_GenericGetAttr((PyObject *)m, name); > if (attr || !PyErr_ExceptionMatches(PyExc_AttributeError)) > return attr; > PyErr_Clear(); > if (m->md_dict) { > _Py_IDENTIFIER(__name__); > mod_name = _PyDict_GetItemId(m->md_dict, &PyId___name__); > if (mod_name) { > - PyErr_Format(PyExc_AttributeError, > + _Py_IDENTIFIER(__file__); > + mod_file = _PyDict_GetItemId(m->md_dict, &PyId___file__); > + if (mod_file && PyUnicode_Check(mod_file)) { > + PyErr_Format(PyExc_AttributeError, > + "module '%U' (loaded from %U) has no attribute '%U'", mod_name, mod_file, name); > + } else { > + PyErr_Format(PyExc_AttributeError, > "module '%U' has no attribute '%U'", mod_name, name); > + } > return NULL; > } > else if (PyErr_Occurred()) { > > Unfortunately, I do think this might impose **some** performance issue, but on > the other hand, I'd be inclined to think that attribute-errors on module > objects are not that likely to begin with, except for typos and issues > like these. (And of course the case that you have to support older > versions of Python with a slower implementation, but you most often see > those checks being done at the module-level, so it would only impact > load-time and not running-time.) > > The added benefit would be quicker debugging when finally having posted > to a forum: "Ah, I see from the message that the path of the module is > not likely a standard-library path. Maybe you have a name collision? > Check for files or directories named '(.py)' in your > working directory / project / ... . > >> >>> (Of course, another option would be to look for other modules of the >>> same name when you get an attribute-error on a module to aid debugging, >>> but I think that's too heavy-weight.) >> >> If that could be done only when the exception escapes to top level and dumps s traceback, that might be reasonable. And it would _definitely_ be helpful. But I don't think it's possible without major changes. > > No, indeed, that was also my expectation: helpful, but too big a hassle > to be worth it. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Terry Jan Reedy From tjreedy at udel.edu Wed Feb 3 17:30:55 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 3 Feb 2016 17:30:55 -0500 Subject: [Python-ideas] A bit meta In-Reply-To: References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <2067CA09-F205-41A2-A0E0-17065DB1BE00@gmail.com> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> <85bn81eesp.fsf@benfinney.id.au> Message-ID: On 1/31/2016 6:06 PM, Stefan Krah wrote: > I for example think that http://try.discourse.org/ looks cluttered > and distracting. I had the opposite experience. All I see, with Firefox, is a blank page -- even after accepting cookies and temporarily allowing at least one google spyscript. Please let us not move our infrastructure to a finicky spyscript cookie monster site. As near as I can tell, bugs.python.org sets 1 cookie per session, which expires at the end of each session, and does not try to run *any* any foreign scripts. -- Terry Jan Reedy From steve at pearwood.info Wed Feb 3 19:30:43 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 4 Feb 2016 11:30:43 +1100 Subject: [Python-ideas] Prevent importing yourself? In-Reply-To: <56AD5C49.1040406@nedbatchelder.com> References: <56ABEACA.8000707@nedbatchelder.com> <56AC9C47.50304@nedbatchelder.com> <20160130224718.GB31806@ando.pearwood.info> <56AD5C49.1040406@nedbatchelder.com> Message-ID: <20160204003042.GN31806@ando.pearwood.info> On Sat, Jan 30, 2016 at 07:58:49PM -0500, Ned Batchelder wrote: > On 1/30/16 5:47 PM, Steven D'Aprano wrote: > >On Sat, Jan 30, 2016 at 06:19:35AM -0500, Ned Batchelder wrote: > > > >>While we're at it though, re-importing __main__ is a separate kind of > >>behavior that is often a problem, since it means you'll have the same > >>classes defined twice. > > >As far as I can tell, importing __main__ is fine. It's only when you > >import __main__ AND the main module under its real name at the same time > >that you can run into problems -- and even then, not always. The sort of > >errors I've seen involve something like this: > > > >import myscript > >import __main__ # this is actually myscript > >a = myscript.TheClass() > ># later > >assert isinstance(a, __main__.TheClass) > > > >which fails, because myscript and __main__ don't share state, despite > >actually coming from the same source file. > > > >So I think it's pretty rare for something like this to actually happen. > >I've never seen it happen by accident, I've only seen it done > >deliberately as a counter-example to to the "modules are singletons" > >rule. > > Something like this does happen in the real world. A class is defined > in the main module, and then the module is later imported with its real > name. Now you have __main__.Class and module.Class both defined. You > don't need to actually "import __main__" for it to happen. > __main__.Class is used implicitly from the main module simply as Class. Ah, yes of course you are correct, you don't need an explicit "import __main__", but you do need an explicit "import module" somewhere. I think that in order to have a problem, you need something like this set of circumstances: (1) You need a module which is intended to be used as BOTH an executable script and an importable library. (2) Your module ends up directly or indirectly importing itself when running as __main__. (3) Your code relies on the "module is a singleton" invariant that you have just violated. (If you ever do something like `if type(obj) is MyClass`, you're relying on that invariant.) (If your module is never __main__, you have no problem. If your module is always __main__ and never imported under the real file name, you have no problem. But if it is both, you may have a problem.) I'm prepared to believe that actually diagnosing this error can be difficult, especially for those who have never come across this before. But I don't think it is especially common. Rather than trying to ban self imports, could we change sys.modules so it caches the module object under *both* the original name and '__main__'? # running "script.py" as the main module sys.modules['__main__'] = sys.modules['script'] = module_object # now "import script" will pick up the same module object as __main__ If the script name isn't a valid identifier, there's no need for the second entry, since it can't be imported. (I have a vague feeling I've already asked this question before, but I can't find any mail with it. If I have asked it, and it's been answered, my apologies. What was the answer again?) -- Steve From rosuav at gmail.com Wed Feb 3 20:04:51 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 4 Feb 2016 12:04:51 +1100 Subject: [Python-ideas] Prevent importing yourself? In-Reply-To: <20160204003042.GN31806@ando.pearwood.info> References: <56ABEACA.8000707@nedbatchelder.com> <56AC9C47.50304@nedbatchelder.com> <20160130224718.GB31806@ando.pearwood.info> <56AD5C49.1040406@nedbatchelder.com> <20160204003042.GN31806@ando.pearwood.info> Message-ID: On Thu, Feb 4, 2016 at 11:30 AM, Steven D'Aprano wrote: > (3) Your code relies on the "module is a singleton" invariant that you > have just violated. (If you ever do something like `if type(obj) is > MyClass`, you're relying on that invariant.) If ever you do "except MyException:", you're relying on that invariant, too. ChrisA From ncoghlan at gmail.com Thu Feb 4 07:42:24 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 4 Feb 2016 22:42:24 +1000 Subject: [Python-ideas] A bit meta In-Reply-To: References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <20160130041001.GI4619@ando.pearwood.info> <2067CA09-F205-41A2-A0E0-17065DB1BE00@gmail.com> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> <85bn81eesp.fsf@benfinney.id.au> <56AF1367.6010206@egenix.com> <56AFB0A9.2010107@egenix.com> Message-ID: On 2 February 2016 at 06:05, Brett Cannon wrote: > On Mon, 1 Feb 2016 at 11:23 M.-A. Lemburg wrote: >> I'd suggest to postpone any decision until better tools are >> available to make this happen. Email is not going to go away, >> but I certainly have seen a lot of other communication tools >> come and go - and I bet there's something to learn in that :-) > > As I have said before, this switch won't happen until someone takes up an > evaluation to properly show what we would gain or lose from a switch to > something like Discourse or HyperKitty. HyperKitty isn't a switch - it's the native web gateway that becomes available by way of upgrading to Mailman 3. The MM3 upgrade is worth doing for a whole host of reasons independently of gaining access to HyperKitty (e.g. one account per user with optional per-list subscription settings, rather than the MM2 model of user settings being stored independently for each list). If Mark is willing to shepherd an MM3 upgrade on a volunteer basis, that would be excellent, but I also expect it to be a fair bit of work. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From storchaka at gmail.com Thu Feb 4 08:03:23 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 4 Feb 2016 15:03:23 +0200 Subject: [Python-ideas] More compact bytecode Message-ID: Damien George mentioned [1] that in MicroPython there are 16 dedicated opcodes for loading from positions 0-15, and 16 for storing to these positions. This not only makes a bytecode more compact, but might make CPython slightly faster too. I have collected statistic for opcodes generated in CPython during running the compileall module for the Lib/ directory. Raw data file is attached. Here is top-16 most used opcodes. Column 4 is the percentage among all opcodes. Columns 5-9 are percents of opcodes with argument that exceeds specified width. # op name % >2bit >3bit >4bit >8bit >16bit 1. 100 LOAD_CONST 22.2 57.8 39.7 25.8 3.6 0 2. 124 LOAD_FAST 14.7 19.3 5.95 1.04 0 0 3. 106 LOAD_ATTR 10.6 49.5 23.1 6.61 0 0 4. 131 CALL_FUNCTION 10.3 8.54 5.68 5.6 5.6 0 5. 116 LOAD_GLOBAL 5.8 39.2 16.7 4.21 0 0 6. 1 POP_TOP 5.11 7. 90 STORE_NAME 4.04 69.5 50.4 33.1 4.03 0 8. 125 STORE_FAST 3.8 39.4 12.8 2.24 0 0 9. 83 RETURN_VALUE 2.68 10. 132 MAKE_FUNCTION 2.16 1.33 0.861 0.782 0.776 0.23 11. 101 LOAD_NAME 1.81 64 49.9 35.3 2.35 0 12. 114 POP_JUMP_IF_FALSE 1.39 100 100 99.5 27.4 0 13. 102 BUILD_TUPLE 1.28 12.4 2.54 0.211 0 0 14. 107 COMPARE_OP 1.15 56.6 36.3 0 0 0 15. 87 POP_BLOCK 0.89 16. 110 JUMP_FORWARD 0.734 75.1 69.6 58.2 2.36 0 13.7% of opcodes doesn't have an argument. None argument exceeds 24 bits. Only 0.005% opcodes have an argument exceeded 16 bits and required EXTENDED_ARG (these opcodes are MAKE_FUNCTION and MAKE_CLOSURE). Only 2% of opcodes have an argument exceeded 8 bits. This is mostly jumping opcodes. More that every 5th opcode is LOAD_CONST. Only few files like Lib/html/__init__.py or Lib/locale.py contain LOAD_CONST with argument larger than 1024. They contain a large number of constants. Arguments larger than 256 are used mostly in encoding files for constructing mappings that are never used. Mean size of opcode is 2.73 bytes. Here is top-32 of most used opcodes with arguments: # op name arg % cum% 1. 124 LOAD_FAST 0 5.76 5.76 2. 131 CALL_FUNCTION 1 4.27 10.03 3. 124 LOAD_FAST 1 3.10 13.13 4. 131 CALL_FUNCTION 2 2.87 16.00 5. 100 LOAD_CONST 1 2.81 18.80 6. 100 LOAD_CONST 0 2.67 21.47 7. 100 LOAD_CONST 2 2.20 23.67 8. 132 MAKE_FUNCTION 0 1.98 25.64 9. 124 LOAD_FAST 2 1.86 27.51 10. 100 LOAD_CONST 3 1.68 29.19 11. 131 CALL_FUNCTION 0 1.54 30.72 12. 106 LOAD_ATTR 0 1.44 32.16 13. 106 LOAD_ATTR 1 1.39 33.55 14. 106 LOAD_ATTR 2 1.36 34.91 15. 116 LOAD_GLOBAL 0 1.34 36.24 16. 100 LOAD_CONST 4 1.31 37.56 17. 124 LOAD_FAST 3 1.18 38.73 18. 106 LOAD_ATTR 3 1.17 39.91 19. 100 LOAD_CONST 5 1.09 41.00 20. 125 STORE_FAST 1 0.95 41.94 21. 106 LOAD_ATTR 4 0.94 42.88 22. 116 LOAD_GLOBAL 1 0.93 43.81 23. 100 LOAD_CONST 6 0.89 44.69 24. 124 LOAD_FAST 4 0.78 45.48 25. 131 CALL_FUNCTION 3 0.76 46.23 26. 106 LOAD_ATTR 5 0.74 46.98 27. 125 STORE_FAST 2 0.74 47.72 28. 100 LOAD_CONST 7 0.73 48.45 29. 116 LOAD_GLOBAL 2 0.73 49.18 30. 102 BUILD_TUPLE 2 0.68 49.86 31. 106 LOAD_ATTR 6 0.62 50.47 32. 100 LOAD_CONST 8 0.61 51.08 There are two ways to make Python opcodes more compact. 1. Add 1-byte dedicated opcodes for most used opcodes with arguments. If add 32 short opcodes for 6 most popular opcodes: LOAD_CONST and LOAD_ATTR with arguments from 0 to 7, LOAD_FAST, STORE_FAST, LOAD_GLOBAL and CALL_FUNCTION with arguments from 0 to 3, this will decrease mean size of opcode from 2.73 to 1.75 bytes (1.56 times). This might make decoding of 49% opcodes slightly faster due to avoiding decoding argument. This is relative simple and compatible way. But not fully compatible. We have either shift numbers for opcodes with arguments to free space for additional opcodes, or add new range for argumentless opcodes at the end of 8-bit space. Third-party tools that work with CPython bytecode should be updated. 2. Use 16-bit opcodes as in WPython. This will decrease mean size of opcode from 2.73 to 2.044 bytes (1.33 times). But this might make decoding of 85% opcodes slightly faster. Raymond shown a roughly 10+% speedup in code exercising simple and common opcodes that that have a oparg when just optimize oparg decoding. [2] Using 16-bit opcodes with embedded oparg might have even larger effect. This change needs more serious reworking of compiler and interpreter, not only disassembler and third-party tools. [1] http://permalink.gmane.org/gmane.comp.python.devel/156162 [2] http://bugs.python.org/issue25823 -------------- next part -------------- { 1: {None: 128632}, 2: {None: 758}, 3: {None: 437}, 4: {None: 6248}, 5: {None: 103}, 10: {None: 45}, 11: {None: 7096}, 12: {None: 4797}, 15: {None: 103}, 16: {None: 14}, 17: {None: 8}, 19: {None: 922}, 20: {None: 4071}, 22: {None: 5841}, 23: {None: 10504}, 24: {None: 2800}, 25: {None: 16725}, 26: {None: 394}, 27: {None: 918}, 28: {None: 30}, 29: {None: 17}, 50: {None: 18}, 51: {None: 18}, 52: {None: 34}, 55: {None: 1719}, 56: {None: 267}, 57: {None: 114}, 59: {None: 17}, 60: {None: 4190}, 61: {None: 732}, 62: {None: 432}, 63: {None: 155}, 64: {None: 704}, 65: {None: 134}, 66: {None: 814}, 67: {None: 9}, 68: {None: 9847}, 69: {None: 403}, 71: {None: 9407}, 72: {None: 566}, 73: {None: 127}, 75: {None: 12}, 76: {None: 6}, 77: {None: 38}, 78: {None: 26}, 79: {None: 167}, 80: {None: 987}, 81: {None: 6260}, 82: {None: 6260}, 83: {None: 67484}, 84: {None: 242}, 86: {None: 1547}, 87: {None: 22382}, 88: {None: 13282}, 89: {None: 4924}, 90: {0: 1437, 1: 10625, 2: 10644, 3: 8239, 4: 6181, 5: 5284, 6: 4341, 7: 3669, 8: 3234, 9: 2691, 10: 2554, 11: 2155, 12: 1978, 13: 1822, 14: 1650, 15: 1493, 16: 1381, 17: 1244, 18: 1162, 19: 1095, 20: 993, 21: 940, 22: 896, 23: 837, 24: 791, 25: 769, 26: 714, 27: 687, 28: 640, 29: 586, 30: 594, 31: 534, 32: 509, 33: 474, 34: 434, 35: 439, 36: 415, 37: 405, 38: 408, 39: 342, 40: 345, 41: 336, 42: 312, 43: 309, 44: 297, 45: 272, 46: 267, 47: 264, 48: 250, 49: 236, 50: 229, 51: 218, 52: 212, 53: 211, 54: 201, 55: 203, 56: 194, 57: 183, 58: 179, 59: 163, 60: 168, 61: 154, 62: 137, 63: 139, 64: 134, 65: 131, 66: 131, 67: 130, 68: 129, 69: 126, 70: 122, 71: 114, 72: 106, 73: 97, 74: 110, 75: 96, 76: 96, 77: 95, 78: 90, 79: 93, 80: 83, 81: 88, 82: 81, 83: 79, 84: 81, 85: 77, 86: 73, 87: 74, 88: 73, 89: 71, 90: 72, 91: 68, 92: 64, 93: 65, 94: 64, 95: 61, 96: 58, 97: 54, 98: 56, 99: 58, 100: 53, 101: 52, 102: 55, 103: 53, 104: 55, 105: 49, 106: 47, 107: 48, 108: 50, 109: 48, 110: 48, 111: 42, 112: 45, 113: 43, 114: 47, 115: 42, 116: 38, 117: 40, 118: 41, 119: 35, 120: 38, 121: 37, 122: 41, 123: 38, 124: 41, 125: 38, 126: 38, 127: 29, 128: 34, 129: 31, 130: 33, 131: 29, 132: 29, 133: 29, 134: 32, 135: 27, 136: 27, 137: 27, 138: 30, 139: 29, 140: 29, 141: 25, 142: 26, 143: 26, 144: 24, 145: 23, 146: 22, 147: 23, 148: 23, 149: 24, 150: 23, 151: 25, 152: 21, 153: 22, 154: 20, 155: 21, 156: 22, 157: 23, 158: 21, 159: 19, 160: 20, 161: 19, 162: 20, 163: 19, 164: 20, 165: 19, 166: 20, 167: 18, 168: 19, 169: 18, 170: 19, 171: 18, 172: 18, 173: 18, 174: 16, 175: 16, 176: 16, 177: 16, 178: 16, 179: 16, 180: 16, 181: 16, 182: 16, 183: 16, 184: 16, 185: 16, 186: 15, 187: 16, 188: 17, 189: 16, 190: 16, 191: 16, 192: 16, 193: 16, 194: 16, 195: 16, 196: 16, 197: 16, 198: 16, 199: 16, 200: 16, 201: 16, 202: 16, 203: 16, 204: 15, 205: 14, 206: 15, 207: 17, 208: 16, 209: 16, 210: 16, 211: 16, 212: 16, 213: 16, 214: 16, 215: 14, 216: 14, 217: 14, 218: 14, 219: 14, 220: 14, 221: 14, 222: 13, 223: 13, 224: 14, 225: 14, 226: 13, 227: 14, 228: 14, 229: 13, 230: 14, 231: 17, 232: 17, 233: 17, 234: 17, 235: 16, 236: 14, 237: 14, 238: 13, 239: 13, 240: 13, 241: 13, 242: 14, 243: 14, 244: 20, 245: 18, 246: 12, 247: 13, 248: 13, 249: 13, 250: 13, 251: 13, 252: 14, 253: 13, 254: 13, 255: 13, 256: 13, 257: 13, 258: 13, 259: 13, 260: 13, 261: 13, 262: 11, 263: 13, 264: 11, 265: 13, 266: 13, 267: 13, 268: 13, 269: 13, 270: 13, 271: 13, 272: 13, 273: 13, 274: 13, 275: 13, 276: 13, 277: 13, 278: 13, 279: 13, 280: 13, 281: 13, 282: 13, 283: 13, 284: 13, 285: 13, 286: 13, 287: 14, 288: 14, 289: 14, 290: 16, 291: 14, 292: 14, 293: 14, 294: 14, 295: 15, 296: 12, 297: 12, 298: 12, 299: 12, 300: 12, 301: 12, 302: 12, 303: 12, 304: 10, 305: 12, 306: 12, 307: 12, 308: 12, 309: 11, 310: 11, 311: 11, 312: 13, 313: 10, 314: 9, 315: 10, 316: 11, 317: 10, 318: 10, 319: 9, 320: 9, 321: 9, 322: 9, 323: 9, 324: 9, 325: 9, 326: 9, 327: 9, 328: 9, 329: 9, 330: 9, 331: 11, 332: 9, 333: 11, 334: 11, 335: 11, 336: 11, 337: 9, 338: 9, 339: 8, 340: 9, 341: 9, 342: 9, 343: 9, 344: 9, 345: 9, 346: 9, 347: 10, 348: 10, 349: 10, 350: 9, 351: 9, 352: 9, 353: 9, 354: 11, 355: 9, 356: 9, 357: 9, 358: 9, 359: 9, 360: 9, 361: 9, 362: 9, 363: 9, 364: 9, 365: 9, 366: 9, 367: 10, 368: 9, 369: 9, 370: 9, 371: 9, 372: 9, 373: 9, 374: 10, 375: 10, 376: 10, 377: 10, 378: 10, 379: 9, 380: 9, 381: 9, 382: 9, 383: 9, 384: 9, 385: 9, 386: 9, 387: 9, 388: 9, 389: 9, 390: 9, 391: 9, 392: 9, 393: 9, 394: 9, 395: 9, 396: 9, 397: 9, 398: 9, 399: 9, 400: 9, 401: 9, 402: 9, 403: 9, 404: 9, 405: 10, 406: 10, 407: 9, 408: 9, 409: 9, 410: 9, 411: 9, 412: 9, 413: 9, 414: 9, 415: 9, 416: 9, 417: 9, 418: 9, 419: 9, 420: 9, 421: 9, 422: 9, 423: 9, 424: 9, 425: 9, 426: 10, 427: 8, 428: 8, 429: 8, 430: 8, 431: 8, 432: 8, 433: 8, 434: 9, 435: 8, 436: 10, 437: 8, 438: 8, 439: 9, 440: 9, 441: 9, 442: 9, 443: 8, 444: 8, 445: 8, 446: 6, 447: 6, 448: 6, 449: 6, 450: 6, 451: 8, 452: 6, 453: 6, 454: 5, 455: 5, 456: 5, 457: 5, 458: 5, 459: 5, 460: 5, 461: 5, 462: 5, 463: 5, 464: 5, 465: 5, 466: 5, 467: 5, 468: 5, 469: 5, 470: 5, 471: 5, 472: 5, 473: 5, 474: 5, 475: 5, 476: 7, 477: 5, 478: 5, 479: 5, 480: 5, 481: 5, 482: 5, 483: 5, 484: 5, 485: 5, 486: 5, 487: 5, 488: 5, 489: 5, 490: 5, 491: 5, 492: 5, 493: 5, 494: 5, 495: 5, 496: 5, 497: 5, 498: 5, 499: 5, 500: 5, 501: 5, 502: 5, 503: 5, 504: 5, 505: 5, 506: 5, 507: 5, 508: 5, 509: 5, 510: 5, 511: 5, 512: 5, 513: 5, 514: 5, 515: 5, 516: 5, 517: 5, 518: 5, 519: 5, 520: 5, 521: 5, 522: 5, 523: 5, 524: 5, 525: 5, 526: 5, 527: 5, 528: 5, 529: 5, 530: 5, 531: 5, 532: 5, 533: 5, 534: 5, 535: 5, 536: 4, 537: 4, 538: 4, 539: 4, 540: 4, 541: 4, 542: 4, 543: 4, 544: 4, 545: 4, 546: 4, 547: 4, 548: 3, 549: 3, 550: 3, 551: 3, 552: 3, 553: 3, 554: 3, 555: 3, 556: 3, 557: 3, 558: 3, 559: 3, 560: 3, 561: 3, 562: 3, 563: 3, 564: 3, 565: 3, 566: 3, 567: 3, 568: 3, 569: 3, 570: 3, 571: 3, 572: 3, 573: 3, 574: 3, 575: 3, 576: 3, 577: 3, 578: 3, 579: 3, 580: 3, 581: 3, 582: 3, 583: 3, 584: 3, 585: 3, 586: 3, 587: 3, 588: 3, 589: 3, 590: 3, 591: 3, 592: 3, 593: 3, 594: 3, 595: 3, 596: 3, 597: 3, 598: 3, 599: 3, 600: 3, 601: 3, 602: 3, 603: 3, 604: 3, 605: 3, 606: 3, 607: 3, 608: 3, 609: 3, 610: 3, 611: 3, 612: 3, 613: 3, 614: 3, 615: 3, 616: 3, 617: 3, 618: 3, 619: 3, 620: 3, 621: 3, 622: 3, 623: 3, 624: 3, 625: 3, 626: 3, 627: 3, 628: 3, 629: 3, 630: 3, 631: 3, 632: 3, 633: 3, 634: 3, 635: 3, 636: 3, 637: 3, 638: 2, 639: 2, 640: 2, 641: 2, 642: 2, 643: 2, 644: 2, 645: 2, 646: 2, 647: 2, 648: 2, 649: 2, 650: 2, 651: 2, 652: 2, 653: 2, 654: 2, 655: 2, 656: 2, 657: 2, 658: 2, 659: 2, 660: 2, 661: 2, 662: 2, 663: 2, 664: 2, 665: 2, 666: 2, 667: 2, 668: 2, 669: 2, 670: 2, 671: 2, 672: 2, 673: 2, 674: 2, 675: 2, 676: 2, 677: 2, 678: 2, 679: 2, 680: 2, 681: 2, 682: 2, 683: 3, 684: 4, 685: 4, 686: 3, 687: 2, 688: 2, 689: 2, 690: 2, 691: 2, 692: 2, 693: 2, 694: 2, 695: 3, 696: 4, 697: 4, 698: 4, 699: 3, 700: 2, 701: 2, 702: 2, 703: 2, 704: 2, 705: 2, 706: 2, 707: 2, 708: 2, 709: 2, 710: 2, 711: 2, 712: 2, 713: 2, 714: 2, 715: 2, 716: 2, 717: 2, 718: 2, 719: 2, 720: 2, 721: 2, 722: 2, 723: 2, 724: 4, 725: 4, 726: 2, 727: 2, 728: 2, 729: 2, 730: 2, 731: 2, 732: 2, 733: 2, 734: 2, 735: 2, 736: 2, 737: 2, 738: 2, 739: 2, 740: 2, 741: 2, 742: 2, 743: 2, 744: 2, 745: 2, 746: 2, 747: 2, 748: 2, 749: 2, 750: 2, 751: 2, 752: 2, 753: 2, 754: 2, 755: 2, 756: 2, 757: 2, 758: 2, 759: 2, 760: 2, 761: 2, 762: 2, 763: 2, 764: 2, 765: 2, 766: 2, 767: 2, 768: 2, 769: 2, 770: 2, 771: 2, 772: 2, 773: 2, 774: 2, 775: 2, 776: 2, 777: 2, 778: 2, 779: 2, 780: 2, 781: 2, 782: 2, 783: 2, 784: 2, 785: 2, 786: 2, 787: 2, 788: 2, 789: 2, 790: 2, 791: 2, 792: 2, 793: 2, 794: 2, 795: 2, 796: 2, 797: 2, 798: 2, 799: 2, 800: 2, 801: 2, 802: 2, 803: 2, 804: 2, 805: 2, 806: 2, 807: 2, 808: 2, 809: 2, 810: 2, 811: 2, 812: 2, 813: 2, 814: 2, 815: 2, 816: 2, 817: 2, 818: 2, 819: 2, 820: 2, 821: 2, 822: 2, 823: 2, 824: 2, 825: 2, 826: 2, 827: 2, 828: 2, 829: 2, 830: 2, 831: 2, 832: 2, 833: 2, 834: 2, 835: 2, 836: 2, 837: 2, 838: 2, 839: 2, 840: 2, 841: 2, 842: 2, 843: 2, 844: 2, 845: 2, 846: 2, 847: 2, 848: 2, 849: 2, 850: 2, 851: 2, 852: 2, 853: 2, 854: 3, 855: 2, 856: 2, 857: 4, 858: 2, 859: 2, 860: 2, 861: 2, 862: 2, 863: 2, 864: 2, 865: 2, 866: 2, 867: 2, 868: 2, 869: 2, 870: 2, 871: 2, 872: 2, 873: 2, 874: 2, 875: 2, 876: 2, 877: 2, 878: 2, 879: 2, 880: 4, 881: 4, 882: 2, 883: 2, 884: 2, 885: 2, 886: 2, 887: 2, 888: 2, 889: 2, 890: 2, 891: 2, 892: 2, 893: 2, 894: 2, 895: 2, 896: 2, 897: 2, 898: 2, 899: 2, 900: 2, 901: 2, 902: 2, 903: 2, 904: 2, 905: 2, 906: 2, 907: 2, 908: 2, 909: 2, 910: 2, 911: 2, 912: 2, 913: 2, 914: 2, 915: 2, 916: 2, 917: 2, 918: 2, 919: 2, 920: 2, 921: 2, 922: 2, 923: 2, 924: 2, 925: 2, 926: 2, 927: 2, 928: 2, 929: 2, 930: 2, 931: 2, 932: 2, 933: 2, 934: 2, 935: 2, 936: 2, 937: 2, 938: 2, 939: 2, 940: 2, 941: 2, 942: 2, 943: 2, 944: 2, 945: 2, 946: 2, 947: 2, 948: 2, 949: 2, 950: 2, 951: 2, 952: 2, 953: 2, 954: 2, 955: 2, 956: 2, 957: 2, 958: 2, 959: 2, 960: 2, 961: 2, 962: 2, 963: 2, 964: 2, 965: 2, 966: 2, 967: 2, 968: 2, 969: 2, 970: 2, 971: 2, 972: 2, 973: 2, 974: 2, 975: 2, 976: 2, 977: 2, 978: 2, 979: 2, 980: 2, 981: 2, 982: 2, 983: 2, 984: 2, 985: 2, 986: 2, 987: 2, 988: 2, 989: 2, 990: 2, 991: 2, 992: 2, 993: 2, 994: 2, 995: 2, 996: 2, 997: 2, 998: 2, 999: 2, 1000: 2, 1001: 2, 1002: 2, 1003: 2, 1004: 2, 1005: 2, 1006: 2, 1007: 2, 1008: 2, 1009: 2, 1010: 2, 1011: 2, 1012: 2, 1013: 2, 1014: 2, 1015: 2, 1016: 2, 1017: 2, 1018: 2, 1019: 2, 1020: 2, 1021: 2, 1022: 2, 1023: 2, 1024: 2, 1025: 2, 1026: 2, 1027: 2, 1028: 2, 1029: 2, 1030: 2, 1031: 2, 1032: 2, 1033: 2, 1034: 2, 1035: 2, 1036: 2, 1037: 2, 1038: 2, 1039: 2, 1040: 2, 1041: 2, 1042: 2, 1043: 2, 1044: 2, 1045: 2, 1046: 2, 1047: 2, 1048: 2, 1049: 2, 1050: 2, 1051: 3, 1052: 3, 1053: 2, 1054: 2, 1055: 2, 1056: 2, 1057: 2, 1058: 2, 1059: 2, 1060: 2, 1061: 2, 1062: 2, 1063: 2, 1064: 2, 1065: 2, 1066: 2, 1067: 2, 1068: 3, 1069: 3, 1070: 3, 1071: 3, 1072: 3, 1073: 3, 1074: 3, 1075: 3, 1076: 3, 1077: 3, 1078: 3, 1079: 3, 1080: 3, 1081: 2, 1082: 2, 1083: 2, 1084: 2, 1085: 2, 1086: 2, 1087: 2, 1088: 2, 1089: 2, 1090: 2, 1091: 2, 1092: 2, 1093: 2, 1094: 2, 1095: 2, 1096: 2, 1097: 2, 1098: 2, 1099: 2, 1100: 2, 1101: 2, 1102: 2, 1103: 2, 1104: 2, 1105: 2, 1106: 2, 1107: 2, 1108: 2, 1109: 2, 1110: 2, 1111: 2, 1112: 2, 1113: 2, 1114: 2, 1115: 2, 1116: 2, 1117: 2, 1118: 2, 1119: 2, 1120: 2, 1121: 2, 1122: 2, 1123: 2, 1124: 2, 1125: 2, 1126: 2, 1127: 2, 1128: 2, 1129: 2, 1130: 2, 1131: 2, 1132: 2, 1133: 2, 1134: 1, 1135: 1, 1136: 1, 1137: 1, 1138: 1, 1139: 1, 1140: 1, 1141: 1, 1142: 1, 1143: 1, 1144: 1, 1145: 1, 1146: 1, 1147: 1, 1148: 1, 1149: 1, 1150: 1, 1151: 1, 1152: 1, 1153: 1, 1154: 1, 1155: 1, 1156: 1, 1157: 1, 1158: 1, 1159: 1, 1160: 1, 1161: 1, 1162: 1, 1163: 1, 1164: 1, 1165: 1, 1166: 1, 1167: 1, 1168: 1, 1169: 1, 1170: 1, 1171: 1, 1172: 1, 1173: 1, 1174: 1, 1175: 1, 1176: 1, 1177: 1, 1178: 1, 1179: 1, 1180: 1, 1181: 1, 1182: 1, 1183: 1, 1184: 1, 1185: 1, 1186: 1, 1187: 1, 1188: 1, 1189: 1, 1190: 1, 1191: 1, 1192: 1, 1193: 1, 1194: 1, 1195: 1, 1196: 1, 1197: 1, 1198: 1, 1199: 1, 1200: 1, 1201: 1, 1202: 1, 1203: 1, 1204: 1, 1205: 1, 1206: 1, 1207: 1, 1208: 1, 1209: 1, 1210: 1, 1211: 1, 1212: 1, 1213: 1, 1214: 1, 1215: 1, 1216: 1, 1217: 1, 1218: 1, 1219: 1, 1220: 1, 1221: 1, 1222: 1, 1223: 1, 1224: 1, 1225: 1, 1226: 1, 1227: 1, 1228: 1, 1229: 1, 1230: 1, 1231: 1, 1232: 1, 1233: 1, 1234: 1, 1235: 1, 1236: 1, 1237: 1, 1238: 1, 1239: 1, 1240: 1, 1241: 1, 1242: 1, 1243: 1, 1244: 1, 1245: 1, 1246: 1, 1247: 1, 1248: 1, 1249: 1, 1250: 1, 1251: 1, 1252: 1, 1253: 1, 1254: 1, 1255: 1, 1256: 1, 1257: 1, 1258: 1, 1259: 1, 1260: 1, 1261: 1, 1262: 1, 1263: 1, 1264: 1, 1265: 1, 1266: 1, 1267: 1, 1268: 1, 1269: 1, 1270: 1, 1271: 1, 1272: 1, 1273: 1, 1274: 1, 1275: 1, 1276: 1, 1277: 1, 1278: 1, 1279: 1, 1280: 1, 1281: 1, 1282: 1, 1283: 1, 1284: 1, 1285: 1, 1286: 1, 1287: 1, 1288: 1, 1289: 1, 1290: 1, 1291: 1, 1292: 1, 1293: 1, 1294: 1, 1295: 1, 1296: 1, 1297: 1, 1298: 1, 1299: 1, 1300: 1, 1301: 1, 1302: 1, 1303: 1, 1304: 1, 1305: 1, 1306: 1, 1307: 1, 1308: 1, 1309: 1, 1310: 1, 1311: 1, 1312: 1, 1313: 1, 1314: 1, 1315: 1, 1316: 1, 1317: 1, 1318: 1, 1319: 1, 1320: 1, 1321: 1, 1322: 1, 1323: 1, 1324: 1, 1325: 1, 1326: 1, 1327: 1, 1328: 1, 1329: 1, 1330: 1, 1331: 1, 1332: 1, 1333: 1, 1334: 1, 1335: 1, 1336: 1, 1337: 1, 1338: 1, 1339: 1, 1340: 1, 1341: 1, 1342: 1, 1343: 1, 1344: 1, 1345: 1, 1346: 1, 1347: 1, 1348: 1, 1349: 1, 1350: 1, 1351: 1, 1352: 1, 1353: 1, 1354: 1, 1355: 1, 1356: 1, 1357: 1, 1358: 1, 1359: 1, 1360: 1, 1361: 1, 1362: 1, 1363: 1, 1364: 1, 1365: 1, 1366: 1, 1367: 1, 1368: 1, 1369: 1, 1370: 1, 1371: 1, 1372: 1, 1373: 1, 1374: 1, 1375: 1, 1376: 1, 1377: 1, 1378: 1, 1379: 1, 1380: 1, 1381: 1, 1382: 1, 1383: 1, 1384: 1, 1385: 1, 1386: 1, 1387: 1, 1388: 1, 1389: 1, 1390: 1, 1391: 1, 1392: 1, 1393: 1, 1394: 1, 1395: 1, 1396: 1, 1397: 1, 1398: 1, 1399: 1, 1400: 1, 1401: 1, 1402: 1, 1403: 1, 1404: 1, 1405: 1, 1406: 1, 1407: 1, 1408: 1, 1409: 1, 1410: 1, 1411: 1, 1412: 1, 1413: 1, 1414: 1, 1415: 1, 1416: 1, 1417: 1, 1418: 1, 1419: 1, 1420: 1, 1421: 1, 1422: 1, 1423: 1, 1424: 1, 1425: 1, 1426: 1, 1427: 1, 1428: 1, 1429: 1, 1430: 1, 1431: 1, 1432: 1, 1433: 1, 1434: 1, 1435: 1, 1436: 1, 1437: 1, 1438: 1, 1439: 1, 1440: 1, 1441: 1, 1442: 1, 1443: 1, 1444: 1, 1445: 1, 1446: 1, 1447: 1, 1448: 1, 1449: 1, 1450: 1, 1451: 1, 1452: 1, 1453: 1, 1454: 1, 1455: 1, 1456: 1, 1457: 1, 1458: 1, 1459: 1, 1460: 1, 1461: 1, 1462: 1, 1463: 1, 1464: 1, 1465: 1, 1466: 1, 1467: 1, 1468: 1, 1469: 1, 1470: 1}, 91: {1: 8, 2: 3, 3: 5, 4: 3, 5: 7, 6: 6, 7: 8, 8: 10, 9: 7, 10: 10, 11: 3, 12: 7, 13: 4, 14: 4, 15: 10, 16: 1, 17: 6, 18: 6, 19: 6, 20: 4, 21: 9, 22: 8, 23: 2, 24: 1, 25: 2, 26: 2, 27: 7, 28: 1, 29: 2, 30: 1, 31: 1, 32: 1, 33: 4, 34: 3, 35: 5, 37: 2, 41: 1, 42: 2, 44: 2, 45: 2, 46: 1, 47: 1, 50: 1, 52: 1, 53: 2, 59: 1, 71: 2, 74: 2, 78: 2, 79: 2, 91: 1, 95: 1, 97: 1, 98: 1, 100: 1, 101: 2, 106: 1, 133: 1}, 92: {0: 1, 1: 72, 2: 4738, 3: 855, 4: 251, 5: 118, 6: 40, 7: 14, 8: 3, 9: 8, 10: 2, 11: 1, 12: 1, 19: 1}, 93: {6: 22, 11: 106, 12: 69, 13: 37, 14: 20, 15: 136, 16: 328, 17: 141, 18: 311, 19: 311, 20: 145, 21: 168, 22: 257, 23: 127, 24: 213, 25: 231, 26: 106, 27: 146, 28: 310, 29: 105, 30: 84, 31: 239, 32: 127, 33: 78, 34: 170, 35: 89, 36: 75, 37: 178, 38: 108, 39: 58, 40: 154, 41: 77, 42: 78, 43: 117, 44: 109, 45: 47, 46: 109, 47: 83, 48: 62, 49: 81, 50: 71, 51: 42, 52: 79, 53: 63, 54: 64, 55: 60, 56: 52, 57: 45, 58: 57, 59: 53, 60: 67, 61: 43, 62: 68, 63: 57, 64: 63, 65: 58, 66: 47, 67: 29, 68: 75, 69: 40, 70: 44, 71: 38, 72: 44, 73: 42, 74: 40, 75: 51, 76: 38, 77: 45, 78: 32, 79: 39, 80: 36, 81: 43, 82: 42, 83: 31, 84: 38, 85: 29, 86: 26, 87: 36, 88: 40, 89: 26, 90: 42, 91: 38, 92: 27, 93: 28, 94: 34, 95: 36, 96: 23, 97: 32, 98: 19, 99: 28, 100: 30, 101: 28, 102: 32, 103: 30, 104: 36, 105: 43, 106: 37, 107: 27, 108: 35, 109: 28, 110: 20, 111: 21, 112: 15, 113: 23, 114: 15, 115: 23, 116: 22, 117: 27, 118: 25, 119: 22, 120: 14, 121: 24, 122: 21, 123: 17, 124: 18, 125: 14, 126: 14, 127: 15, 128: 16, 129: 14, 130: 17, 131: 14, 132: 22, 133: 25, 134: 15, 135: 20, 136: 15, 137: 17, 138: 12, 139: 11, 140: 14, 141: 8, 142: 18, 143: 10, 144: 19, 145: 9, 146: 13, 147: 20, 148: 16, 149: 10, 150: 9, 151: 11, 152: 6, 153: 13, 154: 14, 155: 18, 156: 12, 157: 17, 158: 5, 159: 8, 160: 10, 161: 14, 162: 11, 163: 11, 164: 13, 165: 11, 166: 5, 167: 5, 168: 11, 169: 12, 170: 13, 171: 10, 172: 9, 173: 8, 174: 8, 175: 14, 176: 12, 177: 10, 178: 13, 179: 4, 180: 9, 181: 7, 182: 7, 183: 13, 184: 11, 185: 7, 186: 8, 187: 8, 188: 9, 189: 10, 190: 4, 191: 8, 192: 10, 193: 5, 194: 4, 195: 10, 196: 6, 197: 7, 198: 8, 199: 7, 200: 8, 201: 10, 202: 6, 203: 5, 204: 8, 205: 2, 206: 1, 207: 10, 208: 7, 209: 5, 210: 12, 211: 9, 212: 4, 213: 9, 214: 7, 215: 6, 216: 6, 217: 9, 218: 10, 219: 4, 220: 5, 221: 4, 222: 6, 223: 7, 224: 9, 225: 8, 226: 4, 227: 8, 228: 1, 229: 3, 230: 9, 231: 4, 232: 8, 233: 3, 234: 2, 235: 3, 236: 6, 237: 5, 238: 2, 239: 2, 240: 7, 241: 1, 242: 7, 243: 6, 244: 6, 245: 8, 246: 6, 247: 3, 248: 6, 249: 5, 250: 4, 251: 2, 252: 4, 253: 5, 254: 2, 255: 5, 256: 1, 257: 1, 258: 5, 259: 3, 260: 1, 261: 4, 262: 4, 263: 2, 264: 8, 265: 3, 266: 2, 267: 2, 268: 3, 269: 4, 270: 3, 271: 8, 272: 4, 273: 3, 274: 4, 275: 2, 276: 5, 279: 1, 280: 3, 281: 4, 282: 5, 283: 4, 284: 4, 285: 3, 286: 3, 287: 1, 288: 3, 289: 3, 290: 2, 291: 2, 292: 5, 293: 2, 294: 3, 295: 4, 296: 4, 297: 5, 299: 3, 300: 2, 301: 2, 302: 2, 303: 2, 304: 1, 305: 4, 306: 1, 307: 3, 309: 2, 310: 3, 311: 6, 313: 4, 314: 2, 316: 1, 317: 3, 320: 1, 321: 4, 322: 1, 323: 1, 324: 3, 325: 2, 326: 1, 327: 6, 328: 2, 330: 2, 331: 3, 332: 1, 333: 3, 337: 3, 338: 1, 341: 1, 342: 7, 343: 1, 344: 2, 345: 1, 346: 1, 347: 2, 348: 1, 349: 2, 350: 1, 351: 1, 352: 2, 353: 2, 354: 1, 355: 1, 356: 2, 357: 2, 358: 1, 359: 1, 360: 1, 362: 2, 364: 3, 365: 1, 366: 1, 368: 3, 369: 2, 370: 1, 372: 2, 373: 1, 374: 2, 377: 1, 378: 4, 379: 1, 380: 2, 382: 1, 384: 2, 385: 1, 386: 1, 387: 1, 388: 2, 389: 1, 390: 3, 391: 1, 392: 1, 393: 1, 395: 2, 396: 1, 397: 2, 398: 3, 399: 1, 400: 2, 407: 2, 408: 1, 412: 2, 418: 2, 419: 2, 422: 1, 424: 1, 425: 1, 427: 1, 428: 1, 433: 1, 436: 1, 439: 1, 440: 1, 442: 1, 444: 1, 447: 1, 449: 1, 451: 1, 453: 1, 454: 1, 455: 1, 456: 2, 458: 1, 462: 2, 465: 1, 471: 2, 472: 1, 475: 1, 477: 1, 478: 1, 481: 1, 482: 1, 483: 3, 484: 1, 488: 1, 491: 1, 492: 1, 494: 2, 501: 2, 503: 2, 504: 1, 505: 2, 509: 2, 510: 1, 511: 2, 514: 1, 517: 2, 519: 1, 521: 1, 522: 1, 525: 2, 527: 1, 528: 1, 536: 1, 539: 2, 540: 1, 541: 1, 543: 1, 545: 2, 546: 1, 554: 3, 557: 1, 563: 2, 569: 1, 574: 1, 575: 2, 576: 1, 577: 2, 579: 1, 592: 2, 603: 1, 608: 1, 611: 3, 614: 1, 616: 1, 622: 1, 628: 1, 639: 1, 642: 2, 646: 1, 662: 1, 664: 2, 667: 1, 672: 1, 674: 1, 680: 1, 687: 1, 694: 1, 695: 1, 705: 1, 707: 1, 717: 1, 719: 1, 757: 1, 761: 1, 767: 2, 791: 1, 812: 1, 852: 1, 874: 1, 954: 1, 1003: 1, 1016: 1, 1043: 1, 1063: 1, 1122: 1, 1307: 1, 1361: 1, 1912: 2, 3548: 1}, 94: {1: 46, 2: 1, 4: 1, 256: 3, 512: 1}, 95: {0: 2625, 1: 1923, 2: 2072, 3: 1754, 4: 1467, 5: 1155, 6: 987, 7: 816, 8: 633, 9: 540, 10: 470, 11: 377, 12: 306, 13: 249, 14: 209, 15: 188, 16: 155, 17: 135, 18: 101, 19: 108, 20: 87, 21: 85, 22: 68, 23: 64, 24: 67, 25: 55, 26: 42, 27: 46, 28: 25, 29: 33, 30: 22, 31: 46, 32: 45, 33: 23, 34: 16, 35: 15, 36: 12, 37: 14, 38: 29, 39: 23, 40: 14, 41: 17, 42: 10, 43: 185, 44: 5, 45: 8, 46: 4, 47: 6, 48: 8, 49: 3, 50: 3, 51: 14, 52: 2, 53: 2, 54: 1, 55: 5, 57: 2, 58: 2, 59: 3, 60: 1, 62: 1, 65: 1, 70: 1, 71: 4, 72: 4, 73: 1, 82: 2, 83: 2, 99: 1, 100: 1, 101: 1, 103: 1, 104: 3, 106: 1, 108: 1, 110: 1, 111: 1, 115: 1, 118: 3, 120: 1, 132: 1, 136: 1, 138: 1, 139: 1}, 96: {0: 59, 1: 43, 2: 50, 3: 28, 4: 29, 5: 12, 6: 10, 7: 8, 8: 15, 9: 11, 10: 2, 12: 2, 13: 2, 14: 2, 15: 4, 16: 1, 19: 1, 57: 2}, 97: {0: 133, 1: 36, 2: 31, 3: 27, 4: 25, 5: 17, 6: 13, 7: 15, 8: 17, 9: 16, 10: 6, 11: 10, 12: 8, 13: 5, 14: 5, 15: 4, 16: 4, 17: 7, 18: 7, 19: 8, 20: 8, 21: 4, 22: 8, 23: 5, 24: 10, 25: 4, 26: 3, 27: 4, 28: 4, 29: 7, 30: 1, 31: 5, 33: 2, 34: 2, 35: 4, 36: 5, 37: 3, 38: 4, 39: 2, 44: 2, 45: 1, 46: 1, 48: 1, 49: 5, 50: 3, 52: 2, 55: 1, 58: 1, 59: 1, 60: 2, 62: 3, 63: 1, 67: 2, 73: 1, 76: 1, 78: 1, 80: 2, 82: 1, 100: 2, 104: 1, 106: 1, 108: 1, 110: 1, 111: 1, 144: 1, 226: 1, 229: 1}, 98: {0: 2, 1: 1, 4: 1}, 100: {0: 67111, 1: 70575, 2: 55317, 3: 42258, 4: 33055, 5: 27369, 6: 22283, 7: 18316, 8: 15260, 9: 12734, 10: 11280, 11: 9488, 12: 8380, 13: 7500, 14: 6589, 15: 5975, 16: 5367, 17: 4926, 18: 4308, 19: 3932, 20: 4036, 21: 3501, 22: 3222, 23: 2937, 24: 2815, 25: 2588, 26: 2402, 27: 2328, 28: 2171, 29: 2057, 30: 1988, 31: 1795, 32: 1985, 33: 1703, 34: 1608, 35: 1633, 36: 1479, 37: 1465, 38: 1352, 39: 1266, 40: 1223, 41: 1322, 42: 1166, 43: 1055, 44: 1052, 45: 975, 46: 947, 47: 993, 48: 886, 49: 844, 50: 875, 51: 813, 52: 800, 53: 772, 54: 724, 55: 752, 56: 708, 57: 720, 58: 663, 59: 654, 60: 755, 61: 659, 62: 845, 63: 560, 64: 674, 65: 737, 66: 705, 67: 489, 68: 657, 69: 665, 70: 617, 71: 638, 72: 435, 73: 452, 74: 447, 75: 543, 76: 430, 77: 490, 78: 427, 79: 408, 80: 417, 81: 434, 82: 447, 83: 425, 84: 685, 85: 402, 86: 366, 87: 381, 88: 393, 89: 402, 90: 386, 91: 364, 92: 360, 93: 329, 94: 337, 95: 684, 96: 376, 97: 316, 98: 326, 99: 295, 100: 395, 101: 333, 102: 308, 103: 311, 104: 304, 105: 294, 106: 290, 107: 274, 108: 282, 109: 299, 110: 261, 111: 312, 112: 261, 113: 243, 114: 281, 115: 227, 116: 247, 117: 256, 118: 342, 119: 254, 120: 239, 121: 231, 122: 272, 123: 227, 124: 325, 125: 220, 126: 207, 127: 202, 128: 192, 129: 225, 130: 183, 131: 188, 132: 253, 133: 205, 134: 178, 135: 184, 136: 169, 137: 182, 138: 174, 139: 185, 140: 162, 141: 161, 142: 202, 143: 161, 144: 168, 145: 171, 146: 187, 147: 175, 148: 160, 149: 170, 150: 223, 151: 157, 152: 170, 153: 211, 154: 151, 155: 263, 156: 195, 157: 210, 158: 224, 159: 139, 160: 143, 161: 143, 162: 165, 163: 134, 164: 133, 165: 147, 166: 138, 167: 150, 168: 128, 169: 144, 170: 139, 171: 121, 172: 141, 173: 148, 174: 118, 175: 132, 176: 112, 177: 112, 178: 114, 179: 112, 180: 102, 181: 108, 182: 110, 183: 130, 184: 109, 185: 102, 186: 104, 187: 105, 188: 118, 189: 102, 190: 103, 191: 99, 192: 96, 193: 104, 194: 119, 195: 110, 196: 90, 197: 90, 198: 96, 199: 90, 200: 93, 201: 110, 202: 100, 203: 118, 204: 89, 205: 93, 206: 95, 207: 93, 208: 97, 209: 89, 210: 91, 211: 100, 212: 86, 213: 87, 214: 89, 215: 89, 216: 87, 217: 91, 218: 85, 219: 106, 220: 91, 221: 97, 222: 88, 223: 99, 224: 85, 225: 91, 226: 104, 227: 87, 228: 94, 229: 91, 230: 86, 231: 81, 232: 91, 233: 85, 234: 93, 235: 82, 236: 102, 237: 84, 238: 78, 239: 106, 240: 113, 241: 85, 242: 77, 243: 79, 244: 92, 245: 85, 246: 93, 247: 74, 248: 76, 249: 85, 250: 79, 251: 80, 252: 92, 253: 75, 254: 84, 255: 87, 256: 83, 257: 83, 258: 83, 259: 72, 260: 76, 261: 120, 262: 77, 263: 76, 264: 79, 265: 76, 266: 64, 267: 76, 268: 79, 269: 72, 270: 67, 271: 67, 272: 64, 273: 67, 274: 63, 275: 75, 276: 71, 277: 64, 278: 60, 279: 76, 280: 64, 281: 59, 282: 68, 283: 60, 284: 61, 285: 60, 286: 62, 287: 61, 288: 90, 289: 60, 290: 62, 291: 60, 292: 72, 293: 63, 294: 62, 295: 64, 296: 77, 297: 64, 298: 64, 299: 69, 300: 66, 301: 72, 302: 56, 303: 61, 304: 53, 305: 61, 306: 60, 307: 51, 308: 55, 309: 64, 310: 52, 311: 62, 312: 49, 313: 49, 314: 50, 315: 57, 316: 50, 317: 55, 318: 53, 319: 304, 320: 1929, 321: 202, 322: 48, 323: 51, 324: 47, 325: 49, 326: 50, 327: 71, 328: 55, 329: 54, 330: 85, 331: 70, 332: 49, 333: 47, 334: 47, 335: 47, 336: 52, 337: 49, 338: 54, 339: 53, 340: 70, 341: 98, 342: 52, 343: 48, 344: 51, 345: 104, 346: 47, 347: 47, 348: 76, 349: 39, 350: 39, 351: 34, 352: 36, 353: 34, 354: 38, 355: 34, 356: 39, 357: 33, 358: 31, 359: 30, 360: 31, 361: 35, 362: 32, 363: 32, 364: 30, 365: 30, 366: 31, 367: 43, 368: 34, 369: 32, 370: 39, 371: 30, 372: 36, 373: 31, 374: 32, 375: 27, 376: 28, 377: 25, 378: 33, 379: 25, 380: 25, 381: 25, 382: 36, 383: 38, 384: 24, 385: 24, 386: 37, 387: 26, 388: 40, 389: 22, 390: 36, 391: 31, 392: 22, 393: 25, 394: 34, 395: 19, 396: 31, 397: 16, 398: 16, 399: 18, 400: 16, 401: 27, 402: 17, 403: 15, 404: 21, 405: 18, 406: 17, 407: 25, 408: 24, 409: 26, 410: 16, 411: 26, 412: 13, 413: 27, 414: 21, 415: 25, 416: 15, 417: 28, 418: 17, 419: 15, 420: 16, 421: 18, 422: 15, 423: 14, 424: 11, 425: 11, 426: 14, 427: 12, 428: 11, 429: 11, 430: 10, 431: 10, 432: 11, 433: 11, 434: 10, 435: 10, 436: 10, 437: 12, 438: 12, 439: 10, 440: 10, 441: 11, 442: 10, 443: 9, 444: 10, 445: 10, 446: 10, 447: 10, 448: 9, 449: 12, 450: 11, 451: 11, 452: 15, 453: 10, 454: 11, 455: 10, 456: 11, 457: 10, 458: 16, 459: 10, 460: 10, 461: 11, 462: 11, 463: 9, 464: 11, 465: 9, 466: 13, 467: 11, 468: 9, 469: 13, 470: 9, 471: 12, 472: 9, 473: 13, 474: 9, 475: 12, 476: 10, 477: 10, 478: 9, 479: 10, 480: 15, 481: 10, 482: 10, 483: 22, 484: 10, 485: 11, 486: 11, 487: 9, 488: 11, 489: 9, 490: 10, 491: 13, 492: 11, 493: 9, 494: 12, 495: 9, 496: 9, 497: 12, 498: 9, 499: 10, 500: 9, 501: 10, 502: 9, 503: 9, 504: 11, 505: 15, 506: 10, 507: 16, 508: 12, 509: 10, 510: 14, 511: 11, 512: 9, 513: 9, 514: 9, 515: 10, 516: 9, 517: 11, 518: 10, 519: 11, 520: 9, 521: 9, 522: 9, 523: 10, 524: 12, 525: 9, 526: 13, 527: 13, 528: 10, 529: 9, 530: 9, 531: 18, 532: 12, 533: 10, 534: 14, 535: 13, 536: 20, 537: 9, 538: 15, 539: 11, 540: 11, 541: 10, 542: 10, 543: 14, 544: 10, 545: 11, 546: 10, 547: 9, 548: 11, 549: 13, 550: 10, 551: 10, 552: 10, 553: 14, 554: 9, 555: 10, 556: 14, 557: 9, 558: 9, 559: 9, 560: 9, 561: 16, 562: 10, 563: 15, 564: 10, 565: 10, 566: 10, 567: 13, 568: 9, 569: 11, 570: 8, 571: 8, 572: 8, 573: 8, 574: 10, 575: 9, 576: 9, 577: 11, 578: 12, 579: 8, 580: 12, 581: 8, 582: 10, 583: 9, 584: 8, 585: 10, 586: 13, 587: 9, 588: 9, 589: 8, 590: 8, 591: 12, 592: 15, 593: 10, 594: 22, 595: 8, 596: 8, 597: 8, 598: 11, 599: 8, 600: 8, 601: 8, 602: 8, 603: 8, 604: 8, 605: 8, 606: 16, 607: 11, 608: 13, 609: 8, 610: 8, 611: 8, 612: 8, 613: 8, 614: 8, 615: 11, 616: 7, 617: 7, 618: 7, 619: 7, 620: 14, 621: 7, 622: 7, 623: 15, 624: 9, 625: 7, 626: 8, 627: 9, 628: 9, 629: 8, 630: 8, 631: 10, 632: 8, 633: 8, 634: 9, 635: 8, 636: 8, 637: 8, 638: 10, 639: 7, 640: 8, 641: 12, 642: 14, 643: 8, 644: 8, 645: 7, 646: 9, 647: 8, 648: 10, 649: 8, 650: 7, 651: 7, 652: 9, 653: 8, 654: 11, 655: 7, 656: 11, 657: 8, 658: 7, 659: 9, 660: 7, 661: 8, 662: 7, 663: 8, 664: 8, 665: 10, 666: 7, 667: 8, 668: 7, 669: 8, 670: 9, 671: 11, 672: 7, 673: 7, 674: 8, 675: 7, 676: 7, 677: 10, 678: 7, 679: 8, 680: 13, 681: 8, 682: 17, 683: 9, 684: 7, 685: 7, 686: 7, 687: 7, 688: 8, 689: 7, 690: 9, 691: 8, 692: 7, 693: 11, 694: 8, 695: 8, 696: 10, 697: 7, 698: 7, 699: 8, 700: 9, 701: 7, 702: 8, 703: 8, 704: 7, 705: 7, 706: 7, 707: 7, 708: 7, 709: 8, 710: 7, 711: 7, 712: 11, 713: 6, 714: 6, 715: 8, 716: 11, 717: 7, 718: 6, 719: 12, 720: 7, 721: 12, 722: 11, 723: 7, 724: 6, 725: 17, 726: 10, 727: 11, 728: 6, 729: 13, 730: 6, 731: 11, 732: 6, 733: 15, 734: 6, 735: 7, 736: 8, 737: 8, 738: 6, 739: 12, 740: 6, 741: 7, 742: 8, 743: 11, 744: 6, 745: 13, 746: 6, 747: 13, 748: 6, 749: 9, 750: 6, 751: 7, 752: 6, 753: 7, 754: 8, 755: 7, 756: 6, 757: 6, 758: 6, 759: 8, 760: 7, 761: 8, 762: 6, 763: 6, 764: 5, 765: 5, 766: 7, 767: 5, 768: 5, 769: 9, 770: 5, 771: 5, 772: 5, 773: 9, 774: 5, 775: 7, 776: 5, 777: 8, 778: 5, 779: 7, 780: 5, 781: 5, 782: 5, 783: 5, 784: 5, 785: 5, 786: 5, 787: 13, 788: 5, 789: 7, 790: 5, 791: 9, 792: 5, 793: 7, 794: 5, 795: 5, 796: 5, 797: 5, 798: 7, 799: 7, 800: 5, 801: 5, 802: 8, 803: 6, 804: 5, 805: 5, 806: 7, 807: 5, 808: 9, 809: 5, 810: 5, 811: 5, 812: 5, 813: 7, 814: 5, 815: 5, 816: 7, 817: 13, 818: 7, 819: 6, 820: 14, 821: 7, 822: 7, 823: 6, 824: 6, 825: 7, 826: 5, 827: 8, 828: 9, 829: 9, 830: 7, 831: 6, 832: 5, 833: 5, 834: 5, 835: 11, 836: 5, 837: 7, 838: 11, 839: 5, 840: 5, 841: 7, 842: 5, 843: 5, 844: 5, 845: 5, 846: 5, 847: 5, 848: 5, 849: 5, 850: 5, 851: 5, 852: 5, 853: 7, 854: 11, 855: 5, 856: 5, 857: 5, 858: 7, 859: 7, 860: 5, 861: 7, 862: 5, 863: 5, 864: 15, 865: 5, 866: 7, 867: 7, 868: 5, 869: 5, 870: 5, 871: 7, 872: 6, 873: 7, 874: 7, 875: 6, 876: 5, 877: 5, 878: 5, 879: 7, 880: 7, 881: 5, 882: 7, 883: 5, 884: 5, 885: 5, 886: 7, 887: 5, 888: 7, 889: 8, 890: 6, 891: 8, 892: 7, 893: 7, 894: 9, 895: 5, 896: 7, 897: 7, 898: 5, 899: 5, 900: 5, 901: 7, 902: 5, 903: 5, 904: 5, 905: 7, 906: 11, 907: 5, 908: 5, 909: 5, 910: 5, 911: 7, 912: 4, 913: 6, 914: 4, 915: 4, 916: 6, 917: 6, 918: 5, 919: 10, 920: 4, 921: 6, 922: 6, 923: 6, 924: 5, 925: 4, 926: 5, 927: 4, 928: 4, 929: 4, 930: 4, 931: 4, 932: 4, 933: 6, 934: 8, 935: 5, 936: 5, 937: 7, 938: 5, 939: 5, 940: 9, 941: 4, 942: 4, 943: 4, 944: 4, 945: 6, 946: 4, 947: 4, 948: 8, 949: 4, 950: 4, 951: 6, 952: 4, 953: 4, 954: 4, 955: 4, 956: 7, 957: 6, 958: 4, 959: 7, 960: 8, 961: 4, 962: 6, 963: 4, 964: 4, 965: 4, 966: 6, 967: 4, 968: 4, 969: 5, 970: 4, 971: 8, 972: 4, 973: 4, 974: 4, 975: 6, 976: 4, 977: 4, 978: 10, 979: 4, 980: 5, 981: 4, 982: 5, 983: 6, 984: 5, 985: 6, 986: 9, 987: 6, 988: 5, 989: 6, 990: 5, 991: 6, 992: 5, 993: 6, 994: 8, 995: 8, 996: 5, 997: 6, 998: 7, 999: 4, 1000: 5, 1001: 4, 1002: 7, 1003: 12, 1004: 12, 1005: 4, 1006: 6, 1007: 4, 1008: 8, 1009: 4, 1010: 6, 1011: 6, 1012: 6, 1013: 5, 1014: 6, 1015: 6, 1016: 6, 1017: 6, 1018: 4, 1019: 7, 1020: 4, 1021: 8, 1022: 5, 1023: 4, 1024: 4, 1025: 4, 1026: 6, 1027: 5, 1028: 4, 1029: 4, 1030: 4, 1031: 6, 1032: 5, 1033: 5, 1034: 6, 1035: 5, 1036: 6, 1037: 4, 1038: 4, 1039: 7, 1040: 4, 1041: 5, 1042: 6, 1043: 5, 1044: 4, 1045: 5, 1046: 6, 1047: 5, 1048: 4, 1049: 6, 1050: 4, 1051: 4, 1052: 4, 1053: 4, 1054: 4, 1055: 5, 1056: 4, 1057: 4, 1058: 7, 1059: 4, 1060: 4, 1061: 4, 1062: 4, 1063: 4, 1064: 6, 1065: 4, 1066: 4, 1067: 4, 1068: 8, 1069: 4, 1070: 4, 1071: 4, 1072: 6, 1073: 4, 1074: 4, 1075: 4, 1076: 4, 1077: 4, 1078: 4, 1079: 8, 1080: 4, 1081: 4, 1082: 4, 1083: 4, 1084: 4, 1085: 4, 1086: 4, 1087: 4, 1088: 4, 1089: 5, 1090: 8, 1091: 5, 1092: 4, 1093: 5, 1094: 6, 1095: 4, 1096: 5, 1097: 5, 1098: 4, 1099: 6, 1100: 4, 1101: 6, 1102: 4, 1103: 4, 1104: 4, 1105: 4, 1106: 6, 1107: 4, 1108: 4, 1109: 6, 1110: 4, 1111: 4, 1112: 6, 1113: 4, 1114: 7, 1115: 4, 1116: 4, 1117: 4, 1118: 5, 1119: 4, 1120: 7, 1121: 4, 1122: 5, 1123: 5, 1124: 4, 1125: 4, 1126: 4, 1127: 6, 1128: 4, 1129: 4, 1130: 6, 1131: 4, 1132: 4, 1133: 4, 1134: 4, 1135: 5, 1136: 4, 1137: 4, 1138: 4, 1139: 4, 1140: 4, 1141: 4, 1142: 4, 1143: 6, 1144: 4, 1145: 9, 1146: 4, 1147: 5, 1148: 4, 1149: 5, 1150: 4, 1151: 5, 1152: 4, 1153: 5, 1154: 4, 1155: 5, 1156: 4, 1157: 4, 1158: 4, 1159: 4, 1160: 4, 1161: 4, 1162: 4, 1163: 4, 1164: 4, 1165: 4, 1166: 5, 1167: 4, 1168: 4, 1169: 5, 1170: 4, 1171: 4, 1172: 4, 1173: 5, 1174: 5, 1175: 4, 1176: 4, 1177: 5, 1178: 4, 1179: 4, 1180: 4, 1181: 4, 1182: 4, 1183: 4, 1184: 4, 1185: 4, 1186: 4, 1187: 4, 1188: 4, 1189: 4, 1190: 4, 1191: 4, 1192: 4, 1193: 6, 1194: 4, 1195: 6, 1196: 4, 1197: 4, 1198: 5, 1199: 4, 1200: 4, 1201: 5, 1202: 4, 1203: 5, 1204: 6, 1205: 6, 1206: 4, 1207: 4, 1208: 4, 1209: 5, 1210: 4, 1211: 4, 1212: 8, 1213: 4, 1214: 4, 1215: 5, 1216: 4, 1217: 4, 1218: 4, 1219: 6, 1220: 4, 1221: 4, 1222: 5, 1223: 4, 1224: 4, 1225: 4, 1226: 4, 1227: 4, 1228: 4, 1229: 5, 1230: 4, 1231: 4, 1232: 4, 1233: 4, 1234: 4, 1235: 4, 1236: 4, 1237: 4, 1238: 4, 1239: 4, 1240: 4, 1241: 4, 1242: 4, 1243: 6, 1244: 4, 1245: 5, 1246: 4, 1247: 4, 1248: 5, 1249: 4, 1250: 5, 1251: 4, 1252: 5, 1253: 4, 1254: 4, 1255: 4, 1256: 4, 1257: 4, 1258: 6, 1259: 4, 1260: 7, 1261: 4, 1262: 4, 1263: 6, 1264: 4, 1265: 6, 1266: 4, 1267: 6, 1268: 4, 1269: 7, 1270: 4, 1271: 5, 1272: 4, 1273: 6, 1274: 4, 1275: 6, 1276: 4, 1277: 8, 1278: 4, 1279: 4, 1280: 4, 1281: 4, 1282: 4, 1283: 4, 1284: 5, 1285: 4, 1286: 4, 1287: 4, 1288: 4, 1289: 4, 1290: 4, 1291: 4, 1292: 4, 1293: 4, 1294: 4, 1295: 6, 1296: 4, 1297: 4, 1298: 4, 1299: 4, 1300: 4, 1301: 6, 1302: 4, 1303: 4, 1304: 4, 1305: 5, 1306: 4, 1307: 5, 1308: 4, 1309: 5, 1310: 4, 1311: 5, 1312: 4, 1313: 4, 1314: 4, 1315: 4, 1316: 4, 1317: 6, 1318: 4, 1319: 6, 1320: 4, 1321: 4, 1322: 4, 1323: 4, 1324: 4, 1325: 4, 1326: 4, 1327: 4, 1328: 4, 1329: 4, 1330: 4, 1331: 5, 1332: 4, 1333: 6, 1334: 4, 1335: 5, 1336: 4, 1337: 4, 1338: 4, 1339: 4, 1340: 4, 1341: 4, 1342: 4, 1343: 4, 1344: 5, 1345: 5, 1346: 4, 1347: 4, 1348: 4, 1349: 4, 1350: 4, 1351: 4, 1352: 4, 1353: 4, 1354: 4, 1355: 5, 1356: 5, 1357: 5, 1358: 4, 1359: 4, 1360: 4, 1361: 5, 1362: 4, 1363: 4, 1364: 4, 1365: 4, 1366: 4, 1367: 4, 1368: 4, 1369: 4, 1370: 4, 1371: 4, 1372: 4, 1373: 6, 1374: 4, 1375: 5, 1376: 4, 1377: 4, 1378: 4, 1379: 4, 1380: 4, 1381: 4, 1382: 5, 1383: 5, 1384: 4, 1385: 4, 1386: 4, 1387: 5, 1388: 4, 1389: 4, 1390: 4, 1391: 4, 1392: 4, 1393: 7, 1394: 4, 1395: 4, 1396: 4, 1397: 4, 1398: 4, 1399: 5, 1400: 4, 1401: 4, 1402: 4, 1403: 4, 1404: 4, 1405: 4, 1406: 4, 1407: 7, 1408: 4, 1409: 4, 1410: 4, 1411: 4, 1412: 6, 1413: 4, 1414: 4, 1415: 4, 1416: 4, 1417: 4, 1418: 4, 1419: 4, 1420: 4, 1421: 4, 1422: 4, 1423: 4, 1424: 4, 1425: 4, 1426: 4, 1427: 4, 1428: 4, 1429: 4, 1430: 4, 1431: 4, 1432: 4, 1433: 4, 1434: 4, 1435: 4, 1436: 2, 1437: 2, 1438: 2, 1439: 2, 1440: 2, 1441: 3, 1442: 2, 1443: 2, 1444: 2, 1445: 2, 1446: 4, 1447: 2, 1448: 2, 1449: 2, 1450: 4, 1451: 2, 1452: 2, 1453: 2, 1454: 2, 1455: 2, 1456: 2, 1457: 2, 1458: 2, 1459: 3, 1460: 2, 1461: 4, 1462: 2, 1463: 2, 1464: 2, 1465: 2, 1466: 2, 1467: 2, 1468: 2, 1469: 2, 1470: 3, 1471: 2, 1472: 3, 1473: 2, 1474: 1, 1475: 1, 1476: 1, 1477: 1, 1478: 1, 1479: 1, 1480: 1, 1481: 1, 1482: 1, 1483: 2, 1484: 2, 1485: 1, 1486: 1, 1487: 2, 1488: 2, 1489: 1, 1490: 1, 1491: 1, 1492: 1, 1493: 1, 1494: 1, 1495: 1, 1496: 2, 1497: 1, 1498: 1, 1499: 1, 1500: 1, 1501: 1, 1502: 1, 1503: 1, 1504: 1, 1505: 1, 1506: 1, 1507: 1, 1508: 1, 1509: 1, 1510: 1, 1511: 1, 1512: 1, 1513: 1, 1514: 1, 1515: 1, 1516: 1, 1517: 1, 1518: 1, 1519: 1, 1520: 1, 1521: 1, 1522: 1, 1523: 1, 1524: 1, 1525: 1, 1526: 1, 1527: 1, 1528: 1, 1529: 1, 1530: 1, 1531: 1, 1532: 1, 1533: 1, 1534: 1, 1535: 1, 1536: 1, 1537: 1, 1538: 2, 1539: 1, 1540: 1, 1541: 2, 1542: 1, 1543: 1, 1544: 1, 1545: 2, 1546: 1, 1547: 1, 1548: 3, 1549: 1, 1550: 1, 1551: 1, 1552: 2, 1553: 1, 1554: 1, 1555: 1, 1556: 1, 1557: 1, 1558: 1, 1559: 1, 1560: 1, 1561: 1, 1562: 1, 1563: 1, 1564: 2, 1565: 1, 1566: 1, 1567: 1, 1568: 1, 1569: 1, 1570: 1, 1571: 1, 1572: 1, 1573: 1, 1574: 1, 1575: 1, 1576: 1, 1577: 1, 1578: 1, 1579: 1, 1580: 1, 1581: 2, 1582: 1, 1583: 1, 1584: 1, 1585: 1, 1586: 1, 1587: 1, 1588: 1, 1589: 1, 1590: 1, 1591: 1, 1592: 1, 1593: 1, 1594: 1, 1595: 2, 1596: 1, 1597: 1, 1598: 1, 1599: 1, 1600: 1, 1601: 1, 1602: 1, 1603: 1, 1604: 1, 1605: 1, 1606: 1, 1607: 1, 1608: 1, 1609: 1, 1610: 1, 1611: 1, 1612: 1, 1613: 1, 1614: 1, 1615: 3, 1616: 1, 1617: 3, 1618: 1, 1619: 2, 1620: 1, 1621: 3, 1622: 1, 1623: 1, 1624: 1, 1625: 3, 1626: 1, 1627: 1, 1628: 1, 1629: 1, 1630: 1, 1631: 1, 1632: 1, 1633: 1, 1634: 1, 1635: 1, 1636: 1, 1637: 1, 1638: 1, 1639: 1, 1640: 1, 1641: 1, 1642: 1, 1643: 1, 1644: 2, 1645: 1, 1646: 3, 1647: 1, 1648: 1, 1649: 1, 1650: 1, 1651: 1, 1652: 1, 1653: 1, 1654: 1, 1655: 3, 1656: 1, 1657: 1, 1658: 1, 1659: 1, 1660: 1, 1661: 1, 1662: 1, 1663: 2, 1664: 1, 1665: 1, 1666: 2, 1667: 1, 1668: 2, 1669: 1, 1670: 1, 1671: 1, 1672: 1, 1673: 1, 1674: 1, 1675: 1, 1676: 1, 1677: 1, 1678: 1, 1679: 1, 1680: 1, 1681: 1, 1682: 1, 1683: 1, 1684: 1, 1685: 1, 1686: 3, 1687: 1, 1688: 1, 1689: 1, 1690: 1, 1691: 1, 1692: 1, 1693: 1, 1694: 1, 1695: 1, 1696: 1, 1697: 4, 1698: 1, 1699: 1, 1700: 1, 1701: 1, 1702: 1, 1703: 1, 1704: 1, 1705: 1, 1706: 2, 1707: 1, 1708: 1, 1709: 1, 1710: 1, 1711: 1, 1712: 1, 1713: 1, 1714: 1, 1715: 1, 1716: 1, 1717: 1, 1718: 1, 1719: 1, 1720: 2, 1721: 1, 1722: 1, 1723: 1, 1724: 2, 1725: 1, 1726: 1, 1727: 3, 1728: 1, 1729: 1, 1730: 1, 1731: 1, 1732: 1, 1733: 1, 1734: 3, 1735: 1, 1736: 1, 1737: 1, 1738: 2, 1739: 1, 1740: 1, 1741: 1, 1742: 4, 1743: 1, 1744: 1, 1745: 1, 1746: 1, 1747: 1, 1748: 2, 1749: 1, 1750: 1, 1751: 2, 1752: 1, 1753: 1, 1754: 1, 1755: 2, 1756: 1, 1757: 1, 1758: 1, 1759: 1, 1760: 2, 1761: 1, 1762: 2, 1763: 1, 1764: 1, 1765: 1, 1766: 1, 1767: 1, 1768: 2, 1769: 1, 1770: 2, 1771: 1, 1772: 2, 1773: 1, 1774: 1, 1775: 1, 1776: 1, 1777: 1, 1778: 1, 1779: 1, 1780: 1, 1781: 1, 1782: 1, 1783: 1, 1784: 1, 1785: 1, 1786: 1, 1787: 1, 1788: 1, 1789: 1, 1790: 1, 1791: 2, 1792: 2, 1793: 1, 1794: 1, 1795: 1, 1796: 2, 1797: 2, 1798: 2, 1799: 1, 1800: 1, 1801: 1, 1802: 1, 1803: 1, 1804: 1, 1805: 1, 1806: 1, 1807: 1, 1808: 1, 1809: 1, 1810: 1, 1811: 2, 1812: 1, 1813: 1, 1814: 1, 1815: 4, 1816: 1, 1817: 1, 1818: 2, 1819: 2, 1820: 1, 1821: 1, 1822: 1, 1823: 2, 1824: 1, 1825: 2, 1826: 1, 1827: 2, 1828: 1, 1829: 1, 1830: 1, 1831: 1, 1832: 1, 1833: 1, 1834: 1, 1835: 1, 1836: 1, 1837: 1, 1838: 1, 1839: 1, 1840: 1, 1841: 1, 1842: 1, 1843: 1, 1844: 2, 1845: 1, 1846: 1, 1847: 2, 1848: 1, 1849: 1, 1850: 1, 1851: 1, 1852: 1, 1853: 1, 1854: 1, 1855: 1, 1856: 1, 1857: 1, 1858: 1, 1859: 1, 1860: 1, 1861: 1, 1862: 1, 1863: 1, 1864: 2, 1865: 1, 1866: 2, 1867: 1, 1868: 2, 1869: 1, 1870: 1, 1871: 1, 1872: 1, 1873: 1, 1874: 1, 1875: 2, 1876: 1, 1877: 1, 1878: 2, 1879: 1, 1880: 1, 1881: 1, 1882: 1, 1883: 1, 1884: 1, 1885: 1, 1886: 1, 1887: 1, 1888: 1, 1889: 1, 1890: 1, 1891: 1, 1892: 1, 1893: 1, 1894: 1, 1895: 1, 1896: 2, 1897: 1, 1898: 1, 1899: 1, 1900: 1, 1901: 1, 1902: 1, 1903: 1, 1904: 1, 1905: 1, 1906: 1, 1907: 1, 1908: 1, 1909: 1, 1910: 1, 1911: 1, 1912: 1, 1913: 1, 1914: 1, 1915: 1, 1916: 1, 1917: 1, 1918: 1, 1919: 1, 1920: 2, 1921: 2, 1922: 1, 1923: 1, 1924: 1, 1925: 1, 1926: 1, 1927: 1, 1928: 1, 1929: 1, 1930: 1, 1931: 1, 1932: 1, 1933: 1, 1934: 1, 1935: 1, 1936: 1, 1937: 1, 1938: 1, 1939: 1, 1940: 1, 1941: 1, 1942: 1, 1943: 1, 1944: 1, 1945: 1, 1946: 1, 1947: 1, 1948: 1, 1949: 1, 1950: 1, 1951: 1, 1952: 1, 1953: 1, 1954: 1, 1955: 1, 1956: 1, 1957: 1, 1958: 1, 1959: 2, 1960: 1, 1961: 1, 1962: 1, 1963: 1, 1964: 1, 1965: 1, 1966: 1, 1967: 1, 1968: 1, 1969: 1, 1970: 1, 1971: 1, 1972: 1, 1973: 1, 1974: 1, 1975: 1, 1976: 1, 1977: 1, 1978: 1, 1979: 1, 1980: 1, 1981: 1, 1982: 1, 1983: 1, 1984: 1, 1985: 1, 1986: 1, 1987: 1, 1988: 1, 1989: 1, 1990: 1, 1991: 2, 1992: 1, 1993: 1, 1994: 1, 1995: 1, 1996: 1, 1997: 1, 1998: 1, 1999: 3, 2000: 1, 2001: 1, 2002: 1, 2003: 1, 2004: 1, 2005: 1, 2006: 1, 2007: 3, 2008: 1, 2009: 1, 2010: 1, 2011: 1, 2012: 2, 2013: 1, 2014: 2, 2015: 1, 2016: 1, 2017: 2, 2018: 1, 2019: 1, 2020: 5, 2021: 1, 2022: 2, 2023: 1, 2024: 1, 2025: 1, 2026: 1, 2027: 1, 2028: 1, 2029: 2, 2030: 1, 2031: 1, 2032: 1, 2033: 1, 2034: 1, 2035: 2, 2036: 1, 2037: 1, 2038: 1, 2039: 1, 2040: 1, 2041: 1, 2042: 1, 2043: 1, 2044: 1, 2045: 1, 2046: 1, 2047: 1, 2048: 1, 2049: 1, 2050: 1, 2051: 1, 2052: 1, 2053: 2, 2054: 1, 2055: 2, 2056: 1, 2057: 1, 2058: 1, 2059: 1, 2060: 1, 2061: 1, 2062: 1, 2063: 1, 2064: 1, 2065: 1, 2066: 1, 2067: 1, 2068: 1, 2069: 1, 2070: 1, 2071: 2, 2072: 1, 2073: 1, 2074: 1, 2075: 1, 2076: 1, 2077: 1, 2078: 1, 2079: 1, 2080: 2, 2081: 1, 2082: 1, 2083: 1, 2084: 1, 2085: 1, 2086: 1, 2087: 1, 2088: 1, 2089: 3, 2090: 1, 2091: 2, 2092: 1, 2093: 1, 2094: 1, 2095: 1, 2096: 1, 2097: 1, 2098: 3, 2099: 1, 2100: 1, 2101: 1, 2102: 2, 2103: 1, 2104: 1, 2105: 1, 2106: 1, 2107: 1, 2108: 1, 2109: 2, 2110: 1, 2111: 1, 2112: 3, 2113: 1, 2114: 2, 2115: 1, 2116: 1, 2117: 1, 2118: 1, 2119: 1, 2120: 3, 2121: 1, 2122: 1, 2123: 1, 2124: 1, 2125: 1, 2126: 2, 2127: 1, 2128: 1, 2129: 1, 2130: 2, 2131: 1, 2132: 3, 2133: 1, 2134: 1, 2135: 1, 2136: 3, 2137: 1, 2138: 1, 2139: 1, 2140: 1, 2141: 1, 2142: 3, 2143: 1, 2144: 1, 2145: 1, 2146: 1, 2147: 1, 2148: 1, 2149: 2, 2150: 1, 2151: 3, 2152: 1, 2153: 1, 2154: 1, 2155: 3, 2156: 1, 2157: 1, 2158: 1, 2159: 1, 2160: 1, 2161: 1, 2162: 1, 2163: 1, 2164: 1, 2165: 1, 2166: 1, 2167: 1, 2168: 1, 2169: 1, 2170: 1, 2171: 2, 2172: 1, 2173: 1, 2174: 1, 2175: 1, 2176: 1, 2177: 3, 2178: 1, 2179: 1, 2180: 1, 2181: 1, 2182: 3, 2183: 1, 2184: 1, 2185: 1, 2186: 1, 2187: 1, 2188: 1, 2189: 1, 2190: 1, 2191: 1, 2192: 1, 2193: 1, 2194: 1, 2195: 1, 2196: 1, 2197: 1, 2198: 1, 2199: 1, 2200: 1, 2201: 1, 2202: 1, 2203: 1, 2204: 1, 2205: 1, 2206: 1, 2207: 1, 2208: 1, 2209: 1, 2210: 3, 2211: 1, 2212: 1, 2213: 1, 2214: 1, 2215: 1, 2216: 1, 2217: 1, 2218: 1, 2219: 1, 2220: 1, 2221: 1, 2222: 1, 2223: 2, 2224: 1, 2225: 1, 2226: 2, 2227: 1, 2228: 1, 2229: 2, 2230: 1, 2231: 2, 2232: 1, 2233: 1, 2234: 1, 2235: 1, 2236: 1, 2237: 1, 2238: 1, 2239: 1, 2240: 1, 2241: 1, 2242: 3, 2243: 1, 2244: 1, 2245: 1, 2246: 3, 2247: 1, 2248: 1, 2249: 1, 2250: 2, 2251: 1, 2252: 3, 2253: 1, 2254: 1, 2255: 1, 2256: 1, 2257: 2, 2258: 1, 2259: 1, 2260: 1, 2261: 1, 2262: 1, 2263: 1, 2264: 1, 2265: 1, 2266: 1, 2267: 1, 2268: 1, 2269: 1, 2270: 1, 2271: 2, 2272: 1, 2273: 3, 2274: 1, 2275: 3, 2276: 1, 2277: 2, 2278: 1, 2279: 1, 2280: 1, 2281: 1, 2282: 1, 2283: 1, 2284: 1, 2285: 1, 2286: 1, 2287: 1, 2288: 1, 2289: 1, 2290: 1, 2291: 1, 2292: 1, 2293: 1, 2294: 1, 2295: 1, 2296: 1, 2297: 1, 2298: 1, 2299: 2, 2300: 1, 2301: 1, 2302: 1, 2303: 1, 2304: 1, 2305: 1, 2306: 1, 2307: 1, 2308: 2, 2309: 1, 2310: 2, 2311: 1, 2312: 1, 2313: 1, 2314: 1, 2315: 1, 2316: 4, 2317: 1, 2318: 1, 2319: 1, 2320: 1, 2321: 1, 2322: 1, 2323: 1, 2324: 1, 2325: 1, 2326: 1, 2327: 1, 2328: 1, 2329: 1, 2330: 1, 2331: 1, 2332: 1, 2333: 2, 2334: 1, 2335: 1, 2336: 1, 2337: 1, 2338: 1, 2339: 1, 2340: 1, 2341: 1, 2342: 1, 2343: 2, 2344: 1, 2345: 3, 2346: 1, 2347: 1, 2348: 1, 2349: 1, 2350: 2, 2351: 1, 2352: 1, 2353: 1, 2354: 1, 2355: 3, 2356: 1, 2357: 1, 2358: 1, 2359: 1, 2360: 2, 2361: 1, 2362: 1, 2363: 1, 2364: 1, 2365: 1, 2366: 1, 2367: 1, 2368: 1, 2369: 1, 2370: 1, 2371: 1, 2372: 1, 2373: 1, 2374: 1, 2375: 1, 2376: 1, 2377: 3, 2378: 1, 2379: 1, 2380: 1, 2381: 1, 2382: 1, 2383: 1, 2384: 2, 2385: 1, 2386: 1, 2387: 4, 2388: 1, 2389: 1, 2390: 1, 2391: 1, 2392: 1, 2393: 1, 2394: 1, 2395: 1, 2396: 1, 2397: 1, 2398: 1, 2399: 3, 2400: 1, 2401: 1, 2402: 1, 2403: 1, 2404: 1, 2405: 1, 2406: 1, 2407: 1, 2408: 1, 2409: 1, 2410: 1, 2411: 1, 2412: 1, 2413: 1, 2414: 1, 2415: 1, 2416: 1, 2417: 1, 2418: 1, 2419: 1, 2420: 2, 2421: 1, 2422: 1, 2423: 1, 2424: 1, 2425: 1, 2426: 1, 2427: 1, 2428: 1, 2429: 1, 2430: 3, 2431: 1, 2432: 1, 2433: 1, 2434: 1, 2435: 1, 2436: 1, 2437: 1, 2438: 1, 2439: 2, 2440: 1, 2441: 1, 2442: 2, 2443: 3, 2444: 1, 2445: 1, 2446: 2, 2447: 1, 2448: 2, 2449: 1, 2450: 1, 2451: 1, 2452: 1, 2453: 1, 2454: 1, 2455: 1, 2456: 1, 2457: 1, 2458: 1, 2459: 1, 2460: 2, 2461: 1, 2462: 1, 2463: 1, 2464: 1, 2465: 1, 2466: 1, 2467: 1, 2468: 1, 2469: 1, 2470: 1, 2471: 1, 2472: 2, 2473: 1, 2474: 1, 2475: 1, 2476: 1, 2477: 1, 2478: 3, 2479: 1, 2480: 1, 2481: 1, 2482: 1, 2483: 5, 2484: 1, 2485: 1, 2486: 1, 2487: 1, 2488: 2, 2489: 1, 2490: 2, 2491: 1, 2492: 2, 2493: 1, 2494: 1, 2495: 1, 2496: 1, 2497: 1, 2498: 3, 2499: 1, 2500: 1, 2501: 1, 2502: 1, 2503: 1, 2504: 1, 2505: 3, 2506: 1, 2507: 3, 2508: 1, 2509: 1, 2510: 1, 2511: 3, 2512: 1, 2513: 1, 2514: 1, 2515: 1, 2516: 2, 2517: 1, 2518: 1, 2519: 1, 2520: 3, 2521: 1, 2522: 1, 2523: 2, 2524: 1, 2525: 2, 2526: 1, 2527: 2, 2528: 1, 2529: 1, 2530: 1, 2531: 4, 2532: 1, 2533: 1, 2534: 1, 2535: 1, 2536: 1, 2537: 1, 2538: 1, 2539: 1, 2540: 1, 2541: 1, 2542: 2, 2543: 1, 2544: 2, 2545: 1, 2546: 1, 2547: 1, 2548: 2, 2549: 1, 2550: 3, 2551: 1, 2552: 1, 2553: 1, 2554: 1, 2555: 1, 2556: 1, 2557: 1, 2558: 3, 2559: 1, 2560: 1, 2561: 3, 2562: 1, 2563: 1, 2564: 1, 2565: 2, 2566: 1, 2567: 1, 2568: 1, 2569: 1, 2570: 3, 2571: 1, 2572: 3, 2573: 1, 2574: 2, 2575: 1, 2576: 4, 2577: 1, 2578: 1, 2579: 1, 2580: 1, 2581: 1, 2582: 1, 2583: 2, 2584: 1, 2585: 1, 2586: 1, 2587: 1, 2588: 1, 2589: 1, 2590: 1, 2591: 5, 2592: 1, 2593: 3, 2594: 1, 2595: 1, 2596: 1, 2597: 1, 2598: 1, 2599: 1, 2600: 1, 2601: 1, 2602: 2, 2603: 1, 2604: 1, 2605: 1, 2606: 1, 2607: 1, 2608: 1, 2609: 1, 2610: 1, 2611: 1, 2612: 1, 2613: 1, 2614: 1, 2615: 1, 2616: 1, 2617: 1, 2618: 1, 2619: 1, 2620: 1, 2621: 1, 2622: 1, 2623: 1, 2624: 2, 2625: 1, 2626: 1, 2627: 1, 2628: 1, 2629: 1, 2630: 1, 2631: 1, 2632: 1, 2633: 3, 2634: 1, 2635: 1, 2636: 1, 2637: 1, 2638: 1, 2639: 1, 2640: 3, 2641: 1, 2642: 3, 2643: 1, 2644: 2, 2645: 1, 2646: 1, 2647: 3, 2648: 1, 2649: 1, 2650: 1, 2651: 3, 2652: 1, 2653: 1, 2654: 1, 2655: 2, 2656: 1, 2657: 1, 2658: 1, 2659: 2, 2660: 1, 2661: 3, 2662: 1, 2663: 3, 2664: 1, 2665: 3, 2666: 1, 2667: 3, 2668: 1, 2669: 2, 2670: 1, 2671: 1, 2672: 1, 2673: 3, 2674: 1, 2675: 3, 2676: 1, 2677: 2, 2678: 1, 2679: 3, 2680: 1, 2681: 1, 2682: 1, 2683: 1, 2684: 1, 2685: 1, 2686: 1, 2687: 1, 2688: 1, 2689: 1, 2690: 1, 2691: 1, 2692: 1, 2693: 1, 2694: 1, 2695: 1, 2696: 1, 2697: 2, 2698: 1, 2699: 2, 2700: 1, 2701: 1, 2702: 1, 2703: 1, 2704: 1, 2705: 1, 2706: 1, 2707: 1, 2708: 1, 2709: 1, 2710: 1, 2711: 1, 2712: 1, 2713: 1, 2714: 1, 2715: 1, 2716: 1, 2717: 1, 2718: 1, 2719: 1, 2720: 1, 2721: 1, 2722: 1, 2723: 1, 2724: 1, 2725: 1, 2726: 1, 2727: 2, 2728: 1, 2729: 1, 2730: 1, 2731: 1, 2732: 1, 2733: 1, 2734: 1, 2735: 1, 2736: 1, 2737: 2, 2738: 1, 2739: 1, 2740: 1, 2741: 1, 2742: 1, 2743: 2, 2744: 2, 2745: 1, 2746: 1, 2747: 1, 2748: 1, 2749: 1, 2750: 1, 2751: 1, 2752: 1, 2753: 1, 2754: 1, 2755: 1, 2756: 1, 2757: 1, 2758: 1, 2759: 1, 2760: 1, 2761: 1, 2762: 1, 2763: 1, 2764: 1, 2765: 1, 2766: 1, 2767: 1, 2768: 1, 2769: 1, 2770: 1, 2771: 1, 2772: 1, 2773: 1, 2774: 1, 2775: 1, 2776: 1, 2777: 1, 2778: 1, 2779: 1, 2780: 1, 2781: 1, 2782: 1, 2783: 1, 2784: 1, 2785: 1, 2786: 1, 2787: 1, 2788: 1, 2789: 1, 2790: 1, 2791: 1, 2792: 1, 2793: 1, 2794: 1, 2795: 1, 2796: 1, 2797: 1, 2798: 1, 2799: 3, 2800: 1, 2801: 1, 2802: 1, 2803: 2, 2804: 2, 2805: 1, 2806: 1, 2807: 1, 2808: 1, 2809: 2, 2810: 2, 2811: 1, 2812: 1, 2813: 1, 2814: 1, 2815: 1, 2816: 1, 2817: 1, 2818: 1, 2819: 1, 2820: 1, 2821: 1, 2822: 1, 2823: 1, 2824: 1, 2825: 1, 2826: 1, 2827: 1, 2828: 1, 2829: 1, 2830: 1, 2831: 1, 2832: 1, 2833: 1, 2834: 1, 2835: 1, 2836: 1, 2837: 1, 2838: 1, 2839: 2, 2840: 2, 2841: 1, 2842: 1, 2843: 1, 2844: 1, 2845: 1, 2846: 1, 2847: 1, 2848: 2, 2849: 1, 2850: 1, 2851: 1, 2852: 1, 2853: 1, 2854: 1, 2855: 1, 2856: 2, 2857: 1, 2858: 1, 2859: 1, 2860: 1, 2861: 1, 2862: 1, 2863: 1, 2864: 1, 2865: 1, 2866: 1, 2867: 1, 2868: 1, 2869: 1, 2870: 1, 2871: 1, 2872: 1, 2873: 1, 2874: 1, 2875: 1, 2876: 1, 2877: 1, 2878: 1, 2879: 1, 2880: 1, 2881: 1, 2882: 1, 2883: 1, 2884: 1, 2885: 1, 2886: 1, 2887: 2, 2888: 1, 2889: 1, 2890: 1, 2891: 1, 2892: 3, 2893: 1, 2894: 2, 2895: 1, 2896: 2, 2897: 1, 2898: 1, 2899: 1, 2900: 1, 2901: 1, 2902: 1, 2903: 1, 2904: 1, 2905: 1, 2906: 1, 2907: 1, 2908: 1, 2909: 1, 2910: 2, 2911: 2, 2912: 1, 2913: 1, 2914: 1, 2915: 1, 2916: 2, 2917: 2, 2918: 1, 2919: 1, 2920: 1, 2921: 1, 2922: 1, 2923: 1, 2924: 1, 2925: 2, 2926: 2, 2927: 1, 2928: 1, 2929: 1, 2930: 1, 2931: 1, 2932: 1, 2933: 1, 2934: 1, 2935: 2, 2936: 1, 2937: 1, 2938: 1, 2939: 2, 2940: 1, 2941: 1, 2942: 1, 2943: 1, 2944: 1, 2945: 1, 2946: 1, 2947: 2, 2948: 1, 2949: 1, 2950: 1, 2951: 1, 2952: 1, 2953: 1, 2954: 1, 2955: 1, 2956: 1, 2957: 1, 2958: 1, 2959: 1, 2960: 1, 2961: 1, 2962: 1, 2963: 1, 2964: 1, 2965: 1, 2966: 1, 2967: 1, 2968: 1, 2969: 1, 2970: 1, 2971: 3, 2972: 1, 2973: 1, 2974: 1, 2975: 1, 2976: 1, 2977: 1, 2978: 1, 2979: 1, 2980: 1, 2981: 2, 2982: 1, 2983: 1, 2984: 1, 2985: 1, 2986: 1, 2987: 1, 2988: 1, 2989: 1, 2990: 1, 2991: 1, 2992: 1, 2993: 1, 2994: 1, 2995: 1, 2996: 1, 2997: 1, 2998: 1, 2999: 4, 3000: 1, 3001: 1, 3002: 1, 3003: 1, 3004: 1, 3005: 1, 3006: 1, 3007: 1, 3008: 1, 3009: 1, 3010: 2, 3011: 1, 3012: 1, 3013: 2, 3014: 1, 3015: 1, 3016: 1, 3017: 1, 3018: 3, 3019: 1, 3020: 2, 3021: 1, 3022: 3, 3023: 1, 3024: 1, 3025: 1, 3026: 3, 3027: 1, 3028: 1, 3029: 1, 3030: 1, 3031: 1, 3032: 1, 3033: 1, 3034: 3, 3035: 1, 3036: 1, 3037: 2, 3038: 1, 3039: 2, 3040: 1, 3041: 2, 3042: 1, 3043: 1, 3044: 1, 3045: 1, 3046: 1, 3047: 1, 3048: 1, 3049: 1, 3050: 1, 3051: 1, 3052: 2, 3053: 1, 3054: 1, 3055: 1, 3056: 1, 3057: 1, 3058: 1, 3059: 1, 3060: 1, 3061: 5, 3062: 1, 3063: 1, 3064: 1, 3065: 1, 3066: 1, 3067: 1, 3068: 1, 3069: 1, 3070: 1, 3071: 1, 3072: 1, 3073: 1, 3074: 1, 3075: 1, 3076: 1, 3077: 1, 3078: 1, 3079: 1, 3080: 1, 3081: 1, 3082: 1, 3083: 1, 3084: 2, 3085: 1, 3086: 1, 3087: 1, 3088: 1, 3089: 1, 3090: 1, 3091: 1, 3092: 1, 3093: 1, 3094: 1, 3095: 1, 3096: 1, 3097: 1, 3098: 1, 3099: 1, 3100: 4, 3101: 1, 3102: 1, 3103: 1, 3104: 2, 3105: 1, 3106: 1, 3107: 1, 3108: 1, 3109: 1, 3110: 1, 3111: 1, 3112: 2, 3113: 1, 3114: 1, 3115: 1, 3116: 1, 3117: 1, 3118: 3, 3119: 1, 3120: 1, 3121: 1, 3122: 1, 3123: 1, 3124: 2, 3125: 1, 3126: 1, 3127: 2, 3128: 1, 3129: 1, 3130: 5, 3131: 1, 3132: 1, 3133: 1, 3134: 2, 3135: 1, 3136: 1, 3137: 1, 3138: 1, 3139: 1, 3140: 1, 3141: 1, 3142: 1, 3143: 1, 3144: 1, 3145: 1, 3146: 1, 3147: 1, 3148: 1, 3149: 1, 3150: 2, 3151: 1, 3152: 2, 3153: 1, 3154: 1, 3155: 1, 3156: 1, 3157: 1, 3158: 1, 3159: 1, 3160: 1, 3161: 1, 3162: 1, 3163: 1, 3164: 1, 3165: 1, 3166: 2, 3167: 1, 3168: 2, 3169: 1, 3170: 1, 3171: 1, 3172: 1, 3173: 1, 3174: 1, 3175: 1, 3176: 1, 3177: 1, 3178: 1, 3179: 1, 3180: 1, 3181: 1, 3182: 1, 3183: 1, 3184: 2, 3185: 1, 3186: 1, 3187: 1, 3188: 1, 3189: 1, 3190: 1, 3191: 1, 3192: 1, 3193: 1, 3194: 1, 3195: 1, 3196: 1, 3197: 1, 3198: 1, 3199: 4, 3200: 1, 3201: 1, 3202: 2, 3203: 1, 3204: 1, 3205: 2, 3206: 1, 3207: 1, 3208: 1, 3209: 1, 3210: 1, 3211: 1, 3212: 1, 3213: 1, 3214: 1, 3215: 1, 3216: 1, 3217: 2, 3218: 1, 3219: 1, 3220: 1, 3221: 1, 3222: 1, 3223: 1, 3224: 1, 3225: 3, 3226: 1, 3227: 1, 3228: 1, 3229: 1, 3230: 1, 3231: 1, 3232: 1, 3233: 2, 3234: 1, 3235: 1, 3236: 1, 3237: 1, 3238: 1, 3239: 1, 3240: 3, 3241: 1, 3242: 1, 3243: 1, 3244: 2, 3245: 1, 3246: 1, 3247: 1, 3248: 1, 3249: 1, 3250: 1, 3251: 1, 3252: 1, 3253: 1, 3254: 1, 3255: 1, 3256: 2, 3257: 1, 3258: 1, 3259: 2, 3260: 1, 3261: 1, 3262: 1, 3263: 1, 3264: 2, 3265: 1, 3266: 3, 3267: 1, 3268: 1, 3269: 1, 3270: 3, 3271: 1, 3272: 1, 3273: 1, 3274: 1, 3275: 1, 3276: 3, 3277: 1, 3278: 1, 3279: 1, 3280: 1, 3281: 1, 3282: 1, 3283: 1, 3284: 1, 3285: 1, 3286: 1, 3287: 1, 3288: 1, 3289: 1, 3290: 2, 3291: 1, 3292: 1, 3293: 1, 3294: 1, 3295: 1, 3296: 1, 3297: 1, 3298: 1, 3299: 1, 3300: 1, 3301: 1, 3302: 1, 3303: 1, 3304: 1, 3305: 1, 3306: 1, 3307: 1, 3308: 1, 3309: 1, 3310: 1, 3311: 1, 3312: 1, 3313: 1, 3314: 1, 3315: 1, 3316: 1, 3317: 1, 3318: 1, 3319: 1, 3320: 1, 3321: 1, 3322: 1, 3323: 1, 3324: 2, 3325: 1, 3326: 1, 3327: 1, 3328: 1, 3329: 1, 3330: 1, 3331: 1, 3332: 1, 3333: 2, 3334: 1, 3335: 1, 3336: 1, 3337: 1, 3338: 1, 3339: 1, 3340: 1, 3341: 1, 3342: 1, 3343: 1, 3344: 1, 3345: 1, 3346: 1, 3347: 1, 3348: 1, 3349: 1, 3350: 1, 3351: 1, 3352: 3, 3353: 1, 3354: 2, 3355: 1, 3356: 1, 3357: 1, 3358: 1, 3359: 1, 3360: 3, 3361: 1, 3362: 1, 3363: 1, 3364: 3, 3365: 1, 3366: 1, 3367: 1, 3368: 1, 3369: 1, 3370: 1, 3371: 1, 3372: 1, 3373: 1, 3374: 2, 3375: 1, 3376: 2, 3377: 1, 3378: 2, 3379: 1, 3380: 1, 3381: 1, 3382: 3, 3383: 1, 3384: 1, 3385: 1, 3386: 1, 3387: 1, 3388: 1, 3389: 1, 3390: 1, 3391: 1, 3392: 1, 3393: 1, 3394: 1, 3395: 1, 3396: 1, 3397: 2, 3398: 1, 3399: 1, 3400: 1, 3401: 1, 3402: 2, 3403: 1, 3404: 1, 3405: 1, 3406: 1, 3407: 1, 3408: 1, 3409: 1, 3410: 1, 3411: 1, 3412: 1, 3413: 1, 3414: 1, 3415: 1, 3416: 1, 3417: 1, 3418: 1, 3419: 1, 3420: 1, 3421: 1, 3422: 1, 3423: 1, 3424: 1, 3425: 1, 3426: 1, 3427: 1, 3428: 4, 3429: 2, 3430: 1, 3431: 1, 3432: 1, 3433: 1, 3434: 1, 3435: 1, 3436: 3, 3437: 1, 3438: 1, 3439: 4, 3440: 1, 3441: 1, 3442: 1, 3443: 3, 3444: 1, 3445: 1, 3446: 1, 3447: 1, 3448: 1, 3449: 1, 3450: 1, 3451: 1, 3452: 1, 3453: 1, 3454: 1, 3455: 1, 3456: 1, 3457: 1, 3458: 1, 3459: 1, 3460: 1, 3461: 1, 3462: 1, 3463: 1, 3464: 1, 3465: 1, 3466: 1, 3467: 1, 3468: 2, 3469: 1, 3470: 1, 3471: 1, 3472: 1, 3473: 1, 3474: 1, 3475: 1, 3476: 1, 3477: 1, 3478: 1, 3479: 1, 3480: 1, 3481: 1, 3482: 1, 3483: 1, 3484: 1, 3485: 1, 3486: 1, 3487: 1, 3488: 1, 3489: 1, 3490: 2, 3491: 1, 3492: 1, 3493: 1, 3494: 2, 3495: 1, 3496: 1, 3497: 1, 3498: 2, 3499: 1, 3500: 1, 3501: 1, 3502: 1, 3503: 3, 3504: 1, 3505: 3, 3506: 1, 3507: 1, 3508: 1, 3509: 3, 3510: 1, 3511: 3, 3512: 1, 3513: 1, 3514: 1, 3515: 3, 3516: 1, 3517: 1, 3518: 1, 3519: 1, 3520: 1, 3521: 1, 3522: 1, 3523: 1, 3524: 1, 3525: 1, 3526: 1, 3527: 1, 3528: 1, 3529: 1, 3530: 1, 3531: 1, 3532: 1, 3533: 1, 3534: 2, 3535: 1, 3536: 1, 3537: 1, 3538: 1, 3539: 1, 3540: 1, 3541: 1, 3542: 1, 3543: 2, 3544: 1, 3545: 2, 3546: 1, 3547: 1, 3548: 1, 3549: 2, 3550: 1, 3551: 3, 3552: 1, 3553: 1, 3554: 1, 3555: 1, 3556: 1, 3557: 2, 3558: 1, 3559: 2, 3560: 1, 3561: 1, 3562: 1, 3563: 1, 3564: 1, 3565: 1, 3566: 1, 3567: 1, 3568: 1, 3569: 1, 3570: 1, 3571: 1, 3572: 1, 3573: 1, 3574: 1, 3575: 1, 3576: 1, 3577: 1, 3578: 1, 3579: 1, 3580: 1, 3581: 1, 3582: 1, 3583: 1, 3584: 1, 3585: 1, 3586: 1, 3587: 1, 3588: 1, 3589: 1, 3590: 1, 3591: 2, 3592: 1, 3593: 1, 3594: 1, 3595: 2, 3596: 1, 3597: 2, 3598: 1, 3599: 2, 3600: 1, 3601: 1, 3602: 2, 3603: 1, 3604: 3, 3605: 1, 3606: 1, 3607: 1, 3608: 1, 3609: 1, 3610: 2, 3611: 1, 3612: 3, 3613: 1, 3614: 1, 3615: 1, 3616: 1, 3617: 1, 3618: 1, 3619: 1, 3620: 1, 3621: 1, 3622: 1, 3623: 1, 3624: 1, 3625: 1, 3626: 2, 3627: 1, 3628: 2, 3629: 1, 3630: 1, 3631: 1, 3632: 1, 3633: 1, 3634: 1, 3635: 1, 3636: 1, 3637: 1, 3638: 1, 3639: 1, 3640: 1, 3641: 1, 3642: 1, 3643: 1, 3644: 1, 3645: 1, 3646: 1, 3647: 1, 3648: 1, 3649: 1, 3650: 2, 3651: 1, 3652: 1, 3653: 1, 3654: 1, 3655: 1, 3656: 1, 3657: 1, 3658: 1, 3659: 1, 3660: 1, 3661: 1, 3662: 1, 3663: 1, 3664: 1, 3665: 1, 3666: 1, 3667: 1, 3668: 1, 3669: 1, 3670: 1, 3671: 1, 3672: 1, 3673: 1, 3674: 2, 3675: 1, 3676: 1, 3677: 1, 3678: 1, 3679: 1, 3680: 1, 3681: 1, 3682: 3, 3683: 1, 3684: 1, 3685: 1, 3686: 1, 3687: 1, 3688: 1, 3689: 1, 3690: 3, 3691: 1, 3692: 1, 3693: 1, 3694: 1, 3695: 1, 3696: 1, 3697: 2, 3698: 1, 3699: 1, 3700: 1, 3701: 2, 3702: 2, 3703: 1, 3704: 1, 3705: 1, 3706: 1, 3707: 1, 3708: 1, 3709: 1, 3710: 2, 3711: 1, 3712: 1, 3713: 1, 3714: 1, 3715: 1, 3716: 1, 3717: 1, 3718: 1, 3719: 1, 3720: 1, 3721: 1, 3722: 1, 3723: 1, 3724: 1, 3725: 1, 3726: 1, 3727: 1, 3728: 1, 3729: 1, 3730: 1, 3731: 1, 3732: 1, 3733: 1, 3734: 2, 3735: 1, 3736: 1, 3737: 2, 3738: 1, 3739: 1, 3740: 1, 3741: 1, 3742: 2, 3743: 1, 3744: 1, 3745: 1, 3746: 1, 3747: 1, 3748: 1, 3749: 1, 3750: 1, 3751: 1, 3752: 1, 3753: 1, 3754: 1, 3755: 1, 3756: 1, 3757: 1, 3758: 1, 3759: 1, 3760: 1, 3761: 1, 3762: 1, 3763: 1, 3764: 1, 3765: 1, 3766: 1, 3767: 1, 3768: 1, 3769: 1, 3770: 1, 3771: 1, 3772: 1, 3773: 1, 3774: 1, 3775: 1, 3776: 1, 3777: 1, 3778: 2, 3779: 2, 3780: 1, 3781: 1, 3782: 1, 3783: 1, 3784: 1, 3785: 1, 3786: 1, 3787: 1, 3788: 1, 3789: 1, 3790: 1, 3791: 1, 3792: 1, 3793: 1, 3794: 1, 3795: 1, 3796: 2, 3797: 2, 3798: 1, 3799: 1, 3800: 1, 3801: 1, 3802: 1, 3803: 1, 3804: 1, 3805: 2, 3806: 1, 3807: 1, 3808: 1, 3809: 1, 3810: 1, 3811: 2, 3812: 1, 3813: 1, 3814: 1, 3815: 1, 3816: 1, 3817: 1, 3818: 2, 3819: 2, 3820: 1, 3821: 1, 3822: 1, 3823: 1, 3824: 1, 3825: 1, 3826: 1, 3827: 1, 3828: 1, 3829: 2, 3830: 1, 3831: 1, 3832: 1, 3833: 1, 3834: 1, 3835: 1, 3836: 1, 3837: 1, 3838: 1, 3839: 1, 3840: 1, 3841: 1, 3842: 1, 3843: 1, 3844: 1, 3845: 1, 3846: 1, 3847: 1, 3848: 2, 3849: 1, 3850: 1, 3851: 1, 3852: 1, 3853: 1, 3854: 1, 3855: 1, 3856: 1, 3857: 1, 3858: 1, 3859: 1, 3860: 1, 3861: 1, 3862: 1, 3863: 1, 3864: 3, 3865: 1, 3866: 1, 3867: 1, 3868: 1, 3869: 1, 3870: 1, 3871: 1, 3872: 1, 3873: 1, 3874: 2, 3875: 1, 3876: 2, 3877: 1, 3878: 1, 3879: 1, 3880: 1, 3881: 1, 3882: 1, 3883: 1, 3884: 2, 3885: 1, 3886: 2, 3887: 1, 3888: 1, 3889: 1, 3890: 1, 3891: 1, 3892: 1, 3893: 1, 3894: 1, 3895: 1, 3896: 1, 3897: 1, 3898: 1, 3899: 1, 3900: 1, 3901: 1, 3902: 1, 3903: 1, 3904: 1, 3905: 1, 3906: 1, 3907: 1, 3908: 1, 3909: 2, 3910: 2, 3911: 1, 3912: 1, 3913: 1, 3914: 1, 3915: 1, 3916: 1, 3917: 1, 3918: 1, 3919: 1, 3920: 1, 3921: 1, 3922: 1, 3923: 1, 3924: 1, 3925: 1, 3926: 1, 3927: 1, 3928: 2, 3929: 1, 3930: 2, 3931: 1, 3932: 2, 3933: 1, 3934: 2, 3935: 1, 3936: 1, 3937: 1, 3938: 1, 3939: 1, 3940: 1, 3941: 1, 3942: 1, 3943: 1, 3944: 1, 3945: 1, 3946: 1, 3947: 1, 3948: 1, 3949: 1, 3950: 1, 3951: 1, 3952: 1, 3953: 1, 3954: 1, 3955: 1, 3956: 1, 3957: 1, 3958: 1, 3959: 1, 3960: 1, 3961: 1, 3962: 1, 3963: 1, 3964: 1, 3965: 2, 3966: 1, 3967: 3, 3968: 1, 3969: 1, 3970: 1, 3971: 1, 3972: 1, 3973: 1, 3974: 1, 3975: 3, 3976: 1, 3977: 1, 3978: 1, 3979: 1, 3980: 1, 3981: 1, 3982: 1, 3983: 1, 3984: 1, 3985: 1, 3986: 1, 3987: 1, 3988: 1, 3989: 1, 3990: 1, 3991: 1, 3992: 1, 3993: 1, 3994: 1, 3995: 1, 3996: 1, 3997: 1, 3998: 1, 3999: 1, 4000: 1, 4001: 1, 4002: 1, 4003: 1, 4004: 1, 4005: 1, 4006: 1, 4007: 1, 4008: 1, 4009: 1, 4010: 1, 4011: 1, 4012: 1, 4013: 2, 4014: 1, 4015: 1, 4016: 1, 4017: 1, 4018: 1, 4019: 1, 4020: 1, 4021: 1, 4022: 1, 4023: 1, 4024: 1, 4025: 1, 4026: 1, 4027: 1, 4028: 1, 4029: 1, 4030: 1, 4031: 1, 4032: 1, 4033: 1, 4034: 1, 4035: 1, 4036: 1, 4037: 1, 4038: 1, 4039: 1, 4040: 1, 4041: 1, 4042: 1, 4043: 1, 4044: 1, 4045: 1, 4046: 1, 4047: 1, 4048: 1, 4049: 1, 4050: 1, 4051: 1, 4052: 1, 4053: 1, 4054: 1, 4055: 1, 4056: 1, 4057: 1, 4058: 1, 4059: 1, 4060: 1, 4061: 1, 4062: 1, 4063: 1, 4064: 1, 4065: 1, 4066: 2, 4067: 2, 4068: 1, 4069: 1, 4070: 1, 4071: 1, 4072: 1, 4073: 1, 4074: 1, 4075: 1, 4076: 1, 4077: 1, 4078: 1, 4079: 1, 4080: 1, 4081: 1, 4082: 2, 4083: 1, 4084: 1, 4085: 1, 4086: 1, 4087: 1, 4088: 1, 4089: 1, 4090: 1, 4091: 1, 4092: 1, 4093: 1, 4094: 1, 4095: 1, 4096: 1, 4097: 1, 4098: 1, 4099: 1, 4100: 1, 4101: 1, 4102: 1, 4103: 1, 4104: 2, 4105: 1, 4106: 1, 4107: 1, 4108: 1, 4109: 1, 4110: 1, 4111: 1, 4112: 1, 4113: 1, 4114: 1, 4115: 1, 4116: 1, 4117: 1, 4118: 1, 4119: 1, 4120: 1, 4121: 1, 4122: 1, 4123: 1, 4124: 1, 4125: 2, 4126: 1, 4127: 1, 4128: 1, 4129: 1, 4130: 1, 4131: 1, 4132: 1, 4133: 1, 4134: 1, 4135: 1, 4136: 1, 4137: 1, 4138: 1, 4139: 1, 4140: 1, 4141: 1, 4142: 1, 4143: 1, 4144: 1, 4145: 1, 4146: 1, 4147: 1, 4148: 1, 4149: 1, 4150: 1, 4151: 1}, 101: {0: 10452, 1: 2095, 2: 1162, 3: 2649, 4: 2142, 5: 1444, 6: 1371, 7: 1484, 8: 1136, 9: 897, 10: 855, 11: 741, 12: 626, 13: 587, 14: 564, 15: 1207, 16: 588, 17: 489, 18: 454, 19: 570, 20: 387, 21: 366, 22: 535, 23: 407, 24: 269, 25: 320, 26: 253, 27: 308, 28: 298, 29: 489, 30: 240, 31: 172, 32: 168, 33: 155, 34: 190, 35: 299, 36: 196, 37: 189, 38: 171, 39: 132, 40: 195, 41: 179, 42: 130, 43: 110, 44: 401, 45: 113, 46: 434, 47: 130, 48: 117, 49: 80, 50: 98, 51: 109, 52: 66, 53: 89, 54: 104, 55: 76, 56: 124, 57: 90, 58: 75, 59: 216, 60: 75, 61: 82, 62: 97, 63: 50, 64: 92, 65: 77, 66: 61, 67: 75, 68: 196, 69: 53, 70: 50, 71: 47, 72: 67, 73: 49, 74: 46, 75: 81, 76: 59, 77: 43, 78: 592, 79: 63, 80: 41, 81: 38, 82: 52, 83: 36, 84: 31, 85: 34, 86: 39, 87: 40, 88: 29, 89: 34, 90: 28, 91: 69, 92: 30, 93: 33, 94: 20, 95: 94, 96: 19, 97: 23, 98: 26, 99: 18, 100: 39, 101: 25, 102: 15, 103: 23, 104: 13, 105: 23, 106: 16, 107: 8, 108: 13, 109: 17, 110: 8, 111: 15, 112: 16, 113: 21, 114: 20, 115: 15, 116: 162, 117: 24, 118: 17, 119: 16, 120: 22, 121: 10, 122: 19, 123: 18, 124: 26, 125: 11, 126: 74, 127: 13, 128: 15, 129: 15, 130: 62, 131: 25, 132: 25, 133: 22, 134: 17, 135: 17, 136: 6, 137: 17, 138: 65, 139: 10, 140: 15, 141: 18, 142: 3, 143: 11, 144: 9, 145: 11, 146: 3, 147: 8, 148: 12, 149: 7, 150: 2, 151: 14, 152: 3, 153: 15, 154: 43, 155: 5, 156: 10, 157: 7, 158: 11, 159: 4, 160: 4, 161: 15, 162: 1, 163: 5, 164: 2, 165: 1, 166: 5, 167: 2, 168: 7, 169: 6, 171: 2, 172: 9, 173: 8, 175: 8, 176: 9, 177: 7, 178: 2, 179: 3, 180: 6, 181: 8, 182: 10, 183: 9, 184: 3, 185: 7, 186: 2, 187: 7, 188: 7, 189: 8, 190: 7, 191: 7, 192: 10, 193: 10, 194: 7, 195: 8, 196: 10, 197: 7, 198: 8, 199: 3, 200: 7, 201: 5, 202: 1, 203: 8, 204: 3, 205: 2, 206: 2, 207: 2, 208: 1, 209: 7, 210: 1, 211: 7, 212: 1, 213: 7, 215: 7, 216: 2, 217: 6, 218: 2, 219: 7, 221: 5, 223: 5, 224: 6, 225: 21, 226: 6, 227: 19, 228: 6, 229: 9, 230: 6, 231: 8, 232: 7, 233: 2, 234: 6, 235: 21, 236: 6, 238: 6, 239: 2, 240: 6, 241: 6, 242: 6, 244: 1, 245: 6, 246: 2, 247: 7, 248: 3, 249: 7, 250: 2, 251: 7, 252: 9, 253: 3, 254: 7, 255: 4, 256: 7, 257: 2, 258: 7, 259: 2, 260: 7, 261: 2, 262: 9, 263: 2, 264: 9, 265: 2, 266: 7, 267: 1, 268: 7, 269: 1, 270: 7, 271: 1, 272: 7, 273: 37, 274: 10, 275: 7, 276: 2, 278: 6, 280: 9, 281: 1, 282: 6, 283: 5, 284: 20, 285: 7, 286: 3, 287: 7, 288: 2, 289: 4, 290: 10, 291: 2, 292: 6, 293: 2, 294: 6, 295: 1, 296: 6, 297: 4, 298: 9, 299: 11, 300: 8, 302: 8, 303: 4, 304: 6, 305: 1, 306: 2, 307: 7, 308: 1, 309: 3, 311: 2, 313: 4, 314: 1, 316: 6, 318: 6, 319: 1, 320: 6, 321: 3, 322: 6, 324: 6, 326: 6, 328: 6, 329: 6, 331: 6, 332: 1, 333: 6, 335: 7, 336: 2, 337: 6, 338: 4, 339: 6, 340: 7, 342: 6, 344: 7, 345: 2, 346: 6, 347: 2, 348: 7, 349: 1, 350: 6, 351: 3, 352: 7, 353: 2, 354: 6, 356: 6, 358: 6, 360: 6, 361: 1, 362: 6, 363: 2, 364: 6, 366: 6, 368: 6, 369: 7, 370: 4, 371: 6, 372: 9, 373: 6, 374: 1, 375: 2, 377: 2, 378: 2, 379: 8, 380: 2, 381: 7, 385: 5, 386: 5, 387: 6, 388: 1, 390: 1, 392: 3, 393: 3, 394: 7, 395: 6, 396: 6, 397: 6, 398: 6, 399: 6, 400: 6, 401: 6, 402: 6, 403: 6, 404: 2, 405: 2, 406: 1, 407: 1, 408: 1, 409: 1, 410: 1, 411: 1, 413: 3, 414: 2, 415: 2, 416: 2, 417: 1, 418: 1, 419: 2, 420: 1, 421: 1, 422: 1, 423: 4, 424: 3, 425: 3, 426: 5, 427: 2, 428: 1, 429: 1, 431: 6, 432: 3, 433: 7, 434: 1, 435: 7, 436: 1, 437: 7, 438: 2, 439: 5, 440: 6, 443: 1, 444: 2, 447: 1, 448: 8, 449: 5, 450: 7, 451: 3, 452: 2, 453: 7, 454: 5, 455: 5, 458: 5, 459: 5, 462: 2, 463: 2, 467: 1, 468: 2, 476: 2, 486: 2, 488: 7, 491: 6, 496: 1, 502: 5, 503: 6, 504: 6, 505: 5, 506: 5, 509: 2, 510: 2, 511: 2, 512: 2, 513: 2, 514: 2, 528: 2, 529: 2, 530: 2, 531: 2, 532: 2, 533: 2, 534: 12, 570: 2, 571: 2, 572: 2, 573: 2, 574: 2, 576: 2, 577: 2, 657: 2, 658: 2, 659: 2, 660: 2, 661: 2, 662: 2, 680: 1, 681: 1, 691: 1, 692: 2, 693: 3, 694: 2, 695: 1, 720: 1, 721: 1, 722: 2, 723: 2, 737: 1, 742: 1, 776: 2, 777: 2, 778: 2, 779: 2, 780: 3, 781: 3, 782: 3, 783: 2, 784: 3, 785: 3, 786: 3, 787: 3, 788: 2, 789: 2, 790: 2, 819: 1, 820: 1, 876: 2, 877: 2, 878: 3, 879: 3, 880: 1, 932: 1, 940: 1, 1064: 1, 1065: 1, 1091: 2, 1092: 2, 1093: 2, 1094: 2, 1095: 3, 1096: 3, 1097: 3, 1098: 2, 1099: 3, 1100: 3, 1101: 3, 1102: 3, 1103: 2, 1104: 2, 1105: 2, 1192: 1, 1193: 1, 1215: 1, 1216: 1, 1218: 1, 1220: 1, 1249: 1, 1253: 1, 1254: 1, 1278: 5, 1279: 4, 1280: 5, 1281: 2, 1290: 1, 1295: 1, 1296: 1, 1352: 2, 1353: 2, 1354: 2, 1355: 2, 1366: 1, 1367: 1, 1368: 1, 1396: 1, 1398: 1, 1399: 1, 1400: 1, 1405: 1, 1425: 1, 1426: 1, 1428: 1, 1429: 1, 1441: 1, 1469: 1}, 102: {0: 893, 1: 5421, 2: 17106, 3: 4819, 4: 1602, 5: 1033, 6: 396, 7: 132, 8: 84, 9: 100, 10: 477, 11: 12, 12: 40, 13: 10, 14: 11, 15: 16, 16: 8, 17: 4, 18: 4, 19: 8, 20: 7, 21: 4, 22: 6, 23: 1, 24: 3, 25: 1, 26: 1, 27: 1, 30: 4, 31: 1, 32: 3, 37: 1, 38: 1, 41: 1, 42: 1, 44: 3, 47: 2, 57: 2, 128: 1}, 103: {0: 5603, 1: 5385, 2: 2671, 3: 1898, 4: 858, 5: 507, 6: 330, 7: 223, 8: 123, 9: 92, 10: 91, 11: 50, 12: 68, 13: 36, 14: 27, 15: 28, 16: 18, 17: 16, 18: 16, 19: 13, 20: 14, 21: 11, 22: 8, 23: 7, 24: 12, 25: 11, 26: 3, 27: 5, 28: 2, 29: 1, 30: 2, 31: 1, 32: 2, 33: 5, 34: 5, 35: 2, 36: 5, 38: 3, 39: 1, 41: 1, 42: 2, 43: 2, 44: 6, 45: 1, 50: 1, 51: 1, 53: 1, 54: 2, 59: 1, 61: 1, 63: 1, 65: 1, 66: 1, 67: 1, 70: 1, 75: 1, 76: 1, 78: 1, 79: 1, 80: 1, 81: 1, 85: 1, 97: 1, 100: 1, 102: 1, 116: 1, 132: 1, 424: 1, 505: 1, 511: 1}, 104: {0: 29, 1: 137, 2: 133, 3: 60, 4: 29, 5: 3, 6: 8, 7: 9, 8: 3, 9: 2, 10: 3, 11: 1, 12: 4, 13: 1, 14: 1, 15: 1, 16: 1, 17: 1, 18: 2, 47: 1, 75: 1, 126: 1}, 105: {0: 1931, 1: 1247, 2: 549, 3: 336, 4: 132, 5: 85, 6: 60, 7: 39, 8: 41, 9: 8, 10: 17, 11: 8, 12: 7, 13: 14, 14: 7, 15: 5, 16: 1, 17: 5, 18: 6, 19: 2, 20: 5, 21: 4, 22: 1, 23: 1, 24: 1, 25: 1, 27: 1, 28: 1, 33: 3, 34: 1, 35: 1, 37: 1, 38: 1, 40: 1, 42: 3, 44: 4, 45: 2, 48: 1, 49: 1, 52: 1, 73: 1, 77: 1, 126: 2, 127: 1, 128: 16, 149: 1, 208: 2, 247: 1, 250: 1, 252: 1, 253: 1, 256: 15, 322: 2, 551: 2, 642: 1, 650: 1, 2231: 1}, 106: {0: 36100, 1: 34967, 2: 34211, 3: 29541, 4: 23552, 5: 18739, 6: 15499, 7: 12622, 8: 10370, 9: 8341, 10: 6738, 11: 5305, 12: 4377, 13: 3702, 14: 2877, 15: 2454, 16: 2034, 17: 1716, 18: 1495, 19: 1289, 20: 1118, 21: 1040, 22: 762, 23: 736, 24: 709, 25: 516, 26: 454, 27: 395, 28: 378, 29: 363, 30: 284, 31: 260, 32: 258, 33: 264, 34: 204, 35: 179, 36: 172, 37: 158, 38: 151, 39: 132, 40: 128, 41: 184, 42: 126, 43: 91, 44: 85, 45: 116, 46: 79, 47: 70, 48: 117, 49: 69, 50: 53, 51: 60, 52: 62, 53: 40, 54: 40, 55: 49, 56: 69, 57: 34, 58: 28, 59: 31, 60: 21, 61: 24, 62: 28, 63: 26, 64: 34, 65: 34, 66: 23, 67: 19, 68: 25, 69: 10, 70: 69, 71: 22, 72: 19, 73: 48, 74: 14, 75: 17, 76: 17, 77: 16, 78: 13, 79: 19, 80: 23, 81: 35, 82: 17, 83: 20, 84: 9, 85: 13, 86: 12, 87: 6, 88: 13, 89: 6, 90: 7, 91: 4, 92: 10, 93: 7, 94: 5, 95: 12, 96: 9, 97: 6, 98: 7, 99: 8, 100: 6, 101: 3, 102: 14, 103: 4, 104: 13, 105: 7, 106: 5, 107: 5, 108: 3, 109: 3, 110: 5, 111: 37, 112: 3, 113: 10, 114: 2, 115: 2, 116: 9, 117: 4, 118: 7, 119: 6, 120: 10, 121: 5, 122: 4, 123: 21, 124: 2, 125: 3, 126: 3, 127: 7, 128: 5, 129: 4, 130: 6, 131: 10, 132: 4, 133: 5, 134: 7, 135: 7, 136: 10, 137: 3, 138: 4, 139: 3, 140: 2, 141: 4, 142: 2, 143: 1, 144: 1, 145: 2, 146: 3, 147: 1, 148: 1, 149: 1, 150: 1, 151: 1, 152: 1, 153: 1, 154: 4, 155: 1, 156: 1, 158: 1, 159: 3, 160: 2, 161: 3, 162: 4, 186: 1, 205: 1, 222: 1, 223: 1}, 107: {0: 1560, 1: 875, 2: 8238, 3: 1899, 4: 1376, 5: 951, 6: 2708, 7: 852, 8: 3627, 9: 2347, 10: 4536}, 108: {0: 1567, 1: 1393, 2: 996, 3: 887, 4: 710, 5: 591, 6: 524, 7: 449, 8: 394, 9: 348, 10: 265, 11: 201, 12: 208, 13: 154, 14: 121, 15: 107, 16: 157, 17: 99, 18: 84, 19: 92, 20: 54, 21: 51, 22: 44, 23: 33, 24: 24, 25: 37, 26: 24, 27: 32, 28: 25, 29: 13, 30: 9, 31: 13, 32: 9, 33: 12, 34: 16, 35: 8, 36: 7, 37: 8, 38: 9, 39: 5, 40: 7, 41: 8, 42: 3, 43: 4, 44: 4, 45: 4, 46: 4, 47: 2, 50: 2, 51: 1, 52: 1, 53: 2, 54: 5, 56: 6, 57: 1, 58: 4, 59: 2, 62: 3, 63: 1, 64: 3, 67: 1, 68: 1, 69: 1, 71: 1, 72: 1, 73: 3, 75: 1, 76: 1, 77: 2, 79: 2, 91: 1, 97: 2, 98: 1, 101: 1, 114: 1, 117: 2, 119: 2, 122: 3, 128: 3, 138: 3, 246: 1}, 109: {0: 14, 1: 544, 2: 402, 3: 326, 4: 329, 5: 338, 6: 325, 7: 268, 8: 270, 9: 251, 10: 244, 11: 229, 12: 174, 13: 175, 14: 139, 15: 130, 16: 108, 17: 84, 18: 89, 19: 72, 20: 71, 21: 56, 22: 54, 23: 40, 24: 39, 25: 22, 26: 33, 27: 24, 28: 16, 29: 20, 30: 21, 31: 18, 32: 21, 33: 13, 34: 16, 35: 20, 36: 12, 37: 11, 38: 11, 39: 9, 40: 9, 41: 7, 42: 6, 43: 8, 44: 4, 45: 4, 46: 2, 47: 2, 48: 5, 49: 3, 50: 4, 51: 3, 52: 4, 53: 1, 54: 3, 55: 3, 56: 3, 57: 2, 58: 3, 59: 3, 60: 5, 61: 1, 65: 3, 66: 2, 71: 2, 72: 2, 74: 1, 76: 1, 79: 1, 80: 2, 81: 1, 92: 1, 95: 1, 96: 1, 97: 1, 99: 1, 100: 1, 106: 1, 107: 1, 111: 1, 112: 1, 113: 1, 114: 1, 118: 1, 119: 1, 120: 2, 121: 2, 122: 3, 123: 2, 124: 2, 125: 2, 129: 1, 130: 1, 156: 1}, 110: {0: 6, 1: 3753, 2: 299, 3: 542, 4: 287, 5: 32, 6: 612, 7: 79, 8: 106, 9: 286, 10: 283, 11: 69, 12: 423, 13: 372, 14: 424, 15: 150, 16: 323, 17: 120, 18: 1269, 19: 267, 20: 108, 21: 343, 22: 450, 23: 89, 24: 554, 25: 166, 26: 80, 27: 167, 28: 224, 29: 67, 30: 222, 31: 231, 32: 64, 33: 168, 34: 205, 35: 92, 36: 138, 37: 127, 38: 86, 39: 88, 40: 137, 41: 110, 42: 86, 43: 133, 44: 75, 45: 72, 46: 116, 47: 83, 48: 57, 49: 123, 50: 94, 51: 58, 52: 82, 53: 111, 54: 88, 55: 60, 56: 85, 57: 36, 58: 67, 59: 120, 60: 74, 61: 63, 62: 81, 63: 97, 64: 35, 65: 82, 66: 63, 67: 47, 68: 45, 69: 55, 70: 37, 71: 39, 72: 43, 73: 35, 74: 27, 75: 47, 76: 46, 77: 33, 78: 55, 79: 28, 80: 30, 81: 36, 82: 36, 83: 22, 84: 38, 85: 29, 86: 24, 87: 24, 88: 41, 89: 25, 90: 33, 91: 30, 92: 24, 93: 33, 94: 34, 95: 29, 96: 25, 97: 24, 98: 21, 99: 29, 100: 12, 101: 30, 102: 12, 103: 19, 104: 23, 105: 14, 106: 21, 107: 22, 108: 14, 109: 25, 110: 12, 111: 21, 112: 19, 113: 14, 114: 27, 115: 20, 116: 17, 117: 8, 118: 10, 119: 25, 120: 11, 121: 14, 122: 17, 123: 15, 124: 16, 125: 17, 126: 11, 127: 14, 128: 16, 129: 12, 130: 10, 131: 21, 132: 15, 133: 11, 134: 8, 135: 15, 136: 12, 137: 13, 138: 6, 139: 18, 140: 8, 141: 11, 142: 10, 143: 10, 144: 4, 145: 13, 146: 3, 147: 13, 148: 14, 149: 4, 150: 6, 151: 14, 152: 12, 153: 8, 154: 4, 155: 10, 156: 9, 157: 6, 158: 6, 159: 14, 160: 3, 161: 8, 162: 3, 163: 10, 164: 3, 165: 7, 166: 2, 167: 6, 168: 8, 169: 6, 170: 4, 171: 10, 172: 12, 173: 9, 174: 9, 175: 5, 176: 8, 177: 2, 178: 5, 179: 6, 180: 6, 181: 6, 182: 8, 183: 7, 184: 6, 185: 6, 186: 6, 187: 5, 188: 10, 189: 4, 190: 4, 191: 3, 192: 4, 193: 2, 194: 10, 195: 5, 196: 5, 197: 4, 198: 4, 199: 7, 200: 3, 201: 7, 202: 6, 203: 6, 204: 5, 205: 4, 206: 7, 207: 2, 208: 4, 209: 6, 210: 4, 211: 2, 212: 2, 213: 5, 215: 5, 216: 6, 217: 2, 218: 1, 219: 6, 220: 3, 221: 1, 222: 3, 223: 4, 224: 1, 226: 1, 227: 3, 228: 4, 229: 5, 230: 6, 231: 1, 232: 2, 233: 2, 234: 6, 235: 5, 236: 1, 237: 6, 238: 3, 239: 5, 240: 1, 241: 3, 242: 2, 243: 2, 244: 3, 245: 1, 246: 3, 247: 1, 248: 2, 249: 1, 250: 2, 251: 7, 252: 3, 253: 1, 254: 5, 255: 1, 256: 4, 257: 5, 258: 3, 259: 1, 260: 4, 261: 2, 262: 3, 263: 2, 264: 5, 265: 1, 266: 3, 267: 7, 269: 5, 270: 1, 271: 1, 273: 2, 274: 2, 275: 7, 276: 1, 277: 1, 278: 1, 279: 3, 280: 1, 282: 2, 283: 2, 285: 2, 286: 1, 287: 4, 288: 3, 289: 2, 290: 3, 292: 3, 295: 1, 296: 6, 297: 2, 298: 4, 301: 1, 303: 4, 304: 4, 305: 2, 306: 3, 307: 4, 308: 3, 310: 1, 311: 1, 312: 3, 313: 3, 314: 1, 315: 5, 316: 1, 317: 1, 319: 2, 320: 2, 321: 1, 323: 3, 324: 1, 326: 2, 327: 1, 328: 1, 330: 5, 331: 2, 332: 1, 335: 2, 337: 2, 338: 2, 341: 1, 342: 2, 345: 1, 346: 1, 347: 1, 348: 1, 351: 3, 352: 1, 353: 1, 354: 5, 355: 2, 359: 1, 360: 5, 361: 1, 365: 2, 366: 3, 368: 6, 369: 3, 371: 3, 372: 2, 374: 2, 379: 1, 380: 2, 381: 3, 382: 2, 383: 2, 386: 1, 387: 2, 388: 5, 389: 3, 391: 1, 392: 1, 394: 1, 395: 1, 396: 3, 399: 1, 401: 2, 405: 4, 406: 3, 408: 1, 415: 2, 416: 1, 417: 2, 419: 1, 421: 2, 422: 4, 423: 1, 431: 2, 432: 2, 433: 2, 434: 2, 436: 1, 439: 1, 440: 2, 441: 1, 445: 1, 446: 1, 448: 1, 450: 1, 451: 1, 454: 1, 458: 1, 462: 2, 467: 1, 470: 1, 471: 1, 477: 2, 478: 1, 479: 1, 480: 2, 482: 1, 484: 1, 485: 1, 486: 1, 489: 2, 496: 1, 501: 1, 502: 1, 503: 1, 507: 2, 510: 1, 511: 3, 513: 1, 515: 1, 520: 1, 524: 1, 527: 2, 528: 2, 536: 1, 540: 1, 543: 1, 544: 1, 546: 1, 551: 1, 552: 1, 558: 1, 564: 1, 566: 1, 567: 2, 573: 1, 579: 2, 582: 1, 583: 2, 590: 1, 605: 1, 609: 1, 612: 1, 614: 1, 626: 1, 632: 2, 635: 2, 636: 1, 640: 1, 649: 1, 651: 1, 654: 1, 655: 2, 667: 1, 668: 1, 676: 1, 687: 1, 699: 2, 705: 2, 706: 1, 723: 1, 728: 2, 730: 1, 732: 2, 741: 1, 754: 1, 757: 1, 759: 1, 770: 1, 782: 1, 783: 1, 784: 2, 798: 2, 808: 1, 809: 3, 821: 1, 840: 1, 844: 1, 849: 1, 868: 2, 871: 1, 880: 1, 889: 2, 914: 3, 935: 1, 957: 1, 969: 1, 977: 1, 980: 2, 988: 1, 1003: 1, 1005: 2, 1006: 1, 1009: 1, 1019: 1, 1034: 1, 1035: 1, 1050: 1, 1128: 2, 1136: 1, 1184: 1, 1215: 1, 1217: 1, 1246: 1, 1472: 2, 1520: 2, 1657: 2, 1703: 2, 1734: 2, 2468: 2, 3275: 2, 3312: 2}, 111: {9: 10, 10: 2, 11: 3, 12: 6, 13: 1, 15: 11, 16: 17, 17: 2, 18: 12, 19: 23, 21: 64, 22: 31, 23: 17, 24: 34, 25: 17, 26: 16, 27: 52, 28: 21, 29: 9, 30: 28, 31: 16, 32: 9, 33: 54, 34: 27, 35: 10, 36: 32, 37: 22, 38: 7, 39: 40, 40: 25, 41: 6, 42: 24, 43: 34, 44: 17, 45: 34, 46: 25, 47: 17, 48: 12, 49: 24, 50: 12, 51: 20, 52: 21, 53: 21, 54: 21, 55: 15, 56: 17, 57: 22, 58: 6, 59: 11, 60: 17, 61: 17, 62: 17, 63: 26, 64: 15, 65: 20, 66: 21, 67: 23, 68: 15, 69: 38, 70: 8, 71: 18, 72: 16, 73: 16, 74: 24, 75: 17, 76: 18, 77: 12, 78: 18, 79: 17, 80: 6, 81: 11, 82: 12, 83: 19, 84: 38, 85: 20, 86: 9, 87: 16, 88: 20, 89: 8, 90: 11, 91: 9, 92: 13, 93: 20, 94: 5, 95: 11, 96: 8, 97: 21, 98: 16, 99: 11, 100: 10, 101: 22, 102: 8, 103: 16, 104: 18, 105: 15, 106: 6, 107: 15, 108: 8, 109: 8, 110: 14, 111: 12, 112: 12, 113: 7, 114: 17, 115: 12, 116: 6, 117: 9, 118: 8, 119: 3, 120: 6, 121: 6, 122: 6, 123: 13, 124: 14, 125: 8, 126: 9, 127: 7, 128: 9, 129: 10, 130: 8, 131: 8, 132: 16, 133: 15, 134: 17, 135: 7, 136: 9, 137: 6, 138: 6, 139: 11, 140: 13, 141: 4, 142: 7, 143: 10, 144: 7, 145: 9, 146: 19, 147: 11, 148: 12, 149: 12, 150: 14, 151: 12, 152: 16, 153: 9, 154: 8, 155: 13, 156: 12, 157: 7, 158: 14, 159: 11, 160: 10, 161: 5, 162: 5, 163: 1, 164: 4, 165: 6, 166: 6, 167: 10, 168: 5, 169: 2, 170: 6, 171: 5, 172: 15, 173: 3, 174: 5, 175: 11, 176: 2, 177: 11, 178: 9, 179: 9, 180: 5, 181: 9, 182: 5, 183: 15, 184: 7, 185: 8, 186: 6, 187: 6, 188: 4, 189: 12, 190: 5, 191: 7, 192: 12, 193: 8, 194: 4, 195: 8, 196: 6, 198: 8, 199: 7, 200: 1, 201: 9, 202: 10, 203: 5, 204: 2, 205: 21, 206: 5, 207: 10, 208: 5, 209: 6, 210: 7, 211: 9, 212: 2, 213: 9, 214: 3, 215: 6, 216: 2, 217: 2, 218: 5, 219: 7, 220: 6, 221: 6, 222: 13, 223: 9, 224: 1, 225: 5, 226: 8, 227: 4, 228: 6, 229: 5, 231: 5, 232: 3, 233: 3, 234: 9, 235: 7, 236: 6, 237: 15, 238: 15, 239: 1, 240: 4, 241: 12, 242: 5, 243: 29, 244: 15, 245: 4, 246: 6, 247: 2, 248: 5, 249: 6, 250: 2, 251: 4, 252: 3, 253: 3, 254: 3, 255: 5, 256: 3, 257: 5, 258: 10, 259: 4, 260: 7, 261: 5, 262: 6, 263: 3, 264: 2, 265: 7, 266: 4, 267: 3, 268: 2, 269: 13, 270: 3, 271: 9, 272: 4, 273: 4, 274: 3, 276: 5, 277: 3, 278: 7, 279: 1, 280: 6, 281: 6, 282: 7, 283: 8, 284: 13, 285: 6, 286: 1, 287: 4, 288: 2, 289: 5, 290: 1, 291: 3, 292: 9, 293: 2, 294: 7, 295: 2, 296: 1, 297: 6, 299: 5, 300: 3, 301: 5, 302: 8, 303: 2, 304: 9, 305: 3, 306: 3, 307: 12, 308: 2, 309: 3, 310: 1, 312: 3, 314: 1, 315: 9, 316: 1, 317: 1, 318: 7, 319: 6, 320: 7, 321: 2, 322: 2, 323: 4, 324: 3, 326: 4, 327: 3, 328: 3, 331: 6, 333: 3, 334: 2, 335: 1, 336: 4, 338: 4, 339: 2, 340: 3, 341: 1, 342: 4, 343: 3, 344: 1, 345: 8, 346: 2, 347: 2, 348: 3, 349: 4, 350: 2, 351: 3, 352: 4, 353: 2, 355: 2, 356: 2, 357: 3, 358: 3, 359: 7, 361: 12, 362: 4, 363: 1, 365: 3, 366: 3, 367: 6, 368: 3, 369: 2, 370: 3, 371: 5, 372: 2, 373: 2, 375: 2, 377: 3, 378: 3, 379: 2, 380: 2, 381: 8, 382: 4, 383: 2, 384: 2, 385: 2, 386: 3, 387: 2, 389: 1, 390: 1, 392: 6, 393: 3, 395: 3, 396: 1, 398: 1, 400: 3, 401: 4, 402: 1, 405: 1, 406: 2, 408: 1, 410: 1, 411: 2, 412: 6, 414: 2, 417: 2, 418: 1, 420: 2, 421: 4, 423: 4, 424: 2, 425: 2, 427: 5, 428: 2, 429: 2, 430: 4, 431: 3, 432: 1, 433: 4, 434: 3, 436: 1, 437: 6, 438: 1, 439: 2, 440: 1, 441: 2, 443: 1, 444: 4, 445: 4, 447: 3, 448: 1, 449: 4, 450: 1, 452: 6, 453: 1, 455: 11, 457: 1, 460: 3, 461: 2, 462: 1, 463: 2, 464: 2, 465: 1, 467: 1, 469: 4, 470: 1, 471: 2, 472: 1, 476: 1, 477: 2, 478: 2, 483: 3, 484: 2, 485: 2, 486: 1, 487: 11, 488: 2, 490: 2, 491: 3, 492: 2, 493: 3, 494: 1, 495: 1, 497: 8, 498: 2, 499: 2, 504: 2, 505: 1, 507: 2, 510: 2, 514: 1, 516: 5, 517: 1, 518: 1, 519: 6, 521: 2, 522: 2, 524: 11, 525: 5, 526: 4, 527: 1, 528: 8, 529: 2, 530: 2, 532: 3, 533: 1, 535: 1, 536: 2, 537: 4, 539: 5, 542: 2, 543: 1, 544: 1, 545: 1, 547: 1, 549: 1, 550: 3, 551: 2, 552: 1, 553: 2, 554: 1, 556: 2, 557: 1, 558: 3, 559: 1, 563: 3, 565: 1, 568: 2, 572: 2, 573: 3, 574: 3, 575: 1, 577: 4, 578: 7, 579: 6, 580: 4, 582: 1, 583: 1, 584: 2, 586: 1, 587: 1, 589: 1, 590: 2, 591: 2, 592: 4, 597: 3, 598: 1, 601: 4, 602: 1, 604: 1, 605: 3, 613: 8, 615: 2, 616: 1, 617: 1, 618: 2, 619: 1, 620: 2, 621: 1, 624: 3, 625: 1, 626: 1, 628: 5, 630: 1, 631: 1, 632: 2, 635: 1, 636: 1, 638: 1, 641: 2, 644: 1, 647: 1, 648: 5, 653: 1, 657: 3, 658: 1, 660: 1, 661: 1, 663: 5, 664: 3, 668: 7, 669: 10, 670: 3, 672: 1, 674: 1, 676: 1, 679: 1, 680: 4, 681: 5, 683: 1, 685: 1, 687: 3, 690: 2, 692: 2, 693: 2, 695: 1, 696: 1, 698: 1, 699: 1, 702: 1, 708: 1, 709: 2, 711: 1, 713: 1, 717: 2, 723: 1, 725: 1, 729: 1, 730: 1, 732: 1, 734: 2, 742: 2, 750: 1, 752: 1, 758: 1, 759: 1, 761: 1, 763: 1, 765: 1, 767: 1, 769: 2, 777: 1, 779: 1, 781: 4, 783: 2, 785: 1, 791: 1, 792: 2, 797: 1, 802: 4, 808: 2, 812: 2, 816: 1, 817: 1, 818: 3, 819: 4, 821: 1, 824: 13, 827: 2, 832: 4, 842: 1, 847: 2, 858: 1, 860: 7, 870: 2, 872: 2, 874: 1, 877: 1, 892: 2, 893: 1, 909: 2, 928: 1, 929: 1, 944: 3, 947: 1, 953: 2, 959: 2, 960: 2, 963: 1, 985: 1, 993: 1, 995: 1, 1001: 2, 1015: 7, 1019: 1, 1027: 1, 1028: 4, 1029: 1, 1033: 1, 1050: 1, 1052: 1, 1056: 1, 1059: 1, 1061: 1, 1093: 1, 1099: 1, 1101: 1, 1105: 12, 1107: 2, 1114: 1, 1116: 2, 1120: 1, 1129: 1, 1144: 3, 1157: 1, 1159: 1, 1165: 1, 1183: 1, 1189: 1, 1192: 1, 1193: 4, 1200: 1, 1202: 2, 1226: 1, 1245: 1, 1306: 1, 1321: 2, 1335: 2, 1339: 1, 1353: 2, 1372: 1, 1375: 1, 1381: 1, 1401: 2, 1406: 1, 1410: 1, 1445: 4, 1453: 1, 1455: 1, 1488: 1, 1538: 1, 1568: 2, 1616: 1, 1642: 2, 1712: 2, 1755: 1, 1842: 1, 1923: 2, 1973: 1, 1980: 1, 2002: 1, 2025: 1, 2041: 1, 2061: 2, 2064: 1, 2137: 2, 2193: 4, 2194: 5, 2332: 1, 2709: 1, 3303: 2, 3325: 1}, 112: {9: 11, 12: 40, 13: 2, 15: 30, 16: 4, 17: 1, 18: 23, 19: 10, 21: 57, 22: 7, 23: 5, 24: 36, 25: 2, 26: 1, 27: 66, 28: 8, 29: 6, 30: 26, 31: 11, 32: 9, 33: 36, 34: 18, 35: 7, 36: 20, 37: 17, 38: 3, 39: 33, 40: 11, 41: 6, 42: 28, 43: 22, 44: 2, 45: 17, 46: 5, 47: 10, 48: 12, 49: 10, 50: 5, 51: 20, 52: 6, 53: 2, 54: 19, 55: 10, 56: 3, 57: 21, 58: 24, 59: 2, 60: 14, 61: 14, 62: 17, 63: 9, 64: 9, 65: 11, 66: 11, 67: 8, 68: 5, 69: 11, 70: 7, 71: 6, 72: 9, 73: 6, 74: 4, 75: 11, 76: 16, 77: 7, 78: 15, 79: 10, 80: 13, 81: 12, 82: 7, 83: 1, 84: 6, 85: 14, 86: 5, 87: 16, 88: 2, 89: 8, 90: 14, 91: 8, 92: 4, 93: 9, 94: 2, 95: 5, 96: 7, 97: 6, 98: 2, 99: 12, 100: 2, 101: 9, 102: 13, 103: 6, 104: 3, 105: 13, 106: 9, 107: 6, 108: 10, 109: 4, 110: 3, 111: 3, 112: 7, 113: 3, 114: 2, 115: 4, 116: 1, 117: 9, 118: 3, 119: 6, 120: 6, 121: 5, 122: 6, 123: 3, 124: 4, 125: 13, 126: 6, 127: 7, 128: 11, 129: 2, 130: 2, 131: 2, 132: 5, 133: 7, 134: 6, 135: 2, 136: 7, 137: 5, 138: 2, 139: 2, 140: 1, 141: 1, 142: 2, 143: 3, 144: 7, 145: 3, 146: 5, 147: 5, 148: 3, 149: 3, 150: 8, 151: 8, 152: 5, 153: 6, 154: 8, 155: 5, 156: 6, 158: 3, 159: 3, 160: 3, 161: 8, 162: 2, 163: 5, 164: 3, 165: 5, 166: 3, 167: 5, 168: 8, 169: 1, 170: 6, 171: 2, 172: 8, 173: 2, 174: 2, 175: 2, 176: 5, 177: 3, 178: 7, 179: 1, 180: 2, 181: 5, 182: 3, 183: 8, 184: 1, 185: 2, 186: 3, 187: 7, 188: 4, 189: 1, 190: 2, 191: 4, 192: 4, 193: 1, 194: 3, 196: 3, 197: 2, 198: 2, 200: 2, 201: 2, 202: 7, 203: 3, 205: 2, 206: 6, 207: 4, 208: 5, 209: 3, 211: 2, 212: 3, 213: 3, 214: 2, 215: 2, 216: 2, 217: 3, 218: 3, 219: 2, 220: 1, 223: 4, 224: 2, 225: 1, 226: 2, 227: 1, 228: 3, 230: 2, 231: 1, 232: 1, 233: 3, 234: 6, 235: 3, 236: 2, 237: 4, 238: 1, 240: 3, 241: 9, 242: 8, 243: 1, 244: 3, 245: 1, 246: 1, 247: 1, 248: 4, 249: 7, 250: 1, 251: 2, 252: 1, 253: 2, 255: 2, 256: 2, 257: 2, 258: 1, 259: 1, 261: 1, 262: 2, 263: 2, 265: 4, 266: 4, 267: 6, 268: 2, 269: 3, 270: 2, 272: 3, 273: 2, 277: 1, 278: 1, 279: 2, 280: 1, 281: 2, 283: 2, 284: 2, 285: 2, 287: 1, 288: 1, 289: 3, 291: 4, 292: 2, 293: 2, 295: 1, 296: 1, 297: 1, 299: 1, 300: 4, 301: 1, 303: 3, 304: 1, 305: 2, 307: 1, 308: 3, 309: 1, 311: 2, 312: 1, 315: 3, 316: 1, 317: 2, 318: 2, 320: 2, 321: 1, 323: 2, 326: 1, 327: 3, 330: 1, 331: 2, 332: 2, 334: 1, 335: 2, 336: 2, 337: 1, 338: 3, 339: 1, 341: 2, 345: 1, 347: 2, 349: 2, 350: 1, 351: 3, 353: 2, 354: 1, 355: 1, 356: 3, 357: 3, 358: 1, 359: 4, 361: 2, 362: 2, 363: 4, 364: 1, 366: 1, 369: 4, 374: 1, 375: 1, 376: 2, 378: 3, 380: 2, 383: 1, 384: 1, 385: 1, 386: 1, 387: 1, 388: 4, 389: 1, 392: 2, 393: 1, 394: 4, 396: 1, 399: 2, 400: 1, 401: 2, 404: 2, 405: 2, 407: 2, 412: 1, 413: 1, 415: 2, 416: 1, 417: 3, 428: 1, 431: 3, 432: 3, 433: 1, 435: 4, 437: 2, 439: 1, 440: 2, 441: 2, 443: 1, 446: 2, 447: 1, 449: 2, 450: 1, 454: 1, 455: 1, 456: 3, 467: 1, 471: 1, 472: 1, 475: 2, 478: 1, 479: 1, 483: 1, 488: 1, 492: 2, 497: 2, 498: 1, 499: 2, 500: 1, 503: 1, 507: 1, 510: 1, 511: 4, 512: 1, 514: 2, 518: 2, 527: 1, 528: 1, 532: 1, 540: 1, 547: 1, 549: 2, 550: 2, 551: 2, 555: 1, 556: 2, 558: 2, 564: 1, 568: 1, 569: 2, 570: 3, 577: 1, 578: 3, 579: 10, 581: 1, 586: 1, 588: 1, 589: 1, 590: 4, 594: 1, 598: 1, 602: 1, 604: 1, 605: 1, 611: 1, 615: 1, 619: 1, 620: 1, 625: 1, 627: 1, 628: 1, 630: 1, 632: 1, 638: 1, 641: 2, 643: 3, 652: 2, 653: 1, 654: 1, 659: 2, 663: 1, 668: 1, 669: 3, 672: 1, 674: 1, 676: 1, 685: 1, 701: 1, 705: 2, 706: 2, 714: 1, 717: 1, 735: 2, 740: 2, 742: 2, 752: 1, 765: 2, 769: 1, 780: 2, 784: 1, 795: 1, 803: 2, 815: 1, 821: 2, 822: 1, 831: 1, 835: 1, 842: 1, 854: 1, 871: 1, 890: 1, 902: 1, 905: 1, 911: 1, 915: 1, 938: 1, 961: 1, 963: 1, 969: 2, 985: 1, 1015: 2, 1029: 1, 1042: 1, 1059: 1, 1080: 1, 1104: 1, 1186: 2, 1252: 1, 1258: 1, 1265: 1, 1298: 2, 1335: 1, 1343: 1, 1365: 1, 1445: 2, 1464: 1, 1516: 1, 1642: 2, 1653: 1, 1740: 2, 1757: 1, 1779: 1, 1807: 2, 1862: 4, 1947: 4, 3610: 1, 3692: 1}, 113: {3: 677, 6: 1396, 7: 195, 9: 62, 10: 203, 11: 10, 12: 32, 13: 443, 14: 13, 15: 42, 16: 254, 17: 20, 18: 27, 19: 233, 20: 54, 21: 41, 22: 178, 23: 41, 24: 38, 25: 188, 26: 55, 27: 33, 28: 106, 29: 65, 30: 37, 31: 126, 32: 58, 33: 31, 34: 102, 35: 76, 36: 24, 37: 98, 38: 68, 39: 25, 40: 88, 41: 82, 42: 29, 43: 98, 44: 66, 45: 43, 46: 73, 47: 57, 48: 25, 49: 71, 50: 53, 51: 45, 52: 48, 53: 81, 54: 59, 55: 54, 56: 59, 57: 29, 58: 60, 59: 48, 60: 28, 61: 45, 62: 41, 63: 37, 64: 43, 65: 59, 66: 30, 67: 67, 68: 53, 69: 39, 70: 56, 71: 38, 72: 35, 73: 39, 74: 36, 75: 25, 76: 39, 77: 42, 78: 28, 79: 35, 80: 38, 81: 32, 82: 35, 83: 20, 84: 30, 85: 33, 86: 17, 87: 26, 88: 27, 89: 29, 90: 22, 91: 45, 92: 29, 93: 39, 94: 29, 95: 29, 96: 24, 97: 31, 98: 30, 99: 20, 100: 18, 101: 29, 102: 15, 103: 38, 104: 28, 105: 21, 106: 37, 107: 40, 108: 17, 109: 19, 110: 20, 111: 25, 112: 35, 113: 24, 114: 25, 115: 18, 116: 19, 117: 21, 118: 31, 119: 13, 120: 17, 121: 33, 122: 10, 123: 18, 124: 20, 125: 37, 126: 28, 127: 15, 128: 26, 129: 23, 130: 20, 131: 20, 132: 22, 133: 28, 134: 23, 135: 23, 136: 20, 137: 12, 138: 22, 139: 21, 140: 11, 141: 15, 142: 22, 143: 17, 144: 17, 145: 25, 146: 21, 147: 12, 148: 15, 149: 15, 150: 12, 151: 19, 152: 9, 153: 12, 154: 22, 155: 15, 156: 15, 157: 13, 158: 10, 159: 10, 160: 14, 161: 24, 162: 21, 163: 13, 164: 13, 165: 15, 166: 15, 167: 8, 168: 15, 169: 12, 170: 14, 171: 15, 172: 14, 173: 6, 174: 17, 175: 11, 176: 23, 177: 6, 178: 13, 179: 12, 180: 12, 181: 6, 182: 21, 183: 13, 184: 20, 185: 10, 186: 7, 187: 11, 188: 14, 189: 11, 190: 11, 191: 17, 192: 15, 193: 9, 194: 8, 195: 13, 196: 10, 197: 24, 198: 15, 199: 15, 200: 10, 201: 16, 202: 16, 203: 11, 204: 10, 205: 10, 206: 10, 207: 10, 208: 12, 209: 16, 210: 9, 211: 7, 212: 11, 213: 19, 214: 15, 215: 9, 216: 15, 217: 16, 218: 9, 219: 13, 220: 12, 221: 11, 222: 13, 223: 5, 224: 3, 225: 4, 226: 8, 227: 9, 228: 11, 229: 9, 230: 8, 231: 8, 232: 8, 233: 9, 234: 13, 235: 10, 236: 5, 237: 14, 238: 5, 239: 10, 240: 9, 241: 5, 242: 15, 243: 2, 244: 3, 245: 5, 246: 12, 247: 3, 248: 7, 249: 5, 250: 13, 251: 11, 252: 8, 253: 6, 254: 9, 255: 12, 256: 8, 257: 6, 258: 4, 259: 9, 260: 5, 261: 8, 262: 17, 263: 12, 264: 13, 265: 8, 266: 8, 267: 12, 268: 7, 269: 8, 270: 9, 271: 3, 272: 8, 273: 9, 274: 7, 275: 5, 276: 7, 277: 7, 278: 7, 279: 4, 280: 6, 281: 2, 282: 2, 283: 8, 284: 7, 285: 6, 286: 7, 287: 6, 288: 8, 289: 4, 290: 9, 291: 3, 292: 7, 293: 8, 294: 2, 295: 11, 296: 3, 297: 9, 298: 11, 299: 12, 300: 6, 301: 7, 302: 7, 303: 2, 304: 12, 305: 4, 306: 7, 307: 6, 308: 12, 309: 6, 310: 5, 311: 5, 312: 9, 313: 1, 314: 9, 315: 5, 316: 9, 317: 6, 318: 4, 319: 7, 320: 6, 321: 1, 322: 7, 323: 6, 324: 6, 325: 14, 326: 3, 327: 5, 328: 5, 329: 5, 330: 5, 331: 5, 332: 3, 333: 6, 334: 11, 335: 7, 336: 6, 337: 3, 338: 6, 339: 8, 340: 6, 341: 5, 342: 2, 343: 5, 344: 4, 345: 4, 346: 3, 347: 2, 348: 1, 349: 3, 350: 3, 351: 2, 353: 7, 354: 5, 355: 4, 356: 7, 357: 5, 358: 1, 359: 2, 360: 10, 361: 5, 362: 6, 363: 1, 364: 2, 365: 4, 366: 5, 367: 5, 368: 4, 369: 1, 370: 1, 371: 3, 372: 6, 373: 7, 374: 1, 375: 5, 376: 5, 377: 5, 378: 4, 379: 2, 380: 1, 381: 3, 382: 7, 383: 2, 384: 6, 385: 7, 386: 5, 387: 6, 389: 4, 390: 3, 391: 4, 392: 6, 394: 5, 395: 1, 396: 6, 397: 2, 398: 3, 399: 2, 400: 3, 401: 3, 403: 9, 405: 1, 406: 4, 407: 1, 408: 5, 409: 4, 410: 2, 411: 1, 412: 3, 413: 6, 414: 3, 415: 3, 416: 5, 417: 5, 418: 7, 419: 1, 420: 1, 421: 2, 422: 1, 423: 3, 424: 5, 425: 1, 426: 3, 427: 5, 428: 3, 429: 9, 430: 2, 432: 5, 433: 4, 434: 3, 435: 5, 436: 7, 437: 1, 438: 4, 439: 4, 440: 4, 441: 4, 442: 2, 443: 2, 444: 5, 445: 3, 446: 5, 447: 5, 448: 6, 449: 6, 450: 2, 451: 1, 452: 5, 453: 2, 454: 1, 455: 6, 456: 2, 458: 5, 459: 1, 460: 3, 462: 1, 463: 2, 464: 4, 465: 4, 466: 2, 468: 4, 469: 5, 470: 1, 471: 2, 472: 7, 473: 2, 474: 8, 475: 5, 476: 2, 477: 5, 478: 4, 479: 3, 480: 4, 482: 2, 483: 1, 486: 5, 487: 1, 488: 1, 489: 2, 490: 2, 491: 3, 492: 2, 493: 4, 494: 1, 495: 2, 497: 6, 498: 3, 499: 1, 500: 2, 501: 2, 502: 3, 503: 1, 504: 4, 506: 6, 507: 3, 508: 2, 510: 3, 511: 2, 512: 1, 513: 1, 514: 2, 515: 2, 517: 6, 518: 2, 520: 3, 521: 1, 522: 1, 523: 2, 524: 2, 525: 8, 526: 3, 527: 1, 528: 3, 529: 2, 530: 2, 533: 3, 534: 1, 536: 2, 537: 1, 538: 2, 539: 2, 540: 1, 541: 3, 542: 3, 543: 2, 544: 7, 546: 3, 547: 1, 549: 4, 551: 2, 552: 2, 553: 1, 554: 2, 555: 2, 556: 2, 557: 3, 558: 1, 559: 6, 560: 1, 561: 3, 562: 2, 564: 1, 565: 2, 566: 4, 567: 3, 569: 2, 570: 1, 571: 3, 572: 2, 573: 3, 574: 1, 576: 2, 577: 1, 578: 3, 579: 2, 582: 3, 583: 1, 584: 1, 586: 1, 587: 1, 588: 2, 589: 3, 590: 2, 591: 2, 593: 1, 594: 2, 596: 1, 597: 1, 598: 3, 599: 2, 600: 2, 602: 1, 604: 1, 605: 1, 606: 1, 608: 2, 609: 1, 612: 2, 613: 2, 614: 1, 615: 1, 616: 1, 617: 4, 618: 1, 619: 1, 620: 1, 621: 2, 622: 1, 623: 1, 624: 1, 625: 4, 626: 1, 627: 2, 628: 2, 631: 3, 632: 3, 633: 1, 634: 1, 635: 3, 636: 2, 640: 1, 641: 2, 642: 4, 643: 1, 646: 2, 647: 1, 648: 1, 649: 2, 652: 1, 653: 1, 654: 2, 655: 1, 656: 1, 658: 1, 659: 1, 660: 3, 661: 3, 663: 1, 667: 1, 668: 1, 669: 1, 670: 4, 671: 2, 672: 3, 674: 1, 675: 1, 677: 2, 680: 2, 683: 1, 685: 1, 687: 3, 689: 3, 690: 3, 692: 1, 696: 1, 698: 2, 700: 2, 701: 2, 702: 1, 703: 3, 705: 2, 706: 1, 707: 1, 708: 3, 710: 1, 712: 1, 714: 2, 715: 1, 716: 1, 717: 2, 719: 1, 720: 3, 723: 3, 727: 1, 730: 2, 732: 1, 735: 1, 736: 2, 737: 4, 739: 1, 740: 1, 741: 4, 742: 2, 743: 1, 744: 1, 747: 2, 750: 1, 753: 1, 754: 1, 755: 1, 757: 1, 758: 1, 764: 1, 765: 1, 766: 4, 768: 1, 769: 1, 770: 4, 771: 1, 772: 1, 773: 1, 780: 2, 781: 2, 784: 1, 785: 1, 786: 1, 787: 1, 789: 1, 792: 1, 793: 3, 796: 3, 800: 2, 802: 1, 804: 3, 806: 1, 809: 2, 812: 2, 815: 1, 818: 1, 823: 1, 826: 1, 828: 1, 829: 1, 836: 2, 838: 2, 840: 1, 843: 1, 844: 2, 848: 1, 850: 1, 851: 1, 852: 2, 855: 1, 857: 3, 863: 2, 864: 1, 865: 2, 866: 1, 868: 3, 870: 1, 871: 4, 873: 1, 874: 1, 875: 1, 877: 1, 878: 1, 884: 1, 886: 2, 891: 2, 892: 1, 896: 2, 897: 1, 898: 1, 900: 2, 906: 3, 909: 1, 910: 1, 911: 3, 912: 2, 913: 1, 915: 2, 918: 2, 919: 1, 921: 1, 924: 3, 928: 1, 929: 2, 931: 2, 935: 1, 936: 1, 938: 1, 940: 1, 946: 1, 949: 4, 952: 1, 953: 1, 954: 2, 955: 1, 957: 1, 958: 1, 959: 1, 966: 1, 973: 2, 974: 3, 988: 1, 992: 1, 1002: 1, 1003: 1, 1011: 1, 1012: 1, 1014: 2, 1016: 1, 1017: 1, 1019: 1, 1022: 1, 1025: 1, 1030: 1, 1031: 1, 1034: 1, 1037: 3, 1038: 1, 1043: 1, 1048: 2, 1052: 2, 1054: 2, 1056: 1, 1058: 1, 1059: 2, 1061: 1, 1063: 1, 1065: 1, 1068: 1, 1070: 2, 1073: 1, 1078: 1, 1084: 1, 1085: 1, 1089: 1, 1092: 1, 1093: 2, 1094: 1, 1097: 1, 1099: 1, 1101: 2, 1107: 1, 1111: 2, 1114: 1, 1118: 1, 1127: 1, 1134: 1, 1142: 2, 1147: 1, 1152: 2, 1155: 2, 1157: 1, 1159: 1, 1161: 1, 1171: 1, 1176: 1, 1177: 3, 1181: 1, 1184: 1, 1189: 1, 1196: 1, 1199: 1, 1203: 1, 1204: 2, 1206: 1, 1207: 1, 1215: 1, 1216: 1, 1218: 1, 1225: 1, 1226: 1, 1229: 1, 1234: 1, 1239: 1, 1240: 1, 1243: 1, 1244: 1, 1249: 2, 1250: 1, 1252: 3, 1254: 8, 1265: 1, 1271: 1, 1273: 1, 1276: 2, 1283: 3, 1292: 1, 1297: 1, 1308: 1, 1314: 2, 1319: 2, 1320: 2, 1322: 1, 1324: 1, 1330: 2, 1338: 1, 1340: 1, 1347: 2, 1358: 1, 1367: 1, 1370: 1, 1372: 1, 1385: 1, 1398: 1, 1399: 1, 1415: 1, 1417: 1, 1423: 1, 1431: 1, 1432: 1, 1448: 1, 1456: 2, 1460: 1, 1469: 2, 1471: 1, 1474: 1, 1475: 1, 1497: 1, 1499: 1, 1503: 1, 1511: 1, 1517: 1, 1518: 1, 1520: 3, 1536: 1, 1557: 1, 1580: 1, 1589: 1, 1590: 2, 1609: 1, 1615: 1, 1624: 1, 1638: 1, 1651: 1, 1661: 1, 1663: 2, 1677: 1, 1714: 1, 1730: 1, 1732: 1, 1780: 1, 1784: 1, 1786: 2, 1789: 1, 1797: 2, 1800: 2, 1814: 1, 1816: 1, 1865: 1, 1928: 1, 1953: 2, 1957: 3, 1968: 1, 2065: 1, 2074: 1, 2102: 1, 2143: 1, 2164: 1, 2219: 1, 2238: 1, 2349: 1, 2364: 2, 2391: 1, 2398: 1, 2416: 1, 2486: 2, 2505: 1, 2531: 1, 2716: 1, 2750: 1, 3058: 2, 5254: 1, 5421: 1, 6060: 1, 14959: 1}, 114: {7: 2, 10: 6, 11: 53, 12: 8, 13: 61, 14: 33, 15: 15, 16: 164, 17: 24, 18: 129, 19: 288, 20: 127, 21: 246, 22: 216, 23: 75, 24: 274, 25: 234, 26: 91, 27: 202, 28: 264, 29: 63, 30: 267, 31: 299, 32: 124, 33: 191, 34: 270, 35: 117, 36: 215, 37: 246, 38: 137, 39: 197, 40: 278, 41: 193, 42: 207, 43: 257, 44: 200, 45: 172, 46: 204, 47: 171, 48: 186, 49: 222, 50: 150, 51: 154, 52: 181, 53: 165, 54: 120, 55: 191, 56: 163, 57: 135, 58: 154, 59: 168, 60: 180, 61: 167, 62: 167, 63: 146, 64: 138, 65: 154, 66: 155, 67: 133, 68: 171, 69: 148, 70: 173, 71: 162, 72: 169, 73: 160, 74: 143, 75: 121, 76: 125, 77: 137, 78: 160, 79: 147, 80: 144, 81: 151, 82: 130, 83: 132, 84: 120, 85: 153, 86: 130, 87: 159, 88: 121, 89: 122, 90: 143, 91: 130, 92: 121, 93: 131, 94: 131, 95: 118, 96: 130, 97: 125, 98: 113, 99: 124, 100: 134, 101: 133, 102: 125, 103: 116, 104: 101, 105: 124, 106: 109, 107: 130, 108: 137, 109: 116, 110: 109, 111: 109, 112: 102, 113: 88, 114: 105, 115: 113, 116: 104, 117: 108, 118: 113, 119: 76, 120: 110, 121: 79, 122: 94, 123: 97, 124: 119, 125: 88, 126: 112, 127: 103, 128: 95, 129: 82, 130: 113, 131: 108, 132: 124, 133: 94, 134: 87, 135: 81, 136: 82, 137: 82, 138: 74, 139: 105, 140: 94, 141: 103, 142: 102, 143: 94, 144: 88, 145: 98, 146: 79, 147: 82, 148: 81, 149: 75, 150: 78, 151: 82, 152: 86, 153: 66, 154: 98, 155: 58, 156: 75, 157: 77, 158: 59, 159: 101, 160: 64, 161: 69, 162: 66, 163: 85, 164: 70, 165: 81, 166: 67, 167: 82, 168: 65, 169: 78, 170: 78, 171: 68, 172: 71, 173: 59, 174: 76, 175: 64, 176: 78, 177: 68, 178: 77, 179: 84, 180: 70, 181: 65, 182: 73, 183: 67, 184: 66, 185: 63, 186: 67, 187: 64, 188: 69, 189: 65, 190: 50, 191: 65, 192: 74, 193: 74, 194: 69, 195: 58, 196: 60, 197: 53, 198: 63, 199: 64, 200: 71, 201: 51, 202: 62, 203: 65, 204: 60, 205: 51, 206: 53, 207: 43, 208: 67, 209: 62, 210: 55, 211: 48, 212: 63, 213: 49, 214: 53, 215: 50, 216: 46, 217: 33, 218: 60, 219: 44, 220: 56, 221: 54, 222: 66, 223: 50, 224: 72, 225: 52, 226: 53, 227: 42, 228: 46, 229: 64, 230: 55, 231: 48, 232: 37, 233: 51, 234: 44, 235: 52, 236: 40, 237: 28, 238: 38, 239: 35, 240: 44, 241: 38, 242: 47, 243: 34, 244: 45, 245: 32, 246: 43, 247: 48, 248: 54, 249: 48, 250: 39, 251: 40, 252: 43, 253: 49, 254: 35, 255: 41, 256: 30, 257: 30, 258: 52, 259: 28, 260: 44, 261: 33, 262: 33, 263: 47, 264: 35, 265: 43, 266: 37, 267: 39, 268: 49, 269: 31, 270: 39, 271: 33, 272: 44, 273: 34, 274: 45, 275: 36, 276: 44, 277: 29, 278: 40, 279: 41, 280: 35, 281: 33, 282: 35, 283: 34, 284: 33, 285: 37, 286: 39, 287: 47, 288: 36, 289: 25, 290: 42, 291: 36, 292: 35, 293: 40, 294: 33, 295: 28, 296: 33, 297: 39, 298: 27, 299: 34, 300: 20, 301: 23, 302: 48, 303: 25, 304: 29, 305: 23, 306: 47, 307: 28, 308: 30, 309: 25, 310: 30, 311: 34, 312: 31, 313: 25, 314: 28, 315: 26, 316: 28, 317: 17, 318: 23, 319: 33, 320: 20, 321: 25, 322: 29, 323: 23, 324: 32, 325: 20, 326: 23, 327: 22, 328: 27, 329: 24, 330: 36, 331: 30, 332: 26, 333: 23, 334: 36, 335: 26, 336: 33, 337: 20, 338: 19, 339: 27, 340: 27, 341: 27, 342: 21, 343: 19, 344: 31, 345: 20, 346: 26, 347: 41, 348: 21, 349: 18, 350: 24, 351: 38, 352: 25, 353: 32, 354: 21, 355: 22, 356: 30, 357: 24, 358: 25, 359: 26, 360: 36, 361: 27, 362: 28, 363: 21, 364: 28, 365: 25, 366: 32, 367: 19, 368: 26, 369: 28, 370: 21, 371: 23, 372: 18, 373: 17, 374: 28, 375: 25, 376: 31, 377: 23, 378: 30, 379: 24, 380: 22, 381: 16, 382: 19, 383: 28, 384: 17, 385: 12, 386: 16, 387: 24, 388: 28, 389: 25, 390: 21, 391: 22, 392: 13, 393: 26, 394: 15, 395: 13, 396: 27, 397: 26, 398: 19, 399: 18, 400: 29, 401: 24, 402: 12, 403: 18, 404: 24, 405: 19, 406: 22, 407: 17, 408: 18, 409: 32, 410: 15, 411: 14, 412: 15, 413: 23, 414: 14, 415: 14, 416: 14, 417: 30, 418: 15, 419: 16, 420: 10, 421: 15, 422: 16, 423: 14, 424: 13, 425: 11, 426: 23, 427: 20, 428: 6, 429: 18, 430: 16, 431: 11, 432: 18, 433: 14, 434: 22, 435: 20, 436: 16, 437: 17, 438: 26, 439: 18, 440: 8, 441: 15, 442: 17, 443: 20, 444: 20, 445: 17, 446: 17, 447: 11, 448: 22, 449: 10, 450: 12, 451: 15, 452: 11, 453: 10, 454: 18, 455: 15, 456: 27, 457: 18, 458: 18, 459: 26, 460: 14, 461: 20, 462: 7, 463: 25, 464: 15, 465: 6, 466: 14, 467: 22, 468: 11, 469: 15, 470: 14, 471: 17, 472: 12, 473: 22, 474: 12, 475: 22, 476: 5, 477: 19, 478: 12, 479: 19, 480: 15, 481: 14, 482: 17, 483: 15, 484: 18, 485: 20, 486: 13, 487: 16, 488: 14, 489: 13, 490: 13, 491: 11, 492: 13, 493: 12, 494: 11, 495: 16, 496: 12, 497: 18, 498: 16, 499: 17, 500: 11, 501: 7, 502: 14, 503: 14, 504: 15, 505: 14, 506: 17, 507: 14, 508: 13, 509: 9, 510: 14, 511: 12, 512: 14, 513: 16, 514: 12, 515: 10, 516: 11, 517: 16, 518: 8, 519: 15, 520: 12, 521: 16, 522: 15, 523: 11, 524: 10, 525: 13, 526: 18, 527: 14, 528: 10, 529: 7, 530: 13, 531: 7, 532: 9, 533: 7, 534: 19, 535: 11, 536: 17, 537: 18, 538: 9, 539: 13, 540: 5, 541: 8, 542: 15, 543: 15, 544: 11, 545: 10, 546: 8, 547: 12, 548: 10, 549: 10, 550: 12, 551: 9, 552: 5, 553: 9, 554: 17, 555: 10, 556: 13, 557: 10, 558: 10, 559: 8, 560: 11, 561: 7, 562: 15, 563: 5, 564: 6, 565: 11, 566: 5, 567: 20, 568: 6, 569: 6, 570: 13, 571: 6, 572: 13, 573: 7, 574: 11, 575: 14, 576: 12, 577: 16, 578: 10, 579: 5, 580: 11, 581: 11, 582: 12, 583: 12, 584: 8, 585: 8, 586: 9, 587: 10, 588: 10, 589: 6, 590: 14, 591: 14, 592: 14, 593: 8, 594: 8, 595: 9, 596: 13, 597: 15, 598: 5, 599: 8, 600: 13, 601: 3, 602: 10, 603: 13, 604: 9, 605: 4, 606: 6, 607: 7, 608: 6, 609: 5, 610: 5, 611: 9, 612: 7, 613: 8, 614: 11, 615: 6, 616: 15, 617: 9, 618: 14, 619: 11, 620: 4, 621: 6, 622: 6, 623: 16, 624: 13, 625: 8, 626: 5, 627: 8, 628: 8, 629: 9, 630: 4, 631: 5, 632: 10, 633: 8, 634: 7, 635: 3, 636: 8, 637: 14, 638: 3, 639: 6, 640: 7, 641: 8, 642: 4, 643: 1, 644: 8, 645: 4, 646: 5, 647: 6, 648: 4, 649: 8, 650: 5, 651: 7, 652: 8, 653: 8, 654: 12, 655: 8, 656: 9, 657: 8, 658: 7, 659: 5, 660: 3, 661: 3, 662: 8, 663: 1, 664: 4, 665: 9, 666: 3, 667: 7, 668: 7, 669: 9, 670: 4, 671: 5, 672: 9, 673: 9, 674: 6, 675: 10, 676: 12, 677: 7, 678: 7, 679: 5, 680: 9, 681: 11, 682: 8, 683: 4, 684: 8, 685: 4, 686: 6, 687: 4, 688: 12, 689: 7, 690: 9, 691: 8, 692: 5, 693: 6, 694: 12, 695: 5, 696: 2, 697: 6, 698: 6, 699: 10, 700: 2, 701: 6, 702: 3, 703: 4, 704: 7, 705: 11, 706: 1, 707: 6, 708: 11, 709: 4, 710: 5, 711: 6, 712: 6, 713: 4, 714: 7, 715: 3, 716: 9, 717: 4, 718: 7, 719: 4, 720: 1, 721: 6, 722: 3, 723: 4, 724: 5, 725: 3, 726: 7, 727: 9, 728: 2, 729: 13, 730: 8, 731: 3, 732: 5, 733: 5, 734: 1, 735: 2, 736: 8, 737: 6, 738: 5, 739: 11, 740: 9, 741: 5, 742: 8, 743: 4, 744: 7, 745: 3, 746: 10, 747: 7, 748: 5, 749: 6, 750: 2, 751: 7, 752: 3, 753: 9, 754: 7, 755: 3, 756: 6, 757: 7, 758: 3, 759: 4, 760: 7, 761: 9, 762: 5, 763: 9, 764: 4, 765: 4, 766: 4, 767: 3, 768: 6, 769: 5, 770: 5, 771: 7, 772: 4, 773: 4, 774: 3, 776: 3, 777: 3, 778: 5, 780: 3, 781: 4, 782: 1, 783: 5, 784: 5, 785: 6, 786: 2, 787: 5, 788: 4, 789: 7, 790: 4, 791: 3, 792: 3, 793: 1, 794: 7, 795: 4, 796: 6, 797: 3, 798: 1, 799: 3, 800: 3, 801: 8, 802: 2, 803: 1, 804: 2, 805: 4, 806: 5, 807: 3, 808: 3, 809: 3, 810: 2, 811: 2, 812: 2, 813: 5, 814: 1, 815: 6, 816: 3, 817: 3, 818: 7, 819: 5, 820: 4, 821: 3, 822: 2, 823: 6, 824: 5, 825: 6, 826: 1, 827: 4, 828: 1, 829: 2, 830: 3, 831: 4, 832: 2, 833: 1, 834: 3, 835: 7, 837: 8, 838: 11, 839: 2, 841: 1, 842: 5, 843: 2, 844: 2, 845: 1, 846: 4, 847: 1, 848: 6, 849: 4, 850: 3, 851: 6, 852: 4, 853: 5, 854: 4, 855: 4, 858: 3, 859: 3, 860: 1, 861: 2, 862: 4, 863: 7, 864: 3, 865: 5, 866: 4, 868: 2, 869: 5, 872: 4, 873: 2, 874: 3, 875: 6, 876: 2, 877: 3, 879: 1, 880: 3, 881: 6, 882: 5, 883: 8, 884: 2, 885: 1, 886: 4, 887: 4, 888: 4, 889: 2, 890: 6, 891: 3, 892: 4, 893: 4, 895: 6, 896: 4, 897: 5, 898: 1, 899: 2, 901: 5, 902: 4, 903: 2, 904: 2, 905: 3, 906: 2, 907: 5, 908: 3, 909: 1, 910: 4, 911: 3, 912: 2, 913: 6, 914: 1, 916: 4, 917: 3, 918: 5, 919: 8, 921: 2, 922: 4, 923: 4, 924: 1, 925: 3, 926: 4, 927: 1, 928: 2, 929: 1, 930: 6, 931: 6, 933: 2, 934: 1, 935: 2, 936: 8, 937: 2, 938: 1, 939: 3, 940: 1, 941: 1, 942: 2, 943: 2, 944: 4, 945: 2, 946: 4, 948: 1, 949: 9, 950: 2, 952: 2, 953: 2, 954: 3, 956: 2, 957: 1, 958: 1, 959: 1, 960: 1, 965: 1, 966: 2, 967: 1, 968: 2, 969: 3, 970: 1, 971: 2, 972: 1, 973: 4, 974: 2, 975: 1, 976: 1, 977: 2, 978: 2, 980: 2, 982: 1, 983: 2, 984: 1, 985: 1, 986: 1, 987: 1, 988: 1, 989: 5, 990: 2, 991: 1, 992: 4, 993: 3, 994: 1, 995: 1, 996: 3, 998: 1, 999: 8, 1000: 7, 1001: 9, 1003: 1, 1004: 3, 1005: 1, 1006: 1, 1008: 2, 1009: 1, 1011: 6, 1012: 2, 1014: 2, 1016: 3, 1017: 1, 1018: 2, 1019: 1, 1021: 1, 1022: 1, 1023: 1, 1024: 4, 1025: 1, 1026: 2, 1028: 1, 1029: 2, 1030: 3, 1031: 1, 1033: 3, 1034: 2, 1036: 1, 1037: 1, 1039: 2, 1041: 1, 1043: 3, 1045: 1, 1046: 4, 1048: 1, 1049: 3, 1050: 1, 1051: 3, 1053: 6, 1054: 1, 1055: 2, 1056: 1, 1057: 4, 1058: 2, 1059: 1, 1060: 2, 1063: 2, 1066: 1, 1067: 1, 1069: 2, 1070: 6, 1072: 4, 1074: 1, 1077: 2, 1079: 1, 1080: 4, 1081: 1, 1082: 1, 1083: 3, 1085: 3, 1086: 1, 1089: 2, 1090: 5, 1091: 2, 1092: 4, 1093: 2, 1094: 1, 1095: 3, 1096: 1, 1097: 2, 1098: 3, 1102: 1, 1103: 3, 1104: 1, 1107: 3, 1108: 1, 1109: 1, 1113: 3, 1114: 1, 1115: 1, 1116: 1, 1117: 1, 1119: 5, 1121: 1, 1122: 2, 1124: 1, 1125: 2, 1128: 3, 1129: 1, 1131: 2, 1132: 1, 1133: 3, 1134: 6, 1135: 3, 1137: 1, 1139: 2, 1140: 1, 1141: 1, 1142: 2, 1143: 4, 1144: 3, 1145: 1, 1146: 1, 1147: 1, 1148: 1, 1150: 4, 1153: 1, 1157: 2, 1160: 2, 1161: 2, 1164: 1, 1165: 1, 1166: 4, 1167: 2, 1168: 1, 1176: 1, 1177: 2, 1178: 1, 1183: 1, 1185: 1, 1186: 2, 1187: 1, 1188: 1, 1189: 1, 1191: 2, 1197: 1, 1200: 2, 1201: 1, 1202: 1, 1203: 3, 1204: 1, 1206: 2, 1207: 2, 1208: 2, 1209: 1, 1210: 1, 1211: 1, 1214: 4, 1215: 1, 1216: 2, 1217: 2, 1218: 2, 1221: 1, 1222: 2, 1223: 1, 1226: 1, 1228: 1, 1229: 1, 1231: 1, 1232: 2, 1233: 1, 1234: 1, 1236: 1, 1237: 2, 1238: 1, 1242: 2, 1243: 1, 1244: 1, 1245: 3, 1246: 2, 1247: 1, 1248: 3, 1249: 1, 1251: 1, 1252: 3, 1257: 1, 1258: 1, 1259: 1, 1262: 2, 1264: 1, 1267: 1, 1268: 1, 1270: 1, 1271: 1, 1272: 1, 1274: 1, 1278: 2, 1281: 1, 1282: 1, 1283: 1, 1285: 4, 1289: 2, 1292: 1, 1294: 1, 1295: 1, 1297: 1, 1299: 4, 1300: 1, 1302: 4, 1304: 1, 1306: 1, 1310: 1, 1312: 2, 1313: 2, 1314: 3, 1316: 1, 1317: 1, 1318: 1, 1321: 1, 1322: 1, 1323: 2, 1327: 1, 1330: 1, 1334: 5, 1338: 1, 1339: 1, 1340: 1, 1342: 3, 1343: 1, 1344: 1, 1347: 3, 1349: 2, 1352: 2, 1354: 2, 1355: 2, 1358: 1, 1359: 1, 1360: 1, 1361: 3, 1363: 1, 1364: 2, 1365: 3, 1368: 2, 1373: 1, 1374: 2, 1375: 1, 1377: 1, 1380: 2, 1383: 1, 1384: 1, 1386: 1, 1388: 2, 1389: 2, 1390: 1, 1397: 1, 1400: 2, 1402: 1, 1406: 2, 1407: 1, 1408: 3, 1410: 1, 1411: 2, 1412: 2, 1414: 1, 1419: 3, 1420: 4, 1421: 2, 1422: 4, 1423: 2, 1424: 1, 1425: 1, 1428: 2, 1429: 1, 1430: 1, 1431: 2, 1434: 3, 1438: 1, 1448: 1, 1452: 1, 1453: 3, 1456: 1, 1458: 1, 1459: 1, 1461: 1, 1462: 1, 1464: 4, 1465: 1, 1467: 1, 1469: 1, 1470: 1, 1472: 1, 1474: 2, 1480: 1, 1481: 1, 1491: 2, 1495: 1, 1501: 1, 1502: 2, 1503: 3, 1506: 2, 1507: 2, 1515: 4, 1518: 1, 1526: 1, 1529: 1, 1531: 2, 1541: 3, 1546: 4, 1549: 2, 1553: 1, 1557: 1, 1560: 1, 1568: 1, 1570: 1, 1571: 1, 1576: 3, 1577: 1, 1579: 2, 1583: 1, 1585: 3, 1587: 1, 1591: 2, 1592: 1, 1594: 3, 1595: 1, 1605: 1, 1608: 1, 1611: 1, 1620: 1, 1623: 2, 1627: 1, 1629: 2, 1630: 2, 1646: 1, 1649: 3, 1650: 1, 1656: 2, 1658: 2, 1659: 1, 1663: 2, 1668: 2, 1670: 2, 1683: 3, 1684: 1, 1687: 1, 1688: 2, 1689: 1, 1694: 1, 1701: 2, 1704: 1, 1708: 1, 1711: 1, 1716: 1, 1718: 1, 1719: 1, 1721: 1, 1726: 1, 1729: 1, 1730: 1, 1731: 1, 1740: 3, 1741: 3, 1742: 4, 1744: 1, 1750: 1, 1755: 1, 1761: 1, 1762: 1, 1763: 1, 1764: 1, 1767: 2, 1771: 2, 1781: 1, 1782: 2, 1787: 1, 1794: 2, 1808: 1, 1809: 2, 1811: 2, 1816: 1, 1818: 1, 1819: 1, 1820: 1, 1821: 1, 1823: 1, 1824: 1, 1836: 1, 1840: 1, 1842: 2, 1848: 2, 1849: 1, 1861: 1, 1864: 1, 1865: 1, 1872: 1, 1874: 1, 1877: 2, 1881: 1, 1884: 1, 1888: 1, 1889: 1, 1903: 3, 1905: 1, 1912: 2, 1913: 1, 1926: 1, 1927: 1, 1935: 1, 1936: 2, 1937: 1, 1941: 2, 1942: 1, 1945: 2, 1954: 1, 1958: 2, 1959: 1, 1962: 1, 1965: 3, 1966: 1, 1971: 2, 1979: 1, 1985: 2, 1991: 2, 2002: 1, 2004: 1, 2010: 1, 2013: 1, 2014: 1, 2017: 3, 2018: 1, 2019: 1, 2026: 2, 2029: 2, 2037: 1, 2040: 2, 2043: 1, 2044: 1, 2045: 1, 2054: 1, 2065: 1, 2074: 2, 2095: 3, 2097: 5, 2102: 1, 2107: 2, 2111: 1, 2114: 1, 2115: 1, 2123: 2, 2129: 2, 2134: 2, 2137: 1, 2140: 1, 2141: 1, 2142: 1, 2148: 1, 2149: 2, 2160: 1, 2175: 2, 2177: 1, 2196: 1, 2197: 1, 2200: 2, 2206: 1, 2212: 1, 2219: 2, 2234: 1, 2236: 1, 2237: 1, 2244: 1, 2249: 1, 2252: 1, 2255: 4, 2257: 3, 2259: 2, 2262: 1, 2264: 1, 2272: 2, 2274: 1, 2276: 1, 2280: 2, 2285: 1, 2293: 2, 2298: 1, 2299: 1, 2301: 2, 2304: 1, 2319: 2, 2322: 1, 2326: 2, 2328: 2, 2345: 1, 2348: 2, 2349: 2, 2352: 1, 2365: 2, 2366: 1, 2372: 1, 2374: 3, 2391: 2, 2397: 1, 2404: 1, 2407: 2, 2423: 2, 2429: 3, 2431: 1, 2432: 1, 2433: 2, 2447: 2, 2458: 1, 2469: 2, 2482: 1, 2499: 2, 2538: 1, 2539: 1, 2543: 2, 2547: 1, 2565: 1, 2583: 4, 2584: 1, 2589: 1, 2631: 4, 2632: 1, 2636: 1, 2672: 2, 2675: 1, 2694: 1, 2709: 2, 2737: 2, 2781: 1, 2822: 1, 2850: 2, 2853: 2, 2867: 1, 2886: 2, 2903: 1, 2948: 2, 2984: 2, 3024: 2, 3043: 2, 3081: 1, 3113: 2, 3126: 2, 3162: 2, 3185: 2, 3217: 2, 3321: 2, 3322: 2, 3349: 2, 3408: 2, 3436: 2, 3461: 2, 3492: 2, 3526: 2, 4367: 1, 4863: 1, 5316: 1, 5366: 1, 5757: 2, 5838: 2, 5926: 1, 5941: 1, 5970: 1, 6029: 1, 6044: 1, 6142: 1, 9132: 1}, 115: {12: 10, 13: 2, 15: 6, 16: 8, 18: 59, 19: 2, 21: 56, 22: 6, 24: 43, 25: 1, 27: 23, 28: 6, 29: 1, 30: 36, 31: 4, 33: 26, 34: 4, 35: 3, 36: 24, 37: 8, 38: 2, 39: 13, 40: 14, 41: 4, 42: 12, 43: 9, 44: 4, 45: 12, 46: 1, 47: 5, 48: 13, 49: 18, 50: 1, 51: 10, 52: 12, 53: 6, 54: 10, 55: 11, 57: 1, 58: 3, 59: 5, 60: 10, 61: 10, 62: 2, 63: 3, 64: 4, 65: 3, 66: 6, 67: 5, 68: 4, 69: 2, 70: 8, 71: 4, 72: 8, 73: 5, 74: 3, 75: 4, 76: 3, 77: 4, 78: 4, 79: 1, 80: 1, 81: 4, 82: 9, 83: 2, 84: 2, 85: 2, 87: 3, 88: 7, 89: 3, 90: 4, 91: 5, 92: 6, 93: 2, 94: 5, 95: 4, 96: 7, 97: 4, 98: 3, 99: 5, 100: 7, 101: 4, 102: 1, 103: 4, 104: 6, 105: 4, 106: 5, 107: 4, 108: 3, 109: 1, 110: 5, 111: 3, 112: 7, 113: 3, 114: 4, 115: 3, 116: 2, 117: 1, 118: 1, 120: 2, 121: 2, 122: 1, 123: 2, 124: 4, 125: 2, 126: 2, 128: 1, 129: 6, 130: 3, 131: 2, 132: 10, 133: 4, 134: 1, 135: 2, 137: 1, 138: 1, 139: 2, 140: 3, 141: 1, 142: 8, 143: 1, 144: 5, 147: 1, 149: 3, 150: 4, 152: 2, 153: 5, 154: 2, 155: 1, 156: 2, 157: 1, 158: 1, 159: 3, 160: 3, 161: 1, 162: 4, 163: 4, 164: 2, 165: 2, 166: 4, 167: 1, 168: 4, 169: 4, 170: 3, 171: 1, 172: 1, 173: 2, 174: 2, 175: 2, 176: 2, 177: 1, 178: 1, 179: 1, 180: 1, 181: 1, 183: 1, 184: 1, 185: 1, 187: 4, 188: 3, 189: 1, 190: 1, 191: 3, 192: 7, 193: 4, 195: 1, 196: 1, 197: 1, 198: 1, 199: 2, 200: 1, 201: 4, 202: 1, 203: 5, 204: 2, 205: 2, 207: 2, 208: 2, 211: 3, 212: 1, 214: 4, 215: 2, 216: 4, 218: 2, 219: 1, 220: 1, 222: 1, 223: 1, 224: 1, 225: 4, 226: 1, 227: 1, 229: 2, 231: 2, 232: 1, 234: 2, 235: 1, 237: 3, 238: 1, 239: 1, 240: 3, 241: 1, 243: 1, 245: 1, 247: 2, 249: 4, 251: 2, 253: 2, 254: 1, 255: 1, 256: 2, 257: 1, 258: 1, 261: 1, 262: 2, 263: 1, 264: 1, 266: 1, 268: 2, 270: 2, 271: 1, 272: 1, 273: 2, 274: 2, 277: 2, 278: 2, 282: 1, 285: 1, 286: 1, 287: 2, 288: 2, 289: 1, 290: 2, 291: 1, 292: 3, 293: 1, 299: 2, 300: 3, 301: 3, 302: 1, 303: 3, 306: 1, 309: 1, 310: 3, 311: 1, 312: 1, 314: 1, 316: 2, 317: 1, 319: 1, 320: 1, 324: 1, 325: 1, 326: 4, 328: 1, 330: 1, 334: 2, 339: 2, 340: 1, 343: 1, 344: 1, 345: 1, 350: 1, 352: 2, 354: 1, 363: 1, 365: 1, 367: 2, 369: 1, 370: 1, 371: 1, 387: 1, 388: 2, 389: 1, 395: 1, 396: 1, 398: 1, 405: 1, 406: 1, 409: 1, 411: 3, 413: 1, 416: 1, 419: 1, 420: 1, 422: 1, 437: 1, 438: 3, 440: 1, 443: 1, 446: 1, 447: 1, 451: 1, 454: 1, 457: 1, 470: 1, 483: 3, 488: 1, 491: 1, 496: 1, 511: 1, 520: 1, 530: 1, 532: 1, 535: 1, 536: 1, 544: 1, 554: 1, 563: 1, 565: 1, 589: 1, 590: 1, 592: 1, 595: 1, 600: 1, 602: 1, 607: 2, 629: 1, 643: 1, 650: 1, 655: 1, 668: 1, 695: 2, 724: 1, 743: 1, 745: 1, 761: 1, 772: 1, 775: 2, 796: 1, 797: 1, 801: 1, 805: 1, 814: 1, 816: 1, 819: 1, 838: 1, 844: 1, 847: 1, 850: 1, 852: 1, 865: 1, 875: 1, 897: 1, 901: 1, 905: 1, 907: 1, 920: 1, 921: 1, 955: 1, 962: 1, 995: 1, 1008: 1, 1042: 1, 1054: 1, 1061: 1, 1084: 1, 1089: 1, 1090: 1, 1110: 1, 1118: 1, 1194: 1, 1204: 2, 1223: 1, 1259: 1, 1265: 2, 1297: 1, 1324: 1, 1334: 1, 1364: 1, 1374: 1, 1385: 1, 1410: 2, 1480: 1, 1486: 1, 1496: 1, 1516: 1, 1552: 1, 1565: 1, 1613: 2, 1670: 1, 1681: 1, 1706: 1, 1766: 1, 1876: 1, 2020: 1, 2087: 1, 2157: 1, 2205: 1, 2263: 1, 2333: 2, 2369: 1, 2393: 1, 2432: 1, 2463: 1, 2468: 1, 2517: 1, 2584: 1, 2601: 1, 2637: 1, 2661: 1, 2749: 1, 2785: 1, 2885: 1, 2921: 1, 3310: 1, 3346: 1, 3471: 1, 3507: 1, 3741: 1, 3777: 1, 3916: 1, 3952: 1, 4082: 1, 4118: 1, 4242: 1, 4278: 1, 4451: 1, 4487: 1, 4614: 1, 4650: 1, 4768: 1, 4804: 1, 4904: 1, 4940: 1, 5035: 1, 5071: 1, 5252: 1, 5288: 1, 5421: 1, 5457: 1, 5585: 1, 5621: 1, 5871: 1, 5907: 1, 6108: 1, 6144: 1, 6202: 1, 6238: 1, 6365: 1, 6401: 1, 6455: 1, 6491: 1, 6815: 1, 6851: 1, 7152: 1, 7188: 1, 7313: 1, 7349: 1, 7821: 1, 7857: 1, 8106: 1, 8142: 1, 8307: 1, 8343: 1, 8448: 1, 8484: 1, 8820: 1, 8856: 1, 8955: 1, 8991: 1, 9362: 1, 9398: 1, 9499: 1, 9535: 1, 9749: 1, 9785: 1, 9901: 1, 9937: 1, 10100: 1, 10136: 1, 10503: 1, 10539: 1, 10660: 1, 10696: 1, 10845: 1, 10881: 1, 11030: 1, 11066: 1, 11188: 1, 11224: 1, 11333: 1, 11369: 1, 11664: 1, 11700: 1, 11815: 1, 11851: 1, 11964: 1, 12000: 1, 12076: 1, 12112: 1, 12367: 1, 12403: 1, 12530: 1, 12566: 1, 12759: 1, 12795: 1, 12943: 1, 12979: 1, 13164: 1, 13200: 1, 13333: 1, 13369: 1, 13487: 1, 13523: 1, 13701: 1, 13737: 1, 13870: 1, 13906: 1, 14009: 1, 14045: 1, 14232: 1, 14268: 1, 14379: 1, 14415: 1, 14501: 1, 14537: 1, 14641: 1, 14677: 1, 14772: 1, 14808: 1, 14966: 1, 15002: 1, 15567: 1, 15603: 1, 15918: 1, 15954: 1, 16118: 1, 16154: 1, 16351: 1, 16387: 1, 17140: 1, 17176: 1, 17539: 1, 17575: 1, 17729: 1, 17765: 1, 17937: 1, 17973: 1, 18755: 1, 18791: 1, 19622: 1, 19658: 1, 19828: 1, 19864: 1, 20281: 1, 20317: 1, 20462: 1, 20498: 1, 20619: 1, 20655: 1, 20984: 1, 21020: 1, 21422: 1, 21458: 1, 21584: 1, 21620: 1, 21843: 1, 21879: 1, 21960: 1, 21996: 1, 22093: 1, 22129: 1, 22247: 1, 22283: 1, 22386: 1, 22422: 1, 22543: 1, 22579: 1, 22678: 1, 22714: 1, 22820: 1, 22856: 1, 22990: 1, 23026: 1, 23104: 1, 23140: 1, 23282: 1, 23318: 1, 23445: 1, 23481: 1, 23704: 1, 23740: 1, 23888: 1, 23924: 1, 24181: 1, 24217: 1, 24374: 1, 24410: 1, 24537: 1, 24573: 1, 24760: 1, 24796: 1, 24901: 1, 24937: 1, 25103: 1, 25139: 1, 25293: 1, 25329: 1, 25432: 1, 25468: 1, 25586: 1, 25622: 1, 25725: 1, 25761: 1, 25864: 1, 25900: 1, 26003: 1, 26039: 1, 26172: 1, 26208: 1, 26350: 1, 26386: 1, 26507: 1, 26543: 1, 26664: 1, 26700: 1, 26849: 1, 26885: 1, 26948: 1, 26984: 1, 27421: 1, 27457: 1, 27913: 1, 27949: 1, 28051: 1, 28087: 1, 28663: 1, 28699: 1, 28808: 1, 28844: 1, 29205: 1, 29241: 1, 29344: 1, 29380: 1, 29465: 1, 29501: 1, 29595: 1, 29631: 1, 29826: 1, 29862: 1, 29951: 1, 29987: 1, 30076: 1, 30112: 1, 30201: 1, 30237: 1, 30326: 1, 30362: 1, 30451: 1, 30487: 1, 30576: 1, 30612: 1, 30701: 1, 30737: 1, 30826: 1, 30862: 1, 30966: 1, 31002: 1, 31280: 1, 31316: 1, 31449: 1, 31485: 1, 31618: 1, 31654: 1, 31745: 1, 31781: 1, 31998: 1, 32034: 1, 32128: 1, 32164: 1, 32291: 1, 32327: 1, 32454: 1, 32490: 1, 32657: 1, 32693: 1, 32814: 1, 32850: 1, 32980: 1, 33016: 1, 33110: 1, 33146: 1, 33270: 1, 33306: 1, 33391: 1, 33427: 1, 33681: 1, 33717: 1, 33932: 1, 33968: 1, 34071: 1, 34107: 1, 34179: 1, 34215: 1, 34300: 1, 34336: 1, 34502: 1, 34538: 1}, 116: {0: 33605, 1: 23351, 2: 18247, 3: 13518, 4: 10904, 5: 9297, 6: 7177, 7: 5456, 8: 4371, 9: 3370, 10: 2787, 11: 2157, 12: 1815, 13: 1475, 14: 1268, 15: 933, 16: 824, 17: 673, 18: 592, 19: 472, 20: 355, 21: 315, 22: 290, 23: 254, 24: 198, 25: 185, 26: 185, 27: 155, 28: 108, 29: 117, 30: 109, 31: 95, 32: 76, 33: 51, 34: 67, 35: 60, 36: 43, 37: 49, 38: 40, 39: 43, 40: 37, 41: 27, 42: 35, 43: 29, 44: 48, 45: 354, 46: 26, 47: 22, 48: 11, 49: 20, 50: 31, 51: 16, 52: 9, 53: 23, 54: 19, 55: 5, 56: 3, 57: 8, 58: 5, 59: 6, 60: 7, 61: 4, 62: 3, 63: 2, 64: 2, 65: 8, 66: 1, 67: 1, 68: 1, 72: 1, 77: 3, 78: 1, 79: 1, 91: 1, 92: 1, 95: 1, 96: 2, 97: 1, 135: 2, 137: 1, 144: 2}, 119: {3: 5, 9: 3, 10: 2, 13: 8, 16: 6, 19: 5, 20: 6, 22: 5, 24: 2, 25: 3, 27: 1, 28: 1, 31: 1, 34: 1, 37: 8, 40: 3, 41: 3, 44: 1, 45: 1, 46: 2, 47: 1, 48: 1, 49: 1, 50: 2, 51: 1, 52: 8, 54: 2, 56: 2, 58: 1, 60: 3, 66: 1, 68: 2, 73: 1, 77: 2, 78: 1, 80: 4, 82: 4, 90: 1, 94: 2, 97: 2, 99: 1, 101: 1, 107: 1, 114: 2, 118: 3, 121: 2, 123: 1, 133: 1, 137: 4, 138: 1, 142: 3, 144: 2, 150: 1, 168: 2, 179: 1, 182: 2, 184: 4, 192: 2, 198: 1, 199: 1, 203: 1, 221: 1, 222: 4, 223: 1, 238: 2, 268: 1, 293: 1, 299: 2, 311: 1, 320: 3, 327: 1, 363: 1, 411: 1, 413: 1, 447: 1, 517: 1, 534: 1, 909: 1, 1002: 1, 1216: 1, 1517: 1, 1789: 1}, 120: {4: 3, 5: 2, 8: 1, 9: 4, 14: 14, 15: 2, 16: 2, 17: 10, 18: 2, 19: 13, 20: 8, 21: 16, 22: 15, 23: 14, 24: 85, 25: 24, 26: 17, 27: 140, 28: 37, 29: 29, 30: 196, 31: 35, 32: 35, 33: 187, 34: 66, 35: 44, 36: 193, 37: 86, 38: 37, 39: 174, 40: 121, 41: 48, 42: 182, 43: 98, 44: 49, 45: 137, 46: 82, 47: 58, 48: 160, 49: 89, 50: 48, 51: 145, 52: 126, 53: 35, 54: 125, 55: 95, 56: 70, 57: 93, 58: 96, 59: 68, 60: 109, 61: 82, 62: 58, 63: 96, 64: 71, 65: 53, 66: 89, 67: 75, 68: 74, 69: 76, 70: 58, 71: 61, 72: 89, 73: 78, 74: 67, 75: 72, 76: 74, 77: 50, 78: 49, 79: 41, 80: 68, 81: 57, 82: 77, 83: 45, 84: 62, 85: 50, 86: 46, 87: 47, 88: 47, 89: 48, 90: 42, 91: 41, 92: 49, 93: 50, 94: 43, 95: 47, 96: 39, 97: 28, 98: 42, 99: 45, 100: 39, 101: 45, 102: 48, 103: 37, 104: 46, 105: 42, 106: 35, 107: 29, 108: 47, 109: 28, 110: 29, 111: 31, 112: 38, 113: 41, 114: 38, 115: 32, 116: 32, 117: 37, 118: 35, 119: 40, 120: 32, 121: 43, 122: 22, 123: 33, 124: 22, 125: 19, 126: 21, 127: 31, 128: 21, 129: 29, 130: 23, 131: 28, 132: 27, 133: 26, 134: 18, 135: 27, 136: 17, 137: 20, 138: 21, 139: 22, 140: 32, 141: 23, 142: 11, 143: 18, 144: 22, 145: 22, 146: 10, 147: 20, 148: 23, 149: 20, 150: 26, 151: 17, 152: 25, 153: 12, 154: 18, 155: 17, 156: 17, 157: 11, 158: 22, 159: 21, 160: 13, 161: 18, 162: 18, 163: 16, 164: 10, 165: 21, 166: 13, 167: 15, 168: 12, 169: 11, 170: 9, 171: 16, 172: 20, 173: 14, 174: 11, 175: 13, 176: 14, 177: 15, 178: 11, 179: 11, 180: 11, 181: 10, 182: 14, 183: 19, 184: 15, 185: 9, 186: 10, 187: 10, 188: 10, 189: 11, 190: 10, 191: 10, 192: 22, 193: 8, 194: 11, 195: 8, 196: 13, 197: 6, 198: 11, 199: 11, 200: 10, 201: 12, 202: 8, 203: 8, 204: 14, 205: 9, 206: 11, 207: 10, 208: 8, 209: 12, 210: 6, 211: 7, 212: 13, 213: 7, 214: 11, 215: 11, 216: 14, 217: 3, 218: 9, 219: 11, 220: 3, 221: 8, 222: 3, 223: 4, 224: 10, 225: 14, 226: 7, 227: 10, 228: 6, 229: 5, 230: 9, 231: 16, 232: 10, 233: 8, 234: 9, 235: 11, 236: 7, 237: 2, 238: 3, 239: 9, 240: 7, 241: 2, 242: 3, 243: 7, 244: 11, 245: 7, 246: 10, 247: 14, 248: 1, 249: 1, 250: 7, 251: 8, 252: 2, 253: 8, 254: 5, 255: 5, 256: 6, 257: 4, 258: 12, 259: 2, 260: 6, 261: 7, 262: 5, 263: 7, 264: 5, 265: 5, 266: 3, 267: 5, 268: 1, 269: 5, 270: 2, 271: 2, 272: 6, 273: 3, 274: 6, 275: 4, 276: 5, 277: 4, 278: 5, 279: 5, 280: 2, 281: 4, 282: 1, 283: 4, 284: 8, 285: 4, 286: 3, 287: 5, 288: 6, 289: 6, 290: 4, 291: 4, 292: 3, 293: 4, 294: 4, 295: 1, 296: 3, 297: 4, 298: 5, 299: 4, 300: 5, 301: 6, 302: 3, 303: 3, 304: 3, 305: 3, 306: 6, 307: 8, 308: 1, 309: 4, 310: 2, 311: 6, 312: 1, 314: 3, 315: 2, 316: 2, 317: 3, 318: 3, 319: 6, 320: 2, 321: 3, 322: 2, 323: 2, 324: 4, 325: 2, 326: 6, 327: 4, 328: 1, 329: 2, 330: 3, 331: 3, 332: 2, 333: 2, 335: 7, 337: 1, 338: 2, 339: 3, 341: 5, 342: 1, 344: 2, 345: 2, 346: 3, 347: 1, 348: 6, 349: 2, 350: 4, 351: 3, 352: 2, 353: 2, 354: 3, 355: 3, 356: 6, 358: 1, 359: 2, 360: 1, 361: 3, 363: 3, 365: 1, 366: 2, 367: 4, 368: 2, 369: 2, 370: 3, 371: 2, 372: 3, 373: 1, 374: 3, 375: 1, 376: 1, 377: 2, 378: 3, 379: 1, 380: 1, 382: 4, 383: 1, 384: 1, 385: 1, 387: 2, 388: 4, 389: 1, 390: 1, 391: 2, 392: 4, 393: 1, 395: 2, 396: 2, 397: 2, 398: 3, 399: 2, 403: 2, 404: 1, 406: 2, 407: 1, 408: 3, 409: 1, 412: 1, 414: 1, 416: 1, 418: 6, 419: 1, 420: 2, 421: 1, 422: 1, 424: 1, 425: 1, 426: 1, 428: 2, 430: 3, 433: 1, 439: 2, 440: 1, 441: 3, 445: 1, 446: 1, 447: 2, 452: 1, 453: 1, 454: 3, 456: 1, 457: 1, 458: 1, 459: 1, 461: 2, 462: 4, 463: 1, 464: 2, 466: 1, 469: 2, 470: 2, 476: 1, 477: 1, 479: 1, 480: 2, 485: 2, 486: 1, 489: 2, 490: 1, 491: 2, 492: 1, 493: 1, 494: 3, 497: 2, 499: 1, 502: 3, 506: 1, 508: 2, 509: 1, 514: 1, 515: 2, 516: 1, 517: 2, 519: 1, 522: 1, 523: 1, 524: 1, 525: 2, 526: 2, 528: 2, 529: 1, 530: 2, 531: 1, 534: 1, 536: 1, 537: 2, 539: 1, 541: 2, 543: 1, 544: 1, 548: 1, 550: 1, 554: 2, 556: 2, 558: 2, 559: 1, 562: 1, 563: 1, 565: 4, 568: 2, 572: 1, 574: 2, 576: 1, 577: 1, 582: 1, 583: 1, 586: 1, 587: 1, 589: 1, 591: 1, 593: 2, 594: 1, 596: 1, 600: 2, 601: 1, 603: 1, 615: 1, 617: 1, 620: 1, 622: 1, 625: 3, 630: 2, 638: 1, 646: 1, 650: 4, 653: 1, 656: 1, 657: 1, 658: 1, 663: 1, 670: 1, 676: 3, 678: 1, 681: 2, 682: 1, 686: 1, 694: 1, 695: 2, 700: 2, 706: 1, 708: 1, 709: 1, 713: 1, 718: 1, 725: 1, 727: 1, 733: 1, 750: 1, 751: 1, 769: 1, 774: 1, 778: 3, 782: 1, 785: 1, 793: 1, 803: 1, 824: 1, 826: 1, 857: 1, 863: 1, 869: 1, 873: 1, 895: 1, 968: 1, 1023: 1, 1024: 1, 1034: 1, 1057: 1, 1071: 1, 1136: 1, 1176: 1, 1194: 2, 1263: 1, 1298: 1, 1315: 1, 1384: 1, 1474: 1, 1920: 2, 2258: 1, 2307: 2, 3482: 2, 3565: 1}, 121: {4: 6, 7: 4, 8: 24, 9: 13, 10: 63, 11: 90, 12: 90, 13: 156, 14: 353, 15: 25, 16: 468, 17: 409, 18: 50, 19: 314, 20: 376, 21: 32, 22: 220, 23: 168, 24: 30, 25: 120, 26: 111, 27: 47, 28: 94, 29: 82, 30: 45, 31: 63, 32: 74, 33: 52, 34: 61, 35: 53, 36: 31, 37: 29, 38: 51, 39: 18, 40: 22, 41: 32, 42: 24, 43: 20, 44: 25, 45: 13, 46: 32, 47: 13, 48: 22, 49: 14, 50: 14, 51: 18, 52: 16, 53: 25, 54: 12, 55: 6, 56: 12, 57: 15, 58: 12, 59: 7, 60: 15, 61: 7, 62: 12, 63: 11, 64: 19, 65: 11, 66: 10, 67: 7, 68: 13, 69: 3, 70: 12, 71: 6, 72: 9, 73: 6, 74: 10, 75: 12, 76: 8, 77: 4, 78: 8, 79: 4, 80: 5, 81: 6, 82: 1, 83: 3, 84: 1, 85: 2, 86: 3, 87: 5, 88: 5, 89: 3, 90: 4, 91: 2, 92: 5, 93: 6, 94: 3, 95: 3, 96: 1, 97: 4, 98: 10, 100: 2, 101: 5, 102: 2, 103: 3, 104: 3, 106: 4, 107: 6, 108: 4, 110: 2, 111: 3, 112: 2, 113: 5, 114: 1, 115: 1, 116: 1, 117: 3, 118: 6, 119: 3, 121: 2, 122: 2, 123: 1, 124: 1, 125: 2, 126: 1, 127: 3, 128: 3, 130: 1, 131: 3, 132: 1, 133: 1, 134: 1, 135: 4, 137: 1, 138: 4, 140: 2, 141: 1, 143: 1, 144: 4, 147: 3, 153: 1, 154: 2, 156: 1, 157: 1, 158: 3, 159: 1, 160: 1, 167: 1, 168: 1, 169: 1, 174: 1, 176: 1, 177: 1, 180: 1, 188: 1, 192: 2, 193: 1, 197: 1, 198: 3, 199: 3, 200: 1, 214: 1, 215: 2, 216: 1, 220: 1, 221: 1, 222: 1, 223: 2, 225: 2, 226: 1, 227: 1, 228: 2, 229: 2, 233: 1, 235: 2, 237: 1, 245: 1, 248: 1, 251: 1, 255: 1, 256: 2, 260: 1, 262: 1, 265: 5, 269: 1, 270: 1, 271: 2, 278: 1, 294: 2, 299: 1, 300: 1, 301: 1, 308: 1, 309: 1, 312: 1, 317: 1, 320: 1, 323: 2, 330: 1, 332: 1, 337: 2, 342: 1, 367: 1, 371: 1, 374: 1, 377: 1, 392: 1, 393: 1, 412: 1, 423: 1, 426: 1, 427: 2, 471: 1, 479: 1, 484: 1, 502: 1, 568: 1, 608: 3, 683: 1, 852: 2, 1322: 1}, 122: {4: 5, 5: 21, 7: 3, 8: 1, 9: 31, 10: 1, 11: 35, 12: 8, 13: 3, 14: 78, 15: 17, 16: 25, 17: 126, 18: 47, 19: 13, 20: 38, 21: 43, 22: 31, 23: 68, 24: 45, 25: 24, 26: 63, 27: 83, 28: 47, 29: 48, 30: 42, 31: 35, 32: 23, 33: 44, 34: 27, 35: 29, 36: 25, 37: 23, 38: 32, 39: 32, 40: 22, 41: 24, 42: 16, 43: 21, 44: 29, 45: 29, 46: 18, 47: 20, 48: 15, 49: 14, 50: 15, 51: 21, 52: 16, 53: 27, 54: 14, 55: 11, 56: 15, 57: 16, 58: 24, 59: 11, 60: 13, 61: 16, 62: 9, 63: 9, 64: 12, 65: 7, 66: 13, 67: 6, 68: 9, 69: 5, 70: 16, 71: 11, 72: 8, 73: 7, 74: 10, 75: 7, 76: 13, 77: 14, 78: 7, 79: 14, 80: 7, 81: 3, 82: 11, 83: 3, 84: 7, 85: 6, 86: 11, 87: 9, 88: 5, 89: 12, 90: 8, 91: 8, 92: 7, 93: 5, 94: 5, 95: 6, 96: 9, 97: 11, 98: 5, 99: 11, 100: 2, 101: 8, 102: 6, 103: 9, 104: 7, 105: 7, 106: 7, 107: 11, 108: 7, 110: 6, 111: 9, 112: 2, 113: 2, 114: 4, 115: 2, 116: 6, 117: 3, 118: 3, 119: 4, 120: 2, 121: 2, 122: 1, 123: 1, 125: 3, 126: 2, 127: 10, 128: 4, 129: 9, 130: 1, 131: 5, 132: 5, 133: 1, 134: 4, 135: 5, 136: 2, 137: 3, 138: 1, 139: 2, 140: 1, 141: 3, 142: 2, 143: 5, 144: 3, 145: 2, 146: 5, 147: 1, 148: 1, 149: 2, 150: 2, 151: 4, 152: 3, 153: 1, 154: 1, 155: 2, 156: 3, 157: 2, 158: 2, 159: 1, 160: 1, 161: 2, 162: 2, 164: 1, 165: 2, 166: 2, 168: 3, 169: 1, 170: 5, 171: 1, 172: 1, 173: 4, 174: 2, 175: 3, 177: 1, 178: 3, 179: 1, 180: 3, 181: 2, 182: 3, 183: 1, 184: 3, 186: 1, 187: 4, 188: 2, 189: 1, 190: 2, 191: 5, 192: 1, 195: 2, 196: 2, 197: 1, 198: 1, 199: 4, 200: 2, 201: 3, 202: 1, 204: 3, 206: 1, 207: 1, 208: 2, 209: 1, 213: 1, 214: 1, 215: 1, 217: 4, 220: 1, 221: 1, 222: 3, 226: 1, 227: 4, 228: 1, 229: 2, 230: 2, 231: 2, 233: 1, 237: 1, 238: 1, 240: 1, 241: 4, 242: 1, 243: 3, 244: 2, 247: 2, 249: 2, 253: 1, 254: 1, 256: 2, 260: 1, 262: 1, 265: 1, 270: 1, 274: 1, 277: 1, 280: 1, 283: 1, 284: 1, 287: 1, 288: 1, 289: 2, 290: 3, 293: 1, 295: 1, 296: 1, 298: 2, 301: 1, 303: 1, 305: 1, 310: 2, 311: 2, 313: 2, 315: 1, 323: 3, 330: 1, 335: 1, 342: 1, 350: 1, 351: 1, 354: 1, 356: 1, 372: 1, 377: 1, 389: 1, 391: 1, 398: 1, 401: 1, 407: 1, 408: 1, 410: 1, 416: 2, 424: 2, 427: 3, 434: 1, 440: 1, 447: 1, 449: 1, 453: 1, 508: 2, 509: 1, 513: 1, 519: 1, 537: 2, 544: 1, 561: 1, 565: 1, 569: 1, 592: 1, 594: 1, 605: 1, 615: 1, 619: 1, 667: 1, 668: 1, 740: 1, 796: 1, 837: 1, 979: 1, 1475: 1}, 124: {0: 144976, 1: 77960, 2: 46846, 3: 29612, 4: 19727, 5: 13669, 6: 9345, 7: 6632, 8: 4704, 9: 3637, 10: 2733, 11: 2185, 12: 1703, 13: 1390, 14: 1033, 15: 798, 16: 601, 17: 513, 18: 415, 19: 375, 20: 275, 21: 232, 22: 184, 23: 221, 24: 142, 25: 131, 26: 156, 27: 97, 28: 153, 29: 78, 30: 68, 31: 55, 32: 54, 33: 49, 34: 28, 35: 20, 36: 8, 37: 3, 38: 1, 39: 2, 40: 2}, 125: {0: 1756, 1: 23836, 2: 18727, 3: 13584, 4: 9750, 5: 6922, 6: 5057, 7: 3637, 8: 2694, 9: 2075, 10: 1571, 11: 1215, 12: 885, 13: 739, 14: 511, 15: 409, 16: 346, 17: 267, 18: 250, 19: 200, 20: 164, 21: 155, 22: 122, 23: 106, 24: 113, 25: 57, 26: 48, 27: 42, 28: 46, 29: 33, 30: 38, 31: 34, 32: 34, 33: 33, 34: 23, 35: 12, 36: 6, 37: 5, 38: 1, 39: 1, 40: 1}, 126: {0: 26, 1: 202, 2: 270, 3: 283, 4: 168, 5: 113, 6: 91, 7: 70, 8: 50, 9: 35, 10: 34, 11: 20, 12: 13, 13: 23, 14: 18, 15: 13, 16: 12, 17: 1, 18: 3, 19: 11, 20: 3, 21: 3, 22: 5, 23: 6, 27: 1, 31: 1, 32: 1, 33: 2, 34: 2}, 130: {0: 478, 1: 6277, 2: 132}, 131: {0: 38649, 1: 107380, 2: 72078, 3: 19039, 4: 4768, 5: 1567, 6: 838, 7: 244, 8: 108, 9: 28, 10: 34, 11: 22, 12: 4, 13: 6, 14: 2, 15: 4, 16: 5, 17: 6, 18: 2, 19: 3, 20: 1, 24: 1, 38: 1, 256: 3178, 257: 3714, 258: 1762, 259: 636, 260: 171, 261: 36, 262: 34, 263: 22, 264: 3, 267: 3, 270: 1, 512: 1307, 513: 1031, 514: 455, 515: 121, 516: 41, 517: 8, 518: 2, 519: 1, 521: 1, 768: 491, 769: 380, 770: 124, 771: 15, 772: 6, 773: 2, 1024: 212, 1025: 112, 1026: 51, 1027: 9, 1028: 1, 1280: 69, 1281: 37, 1282: 1, 1283: 1, 1536: 36, 1537: 6, 1538: 2, 1792: 199, 1793: 1, 1794: 1, 2048: 176, 2049: 26, 2050: 1, 2304: 4, 2305: 10, 2560: 7, 2561: 1, 2816: 1, 3072: 1, 3584: 1, 3840: 1, 5888: 1}, 132: {0: 49708, 1: 2862, 2: 768, 3: 308, 4: 135, 5: 66, 6: 35, 7: 19, 8: 14, 9: 9, 10: 12, 11: 3, 12: 4, 14: 1, 16: 2, 17: 1, 256: 120, 257: 34, 258: 11, 259: 3, 260: 4, 261: 6, 262: 1, 263: 1, 267: 1, 512: 25, 513: 10, 514: 4, 515: 3, 518: 1, 522: 2, 768: 34, 769: 3, 770: 2, 1024: 8, 1025: 7, 1026: 3, 1280: 1, 1536: 3, 1793: 2, 1794: 6, 2049: 1, 2051: 1, 131072: 74, 131073: 4, 131328: 2, 131329: 1, 196608: 22, 196609: 5, 196864: 4, 196865: 3, 262144: 1, 262145: 2, 262146: 1, 262147: 1, 327682: 2, 327937: 1, 590338: 2}, 133: {2: 4222, 3: 347}, 134: {0: 4190, 1: 134, 2: 34, 3: 27, 4: 8, 5: 1, 6: 1, 8: 5, 12: 2, 256: 17, 257: 2, 258: 1, 512: 2, 517: 1, 768: 2, 769: 1, 131072: 6}, 135: {0: 4380, 1: 1134, 2: 438, 3: 188, 4: 102, 5: 57, 6: 39, 7: 17, 8: 19, 9: 17, 10: 9, 11: 12, 12: 14, 13: 8, 14: 6, 15: 8, 16: 9, 17: 3, 18: 3, 19: 3, 20: 3}, 136: {0: 8820, 1: 3399, 2: 1154, 3: 439, 4: 280, 5: 124, 6: 58, 7: 29, 8: 31, 9: 29, 10: 26, 11: 23, 12: 34, 13: 17, 14: 13, 15: 11, 16: 3, 17: 5, 18: 4, 19: 2}, 137: {0: 1765, 1: 646, 2: 233, 3: 128, 4: 59, 5: 32, 6: 22, 7: 12, 8: 11, 9: 6, 10: 14, 11: 5, 12: 5, 13: 3, 14: 3, 15: 1}, 138: {0: 5, 2: 1}, 140: {0: 587, 1: 193, 2: 80, 3: 32, 4: 19, 5: 3, 6: 3, 256: 42, 257: 13, 258: 2, 259: 1, 260: 3, 512: 9, 513: 5, 768: 5, 1024: 2}, 141: {0: 112, 1: 137, 2: 66, 3: 53, 4: 13, 5: 2, 6: 1, 7: 2, 8: 1, 12: 1, 256: 12, 257: 12, 258: 2, 259: 4, 512: 5, 513: 1, 515: 2, 520: 2, 768: 1, 769: 4, 770: 2, 771: 3, 1024: 2, 1281: 1, 1536: 1, 1537: 2, 2048: 2}, 142: {0: 276, 1: 109, 2: 36, 3: 3, 4: 3, 256: 3, 257: 7, 258: 2, 513: 1, 768: 2, 770: 1}, 143: {5: 92, 7: 34, 10: 7, 11: 25, 12: 67, 13: 47, 14: 73, 15: 213, 16: 17, 17: 177, 18: 371, 19: 171, 20: 416, 21: 320, 22: 70, 23: 195, 24: 225, 25: 45, 26: 147, 27: 181, 28: 50, 29: 201, 30: 130, 31: 60, 32: 86, 33: 141, 34: 48, 35: 51, 36: 87, 37: 53, 38: 42, 39: 83, 40: 34, 41: 39, 42: 70, 43: 37, 44: 35, 45: 51, 46: 28, 47: 30, 48: 39, 49: 30, 50: 38, 51: 46, 52: 23, 53: 20, 54: 45, 55: 26, 56: 27, 57: 54, 58: 20, 59: 28, 60: 23, 61: 25, 62: 17, 63: 29, 64: 24, 65: 17, 66: 17, 67: 26, 68: 19, 69: 35, 70: 21, 71: 21, 72: 22, 73: 23, 74: 22, 75: 22, 76: 23, 77: 23, 78: 23, 79: 12, 80: 15, 81: 17, 82: 17, 83: 21, 84: 15, 85: 21, 86: 22, 87: 9, 88: 22, 89: 23, 90: 15, 91: 16, 92: 24, 93: 6, 94: 20, 95: 15, 96: 11, 97: 9, 98: 12, 99: 12, 100: 15, 101: 11, 102: 9, 103: 21, 104: 12, 105: 10, 106: 6, 107: 16, 108: 7, 109: 12, 110: 8, 111: 9, 112: 12, 113: 7, 114: 7, 115: 9, 116: 10, 117: 13, 118: 10, 119: 7, 120: 6, 121: 17, 122: 7, 123: 9, 124: 3, 125: 3, 126: 9, 127: 7, 128: 6, 129: 8, 130: 5, 131: 11, 132: 9, 133: 10, 134: 5, 135: 6, 136: 6, 137: 5, 138: 9, 139: 7, 140: 7, 141: 3, 142: 2, 143: 6, 144: 5, 145: 2, 146: 5, 147: 3, 148: 4, 149: 3, 150: 2, 151: 4, 152: 7, 153: 5, 154: 3, 155: 8, 156: 9, 157: 10, 158: 6, 159: 4, 160: 7, 161: 1, 162: 1, 163: 6, 164: 7, 165: 6, 166: 3, 167: 8, 168: 1, 169: 6, 170: 5, 171: 1, 172: 2, 173: 4, 174: 1, 175: 1, 176: 2, 177: 1, 178: 4, 180: 1, 181: 1, 182: 5, 183: 4, 184: 3, 185: 2, 186: 5, 187: 6, 188: 6, 189: 3, 190: 4, 192: 4, 193: 5, 194: 5, 195: 2, 196: 2, 197: 1, 200: 1, 201: 5, 202: 4, 203: 1, 204: 1, 205: 3, 206: 3, 207: 2, 208: 2, 209: 4, 210: 5, 211: 2, 213: 2, 214: 3, 215: 2, 217: 3, 218: 3, 219: 2, 220: 2, 221: 1, 222: 2, 223: 1, 225: 2, 226: 1, 227: 1, 228: 2, 229: 1, 231: 4, 232: 4, 233: 1, 234: 2, 235: 3, 237: 1, 239: 2, 240: 2, 241: 2, 242: 1, 244: 1, 245: 1, 246: 2, 247: 1, 248: 1, 249: 1, 250: 3, 253: 2, 256: 2, 259: 1, 261: 1, 262: 2, 263: 1, 264: 2, 265: 2, 266: 3, 267: 3, 269: 3, 272: 1, 273: 1, 274: 1, 275: 1, 276: 2, 277: 1, 278: 2, 280: 1, 282: 2, 285: 2, 286: 1, 289: 3, 290: 1, 292: 1, 293: 2, 294: 1, 296: 3, 297: 1, 299: 1, 300: 1, 301: 1, 305: 2, 306: 1, 308: 1, 310: 1, 311: 1, 314: 1, 316: 2, 317: 1, 319: 1, 320: 1, 322: 1, 335: 1, 337: 1, 341: 1, 343: 1, 345: 3, 349: 4, 350: 1, 352: 1, 356: 1, 358: 2, 361: 1, 364: 1, 366: 1, 368: 1, 372: 1, 376: 1, 377: 2, 380: 2, 384: 1, 385: 1, 391: 1, 406: 1, 413: 1, 422: 1, 427: 1, 428: 1, 439: 1, 444: 1, 451: 1, 459: 1, 463: 1, 465: 1, 469: 1, 477: 1, 485: 1, 487: 1, 490: 1, 491: 1, 500: 1, 519: 1, 525: 2, 527: 2, 531: 1, 562: 1, 566: 1, 586: 1, 593: 1, 604: 1, 609: 1, 634: 1, 650: 1, 673: 1, 676: 1, 680: 1, 703: 1, 728: 1, 733: 1, 737: 1, 859: 1, 903: 1, 915: 1, 972: 1, 2436: 1, 3759: 1}, 145: {2: 1243, 3: 37, 4: 7}, 146: {2: 27, 3: 2}, 147: {2: 60, 3: 6}, 148: {0: 160, 1: 26, 2: 5, 3: 2, 4: 1}, 149: {1: 2, 2: 12, 3: 3, 4: 7}, 150: {2: 1}, 152: {2: 1}, 154: {5: 7, 7: 3, 11: 2, 13: 2, 15: 3, 26: 1, 31: 1, 32: 1, 33: 1, 35: 1, 37: 1, 39: 1, 48: 2, 67: 1, 70: 2, 81: 2, 88: 1, 90: 2}, 155: {0: 164, 1: 1, 2: 13, 3: 2, 4: 31, 5: 4, 6: 4, 7: 1}, } From victor.stinner at gmail.com Thu Feb 4 11:16:07 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 4 Feb 2016 17:16:07 +0100 Subject: [Python-ideas] More compact bytecode In-Reply-To: References: Message-ID: 2016-02-04 14:03 GMT+01:00 Serhiy Storchaka : > Damien George mentioned [1] that in MicroPython there are 16 dedicated > opcodes for loading from positions 0-15, and 16 for storing to these > positions. This not only makes a bytecode more compact, but might make > CPython slightly faster too. MicroPython has a very important requirements on disk and memory footprint. I don't think that it's the case for CPython. > 1. Add 1-byte dedicated opcodes for most used opcodes with arguments. It means duplicating a lot of code in ceval.c, no? (Even if we use C #define "templates"). I dislike this option. > 2. Use 16-bit opcodes as in WPython. IMHO this is a much more interesting option. It would be great to remove "if (HAS_ARG(opcode)) oparg = NEXTARG();" and always get the argument from the 16-bit opcode. This if() adds more work to the CPU which has to flush the pipeline on branch misprediction. Usually, it doesn't matter. For this if() is really part of the hot path code Python, it's the most important loop running the bytecode. So any instruction matters here ;-) You just have to handle correctly EXTENDED_ARG to large arguments. Victor From storchaka at gmail.com Thu Feb 4 11:55:43 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 4 Feb 2016 18:55:43 +0200 Subject: [Python-ideas] More compact bytecode In-Reply-To: References: Message-ID: On 04.02.16 18:16, Victor Stinner wrote: >> 1. Add 1-byte dedicated opcodes for most used opcodes with arguments. > > It means duplicating a lot of code in ceval.c, no? (Even if we use C > #define "templates"). I dislike this option. This needs to add less than 40 lines of code in ceval.c. 5-line macro and 1 line per new opcode. A little more lines need to be added to compiler, peephole optimizer, opcode.py, opcode.h and the documentation. >> 2. Use 16-bit opcodes as in WPython. > > IMHO this is a much more interesting option. > > It would be great to remove "if (HAS_ARG(opcode)) oparg = NEXTARG();" > and always get the argument from the 16-bit opcode. This if() adds > more work to the CPU which has to flush the pipeline on branch > misprediction. Usually, it doesn't matter. For this if() is really > part of the hot path code Python, it's the most important loop running > the bytecode. So any instruction matters here ;-) Actually in common case this "if" is executed at compile time. But I expect a benefit from using simple read of 16-bit value instead of 3 reads of 8-bit values. From srkunze at mail.de Thu Feb 4 12:27:39 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 4 Feb 2016 18:27:39 +0100 Subject: [Python-ideas] More compact bytecode In-Reply-To: References: Message-ID: <56B38A0B.1090508@mail.de> On 04.02.2016 17:55, Serhiy Storchaka wrote: > On 04.02.16 18:16, Victor Stinner wrote: >>> 1. Add 1-byte dedicated opcodes for most used opcodes with arguments. >> >> It means duplicating a lot of code in ceval.c, no? (Even if we use C >> #define "templates"). I dislike this option. > > This needs to add less than 40 lines of code in ceval.c. 5-line macro > and 1 line per new opcode. A little more lines need to be added to > compiler, peephole optimizer, opcode.py, opcode.h and the documentation. > Do you have a working CPython to showcast this? >>> 2. Use 16-bit opcodes as in WPython. >> >> IMHO this is a much more interesting option. >> >> It would be great to remove "if (HAS_ARG(opcode)) oparg = NEXTARG();" >> and always get the argument from the 16-bit opcode. This if() adds >> more work to the CPU which has to flush the pipeline on branch >> misprediction. Usually, it doesn't matter. For this if() is really >> part of the hot path code Python, it's the most important loop running >> the bytecode. So any instruction matters here ;-) > > Actually in common case this "if" is executed at compile time. But I > expect a benefit from using simple read of 16-bit value instead of 3 > reads of 8-bit values. Same question as above + does it make sense to combine those two options? Best, Sven From abarnert at yahoo.com Thu Feb 4 12:28:45 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 4 Feb 2016 17:28:45 +0000 (UTC) Subject: [Python-ideas] More compact bytecode In-Reply-To: References: Message-ID: <2075040798.1404427.1454606925874.JavaMail.yahoo@mail.yahoo.com> On Thursday, February 4, 2016 8:55 AM, Serhiy Storchaka wrote: > > On 04.02.16 18:16, Victor Stinner wrote: >>> 2. Use 16-bit opcodes as in WPython. >> >> IMHO this is a much more interesting option. >> >> It would be great to remove "if (HAS_ARG(opcode)) oparg = > NEXTARG();" >> and always get the argument from the 16-bit opcode. This if() adds >> more work to the CPU which has to flush the pipeline on branch >> misprediction. Usually, it doesn't matter. For this if() is really >> part of the hot path code Python, it's the most important loop running >> the bytecode. So any instruction matters here ;-) > > Actually in common case this "if" is executed at compile time. But I > expect a benefit from using simple read of 16-bit value instead of 3 > reads of 8-bit values. Only if the 16-bit reads are aligned. Can that be guaranteed somehow? If you change co_code to some type that's like bytes, but an immutable array of shorts instead of bytes, that would certainly do it. And it would be easier to inspect/hack 16-bit bytecodes from Python if you didn't have to look at them as bytes. But that implies creating the new type, exposing it to Python, and changing what co_code returns. Or it could just be a simple switch when constructing a code object: if the code argument is aligned, use it as-is; if not, copy its contents into an aligned array, build a new bytes around that, and store that instead. (Would that significantly slow down .pyc reads and other marshaling uses, if half of all code objects have to copy their contents?) From python at mrabarnett.plus.com Thu Feb 4 13:26:50 2016 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 04 Feb 2016 18:26:50 +0000 Subject: [Python-ideas] More compact bytecode In-Reply-To: <2075040798.1404427.1454606925874.JavaMail.yahoo@mail.yahoo.com> Message-ID: On 2016-02-04 17:28:45, "Andrew Barnert via Python-ideas" wrote: >On Thursday, February 4, 2016 8:55 AM, Serhiy Storchaka > wrote: > >> > On 04.02.16 18:16, Victor Stinner wrote: > > >>>> 2. Use 16-bit opcodes as in WPython. >>> >>> IMHO this is a much more interesting option. >>> >>> It would be great to remove "if (HAS_ARG(opcode)) oparg = >> NEXTARG();" >>> and always get the argument from the 16-bit opcode. This if() adds >>> more work to the CPU which has to flush the pipeline on branch >>> misprediction. Usually, it doesn't matter. For this if() is really >>> part of the hot path code Python, it's the most important loop >>>running >>> the bytecode. So any instruction matters here ;-) >> >> Actually in common case this "if" is executed at compile time. But I >> expect a benefit from using simple read of 16-bit value instead of 3 >> reads of 8-bit values. > >Only if the 16-bit reads are aligned. Can that be guaranteed somehow? > Aren't allocated blocks usually aligned to n-byte boundaries (or multiples thereof) anyway, where n is the number of bytes needed to hold an address? >If you change co_code to some type that's like bytes, but an immutable >array of shorts instead of bytes, that would certainly do it. And it >would be easier to inspect/hack 16-bit bytecodes from Python if you >didn't have to look at them as bytes. But that implies creating the new >type, exposing it to Python, and changing what co_code returns. > >Or it could just be a simple switch when constructing a code object: if >the code argument is aligned, use it as-is; if not, copy its contents >into an aligned array, build a new bytes around that, and store that >instead. (Would that significantly slow down .pyc reads and other >marshaling uses, if half of all code objects have to copy their >contents?) > From storchaka at gmail.com Thu Feb 4 13:48:46 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 4 Feb 2016 20:48:46 +0200 Subject: [Python-ideas] More compact bytecode In-Reply-To: <2075040798.1404427.1454606925874.JavaMail.yahoo@mail.yahoo.com> References: <2075040798.1404427.1454606925874.JavaMail.yahoo@mail.yahoo.com> Message-ID: On 04.02.16 19:28, Andrew Barnert via Python-ideas wrote: >> Actually in common case this "if" is executed at compile time. But I >> expect a benefit from using simple read of 16-bit value instead of 3 >> reads of 8-bit values. > > Only if the 16-bit reads are aligned. Can that be guaranteed somehow? Yes, we can. On common platforms the start of bytes data is at least 32-bit aligned. Other code depends on this. It can be not aligned only on platform where the alignment doesn't matter. From storchaka at gmail.com Thu Feb 4 13:55:39 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 4 Feb 2016 20:55:39 +0200 Subject: [Python-ideas] More compact bytecode In-Reply-To: <56B38A0B.1090508@mail.de> References: <56B38A0B.1090508@mail.de> Message-ID: On 04.02.16 19:27, Sven R. Kunze wrote: > On 04.02.2016 17:55, Serhiy Storchaka wrote: >>>> 1. Add 1-byte dedicated opcodes for most used opcodes with arguments. > > Do you have a working CPython to showcast this? Not yet. The interpreter is easy, but the compiler is more complicated in this case. >>>> 2. Use 16-bit opcodes as in WPython. > > Same question as above + does it make sense to combine those two options? Of course not. Having some opcodes to be 8-bit kills the benefit of all 16-bit opcodes. This options needs to rewrite more code, but resulting code can be easier. I afraid that might be problem with bootstrapping, generating freezed modules. From barry at python.org Thu Feb 4 14:21:01 2016 From: barry at python.org (Barry Warsaw) Date: Thu, 4 Feb 2016 14:21:01 -0500 Subject: [Python-ideas] A bit meta References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <56AB94A2.20901@stoneleaf.us> <20160130041001.GI4619@ando.pearwood.info> <2067CA09-F205-41A2-A0E0-17065DB1BE00@gmail.com> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> Message-ID: <20160204142101.23da31f6@anarchist.wooz.org> On Jan 31, 2016, at 06:27 PM, Brett Cannon wrote: >relating to Python. But go into any university around the world and ask >some CS student, "what is Usenet?" -- let alone NNTP -- and it's quite >possible you will get a blank stare. Which is too bad because NNTP is a pretty decent protocol, apart from any historical connection to Usenet. >For instance, people have said they don't want to set up another account. >But people forget that *every* mailing list on mail.python.org requires its >own account to post (I personally have near a bazillion at this point). Which will go away as we start to migrate to Mailman 3. Sadly, one of the more interesting SSO approaches was Persona which Mozilla has EOLd. There are ongoing discussions on how best to replace that. At worst, you'll have a single login per MM3 installation. Maybe you'll be able to use something else like a OpenID/Google/FB or other login but you won't be able to rely on Persona to provide a verified email address on first login. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Thu Feb 4 14:38:54 2016 From: barry at python.org (Barry Warsaw) Date: Thu, 4 Feb 2016 14:38:54 -0500 Subject: [Python-ideas] A bit meta References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> <85bn81eesp.fsf@benfinney.id.au> Message-ID: <20160204143854.4bb6a05a@anarchist.wooz.org> On Feb 01, 2016, at 01:59 AM, Nicholas Chammas wrote: >I *have* been arguing that a modern web-based forum solves common discussion >issues in a way that mailing lists cannot match. And yet, web based forums come with their own limitations and problems. I won't outline all of my complaints about things like Gmail and other such web-based solutions because they'll likely devolve into the general issue with all of our communication mediums: it is a highly religious argument. I have workflows and tools I use to efficiently engage in discussions with open source communities. They aren't perfect, but they are damn good and very comfortable. You have or prefer different tools and workflows that make your engagement more fun and productive. Are yours wrong and mine right? Or vice versa? Of course not! There's always going to be a conflict between status quo and change, and in general I think that's healthy. It's important for new arrivals to try to understand why there's so much passion in the status quo, and it's good for old-timers to be challenged now and then. I think success depends on balance and evolution rather than revolution. In any case, all this is very interesting, but kind of pointless until and unless people take ownership and responsibility for implementing something different. Take a good look at what Brett is doing to migrate us to git. It doesn't matter if I disagree with his choices, he's going to do the very hard work of actually making it happen, bridge the gap between evolution and revolution, and be the guy who ensures all the kinks are smoothed out. And that's the only way it will work -- kudo to him and the other folks who will be making this happen. Such changes are not short-term commitments, but I encourage those who really want to see a change in how we communicate, to figure out how to make that happen. As I've said, for the part that interests me, I will work with anybody who wants to make Mailman 3 and Discourse work better together. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Thu Feb 4 14:42:10 2016 From: barry at python.org (Barry Warsaw) Date: Thu, 4 Feb 2016 14:42:10 -0500 Subject: [Python-ideas] A better python mode for Emacs (was Re: A bit meta) References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <2067CA09-F205-41A2-A0E0-17065DB1BE00@gmail.com> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> <85bn81eesp.fsf@benfinney.id.au> <506f4995-f5ca-4431-abe9-ed83f33c0406@googlegroups.com> Message-ID: <20160204144210.6186a59e@anarchist.wooz.org> On Feb 01, 2016, at 05:22 AM, Rustom Mody wrote: >[Since you mentioned emacs...] Things can get bitrotten and worse simply by >passage of time Here's my mail to emacs list about python mode(s) that seems >to be getting brokener and brokener by the day >http://lists.gnu.org/archive/html/help-gnu-emacs/2016-01/msg00372.html You might have better luck with https://gitlab.com/python-mode-devs/python-mode Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Thu Feb 4 14:46:28 2016 From: barry at python.org (Barry Warsaw) Date: Thu, 4 Feb 2016 14:46:28 -0500 Subject: [Python-ideas] A bit meta References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <20160130041001.GI4619@ando.pearwood.info> <2067CA09-F205-41A2-A0E0-17065DB1BE00@gmail.com> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> <85bn81eesp.fsf@benfinney.id.au> <56AF1367.6010206@egenix.com> Message-ID: <20160204144628.10c00d26@anarchist.wooz.org> On Feb 01, 2016, at 06:02 PM, Brett Cannon wrote: >That assumes you can even do that, e.g., I use Google Inbox and there is no >plain text option. I'm afraid this is an example of the OSS community >trying to swim against the stream where the rest of the world has moved on >and it is slowly making it harder to get new people to participate in OSS. Or alternatively, an 800lb gorilla deciding what is best for everyone --even those not using its tools-- and ignoring decades of interoperability and standards, slowly making it less fun for long-time contributors to continue to participate. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Thu Feb 4 15:04:51 2016 From: barry at python.org (Barry Warsaw) Date: Thu, 4 Feb 2016 15:04:51 -0500 Subject: [Python-ideas] A bit meta References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> <85bn81eesp.fsf@benfinney.id.au> <56AF1367.6010206@egenix.com> <56AFB0A9.2010107@egenix.com> Message-ID: <20160204150451.53fca430@anarchist.wooz.org> On Feb 04, 2016, at 10:42 PM, Nick Coghlan wrote: >If Mark is willing to shepherd an MM3 upgrade on a volunteer basis, >that would be excellent, but I also expect it to be a fair bit of >work. We are talking about it. Stay tuned. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From srkunze at mail.de Thu Feb 4 15:13:00 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 4 Feb 2016 21:13:00 +0100 Subject: [Python-ideas] More compact bytecode In-Reply-To: References: <56B38A0B.1090508@mail.de> Message-ID: <56B3B0CC.7010205@mail.de> On 04.02.2016 19:55, Serhiy Storchaka wrote: > On 04.02.16 19:27, Sven R. Kunze wrote: >> On 04.02.2016 17:55, Serhiy Storchaka wrote: >>>>> 1. Add 1-byte dedicated opcodes for most used opcodes with arguments. >> >> Do you have a working CPython to showcast this? > > Not yet. The interpreter is easy, but the compiler is more complicated > in this case. > >>>>> 2. Use 16-bit opcodes as in WPython. >> >> Same question as above + does it make sense to combine those two >> options? > > Of course not. Having some opcodes to be 8-bit kills the benefit of > all 16-bit opcodes. Are you sure? I still could imagine some benefit. > This options needs to rewrite more code, but resulting code can be > easier. I afraid that might be problem with bootstrapping, generating > freezed modules. What option do you prefer then? From random832 at fastmail.com Thu Feb 4 15:27:05 2016 From: random832 at fastmail.com (Random832) Date: Thu, 04 Feb 2016 15:27:05 -0500 Subject: [Python-ideas] A bit meta In-Reply-To: <20160204144628.10c00d26@anarchist.wooz.org> References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <20160130041001.GI4619@ando.pearwood.info> <2067CA09-F205-41A2-A0E0-17065DB1BE00@gmail.com> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> <85bn81eesp.fsf@benfinney.id.au> <56AF1367.6010206@egenix.com> <20160204144628.10c00d26@anarchist.wooz.org> Message-ID: <1454617625.550176.512255274.0B319EC5@webmail.messagingengine.com> On Thu, Feb 4, 2016, at 14:46, Barry Warsaw wrote: > On Feb 01, 2016, at 06:02 PM, Brett Cannon wrote: >> That assumes you can even do that, e.g., I use Google Inbox and there >> is no plain text option. I'm afraid this is an example of the OSS >> community trying to swim against the stream where the rest of the >> world has moved on and it is slowly making it harder to get new >> people to participate in OSS. > > Or alternatively, an 800lb gorilla deciding what is best for everyone > --even those not using its tools-- and ignoring decades of > interoperability and standards, slowly making it less fun for long- > time contributors to continue to participate.? There's no singular 800lb gorilla pushing the trend towards HTML mail. There's the fact that real users want to be able to use font styles, inline images, and tables, and anyone who wanted that to be handled via something other than HTML (text/enriched or whatever) has *completely* dropped the ball. (I don't regularly post here, but I've written this response in HTML on general principle) -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Thu Feb 4 15:29:02 2016 From: random832 at fastmail.com (Random832) Date: Thu, 04 Feb 2016 15:29:02 -0500 Subject: [Python-ideas] A bit meta In-Reply-To: <1454617625.550176.512255274.0B319EC5@webmail.messagingengine.com> References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <20160130041001.GI4619@ando.pearwood.info> <2067CA09-F205-41A2-A0E0-17065DB1BE00@gmail.com> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> <85bn81eesp.fsf@benfinney.id.au> <56AF1367.6010206@egenix.com> <20160204144628.10c00d26@anarchist.wooz.org> <1454617625.550176.512255274.0B319EC5@webmail.messagingengine.com> Message-ID: <1454617742.550346.512264626.3CE0A764@webmail.messagingengine.com> On Thu, Feb 4, 2016, at 15:27, Random832 wrote: > (I don't regularly post here, but I've written this response in HTML on > general principle) I meant to say, of course, that I don't regularly post *in HTML* here. From abarnert at yahoo.com Thu Feb 4 16:43:19 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 4 Feb 2016 21:43:19 +0000 (UTC) Subject: [Python-ideas] More compact bytecode References: <1839413661.1594086.1454622199260.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <1839413661.1594086.1454622199260.JavaMail.yahoo@mail.yahoo.com> On Feb 4, 2016, at 10:55, Serhiy Storchaka wrote: >> On 04.02.16 19:27, Sven R. Kunze wrote: >> On 04.02.2016 17:55, Serhiy Storchaka wrote: >>>>> 1. Add 1-byte dedicated opcodes for most used opcodes with arguments. >> >> Do you have a working CPython to showcast this? > > Not yet. The interpreter is easy, but the compiler is more complicated in this case. > >>>>> 2. Use 16-bit opcodes as in WPython. >> >> Same question as above + does it make sense to combine those two options? > > Of course not. Having some opcodes to be 8-bit kills the benefit of all 16-bit opcodes. > > This options needs to rewrite more code, but resulting code can be easier. I afraid that might be problem with bootstrapping, generating freezed modules. I just tried it out to see what's hard. And it seems like it's all easy. I was able to get it working (not passing all tests, of course--you'd have to fix dis, pdb, etc. for that) on my lunch hour. I didn't try to be smart anywhere: every opcode gets 8 bits for an arg, and uses up to three EXTENDED_ARG ops if it needs more. I didn't do any simplification, nor did I cast the bytecode to unsigned short *. The hardest changes are in ceval (plus frameobject--I didn't realize lasti has to start lasti at -2 instead of -1 until I tried to run it and immediately got errors about unknown opcodes at odd offsets). Compile is pretty easy by comparison. Nothing at all has to be done for bootstrap, freeze, marshal, etc. (unless you want to add backward-compat code to load non-word-based .pyc files). You'd also need to fix at least peephole (I just disabled it), dis, and pdb, but none of them look too hard. The bootstrap bytecode and stdlib .pyc files ends up about 5% smaller (and that's with the peephole optimizer disabled, so presumably it would be better in real life). If anyone wants to look at or play with the code, I'm in the process of regenerating my github fork, but it'll be branch wpy on https://github.com/abarnert/cpython once that's done (maybe 20 minutes? I forget how long cloning takes). From storchaka at gmail.com Thu Feb 4 17:11:07 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 5 Feb 2016 00:11:07 +0200 Subject: [Python-ideas] More compact bytecode In-Reply-To: <1839413661.1594086.1454622199260.JavaMail.yahoo@mail.yahoo.com> References: <1839413661.1594086.1454622199260.JavaMail.yahoo.ref@mail.yahoo.com> <1839413661.1594086.1454622199260.JavaMail.yahoo@mail.yahoo.com> Message-ID: On 04.02.16 23:43, Andrew Barnert via Python-ideas wrote: > I just tried it out to see what's hard. And it seems like it's all easy. I was able to get it working (not passing all tests, of course--you'd have to fix dis, pdb, etc. for that) on my lunch hour. > > > I didn't try to be smart anywhere: every opcode gets 8 bits for an arg, and uses up to three EXTENDED_ARG ops if it needs more. I didn't do any simplification, nor did I cast the bytecode to unsigned short *. > The hardest changes are in ceval (plus frameobject--I didn't realize lasti has to start lasti at -2 instead of -1 until I tried to run it and immediately got errors about unknown opcodes at odd offsets). > > Compile is pretty easy by comparison. > > Nothing at all has to be done for bootstrap, freeze, marshal, etc. (unless you want to add backward-compat code to load non-word-based .pyc files). > > You'd also need to fix at least peephole (I just disabled it), dis, and pdb, but none of them look too hard. > > The bootstrap bytecode and stdlib .pyc files ends up about 5% smaller (and that's with the peephole optimizer disabled, so presumably it would be better in real life). > > > If anyone wants to look at or play with the code, I'm in the process of regenerating my github fork, but it'll be branch wpy on https://github.com/abarnert/cpython once that's done (maybe 20 minutes? I forget how long cloning takes). Looks good. Right now I'm implementing the first option (the hard part is peephole optimizer). We could compare both approaches later. From rosuav at gmail.com Thu Feb 4 17:22:09 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 5 Feb 2016 09:22:09 +1100 Subject: [Python-ideas] A bit meta In-Reply-To: <1454617625.550176.512255274.0B319EC5@webmail.messagingengine.com> References: <9AFD3488-88D9-4CE2-9772-EB1FDD65615A@selik.org> <20160130041001.GI4619@ando.pearwood.info> <2067CA09-F205-41A2-A0E0-17065DB1BE00@gmail.com> <22189.23440.832633.260311@turnbull.sk.tsukuba.ac.jp> <85bn81eesp.fsf@benfinney.id.au> <56AF1367.6010206@egenix.com> <20160204144628.10c00d26@anarchist.wooz.org> <1454617625.550176.512255274.0B319EC5@webmail.messagingengine.com> Message-ID: On Fri, Feb 5, 2016 at 7:27 AM, Random832 wrote: > There's no singular 800lb gorilla pushing the trend towards HTML mail. > There's the fact that real users want to be able to use font styles, inline > images, and tables, and anyone who wanted that to be handled via something > other than HTML (text/enriched or whatever) has completely dropped the ball. Sure, nobody's pushing for PDF Emails or anything like that. But it's not that hard to use a bit of casual text markup instead of actual font styles; and honestly, the number of times tables/images/etc are used well is utterly dwarfed by the number of times they're used uselessly. Epitomizing the inanity of a lot of people's email usage is one person I know who will reply-all to a long post, quoting the whole thing, and top-posting a "Thanks!" with a signature attached. That's the entire content of the post, but it has a whole lot of color and formatting and stuff. If text-only emails lose us that kind of thing at the cost of also losing us the occasional inline image, I'm okay with that price. ChrisA From abarnert at yahoo.com Thu Feb 4 17:22:09 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 4 Feb 2016 22:22:09 +0000 (UTC) Subject: [Python-ideas] More compact bytecode In-Reply-To: <56B3B0CC.7010205@mail.de> References: <56B3B0CC.7010205@mail.de> Message-ID: <1071242114.1621780.1454624529906.JavaMail.yahoo@mail.yahoo.com> On Thursday, February 4, 2016 12:13 PM, Sven R. Kunze wrote: > On 04.02.2016 19:55, Serhiy Storchaka wrote: >> On 04.02.16 19:27, Sven R. Kunze wrote: >>> On 04.02.2016 17:55, Serhiy Storchaka wrote: >>>>>> 1. Add 1-byte dedicated opcodes for most used opcodes with > arguments. >>> >>> Do you have a working CPython to showcast this? >> >> Not yet. The interpreter is easy, but the compiler is more complicated >> in this case. >> >>>>>> 2. Use 16-bit opcodes as in WPython. >>> >>> Same question as above + does it make sense to combine those two >>> options? >> >> Of course not. Having some opcodes to be 8-bit kills the benefit of >> all 16-bit opcodes. > > Are you sure? I still could imagine some benefit. The only benefits I can see for using 16-bit opcodes are: * You can treat the bytecode as an (unsigned short *) instead of (unsigned char *) and do 16-bit reads everywhere instead of 8-bit reads. This will no longer be true if you have to do an 8-bit read and then conditionally read more bits. * The ceval loop can be simplified if it doesn't have to deal with variable lengths. This will no longer be true if you have to deal with variable lengths. So, unless you're imagining something else, I don't think there is any benefit to be gotten by directly combining the two. You may be able to import some of the "spirit" of the first one into the second, but I'm not sure it's worth it. One of the minor problems with wordcode is that args in range [256, 65536) now take 4 bytes instead of 3. If you cram a few bits into the opcode byte, you can push the range farther back. I'm guessing it's pretty rare to do LOAD_CONST 256, but giving, say, 5 bits to JUMP_ABSOLUTE, and 2 bits to each of the various relative jumps, would cut down the need for EXTENDED_ARGS dramatically. From abarnert at yahoo.com Thu Feb 4 17:45:38 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 4 Feb 2016 22:45:38 +0000 (UTC) Subject: [Python-ideas] A bit meta In-Reply-To: References: Message-ID: <1264636109.1628200.1454625938457.JavaMail.yahoo@mail.yahoo.com> On Thursday, February 4, 2016 2:22 PM, Chris Angelico wrote: > Sure, nobody's pushing for PDF Emails or anything like that. But it's > not that hard to use a bit of casual text markup instead of actual > font styles; and honestly, the number of times tables/images/etc are > used well is utterly dwarfed by the number of times they're used > uselessly. Epitomizing the inanity of a lot of people's email usage is > one person I know who will reply-all to a long post, quoting the whole > thing, and top-posting a "Thanks!" with a signature attached. On the other hand, I occasionally have to deal with a guy who used to wRITE aLL hIS tEXT lIKE tHIS fOR sOME rEASON, but since he discovered HTML mail, he now instead uses tons of bizarre formatting with normal capitalization. I can just view as plain text and his mail is perfectly readable. But he doesn't _know_ it's readable, so presumably the voices in his head are satisfied. From rosuav at gmail.com Thu Feb 4 21:57:57 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 5 Feb 2016 13:57:57 +1100 Subject: [Python-ideas] More compact bytecode In-Reply-To: <1071242114.1621780.1454624529906.JavaMail.yahoo@mail.yahoo.com> References: <56B3B0CC.7010205@mail.de> <1071242114.1621780.1454624529906.JavaMail.yahoo@mail.yahoo.com> Message-ID: On Fri, Feb 5, 2016 at 9:22 AM, Andrew Barnert via Python-ideas wrote: > You may be able to import some of the "spirit" of the first one into the second, but I'm not sure it's worth it. One of the minor problems with wordcode is that args in range [256, 65536) now take 4 bytes instead of 3. If you cram a few bits into the opcode byte, you can push the range farther back. I'm guessing it's pretty rare to do LOAD_CONST 256, but giving, say, 5 bits to JUMP_ABSOLUTE, and 2 bits to each of the various relative jumps, would cut down the need for EXTENDED_ARGS dramatically. > At the cost of complexity, possibly including less readable code. This change can always be done later, if and only if it proves to make enough difference in performance. ChrisA From abarnert at yahoo.com Thu Feb 4 23:25:39 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 5 Feb 2016 04:25:39 +0000 (UTC) Subject: [Python-ideas] More compact bytecode In-Reply-To: References: Message-ID: <16900900.1740372.1454646339061.JavaMail.yahoo@mail.yahoo.com> On Thursday, February 4, 2016 6:57 PM, Chris Angelico wrote: > On Fri, Feb 5, 2016 at 9:22 AM, Andrew Barnert via Python-ideas > wrote: >> You may be able to import some of the "spirit" of the first one >> into the second, but I'm not sure it's worth it. One of the minor >> problems with wordcode is that args in range [256, 65536) now take 4 bytes >> instead of 3. If you cram a few bits into the opcode byte, you can push the >> range farther back. I'm guessing it's pretty rare to do LOAD_CONST 256, >> but giving, say, 5 bits to JUMP_ABSOLUTE, and 2 bits to each of the various >> relative jumps, would cut down the need for EXTENDED_ARGS dramatically. > > At the cost of complexity, possibly including less readable code. This > change can always be done later, if and only if it proves to make > enough difference in performance. Definitely. And, of course, the extra complexity may actually slow things down. I'm just saying it _might_ be worth prototyping and testing if the wordcode experiment turns out to be worth pursuing[^1], and someone produces profiles showing that jumps are now a bigger bottleneck than they used to be. [^1]: We already knew that it was the starting point of a 2.6 fork that claimed to be faster than stock CPython, but that fork had lots of other optimizations, and I don't know of any serious benchmarking or profiling results from it in the first place. We now also know that changing to wordcode is probably easier than it looks. That may add up to "worth a few more hours to build a complete prototype", but it doesn't add up to "do it and merge it and start planning further changes around it" just yet. :) From rosuav at gmail.com Fri Feb 5 00:01:27 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 5 Feb 2016 16:01:27 +1100 Subject: [Python-ideas] More compact bytecode In-Reply-To: <16900900.1740372.1454646339061.JavaMail.yahoo@mail.yahoo.com> References: <16900900.1740372.1454646339061.JavaMail.yahoo@mail.yahoo.com> Message-ID: On Fri, Feb 5, 2016 at 3:25 PM, Andrew Barnert wrote: > That may add up to "worth a few more hours to build a complete prototype", but it doesn't add up to "do it and merge it and start planning further changes around it" just yet. :) > Yep! But also, I'm inclined to reduce, rather than expand, the scope of the change - it's a few hours followed by a few more to do the jumps-with-some-bits-in-opcode, etc, etc. ChrisA From steve.dower at python.org Fri Feb 5 00:21:11 2016 From: steve.dower at python.org (Steve Dower) Date: Thu, 4 Feb 2016 21:21:11 -0800 Subject: [Python-ideas] List Comprehensions In-Reply-To: References: Message-ID: In Python 3.5: >>> [*(x for x in range(10)), 100] Should give [0, 1 ... 10, 100]. You can now expand sequence and dict items with * and **, similar to how function calls work. Top-posted from my Windows Phone -----Original Message----- From: "shiva prasanth" Sent: ?2/?3/?2016 1:00 To: "python-ideas at python.org" Subject: [Python-ideas] List Comprehensions python input and output In [2]: z=[ x for x in range(10)] In [3]: z Out[3]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] In [4]: z=[x for x in range(10),100] # should be [ 0 , 1, 2, 3, 4, 5, 6, 7, 8, 9, 100] # but output is not as expected # following is the output # other wise it should show error In [5]: z Out[5]: [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], 100] -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Feb 5 14:58:58 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 5 Feb 2016 19:58:58 +0000 (UTC) Subject: [Python-ideas] Exposing flat bytecode representation to optimizers References: <1763502835.2002313.1454702338876.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <1763502835.2002313.1454702338876.JavaMail.yahoo@mail.yahoo.com> The biggest pain with dealing with the peephole optimizer is that it happens after all the complicated flattening and fixup[^1] the compiler does, which means you have to hack up the jump targets as you go along. The future bytecode optimizers that PEP 511 enables will have the same headache. But this isn't actually necessary. The optimizer could work on a flat array of instructions[^2] instead of an array of bytes, with relocatable jump targets instead of fixed byte offsets, and then the compiler could do the fixup _after_ the optimization.[^3] It would break the optimizer APIs, but `PyCode_Optimize` isn't public, and the API proposed by PEP 511 is public, but that PEP isn't even finalized, much less accepted yet. I don't think we need to expose the intermediate representation any farther along than the `PyCode_Optimize` step.[^4] Just moving the optimize one step earlier in the chain solves more than enough to be worth it. [^1]: If you think numbering the offsets and emitting the jump targets is easy: Every time you fix up a jump, that may require adding an `EXTENDED_ARG`, which means you have to redo any fixups that cross the the current instruction. The compiler does this by just looping until it hasn't added any more `EXTENDED_ARG`s. [^2]: In case anyone's thinking that wordcode would solve this problem, it doesn't. The `EXTENDED_ARG` jump targets are a much bigger hassle than the 1-or-3-byte-ops, and wordcode not only can't eliminate those, but makes `EXTENDED_ARG` more common. [^3]: The compiler doesn't actually have exactly what the optimizers would want, but it's pretty close: it has a linked list of block objects, each of which has an array of instruction objects, with jump targets being pointers to blocks. That's actually even better to work with, but too complicated to expose to optimizers. Flattening it would be trivial. Or, if that's too expensive, we could do something almost as simple and much cheaper: convert it in-line to a deque-like linked list of arrays, with jump targets being indices or pointers into that. Or we could just expose the list of blocks as-is, as an opaque thing with a mutable-deque-of-instructions API around it. [^4]: Later stages--import hooks, optimizing decorators, etc.--have the same pain as the peephole optimizer, but they also have code objects that contain other code objects, and they can be written in Python, and so on, so the same solution isn't feasible there. Of course we could add a function to convert bytecode back to a list of instructions, in some format that can be exposed to Python (in fact, `dis` already does 90% of that), and then another one to convert that back to bytecode and redo the fixup (which is basically the third-party `byteplay` module). But that's almost certainly overkill. (If we wanted that, I'd rather add `byteplay` to the stdlib, port it and `dis` to C, and expose a C API for them. And then we could use that for everything, including the peephole optimizer and PEP 511 optimizers. Which would all be really cool, but I think it's more work than we want to do, and I don't know if we'd actually want something like `byteplay` builtin even if it were easy...) From storchaka at gmail.com Fri Feb 5 15:35:50 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 5 Feb 2016 22:35:50 +0200 Subject: [Python-ideas] Exposing flat bytecode representation to optimizers In-Reply-To: <1763502835.2002313.1454702338876.JavaMail.yahoo@mail.yahoo.com> References: <1763502835.2002313.1454702338876.JavaMail.yahoo.ref@mail.yahoo.com> <1763502835.2002313.1454702338876.JavaMail.yahoo@mail.yahoo.com> Message-ID: On 05.02.16 21:58, Andrew Barnert via Python-ideas wrote: > The biggest pain with dealing with the peephole optimizer is that it happens after all the complicated flattening and fixup[^1] the compiler does, which means you have to hack up the jump targets as you go along. The future bytecode optimizers that PEP 511 enables will have the same headache. > > But this isn't actually necessary. The optimizer could work on a flat array of instructions[^2] instead of an array of bytes, with relocatable jump targets instead of fixed byte offsets, and then the compiler could do the fixup _after_ the optimization.[^3] > > It would break the optimizer APIs, but `PyCode_Optimize` isn't public, and the API proposed by PEP 511 is public, but that PEP isn't even finalized, much less accepted yet. > > > I don't think we need to expose the intermediate representation any farther along than the `PyCode_Optimize` step.[^4] Just moving the optimize one step earlier in the chain solves more than enough to be worth it. LGTM. I did have the same idea when added specialized 8-bit opcodes. Some optimization (like constant folding) it is worth to move yet one step earlier, to AST. Other idea - instead of EXTENDED_ARG have two sets of instructions: short 16-bit with 8-bit arg, and long 32-bit with 24-bit arg. For simplicity initially only long instructions are emitted (if it makes sense). After optimization the code is packed using short instructions if possible (now we have all needed information and can do this in one pass). At the same time NOPs are removed. For later stages we easy can unpack the code back to long form. From abarnert at yahoo.com Fri Feb 5 16:03:20 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 5 Feb 2016 21:03:20 +0000 (UTC) Subject: [Python-ideas] Exposing flat bytecode representation to optimizers In-Reply-To: References: Message-ID: <1080499058.2022676.1454706200143.JavaMail.yahoo@mail.yahoo.com> On Friday, February 5, 2016 12:36 PM, Serhiy Storchaka wrote: > On 05.02.16 21:58, Andrew Barnert via Python-ideas wrote: >> The biggest pain with dealing with the peephole optimizer is that it >> happens after all the complicated flattening and fixup[^1] the compiler does, >> which means you have to hack up the jump targets as you go along. The future >> bytecode optimizers that PEP 511 enables will have the same headache. >> >> But this isn't actually necessary. The optimizer could work on a flat >> array of instructions[^2] instead of an array of bytes, with relocatable jump >> targets instead of fixed byte offsets, and then the compiler could do the fixup >> _after_ the optimization.[^3] > > LGTM. I did have the same idea when added specialized 8-bit opcodes. > > Some optimization (like constant folding) it is worth to move yet one > step earlier, to AST. Definitely. And if we could only move _all_ optimization earlier, that would make this change unnecessary. But, sadly, I don't think we can; there are some steps the peephole optimizer does that have to stay bytecode-based, and post-PEP 511, there will always be new ideas people come up with that also have to stay bytecode-based.[^1] > Other idea - instead of EXTENDED_ARG have two sets of instructions: > short 16-bit with 8-bit arg, and long 32-bit with 24-bit arg. For > simplicity initially only long instructions are emitted (if it makes > sense). After optimization the code is packed using short instructions > if possible (now we have all needed information and can do this in one > pass). At the same time NOPs are removed. For later stages we easy can > unpack the code back to long form. I like that. Unpacking/packing is a lot simpler, and less to explain, than exposing an array of structs, and we can even expose those functions to Python for bytecode decorators (maybe hidden in the dis module?), which would be a lot simpler than everything byteplay does, and and 90% of the time it would be all you need. I also like that you can cram NOP removal into the packing step, so optimizers that just remove code don't have to actually remove or renumber anything; only those that insert or move code have to worry about that. The CPU cost of going from list of arrays of instructions to unpacked opcode array to packed opcode array will be a little higher than going straight from list of arrays of instructions to packed opcode array, but I'll bet it will be negligible. The only downside I can think of is that arguments are now restricted to 2**31 (or signed 32-bit), and this would restrict them to 2**23. Has anyone ever had anything close to 4 million instructions (or consts, or whatever), in a code object? I doubt it. But why not make it even simpler and just have all unpacked instructions be 32-bit? Sure, that means unpacked code arrays are bigger, but it's not like the optimizers are going to be looping over the same array a zillion times and worrying about cache spill (or that the optimizers will be in a hotspot in the first place). Then we've just got an int32*, and a jump to offset 76 is a jump to the 4 bytes at bytecode[76] (or, in Python, where we may still have to use a bytes object, it's at worst a jump to bytecode[76<<2]). Anyway, this seems doable, and worth doing, independent of the wordcode idea, the packed-args idea you're working on, AST-ifying most of the peephole optimizer, PEP 511, or anything else, so if no comes up with any objections, I'll add a tracker issue for it and work on this before going back to wordcode this weekend. [^1]: For example, Victor's const optimization can only be done at the AST level by adding a new kind of AST node, and compiler support for it. That's obviously not something a plug-in optimizer could do, so it would instead have to be written as a bytecode optimizer. From python at mrabarnett.plus.com Fri Feb 5 16:10:11 2016 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 05 Feb 2016 21:10:11 +0000 Subject: [Python-ideas] Exposing flat bytecode representation to optimizers In-Reply-To: <1763502835.2002313.1454702338876.JavaMail.yahoo@mail.yahoo.com> Message-ID: On 2016-02-05 19:58:58, "Andrew Barnert via Python-ideas" wrote: >The biggest pain with dealing with the peephole optimizer is that it >happens after all the complicated flattening and fixup[^1] the compiler >does, which means you have to hack up the jump targets as you go along. >The future bytecode optimizers that PEP 511 enables will have the same >headache. > >But this isn't actually necessary. The optimizer could work on a flat >array of instructions[^2] instead of an array of bytes, with >relocatable jump targets instead of fixed byte offsets, and then the >compiler could do the fixup _after_ the optimization.[^3] > >It would break the optimizer APIs, but `PyCode_Optimize` isn't public, >and the API proposed by PEP 511 is public, but that PEP isn't even >finalized, much less accepted yet. > > >I don't think we need to expose the intermediate representation any >farther along than the `PyCode_Optimize` step.[^4] Just moving the >optimize one step earlier in the chain solves more than enough to be >worth it. > > > [^1]: If you think numbering the offsets and emitting the jump >targets is easy: Every time you fix up a jump, that may require adding >an `EXTENDED_ARG`, which means you have to redo any fixups that cross >the the current instruction. The compiler does this by just looping >until it hasn't added any more `EXTENDED_ARG`s. > > [^2]: In case anyone's thinking that wordcode would solve this >problem, it doesn't. The `EXTENDED_ARG` jump targets are a much bigger >hassle than the 1-or-3-byte-ops, and wordcode not only can't eliminate >those, but makes `EXTENDED_ARG` more common. > > [^3]: The compiler doesn't actually have exactly what the optimizers >would want, but it's pretty close: it has a linked list of block >objects, each of which has an array of instruction objects, with jump >targets being pointers to blocks. That's actually even better to work >with, but too complicated to expose to optimizers. Flattening it would >be trivial. Or, if that's too expensive, we could do something almost >as simple and much cheaper: convert it in-line to a deque-like linked >list of arrays, with jump targets being indices or pointers into that. >Or we could just expose the list of blocks as-is, as an opaque thing >with a mutable-deque-of-instructions API around it. > > [^4]: Later stages--import hooks, optimizing decorators, etc.--have >the same pain as the peephole optimizer, but they also have code >objects that contain other code objects, and they can be written in >Python, and so on, so the same solution isn't feasible there. Of course >we could add a function to convert bytecode back to a list of >instructions, in some format that can be exposed to Python (in fact, >`dis` already does 90% of that), and then another one to convert that >back to bytecode and redo the fixup (which is basically the third-party >`byteplay` module). But that's almost certainly overkill. (If we wanted >that, I'd rather add `byteplay` to the stdlib, port it and `dis` to C, >and expose a C API for them. And then we could use that for everything, >including the peephole optimizer and PEP 511 optimizers. Which would >all be really cool, but I think it's more work than we want to do, and >I don't know if we'd actually want something like `byteplay` builtin >even if it were easy...) > What I've done in similar situations is have jumps refer to labels, which are also instructions in the code. There would then be a pass to copy the code, recording the position of each label, but not copying it, followed by a pass to change the jumps to refer to their target's position. (The jumps were fixed-size, though.) From lkb.teichmann at gmail.com Fri Feb 5 16:20:46 2016 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Fri, 5 Feb 2016 22:20:46 +0100 Subject: [Python-ideas] Simpler Customization of Class Creation - PEP 487 Message-ID: Hi List, about a year ago I started a discussion on how to simplify metaclasses, which led to PEP 487. I got some good ideas from this list, but couldn't follow up on this because I was bound in other projects. In short, metaclasses are often not used as they are considered very complicated. Indeed they are, especially if you need to use two of them at the same time in a multiple inheritance context. Most metaclasses, however, serve only some of the following three purposes: a) run some code after a class is created b) initialize descriptors of a class or c) keep the order in which class attributes have been defined. PEP 487 now proposes to put a metaclass into the standard library, which can be used for all those three purposes. If now libraries start to use this metaclass, we won't need any metaclass mixing anymore. What has changed since the last time I posted PEP 487? Firstly, I re-wrote large parts of the PEP to make it easier to read. Those who liked the old text, that's still existing in PEP 422. Secondly, I modified the proposal following suggestions from this list: I added the descriptor initialization (purpose b)), as this was considered particularly useful, even if it could in principle be done using purpose a) from above. The order-keeping of the class attributes is the leftover from a much more ambitious previous idea that would have allowed for custom namespaces during class creation. But this additional feature would have rendered the most common usecase - getting the order of attributes - much more complicated, so I opted for usability over flexibility. I have put the new version of the PEP here: https://github.com/tecki/metaclasses/blob/pep487/pep-0487.txt and also added it to this posting. An implementation of this PEP can be found at: https://pypi.python.org/pypi/metaclass Greetings Martin PEP: 487 Title: Simpler customisation of class creation Version: $Revision$ Last-Modified: $Date$ Author: Martin Teichmann , Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 27-Feb-2015 Python-Version: 3.6 Post-History: 27-Feb-2015, 5-Feb-2016 Replaces: 422 Abstract ======== Currently, customising class creation requires the use of a custom metaclass. This custom metaclass then persists for the entire lifecycle of the class, creating the potential for spurious metaclass conflicts. This PEP proposes to instead support a wide range of customisation scenarios through a new ``__init_subclass__`` hook in the class body, a hook to initialize descriptors, and a way to keep the order in which attributes are defined. Those hooks should at first be defined in a metaclass in the standard library, with the option that this metaclass eventually becomes the default ``type`` metaclass. The new mechanism should be easier to understand and use than implementing a custom metaclass, and thus should provide a gentler introduction to the full power Python's metaclass machinery. Background ========== Metaclasses are a powerful tool to customize class creation. They have, however, the problem that there is no automatic way to combine metaclasses. If one wants to use two metaclasses for a class, a new metaclass combining those two needs to be created, typically manually. This need often occurs as a surprise to a user: inheriting from two base classes coming from two different libraries suddenly raises the necessity to manually create a combined metaclass, where typically one is not interested in those details about the libraries at all. This becomes even worse if one library starts to make use of a metaclass which it has not done before. While the library itself continues to work perfectly, suddenly every code combining those classes with classes from another library fails. Proposal ======== While there are many possible ways to use a metaclass, the vast majority of use cases falls into just three categories: some initialization code running after class creation, the initalization of descriptors and keeping the order in which class attributes were defined. Those three use cases can easily be performed by just one metaclass. If this metaclass is put into the standard library, and all libraries that wish to customize class creation use this very metaclass, no combination of metaclasses is necessary anymore. The three use cases are achieved as follows: 1. The metaclass contains an ``__init_subclass__`` hook that initializes all subclasses of a given class, 2. the metaclass calls an ``__init_descriptor__`` hook for all descriptors defined in the class, and 3. an ``__attribute_order__`` tuple is left in the class in order to inspect the order in which attributes were defined. For ease of use, a base class ``SubclassInit`` is defined, which uses said metaclass and contains an empty stub for the hook described for use case 1. As an example, the first use case looks as follows:: class SpamBase(SubclassInit): # this is implicitly a @classmethod def __init_subclass__(cls, **kwargs): # This is invoked after a subclass is created, but before # explicit decorators are called. # The usual super() mechanisms are used to correctly support # multiple inheritance. # **kwargs are the keyword arguments to the subclasses' # class creation statement super().__init_subclass__(cls, **kwargs) class Spam(SpamBase): pass # the new hook is called on Spam The base class ``SubclassInit`` contains an empty ``__init_subclass__`` method which serves as an endpoint for cooperative multiple inheritance. Note that this method has no keyword arguments, meaning that all methods which are more specialized have to process all keyword arguments. This general proposal is not a new idea (it was first suggested for inclusion in the language definition `more than 10 years ago`_, and a similar mechanism has long been supported by `Zope's ExtensionClass`_), but the situation has changed sufficiently in recent years that the idea is worth reconsidering for inclusion. The second part of the proposal adds an ``__init_descriptor__`` initializer for descriptors. Descriptors are defined in the body of a class, but they do not know anything about that class, they do not even know the name they are accessed with. They do get to know their owner once ``__get__`` is called, but still they do not know their name. This is unfortunate, for example they cannot put their associated value into their object's ``__dict__`` under their name, since they do not know that name. This problem has been solved many times, and is one of the most important reasons to have a metaclass in a library. While it would be easy to implement such a mechanism using the first part of the proposal, it makes sense to have one solution for this problem for everyone. To give an example of its usage, imagine a descriptor representing weak referenced values (this is an insanely simplified, yet working example):: import weakref class WeakAttribute: def __get__(self, instance, owner): return instance.__dict__[self.name] def __set__(self, instance, value): instance.__dict__[self.name] = weakref.ref(value) # this is the new initializer: def __init_descriptor__(self, owner, name): self.name = name The third part of the proposal is to leave a tuple called ``__attribute_order__`` in the class that contains the order in which the attributes were defined. This is a very common usecase, many libraries use an ``OrderedDict`` to store this order. This is a very simple way to achieve the same goal. Key Benefits ============ Easier inheritance of definition time behaviour ----------------------------------------------- Understanding Python's metaclasses requires a deep understanding of the type system and the class construction process. This is legitimately seen as challenging, due to the need to keep multiple moving parts (the code, the metaclass hint, the actual metaclass, the class object, instances of the class object) clearly distinct in your mind. Even when you know the rules, it's still easy to make a mistake if you're not being extremely careful. Understanding the proposed implicit class initialization hook only requires ordinary method inheritance, which isn't quite as daunting a task. The new hook provides a more gradual path towards understanding all of the phases involved in the class definition process. Reduced chance of metaclass conflicts ------------------------------------- One of the big issues that makes library authors reluctant to use metaclasses (even when they would be appropriate) is the risk of metaclass conflicts. These occur whenever two unrelated metaclasses are used by the desired parents of a class definition. This risk also makes it very difficult to *add* a metaclass to a class that has previously been published without one. By contrast, adding an ``__init_subclass__`` method to an existing type poses a similar level of risk to adding an ``__init__`` method: technically, there is a risk of breaking poorly implemented subclasses, but when that occurs, it is recognised as a bug in the subclass rather than the library author breaching backwards compatibility guarantees. A path of introduction into Python ================================== Most of the benefits of this PEP can already be implemented using a simple metaclass. For the ``__init_subclass__`` hook this works all the way down to Python 2.7, while the attribute order needs Python 3.0 to work. Such a class has been `uploaded to PyPI`_. The only drawback of such a metaclass are the mentioned problems with metaclasses and multiple inheritance. Two classes using such a metaclass can only be combined, if they use exactly the same such metaclass. This fact calls for the inclusion of such a class into the standard library, let's call it ``SubclassMeta``, with the base class using it called ``SubclassInit``. Once all users use this standard library metaclass, classes from different packages can easily be combined. But still such classes cannot be easily combined with other classes using other metaclasses. Authors of metaclasses should bear that in mind and inherit from the standard metaclass if it seems useful for users of the metaclass to add more functionality. Ultimately, if the need for combining with other metaclasses is strong enough, the proposed functionality may be introduced into Python's ``type``. Those arguments strongly hint to the following procedure to include the proposed functionality into Python: 1. The metaclass implementing this proposal is put onto PyPI, so that it can be used and scrutinized. 2. Once the code is properly mature, it can be added to the Python standard library. There should be a new module called ``metaclass`` which collects tools for metaclass authors, as well as a documentation of the best practices of how to write metaclasses. 3. If the need of combining this metaclass with other metaclasses is strong enough, it may be included into Python itself. While the metaclass is still in the standard library and not in the language, it may still clash with other metaclasses. The most prominent metaclass in use is probably ABCMeta. It is also a particularly good example for the need of combining metaclasses. For users who want to define a ABC with subclass initialization, we should support a ``ABCSubclassInit`` class, or let ABCMeta inherit from this PEP's metaclass. Extensions written in C or C++ also often define their own metaclass. It would be very useful if those could also inherit from the metaclass defined here, but this is probably not possible. New Ways of Using Classes ========================= This proposal has many usecases like the following. In the examples, we still inherit from the ``SubclassInit`` base class. This would become unnecessary once this PEP is included in Python directly. Subclass registration --------------------- Especially when writing a plugin system, one likes to register new subclasses of a plugin baseclass. This can be done as follows:: class PluginBase(SubclassInit): subclasses = [] def __init_subclass__(cls, **kwargs): super().__init_subclass__(**kwargs) cls.subclasses.append(cls) One should note that this also works nicely as a mixin class. Trait descriptors ----------------- There are many designs of Python descriptors in the wild which, for example, check boundaries of values. Often those "traits" need some support of a metaclass to work. This is how this would look like with this PEP:: class Trait: def __get__(self, instance, owner): return instance.__dict__[self.key] def __set__(self, instance, value): instance.__dict__[self.key] = value def __init_descriptor__(self, owner, name): self.key = name class Int(Trait): def __set__(self, instance, value): # some boundary check code here super().__set__(instance, value) Rejected Design Options ======================= Calling the hook on the class itself ------------------------------------ Adding an ``__autodecorate__`` hook that would be called on the class itself was the proposed idea of PEP 422. Most examples work the same way or even better if the hook is called on the subclass. In general, it is much easier to explicitly call the hook on the class in which it is defined (to opt-in to such a behavior) than to opt-out, meaning that one does not want the hook to be called on the class it is defined in. This becomes most evident if the class in question is designed as a mixin: it is very unlikely that the code of the mixin is to be executed for the mixin class itself, as it is not supposed to be a complete class on its own. The original proposal also made major changes in the class initialization process, rendering it impossible to back-port the proposal to older Python versions. Other variants of calling the hook ---------------------------------- Other names for the hook were presented, namely ``__decorate__`` or ``__autodecorate__``. This proposal opts for ``__init_subclass__`` as it is very close to the ``__init__`` method, just for the subclass, while it is not very close to decorators, as it does not return the class. Requiring an explicit decorator on ``__init_subclass__`` -------------------------------------------------------- One could require the explicit use of ``@classmethod`` on the ``__init_subclass__`` decorator. It was made implicit since there's no sensible interpretation for leaving it out, and that case would need to be detected anyway in order to give a useful error message. This decision was reinforced after noticing that the user experience of defining ``__prepare__`` and forgetting the ``@classmethod`` method decorator is singularly incomprehensible (particularly since PEP 3115 documents it as an ordinary method, and the current documentation doesn't explicitly say anything one way or the other). Defining arbitrary namespaces ----------------------------- PEP 422 defined a generic way to add arbitrary namespaces for class definitions. This approach is much more flexible than just leaving the definition order in a tuple. The ``__prepare__`` method in a metaclass supports exactly this behavior. But given that effectively the only use cases that could be found out in the wild were the ``OrderedDict`` way of determining the attribute order, it seemed reasonable to only support this special case. The metaclass described in this PEP has been designed to be very simple such that it could be reasonably made the default metaclass. This was especially important when designing the attribute order functionality: This was a highly demanded feature and has been enabled through the ``__prepare__`` method of metaclasses. This method can be abused in very weird ways, making it hard to correctly maintain this feature in CPython. This is why it has been proposed to deprecated this feature, and instead use ``OrderedDict`` as the standard namespace, supporting the most important feature while dropping most of the complexity. But this would have meant that ``OrderedDict`` becomes a language builtin like dict and set, and not just a standard library class. The choice of the ``__attribute_order__`` tuple is a much simpler solution to the problem. A more ``__new__``-like hook ---------------------------- In PEP 422 the hook worked more like the ``__new__`` method than the ``__init__`` method, meaning that it returned a class instead of modifying one. This allows a bit more flexibility, but at the cost of much harder implementation and undesired side effects. History ======= This used to be a competing proposal to PEP 422 by Nick Coughlan and Daniel Urban. It shares both most of the PEP text and proposed code, but has major differences in how to achieve its goals. In the meantime, PEP 422 has been withdrawn favouring this approach. References ========== .. _published code: http://mail.python.org/pipermail/python-dev/2012-June/119878.html .. _more than 10 years ago: http://mail.python.org/pipermail/python-dev/2001-November/018651.html .. _Zope's ExtensionClass: http://docs.zope.org/zope_secrets/extensionclass.html .. _uploaded to PyPI: https://pypi.python.org/pypi/metaclass Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From abarnert at yahoo.com Fri Feb 5 16:25:29 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 5 Feb 2016 21:25:29 +0000 (UTC) Subject: [Python-ideas] Exposing flat bytecode representation to optimizers In-Reply-To: References: Message-ID: <967642016.2011381.1454707529104.JavaMail.yahoo@mail.yahoo.com> On Friday, February 5, 2016 1:10 PM, MRAB wrote: > On 2016-02-05 19:58:58, "Andrew Barnert via Python-ideas" > wrote: > >> The biggest pain with dealing with the peephole optimizer is that it >> happens after all the complicated flattening and fixup[^1] the compiler >> does, which means you have to hack up the jump targets as you go along. >> The future bytecode optimizers that PEP 511 enables will have the same >> headache. >> >> But this isn't actually necessary. The optimizer could work on a flat >> array of instructions[^2] instead of an array of bytes, with >> relocatable jump targets instead of fixed byte offsets, and then the >> compiler could do the fixup _after_ the optimization.[^3] > What I've done in similar situations is have jumps refer to labels, > which are also instructions in the code. > > There would then be a pass to copy the code, recording the position of > each label, but not copying it, followed by a pass to change the jumps > to refer to their target's position. (The jumps were fixed-size, > though.) Yes, that's basically what byteplay does, and effectively what the compiler is doing by pointing at blocks instead of instructions (because Python has no way to jump to anything that isn't the start or end of a block). But that still doesn't make the fixups trivial, unless either (a) you never insert, delete, or move code so the jumps never change size, or (b) you're willing to punt and revert optimizing any time you push an arg into EXTENDED_ARG territory (which is reasonable for experimentation, but not for something in the core, or that you intend to share and deploy widely). Making each optimizer do the same work seems kind of silly. Many of them will punt on uncommon (but not _that_ rare) cases--or, worse, will get those cases wrong and never test them. Plus, with PEP 511, it means if you have 6 optimizers plugged in, you're doing these extra passes 6 times instead of once. We _could_ expose a higher-level label-based representation to the optimizers rather than making them all do it themselves. However, I think the unpack/pack that Serhiy suggested is simpler, and good enough. You _do_ still need fixups if you insert or move code (but not if you delete code, since his suggested pack does NOP removal), but those fixups will be so easy that you won't need labels unless you're doing some really complicated reordering. From abarnert at yahoo.com Fri Feb 5 16:53:13 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 5 Feb 2016 21:53:13 +0000 (UTC) Subject: [Python-ideas] Simpler Customization of Class Creation - PEP 487 In-Reply-To: References: Message-ID: <1814585983.22893.1454709193740.JavaMail.yahoo@mail.yahoo.com> Two small things. On Friday, February 5, 2016 1:21 PM, Martin Teichmann wrote: > Most metaclasses, however, serve only some of the following three > purposes: a) run some code after a class is created b) initialize descriptors > of a class or c) keep the order in which class attributes have been defined. How is c) not served by just defining a __prepare__ method that returns an OrderedDict? That seems easier to use than a separate dict and __attribute_order__ tuple. You later mention that many people already do this, but don't explain what's wrong with it or why your idea is better. It's not like a one-line __prepare__ is difficult to write, or like OrderedDict is too slow (especially in 3.6), so you're just adding a cost (less convenient to use) for no benefit that I can see. Also: > A path of introduction into Python > ================================== > > Most of the benefits of this PEP can already be implemented using > a simple metaclass. For the ``__init_subclass__`` hook this works > all the way down to Python 2.7, while the attribute order needs Python 3.0 > to work. Such a class has been `uploaded to PyPI`_. > > The only drawback of such a metaclass are the mentioned problems with > metaclasses and multiple inheritance. It sounds like you're trying to have it both ways here: it seems like if your proposal really is better because it works with Python 2.7 rather than just 3.x, then your proposal being in the stdlib is probably a bad idea, because it will mislead rather than help people trying to write version-straddling code. From yselivanov.ml at gmail.com Fri Feb 5 17:26:02 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 5 Feb 2016 17:26:02 -0500 Subject: [Python-ideas] Exposing flat bytecode representation to optimizers In-Reply-To: References: <1763502835.2002313.1454702338876.JavaMail.yahoo.ref@mail.yahoo.com> <1763502835.2002313.1454702338876.JavaMail.yahoo@mail.yahoo.com> Message-ID: <56B5217A.6060501@gmail.com> On 2016-02-05 3:35 PM, Serhiy Storchaka wrote: >> >> I don't think we need to expose the intermediate representation any >> farther along than the `PyCode_Optimize` step.[^4] Just moving the >> optimize one step earlier in the chain solves more than enough to be >> worth it. > > LGTM. I did have the same idea when added specialized 8-bit opcodes. Agree. > > Some optimization (like constant folding) it is worth to move yet one > step earlier, to AST. > > Other idea - instead of EXTENDED_ARG have two sets of instructions: > short 16-bit with 8-bit arg, and long 32-bit with 24-bit arg. For > simplicity initially only long instructions are emitted (if it makes > sense). After optimization the code is packed using short instructions > if possible (now we have all needed information and can do this in one > pass). At the same time NOPs are removed. For later stages we easy can > unpack the code back to long form. I also had this idea (don't know if it's good or not): 1. have 16 bits per opcode 2. first 8 bits encode the opcode number 3. next 7 bits encode the arg (most args don't need more than 7 bits anyways) 4. if the 16th bit is 1 then the next opcode is the EXTENDED_ARG We can then encode EXTENDED_ARG to follow the same format -- first 15 bits are for the arg, the last one is to add one more EXTENDED_ARG. Yury From yselivanov.ml at gmail.com Fri Feb 5 17:46:31 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 5 Feb 2016 17:46:31 -0500 Subject: [Python-ideas] More compact bytecode In-Reply-To: References: Message-ID: <56B52647.2060403@gmail.com> Serhiy, Victor, Andrew, What do you think about turning constant slice objects into actual constants? Right now: def a(): return a[:1] dis.dis(a) 1 0 LOAD_GLOBAL 0 (a) 3 LOAD_CONST 0 (None) 6 LOAD_CONST 1 (1) 9 BUILD_SLICE 2 12 BINARY_SUBSCR 13 RETURN_VALUE With this optimization we could just have: 1 0 LOAD_GLOBAL 0 (a) 3 LOAD_CONST 0 (:1) 6 BINARY_SUBSCR 7 RETURN_VALUE Yury From abarnert at yahoo.com Fri Feb 5 18:14:25 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 5 Feb 2016 23:14:25 +0000 (UTC) Subject: [Python-ideas] Exposing flat bytecode representation to optimizers In-Reply-To: <56B5217A.6060501@gmail.com> References: <56B5217A.6060501@gmail.com> Message-ID: <1112741469.61236.1454714065734.JavaMail.yahoo@mail.yahoo.com> On Friday, February 5, 2016 2:26 PM, Yury Selivanov wrote: > > I also had this idea (don't know if it's good or not): > > 1. have 16 bits per opcode > 2. first 8 bits encode the opcode number > 3. next 7 bits encode the arg (most args don't need more than 7 bits > anyways) > 4. if the 16th bit is 1 then the next opcode is the EXTENDED_ARG Why? The advantage of Serhiy's idea (or my original one) is that it makes it much easier to write bytecode processors (like the peephole optimizer) when everything is fixed width, because jump target and lnotab fixups become trivial. Your idea leaves jump target and lnotab fixups just as hard as they are today, so we get the cost of having to transform back and forth between two representations, without any benefit. If you were instead suggesting that as an alternative to the version of wordcode I proposed and prototyped in the other thread, to be actually used by the interpreter, then there _is_ a small benefit to your version: arguments in range [2**17, 2**23) need 4 bytes instead of 6. But I think the cost of having arguments in range [2**7, 2**8) take 4 bytes instead of 2, and the extra complexity and CPU cost in the fetch and peek at the core of the eval loop, and being a bigger change from today, make it a lot less attractive. From victor.stinner at gmail.com Fri Feb 5 18:30:09 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 6 Feb 2016 00:30:09 +0100 Subject: [Python-ideas] More compact bytecode In-Reply-To: <56B52647.2060403@gmail.com> References: <56B52647.2060403@gmail.com> Message-ID: 2016-02-05 23:46 GMT+01:00 Yury Selivanov : > What do you think about turning constant slice objects into actual > constants? It doesn't look interesting. bench$ python3 -m timeit -s 'def f(list):' -s ' list[:1]; list[:1]; list[:1]; list[:1]; list[:1]' -s 'l=list("hello")' 'f(l)' 1000000 loops, best of 3: 1.34 usec per loop bench$ python3 -m timeit -s 'def f(list):' -s ' list["whatever"]; list["whatever"]; list["whatever"]; list["whatever"]; list["whatever"]' -s 'import types; co=f.__code__; consts=[slice(0, 1, None) if const == "whatever" else const for const in co.co_consts]; f.__code__ = types.CodeType(co.co_argcount, co.co_kwonlyargcount, co.co_nlocals, co.co_stacksize, co.co_flags, co.co_code, tuple(consts), co.co_names, co.co_varnames, co.co_filename, co.co_name, co.co_firstlineno, co.co_lnotab, co.co_freevars, co.co_cellvars); l=list("hello")' 'f(l)' 1000000 loops, best of 3: 1.3 usec per loop It's only 3% faster on a microbenchmark. (On my laptop: 687 ns => 658 ns: 4% faster) I'm not sure that the benchmark is "fair" since the slice object has a free list of 1 item which is probably always used in this benchmark. But you will be able to easily implement suck "hack" in fatoptimizer which already has similar optimizations. IMHO it's ok for an optimizer like fatoptimizer, it's not worth for the default builtin optimizer in CPython. Victor From yselivanov.ml at gmail.com Fri Feb 5 18:25:40 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 5 Feb 2016 18:25:40 -0500 Subject: [Python-ideas] Exposing flat bytecode representation to optimizers In-Reply-To: <1112741469.61236.1454714065734.JavaMail.yahoo@mail.yahoo.com> References: <56B5217A.6060501@gmail.com> <1112741469.61236.1454714065734.JavaMail.yahoo@mail.yahoo.com> Message-ID: <56B52F74.6090201@gmail.com> On 2016-02-05 6:14 PM, Andrew Barnert wrote: > On Friday, February 5, 2016 2:26 PM, Yury Selivanov wrote: > >> I also had this idea (don't know if it's good or not): >> >> 1. have 16 bits per opcode >> 2. first 8 bits encode the opcode number >> 3. next 7 bits encode the arg (most args don't need more than 7 bits >> anyways) >> 4. if the 16th bit is 1 then the next opcode is the EXTENDED_ARG > > Why? The advantage of Serhiy's idea (or my original one) is that it makes it much easier to write bytecode processors (like the peephole optimizer) when everything is fixed width, because jump target and lnotab fixups become trivial. Your idea leaves jump target and lnotab fixups just as hard as they are today, so we get the cost of having to transform back and forth between two representations, without any benefit. Yes, this was related to how Serhiy original proposal on how we can pack opcodes. It's completely unrelated to the peephole optimizer. Sorry for the confusion. > > If you were instead suggesting that as an alternative to the version of wordcode I proposed and prototyped in the other thread, to be actually used by the interpreter, then there _is_ a small benefit to your version: arguments in range [2**17, 2**23) need 4 bytes instead of 6. But I think the cost of having arguments in range [2**7, 2**8) take 4 bytes instead of 2, and the extra complexity and CPU cost in the fetch and peek at the core of the eval loop, and being a bigger change from today, make it a lot less attractive. I'm sorry, I can't parse what you said there... My idea is based on the fact that most opcode arguments are in [0, 127] range. EXTENDED_ARG isn't used too often, but when it is used, it will be packed more efficiently in my scheme. Yury From victor.stinner at gmail.com Fri Feb 5 18:54:56 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 6 Feb 2016 00:54:56 +0100 Subject: [Python-ideas] Exposing flat bytecode representation to optimizers In-Reply-To: <1763502835.2002313.1454702338876.JavaMail.yahoo@mail.yahoo.com> References: <1763502835.2002313.1454702338876.JavaMail.yahoo.ref@mail.yahoo.com> <1763502835.2002313.1454702338876.JavaMail.yahoo@mail.yahoo.com> Message-ID: 2016-02-05 20:58 GMT+01:00 Andrew Barnert via Python-ideas : > It would break the optimizer APIs, but `PyCode_Optimize` isn't public Sadly, PyCode_Optimize() is public and part of the stable ABI. But I'm ok to take the responsability of breaking it if we implement the PEP 511 since we will probably have a much better API to optimize AST and bytecode, and I would prefer that no one directly uses PyCode_Optimize()! > and the API proposed by PEP 511 is public, but that PEP isn't even finalized, much less accepted yet. To be clear: the PEP 511 is a draft still stuck in python-ideas, it's not ready for a review on python-dev. I still expect changes on the API. I'm very interested by your feedback on the bytecode transformer API :-) > [^3]: The compiler doesn't actually have exactly what the optimizers would want, but it's pretty close: it has a linked list of block objects, each of which has an array of instruction objects, with jump targets being pointers to blocks. That's actually even better to work with, but too complicated to expose to optimizers. The current implementation of the peephole optimizer has a complex code to update jumps. IMHO it would be simpler if the code would be blocks of instructions. My plan is to implement most implementations at AST level and only optimizer jumps at "bytecode" level. So I don't need much informations on instructions, just a structure easy to manipulate to optimize jumps. For my old "registervm" project, I also wrote my own API to have a higher level structure for bytecode: https://hg.python.org/sandbox/registervm/file/tip/Lib/registervm.py * Block: list of sequential instructions. Blocks are linked together by jumps (and conditional jumps) * Instruction * Instruction argument: Immediate, Immediate8 (8 bits), Label (for jumps), Register (well, something specific to my register-based instructions), Local (local variable) Basically, the code is a list of blocks where a block is a list of instructions + arguments (in registervm, an instruction can have many arguments, not only one). In registervm, the Instruction class stores directly arguments, but that's more an implementation detail ;-) Do you know if byteplay and other projects have a similar design? It would be nice to find the root common structure to be a generic API. > Flattening it would be trivial. Or, if that's too expensive, we could do something almost as simple and much cheaper: convert it in-line to a deque-like linked list of arrays, with jump targets being indices or pointers into that. Or we could just expose the list of blocks as-is, as an opaque thing with a mutable-deque-of-instructions API around it. For the PEP 511 we can imagine multiple levels for code tranformers: block of instructions, flatten bytecode, AST, etc. The cost of the transformation is to convert internal from CPython objects to Python objects, and then convert the result of the code transformer back to internal CPython objects. For AST, I expect the cost of the double conversions to be non-negligible since AST uses a lot of small objects. Hopefully, no conversion is needed if no code transformer is registered. Victor From abarnert at yahoo.com Fri Feb 5 18:58:06 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 5 Feb 2016 15:58:06 -0800 Subject: [Python-ideas] Exposing flat bytecode representation to optimizers In-Reply-To: <56B52F74.6090201@gmail.com> References: <56B5217A.6060501@gmail.com> <1112741469.61236.1454714065734.JavaMail.yahoo@mail.yahoo.com> <56B52F74.6090201@gmail.com> Message-ID: <55DD0494-DF2F-4F32-8D63-E248A37096A4@yahoo.com> On Feb 5, 2016, at 15:25, Yury Selivanov wrote: > > > >> On 2016-02-05 6:14 PM, Andrew Barnert wrote: >>> On Friday, February 5, 2016 2:26 PM, Yury Selivanov wrote: >>> >>> I also had this idea (don't know if it's good or not): >>> >>> 1. have 16 bits per opcode >>> 2. first 8 bits encode the opcode number >>> 3. next 7 bits encode the arg (most args don't need more than 7 bits >>> anyways) >>> 4. if the 16th bit is 1 then the next opcode is the EXTENDED_ARG >> >> Why? The advantage of Serhiy's idea (or my original one) is that it makes it much easier to write bytecode processors (like the peephole optimizer) when everything is fixed width, because jump target and lnotab fixups become trivial. Your idea leaves jump target and lnotab fixups just as hard as they are today, so we get the cost of having to transform back and forth between two representations, without any benefit. > > Yes, this was related to how Serhiy original proposal on how we can pack opcodes. It's completely unrelated to the peephole optimizer. Sorry for the confusion. OK, we're on the wrong thread, then, but no big deal. >> If you were instead suggesting that as an alternative to the version of wordcode I proposed and prototyped in the other thread, to be actually used by the interpreter, then there _is_ a small benefit to your version: arguments in range [2**17, 2**23) need 4 bytes instead of 6. But I think the cost of having arguments in range [2**7, 2**8) take 4 bytes instead of 2, and the extra complexity and CPU cost in the fetch and peek at the core of the eval loop, and being a bigger change from today, make it a lot less attractive. > > I'm sorry, I can't parse what you said there... OK, let's compare my idea, from the thread you meant to reply to, and your idea. I use 16-bit opcodes, with 8 bits for the op and 8 for the argument. I use the existing EXTENDED_ARG mechanism to handle values that don't fit in 8 bits (each 16-bit EXTENDED_ARGS gets you another 8 bits). You use 16-bit opcodes, with only 8 bits for the op, 7 for the argument, and 1 for a continuation bit. By using the continuation bit instead of existing EXTENDED_ARGs, you can fit 22 bits worth of argument in 4 bytes, while I can only fit 16 bits worth. So, your idea uses fewer bytes than mine (4 instead of 6) for any argument in range [65536, 4194303]. But it uses more bytes than mine (4 instead of 2) for any argument in range [128, 255]. Although [128, 255] is a smaller range than [65536, 4194303], it's much more common in arguments, so I think your idea will save less space than mine overall. Your idea is also more complicated conceptually, would require more changes to the interpreter (and to third-party code like byteplay or decompyle), and will probably be slower in the eval loop, which also all make it less attractive. As for the idea Serhiy is working on, I suspect it will save more space than either yours or mine, but mine might have performance and/or simplicity benefits that make it a better choice. (Hopefully we'll both get far enough that we can prototype it and see.) He's stealing extra bits from the opcode byte to allow skipping the argument entirely in many cases (e.g., LOAD_CONST_0 would be a no-argument opcode that does the same thing as LOAD_CONST with argument 0). So many common opcodes will be only a single byte. From yselivanov.ml at gmail.com Fri Feb 5 19:05:59 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 5 Feb 2016 19:05:59 -0500 Subject: [Python-ideas] Exposing flat bytecode representation to optimizers In-Reply-To: <55DD0494-DF2F-4F32-8D63-E248A37096A4@yahoo.com> References: <56B5217A.6060501@gmail.com> <1112741469.61236.1454714065734.JavaMail.yahoo@mail.yahoo.com> <56B52F74.6090201@gmail.com> <55DD0494-DF2F-4F32-8D63-E248A37096A4@yahoo.com> Message-ID: <56B538E7.3070604@gmail.com> On 2016-02-05 6:58 PM, Andrew Barnert wrote: > On Feb 5, 2016, at 15:25, Yury Selivanov wrote: >> >> >>> On 2016-02-05 6:14 PM, Andrew Barnert wrote: >>>> On Friday, February 5, 2016 2:26 PM, Yury Selivanov wrote: >>>> >>>> I also had this idea (don't know if it's good or not): >>>> >>>> 1. have 16 bits per opcode >>>> 2. first 8 bits encode the opcode number >>>> 3. next 7 bits encode the arg (most args don't need more than 7 bits >>>> anyways) >>>> 4. if the 16th bit is 1 then the next opcode is the EXTENDED_ARG >>> Why? The advantage of Serhiy's idea (or my original one) is that it makes it much easier to write bytecode processors (like the peephole optimizer) when everything is fixed width, because jump target and lnotab fixups become trivial. Your idea leaves jump target and lnotab fixups just as hard as they are today, so we get the cost of having to transform back and forth between two representations, without any benefit. >> Yes, this was related to how Serhiy original proposal on how we can pack opcodes. It's completely unrelated to the peephole optimizer. Sorry for the confusion. > OK, we're on the wrong thread, then, but no big deal. > >>> If you were instead suggesting that as an alternative to the version of wordcode I proposed and prototyped in the other thread, to be actually used by the interpreter, then there _is_ a small benefit to your version: arguments in range [2**17, 2**23) need 4 bytes instead of 6. But I think the cost of having arguments in range [2**7, 2**8) take 4 bytes instead of 2, and the extra complexity and CPU cost in the fetch and peek at the core of the eval loop, and being a bigger change from today, make it a lot less attractive. >> I'm sorry, I can't parse what you said there... > OK, let's compare my idea, from the thread you meant to reply to, and your idea. > > I use 16-bit opcodes, with 8 bits for the op and 8 for the argument. I use the existing EXTENDED_ARG mechanism to handle values that don't fit in 8 bits (each 16-bit EXTENDED_ARGS gets you another 8 bits). > > You use 16-bit opcodes, with only 8 bits for the op, 7 for the argument, and 1 for a continuation bit. By using the continuation bit instead of existing EXTENDED_ARGs, you can fit 22 bits worth of argument in 4 bytes, while I can only fit 16 bits worth. > > So, your idea uses fewer bytes than mine (4 instead of 6) for any argument in range [65536, 4194303]. But it uses more bytes than mine (4 instead of 2) for any argument in range [128, 255]. Although [128, 255] is a smaller range than [65536, 4194303], it's much more common in arguments, so I think your idea will save less space than mine overall. > > Your idea is also more complicated conceptually, would require more changes to the interpreter (and to third-party code like byteplay or decompyle), and will probably be slower in the eval loop, which also all make it less attractive. Got it. Thanks for clarification. I actually thought Serhiy is going to work on 16bit opcodes, but it seems that he wants to add specialized opcodes like LOAD_ATTR0, LOAD_ATTR1, etc. Yury From victor.stinner at gmail.com Fri Feb 5 19:06:04 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 6 Feb 2016 01:06:04 +0100 Subject: [Python-ideas] Exposing flat bytecode representation to optimizers In-Reply-To: <1763502835.2002313.1454702338876.JavaMail.yahoo@mail.yahoo.com> References: <1763502835.2002313.1454702338876.JavaMail.yahoo.ref@mail.yahoo.com> <1763502835.2002313.1454702338876.JavaMail.yahoo@mail.yahoo.com> Message-ID: 2016-02-05 20:58 GMT+01:00 Andrew Barnert via Python-ideas : > [^3]: The compiler doesn't actually have exactly what the optimizers would want, but it's pretty close: it has a linked list of block objects, each of which has an array of instruction objects, with jump targets being pointers to blocks. This thread was hijacked by discussion the bytecode bytes format. I was confused by the discussion on extended arguments and size of bytecode instructions in bytes. Hopefully, the annoying case of "extended arguments" does not matter here! Compiler instructions use 32-bit signed integer (let's say "integers without arbitrary limit :-D"), and EXTENDED_ARG instructions are only emitted later when the blocks of instructions are compiled to effective bytecode. That's another nice advantage to work on blocks of instructions :-) Victor From storchaka at gmail.com Fri Feb 5 19:49:05 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 6 Feb 2016 02:49:05 +0200 Subject: [Python-ideas] More compact bytecode In-Reply-To: <56B52647.2060403@gmail.com> References: <56B52647.2060403@gmail.com> Message-ID: On 06.02.16 00:46, Yury Selivanov wrote: > What do you think about turning constant slice objects into actual > constants? Slices are not supported by marshal. From yselivanov.ml at gmail.com Fri Feb 5 20:27:46 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 5 Feb 2016 20:27:46 -0500 Subject: [Python-ideas] More compact bytecode In-Reply-To: References: <56B52647.2060403@gmail.com> Message-ID: <56B54C12.6020800@gmail.com> On 2016-02-05 7:49 PM, Serhiy Storchaka wrote: > On 06.02.16 00:46, Yury Selivanov wrote: >> What do you think about turning constant slice objects into actual >> constants? > > Slices are not supported by marshal. That should be fixable, but in any case -- this is a nano-optimization, which is nice to have when there is nothing left to optimize ;) Yury From victor.stinner at gmail.com Fri Feb 5 21:31:08 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 6 Feb 2016 03:31:08 +0100 Subject: [Python-ideas] More compact bytecode In-Reply-To: References: <56B52647.2060403@gmail.com> Message-ID: 2016-02-06 1:49 GMT+01:00 Serhiy Storchaka : > Slices are not supported by marshal. It's possible to inject arbitrary objects to code constants at runtime if you replace the __code__ attribute of a function. I'm using that in FAT Python to inject builtin functions in constants. Later if we consider that the optimization has been proven to be efficient enough, we may consider to implement it directly in Python ;-) Victor From abarnert at yahoo.com Fri Feb 5 21:37:29 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 5 Feb 2016 18:37:29 -0800 Subject: [Python-ideas] Exposing flat bytecode representation to optimizers In-Reply-To: References: <1763502835.2002313.1454702338876.JavaMail.yahoo.ref@mail.yahoo.com> <1763502835.2002313.1454702338876.JavaMail.yahoo@mail.yahoo.com> Message-ID: On Feb 5, 2016, at 15:54, Victor Stinner wrote: > > 2016-02-05 20:58 GMT+01:00 Andrew Barnert via Python-ideas > : >> It would break the optimizer APIs, but `PyCode_Optimize` isn't public > > Sadly, PyCode_Optimize() is public and part of the stable ABI. Oops, you're right--I thought it was inside the Py_LIMITED_API check, but it's after the #endif. Anyway, it's not documented anywhere--no mention in the C API docs, and no explanation of what it does in the header (not even the usual one-liner comment). So, I think if you break that in PEP 511, hopefully nobody will complain :) > But I'm > ok to take the responsability of breaking it if we implement the PEP > 511 since we will probably have a much better API to optimize AST and > bytecode, and I would prefer that no one directly uses > PyCode_Optimize()! Great! >> and the API proposed by PEP 511 is public, but that PEP isn't even finalized, much less accepted yet. > > To be clear: the PEP 511 is a draft still stuck in python-ideas, it's > not ready for a review on python-dev. I still expect changes on the > API. > > I'm very interested by your feedback on the bytecode transformer API :-) As I mentioned before, I think it probably has to take and return a complete code object, not just the four pieces of the code object that the peephole optimizer happens to need. But meanwhile: >> [^3]: The compiler doesn't actually have exactly what the optimizers would want, but it's pretty close: it has a linked list of block objects, each of which has an array of instruction objects, with jump targets being pointers to blocks. That's actually even better to work with, but too complicated to expose to optimizers. > > The current implementation of the peephole optimizer has a complex > code to update jumps. Yes, and it has to be complex, even despite punting on all the hard cases. :) > IMHO it would be simpler if the code would be > blocks of instructions. Sure. But I didn't want to expose the compiler's internal blocks concept to the optimizer; that seems way too brittle. We could design and build a new optimizer-friendly concept of blocks that's independent of the compiler's (as it sounds like you've done in the past), or something simpler like MRAB's labels, or some other "macro-assembler"-style feature that nobody's yet thought of in this discussion. If we really need to do that, I think it's seriously worth looking at incorporating byteplay, porting it to C, and providing a C API for it rather than designing something new. Besides the fact that it hews pretty closely to the design of the dis module, which is already in the stdlib, a variety of people have been using it for years, and I think experience shows that it makes transforming code objects, including inserting and moving ops, easy enough. It also takes care of other annoying things, like getting all 17 of the constructor arguments to code right. But I think Serhiy's much simpler suggestion may be good enough: making NOP removal part of the repack process means that a simple optimizer (like the existing peephole optimizer) never has to renumber jumps or lnotab, and using a flat array of fixed-size instructions means that even if more complicated optimizers do have to renumber, it's easy instead of complex. I won't really know how his idea pans out until I build it and play with it, but I'm hopeful. At any rate, whatever format we expose (unpacked bytecode, a macro-assembler-ish format, the internal compiler blocks, etc.) obviously determines the API for PEP 511 bytecode processors. > My plan is to implement most implementations at AST level and only > optimizer jumps at "bytecode" level. Sure, you've already explained (and Serhiy recently re-explained) that most of what the peephole optimizer does, and much of what you'd want it to do but it doesn't yet do, can be done at the AST level. But there will still be a few things that have to be done with bytecode. (Also, consider that things like decorators have to work on bytecode. And that, while your const optimization can be done at the AST level if we add a new AST node, that wouldn't help for anyone who had the same idea and just wanted to experiment with it, or deploy it locally, etc., without patching the compiler and the ast module.) > Do you know if byteplay and other projects have a similar design? The design is pretty well documented. But it's basically a label-based design. Each original jump target is given a label, and you can add new labels anywhere (or give them nice names). At the end, when you convert a byteplay.Code object back to a code object, it maps those labels to offsets. > It > would be nice to find the root common structure to be a generic API. I don't know if there's anything common between labels, blocks, generalized trees, ... that's worth capturing here. We just want to find the simplest structure that works. >> Flattening it would be trivial. Or, if that's too expensive, we could do something almost as simple and much cheaper: convert it in-line to a deque-like linked list of arrays, with jump targets being indices or pointers into that. Or we could just expose the list of blocks as-is, as an opaque thing with a mutable-deque-of-instructions API around it. > > For the PEP 511 we can imagine multiple levels for code tranformers: > block of instructions, flatten bytecode, AST, etc. I don't think anyone will ever want to write transformers at all possible levels. If you can access source bytes, source text, token stream, AST, and some friendlier version of bytecode, when would you want to access the ASDL, or the unfriendly bytecode, or any other stage in the process? > The cost of the transformation is to convert internal from CPython > objects to Python objects, and then convert the result of the code > transformer back to internal CPython objects. For bytecode, the compiler is already creating Python bytes and list objects to pass to the optimizer. (The AST is another story.) From abarnert at yahoo.com Fri Feb 5 21:40:59 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 5 Feb 2016 18:40:59 -0800 Subject: [Python-ideas] Exposing flat bytecode representation to optimizers In-Reply-To: References: <1763502835.2002313.1454702338876.JavaMail.yahoo.ref@mail.yahoo.com> <1763502835.2002313.1454702338876.JavaMail.yahoo@mail.yahoo.com> Message-ID: <7EEAC686-70CC-4798-940F-C22F551A4DD0@yahoo.com> On Feb 5, 2016, at 16:06, Victor Stinner wrote: > > 2016-02-05 20:58 GMT+01:00 Andrew Barnert via Python-ideas > : >> [^3]: The compiler doesn't actually have exactly what the optimizers would want, but it's pretty close: it has a linked list of block objects, each of which has an array of instruction objects, with jump targets being pointers to blocks. > > This thread was hijacked by discussion the bytecode bytes format. I > was confused by the discussion on extended arguments and size of > bytecode instructions in bytes. > > Hopefully, the annoying case of "extended arguments" does not matter > here! Compiler instructions use 32-bit signed integer (let's say > "integers without arbitrary limit :-D"), and EXTENDED_ARG instructions > are only emitted later when the blocks of instructions are compiled to > effective bytecode. That's exactly how it works today--except that the optimizer is on the wrong side of the boundary; it gets the emitted EXTENDED_ARG instructions, and had to do jump target and lnotab fixups that way. So, either we want to move the optimizer across that boundary (meaning we have to expose some of fragile compiler internals), or we have to come up with a public representation that's easier to work on. I think Serhiy's "unpacked bytecode" may be a good enough version of the latter--and it's dead easy to build. From ncoghlan at gmail.com Sat Feb 6 01:44:12 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 6 Feb 2016 16:44:12 +1000 Subject: [Python-ideas] Simpler Customization of Class Creation - PEP 487 In-Reply-To: References: Message-ID: On 6 February 2016 at 07:20, Martin Teichmann wrote: > Hi List, > > about a year ago I started a discussion on how to simplify metaclasses, > which led to PEP 487. I got some good ideas from this list, but couldn't > follow up on this because I was bound in other projects. Thanks for taking this up again! > In short, metaclasses are often not used as they are considered very > complicated. Indeed they are, especially if you need to use two of them > at the same time in a multiple inheritance context. > > Most metaclasses, however, serve only some of the following three > purposes: a) run some code after a class is created b) initialize descriptors > of a class or c) keep the order in which class attributes have been defined. > > PEP 487 now proposes to put a metaclass into the standard library, which > can be used for all those three purposes. If now libraries start to use this > metaclass, we won't need any metaclass mixing anymore. > > What has changed since the last time I posted PEP 487? Firstly, I re-wrote > large parts of the PEP to make it easier to read. Those who liked the > old text, that's still existing in PEP 422. > > Secondly, I modified the proposal following suggestions from this list: > I added the descriptor initialization (purpose b)), as this was considered > particularly useful, even if it could in principle be done using purpose a) from > above. The order-keeping of the class attributes is the leftover from a much > more ambitious previous idea that would have allowed for custom namespaces > during class creation. But this additional feature would have rendered the > most common usecase - getting the order of attributes - much more > complicated, so I opted for usability over flexibility. I like this updated approach in general - more detailed comments are inline below. > I have put the new version of the PEP here: > > https://github.com/tecki/metaclasses/blob/pep487/pep-0487.txt I also just pushed this version to the PEPs repo. > Proposal > ======== > > While there are many possible ways to use a metaclass, the vast majority > of use cases falls into just three categories: some initialization code > running after class creation, the initalization of descriptors and > keeping the order in which class attributes were defined. > > Those three use cases can easily be performed by just one metaclass. If > this metaclass is put into the standard library, and all libraries that > wish to customize class creation use this very metaclass, no combination > of metaclasses is necessary anymore. While you do cover it later, it's worth mentioning up front that there's a reasonable case to be made that type should just work this way by default. However, changing type *again* is difficult if we decide we made a mistake, so the currently proposed plan is: 1. Introduce a PyPI package (metaclass) for initial iteration on the API 2. Introduce a stdlib metaclass as a provisional in Python 3.6 3. Consider this as possible default behaviour for type in Python 3.7. If type changes, the old type name will just become a legacy alias for type Steps 2 & 3 would be similar to the way the set datatype was first introduced as sets.Set, and only later made a builtin type (with a slightly different API) based on wider experience with the sets module. Step 2 mostly serves as a signalling mechanism that unequivocally blesses the PyPI module created in 1 as setting the future direction of the default behaviour of the builtin "type". > The three use cases are achieved as follows: > > 1. The metaclass contains an ``__init_subclass__`` hook that initializes > all subclasses of a given class, > 2. the metaclass calls an ``__init_descriptor__`` hook for all descriptors > defined in the class, and "__init_descriptor__" confused me, as it wasn't clear to me until much later in the PEP that it's a proposed addition to the *descriptor* API, rather than something you implement on the metaclass you're defining. It's also not really restricted to descriptors - as a new hook, it could be implemented by any attribute, regardless of whether it supported any other part of the descriptor protocol. As such, a possible way to go here is to instead call this a new "attribute ownership protocol", and make the hook name "__set_owner__". It should also be called out that implementations of __set_owner__ will need to handle the case of attribute re-use, and handle things appropriately if the owner has already been set (e.g. in the simplest case, by throwing a RuntimeError indicating that shared ownership isn't supported). > 3. an ``__attribute_order__`` tuple is left in the class in order to inspect > the order in which attributes were defined. > > For ease of use, a base class ``SubclassInit`` is defined, which uses said > metaclass and contains an empty stub for the hook described for use case 1. You should specify the metaclass name here as well. Given the three-fold difference in behaviour, naming the new metaclass and class after only one of those behaviours seems misleading. On the other hand, "OrderedAttributeOwningSubclassInitialisingMeta" would be silly, so it might be worth instead calling them something like "ProvisionalMeta" and "ProvisionalClass" - that is, you're opting in to the provisional future behaviour of the "type" and "object" builtins, without specifying exactly what the current differences are. I'd also suggest putting the new types in the existing "types" module, rather than defining a new module for them (aside from the module on PyPI). > As an example, the first use case looks as follows:: > > class SpamBase(SubclassInit): > # this is implicitly a @classmethod > def __init_subclass__(cls, **kwargs): > # This is invoked after a subclass is created, but before > # explicit decorators are called. > # The usual super() mechanisms are used to correctly support > # multiple inheritance. > # **kwargs are the keyword arguments to the subclasses' > # class creation statement > super().__init_subclass__(cls, **kwargs) > > class Spam(SpamBase): > pass > # the new hook is called on Spam This example isn't particularly clear, since the __init_subclass__ doesn't *do* anything. An example that preserves the class keyword arguments as an attribute may be more comprehensible: >>> class ExampleBase(metaclass.SubclassInit): ... def __init_subclass__(cls, **kwds): ... cls.class_args = kwds ... super().__init_subclass__() ... >>> class Example(ExampleBase, a=1, b=2, c=3): ... pass ... >>> Example.class_args {'b': 2, 'a': 1, 'c': 3} > The second part of the proposal adds an ``__init_descriptor__`` > initializer for descriptors. Descriptors are defined in the body of a > class, but they do not know anything about that class, they do not > even know the name they are accessed with. They do get to know their > owner once ``__get__`` is called, but still they do not know their > name. This is unfortunate, for example they cannot put their > associated value into their object's ``__dict__`` under their name, > since they do not know that name. This problem has been solved many > times, and is one of the most important reasons to have a metaclass in > a library. While it would be easy to implement such a mechanism using > the first part of the proposal, it makes sense to have one solution > for this problem for everyone. This is the part I suggest renaming as an "attribute ownership protocol", with the hook name as "__set_owner__". > To give an example of its usage, imagine a descriptor representing weak > referenced values (this is an insanely simplified, yet working example):: > > import weakref > > class WeakAttribute: > def __get__(self, instance, owner): > return instance.__dict__[self.name] > > def __set__(self, instance, value): > instance.__dict__[self.name] = weakref.ref(value) > > # this is the new initializer: > def __init_descriptor__(self, owner, name): > self.name = name Similar to the __subclass_init__ case, a more meaningful usage example may help here, such as allowing owning classes to define a hook that gets called when the weak reference goes away, while still having useful default behaviour. For example (untested): class WeakAttribute: def __init__(self): self._owner = None self._name = None self._callback = None def __get__(self, instance, owner): if instance is None: return self return instance.__dict__[self.name] def __set__(self, instance, value): instance.__dict__[self.name] = weakref.proxy(value, self._callback) def __set_owner__(self, owner, attr): if self._owner is not None: raise RuntimeError("{!r} already owned by {!r}".format(self, self._owner()) self._owner = weakref.ref(owner) self._name = attr callback = getattr(owner, "attribute_collected", None) if callback is not None: self._callback = functools.partial(callback, attr) class Example(metaclass.SubclassInit): proxy = WeakAttribute() def attribute_collected(self, attr): print("{} was garbage collected".format()) > The third part of the proposal is to leave a tuple called > ``__attribute_order__`` in the class that contains the order in which > the attributes were defined. This is a very common usecase, many > libraries use an ``OrderedDict`` to store this order. This is a very > simple way to achieve the same goal. This should spell out the underlying mechanism here - the new metaclass will *also* use OrderedDict to preserve the order during class construction, so the extra bit the provisional metaclass adds over the DIY __prepare__ method is taking the original ordered dicts keys and saving them in an attribute, while __dict__ itself will remain an ordinary unordered dict. > New Ways of Using Classes > ========================= > > This proposal has many usecases like the following. In the examples, > we still inherit from the ``SubclassInit`` base class. This would > become unnecessary once this PEP is included in Python directly. > > Subclass registration > --------------------- > > Especially when writing a plugin system, one likes to register new > subclasses of a plugin baseclass. This can be done as follows:: > > class PluginBase(SubclassInit): > subclasses = [] > > def __init_subclass__(cls, **kwargs): > super().__init_subclass__(**kwargs) > cls.subclasses.append(cls) > > One should note that this also works nicely as a mixin class. This should explain that the difference between this and just calling PluginBase.__subclasses__() is that this example flattens the inheritance tree into a simple list. > Trait descriptors > ----------------- > > There are many designs of Python descriptors in the wild which, for > example, check boundaries of values. Often those "traits" need some support > of a metaclass to work. This is how this would look like with this > PEP:: > > class Trait: > def __get__(self, instance, owner): > return instance.__dict__[self.key] > > def __set__(self, instance, value): > instance.__dict__[self.key] = value > > def __init_descriptor__(self, owner, name): > self.key = name > > class Int(Trait): > def __set__(self, instance, value): > # some boundary check code here > super().__set__(instance, value) This doesn't show the descriptor subclass making use of the state set up by the new hook on the parent class, so the subclass ends up making the example more confusing, rather than improving it. > Rejected Design Options > ======================= > > > Calling the hook on the class itself > ------------------------------------ > > Adding an ``__autodecorate__`` hook that would be called on the class > itself was the proposed idea of PEP 422. Most examples work the same > way or even better if the hook is called on the subclass. In general, > it is much easier to explicitly call the hook on the class in which it > is defined (to opt-in to such a behavior) than to opt-out, meaning > that one does not want the hook to be called on the class it is > defined in. > > This becomes most evident if the class in question is designed as a > mixin: it is very unlikely that the code of the mixin is to be > executed for the mixin class itself, as it is not supposed to be a > complete class on its own. > > The original proposal also made major changes in the class > initialization process, rendering it impossible to back-port the > proposal to older Python versions. This should be elaborated on: * we *do* want to change the default behaviour of type in the future * we *also* want to be able to validate the usability of those changes before we make them Unlike PEP 422, this PEP lets us take two preliminary steps (library on PyPI, provisional API in the standard library) *before* making any changes to type itself. > History > ======= > > This used to be a competing proposal to PEP 422 by Nick Coughlan and Coghlan :) > Daniel Urban. It shares both most of the PEP text and proposed code, I think the code and text have diverged significantly now, but the major shared aspect was always common *goals*, rather than any of the technical details. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From lkb.teichmann at gmail.com Sat Feb 6 06:02:44 2016 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Sat, 6 Feb 2016 12:02:44 +0100 Subject: [Python-ideas] Simpler Customization of Class Creation - PEP 487 In-Reply-To: <1814585983.22893.1454709193740.JavaMail.yahoo@mail.yahoo.com> References: <1814585983.22893.1454709193740.JavaMail.yahoo@mail.yahoo.com> Message-ID: Hi Andrew, Hi List, thanks for the quick review. You brought up very interesting points which should be discussed here. >> Most metaclasses, however, serve only some of the following three >> purposes: a) run some code after a class is created b) initialize descriptors >> of a class or c) keep the order in which class attributes have been defined. > > How is c) not served by just defining a __prepare__ method that returns an OrderedDict? That seems easier to use than a separate dict and __attribute_order__ tuple. You later mention that many people already do this, but don't explain what's wrong with it or why your idea is better. It's not like a one-line __prepare__ is difficult to write, or like OrderedDict is too slow (especially in 3.6), so you're just adding a cost (less convenient to use) for no benefit that I can see. In my implementation this is exactly what I do: I create a OrderedDict in __prepare__. The question is just: how dow we get that information to the user? The __prepare__d namespace only exists during class creation, type.__new__ cripples it into a normal dict. (that is done here: https://github.com/python/cpython/blob/master/Objects/typeobject.c#L2343) An older version of the PEP introduced a namespace parameter to the __init_subclass__ hook which would still get the OrderedDict. This is a neat way of doing so, but I did not want to exclude the option that this metaclass once becomes the default metaclass. And then suddenly OrderedDict turns from just another standard library class into a type built into the language. Maybe this is not a big deal, maybe it is, what are other people thinking about that? > It sounds like you're trying to have it both ways here: it seems like if your proposal really is better because it works with Python 2.7 rather than just 3.x, then your proposal being in the stdlib is probably a bad idea, because it will mislead rather than help people trying to write version-straddling code. This is a very complicated issue. In order for my proposal to make any sense, everyone has to use exactly the same metaclass, not just a look-alike. I don't see another way to do that than putting it into the standard library, or directly into the Python language. While Python 2 compatibility is not such a big deal anymore (this is 2016, and I figure 2015 really has been the year of Python 3), the problem is that this proposal is targetting authors of libraries, not just applications. Libraries are, for a good reason, very reluctant to drop backwards compatibility. So trying to convince them to do something which is not backwards portable won't fly. My idea is that libraries just use the PyPI implementation of this PEP, which will use the standard library implementation once it is existing. But library developers are also very reluctant to add any additional dependencies, my module is certainly no exception. Instead they have the option to copy my library into theirs. That means that they won't have the benefit of compatibility with other libraries while the metaclass is not in the standard library, but once it is, they are automatically compatible with the rest of the world. The best option for compatiblity probably is that this metaclass is also added to compatibility libraries like six, so that different libraries are at least compatible if they use the same compatibility library (and are automatically compatible with everyone else once the metaclass is in the standard library). Greetings Martin From storchaka at gmail.com Sat Feb 6 06:05:06 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 6 Feb 2016 13:05:06 +0200 Subject: [Python-ideas] Exposing flat bytecode representation to optimizers In-Reply-To: <1080499058.2022676.1454706200143.JavaMail.yahoo@mail.yahoo.com> References: <1080499058.2022676.1454706200143.JavaMail.yahoo@mail.yahoo.com> Message-ID: On 05.02.16 23:03, Andrew Barnert via Python-ideas wrote: > But why not make it even simpler and just have all unpacked > instructions be 32-bit? Sure, that means unpacked code arrays are > bigger, but it's not like the optimizers are going to be looping over > the same array a zillion times and worrying about cache spill (or > that the optimizers will be in a hotspot in the first place). Then > we've just got an int32*, and a jump to offset 76 is a jump to the 4 > bytes at bytecode[76] (or, in Python, where we may still have to use > a bytes object, it's at worst a jump to bytecode[76<<2]). My idea was to not add new opcodes for unpacked form and keep unpacked form executable. Thus we have 16-bit LOAD_CONST and 32-bit LONG_LOAD_CONST, but only 16-bit POP_TOP and COMPARE_OP since POP_TOP has no argument and the argument of COMPARE_OP always fits in 8 bit. Unpacked form always uses long variant if it exists. Alternative variant - always use 32-bit instructions and don't pack them to 8 or 16 bits. This will increase bytecode size by 4/2.73 = 1.5 times, but will make some parts of compiler, optimizer and interpreter simpler. From abarnert at yahoo.com Sat Feb 6 12:47:58 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 6 Feb 2016 09:47:58 -0800 Subject: [Python-ideas] Simpler Customization of Class Creation - PEP 487 In-Reply-To: References: <1814585983.22893.1454709193740.JavaMail.yahoo@mail.yahoo.com> Message-ID: On Feb 6, 2016, at 03:02, Martin Teichmann wrote: > > While Python 2 compatibility is not such a big deal anymore (this is > 2016, and I figure 2015 really has been the > year of Python 3), the problem is that this proposal is targetting > authors of libraries, not just applications. > Libraries are, for a good reason, very reluctant to drop backwards > compatibility. So trying to convince them > to do something which is not backwards portable won't fly. My idea is > that libraries just use the PyPI > implementation of this PEP, which will use the standard library > implementation once it is existing. OK, now this makes sense. The PyPI module does something like this: try: from types import Meta except ImportError: pass So, in 2.7 or 3.5, this has no effect, and you get the Meta defined in the module, but if 3.6 or 3.7 adds types.Meta, people using the PyPI module get the stdlib type, so they get the benefits of a shared metaclass. And if 3.7 or 3.8 replaces type with your metaclass, and just has types.Meta = type, the PyPI module now gets the builtin if possible, the stdlib type as a fallback, and the PyPI type as a double-fallback. Assuming that understanding was right, this does seem like the best solution to the problem--not perfect, but as good as anything could possibly be. One thing on the six idea: if some libraries get the type from six and others from your module, they'll have incompatible metaclasses for no good reason (when used on 3.5 and earlier). It seems like if this is going to end up in six, it should be in six as soon as possible, and direct use of the separate library should be strongly discouraged. From python at mrabarnett.plus.com Sat Feb 6 14:16:05 2016 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 06 Feb 2016 19:16:05 +0000 Subject: [Python-ideas] Exposing flat bytecode representation to optimizers In-Reply-To: Message-ID: On 2016-02-06 11:05:06, "Serhiy Storchaka" wrote: >On 05.02.16 23:03, Andrew Barnert via Python-ideas wrote: >>But why not make it even simpler and just have all unpacked >>instructions be 32-bit? Sure, that means unpacked code arrays are >>bigger, but it's not like the optimizers are going to be looping over >>the same array a zillion times and worrying about cache spill (or >>that the optimizers will be in a hotspot in the first place). Then >>we've just got an int32*, and a jump to offset 76 is a jump to the 4 >>bytes at bytecode[76] (or, in Python, where we may still have to use >>a bytes object, it's at worst a jump to bytecode[76<<2]). > >My idea was to not add new opcodes for unpacked form and keep unpacked >form executable. Thus we have 16-bit LOAD_CONST and 32-bit >LONG_LOAD_CONST, but only 16-bit POP_TOP and COMPARE_OP since POP_TOP >has no argument and the argument of COMPARE_OP always fits in 8 bit. >Unpacked form always uses long variant if it exists. > >Alternative variant - always use 32-bit instructions and don't pack >them to 8 or 16 bits. This will increase bytecode size by 4/2.73 = 1.5 >times, but will make some parts of compiler, optimizer and interpreter >simpler. > If, at some point, we find that 32 bits aren't enough (because we're starting to get code objects containing more than 4 million instructions), we could then add a 'wide' form with 64-bit instructions. Just a thought... From antoine at python.org Sat Feb 6 14:18:12 2016 From: antoine at python.org (Antoine Pitrou) Date: Sat, 6 Feb 2016 19:18:12 +0000 (UTC) Subject: [Python-ideas] More compact bytecode References: Message-ID: Serhiy Storchaka writes: > > 2. Use 16-bit opcodes as in WPython. > > This will decrease mean size of opcode from 2.73 to 2.044 bytes (1.33 > times). But this might make decoding of 85% opcodes slightly faster. It sounds like, by 16-bit opcodes, you mean combine the opcode and the argument in a single 16-bit word. But that doesn't solve the issue you want to solve: you still have to decode the argument encoded in the 16-bit word. I don't see where the benefit is. The *byte* size of bytecode is IMO largely unimportant. Python bytecode is synthetic and high-level, its expressive density is much higher than that of native code. I don't think the cache occupancy of bytecode is a significant contributor to Python performance (certainly much less so than the memory consumption of *objects*). (as for the disk occupancy of pyc files, if that is a concern we could actually compress those files; there are very fast compressors with decent efficiency these days, such as lzo or snappy) It is generally estimated the overhead of bytecode dispatch and decoding is around 10-30% for CPython. You cannot hope to eliminate that overhead entirely without writing a (JIT or AOT) compiler, so any heroic effort to restructure the current opcode space and structure will at best win 5 to 20% on select benchmarks. Regards Antoine. From abarnert at yahoo.com Sat Feb 6 19:21:11 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 6 Feb 2016 16:21:11 -0800 Subject: [Python-ideas] More compact bytecode In-Reply-To: References: Message-ID: <172A4A4B-0B53-4BFE-83B7-38E945A79246@yahoo.com> On Feb 6, 2016, at 11:18, Antoine Pitrou wrote: > > Serhiy Storchaka writes: >> >> 2. Use 16-bit opcodes as in WPython. >> >> This will decrease mean size of opcode from 2.73 to 2.044 bytes (1.33 >> times). But this might make decoding of 85% opcodes slightly faster. > > It sounds like, by 16-bit opcodes, you mean combine the opcode and the > argument in > a single 16-bit word. But that doesn't solve the issue you want to solve: you > still have to decode the argument encoded in the 16-bit word. I don't see > where the benefit is. If (at least most of the time) the argument is just the second byte of the two bytes, that decoding is trivial. The cost should be less than the savings of doing one short read instead of three byte reads. (Of course we won't know that for sure until we try. It should also lets us simplify the eval loop a bit, and simplify all other code that has to deal with bytecode (although I'd like to simplify it even further, as discussed in the other thread). > It is generally estimated the overhead of bytecode dispatch and decoding is > around > 10-30% for CPython. You cannot hope to eliminate that overhead entirely without > writing a (JIT or AOT) compiler, so any heroic effort to restructure the current > opcode space and structure will at best win 5 to 20% on select benchmarks. To be honest, unlike everyone else on this thread, I'm actually more interested in the simplicity gains than the performance gains. When I'm running CPU-bound code that spends most of its time in CPython itself, it's usually more than fast enough--and, when it isn't, it's almost always an order of magnitude too slow, so adding up all these little 5-10% gains is never going to get us near the point where I can stop using PyPy or Cython or something. I don't know whether other people have different use cases where these speedups really do matter, or if they're just chasing micro-optimization of the CPython core for its own sake, but I'm willing to cynically use their desire for an 8% speedup to get them onboard with selling my simplification. :) (Plus, adding up all these little gains could soon get us to the point where 3.7 finally beats 2.7 in every benchmark, instead of just most of them, which would kill off an annoying source of FUD.) From storchaka at gmail.com Sun Feb 7 02:18:18 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 7 Feb 2016 09:18:18 +0200 Subject: [Python-ideas] More compact bytecode In-Reply-To: References: Message-ID: On 06.02.16 21:18, Antoine Pitrou wrote: > It sounds like, by 16-bit opcodes, you mean combine the opcode and the > argument in > a single 16-bit word. But that doesn't solve the issue you want to solve: you > still have to decode the argument encoded in the 16-bit word. I don't see > where the benefit is. Current code uses 3 read operations: opcode = *next_instr++; next_instr += 2; oparg = (next_instr[-1]<<8) + next_instr[-2]; Even combining the latter two operations in one read operation give as 10% gain in the microbenchmark (see http://bugs.python.org/issue25823): opcode = *next_instr++; oparg = *(unsigned short *)next_instr; next_instr += 2; With combining the opcode and the argument in always aligned 16-bit word I expect larger gain. word = *(unsigned short *)next_instr; next_instr += 2; opcode = word & 0xff; oparg = word >> 8; > It is generally estimated the overhead of bytecode dispatch and decoding is > around > 10-30% for CPython. You cannot hope to eliminate that overhead entirely without > writing a (JIT or AOT) compiler, so any heroic effort to restructure the current > opcode space and structure will at best win 5 to 20% on select benchmarks. This would be awesome result. From solipsis at pitrou.net Sun Feb 7 07:53:35 2016 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 7 Feb 2016 13:53:35 +0100 Subject: [Python-ideas] More compact bytecode References: <172A4A4B-0B53-4BFE-83B7-38E945A79246@yahoo.com> Message-ID: <20160207135335.0809231a@fsol> On Sat, 6 Feb 2016 16:21:11 -0800 Andrew Barnert via Python-ideas wrote: > > To be honest, unlike everyone else on this thread, I'm actually more interested in the simplicity gains than the performance gains. It is a laudable goal, but what is proposed here wouldn't simplify much if anything, since some opcodes need more than 8 bits of arguments (typically the opcodes that have two logical 8-bit arguments packed in the 16-bit word). So you'll still get variable-sized opcode and need an adequate decoding machinery for them. So perhaps bytecode should be processed by the compile chain in the form of: typedef struct { int opcode; int operand; } instruction_t; instruction_t *bytecode; And then get packed at the end, when creating the code object. > When I'm running CPU-bound code that spends most of its time in CPython itself, it's usually more than fast enough--and, when it isn't, it's almost always an order of magnitude too slow, so adding up all these little 5-10% gains is never going to get us near the point where I can stop using PyPy or Cython or something. I think that mirrors the experience of many people. Small perf improvements are not useless either, because they can make e.g. command-line tools feel a bit more "snappy". > (Plus, adding up all these little gains could soon get us to the point where 3.7 finally beats 2.7 in every benchmark, instead of just most of them, which would kill off an annoying source of FUD.) I think that FUD is very tired now and very few people pay attention to it. The only major factor slowing down the 2->3 migration is the effort required for porting. (also, the breadth of new features and improvements in the 3.x line massively offsets small degradations on micro-benchmarks) Regards Antoine. From solipsis at pitrou.net Sun Feb 7 07:54:29 2016 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 7 Feb 2016 13:54:29 +0100 Subject: [Python-ideas] More compact bytecode References: Message-ID: <20160207135429.470966c1@fsol> On Sun, 7 Feb 2016 09:18:18 +0200 Serhiy Storchaka wrote: > On 06.02.16 21:18, Antoine Pitrou wrote: > > It sounds like, by 16-bit opcodes, you mean combine the opcode and the > > argument in > > a single 16-bit word. But that doesn't solve the issue you want to solve: you > > still have to decode the argument encoded in the 16-bit word. I don't see > > where the benefit is. > > Current code uses 3 read operations: > > opcode = *next_instr++; > next_instr += 2; > oparg = (next_instr[-1]<<8) + next_instr[-2]; > > Even combining the latter two operations in one read operation give as > 10% gain in the microbenchmark (see http://bugs.python.org/issue25823): > > opcode = *next_instr++; > oparg = *(unsigned short *)next_instr; > next_instr += 2; So it remains to be seen how much performance is won on actual non-silly benchmarks ;-) Regards Antoine. From python at mrabarnett.plus.com Sun Feb 7 14:22:03 2016 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 7 Feb 2016 19:22:03 +0000 Subject: [Python-ideas] More compact bytecode In-Reply-To: References: Message-ID: <56B7995B.4010701@mrabarnett.plus.com> On 2016-02-07 07:18, Serhiy Storchaka wrote: > On 06.02.16 21:18, Antoine Pitrou wrote: >> It sounds like, by 16-bit opcodes, you mean combine the opcode and the >> argument in >> a single 16-bit word. But that doesn't solve the issue you want to solve: you >> still have to decode the argument encoded in the 16-bit word. I don't see >> where the benefit is. > > Current code uses 3 read operations: > > opcode = *next_instr++; > next_instr += 2; > oparg = (next_instr[-1]<<8) + next_instr[-2]; > > Even combining the latter two operations in one read operation give as > 10% gain in the microbenchmark (see http://bugs.python.org/issue25823): > > opcode = *next_instr++; > oparg = *(unsigned short *)next_instr; > next_instr += 2; > The previous code is big-endian, whereas this code's endianness is processor-dependant. > With combining the opcode and the argument in always aligned 16-bit word > I expect larger gain. > > word = *(unsigned short *)next_instr; > next_instr += 2; > opcode = word & 0xff; > oparg = word >> 8; > >> It is generally estimated the overhead of bytecode dispatch and decoding is >> around >> 10-30% for CPython. You cannot hope to eliminate that overhead entirely without >> writing a (JIT or AOT) compiler, so any heroic effort to restructure the current >> opcode space and structure will at best win 5 to 20% on select benchmarks. > > This would be awesome result. > From abarnert at yahoo.com Sun Feb 7 20:07:43 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 7 Feb 2016 17:07:43 -0800 Subject: [Python-ideas] More compact bytecode In-Reply-To: <20160207135335.0809231a@fsol> References: <172A4A4B-0B53-4BFE-83B7-38E945A79246@yahoo.com> <20160207135335.0809231a@fsol> Message-ID: On Feb 7, 2016, at 04:53, Antoine Pitrou wrote: > > On Sat, 6 Feb 2016 16:21:11 -0800 > Andrew Barnert via Python-ideas > wrote: >> >> To be honest, unlike everyone else on this thread, I'm actually more interested in the simplicity gains than the performance gains. > > It is a laudable goal, but what is proposed here wouldn't simplify much > if anything, since some opcodes need more than 8 bits of arguments > (typically the opcodes that have two logical 8-bit arguments packed in > the 16-bit word). So you'll still get variable-sized opcode and need > an adequate decoding machinery for them. With "wordcode" (at least as I've implemented it), EXTENDED_ARG code doesn't get any simpler (or more complicated)--but the main loop can just read 2 bytes (and then add in any extended arg value) instead of reading 1 or 3, which is definitely simpler. (As implemented in my patch, it's not actually much simpler, because I use all the same macros, I just redefine them, but that's because my initial goal was to change as little as possible--which turns out to be very little. If it seems promising, we can do a slightly larger change that will simplify things more, and possible also improve performance, but I don't want to do that until I've proven it's worth trying.) > So perhaps bytecode should be processed by the compile chain in the > form of: > > typedef struct { > int opcode; > int operand; > } instruction_t; > > instruction_t *bytecode; > > And then get packed at the end, when creating the code object. Yes, there's another thread on that, with a few different variations. This is actually independent of the wordcode change, so I'm doing it on a separate branch. The variation I'm currently working on (based on suggestions by Serhiy) just uses a 32-bit "unpacked bytecode" format (1 byte opcode, 3 bytes arg, no EXTENDED_ARG). The compiler flattens its blocks of instruction objects into unpacked bytecode, gives that to the optimizer (currently that's the peephole optimizer, but this is also where Victor's PEP 511 bytecode transformer hooks fit in), and then packs the bytecode, removing NOPs and fixing up jump targets and lnotab. This makes the peephole optimizer a lot simpler. I've also exposed unpack and pack-and-fixup functions by C API and the dis module, which means bytecode-hacking decorators, import hooks, etc. can work on unpacked bytecode, and usually won't need a third-party module like byteplay to be readable. The big question is whether it's acceptable to limit args to 24 bits. I don't think jumps > 4 million are an issue, but > 255 annotations might be? I'll test it with as much code as possible to see; if not, unpacking to 64 bits (whether with a struct or just an int64) is the obvious answer. Also, it may turn out that bytecode processing is still too painful even with the "unpacked" format. In which case we'll need something different--labels, a tree of blocks similar to what the compiler already has, or something else. But I think this will be sufficient. >> (Plus, adding up all these little gains could soon get us to the point where 3.7 finally beats 2.7 in every benchmark, instead of just most of them, which would kill off an annoying source of FUD.) > > I think that FUD is very tired now and very few people pay attention to it. Well, it spawned a thread on -ideas with dozens of replies just a week or two ago... > (also, the breadth of new features and improvements in the 3.x line > massively offsets small degradations on micro-benchmarks) You don't have to convince me; as far as I'm concerned, the barrier was crossed with 3.2. From ncoghlan at gmail.com Mon Feb 8 01:50:43 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 8 Feb 2016 16:50:43 +1000 Subject: [Python-ideas] Simpler Customization of Class Creation - PEP 487 In-Reply-To: References: <1814585983.22893.1454709193740.JavaMail.yahoo@mail.yahoo.com> Message-ID: On 7 February 2016 at 03:47, Andrew Barnert via Python-ideas wrote: > One thing on the six idea: if some libraries get the type from six and others from your module, they'll have incompatible metaclasses for no good reason (when used on 3.5 and earlier). It seems like if this is going to end up in six, it should be in six as soon as possible, and direct use of the separate library should be strongly discouraged. The daisy chain of fallbacks approach holds - six would just become another stepping stone on the path to the standard library (and eventually the type builtin). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From lkb.teichmann at gmail.com Mon Feb 8 04:28:17 2016 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Mon, 8 Feb 2016 10:28:17 +0100 Subject: [Python-ideas] Simpler Customization of Class Creation - PEP 487 In-Reply-To: References: Message-ID: Hi Nick, Hi List, Thanks for the good comments, I tried to incorporate all of them into the new version of my PEP, which can be found here: https://github.com/tecki/peps/blob/pep487/pep-0487.txt I'm not sure whether it is a good idea to post the entire PEP here all the time, but actually your inline comments were a very helpful technique. I'm not sure how to proceed. Just some comment about the changes I made: > Given the three-fold difference in behaviour, naming the new metaclass > and class after only one of those behaviours seems misleading. On the > other hand, "OrderedAttributeOwningSubclassInitialisingMeta" would be > silly, so it might be worth instead calling them something like > "ProvisionalMeta" and "ProvisionalClass" - that is, you're opting in > to the provisional future behaviour of the "type" and "object" > builtins, without specifying exactly what the current differences are. I had already figured that SubclassInit is not such a great naming idea, but couldn't find anything better. Calling something Provisional gives bad Karma, nothing sticks around longer than something marked "Provisional". So, I propose to call the metaclass "types.Type" and the base class "types.Object". Maybe we can even go for "types.type" and "types.object"? I thought this would be a bit, well, blunt, so I chose the former. >> To give an example of its usage, imagine a descriptor representing weak >> referenced values (this is an insanely simplified, yet working example):: >> >> import weakref >> >> class WeakAttribute: >> def __get__(self, instance, owner): >> return instance.__dict__[self.name] >> >> def __set__(self, instance, value): >> instance.__dict__[self.name] = weakref.ref(value) >> >> # this is the new initializer: >> def __init_descriptor__(self, owner, name): >> self.name = name > > Similar to the __subclass_init__ case, a more meaningful usage example > may help here, such as allowing owning classes to define a hook that > gets called when the weak reference goes away, while still having > useful default behaviour. For example (untested): > > class WeakAttribute: > def __init__(self): > self._owner = None > self._name = None > self._callback = None > > def __get__(self, instance, owner): > if instance is None: > return self > return instance.__dict__[self.name] > > def __set__(self, instance, value): > instance.__dict__[self.name] = weakref.proxy(value, self._callback) > > def __set_owner__(self, owner, attr): > if self._owner is not None: > raise RuntimeError("{!r} already owned by > {!r}".format(self, self._owner()) > self._owner = weakref.ref(owner) > self._name = attr > callback = getattr(owner, "attribute_collected", None) > if callback is not None: > self._callback = functools.partial(callback, attr) > > class Example(metaclass.SubclassInit): > proxy = WeakAttribute() > def attribute_collected(self, attr): > print("{} was garbage collected".format()) While I think that you're correct that a better example would be a good idea, you example for me hints to that __set_owner__ does an initialization of the attribute at object creation time, but it does it at class creation time. I'll sleep over it, maybe I find a nice example. It's also important to note that the new thing about the metaclass is not that it supplies the attribute with the owner, but that it tell the attribute about its name. The owner was already shipped to __get__ and __set__, but not the name within that class. I am not aware of a way to find out the attribute name from within a descriptor. (That's not true: a year ago some ideas how to do that were posted here, but they used undocumented internal features of CPython, not necessarily a reasonable way). This is why the name __set_owner__ also might be misleading, but __set_name__ also doesn't sound right to me, any better calls? For the meantime I stick with __set_owner__. >> This used to be a competing proposal to PEP 422 by Nick Coughlan and > > Coghlan :) Sorry... Greetings Martin Teichmann From sjoerdjob at sjec.nl Mon Feb 8 05:00:39 2016 From: sjoerdjob at sjec.nl (Sjoerd Job Postmus) Date: Mon, 8 Feb 2016 11:00:39 +0100 Subject: [Python-ideas] Simpler Customization of Class Creation - PEP 487 In-Reply-To: References: Message-ID: <20160208100039.GA12662@sjoerdjob.com> On Mon, Feb 08, 2016 at 10:28:17AM +0100, Martin Teichmann wrote: > While I think that you're correct that a better example would be a good > idea, you example for me hints to that __set_owner__ does an initialization > of the attribute at object creation time, but it does it at class creation time. > I'll sleep over it, maybe I find a nice example. > > It's also important to note that the new thing about the metaclass is > not that it supplies the attribute with the owner, but that it tell > the attribute > about its name. The owner was already shipped to __get__ and __set__, > but not the name within that class. I am not aware of a way to find out > the attribute name from within a descriptor. (That's not true: a year ago > some ideas how to do that were posted here, but they used undocumented > internal features of CPython, not necessarily a reasonable way). > This is why the name __set_owner__ also might be misleading, but > __set_name__ also doesn't sound right to me, any better calls? For > the meantime I stick with __set_owner__. It would cause backwards-incompatibility, but would it not be an option to call __get__ and __set__ (and __del__) with an extra parameter: the name of the attribute that is supposed to be accessed? (This can probably be worked around by inspecting the signature of __get__/... to see how many arguments it takes). Because right now, if the descriptor itself does not maintain any state, it is possible to assign the same descriptor to multiple class attributes, maybe even to multiple classes. (On the other hand, in that case it would probably not need `__set_owner__` either). The question to me is: can __set_owner__ give us something that can not be done by adding a name parameter to __get__/... . The weakref proxy (with callback) looks to be one of those cases, but is not really: def __set__(self, owner, value, attr): def _cb(arg): callback = getattr(owner, 'attribute_collected') if callback is note None: callback(attr) instance.__dict__[self.name] = weakref.proxy(value, callback) Calling desc.__set__(owner, value, attr) instead of desc.__set__(owner, value) can probably also be implemented by a custom metaclass which walks over all the values in the generated namespace, and replaces a descriptor with a NameCallingDescriptor(val, attr). for attr, val in namespace.items(): if is_descriptor(val): if descriptor_wants_attribute_name(val): namespace[attr] = NameCallingDescriptor(val, attr) Is there anything that __set_owner__ brings that can not be done by changing __get__/__set__/...? From lkb.teichmann at gmail.com Mon Feb 8 08:16:29 2016 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Mon, 8 Feb 2016 14:16:29 +0100 Subject: [Python-ideas] Simpler Customization of Class Creation - PEP 487 In-Reply-To: <20160208100039.GA12662@sjoerdjob.com> References: <20160208100039.GA12662@sjoerdjob.com> Message-ID: Hi Sjoerd, Hi List, > It would cause backwards-incompatibility, but would it not be an option > to call __get__ and __set__ (and __del__) with an extra parameter: the > name of the attribute that is supposed to be accessed? (This can > probably be worked around by inspecting the signature of __get__/... to > see how many arguments it takes). > > Because right now, if the descriptor itself does not maintain any state, > it is possible to assign the same descriptor to multiple class > attributes, maybe even to multiple classes. (On the other hand, in that > case it would probably not need `__set_owner__` either). I think the backwards-incompatibility is a show stopper for this idea. I actually think it's a good idea, but too late. I checked PEP 252, which invented __get__, and it also states that every attribute should have a __name__ and __objclass__. But it is silent on how an attribute should acquire that. __set_owner__ fills this silence. We could try to actually implement PEP 252 and set __name__ and __objclass__. I guess this would give a big backwards compatibility headache. We could require descriptors to inherit from a class types.Descriptor, and set the __name__ and __objclass__ only on them. That's a little less flexible than the current proposal. > The question to me is: can __set_owner__ give us something that can not > be done by adding a name parameter to __get__/... . The weakref proxy > (with callback) looks to be one of those cases, but is not really: Well, __set_owner__ gets called much earlier. This means we might avoid having to write some lazy initialization code. I know, that is a weak argument, but I think it's actually nicer to have a __set_owner__ method over having an overloaded __get__. But this "nice" certainly is a question of taste. Greetings Martin From mike at selik.org Mon Feb 8 16:24:36 2016 From: mike at selik.org (Michael Selik) Date: Mon, 08 Feb 2016 21:24:36 +0000 Subject: [Python-ideas] Simpler Customization of Class Creation - PEP 487 In-Reply-To: References: Message-ID: Could you help me better understand the 3 purposes you described in the context of multiple metaclasses? On Fri, Feb 5, 2016 at 4:20 PM Martin Teichmann wrote: > a) run some code after a class is created > b) initialize descriptors > c) keep the order in which class attributes have been defined This new ProvisionalMeta will allow multiple inheritance of metaclasses using standard left-to-right priority (or the MRO for more depth). In the case of two metaclasses, the code from Left will run immediately after the class is created and the code from Right will go next. Right? (joking. I should say "Correct?") If the user doesn't want to learn about the metaclasses inside the hierarchy, as per your explanation of the trouble with metaclass conflicts, will they be able to make the decision about which should be Left and which should be Right? What about cases where Left and Right interfere with each other? This reminds me of version control merge conflicts. It's frustrating that the automatic merge fails, but it's better than allowing mistakes. Or am I veering too much towards Java-land? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mahmoud at hatnote.com Mon Feb 8 18:36:26 2016 From: mahmoud at hatnote.com (Mahmoud Hashemi) Date: Mon, 8 Feb 2016 15:36:26 -0800 Subject: [Python-ideas] Compact repr operator (i.e., __short_repr__) Message-ID: I was curious if what kind of interest there would be in adding an overridable method to types to allow getting back a repr, but a predictably short one. The motivation is that there are many cases where producing the full, round-trippable repr is possible, but would take up significant resources or be too huge to be useful for human consumption. The built-in [reprlib module][1] certainly shows there's an interest for many built-in types, and I think arguably that same functionality is desirable for user types. As mentioned in the subject a possible API could be: class ValueObject(object): def __init__(self, content): self.content = content def __repr__(self): return '%s(%r)' % (self.__class__.__name__, self.content) def __short_repr__(self, size=None, depth=None): # TODO: should interpretation of size/depth be casual or strict? if size and len(self.content) > size: short_content = self.content[:size] + '...' else: pass # TODO: could just return normal repr possibly return '<%s content=%r)' % (self.__class__.__name__, short_content) Without something like this, there's no way to ask an object if it's repr is of manageable length, or how its repr could be meaningfully shortened. Usually I just chop out the middle and add an ellipsis, but as for the time spent generating that middle, I'll never get those cycles back. Anyways, thanks for your attention, and happy Monday to all! Mahmoud https://github.com/mahmoud https://twitter.com/mhashemi http://sedimental.org [1]: https://docs.python.org/2/library/functions.html#func-repr [2]: https://docs.python.org/3.4/library/reprlib.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Feb 8 18:43:58 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 8 Feb 2016 15:43:58 -0800 Subject: [Python-ideas] Compact repr operator (i.e., __short_repr__) In-Reply-To: References: Message-ID: On Mon, Feb 8, 2016 at 3:36 PM, Mahmoud Hashemi wrote: > The motivation is that there are many cases where producing the full, > round-trippable repr is possible, but would take up significant resources > or be too huge to be useful for human consumption. > numpy as realized this, and produced a __repr__ (and __str__) that truncates: In [*26*]: len(arr) Out[*26*]: 100000 In [*27*]: repr(arr) Out[*27*]: 'array([ 0.00000000e+00, 1.00001000e-02, 2.00002000e-02, ...,\n 9.99980000e+02, 9.99990000e+02, 1.00000000e+03])' I"m not sure that a full-sized repr is ever useful, so this seems fine to me. I wonder how often anyone actually counts on eval(repr(obj)) == obj ? In short, I don't see that this would be all that useful. -CHB > The built-in [reprlib module][1] certainly shows there's an interest for > many built-in types, and I think arguably that same functionality is > desirable for user types. > > As mentioned in the subject a possible API could be: > > class ValueObject(object): > def __init__(self, content): > self.content = content > > def __repr__(self): > return '%s(%r)' % (self.__class__.__name__, self.content) > > def __short_repr__(self, size=None, depth=None): > # TODO: should interpretation of size/depth be casual or strict? > if size and len(self.content) > size: > short_content = self.content[:size] + '...' > else: > pass # TODO: could just return normal repr possibly > return '<%s content=%r)' % (self.__class__.__name__, short_content) > > Without something like this, there's no way to ask an object if it's repr > is of manageable length, or how its repr could be meaningfully shortened. > Usually I just chop out the middle and add an ellipsis, but as for the time > spent generating that middle, I'll never get those cycles back. > > Anyways, thanks for your attention, and happy Monday to all! > > Mahmoud > https://github.com/mahmoud > https://twitter.com/mhashemi > http://sedimental.org > > [1]: https://docs.python.org/2/library/functions.html#func-repr > [2]: https://docs.python.org/3.4/library/reprlib.html > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at selik.org Mon Feb 8 18:54:37 2016 From: mike at selik.org (Michael Selik) Date: Mon, 08 Feb 2016 23:54:37 +0000 Subject: [Python-ideas] List Comprehensions In-Reply-To: References: <20160203110403.GK31806@ando.pearwood.info> Message-ID: On Wed, Feb 3, 2016 at 6:14 AM Chris Angelico wrote: > On Wed, Feb 3, 2016 at 10:04 PM, Steven D'Aprano > wrote: > > I've sometimes thought that Python should have a iterator concatenation > > operator (perhaps &) which we could use: > > > > [x for x in range(5) & [100] & "spam"] > > => returns [0, 1, 2, 3, 4, 100, 's', 'p', 'a', 'm'] > > Might be problematic for generic iterables, as your three examples > are; but if you explicitly request an iterator, it's not hard to make > it support + or & for chaining: > > from itertools import chain > > _iter = iter > class iter: > def __init__(self, *args): > self.iter = _iter(*args) > def __add__(self, other): > return type(self)(chain(self.iter, _iter(other))) > __and__ = __add__ > def __iter__(self): return self > def __next__(self): return next(self.iter) > > print(list(iter(range(5)) & [100] & "spam")) > > Is that good enough? > For some reason, the use of an operator there makes me feel like the right-hand argument could be a non-iterable that is appended. ChainableRange(10) & [100] # versus ChainableRange(10) & 100 As opposed to the more explicit use of ``itertools.chain`` which makes it more clear (in my head) that the second argument must be an iterable chain(range(10), [100]) chain(range(10), 100) # will cause TypeError: 'int' object is not iterable I could imagine a module of fancy iterables overloading __and__ for that purpose, but it ought to be a large enough chunk of functionality to warrant a mini-language. Like NumPy. As much as I dislike the term "domain-specific language", I think this example of overloading operators falls into that category. One could take the itertools module and map all sorts of operators to its features: a & b ==> chain(a, b) it[j:k] ==> islice(it, j, k) +it ==> tee(it) a * b ==> product(a, b) 2-it ==> pairwise(it) etc. Not saying it's a good idea for the standard library, but I could see a module that makes extensive use of itertools doing so. It's worth repeating that I prefer using the word instead of the operator for most if not all of these cases. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Feb 8 18:57:19 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 9 Feb 2016 10:57:19 +1100 Subject: [Python-ideas] List Comprehensions In-Reply-To: References: <20160203110403.GK31806@ando.pearwood.info> Message-ID: On Tue, Feb 9, 2016 at 10:54 AM, Michael Selik wrote: > I could imagine a module of fancy iterables overloading __and__ for that > purpose, but it ought to be a large enough chunk of functionality to warrant > a mini-language. Like NumPy. As much as I dislike the term "domain-specific > language", I think this example of overloading operators falls into that > category. One could take the itertools module and map all sorts of operators > to its features: > > a & b ==> chain(a, b) > it[j:k] ==> islice(it, j, k) > +it ==> tee(it) > a * b ==> product(a, b) > 2-it ==> pairwise(it) > > etc. Not saying it's a good idea for the standard library, but I could see a > module that makes extensive use of itertools doing so. It's worth repeating > that I prefer using the word instead of the operator for most if not all of > these cases. And the cool part of it is that anyone can do this simply by creating a class called "iter", and using that everywhere. A simple "from magic import iter" will enable this for the people who want it AND make it possible for everyone else to figure out what's going on. Personally, I don't use the itertools functions anywhere near enough to justify magic syntax, but there are people who do. ChrisA From mike at selik.org Mon Feb 8 19:28:30 2016 From: mike at selik.org (Michael Selik) Date: Tue, 09 Feb 2016 00:28:30 +0000 Subject: [Python-ideas] Compact repr operator (i.e., __short_repr__) In-Reply-To: References: Message-ID: On Mon, Feb 8, 2016 at 6:36 PM Mahmoud Hashemi wrote: > Without something like this, there's no way to ask an object if it's repr > is of manageable length, or how its repr could be meaningfully shortened. > Usually I just chop out the middle and add an ellipsis, but as for the time > spent generating that middle, I'll never get those cycles back. > With the addition of ``__short_repr__`` would there be a change to the default Python REPL? If not, how would you use the new magic method? If so, then why not just change __repr__ for all the built-ins to truncate automatically? As far as I know, reprs are not part of the backwards compatibility guarantee. (Note, I'm not advocating for changing reprs.) If the usage would be to write a function that checks for the existence of __short_repr__, then why not simply move the implementation of ``if len(s) < maxlen`` etc. to that function? -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Mon Feb 8 21:12:25 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 9 Feb 2016 11:12:25 +0900 Subject: [Python-ideas] List Comprehensions In-Reply-To: References: <20160203110403.GK31806@ando.pearwood.info> Message-ID: <22201.19209.716288.387770@turnbull.sk.tsukuba.ac.jp> Michael Selik writes: > One could take the itertools module and map all sorts of operators > to its features: One could, but I find it hard to think of use cases where you need a concise operator language for dealing with general iterables. The only case I can think of is array algebra, which could deal with general iterables but in practice limits itself to fixed-dimension arrays. Even that has been perennially controversial despite the wealth of applications (and the passion of its advocates :-). From mahmoud at hatnote.com Mon Feb 8 21:49:18 2016 From: mahmoud at hatnote.com (Mahmoud Hashemi) Date: Mon, 8 Feb 2016 18:49:18 -0800 Subject: [Python-ideas] Compact repr operator (i.e., __short_repr__) In-Reply-To: References: Message-ID: Roundtrippable reprs are certainly part of Python canon, whether or not they are universally used (Chris), or guaranteed (Mike). I could see __short_repr__ (and associated reprlib) being the desired behavior in some console environments, but I'm not one to say it should be the default in the main Python REPL. My use cases are for 1) a web based console/REPL and 2) a configuration store that maintains a human readable history of past values (i.e., it does not maintain references to the objects themselves). But mostly I wanted to kick off the discussion of how to update reprlib (and pprint) to be more efficient and applicable. Mahmoud On Mon, Feb 8, 2016 at 4:28 PM, Michael Selik wrote: > On Mon, Feb 8, 2016 at 6:36 PM Mahmoud Hashemi > wrote: > >> Without something like this, there's no way to ask an object if it's repr >> is of manageable length, or how its repr could be meaningfully shortened. >> Usually I just chop out the middle and add an ellipsis, but as for the time >> spent generating that middle, I'll never get those cycles back. >> > > With the addition of ``__short_repr__`` would there be a change to the > default Python REPL? If not, how would you use the new magic method? If so, > then why not just change __repr__ for all the built-ins to truncate > automatically? As far as I know, reprs are not part of the backwards > compatibility guarantee. (Note, I'm not advocating for changing reprs.) > > If the usage would be to write a function that checks for the existence of > __short_repr__, then why not simply move the implementation of ``if len(s) > < maxlen`` etc. to that function? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon Feb 8 23:19:44 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 09 Feb 2016 17:19:44 +1300 Subject: [Python-ideas] Compact repr operator (i.e., __short_repr__) In-Reply-To: References: Message-ID: <56B968E0.1070709@canterbury.ac.nz> Mahmoud Hashemi wrote: > I was curious if what kind of interest there would be in adding an > overridable method to types to allow getting back a repr, but a > predictably short one. If you just want to know the type and identity of an object, you can use object.__repr__(x). -- Greg From ben+python at benfinney.id.au Mon Feb 8 23:47:43 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 09 Feb 2016 15:47:43 +1100 Subject: [Python-ideas] Compact repr operator (i.e., __short_repr__) References: <56B968E0.1070709@canterbury.ac.nz> Message-ID: <8560xy8p74.fsf@benfinney.id.au> Greg Ewing writes: > Mahmoud Hashemi wrote: > > I was curious if what kind of interest there would be in adding an > > overridable method to types to allow getting back a repr, but a > > predictably short one. > > If you just want to know the type and identity of an object, > you can use object.__repr__(x). Or, to avoid calling dunder methods directly, use ?type(x)? and ?id(x)?. Whether all these suggestions address the stated requirement will depend on how predictable is meant by ?predictably short?. -- \ ?All persons, living and dead, are purely coincidental.? | `\ ?_Timequake_, Kurt Vonnegut | _o__) | Ben Finney From lkb.teichmann at gmail.com Tue Feb 9 02:44:39 2016 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Tue, 9 Feb 2016 08:44:39 +0100 Subject: [Python-ideas] Simpler Customization of Class Creation - PEP 487 In-Reply-To: References: Message-ID: > This new ProvisionalMeta will allow multiple inheritance of metaclasses > using standard left-to-right priority (or the MRO for more depth). In the > case of two metaclasses, the code from Left will run immediately after the > class is created and the code from Right will go next. Right? (joking. I > should say "Correct?") No. The upshot is: metaclasses are complicated, especially if you want to combine two of them. But most uses of the usecases fall into one of the three mentioned categories. So we just write one metaclass that serves those use cases, everyone can simply use the very same metaclass, so the are no conflicts anymore as there *is* only one metaclass. Eventually, this one metaclass shall become the builtin type, and so we don't need metaclasses anymore in most usecases. The proposed metaclass (current working title: types.Type) will call a hook for users to customize its behavior. This hook will use the normal Python cooperative multiple inheritance scheme, which also has its very own problems, but known and solvable problems. The current metaclasses, on the other hand, once you use two of them open a complete new box of pandora. Greetings Martin From cory at lukasa.co.uk Tue Feb 9 03:56:46 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Tue, 9 Feb 2016 08:56:46 +0000 Subject: [Python-ideas] Compact repr operator (i.e., __short_repr__) In-Reply-To: References: Message-ID: <28D7B647-C173-4B1D-BD74-DE62768FFCC0@lukasa.co.uk> > On 9 Feb 2016, at 02:49, Mahmoud Hashemi wrote: > > Roundtrippable reprs are certainly part of Python canon, whether or not they are universally used (Chris), or guaranteed (Mike). > They can be part of the canon all they want, but if they?re not universally guaranteed then I don?t know that this is a real problem. It means that the world of Python objects divides into two kinds: first, those with __repr__ return values that can be round tripped and those with __repr__ return values that cannot be round tripped. Given that objects of the second type already exist (I know this for a fact because I have written some quite recently!), it would be an error to assume that the identity 'eval(repr(x)) == x? holds for arbitrary types. In fact, not only does it not hold for all third-party types, it doesn?t even hold for all built-in types: >>> x = float(?nan?) >>> repr(x) ?nan? >>> eval(repr(x)) Traceback (most recent call last): File "", line 1, in File "", line 1, in NameError: name 'nan' is not defined I think the reality is that there is no constraint on the representation of arbitrary types to be round-trippable in any way. Again, all custom types have non-round-trippable representations by default, many more eclectic built-in types have non-round-tripppable representations (in addition to NaN, the memoryview object leaps to mind). I can also note the Python documentation on repr: > For many types, this function makes an attempt to return a string that would yield an object with the same value when passed to eval(), otherwise the representation is a string enclosed in angle brackets that contains the name of the type of the object together with additional information often including the name and address of the object. If the language doesn?t even try to enforce the idea that representations will be round-trippable, I think there?s just no problem here. Cory -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mahmoud at hatnote.com Tue Feb 9 04:09:04 2016 From: mahmoud at hatnote.com (Mahmoud Hashemi) Date: Tue, 9 Feb 2016 01:09:04 -0800 Subject: [Python-ideas] Compact repr operator (i.e., __short_repr__) In-Reply-To: <28D7B647-C173-4B1D-BD74-DE62768FFCC0@lukasa.co.uk> References: <28D7B647-C173-4B1D-BD74-DE62768FFCC0@lukasa.co.uk> Message-ID: If you want to make the case that default reprs should be non-roundtrippable in the case that they're too long, that's a fine and separate discussion. Though I have argued in the past that float('nan') would be a less surprising/more useful repr. And that's what it's about, usefulness. It's useful to be able to copy in and out of the REPL, even for very large reprs. I've done it, my coworkers and students do it, and you've probably done it as well. But there are other times (often outside the REPL), where that is not the case, and being able to address them explicitly, in the vein of reprlib and pprint -- but better -- would be *useful*. Who hasn't wished that the built-in defaultdict and OrderedDict were as pprintable or reprlib.repr-able as dict. There's plenty of room to improve. On Tue, Feb 9, 2016 at 12:56 AM, Cory Benfield wrote: > > > On 9 Feb 2016, at 02:49, Mahmoud Hashemi wrote: > > > > Roundtrippable reprs are certainly part of Python canon, whether or not > they are universally used (Chris), or guaranteed (Mike). > > > > They can be part of the canon all they want, but if they?re not > universally guaranteed then I don?t know that this is a real problem. It > means that the world of Python objects divides into two kinds: first, those > with __repr__ return values that can be round tripped and those with > __repr__ return values that cannot be round tripped. > > Given that objects of the second type already exist (I know this for a > fact because I have written some quite recently!), it would be an error to > assume that the identity 'eval(repr(x)) == x? holds for arbitrary types. In > fact, not only does it not hold for all third-party types, it doesn?t even > hold for all built-in types: > > >>> x = float(?nan?) > >>> repr(x) > ?nan? > >>> eval(repr(x)) > Traceback (most recent call last): > File "", line 1, in > File "", line 1, in > NameError: name 'nan' is not defined > > I think the reality is that there is no constraint on the representation > of arbitrary types to be round-trippable in any way. Again, all custom > types have non-round-trippable representations by default, many more > eclectic built-in types have non-round-tripppable representations (in > addition to NaN, the memoryview object leaps to mind). > > I can also note the Python documentation on repr: > > > For many types, this function makes an attempt to return a string that > would yield an object with the same value when passed to eval(), otherwise > the representation is a string enclosed in angle brackets that contains the > name of the type of the object together with additional information often > including the name and address of the object. > > If the language doesn?t even try to enforce the idea that representations > will be round-trippable, I think there?s just no problem here. > > Cory > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at selik.org Tue Feb 9 04:54:27 2016 From: mike at selik.org (Michael Selik) Date: Tue, 09 Feb 2016 09:54:27 +0000 Subject: [Python-ideas] Simpler Customization of Class Creation - PEP 487 In-Reply-To: References: Message-ID: On Tue, Feb 9, 2016, 2:44 AM Martin Teichmann wrote: > > This new ProvisionalMeta will allow multiple inheritance of metaclasses > > using standard left-to-right priority (or the MRO for more depth). In the > > case of two metaclasses, the code from Left will run immediately after > the > > class is created and the code from Right will go next. Right? (joking. I > > should say "Correct?") > > No. The upshot is: metaclasses are complicated, especially if you want > to combine two of them. But most uses of the usecases fall into one of > the three mentioned categories. So we just write one metaclass that > serves those use cases, everyone can simply use the very same metaclass, > so the are no conflicts anymore as there *is* only one metaclass. > Eventually, > this one metaclass shall become the builtin type, and so we don't need > metaclasses anymore in most usecases. > Gotcha. I misread the PEP. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Tue Feb 9 05:13:22 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 9 Feb 2016 02:13:22 -0800 Subject: [Python-ideas] Compact repr operator (i.e., __short_repr__) In-Reply-To: <28D7B647-C173-4B1D-BD74-DE62768FFCC0@lukasa.co.uk> References: <28D7B647-C173-4B1D-BD74-DE62768FFCC0@lukasa.co.uk> Message-ID: <5FC0322A-7ECF-448A-8D87-4A9341EBBA88@yahoo.com> On Feb 9, 2016, at 00:56, Cory Benfield wrote: > > >> On 9 Feb 2016, at 02:49, Mahmoud Hashemi wrote: >> >> Roundtrippable reprs are certainly part of Python canon, whether or not they are universally used (Chris), or guaranteed (Mike). > > They can be part of the canon all they want, but if they?re not universally guaranteed then I don?t know that this is a real problem. It means that the world of Python objects divides into two kinds: first, those with __repr__ return values that can be round tripped and those with __repr__ return values that cannot be round tripped. So? One use of round-trippable reprs is to copy them from output and paste them into source code or an interactive session. When doing so, you almost always know that you're dealing with a builtin or third-party type that's round-trippable--and, when you're surprised, it's almost always obvious, because you get something that looks nothing at all like the source equivalent, and that raises a SyntaxError if you try to evaluate it anyway. And even those "almost"s aren't a problem in practice. Sure, a list that recursively contains itself looks misleadingly round-trippable, and will evaluate successfully into the wrong thing. But this rarely comes up in practice--and, if it does, because there's a human being inspecting, debugging, or playing with things, rather than a program, it's easy to deal with. This isn't theoretical--I do this all the time when debugging code, I write my own types to make them easier to debug this way, and it saves me time and hassle. In fact, it's one of the benefits of using Python over some of the other languages I use, where repr or toDebugString or whatever is never useful, instead of being usually useful especially with common types. The only thing the non-universality of round-tripping means is that you can't use repr with eval as a persistence format. Which is a good thing--you *shouldn't* use it as a persistence format, and that would be true even if it did work. But using it as an inspecting/debugging format is not a problem, and breaking that would be a bad idea. In fact, breaking it would make repr nearly pointless. Except for a types that define __repr__ but block __str__ with a TypeError (which is rare, and it's debatable whether those types are even valid), when would you ever use repr otherwise? From cory at lukasa.co.uk Tue Feb 9 06:13:13 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Tue, 9 Feb 2016 11:13:13 +0000 Subject: [Python-ideas] Compact repr operator (i.e., __short_repr__) In-Reply-To: <5FC0322A-7ECF-448A-8D87-4A9341EBBA88@yahoo.com> References: <28D7B647-C173-4B1D-BD74-DE62768FFCC0@lukasa.co.uk> <5FC0322A-7ECF-448A-8D87-4A9341EBBA88@yahoo.com> Message-ID: > On 9 Feb 2016, at 10:13, Andrew Barnert wrote: > > So? > > One use of round-trippable reprs is to copy them from output and paste them into source code or an interactive session. When doing so, you almost always know that you're dealing with a builtin or third-party type that's round-trippable--and, when you're surprised, it's almost always obvious, because you get something that looks nothing at all like the source equivalent, and that raises a SyntaxError if you try to evaluate it anyway. > > And even those "almost"s aren't a problem in practice. Sure, a list that recursively contains itself looks misleadingly round-trippable, and will evaluate successfully into the wrong thing. But this rarely comes up in practice--and, if it does, because there's a human being inspecting, debugging, or playing with things, rather than a program, it's easy to deal with. > > This isn't theoretical--I do this all the time when debugging code, I write my own types to make them easier to debug this way, and it saves me time and hassle. In fact, it's one of the benefits of using Python over some of the other languages I use, where repr or toDebugString or whatever is never useful, instead of being usually useful especially with common types. > > The only thing the non-universality of round-tripping means is that you can't use repr with eval as a persistence format. Which is a good thing--you *shouldn't* use it as a persistence format, and that would be true even if it did work. But using it as an inspecting/debugging format is not a problem, and breaking that would be a bad idea. > > In fact, breaking it would make repr nearly pointless. Except for a types that define __repr__ but block __str__ with a TypeError (which is rare, and it's debatable whether those types are even valid), when would you ever use repr otherwise? I don?t think we?re arguing the same point. I?m not saying that __repr__ shouldn?t be round-trippable: if that makes sense for your type then totally fine, go for it. However, I am saying that I don?t see the advantage in having *both* a round-trippable and non-round-trippable repr for the same type. If you?re copying and pasting then the length of the round-trippable representation is a non-issue, and you actively want it to appear in your debug output rather than a shorter version that elides information required to round-trip the data. So in this case, what purpose does the shorter version serve? Cory -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ceronman at gmail.com Tue Feb 9 16:40:18 2016 From: ceronman at gmail.com (=?UTF-8?Q?Manuel_Cer=C3=B3n?=) Date: Tue, 9 Feb 2016 22:40:18 +0100 Subject: [Python-ideas] Improve readability of long numeric literals Message-ID: Hi everyone! Sometimes it's hard to read long numbers. For example: >>> opts.write_buffer_size = 67108864 Some languages (Ruby, Perl, Swift) allow the use of underscores in numeric literals, which are ignored. They are typically used as thousands separators. The example above would look like this: >>> opts.write_buffer_size = 67_108_864 Which helps to quickly identify that this is around 67 million. Another option is to use spaces instead of underscores: >>> opts.write_buffer_size = 67 108 864 This has two advantages: 1. is analog to the way string literals work, which are concatenated if put next to each other. 2. spaces are already used as thousands separator in many european languages [1]. The disadvantage is that, as far as I known, no other languages do this. I have seen some old discussions around this, but nothing on this list or a PEP. With Python being use more and more for scientific and numeric computation, this is a small change that will help with readability a lot. And, as far as I can tell, it doesn't break compatibility in any way. Thoughts? Manuel. [1] https://docs.oracle.com/cd/E19455-01/806-0169/overview-9/index.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From ian.g.kelly at gmail.com Tue Feb 9 17:38:10 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Tue, 9 Feb 2016 15:38:10 -0700 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: Message-ID: On Tue, Feb 9, 2016 at 2:40 PM, Manuel Cer?n wrote: > Another option is to use spaces instead of underscores: > >>>> opts.write_buffer_size = 67 108 864 > > This has two advantages: 1. is analog to the way string literals work, which > are concatenated if put next to each other. 2. spaces are already used as > thousands separator in many european languages [1]. > > The disadvantage is that, as far as I known, no other languages do this. Another disadvantage to using spaces is that it could mask an inadvertently omitted operator that previously would have been a SyntaxError. From ethan at stoneleaf.us Tue Feb 9 18:02:38 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 09 Feb 2016 15:02:38 -0800 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: Message-ID: <56BA700E.7010305@stoneleaf.us> On 02/09/2016 01:40 PM, Manuel Cer?n wrote: > Sometimes it's hard to read long numbers. For example: > > >>> opts.write_buffer_size = 67108864 > > Some languages (Ruby, Perl, Swift) allow the use of underscores in > numeric literals, which are ignored. They are typically used as > thousands separators. The example above would look like this: > > >>> opts.write_buffer_size = 67_108_864 As I recall, a number of years ago we had this discussion and Guido approved the idea. The only email I could locate at the moment, though, shows his support of the idea, but not outright approval. [1] I dare say if somebody submitted a patch it would fare well. (As in: be accepted, not gone forever.) -- ~Ethan~ [1] https://mail.python.org/pipermail/python-ideas/2011-May/010157.html From guido at python.org Tue Feb 9 18:17:36 2016 From: guido at python.org (Guido van Rossum) Date: Tue, 9 Feb 2016 15:17:36 -0800 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: <56BA700E.7010305@stoneleaf.us> References: <56BA700E.7010305@stoneleaf.us> Message-ID: Indeed, "123 456" is a no-no, but "123_456" sound good. (Not sure about "12_34_56" but there are probably use cases for that too.) On Tue, Feb 9, 2016 at 3:02 PM, Ethan Furman wrote: > On 02/09/2016 01:40 PM, Manuel Cer?n wrote: > >> Sometimes it's hard to read long numbers. For example: >> >> >>> opts.write_buffer_size = 67108864 >> >> Some languages (Ruby, Perl, Swift) allow the use of underscores in >> numeric literals, which are ignored. They are typically used as >> thousands separators. The example above would look like this: >> >> >>> opts.write_buffer_size = 67_108_864 > > > As I recall, a number of years ago we had this discussion and Guido approved > the idea. The only email I could locate at the moment, though, shows his > support of the idea, but not outright approval. [1] > > I dare say if somebody submitted a patch it would fare well. (As in: be > accepted, not gone forever.) > > -- > ~Ethan~ > > > > [1] https://mail.python.org/pipermail/python-ideas/2011-May/010157.html > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From oscar.j.benjamin at gmail.com Tue Feb 9 18:27:38 2016 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Tue, 9 Feb 2016 23:27:38 +0000 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: <56BA700E.7010305@stoneleaf.us> Message-ID: On 9 Feb 2016 23:18, "Guido van Rossum" wrote: > > Indeed, "123 456" is a no-no, but "123_456" sound good. (Not sure > about "12_34_56" but there are probably use cases for that too.) > It would be useful for hex literals. There are other more confusing possibilities such as 1_._0_e_-_1_0. -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.cliffe at btinternet.com Tue Feb 9 18:44:49 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Tue, 9 Feb 2016 23:44:49 +0000 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: <56BA700E.7010305@stoneleaf.us> Message-ID: <56BA79F1.3050405@btinternet.com> On 09/02/2016 23:17, Guido van Rossum wrote: > Indeed, "123 456" is a no-no, but "123_456" sound good. (Not sure > about "12_34_56" but there are probably use cases for that too.) Looks to me like a bank sort code. Probably not Guido's, to judge by his comment. From guido at python.org Tue Feb 9 18:51:35 2016 From: guido at python.org (Guido van Rossum) Date: Tue, 9 Feb 2016 15:51:35 -0800 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: <56BA79F1.3050405@btinternet.com> References: <56BA700E.7010305@stoneleaf.us> <56BA79F1.3050405@btinternet.com> Message-ID: I don't know what a bank sort code is (maybe a UK thing?) FWIW there are some edge cases to be decided: is _123 valid? or 123_? or 123__456? On Tue, Feb 9, 2016 at 3:44 PM, Rob Cliffe wrote: > > > On 09/02/2016 23:17, Guido van Rossum wrote: >> >> Indeed, "123 456" is a no-no, but "123_456" sound good. (Not sure >> about "12_34_56" but there are probably use cases for that too.) > > Looks to me like a bank sort code. Probably not Guido's, to judge by his > comment. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From ben+python at benfinney.id.au Tue Feb 9 19:06:20 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 10 Feb 2016 11:06:20 +1100 Subject: [Python-ideas] Improve readability of long numeric literals References: Message-ID: <85ziv977k3.fsf@benfinney.id.au> Ian Kelly writes: > On Tue, Feb 9, 2016 at 2:40 PM, Manuel Cer?n wrote: > > Another option is to use spaces instead of underscores: > > > >>>> opts.write_buffer_size = 67 108 864 > > > > [?] > > The disadvantage is that, as far as I known, no other languages do > > this. > > Another disadvantage to using spaces is that it could mask an > inadvertently omitted operator that previously would have been a > SyntaxError. The exact same fact ? that a proposed new syntax was previously a syntax error ? is commonly presented as a positive. We know there are no valid Python programs already using the construct to mean something else. So I don't think it's reasonable to present that now as though it were negative. (good sigmonster, have a cookie) -- \ ?? correct code is great, code that crashes could use | `\ improvement, but incorrect code that doesn?t crash is a | _o__) horrible nightmare.? ?Chris Smith, 2008-08-22 | Ben Finney From guido at python.org Tue Feb 9 19:16:19 2016 From: guido at python.org (Guido van Rossum) Date: Tue, 9 Feb 2016 16:16:19 -0800 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: <85ziv977k3.fsf@benfinney.id.au> References: <85ziv977k3.fsf@benfinney.id.au> Message-ID: On Tue, Feb 9, 2016 at 4:06 PM, Ben Finney wrote: > Ian Kelly writes: > >> On Tue, Feb 9, 2016 at 2:40 PM, Manuel Cer?n wrote: >> > Another option is to use spaces instead of underscores: >> > >> >>>> opts.write_buffer_size = 67 108 864 >> > >> > [?] >> > The disadvantage is that, as far as I known, no other languages do >> > this. >> >> Another disadvantage to using spaces is that it could mask an >> inadvertently omitted operator that previously would have been a >> SyntaxError. > > The exact same fact ? that a proposed new syntax was previously a syntax > error ? is commonly presented as a positive. We know there are no valid > Python programs already using the construct to mean something else. > > So I don't think it's reasonable to present that now as though it were > negative. I think you misunderstand. The argument (which I agree with) is that the syntax error was considered beneficial, since it would catch common typos -- so that we're loath to make that valid code. -- --Guido van Rossum (python.org/~guido) From oscar.j.benjamin at gmail.com Tue Feb 9 19:16:43 2016 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 10 Feb 2016 00:16:43 +0000 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: <56BA700E.7010305@stoneleaf.us> <56BA79F1.3050405@btinternet.com> Message-ID: On 9 February 2016 at 23:51, Guido van Rossum wrote: > I don't know what a bank sort code is (maybe a UK thing?) It is a UK thing. It identifies the bank you opened your account with. > FWIW there are some edge cases to be decided: is _123 valid? or 123_? > or 123__456? _123 is currently a valid identifier: >>> _123 = 1 >>> _123 1 123_ is not. There's no good reason to allow either though. If the purpose is to separate the digits for clarity then the underscore doesn't need to be at the beginning or the end. -- Oscar From ceronman at gmail.com Tue Feb 9 19:22:23 2016 From: ceronman at gmail.com (=?UTF-8?Q?Manuel_Cer=C3=B3n?=) Date: Wed, 10 Feb 2016 01:22:23 +0100 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: <56BA700E.7010305@stoneleaf.us> <56BA79F1.3050405@btinternet.com> Message-ID: On Wed, Feb 10, 2016 at 12:51 AM, Guido van Rossum wrote: > I don't know what a bank sort code is (maybe a UK thing?) > > FWIW there are some edge cases to be decided: is _123 valid? or 123_? > or 123__456? > _123 is a valid identifier name, so no. For consistency, I think the leading underscore should be out too. Multiple underscores in the middle might be useful for separating millions and thousands: 700__000_000 but perhaps it's too much. -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Tue Feb 9 19:45:58 2016 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 10 Feb 2016 00:45:58 +0000 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: <56BA700E.7010305@stoneleaf.us> Message-ID: <56BA8846.8060904@mrabarnett.plus.com> On 2016-02-09 23:27, Oscar Benjamin wrote: > On 9 Feb 2016 23:18, "Guido van Rossum" > wrote: > > > > Indeed, "123 456" is a no-no, but "123_456" sound good. (Not sure > > about "12_34_56" but there are probably use cases for that too.) > > > > It would be useful for hex literals. There are other more confusing > possibilities such as 1_._0_e_-_1_0. > The Ada programming language allows underscores in numerals, but requires there to be a digit on both sides of the underscore. From ethan at stoneleaf.us Tue Feb 9 19:54:56 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 09 Feb 2016 16:54:56 -0800 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: <85ziv977k3.fsf@benfinney.id.au> References: <85ziv977k3.fsf@benfinney.id.au> Message-ID: <56BA8A60.9040508@stoneleaf.us> On 02/09/2016 04:06 PM, Ben Finney wrote: > Ian Kelly writes: >> Another disadvantage to using spaces is that it could mask an >> inadvertently omitted operator that previously would have been a >> SyntaxError. > > The exact same fact ? that a proposed new syntax was previously a > syntax error ? is commonly presented as a positive. We know there are > no valid Python programs already using the construct to mean > something else. > > So I don't think it's reasonable to present that now as though it were > negative. If the SyntaxError is also a common mistake, then suddenly having it be working, but wrong, code is a bad thing. > (good sigmonster, have a cookie) ?? correct code is great, code that crashes could use | improvement, but incorrect code that doesn?t crash is a | horrible nightmare.? ?Chris Smith, 2008-08-22 | Yes, good sigmonster - "incorrect code that doesn't crash is a horrible nightmare" -- which is what could happen when a SyntaxError suddenly becomes syntacticly correct. -- ~Ethan~ From steve at pearwood.info Tue Feb 9 19:50:41 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 10 Feb 2016 11:50:41 +1100 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: <56BA700E.7010305@stoneleaf.us> <56BA79F1.3050405@btinternet.com> Message-ID: <20160210005041.GD31806@ando.pearwood.info> On Wed, Feb 10, 2016 at 12:16:43AM +0000, Oscar Benjamin wrote: > _123 is currently a valid identifier: > > >>> _123 = 1 > >>> _123 > 1 > > 123_ is not. There's no good reason to allow either though. If the > purpose is to separate the digits for clarity then the underscore > doesn't need to be at the beginning or the end. Agreed. Disallow leading and trailing underscores, otherwise allow and ignore any number of underscores in integer literals so that all of these are legal: 123_456_789 0x1234_ABCD 0b1111_0000_1010_0101 0o12_34 For avoidance of doubt, there must be at least one digit before the first underscore. These are not allowed: -_123_456 +_123_456 (actually, they are allowed, since they're legal identifiers). Consecutive underscores will be allowed: 1234____5678 but the docs (PEP 8?) should say "don't do that". Likewise for excessive underscores: 1_2_3_4_5_6_7_8_9_0 These sorts of abuses are a style issue, not a syntax issue. Floats are more complex: 123_456.000_001e-23 looks okay to me, but what about this? 123_456_._000_001_e_-_23 I think that's ugly. Should we restrict underscores to being only between digits? Or just call that a style issue too? -- Steve From ethan at stoneleaf.us Tue Feb 9 20:07:53 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 09 Feb 2016 17:07:53 -0800 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: <56BA700E.7010305@stoneleaf.us> Message-ID: <56BA8D69.8040108@stoneleaf.us> To excerpt the email I referred to earlier: Guido said: ---------- > Fine points about _ in floats: IMO the _ should be allowed to appear > between any two digits, or between the last digit and the 'e' in the > exponent, or between the 'e' and a following digit. But not adjacent > to the '.' or to the '+' or '-' in the exponent. So 3.141_593 yes, > 3_.14 no. > > Fine points about _ in bin/oct/hex literals: 0x_dead_beef yes, 0_xdeadbeef no. > > (The overall rule seems to be that it must be internal to alphanumeric > strings, except that leading 0x, 0o or 0b must not be separated -- > somehow I find 0_x_dead_beef would be a disservice to human readers.) and in talking about int() accepting underscored inputs: > It seems entirely harmless here. Also for float(). -- ~Ethan~ From abarnert at yahoo.com Tue Feb 9 20:35:42 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 9 Feb 2016 17:35:42 -0800 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: Message-ID: One possible objection that nobody's raised: Separating groups of three is all well and good; to my western eyes, 10_000_000_000 is obviously 10 billion.* But someone from China is likely to use groups of four, and 100_0000_0000 is not obviously anything--my first thought is around 100 billion, but that can't be right, so I have to count up the digits. I still think this is a good suggestion, because 100000000000 is even more useless to me as 100_0000_0000, and far more likely to be concealing a typo. I just wanted to make sure everyone knew the issue. * If you're going to say "no, it's a milliard, you stupid American", go back to the early 70s, and bring your non-decimal currency with you. It's billion in English, and has been for 40+ years. Leave the fighting to the languages where it's ambiguous, like Portuguese or Finnish. Sent from my iPhone > On Feb 9, 2016, at 13:40, Manuel Cer?n wrote: > > Hi everyone! > > Sometimes it's hard to read long numbers. For example: > > >>> opts.write_buffer_size = 67108864 > > Some languages (Ruby, Perl, Swift) allow the use of underscores in numeric literals, which are ignored. They are typically used as thousands separators. The example above would look like this: > > >>> opts.write_buffer_size = 67_108_864 > > Which helps to quickly identify that this is around 67 million. > > Another option is to use spaces instead of underscores: > > >>> opts.write_buffer_size = 67 108 864 > > This has two advantages: 1. is analog to the way string literals work, which are concatenated if put next to each other. 2. spaces are already used as thousands separator in many european languages [1]. > > The disadvantage is that, as far as I known, no other languages do this. > > I have seen some old discussions around this, but nothing on this list or a PEP. With Python being use more and more for scientific and numeric computation, this is a small change that will help with readability a lot. And, as far as I can tell, it doesn't break compatibility in any way. > > Thoughts? > > Manuel. > > [1] https://docs.oracle.com/cd/E19455-01/806-0169/overview-9/index.html > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Tue Feb 9 20:35:29 2016 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 10 Feb 2016 01:35:29 +0000 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: <56BA8D69.8040108@stoneleaf.us> References: <56BA700E.7010305@stoneleaf.us> <56BA8D69.8040108@stoneleaf.us> Message-ID: On 10 February 2016 at 01:07, Ethan Furman wrote: > > and in talking about int() accepting underscored inputs: > >> It seems entirely harmless here. Also for float(). I don't agree with either of those. Syntax accepted by int() is less permissive than for int literals (e.g. int('0x1')) which is good because int is often used to process data form external sources. In this vain I'm not sure how I feel about int accepting non-ascii characters - perhaps there should be a separate int.from_ascii function for this purpose but that's a different subject. Having float() accept underscored inputs violates IEEE754. That doesn't mean it's impossible but why bother? -- Oscar From rosuav at gmail.com Tue Feb 9 20:39:08 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 10 Feb 2016 12:39:08 +1100 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: <56BA700E.7010305@stoneleaf.us> <56BA8D69.8040108@stoneleaf.us> Message-ID: On Wed, Feb 10, 2016 at 12:35 PM, Oscar Benjamin wrote: > On 10 February 2016 at 01:07, Ethan Furman wrote: >> >> and in talking about int() accepting underscored inputs: >> >>> It seems entirely harmless here. Also for float(). > > I don't agree with either of those. Syntax accepted by int() is less > permissive than for int literals (e.g. int('0x1')) which is good > because int is often used to process data form external sources. +1. Keep int() as it is, so we don't get weird stuff happening and causing confusion. But I'm +1 on allowing underscores between digits (not straight after a decimal point in a float, though, as it looks like attribute access). ChrisA From abarnert at yahoo.com Tue Feb 9 20:45:11 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 9 Feb 2016 17:45:11 -0800 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: <56BA8846.8060904@mrabarnett.plus.com> References: <56BA700E.7010305@stoneleaf.us> <56BA8846.8060904@mrabarnett.plus.com> Message-ID: <3F9F566F-7521-43CC-A9AD-97EDA81552EA@yahoo.com> On Feb 9, 2016, at 16:45, MRAB wrote: > > The Ada programming language allows underscores in numerals, but requires there to be a digit on both sides of the underscore. I think Swift, Ruby, and most other languages allow runs of multiple underscores, and even trailing underscores. It seems like it's a lot easier to lex "digit (digit-or-underscore)*", "0x (hex-digit-or-underscore)+", etc. than to try to add restrictions. And not just for Python itself, but for anyone who wants to write a Python tokenizer or parser. And it's a shorter rule to document, and easier to remember. So, unless there's a really compelling reason for an extra restriction, I think it's better to leave the restrictions out (and make them style issues). From joshua at landau.ws Tue Feb 9 21:18:53 2016 From: joshua at landau.ws (Joshua Landau) Date: Wed, 10 Feb 2016 02:18:53 +0000 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: <56BA8846.8060904@mrabarnett.plus.com> References: <56BA700E.7010305@stoneleaf.us> <56BA8846.8060904@mrabarnett.plus.com> Message-ID: On 10 February 2016 at 00:45, MRAB wrote: > The Ada programming language allows underscores in numerals, but requires > there to be a digit on both sides of the underscore. +1 to this. Nobody's given an important use-case for any of the odd cases (doubled or trailing underscores, or those oddly placed in floats) but there are legitimate reasons to want groups of different sizes, especially with non-decimal bases. Saying a digit is needed either side is both obvious and sufficient. From rosuav at gmail.com Tue Feb 9 21:46:18 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 10 Feb 2016 13:46:18 +1100 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: <56BA700E.7010305@stoneleaf.us> <56BA8846.8060904@mrabarnett.plus.com> Message-ID: On Wed, Feb 10, 2016 at 1:18 PM, Joshua Landau wrote: > On 10 February 2016 at 00:45, MRAB wrote: >> The Ada programming language allows underscores in numerals, but requires >> there to be a digit on both sides of the underscore. > > +1 to this. > > Nobody's given an important use-case for any of the odd cases (doubled > or trailing underscores, or those oddly placed in floats) but there > are legitimate reasons to want groups of different sizes, especially > with non-decimal bases. Saying a digit is needed either side is both > obvious and sufficient. I'm not sure how that would get encoded into the grammar, but certainly that's the advice I would recommend for style guides. ChrisA From joshua at landau.ws Tue Feb 9 22:32:51 2016 From: joshua at landau.ws (Joshua Landau) Date: Wed, 10 Feb 2016 03:32:51 +0000 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: <56BA700E.7010305@stoneleaf.us> <56BA8846.8060904@mrabarnett.plus.com> Message-ID: On 10 February 2016 at 02:46, Chris Angelico wrote: > On Wed, Feb 10, 2016 at 1:18 PM, Joshua Landau wrote: >> On 10 February 2016 at 00:45, MRAB wrote: >>> The Ada programming language allows underscores in numerals, but requires >>> there to be a digit on both sides of the underscore. >> >> +1 to this. > > I'm not sure how that would get encoded into the grammar, but > certainly that's the advice I would recommend for style guides. You do the BNF equivalent of turning \d\d* to \d(\d|_\d)*. It should be really simple. From stephen at xemacs.org Tue Feb 9 23:11:38 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 10 Feb 2016 13:11:38 +0900 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: Message-ID: <22202.47226.469869.136324@turnbull.sk.tsukuba.ac.jp> Andrew Barnert via Python-ideas writes: > One possible objection that nobody's raised: > > Separating groups of three is all well and good; to my western > eyes, 10_000_000_000 is obviously 10 billion.* > > But someone from China is likely to use groups of four, and > 100_0000_0000 is not obviously anything--my first thought is around > 100 billion, but that can't be right, so I have to count up the > digits. Not a problem for me any more, that's quite obviously "100-oku" (and if the unit is "yen", that roughly converts to USD100 million, very convenient). I suspect others will learn equally quickly if this becomes at all common and they need to read "Chinese" or "Japanese" code where it's used. Anyway, for me (YMMV) this really is a for-the-writer readability problem. Personally I can't imagine using it except interactively. If I think a number needs checking, I make it a named value, and often computed. (Eg, the OP's example would be 1 << 26). From dan at tombstonezero.net Tue Feb 9 23:30:11 2016 From: dan at tombstonezero.net (Dan Sommers) Date: Wed, 10 Feb 2016 04:30:11 +0000 (UTC) Subject: [Python-ideas] Improve readability of long numeric literals References: <56BA700E.7010305@stoneleaf.us> <56BA79F1.3050405@btinternet.com> <20160210005041.GD31806@ando.pearwood.info> Message-ID: On Wed, 10 Feb 2016 11:50:41 +1100, Steven D'Aprano wrote: > Floats are more complex: > > 123_456.000_001e-23 Floats are less complex. Complexes are more complex: 22j 123_456.000_001e-23j :-) > looks okay to me, but what about this? > > 123_456_._000_001_e_-_23 Agreed: that is no longer more readable. All I can think of for a use case for leading underscores would be to line up values of different lengths: x4 = __4 x5 = _33 x6 = __4 x7 = 220 milli = 1e__-3 micro = 1e__-6 nano = 1e__-9 pico = 1e_-12 femto = 1e_-15 But PEP 8 already suggests otherwise: milli = 1e-3 micro = 1e-6 nano = 1e-9 pico = 1e-12 femto = 1e-15 From guido at python.org Tue Feb 9 23:53:16 2016 From: guido at python.org (Guido van Rossum) Date: Tue, 9 Feb 2016 20:53:16 -0800 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: <56BA700E.7010305@stoneleaf.us> <56BA79F1.3050405@btinternet.com> <20160210005041.GD31806@ando.pearwood.info> Message-ID: Let me show you how silly this looked to me... On Tuesday, February 9, 2016, Dan Sommers > wrote: > On Wed, 10 Feb 2016 11:50:41 +1100, Steven D'Aprano wrote: > > > Floats are more complex: > > > > 123_456.000_001e-23 > > Floats are less complex. Complexes are more complex: > > 22j > > 123_456.000_001e-23j > > :-) > > > looks okay to me, but what about this? > > > > 123_456_._000_001_e_-_23 > > Agreed: that is no longer more readable. > > All I can think of for a use case for leading underscores would be to > line up values of different lengths: > > x4 = __4 > x5 = _33 > x6 = __4 > x7 = 220 > > milli = 1e__-3 > micro = 1e__-6 > nano = 1e__-9 > pico = 1e_-12 > femto = 1e_-15 > > But PEP 8 already suggests otherwise: > > milli = 1e-3 > micro = 1e-6 > nano = 1e-9 > pico = 1e-12 > femto = 1e-15 > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Wed Feb 10 03:53:10 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 10 Feb 2016 09:53:10 +0100 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: Message-ID: 2016-02-09 22:40 GMT+01:00 Manuel Cer?n : > Hi everyone! > > Sometimes it's hard to read long numbers. For example: > >>>> opts.write_buffer_size = 67108864 > > Some languages (Ruby, Perl, Swift) allow the use of underscores in numeric > literals, which are ignored. Yeah, I saw this (in Perl) and I think that it's a good idea. > Another option is to use spaces instead of underscores: > >>>> opts.write_buffer_size = 67 108 864 It sounds error-prone to me. It's common that I forget a comma in a tuple like : x = (1, 2 3) I expect a SyntaxError, not x = (1, 23). It can also occur on a single line: x = (1, 2 3) > I have seen some old discussions around this, but nothing on this list or a > PEP. With Python being use more and more for scientific and numeric > computation, this is a small change that will help with readability a lot. > And, as far as I can tell, it doesn't break compatibility in any way. I'm not sure that a PEP is required. We just have to clarify where underscore are allowed exactly. See the discussion above they are corner cases on float and complex numbers. Victor From p.f.moore at gmail.com Wed Feb 10 04:44:02 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 10 Feb 2016 09:44:02 +0000 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: <56BA700E.7010305@stoneleaf.us> <56BA8846.8060904@mrabarnett.plus.com> Message-ID: On 10 February 2016 at 03:32, Joshua Landau wrote: > On 10 February 2016 at 02:46, Chris Angelico wrote: >> On Wed, Feb 10, 2016 at 1:18 PM, Joshua Landau wrote: >>> On 10 February 2016 at 00:45, MRAB wrote: >>>> The Ada programming language allows underscores in numerals, but requires >>>> there to be a digit on both sides of the underscore. >>> >>> +1 to this. >> >> I'm not sure how that would get encoded into the grammar, but >> certainly that's the advice I would recommend for style guides. > > You do the BNF equivalent of turning \d\d* to \d(\d|_\d)*. It should > be really simple. It's possible to get it right, but I think keeping the grammar simple and making the rest a style issue is the best approach. We don't disallow 0x6AfEbbC for example, but mixing case like that is ugly to read too. (I was originally going to say "Under that change, "23" becomes invalid" but then I realised I'd misread the grammar. Which sort of makes my point that we want to keep the rules simple :-)) Paul From guido at python.org Tue Feb 9 23:52:58 2016 From: guido at python.org (Guido van Rossum) Date: Tue, 9 Feb 2016 20:52:58 -0800 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: <56BA700E.7010305@stoneleaf.us> <56BA79F1.3050405@btinternet.com> <20160210005041.GD31806@ando.pearwood.info> Message-ID: Let me show you how silly this looked to me... On Tuesday, February 9, 2016, Dan Sommers wrote: > On Wed, 10 Feb 2016 11:50:41 +1100, Steven D'Aprano wrote: > > > Floats are more complex: > > > > 123_456.000_001e-23 > > Floats are less complex. Complexes are more complex: > > 22j > > 123_456.000_001e-23j > > :-) > > > looks okay to me, but what about this? > > > > 123_456_._000_001_e_-_23 > > Agreed: that is no longer more readable. > > All I can think of for a use case for leading underscores would be to > line up values of different lengths: > > x4 = __4 > x5 = _33 > x6 = __4 > x7 = 220 > > milli = 1e__-3 > micro = 1e__-6 > nano = 1e__-9 > pico = 1e_-12 > femto = 1e_-15 > > But PEP 8 already suggests otherwise: > > milli = 1e-3 > micro = 1e-6 > nano = 1e-9 > pico = 1e-12 > femto = 1e-15 > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: IMG_0253.jpg Type: image/jpeg Size: 110669 bytes Desc: not available URL: From python at 2sn.net Wed Feb 10 06:37:30 2016 From: python at 2sn.net (Alexander Heger) Date: Wed, 10 Feb 2016 22:37:30 +1100 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: Message-ID: >>>> opts.write_buffer_size = 67108864 > > The disadvantage is that, as far as I known, no other languages do this. This is not true. It is absolutely legal in FORTRAN program f print*, 123 456 end will just print the number 123456. Hence for me as a FORTRAN this would seem a natural thing to do. Better than the underscores ... I would associate the with the LaTeX maths mode index operator, and then I would read 100_002 as the number 4. -Alexander From greg.ewing at canterbury.ac.nz Wed Feb 10 06:57:22 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 11 Feb 2016 00:57:22 +1300 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: <56BA700E.7010305@stoneleaf.us> Message-ID: <56BB25A2.7010103@canterbury.ac.nz> Guido van Rossum wrote: > (Not sure > about "12_34_56" but there are probably use cases for that too.) I think the Chinese group by 10,000s rather than 1000s, so they might want to write 1234_5678. -- Greg From ncoghlan at gmail.com Wed Feb 10 07:38:57 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 10 Feb 2016 22:38:57 +1000 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: <56BB25A2.7010103@canterbury.ac.nz> References: <56BA700E.7010305@stoneleaf.us> <56BB25A2.7010103@canterbury.ac.nz> Message-ID: On 10 February 2016 at 21:57, Greg Ewing wrote: > Guido van Rossum wrote: >> >> (Not sure >> about "12_34_56" but there are probably use cases for that too.) > > I think the Chinese group by 10,000s rather than 1000s, > so they might want to write 1234_5678. As others have suggested, I like the idea of keeping the grammar simple (i.e. numeric literals must start with a base appropriate digit, but may subsequently contain digits or underscores). I'd even apply that to float literals, with the avoidance of putting an underscore just before the floating point being a style issue, rather than a syntactic one. What kind of numeric grouping to use is also a style question - if it's an English-language project or an international project using metric values, then it would make sense to group by thousands. If it's a project written assuming maintainers can follow Chinese or Japanese, then it would make sense to group according to the conventions of those language communities, just as folks may already decide to do with variable names and comments. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From storchaka at gmail.com Wed Feb 10 07:59:00 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 10 Feb 2016 14:59:00 +0200 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: Message-ID: On 10.02.16 13:37, Alexander Heger wrote: > will just print the number 123456. Hence for me as a FORTRAN this > would seem a natural thing to do. Better than the underscores ... I > would associate the with the LaTeX maths mode index operator, and then > I would read 100_002 as the number 4. No, 100_{002} is the number 4. From storchaka at gmail.com Wed Feb 10 08:09:23 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 10 Feb 2016 15:09:23 +0200 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: Message-ID: On 09.02.16 23:40, Manuel Cer?n wrote: > Sometimes it's hard to read long numbers. For example: > > >>> opts..write_buffer_size = 67108864 > > Some languages (Ruby, Perl, Swift) allow the use of underscores in > numeric literals, which are ignored. They are typically used as > thousands separators. Some languages allow the use of ' (an apostrophe) as thousands separators: 67'108'864 But I prefer underscores as more common variant. From p.f.moore at gmail.com Wed Feb 10 08:10:48 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 10 Feb 2016 13:10:48 +0000 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: Message-ID: On 10 February 2016 at 11:37, Alexander Heger wrote: >>>>> opts.write_buffer_size = 67108864 >> >> The disadvantage is that, as far as I known, no other languages do this. > > This is not true. It is absolutely legal in FORTRAN > > program f > print*, 123 456 > end > > will just print the number 123456. Hence for me as a FORTRAN this > would seem a natural thing to do. Better than the underscores ... I > would associate the with the LaTeX maths mode index operator, and then > I would read 100_002 as the number 4. But to be fair, in older fortrans at least (I'd like to hope it's got more sane these days) wasn't pro gr a m f p r i n t*,1 2 3 4 5 6 e nd just as valid? (IIRC, whitespace was ignored everywhere, even within keywords...) Paul From python at 2sn.net Wed Feb 10 08:21:09 2016 From: python at 2sn.net (Alexander Heger) Date: Thu, 11 Feb 2016 00:21:09 +1100 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: Message-ID: >> will just print the number 123456. Hence for me as a FORTRAN this >> would seem a natural thing to do. Better than the underscores ... I >> would associate the with the LaTeX maths mode index operator, and then >> I would read 100_002 as the number 4. > > No, 100_{002} is the number 4. Well, as you know, even more strictly speaking, in a LaTeX text you'd have to write $100_{002}$. But in more colloquial use of LaTeX syntax, e.g., in plain text abstracts, the braces are often dropped for the sake of readability. Beside, base 0 or base 1 make little sense. -Alexander Sent from my Pixel C. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Wed Feb 10 09:50:57 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 10 Feb 2016 23:50:57 +0900 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: Message-ID: <22203.20049.990491.841849@turnbull.sk.tsukuba.ac.jp> Alexander Heger writes: > This is not true. It is absolutely legal in FORTRAN > > program f > print*, 123 456 > end In traditional FORTRAN, so is this: print*,123456 and this p r i n t * , 1 2 3 4 5 6 and all three print statements have exactly the same meaning, because in traditional FORTRAN spaces are completely insignificant. That's kinda un-Pythonic. From srkunze at mail.de Wed Feb 10 12:35:16 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 10 Feb 2016 18:35:16 +0100 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: Message-ID: <56BB74D4.2090608@mail.de> That's one reason I am -1 on this proposal. On 10.02.2016 02:35, Andrew Barnert via Python-ideas wrote: > One possible objection that nobody's raised: > > Separating groups of three is all well and good; to my western eyes, > 10_000_000_000 is obviously 10 billion.* > > But someone from China is likely to use groups of four, and > 100_0000_0000 is not obviously anything--my first thought is around > 100 billion, but that can't be right, so I have to count up the digits. > > I still think this is a good suggestion, because 100000000000 is even > more useless to me as 100_0000_0000, and far more likely to be > concealing a typo. I just wanted to make sure everyone knew the issue. > > > * If you're going to say "no, it's a milliard, you stupid American", > go back to the early 70s, and bring your non-decimal currency with > you. It's billion in English, and has been for 40+ years. Leave the > fighting to the languages where it's ambiguous, like Portuguese or > Finnish. > > Sent from my iPhone > > On Feb 9, 2016, at 13:40, Manuel Cer?n > wrote: > >> Hi everyone! >> >> Sometimes it's hard to read long numbers. For example: >> >> >>> opts.write_buffer_size = 67108864 >> >> Some languages (Ruby, Perl, Swift) allow the use of underscores in >> numeric literals, which are ignored. They are typically used as >> thousands separators. The example above would look like this: >> >> >>> opts.write_buffer_size = 67_108_864 >> >> Which helps to quickly identify that this is around 67 million. >> >> Another option is to use spaces instead of underscores: >> >> >>> opts.write_buffer_size = 67 108 864 >> >> This has two advantages: 1. is analog to the way string literals >> work, which are concatenated if put next to each other. 2. spaces are >> already used as thousands separator in many european languages [1]. >> >> The disadvantage is that, as far as I known, no other languages do this. >> >> I have seen some old discussions around this, but nothing on this >> list or a PEP. With Python being use more and more for scientific and >> numeric computation, this is a small change that will help with >> readability a lot. And, as far as I can tell, it doesn't break >> compatibility in any way. >> >> Thoughts? >> >> Manuel. >> >> [1] https://docs.oracle.com/cd/E19455-01/806-0169/overview-9/index.html >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Wed Feb 10 12:42:14 2016 From: random832 at fastmail.com (Random832) Date: Wed, 10 Feb 2016 12:42:14 -0500 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: <56BA700E.7010305@stoneleaf.us> Message-ID: <1455126134.3208445.517523930.7496A8E0@webmail.messagingengine.com> On Tue, Feb 9, 2016, at 18:17, Guido van Rossum wrote: > Indeed, "123 456" is a no-no, but "123_456" sound good. (Not sure > about "12_34_56" but there are probably use cases for that too.) For one, not all languages use the same divisions. In many Indian languages, the natural divisions would be like 1_23_45_678, whereas in Chinese and Japanese some people may want to use 1_2345_6789, though the western system is also common. Also, dare I suggest, 0x_0123_4567_89AB_CDEF? Arbitrary grouping may be useful in binary constants representing bit fields. For that matter, for an integer representing a fixed-point fractional quantity, one may want to use 1_234_567__0000, where the last separator represents the decimal place. Mathematical publications sometimes group digits after a decimal place into groups of five, e.g. Decimal("3.14159_26535_89793_84626") or combine these ideas for Fraction(3__14159_26535_89793_84626, 10**20) I don't know that there's any reason to build restrictions into the language, any more than to require certain integer constants to be in decimal and others to be in hexadecimal. From srkunze at mail.de Wed Feb 10 12:42:50 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 10 Feb 2016 18:42:50 +0100 Subject: [Python-ideas] Compact repr operator (i.e., __short_repr__) In-Reply-To: References: <28D7B647-C173-4B1D-BD74-DE62768FFCC0@lukasa.co.uk> <5FC0322A-7ECF-448A-8D87-4A9341EBBA88@yahoo.com> Message-ID: <56BB769A.30602@mail.de> On 09.02.2016 12:13, Cory Benfield wrote: >> On 9 Feb 2016, at 10:13, Andrew Barnert wrote: >> >> So? >> >> One use of round-trippable reprs is to copy them from output and paste them into source code or an interactive session. When doing so, you almost always know that you're dealing with a builtin or third-party type that's round-trippable--and, when you're surprised, it's almost always obvious, because you get something that looks nothing at all like the source equivalent, and that raises a SyntaxError if you try to evaluate it anyway. >> >> And even those "almost"s aren't a problem in practice. Sure, a list that recursively contains itself looks misleadingly round-trippable, and will evaluate successfully into the wrong thing. But this rarely comes up in practice--and, if it does, because there's a human being inspecting, debugging, or playing with things, rather than a program, it's easy to deal with. >> >> This isn't theoretical--I do this all the time when debugging code, I write my own types to make them easier to debug this way, and it saves me time and hassle. In fact, it's one of the benefits of using Python over some of the other languages I use, where repr or toDebugString or whatever is never useful, instead of being usually useful especially with common types. >> >> The only thing the non-universality of round-tripping means is that you can't use repr with eval as a persistence format. Which is a good thing--you *shouldn't* use it as a persistence format, and that would be true even if it did work. But using it as an inspecting/debugging format is not a problem, and breaking that would be a bad idea. >> >> In fact, breaking it would make repr nearly pointless. Except for a types that define __repr__ but block __str__ with a TypeError (which is rare, and it's debatable whether those types are even valid), when would you ever use repr otherwise? > I don?t think we?re arguing the same point. I?m not saying that __repr__ shouldn?t be round-trippable: if that makes sense for your type then totally fine, go for it. However, I am saying that I don?t see the advantage in having *both* a round-trippable and non-round-trippable repr for the same type. If you?re copying and pasting then the length of the round-trippable representation is a non-issue, and you actively want it to appear in your debug output rather than a shorter version that elides information required to round-trip the data. So in this case, what purpose does the shorter version serve? Same question here. I don't see much use for __short_repr__ being worth the hassle of maintaining possibly two implementations of almost the same thing. All in all, I don't think __repr__ needs optimization in the first place. Best, Sven From random832 at fastmail.com Wed Feb 10 12:47:16 2016 From: random832 at fastmail.com (Random832) Date: Wed, 10 Feb 2016 12:47:16 -0500 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: <1455126134.3208445.517523930.7496A8E0@webmail.messagingengine.com> References: <56BA700E.7010305@stoneleaf.us> <1455126134.3208445.517523930.7496A8E0@webmail.messagingengine.com> Message-ID: <1455126436.3209486.517537346.16601817@webmail.messagingengine.com> On Wed, Feb 10, 2016, at 12:42, Random832 wrote: > 3__14159_26535_89793_84626 ...And before anyone else points it out, I copied the fourth digit group from the wrong position. (I happen to have the first 17 digits memorized, so I typed the first three groups, and then mistakenly copied from after the last one I had memorized rather than the last one I'd typed). From g.brandl at gmx.net Wed Feb 10 12:51:02 2016 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 10 Feb 2016 18:51:02 +0100 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: Message-ID: On 02/09/2016 10:40 PM, Manuel Cer?n wrote: > Hi everyone! > > Sometimes it's hard to read long numbers. For example: > >>>> opts.write_buffer_size = 67108864 > > Some languages (Ruby, Perl, Swift) allow the use of underscores in numeric > literals, which are ignored. They are typically used as thousands separators. > The example above would look like this: > >>>> opts.write_buffer_size = 67_108_864 > > Which helps to quickly identify that this is around 67 million. > Thoughts? I like it, and for everybody to try it out, I posted a draft patch here: http://bugs.python.org/issue26331 Underscores are allowed anywhere in numeric literals, except: * at the beginning of a literal (obviously) * at the end of a literal * directly after a dot (since the underscore could start an attribute name) * directly after a sign in exponents (for consistency with leading signs) * in the middle of the "0x", "0o" or "0b" base specifiers Reviewers welcome! cheers, Georg From random832 at fastmail.com Wed Feb 10 13:00:20 2016 From: random832 at fastmail.com (Random832) Date: Wed, 10 Feb 2016 13:00:20 -0500 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: Message-ID: <1455127220.3213124.517547370.00DB2A0F@webmail.messagingengine.com> On Wed, Feb 10, 2016, at 12:51, Georg Brandl wrote: > Underscores are allowed anywhere in numeric literals, except: > > * at the beginning of a literal (obviously) > * at the end of a literal > * directly after a dot (since the underscore could start an attribute > name) I don't think it's particularly important to support this case, but the sequence digit/dot/name with no spaces between is a syntax error now, because the digit/dot is interpreted as a floating point constant. > * directly after a sign in exponents (for consistency with leading signs) > * in the middle of the "0x", "0o" or "0b" base specifiers Do you allow multiple underscores in a row? I mentioned a couple possible use cases for that. From g.brandl at gmx.net Wed Feb 10 13:10:38 2016 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 10 Feb 2016 19:10:38 +0100 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: <1455127220.3213124.517547370.00DB2A0F@webmail.messagingengine.com> References: <1455127220.3213124.517547370.00DB2A0F@webmail.messagingengine.com> Message-ID: On 02/10/2016 07:00 PM, Random832 wrote: > On Wed, Feb 10, 2016, at 12:51, Georg Brandl wrote: >> Underscores are allowed anywhere in numeric literals, except: >> >> * at the beginning of a literal (obviously) >> * at the end of a literal >> * directly after a dot (since the underscore could start an attribute >> name) > > I don't think it's particularly important to support this case, but the > sequence digit/dot/name with no spaces between is a syntax error now, > because the digit/dot is interpreted as a floating point constant. Don't forget that float literals can also start with just the dot. Therefore this case can get quite ambiguous. expr ._2 is an attribute access expr 0._2 is a syntax error >> * directly after a sign in exponents (for consistency with leading signs) >> * in the middle of the "0x", "0o" or "0b" base specifiers > > Do you allow multiple underscores in a row? I mentioned a couple > possible use cases for that. Yes, there is no restriction. Georg From mike at selik.org Wed Feb 10 14:18:25 2016 From: mike at selik.org (Michael Selik) Date: Wed, 10 Feb 2016 19:18:25 +0000 Subject: [Python-ideas] Compact repr operator (i.e., __short_repr__) In-Reply-To: References: <28D7B647-C173-4B1D-BD74-DE62768FFCC0@lukasa.co.uk> Message-ID: The appropriate truncation for any repr that is too long would depend on the particulars of both the object and the purpose of the output. If reprlib and pprint were improved for say, defaultdict and OrderedDict, or more generic things, what would its features be? I'm guessing you wouldn't just duplicate the functionality of ``pprint(dict(x))``. On Tue, Feb 9, 2016, 4:09 AM Mahmoud Hashemi wrote: > If you want to make the case that default reprs should be > non-roundtrippable in the case that they're too long, that's a fine and > separate discussion. Though I have argued in the past that float('nan') > would be a less surprising/more useful repr. > > And that's what it's about, usefulness. It's useful to be able to copy in > and out of the REPL, even for very large reprs. I've done it, my coworkers > and students do it, and you've probably done it as well. > > But there are other times (often outside the REPL), where that is not the > case, and being able to address them explicitly, in the vein of reprlib and > pprint -- but better -- would be *useful*. Who hasn't wished that the > built-in defaultdict and OrderedDict were as pprintable or > reprlib.repr-able as dict. > > There's plenty of room to improve. > > On Tue, Feb 9, 2016 at 12:56 AM, Cory Benfield wrote: > >> >> > On 9 Feb 2016, at 02:49, Mahmoud Hashemi wrote: >> > >> > Roundtrippable reprs are certainly part of Python canon, whether or not >> they are universally used (Chris), or guaranteed (Mike). >> > >> >> They can be part of the canon all they want, but if they?re not >> universally guaranteed then I don?t know that this is a real problem. It >> means that the world of Python objects divides into two kinds: first, those >> with __repr__ return values that can be round tripped and those with >> __repr__ return values that cannot be round tripped. >> >> Given that objects of the second type already exist (I know this for a >> fact because I have written some quite recently!), it would be an error to >> assume that the identity 'eval(repr(x)) == x? holds for arbitrary types. In >> fact, not only does it not hold for all third-party types, it doesn?t even >> hold for all built-in types: >> >> >>> x = float(?nan?) >> >>> repr(x) >> ?nan? >> >>> eval(repr(x)) >> Traceback (most recent call last): >> File "", line 1, in >> File "", line 1, in >> NameError: name 'nan' is not defined >> >> I think the reality is that there is no constraint on the representation >> of arbitrary types to be round-trippable in any way. Again, all custom >> types have non-round-trippable representations by default, many more >> eclectic built-in types have non-round-tripppable representations (in >> addition to NaN, the memoryview object leaps to mind). >> >> I can also note the Python documentation on repr: >> >> > For many types, this function makes an attempt to return a string that >> would yield an object with the same value when passed to eval(), otherwise >> the representation is a string enclosed in angle brackets that contains the >> name of the type of the object together with additional information often >> including the name and address of the object. >> >> If the language doesn?t even try to enforce the idea that representations >> will be round-trippable, I think there?s just no problem here. >> >> Cory >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mahmoud at hatnote.com Wed Feb 10 14:31:05 2016 From: mahmoud at hatnote.com (Mahmoud Hashemi) Date: Wed, 10 Feb 2016 11:31:05 -0800 Subject: [Python-ideas] Compact repr operator (i.e., __short_repr__) In-Reply-To: References: <28D7B647-C173-4B1D-BD74-DE62768FFCC0@lukasa.co.uk> Message-ID: For those examples, the pprint/reprlib behavior probably wouldn't be much different than dict/list of tuples. I can put my vote in, but I prefer to defer design specifics to the creators/maintainers of those types. That's my point. Right now I feel like repr() is a powerful, but underleveraged function in the builtin ecosystem. Instead of pprint.pformat(), why not add a keyword argument, and have repr(obj, pretty=True), and have that call through with arguments to a more full featured __repr__? Same could go for repr(obj, length=150, depth=2). These could have defaults set for REPL sessions. Everyone is doing hacky, incomplete heuristic introspection, from IPython to the standard library. Instead, let's embrace the complexities of viewing application state, and allow for formalization of community efforts. Mahmoud On Wed, Feb 10, 2016 at 11:18 AM, Michael Selik wrote: > The appropriate truncation for any repr that is too long would depend on > the particulars of both the object and the purpose of the output. If > reprlib and pprint were improved for say, defaultdict and OrderedDict, or > more generic things, what would its features be? > > I'm guessing you wouldn't just duplicate the functionality of > ``pprint(dict(x))``. > > On Tue, Feb 9, 2016, 4:09 AM Mahmoud Hashemi wrote: > >> If you want to make the case that default reprs should be >> non-roundtrippable in the case that they're too long, that's a fine and >> separate discussion. Though I have argued in the past that float('nan') >> would be a less surprising/more useful repr. >> >> And that's what it's about, usefulness. It's useful to be able to copy in >> and out of the REPL, even for very large reprs. I've done it, my coworkers >> and students do it, and you've probably done it as well. >> >> But there are other times (often outside the REPL), where that is not the >> case, and being able to address them explicitly, in the vein of reprlib and >> pprint -- but better -- would be *useful*. Who hasn't wished that the >> built-in defaultdict and OrderedDict were as pprintable or >> reprlib.repr-able as dict. >> >> There's plenty of room to improve. >> >> On Tue, Feb 9, 2016 at 12:56 AM, Cory Benfield wrote: >> >>> >>> > On 9 Feb 2016, at 02:49, Mahmoud Hashemi wrote: >>> > >>> > Roundtrippable reprs are certainly part of Python canon, whether or >>> not they are universally used (Chris), or guaranteed (Mike). >>> > >>> >>> They can be part of the canon all they want, but if they?re not >>> universally guaranteed then I don?t know that this is a real problem. It >>> means that the world of Python objects divides into two kinds: first, those >>> with __repr__ return values that can be round tripped and those with >>> __repr__ return values that cannot be round tripped. >>> >>> Given that objects of the second type already exist (I know this for a >>> fact because I have written some quite recently!), it would be an error to >>> assume that the identity 'eval(repr(x)) == x? holds for arbitrary types. In >>> fact, not only does it not hold for all third-party types, it doesn?t even >>> hold for all built-in types: >>> >>> >>> x = float(?nan?) >>> >>> repr(x) >>> ?nan? >>> >>> eval(repr(x)) >>> Traceback (most recent call last): >>> File "", line 1, in >>> File "", line 1, in >>> NameError: name 'nan' is not defined >>> >>> I think the reality is that there is no constraint on the representation >>> of arbitrary types to be round-trippable in any way. Again, all custom >>> types have non-round-trippable representations by default, many more >>> eclectic built-in types have non-round-tripppable representations (in >>> addition to NaN, the memoryview object leaps to mind). >>> >>> I can also note the Python documentation on repr: >>> >>> > For many types, this function makes an attempt to return a string that >>> would yield an object with the same value when passed to eval(), otherwise >>> the representation is a string enclosed in angle brackets that contains the >>> name of the type of the object together with additional information often >>> including the name and address of the object. >>> >>> If the language doesn?t even try to enforce the idea that >>> representations will be round-trippable, I think there?s just no problem >>> here. >>> >>> Cory >>> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Wed Feb 10 15:26:02 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 11 Feb 2016 09:26:02 +1300 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: Message-ID: <56BB9CDA.6090404@canterbury.ac.nz> Alexander Heger wrote: > Well, as you know, even more strictly speaking, in a LaTeX text you'd > have to write $100_{002}$. +1 on allowing Python expressions to be written in LaTeX math mode. This is obviously why Guido has reserved $ for such a long time. He's just popped back in the time machine and told himself about this thread! -- Greg From random832 at fastmail.com Wed Feb 10 15:29:24 2016 From: random832 at fastmail.com (Random832) Date: Wed, 10 Feb 2016 15:29:24 -0500 Subject: [Python-ideas] Round-trippable repr for everything In-Reply-To: <28D7B647-C173-4B1D-BD74-DE62768FFCC0@lukasa.co.uk> References: <28D7B647-C173-4B1D-BD74-DE62768FFCC0@lukasa.co.uk> Message-ID: <1455136164.3245438.517660898.1020A699@webmail.messagingengine.com> On Tue, Feb 9, 2016, at 03:56, Cory Benfield wrote: > I think the reality is that there is no constraint on the representation > of arbitrary types to be round-trippable in any way. Again, all custom > types have non-round-trippable representations by default, many more > eclectic built-in types have non-round-tripppable representations (in > addition to NaN, the memoryview object leaps to mind). One other example is classes and functions though in many cases it's not clear why this should be the case. Have the default for top-level functions and classes check whether it's reachable through [module].[name] and if so return that. The default for methods could use [class].[name], or [obj].[name] for bound methods. Instead of the current representation, you could have the default repr on objects use pickle. def __repr__(self): return 'pickle.loads(%r)' % pickle.dumps(self) Or maybe have a helper function e.g. def unrepr(description, pickled) return pickle.dumps(pickled) # ignore description and default to "unrepr('MyClass object at 0x12345678', b'pickledstuff')" [Speaking of pickle and since you mentioned memoryview, I noticed a bug: pickle apparently successfully generates output for a memoryview, but it fails on attempting to unpickle it - I think that it should throw an exception immediately on attempting to pickle it.] Certainly there's the odd edge case where it doesn't make sense for it to be round-trippable, and there's a tradeoff between human-readability and round-trippability (see e.g. all the classes that can't be round-tripped unless the class is imported)... but that's part of where a short-repr vs long-repr could make sense - short-repr could Decimal('123.456') or even whereas long-repr returns __import__('decimal').Decimal('123.456') From abarnert at yahoo.com Wed Feb 10 16:04:13 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 10 Feb 2016 21:04:13 +0000 (UTC) Subject: [Python-ideas] Round-trippable repr for everything In-Reply-To: <1455136164.3245438.517660898.1020A699@webmail.messagingengine.com> References: <1455136164.3245438.517660898.1020A699@webmail.messagingengine.com> Message-ID: <390162991.1729100.1455138253237.JavaMail.yahoo@mail.yahoo.com> On Wednesday, February 10, 2016 12:29 PM, Random832 wrote: > On Tue, Feb 9, 2016, at 03:56, Cory Benfield wrote: >> I think the reality is that there is no constraint on the representation >> of arbitrary types to be round-trippable in any way. Again, all custom >> types have non-round-trippable representations by default, many more >> eclectic built-in types have non-round-tripppable representations (in >> addition to NaN, the memoryview object leaps to mind). > > One other example is classes and functions though in many cases it's not > clear why this should be the case. Have the default for top-level > functions and classes check whether it's reachable through > [module].[name] and if so return that. The default for methods could use > [class].[name], or [obj].[name] for bound methods. > > Instead of the current representation, you could have the default repr > on objects use pickle. That seems like a bad idea. First, as a human programmer, the repr "[1, 2, (3, 'spam')]" means something to me--and the same thing to me as to Python; the repr "unrepr('silly.C object at 0x106ec2b38', b'\x80\x03csilly\nC\nq\x00)\x81q\x01}q\x02X\x01\x00\x00\x00xq\x03K\x02sb.')" means less to me than the current "", while taking up a lot more space. Meanwhile, if I've changed the silly module since printing out the repr, that pickle will fail--or, worse, and more likely, succeed but give me the wrong value. And, since eval'ing reprs is generally something you do when experimenting or when debugging code that you're actively working on, this will be very common. Meanwhile, eval("[1, 2, (3, 'spam')]") == [1, 2, (3, 'spam')], and that's true for most containers and "value-type" objects, which tend to be the kinds of things that have round-trippable reprs today. That probably won't be true for instances of arbitrary types where the creator didn't think of a repr. Finally, when I paste a repr into the REPL or into my source and it needs further qualification, as with datetime or Decimal, I can almost always figure this out pretty easily. And I would much rather have to figure it out and then paste "decimal.Decimal('123.456')" into my source than have the computer figure it out and paste "__import__('decimal').Decimal('123.456')" or something with a pickle in the middle of it into my source. It's true that pasting reprs is just something that tends to often work when you'd expect it to today, not something guaranteed--but since repr is used for exploration and debugging by humans, not for automated processing by scripts, "tends to often work" tends to often be very good. So there really isn't a problem here. And your suggestion doesn't really help that use anyway; the only use it helps is using repr as an automated serialization format, which we explicitly _don't_ want to help. If you want automated serialization, just use pickle. There's no reason to twist repr and eval to support that use case almost but not quite as well (and even less securely and efficiently). If you want something more readable, use something like YAML with a custom type library, or jsonpickle, etc. Or, if you think we need a more human-readable pickle format, that might be an interesting idea, but there's no reason to twist repr into it, or to force it to be readable by eval instead of loads. From rosuav at gmail.com Wed Feb 10 16:42:08 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 11 Feb 2016 08:42:08 +1100 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: <1455127220.3213124.517547370.00DB2A0F@webmail.messagingengine.com> References: <1455127220.3213124.517547370.00DB2A0F@webmail.messagingengine.com> Message-ID: On Thu, Feb 11, 2016 at 5:00 AM, Random832 wrote: > On Wed, Feb 10, 2016, at 12:51, Georg Brandl wrote: >> Underscores are allowed anywhere in numeric literals, except: >> >> * at the beginning of a literal (obviously) >> * at the end of a literal >> * directly after a dot (since the underscore could start an attribute >> name) > > I don't think it's particularly important to support this case, but the > sequence digit/dot/name with no spaces between is a syntax error now, > because the digit/dot is interpreted as a floating point constant. Only if that's the only dot. >>> 1._ File "", line 1 1._ ^ SyntaxError: invalid syntax >>> 1.1._ Traceback (most recent call last): File "", line 1, in AttributeError: 'float' object has no attribute '_' I don't want to see the first one have an arbitrary new meaning. It'd be confusing. ChrisA From random832 at fastmail.com Wed Feb 10 23:55:18 2016 From: random832 at fastmail.com (Random832) Date: Wed, 10 Feb 2016 23:55:18 -0500 Subject: [Python-ideas] Improve readability of long numeric literals In-Reply-To: References: <1455127220.3213124.517547370.00DB2A0F@webmail.messagingengine.com> Message-ID: <1455166518.3687160.518039338.7CBE43E9@webmail.messagingengine.com> On Wed, Feb 10, 2016, at 16:42, Chris Angelico wrote: > On Thu, Feb 11, 2016 at 5:00 AM, Random832 > wrote: > > I don't think it's particularly important to support this case, but the > > sequence digit/dot/name with no spaces between is a syntax error now, > > because the digit/dot is interpreted as a floating point constant. > > Only if that's the only dot. But only the first dot is part of the numeric literal and therefore even theoretically eligible to have an underscore before or after it accepted as part of the literal, so that goes without saying. I was just pointing out that the underscore this rule disallows can't actually start an attribute name. From cory at lukasa.co.uk Thu Feb 11 04:00:42 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 11 Feb 2016 09:00:42 +0000 Subject: [Python-ideas] Compact repr operator (i.e., __short_repr__) In-Reply-To: References: <28D7B647-C173-4B1D-BD74-DE62768FFCC0@lukasa.co.uk> Message-ID: <2603FD3A-035D-4678-B9F7-F08F2E3A0127@lukasa.co.uk> > On 10 Feb 2016, at 19:31, Mahmoud Hashemi wrote: > > For those examples, the pprint/reprlib behavior probably wouldn't be much different than dict/list of tuples. I can put my vote in, but I prefer to defer design specifics to the creators/maintainers of those types. That's my point. > > Right now I feel like repr() is a powerful, but underleveraged function in the builtin ecosystem. Instead of pprint.pformat(), why not add a keyword argument, and have repr(obj, pretty=True), and have that call through with arguments to a more full featured __repr__? Same could go for repr(obj, length=150, depth=2). These could have defaults set for REPL sessions. > > Everyone is doing hacky, incomplete heuristic introspection, from IPython to the standard library. Instead, let's embrace the complexities of viewing application state, and allow for formalization of community efforts. If we want to embrace the complexities of viewing application state, perhaps we should divorce this conversation from the repr altogether. There?s plenty of applications where ?viewing application state? will require more than the repr generally provides: you?ll want insight into private member variables, into the state of composed objects, and potentially into the state of some of those private member variables (e.g. is this lock held at this moment?). This is outside the generally accepted scope of the repr as I understand it: most people don?t provide the repr in this form. If you want the complexity of application state then you probably want some way to dump the graph of objects starting with a specific one, along with their state. That?s totally beyond the scope of the repr, and is closer to what pickle would do for you (though as Andrew points out, pickle is not the right tool for this job either). What I don?t understand is: if you are in the REPL already, what does any of this buy you? The REPL has plenty of powerful tools for introspecting Python objects: specifically, it has the Python programming language! If you want to introspect application state from the REPL, then just grab the object you want and start interrogating its state. dir() works, printing fields works, calling methods works, and all of this is far more powerful than anything you can get from the repr. Cory -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mahmoud at hatnote.com Thu Feb 11 04:24:42 2016 From: mahmoud at hatnote.com (Mahmoud Hashemi) Date: Thu, 11 Feb 2016 01:24:42 -0800 Subject: [Python-ideas] Compact repr operator (i.e., __short_repr__) In-Reply-To: <2603FD3A-035D-4678-B9F7-F08F2E3A0127@lukasa.co.uk> References: <28D7B647-C173-4B1D-BD74-DE62768FFCC0@lukasa.co.uk> <2603FD3A-035D-4678-B9F7-F08F2E3A0127@lukasa.co.uk> Message-ID: Michael: Yes, I have several projects on PyPI, some of which are related and somewhat popular, but I hardly think that a PyPI package can encompass the API changes I'm describing. We already have several built-in modules (reprlib, pprint) and 3rd party modules (IPython Notebook, Django 500 pages, the Werkzeug debugger) that see a great deal of usage, but only support a few builtinn types and have to hack heuristics for the rest. Cory: Introspecting application state happens at many levels, one step at a time. Not only is perfect the enemy of good here, but I really don't think an omni-solution like the one you're describing is a realistic, Pythonic goal. Besides, graphs are powerful, but don't go underestimating linear text. An enhanced repr allows developer-users to communicate the context of their usage so that the library authors can programmatically provide and prioritize information in their types. I've seen (and written) plenty of reprs that encode useful information in my types. To your example, I might have: . I still think you're selling repr short. It can be much more. One day we might have: >>> repr(obj, pretty=True, depth=2, width=120) Though, as I said before, my primary use case isn't the REPL, and those arguments should be part of the REPL config (pprint defaults to width=80 for consoles). The point is that repr has room to evolve. Mahmoud On Thu, Feb 11, 2016 at 1:00 AM, Cory Benfield wrote: > > > On 10 Feb 2016, at 19:31, Mahmoud Hashemi wrote: > > > > For those examples, the pprint/reprlib behavior probably wouldn't be > much different than dict/list of tuples. I can put my vote in, but I prefer > to defer design specifics to the creators/maintainers of those types. > That's my point. > > > > Right now I feel like repr() is a powerful, but underleveraged function > in the builtin ecosystem. Instead of pprint.pformat(), why not add a > keyword argument, and have repr(obj, pretty=True), and have that call > through with arguments to a more full featured __repr__? Same could go for > repr(obj, length=150, depth=2). These could have defaults set for REPL > sessions. > > > > Everyone is doing hacky, incomplete heuristic introspection, from > IPython to the standard library. Instead, let's embrace the complexities of > viewing application state, and allow for formalization of community efforts. > > If we want to embrace the complexities of viewing application state, > perhaps we should divorce this conversation from the repr altogether. > > There?s plenty of applications where ?viewing application state? will > require more than the repr generally provides: you?ll want insight into > private member variables, into the state of composed objects, and > potentially into the state of some of those private member variables (e.g. > is this lock held at this moment?). This is outside the generally accepted > scope of the repr as I understand it: most people don?t provide the repr in > this form. > > If you want the complexity of application state then you probably want > some way to dump the graph of objects starting with a specific one, along > with their state. That?s totally beyond the scope of the repr, and is > closer to what pickle would do for you (though as Andrew points out, pickle > is not the right tool for this job either). > > What I don?t understand is: if you are in the REPL already, what does any > of this buy you? The REPL has plenty of powerful tools for introspecting > Python objects: specifically, it has the Python programming language! If > you want to introspect application state from the REPL, then just grab the > object you want and start interrogating its state. dir() works, printing > fields works, calling methods works, and all of this is far more powerful > than anything you can get from the repr. > > Cory > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Feb 11 05:11:34 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 11 Feb 2016 20:11:34 +1000 Subject: [Python-ideas] Compact repr operator (i.e., __short_repr__) In-Reply-To: References: <28D7B647-C173-4B1D-BD74-DE62768FFCC0@lukasa.co.uk> <2603FD3A-035D-4678-B9F7-F08F2E3A0127@lukasa.co.uk> Message-ID: On 11 February 2016 at 19:24, Mahmoud Hashemi wrote: > Michael: Yes, I have several projects on PyPI, some of which are related and > somewhat popular, but I hardly think that a PyPI package can encompass the > API changes I'm describing. We already have several built-in modules > (reprlib, pprint) and 3rd party modules (IPython Notebook, Django 500 pages, > the Werkzeug debugger) that see a great deal of usage, but only support a > few builtinn types and have to hack heuristics for the rest. I generally haven't been following these threads, but this one caught my eye. Making pprint extensible was one of the rationales for adding functools.singledispatch to the standard library. However, as far as I am aware, we haven't had anyone find the time to follow up on that change by actually applying it where it would make sense to do so. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve at pearwood.info Thu Feb 11 05:59:41 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 11 Feb 2016 21:59:41 +1100 Subject: [Python-ideas] Compact repr operator (i.e., __short_repr__) In-Reply-To: References: <28D7B647-C173-4B1D-BD74-DE62768FFCC0@lukasa.co.uk> <2603FD3A-035D-4678-B9F7-F08F2E3A0127@lukasa.co.uk> Message-ID: <20160211105941.GN31806@ando.pearwood.info> On Thu, Feb 11, 2016 at 01:24:42AM -0800, Mahmoud Hashemi wrote: > I still think you're selling repr short. It can be much more. One day we > might have: > > >>> repr(obj, pretty=True, depth=2, width=120) I'm sympathetic, but I'm not sure that repr() is the right place to be focusing. repr() is too closely tied to the REPL and the existing string formatting codes to start making radical changes. But we do have the pprint library, which (in my opinion) is underpowered and underused. I know that some people think that Python already has too many string conversion functions ( str()/__str__ and repr()/__repr__ ) but for my own use I've occasionally found that it has too few. Some time ago, I started working on a library for continued fractions. I count at least five useful/standard representations for a continued fraction. All of these would be equivalent: ContinuedFraction(1, 2, 3, 4, 5) [1; 2, 3, 4, 5] 1 + 1/(2 + 1/(3 + 1/(4 + 1/5))) 1 1 1 1 1 + ----- ----- ----- ----- 2 + 3 + 4 + 5 1 1 + ----------------- 1 2 + ------------- 1 3 + --------- 1 4 + ----- 5 (Apart from the first, the remaining four are standard notations used in mathematics.) Now obviously I can just add my own string conversion methods to the class, but it would be nice to be able to integrate with the pprint (and maybe reprlib) libraries in some way rather than an ad hoc set of arbitrarily named methods? I must admit that I'm just thinking aloud here, I don't actually have a concrete idea. But maybe my use-case will spark an idea in someone else. -- Steve From cory at lukasa.co.uk Thu Feb 11 08:50:38 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 11 Feb 2016 13:50:38 +0000 Subject: [Python-ideas] Compact repr operator (i.e., __short_repr__) In-Reply-To: References: <28D7B647-C173-4B1D-BD74-DE62768FFCC0@lukasa.co.uk> <2603FD3A-035D-4678-B9F7-F08F2E3A0127@lukasa.co.uk> Message-ID: > On 11 Feb 2016, at 09:24, Mahmoud Hashemi wrote: > > An enhanced repr allows developer-users to communicate the context of their usage so that the library authors can programmatically provide and prioritize information in their types. I've seen (and written) plenty of reprs that encode useful information in my types. To your example, I might have: . > To be clear, I absolutely do that as well. In fact, if I?m writing a repr that?s almost always the form I give it. Again, I?m not arguing that the repr isn?t useful. I?m arguing that it *is* useful, and that attempting to extend its utility with flags and complex functionality dilutes its utility. I?m in opposition to trying to shovel more into the repr: to try to extend it with more flexibility and therefore increase the burden on developers to be useful in that scenario. I don?t disagree that the repr *can* be more: I just don?t think it should be. The repr is, like it or not, implicitly unstructured. Because the return value of __repr__ is a string, we cannot easily extend it for utility without pushing that cost out to developers. That is, if we wanted to support your ?pretty=True? argument to repr(), all developers would need to start writing __repr__ functions that respond to that flag. This is, IMO, a bad way to obtain the feature you want, because at this point ?pretty? becomes constrained by the original author. It also means that some flags may not work if the author has not supported them: ?pretty? may do nothing, ?pretty? may work but ?depth? may be ignored, or ?width?. What I?ve understood from your mails is that you want is the ability to provide a *structured* debug representation of an object. That is, a function to hang off an object that is formatted according to the preferences of the user, not the developer. I think this is a good thing to have, because structured debug representations allow for more automated processing, as well as additional utility for users: see for example the structlog[0] module, which applies this concept to logging. This is a good and reasonable thing to want, but it?s a different thing to the repr. The repr is fundamentally an unstructured representation of whatever the author believed was useful to describe about the object. I don?t see any reason to remove that or to put pressure on that mechanism to be more than it is. Cory [0]: https://structlog.readthedocs.org/en/stable/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From random832 at fastmail.com Thu Feb 11 11:28:58 2016 From: random832 at fastmail.com (Random832) Date: Thu, 11 Feb 2016 11:28:58 -0500 Subject: [Python-ideas] Compact repr operator (i.e., __short_repr__) In-Reply-To: <20160211105941.GN31806@ando.pearwood.info> References: <28D7B647-C173-4B1D-BD74-DE62768FFCC0@lukasa.co.uk> <2603FD3A-035D-4678-B9F7-F08F2E3A0127@lukasa.co.uk> <20160211105941.GN31806@ando.pearwood.info> Message-ID: <1455208138.3512878.518527010.66BCA5EF@webmail.messagingengine.com> On Thu, Feb 11, 2016, at 05:59, Steven D'Aprano wrote: > Some time ago, I started working on a library for continued fractions. I > count at least five useful/standard representations for a continued > fraction. All of these would be equivalent: > (Apart from the first, the remaining four are standard notations used > in mathematics.) How about 1+\dfrac 1{2+\dfrac 1{3+\dfrac 1{4+\dfrac 1 5}}}? A way to get a "mathematical abstract syntax tree" that can be operated on and transformed into that or any of your notations [1+1/(2+1/(3+1/(4+1/5))) might be the default] for numeric objects (and other things such as matrices and vectors) might be nice. From jbvsmo at gmail.com Thu Feb 11 21:14:42 2016 From: jbvsmo at gmail.com (=?UTF-8?Q?Jo=C3=A3o_Bernardo?=) Date: Fri, 12 Feb 2016 00:14:42 -0200 Subject: [Python-ideas] Dict with inverse form Message-ID: Many times I have the need for a dictionary and its inverse form (keys and values swapped). For static dictionaries, it is easy, but sometimes I wish there were a class to hold and update both dictionaries at the same time. Also it will avoid creating 2 variables. Sample: # Current code d = {'foo': 10, 'bar': 12, 'baz': 17} d_inv = {v: k for k, v in d.items()} # My proposal d = InverseDict({'foo': 10, 'bar': 12, 'baz': 17}) d['foo'] == 10 d.inv[10] == 'foo' This class could exist in the collections module. The way the inverse dictionary is accessed could be different (another attribute name or another form): d.I[10] # Like numpy's matrix.T for transpose d.inverse[10] # More verbose d[10:1] # d[10:0] and d[10] would mean the direct dict # while d[10:1] the inverse one. # This would limit the usage to get/set/del only... (-d)[10] # negative symbol but it would require parenthesis. # (~d)[10] should be more correct because of # the operator's name The inverse version of the dict may or may not allow repeated items (overwrite vs error). The important part would be to remove keys from the inverse dict when the value is changed in the direct one: d['foo'] = 1 d.inv[1] == 'foo' d.inv[10] -> will raise KeyError instead of returning 'foo' Regards, Jo?o Bernardo -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Feb 11 21:18:42 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 12 Feb 2016 13:18:42 +1100 Subject: [Python-ideas] Dict with inverse form In-Reply-To: References: Message-ID: On Fri, Feb 12, 2016 at 1:14 PM, Jo?o Bernardo wrote: > Many times I have the need for a dictionary and its inverse form (keys and > values swapped). ... The important part would be to remove keys from the > inverse dict when the value is changed in the direct one: > > d['foo'] == 10 > d.inv[10] == 'foo' > d['foo'] = 1 > d.inv[1] == 'foo' > d.inv[10] -> will raise KeyError instead of returning 'foo' Sounds like what you're looking for is a view, rather than an additional concrete dictionary. In other words, it's a (presumably more efficient) way of doing this: def d.inv[x]: for key in d: if d[key] == x: return key raise KeyError Does that sound right? ChrisA From mahmoud at hatnote.com Thu Feb 11 21:20:24 2016 From: mahmoud at hatnote.com (Mahmoud Hashemi) Date: Thu, 11 Feb 2016 18:20:24 -0800 Subject: [Python-ideas] Dict with inverse form In-Reply-To: References: Message-ID: You hit the nail on the head that inverting dictionaries has a big problem with collisions, as values become keys. One solution to this is to use a MultiDict, a la boltons' OrderedMultiDict: https://boltons.readthedocs.org/en/latest/dictutils.html#boltons.dictutils.OrderedMultiDict.inverted But yes, I use inversions all the time, and would love to see a MultiDict in the collections module to support that use case. Mahmoud https://github.com/mahmoud https://twitter.com/mhashemi http://sedimental.org On Thu, Feb 11, 2016 at 6:14 PM, Jo?o Bernardo wrote: > Many times I have the need for a dictionary and its inverse form (keys and > values swapped). For static dictionaries, it is easy, but sometimes I wish > there were a class to hold and update both dictionaries at the same time. > Also it will avoid creating 2 variables. > > Sample: > > # Current code > d = {'foo': 10, 'bar': 12, 'baz': 17} > d_inv = {v: k for k, v in d.items()} > > # My proposal > d = InverseDict({'foo': 10, 'bar': 12, 'baz': 17}) > > d['foo'] == 10 > d.inv[10] == 'foo' > > This class could exist in the collections module. > The way the inverse dictionary is accessed could be different (another > attribute name or another form): > > d.I[10] # Like numpy's matrix.T for transpose > d.inverse[10] # More verbose > > d[10:1] # d[10:0] and d[10] would mean the direct dict > # while d[10:1] the inverse one. > # This would limit the usage to get/set/del only... > > (-d)[10] # negative symbol but it would require parenthesis. > # (~d)[10] should be more correct because of > # the operator's name > > The inverse version of the dict may or may not allow repeated items > (overwrite vs error). The important part would be to remove keys from the > inverse dict when the value is changed in the direct one: > > d['foo'] = 1 > d.inv[1] == 'foo' > d.inv[10] -> will raise KeyError instead of returning 'foo' > > > Regards, > Jo?o Bernardo > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jbvsmo at gmail.com Thu Feb 11 21:36:15 2016 From: jbvsmo at gmail.com (=?UTF-8?Q?Jo=C3=A3o_Bernardo?=) Date: Fri, 12 Feb 2016 00:36:15 -0200 Subject: [Python-ideas] Dict with inverse form In-Reply-To: References: Message-ID: On Fri, Feb 12, 2016 at 12:18 AM, Chris Angelico wrote: > > Sounds like what you're looking for is a view, rather than an > additional concrete dictionary. If the view offers the same O(1) complexity on amortized case, then it is OK. For the typical use in my experience (small dictionaries and several accesses) space is not a concern, so really adding a concrete dictionary like I've been doing is also fine. Jo?o Bernardo -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Feb 11 22:58:45 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 12 Feb 2016 03:58:45 +0000 (UTC) Subject: [Python-ideas] Incorporating something like byteplay into the stdlib References: <1465993608.2439410.1455249525441.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <1465993608.2439410.1455249525441.JavaMail.yahoo@mail.yahoo.com> tl;dr: We should turn dis.Bytecode into a builtin mutable structure similar to byteplay.Code, to make PEP 511 bytecode transformers implementable. Why? ---- The first problem is that the API for bytecode transformers is incomplete. You don't get, e.g., varnames, so you can't even write a transformer that looks up or modifies locals. But that part is easy to fix, so I won't dwell on it. The second problem is that bytecode is just painful to work with. The peephole optimizer deals with this by just punting and returning the original code whenever it sees anything remotely complicated (which I don't think we want to encourage for all bytecode transformers), and it's _still_ pretty hairy code. And it's the kind of code that can easily harbor hard-to-spot and harder-to-debug bugs (line numbers occasionally off on tracebacks, segfaults on about 1/255 programs that do something uncommon, that kind of fun stuff). The compiler is already doing this work. There's no reason every bytecode processor should have to repeat all of it. I'm not so worried about performance here--technically, fixup is worst-case quadratic, but practically I doubt we spend much time on it--but simplicity. Why should everyone have to repeat dozens of lines of complicated code to write a simple 10-line transformer? What? ----- I played around with a few possibilities from fixed-width bytecode and uncompressed lnotab to a public version of the internal assembler structs, but I think the best one is a flat sequence of instructions, with pseudo-instructions for labels and line numbers, and jump targets just references to those label instructions. The compiler, instead of generating and fixing up bytecode and then calling PyCode_Optimize, would generate this structure, call PyCode_Optimize, and then generate and fix up the result. The object should look a lot like dis.Bytecode, which is basically an iterable of Instruction objects with some extra attributes. But not identical to dis.Bytecode. * It needs a C API, and probably a C implementation. * It should be a mutable sequence, not just an iterable. * Instruction objects should be namedtuples. * It should be possible to replace one with a plain (op, arg) tuple (as with tokenizer.Token objects). * It should be possible to return any iterable of such tuples instead of a Bytecode (again like tokenizer.untokenize). * It doesn't need the "low-level" stuff that dis carries around, because that low level doesn't exist yet. For example, a dis.Instruction for a LOAD_CONST has the actual const value in argval, but it also has the co_consts index in arg. We don't need the latter. * Instead of separate opname and opval fields (and opmap and opname to map between them), we should just make the opcodes an enum. (Actually, this one could probably be pulled out into a standalone suggestion for the dis module.) I think we should just modify dis.Bytecode to handle this use case as well as the existing one (which means it would retain things like Instruction.arg and Bytecode.first_line, they'll just be None if the object wasn't created from a live code object--just like the existing Bytecode.current_offset is None if it's created from a code or function instead of a frame or traceback). The reason we want a mutable sequence is that it makes the most sense for processors that do random-access-y stuff. (Also, as Serhiy suggested, if the fixup code does NOP removal, this can be good for some simple processors, too--you can do bytecode[i] = (dis.NOP, None) while in the middle of iterating bytecode.) But for simpler one-pass processors, yielding instructions is really handy: def constify_globals(bytecode: dis.Bytecode): for op, arg in bytecode: if op == dis.LOAD_GLOBAL: yield (dis.LOAD_CONST, eval(arg, globals())) else: yield (op, arg) def eliminate_double_jumps(bytecode: dis.Bytecode): for instr in bytecode: if dis.hasjump(instr.opcode): target = bytecode[instr.argval.offset + 1] if target.opcode in {dis.JUMP_FORWARD, dis.JUMP_ABSOLUTE}: yield (instr.opcode, target.argval) continue yield instr Anyway, I realize this API is still a little vague, but if people don't hate the idea, I'll try to finish up both the design and the proof of concept this weekend. But obviously, any suggestions or holes poked in the idea would be even better before I do that work than after. Byteplay? --------- Once I started building a proof of concept, I realized what I was building is almost exactly the functionality of byteplay, but with a more dis-like interface, except without the to_code method. Of course that method is the hard part--but it's pretty much the same code I already wrote in the compiler; it's just a matter of separating it out and exposing it nicely. Byteplay.to_code does one additional thing the compiler doesn't do: it verifies that the stack effects of the assembled code are balanced. And, while this used to be a huge pain, the 3.4+ version (which leans on the dis module) is a lot simpler. So, maybe that's worth adding to. One thing Byteplay.from_code does that I _don't_ think we want is recursively transforming any code consts into Byteplay objects. (Because the compiler works inside-out; your nested functions are already optimized by the time you're being called.) Anyway, the advantage here would be that import hooks and decorators can use the exact same API as PEP 511 transformers, except they have to call to_code at the end instead of just returning an object and letting the compiler call it for them. (They'll also probably have to recurse manually on (dis.Bytecode(const) for const in co_consts if isinstance(const, types.CodeType)), while PEP 511 transformers won't.) Alternatives ------------ We could just pass code objects (or all the separate pieces, instead of some of them), and then the docs could suggest using byteplay for non-trivial bytecode transformers, and then everyone will just end up using byteplay. So, what's wrong with that? The biggest problem is that, after each new Python release, anyone using a bytecode transformer will have to wait until byteplay is updated before they can update Python. Also, some people are going to try to do it from scratch instead of adding a dependency, and get it wrong, and publish processors that work on most code but in weird edge cases cause a SystemError or even a segfault which will be a nightmare to debug. And it's annoying that anyone who wants to hack on the bytecode representation pretty much has to disable the peephole optimizer. Or we could just remove bytecode transformers from PEP 511. PEP 511 still seems worth doing to me, even if it only has AST transformers, especially since all or nearly all of the examples anyone's come up with for it are implementable (and easier to implement) at the AST level. From abarnert at yahoo.com Thu Feb 11 23:07:47 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 12 Feb 2016 04:07:47 +0000 (UTC) Subject: [Python-ideas] Dict with inverse form In-Reply-To: References: Message-ID: <935127780.2464284.1455250067522.JavaMail.yahoo@mail.yahoo.com> On Thursday, February 11, 2016 6:15 PM, Jo?o Bernardo wrote: >Many times I have the need for a dictionary and its inverse form (keys and values swapped). For static dictionaries, it is easy, but sometimes I wish there were a class to hold and update both dictionaries at the same time. Also it will avoid creating 2 variables. This really sounds like something you should just flesh out and put on PyPI, and then suggest for stdlib inclusion once it's stable. And not just for the usual reasons, but because I think there are some design decisions that you'll want to work through and then play with in real code before you're sure you're happy with it. For example, when I've built something like this, sometimes I want errors on duplicate values, sometimes just pick one arbitrarily, sometimes I want the inverse dict to store sets of values, and sometimes I even want _both_ to store sets of values. Do you want to do just one of those, or two or more? Controlled by flags? Or different attribute names? Or different classes? From phyks at phyks.me Fri Feb 12 05:00:53 2016 From: phyks at phyks.me (Phyks) Date: Fri, 12 Feb 2016 11:00:53 +0100 Subject: [Python-ideas] Sharing modules across virtualenvs Message-ID: <56BDAD55.9040608@phyks.me> Hi, I was wondering recently about the way virtualenvs work, and did not find much information on the design choices. Sorry if this is not the right mailing-list to post about it, but it seemed to be. As far as I understand, virtualenvs work by modifying the PYTHONHOME and then the folders in which to look up to import modules. Are there any specific reasons for this implementation, and not a "/lib"-like implementation, putting the various versions of the module in a single folder, with a version naming, and symlinking the up-to-date one? Then, virtualenvs could simply be applied as filters on this specific folder, to hide part of the available module. Contrary to the current implementation, this means building (and storing, although storage is said to be cheap) large Python modules only once for your system, and share them. I do not doubt there were great reasons to choose to move the lookup path, however. But I do not know which one they are. Thanks! -- Phyks From ncoghlan at gmail.com Fri Feb 12 05:53:38 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 12 Feb 2016 20:53:38 +1000 Subject: [Python-ideas] Sharing modules across virtualenvs In-Reply-To: <56BDAD55.9040608@phyks.me> References: <56BDAD55.9040608@phyks.me> Message-ID: On 12 February 2016 at 20:00, Phyks wrote: > Hi, > > I was wondering recently about the way virtualenvs work, and did not > find much information on the design choices. Sorry if this is not the > right mailing-list to post about it, but it seemed to be. > > As far as I understand, virtualenvs work by modifying the PYTHONHOME and > then the folders in which to look up to import modules. > > Are there any specific reasons for this implementation, and not a > "/lib"-like implementation, putting the various versions of the module > in a single folder, with a version naming, and symlinking the up-to-date > one? Symlinking unfortunately doesn't solve the problem for a couple of reasons: * symlinks are still basically unusable on Windows * symlinks and https://packaging.python.org/en/latest/specifications/#recording-installed-distributions don't get along Module sharing between virtual environments *could* still be implemented based on *.pth files, but: * you'd need to update an installer (probably pip) to handle shared installs and uninstalls * you'd have a refcounting problem on deciding when it was OK to uninstall versions * you'd run into the startup performance problems that can arise with .pth files and adding lots of entries to sys.path And even if you do manage to solve all those problems, at the end of it, you'll have essentially reinvented some of the core components of nix: https://nixos.org/nix/ > Then, virtualenvs could simply be applied as filters on this specific > folder, to hide part of the available module. Contrary to the current > implementation, this means building (and storing, although storage is > said to be cheap) large Python modules only once for your system, and > share them. The implicit wheel cache used in recent versions of pip tackles that problem in a different way (the built wheel fields are cached, so you don't have to build them again, but they still get unpacked separately into each virtualenv that needs them) Cheers, Nick. P.S. This is more a distutils-sig topic than a python-ideas one, so if you'd like to learn more or explore the idea further, I'd suggest following up there -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From encukou at gmail.com Fri Feb 12 05:57:12 2016 From: encukou at gmail.com (Petr Viktorin) Date: Fri, 12 Feb 2016 11:57:12 +0100 Subject: [Python-ideas] Incorporating something like byteplay into the stdlib In-Reply-To: <1465993608.2439410.1455249525441.JavaMail.yahoo@mail.yahoo.com> References: <1465993608.2439410.1455249525441.JavaMail.yahoo.ref@mail.yahoo.com> <1465993608.2439410.1455249525441.JavaMail.yahoo@mail.yahoo.com> Message-ID: <56BDBA88.4010502@gmail.com> On 02/12/2016 04:58 AM, Andrew Barnert via Python-ideas wrote: > tl;dr: We should turn dis.Bytecode into a builtin mutable structure similar to byteplay.Code, to make PEP 511 bytecode transformers implementable. > > Why? > > ---- > > The first problem is that the API for bytecode transformers is incomplete. You don't get, e.g., varnames, so you can't even write a transformer that looks up or modifies locals. But that part is easy to fix, so I won't dwell on it. > > > The second problem is that bytecode is just painful to work with. The peephole optimizer deals with this by just punting and returning the original code whenever it sees anything remotely complicated (which I don't think we want to encourage for all bytecode transformers), and it's _still_ pretty hairy code. And it's the kind of code that can easily harbor hard-to-spot and harder-to-debug bugs (line numbers occasionally off on tracebacks, segfaults on about 1/255 programs that do something uncommon, that kind of fun stuff). > > The compiler is already doing this work. There's no reason every bytecode processor should have to repeat all of it. I'm not so worried about performance here--technically, fixup is worst-case quadratic, but practically I doubt we spend much time on it--but simplicity. Why should everyone have to repeat dozens of lines of complicated code to write a simple 10-line transformer? tl;dr: Why would they, indeed? Couldln't just import a library that does it for them? That's what libraries are for. > What? > ----- > > I played around with a few possibilities from fixed-width bytecode and uncompressed lnotab to a public version of the internal assembler structs, but I think the best one is a flat sequence of instructions, with pseudo-instructions for labels and line numbers, and jump targets just references to those label instructions. > > The compiler, instead of generating and fixing up bytecode and then calling PyCode_Optimize, would generate this structure, call PyCode_Optimize, and then generate and fix up the result. > > The object should look a lot like dis.Bytecode, which is basically an iterable of Instruction objects with some extra attributes. But not identical to dis.Bytecode. > > * It needs a C API, and probably a C implementation. > * It should be a mutable sequence, not just an iterable. > * Instruction objects should be namedtuples. > * It should be possible to replace one with a plain (op, arg) tuple (as with tokenizer.Token objects). > * It should be possible to return any iterable of such tuples instead of a Bytecode (again like tokenizer.untokenize). > * It doesn't need the "low-level" stuff that dis carries around, because that low level doesn't exist yet. For example, a dis.Instruction for a LOAD_CONST has the actual const value in argval, but it also has the co_consts index in arg. We don't need the latter. > * Instead of separate opname and opval fields (and opmap and opname to map between them), we should just make the opcodes an enum. (Actually, this one could probably be pulled out into a standalone suggestion for the dis module.) > > I think we should just modify dis.Bytecode to handle this use case as well as the existing one (which means it would retain things like Instruction.arg and Bytecode.first_line, they'll just be None if the object wasn't created from a live code object--just like the existing Bytecode.current_offset is None if it's created from a code or function instead of a frame or traceback). > > > The reason we want a mutable sequence is that it makes the most sense for processors that do random-access-y stuff. (Also, as Serhiy suggested, if the fixup code does NOP removal, this can be good for some simple processors, too--you can do bytecode[i] = (dis.NOP, None) while in the middle of iterating bytecode.) > > But for simpler one-pass processors, yielding instructions is really handy: > > def constify_globals(bytecode: dis.Bytecode): > for op, arg in bytecode: > if op == dis.LOAD_GLOBAL: > yield (dis.LOAD_CONST, eval(arg, globals())) > else: > yield (op, arg) > > def eliminate_double_jumps(bytecode: dis.Bytecode): > for instr in bytecode: > if dis.hasjump(instr.opcode): > target = bytecode[instr.argval.offset + 1] > if target.opcode in {dis.JUMP_FORWARD, dis.JUMP_ABSOLUTE}: > yield (instr.opcode, target.argval) > continue > yield instr > > Anyway, I realize this API is still a little vague, but if people don't hate the idea, I'll try to finish up both the design and the proof of concept this weekend. But obviously, any suggestions or holes poked in the idea would be even better before I do that work than after. Are you sure the API you come up with will be good enough to be immortalized in CPython itself? Why not put it on PyPI, and only look into immortalizing it after it survives some real use (and competition)? Sure, your ideas look great now, but any new API needs some validation. > Byteplay? > --------- > > Once I started building a proof of concept, I realized what I was building is almost exactly the functionality of byteplay, but with a more dis-like interface, except without the to_code method. > > Of course that method is the hard part--but it's pretty much the same code I already wrote in the compiler; it's just a matter of separating it out and exposing it nicely. > > Byteplay.to_code does one additional thing the compiler doesn't do: it verifies that the stack effects of the assembled code are balanced. And, while this used to be a huge pain, the 3.4+ version (which leans on the dis module) is a lot simpler. So, maybe that's worth adding to. > > One thing Byteplay.from_code does that I _don't_ think we want is recursively transforming any code consts into Byteplay objects. (Because the compiler works inside-out; your nested functions are already optimized by the time you're being called.) > > Anyway, the advantage here would be that import hooks and decorators can use the exact same API as PEP 511 transformers, except they have to call to_code at the end instead of just returning an object and letting the compiler call it for them. (They'll also probably have to recurse manually on (dis.Bytecode(const) for const in co_consts if isinstance(const, types.CodeType)), while PEP 511 transformers won't.) Why can't Byteplay be fixed? (Or modularized, or rewritten?) > > Alternatives > ------------ > > We could just pass code objects (or all the separate pieces, instead of some of them), and then the docs could suggest using byteplay for non-trivial bytecode transformers, and then everyone will just end up using byteplay. > > So, what's wrong with that? The biggest problem is that, after each new Python release, anyone using a bytecode transformer will have to wait until byteplay is updated before they can update Python. That's a problem common to any library. You always need to wait until your dependencies are ported to a new version of a language. Byteplay looks special because it explicitly depends on an interface with no backwards compatibility guarantees. But once everyone starts writing bytecode transformers, it won't be special any more: *any* bytecode utility library will have the same problem. Do we add them all to force them to follow Python's release cycle? > Also, some people are going to try to do it from scratch instead of adding a dependency, and get it wrong, and publish processors that work on most code but in weird edge cases cause a SystemError or even a segfault which will be a nightmare to debug. Shame on them. But is this really a reason to include one blessed convenience interface in the language itself? > And it's annoying that anyone who wants to hack on the bytecode representation pretty much has to disable the peephole optimizer. I'm sure this individual problem can be solved in a less invasive way. > Or we could just remove bytecode transformers from PEP 511. PEP 511 still seems worth doing to me, even if it only has AST transformers, especially since all or nearly all of the examples anyone's come up with for it are implementable (and easier to implement) at the AST level. This looks like an OK solution. It can always be added back. From victor.stinner at gmail.com Fri Feb 12 07:05:21 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 12 Feb 2016 13:05:21 +0100 Subject: [Python-ideas] Incorporating something like byteplay into the stdlib In-Reply-To: <1465993608.2439410.1455249525441.JavaMail.yahoo@mail.yahoo.com> References: <1465993608.2439410.1455249525441.JavaMail.yahoo.ref@mail.yahoo.com> <1465993608.2439410.1455249525441.JavaMail.yahoo@mail.yahoo.com> Message-ID: Hi, 2016-02-12 4:58 GMT+01:00 Andrew Barnert via Python-ideas : > tl;dr: We should turn dis.Bytecode into a builtin mutable structure similar to byteplay.Code, to make PEP 511 bytecode transformers implementable. Hum, it looks like your email is highly coupled to the PEP 511. First of all, I really want to support bytecode transformer because I would like to be able to disable the peephole optimizer. Having the peephole registered in code transformers as AST transformers make the whole PEP more consistent. There is no more special case for peephole optimizer. > Why? > > ---- > > The first problem is that the API for bytecode transformers is incomplete. You don't get, e.g., varnames, so you can't even write a transformer that looks up or modifies locals. But that part is easy to fix, so I won't dwell on it. I agree that we can enhance the Python stdlib to ease manipulation of bytecode, but I disagree that it's a requirement. It's ok to use an external library (like byteplay) for that. > The second problem is that bytecode is just painful to work with. The peephole optimizer deals with this by just punting and returning the original code whenever it sees anything remotely complicated (which I don't think we want to encourage for all bytecode transformers), and it's _still_ pretty hairy code. And it's the kind of code that can easily harbor hard-to-spot and harder-to-debug bugs (line numbers occasionally off on tracebacks, segfaults on about 1/255 programs that do something uncommon, that kind of fun stuff). Sorry, I don't understand. Are you writing that the CPython peephole optimizer produces invalid code? Or are you talking about bugs in your own code? I'm not aware of bugs in the peephole optimizer. > The compiler is already doing this work. There's no reason every bytecode processor should have to repeat all of it. I'm not so worried about performance here--technically, fixup is worst-case quadratic, but practically I doubt we spend much time on it--but simplicity. Why should everyone have to repeat dozens of lines of complicated code to write a simple 10-line transformer? Hum, are you talking about the API proposed in the PEP 511? I understand that you are saying the API only takes a whole code object as input and produces a code object as output. An optimizer usually needs a different structure to be able to modify the code. If you have multiple bytecode transformers, you have to repeat these "disassemble" and "assemble" steps, right? I don't think that we will have plently bytecode optimizers in the wild. Even if two or three major bytecode optimizers become popular, are you sure that we will want to combine them? I expect that a single optimizer implements *all* optimizations. I don't see the point of running multiple optimizers to implement multiple optimizations steps. For example, my fatoptimizer AST optimizer implements multiple steps, but *internally*. It is only called once on the AST. I don't think that performance of importing modules really matters. My PEP 511 is mostly written for compilation ahead of time, to support complex and expensive optimizers. The real problem is to run a script: "python script.py" always has to execute all code transformers. For scripts, I hesitate to simply disable expensive optimizers, or maybe even disable all optimizers. For example, a script can run less than 50 ms, is it worth to spend 10 ms to optimize it to a get speedup of 1 ms? (no) The problem of using a specific format for bytecode rather than a code object is that we will have to maintain it. I'm not sure that all bytecode optimizer want the same internal structures. For some kind of optimizations, a sequential list of instructions is enough. For some other optimizations, you need to split blocks of code to have a representation of the exacty "control flow". I'm not sure that one structure is enough to cover all cases. So I prefer to let optimizers "disassemble" and "assemble" themself the bytecode. Last point, the PEP 511 has to take in account the existing peephole optimizer implement in C. If you really want to use a different structure, you will have to reimplement the peephole optimizer with your new API. Since my target is AST, I'm not really interested by that :-) What do you think? > I played around with a few possibilities from fixed-width bytecode and uncompressed lnotab to a public version of the internal assembler structs, but I think the best one is a flat sequence of instructions, with pseudo-instructions for labels and line numbers, and jump targets just references to those label instructions. Could you try to reimplement the whole peephole optimizer to see if it benefit of your design? I played with bytecode in the past. At the send, I started to implement optimizations which can be implemented simpler at AST level. Why do you prefer bytecode over AST? Your example of converting globals to constants became trivial to implement using my new ast.Constant node. > * It needs a C API, and probably a C implementation. I don't like extending the Python C API, it is already very large, we have too many functions :-p A C API is more expensive to maintain than a Python API. I would prefer to continue to play with an external module (hosted on PyPI) to not pay the price of maintenance! By the way, what is the final goal? Do you plan to implement a new ultra optimized bytecode optimizer? If yes, do you plan to integrate it into CPython? If no, I don't think that we have to pay the price of maintenance for such "toy" project. The design of my PEP 511 is to allow to support pluggable and *external* optimizers. I don't think that any code optimizer in the wild is mature enough to enter into CPython directly. > Anyway, I realize this API is still a little vague, (...) It doesn't fit requirements to put something into the Python stdlib. Usually, we experiment stuff on PyPI, wait until it becomes mature, and then propose to integrate it. It looks like you are talking about creating a new API and directly put it into the stdlib, right? Are you sure that it will not change next 2 years? Not any single minor change? > We could just pass code objects (or all the separate pieces, instead of some of them), and then the docs could suggest using byteplay for non-trivial bytecode transformers, and then everyone will just end up using byteplay. Again, you need to elaborate your rationale. What are your use cases? Which kind of optimizations do you want to implement? > So, what's wrong with that? The biggest problem is that, after each new Python release, anyone using a bytecode transformer will have to wait until byteplay is updated before they can update Python. Why not contributing to byteplay to support the next CPython release? I don't understand your problem here. > Or we could just remove bytecode transformers from PEP 511. PEP 511 still seems worth doing to me, even if it only has AST transformers, especially since all or nearly all of the examples anyone's come up with for it are implementable (and easier to implement) at the AST level. I want to plug the existing peephole optimizer into my PEP 511 since it is an obvious *code* transformer. Even if changes are minor, some users want to disable it because it really changes the code. It would help code coverage for example. Victor From egregius313 at gmail.com Fri Feb 12 07:09:38 2016 From: egregius313 at gmail.com (Edward Minnix) Date: Fri, 12 Feb 2016 07:09:38 -0500 Subject: [Python-ideas] Generator unpacking Message-ID: Hello, I am a new programmer and have been using Python for a few months. I was experimenting the other day with unpacking (lists, tuples, etc.) And I realized something: when you type: >>> a, b, *rest = count() The interpreter gets caught in an infinite loop which I could not kill without terminating my REPL. Would there be a way to add generators to the unpackables, even if it was only in the front? Thanks, Ed M -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Feb 12 07:46:55 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 12 Feb 2016 23:46:55 +1100 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: Message-ID: <20160212124650.GU31806@ando.pearwood.info> On Fri, Feb 12, 2016 at 07:09:38AM -0500, Edward Minnix wrote: > Hello, > > I am a new programmer and have been using Python for a few months. > > I was experimenting the other day with unpacking (lists, tuples, etc.) And > I realized something: > > when you type: > > >>> a, b, *rest = count() > > The interpreter gets caught in an infinite loop which I could not kill > without terminating my REPL. That's because count() is an infinite generator. > Would there be a way to add generators to the unpackables, even if it was > only in the front? Generators can already be unpacked. They just have to be finite: py> def gen(): ... yield 1 ... yield 2 ... yield 3 ... yield 4 ... py> a, b, *c = gen() py> a 1 py> b 2 py> c [3, 4] I'm surprised that the unpacking can't be interrupted with Ctrl-C. I think that is a bug. -- Steve From ncoghlan at gmail.com Fri Feb 12 08:22:28 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 12 Feb 2016 23:22:28 +1000 Subject: [Python-ideas] Generator unpacking In-Reply-To: <20160212124650.GU31806@ando.pearwood.info> References: <20160212124650.GU31806@ando.pearwood.info> Message-ID: On 12 February 2016 at 22:46, Steven D'Aprano wrote: > I'm surprised that the unpacking can't be interrupted with Ctrl-C. I > think that is a bug. It's a consequence of looping in C over an infinite iterator also implemented in C - "sum(itertools.count())" will hang the same way, since control never gets back to the eval loop to notice that Ctrl-C has been pressed. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From jbvsmo at gmail.com Fri Feb 12 08:27:00 2016 From: jbvsmo at gmail.com (=?UTF-8?Q?Jo=C3=A3o_Bernardo?=) Date: Fri, 12 Feb 2016 11:27:00 -0200 Subject: [Python-ideas] Dict with inverse form In-Reply-To: <935127780.2464284.1455250067522.JavaMail.yahoo@mail.yahoo.com> References: <935127780.2464284.1455250067522.JavaMail.yahoo@mail.yahoo.com> Message-ID: On Fri, Feb 12, 2016 at 2:07 AM, Andrew Barnert wrote: > > For example, when I've built something like this, sometimes I want errors > on duplicate values, sometimes just pick one arbitrarily, sometimes I want > the inverse dict to store sets of values, and sometimes I even want _both_ > to store sets of values. Do you want to do just one of those, or two or > more? Controlled by flags? Or different attribute names? Or different > classes? > My use case is quite simple. I believe I always used this pattern on a one-to-one mapping, so allowing repeated values to overwrite or throwing an error is not exactly important to me, but it may be to someone else. Someone suggested a MultiDict sort of thing, but I don't think I need that amount of complexity here. I will write a concrete class and post on PyPI then. Jo?o Bernardo -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmueller at python-academy.de Fri Feb 12 08:53:45 2016 From: mmueller at python-academy.de (=?UTF-8?Q?Mike_M=c3=bcller?=) Date: Fri, 12 Feb 2016 14:53:45 +0100 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: Message-ID: <56BDE3E9.6030904@python-academy.de> Am 12.02.16 um 13:09 schrieb Edward Minnix: > Hello, > > I am a new programmer and have been using Python for a few months. > > I was experimenting the other day with unpacking (lists, tuples, etc.) And I > realized something: > > when you type: > >>>> a, b, *rest = count() > > The interpreter gets caught in an infinite loop which I could not kill without > terminating my REPL. > > Would there be a way to add generators to the unpackables, even if it was only > in the front? You can use `islice` to get the first two value from `count`: from itertools import count, islice a, b = islice(count(), None, 2) Mike > Thanks, > Ed M > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From rymg19 at gmail.com Fri Feb 12 11:15:40 2016 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Fri, 12 Feb 2016 10:15:40 -0600 Subject: [Python-ideas] Making (?x) less scary Message-ID: <2D44226C-B26E-445E-91AB-C068EE041554@gmail.com> I was reading re's docs and came across this: Note that the?(?x)?flag changes how the expression is parsed. It should be used first in the expression string, or after one or more whitespace characters. If there are non-whitespace characters before the flag, the results are undefined. Is there a particular reason for this being undefined? It seems kind of un-Pythonic to me and more like C/C++. Would it be possible to instead throw an exception, e.g. ValueError? -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Feb 12 11:39:53 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 12 Feb 2016 08:39:53 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <20160212124650.GU31806@ando.pearwood.info> Message-ID: On Fri, Feb 12, 2016 at 5:22 AM, Nick Coghlan wrote: > On 12 February 2016 at 22:46, Steven D'Aprano wrote: >> I'm surprised that the unpacking can't be interrupted with Ctrl-C. I >> think that is a bug. > > It's a consequence of looping in C over an infinite iterator also > implemented in C - "sum(itertools.count())" will hang the same way, > since control never gets back to the eval loop to notice that Ctrl-C > has been pressed. We should occasionally check for signals in such loops. We do that in other places. -- --Guido van Rossum (python.org/~guido) From abarnert at yahoo.com Fri Feb 12 12:04:45 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 12 Feb 2016 09:04:45 -0800 Subject: [Python-ideas] Incorporating something like byteplay into the stdlib In-Reply-To: <56BDBA88.4010502@gmail.com> References: <1465993608.2439410.1455249525441.JavaMail.yahoo.ref@mail.yahoo.com> <1465993608.2439410.1455249525441.JavaMail.yahoo@mail.yahoo.com> <56BDBA88.4010502@gmail.com> Message-ID: On Feb 12, 2016, at 02:57, Petr Viktorin wrote: > > Are you sure the API you come up with will be good enough to be > immortalized in CPython itself? That's exactly why I'm sticking as close as possible to the existing dis module--which is already immortalized in CPython--and leaning on the byteplay module--which has clearly won the competition in the wild--for things that dis doesn't cover. > Why not put it on PyPI, and only look into immortalizing it after it > survives some real use (and competition)? You want me to put a module on PyPI that patches the compiler and the peephole optimizer? I doubt that's even possible, and it would be a very bad idea if it were. If you weren't reading the proposal, just skimming for individual things to disagree with, you might have missed that the main idea is to provide a useful interface for the compiler to pass to PyCode_Optimize, and exposing functions to convert back and forth to that format a la byteplay is a secondary idea that adds the benefits that import hooks and decorators could then use the same interface as PEP 511 optimizers. You can't do the secondary thing without the main thing. >> Anyway, the advantage here would be that import hooks and decorators can use the exact same API as PEP 511 transformers, except they have to call to_code at the end instead of just returning an object and letting the compiler call it for them. (They'll also probably have to recurse manually on (dis.Bytecode(const) for const in co_consts if isinstance(const, types.CodeType)), while PEP 511 transformers won't.) > > Why can't Byteplay be fixed? (Or modularized, or rewritten?) If the problem is the interface to the peephole optimizer and PEP 511 optimizers, fixing byteplay doesn't help that. If we did fix the PyCode_Optimize interface, then sure, instead of exposing the same code, we could keep it hidden and separately change byteplay to duplicate the same interface we could have exposed. But why? Also, byteplay is actually a very small module, and the only hard part of it is the part that keeps in sync with the dis module across each new Python version and does the same things backward-compatibly for pre-3.4 versions. There's really nothing to "modularize" there. If 3.6 had an extended dis module and C API as I've suggested, in 3.6+ it could just become a shim around what's already in the stdlib, which would then be useful for backward compat for anyone who wants to write bytecode processing import hooks or decorators that work with 3.5 and 3.6, or maybe even with 2.7 and 3.6. I think that probably _is_ worth doing, but that's not a proposal for stdlib, it's a proposal for byteplay (and only if this proposal--and PEP 511, of course--goes through). >> So, what's wrong with that? The biggest problem is that, after each new Python release, anyone using a bytecode transformer will have to wait until byteplay is updated before they can update Python. > > That's a problem common to any library. You always need to wait until > your dependencies are ported to a new version of a language. In theory, yes. In practice, when Python 3.6 comes out, every library in my site-packages from 3.5 will continue to work, most of them without even needing a recompile, except for byteplay, and that's been the same for almost every release in Python's history. So that does make it special in practice. > Byteplay looks special because it explicitly depends on an interface > with no backwards compatibility guarantees. But once everyone starts > writing bytecode transformers, it won't be special any more: *any* > bytecode utility library will have the same problem. Do we add them all > to force them to follow Python's release cycle? Since byteplay is the only one that's still alive and being updated, and "all of them" is just one, yes, that's exactly what we do. And "forcing it to follow Python's release cycle" is not a problem. The only time there are significant updates to the library is when there's a new version of Python to deal with, or when someone finds a bug in the way it deals with the latest version of Python that wasn't discovered immediately. At any rate, if everyone begins writing bytecode transformers, that makes the existing problem worse, it doesn't solve it. >> Also, some people are going to try to do it from scratch instead of adding a dependency, and get it wrong, and publish processors that work on most code but in weird edge cases cause a SystemError or even a segfault which will be a nightmare to debug. > > Shame on them. But is this really a reason to include one blessed > convenience interface in the language itself? Yes. Expecting people to update the lnotab, etc., as currently designed is a bad idea, especially given the way things go wrong if they get it wrong. If we're providing an interface that exposes this stuff to normal developers and expects them to deal with it, we should have tools to make it tractable. >> And it's annoying that anyone who wants to hack on the bytecode representation pretty much has to disable the peephole optimizer. > > I'm sure this individual problem can be solved in a less invasive way. I'm not. Short of rewriting the peephole optimizer to a new interface, how would you solve it? I think this is a pretty minor problem in the first place (how often do people want to hack on the bytecode representation? and how often do people do so who aren't pretty experienced with Python and CPython?), which is why I tossed it in at the end, after the real problems. >> Or we could just remove bytecode transformers from PEP 511. PEP 511 still seems worth doing to me, even if it only has AST transformers, especially since all or nearly all of the examples anyone's come up with for it are implementable (and easier to implement) at the AST level. > > This looks like an OK solution. It can always be added back. Well, I assume Victor has an argument for why it's not OK, but I think it would make things simpler. From g.rodola at gmail.com Fri Feb 12 12:51:44 2016 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Fri, 12 Feb 2016 18:51:44 +0100 Subject: [Python-ideas] atexit.register_w_signals() Message-ID: http://grodola.blogspot.com/2016/02/how-to-always-execute-exit-functions-in-py.html Thoughts? -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Feb 12 12:49:08 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 13 Feb 2016 04:49:08 +1100 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <20160212124650.GU31806@ando.pearwood.info> Message-ID: <20160212174907.GV31806@ando.pearwood.info> On Fri, Feb 12, 2016 at 08:39:53AM -0800, Guido van Rossum wrote: > On Fri, Feb 12, 2016 at 5:22 AM, Nick Coghlan wrote: > > On 12 February 2016 at 22:46, Steven D'Aprano wrote: > >> I'm surprised that the unpacking can't be interrupted with Ctrl-C. I > >> think that is a bug. > > > > It's a consequence of looping in C over an infinite iterator also > > implemented in C - "sum(itertools.count())" will hang the same way, > > since control never gets back to the eval loop to notice that Ctrl-C > > has been pressed. > > We should occasionally check for signals in such loops. We do that in > other places. http://bugs.python.org/issue26351 -- Steve From ethan at stoneleaf.us Fri Feb 12 13:11:02 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 12 Feb 2016 10:11:02 -0800 Subject: [Python-ideas] atexit.register_w_signals() In-Reply-To: References: Message-ID: <56BE2036.8010503@stoneleaf.us> On 02/12/2016 09:51 AM, Giampaolo Rodola' wrote: > http://... > Thoughts? If you have an idea, please post that idea here, not a link. -- ~Ethan~ From g.rodola at gmail.com Fri Feb 12 13:36:28 2016 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Fri, 12 Feb 2016 19:36:28 +0100 Subject: [Python-ideas] atexit.register_w_signals() In-Reply-To: <56BE2036.8010503@stoneleaf.us> References: <56BE2036.8010503@stoneleaf.us> Message-ID: On Fri, Feb 12, 2016 at 7:11 PM, Ethan Furman wrote: > On 02/12/2016 09:51 AM, Giampaolo Rodola' wrote: > > http://... >> Thoughts? >> > > If you have an idea, please post that idea here, not a link. > > -- > ~Ethan~ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > Of course you are right, I'm sorry. So, one problem with atexit.register() is that it won't run the registered function in case the process is killed by a signal. import atexit, os, signal @atexit.register def cleanup(): print("on exit") # XXX this never gets printed os.kill(os.getpid(), signal.SIGTERM) The correct way to do that is to use signal.signal(). The problem with that though is that in case a third-party module has already registered a function for that signal, the new function will overwrite the old one: import os, signal def old(*args): print("old") # XXX this never gets printed def new(*args): print("new") signal.signal(signal.SIGTERM, old) signal.signal(signal.SIGTERM, new) os.kill(os.getpid(), signal.SIGTERM) Also, we would still have to use atexit.register() so that the new function is called also on "clean" interpreter exit and take into account other signals other than SIGTERM which would cause the process to terminate (SIGINT, etc.). My proposal is to evaluate adding a higher-level function which takes care of all these details, providing better chances to execute the exit function than atexit.register or signal.signal alone. I also noticed that every time I end up working on a new code base, most of the time people are using either atexit or signal module without knowing these implications. It also took quite a bit to do the same and come up with this code. Code: https://gist.github.com/giampaolo/6f07f6cd7dd1c667b0bb Relevant blog post: http://grodola.blogspot.com/2016/02/how-to-always-execute-exit-functions-in-py.html Thoughts? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Feb 12 13:46:13 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 12 Feb 2016 10:46:13 -0800 Subject: [Python-ideas] atexit.register_w_signals() In-Reply-To: References: <56BE2036.8010503@stoneleaf.us> Message-ID: <56BE2875.3000702@stoneleaf.us> On 02/12/2016 10:36 AM, Giampaolo Rodola' wrote: > So, one problem with atexit.register() is that it won't run the > registered function in case the process is killed by a signal. [...] > The correct way to do that is to use signal.signal(). The problem with > that though is that in case a third-party module has > already registered a function for that signal, the new function will > overwrite the old one: [...] > I also noticed that every time I end up working on a new code base, most > of the time people are using either atexit or signal module without > knowing these implications. It also took quite a bit to do the same and > come up with this code. > > Code: > https://gist.github.com/giampaolo/6f07f6cd7dd1c667b0bb > > Relevant blog post: > http://grodola.blogspot.com/2016/02/how-to-always-execute-exit-functions-in-py.html The idea seems sound. Have you considered releasing your code as a pypi package? -- ~Ethan~ From rosuav at gmail.com Fri Feb 12 14:00:25 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 13 Feb 2016 06:00:25 +1100 Subject: [Python-ideas] atexit.register_w_signals() In-Reply-To: References: <56BE2036.8010503@stoneleaf.us> Message-ID: On Sat, Feb 13, 2016 at 5:36 AM, Giampaolo Rodola' wrote: > Also, we would still have to use atexit.register() so that the new function > is called also on "clean" interpreter exit and take into account other > signals other than SIGTERM which would cause the process to terminate > (SIGINT, etc.). My proposal is to evaluate adding a higher-level function > which takes care of all these details, providing better chances to execute > the exit function than atexit.register or signal.signal alone. The trouble is, you can't know which signals will cause a process to terminate. According to 'man signal', my system has these signals set to terminate the process by default: HUP, INT, KILL, PIPE, ALRM, TERM, and USR1/USR2 etc; and these will terminate and dump core: QUIT, ILL, ABRT, and FPE. Which of them is it correct to register your atexit handler for? If you get SIGILL or SIGFPE, you can't depend on program state (they should never be sent to a Python program, so receiving one almost certainly means an interpreter bug or other major issue). SIGINT should result in KeyboardInterrupt, so I wouldn't hook that - let normal exception handling take care of it. SIGHUP, SIGPIPE, and SIGALRM are often changed in disposition; will you hook them only if they're on their defaults of terminating the process? (BTW, the article you link to suggests that Ctrl-D sends SIGQUIT. It doesn't; what you'll normally get from Ctrl-D is EOF on stdin.) Ultimately, "exit functions" are a bit of a hairy area to get into, because so much isn't guaranteed. There's no one best way to ensure that something happens on termination; just off the top of my head, I can think of a few ways (on Linux, and most other POSIX systems; no idea about Windows): * Create a process that just creates a (some) subprocess(es) and waits on it (them). When it gets told that its child is dead, it does the cleanup work (removing a PID file, for instance). * Divide the process into a client and a server. If the server notices that the client's missing, it performs an unclean termination cleanup. (Example: PostgreSQL will roll back your transaction if you disconnect without committing.) * Get notified by the system (eg Upstart, SystemD) when the process is dead, with an event trigger or state trigger. * Connect to the process via a debugger harness and monitor it for termination. Not one of these is a perfect solution (they can't use program internals the way a registered exit function can), but neither is anything inside the process (for starters, you'll never be able to handle SIGKILL that way). I'm not sure how useful it is to make a simple one-liner that claims to make your function run on termination, but still can't guarantee it; you'll still need to dig into the docs to figure out exactly what circumstances will and won't cause it to run. I agree with Ethan about PyPI though. Give the code a bit of visibility, see who picks it up. ChrisA From abarnert at yahoo.com Fri Feb 12 14:16:16 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 12 Feb 2016 11:16:16 -0800 Subject: [Python-ideas] Incorporating something like byteplay into the stdlib In-Reply-To: References: <1465993608.2439410.1455249525441.JavaMail.yahoo.ref@mail.yahoo.com> <1465993608.2439410.1455249525441.JavaMail.yahoo@mail.yahoo.com> Message-ID: On Feb 12, 2016, at 04:05, Victor Stinner wrote: > > Hi, > > 2016-02-12 4:58 GMT+01:00 Andrew Barnert via Python-ideas > : >> tl;dr: We should turn dis.Bytecode into a builtin mutable structure similar to byteplay.Code, to make PEP 511 bytecode transformers implementable. > > Hum, it looks like your email is highly coupled to the PEP 511. Very much so. The basic idea could be done without PEP 511, but there wouldn't be any urgency to doing it if the only client is the peephole optimizer... > First of all, I really want to support bytecode transformer because I > would like to be able to disable the peephole optimizer. Having the > peephole registered in code transformers as AST transformers make the > whole PEP more consistent. There is no more special case for peephole > optimizer. OK. But if the peephole optimizer will almost always be the only bytecode processor present, with everything else being AST processors, then it already is special, and making it more generic in a way that still doesn't encompass anything else doesn't change that, it just adds complexity. Personally, I think there probably are useful optimizations that can only be done by bytecode. But the fact that nobody has come up with any compelling examples (while there are lots of compelling examples for AST processors) might imply that we should put it off until someone needs it. >> Why? >> >> ---- >> >> The first problem is that the API for bytecode transformers is incomplete. You don't get, e.g., varnames, so you can't even write a transformer that looks up or modifies locals. But that part is easy to fix, so I won't dwell on it. > > I agree that we can enhance the Python stdlib to ease manipulation of > bytecode, but I disagree that it's a requirement. It's ok to use an > external library (like byteplay) for that. You're either answering out of order, or you missed the point of this paragraph. Your API passes only the bytecode string, consts, lnotab, and global names. There is no way to write a bytecode processor that looks up locals with that interface, because you didn't pass the locals. As I said, that's easy to fix--we could pass all 12 or so necessary parameters, or wrap them up with the 5 or so unnecessary ones in a code object one step earlier in the process and pass that, etc. But that just gets us to the second problem, which is harder to fix. >> The second problem is that bytecode is just painful to work with. The peephole optimizer deals with this by just punting and returning the original code whenever it sees anything remotely complicated (which I don't think we want to encourage for all bytecode transformers), and it's _still_ pretty hairy code. And it's the kind of code that can easily harbor hard-to-spot and harder-to-debug bugs (line numbers occasionally off on tracebacks, segfaults on about 1/255 programs that do something uncommon, that kind of fun stuff). > > Sorry, I don't understand. Are you writing that the CPython peephole > optimizer produces invalid code? No. As I said, the peephole optimizer deals with this by punting and returning the original code whenever things get remotely complicated (e.g., long functions). I don't think we want third-party optimizers to similarly punt on anything complicated. So, they _will_ have bugs like this, unless there's some way of making it tractable for them. >> The compiler is already doing this work. There's no reason every bytecode processor should have to repeat all of it. I'm not so worried about performance here--technically, fixup is worst-case quadratic, but practically I doubt we spend much time on it--but simplicity. Why should everyone have to repeat dozens of lines of complicated code to write a simple 10-line transformer? > > Hum, are you talking about the API proposed in the PEP 511? I understand that you are saying the API only takes a whole code > object as input and produces a code object as output. An optimizer > usually needs a different structure to be able to modify the code. If > you have multiple bytecode transformers, you have to repeat these > "disassemble" and "assemble" steps, right? Well, the API currently doesn't even take a whole code object--but other than that, yes, that's exactly what I'm saying. > I don't think that we will have plently bytecode optimizers in the > wild. Even if two or three major bytecode optimizers become popular, > are you sure that we will want to combine them? I expect that a single > optimizer implements *all* optimizations. I don't see the point of > running multiple optimizers to implement multiple optimizations steps. > For example, my fatoptimizer AST optimizer implements multiple steps, > but *internally*. It is only called once on the AST. You seem to be focusing on the performance issue here, while I think that's a minor consideration at best. The problem (probably) isn't that it's too costly for the CPU to run the disassembly and assembly code multiple times, but that it's too costly for the Python community to write and maintain disassembly and assembly code multiple times. I suppose if you're expecting that everyone experimenting with a new optimizer will do so by writing it as a patch to one of the three major optimizer projects rather than a standalone project, that might still solve much of the real problem. But do you really think (a) that's the way things should go, and (b) that's the way things probably _will_ go? > I don't think that performance of importing modules really matters. My > PEP 511 is mostly written for compilation ahead of time, to support > complex and expensive optimizers. > > The real problem is to run a script: "python script.py" always has to > execute all code transformers. For scripts, I hesitate to simply > disable expensive optimizers, or maybe even disable all optimizers. > For example, a script can run less than 50 ms, is it worth to spend 10 > ms to optimize it to a get speedup of 1 ms? (no) I didn't think the performance of optimizers would matter _at all_, which is why I was most focused on the simplicity and DRY argument. If it really is an issue for startup time for small scripts, then repeating the same disassemble/assemble steps multiple times may make that issue worse, but I don't think we'd know that without profiling. So I still think this is probably the least important of my arguments for the idea. > The problem of using a specific format for bytecode rather than a code > object is that we will have to maintain it. We're already maintaining such a format in the stdlib today, and have been for years: the dis module. The problem is that it's an immutable format, and has only a disassembler but no assembler, and no way to pass code into the assembler we already have under the covers. My proposal is essentially just to fix that. Everything else is just gravy. For example, for very little extra work, we could make the mutable dis module replace _all_ need for byteplay, so I think we should do that very little extra work--but if we don't, the core idea is still useful on its own. > I'm not sure that all > bytecode optimizer want the same internal structures. For some kind of > optimizations, a sequential list of instructions is enough. For some > other optimizations, you need to split blocks of code to have a > representation of the exacty "control flow". Even then, it's far easier to build a tree of blocks from the dis structure than from the bytecode-and-lnotab-and-arrays structure--and even more so to do the assembly and fixups if you're just assembling the dis structure rather than the bytecode-and-etc. structure. (And, if optimizer performance is really an issue--again, I doubt it is, but just in case--it should also be much less expensive to chain your third-party optimizer and the peephole optimizer if you don't need to do the fixups in between.) > Last point, the PEP 511 has to take in account the existing peephole > optimizer implement in C. If you really want to use a different > structure, you will have to reimplement the peephole optimizer with > your new API. Yes. But that's more part of the motivation than a problem. From my exploratory steps, rewriting the peephole optimizer around the dis API is less work than almost any nontrivial change to the peephole optimizer as-is (far less work than changing it to understand wordcode, which I was playing with last weekend). Also, I'm curious if there would be any measurable performance benefit from the peephole optimizer not punting on large functions. No huge benefits here, but as long as someone's willing to do the work--and I am willing, unless Serhiy (who definitely knows CPython better than me, and seems just as motivated here) wants to do it first. > Since my target is AST, I'm not really interested by > that :-) And that brings us back to the alternative idea of just not dealing with bytecode in PEP 511. >> I played around with a few possibilities from fixed-width bytecode and uncompressed lnotab to a public version of the internal assembler structs, but I think the best one is a flat sequence of instructions, with pseudo-instructions for labels and line numbers, and jump targets just references to those label instructions. > > Could you try to reimplement the whole peephole optimizer to see if it > benefit of your design? Yes, if I work on this over the weekend, reimplementing the peephole optimizer will definitely be part of the PoC. After all, we don't have a large code base of other bytecode optimizers used in the wild to play with yet. :) > I played with bytecode in the past. At the send, I started to > implement optimizations which can be implemented simpler at AST level. > > Why do you prefer bytecode over AST? I don't prefer bytecode over AST. As I've said multiple times, including in the same email you're responding to, I think 90% of the optimizations you could do in bytecode could instead be done, usually more simply (even with something like byteplay), at the AST level. The only question is whether that last 10% matters. > Your example of converting > globals to constants became trivial to implement using my new > ast.Constant node. Imagine you hadn't thought of that optimization (not likely, since it's the first example everyone uses, but you know what I mean), and we'd released Python 3.6 with PEP 511 and without ast.Constant, and now I thought of it. How would I implement it in a third-party optimizer? I can't patch the compiler to add a new AST node. (Nobody's going to install my optimizer if the steps are "get the CPython source, apply this patch, rebuild with the same config as your existing compiler, ...") And there's no way to implement it with the AST with the existing nodes. So I'd have to write it as a bytecode transformer. Of course once I released that to PyPI and everyone loved it and we had benchmarks to back it up, then I could come back and say, "We should add an ast.Constant node in Python 3.7, because that would let me rewrite my global optimizer as a simpler AST processor." And a few years later Python 3.6 would stop being relevant enough to keep maintaining the bytecode version. Meanwhile, here's a completely different example: the Python compiler emits a lot of jumps to jump instructions. The peephole optimizer takes care of some, but not all, of these. Maybe with an ast.Goto node, or extra fields in the various for/try/etc. nodes that don't get generated by parse but can be added by an AST processor, etc., they could all be handled at the AST level--but I'm not sure about that, and I think it would be significantly more complicated than doing it at the bytecode level. So, it's still one of the 10% cases. Again, I think it's possible that the Python community could live without that 10%. If the globals idea had to wait a year and a half for a new AST node to be added, and the double-jump fixes in the existing peephole optimizer were all we ever had, I think CPython would do just fine. (In fact, I think the whole mania for micro-optimizing CPython is mostly misdirected energy in the first place... But I could be wrong about that, and if other people think it's useful, I think it's fun.:)) Which is why I think maybe removing bytecode processors from PEP 511 is a real alternative. It's only if we think we _do_ need bytecode processors that we need a way to write them. >> * It needs a C API, and probably a C implementation. > > I don't like extending the Python C API, it is already very large, we > have too many functions :-p > > A C API is more expensive to maintain than a Python API. Well, if PEP 511 only supports optimizers written in Python (or written in C, but using the Python API), then the only real need for the C API is the peephole optimizer. So (following my own advice about not overgeneralizing a special case, which I should have thought of before...), I think you're right here. Leave the dis API as pure Python, and keep anything that has to be done in C private to the compiler (maybe shared with the peephole optimizer if necessary, but not exposed as a public and documented C API). So, scratch that line--which makes the proposed change a lot smaller. > I would prefer to continue to play with an external module (hosted on > PyPI) to not pay the price of maintenance! The dis module is already in the stdlib. The byteplay module is a 8 years old in its current form, even older in its original form, and it's become abundantly clear that the only problem with maintaining byteplay is syncing up with changes in dis and the rest of CPython. (The dis module, on the other hand, actually _does_ change, far more often and more substantially than byteplay, and it fits into the CPython release schedule pretty nicely.) > By the way, what is the final goal? Do you plan to implement a new > ultra optimized bytecode optimizer? If yes, do you plan to integrate > it into CPython? If no, I don't think that we have to pay the price of > maintenance for such "toy" project. I don't think it's a "toy" project any more than the existing dis module is. At any rate, the end goals are, in rapidly-descending order of importance: 1. Make it feasible for people to write bytecode transformers. 2. Encourage experimentation by allowing people to move mostly the same code between decorators and PEP 511 optimizers and even built in to CPython. 3. Make the compiler a little simpler, and the peephole optimizer a lot simpler. 4. Possibly improve the performance of the optimization step, and possibly improve the performance benefits of the peephole optimizer (by having it not punt on complicated functions). 5. Enable experimentation on low-level parts of CPython (e.g., the only hard part of the wordcode patch in 3.5 is the peephole optimizer--if that went away, we could have more experiments in that vein). 6. Remove the maintenance headaches for byteplay and modules that depend on it. As I said, the importance decreases rapidly, and if we decide that bytecode optimizers aren't that important, and leave them out of PEP 511, that pretty much knocks out the top 2, which I think is more than enough to kill my proposal. >> Anyway, I realize this API is still a little vague, (...) > > It doesn't fit requirements to put something into the Python stdlib. > Usually, we experiment stuff on PyPI, wait until it becomes mature, > and then propose to integrate it. > > It looks like you are talking about creating a new API and directly > put it into the stdlib, right? I'm talking about making the smallest possible changes to the _existing_ dis API that's _already_ in the stdlib, and basing those changes on a very mature third-party library that's long since defeated all of its competition. Meanwhile, "just put it on PyPI" isn't an option. A third-party module could monkeypatch dis, but it can't change the order in which the builtin compiler does the assembly, optimizer, and fixup steps, or change the interface to PEP 511 processors and the peephole optimizer. As I said, there _is_ the alternative of making a smaller change (like handing complete code objects to processors and then recommending byteplay), which solves not all but some of the problems for less work, and there's also the alternative of just dropping the bytecode processor API, which means most of the problems don't need to be solved, for no work. >> We could just pass code objects (or all the separate pieces, instead of some of them), and then the docs could suggest using byteplay for non-trivial bytecode transformers, and then everyone will just end up using byteplay. > > Again, you need to elaborate your rationale. What are your use cases? > Which kind of optimizations do you want to implement? Any kind of optimizations that need to work on bytecode. My rationale for that is that PEP 511 has a rationale for such optimizations. If that rationale isn't good enough, PEP 511 should only handle AST processors, in which case most of the problem I'm trying to solve won't exist in the first place. >> So, what's wrong with that? The biggest problem is that, after each new Python release, anyone using a bytecode transformer will have to wait until byteplay is updated before they can update Python. > > Why not contributing to byteplay to support the next CPython release? I've done that before. So have other people. The problem is that someone has to notice that something about CPython bytecode and/or the dis module has changed, and figure out how byteplay needs to change to handle that, and then figure out how to make the change backward-compatibly. If it were integrated in the dis module, then every change that anyone makes would automatically be handled, because you already have to update the dis module (which is almost always very simple--and would continue to be so). And it would be the person who best understands the change making the update, instead of someone who just discovered that 3.7 makes his code raise a SystemError if run through a bytecode processor that he's been using for the last two years. Essentially, integrating this into the existing dis module means we track changes for free, and it's hard to beat free. > I don't understand your problem here. I think that's because you've never tried to write a non-trivial bytecode processor. >> Or we could just remove bytecode transformers from PEP 511. PEP 511 still seems worth doing to me, even if it only has AST transformers, especially since all or nearly all of the examples anyone's come up with for it are implementable (and easier to implement) at the AST level. > > I want to plug the existing peephole optimizer into my PEP 511 since > it is an obvious *code* transformer. Even if changes are minor, some > users want to disable it because it really changes the code. It would > help code coverage for example. If it really is a special case that needs to be handled, don't over generalize it to cover other cases that you're never going to need. I think we might want those other cases. If so, we need to make them writable. If not, we shouldn't enable them half-way. From abarnert at yahoo.com Fri Feb 12 14:54:14 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 12 Feb 2016 11:54:14 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: Message-ID: On Feb 12, 2016, at 04:09, Edward Minnix wrote: > > Hello, > > I am a new programmer and have been using Python for a few months. > > I was experimenting the other day with unpacking (lists, tuples, etc.) And I realized something: > > when you type: > > >>> a, b, *rest = count() > > The interpreter gets caught in an infinite loop which I could not kill without terminating my REPL. > > Would there be a way to add generators to the unpackables, even if it was only in the front? I think what you're _really_ suggesting here is that there should be a way to unpack iterators lazily, presumably by unpacking them into iterators. And if you could come up with a good intuitive way to make that work, that would be a great suggestion. But I don't know of such a way. Here's some background: You already_can_ unpack generators and other iterators: >>> c = (i*2 for i in range(5)) >>> a, b, *rest = c >>> rest [4. 6, 8] It works something like this: >>> _c = iter(c) >>> a = next(_c) >>> b = next(_c) >>> rest = list(_c) The problem is that when you give it an infinite iterator, like count(), that last step is attempting to create an infinite list, which takes infinite time (well, it'll raise a MemoryError at some point--but that could take a long time, especially if it drives your system into swap hell along the way). Even for non-infinite iterators, this is sometimes not what you want, because it means you lose all laziness. For example: >>> f = sock.makefile('r') >>> a, b, *rest = f In these last two cases, what you actually want is something like this: >>> _c = iter(c) >>> a = next(_c) >>> b = next(_c) >>> rest = _c That would finish immediately, with rest as a lazy iterator instead of a list. But is that what you want in the first case? Would it still be what you wanted in "a, b, *rest = [1, 2, 3, 4]"? Many novices would be confused if they wrote that, and then printed out rest and got "" instead of "[3, 4]". Maybe that would be acceptable; people would just have to learn to use list(rest) when that's what they want (the same way they do with map, etc.). But it's probably too late for that, for backward compatibility reasons. Also, consider "a, *rest, b = [1, 2, 3, 4]". I don't think there's any way that could be done lazily. Meanwhile, you can always write the expanded version out explicitly. (And you can leave off the first line when you know c is already an iterator.) Or you can use itertools.islice to make it more compact: >>> a, b = itertools.islice(c, 2) >>> rest = c If you can come up with intuitive syntax, or an automated rule, or something else, to distinguish the case where you want a list from the case where you want a lazy iterator (and without breaking backward compatibility or over-complicating the language), then we could avoid that. If not, learn to love itertools. :) (That's actually good advise anyway--anyone writing the kind of code that needs infinite lists will get a lot out of digging deep into what itertools can do for you.) From chris.barker at noaa.gov Fri Feb 12 15:12:21 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 12 Feb 2016 12:12:21 -0800 Subject: [Python-ideas] Dict with inverse form In-Reply-To: References: <935127780.2464284.1455250067522.JavaMail.yahoo@mail.yahoo.com> Message-ID: the other restriction is that both keys and values (or is it values and keys?) both need to be immutable. but yes, I've needed this a lot, and I think that fact that both sets of keys need to be unique isn't really a restriction -- if you need to look it up either way, then it's probably naturally a one-to-one mapping anyway. I've needed this a few times, and have always kludged something together, rather than make a proper, robust tested class. So I look forward to seeing what you come up with. -CHB On Fri, Feb 12, 2016 at 5:27 AM, Jo?o Bernardo wrote: > > On Fri, Feb 12, 2016 at 2:07 AM, Andrew Barnert > wrote: >> >> For example, when I've built something like this, sometimes I want errors >> on duplicate values, sometimes just pick one arbitrarily, sometimes I want >> the inverse dict to store sets of values, and sometimes I even want _both_ >> to store sets of values. Do you want to do just one of those, or two or >> more? Controlled by flags? Or different attribute names? Or different >> classes? >> > > My use case is quite simple. I believe I always used this pattern on a > one-to-one mapping, so allowing repeated values to overwrite or throwing an > error is not exactly important to me, but it may be to someone else. > > Someone suggested a MultiDict sort of thing, but I don't think I need that > amount of complexity here. > > I will write a concrete class and post on PyPI then. > > Jo?o Bernardo > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Feb 12 15:19:05 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 12 Feb 2016 12:19:05 -0800 Subject: [Python-ideas] atexit.register_w_signals() In-Reply-To: References: <56BE2036.8010503@stoneleaf.us> Message-ID: On Feb 12, 2016, at 10:36, Giampaolo Rodola' wrote: > So, one problem with atexit.register() is that it won't run the registered function in case the process is killed by a signal. I think atexit is meant to be similar to the C feature of the same name. C atexit handlers are explicitly called only on "normal termination"--that is, returning from main(), or calling exit(). The reason C has other termination mechanisms like _exit() is to give you a way to skip the atexit handlers. And POSIX defines signal handling in terms of _exit. All of the signals whose default action is T or A terminate the process "with all the consequences of _exit()"--meaning, specifically, that atexit handlers don't get called. Maybe for some reason that design doesn't work as well for Python as it does for C. But if so, (a) I think you need to explain why, and (b) I think the replacement should look a lot less like the C mechanism, so it doesn't mislead people. But really, Python already has a higher-level wrapper. Put a try/except, try/finally, with Exit Stack, plain with, etc. around your main function. That way, you can decide whether to handle only normal exits, normal exits plus Exception, normal edits plus BaseException, etc. And you can transform signals into exceptions (the way Python already does for SIGINT, and the way the example in the docs does for SIGALRM). > import atexit, os, signal > @atexit.register > def cleanup(): > print("on exit") # XXX this never gets printed > os.kill(os.getpid(), signal.SIGTERM) > > The correct way to do that is to use signal.signal(). The problem with that though is that in case a third-party module has > already registered a function for that signal, the new function will overwrite the old one: There's a solution for that: chaining signal handlers. When you call signal.signal to register your handler, you get the old handler back as a result. You can store that and have your handler call the old one, unless it's one of the special ones. Or course that doesn't help when one library wants to ignore a signal with SIG_IGN and another wants to process it. Or when a library wants to dynamically register and unregister handlers after startup. It doesn't even work in the simple case unless everyone plays nicely--one library that doesn't do chaining breaks all the libraries that do. Maybe a library to wrap up all those complexities so multiple libraries could cooperate better in signal handling would be useful? > Also, we would still have to use atexit.register() so that the new function is called also on "clean" interpreter exit Or, more simply, do the same thing people do in C: install a signal handler that calls exit() instead of doing explicit cleanup. Then you only have to register your cleanup in one place (whether that's atexit handlers, or something more Pythonic like a with or finally that gets triggered as the SystemExit passes along the exception handler chain). From abarnert at yahoo.com Fri Feb 12 15:27:08 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 12 Feb 2016 12:27:08 -0800 Subject: [Python-ideas] Dict with inverse form In-Reply-To: References: <935127780.2464284.1455250067522.JavaMail.yahoo@mail.yahoo.com> Message-ID: <4BA45D4A-FAFB-4545-8570-F039039679AD@yahoo.com> On Feb 12, 2016, at 05:27, Jo?o Bernardo wrote: > > >> On Fri, Feb 12, 2016 at 2:07 AM, Andrew Barnert wrote: >> For example, when I've built something like this, sometimes I want errors on duplicate values, sometimes just pick one arbitrarily, sometimes I want the inverse dict to store sets of values, and sometimes I even want _both_ to store sets of values. Do you want to do just one of those, or two or more? Controlled by flags? Or different attribute names? Or different classes? > > My use case is quite simple. I believe I always used this pattern on a one-to-one mapping, so allowing repeated values to overwrite or throwing an error is not exactly important to me, but it may be to someone else. > > Someone suggested a MultiDict sort of thing, but I don't think I need that amount of complexity here. > > I will write a concrete class and post on PyPI then. I look forward to it. Next time I need some variation of this, even if it *isn't* the same variation you end up building, the fact that there's a potential de facto standard for what to call the ".inv" or whatever still helps me, right? Also, even if nobody ever agrees to put this in the stdlib, the collections module linked to some outside recipes/modules like OrderedSet long before Nick and the other packaging guys started pushing the idea that it was a good idea for the stdlib in general. I think if this gets sufficient uptake, it deserves such a link. As you say, it's something many people end up building for themselves. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Fri Feb 12 15:36:20 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 12 Feb 2016 15:36:20 -0500 Subject: [Python-ideas] Incorporating something like byteplay into the stdlib In-Reply-To: <1465993608.2439410.1455249525441.JavaMail.yahoo@mail.yahoo.com> References: <1465993608.2439410.1455249525441.JavaMail.yahoo.ref@mail.yahoo.com> <1465993608.2439410.1455249525441.JavaMail.yahoo@mail.yahoo.com> Message-ID: <56BE4244.6060504@gmail.com> On 2016-02-11 10:58 PM, Andrew Barnert via Python-ideas wrote: > tl;dr: We should turn dis.Bytecode into a builtin mutable structure similar to byteplay.Code, to make PEP 511 bytecode transformers implementable. Big -1 on the idea, sorry. CPython's bytecode is the implementation detail of CPython. PyPy has some opcodes that CPython doesn't have, for example. Who knows, maybe in CPython 4.0 we won't have code objects at all :) Adding something to the standard library means that it will be supported for years to come. It means that the code is safe to use. Which, in turn, guarantees that there will be plenty of code that depends on this new functionality. At first some of that code will be bytecode optimizers, later someone implements LINQ-like extension, and in no time we lose our freedom to work with opcodes. If this "new functionality" is something that depends on CPython's internals, it will only fracture the ecosystem. PyPy, or Pyston, or IronPython developers will either have to support byteplay-like stuff (which might be impossible), or explain their users why some libraries don't work on their platform. Yury From chris.barker at noaa.gov Fri Feb 12 15:36:38 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 12 Feb 2016 12:36:38 -0800 Subject: [Python-ideas] Dict with inverse form In-Reply-To: <4BA45D4A-FAFB-4545-8570-F039039679AD@yahoo.com> References: <935127780.2464284.1455250067522.JavaMail.yahoo@mail.yahoo.com> <4BA45D4A-FAFB-4545-8570-F039039679AD@yahoo.com> Message-ID: On Fri, Feb 12, 2016 at 12:27 PM, Andrew Barnert via Python-ideas < python-ideas at python.org> wrote: > I look forward to it. Next time I need some variation of this, even if it > *isn't* the same variation you end up building, the fact that there's a > potential de facto standard for what to call the ".inv" or whatever still > helps me, right? > yeah, though I'm not sure I like that name... (can't think of a better one right now, though). But what I would like is for the "inverse" to be available as an object itself, so: my_double_dict = DoubleDict( ( (1:'a'), (2:'b'), (3:'c) ) ) my_inverse = my_double_dict.inv my_double_dict[1] == 'a' my_inverse['a'] == 1 i.e you now have two objects, which are essentially the same object, but with inverse referencing semantics. Note how I cleverly introduced a name for this object -- I think it more as doubling than inversing, or really a one-to-one mapping, but haven't thought of a clever name for from that... And for my part, re-using a key on either side should simply write-over the existing item, just like a dict. -CHB > Also, even if nobody ever agrees to put this in the stdlib, the > collections module linked to some outside recipes/modules like OrderedSet > long before Nick and the other packaging guys started pushing the idea that > it was a good idea for the stdlib in general. I think if this gets > sufficient uptake, it deserves such a link. As you say, it's something many > people end up building for themselves. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ian.g.kelly at gmail.com Fri Feb 12 15:50:40 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Fri, 12 Feb 2016 13:50:40 -0700 Subject: [Python-ideas] Dict with inverse form In-Reply-To: References: <935127780.2464284.1455250067522.JavaMail.yahoo@mail.yahoo.com> <4BA45D4A-FAFB-4545-8570-F039039679AD@yahoo.com> Message-ID: On Fri, Feb 12, 2016 at 1:36 PM, Chris Barker wrote: > On Fri, Feb 12, 2016 at 12:27 PM, Andrew Barnert via Python-ideas > wrote: >> >> I look forward to it. Next time I need some variation of this, even if it >> *isn't* the same variation you end up building, the fact that there's a >> potential de facto standard for what to call the ".inv" or whatever still >> helps me, right? > > > yeah, though I'm not sure I like that name... (can't think of a better one > right now, though). > > But what I would like is for the "inverse" to be available as an object > itself, so: > > my_double_dict = DoubleDict( ( (1:'a'), (2:'b'), (3:'c) ) ) > my_inverse = my_double_dict.inv > > my_double_dict[1] == 'a' > my_inverse['a'] == 1 > > > i.e you now have two objects, which are essentially the same object, but > with inverse referencing semantics. I have some unpublished (lightly tested) code that basically does this (inspired by Guava's BiMap). class BiDict(dict): def __init__(self, *args, **kwargs): self._inverse = _BiDictInverse(self) self._super_inverse = super(BiDict, self._inverse) self.update(*args, **kwargs) @property def inverse(self): return self._inverse def __repr__(self): return 'BiDict(%s)' % super().__repr__() def __setitem__(self, key, value): if value in self._inverse and self._inverse[value] != key: raise ValueError( '%r already bound to %r' % (value, self._inverse[value])) if key in self: self._super_inverse.__delitem__(self[key]) super().__setitem__(key, value) self._super_inverse.__setitem__(value, key) def __delitem__(self, key): self._super_inverse.__delitem__(self[key]) super().__delitem__(key) def clear(self): super().clear() self._super_inverse.clear() def pop(self, key, *args): key_was_present = key in self value = super().pop(key, *args) if key_was_present: self._super_inverse.__delitem__(value) return value def popitem(self): (key, value) = super().popitem() self._super_inverse.__delitem__(value) return (key, value) def setdefault(self, key, *args): key_was_present = key in self value = super().setdefault(key, *args) if not key_was_present: self._super_inverse.__setitem__(value, key) return value def update(self, *args, **kwargs): if len(args) > 1: raise TypeError( 'update expected at most 1 arguments, got %d' % len(args)) if args and hasattr(args[0], 'keys'): for key in args[0]: self[key] = args[0][key] elif args: for key, value in args[0]: self[key] = value for key in kwargs: self[key] = kwargs[key] class _BiDictInverse(BiDict): def __init__(self, forward_dict): self._inverse = forward_dict self._super_inverse = super(BiDict, self._inverse) From twangist at gmail.com Fri Feb 12 15:53:38 2016 From: twangist at gmail.com (Brian O'Neill) Date: Fri, 12 Feb 2016 15:53:38 -0500 Subject: [Python-ideas] Dict with inverse form Message-ID: <5C0CE62E-7BB4-414B-B617-1C4E9AAA9F96@gmail.com> FYI, there is at least one library that provides bidirectional dicts: bidict. Github page: https://github.com/jab/bidict -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Fri Feb 12 15:59:40 2016 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 12 Feb 2016 20:59:40 +0000 Subject: [Python-ideas] Making (?x) less scary In-Reply-To: <2D44226C-B26E-445E-91AB-C068EE041554@gmail.com> References: <2D44226C-B26E-445E-91AB-C068EE041554@gmail.com> Message-ID: <56BE47BC.7050902@mrabarnett.plus.com> On 2016-02-12 16:15, Ryan Gonzalez wrote: > I was reading re's docs and came across this: > > > Note that the (?x) flag changes how the expression is parsed. It should > be used first in the expression string, or after one or more whitespace > characters. If there are non-whitespace characters before the flag, the > results are undefined. > > > Is there a particular reason for this being undefined? It seems kind of > un-Pythonic to me and more like C/C++. Would it be possible to instead > throw an exception, e.g. ValueError? > In Python 3.5 I get this: >>> re.compile(" (?x)a b", re.DEBUG) LITERAL 32 LITERAL 97 LITERAL 98 LITERAL 97 LITERAL 98 re.compile(' (?x)a b', re.VERBOSE|re.DEBUG) So, apparently, if the 'flags' argument doesn't specify VERBOSE, it parses the pattern, and if it then finds that it's now marked as VERBOSE because of a "(?m)" in the pattern, it parses it again. Unfortunately, it generates the debug output while it parses, so you get output from both passes. From ned at nedbatchelder.com Fri Feb 12 16:12:10 2016 From: ned at nedbatchelder.com (Ned Batchelder) Date: Fri, 12 Feb 2016 16:12:10 -0500 Subject: [Python-ideas] Making (?x) less scary In-Reply-To: <2D44226C-B26E-445E-91AB-C068EE041554@gmail.com> References: <2D44226C-B26E-445E-91AB-C068EE041554@gmail.com> Message-ID: <56BE4AAA.8000201@nedbatchelder.com> On 2/12/16 11:15 AM, Ryan Gonzalez wrote: > I was reading re's docs and came across this: > > > Note that the (?x) flag changes how the expression is parsed. It > should be used first in the expression string, or after one or more > whitespace characters. If there are non-whitespace characters before > the flag, the results are undefined. > > > Is there a particular reason for this being undefined? It seems kind > of un-Pythonic to me and more like C/C++. Would it be possible to > instead throw an exception, e.g. ValueError? Is there a reason to deal with this case at all? Throwing an exception means having to add code to detect the condition and raise the exception. I'd rather see the docs changed to say, "It must be used first in the string, or after one or more whitespace characters." No mention of what happens otherwise is needed. --Ned. From abarnert at yahoo.com Fri Feb 12 16:13:15 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 12 Feb 2016 13:13:15 -0800 Subject: [Python-ideas] Incorporating something like byteplay into the stdlib In-Reply-To: <56BE4244.6060504@gmail.com> References: <1465993608.2439410.1455249525441.JavaMail.yahoo.ref@mail.yahoo.com> <1465993608.2439410.1455249525441.JavaMail.yahoo@mail.yahoo.com> <56BE4244.6060504@gmail.com> Message-ID: <34117C96-D1E9-4A73-B3EB-B670C152C2BE@yahoo.com> On Feb 12, 2016, at 12:36, Yury Selivanov wrote: > >> On 2016-02-11 10:58 PM, Andrew Barnert via Python-ideas wrote: >> tl;dr: We should turn dis.Bytecode into a builtin mutable structure similar to byteplay.Code, to make PEP 511 bytecode transformers implementable. > > Big -1 on the idea, sorry. > > CPython's bytecode is the implementation detail of CPython. PyPy has some opcodes that CPython doesn't have, for example. Who knows, maybe in CPython 4.0 we won't have code objects at all :) > > Adding something to the standard library means that it will be supported for years to come. It means that the code is safe to use. Which, in turn, guarantees that there will be plenty of code that depends on this new functionality. At first some of that code will be bytecode optimizers, later someone implements LINQ-like extension, and in no time we lose our freedom to work with opcodes. > > If this "new functionality" is something that depends on CPython's internals, it will only fracture the ecosystem. PyPy, or Pyston, or IronPython developers will either have to support byteplay-like stuff (which might be impossible), or explain their users why some libraries don't work on their platform. This sounds like potentially a argument against adding bytecode processors in PEP 511.[^1] But if PEP 511 *does* add bytecode processors, I don't see how my proposal makes things any worse. Having dis (and inspect, and types.CodeType, and so on) be part of the stdlib makes it easier, not harder, to change CPython without breaking code that may need to introspect it for some reason. In the same way, having a mutable dis would make it easier, not harder, to change CPython without breaking bytecode processors. [^1]: Then again, it's just as good an argument against import hooks, and exposing the __code__ member on function objects so decorators can change it, and so on, and years with those features hasn't created a catastrophe... From storchaka at gmail.com Fri Feb 12 16:39:45 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 12 Feb 2016 23:39:45 +0200 Subject: [Python-ideas] Making (?x) less scary In-Reply-To: <56BE4AAA.8000201@nedbatchelder.com> References: <2D44226C-B26E-445E-91AB-C068EE041554@gmail.com> <56BE4AAA.8000201@nedbatchelder.com> Message-ID: On 12.02.16 23:12, Ned Batchelder wrote: > On 2/12/16 11:15 AM, Ryan Gonzalez wrote: >> I was reading re's docs and came across this: >> >> Note that the (?x) flag changes how the expression is parsed. It >> should be used first in the expression string, or after one or more >> whitespace characters. If there are non-whitespace characters before >> the flag, the results are undefined. >> >> Is there a particular reason for this being undefined? It seems kind >> of un-Pythonic to me and more like C/C++. Would it be possible to >> instead throw an exception, e.g. ValueError? > > Is there a reason to deal with this case at all? Throwing an exception > means having to add code to detect the condition and raise the exception. http://bugs.python.org/issue22493 From yselivanov.ml at gmail.com Fri Feb 12 16:45:40 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 12 Feb 2016 16:45:40 -0500 Subject: [Python-ideas] Incorporating something like byteplay into the stdlib In-Reply-To: <34117C96-D1E9-4A73-B3EB-B670C152C2BE@yahoo.com> References: <1465993608.2439410.1455249525441.JavaMail.yahoo.ref@mail.yahoo.com> <1465993608.2439410.1455249525441.JavaMail.yahoo@mail.yahoo.com> <56BE4244.6060504@gmail.com> <34117C96-D1E9-4A73-B3EB-B670C152C2BE@yahoo.com> Message-ID: <56BE5284.1010204@gmail.com> On 2016-02-12 4:13 PM, Andrew Barnert wrote: > On Feb 12, 2016, at 12:36, Yury Selivanov wrote: >>> On 2016-02-11 10:58 PM, Andrew Barnert via Python-ideas wrote: >>> tl;dr: We should turn dis.Bytecode into a builtin mutable structure similar to byteplay.Code, to make PEP 511 bytecode transformers implementable. >> Big -1 on the idea, sorry. >> >> CPython's bytecode is the implementation detail of CPython. PyPy has some opcodes that CPython doesn't have, for example. Who knows, maybe in CPython 4.0 we won't have code objects at all :) >> >> Adding something to the standard library means that it will be supported for years to come. It means that the code is safe to use. Which, in turn, guarantees that there will be plenty of code that depends on this new functionality. At first some of that code will be bytecode optimizers, later someone implements LINQ-like extension, and in no time we lose our freedom to work with opcodes. >> >> If this "new functionality" is something that depends on CPython's internals, it will only fracture the ecosystem. PyPy, or Pyston, or IronPython developers will either have to support byteplay-like stuff (which might be impossible), or explain their users why some libraries don't work on their platform. > This sounds like potentially a argument against adding bytecode processors in PEP 511.[^1] But if PEP 511 *does* add bytecode processors, I don't see how my proposal makes things any worse. The main (and only?) motivation behind PEP 511 is the optimization of CPython. Maybe the new APIs will only be exposed at C level. > Having dis (and inspect, and types.CodeType, and so on) be part of the stdlib makes it easier, not harder, to change CPython without breaking code that may need to introspect it for some reason. You don't need mutability for introspection. > In the same way, having a mutable dis would make it easier, not harder, to change CPython without breaking bytecode processors. PEP 492 added a bunch of new opcodes. Serhiy is exploring an opportunity of adding few more LOAD_CONST_N opcodes. How would a mutable byteplay-code-like object in the dis module help that? Interacting with bytecode in Python is generally considered unsafe, and used mostly for the purposes of experimentation, for which a PyPI module is enough. FWIW I have lots of experience working with bytecodes. For example, I have code in production systems in which decorators patch functions' code objects: they guard 'yield' expressions in 'finally' blocks with some function calls. So far that code had only caused me maintenance pain. It's the only code that I have to upgrade each time a new version of Python is released. > [^1]: Then again, it's just as good an argument against import hooks, and exposing the __code__ member on function objects so decorators can change it, and so on, and years with those features hasn't created a catastrophe... Import hooks (and even AST/parse/compile) is a much more high-level API. I'm not sure we can compare them to byteplay. Yury From yselivanov.ml at gmail.com Fri Feb 12 16:57:23 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 12 Feb 2016 16:57:23 -0500 Subject: [Python-ideas] Incorporating something like byteplay into the stdlib In-Reply-To: <56BE5284.1010204@gmail.com> References: <1465993608.2439410.1455249525441.JavaMail.yahoo.ref@mail.yahoo.com> <1465993608.2439410.1455249525441.JavaMail.yahoo@mail.yahoo.com> <56BE4244.6060504@gmail.com> <34117C96-D1E9-4A73-B3EB-B670C152C2BE@yahoo.com> <56BE5284.1010204@gmail.com> Message-ID: <56BE5543.7000302@gmail.com> On 2016-02-12 4:45 PM, Yury Selivanov wrote: >> In the same way, having a mutable dis would make it easier, not >> harder, to change CPython without breaking bytecode processors. > > PEP 492 added a bunch of new opcodes. Serhiy is exploring an > opportunity of adding few more LOAD_CONST_N opcodes. How would a > mutable byteplay-code-like object in the dis module help that? The key point here is not the API of your mutable code-object-abstraction. The problem is that bytecode modifiers rely on: 1. existence of certain bytecodes; 2. knowing their precise behaviour and side-effects; 3. matching/patching/analyzing exact sequences of bytecodes. High-level abstractions won't help with the above. Say we want to remove a few opcodes in favour of adding a few new ones. If we do that, most of code optimizers will break. That's why our low-level peephole optimizer is private - we can update it ourselves when we need it. It's completely in our control. Also, AFAIK, FAT Python analyzes/transforms AST. I'm not sure how byteplay could help FAT Python specifically. Yury From wes.turner at gmail.com Fri Feb 12 17:20:06 2016 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 12 Feb 2016 16:20:06 -0600 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict In-Reply-To: References: <5620D33A.3050709@feete.org> <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> Message-ID: This seems to keep a consistent __init__ signature with OrderedDict (by .pop()-ing 'default_factory' from kwargs instead of specifying as a positionalkwarg): class OrderedDefaultDict(OrderedDict): def __init__(self, *a, **kw): default_factory = kw.pop('default_factory', self.__class__) OrderedDict.__init__(self, *a, **kw) self.default_factory = default_factory def __missing__(self, key): self[key] = value = self.default_factory() return value I've added a few tests (as well as to_json, and _repr_json_ https://gist.github.com/westurner/be22dba8110be099a35e/c1a3a7394e401d4742df0617900bde6ab2643300#file-ordereddefaultdict-py-L120-L122 (Without this fix, json.loads(output_json, object_pairs_hook=OrderedDefaultDict) doesn't seem to work). On Fri, Nov 13, 2015 at 11:17 PM, Andrew Barnert wrote: > On Nov 13, 2015, at 20:46, Wes Turner wrote: > > On Fri, Oct 16, 2015 at 9:08 PM, Andrew Barnert via Python-ideas < > python-ideas at python.org> wrote: > >> Actually, forget all that; it's even simpler. >> >> At least in recent 3.x, the only thing wrong with inheriting from both >> types, assuming you put OrderedDict first, is the __init__ signature. So: >> >> class OrderedDefaultDict(OrderedDict, defaultdict): >> def __init__(self, default_factory=None, *a, **kw): >> OrderedDict.__init__(self, *a, **kw) >> self.default_factory = default_factory >> >> More importantly, because __missing__ support is built into dict, despite >> the confusing docs for defaultdict, you don't really need defaultdict at >> all here: >> >> class OrderedDefaultDict(OrderedDict): >> def __init__(self, default_factory=None, *a, **kw): >> OrderedDict.__init__(self, *a, **kw) >> self.default_factory = default_factory >> def __missing__(self, key): >> self[key] = value = default_factory() >> return value >> >> And either of these should work with 2.5+ (according to >> https://docs.python.org/2/library/stdtypes.html#dict that's when >> dict.__missing__ was added). >> >> > Thanks! > > - [ ] Could/should maybe either of these make it into the standard library, > that would save a fair amount of copying. > > > You could say the same about everything in the recipes, every one-liner > like "def identity(x): return x", and so on. But most such things aren't > used that often, and there's a cost to putting them in the stdlib?more for > people to learn and remember, more code to be maintained, bigger downloads, > etc. So just saying "maybe it would save some copying" isn't an argument > for adding it to the stdlib. > > Adding OrderedDefaultDict as a docs recipe (as OrderedCounter already is) > might be worth doing. > Reorganizing the docs a bit to make it more obvious (by better > highlighting __missing__, and making it clear that it's a method of dict > rather than something special about defaultdict) seems even more likely to > be worth it. If someone has an idea of how it should read, just file a docs > bug and submit a patch and see if whoever's in charge of that area says it > needs to come back here. > > .. Great in combination w/ dict views: > https://docs.python.org/2/library/stdtypes.html#dictionary-view-objects > > > Of course, and that works out of the box. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Feb 12 17:27:56 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 12 Feb 2016 14:27:56 -0800 Subject: [Python-ideas] Incorporating something like byteplay into the stdlib In-Reply-To: <56BE5284.1010204@gmail.com> References: <1465993608.2439410.1455249525441.JavaMail.yahoo.ref@mail.yahoo.com> <1465993608.2439410.1455249525441.JavaMail.yahoo@mail.yahoo.com> <56BE4244.6060504@gmail.com> <34117C96-D1E9-4A73-B3EB-B670C152C2BE@yahoo.com> <56BE5284.1010204@gmail.com> Message-ID: On Feb 12, 2016, at 13:45, Yury Selivanov wrote: > >> On 2016-02-12 4:13 PM, Andrew Barnert wrote: >> On Feb 12, 2016, at 12:36, Yury Selivanov wrote: >>>> On 2016-02-11 10:58 PM, Andrew Barnert via Python-ideas wrote: >>>> tl;dr: We should turn dis.Bytecode into a builtin mutable structure similar to byteplay.Code, to make PEP 511 bytecode transformers implementable. >>> Big -1 on the idea, sorry. >>> >>> CPython's bytecode is the implementation detail of CPython. PyPy has some opcodes that CPython doesn't have, for example. Who knows, maybe in CPython 4.0 we won't have code objects at all :) >>> >>> Adding something to the standard library means that it will be supported for years to come. It means that the code is safe to use. Which, in turn, guarantees that there will be plenty of code that depends on this new functionality. At first some of that code will be bytecode optimizers, later someone implements LINQ-like extension, and in no time we lose our freedom to work with opcodes. >>> >>> If this "new functionality" is something that depends on CPython's internals, it will only fracture the ecosystem. PyPy, or Pyston, or IronPython developers will either have to support byteplay-like stuff (which might be impossible), or explain their users why some libraries don't work on their platform. >> This sounds like potentially a argument against adding bytecode processors in PEP 511.[^1] But if PEP 511 *does* add bytecode processors, I don't see how my proposal makes things any worse. > > The main (and only?) motivation behind PEP 511 is the optimization of CPython. Maybe the new APIs will only be exposed at C level. Have you read PEP 511? It exposes an API for adding bytecode processors, explicitly explains that the reason for this API is to allow people to write new bytecode optimizers in Python, and includes a toy example of a bytecode transformer. I'm not imagining some far-fetched idea that someone might suggest in the future, I'm responding to what's actually written in the PEP. >> Having dis (and inspect, and types.CodeType, and so on) be part of the stdlib makes it easier, not harder, to change CPython without breaking code that may need to introspect it for some reason. > > You don't need mutability for introspection. Of course. When you split an analogy in half and only reply to the first half of it like this, the half-analogy has no content. So what? >> In the same way, having a mutable dis would make it easier, not harder, to change CPython without breaking bytecode processors. > > PEP 492 added a bunch of new opcodes. Serhiy is exploring an opportunity of adding few more LOAD_CONST_N opcodes. How would a mutable byteplay-code-like object in the dis module help that? If someone had written a bytecode processor on top of the dis module, and wanted to update it to take advantage of LOAD_CONST_N, it would be easy to do so--even on a local copy of CPython patched with Serhiy's changes. If they'd instead written it on top of a third-party module, they'd have to wait for that module to be updated (probably after the next major version of Python comes out), or update it locally. Which one of those sounds easiest to you? > Interacting with bytecode in Python is generally considered unsafe, and used mostly for the purposes of experimentation, for which a PyPI module is enough. That's an argument against the PEP 511 API for adding bytecode processors--and, again, also possibly an argument against mutable function.__code__ and so on. But how is it an argument against my proposal? >> [^1]: Then again, it's just as good an argument against import hooks, and exposing the __code__ member on function objects so decorators can change it, and so on, and years with those features hasn't created a catastrophe... > > Import hooks (and even AST/parse/compile) is a much more high-level API. I'm not sure we can compare them to byteplay. You're responding selectively here. Your argument is that people shouldn't mess with bytecode. If we don't want people to mess with bytecode, we shouldn't expose bytecode to be messed with. But you can write a decorator that sets f.__code__ = types.CodeType(...) with a replaced bytecode string, and all of the details on how to do that are fully documented in the dis and inspect modules. Making it tedious and error-prone is not a good way to discourage something. Meanwhile, the "low-level" part of this already exists: the dis module lists all the opcodes, disassembles bytecode, represents that disassembled form, etc. From abarnert at yahoo.com Fri Feb 12 17:38:31 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 12 Feb 2016 14:38:31 -0800 Subject: [Python-ideas] Incorporating something like byteplay into the stdlib In-Reply-To: <56BE5543.7000302@gmail.com> References: <1465993608.2439410.1455249525441.JavaMail.yahoo.ref@mail.yahoo.com> <1465993608.2439410.1455249525441.JavaMail.yahoo@mail.yahoo.com> <56BE4244.6060504@gmail.com> <34117C96-D1E9-4A73-B3EB-B670C152C2BE@yahoo.com> <56BE5284.1010204@gmail.com> <56BE5543.7000302@gmail.com> Message-ID: On Feb 12, 2016, at 13:57, Yury Selivanov wrote: > > Also, AFAIK, FAT Python analyzes/transforms AST. I'm not sure how byteplay could help FAT Python specifically. So far, all three people who've responded have acted like I invented the idea that PEP 511 might include bytecode optimizers. Even Victor, who wrote the PEP. Am I going crazy here? I'm looking right at http://www.python.org/dev/peps/pep-0511/. It has a "Usage 4" section that has a rationale for why we should allow writing bytecode optimizers in Python, and an example of something that can't be done by an AST optimizer. It has a "code_transformer() method" section, showing the API designed for writing those optimizers. It has an "API to get/set code transformers" section that explains when and how those transformers get run. It has a "Bytecode transformer" section that gives a toy example. Assuming I'm not imagining all that, why are people demanding that I provide a rationale for why we should add bytecode optimizers, or telling me that adding bytecode optimizers isn't going to help FAT Python, etc.? My proposal is that if we add bytecode optimizers, we should make it possible to write them. I don't need to justify that "if". If nobody things we should add bytecode optimizers, not even the author of the PEP that suggests them, then the answer is simple: just remove them from the PEP, and then my proposal becomes void. From ethan at stoneleaf.us Fri Feb 12 17:55:55 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 12 Feb 2016 14:55:55 -0800 Subject: [Python-ideas] Incorporating something like byteplay into the stdlib In-Reply-To: References: <1465993608.2439410.1455249525441.JavaMail.yahoo.ref@mail.yahoo.com> <1465993608.2439410.1455249525441.JavaMail.yahoo@mail.yahoo.com> <56BE4244.6060504@gmail.com> <34117C96-D1E9-4A73-B3EB-B670C152C2BE@yahoo.com> <56BE5284.1010204@gmail.com> <56BE5543.7000302@gmail.com> Message-ID: <56BE62FB.5090308@stoneleaf.us> On 02/12/2016 02:38 PM, Andrew Barnert via Python-ideas wrote: > My proposal is that if we add bytecode optimizers, we should make it > possible to write them. +1 -- ~Ethan~ From yselivanov.ml at gmail.com Fri Feb 12 17:58:31 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 12 Feb 2016 17:58:31 -0500 Subject: [Python-ideas] Incorporating something like byteplay into the stdlib In-Reply-To: References: <1465993608.2439410.1455249525441.JavaMail.yahoo.ref@mail.yahoo.com> <1465993608.2439410.1455249525441.JavaMail.yahoo@mail.yahoo.com> <56BE4244.6060504@gmail.com> <34117C96-D1E9-4A73-B3EB-B670C152C2BE@yahoo.com> <56BE5284.1010204@gmail.com> Message-ID: <56BE6397.7090908@gmail.com> On 2016-02-12 5:27 PM, Andrew Barnert wrote: > On Feb 12, 2016, at 13:45, Yury Selivanov wrote: >>> On 2016-02-12 4:13 PM, Andrew Barnert wrote: >>> On Feb 12, 2016, at 12:36, Yury Selivanov wrote: >>>>> On 2016-02-11 10:58 PM, Andrew Barnert via Python-ideas wrote: >>>>> tl;dr: We should turn dis.Bytecode into a builtin mutable structure similar to byteplay.Code, to make PEP 511 bytecode transformers implementable. >>>> Big -1 on the idea, sorry. >>>> >>>> CPython's bytecode is the implementation detail of CPython. PyPy has some opcodes that CPython doesn't have, for example. Who knows, maybe in CPython 4.0 we won't have code objects at all :) >>>> >>>> Adding something to the standard library means that it will be supported for years to come. It means that the code is safe to use. Which, in turn, guarantees that there will be plenty of code that depends on this new functionality. At first some of that code will be bytecode optimizers, later someone implements LINQ-like extension, and in no time we lose our freedom to work with opcodes. >>>> >>>> If this "new functionality" is something that depends on CPython's internals, it will only fracture the ecosystem. PyPy, or Pyston, or IronPython developers will either have to support byteplay-like stuff (which might be impossible), or explain their users why some libraries don't work on their platform. >>> This sounds like potentially a argument against adding bytecode processors in PEP 511.[^1] But if PEP 511 *does* add bytecode processors, I don't see how my proposal makes things any worse. >> The main (and only?) motivation behind PEP 511 is the optimization of CPython. Maybe the new APIs will only be exposed at C level. > Have you read PEP 511? It exposes an API for adding bytecode processors, explicitly explains that the reason for this API is to allow people to write new bytecode optimizers in Python, and includes a toy example of a bytecode transformer. I'm not imagining some far-fetched idea that someone might suggest in the future, I'm responding to what's actually written in the PEP. I guess I read an earlier version which was focused only on AST transformations. Maybe PEP 511 should be focused just on just one thing. > >>> Having dis (and inspect, and types.CodeType, and so on) be part of the stdlib makes it easier, not harder, to change CPython without breaking code that may need to introspect it for some reason. >> You don't need mutability for introspection. > Of course. When you split an analogy in half and only reply to the first half of it like this, the half-analogy has no content. So what? Sorry, somehow I failed to read that paragraph in one piece. My bad. > >>> In the same way, having a mutable dis would make it easier, not harder, to change CPython without breaking bytecode processors. >> PEP 492 added a bunch of new opcodes. Serhiy is exploring an opportunity of adding few more LOAD_CONST_N opcodes. How would a mutable byteplay-code-like object in the dis module help that? > If someone had written a bytecode processor on top of the dis module, and wanted to update it to take advantage of LOAD_CONST_N, it would be easy to do so--even on a local copy of CPython patched with Serhiy's changes. If they'd instead written it on top of a third-party module, they'd have to wait for that module to be updated (probably after the next major version of Python comes out), or update it locally. Which one of those sounds easiest to you? My point (which is the *key* point) is that if we decide to have only LOAD_CONST_N opcodes and remove plain old LOAD_CONST -- all optimizers will break, no matter what library they use. That's just a sad reality of working on the bytecode level. For instance, PEP 492 split WITH_CLEANUP opcode into WITH_CLEANUP_START and WITH_CLEANUP_FINISH. *Any* bytecode manipulation code that expected to see WITH_CLEANUP after SETUP_WITH *was* broken. In short: I don't want to add more stuff to CPython that can make it harder for us to modify its low-level internals. >> Interacting with bytecode in Python is generally considered unsafe, and used mostly for the purposes of experimentation, for which a PyPI module is enough. > That's an argument against the PEP 511 API for adding bytecode processors--and, again, also possibly an argument against mutable function.__code__ and so on. But how is it an argument against my proposal? function.__code__ exists and mutable regardless of PEP 511 and byteplay :) Let's not add it to the mix. You're right, I guess this is a common argument for both PEP511's code_transformer and a byteplay in stdlib. > >>> [^1]: Then again, it's just as good an argument against import hooks, and exposing the __code__ member on function objects so decorators can change it, and so on, and years with those features hasn't created a catastrophe... >> Import hooks (and even AST/parse/compile) is a much more high-level API. I'm not sure we can compare them to byteplay. > You're responding selectively here. Your argument is that people shouldn't mess with bytecode. If we don't want people to mess with bytecode, we shouldn't expose bytecode to be messed with. But you can write a decorator that sets f.__code__ = types.CodeType(...) with a replaced bytecode string, and all of the details on how to do that are fully documented in the dis and inspect modules. Making it tedious and error-prone is not a good way to discourage something. > > Meanwhile, the "low-level" part of this already exists: the dis module lists all the opcodes, disassembles bytecode, represents that disassembled form, etc. Although function.__code__ is mutable, almost nobody actually mutates it. We have dis module primarily for introspection and research purposes, view it as a handy tool to see how CPython really works. I'm OK if PEP 511 adds some AST transformation hooks (because AST is a higher-level abstraction). Adding code-object transformation hooks and a library to mutate (or produce new) code objects seems very wrong to me. Yury From yselivanov.ml at gmail.com Fri Feb 12 17:58:37 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 12 Feb 2016 17:58:37 -0500 Subject: [Python-ideas] Incorporating something like byteplay into the stdlib In-Reply-To: References: <1465993608.2439410.1455249525441.JavaMail.yahoo.ref@mail.yahoo.com> <1465993608.2439410.1455249525441.JavaMail.yahoo@mail.yahoo.com> <56BE4244.6060504@gmail.com> <34117C96-D1E9-4A73-B3EB-B670C152C2BE@yahoo.com> <56BE5284.1010204@gmail.com> Message-ID: <56BE639D.1010303@gmail.com> On 2016-02-12 5:27 PM, Andrew Barnert wrote: > On Feb 12, 2016, at 13:45, Yury Selivanov wrote: >>> On 2016-02-12 4:13 PM, Andrew Barnert wrote: >>> On Feb 12, 2016, at 12:36, Yury Selivanov wrote: >>>>> On 2016-02-11 10:58 PM, Andrew Barnert via Python-ideas wrote: >>>>> tl;dr: We should turn dis.Bytecode into a builtin mutable structure similar to byteplay.Code, to make PEP 511 bytecode transformers implementable. >>>> Big -1 on the idea, sorry. >>>> >>>> CPython's bytecode is the implementation detail of CPython. PyPy has some opcodes that CPython doesn't have, for example. Who knows, maybe in CPython 4.0 we won't have code objects at all :) >>>> >>>> Adding something to the standard library means that it will be supported for years to come. It means that the code is safe to use. Which, in turn, guarantees that there will be plenty of code that depends on this new functionality. At first some of that code will be bytecode optimizers, later someone implements LINQ-like extension, and in no time we lose our freedom to work with opcodes. >>>> >>>> If this "new functionality" is something that depends on CPython's internals, it will only fracture the ecosystem. PyPy, or Pyston, or IronPython developers will either have to support byteplay-like stuff (which might be impossible), or explain their users why some libraries don't work on their platform. >>> This sounds like potentially a argument against adding bytecode processors in PEP 511.[^1] But if PEP 511 *does* add bytecode processors, I don't see how my proposal makes things any worse. >> The main (and only?) motivation behind PEP 511 is the optimization of CPython. Maybe the new APIs will only be exposed at C level. > Have you read PEP 511? It exposes an API for adding bytecode processors, explicitly explains that the reason for this API is to allow people to write new bytecode optimizers in Python, and includes a toy example of a bytecode transformer. I'm not imagining some far-fetched idea that someone might suggest in the future, I'm responding to what's actually written in the PEP. I guess I read an earlier version which was focused only on AST transformations. Maybe PEP 511 should be focused just on just one thing. > >>> Having dis (and inspect, and types.CodeType, and so on) be part of the stdlib makes it easier, not harder, to change CPython without breaking code that may need to introspect it for some reason. >> You don't need mutability for introspection. > Of course. When you split an analogy in half and only reply to the first half of it like this, the half-analogy has no content. So what? Sorry, somehow I failed to read that paragraph in one piece. My bad. > >>> In the same way, having a mutable dis would make it easier, not harder, to change CPython without breaking bytecode processors. >> PEP 492 added a bunch of new opcodes. Serhiy is exploring an opportunity of adding few more LOAD_CONST_N opcodes. How would a mutable byteplay-code-like object in the dis module help that? > If someone had written a bytecode processor on top of the dis module, and wanted to update it to take advantage of LOAD_CONST_N, it would be easy to do so--even on a local copy of CPython patched with Serhiy's changes. If they'd instead written it on top of a third-party module, they'd have to wait for that module to be updated (probably after the next major version of Python comes out), or update it locally. Which one of those sounds easiest to you? My point (which is the *key* point) is that if we decide to have only LOAD_CONST_N opcodes and remove plain old LOAD_CONST -- all optimizers will break, no matter what library they use. That's just a sad reality of working on the bytecode level. For instance, PEP 492 split WITH_CLEANUP opcode into WITH_CLEANUP_START and WITH_CLEANUP_FINISH. *Any* bytecode manipulation code that expected to see WITH_CLEANUP after SETUP_WITH *was* broken. In short: I don't want to add more stuff to CPython that can make it harder for us to modify its low-level internals. >> Interacting with bytecode in Python is generally considered unsafe, and used mostly for the purposes of experimentation, for which a PyPI module is enough. > That's an argument against the PEP 511 API for adding bytecode processors--and, again, also possibly an argument against mutable function.__code__ and so on. But how is it an argument against my proposal? function.__code__ exists and mutable regardless of PEP 511 and byteplay :) Let's not add it to the mix. You're right, I guess this is a common argument for both PEP511's code_transformer and a byteplay in stdlib. > >>> [^1]: Then again, it's just as good an argument against import hooks, and exposing the __code__ member on function objects so decorators can change it, and so on, and years with those features hasn't created a catastrophe... >> Import hooks (and even AST/parse/compile) is a much more high-level API. I'm not sure we can compare them to byteplay. > You're responding selectively here. Your argument is that people shouldn't mess with bytecode. If we don't want people to mess with bytecode, we shouldn't expose bytecode to be messed with. But you can write a decorator that sets f.__code__ = types.CodeType(...) with a replaced bytecode string, and all of the details on how to do that are fully documented in the dis and inspect modules. Making it tedious and error-prone is not a good way to discourage something. > > Meanwhile, the "low-level" part of this already exists: the dis module lists all the opcodes, disassembles bytecode, represents that disassembled form, etc. Although function.__code__ is mutable, almost nobody actually mutates it. We have dis module primarily for introspection and research purposes, view it as a handy tool to see how CPython really works. I'm OK if PEP 511 adds some AST transformation hooks (because AST is a higher-level abstraction). Adding code-object transformation hooks and a library to mutate (or produce new) code objects seems very wrong to me. Yury From python at lucidity.plus.com Fri Feb 12 18:01:11 2016 From: python at lucidity.plus.com (Erik) Date: Fri, 12 Feb 2016 23:01:11 +0000 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: Message-ID: <56BE6437.20305@lucidity.plus.com> On 12/02/16 19:54, Andrew Barnert via Python-ideas wrote: > Meanwhile, you can always write the expanded version out explicitly. (And you can leave off the first line when you know c is already an iterator.) Or you can use itertools.islice to make it more compact: > > >>> a, b = itertools.islice(c, 2) > >>> rest = c Why not just have an itertools.unpack() - a simple version without argument checking: def unpack(seq, num): it = iter(seq) yield from (i[1] for i in zip(range(num), it)) yield it foo, bar, rest = unpack([1, 2, 3, 4, 5, 6], 2) Because it's in itertools, the expectation is that it has something to do with iterators so the final return value always being an iterator regardless of the original sequence type is reasonable (and is perhaps the only justification for putting it in itertools in the first place ;) ). E. From Nikolaus at rath.org Fri Feb 12 18:02:54 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Fri, 12 Feb 2016 15:02:54 -0800 Subject: [Python-ideas] Dict with inverse form In-Reply-To: (Chris Barker's message of "Fri, 12 Feb 2016 12:36:38 -0800") References: <935127780.2464284.1455250067522.JavaMail.yahoo@mail.yahoo.com> <4BA45D4A-FAFB-4545-8570-F039039679AD@yahoo.com> Message-ID: <8737sx4jmp.fsf@thinkpad.rath.org> On Feb 12 2016, Chris Barker wrote: > On Fri, Feb 12, 2016 at 12:27 PM, Andrew Barnert via Python-ideas < > python-ideas-+ZN9ApsXKcEdnm+yROfE0A at public.gmane.org> wrote: > >> I look forward to it. Next time I need some variation of this, even if it >> *isn't* the same variation you end up building, the fact that there's a >> potential de facto standard for what to call the ".inv" or whatever still >> helps me, right? >> > > yeah, though I'm not sure I like that name... (can't think of a better one > right now, though). Are there use cases where the set of keys and the set of values is overlapping? In most cases you can probably get away by not offering an "inverse" object at all, just let regular lookup fall back on value lookup. Best, -Nikolaus (No Cc on replies please, I'm reading the list) -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From yselivanov.ml at gmail.com Fri Feb 12 18:17:48 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 12 Feb 2016 18:17:48 -0500 Subject: [Python-ideas] Incorporating something like byteplay into the stdlib In-Reply-To: References: <1465993608.2439410.1455249525441.JavaMail.yahoo.ref@mail.yahoo.com> <1465993608.2439410.1455249525441.JavaMail.yahoo@mail.yahoo.com> <56BE4244.6060504@gmail.com> <34117C96-D1E9-4A73-B3EB-B670C152C2BE@yahoo.com> <56BE5284.1010204@gmail.com> <56BE5543.7000302@gmail.com> Message-ID: <56BE681C.9070401@gmail.com> On 2016-02-12 5:38 PM, Andrew Barnert wrote: > Assuming I'm not imagining all that, why are people demanding that I provide a rationale for why we should add bytecode optimizers, or telling me that adding bytecode optimizers isn't going to help FAT Python, etc.? > > My proposal is that if we add bytecode optimizers, we should make it possible to write them. I don't need to justify that "if". If nobody things we should add bytecode optimizers, not even the author of the PEP that suggests them, then the answer is simple: just remove them from the PEP, and then my proposal becomes void. Perhaps I and other people don't understand the "if we add bytecode optimizers, we should make it possible to write them" part. I don't expect to see more than 1 or 2 optimizers out there. Writing a bytecode optimizer that can yield significant speedups is an extremely challenging task. Adding a lot of new functionality to the stdlib *just* for those few optimizers doesn't make a lot of sense. Moreover, having such tools in the stdlib, might cause people to start using them for things other than optimization -- something I don't like at all. We already have import hooks, AST etc for most of such purposes. Yury From mike at selik.org Fri Feb 12 18:35:25 2016 From: mike at selik.org (Michael Selik) Date: Fri, 12 Feb 2016 23:35:25 +0000 Subject: [Python-ideas] Generator unpacking In-Reply-To: <56BE6437.20305@lucidity.plus.com> References: <56BE6437.20305@lucidity.plus.com> Message-ID: On Fri, Feb 12, 2016 at 6:01 PM Erik wrote: > On 12/02/16 19:54, Andrew Barnert via Python-ideas wrote: > > Meanwhile, you can always write the expanded version out explicitly. > (And you can leave off the first line when you know c is already an > iterator.) Or you can use itertools.islice to make it more compact: > > > > >>> a, b = itertools.islice(c, 2) > > >>> rest = c > > Why not just have an itertools.unpack() - a simple version without > argument checking: > > def unpack(seq, num): > it = iter(seq) > yield from (i[1] for i in zip(range(num), it)) > yield it > > foo, bar, rest = unpack([1, 2, 3, 4, 5, 6], 2) > > Because it's in itertools, the expectation is that it has something to > do with iterators so the final return value always being an iterator > regardless of the original sequence type is reasonable (and is perhaps > the only justification for putting it in itertools in the first place ;) ). There's some visual dissonance since the ``num`` argument is asking for the number of elements to unpack, but the left-hand of the assignment has num+1 variables. What do you think about just using islice? >>> from itertools import islice >>> it = iter(range(5)) >>> (first, second), rest = islice(it, 0, 2), it >>> first 0 >>> second 1 >>> rest I suppose it might read better broken apart into two lines to emphasize that the state changed. >>> first, second = islice(it, 0, 2) >>> rest = it -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at lucidity.plus.com Fri Feb 12 18:36:27 2016 From: python at lucidity.plus.com (Erik) Date: Fri, 12 Feb 2016 23:36:27 +0000 Subject: [Python-ideas] zip() problem. Message-ID: <56BE6C7B.4040203@lucidity.plus.com> Hi. In writing my previous email, I noticed something about zip() that I'd not seen before (but is obvious, I guess) - when it reaches the shortest sequence and terminates, any iterators already processed in that pass will have generated one extra value than the others. Those additional values are discarded. For example: h = iter("Hello") w = iter("World") s = iter("Spam") e = iter("Eggs") for i in zip(h, w, s, e): print(i) for i in (h, w, s, e): print(list(i)) ---> All iterators are exhausted. h = iter("Hello") w = iter("World") s = iter("Spam") e = iter("Eggs") for i in zip(h, s, e, w): print(i) for i in (h, w, s, e): print(list(i)) ---> "w" still has the trailing 'd' character. So, if you're using zip() over itertools.zip_longest() then you have to be careful of the order of your arguments and try to put the probably-shortest one first if this would otherwise cause problems. The reason I'm posting to 'ideas' is: what should/could be done about it? 1) A simple warning in the docstring for zip()? 2) Something to prevent it (for example a keyword argument to zip() to switch on some behaviour where the iterators are first queried that they have more items to generate before the values start being consumed)? 3) Nothing. There are bigger things to worry about ;) WRT (2), I thought that perhaps __len__ was part of the iterator protocol, but it's not (just __iter__ and __next__), hence: >>> len(range(5, 40)) 35 >>> len(iter(range(5, 40))) Traceback (most recent call last): File "", line 1, in TypeError: object of type 'range_iterator' has no len() >>> len(iter("FooBar")) Traceback (most recent call last): File "", line 1, in TypeError: object of type 'str_iterator' has no len() ... though would that also be something to consider (I guess all iterators would have to keep some state regarding the amount of values previously generated and then apply that offset to the result of len() on the underlying object)? Perhaps that would just be too heavyweight for what is a relatively minor wart. E. From mike at selik.org Fri Feb 12 18:50:23 2016 From: mike at selik.org (Michael Selik) Date: Fri, 12 Feb 2016 23:50:23 +0000 Subject: [Python-ideas] zip() problem. In-Reply-To: <56BE6C7B.4040203@lucidity.plus.com> References: <56BE6C7B.4040203@lucidity.plus.com> Message-ID: On Fri, Feb 12, 2016 at 6:39 PM Erik wrote: > Hi. > > In writing my previous email, I noticed something about zip() that I'd > not seen before (but is obvious, I guess) - when it reaches the shortest > sequence and terminates, any iterators already processed in that pass > will have generated one extra value than the others. Those additional > values are discarded. > > For example: > > h = iter("Hello") > w = iter("World") > s = iter("Spam") > e = iter("Eggs") > > for i in zip(h, w, s, e): > print(i) > > for i in (h, w, s, e): > print(list(i)) > > ---> All iterators are exhausted. > > h = iter("Hello") > w = iter("World") > s = iter("Spam") > e = iter("Eggs") > > for i in zip(h, s, e, w): > print(i) > > for i in (h, w, s, e): > print(list(i)) > > > ---> "w" still has the trailing 'd' character. > > > So, if you're using zip() over itertools.zip_longest() then you have to > be careful of the order of your arguments and try to put the > probably-shortest one first if this would otherwise cause problems. > > > The reason I'm posting to 'ideas' is: what should/could be done about it? > > 1) A simple warning in the docstring for zip()? > I wouldn't want to clutter the docstring, but a note in the long-form documentation could be useful. > 2) Something to prevent it (for example a keyword argument to zip() to > switch on some behaviour where the iterators are first queried that they > have more items to generate before the values start being consumed)? > How can you query whether an iterator has another value without consuming that value? > 3) Nothing. There are bigger things to worry about ;) > > WRT (2), I thought that perhaps __len__ was part of the iterator > protocol, but it's not (just __iter__ and __next__), hence: > > >>> len(range(5, 40)) > 35 > >>> len(iter(range(5, 40))) > Traceback (most recent call last): > File "", line 1, in > TypeError: object of type 'range_iterator' has no len() > >>> len(iter("FooBar")) > Traceback (most recent call last): > File "", line 1, in > TypeError: object of type 'str_iterator' has no len() > > ... though would that also be something to consider (I guess all > iterators would have to keep some state regarding the amount of values > previously generated and then apply that offset to the result of len() > on the underlying object)? Perhaps that would just be too heavyweight > for what is a relatively minor wart. > How would you handle the length of an infinite iterator? Or one that *might* be infinite, depending on current state of the program? A more realistic example: if I'm looking up N records from a distributed database, I might do that in parallel and get the results back unordered, as an iterator. If M of the queries timeout, I might choose to ignore those records and exclude them from the resulting iterator. So, when I kick off the queries, the length of that iterator might be N. When the timeouts are finished, the length is N-M. Further, if I've consumed 2 records, is the length still N-M or N-M-2? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at selik.org Fri Feb 12 18:51:34 2016 From: mike at selik.org (Michael Selik) Date: Fri, 12 Feb 2016 23:51:34 +0000 Subject: [Python-ideas] zip() problem. In-Reply-To: References: <56BE6C7B.4040203@lucidity.plus.com> Message-ID: BTW, from the documentation ( https://docs.python.org/3/library/functions.html#zip): "zip() should only be used with unequal length inputs when you don?t care about trailing, unmatched values from the longer iterables. If those values are important, useitertools.zip_longest() instead." On Fri, Feb 12, 2016 at 6:50 PM Michael Selik wrote: > On Fri, Feb 12, 2016 at 6:39 PM Erik wrote: > >> Hi. >> >> In writing my previous email, I noticed something about zip() that I'd >> not seen before (but is obvious, I guess) - when it reaches the shortest >> sequence and terminates, any iterators already processed in that pass >> will have generated one extra value than the others. Those additional >> values are discarded. >> >> For example: >> >> h = iter("Hello") >> w = iter("World") >> s = iter("Spam") >> e = iter("Eggs") >> >> for i in zip(h, w, s, e): >> print(i) >> >> for i in (h, w, s, e): >> print(list(i)) >> >> ---> All iterators are exhausted. >> >> h = iter("Hello") >> w = iter("World") >> s = iter("Spam") >> e = iter("Eggs") >> >> for i in zip(h, s, e, w): >> print(i) >> >> for i in (h, w, s, e): >> print(list(i)) >> >> >> ---> "w" still has the trailing 'd' character. >> >> >> So, if you're using zip() over itertools.zip_longest() then you have to >> be careful of the order of your arguments and try to put the >> probably-shortest one first if this would otherwise cause problems. >> >> >> The reason I'm posting to 'ideas' is: what should/could be done about it? >> >> 1) A simple warning in the docstring for zip()? >> > > I wouldn't want to clutter the docstring, but a note in the long-form > documentation could be useful. > > >> 2) Something to prevent it (for example a keyword argument to zip() to >> switch on some behaviour where the iterators are first queried that they >> have more items to generate before the values start being consumed)? >> > > How can you query whether an iterator has another value without consuming > that value? > > >> 3) Nothing. There are bigger things to worry about ;) >> >> WRT (2), I thought that perhaps __len__ was part of the iterator >> protocol, but it's not (just __iter__ and __next__), hence: >> >> >>> len(range(5, 40)) >> 35 >> >>> len(iter(range(5, 40))) >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: object of type 'range_iterator' has no len() >> >>> len(iter("FooBar")) >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: object of type 'str_iterator' has no len() >> >> ... though would that also be something to consider (I guess all >> iterators would have to keep some state regarding the amount of values >> previously generated and then apply that offset to the result of len() >> on the underlying object)? Perhaps that would just be too heavyweight >> for what is a relatively minor wart. >> > > How would you handle the length of an infinite iterator? Or one that > *might* be infinite, depending on current state of the program? > > A more realistic example: if I'm looking up N records from a distributed > database, I might do that in parallel and get the results back unordered, > as an iterator. If M of the queries timeout, I might choose to ignore those > records and exclude them from the resulting iterator. So, when I kick off > the queries, the length of that iterator might be N. When the timeouts are > finished, the length is N-M. Further, if I've consumed 2 records, is the > length still N-M or N-M-2? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Feb 12 18:59:41 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 12 Feb 2016 15:59:41 -0800 Subject: [Python-ideas] Dict with inverse form In-Reply-To: <8737sx4jmp.fsf@thinkpad.rath.org> References: <935127780.2464284.1455250067522.JavaMail.yahoo@mail.yahoo.com> <4BA45D4A-FAFB-4545-8570-F039039679AD@yahoo.com> <8737sx4jmp.fsf@thinkpad.rath.org> Message-ID: On Fri, Feb 12, 2016 at 3:02 PM, Nikolaus Rath wrote: > > Are there use cases where the set of keys and the set of values is > overlapping? > > In most cases you can probably get away by not offering an "inverse" > object at all, just let regular lookup fall back on value lookup. > hmm, interesting. my use case at hand is mapping color name strings to RGB values -- it should be a one to one relationship, and I need to look for either one. But you are quite right, they would never overlap, unless someone decided a nice name for a color would be "(125, 15, 235)" -- even then they wouldn't because the RGB value sare a tuple, not a string. So that may be the way to get the easiest interface. -Thanks, - Chris > > Best, > -Nikolaus > > (No Cc on replies please, I'm reading the list) > -- > GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F > Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F > > ?Time flies like an arrow, fruit flies like a Banana.? > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Feb 12 19:03:31 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 12 Feb 2016 16:03:31 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BE6437.20305@lucidity.plus.com> Message-ID: <761F772B-C257-4F08-A52D-74070199AEB1@yahoo.com> On Feb 12, 2016, at 15:35, Michael Selik wrote: > >> On Fri, Feb 12, 2016 at 6:01 PM Erik wrote: >> On 12/02/16 19:54, Andrew Barnert via Python-ideas wrote: >> > Meanwhile, you can always write the expanded version out explicitly. (And you can leave off the first line when you know c is already an iterator.) Or you can use itertools.islice to make it more compact: >> > >> > >>> a, b = itertools.islice(c, 2) >> > >>> rest = c >> >> Why not just have an itertools.unpack() - a simple version without >> argument checking: >> >> def unpack(seq, num): >> it = iter(seq) >> yield from (i[1] for i in zip(range(num), it)) >> yield it Creating a range just to zip with just to throw away the values seems like overcomplicating things. Unless there's some performance benefit to doing it that way, why not just keep it simple? def unpack(seq, num): it = iter(seq) yield from islice(it, num) yield it Or, to be super novice-friendly: def unpack(seq, num): it = iter(seq) for _ in range(num): yield next(it) yield it >> foo, bar, rest = unpack([1, 2, 3, 4, 5, 6], 2) That doesn't really distinguish the "rest" very clearly. Of course we could just change the last line to "yield [it]", and then call it with "foo, bar, *rest =", but that seems cheesy to me. I don't know; if there is something better than islice to be found here, I think you're probably on the right track, but I don't think you're there yet, and I'm not sure there is anything to find. Unpacking syntax just feels "sequency" to me, and in a language with distinct sequences and iterators (instead of, say, lazy sequences like Haskell, or views wrapping many iterators like Swift), I think that means non-lazy. But hopefully I'm wrong. :) >> Because it's in itertools, the expectation is that it has something to >> do with iterators so the final return value always being an iterator >> regardless of the original sequence type is reasonable (and is perhaps >> the only justification for putting it in itertools in the first place ;) ). > > There's some visual dissonance since the ``num`` argument is asking for the number of elements to unpack, but the left-hand of the assignment has num+1 variables. > > What do you think about just using islice? > > >>> from itertools import islice > >>> it = iter(range(5)) > >>> (first, second), rest = islice(it, 0, 2), it > >>> first > 0 > >>> second > 1 > >>> rest > > > I suppose it might read better broken apart into two lines to emphasize that the state changed. > > >>> first, second = islice(it, 0, 2) > >>> rest = it That's exactly what was in my email, as the way to do things today, which he was replying to: >> Or you can use itertools.islice to make it more compact: >> > >> > >>> a, b = itertools.islice(c, 2) >> > >>> rest = c So I think we can assume that he thinks his version improves over that, or he wouldn't have suggested it... -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Feb 12 19:09:24 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 12 Feb 2016 16:09:24 -0800 Subject: [Python-ideas] zip() problem. In-Reply-To: References: <56BE6C7B.4040203@lucidity.plus.com> Message-ID: On Feb 12, 2016, at 15:51, Michael Selik wrote: > > BTW, from the documentation (https://docs.python.org/3/library/functions.html#zip): > > "zip() should only be used with unequal length inputs when you don?t care about trailing, unmatched values from the longer iterables. If those values are important, useitertools.zip_longest() instead." I think what's missing (from his point of view) is some statement that if you call it with iterators, it's not just the trailing values, but also the iterators' states that you shouldn't care about. I always took that as read without it needing to be stated. But maybe it does need stating? -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at lucidity.plus.com Fri Feb 12 19:22:58 2016 From: python at lucidity.plus.com (Erik) Date: Sat, 13 Feb 2016 00:22:58 +0000 Subject: [Python-ideas] Generator unpacking In-Reply-To: <761F772B-C257-4F08-A52D-74070199AEB1@yahoo.com> References: <56BE6437.20305@lucidity.plus.com> <761F772B-C257-4F08-A52D-74070199AEB1@yahoo.com> Message-ID: <56BE7762.1030804@lucidity.plus.com> On 13/02/16 00:03, Andrew Barnert wrote: > On Feb 12, 2016, at 15:35, Michael Selik > wrote: > Creating a range just to zip with just to throw away the values seems > like overcomplicating things. Unless there's some performance benefit to > doing it that way, why not just keep it simple? I agree. I've never used islice() before, so I missed that as a better way of yielding the first 'n' values. > That's exactly what was in my email, as the way to do things today, > which he was replying to: > >> Or you can use itertools.islice to make it more compact: >> > >> > >>> a, b = itertools.islice(c, 2) >> > >>> rest = c > > So I think we can assume that he thinks his version improves over that, > or he wouldn't have suggested it... My "suggestion" was simply that perhaps creating a very short wrapper function somewhere that handles whether the sequence is already an iterator or not etc (and using islice() or whatever - I don't really care ;)) would perhaps be a more pragmatic option than trying to squeeze in some syntax change or underlying unpack heuristic/mechanic (which is where I thought the thread was heading). Perhaps I didn't express that very clearly. I'm happy do drop it ;) E. From abarnert at yahoo.com Fri Feb 12 19:23:05 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 12 Feb 2016 16:23:05 -0800 Subject: [Python-ideas] Dict with inverse form In-Reply-To: References: <935127780.2464284.1455250067522.JavaMail.yahoo@mail.yahoo.com> <4BA45D4A-FAFB-4545-8570-F039039679AD@yahoo.com> <8737sx4jmp.fsf@thinkpad.rath.org> Message-ID: On Feb 12, 2016, at 15:59, Chris Barker wrote: > >> On Fri, Feb 12, 2016 at 3:02 PM, Nikolaus Rath wrote: >> >> Are there use cases where the set of keys and the set of values is >> overlapping? >> >> In most cases you can probably get away by not offering an "inverse" >> object at all, just let regular lookup fall back on value lookup. > > hmm, interesting. my use case at hand is mapping color name strings to RGB values -- it should be a one to one relationship, and I need to look for either one. > > But you are quite right, they would never overlap, unless someone decided a nice name for a color would be "(125, 15, 235)" -- even then they wouldn't because the RGB value sare a tuple, not a string. > > So that may be the way to get the easiest interface. If that's your use case, there's an even easier interface: just toss the forward and reverse mappings in the same dict. If you don't need to iterate the dict[^1] or print in user-friendly form, it's hard to beat that for simplicity or compactness.[^2] Of the top of my head (untested): class SelfInvertingDict(dict): def __delitem__(self, key): super().__delitem__(self[key]) super().__delitem__(key) def __setitem__(self, key, value): if value in self and self[value] != key: raise ValueError("duplicate key: '{}'".format(value) if key in self: del self[key] super().__setitem__(key, value) super().__setitem__(value, key) [^1]: Even if you do need to iterate, you can always do "(or don't mind iterating with a type switch like "names = (key for key in d if isinstance(key, str))" [^2]: On the other hand, it's pretty easy to beat for static-type-checking purposes. The actual type of that dual dict is pretty ugly, and probably not describable in mypy terms, so presumably you'd just punt and label it Dict[Any, Any], or at best stick Union types in for key and value. But that's true for the double-dict-with-fall-through design too. -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at lucidity.plus.com Fri Feb 12 19:52:41 2016 From: python at lucidity.plus.com (Erik) Date: Sat, 13 Feb 2016 00:52:41 +0000 Subject: [Python-ideas] zip() problem. In-Reply-To: References: <56BE6C7B.4040203@lucidity.plus.com> Message-ID: <56BE7E59.6010301@lucidity.plus.com> On 13/02/16 00:09, Andrew Barnert wrote: > On Feb 12, 2016, at 15:51, Michael Selik > wrote: > >> BTW, from the documentation >> (https://docs.python.org/3/library/functions.html#zip): >> [snip] > I think what's missing (from his point of view) is some statement that > if you call it with iterators, it's not just the trailing values, but > also the iterators' states that you shouldn't care about. Yes. And also, as someone else pointed out to me privately, the docstring can be interpreted as already covering this: """ Return a zip object whose .__next__() method returns a tuple where the i-th element comes from the i-th iterable argument. The .__next__() method continues until the shortest iterable in the argument sequence is exhausted and then it raises StopIteration. """ It just depends on what "The .__next__() method continues" is supposed to mean. I can see that for the obvious implementation it means what actually happens (because the method is implemented in the obvious way ;)), but it _could_ be interpreted as meaning that when __next__ is called when the shortest iterable is exhausted then it does not do anything at all. This is in the docstring of a function that will be called by casual and newbie users. Are they expected to read up on what __next__ means and mentally imagine the mechanics of the loop that is implementing this function for them so they understand all the side-effects? > I always took that as read without it needing to be stated. But maybe it > does need stating? There is also the issue that the CPython implementation of this is not necessarily the _only_ way of implementing this. Another implementation might construct the tuple in reverse order for example, and a different set of iterators have the extra value consumed. I don't think it's unreasonable to state clearly that _any_ iterator longer than the shortest may or may not have at least one extra value extracted from it, which will then be discarded. Anyway, I'm over it now. I just thought I'd mention it. I've obviously never run into a real problem with it in the wild so perhaps it's really not an issue. E. From victor.stinner at gmail.com Fri Feb 12 20:28:20 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 13 Feb 2016 02:28:20 +0100 Subject: [Python-ideas] Incorporating something like byteplay into the stdlib In-Reply-To: References: <1465993608.2439410.1455249525441.JavaMail.yahoo.ref@mail.yahoo.com> <1465993608.2439410.1455249525441.JavaMail.yahoo@mail.yahoo.com> Message-ID: Hi, I understand that you have 3 major points: (1) byteplay lags behind CPython, it's difficult to maintain it (2) you want to integrate the code features of byteplay into the dis module (3) you want to use a new API of the dis module in the PEP 511 for bytecode transformers For the point (1), it may be fixed by the point (2). Otherwise, I'm not interested to add byteplay into CPython. I prefer to not promote too much the usage of bytecode transformers, since the bytecode is very low-level: it depends on the Python minor version and is not portable accross implementations of Python. We want to be free to modify deeply the bytecode in CPython. Yury explained that before than me ;-) IMHO the point (2) is just fine. Go ahead! I'm opposed to the point (3) because it would couple too much the exact implementation of bytecodes to the PEP 511 API. I tried to write an API which can be implemented by all Python implementations, not only CPython. See: https://www.python.org/dev/peps/pep-0511/#other-python-implementations IMHO a "black-box" bytecode transformer API is the best we can do to support all Python implementations. By the way, (3) requires to reimplement the dis module in C to bootstrap Python. IMHO it will be boring to write the C code, and much more annoying to maintain it. So quickly, we will refuse any enhancement on the API. I don't think that you want that, since you want to experiment new things, right? It is already to build the API you described *on top of the PEP 511*. Such API would probably be specific to CPython and to byteplay (or dis module if you enhance it). Example: --- import dis import sys # PEP 511 code transformer class ByteplayRegistry: name = "bytecode" def __init__(self): self._transformers = [] def register(self, name, transformer): # FIXME: update self.name? self._transformers.append(transformer) def disassemble(self, code): # FIXME: optional support for byteplay? return dis.dis(code) def assemble(self, asm, code): # FIXME: implement assembler return ... def code_transformer(self, code, context): # disassemble() and assemble() is only called once for all transformers asm = self.disassemble(code) for transformer in self._transformers: asm = transformer(asm, code, context) return self.assemble(asm, code) def global_to_const(asm, code, context): # FIXME: implement optimization return asm byteplay_registry = ByteplayRegistry() byteplay_registry.register("global_to_const", global_to_const) sys.set_code_transformers([byteplay_registry]) --- 2016-02-12 20:16 GMT+01:00 Andrew Barnert : > Your API passes only the bytecode string, consts, lnotab, and global names. There is no way to write a bytecode processor that looks up locals with that interface, because you didn't pass the locals. Oh, it looks like you are referring to an old version of the PEP 511 which passed 5 parameters to code_transformer(). The latest version now pass a whole code object to the transformer, and the transformer must return a new code object: https://www.python.org/dev/peps/pep-0511/#code-transformer-method I just saw that I forgot to update the example of bytecode transformer. It's now fixed. https://www.python.org/dev/peps/pep-0511/#bytecode-transformer I also updated the implementation: https://bugs.python.org/issue26145 Victor From g.rodola at gmail.com Sat Feb 13 06:35:03 2016 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Sat, 13 Feb 2016 12:35:03 +0100 Subject: [Python-ideas] atexit.register_w_signals() In-Reply-To: References: <56BE2036.8010503@stoneleaf.us> Message-ID: On Fri, Feb 12, 2016 at 8:00 PM, Chris Angelico wrote: > On Sat, Feb 13, 2016 at 5:36 AM, Giampaolo Rodola' > wrote: > > Also, we would still have to use atexit.register() so that the new > function > > is called also on "clean" interpreter exit and take into account other > > signals other than SIGTERM which would cause the process to terminate > > (SIGINT, etc.). My proposal is to evaluate adding a higher-level function > > which takes care of all these details, providing better chances to > execute > > the exit function than atexit.register or signal.signal alone. > > The trouble is, you can't know which signals will cause a process to > terminate. According to 'man signal', my system has these signals set > to terminate the process by default: HUP, INT, KILL, PIPE, ALRM, TERM, > and USR1/USR2 etc; and these will terminate and dump core: QUIT, ILL, > ABRT, and FPE. Which of them is it correct to register your atexit > handler for? Yeah, that is true. But there are certain signals which are undoubtedly designed to terminate a process: SIGTERM / SIGINT, which are the most used, and SIGQUIT / SIGABRT, which I've personally never used but their meaning is quite obvious. What people want most of the times is to address *those* signal, and if they want to do something different (e.g. associate SIGHUP) they can simply pass it as an argument (register_exit_func(signals=[SIGTERM, SIGHUP])). > SIGINT should result in KeyboardInterrupt, so I wouldn't hook that - > let normal exception handling take care of it. Agreed. > SIGHUP, SIGPIPE, and > SIGALRM are often changed in disposition; will you hook them only if > they're on their defaults of terminating the process. I would say hooking these up should be up to the user. -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sat Feb 13 02:59:55 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 13 Feb 2016 20:59:55 +1300 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: Message-ID: <56BEE27B.9000804@canterbury.ac.nz> Andrew Barnert via Python-ideas wrote: > I think what you're _really_ suggesting here is that there should be a way to > unpack iterators lazily, Seems to me you should be able to write a, b, * = some_iter as a way of saying "I don't care about the rest of the values". -- Greg From rosuav at gmail.com Sat Feb 13 07:26:18 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 13 Feb 2016 23:26:18 +1100 Subject: [Python-ideas] atexit.register_w_signals() In-Reply-To: References: <56BE2036.8010503@stoneleaf.us> Message-ID: On Sat, Feb 13, 2016 at 10:35 PM, Giampaolo Rodola' wrote: > Yeah, that is true. But there are certain signals which are undoubtedly > designed to terminate a process: SIGTERM / SIGINT, which are the most used, > and SIGQUIT / SIGABRT, which I've personally never used but their meaning is > quite obvious. What people want most of the times is to address *those* > signal, and if they want to do something different (e.g. associate SIGHUP) > they can simply pass it as an argument (register_exit_func(signals=[SIGTERM, > SIGHUP])). > SIGTERM is generally expected to terminate a process cleanly. SIGQUIT is an unclean termination, and by default will dump core as well as terminating. SIGABRT is often sent from the process itself (via the abort() C function), and normally means a critical invariant has been broken (eg trampling all over vital memory structures like the stack or heap). SIGINT is a standard "break" signal, and doesn't always mean process termination at all (start an infinite loop in interactive Python, then hit Ctrl-C - you expect to go back to the prompt). Of them, only SIGTERM can consistently be interpreted as a termination signal. Some programs treat SIGINT and SIGTERM as different forms of termination (consider PostgreSQL - SIGTERM means "stop accepting connections, and terminate when current connections finish", and SIGINT means "abort current connections and shut down immediately"), but a general library function can't assume that. Ultimately, signals belong to the application, not a library. It's impossible for a library to be able to implement an atexit system in any sort of general way. If I were faced with this problem, I'd probably make a simple way to turn a bunch of signals into raised exceptions and do all the atexit work with context managers or finally blocks. That would be reasonably clean, without trying to promise something that won't always make sense. ChrisA From g.rodola at gmail.com Sat Feb 13 07:53:08 2016 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Sat, 13 Feb 2016 13:53:08 +0100 Subject: [Python-ideas] atexit.register_w_signals() In-Reply-To: References: <56BE2036.8010503@stoneleaf.us> Message-ID: On Sat, Feb 13, 2016 at 1:26 PM, Chris Angelico wrote: > On Sat, Feb 13, 2016 at 10:35 PM, Giampaolo Rodola' > wrote: > > Yeah, that is true. But there are certain signals which are undoubtedly > > designed to terminate a process: SIGTERM / SIGINT, which are the most > used, > > and SIGQUIT / SIGABRT, which I've personally never used but their > meaning is > > quite obvious. What people want most of the times is to address *those* > > signal, and if they want to do something different (e.g. associate > SIGHUP) > > they can simply pass it as an argument > (register_exit_func(signals=[SIGTERM, > > SIGHUP])). > > > > SIGTERM is generally expected to terminate a process cleanly. SIGQUIT > is an unclean termination, and by default will dump core as well as > terminating. SIGABRT is often sent from the process itself (via the > abort() C function), and normally means a critical invariant has been > broken (eg trampling all over vital memory structures like the stack > or heap). SIGINT is a standard "break" signal, and doesn't always mean > process termination at all (start an infinite loop in interactive > Python, then hit Ctrl-C - you expect to go back to the prompt). > > Of them, only SIGTERM can consistently be interpreted as a termination > signal. Some programs treat SIGINT and SIGTERM as different forms of > termination (consider PostgreSQL - SIGTERM means "stop accepting > connections, and terminate when current connections finish", and > SIGINT means "abort current connections and shut down immediately"), > but a general library function can't assume that. > > Ultimately, signals belong to the application, not a library. It's > impossible for a library to be able to implement an atexit system in > any sort of general way. If I were faced with this problem, I'd > probably make a simple way to turn a bunch of signals into raised > exceptions and do all the atexit work with context managers or finally > blocks. That would be reasonably clean, without trying to promise > something that won't always make sense. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > OK, I am convinced it makes sense to look for SIGTERM only. Also, I now have a better understanding of how different signal.signal() works on Windows http://bugs.python.org/msg260201 and it probably makes no sense to make an effort for trying to support Windows at all. > Of them, only SIGTERM can consistently be interpreted as a termination > signal. Some programs treat SIGINT and SIGTERM as different forms of > termination (consider PostgreSQL - SIGTERM means "stop accepting > connections, and terminate when current connections finish", and > SIGINT means "abort current connections and shut down immediately"), > but a general library function can't assume that. In this case such a kind of app wouldn't use something like register_exit_fun(). The problems I'm trying to address are 2: run the exit function for the most used/obvious/portable termination signal (SIGTERM), which is what atexit lacks, and at the same time avoid to overwrite any existing signal.signal() handler. In order to do this right now I'm supposed to use `atexit.register` *and* a wrapper around signal.signal(). -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Feb 13 08:06:22 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 13 Feb 2016 23:06:22 +1000 Subject: [Python-ideas] Generator unpacking In-Reply-To: <56BEE27B.9000804@canterbury.ac.nz> References: <56BEE27B.9000804@canterbury.ac.nz> Message-ID: On 13 February 2016 at 17:59, Greg Ewing wrote: > Andrew Barnert via Python-ideas wrote: >> >> I think what you're _really_ suggesting here is that there should be a way >> to >> unpack iterators lazily, > > > Seems to me you should be able to write > > a, b, * = some_iter > > as a way of saying "I don't care about the rest of the values". The main problem with that specific spelling of the idea is that it's the inverse of what the bare "*" means when declaring function parameters - there, it's a way of marking the end of the positional arguments, when you want to put keyword only arguments after it. The other problem is that it makes: a *= value and a, b, *= value mean wildly different things. Where these discussions generally end up is: 1. The cases where you actually want "unpack this many values, ignore the rest" are pretty rare 2. When you do really need it, islice handles it 3. Adding new syntax isn't warranted for a relatively rare use case the stdlib already covers Probably the most plausible idea would be a "head()" recipe that does something like: def head(iterable, n): itr = iter(iterable) return tuple(islice(itr, n)), itr Usable as: (a, b), rest = head(iterable, 2) Making it a recipe means folks can customise it as they wish (e.g. omitting the tuple call) However, I'm not sure how much that would actually help, since using islice directly here is already pretty straightforward (once you know about it), and for small numbers of items, you can also just use the next builtin if you know you have an iterator: itr = iter(iterable) a = next(itr) b, c = next(itr), next(itr) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rosuav at gmail.com Sat Feb 13 08:22:07 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 14 Feb 2016 00:22:07 +1100 Subject: [Python-ideas] atexit.register_w_signals() In-Reply-To: References: <56BE2036.8010503@stoneleaf.us> Message-ID: On Sat, Feb 13, 2016 at 11:53 PM, Giampaolo Rodola' wrote: > In this case such a kind of app wouldn't use something like > register_exit_fun(). The problems I'm trying to address are 2: run the exit > function for the most used/obvious/portable termination signal (SIGTERM), > which is what atexit lacks, and at the same time avoid to overwrite any > existing signal.signal() handler. In order to do this right now I'm supposed > to use `atexit.register` *and* a wrapper around signal.signal(). import signal class SigTerm(SystemExit): pass def sigterm(sig,frm): raise SigTerm signal.signal(15,sigterm) import atexit atexit.register(print,"atexit called") import os print("I am",os.getpid()) print("Hit enter to exit.") input() Once you turn the signal into an exception, atexit and finally both work as normal. Add a bit more sophistication to the first four lines, wrap 'em up in a library function, and then you don't have to worry about the complexity of a still-not-guaranteed atexit. ChrisA From oscar.j.benjamin at gmail.com Sat Feb 13 11:36:25 2016 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sat, 13 Feb 2016 16:36:25 +0000 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BEE27B.9000804@canterbury.ac.nz> Message-ID: On 13 February 2016 at 13:06, Nick Coghlan wrote: > 1. The cases where you actually want "unpack this many values, ignore > the rest" are pretty rare It's not so much ignore the rest but rather retain the rest for separate consumption. This happens when you want to either peek or split the first item. For example in parsing a csv file: def readcsv(csvfile): csvfile = map(str.split, csvfile) try: fieldnames = next(csvfile) except StopIteration: raise ValueError('Bad csv file') return [dict(zip(fieldnames, line)) for line in csvfile] It would be nicer to write that as something like fieldnames, * = csvfile Another situation where I've wanted that is given an iterable that yields sequences all of the same length I might want to peek the first item to check its length before the loop begins. > 2. When you do really need it, islice handles it That's true so you can do fieldnames, = islice(csvfile, 1) Somehow I don't like that but really it's fine. -- Oscar From abarnert at yahoo.com Sat Feb 13 13:00:53 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 13 Feb 2016 10:00:53 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BEE27B.9000804@canterbury.ac.nz> Message-ID: On Feb 13, 2016, at 08:36, Oscar Benjamin wrote: > >> On 13 February 2016 at 13:06, Nick Coghlan wrote: >> 1. The cases where you actually want "unpack this many values, ignore >> the rest" are pretty rare > > It's not so much ignore the rest but rather retain the rest for > separate consumption. This happens when you want to either peek or > split the first item. For example in parsing a csv file: > > def readcsv(csvfile): > csvfile = map(str.split, csvfile) > try: > fieldnames = next(csvfile) > except StopIteration: > raise ValueError('Bad csv file') > return [dict(zip(fieldnames, line)) for line in csvfile] > > It would be nicer to write that as something like > > fieldnames, * = csvfile So instead of getting an exception that says "bad csv file" you get one that says "unpacking expected at least 1 value, found 0" or something? That doesn't seem nicer. But if we really do need such functionality commonly, just adding an exception type parameter to next would cover it. Or just putting an exception-wrapping function in the stdlib, so you can just write "fieldnames = exwrap(next, StopIteration, ValueError)(csvfile)". But then anyone can write exwrap as a three-line function today, and I don't think anyone ever does, so I doubt it needs to be standardized... Also, is it worth mentioning that this doesn't actually parse csv files, but whitespace-separated files, without any form of escaping or quoting, and probably doing the wrong thing on short rows? Because there is a really easy way of writing this function that's a lot nicer and a lot shorter and raises meaningful exceptions and actually works: def readcsv(csvfile): return list(csv.DictReader(csvfile)) (Although I'm not sure why you want a list rather than an iterator in the first place.) > Another situation where I've wanted that is given an iterable that > yields sequences all of the same length I might want to peek the first > item to check its length before the loop begins. Again, next. >> 2. When you do really need it, islice handles it > > That's true so you can do > > fieldnames, = islice(csvfile, 1) > > Somehow I don't like that but really it's fine. From greg.ewing at canterbury.ac.nz Sat Feb 13 16:40:26 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 14 Feb 2016 10:40:26 +1300 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BEE27B.9000804@canterbury.ac.nz> Message-ID: <56BFA2CA.2080404@canterbury.ac.nz> Nick Coghlan wrote: > On 13 February 2016 at 17:59, Greg Ewing wrote: > >> a, b, * = some_iter > > The main problem with that specific spelling of the idea is that it's > the inverse of what the bare "*" means when declaring function > parameters I know, but despite that, this still seems like tbe "obvious" way to spell it to me. > a *= value > > and > > a, b, *= value > > mean wildly different things. Another possibility is a, b, ... = value > 2. When you do really need it, islice handles it I find that answer unsatisfying, because by using islice I'm telling it to do *more* work, whereas I really want to tell it to do *less* work. It just seems wrong, like a kind of abstraction inversion. -- Greg From ncoghlan at gmail.com Sun Feb 14 02:43:53 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 14 Feb 2016 17:43:53 +1000 Subject: [Python-ideas] Generator unpacking In-Reply-To: <56BFA2CA.2080404@canterbury.ac.nz> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> Message-ID: On 14 February 2016 at 07:40, Greg Ewing wrote: > > Another possibility is > > a, b, ... = value Now, *that* spelling to turn off the "implicit peek" behaviour in iterable unpacking I quite like. arg_iter = iter(args) command, ... = arg_iter run_command(command, arg_iter) Although again, the main downside would be that "..." here means something rather different from what it means as a subscript element. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From phyks at phyks.me Sun Feb 14 08:23:37 2016 From: phyks at phyks.me (Phyks) Date: Sun, 14 Feb 2016 14:23:37 +0100 Subject: [Python-ideas] Sharing modules across virtualenvs In-Reply-To: References: <56BDAD55.9040608@phyks.me> Message-ID: <56C07FD9.6030309@phyks.me> Thanks for the explanations! Will look at distuil-sig for further infos. -- Phyks Le 12/02/2016 11:53, Nick Coghlan a ?crit : > On 12 February 2016 at 20:00, Phyks wrote: >> Hi, >> >> I was wondering recently about the way virtualenvs work, and did not >> find much information on the design choices. Sorry if this is not the >> right mailing-list to post about it, but it seemed to be. >> >> As far as I understand, virtualenvs work by modifying the PYTHONHOME and >> then the folders in which to look up to import modules. >> >> Are there any specific reasons for this implementation, and not a >> "/lib"-like implementation, putting the various versions of the module >> in a single folder, with a version naming, and symlinking the up-to-date >> one? > > Symlinking unfortunately doesn't solve the problem for a couple of reasons: > > * symlinks are still basically unusable on Windows > * symlinks and https://packaging.python.org/en/latest/specifications/#recording-installed-distributions > don't get along > > Module sharing between virtual environments *could* still be > implemented based on *.pth files, but: > > * you'd need to update an installer (probably pip) to handle shared > installs and uninstalls > * you'd have a refcounting problem on deciding when it was OK to > uninstall versions > * you'd run into the startup performance problems that can arise with > .pth files and adding lots of entries to sys.path > > And even if you do manage to solve all those problems, at the end of > it, you'll have essentially reinvented some of the core components of > nix: https://nixos.org/nix/ > >> Then, virtualenvs could simply be applied as filters on this specific >> folder, to hide part of the available module. Contrary to the current >> implementation, this means building (and storing, although storage is >> said to be cheap) large Python modules only once for your system, and >> share them. > > The implicit wheel cache used in recent versions of pip tackles that > problem in a different way (the built wheel fields are cached, so you > don't have to build them again, but they still get unpacked separately > into each virtualenv that needs them) > > Cheers, > Nick. > > P.S. This is more a distutils-sig topic than a python-ideas one, so if > you'd like to learn more or explore the idea further, I'd suggest > following up there > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > From stephen at xemacs.org Sun Feb 14 11:39:10 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 15 Feb 2016 01:39:10 +0900 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> Message-ID: <22208.44462.871604.315132@turnbull.sk.tsukuba.ac.jp> Nick Coghlan writes: > On 14 February 2016 at 07:40, Greg Ewing wrote: > > Another possibility is > > > > a, b, ... = value +1 It may be TOOWTDI, I thought of it independently. > Although again, the main downside would be that "..." here means > something rather different from what it means as a subscript > element. But not what it means in natural language, which is basically "continues as expected". That's somewhat different from "*" which has no established meaning in natural language (except "note here"), and which is already heavily used in Python. Steve From g.brandl at gmx.net Sun Feb 14 13:57:09 2016 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 14 Feb 2016 19:57:09 +0100 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> Message-ID: On 02/14/2016 08:43 AM, Nick Coghlan wrote: > On 14 February 2016 at 07:40, Greg Ewing > > wrote: > > > Another possibility is > > a, b, ... = value > > > Now, *that* spelling to turn off the "implicit peek" behaviour in iterable > unpacking I quite like. > > arg_iter = iter(args) > command, ... = arg_iter > run_command(command, arg_iter) > Assigning to Ellipsis? Interesting idea, but I'd probably go a bit further in the similarity to star-assignment: a, b, *... = value Ellipsis could then be a general "throw-away" lvalue. This would make it possible to say a, ..., b, ... = some_function() i.e. skip exactly one item in unpacking. > Although again, the main downside would be that "..." here means something > rather different from what it means as a subscript element. Keep in mind that Ellipsis is also a legal rvalue anywhere else. I.e. this would be legal (and a no-op): ... = ... But thinking about it, this is also legal at the moment: [] = [] Interestingly, this raises: () = () cheers, Georg From greg.ewing at canterbury.ac.nz Sun Feb 14 15:44:55 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 15 Feb 2016 09:44:55 +1300 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> Message-ID: <56C0E747.2040806@canterbury.ac.nz> Georg Brandl wrote: > a, b, *... = value > > Ellipsis could then be a general "throw-away" lvalue. This would make it > possible to say > > a, ..., b, ... = some_function() > > i.e. skip exactly one item in unpacking. I thought about that too, but it seemed like it would be too confusing -- the above *looks* like it should be skipping an arbitrary number of items. I think this interpretation would be even more inconsistent with existing uses of ... in both Python and English. -- Greg From greg.ewing at canterbury.ac.nz Sun Feb 14 15:45:06 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 15 Feb 2016 09:45:06 +1300 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> Message-ID: <56C0E752.1060501@canterbury.ac.nz> Georg Brandl wrote: > But thinking about it, this is also legal at the moment: > > [] = [] > > Interestingly, this raises: > > () = () That's weird. I'm guessing it's because () gets treated as a constant in some part of the compilation pipeline, and then the assignment is rejected on the grounds that you can't assign to a constant, -- Greg From oscar.j.benjamin at gmail.com Sun Feb 14 19:30:57 2016 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Mon, 15 Feb 2016 00:30:57 +0000 Subject: [Python-ideas] Dict(x, y) -> Dict(zip(x, y)) Message-ID: I often find myself writing dict(zip(x, y)). Maybe that's just me or maybe not but I would like it if it were possible to spell that simply as dict(x, y). Currently dict() with two arguments is an error: $ python3 Python 3.4.3 (default, Mar 26 2015, 22:03:40) [GCC 4.9.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> dict(['foo', 'bar'], [1, 2]) Traceback (most recent call last): File "", line 1, in TypeError: dict expected at most 1 arguments, got 2 I would prefer it if dict(x, y) would treat x as an iterable of keys and y as an iterable of corresponding values. This would be semantically equivalent to dict(zip(x, y)) except that it would raise an error if x and y don't yield the same number of items (zip truncates the longer argument). Although mostly equivalent the new construction might be more efficient, would reduce the visual noise of the redundant call to zip, and would be slightly more robust to errors. -- Oscar From jared.grubb at gmail.com Sun Feb 14 19:47:20 2016 From: jared.grubb at gmail.com (Jared Grubb) Date: Sun, 14 Feb 2016 16:47:20 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> Message-ID: <8E178DB4-AC3A-472F-931A-09F05B5379E2@gmail.com> > On Feb 14, 2016, at 10:57, Georg Brandl wrote: > > On 02/14/2016 08:43 AM, Nick Coghlan wrote: >> On 14 February 2016 at 07:40, Greg Ewing >> > > wrote: >> >> >> Another possibility is >> >> a, b, ... = value >> >> >> Now, *that* spelling to turn off the "implicit peek" behaviour in iterable >> unpacking I quite like. >> >> arg_iter = iter(args) >> command, ... = arg_iter >> run_command(command, arg_iter) >> > > Assigning to Ellipsis? Interesting idea, but I'd probably go a bit further > in the similarity to star-assignment: > > a, b, *... = value > > Ellipsis could then be a general "throw-away" lvalue. This would make it > possible to say > > a, ..., b, ... = some_function() > > i.e. skip exactly one item in unpacking. That looks harder to read than what you can currently do: a, _, b, _ = some_function() But I do like the asterisk+ellipsis. >> Although again, the main downside would be that "..." here means something >> rather different from what it means as a subscript element. > > Keep in mind that Ellipsis is also a legal rvalue anywhere else. I.e. > this would be legal (and a no-op): > > ... = ... > > But thinking about it, this is also legal at the moment: > > [] = [] > > Interestingly, this raises: > > () = () > > cheers, > Georg > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sun Feb 14 20:23:35 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 15 Feb 2016 10:23:35 +0900 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> Message-ID: <22209.10391.941943.514577@turnbull.sk.tsukuba.ac.jp> Georg Brandl writes: > Assigning to Ellipsis? Interesting idea, but I'd probably go a bit > further in the similarity to star-assignment: > > a, b, *... = value I don't get the "*". Normally that means create or unpack a container, but here the semantics is "leave the container alone". And to my eyes, it's not an assignment to Ellipsis semantically. It's an operator that says "b gets an element, and the tail of value is used somewhere else, don't copy it as a sequence to b". Perhaps a, b ...= consumable_value would make that clearer but it looks strange and ugly to me. What would a, b, ... = some_list mean? (Using the OP's notation without prejudice to other notations.) Would it pop a and b off the list? > Ellipsis could then be a general "throw-away" lvalue. This would make it > possible to say > > a, ..., b, ... = some_function() > > i.e. skip exactly one item in unpacking. That has a rather different meaning in math texts, though. It means "an infinite sequence starting at a with a generic element denoted 'b'". The number of elements between a and b is arbitrary, and typically indicated by writing b as an expression involving an index variable such as i or n. > Keep in mind that Ellipsis is also a legal rvalue anywhere else. That might kill the operator interpretation. From bruce at leban.us Sun Feb 14 21:43:29 2016 From: bruce at leban.us (Bruce Leban) Date: Sun, 14 Feb 2016 18:43:29 -0800 Subject: [Python-ideas] Dict(x, y) -> Dict(zip(x, y)) In-Reply-To: References: Message-ID: this would have: >>> dict(((1,2),(3,4))) {1: 2, 3: 4} >>> dict(((1,2),(3,4)), ((5,6))) {(1, 2): 5, (3, 4): 6} in other words, the interpretation of the first argument to dict is entirely different if there is a second argument than if there isn't one. It's quite reasonable to expect the reverse, namely that the result would be: >>> dict(((1,2),(3,4)), ((5,6))) {1: 2, 3: 4, 5: 6} So adding this saves 5 characters and makes it easier to confuse readers of the code. Doesn't seem like a good idea to me. --- Bruce Check out my puzzle book and get it free here: http://J.mp/ingToConclusionsFree (available on iOS) On Sun, Feb 14, 2016 at 4:30 PM, Oscar Benjamin wrote: > I often find myself writing dict(zip(x, y)). Maybe that's just me or > maybe not but I would like it if it were possible to spell that simply > as dict(x, y). > > Currently dict() with two arguments is an error: > > $ python3 > Python 3.4.3 (default, Mar 26 2015, 22:03:40) > [GCC 4.9.2] on linux > Type "help", "copyright", "credits" or "license" for more information. > >>> dict(['foo', 'bar'], [1, 2]) > Traceback (most recent call last): > File "", line 1, in > TypeError: dict expected at most 1 arguments, got 2 > > I would prefer it if dict(x, y) would treat x as an iterable of keys > and y as an iterable of corresponding values. This would be > semantically equivalent to dict(zip(x, y)) except that it would raise > an error if x and y don't yield the same number of items (zip > truncates the longer argument). > > Although mostly equivalent the new construction might be more > efficient, would reduce the visual noise of the redundant call to zip, > and would be slightly more robust to errors. > > -- > Oscar > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcgoble3 at gmail.com Mon Feb 15 00:48:24 2016 From: jcgoble3 at gmail.com (Jonathan Goble) Date: Mon, 15 Feb 2016 00:48:24 -0500 Subject: [Python-ideas] Exposing regular expression bytecode Message-ID: (This was previously sent to python-dev [1], but it was suggested that I bring it here first.) I filed http://bugs.python.org/issue26336 a few days ago, but now I think this list might be a better place to get discussion going. Basically, I'd like to see the bytecode of a compiled regex object exposed as a public (probably read-only) attribute of the object. Currently, although compiled in pure Python through modules sre_compile and sre_parse, the list of opcodes is then passed into C and copied into an array in a C struct, without being publicly exposed in any way. The only way for a user to get an internal representation of the regex is the re.DEBUG flag, which only produces an intermediate representation rather than the actual bytecode and only goes to stdout, which makes it useless for someone who wants to examine it programmatically. I'm sure others can think of other potential use cases for this, but one in particular would be that someone could write a debugger that can allow a user to step through a regex one opcode at a time to see exactly where it is failing. It would also perhaps be nice to have a public constructor for the regex object type, which would enable users to modify the bytecode and directly create a new regex object from it, similar to what is currently possible for function bytecode through the types.FunctionType and types.CodeType constructors. This would make possible things such as optimizers. In addition to exposing the code in a public attribute, a helper module written in Python similar to the dis module (which is for Python's own bytecode) would be very helpful, allowing the code to be easily disassembled and examined at a higher level. Is this a good idea, or am I barking up the wrong tree? I think it's a great idea, but I'm open to being told this is a horrible idea. :) I welcome any and all comments both here and on the bug tracker. [1] https://mail.python.org/pipermail/python-dev/2016-February/143355.html From abarnert at yahoo.com Mon Feb 15 01:31:31 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 14 Feb 2016 22:31:31 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> Message-ID: <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> On Feb 14, 2016, at 10:57, Georg Brandl wrote: > > Assigning to Ellipsis? Interesting idea, but I'd probably go a bit further > in the similarity to star-assignment: > > a, b, *... = value > > Ellipsis could then be a general "throw-away" lvalue. This would make it > possible to say > > a, ..., b, ... = some_function() > > i.e. skip exactly one item in unpacking. But we already have a general "throw-away": underscore. You can already write "a, _, b, _ = it" and it will unpack 4 values and throw away 2 of them. And you can also write "a, b, *_ = it" to unpack 2 or more values and throw away all but the first 2. And, for that matter, "a, *_, b= it". There's nothing special about the underscore--you could get the same effect by writing "a, dummy, b, evendummier = it"--but it's conventional. Anyway, what people are looking for in this thread is almost the exact opposite: they don't want to unpack the value and throw it away, they want to leave the value there for later unpacking. (Of course for collections, there's no such distinction, but for iterators there is.) >> Although again, the main downside would be that "..." here means something >> rather different from what it means as a subscript element. > > Keep in mind that Ellipsis is also a legal rvalue anywhere else. I.e. > this would be legal (and a no-op): > > ... = ... With your version, where ... is a normal target that just ignores its value, sure. With the original version, where ... means "unpack 0 elements from the iterable and stop", it would presumably raise a TypeError("'ellipsis' object is not iterable"). > But thinking about it, this is also legal at the moment: > > [] = [] Yes, but that's completely different. The [] on the left isn't an expression, or even a target, but a target list with 0 targets in it. Assignment to target lists is defined recursively, so assigning to 0 targets is legal iff you're unpacking 0 values. The fact that you have specifically [] on the right side is irrelevant. You can get the same effect by writing [] = (), or [] = {}, or [] = (i for i in range(5) if i<0). And clearly, you're not assigning to "the empty list", because each empty list created with [] is a distinct object. Anyway, ... is a constant; it's currently illegal as a target because of the rule added in 3.0 banning assignment to a handful of special constants (..., None, etc.). If you have it a special meaning as a target, then of course that rule no longer applies. > Interestingly, this raises: > > () = () This just doesn't parse, because the target-list grammar doesn't have the same special case for () as the parenthesized expression grammar. (Remember, both tuples and target lists are made by commas, not parens, and () really is a special case.) If such a rule were added, then it would presumably mean the same thing as [] = (). But what would be the point of adding that rule? (You could argue that the inconsistency makes the language harder to remember, but it's been this way for decades, and nobody notices it until they've been using Python for years, so that's not very compelling.) From greg.ewing at canterbury.ac.nz Mon Feb 15 02:12:33 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 15 Feb 2016 20:12:33 +1300 Subject: [Python-ideas] Generator unpacking In-Reply-To: <22209.10391.941943.514577@turnbull.sk.tsukuba.ac.jp> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <22209.10391.941943.514577@turnbull.sk.tsukuba.ac.jp> Message-ID: <56C17A61.9050708@canterbury.ac.nz> Stephen J. Turnbull wrote: > What would > > a, b, ... = some_list > > mean? (Using the OP's notation without prejudice to other notations.) > Would it pop a and b off the list? No, it would just assign the first two items of the list to a and b and not care whether there were any more. -- Greg From wes.turner at gmail.com Mon Feb 15 02:26:02 2016 From: wes.turner at gmail.com (Wes Turner) Date: Mon, 15 Feb 2016 01:26:02 -0600 Subject: [Python-ideas] Exposing regular expression bytecode In-Reply-To: References: Message-ID: the new 'regex' module (with a re compatability mode and unicode) may be the place to find/add more debugging syms * | PyPI: https://pypi.python.org/pypi/regex * | Src: https://bitbucket.org/mrabarnett/mrab-regex On Feb 14, 2016 11:49 PM, "Jonathan Goble" wrote: > (This was previously sent to python-dev [1], but it was suggested that > I bring it here first.) > > I filed http://bugs.python.org/issue26336 a few days ago, but now I > think this list might be a better place to get discussion going. > Basically, I'd like to see the bytecode of a compiled regex object > exposed as a public (probably read-only) attribute of the object. > > Currently, although compiled in pure Python through modules > sre_compile and sre_parse, the list of opcodes is then passed into C > and copied into an array in a C struct, without being publicly exposed > in any way. The only way for a user to get an internal representation > of the regex is the re.DEBUG flag, which only produces an intermediate > representation rather than the actual bytecode and only goes to > stdout, which makes it useless for someone who wants to examine it > programmatically. > > I'm sure others can think of other potential use cases for this, but > one in particular would be that someone could write a debugger that > can allow a user to step through a regex one opcode at a time to see > exactly where it is failing. It would also perhaps be nice to have a > public constructor for the regex object type, which would enable users > to modify the bytecode and directly create a new regex object from it, > similar to what is currently possible for function bytecode through > the types.FunctionType and types.CodeType constructors. This would > make possible things such as optimizers. > > In addition to exposing the code in a public attribute, a helper > module written in Python similar to the dis module (which is for > Python's own bytecode) would be very helpful, allowing the code to be > easily disassembled and examined at a higher level. > > Is this a good idea, or am I barking up the wrong tree? I think it's a > great idea, but I'm open to being told this is a horrible idea. :) I > welcome any and all comments both here and on the bug tracker. > > [1] https://mail.python.org/pipermail/python-dev/2016-February/143355.html > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Mon Feb 15 02:30:29 2016 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 15 Feb 2016 08:30:29 +0100 Subject: [Python-ideas] Generator unpacking In-Reply-To: <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> Message-ID: On 02/15/2016 07:31 AM, Andrew Barnert via Python-ideas wrote: > On Feb 14, 2016, at 10:57, Georg Brandl > wrote: >> >> Assigning to Ellipsis? Interesting idea, but I'd probably go a bit >> further in the similarity to star-assignment: >> >> a, b, *... = value >> >> Ellipsis could then be a general "throw-away" lvalue. This would make it >> possible to say >> >> a, ..., b, ... = some_function() >> >> i.e. skip exactly one item in unpacking. > > But we already have a general "throw-away": underscore. You can already write > "a, _, b, _ = it" and it will unpack 4 values and throw away 2 of them. And > you can also write "a, b, *_ = it" to unpack 2 or more values and throw away > all but the first 2. And, for that matter, "a, *_, b= it". I kinda know, having written the PEP and implementation for the latter two. It was just a quick, not very well thought-out idea of how to generalize Nick's interesting suggestion. >> But thinking about it, this is also legal at the moment: >> >> [] = [] > > Yes, but that's completely different. The [] on the left isn't an expression, > or even a target, but a target list with 0 targets in it. Assignment to > target lists is defined recursively, so assigning to 0 targets is legal iff > you're unpacking 0 values. > > The fact that you have specifically [] on the right side is irrelevant. You > can get the same effect by writing [] = (), or [] = {}, or [] = (i for i in > range(5) if i<0). And clearly, you're not assigning to "the empty list", > because each empty list created with [] is a distinct object. Yes, what you're saying can be expressed as "of course it's legal, since it's legal". Please step back a bit and look at this from the "how it looks" perspective, not from the "what the grammar rules" perspective. My observation was simple that we already have the case of X = X where both Xes are the same syntax, but have a wildly differing interpretation. > Anyway, ... is a constant; it's currently illegal as a target because of the > rule added in 3.0 banning assignment to a handful of special constants (..., > None, etc.). If you have it a special meaning as a target, then of course > that rule no longer applies. > >> Interestingly, this raises: >> >> () = () > > This just doesn't parse, because the target-list grammar doesn't have the > same special case for () as the parenthesized expression grammar. (Remember, > both tuples and target lists are made by commas, not parens, and () really is > a special case.) > > If such a rule were added, then it would presumably mean the same thing as [] > = (). But what would be the point of adding that rule? (You could argue that > the inconsistency makes the language harder to remember, but it's been this > way for decades, and nobody notices it until they've been using Python for > years, so that's not very compelling.) > > _______________________________________________ Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: > http://python.org/psf/codeofconduct/ > From greg.ewing at canterbury.ac.nz Mon Feb 15 02:33:30 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 15 Feb 2016 20:33:30 +1300 Subject: [Python-ideas] Generator unpacking In-Reply-To: <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> Message-ID: <56C17F4A.3020607@canterbury.ac.nz> Andrew Barnert via Python-ideas wrote: > On Feb 14, 2016, at 10:57, Georg Brandl wrote: > >> ... = ... > > With the original version, where ... means "unpack 0 elements from the > iterable and stop", it would presumably raise a TypeError("'ellipsis' object > is not iterable"). Curious about what would happen, I tried >>> [] = 42 Traceback (most recent call last): File "", line 1, in TypeError: 'int' object is not iterable So it appears that [] = something is a cute way of asserting that the rhs is iterable without consuming anything from it! -- Greg From jcgoble3 at gmail.com Mon Feb 15 02:47:28 2016 From: jcgoble3 at gmail.com (Jonathan Goble) Date: Mon, 15 Feb 2016 02:47:28 -0500 Subject: [Python-ideas] Exposing regular expression bytecode In-Reply-To: References: Message-ID: On Mon, Feb 15, 2016 at 2:26 AM, Wes Turner wrote: > the new 'regex' module (with a re compatability mode and unicode) may be the > place to find/add more debugging syms Thanks, but I'm already well aware of that module, as I'm already using it for another project of mine that requires some of its new features. But this proposal, I think, is simple enough that it would be worth doing for the built-in 're' modules. Also, the things that implementing it would enable, such as optimizers and especially debuggers, would be available to a much larger number of people if the built-in module supported them. I'm not really asking for much; just take an array already in the struct that represents the object, expose it as a public attribute, and write a simple 'dis'-like module to enable low-level programmatic analysis. Once that's available, third parties are likely to build on it from there; that's much less likely to happen if such tools only worked on a different regex module (albeit a great one) from PyPI and not on the built-in version. From p.f.moore at gmail.com Mon Feb 15 03:12:16 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 15 Feb 2016 08:12:16 +0000 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> Message-ID: On 14 February 2016 at 07:43, Nick Coghlan wrote: > On 14 February 2016 at 07:40, Greg Ewing > wrote: >> >> >> Another possibility is >> >> a, b, ... = value > > > Now, *that* spelling to turn off the "implicit peek" behaviour in iterable > unpacking I quite like. > > arg_iter = iter(args) > command, ... = arg_iter > run_command(command, arg_iter) > > Although again, the main downside would be that "..." here means something > rather different from what it means as a subscript element. IMO, the other downside is that the semantic difference between a, b, ... = value and a, b, *_ = value is very subtle, and (even worse) only significant if value is an iterable as opposed to a concrete container such as a list. IMO, explicit is better than implicit here, and itertools.islice is the right way to go: >>> i = iter(range(10)) >>> a, b = islice(i,2) >>> a 0 >>> b 1 >>> list(i) [2, 3, 4, 5, 6, 7, 8, 9] (And of course in my first attempt I forgot I needed iter(range(...)) as range is not a pure iterable - proving my point about the subtle semantics!) Paul From abarnert at yahoo.com Mon Feb 15 03:24:17 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 15 Feb 2016 00:24:17 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> Message-ID: <8D51FAAA-73E0-436E-8808-8F990E70DD40@yahoo.com> On Feb 14, 2016, at 23:30, Georg Brandl wrote: > >> On 02/15/2016 07:31 AM, Andrew Barnert via Python-ideas wrote: >> On Feb 14, 2016, at 10:57, Georg Brandl >> wrote: > >>> But thinking about it, this is also legal at the moment: >>> >>> [] = [] >> >> Yes, but that's completely different. The [] on the left isn't an expression, >> or even a target, but a target list with 0 targets in it. Assignment to >> target lists is defined recursively, so assigning to 0 targets is legal iff >> you're unpacking 0 values. >> >> The fact that you have specifically [] on the right side is irrelevant. You >> can get the same effect by writing [] = (), or [] = {}, or [] = (i for i in >> range(5) if i<0). And clearly, you're not assigning to "the empty list", >> because each empty list created with [] is a distinct object. > > Yes, what you're saying can be expressed as "of course it's legal, since it's > legal". No; common sense isn't the same thing as tautology. It has an obvious semantics, there's no good reason to ban it, and it comes for free with the simplest grammar--therefore, it's common sense that the language should allow it. It's only because Python generally does such a great job following common sense that you don't notice. :) And it's similar common sense that your version of "..." should make "... = ..." legal, while Nick's version should make it illegal. > Please step back a bit and look at this from the "how it looks" perspective, > not from the "what the grammar rules" perspective. My observation was simple > that we already have the case of > > X = X > > where both Xes are the same syntax, but have a wildly differing interpretation. x = x" is a no-op (assuming x was already bound) and "False = False" is an error. Given that, and the fact that (unlike those cases) "[] = []" _doesn't_ have the same syntax for the two Xes, I don't see what the observation demonstrates. From lkb.teichmann at gmail.com Mon Feb 15 03:31:07 2016 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Mon, 15 Feb 2016 09:31:07 +0100 Subject: [Python-ideas] Simpler Customization of Class Creation - PEP 487 In-Reply-To: References: Message-ID: Hi List, I thought it was a good idea to show that projects can benefit from PEP 487 out there in the real world. As a showcase example I chose IPython traitlets, and it was a nice success. With only little modifications to the traitlets code it uses a PEP 487 style metaclass, and all tests pass unmodified with both Python 2 and 3. You can find the PEP 487 style IPython traitlets here: https://github.com/tecki/traitlets/tree/pep487 Greetings Martin From g.brandl at gmx.net Mon Feb 15 03:34:07 2016 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 15 Feb 2016 09:34:07 +0100 Subject: [Python-ideas] Generator unpacking In-Reply-To: <8D51FAAA-73E0-436E-8808-8F990E70DD40@yahoo.com> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <8D51FAAA-73E0-436E-8808-8F990E70DD40@yahoo.com> Message-ID: On 02/15/2016 09:24 AM, Andrew Barnert via Python-ideas wrote: > On Feb 14, 2016, at 23:30, Georg Brandl > wrote: >> >>> On 02/15/2016 07:31 AM, Andrew Barnert via Python-ideas wrote: On Feb 14, >>> 2016, at 10:57, Georg Brandl >>> wrote: >> >>>> But thinking about it, this is also legal at the moment: >>>> >>>> [] = [] >>> >>> Yes, but that's completely different. The [] on the left isn't an >>> expression, or even a target, but a target list with 0 targets in it. >>> Assignment to target lists is defined recursively, so assigning to 0 >>> targets is legal iff you're unpacking 0 values. >>> >>> The fact that you have specifically [] on the right side is irrelevant. >>> You can get the same effect by writing [] = (), or [] = {}, or [] = (i >>> for i in range(5) if i<0). And clearly, you're not assigning to "the >>> empty list", because each empty list created with [] is a distinct >>> object. >> >> Yes, what you're saying can be expressed as "of course it's legal, since >> it's legal". > > No; common sense isn't the same thing as tautology. It has an obvious > semantics, there's no good reason to ban it, and it comes for free with the > simplest grammar--therefore, it's common sense that the language should allow > it. It's only because Python generally does such a great job following common > sense that you don't notice. :) > > And it's similar common sense that your version of "..." should make "... = > ..." legal, while Nick's version should make it illegal. Well, that's all I wanted to express. If what was noteworthy to me was all just common sense to you, so much the better. :) Keep in mind that it might not be to everyone. Georg From abarnert at yahoo.com Mon Feb 15 03:39:58 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 15 Feb 2016 00:39:58 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> Message-ID: <40592F59-371A-4440-B58E-7EC83DE54713@yahoo.com> On Feb 15, 2016, at 00:12, Paul Moore wrote: > > IMO, the other downside is that the semantic difference between > > a, b, ... = value > > and > > a, b, *_ = value > > is very subtle, and (even worse) only significant if value is an > iterable as opposed to a concrete container such as a list. You mean iterator, not iterable. And being "concrete" has nothing to do with it--a dict view, a memoryview, a NumPy slice, etc. aren't iterators any more than a list is. This is exactly why I think we need an official term like "collection" for "iterables that are not iterators" (or "iterables whose __iter__ doesn't return self" or similar). People struggling to come up with terms end up confusing themselves--not just about wording, but about actual concepts. As proven below: > (And of course in my first attempt I forgot I needed iter(range(...)) > as range is not a pure iterable - proving my point about the subtle > semantics!) Ranges are as iterable as both lists and iterators. You're a smart guy, and you know Python. So why do you make this mistake? Because you don't have a term to fit "range" into, so your brain struggles between two prototypes--is it like a list, or like a generator? Well, it's lazy, so it's like a generator, so you don't need to call iter here, right? Nope. This is the same as the experiment where they give people the English cousin rule and asked them to evaluate which family relationships count as cousins. People whose native language has no word for "cousin"--e.g., because they have unrelated words for "maternal cousin" and "paternal cousin"--make a lot more mistakes than people whose language has a word that matches the rule. From p.f.moore at gmail.com Mon Feb 15 03:49:33 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 15 Feb 2016 08:49:33 +0000 Subject: [Python-ideas] Generator unpacking In-Reply-To: <40592F59-371A-4440-B58E-7EC83DE54713@yahoo.com> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <40592F59-371A-4440-B58E-7EC83DE54713@yahoo.com> Message-ID: On 15 February 2016 at 08:39, Andrew Barnert wrote: >> is very subtle, and (even worse) only significant if value is an >> iterable as opposed to a concrete container such as a list. > > You mean iterator, not iterable. And being "concrete" has nothing to do with it--a dict view, a memoryview, a NumPy slice, etc. aren't iterators any more than a list is. Precisely :-) Nor is a range object, see the (other!) mistake I made. > > This is exactly why I think we need an official term like "collection" for "iterables that are not iterators" (or "iterables whose __iter__ doesn't return self" or similar). People struggling to come up with terms end up confusing themselves--not just about wording, but about actual concepts. As proven below: Indeed. >> (And of course in my first attempt I forgot I needed iter(range(...)) >> as range is not a pure iterable - proving my point about the subtle >> semantics!) > > Ranges are as iterable as both lists and iterators. You're a smart guy, and you know Python. So why do you make this mistake? Because you don't have a term to fit "range" into, so your brain struggles between two prototypes--is it like a list, or like a generator? Well, it's lazy, so it's like a generator, so you don't need to call iter here, right? Nope. Thanks for providing another perspective on my point. This whole area is rather too full of semantic confusion (I'm not as sure as you seem to be that agreeing on a terminology will address that issue, but that's a side matter) and I think we should be very careful about introducing syntax that requires a good understanding of the concepts to use correctly, when there's a perfectly acceptable explicit form (even if it is slightly more verbose/repetitive). Paul From rosuav at gmail.com Mon Feb 15 03:50:57 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 15 Feb 2016 19:50:57 +1100 Subject: [Python-ideas] Generator unpacking In-Reply-To: <56C17F4A.3020607@canterbury.ac.nz> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C17F4A.3020607@canterbury.ac.nz> Message-ID: On Mon, Feb 15, 2016 at 6:33 PM, Greg Ewing wrote: > Curious about what would happen, I tried > >>>> [] = 42 > Traceback (most recent call last): > File "", line 1, in > TypeError: 'int' object is not iterable > > So it appears that [] = something is a cute way of asserting > that the rhs is iterable without consuming anything from it! Yeah... uhh... "iter(x)" is a less cute way of asserting that it's iterable :D ChrisA From abarnert at yahoo.com Mon Feb 15 03:54:03 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 15 Feb 2016 00:54:03 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C17F4A.3020607@canterbury.ac.nz> Message-ID: <50CA8BDA-0888-422D-9BF2-9A2766D44B15@yahoo.com> On Feb 15, 2016, at 00:50, Chris Angelico wrote: > >> On Mon, Feb 15, 2016 at 6:33 PM, Greg Ewing wrote: >> Curious about what would happen, I tried >> >>>>> [] = 42 >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: 'int' object is not iterable >> >> So it appears that [] = something is a cute way of asserting >> that the rhs is iterable without consuming anything from it! > > Yeah... uhh... "iter(x)" is a less cute way of asserting that it's iterable :D With the side benefit of not also asserting that it's empty, and not possibly consuming elements. :) From rosuav at gmail.com Mon Feb 15 03:54:14 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 15 Feb 2016 19:54:14 +1100 Subject: [Python-ideas] Generator unpacking In-Reply-To: <40592F59-371A-4440-B58E-7EC83DE54713@yahoo.com> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <40592F59-371A-4440-B58E-7EC83DE54713@yahoo.com> Message-ID: On Mon, Feb 15, 2016 at 7:39 PM, Andrew Barnert via Python-ideas wrote: > On Feb 15, 2016, at 00:12, Paul Moore wrote: >> >> IMO, the other downside is that the semantic difference between >> >> a, b, ... = value >> >> and >> >> a, b, *_ = value >> >> is very subtle, and (even worse) only significant if value is an >> iterable as opposed to a concrete container such as a list. > > You mean iterator, not iterable. And being "concrete" has nothing to do with it--a dict view, a memoryview, a NumPy slice, etc. aren't iterators any more than a list is. > Allow me to take this opportunity to reiterate the recommendation for "reiterable", which has come up enough times that I've completely forgotten who first came up with it. An iterable can be iterated over at least once; a reiterable can be iterated over more than once, and will often produce the same sequence of values each time. ChrisA From leewangzhong+python at gmail.com Mon Feb 15 04:44:13 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Mon, 15 Feb 2016 04:44:13 -0500 Subject: [Python-ideas] Dict(x, y) -> Dict(zip(x, y)) In-Reply-To: References: Message-ID: Does it make code more readable? What about simply defining `dictzip` or `dict2` for yourself? On Feb 14, 2016 9:44 PM, "Bruce Leban" wrote: > > this would have: > > >>> dict(((1,2),(3,4))) > {1: 2, 3: 4} > > >>> dict(((1,2),(3,4)), ((5,6))) > {(1, 2): 5, (3, 4): 6} > > in other words, the interpretation of the first argument to dict is entirely different if there is a second argument than if there isn't one. There is precedent for that in ``range``. (Sorry for the double mail, Bruce.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Feb 15 04:53:20 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 15 Feb 2016 01:53:20 -0800 Subject: [Python-ideas] Dict(x, y) -> Dict(zip(x, y)) In-Reply-To: References: Message-ID: On Feb 14, 2016, at 18:43, Bruce Leban wrote: > > this would have: > > >>> dict(((1,2),(3,4))) > {1: 2, 3: 4} > > >>> dict(((1,2),(3,4)), ((5,6))) > {(1, 2): 5, (3, 4): 6} > > in other words, the interpretation of the first argument to dict is entirely different if there is a second argument than if there isn't one. It would be nice if we could make Oscar's new parameters keyword-only: dict(keys=spam, values=eggs) Then it's _not_ the first argument that's being interpreted differently. Unfortunately, we can't do this, because it would be ambiguous with the dict(a=1, b=2) constructor. But what about a classmethod named constructor (with or without keyword-only params, depending on how long the name is): dict.kv(keys=spam, values=eggs) dict.from_keys_values(spam, eggs) That's pretty close to the idiomatic way to build dictionaries in ObjectiveC: [NSDict dictionaryWithValues:eggs forKeys:spam] ... so it's not totally unprecedented. Still, it raises the obvious question of whether "dict.kv" is much better than "dictkv", which anyone can write for himself as a one-liner if he wants it... -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Mon Feb 15 07:53:08 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Mon, 15 Feb 2016 13:53:08 +0100 Subject: [Python-ideas] Simpler Customization of Class Creation - PEP 487 In-Reply-To: References: Message-ID: <56C1CA34.7020502@mail.de> Nice. I am glad to see movement and practical usage in that area. Metaclasses have been a headache for a long time. We introduced them into our production for a short period of time just to remove them later. As the PEP expected, ABCMeta was the issue. We will see if the proposal will solve that. On 15.02.2016 09:31, Martin Teichmann wrote: > Hi List, > > I thought it was a good idea to show that projects can benefit from PEP 487 out > there in the real world. As a showcase example I chose IPython traitlets, and it > was a nice success. With only little modifications to the traitlets code it uses > a PEP 487 style metaclass, and all tests pass unmodified with both Python 2 > and 3. You can find the PEP 487 style IPython traitlets here: > https://github.com/tecki/traitlets/tree/pep487 > > Greetings > > Martin > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From srkunze at mail.de Mon Feb 15 08:55:26 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Mon, 15 Feb 2016 14:55:26 +0100 Subject: [Python-ideas] Generator unpacking In-Reply-To: <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> Message-ID: <56C1D8CE.4020709@mail.de> On 15.02.2016 07:31, Andrew Barnert via Python-ideas wrote: > On Feb 14, 2016, at 10:57, Georg Brandl wrote: >> Assigning to Ellipsis? Interesting idea, but I'd probably go a bit further >> in the similarity to star-assignment: >> >> a, b, *... = value >> >> Ellipsis could then be a general "throw-away" lvalue. This would make it >> possible to say >> >> a, ..., b, ... = some_function() >> >> i.e. skip exactly one item in unpacking. > But we already have a general "throw-away": underscore. You can already write "a, _, b, _ = it" and it will unpack 4 values and throw away 2 of them. And you can also write "a, b, *_ = it" to unpack 2 or more values and throw away all but the first 2. And, for that matter, "a, *_, b= it". > > There's nothing special about the underscore--you could get the same effect by writing "a, dummy, b, evendummier = it"--but it's conventional. I agree. >> But thinking about it, this is also legal at the moment: >> >> [] = [] > Yes, but that's completely different. The [] on the left isn't an expression, or even a target, but a target list with 0 targets in it. Assignment to target lists is defined recursively, so assigning to 0 targets is legal iff you're unpacking 0 values. > > The fact that you have specifically [] on the right side is irrelevant. You can get the same effect by writing [] = (), or [] = {}, or [] = (i for i in range(5) if i<0). And clearly, you're not assigning to "the empty list", because each empty list created with [] is a distinct object. > > Anyway, ... is a constant; it's currently illegal as a target because of the rule added in 3.0 banning assignment to a handful of special constants (..., None, etc.). If you have it a special meaning as a target, then of course that rule no longer applies. > >> Interestingly, this raises: >> >> () = () > This just doesn't parse, because the target-list grammar doesn't have the same special case for () as the parenthesized expression grammar. (Remember, both tuples and target lists are made by commas, not parens, and () really is a special case.) > > If such a rule were added, then it would presumably mean the same thing as [] = (). But what would be the point of adding that rule? (You could argue that the inconsistency makes the language harder to remember, but it's been this way for decades, and nobody notices it until they've been using Python for years, so that's not very compelling.) Interestingly, doing the following results in an syntax error. >>> [1,2]=[3,4] File "", line 1 SyntaxError: can't assign to literal So, if it's illegal to assign to the literal [1,2], I don't see why it should be legal for []. But as you said, that's a highly theoretical problem. That just reminds me of the js part of https://www.destroyallsoftware.com/talks/wat Best, Sven From steve at pearwood.info Mon Feb 15 09:42:52 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 16 Feb 2016 01:42:52 +1100 Subject: [Python-ideas] Generator unpacking In-Reply-To: <56C1D8CE.4020709@mail.de> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> Message-ID: <20160215144251.GY31806@ando.pearwood.info> On Mon, Feb 15, 2016 at 02:55:26PM +0100, Sven R. Kunze wrote: > On 15.02.2016 07:31, Andrew Barnert via Python-ideas wrote: > >> [] = [] > > > >Yes, but that's completely different. The [] on the left isn't an > >expression, or even a target, but a target list with 0 targets in it. [..] > Interestingly, doing the following results in an syntax error. > > >>> [1,2]=[3,4] > > File "", line 1 > SyntaxError: can't assign to literal > > > So, if it's illegal to assign to the literal [1,2], I don't see why it > should be legal for []. But as you said, that's a highly theoretical > problem. I believe you are misinterpreting the error. The error isn't that you are trying to assign to the literal [1,2]. The error is that you are trying to assign to the literal 1. It's a subtle difference, but important to understand that difference in order to understand why [] is a legal assignment target. py> [a, b, c, 1] = "abcd" File "", line 1 SyntaxError: can't assign to literal Obviously [a, b, c, 1] is not a literal, but 1 is. If there is any doubt: py> [a, b, c, None] = "abcd" File "", line 1 SyntaxError: cannot assign to None Since [1,2] is a *list of assignment targets*, not a single target, the assignment you attempted [1, 2] = [3, 4] is roughly equivalent to: 1 = 3 2 = 4 which obviously gives a syntax error. [] is a valid assignment target so long as the right hand side is an empty iterable. If it is not empty, you get a TypeError: py> [] = "abc" Traceback (most recent call last): File "", line 1, in ValueError: too many values to unpack There are three values on the right hand side, and zero targets. This might even be useful. You can confirm that an iterable is empty (why you might want to do this, I can't imagine, but suppose you did) by assigning it to zero targets: [] = iterable succeeds only if iterable has zero items, otherwise it raises TypeError. -- Steve From srkunze at mail.de Mon Feb 15 10:00:44 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Mon, 15 Feb 2016 16:00:44 +0100 Subject: [Python-ideas] Incorporating something like byteplay into the stdlib In-Reply-To: <56BE62FB.5090308@stoneleaf.us> References: <1465993608.2439410.1455249525441.JavaMail.yahoo.ref@mail.yahoo.com> <1465993608.2439410.1455249525441.JavaMail.yahoo@mail.yahoo.com> <56BE4244.6060504@gmail.com> <34117C96-D1E9-4A73-B3EB-B670C152C2BE@yahoo.com> <56BE5284.1010204@gmail.com> <56BE5543.7000302@gmail.com> <56BE62FB.5090308@stoneleaf.us> Message-ID: <56C1E81C.7040602@mail.de> On 12.02.2016 23:55, Ethan Furman wrote: > On 02/12/2016 02:38 PM, Andrew Barnert via Python-ideas wrote: > > > My proposal is that if we add bytecode optimizers, we should make it > > possible to write them. > > +1 +1 from me. Despite all of Yury's concerns, I find that step logical and not overly problematic. Yury seems to be anxious about bytecode optimizers being used as one of the first-used features. However, without some proper docs this kind of feature is useless anyway. Thus, the docs just need to make is explicit what use-cases are supported (research, experimentation, no production, etc.). Furthermore, after you've gained experience with Python, you already know (aka feel) that you cannot solve most of your (production) problems by tweaking bytecode. I, moreover, don't agree with Yury about the number of potential optimizers. Sure, in the first years there will only be a few but long-term the Python ecosystem can only benefit from competition among bytecode optimizers. There are a lot of smart people out there who have some time to spare (as can be seen from the current dynamic debate about optimizing CPython). Best, Sven From srkunze at mail.de Mon Feb 15 10:13:17 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Mon, 15 Feb 2016 16:13:17 +0100 Subject: [Python-ideas] Generator unpacking In-Reply-To: <20160215144251.GY31806@ando.pearwood.info> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> <20160215144251.GY31806@ando.pearwood.info> Message-ID: <56C1EB0D.70505@mail.de> On 15.02.2016 15:42, Steven D'Aprano wrote: > I believe you are misinterpreting the error. The error isn't that you > are trying to assign to the literal [1,2]. The error is that you are > trying to assign to the literal 1. It's a subtle difference, You are indeed right. And while we are on it, that reminds me of KeyError. I always feel annoyed by the fact that Python doesn't tell me what key a dict is missing. So, if I were to make a suggestion, I would like to see the issue-causing thing to be mentioned in those two types of exceptions. > but > important to understand that difference in order to understand why [] is > a legal assignment target. > > py> [a, b, c, 1] = "abcd" > File "", line 1 > SyntaxError: can't assign to literal > > Obviously [a, b, c, 1] is not a literal, but 1 is. If there is any > doubt: > > py> [a, b, c, None] = "abcd" > File "", line 1 > SyntaxError: cannot assign to None > > > Since [1,2] is a *list of assignment targets*, not a single target, the > assignment you attempted > > [1, 2] = [3, 4] > > is roughly equivalent to: > > 1 = 3 > 2 = 4 > > which obviously gives a syntax error. > > [] is a valid assignment target so long as the right hand side is an > empty iterable. If it is not empty, you get a TypeError: > > py> [] = "abc" > Traceback (most recent call last): > File "", line 1, in > ValueError: too many values to unpack > > > There are three values on the right hand side, and zero targets. > > This might even be useful. You can confirm that an iterable is empty > (why you might want to do this, I can't imagine, but suppose you did) by > assigning it to zero targets: > > [] = iterable > > succeeds only if iterable has zero items, otherwise it raises > TypeError. Completely true and nothing to add; except, as Georg already noted, the "as it looks" experience is somewhat weird and I always would consider this a syntax error (don't ask me why). Best, Sven From ian.g.kelly at gmail.com Mon Feb 15 10:59:25 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Mon, 15 Feb 2016 08:59:25 -0700 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <40592F59-371A-4440-B58E-7EC83DE54713@yahoo.com> Message-ID: On Feb 15, 2016 1:55 AM, "Chris Angelico" wrote: > Allow me to take this opportunity to reiterate the recommendation for > "reiterable", which has come up enough times that I've completely > forgotten who first came up with it. An iterable can be iterated over > at least once; a reiterable can be iterated over more than once, and > will often produce the same sequence of values each time. Iterators can be iterated over more than once. They're just required to be empty on the second and subsequent iterations. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Feb 15 12:39:34 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 15 Feb 2016 09:39:34 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: <20160215144251.GY31806@ando.pearwood.info> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> <20160215144251.GY31806@ando.pearwood.info> Message-ID: <56C20D56.2030605@stoneleaf.us> On 02/15/2016 06:42 AM, Steven D'Aprano wrote: > py> [] = "abc" > Traceback (most recent call last): > File "", line 1, in > ValueError: too many values to unpack > > > There are three values on the right hand side, and zero targets. > > This might even be useful. You can confirm that an iterable is empty > (why you might want to do this, I can't imagine, but suppose you did) by > assigning it to zero targets: > > [] = iterable > > succeeds only if iterable has zero items, otherwise it raises > TypeError. I /think/ I've used that trick to confirm an iterable was empty; I /know/ I've used [some_var] = some_var to extract the only item in a one-item list (and verify it had, in fact, only one item). -- ~Ethan~ From srkunze at mail.de Mon Feb 15 12:53:45 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Mon, 15 Feb 2016 18:53:45 +0100 Subject: [Python-ideas] Generator unpacking In-Reply-To: <56C20D56.2030605@stoneleaf.us> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> <20160215144251.GY31806@ando.pearwood.info> <56C20D56.2030605@stoneleaf.us> Message-ID: <56C210A9.4020002@mail.de> On 15.02.2016 18:39, Ethan Furman wrote: > I /think/ I've used that trick to confirm an iterable was empty; I > /know/ I've used > > [some_var] = some_var Is that an unknown Python idiom? It looks short but still insane. Best, Sven From jcgoble3 at gmail.com Mon Feb 15 13:00:27 2016 From: jcgoble3 at gmail.com (Jonathan Goble) Date: Mon, 15 Feb 2016 13:00:27 -0500 Subject: [Python-ideas] Generator unpacking In-Reply-To: <56C210A9.4020002@mail.de> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> <20160215144251.GY31806@ando.pearwood.info> <56C20D56.2030605@stoneleaf.us> <56C210A9.4020002@mail.de> Message-ID: On Mon, Feb 15, 2016 at 12:53 PM, Sven R. Kunze wrote: > On 15.02.2016 18:39, Ethan Furman wrote: >> >> I /think/ I've used that trick to confirm an iterable was empty; I /know/ >> I've used >> >> [some_var] = some_var > > > Is that an unknown Python idiom? > > It looks short but still insane. It's actually a useful way to quickly unpack nested lists and tuples. Here's a quick example: >>> x = [] >>> x.append((1, 2)) >>> x.append(3) >>> x.append((4, 5)) >>> x.append([6, 7, 8]) >>> x [(1, 2), 3, (4, 5), [6, 7, 8]] >>> [(a, b), c, (d, e), [f, g, h]] = x >>> a 1 >>> b 2 >>> c 3 >>> d 4 >>> e 5 >>> f 6 >>> g 7 >>> h 8 From greg.ewing at canterbury.ac.nz Mon Feb 15 15:11:55 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 16 Feb 2016 09:11:55 +1300 Subject: [Python-ideas] Generator unpacking In-Reply-To: <50CA8BDA-0888-422D-9BF2-9A2766D44B15@yahoo.com> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C17F4A.3020607@canterbury.ac.nz> <50CA8BDA-0888-422D-9BF2-9A2766D44B15@yahoo.com> Message-ID: <56C2310B.7090802@canterbury.ac.nz> Andrew Barnert via Python-ideas wrote: > On Feb 15, 2016, at 00:50, Chris Angelico wrote: > >>>Yeah... uhh... "iter(x)" is a less cute way of asserting that it's iterable :D > > With the side benefit of not also asserting that it's empty, and not possibly consuming elements. :) Er, yeah, there is that. :-) But with the proposed syntax it could be [...] = something assuming the compiler doesn't decide to optimize out the iter() in that case. -- Greg From greg.ewing at canterbury.ac.nz Mon Feb 15 15:23:03 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 16 Feb 2016 09:23:03 +1300 Subject: [Python-ideas] Generator unpacking In-Reply-To: <56C1D8CE.4020709@mail.de> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> Message-ID: <56C233A7.4040707@canterbury.ac.nz> Sven R. Kunze wrote: > So, if it's illegal to assign to the literal [1,2], I don't see why it > should be legal for []. But as you said, that's a highly theoretical > problem. But [1,2] is not a literal -- the individual elements 1 and 2 are. It's the inability to assign to those elements that makes it illegal. On the other hand, [] doesn't contain any elements that it's illegal to assign to, so there's no reason to reject it. But it doesn't contain any elements that it's legal to assign to either, so you could say there's no reason to accept it. This is a philosophical question. When you've eaten the last chocolate, do you have an empty box of chocolates, or just an empty box? -- Greg From brenbarn at brenbarn.net Mon Feb 15 04:21:01 2016 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Mon, 15 Feb 2016 01:21:01 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: <40592F59-371A-4440-B58E-7EC83DE54713@yahoo.com> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <40592F59-371A-4440-B58E-7EC83DE54713@yahoo.com> Message-ID: <56C1987D.7050209@brenbarn.net> On 2016-02-15 00:39, Andrew Barnert via Python-ideas wrote: > On Feb 15, 2016, at 00:12, Paul Moore wrote: >> >> IMO, the other downside is that the semantic difference between >> >> a, b, ... = value >> >> and >> >> a, b, *_ = value >> >> is very subtle, and (even worse) only significant if value is an >> iterable as opposed to a concrete container such as a list. > > You mean iterator, not iterable. And being "concrete" has nothing to > do with it--a dict view, a memoryview, a NumPy slice, etc. aren't > iterators any more than a list is. > > This is exactly why I think we need an official term like > "collection" for "iterables that are not iterators" (or "iterables > whose __iter__ doesn't return self" or similar). People struggling to > come up with terms end up confusing themselves--not just about > wording, but about actual concepts. As proven below: I still don't think that is at all what we need, as this example shows. Whether the value is an iterable or an iterator is not relevant. Whether the iterable's iterator is self is not relevant. What is relevant is the difference in *behavior* --- namely, whether you can rewind, restart, or otherwise retrieve already-obtained values from the object, or whether advancing it is an irreversible operation and there is no way to get old values except by storing them yourself. In this example, being able to say that value was or was not an iterator or an iterable would in no way help to clarify how the code would behave differently. Saying that it is an iterable or an iterator is just saying that it has or doesn't have .next() and/or .__iter__() methods that follow certain very broad protocols, but what matters for understanding examples like this is what those methods actually DO. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From abarnert at yahoo.com Mon Feb 15 16:55:32 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 15 Feb 2016 13:55:32 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: <56C1EB0D.70505@mail.de> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> <20160215144251.GY31806@ando.pearwood.info> <56C1EB0D.70505@mail.de> Message-ID: On Feb 15, 2016, at 07:13, Sven R. Kunze wrote: > >> On 15.02.2016 15:42, Steven D'Aprano wrote: >> I believe you are misinterpreting the error. The error isn't that you >> are trying to assign to the literal [1,2]. The error is that you are >> trying to assign to the literal 1. It's a subtle difference, > > You are indeed right. > > And while we are on it, that reminds me of KeyError. I always feel annoyed by the fact that Python doesn't tell me what key a dict is missing. > > So, if I were to make a suggestion, I would like to see the issue-causing thing to be mentioned in those two types of exceptions. IIRC, there's a series of bugs on adding more useful information to the builtin exceptions, which includes putting the key in the error message _and_ as an attribute of the exception (a la filename in the filesystem exceptions), but they've been waiting on someone to do the work for a few years now. If so, there's an obvious way to kick-start the solution: find the bug and write a patch. (I suspect some people would object to fixing KeyError without also fixing IndexError, and want to argue about whether the key member goes in each exception directly or is part of LookupError, and so on, and there's the more substantive issue of keeping alive potentially-large keys, and so on, so it probably won't be as simple as "submit a trivial patch and it gets accepted", but at least having a patch would get the process moving.) But for SyntaxError, there is no object to stick in the error object or error message, just a source token or an AST node. Both of those have pointers back to start and end in the source string, and that's what goes into the SyntaxError, which is already shown to the user by printing the relevant line of source and pointing a caret at it. When that's insufficient or misleading (as with the ever-popular missed-')' errors), printing the token or AST node itself would probably just make it more misleading. And, as for putting it in the error object, it's pretty rare that anyone handles SyntaxError in code (unlike KeyError, FileNotFoundError, etc.), so that seems a lot less useful. Maybe there is a clean way to improve SyntaxError akin to KeyError, but it would take a lot more thought and design (and probably argument), so I'd work on KeyError first. From srkunze at mail.de Mon Feb 15 17:11:24 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Mon, 15 Feb 2016 23:11:24 +0100 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> <20160215144251.GY31806@ando.pearwood.info> <56C20D56.2030605@stoneleaf.us> <56C210A9.4020002@mail.de> Message-ID: <56C24D0C.5080905@mail.de> On 15.02.2016 19:00, Jonathan Goble wrote: > On Mon, Feb 15, 2016 at 12:53 PM, Sven R. Kunze wrote: >> On 15.02.2016 18:39, Ethan Furman wrote: >>> I /think/ I've used that trick to confirm an iterable was empty; I /know/ >>> I've used >>> >>> [some_var] = some_var >> >> Is that an unknown Python idiom? >> >> It looks short but still insane. > It's actually a useful way to quickly unpack nested lists and tuples. > Here's a quick example: > > [snip] I am aware of (nested) unpacking in general. :) The use-case of checking if a list is has only a single element and to unpack it in a single statement has simply never occurred to me. Best, Sven From srkunze at mail.de Mon Feb 15 17:23:39 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Mon, 15 Feb 2016 23:23:39 +0100 Subject: [Python-ideas] Generator unpacking In-Reply-To: <56C24D0C.5080905@mail.de> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> <20160215144251.GY31806@ando.pearwood.info> <56C20D56.2030605@stoneleaf.us> <56C210A9.4020002@mail.de> <56C24D0C.5080905@mail.de> Message-ID: <56C24FEB.10103@mail.de> On 15.02.2016 23:11, Sven R. Kunze wrote: > On 15.02.2016 19:00, Jonathan Goble wrote: >> On Mon, Feb 15, 2016 at 12:53 PM, Sven R. Kunze wrote: >>> On 15.02.2016 18:39, Ethan Furman wrote: >>>> I /think/ I've used that trick to confirm an iterable was empty; I >>>> /know/ >>>> I've used >>>> >>>> [some_var] = some_var >>> >>> Is that an unknown Python idiom? >>> >>> It looks short but still insane. >> It's actually a useful way to quickly unpack nested lists and tuples. >> Here's a quick example: >> >> [snip] > > I am aware of (nested) unpacking in general. :) > > > The use-case of checking if a list is has only a single element and to > unpack it in a single statement has simply never occurred to me. In fact, I have been asked several times by some older Python developers on what the Pythonic way of doing exactly this is. Like this: Dev: Sven, do you know a better way to get THE item from single-item list rather than just [0]? Sven: huh? *distracted* yeah *thinking* why not [0]? seems simple enough to get the intent *back to topic* I think because of that thread, I won't forget anymore. ;-) Best, Sven From abarnert at yahoo.com Mon Feb 15 17:27:30 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 15 Feb 2016 14:27:30 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: <56C1987D.7050209@brenbarn.net> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <40592F59-371A-4440-B58E-7EC83DE54713@yahoo.com> <56C1987D.7050209@brenbarn.net> Message-ID: <083415BF-CD35-46BD-B272-AA8F8E4C1D7E@yahoo.com> On Feb 15, 2016, at 01:21, Brendan Barnwell wrote: > >> On 2016-02-15 00:39, Andrew Barnert via Python-ideas wrote: >>> On Feb 15, 2016, at 00:12, Paul Moore wrote: >>> >>> IMO, the other downside is that the semantic difference between >>> >>> a, b, ... = value >>> >>> and >>> >>> a, b, *_ = value >>> >>> is very subtle, and (even worse) only significant if value is an >>> iterable as opposed to a concrete container such as a list. >> >> You mean iterator, not iterable. And being "concrete" has nothing to >> do with it--a dict view, a memoryview, a NumPy slice, etc. aren't >> iterators any more than a list is. >> >> This is exactly why I think we need an official term like >> "collection" for "iterables that are not iterators" (or "iterables >> whose __iter__ doesn't return self" or similar). People struggling to >> come up with terms end up confusing themselves--not just about >> wording, but about actual concepts. As proven below: > > I still don't think that is at all what we need, as this example shows. Whether the value is an iterable or an iterator is not relevant. You're making the same mistake that this idea was meant to cure: iterators _are_ iterables. Which means this is vacuously true. But whether the value is a collection/reiterable/whatever or an iterator is exactly what's relevant. > Whether the iterable's iterator is self is not relevant. What is relevant is the difference in *behavior* --- namely, whether you can rewind, restart, or otherwise retrieve already-obtained values from the object, or whether advancing it is an irreversible operation and there is no way to get old values except by storing them yourself. An iterator returns self from iter, which means advancing the iterator is consuming self, so there is no way to get the old values again. A non-iterator iterable may return a different object from iter, which means advancing the iterator doesn't have to consume self; you can get the old values again just by calling iter to get a new iterator at the start. A collection, or reiterable, or whatever, could be defined as an iterable that _doesn't_ return self. Or, nearly-equivalently, it could be defined as an iterable that returns a new iterator over the same values (unless mutated in between iter calls), borrowing the distinction that's already in the docs for defining dict and dict view semantics. Of course any useful definition would leave pathological types that are neither iterator nor collection (e.g., an object that returns a new iterator each time, but those iterators destructively modify self), or maybe where they do qualify but misleadingly so (I can't think of any examples). And there may also be cases where it isn't clear (e.g., is a collection of shared memory not a collection because some other process can change its values, or does that count as "unless mutated"? that probably depends on how and why your app is using that shared memory). But that isn't a problem; we're not trying to come up with a definition that could be used to write type-theoretic behavior proofs for Python semantics, but something that's useful in practice for discussing almost all real Python programs with other humans. > In this example, being able to say that value was or was not an iterator or an iterable would in no way help to clarify how the code would behave differently. Saying that it is an iterable or an iterator is just saying that it has or doesn't have .next() and/or .__iter__() methods that follow certain very broad protocols, but what matters for understanding examples like this is what those methods actually DO. The iterator protocol defines what those methods do. The __iter__ method returns self, and the __next__ method advances self to return the next value. The fact that these semantics aren't checked (even by the ABC or by mypy) doesn't mean they don't exist or are meaningless. The iterable protocol, on the other hand, just tells you that __iter__ returns an iterator. Because iterators and collections are both iterable, just knowing that something is an iterable doesn't tell you whether it's reusable. But knowing that something is a collection (assuming we have a well-defined term "collection") would. Which is exactly the point of the proposal. From santagada at gmail.com Mon Feb 15 17:30:15 2016 From: santagada at gmail.com (Leonardo Santagada) Date: Mon, 15 Feb 2016 23:30:15 +0100 Subject: [Python-ideas] A bit meta Message-ID: Is there any way for me to help in evaluating/testing/deploying discourse? I really think trying it out and seeing if people flock around a discourse instance is beneficial for the community. -- Leonardo Santagada -------------- next part -------------- An HTML attachment was scrubbed... URL: From brenbarn at brenbarn.net Mon Feb 15 17:41:22 2016 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Mon, 15 Feb 2016 14:41:22 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: <083415BF-CD35-46BD-B272-AA8F8E4C1D7E@yahoo.com> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <40592F59-371A-4440-B58E-7EC83DE54713@yahoo.com> <56C1987D.7050209@brenbarn.net> <083415BF-CD35-46BD-B272-AA8F8E4C1D7E@yahoo.com> Message-ID: <56C25412.8050906@brenbarn.net> On 2016-02-15 14:27, Andrew Barnert wrote: > Of course any useful definition would leave pathological types that > are neither iterator nor collection (e.g., an object that returns a > new iterator each time, but those iterators destructively modify > self), or maybe where they do qualify but misleadingly so (I can't > think of any examples). And there may also be cases where it isn't > clear (e.g., is a collection of shared memory not a collection > because some other process can change its values, or does that count > as "unless mutated"? that probably depends on how and why your app is > using that shared memory). But that isn't a problem; we're not trying > to come up with a definition that could be used to write > type-theoretic behavior proofs for Python semantics, but something > that's useful in practice for discussing almost all real Python > programs with other humans. > Because iterators and collections are > both iterable, just knowing that something is an iterable doesn't > tell you whether it's reusable. But knowing that something is a > collection (assuming we have a well-defined term "collection") would. > Which is exactly the point of the proposal. I still don't understand why you want to use the term "collection" when everything you say about the purpose of the term (as for instance your last paragraph here) suggests you want to know whether it is reusable. To ordinary humans "collection" says nothing about iteration or reusability. If you want to say it's reusable, just say it's reusable. As a bonus, you don't have to get anyone to buy into a term like "collection", or even know the difference between iterators and iterables, because "reusable iterator" and "reusable iterable" (or "re-iterable", or "restartable" or other such terms) already have an obvious interpretation (even if one of them is technically impossible). -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From steve at pearwood.info Mon Feb 15 18:43:22 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 16 Feb 2016 10:43:22 +1100 Subject: [Python-ideas] Generator unpacking In-Reply-To: <56C210A9.4020002@mail.de> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> <20160215144251.GY31806@ando.pearwood.info> <56C20D56.2030605@stoneleaf.us> <56C210A9.4020002@mail.de> Message-ID: <20160215234322.GZ31806@ando.pearwood.info> On Mon, Feb 15, 2016 at 06:53:45PM +0100, Sven R. Kunze wrote: > On 15.02.2016 18:39, Ethan Furman wrote: > >I /think/ I've used that trick to confirm an iterable was empty; I > >/know/ I've used > > > >[some_var] = some_var > > Is that an unknown Python idiom? No, it's exactly the same Python idiom (assignment to a list of targets) as we've been talking about for the last few posts. We've had examples with four targets, three targets, two targets and zero targets. This is an example with one target. [a] = iterable requires that the right-hand side be iterable, and after unpacking it must contain exactly one item, to match the one assignment target given on the left. -- Steve From abarnert at yahoo.com Mon Feb 15 18:54:22 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 15 Feb 2016 15:54:22 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: <56C25412.8050906@brenbarn.net> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <40592F59-371A-4440-B58E-7EC83DE54713@yahoo.com> <56C1987D.7050209@brenbarn.net> <083415BF-CD35-46BD-B272-AA8F8E4C1D7E@yahoo.com> <56C25412.8050906@brenbarn.net> Message-ID: <05B2AA37-18B1-4402-BE91-8893077CD4C4@yahoo.com> On Feb 15, 2016, at 14:41, Brendan Barnwell wrote: > >> On 2016-02-15 14:27, Andrew Barnert wrote: >> Of course any useful definition would leave pathological types that >> are neither iterator nor collection (e.g., an object that returns a >> new iterator each time, but those iterators destructively modify >> self), or maybe where they do qualify but misleadingly so (I can't >> think of any examples). And there may also be cases where it isn't >> clear (e.g., is a collection of shared memory not a collection >> because some other process can change its values, or does that count >> as "unless mutated"? that probably depends on how and why your app is >> using that shared memory). But that isn't a problem; we're not trying >> to come up with a definition that could be used to write >> type-theoretic behavior proofs for Python semantics, but something >> that's useful in practice for discussing almost all real Python >> programs with other humans. > >> Because iterators and collections are >> both iterable, just knowing that something is an iterable doesn't >> tell you whether it's reusable. But knowing that something is a >> collection (assuming we have a well-defined term "collection") would. >> Which is exactly the point of the proposal. > > I still don't understand why you want to use the term "collection" when everything you say about the purpose of the term (as for instance your last paragraph here) suggests you want to know whether it is reusable. To ordinary humans "collection" says nothing about iteration or reusability. First, the word "reiterable" is acceptable to me (and certainly better than not having any such term)--that's why I said "like 'reiterable' or 'collection'" multiple times. It's an easy-to-remember term, and it's name helps keep it straight. The biggest problem is that it's not an actual word, and it looks and sounds ugly. It's much better than not having a word at all, and forcing people to use phrases like "reusable non-iterator iterable", but it's not ideal. The reason "collection" is better is that it matches an intuitive category that actually means what we want. And the fact that it's not immediately obvious that it does what we want is a _good_ thing, because it means people have one simple thing to learn and remember: you can iterate collections over and over without affecting them. And "collection" stays intuitive when used in compounds like "virtual collection" or "lazy collection" or "infinite collection". So, I can say, "a range is a lazy collection", and you know that means it's not a one-shot iterator, because collections aren't one-shot iterators--"lazy" ones, don't physically hold the stuff, but come up with it when asked, but they're still collections, so they're still not one-shot iterators. Also, notice that we already to use the term "collection" informally to mean exactly what people intuitively think of--including in the documentation of the "collections" module in the stdlib. We certainly don't use the term "reiterable" anywhere (especially when typing with autocorrect on). I think people worry about the case of things that aren't collections but also aren't iterators. But let's look at one: class Random3to6d6: def __iter__(self): for _ in range(randint(3, 6)): yield randint(1, 6) That's clearly not a collection. It's also clearly not an iterator. So, if I want to use it, that's a clear sign that I have to think about it carefully. Is it a reusable or a reiterable? Depends how you define the term. Which means I have to know, and have internalized, the exact definition before "reusable" is any help here. I still have to think about it carefully, but it's less obvious that I have to do so. Again, I think adding "reiterable" to the glossary (and maybe collections.abc and typing) would be a big improvement; I just think adding "collection" would be a slightly better one. From abarnert at yahoo.com Mon Feb 15 19:11:16 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 15 Feb 2016 16:11:16 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: <20160215234322.GZ31806@ando.pearwood.info> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> <20160215144251.GY31806@ando.pearwood.info> <56C20D56.2030605@stoneleaf.us> <56C210A9.4020002@mail.de> <20160215234322.GZ31806@ando.pearwood.info> Message-ID: <6E6AD40C-826A-4159-801F-F23EF12FF6E7@yahoo.com> On Feb 15, 2016, at 15:43, Steven D'Aprano wrote: > >> On Mon, Feb 15, 2016 at 06:53:45PM +0100, Sven R. Kunze wrote: >>> On 15.02.2016 18:39, Ethan Furman wrote: >>> I /think/ I've used that trick to confirm an iterable was empty; I >>> /know/ I've used >>> >>> [some_var] = some_var >> >> Is that an unknown Python idiom? > > No, it's exactly the same Python idiom (assignment to a list of targets) > as we've been talking about for the last few posts. We've had examples > with four targets, three targets, two targets and zero targets. This is > an example with one target. > > [a] = iterable > > requires that the right-hand side be iterable, and after unpacking it > must contain exactly one item, to match the one assignment target given > on the left. I see this more often written in tuple-ish form: index, value = struct.unpack_from('!HH', buf, off) namelen, = struct.unpack_from('!H', buf, off+4) Somehow, that feels more natural and pythonic than using [index, value] and [namelen], despite the fact that without the brackets it's easy to miss the trailing comma and end up with (16,) instead of 16 as your length and a confusing TypeError a few lines down. From steve at pearwood.info Mon Feb 15 19:34:43 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 16 Feb 2016 11:34:43 +1100 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> <20160215144251.GY31806@ando.pearwood.info> <56C1EB0D.70505@mail.de> Message-ID: <20160216003443.GA31806@ando.pearwood.info> On Mon, Feb 15, 2016 at 01:55:32PM -0800, Andrew Barnert via Python-ideas wrote: > On Feb 15, 2016, at 07:13, Sven R. Kunze wrote: > > > >> On 15.02.2016 15:42, Steven D'Aprano wrote: > >> I believe you are misinterpreting the error. The error isn't that you > >> are trying to assign to the literal [1,2]. The error is that you are > >> trying to assign to the literal 1. It's a subtle difference, > > > > You are indeed right. > > > > And while we are on it, that reminds me of KeyError. I always feel > > annoyed by the fact that Python doesn't tell me what key a dict is > > missing. Showing the missing key in the error message goes all the way back to Python 1.5: [steve at ando ~]$ python1.5 -c '{}["spam"]' Traceback (innermost last): File "", line 1, in ? KeyError: spam Admittedly, the exception object itself doesn't keep a reference to the missing key, so you can't programmatically query it for the key, but normally that's not a problem since you just tried to look it up so you should still have the key. > > So, if I were to make a suggestion, I would like to see the > > issue-causing thing to be mentioned in those two types of > > exceptions. > > IIRC, there's a series of bugs on adding more useful information to > the builtin exceptions, which includes putting the key in the error > message _and_ as an attribute of the exception (a la filename in the > filesystem exceptions), but they've been waiting on someone to do the > work for a few years now. If so, there's an obvious way to kick-start > the solution: find the bug and write a patch. (I suspect some people > would object to fixing KeyError without also fixing IndexError, and > want to argue about whether the key member goes in each exception > directly or is part of LookupError, and so on, and there's the more > substantive issue of keeping alive potentially-large keys, and so on, > so it probably won't be as simple as "submit a trivial patch and it > gets accepted", but at least having a patch would get the process > moving.) See discussion here: http://bugs.python.org/issue18162 Also http://bugs.python.org/issue1182143 And the PEP: https://www.python.org/dev/peps/pep-0473/ > But for SyntaxError, there is no object to stick in the error object > or error message, just a source token or an AST node. Both of those > have pointers back to start and end in the source string, and that's > what goes into the SyntaxError, which is already shown to the user by > printing the relevant line of source and pointing a caret at it. Sadly, CPython doesn't manage to even display the caret at all: [steve at ando ~]$ python -c "[a, 2] = 'xy'" File "", line 1 SyntaxError: can't assign to literal Jython gives a much more informative error: steve at orac:~$ jython -c "[a, 2] = 'ab'" File "", line 1 [a, 2] = 'ab' ^ SyntaxError: can't assign to number -- Steve From vgr255 at live.ca Mon Feb 15 19:49:33 2016 From: vgr255 at live.ca (=?iso-8859-1?Q?=C9manuel_Barry?=) Date: Mon, 15 Feb 2016 19:49:33 -0500 Subject: [Python-ideas] Generator unpacking In-Reply-To: <20160216003443.GA31806@ando.pearwood.info> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> <20160215144251.GY31806@ando.pearwood.info> <56C1EB0D.70505@mail.de> <20160216003443.GA31806@ando.pearwood.info> Message-ID: > From: Steven D'Aprano > Sent: Monday, February 15, 2016 7:35 PM > Subject: Re: [Python-ideas] Generator unpacking > > Showing the missing key in the error message goes all the way back to > Python 1.5: > > [steve at ando ~]$ python1.5 -c '{}["spam"]' > Traceback (innermost last): > File "", line 1, in ? > KeyError: spam > > Admittedly, the exception object itself doesn't keep a reference to the > missing key, so you can't programmatically query it for the key [...] Well, there's no `key` attribute for example, but the KeyError exception has exactly one argument, the missing key. >>> try: ... {}[42] ... except KeyError as e: ... print(e.args[0]) ... 42 So, you *can* query it for the missing key, even though it's a bit ugly. But as far as TOOWTDI goes, I think this is fine as-is. This is just my opinion, though :) -- Emanuel From stephen at xemacs.org Mon Feb 15 22:42:11 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 16 Feb 2016 12:42:11 +0900 Subject: [Python-ideas] Generator unpacking In-Reply-To: <083415BF-CD35-46BD-B272-AA8F8E4C1D7E@yahoo.com> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <40592F59-371A-4440-B58E-7EC83DE54713@yahoo.com> <56C1987D.7050209@brenbarn.net> <083415BF-CD35-46BD-B272-AA8F8E4C1D7E@yahoo.com> Message-ID: <22210.39571.987984.507976@turnbull.sk.tsukuba.ac.jp> Andrew Barnert via Python-ideas writes: > But that isn't a problem; we're not trying to come up with a > definition that could be used to write type-theoretic behavior > proofs for Python semantics, but something that's useful in > practice for discussing almost all real Python programs with other > humans. Are there really places where we *want* "foo(range(n))" to raise but "foo(iter(range(n)))" to do something useful? In other words, is this really a terminology problem, rather than a problem with the design of APIs that take iterators rather than iterables? From rob.cliffe at btinternet.com Mon Feb 15 22:47:14 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Tue, 16 Feb 2016 03:47:14 +0000 Subject: [Python-ideas] Generator unpacking In-Reply-To: <05B2AA37-18B1-4402-BE91-8893077CD4C4@yahoo.com> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <40592F59-371A-4440-B58E-7EC83DE54713@yahoo.com> <56C1987D.7050209@brenbarn.net> <083415BF-CD35-46BD-B272-AA8F8E4C1D7E@yahoo.com> <56C25412.8050906@brenbarn.net> <05B2AA37-18B1-4402-BE91-8893077CD4C4@yahoo.com> Message-ID: <56C29BC2.8090607@btinternet.com> On 15/02/2016 23:54, Andrew Barnert via Python-ideas wrote: > On Feb 15, 2016, at 14:41, Brendan Barnwell wrote: >>> On 2016-02-15 14:27, Andrew Barnert wrote: >>> Of course any useful definition would leave pathological types that >>> are neither iterator nor collection (e.g., an object that returns a >>> new iterator each time, but those iterators destructively modify >>> self), or maybe where they do qualify but misleadingly so (I can't >>> think of any examples). And there may also be cases where it isn't >>> clear (e.g., is a collection of shared memory not a collection >>> because some other process can change its values, or does that count >>> as "unless mutated"? that probably depends on how and why your app is >>> using that shared memory). But that isn't a problem; we're not trying >>> to come up with a definition that could be used to write >>> type-theoretic behavior proofs for Python semantics, but something >>> that's useful in practice for discussing almost all real Python >>> programs with other humans. >> >>> Because iterators and collections are >>> both iterable, just knowing that something is an iterable doesn't >>> tell you whether it's reusable. But knowing that something is a >>> collection (assuming we have a well-defined term "collection") would. >>> Which is exactly the point of the proposal. >> I still don't understand why you want to use the term "collection" when everything you say about the purpose of the term (as for instance your last paragraph here) suggests you want to know whether it is reusable. To ordinary humans "collection" says nothing about iteration or reusability. > First, the word "reiterable" is acceptable to me (and certainly better than not having any such term)--that's why I said "like 'reiterable' or 'collection'" multiple times. It's an easy-to-remember term, and it's name helps keep it straight. The biggest problem is that it's not an actual word, and it looks and sounds ugly. It's much better than not having a word at all, and forcing people to use phrases like "reusable non-iterator iterable", but it's not ideal. > > The reason "collection" is better is that it matches an intuitive category that actually means what we want. > > And the fact that it's not*immediately obvious* [my emphasis - RC] that it does what we want is a _good_ thing, because it means people have one simple thing to learn and remember: you can iterate collections over and over without affecting them. So you'd rather people learned the word "collection" (which does not convey that they are reiterable) and then *learned* that collections can be iterated over repeatedly, than learn the word "reiterable" that makes it *immediately obvious* that something can be iterated over repeatedly. Why make life more difficult? How is that a "_good_ thing"? Rob Cliffe > > And "collection" stays intuitive when used in compounds like "virtual collection" or "lazy collection" or "infinite collection". So, I can say, "a range is a lazy collection", and you know that means it's not a one-shot iterator, because collections aren't one-shot iterators--"lazy" ones, don't physically hold the stuff, but come up with it when asked, but they're still collections, so they're still not one-shot iterators. > > Also, notice that we already to use the term "collection" informally to mean exactly what people intuitively think of--including in the documentation of the "collections" module in the stdlib. We certainly don't use the term "reiterable" anywhere (especially when typing with autocorrect on). > > I think people worry about the case of things that aren't collections but also aren't iterators. But let's look at one: > > class Random3to6d6: > def __iter__(self): > for _ in range(randint(3, 6)): > yield randint(1, 6) > > That's clearly not a collection. It's also clearly not an iterator. So, if I want to use it, that's a clear sign that I have to think about it carefully. > > Is it a reusable or a reiterable? Depends how you define the term. Which means I have to know, and have internalized, the exact definition before "reusable" is any help here. I still have to think about it carefully, but it's less obvious that I have to do so. > > Again, I think adding "reiterable" to the glossary (and maybe collections.abc and typing) would be a big improvement; I just think adding "collection" would be a slightly better one. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Tue Feb 16 00:56:14 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 15 Feb 2016 21:56:14 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: <22210.39571.987984.507976@turnbull.sk.tsukuba.ac.jp> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <40592F59-371A-4440-B58E-7EC83DE54713@yahoo.com> <56C1987D.7050209@brenbarn.net> <083415BF-CD35-46BD-B272-AA8F8E4C1D7E@yahoo.com> <22210.39571.987984.507976@turnbull.sk.tsukuba.ac.jp> Message-ID: <4CAB96F6-F814-43D8-93A6-A8AC00BF06F5@yahoo.com> On Feb 15, 2016, at 19:42, Stephen J. Turnbull wrote: > > Andrew Barnert via Python-ideas writes: > >> But that isn't a problem; we're not trying to come up with a >> definition that could be used to write type-theoretic behavior >> proofs for Python semantics, but something that's useful in >> practice for discussing almost all real Python programs with other >> humans. > > Are there really places where we *want* "foo(range(n))" to raise but > "foo(iter(range(n)))" to do something useful? In other words, is this > really a terminology problem, rather than a problem with the design of > APIs that take iterators rather than iterables? Sure. Any multi-pass algorithm should raise when given an iterator. (Maybe some could instead copy the iterator to a list or something, but as a general rule, silently and unexpectedly jumping from 0 to O(N) space depending on the input type isn't very friendly.) In the other direction, of course, the next function requires an iterator, and should and does raise when given something else. And the slightly higher-level functions discussed in this thread are the same. A function intended to consume and return the first few values in an iterator while leaving the iterator holding the rest is basically just a "multinext", and it would be very confusing if it took a collection and left it holding all the values, including the two you already used. And how do you explain that to a human? You could say "While it's an iterable, it's not an iterator." And that's exactly what Python has said for the last however-many years. And that's exactly what leads to the confusion Paul was talking about (and the additional confusion he inadvertently revealed). And the problem isn't "Paul is an idiot", because I've seen multiple core devs, and other highly experienced Python programmers, make the same mistake. Even the official docs made a similar mistake, and nobody caught it for 4 years. If the current terminology were sufficiently clear and intuitive that nothing needed to be done, these problems wouldn't exist. From mike at selik.org Tue Feb 16 00:59:58 2016 From: mike at selik.org (Michael Selik) Date: Tue, 16 Feb 2016 05:59:58 +0000 Subject: [Python-ideas] Generator unpacking In-Reply-To: <56C29BC2.8090607@btinternet.com> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <40592F59-371A-4440-B58E-7EC83DE54713@yahoo.com> <56C1987D.7050209@brenbarn.net> <083415BF-CD35-46BD-B272-AA8F8E4C1D7E@yahoo.com> <56C25412.8050906@brenbarn.net> <05B2AA37-18B1-4402-BE91-8893077CD4C4@yahoo.com> <56C29BC2.8090607@btinternet.com> Message-ID: On Mon, Feb 15, 2016 at 10:59 PM Rob Cliffe wrote: > > > On 15/02/2016 23:54, Andrew Barnert via Python-ideas wrote: > > On Feb 15, 2016, at 14:41, Brendan Barnwell wrote: > > On 2016-02-15 14:27, Andrew Barnert wrote: > Of course any useful definition would leave pathological types that > are neither iterator nor collection (e.g., an object that returns a > new iterator each time, but those iterators destructively modify > self), or maybe where they do qualify but misleadingly so (I can't > think of any examples). And there may also be cases where it isn't > clear (e.g., is a collection of shared memory not a collection > because some other process can change its values, or does that count > as "unless mutated"? that probably depends on how and why your app is > using that shared memory). But that isn't a problem; we're not trying > to come up with a definition that could be used to write > type-theoretic behavior proofs for Python semantics, but something > that's useful in practice for discussing almost all real Python > programs with other humans. > > > > Because iterators and collections are > both iterable, just knowing that something is an iterable doesn't > tell you whether it's reusable. But knowing that something is a > collection (assuming we have a well-defined term "collection") would. > Which is exactly the point of the proposal. > > I still don't understand why you want to use the term "collection" when everything you say about the purpose of the term (as for instance your last paragraph here) suggests you want to know whether it is reusable. To ordinary humans "collection" says nothing about iteration or reusability. > > First, the word "reiterable" is acceptable to me (and certainly better than not having any such term)--that's why I said "like 'reiterable' or 'collection'" multiple times. It's an easy-to-remember term, and it's name helps keep it straight. The biggest problem is that it's not an actual word, and it looks and sounds ugly. It's much better than not having a word at all, and forcing people to use phrases like "reusable non-iterator iterable", but it's not ideal. > > The reason "collection" is better is that it matches an intuitive category that actually means what we want. > > And the fact that it's not *immediately obvious* [my emphasis - RC] that it does what we want is a _good_ thing, because it means people have one simple thing to learn and remember: you can iterate collections over and over without affecting them. > > So you'd rather people learned the word "collection" (which does not > convey that they are reiterable) and then *learned* that collections can be > iterated over repeatedly, than learn the word "reiterable" that makes it *immediately > obvious* that something can be iterated over repeatedly. > Why make life more difficult? How is that a "_good_ thing"? > > Rob Cliffe > > And "collection" stays intuitive when used in compounds like "virtual collection" or "lazy collection" or "infinite collection". So, I can say, "a range is a lazy collection", and you know that means it's not a one-shot iterator, because collections aren't one-shot iterators--"lazy" ones, don't physically hold the stuff, but come up with it when asked, but they're still collections, so they're still not one-shot iterators. > > Also, notice that we already to use the term "collection" informally to mean exactly what people intuitively think of--including in the documentation of the "collections" module in the stdlib. We certainly don't use the term "reiterable" anywhere (especially when typing with autocorrect on). > > I think people worry about the case of things that aren't collections but also aren't iterators. But let's look at one: > > class Random3to6d6: > def __iter__(self): > for _ in range(randint(3, 6)): > yield randint(1, 6) > > That's clearly not a collection. It's also clearly not an iterator. So, if I want to use it, that's a clear sign that I have to think about it carefully. > > Is it a reusable or a reiterable? Depends how you define the term. Which means I have to know, and have internalized, the exact definition before "reusable" is any help here. I still have to think about it carefully, but it's less obvious that I have to do so. > > Again, I think adding "reiterable" to the glossary (and maybe collections.abc and typing) would be a big improvement; I just think adding "collection" would be a slightly better one. > > The glossary entry for "iterator" already says, "A container object (such as a list) produces a fresh new iterator each time you pass it to the iter() function or use it in a for loop." I expect that no one will misunderstand you if you say the word "reiterable" in the context of checking for an exhausted iterator or an iterable that returns a new iterator each time. A glossary entry just opens the opportunity for nitpicks and confusion between a natural interpretation of the word and the glossary definition. If not in the glossary, conversants will know to be careful to (re)define the term as appropriate to the conversation if necessary. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Tue Feb 16 01:02:26 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 15 Feb 2016 22:02:26 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: <56C29BC2.8090607@btinternet.com> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <40592F59-371A-4440-B58E-7EC83DE54713@yahoo.com> <56C1987D.7050209@brenbarn.net> <083415BF-CD35-46BD-B272-AA8F8E4C1D7E@yahoo.com> <56C25412.8050906@brenbarn.net> <05B2AA37-18B1-4402-BE91-8893077CD4C4@yahoo.com> <56C29BC2.8090607@btinternet.com> Message-ID: <9C78842E-B30A-43D4-BDC2-16CCA19B4656@yahoo.com> On Feb 15, 2016, at 19:47, Rob Cliffe wrote: > >> On 15/02/2016 23:54, Andrew Barnert via Python-ideas wrote: >>> On Feb 15, 2016, at 14:41, Brendan Barnwell wrote: >>>>> On 2016-02-15 14:27, Andrew Barnert wrote: >>>>> Of course any useful definition would leave pathological types that >>>>> are neither iterator nor collection (e.g., an object that returns a >>>>> new iterator each time, but those iterators destructively modify >>>>> self), or maybe where they do qualify but misleadingly so (I can't >>>>> think of any examples). And there may also be cases where it isn't >>>>> clear (e.g., is a collection of shared memory not a collection >>>>> because some other process can change its values, or does that count >>>>> as "unless mutated"? that probably depends on how and why your app is >>>>> using that shared memory). But that isn't a problem; we're not trying >>>>> to come up with a definition that could be used to write >>>>> type-theoretic behavior proofs for Python semantics, but something >>>>> that's useful in practice for discussing almost all real Python >>>>> programs with other humans. >>>> >>>> Because iterators and collections are >>>> both iterable, just knowing that something is an iterable doesn't >>>> tell you whether it's reusable. But knowing that something is a >>>> collection (assuming we have a well-defined term "collection") would. >>>> Which is exactly the point of the proposal. >>> I still don't understand why you want to use the term "collection" when everything you say about the purpose of the term (as for instance your last paragraph here) suggests you want to know whether it is reusable. To ordinary humans "collection" says nothing about iteration or reusability. >> First, the word "reiterable" is acceptable to me (and certainly better than not having any such term)--that's why I said "like 'reiterable' or 'collection'" multiple times. It's an easy-to-remember term, and it's name helps keep it straight. The biggest problem is that it's not an actual word, and it looks and sounds ugly. It's much better than not having a word at all, and forcing people to use phrases like "reusable non-iterator iterable", but it's not ideal. >> >> The reason "collection" is better is that it matches an intuitive category that actually means what we want. >> >> And the fact that it's not immediately obvious [my emphasis - RC] that it does what we want is a _good_ thing, because it means people have one simple thing to learn and remember: you can iterate collections over and over without affecting them. > So you'd rather people learned the word "collection" (which does not convey that they are reiterable) and then *learned* that collections can be iterated over repeatedly, than learn the word "reiterable" that makes it immediately obvious that something can be iterated over repeatedly. > Why make life more difficult? How is that a "_good_ thing"? Because "reiterable" is really not immediately obvious. It has no intuitive meaning; it's not even a real word. You have to learn what a reiterable is, by definition. That's no easier than learning a simple fact about collections, and you end up knowing less after that learning. Think of my random example. Is that a reiterable? Well, you can call iter on its multiple times and get different iterators. But they don't iterate the same value. So, yes or no? There's no intuition to guide you there; all you have is a definitional rule that you had to memorize. Similarly, you can't meaningfully attach adjectives to "reiterable" that restrict or expand its scope. "Lazy reiterable" doesn't mean anything; "lazy collection" does. But I've already made all these points and you just skipped over them. And more importantly, as I've already said, I'd be very happy for "reiterable" to be added to the glossary and maybe the ABCs--it's not as good as "collection", but it's a lot better than nothing--so I don't really want to argue against it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcgoble3 at gmail.com Tue Feb 16 01:04:04 2016 From: jcgoble3 at gmail.com (Jonathan Goble) Date: Tue, 16 Feb 2016 01:04:04 -0500 Subject: [Python-ideas] Exposing regular expression bytecode In-Reply-To: References: Message-ID: Hello? Anyone? I'd really like to see this idea move forward, but I need some comments for that. Original message: https://mail.python.org/pipermail/python-ideas/2016-February/038488.html On Mon, Feb 15, 2016 at 2:47 AM, Jonathan Goble wrote: > On Mon, Feb 15, 2016 at 2:26 AM, Wes Turner wrote: >> the new 'regex' module (with a re compatability mode and unicode) may be the >> place to find/add more debugging syms > > Thanks, but I'm already well aware of that module, as I'm already > using it for another project of mine that requires some of its new > features. But this proposal, I think, is simple enough that it would > be worth doing for the built-in 're' modules. Also, the things that > implementing it would enable, such as optimizers and especially > debuggers, would be available to a much larger number of people if the > built-in module supported them. > > I'm not really asking for much; just take an array already in the > struct that represents the object, expose it as a public attribute, > and write a simple 'dis'-like module to enable low-level programmatic > analysis. Once that's available, third parties are likely to build on > it from there; that's much less likely to happen if such tools only > worked on a different regex module (albeit a great one) from PyPI and > not on the built-in version. From abarnert at yahoo.com Tue Feb 16 01:12:55 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 15 Feb 2016 22:12:55 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <40592F59-371A-4440-B58E-7EC83DE54713@yahoo.com> <56C1987D.7050209@brenbarn.net> <083415BF-CD35-46BD-B272-AA8F8E4C1D7E@yahoo.com> <56C25412.8050906@brenbarn.net> <05B2AA37-18B1-4402-BE91-8893077CD4C4@yahoo.com> <56C29BC2.8090607@btinternet.com> Message-ID: <247A0783-58BB-4C3E-85CC-C654C35F4400@yahoo.com> On Feb 15, 2016, at 21:59, Michael Selik wrote: > > The glossary entry for "iterator" already says, "A container object (such as a list) produces a fresh new iterator each time you pass it to the iter() function or use it in a for loop." Well, "container" would be a perfectly good word if we didn't already use it to mean something different: an object with the "in" operator. You can have collections of things that can't efficiently test containment (a lazy linked list would take linear time and space for __contains__), and containment of things that aren't even iterables (e.g., a real interval). > I expect that no one will misunderstand you if you say the word "reiterable" in the context of checking for an exhausted iterator or an iterable that returns a new iterator each time. The problem isn't so much whether I can use it--I do, in fact, use both "collection" and "reiterable", defining them when necessary (it often isn't necessary--but when it is, at least I only have to define them once). It's whether other people know there's a word they can use. When they don't, they struggle to explain what they mean, and frequently make mistakes, like Paul saying that a range is not an iterable while trying to explain someone else's confusion about iterables, or whoever wrote the docs on dict views saying that they're not iterators but sequences (which took years for someone to notice and fix). From abarnert at yahoo.com Tue Feb 16 01:15:56 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 15 Feb 2016 22:15:56 -0800 Subject: [Python-ideas] Exposing regular expression bytecode In-Reply-To: References: Message-ID: Isn't the regex module intended to replace the re module "one of these days", once it's actually whipped into stdlibable shape? If so, it seems like it would be better to help Matthew finish that than to add to the re module. Sent from my iPhone > On Feb 15, 2016, at 22:04, Jonathan Goble wrote: > > Hello? Anyone? I'd really like to see this idea move forward, but I > need some comments for that. > > Original message: > https://mail.python.org/pipermail/python-ideas/2016-February/038488.html > >> On Mon, Feb 15, 2016 at 2:47 AM, Jonathan Goble wrote: >>> On Mon, Feb 15, 2016 at 2:26 AM, Wes Turner wrote: >>> the new 'regex' module (with a re compatability mode and unicode) may be the >>> place to find/add more debugging syms >> >> Thanks, but I'm already well aware of that module, as I'm already >> using it for another project of mine that requires some of its new >> features. But this proposal, I think, is simple enough that it would >> be worth doing for the built-in 're' modules. Also, the things that >> implementing it would enable, such as optimizers and especially >> debuggers, would be available to a much larger number of people if the >> built-in module supported them. >> >> I'm not really asking for much; just take an array already in the >> struct that represents the object, expose it as a public attribute, >> and write a simple 'dis'-like module to enable low-level programmatic >> analysis. Once that's available, third parties are likely to build on >> it from there; that's much less likely to happen if such tools only >> worked on a different regex module (albeit a great one) from PyPI and >> not on the built-in version. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From jcgoble3 at gmail.com Tue Feb 16 01:36:34 2016 From: jcgoble3 at gmail.com (Jonathan Goble) Date: Tue, 16 Feb 2016 01:36:34 -0500 Subject: [Python-ideas] Exposing regular expression bytecode In-Reply-To: References: Message-ID: On Tue, Feb 16, 2016 at 1:15 AM, Andrew Barnert wrote: > Isn't the regex module intended to replace the re module "one of these days", once it's actually whipped into stdlibable shape? If so, it seems like it would be better to help Matthew finish that than to add to the re module. As I said roughly 24 hours ago, I'd like to see this in core now (meaning 3.6) because of its usefulness and (I presume) its simplicity. The regex module has been "intended" to replace re for what seems like (and probably has been) years, and could take several more years at this pace. I really don't see what is accomplished by excluding a simple, useful feature from this year's version of Python core in favor of shunting it to (what is currently) a third-party PyPI module in the hopes that it will eventually be approved to replace the core module some years down the road. All that does is drastically reduce the number of people who have access to the feature in the time being. Don't get me wrong here; I completely understand why the regex module is being developed separately, because the changes being made are massive. But this proposed feature is not even remotely on the scale of those changes and can not only easily be made to core now, but can benefit many, many users of regular expressions, especially after third-party debuggers and optimizers come along (which is not likely to happen *until* it's in core). Why arbitrarily limit the number of people with access to it? And why does the fact that another regex module has been in development for several years have to preclude any possibility of changes to the current one, especially when the other one doesn't seem likely to get approved any time soon? As a final note, I ALREADY argued against shunting this to regex last night, and quoted that response in my bump tonight. As such, you're not leaving a good first impression on newbie me by totally ignoring that and repeating the same suggestion without any attempt to counter my arguments against it. If you have actual counter-arguments to either of these responses, I'm open to listening and possibly being convinced otherwise, but please don't just ignore what I've already said. From mike at selik.org Tue Feb 16 01:50:02 2016 From: mike at selik.org (Michael Selik) Date: Tue, 16 Feb 2016 06:50:02 +0000 Subject: [Python-ideas] Generator unpacking In-Reply-To: <9C78842E-B30A-43D4-BDC2-16CCA19B4656@yahoo.com> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <40592F59-371A-4440-B58E-7EC83DE54713@yahoo.com> <56C1987D.7050209@brenbarn.net> <083415BF-CD35-46BD-B272-AA8F8E4C1D7E@yahoo.com> <56C25412.8050906@brenbarn.net> <05B2AA37-18B1-4402-BE91-8893077CD4C4@yahoo.com> <56C29BC2.8090607@btinternet.com> <9C78842E-B30A-43D4-BDC2-16CCA19B4656@yahoo.com> Message-ID: On Tue, Feb 16, 2016 at 1:02 AM Andrew Barnert via Python-ideas < python-ideas at python.org> wrote: > On Feb 15, 2016, at 19:47, Rob Cliffe wrote: > > So you'd rather people learned the word "collection" (which does not > convey that they are reiterable) and then *learned* that collections can be > iterated over repeatedly, than learn the word "reiterable" that makes it *immediately > obvious* that something can be iterated over repeatedly. > > > Because "reiterable" is really not immediately obvious. It has no > intuitive meaning; > Seems intuitive to me that reiterable is repeatedly iterable, or at least iterable more than once. I suppose the dictionary would say that the "re-" prefix means "again". The word "non-iterable" isn't in the glossary and people seem to get that it means not iterable. If something is autoiterable it'd iterate over itself (so an iterator is autoiterable?), if something is semiiterable, it'd be kinda-iterable or half-iterable. I'm not sure what that means, but I'll bet I'd understand it in context. Maybe when you iterate over it you don't actually get all the elements. The English language is very flexible and can match Python's flexibility. I think it's the desire to use an inflexible glossary word that led people into the confusions/misuse you've described. > It's whether other people know there's a word they can use. I doubt a glossary entry would solve that problem. Instead I think premature glossarizing (a new word!) would create confusion as some people use a word with the glossary definition and some use it however the word makes sense in that particular conversation. Perhaps we could try something similar to the progression of a module from personal to PyPI to popularity to standard library. Jargon should first be popular in natural usage (without long debates) on the mailing lists before it gets added to the glossary. I know you really want to add something like this to the glossary, as you've brought it up before. I think a very convincing argument would be referencing a few conversations where you used a term, either word or phrase, to successfully clarify a conversation without needing to debate its definition. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Tue Feb 16 02:28:45 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 15 Feb 2016 23:28:45 -0800 Subject: [Python-ideas] Exposing regular expression bytecode In-Reply-To: References: Message-ID: <475EF1D2-E27E-4AB6-9729-66B00470D60F@yahoo.com> On Feb 15, 2016, at 22:36, Jonathan Goble wrote: > >> On Tue, Feb 16, 2016 at 1:15 AM, Andrew Barnert wrote: >> Isn't the regex module intended to replace the re module "one of these days", once it's actually whipped into stdlibable shape? If so, it seems like it would be better to help Matthew finish that than to add to the re module. > > As I said roughly 24 hours ago, I'd like to see this in core now > (meaning 3.6) because of its usefulness and (I presume) its > simplicity. The regex module has been "intended" to replace re for > what seems like (and probably has been) years, and could take several > more years at this pace. Sorry; I was assuming it was close because it almost made it into 3.3 and then again 3.4. But I guess the alternative position--that almost but not quite making 3.3 and 3.4 and then not even being mentioned for 3.5--implies that it may not be that near-future, even if you were to help with it. Anyway, assuming it's not that complicated, and you did the work, the only negative is that it might encourage people to write code that depends on re that regex can't be backward-compatible with. And at this point, that doesn't seem like much of a negative. (At the very least, nobody's made a good argument for it being a major negative.) > As such, you're > not leaving a good first impression on newbie me On a personal note: please keep in mind that I'm just some random guy, like you, and so are many other people on this list. Please don't hold anything I say against the core devs, any more than you'd want something you say to represent them. It can be frustrating to push ideas through -ideas--but if you think of the long-term consequences of keeping Python simple and conservative vs. keeping the discussion fair to new ideas, it helps with the frustration. I hope I haven't contributed to pushing you away from contributing to Python, because it needs people willing to do the work and put together complete ideas, with complete implementations. From jcgoble3 at gmail.com Tue Feb 16 02:42:35 2016 From: jcgoble3 at gmail.com (Jonathan Goble) Date: Tue, 16 Feb 2016 02:42:35 -0500 Subject: [Python-ideas] Exposing regular expression bytecode In-Reply-To: <475EF1D2-E27E-4AB6-9729-66B00470D60F@yahoo.com> References: <475EF1D2-E27E-4AB6-9729-66B00470D60F@yahoo.com> Message-ID: On Tue, Feb 16, 2016 at 2:28 AM, Andrew Barnert wrote: > On Feb 15, 2016, at 22:36, Jonathan Goble wrote: >> >>> On Tue, Feb 16, 2016 at 1:15 AM, Andrew Barnert wrote: >>> Isn't the regex module intended to replace the re module "one of these days", once it's actually whipped into stdlibable shape? If so, it seems like it would be better to help Matthew finish that than to add to the re module. >> >> As I said roughly 24 hours ago, I'd like to see this in core now >> (meaning 3.6) because of its usefulness and (I presume) its >> simplicity. The regex module has been "intended" to replace re for >> what seems like (and probably has been) years, and could take several >> more years at this pace. > > Sorry; I was assuming it was close because it almost made it into 3.3 and then again 3.4. But I guess the alternative position--that almost but not quite making 3.3 and 3.4 and then not even being mentioned for 3.5--implies that it may not be that near-future, even if you were to help with it. That's precisely what I was getting at. :-) > Anyway, assuming it's not that complicated, and you did the work, the only negative is that it might encourage people to write code that depends on re that regex can't be backward-compatible with. And at this point, that doesn't seem like much of a negative. (At the very least, nobody's made a good argument for it being a major negative.) I would imagine that a change this simple could be easily made to regex as well as re. In other words, I'm not opposed to having the change made in regex; I just want to see it in re at the same time. As for doing the work, I'm willing to look into that possibility, but I had never, ever written C code until last month, and I know nothing about the Python/C API. Literally all I've done in C is a fork and extension of Lua's pattern matching [1]; considering a feature for that is what led me to inspect Python's re internals and made me realize that exposing the code could be very beneficial. [1] https://github.com/jcgoble3/lua-matchext >> As such, you're >> not leaving a good first impression on newbie me > > On a personal note: please keep in mind that I'm just some random guy, like you, and so are many other people on this list. Please don't hold anything I say against the core devs, any more than you'd want something you say to represent them. It can be frustrating to push ideas through -ideas--but if you think of the long-term consequences of keeping Python simple and conservative vs. keeping the discussion fair to new ideas, it helps with the frustration. I hope I haven't contributed to pushing you away from contributing to Python, because it needs people willing to do the work and put together complete ideas, with complete implementations. I understand, and probably shouldn't have added that last paragraph, but it is frustrating to argue against something and then have the same thing thrown back at me without counter-argument. I guess I'm just annoyed that I haven't gotten any substantive comments regarding the usefulness of it, while other threads are highly active (e.g. generator unpacking). Perhaps I should be a bit more patient; patience has never been one of my positive qualities. :-P > I hope I haven't contributed to pushing you away from contributing to Python, because it needs people willing to do the work and put together complete ideas, with complete implementations. You haven't. :-) And again, I'm willing to do some coding work on this, but it may take some time for me to learn my way around the Python/C API, as a cursory look at the docs immediately shows that it's vastly different from the Lua/C API that I've recently become accustomed to. From abarnert at yahoo.com Tue Feb 16 03:26:42 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 16 Feb 2016 00:26:42 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <40592F59-371A-4440-B58E-7EC83DE54713@yahoo.com> <56C1987D.7050209@brenbarn.net> <083415BF-CD35-46BD-B272-AA8F8E4C1D7E@yahoo.com> <56C25412.8050906@brenbarn.net> <05B2AA37-18B1-4402-BE91-8893077CD4C4@yahoo.com> <56C29BC2.8090607@btinternet.com> <9C78842E-B30A-43D4-BDC2-16CCA19B4656@yahoo.com> Message-ID: <917E2CF5-E0A9-420B-B312-F6FE9F9A0D57@yahoo.com> On Feb 15, 2016, at 22:50, Michael Selik wrote: > >> On Tue, Feb 16, 2016 at 1:02 AM Andrew Barnert via Python-ideas wrote: > > > I doubt a glossary entry would solve that problem. Instead I think premature glossarizing (a new word!) I don't think it's at all premature. This was first suggested somewhere before 2006, and came close to being approved in 2006, but fell apart over bikeshedding issues. A decade later--a decade of watching people make exactly the mistakes that we've seen in this thread--has shown that the problem still needs a solution. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Feb 16 04:22:38 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 16 Feb 2016 20:22:38 +1100 Subject: [Python-ideas] Exposing regular expression bytecode In-Reply-To: References: <475EF1D2-E27E-4AB6-9729-66B00470D60F@yahoo.com> Message-ID: On Tue, Feb 16, 2016 at 6:42 PM, Jonathan Goble wrote: > I understand, and probably shouldn't have added that last paragraph, > but it is frustrating to argue against something and then have the > same thing thrown back at me without counter-argument. I guess I'm > just annoyed that I haven't gotten any substantive comments regarding > the usefulness of it, while other threads are highly active (e.g. > generator unpacking). Perhaps I should be a bit more patient; patience > has never been one of my positive qualities. :-P > For what it's worth, I read your post with interest, but didn't have anything substantive to reply - mainly because I don't use regexes much. But it would be rather cool to be able to decompile a regex. Imagine a regex pretty-printer: compile an arbitrary string, and if it's a valid regex, decompile it to a valid source code form, using re.VERBOSE. That could help _hugely_ with debugging, if the trick can be pulled off. ChrisA From stephen at xemacs.org Tue Feb 16 04:25:59 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 16 Feb 2016 18:25:59 +0900 Subject: [Python-ideas] Generator unpacking In-Reply-To: <56C233A7.4040707@canterbury.ac.nz> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> <56C233A7.4040707@canterbury.ac.nz> Message-ID: <22210.60199.52543.191353@turnbull.sk.tsukuba.ac.jp> Greg Ewing writes: > This is a philosophical question. When you've eaten > the last chocolate, do you have an empty box of > chocolates, or just an empty box? That depends on whether you've lifted to the chocolate monad or not. Wondering-if-Saunders-Mac-Lane-liked-chocolate-ly y'rs, From p.f.moore at gmail.com Tue Feb 16 04:55:06 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 16 Feb 2016 09:55:06 +0000 Subject: [Python-ideas] Exposing regular expression bytecode In-Reply-To: References: Message-ID: On 16 February 2016 at 06:04, Jonathan Goble wrote: > Hello? Anyone? I'd really like to see this idea move forward, but I > need some comments for that. Sorry. I don't personally have any issue with the proposal, and it sounds like a reasonable idea. I don't think it's likely to be *hugely* controversial - although it will likely need a little care in documenting the feature to ensure that we are clear that there's no guarantees of backward compatibility that we don't want to commit to on the newly - exposed data. And we should also ensure that by exposing this information, we don't preclude changes such as the incorporation of the regex module (I don't know if the regex module has a bytecode implementation like the re module does). The next step is probably simply to raise a tracker issue for this. I know you said you have little C experience, but the big problem is that it's unlikely that any of the core devs with C experience will have the time or motivation to code up your idea. So without a working patch, and someone willing and able to respond to comments on the patch, it's not likely to progress. But if you are willing to dig into Python's C API yourself (and it sounds like you are) there are definitely people who will help you. You might want to join the core mentorship list (see http://pythonmentors.com/) where you should get plenty of assistance. This proposal sounds like a great "beginner" task, as well - so even if you don't want to implement it yourself, still put it on the tracker, and mark it as an "easy" change, and maybe some other newcomer who wants a task to help them learn the C API will pick it up. Hope that helps - thanks for the suggestion and sorry if it seems like no-one was interested at first. It's an unfortunate fact of life around here that things *do* take time to get people's interest. You mention patience in one of your messages - that's definitely something you'll need to cultivate, I'm afraid... :-) Paul From srkunze at mail.de Tue Feb 16 05:07:07 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 16 Feb 2016 11:07:07 +0100 Subject: [Python-ideas] Generator unpacking In-Reply-To: <20160215234322.GZ31806@ando.pearwood.info> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> <20160215144251.GY31806@ando.pearwood.info> <56C20D56.2030605@stoneleaf.us> <56C210A9.4020002@mail.de> <20160215234322.GZ31806@ando.pearwood.info> Message-ID: <56C2F4CB.7030709@mail.de> On 16.02.2016 00:43, Steven D'Aprano wrote: > No, it's exactly the same Python idiom (assignment to a list of targets) > as we've been talking about for the last few posts. I think we better distinguish between idioms and language features. > We've had examples > with four targets, three targets, two targets and zero targets. This is > an example with one target. > > [a] = iterable > > requires that the right-hand side be iterable, and after unpacking it > must contain exactly one item, to match the one assignment target given > on the left. Of course, it's quite straightforward once you ponder about it. I recently talked to a coworker about this. The concrete example is about "How do I get the one-and-only element of a **set** which obviously does not support subscripting". Another aspect, I came to think of is the following asymmetry: a, b, c, d = mylist4 # works a, b, c = mylist3 # also works a, b = mylist2 # works too [a] = mylist1 # special case? One might resolve the asymmetry by writing: [a, b, c, d] = mylist4 [a, b, c] = mylist3 [a, b] = mylist2 [a] = mylist1 # fits in [] = mylist0 # even that is possible now Thus, the parentheses-less variants are special cases. However, when glancing over our production source, the parentheses-less variant is rather the norm than a special case (haven't even seen nested ones). I suspect a special-character-phobia (ever used a German keyboard ;-) ?) -- Why should I write '''[a, b] = [b, a]''' when '''a, b = b, a''' suffices? So, it seems to me that inducing the 1-target and 0-target concepts from the norm is not that as easy as you might believe. Last but not least, most devs are not used to magic on the lhs. Imagine how weird and interesting you would find that the following being possible in Python: mylist = [6, 7] a*3, b+5 = mylist Best, Sven From srkunze at mail.de Tue Feb 16 05:11:42 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 16 Feb 2016 11:11:42 +0100 Subject: [Python-ideas] Generator unpacking In-Reply-To: <6E6AD40C-826A-4159-801F-F23EF12FF6E7@yahoo.com> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> <20160215144251.GY31806@ando.pearwood.info> <56C20D56.2030605@stoneleaf.us> <56C210A9.4020002@mail.de> <20160215234322.GZ31806@ando.pearwood.info> <6E6AD40C-826A-4159-801F-F23EF12FF6E7@yahoo.com> Message-ID: <56C2F5DE.7080004@mail.de> On 16.02.2016 01:11, Andrew Barnert via Python-ideas wrote: > I see this more often written in tuple-ish form: > > index, value = struct.unpack_from('!HH', buf, off) > namelen, = struct.unpack_from('!H', buf, off+4) > > Somehow, that feels more natural and pythonic than using [index, value] and [namelen], despite the fact that without the brackets it's easy to miss the trailing comma and end up with (16,) instead of 16 as your length and a confusing TypeError a few lines down. You are right. It seems to me that without the parentheses it's not regarded a tuple but a special feature syntax; so all the known peculiarities of tuples aren't recognized properly. Best, Sven From srkunze at mail.de Tue Feb 16 05:21:19 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 16 Feb 2016 11:21:19 +0100 Subject: [Python-ideas] Generator unpacking In-Reply-To: <20160216003443.GA31806@ando.pearwood.info> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> <20160215144251.GY31806@ando.pearwood.info> <56C1EB0D.70505@mail.de> <20160216003443.GA31806@ando.pearwood.info> Message-ID: <56C2F81F.9010608@mail.de> On 16.02.2016 01:34, Steven D'Aprano wrote: > On Mon, Feb 15, 2016 at 01:55:32PM -0800, Andrew Barnert via Python-ideas wrote: >> On Feb 15, 2016, at 07:13, Sven R. Kunze wrote: >>> And while we are on it, that reminds me of KeyError. I always feel >>> annoyed by the fact that Python doesn't tell me what key a dict is >>> missing. > Showing the missing key in the error message goes all the way back to > Python 1.5: > > [steve at ando ~]$ python1.5 -c '{}["spam"]' > Traceback (innermost last): > File "", line 1, in ? > KeyError: spam Sorry. I confused KeyError with IndexError (bracket all over the place). >>> [1][1] Traceback (most recent call last): File "", line 1, in IndexError: list index out of range > Admittedly, the exception object itself doesn't keep a reference to the > missing key, so you can't programmatically query it for the key, but > normally that's not a problem since you just tried to look it up so you > should still have the key. That's true. However, most the time I see such tracebacks while sifting through server logs. So, the more data an exception exposes, the better. I wouldn't even mind an excerpt of the dict/list itself in order to get a faster understanding of the problem domain (wrong type of keys, etc.). >> But for SyntaxError, there is no object to stick in the error object >> or error message, just a source token or an AST node. Both of those >> have pointers back to start and end in the source string, and that's >> what goes into the SyntaxError, which is already shown to the user by >> printing the relevant line of source and pointing a caret at it. > Sadly, CPython doesn't manage to even display the caret at all: > > [steve at ando ~]$ python -c "[a, 2] = 'xy'" > File "", line 1 > SyntaxError: can't assign to literal > > > > Jython gives a much more informative error: > > steve at orac:~$ jython -c "[a, 2] = 'ab'" > File "", line 1 > [a, 2] = 'ab' > ^ > SyntaxError: can't assign to number That seems to be a great idea. Best, Sven From stephen at xemacs.org Tue Feb 16 05:49:11 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 16 Feb 2016 19:49:11 +0900 Subject: [Python-ideas] Generator unpacking In-Reply-To: <4CAB96F6-F814-43D8-93A6-A8AC00BF06F5@yahoo.com> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <40592F59-371A-4440-B58E-7EC83DE54713@yahoo.com> <56C1987D.7050209@brenbarn.net> <083415BF-CD35-46BD-B272-AA8F8E4C1D7E@yahoo.com> <22210.39571.987984.507976@turnbull.sk.tsukuba.ac.jp> <4CAB96F6-F814-43D8-93A6-A8AC00BF06F5@yahoo.com> Message-ID: <22210.65191.32136.404624@turnbull.sk.tsukuba.ac.jp> Andrew Barnert writes: > On Feb 15, 2016, at 19:42, Stephen J. Turnbull wrote: > > Are there really places where we *want* "foo(range(n))" to raise but > > "foo(iter(range(n)))" to do something useful? In other words, is this > > really a terminology problem, rather than a problem with the design of > > APIs that take iterators rather than iterables? > > Sure. Any multi-pass algorithm should raise when given an > iterator. That's not my problem. I'm not suggesting that the coercion between iterators and non-iterators should go both directions. If a function that wants a Sequence is handed an iterator it should feel free to raise. > In the other direction, of course, the next function requires an > iterator, and should and does raise when given something else. Sure. But next() *can't* work if the object passed doesn't maintain internal state. I don't think it will bother anybody if next raises on a non-iterator, at least no more than it bothers people when something raises on an attempt to assign to a tuple element. My question is "why are they iter[ator]-tools, not iter[able]-tools?" Mostly because sequences don't need those tools very often, I guess. > And the slightly higher-level functions discussed in this thread > are the same. Their implementations are, yes (except for the lhs ellipsis syntax, which doesn't exist yet). > A function intended to consume and return the first few values in > an iterator while leaving the iterator holding the rest is > basically just a "multinext", and it would be very confusing if it > took a collection and left it holding all the values, including the > two you already used. Greg Ewing evidently thinks that's the natural thing, so I accept that it's not un-natural. And you can easily change that behavior *without* perverse effects on space or time with iter(), at the expense of choosing a name, and an assignment statement. But there's an alternative interpretion of what we're looking at here with ellipsis, at least, and that's "multipop", in which case you do expect it to consume a Sequence, and all iterables would behave the same in that respect. Again, you can change that behavior with an appropriate call to iter() (and this time you don't even need to choose a name). I agree that we can't get to win-win-win here, after all both iterators and lists are optimizations of iterable, and that always costs you something. But I wonder if it wouldn't be possible to take the attitude that we should do our best to optimize either list() or iter() out of Python programs. Since the costs of willy-nilly applying list() to arbitrary iterables are obvious, iter() is the one we should try to make implicit where possible. We can't get rid of it entirely (as with multinext vs. multipop, where either choice leaves us wanting to call iter() for some use cases). But maybe we could reduce the frequency of cases where it's all too easy to forget to call it. > If the current terminology were sufficiently clear and intuitive > that nothing needed to be done, these problems wouldn't exist. That takes the current implementation as given, which is exactly what I'm questioning. From abarnert at yahoo.com Tue Feb 16 06:04:47 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 16 Feb 2016 03:04:47 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: <56C2F5DE.7080004@mail.de> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> <20160215144251.GY31806@ando.pearwood.info> <56C20D56.2030605@stoneleaf.us> <56C210A9.4020002@mail.de> <20160215234322.GZ31806@ando.pearwood.info> <6E6AD40C-826A-4159-801F-F23EF12FF6E7@yahoo.com> <56C2F5DE.7080004@mail.de> Message-ID: <92DCECC7-2E98-49F8-AC25-515473BBD8E0@yahoo.com> On Feb 16, 2016, at 02:11, Sven R. Kunze wrote: > >> On 16.02.2016 01:11, Andrew Barnert via Python-ideas wrote: >> I see this more often written in tuple-ish form: >> >> index, value = struct.unpack_from('!HH', buf, off) >> namelen, = struct.unpack_from('!H', buf, off+4) >> >> Somehow, that feels more natural and pythonic than using [index, value] and [namelen], despite the fact that without the brackets it's easy to miss the trailing comma and end up with (16,) instead of 16 as your length and a confusing TypeError a few lines down. > > You are right. It seems to me that without the parentheses it's not regarded a tuple but a special feature syntax; so all the known peculiarities of tuples aren't recognized properly. It's not a tuple _with_ the parentheses either: it's a parenthesized target list, which has a syntax and semantics that are, while not completely unrelated to tuple, definitely not the same. One way in which it _is_ like a tuple is that this doesn't help: (namelen) = struct.unpack_from('!H', buf, off+4) Just like tuples, parenthesized target lists need commas, so this will bind namelen to the tuple (16,) instead of to 16: From srkunze at mail.de Tue Feb 16 07:06:29 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 16 Feb 2016 13:06:29 +0100 Subject: [Python-ideas] Generator unpacking In-Reply-To: <4CAB96F6-F814-43D8-93A6-A8AC00BF06F5@yahoo.com> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <40592F59-371A-4440-B58E-7EC83DE54713@yahoo.com> <56C1987D.7050209@brenbarn.net> <083415BF-CD35-46BD-B272-AA8F8E4C1D7E@yahoo.com> <22210.39571.987984.507976@turnbull.sk.tsukuba.ac.jp> <4CAB96F6-F814-43D8-93A6-A8AC00BF06F5@yahoo.com> Message-ID: <56C310C5.6030404@mail.de> On 16.02.2016 06:56, Andrew Barnert via Python-ideas wrote: > On Feb 15, 2016, at 19:42, Stephen J. Turnbull wrote: >> Are there really places where we *want* "foo(range(n))" to raise but >> "foo(iter(range(n)))" to do something useful? In other words, is this >> really a terminology problem, rather than a problem with the design of >> APIs that take iterators rather than iterables? > Sure. Any multi-pass algorithm should raise when given an iterator. (Maybe some could instead copy the iterator to a list or something, but as a general rule, silently and unexpectedly jumping from 0 to O(N) space depending on the input type isn't very friendly.) > > In the other direction, of course, the next function requires an iterator, and should and does raise when given something else. And the slightly higher-level functions discussed in this thread are the same. A function intended to consume and return the first few values in an iterator while leaving the iterator holding the rest is basically just a "multinext", and it would be very confusing if it took a collection and left it holding all the values, including the two you already used. And how do you explain that to a human? You could say "While it's an iterable, it's not an iterator." And that's exactly what Python has said for the last however-many years. And that's exactly what leads to the confusion Paul was talking about (and the additional confusion he inadvertently revealed). And the problem isn't "Paul is an idiot", because I've seen multiple core devs, and other highly experienced Python programmers, make the same mistake. Even the official docs made a similar mistake, and nob > ody caught it for 4 years. If the current terminology were sufficiently clear and intuitive that nothing needed to be done, these problems wouldn't exist. I think both is true. The current API, being one of the best available out there, exposes some warts (as you described) which in turn lead to a plethora of abstract wording expressing almost the same: - lists, iterables, iterators, sets, generators, views, range, etc. These datastructures evolved over time and proved useful. In my point of view, what's missing a structure to put them into. If one looks closely, one can see how they differ in usage: * *1) Is it iterable?* *- yes to all (as far as I know)* ** *2) Is it subscriptable?* * - yes: list, range - no: iterator, generator, set, view * *3) Has it a length?* * - yes: list, range, set, view - no: iterator, generator *4) Has it a contains test?* - yes: list, range, set, view - no: iterator, generator * 5) Are items materialized?* - yes: list, set - no: iterator, generator, range, view *6) Can it have mutable underlying data?* - yes: generator, iterator, view - no: list, set, range *7) Does iteration change it?* - yes: iterator, generator - no: list, set, range, view As expected, they almost always have nothing in common except 1). I am perfectly fine with calling these things collections; if I have a container of n things, I almost certainly can go through them one by one. 2) to 4) are usually the most common operations and the safe when implemented. range is like list; view is like set. 5) to 7) are usually things you worry about when you are concerned with performance. As usual, those things are hard to do right. *My suggestions:* Let's call all of them *collections*. Let's call collections *list-like* if they support 2) to 4) and *set-like* if they support 4). Let's call collections *lazy* when they don't fit 5). Let's call collections *view-like* when they feature 6). Let's call collections *iteration-stable* when they don't fit 7). As you can see, for me it's not a matter of inventing new nouns, but a matter of finding the right adjective to further specify the type of collection. Does it help? I don't know but it helped me to structure the concepts. Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Feb 16 07:28:50 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 16 Feb 2016 23:28:50 +1100 Subject: [Python-ideas] Generator unpacking In-Reply-To: <9C78842E-B30A-43D4-BDC2-16CCA19B4656@yahoo.com> References: <56BFA2CA.2080404@canterbury.ac.nz> <40592F59-371A-4440-B58E-7EC83DE54713@yahoo.com> <56C1987D.7050209@brenbarn.net> <083415BF-CD35-46BD-B272-AA8F8E4C1D7E@yahoo.com> <56C25412.8050906@brenbarn.net> <05B2AA37-18B1-4402-BE91-8893077CD4C4@yahoo.com> <56C29BC2.8090607@btinternet.com> <9C78842E-B30A-43D4-BDC2-16CCA19B4656@yahoo.com> Message-ID: <20160216122849.GB31806@ando.pearwood.info> On Mon, Feb 15, 2016 at 10:02:26PM -0800, Andrew Barnert via Python-ideas wrote: > Because "reiterable" is really not immediately obvious. It has no > intuitive meaning; it's not even a real word. You have to learn what a > reiterable is, by definition. What's a "real word"? Is "closure" a real word? How about "iterator"? "Closure" is in the Oxford dictionary, but it doesn't mention anything about functions and variables from surrounding scopes. "Iterator" isn't in the Oxford dictionary at all; neither is "iterable". As far as intuitive meanings go, "re-" is a standard English prefix meaning (among other things) "again", as in reborn, renew, retake, rewind, retry -- and "reiterate". > That's no easier than learning a simple > fact about collections, and you end up knowing less after that > learning. I'm not sure I understand how learning the definition of "reiterable" means I know *less* than when I started. Presumably that means I have forgotten something that I previously knew. I hope it wasn't something important. > Think of my random example. I would, but I've forgotten what it is. > Is that a reiterable? Well, you can call > iter on its multiple times and get different iterators. But they don't > iterate the same value. So, yes or no? There's no intuition to guide > you there; all you have is a definitional rule that you had to > memorize. I'm honestly not sure what actual problem you think this word is going to solve. I can see that there is some difficulty with people confusing the concepts of iterables and iterators, and some edge cases where people mistake "lazy sequences" like (x)range for an iterator. I acknowledge those difficulties. But: (1) I don't think those difficulties are worth the thousands of words written in this thread so far. (2) I don't understand how redefining (there's that re- prefix again) "collection" or adding "reiterable" will help. I think the meanings of iterable and iterator are already clearly documented, and people still get them confused, so how will adding "reiterable" help? (3) I don't think that the confusion between lazy sequences and iterators is especially harmful. 9 times out of 10, it's a difference that makes no difference. I like to be pedantic and tell people that (x)range is not an iterator, but if I'm honest to myself, that fact is rarely going to help them write better code. (It won't make them write worse code either.) (4) You've already acknowledged that "collection" currently has a meaning in Python. It's normally used for lists, tuples and dicts, as well as the things in the `collections` module. I don't understand why "collection" is your preferred term for an iterable that can be iterated over multiple times (a *re-iterable* in plain English). I think that this problem is similtaneously too big to solve, and too trivial to care about solving. As I see it, we have a great many related categories of things that can be iterated over. Some of them are represented by concrete or abstract classes in the standard library, some of them aren't. Many of them overlap: * eager sequences, mappings and sets; * lazy sequences, mappings or sets; * things which support the `in` operator (collections); * things that obey the iterator protocol ("iterators"); * things which obey the old sequence protocol; * things which obey the iterator protocol, except for the part about "once they become empty and raise StopIteration, they should always raise StopIteration" -- these are officially called "broken iterators"; * things which are broken iterators with an official API for restarting them (perhaps called "reiterators"?); * reiterators which can be restarted, but won't necessarily give the same results each time; * iterators that provide a look-ahead or look-behind method; * sequences that use the function-call API, like random.random(); * things that you can pass to iter(); * things which you can pass to iter() multiple times, and get iterators which yield the same values each time; * things which you can pass to iter() multiple times, and get iterators which DON'T yield the same values each time; and probably more. I don't think it is practical, or necessary, to try naming them all. We have a good set of names already: - collection for things that support `in`; - iterable for things that can be iterated over; - iterator for a particular subset of iterables; - sequence for things like lists; - lazy sequence for things that are sequences where the items are calculated on demand rather than in advance. I think adding to that set of names is a case of YAGNI. I do see some sense in having a name for iterators with a "restart" method, and the obvious name for that would be reiterator. But I don't think the glossary needs to have that entry until such a time there actually is a restartable iterator in the std lib. I *don't* think there's any need for a name for "iterables apart from iterators". While it is occasionally useful to be able to talk about such, that's not frequent enough that we need a name for it. We can just say "iterables apart from iterators", or we can give concrete examples: "sequences, mappings and sets". > Similarly, you can't meaningfully attach adjectives to "reiterable" > that restrict or expand its scope. "Lazy reiterable" doesn't mean > anything; "lazy collection" does. I don't see why you think lazy reiterable doesn't mean anything. It is clearly a reiterable where the items are calculated on demand rather than in advance. What else could it mean? > But I've already made all these points and you just skipped over them. It's a huge thread, with (no offense) a lot of rambling, pedantic arguments over the number of angels that can dance on the head of a pin (what sort of angels? seraphim, cherubim, ophanim, malakhim, dominions, powers, virtues, or something else? what size pin?). It's easy to get lost in the discussion, partly (I believe) because the problem being solved is so abstract. This seems to me to be a debate over definitions for their own sake, not in order to solve an actual problem. I could be wrong, of course. But if there is anything actually concrete and important being solved by this proposal, it is well and truly hidden in the thousands of words of abstract argument. -- Steve From rob.cliffe at btinternet.com Tue Feb 16 08:24:46 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Tue, 16 Feb 2016 13:24:46 +0000 Subject: [Python-ideas] Dict(x, y) -> Dict(zip(x, y)) In-Reply-To: References: Message-ID: <56C3231E.5060100@btinternet.com> I think the zip is meaningful enough that it should not be omitted. Someone seeing dict(x, y) for the first time might think it meant { x : y } or the combination of 2 dictionaries, x and y. Rob Cliffe On 15/02/2016 00:30, Oscar Benjamin wrote: > I often find myself writing dict(zip(x, y)). Maybe that's just me or > maybe not but I would like it if it were possible to spell that simply > as dict(x, y). > > Currently dict() with two arguments is an error: > > $ python3 > Python 3.4.3 (default, Mar 26 2015, 22:03:40) > [GCC 4.9.2] on linux > Type "help", "copyright", "credits" or "license" for more information. >>>> dict(['foo', 'bar'], [1, 2]) > Traceback (most recent call last): > File "", line 1, in > TypeError: dict expected at most 1 arguments, got 2 > > I would prefer it if dict(x, y) would treat x as an iterable of keys > and y as an iterable of corresponding values. This would be > semantically equivalent to dict(zip(x, y)) except that it would raise > an error if x and y don't yield the same number of items (zip > truncates the longer argument). > > Although mostly equivalent the new construction might be more > efficient, would reduce the visual noise of the redundant call to zip, > and would be slightly more robust to errors. > > -- > Oscar > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From g.brandl at gmx.net Tue Feb 16 12:54:21 2016 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 16 Feb 2016 18:54:21 +0100 Subject: [Python-ideas] Generator unpacking In-Reply-To: <56C2F4CB.7030709@mail.de> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> <20160215144251.GY31806@ando.pearwood.info> <56C20D56.2030605@stoneleaf.us> <56C210A9.4020002@mail.de> <20160215234322.GZ31806@ando.pearwood.info> <56C2F4CB.7030709@mail.de> Message-ID: On 02/16/2016 11:07 AM, Sven R. Kunze wrote: > On 16.02.2016 00:43, Steven D'Aprano wrote: >> No, it's exactly the same Python idiom (assignment to a list of targets) >> as we've been talking about for the last few posts. > > I think we better distinguish between idioms and language features. > >> We've had examples >> with four targets, three targets, two targets and zero targets. This is >> an example with one target. >> >> [a] = iterable >> >> requires that the right-hand side be iterable, and after unpacking it >> must contain exactly one item, to match the one assignment target given >> on the left. > > Of course, it's quite straightforward once you ponder about it. I > recently talked to a coworker about this. The concrete example is about > "How do I get the one-and-only element of a **set** which obviously does > not support subscripting". > > > Another aspect, I came to think of is the following asymmetry: > > a, b, c, d = mylist4 # works > a, b, c = mylist3 # also works > a, b = mylist2 # works too > [a] = mylist1 # special case? a, = mylist1 actually. Same "special case" as with 1-element tuples. Georg From ethan at stoneleaf.us Tue Feb 16 13:29:19 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 16 Feb 2016 10:29:19 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> <20160215144251.GY31806@ando.pearwood.info> <56C20D56.2030605@stoneleaf.us> <56C210A9.4020002@mail.de> <20160215234322.GZ31806@ando.pearwood.info> <56C2F4CB.7030709@mail.de> Message-ID: <56C36A7F.3030908@stoneleaf.us> On 02/16/2016 11:07 AM, Sven R. Kunze wrote: > Another aspect, I came to think of is the following asymmetry: > > a, b, c, d = mylist4 # works > a, b, c = mylist3 # also works > a, b = mylist2 # works too > [a] = mylist1 # special case? The asymmetry is because you are mixing two different techniques: a, b, c, d = mylist4 a, b, c = mylist3 a, b = mylist2 a, = mylist1 or [a, b, c, d] = mylist4 [a, b, c] = mylist3 [a, b] = mylist2 [a] = mylist1 -- ~Ethan~ From jcgoble3 at gmail.com Tue Feb 16 13:29:33 2016 From: jcgoble3 at gmail.com (Jonathan Goble) Date: Tue, 16 Feb 2016 13:29:33 -0500 Subject: [Python-ideas] Exposing regular expression bytecode In-Reply-To: <56C34D84.40509@gmail.com> References: <475EF1D2-E27E-4AB6-9729-66B00470D60F@yahoo.com> <56C34D84.40509@gmail.com> Message-ID: OK, thanks for the comments, everyone. I'm glad to hear that people generally think this is a useful idea. Some specific replies: On Tue, Feb 16, 2016 at 4:22 AM, Chris Angelico wrote: > For what it's worth, I read your post with interest, but didn't have > anything substantive to reply - mainly because I don't use regexes > much. But it would be rather cool to be able to decompile a regex. > Imagine a regex pretty-printer: compile an arbitrary string, and if > it's a valid regex, decompile it to a valid source code form, using > re.VERBOSE. That could help _hugely_ with debugging, if the trick can > be pulled off. > > ChrisA That's exactly the type of tools I envision being made available by third parties. Depending on how much I get invested into this project, I may even write such a tool myself (though that's not guaranteed). On Tue, Feb 16, 2016 at 4:55 AM, Paul Moore wrote: > Sorry. I don't personally have any issue with the proposal, and it > sounds like a reasonable idea. I don't think it's likely to be > *hugely* controversial - although it will likely need a little care in > documenting the feature to ensure that we are clear that there's no > guarantees of backward compatibility that we don't want to commit to > on the newly - exposed data. And we should also ensure that by > exposing this information, we don't preclude changes such as the > incorporation of the regex module (I don't know if the regex module > has a bytecode implementation like the re module does). The regex implementation is indeed something I would need to investigate here, and will do so before I go too far. > The next step is probably simply to raise a tracker issue for this. I > know you said you have little C experience, but the big problem is > that it's unlikely that any of the core devs with C experience will > have the time or motivation to code up your idea. So without a working > patch, and someone willing and able to respond to comments on the > patch, it's not likely to progress. Tracker issue is already filed: http://bugs.python.org/issue26336 I actually filed the issue before I realized that the mailing lists were a better place to discuss it. > But if you are willing to dig into Python's C API yourself (and it > sounds like you are) there are definitely people who will help you. > You might want to join the core mentorship list (see > http://pythonmentors.com/) where you should get plenty of assistance. > This proposal sounds like a great "beginner" task, as well - so even > if you don't want to implement it yourself, still put it on the > tracker, and mark it as an "easy" change, and maybe some other > newcomer who wants a task to help them learn the C API will pick it > up. I'll look into the mentorship list; thanks for the link. As for marking it "easy", I don't seem to have the necessary permissions to change the Keywords field; perhaps you or someone else can set that flag for me? If so, I'd appreciate it. :-) > Hope that helps - thanks for the suggestion and sorry if it seems like > no-one was interested at first. It's an unfortunate fact of life > around here that things *do* take time to get people's interest. You > mention patience in one of your messages - that's definitely something > you'll need to cultivate, I'm afraid... :-) Patience is something I've been working on since I was a little kid. I'm 29 years old now, and it still eludes me from time to time. But yes, it's something I'll have to work on. :-P Also, I received a small patch off-list from Petr Viktorin implementing a getter for the code list (thanks, Petr). I'll need to test it, but from the little I know of the C API it looks like it will get me started in the right direction. Assuming that works, what's left is a public constructor for the regex type (to enable optimizers), a dis-like module, and docs and tests. I don't think this would be major enough to require a PEP, but of course being new here, I'm open to being told I'm wrong. :-) From tjreedy at udel.edu Tue Feb 16 16:07:34 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 16 Feb 2016 16:07:34 -0500 Subject: [Python-ideas] Explicit variable capture list In-Reply-To: References: <20160120003712.GZ10854@ando.pearwood.info> <20160121001027.GB4619@ando.pearwood.info> <56A3ED9B.6030009@canterbury.ac.nz> Message-ID: On 1/25/2016 1:52 PM, Guido van Rossum wrote: > On Sun, Jan 24, 2016 at 10:32 PM, Terry Reedy wrote: >> What I've concluded from this thread is that function definitions (with >> direct use 'def' or 'lambda') do not fit well within loops, though I used >> them there myself. > > Right. When you can avoid them, you avoid extra work in an inner loop, > which is often a good idea. > >> When delayed function calls are are needed, what belongs within loops is >> packaging of a pre-defined function with one or more arguments within a >> callable. Instance.method is an elegant syntax for doing so. >> functools.partial(func, args, ...) is a much clumsier generalized >> expression, which requires an import. Note that 'partial' returns a >> function for delayed execution even when a complete, not partial, set of >> arguments is passed. > > Right. I've always hated partial() (which is why it's not a builtin) > because usually a lambda is clearer (it's difficult to calculate in > your head the signature of the thing it returns from the arguments > passed), but this is one thing where partial() wins, since it captures > values. I agree that the difficulty of immediately grokking the signature partials that binds arbitrary parameters is a downside to their use. Fake parameters set to a constant necessarily go at the end of the parameter list. The actual signature is the list with those parameters chopped off and ignored. To eliminate the possibility of accidentally supplying a different value positionally, such parameters could (and I think should) be made keyword-only. def f(a, b='default', *, int=int): pass Bound methods necessarily bind the first parameter, often called 'self'. This again makes the actual signature easy to determine. >> A major attempted (and tempting) use for definitions within a loop is >> multiple callbacks for multiple gui widgets, where delayed execution is >> needed. The three answers to multiple 'why doesn't this work' on both >> python-list and Stackoverflow are multiple definitions with variant 'default >> args', a custom make_function function outside the loop called multiple >> times within the loop, and a direct function outside the loop called with >> partial within the loop. I am going to start using partial more. Since writing this, I realized that defining a custom class and using bound methods is a fourth option, which I also like. This binds the differentiating data to an instance, which is then bound to the function, rather than to the function directly. A toy example: ---- import tkinter as tk root = tk.Tk() class Card(tk.Button): hide = 'XXX' def __init__(self, txt): tk.Button.__init__(self, root, text=self.hide) # or # super().__init__(root, text=self.hide) self.txt = txt self.exposed = False def flip(self): self['text'] = self.hide if self.exposed else self.txt self.exposed = not self.exposed for i, txt in enumerate(('one', 'two')): card = Card(txt) card['command'] = card.flip card.grid(row=0, column=i) #root.mainloop() # uncomment if run on command line without -i ---- The main problem with this is that some beginners are trying to write (or being told to write) tkinter guis before they learn about class statements. The super() form is easier to write, but its use is even more 'advanced'. -- Terry Jan Reedy From k7hoven at gmail.com Tue Feb 16 17:03:52 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Wed, 17 Feb 2016 00:03:52 +0200 Subject: [Python-ideas] Dict(x, y) -> Dict(zip(x, y)) In-Reply-To: References: Message-ID: On Feb 15, 2016 02:31, "Oscar Benjamin" wrote: > > I often find myself writing dict(zip(x, y)). Maybe that's just me or > maybe not but I would like it if it were possible to spell that simply > as dict(x, y). > Not sure how useful this is, but what about {*keys : *values} ? -- Koos -------------- next part -------------- An HTML attachment was scrubbed... URL: From ian at feete.org Tue Feb 16 18:48:02 2016 From: ian at feete.org (Ian Foote) Date: Tue, 16 Feb 2016 23:48:02 +0000 Subject: [Python-ideas] Dict(x, y) -> Dict(zip(x, y)) In-Reply-To: References: Message-ID: <56C3B532.8020201@feete.org> On 15/02/16 09:53, Andrew Barnert via Python-ideas wrote: > > But what about a classmethod named constructor (with or without > keyword-only params, depending on how long the name is): > > dict.kv(keys=spam, values=eggs) > dict.from_keys_values(spam, eggs) > I would spell this dict.zip(keys, values). It brings to mind the current implementation, but allows for optimisation. It's also slightly less parenthesis heavy, which I think makes it a little more readable. > > Still, it raises the obvious question of whether "dict.kv" is much > better than "dictkv", which anyone can write for himself as a one-liner > if he wants it... > I think this is one of those cases where the gain is small enough that writing the helper function is often not worth it, but big enough that people would use it if it already existed. That might be a small window, but it's there. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: OpenPGP digital signature URL: From python at mrabarnett.plus.com Tue Feb 16 19:34:44 2016 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 17 Feb 2016 00:34:44 +0000 Subject: [Python-ideas] Dict(x, y) -> Dict(zip(x, y)) In-Reply-To: <56C3B532.8020201@feete.org> References: <56C3B532.8020201@feete.org> Message-ID: <56C3C024.2030604@mrabarnett.plus.com> On 2016-02-16 23:48, Ian Foote wrote: > On 15/02/16 09:53, Andrew Barnert via Python-ideas wrote: > >> >> But what about a classmethod named constructor (with or without >> keyword-only params, depending on how long the name is): >> >> dict.kv(keys=spam, values=eggs) >> dict.from_keys_values(spam, eggs) >> Going the other way it's "my_dict.items()", so perhaps "dict.from_items(spam, eggs)"? > > I would spell this dict.zip(keys, values). It brings to mind the current > implementation, but allows for optimisation. It's also slightly less > parenthesis heavy, which I think makes it a little more readable. > >> >> Still, it raises the obvious question of whether "dict.kv" is much >> better than "dictkv", which anyone can write for himself as a one-liner >> if he wants it... >> > > I think this is one of those cases where the gain is small enough that > writing the helper function is often not worth it, but big enough that > people would use it if it already existed. That might be a small window, > but it's there. > From abarnert at yahoo.com Tue Feb 16 20:05:13 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 16 Feb 2016 17:05:13 -0800 Subject: [Python-ideas] Dict(x, y) -> Dict(zip(x, y)) In-Reply-To: <56C3C024.2030604@mrabarnett.plus.com> References: <56C3B532.8020201@feete.org> <56C3C024.2030604@mrabarnett.plus.com> Message-ID: On Feb 16, 2016, at 16:34, MRAB wrote: > >> On 2016-02-16 23:48, Ian Foote wrote: >>> On 15/02/16 09:53, Andrew Barnert via Python-ideas wrote: >>> >>> >>> But what about a classmethod named constructor (with or without >>> keyword-only params, depending on how long the name is): >>> >>> dict.kv(keys=spam, values=eggs) >>> dict.from_keys_values(spam, eggs) > Going the other way it's "my_dict.items()", so perhaps "dict.from_items(spam, eggs)"? But items() is the inverse of the plain constructor, which takes an iterable of items. In other words, both items() and the constructor deal in a single iterable of zipped key-value pairs. The proposed new method or function instead deals in unzipped, separate key and value iterables. So it's the inverse of "my_dict.keys(), my_dict.values()". Of course you can also write that as "zip(*my_dict.items())", just as you can also write the new method as "dict(zip(keys, values))". At any rate, since you can write any variation of this as a one-liner, as long as you're willing to spell it "dict_spam" instead of "dict.spam", each person can call it whatever they want. :) From ian at feete.org Tue Feb 16 20:15:51 2016 From: ian at feete.org (Ian Foote) Date: Wed, 17 Feb 2016 01:15:51 +0000 Subject: [Python-ideas] Dict(x, y) -> Dict(zip(x, y)) In-Reply-To: <56C3C024.2030604@mrabarnett.plus.com> References: <56C3B532.8020201@feete.org> <56C3C024.2030604@mrabarnett.plus.com> Message-ID: <56C3C9C7.6090100@feete.org> On 17/02/16 00:34, MRAB wrote: > On 2016-02-16 23:48, Ian Foote wrote: >> On 15/02/16 09:53, Andrew Barnert via Python-ideas wrote: >> >>> >>> But what about a classmethod named constructor (with or without >>> keyword-only params, depending on how long the name is): >>> >>> dict.kv(keys=spam, values=eggs) >>> dict.from_keys_values(spam, eggs) >>> > Going the other way it's "my_dict.items()", so perhaps > "dict.from_items(spam, eggs)"? > Given a dictionary d, I would expect your dict.from_items to satisfy: dict.from_items(d.items()) == d But this is how dict already behaves: dict(d.items()) == d. What a new classmethod would satisfy is: dict.zip(d.keys(), d.values()) == d -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: OpenPGP digital signature URL: From python at mrabarnett.plus.com Tue Feb 16 20:36:21 2016 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 17 Feb 2016 01:36:21 +0000 Subject: [Python-ideas] Dict(x, y) -> Dict(zip(x, y)) In-Reply-To: References: <56C3B532.8020201@feete.org> <56C3C024.2030604@mrabarnett.plus.com> Message-ID: <56C3CE94.2070109@mrabarnett.plus.com> On 2016-02-17 01:05, Andrew Barnert wrote: > On Feb 16, 2016, at 16:34, MRAB wrote: > > > >> On 2016-02-16 23:48, Ian Foote wrote: > >>> On 15/02/16 09:53, Andrew Barnert via Python-ideas wrote: > >>> > >>> > >>> But what about a classmethod named constructor (with or without > >>> keyword-only params, depending on how long the name is): > >>> > >>> dict.kv(keys=spam, values=eggs) > >>> dict.from_keys_values(spam, eggs) > > Going the other way it's "my_dict.items()", so perhaps "dict.from_items(spam, eggs)"? > > But items() is the inverse of the plain constructor, which takes an iterable of items. In other words, both items() and the constructor deal in a single iterable of zipped key-value pairs. [snip] Ah, of course! Obvious when you point it out... From stephen at xemacs.org Tue Feb 16 21:25:04 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 17 Feb 2016 11:25:04 +0900 Subject: [Python-ideas] Exposing regular expression bytecode In-Reply-To: References: <475EF1D2-E27E-4AB6-9729-66B00470D60F@yahoo.com> <56C34D84.40509@gmail.com> Message-ID: <22211.55808.193648.723977@turnbull.sk.tsukuba.ac.jp> Executive summary: Very process-meta. Suggestion: good GSoC project? Jonathan Goble writes: > That's exactly the type of tools I envision being made available by > third parties. Experience shows that such visions mostly remain dreams. Dreaming is good, but Python demands somewhat more for inclusion in the core. Not that much -- Victor Stinner's FAT Python is a good example. (I think that's being discussed on the python-dev list, easy to find in the archives.) What he's *actually* doing is (conceptually, I haven't looked at the actual patch) a somewhat invasive modification of the core compilation process. But along with that he's demonstrated several practical optimizations that are enabled by his change.[1] (Note that he brought the patch with him when proposing his change, too. I guess he wrote the code when he woke up in the morning. :-) That created a certain amount of buzz -- optimization proposals always do :-), and some people see the needed changes as simplifying the whole process, which brought them on board. > Depending on how much I get invested into this project, > I may even write such a tool myself (though that's not guaranteed). This is exactly backwards from the point of view of getting it into the stdlib. What the deafening silence was saying (and what the actual posts say!) is that nobody else is going to do it. Features that aren't going to be exploited fairly soon are complications, and that is against a fundamental design principle (the "Zen of Python", try "python -m this | grep -i comp" if you haven't seen it before). Which gives me an idea: Victor proposed writing optimizations for FAT Python as a Google Summer of Code project for students. If you can find an experienced developer to mentor with you, the basic change sounds like an easy project for a student, and the tool like something quite advanced but still feasible. If you want to know more, write me off-list, or better yet ask on core-mentorship. I can't mentor, but I can help with the admin details. FMI: https://wiki.python.org/moin/SummerOfCode/2016 > On Tue, Feb 16, 2016 at 4:55 AM, Paul Moore wrote: > > Sorry. I don't personally have any issue with the proposal, and it > > sounds like a reasonable idea. I don't think it's likely to be > > *hugely* controversial Agreed. The only real risks in exposing an existing internal attribute are (1) complication and (2) future maintenance cost for a little-used feature (eg, if Python decides to anoint regex). It might languish in the tracker for quite a while if you can't demonstrate real use cases, though. (The educational aspect of being able to merely list the compiled bytecodes readably might be enough, but I would bet against it.) > I don't think this would be major enough to require a PEP, Definitely, bet against that. This doesn't change the language or violate backward compatibility at all, and design looks quite straightforward, including the potential tools (ISTR you saying you've seen them for other regexp engines? at least the UI, and maybe the algorithms, can be borrowed). Footnotes: [1] They all give 1-10% on microbenchmarks, so individually they're insignificant. But as the apocryphal congressman said about billions of dollars, "1% here, 5% there, and pretty soon you're talking about perceptible speedups", and that gets certain people excited. From jcgoble3 at gmail.com Tue Feb 16 21:45:20 2016 From: jcgoble3 at gmail.com (Jonathan Goble) Date: Tue, 16 Feb 2016 21:45:20 -0500 Subject: [Python-ideas] Exposing regular expression bytecode In-Reply-To: <22211.55808.193648.723977@turnbull.sk.tsukuba.ac.jp> References: <475EF1D2-E27E-4AB6-9729-66B00470D60F@yahoo.com> <56C34D84.40509@gmail.com> <22211.55808.193648.723977@turnbull.sk.tsukuba.ac.jp> Message-ID: On Tue, Feb 16, 2016 at 9:25 PM, Stephen J. Turnbull wrote: > Executive summary: Very process-meta. Suggestion: good GSoC project? Thanks for the details about Python's processes; I'm learning about them as I go. I now see that this may be a bit harder to get pushed through than I had initially thought, but at least for now, I'll keep slowly pushing toward it. As for GSoC, that's an interesting idea. I'm not a student myself (29 years old and on disability), though, but that does seem like a possible way forward if a student wants to take it on. I'll see about putting something together to post on core-mentorship tonight or tomorrow (and of course, I'm still open to doing work on it myself as well). From stephen at xemacs.org Tue Feb 16 21:02:34 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 17 Feb 2016 11:02:34 +0900 Subject: [Python-ideas] Exposing regular expression bytecode In-Reply-To: References: <475EF1D2-E27E-4AB6-9729-66B00470D60F@yahoo.com> <56C34D84.40509@gmail.com> Message-ID: Executive summary: Very process-meta. Suggestion: good GSoC project? Jonathan Goble writes: > That's exactly the type of tools I envision being made available by > third parties. Experience shows that such visions mostly remain dreams. Dreaming is good, but Python demands somewhat more for inclusion in the core. Not that much -- Victor Stinner's FAT Python is a good example. (I think that's being discussed on the python-dev list, easy to find in the archives.) What he's *actually* doing is (conceptually, I haven't looked at the actual patch) a somewhat invasive modification of the core compilation process. But along with that he's demonstrated several practical optimizations that are enabled by his change.[1] (Note that he brought the patch with him when proposing his change, too. I guess he wrote the code when he woke up in the morning. :-) That created a certain amount of buzz, and some people see the needed changes as simplifying the whole process, which brought them on board. > Depending on how much I get invested into this project, > I may even write such a tool myself (though that's not guaranteed). This is exactly backwards from the point of view of getting it into the stdlib. What the deafening silence was saying (and what the actual posts say!) is that nobody else is going to do it. Features that aren't going to be exploited fairly soon are complications, and that is against a fundamental design principle (the "Zen of Python", try "python -m this | grep -i comp" if you haven't seen it before). Which gives me an idea: Victor proposed writing optimizations for FAT Python as a Google Summer of Code project for students. If you can find an experienced developer to mentor with you, the basic change sounds like an easy project for a student, and the tool like something quite advanced but still feasible. If you want to know more, write me off-list. I can't mentor, but I can help with the admin details. FMI: https://wiki.python.org/moin/SummerOfCode/2016 > On Tue, Feb 16, 2016 at 4:55 AM, Paul Moore wrote: > > Sorry. I don't personally have any issue with the proposal, and it > > sounds like a reasonable idea. I don't think it's likely to be > > *hugely* controversial Agreed. The only real risks in exposing an existing internal attribute are (1) complication and (2) future maintenance cost for a little-used feature (eg, if Python decides to anoint regex). It might languish in the tracker for quite a while if you can't demonstrate real use cases, though. (The educational aspect of being able to merely list the compiled bytecodes readably might be enough, but I would bet against it.) > I don't think this would be major enough to require a PEP, Definitely, bet against that. This doesn't change the language or violate backward compatibility at all, and design looks quite straightforward, including the potential tools (ISTR you saying you've seen them for other regexp engines? at least the UI, and maybe the algorithms, can be borrowed). Footnotes: [1] They all give 1-10% on microbenchmarks, so individually they're insignificant. But as the apocryphal congressman said about billions of dollars, "1% here, 5% there, and pretty soon you're talking about perceptible speedups", and that gets certain people excited. From p.f.moore at gmail.com Wed Feb 17 04:55:04 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 17 Feb 2016 09:55:04 +0000 Subject: [Python-ideas] Exposing regular expression bytecode In-Reply-To: References: <475EF1D2-E27E-4AB6-9729-66B00470D60F@yahoo.com> <56C34D84.40509@gmail.com> Message-ID: On 16 February 2016 at 18:29, Jonathan Goble wrote: >> But if you are willing to dig into Python's C API yourself (and it >> sounds like you are) there are definitely people who will help you. >> You might want to join the core mentorship list (see >> http://pythonmentors.com/) where you should get plenty of assistance. >> This proposal sounds like a great "beginner" task, as well - so even >> if you don't want to implement it yourself, still put it on the >> tracker, and mark it as an "easy" change, and maybe some other >> newcomer who wants a task to help them learn the C API will pick it >> up. > > I'll look into the mentorship list; thanks for the link. As for > marking it "easy", I don't seem to have the necessary permissions to > change the Keywords field; perhaps you or someone else can set that > flag for me? If so, I'd appreciate it. :-) Done (although the "easy" keyword is mainly to help others know it's something they could work on, so if you're working on it it's less relevant :-)) > Also, I received a small patch off-list from Petr Viktorin > implementing a getter for the code list (thanks, Petr). I'll need to > test it, but from the little I know of the C API it looks like it will > get me started in the right direction. Assuming that works, what's > left is a public constructor for the regex type (to enable > optimizers), a dis-like module, and docs and tests. I don't think this > would be major enough to require a PEP, but of course being new here, > I'm open to being told I'm wrong. :-) IMO, this shouldn't need a PEP, unless someone feels that exposing the code list implies a compatibility guarantee (personally, I don't - I think that it should be seen in the same light as the dis module, and as a CPython implementation detail). Paul From rosuav at gmail.com Wed Feb 17 08:58:08 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 18 Feb 2016 00:58:08 +1100 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> <20160215144251.GY31806@ando.pearwood.info> <56C20D56.2030605@stoneleaf.us> <56C210A9.4020002@mail.de> <20160215234322.GZ31806@ando.pearwood.info> <56C2F4CB.7030709@mail.de> Message-ID: On Wed, Feb 17, 2016 at 4:54 AM, Georg Brandl wrote: >> Another aspect, I came to think of is the following asymmetry: >> >> a, b, c, d = mylist4 # works >> a, b, c = mylist3 # also works >> a, b = mylist2 # works too >> [a] = mylist1 # special case? > > a, = mylist1 > > actually. Same "special case" as with 1-element tuples. The one-target case is actually available to everything else; the only difference is that the trailing comma is mandatory, not optional: >>> a,b,c, = range(3) Which is, again, the same as with tuples. ChrisA From ethan at stoneleaf.us Wed Feb 17 11:26:29 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 17 Feb 2016 08:26:29 -0800 Subject: [Python-ideas] Generator unpacking In-Reply-To: References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> <20160215144251.GY31806@ando.pearwood.info> <56C20D56.2030605@stoneleaf.us> <56C210A9.4020002@mail.de> <20160215234322.GZ31806@ando.pearwood.info> <56C2F4CB.7030709@mail.de> Message-ID: <56C49F35.9060508@stoneleaf.us> On 02/17/2016 05:58 AM, Chris Angelico wrote: > On Wed, Feb 17, 2016 at 4:54 AM, Georg Brandl wrote: >>> Another aspect, I came to think of is the following asymmetry: >>> >>> a, b, c, d = mylist4 # works >>> a, b, c = mylist3 # also works >>> a, b = mylist2 # works too >>> [a] = mylist1 # special case? >> >> a, = mylist1 >> >> actually. Same "special case" as with 1-element tuples. > > The one-target case is actually available to everything else; the only > difference is that the trailing comma is mandatory, not optional: > >>>> a,b,c, = range(3) > > Which is, again, the same as with tuples. Did you mean the trailing comma is necessary in the one-tuple case? --> a, b, c = range(3) --> a 0 --> b 1 --> c 2 --> a = range(1) --> a [0] --> a, = range(1) --> a 0 -- ~Ethan~ From rosuav at gmail.com Wed Feb 17 11:36:08 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 18 Feb 2016 03:36:08 +1100 Subject: [Python-ideas] Generator unpacking In-Reply-To: <56C49F35.9060508@stoneleaf.us> References: <56BEE27B.9000804@canterbury.ac.nz> <56BFA2CA.2080404@canterbury.ac.nz> <9E6AEE84-1D5F-4763-B2D9-AAE3871D8692@yahoo.com> <56C1D8CE.4020709@mail.de> <20160215144251.GY31806@ando.pearwood.info> <56C20D56.2030605@stoneleaf.us> <56C210A9.4020002@mail.de> <20160215234322.GZ31806@ando.pearwood.info> <56C2F4CB.7030709@mail.de> <56C49F35.9060508@stoneleaf.us> Message-ID: On Thu, Feb 18, 2016 at 3:26 AM, Ethan Furman wrote: > Did you mean the trailing comma is necessary in the one-tuple case? > > --> a, b, c = range(3) > --> a > 0 > --> b > 1 > --> c > 2 > > --> a = range(1) > --> a > [0] > --> a, = range(1) > --> a > 0 > > -- > ~Ethan~ Correct. It's optional in every other case, but mandatory in the one-element case. Chrisa From foley12723 at gmail.com Wed Feb 17 12:01:59 2016 From: foley12723 at gmail.com (Stephan Foley) Date: Wed, 17 Feb 2016 12:01:59 -0500 Subject: [Python-ideas] A tree data structure for Python Message-ID: Hello, I would love to see a tree data structure out of the box for python. I'm thinking of a n-ary or multiway tree. It's a useful data structure to model things that are difficult to model with lists, dicts, etc. Recursion can be avoided by using a stack for traversal. From abarnert at yahoo.com Wed Feb 17 12:30:11 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 17 Feb 2016 09:30:11 -0800 Subject: [Python-ideas] A tree data structure for Python In-Reply-To: References: Message-ID: On Feb 17, 2016, at 09:01, Stephan Foley wrote: > > Hello, I would love to see a tree data structure out of the box for > python. I'm thinking of a n-ary or multiway tree. It's a useful data > structure to model things that are difficult to model with lists, > dicts, etc. Recursion can be avoided by using a stack for traversal. Do you want it to be fixed-n-ary or arbitrary branching? Values in all nodes, or only the leaves? Does it need O(1) size or depth? Single-linked, back-pointers, or threaded? Mutable or copying? Are trees and nodes separate types (and, if so, is a subtree a tree in its own right)? What algorithms do you need besides BFS and DFS (pre-, in-, and post-order) visits? All of these are really easy to build a class for--or to just represent with a list of lists (or a tuple of tuples)--but coming up with a class that can handle all of the different possibilities in a single type is a lot harder. What might be more useful is to write generic algorithms. Then you can implement very different kinds of trees, and even use existing tree types, with a single implementation of the algorithm: "for node in bfs(root, itemgetter(1)):" for a simple "(value, (child, ...))" recursive tuple, "for div in bfs(h.body, lambda node: node.find_children('div'))" for an HTML document, etc. (I'd bet this already exists on PyPI.) From foley12723 at gmail.com Wed Feb 17 12:52:51 2016 From: foley12723 at gmail.com (Stephan Foley) Date: Wed, 17 Feb 2016 12:52:51 -0500 Subject: [Python-ideas] A tree data structure for Python In-Reply-To: References: Message-ID: Hello and thanks for the reply. On Wed, Feb 17, 2016 at 12:30 PM, Andrew Barnert wrote: > Do you want it to be fixed-n-ary or arbitrary branching? Values in all nodes, or only the leaves? Does it need O(1) size or depth? Single-linked, back-pointers, or threaded? Mutable or copying? Are trees and nodes separate types (and, if so, is a subtree a tree in its own right)? What algorithms do you need besides BFS and DFS (pre-, in-, and post-order) visits? Yes, I'm looking online and neither C++ nor Java have a tree class for the same reason. Rolling your own tree is very easy, but developing something useful for everyone can involve a lot of overhead. But, to answer your questions, a lightweight tree that could represent either a file system or configuration data. Arbitrary branching. Values in all nodes. Pointers to first child, next sibling and parent. Based on a node class. Pre-order, post-order and level-order traversal. From grant.jenks at gmail.com Wed Feb 17 13:57:22 2016 From: grant.jenks at gmail.com (Grant Jenks) Date: Wed, 17 Feb 2016 10:57:22 -0800 Subject: [Python-ideas] A tree data structure for Python In-Reply-To: References: Message-ID: On Wed, Feb 17, 2016 at 9:01 AM, Stephan Foley wrote: > Hello, I would love to see a tree data structure out of the box for > python. I'm thinking of a n-ary or multiway tree. Reminds me of the "one-line tree in Python" hack: https://gist.github.com/hrldcpr/2012250 Here's the gist of it: from collections import defaultdict def tree(): return defaultdict(tree) It's almost too flexible but writing methods to traverse is relatively easy. Grant From brett at python.org Wed Feb 17 14:34:06 2016 From: brett at python.org (Brett Cannon) Date: Wed, 17 Feb 2016 19:34:06 +0000 Subject: [Python-ideas] A bit meta In-Reply-To: References: Message-ID: All the owners on the list are too busy to drive this. You will need to figure out what people who use email will gain/lose in terms of features. You will also need to figure out hosting. It might be easiest to see if Donald has specific tasks necessary to experiment with distutils-sig since that is a smaller mailing list and if it's successful there then we can consider a change here. On Mon, 15 Feb 2016 at 14:30 Leonardo Santagada wrote: > Is there any way for me to help in evaluating/testing/deploying discourse? > I really think trying it out and seeing if people flock around a discourse > instance is beneficial for the community. > > -- > > Leonardo Santagada > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Feb 17 14:51:47 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 17 Feb 2016 11:51:47 -0800 Subject: [Python-ideas] A tree data structure for Python In-Reply-To: References: Message-ID: <1C6CC017-BB84-4C20-AA60-72B682A5ACA1@yahoo.com> On Feb 17, 2016, at 09:52, Stephan Foley wrote: > > Hello and thanks for the reply. > >> On Wed, Feb 17, 2016 at 12:30 PM, Andrew Barnert wrote: >> >> Do you want it to be fixed-n-ary or arbitrary branching? Values in all nodes, or only the leaves? Does it need O(1) size or depth? Single-linked, back-pointers, or threaded? Mutable or copying? Are trees and nodes separate types (and, if so, is a subtree a tree in its own right)? What algorithms do you need besides BFS and DFS (pre-, in-, and post-order) visits? > > Yes, I'm looking online and neither C++ nor Java have a tree class for > the same reason. Rolling your own tree is very easy, but developing > something useful for everyone can involve a lot of overhead. It's not so much the overhead, as the design decisions. You need parent pointers; I not only don't need them, but need subtrees that don't keep their parents alive. I need threading, you don't want it, and don't want on do the log N work on each update to fix up the threads you're never going to use but the type demands them. You want first child and next sibling, I want all children. And so on. In most of there cases, there's no structure or API that's going to make us both happy. That's why a functional design works much better than an OO design for this particular problem. It's hard to explain in text, but easy to show in code, so check https://github.com/abarnert/treestuff From breamoreboy at yahoo.co.uk Wed Feb 17 15:48:33 2016 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Wed, 17 Feb 2016 20:48:33 +0000 Subject: [Python-ideas] A tree data structure for Python In-Reply-To: References: Message-ID: On 17/02/2016 17:01, Stephan Foley wrote: > Hello, I would love to see a tree data structure out of the box for > python. I'm thinking of a n-ary or multiway tree. It's a useful data > structure to model things that are difficult to model with lists, > dicts, etc. Recursion can be avoided by using a stack for traversal. It won't happen as there are far too many conflicting options, see for example the list here http://kmike.ru/python-data-structures/ -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From python-ideas at mgmiller.net Wed Feb 17 16:54:18 2016 From: python-ideas at mgmiller.net (Mike Miller) Date: Wed, 17 Feb 2016 13:54:18 -0800 Subject: [Python-ideas] A tree data structure for Python In-Reply-To: References: Message-ID: <56C4EC0A.6090806@mgmiller.net> How about a toolbox of primitives to build the structure needed? With say the top-five most common high-level combinations included? Perhaps something as flexible as the logging module, with a modern design take. Sounds like a good package to add to PyPI until it matures, if something similar doesn't already exist. -Mike On 2016-02-17 12:48, Mark Lawrence wrote: > > It won't happen as there are far too many conflicting options, see for example > the list here http://kmike.ru/python-data-structures/ > From mike at selik.org Wed Feb 17 18:22:32 2016 From: mike at selik.org (Michael Selik) Date: Wed, 17 Feb 2016 18:22:32 -0500 Subject: [Python-ideas] Dict(x, y) -> Dict(zip(x, y)) In-Reply-To: <56C3B532.8020201@feete.org> References: <56C3B532.8020201@feete.org> Message-ID: <6B0078AA-FB17-43BD-8DF2-F4DDDE17C3E6@selik.org> > On Feb 16, 2016, at 6:48 PM, Ian Foote wrote: > > I think this is one of those cases where the gain is small enough that > writing the helper function is often not worth it, but big enough that > people would use it if it already existed. That might be a small window, > but it's there. I?m pretty sure I would stick with ``dict(zip(keys, values))``. My code would then be backwards-compatible and wouldn?t introduce yet another thing for a newbie to need to learn about when reading my code. A new alternate constructor should follow the pattern ``dict.from_stuff`` or ``dict.fromstuff`` like ``dict.fromkeys`` or ``datetime.fromordinal``. What would you call this one? The method ``dict.setdefault`` is already awkwardly named. I suppose ``dict.get_value_and_insert_key_if_missing`` is prohibitively long. Would this new one be ``dict.from_keys_and_values``? Actually, that makes me think of one version I might be OK with: dict.fromkeys('abc', values=[0, 1, 2]) But then what would be the benefit versus ``dict(zip(keys, values))``? From mike at selik.org Wed Feb 17 18:35:33 2016 From: mike at selik.org (Michael Selik) Date: Wed, 17 Feb 2016 18:35:33 -0500 Subject: [Python-ideas] Explicit variable capture list In-Reply-To: References: <20160120003712.GZ10854@ando.pearwood.info> <20160121001027.GB4619@ando.pearwood.info> <56A3ED9B.6030009@canterbury.ac.nz> Message-ID: <28B4E450-C682-42BC-B270-384CF4DA6730@selik.org> > On Feb 16, 2016, at 4:07 PM, Terry Reedy wrote: > > Since writing this, I realized that defining a custom class and using bound methods is a fourth option, which I also like. This binds the differentiating data to an instance, which is then bound to the function, rather than to the function directly. A toy example: Does this deserve a link to the closures == classes koan? I think so :-) https://people.csail.mit.edu/gregs/ll1-discuss-archive-html/msg03277.html Anywhere someone suggests a closure, a class can probably do the work. And vice-versa, though one will often be more elegant than the other for a particular circumstance. From foley12723 at gmail.com Thu Feb 18 20:53:04 2016 From: foley12723 at gmail.com (Stephan Foley) Date: Thu, 18 Feb 2016 20:53:04 -0500 Subject: [Python-ideas] A tree data structure for Python In-Reply-To: <1C6CC017-BB84-4C20-AA60-72B682A5ACA1@yahoo.com> References: <1C6CC017-BB84-4C20-AA60-72B682A5ACA1@yahoo.com> Message-ID: Hi Andrew, On Wed, Feb 17, 2016 at 2:51 PM, Andrew Barnert wrote: > It's not so much the overhead, as the design decisions. You need parent pointers; I not only don't need them, but need subtrees that don't keep their parents alive. I need threading, you don't want it, and don't want on do the log N work on each update to fix up the threads you're never going to use but the type demands them. You want first child and next sibling, I want all children. And so on. In most of there cases, there's no structure or API that's going to make us both happy. I get what you're saying, but something simple like the C++ Boost Property Tree might be general enough to be useful: http://www.boost.org/doc/libs/1_60_0/doc/html/property_tree.html#property_tree.intro > That's why a functional design works much better than an OO design for this particular problem. It's hard to explain in text, but easy to show in code, so check https://github.com/abarnert/treestuff Ahhh....functional design....I'm going to hit my Scheme books :-) In general, I only use OO for data and try to make everything else functions. Classes work nicely to hold the pointers. Really, you can also program a tree in a memory buffer with integer pointers as offsets. From foley12723 at gmail.com Thu Feb 18 21:01:24 2016 From: foley12723 at gmail.com (Stephan Foley) Date: Thu, 18 Feb 2016 21:01:24 -0500 Subject: [Python-ideas] A tree data structure for Python In-Reply-To: References: Message-ID: Hi Mark, On Wed, Feb 17, 2016 at 3:48 PM, Mark Lawrence wrote: > > It won't happen as there are far too many conflicting options, see for > example the list here http://kmike.ru/python-data-structures/ Cool site...I once programmed a trie in Python. As for the trees, my request is for a very specific tree...a n-ary or multiway tree. The other trees listed, like red-black and binary, are for sorting and searching efficiently. A Patricia-Tree is actually a trie. BTrees are for external storage. From guido at python.org Thu Feb 18 21:07:29 2016 From: guido at python.org (Guido van Rossum) Date: Thu, 18 Feb 2016 18:07:29 -0800 Subject: [Python-ideas] A tree data structure for Python In-Reply-To: References: Message-ID: Stephan, you can't just request a new stdlib module and expect someone to make one for you. Is there an existing PyPI package that you think is mature enough to add to the stdlib? Then say so. If you want to write one yourself, go right ahead. But this thread has run its course. On Thu, Feb 18, 2016 at 6:01 PM, Stephan Foley wrote: > Hi Mark, > > On Wed, Feb 17, 2016 at 3:48 PM, Mark Lawrence wrote: > >> >> It won't happen as there are far too many conflicting options, see for >> example the list here http://kmike.ru/python-data-structures/ > > Cool site...I once programmed a trie in Python. As for the trees, my > request is for a very specific tree...a n-ary or multiway tree. The > other trees listed, like red-black and binary, are for sorting and > searching efficiently. A Patricia-Tree is actually a trie. BTrees are > for external storage. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From foley12723 at gmail.com Thu Feb 18 21:08:36 2016 From: foley12723 at gmail.com (Stephan Foley) Date: Thu, 18 Feb 2016 21:08:36 -0500 Subject: [Python-ideas] A tree data structure for Python In-Reply-To: <56C4EC0A.6090806@mgmiller.net> References: <56C4EC0A.6090806@mgmiller.net> Message-ID: Hi Mike, On Wed, Feb 17, 2016 at 4:54 PM, Mike Miller wrote: > How about a toolbox of primitives to build the structure needed? With say > the top-five most common high-level combinations included? Perhaps > something as flexible as the logging module, with a modern design take. Sounds great. I like lists, dicts, etc, but they are hard to use building up complex structures. Something like linked list, node and visitor pattern would be nice. From foley12723 at gmail.com Thu Feb 18 21:13:08 2016 From: foley12723 at gmail.com (Stephan Foley) Date: Thu, 18 Feb 2016 21:13:08 -0500 Subject: [Python-ideas] A tree data structure for Python In-Reply-To: References: Message-ID: Hi Guido, On Thu, Feb 18, 2016 at 9:07 PM, Guido van Rossum wrote: > Stephan, you can't just request a new stdlib module and expect someone > to make one for you. Is there an existing PyPI package that you think > is mature enough to add to the stdlib? Then say so. If you want to > write one yourself, go right ahead. But this thread has run its > course. OK, sorry...I'm new here and thought this was a mailing list to propose ideas, but I don't mean to use up other people's bandwidth. I'll go look at PyPI and I'll be happy to help out if I can. From jcgoble3 at gmail.com Fri Feb 19 14:42:09 2016 From: jcgoble3 at gmail.com (Jonathan Goble) Date: Fri, 19 Feb 2016 14:42:09 -0500 Subject: [Python-ideas] Exposing regular expression bytecode In-Reply-To: References: <475EF1D2-E27E-4AB6-9729-66B00470D60F@yahoo.com> <56C34D84.40509@gmail.com> Message-ID: FWIW, I've decided to shelve this idea for the time being, at least, as I've had some things come up unexpectedly that are going to eat into my available time for the foreseeable future, so I no longer have time to pursue this myself. (Suffice it to say that Real Life always has crappy timing. :-P) Maybe I'll have time to resurrect it in the future; in the meantime, the issue on the bug tracker remains open in the event someone gets bored and decides to take a crack at it. From gunkmute at gmail.com Sun Feb 21 08:35:29 2016 From: gunkmute at gmail.com (Demur Rumed) Date: Sun, 21 Feb 2016 13:35:29 +0000 Subject: [Python-ideas] Wordcode v2, moved from -dev Message-ID: Continuing discussion from https://mail.python.org/pipermail/python-dev/2016-February/143357.html I'm currently waiting for abarnert to accept my PR: https://github.com/abarnert/cpython/pull/1 which fixes tests. The only really failing test (ie which doesn't also fail on my stock python) is test_importlib since it seems I don't know how to update frozen modules or something Antoine wanted to know what I benchmarked. Fact is that I haven't been rigorous; I'm comparing Archlinux's stock python to my ./configure && make. That said pybench is 20s on stock python compared to 17s on wordcode On the matter of bytecode projects having to be updated, I've updated my Befunge Python JIT. I post some analysis in my PR on why this particular benchmark should go in the old one's favor @ https://github.com/abarnert/cpython/pull/1#issuecomment-186679849 Current funge.py: https://github.com/serprex/Befunge/blob/master/funge.py Wordcode wfunge.py: https://github.com/serprex/Befunge/blob/master/wfunge.py Stock python runs Mandelbrot in 30s vs 28s on wfunge, & 3.3s vs 2.9 for beer6.bf (Mandel is a benchmark of generated code, whereas beer6 is mostly a benchmark of the JIT recompiling This also serves to demonstrate that current projects can be updated quite straightforwardly, worst case they change to always emitting EXTENDED_ARG for opcodes >= STORE_FAST In the pull request I've removed HAVE_ARGUMENT. I believe this'll help in porting things to fail-fast since HAVE_ARGUMENT's usage in code essentially must change, but perhaps one could add \x90 back as a constant named DOES_NOT_IGNORE_ARGUMENT I'd like to ask someone who isn't involved in development of this branch to verify my benchmark results Befunge-93 programs I mentioned: http://quadium.net/funge/downloads/bef93src/beer6.bf http://quadium.net/funge/downloads/bef93src/mandel.bf Still to do is at least investigate halving opargs for jump instructions -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Feb 21 12:41:57 2016 From: brett at python.org (Brett Cannon) Date: Sun, 21 Feb 2016 17:41:57 +0000 Subject: [Python-ideas] Wordcode v2, moved from -dev In-Reply-To: References: Message-ID: On Sun, 21 Feb 2016 at 05:37 Demur Rumed wrote: > Continuing discussion from > https://mail.python.org/pipermail/python-dev/2016-February/143357.html > > I'm currently waiting for abarnert to accept my PR: > https://github.com/abarnert/cpython/pull/1 which fixes tests. The only > really failing test (ie which doesn't also fail on my stock python) is > test_importlib since it seems I don't know how to update frozen modules or > something > It's automatic when you run `make` and it updates Python/importlib.h as part of the process. I don't think this kind of change would have broken freezing, but I guess it's possible. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sun Feb 21 14:14:00 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 21 Feb 2016 11:14:00 -0800 Subject: [Python-ideas] Wordcode v2, moved from -dev In-Reply-To: References: Message-ID: I haven't been able to build and test this, because I can't build Python on Mac at all anymore. I believe this is because #25136 is much more serious now (Xcode 7 is the released version, and the 10.11 CLT corresponds to it so that workaround no longer works). So I just accepted the PR so anyone else who wants to play with and/or test the code can do so on my fork (https://github.com/abarnert/cpython branch wpy). Sent from my iPhone > On Feb 21, 2016, at 05:35, Demur Rumed wrote: > > Continuing discussion from https://mail.python.org/pipermail/python-dev/2016-February/143357.html > > I'm currently waiting for abarnert to accept my PR: https://github.com/abarnert/cpython/pull/1 which fixes tests. The only really failing test (ie which doesn't also fail on my stock python) is test_importlib since it seems I don't know how to update frozen modules or something > > Antoine wanted to know what I benchmarked. Fact is that I haven't been rigorous; I'm comparing Archlinux's stock python to my ./configure && make. That said pybench is 20s on stock python compared to 17s on wordcode > > On the matter of bytecode projects having to be updated, I've updated my Befunge Python JIT. I post some analysis in my PR on why this particular benchmark should go in the old one's favor @ https://github.com/abarnert/cpython/pull/1#issuecomment-186679849 > > Current funge.py: https://github.com/serprex/Befunge/blob/master/funge.py > Wordcode wfunge.py: https://github.com/serprex/Befunge/blob/master/wfunge.py > > Stock python runs Mandelbrot in 30s vs 28s on wfunge, & 3.3s vs 2.9 for beer6.bf (Mandel is a benchmark of generated code, whereas beer6 is mostly a benchmark of the JIT recompiling > > This also serves to demonstrate that current projects can be updated quite straightforwardly, worst case they change to always emitting EXTENDED_ARG for opcodes >= STORE_FAST > > In the pull request I've removed HAVE_ARGUMENT. I believe this'll help in porting things to fail-fast since HAVE_ARGUMENT's usage in code essentially must change, but perhaps one could add \x90 back as a constant named DOES_NOT_IGNORE_ARGUMENT I think that it might be a good idea to add something like this back in on the Python side, but not the C equivalent. There are really two different reasons to use HAVE_ARGUMENT: to read 1 bytes instead of 3, and to skip any figuring-out-what-the-arg-means code. The former is no longer valid with wordcode, but the latter still is. Or, alternatively, maybe just add "hasarg = range(90, 256)" to dis, as a sequence of bytecodes that have any meaningful argument (to go with dis.hasjrel, etc.). > I'd like to ask someone who isn't involved in development of this branch to verify my benchmark results > Befunge-93 programs I mentioned: > http://quadium.net/funge/downloads/bef93src/beer6.bf > http://quadium.net/funge/downloads/bef93src/mandel.bf > > Still to do is at least investigate halving opargs for jump instructions > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Tue Feb 23 16:05:16 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 23 Feb 2016 22:05:16 +0100 Subject: [Python-ideas] How the heck does async/await work in Python 3.5 In-Reply-To: References: <56c7d145$0$1597$c3e8da3$5496439d@news.astraweb.com> Message-ID: <56CCC98C.5060504@mail.de> On 20.02.2016 07:53, Christian Gollwitzer wrote: > If you have difficulties wit hthe overall concept, and if you are open > to discussions in another language, take a look at this video: > > https://channel9.msdn.com/Shows/C9-GoingNative/GoingNative-39-await-co-routines > > > MS has added coroutine support with very similar syntax to VC++ > recently, and the developer tries to explain it to the "stackful" > programmers. Because of this thread, I finally finished an older post collecting valuable insights from last year discussions regarding concurrency modules available in Python: http://srkunze.blogspot.com/2016/02/concurrency-in-python.html It appears to me that it would fit here well. @python-ideas Back then, the old thread ("Concurrency Modules") was like basically meant to result in something useful. I hope the post covers the essence of the discussion. Some even suggested putting the table into the Python docs. I am unaware of the formal procedure here but I would be glad if somebody could point be at the right direction if that the survey table is wanted in the docs. Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsbueno at python.org.br Tue Feb 23 16:25:47 2016 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Tue, 23 Feb 2016 18:25:47 -0300 Subject: [Python-ideas] How the heck does async/await work in Python 3.5 In-Reply-To: <56CCC98C.5060504@mail.de> References: <56c7d145$0$1597$c3e8da3$5496439d@news.astraweb.com> <56CCC98C.5060504@mail.de> Message-ID: Today I also stumbled on this helpful "essay" from Brett Cannon about the same subject http://www.snarky.ca/how-the-heck-does-async-await-work-in-python-3-5 On 23 February 2016 at 18:05, Sven R. Kunze wrote: > On 20.02.2016 07:53, Christian Gollwitzer wrote: > > If you have difficulties wit hthe overall concept, and if you are open to > discussions in another language, take a look at this video: > > https://channel9.msdn.com/Shows/C9-GoingNative/GoingNative-39-await-co-routines > > MS has added coroutine support with very similar syntax to VC++ recently, > and the developer tries to explain it to the "stackful" programmers. > > > Because of this thread, I finally finished an older post collecting valuable > insights from last year discussions regarding concurrency modules available > in Python: http://srkunze.blogspot.com/2016/02/concurrency-in-python.html It > appears to me that it would fit here well. > > @python-ideas > Back then, the old thread ("Concurrency Modules") was like basically meant > to result in something useful. I hope the post covers the essence of the > discussion. > Some even suggested putting the table into the Python docs. I am unaware of > the formal procedure here but I would be glad if somebody could point be at > the right direction if that the survey table is wanted in the docs. > > Best, > Sven > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From victor.stinner at gmail.com Tue Feb 23 18:33:00 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 24 Feb 2016 00:33:00 +0100 Subject: [Python-ideas] PEP 511 and bytecode optimizers Message-ID: Hi, I played with bytecode optimizers. I still have the same opinion: for the PEP 511, code_transformer() should take a code object as input and return a new code object as output. I don't want to put one specific API in the PEP 511 since it means that we will have to maintain this API. If you want a different data structure, you need extra decode/encode steps in your code :-/ Existing projects take a whole Python code object as input. In my experience, there is no such "grand unified API" to manipulate bytecode. The byteplay and codetransformers projects (and now my bytecode project) use a different and somehow incompatible API. You should pick the one which fits best your use case (and your coding style). I wrote a new bytecode project, my own "high-level" API to modify bytecode. I also reimplemented the peephole optimizer of CPython 3.6 in pure Python. More details below. -- Recently, Andrew Barnert suggested to modify the code_transformer() of the PEP 511: https://www.python.org/dev/peps/pep-0511/#code-transformer-method Currently, input and output are Python code objects: def code_transformer(self, code, context): ... return new_code Andrew proposed to use an higher level API (which don't exist yet, he wants to put something into the dis module). I rewrote the peephole optimizer (currently implemented C) in pure Python: https://hg.python.org/sandbox/fatpython/file/6b01409f2e10/Lib/peephole_opt.py The Python code is still low-level. It is based on the C code which modifies directly bytes (a bytearray object in my code). I added an "Instr" (bytecode instruction) class but it's a minor abstraction. The C peephole optimizer has many "is_basic_block(offset, size)" checks to ensure that we respect the control flow (don't modify two instructions of two different code paths). I wrote my own bytecode API using blocks: a block is a list of instructions. Jumps use labels, each block has an unique label. The line number is stored directly in an instruction. This API fits well with the peephole optimizer. Any instruction can be removed. Respecting the control flow is obvious: restrict optimizations to one block, optimize all blocks independently. I released bytecode 0.0 which is mostly a proof-of-concept: https://github.com/haypo/bytecode I adapted my Python peephole optimizer on top of my bytecode project: https://github.com/haypo/bytecode/blob/a17baedf5ce89622e16dcf7bf1de8094525f4a93/peephole_opt.py bytecode should be enhanced. For example, it is unable to compute the stack level. The API may move to something closer to byteplay: manipulate directly variable names for LOAD_FAST, pass directly the constant value to LOAD_CONST, ... rather than having to use an integer index of a separated list. Victor From tjreedy at udel.edu Tue Feb 23 21:37:12 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 23 Feb 2016 21:37:12 -0500 Subject: [Python-ideas] How the heck does async/await work in Python 3.5 In-Reply-To: References: <56c7d145$0$1597$c3e8da3$5496439d@news.astraweb.com> <56CCC98C.5060504@mail.de> Message-ID: On 2/23/2016 4:25 PM, Joao S. O. Bueno wrote: > Today I also stumbled on this helpful "essay" from Brett Cannon about > the same subject > > http://www.snarky.ca/how-the-heck-does-async-await-work-in-python-3-5 In this essay, Brett says that asyncio added an event loop to Python. It did, but it was the second. The tk event loop was added about 20 years ago with tkinter. He also said that this enabled asynchronous programming to Python. It actually added async i/o to go along with async key-mouse-screen user interface. To illustrate, Brett gives this example, which only uses screen output. import asyncio # Borrowed from http://curio.readthedocs.org/en/latest/tutorial.html. @asyncio.coroutine def countdown(number, n): while n > 0: print('T-minus', n, '({})'.format(number)) yield from asyncio.sleep(1) n -= 1 loop = asyncio.get_event_loop() tasks = [ asyncio.ensure_future(countdown("A", 2)), asyncio.ensure_future(countdown("B", 3))] loop.run_until_complete(asyncio.wait(tasks)) loop.close() It outputs 2 or 1 lines at 1 second intervals. T-minus 2 (A) T-minus 3 (B) T-minus 1 (A) T-minus 2 (B) T-minus 1 (B) and stops after 2 seconds. The tkinter equivalent import tkinter root = tkinter.Tk() #root.withdraw() # optionally hide the window def countdown(number, n): if n > 0: print('T-minus', n, '({})'.format(number)) n -= 1 root.after(1000, countdown, number, n) root.after(1, countdown, 'A', 2) root.after(1, countdown, 'B', 3) root.mainloop() has the same output, but needs either the window close button [X] or other means to stop. (Tk mainloop() is equivalent to asyncio loop.run_forever()) Of course, in normal use, the output would go to a widget in the window, so having the window is not a problem. It would be nice to be able to use a repeatedly resumed generator instead of a repeatedly called callback with tkinter .after also. I am thinking about how that might be done. Maybe a generic callback that calls next(generator) when called. I may also look as Brett's example of interfacing with async and await keywords. -- Terry Jan Reedy From p.f.moore at gmail.com Wed Feb 24 04:59:24 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 24 Feb 2016 09:59:24 +0000 Subject: [Python-ideas] How the heck does async/await work in Python 3.5 In-Reply-To: References: <56c7d145$0$1597$c3e8da3$5496439d@news.astraweb.com> <56CCC98C.5060504@mail.de> Message-ID: On 24 February 2016 at 02:37, Terry Reedy wrote: > > In this essay, Brett says that asyncio added an event loop to Python. It > did, but it was the second. The tk event loop was added about 20 years ago > with tkinter. One of the things I would love to see (but don't have the time to work on) is a GUI event loop based around async/await. It would be a very useful example to make it clear to people that async/await isn't just about network protocols. Paul From victor.stinner at gmail.com Wed Feb 24 05:01:26 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 24 Feb 2016 11:01:26 +0100 Subject: [Python-ideas] How the heck does async/await work in Python 3.5 In-Reply-To: References: <56c7d145$0$1597$c3e8da3$5496439d@news.astraweb.com> <56CCC98C.5060504@mail.de> Message-ID: See also Doug Hellmann article on asyncio, from its serie of "Python 3 Module of the Week" articles: https://pymotw.com/3/asyncio/index.html Victor 2016-02-23 22:25 GMT+01:00 Joao S. O. Bueno : > Today I also stumbled on this helpful "essay" from Brett Cannon about > the same subject > > http://www.snarky.ca/how-the-heck-does-async-await-work-in-python-3-5 > > On 23 February 2016 at 18:05, Sven R. Kunze wrote: >> On 20.02.2016 07:53, Christian Gollwitzer wrote: >> >> If you have difficulties wit hthe overall concept, and if you are open to >> discussions in another language, take a look at this video: >> >> https://channel9.msdn.com/Shows/C9-GoingNative/GoingNative-39-await-co-routines >> >> MS has added coroutine support with very similar syntax to VC++ recently, >> and the developer tries to explain it to the "stackful" programmers. >> >> >> Because of this thread, I finally finished an older post collecting valuable >> insights from last year discussions regarding concurrency modules available >> in Python: http://srkunze.blogspot.com/2016/02/concurrency-in-python.html It >> appears to me that it would fit here well. >> >> @python-ideas >> Back then, the old thread ("Concurrency Modules") was like basically meant >> to result in something useful. I hope the post covers the essence of the >> discussion. >> Some even suggested putting the table into the Python docs. I am unaware of >> the formal procedure here but I would be glad if somebody could point be at >> the right direction if that the survey table is wanted in the docs. >> >> Best, >> Sven >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From tjreedy at udel.edu Wed Feb 24 15:57:16 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 24 Feb 2016 15:57:16 -0500 Subject: [Python-ideas] How the heck does async/await work in Python 3.5 In-Reply-To: References: <56c7d145$0$1597$c3e8da3$5496439d@news.astraweb.com> <56CCC98C.5060504@mail.de> Message-ID: On 2/24/2016 4:59 AM, Paul Moore wrote: > On 24 February 2016 at 02:37, Terry Reedy wrote: >> >> In this essay, Brett says that asyncio added an event loop to Python. It >> did, but it was the second. The tk event loop was added about 20 years ago >> with tkinter. > > One of the things I would love to see (but don't have the time to work > on) is a GUI event loop based around async/await. It would be a very > useful example to make it clear to people that async/await isn't just > about network protocols. Aiming at this was part of the point of my closing comment about adapting generators to the tk mainloop callback interface. The next step would be to do the same for awaitables. However, I need more study of the details of asyncio and the async/await protocol. -- Terry Jan Reedy From ncoghlan at gmail.com Wed Feb 24 20:05:33 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 25 Feb 2016 11:05:33 +1000 Subject: [Python-ideas] How the heck does async/await work in Python 3.5 In-Reply-To: References: <56c7d145$0$1597$c3e8da3$5496439d@news.astraweb.com> <56CCC98C.5060504@mail.de> Message-ID: On 25 February 2016 at 06:57, Terry Reedy wrote: > On 2/24/2016 4:59 AM, Paul Moore wrote: >> >> On 24 February 2016 at 02:37, Terry Reedy wrote: >>> >>> >>> In this essay, Brett says that asyncio added an event loop to Python. It >>> did, but it was the second. The tk event loop was added about 20 years >>> ago >>> with tkinter. >> >> One of the things I would love to see (but don't have the time to work >> on) is a GUI event loop based around async/await. It would be a very >> useful example to make it clear to people that async/await isn't just >> about network protocols. > > Aiming at this was part of the point of my closing comment about adapting > generators to the tk mainloop callback interface. The next step would be to > do the same for awaitables. However, I need more study of the details of > asyncio and the async/await protocol. Letting GUI event loops take over running awaitables is part of the rationale for the event loop management features in asyncio: https://docs.python.org/3/library/asyncio-eventloops.html Having an event loop interface adapter in the stdlib tk modules that can be used with asyncio.set_event_loop() would provide a much clearer demonstration of the full power of that approach than the current "*nix selectors vs Windows IOCP" capability (the latter is useful, but doesn't cover the "integration with an existing 3rd party event loop" use case). Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tjreedy at udel.edu Thu Feb 25 01:49:56 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 25 Feb 2016 01:49:56 -0500 Subject: [Python-ideas] How the heck does async/await work in Python 3.5 In-Reply-To: References: <56c7d145$0$1597$c3e8da3$5496439d@news.astraweb.com> <56CCC98C.5060504@mail.de> Message-ID: <56CEA414.7070609@udel.edu> On 2/24/2016 8:05 PM, Nick Coghlan wrote: > On 25 February 2016 at 06:57, Terry Reedy wrote: >> On 2/24/2016 4:59 AM, Paul Moore wrote: >>> >>> On 24 February 2016 at 02:37, Terry Reedy wrote: >>>> >>>> >>>> In this essay, Brett says that asyncio added an event loop to Python. It >>>> did, but it was the second. The tk event loop was added about 20 years >>>> ago >>>> with tkinter. >>> >>> One of the things I would love to see (but don't have the time to work >>> on) is a GUI event loop based around async/await. It would be a very >>> useful example to make it clear to people that async/await isn't just >>> about network protocols. >> >> Aiming at this was part of the point of my closing comment about adapting >> generators to the tk mainloop callback interface. The next step would be to >> do the same for awaitables. However, I need more study of the details of >> asyncio and the async/await protocol. > > Letting GUI event loops take over running awaitables is part of the > rationale for the event loop management features in asyncio: > https://docs.python.org/3/library/asyncio-eventloops.html One issue I have with asyncio.base_events.BaseEventLoop is that it is not a base event loop. It is a derived network IO event loop which could have been called NetworkIOEventLoop. It 'subclasses' the true base event loop by containment. The latter, call it MinimalEventLoop, comprises the run and call methods listed in https://docs.python.org/3/library/asyncio-eventloop.html ('loop', not 'loops', as above) and perhaps the task methods. In any case, it is only about 1/4 the derived class. Integrating tkinter's event loop only involves this subset. I might have found it easier to think about this if there were a separate base class. > Having an event loop interface adapter in the stdlib tk modules that > can be used with asyncio.set_event_loop() My memory is that any replacement event loop is expected to be a full implmentation of the network io loop, so one cannot just make a gui loop that works with coroutines. > would provide a much clearer > demonstration of the full power of that approach than the current > "*nix selectors vs Windows IOCP" capability (the latter is useful, but > doesn't cover the "integration with an existing 3rd party event loop" > use case). I believe the selector and proactor subclass only the network part of BaseEventLoop. A cross-platform gui might to modify the event part of either. I guess the following should work --- EventLoop = select_event_loop(OS, needs) class TkEventLoop: # modify loop methods as needed set_event_loop(TkEventLoop) --- The heart of BaseEventLoop is .run_forever, and the heart of this is while True: self._run_once() if self._stopping: break The tk equivalent of _run_once, call it tkupdate, is tkinter.Tk().update ("Enter event loop until all pending events have been processed by Tcl. [and return]). If 'while True' is kept as the master loop, it might be sufficient to add a call to tkupdate either before or after the _run_once_ call. I will have to try this and see what happens. This would be easiest to do is there were a self.updater attribute, initially None, which a user could set to an external update function to be called once in each loop. The opposite approach, far more difficult, would be to replace the Python while loop with the tk loop and change other methods as needed. Here is where having a giant class is awkward. There are 13 private attributes. Do the network methods depend on these? Another issue is that the the tk loop object is accessible from python only indirectly. At this point, I am wondering whether it would be easier to forget the asynio network io loop and write from scratch a tkinter-based loop class that has the minimum needed to work with async and await code. But what is that minimum? PEP 492 "assumes that the asynchronous tasks are scheduled and coordinated by an Event Loop similar to that of stdlib module asyncio.events.AbstractEventLoop." However, AsstractEventLoop is not that, but an abstract network/socket io event loop and a majority of the methods cannot be needed for non-network, non-socket code. Let me try again. What is the actual minimal event loop api required by await? -- Terry Jan Reedy From guido at python.org Thu Feb 25 11:47:20 2016 From: guido at python.org (Guido van Rossum) Date: Thu, 25 Feb 2016 08:47:20 -0800 Subject: [Python-ideas] How the heck does async/await work in Python 3.5 In-Reply-To: <56CEA414.7070609@udel.edu> References: <56c7d145$0$1597$c3e8da3$5496439d@news.astraweb.com> <56CCC98C.5060504@mail.de> <56CEA414.7070609@udel.edu> Message-ID: On Wed, Feb 24, 2016 at 10:49 PM, Terry Reedy wrote: > My memory is that any replacement event loop is expected to be a full > implmentation of the network io loop, so one cannot just make a gui loop > that works with coroutines. That's not correct. If an I/O loop does not implement the network I/O APIs it just cannot be used for network I/O, but it can still implement all the other functionality (like call_later). There are other loops that don't implement everything (e.g. the subprocess and datagram APIs are not supported by Proactor, and signal support is spotty, too). If you find some implementation barrier that prevents that from working it should be fixed. [...] > At this point, I am wondering whether it would be easier to forget the > asynio network io loop and write from scratch a tkinter-based loop class > that has the minimum needed to work with async and await code. But what is > that minimum? PEP 492 "assumes that the asynchronous tasks are scheduled > and coordinated by an Event Loop similar to that of stdlib module > asyncio.events.AbstractEventLoop." However, AsstractEventLoop is not that, > but an abstract network/socket io event loop and a majority of the methods > cannot be needed for non-network, non-socket code. > > Let me try again. What is the actual minimal event loop api required by > await? The await expression (or yield from) does not require an event loop. Heck, it doesn't require futures. It uses coroutines and it is totally up to the manager of those how they are scheduled (read Greg Ewing's very early explanations of yield from -- some have no I/O at all). But there's something else that you might be after. The APIs related to scheduling callbacks (timed or not) are the most fundamental. I find run_until_complete() also fundamental, and requires a Future class. So you need that (but you could have your own Future implementation if you start from scratch or from AbstractEventLoop, leaving the network I/O methods unimplemented. (After all they don't use @abstractmethod so as long as you don't call them you should be okay leaving them unimplemented.) You really should be able to pick and choose, and the comments with section names in AbstractEventLoop are meant to provide you with guidance as to what to implement and what to leave out. When you're implementing this API on top of tkinter, you'll probably find that you'll have to use tkinter's way of sleeping anyways, so the implementation of waiting in BaseEventLoop using a selector is not useful for this scenario. There are probably some possible refactorings in the asyncio package to help you reuse a little more code, but all in all I still think it would be very useful to have an asyncio loop integrated with Tkinter. (Of course Tkinter does support network I/O, so it would be possible to integrate with that, too. Or some hybrid where you somehow figure out how to wait using a Selector *or* tkinter events in the same loop.) -- --Guido van Rossum (python.org/~guido) From abarnert at yahoo.com Thu Feb 25 13:42:01 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 25 Feb 2016 10:42:01 -0800 Subject: [Python-ideas] How the heck does async/await work in Python 3.5 In-Reply-To: References: <56c7d145$0$1597$c3e8da3$5496439d@news.astraweb.com> <56CCC98C.5060504@mail.de> <56CEA414.7070609@udel.edu> Message-ID: On Feb 25, 2016, at 08:47, Guido van Rossum wrote: > > (Of course Tkinter does support network I/O, so it would be possible > to integrate with that, too. Or some hybrid where you somehow figure > out how to wait using a Selector *or* tkinter events in the same > loop.) I remember that back in the late 90s, you often needed a loop like this pseudocode: next_ui = now() + 20ms while True: livesocks = poll(socks, timeout=next_ui-now()) for sock in livesocks: handle(sock) if now() >= next_ui: ui_loop_once() next_ui = now() + 20ms The idea is that GUI/audio/video/game code expects to only fire 20-60 times/second, and may do a lot of work each time; network code dealing with dozens of sockets expects to be fired a lot more often, and to do a lot less work each time. Is this still an issue? If I wanted to write, say, a BitTorrent client like Transmission or uTorrrent using Tkinter, and I fired the Tkinter loop after every selector poll, would it waste too much time checking for events from the OS and onidle handlers and so on thousands of times/second? (And if this is still an issue, is waking 50 times/sec still an acceptable way to solve it, or will you be keeping laptops from sleeping and so on?) I feel like I must have worked on something over the past decade that would tell me that... but everything I can think of (that didn't use something like Qt to abstract it out), the network stuff goes on a different thread or threads, and communicates with the UI by posting events in one direction and kicking a pipe in the other. From jsbueno at python.org.br Thu Feb 25 13:52:45 2016 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Thu, 25 Feb 2016 15:52:45 -0300 Subject: [Python-ideas] How the heck does async/await work in Python 3.5 In-Reply-To: References: <56c7d145$0$1597$c3e8da3$5496439d@news.astraweb.com> <56CCC98C.5060504@mail.de> <56CEA414.7070609@udel.edu> Message-ID: On 25 February 2016 at 15:42, Andrew Barnert via Python-ideas wrote: > Is this still an issue? If I wanted to write, say, a BitTorrent client like Transmission or uTorrrent using Tkinter, and I fired the Tkinter loop after every selector poll, would it waste too much time checking for events from the OS and onidle handlers and so on thousands of times/second? (And if this is still an issue, is waking 50 times/sec still an acceptable way to solve it, or will you be keeping laptops from sleeping and so on?) This sounds like we will need a "gear reduction" adapter function to be thrown in: a function that would count the maximum calls-per-second for the low-update part of the code - maybe it could go as a decorator in a helper project for AsyncIO. (disclaimer: I was on a bad day, and yesterday on the local hackerspace Python meeting I help coodinate, people asked for an example tic-tac-toe after I suggested exploring something with Pygame. We ended the night with possibly the first 30 frames/sec tic-tac-toe ever) From abarnert at yahoo.com Fri Feb 26 00:27:02 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 26 Feb 2016 05:27:02 +0000 (UTC) Subject: [Python-ideas] Standard (portable) bytecode "assembly" format References: <1715964377.2673152.1456464422259.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <1715964377.2673152.1456464422259.JavaMail.yahoo@mail.yahoo.com> On -dev (http://article.gmane.org/gmane.comp.python.devel/156543), Demur Rumed asked what it would take to get the "wordcode" patch into Python. Obviously, we need to finish it, benchmark it, etc., but on top of that, as Guido pointed out: > An unfortunate issue however is that many projects seem to make a > hobby of hacking bytecode. All those projects would have to be totally > rewritten in order to support the new wordcode format (as opposed to > just having to be slightly adjusted to support the occasional new > bytecode opcode). Those projects of course don't work with Pypy or > Jython either, but they do work for mainstream CPython, and it's > unacceptable to just leave them all behind. Greg Ewing replied: > Maybe this argues for having an assembly-language-like > intermediate form between the AST and the actual code > used by the interpreter? Done properly it could make > things easier for bytecode-hacking projects as well as > providing some insulation from implementation details. I think he's right. Of course we already have such a format today: dis.Bytecode. But it doesn't quite solve the problem, for three reasons: * Not accessible from C. * Not mutable, and no assembler. * A few things (mainly jump arguments) are still in terms of bytecode bytes. But fix that, and we have a format that will be unchanged with wordcode, and that can work out of the box in MicroPython (which has a not-quite-CPython bytecode format), and so on. I think if we do that for 3.6, then it's plausible to consider wordcode for 3.7. And, fix it well enough, and it also solves the problem I brought up a few weeks ago (http://article.gmane.org/gmane.comp.python.ideas/38431): if PEP 511 is going to provide a builtin API for registering bytecode processors, we should make it feasible to write them. I have a somewhat complete proposal (at http://stupidpythonideas.blogspot.com/2016/02/a-standard-assembly-format-for-python.html), but until I actually implement it, most people should only care about this summary: * Iterable of (opcode, argval [, line [, ...]]) tuples. The argval is the actual global name, constant value, etc., not the encoded index, etc. For jumps, the argval is just the target instruction itself. The existing dis.Bytecode (with a few minor changes) already fits this type--but so does, say, a list of 3-tuples, which we can much more easily build in C. * The assemble function from compile.c doesn't need that much work to convert it into a PyCode_Assemble/dis.assemble that takes such an iterable (plus optional name, filename, and first_line) and generates a code object. The compiler can then use the same function as pure Python code. And PyCode_Assemble is the only new C API function needed. * We already have a disassembler for this format in the stdlib since 3.4. It does need a few minor changes, and there are a few simple extensions that I think are worth adding (like making Bytecode a MutableSequence), but that's it. * Assuming the assembler drops NOPs, we can use NOPs as pseudo-instructions for when you want byteplay-like Label and SetLineNo. The disassembler can optionally even generate them. So, we don't need explicit pseudo-instructions. * Any higher-level representations, like a graph of blocks with edges for the jumps between them, are easy enough to build on top of the dis representation (and to flatten back into that representation), so we don't need anything more complicated in the stdlib. From victor.stinner at gmail.com Fri Feb 26 05:27:00 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 26 Feb 2016 11:27:00 +0100 Subject: [Python-ideas] Standard (portable) bytecode "assembly" format In-Reply-To: <1715964377.2673152.1456464422259.JavaMail.yahoo@mail.yahoo.com> References: <1715964377.2673152.1456464422259.JavaMail.yahoo.ref@mail.yahoo.com> <1715964377.2673152.1456464422259.JavaMail.yahoo@mail.yahoo.com> Message-ID: 2016-02-26 6:27 GMT+01:00 Andrew Barnert via Python-ideas : > Of course we already have such a format today: dis.Bytecode. But it doesn't quite solve the problem, for three reasons: > > * Not accessible from C. I don't think that it's a real issue. The current trend is more to rewrite pieces of CPython in Python. importlib is a great example of that. importlib is also an interested case because it is a Python module to improts modules, but we need importlib to import importlib. Hum. Brett Canon solved this issue by compiling the Python code to a frozen module. It means that we can do something similar if we want to rewrite the peephole optimizer in Python. > * Not mutable, and no assembler. I looked at Bytecode & Instruction objects of dis. They look nice to "read" bytecode, but not to modify bytecode. dis.Instruction is not mutable and informations are duplicated. For example, the operator is stored as name (LOAD_CONST) and code (100). Argument is stored as int (1), value ("hello") and representation ('"hello"'). It has no methods but attributes like is_jump_target. dis.Instruction doesn't seem extensible to add new features. Adding more items to such namedtuple doesn't seem like a nice API to me. > * A few things (mainly jump arguments) are still in terms of bytecode bytes. Hum, this is a problem. The dis is already in the stdlib, you cannot modify its API in a backward incompatible way. IMHO it's safer and simpler to add something new (maybe in a new module), not modify the existing Bytecode & Instruction classes. To modify bytecode, you need a different structure. For example, jump targets must be abstracted with labels (or something else). In my bytecode module, I provide 3 different ways to expose bytecode: * ConcreteBytecode: instructions close to raw bytes structure, arguments must be integers * Bytecode: list of abstract instructions using labels * BytecodeBlocks: use blocks, a block is a list of instructions with a label, jumps point to blocks An instruction is an object which contains a line number, has methods like is_jump(). Abstract instructions can be modified (lineno, name, op, arg), they have no size. Concrete instructions have size, attributes cannot be modified. Concrete bytecode & instructions is closer to what we already have in the dis module. I'm not sure that it's useful, maybe I should keep it private. It's an intermediate format to disassemble and assemble code objects. Instr('LOAD_CONST') argument is directly the constant value, so Bytecode has no "consts" attribute. ConcreteInstr('LOAD_CONST') agument is an integer: the index to the consts list of the ConcreteBytecode. BytecodeBlocks is a "flat" control flow graph (CFG). It is required by the peephole optimizer to not modify two instructions which are part of two code paths (two blocks). Side-effect, with blocks, it's trivial to detect dead code. As you wrote, it's also possible to reorder blocks to try to avoid jumps. Note: the current peephole optimizer miss a lot of optimizations on jumps :-/ Python 3.5 is a little bit better. > And, fix it well enough, and it also solves the problem I brought up a few weeks ago (http://article.gmane.org/gmane.comp.python.ideas/38431): if PEP 511 is going to provide a builtin API for registering bytecode processors, we should make it feasible to write them. Again, you don't need to add anything to the stdlib to write a bytecode optimizer. byteplay, codetransformer, bytecode, etc. projects are already available. By the way, I wrote PEP 511 for AST optimizers, not for bytecode optimizers. Since we can modify AST, bytecode is less interesting. Writing an optimizer on bytecode depends too much on the implementation. It may break if we add new bytecode instructions or modify the format of instructions (as you said). It's a deliberate choice to leave optimizers out of the stdlib. I expect that it will take months or years to stabilize the API of an optimizer, test it with various kinds of applications, etc. > * Iterable of (opcode, argval [, line [, ...]]) tuples. The argval is the actual global name, constant value, etc., not the encoded index, etc. For jumps, the argval is just the target instruction itself. The existing dis.Bytecode (with a few minor changes) already fits this type--but so does, say, a list of 3-tuples, which we can much more easily build in C. You should not use "be usable in C" constraint, or I expect a bad API :-/ When you modify bytecode, you need many functions which are revelant to be put in instruction objects. See existing codetransformer & bytecode projects for examples of methods. Note: my bytecode.Instr has a simple constructor, it takes 2 or 3 parameters: Instr(lineno, name, arg=UNSET). I don't think that it's hard to write an helper function in C to emit Instr object if you *really* want to write C code. > And PyCode_Assemble is the only new C API function needed. I don't understand why do you care so much of having a C API. What do you want to do? The only need for CPython is to have the most simple peephole optimizer, basically only optimize jumps. An AST optimizer can do everything else. I would like to experiment such peephole optimizer implemented in pure Python. I'm not sure that writing it in pure Python will kill performances. The cost of import matters, but only in few use cases. In general, applications run longer than 1 second and so the cost of import is negligible. Moreover, .py are only compiled once to .pyc. If .pyc are precompiled, the speed of the optimizer doesn't matter :-) > * Assuming the assembler drops NOPs, we can use NOPs as pseudo-instructions for when you want byteplay-like Label and SetLineNo. The disassembler can optionally even generate them. So, we don't need explicit pseudo-instructions. For pattern matching, inline Label or SetLineno instrutions are annoying. For example, if you use the pattern "LOAD_CONST ; UNARY_NOT", "SetLineno 3; LOAD_CONST ; SetLineno 3; UNARY_NOT" will not match. You *can* modify the algorithm to match patterns, but putting line numbers in instructions avoid this issue. Using multiple blocks rather than a single list of instructions avoid the need of inline labels. In my bytecode project, I tried to support both API: inline labels in Bytecode, labels in blocks in BytecodeBlocks. I may add support for Setlineno later. I'm still working on the API. > * Any higher-level representations, like a graph of blocks with edges for the jumps between them, are easy enough to build on top of the dis representation (and to flatten back into that representation), so we don't need anything more complicated in the stdlib. Yeah, you should start with something simple but extensible. An API generic enough to be usable as a low-level API by existing byteplay, codetransformer, bytecode projects, and then build an higher-level API on top of that. Or maybe I'm right and it's a bad idea :-) codetransformer is more than just an API to modify bytecode. It has an API to match instructions using patterns. Such stuff should be kept in codetransformer. Victor From ncoghlan at gmail.com Fri Feb 26 06:36:15 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 26 Feb 2016 21:36:15 +1000 Subject: [Python-ideas] Standard (portable) bytecode "assembly" format In-Reply-To: <1715964377.2673152.1456464422259.JavaMail.yahoo@mail.yahoo.com> References: <1715964377.2673152.1456464422259.JavaMail.yahoo.ref@mail.yahoo.com> <1715964377.2673152.1456464422259.JavaMail.yahoo@mail.yahoo.com> Message-ID: On 26 February 2016 at 15:27, Andrew Barnert via Python-ideas wrote: > Of course we already have such a format today: dis.Bytecode. I wouldn't feel beholden to hewing too closely to the existing dis API - that was defined to solve a specific problem with making it easier to test CPython's code generation pipeline, while also providing an improved foundation for the dis text output. For those use cases, the redundancy in the API is a help, rather than a hindrance, since we can easily test and display all the values of interest. For manipulation though, the redundancy is a problem - you need to either declare some fields authoritative and implicitly derive the others, or else expect users to keep things in sync manually (which would be a pretty user hostile API). I do think it's reasonable to seek to define a standard MutableBytecode format specifically to make manipulation easier, but I don't think it makes sense to couple that to PEP 511's definition of bytecode processing. The reason I feel that way is that I consider it *entirely acceptable* for the first generation of bytecode post-processors to be based on the disassemble-manipulate-reassemble model that folks already use for bytecode manipulating function decorators, and for doing that conveniently to be dependent on 3rd party libraries, at least for the time being. If we later settle on a standard mutable bytecode format, we may also decide to introduce bytecode pre-processors that accept and produce the pre-assembly form of the bytecode, but that's something to be done as a possible compile time reduction measure *after* folks have practical experience with the easier to define post-processing approach, rather than before. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Fri Feb 26 09:33:42 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 26 Feb 2016 15:33:42 +0100 Subject: [Python-ideas] Standard (portable) bytecode "assembly" format In-Reply-To: References: <1715964377.2673152.1456464422259.JavaMail.yahoo.ref@mail.yahoo.com> <1715964377.2673152.1456464422259.JavaMail.yahoo@mail.yahoo.com> Message-ID: I just released bytecode 0.1, to discuss a "stable" (released) API :-) https://pypi.python.org/pypi/bytecode Instr constructor is now: Instr(name, arg=UNSET, *, lineno=None). I added SetLineno pseudo-instruction. If Instr is created with no line number, the line number is inherited from previous instructions, from SetLineno, or from the first line number of the bytecode object (default: 1). Mandatory "Hello World" example: from bytecode import Instr, Bytecode bytecode = Bytecode() bytecode.extend([Instr("LOAD_NAME", 'print'), Instr("LOAD_CONST", 'Hello World!'), Instr("CALL_FUNCTION", 1), Instr("POP_TOP"), Instr("LOAD_CONST", None), Instr("RETURN_VALUE")]) code = bytecode.to_code() exec(code) Victor From brett at python.org Fri Feb 26 11:50:21 2016 From: brett at python.org (Brett Cannon) Date: Fri, 26 Feb 2016 16:50:21 +0000 Subject: [Python-ideas] Standard (portable) bytecode "assembly" format In-Reply-To: References: <1715964377.2673152.1456464422259.JavaMail.yahoo.ref@mail.yahoo.com> <1715964377.2673152.1456464422259.JavaMail.yahoo@mail.yahoo.com> Message-ID: On Fri, 26 Feb 2016 at 02:27 Victor Stinner wrote: > 2016-02-26 6:27 GMT+01:00 Andrew Barnert via Python-ideas > : > [SNIP] > > > And PyCode_Assemble is the only new C API function needed. > > I don't understand why do you care so much of having a C API. What do > you want to do? > > The only need for CPython is to have the most simple peephole > optimizer, basically only optimize jumps. An AST optimizer can do > everything else. I would like to experiment such peephole optimizer > implemented in pure Python. > > I'm not sure that writing it in pure Python will kill performances. > The cost of import matters, but only in few use cases. In general, > applications run longer than 1 second and so the cost of import is > negligible. Moreover, .py are only compiled once to .pyc. If .pyc are > precompiled, the speed of the optimizer doesn't matter :-) > So one thing to point out (that I know Raymond Hettinger would ;) ), is that Python scripts passed by file path on the command-line are not written out to a .pyc file, and so at least the initial entry point will still have to pay for any optimizer overhead no matter what. And if you write your entire app in a single file for ease-of-shipment then you will pay the penalty 100% of the time and not just for some small __main__ module. Now if the balance between overhead vs. the optimization benefit for everyone else balances out then it's worth the cost, but the question is what exactly that cost works out to be. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Feb 26 11:51:39 2016 From: brett at python.org (Brett Cannon) Date: Fri, 26 Feb 2016 16:51:39 +0000 Subject: [Python-ideas] Standard (portable) bytecode "assembly" format In-Reply-To: References: <1715964377.2673152.1456464422259.JavaMail.yahoo.ref@mail.yahoo.com> <1715964377.2673152.1456464422259.JavaMail.yahoo@mail.yahoo.com> Message-ID: On Fri, 26 Feb 2016 at 06:34 Victor Stinner wrote: > I just released bytecode 0.1, to discuss a "stable" (released) API :-) > https://pypi.python.org/pypi/bytecode > > Instr constructor is now: Instr(name, arg=UNSET, *, lineno=None). > > I added SetLineno pseudo-instruction. If Instr is created with no line > number, the line number is inherited from previous instructions, from > SetLineno, or from the first line number of the bytecode object > (default: 1). > > Mandatory "Hello World" example: > > from bytecode import Instr, Bytecode > > bytecode = Bytecode() > bytecode.extend([Instr("LOAD_NAME", 'print'), > Instr("LOAD_CONST", 'Hello World!'), > Instr("CALL_FUNCTION", 1), > Instr("POP_TOP"), > Instr("LOAD_CONST", None), > Instr("RETURN_VALUE")]) > code = bytecode.to_code() > exec(code) > Any reason you went with string constants instead of enums? -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Fri Feb 26 12:19:14 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 26 Feb 2016 18:19:14 +0100 Subject: [Python-ideas] Standard (portable) bytecode "assembly" format In-Reply-To: References: <1715964377.2673152.1456464422259.JavaMail.yahoo.ref@mail.yahoo.com> <1715964377.2673152.1456464422259.JavaMail.yahoo@mail.yahoo.com> Message-ID: 2016-02-26 17:51 GMT+01:00 Brett Cannon : >> bytecode.extend([Instr("LOAD_NAME", 'print'), >> Instr("LOAD_CONST", 'Hello World!'), >> Instr("CALL_FUNCTION", 1), >> Instr("POP_TOP"), >> Instr("LOAD_CONST", None), >> Instr("RETURN_VALUE")]) > > Any reason you went with string constants instead of enums? My API is still a work-in-progress :-) Maybe the API can be changed to: LOAD_CONST('Hello World!') POP_TOP() But it means that your code will probably starts with "from bytecode import *" or "from bytecode import LOAD_CONST, POP_TOP". There are something like 155 opcodes, so I would prefer to not have to write the exhaustive list of imports. Another option is something like: Instr.LOAD_CONST('Hello World!') Instr.POP_TOP() or whatever.LOAD_CONST('Hello World!') whatever.POP_TOP() I don't know what is the best. codetransformers uses instructions.LOAD_CONST("Hello World!") and instructions.LOAD_FAST is a type (it used for pattern matching). Victor From abarnert at yahoo.com Fri Feb 26 13:15:58 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 26 Feb 2016 10:15:58 -0800 Subject: [Python-ideas] Standard (portable) bytecode "assembly" format In-Reply-To: References: <1715964377.2673152.1456464422259.JavaMail.yahoo.ref@mail.yahoo.com> <1715964377.2673152.1456464422259.JavaMail.yahoo@mail.yahoo.com> Message-ID: <471F6FEC-6F50-42A6-B489-B8091C1A8BF3@yahoo.com> On Feb 26, 2016, at 02:27, Victor Stinner wrote: > > 2016-02-26 6:27 GMT+01:00 Andrew Barnert via Python-ideas > : >> Of course we already have such a format today: dis.Bytecode. But it doesn't quite solve the problem, for three reasons: >> >> * Not accessible from C. > > I don't think that it's a real issue. The current trend is more to > rewrite pieces of CPython in Python. Sure, we could either (a) have duplicate code in C and Python that do virtually the same assembly and fixup work, (b) rewrite the peephole optimizer and part of the compiler in Python and freeze both them and the dis module (or whatever), or (c) use a format that's accessible from both C and Python and change as little as possible to get what we want. I think the last one is clearly the best solution, but it's not because the other two aren't impossible. >> * Not mutable, and no assembler. > > I looked at Bytecode & Instruction objects of dis. They look nice to > "read" bytecode, but not to modify bytecode. > > dis.Instruction is not mutable and informations are duplicated. Which is exactly why I suggested the very alternative that you're replying to: tuples of (opcode, argval [, line [, ...]]) are trivial to build. Instruction (with a minor, backward-compatible change) is compatible with that, but you don't need to use Instruction. Similarly, an iterable of such tuples is trivial to build; Bytecode is compatible with that, but you don't need to use Bytecode. Here's an example of what a bytecode processor could look like: for opcode, argval, *rest in instructions: if opcode == dis.LOAD_GLOBAL: yield (dis.LOAD_CONST, eval(argval, globals(), *rest) else: yield (opcode, argval, *rest) If you want to use the dis structures instead, you don't have to, but you can: bc = dis.Bytecode(instructions) for i, instr in enumerate(bc): if instr.opcode == dis.LOAD_GLOBAL: bc[i] = instr.replace(opcode=dis.LOAD_CONST, eval(instr.argval, globals())) return bc And notice that, even if you _do_ want to use those structures, the problems you're imagining don't arise. There are more complicated examples on the linked blog post. > For > example, the operator is stored as name (LOAD_CONST) and code (100). > Argument is stored as int (1), value ("hello") and representation > ('"hello"'). It has no methods but attributes like is_jump_target. And, as I said, you only have to supply opcode, argval, and sometimes line. The other attributes are there for reading existing bytecode, but aren't needed for emitting it. This is the same model that's used successfully in the tokenize module. (Of course that module has some other API nightmares, but _this_ part of it is very nice.) Tokens are a namedtuple with 6 attributes, but you can construct them with just the first 2, 3, or 4, or you can just substitute a tuple of 2, 3, or 4 elements in place of a Token. > dis.Instruction doesn't seem extensible to add new features. Why not? I added hasjrel to see how easy it is: there's one obvious way to do it, which took a few seconds, and it works exactly as I'd want it to. What kind of new features do you think would be difficult to add? >> * A few things (mainly jump arguments) are still in terms of bytecode bytes. > > Hum, this is a problem. The dis is already in the stdlib, you cannot > modify its API in a backward incompatible way. Again, already covered, and covered in more detail in the blog post. > Abstract instructions can be modified (lineno, name, op, arg), they > have no size. Concrete instructions have size, attributes cannot be > modified. > > Concrete bytecode & instructions is closer to what we already have in > the dis module. No it isn't. What we have in the dis module does _not_ have size; it's a flat sequence of instructions. If you've missed that, you probably need to go back and reread the proposal, because it doesn't really make sense if you think this is what it's suggesting. > BytecodeBlocks is a "flat" control flow graph (CFG). It is required by > the peephole optimizer to not modify two instructions which are part > of two code paths (two blocks). Here we get to the core of the proposal. As I show in the linked blog post, it takes a handful of lines to go back and forth between the proposed format and a block-graph format. It's just as easy to go back and forth between having pseudo-instructions and not having them. Or any other format you come up with. That's not true for raw bytecode--going back and forth requires writing a complicated disassembler and even more complicated assembler. But, even more important, the proposed format is the same between CPython 3.6 and MicroPython 3.6, and it stays the same even if CPython 3.7 switches to wordcode. And any code you've written that builds a block graph out of the proposed format still works. That's what makes the proposed format a portable, resilient format. And I believe it's the simplest possible portable, resilient format. It's not the ideal format to use for every possible kind of bytecode manipulation. That isn't the goal. The fact that it happens to be good enough for a lot of kinds of bytecode manipulation is a nice side benefit, but it's not the point. The fact that it integrates nicely with dis is also very nice, but it's not the point. So, "let's build yet another third-party assembler and disassembler with a different API" is not a competing solution to this proposal; it's part of the problem I'm trying to solve. > By the way, I wrote PEP 511 for AST optimizers, not for bytecode > optimizers. As I've said before: you included bytecode optimizers in PEP 511, you made the API more complicated so you could allow them, you provide a rationale for why we need to allow them, and you gave an example of one. If the PEP is wrong, you don't have to convince anyone; it's your PEP, go change it. Anyway, from here you go off onto a long tangent arguing that my proposed format is not the ideal once-and-for-all-best format to use for every possible kind of bytecode manipulation. I already granted that above, and I'll grant it again and snip all the arguments. >> And PyCode_Assemble is the only new C API function needed. > > I don't understand why do you care so much of having a C API. What do > you want to do? As explained near the top, I want to share code between the assemble function in the compiler and the assemble function used in Python code. Ideally, I'd like to do this without having to expose any new types or utility functions or anything else to C. And, as it turns out, that's doable. I can write a PyCode_Assemble function that's used by the compiler and by Python code without having to add a single other new thing to the C API. >> * Any higher-level representations, like a graph of blocks with edges for the jumps between them, are easy enough to build on top of the dis representation (and to flatten back into that representation), so we don't need anything more complicated in the stdlib. > > Yeah, you should start with something simple but extensible. An API > generic enough to be usable as a low-level API by existing byteplay, > codetransformer, bytecode projects, and then build an higher-level API > on top of that. Or maybe I'm right and it's a bad idea :-) I don't understand the last sentence. Are you contradicting the rest of the paragraph, and suggesting that a simple but extensible API that can be used by byteplay, etc. and new projects is a bad thing? If so, why? Do you think it would be better to bless one of those projects, and keep all the others as hard to write as they are today? From ethan at stoneleaf.us Fri Feb 26 13:23:45 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 26 Feb 2016 10:23:45 -0800 Subject: [Python-ideas] Standard (portable) bytecode "assembly" format In-Reply-To: References: <1715964377.2673152.1456464422259.JavaMail.yahoo.ref@mail.yahoo.com> <1715964377.2673152.1456464422259.JavaMail.yahoo@mail.yahoo.com> Message-ID: <56D09831.3010104@stoneleaf.us> On 02/26/2016 09:19 AM, Victor Stinner wrote: > But it means that your code will probably starts with "from bytecode > import *" or "from bytecode import LOAD_CONST, POP_TOP". There are > something like 155 opcodes, so I would prefer to not have to write the > exhaustive list of imports. There is a reason that `from module import *` is still available, and this is one of them. You could also put all the opcodes (and just the opcodes) into their own module to limit the `import *` reach: from bytecode.opcodes import * -- ~Ethan~ From abarnert at yahoo.com Fri Feb 26 14:21:59 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 26 Feb 2016 11:21:59 -0800 Subject: [Python-ideas] Standard (portable) bytecode "assembly" format In-Reply-To: References: <1715964377.2673152.1456464422259.JavaMail.yahoo.ref@mail.yahoo.com> <1715964377.2673152.1456464422259.JavaMail.yahoo@mail.yahoo.com> Message-ID: <7EBE5C45-5EEA-4715-9266-ECD927F8940A@yahoo.com> On Feb 26, 2016, at 03:36, Nick Coghlan wrote: > > On 26 February 2016 at 15:27, Andrew Barnert via Python-ideas > wrote: >> Of course we already have such a format today: dis.Bytecode. > > I wouldn't feel beholden to hewing too closely to the existing dis API I don't feel too beholden to it. In fact, I only came back to it by accident. A few weeks ago, I proposed extending dis into something like byteplay and giving it a C API so the compiler could use it too. I decided that was a mistake, and went back to trying to come up with the simplest portable format that both the compiler and Python code could use, and came up with the iterable of tuples. And then I went and looked at what it would take to make dis support that format, and it turns out to take very little. So, rather than building a new Python convenience module for portable assembly (which I don't think we want to do), or not having one and just making people deal with the tuples with no convenience features (which is also viable, but I think less desirable), let's just use dis. > For manipulation though, the redundancy is a problem - you need to > either declare some fields authoritative and implicitly derive the > others, The portable assembly format doesn't use Instructions, just tuples of 2 or more elements where the first three are opcode, argval, and line. A dis.Instruction object (if we reorder its attributes and make the fact that it's a namedtuple type public) fits that, which means you can use it for convenience when it's convenient (probably leaving the other fields None), but you can also just ignore it and use a plain tuple, and often that's the simplest thing. Or, of course, you can use a library like byteplay that uses dis where it's helpful but provides whatever API it wants. > I do think it's reasonable to seek to define a standard > MutableBytecode format specifically to make manipulation easier I'm not sure it is. After failing to convince Victor that byteplay really can do what he wants, I now think that it's more reasonable to define a standard MutableBytecode format only to make it easier to build third-party libraries that each make manipulation easier in different ways. > , but I > don't think it makes sense to couple that to PEP 511's definition of > bytecode processing. Remember the starting motivation. What's the biggest stumbling block to switching to wordcode? It breaks code that manipulates bytecode directly. Adding a portable, future-proof format doesn't do any good if, at the same time, we also add something that encourages new code that ignores that format and instead manipulates bytecode directly. > The reason I feel that way is that I consider it *entirely acceptable* > for the first generation of bytecode post-processors to be based on > the disassemble-manipulate-reassemble model that folks already use for > bytecode manipulating function decorators, and for doing that > conveniently to be dependent on 3rd party libraries, at least for the > time being. My goal is to update byteplay to use the portable iterable-of-tuples format and lean on the built-in assembler if 3.6+. Even though plenty of things are actually simple enough to write with the iterable-of-tuples portable format, byteplay still has the advantage of (a) working back to Python 2.6, and (b) working with all of the existing code I've written for it over the last half-decade, so I will continue to use it. And I'm sure many other people will use it, or other third-party libraries. But, if most of those libraries (and most code people write without third-party libraries) rely on the portable format, then they'll continue to work throughout the 3.7 development cycle rather than making users wait 3-24 months before upgrading to 3.7. And it will allow us to make more radical changes in 3.7. And it'll allow MicroPython to support most of those libraries despite a slightly different internal format. And so on. > If we later settle on a standard mutable bytecode format, we may also > decide to introduce bytecode pre-processors that accept and produce > the pre-assembly form of the bytecode, but that's something to be done > as a possible compile time reduction measure *after* folks have > practical experience with the easier to define post-processing > approach, rather than before. If compile-time performance were the issue here, I'd agree. But it's not an issue--or, if it is, it's a distant fourth place behind resilience, portability, and simplicity. And to get resilience and portability, we have to use a resilient and portable format from the start, not bolt one on later as an option. From abarnert at yahoo.com Fri Feb 26 14:28:54 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 26 Feb 2016 11:28:54 -0800 Subject: [Python-ideas] Standard (portable) bytecode "assembly" format In-Reply-To: References: <1715964377.2673152.1456464422259.JavaMail.yahoo.ref@mail.yahoo.com> <1715964377.2673152.1456464422259.JavaMail.yahoo@mail.yahoo.com> Message-ID: <9524912B-A2BE-4402-B1FA-9BF16070AE1C@yahoo.com> Can we change the subject for this? Bikeshedding one of multiple different higher-level APIs that could be used for different kinds of bytecode processing is off-topic from having a simple portable format for representing bytecode, except tangentially in that I hope (and am pretty sure) that all such higher-level APIs can be built on top of the portable format. Sent from my iPhone > On Feb 26, 2016, at 09:19, Victor Stinner wrote: > > 2016-02-26 17:51 GMT+01:00 Brett Cannon : >>> bytecode.extend([Instr("LOAD_NAME", 'print'), >>> Instr("LOAD_CONST", 'Hello World!'), >>> Instr("CALL_FUNCTION", 1), >>> Instr("POP_TOP"), >>> Instr("LOAD_CONST", None), >>> Instr("RETURN_VALUE")]) >> >> Any reason you went with string constants instead of enums? > > My API is still a work-in-progress :-) Maybe the API can be changed to: > > LOAD_CONST('Hello World!') > POP_TOP() > > But it means that your code will probably starts with "from bytecode > import *" or "from bytecode import LOAD_CONST, POP_TOP". There are > something like 155 opcodes, so I would prefer to not have to write the > exhaustive list of imports. > > Another option is something like: > > Instr.LOAD_CONST('Hello World!') > Instr.POP_TOP() > > or > > whatever.LOAD_CONST('Hello World!') > whatever.POP_TOP() > > I don't know what is the best. codetransformers uses > instructions.LOAD_CONST("Hello World!") and instructions.LOAD_FAST is > a type (it used for pattern matching). > > Victor From victor.stinner at gmail.com Fri Feb 26 17:05:51 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 26 Feb 2016 23:05:51 +0100 Subject: [Python-ideas] Standard (portable) bytecode "assembly" format In-Reply-To: <471F6FEC-6F50-42A6-B489-B8091C1A8BF3@yahoo.com> References: <1715964377.2673152.1456464422259.JavaMail.yahoo.ref@mail.yahoo.com> <1715964377.2673152.1456464422259.JavaMail.yahoo@mail.yahoo.com> <471F6FEC-6F50-42A6-B489-B8091C1A8BF3@yahoo.com> Message-ID: 2016-02-26 19:15 GMT+01:00 Andrew Barnert : > Sure, we could either (a) have duplicate code in C and Python that do virtually the same assembly and fixup work, (b) rewrite the peephole optimizer and part of the compiler in Python and freeze both them and the dis module (or whatever), or (c) use a format that's accessible from both C and Python and change as little as possible to get what we want. I think the last one is clearly the best solution, but it's not because the other two aren't impossible. > (...) > As explained near the top, I want to share code between the assemble function in the compiler and the assemble function used in Python code. > > Ideally, I'd like to do this without having to expose any new types or utility functions or anything else to C. > > And, as it turns out, that's doable. I can write a PyCode_Assemble function that's used by the compiler and by Python code without having to add a single other new thing to the C API. Currently, Python/compile.c uses specific C structures: * struct instr: opcode, oparg, ... a jump target is pointer to a basicblock * struct basicblock: list of instructions, ... * struct fblockinfo * struct compiler_unit: list of constants, list of names, blocks, etc. * struct compiler: filename, compiler_unit, ... * ... Your proposal looks more like a flat list of instructions, it doesn't fit well with the current code (blocks). The structures contain many information which are specific to the compiler, I'm not sure that it would make sense to put them in your generic API. Or maybe you can rebuild the current structures on top of your API. My opinion on that is that it's not worth to modify Python/compile.c and leave it unchanged. >>> * Not mutable, and no assembler. >> >> I looked at Bytecode & Instruction objects of dis. They look nice to >> "read" bytecode, but not to modify bytecode. >> >> dis.Instruction is not mutable and informations are duplicated. > > Which is exactly why I suggested the very alternative that you're replying to: tuples of (opcode, argval [, line [, ...]]) are trivial to build. A tuple cannot be modified. By mutable, I mean being able to replace an attribute without having to create a new instruction: instr.arg = new_arg instead of bytecode[index] = instr.replace_arg(arg) In the first version in my bytecode project, I hesitated between abstract instruction and concrete instruction. I wanted to put checks, so I started with immutable instructions. But it's not really convenient. I would prefer mutable instructions. I left concrete instructions immutable, because arguments depend on a bytecode object. For example, LOAD_CONST uses an index in a list of constants. And jump targets depend on the exact size of other instructions. Maybe concrete bytecode should be made mutable too. But it's not too hard to create a new concrete instruction to replace an existing one. > Here's an example of what a bytecode processor could look like: > > for opcode, argval, *rest in instructions: Hum, "*rest" doesn't look good to me. What is the exact size of an instruction? (how many fields) What if we want to add a new field later? Will it break existing code relying on the minimum/maximum number of fields of an instruction? > If you want to use the dis structures instead, you don't have to, but you can: > > bc = dis.Bytecode(instructions) > for i, instr in enumerate(bc): Yeah, this API looks better: a single object which contains all information. It's more future-proof. (I just talking about the for "instr in bytecode:" :-)) >> For >> example, the operator is stored as name (LOAD_CONST) and code (100). >> Argument is stored as int (1), value ("hello") and representation >> ('"hello"'). It has no methods but attributes like is_jump_target. > > And, as I said, you only have to supply opcode, argval, and sometimes line. The other attributes are there for reading existing bytecode, but aren't needed for emitting it. Hum, I understand that an instruction is a named tuple. So if I create an instruction only with the opcode (ex: 100), the name field is not set, right? Which fields are "mandatory"? Which fields are optional? In my bytecode API, you provide a name, the opcode is computed from the name. You can modify the name, opcode is updated. If you modify opcode, name is updated. There are checks on lineno attributes (must be an int >= 1, or None). ConcreteInstr has strict checks on the argument. >> dis.Instruction doesn't seem extensible to add new features. > > Why not? I added hasjrel to see how easy it is: there's one obvious way to do it, which took a few seconds, and it works exactly as I'd want it to. What kind of new features do you think would be difficult to add? In bytecode 0.1, I have the following methods on Instr: * format(labels) * __repr__() * __eq__(): smart comparison. For LOAD_CONST, it understands that -0.0 argument is different than +0.0 for example. * is_jump() * is_cond_jump() ConcreteInstr has additional methods: * assemble() * ConcreteInstr.disassemble() (static method) * get_jump_target(instr_offset) About your hasjrel example: do you mean that you added a new field to the namedtuple? Does the constructor of the instruction have to fill this field manually? What if the field is not set? >> Concrete bytecode & instructions is closer to what we already have in >> the dis module. > > No it isn't. What we have in the dis module does _not_ have size; it's a flat sequence of instructions. If you've missed that, you probably need to go back and reread the proposal, because it doesn't really make sense if you think this is what it's suggesting. The offset attribute doesn't seem revelant for an abstract instruciton. If you remove an instruction before, the offset becomes inconsistent. I chose to not store the offset inside instructions, but recompute it each time that I iterate on concrete instructions (offset += instr.size). >> BytecodeBlocks is a "flat" control flow graph (CFG). It is required by >> the peephole optimizer to not modify two instructions which are part >> of two code paths (two blocks). > > Here we get to the core of the proposal. > > As I show in the linked blog post, it takes a handful of lines to go back and forth between the proposed format and a block-graph format. It's just as easy to go back and forth between having pseudo-instructions and not having them. Or any other format you come up with. I saw your "def blockify(instructions):" function, but I don't understand how do you store labels. You use a "is_jump_target" attribute. If you remove the target of a jump (an instruction with is_jump_target=True), I guess that you have to mark the following instrution with is_jump_target=True, right? What if the block only contains one instruction? I identified a requirement when you manipulate jumps: being able to "resolve jumps". From a jump, you want to know the target instruction. With bytecode.BytecodeBlocks, you get the target block with: "target_block = bytecode[jump.label]; target_instr = target_block[0]" (with a complexity of O(1), bycode[label] gets an item of an list, it uses a mapping label => block index to get the index.) With bytecode.Bytecode (list of instructions), you have to iterate on all iterations to search for the label. I can maybe optimize that later to build an internal cache, updated when the list is modified. I'm not sure that a label is the most convenient abstraction for blocks. In CFG, a jump points directly to a subtree (the instruction argument is directly the block), there is no indirection like my label object. In bytecode, you can also convert bytecode between the 3 formats (concrete, bytecode, blocks), the 3 classes have 5 conversion methods: * from_code() * to_code() * to_concrete_bytecode() * to_bytecode() * to_bytecode_blocks() > So, "let's build yet another third-party assembler and disassembler with a different API" is not a competing solution to this proposal; it's part of the problem I'm trying to solve. I wrote the bytecode project to try to implement you idea. It looks like we don't want the same API :-) >> Yeah, you should start with something simple but extensible. An API >> generic enough to be usable as a low-level API by existing byteplay, >> codetransformer, bytecode projects, and then build an higher-level API >> on top of that. Or maybe I'm right and it's a bad idea :-) > > I don't understand the last sentence. Sorry, I wanted to write "maybe I'm wrong and it's a bad idea". Victor From maxischmeii at gmail.com Fri Feb 26 17:12:16 2016 From: maxischmeii at gmail.com (Maxime S) Date: Fri, 26 Feb 2016 23:12:16 +0100 Subject: [Python-ideas] How the heck does async/await work in Python 3.5 In-Reply-To: References: <56c7d145$0$1597$c3e8da3$5496439d@news.astraweb.com> <56CCC98C.5060504@mail.de> <56CEA414.7070609@udel.edu> Message-ID: 2016-02-25 17:47 GMT+01:00 Guido van Rossum : > When you're implementing this API on top of tkinter, you'll probably > find that you'll have to use tkinter's way of sleeping anyways, so the > implementation of waiting in BaseEventLoop using a selector is not > useful for this scenario. > > There are probably some possible refactorings in the asyncio package > to help you reuse a little more code, but all in all I still think it > would be very useful to have an asyncio loop integrated with Tkinter. > (Of course Tkinter does support network I/O, so it would be possible > to integrate with that, too. Or some hybrid where you somehow figure > out how to wait using a Selector *or* tkinter events in the same > loop.) > It is actually quite easy to implement an asyncio loop over tkinter once you realise that tkapp.dooneevent() is very similar to poll(), and tkapp.createfilehandler() is very similar to register() (which is not that surprising since tcl use poll() internally if available). Thus it is possible to create a tk selector and reuse all the code in SelectorEventLoop. Unfortunately createfilehandler() is only available on UNIX, so some form of threading is probably innevitable on Windows. class TkSelector(selectors._BaseSelectorImpl): """Selector based on the Tk event loop.""" def __init__(self, app): super().__init__() self.app = app self.ready = [] self.is_timeout = False self.after_key = None def _file_cb(self, fileobj, mask): fd = self._fileobj_lookup(fileobj) self.ready.append((fd, mask)) def _timeout_cb(self): self.is_timeout = True def _reset_state(self): del self.ready[:] self.is_timeout = False if self.after_key: self.app.after_cancel(self.after_key) self.after_key = None def register(self, fileobj, events, data=None): key = super().register(fileobj, events, data) flags = 0 if events & EVENT_READ: flags |= tkinter.READABLE if events & EVENT_WRITE: flags |= tkinter.WRITABLE self.app.createfilehandler(fileobj, flags, self._file_cb) return key def unregister(self, fileobj): key = super().unregister(fileobj) self.app.deletefilehandler(fileobj) return key def select(self, timeout=None): if timeout is None: pass elif timeout <= 0: while self.app.dooneevent(_tkinter.DONT_WAIT): pass self.is_timeout = True else: self.after_key = self.app.after(math.ceil(timeout*1000), self._timeout_cb) while not (self.is_timeout or self.ready): self.app.dooneevent() ret = [] for fd, mask in self.ready: events = 0 if mask & tkinter.WRITABLE: events |= EVENT_WRITE if mask & tkinter.READABLE: events |= EVENT_READ key = self._key_from_fd(fd) if key: ret.append((key, events & key.events)) self._reset_state() return ret class TkEventLoop(asyncio.SelectorEventLoop): """Asyncio-compatible tkinter event loop.""" def __init__(self, app): selector = TkSelector(app) super().__init__(selector=selector) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Feb 26 18:47:57 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 26 Feb 2016 15:47:57 -0800 Subject: [Python-ideas] Standard (portable) bytecode "assembly" format In-Reply-To: References: <1715964377.2673152.1456464422259.JavaMail.yahoo.ref@mail.yahoo.com> <1715964377.2673152.1456464422259.JavaMail.yahoo@mail.yahoo.com> <471F6FEC-6F50-42A6-B489-B8091C1A8BF3@yahoo.com> Message-ID: On Feb 26, 2016, at 14:05, Victor Stinner wrote: > > 2016-02-26 19:15 GMT+01:00 Andrew Barnert : >> Sure, we could either (a) have duplicate code in C and Python that do virtually the same assembly and fixup work, (b) rewrite the peephole optimizer and part of the compiler in Python and freeze both them and the dis module (or whatever), or (c) use a format that's accessible from both C and Python and change as little as possible to get what we want. I think the last one is clearly the best solution, but it's not because the other two aren't impossible. >> (...) >> As explained near the top, I want to share code between the assemble function in the compiler and the assemble function used in Python code. >> >> Ideally, I'd like to do this without having to expose any new types or utility functions or anything else to C. >> >> And, as it turns out, that's doable. I can write a PyCode_Assemble function that's used by the compiler and by Python code without having to add a single other new thing to the C API. > > Currently, Python/compile.c uses specific C structures: Yes. But most of that information is only needed earlier in the process--building the blocks, linearizing them, making sure each one ends in a return, etc. It requires a bit of reorganization to cleanly separate out the final assembly/fixup step, but not that much. And, while that last step does use the current structures today, it doesn't actually need anything from them but a way to iterate instructions. (I learned this during an earlier experiment, where I shared all of the compiler structures directly with the peephole optimizer and then tried to limit the sharing as much as possible.) > My opinion on that is that it's not worth to modify Python/compile.c > and leave it unchanged. DRY. With a single assembler rather than two, anyone who wants to change anything about the internal bytecode format or how fixup works or anything else only has to do it once, rather than figuring out how to do the same thing in two completely different pieces of code that are intended to accomplish the same thing. >>>> * Not mutable, and no assembler. >>> >>> I looked at Bytecode & Instruction objects of dis. They look nice to >>> "read" bytecode, but not to modify bytecode. >>> >>> dis.Instruction is not mutable and informations are duplicated. >> >> Which is exactly why I suggested the very alternative that you're replying to: tuples of (opcode, argval [, line [, ...]]) are trivial to build. > > A tuple cannot be modified. So what? Even your examples don't mutate the Instr object; they build a new one instead. It's not like "bc[i] = Instruction(LOAD_CONST, constval)" or "yield (LOAD_CONST, constval)" and so on are less readable/Pythonic/concise than "bc[i].opcode = LOAD_CONST; bc[i].argval = constval". If you really think this is important, changing the format to any iterable of iterables instead of iterable of tuples is trivial, so you can use lists instead of tuples, and make Instruction mutable, and so on. But I don't see what it buys you. Also, one more time: I'm not trying to invent the all-singing, all-dancing best-possible interface for all kinds of bytecode manipulation; I'm trying to invent the simplest portable and resilient format that people can build other APIs on top of. The dis module happens to provide a somewhat useful such API for simple manipulations, which is nice, but it will never be the best one for all manipulations, and that's fine. So I don't want to change dis any more than necessary. Your own API can diverge much more radically from dis if it wants. It just has to take the same iterable-of-tuples format as input and output; it can store things internally however it wants, which can include mutable instruction objects if you want. > In my bytecode API, you provide a name, the opcode is computed from > the name. To me, requiring something that duck-types like an int, and then providing an IntEnum of all the opcodes, seems like a much nicer API than requiring strings and providing str<->int maps. That's exactly what enums are for. But if you want to build a string API on top of the portable duck-types-as-int format instead, you can. All you have to do is emit the ints in the iterable you pass to assemble. More generally, I don't think debating all, or even any, of the design decisions of your in-progress library is at all relevant to this discussion. Unless you think you have an example of something you can do with raw bytecode that you can't do with the portable format, it doesn't affect this proposal at all. >>> BytecodeBlocks is a "flat" control flow graph (CFG). It is required by >>> the peephole optimizer to not modify two instructions which are part >>> of two code paths (two blocks). >> >> Here we get to the core of the proposal. >> >> As I show in the linked blog post, it takes a handful of lines to go back and forth between the proposed format and a block-graph format. It's just as easy to go back and forth between having pseudo-instructions and not having them. Or any other format you come up with. > > I saw your "def blockify(instructions):" function, but I don't > understand how do you store labels. I gave examples that show how to do various things with and without storing labels. You're looking at one of the examples without storing labels, and asking where the labels are stored in that example. For that example, I instead make sure I have a complete dis.Bytecode with its is_jump_target fields filled in, and use that, to show that labels aren't always necessary. The next few questions are mostly irrelevant, so I'm skipping most of them. In general, you're asking how your library could do block-related things directly on the portable format. In many cases, what you want to do is actually easy, but it doesn't matter, because your library isn't going to do that; it's going to take the portable format as input and output, and do things on a graph of blocks in between, and the format you use for that graph of blocks is entirely up to you, as long as you can linearize that back to an iterable of tuples as the end. > I identified a requirement when you manipulate jumps: being able to > "resolve jumps". ... > With bytecode.Bytecode (list of instructions), you have to iterate on > all iterations to search for the label. No you don't. I gave examples that resolve jumps in O(1) time, both with and without labels. But, again, who cares? Your code won't be doing this. > I'm not sure that a label is the most convenient abstraction for > blocks. And it doesn't have to be. As long as it's sufficient for you to build whatever more convenient abstraction you think you need. >>> Yeah, you should start with something simple but extensible. An API >>> generic enough to be usable as a low-level API by existing byteplay, >>> codetransformer, bytecode projects, and then build an higher-level API >>> on top of that. Or maybe I'm right and it's a bad idea :-) >> >> I don't understand the last sentence. > > Sorry, I wanted to write "maybe I'm wrong and it's a bad idea". OK, then I think you're right, and it was a good idea. :) And that's exactly what I've attempted to do: come up with the simplest API that can support things like byteplay, codetransformer, and bytecode so they don't have to directly manipulate the byte strings anymore, giving us more freedom to change the internal CPython format without breaking every bytecode processor in the world. From tjreedy at udel.edu Sat Feb 27 03:08:53 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 27 Feb 2016 03:08:53 -0500 Subject: [Python-ideas] How the heck does async/await work in Python 3.5 In-Reply-To: References: <56c7d145$0$1597$c3e8da3$5496439d@news.astraweb.com> <56CCC98C.5060504@mail.de> <56CEA414.7070609@udel.edu> Message-ID: On 2/26/2016 5:12 PM, Maxime S wrote: > > 2016-02-25 17:47 GMT+01:00 Guido van Rossum > >: > > When you're implementing this API on top of tkinter, you'll probably > find that you'll have to use tkinter's way of sleeping anyways, so the > implementation of waiting in BaseEventLoop using a selector is not > useful for this scenario. > > There are probably some possible refactorings in the asyncio package > to help you reuse a little more code, but all in all I still think it > would be very useful to have an asyncio loop integrated with Tkinter. > (Of course Tkinter does support network I/O, so it would be possible > to integrate with that, too. Or some hybrid where you somehow figure > out how to wait using a Selector *or* tkinter events in the same > loop.) > > > It is actually quite easy to implement an asyncio loop over tkinter once > you realise that tkapp.dooneevent() is very similar to poll(), and It took me awhile to understand what you mean by 'tkapp'. Instances of tkinter.Tk get an undocumented .tk attribute that is an instance of the undocumented and hidden _tkinter class that is called 'tkapp' in printed representations. In other words, after import tkinter as tk root = tk.Tk() tkapp = root.tk tkapp has a dooneevent method. I found the tcl doc for it at https://www.tcl.tk/man/tcl/TclLib/DoOneEvent.htm Does calling it with DONT_WAIT "TCL_DONT_WAIT - Do not sleep: process only events that are ready at the time of the call." differ from calling root.update? def update(self): """Enter event loop until all pending events have been processed by Tcl.""" self.tk.call('update') If so, how? I interpret 'pending events' as 'events that are ready'. To add to the confusion, 2.7 aliased the tkapp methods (and attributes?) as Tkinter.tkinter functions, where Tkinter.tkinter was the C-coded _tkinter module. At least some people used the alieases. In 3.0, the aliases were removed, and when 'Tkinter' became 'tkinter', 'tkinter because '_tkinter'. (You may know all this, but I want it recorded and possibly added to the docs someday.) > tkapp.createfilehandler() is very similar to register() (which is not > that surprising since tcl use poll() internally if available). As you note below, tkapp.createfilehandler is unix-only and does not exist on Windows in 2.x or 3.x. In 2.7, however, the Tkinter.tkinter.createfilehandle alias did exist on Windows, but with with a value of None. > Thus it is possible to create a tk selector and reuse all the code in > SelectorEventLoop. Unfortunately createfilehandler() is only available > on UNIX, so some form of threading is probably innevitable on Windows. > > class TkSelector(selectors._BaseSelectorImpl): > """Selector based on the Tk event loop.""" > > def __init__(self, app): > super().__init__() > self.app = app > self.ready = [] > self.is_timeout = False > self.after_key = None > def _file_cb(self, fileobj, mask): > fd = self._fileobj_lookup(fileobj) > self.ready.append((fd, mask)) > def _timeout_cb(self): > self.is_timeout = True > def _reset_state(self): > del self.ready[:] > self.is_timeout = False > if self.after_key: > self.app.after_cancel(self.after_key) > self.after_key = None > > def register(self, fileobj, events, data=None): > key = super().register(fileobj, events, data) > flags = 0 > if events & EVENT_READ: > flags |= tkinter.READABLE > if events & EVENT_WRITE: > flags |= tkinter.WRITABLE > self.app.createfilehandler(fileobj, flags, self._file_cb) > return key > > def unregister(self, fileobj): > key = super().unregister(fileobj) > self.app.deletefilehandler(fileobj) > return key > > def select(self, timeout=None): > if timeout is None: > pass > elif timeout <= 0: > while self.app.dooneevent(_tkinter.DONT_WAIT): > pass > self.is_timeout = True > else: > self.after_key = self.app.after(math.ceil(timeout*1000), > self._timeout_cb) > while not (self.is_timeout or self.ready): > self.app.dooneevent() > ret = [] > for fd, mask in self.ready: > events = 0 > if mask & tkinter.WRITABLE: > events |= EVENT_WRITE > if mask & tkinter.READABLE: > events |= EVENT_READ > > key = self._key_from_fd(fd) > if key: > ret.append((key, events & key.events)) > self._reset_state() > return ret > > class TkEventLoop(asyncio.SelectorEventLoop): > """Asyncio-compatible tkinter event loop.""" > def __init__(self, app): > selector = TkSelector(app) > super().__init__(selector=selector) I am saving this for future reference. -- Terry Jan Reedy From maxischmeii at gmail.com Sat Feb 27 04:42:18 2016 From: maxischmeii at gmail.com (Maxime S) Date: Sat, 27 Feb 2016 10:42:18 +0100 Subject: [Python-ideas] How the heck does async/await work in Python 3.5 In-Reply-To: References: <56c7d145$0$1597$c3e8da3$5496439d@news.astraweb.com> <56CCC98C.5060504@mail.de> <56CEA414.7070609@udel.edu> Message-ID: 2016-02-27 9:08 GMT+01:00 Terry Reedy : > On 2/26/2016 5:12 PM, Maxime S wrote: > >> >> 2016-02-25 17:47 GMT+01:00 Guido van Rossum >> > >: >> >> When you're implementing this API on top of tkinter, you'll probably >> find that you'll have to use tkinter's way of sleeping anyways, so the >> implementation of waiting in BaseEventLoop using a selector is not >> useful for this scenario. >> >> There are probably some possible refactorings in the asyncio package >> to help you reuse a little more code, but all in all I still think it >> would be very useful to have an asyncio loop integrated with Tkinter. >> (Of course Tkinter does support network I/O, so it would be possible >> to integrate with that, too. Or some hybrid where you somehow figure >> out how to wait using a Selector *or* tkinter events in the same >> loop.) >> >> >> It is actually quite easy to implement an asyncio loop over tkinter once >> you realise that tkapp.dooneevent() is very similar to poll(), and >> > > It took me awhile to understand what you mean by 'tkapp'. Instances of > tkinter.Tk get an undocumented .tk attribute that is an instance of the > undocumented and hidden _tkinter class that is called 'tkapp' in printed > representations. In other words, after > > I actually meant an instance of Tk(), which also have dooneevent(). Sorry this wasn't clear. > import tkinter as tk > root = tk.Tk() > tkapp = root.tk > > tkapp has a dooneevent method. I found the tcl doc for it at > https://www.tcl.tk/man/tcl/TclLib/DoOneEvent.htm > Does calling it with DONT_WAIT "TCL_DONT_WAIT - Do not sleep: process only > events that are ready at the time of the call." differ from calling > root.update? > > Good point. The code under update() is essentially this (plus some error cheking): if (nargs == 1) { flags = TCL_DONT_WAIT; } else { flags = TCL_IDLE_EVENTS; } while (Tcl_DoOneEvent(flags) != 0) {} So, it is probably much more efficent to call update() than to do the same loop in python as I did, and it avoid messing around with _tkinter undocumented flags. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Feb 27 06:08:08 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 27 Feb 2016 06:08:08 -0500 Subject: [Python-ideas] How the heck does async/await work in Python 3.5 In-Reply-To: References: <56c7d145$0$1597$c3e8da3$5496439d@news.astraweb.com> <56CCC98C.5060504@mail.de> <56CEA414.7070609@udel.edu> Message-ID: On 2/27/2016 4:42 AM, Maxime S wrote: > 2016-02-27 9:08 GMT+01:00 Terry Reedy > >: > It took me awhile to understand what you mean by 'tkapp'. Instances > of tkinter.Tk get an undocumented .tk attribute that is an instance > of the undocumented and hidden _tkinter class that is called 'tkapp' > in printed representations. In other words, after > > > I actually meant an instance of Tk(), which also have dooneevent(). > Sorry this wasn't clear. I was fooled because in 3.x dooneevent does not show up on the root completion list. 3.x Tk must have a __getattr__ that pulls attributes from its tk (tkapp) attribute. # 3.5.1 >>> import tkinter as tk >>> root = tk.Tk() >>> 'dooneevent' in tk.Tk.__dict__ False >>> 'dooneevent' in root.__dict__ False >>> root.dooneevent For 2.x, dooneevent does appear as a completion, though I don't know how or why, as I get the same False and False as above. (I will have to see if the completion code changed, or if something else changed to change its meaning.) > import tkinter as tk > root = tk.Tk() > tkapp = root.tk > > tkapp has a dooneevent method. I found the tcl doc for it at > https://www.tcl.tk/man/tcl/TclLib/DoOneEvent.htm > Does calling it with DONT_WAIT "TCL_DONT_WAIT - Do not sleep: > process only events that are ready at the time of the call." differ > from calling root.update? > > > Good point. The code under update() is essentially this (plus some error > cheking): > > if (nargs == 1) { > flags = TCL_DONT_WAIT; > } else { > flags = TCL_IDLE_EVENTS; > } > > while (Tcl_DoOneEvent(flags) != 0) {} Thanks for checking this. > So, it is probably much more efficent to call update() than to do the > same loop in python as I did, and it avoid messing around with _tkinter > undocumented flags. -- Terry Jan Reedy