From steve at pearwood.info Thu Jan 1 00:14:19 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 1 Jan 2015 10:14:19 +1100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <DA710CDC-41FF-4097-B58A-83B711B8F8B7@yahoo.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <20141230234702.GE24472@ando.pearwood.info> <DA710CDC-41FF-4097-B58A-83B711B8F8B7@yahoo.com> Message-ID: <20141231231419.GG24472@ando.pearwood.info> On Wed, Dec 31, 2014 at 11:50:04AM +0100, Andrew Barnert wrote: > On Dec 31, 2014, at 0:47, Steven D'Aprano <steve at pearwood.info> wrote: > > What's wrong with asking the Finder to move the file? Under what > > circumstances would that be unacceptable? Are there scenarios on OS X > > where the Finder isn't available but the trash is? > > I already answered this briefly earlier in the thread, but in more detail: [...] Thank you for the detailed and extensive answer! -- Steven From ram at rachum.com Thu Jan 1 11:37:55 2015 From: ram at rachum.com (Ram Rachum) Date: Thu, 1 Jan 2015 12:37:55 +0200 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <20141231231419.GG24472@ando.pearwood.info> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <20141230234702.GE24472@ando.pearwood.info> <DA710CDC-41FF-4097-B58A-83B711B8F8B7@yahoo.com> <20141231231419.GG24472@ando.pearwood.info> Message-ID: <CANXboVbr-HVfA2adk7Vpog-63EeAtyP9W=mvnN2THbpyyM-CmQ@mail.gmail.com> Hi everyone, I'm a complete ignoramus when it comes to Mac, so I can't say anything about that, but as I understand the discussion, there are problems with implementing this on Mac. Okay. So maybe this should be available only on Windows and Linux? (If it's not a problem to do on Linux.) As much as I don't like having functionality that's only available on certain OSs, it makes more sense than depriving Windows users of functionality that they could be using just because Mac doesn't support it. (And of course, Python has lots of OS-specific modules.) So maybe we can add a function to the `os` module that sends a file to the recycle bin, and a constant that will say whether the current OS supports this? Then we could have code like this: if os.RECYCLE_BIN_SUPPORT: os.recycle(my_file) else: os.remove(my_file) What do you think? Thanks, Ram. On Thu, Jan 1, 2015 at 1:14 AM, Steven D'Aprano <steve at pearwood.info> wrote: > On Wed, Dec 31, 2014 at 11:50:04AM +0100, Andrew Barnert wrote: > > On Dec 31, 2014, at 0:47, Steven D'Aprano <steve at pearwood.info> wrote: > > > > What's wrong with asking the Finder to move the file? Under what > > > circumstances would that be unacceptable? Are there scenarios on OS X > > > where the Finder isn't available but the trash is? > > > > I already answered this briefly earlier in the thread, but in more > detail: > [...] > > Thank you for the detailed and extensive answer! > > > -- > Steven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150101/2626e926/attachment.html> From rosuav at gmail.com Thu Jan 1 11:46:37 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 1 Jan 2015 21:46:37 +1100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <CANXboVbr-HVfA2adk7Vpog-63EeAtyP9W=mvnN2THbpyyM-CmQ@mail.gmail.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <20141230234702.GE24472@ando.pearwood.info> <DA710CDC-41FF-4097-B58A-83B711B8F8B7@yahoo.com> <20141231231419.GG24472@ando.pearwood.info> <CANXboVbr-HVfA2adk7Vpog-63EeAtyP9W=mvnN2THbpyyM-CmQ@mail.gmail.com> Message-ID: <CAPTjJmq7A4w889PM3wdWjyMcxFGDXWoMs2Z5_8N5H4+vnFeV5Q@mail.gmail.com> On Thu, Jan 1, 2015 at 9:37 PM, Ram Rachum <ram at rachum.com> wrote: > As much as I don't like having functionality that's only available on > certain OSs, it makes more sense than depriving Windows users of > functionality that they could be using just because Mac doesn't support it. > (And of course, Python has lots of OS-specific modules.) Most OS-specific stuff is in the os module, but path handling is inherently platform-specific to some extent, so either place would be reasonable. > So maybe we can add a function to the `os` module that sends a file to the > recycle bin, and a constant that will say whether the current OS supports > this? Then we could have code like this: > > if os.RECYCLE_BIN_SUPPORT: > os.recycle(my_file) > else: > os.remove(my_file) > > What do you think? Or: try: delete = os.recycle except AttributeError: delete = os.remove delete(my_file) ChrisA From steve at pearwood.info Thu Jan 1 13:35:26 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 1 Jan 2015 23:35:26 +1100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <CANXboVbr-HVfA2adk7Vpog-63EeAtyP9W=mvnN2THbpyyM-CmQ@mail.gmail.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <20141230234702.GE24472@ando.pearwood.info> <DA710CDC-41FF-4097-B58A-83B711B8F8B7@yahoo.com> <20141231231419.GG24472@ando.pearwood.info> <CANXboVbr-HVfA2adk7Vpog-63EeAtyP9W=mvnN2THbpyyM-CmQ@mail.gmail.com> Message-ID: <20150101123526.GC15262@ando.pearwood.info> On Thu, Jan 01, 2015 at 12:37:55PM +0200, Ram Rachum wrote: > So maybe we can add a function to the `os` module that sends a file to the > recycle bin, and a constant that will say whether the current OS supports > this? Then we could have code like this: The os module is for low-level operating-system functions. "Send to trash" is neither low-level nor part of the OS per se, it is part of the desktop environment. I'm not convinced that this needs to be in the standard library, but if it is, I think that the os module is completely the wrong place for it. I think, in order of preference: 1) shutil 2) pathlib is the right place. (The actual implementation for the move_to_trash function could come from another, platform-specific, module.) > if os.RECYCLE_BIN_SUPPORT: > os.recycle(my_file) > else: > os.remove(my_file) "Recycle" is not a good name for the function, because it doesn't recycle the file. It does the opposite of recycle: it prevents the file from being deleted and over-written. I think an explicit name like move_to_trash is better than something misleading like "recycle". No need for a special flag to check if the function exists, just check for the function: try: f = shutil.send_to_trash except AttributeError: f = os.remove f(my_file) But I'm still not sure that this needs to be in the standard library. For such a specialist need, what's wrong with using a third party solution? -- Steven From ram at rachum.com Thu Jan 1 13:43:09 2015 From: ram at rachum.com (Ram Rachum) Date: Thu, 1 Jan 2015 14:43:09 +0200 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <20150101123526.GC15262@ando.pearwood.info> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <20141230234702.GE24472@ando.pearwood.info> <DA710CDC-41FF-4097-B58A-83B711B8F8B7@yahoo.com> <20141231231419.GG24472@ando.pearwood.info> <CANXboVbr-HVfA2adk7Vpog-63EeAtyP9W=mvnN2THbpyyM-CmQ@mail.gmail.com> <20150101123526.GC15262@ando.pearwood.info> Message-ID: <CANXboVa_goTaMngPTswkeVe+cmCDWuj4+BqZSQ96wXkXiCo3nQ@mail.gmail.com> I take your point about it not being in the `os` module, I have no issue with it being in shutil or pathlib. I also agree about the name, "recycle" doesn't really make sense. Regarding your question: "But I'm still not sure that this needs to be in the standard library. For such a specialist need, what's wrong with using a third party solution?" Well, I can use a third-party solution for everything, but it would be nicer for me if it was in the standard library because then I could count on it always being there and the API not changing. I could say the same thing you said about the `webbrowser` module, that opening a new browser tab is quite a specialist need, but I do enjoy having it available in the standard library and I feel the same about sending a file to trash, which I think is a basic feature that's we've all been taking for granted for the last decade or two. Happy new year, Ram. On Thu, Jan 1, 2015 at 2:35 PM, Steven D'Aprano <steve at pearwood.info> wrote: > On Thu, Jan 01, 2015 at 12:37:55PM +0200, Ram Rachum wrote: > > > So maybe we can add a function to the `os` module that sends a file to > the > > recycle bin, and a constant that will say whether the current OS supports > > this? Then we could have code like this: > > The os module is for low-level operating-system functions. "Send to > trash" is neither low-level nor part of the OS per se, it is part of the > desktop environment. > > I'm not convinced that this needs to be in the standard library, but if > it is, I think that the os module is completely the wrong place for it. > I think, in order of preference: > > 1) shutil > 2) pathlib > > is the right place. > > (The actual implementation for the move_to_trash function could come > from another, platform-specific, module.) > > > > if os.RECYCLE_BIN_SUPPORT: > > os.recycle(my_file) > > else: > > os.remove(my_file) > > > "Recycle" is not a good name for the function, because it doesn't > recycle the file. It does the opposite of recycle: it prevents the file > from being deleted and over-written. I think an explicit name like > move_to_trash is better than something misleading like "recycle". > > No need for a special flag to check if the function exists, just check > for the function: > > try: > f = shutil.send_to_trash > except AttributeError: > f = os.remove > f(my_file) > > > But I'm still not sure that this needs to be in the standard library. > For such a specialist need, what's wrong with using a third party > solution? > > > -- > Steven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150101/87e2b1a6/attachment.html> From liam.marsh.home at gmail.com Thu Jan 1 13:47:54 2015 From: liam.marsh.home at gmail.com (Liam Marsh) Date: Thu, 1 Jan 2015 13:47:54 +0100 Subject: [Python-ideas] pep 397 In-Reply-To: <CAPTjJmoav+k8xa3u2LU84c8_UXMkEerFmMSnhgwkM=g2vtG4AQ@mail.gmail.com> References: <CACPPHzsuATV4Gkn2fCE-P3Tr4frjQ9TuEATU5cg6q3vzR9pS0w@mail.gmail.com> <CAPTjJmoav+k8xa3u2LU84c8_UXMkEerFmMSnhgwkM=g2vtG4AQ@mail.gmail.com> Message-ID: <CACPPHzuMqWfFqOFdRQbqpXMJLr078asqeBQv4k2Fn=S9KWmyEg@mail.gmail.com> "I suspect you don't need any changes to the language to make this work" - you are right, this problem does not affect the language itself, but the idlelib library.... - I was thinking of an "pyidle" executable next to the "py" and "pyw" executables. I am also trying to create a small script to do this... 2014-12-30 23:38 GMT+01:00 Chris Angelico <rosuav at gmail.com>: > On Wed, Dec 31, 2014 at 7:31 AM, Liam Marsh <liam.marsh.home at gmail.com> > wrote: > > the pep 397 says that any python script is able to choose the language > > version which will run it, between all the versions installed on the > > computer, using on windows a launcher in the "C:\windows" folder. > > > > can the idle version be chosen like this too, or can the idle "run" > command > > do it? > > I don't know of a way off-hand. However, if the information is already > there in the shebang (as per PEP 397), it ought to be possible to rig > up a launcher/chooser script. This might be something to discuss on > python-list at python.org, rather than here; there are a lot more people > on python-list than there are here on -ideas, and I suspect you don't > need any changes to the language to make this work. > > It's also entirely possible that someone there already knows exactly > what you need to do! > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150101/20cb9c46/attachment.html> From markus at unterwaditzer.net Thu Jan 1 13:47:15 2015 From: markus at unterwaditzer.net (Markus Unterwaditzer) Date: Thu, 1 Jan 2015 13:47:15 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <20150101123526.GC15262@ando.pearwood.info> References: <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <20141230234702.GE24472@ando.pearwood.info> <DA710CDC-41FF-4097-B58A-83B711B8F8B7@yahoo.com> <20141231231419.GG24472@ando.pearwood.info> <CANXboVbr-HVfA2adk7Vpog-63EeAtyP9W=mvnN2THbpyyM-CmQ@mail.gmail.com> <20150101123526.GC15262@ando.pearwood.info> Message-ID: <20150101124715.GA7009@chromebot.lan> On Thu, Jan 01, 2015 at 11:35:26PM +1100, Steven D'Aprano wrote: > On Thu, Jan 01, 2015 at 12:37:55PM +0200, Ram Rachum wrote: > > > So maybe we can add a function to the `os` module that sends a file to the > > recycle bin, and a constant that will say whether the current OS supports > > this? Then we could have code like this: > > The os module is for low-level operating-system functions. "Send to > trash" is neither low-level nor part of the OS per se, it is part of the > desktop environment. > > I'm not convinced that this needs to be in the standard library, but if > it is, I think that the os module is completely the wrong place for it. > I think, in order of preference: > > 1) shutil > 2) pathlib > > is the right place. Maybe the right way to do this is to create a new module, stdlib or not, for desktop-related APIs. Besides a `recycle` function this might, for example, include a subset of pyxdg's functionality. -- Markus From rosuav at gmail.com Thu Jan 1 14:27:18 2015 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 2 Jan 2015 00:27:18 +1100 Subject: [Python-ideas] pep 397 In-Reply-To: <CACPPHzuMqWfFqOFdRQbqpXMJLr078asqeBQv4k2Fn=S9KWmyEg@mail.gmail.com> References: <CACPPHzsuATV4Gkn2fCE-P3Tr4frjQ9TuEATU5cg6q3vzR9pS0w@mail.gmail.com> <CAPTjJmoav+k8xa3u2LU84c8_UXMkEerFmMSnhgwkM=g2vtG4AQ@mail.gmail.com> <CACPPHzuMqWfFqOFdRQbqpXMJLr078asqeBQv4k2Fn=S9KWmyEg@mail.gmail.com> Message-ID: <CAPTjJmrGwWr7G84S_9maK6Su9GudAHwLF7Omu-v7ha0SVyLkbw@mail.gmail.com> On Thu, Jan 1, 2015 at 11:47 PM, Liam Marsh <liam.marsh.home at gmail.com> wrote: > "I suspect you don't > need any changes to the language to make this work" > > - you are right, this problem does not affect the language itself, but the > idlelib library.... > - I was thinking of an "pyidle" executable next to the "py" and "pyw" > executables. > I am also trying to create a small script to do this... A small script ought to be able to handle it. All it needs to do is: 1) Read in the first line of the file. 2) See if it can be parsed according to PEP 397 3) Invoke IDLE. Since this is primarily for your own personal use, rather than being a part of the language or standard library, you could make a few assumptions - for instance, assume an ASCII-compatible file encoding, assume that it won't get any arguments other than the script to open, anything else that'll make your life easier. Knock something together, and if you have trouble with the details, python-list will undoubtedly be willing to help out. All the best! ChrisA From abarnert at yahoo.com Thu Jan 1 16:37:52 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 1 Jan 2015 16:37:52 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <CANXboVbr-HVfA2adk7Vpog-63EeAtyP9W=mvnN2THbpyyM-CmQ@mail.gmail.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <20141230234702.GE24472@ando.pearwood.info> <DA710CDC-41FF-4097-B58A-83B711B8F8B7@yahoo.com> <20141231231419.GG24472@ando.pearwood.info> <CANXboVbr-HVfA2adk7Vpog-63EeAtyP9W=mvnN2THbpyyM-CmQ@mail.gmail.com> Message-ID: <D35AD2FB-8E6C-47B5-9954-08E110D1E392@yahoo.com> On Jan 1, 2015, at 11:37, Ram Rachum <ram at rachum.com> wrote: > Hi everyone, > > I'm a complete ignoramus when it comes to Mac, so I can't say anything about that, but as I understand the discussion, there are problems with implementing this on Mac. Okay. So maybe this should be available only on Windows and Linux? (If it's not a problem to do on Linux.) On Linux, it's not exactly like Mac, but it's similar. Linux has no notion of trash at all; desktop environments like GNOME do. And if you want it to work exactly like the way GNOME does, you call the API via GObject; otherwise you have to implement a complicated approximation based on rename that won't be fully functional. Really, on all three platforms, the answer is the same: you call an API from your desktop manager. The only difference is that Windows merges desktop and kernel together to the point where they can't be separated--so Python is already using Win32, unlike Cocoa or GObject. > As much as I don't like having functionality that's only available on certain OSs, it makes more sense than depriving Windows users of functionality that they could be using just because Mac doesn't support it. No one is "depriving" Windows users of anything. Python comes with pip, send2trash installs without needing a compiler (at least I hope so; if not, someone can write a new third-party module that does). Not everything needs to be in the stdlib. So far, there aren't many third-party modules mentioned in the stdlib docs, but that's only because we didn't have preinstalled pip, wheels, etc. until 3.4. > (And of course, Python has lots of OS-specific modules.) > > So maybe we can add a function to the `os` module that sends a file to the recycle bin, and a constant that will say whether the current OS supports this? Then we could have code like this: > > if os.RECYCLE_BIN_SUPPORT: > os.recycle(my_file) > else: > os.remove(my_file) > > What do you think? There's no such flag for anything else platform-contingent. EAFP. > > > Thanks, > Ram. > > On Thu, Jan 1, 2015 at 1:14 AM, Steven D'Aprano <steve at pearwood.info> wrote: >> On Wed, Dec 31, 2014 at 11:50:04AM +0100, Andrew Barnert wrote: >> > On Dec 31, 2014, at 0:47, Steven D'Aprano <steve at pearwood.info> wrote: >> >> > > What's wrong with asking the Finder to move the file? Under what >> > > circumstances would that be unacceptable? Are there scenarios on OS X >> > > where the Finder isn't available but the trash is? >> > >> > I already answered this briefly earlier in the thread, but in more detail: >> [...] >> >> Thank you for the detailed and extensive answer! >> >> >> -- >> Steven >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150101/d38e6d4e/attachment.html> From abarnert at yahoo.com Thu Jan 1 16:54:59 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 1 Jan 2015 16:54:59 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <20150101124715.GA7009@chromebot.lan> References: <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <20141230234702.GE24472@ando.pearwood.info> <DA710CDC-41FF-4097-B58A-83B711B8F8B7@yahoo.com> <20141231231419.GG24472@ando.pearwood.info> <CANXboVbr-HVfA2adk7Vpog-63EeAtyP9W=mvnN2THbpyyM-CmQ@mail.gmail.com> <20150101123526.GC15262@ando.pearwood.info> <20150101124715.GA7009@chromebot.lan> Message-ID: <78E3056F-D649-4777-9755-6CBEF5598CA1@yahoo.com> On Jan 1, 2015, at 13:47, Markus Unterwaditzer <markus at unterwaditzer.net> wrote: > On Thu, Jan 01, 2015 at 11:35:26PM +1100, Steven D'Aprano wrote: >> On Thu, Jan 01, 2015 at 12:37:55PM +0200, Ram Rachum wrote: >> >>> So maybe we can add a function to the `os` module that sends a file to the >>> recycle bin, and a constant that will say whether the current OS supports >>> this? Then we could have code like this: >> >> The os module is for low-level operating-system functions. "Send to >> trash" is neither low-level nor part of the OS per se, it is part of the >> desktop environment. >> >> I'm not convinced that this needs to be in the standard library, but if >> it is, I think that the os module is completely the wrong place for it. >> I think, in order of preference: >> >> 1) shutil >> 2) pathlib >> >> is the right place. > > Maybe the right way to do this is to create a new module, stdlib or not, for > desktop-related APIs. Besides a `recycle` function this might, for example, > include a subset of pyxdg's functionality. That's a great idea. To do something as simple as getting special directory names or creating a shortcut/alias/bookmark (as opposed to a symlink) or, yes, trashing a file in a cross-platform way requires something way too heavy-duty (like Qt or JUCE)... or copying code off some blog (that doesn't actually work on localized Windows or in a sandboxed OS X app or on non-Debianish Linux, but you won't know that until you think to test it). A third-party module can get away with 90% support. Most Mac users are on 10.8+, most apps don't fork, most Linux users who care about the desktop have a GNOME-compatible one; etc. > > -- Markus > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From markus at unterwaditzer.net Thu Jan 1 17:00:52 2015 From: markus at unterwaditzer.net (Markus Unterwaditzer) Date: Thu, 1 Jan 2015 17:00:52 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <D35AD2FB-8E6C-47B5-9954-08E110D1E392@yahoo.com> References: <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <20141230234702.GE24472@ando.pearwood.info> <DA710CDC-41FF-4097-B58A-83B711B8F8B7@yahoo.com> <20141231231419.GG24472@ando.pearwood.info> <CANXboVbr-HVfA2adk7Vpog-63EeAtyP9W=mvnN2THbpyyM-CmQ@mail.gmail.com> <D35AD2FB-8E6C-47B5-9954-08E110D1E392@yahoo.com> Message-ID: <20150101160052.GA8524@chromebot.lan> On Thu, Jan 01, 2015 at 04:37:52PM +0100, Andrew Barnert wrote: > On Jan 1, 2015, at 11:37, Ram Rachum <ram at rachum.com> wrote: > > > Hi everyone, > > > > I'm a complete ignoramus when it comes to Mac, so I can't say anything > > about that, but as I understand the discussion, there are problems with > > implementing this on Mac. Okay. So maybe this should be available only on > > Windows and Linux? (If it's not a problem to do on Linux.) > > On Linux, it's not exactly like Mac, but it's similar. Linux has no notion of > trash at all; desktop environments like GNOME do. And if you want it to work > exactly like the way GNOME does, you call the API via GObject; otherwise you > have to implement a complicated approximation based on rename that won't be > fully functional. Actually it seems that the trash location is standardized by the XDG Desktop Specification: https://github.com/kevva/xdg-trash http://www.ramendik.ru/docs/trashspec.html From abarnert at yahoo.com Thu Jan 1 23:59:59 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 1 Jan 2015 23:59:59 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <20150101160052.GA8524@chromebot.lan> References: <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <20141230234702.GE24472@ando.pearwood.info> <DA710CDC-41FF-4097-B58A-83B711B8F8B7@yahoo.com> <20141231231419.GG24472@ando.pearwood.info> <CANXboVbr-HVfA2adk7Vpog-63EeAtyP9W=mvnN2THbpyyM-CmQ@mail.gmail.com> <D35AD2FB-8E6C-47B5-9954-08E110D1E392@yahoo.com> <20150101160052.GA8524@chromebot.lan> Message-ID: <BE0BA8DD-3E25-47C9-A0BD-868F9EEA8983@yahoo.com> On Jan 1, 2015, at 17:00, Markus Unterwaditzer <markus at unterwaditzer.net> wrote: > On Thu, Jan 01, 2015 at 04:37:52PM +0100, Andrew Barnert wrote: >> On Jan 1, 2015, at 11:37, Ram Rachum <ram at rachum.com> wrote: >> >>> Hi everyone, >>> >>> I'm a complete ignoramus when it comes to Mac, so I can't say anything >>> about that, but as I understand the discussion, there are problems with >>> implementing this on Mac. Okay. So maybe this should be available only on >>> Windows and Linux? (If it's not a problem to do on Linux.) >> >> On Linux, it's not exactly like Mac, but it's similar. Linux has no notion of >> trash at all; desktop environments like GNOME do. And if you want it to work >> exactly like the way GNOME does, you call the API via GObject; otherwise you >> have to implement a complicated approximation based on rename that won't be >> fully functional. > > Actually it seems that the trash location is standardized by the XDG Desktop > Specification: > > https://github.com/kevva/xdg-trash > http://www.ramendik.ru/docs/trashspec.html Yes, as I just said: Linux has no notion of trash; desktop environments do. The FreeDesktop standard is not Linux; it's a standard that at least two desktop environments (which also work on multiple non-Linux systems) support. On most Linux systems, you will have Gtk libraries; on most Linux systems without Gtk libraries, you will not have a FreeDesktop-compliant trash. So, using GObject is a good 90% solution (like using NSFileManager is a good 90% solution on Mac). Of course you _could_ implement that full spec yourself in Python, right? Well, read it. There are references to things "the implementation" may do, like providing a way for the administrator to disable sticky-bit checking for filesystems that don't support it. Do you know how to check whether the administrator has disabled sticky-bit checking on a filesystem? Or do you just want your code to say a file can't be trashed, or trash by moving it across the network to ~, even when the desktop and other apps have no problem trashing it properly? For that matter, do you know how every FreeDesktop-compatible system wants to report a problem to the administrator? You may meet the XDG Trash spec and still not satisfy user expectations. And it'll be a lot of work. In other words, "you have to implement a complicated approximation based on rename that won't be fully functional". Is that really a better solution than using GObject? If you're writing your own portability library like Qt or JUCE, probably; otherwise, I doubt it. From steve at pearwood.info Fri Jan 2 01:09:14 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 2 Jan 2015 11:09:14 +1100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <CANXboVa_goTaMngPTswkeVe+cmCDWuj4+BqZSQ96wXkXiCo3nQ@mail.gmail.com> References: <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <20141230234702.GE24472@ando.pearwood.info> <DA710CDC-41FF-4097-B58A-83B711B8F8B7@yahoo.com> <20141231231419.GG24472@ando.pearwood.info> <CANXboVbr-HVfA2adk7Vpog-63EeAtyP9W=mvnN2THbpyyM-CmQ@mail.gmail.com> <20150101123526.GC15262@ando.pearwood.info> <CANXboVa_goTaMngPTswkeVe+cmCDWuj4+BqZSQ96wXkXiCo3nQ@mail.gmail.com> Message-ID: <20150102000913.GD15262@ando.pearwood.info> On Thu, Jan 01, 2015 at 02:43:09PM +0200, Ram Rachum wrote: > Regarding your question: "But I'm still not sure that this needs to be in > the standard library. For such a specialist need, what's wrong with using a > third party solution?" > > Well, I can use a third-party solution for everything, but it would be > nicer for me if it was in the standard library because then I could count > on it always being there and the API not changing. Which is exactly the point. It has been said that the standard library is where good modules go to die. Do you think that the "move to trash" API is ready to die? I don't think so. I think that there is a lot more than just moving files to the trash. I've spent a little bit of time over the last day or so thinking about what sort of trash-related functionality I would like, as a Linux user: - move to trash, of course, but which trash? - return the name of the trash directory in my home directory - list trash directories on other partitions and disks - list trash directories for all users (if I have read permission) - empty one or more of those trash directories - optionally only deleting sufficiently old and/or large files - validate and/or repair the trash metadata - report how much space is used by the trash, with or without metadata - support legacy .Trash directories as well as .local/share/Trash For some operations, I would want to specify either a specific trash directory, or all trash directories for a given user (defaulting to the current user), or all trash directories for all users. (The later would only work if I was running as the root/Administrator user, of course.) The point being that the API for all this functionality isn't obvious, and it is very unlikely to be so stable that it is ready to move into the standard library. -- Steven From random832 at fastmail.us Fri Jan 2 14:24:26 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Fri, 02 Jan 2015 08:24:26 -0500 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> Message-ID: <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> On Tue, Dec 30, 2014, at 18:12, Andrew Barnert wrote: > By "suitable" I mean "suitable for widespread-enough use that it should > be in the stdlib". As I said earlier in the thread, I think it's > perfectly reasonable to have a third-party library that uses a deprecated > API, requires 10.8, requires a runloop, directly or indirectly asks the > Finder to do the work for you, or doesn't quite meet Apple's HIG rules, > because all of those could be acceptable in _many_ applications; I just > don't think any of those should be in the stdlib, because none of those > are acceptable in _most_ applications. Not having one isn't acceptable either. Because what _not_ having a cross-platform wrapper gets you is windows and mac left behind by applications with badly-written direct implementations of the way you do it on Unix. Can you define "doesn't quite meet Apple's HIG" rules and link the relevant section of the HIG? From random832 at fastmail.us Fri Jan 2 14:30:43 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Fri, 02 Jan 2015 08:30:43 -0500 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <20150101123526.GC15262@ando.pearwood.info> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <20141230234702.GE24472@ando.pearwood.info> <DA710CDC-41FF-4097-B58A-83B711B8F8B7@yahoo.com> <20141231231419.GG24472@ando.pearwood.info> <CANXboVbr-HVfA2adk7Vpog-63EeAtyP9W=mvnN2THbpyyM-CmQ@mail.gmail.com> <20150101123526.GC15262@ando.pearwood.info> Message-ID: <1420205443.2838218.208808325.450952DE@webmail.messagingengine.com> On Thu, Jan 1, 2015, at 07:35, Steven D'Aprano wrote: > The os module is for low-level operating-system functions. "Send to > trash" is neither low-level nor part of the OS per se, it is part of the > desktop environment. What is the correct place for high-level functions that need to be implemented in a different way on each OS? I think the problem is that some people view os as "low-level functions, like the ones found in unistd.h" and other people view it as "where to put stuff that needs a separate implementation per platform". The actual description in the documentation is "This module provides a portable way of using operating system dependent functionality." Nothing about low-level. From random832 at fastmail.us Fri Jan 2 14:36:24 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Fri, 02 Jan 2015 08:36:24 -0500 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <D35AD2FB-8E6C-47B5-9954-08E110D1E392@yahoo.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <20141230234702.GE24472@ando.pearwood.info> <DA710CDC-41FF-4097-B58A-83B711B8F8B7@yahoo.com> <20141231231419.GG24472@ando.pearwood.info> <CANXboVbr-HVfA2adk7Vpog-63EeAtyP9W=mvnN2THbpyyM-CmQ@mail.gmail.com> <D35AD2FB-8E6C-47B5-9954-08E110D1E392@yahoo.com> Message-ID: <1420205784.2839657.208808873.1C78CB92@webmail.messagingengine.com> On Thu, Jan 1, 2015, at 10:37, Andrew Barnert wrote: > On Linux, it's not exactly like Mac, but it's similar. Linux has no > notion of trash at all; desktop environments like GNOME do. And if you > want it to work exactly like the way GNOME does, you call the API via > GObject; otherwise you have to implement a complicated approximation > based on rename that won't be fully functional. There is an actual standard for the Trash system that all three major desktop environments implement, so your "complicated approximation based on rename" can actually be correct rather than an approximation. If that's not exactly like how GNOME does it, then GNOME is incorrect. From random832 at fastmail.us Fri Jan 2 14:43:44 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Fri, 02 Jan 2015 08:43:44 -0500 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <BE0BA8DD-3E25-47C9-A0BD-868F9EEA8983@yahoo.com> References: <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <20141230234702.GE24472@ando.pearwood.info> <DA710CDC-41FF-4097-B58A-83B711B8F8B7@yahoo.com> <20141231231419.GG24472@ando.pearwood.info> <CANXboVbr-HVfA2adk7Vpog-63EeAtyP9W=mvnN2THbpyyM-CmQ@mail.gmail.com> <D35AD2FB-8E6C-47B5-9954-08E110D1E392@yahoo.com> <20150101160052.GA8524@chromebot.lan> <BE0BA8DD-3E25-47C9-A0BD-868F9EEA8983@yahoo.com> Message-ID: <1420206224.2842545.208810201.5532DB2B@webmail.messagingengine.com> On Thu, Jan 1, 2015, at 17:59, Andrew Barnert wrote: > Of course you _could_ implement that full spec yourself in Python, right? > Well, read it. There are references to things "the implementation" may > do, like providing a way for the administrator to disable sticky-bit > checking for filesystems that don't support it. Do you know how to check > whether the administrator has disabled sticky-bit checking on a > filesystem? A python implementation would be an independent implementation, and not bound by another implementation's (even on the same machine) implementation-specific decisions. There is therefore no reason the python implementation couldn't simply not provide such a method, and therefore any .Trash directory without the sticky bit set does not pass the checks, and all that will happen is .Trash-$uid gets used instead - no harm done. From ncoghlan at gmail.com Fri Jan 2 19:40:52 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Jan 2015 04:40:52 +1000 Subject: [Python-ideas] python on mobile In-Reply-To: <CAJxq848T09_c-jrHhtyyWUBFk7By7xtUphCExom7wNd36ZCtow@mail.gmail.com> References: <CADjSo4TssZHGiOCm110co-BQxLAXZpCZbT1T=zdd8eBavnyVzQ@mail.gmail.com> <CAJxq848RyRhbeD4kHssFo1jHUtVX5N1_G=RB1rB+DkmDaxxA2A@mail.gmail.com> <CADiSq7enX0nmNd2AEHGwCgdDsgaXWzdz0A1w5iU_m1V+JdHe6g@mail.gmail.com> <CAJxq84-EeN0v49sHpPYZStJJ_615quqMYmmq2OPvhcaae6Hfiw@mail.gmail.com> <CADjSo4Seed-B0FmTUikrtvDux_LprdvedVrR1-RvQS8fPE=LiQ@mail.gmail.com> <87k31833vw.fsf@uwakimon.sk.tsukuba.ac.jp> <CAJxq848T09_c-jrHhtyyWUBFk7By7xtUphCExom7wNd36ZCtow@mail.gmail.com> Message-ID: <CADiSq7ewkZwRZhkeyuJC32KeaaPuq9FTxQXHiY5uhOZBA+Un=g@mail.gmail.com> On 31 December 2014 at 13:42, Russell Keith-Magee <russell at keith-magee.com> wrote: > > > On Wed, Dec 31, 2014 at 11:16 AM, Stephen J. Turnbull <stephen at xemacs.org> > wrote: >> >> Fetchinson . writes: >> >> > > I'm not sure exactly what would be discussed [on mobile-sig], >> > > especially in the short term when the big ticket item is getting >> > > the patches into the main Python tree. >> >> It probably matters more that there *is* substantive discussion than >> *what* is discussed. There are probably a lot of people who are far >> more interested in "how to do 'it' *now*" than "get 'it' into the main >> Python tree 'someday'", for example. >> > > I'm not sure if a mailing list would help with this as much as a good, > simple set of docs. It's not like there's unknown territory here; once > you've got Python installed on a mobile device, the "how" is reasonably > easy. The catch is that someone with Python experience won't necessarily > have experience managing a cross-platform C/Automake build, which is what > you need to get Python installed in the first place. > One of the benefits of a SIG over python-ideas is that their remit can be broader than just CPython. distutils-sig is probably the most expansive on that front, but As far as setting one up goes, I believe the page at https://www.python.org/community/sigs/guidelines/ is still accurate (although the blurb isn't quite right when it comes to long lived topical sigs like distutils-sig, or intermittently active sigs like import-sig) Although it's a couple of months off, I would suggest a talk (if it's >> still possible to schedule), a BOF, and a sprint at Pycon North >> America in March. That should get things moving, and several of the >> "important" folks who have already expressed interest should be there, >> too. >> > > Unfortunately, due to the tyranny of distance (I'm based on the west coast > of Australia), plus work commitments, it's unlikely that I'll be attending > PyCon in Montreal. > Will you be at LCA this year? At the very least we can continue the various discussions from PyCon AU :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150103/1e84b667/attachment-0001.html> From abarnert at yahoo.com Fri Jan 2 20:46:51 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 2 Jan 2015 20:46:51 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> Message-ID: <11C6719F-3F28-4311-966E-CFEA6CD10467@yahoo.com> On Jan 2, 2015, at 14:24, random832 at fastmail.us wrote: > On Tue, Dec 30, 2014, at 18:12, Andrew Barnert wrote: >> By "suitable" I mean "suitable for widespread-enough use that it should >> be in the stdlib". As I said earlier in the thread, I think it's >> perfectly reasonable to have a third-party library that uses a deprecated >> API, requires 10.8, requires a runloop, directly or indirectly asks the >> Finder to do the work for you, or doesn't quite meet Apple's HIG rules, >> because all of those could be acceptable in _many_ applications; I just >> don't think any of those should be in the stdlib, because none of those >> are acceptable in _most_ applications. > > Not having one isn't acceptable either. Because what _not_ having a > cross-platform wrapper gets you is windows and mac left behind by > applications with badly-written direct implementations of the way you do > it on Unix. I'm not sure what "the way you do it on Unix" means. There is no Unix/POSIX/etc. notion of trashing a file. Of course some Unix desktop environments (going back at least as far as NeXTStep) have had such a thing, but that can't be what you mean. If you're referring to XDG, I don't know of a single application that uses a badly-written direct implementation of the XDG trash spec on Mac or Windows. Do you? More importantly, which of those other options do you think is better than continuing to have no trash API? Drop 10.6-10.7 compat? Use a deprecated API and break Mac App Store compatibility? Require all scripts to have a runloop? > Can you define "doesn't quite meet Apple's HIG" rules and link the > relevant section of the HIG? Of course not, because I have no idea what exactly you'd write in place of calling the appropriate API. Give me a design or some code, and I'll tell you what it does wrong. From abarnert at yahoo.com Fri Jan 2 21:01:08 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 2 Jan 2015 21:01:08 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <1420205784.2839657.208808873.1C78CB92@webmail.messagingengine.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <20141230234702.GE24472@ando.pearwood.info> <DA710CDC-41FF-4097-B58A-83B711B8F8B7@yahoo.com> <20141231231419.GG24472@ando.pearwood.info> <CANXboVbr-HVfA2adk7Vpog-63EeAtyP9W=mvnN2THbpyyM-CmQ@mail.gmail.com> <D35AD2FB-8E6C-47B5-9954-08E110D1E392@yahoo.com> <1420205784.2839657.208808873.1C78CB92@webmail.messagingengine.com> Message-ID: <DF006B99-238A-448D-B505-BC526242DDD1@yahoo.com> On Jan 2, 2015, at 14:36, random832 at fastmail.us wrote: > On Thu, Jan 1, 2015, at 10:37, Andrew Barnert wrote: >> On Linux, it's not exactly like Mac, but it's similar. Linux has no >> notion of trash at all; desktop environments like GNOME do. And if you >> want it to work exactly like the way GNOME does, you call the API via >> GObject; otherwise you have to implement a complicated approximation >> based on rename that won't be fully functional. > > There is an actual standard for the Trash system that all three major > desktop environments implement, so your "complicated approximation based > on rename" can actually be correct rather than an approximation. If > that's not exactly like how GNOME does it, then GNOME is incorrect. First, XDG specifications are explicitly not standards (and XDG Trash is a draft extension proposal to those specs) so there is not an actual standard. But, far more importantly, there are large areas that the spec leaves open to the implementation, or open for future expansion, that GNOME (and KDE) fills in with appropriate behavior. I gave one example earlier in the thread, which I picked by pointing my finger at one random paragraph from the spec. If you read through it, there are plenty of others. If you call the APIs via GObject or Qt on a GNOME or KDE system, you will get the exact same behavior as everything else on the system. If you implement it yourself and ignore what GNOME and KDE do, nobody's going to care that you're 100% compliant with a draft spec proposal, they're going to care that your app refuses to trash something that everything else can trash (or, worse, that your app trashes it by copying 8GB over the network). From abarnert at yahoo.com Fri Jan 2 21:05:51 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 2 Jan 2015 21:05:51 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <1420206224.2842545.208810201.5532DB2B@webmail.messagingengine.com> References: <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <20141230234702.GE24472@ando.pearwood.info> <DA710CDC-41FF-4097-B58A-83B711B8F8B7@yahoo.com> <20141231231419.GG24472@ando.pearwood.info> <CANXboVbr-HVfA2adk7Vpog-63EeAtyP9W=mvnN2THbpyyM-CmQ@mail.gmail.com> <D35AD2FB-8E6C-47B5-9954-08E110D1E392@yahoo.com> <20150101160052.GA8524@chromebot.lan> <BE0BA8DD-3E25-47C9-A0BD-868F9EEA8983@yahoo.com> <1420206224.2842545.208810201.5532DB2B@webmail.messagingengine.com> Message-ID: <73AA04B8-4D4A-4626-8381-9A071C78EA5B@yahoo.com> On Jan 2, 2015, at 14:43, random832 at fastmail.us wrote: > On Thu, Jan 1, 2015, at 17:59, Andrew Barnert wrote: >> Of course you _could_ implement that full spec yourself in Python, right? >> Well, read it. There are references to things "the implementation" may >> do, like providing a way for the administrator to disable sticky-bit >> checking for filesystems that don't support it. Do you know how to check >> whether the administrator has disabled sticky-bit checking on a >> filesystem? > > A python implementation would be an independent implementation, and not > bound by another implementation's (even on the same machine) > implementation-specific decisions. Then a Python implementation would be a bad implementation. The whole point of XDG is to make it easier to write desktop software that's consistent with other desktop software. So if you write some Qt software and I run it on a GNOME system, Qt follows (as best it can--which is better each version) the GNOME rules, and your GNOME-based settings, rather than ignoring them and pretending it's on a KDE system. > There is therefore no reason the > python implementation couldn't simply not provide such a method, and > therefore any .Trash directory without the sticky bit set does not pass > the checks, and all that will happen is .Trash-$uid gets used instead - > no harm done. Except that if you can't create a .Trash-$uid, then you can't trash files on that filesystem even though Nautilus, etc. can. (And, needless to say, if you try to implement any other trash functionality, like listing trash contents or undeleting, you're going to be wrong there too.) From ncoghlan at gmail.com Sat Jan 3 02:36:01 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Jan 2015 11:36:01 +1000 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <73AA04B8-4D4A-4626-8381-9A071C78EA5B@yahoo.com> References: <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <20141230234702.GE24472@ando.pearwood.info> <DA710CDC-41FF-4097-B58A-83B711B8F8B7@yahoo.com> <20141231231419.GG24472@ando.pearwood.info> <CANXboVbr-HVfA2adk7Vpog-63EeAtyP9W=mvnN2THbpyyM-CmQ@mail.gmail.com> <D35AD2FB-8E6C-47B5-9954-08E110D1E392@yahoo.com> <20150101160052.GA8524@chromebot.lan> <BE0BA8DD-3E25-47C9-A0BD-868F9EEA8983@yahoo.com> <1420206224.2842545.208810201.5532DB2B@webmail.messagingengine.com> <73AA04B8-4D4A-4626-8381-9A071C78EA5B@yahoo.com> Message-ID: <CADiSq7dME4OP7VXoC6rZEWOvHgLmj1H52zzeoOqvEniWBrV_3g@mail.gmail.com> On 3 Jan 2015 06:06, "Andrew Barnert" <abarnert at yahoo.com.dmarc.invalid> wrote: > > Except that if you can't create a .Trash-$uid, then you can't trash files on that filesystem even though Nautilus, etc. can. (And, needless to say, if you try to implement any other trash functionality, like listing trash contents or undeleting, you're going to be wrong there too.) It sounds to me that what is needed before changes to pathlib can be discussed is a reference third party module that appropriately respects desktop conventions (whether Windows, Mac OS X, GNOME, KDE, or other XDG based Linux window manager), and associated clear documentation on how to delete files in a way that integrates well with underlying platforms. The standard library isn't the place to work through that standardisation activity, although a PEP is certainly a reasonable place to track it. Guido's work on PEP 3156 and the asnycio module is a good point of reference here, as is Antoine's work on pathlib itself. Once a pathlib independent reference implementation of imperative desktop trash management exists, and there is broad acknowledgement that it is a suitable candidate for inclusion in the standard library (perhaps as part of shutil, perhaps as a new independent module, depending on API complexity), *then* it becomes feasible to discuss adding trash management support to the pathlib abstraction layer. Regards, Nick. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150103/f6ae6eb8/attachment-0001.html> From steve at pearwood.info Sat Jan 3 02:50:45 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 3 Jan 2015 12:50:45 +1100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> Message-ID: <20150103015045.GB27426@ando.pearwood.info> On Fri, Jan 02, 2015 at 08:24:26AM -0500, random832 at fastmail.us wrote: > Not having one [move to trash function] isn't acceptable either. I'm not sure about that. Do other languages offer a move-to-trash functionality? Say, PHP, Javascript, Java, Ruby, Boo, Go, Lua? If Python is the odd one out that DOESN'T offer such functionality, and all the others do, then you might have a strong case. Otherwise, consider that the great bulk of scripts that are written in Python don't do any file management at all, other than perhaps reading or writing a few files. So even if Python gained this functionality, it wouldn't have any impact on most scripts and applications.[1] > Because what _not_ having a > cross-platform wrapper gets you is windows and mac left behind Really? I would expect that Windows is probably the *easiest* platform to implement this functionality, as it has a single, standard, stable API for moving files to trash. (Or so I am lead to believe.) You can probably use that API via ctypes, or via pywin32. Getting OS X right is very complicated, as Andrew has so ably explained. In Linux and other Unixes, the situation is like OS X, only more so since there's no one single authority. You have a choice of desktop environments, which may or may not provide a move-to-trash API, including no desktop environment at all. Gnome provides an API for moving to trash, but I don't know how well it supports the freedesktop standard; KDE supports the freedesktop standard, but I don't know if it provides an API that can be called. XFCE has partial support. It seems to me that "move to trash" is at least straight forward on Windows, but tricky and complicated everywhere else. Or, there is always the option of using a third-party module: https://pypi.python.org/pypi/Send2Trash [1] With one possible exception. If Andrew is correct, then the wrong implementation of move-to-trash might effect in a bad way anyone trying to put their app on the Apple Store, whether they use that move-to-trash functionality or not. -- Steve From rosuav at gmail.com Sat Jan 3 03:18:24 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 3 Jan 2015 13:18:24 +1100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <20150103015045.GB27426@ando.pearwood.info> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> Message-ID: <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> On Sat, Jan 3, 2015 at 12:50 PM, Steven D'Aprano <steve at pearwood.info> wrote: > Otherwise, consider that the great bulk of scripts that are written in > Python don't do any file management at all, other than perhaps reading > or writing a few files. So even if Python gained this functionality, it > wouldn't have any impact on most scripts and applications.[1] I've always thought of a trash can as an inherently human-controlled feature. When you go in using a GUI file manager and delete a file, it lands in trash. When you use another application, it's always felt surprising - for instance, BitTorrent on Windows deletes things by moving them to trash. (Which has abysmal performance if you select a huge bunch of things and wipe them all out at once; trashing gobs of stuff across thousands of files seems to trigger an O(N*N) repeated search for which files "fall out" of trash - not to mention that this probably means a fairly arbitrary subset of the freshly-deleted is in trash, and little or nothing from previous deletions remains.) Most apps should do file management at the file system level, unless they're specifically replicating/replacing the file manager. So a "move to trash" feature is something that (a) isn't going to be needed by most apps, and especially (b) would be an attractive nuisance. A third-party module has that extra little hump of having to be consciously sought, so it can be there for those who want it but not waving itself around at those who really should be looking at os.remove(). I'm -0.5 on adding a "move to trash" function to the stdlib, ever, and a definite -1 on adding one now. ChrisA From breamoreboy at yahoo.co.uk Sat Jan 3 09:07:47 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Sat, 03 Jan 2015 08:07:47 +0000 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> Message-ID: <m8880n$mdo$1@ger.gmane.org> On 03/01/2015 02:18, Chris Angelico wrote: > On Sat, Jan 3, 2015 at 12:50 PM, Steven D'Aprano <steve at pearwood.info> wrote: >> Otherwise, consider that the great bulk of scripts that are written in >> Python don't do any file management at all, other than perhaps reading >> or writing a few files. So even if Python gained this functionality, it >> wouldn't have any impact on most scripts and applications.[1] > > I've always thought of a trash can as an inherently human-controlled > feature. As, for example, holding down the SHIFT key on Windows and skipping the recycle bin completely? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From rosuav at gmail.com Sat Jan 3 09:10:55 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 3 Jan 2015 19:10:55 +1100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <m8880n$mdo$1@ger.gmane.org> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <m8880n$mdo$1@ger.gmane.org> Message-ID: <CAPTjJmrx=Rr4Pv7-yKggiMtihFj5fHGAZ5txdW+hb7KFpbjGpQ@mail.gmail.com> On Sat, Jan 3, 2015 at 7:07 PM, Mark Lawrence <breamoreboy at yahoo.co.uk> wrote: > On 03/01/2015 02:18, Chris Angelico wrote: >> >> On Sat, Jan 3, 2015 at 12:50 PM, Steven D'Aprano <steve at pearwood.info> >> wrote: >>> >>> Otherwise, consider that the great bulk of scripts that are written in >>> Python don't do any file management at all, other than perhaps reading >>> or writing a few files. So even if Python gained this functionality, it >>> wouldn't have any impact on most scripts and applications.[1] >> >> >> I've always thought of a trash can as an inherently human-controlled >> feature. > > As, for example, holding down the SHIFT key on Windows and skipping the > recycle bin completely? That's a human deciding _not_ to use it; but yes, it should be under human control. ChrisA From abarnert at yahoo.com Sat Jan 3 11:28:52 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 3 Jan 2015 11:28:52 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> Message-ID: <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> On Jan 3, 2015, at 3:18, Chris Angelico <rosuav at gmail.com> wrote: > On Sat, Jan 3, 2015 at 12:50 PM, Steven D'Aprano <steve at pearwood.info> wrote: >> Otherwise, consider that the great bulk of scripts that are written in >> Python don't do any file management at all, other than perhaps reading >> or writing a few files. So even if Python gained this functionality, it >> wouldn't have any impact on most scripts and applications.[1] > > I've always thought of a trash can as an inherently human-controlled > feature. I think this may be an accident of the fact that cmd.exe didn't include a trash command, neither did OS X, and the idea was dropped from xdg-trash; we've all been trained for a couple decades now that you can't trash from the command line. I'm not sure that's a good thing. But it is nevertheless a thing that can't be ignored. > When you go in using a GUI file manager and delete a file, it > lands in trash. When you use another application, it's always felt > surprising In some GUI applications, it does make sense to trash things via a menu or other user interaction. (Except maybe when you're using Apple's versions/history thingy or some similar mechanism.) One of the things Aquamacs adds to the standard emacs distribution is a move-to-trash command that you can use in diredit mode, for example. Although I guess you could argue that counts as "unless you're replacing the file manager", and maybe the same is true of most other such examples... But not all. When I worked on TuneUp, we had a feature to move selected duplicate tracks from quarantine to trash, which I thought was stupid and unnecessary (the quarantine folder is already effectively an app-level trash can...). But when we removed it, we got hundreds of user complaints, and a nasty blog post saying we were trying to deliberately destroy out users' music collections on behalf of the RIAA or something. And: > for instance, BitTorrent on Windows deletes things by > moving them to trash. (Which has abysmal performance if you select a > huge bunch of things and wipe them all out at once; trashing gobs of > stuff across thousands of files seems to trigger an O(N*N) repeated > search for which files "fall out" of trash - not to mention that this > probably means a fairly arbitrary subset of the freshly-deleted is in > trash, and little or nothing from previous deletions remains.) The Mac version also does the move-to-trash; it has perfectly acceptable performance. (Actually, old versions were terribly slow at trashing a torrent with a lot of top-level files or dirs, but newer versions seem to do a single operation per torrent instead, which works fine.) And I'm actually glad it does this. At least twice, I've downloaded an ISO, mounted it, then trashed the torrent with all files; if they were deleted instead of trashed, I would have had to repeat the download to update my VMs... You could argue that I'm an idiot for doing this, and you'd be right, but I'm probably not the only such idiot; in fact, that's kind of what trash is for in the first place. > Most > apps should do file management at the file system level, unless > they're specifically replicating/replacing the file manager. So a > "move to trash" feature is something that (a) isn't going to be needed > by most apps, and especially (b) would be an attractive nuisance. A > third-party module has that extra little hump of having to be > consciously sought, so it can be there for those who want it but not > waving itself around at those who really should be looking at > os.remove(). I agree with this, but it still might be nice to mention that third-party module in shutil and/or pathlib. Whether that's send2trash or a complete trash management module as Steven (I think?) envisioned or a reference to the platform APIs and the various ways to get at them, it would be useful. (In fact, some prominent reference to PyWin32, PyObjC, GObject, and PyQt/PySide for non-GUI-but-still-desktop apps in the docs might be helpful for lots of reasons.) Honestly, I think one of the big things people want is the ability to write the "trash" tool for use on the command line that all of the major desktop systems are missing. It feels like it should be a 3-line Python script, and the fact that it isn't possible looks like a weakness. But really, I don't think that's a problem. OS X comes with a Python with PyObjC; a GNOME or KDE system can install python-gobject or PySide in one apt-get/urpmi/port/etc. command; Windows' function is accessible via the built-in ctypes. I have such scripts on my Mac and Fedora boxes; the fact that they're _different_ 3-line scripts is hardly a serious problem. From rosuav at gmail.com Sat Jan 3 11:39:46 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 3 Jan 2015 21:39:46 +1100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> Message-ID: <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> On Sat, Jan 3, 2015 at 9:28 PM, Andrew Barnert <abarnert at yahoo.com> wrote: > The Mac version also does the move-to-trash; it has perfectly acceptable performance. > Good. The note about performance was a mere parenthesis, and the last time I did anything like that on Windows was on XP, so it's quite possibly different on Win7/8. > I agree with this, but it still might be nice to mention that third-party module in shutil and/or pathlib. > Maybe. Is there a mention of other third-party packages in stdlib docs? It's a slippery slope; which ones deserve mention? Maybe just something like "Note that moving objects to the trash can is possible in OS-dependent ways; packages are available on PyPI to make this easier", without naming any. > Honestly, I think one of the big things people want is the ability to write the "trash" tool for use on the command line that all of the major desktop systems are missing. It feels like it should be a 3-line Python script, and the fact that it isn't possible looks like a weakness. But really, I don't think that's a problem. OS X comes with a Python with PyObjC; a GNOME or KDE system can install python-gobject or PySide in one apt-get/urpmi/port/etc. command; Windows' function is accessible via the built-in ctypes. I have such scripts on my Mac and Fedora boxes; the fact that they're _different_ 3-line scripts is hardly a serious problem. > As an explicit "trash" tool, sure. Typing "trash some_file" is under human control just as much as hitting Delete without holding Shift is. I just don't think it should be the obvious way for an application to remove files. ChrisA From abarnert at yahoo.com Sat Jan 3 12:04:24 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 3 Jan 2015 12:04:24 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> Message-ID: <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> On Jan 3, 2015, at 11:39, Chris Angelico <rosuav at gmail.com> wrote: > On Sat, Jan 3, 2015 at 9:28 PM, Andrew Barnert <abarnert at yahoo.com> wrote: >> I agree with this, but it still might be nice to mention that third-party module in shutil and/or pathlib. > > Maybe. Is there a mention of other third-party packages in stdlib > docs? It's a slippery slope; which ones deserve mention? Maybe just > something like "Note that moving objects to the trash can is possible > in OS-dependent ways; packages are available on PyPI to make this > easier", without naming any. The impression I get is that this is a slope that Python (or at least Nick Coghlan) wants to start sliding down--with the advent of wheels and pre-installed pip, keeping things out of the stdlib but still accessible for most users is now possible, and often desirable. But I could be putting words (or complete ideas) in people's mouths. From tomirendo at gmail.com Sat Jan 3 14:37:03 2015 From: tomirendo at gmail.com (yotam vaknin) Date: Sat, 3 Jan 2015 15:37:03 +0200 Subject: [Python-ideas] Slicing and Chainging iterables. Message-ID: <CANMcLn2d2S=8WSZ3L7kfYk0JaU=3wFg93n=D7Pr1kwh_jK-XGg@mail.gmail.com> I would like to purpose that in python 3 it will be easy to chain and slice Iterators just like you can add and slice lists and tuples easily. I think there could be 2 methods to do this: 1. Implementing '+' and slicing ([1:2:3]) for iterators and generators by default, Resulting in Itertool's chain(a,b) and islice(a,b,c) respectively. 2. Having itertool's chain and islice imported by default. I think since python 3 return zip,map, dict.items and so on as iterators, it makes working with those kind of objects more difficult without having these methods around. And since those methods are important enough to have for lists, it seems important enough for iterators too. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150103/1cbd2ac1/attachment.html> From steve at pearwood.info Sat Jan 3 16:28:44 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 4 Jan 2015 02:28:44 +1100 Subject: [Python-ideas] Slicing and Chainging iterables. In-Reply-To: <CANMcLn2d2S=8WSZ3L7kfYk0JaU=3wFg93n=D7Pr1kwh_jK-XGg@mail.gmail.com> References: <CANMcLn2d2S=8WSZ3L7kfYk0JaU=3wFg93n=D7Pr1kwh_jK-XGg@mail.gmail.com> Message-ID: <20150103152844.GG27426@ando.pearwood.info> On Sat, Jan 03, 2015 at 03:37:03PM +0200, yotam vaknin wrote: > I would like to purpose that in python 3 it will be easy to chain and slice > Iterators just like you can add and slice lists and tuples easily. > > I think there could be 2 methods to do this: > 1. Implementing '+' and slicing ([1:2:3]) for iterators and generators by > default, Resulting in Itertool's chain(a,b) and islice(a,b,c) respectively. > 2. Having itertool's chain and islice imported by default. The iterator protocol is intentionally simple. To create an iterator, you need to only define two methods: (1) __iter__, which returns the instance itself; (2) __next__, which returns the next item and raises StopIterator when done. With your proposal, you would have to define two or three more methods: (3) __add__ and possibly __radd__, to chain iterators; (4) __getitem__, for slicing. There is nothing stopping you from adding these methods to your own iterator classes, but with your proposal that would be compulsory for all iterators. > I think since python 3 return zip,map, dict.items and so on as iterators, > it makes working with those kind of objects more difficult without having > these methods around. And since those methods are important enough to have > for lists, it seems important enough for iterators too. It only takes a single line of code to get iterator chaining and slicing: from itertools import chain, islice And now you can chain and slice any iterator, regardless of where it came from. I don't think that is difficult. *If* iterators where instances of a concrete base class, then it would make sense to add chain and slice methods to that base class so all other iterators could inherit from it. But they aren't, iterators in Python are values which obey a protocol, not inheritence. That means that a functional approach, like itertools, is more appropriate. I can see the appeal of using + and [a:b:c] syntax instead of function syntax chain() and islice(), but I don't think the advantage is enough to out-weigh the costs. -- Steve From tomirendo at gmail.com Sat Jan 3 16:53:12 2015 From: tomirendo at gmail.com (yotam vaknin) Date: Sat, 3 Jan 2015 17:53:12 +0200 Subject: [Python-ideas] Slicing and Chainging iterables. In-Reply-To: <20150103152844.GG27426@ando.pearwood.info> References: <CANMcLn2d2S=8WSZ3L7kfYk0JaU=3wFg93n=D7Pr1kwh_jK-XGg@mail.gmail.com> <20150103152844.GG27426@ando.pearwood.info> Message-ID: <CANMcLn2Qnh2bRG5mcqcrg-VuOYDY8Vp65ENK9WkT=py4aix8Lg@mail.gmail.com> Sorry, I probably wasn't clear enough. My idea was to add these method (__add__, __getitem__) to the already available iterators (map,zip,dict.items) and generators by default. Not to make these methods part of the iterator protocol. 2015-01-03 17:28 GMT+02:00 Steven D'Aprano <steve at pearwood.info>: > On Sat, Jan 03, 2015 at 03:37:03PM +0200, yotam vaknin wrote: > > I would like to purpose that in python 3 it will be easy to chain and > slice > > Iterators just like you can add and slice lists and tuples easily. > > > > I think there could be 2 methods to do this: > > 1. Implementing '+' and slicing ([1:2:3]) for iterators and generators by > > default, Resulting in Itertool's chain(a,b) and islice(a,b,c) > respectively. > > 2. Having itertool's chain and islice imported by default. > > The iterator protocol is intentionally simple. To create an iterator, > you need to only define two methods: > > (1) __iter__, which returns the instance itself; > > (2) __next__, which returns the next item and raises StopIterator when > done. > > > With your proposal, you would have to define two or three more methods: > > (3) __add__ and possibly __radd__, to chain iterators; > > (4) __getitem__, for slicing. > > There is nothing stopping you from adding these methods to your own > iterator classes, but with your proposal that would be compulsory for > all iterators. > > > > I think since python 3 return zip,map, dict.items and so on as iterators, > > it makes working with those kind of objects more difficult without having > > these methods around. And since those methods are important enough to > have > > for lists, it seems important enough for iterators too. > > It only takes a single line of code to get iterator chaining and > slicing: > > from itertools import chain, islice > > And now you can chain and slice any iterator, regardless of where it > came from. I don't think that is difficult. > > *If* iterators where instances of a concrete base class, then it would > make sense to add chain and slice methods to that base class so all > other iterators could inherit from it. But they aren't, iterators in > Python are values which obey a protocol, not inheritence. That means > that a functional approach, like itertools, is more appropriate. > > I can see the appeal of using + and [a:b:c] syntax instead of function > syntax chain() and islice(), but I don't think the advantage is enough > to out-weigh the costs. > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150103/700acccf/attachment.html> From abarnert at yahoo.com Sat Jan 3 21:40:59 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 3 Jan 2015 21:40:59 +0100 Subject: [Python-ideas] Slicing and Chainging iterables. In-Reply-To: <CANMcLn2Qnh2bRG5mcqcrg-VuOYDY8Vp65ENK9WkT=py4aix8Lg@mail.gmail.com> References: <CANMcLn2d2S=8WSZ3L7kfYk0JaU=3wFg93n=D7Pr1kwh_jK-XGg@mail.gmail.com> <20150103152844.GG27426@ando.pearwood.info> <CANMcLn2Qnh2bRG5mcqcrg-VuOYDY8Vp65ENK9WkT=py4aix8Lg@mail.gmail.com> Message-ID: <9BA6ABD2-2F58-4F95-959D-F0D3ECB3F3E8@yahoo.com> On Jan 3, 2015, at 16:53, yotam vaknin <tomirendo at gmail.com> wrote: > Sorry, I probably wasn't clear enough. > My idea was to add these method (__add__, __getitem__) to the already available iterators (map,zip,dict.items) and generators by default. Not to make these methods part of the iterator protocol. Well, dict.items doesn't return an iterator, it returns an iterable view. More importantly, if you add this just to generators and map and zip, it won't apply to (among other things) filter; the built-in iterators for list, tuple, set, dict, and the dict views; the special type used to iterate types that implement __getitem__ but not __iter__; any of the iterators from itertools--including the result of slicing or chaining two map iterators; files; csv readers; XML iterparse; ? I think that would make the language a lot worse. In general, code doesn't care whether it's gotten an iterator from mapping a function over a file, chaining two iterators together, or iterating a sequence. If you wrote some code that expected to get a generator, used the + operator, and then changed the calling code to filter that generator, it would break, for no good reason. If you really wanted to, you could make this general pretty easily: change the meaning of the + operator so that, after checking __add__ and __radd__, before raising a TypeError, it checks whether both operand have __iter__ methods and, if so, it returns a chain of the two. Slicing has another problem, however. First, it's a bit odd to be able to do i[5:7] but not i[5]. But, more seriously, I wouldn't expect i[5:7] to give me an iterable that, when accessed, discards the first 5 elements of i. And imagine how confusing this code would be: >>> i = iter(range(10)) >>> a = i[2:4] >>> b = i[4:6] >>> list(a) [2, 3] >>> list(b) [8, 9] Hiding the fact that i is an iterable rather than a sequence will just confuse your readers. > 2015-01-03 17:28 GMT+02:00 Steven D'Aprano <steve at pearwood.info>: >> On Sat, Jan 03, 2015 at 03:37:03PM +0200, yotam vaknin wrote: >> > I would like to purpose that in python 3 it will be easy to chain and slice >> > Iterators just like you can add and slice lists and tuples easily. >> > >> > I think there could be 2 methods to do this: >> > 1. Implementing '+' and slicing ([1:2:3]) for iterators and generators by >> > default, Resulting in Itertool's chain(a,b) and islice(a,b,c) respectively. >> > 2. Having itertool's chain and islice imported by default. >> >> The iterator protocol is intentionally simple. To create an iterator, >> you need to only define two methods: >> >> (1) __iter__, which returns the instance itself; >> >> (2) __next__, which returns the next item and raises StopIterator when >> done. >> >> >> With your proposal, you would have to define two or three more methods: >> >> (3) __add__ and possibly __radd__, to chain iterators; >> >> (4) __getitem__, for slicing. >> >> There is nothing stopping you from adding these methods to your own >> iterator classes, but with your proposal that would be compulsory for >> all iterators. >> >> >> > I think since python 3 return zip,map, dict.items and so on as iterators, >> > it makes working with those kind of objects more difficult without having >> > these methods around. And since those methods are important enough to have >> > for lists, it seems important enough for iterators too. >> >> It only takes a single line of code to get iterator chaining and >> slicing: >> >> from itertools import chain, islice >> >> And now you can chain and slice any iterator, regardless of where it >> came from. I don't think that is difficult. >> >> *If* iterators where instances of a concrete base class, then it would >> make sense to add chain and slice methods to that base class so all >> other iterators could inherit from it. But they aren't, iterators in >> Python are values which obey a protocol, not inheritence. That means >> that a functional approach, like itertools, is more appropriate. >> >> I can see the appeal of using + and [a:b:c] syntax instead of function >> syntax chain() and islice(), but I don't think the advantage is enough >> to out-weigh the costs. >> >> >> >> -- >> Steve >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150103/32d86fd7/attachment-0001.html> From abarnert at yahoo.com Sat Jan 3 22:14:54 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 3 Jan 2015 22:14:54 +0100 Subject: [Python-ideas] Slicing and Chainging iterables. In-Reply-To: <CANMcLn2Qnh2bRG5mcqcrg-VuOYDY8Vp65ENK9WkT=py4aix8Lg@mail.gmail.com> References: <CANMcLn2d2S=8WSZ3L7kfYk0JaU=3wFg93n=D7Pr1kwh_jK-XGg@mail.gmail.com> <20150103152844.GG27426@ando.pearwood.info> <CANMcLn2Qnh2bRG5mcqcrg-VuOYDY8Vp65ENK9WkT=py4aix8Lg@mail.gmail.com> Message-ID: <892E5E24-CB4D-448F-8B76-239E80EF6C90@yahoo.com> On Jan 3, 2015, at 16:53, yotam vaknin <tomirendo at gmail.com> wrote: > Sorry, I probably wasn't clear enough. > My idea was to add these method (__add__, __getitem__) to the already available iterators (map,zip,dict.items) and generators by default. Not to make these methods part of the iterator protocol. Reading this again, I think what you _really_ want is probably a generalized view protocol, where views of sequences act like (in fact, _are_) sequences, but with the main advantages of iterators (no storage or upfront computation). Like dictionary views (which is why I chose the name "view", of course). And there's no reason map _couldn't_ return a view instead of an iterator. When called on (all) sequences it would be a sequence view; when called on (any) iterators it would be an iterator view; otherwise it would be a non-sequence non-iterator (that is, reusable, but not indexable) iterable view. But in any case, it would never call the mapped function on a given mapped value until the result is requested. In other words: >>> def spam(x): ... print(x) ... return x+1 >>> m = map(spam, range(5)) >>> m <MapSequenceView at 0x12345678> >>> m[2] 2 3 >>> ms = m[2:4] >>> ms[1] 3 4 You can build view-based zip similarly, and filter, and most of the functions in itertools, and so on (although note that in some cases--like filter--a view of a sequence wouldn't be a sequence). You could also make slices into lazy views instead of copying (as NumPy already does with its arrays). In fact, Python could have designed all of its collections and higher-order functions around views instead of iterators. That's what Swift did. (Swift goes farther, with a hierarchy of kinds of indexing, based on C++, instead of just iterables and sequences.) I wrote a blog post last year (based on the first beta of Swift, which had some oddities in its implementation) examining how this could work in Python terms (http://stupidpythonideas.blogspot.com/2014/07/swift-style-map-and-filter-views.html). I think I've also got a partial implementation somewhere on Github of map, filter, and maybe a few more functions implemented as views. But it would be a pretty major change to Python to move from iterators to views. And iterators are much simpler to create than views, so the tradeoff probably wouldn't be worth it, even if it weren't for the historical/compatibility issue. (It's much the same with Haskell-style lazy lists; Python iterables can only substitute for lazy lists 90% of the time, but that doesn't mean lazy lists are a better language choice.) From sturla.molden at gmail.com Sat Jan 3 23:14:53 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Sat, 3 Jan 2015 22:14:53 +0000 (UTC) Subject: [Python-ideas] Slicing and Chainging iterables. References: <CANMcLn2d2S=8WSZ3L7kfYk0JaU=3wFg93n=D7Pr1kwh_jK-XGg@mail.gmail.com> Message-ID: <605337341442016047.995305sturla.molden-gmail.com@news.gmane.org> yotam vaknin <tomirendo at gmail.com> wrote: > 1. Implementing '+' and slicing ([1:2:3]) for iterators and generators by > default, Resulting in Itertool's chain(a,b) and islice(a,b,c) respectively. This will break NumPy. Sturla From ncoghlan at gmail.com Sun Jan 4 07:05:23 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 4 Jan 2015 16:05:23 +1000 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> Message-ID: <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> On 3 January 2015 at 21:04, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: > On Jan 3, 2015, at 11:39, Chris Angelico <rosuav at gmail.com> wrote: > >> On Sat, Jan 3, 2015 at 9:28 PM, Andrew Barnert <abarnert at yahoo.com> wrote: >>> I agree with this, but it still might be nice to mention that third-party module in shutil and/or pathlib. >> >> Maybe. Is there a mention of other third-party packages in stdlib >> docs? It's a slippery slope; which ones deserve mention? Maybe just >> something like "Note that moving objects to the trash can is possible >> in OS-dependent ways; packages are available on PyPI to make this >> easier", without naming any. > > The impression I get is that this is a slope that Python (or at least Nick Coghlan) wants to start sliding down--with the advent of wheels and pre-installed pip, keeping things out of the stdlib but still accessible for most users is now possible, and often desirable. But I could be putting words (or complete ideas) in people's mouths. Your parenthetical qualification is correct - while I personally think providing authoritative recommendations of third party modules is a good idea, this is far from a universally held opinion amongst the core development team. My own caveat on the idea is that any such explicit recommendations should only cover projects that potentially *could* be included in the standard library from an API design perspective, but we don't want to include them (yet, or perhaps ever) for project governance reasons, such as their update cycles for feature releases needing to be faster than those of the standard library. The "pro" case for this approach is primarily about lowering barriers to entry: for folks just learning Python, "not in the standard library" is currently functionally equivalent in many cases to "Python can't do this". New users often won't yet have the skills needed to do their own research and evaluate competing solutions to their problem, and teaching someone how to find a module on the internet, evaluate the trustworthiness of the people publishing it, get it onto their computer, and update it appropriately over time is genuinely difficult (even with pip provided by default - that helps with the last two steps, but the first two are still genuinely difficult, in a way experienced open source developers often take for granted). Providing authoritative recommendations of third party projects in the standard library documentation helps resolve that situation without needing to resort to making the standard library larger (with the additional long term maintenance complications the latter approach entails). You see a similar dynamic operating in a slightly different way with a project like Django - when the Django developers are significantly more experienced in web development than you are, their opinionated, all-inclusive approach to web framework design is a *feature*, not a defect. It's only when you become more experienced with web service development yourself, and need to deal with problems that don't fit neatly into the SQL-backed three-tier web application model, or need to integrate with other systems which weren't built with Django in mind, that less opinionated frameworks like Flask or Pyramid become interesting - those frameworks leave many more decisions to the developers using them, which is exactly what you want in many situations as an experienced developer, but constantly asking people to make decisions they aren't yet qualified to make poses a huge barrier to entry. It's a potential problem when less experienced folks are put off by the request to make those decisions (since not understanding the questions you're being asked by your tools makes for a terrible user experience), and decide to go learn something else that places fewer demands on them, but it's even worse for everyone trying to deploy and manage the resulting software systems when folks happily make the decisions these advanced DIY integration frameworks ask of them without even realising they're not yet qualified to make them. djangopackages.com is also incredibly useful as a tool for finding peer evaluations of projects in the Django ecosystem. I'm not yet aware of any successful attempts to expand that model to other plugin ecosystems, but the core grid concept of assembling a set of criteria, providing answers across a set of related projects, and publishing the results to provide a starting point for anyone doing their own research in that topic area is a wonderful one. The "con" case is that providing default recommendations can be controversial - when there are multiple competing projects in an area, being the default recommendation of the CPython core development team can be a huge popularity boost, since reputation is to some degree transitive. If someone trusts CPython, and we trust another project enough to recommend it, then a lot of folks will use that recommendation as a substitute for doing their own research (even when they *can* do that research themselves, the default recommendation is likely to be good enough, letting them defer investigation of alternatives until after they've tried the default option). I suspect that objection could potentially be dealt with the same way we deal with any other standard library design decisions: if a module maintainer is comfortable making a recommendation based on their own expertise they can, but broader recommendations spanning several modules would require at least a discussion on python-dev, and potentially even a PEP. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Jan 4 08:23:08 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 4 Jan 2015 17:23:08 +1000 Subject: [Python-ideas] Slicing and Chainging iterables. In-Reply-To: <892E5E24-CB4D-448F-8B76-239E80EF6C90@yahoo.com> References: <CANMcLn2d2S=8WSZ3L7kfYk0JaU=3wFg93n=D7Pr1kwh_jK-XGg@mail.gmail.com> <20150103152844.GG27426@ando.pearwood.info> <CANMcLn2Qnh2bRG5mcqcrg-VuOYDY8Vp65ENK9WkT=py4aix8Lg@mail.gmail.com> <892E5E24-CB4D-448F-8B76-239E80EF6C90@yahoo.com> Message-ID: <CADiSq7csQHN3Yh==A5=DZe-MT959ANvg=8AAWuxikkpJ-F4A6g@mail.gmail.com> On 4 January 2015 at 07:14, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: > But it would be a pretty major change to > Python to move from iterators to views. And iterators are much simpler to create than views, so the tradeoff probably wouldn't be worth it, even if it weren't for the historical/compatibility issue. (It's much the same with Haskell-style lazy lists; Python iterables can only substitute for lazy lists 90% of the time, but that doesn't mean lazy lists are a better language choice.) It's worth noting that many types implement Mapping.(keys,values,items) as iterators in Python 3 rather than as views, and generally don't receive any complaints from users. Iterators are very simple to implement, cover 90% of the use cases, and in those cases where they don't, you can usually write a custom wrapper around the original containers with view-like behaviour. Providing an operator based spelling for itertools.chain, itertools.islice and itertools.repeat is tempting enough on the surface to be suggested every few years (e.g. [1]), but it creates so many new complications on the *implementation* side that the benefits just aren't worth the cost in additional complexity. Cheers, Nick. [1] https://mail.python.org/pipermail/python-ideas/2010-April/006983.html -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From jeanpierreda at gmail.com Sun Jan 4 08:45:16 2015 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sat, 3 Jan 2015 23:45:16 -0800 Subject: [Python-ideas] Slicing and Chainging iterables. In-Reply-To: <CADiSq7csQHN3Yh==A5=DZe-MT959ANvg=8AAWuxikkpJ-F4A6g@mail.gmail.com> References: <CANMcLn2d2S=8WSZ3L7kfYk0JaU=3wFg93n=D7Pr1kwh_jK-XGg@mail.gmail.com> <20150103152844.GG27426@ando.pearwood.info> <CANMcLn2Qnh2bRG5mcqcrg-VuOYDY8Vp65ENK9WkT=py4aix8Lg@mail.gmail.com> <892E5E24-CB4D-448F-8B76-239E80EF6C90@yahoo.com> <CADiSq7csQHN3Yh==A5=DZe-MT959ANvg=8AAWuxikkpJ-F4A6g@mail.gmail.com> Message-ID: <CABicbJ+o-N9XRm5rfzo8aMJ5Pi5oa2MDPQ3=Nu6WmzpeYVaN8w@mail.gmail.com> On Sat, Jan 3, 2015 at 11:23 PM, Nick Coghlan <ncoghlan at gmail.com> wrote: > On 4 January 2015 at 07:14, Andrew Barnert > <abarnert at yahoo.com.dmarc.invalid> wrote: >> But it would be a pretty major change to >> Python to move from iterators to views. And iterators are much simpler to create than views, so the tradeoff probably wouldn't be worth it, even if it weren't for the historical/compatibility issue. (It's much the same with Haskell-style lazy lists; Python iterables can only substitute for lazy lists 90% of the time, but that doesn't mean lazy lists are a better language choice.) > > It's worth noting that many types implement > Mapping.(keys,values,items) as iterators in Python 3 rather than as > views, and generally don't receive any complaints from users. They probably should implement keys/etc. properly. collections.abc.[Mutable]Mapping make it easy to support views if you otherwise implement a dictionary-like object. -- Devin From ncoghlan at gmail.com Sun Jan 4 09:01:57 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 4 Jan 2015 18:01:57 +1000 Subject: [Python-ideas] Slicing and Chainging iterables. In-Reply-To: <CABicbJ+o-N9XRm5rfzo8aMJ5Pi5oa2MDPQ3=Nu6WmzpeYVaN8w@mail.gmail.com> References: <CANMcLn2d2S=8WSZ3L7kfYk0JaU=3wFg93n=D7Pr1kwh_jK-XGg@mail.gmail.com> <20150103152844.GG27426@ando.pearwood.info> <CANMcLn2Qnh2bRG5mcqcrg-VuOYDY8Vp65ENK9WkT=py4aix8Lg@mail.gmail.com> <892E5E24-CB4D-448F-8B76-239E80EF6C90@yahoo.com> <CADiSq7csQHN3Yh==A5=DZe-MT959ANvg=8AAWuxikkpJ-F4A6g@mail.gmail.com> <CABicbJ+o-N9XRm5rfzo8aMJ5Pi5oa2MDPQ3=Nu6WmzpeYVaN8w@mail.gmail.com> Message-ID: <CADiSq7fNTOza0arKmCXM2Wmtm6f2cQ7yqz=Mzs5+ta=m1T+vfw@mail.gmail.com> On 4 January 2015 at 17:45, Devin Jeanpierre <jeanpierreda at gmail.com> wrote: > On Sat, Jan 3, 2015 at 11:23 PM, Nick Coghlan <ncoghlan at gmail.com> wrote: >> On 4 January 2015 at 07:14, Andrew Barnert >> <abarnert at yahoo.com.dmarc.invalid> wrote: >>> But it would be a pretty major change to >>> Python to move from iterators to views. And iterators are much simpler to create than views, so the tradeoff probably wouldn't be worth it, even if it weren't for the historical/compatibility issue. (It's much the same with Haskell-style lazy lists; Python iterables can only substitute for lazy lists 90% of the time, but that doesn't mean lazy lists are a better language choice.) >> >> It's worth noting that many types implement >> Mapping.(keys,values,items) as iterators in Python 3 rather than as >> views, and generally don't receive any complaints from users. > > They probably should implement keys/etc. properly. > collections.abc.[Mutable]Mapping make it easy to support views if you > otherwise implement a dictionary-like object. What people "should" do and what they actually do in practice often differ wildly. In this particular case, folks migrating from Python 2 will frequently rename existing iter* methods to be the Python 3 implementations of the base methods. It's technically a non-compliant implementation of the Python mapping protocol, but you'll only encounter problems if you attempt to use that container implementation with code that relies on the Python 3 view behaviour. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From wes.turner at gmail.com Sun Jan 4 10:27:43 2015 From: wes.turner at gmail.com (Wes Turner) Date: Sun, 4 Jan 2015 03:27:43 -0600 Subject: [Python-ideas] Slicing and Chainging iterables. In-Reply-To: <CANMcLn2d2S=8WSZ3L7kfYk0JaU=3wFg93n=D7Pr1kwh_jK-XGg@mail.gmail.com> References: <CANMcLn2d2S=8WSZ3L7kfYk0JaU=3wFg93n=D7Pr1kwh_jK-XGg@mail.gmail.com> Message-ID: <CACfEFw8DFV_Frv--Nj8ewb--y2yQ8PO9ObgG=yky3xS1C55qZw@mail.gmail.com> In addition to itertools.chain and itertools.islice, this is possible with a number of third-party packages: * http://toolz.readthedocs.org/en/latest/api.html#itertoolz (concat, pluck, first, last, take) * https://github.com/kachayev/fn.py#streams-and-infinite-sequences-declaration (<< 'Stream' operator) * http://funcy.readthedocs.org/en/latest/seqs.html Stdlib docs for this: * https://docs.python.org/2/howto/functional.html#iterators * https://docs.python.org/2/tutorial/classes.html#iterators * https://docs.python.org/2/library/stdtypes.html#iterator-types * https://docs.python.org/2/reference/datamodel.html#object.__iter__ * On Sat, Jan 3, 2015 at 7:37 AM, yotam vaknin <tomirendo at gmail.com> wrote: > I would like to purpose that in python 3 it will be easy to chain and > slice Iterators just like you can add and slice lists and tuples easily. > > I think there could be 2 methods to do this: > 1. Implementing '+' and slicing ([1:2:3]) for iterators and generators by > default, Resulting in Itertool's chain(a,b) and islice(a,b,c) respectively. > 2. Having itertool's chain and islice imported by default. > > I think since python 3 return zip,map, dict.items and so on as iterators, > it makes working with those kind of objects more difficult without having > these methods around. And since those methods are important enough to have > for lists, it seems important enough for iterators too. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150104/d9608e7e/attachment.html> From abarnert at yahoo.com Sun Jan 4 11:32:13 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 4 Jan 2015 11:32:13 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> Message-ID: <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> On Jan 4, 2015, at 7:05, Nick Coghlan <ncoghlan at gmail.com> wrote: > On 3 January 2015 at 21:04, Andrew Barnert > <abarnert at yahoo.com.dmarc.invalid> wrote: >> On Jan 3, 2015, at 11:39, Chris Angelico <rosuav at gmail.com> wrote: >> >>> On Sat, Jan 3, 2015 at 9:28 PM, Andrew Barnert <abarnert at yahoo.com> wrote: >>>> I agree with this, but it still might be nice to mention that third-party module in shutil and/or pathlib. >>> >>> Maybe. Is there a mention of other third-party packages in stdlib >>> docs? It's a slippery slope; which ones deserve mention? Maybe just >>> something like "Note that moving objects to the trash can is possible >>> in OS-dependent ways; packages are available on PyPI to make this >>> easier", without naming any. >> >> The impression I get is that this is a slope that Python (or at least Nick Coghlan) wants to start sliding down--with the advent of wheels and pre-installed pip, keeping things out of the stdlib but still accessible for most users is now possible, and often desirable. But I could be putting words (or complete ideas) in people's mouths. > > Your parenthetical qualification is correct - while I personally think > providing authoritative recommendations of third party modules is a > good idea, this is far from a universally held opinion amongst the > core development team. My own caveat on the idea is that any such > explicit recommendations should only cover projects that potentially > *could* be included in the standard library from an API design > perspective, but we don't want to include them (yet, or perhaps ever) > for project governance reasons, such as their update cycles for > feature releases needing to be faster than those of the standard > library. > > The "pro" case for this approach is primarily about lowering barriers > to entry: for folks just learning Python, "not in the standard > library" is currently functionally equivalent in many cases to "Python > can't do this". New users often won't yet have the skills needed to do > their own research and evaluate competing solutions to their problem, > and teaching someone how to find a module on the internet, evaluate > the trustworthiness of the people publishing it, get it onto their > computer, and update it appropriately over time is genuinely difficult > (even with pip provided by default - that helps with the last two > steps, but the first two are still genuinely difficult, in a way > experienced open source developers often take for granted). Providing > authoritative recommendations of third party projects in the standard > library documentation helps resolve that situation without needing to > resort to making the standard library larger (with the additional long > term maintenance complications the latter approach entails). > > You see a similar dynamic operating in a slightly different way with a > project like Django - when the Django developers are significantly > more experienced in web development than you are, their opinionated, > all-inclusive approach to web framework design is a *feature*, not a > defect. It's only when you become more experienced with web service > development yourself, and need to deal with problems that don't fit > neatly into the SQL-backed three-tier web application model, or need > to integrate with other systems which weren't built with Django in > mind, that less opinionated frameworks like Flask or Pyramid become > interesting - those frameworks leave many more decisions to the > developers using them, which is exactly what you want in many > situations as an experienced developer, but constantly asking people > to make decisions they aren't yet qualified to make poses a huge > barrier to entry. It's a potential problem when less experienced folks > are put off by the request to make those decisions (since not > understanding the questions you're being asked by your tools makes for > a terrible user experience), and decide to go learn something else > that places fewer demands on them, but it's even worse for everyone > trying to deploy and manage the resulting software systems when folks > happily make the decisions these advanced DIY integration frameworks > ask of them without even realising they're not yet qualified to make > them. > > djangopackages.com is also incredibly useful as a tool for finding > peer evaluations of projects in the Django ecosystem. I'm not yet > aware of any successful attempts to expand that model to other plugin > ecosystems, but the core grid concept of assembling a set of criteria, > providing answers across a set of related projects, and publishing the > results to provide a starting point for anyone doing their own > research in that topic area is a wonderful one. > > The "con" case is that providing default recommendations can be > controversial - when there are multiple competing projects in an area, > being the default recommendation of the CPython core development team > can be a huge popularity boost, since reputation is to some degree > transitive. If someone trusts CPython, and we trust another project > enough to recommend it, then a lot of folks will use that > recommendation as a substitute for doing their own research (even when > they *can* do that research themselves, the default recommendation is > likely to be good enough, letting them defer investigation of > alternatives until after they've tried the default option). > > I suspect that objection could potentially be dealt with the same way > we deal with any other standard library design decisions: if a module > maintainer is comfortable making a recommendation based on their own > expertise they can, but broader recommendations spanning several > modules would require at least a discussion on python-dev, and > potentially even a PEP. The question here, then, is how this applies to desktop/shell functionality like moving to trash, administering trash in all the ways someone (Steven?) wanted, other features that someone else raised (aliases/bookmarks/shortcuts/desktop files), and maybe additional stuff (accessing the desktop's extension/file-type mapping). Some of that stuff is already in the stdlib (or might be confusing to novices--e.g., IIRC, on XDG desktops, some things that are done with aliases on Mac are done with desktop files, but in some cases you just use symlinks, so a Linux novice might not even realize he's not writing cross-platform code...). For a GUI app, it's not at all clear that choosing Qt or Wx or one of the platform-specific libraries automatically solves all of these problems for you, while choosing Tk doesn't. For a command-line app, it's even worse. Most Windows users probably have no idea they're asking for desktop integration rather than OS support when they ask how to move a file to the trash or create a shortcut, because Windows intentionally buries that distinction. So when they see that neither os nor shutil nor anything else in the stdlib can help them, they're not going to think, "Oh, I need to go find a platform-specific or cross-platform desktop integration library." (And they're also not going to think "Well, Perl and Ruby and Node don't have these either, so why should I expect Python to?") So, I think in this case it would be useful to recommend something. The question is _what_ to recommend (and where). I think for platform-specific code there are pretty clear winners (PyWin32, PyObjC, GObject, and PyQt/PySide, for the big four). Some might argue for including the XDG package (e.g., if you want to do the same thing as one of the XDG-utils tools, you don't need Gtk or Qt for that), and Mac users might also want a link to the ScriptingBridge guide or something (because some things that are done by API on Windows are done by scripting System Events or Finder or something else), and people using a third-tier desktop on Linux may not have Gtk or Qt libraries to bind to, etc., but I think those are minor questions compared to whether it would be useful to have pointers to these packages for novices. (I think it goes without saying that none of these packages should be added to the stdlib, right?) But for cross-platform code there really isn't an answer. (Well, PyQt/PySide would work, but it's still non-trivial to install and configure on Mac and Windows, and it's overkill.) So, maybe what we need here is for someone to first create the desktop package, and then we can worry about whether it should be added to the stdlib, recommended in the stdlib, or neither. And that still leaves open the possibility that some small subset of this functionality might be separable enough and useful enough to put in the stdlib, or in a small package linked from the stdlib. After all, the equivalent of start/open/xdg-open is already there for some platforms, and webbrowser.open and some of the stuff in shutil. I don't think moving to trash is part of such a subset (for the reasons I gave earlier), but I could be wrong. From ncoghlan at gmail.com Sun Jan 4 14:01:08 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 4 Jan 2015 23:01:08 +1000 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> Message-ID: <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> On 4 January 2015 at 20:32, Andrew Barnert <abarnert at yahoo.com> wrote: > The question here, then, is how this applies to desktop/shell functionality like moving to trash, administering trash in all the ways someone (Steven?) wanted, other features that someone else raised (aliases/bookmarks/shortcuts/desktop files), and maybe additional stuff (accessing the desktop's extension/file-type mapping). Some of that stuff is already in the stdlib (or might be confusing to novices--e.g., IIRC, on XDG desktops, some things that are done with aliases on Mac are done with desktop files, but in some cases you just use symlinks, so a Linux novice might not even realize he's not writing cross-platform code...). > > For a GUI app, it's not at all clear that choosing Qt or Wx or one of the platform-specific libraries automatically solves all of these problems for you, while choosing Tk doesn't. For a command-line app, it's even worse. Most Windows users probably have no idea they're asking for desktop integration rather than OS support when they ask how to move a file to the trash or create a shortcut, because Windows intentionally buries that distinction. So when they see that neither os nor shutil nor anything else in the stdlib can help them, they're not going to think, "Oh, I need to go find a platform-specific or cross-platform desktop integration library." (And they're also not going to think "Well, Perl and Ruby and Node don't have these either, so why should I expect Python to?") So, I think in this case it would be useful to recommend something. The question is _what_ to recommend (and where). > > I think for platform-specific code there are pretty clear winners (PyWin32, PyObjC, GObject, and PyQt/PySide, for the big four). Some might argue for including the XDG package (e.g., if you want to do the same thing as one of the XDG-utils tools, you don't need Gtk or Qt for that), and Mac users might also want a link to the ScriptingBridge guide or something (because some things that are done by API on Windows are done by scripting System Events or Finder or something else), and people using a third-tier desktop on Linux may not have Gtk or Qt libraries to bind to, etc., but I think those are minor questions compared to whether it would be useful to have pointers to these packages for novices. (I think it goes without saying that none of these packages should be added to the stdlib, right?) Right, a useful first step might be a "Tools for writing rich client applications in Python" overview page. That's out of scope for any of the current *.python.org documentation set (it doesn't fit under docs.python.org *or* packaging.python.org, nor anywhere on the main site), but if it's written in an appropriate tone (i.e. making it clear that these are default starting points, and recommendations for further evaluation, rather than universally applicable definitive conclusions), then a future "ecosystem.python.org" guide could start the same way p.p.o did, as a separate Sphinx project hosted on ReadTheDocs that was later blessed with a python.org subdomain. https://packaging.python.org/en/latest/deployment.html is an example skeleton of such a page for package deployment tools. We haven't filled that one *in* yet, but hopefully it conveys the general idea. https://packaging.python.org/en/latest/extensions.html is another one reviewing binary extensions, and some of the tools available in that space. For those curious as to "Why not the wiki?", a Sphinx project hosted on a pull request capable service like GitHub, BitBucket or Kallithea offers a much nicer workflow for reviewing of proposed changes, together with an integrated issue tracker for submitting proposals for updates (https://github.com/pypa/python-packaging-user-guide/ is the project behind packaging.python.org, for example). > But for cross-platform code there really isn't an answer. (Well, PyQt/PySide would work, but it's still non-trivial to install and configure on Mac and Windows, and it's overkill.) So, maybe what we need here is for someone to first create the desktop package, and then we can worry about whether it should be added to the stdlib, recommended in the stdlib, or neither. Just describing the problem space can be incredibly valuable for newcomers - one of the hardest things to learn as a beginner is the difference between "this is hard because I don't know what I'm doing" and "this is hard because I'm working on an unsolved problem" (with "this is deceptively easy because I'm setting myself up for problems later on" being an interesting variant that's hard to detect no matter how experienced you are). I see this is a *lot* when working with the scientific community - they actually have much higher expectations of cross-platform tooling than professional developers do. From the outside, the political, personal and commercial meta-issues that combine to make cross-platform standards development an unholy nightmare aren't immediately obvious, and nor are the trade-offs between "lowest common denominator" (or Java-style "reinvent all the wheels") functionality that works everywhere, and providing access to advanced platform specific functionality. > And that still leaves open the possibility that some small subset of this functionality might be separable enough and useful enough to put in the stdlib, or in a small package linked from the stdlib. After all, the equivalent of start/open/xdg-open is already there for some platforms, and webbrowser.open and some of the stuff in shutil. I don't think moving to trash is part of such a subset (for the reasons I gave earlier), but I could be wrong. While I think it's better to leave the question open rather than try to pre-empt an answer, I do believe we can at least say *now* isn't the right time to be adding "move to trash" functionality to the standard library - the functionality isn't mature enough within the broader ecosystem to determine an appropriate answer. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From random832 at fastmail.us Sun Jan 4 20:24:36 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Sun, 04 Jan 2015 14:24:36 -0500 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <20150103015045.GB27426@ando.pearwood.info> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> Message-ID: <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> On Fri, Jan 2, 2015, at 20:50, Steven D'Aprano wrote: > > Because what _not_ having a > > cross-platform wrapper gets you is windows and mac left behind > > Really? I would expect that Windows is probably the *easiest* platform > to implement this functionality, as it has a single, standard, stable > API for moving files to trash. (Or so I am lead to believe.) You can > probably use that API via ctypes, or via pywin32. Why would someone who is developing on Linux and has no pre-made library function to call bother with that? If being cross-platform isn't easy, it won't happen. You see it now with the lack of any support for "call glob on arguments on windows and not on unix" [because the shell handles it on unix] whether directly, in argparse, or in fileinput - today, nothing ever calls glob, and so people calling such scripts on windows can't use wildcards (glob is also different from windows wildcards in subtle ways) > Getting OS X right is very complicated, as Andrew has so ably explained. > > In Linux and other Unixes, the situation is like OS X, only more so > since there's no one single authority. However, what authority there is does not allegedly demand that you present dialog boxes and admin elevation prompts to the user (this has been asserted, but no citation has been given). > You have a choice of desktop > environments, which may or may not provide a move-to-trash API, > including no desktop environment at all. Gnome provides an API for > moving to trash, but I don't know how well it supports the freedesktop > standard; KDE supports the freedesktop standard, but I don't know if it > provides an API that can be called. XFCE has partial support. I don't see why you need to call an API to the desktop enviroment. The entire point of the spec is to provide compatibility _between_ implementations on the same filesystem - you can quit gnome and log into KDE and see the same trash. Python would just be a separate implementation that stands by itself. From ncoghlan at gmail.com Sun Jan 4 21:25:01 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 5 Jan 2015 06:25:01 +1000 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> Message-ID: <CADiSq7fHy94T6CNbyzv+NFUca05MqENkMrpYteU5w09GGz+SHQ@mail.gmail.com> On 5 Jan 2015 05:25, <random832 at fastmail.us> wrote: > > On Fri, Jan 2, 2015, at 20:50, Steven D'Aprano wrote: > > You have a choice of desktop > > environments, which may or may not provide a move-to-trash API, > > including no desktop environment at all. Gnome provides an API for > > moving to trash, but I don't know how well it supports the freedesktop > > standard; KDE supports the freedesktop standard, but I don't know if it > > provides an API that can be called. XFCE has partial support. > > I don't see why you need to call an API to the desktop enviroment. The > entire point of the spec is to provide compatibility _between_ > implementations on the same filesystem - you can quit gnome and log into > KDE and see the same trash. That compatibility is implemented *by* the desktop environments. Yes, you could write your own, but why would you, when the whole point of providing a standard API would be to integrate more cleanly with the underlying OS. > Python would just be a separate > implementation that stands by itself. The word "just" does not belong in that sentence. Reliable cross-platform development & testing for operations that are tightly coupled to operating system services is hard, period - the associated test matrix of platforms and permission levels ensures that. Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150105/10de07ae/attachment.html> From solipsis at pitrou.net Sun Jan 4 21:35:07 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 4 Jan 2015 21:35:07 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <CADiSq7fHy94T6CNbyzv+NFUca05MqENkMrpYteU5w09GGz+SHQ@mail.gmail.com> Message-ID: <20150104213507.708c3548@fsol> On Mon, 5 Jan 2015 06:25:01 +1000 Nick Coghlan <ncoghlan at gmail.com> wrote: > On 5 Jan 2015 05:25, <random832 at fastmail.us> wrote: > > > > On Fri, Jan 2, 2015, at 20:50, Steven D'Aprano wrote: > > > > You have a choice of desktop > > > environments, which may or may not provide a move-to-trash API, > > > including no desktop environment at all. Gnome provides an API for > > > moving to trash, but I don't know how well it supports the freedesktop > > > standard; KDE supports the freedesktop standard, but I don't know if it > > > provides an API that can be called. XFCE has partial support. > > > > I don't see why you need to call an API to the desktop enviroment. The > > entire point of the spec is to provide compatibility _between_ > > implementations on the same filesystem - you can quit gnome and log into > > KDE and see the same trash. > > That compatibility is implemented *by* the desktop environments. Yes, you > could write your own, but why would you, when the whole point of providing > a standard API would be to integrate more cleanly with the underlying OS. For the same reason that we ship a HTTP server while we could integrate more cleanly with Apache? :-) When there's a well-defined spec, it can be simpler to implement the spec yourself rather than pull third-party dependencies, especially when those happen to be huge as may be the case with desktop environments. Regards Antoine. From abarnert at yahoo.com Sun Jan 4 22:02:27 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 4 Jan 2015 22:02:27 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> Message-ID: <6A8A7845-7865-4746-9859-CD51FC695DDA@yahoo.com> On Jan 4, 2015, at 20:24, random832 at fastmail.us wrote: > On Fri, Jan 2, 2015, at 20:50, Steven D'Aprano wrote: > >> Getting OS X right is very complicated, as Andrew has so ably explained. >> >> In Linux and other Unixes, the situation is like OS X, only more so >> since there's no one single authority. > > However, what authority there is does not allegedly demand that you > present dialog boxes and admin elevation prompts to the user (this has > been asserted, but no citation has been given). Where has that, or anything remotely similar, been alleged or asserted? What _has_ been asserted is that an app has to fit in with the choices made by the platform implementation. And the only example given was given with a direct quote from the spec linked in the preceding email. I'm not sure how much simpler a citation you need in order to be able to follow it. The example was about platform-specific ways to allow admins to disable sticky-bit checking on network mounts. There are more such things that are left incompletely specified, or explicitly left unspecified for future development. I don't think any of them have anything to do with dialog boxes or other user interaction, and I don't recall anyone saying otherwise. (I suppose if you didn't read carefully, you might take something like "SHOULD warn the administrator" as implying a prompt saying "Go get your sys admin and drag him over to the screen" rather than, say, a syslog warning with a string the admin can see later and look up in help or google). >> You have a choice of desktop >> environments, which may or may not provide a move-to-trash API, >> including no desktop environment at all. Gnome provides an API for >> moving to trash, but I don't know how well it supports the freedesktop >> standard; KDE supports the freedesktop standard, but I don't know if it >> provides an API that can be called. XFCE has partial support. > > I don't see why you need to call an API to the desktop enviroment. The > entire point of the spec is to provide compatibility _between_ > implementations on the same filesystem - you can quit gnome and log into > KDE and see the same trash. Python would just be a separate > implementation that stands by itself. The last sentence is exactly the same thing you already got two detailed answers to that you haven't responded to in any way. So I suspect the reason you don't understand why calling an API is a good idea is that you skimmed over the explanations that would explain exactly that. Go back and read those emails. From ethan at stoneleaf.us Sun Jan 4 22:02:39 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 04 Jan 2015 13:02:39 -0800 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> Message-ID: <54A9AA6F.8020400@stoneleaf.us> On 01/04/2015 11:24 AM, random832 at fastmail.us wrote: > If being cross-platform isn't easy, it won't happen. You see it now with > the lack of any support for "call glob on arguments on windows and not > on unix" [because the shell handles it on unix] whether directly, in > argparse, or in fileinput Could you elaborate on this point? > I don't see why you need to call an API to the desktop enviroment. The > entire point of the spec is to provide compatibility _between_ > implementations on the same filesystem - you can quit gnome and log into > KDE and see the same trash. Python would just be a separate > implementation that stands by itself. If by "stands by itself" you mean "desktops can't interact with it", how would that be useful? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150104/2d4ed31e/attachment.sig> From abarnert at yahoo.com Sun Jan 4 23:06:42 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 4 Jan 2015 23:06:42 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <20150104213507.708c3548@fsol> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <CADiSq7fHy94T6CNbyzv+NFUca05MqENkMrpYteU5w09GGz+SHQ@mail.gmail.com> <20150104213507.708c3548@fsol> Message-ID: <8AB561C3-EDFD-4F6B-8AE9-9115627AF6A4@yahoo.com> On Jan 4, 2015, at 21:35, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Mon, 5 Jan 2015 06:25:01 +1000 > Nick Coghlan <ncoghlan at gmail.com> wrote: >> On 5 Jan 2015 05:25, <random832 at fastmail.us> wrote: >>> >>> On Fri, Jan 2, 2015, at 20:50, Steven D'Aprano wrote: >> >>>> You have a choice of desktop >>>> environments, which may or may not provide a move-to-trash API, >>>> including no desktop environment at all. Gnome provides an API for >>>> moving to trash, but I don't know how well it supports the freedesktop >>>> standard; KDE supports the freedesktop standard, but I don't know if it >>>> provides an API that can be called. XFCE has partial support. >>> >>> I don't see why you need to call an API to the desktop enviroment. The >>> entire point of the spec is to provide compatibility _between_ >>> implementations on the same filesystem - you can quit gnome and log into >>> KDE and see the same trash. >> >> That compatibility is implemented *by* the desktop environments. Yes, you >> could write your own, but why would you, when the whole point of providing >> a standard API would be to integrate more cleanly with the underlying OS. > > For the same reason that we ship a HTTP server while we could integrate > more cleanly with Apache? :-) That's an interesting point--but note that Python _also_ has WSGI and various other ways to integrate more cleanly with Apache (and various other web servers). If you just want to put up a simple web service to handle a few users (e.g., as a user interface to your application), SimpleHTTPServer is sometimes the answer. But if you want to add a web service to a major Apache deployment, you generally don't want to do that by deploying SimpleHTTPServer alongside it. It's not _impossible_ to share ports, SSL certs, configs, logs, deployment scripts, monitoring tools, etc. between the two, but it's not the way you'd want to do things. (There's also the fact that SimpleHTTPServer won't be able to handle the same load as Apache?) > When there's a well-defined spec, I suppose it depends on your definition of "well-defined spec". As Nick said, XFCE only partly supports the spec, and he's not sure how completely GNOME does. As I pointed out, there are gaps in the spec for both implementation-defined and future-expansion behavior. That isn't the same as the case with HTTP/1.1. Look at http://www.freedesktop.org/wiki/Software. Notice that xdg-utils does not yet have a trash command, and there is no separate trash program or library. (As far as I know, there are at least two XDG trash programs on github which have not yet been submitted to be part of the XDG software collection, which don't appear to be 100% compatible with each other. (For example, one of them requires top trash directories to have exactly mode 41777, the other just requires them to be executable for the current user and have the sticky bit on.) Does creating a third one--and putting it in the Python stdlib forever--really sound like a good idea? (As a side note, one of the two above-mentioned trash programs is written in Node, and I don't see anyone suggesting that the functionality be moved into the Node stdlib.) From stephen at xemacs.org Sun Jan 4 23:18:07 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 05 Jan 2015 07:18:07 +0900 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> Message-ID: <87tx06197k.fsf@uwakimon.sk.tsukuba.ac.jp> random832 at fastmail.us writes: > I don't see why you need to call an API to the desktop enviroment. The > entire point of the spec is to provide compatibility _between_ > implementations on the same filesystem - you can quit gnome and log into > KDE and see the same trash. Python would just be a separate > implementation that stands by itself. But Python doesn't much care about similarities or differences between GNOME and KDE AFAICS -- the relationships to Mac and Windows are much more interesting. Emacs has had a policy of implementing everything itself for decades, and its mindshare has been decreasing for decades.[1] Many Emacs developers are cross-platform users, and the common cross-platform implementation is great for them. Most potential users are not, and the small differences between Emacs's behavior and the platform API mean that Emacs is not a separate implementation that stands by itself -- it's a bad implementation that spends a lot of time standing in a corner.[2] This is not the only reason for Emacs's relative loss of popularity, but it's definitely a common complaint -- and the complainers often never return to Emacs channels when it becomes clear that the Emacs way is the only way. In any case, the small platform differences and the odd problem with the spec mean that this is definitely suited for out-of-stdlib experimentation. Footnotes: [1] The former is a consequence of a policy that (official) Emacs on non-GNU systems can't have any feature that Emacs on the GNU System doesn't have -- if you want a move-to-trash command on Mac, you have to implement for GNU. [2] A traditional punishment for mischievous children, especially those who don't finish their chores or homework, where I grew up. From random832 at fastmail.us Sun Jan 4 23:50:42 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Sun, 04 Jan 2015 17:50:42 -0500 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <54A9AA6F.8020400@stoneleaf.us> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> Message-ID: <1420411842.1291347.209489837.52B6E2E7@webmail.messagingengine.com> On Sun, Jan 4, 2015, at 16:02, Ethan Furman wrote: > If by "stands by itself" you mean "desktops can't interact with it", how > would that be useful? They interact with it _through the filesystem_, because it does the same things with the filesystem as all the desktop environments do and as the spec says. Just like Gnome doesn't have to interact directly with KDE - programs from each environment don't have to call the other's library APIs - for both Nautilus and Dolphin to display the same trash contents and to do the same thing when you trash a file. From random832 at fastmail.us Mon Jan 5 00:10:01 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Sun, 04 Jan 2015 18:10:01 -0500 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <54A9AA6F.8020400@stoneleaf.us> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> Message-ID: <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> On Sun, Jan 4, 2015, at 16:02, Ethan Furman wrote: > On 01/04/2015 11:24 AM, random832 at fastmail.us wrote: > > > If being cross-platform isn't easy, it won't happen. You see it now with > > the lack of any support for "call glob on arguments on windows and not > > on unix" [because the shell handles it on unix] whether directly, in > > argparse, or in fileinput > > Could you elaborate on this point? On Unix, as I assume you know, the shell is responsible for interpreting the kind of wildcard patterns that glob uses (plus shell-specific extensions), and passing a list of proper filenames as the child process's argv. On Windows, this does not happen - the child process is simply passed a single string with the whole command line. Python (or the C Runtime Library that the python interpreter is linked against) converts this to a list of strings for individual arguments based on spaces and quotes, but does not interpret wildcard patterns in any of the arguments. The C Runtime Library _can_ do this automatically - this is done on MSVC by linking the "setargv.obj" library routine, which replaces a standard internal routine that does not expand wildcards, but it does a poor job because it does not know which arguments are intended to be filenames vs other strings, and there is traditionally no way to escape them [since * and ? aren't allowed in filenames, there's no reason not to allow "some directory\*.txt", all in quotes, as an argument that will be handled as a wildcard] The appropriate place to expand them would be after you know you intend to treat a list of arguments as a list of filenames, rather than at program start - after options are parsed, for example (so an option with an argument with an asterisk in it doesn't get turned into multiple arguments), or if a list is being passed in to the fileinput module. This should also only be done on windows, and not on other platforms (since on other platforms this is supposed to be handled by the shell rather than the child process). Right now, none of this is done. If you pass *.txt on the command line to a python script, it will attempt to open a file called "*.txt". ---- Another separate but related issue is the fact that windows wildcards do not behave in the same way as python glob patterns. Bracketed character classes are not supported, and left bracket is a valid character in filenames unlike ? and *. There are some subtleties around dots ("*.*" will match all filenames, even with no dot. "*." matches filenames without any dot.), they're case-insensitive (I think glob does handle this part, but not in the same way as the platform in some cases), and they can match the short-form alternate filenames [8 characters, dot, 3 characters], so "*.htm" will typically match most files ending in ".html" as well as those ending in ".htm". It might be useful to provide a way to make glob behave in the windows-specific way (using the platform-specific functions FindFirstFileEx and RtlIsNameInExpression on windows.) From rosuav at gmail.com Mon Jan 5 01:42:42 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 5 Jan 2015 11:42:42 +1100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <54A9AA6F.8020400@stoneleaf.us> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> Message-ID: <CAPTjJmq0F7drKJ1UkJ7V1PF=5zA=AoSMXdxqrjVBUjB9B+Bdiw@mail.gmail.com> On Mon, Jan 5, 2015 at 8:02 AM, Ethan Furman <ethan at stoneleaf.us> wrote: > On 01/04/2015 11:24 AM, random832 at fastmail.us wrote: > >> If being cross-platform isn't easy, it won't happen. You see it now with >> the lack of any support for "call glob on arguments on windows and not >> on unix" [because the shell handles it on unix] whether directly, in >> argparse, or in fileinput > > Could you elaborate on this point? On Unix, a command like "python some_script.py *.txt" is parsed by the shell[1] into a bunch of extra arguments, eg "python some_script.py file_one.txt file_two.txt file_three.txt". On Windows[2], this expansion isn't done, so the script gets a literal "*.txt" as an argument. So a Unix program is usually written to accept multiple args, but a Windows program usually expects to do its own globbing. Now, what happens when you want to reference a file whose name actually includes U+002A (or \x2A, given that file names on most Linux systems consist of bytes)? On Unix, you put a backslash in front of the name, to prevent the shell from expanding it ("\*.txt"), or use some other means of calling up the program (subprocess.call(["python","some_script.py","*.txt"])). Your expectation then is that the script will NOT expand globs; but the script will see the exact same thing in sys.argv as it does on Windows, where the expectation is that it WILL expand globs (and, if I'm not mistaken, asterisks are simply disallowed in file names - though that might be a filesystem-specific rule). So somehow, your script has to know whether it's supposed to call glob or not. Or, more likely, it'll do whatever is appropriate for the platform its developer uses, and then misbehaves on the other one. ChrisA [1] Assuming you're using one of the standard shells, and assuming you haven't configured this feature away, etc, etc. Normal user expectations. [2] Again, assuming you're using the default shell; but again, normal user expectation, ergo replacement shells often replicate this. From steve at pearwood.info Mon Jan 5 01:57:30 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 5 Jan 2015 11:57:30 +1100 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> References: <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> Message-ID: <20150105005730.GR27426@ando.pearwood.info> On Sun, Jan 04, 2015 at 06:10:01PM -0500, random832 at fastmail.us wrote: [...] > Right now, none of this is done. If you pass *.txt on the command line > to a python script, it will attempt to open a file called "*.txt". The important thing is that shells have different behaviour. A way, perhaps not the best way, but a way, of getting close to platform independent behaviour when it comes to globbing is to use the glob module. On Windows, you will get Python's definition of globbing. On POSIX systems, the globbing module will do nothing, because the shell will most likely have already expanded the wild-cards. (I don't know of any Unix shells which behave like Windows.) Windows users are unlikely to try to use Unix-shell-specific wildcards, because they are Windows users and they won't have any expectation that they will work. Unix users will, and they will work because the shell interprets the wildcards before Python sees them. > Another separate but related issue is the fact that windows wildcards do > not behave in the same way as python glob patterns. I don't understand why you say this. As I understand it, there is no such thing as "Windows wildcards" as every application which wants to support wildcards has to implement their own. If you want to know what kinds of globbing wildcards the application supports, you have to read the application documentation. (Or guess.) I am not a Windows expert, so I may have been misinformed. Anyone care to comment? > Bracketed character classes are not supported, Applications which use the Python glob module do support them. Other applications may support them, if they so choose to implement it. > and left bracket is a valid character in > filenames unlike ? and *. And on POSIX systems, *all* wildcards are valid characters in file names. If you want to specify a file called literally "*.txt", or "spam[and eggs?].jpg", you have to escape the wildcards. Windows is no different. The glob module supports escaping of wildcards, doesn't it? If so, we have no problem. If not, that's a bug, or at least an obvious and important piece of missing functionality. > There are some subtleties around dots ("*.*" > will match all filenames, even with no dot. "*." matches filenames > without any dot.), they're case-insensitive (I think glob does handle > this part, but not in the same way as the platform in some cases), and > they can match the short-form alternate filenames [8 characters, dot, 3 > characters], so "*.htm" will typically match most files ending in > ".html" as well as those ending in ".htm". As I said, I don't believe that there is any standard Windows filename wildcard handling, so the behaviour you describe may apply to some applications but not all. Anyone like to comment? > It might be useful to provide a way to make glob behave in the > windows-specific way (using the platform-specific functions > FindFirstFileEx and RtlIsNameInExpression on windows.) This may be a good idea. Maybe glob needs to stay as it is, for backwards compatibility, and a new module osglob be created that aims to implement globbing according to the expected rules of the operating system. osglob could work like the os module and delegate to platform-specific modules posix_glob, windows_glob, osx_glob etc., with the current glob module remaining for applications which want to present a more-or-less equivalent globbing behaviour. -- Steven From rosuav at gmail.com Mon Jan 5 02:08:06 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 5 Jan 2015 12:08:06 +1100 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <20150105005730.GR27426@ando.pearwood.info> References: <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> <20150105005730.GR27426@ando.pearwood.info> Message-ID: <CAPTjJmozZ04qcrFLbGGrEWuC7kPDAr71g3aUWATc5tL9v4sX=g@mail.gmail.com> On Mon, Jan 5, 2015 at 11:57 AM, Steven D'Aprano <steve at pearwood.info> wrote: > On Sun, Jan 04, 2015 at 06:10:01PM -0500, random832 at fastmail.us wrote: > [...] >> Right now, none of this is done. If you pass *.txt on the command line >> to a python script, it will attempt to open a file called "*.txt". > > On Windows, you will get Python's definition of globbing. On POSIX > systems, the globbing module will do nothing, because the shell will > most likely have already expanded the wild-cards You're assuming that there are no wildcard characters actually included in the file names, which on Unix systems is perfectly possible. How can you reliably manipulate a file called *.txt when there are foo.txt and bar.txt in the same directory? With most Unix programs, you escape the asterisk to get it past the shell, the program does no globbing of its own, and you're safe. If your Python script unconditionally globs its file names, you're stuck. >> Another separate but related issue is the fact that windows wildcards do >> not behave in the same way as python glob patterns. > > I don't understand why you say this. As I understand it, there is no > such thing as "Windows wildcards" as every application which wants to > support wildcards has to implement their own. If you want to know what > kinds of globbing wildcards the application supports, you have to read > the application documentation. (Or guess.) > > I am not a Windows expert, so I may have been misinformed. Anyone care > to comment? There's a very standard interpretation, which I think is codified in the FindFirstFile API, although I'm not sure of the details. >> and left bracket is a valid character in >> filenames unlike ? and *. > > And on POSIX systems, *all* wildcards are valid characters in file > names. If you want to specify a file called literally "*.txt", or > "spam[and eggs?].jpg", you have to escape the wildcards. Windows is no > different. No, Windows *is* different, because you simply aren't allowed to have those characters in file names: >>> open("*.txt","w") Traceback (most recent call last): File "<pyshell#0>", line 1, in <module> open("*.txt","w") OSError: [Errno 22] Invalid argument: '*.txt' Unix allows anything other than a slash or NUL, so all wildcards have to be escaped to get them past the shell; or, alternatively, you can just use a non-shell way of starting a program, like Python's own subprocess module. > The glob module supports escaping of wildcards, doesn't it? If so, we > have no problem. If not, that's a bug, or at least an obvious and > important piece of missing functionality. I'm not sure that's the solution, because then Unix users would have to double-escape everything. It can be done, of course, but this is the exact sort of complication that single-platform developers often don't even think of. Suppose you develop on Windows, and just unconditionally glob all your arguments... your program will work fine on Unix, until it's asked to deal with a file with an asterisk in it, and your user complains very loudly about how it misbehaved... and maybe destroyed a bunch of files. Oops. Not good. ChrisA From steve at pearwood.info Mon Jan 5 02:13:12 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 5 Jan 2015 12:13:12 +1100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> Message-ID: <20150105011312.GS27426@ando.pearwood.info> On Sun, Jan 04, 2015 at 11:01:08PM +1000, Nick Coghlan wrote: > For those curious as to "Why not the wiki?", a Sphinx project hosted > on a pull request capable service like GitHub, BitBucket or Kallithea > offers a much nicer workflow for reviewing of proposed changes, > together with an integrated issue tracker for submitting proposals for > updates (https://github.com/pypa/python-packaging-user-guide/ is the > project behind packaging.python.org, for example). The concept of "proposed changes" goes completely against the grain of community-managed content. Imagine if Wikipedia required you to make pull requests. The question to be asked is not whether "pull requests" are "nicer" than a wiki, but whether this information needs to be owned by a single person (or a small group thereof), and hence use the packaging.python.org module, or owned by the whole community, and hence use the wiki model. At the moment, the wiki is horribly underused. I'm not blaming anyone for that, I'm just as much to blame as anyone else. I have made far more contributions to the ActiveState recipes than I have to the Python wiki. But I'd really like to see the wiki become the second place people look for information, after the official docs. And hopefully there won't be a need for a third place :-) Although the quality of contrabutions is variable, the PHP community has made a very interesting decision to integrate community-supplied information with official docs: http://php.net/manual/en/function.dechex.php I wonder how we might do something similar? -- Steve From donald at stufft.io Mon Jan 5 02:17:57 2015 From: donald at stufft.io (Donald Stufft) Date: Sun, 4 Jan 2015 20:17:57 -0500 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <20150105011312.GS27426@ando.pearwood.info> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> Message-ID: <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> > On Jan 4, 2015, at 8:13 PM, Steven D'Aprano <steve at pearwood.info> wrote: > > On Sun, Jan 04, 2015 at 11:01:08PM +1000, Nick Coghlan wrote: > >> For those curious as to "Why not the wiki?", a Sphinx project hosted >> on a pull request capable service like GitHub, BitBucket or Kallithea >> offers a much nicer workflow for reviewing of proposed changes, >> together with an integrated issue tracker for submitting proposals for >> updates (https://github.com/pypa/python-packaging-user-guide/ is the >> project behind packaging.python.org, for example). > > The concept of "proposed changes" goes completely against the grain of > community-managed content. Imagine if Wikipedia required you to make > pull requests. > > The question to be asked is not whether "pull requests" are "nicer" than > a wiki, but whether this information needs to be owned by a single > person (or a small group thereof), and hence use the > packaging.python.org module, or owned by the whole community, and hence > use the wiki model. > > At the moment, the wiki is horribly underused. I'm not blaming anyone > for that, I'm just as much to blame as anyone else. I have made far more > contributions to the ActiveState recipes than I have to the Python wiki. > But I'd really like to see the wiki become the second place people look > for information, after the official docs. And hopefully there won't be a > need for a third place :-) > > Although the quality of contrabutions is variable, the PHP community has > made a very interesting decision to integrate community-supplied > information with official docs: > > http://php.net/manual/en/function.dechex.php > > I wonder how we might do something similar? > On the other hand, a number of pages on the Wiki are either completely out of date because nobody cares about them or need to be locked because too many people care about it and they fight over which project should be named as recommended. Wiki content can be good for more factual based information but I think the format is relatively poor for things which are more opinion based. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From ben+python at benfinney.id.au Mon Jan 5 02:22:35 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 05 Jan 2015 12:22:35 +1100 Subject: [Python-ideas] Encouraging more use of the Python wiki (was: Adding `pathlib.Path` method that would send file to recycle bin) References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> Message-ID: <85387qyqas.fsf_-_@benfinney.id.au> Donald Stufft <donald at stufft.io> writes: > On the other hand, a number of pages on the Wiki are either completely > out of date because nobody cares about them or need to be locked > because too many people care about it and they fight over which > project should be named as recommended. In recent years (I noticed this since about 2013), the Python wiki does not offer an ?Edit? function when I am logged in; every page presents ?Immutable Page? instead. That has prevented me from improving pages I've wanted to. And no, I haven't gone to the effort of constructing a bug report for this. Perhaps the same is true for many other logged-in users as well? -- \ ?It is well to remember that the entire universe, with one | `\ trifling exception, is composed of others.? ?John Andrew Holmes | _o__) | Ben Finney From steve at pearwood.info Mon Jan 5 02:31:23 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 5 Jan 2015 12:31:23 +1100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> Message-ID: <20150105013123.GT27426@ando.pearwood.info> On Sun, Jan 04, 2015 at 02:24:36PM -0500, random832 at fastmail.us wrote: > On Fri, Jan 2, 2015, at 20:50, Steven D'Aprano wrote: > > > Because what _not_ having a > > > cross-platform wrapper gets you is windows and mac left behind > > > > Really? I would expect that Windows is probably the *easiest* platform > > to implement this functionality, as it has a single, standard, stable > > API for moving files to trash. (Or so I am lead to believe.) You can > > probably use that API via ctypes, or via pywin32. > > Why would someone who is developing on Linux and has no pre-made library > function to call bother with that? Because they want to write cross-platform code. For my own personal scripts, I wouldn't bother. If I had a Windows system which I intended to use the script on, I would, or if I intended to make the script available to others, I might. > If being cross-platform isn't easy, it won't happen. You see it now with > the lack of any support for "call glob on arguments on windows and not > on unix" [because the shell handles it on unix] whether directly, in > argparse, or in fileinput - today, nothing ever calls glob, and so > people calling such scripts on windows can't use wildcards (glob is also > different from windows wildcards in subtle ways) I think you are mistaken that nothing uses glob: https://searchcode.com/?q=import+glob+lang%3Apython [...] > > You have a choice of desktop > > environments, which may or may not provide a move-to-trash API, > > including no desktop environment at all. Gnome provides an API for > > moving to trash, but I don't know how well it supports the freedesktop > > standard; KDE supports the freedesktop standard, but I don't know if it > > provides an API that can be called. XFCE has partial support. > > I don't see why you need to call an API to the desktop enviroment. You don't *need* to do so, but doing so guarantees that your code will match the expected behaviour as provided by the API, and avoids needing to re-invent the wheel. I would expect that for something which eventually makes it into the standard library, if it ever does, it will have multiple implementations: - call the platform API, if one exists and it is available - fall back to a Python implementation, if no such API exists That is exactly what the third party Send2Trash package does. If Gnome is installed and the GIO library is installed, it uses that, otherwise it falls back on its own (possibly inferior?) implementation. -- Steven From random832 at fastmail.us Mon Jan 5 02:33:48 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Sun, 04 Jan 2015 20:33:48 -0500 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <20150105005730.GR27426@ando.pearwood.info> References: <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> <20150105005730.GR27426@ando.pearwood.info> Message-ID: <1420421628.1098818.209525929.58232F93@webmail.messagingengine.com> On Sun, Jan 4, 2015, at 19:57, Steven D'Aprano wrote: > On Sun, Jan 04, 2015 at 06:10:01PM -0500, random832 at fastmail.us wrote: > > Another separate but related issue is the fact that windows wildcards do > > not behave in the same way as python glob patterns. > > I don't understand why you say this. As I understand it, there is no > such thing as "Windows wildcards" as every application which wants to > support wildcards has to implement their own. If you want to know what > kinds of globbing wildcards the application supports, you have to read > the application documentation. (Or guess.) Wildcards are built into the FindFirstFile/FindNextFile (and thereby the _tfindfirst/next) API used for enumerating directories, and users do not generally expect to be able to apply wildcards to path components other than the last one. These functions are used by applications, and generally by libraries provided in higher-level languages. > I am not a Windows expert, so I may have been misinformed. Anyone care > to comment? From random832 at fastmail.us Mon Jan 5 02:40:16 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Sun, 04 Jan 2015 20:40:16 -0500 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <20150105013123.GT27426@ando.pearwood.info> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <20150105013123.GT27426@ando.pearwood.info> Message-ID: <1420422016.1099647.209526789.52D57044@webmail.messagingengine.com> On Sun, Jan 4, 2015, at 20:31, Steven D'Aprano wrote: > I think you are mistaken that nothing uses glob: I am very confident in saying that programs that do not use glob _to process lists of filenames given on the command line_ outnumber those that do at least 1000:1, and that the vast majority of your search consists of uses for processing things other than command line arguments. > https://searchcode.com/?q=import+glob+lang%3Apython The vast majority of results (I used https://searchcode.com/?q=glob.glob+lang%3Apython so I could see the actual call in context) are using it for hardcoded strings, not command line arguments (and to do it correctly they should do even that conditionally on os.name == 'nt' [or os2 or ce, not sure about the other non-posix values]) From ethan at stoneleaf.us Mon Jan 5 02:41:24 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 04 Jan 2015 17:41:24 -0800 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <20150105011312.GS27426@ando.pearwood.info> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> Message-ID: <54A9EBC4.2010305@stoneleaf.us> On 01/04/2015 05:13 PM, Steven D'Aprano wrote: > Although the quality of contributions is variable, the PHP community has > made a very interesting decision to integrate community-supplied > information with official docs: > > http://php.net/manual/en/function.dechex.php > > I wonder how we might do something similar? I hope that we do not. A separate wiki is fine, but the official docs should just have the official stuff in them. If the docs were a wiki, we'd have scads of entries on "how to program <language-x> in Python", and IMNSHO the official docs should just be about programming Python in Python. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150104/764cda18/attachment.sig> From steve at pearwood.info Mon Jan 5 03:07:35 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 5 Jan 2015 13:07:35 +1100 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <CAPTjJmozZ04qcrFLbGGrEWuC7kPDAr71g3aUWATc5tL9v4sX=g@mail.gmail.com> References: <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> <20150105005730.GR27426@ando.pearwood.info> <CAPTjJmozZ04qcrFLbGGrEWuC7kPDAr71g3aUWATc5tL9v4sX=g@mail.gmail.com> Message-ID: <20150105020735.GU27426@ando.pearwood.info> On Mon, Jan 05, 2015 at 12:08:06PM +1100, Chris Angelico wrote: > On Mon, Jan 5, 2015 at 11:57 AM, Steven D'Aprano <steve at pearwood.info> wrote: > > On Sun, Jan 04, 2015 at 06:10:01PM -0500, random832 at fastmail.us wrote: > > [...] > >> Right now, none of this is done. If you pass *.txt on the command line > >> to a python script, it will attempt to open a file called "*.txt". > > > > On Windows, you will get Python's definition of globbing. On POSIX > > systems, the globbing module will do nothing, because the shell will > > most likely have already expanded the wild-cards > > You're assuming that there are no wildcard characters actually > included in the file names, which on Unix systems is perfectly > possible. How can you reliably manipulate a file called *.txt when > there are foo.txt and bar.txt in the same directory? With most Unix > programs, you escape the asterisk to get it past the shell, the > program does no globbing of its own, and you're safe. If your Python > script unconditionally globs its file names, you're stuck. You seem to be right. With double wildcard expansion (by the shell and by the python script) there doesn't seem to be a straightforward way to get just the file name with the wildcard in it: [steve at ando junk]$ cat testglob.py import sys import glob for arg in sys.argv[1:]: print(arg + ":-") for name in glob.glob(arg): print(" " + name) [steve at ando junk]$ ls eggs.jpg *.jpg testglob.py [steve at ando junk]$ python3.3 testglob.py *.jpg eggs.jpg:- eggs.jpg *.jpg:- eggs.jpg *.jpg [steve at ando junk]$ python3.3 testglob.py \*.jpg *.jpg:- eggs.jpg *.jpg [steve at ando junk]$ python3.3 testglob.py \\*.jpg \*.jpg:- I guess the simple solutions are to either do a platform check first, or to provide a --no-glob command-line switch to turn globbing off. But I guess people won't think of that until its reported as a bug :-) > >> Another separate but related issue is the fact that windows wildcards do > >> not behave in the same way as python glob patterns. > > > > I don't understand why you say this. As I understand it, there is no > > such thing as "Windows wildcards" as every application which wants to > > support wildcards has to implement their own. If you want to know what > > kinds of globbing wildcards the application supports, you have to read > > the application documentation. (Or guess.) > > > > I am not a Windows expert, so I may have been misinformed. Anyone care > > to comment? > > There's a very standard interpretation, which I think is codified in > the FindFirstFile API, although I'm not sure of the details. Do all Windows apps call FindFirstFile? > >> and left bracket is a valid character in > >> filenames unlike ? and *. > > > > And on POSIX systems, *all* wildcards are valid characters in file > > names. If you want to specify a file called literally "*.txt", or > > "spam[and eggs?].jpg", you have to escape the wildcards. Windows is no > > different. > > No, Windows *is* different, because you simply aren't allowed to have > those characters in file names: I know that. The point that Random raised is that some wildcards are legal in Windows file names. Yes, and *all* wildcards are legal in POSIX filenames, so whatever problems we have with escaping are going to occur on both platforms, as you have already pointed out above. In any case, having a platform specific globbing module is looking more and more useful. Does this need a PEP? -- Steven From random832 at fastmail.us Mon Jan 5 03:41:55 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Sun, 04 Jan 2015 21:41:55 -0500 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <20150105020735.GU27426@ando.pearwood.info> References: <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> <20150105005730.GR27426@ando.pearwood.info> <CAPTjJmozZ04qcrFLbGGrEWuC7kPDAr71g3aUWATc5tL9v4sX=g@mail.gmail.com> <20150105020735.GU27426@ando.pearwood.info> Message-ID: <1420425715.1110144.209535413.2CD84ED2@webmail.messagingengine.com> On Sun, Jan 4, 2015, at 21:07, Steven D'Aprano wrote: > Do all Windows apps call FindFirstFile? Most Windows apps' only interaction with wildcards is via the system file selection dialog. Ultimately, FindFirstFile/FindNextFile is the only way to enumerate files in a directory - it's the equivalent of opendir/readdir. You're expected to pass in directory\* or *.* if you want to enumerate all files in a directory with no filtering. I think it's rare for programs not deliberately written to emulate some other behavior or ported from unix (with canned implementations of opendir, readdir, glob, and fnmatch being easier than rewriting everything to use the windows functions) to implement their own wildcard handling rather than using the Windows API - the C Runtime functions findfirst/findnext date back to early DOS C compilers (the OS functions were int 21,4E and 4F.) > I know that. The point that Random raised is that some wildcards are > legal in Windows file names. Well, more specifically, that the square bracket is both allowed in Windows filenames and not expected to be used as a wildcard, so there is no escape mechanism (since the only standard wildcards are ? and *, which are not allowed in filenames) >Yes, and *all* wildcards are legal in > POSIX filenames, so whatever problems we have with escaping are going to > occur on both platforms, as you have already pointed out above. Well, yes, except for the fact that escaping and quoting is handled by the shell on POSIX. My _main_ reason for mentioning it was to forestall objections involving applying wildcards to double-quoted strings, since double quote does _not_ normally escape wildcards on Windows. From yawar.amin at gmail.com Mon Jan 5 04:47:06 2015 From: yawar.amin at gmail.com (Yawar Amin) Date: Sun, 04 Jan 2015 22:47:06 -0500 Subject: [Python-ideas] Syntax: 'return: ...' expressions Message-ID: <54AA093A.2050508@gmail.com> Note: I'm reposting this here after posting on the /r/Python reddit as I've realised this is a better venue: http://www.reddit.com/r/Python/comments/2ra8yc/return_expressions/ I have an idea for a new syntax element that combines any number of statements and expressions into a single expression. I'm partial to calling it a 'return' expression because that should be compatible with existing code--there's guaranteed to be no existing code of the form 'return:'. But we could call it 'do' or 'val' or 'expr' or whatever. expression ::= ... | return_expr return_expr ::= "return" ":" statement* expression+ We can define all expression-oriented syntax in terms of return expressions. I.e., wrap up all statement-oriented syntax inside an expression. E.g., x = ( return: if y == 1: z = 'a' elif y == 2: z = 'b' elif y == 3: z = 'c' else: z = 'd' z ) Or a single-line example, x = return: print "Blah"; 5 I know, it's the Python programmer's dreaded multi-line expression. Note here that I'm not proposing any change to indentation rules. I'm relying on parens to relax the rules. There's precedent for using parens in new kinds of expressions--e.g. generator expressions. So the usage shouldn't look alien in Python code. Now the controversy. We can use a return expression to get a multi-line, multi-statement, lambda, as long as lambda only cares that its body is a single exprssion. Which I believe is the case, e.g. this is valid Python today: f = lambda x: ( x, x + 1, x + 2 ) Anyway, an imaginary lambda example: txtName.text.signal.map( # Need to wrap the return expr in parens to separate the lambdas. lambda s: ( return: l = len(s) l >= 5 ), # The next argument, a lambda without a return expr, happily lives # alongside. lambda err: notifications.send(err) ).subscribe( # Parens not needed here, only a single argument. lambda flag: return: pnlMain.back_color = "green" if flag else "red" # Must end with an expr, unlike a normal function. Think of it # this way: we're 'inside' a return, we _have to_ return # some value. None ) I believe 'return: ...' is an unintrusive and versatile solution. It's _not_ meant to just forcefully shoehorn full functionality into lambdas as I believe I show above; it doesn't break compatibility; it doesn't require any indent/whitespace rule changes; and it's guaranteed to not affect _any_ existing code. Regards, Yawar P.S. Relevant: https://groups.google.com/d/msg/python-ideas/kYQbvsmyM-4/ufU26RPTjLQJ 'A much better idea would be to find a way to make all compound statements into expressions, future-proofing the decision and avoiding the redundancy between "compound statement statement" and "compound statement expression".' https://groups.google.com/d/msg/python-ideas/EQQq3--DDu0/UTcfI34sVwwJ 'Something that has started to annoy me in the last couple of years is the fact that most Python control statements cannot be used as expressions. I feel this is a pretty deep limitation and personally I don't feel it's well-justified.' -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 834 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150104/6dccc00f/attachment-0001.sig> From stephen at xemacs.org Mon Jan 5 05:19:16 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 05 Jan 2015 13:19:16 +0900 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <20150105020735.GU27426@ando.pearwood.info> References: <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> <20150105005730.GR27426@ando.pearwood.info> <CAPTjJmozZ04qcrFLbGGrEWuC7kPDAr71g3aUWATc5tL9v4sX=g@mail.gmail.com> <20150105020735.GU27426@ando.pearwood.info> Message-ID: <87lhlh2723.fsf@uwakimon.sk.tsukuba.ac.jp> Steven D'Aprano writes: > In any case, having a platform specific globbing module By "platform-specific", do you mean one module that handles all platforms appropriately (what this thread is about), or do you mean per-platform modules (what "platform-specific" means to me)? > is looking more and more useful. Does this need a PEP? If you're really serious about putting it in the stdlib, I would certainly think so. It's absolutely clear that the solution is going to be complicated and hard to understand. Without a PEP, nobody will know what is and what is not a bug. Likely even with a PEP a lot of people will claim the PEP is buggy no matter what it says! I would hope the different specs will partition the complainers, but I'm not even sure of that. ;-) From abarnert at yahoo.com Mon Jan 5 10:49:03 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 5 Jan 2015 10:49:03 +0100 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <54AA093A.2050508@gmail.com> References: <54AA093A.2050508@gmail.com> Message-ID: <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> On Jan 5, 2015, at 4:47, Yawar Amin <yawar.amin at gmail.com> wrote: > I know, it's the Python programmer's dreaded multi-line expression. Note > here that I'm not proposing any change to indentation rules. But that _is_ a new indentation rule. You're using multiple statements inside parentheses, indented as if they were a block. (This is effectively just a suggestion for brace syntax, and I'm not sure what the return adds to it that you couldn't get just from brace syntax. You also seem to have invented a rule that a sequence of statements has the value of the last statement as its value. Which requires first inventing a rule that gives values to statements. I'll assume you wanted to go with the same rule the interactive interpreter uses to display a last value--an expression statement has its expression's value, and any other kind of statement has None?) You're assuming that parentheses have an existing indentation rule that you can piggyback on, but they don't. Superficially, parenthesis continuation may look similar to block indentation, but it's not at all the same. Compare: >>> (spam, ... eggs) >>> if spam: ... eggs The first is a perfectly valid tuple display; the second raises an IndentationError. Because suites begin with a new indent, but paren continuation just concatenates the lines together. So, unless you're suggesting that any free-form sequence of statements is now legal within parens, you must be inventing a new indent rule to use within those parens. There are also problems with nesting indents and colons. Note that compound statements (like if) include either by an inline simple statement list, or an indented block of one or more statements. This means you can't have two possibly-indenting colons on the same line. Unless your new expression can only be used in simple statements, and a simple statement with your new expression can't be used in an inline simple statement list, you're changing the way colons and indents are parsed by statements. Finally, are you sure your new return: isn't ambiguous without look ahead or context? When the parser reads "if spam: return", is it starting a return simple statement, or an expression statement that starts with a return expression? Have you tried actually writing out the grammar? From rosuav at gmail.com Mon Jan 5 10:55:41 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 5 Jan 2015 20:55:41 +1100 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> Message-ID: <CAPTjJmoikqFxa0hXJp435hi3nrMOHgqpt_sqdu_MX6-_gUXGqw@mail.gmail.com> On Mon, Jan 5, 2015 at 8:49 PM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: > Finally, are you sure your new return: isn't ambiguous without look ahead or context? When the parser reads "if spam: return", is it starting a return simple statement, or an expression statement that starts with a return expression? Have you tried actually writing out the grammar? > The way I see it, the return-expression is "return:", whereas the simple return statement would have no colon. Your other concerns, though, are quite valid. ChrisA From abarnert at yahoo.com Mon Jan 5 11:10:40 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 5 Jan 2015 11:10:40 +0100 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <87lhlh2723.fsf@uwakimon.sk.tsukuba.ac.jp> References: <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> <20150105005730.GR27426@ando.pearwood.info> <CAPTjJmozZ04qcrFLbGGrEWuC7kPDAr71g3aUWATc5tL9v4sX=g@mail.gmail.com> <20150105020735.GU27426@ando.pearwood.info> <87lhlh2723.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <160BFDF9-033E-43C2-AF9E-18740FB1E90A@yahoo.com> It seems to me that if you want auto-globbing support on Windows, the "right" way to do it is to link in the fancy setargv instead of the default one. This handles wildcards exactly the same way the command.com/cmd.exe builtin commands do, which I suspect is what Windows users would actually be expecting of they passed a wildcard on the command line. (I think the docs no longer guarantee that this is true, but it's probably still true, and certainly closer to true than if you try to do it manually.) The only problem is that you can't add a command-line switch that controls at runtime which implementation of setargv got linked into the msvcrt prequel at build time. There used to be a way around this for DOS/Win16 programs, but I don't think there is for NT/Win32 (short of ignoring the argv and asking for and parsing the command line instead), so this would be an all-or-nothing change. Are there programs that depend on _not_ having auto-globbing on Windows? There might be... At any rate, I can't imagine many programs want POSIX (sh-style) globbing but without other POSIX shell features (at least the different split/quoting/escape rules, if not the various kinds of expansion, etc.); the only reason anyone calls glob.glob for Windows is because it's relatively easy (compared to using win32api to call FindFirstFile, or using win32api to get the whole command line and then using shlex and os to process it, depending on which one they actually wanted) and not _too_ terrible for simple cases. If there's a way to give people something that's not only easier but better, I don't think they'd complain. Sent from a random iPhone On Jan 5, 2015, at 5:19, "Stephen J. Turnbull" <stephen at xemacs.org> wrote: > Steven D'Aprano writes: > >> In any case, having a platform specific globbing module > > By "platform-specific", do you mean one module that handles all > platforms appropriately (what this thread is about), or do you mean > per-platform modules (what "platform-specific" means to me)? > >> is looking more and more useful. Does this need a PEP? > > If you're really serious about putting it in the stdlib, I would > certainly think so. It's absolutely clear that the solution is going > to be complicated and hard to understand. Without a PEP, nobody will > know what is and what is not a bug. Likely even with a PEP a lot of > people will claim the PEP is buggy no matter what it says! I would > hope the different specs will partition the complainers, but I'm not > even sure of that. ;-) > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From abarnert at yahoo.com Mon Jan 5 11:28:21 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 5 Jan 2015 11:28:21 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <54A9EBC4.2010305@stoneleaf.us> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <54A9EBC4.2010305@stoneleaf.us> Message-ID: <31C37528-4EE5-4F52-827D-425D5FB1999E@yahoo.com> On Jan 5, 2015, at 2:41, Ethan Furman <ethan at stoneleaf.us> wrote: > On 01/04/2015 05:13 PM, Steven D'Aprano wrote: > >> Although the quality of contributions is variable, the PHP community has >> made a very interesting decision to integrate community-supplied >> information with official docs: >> >> http://php.net/manual/en/function.dechex.php >> >> I wonder how we might do something similar? > > I hope that we do not. A separate wiki is fine, but the official docs should just have the official stuff in them. MSDN has something similar. They've gone back and forth between a half-dozen different models, and I think at present it's just (formatted) comments on each documentation page. At any rate, it's very clear what's official documentation and what's some guy saying "the docs say you can't do Foo, but here's some code that simulates Foo in most cases". I've found useful stuff there. Then again, I've also found live code (and StackOverflow questions, etc.) that's broken because someone took some sample fragment from a comment and used it inappropriately (often ignoring the very next comment that explains why you shouldn't do that if X, Y, or Z). Both PHP and MSDN have a separate page for each function/method/attribute, while Python has a page per module (broken down into sections that are usually very nicely organized, so I wouldn't want to lose that...), so I'm not sure a similar comment format would work in the first place. Maybe allowing inline comments at every section break, entry, or even paragraph would work (like the way PyGame docs have inline example-search and other links)? But that could easily lead to docs that are so cluttered with comments (or with disclosure triangles, if they start off hidden) that the actual documentation is hard to read. > If the docs were a wiki, we'd have scads of entries on "how to program <language-x> in Python", In other fora (python-list, python-tutor, StackOverflow, comments on widely-followed blogs, etc.) when someone suggests writing Java-style code instead of Pythonic code they usually get quickly shouted down, often even with good explanations, so I'm not sure things would be any worse on a wiki attached to the Python docs. Also, the Python wiki is already a wiki, and isn't dominated by that kind of cruft. So I'm not sure that problem is as much of a threat. That being said, I agree with your conclusion. Docs seem to work best information that's managed like code; things that aren't quite docs but are useful may work best in other ways, but people have already found the wiki, StackOverflow, ActiveState, blogs, etc. and aren't having any problem using them. And, of all the alternate places to find elaborated documentation, the wiki seems to be among the least popular, which doesn't seem to imply that more wikification is the answer to any relevant question. From abarnert at yahoo.com Mon Jan 5 11:37:22 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 5 Jan 2015 11:37:22 +0100 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <CAPTjJmoikqFxa0hXJp435hi3nrMOHgqpt_sqdu_MX6-_gUXGqw@mail.gmail.com> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> <CAPTjJmoikqFxa0hXJp435hi3nrMOHgqpt_sqdu_MX6-_gUXGqw@mail.gmail.com> Message-ID: <B36D6AB2-BA0C-4162-B1DC-3FD89504938B@yahoo.com> On Jan 5, 2015, at 10:55, Chris Angelico <rosuav at gmail.com> wrote: > On Mon, Jan 5, 2015 at 8:49 PM, Andrew Barnert > <abarnert at yahoo.com.dmarc.invalid> wrote: >> Finally, are you sure your new return: isn't ambiguous without look ahead or context? When the parser reads "if spam: return", is it starting a return simple statement, or an expression statement that starts with a return expression? Have you tried actually writing out the grammar? > > The way I see it, the return-expression is "return:", whereas the > simple return statement would have no colon. Are you suggesting that "return:" would be a single token (as opposed to all the other colons in the tokenizer), or that you'd use look ahead to disambiguate? I suspect it probably _is_ possible to make something work that fits into Python's parser, but it isn't immediately obvious (at least to me) what that would be, which is why I wanted to see a grammar. (And also to know which "level" this comes at, to automatically answer questions like how it fits into things like conditional expressions and comprehension clauses and so forth.) From dw+python-ideas at hmmz.org Mon Jan 5 11:38:29 2015 From: dw+python-ideas at hmmz.org (David Wilson) Date: Mon, 5 Jan 2015 10:38:29 +0000 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> Message-ID: <20150105103828.GA22528@k3> I have no problem with this being included somehow in the standard library, but pathlib to me is all about the filesystem, whereas (as others have pointed out), the Recycle Bin and associated concepts on other OS relate pretty uniformly to the OS shell. There are many shell-related concepts that could be nicely exposed, but again I don't think any of them belong in pathlib. For example, labels/tags, owner application, icon, visibility flags, etc. David On Mon, Dec 29, 2014 at 06:54:46PM +0200, Ram Rachum wrote: > Hi guys, > > I want to have a method of?`pathlib.Path` that would send a file to the recycle > bin. (i.e. soft delete.) > > What do you think about adding this?? > > I see there's a PyPI package `Send2Trash` that does this, but it would be nice > if this was in the standard library. > > > Thanks, > Ram. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From rosuav at gmail.com Mon Jan 5 11:46:46 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 5 Jan 2015 21:46:46 +1100 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <B36D6AB2-BA0C-4162-B1DC-3FD89504938B@yahoo.com> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> <CAPTjJmoikqFxa0hXJp435hi3nrMOHgqpt_sqdu_MX6-_gUXGqw@mail.gmail.com> <B36D6AB2-BA0C-4162-B1DC-3FD89504938B@yahoo.com> Message-ID: <CAPTjJmrtT3SPF965Q+oUxf3bA=HiXsj8JYKaJvd+dqfc7DLrQw@mail.gmail.com> On Mon, Jan 5, 2015 at 9:37 PM, Andrew Barnert <abarnert at yahoo.com> wrote: > Are you suggesting that "return:" would be a single token (as opposed to all the other colons in the tokenizer), or that you'd use look ahead to disambiguate? Oh, I see what you mean. I'm sure this can be dealt with, though; if nothing else, requiring that this kind of expression be parenthesized would do it, as 'return' inside parens has no meaning. It'd be like a genexp. But I still don't think it's a good idea :) ChrisA From alexisl at hp.com Mon Jan 5 12:20:11 2015 From: alexisl at hp.com (Alexis Lee) Date: Mon, 5 Jan 2015 11:20:11 +0000 Subject: [Python-ideas] inspect.getclassdistance Message-ID: <20150105112011.GB7357@hp.com> Hopefully sufficiently documented to stand without introduction: def getclassdistance(srcs, dst): """srcs may be either a single class or a list of (class, distance) pairs. dst is the superclass to find. Performs a breadth-first search for dst, returning the shortest distance. """ if type(srcs) != list: srcs = [(srcs, 0)] if len(srcs) == 0: return None here, distance = srcs[0] if here == dst: return distance newSrc = srcs[1:] + [(c, distance + 1) for c in list(here.__bases__)] return classDistance(newSrc, dst) If this is already implemented and I just couldn't find it, I'm more than happy to withdraw the idea. Alexis -- Nova Engineer, HP Cloud. AKA lealexis, lxsli. From steve at pearwood.info Mon Jan 5 12:43:15 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 5 Jan 2015 22:43:15 +1100 Subject: [Python-ideas] inspect.getclassdistance In-Reply-To: <20150105112011.GB7357@hp.com> References: <20150105112011.GB7357@hp.com> Message-ID: <20150105114315.GZ27426@ando.pearwood.info> On Mon, Jan 05, 2015 at 11:20:11AM +0000, Alexis Lee wrote: > Hopefully sufficiently documented to stand without introduction: Er, not really. What does distance mean in the context of classes? When would you use this? Can you give some examples, e.g. what's the distance between int and str? > def getclassdistance(srcs, dst): > """srcs may be either a single class or a list of (class, distance) pairs. > dst is the superclass to find. > Performs a breadth-first search for dst, returning the shortest distance. > """ > if type(srcs) != list: > srcs = [(srcs, 0)] > if len(srcs) == 0: > return None Shouldn't that be an exception? > here, distance = srcs[0] > if here == dst: > return distance > newSrc = srcs[1:] + [(c, distance + 1) for c in list(here.__bases__)] > return classDistance(newSrc, dst) What's classDistance? > If this is already implemented and I just couldn't find it, I'm more > than happy to withdraw the idea. Explain it first :-) -- Steven From rosuav at gmail.com Mon Jan 5 12:51:33 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 5 Jan 2015 22:51:33 +1100 Subject: [Python-ideas] inspect.getclassdistance In-Reply-To: <20150105112011.GB7357@hp.com> References: <20150105112011.GB7357@hp.com> Message-ID: <CAPTjJmqxhSZYyewxBm600hwieeYhiFM9F+0kY7JZGiCJ0bsAdw@mail.gmail.com> On Mon, Jan 5, 2015 at 10:20 PM, Alexis Lee <alexisl at hp.com> wrote: > def getclassdistance(srcs, dst): > """srcs may be either a single class or a list of (class, distance) pairs. > dst is the superclass to find. > Performs a breadth-first search for dst, returning the shortest distance. > """ > > if type(srcs) != list: > srcs = [(srcs, 0)] > if len(srcs) == 0: > return None > here, distance = srcs[0] > if here == dst: > return distance > newSrc = srcs[1:] + [(c, distance + 1) for c in list(here.__bases__)] > return classDistance(newSrc, dst) I'm not 100% sure that I understand what this is doing, so here's a reimplementation of what I think it's doing. Tell me where this one's wrong, and you'll be a long way toward explaining what yours is accomplishing :) def getclassdistance(src, dst): """Measure the shortest parent chain which leads from src to dst Returns the length of the shortest such chain, or None if no chain exists.""" srcs = [(src, 0)] while srcs: here, distance = srcs.pop(0) if here is dst: return distance for c in here.__bases__: srcs.append((c, distance+1)) I'm still not sure what the value of this is, and therefore why it shouldn't just be a personal utility function (not everything needs to be in the stdlib), but if this is an iterative equivalent to your recursive function (assuming that classDistance is getclassdistance), then it's at least reasonably simple. I made a couple of other changes along the way, including requiring that the first arg be a single class (is there any reason, other than recursion, to provide a pre-populated list?), and using 'is' rather than '==' to compare classes. But since I can't test this against your use-case, I can't verify that it's doing the right job. ChrisA From abarnert at yahoo.com Mon Jan 5 13:09:37 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 5 Jan 2015 13:09:37 +0100 Subject: [Python-ideas] inspect.getclassdistance In-Reply-To: <20150105112011.GB7357@hp.com> References: <20150105112011.GB7357@hp.com> Message-ID: <3D63844A-E1EA-4708-ADF0-8AB4D8453EDD@yahoo.com> On Jan 5, 2015, at 12:20, Alexis Lee <alexisl at hp.com> wrote: > Hopefully sufficiently documented to stand without introduction: It doesn't actually run as written (maybe you just used the wrong name in what was supposed to be a recursive call?), and I'm not sure you're doing the BFS right, but I think I can guess what you're after: You're trying to find out if dst is an ancestor of src (the option of taking a list of pairs, despite being documented, seems to be there only for internal use by recursive calls) and, if so, how long the path is between them. And: * Only actual inheritance counts; abc.register or other subclass hooks are ignored. * In case of diamond inheritance where the sides are of unequal length, the shorter one counts (even if that shorter path would never be followed by the c3 rule and therefore isn't even relevant at runtime). * More generally, you don't care about the c3 MRO path, or the actual path taken for any particular method's lookup, just the inheritance path itself. * The actual classes on the path are irrelevant to you, you just want its length. Also, it seems like it would be simpler to just create a general inheritance-BFS-walker function that yields each base as it goes, which would allow you to write this function as a one-liner. I also don't get why you're using recursion rather than a loop here, given that there's nothing interesting on the stack frames. I suspect you wrote this in Common Lisp or OCaml first and then ported it to Python? That might also explain why it's such a strange fit for the way inheritance works in Python. But, most importantly, if I'm right about what this does, I can't think of anywhere I'd want to use it. I can imagine cases (although not very Pythonic ones) where I'd want to know, say, the MRO path (as in dst.__mro__[:dst.__mro__.find(src)][::-1]), but this... When would it ever be interesting? > def getclassdistance(srcs, dst): > """srcs may be either a single class or a list of (class, distance) pairs. > dst is the superclass to find. > Performs a breadth-first search for dst, returning the shortest distance. > """ > > if type(srcs) != list: > srcs = [(srcs, 0)] > if len(srcs) == 0: > return None > here, distance = srcs[0] > if here == dst: > return distance > newSrc = srcs[1:] + [(c, distance + 1) for c in list(here.__bases__)] > return classDistance(newSrc, dst) > > If this is already implemented and I just couldn't find it, I'm more > than happy to withdraw the idea. > > > Alexis > -- > Nova Engineer, HP Cloud. AKA lealexis, lxsli. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From rosuav at gmail.com Mon Jan 5 13:30:28 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 5 Jan 2015 23:30:28 +1100 Subject: [Python-ideas] inspect.getclassdistance In-Reply-To: <3D63844A-E1EA-4708-ADF0-8AB4D8453EDD@yahoo.com> References: <20150105112011.GB7357@hp.com> <3D63844A-E1EA-4708-ADF0-8AB4D8453EDD@yahoo.com> Message-ID: <CAPTjJmqkQq2rRYj+6ywyyQKVwMctUrr+4aSyH9TdLu8PpciT9A@mail.gmail.com> On Mon, Jan 5, 2015 at 11:09 PM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: > Also, it seems like it would be simpler to just create a general inheritance-BFS-walker function that yields each base as it goes, which would allow you to write this function as a one-liner. > def bfs_bases(cls): """Yield the base which are belong to us""" yield from cls.__bases__ for cls in cls.__bases__: yield from bfs_bases(cls) :) ChrisA From jdhardy at gmail.com Mon Jan 5 14:25:08 2015 From: jdhardy at gmail.com (Jeff Hardy) Date: Mon, 5 Jan 2015 13:25:08 +0000 Subject: [Python-ideas] python on mobile In-Reply-To: <CADiSq7ewkZwRZhkeyuJC32KeaaPuq9FTxQXHiY5uhOZBA+Un=g@mail.gmail.com> References: <CADjSo4TssZHGiOCm110co-BQxLAXZpCZbT1T=zdd8eBavnyVzQ@mail.gmail.com> <CAJxq848RyRhbeD4kHssFo1jHUtVX5N1_G=RB1rB+DkmDaxxA2A@mail.gmail.com> <CADiSq7enX0nmNd2AEHGwCgdDsgaXWzdz0A1w5iU_m1V+JdHe6g@mail.gmail.com> <CAJxq84-EeN0v49sHpPYZStJJ_615quqMYmmq2OPvhcaae6Hfiw@mail.gmail.com> <CADjSo4Seed-B0FmTUikrtvDux_LprdvedVrR1-RvQS8fPE=LiQ@mail.gmail.com> <87k31833vw.fsf@uwakimon.sk.tsukuba.ac.jp> <CAJxq848T09_c-jrHhtyyWUBFk7By7xtUphCExom7wNd36ZCtow@mail.gmail.com> <CADiSq7ewkZwRZhkeyuJC32KeaaPuq9FTxQXHiY5uhOZBA+Un=g@mail.gmail.com> Message-ID: <CAF7AXFFh0voFPLPtYxv0qC8yDXmxdBn5K-PT_5oZSXC8RT7jLg@mail.gmail.com> On Fri, Jan 2, 2015 at 6:40 PM, Nick Coghlan <ncoghlan at gmail.com> wrote: > On 31 December 2014 at 13:42, Russell Keith-Magee <russell at keith-magee.com> > wrote: >> >> >> >> On Wed, Dec 31, 2014 at 11:16 AM, Stephen J. Turnbull <stephen at xemacs.org> >> wrote: >>> >>> Fetchinson . writes: >>> >>> > > I'm not sure exactly what would be discussed [on mobile-sig], >>> > > especially in the short term when the big ticket item is getting >>> > > the patches into the main Python tree. >>> >>> It probably matters more that there *is* substantive discussion than >>> *what* is discussed. There are probably a lot of people who are far >>> more interested in "how to do 'it' *now*" than "get 'it' into the main >>> Python tree 'someday'", for example. >> >> >> I'm not sure if a mailing list would help with this as much as a good, >> simple set of docs. It's not like there's unknown territory here; once >> you've got Python installed on a mobile device, the "how" is reasonably >> easy. The catch is that someone with Python experience won't necessarily >> have experience managing a cross-platform C/Automake build, which is what >> you need to get Python installed in the first place. > > > One of the benefits of a SIG over python-ideas is that their remit can be > broader than just CPython. I'd like to see a mobile-sig, because I think there's enough to discuss that isn't relevant to -dev. I plan to get IronPython running on Xamarin Android/iOS sometime this year so it would be helpful to coordinate things with CPython and Jython - for example, what the values will be for things like sys.platform and os.name will be, and issues around file system emulation, supported modules, etc. that should be compatible for all implementations. Plus there's the issue of new libraries for sensors, GPS, cameras, etc. that provide Pythonic wrappers over the platform APIs. It's not just "get CPython working on mobile" but "how should Python look & feel on mobile devices". (Of course, CPython is special so it's implementation details are still fair game.) So +1 on mobile-sig from me. I'll even volunteer to get it set up & administer it, if no one else is dying to. - Jeff From alexisl at hp.com Mon Jan 5 14:44:50 2015 From: alexisl at hp.com (Alexis Lee) Date: Mon, 5 Jan 2015 13:44:50 +0000 Subject: [Python-ideas] inspect.getclassdistance In-Reply-To: <20150105114315.GZ27426@ando.pearwood.info> References: <20150105112011.GB7357@hp.com> <20150105114315.GZ27426@ando.pearwood.info> Message-ID: <20150105134450.GC7357@hp.com> Steven D'Aprano said on Mon, Jan 05, 2015 at 10:43:15PM +1100: > Er, not really. What does distance mean in the context of classes? When > would you use this? Can you give some examples, e.g. what's the distance > between int and str? Thanks for your fast response! Distance means the number of classes in the inheritance hierarchy between classes A and B. EG given a system: A <-- B1 <-- B2 <-- D ^ / '--------C-------" Given by: class A(object) class B1(A) class B2(B1) class C(A) class D(B2, C) The distances should be: dst A B1 B2 C D A 0 None None None None B1 1 0 None None None src B2 2 1 0 None None C 1 None None 0 None D 2 2 1 1 0 > > def getclassdistance(srcs, dst): > > """srcs may be either a single class or a list of (class, distance) pairs. > > dst is the superclass to find. > > Performs a breadth-first search for dst, returning the shortest distance. > > """ Maybe the above could be inferred if this last line was changed to: Performs a breadth-first search for dst, either through all the bases of src (if a single class is given) or through the bases of each class given by a pair. Returns the minimal number of jumps required to reach a base class equal to dst. > > if type(srcs) != list: > > srcs = [(srcs, 0)] > > if len(srcs) == 0: > > return None > > Shouldn't that be an exception? It certainly could be. Does TypeError sound appropriate? > > here, distance = srcs[0] > > if here == dst: > > return distance > > newSrc = srcs[1:] + [(c, distance + 1) for c in list(here.__bases__)] > > return classDistance(newSrc, dst) > > What's classDistance? Oops - the old method name. Should be getclassdistance. Alexis -- Nova Engineer, HP Cloud. AKA lealexis, lxsli. From solipsis at pitrou.net Mon Jan 5 14:58:05 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 5 Jan 2015 14:58:05 +0100 Subject: [Python-ideas] inspect.getclassdistance References: <20150105112011.GB7357@hp.com> <20150105114315.GZ27426@ando.pearwood.info> <20150105134450.GC7357@hp.com> Message-ID: <20150105145805.5259c8da@fsol> On Mon, 5 Jan 2015 13:44:50 +0000 Alexis Lee <alexisl at hp.com> wrote: > > Maybe the above could be inferred if this last line was changed to: > > Performs a breadth-first search for dst, either through all the > bases of src (if a single class is given) or through the > bases of each class given by a pair. > Returns the minimal number of jumps required to reach a base class > equal to dst. Can you explain the use case? Regards Antoine. From alexisl at hp.com Mon Jan 5 15:05:20 2015 From: alexisl at hp.com (Alexis Lee) Date: Mon, 5 Jan 2015 14:05:20 +0000 Subject: [Python-ideas] inspect.getclassdistance In-Reply-To: <3D63844A-E1EA-4708-ADF0-8AB4D8453EDD@yahoo.com> References: <20150105112011.GB7357@hp.com> <3D63844A-E1EA-4708-ADF0-8AB4D8453EDD@yahoo.com> Message-ID: <20150105140520.GD7357@hp.com> Andrew Barnert said on Mon, Jan 05, 2015 at 01:09:37PM +0100: > On Jan 5, 2015, at 12:20, Alexis Lee <alexisl at hp.com> wrote: > You're trying to find out if dst is an ancestor of src (the option of > taking a list of pairs, despite being documented, seems to be there > only for internal use by recursive calls) and, if so, how long the > path is between them. All correct. (moved for clarity) > When would it ever be interesting? The usecase is here (comments at line 861, also see 866-894): https://review.openstack.org/#/c/142835/5/nova/api/openstack/compute/servers.py IE I receive a NeutronException of some specialised type; based on that there are multiple Nova exceptions I might want to throw. I would use getclassdistance (or rather, your walker) to sort those and throw the first, most-specific exception. EG I might be considering throwing either HTTPConflict or HTTPBadRequest, but the former is-a latter, hence it's more specific and the one which should be thrown. > * Only actual inheritance counts; abc.register or other subclass hooks are ignored. Never used abc; happy to enhance if the usecase is useful. > * In case of diamond inheritance where the sides are of unequal > length, the shorter one counts (even if that shorter path would never > be followed by the c3 rule and therefore isn't even relevant at > runtime). Yes. Although thinking about it, I actually want the longest path to count. Curses. This behaviour should be a toggle parameter. > * The actual classes on the path are irrelevant to you, you just want > its length. Yes. > * More generally, you don't care about the c3 MRO path, or the actual > path taken for any particular method's lookup, just the inheritance > path itself. Yes. MRO distance is trivial to obtain from __mro__. > Also, it seems like it would be simpler to just create a general > inheritance-BFS-walker function that yields each base as it goes, > which would allow you to write this function as a one-liner. Yes, that's fair. I'll wait for some more feedback on whether such a thing would be useful in the stdlib at all before writing that. > I also don't get why you're using recursion rather than a loop here, Just how I think. Alexis -- Nova Engineer, HP Cloud. AKA lealexis, lxsli. From p.f.moore at gmail.com Mon Jan 5 15:25:34 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 5 Jan 2015 14:25:34 +0000 Subject: [Python-ideas] inspect.getclassdistance In-Reply-To: <20150105140520.GD7357@hp.com> References: <20150105112011.GB7357@hp.com> <3D63844A-E1EA-4708-ADF0-8AB4D8453EDD@yahoo.com> <20150105140520.GD7357@hp.com> Message-ID: <CACac1F9fRtgyLyM-S3fkHWym2vv61sGSD0UMzazczKoJreo_bg@mail.gmail.com> On 5 January 2015 at 14:05, Alexis Lee <alexisl at hp.com> wrote: >> When would it ever be interesting? > > The usecase is here (comments at line 861, also see 866-894): > https://review.openstack.org/#/c/142835/5/nova/api/openstack/compute/servers.py Why is it not simply acceptable for you to use your code? I presume you are implying that this would be useful in the stdlib, but you'd need far more than a single use case to justify that. And you'd need to clarify why it would be useful to have it in 3.5+ and not in earlier versions - presumably the code you pointed to would need to support Python 3.4 at least, so a stdlib addition wouldn't be of any use unless there's a reasonable probability that new code would need something like this (and writing what seems to be a pretty trivial function isn't a better option). Paul From alexisl at hp.com Mon Jan 5 15:35:28 2015 From: alexisl at hp.com (Alexis Lee) Date: Mon, 5 Jan 2015 14:35:28 +0000 Subject: [Python-ideas] inspect.getclassdistance In-Reply-To: <CACac1F9fRtgyLyM-S3fkHWym2vv61sGSD0UMzazczKoJreo_bg@mail.gmail.com> References: <20150105112011.GB7357@hp.com> <3D63844A-E1EA-4708-ADF0-8AB4D8453EDD@yahoo.com> <20150105140520.GD7357@hp.com> <CACac1F9fRtgyLyM-S3fkHWym2vv61sGSD0UMzazczKoJreo_bg@mail.gmail.com> Message-ID: <20150105143528.GF7357@hp.com> Paul Moore said on Mon, Jan 05, 2015 at 02:25:34PM +0000: > On 5 January 2015 at 14:05, Alexis Lee <alexisl at hp.com> wrote: > >> When would it ever be interesting? > > > > The usecase is here (comments at line 861, also see 866-894): > > https://review.openstack.org/#/c/142835/5/nova/api/openstack/compute/servers.py > > Why is it not simply acceptable for you to use your code? I presume > you are implying that this would be useful in the stdlib, but you'd > need far more than a single use case to justify that. Well, that's it basically. I thought I'd offer it - if it isn't considered generally useful (and the reactions so far imply it isn't, tbh) it's no skin off my nose. I was just surprised it wasn't already available. Alexis -- Nova Engineer, HP Cloud. AKA lealexis, lxsli. From alexisl at hp.com Mon Jan 5 15:43:39 2015 From: alexisl at hp.com (Alexis Lee) Date: Mon, 5 Jan 2015 14:43:39 +0000 Subject: [Python-ideas] inspect.getclassdistance In-Reply-To: <CAPTjJmqkQq2rRYj+6ywyyQKVwMctUrr+4aSyH9TdLu8PpciT9A@mail.gmail.com> References: <20150105112011.GB7357@hp.com> <3D63844A-E1EA-4708-ADF0-8AB4D8453EDD@yahoo.com> <CAPTjJmqkQq2rRYj+6ywyyQKVwMctUrr+4aSyH9TdLu8PpciT9A@mail.gmail.com> Message-ID: <20150105144339.GG7357@hp.com> Chris Angelico said on Mon, Jan 05, 2015 at 11:30:28PM +1100: > def bfs_bases(cls): > """Yield the base which are belong to us""" > yield from cls.__bases__ > for cls in cls.__bases__: > yield from bfs_bases(cls) Much as I appreciate the pun, that isn't properly breadth-first, alas. This is and returns the classes in tranches by distance in case you care about the specific distance rather than a distance ordering: def bfs_bases2(cls): bases = [cls] while len(bases) > 0: yield bases bases = reduce(lambda x, y: x+y, (base.__bases__ for base in bases)) Alexis -- Nova Engineer, HP Cloud. AKA lealexis, lxsli. From rosuav at gmail.com Mon Jan 5 15:47:55 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 6 Jan 2015 01:47:55 +1100 Subject: [Python-ideas] inspect.getclassdistance In-Reply-To: <20150105144339.GG7357@hp.com> References: <20150105112011.GB7357@hp.com> <3D63844A-E1EA-4708-ADF0-8AB4D8453EDD@yahoo.com> <CAPTjJmqkQq2rRYj+6ywyyQKVwMctUrr+4aSyH9TdLu8PpciT9A@mail.gmail.com> <20150105144339.GG7357@hp.com> Message-ID: <CAPTjJmpwUr+y7jDvdZ-mosRY04coSuSh4Euw97NS8mMVvR_MRg@mail.gmail.com> On Tue, Jan 6, 2015 at 1:43 AM, Alexis Lee <alexisl at hp.com> wrote: > Chris Angelico said on Mon, Jan 05, 2015 at 11:30:28PM +1100: >> def bfs_bases(cls): >> """Yield the base which are belong to us""" >> yield from cls.__bases__ >> for cls in cls.__bases__: >> yield from bfs_bases(cls) > > Much as I appreciate the pun, that isn't properly breadth-first, alas. Hmm, good point. Once it's done one level (this class's bases), it then goes on to recurse depth-first. Whoops. Buggy. Your version's roughly as short, though, which was what I was aiming for - to point out how trivially simple this could be. ChrisA From skip.montanaro at gmail.com Mon Jan 5 15:49:07 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Mon, 5 Jan 2015 08:49:07 -0600 Subject: [Python-ideas] Encouraging more use of the Python wiki (was: Adding `pathlib.Path` method that would send file to recycle bin) In-Reply-To: <85387qyqas.fsf_-_@benfinney.id.au> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> Message-ID: <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> On Sun, Jan 4, 2015 at 7:22 PM, Ben Finney <ben+python at benfinney.id.au> wrote: > In recent years (I noticed this since about 2013), the Python wiki does > not offer an ?Edit? function when I am logged in; every page presents > ?Immutable Page? instead. Ben (and others), I just noticed this small paragraph on the FrontPage: If you want to edit a page and have just signed up, or you find that you can no longer edit a page that you could edit before, please write to the pydotorg-www mailing list, stating your account name and your intended edits and we'll add you to the EditorsGroup. I suspect that has something to do with your inability to edit. Somewhere along the way, someone must have made the executive decision that open editing no longer worked (too much spam?) and that editing needed to be a by-invitation or by-request sort of thing. I'd add you, but am getting nothing but Gateway Timeout responses at the moment. Skip From mal at egenix.com Mon Jan 5 16:27:59 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 05 Jan 2015 16:27:59 +0100 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> Message-ID: <54AAAD7F.6080201@egenix.com> On 05.01.2015 15:49, Skip Montanaro wrote: > On Sun, Jan 4, 2015 at 7:22 PM, Ben Finney <ben+python at benfinney.id.au> wrote: >> In recent years (I noticed this since about 2013), the Python wiki does >> not offer an ?Edit? function when I am logged in; every page presents >> ?Immutable Page? instead. > > Ben (and others), > > I just noticed this small paragraph on the FrontPage: > > If you want to edit a page and have just signed up, or you find that > you can no longer edit a page that you could edit before, please write > to the pydotorg-www mailing list, stating your account name and your > intended edits and we'll add you to the EditorsGroup. > > I suspect that has something to do with your inability to edit. > Somewhere along the way, someone must have made the executive decision > that open editing no longer worked (too much spam?) and that editing > needed to be a by-invitation or by-request sort of thing. This was discussed and announced on the pydotorg-www list at the time. The step was necessary, because of excessive spam and vandalim - editors were spending more time cleaning up than editing pages and were losing interest in the wiki altogether because of this. The first step was to require logins for editing. This worked for a (short) while. The second was adding more complicated textchas, the third step was requiring asking for editing rights on the pydotorg-www mailing list. Which is a little annoying and some work for the admins, but has resulted in the amount of spam to go down to zero. We usually grant the editing requests within a day. FWIW: I've maintained the wiki VM ever since it came under attack in 2013. > I'd add you, but am getting nothing but Gateway Timeout responses at the moment. If you get those, simply try to resend/reload. The wiki will notice duplicate editing requests. In my experience, the editing requests typically make it through to the wiki the first time and you get a duplicate editing warning on the second one. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 05 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From skip.montanaro at gmail.com Mon Jan 5 16:42:39 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Mon, 5 Jan 2015 09:42:39 -0600 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <54AAAD7F.6080201@egenix.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> Message-ID: <CANc-5UwKb9Zqy9gLuPFs5tVqfzBQRz2xqwMzvkAMDqWCH_c-TQ@mail.gmail.com> On Mon, Jan 5, 2015 at 9:27 AM, M.-A. Lemburg <mal at egenix.com> wrote: >> I'd add you, but am getting nothing but Gateway Timeout responses at the moment. > > If you get those, simply try to resend/reload. The wiki will notice > duplicate editing requests. In my experience, the editing requests > typically make it through to the wiki the first time and you get > a duplicate editing warning on the second one. Unfortunately, I couldn't even get to the EditorsGroup page to edit it. Now, about an hour later, I finally succeeded. Ben, send me your wiki login and I'll add you. MAL, I thought the spam problem had mostly been solved by Martin's modification several years ago to require a certain time to pass between a new login being created and their first attempt to edit. Skip From mal at egenix.com Mon Jan 5 17:05:27 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 05 Jan 2015 17:05:27 +0100 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <CANc-5UwKb9Zqy9gLuPFs5tVqfzBQRz2xqwMzvkAMDqWCH_c-TQ@mail.gmail.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <CANc-5UwKb9Zqy9gLuPFs5tVqfzBQRz2xqwMzvkAMDqWCH_c-TQ@mail.gmail.com> Message-ID: <54AAB647.6080507@egenix.com> On 05.01.2015 16:42, Skip Montanaro wrote: > On Mon, Jan 5, 2015 at 9:27 AM, M.-A. Lemburg <mal at egenix.com> wrote: >>> I'd add you, but am getting nothing but Gateway Timeout responses at the moment. >> >> If you get those, simply try to resend/reload. The wiki will notice >> duplicate editing requests. In my experience, the editing requests >> typically make it through to the wiki the first time and you get >> a duplicate editing warning on the second one. > > Unfortunately, I couldn't even get to the EditorsGroup page to edit > it. Now, about an hour later, I finally succeeded. Hmm, looks like the VM is overloaded again :-( > Ben, send me your wiki login and I'll add you. > > MAL, I thought the spam problem had mostly been solved by Martin's > modification several years ago to require a certain time to pass > between a new login being created and their first attempt to edit. I'm not aware of such a modification in the installation. In any case, the problem was real and the editors group solved it for good now :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 05 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From fetchinson at googlemail.com Mon Jan 5 17:28:37 2015 From: fetchinson at googlemail.com (Fetchinson .) Date: Mon, 5 Jan 2015 17:28:37 +0100 Subject: [Python-ideas] python on mobile In-Reply-To: <CAF7AXFFh0voFPLPtYxv0qC8yDXmxdBn5K-PT_5oZSXC8RT7jLg@mail.gmail.com> References: <CADjSo4TssZHGiOCm110co-BQxLAXZpCZbT1T=zdd8eBavnyVzQ@mail.gmail.com> <CAJxq848RyRhbeD4kHssFo1jHUtVX5N1_G=RB1rB+DkmDaxxA2A@mail.gmail.com> <CADiSq7enX0nmNd2AEHGwCgdDsgaXWzdz0A1w5iU_m1V+JdHe6g@mail.gmail.com> <CAJxq84-EeN0v49sHpPYZStJJ_615quqMYmmq2OPvhcaae6Hfiw@mail.gmail.com> <CADjSo4Seed-B0FmTUikrtvDux_LprdvedVrR1-RvQS8fPE=LiQ@mail.gmail.com> <87k31833vw.fsf@uwakimon.sk.tsukuba.ac.jp> <CAJxq848T09_c-jrHhtyyWUBFk7By7xtUphCExom7wNd36ZCtow@mail.gmail.com> <CADiSq7ewkZwRZhkeyuJC32KeaaPuq9FTxQXHiY5uhOZBA+Un=g@mail.gmail.com> <CAF7AXFFh0voFPLPtYxv0qC8yDXmxdBn5K-PT_5oZSXC8RT7jLg@mail.gmail.com> Message-ID: <CADjSo4QnhX4bSNuiBExNx7pMyffzBT6J0pCZGfvk-2cv_hgM1Q@mail.gmail.com> > I'd like to see a mobile-sig, because I think there's enough to > discuss that isn't relevant to -dev. I plan to get IronPython running > on Xamarin Android/iOS sometime this year so it would be helpful to > coordinate things with CPython and Jython - for example, what the > values will be for things like sys.platform and os.name will be, and > issues around file system emulation, supported modules, etc. that > should be compatible for all implementations. > > Plus there's the issue of new libraries for sensors, GPS, cameras, > etc. that provide Pythonic wrappers over the platform APIs. It's not > just "get CPython working on mobile" but "how should Python look & > feel on mobile devices". (Of course, CPython is special so it's > implementation details are still fair game.) > > So +1 on mobile-sig from me. I'll even volunteer to get it set up & > administer it +1 -- Psss, psss, put it down! - http://www.cafepress.com/putitdown From abarnert at yahoo.com Mon Jan 5 17:36:28 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 5 Jan 2015 17:36:28 +0100 Subject: [Python-ideas] inspect.getclassdistance In-Reply-To: <20150105140520.GD7357@hp.com> References: <20150105112011.GB7357@hp.com> <3D63844A-E1EA-4708-ADF0-8AB4D8453EDD@yahoo.com> <20150105140520.GD7357@hp.com> Message-ID: <A80F1440-13F9-40ED-8D96-0D9D1F789A24@yahoo.com> On Jan 5, 2015, at 15:05, Alexis Lee <alexisl at hp.com> wrote: > Andrew Barnert said on Mon, Jan 05, 2015 at 01:09:37PM +0100: >> On Jan 5, 2015, at 12:20, Alexis Lee <alexisl at hp.com> wrote: >> You're trying to find out if dst is an ancestor of src (the option of >> taking a list of pairs, despite being documented, seems to be there >> only for internal use by recursive calls) and, if so, how long the >> path is between them. > > All correct. > > (moved for clarity) >> When would it ever be interesting? > > The usecase is here (comments at line 861, also see 866-894): > https://review.openstack.org/#/c/142835/5/nova/api/openstack/compute/servers.py > > IE I receive a NeutronException of some specialised type; based on that > there are multiple Nova exceptions I might want to throw. I would use > getclassdistance (or rather, your walker) to sort those and throw the > first, most-specific exception. EG I might be considering throwing > either HTTPConflict or HTTPBadRequest, but the former is-a latter, hence > it's more specific and the one which should be thrown. I'm afraid I still don't understand the use case. To know that the former is-a latter, all you need is issubclass; what does the distance add? Maybe you're thinking that, in the case of two unrelated exceptions, the distance from some base class (like Exception) tells you which one is "most specific"? I don't think that's true in general (the hierarchy of Python's standard exceptions and other stdlib exceptions definitely hasn't been designed that way) but maybe your project designed its own exceptions that way on purpose. But if that's true, there ought to be a much simpler option: The "depth" of any exception class is 1 more than the depth of its deepest ancestor, with the root being 0. Do you actually have a use case where you need to know that exception class A is 2 levels deep from some arbitrary exception class B (as opposed to from the root)? >> I also don't get why you're using recursion rather than a loop here, > > Just how I think. Well, if you're you're going to suggest ideas for the Python stdlib, you may want to learn to think Pythonically. Not that recursion is never a good idea in Python, but deliberately using it in place of a for or while loop isn't. From steve at pearwood.info Mon Jan 5 18:10:55 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 6 Jan 2015 04:10:55 +1100 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <87lhlh2723.fsf@uwakimon.sk.tsukuba.ac.jp> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> <20150105005730.GR27426@ando.pearwood.info> <CAPTjJmozZ04qcrFLbGGrEWuC7kPDAr71g3aUWATc5tL9v4sX=g@mail.gmail.com> <20150105020735.GU27426@ando.pearwood.info> <87lhlh2723.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20150105171042.GA27426@ando.pearwood.info> On Mon, Jan 05, 2015 at 01:19:16PM +0900, Stephen J. Turnbull wrote: > Steven D'Aprano writes: > > > In any case, having a platform specific globbing module > > By "platform-specific", do you mean one module that handles all > platforms appropriately (what this thread is about), or do you mean > per-platform modules (what "platform-specific" means to me)? Yes to both. The model I am proposing is that of the os and os.path modules: we import os and it does the right thing for the current platform. In particular, os.path will do the right thing for the current platform, but you can import ntpath, posixpath, genericpath etc. to get behaviour for a specific platform. I think we have to leave the glob module as-is, for backwards compatibility, and add a new package, say globbing (or osglob, the name can be bike-shedded to death later *wink*) with the new behaviour. globbing will look something like this: globbing/ __init__.py posix.py nt.py generic.py (plus any other platforms that may need special treatment, if any) and the globbing/__init__.py implementation could potentially be as simple as "from foo import *" from the appropriate sub-module. The globbing.generic implementation might be as simple as: from glob import * Or perhaps the relationship should go the other way? Move the current glob.py implementation into globbing.generic, and replace it with: from globbing.generic import * These little details are to be decided later, but the aim is: * Current code that imports glob will continue to work the same as it does now, warts and all. * Code that imports globbing will do the right thing for the platform. * If you want to apply globs to something other than file names, the right module to use would probably be globbing.generic (or possible fnmatch directly). -- Steve From rosuav at gmail.com Mon Jan 5 18:14:38 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 6 Jan 2015 04:14:38 +1100 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <20150105171042.GA27426@ando.pearwood.info> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> <20150105005730.GR27426@ando.pearwood.info> <CAPTjJmozZ04qcrFLbGGrEWuC7kPDAr71g3aUWATc5tL9v4sX=g@mail.gmail.com> <20150105020735.GU27426@ando.pearwood.info> <87lhlh2723.fsf@uwakimon.sk.tsukuba.ac.jp> <20150105171042.GA27426@ando.pearwood.info> Message-ID: <CAPTjJmpwgiJ2Y2JsWz66EiGKBeQj0VvJSsNyZMqU5AJrEONxgA@mail.gmail.com> On Tue, Jan 6, 2015 at 4:10 AM, Steven D'Aprano <steve at pearwood.info> wrote: > * If you want to apply globs to something other than file names, > the right module to use would probably be globbing.generic > (or possible fnmatch directly). To clarify: Do you mean "something other than names of currently-existing files", or "something other than valid names for the local file system", or something else? For instance, suppose you write an unarchiver (in case we don't have enough of them already, of course), and you can "python unarchive.py archivename *.txt" to extract all files matching *.txt. The globbing would be done against some sort of internal index, but the names would have to be valid for the local file system, or you wouldn't be able to create them. Which module should you use? ChrisA From turnbull at sk.tsukuba.ac.jp Mon Jan 5 19:09:39 2015 From: turnbull at sk.tsukuba.ac.jp (Stephen J. Turnbull) Date: Tue, 06 Jan 2015 03:09:39 +0900 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <20150105171042.GA27426@ando.pearwood.info> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> <20150105005730.GR27426@ando.pearwood.info> <CAPTjJmozZ04qcrFLbGGrEWuC7kPDAr71g3aUWATc5tL9v4sX=g@mail.gmail.com> <20150105020735.GU27426@ando.pearwood.info> <87lhlh2723.fsf@uwakimon.sk.tsukuba.ac.jp> <20150105171042.GA27426@ando.pearwood.info> Message-ID: <87fvbp14m4.fsf@uwakimon.sk.tsukuba.ac.jp> Steven D'Aprano writes: > The model I am proposing is that of the os and os.path modules: we > import os and it does the right thing for the current platform. Which is normally "nothing" on POSIX, since the shell does it for you. Or, if you're talking about what the shell does for you, while I suppose there is a basic globbing defined by POSIX, but bash and zsh go well beyond that, and they behave somewhat differently in corner cases IIRC. If the program is going to invoke a globbing function specifically, I suppose it makes sense to default to the same behavior as on the current platform, but it's not obvious that that's what the user wants. I use bash on Windows; why settle for cmd.exe if you want the power of the command line? It could even cause problems (is there a reliable way to determine what the shell is, let alone whether it implements globbing or not, on Windows?) > In particular, os.path will do the right thing for the current > platform, but you can import ntpath, posixpath, genericpath etc. to > get behaviour for a specific platform. But there *is* a RightThang for paths; the OS decides. Globbing is a user preference thing. From skip.montanaro at gmail.com Mon Jan 5 19:45:38 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Mon, 5 Jan 2015 12:45:38 -0600 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <87fvbp14m4.fsf@uwakimon.sk.tsukuba.ac.jp> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> <20150105005730.GR27426@ando.pearwood.info> <CAPTjJmozZ04qcrFLbGGrEWuC7kPDAr71g3aUWATc5tL9v4sX=g@mail.gmail.com> <20150105020735.GU27426@ando.pearwood.info> <87lhlh2723.fsf@uwakimon.sk.tsukuba.ac.jp> <20150105171042.GA27426@ando.pearwood.info> <87fvbp14m4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <CANc-5UyqrtrSj4wmvUHSP2OHQBLyU_CnASEEDxPd+wYafRXVPA@mail.gmail.com> On Mon, Jan 5, 2015 at 12:09 PM, Stephen J. Turnbull < turnbull at sk.tsukuba.ac.jp> wrote: > Which is normally "nothing" on POSIX, since the shell does it for > you. Or, if you're talking about what the shell does for you, while I > suppose there is a basic globbing defined by POSIX, but bash and zsh > go well beyond that, and they behave somewhat differently in corner > cases IIRC. > Indeed. I only have 2.7.2 available here at work. Here's what bash tells me on a Linux box: % ls yen{2,3}.* yen2.out yen2.png yen2.trade yen3.out yen3.png yen3.trade % ls yen[23].* yen2.out yen2.png yen2.trade yen3.out yen3.png yen3.trade Here's what /bin/sh tells me on a Solaris 10 box: $ ls yen{2,3}.* yen{2,3}.*: No such file or directory $ ls yen[23].* yen2.out yen2.png yen2.trade yen3.out yen3.png yen3.trade Here's what the glob module tells me: % python Python 2.7.2 (default, Nov 14 2012, 05:07:35) [GCC 4.4.6 [TWW]] on linux3 Type "help", "copyright", "credits" or "license" for more information. >>> import glob >>> glob.glob("yen{2,3}.*") [] >>> glob.glob("yen[23].*") ['yen3.trade', 'yen2.out', 'yen2.trade', 'yen3.out', 'yen3.png', 'yen2.png'] I only discovered this "shortcoming" (or "Bourne Shell dependency") relatively recently. I've been using bash for so long it never even occurred to me that {...} notation wasn't available in all shells. Skip -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150105/b3812284/attachment-0001.html> From random832 at fastmail.us Mon Jan 5 21:33:08 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Mon, 05 Jan 2015 15:33:08 -0500 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <87lhlh2723.fsf@uwakimon.sk.tsukuba.ac.jp> References: <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> <20150105005730.GR27426@ando.pearwood.info> <CAPTjJmozZ04qcrFLbGGrEWuC7kPDAr71g3aUWATc5tL9v4sX=g@mail.gmail.com> <20150105020735.GU27426@ando.pearwood.info> <87lhlh2723.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <1420489988.4157893.209883497.15AB7246@webmail.messagingengine.com> On Sun, Jan 4, 2015, at 23:19, Stephen J. Turnbull wrote: > By "platform-specific", do you mean one module that handles all > platforms appropriately (what this thread is about), or do you mean > per-platform modules (what "platform-specific" means to me)? Windows technically requires calls to the actual filesystem driver for correct behavior - different filesystems may have different case folding, theoretically other different rules, etc. Plus you've got to examine both the long and short filenames - if you have foobar.html, *.htm will match it because the short filename is FOOBAR~1.HTM (it, however, _returns_ the long filename.) This means that the equivalent function to glob can't simply be the equivalent of listdir filtered by fnmatch. On Mon, Jan 5, 2015, at 05:10, Andrew Barnert wrote: > It seems to me that if you want auto-globbing support on Windows, the > "right" way to do it is to link in the fancy setargv instead of the > default one. This handles wildcards exactly the same way the > command.com/cmd.exe builtin commands do, which I suspect is what Windows > users would actually be expecting of they passed a wildcard on the > command line. (I think the docs no longer guarantee that this is true, > but it's probably still true, and certainly closer to true than if you > try to do it manually.) The problem with that is that if you do that you can no longer pass in _non_-filename arguments that contain a question mark or asterisk (and happen to match a file). Better to do it inside the program, when you actually know the argument you're looking at is intended to be a filename spec. Which, I assume, is _why_ it's not done by default when you compile a C program. A program like grep "a.*b" "Some Directory\*.txt" should handle wildcards in the filename and not in the regex. That is the user expectation on windows, and that is something that windows programs (and not unix programs) can and do deliver. Setargv doesn't, which is why you have to opt in to it (if you know your program doesn't need non-filename arguments that look like wildcards). It should be possible to write Python programs that can do the same, in a way that is relatively painless. A setargv-like solution is insufficient for that. On Mon, Jan 5, 2015, at 13:45, Skip Montanaro wrote: > I only discovered this "shortcoming" (or "Bourne Shell dependency") > relatively recently. I've been using bash for so long it never even > occurred to me that {...} notation wasn't available in all shells. Brace expansion is technically not globbing. yen{2,3}.txt will return yen2.txt yen3.txt even if one or both of those files don't exist. The reason it works with globs is that yen{2,3}.* literally expands to yen2.* yen3.* which is then globbed. From rosuav at gmail.com Mon Jan 5 22:33:41 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 6 Jan 2015 08:33:41 +1100 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <1420489988.4157893.209883497.15AB7246@webmail.messagingengine.com> References: <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> <20150105005730.GR27426@ando.pearwood.info> <CAPTjJmozZ04qcrFLbGGrEWuC7kPDAr71g3aUWATc5tL9v4sX=g@mail.gmail.com> <20150105020735.GU27426@ando.pearwood.info> <87lhlh2723.fsf@uwakimon.sk.tsukuba.ac.jp> <1420489988.4157893.209883497.15AB7246@webmail.messagingengine.com> Message-ID: <CAPTjJmpOVP0++L+aYJCaLgwvuwSrHQurnQDUb6d+i49eOJ5CQQ@mail.gmail.com> On Tue, Jan 6, 2015 at 7:33 AM, <random832 at fastmail.us> wrote: > On Mon, Jan 5, 2015, at 13:45, Skip Montanaro wrote: >> I only discovered this "shortcoming" (or "Bourne Shell dependency") >> relatively recently. I've been using bash for so long it never even >> occurred to me that {...} notation wasn't available in all shells. > > Brace expansion is technically not globbing. yen{2,3}.txt will return > yen2.txt yen3.txt even if one or both of those files don't exist. The > reason it works with globs is that yen{2,3}.* literally expands to > yen2.* yen3.* which is then globbed. Which, by the way, is the main reason I use brace expansion. Type "mv" and then fill in a file name using tab completion, then insert or delete something by editing it to have braces and a comma: mv some-long-file-name-blah-blah-blah mv some-{long-,}file-name-blah-blah-blah Et voila! Remove one word from a tab-completed file name. There's no way that a generic glob expansion tool can do everything that bash does, and I doubt anyone expects it. ChrisA From p.f.moore at gmail.com Mon Jan 5 23:20:49 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 5 Jan 2015 22:20:49 +0000 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <1420489988.4157893.209883497.15AB7246@webmail.messagingengine.com> References: <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> <20150105005730.GR27426@ando.pearwood.info> <CAPTjJmozZ04qcrFLbGGrEWuC7kPDAr71g3aUWATc5tL9v4sX=g@mail.gmail.com> <20150105020735.GU27426@ando.pearwood.info> <87lhlh2723.fsf@uwakimon.sk.tsukuba.ac.jp> <1420489988.4157893.209883497.15AB7246@webmail.messagingengine.com> Message-ID: <CACac1F8GrRiEreksxX79QZrs8PaLCE-Y2dFOb0sY44HREnGKSQ@mail.gmail.com> On 5 January 2015 at 20:33, <random832 at fastmail.us> wrote: > On Mon, Jan 5, 2015, at 05:10, Andrew Barnert wrote: >> It seems to me that if you want auto-globbing support on Windows, the >> "right" way to do it is to link in the fancy setargv instead of the >> default one. This handles wildcards exactly the same way the >> command.com/cmd.exe builtin commands do, which I suspect is what Windows >> users would actually be expecting of they passed a wildcard on the >> command line. (I think the docs no longer guarantee that this is true, >> but it's probably still true, and certainly closer to true than if you >> try to do it manually.) > > The problem with that is that if you do that you can no longer pass in > _non_-filename arguments that contain a question mark or asterisk (and > happen to match a file). Better to do it inside the program, when you > actually know the argument you're looking at is intended to be a > filename spec. Which, I assume, is _why_ it's not done by default when > you compile a C program. Just as a note - I find the fact that Python *doesn't* allow the C runtime to do its glob-mangling of the supplied command arguments a very useful feature, and I would not want it to be "fixed". The setargv behaviour in the MSVC runtime is full of arcane corner cases and quoting gotchas, and I wouldn't want it imposed on me by default. Paul From g.brandl at gmx.net Tue Jan 6 00:40:19 2015 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 06 Jan 2015 00:40:19 +0100 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <CANc-5UyqrtrSj4wmvUHSP2OHQBLyU_CnASEEDxPd+wYafRXVPA@mail.gmail.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> <20150105005730.GR27426@ando.pearwood.info> <CAPTjJmozZ04qcrFLbGGrEWuC7kPDAr71g3aUWATc5tL9v4sX=g@mail.gmail.com> <20150105020735.GU27426@ando.pearwood.info> <87lhlh2723.fsf@uwakimon.sk.tsukuba.ac.jp> <20150105171042.GA27426@ando.pearwood.info> <87fvbp14m4.fsf@uwakimon.sk.tsukuba.ac.jp> <CANc-5UyqrtrSj4wmvUHSP2OHQBLyU_CnASEEDxPd+wYafRXVPA@mail.gmail.com> Message-ID: <m8f7d4$ik$1@ger.gmane.org> On 01/05/2015 07:45 PM, Skip Montanaro wrote: > Indeed. I only have 2.7.2 available here at work. Here's what bash tells me on a > Linux box: > > % ls yen{2,3}.* > yen2.out yen2.png yen2.tradeyen3.out yen3.png yen3.trade > % ls yen[23].* > yen2.out yen2.png yen2.tradeyen3.out yen3.png yen3.trade [...] > Here's what the glob module tells me: > > % python > Python 2.7.2 (default, Nov 14 2012, 05:07:35) > [GCC 4.4.6 [TWW]] on linux3 > Type "help", "copyright", "credits" or "license" for more information. >>>> import glob >>>> glob.glob("yen{2,3}.*") > [] >>>> glob.glob("yen[23].*") > ['yen3.trade', 'yen2.out', 'yen2.trade', 'yen3.out', 'yen3.png', 'yen2.png'] > > I only discovered this "shortcoming" (or "Bourne Shell dependency") relatively > recently. I've been using bash for so long it never even occurred to me that > {...} notation wasn't available in all shells. Note that the {...} notation is not a part of globbing, it's a different mechanism (bash calls it brace expansion IIRC). With brace expansion, the different choices are *always* expanded regardless of the existence of matching filenames. Your pattern first gets expanded to "yen2.* yen3.*" and then globbing ensues (with the standard rule that if there is nothing matching e.g. yen2.* it is either given literally to the program or the command line is rejected, depending on the shell). => I wouldn't expect the Python glob module to perform brace expansion. cheers, Georg From steve at pearwood.info Tue Jan 6 00:57:10 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 6 Jan 2015 10:57:10 +1100 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <CAPTjJmpwgiJ2Y2JsWz66EiGKBeQj0VvJSsNyZMqU5AJrEONxgA@mail.gmail.com> References: <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> <20150105005730.GR27426@ando.pearwood.info> <CAPTjJmozZ04qcrFLbGGrEWuC7kPDAr71g3aUWATc5tL9v4sX=g@mail.gmail.com> <20150105020735.GU27426@ando.pearwood.info> <87lhlh2723.fsf@uwakimon.sk.tsukuba.ac.jp> <20150105171042.GA27426@ando.pearwood.info> <CAPTjJmpwgiJ2Y2JsWz66EiGKBeQj0VvJSsNyZMqU5AJrEONxgA@mail.gmail.com> Message-ID: <20150105235710.GB27426@ando.pearwood.info> On Tue, Jan 06, 2015 at 04:14:38AM +1100, Chris Angelico wrote: > On Tue, Jan 6, 2015 at 4:10 AM, Steven D'Aprano <steve at pearwood.info> wrote: > > * If you want to apply globs to something other than file names, > > the right module to use would probably be globbing.generic > > (or possible fnmatch directly). > > To clarify: Do you mean "something other than names of > currently-existing files", or "something other than valid names for > the local file system", or something else? For instance, suppose you > write an unarchiver (in case we don't have enough of them already, of > course), and you can "python unarchive.py archivename *.txt" to > extract all files matching *.txt. The globbing would be done against > some sort of internal index, but the names would have to be valid for > the local file system, or you wouldn't be able to create them. Which > module should you use? fnmatch. I've had a look inside glob.py and it calls os.listdir directly, so you cannot use glob to match things other than actual existing files. (Well, I suppose you could if you monkey-patched the module, but lets not go there.) fnmatch, on the other hand, provides the right tool for matching names other than actual file names: fnmatch.filter(names, pattern). (Well, almost the right tool -- it has a few issues too.) To summarise: - the glob module returns only actual files in the file system; - the new globbing package will do the same, except it will use platform-specific wildcards where possible; - fnmatch continues as Python's generic "match globs against arbitrary strings" module (despite the name). -- Steven From steve at pearwood.info Tue Jan 6 01:14:09 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 6 Jan 2015 11:14:09 +1100 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <87fvbp14m4.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> <20150105005730.GR27426@ando.pearwood.info> <CAPTjJmozZ04qcrFLbGGrEWuC7kPDAr71g3aUWATc5tL9v4sX=g@mail.gmail.com> <20150105020735.GU27426@ando.pearwood.info> <87lhlh2723.fsf@uwakimon.sk.tsukuba.ac.jp> <20150105171042.GA27426@ando.pearwood.info> <87fvbp14m4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20150106001408.GC27426@ando.pearwood.info> On Tue, Jan 06, 2015 at 03:09:39AM +0900, Stephen J. Turnbull wrote: > Steven D'Aprano writes: > > > The model I am proposing is that of the os and os.path modules: we > > import os and it does the right thing for the current platform. > > Which is normally "nothing" on POSIX, since the shell does it for > you. Or, if you're talking about what the shell does for you, while I > suppose there is a basic globbing defined by POSIX, but bash and zsh > go well beyond that, and they behave somewhat differently in corner > cases IIRC. All correct. > If the program is going to invoke a globbing function specifically, I > suppose it makes sense to default to the same behavior as on the > current platform, but it's not obvious that that's what the user > wants. I use bash on Windows; why settle for cmd.exe if you want the > power of the command line? It could even cause problems (is there a > reliable way to determine what the shell is, let alone whether it > implements globbing or not, on Windows?) I see that as an application issue, not a Python issue. As the author of the application, I have some flexibility: - Currently, I can not use glob.py at all, and my Unix users will be able to use globs, but most of my Windows users won't. - If I unconditionally use glob.py, my Unix users can not reliably specify filenames containing wildcards but my Windows users may (assuming that glob supports escaping, which it currently doesn't). - I can conditionally use glob or not, depending on the platform. - Or I can provide an application switch/preference which allows the user to decide whether to use wildcards or not. I don't know of any reliable way for a Python script to determine the current shell (there may not even be a shell!). If such a way exists, that would be good functionality to have. > > In particular, os.path will do the right thing for the current > > platform, but you can import ntpath, posixpath, genericpath etc. to > > get behaviour for a specific platform. > > But there *is* a RightThang for paths; the OS decides. Globbing is a > user preference thing. Not entirely user-preference. As Random's discussion has pointed out, there are issues with the way globbing is currently handled. For example, Windows doesn't support [...] wildcards and so most Windows users will expect to be able to specify [ and ] as ordinary characters. You can't do that with glob.py. Windows users who have bash installed are a very small minority :-) -- Steven From rosuav at gmail.com Tue Jan 6 01:25:37 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 6 Jan 2015 11:25:37 +1100 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <20150105235710.GB27426@ando.pearwood.info> References: <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> <20150105005730.GR27426@ando.pearwood.info> <CAPTjJmozZ04qcrFLbGGrEWuC7kPDAr71g3aUWATc5tL9v4sX=g@mail.gmail.com> <20150105020735.GU27426@ando.pearwood.info> <87lhlh2723.fsf@uwakimon.sk.tsukuba.ac.jp> <20150105171042.GA27426@ando.pearwood.info> <CAPTjJmpwgiJ2Y2JsWz66EiGKBeQj0VvJSsNyZMqU5AJrEONxgA@mail.gmail.com> <20150105235710.GB27426@ando.pearwood.info> Message-ID: <CAPTjJmppdc4Fw=o2RHSxmnJQmcVgUs2Q6eYdEzzdLfhEEDezGA@mail.gmail.com> On Tue, Jan 6, 2015 at 10:57 AM, Steven D'Aprano <steve at pearwood.info> wrote: > On Tue, Jan 06, 2015 at 04:14:38AM +1100, Chris Angelico wrote: >> On Tue, Jan 6, 2015 at 4:10 AM, Steven D'Aprano <steve at pearwood.info> wrote: >> > * If you want to apply globs to something other than file names, >> > the right module to use would probably be globbing.generic >> > (or possible fnmatch directly). >> >> Which module should you use [to match non-existing files]? > > fnmatch. I've had a look inside glob.py and it calls os.listdir > directly, so you cannot use glob to match things other than actual > existing files. (Well, I suppose you could if you monkey-patched the > module, but lets not go there.) > > fnmatch, on the other hand, provides the right tool for matching names > other than actual file names: fnmatch.filter(names, pattern). (Well, > almost the right tool -- it has a few issues too.) Ugh. That's poor naming, then. If this new globbing module happens, I would be a strong +1 on having 'globbing.generic' (or somesuch) to do this kind of thing, if only to fix the name. ChrisA From yawar.amin at gmail.com Tue Jan 6 02:59:20 2015 From: yawar.amin at gmail.com (Yawar Amin) Date: Mon, 05 Jan 2015 20:59:20 -0500 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> Message-ID: <54AB4178.3050508@gmail.com> On 2015-01-05 04:49, Andrew Barnert wrote: > ... paren continuation just concatenates the lines together. I'll quote this first, because it's the crux of the matter and what I failed to understand up until now. IIUC now, Python never actually 'relaxes' any indentation rules--a preprocessor just concatenates split lines (i.e. lines split with parens etc.) and then passes the result on to the parser? I guess pretty much everything else is made redundant by this one point. > ... I'm not sure what the return adds to it that you couldn't get just > from brace syntax. I was trying to avoid brace syntax as un-Pythonic. > You also seem to have invented a rule that a sequence of statements > has the value of the last statement as its value. Which requires first > inventing a rule that gives values to statements. I'll assume you > wanted to go with the same rule the interactive interpreter uses to > display a last value--an expression statement has its expression's > value, and any other kind of statement has None?) No, I was trying to say that whatever's inside the 'return: ...' block is evaluated, and then the last expression inside the block becomes the value of the block as a whole. No change would be required to any existing expressions or statements, or to the result of a normal sequence of statements. > Finally, are you sure your new return: isn't ambiguous without look > ahead or context? When the parser reads "if spam: return", is it > starting a return simple statement, or an expression statement that > starts with a return expression? Have you tried actually writing out > the grammar? Having the keyword be 'return' wasn't important to me; it could just as easily be a new one like 'do' or 'begin'. It's back to the drawing board for me, I guess :-) Regards, Yawar -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 834 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150105/a4f8c0c6/attachment.sig> From ncoghlan at gmail.com Tue Jan 6 05:36:01 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 6 Jan 2015 14:36:01 +1000 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <20150105011312.GS27426@ando.pearwood.info> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> Message-ID: <CADiSq7exQGaiNVVdCajjQBZ0_VjMwMjFQvG-E3UyFt2RD3xxrQ@mail.gmail.com> On 5 January 2015 at 11:13, Steven D'Aprano <steve at pearwood.info> wrote: > On Sun, Jan 04, 2015 at 11:01:08PM +1000, Nick Coghlan wrote: > >> For those curious as to "Why not the wiki?", a Sphinx project hosted >> on a pull request capable service like GitHub, BitBucket or Kallithea >> offers a much nicer workflow for reviewing of proposed changes, >> together with an integrated issue tracker for submitting proposals for >> updates (https://github.com/pypa/python-packaging-user-guide/ is the >> project behind packaging.python.org, for example). > > The concept of "proposed changes" goes completely against the grain of > community-managed content. Imagine if Wikipedia required you to make > pull requests. Wikipedia's typical equivalent of pull requests is when editors revert a page to an earlier version and move the discussion of the proposed change to an earlier version. This approach can be escalated to *actual* pull requests when the editors lock a page to disrupt an ongoing edit war. The "anyone can publish by default" approach has the advantage of significantly increasing editing throughput by streamlining the handling of non-controversial cases. The downside as a reader is that it achieves this by allowing a brief window where controversial changes can be published without first establishing consensus, which means you may be presented with inaccurate information on controversial topics depending on when you check a page. I think that model works well for the task of creating a collaborative encyclopaedia (where "eventually accurate" is good enough for almost all purposes, and lowering barriers to entry for casual contribution of corrections is a high priority), but I don't believe it's appropriate for the task of delegating the CPython core development team's reputation and authority to other groups. For that, a pre-commit review process, or a more explicit topic area delegation (like the Python Packaging Authority handling packaging.python.org), makes more sense to me: if we trust folks to maintain a module or topic area *in* the standard library, we should be able to trust them to provide reasonable and balanced recommendations regarding applications and libraries that are maintained *outside* the standard library. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Jan 6 05:43:33 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 6 Jan 2015 14:43:33 +1000 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <20150105103828.GA22528@k3> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20150105103828.GA22528@k3> Message-ID: <CADiSq7fLPxvBvfek1yTJE53bwhXsVfEpBrArRU1eC_0DFqMwHA@mail.gmail.com> On 5 January 2015 at 20:38, David Wilson <dw+python-ideas at hmmz.org> wrote: > I have no problem with this being included somehow in the standard > library, but pathlib to me is all about the filesystem, whereas (as > others have pointed out), the Recycle Bin and associated concepts on > other OS relate pretty uniformly to the OS shell. > > There are many shell-related concepts that could be nicely exposed, but > again I don't think any of them belong in pathlib. For example, > labels/tags, owner application, icon, visibility flags, etc. Better tools for accessing typical extended file metadata would indeed be highly desirable (thumbnails/previews are another one that comes to mind). As with the recycle bin though, the generally preferred approach for standard library additions is incorporating existing modules that have been available through PyPI for some time, or else creating a derivative of such a project that meets other constraints of standard library inclusion (e.g. ipaddress vs the project it was based on, ipaddr). Unfortunately, most of the folks that need these cross-platform integration features are likely to be using a C/C++ toolkit that provides them (e.g. Gtk, Qt, wxWidgets), which significantly reduces the incentive for anyone to create a Python-specific abstraction that would be suitable for eventual inclusion in the standard library. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ben+python at benfinney.id.au Tue Jan 6 07:26:52 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 06 Jan 2015 17:26:52 +1100 Subject: [Python-ideas] Encouraging more use of the Python wiki References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> Message-ID: <85tx04xw43.fsf@benfinney.id.au> "M.-A. Lemburg" <mal at egenix.com> writes: > [editing the Python wiki] requiring asking for editing rights > on the pydotorg-www mailing list. Which is a little annoying and > some work for the admins, but has resulted in the amount of > spam to go down to zero. We usually grant the editing requests > within a day. I quite understand the motivation for this. It does need to be recognised that this will dramatically reduce the number of pople who will pass that hurdle to edit the wiki, and thus the wiki loses a lot of edits (both helpful and unhelpful) it might otherwise have. Thanks to those doing the work to maintain Python community infrastructure. -- \ ?If you can do no good, at least do no harm.? ?_Slapstick_, | `\ Kurt Vonnegut | _o__) | Ben Finney From ethan at stoneleaf.us Tue Jan 6 07:34:02 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 05 Jan 2015 22:34:02 -0800 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <85tx04xw43.fsf@benfinney.id.au> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> Message-ID: <54AB81DA.7050502@stoneleaf.us> On 01/05/2015 10:26 PM, Ben Finney wrote: > "M.-A. Lemburg" writes: >> [editing the Python wiki] requiring asking for editing rights >> on the pydotorg-www mailing list. Which is a little annoying and >> some work for the admins, but has resulted in the amount of >> spam to go down to zero. We usually grant the editing requests >> within a day. > It does need to be recognised that this will dramatically reduce the > number of pople who will pass that hurdle to edit the wiki, and thus the > wiki loses a lot of edits (both helpful and unhelpful) it might > otherwise have. At least the edits being made now are by folks who care. Better fewer edits than a site full of little but spam. > Thanks to those doing the work to maintain Python community > infrastructure. Seconded. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150105/e11e66e4/attachment.sig> From abarnert at yahoo.com Tue Jan 6 09:25:25 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 6 Jan 2015 09:25:25 +0100 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <54AB4178.3050508@gmail.com> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> <54AB4178.3050508@gmail.com> Message-ID: <97D50920-B7D2-4125-9389-224C4CBC83CA@yahoo.com> On Jan 6, 2015, at 2:59, Yawar Amin <yawar.amin at gmail.com> wrote: > On 2015-01-05 04:49, Andrew Barnert wrote: > >> You also seem to have invented a rule that a sequence of statements >> has the value of the last statement as its value. Which requires first >> inventing a rule that gives values to statements. I'll assume you >> wanted to go with the same rule the interactive interpreter uses to >> display a last value--an expression statement has its expression's >> value, and any other kind of statement has None?) > > No, I was trying to say that whatever's inside the 'return: ...' block > is evaluated, and then the last expression inside the block becomes the > value of the block as a whole. No change would be required to any > existing expressions or statements, or to the result of a normal > sequence of statements. I think you're missing another important point here: statements and expressions are different things. Blocks are made up of statements, not expressions, and statements don't have values. This is different from many other languages like C, Ruby, or JavaScript, where as many things as possible are expressions (and therefore have values), so a block is (oversimplifying a bit) just a sequence of expressions separated by semicolons and stuck inside braces, and therefore it has a last value. One type of statement, the expression statement, is just an expression on a line by itself (or between semicolons), so you _could_ invent a rule pretty easily that expression statements have the value of their expression, and all other statements have a value of None, and a block has the value of its last statement. This is basically the same rule used by the interactive REPL for displaying the repr of what you've typed and setting the _ variable, so it wouldn't be a huge stretch to add the same rule to the language itself--but it would still be a new rule, and a significant change to the internals of the interpreter, even though it would only be useful in this new context. From mal at egenix.com Tue Jan 6 09:31:50 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 06 Jan 2015 09:31:50 +0100 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <85tx04xw43.fsf@benfinney.id.au> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> Message-ID: <54AB9D76.7030105@egenix.com> On 06.01.2015 07:26, Ben Finney wrote: > "M.-A. Lemburg" <mal at egenix.com> writes: > >> [editing the Python wiki] requiring asking for editing rights >> on the pydotorg-www mailing list. Which is a little annoying and >> some work for the admins, but has resulted in the amount of >> spam to go down to zero. We usually grant the editing requests >> within a day. > > I quite understand the motivation for this. > > It does need to be recognised that this will dramatically reduce the > number of pople who will pass that hurdle to edit the wiki, and thus the > wiki loses a lot of edits (both helpful and unhelpful) it might > otherwise have. We might lose some occasional quick typo fixes, but it seems that those genuinely interested in helping with the wiki don't have a problem asking on the list. To reduce the problem I had created a list of user names who had made good edits in the months before the change. The only nit I still have with the setup is that it's not necessarily apparent for new users how to get editing rights. There's a section on the front page, but you don't see this when looking at a random page. > Thanks to those doing the work to maintain Python community > infrastructure. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 06 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From abarnert at yahoo.com Tue Jan 6 10:29:32 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 6 Jan 2015 10:29:32 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <CADiSq7fLPxvBvfek1yTJE53bwhXsVfEpBrArRU1eC_0DFqMwHA@mail.gmail.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20150105103828.GA22528@k3> <CADiSq7fLPxvBvfek1yTJE53bwhXsVfEpBrArRU1eC_0DFqMwHA@mail.gmail.com> Message-ID: <9A57680C-9629-4A22-91FF-E0625EA95457@yahoo.com> On Jan 6, 2015, at 5:43, Nick Coghlan <ncoghlan at gmail.com> wrote: > On 5 January 2015 at 20:38, David Wilson <dw+python-ideas at hmmz.org> wrote: >> I have no problem with this being included somehow in the standard >> library, but pathlib to me is all about the filesystem, whereas (as >> others have pointed out), the Recycle Bin and associated concepts on >> other OS relate pretty uniformly to the OS shell. >> >> There are many shell-related concepts that could be nicely exposed, but >> again I don't think any of them belong in pathlib. For example, >> labels/tags, owner application, icon, visibility flags, etc. > > Better tools for accessing typical extended file metadata would indeed > be highly desirable (thumbnails/previews are another one that comes to > mind). As with the recycle bin though, the generally preferred > approach for standard library additions is incorporating existing > modules that have been available through PyPI for some time, or else > creating a derivative of such a project that meets other constraints > of standard library inclusion (e.g. ipaddress vs the project it was > based on, ipaddr). > > Unfortunately, most of the folks that need these cross-platform > integration features are likely to be using a C/C++ toolkit that > provides them (e.g. Gtk, Qt, wxWidgets), which significantly reduces > the incentive for anyone to create a Python-specific abstraction that > would be suitable for eventual inclusion in the standard library. I'm not sure that's really unfortunate. First, a framework like Qt or wx is maintained by people who think about nothing but desktop and GUI integration on the major platforms. They track changes pretty quickly (especially compared to the Python stdlib). A few years back, when Apple got rid of legacy-style preference files, I had a Qt app that I was able to fix just by updating to the next version of Qt, and a non-Qt app that I had to fix by reseatching, writing, and testing, a wrapper that "temporarily" (of course it's still there in 2015...) stashes the entire legacy file as an encoded string in the new-style prefs database. What are the odds a Python stdlib solution (or even a third-party Python lib with less than 1/20th the user base, community, and corporate support) would have fixed it for me automatically in time for my next release the way Qt did? Also, the fact that Qt and wx are complete frameworks means they can make assumptions that allow for simpler solutions. A Qt library knows how to check whether there's a runloop and, if there is, knows it'll be a Qt runloop, and, if not, knows how to create a short-term blocking runloop and background thread, and it also knows how to pass signals from background threads to the runloop, and so on; this means Qt could just use the (10.5+, and still not deprecated as of 10.10) NSWorkspace API for trashing on Mac, because none of its deficiencies apply to any Qt app. A library meant to be used generally across all kinds of apps can't do that. Obviously this is also a weakness, because not all cross-platform desktop-integration apps really need a full framework? but many do. And again, many simple tools don't need to be cross-platform (like my completely separate 3-line trash tools for each platform that I use), or don't need full generality (my Mac trash tool doesn't need to work on 10.6, or to be App Store compliant; my Ubuntu trash tool doesn't have to know how Unity deals with trashing over CIFS because I only have two CIFS mounts and they both do server-side automatic recycling; etc.). From ncoghlan at gmail.com Tue Jan 6 11:25:01 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 6 Jan 2015 20:25:01 +1000 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <9A57680C-9629-4A22-91FF-E0625EA95457@yahoo.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20150105103828.GA22528@k3> <CADiSq7fLPxvBvfek1yTJE53bwhXsVfEpBrArRU1eC_0DFqMwHA@mail.gmail.com> <9A57680C-9629-4A22-91FF-E0625EA95457@yahoo.com> Message-ID: <CADiSq7eBBhoEpJp912OpFSGpEFyet7=djfGEpOfP1M=ERtmOsw@mail.gmail.com> On 6 January 2015 at 19:29, Andrew Barnert <abarnert at yahoo.com> wrote: > On Jan 6, 2015, at 5:43, Nick Coghlan <ncoghlan at gmail.com> wrote: >> Unfortunately, most of the folks that need these cross-platform >> integration features are likely to be using a C/C++ toolkit that >> provides them (e.g. Gtk, Qt, wxWidgets), which significantly reduces >> the incentive for anyone to create a Python-specific abstraction that >> would be suitable for eventual inclusion in the standard library. > > I'm not sure that's really unfortunate. > > First, a framework like Qt or wx is maintained by people who think about nothing but desktop and GUI integration on the major platforms. They track changes pretty quickly (especially compared to the Python stdlib). A few years back, when Apple got rid of legacy-style preference files, I had a Qt app that I was able to fix just by updating to the next version of Qt, and a non-Qt app that I had to fix by reseatching, writing, and testing, a wrapper that "temporarily" (of course it's still there in 2015...) stashes the entire legacy file as an encoded string in the new-style prefs database. What are the odds a Python stdlib solution (or even a third-party Python lib with less than 1/20th the user base, community, and corporate support) would have fixed it for me automatically in time for my next release the way Qt did? > > Also, the fact that Qt and wx are complete frameworks means they can make assumptions that allow for simpler solutions. A Qt library knows how to check whether there's a runloop and, if there is, knows it'll be a Qt runloop, and, if not, knows how to create a short-term blocking runloop and background thread, and it also knows how to pass signals from background threads to the runloop, and so on; this means Qt could just use the (10.5+, and still not deprecated as of 10.10) NSWorkspace API for trashing on Mac, because none of its deficiencies apply to any Qt app. A library meant to be used generally across all kinds of apps can't do that. Obviously this is also a weakness, because not all cross-platform desktop-integration apps really need a full framework? but many do. > > And again, many simple tools don't need to be cross-platform (like my completely separate 3-line trash tools for each platform that I use), or don't need full generality (my Mac trash tool doesn't need to work on 10.6, or to be App Store compliant; my Ubuntu trash tool doesn't have to know how Unity deals with trashing over CIFS because I only have two CIFS mounts and they both do server-side automatic recycling; etc.). Yes, there's certainly a lot of value in adopting an existing specialist toolkit when writing rich client applications. The key problem is that it isn't easy to start with the provided Tcl/Tk support in the standard library (to avoid introducing a large external dependency) and then later migrate to one of the other third party toolkits (if the basic functionality proves insufficient). While an asyncio-style pluggable solution may be one possible option for that (i.e. provide default implementations of features which may be fairly basic, but allow them to be replaced by full featured implementations backed by Gtk/Qt/wxWidgets/etc), that kind of abstraction layer design is extraordinarily difficult to design well. It's hard enough even in cases like wsgiref and asyncio that are relatively simple compared to the services a full rich client application will expect to have available. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From abarnert at yahoo.com Tue Jan 6 14:43:54 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 6 Jan 2015 14:43:54 +0100 Subject: [Python-ideas] Adding `pathlib.Path` method that would send file to recycle bin In-Reply-To: <CADiSq7eBBhoEpJp912OpFSGpEFyet7=djfGEpOfP1M=ERtmOsw@mail.gmail.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20150105103828.GA22528@k3> <CADiSq7fLPxvBvfek1yTJE53bwhXsVfEpBrArRU1eC_0DFqMwHA@mail.gmail.com> <9A57680C-9629-4A22-91FF-E0625EA95457@yahoo.com> <CADiSq7eBBhoEpJp912OpFSGpEFyet7=djfGEpOfP1M=ERtmOsw@mail.gmail.com> Message-ID: <F59CAFE9-FDEE-40C1-9E36-6A08DEF68F04@yahoo.com> On Jan 6, 2015, at 11:25, Nick Coghlan <ncoghlan at gmail.com> wrote: > On 6 January 2015 at 19:29, Andrew Barnert <abarnert at yahoo.com> wrote: >> On Jan 6, 2015, at 5:43, Nick Coghlan <ncoghlan at gmail.com> wrote: >>> Unfortunately, most of the folks that need these cross-platform >>> integration features are likely to be using a C/C++ toolkit that >>> provides them (e.g. Gtk, Qt, wxWidgets), which significantly reduces >>> the incentive for anyone to create a Python-specific abstraction that >>> would be suitable for eventual inclusion in the standard library. >> >> I'm not sure that's really unfortunate. >> >> First, a framework like Qt or wx is maintained by people who think about nothing but desktop and GUI integration on the major platforms. They track changes pretty quickly (especially compared to the Python stdlib). A few years back, when Apple got rid of legacy-style preference files, I had a Qt app that I was able to fix just by updating to the next version of Qt, and a non-Qt app that I had to fix by reseatching, writing, and testing, a wrapper that "temporarily" (of course it's still there in 2015...) stashes the entire legacy file as an encoded string in the new-style prefs database. What are the odds a Python stdlib solution (or even a third-party Python lib with less than 1/20th the user base, community, and corporate support) would have fixed it for me automatically in time for my next release the way Qt did? >> >> Also, the fact that Qt and wx are complete frameworks means they can make assumptions that allow for simpler solutions. A Qt library knows how to check whether there's a runloop and, if there is, knows it'll be a Qt runloop, and, if not, knows how to create a short-term blocking runloop and background thread, and it also knows how to pass signals from background threads to the runloop, and so on; this means Qt could just use the (10.5+, and still not deprecated as of 10.10) NSWorkspace API for trashing on Mac, because none of its deficiencies apply to any Qt app. A library meant to be used generally across all kinds of apps can't do that. Obviously this is also a weakness, because not all cross-platform desktop-integration apps really need a full framework? but many do. >> >> And again, many simple tools don't need to be cross-platform (like my completely separate 3-line trash tools for each platform that I use), or don't need full generality (my Mac trash tool doesn't need to work on 10.6, or to be App Store compliant; my Ubuntu trash tool doesn't have to know how Unity deals with trashing over CIFS because I only have two CIFS mounts and they both do server-side automatic recycling; etc.). > > Yes, there's certainly a lot of value in adopting an existing > specialist toolkit when writing rich client applications. The key > problem is that it isn't easy to start with the provided Tcl/Tk > support in the standard library (to avoid introducing a large external > dependency) and then later migrate to one of the other third party > toolkits (if the basic functionality proves insufficient). > While an asyncio-style pluggable solution may be one possible option > for that (i.e. provide default implementations of features which may > be fairly basic, but allow them to be replaced by full featured > implementations backed by Gtk/Qt/wxWidgets/etc), that kind of > abstraction layer design is extraordinarily difficult to design well. > It's hard enough even in cases like wsgiref and asyncio that are > relatively simple compared to the services a full rich client > application will expect to have available. Yes, and even besides the difference in complexity, even the most basic elements--the API for the runloop and message maps (wx), signals/slots (Qt), event handlers (Tk), etc.--are pretty hard to abstract away. For servers, a reactor is a reactor, but for GUIs, a signal/slot system is not a message pump/message map system. And whichever abstraction you pick, adapting it to work with the other abstractions is no fun. (Look at how long the TkAqua Cocoa port took, and how much Cocoa functionality is still missing.) From ncoghlan at gmail.com Tue Jan 6 16:25:39 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 7 Jan 2015 01:25:39 +1000 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <54AB4178.3050508@gmail.com> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> <54AB4178.3050508@gmail.com> Message-ID: <CADiSq7cwJn9NZnX=wCTfHs6s+yN_8tuFvbu8P8yvwtjP1Zp=0w@mail.gmail.com> On 6 January 2015 at 11:59, Yawar Amin <yawar.amin at gmail.com> wrote: > On 2015-01-05 04:49, Andrew Barnert wrote: >> ... paren continuation just concatenates the lines together. > > I'll quote this first, because it's the crux of the matter and what I > failed to understand up until now. IIUC now, Python never actually > 'relaxes' any indentation rules--a preprocessor just concatenates split > lines (i.e. lines split with parens etc.) and then passes the result on > to the parser? Not quite. INDENT, DEDENT and NEWLINE are possible tokens generated by the tokeniser. They're only generated to delimit statements, never inside expressions - by the time the parser itself gets involved, the original whitespace has been elided by the tokenisation process. > It's back to the drawing board for me, I guess :-) You may find http://python-notes.curiousefficiency.org/en/latest/pep_ideas/suite_expr.html an interesting read. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Tue Jan 6 17:22:23 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 6 Jan 2015 16:22:23 +0000 Subject: [Python-ideas] Possible new itertool: comm() Message-ID: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> In writing a utility script today, I found myself needing to do something similar to what the Unix "comm" utility does - take two sorted iterators, and partition the values into "only in the first", "only in the second", and "in both" groups. I couldn't find an obvious implementation around, so I ended up having to write my own (see below). Would this be something useful to add to the itertools module? The reasons I think it might be worth including in the stdlib are: 1. The implementation is somewhat tricky to get right, especially the edge cases (I'm not 100% sure my implementation is completely right in this regard). So implementing it yourself each time is error-prone and time consuming. 2. The times I've needed this have been ad-hoc scripts (I'm on Windows, so while a Unix user might use a quick shell pipeline with comm, that's less convenient for me) where depending on a 3rd party distribution from PyPI is less ideal. There's plenty of room for API bikeshedding here, but the basic algorithm remains the same. What do people think? Paul def comm(it1, it2): """Partition 2 sorted iterators into common, only in 1st, and only in 2nd. >>> list(comm([1,2,3], [2,3,4])) [('<', 1), ('=', 2), ('=', 3), ('>', 4)] >>> list(comm([0,1,2,3], [2,3,4,5])) [('<', 0), ('<', 1), ('=', 2), ('=', 3), ('>', 4), ('>', 5)] >>> list(comm([0,1,2,3,6], [2,3,4,5])) [('<', 0), ('<', 1), ('=', 2), ('=', 3), ('>', 4), ('>', 5), ('<', 6)] >>> list(comm([0,1], [])) [('<', 0), ('<', 1)] >>> list(comm([], [0,1])) [('>', 0), ('>', 1)] >>> list(comm([0,1], [0,1])) [('=', 0), ('=', 1)] >>> list(comm([], [])) [] """ it1 = iter(it1) it2 = iter(it2) it1_finished = False it2_finished = False try: v1 = next(it1) except StopIteration: it1_finished = True try: v2 = next(it2) except StopIteration: it2_finished = True while not (it1_finished or it2_finished): if v1 < v2: yield ('<', v1) try: v1 = next(it1) except StopIteration: it1_finished = True elif v1 > v2: yield ('>', v2) try: v2 = next(it2) except StopIteration: it2_finished = True else: yield ('=', v1) try: v1 = next(it1) except StopIteration: it1_finished = True try: v2 = next(it2) except StopIteration: it2_finished = True if it1_finished and not it2_finished: yield ('>', v2) for v2 in it2: yield ('>', v2) if it2_finished and not it1_finished: yield ('<', v1) for v1 in it1: yield ('<', v1) if __name__ == '__main__': import doctest doctest.testmod() From graffatcolmingov at gmail.com Tue Jan 6 17:33:16 2015 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Tue, 6 Jan 2015 10:33:16 -0600 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> Message-ID: <CAN-Kwu12CwKtWjPWWB40tnvXV-r95PQQmvvmgHr2EZyJdbMpmQ@mail.gmail.com> On Tue, Jan 6, 2015 at 10:22 AM, Paul Moore <p.f.moore at gmail.com> wrote: > In writing a utility script today, I found myself needing to do > something similar to what the Unix "comm" utility does - take two > sorted iterators, and partition the values into "only in the first", > "only in the second", and "in both" groups. > > I couldn't find an obvious implementation around, so I ended up having > to write my own (see below). > > Would this be something useful to add to the itertools module? The > reasons I think it might be worth including in the stdlib are: > > 1. The implementation is somewhat tricky to get right, especially the > edge cases (I'm not 100% sure my implementation is completely right in > this regard). So implementing it yourself each time is error-prone and > time consuming. > 2. The times I've needed this have been ad-hoc scripts (I'm on > Windows, so while a Unix user might use a quick shell pipeline with > comm, that's less convenient for me) where depending on a 3rd party > distribution from PyPI is less ideal. > > There's plenty of room for API bikeshedding here, but the basic > algorithm remains the same. > > What do people think? > > Paul > > def comm(it1, it2): > """Partition 2 sorted iterators into common, only in 1st, and only in 2nd. > > >>> list(comm([1,2,3], [2,3,4])) > [('<', 1), ('=', 2), ('=', 3), ('>', 4)] > >>> list(comm([0,1,2,3], [2,3,4,5])) > [('<', 0), ('<', 1), ('=', 2), ('=', 3), ('>', 4), ('>', 5)] > >>> list(comm([0,1,2,3,6], [2,3,4,5])) > [('<', 0), ('<', 1), ('=', 2), ('=', 3), ('>', 4), ('>', 5), ('<', 6)] > >>> list(comm([0,1], [])) > [('<', 0), ('<', 1)] > >>> list(comm([], [0,1])) > [('>', 0), ('>', 1)] > >>> list(comm([0,1], [0,1])) > [('=', 0), ('=', 1)] > >>> list(comm([], [])) > [] > """ > > it1 = iter(it1) > it2 = iter(it2) > > it1_finished = False > it2_finished = False > try: > v1 = next(it1) > except StopIteration: > it1_finished = True > try: > v2 = next(it2) > except StopIteration: > it2_finished = True > while not (it1_finished or it2_finished): > if v1 < v2: > yield ('<', v1) > try: > v1 = next(it1) > except StopIteration: > it1_finished = True > elif v1 > v2: > yield ('>', v2) > try: > v2 = next(it2) > except StopIteration: > it2_finished = True > else: > yield ('=', v1) > try: > v1 = next(it1) > except StopIteration: > it1_finished = True > try: > v2 = next(it2) > except StopIteration: > it2_finished = True > > if it1_finished and not it2_finished: > yield ('>', v2) > for v2 in it2: > yield ('>', v2) > if it2_finished and not it1_finished: > yield ('<', v1) > for v1 in it1: > yield ('<', v1) > > if __name__ == '__main__': > import doctest > doctest.testmod() > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ So my question is how well with this work with generators/iterators? Your examples use lists, but it would be impossible to use this with anything that isn't finite right? From p.f.moore at gmail.com Tue Jan 6 17:44:04 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 6 Jan 2015 16:44:04 +0000 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <CAN-Kwu12CwKtWjPWWB40tnvXV-r95PQQmvvmgHr2EZyJdbMpmQ@mail.gmail.com> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <CAN-Kwu12CwKtWjPWWB40tnvXV-r95PQQmvvmgHr2EZyJdbMpmQ@mail.gmail.com> Message-ID: <CACac1F_2ns+-C=9if959sgzJaeYNBnQF0nNtRdBYTpaueuEw4A@mail.gmail.com> On 6 January 2015 at 16:33, Ian Cordasco <graffatcolmingov at gmail.com> wrote: > So my question is how well with this work with generators/iterators? > Your examples use lists, but it would be impossible to use this with > anything that isn't finite right? The examples use lists because I was using doctest and needed checkable output. But I think it should work fine with any (finite) iterator. I don't see a problem with the algorithm if I use infinite iterators (the algorithm is one-pass and generates results on demand) but I haven't tested it explicitly. [pause, test...] Yep, looks OK: >>> import itertools >>> c1 = itertools.count() >>> c2 = itertools.count(3) >>> output = comm(c1, c2) >>> next(output) ('<', 0) >>> next(output) ('<', 1) >>> next(output) ('<', 2) >>> next(output) ('=', 3) Paul From rosuav at gmail.com Tue Jan 6 17:46:21 2015 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 7 Jan 2015 03:46:21 +1100 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <CAN-Kwu12CwKtWjPWWB40tnvXV-r95PQQmvvmgHr2EZyJdbMpmQ@mail.gmail.com> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <CAN-Kwu12CwKtWjPWWB40tnvXV-r95PQQmvvmgHr2EZyJdbMpmQ@mail.gmail.com> Message-ID: <CAPTjJmqyuNE3t9H6-VXTQRazCCn_6FDA7uonw81FGN+7uUzHsA@mail.gmail.com> On Wed, Jan 7, 2015 at 3:33 AM, Ian Cordasco <graffatcolmingov at gmail.com> wrote: > So my question is how well with this work with generators/iterators? > Your examples use lists, but it would be impossible to use this with > anything that isn't finite right? I think it ought to work fine on infinite iterators, based on my reading of the code. Obviously it would itself be infinite in that case. With one infinite and one finite iterator, it'll eventually get into one of the loops at the end, and forever yield from the infinite iterator; if both are infinite, it'll never break out of the primary loop, and continue consuming values from one or the other and yielding tuples: >>> def count_by(x): n = 0 while True: n += x yield n >>> it = comm(count_by(2), count_by(3)) >>> next(it) ('<', 2) >>> next(it) ('>', 3) >>> next(it) ('<', 4) >>> next(it) ('=', 6) ... etc etc etc ... ChrisA From rosuav at gmail.com Tue Jan 6 17:47:52 2015 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 7 Jan 2015 03:47:52 +1100 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <CAPTjJmqyuNE3t9H6-VXTQRazCCn_6FDA7uonw81FGN+7uUzHsA@mail.gmail.com> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <CAN-Kwu12CwKtWjPWWB40tnvXV-r95PQQmvvmgHr2EZyJdbMpmQ@mail.gmail.com> <CAPTjJmqyuNE3t9H6-VXTQRazCCn_6FDA7uonw81FGN+7uUzHsA@mail.gmail.com> Message-ID: <CAPTjJmoGqypqWB8sQ+1nAi3KJaMouWygB=Vx-MX=mQPRsCheoA@mail.gmail.com> On Wed, Jan 7, 2015 at 3:46 AM, Chris Angelico <rosuav at gmail.com> wrote: >>>> def count_by(x): > n = 0 > while True: > n += x > yield n And thanks to the magic of sloppy editing, I just rewrote itertools.count(). In my own testing, there was a print call to show what the internal iterators were yielding, but that got left out, leaving this looking a little silly. Anyhow. ChrisA From raymond.hettinger at gmail.com Tue Jan 6 18:14:01 2015 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Tue, 6 Jan 2015 09:14:01 -0800 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> Message-ID: <1BC2F2DA-DC73-4BE3-8550-37F5EA872C07@gmail.com> > On Jan 6, 2015, at 8:22 AM, Paul Moore <p.f.moore at gmail.com> wrote: > > In writing a utility script today, I found myself needing to do > something similar to what the Unix "comm" utility does - take two > sorted iterators, and partition the values into "only in the first", > "only in the second", and "in both" groups. As far as I can tell, this would be a very rare need. Raymond From abarnert at yahoo.com Tue Jan 6 18:39:57 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 6 Jan 2015 17:39:57 +0000 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> Message-ID: <EBC9FD14-4E89-4ADA-9113-3B1CC365B47E@yahoo.com> On Jan 6, 2015, at 16:22, Paul Moore <p.f.moore at gmail.com> wrote: > 2. The times I've needed this have been ad-hoc scripts (I'm on > Windows, so while a Unix user might use a quick shell pipeline with > comm, that's less convenient for me) where depending on a 3rd party > distribution from PyPI is less ideal. Why? Current standard Windows installers include pip, and being pure Python you won't need a compiler, so what's wrong with requiring a PyPI distribution? (Of course that means you need to be able to count on a relatively recent Python 3.4+/2.7+, but it's hard to see how that's worse than something in a future version of the stdlib, which would mean you need to be able to count on 3.5+.) And I'll bet if you submit this as a pull request to more-itertools, it'll be accepted, meaning you don't even have to create or maintain a PyPI project. From steve at pearwood.info Tue Jan 6 19:00:05 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 7 Jan 2015 05:00:05 +1100 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> Message-ID: <20150106180005.GE19387@ando.pearwood.info> On Tue, Jan 06, 2015 at 04:22:23PM +0000, Paul Moore wrote: > In writing a utility script today, I found myself needing to do > something similar to what the Unix "comm" utility does - take two > sorted iterators, and partition the values into "only in the first", > "only in the second", and "in both" groups. It seems awfully specialised to me, so I'm not sure that it belongs in itertools. But for what it's worth, here's my implementation which I think is easier to understand than yours, and covers the unfortunate case where you have unorderable items such as float NANs in your data. def comm(values1, values2): """Partition 2 sorted iterators into common, only in 1st, and only in 2nd. >>> list(comm([1,2,3], [2,3,4])) [('<', 1), ('=', 2), ('=', 3), ('>', 4)] >>> list(comm([0,1,2,3], [2,3,4,5])) [('<', 0), ('<', 1), ('=', 2), ('=', 3), ('>', 4), ('>', 5)] >>> list(comm([0,1,2,3,6], [2,3,4,5])) [('<', 0), ('<', 1), ('=', 2), ('=', 3), ('>', 4), ('>', 5), ('<', 6)] >>> list(comm([0,1], [])) [('<', 0), ('<', 1)] >>> list(comm([], [0,1])) [('>', 0), ('>', 1)] >>> list(comm([0,1], [0,1])) [('=', 0), ('=', 1)] >>> list(comm([], [])) [] >>> list(comm([1,2,3,4], [2,3,float('nan'),4])) [('<', 1), ('=', 2), ('=', 3), ('?', (4, nan)), ('>', 4)] """ MISSING = object() it1 = iter(values1) it2 = iter(values2) a = next(it1, MISSING) b = next(it2, MISSING) while True: if a is MISSING: if b is MISSING: return yield ('>', b) for x in it2: yield ('>', x) return if b is MISSING: assert a is not MISSING yield ('<', a) for x in it1: yield ('<', x) return if a == b: yield ('=', a) # Or could use b. a = next(it1, MISSING) b = next(it2, MISSING) elif a < b: yield ('<', a) a = next(it1, MISSING) elif a > b: yield ('>', b) b = next(it2, MISSING) else: # Unorderable a and b, like float NANs. # Maybe we should raise instead? yield ('?', (a, b)) a = next(it1, MISSING) b = next(it2, MISSING) If you prefer this version, feel free to just use it. If you need a licence, consider it being under the MIT licence. (Over the next day or so, time permitting, I'll try to put this on ActiveState's Python recipes and make the licence more official.) -- Steve From p.f.moore at gmail.com Tue Jan 6 19:22:44 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 6 Jan 2015 18:22:44 +0000 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <1BC2F2DA-DC73-4BE3-8550-37F5EA872C07@gmail.com> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <1BC2F2DA-DC73-4BE3-8550-37F5EA872C07@gmail.com> Message-ID: <CACac1F8kwFs5g4VbCaKwbwGSE=gHG8JHkZ=ZTcCgpyZ9WxRqyA@mail.gmail.com> On 6 January 2015 at 17:14, Raymond Hettinger <raymond.hettinger at gmail.com> wrote: >> On Jan 6, 2015, at 8:22 AM, Paul Moore <p.f.moore at gmail.com> wrote: >> >> In writing a utility script today, I found myself needing to do >> something similar to what the Unix "comm" utility does - take two >> sorted iterators, and partition the values into "only in the first", >> "only in the second", and "in both" groups. > > As far as I can tell, this would be a very rare need. It's come up for me a few times, usually when trying to check two lists of files to see which ones have been missed by a program, and which ones the program thinks are present but no longer exist. Its use cases are about as common as those of the Unix "comm" tool :-) But I take your point, it's not *that* common. Paul From ethan at stoneleaf.us Tue Jan 6 19:28:23 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 06 Jan 2015 10:28:23 -0800 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <20150106180005.GE19387@ando.pearwood.info> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <20150106180005.GE19387@ando.pearwood.info> Message-ID: <54AC2947.3000209@stoneleaf.us> On 01/06/2015 10:00 AM, Steven D'Aprano wrote: > a = next(it1, MISSING) > b = next(it2, MISSING) Don't forget to guard your next calls -- RunTimeError would not be a friendly way to stop iterating. ;) -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150106/dd3e8b8e/attachment.sig> From p.f.moore at gmail.com Tue Jan 6 19:29:33 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 6 Jan 2015 18:29:33 +0000 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <EBC9FD14-4E89-4ADA-9113-3B1CC365B47E@yahoo.com> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <EBC9FD14-4E89-4ADA-9113-3B1CC365B47E@yahoo.com> Message-ID: <CACac1F_yTKhnQ9_BcoLAm41b3cyqfrjRNkh6A3fRh8M_aT9NtA@mail.gmail.com> On 6 January 2015 at 17:39, Andrew Barnert <abarnert at yahoo.com> wrote: >> 2. The times I've needed this have been ad-hoc scripts (I'm on >> Windows, so while a Unix user might use a quick shell pipeline with >> comm, that's less convenient for me) where depending on a 3rd party >> distribution from PyPI is less ideal. > > Why? Current standard Windows installers include pip, and being pure Python you won't need a compiler, so what's wrong with requiring a PyPI distribution? (Of course that means you need to be able to count on a relatively recent Python 3.4+/2.7+, but it's hard to see how that's worse than something in a future version of the stdlib, which would mean you need to be able to count on 3.5+.) Well, I usually write my "little utility scripts" as simple .py files to be run with the system Python. I tend to use them on multiple machines. So unless a dependency is one of the modules I routinely install (things like requests) the process goes run script, oops, needed that distribution, pip install dist, run it again. Not a big issue certainly (and hardly a showstopper) but annoying. And it does mean I'd need to make a PyPI project for my personal utility functions, which doesn't really seem an appropriate use for PyPI, tbh. Requiring Python 3.5+ isn't a big deal, I routinely put the newest version of Python on all my machines. The ones I can't tend to be "secure", meaning I have no access to PyPI either :-( > And I'll bet if you submit this as a pull request to more-itertools, it'll be accepted, meaning you don't even have to create or maintain a PyPI project. Thanks for the suggestion, I might do that. Paul From p.f.moore at gmail.com Tue Jan 6 19:30:11 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 6 Jan 2015 18:30:11 +0000 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <54AC2947.3000209@stoneleaf.us> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <20150106180005.GE19387@ando.pearwood.info> <54AC2947.3000209@stoneleaf.us> Message-ID: <CACac1F_2XQvVPpeztVTKsp7asJgu-opqEXVCp+swA2_wJqjLqw@mail.gmail.com> On 6 January 2015 at 18:28, Ethan Furman <ethan at stoneleaf.us> wrote: > On 01/06/2015 10:00 AM, Steven D'Aprano wrote: > >> a = next(it1, MISSING) >> b = next(it2, MISSING) > > Don't forget to guard your next calls -- RunTimeError would not be a friendly way to stop iterating. ;) Doesn't the second argument get used instead of raising StopIteration? Paul From solipsis at pitrou.net Tue Jan 6 19:36:55 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 6 Jan 2015 19:36:55 +0100 Subject: [Python-ideas] Possible new itertool: comm() References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <1BC2F2DA-DC73-4BE3-8550-37F5EA872C07@gmail.com> <CACac1F8kwFs5g4VbCaKwbwGSE=gHG8JHkZ=ZTcCgpyZ9WxRqyA@mail.gmail.com> Message-ID: <20150106193655.2be8dcf0@fsol> On Tue, 6 Jan 2015 18:22:44 +0000 Paul Moore <p.f.moore at gmail.com> wrote: > On 6 January 2015 at 17:14, Raymond Hettinger > <raymond.hettinger at gmail.com> wrote: > >> On Jan 6, 2015, at 8:22 AM, Paul Moore <p.f.moore at gmail.com> wrote: > >> > >> In writing a utility script today, I found myself needing to do > >> something similar to what the Unix "comm" utility does - take two > >> sorted iterators, and partition the values into "only in the first", > >> "only in the second", and "in both" groups. > > > > As far as I can tell, this would be a very rare need. > > It's come up for me a few times, usually when trying to check two > lists of files to see which ones have been missed by a program, and > which ones the program thinks are present but no longer exist. Why don't you use sets for such things? Your iterator is really only useful for huge or unhashable inputs. Regards Antoine. From p.f.moore at gmail.com Tue Jan 6 19:40:40 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 6 Jan 2015 18:40:40 +0000 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <20150106180005.GE19387@ando.pearwood.info> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <20150106180005.GE19387@ando.pearwood.info> Message-ID: <CACac1F9q6+MV+345d2byLFbbUhNbyq_hthAA8K3WV_88H1ooKQ@mail.gmail.com> On 6 January 2015 at 18:00, Steven D'Aprano <steve at pearwood.info> wrote: > But for what it's worth, here's my implementation which I > think is easier to understand than yours, Possibly - it's a trade-off between using a sentinel vs explicitly catching the exceptions. I'm on the fence, personally. > and covers the unfortunate > case where you have unorderable items such as float NANs in your data. Technically I don't think this is possible, as the input iterators need to be sorted, and that property will be violated if there are unorderable items in there... Paul From ethan at stoneleaf.us Tue Jan 6 19:43:23 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 06 Jan 2015 10:43:23 -0800 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <CACac1F_2XQvVPpeztVTKsp7asJgu-opqEXVCp+swA2_wJqjLqw@mail.gmail.com> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <20150106180005.GE19387@ando.pearwood.info> <54AC2947.3000209@stoneleaf.us> <CACac1F_2XQvVPpeztVTKsp7asJgu-opqEXVCp+swA2_wJqjLqw@mail.gmail.com> Message-ID: <54AC2CCB.5040703@stoneleaf.us> On 01/06/2015 10:30 AM, Paul Moore wrote: > On 6 January 2015 at 18:28, Ethan Furman <ethan at stoneleaf.us> wrote: >> On 01/06/2015 10:00 AM, Steven D'Aprano wrote: >> >>> a = next(it1, MISSING) >>> b = next(it2, MISSING) >> >> Don't forget to guard your next calls -- RunTimeError would not be a friendly way to stop iterating. ;) > > Doesn't the second argument get used instead of raising StopIteration? Yup, it sure does. Guess I should have paid more attention to that niggling feeling that I was missing something. ;) On the other hand, I figured even if I was wrong it wouldn't hurt to increase awareness about the change to generators. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150106/3ef9a927/attachment.sig> From p.f.moore at gmail.com Tue Jan 6 19:44:22 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 6 Jan 2015 18:44:22 +0000 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <20150106193655.2be8dcf0@fsol> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <1BC2F2DA-DC73-4BE3-8550-37F5EA872C07@gmail.com> <CACac1F8kwFs5g4VbCaKwbwGSE=gHG8JHkZ=ZTcCgpyZ9WxRqyA@mail.gmail.com> <20150106193655.2be8dcf0@fsol> Message-ID: <CACac1F9xnLe5Sv_-vCMH6FBurJRrfU7VONXW5rsWbsfWzozb2g@mail.gmail.com> On 6 January 2015 at 18:36, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Tue, 6 Jan 2015 18:22:44 +0000 > Paul Moore <p.f.moore at gmail.com> wrote: >> On 6 January 2015 at 17:14, Raymond Hettinger >> <raymond.hettinger at gmail.com> wrote: >> >> On Jan 6, 2015, at 8:22 AM, Paul Moore <p.f.moore at gmail.com> wrote: >> >> >> >> In writing a utility script today, I found myself needing to do >> >> something similar to what the Unix "comm" utility does - take two >> >> sorted iterators, and partition the values into "only in the first", >> >> "only in the second", and "in both" groups. >> > >> > As far as I can tell, this would be a very rare need. >> >> It's come up for me a few times, usually when trying to check two >> lists of files to see which ones have been missed by a program, and >> which ones the program thinks are present but no longer exist. > > Why don't you use sets for such things? Your iterator is really only > useful for huge or unhashable inputs. For the case described you're right. I had a case the other day where one of the problems was that one list had duplicates, and I needed to see that ([1,1,2,2] vs [1,2] needed to show [1,2] as "only in the first list" and [1,2] as "in both"). I could probably have used counters. But I was starting from thinking that I had a pair of sorted lists, and "in the Unix shell I'd use comm"...) Paul From p.f.moore at gmail.com Tue Jan 6 19:47:46 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 6 Jan 2015 18:47:46 +0000 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <54AC2CCB.5040703@stoneleaf.us> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <20150106180005.GE19387@ando.pearwood.info> <54AC2947.3000209@stoneleaf.us> <CACac1F_2XQvVPpeztVTKsp7asJgu-opqEXVCp+swA2_wJqjLqw@mail.gmail.com> <54AC2CCB.5040703@stoneleaf.us> Message-ID: <CACac1F9BMouTa2MP8zAOGsHtL6utTL8eSv3tuePg=VHTq5-0CA@mail.gmail.com> On 6 January 2015 at 18:43, Ethan Furman <ethan at stoneleaf.us> wrote: > Yup, it sure does. Guess I should have paid more attention to that niggling feeling that I was missing something. ;) > On the other hand, I figured even if I was wrong it wouldn't hurt to increase awareness about the change to generators. I should probably point out that when writing my code, I was conscious of the generator change and took special case of catching all of the StopIterations that might leak out. It's probably good that I was made to think carefully about it, but all those try...except blocks sure look ugly. (And Steven's version with its sentinel value doesn't look much better, with all those comparisons against a magic value - it's just a generally annoying algorithm with all the code paths for corner cases). Paul From guido at python.org Tue Jan 6 19:47:44 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 6 Jan 2015 10:47:44 -0800 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <CACac1F9xnLe5Sv_-vCMH6FBurJRrfU7VONXW5rsWbsfWzozb2g@mail.gmail.com> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <1BC2F2DA-DC73-4BE3-8550-37F5EA872C07@gmail.com> <CACac1F8kwFs5g4VbCaKwbwGSE=gHG8JHkZ=ZTcCgpyZ9WxRqyA@mail.gmail.com> <20150106193655.2be8dcf0@fsol> <CACac1F9xnLe5Sv_-vCMH6FBurJRrfU7VONXW5rsWbsfWzozb2g@mail.gmail.com> Message-ID: <CAP7+vJ+-PrJh07Ra-ru4QyCwMwMKNaRFbPCPmnxXPV5bybBaHA@mail.gmail.com> Folks, I realize I'm sounding a bit like Raymond here, but can you all please find a different forum to discuss algorithms and coding problems and pick each other's solutions apart? The focus on python-ideas should be to quickly validate ideas for the language or the stdlib, not to discuss arbitrary snippets of code. On Tue, Jan 6, 2015 at 10:44 AM, Paul Moore <p.f.moore at gmail.com> wrote: > On 6 January 2015 at 18:36, Antoine Pitrou <solipsis at pitrou.net> wrote: > > On Tue, 6 Jan 2015 18:22:44 +0000 > > Paul Moore <p.f.moore at gmail.com> wrote: > >> On 6 January 2015 at 17:14, Raymond Hettinger > >> <raymond.hettinger at gmail.com> wrote: > >> >> On Jan 6, 2015, at 8:22 AM, Paul Moore <p.f.moore at gmail.com> wrote: > >> >> > >> >> In writing a utility script today, I found myself needing to do > >> >> something similar to what the Unix "comm" utility does - take two > >> >> sorted iterators, and partition the values into "only in the first", > >> >> "only in the second", and "in both" groups. > >> > > >> > As far as I can tell, this would be a very rare need. > >> > >> It's come up for me a few times, usually when trying to check two > >> lists of files to see which ones have been missed by a program, and > >> which ones the program thinks are present but no longer exist. > > > > Why don't you use sets for such things? Your iterator is really only > > useful for huge or unhashable inputs. > > For the case described you're right. I had a case the other day where > one of the problems was that one list had duplicates, and I needed to > see that ([1,1,2,2] vs [1,2] needed to show [1,2] as "only in the > first list" and [1,2] as "in both"). I could probably have used > counters. But I was starting from thinking that I had a pair of sorted > lists, and "in the Unix shell I'd use comm"...) > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150106/31919aea/attachment.html> From ethan at stoneleaf.us Tue Jan 6 20:02:16 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 06 Jan 2015 11:02:16 -0800 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <CACac1F9BMouTa2MP8zAOGsHtL6utTL8eSv3tuePg=VHTq5-0CA@mail.gmail.com> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <20150106180005.GE19387@ando.pearwood.info> <54AC2947.3000209@stoneleaf.us> <CACac1F_2XQvVPpeztVTKsp7asJgu-opqEXVCp+swA2_wJqjLqw@mail.gmail.com> <54AC2CCB.5040703@stoneleaf.us> <CACac1F9BMouTa2MP8zAOGsHtL6utTL8eSv3tuePg=VHTq5-0CA@mail.gmail.com> Message-ID: <54AC3138.40107@stoneleaf.us> On 01/06/2015 10:47 AM, Paul Moore wrote: > On 6 January 2015 at 18:43, Ethan Furman wrote: >> >> Yup, it sure does. Guess I should have paid more attention to that niggling feeling that I was missing something. ;) >> On the other hand, I figured even if I was wrong it wouldn't hurt to increase awareness about the change to generators. > > I should probably point out that when writing my code, I was conscious > of the generator change and took special case of catching all of the > StopIterations that might leak out. It's probably good that I was made > to think carefully about it, but all those try...except blocks sure > look ugly. Having looked more closely at your code, I don't see any thing that wouldn't have had to be there even without the generator change because your exhausting two iterators. > (And Steven's version with its sentinel value doesn't look > much better, with all those comparisons against a magic value I think I prefer his, although I would rename MISSING to FINISHED. ;) -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150106/9462187d/attachment.sig> From abarnert at yahoo.com Tue Jan 6 19:59:08 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 6 Jan 2015 18:59:08 +0000 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <CACac1F_yTKhnQ9_BcoLAm41b3cyqfrjRNkh6A3fRh8M_aT9NtA@mail.gmail.com> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <EBC9FD14-4E89-4ADA-9113-3B1CC365B47E@yahoo.com> <CACac1F_yTKhnQ9_BcoLAm41b3cyqfrjRNkh6A3fRh8M_aT9NtA@mail.gmail.com> Message-ID: <98F84827-B54F-4163-B3EF-C3AFEF5BA811@yahoo.com> On Jan 6, 2015, at 18:29, Paul Moore <p.f.moore at gmail.com> wrote: > On 6 January 2015 at 17:39, Andrew Barnert <abarnert at yahoo.com> wrote: >>> 2. The times I've needed this have been ad-hoc scripts (I'm on >>> Windows, so while a Unix user might use a quick shell pipeline with >>> comm, that's less convenient for me) where depending on a 3rd party >>> distribution from PyPI is less ideal. >> >> Why? Current standard Windows installers include pip, and being pure Python you won't need a compiler, so what's wrong with requiring a PyPI distribution? (Of course that means you need to be able to count on a relatively recent Python 3.4+/2.7+, but it's hard to see how that's worse than something in a future version of the stdlib, which would mean you need to be able to count on 3.5+.) > > Well, I usually write my "little utility scripts" as simple .py files > to be run with the system Python. I tend to use them on multiple > machines. So unless a dependency is one of the modules I routinely > install (things like requests) the process goes run script, oops, > needed that distribution, pip install dist, run it again. I do the same thing; I just long ago got into the habit of including more-itertools as one of my routine installs alongside requests, etc. :) I wonder if there's a common-enough set of things that don't really belong in stdlib (often just because they update too frequently) but are often worth having for many people. If I could just "pip install extralib" on every machine, even at the cost of getting a few libs I don't actually need, it would be worth it. On the other hand, if the common set isn't common enough, so for many people extralib only had half of what they want and was 70% stuff they didn't care shouted it wouldn't be that useful... > Not a big > issue certainly (and hardly a showstopper) but annoying. And it does > mean I'd need to make a PyPI project for my personal utility > functions, which doesn't really seem an appropriate use for PyPI, tbh. > > Requiring Python 3.5+ isn't a big deal, I routinely put the newest > version of Python on all my machines. The ones I can't tend to be > "secure", meaning I have no access to PyPI either :-( > >> And I'll bet if you submit this as a pull request to more-itertools, it'll be accepted, meaning you don't even have to create or maintain a PyPI project. > > Thanks for the suggestion, I might do that. > > Paul From wes.turner at gmail.com Tue Jan 6 20:25:06 2015 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 6 Jan 2015 13:25:06 -0600 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <54AB9D76.7030105@egenix.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> Message-ID: <CACfEFw9mUcf11bbPz0r4rBASNOdbgTdTbzN0jV=Q7wZLBa9yxg@mail.gmail.com> Fair disclosure: I'm not a big fan of Moin syntax (can Pandoc convert this?). I don't like having to login to share with the Python community. It would be great to gain maximum synergy from the community on this. Some interesting perspectives on wikis, and where wikis are going: [1] https://github.com/blog/1939-how-github-uses-github-to-document-github [2] https://github.com/westurner/wiki/blob/master/Makefile [2] is a Sphinx Makefile for a GitHub ReStructuredText wiki. It seems to work okay for my purposes. What could prevent this from scaling to handle the Python wiki? >From https://github.com/westurner/wiki/issues/5 , in Markdown (which works on both BitBucket and GitHub): # Advantages > * Beautiful Sphinx output: > https://github.com/westurner/wiki/blob/master/conf.py > * Search > * Context-focused pages > * Navbar > * **Responsive** CSS Freedom; for my cell phone > * Pull Requests > # Challenges / Disadvantages > * New wiki pages created through the GitHub wiki interface add the `.rest` > suffix, rather than `.rst` > * Sort of an advantage (`.rest` files are docutils-only) > * This precludes inclusion of the wiki pages into a standard Sphinx > docset as a submodule or subtree. > * Sphinx ReStructuredText is a superset of GitHub ReStructuredText > (docutils), so Sphinx markup is, in the best case not supported, and in the > worst case breaks the wiki render > * Notably: `:ref:`, `.. index:`, and `.. toctree::` are only supported > by Sphinx > * To test this: `make rst2html_all` > * Double Tables of Contents (#4) > * Double pulls and pushes (`make pull push`): > https://github.com/westurner/wiki/blob/master/Makefile > * Sphinx requires a manual build step in order to publish in-progress > changes On Tue, Jan 6, 2015 at 2:31 AM, M.-A. Lemburg <mal at egenix.com> wrote: > On 06.01.2015 07:26, Ben Finney wrote: > > "M.-A. Lemburg" <mal at egenix.com> writes: > > > >> [editing the Python wiki] requiring asking for editing rights > >> on the pydotorg-www mailing list. Which is a little annoying and > >> some work for the admins, but has resulted in the amount of > >> spam to go down to zero. We usually grant the editing requests > >> within a day. > > > > I quite understand the motivation for this. > > > > It does need to be recognised that this will dramatically reduce the > > number of pople who will pass that hurdle to edit the wiki, and thus the > > wiki loses a lot of edits (both helpful and unhelpful) it might > > otherwise have. > > We might lose some occasional quick typo fixes, but it seems that > those genuinely interested in helping with the wiki don't have > a problem asking on the list. > > To reduce the problem I had created a list of user names > who had made good edits in the months before the change. > > The only nit I still have with the setup is that it's not > necessarily apparent for new users how to get editing rights. > There's a section on the front page, but you don't see this > when looking at a random page. > > > Thanks to those doing the work to maintain Python community > > infrastructure. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Jan 06 2015) > >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ > >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150106/486c058e/attachment-0001.html> From wes.turner at gmail.com Tue Jan 6 20:27:47 2015 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 6 Jan 2015 13:27:47 -0600 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <CACfEFw9mUcf11bbPz0r4rBASNOdbgTdTbzN0jV=Q7wZLBa9yxg@mail.gmail.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CACfEFw9mUcf11bbPz0r4rBASNOdbgTdTbzN0jV=Q7wZLBa9yxg@mail.gmail.com> Message-ID: <CACfEFw_h=1=Qtr42L+Xc25Z_fm5w0NRG3xQb1nKEg0=kxY3CkQ@mail.gmail.com> ... I had created an awesome-python-android page, but I seem to have `push -f`'ed over it and the reflog is too hard. On Tue, Jan 6, 2015 at 1:25 PM, Wes Turner <wes.turner at gmail.com> wrote: > Fair disclosure: I'm not a big fan of Moin syntax (can Pandoc convert > this?). I don't like having to login to share with the Python community. It > would be great to gain maximum synergy from the community on this. > > Some interesting perspectives on wikis, and where wikis are going: > > [1] https://github.com/blog/1939-how-github-uses-github-to-document-github > [2] https://github.com/westurner/wiki/blob/master/Makefile > > [2] is a Sphinx Makefile for a GitHub ReStructuredText wiki. It seems to > work okay for my purposes. What could prevent this from scaling to handle > the Python wiki? > > From https://github.com/westurner/wiki/issues/5 , in Markdown (which > works on both BitBucket and GitHub): > > # Advantages >> > > > * Beautiful Sphinx output: >> https://github.com/westurner/wiki/blob/master/conf.py >> * Search >> * Context-focused pages >> * Navbar >> * **Responsive** CSS Freedom; for my cell phone >> * Pull Requests >> # Challenges / Disadvantages >> * New wiki pages created through the GitHub wiki interface add the >> `.rest` suffix, rather than `.rst` >> * Sort of an advantage (`.rest` files are docutils-only) >> * This precludes inclusion of the wiki pages into a standard Sphinx >> docset as a submodule or subtree. >> * Sphinx ReStructuredText is a superset of GitHub ReStructuredText >> (docutils), so Sphinx markup is, in the best case not supported, and in the >> worst case breaks the wiki render >> * Notably: `:ref:`, `.. index:`, and `.. toctree::` are only supported >> by Sphinx >> * To test this: `make rst2html_all` >> * Double Tables of Contents (#4) >> * Double pulls and pushes (`make pull push`): >> https://github.com/westurner/wiki/blob/master/Makefile >> * Sphinx requires a manual build step in order to publish in-progress >> changes > > > > > > On Tue, Jan 6, 2015 at 2:31 AM, M.-A. Lemburg <mal at egenix.com> wrote: > >> On 06.01.2015 07:26, Ben Finney wrote: >> > "M.-A. Lemburg" <mal at egenix.com> writes: >> > >> >> [editing the Python wiki] requiring asking for editing rights >> >> on the pydotorg-www mailing list. Which is a little annoying and >> >> some work for the admins, but has resulted in the amount of >> >> spam to go down to zero. We usually grant the editing requests >> >> within a day. >> > >> > I quite understand the motivation for this. >> > >> > It does need to be recognised that this will dramatically reduce the >> > number of pople who will pass that hurdle to edit the wiki, and thus the >> > wiki loses a lot of edits (both helpful and unhelpful) it might >> > otherwise have. >> >> We might lose some occasional quick typo fixes, but it seems that >> those genuinely interested in helping with the wiki don't have >> a problem asking on the list. >> >> To reduce the problem I had created a list of user names >> who had made good edits in the months before the change. >> >> The only nit I still have with the setup is that it's not >> necessarily apparent for new users how to get editing rights. >> There's a section on the front page, but you don't see this >> when looking at a random page. >> >> > Thanks to those doing the work to maintain Python community >> > infrastructure. >> >> -- >> Marc-Andre Lemburg >> eGenix.com >> >> Professional Python Services directly from the Source (#1, Jan 06 2015) >> >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >> >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >> >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ >> ________________________________________________________________________ >> >> ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: >> >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >> Registered at Amtsgericht Duesseldorf: HRB 46611 >> http://www.egenix.com/company/contact/ >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150106/410321f0/attachment.html> From guido at python.org Tue Jan 6 20:31:41 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 6 Jan 2015 11:31:41 -0800 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <54AB9D76.7030105@egenix.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> Message-ID: <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> If you ask me the bigger problem with the Wiki is not how to encourage or manage small edits, it is how to deal with the entire wiki gradually getting out of date due to page "owners" losing interest or topics becoming irrelevant. (A random example: https://wiki.python.org/moin/MostPopularPythonProjects is 10 years old, which makes it actively harmful given its lofty title and linkage from other pages.) An early Python contributor, Ken Manheimer, described the necessary activity as "wiki gardening". We need more wiki gardeners, people who have an eye for the big picture and actively edit content, not just guards who passively judge contribute proposed changes (even that's also needed -- Python is sufficiently well-known that the wiki would attract a lot of spam and occasional vandalism if it was completely open). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150106/2081f837/attachment.html> From eltoder at gmail.com Tue Jan 6 20:35:54 2015 From: eltoder at gmail.com (Eugene Toder) Date: Tue, 6 Jan 2015 14:35:54 -0500 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <97D50920-B7D2-4125-9389-224C4CBC83CA@yahoo.com> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> <54AB4178.3050508@gmail.com> <97D50920-B7D2-4125-9389-224C4CBC83CA@yahoo.com> Message-ID: <CA+KNMznGRf=Q3Cv_6+e7mJXtTmwR37ozw+ocFCLEXkf=9HHb+g@mail.gmail.com> On Tue, Jan 6, 2015 at 3:25 AM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: > I think you're missing another important point here: statements and expressions are different things. Blocks are made up of statements, not expressions, and statements don't have values. This is different from many other languages like C, Ruby, or JavaScript, where as many things as possible are expressions (and therefore have values), so a block is (oversimplifying a bit) just a sequence of expressions separated by semicolons and stuck inside braces, and therefore it has a last value. Statements and expressions are separate in C and Javascript, and statements don't have values. This is the reason, for example, for having the ?: operator, and for having statement expressions in GNU C [1]. This separation is common in C-like languages. Some C-like languages that have REPL invent a rule for statement values similar to Python's REPL. > One type of statement, the expression statement, is just an expression on a line by itself (or between semicolons), so you _could_ invent a rule pretty easily that expression statements have the value of their expression, and all other statements have a value of None, and a block has the value of its last statement. This is basically the same rule used by the interactive REPL for displaying the repr of what you've typed and setting the _ variable, so it wouldn't be a huge stretch to add the same rule to the language itself--but it would still be a new rule, and a significant change to the internals of the interpreter, even though it would only be useful in this new context. I don't think one needs to invent this rule in the context of this idea. All that is needed is to end the "return expression" (or whatever you want to call it) with an expression rather than a statement (i.e. return_expr ::= "keyword" statement* expression). The value of this last expression is the value of the whole thing. [1] https://gcc.gnu.org/onlinedocs/gcc-3.1/gcc/Statement-Exprs.html From p.f.moore at gmail.com Tue Jan 6 20:39:39 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 6 Jan 2015 19:39:39 +0000 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <CAP7+vJ+-PrJh07Ra-ru4QyCwMwMKNaRFbPCPmnxXPV5bybBaHA@mail.gmail.com> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <1BC2F2DA-DC73-4BE3-8550-37F5EA872C07@gmail.com> <CACac1F8kwFs5g4VbCaKwbwGSE=gHG8JHkZ=ZTcCgpyZ9WxRqyA@mail.gmail.com> <20150106193655.2be8dcf0@fsol> <CACac1F9xnLe5Sv_-vCMH6FBurJRrfU7VONXW5rsWbsfWzozb2g@mail.gmail.com> <CAP7+vJ+-PrJh07Ra-ru4QyCwMwMKNaRFbPCPmnxXPV5bybBaHA@mail.gmail.com> Message-ID: <CACac1F8CBjDe3UaNsEm7__hncL97XUyHjiWP_KNjB_dFQsEv8w@mail.gmail.com> On 6 January 2015 at 18:47, Guido van Rossum <guido at python.org> wrote: > Folks, I realize I'm sounding a bit like Raymond here, but can you all > please find a different forum to discuss algorithms and coding problems and > pick each other's solutions apart? The focus on python-ideas should be to > quickly validate ideas for the language or the stdlib, not to discuss > arbitrary snippets of code. Agreed. I'm happy with the consensus (reached pretty quickly) that this isn't useful enough for the stdlib. Thanks all. Paul From skip.montanaro at gmail.com Tue Jan 6 20:39:38 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Tue, 6 Jan 2015 13:39:38 -0600 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> Message-ID: <CANc-5UzU+o8K1QXLGnybVnj8LvW4HgMA8ZVkM+D+KAyxYXt+pQ@mail.gmail.com> On Tue, Jan 6, 2015 at 1:31 PM, Guido van Rossum <guido at python.org> wrote: > it is how to deal with the entire wiki gradually getting out of date due > to page "owners" losing interest or topics becoming irrelevant. Even if you can recruit lots of gardeners, you need a few master gardeners to lay things out (define the structure of the garden). Otherwise you wind up with just a bag of pages. Pushing the gardening metaphor to its limit: no planning means the lettuce is always shaded by the corn. Skip -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150106/f3ee53fe/attachment.html> From guido at python.org Tue Jan 6 20:44:34 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 6 Jan 2015 11:44:34 -0800 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <CANc-5UzU+o8K1QXLGnybVnj8LvW4HgMA8ZVkM+D+KAyxYXt+pQ@mail.gmail.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> <CANc-5UzU+o8K1QXLGnybVnj8LvW4HgMA8ZVkM+D+KAyxYXt+pQ@mail.gmail.com> Message-ID: <CAP7+vJLg6Guyusw6DhSYrAP1VYXsWnM-1RaYoURbr14k+2Gq-A@mail.gmail.com> On Tue, Jan 6, 2015 at 11:39 AM, Skip Montanaro <skip.montanaro at gmail.com> wrote: > > > On Tue, Jan 6, 2015 at 1:31 PM, Guido van Rossum <guido at python.org> wrote: > >> it is how to deal with the entire wiki gradually getting out of date due >> to page "owners" losing interest or topics becoming irrelevant. > > > Even if you can recruit lots of gardeners, you need a few master gardeners > to lay things out (define the structure of the garden). Otherwise you wind > up with just a bag of pages. Pushing the gardening metaphor to its limit: > no planning means the lettuce is always shaded by the corn. > Nice one, and agreed. I don't see anyone with a serious wish to be a master gardener for wiki.python.org in this sense though. :-( Perhaps we should advertise the position? It's a volunteer role, but will require a lot of motivation. Ideally the master gardener team should be allowed to select the tool suite and be given permission to switch to a new suite pretty aggressively. The team should also be responsible for deciding on the policy for edit access. This seems more workable than having an open-ended discussion on python-ideas. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150106/f13a7743/attachment.html> From skip.montanaro at gmail.com Tue Jan 6 20:51:19 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Tue, 6 Jan 2015 13:51:19 -0600 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <CACfEFw9mUcf11bbPz0r4rBASNOdbgTdTbzN0jV=Q7wZLBa9yxg@mail.gmail.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CACfEFw9mUcf11bbPz0r4rBASNOdbgTdTbzN0jV=Q7wZLBa9yxg@mail.gmail.com> Message-ID: <CANc-5UxAMdJCAa58TsJNi16U1r++wpoRvjKo1XhDLjH-nLjOQA@mail.gmail.com> On Tue, Jan 6, 2015 at 1:25 PM, Wes Turner <wes.turner at gmail.com> wrote: > I'm not a big fan of Moin syntax (can Pandoc convert this?). I haven't followed wiki markup at all in the past several years. Back when Moin came along, it was (arguably?) the correct choice for wiki.python.org. I think most of us active at the time were very much in the "eat your own dog food" camp at the time. I've never heard of Pandoc (I grow old, can't follow every trend), but I thought there were some tools which could convert Moin into other formats. I can't find much now, though this https://ikiwiki.info/tips/convert_moinmoin_to_ikiwiki/ looks like it might be a reasonable starting point. Skip -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150106/02f72d5f/attachment.html> From wes.turner at gmail.com Tue Jan 6 20:59:00 2015 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 6 Jan 2015 13:59:00 -0600 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> Message-ID: <CACfEFw-ZaP-u3TVS+EzVPY5=cyjWrvCfWD4MSFZ9H_QE-EiYjg@mail.gmail.com> On Tue, Jan 6, 2015 at 1:31 PM, Guido van Rossum <guido at python.org> wrote: > > An early Python contributor, Ken Manheimer, described the necessary > activity as "wiki gardening". We need more wiki gardeners, people who have > an eye for the big picture and actively edit content, not just guards who > passively judge contribute proposed changes (even that's also needed -- > Python is sufficiently well-known that the wiki would attract a lot of spam > and occasional vandalism if it was completely open). > > The edx wiki seems to avoid many of these challenges. https://github.com/edx/edx-platform/wiki/How-to-Rebase-a-Pull-Request Other than versioning docs/ with the code and encouraging helpful community members to provide a valuable service, IDK of any other good solutions. As far as change control, pull requests (of the wiki branch) seem to work great for just me. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150106/e2b1001d/attachment-0001.html> From donald at stufft.io Tue Jan 6 21:24:53 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 6 Jan 2015 15:24:53 -0500 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <CAP7+vJLg6Guyusw6DhSYrAP1VYXsWnM-1RaYoURbr14k+2Gq-A@mail.gmail.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> <CANc-5UzU+o8K1QXLGnybVnj8LvW4HgMA8ZVkM+D+KAyxYXt+pQ@mail.gmail.com> <CAP7+vJLg6Guyusw6DhSYrAP1VYXsWnM-1RaYoURbr14k+2Gq-A@mail.gmail.com> Message-ID: <80F2EFB1-07F4-4966-9DCB-6BBD50B50D42@stufft.io> > On Jan 6, 2015, at 2:44 PM, Guido van Rossum <guido at python.org> wrote: > > On Tue, Jan 6, 2015 at 11:39 AM, Skip Montanaro <skip.montanaro at gmail.com <mailto:skip.montanaro at gmail.com>> wrote: > > > On Tue, Jan 6, 2015 at 1:31 PM, Guido van Rossum <guido at python.org <mailto:guido at python.org>> wrote: > it is how to deal with the entire wiki gradually getting out of date due to page "owners" losing interest or topics becoming irrelevant. > > Even if you can recruit lots of gardeners, you need a few master gardeners to lay things out (define the structure of the garden). Otherwise you wind up with just a bag of pages. Pushing the gardening metaphor to its limit: no planning means the lettuce is always shaded by the corn. > > Nice one, and agreed. I don't see anyone with a serious wish to be a master gardener for wiki.python.org <http://wiki.python.org/> in this sense though. :-( Perhaps we should advertise the position? It's a volunteer role, but will require a lot of motivation. Ideally the master gardener team should be allowed to select the tool suite and be given permission to switch to a new suite pretty aggressively. The team should also be responsible for deciding on the policy for edit access. This seems more workable than having an open-ended discussion on python-ideas. I have very little opinion on what the wiki is or does but I wanted to just say if at some point we do switch software then with my infrastructure team hat on I would ask for two properties to exist in the software (not really specific to the wiki tbh): 1. Configuration that is able to be handled via config management like salt/chef/puppet etc. Typically this just means that the configuration is handled via a text file and the software itself doesn?t attempt to write to the configuration as part of it?s normal execution. 2. State does not need to be stored on the local file system and instead can be stored in something like a database (preferably PostgreSQL) or an object store (preferably cloudfiles). This allows us to treat the servers running the software as emphereal with all of the state stored elsewhere. This makes it easier to manage the servers, makes it easier to recover from security issues on a server, makes it easier to upgrade the server, and makes it easier to scale the service in general. Of the two, the first property is the most important, but the second one is a really good idea too. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150106/53e8e9fc/attachment.html> From cs at zip.com.au Tue Jan 6 22:04:55 2015 From: cs at zip.com.au (Cameron Simpson) Date: Wed, 7 Jan 2015 08:04:55 +1100 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <1BC2F2DA-DC73-4BE3-8550-37F5EA872C07@gmail.com> References: <1BC2F2DA-DC73-4BE3-8550-37F5EA872C07@gmail.com> Message-ID: <20150106210455.GA65122@cskk.homeip.net> On 06Jan2015 09:14, Raymond Hettinger <raymond.hettinger at gmail.com> wrote: >> On Jan 6, 2015, at 8:22 AM, Paul Moore <p.f.moore at gmail.com> wrote: >> In writing a utility script today, I found myself needing to do >> something similar to what the Unix "comm" utility does - take two >> sorted iterators, and partition the values into "only in the first", >> "only in the second", and "in both" groups. > >As far as I can tell, this would be a very rare need. Really? I do this on an ad hoc basis in shell scripts a lot. I think it might just be rare for you. In Python I would generally be choosing to use sets, but that is at least partially because sets are there and the stdlib doesn't have a comm. With sets, I inherently need finite sources, and I also don't get to yield results progressively from ordered iterators. Also, it needs to fit into memory. The most obvious Python use case I happen to actually have to hand would almost be an abuse of the suggested comm(): a log merge tool I wrote for merging multiple logs; in that case the "common" set is always empty, hence the "abuse" idea; it isn't really abuse, just a corner use case. It gets run many times every day. Reviewing the code, I notice it starts with: from heapq import merge It strikes me that it might be easier to write comm() as a wrapper to heapq.merge. Though that wouldn't handle Steven's "unorderable items" case. Cheers, Cameron Simpson <cs at zip.com.au> If it sticks, force it. If it breaks, it needed replacing anyway. From cs at zip.com.au Tue Jan 6 22:09:59 2015 From: cs at zip.com.au (Cameron Simpson) Date: Wed, 7 Jan 2015 08:09:59 +1100 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <20150106193655.2be8dcf0@fsol> References: <20150106193655.2be8dcf0@fsol> Message-ID: <20150106210959.GA87112@cskk.homeip.net> On 06Jan2015 19:36, Antoine Pitrou <solipsis at pitrou.net> wrote: >On Tue, 6 Jan 2015 18:22:44 +0000 >Paul Moore <p.f.moore at gmail.com> wrote: >> On 6 January 2015 at 17:14, Raymond Hettinger >> <raymond.hettinger at gmail.com> wrote: >> >> On Jan 6, 2015, at 8:22 AM, Paul Moore <p.f.moore at gmail.com> wrote: >> >> >> >> In writing a utility script today, I found myself needing to do >> >> something similar to what the Unix "comm" utility does - take two >> >> sorted iterators, and partition the values into "only in the first", >> >> "only in the second", and "in both" groups. >> > >> > As far as I can tell, this would be a very rare need. >> >> It's come up for me a few times, usually when trying to check two >> lists of files to see which ones have been missed by a program, and >> which ones the program thinks are present but no longer exist. > >Why don't you use sets for such things? Your iterator is really only >useful for huge or unhashable inputs. In my use case (an existing tool): 1) I'm merging log files of arbitrary size; I am _not_ going to suck them into memory. A comm()-like function has a tiny and fixed memory footprint, versus an unbounded out. 2) I want ordered output, and my inputs are already ordered; why on earth would I impose a pointless sorting cost on my (currently linear) runtime? Sets are the "obvious" Python way to do this, because comm() is more or less a set intersection operation and sets are right there in Python. But for unbounded sorted inputs and progressive output, they are a _bad_ choice. Cheers, Cameron Simpson <cs at zip.com.au> Yesterday, I was running a CNC plasma cutter that's controlled by Windows XP. This is a machine that moves around a plasma torch that cuts thick steel plate. ?A "New Java update is available" window popped up while I was working. ?Not good. - John Nagle From solipsis at pitrou.net Tue Jan 6 22:24:48 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 6 Jan 2015 22:24:48 +0100 Subject: [Python-ideas] Possible new itertool: comm() References: <20150106193655.2be8dcf0@fsol> <20150106210959.GA87112@cskk.homeip.net> Message-ID: <20150106222448.574f82fa@fsol> On Wed, 7 Jan 2015 08:09:59 +1100 Cameron Simpson <cs at zip.com.au> wrote: > On 06Jan2015 19:36, Antoine Pitrou <solipsis at pitrou.net> wrote: > >On Tue, 6 Jan 2015 18:22:44 +0000 > >Paul Moore <p.f.moore at gmail.com> wrote: > >> On 6 January 2015 at 17:14, Raymond Hettinger > >> <raymond.hettinger at gmail.com> wrote: > >> >> On Jan 6, 2015, at 8:22 AM, Paul Moore <p.f.moore at gmail.com> wrote: > >> >> > >> >> In writing a utility script today, I found myself needing to do > >> >> something similar to what the Unix "comm" utility does - take two > >> >> sorted iterators, and partition the values into "only in the first", > >> >> "only in the second", and "in both" groups. > >> > > >> > As far as I can tell, this would be a very rare need. > >> > >> It's come up for me a few times, usually when trying to check two > >> lists of files to see which ones have been missed by a program, and > >> which ones the program thinks are present but no longer exist. > > > >Why don't you use sets for such things? Your iterator is really only > >useful for huge or unhashable inputs. > > In my use case (an existing tool): > > 1) I'm merging log files of arbitrary size; I am _not_ going to suck them into > memory. A comm()-like function has a tiny and fixed memory footprint, versus an > unbounded out. I don't understand what your use case has to do with comm(). If you just want to merge sorted iterators you don't need all the complication this function has. Regards Antoine. From wes.turner at gmail.com Tue Jan 6 22:48:25 2015 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 6 Jan 2015 15:48:25 -0600 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <80F2EFB1-07F4-4966-9DCB-6BBD50B50D42@stufft.io> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> <CANc-5UzU+o8K1QXLGnybVnj8LvW4HgMA8ZVkM+D+KAyxYXt+pQ@mail.gmail.com> <CAP7+vJLg6Guyusw6DhSYrAP1VYXsWnM-1RaYoURbr14k+2Gq-A@mail.gmail.com> <80F2EFB1-07F4-4966-9DCB-6BBD50B50D42@stufft.io> Message-ID: <CACfEFw_xj9RtYJ5urPfZKUvHkPL501JmvtX7E7DNBrL6N_Eqag@mail.gmail.com> On Tue, Jan 6, 2015 at 2:24 PM, Donald Stufft <donald at stufft.io> wrote: > > I have very little opinion on what the wiki is or does but I wanted to > just say if at some point we do switch software then with my infrastructure > team hat on I would ask for two properties to exist in the software (not > really specific to the wiki tbh): > > 1. Configuration that is able to be handled via config management like > salt/chef/puppet etc. Typically this just means that the configuration is > handled via a text file and the software itself doesn?t attempt to write to > the configuration as part of it?s normal execution. > +1 * http://psf-salt.readthedocs.org/en/latest/ * https://github.com/python/psf-salt > > 2. State does not need to be stored on the local file system and instead > can be stored in something like a database (preferably PostgreSQL) or an > object store (preferably cloudfiles). This allows us to treat the servers > running the software as emphereal with all of the state stored elsewhere. > This makes it easier to manage the servers, makes it easier to recover from > security issues on a server, makes it easier to upgrade the server, and > makes it easier to scale the service in general. > > Of the two, the first property is the most important, but the second one > is a really good idea too. > > re: persistence * difflib * hg diff * git diff structured data * http://www.w3.org/2001/sw/wiki/Tools (RDF; FreeBase is moving to WikiData) * https://westurner.github.io/wiki/tools * https://westurner.github.io/dotfiles/tools.html#python * https://westurner.github.io/dotfiles/_sources/tools.txt (this is the simplest ReST markup I could think of, a sphinxcontrib-rdf directive/role could be really useful, but wouldn't work w/ GitHub wikis without a Sphinx compile step) : .. index:: Python > .. _python: Python > ================= > | Wikipedia: `<https://en.wikipedia.org/wiki/Python_(programming_language) > >`_ > | Homepage: https://www.python.org/ > | Docs: https://docs.python.org/2/ > | Docs: https://docs.python.org/devguide/ > | Docs: https://docs.python.org/devguide/documenting.html > | Docs: http://learnxinyminutes.com/docs/python/ > | Source: hg https://hg.python.org/cpython Python is a dynamically-typed, :ref:`C`-based third-generation > programming language. [...] -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150106/2aff56ca/attachment.html> From mal at egenix.com Tue Jan 6 23:01:19 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 06 Jan 2015 23:01:19 +0100 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <CAP7+vJLg6Guyusw6DhSYrAP1VYXsWnM-1RaYoURbr14k+2Gq-A@mail.gmail.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> <CANc-5UzU+o8K1QXLGnybVnj8LvW4HgMA8ZVkM+D+KAyxYXt+pQ@mail.gmail.com> <CAP7+vJLg6Guyusw6DhSYrAP1VYXsWnM-1RaYoURbr14k+2Gq-A@mail.gmail.com> Message-ID: <54AC5B2F.6090504@egenix.com> On 06.01.2015 20:44, Guido van Rossum wrote: > On Tue, Jan 6, 2015 at 11:39 AM, Skip Montanaro <skip.montanaro at gmail.com> > wrote: >> On Tue, Jan 6, 2015 at 1:31 PM, Guido van Rossum <guido at python.org> wrote: >> >>> it is how to deal with the entire wiki gradually getting out of date due >>> to page "owners" losing interest or topics becoming irrelevant. >> >> >> Even if you can recruit lots of gardeners, you need a few master gardeners >> to lay things out (define the structure of the garden). Otherwise you wind >> up with just a bag of pages. Pushing the gardening metaphor to its limit: >> no planning means the lettuce is always shaded by the corn. >> > > Nice one, and agreed. I don't see anyone with a serious wish to be a master > gardener for wiki.python.org in this sense though. :-( Perhaps we should > advertise the position? It's a volunteer role, but will require a lot of > motivation. Ideally the master gardener team should be allowed to select > the tool suite and be given permission to switch to a new suite pretty > aggressively. The team should also be responsible for deciding on the > policy for edit access. This seems more workable than having an open-ended > discussion on python-ideas. Such discussions should really happen on pydotorg-www where that team already works. We do have several people who maintain pages or page sets on the wiki, but more editorial help is always welcome. We need people who have experience in technical writing and a passion to maintain informational resources. FWIW: A change of tools won't magically give us better content. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 06 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From wes.turner at gmail.com Tue Jan 6 23:11:44 2015 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 6 Jan 2015 16:11:44 -0600 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <54AC5B2F.6090504@egenix.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <CAPTjJmpOEeUhPqqqBHP1ykDEMGED92ntw5ENp0w-CwKtuSSDUQ@mail.gmail.com> <29AE11E7-7347-4C9C-BC4E-996DFA6C1754@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> <CANc-5UzU+o8K1QXLGnybVnj8LvW4HgMA8ZVkM+D+KAyxYXt+pQ@mail.gmail.com> <CAP7+vJLg6Guyusw6DhSYrAP1VYXsWnM-1RaYoURbr14k+2Gq-A@mail.gmail.com> <54AC5B2F.6090504@egenix.com> Message-ID: <CACfEFw8uQwBq7ThcaeDtvUMmBRZnVZE1ZKTLefU0Da+6JXxR1Q@mail.gmail.com> On Tue, Jan 6, 2015 at 4:01 PM, M.-A. Lemburg <mal at egenix.com> wrote: > On 06.01.2015 20:44, Guido van Rossum wrote: > > On Tue, Jan 6, 2015 at 11:39 AM, Skip Montanaro < > skip.montanaro at gmail.com> > > wrote: > >> On Tue, Jan 6, 2015 at 1:31 PM, Guido van Rossum <guido at python.org> > wrote: > >> > >>> it is how to deal with the entire wiki gradually getting out of date > due > >>> to page "owners" losing interest or topics becoming irrelevant. > >> > >> > >> Even if you can recruit lots of gardeners, you need a few master > gardeners > >> to lay things out (define the structure of the garden). Otherwise you > wind > >> up with just a bag of pages. Pushing the gardening metaphor to its > limit: > >> no planning means the lettuce is always shaded by the corn. > >> > > > > Nice one, and agreed. I don't see anyone with a serious wish to be a > master > > gardener for wiki.python.org in this sense though. :-( Perhaps we should > > advertise the position? It's a volunteer role, but will require a lot of > > motivation. Ideally the master gardener team should be allowed to select > > the tool suite and be given permission to switch to a new suite pretty > > aggressively. The team should also be responsible for deciding on the > > policy for edit access. This seems more workable than having an > open-ended > > discussion on python-ideas. > > Such discussions should really happen on pydotorg-www where > that team already works. We do have several people who maintain pages > or page sets on the wiki, but more editorial help is always welcome. > https://mail.python.org/mailman/listinfo/pydotorg-www https://www.python.org/dev/pydotorg/website/ > > We need people who have experience in technical writing and > a passion to maintain informational resources. > > FWIW: A change of tools won't magically give us better content. > A boost in familiar usability could encourage effortless contributions. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150106/0ebe832f/attachment-0001.html> From ethan at stoneleaf.us Tue Jan 6 23:48:18 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 06 Jan 2015 14:48:18 -0800 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <CACfEFw8uQwBq7ThcaeDtvUMmBRZnVZE1ZKTLefU0Da+6JXxR1Q@mail.gmail.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> <CANc-5UzU+o8K1QXLGnybVnj8LvW4HgMA8ZVkM+D+KAyxYXt+pQ@mail.gmail.com> <CAP7+vJLg6Guyusw6DhSYrAP1VYXsWnM-1RaYoURbr14k+2Gq-A@mail.gmail.com> <54AC5B2F.6090504@egenix.com> <CACfEFw8uQwBq7ThcaeDtvUMmBRZnVZE1ZKTLefU0Da+6JXxR1Q@mail.gmail.com> Message-ID: <54AC6632.5040204@stoneleaf.us> On 01/06/2015 02:11 PM, Wes Turner wrote: > On Tue, Jan 6, 2015 at 4:01 PM, M.-A. Lemburg wrote: >> >> Such discussions should really happen on pydotorg-www where >> that team already works. We do have several people who maintain pages >> or page sets on the wiki, but more editorial help is always welcome. > > https://www.python.org/dev/pydotorg/website/ Subversion, huh? > A boost in familiar usability could encourage effortless contributions. Not even breathing is effortless. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150106/e82b2aa9/attachment.sig> From mal at egenix.com Wed Jan 7 00:04:02 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 07 Jan 2015 00:04:02 +0100 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <CACfEFw8uQwBq7ThcaeDtvUMmBRZnVZE1ZKTLefU0Da+6JXxR1Q@mail.gmail.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> <CANc-5UzU+o8K1QXLGnybVnj8LvW4HgMA8ZVkM+D+KAyxYXt+pQ@mail.gmail.com> <CAP7+vJLg6Guyusw6DhSYrAP1VYXsWnM-1RaYoURbr14k+2Gq-A@mail.gmail.com> <54AC5B2F.6090504@egenix.com> <CACfEFw8uQwBq7ThcaeDtvUMmBRZnVZE1ZKTLefU0Da+6JXxR1Q@mail.gmail.com> Message-ID: <54AC69E2.6070303@egenix.com> On 06.01.2015 23:11, Wes Turner wrote: > On Tue, Jan 6, 2015 at 4:01 PM, M.-A. Lemburg <mal at egenix.com> wrote: > >> On 06.01.2015 20:44, Guido van Rossum wrote: >>> On Tue, Jan 6, 2015 at 11:39 AM, Skip Montanaro < >> skip.montanaro at gmail.com> >>> wrote: >>>> On Tue, Jan 6, 2015 at 1:31 PM, Guido van Rossum <guido at python.org> >> wrote: >>>> >>>>> it is how to deal with the entire wiki gradually getting out of date >> due >>>>> to page "owners" losing interest or topics becoming irrelevant. >>>> >>>> >>>> Even if you can recruit lots of gardeners, you need a few master >> gardeners >>>> to lay things out (define the structure of the garden). Otherwise you >> wind >>>> up with just a bag of pages. Pushing the gardening metaphor to its >> limit: >>>> no planning means the lettuce is always shaded by the corn. >>>> >>> >>> Nice one, and agreed. I don't see anyone with a serious wish to be a >> master >>> gardener for wiki.python.org in this sense though. :-( Perhaps we should >>> advertise the position? It's a volunteer role, but will require a lot of >>> motivation. Ideally the master gardener team should be allowed to select >>> the tool suite and be given permission to switch to a new suite pretty >>> aggressively. The team should also be responsible for deciding on the >>> policy for edit access. This seems more workable than having an >> open-ended >>> discussion on python-ideas. >> >> Such discussions should really happen on pydotorg-www where >> that team already works. We do have several people who maintain pages >> or page sets on the wiki, but more editorial help is always welcome. >> > > https://mail.python.org/mailman/listinfo/pydotorg-www Yep, that's the list. > https://www.python.org/dev/pydotorg/website/ That page explains the old python.org website system. It doesn't have anything to do with wiki.python.org. >> We need people who have experience in technical writing and >> a passion to maintain informational resources. >> >> FWIW: A change of tools won't magically give us better content. >> > > A boost in familiar usability could encourage effortless contributions. The wiki does support ReST syntax if you don't like the wiki markup: http://moinmo.in/HelpOnParsers/ReStructuredText and it also supports WYSIWYG editing using FCKeditor (even though the generated markup code doesn't look all that nice). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 06 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From victor.stinner at gmail.com Wed Jan 7 00:11:51 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 7 Jan 2015 00:11:51 +0100 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> Message-ID: <CAMpsgwZc4+iZ5z5EzAO4JZ_R4-fVHKppxRXrGqaOUkWzbdT1Ww@mail.gmail.com> Hi, This idea was already proposed as part of a larger PEP: https://www.python.org/dev/peps/pep-0471/#wildcard-support The PEP explains why the Windows wildcard idea was rejected. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/82845b71/attachment.html> From mal at egenix.com Wed Jan 7 00:12:33 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 07 Jan 2015 00:12:33 +0100 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <54AC69E2.6070303@egenix.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> <CANc-5UzU+o8K1QXLGnybVnj8LvW4HgMA8ZVkM+D+KAyxYXt+pQ@mail.gmail.com> <CAP7+vJLg6Guyusw6DhSYrAP1VYXsWnM-1RaYoURbr14k+2Gq-A@mail.gmail.com> <54AC5B2F.6090504@egenix.com> <CACfEFw8uQwBq7ThcaeDtvUMmBRZnVZE1ZKTLefU0Da+6JXxR1Q@mail.gmail.com> <54AC69E2.6070303@egenix.co m> Message-ID: <54AC6BE1.5000505@egenix.com> While we're in wishlist mode: A contribution we'd really appreciate is a moin 1.9 theme adapted to the new python.org style :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 07 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From wes.turner at gmail.com Wed Jan 7 00:21:33 2015 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 6 Jan 2015 17:21:33 -0600 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <54AC69E2.6070303@egenix.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> <CANc-5UzU+o8K1QXLGnybVnj8LvW4HgMA8ZVkM+D+KAyxYXt+pQ@mail.gmail.com> <CAP7+vJLg6Guyusw6DhSYrAP1VYXsWnM-1RaYoURbr14k+2Gq-A@mail.gmail.com> <54AC5B2F.6090504@egenix.com> <CACfEFw8uQwBq7ThcaeDtvUMmBRZnVZE1ZKTLefU0Da+6JXxR1Q@mail.gmail.com> <54AC69E2.6070303@egenix.com> Message-ID: <CACfEFw9sQAd1NrfM7YOLwyjmgPPO9e_5iLtdQkTsDN17Ff13Lg@mail.gmail.com> On Tue, Jan 6, 2015 at 5:04 PM, M.-A. Lemburg <mal at egenix.com> wrote: > > > https://mail.python.org/mailman/listinfo/pydotorg-www > > Yep, that's the list. > > > https://www.python.org/dev/pydotorg/website/ > > That page explains the old python.org website system. It > doesn't have anything to do with wiki.python.org. > My mistake. That page does seem outdated. I must've been looking for one level up: https://www.python.org/dev/pydotorg/ What content should be version controlled with the website? > > >> We need people who have experience in technical writing and > >> a passion to maintain informational resources. > >> > >> FWIW: A change of tools won't magically give us better content. > >> > > > > A boost in familiar usability could encourage effortless contributions. > > The wiki does support ReST syntax if you don't like the wiki > markup: > > http://moinmo.in/HelpOnParsers/ReStructuredText Ah, thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150106/62c2bd90/attachment-0001.html> From random832 at fastmail.us Wed Jan 7 00:35:05 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Tue, 06 Jan 2015 18:35:05 -0500 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <CAMpsgwZc4+iZ5z5EzAO4JZ_R4-fVHKppxRXrGqaOUkWzbdT1Ww@mail.gmail.com> References: <CANXboVbxWRJgFmUahq4A32aALvZRp3Qc-Ah+vrHJ3v0ipJY2bw@mail.gmail.com> <20141229185326.19663a01@fsol> <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> <CAMpsgwZc4+iZ5z5EzAO4JZ_R4-fVHKppxRXrGqaOUkWzbdT1Ww@mail.gmail.com> Message-ID: <1420587305.879324.210454157.1BA5CE04@webmail.messagingengine.com> On Tue, Jan 6, 2015, at 18:11, Victor Stinner wrote: > Hi, > > This idea was already proposed as part of a larger PEP: > https://www.python.org/dev/peps/pep-0471/#wildcard-support Not the same idea, really. The _main_ thrust of this idea is to have a way to have a single function (OS-specific algorithms or otherwise) which acts (roughly) like itertools.chain(map(glob,lst)) on windows and simply uses the original list unmodified on unix, and to use this in places like fileinput. The point is that command line argument wildcard processing belongs in a different place on each platform. The windows-specific matching algorithm is just an aside. From steve at pearwood.info Wed Jan 7 01:42:50 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 7 Jan 2015 11:42:50 +1100 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <CAMpsgwZc4+iZ5z5EzAO4JZ_R4-fVHKppxRXrGqaOUkWzbdT1Ww@mail.gmail.com> References: <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> <CAMpsgwZc4+iZ5z5EzAO4JZ_R4-fVHKppxRXrGqaOUkWzbdT1Ww@mail.gmail.com> Message-ID: <20150107004249.GF19387@ando.pearwood.info> On Wed, Jan 07, 2015 at 12:11:51AM +0100, Victor Stinner wrote: > Hi, > > This idea was already proposed as part of a larger PEP: > https://www.python.org/dev/peps/pep-0471/#wildcard-support > > The PEP explains why the Windows wildcard idea was rejected. The Windows wildcard idea was rejected in the context of a faster os.walk, not globbing. The PEP is inaccurate: it gives the wrong Python-Ideas thread for discussions on the wildcard idea. The wildcard idea is discussed in this thread: https://mail.python.org/pipermail/python-ideas/2012-November/017965.html not in the thread given in the PEP. Well, I say "discussed", but it's mostly ignored, with only a couple of comments. There's somewhat more discussion on Python-Dev: https://mail.python.org/pipermail/python-dev/2014-June/135217.html but it's mostly people just saying "-1 on wildcards". I agree with this decision, for PEP 471. But I don't believe this is relevant to the question of explicit globbing. -- Steve From stephen at xemacs.org Wed Jan 7 02:16:15 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 07 Jan 2015 10:16:15 +0900 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <54AC6632.5040204@stoneleaf.us> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <CAPTjJmraupWUvrTpBt=f=sof7+ibpQx++V3aEzO4sYFd-s8OAg@mail.gmail.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> <CANc-5UzU+o8K1QXLGnybVnj8LvW4HgMA8ZVkM+D+KAyxYXt+pQ@mail.gmail.com> <CAP7+vJLg6Guyusw6DhSYrAP1VYXsWnM-1RaYoURbr14k+2Gq-A@mail.gmail.com> <54AC5B2F.6090504@egenix.com> <CACfEFw8uQwBq7ThcaeDtvUMmBRZnVZE1ZKTLefU0Da+6JXxR1Q@mail.gmail.com> <54AC6632.5040204@stoneleaf.us> Message-ID: <87387n1jc0.fsf@uwakimon.sk.tsukuba.ac.jp> Ethan Furman writes: > On 01/06/2015 02:11 PM, Wes Turner wrote: > > A boost in familiar usability could encourage effortless contributions. > > Not even breathing is effortless. True, but that's the wrong fallacy. The energy required is not the point, it's the thinking. Wes's point is that he'd like it to be possible to contribute to Python without being aware of the mechanics of contribution, just as it's possible to breath without being aware of your lungs most of the time. That's reasonable, and wikis actually do work in some contexts. The fallacy in applying wiki-think to Python is that for new and small projects with SEI "Level Zero" processes, *increase in quantity* of documentation is *improvement in quality* of documentation. That's no longer true of Python, and hasn't been for two decades (at least). It isn't even clear that filling an obvious hole with an accurate-as- far-as-it-goes drive-by contribution is a good thing. Not only may the residual inaccuracy mislead the reader, it may also contribute to an impression that the Python docs are lower quality on average than they actually are. From ethan at stoneleaf.us Wed Jan 7 02:28:13 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 06 Jan 2015 17:28:13 -0800 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <87387n1jc0.fsf@uwakimon.sk.tsukuba.ac.jp> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> <CANc-5UzU+o8K1QXLGnybVnj8LvW4HgMA8ZVkM+D+KAyxYXt+pQ@mail.gmail.com> <CAP7+vJLg6Guyusw6DhSYrAP1VYXsWnM-1RaYoURbr14k+2Gq-A@mail.gmail.com> <54AC5B2F.6090504@egenix.com> <CACfEFw8uQwBq7ThcaeDtvUMmBRZnVZE1ZKTLefU0Da+6JXxR1Q@mail.gmail.com> <54AC6632.5040204@stoneleaf.us> <87387n1jc0.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <54AC8BAD.8000203@stoneleaf.us> On 01/06/2015 05:16 PM, Stephen J. Turnbull wrote: > Ethan Furman writes: >> On 01/06/2015 02:11 PM, Wes Turner wrote: >>> >>> A boost in familiar usability could encourage effortless contributions. >> >> Not even breathing is effortless. > > True, but that's the wrong fallacy. [...] > > The fallacy in applying wiki-think to Python is that [an] *increase in > quantity* of documentation is [not necessarily an] *improvement in quality* > of documentation. [...] > > It isn't even clear that filling an obvious hole with an accurate-as- > far-as-it-goes drive-by contribution is a good thing. Not only may > the residual inaccuracy mislead the reader, it may also contribute to > an impression that the Python docs are lower quality on average than > they actually are. Thanks -- that's worded much better than my own feeble attempt. :) -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150106/dc41c9c2/attachment.sig> From yawar.amin at gmail.com Wed Jan 7 03:08:46 2015 From: yawar.amin at gmail.com (Yawar Amin) Date: Tue, 06 Jan 2015 21:08:46 -0500 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <97D50920-B7D2-4125-9389-224C4CBC83CA@yahoo.com> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> <54AB4178.3050508@gmail.com> <97D50920-B7D2-4125-9389-224C4CBC83CA@yahoo.com> Message-ID: <54AC952E.5060301@gmail.com> On 2015-01-06 03:25, Andrew Barnert wrote: > [...] > I think you're missing another important point here: statements and > expressions are different things. Blocks are made up of statements, > not expressions, ... Actually I can offer counter-examples to that: def p(x): print x class A: p("Hello!") class B: 1 # Etc. But I think I see what's happening here: statements are the top dogs in Python, and when Python wants a statement but only has an expression, it 'promotes' the expression into a statement by evaluating it, throwing away its value, and pretending nothing happened (i.e., that there was a 'pass' statement there). Thus defining a class can have the nonsensical effect of calling a function. But there's currently no way of going in the other direction, i.e. demoting a statement to an expression. Which is what I was trying to do. I _still_ think at least something like the following would work (in terms of Python's grammar[1]): expr_expr: 'expr' ':' small_stmt (';' small_stmt)* [';' expr] So e.g.: x = expr: import os; os.system("date") I'll explore this further and see where it goes. Regards, Yawar [1] https://docs.python.org/3/reference/grammar.html -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 834 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150106/780627c4/attachment.sig> From cs at zip.com.au Wed Jan 7 03:14:40 2015 From: cs at zip.com.au (Cameron Simpson) Date: Wed, 7 Jan 2015 13:14:40 +1100 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <20150106222448.574f82fa@fsol> References: <20150106222448.574f82fa@fsol> Message-ID: <20150107021440.GA34389@cskk.homeip.net> On 06Jan2015 22:24, Antoine Pitrou <solipsis at pitrou.net> wrote: >On Wed, 7 Jan 2015 08:09:59 +1100 >Cameron Simpson <cs at zip.com.au> wrote: >> On 06Jan2015 19:36, Antoine Pitrou <solipsis at pitrou.net> wrote: >> >On Tue, 6 Jan 2015 18:22:44 +0000 >> >Paul Moore <p.f.moore at gmail.com> wrote: >> >> On 6 January 2015 at 17:14, Raymond Hettinger >> >> <raymond.hettinger at gmail.com> wrote: >> >> >> On Jan 6, 2015, at 8:22 AM, Paul Moore <p.f.moore at gmail.com> wrote: >> >> >> >> >> >> In writing a utility script today, I found myself needing to do >> >> >> something similar to what the Unix "comm" utility does - take two >> >> >> sorted iterators, and partition the values into "only in the first", >> >> >> "only in the second", and "in both" groups. >> >> > >> >> > As far as I can tell, this would be a very rare need. >> >> >> >> It's come up for me a few times, usually when trying to check two >> >> lists of files to see which ones have been missed by a program, and >> >> which ones the program thinks are present but no longer exist. >> > >> >Why don't you use sets for such things? Your iterator is really only >> >useful for huge or unhashable inputs. >> >> In my use case (an existing tool): >> >> 1) I'm merging log files of arbitrary size; I am _not_ going to suck them into >> memory. A comm()-like function has a tiny and fixed memory footprint, versus an >> unbounded out. > >I don't understand what your use case has to do with comm(). I've got two! I, like another poster, also very commonly compare two similar directory trees for commonality, and equivalent list comparisons. >If you >just want to merge sorted iterators you don't need all the complication >this function has. Indeed not, but they are very similar tasks: you're pulling in 2 or more streams of sorted inputs and classifying them. In my direct use case, all into the same output stream, in order. In comm(), into three streams (well, being an iterator output: one stream with three classifications). Cheers, Cameron Simpson <cs at zip.com.au> The double cam chain setup on the 1980's DOHC CB750 was another one of Honda's pointless engineering breakthroughs. You know the cycle (if you'll pardon the pun :-), Wonderful New Feature is introduced with much fanfare, WNF is fawned over by the press, WNF is copied by the other three Japanese makers (this step is sometimes optional), and finally, WNF is quietly dropped by Honda. - Blaine Gardner, <blgardne at sim.es.com> From rosuav at gmail.com Wed Jan 7 03:28:34 2015 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 7 Jan 2015 13:28:34 +1100 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <54AC952E.5060301@gmail.com> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> <54AB4178.3050508@gmail.com> <97D50920-B7D2-4125-9389-224C4CBC83CA@yahoo.com> <54AC952E.5060301@gmail.com> Message-ID: <CAPTjJmp3h-zOgJKg7=zRMCfyjNx0S_oh7+wSxw+zpeThYkcQKQ@mail.gmail.com> On Wed, Jan 7, 2015 at 1:08 PM, Yawar Amin <yawar.amin at gmail.com> wrote: > But I think I see what's happening here: statements are the top dogs in > Python, and when Python wants a statement but only has an expression, it > 'promotes' the expression into a statement by evaluating it, throwing > away its value, and pretending nothing happened (i.e., that there was a > 'pass' statement there). Thus defining a class can have the nonsensical > effect of calling a function. One type of statement is (via a couple of levels of indirection) the expr_stmt, which contains an expression and (if I'm not misreading the grammar) may or may not assign it to anything. https://docs.python.org/3/reference/grammar.html It's not that an expression gets "promoted", it's that one valid form of statement is an expression with nothing around it. In C, the definition is more like "an expression followed by a semicolon", but in Python, you don't need any adornment, so it looks identical. However, there is (as far as I know) no way in Python to wrap a statement into an expression. Some languages have such a thing (eg a lambda function syntax, which is an expression returning a function, and in which statements are allowed), but I don't think Python does, anywhere; all compound units that allow statements are themselves statements. ChrisA From jim.baker at python.org Wed Jan 7 04:30:56 2015 From: jim.baker at python.org (Jim Baker) Date: Tue, 6 Jan 2015 20:30:56 -0700 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <20150107021440.GA34389@cskk.homeip.net> References: <20150106222448.574f82fa@fsol> <20150107021440.GA34389@cskk.homeip.net> Message-ID: <CAOhO=aN7R3Cac0T69q0WTndS+CPzsRpWsZeMYXrz8+OYTw7Onw@mail.gmail.com> The comm functionality sounds very reasonable for inclusion. But please call it itertools.common. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150106/c24509f6/attachment.html> From python at mrabarnett.plus.com Wed Jan 7 04:41:24 2015 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 07 Jan 2015 03:41:24 +0000 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <54AC952E.5060301@gmail.com> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> <54AB4178.3050508@gmail.com> <97D50920-B7D2-4125-9389-224C4CBC83CA@yahoo.com> <54AC952E.5060301@gmail.com> Message-ID: <54ACAAE4.3090505@mrabarnett.plus.com> On 2015-01-07 02:08, Yawar Amin wrote: > On 2015-01-06 03:25, Andrew Barnert wrote: >> [...] >> I think you're missing another important point here: statements and >> expressions are different things. Blocks are made up of statements, >> not expressions, ... > > Actually I can offer counter-examples to that: > > def p(x): print x > class A: p("Hello!") > class B: 1 > # Etc. > > But I think I see what's happening here: statements are the top dogs in > Python, and when Python wants a statement but only has an expression, it > 'promotes' the expression into a statement by evaluating it, throwing > away its value, and pretending nothing happened (i.e., that there was a > 'pass' statement there). Thus defining a class can have the nonsensical > effect of calling a function. > > But there's currently no way of going in the other direction, i.e. > demoting a statement to an expression. Which is what I was trying to do. > I _still_ think at least something like the following would work (in > terms of Python's grammar[1]): > > expr_expr: 'expr' ':' small_stmt (';' small_stmt)* [';' expr] > > So e.g.: > > x = expr: import os; os.system("date") > [snip] In BCPL, the body of a routine can be only a single statement, but that statement can be a block: LET foo() BE $( .... $) Similarly, the body of a function can be only an expression, but that expression can include VALOF. VALOF is followed by a block, and the value of VALOF is given by the expression after RESULTIS: LET foo() = VALOF $( ... RESULTIS ... ... $) Of course, in Python, the block would be indented, so it's not entirely suitable, but you could imagine this: x = valof: import os resultis os.system("date") I think you're better off using a function instead! From abarnert at yahoo.com Wed Jan 7 05:43:49 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 6 Jan 2015 23:43:49 -0500 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <CA+KNMznGRf=Q3Cv_6+e7mJXtTmwR37ozw+ocFCLEXkf=9HHb+g@mail.gmail.com> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> <54AB4178.3050508@gmail.com> <97D50920-B7D2-4125-9389-224C4CBC83CA@yahoo.com> <CA+KNMznGRf=Q3Cv_6+e7mJXtTmwR37ozw+ocFCLEXkf=9HHb+g@mail.gmail.com> Message-ID: <D003557A-AC4C-464D-8593-25ED659E2503@yahoo.com> LOn Jan 6, 2015, at 14:35, Eugene Toder <eltoder at gmail.com> wrote: > On Tue, Jan 6, 2015 at 3:25 AM, Andrew Barnert > <abarnert at yahoo.com.dmarc.invalid> wrote: >> I think you're missing another important point here: statements and expressions are different things. Blocks are made up of statements, not expressions, and statements don't have values. This is different from many other languages like C, Ruby, or JavaScript, where as many things as possible are expressions (and therefore have values), so a block is (oversimplifying a bit) just a sequence of expressions separated by semicolons and stuck inside braces, and therefore it has a last value. > Statements and expressions are separate in C and Javascript, Well, obviously; otherwise "as many things as possible are expressions" would be pointless. In C, unlike Python (and, more to the point, unlike many of C's predecessors), assignment, augmented assignment, and increment are expressions. Different languages have since taken that farther: Ruby makes most flow control into expressions, JavaScript makes function definitions into expressions (which means you can wrap any statement in an expression by defining and calling a function around it), CoffeeScript makes everything but return, continue, and break into expressions. >> One type of statement, the expression statement, is just an expression on a line by itself (or between semicolons), so you _could_ invent a rule pretty easily that expression statements have the value of their expression, and all other statements have a value of None, and a block has the value of its last statement. This is basically the same rule used by the interactive REPL for displaying the repr of what you've typed and setting the _ variable, so it wouldn't be a huge stretch to add the same rule to the language itself--but it would still be a new rule, and a significant change to the internals of the interpreter, even though it would only be useful in this new context. > I don't think one needs to invent this rule in the context of this > idea. All that is needed is to end the "return expression" (or > whatever you want to call it) with an expression rather than a > statement (i.e. return_expr ::= "keyword" statement* expression). The > value of this last expression is the value of the whole thing. That's equivalent to arguing that function bodies don't need return, break, or continue, because you can theoretically write anything to fall off the end. I'm assuming that the OP wanted something that works intuitively similar to other languages, the REPL, etc.--e.g., if the last statement executed is in the true block of an if/else, it doesn't matter that it's not textually the last statement. From yawar.amin at gmail.com Wed Jan 7 06:29:27 2015 From: yawar.amin at gmail.com (Yawar Amin) Date: Wed, 07 Jan 2015 00:29:27 -0500 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <D003557A-AC4C-464D-8593-25ED659E2503@yahoo.com> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> <54AB4178.3050508@gmail.com> <97D50920-B7D2-4125-9389-224C4CBC83CA@yahoo.com> <CA+KNMznGRf=Q3Cv_6+e7mJXtTmwR37ozw+ocFCLEXkf=9HHb+g@mail.gmail.com> <D003557A-AC4C-464D-8593-25ED659E2503@yahoo.com> Message-ID: <54ACC437.20608@gmail.com> On 2015-01-06 23:43, Andrew Barnert wrote: > [...] > That's equivalent to arguing that function bodies don't need return, > break, or continue, because you can theoretically write anything to > fall off the end. Not really; it's more like arguing that the particular proposed new syntax doesn't need a return statement because it is defined to automatically return its last expression (i.e., 'fall off the end'). All other control structures and syntax keep their existing behaviour, including functions. And in any case, many languages don't have any concept of a return statement; they _do_ actually return the last expression in their function bodies. > I'm assuming that the OP wanted something that works intuitively > similar to other languages, the REPL, etc.--e.g., if the last > statement executed is in the true block of an if/else, it doesn't > matter that it's not textually the last statement. Not exactly, I think ... I just wanted a way to 'wrap' statements inside expressions. That would unify the two concepts in Python and make it much more expressive. Regards, Yawar -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 834 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/d6691a09/attachment-0001.sig> From ncoghlan at gmail.com Wed Jan 7 07:15:32 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 7 Jan 2015 16:15:32 +1000 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <54ACC437.20608@gmail.com> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> <54AB4178.3050508@gmail.com> <97D50920-B7D2-4125-9389-224C4CBC83CA@yahoo.com> <CA+KNMznGRf=Q3Cv_6+e7mJXtTmwR37ozw+ocFCLEXkf=9HHb+g@mail.gmail.com> <D003557A-AC4C-464D-8593-25ED659E2503@yahoo.com> <54ACC437.20608@gmail.com> Message-ID: <CADiSq7dj_wKmCt2TpzrRQUmQ1HKiXi_WNLztuB=XjykDzTjNjA@mail.gmail.com> On 7 January 2015 at 15:29, Yawar Amin <yawar.amin at gmail.com> wrote: > Not exactly, I think ... I just wanted a way to 'wrap' statements inside > expressions. That would unify the two concepts in Python The data modelling and control flow aspects of Python, as represented by statements, are deliberately distinct from the computational aspects, as represented by expressions. Some constructs blur the boundaries between computation, data modelling and control flow, and hence may exist in both statement and expression forms. The fact that it permits data modelling and control flow constructs to be embedded inside computational ones is an argument *against* your proposal, not one in its favour. > and make it much more expressive. Allowing data modelling and control flow constructs to be embedded inside expressions doesn't make the language more expressive overall, it just lets you do more within a single expression without giving the results of suboperations names and/or factoring them out into separate usable functions. The number of cases where executing a suboperation first and binding the result to a name even arguably reduces clarity are relatively few and far between, and better represented in PEPs like 403 and 3150 than they are in a general purpose statement-as-expression syntax. So if you're interested in pursuing this further, I suggest focusing on the possible pragmatic benefits of defining a standard way to tunnel Python code through whitespace insensitive contexts and reformat it with a pretty printer at the far end, rather than the far more nebulous concept of expressiveness. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mal at egenix.com Wed Jan 7 09:49:55 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 07 Jan 2015 09:49:55 +0100 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <CACfEFw9sQAd1NrfM7YOLwyjmgPPO9e_5iLtdQkTsDN17Ff13Lg@mail.gmail.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> <CANc-5UzU+o8K1QXLGnybVnj8LvW4HgMA8ZVkM+D+KAyxYXt+pQ@mail.gmail.com> <CAP7+vJLg6Guyusw6DhSYrAP1VYXsWnM-1RaYoURbr14k+2Gq-A@mail.gmail.com> <54AC5B2F.6090504@egenix.com> <CACfEFw8uQwBq7ThcaeDtvUMmBRZnVZE1ZKTLefU0Da+6JXxR1Q@mail.gmail.com> <54AC69E2.6070303@egenix.com> <CACfEFw9sQAd1NrfM7YOLwyjmgPPO9e_5iLtdQkTsDN17Ff13Lg@mail.gmail.co m> Message-ID: <54ACF333.80307@egenix.com> On 07.01.2015 00:21, Wes Turner wrote: > On Tue, Jan 6, 2015 at 5:04 PM, M.-A. Lemburg <mal at egenix.com> wrote: > >> >>> https://mail.python.org/mailman/listinfo/pydotorg-www >> >> Yep, that's the list. >> >>> https://www.python.org/dev/pydotorg/website/ >> >> That page explains the old python.org website system. It >> doesn't have anything to do with wiki.python.org. >> > > My mistake. That page does seem outdated. > > I must've been looking for one level up: > https://www.python.org/dev/pydotorg/ Sorry to say, but that content is outdated as well :-( > What content should be version controlled with the website? The current python.org website is a mix of code and templates in github repo, content/data stored in a database and a (for various reasons) homegrown CMS to manage pages such as the ones you found. There's work underway to replace the homegrown CMS with a standard more feature rich one, which will hopefully allow us to open up site editing to way more volunteers than we currently have. wiki.python.org is a separate VM, which doesn't have anything to do with www.python.org. The VM runs a more or less standard moin installation, which uses its own version control system (you can access this via the "info" link on the pages). >>>> We need people who have experience in technical writing and >>>> a passion to maintain informational resources. >>>> >>>> FWIW: A change of tools won't magically give us better content. >>>> >>> >>> A boost in familiar usability could encourage effortless contributions. >> >> The wiki does support ReST syntax if you don't like the wiki >> markup: >> >> http://moinmo.in/HelpOnParsers/ReStructuredText > > > Ah, thanks! -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 07 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From wes.turner at gmail.com Wed Jan 7 12:00:24 2015 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 7 Jan 2015 05:00:24 -0600 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <54ACF333.80307@egenix.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> <CANc-5UzU+o8K1QXLGnybVnj8LvW4HgMA8ZVkM+D+KAyxYXt+pQ@mail.gmail.com> <CAP7+vJLg6Guyusw6DhSYrAP1VYXsWnM-1RaYoURbr14k+2Gq-A@mail.gmail.com> <54AC5B2F.6090504@egenix.com> <CACfEFw8uQwBq7ThcaeDtvUMmBRZnVZE1ZKTLefU0Da+6JXxR1Q@mail.gmail.com> <54AC69E2.6070303@egenix.com> <CACfEFw9sQAd1NrfM7YOLwyjmgPPO9e_5iLtdQkTsDN17Ff13Lg@mail.gmail.com> <54ACF333.80307@egenix.com> Message-ID: <CACfEFw-hB_YSrSEp3w7PKxaJisGZW--+HikZwRvjpPeppdY4UQ@mail.gmail.com> On Wed, Jan 7, 2015 at 2:49 AM, M.-A. Lemburg <mal at egenix.com> wrote: > > > > What content should be version controlled with the website? > > The current python.org website is a mix of code and templates > in github repo, content/data stored in a database and a (for > various reasons) homegrown CMS to manage pages such as > the ones you found. > > There's work underway to replace the homegrown CMS with a > standard more feature rich one, which will hopefully allow us > to open up site editing to way more volunteers than we > currently have. > > So there are no topical guidelines? > wiki.python.org is a separate VM, which doesn't have anything > to do with www.python.org. The VM runs a more or less standard > moin installation, which uses its own version control system > (you can access this via the "info" link on the pages). > > https://github.com/dwf/moin2rst Formatter plugin > Simply put ``RenderAsRestructuredtext.py`` to MoinMoin's > ``plugin/formatter`` directory. > Action plugin > Simply put ``RenderAsRestructuredtext.py`` to MoinMoin's ``plugin/action`` > directory. > >>>> We need people who have experience in technical writing and > >>>> a passion to maintain informational resources. > >>>> > >>>> FWIW: A change of tools won't magically give us better content. > >>>> > >>> > >>> A boost in familiar usability could encourage effortless contributions. > >> > >> The wiki does support ReST syntax if you don't like the wiki > >> markup: > >> > >> http://moinmo.in/HelpOnParsers/ReStructuredText > > > > > > Ah, thanks! > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/ad991e47/attachment.html> From abarnert at yahoo.com Wed Jan 7 08:11:32 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 7 Jan 2015 02:11:32 -0500 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <54AC952E.5060301@gmail.com> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> <54AB4178.3050508@gmail.com> <97D50920-B7D2-4125-9389-224C4CBC83CA@yahoo.com> <54AC952E.5060301@gmail.com> Message-ID: <74164843-0774-441F-A898-4776E15EFC8D@yahoo.com> On Jan 6, 2015, at 21:08, Yawar Amin <yawar.amin at gmail.com> wrote: > Actually I can offer counter-examples to that: > > def p(x): print x > class A: p("Hello!") > class B: 1 > # Etc. No, there are no counter-examples there. A top-level program is a sequence of statements. Both def and class are compound statements, meaning the colon is followed by either a simple statement list or an indented block of statements. An expression statement is a kind of simple statement containing nothing but an expression. > But I think I see what's happening here: statements are the top dogs in > Python, and when Python wants a statement but only has an expression, it > 'promotes' the expression into a statement by evaluating it, throwing > away its value, and pretending nothing happened (i.e., that there was a > 'pass' statement there). Thus defining a class can have the nonsensical > effect of calling a function. > > But there's currently no way of going in the other direction, i.e. > demoting a statement to an expression. Which is what I was trying to do. That's the key. A function is a way to wrap a sequence of statements (the function body) so it can be used in an expression (a function call). Because JavaScript lets you define functions (with full statement syntax) in expressions, that gives you a way to "demote" an expression inline, while Python can only do so out-of-line. At the core, this is what all of the multiline lambda attempts and similar proposals are trying to accomplish. From skip.montanaro at gmail.com Wed Jan 7 14:27:23 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Wed, 7 Jan 2015 07:27:23 -0600 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <CACfEFw-hB_YSrSEp3w7PKxaJisGZW--+HikZwRvjpPeppdY4UQ@mail.gmail.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> <CANc-5UzU+o8K1QXLGnybVnj8LvW4HgMA8ZVkM+D+KAyxYXt+pQ@mail.gmail.com> <CAP7+vJLg6Guyusw6DhSYrAP1VYXsWnM-1RaYoURbr14k+2Gq-A@mail.gmail.com> <54AC5B2F.6090504@egenix.com> <CACfEFw8uQwBq7ThcaeDtvUMmBRZnVZE1ZKTLefU0Da+6JXxR1Q@mail.gmail.com> <54AC69E2.6070303@egenix.com> <CACfEFw9sQAd1NrfM7YOLwyjmgPPO9e_5iLtdQkTsDN17Ff13Lg@mail.gmail.com> <54ACF333.80307@egenix.com> <CACfEFw-hB_YSrSEp3w7PKxaJisGZW--+HikZwRvjpPeppdY4UQ@mail.gmail.com> Message-ID: <CANc-5UyZRAZbZhhX6qXyfxfJTUa9EsbSOUvqGqxkObPjCne6dQ@mail.gmail.com> I'm going to add one more bit here about wiki gardening. BITD, the main website had many pages which today would be called listicles. The problem was that those lists were dynamic. They weren't just the "top ten scientific Python modules of all time" or the "seven best packaging tools." They were whatever was at the front of the author's brain at some point in the past. Having these static pages given the imprimatur of the PSF when in fact none of the site maintainers were obviously responsible for (or interested in) keeping them up-to-date did a disservice to the community and to authors of other packages which weren't represented in those lists. In addition, there were more barriers to update than necessary (essentially, figure out how to report the problem, offer suggested fix(es), then have them swallowed up into the site update mechanism). One of the main uses envisioned for the wiki was as a place where these listicles could be maintained by the greater Python community. For some things, that's worked out pretty well. The most obvious thing that comes to mind is the PythonTraining page. That works because the people whose skills are represented on that page have a very good reason for keeping things up-to-date: it's free advertising for their businesses. Other listicle type pages haven't been keep as up-to-date. For example, the PythonEditors page was last updated in Feb 2014, is huge (and might benefit from being split into multiple pages), and probably no longer accurately represents the available editors or IDEs which support Python. The takeaway in my mind is that we could probably use "gardeners" to take over active maintenance of these listicle pages. That, coupled with a couple "master gardeners" to develop some suitable structure, and some landscape crews to prune dead/outdated/no-longer-useful pages, would likely go a long way to improving the quality of the wiki. Skip From toddrjen at gmail.com Wed Jan 7 14:33:54 2015 From: toddrjen at gmail.com (Todd) Date: Wed, 7 Jan 2015 14:33:54 +0100 Subject: [Python-ideas] math.inf and math.nan constants Message-ID: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> The current way to get infinite and nan floats are to do float("inf") and float("nan"), respectively. For many applications, this is sufficient. However, for scientific and mathematical computing, using inf and nan values is very common, and the current approach is very verbose. So for those applications, I think it would be useful if the math standard library module provided "math.nan" and "math.inf", which would return the same thing as float("nan") and float("inf"), respectively. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/b6e9f357/attachment.html> From marky1991 at gmail.com Wed Jan 7 14:49:32 2015 From: marky1991 at gmail.com (Mark Young) Date: Wed, 7 Jan 2015 08:49:32 -0500 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> Message-ID: <CAG3cHaaXAXs87ZK=wk1q191HR2tt0EVdZTFSgo=2xLO_feujbw@mail.gmail.com> So with the proposal, you'd likely do something like from math import nan, inf Couldn't you just instead put nan, inf = float("nan"), float("inf") at the top of your script? I don't really think this is any more verbose than an import from the math module. (Unless you're already doing an import star from math) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/ccf2607b/attachment-0001.html> From victor.stinner at gmail.com Wed Jan 7 15:19:31 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 7 Jan 2015 15:19:31 +0100 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <1BC2F2DA-DC73-4BE3-8550-37F5EA872C07@gmail.com> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <1BC2F2DA-DC73-4BE3-8550-37F5EA872C07@gmail.com> Message-ID: <CAMpsgwZGZ14KyOWOSAZYbocU78sQ87Rgo4hB6Ba+mb=xm7Stsg@mail.gmail.com> Hi, 2015-01-06 18:14 GMT+01:00 Raymond Hettinger <raymond.hettinger at gmail.com>: >> On Jan 6, 2015, at 8:22 AM, Paul Moore <p.f.moore at gmail.com> wrote: >> >> In writing a utility script today, I found myself needing to do >> something similar to what the Unix "comm" utility does - take two >> sorted iterators, and partition the values into "only in the first", >> "only in the second", and "in both" groups. > > As far as I can tell, this would be a very rare need. I never used the UNIX comm tool. I didn't know that it exists :-) Can you maybe explain the purpose the tool/your function? Show an example. difflib may be a better candidate to add such function than contextlib. Victor From victor.stinner at gmail.com Wed Jan 7 15:29:14 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 7 Jan 2015 15:29:14 +0100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> Message-ID: <CAMpsgwavyQc5QL91GfbgLG=9G9c3mpg5JD_7BNk-FWUvy5Y+TA@mail.gmail.com> 2015-01-07 14:33 GMT+01:00 Todd <toddrjen at gmail.com>: > The current way to get infinite and nan floats are to do float("inf") and > float("nan"), respectively. For many applications, this is sufficient. > However, for scientific and mathematical computing, using inf and nan values > is very common, and the current approach is very verbose. So for those > applications, I think it would be useful if the math standard library module > provided "math.nan" and "math.inf", which would return the same thing as > float("nan") and float("inf"), respectively. Do you expect singletons? I mean that float("nan") is math.nan would be True. Request of float("0.0") singleton: http://bugs.python.org/issue4024 I dislike the idea of adding math.<float constant>. I agree with Mark Young, you are already create you own symbols in a module. Victor From random832 at fastmail.us Wed Jan 7 15:44:26 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Wed, 07 Jan 2015 09:44:26 -0500 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <20150107004249.GF19387@ando.pearwood.info> References: <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> <CAMpsgwZc4+iZ5z5EzAO4JZ_R4-fVHKppxRXrGqaOUkWzbdT1Ww@mail.gmail.com> <20150107004249.GF19387@ando.pearwood.info> Message-ID: <1420641866.1163622.210704857.0D786745@webmail.messagingengine.com> On Tue, Jan 6, 2015, at 19:42, Steven D'Aprano wrote: > The PEP is inaccurate: it gives the wrong Python-Ideas thread for > discussions on the wildcard idea. The wildcard idea is discussed in this > thread: > > https://mail.python.org/pipermail/python-ideas/2012-November/017965.html I think the fundamental difference between these cases is in handling user-entered strings, especially command-line arguments (where the user will expect them to behave a certain way, in particular "*.*" matching all files even those with no dot - most of the other quirks are obscure anyway) vs hardcoded (where expectations are determined by the programmer, not the user). From ethan at stoneleaf.us Wed Jan 7 15:55:51 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 07 Jan 2015 06:55:51 -0800 Subject: [Python-ideas] Add OS-dependent automatic glob support In-Reply-To: <1420641866.1163622.210704857.0D786745@webmail.messagingengine.com> References: <7D7CB381-EE2A-4931-89DB-E6661A541715@yahoo.com> <54A1E167.1050605@canterbury.ac.nz> <B2B07DC1-C34E-4977-984E-1141E8579C6A@mac.com> <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1420205066.2836204.208806917.69EED9B5@webmail.messagingengine.com> <20150103015045.GB27426@ando.pearwood.info> <1420399476.1260565.209437029.38EF0C44@webmail.messagingengine.com> <54A9AA6F.8020400@stoneleaf.us> <1420413001.1294229.209490357.7C0E4C45@webmail.messagingengine.com> <CAMpsgwZc4+iZ5z5EzAO4JZ_R4-fVHKppxRXrGqaOUkWzbdT1Ww@mail.gmail.com> <20150107004249.GF19387@ando.pearwood.info> <1420641866.1163622.210704857.0D786745@webmail.messagingengine.com> Message-ID: <54AD48F7.6000407@stoneleaf.us> On 01/07/2015 06:44 AM, random832 at fastmail.us wrote: > On Tue, Jan 6, 2015, at 19:42, Steven D'Aprano wrote: >> The PEP is inaccurate: it gives the wrong Python-Ideas thread for >> discussions on the wildcard idea. The wildcard idea is discussed in this >> thread: >> >> https://mail.python.org/pipermail/python-ideas/2012-November/017965.html > > I think the fundamental difference between these cases is in handling > user-entered strings, especially command-line arguments (where the user > will expect them to behave a certain way, in particular "*.*" matching > all files even those with no dot - most of the other quirks are obscure > anyway) vs hardcoded (where expectations are determined by the > programmer, not the user). But is the programmer a *nix programmer or a Win programmer? ;) -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/c43e301a/attachment.sig> From jeanpierreda at gmail.com Wed Jan 7 15:55:57 2015 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Wed, 7 Jan 2015 08:55:57 -0600 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAMpsgwavyQc5QL91GfbgLG=9G9c3mpg5JD_7BNk-FWUvy5Y+TA@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <CAMpsgwavyQc5QL91GfbgLG=9G9c3mpg5JD_7BNk-FWUvy5Y+TA@mail.gmail.com> Message-ID: <CABicbJJDGRGYMdrhj+tTo_Qz+QdxP0axnQcVefFbSXP7X-Kayg@mail.gmail.com> On Wed, Jan 7, 2015 at 8:29 AM, Victor Stinner <victor.stinner at gmail.com> wrote: > 2015-01-07 14:33 GMT+01:00 Todd <toddrjen at gmail.com>: >> The current way to get infinite and nan floats are to do float("inf") and >> float("nan"), respectively. For many applications, this is sufficient. >> However, for scientific and mathematical computing, using inf and nan values >> is very common, and the current approach is very verbose. So for those >> applications, I think it would be useful if the math standard library module >> provided "math.nan" and "math.inf", which would return the same thing as >> float("nan") and float("inf"), respectively. > > Do you expect singletons? I mean that float("nan") is math.nan would be True. That seems like a bad idea. There are millions of different NaNs, so xxx is math.nan would only work sometimes. (math.isnan is the way to deal with this.) -- Devin From toddrjen at gmail.com Wed Jan 7 15:58:37 2015 From: toddrjen at gmail.com (Todd) Date: Wed, 7 Jan 2015 15:58:37 +0100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAG3cHaaXAXs87ZK=wk1q191HR2tt0EVdZTFSgo=2xLO_feujbw@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <CAG3cHaaXAXs87ZK=wk1q191HR2tt0EVdZTFSgo=2xLO_feujbw@mail.gmail.com> Message-ID: <CAFpSVp+1ee8mM_xbPkANeNjWJGRr8zAd=SFe+VC5EO-2L4P2Dg@mail.gmail.com> Yes, this would be a shortcut for that. math.nan and math.inf are less verbose than float("nan") and float("inf"), even if you aren't already using the math module (fewer characters total, and in particular fewer special characters). It would also, at least in my opinion, be more readable. Plus there is already the math.isnan and math.isinf, so I don't think it would be totally out-of-place. On Wed, Jan 7, 2015 at 2:49 PM, Mark Young <marky1991 at gmail.com> wrote: > So with the proposal, you'd likely do something like > > from math import nan, inf > > Couldn't you just instead put > > nan, inf = float("nan"), float("inf") > > at the top of your script? I don't really think this is any more verbose > than an import from the math module. (Unless you're already doing an import > star from math) > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/dc15c470/attachment-0001.html> From solipsis at pitrou.net Wed Jan 7 16:03:08 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 7 Jan 2015 16:03:08 +0100 Subject: [Python-ideas] math.inf and math.nan constants References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> Message-ID: <20150107160308.7415d5dd@fsol> On Wed, 7 Jan 2015 14:33:54 +0100 Todd <toddrjen at gmail.com> wrote: > The current way to get infinite and nan floats are to do float("inf") and > float("nan"), respectively. For many applications, this is sufficient. > However, for scientific and mathematical computing, using inf and nan > values is very common, and the current approach is very verbose. So for > those applications, I think it would be useful if the math standard library > module provided "math.nan" and "math.inf", which would return the same > thing as float("nan") and float("inf"), respectively. Note if you are using Numpy, you already have np.nan and np.inf. Regards Antoine. From solipsis at pitrou.net Wed Jan 7 16:02:02 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 7 Jan 2015 16:02:02 +0100 Subject: [Python-ideas] math.inf and math.nan constants References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <CAMpsgwavyQc5QL91GfbgLG=9G9c3mpg5JD_7BNk-FWUvy5Y+TA@mail.gmail.com> Message-ID: <20150107160202.531577d8@fsol> On Wed, 7 Jan 2015 15:29:14 +0100 Victor Stinner <victor.stinner at gmail.com> wrote: > 2015-01-07 14:33 GMT+01:00 Todd <toddrjen at gmail.com>: > > The current way to get infinite and nan floats are to do float("inf") and > > float("nan"), respectively. For many applications, this is sufficient. > > However, for scientific and mathematical computing, using inf and nan values > > is very common, and the current approach is very verbose. So for those > > applications, I think it would be useful if the math standard library module > > provided "math.nan" and "math.inf", which would return the same thing as > > float("nan") and float("inf"), respectively. > > Do you expect singletons? I mean that float("nan") is math.nan would be True. > > Request of float("0.0") singleton: http://bugs.python.org/issue4024 > > I dislike the idea of adding math.<float constant>. You mean like math.pi and math.e? :) From marco.buttu at gmail.com Wed Jan 7 16:23:29 2015 From: marco.buttu at gmail.com (Marco Buttu) Date: Wed, 07 Jan 2015 16:23:29 +0100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAFpSVp+1ee8mM_xbPkANeNjWJGRr8zAd=SFe+VC5EO-2L4P2Dg@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <CAG3cHaaXAXs87ZK=wk1q191HR2tt0EVdZTFSgo=2xLO_feujbw@mail.gmail.com> <CAFpSVp+1ee8mM_xbPkANeNjWJGRr8zAd=SFe+VC5EO-2L4P2Dg@mail.gmail.com> Message-ID: <54AD4F71.7060008@oa-cagliari.inaf.it> On 07/01/2015 15:58, Todd wrote: > Yes, this would be a shortcut for that. math.nan and math.inf are > less verbose than float("nan") and float("inf"), even if you aren't > already using the math module (fewer characters total, and in > particular fewer special characters). It would also, at least in my > opinion, be more readable. Plus there is already the math.isnan and > math.isinf, so I don't think it would be totally out-of-place. I totally agree. +1 -- Marco Buttu INAF-Osservatorio Astronomico di Cagliari Via della Scienza n. 5, 09047 Selargius (CA) Phone: 070 711 80 217 Email: mbuttu at oa-cagliari.inaf.it From wes.turner at gmail.com Wed Jan 7 16:24:26 2015 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 7 Jan 2015 09:24:26 -0600 Subject: [Python-ideas] build:// Path / URI / URN with the __div__ operator Message-ID: <CACfEFw9Fes0iW9cz-juk8jk9pchC7wcQhDUKiVX+2tDcgUBS2Q@mail.gmail.com> * [ ] build Path / URI / URN with the __div__ operator * [ ] pick some combination of operators for URI schema, query, fragment conjunction (like URLObject.with_<*>):: # scheme://user at host.netloc:port/path/?query#fragment from _ import URI, https, //, http https // 'user at host.netloc:port' / path ** query % fragment * __div__ / * __floordiv__ // * __and__ && * __mod__ % * __invert__ ~ * __or__ | * __xor__ ^ * __and__ + * __mul__ * * __pow__ ** * __lshift__ << * __rshift__ >> References ----------------- * https://pypi.python.org/pypi/path.py -- __div__ * https://pypi.python.org/pypi/pathlib -- [ Standard Library ] * https://pypi.python.org/pypi/furl * https://pypi.python.org/pypi/URLObject - [ with_scheme, with_netloc, with_path, ... ] * https://urlobject.readthedocs.org/en/latest/quickstart.html * https://pypi.python.org/pypi/rdflib -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/358439a8/attachment.html> From p.f.moore at gmail.com Wed Jan 7 16:32:45 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 7 Jan 2015 15:32:45 +0000 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <CAMpsgwZGZ14KyOWOSAZYbocU78sQ87Rgo4hB6Ba+mb=xm7Stsg@mail.gmail.com> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <1BC2F2DA-DC73-4BE3-8550-37F5EA872C07@gmail.com> <CAMpsgwZGZ14KyOWOSAZYbocU78sQ87Rgo4hB6Ba+mb=xm7Stsg@mail.gmail.com> Message-ID: <CACac1F-vLSi1=CzSnKg29Yg0o4KkSvZ0fPgbqJpUEFjMQFA=Og@mail.gmail.com> On 7 January 2015 at 14:19, Victor Stinner <victor.stinner at gmail.com> wrote: > I never used the UNIX comm tool. I didn't know that it exists :-) > > Can you maybe explain the purpose the tool/your function? Show an example. The function has doctests included which may help. Basically, take 2 sorted lists (the Unix tool takes 2 sorted files of text), and merge them, preserving order, with each line marked as "only occurs in the first input", "only occurs in the second input" and "occurs in both inputs". So, for example, with inputs 1,1,2,4 and 1,3,4 we would get both: 1 first: 1 first: 2 second: 3 both: 4 The problem with difflib/diff is mainly that it does a *lot* more work than is needed - for sorted input, a fast single-pass algorithm is fine, whereas a general diff needs to work to find common subsequences. Paul From p.f.moore at gmail.com Wed Jan 7 16:36:02 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 7 Jan 2015 15:36:02 +0000 Subject: [Python-ideas] build:// Path / URI / URN with the __div__ operator In-Reply-To: <CACfEFw9Fes0iW9cz-juk8jk9pchC7wcQhDUKiVX+2tDcgUBS2Q@mail.gmail.com> References: <CACfEFw9Fes0iW9cz-juk8jk9pchC7wcQhDUKiVX+2tDcgUBS2Q@mail.gmail.com> Message-ID: <CACac1F9sRUndHk8WgEP4wCEMArPxg_w=nC4QLzg2xCSx2rLg8A@mail.gmail.com> On 7 January 2015 at 15:24, Wes Turner <wes.turner at gmail.com> wrote: > * [ ] build Path / URI / URN with the __div__ operator > * [ ] pick some combination of operators for URI schema, query, fragment > conjunction (like URLObject.with_<*>):: [...] That was way too terse. What are you proposing? Paul From steve at pearwood.info Wed Jan 7 16:38:48 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 8 Jan 2015 02:38:48 +1100 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <CAMpsgwZGZ14KyOWOSAZYbocU78sQ87Rgo4hB6Ba+mb=xm7Stsg@mail.gmail.com> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <1BC2F2DA-DC73-4BE3-8550-37F5EA872C07@gmail.com> <CAMpsgwZGZ14KyOWOSAZYbocU78sQ87Rgo4hB6Ba+mb=xm7Stsg@mail.gmail.com> Message-ID: <20150107153847.GG19387@ando.pearwood.info> On Wed, Jan 07, 2015 at 03:19:31PM +0100, Victor Stinner wrote: > Hi, > > 2015-01-06 18:14 GMT+01:00 Raymond Hettinger <raymond.hettinger at gmail.com>: > >> On Jan 6, 2015, at 8:22 AM, Paul Moore <p.f.moore at gmail.com> wrote: > >> > >> In writing a utility script today, I found myself needing to do > >> something similar to what the Unix "comm" utility does - take two > >> sorted iterators, and partition the values into "only in the first", > >> "only in the second", and "in both" groups. > > > > As far as I can tell, this would be a very rare need. > > I never used the UNIX comm tool. I didn't know that it exists :-) > > Can you maybe explain the purpose the tool/your function? Show an example. It might help to realise that comm is abbreviated from "common". It took me an embarrassingly long time to work that out. This example of using comm in Unix might help: http://www.theunixschool.com/2011/03/comm-beautiful-comparison.html > difflib may be a better candidate to add such function than contextlib. I think you mean itertools rather than contextlib. It seems to me that difflib would be a good place for it. -- Steve From senthil at uthcode.com Wed Jan 7 16:53:39 2015 From: senthil at uthcode.com (Senthil Kumaran) Date: Wed, 7 Jan 2015 07:53:39 -0800 Subject: [Python-ideas] build:// Path / URI / URN with the __div__ operator In-Reply-To: <CACac1F9sRUndHk8WgEP4wCEMArPxg_w=nC4QLzg2xCSx2rLg8A@mail.gmail.com> References: <CACfEFw9Fes0iW9cz-juk8jk9pchC7wcQhDUKiVX+2tDcgUBS2Q@mail.gmail.com> <CACac1F9sRUndHk8WgEP4wCEMArPxg_w=nC4QLzg2xCSx2rLg8A@mail.gmail.com> Message-ID: <CAPOVWOQR3v9Pe1WFnRuEZODdTBq439r1gKVriFr2p1rFwq0n-Q@mail.gmail.com> On Wed, Jan 7, 2015 at 7:36 AM, Paul Moore <p.f.moore at gmail.com> wrote: > That was way too terse. What are you proposing? His proposal seems like a __div__ method an (no existing yet) URI Object. But __div__ method for an existing Path Object. I get the idea, but I am having a hard time understanding the use cases which may not be confusing for common cases. Question: Does __div__ makes sense for strings? Why does it? I dont think so, but this question I pose has marginal relationship to the proposal. thanks, Senthil -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/7c53bca5/attachment.html> From alexander.belopolsky at gmail.com Wed Jan 7 17:07:28 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 7 Jan 2015 11:07:28 -0500 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <54AD4F71.7060008@oa-cagliari.inaf.it> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <CAG3cHaaXAXs87ZK=wk1q191HR2tt0EVdZTFSgo=2xLO_feujbw@mail.gmail.com> <CAFpSVp+1ee8mM_xbPkANeNjWJGRr8zAd=SFe+VC5EO-2L4P2Dg@mail.gmail.com> <54AD4F71.7060008@oa-cagliari.inaf.it> Message-ID: <CAP7h-xZFd3DGvhr0D6D3rzJBm5BgZ3yX9jaOCQaFGPM36GD74g@mail.gmail.com> On Wed, Jan 7, 2015 at 10:23 AM, Marco Buttu <marco.buttu at gmail.com> wrote: > On 07/01/2015 15:58, Todd wrote: > > Yes, this would be a shortcut for that. math.nan and math.inf are less >> verbose than float("nan") and float("inf"), even if you aren't already >> using the math module (fewer characters total, and in particular fewer >> special characters). It would also, at least in my opinion, be more >> readable. Plus there is already the math.isnan and math.isinf, so I don't >> think it would be totally out-of-place. >> > > I totally agree. +1 Another +1. Two points: 1. The current ways to create inf and nan are not obvious and hard to discover. It will be nice to have these constants discoverable in tab completion from math. 2. The repr()s of nan and inf are simply 'nan' and 'inf', not 'float("nan")' and 'float("inf")', so it is natural to expect nan and inf constants in some namespace. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/959b2a0f/attachment.html> From guido at python.org Wed Jan 7 17:07:45 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 7 Jan 2015 08:07:45 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <54AD4F71.7060008@oa-cagliari.inaf.it> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <CAG3cHaaXAXs87ZK=wk1q191HR2tt0EVdZTFSgo=2xLO_feujbw@mail.gmail.com> <CAFpSVp+1ee8mM_xbPkANeNjWJGRr8zAd=SFe+VC5EO-2L4P2Dg@mail.gmail.com> <54AD4F71.7060008@oa-cagliari.inaf.it> Message-ID: <CAP7+vJJnwWeV5H-k-pwOfmnkazw6LtP4yUMaoOiB8=2Cfdt1ww@mail.gmail.com> +1. Let's do it. On Wed, Jan 7, 2015 at 7:23 AM, Marco Buttu <marco.buttu at gmail.com> wrote: > On 07/01/2015 15:58, Todd wrote: > > Yes, this would be a shortcut for that. math.nan and math.inf are less >> verbose than float("nan") and float("inf"), even if you aren't already >> using the math module (fewer characters total, and in particular fewer >> special characters). It would also, at least in my opinion, be more >> readable. Plus there is already the math.isnan and math.isinf, so I don't >> think it would be totally out-of-place. >> > > I totally agree. +1 > > -- > Marco Buttu > > INAF-Osservatorio Astronomico di Cagliari > Via della Scienza n. 5, 09047 Selargius (CA) > Phone: 070 711 80 217 > Email: mbuttu at oa-cagliari.inaf.it > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/396d4e12/attachment.html> From steve at pearwood.info Wed Jan 7 17:09:26 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 8 Jan 2015 03:09:26 +1100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> Message-ID: <20150107160925.GH19387@ando.pearwood.info> On Wed, Jan 07, 2015 at 02:33:54PM +0100, Todd wrote: > The current way to get infinite and nan floats are to do float("inf") and > float("nan"), respectively. For many applications, this is sufficient. > However, for scientific and mathematical computing, using inf and nan > values is very common, and the current approach is very verbose. So for > those applications, I think it would be useful if the math standard library > module provided "math.nan" and "math.inf", which would return the same > thing as float("nan") and float("inf"), respectively. Compare: inf = float('inf') func(inf) versus: from math import inf func(inf) Defining your module's own constant actually takes two characters fewer than importing it from the math module. In any case, we're talking about an utterly trivial micro-optimization. This is Python, not Unix shell scripting. We don't use names like "umount" to save one character over "unmount", we encouraging writing for clarity and correctness over brevity. The math module has constants pi and e because it would be truly an inconvenience to have to create them yourself: pi = 3.14159265389793 I can remember the first six digits of pi, 3.1415, anything more than that I have to look up, and of course there is the considerable risk of getting it wrong, as I did above. But the same does not apply to creating your own inf constant. There is no long series of digits to remember, just three letters, and it's hard to get it wrong. Whether you write import math process(math.inf) or inf = float('inf') process(inf) is much the same effort. At worst, we might say that it is a matter of personal taste which you prefer. But frankly, the difference is trivial. Of course, writing float('inf') every single time you want an infinity is wasteful and silly, but that's a matter of education. If you can't teach people to write inf = float('inf') you won't be able to teach them to use math.inf either. Up to now, I have only discussed inf, not nan, because there is only one inf value in floats. (Two if you count -inf.) But the IEEE 754 standard provides many different NAN values, and offers a possible interpretion of them: the extra "payload" on each NAN can be used to carry diagnostic information on which numeric operation failed. Back in the 1980s, I was a user of the Apple Standard Numerics Environment (SANE), which provided easy access to NAN payloads and a standard interpretation for them. It is a system which can work well and help in debugging. Python today doesn't give us any simple way to access all those different float NANs, or offer any interpretation of what they might mean, but it might some day. I am strongly opposed to anything which might pre-empt that or discourage future advances. In summary: - offering a math.inf is harmless but not especially useful; - offering a single math.nan is harmful and I strongly oppose it. -- Steve From p.f.moore at gmail.com Wed Jan 7 17:09:51 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 7 Jan 2015 16:09:51 +0000 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <20150107153847.GG19387@ando.pearwood.info> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <1BC2F2DA-DC73-4BE3-8550-37F5EA872C07@gmail.com> <CAMpsgwZGZ14KyOWOSAZYbocU78sQ87Rgo4hB6Ba+mb=xm7Stsg@mail.gmail.com> <20150107153847.GG19387@ando.pearwood.info> Message-ID: <CACac1F-ZPccxP0bhDPwBt81tpFSpyHK5OhQABdLNvUORnNftYg@mail.gmail.com> On 7 January 2015 at 15:38, Steven D'Aprano <steve at pearwood.info> wrote: > This example of using comm in Unix might help: > > http://www.theunixschool.com/2011/03/comm-beautiful-comparison.html The example on that page is the exact reason I wrote my own implementation (the Windows version of comm that I have locally got confused by non-ASCII content in some of the filenames I had). Paul From wes.turner at gmail.com Wed Jan 7 17:11:46 2015 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 7 Jan 2015 10:11:46 -0600 Subject: [Python-ideas] build:// Path / URI / URN with the __div__ operator In-Reply-To: <CAPOVWOQR3v9Pe1WFnRuEZODdTBq439r1gKVriFr2p1rFwq0n-Q@mail.gmail.com> References: <CACfEFw9Fes0iW9cz-juk8jk9pchC7wcQhDUKiVX+2tDcgUBS2Q@mail.gmail.com> <CACac1F9sRUndHk8WgEP4wCEMArPxg_w=nC4QLzg2xCSx2rLg8A@mail.gmail.com> <CAPOVWOQR3v9Pe1WFnRuEZODdTBq439r1gKVriFr2p1rFwq0n-Q@mail.gmail.com> Message-ID: <CACfEFw8=MBC-avyL2LaxZ9K9rA8DRYXUnu5c+_8QmN9YwQN3gw@mail.gmail.com> %doctest_mode >>> p = pathlib.Path('//') / 'two' / 'three' >>> p PosixPath('//two/three') >>> 'https' + p Traceback (most recent call last): File "<ipython-input-37-007abdbe7c66>", line 1, in <module> 'https' + p TypeError: cannot concatenate 'str' and 'PosixPath' objects >>> 'root(//)' + p Traceback (most recent call last): File "<ipython-input-38-e2654adaa7b1>", line 1, in <module> 'root(//)' + p TypeError: cannot concatenate 'str' and 'PosixPath' objects >>> p = pathlib.Path('https://example.org/') / 'two' / 'three' >>> p PosixPath('https:/example.org/two/three') >>> p + "?query=value" + "#fragment" Traceback (most recent call last): File "<ipython-input-41-df5e0b32be61>", line 1, in <module> p + "?query=value" + "#fragment" TypeError: unsupported operand type(s) for +: 'PosixPath' and 'str' So, something like this would/could be really helpful, for usability: from ns/uri/url import URI, http, https from _ import base # "//" _url = https // 'user at host.netloc:port' / path ** query % fragment _url = base // path ** query % fragment # or, just: _url = https / 'user at host.netloc:port' / path ** query % fragment _url = base / path ** query % fragment An OrderedMultiDict of query vars would also be helpful for building a union set of GET params joined by not os.path.sep: * ?key=val args and/or * #{'key':'val'} * OrderedDict.setdefault(key, list) On Wed, Jan 7, 2015 at 9:53 AM, Senthil Kumaran <senthil at uthcode.com> wrote: > > On Wed, Jan 7, 2015 at 7:36 AM, Paul Moore <p.f.moore at gmail.com> wrote: > >> That was way too terse. What are you proposing? > > > His proposal seems like a __div__ method an (no existing yet) URI Object. > But __div__ method for an existing Path Object. > > I get the idea, but I am having a hard time understanding the use cases > which may not be confusing for common cases. > > Question: Does __div__ makes sense for strings? Why does it? I dont think > so, but this question I pose has marginal relationship to the proposal. > > thanks, > Senthil > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/9a09f39a/attachment.html> From rosuav at gmail.com Wed Jan 7 17:14:17 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 8 Jan 2015 03:14:17 +1100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <20150107160925.GH19387@ando.pearwood.info> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> Message-ID: <CAPTjJmq6kOBZC3MWwRigYNOPYfSC9T58g_OZhyiELzFai59KYw@mail.gmail.com> On Thu, Jan 8, 2015 at 3:09 AM, Steven D'Aprano <steve at pearwood.info> wrote: > I can remember the first six digits of pi, 3.1415... Aside: That's technically an incorrect approximation. Since the next digit is a 9, you ought to round that up... or just use 3.14159, stopping before the 2 and thus correctly rounding down. ChrisA From guido at python.org Wed Jan 7 17:16:18 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 7 Jan 2015 08:16:18 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <20150107160925.GH19387@ando.pearwood.info> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> Message-ID: <CAP7+vJJ_XHREaimQ=rSmkr0K24Hrme0c3gX+JrgoODD_9MD2Vw@mail.gmail.com> On Wed, Jan 7, 2015 at 8:09 AM, Steven D'Aprano <steve at pearwood.info> wrote: > On Wed, Jan 07, 2015 at 02:33:54PM +0100, Todd wrote: > > > The current way to get infinite and nan floats are to do float("inf") and > > float("nan"), respectively. For many applications, this is sufficient. > > However, for scientific and mathematical computing, using inf and nan > > values is very common, and the current approach is very verbose. So for > > those applications, I think it would be useful if the math standard > library > > module provided "math.nan" and "math.inf", which would return the same > > thing as float("nan") and float("inf"), respectively. > > Compare: > > inf = float('inf') > func(inf) > > versus: > > from math import inf > func(inf) > But using a more common style, you already have math imported anyway, and you can just write math.inf rather than having to interrupt your coding flow, see if you already have an inf variable defined, if not define it, etc. > > Defining your module's own constant actually takes two characters fewer > than importing it from the math module. > But that's not the (whole) point. The existing idioms are commonly used and unintuitive. I suppose you would also be in favor of defining e = math.exp(1) in favor of math.e? > > In any case, we're talking about an utterly trivial micro-optimization. > This is Python, not Unix shell scripting. We don't use names like > "umount" to save one character over "unmount", we encouraging writing > for clarity and correctness over brevity. > > The math module has constants pi and e because it would be truly an > inconvenience to have to create them yourself: > > pi = 3.14159265389793 > > I can remember the first six digits of pi, 3.1415, anything more than > that I have to look up, and of course there is the considerable risk of > getting it wrong, as I did above. > > But the same does not apply to creating your own inf constant. There is > no long series of digits to remember, just three letters, and it's hard > to get it wrong. Whether you write > > import math > process(math.inf) > > or > > inf = float('inf') > process(inf) > > is much the same effort. At worst, we might say that it is a matter of > personal taste which you prefer. But frankly, the difference is trivial. > > I say bah to that long-winded diatribe. > Of course, writing float('inf') every single time you want an infinity > is wasteful and silly, but that's a matter of education. If you can't > teach people to write inf = float('inf') you won't be able to teach them > to use math.inf either. > > Up to now, I have only discussed inf, not nan, because there is only one > inf value in floats. (Two if you count -inf.) But the IEEE 754 standard > provides many different NAN values, and offers a possible interpretion > of them: the extra "payload" on each NAN can be used to carry diagnostic > information on which numeric operation failed. Back in the 1980s, I was > a user of the Apple Standard Numerics Environment (SANE), which provided > easy access to NAN payloads and a standard interpretation for them. It > is a system which can work well and help in debugging. > > Python today doesn't give us any simple way to access all those > different float NANs, or offer any interpretation of what they might > mean, but it might some day. I am strongly opposed to anything which > might pre-empt that or discourage future advances. > I don't see how the existence of a nan constant pre-empts anything. Your imagined new mechanism will have to co-exist with float("nan") anyway. > In summary: > > - offering a math.inf is harmless but not especially useful; > > - offering a single math.nan is harmful and I strongly oppose it. That's your opinion and you're entitled to it. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/29500d27/attachment.html> From yawar.amin at gmail.com Wed Jan 7 17:20:46 2015 From: yawar.amin at gmail.com (Yawar Amin) Date: Wed, 7 Jan 2015 11:20:46 -0500 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <74164843-0774-441F-A898-4776E15EFC8D@yahoo.com> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> <54AB4178.3050508@gmail.com> <97D50920-B7D2-4125-9389-224C4CBC83CA@yahoo.com> <54AC952E.5060301@gmail.com> <74164843-0774-441F-A898-4776E15EFC8D@yahoo.com> Message-ID: <8C6A6BAA-197B-4796-9163-6EE3AE3E1553@gmail.com> On 2015-01-07, at 2:11, Andrew Barnert <abarnert at yahoo.com> wrote: > [...] >> > > That's the key. A function is a way to wrap a sequence of statements (the function body) so it can be used in an expression (a function call). Because JavaScript lets you define functions (with full statement syntax) in expressions, that gives you a way to "demote" an expression inline, while Python can only do so out-of-line. At the core, this is what all of the multiline lambda attempts and similar proposals are trying to accomplish. Thanks. This has given me an idea that's on a slightly different tangent than this one. Will write it up (this time with some actual runnable code) when I get home tonight. Regards, Yawar From alexander.belopolsky at gmail.com Wed Jan 7 17:21:15 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 7 Jan 2015 11:21:15 -0500 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAP7+vJJ_XHREaimQ=rSmkr0K24Hrme0c3gX+JrgoODD_9MD2Vw@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CAP7+vJJ_XHREaimQ=rSmkr0K24Hrme0c3gX+JrgoODD_9MD2Vw@mail.gmail.com> Message-ID: <CAP7h-xbzQ_bhcu5DYnbcwgSnfnbumeV3RzbiSb-tcLhhZixGJA@mail.gmail.com> On Wed, Jan 7, 2015 at 11:16 AM, Guido van Rossum <guido at python.org> wrote: > But that's not the (whole) point. The existing idioms are commonly used > and unintuitive. I suppose you would also be in favor of defining e = > math.exp(1) in favor of math.e? .. math.acos(-1) in favor of math.pi? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/c58d4a1f/attachment.html> From wes.turner at gmail.com Wed Jan 7 17:23:06 2015 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 7 Jan 2015 10:23:06 -0600 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <CANc-5UyZRAZbZhhX6qXyfxfJTUa9EsbSOUvqGqxkObPjCne6dQ@mail.gmail.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> <CANc-5UzU+o8K1QXLGnybVnj8LvW4HgMA8ZVkM+D+KAyxYXt+pQ@mail.gmail.com> <CAP7+vJLg6Guyusw6DhSYrAP1VYXsWnM-1RaYoURbr14k+2Gq-A@mail.gmail.com> <54AC5B2F.6090504@egenix.com> <CACfEFw8uQwBq7ThcaeDtvUMmBRZnVZE1ZKTLefU0Da+6JXxR1Q@mail.gmail.com> <54AC69E2.6070303@egenix.com> <CACfEFw9sQAd1NrfM7YOLwyjmgPPO9e_5iLtdQkTsDN17Ff13Lg@mail.gmail.com> <54ACF333.80307@egenix.com> <CACfEFw-hB_YSrSEp3w7PKxaJisGZW--+HikZwRvjpPeppdY4UQ@mail.gmail.com> <CANc-5UyZRAZbZhhX6qXyfxfJTUa9EsbSOUvqGqxkObPjCne6dQ@mail.gmail.com> Message-ID: <CACfEFw_GSsJsTkJbREQqdWxB=Rqzku-Arct+qw0Gw7U-qcXz6g@mail.gmail.com> On Wed, Jan 7, 2015 at 7:27 AM, Skip Montanaro <skip.montanaro at gmail.com> wrote: > I'm going to add one more bit here about wiki gardening. > In order to test parsing code in order to produce Linked Data http://5stardata.info , it could be helpful to produce a periodic archive/dump sort of like http://wiki.dbpedia.org/Datasets . A github repository could be useful for hosting these types of releases. python/wiki___ ? > > BITD, the main website had many pages which today would be called > listicles. The problem was that those lists were dynamic. They weren't > just the "top ten scientific Python modules of all time" or the "seven > best packaging tools." They were whatever was at the front of the > author's brain at some point in the past. Having these static pages > given the imprimatur of the PSF when in fact none of the site > maintainers were obviously responsible for (or interested in) keeping > them up-to-date did a disservice to the community and to authors of > other packages which weren't represented in those lists. In addition, > there were more barriers to update than necessary (essentially, figure > out how to report the problem, offer suggested fix(es), then have them > swallowed up into the site update mechanism). > Are the links out of date, or the descriptions and exclusive clusters? * http://schema.org/SoftwareApplication (RDFa in HTML5, Microdata) * http://json-ld.org/ * URIs are tags > > One of the main uses envisioned for the wiki was as a place where > these listicles could be maintained by the greater Python > community. For some things, that's worked out pretty well. The most > obvious thing that comes to mind is the PythonTraining page. That > works because the people whose skills are represented on that page > have a very good reason for keeping things up-to-date: it's free > advertising for their businesses. > > Other listicle type pages haven't been keep as up-to-date. For > example, the PythonEditors page was last updated in Feb 2014, is huge > (and might benefit from being split into multiple pages), and probably > no longer accurately represents the available editors or IDEs which > support Python. > The takeaway in my mind is that we could probably use "gardeners" to > take over active maintenance of these listicle pages. That, coupled > with a couple "master gardeners" to develop some suitable structure, > and some landscape crews to prune dead/outdated/no-longer-useful > pages, would likely go a long way to improving the quality of the > wiki. > > Skip > Are there tools for working with a regular editor (such as vim) and MoinMoin pages? With ReStructuredText, the Riv.vim syntax helpers are great for things like tables; Voom is great for outlines (:Voom rest). -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/f5312bea/attachment.html> From chris.barker at noaa.gov Wed Jan 7 17:25:22 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 7 Jan 2015 08:25:22 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <20150107160925.GH19387@ando.pearwood.info> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> Message-ID: <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> On Wed, Jan 7, 2015 at 8:09 AM, Steven D'Aprano <steve at pearwood.info> wrote: > import math > process(math.inf) > > or > > inf = float('inf') > process(inf) > > is much the same effort. At worst, we might say that it is a matter of > personal taste which you prefer. But frankly, the difference is trivial. > exactly -- but the cost is trivial too. Note that numpy provides np.inf and np.nan for a reason, and it s a nice thing to have. > - offering a math.inf is harmless but not especially useful; > but nice. - offering a single math.nan is harmful and I strongly oppose it. defining math.nan as a singleton could be considered harmful, but I don't see any harm at all in offering it as a pre-defined constant. We NEED to have a way to create a NaN-valued float in any case (and do with float('nan')) -- this would be no different. And as nan (and inf, and -inf) can result from computation, rather than python-level object creation, they are different than None anyway (they are also particular values or a float, rather than a particular type). +1 -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/d5814905/attachment.html> From steve at pearwood.info Wed Jan 7 17:48:35 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 8 Jan 2015 03:48:35 +1100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAPTjJmq6kOBZC3MWwRigYNOPYfSC9T58g_OZhyiELzFai59KYw@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CAPTjJmq6kOBZC3MWwRigYNOPYfSC9T58g_OZhyiELzFai59KYw@mail.gmail.com> Message-ID: <20150107164834.GI19387@ando.pearwood.info> On Thu, Jan 08, 2015 at 03:14:17AM +1100, Chris Angelico wrote: > On Thu, Jan 8, 2015 at 3:09 AM, Steven D'Aprano <steve at pearwood.info> wrote: > > I can remember the first six digits of pi, 3.1415... > > Aside: That's technically an incorrect approximation. I know. I didn't say it was a good approximation, I said I could remember those digits. -- Steve From skip.montanaro at gmail.com Wed Jan 7 17:55:55 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Wed, 7 Jan 2015 10:55:55 -0600 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <CACfEFw_GSsJsTkJbREQqdWxB=Rqzku-Arct+qw0Gw7U-qcXz6g@mail.gmail.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <1752C815-970E-4588-8DEE-523833EC4E8A@yahoo.com> <CADiSq7cT83nqaCyX_Mjj07u3AW0wfvGhq5VbG_BYi61f1R915w@mail.gmail.com> <822B30F1-3A5C-4CF9-BE10-0A818A5241DB@yahoo.com> <CADiSq7e1mjnRo2AWfsz4Bsfm1hU6EDOBqyXUiHVFNj8gkFOo=w@mail.gmail.com> <20150105011312.GS27426@ando.pearwood.info> <E862C12F-E2AF-4BEB-9661-CB8823DAE13E@stufft.io> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> <CANc-5UzU+o8K1QXLGnybVnj8LvW4HgMA8ZVkM+D+KAyxYXt+pQ@mail.gmail.com> <CAP7+vJLg6Guyusw6DhSYrAP1VYXsWnM-1RaYoURbr14k+2Gq-A@mail.gmail.com> <54AC5B2F.6090504@egenix.com> <CACfEFw8uQwBq7ThcaeDtvUMmBRZnVZE1ZKTLefU0Da+6JXxR1Q@mail.gmail.com> <54AC69E2.6070303@egenix.com> <CACfEFw9sQAd1NrfM7YOLwyjmgPPO9e_5iLtdQkTsDN17Ff13Lg@mail.gmail.com> <54ACF333.80307@egenix.com> <CACfEFw-hB_YSrSEp3w7PKxaJisGZW--+HikZwRvjpPeppdY4UQ@mail.gmail.com> <CANc-5UyZRAZbZhhX6qXyfxfJTUa9EsbSOUvqGqxkObPjCne6dQ@mail.gmail.com> <CACfEFw_GSsJsTkJbREQqdWxB=Rqzku-Arct+qw0Gw7U-qcXz6g@mail.gmail.com> Message-ID: <CANc-5UwbK7tKJ2+osb4x1P2hU05wJoXTwBw05S7veweh68UXDg@mail.gmail.com> On Wed, Jan 7, 2015 at 10:23 AM, Wes Turner <mailto:wes.turner at gmail.com> wrote: > Are there tools for working with a regular editor (such as vim) and > MoinMoin pages? I'm not at all concerned with this, as I see the editing step being pretty much orthogonal to the markup language and the version control tools. Editing through a web browser pretty much sucks no matter what you're trying to edit, especially if you are used to using highly functional, mature editors like vim and Emacs. If you want your editing to happen outside a glorified <textarea> widget, find an "offline" editing plugin. (I happen to use Edit with Emacs from Chrome, but there are other extensions available.) I assume that MoinMoin will eventually be supplanted by some other (wiki?) technology. I do think it's worth noting though, that while Github and other version control services are all the rage for all sorts of collaborative editing, wikis in general predate that sort of technology by a long way. If I read correctly, the original C2 Wiki was launched in 1995, MoinMoin in 2000, Git in 2005, and Github not until 2007. See: * http://c2.com/cgi/wiki?WikiHistory * http://en.wikipedia.org/wiki/MoinMoin * http://en.wikipedia.org/wiki/Git_(software) * http://en.wikipedia.org/wiki/GitHub It's only more recently developed wiki systems (or perhaps a few wiki systems whose authors were prescient and saw DVCSs coming) which support generic version control backends. I think I posted a link yesterday which looked like a reasonable starting point for converting MoinMoin markup to Markdown. Run with that if you want. If you can parse MoinMoin markup and generate Markdown, it shouldn't be too difficult to iterate over all the pages and their iterations in wiki.python.org, convert to Markdown, then commit to a Github repo. I think MoinMoin has served the Python community pretty well. Despite its apparent unpopularity, I think structural and content issues are more important than the markup syntax used for that content or the version control scheme implemented in the backend. Skip -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/0b94a9fa/attachment-0001.html> From ethan at stoneleaf.us Wed Jan 7 17:55:43 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 07 Jan 2015 08:55:43 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAP7+vJJnwWeV5H-k-pwOfmnkazw6LtP4yUMaoOiB8=2Cfdt1ww@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <CAG3cHaaXAXs87ZK=wk1q191HR2tt0EVdZTFSgo=2xLO_feujbw@mail.gmail.com> <CAFpSVp+1ee8mM_xbPkANeNjWJGRr8zAd=SFe+VC5EO-2L4P2Dg@mail.gmail.com> <54AD4F71.7060008@oa-cagliari.inaf.it> <CAP7+vJJnwWeV5H-k-pwOfmnkazw6LtP4yUMaoOiB8=2Cfdt1ww@mail.gmail.com> Message-ID: <54AD650F.905@stoneleaf.us> On 01/07/2015 08:07 AM, Guido van Rossum wrote: > On Wed, Jan 7, 2015 at 7:23 AM, Marco Buttu wrote: >> On 07/01/2015 15:58, Todd wrote: >>> >>> Yes, this would be a shortcut for that. math.nan and math.inf are less verbose than float("nan") and >>> float("inf"), even if you aren't already using the math module (fewer characters total, and in particular fewer >>> special characters). It would also, at least in my opinion, be more readable. Plus there is already the >>> math.isnan and math.isinf, so I don't think it would be totally out-of-place. >> >> I totally agree. +1 > > +1. Let's do it. Anything special for float('-inf') ? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/ae264c8d/attachment.sig> From jeanpierreda at gmail.com Wed Jan 7 17:57:50 2015 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Wed, 7 Jan 2015 10:57:50 -0600 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> Message-ID: <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> On Wed, Jan 7, 2015 at 10:25 AM, Chris Barker <chris.barker at noaa.gov> wrote: > On Wed, Jan 7, 2015 at 8:09 AM, Steven D'Aprano <steve at pearwood.info> wrote: >> - offering a single math.nan is harmful and I strongly oppose it. > > defining math.nan as a singleton could be considered harmful, but I don't > see any harm at all in offering it as a pre-defined constant. We NEED to > have a way to create a NaN-valued float in any case (and do with > float('nan')) -- this would be no different. Maybe this would justify a note in the documentation, that this is one NaN, but there are many others that are different. As long as people know not to rely on math.nan being the same object or having the same bits as every nan, then there's no harm. -- Devin From jeanpierreda at gmail.com Wed Jan 7 17:58:27 2015 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Wed, 7 Jan 2015 10:58:27 -0600 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <54AD650F.905@stoneleaf.us> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <CAG3cHaaXAXs87ZK=wk1q191HR2tt0EVdZTFSgo=2xLO_feujbw@mail.gmail.com> <CAFpSVp+1ee8mM_xbPkANeNjWJGRr8zAd=SFe+VC5EO-2L4P2Dg@mail.gmail.com> <54AD4F71.7060008@oa-cagliari.inaf.it> <CAP7+vJJnwWeV5H-k-pwOfmnkazw6LtP4yUMaoOiB8=2Cfdt1ww@mail.gmail.com> <54AD650F.905@stoneleaf.us> Message-ID: <CABicbJJek+KuBS9WiHwy9XhD6DTkPsMF+WsS4ZqCdv3CZ=uaxQ@mail.gmail.com> On Wed, Jan 7, 2015 at 10:55 AM, Ethan Furman <ethan at stoneleaf.us> wrote: > On 01/07/2015 08:07 AM, Guido van Rossum wrote: >> On Wed, Jan 7, 2015 at 7:23 AM, Marco Buttu wrote: >>> On 07/01/2015 15:58, Todd wrote: >>>> >>>> Yes, this would be a shortcut for that. math.nan and math.inf are less verbose than float("nan") and >>>> float("inf"), even if you aren't already using the math module (fewer characters total, and in particular fewer >>>> special characters). It would also, at least in my opinion, be more readable. Plus there is already the >>>> math.isnan and math.isinf, so I don't think it would be totally out-of-place. >>> >>> I totally agree. +1 >> >> +1. Let's do it. > > Anything special for float('-inf') ? There is no constant name as short and obvious as -math.inf. -- Devin From mertz at gnosis.cx Wed Jan 7 19:20:22 2015 From: mertz at gnosis.cx (David Mertz) Date: Wed, 7 Jan 2015 10:20:22 -0800 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <CACac1F-ZPccxP0bhDPwBt81tpFSpyHK5OhQABdLNvUORnNftYg@mail.gmail.com> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <1BC2F2DA-DC73-4BE3-8550-37F5EA872C07@gmail.com> <CAMpsgwZGZ14KyOWOSAZYbocU78sQ87Rgo4hB6Ba+mb=xm7Stsg@mail.gmail.com> <20150107153847.GG19387@ando.pearwood.info> <CACac1F-ZPccxP0bhDPwBt81tpFSpyHK5OhQABdLNvUORnNftYg@mail.gmail.com> Message-ID: <CAEbHw4bjdKL_vhsTK=Et4i-8VoUkiiEAgjmgczPn1Wnyq3XHcg@mail.gmail.com> Adding a itertools.comm() seems unnecessary, although adding it to the widely used more-itertools feels more reasonable. However, the problem with the functionality is that it assumes that the iterables given to it return pre-sorted values. Which is fine it that assumption is satisfied, but one really can't know in advance of exhausting the iterators whether that will prove true. The desired behavior in the absence of this pre-condition being satisfied is unclear to me. I could plausibly see the tool raising some exception when the pre-condition was violated. Of course, in that case, we might well have already returned multiple elements on next(comm(it1, it2)) calls before the exception is encountered, and the initial values might be deceptive or retroactively invalidated. I could plausibly see it ignoring the non-sorted elements in some way. Maybe just pretend they are sorted rather than checking, returning somewhat ill-defined values, I'd think. Or maybe just discarding the missorted values? Or flagging them with some other indicator indicating the missorting? I dunno. In any case, this is not something that can be applied to iterators generically, but only to iterators about which one makes lots of assumptions in advance, and hence seems ill-suited for the stdlib itertools. On Wed, Jan 7, 2015 at 8:09 AM, Paul Moore <p.f.moore at gmail.com> wrote: > On 7 January 2015 at 15:38, Steven D'Aprano <steve at pearwood.info> wrote: > > This example of using comm in Unix might help: > > > > http://www.theunixschool.com/2011/03/comm-beautiful-comparison.html > > The example on that page is the exact reason I wrote my own > implementation (the Windows version of comm that I have locally got > confused by non-ASCII content in some of the filenames I had). > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/03368b87/attachment.html> From python at mrabarnett.plus.com Wed Jan 7 19:23:37 2015 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 07 Jan 2015 18:23:37 +0000 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> Message-ID: <54AD79A9.1020000@mrabarnett.plus.com> On 2015-01-07 16:57, Devin Jeanpierre wrote: > On Wed, Jan 7, 2015 at 10:25 AM, Chris Barker <chris.barker at noaa.gov> wrote: >> On Wed, Jan 7, 2015 at 8:09 AM, Steven D'Aprano <steve at pearwood.info> wrote: >>> - offering a single math.nan is harmful and I strongly oppose it. >> >> defining math.nan as a singleton could be considered harmful, but I don't >> see any harm at all in offering it as a pre-defined constant. We NEED to >> have a way to create a NaN-valued float in any case (and do with >> float('nan')) -- this would be no different. > > Maybe this would justify a note in the documentation, that this is one > NaN, but there are many others that are different. > > As long as people know not to rely on math.nan being the same object > or having the same bits as every nan, then there's no harm. > Does that mean it should be a function that returns a NaN, i.e. math.nan()? From eric at trueblade.com Wed Jan 7 19:24:19 2015 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 07 Jan 2015 13:24:19 -0500 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <CAEbHw4bjdKL_vhsTK=Et4i-8VoUkiiEAgjmgczPn1Wnyq3XHcg@mail.gmail.com> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <1BC2F2DA-DC73-4BE3-8550-37F5EA872C07@gmail.com> <CAMpsgwZGZ14KyOWOSAZYbocU78sQ87Rgo4hB6Ba+mb=xm7Stsg@mail.gmail.com> <20150107153847.GG19387@ando.pearwood.info> <CACac1F-ZPccxP0bhDPwBt81tpFSpyHK5OhQABdLNvUORnNftYg@mail.gmail.com> <CAEbHw4bjdKL_vhsTK=Et4i-8VoUkiiEAgjmgczPn1Wnyq3XHcg@mail.gmail.com> Message-ID: <54AD79D3.6090104@trueblade.com> On 01/07/2015 01:20 PM, David Mertz wrote: > Adding a itertools.comm() seems unnecessary, although adding it to the > widely used more-itertools feels more reasonable. > > However, the problem with the functionality is that it assumes that the > iterables given to it return pre-sorted values. Which is fine it that > assumption is satisfied, but one really can't know in advance of > exhausting the iterators whether that will prove true. The desired > behavior in the absence of this pre-condition being satisfied is unclear > to me. itertools.groupby has the same requirement for a pre-sorted input iterator. But I do agree that itertools isn't the best place in the stdlib for common(). difflib seems reasonable to me, if it's added at all. Eric. From mertz at gnosis.cx Wed Jan 7 19:26:32 2015 From: mertz at gnosis.cx (David Mertz) Date: Wed, 7 Jan 2015 10:26:32 -0800 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <CAEbHw4bjdKL_vhsTK=Et4i-8VoUkiiEAgjmgczPn1Wnyq3XHcg@mail.gmail.com> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <1BC2F2DA-DC73-4BE3-8550-37F5EA872C07@gmail.com> <CAMpsgwZGZ14KyOWOSAZYbocU78sQ87Rgo4hB6Ba+mb=xm7Stsg@mail.gmail.com> <20150107153847.GG19387@ando.pearwood.info> <CACac1F-ZPccxP0bhDPwBt81tpFSpyHK5OhQABdLNvUORnNftYg@mail.gmail.com> <CAEbHw4bjdKL_vhsTK=Et4i-8VoUkiiEAgjmgczPn1Wnyq3XHcg@mail.gmail.com> Message-ID: <CAEbHw4YY0z1YpkMH4s8z1A+Og=vRU=13UMrJL2heMY7z6cBUDA@mail.gmail.com> I don't know if this clarifies anything, but I tried out such a mis-sorted example using the actual 'comm' tool 511-tmp % comm comm1 comm2 A B C E F J A K M N T U V 512-tmp % cat comm[12] A B C E F J A K T U V A B M N T U V On Wed, Jan 7, 2015 at 10:20 AM, David Mertz <mertz at gnosis.cx> wrote: > Adding a itertools.comm() seems unnecessary, although adding it to the > widely used more-itertools feels more reasonable. > > However, the problem with the functionality is that it assumes that the > iterables given to it return pre-sorted values. Which is fine it that > assumption is satisfied, but one really can't know in advance of exhausting > the iterators whether that will prove true. The desired behavior in the > absence of this pre-condition being satisfied is unclear to me. > > I could plausibly see the tool raising some exception when the > pre-condition was violated. Of course, in that case, we might well have > already returned multiple elements on next(comm(it1, it2)) calls before the > exception is encountered, and the initial values might be deceptive or > retroactively invalidated. > > I could plausibly see it ignoring the non-sorted elements in some way. > Maybe just pretend they are sorted rather than checking, returning somewhat > ill-defined values, I'd think. Or maybe just discarding the missorted > values? Or flagging them with some other indicator indicating the > missorting? I dunno. > > In any case, this is not something that can be applied to iterators > generically, but only to iterators about which one makes lots of > assumptions in advance, and hence seems ill-suited for the stdlib itertools. > > > On Wed, Jan 7, 2015 at 8:09 AM, Paul Moore <p.f.moore at gmail.com> wrote: > >> On 7 January 2015 at 15:38, Steven D'Aprano <steve at pearwood.info> wrote: >> > This example of using comm in Unix might help: >> > >> > http://www.theunixschool.com/2011/03/comm-beautiful-comparison.html >> >> The example on that page is the exact reason I wrote my own >> implementation (the Windows version of comm that I have locally got >> confused by non-ASCII content in some of the filenames I had). >> >> Paul >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/93c05075/attachment.html> From ethan at stoneleaf.us Wed Jan 7 19:27:58 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 07 Jan 2015 10:27:58 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <54AD79A9.1020000@mrabarnett.plus.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <54AD79A9.1020000@mrabarnett.plus.com> Message-ID: <54AD7AAE.4020107@stoneleaf.us> On 01/07/2015 10:23 AM, MRAB wrote: > On 2015-01-07 16:57, Devin Jeanpierre wrote: >> >> Maybe this would justify a note in the documentation, that this is one >> NaN, but there are many others that are different. >> >> As long as people know not to rely on math.nan being the same object >> or having the same bits as every nan, then there's no harm. > > Does that mean it should be a function that returns a NaN, i.e. math.nan()? No, it should be similar to math.e and math.pi -- that is, a constant value (which NaN is bound to math.nan is subject to change without notice, but it will be a NaN). -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/5a069e13/attachment.sig> From solipsis at pitrou.net Wed Jan 7 19:31:00 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 7 Jan 2015 19:31:00 +0100 Subject: [Python-ideas] math.inf and math.nan constants References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <54AD79A9.1020000@mrabarnett.plus.com> Message-ID: <20150107193100.784292b8@fsol> On Wed, 07 Jan 2015 18:23:37 +0000 MRAB <python at mrabarnett.plus.com> wrote: > On 2015-01-07 16:57, Devin Jeanpierre wrote: > > On Wed, Jan 7, 2015 at 10:25 AM, Chris Barker <chris.barker at noaa.gov> wrote: > >> On Wed, Jan 7, 2015 at 8:09 AM, Steven D'Aprano <steve at pearwood.info> wrote: > >>> - offering a single math.nan is harmful and I strongly oppose it. > >> > >> defining math.nan as a singleton could be considered harmful, but I don't > >> see any harm at all in offering it as a pre-defined constant. We NEED to > >> have a way to create a NaN-valued float in any case (and do with > >> float('nan')) -- this would be no different. > > > > Maybe this would justify a note in the documentation, that this is one > > NaN, but there are many others that are different. > > > > As long as people know not to rely on math.nan being the same object > > or having the same bits as every nan, then there's no harm. > > > Does that mean it should be a function that returns a NaN, i.e. math.nan()? If it returns always the same NaN it won't make a difference. However we could have a random.nan() function to get random NaNs (perhaps with different distributions). Regards Antoine. From steve at pearwood.info Wed Jan 7 19:35:33 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 8 Jan 2015 05:35:33 +1100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAP7+vJJ_XHREaimQ=rSmkr0K24Hrme0c3gX+JrgoODD_9MD2Vw@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CAP7+vJJ_XHREaimQ=rSmkr0K24Hrme0c3gX+JrgoODD_9MD2Vw@mail.gmail.com> Message-ID: <20150107183520.GK19387@ando.pearwood.info> On Wed, Jan 07, 2015 at 08:16:18AM -0800, Guido van Rossum wrote: > I suppose you would also be in favor of defining e = math.exp(1) in > favor of math.e? Euler's Number e is an important mathematical constant, but I'm not sure that it is a useful *programming* constant, more useful than (say) ?, the Euler-Mascheroni constant which is also ubiquitous in mathematics. I can't think of any reason to use e on its own, apart from exponentials and logs, where we have functions which are more accurate than calling e**x directly. If e wasn't already in the math module, I don't think there are enough use-cases to justify adding it. Plenty of other languages don't (e.g. Haskell and Lua). I think math.tau would be much more useful and important than e :-) http://bugs.python.org/issue12345 > I say bah to that long-winded diatribe. Long-winded, perhaps, but a diatribe? (Definition on WordNet: "thunderous verbal attack".) -- Steve From python at 2sn.net Wed Jan 7 19:43:22 2015 From: python at 2sn.net (Alexander Heger) Date: Thu, 8 Jan 2015 05:43:22 +1100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAP7h-xbzQ_bhcu5DYnbcwgSnfnbumeV3RzbiSb-tcLhhZixGJA@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CAP7+vJJ_XHREaimQ=rSmkr0K24Hrme0c3gX+JrgoODD_9MD2Vw@mail.gmail.com> <CAP7h-xbzQ_bhcu5DYnbcwgSnfnbumeV3RzbiSb-tcLhhZixGJA@mail.gmail.com> Message-ID: <CAN3CYHxD66ixmTYqUJ4DxW9nu-f5OYYnbL6f7BYxpvz00+dGxQ@mail.gmail.com> On 8 January 2015 at 03:21, Alexander Belopolsky <alexander.belopolsky at gmail.com> wrote: > .. math.acos(-1) in favor of math.pi? fastest is pi = 4 * math.atan(1) which I use in old FORTRAN codes From steve at pearwood.info Wed Jan 7 19:46:56 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 8 Jan 2015 05:46:56 +1100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <20150107193100.784292b8@fsol> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <54AD79A9.1020000@mrabarnett.plus.com> <20150107193100.784292b8@fsol> Message-ID: <20150107184656.GL19387@ando.pearwood.info> On Wed, Jan 07, 2015 at 07:31:00PM +0100, Antoine Pitrou wrote: > However we could have a random.nan() function to get random NaNs > (perhaps with different distributions). Do you have a use case for random NANs? I'd rather have the float constructor allow you to specify the payload, as decimal already does: py> decimal.Decimal('nan123') Decimal('NaN123') or possible a class method: float.nan(123) -- Steve From python at 2sn.net Wed Jan 7 19:58:03 2015 From: python at 2sn.net (Alexander Heger) Date: Thu, 8 Jan 2015 05:58:03 +1100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> Message-ID: <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> > Maybe this would justify a note in the documentation, that this is one > NaN, but there are many others that are different. > > As long as people know not to rely on math.nan being the same object > or having the same bits as every nan, then there's no harm. comparison with NaNs should be tough and can only be done with the maths.isnan functions. IEEE754 requires that all comparisons with NaN give false, i.e., nan == 0: False nan != 0: False (I use in FORTRAN codes to find NaNs in old code) Same for comparing nan to itself. >>> x = float('nan') >>> x is x True >>> x == x False Unless math.nan was a singleton, the "is" operator would also fail, and even that would not compare to a nan generated in computation in general. But I agree, having *a* nan constant in math would be good, as long as users know what they can do with it and what not. It should not be a function. -Alexander From bruce at leban.us Wed Jan 7 20:02:50 2015 From: bruce at leban.us (Bruce Leban) Date: Wed, 7 Jan 2015 11:02:50 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <20150107160925.GH19387@ando.pearwood.info> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> Message-ID: <CAGu0Ansg=r5eTBWOsRztYyBVPYzBJvT9gUU6x9a45-jYPAbSoA@mail.gmail.com> On Wed, Jan 7, 2015 at 8:09 AM, Steven D'Aprano <steve at pearwood.info> wrote: > In any case, we're talking about an utterly trivial micro-optimization. > This is Python, not Unix shell scripting. We don't use names like > "umount" to save one character over "unmount", we encouraging writing > for clarity and correctness over brevity. > > The math module has constants pi and e because it would be truly an > inconvenience to have to create them yourself: > > pi = 3.14159265389793 > > You're missing one key detail. If I see math.nan or math.inf in code, I know what those values are. If I see a bare nan or inf in the code I don't. That's an inconvenience too. > Up to now, I have only discussed inf, not nan, because there is only one > inf value in floats. (Two if you count -inf.) But the IEEE 754 standard > provides many different NAN values, and offers a possible interpretion > of them: the extra "payload" on each NAN can be used to carry diagnostic > information on which numeric operation failed. > This is a red herring as any future change to support this won't automatically touch code that uses float("nan") just as it won't touch code that uses float.nan. There is a problem here but that's not it. The problem is that float("nan") is float("nan") => False math.nan is math.nan => True So is this an attractive nuisance that will make people think they can should use x is math.nan instead of math.isnan(x)? Hopefully not but something to consider. Pylint should flag this usage. --- Bruce Check out my new puzzle book: http://Ju.mp/ingToConclusions Get it free here: http://Ju.mp/ingToConclusionsFree (available on iOS) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/3bc94f1d/attachment.html> From ethan at stoneleaf.us Wed Jan 7 20:49:37 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 07 Jan 2015 11:49:37 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAGu0Ansg=r5eTBWOsRztYyBVPYzBJvT9gUU6x9a45-jYPAbSoA@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CAGu0Ansg=r5eTBWOsRztYyBVPYzBJvT9gUU6x9a45-jYPAbSoA@mail.gmail.com> Message-ID: <54AD8DD1.8030402@stoneleaf.us> On 01/07/2015 11:02 AM, Bruce Leban wrote: > > There is a problem here but that's not it. The problem is that > > float("nan") is float("nan") => False > math.nan is math.nan => True This is not a new problem: --> nan = float('nan') --> nan is nan True > So is this an attractive nuisance that will make people think they can should use x is math.nan instead of > math.isnan(x)? Hopefully not but something to consider. Pylint should flag this usage. Agreed, for all linters. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/90ebbfd4/attachment.sig> From mal at egenix.com Wed Jan 7 21:45:00 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 07 Jan 2015 21:45:00 +0100 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <CANc-5UwbK7tKJ2+osb4x1P2hU05wJoXTwBw05S7veweh68UXDg@mail.gmail.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> <CANc-5UzU+o8K1QXLGnybVnj8LvW4HgMA8ZVkM+D+KAyxYXt+pQ@mail.gmail.com> <CAP7+vJLg6Guyusw6DhSYrAP1VYXsWnM-1RaYoURbr14k+2Gq-A@mail.gmail.com> <54AC5B2F.6090504@egenix.com> <CACfEFw8uQwBq7ThcaeDtvUMmBRZnVZE1ZKTLefU0Da+6JXxR1Q@mail.gmail.com> <54AC69E2.6070303@egenix.com> <CACfEFw9sQAd1NrfM7YOLwyjmgPPO9e_5iLtdQkTsDN17Ff13Lg@mail.gmail.com> <54ACF333.80307@egenix.com> <CACfEFw-hB_YSrSEp3w7PKxaJisGZW--+HikZwRvjpPeppdY4UQ@mail.gmail.com> <CANc-5UyZRAZbZhhX6qXyfxfJTUa9EsbSOUvqGqxkObPjCne6dQ@mail.gmail.com> <CACfEFw_GSsJsTkJbREQqdWxB=Rqzku-Arct+qw0Gw7U-qcXz6g@mail.gmail.com> <CANc-5UwbK7tKJ2+osb4x1P2hU05wJoXTwBw05S7veweh68UXDg@mail.gmail.com> Message-ID: <54AD9ACC.9050409@egenix.com> On 07.01.2015 17:55, Skip Montanaro wrote: > On Wed, Jan 7, 2015 at 10:23 AM, Wes Turner <mailto:wes.turner at gmail.com> wrote: > >> Are there tools for working with a regular editor (such as vim) and >> MoinMoin pages? > > I'm not at all concerned with this, as I see the editing step being > pretty much orthogonal to the markup language and the version control > tools. Editing through a web browser pretty much sucks no matter what > you're trying to edit, especially if you are used to using highly > functional, mature editors like vim and Emacs. If you want your editing > to happen outside a glorified <textarea> widget, find an "offline" > editing plugin. (I happen to use Edit with Emacs from Chrome, but > there are other extensions available.) > > I assume that MoinMoin will eventually be supplanted by some other > (wiki?) technology. I do think it's worth noting though, that while > Github and other version control services are all the rage for all > sorts of collaborative editing, wikis in general predate that sort of > technology by a long way. If I read correctly, the original C2 Wiki > was launched in 1995, MoinMoin in 2000, Git in 2005, and Github > not until 2007. See: > > * http://c2.com/cgi/wiki?WikiHistory > * http://en.wikipedia.org/wiki/MoinMoin > * http://en.wikipedia.org/wiki/Git_(software) > * http://en.wikipedia.org/wiki/GitHub > > It's only more recently developed wiki systems (or perhaps a few wiki > systems whose authors were prescient and saw DVCSs coming) which > support generic version control backends. > > I think I posted a link yesterday which looked like a reasonable > starting point for converting MoinMoin markup to Markdown. Run with > that if you want. If you can parse MoinMoin markup and generate > Markdown, it shouldn't be too difficult to iterate over all the pages > and their iterations in wiki.python.org, convert to Markdown, then > commit to a Github repo. ... and then you've changed the technology, lost the dynamic features of wikis, the permission controls, full text search, but have not solved the real issues, only created more work that's not relevant to the content. > I think MoinMoin has served the Python community pretty well. Despite > its apparent unpopularity, I think structural and content issues are > more important than the markup syntax used for that content or the > version control scheme implemented in the backend. In the end, it all boils down to people curating content. The tools, the markup and version control are all secondary. My recommendation: look through the wiki pages, grab some that are older and some care, and improve them. That's going help much more than this tools discussion. Here's starting point: https://wiki.python.org/moin/WikiGuidelines -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 07 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From skip.montanaro at gmail.com Wed Jan 7 21:55:36 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Wed, 7 Jan 2015 14:55:36 -0600 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <54AD9ACC.9050409@egenix.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <85387qyqas.fsf_-_@benfinney.id.au> <CANc-5Uy0o7f+avet5MM2Jnx7A32065za=PJ+LQy7g4Mocs1dDQ@mail.gmail.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> <CANc-5UzU+o8K1QXLGnybVnj8LvW4HgMA8ZVkM+D+KAyxYXt+pQ@mail.gmail.com> <CAP7+vJLg6Guyusw6DhSYrAP1VYXsWnM-1RaYoURbr14k+2Gq-A@mail.gmail.com> <54AC5B2F.6090504@egenix.com> <CACfEFw8uQwBq7ThcaeDtvUMmBRZnVZE1ZKTLefU0Da+6JXxR1Q@mail.gmail.com> <54AC69E2.6070303@egenix.com> <CACfEFw9sQAd1NrfM7YOLwyjmgPPO9e_5iLtdQkTsDN17Ff13Lg@mail.gmail.com> <54ACF333.80307@egenix.com> <CACfEFw-hB_YSrSEp3w7PKxaJisGZW--+HikZwRvjpPeppdY4UQ@mail.gmail.com> <CANc-5UyZRAZbZhhX6qXyfxfJTUa9EsbSOUvqGqxkObPjCne6dQ@mail.gmail.com> <CACfEFw_GSsJsTkJbREQqdWxB=Rqzku-Arct+qw0Gw7U-qcXz6g@mail.gmail.com> <CANc-5UwbK7tKJ2+osb4x1P2hU05wJoXTwBw05S7veweh68UXDg@mail.gmail.com> <54AD9ACC.9050409@egenix.com> Message-ID: <CANc-5Uxe+FOt8LPzkL6eWU4xSGUW0tp=_7nKQ=8s5c5iwACung@mail.gmail.com> On Wed, Jan 7, 2015 at 2:45 PM, M.-A. Lemburg <mal at egenix.com> wrote: > > ... and then you've changed the technology, lost the dynamic features > of wikis, the permission controls, full text search, but have not > solved the real issues, only created more work that's not relevant > to the content. I'm not sure what the disagreement is here. I wasn't advocating moving away from wiki technology generally, just observing that the current tool (MoinMoin) has different markup syntax and version control than people appear to like (Markdown and GitHub). I imagine you could find a wiki system which uses those two frontend and backend technologies. I was only pointing out that converting from one to the other shouldn't be terribly onerous, as long as you could iterate over the versions and parse the existing syntax into the new form. In any event, I agree that the content and structure is more important than the implementation technology. If you can't solve those problems, changing technology winds up being just a bunch of useless effort. Skip From jeanpierreda at gmail.com Wed Jan 7 22:04:28 2015 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Wed, 7 Jan 2015 15:04:28 -0600 Subject: [Python-ideas] Possible new itertool: comm() In-Reply-To: <54AD79D3.6090104@trueblade.com> References: <CACac1F_7fXwx1diP7QCxQtEPWHMKF9gAaS1+y7bZEKN1FDfj3A@mail.gmail.com> <1BC2F2DA-DC73-4BE3-8550-37F5EA872C07@gmail.com> <CAMpsgwZGZ14KyOWOSAZYbocU78sQ87Rgo4hB6Ba+mb=xm7Stsg@mail.gmail.com> <20150107153847.GG19387@ando.pearwood.info> <CACac1F-ZPccxP0bhDPwBt81tpFSpyHK5OhQABdLNvUORnNftYg@mail.gmail.com> <CAEbHw4bjdKL_vhsTK=Et4i-8VoUkiiEAgjmgczPn1Wnyq3XHcg@mail.gmail.com> <54AD79D3.6090104@trueblade.com> Message-ID: <CABicbJLi1bEkd4A617Pc4F7JhO2AyZeP3SEVApR7m19yyzz_Ag@mail.gmail.com> On Wed, Jan 7, 2015 at 12:24 PM, Eric V. Smith <eric at trueblade.com> wrote: > On 01/07/2015 01:20 PM, David Mertz wrote: >> However, the problem with the functionality is that it assumes that the >> iterables given to it return pre-sorted values. Which is fine it that >> assumption is satisfied, but one really can't know in advance of >> exhausting the iterators whether that will prove true. The desired >> behavior in the absence of this pre-condition being satisfied is unclear >> to me. > > itertools.groupby has the same requirement for a pre-sorted input iterator. groupby doesn't require sorted input, and behaves in an obvious way for jumbled input. But you're right, of all the functions, it's the most like comm() in terms of the kind of requirements placed on input. -- Devin From mal at egenix.com Wed Jan 7 22:11:40 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 07 Jan 2015 22:11:40 +0100 Subject: [Python-ideas] Encouraging more use of the Python wiki In-Reply-To: <CANc-5Uxe+FOt8LPzkL6eWU4xSGUW0tp=_7nKQ=8s5c5iwACung@mail.gmail.com> References: <DC50B4E7-62F7-4756-B08C-2FA928119DEF@yahoo.com> <54AAAD7F.6080201@egenix.com> <85tx04xw43.fsf@benfinney.id.au> <54AB9D76.7030105@egenix.com> <CAP7+vJ+JUC9xg1_o_KoFAJ-AyJj_AZfytPbPh2MRvd5YfzeMkw@mail.gmail.com> <CANc-5UzU+o8K1QXLGnybVnj8LvW4HgMA8ZVkM+D+KAyxYXt+pQ@mail.gmail.com> <CAP7+vJLg6Guyusw6DhSYrAP1VYXsWnM-1RaYoURbr14k+2Gq-A@mail.gmail.com> <54AC5B2F.6090504@egenix.com> <CACfEFw8uQwBq7ThcaeDtvUMmBRZnVZE1ZKTLefU0Da+6JXxR1Q@mail.gmail.com> <54AC69E2.6070303@egenix.com> <CACfEFw9sQAd1NrfM7YOLwyjmgPPO9e_5iLtdQkTsDN17Ff13Lg@mail.gmail.com> <54ACF333.80307@egenix.com> <CACfEFw-hB_YSrSEp3w7PKxaJisGZW--+HikZwRvjpPeppdY4UQ@mail.gmail.com> <CANc-5UyZRAZbZhhX6qXyfxfJTUa9EsbSOUvqGqxkObPjCne6dQ@mail.gmail.com> <CACfEFw_GSsJsTkJbREQqdWxB=Rqzku-Arct+qw0Gw7U-qcXz6g@mail.gmail.com> <CANc-5UwbK7tKJ2+osb4x1P2hU05wJoXTwBw05S7veweh68UXDg@mail.gmail.com> <54AD9ACC.9050409@egenix.com> <CANc-5Uxe+FOt8LPzkL6eWU4xSGUW0tp=_7nKQ=8s5c5iwACung@mail.gmail.com> Message-ID: <54ADA10C.7060609@egenix.com> On 07.01.2015 21:55, Skip Montanaro wrote: > On Wed, Jan 7, 2015 at 2:45 PM, M.-A. Lemburg <mal at egenix.com> wrote: >> >> ... and then you've changed the technology, lost the dynamic features >> of wikis, the permission controls, full text search, but have not >> solved the real issues, only created more work that's not relevant >> to the content. > > I'm not sure what the disagreement is here. I wasn't advocating moving > away from wiki technology generally, just observing that the current > tool (MoinMoin) has different markup syntax and version control than > people appear to like (Markdown and GitHub). I imagine you could find > a wiki system which uses those two frontend and backend technologies. > I was only pointing out that converting from one to the other > shouldn't be terribly onerous, as long as you could iterate over the > versions and parse the existing syntax into the new form. Ah, sorry. I was obviously misunderstanding your emails :-) Moin does support ReST out of the box and there's also an addon parser for Markdown, we could install, if it's stable enough. > In any event, I agree that the content and structure is more important > than the implementation technology. If you can't solve those problems, > changing technology winds up being just a bunch of useless effort. Yep. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 07 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From rosuav at gmail.com Wed Jan 7 23:13:20 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 8 Jan 2015 09:13:20 +1100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAGu0Ansg=r5eTBWOsRztYyBVPYzBJvT9gUU6x9a45-jYPAbSoA@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CAGu0Ansg=r5eTBWOsRztYyBVPYzBJvT9gUU6x9a45-jYPAbSoA@mail.gmail.com> Message-ID: <CAPTjJmqf2s2U7DsjkpyenVHHb8QBbzJUOiLE2cw1xyTGF+_SfA@mail.gmail.com> On Thu, Jan 8, 2015 at 6:02 AM, Bruce Leban <bruce at leban.us> wrote: > There is a problem here but that's not it. The problem is that > > float("nan") is float("nan") => False > math.nan is math.nan => True > > > So is this an attractive nuisance that will make people think they can > should use x is math.nan instead of math.isnan(x)? Hopefully not but > something to consider. Pylint should flag this usage. I don't think that's a problem. Using 'is' to test floating-point values is already broken: >>> math.acos(-1) is math.pi False >>> math.atan(1)*4 is math.pi False >>> 1.0 + 2.0 is 3.0 False There are plenty of calculations that will result in infinity or NaN, and the correct thing to do will be either an equality check or the math.is* functions. Or, as everyone keeps spouting, some kind of epsilon-based "almost equal" check. Or something. But not object identity. ChrisA From guido at python.org Wed Jan 7 23:18:37 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 7 Jan 2015 14:18:37 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAPTjJmqf2s2U7DsjkpyenVHHb8QBbzJUOiLE2cw1xyTGF+_SfA@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CAGu0Ansg=r5eTBWOsRztYyBVPYzBJvT9gUU6x9a45-jYPAbSoA@mail.gmail.com> <CAPTjJmqf2s2U7DsjkpyenVHHb8QBbzJUOiLE2cw1xyTGF+_SfA@mail.gmail.com> Message-ID: <CAP7+vJLuoR2d3+ndpz=8cu4=PLPaJqd5AteZe75wZfF41AbExw@mail.gmail.com> On Wed, Jan 7, 2015 at 2:13 PM, Chris Angelico <rosuav at gmail.com> wrote: > On Thu, Jan 8, 2015 at 6:02 AM, Bruce Leban <bruce at leban.us> wrote: > > There is a problem here but that's not it. The problem is that > > > > float("nan") is float("nan") => False > > math.nan is math.nan => True > > > > > > So is this an attractive nuisance that will make people think they can > > should use x is math.nan instead of math.isnan(x)? Hopefully not but > > something to consider. Pylint should flag this usage. > > I don't think that's a problem. Using 'is' to test floating-point > values is already broken: [...] > Correct. And the first half of the thread was comparing the convenience of defining a nan constant at the top of your module using "nan = float('nan')" with "from math import nan". That would have the exact same 'is' problem. There is no attractive nuisance here except that this thread itself is wasting everybody's time. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/24faebc9/attachment-0001.html> From steve at pearwood.info Thu Jan 8 00:23:05 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 8 Jan 2015 10:23:05 +1100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> Message-ID: <20150107232304.GM19387@ando.pearwood.info> On Thu, Jan 08, 2015 at 05:58:03AM +1100, Alexander Heger wrote: > IEEE754 requires that all comparisons with NaN give false, i.e., Not quite. "Not equal" comparisons should return true. NANs should compare unordered and unequal to everything including themselves. -- Steve From chris.barker at noaa.gov Thu Jan 8 00:31:31 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 7 Jan 2015 15:31:31 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAGu0Ansg=r5eTBWOsRztYyBVPYzBJvT9gUU6x9a45-jYPAbSoA@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CAGu0Ansg=r5eTBWOsRztYyBVPYzBJvT9gUU6x9a45-jYPAbSoA@mail.gmail.com> Message-ID: <CALGmxELPMwiDGphEGQiZBOmaNLZ4EK_t14jMVYx0g7hXBTEXnA@mail.gmail.com> On Wed, Jan 7, 2015 at 11:02 AM, Bruce Leban <bruce at leban.us> wrote: > There is a problem here but that's not it. The problem is that > > float("nan") is float("nan") => False > math.nan is math.nan => True > > > So is this an attractive nuisance that will make people think they can > should use x is math.nan instead of math.isnan(x)? > This "attractive nuisance" exists for ALL python objects, and is especially attractive for interned ints and strings. I have a much harder time getting students to use "is None" than getting them to not use "is" elsewhere. My summary: there isn't a huge reason to add math.nan, but there is essentially zero cost -- what's the problem? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/08c36d8f/attachment.html> From yawar.amin at gmail.com Thu Jan 8 02:05:13 2015 From: yawar.amin at gmail.com (Yawar Amin) Date: Wed, 07 Jan 2015 20:05:13 -0500 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <8C6A6BAA-197B-4796-9163-6EE3AE3E1553@gmail.com> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> <54AB4178.3050508@gmail.com> <97D50920-B7D2-4125-9389-224C4CBC83CA@yahoo.com> <54AC952E.5060301@gmail.com> <74164843-0774-441F-A898-4776E15EFC8D@yahoo.com> <8C6A6BAA-197B-4796-9163-6EE3AE3E1553@gmail.com> Message-ID: <54ADD7C9.9030709@gmail.com> On 2015-01-07 11:20, Yawar Amin wrote: > On 2015-01-07, at 2:11, Andrew Barnert <abarnert at yahoo.com> wrote: >> [...] >> That's the key. A function is a way to wrap a sequence of statements >> (the function body) so it can be used in an expression (a function >> call). Because JavaScript lets you define functions (with full >> statement syntax) in expressions, that gives you a way to "demote" an >> expression inline, while Python can only do so out-of-line. At the >> core, this is what all of the multiline lambda attempts and similar >> proposals are trying to accomplish. > > Thanks. This has given me an idea that's on a slightly different > tangent than this one. Will write it up (this time with some actual > runnable code) when I get home tonight. As promised, a write-up of my idea: http://yawar.blogspot.ca/2015/01/expressive-functional-programming-with.html I have a feeling that some Pythoneers will really hate it, but hey, what can you do :-) Regards, Yawar -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 834 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/f2cd2184/attachment.sig> From rosuav at gmail.com Thu Jan 8 02:12:56 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 8 Jan 2015 12:12:56 +1100 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <54ADD7C9.9030709@gmail.com> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> <54AB4178.3050508@gmail.com> <97D50920-B7D2-4125-9389-224C4CBC83CA@yahoo.com> <54AC952E.5060301@gmail.com> <74164843-0774-441F-A898-4776E15EFC8D@yahoo.com> <8C6A6BAA-197B-4796-9163-6EE3AE3E1553@gmail.com> <54ADD7C9.9030709@gmail.com> Message-ID: <CAPTjJmq=YvDSeXnyoppJ378VJ3umU3mgNoe4UWt2YunJPjp8KA@mail.gmail.com> On Thu, Jan 8, 2015 at 12:05 PM, Yawar Amin <yawar.amin at gmail.com> wrote: > As promised, a write-up of my idea: > http://yawar.blogspot.ca/2015/01/expressive-functional-programming-with.html > > I have a feeling that some Pythoneers will really hate it, but hey, what > can you do :-) If you're coming to python-ideas, you probably should have in your head that 'print' is a function and not a statement :) I'm really not seeing the advantage of this system; instead of expressing your code as a series of statements, you have to nest everything inside everything else, and the advantage is... the purity of having everything be an expression. Seems to me this has purpose in penetration testing (eg proving that you can get past someone else's sandboxing), but not practical programming. ChrisA From yawar.amin at gmail.com Thu Jan 8 02:41:47 2015 From: yawar.amin at gmail.com (Yawar Amin) Date: Wed, 07 Jan 2015 20:41:47 -0500 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <CAPTjJmq=YvDSeXnyoppJ378VJ3umU3mgNoe4UWt2YunJPjp8KA@mail.gmail.com> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> <54AB4178.3050508@gmail.com> <97D50920-B7D2-4125-9389-224C4CBC83CA@yahoo.com> <54AC952E.5060301@gmail.com> <74164843-0774-441F-A898-4776E15EFC8D@yahoo.com> <8C6A6BAA-197B-4796-9163-6EE3AE3E1553@gmail.com> <54ADD7C9.9030709@gmail.com> <CAPTjJmq=YvDSeXnyoppJ378VJ3umU3mgNoe4UWt2YunJPjp8KA@mail.gmail.com> Message-ID: <54ADE05B.6060209@gmail.com> On 2015-01-07 20:12, Chris Angelico wrote: > [...] > If you're coming to python-ideas, you probably should have in your > head that 'print' is a function and not a statement :) Ah, I'll have to remember that from next time onwards to continue hiding my shame: I'm still on 2.7.3. > I'm really not seeing the advantage of this system; instead of > expressing your code as a series of statements, you have to nest > everything inside everything else, and the advantage is... the purity > of having everything be an expression. Seems to me this has purpose in > penetration testing (eg proving that you can get past someone else's > sandboxing), but not practical programming. Agreed, it doesn't look too great, but then again every idea starts out with someone saying 'That's never going to have a practical application', so I could take that as a good sign :-) Regards, Yawar -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 834 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/a230aac8/attachment.sig> From ben+python at benfinney.id.au Thu Jan 8 02:47:35 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 08 Jan 2015 12:47:35 +1100 Subject: [Python-ideas] Syntax: 'return: ...' expressions References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> <54AB4178.3050508@gmail.com> <97D50920-B7D2-4125-9389-224C4CBC83CA@yahoo.com> <54AC952E.5060301@gmail.com> <74164843-0774-441F-A898-4776E15EFC8D@yahoo.com> <8C6A6BAA-197B-4796-9163-6EE3AE3E1553@gmail.com> <54ADD7C9.9030709@gmail.com> <CAPTjJmq=YvDSeXnyoppJ378VJ3umU3mgNoe4UWt2YunJPjp8KA@mail.gmail.com> <54ADE05B.6060209@gmail.com> Message-ID: <85a91uxcug.fsf@benfinney.id.au> Yawar Amin <yawar.amin at gmail.com> writes: > Ah, I'll have to remember that from next time onwards to continue > hiding my shame: I'm still on 2.7.3. If you are interested in the future of Python (which is what this forum entails), you will be at cross purposes until you switch to Python 3. Those who want to continue writing for Python 2 are welcome to do so, but Python 2 is not a consideration for discussing improvements to Python. -- \ ?A life spent making mistakes is not only most honorable but | `\ more useful than a life spent doing nothing.? ?anonymous | _o__) | Ben Finney From yawar.amin at gmail.com Thu Jan 8 03:27:15 2015 From: yawar.amin at gmail.com (Yawar Amin) Date: Wed, 07 Jan 2015 21:27:15 -0500 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <85a91uxcug.fsf@benfinney.id.au> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> <54AB4178.3050508@gmail.com> <97D50920-B7D2-4125-9389-224C4CBC83CA@yahoo.com> <54AC952E.5060301@gmail.com> <74164843-0774-441F-A898-4776E15EFC8D@yahoo.com> <8C6A6BAA-197B-4796-9163-6EE3AE3E1553@gmail.com> <54ADD7C9.9030709@gmail.com> <CAPTjJmq=YvDSeXnyoppJ378VJ3umU3mgNoe4UWt2YunJPjp8KA@mail.gmail.com> <54ADE05B.6060209@gmail.com> <85a91uxcug.fsf@benfinney.id.au> Message-ID: <54ADEB03.609@gmail.com> Hi Ben, On 2015-01-07 20:47, Ben Finney wrote: > [...] > If you are interested in the future of Python (which is what this forum > entails), you will be at cross purposes until you switch to Python 3. > > Those who want to continue writing for Python 2 are welcome to do so, > but Python 2 is not a consideration for discussing improvements to > Python. I'm aware of the current Python version and, as far as I can tell, have never proposed anything that would only work with an older version. The comments you're replying to are about something I posted on my own blog, which you'll agree is not python-ideas. Regards, Yawar -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 834 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150107/c3cd780a/attachment.sig> From rosuav at gmail.com Thu Jan 8 04:26:50 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 8 Jan 2015 14:26:50 +1100 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <54ADEB03.609@gmail.com> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> <54AB4178.3050508@gmail.com> <97D50920-B7D2-4125-9389-224C4CBC83CA@yahoo.com> <54AC952E.5060301@gmail.com> <74164843-0774-441F-A898-4776E15EFC8D@yahoo.com> <8C6A6BAA-197B-4796-9163-6EE3AE3E1553@gmail.com> <54ADD7C9.9030709@gmail.com> <CAPTjJmq=YvDSeXnyoppJ378VJ3umU3mgNoe4UWt2YunJPjp8KA@mail.gmail.com> <54ADE05B.6060209@gmail.com> <85a91uxcug.fsf@benfinney.id.au> <54ADEB03.609@gmail.com> Message-ID: <CAPTjJmqfURFmLHSC2Yd=xm4hKgxYbA8=YmQDfjpP36mGj2Vvqw@mail.gmail.com> On Thu, Jan 8, 2015 at 1:27 PM, Yawar Amin <yawar.amin at gmail.com> wrote: > On 2015-01-07 20:47, Ben Finney wrote: >> [...] >> If you are interested in the future of Python (which is what this forum >> entails), you will be at cross purposes until you switch to Python 3. >> >> Those who want to continue writing for Python 2 are welcome to do so, >> but Python 2 is not a consideration for discussing improvements to >> Python. > > I'm aware of the current Python version and, as far as I can tell, have > never proposed anything that would only work with an older version. > > The comments you're replying to are about something I posted on my own > blog, which you'll agree is not python-ideas. My apologies, I started this sub-thread in semi-jocular response to your assumption (on the blog) that 'print' was a statement. If you use other statements, your code will (as far as I know) work equally on 2.7 and 3.x, which is an advantage, not a disadvantage. The content of your post is all stuff that can be done _without_ proposing changes to the language, so it's perfectly appropriate to talk about both branches. ChrisA From abarnert at yahoo.com Thu Jan 8 16:32:41 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 8 Jan 2015 07:32:41 -0800 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <54ADD7C9.9030709@gmail.com> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> <54AB4178.3050508@gmail.com> <97D50920-B7D2-4125-9389-224C4CBC83CA@yahoo.com> <54AC952E.5060301@gmail.com> <74164843-0774-441F-A898-4776E15EFC8D@yahoo.com> <8C6A6BAA-197B-4796-9163-6EE3AE3E1553@gmail.com> <54ADD7C9.9030709@gmail.com> Message-ID: <F86DAA65-1E17-4BDE-9D21-D54AF9567857@yahoo.com> On Jan 7, 2015, at 17:05, Yawar Amin <yawar.amin at gmail.com> wrote: > On 2015-01-07 11:20, Yawar Amin wrote: >> On 2015-01-07, at 2:11, Andrew Barnert <abarnert at yahoo.com> wrote: >>> [...] > >>> That's the key. A function is a way to wrap a sequence of statements >>> (the function body) so it can be used in an expression (a function >>> call). Because JavaScript lets you define functions (with full >>> statement syntax) in expressions, that gives you a way to "demote" an >>> expression inline, while Python can only do so out-of-line. At the >>> core, this is what all of the multiline lambda attempts and similar >>> proposals are trying to accomplish. >> >> Thanks. This has given me an idea that's on a slightly different >> tangent than this one. Will write it up (this time with some actual >> runnable code) when I get home tonight. > > As promised, a write-up of my idea: > http://yawar.blogspot.ca/2015/01/expressive-functional-programming-with.html > > I have a feeling that some Pythoneers will really hate it, but hey, what > can you do :-) Well, I think you've successfully proven that for imperative-style code, a statement-expression language is more readable than a CPS pure-expression language. :) The difference isn't just a lack of syntactic sugar: in a pure expression language, "is followed by" has to be modeled as "is controlled by" (with CPS that's made explicit), meaning (in Python) it has to be indented. Haskell's do statement has a nice way out of that (if you really want to turn off Pythonistas by using the word "monad"). Promise-based sequencing (as with most JS Promises/A-based code) offers a different solution with a similar effect. It might be worth following one of those ideas to see where it leads you. In fact, the latter raises another possibility: once you're yielding from promises, you've got coroutines for free. In at some cases, explicit CPS code can be turned into implicit coroutine code very easily. If that works for everything, it also means you no longer need the manual trampoline and/or you can make all of your sequencing asynchronous and therefore add in concurrency for free (as in asyncio). Another trick that might be worth playing with explicit native globals and locals for your functions (e.g., by constructing a new types.FunctionType), so let bindings (which also take care of def, by just binding a lambda, and class, by binding a call to type or the appropriate metaclass) can use native Python environments (and closures). This would be more interesting if you wanted to add state later (because you'd get it for free), but it might be interesting just to see where you run into problems. Anyway, while this raises some interesting ideas, I don't think it raises any ideas for the future of the Python language, so I'm not sure Python-ideas is the right place to discuss it. From toddrjen at gmail.com Thu Jan 8 17:01:03 2015 From: toddrjen at gmail.com (Todd) Date: Thu, 8 Jan 2015 17:01:03 +0100 Subject: [Python-ideas] Float range class Message-ID: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> Currently, the range class is limited to integers. However, in many applications, especially science and mathematics, ranges of float values are the norm. The current approaches for float ranges all have significant limitations. numpy.arange pre-allocates the values and doesn't use python floats, both of which affect performance. List comprehensions also have to be pre-allocated. Generators and values created in the body of a loop cannot be indexed or re-used and are much more verbose and harder to read. So it would be nice to have a float equivalent of the existing range class. Assuming such a class is worthwhile, I see three possible ways to handle float ranges, each with its own strengths and weaknesses. I am not sure which, if any, is preferable. The first is to extend the existing range class so it can support floats or to use duck-typing to support any class that has the necessary methods. However, I would be worried about making such a fundamental change to such a core class, and either approach could hurt performance. The second approach is to create a new class that handles floating-point ranges (perhaps called "frange") and/or a new class that handles duck-typed ranges (perhaps called "drange"). However, there isn't really a good place right now to put these as far as I can see. The third approach is to create a new module for various range classes. For example there could be a float range, a datetime range, a decimal range, a duck-typed range, an abstract base class for making new range classes, etc. However, this is a lot more work (although it wouldn't need to all be done at one time). -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/3c93291f/attachment.html> From mal at egenix.com Thu Jan 8 17:18:32 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 08 Jan 2015 17:18:32 +0100 Subject: [Python-ideas] Float range class In-Reply-To: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> Message-ID: <54AEADD8.4030705@egenix.com> On 08.01.2015 17:01, Todd wrote: > Currently, the range class is limited to integers. However, in many > applications, especially science and mathematics, ranges of float values > are the norm. > > The current approaches for float ranges all have significant limitations. > numpy.arange pre-allocates the values and doesn't use python floats, both > of which affect performance. List comprehensions also have to be > pre-allocated. Generators and values created in the body of a loop cannot > be indexed or re-used and are much more verbose and harder to read. So it > would be nice to have a float equivalent of the existing range class. > > Assuming such a class is worthwhile, I see three possible ways to handle > float ranges, each with its own strengths and weaknesses. I am not sure > which, if any, is preferable. > > The first is to extend the existing range class so it can support floats or > to use duck-typing to support any class that has the necessary methods. > However, I would be worried about making such a fundamental change to such > a core class, and either approach could hurt performance. > > The second approach is to create a new class that handles floating-point > ranges (perhaps called "frange") and/or a new class that handles duck-typed > ranges (perhaps called "drange"). However, there isn't really a good place > right now to put these as far as I can see. > > The third approach is to create a new module for various range classes. > For example there could be a float range, a datetime range, a decimal > range, a duck-typed range, an abstract base class for making new range > classes, etc. However, this is a lot more work (although it wouldn't need > to all be done at one time). I think you'd first have to describe some use cases, since it's not at all clear what you mean with "float range", e.g. you could be referring to: * interval arithmetics: http://en.wikipedia.org/wiki/Interval_arithmetic * ranges of numbers with a fixed stepping (which can create problems near the edges due to rounding issues) * ranges in the sense of mathematical sets, where you're mostly interested in using set operations etc. Looping over such ranges doesn't sound like a useful operation, except for the case where you have a fixed stepping (and those can easily be implemented by mapping an integer to a stepping in your interval). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 08 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From chris.barker at noaa.gov Thu Jan 8 17:31:43 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 8 Jan 2015 08:31:43 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> Message-ID: <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> On Thu, Jan 8, 2015 at 8:01 AM, Todd <toddrjen at gmail.com> wrote: > Currently, the range class is limited to integers. However, in many > applications, especially science and mathematics, ranges of float values > are the norm. > almost -- see below. > So it would be nice to have a float equivalent of the existing range > class. > +1 > The second approach is to create a new class that handles floating-point > ranges > I think this is the way to go, but not for the same reasons you gave. from numpy experience: numpy provides an arange() ("array range", so names back in the days of import *) with integer inputs is is essentially the same as the py2 range() function -- i.e. creates a 1-d numpy array of integers using start, stop, step. Called with floating point arguments, it does much the same thing, but with floats, and, of course, start, stop and step need not be integers -- but here's the rub: it does this in the naive way: starting with start, continuing to add step, until the value is >= stop, at which point it terminates without including that last value. This is fine and dandy for integers, but is a mess with floats, due to rounding and binary storage under the hood (i.e floats can not precisely represent common base-10 steps). This is particularly a problem when you ant a particular end point to be included, you then need to increase stop to a bit more than you want, so as to make sure it gets included, but then you can accidentally get one too many. This is almost never what people really want, though it does work as expected most of the time -- that most of the time is the catch -- future bugs lurking. So: the accepted practice in numpy (and often espoused to newbies on the mailing list) is to not use arange() for floats. Instead, the "linspace()" function is recommended. This function returns evenly spaced numbers over a specified interval, guaranteeing that you get the start an end points you want included. It has optional arguments that let you define the spacing in various ways, but the key is that the algorithms are designed to give the user what is expected, regardless of floating point nuance. Anyway, if such an object were to be added to the pyton standard libary, it should be more like the numpy linspace(), rather than range. -Chris linspace docstring:: --------------------------- Return evenly spaced numbers over a specified interval. Returns `num` evenly spaced samples, calculated over the interval [`start`, `stop` ]. The endpoint of the interval can optionally be excluded. Parameters ---------- start : scalar The starting value of the sequence. stop : scalar The end value of the sequence, unless `endpoint` is set to False. In that case, the sequence consists of all but the last of ``num + 1`` evenly spaced samples, so that `stop` is excluded. Note that the step size changes when `endpoint` is False. num : int, optional Number of samples to generate. Default is 50. endpoint : bool, optional If True, `stop` is the last sample. Otherwise, it is not included. Default is True. retstep : bool, optional If True, return (`samples`, `step`), where `step` is the spacing between samples. dtype : dtype The type of the output array. If `dtype` is not given, infer the data type from the other input arguments. .. versionadded:: 1.9.0 Returns ------- samples : ndarray There are `num` equally spaced samples in the closed interval ``[start, stop]`` or the half-open interval ``[start, stop)`` (depending on whether `endpoint` is True or False). step : float (only if `retstep` is True) Size of spacing between samples. > (perhaps called "frange") and/or a new class that handles duck-typed > ranges (perhaps called "drange"). However, there isn't really a good place > right now to put these as far as I can see. > > > The third approach is to create a new module for various range classes. > For example there could be a float range, a datetime range, a decimal > range, a duck-typed range, an abstract base class for making new range > classes, etc. However, this is a lot more work (although it wouldn't need > to all be done at one time). > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/ca1ed572/attachment-0001.html> From toddrjen at gmail.com Thu Jan 8 18:12:32 2015 From: toddrjen at gmail.com (Todd) Date: Thu, 8 Jan 2015 18:12:32 +0100 Subject: [Python-ideas] Float range class In-Reply-To: <54AEADD8.4030705@egenix.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <54AEADD8.4030705@egenix.com> Message-ID: <CAFpSVp+sVKBqRgsCShGKotwGdViTj2WsG0KFE2fANwEADtMH-w@mail.gmail.com> On Thu, Jan 8, 2015 at 5:18 PM, M.-A. Lemburg <mal at egenix.com> wrote: > On 08.01.2015 17:01, Todd wrote: > > Currently, the range class is limited to integers. However, in many > > applications, especially science and mathematics, ranges of float values > > are the norm. > > > > The current approaches for float ranges all have significant limitations. > > numpy.arange pre-allocates the values and doesn't use python floats, both > > of which affect performance. List comprehensions also have to be > > pre-allocated. Generators and values created in the body of a loop > cannot > > be indexed or re-used and are much more verbose and harder to read. So > it > > would be nice to have a float equivalent of the existing range class. > > > > Assuming such a class is worthwhile, I see three possible ways to handle > > float ranges, each with its own strengths and weaknesses. I am not sure > > which, if any, is preferable. > > > > The first is to extend the existing range class so it can support floats > or > > to use duck-typing to support any class that has the necessary methods. > > However, I would be worried about making such a fundamental change to > such > > a core class, and either approach could hurt performance. > > > > The second approach is to create a new class that handles floating-point > > ranges (perhaps called "frange") and/or a new class that handles > duck-typed > > ranges (perhaps called "drange"). However, there isn't really a good > place > > right now to put these as far as I can see. > > > > The third approach is to create a new module for various range classes. > > For example there could be a float range, a datetime range, a decimal > > range, a duck-typed range, an abstract base class for making new range > > classes, etc. However, this is a lot more work (although it wouldn't > need > > to all be done at one time). > > I think you'd first have to describe some use cases, since it's > not at all clear what you mean with "float range", e.g. you > could be referring to: > > * interval arithmetics: http://en.wikipedia.org/wiki/Interval_arithmetic > > * ranges of numbers with a fixed stepping (which can create problems > near the edges due to rounding issues) > This is what I am talking about. The idea would be to have something that is, as much as possible, identical to the existing range. I understand it has floating-point issues, but any other approach would as well. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/0456dd29/attachment.html> From toddrjen at gmail.com Thu Jan 8 18:22:27 2015 From: toddrjen at gmail.com (Todd) Date: Thu, 8 Jan 2015 18:22:27 +0100 Subject: [Python-ideas] Float range class In-Reply-To: <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> Message-ID: <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> On Thu, Jan 8, 2015 at 5:31 PM, Chris Barker <chris.barker at noaa.gov> wrote: > it does this in the naive way: starting with start, continuing to add > step, until the value is >= stop, at which point it terminates without > including that last value. This is fine and dandy for integers, but is a > mess with floats, due to rounding and binary storage under the hood (i.e > floats can not precisely represent common base-10 steps). This is > particularly a problem when you ant a particular end point to be included, > you then need to increase stop to a bit more than you want, so as to make > sure it gets included, but then you can accidentally get one too many. > I would think that a floating range class would necessarily use multiplication rather than repeated addition (to allow indexing at arbitrary point), which would avoid cumulative floating-point errors (although it would still have a smaller floating point error at the end), and for the same reason the final value would have to be pre-computed rather than using a naive ">=" which would allow it to be a bit smarter. > This is almost never what people really want, though it does work as > expected most of the time -- that most of the time is the catch -- future > bugs lurking. > > So: the accepted practice in numpy (and often espoused to newbies on the > mailing list) is to not use arange() for floats. Instead, the "linspace()" > function is recommended. This function returns evenly spaced numbers over a > specified interval, guaranteeing that you get the start an end points you > want included. It has optional arguments that let you define the spacing in > various ways, but the key is that the algorithms are designed to give the > user what is expected, regardless of floating point nuance. > > If there was a range module (rangetools?), it could also contain a count-based floating point class. But, at least in my own experience, I use arange when I want an interval-based range, and linspace when I want a count-based range. It just depends on what, exactly, I am trying to do. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/01dae22c/attachment.html> From abarnert at yahoo.com Thu Jan 8 18:41:00 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 8 Jan 2015 09:41:00 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> Message-ID: <BF30F199-DFDA-4003-B160-6CFFBD0DD14D@yahoo.com> On Jan 8, 2015, at 8:01, Todd <toddrjen at gmail.com> wrote: > The current approaches for float ranges all have significant limitations. numpy.arange pre-allocates the values and doesn't use python floats, both of which affect performance. > How does using numpy floats instead of python floats affect performance, unless "it's too fast" or "it uses too little memory" are problems? > The third approach is to create a new module for various range classes. For example there could be a float range, a datetime range, a decimal range, a duck-typed range, an abstract base class for making new range classes, etc. However, this is a lot more work (although it wouldn't need to all be done at one time). > I think what you'd really want is a full suite of interval/count/range APIs over various types. For example, a datetime range that doesn't even allow offsetting or comparing for overlap doesn't seem all that useful. I can think of lots of uses for all kinds of range-like things, but none of them are well satisfied by just a range view that acts like a sequence and has no other methods, short of iterating them (for which a generator expression is sufficient--or, if you're doing a lot of them in some app for some reason, a one-liner function that wraps up the genexpr). More generally: if an frange that works exactly like range but with floats (and rounding errors) is all you want, that's so trivial to write that it doesn't seem worth putting in the stdlib, unless the use case is very common. If you want more than that, then whether it belongs in the stdlib probably depends on exactly what you want (and whether you or someone else has a good design and implementation, of course). -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/20896974/attachment.html> From solipsis at pitrou.net Thu Jan 8 18:45:16 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 8 Jan 2015 18:45:16 +0100 Subject: [Python-ideas] Float range class References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> Message-ID: <20150108184516.27ee2d84@fsol> On Thu, 8 Jan 2015 17:01:03 +0100 Todd <toddrjen at gmail.com> wrote: > Currently, the range class is limited to integers. However, in many > applications, especially science and mathematics, ranges of float values > are the norm. > > The current approaches for float ranges all have significant limitations. > numpy.arange pre-allocates the values and doesn't use python floats, both > of which affect performance. List comprehensions also have to be > pre-allocated. Generators and values created in the body of a loop cannot > be indexed or re-used and are much more verbose and harder to read. So it > would be nice to have a float equivalent of the existing range class. What use do you have for a lazily allocated floating-point range? Regards Antoine. From toddrjen at gmail.com Thu Jan 8 18:58:41 2015 From: toddrjen at gmail.com (Todd) Date: Thu, 8 Jan 2015 18:58:41 +0100 Subject: [Python-ideas] Float range class In-Reply-To: <BF30F199-DFDA-4003-B160-6CFFBD0DD14D@yahoo.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <BF30F199-DFDA-4003-B160-6CFFBD0DD14D@yahoo.com> Message-ID: <CAFpSVpKxUbY+azZRPY+=vo3Ht8RuchruUh1AN56EzKCjcwTiiQ@mail.gmail.com> On Thu, Jan 8, 2015 at 6:41 PM, Andrew Barnert <abarnert at yahoo.com> wrote: > On Jan 8, 2015, at 8:01, Todd <toddrjen at gmail.com> wrote: > > The current approaches for float ranges all have significant limitations. > numpy.arange pre-allocates the values and doesn't use python floats, both > of which affect performance. > > How does using numpy floats instead of python floats affect performance, > unless "it's too fast" or "it uses too little memory" are problems? > If you are using numpy arrays with numpy functions, it isn't an issue. But if you are using it in a similar manner to the existing range class, at some point it is likely going to interact with python native types (such as adding a numpy scalar to a python float). This require a type conversion, either python float to numpy scalar or vice versus, and that is relatively slow. I have profiled it, this can be the primary bottleneck in numerical code that can't be vectorized for whatever reason. The third approach is to create a new module for various range classes. > For example there could be a float range, a datetime range, a decimal > range, a duck-typed range, an abstract base class for making new range > classes, etc. However, this is a lot more work (although it wouldn't need > to all be done at one time). > > I think what you'd really want is a full suite of interval/count/range > APIs over various types. For example, a datetime range that doesn't even > allow offsetting or comparing for overlap doesn't seem all that useful. I > can think of lots of uses for all kinds of range-like things, but none of > them are well satisfied by just a range view that acts like a sequence and > has no other methods, short of iterating them (for which a generator > expression is sufficient--or, if you're doing a lot of them in some app for > some reason, a one-liner function that wraps up the genexpr). > > The capabilities of the classes in the module would need to be discussed if there was actually a desire for such a module. I personally only commonly see float ranges, so I limited the discussion mostly to those. However, I wanted to be as comprehensive as possible about the options, so provided some rough examples of the sorts of things that could potentially appear in such a module. Since I don't really have a use for the additional classes I don't really know what their specific needs would be. > More generally: if an frange that works exactly like range but with floats > (and rounding errors) is all you want, that's so trivial to write that it > doesn't seem worth putting in the stdlib, unless the use case is very > common. > In numerical computing the use-case is extremely common, in fact I would say it practically universal. Ideally it would be vectorized, but in practice this is often not possible, requiring the use of loops. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/a4df8c34/attachment.html> From toddrjen at gmail.com Thu Jan 8 19:05:53 2015 From: toddrjen at gmail.com (Todd) Date: Thu, 8 Jan 2015 19:05:53 +0100 Subject: [Python-ideas] Float range class In-Reply-To: <20150108184516.27ee2d84@fsol> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <20150108184516.27ee2d84@fsol> Message-ID: <CAFpSVpJH+yNpU7EXAB9g2p1tz0WscY+DW2FAu7w60h+SKLTKyQ@mail.gmail.com> On Thu, Jan 8, 2015 at 6:45 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Thu, 8 Jan 2015 17:01:03 +0100 > Todd <toddrjen at gmail.com> wrote: > > Currently, the range class is limited to integers. However, in many > > applications, especially science and mathematics, ranges of float values > > are the norm. > > > > The current approaches for float ranges all have significant limitations. > > numpy.arange pre-allocates the values and doesn't use python floats, both > > of which affect performance. List comprehensions also have to be > > pre-allocated. Generators and values created in the body of a loop > cannot > > be indexed or re-used and are much more verbose and harder to read. So > it > > would be nice to have a float equivalent of the existing range class. > > What use do you have for a lazily allocated floating-point range? > > Regards > > Antoine. > > The most common use-case is similar to the existing range: looping or iterating over values needed for some computation. This can become much more complicated, however, in many numerical algorithms where the exact order of the values you access cannot be known beforehand. You can save a huge amount of memory by not pre-allocating an array of such values, but that requires making the range indexable. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/babc0d98/attachment.html> From njs at pobox.com Thu Jan 8 19:40:12 2015 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 8 Jan 2015 18:40:12 +0000 Subject: [Python-ideas] Float range class In-Reply-To: <CAFpSVpKxUbY+azZRPY+=vo3Ht8RuchruUh1AN56EzKCjcwTiiQ@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <BF30F199-DFDA-4003-B160-6CFFBD0DD14D@yahoo.com> <CAFpSVpKxUbY+azZRPY+=vo3Ht8RuchruUh1AN56EzKCjcwTiiQ@mail.gmail.com> Message-ID: <CAPJVwB=HXy6P7tGV_wiPo1P2Fsz1jEP5ePcECw_4LMSLQu-3hA@mail.gmail.com> On Thu, Jan 8, 2015 at 5:58 PM, Todd <toddrjen at gmail.com> wrote: > On Thu, Jan 8, 2015 at 6:41 PM, Andrew Barnert <abarnert at yahoo.com> wrote: >> >> More generally: if an frange that works exactly like range but with floats >> (and rounding errors) is all you want, that's so trivial to write that it >> doesn't seem worth putting in the stdlib, unless the use case is very >> common. > > In numerical computing the use-case is extremely common, in fact I would say > it practically universal. Ideally it would be vectorized, but in practice > this is often not possible, requiring the use of loops. Can you give a more specific example? I believe your problem is real, but as a numerical programmer I can't think of an example of where I've ever wanted anything like this, so it's hard for me to visualize what exactly you're talking about. And is what you want just: class FRange: def __init__(start, stop, step): self.start = start self.stop = stop self.step = step def __getitem__(self, i): x = self.start + i * self.step if x > self.stop: raise IndexError def __iter__(self): i = 0 while True: try: yield self[i] except IndexError: return i += 1 ? -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From ethan at stoneleaf.us Thu Jan 8 20:17:50 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 08 Jan 2015 11:17:50 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <CAPJVwB=HXy6P7tGV_wiPo1P2Fsz1jEP5ePcECw_4LMSLQu-3hA@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <BF30F199-DFDA-4003-B160-6CFFBD0DD14D@yahoo.com> <CAFpSVpKxUbY+azZRPY+=vo3Ht8RuchruUh1AN56EzKCjcwTiiQ@mail.gmail.com> <CAPJVwB=HXy6P7tGV_wiPo1P2Fsz1jEP5ePcECw_4LMSLQu-3hA@mail.gmail.com> Message-ID: <54AED7DE.7080708@stoneleaf.us> On 01/08/2015 10:40 AM, Nathaniel Smith wrote: > class FRange: > def __init__(start, stop, step): > self.start = start > self.stop = stop > self.step = step > > def __getitem__(self, i): > x = self.start + i * self.step > if x > self.stop: > raise IndexError return x # ;) > > def __iter__(self): > i = 0 > while True: > try: > yield self[i] > except IndexError: > return > i += 1 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/2e601a15/attachment.sig> From chris.barker at noaa.gov Thu Jan 8 20:14:04 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 8 Jan 2015 11:14:04 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <CAFpSVpKxUbY+azZRPY+=vo3Ht8RuchruUh1AN56EzKCjcwTiiQ@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <BF30F199-DFDA-4003-B160-6CFFBD0DD14D@yahoo.com> <CAFpSVpKxUbY+azZRPY+=vo3Ht8RuchruUh1AN56EzKCjcwTiiQ@mail.gmail.com> Message-ID: <CALGmxEJPANRfJ5rSYWPNeCXEi7-aed6Rj6L9cuQ2TMqgPMzRcg@mail.gmail.com> On Thu, Jan 8, 2015 at 9:58 AM, Todd <toddrjen at gmail.com> wrote: > How does using numpy floats instead of python floats affect performance, >> unless "it's too fast" or "it uses too little memory" are problems? >> > > ... > If you are using numpy arrays with numpy functions, it isn't an issue. > But if you are using it in a similar manner to the existing range class, at > some point it is likely going to interact with python native types (such as > adding a numpy scalar to a python float). This require a type conversion, > either python float to numpy scalar or vice versus, and that is relatively > slow. I have profiled it, this can be the primary bottleneck in numerical > code that can't be vectorized for whatever reason. > Sure, but this is a core issue with numpy, period. There are, in fact, some things that work faster with python native data types than numpy. But if you are interacting with numpy arrays, it's hard for me to imagine that this one bottleneck is going to be key. I think the real use case here is for code that is not using numpy at all. The third approach is to create a new module for various range classes. >> For example there could be a float range, a datetime range, a decimal >> range, a duck-typed range, >> >> wouldn't a single duck-typed range cover all the bases, though perhaps with less than optimum performance? By the way, numpy's range works with virtually all types, including datetime64: In [21]: np.arange(np.datetime64('2014-01-01'), np.datetime64('2014-01-10')) Out[21]: array(['2014-01-01', '2014-01-02', '2014-01-03', '2014-01-04', '2014-01-05', '2014-01-06', '2014-01-07', '2014-01-08', '2014-01-09'], dtype='datetime64[D]') -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/20f52390/attachment-0001.html> From chris.barker at noaa.gov Thu Jan 8 20:05:46 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 8 Jan 2015 11:05:46 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> Message-ID: <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> > > I would think that a floating range class would necessarily use > multiplication rather than repeated addition (to allow indexing at > arbitrary point), which would avoid cumulative floating-point errors > (although it would still have a smaller floating point error at the end), > and for the same reason the final value would have to be pre-computed > rather than using a naive ">=" which would allow it to be a bit smarter. > That's the trick -- range() (and arange) does not specify a final value, it specifies a final value not to include. This is well defined and easy to understand for integers, but not so for floating point. But you are right about the multiplication and pre-computing of final value -- that's a good reason to provide this as a built-in -- it's very easy to implement, but even easier to implement badly. But, at least in my own experience, I use arange when I want an >> interval-based range, and linspace when I want a count-based range. >> > I would argue (and do!) that you should not do this -- if you know what you are doing with FP, then fine, but it really is tricky. You would be better off computing the count you want then then using linspace anyway. I suppose an interval-based API to something like linspace would be a nice convenience, though. I haven't managed to come up with a quick an easy example where this matters, but they DO happen. I guess I'm arguing that a range-like object for FP should be a closed rather than open interval -- specifying the starting and end points. That is because defining an open interval where numbers are of finite, but hard to know know in advance the interval, is just too ugly and complex. I see your point that sometimes you want a specific delta, and sometimes you want a specific end point, but I supect that most of the time you want a specific delta you ALSO want a specific and point and/or want to know how many values you are going to get. In fact, in the most common use of integer range, you really are defining the number of values you want. And note that the range convention of starting at zero and not including the stop value was designed to match python indexing convention, i.e.: for i in range( len(sequence) ): ... is natural an easy to write, and does what's expected. And also: for i in range(n): .... will loop n times. but neither of these apply to floating point ranges. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/71c2c75b/attachment.html> From chris.barker at noaa.gov Thu Jan 8 17:42:06 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 8 Jan 2015 08:42:06 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <54AEADD8.4030705@egenix.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <54AEADD8.4030705@egenix.com> Message-ID: <CALGmxEKqOoLYpDsvCyytYvRKVC3BvDTVHivF9p8UzYa0=AknaQ@mail.gmail.com> On Thu, Jan 8, 2015 at 8:18 AM, M.-A. Lemburg <mal at egenix.com> wrote: > I think you'd first have to describe some use cases, > <snip> > * ranges of numbers with a fixed stepping (which can create problems > near the edges due to rounding issues) > <snip> > Looping over such ranges doesn't sound like a useful operation, > Good point -- it is very common to want such a range of number in a sequence -- but does it need to be an iterator? Though in keeping with python's move to iterators by default, why not? > except for the case where you have a fixed stepping (and those > can easily be implemented by mapping an integer to a stepping > in your interval). > and range can be easily implemented with a while loop. Such fixed stepping sequences are really common, so convenience functions make sense. But more to the point, as you mention above (and I mentioned in a previous email), there can be problems near the edges due to rounding issues, so it would be nice to provide a built-in that addresses those problems consistently and predictably, rather than having each user write their own naive implementation and get it wrong much of the time. NOTE: for many of us those common use-cases also involve doing a bunch of computation of arrays of numbers, and thus involve numpy anyway. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/3709c00f/attachment.html> From Russell.S.Stewart at gmail.com Thu Jan 8 22:11:42 2015 From: Russell.S.Stewart at gmail.com (Russell Stewart) Date: Thu, 8 Jan 2015 13:11:42 -0800 Subject: [Python-ideas] Better comand line version of python -c Message-ID: <CAATkbzBPf_u3CqYMtpSUoEJFV56ouqAa5UGAUNK2ksgXxZ--AQ@mail.gmail.com> I am the original dev for pypi.python.org/pypi/pythonpy, which offers much of the same functionality as python -c, but in a more convenient form. I'd be really interested in figuring out whether the core python tools could support many of the use cases that pythonpy covers, so that users wouldn't have to install it as a separate tool. I don't have a concrete idea of what it would look like, but many of my users have said they would much prefer an officially supported tool. Perhaps someone on this mailing list can provide some insight into the available options. Best, Russell -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/df27bb7d/attachment.html> From abarnert at yahoo.com Thu Jan 8 22:30:15 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 8 Jan 2015 13:30:15 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <CAPJVwB=HXy6P7tGV_wiPo1P2Fsz1jEP5ePcECw_4LMSLQu-3hA@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <BF30F199-DFDA-4003-B160-6CFFBD0DD14D@yahoo.com> <CAFpSVpKxUbY+azZRPY+=vo3Ht8RuchruUh1AN56EzKCjcwTiiQ@mail.gmail.com> <CAPJVwB=HXy6P7tGV_wiPo1P2Fsz1jEP5ePcECw_4LMSLQu-3hA@mail.gmail.com> Message-ID: <08B511D5-9594-4C02-A1CA-BFC44F26E306@yahoo.com> On Jan 8, 2015, at 10:40, Nathaniel Smith <njs at pobox.com> wrote: > On Thu, Jan 8, 2015 at 5:58 PM, Todd <toddrjen at gmail.com> wrote: >> On Thu, Jan 8, 2015 at 6:41 PM, Andrew Barnert <abarnert at yahoo.com> wrote: >>> >>> More generally: if an frange that works exactly like range but with floats >>> (and rounding errors) is all you want, that's so trivial to write that it >>> doesn't seem worth putting in the stdlib, unless the use case is very >>> common. >> >> In numerical computing the use-case is extremely common, in fact I would say >> it practically universal. Ideally it would be vectorized, but in practice >> this is often not possible, requiring the use of loops. > > Can you give a more specific example? I believe your problem is real, > but as a numerical programmer I can't think of an example of where > I've ever wanted anything like this, so it's hard for me to visualize > what exactly you're talking about. > > And is what you want just: > > class FRange: > def __init__(start, stop, step): > self.start = start > self.stop = stop > self.step = step > > def __getitem__(self, i): > x = self.start + i * self.step > if x > self.stop: > raise IndexError > def __iter__(self): > i = 0 > while True: > try: > yield self[i] > except IndexError: > return > i += 1 This isn't really necessary; any type whose __getitem__ takes int values from 0 up to some point and then raises IndexError beyond that is automatically iterable (and the built-in sequence iterator is basically a C-optimized version of what you just wrote). While we're at it, you probably also want __len__, __contains__ (one more place for naive float rounding problems?), __reversed__, index, count, a __getitem__ that handles slices and negative values, and __hash__. All pretty simple to add. (In fact, if you inherit collections.abc.Sequence you'll get everything but __getitem__, __len__, and __hash__ for free, but you'd probably want to implement __contains__, index, and count anyway, because the default versions will be linear instead of constant.) From abarnert at yahoo.com Thu Jan 8 22:33:23 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 8 Jan 2015 13:33:23 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> Message-ID: <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> On Jan 8, 2015, at 11:05, Chris Barker <chris.barker at noaa.gov> wrote: >> I would think that a floating range class would necessarily use multiplication rather than repeated addition (to allow indexing at arbitrary point), which would avoid cumulative floating-point errors (although it would still have a smaller floating point error at the end), and for the same reason the final value would have to be pre-computed rather than using a naive ">=" which would allow it to be a bit smarter. > > That's the trick -- range() (and arange) does not specify a final value, it specifies a final value not to include. This is well defined and easy to understand for integers, but not so for floating point. But you are right about the multiplication and pre-computing of final value -- that's a good reason to provide this as a built-in -- it's very easy to implement, but even easier to implement badly. Implementing it badly isn't much of a problem; even if you don't precompute the stop, you're just adding the cost of a multiplication to every __getitem__, and I doubt that's going to be a bottleneck in most applications. It's _using_ it badly that's the more serious problem. Just as with np.arange, a naive use of frange will lead to rounding problems, and I can't see how a builtin could provide any API that avoids that problem? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/cfe43b1a/attachment.html> From warren.weckesser at gmail.com Thu Jan 8 23:14:24 2015 From: warren.weckesser at gmail.com (Warren Weckesser) Date: Thu, 8 Jan 2015 17:14:24 -0500 Subject: [Python-ideas] Float range class In-Reply-To: <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> Message-ID: <CAGzF1ueVK7LJyQdD-YB8tP691Nfuax8w_mCGh_qsL_q_rSBy-A@mail.gmail.com> On Thu, Jan 8, 2015 at 2:05 PM, Chris Barker <chris.barker at noaa.gov> wrote: > I would think that a floating range class would necessarily use >> multiplication rather than repeated addition (to allow indexing at >> arbitrary point), which would avoid cumulative floating-point errors >> (although it would still have a smaller floating point error at the end), >> and for the same reason the final value would have to be pre-computed >> rather than using a naive ">=" which would allow it to be a bit smarter. >> > > That's the trick -- range() (and arange) does not specify a final value, > it specifies a final value not to include. This is well defined and easy to > understand for integers, but not so for floating point. But you are right > about the multiplication and pre-computing of final value -- that's a good > reason to provide this as a built-in -- it's very easy to implement, but > even easier to implement badly. > The implementation based on `start + i*step` doesn't solve the problem of "unexpected" off-by-one sequences ("unexpected", that is, for those not familiar with the vagaries of floating point calculations). Suppose frange uses the "half open" convention, and consider frange(start=0.3, stop=0.9, step=0.3) With perfect math, it should be [0.3, 0.6]. But check: >>> start = 0.3 >>> stop = 0.9 >>> step = 0.3 >>> [start + i*step for i in range(4) if start + i*step < stop] [0.3, 0.6, 0.8999999999999999] Now suppose frange uses the "closed" convention, and consider frange(start=0.4, stop=1.2, step=0.4) Here we naively expect [0.4, 0.8, 1.2]. Take a look at what we get: >>> start = 0.4 >>> stop = 1.2 >>> step = 0.4 >>> [start + i*step for i in range(4) if start + i*step <= stop] [0.4, 0.8] That's because the third value is actually 1.2000000000000002: >>> [start + k*step for k in range(3)] [0.4, 0.8, 1.2000000000000002] I've fixed at least two bugs in scipy because of exactly this type of naive use of numpy's arange function. The fix is to use np.linspace, as Chris explained. These days, I only use np.arange with integers (or integral floating point values); if I want a uniformally spaced sequence of floats with a non-integer step, I use np.linspace. If I don't want the last point, I use the `endpoint=False` argument in `np.linspace` (e.g. `x = np.linspace(0, 1, num=4, endpoint=False)` generates [0, 0.25, 0.5, 0.75]. Warren (Chris, sorry for the duplicate email. Once again I forgot to "reply all".) > But, at least in my own experience, I use arange when I want an >>> interval-based range, and linspace when I want a count-based range. >>> >> > I would argue (and do!) that you should not do this -- if you know what > you are doing with FP, then fine, but it really is tricky. You would be > better off computing the count you want then then using linspace anyway. I > suppose an interval-based API to something like linspace would be a nice > convenience, though. > > I haven't managed to come up with a quick an easy example where this > matters, but they DO happen. > > I guess I'm arguing that a range-like object for FP should be a closed > rather than open interval -- specifying the starting and end points. That > is because defining an open interval where numbers are of finite, but hard > to know know in advance the interval, is just too ugly and complex. > > I see your point that sometimes you want a specific delta, and sometimes > you want a specific end point, but I supect that most of the time you want > a specific delta you ALSO want a specific and point and/or want to know how > many values you are going to get. > > In fact, in the most common use of integer range, you really are defining > the number of values you want. > > And note that the range convention of starting at zero and not including > the stop value was designed to match python indexing convention, i.e.: > > for i in range( len(sequence) ): > ... > > is natural an easy to write, and does what's expected. And also: > > for i in range(n): > .... > > will loop n times. > > but neither of these apply to floating point ranges. > > -Chris > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/7db3a4e3/attachment.html> From chris.barker at noaa.gov Thu Jan 8 23:35:46 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 8 Jan 2015 14:35:46 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> Message-ID: <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> On Thu, Jan 8, 2015 at 1:33 PM, Andrew Barnert <abarnert at yahoo.com> wrote: > Implementing it badly isn't much of a problem; even if you don't > precompute the stop, you're just adding the cost of a multiplication to > every __getitem__, and I doubt that's going to be a bottleneck in most > applications. > I wasn't talking about performance, I was talking about the vagaries of floating point. for example, an obvious (easy, anyway) thing to do is somethign like: val = start while val < stop: val += step but that's not a great idea with fp. > It's _using_ it badly that's the more serious problem. Just as with > np.arange, a naive use of frange will lead to rounding problems, and I > can't see how a builtin could provide any API that avoids that problem? > which is why I'm suggesting that a built in be more like numpy's linspace than like range. See Warren's note for a better explanation with examples. Another of my points is that while ranges of floating point numbers are very commonly used, the use cases are generally quite different than for ranges of integers, so we may not want the same API, floating point issues aside. The most common cases for range are: a) Give me n numbers b) Loop n times c) Give me all the indexes for a sequence of length n d) Give me all the numbers from a to b - then you need to think about whether you are including b or not, and add 1 if you want it. e) Give me every j-th number from a to b. - then you need to think even more about the end point (if you care) Note that you need to think about the last value in (d) and (e), but that's pretty easy to do, and not prone to error with integers. But for floating point range-like things: a -- c are not relevant. d) is more likely to be spelled as: Give me n numbers evenly spaced from a to b -- that's np.linspace() e) is essentially: give me all the numbers form a to be, using this delta. - this seems like a floating point range, but again, you need to think carefully about the end points, so you're really better off with llnspace, and then once the end points have been defined, you can write FP code that will do the expected thing. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/390611e7/attachment-0001.html> From abarnert at yahoo.com Thu Jan 8 23:35:06 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 8 Jan 2015 22:35:06 +0000 (UTC) Subject: [Python-ideas] Better comand line version of python -c In-Reply-To: <CAATkbzBPf_u3CqYMtpSUoEJFV56ouqAa5UGAUNK2ksgXxZ--AQ@mail.gmail.com> References: <CAATkbzBPf_u3CqYMtpSUoEJFV56ouqAa5UGAUNK2ksgXxZ--AQ@mail.gmail.com> Message-ID: <1272917759.3995303.1420756506710.JavaMail.yahoo@jws100140.mail.ne1.yahoo.com> On Thursday, January 8, 2015 1:11 PM, Russell Stewart <Russell.S.Stewart at gmail.com> wrote: >I am the original dev for pypi.python.org/pypi/pythonpy, which offers much of the same functionality as python -c, but in a more convenient form. I'd be really interested in figuring out whether the core python tools could support many of the use cases that pythonpy covers, so that users wouldn't have to install it as a separate tool. I don't have a concrete idea of what it would look like, but many of my users have said they would much prefer an officially supported tool. Perhaps someone on this mailing list can provide some insight into the available options. This looks pretty cool to me?so I have lots of questions and comments. :) The obvious problem is what to call it. `py` (well, `py.exe`) is already the name of the PEP 397 launcher tool (the thing that lets you use shbang lines on Windows), so this would need a new name. Maybe it would be better as just a different flag or set of flags on `python` itself, like `--pprint`/`-p`? Of course then there's the problem of where to stick the flags for `-p` (for GNU-style long names this is pretty easy: `--print=filter,lines`, but obviously for something meant to be used for quick&dirty command-line usage you need short names as well?). And the "backport" to 3.4 (and 2.x) would act differently from the standard 3.5+ version. But it might be worth looking at anyway. Or, of course, just make this a stdlib module, so it's just `python [PYTHON OPTIONS] -m YOUR_MODULENAME [YOUR MODULE OPTIONS]`, and you can alias that to whatever you want. Is `-fx` a multi-character single-hyphen flag (in which case it's very weird to mix those with GNU-style long arguments in the same program, or is it a combination of `-f` and `-x`? And is the filter expression an argument to `-fx`, or does `-f` just change the interpretation of the argument? Speaking of GNU-style long arguments, why `--i` instead of just `-i` (as a short name for `--ignore_exceptions` or something) The documentation is pretty Python 2-specific: the majority of the examples use the `range` function and depend on it returning a list. You seem to have built a non-trivial custom pretty-printer for this tool (e.g., to print lists row by row) as well as an auto-importer (e.g., to use `collections.Counter` without an `import collections`); maybe some or all of that should be separated out and made available to Python code somewhere in the stdlib, and then the script (or flag) could just use that function? This seems to be pretty strongly list-based for no good reason. Why not print _any_ iterable line by line, make -l set `l = sys.stdin` instead of `l = list(sys.stdin)` (I realize that doesn't work for the specific example of `py -l 'l[::-1]'`, but `py -l reversed(l)` would work just as well in that case, and it's hard to think of other examples where there would even be a problem), etc.? It might be nice to have an option to see the `repr` instead of the `str` of everything, to match what you'd see from the interactive interpreter. While --si is cool, people already misuse `str.split` and `re.split` to try to parse CSV and similar input that has quotes or escapes; it might be nice to have a `--csv` mode to parse the input with `csv.reader` (and that could still take an optional delimiter, of course; passing other dialect flags to `reader` might be out of scope). A columnar reading mode might also be nice, since that's one of those things that's novices have a hard time writing without statements. An option for pretty JSON output instead of compact JSON output might be nice. And maybe an option to read multi-line JSON input?not just a single JSON value (which is trivial with `json.loads(l)`, but, since JSON is self-delimiting, still a stream of them, without the requirement of one/line (e.g., by looping over `json.JSONDecoder.raw_decode`). I assume you're evaluating the expression with `eval`. Are you passing it a custom locals and globals (so any internals of your script itself aren't available), or not? I could see disadvantages of both (the former may prevent some useful quick&dirty hacks; the latter could raise safety concerns), but either way, it's probably worth documenting. >From the summary help, it looks like you can apply both `-x` and `-fx`. If you do that, do they run in a specific order, or in the order specified? For that matter, can you apply multiple mappings and/or filterings? (If not, it seems like that could be handy.) It might be useful to have some kind of "daemon mode", where you start up an interpreter in the background and then pipe input to it. Then you can pipe multiple things to the same interpreter session, both for persistence, and to avoid the process startup cost on platforms where it matters (like Windows). Maybe with an optional timeout, where the daemon stops a few minutes after last use, so you don't have to remember to `py -d sys.exit(0)` or similar. And maybe the magic `_` should work in daemon mode, as it does in the interactive interpreter. (I could see writing some one-liner that takes 90 seconds to run, then realizing I wanted it in JSON format, so `py --daemon -jo _` would be handy to avoid re-running everything?) It would be really cool if, when run under PowerShell, this could handle scriptlet input and output instead of plain text (so, e.g., you pass it an array of strings and it processes each string). Being able to somehow pass command-line arguments through to `python` itself (assuming this isn't merged into the main interpreter, of course) might be handy. In particular, I could see `-u` being useful, but there might be others (including platform/implementation-specific `-X` options). Of course that would come for free with the `-m` interface, but since I'm not sure that's the best option... Instead of just being able to ignore exceptions, it might be nice to enable one-line exception output (e.g., print just the type and message, no traceback, and to stdout instead of stderr). With `-x`, an option to prefix each line of output with the corresponding line of input could be handy, similar to the `-v` option to commands like `cp`, `tar`, etc. A way to feed input files in could be handy to allow the kind of one-liners people often fall back to perl for, although I'm not sure exactly what you'd want this to look like. Some of these are probably way out of scope even for a third-party tool, much less for a built-in stdlib version, but I'm not sure exactly which, so I've just dumped everything on you to let you sort them out. :) From Russell.S.Stewart at gmail.com Fri Jan 9 00:14:42 2015 From: Russell.S.Stewart at gmail.com (Russell Stewart) Date: Thu, 8 Jan 2015 15:14:42 -0800 Subject: [Python-ideas] Better comand line version of python -c In-Reply-To: <1272917759.3995303.1420756506710.JavaMail.yahoo@jws100140.mail.ne1.yahoo.com> References: <CAATkbzBPf_u3CqYMtpSUoEJFV56ouqAa5UGAUNK2ksgXxZ--AQ@mail.gmail.com> <1272917759.3995303.1420756506710.JavaMail.yahoo@jws100140.mail.ne1.yahoo.com> Message-ID: <CAATkbzAD96W-suKCu1-pDrJdTF_qOefXPUPXTH8+optLyUXf+w@mail.gmail.com> Andrew, You've brought up many design points that I might have done differently if I were to start over. Things got locked in after the first few hundred users, but bringing things under the primary python command would free up a lot more room without backwards compatibility. Heres some thoughts on a few things you said, assuming a brand new version without backwards compatibility would be the way forward: Json handling: I think I would remove this if I did it again. I originally found it quite useful, but the fact that it doesn't support multi-line json inputs makes it weird. Also, all the functionality can be accessed through the json module with python (e.g. py -l 'json.loads(l)'). Split input/output: This one is actually super useful for parsing lines of text that you get from other unix commands (e.g. py --si 'special: ' -x x[1][:4] will grab the first 4 characters after the word special in any input file). It's also much easier to provide complete access to what people want to do with the single delimeter flag, so its much better than the --ji --jo flags in that respect. -csv option: I considered this at length, and originally the --si was motivated by commas. But ultimately --si is good for lots of things other than commas. A separate csv command seems nice, until you realize that there are too many options to specify in parameters, just like in the json case. It's better to let people parse csv docs with py -l 'csv.Reader(l) ... '. Windows py.exe issue: Yea, I'm aware of this. I have a pythonwpy for windows that is run with wpy, but no one seems to care about comand line tools on windows. py vs python -m pythonpy ...: Yes, using this with python -m is a very real option. People could simply alias py='python -m pythonpy' to avoid typing issues. This is probably the most straightforward distribution option. It would be great to keep the ipython style tab completion, but I'm not sure that its necessary. py -fx flag: Originally, I didn't want to name this -f because I didn't want the active variable x to be magically introduced. When you type -fx, its clear where the variable x comes from. This was probably a mistake, because everyone hates 2 character single-dash flags. Should be changed in the case of an overhaul. This command is also only marginally useful, due to the fact that grep already handles this use case so well, and because you can get the functionality with py -x 'x if foo(x) else None' just as well. pythonpy should be Iterable based, not list-based: I agree somewhat. The current dictionary functionality is weird, only printing the list of keys. Again, I don't want to change this for backwards compat reasons. This could be done a lot of different ways, and I think the current strategy is pretty good, but leaves a little bit to be desired in terms of simplicity. The -x and -l flags capture a lot of the list-printing that you would want to do, so its not clear you need to do list-printing with no arguments. Using a separate flag for list printing would be a good design consideration here. Keep in mind that the current motivation for all the list strangeness is so that pythonpy will play well with unix pipes, which it does. That is really important for some core usecases. Daemon mode: I hadn't thought of this one before. Maybe it could be done well, but daemon modes tend incur a high cost of mental overhead on the user. I've worked hard to keep the startup time within a factor of 2 from the python interpreter, so I do care a lot about that. --i flag: Again, the naming here is up for change. It's useful in a few cases, where python throws exceptions easily (e.g. py -x re.match(r"b", x).group(0)). But I haven't used it all that much, and it really could be removed. So in general I like the python -m idea. Support for tab completion under this would be really nice. I've long thought that python -c and python -m should have ipython style tab completion anyways, so that one could type python -m Sim<tab> and get python -m SimpleHTTPServer. Another idea would be to try and get a shorter binary name distributed with python. It sounds superficial, but is ultimately quite important if python wants to compete with ruby, perl, and sed. Best, Russell On Thu, Jan 8, 2015 at 2:35 PM, Andrew Barnert <abarnert at yahoo.com> wrote: > On Thursday, January 8, 2015 1:11 PM, Russell Stewart < > Russell.S.Stewart at gmail.com> wrote: > > > >I am the original dev for pypi.python.org/pypi/pythonpy, which offers > much of the same functionality as python -c, but in a more convenient form. > I'd be really interested in figuring out whether the core python tools > could support many of the use cases that pythonpy covers, so that users > wouldn't have to install it as a separate tool. I don't have a concrete > idea of what it would look like, but many of my users have said they would > much prefer an officially supported tool. Perhaps someone on this mailing > list can provide some insight into the available options. > > This looks pretty cool to me?so I have lots of questions and comments. :) > > The obvious problem is what to call it. `py` (well, `py.exe`) is already > the name of the PEP 397 launcher tool (the thing that lets you use shbang > lines on Windows), so this would need a new name. > > Maybe it would be better as just a different flag or set of flags on > `python` itself, like `--pprint`/`-p`? Of course then there's the problem > of where to stick the flags for `-p` (for GNU-style long names this is > pretty easy: `--print=filter,lines`, but obviously for something meant to > be used for quick&dirty command-line usage you need short names as well?). > And the "backport" to 3.4 (and 2.x) would act differently from the standard > 3.5+ version. But it might be worth looking at anyway. > > > Or, of course, just make this a stdlib module, so it's just `python > [PYTHON OPTIONS] -m YOUR_MODULENAME [YOUR MODULE OPTIONS]`, and you can > alias that to whatever you want. > > Is `-fx` a multi-character single-hyphen flag (in which case it's very > weird to mix those with GNU-style long arguments in the same program, or is > it a combination of `-f` and `-x`? And is the filter expression an argument > to `-fx`, or does `-f` just change the interpretation of the argument? > > > Speaking of GNU-style long arguments, why `--i` instead of just `-i` (as a > short name for `--ignore_exceptions` or something) > > The documentation is pretty Python 2-specific: the majority of the > examples use the `range` function and depend on it returning a list. > > > You seem to have built a non-trivial custom pretty-printer for this tool > (e.g., to print lists row by row) as well as an auto-importer (e.g., to use > `collections.Counter` without an `import collections`); maybe some or all > of that should be separated out and made available to Python code somewhere > in the stdlib, and then the script (or flag) could just use that function? > > This seems to be pretty strongly list-based for no good reason. Why not > print _any_ iterable line by line, make -l set `l = sys.stdin` instead of > `l = list(sys.stdin)` (I realize that doesn't work for the specific example > of `py -l 'l[::-1]'`, but `py -l reversed(l)` would work just as well in > that case, and it's hard to think of other examples where there would even > be a problem), etc.? > > It might be nice to have an option to see the `repr` instead of the `str` > of everything, to match what you'd see from the interactive interpreter. > > While --si is cool, people already misuse `str.split` and `re.split` to > try to parse CSV and similar input that has quotes or escapes; it might be > nice to have a `--csv` mode to parse the input with `csv.reader` (and that > could still take an optional delimiter, of course; passing other dialect > flags to `reader` might be out of scope). > > A columnar reading mode might also be nice, since that's one of those > things that's novices have a hard time writing without statements. > > An option for pretty JSON output instead of compact JSON output might be > nice. > > And maybe an option to read multi-line JSON input?not just a single JSON > value (which is trivial with `json.loads(l)`, but, since JSON is > self-delimiting, still a stream of them, without the requirement of > one/line (e.g., by looping over `json.JSONDecoder.raw_decode`). > > I assume you're evaluating the expression with `eval`. Are you passing it > a custom locals and globals (so any internals of your script itself aren't > available), or not? I could see disadvantages of both (the former may > prevent some useful quick&dirty hacks; the latter could raise safety > concerns), but either way, it's probably worth documenting. > > From the summary help, it looks like you can apply both `-x` and `-fx`. If > you do that, do they run in a specific order, or in the order specified? > For that matter, can you apply multiple mappings and/or filterings? (If > not, it seems like that could be handy.) > > It might be useful to have some kind of "daemon mode", where you start up > an interpreter in the background and then pipe input to it. Then you can > pipe multiple things to the same interpreter session, both for persistence, > and to avoid the process startup cost on platforms where it matters (like > Windows). Maybe with an optional timeout, where the daemon stops a few > minutes after last use, so you don't have to remember to `py -d > sys.exit(0)` or similar. And maybe the magic `_` should work in daemon > mode, as it does in the interactive interpreter. (I could see writing some > one-liner that takes 90 seconds to run, then realizing I wanted it in JSON > format, so `py --daemon -jo _` would be handy to avoid re-running > everything?) > > It would be really cool if, when run under PowerShell, this could handle > scriptlet input and output instead of plain text (so, e.g., you pass it an > array of strings and it processes each string). > > Being able to somehow pass command-line arguments through to `python` > itself (assuming this isn't merged into the main interpreter, of course) > might be handy. In particular, I could see `-u` being useful, but there > might be others (including platform/implementation-specific `-X` options). > Of course that would come for free with the `-m` interface, but since I'm > not sure that's the best option... > > Instead of just being able to ignore exceptions, it might be nice to > enable one-line exception output (e.g., print just the type and message, no > traceback, and to stdout instead of stderr). > > With `-x`, an option to prefix each line of output with the corresponding > line of input could be handy, similar to the `-v` option to commands like > `cp`, `tar`, etc. > > A way to feed input files in could be handy to allow the kind of > one-liners people often fall back to perl for, although I'm not sure > exactly what you'd want this to look like. > > Some of these are probably way out of scope even for a third-party tool, > much less for a built-in stdlib version, but I'm not sure exactly which, so > I've just dumped everything on you to let you sort them out. :) > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/ef14fc78/attachment-0001.html> From edk141 at gmail.com Fri Jan 9 00:17:01 2015 From: edk141 at gmail.com (Ed Kellett) Date: Thu, 08 Jan 2015 23:17:01 +0000 Subject: [Python-ideas] Better comand line version of python -c References: <CAATkbzBPf_u3CqYMtpSUoEJFV56ouqAa5UGAUNK2ksgXxZ--AQ@mail.gmail.com> <1272917759.3995303.1420756506710.JavaMail.yahoo@jws100140.mail.ne1.yahoo.com> Message-ID: <CABmzr0jQYofk=oGq0UWfZPe5xDdGJ7=rqqBASzWb=jitnpa1ZA@mail.gmail.com> IMV, pythonpy relies too much on specific CLI options, rather than making the Python ways to do those things more accessible. I wrote a tool[1] that attempts to solve that problem, although it ended up being heavily geared towards "take a stream of things and process each of them"-type tasks. Things it does that might be of interest here: - handles exceptions pretty well, I think - has a very small set of primitive operations (so there's less stuff to remember about the tool itself) - runs code in a custom environment, so it's isolated from my own machinery. There is a tiny bit of API for controlling data flow; this is accessed using auto-imports just like any other API. Having played extensively with both, I think I'd like a standard tool to adopt a philosophy more like mine with regard to the supported feature set. Not just because that's how I did it; I feel like special support for e.g. mutliline json input hurts use cases none of the developers thought of yet. That said, it's possible my thing ended up a bit too ideological. Ed Kellett 1. https://github.com/edk0/spy -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/8b535a71/attachment.html> From rosuav at gmail.com Fri Jan 9 00:39:35 2015 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 9 Jan 2015 10:39:35 +1100 Subject: [Python-ideas] Better comand line version of python -c In-Reply-To: <CAATkbzAD96W-suKCu1-pDrJdTF_qOefXPUPXTH8+optLyUXf+w@mail.gmail.com> References: <CAATkbzBPf_u3CqYMtpSUoEJFV56ouqAa5UGAUNK2ksgXxZ--AQ@mail.gmail.com> <1272917759.3995303.1420756506710.JavaMail.yahoo@jws100140.mail.ne1.yahoo.com> <CAATkbzAD96W-suKCu1-pDrJdTF_qOefXPUPXTH8+optLyUXf+w@mail.gmail.com> Message-ID: <CAPTjJmpMmzM1wnheEqRUZD-O_Psq_VJ1OdK31PRLcmFzrXVzCg@mail.gmail.com> On Fri, Jan 9, 2015 at 10:14 AM, Russell Stewart <Russell.S.Stewart at gmail.com> wrote: > I've long thought that python -c and python -m should have ipython style tab > completion anyways, so that one could type python -m Sim<tab> and get python > -m SimpleHTTPServer. That ought to be possible, but it would be a bash (and other) feature, rather than a Python one. Recent versions of git come with a fairly comprehensive tab-completion setup, written for multiple shells, and it should be possible to do something a lot simpler for Python. Of course, the key here is knowing every possible completion for "python -m <blah>", without taking three parts of forever to build the list... but that's largely in the hands of individual sysadmins. It'd definitely be handy. ChrisA From jeanpierreda at gmail.com Fri Jan 9 00:53:21 2015 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Thu, 8 Jan 2015 15:53:21 -0800 Subject: [Python-ideas] Better comand line version of python -c In-Reply-To: <CAATkbzAD96W-suKCu1-pDrJdTF_qOefXPUPXTH8+optLyUXf+w@mail.gmail.com> References: <CAATkbzBPf_u3CqYMtpSUoEJFV56ouqAa5UGAUNK2ksgXxZ--AQ@mail.gmail.com> <1272917759.3995303.1420756506710.JavaMail.yahoo@jws100140.mail.ne1.yahoo.com> <CAATkbzAD96W-suKCu1-pDrJdTF_qOefXPUPXTH8+optLyUXf+w@mail.gmail.com> Message-ID: <CABicbJKAyYEwk9Df8p0+5XD3ADkuupDjHc8bJftH3yiVgquRjg@mail.gmail.com> On Thu, Jan 8, 2015 at 3:14 PM, Russell Stewart <Russell.S.Stewart at gmail.com> wrote: > Originally, I didn't want to name this -f because I didn't want the active > variable x to be magically introduced. When you type -fx, its clear where > the variable x comes from. This was probably a mistake, because everyone > hates 2 character single-dash flags. Should be changed in the case of an > overhaul. This command is also only marginally useful, due to the fact that > grep already handles this use case so well, and because you can get the > functionality with py -x 'x if foo(x) else None' just as well. You can make -f a flag that takes one mandatory argument. This lets people do -fy too. -- Devin From Russell.S.Stewart at gmail.com Fri Jan 9 00:59:40 2015 From: Russell.S.Stewart at gmail.com (Russell Stewart) Date: Thu, 8 Jan 2015 15:59:40 -0800 Subject: [Python-ideas] Better comand line version of python -c In-Reply-To: <CABmzr0jQYofk=oGq0UWfZPe5xDdGJ7=rqqBASzWb=jitnpa1ZA@mail.gmail.com> References: <CAATkbzBPf_u3CqYMtpSUoEJFV56ouqAa5UGAUNK2ksgXxZ--AQ@mail.gmail.com> <1272917759.3995303.1420756506710.JavaMail.yahoo@jws100140.mail.ne1.yahoo.com> <CABmzr0jQYofk=oGq0UWfZPe5xDdGJ7=rqqBASzWb=jitnpa1ZA@mail.gmail.com> Message-ID: <CAATkbzDcM5_42cszJDUzKL8eqhgM3-f4hPRvDVoUSWz53_NwJg@mail.gmail.com> Interesting tool Ed. I think we both agree that fewer options is better. I'd actually advocate for a tool with just the (-x, -l, --si, and --so options). I marginally prefer "py -x x" to "spy pipe", but that's a moot point. Running things in a separated environment is definitely the way to go. This is especially true when you are doing auto-importing or tab completion. I would run pythonpy with python -E if only I could figure out how to force pip to install it that way. An officially supported tool wouldn't have these problems though. Exception handling/printing is an obvious area for improvement with pythonpy. As I said above, I don't think the --i command panned out very well, and it isn't necessary. Likewise, it's not clear that your --no-exception-handling flag is necessary either. Also, I'm not sure why you have multiple arguments for different pipes, when you could do that just as well with spy '3' | spy 'pipe * 2'. Simple tools that are composable is the unix way. Re Chris: You should check out the pythonpy tab completion if you are curious. It already does tab completion for SimpleHTTPServer when you type py Sim<tab>. It handles this with a custom script for bash, but there is no reason that python couldn't have a custom bash script, just as git does. On Thu, Jan 8, 2015 at 3:17 PM, Ed Kellett <edk141 at gmail.com> wrote: > IMV, pythonpy relies too much on specific CLI options, rather than making > the Python ways to do those things more accessible. > > I wrote a tool[1] that attempts to solve that problem, although it ended > up being heavily geared towards "take a stream of things and process each > of them"-type tasks. > > Things it does that might be of interest here: > - handles exceptions pretty well, I think > - has a very small set of primitive operations (so there's less stuff to > remember about the tool itself) > - runs code in a custom environment, so it's isolated from my own > machinery. There is a tiny bit of API for controlling data flow; this is > accessed using auto-imports just like any other API. > > Having played extensively with both, I think I'd like a standard tool to > adopt a philosophy more like mine with regard to the supported feature set. > Not just because that's how I did it; I feel like special support for e.g. > mutliline json input hurts use cases none of the developers thought of yet. > > That said, it's possible my thing ended up a bit too ideological. > > Ed Kellett > > 1. https://github.com/edk0/spy > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/4284acaa/attachment.html> From edk141 at gmail.com Fri Jan 9 01:13:13 2015 From: edk141 at gmail.com (Ed Kellett) Date: Fri, 09 Jan 2015 00:13:13 +0000 Subject: [Python-ideas] Better comand line version of python -c References: <CAATkbzBPf_u3CqYMtpSUoEJFV56ouqAa5UGAUNK2ksgXxZ--AQ@mail.gmail.com> <1272917759.3995303.1420756506710.JavaMail.yahoo@jws100140.mail.ne1.yahoo.com> <CABmzr0jQYofk=oGq0UWfZPe5xDdGJ7=rqqBASzWb=jitnpa1ZA@mail.gmail.com> <CAATkbzDcM5_42cszJDUzKL8eqhgM3-f4hPRvDVoUSWz53_NwJg@mail.gmail.com> Message-ID: <CABmzr0iM8UJN9eOY74uOmLrS8=eu+uvmkXB8m-iOwyQv2mR5tw@mail.gmail.com> On Fri Jan 09 2015 at 12:00:20 AM Russell Stewart < Russell.S.Stewart at gmail.com> wrote: > > Running things in a separated environment is definitely the way to go. > This is especially true when you are doing auto-importing or tab > completion. I would run pythonpy with python -E if only I could figure out > how to force pip to install it that way. > Oops... 'environment' was my shorter way of saying global and local dictionaries. > Exception handling/printing is an obvious area for improvement with > pythonpy. As I said above, I don't think the --i command panned out very > well, and it isn't necessary. Likewise, it's not clear that > your --no-exception-handling flag is necessary either. > It's for debugging spy itself (I think the documentation says something to that effect), because the exception handling removes frames that belong to spy from the traceback. I wouldn't include this in a standard tool, and I'll probably remove it from spy when I decide to commit to a not-development version. > Also, I'm not sure why you have multiple arguments for different pipes, > when you could do that just as well with spy '3' | spy 'pipe * 2'. Simple > tools that are composable is the unix way. > Because passing Python objects to the next things is often helpful, especially given the lack of pre- or post-command or built-in json processing (you're expected to achieve the same end by piping your objects). In fact, generalizing the three things above was how the piping idea came about in the first place. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/6c497f57/attachment-0001.html> From mistersheik at gmail.com Fri Jan 9 04:01:55 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 8 Jan 2015 19:01:55 -0800 (PST) Subject: [Python-ideas] Float range class In-Reply-To: <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> Message-ID: <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> I agree with everyone above. At first I was +1 on this proposal, but why not suggest to the numpy people that arange and linespace should return Sequence objects rather than numpy arrays? I'm pretty sure there's a way to make sure objects work in numpy. Also, didn't realize you could do this: In [9]: np.linspace(0, 10, 4, dtype=np.int) Out[9]: array([ 0, 3, 6, 10]) Kind of cool to have that for the integers. Too bad it's an array and not a Sequence object. Best, Neil On Thursday, January 8, 2015 at 5:37:09 PM UTC-5, Chris Barker wrote: > > On Thu, Jan 8, 2015 at 1:33 PM, Andrew Barnert <abar... at yahoo.com > <javascript:>> wrote: > > >> Implementing it badly isn't much of a problem; even if you don't >> precompute the stop, you're just adding the cost of a multiplication to >> every __getitem__, and I doubt that's going to be a bottleneck in most >> applications. >> > > I wasn't talking about performance, I was talking about the vagaries of > floating point. for example, an obvious (easy, anyway) thing to do is > somethign like: > > val = start > while val < stop: > val += step > > but that's not a great idea with fp. > > >> It's _using_ it badly that's the more serious problem. Just as with >> np.arange, a naive use of frange will lead to rounding problems, and I >> can't see how a builtin could provide any API that avoids that problem? >> > > which is why I'm suggesting that a built in be more like numpy's linspace > than like range. See Warren's note for a better explanation with examples. > > Another of my points is that while ranges of floating point numbers are > very commonly used, the use cases are generally quite different than for > ranges of integers, so we may not want the same API, floating point issues > aside. The most common cases for range are: > > a) Give me n numbers > > b) Loop n times > > c) Give me all the indexes for a sequence of length n > > d) Give me all the numbers from a to b > - then you need to think about whether you are including b or not, and > add 1 if you want it. > > e) Give me every j-th number from a to b. > - then you need to think even more about the end point (if you care) > > Note that you need to think about the last value in (d) and (e), but > that's pretty easy to do, and not prone to error with integers. > > But for floating point range-like things: > > a -- c are not relevant. > > d) is more likely to be spelled as: Give me n numbers evenly spaced from a > to b -- that's np.linspace() > > e) is essentially: give me all the numbers form a to be, using this delta. > - this seems like a floating point range, but again, you need to think > carefully about the end points, so you're really better off with llnspace, > and then once the end points have been defined, you can write FP code that > will do the expected thing. > > -Chris > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.... at noaa.gov <javascript:> > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/09a2048f/attachment.html> From mistersheik at gmail.com Fri Jan 9 04:08:34 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 8 Jan 2015 19:08:34 -0800 (PST) Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAP7h-xZFd3DGvhr0D6D3rzJBm5BgZ3yX9jaOCQaFGPM36GD74g@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <CAG3cHaaXAXs87ZK=wk1q191HR2tt0EVdZTFSgo=2xLO_feujbw@mail.gmail.com> <CAFpSVp+1ee8mM_xbPkANeNjWJGRr8zAd=SFe+VC5EO-2L4P2Dg@mail.gmail.com> <54AD4F71.7060008@oa-cagliari.inaf.it> <CAP7h-xZFd3DGvhr0D6D3rzJBm5BgZ3yX9jaOCQaFGPM36GD74g@mail.gmail.com> Message-ID: <e47dc683-8d85-4b18-a2c3-cba5ef09b4b4@googlegroups.com> Also +1. I sometimes import numpy just for this because I find float("inf") so ugly. Best, Neil On Wednesday, January 7, 2015 at 11:08:23 AM UTC-5, Alexander Belopolsky wrote: > > > > On Wed, Jan 7, 2015 at 10:23 AM, Marco Buttu <marco... at gmail.com > <javascript:>> wrote: > >> On 07/01/2015 15:58, Todd wrote: >> >> Yes, this would be a shortcut for that. math.nan and math.inf are less >>> verbose than float("nan") and float("inf"), even if you aren't already >>> using the math module (fewer characters total, and in particular fewer >>> special characters). It would also, at least in my opinion, be more >>> readable. Plus there is already the math.isnan and math.isinf, so I don't >>> think it would be totally out-of-place. >>> >> >> I totally agree. +1 > > > Another +1. > > Two points: > > 1. The current ways to create inf and nan are not obvious and hard to > discover. It will be nice to have these constants discoverable in tab > completion from math. > > 2. The repr()s of nan and inf are simply 'nan' and 'inf', not > 'float("nan")' and 'float("inf")', so it is natural to expect nan and inf > constants in some namespace. > > Both good points? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/9e6a51f9/attachment.html> From steve at pearwood.info Fri Jan 9 04:13:50 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 9 Jan 2015 14:13:50 +1100 Subject: [Python-ideas] Float range class In-Reply-To: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> Message-ID: <20150109031344.GP19387@ando.pearwood.info> On Thu, Jan 08, 2015 at 05:01:03PM +0100, Todd wrote: > Currently, the range class is limited to integers. However, in many > applications, especially science and mathematics, ranges of float values > are the norm. [...] Solving this problem for the standard library would be a good idea. The math module might be the best place for it, although that would require giving up the concept that it is a thin wrapper around your platform's C math routines. (I'm not sure just how much that idea still applies.) I experimented with a few different ways of doing non-integral ranges: This is probably the least correct way: https://code.activestate.com/recipes/577068-floating-point-range/ This recipe was more accurate than numpy's linspace, at least at the time: https://code.activestate.com/recipes/577878-generate-equally-spaced-floats/ This recipe tries to combine them both: https://code.activestate.com/recipes/577881-equally-spaced-floats-part-2/ If something like this makes it into the standard library, I think it is important that it isn't limited to just floats. If you pass it Fractions, or Decimals, it should return Fractions or Decimals, and not just convert everything to float. > The first is to extend the existing range class so it can support floats or > to use duck-typing to support any class that has the necessary methods. > However, I would be worried about making such a fundamental change to such > a core class, and either approach could hurt performance. A very strong -1 on that. > The second approach is to create a new class that handles floating-point > ranges (perhaps called "frange") and/or a new class that handles duck-typed > ranges (perhaps called "drange"). However, there isn't really a good place > right now to put these as far as I can see. No need to have two such classes. There should be one, and it should be polymorphic. If you want floats, pass it float arguments, if you want some other numeric type, pass it the type you want. As for where it should live, the math module is the obvious place, but it is written in C (which is a barrier to adding new functionality, as opposed to Python modules). Maybe it's time for the math.so library to be re-named to _math.so (or .dll on Windows), and a new math.py module created which ends with from _math import * That will allow new functionality to be written in Python first, and only written in C if necessary. > The third approach is to create a new module for various range classes. > For example there could be a float range, a datetime range, a decimal > range, a duck-typed range, an abstract base class for making new range > classes, etc. However, this is a lot more work (although it wouldn't need > to all be done at one time). YAGNI. -- Steven From ethan at stoneleaf.us Fri Jan 9 04:19:23 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 08 Jan 2015 19:19:23 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> Message-ID: <54AF48BB.5050303@stoneleaf.us> On 01/08/2015 08:01 AM, Todd wrote: > > Currently, the range class is limited to integers. However, in many applications, especially science and mathematics, > ranges of float values are the norm. Here is a proposed range replacement -- it works with int and float directly, and will attempt to work with any other type for which sufficient information can be gleaned from the provided parameters: - range(Date(2015, 1, 1), step=ONE_DAY, count=31) # step must be specified; gives every day in January 2015 - range(Fraction(5, 10), count=3) # step is deduced as 10/10 (aka 1) - range(100.0) # 0.0, 1.0, 2.0, ..., 99.0 For more flexibility, an epsilon keyword argument is available, and step can be a function: range([start,] stop [,step] [,count] [,epsilon]) If step is a function, on each iteration it is called with start_value, current_iteration, last_value. epsilon is used to check for the stop point, and for containment checks. Containment is only supported when the iteration type and step value support addition, subtraction, and division, and the step cannot be a function (in these cases, use list() to get a concrete sequence). This particular version works from 2.6+ and 3.2+ (3.1 should work if you add a 'callable' function), which is why it's called xrange in the code. --- 8< xrange.py -------------------------------------------------------------- from decimal import Decimal class xrange(object): ''' accepts arbitrary objects to use to produce sequences ''' def __init__(self, start, stop=None, step=None, count=None, epsilon=None): if stop is not None and count is not None: raise ValueError("cannot specify both stop and count") if stop is None and count is None: # check for default start based on type start, stop = None, start if isinstance(stop, (int, float, Decimal, complex)): start = type(stop)(0) else: raise ValueError("start must be specified for type %r" % type(stop)) if start is None: ref = type(stop) else: ref = type(start) if step is None: try: step = ref(1) except TypeError: raise ValueError("step must be specified for type %r" % type(stop)) if not callable(step): ref = type(step) if epsilon is None: if isinstance(start, (float, Decimal, complex)): try: epsilon = .0000001 * step except TypeError: epsilon = ref(0) else: epsilon = ref(0) if count is None: try: stop = stop - epsilon except TypeError: pass self.start = self.init_start = start self.stop = stop self.step = step self.count = self.init_count = count self.epsilon = epsilon def __contains__(self, value): start, stop, step = self.start, self.stop, self.step count, epsilon = self.count, self.epsilon if callable(step): raise TypeError( "range with step %r does not support containment checks" % step) try: distance = round((value - start) / step) except TypeError: raise TypeError( "range of %s with step %s does not support " "containment checks" % (type(start), type(step))) if start - epsilon <= value <= start + epsilon: return True if step > type(step)(0): if start + epsilon > value: return False elif stop is not None and stop - epsilon <= value: return False elif count is not None and distance >= count: return False else: if start + epsilon < value: return False elif stop is not None and stop - epsilon >= value: return False elif count is not None and distance >= count: return False target = start + distance * step if target - epsilon <= value <= target + epsilon: return True return False def __iter__(self): start = self.start stop = self.stop step = self.step count = self.count epsilon = self.epsilon i = -1 while 'more values to yield': i += 1 if callable(step): if i: value = step(start, i, value) else: value = start else: value = start + i * step if count is not None: if count < 1: break count -= 1 else: if stop > start and value >= stop: break if stop < start and value <= stop: break yield value def __repr__(self): values = [ '%s=%s' % (k,v) for k,v in ( ('start',self.start), ('stop',self.stop), ('step', self.step), ('count', self.count), ('epsilon', self.epsilon), ) if v is not None ] return '<%s(%s)>' % (self.__class__.__name__, ', '.join(values)) --- 8< xrange.py -------------------------------------------------------------- --- 8< test_xrange.py --------------------------------------------------------- from unittest import TestCase, main from xrange import xrange import datetime class Test_xrange(TestCase): def test_int_iter_forwards(self): self.assertEqual( list(range(10)), list(xrange(10))) self.assertEqual( list(range(0, 10)), list(xrange(0, 10))) self.assertEqual( list(range(0, 10, 1)), list(xrange(0, 10, 1))) self.assertEqual( list(range(0, 10, 1)), list(xrange(0, count=10))) self.assertEqual( list(range(0, 10, 1)), list(xrange(10, step=lambda s, i, v: v+1))) self.assertEqual( list(range(0, 10, 1)), list(xrange(0, 10, step=lambda s, i, v: v+1))) self.assertEqual( list(range(5, 15)), list(xrange(5, count=10))) self.assertEqual( list(range(-10, 0)), list(xrange(-10, 0))) self.assertEqual( list(range(-9, 1)), list(xrange(-9, 1))) self.assertEqual( list(range(-20, 20, 1)), list(xrange(-20, 20, 1))) self.assertEqual( list(range(-20, 20, 2)), list(xrange(-20, 20, 2))) self.assertEqual( list(range(-20, 20, 3)), list(xrange(-20, 20, 3))) self.assertEqual( list(range(-20, 20, 4)), list(xrange(-20, 20, 4))) self.assertEqual( list(range(-20, 20, 5)), list(xrange(-20, 20, 5))) def test_int_iter_backwards(self): self.assertEqual( list(range(9, -1, -1)), list(xrange(9, -1, -1))) self.assertEqual( list(range(9, -9, -1)), list(xrange(9, -9, -1))) self.assertEqual( list(range(9, -9, -2)), list(xrange(9, -9, -2))) self.assertEqual( list(range(9, -9, -3)), list(xrange(9, -9, -3))) self.assertEqual( list(range(9, 0, -1)), list(xrange(9, 0, -1))) self.assertEqual( list(range(9, -1, -1)), list(xrange(9, step=-1, count=10))) def test_int_containment(self): robj = xrange(10) for i in range(10): self.assertTrue(i in robj) self.assertFalse(-1 in robj) self.assertFalse(10 in robj) self.assertFalse(5.23 in robj) def test_float_iter(self): floats = [float(i) for i in range(100)] self.assertEqual( floats, list(xrange(100.0))) self.assertEqual( floats, list(xrange(0, 100.0))) self.assertEqual( floats, list(xrange(0, 100.0, 1.0))) self.assertEqual( floats, list(xrange(100.0, step=lambda s, i, v: v + 1.0))) self.assertEqual( floats, list(xrange(100.0, step=lambda s, i, v: s + i * 1.0))) self.assertEqual( floats, list(xrange(0.0, count=100))) self.assertEqual( [0.3, 0.6], list(xrange(0.3, 0.9, 0.3))) self.assertEqual( [0.4, 0.8], list(xrange(0.4, 1.2, 0.4))) def test_float_iter_backwards(self): floats = [float(i) for i in range(99, -1, -1)] self.assertEqual( floats, list(xrange(99, -1, -1))) self.assertEqual( floats, list(xrange(99, step=lambda s, i, v: v - 1.0, count=100))) self.assertEqual( [0.6, 0.3], list(xrange(0.6, 0.0, -0.3))) self.assertEqual( [0.8, 0.4] , list(xrange(0.8, 0.0, -0.4))) def test_float_containment(self): robj = xrange(100000000.0) for i in [float(i) for i in range(10000)]: self.assertTrue(i in robj) self.assertFalse(0.000001 in robj) self.assertFalse(100000000.0 in robj) self.assertFalse(50.23 in robj) def test_date_iter(self): ONE_DAY = datetime.timedelta(1) ONE_WEEK = datetime.timedelta(7) robj = xrange(datetime.date(2014, 1, 1), step=ONE_DAY, count=31) day1 = datetime.date(2014, 1, 1) riter = iter(robj) try: datetime.timedelta(7) / datetime.timedelta(1) containment = True except TypeError: containment = False for i in range(31): day = day1 + i * ONE_DAY rday = next(riter) self.assertEqual(day, rday) if containment: self.assertTrue(day in robj) else: self.assertRaises(TypeError, robj.__contains__, day) self.assertRaises(StopIteration, next, riter) if containment: self.assertFalse(day + ONE_DAY in robj) else: self.assertRaises(TypeError, robj.__contains__, day + ONE_DAY) def test_fraction_iter(self): from fractions import Fraction as F f = xrange(F(5, 10), count=3) self.assertEqual([F(5, 10), F(15, 10), F(25, 10)], list(f)) if __name__ == '__main__': main() --- 8< test_xrange.py --------------------------------------------------------- -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/82f92384/attachment-0001.sig> From njs at pobox.com Fri Jan 9 04:29:13 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 9 Jan 2015 03:29:13 +0000 Subject: [Python-ideas] Float range class In-Reply-To: <08B511D5-9594-4C02-A1CA-BFC44F26E306@yahoo.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <BF30F199-DFDA-4003-B160-6CFFBD0DD14D@yahoo.com> <CAFpSVpKxUbY+azZRPY+=vo3Ht8RuchruUh1AN56EzKCjcwTiiQ@mail.gmail.com> <CAPJVwB=HXy6P7tGV_wiPo1P2Fsz1jEP5ePcECw_4LMSLQu-3hA@mail.gmail.com> <08B511D5-9594-4C02-A1CA-BFC44F26E306@yahoo.com> Message-ID: <CAPJVwBnkmjN5Xv4dkvskkjH3MvPDrpMiCurP_B2C7Erk7akHWQ@mail.gmail.com> On 8 Jan 2015 21:30, "Andrew Barnert" <abarnert at yahoo.com> wrote: > > On Jan 8, 2015, at 10:40, Nathaniel Smith <njs at pobox.com> wrote: > > > On Thu, Jan 8, 2015 at 5:58 PM, Todd <toddrjen at gmail.com> wrote: > >> On Thu, Jan 8, 2015 at 6:41 PM, Andrew Barnert <abarnert at yahoo.com> wrote: > >>> > >>> More generally: if an frange that works exactly like range but with floats > >>> (and rounding errors) is all you want, that's so trivial to write that it > >>> doesn't seem worth putting in the stdlib, unless the use case is very > >>> common. > >> > >> In numerical computing the use-case is extremely common, in fact I would say > >> it practically universal. Ideally it would be vectorized, but in practice > >> this is often not possible, requiring the use of loops. > > > > Can you give a more specific example? I believe your problem is real, > > but as a numerical programmer I can't think of an example of where > > I've ever wanted anything like this, so it's hard for me to visualize > > what exactly you're talking about. > > > > And is what you want just: > > > > class FRange: > > def __init__(start, stop, step): > > self.start = start > > self.stop = stop > > self.step = step > > > > def __getitem__(self, i): > > x = self.start + i * self.step > > if x > self.stop: > > raise IndexError > > > def __iter__(self): > > i = 0 > > while True: > > try: > > yield self[i] > > except IndexError: > > return > > i += 1 > > This isn't really necessary; any type whose __getitem__ takes int values from 0 up to some point and then raises IndexError beyond that is automatically iterable (and the built-in sequence iterator is basically a C-optimized version of what you just wrote). > > While we're at it, you probably also want __len__, __contains__ (one more place for naive float rounding problems?), __reversed__, index, count, a __getitem__ that handles slices and negative values, and __hash__. All pretty simple to add. (In fact, if you inherit collections.abc.Sequence you'll get everything but __getitem__, __len__, and __hash__ for free, but you'd probably want to implement __contains__, index, and count anyway, because the default versions will be linear instead of constant.) Most of those I forgot about :-), but leaving out __contains__ and friends was intentional, because my feeling is that there's *really* no valid use case for those in the floating point case. Best to pretend that == just doesn't exist on FP numbers. I'd still like an answer to the question about what some or all of this is good for, though. Certainly you *can* implement all of those things -- but why? Concretely, what's the use case? -n -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/2386cc9c/attachment.html> From jeanpierreda at gmail.com Fri Jan 9 04:20:39 2015 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Thu, 8 Jan 2015 19:20:39 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <20150109031344.GP19387@ando.pearwood.info> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <20150109031344.GP19387@ando.pearwood.info> Message-ID: <CABicbJLDR79Hot7Auk1Tk+DiA9ue=QmErsAFjf-d+zB5QTCo0g@mail.gmail.com> On Thu, Jan 8, 2015 at 7:13 PM, Steven D'Aprano <steve at pearwood.info> wrote: > On Thu, Jan 08, 2015 at 05:01:03PM +0100, Todd wrote: >> Currently, the range class is limited to integers. However, in many >> applications, especially science and mathematics, ranges of float values >> are the norm. > [...] > > Solving this problem for the standard library would be a good idea. The > math module might be the best place for it, although that would require > giving up the concept that it is a thin wrapper around your platform's C > math routines. (I'm not sure just how much that idea still applies.) > -snip- > If something like this makes it into the standard library, I think it is > important that it isn't limited to just floats. If you pass it > Fractions, or Decimals, it should return Fractions or Decimals, and not > just convert everything to float. -snip- > As for where it should live, the math module is the obvious place, but > it is written in C (which is a barrier to adding new functionality, as > opposed to Python modules). I actually expect the math module to convert to float, just like I expect the cmath module to convert to complex. For example: # python 3.4 >>> math.sqrt(decimal.Decimal(0)) 0.0 Are there any exceptions to this? -- Devin From steve at pearwood.info Fri Jan 9 04:34:10 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 9 Jan 2015 14:34:10 +1100 Subject: [Python-ideas] Float range class In-Reply-To: <CABicbJLDR79Hot7Auk1Tk+DiA9ue=QmErsAFjf-d+zB5QTCo0g@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <20150109031344.GP19387@ando.pearwood.info> <CABicbJLDR79Hot7Auk1Tk+DiA9ue=QmErsAFjf-d+zB5QTCo0g@mail.gmail.com> Message-ID: <20150109033410.GQ19387@ando.pearwood.info> On Thu, Jan 08, 2015 at 07:20:39PM -0800, Devin Jeanpierre wrote: [...] > > As for where it should live, the math module is the obvious place, but > > it is written in C (which is a barrier to adding new functionality, as > > opposed to Python modules). > > I actually expect the math module to convert to float, just like I > expect the cmath module to convert to complex. For example: > > # python 3.4 > >>> math.sqrt(decimal.Decimal(0)) > 0.0 > > Are there any exceptions to this? py> from math import factorial py> factorial(5.0) 120 -- Steve From njs at pobox.com Fri Jan 9 04:33:02 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 9 Jan 2015 03:33:02 +0000 Subject: [Python-ideas] Float range class In-Reply-To: <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> Message-ID: <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> On 9 Jan 2015 03:02, "Neil Girdhar" <mistersheik at gmail.com> wrote: > > I agree with everyone above. At first I was +1 on this proposal, but why not suggest to the numpy people that arange and linespace should return Sequence objects rather than numpy arrays? I'm pretty sure there's a way to make sure objects work in numpy. It'd be technically possible for numpy.arange to return a special object that was coerceable on demand to a real array, but it'd break tons of code and isn't going to happen. (There's a reason it took py3k to get rid of xrange. There are no plans for numpy3k at this time.) -n -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/66b17892/attachment.html> From jeanpierreda at gmail.com Fri Jan 9 04:42:32 2015 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Thu, 8 Jan 2015 19:42:32 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <20150109033410.GQ19387@ando.pearwood.info> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <20150109031344.GP19387@ando.pearwood.info> <CABicbJLDR79Hot7Auk1Tk+DiA9ue=QmErsAFjf-d+zB5QTCo0g@mail.gmail.com> <20150109033410.GQ19387@ando.pearwood.info> Message-ID: <CABicbJK-O79UDu=KT7OgSh=EA6Qr9iwN=S-2=f9EPu_6NWcbqA@mail.gmail.com> On Thu, Jan 8, 2015 at 7:34 PM, Steven D'Aprano <steve at pearwood.info> wrote: > py> from math import factorial > py> factorial(5.0) > 120 This kind of grosses me out, but OK. This is sufficient as an argument to demonstrate that math is not only for floating point math, but for math in general. Although I still find it doubtful that we'd ever replace, say, math.sqrt with a polymorphic sqrt -- in my head somebody says "no, this would slow down existing Python programs by XYZ%" and then it doesn't happen. Maybe I'm being pessimistic. -- Devin From ethan at stoneleaf.us Fri Jan 9 04:50:44 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 08 Jan 2015 19:50:44 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <54AF48BB.5050303@stoneleaf.us> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <54AF48BB.5050303@stoneleaf.us> Message-ID: <54AF5014.1070408@stoneleaf.us> On 01/08/2015 07:19 PM, Ethan Furman wrote: > For more flexibility, an epsilon keyword argument is available, and step can be a function: > > range([start,] stop [,step] [,count] [,epsilon]) On second and third thought, having step be a function makes implementing __len__, __index__, find, etc., etc., either impossible or expensive, so nevermind that part. ;) -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/851df18c/attachment.sig> From abarnert at yahoo.com Fri Jan 9 05:27:33 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 8 Jan 2015 20:27:33 -0800 Subject: [Python-ideas] Better comand line version of python -c In-Reply-To: <CAATkbzAD96W-suKCu1-pDrJdTF_qOefXPUPXTH8+optLyUXf+w@mail.gmail.com> References: <CAATkbzBPf_u3CqYMtpSUoEJFV56ouqAa5UGAUNK2ksgXxZ--AQ@mail.gmail.com> <1272917759.3995303.1420756506710.JavaMail.yahoo@jws100140.mail.ne1.yahoo.com> <CAATkbzAD96W-suKCu1-pDrJdTF_qOefXPUPXTH8+optLyUXf+w@mail.gmail.com> Message-ID: <CC7248E2-FD0A-42AD-B9E4-F0AAD852D316@yahoo.com> On Jan 8, 2015, at 15:14, Russell Stewart <Russell.S.Stewart at gmail.com> wrote: [lots of unmarked snips below] > Json handling: > I think I would remove this if I did it again. I originally found it quite useful, but the fact that it doesn't support multi-line json inputs makes it weird. Also, all the functionality can be accessed through the json module with python (e.g. py -l 'json.loads(l)'). But, as I pointed out, it would be pretty easy to make it handle a stream of multi-line JSON inputs by looping over raw_decode--and that would _not_ be easy to access in a one-liner by the user. So with that change, it could be useful. > -csv option: > I considered this at length, and originally the --si was motivated by commas. But ultimately --si is good for lots of things other than commas. A separate csv command seems nice, until you realize that there are too many options to specify in parameters, just like in the json case. I don't understand what you mean here. In the JSON case, there aren't any options to specify at all. In the CSV case, there are lots of dialect options, but most of them are rarely used; being able to specify a delimiter (which you already need for --si) or a named dialect covers 90% of what people need. > py vs python -m pythonpy ...: > Yes, using this with python -m is a very real option. People could simply alias py='python -m pythonpy' to avoid typing issues. This is probably the most straightforward distribution option. It would be great to keep the ipython style tab completion, but I'm not sure that its necessary. I don't understand what you mean about tab completion. If you're talking about having bash (or zsh or whatever) do IPython-style tab completion in the expression, you can write a completion script that remembers that sets a variable when it sees "-m pythonpy" and completes non-option arguments with your Python completion instead of with files if that variable is set. The only trick with the alias is that you have to pass Python's arguments before the -m and pythonpy's after it? but I suspect that most people will never need Python arguments, and those who do will need them often enough that they'll just set up a separate alias for `python -u -m pythonpy`. > py -fx flag: > Originally, I didn't want to name this -f because I didn't want the active variable x to be magically introduced. When you type -fx, its clear where the variable x comes from. This was probably a mistake, because everyone hates 2 character single-dash flags. It's not that everyone hates 2 character single-dash flags, it's that everyone hates inconsistency. Your script is _almost_ compatible with the GNU standard (single-char single-dash arguments and multi-char double-dash arguments), but in a few places (-fx instead of --fx, --i instead of -i) you violate it. Why not just follow it consistently? > Should be changed in the case of an overhaul. This command is also only marginally useful, due to the fact that grep already handles this use case so well, and because you can get the functionality with py -x 'x if foo(x) else None' just as well. Does that mean your script doesn't print out None values? I can see how that's useful in some cases, but it could be pretty bad--and surprising--in others. > pythonpy should be Iterable based, not list-based: > I agree somewhat. The current dictionary functionality is weird, only printing the list of keys. Again, I don't want to change this for backwards compat reasons. This could be done a lot of different ways, and I think the current strategy is pretty good, but leaves a little bit to be desired in terms of simplicity. The -x and -l flags capture a lot of the list-printing that you would want to do, so its not clear you need to do list-printing with no arguments. Using a separate flag for list printing would be a good design consideration here. Keep in mind that the current motivation for all the list strangeness is so that pythonpy will play well with unix pipes, which it does. That is really important for some core usecases. I don't understand this paragraph. What does any of this have to do with just changing all the stuff that requires lists to require any iterable, and/or changing -l to give you an iterator instead of a list? In fact, the reason I want to be able to output any iterable--in particular, an iterator--line by line without having to convert to a list is to use it with standard Unix pipelines. If I have a huge input file to filter, I'd rather use, say, a generator expression and have it read and output lines one at a time than have to process the whole thing into a list in memory before outputting anything. (Yes, I realize that -x covers many of the same cases--but just as map doesn't make genexprs useless in scripts, I don't think -x would make them useless in one-liners.) > Daemon mode: > I hadn't thought of this one before. Maybe it could be done well, but daemon modes tend incur a high cost of mental overhead on the user. If designed right, there really isn't any overhead as long as either (a) the user doesn't mind leaving it running forever, or (b) there's a timeout in case you forget to kill it. Then I'd just alias `py` to `python -m pythonpy --daemon=300` and everything would work exactly the same as before, except that it would start faster, and I could use variables set by `-c`/`-C` or the previous result (as `_`) if I wanted to. > I've worked hard to keep the startup time within a factor of 2 from the python interpreter, so I do care a lot about that. I know "Windows users don't care about the command line"--but Cygwin users do, and they still have to deal with the Windows process launch delay, and there are some Unixes that people still use that can't launch processes as fast as Linux or BSD. > --i flag: > Again, the naming here is up for change. It's useful in a few cases, where python throws exceptions easily (e.g. py -x re.match(r"b", x).group(0)). But I haven't used it all that much, and it really could be removed. I wasn't arguing that it was useless--on the contrary, I think I'd use it often. (Although again, I'd also like a "short exception output" option.) I was just suggesting that it should be `-i` rather than `--i` (both to stick with the standard you're almost following, and to save a keystroke on something I think would be useful often, but not often enough to make a new alias for...). > So in general I like the python -m idea. Support for tab completion under this would be really nice. I've long thought that python -c and python -m should have ipython style tab completion anyways, On my system, `python -m` does have tab completion (although it's not as smart as it could be--e.g., even with -s it still searches user site for completions). I'm not sure what package this comes from, but it's pretty trivial. Making `python -c` do completion for Python statements is cool, it just requires someone writing the completer. It sounds like you've already written one for pythonpy; it's just a matter of modifying it to handle statements instead of expressions and hooking it to the `-c` argument of your python completion. That seems like it would be a useful thing in its own right (although I'm not sure the bash-completions and zsh people would take it upstream, so it might have to be a separate package). > so that one could type python -m Sim<tab> and get python -m SimpleHTTPServer. Another idea would be to try and get a shorter binary name distributed with python. It sounds superficial, but is ultimately quite important if python wants to compete with ruby, perl, and sed. I really don't think that's important. The reason people use perl and sed isn't that the command is 2 or 3 characters shorter (if it were, they'd just create an alias), it's that many trivial one-liners are more verbose in Python. For example, consider looping over each line in each filename passed on the command line. In perl that's a few characters; in python (even pythonpy) you have to loop over fileinput.input to do the same. And now imagine that what you want to do with each line is an re.sub; in Python, just the quoting is longer than the whole thing in Perl. In general, this isn't a weakness (it's what makes Python code readable), but if you're really interested in "competing with Perl" on its own terms, those are the parts you have to compete with, not the name of the command. > On Thu, Jan 8, 2015 at 2:35 PM, Andrew Barnert <abarnert at yahoo.com> wrote: >> On Thursday, January 8, 2015 1:11 PM, Russell Stewart <Russell.S.Stewart at gmail.com> wrote: >> >> >> >I am the original dev for pypi.python.org/pypi/pythonpy, which offers much of the same functionality as python -c, but in a more convenient form. I'd be really interested in figuring out whether the core python tools could support many of the use cases that pythonpy covers, so that users wouldn't have to install it as a separate tool. I don't have a concrete idea of what it would look like, but many of my users have said they would much prefer an officially supported tool. Perhaps someone on this mailing list can provide some insight into the available options. >> >> This looks pretty cool to me?so I have lots of questions and comments. :) >> >> The obvious problem is what to call it. `py` (well, `py.exe`) is already the name of the PEP 397 launcher tool (the thing that lets you use shbang lines on Windows), so this would need a new name. >> >> Maybe it would be better as just a different flag or set of flags on `python` itself, like `--pprint`/`-p`? Of course then there's the problem of where to stick the flags for `-p` (for GNU-style long names this is pretty easy: `--print=filter,lines`, but obviously for something meant to be used for quick&dirty command-line usage you need short names as well?). And the "backport" to 3.4 (and 2.x) would act differently from the standard 3.5+ version. But it might be worth looking at anyway. >> >> >> Or, of course, just make this a stdlib module, so it's just `python [PYTHON OPTIONS] -m YOUR_MODULENAME [YOUR MODULE OPTIONS]`, and you can alias that to whatever you want. >> >> Is `-fx` a multi-character single-hyphen flag (in which case it's very weird to mix those with GNU-style long arguments in the same program, or is it a combination of `-f` and `-x`? And is the filter expression an argument to `-fx`, or does `-f` just change the interpretation of the argument? >> >> >> Speaking of GNU-style long arguments, why `--i` instead of just `-i` (as a short name for `--ignore_exceptions` or something) >> >> The documentation is pretty Python 2-specific: the majority of the examples use the `range` function and depend on it returning a list. >> >> >> You seem to have built a non-trivial custom pretty-printer for this tool (e.g., to print lists row by row) as well as an auto-importer (e.g., to use `collections.Counter` without an `import collections`); maybe some or all of that should be separated out and made available to Python code somewhere in the stdlib, and then the script (or flag) could just use that function? >> >> This seems to be pretty strongly list-based for no good reason. Why not print _any_ iterable line by line, make -l set `l = sys.stdin` instead of `l = list(sys.stdin)` (I realize that doesn't work for the specific example of `py -l 'l[::-1]'`, but `py -l reversed(l)` would work just as well in that case, and it's hard to think of other examples where there would even be a problem), etc.? >> >> It might be nice to have an option to see the `repr` instead of the `str` of everything, to match what you'd see from the interactive interpreter. >> >> While --si is cool, people already misuse `str.split` and `re.split` to try to parse CSV and similar input that has quotes or escapes; it might be nice to have a `--csv` mode to parse the input with `csv.reader` (and that could still take an optional delimiter, of course; passing other dialect flags to `reader` might be out of scope). >> >> A columnar reading mode might also be nice, since that's one of those things that's novices have a hard time writing without statements. >> >> An option for pretty JSON output instead of compact JSON output might be nice. >> >> And maybe an option to read multi-line JSON input?not just a single JSON value (which is trivial with `json.loads(l)`, but, since JSON is self-delimiting, still a stream of them, without the requirement of one/line (e.g., by looping over `json.JSONDecoder.raw_decode`). >> >> I assume you're evaluating the expression with `eval`. Are you passing it a custom locals and globals (so any internals of your script itself aren't available), or not? I could see disadvantages of both (the former may prevent some useful quick&dirty hacks; the latter could raise safety concerns), but either way, it's probably worth documenting. >> >> From the summary help, it looks like you can apply both `-x` and `-fx`. If you do that, do they run in a specific order, or in the order specified? For that matter, can you apply multiple mappings and/or filterings? (If not, it seems like that could be handy.) >> >> It might be useful to have some kind of "daemon mode", where you start up an interpreter in the background and then pipe input to it. Then you can pipe multiple things to the same interpreter session, both for persistence, and to avoid the process startup cost on platforms where it matters (like Windows). Maybe with an optional timeout, where the daemon stops a few minutes after last use, so you don't have to remember to `py -d sys.exit(0)` or similar. And maybe the magic `_` should work in daemon mode, as it does in the interactive interpreter. (I could see writing some one-liner that takes 90 seconds to run, then realizing I wanted it in JSON format, so `py --daemon -jo _` would be handy to avoid re-running everything?) >> >> It would be really cool if, when run under PowerShell, this could handle scriptlet input and output instead of plain text (so, e.g., you pass it an array of strings and it processes each string). >> >> Being able to somehow pass command-line arguments through to `python` itself (assuming this isn't merged into the main interpreter, of course) might be handy. In particular, I could see `-u` being useful, but there might be others (including platform/implementation-specific `-X` options). Of course that would come for free with the `-m` interface, but since I'm not sure that's the best option... >> >> Instead of just being able to ignore exceptions, it might be nice to enable one-line exception output (e.g., print just the type and message, no traceback, and to stdout instead of stderr). >> >> With `-x`, an option to prefix each line of output with the corresponding line of input could be handy, similar to the `-v` option to commands like `cp`, `tar`, etc. >> >> A way to feed input files in could be handy to allow the kind of one-liners people often fall back to perl for, although I'm not sure exactly what you'd want this to look like. >> >> Some of these are probably way out of scope even for a third-party tool, much less for a built-in stdlib version, but I'm not sure exactly which, so I've just dumped everything on you to let you sort them out. :) > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/545fe4f5/attachment-0001.html> From abarnert at yahoo.com Fri Jan 9 05:46:30 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 8 Jan 2015 20:46:30 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> Message-ID: <18FA5E2F-8FCF-4A47-AA42-C8CF494255E2@yahoo.com> On Jan 8, 2015, at 14:35, Chris Barker <chris.barker at noaa.gov> wrote: > On Thu, Jan 8, 2015 at 1:33 PM, Andrew Barnert <abarnert at yahoo.com> wrote: > >> Implementing it badly isn't much of a problem; even if you don't precompute the stop, you're just adding the cost of a multiplication to every __getitem__, and I doubt that's going to be a bottleneck in most applications. > > I wasn't talking about performance, I was talking about the vagaries of floating point. I think you may be disagreeing with the wrong person here (Todd is the one who proposed this feature; I was pointing out problems with it), but just in case you actually meant that a properly-implemented frange actually _could_ solve problems with floating-point vagaries: > for example, an obvious (easy, anyway) thing to do is somethign like: > > val = start > while val < stop: > val += step > > but that's not a great idea with fp. But this is just as bad an idea--and it's presumably exactly equivalent to what frange would do: for i in count(): val = start + i*step if val >= stop: break For example, try it with start=0, step=0.3, stop=0.9, and you'll get 0, 0.3. 0.6, and 0.8999999999999999--exactly the same (probably-)incorrect result you get from repeated addition. In other words, as I said, the problem isn't that it's difficult to implement correctly; the problem is that it's difficult to _use_ correctly. Which, to me, seems like an argument against it being in the stdlib, not for it. >> It's _using_ it badly that's the more serious problem. Just as with np.arange, a naive use of frange will lead to rounding problems, and I can't see how a builtin could provide any API that avoids that problem? > > which is why I'm suggesting that a built in be more like numpy's linspace than like range. See Warren's note for a better explanation with examples. Sure; I said effectively the same thing. That's why I mentioned that arange has the same problem. A float linspace view might be useful; a float range view is an attractive nuisance. So I think we agree? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/c20148cc/attachment.html> From Russell.S.Stewart at gmail.com Fri Jan 9 06:02:15 2015 From: Russell.S.Stewart at gmail.com (Russell Stewart) Date: Thu, 8 Jan 2015 21:02:15 -0800 Subject: [Python-ideas] Better comand line version of python -c In-Reply-To: <CC7248E2-FD0A-42AD-B9E4-F0AAD852D316@yahoo.com> References: <CAATkbzBPf_u3CqYMtpSUoEJFV56ouqAa5UGAUNK2ksgXxZ--AQ@mail.gmail.com> <1272917759.3995303.1420756506710.JavaMail.yahoo@jws100140.mail.ne1.yahoo.com> <CAATkbzAD96W-suKCu1-pDrJdTF_qOefXPUPXTH8+optLyUXf+w@mail.gmail.com> <CC7248E2-FD0A-42AD-B9E4-F0AAD852D316@yahoo.com> Message-ID: <CAATkbzAK2W9_hrXTzw3DY7zurnSZJExUF+n3-QdkGXiRZkwFMA@mail.gmail.com> I think if you actually used one of the tools in the space for a little while (be it pythonpy or spy, which seems good, or the not so great pyp), you may come to realize why your intuition for the user interface is so flawed. On Thu, Jan 8, 2015 at 8:27 PM, Andrew Barnert <abarnert at yahoo.com> wrote: > On Jan 8, 2015, at 15:14, Russell Stewart <Russell.S.Stewart at gmail.com> > wrote: > > [lots of unmarked snips below] > > Json handling: > I think I would remove this if I did it again. I originally found it quite > useful, but the fact that it doesn't support multi-line json inputs makes > it weird. Also, all the functionality can be accessed through the json > module with python (e.g. py -l 'json.loads(l)'). > > > But, as I pointed out, it would be pretty easy to make it handle a stream > of multi-line JSON inputs by looping over raw_decode--and that would _not_ > be easy to access in a one-liner by the user. So with that change, it could > be useful. > > -csv option: > I considered this at length, and originally the --si was motivated by > commas. But ultimately --si is good for lots of things other than commas. A > separate csv command seems nice, until you realize that there are too many > options to specify in parameters, just like in the json case. > > > I don't understand what you mean here. In the JSON case, there aren't any > options to specify at all. In the CSV case, there are lots of dialect > options, but most of them are rarely used; being able to specify a > delimiter (which you already need for --si) or a named dialect covers 90% > of what people need. > > py vs python -m pythonpy ...: > Yes, using this with python -m is a very real option. People could simply > alias py='python -m pythonpy' to avoid typing issues. This is probably the > most straightforward distribution option. It would be great to keep the > ipython style tab completion, but I'm not sure that its necessary. > > > I don't understand what you mean about tab completion. If you're talking > about having bash (or zsh or whatever) do IPython-style tab completion in > the expression, you can write a completion script that remembers that sets > a variable when it sees "-m pythonpy" and completes non-option arguments > with your Python completion instead of with files if that variable is set. > > The only trick with the alias is that you have to pass Python's arguments > before the -m and pythonpy's after it? but I suspect that most people will > never need Python arguments, and those who do will need them often enough > that they'll just set up a separate alias for `python -u -m pythonpy`. > > py -fx flag: > Originally, I didn't want to name this -f because I didn't want the active > variable x to be magically introduced. When you type -fx, its clear where > the variable x comes from. This was probably a mistake, because everyone > hates 2 character single-dash flags. > > > It's not that everyone hates 2 character single-dash flags, it's that > everyone hates inconsistency. Your script is _almost_ compatible with the > GNU standard (single-char single-dash arguments and multi-char double-dash > arguments), but in a few places (-fx instead of --fx, --i instead of -i) > you violate it. Why not just follow it consistently? > > Should be changed in the case of an overhaul. This command is also only > marginally useful, due to the fact that grep already handles this use case > so well, and because you can get the functionality with py -x 'x if foo(x) > else None' just as well. > > > Does that mean your script doesn't print out None values? I can see how > that's useful in some cases, but it could be pretty bad--and surprising--in > others. > > pythonpy should be Iterable based, not list-based: > I agree somewhat. The current dictionary functionality is weird, only > printing the list of keys. Again, I don't want to change this for backwards > compat reasons. This could be done a lot of different ways, and I think the > current strategy is pretty good, but leaves a little bit to be desired in > terms of simplicity. The -x and -l flags capture a lot of the list-printing > that you would want to do, so its not clear you need to do list-printing > with no arguments. Using a separate flag for list printing would be a good > design consideration here. Keep in mind that the current motivation for all > the list strangeness is so that pythonpy will play well with unix pipes, > which it does. That is really important for some core usecases. > > > I don't understand this paragraph. What does any of this have to do with > just changing all the stuff that requires lists to require any iterable, > and/or changing -l to give you an iterator instead of a list? > > In fact, the reason I want to be able to output any iterable--in > particular, an iterator--line by line without having to convert to a list > is to use it with standard Unix pipelines. If I have a huge input file to > filter, I'd rather use, say, a generator expression and have it read and > output lines one at a time than have to process the whole thing into a list > in memory before outputting anything. (Yes, I realize that -x covers many > of the same cases--but just as map doesn't make genexprs useless in > scripts, I don't think -x would make them useless in one-liners.) > > Daemon mode: > I hadn't thought of this one before. Maybe it could be done well, but > daemon modes tend incur a high cost of mental overhead on the user. > > > If designed right, there really isn't any overhead as long as either (a) > the user doesn't mind leaving it running forever, or (b) there's a timeout > in case you forget to kill it. Then I'd just alias `py` to `python -m > pythonpy --daemon=300` and everything would work exactly the same as > before, except that it would start faster, and I could use variables set by > `-c`/`-C` or the previous result (as `_`) if I wanted to. > > I've worked hard to keep the startup time within a factor of 2 from the > python interpreter, so I do care a lot about that. > > > I know "Windows users don't care about the command line"--but Cygwin users > do, and they still have to deal with the Windows process launch delay, and > there are some Unixes that people still use that can't launch processes as > fast as Linux or BSD. > > --i flag: > Again, the naming here is up for change. It's useful in a few cases, where > python throws exceptions easily (e.g. py -x re.match(r"b", x).group(0)). > But I haven't used it all that much, and it really could be removed. > > > I wasn't arguing that it was useless--on the contrary, I think I'd use it > often. (Although again, I'd also like a "short exception output" option.) I > was just suggesting that it should be `-i` rather than `--i` (both to stick > with the standard you're almost following, and to save a keystroke on > something I think would be useful often, but not often enough to make a new > alias for...). > > So in general I like the python -m idea. Support for tab completion under > this would be really nice. I've long thought that python -c and python -m > should have ipython style tab completion anyways, > > > On my system, `python -m` does have tab completion (although it's not as > smart as it could be--e.g., even with -s it still searches user site for > completions). I'm not sure what package this comes from, but it's pretty > trivial. > > Making `python -c` do completion for Python statements is cool, it just > requires someone writing the completer. It sounds like you've already > written one for pythonpy; it's just a matter of modifying it to handle > statements instead of expressions and hooking it to the `-c` argument of > your python completion. That seems like it would be a useful thing in its > own right (although I'm not sure the bash-completions and zsh people would > take it upstream, so it might have to be a separate package). > > so that one could type python -m Sim<tab> and get python -m > SimpleHTTPServer. Another idea would be to try and get a shorter binary > name distributed with python. It sounds superficial, but is ultimately > quite important if python wants to compete with ruby, perl, and sed. > > > I really don't think that's important. The reason people use perl and sed > isn't that the command is 2 or 3 characters shorter (if it were, they'd > just create an alias), it's that many trivial one-liners are more verbose > in Python. For example, consider looping over each line in each filename > passed on the command line. In perl that's a few characters; in python > (even pythonpy) you have to loop over fileinput.input to do the same. And > now imagine that what you want to do with each line is an re.sub; in > Python, just the quoting is longer than the whole thing in Perl. In > general, this isn't a weakness (it's what makes Python code readable), but > if you're really interested in "competing with Perl" on its own terms, > those are the parts you have to compete with, not the name of the command. > > > On Thu, Jan 8, 2015 at 2:35 PM, Andrew Barnert <abarnert at yahoo.com> wrote: > >> On Thursday, January 8, 2015 1:11 PM, Russell Stewart < >> Russell.S.Stewart at gmail.com> wrote: >> >> >> >I am the original dev for pypi.python.org/pypi/pythonpy, which offers >> much of the same functionality as python -c, but in a more convenient form. >> I'd be really interested in figuring out whether the core python tools >> could support many of the use cases that pythonpy covers, so that users >> wouldn't have to install it as a separate tool. I don't have a concrete >> idea of what it would look like, but many of my users have said they would >> much prefer an officially supported tool. Perhaps someone on this mailing >> list can provide some insight into the available options. >> >> This looks pretty cool to me?so I have lots of questions and comments. :) >> >> The obvious problem is what to call it. `py` (well, `py.exe`) is already >> the name of the PEP 397 launcher tool (the thing that lets you use shbang >> lines on Windows), so this would need a new name. >> >> Maybe it would be better as just a different flag or set of flags on >> `python` itself, like `--pprint`/`-p`? Of course then there's the problem >> of where to stick the flags for `-p` (for GNU-style long names this is >> pretty easy: `--print=filter,lines`, but obviously for something meant to >> be used for quick&dirty command-line usage you need short names as well?). >> And the "backport" to 3.4 (and 2.x) would act differently from the standard >> 3.5+ version. But it might be worth looking at anyway. >> >> >> Or, of course, just make this a stdlib module, so it's just `python >> [PYTHON OPTIONS] -m YOUR_MODULENAME [YOUR MODULE OPTIONS]`, and you can >> alias that to whatever you want. >> >> Is `-fx` a multi-character single-hyphen flag (in which case it's very >> weird to mix those with GNU-style long arguments in the same program, or is >> it a combination of `-f` and `-x`? And is the filter expression an argument >> to `-fx`, or does `-f` just change the interpretation of the argument? >> >> >> Speaking of GNU-style long arguments, why `--i` instead of just `-i` (as >> a short name for `--ignore_exceptions` or something) >> >> The documentation is pretty Python 2-specific: the majority of the >> examples use the `range` function and depend on it returning a list. >> >> >> You seem to have built a non-trivial custom pretty-printer for this tool >> (e.g., to print lists row by row) as well as an auto-importer (e.g., to use >> `collections.Counter` without an `import collections`); maybe some or all >> of that should be separated out and made available to Python code somewhere >> in the stdlib, and then the script (or flag) could just use that function? >> >> This seems to be pretty strongly list-based for no good reason. Why not >> print _any_ iterable line by line, make -l set `l = sys.stdin` instead of >> `l = list(sys.stdin)` (I realize that doesn't work for the specific example >> of `py -l 'l[::-1]'`, but `py -l reversed(l)` would work just as well in >> that case, and it's hard to think of other examples where there would even >> be a problem), etc.? >> >> It might be nice to have an option to see the `repr` instead of the `str` >> of everything, to match what you'd see from the interactive interpreter. >> >> While --si is cool, people already misuse `str.split` and `re.split` to >> try to parse CSV and similar input that has quotes or escapes; it might be >> nice to have a `--csv` mode to parse the input with `csv.reader` (and that >> could still take an optional delimiter, of course; passing other dialect >> flags to `reader` might be out of scope). >> >> A columnar reading mode might also be nice, since that's one of those >> things that's novices have a hard time writing without statements. >> >> An option for pretty JSON output instead of compact JSON output might be >> nice. >> >> And maybe an option to read multi-line JSON input?not just a single JSON >> value (which is trivial with `json.loads(l)`, but, since JSON is >> self-delimiting, still a stream of them, without the requirement of >> one/line (e.g., by looping over `json.JSONDecoder.raw_decode`). >> >> I assume you're evaluating the expression with `eval`. Are you passing it >> a custom locals and globals (so any internals of your script itself aren't >> available), or not? I could see disadvantages of both (the former may >> prevent some useful quick&dirty hacks; the latter could raise safety >> concerns), but either way, it's probably worth documenting. >> >> From the summary help, it looks like you can apply both `-x` and `-fx`. >> If you do that, do they run in a specific order, or in the order specified? >> For that matter, can you apply multiple mappings and/or filterings? (If >> not, it seems like that could be handy.) >> >> It might be useful to have some kind of "daemon mode", where you start up >> an interpreter in the background and then pipe input to it. Then you can >> pipe multiple things to the same interpreter session, both for persistence, >> and to avoid the process startup cost on platforms where it matters (like >> Windows). Maybe with an optional timeout, where the daemon stops a few >> minutes after last use, so you don't have to remember to `py -d >> sys.exit(0)` or similar. And maybe the magic `_` should work in daemon >> mode, as it does in the interactive interpreter. (I could see writing some >> one-liner that takes 90 seconds to run, then realizing I wanted it in JSON >> format, so `py --daemon -jo _` would be handy to avoid re-running >> everything?) >> >> It would be really cool if, when run under PowerShell, this could handle >> scriptlet input and output instead of plain text (so, e.g., you pass it an >> array of strings and it processes each string). >> >> Being able to somehow pass command-line arguments through to `python` >> itself (assuming this isn't merged into the main interpreter, of course) >> might be handy. In particular, I could see `-u` being useful, but there >> might be others (including platform/implementation-specific `-X` options). >> Of course that would come for free with the `-m` interface, but since I'm >> not sure that's the best option... >> >> Instead of just being able to ignore exceptions, it might be nice to >> enable one-line exception output (e.g., print just the type and message, no >> traceback, and to stdout instead of stderr). >> >> With `-x`, an option to prefix each line of output with the corresponding >> line of input could be handy, similar to the `-v` option to commands like >> `cp`, `tar`, etc. >> >> A way to feed input files in could be handy to allow the kind of >> one-liners people often fall back to perl for, although I'm not sure >> exactly what you'd want this to look like. >> >> Some of these are probably way out of scope even for a third-party tool, >> much less for a built-in stdlib version, but I'm not sure exactly which, so >> I've just dumped everything on you to let you sort them out. :) >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/4b872ba7/attachment-0001.html> From abarnert at yahoo.com Fri Jan 9 06:38:14 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 9 Jan 2015 05:38:14 +0000 (UTC) Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <20150107160925.GH19387@ando.pearwood.info> References: <20150107160925.GH19387@ando.pearwood.info> Message-ID: <1556539310.4083422.1420781894547.JavaMail.yahoo@jws100143.mail.ne1.yahoo.com> On Wednesday, January 7, 2015 8:10 AM, Steven D'Aprano <steve at pearwood.info> wrote: > Up to now, I have only discussed inf, not nan, because there is only one > inf value in floats. (Two if you count -inf.) But the IEEE 754 standard > provides many different NAN values, and offers a possible interpretion > of them: the extra "payload" on each NAN can be used to carry > diagnostic > information on which numeric operation failed. Back in the 1980s, I was > a user of the Apple Standard Numerics Environment (SANE), which provided > easy access to NAN payloads and a standard interpretation for them. It > is a system which can work well and help in debugging. > > Python today doesn't give us any simple way to access all those > different float NANs, or offer any interpretation of what they might > mean, but it might some day. I am strongly opposed to anything which > might pre-empt that or discourage future advances. I don't think there's really an issue here; the only thing this preempts is using the plain name `float.nan` for such future advances. What's wrong with, say, `float.make_nan(payload=30, quiet=False, sign=True)`? Given that the former is almost certainly going to be used far more often than the latter, the extra characters to type and read for the uncommon case don't seem like a serious problem. Since you brought up SANE, from what I remember (I tried to dig up the Apple Numerics Manual or the semi-public SANE 68k assembly source, but I give up?), this is roughly equivalent to what SANE did. There was a function to return a NaN with a code from 0-255, with meanings like "square root of a negative" or "division by zero" or "invalid NaN code"?the last being used if you tried to call it with 0 as the code. If you wanted the all-zeros NaN, that was a separate function. Meanwhile, if you only care about platforms that use x86/68k-style representations for IEEE-754 (unfortunately, pre-2008 the bits weren't standardized, so at least MIPS did things differently), `make_nan` is pretty easy. Something like: def make_nan(payload=0, sign=False, quiet=True): payload |= (0x7ff0000000000000 | (sign * 0x8000000000000000) | (quiet * 0x0008000000000000)) return struct.unpack('d', struct.pack('Q', payload))[0] And likewise, you could write a `parse_nan` that returns `payload, sign, quiet`. And you'd probably want to add some parameter validation (and don't forget that payload 0 and quiet-bit 0 means it's not a NaN, it's an inf). From abarnert at yahoo.com Fri Jan 9 06:46:10 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 9 Jan 2015 05:46:10 +0000 (UTC) Subject: [Python-ideas] Better comand line version of python -c In-Reply-To: <CAATkbzAK2W9_hrXTzw3DY7zurnSZJExUF+n3-QdkGXiRZkwFMA@mail.gmail.com> References: <CAATkbzAK2W9_hrXTzw3DY7zurnSZJExUF+n3-QdkGXiRZkwFMA@mail.gmail.com> Message-ID: <1782252028.4088356.1420782370114.JavaMail.yahoo@jws100119.mail.ne1.yahoo.com> On Thursday, January 8, 2015 9:02 PM, Russell Stewart <Russell.S.Stewart at gmail.com> wrote: >I think if you actually used one of the tools in the space for a little while (be it pythonpy or spy, which seems good, or the not so great pyp), you may come to realize why your intuition for the user interface is so flawed. Which part of it is flawed? After playing with it a bit, I still think `-i` would be better than `--i` and `-fx` feels weird; that Cygwin users do care about the command line, that I'd like a daemon mode (or some other magic way to access `_`), that if JSON is useful it would be more useful to parse a stream of multi-line JSON objects than requiring them to be one per line, that I have no need for any CSV options besides delimiter or dialect, that I don't understand what you were trying to say in your iterable-vs.-list paragraph, etc. And I'm even _more_ convinced that I'd like a shortcut for sys.stdin as an iterator rather than as a list (and for iterators to be printed line by line as lists are, but lazily--except that I think you already _do_ that, even though the docs say otherwise); it's easy to stall the `-l` pipeline with a huge input, which other Unix tools don't have a problem with. More generally, it would help if you'd reply inline rather than top-posting; it's hard to figure out what your replies are in reply to. >On Thu, Jan 8, 2015 at 8:27 PM, Andrew Barnert <abarnert at yahoo.com> wrote: > >On Jan 8, 2015, at 15:14, Russell Stewart <Russell.S.Stewart at gmail.com> wrote: >> >> >>[lots of unmarked snips below] >> >> >>Json handling: >>>I think I would remove this if I did it again. I originally found it quite useful, but the fact that it doesn't support multi-line json inputs makes it weird. Also, all the functionality can be accessed through the json module with python (e.g. py -l 'json.loads(l)'). >> >> >>But, as I pointed out, it would be pretty easy to make it handle a stream of multi-line JSON inputs by looping over raw_decode--and that would _not_ be easy to access in a one-liner by the user. So with that change, it could be useful. >> >>-csv option: >>>I considered this at length, and originally the --si was motivated by commas. But ultimately --si is good for lots of things other than commas. A separate csv command seems nice, until you realize that there are too many options to specify in parameters, just like in the json case. >> >> >>I don't understand what you mean here. In the JSON case, there aren't any options to specify at all. In the CSV case, there are lots of dialect options, but most of them are rarely used; being able to specify a delimiter (which you already need for --si) or a named dialect covers 90% of what people need. >> >>py vs python -m pythonpy ...: >>>Yes, using this with python -m is a very real option. People could simply alias py='python -m pythonpy' to avoid typing issues. This is probably the most straightforward distribution option. It would be great to keep the ipython style tab completion, but I'm not sure that its necessary. >> >> >>I don't understand what you mean about tab completion. If you're talking about having bash (or zsh or whatever) do IPython-style tab completion in the expression, you can write a completion script that remembers that sets a variable when it sees "-m pythonpy" and completes non-option arguments with your Python completion instead of with files if that variable is set. >> >> >>The only trick with the alias is that you have to pass Python's arguments before the -m and pythonpy's after it? but I suspect that most people will never need Python arguments, and those who do will need them often enough that they'll just set up a separate alias for `python -u -m pythonpy`. >> >>py -fx flag: >>>Originally, I didn't want to name this -f because I didn't want the active variable x to be magically introduced. When you type -fx, its clear where the variable x comes from. This was probably a mistake, because everyone hates 2 character single-dash flags. >> >> >>It's not that everyone hates 2 character single-dash flags, it's that everyone hates inconsistency. Your script is _almost_ compatible with the GNU standard (single-char single-dash arguments and multi-char double-dash arguments), but in a few places (-fx instead of --fx, --i instead of -i) you violate it. Why not just follow it consistently? >> >>Should be changed in the case of an overhaul. This command is also only marginally useful, due to the fact that grep already handles this use case so well, and because you can get the functionality with py -x 'x if foo(x) else None' just as well. >> >> >>Does that mean your script doesn't print out None values? I can see how that's useful in some cases, but it could be pretty bad--and surprising--in others. >> >>pythonpy should be Iterable based, not list-based: >>>I agree somewhat. The current dictionary functionality is weird, only printing the list of keys. Again, I don't want to change this for backwards compat reasons. This could be done a lot of different ways, and I think the current strategy is pretty good, but leaves a little bit to be desired in terms of simplicity. The -x and -l flags capture a lot of the list-printing that you would want to do, so its not clear you need to do list-printing with no arguments. Using a separate flag for list printing would be a good design consideration here. Keep in mind that the current motivation for all the list strangeness is so that pythonpy will play well with unix pipes, which it does. That is really important for some core usecases. >> >>I don't understand this paragraph. What does any of this have to do with just changing all the stuff that requires lists to require any iterable, and/or changing -l to give you an iterator instead of a list? >> >> >>In fact, the reason I want to be able to output any iterable--in particular, an iterator--line by line without having to convert to a list is to use it with standard Unix pipelines. If I have a huge input file to filter, I'd rather use, say, a generator expression and have it read and output lines one at a time than have to process the whole thing into a list in memory before outputting anything. (Yes, I realize that -x covers many of the same cases--but just as map doesn't make genexprs useless in scripts, I don't think -x would make them useless in one-liners.) >> >> >>Daemon mode: >>>I hadn't thought of this one before. Maybe it could be done well, but daemon modes tend incur a high cost of mental overhead on the user. >> >> >>If designed right, there really isn't any overhead as long as either (a) the user doesn't mind leaving it running forever, or (b) there's a timeout in case you forget to kill it. Then I'd just alias `py` to `python -m pythonpy --daemon=300` and everything would work exactly the same as before, except that it would start faster, and I could use variables set by `-c`/`-C` or the previous result (as `_`) if I wanted to. >> >>I've worked hard to keep the startup time within a factor of 2 from the python interpreter, so I do care a lot about that. >> >> >>I know "Windows users don't care about the command line"--but Cygwin users do, and they still have to deal with the Windows process launch delay, and there are some Unixes that people still use that can't launch processes as fast as Linux or BSD. >> >>--i flag: >>>Again, the naming here is up for change. It's useful in a few cases, where python throws exceptions easily (e.g. py -x re.match(r"b", x).group(0)). But I haven't used it all that much, and it really could be removed. >> >> >>I wasn't arguing that it was useless--on the contrary, I think I'd use it often. (Although again, I'd also like a "short exception output" option.) I was just suggesting that it should be `-i` rather than `--i` (both to stick with the standard you're almost following, and to save a keystroke on something I think would be useful often, but not often enough to make a new alias for...). >> >>So in general I like the python -m idea. Support for tab completion under this would be really nice. I've long thought that python -c and python -m should have ipython style tab completion anyways, >> >> >>On my system, `python -m` does have tab completion (although it's not as smart as it could be--e.g., even with -s it still searches user site for completions). I'm not sure what package this comes from, but it's pretty trivial. >> >> >>Making `python -c` do completion for Python statements is cool, it just requires someone writing the completer. It sounds like you've already written one for pythonpy; it's just a matter of modifying it to handle statements instead of expressions and hooking it to the `-c` argument of your python completion. That seems like it would be a useful thing in its own right (although I'm not sure the bash-completions and zsh people would take it upstream, so it might have to be a separate package). >> >>so that one could type python -m Sim<tab> and get python -m SimpleHTTPServer. Another idea would be to try and get a shorter binary name distributed with python. It sounds superficial, but is ultimately quite important if python wants to compete with ruby, perl, and sed. >> >> >>I really don't think that's important. The reason people use perl and sed isn't that the command is 2 or 3 characters shorter (if it were, they'd just create an alias), it's that many trivial one-liners are more verbose in Python. For example, consider looping over each line in each filename passed on the command line. In perl that's a few characters; in python (even pythonpy) you have to loop over fileinput.input to do the same. And now imagine that what you want to do with each line is an re.sub; in Python, just the quoting is longer than the whole thing in Perl. In general, this isn't a weakness (it's what makes Python code readable), but if you're really interested in "competing with Perl" on its own terms, those are the parts you have to compete with, not the name of the command. >> >> >> >> >>On Thu, Jan 8, 2015 at 2:35 PM, Andrew Barnert <abarnert at yahoo.com> wrote: >>> >>>On Thursday, January 8, 2015 1:11 PM, Russell Stewart <Russell.S.Stewart at gmail.com> wrote: >>>> >>>> >>>>>I am the original dev for pypi.python.org/pypi/pythonpy, which offers much of the same functionality as python -c, but in a more convenient form. I'd be really interested in figuring out whether the core python tools could support many of the use cases that pythonpy covers, so that users wouldn't have to install it as a separate tool. I don't have a concrete idea of what it would look like, but many of my users have said they would much prefer an officially supported tool. Perhaps someone on this mailing list can provide some insight into the available options. >>>> >>>>This looks pretty cool to me?so I have lots of questions and comments. :) >>>> >>>>The obvious problem is what to call it. `py` (well, `py.exe`) is already the name of the PEP 397 launcher tool (the thing that lets you use shbang lines on Windows), so this would need a new name. >>>> >>>>Maybe it would be better as just a different flag or set of flags on `python` itself, like `--pprint`/`-p`? Of course then there's the problem of where to stick the flags for `-p` (for GNU-style long names this is pretty easy: `--print=filter,lines`, but obviously for something meant to be used for quick&dirty command-line usage you need short names as well?). And the "backport" to 3.4 (and 2.x) would act differently from the standard 3.5+ version. But it might be worth looking at anyway. >>>> >>>> >>>>Or, of course, just make this a stdlib module, so it's just `python [PYTHON OPTIONS] -m YOUR_MODULENAME [YOUR MODULE OPTIONS]`, and you can alias that to whatever you want. >>>> >>>>Is `-fx` a multi-character single-hyphen flag (in which case it's very weird to mix those with GNU-style long arguments in the same program, or is it a combination of `-f` and `-x`? And is the filter expression an argument to `-fx`, or does `-f` just change the interpretation of the argument? >>>> >>>> >>>>Speaking of GNU-style long arguments, why `--i` instead of just `-i` (as a short name for `--ignore_exceptions` or something) >>>> >>>>The documentation is pretty Python 2-specific: the majority of the examples use the `range` function and depend on it returning a list. >>>> >>>> >>>>You seem to have built a non-trivial custom pretty-printer for this tool (e.g., to print lists row by row) as well as an auto-importer (e.g., to use `collections.Counter` without an `import collections`); maybe some or all of that should be separated out and made available to Python code somewhere in the stdlib, and then the script (or flag) could just use that function? >>>> >>>>This seems to be pretty strongly list-based for no good reason. Why not print _any_ iterable line by line, make -l set `l = sys.stdin` instead of `l = list(sys.stdin)` (I realize that doesn't work for the specific example of `py -l 'l[::-1]'`, but `py -l reversed(l)` would work just as well in that case, and it's hard to think of other examples where there would even be a problem), etc.? >>>> >>>>It might be nice to have an option to see the `repr` instead of the `str` of everything, to match what you'd see from the interactive interpreter. >>>> >>>>While --si is cool, people already misuse `str.split` and `re.split` to try to parse CSV and similar input that has quotes or escapes; it might be nice to have a `--csv` mode to parse the input with `csv.reader` (and that could still take an optional delimiter, of course; passing other dialect flags to `reader` might be out of scope). >>>> >>>>A columnar reading mode might also be nice, since that's one of those things that's novices have a hard time writing without statements. >>>> >>>>An option for pretty JSON output instead of compact JSON output might be nice. >>>> >>>>And maybe an option to read multi-line JSON input?not just a single JSON value (which is trivial with `json.loads(l)`, but, since JSON is self-delimiting, still a stream of them, without the requirement of one/line (e.g., by looping over `json.JSONDecoder.raw_decode`). >>>> >>>>I assume you're evaluating the expression with `eval`. Are you passing it a custom locals and globals (so any internals of your script itself aren't available), or not? I could see disadvantages of both (the former may prevent some useful quick&dirty hacks; the latter could raise safety concerns), but either way, it's probably worth documenting. >>>> >>>>From the summary help, it looks like you can apply both `-x` and `-fx`. If you do that, do they run in a specific order, or in the order specified? For that matter, can you apply multiple mappings and/or filterings? (If not, it seems like that could be handy.) >>>> >>>>It might be useful to have some kind of "daemon mode", where you start up an interpreter in the background and then pipe input to it. Then you can pipe multiple things to the same interpreter session, both for persistence, and to avoid the process startup cost on platforms where it matters (like Windows). Maybe with an optional timeout, where the daemon stops a few minutes after last use, so you don't have to remember to `py -d sys.exit(0)` or similar. And maybe the magic `_` should work in daemon mode, as it does in the interactive interpreter. (I could see writing some one-liner that takes 90 seconds to run, then realizing I wanted it in JSON format, so `py --daemon -jo _` would be handy to avoid re-running everything?) >>>> >>>>It would be really cool if, when run under PowerShell, this could handle scriptlet input and output instead of plain text (so, e.g., you pass it an array of strings and it processes each string). >>>> >>>>Being able to somehow pass command-line arguments through to `python` itself (assuming this isn't merged into the main interpreter, of course) might be handy. In particular, I could see `-u` being useful, but there might be others (including platform/implementation-specific `-X` options). Of course that would come for free with the `-m` interface, but since I'm not sure that's the best option... >>>> >>>>Instead of just being able to ignore exceptions, it might be nice to enable one-line exception output (e.g., print just the type and message, no traceback, and to stdout instead of stderr). >>>> >>>>With `-x`, an option to prefix each line of output with the corresponding line of input could be handy, similar to the `-v` option to commands like `cp`, `tar`, etc. >>>> >>>>A way to feed input files in could be handy to allow the kind of one-liners people often fall back to perl for, although I'm not sure exactly what you'd want this to look like. >>>> >>>>Some of these are probably way out of scope even for a third-party tool, much less for a built-in stdlib version, but I'm not sure exactly which, so I've just dumped everything on you to let you sort them out. :) >>>> >>> > > > From jeanpierreda at gmail.com Fri Jan 9 06:46:23 2015 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Thu, 8 Jan 2015 21:46:23 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <1556539310.4083422.1420781894547.JavaMail.yahoo@jws100143.mail.ne1.yahoo.com> References: <20150107160925.GH19387@ando.pearwood.info> <1556539310.4083422.1420781894547.JavaMail.yahoo@jws100143.mail.ne1.yahoo.com> Message-ID: <CABicbJ+LqpwBP4YY5C1sLHJPHd+GoyHU06CB3miTb+Q15_mPHw@mail.gmail.com> On Thu, Jan 8, 2015 at 9:38 PM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: > On Wednesday, January 7, 2015 8:10 AM, Steven D'Aprano <steve at pearwood.info> wrote: >> Python today doesn't give us any simple way to access all those >> different float NANs, or offer any interpretation of what they might >> mean, but it might some day. I am strongly opposed to anything which >> might pre-empt that or discourage future advances. > > I don't think there's really an issue here; the only thing this preempts is using the plain name `float.nan` for such future advances. What's wrong with, say, `float.make_nan(payload=30, quiet=False, sign=True)`? Given that the former is almost certainly going to be used far more often than the latter, the extra characters to type and read for the uncommon case don't seem like a serious problem. +1, a classmethod is the right place for this. -- Devin From edk141 at gmail.com Fri Jan 9 07:10:28 2015 From: edk141 at gmail.com (Ed Kellett) Date: Fri, 09 Jan 2015 06:10:28 +0000 Subject: [Python-ideas] Better comand line version of python -c References: <CAATkbzAK2W9_hrXTzw3DY7zurnSZJExUF+n3-QdkGXiRZkwFMA@mail.gmail.com> <1782252028.4088356.1420782370114.JavaMail.yahoo@jws100119.mail.ne1.yahoo.com> Message-ID: <CABmzr0iT675COwNEnKj+omFV081X_H__RGwmqhaMWixY=9ia6w@mail.gmail.com> On Fri Jan 09 2015 at 5:47:02 AM Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: > On Thursday, January 8, 2015 9:02 PM, Russell Stewart < > Russell.S.Stewart at gmail.com> wrote: > > And I'm even _more_ convinced that I'd like a shortcut for sys.stdin as an > iterator rather than as a list (and for iterators to be printed line by > line as lists are, but lazily--except that I think you already _do_ that, > even though the docs say otherwise); it's easy to stall the `-l` pipeline > with a huge input, which other Unix tools don't have a problem with. > spy's handling of multiline inputs might suit you better, although in this case it doesn't seem like a characteristic of the interface as much as an implementation detail. On a completely different note, do you have any specific thoughts about how the daemon mode is going to work? I considered it for spy, but getting it exactly right is non-trivial: - Multiple users on the same system should definitely work - Different pythons or different virtualenvs under the same user should work - Where should the daemon's stdout go? - If `_` is present, what will happen when you inevitably end up working on two things at once in different terminals? - If this requires any manual help from the user, we've lost. It's not convenient enough Ed Kellett -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/cd381ba2/attachment.html> From ncoghlan at gmail.com Fri Jan 9 08:35:37 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 9 Jan 2015 17:35:37 +1000 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <54ADD7C9.9030709@gmail.com> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> <54AB4178.3050508@gmail.com> <97D50920-B7D2-4125-9389-224C4CBC83CA@yahoo.com> <54AC952E.5060301@gmail.com> <74164843-0774-441F-A898-4776E15EFC8D@yahoo.com> <8C6A6BAA-197B-4796-9163-6EE3AE3E1553@gmail.com> <54ADD7C9.9030709@gmail.com> Message-ID: <CADiSq7c0hFqm-XE7nx+r_RBLHmR-_zVinoDD6-9ja54M-UGo2w@mail.gmail.com> On 8 January 2015 at 11:05, Yawar Amin <yawar.amin at gmail.com> wrote: > As promised, a write-up of my idea: > http://yawar.blogspot.ca/2015/01/expressive-functional-programming-with.html The statement/expression distinction is deliberate, for the reasons you describe in the final article of your post. You will have zero chance of persuading anyone if you start from an assumption that the distinction is accidental. That said, there are some *very* limited situations where pulling things out-of-line in order to name them reduces readability (see PEPs 403 and 3150 for further discussion of that), and other situations where semantically significant leading whitespace causes practical problems (see http://python-notes.curiousefficiency.org/en/latest/pep_ideas/suite_expr.html for the discussion of that). If you can come up with a practical proposal that better handles those situations, *without* having a generally negative effect on code readability, and without introducing "two ways to do it" for various problems that will complicate learning, then you may have something signficant. But for something as fundamental as introducing new syntax, you need to start with concrete use cases, rather than hypothetical "it might be useful if..." situations. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From abarnert at yahoo.com Fri Jan 9 09:24:42 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 9 Jan 2015 08:24:42 +0000 (UTC) Subject: [Python-ideas] Better comand line version of python -c In-Reply-To: <CABmzr0iT675COwNEnKj+omFV081X_H__RGwmqhaMWixY=9ia6w@mail.gmail.com> References: <CABmzr0iT675COwNEnKj+omFV081X_H__RGwmqhaMWixY=9ia6w@mail.gmail.com> Message-ID: <1628625043.4104139.1420791882196.JavaMail.yahoo@jws10043.mail.ne1.yahoo.com> On Thursday, January 8, 2015 10:10 PM, Ed Kellett <edk141 at gmail.com> wrote: >On a completely different note, do you have any specific thoughts about how the daemon mode is going to work? I considered it for spy, but getting it exactly right is non-trivial: I was having a hard time trying to explain it, so after my third rewrite, I decided to instead hack up a (very) quick&dirty eval daemon (https://github.com/abarnert/pyd). The daemon part is basically invisible. If a daemon is already running, it's used; if not, a new one is started; to kill an existing daemon, just run with no expression. (Actually, it's not quite invisible; I added in some lines to print the daemon's pid when launched and to tell you whether it was running when killed, for debugging purposes. But those could be removed.) Your expression gets passed to `eval`, your input gets sent to the expression's stdin line by line, its stdout and stderr get sent to your stdout line by line, and then the result of your expression gets stored in `_` for next time, and printed. For example: $ echo | ./pyd 2+3 Started daemon on 8089 5 $ echo | ./pyd _+3 8 $ echo -e 'abc\ndef\n' | ./pyd 'input()' abc $ echo | ./pyd _+3 TypeError: unsupported operand type(s) for +: 'int' and 'str' $ echo | ./pyd '_+"3"' abc3 $ ./pyd Daemon killed $ ./pyd Daemon was not running $ echo | ./pyd _+3 NameError: name '_' is not defined $ echo | ./pyd 2+3 5 $ echo | ./pyd _+3 8 You're probably wondering what's with the `echo`. Proxying `sys.stdin` line by line to a socket in both terminal and non-terminal mode on Mac, Linux, and Cygwin has some annoying problems, and the workaround takes more than a couple lines, so I made the code loop over `input` instead, which means it ends with a blank line rather than EOF, which sucks as a UI but it's fine for testing the functionality. I haven't thought through all of the issues with in what senses this is and isn't a daemon; I just closed stdin/out/err and called `setsid`. I also didn't include a timeout argument; the daemon just runs until you kill it. The `eval` is done in the daemon's own environment. There's probably lots of missing error handling. It's only tested on Mac and Fedora, and definitely won't work on (native) Windows. Of course it doesn't do any of the fancy stuff pythonpy or spy does; it just calls eval. It may not work with very long expressions or very long input lines or lots of separate output writes, and it definitely doesn't work if the output is more than 4K. But all of those things are doable in a real program if you have more than 15 minutes to write it. :) >- Multiple users on the same system should definitely work That's easy; my hack creates the socket under `$HOME`. A real script would hopefully use somewhere XDG/Mac-compliant if relevant. Or you could encode the uid into the socket name and store it in `$TMP` if you prefer. >- Different pythons or different virtualenvs under the same user should work I apologize that I haven't looked at spy yet, but with pythonpy it looks like you have to have a separate script for each Python, which means all you need to do it add the script's name (`basename(argv[0])`) to the socket name. If you want to do something like `python -m pythonpy 2+3`, then you'll probably want the full path to `sys.executable` instead (which means you need to encode it in some way to deal with slashes, of course). >- Where should the daemon's stdout go? Back over the socket. (There shouldn't be any stdout between connections. If there is, it'll go nowhere.) >- If `_` is present, what will happen when you inevitably end up working on two things at once in different terminals? Whoever runs last (note that the hack doesn't allow concurrent connections; I'm not sure if you'd want to or not) overwrites `_`. Personally, the use cases I'm envisioning for `_` are accessing it immediately after the last command--typically because I'm an idiot and forgot a step and don't want to have to re-run the previous step just to add another one (the same reason I usually use `_` in the interactive interpreter). So, I think this is fine. But if you want separate sessions, that's easy. Just encode the sid in the socket name, so you have one daemon per session. If you want to get fancy, you could instead use an environment variable that's set to the sid (or just to a random UUID or whatever) if not present; that way by default each terminal gets its own daemon, but if you want to share a daemon between sessions all you have to do is set the env variable manually to match. You might also want to consider whether the daemon should die when the controlling shell/session/terminal dies, if you have a separate daemon per terminal. >- If this requires any manual help from the user, we've lost. It's not convenient enough It shouldn't. If you want to provide options to let the user get fancy you can (like the last paragraph above), but I don't think any such options are necessary. The only options I might provide are to use the daemon or not, and an idle timeout to kill the daemon. From abarnert at yahoo.com Fri Jan 9 09:46:48 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 9 Jan 2015 08:46:48 +0000 (UTC) Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <CADiSq7c0hFqm-XE7nx+r_RBLHmR-_zVinoDD6-9ja54M-UGo2w@mail.gmail.com> References: <CADiSq7c0hFqm-XE7nx+r_RBLHmR-_zVinoDD6-9ja54M-UGo2w@mail.gmail.com> Message-ID: <818732810.4093303.1420793208359.JavaMail.yahoo@jws100159.mail.ne1.yahoo.com> On Thursday, January 8, 2015 11:35 PM, Nick Coghlan <ncoghlan at gmail.com> wrote: > The statement/expression distinction is deliberate, for the reasons > you describe in the final article of your post. You will have zero > chance of persuading anyone if you start from an assumption that the > distinction is accidental. It strikes me that it might be useful if there were some good official explanation of this (maybe in the Design & History FAQ). Many other "modern" languages (Ruby, CoffeeScript, etc.) try to eliminate or hide the distinction as much as possible. And JavaScript documentation often apologizes for having the distinction and explains how you can "work around the problem" by defining and calling a function inline to wrap any statement. So, many people believe that the distinction is all negatives with no positives. I believe the distinction isn't just there to keep the parser simpler, or because Guido wasn't clever enough to figure out how to embed statements in expressions in an indentation-significant language, or because the examples of Ruby and CoffeeScript didn't exist yet and now it's too late for Python; statements are a big part of what makes Python code more readable than those languages, both in-depth and in a quick scan to see the general flow control and state changes. But it's hard to explain why. And, even if I manage to explain why _I_ think statements make Python more readable, that's just my opinion; it would be nice to have something that Guido endorsed. Also, for people who want to suggest changes to Python (or design their own languages), it would be helpful to understand how the distinction fits in with other things. For example, in a pure functional language, there may be no benefit to having statements. In a language where (almost) everything is an expression, having mutating methods return self instead of None wouldn't be nearly as much of a problem. More generally, expression-ness and not-mutating are deeply connected (e.g., comprehensions aren't designed for mutation, loop statements are), but it's not easy to say how. And so on. From ncoghlan at gmail.com Fri Jan 9 10:12:30 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 9 Jan 2015 19:12:30 +1000 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAP7h-xZFd3DGvhr0D6D3rzJBm5BgZ3yX9jaOCQaFGPM36GD74g@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <CAG3cHaaXAXs87ZK=wk1q191HR2tt0EVdZTFSgo=2xLO_feujbw@mail.gmail.com> <CAFpSVp+1ee8mM_xbPkANeNjWJGRr8zAd=SFe+VC5EO-2L4P2Dg@mail.gmail.com> <54AD4F71.7060008@oa-cagliari.inaf.it> <CAP7h-xZFd3DGvhr0D6D3rzJBm5BgZ3yX9jaOCQaFGPM36GD74g@mail.gmail.com> Message-ID: <CADiSq7ei5ugJd61bN9gcUX__Gz32wPbhCvUTPpq3uph-=-5_4Q@mail.gmail.com> On 8 January 2015 at 02:07, Alexander Belopolsky <alexander.belopolsky at gmail.com> wrote: > > On Wed, Jan 7, 2015 at 10:23 AM, Marco Buttu <marco.buttu at gmail.com> wrote: >> >> On 07/01/2015 15:58, Todd wrote: >> >>> Yes, this would be a shortcut for that. math.nan and math.inf are less >>> verbose than float("nan") and float("inf"), even if you aren't already using >>> the math module (fewer characters total, and in particular fewer special >>> characters). It would also, at least in my opinion, be more readable. Plus >>> there is already the math.isnan and math.isinf, so I don't think it would be >>> totally out-of-place. >> >> >> I totally agree. +1 > > > Another +1. > > Two points: > > 1. The current ways to create inf and nan are not obvious and hard to > discover. It will be nice to have these constants discoverable in tab > completion from math. > > 2. The repr()s of nan and inf are simply 'nan' and 'inf', not 'float("nan")' > and 'float("inf")', so it is natural to expect nan and inf constants in some > namespace. 3. Having these as documented constants provides a natural home for documentation of some of the standard assumptions about the behaviour of containers and comparisons that they can break (this is mostly NaN related - off the top of my head, I don't recall any particularly weird inf-induced behaviour) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Fri Jan 9 10:24:59 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 Jan 2015 09:24:59 +0000 Subject: [Python-ideas] Better comand line version of python -c In-Reply-To: <CAATkbzAD96W-suKCu1-pDrJdTF_qOefXPUPXTH8+optLyUXf+w@mail.gmail.com> References: <CAATkbzBPf_u3CqYMtpSUoEJFV56ouqAa5UGAUNK2ksgXxZ--AQ@mail.gmail.com> <1272917759.3995303.1420756506710.JavaMail.yahoo@jws100140.mail.ne1.yahoo.com> <CAATkbzAD96W-suKCu1-pDrJdTF_qOefXPUPXTH8+optLyUXf+w@mail.gmail.com> Message-ID: <CACac1F-K4UWAF8ez-cm+6DoMA1HmCnMajOWCKZeh1tCxrf960Q@mail.gmail.com> On 8 January 2015 at 23:14, Russell Stewart <Russell.S.Stewart at gmail.com> wrote: > Windows py.exe issue: > Yea, I'm aware of this. I have a pythonwpy for windows that is run with wpy, > but no one seems to care about comand line tools on windows. I do! :-) I'm planning on looking at this module as soon as I get a chance - it sounds very nice! Paul From ncoghlan at gmail.com Fri Jan 9 10:48:34 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 9 Jan 2015 19:48:34 +1000 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <818732810.4093303.1420793208359.JavaMail.yahoo@jws100159.mail.ne1.yahoo.com> References: <CADiSq7c0hFqm-XE7nx+r_RBLHmR-_zVinoDD6-9ja54M-UGo2w@mail.gmail.com> <818732810.4093303.1420793208359.JavaMail.yahoo@jws100159.mail.ne1.yahoo.com> Message-ID: <CADiSq7fZda_pihPgvLLfxH=tbuYiC1QD3f5A_hms9qpFDGoVoQ@mail.gmail.com> On 9 January 2015 at 18:46, Andrew Barnert <abarnert at yahoo.com> wrote: > On Thursday, January 8, 2015 11:35 PM, Nick Coghlan <ncoghlan at gmail.com> wrote: > > > >> The statement/expression distinction is deliberate, for the reasons >> you describe in the final article of your post. You will have zero >> chance of persuading anyone if you start from an assumption that the >> distinction is accidental. > > It strikes me that it might be useful if there were some good official explanation of this (maybe in the Design & History FAQ). > > Many other "modern" languages (Ruby, CoffeeScript, etc.) try to eliminate or hide the distinction as much as possible. And JavaScript documentation often apologizes for having the distinction and explains how you can "work around the problem" by defining and calling a function inline to wrap any statement. So, many people believe that the distinction is all negatives with no positives. > > I believe the distinction isn't just there to keep the parser simpler, or because Guido wasn't clever enough to figure out how to embed statements in expressions in an indentation-significant language, or because the examples of Ruby and CoffeeScript didn't exist yet and now it's too late for Python; statements are a big part of what makes Python code more readable than those languages, both in-depth and in a quick scan to see the general flow control and state changes. > > But it's hard to explain why. And, even if I manage to explain why _I_ think statements make Python more readable, that's just my opinion; it would be nice to have something that Guido endorsed. > > Also, for people who want to suggest changes to Python (or design their own languages), it would be helpful to understand how the distinction fits in with other things. For example, in a pure functional language, there may be no benefit to having statements. In a language where (almost) everything is an expression, having mutating methods return self instead of None wouldn't be nearly as much of a problem. More generally, expression-ness and not-mutating are deeply connected (e.g., comprehensions aren't designed for mutation, loop statements are), but it's not easy to say how. And so on. For myself, the analogy I try drawing these days is to suggest to people that they look at a cookbook written in English, and then try translating that cookbook into their preferred programming language in such a way that anyone who could read the original English version could still read the translated version without needing to be told "Oh, don't worry about that bit, it's just needed so the computer understands what is going on"). When a problem is amenable to being handled in that style, you're often best off coding it in that style, as that opens up future maintenance to the broadest possible base of contributors. Languages that support coding in this style are the ones typically referred to as "scripting languages", and Python is one of those languages. However, there are additional styles of thinking that folks can be trained in (like lambda calculus, object oriented programming, event driven programming, etc) and the reason those styles of thinking were created is because the cookbook style starts to encounter scalability problems when trying to handle higher levels of complexity. Many programming languages are written on the assumption that a single style of thinking should be applied to the entire problem space of an application, and then get referred to by the style of thinking they promote (so functional programming languages, object-oriented programming languages, logical assertion languages, etc). This generally limits the applicability of those languages outside the problem domains where that particular style of thinking is suitable, and also means their barriers to entry are much higher than those of the scripting languages (since you need to learn a new way of thinking first, whereas scripting languages often let you talk to the computer using the same kind of reasoning that you'd use to give detailed step-by-step instructions to another human). The design of Python takes a more pragmatic approach and recognises that different problems and situations are best handled through different kinds of thinking, and so it offers those features as optional add-ons to the cookbook approach, allowing people to use them if they make sense in their particular case, *without* making them the dominant mode of operation of the overall language. Where folks run into trouble with Python (especially when it comes to the design side of things) is in trying to ignore that overarching cookbook element, and *just* use one of the other styles of thinking. It's not designed to work that way - the "cookbook" layer defines the overall structure, and then the other aspects like the object-oriented programming support, the functional programming support and the event driven programming support all reside within that as suitable tools for building *components* of larger systems. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From toddrjen at gmail.com Fri Jan 9 11:20:32 2015 From: toddrjen at gmail.com (Todd) Date: Fri, 9 Jan 2015 11:20:32 +0100 Subject: [Python-ideas] Float range class In-Reply-To: <CAPJVwB=HXy6P7tGV_wiPo1P2Fsz1jEP5ePcECw_4LMSLQu-3hA@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <BF30F199-DFDA-4003-B160-6CFFBD0DD14D@yahoo.com> <CAFpSVpKxUbY+azZRPY+=vo3Ht8RuchruUh1AN56EzKCjcwTiiQ@mail.gmail.com> <CAPJVwB=HXy6P7tGV_wiPo1P2Fsz1jEP5ePcECw_4LMSLQu-3hA@mail.gmail.com> Message-ID: <CAFpSVpKqDNUUETWPcc7Twj_kToQVMduHPY7CsgdCEw07FTu+pw@mail.gmail.com> On Thu, Jan 8, 2015 at 7:40 PM, Nathaniel Smith <njs at pobox.com> wrote: > On Thu, Jan 8, 2015 at 5:58 PM, Todd <toddrjen at gmail.com> wrote: > > On Thu, Jan 8, 2015 at 6:41 PM, Andrew Barnert <abarnert at yahoo.com> > wrote: > >> > >> More generally: if an frange that works exactly like range but with > floats > >> (and rounding errors) is all you want, that's so trivial to write that > it > >> doesn't seem worth putting in the stdlib, unless the use case is very > >> common. > > > > In numerical computing the use-case is extremely common, in fact I would > say > > it practically universal. Ideally it would be vectorized, but in > practice > > this is often not possible, requiring the use of loops. > > Can you give a more specific example? I believe your problem is real, > but as a numerical programmer I can't think of an example of where > I've ever wanted anything like this, so it's hard for me to visualize > what exactly you're talking about. > Here is one right from the scipy cookbook: http://wiki.scipy.org/Cookbook/Theoretical_Ecology/Hastings_and_Powell But in practice, it would be any case where a range of values are needed, and you can't vectorize for whatever reason (such as having mutable values, needing to do things out of order, or relying on some arbitrary data set). Some other examples: 1. Anything done in parallel on a range of values (here is an example, although they don't give the actual algorithm http://stackoverflow.com/questions/11210677/need-help-to-parallelize-a-loop-in-python ) 2. Using individual floats as inputs to another program or library (I am doing this right now for simulation, I only use numpy because it is the only easy way to get the ranges I need) 3. Many types a stair step adaptive approaches in experiments ( http://en.wikipedia.org/wiki/Psychophysics#Staircase_procedures) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/eb7e6471/attachment.html> From toddrjen at gmail.com Fri Jan 9 11:27:17 2015 From: toddrjen at gmail.com (Todd) Date: Fri, 9 Jan 2015 11:27:17 +0100 Subject: [Python-ideas] Float range class In-Reply-To: <CALGmxEJPANRfJ5rSYWPNeCXEi7-aed6Rj6L9cuQ2TMqgPMzRcg@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <BF30F199-DFDA-4003-B160-6CFFBD0DD14D@yahoo.com> <CAFpSVpKxUbY+azZRPY+=vo3Ht8RuchruUh1AN56EzKCjcwTiiQ@mail.gmail.com> <CALGmxEJPANRfJ5rSYWPNeCXEi7-aed6Rj6L9cuQ2TMqgPMzRcg@mail.gmail.com> Message-ID: <CAFpSVp+i4E7ncfsT1fbxyopZtuBSsFeOHOKxPbJh4f3MEgOHBg@mail.gmail.com> On Thu, Jan 8, 2015 at 8:14 PM, Chris Barker <chris.barker at noaa.gov> wrote: > On Thu, Jan 8, 2015 at 9:58 AM, Todd <toddrjen at gmail.com> wrote: > >> How does using numpy floats instead of python floats affect performance, >>> unless "it's too fast" or "it uses too little memory" are problems? >>> >> >> ... > >> If you are using numpy arrays with numpy functions, it isn't an issue. >> But if you are using it in a similar manner to the existing range class, at >> some point it is likely going to interact with python native types (such as >> adding a numpy scalar to a python float). This require a type conversion, >> either python float to numpy scalar or vice versus, and that is relatively >> slow. I have profiled it, this can be the primary bottleneck in numerical >> code that can't be vectorized for whatever reason. >> > > Sure, but this is a core issue with numpy, period. There are, in fact, > some things that work faster with python native data types than numpy. But > if you are interacting with numpy arrays, it's hard for me to imagine that > this one bottleneck is going to be key. > > I think the real use case here is for code that is not using numpy at all. > > Exactly. This is a case where people use numpy just because it is the only commonly-used package that provides any sort of floating-point range, and their performance can suffer considerably because of it. I have had cases where I use numpy.linspace or numpy.arange and then convert it to python types up-front because the values are re-used many times in loops and this hurts performance. That is why I am suggesting having one that doesn't use numpy ranges. > The third approach is to create a new module for various range classes. >>> For example there could be a float range, a datetime range, a decimal >>> range, a duck-typed range, >>> >>> wouldn't a single duck-typed range cover all the bases, though perhaps > with less than optimum performance? > > If all the ranges behaved exactly the same, yes. But there might be special corner cases for particular range types that might require optional arguments or something. And having optimized versions may very well be an important enough issue on its own. Overall, though, I was just trying to make sure all alternative approaches I could think of were mentioned. There might be additional issues with the various approaches I wasn't aware of, so I thought they should all be mentioned. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/429b749e/attachment-0001.html> From toddrjen at gmail.com Fri Jan 9 11:31:05 2015 From: toddrjen at gmail.com (Todd) Date: Fri, 9 Jan 2015 11:31:05 +0100 Subject: [Python-ideas] Float range class In-Reply-To: <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> Message-ID: <CAFpSVpJhqy3EAon4OF-SV3KaaASCdWvGem=x1QJGigLLeM0JrQ@mail.gmail.com> On Fri, Jan 9, 2015 at 4:01 AM, Neil Girdhar <mistersheik at gmail.com> wrote: > I agree with everyone above. At first I was +1 on this proposal, but why > not suggest to the numpy people that arange and linespace should return > Sequence objects rather than numpy arrays? I'm pretty sure there's a way > to make sure objects work in numpy. > Because that will have much worse performance in cases where the algorithm can be vectorized. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/4f346c48/attachment.html> From mal at egenix.com Fri Jan 9 11:55:04 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 09 Jan 2015 11:55:04 +0100 Subject: [Python-ideas] Float range class In-Reply-To: <CAFpSVp+sVKBqRgsCShGKotwGdViTj2WsG0KFE2fANwEADtMH-w@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <54AEADD8.4030705@egenix.com> <CAFpSVp+sVKBqRgsCShGKotwGdViTj2WsG0KFE2fANwEADtMH-w@mail.gmail.com> Message-ID: <54AFB388.7010601@egenix.com> On 08.01.2015 18:12, Todd wrote: > On Thu, Jan 8, 2015 at 5:18 PM, M.-A. Lemburg <mal at egenix.com> wrote: >> I think you'd first have to describe some use cases, since it's >> not at all clear what you mean with "float range", e.g. you >> could be referring to: >> >> * interval arithmetics: http://en.wikipedia.org/wiki/Interval_arithmetic >> >> * ranges of numbers with a fixed stepping (which can create problems >> near the edges due to rounding issues) >> > > This is what I am talking about. The idea would be to have something that > is, as much as possible, identical to the existing range. I understand it > has floating-point issues, but any other approach would as well. This should do the trick: def frange(start, stop, steps): start = float(start) stop = float(stop) steps = int(steps) for i in range(steps + 1): yield ((steps - i) * start + i * stop) / steps It's a generator. If you need a list, wrap it into list(frange(...)). The calculation may look a little awkward. That's on purpose to reduce fp rounding errors. tmp/frange> python3.4 -i frange.py >>> list(frange(0,1,3)) [0.0, 0.3333333333333333, 0.6666666666666666, 1.0] >>> list(frange(0,1,4)) [0.0, 0.25, 0.5, 0.75, 1.0] >>> list(frange(0,1,5)) [0.0, 0.2, 0.4, 0.6, 0.8, 1.0] >>> list(frange(0,1,6)) [0.0, 0.16666666666666666, 0.3333333333333333, 0.5, 0.6666666666666666, 0.8333333333333334, 1.0] >>> Both sides of the range are included. If you need one side open, simply leave out the first or last element of the list. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 09 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From jsbueno at python.org.br Fri Jan 9 11:59:19 2015 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Fri, 9 Jan 2015 08:59:19 -0200 Subject: [Python-ideas] Float range class In-Reply-To: <54AFB388.7010601@egenix.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <54AEADD8.4030705@egenix.com> <CAFpSVp+sVKBqRgsCShGKotwGdViTj2WsG0KFE2fANwEADtMH-w@mail.gmail.com> <54AFB388.7010601@egenix.com> Message-ID: <CAH0mxTQBP4SRnFZ3TMJebKd1W9fHxJgnfn7kG8iEZPe4MkVbbw@mail.gmail.com> Since this is being carried on, I see that many of the problems resulting from a naive use of FP would not happen with Decimals - Why not either allow the frange to have it's type specified (among float/Decimal/fraction/etc...) or, to keep things simple and boring, just think about one *range function for Decimals as well? On 9 January 2015 at 08:55, M.-A. Lemburg <mal at egenix.com> wrote: > On 08.01.2015 18:12, Todd wrote: >> On Thu, Jan 8, 2015 at 5:18 PM, M.-A. Lemburg <mal at egenix.com> wrote: >>> I think you'd first have to describe some use cases, since it's >>> not at all clear what you mean with "float range", e.g. you >>> could be referring to: >>> >>> * interval arithmetics: http://en.wikipedia.org/wiki/Interval_arithmetic >>> >>> * ranges of numbers with a fixed stepping (which can create problems >>> near the edges due to rounding issues) >>> >> >> This is what I am talking about. The idea would be to have something that >> is, as much as possible, identical to the existing range. I understand it >> has floating-point issues, but any other approach would as well. > > This should do the trick: > > def frange(start, stop, steps): > start = float(start) > stop = float(stop) > steps = int(steps) > for i in range(steps + 1): > yield ((steps - i) * start + i * stop) / steps > > It's a generator. If you need a list, wrap it into list(frange(...)). > > The calculation may look a little awkward. That's on purpose to > reduce fp rounding errors. > > tmp/frange> python3.4 -i frange.py >>>> list(frange(0,1,3)) > [0.0, 0.3333333333333333, 0.6666666666666666, 1.0] >>>> list(frange(0,1,4)) > [0.0, 0.25, 0.5, 0.75, 1.0] >>>> list(frange(0,1,5)) > [0.0, 0.2, 0.4, 0.6, 0.8, 1.0] >>>> list(frange(0,1,6)) > [0.0, 0.16666666666666666, 0.3333333333333333, 0.5, 0.6666666666666666, 0.8333333333333334, 1.0] >>>> > > Both sides of the range are included. If you need one side open, simply > leave out the first or last element of the list. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Jan 09 2015) >>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From rosuav at gmail.com Fri Jan 9 12:01:54 2015 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 9 Jan 2015 22:01:54 +1100 Subject: [Python-ideas] Float range class In-Reply-To: <54AFB388.7010601@egenix.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <54AEADD8.4030705@egenix.com> <CAFpSVp+sVKBqRgsCShGKotwGdViTj2WsG0KFE2fANwEADtMH-w@mail.gmail.com> <54AFB388.7010601@egenix.com> Message-ID: <CAPTjJmpBn5NsyShHC=mfy5jnz5d3NGBK1N+Ayn2TR9AVuY651Q@mail.gmail.com> On Fri, Jan 9, 2015 at 9:55 PM, M.-A. Lemburg <mal at egenix.com> wrote: >> This is what I am talking about. The idea would be to have something that >> is, as much as possible, identical to the existing range. I understand it >> has floating-point issues, but any other approach would as well. > > This should do the trick: > > def frange(start, stop, steps): > start = float(start) > stop = float(stop) > steps = int(steps) > for i in range(steps + 1): > yield ((steps - i) * start + i * stop) / steps Thing is, a range object is a lot more than just an iterable. In many ways, it acts just like list(self) would - you can index it, slice it, etc: >>> r = range(10, 101, 10) >>> list(r) [10, 20, 30, 40, 50, 60, 70, 80, 90, 100] >>> r.index(70) 6 >>> r[6] 70 >>> len(r) 10 >>> r[1:-4:2] range(20, 70, 20) Does frange() need to be able to do all these operations? ChrisA From toddrjen at gmail.com Fri Jan 9 12:09:14 2015 From: toddrjen at gmail.com (Todd) Date: Fri, 9 Jan 2015 12:09:14 +0100 Subject: [Python-ideas] Float range class In-Reply-To: <CAH0mxTQBP4SRnFZ3TMJebKd1W9fHxJgnfn7kG8iEZPe4MkVbbw@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <54AEADD8.4030705@egenix.com> <CAFpSVp+sVKBqRgsCShGKotwGdViTj2WsG0KFE2fANwEADtMH-w@mail.gmail.com> <54AFB388.7010601@egenix.com> <CAH0mxTQBP4SRnFZ3TMJebKd1W9fHxJgnfn7kG8iEZPe4MkVbbw@mail.gmail.com> Message-ID: <CAFpSVpKsB9uVg7d0jyzEuR++tH51EDbfKM4+Ax4iJLZ5OQNeqg@mail.gmail.com> On Fri, Jan 9, 2015 at 11:59 AM, Joao S. O. Bueno <jsbueno at python.org.br> wrote: > Since this is being carried on, I see that many of the problems > resulting from a naive use of FP would not happen with > Decimals - > Why not either allow the frange to have it's type specified > (among float/Decimal/fraction/etc...) or, to keep things simple and boring, > just think about one *range function for Decimals as well? > > That was the idea of the "rangetools" module. There would be various range types for various data types. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/e62e28c1/attachment.html> From mal at egenix.com Fri Jan 9 12:28:13 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 09 Jan 2015 12:28:13 +0100 Subject: [Python-ideas] Float range class In-Reply-To: <CAPTjJmpBn5NsyShHC=mfy5jnz5d3NGBK1N+Ayn2TR9AVuY651Q@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <54AEADD8.4030705@egenix.com> <CAFpSVp+sVKBqRgsCShGKotwGdViTj2WsG0KFE2fANwEADtMH-w@mail.gmail.com> <54AFB388.7010601@egenix.com> <CAPTjJmpBn5NsyShHC=mfy5jnz5d3NGBK1N+Ayn2TR9AVuY651Q@mail.gmail.com> Message-ID: <54AFBB4D.2010702@egenix.com> On 09.01.2015 12:01, Chris Angelico wrote: > On Fri, Jan 9, 2015 at 9:55 PM, M.-A. Lemburg <mal at egenix.com> wrote: >>> This is what I am talking about. The idea would be to have something that >>> is, as much as possible, identical to the existing range. I understand it >>> has floating-point issues, but any other approach would as well. >> >> This should do the trick: >> >> def frange(start, stop, steps): >> start = float(start) >> stop = float(stop) >> steps = int(steps) >> for i in range(steps + 1): >> yield ((steps - i) * start + i * stop) / steps > > Thing is, a range object is a lot more than just an iterable. In many > ways, it acts just like list(self) would - you can index it, slice it, > etc: > >>>> r = range(10, 101, 10) >>>> list(r) > [10, 20, 30, 40, 50, 60, 70, 80, 90, 100] >>>> r.index(70) > 6 >>>> r[6] > 70 >>>> len(r) > 10 >>>> r[1:-4:2] > range(20, 70, 20) > > Does frange() need to be able to do all these operations? Probably 95% of all uses of range() in my own code never use range object features beyond that of being iterable. For everything else you can create either a list or throw itertools at it :-) IMO, the only reason for having an frange() function in (probably) the math module is to provide an implementation which tries to reduce fp rounding problems to a minimum. Other than that, it's just too simple to add straight to the application code in a way which exactly fits the purpose and functionality needed by that application. For decimals and date/time values, the naive algorithm is good enough, so the above argument doesn't apply to those. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 09 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From python at 2sn.net Fri Jan 9 12:31:54 2015 From: python at 2sn.net (Alexander Heger) Date: Fri, 9 Jan 2015 22:31:54 +1100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <20150107232304.GM19387@ando.pearwood.info> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> Message-ID: <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> >> IEEE754 requires that all comparisons with NaN give false, i.e., > > Not quite. "Not equal" comparisons should return true. yes. Sorry for the omission. > NANs should compare unordered and unequal to everything including > themselves. correct. From rosuav at gmail.com Fri Jan 9 12:33:31 2015 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 9 Jan 2015 22:33:31 +1100 Subject: [Python-ideas] Float range class In-Reply-To: <54AFBB4D.2010702@egenix.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <54AEADD8.4030705@egenix.com> <CAFpSVp+sVKBqRgsCShGKotwGdViTj2WsG0KFE2fANwEADtMH-w@mail.gmail.com> <54AFB388.7010601@egenix.com> <CAPTjJmpBn5NsyShHC=mfy5jnz5d3NGBK1N+Ayn2TR9AVuY651Q@mail.gmail.com> <54AFBB4D.2010702@egenix.com> Message-ID: <CAPTjJmr9g+kbtPorTKHfnMaf2WduvvvPmFqSpKbOhUo7R5LqLw@mail.gmail.com> On Fri, Jan 9, 2015 at 10:28 PM, M.-A. Lemburg <mal at egenix.com> wrote: > Probably 95% of all uses of range() in my own code never use range > object features beyond that of being iterable. For everything > else you can create either a list or throw itertools at it :-) > > IMO, the only reason for having an frange() function in (probably) > the math module is to provide an implementation which tries to > reduce fp rounding problems to a minimum. Sure, and that's doubtless very useful. But that can't be described as "something that is, as much as possible, identical to the existing range". It's a very specific subset - one that happens to be the most-used aspect of range(), but a subset nonetheless. ChrisA From toddrjen at gmail.com Fri Jan 9 12:40:27 2015 From: toddrjen at gmail.com (Todd) Date: Fri, 9 Jan 2015 12:40:27 +0100 Subject: [Python-ideas] Float range class In-Reply-To: <54AFBB4D.2010702@egenix.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <54AEADD8.4030705@egenix.com> <CAFpSVp+sVKBqRgsCShGKotwGdViTj2WsG0KFE2fANwEADtMH-w@mail.gmail.com> <54AFB388.7010601@egenix.com> <CAPTjJmpBn5NsyShHC=mfy5jnz5d3NGBK1N+Ayn2TR9AVuY651Q@mail.gmail.com> <54AFBB4D.2010702@egenix.com> Message-ID: <CAFpSVpLrNb7mxVfvowcH9MbUX-k_HYo6SNgYrh3Ngjg4eBTp7A@mail.gmail.com> On Fri, Jan 9, 2015 at 12:28 PM, M.-A. Lemburg <mal at egenix.com> wrote: > On 09.01.2015 12:01, Chris Angelico wrote: > > On Fri, Jan 9, 2015 at 9:55 PM, M.-A. Lemburg <mal at egenix.com> wrote: > >>> This is what I am talking about. The idea would be to have something > that > >>> is, as much as possible, identical to the existing range. I > understand it > >>> has floating-point issues, but any other approach would as well. > >> > >> This should do the trick: > >> > >> def frange(start, stop, steps): > >> start = float(start) > >> stop = float(stop) > >> steps = int(steps) > >> for i in range(steps + 1): > >> yield ((steps - i) * start + i * stop) / steps > > > > Thing is, a range object is a lot more than just an iterable. In many > > ways, it acts just like list(self) would - you can index it, slice it, > > etc: > > > >>>> r = range(10, 101, 10) > >>>> list(r) > > [10, 20, 30, 40, 50, 60, 70, 80, 90, 100] > >>>> r.index(70) > > 6 > >>>> r[6] > > 70 > >>>> len(r) > > 10 > >>>> r[1:-4:2] > > range(20, 70, 20) > > > > Does frange() need to be able to do all these operations? > <snip> > IMO, the only reason for having an frange() function in (probably) > the math module is to provide an implementation which tries to > reduce fp rounding problems to a minimum. > > Other than that, it's just too simple to add straight to the > application code in a way which exactly fits the purpose and > functionality needed by that application. > > I think the wide usage of numpy.arange and numpy.linspace is evidence against that conclusion. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/f8afe1fc/attachment-0001.html> From python at 2sn.net Fri Jan 9 12:58:19 2015 From: python at 2sn.net (Alexander Heger) Date: Fri, 9 Jan 2015 22:58:19 +1100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> Message-ID: <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> here another odd issue related to NaN arithmetics: (1) >>> nan = float('nan') >>> nan is nan True >>> nan == nan False this is what we want. Good. But (2) >>> x = (nan,) >>> x is x True >>> x == x True ... hmm ... apparently the comparison uses ID before using == ? for tuple elements? on the other hand (3) >>> (nan,) is (nan,) False >>> (nan,) == (nan,) True I guess we see from (4) >>> float('nan') is float('nan') False >>> (float('nan'),) == (float('nan'),) False I understand why they all give the results they do, however, I think the second comparison in (2) should give False and, more importantly, the first comparison of (4) should be True - maths.nan should be a singleton - even if we allow to float.make_nan() to produce NaNs with different payloads - but maths.nan could be just one thing, one instance, the numerical counterpart to None. -Alexander From mal at egenix.com Fri Jan 9 12:58:55 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 09 Jan 2015 12:58:55 +0100 Subject: [Python-ideas] Float range class In-Reply-To: <CAFpSVpLrNb7mxVfvowcH9MbUX-k_HYo6SNgYrh3Ngjg4eBTp7A@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <54AEADD8.4030705@egenix.com> <CAFpSVp+sVKBqRgsCShGKotwGdViTj2WsG0KFE2fANwEADtMH-w@mail.gmail.com> <54AFB388.7010601@egenix.com> <CAPTjJmpBn5NsyShHC=mfy5jnz5d3NGBK1N+Ayn2TR9AVuY651Q@mail.gmail.com> <54AFBB4D.2010702@egenix.com> <CAFpSVpLrNb7mxVfvowcH9MbUX-k_HYo6SNgYrh3Ngjg4eBTp7A@mail.gmail.com> Message-ID: <54AFC27F.2080309@egenix.com> On 09.01.2015 12:40, Todd wrote: > On Fri, Jan 9, 2015 at 12:28 PM, M.-A. Lemburg <mal at egenix.com> wrote: > >> On 09.01.2015 12:01, Chris Angelico wrote: >>> On Fri, Jan 9, 2015 at 9:55 PM, M.-A. Lemburg <mal at egenix.com> wrote: >>>>> This is what I am talking about. The idea would be to have something >> that >>>>> is, as much as possible, identical to the existing range. I >> understand it >>>>> has floating-point issues, but any other approach would as well. >>>> >>>> This should do the trick: >>>> >>>> def frange(start, stop, steps): >>>> start = float(start) >>>> stop = float(stop) >>>> steps = int(steps) >>>> for i in range(steps + 1): >>>> yield ((steps - i) * start + i * stop) / steps >>> >>> Thing is, a range object is a lot more than just an iterable. In many >>> ways, it acts just like list(self) would - you can index it, slice it, >>> etc: >>> >>>>>> r = range(10, 101, 10) >>>>>> list(r) >>> [10, 20, 30, 40, 50, 60, 70, 80, 90, 100] >>>>>> r.index(70) >>> 6 >>>>>> r[6] >>> 70 >>>>>> len(r) >>> 10 >>>>>> r[1:-4:2] >>> range(20, 70, 20) >>> >>> Does frange() need to be able to do all these operations? >> <snip> >> IMO, the only reason for having an frange() function in (probably) >> the math module is to provide an implementation which tries to >> reduce fp rounding problems to a minimum. >> >> Other than that, it's just too simple to add straight to the >> application code in a way which exactly fits the purpose and >> functionality needed by that application. >> >> > I think the wide usage of numpy.arange and numpy.linspace is evidence > against that conclusion. Well, we're talking about the stdlib here. numpy is used in a specific application space where such questions arise often. The scope for the stdlib is different. It's meant to cover features needed by a very wide range of users in a wide variety of application spaces. Having a stdlib functions which returns evenly spaced floats is useful for people, but more on the grounds that the implementation can be tricky to get right. The above is just a first stab at it. I'm sure Mark Dickinson and Tim Peters will have better ideas on how to reduce fp rounding issues some more. In general, you always have to find a compromise between usefulness of new features in the stdlib and the maintenance effort introduced by them. If there's a very active maintainer, the barrier for entry is lower. If people just submit a patch and then leave the code unmaintained, things are different. That is to say: if you'd volunteer to maintain a rangetools module for at least the next 5 years, chances are higher this will get accepted. See Raymond's itertools and collections as good examples. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 09 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From rosuav at gmail.com Fri Jan 9 13:12:26 2015 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 9 Jan 2015 23:12:26 +1100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> Message-ID: <CAPTjJmqD=wf9TSHv1N5MWKUUTkyFM0OSsu-DHX7B-f-BURRgrQ@mail.gmail.com> On Fri, Jan 9, 2015 at 10:58 PM, Alexander Heger <python at 2sn.net> wrote: > here another odd issue related to NaN arithmetics: > > (1) >>>> nan = float('nan') >>>> nan is nan > True >>>> nan == nan > False > > this is what we want. Good. But > > (2) >>>> x = (nan,) >>>> x is x > True >>>> x == x > True > > ... hmm ... apparently the comparison uses ID before using == ? for > tuple elements? It doesn't even compare them. Consider: >>> class Different: ... def __eq__(self, other): ... print("Comparing",id(self),"with",id(other)) ... return False ... >>> x = Different() >>> x == x Comparing 139945362930712 with 139945362930712 False >>> tup = x, >>> tup == tup True A tuple is equal to itself, for efficiency's sake, and doesn't go in and check that all its elements are equal to themselves. > on the other hand > > (3) >>>> (nan,) is (nan,) > False >>>> (nan,) == (nan,) > True This, however, shows your point better. Likewise, using my Different class shows that the comparison isn't made: >>> (x,) is (x,) False >>> (x,) == (x,) True > I guess we see from > > (4) >>>> float('nan') is float('nan') > False >>>> (float('nan'),) == (float('nan'),) > False Right; they're different objects, so the short-cut isn't taken. It's a shortcut that works safely for every sane object except NaN. You won't find many non-buggy objects that aren't equal to themselves. > I understand why they all give the results they do, however, I think > the second comparison in (2) should give False and, more importantly, > the first comparison of (4) should be True - maths.nan should be a > singleton - even if we allow to float.make_nan() to produce NaNs with > different payloads - but maths.nan could be just one thing, one > instance, the numerical counterpart to None. Why should math.nan be a singleton? Consider: >>> float("1.0") is float("1.0") False >>> (float("1.0"),) == (float("1.0"),) True Even with integers, although CPython caches some small ones: >>> int("12345") is int("12345") False >>> (int("12345"),) == (int("12345"),) True (And CPython also caches literals, so both these tests have to use explicit constructor calls.) I think we can all agree that the number 12345 is unique - there aren't two such numbers. So if we're allowed to have two int objects representing the exact same number, why should NaN (which represents the vastly infinite space of non-numbers) be unique? ChrisA From njs at pobox.com Fri Jan 9 13:20:12 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 9 Jan 2015 12:20:12 +0000 Subject: [Python-ideas] Float range class In-Reply-To: <CAFpSVpKqDNUUETWPcc7Twj_kToQVMduHPY7CsgdCEw07FTu+pw@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <BF30F199-DFDA-4003-B160-6CFFBD0DD14D@yahoo.com> <CAFpSVpKxUbY+azZRPY+=vo3Ht8RuchruUh1AN56EzKCjcwTiiQ@mail.gmail.com> <CAPJVwB=HXy6P7tGV_wiPo1P2Fsz1jEP5ePcECw_4LMSLQu-3hA@mail.gmail.com> <CAFpSVpKqDNUUETWPcc7Twj_kToQVMduHPY7CsgdCEw07FTu+pw@mail.gmail.com> Message-ID: <CAPJVwBnORCXOgFaXiZ8E9qtnxQ=+SndLF2xNPtRHC9JcKyFKQQ@mail.gmail.com> On 9 Jan 2015 10:21, "Todd" <toddrjen at gmail.com> wrote: > > On Thu, Jan 8, 2015 at 7:40 PM, Nathaniel Smith <njs at pobox.com> wrote: >> >> On Thu, Jan 8, 2015 at 5:58 PM, Todd <toddrjen at gmail.com> wrote: >> > On Thu, Jan 8, 2015 at 6:41 PM, Andrew Barnert <abarnert at yahoo.com> wrote: >> >> >> >> More generally: if an frange that works exactly like range but with floats >> >> (and rounding errors) is all you want, that's so trivial to write that it >> >> doesn't seem worth putting in the stdlib, unless the use case is very >> >> common. >> > >> > In numerical computing the use-case is extremely common, in fact I would say >> > it practically universal. Ideally it would be vectorized, but in practice >> > this is often not possible, requiring the use of loops. >> >> Can you give a more specific example? I believe your problem is real, >> but as a numerical programmer I can't think of an example of where >> I've ever wanted anything like this, so it's hard for me to visualize >> what exactly you're talking about. > > > Here is one right from the scipy cookbook: > http://wiki.scipy.org/Cookbook/Theoretical_Ecology/Hastings_and_Powell > But in practice, it would be any case where a range of values are needed, and you can't vectorize for whatever reason (such as having mutable values, needing to do things out of order, or relying on some arbitrary data set). Perhaps I'm being dense, but I can't see how a lazy frange object would help the linked-to code? The only place I can see that looks relevant is where it does enumerate (np.linspace(...)). But AFAICT this neither requires range semantics nor lazy view machinery, which I thought were the two main things you're arguing for. It's also very unlikely that materializing the linspace array here is anywhere close to being a bottleneck. (It calls odeint on every iteration!) So it's a theoretical inefficiency, but we have to accept a lot of those if we want to program in python, and focus our efforts on the ones that really matter. > Some other examples: > 1. Anything done in parallel on a range of values (here is an example, although they don't give the actual algorithm http://stackoverflow.com/questions/11210677/need-help-to-parallelize-a-loop-in-python ) This example also seems to just be iterating over the return value from linspace, and performing very slow computations for each entry? So same comments as above. > 2. Using individual floats as inputs to another program or library (I am doing this right now for simulation, I only use numpy because it is the only easy way to get the ranges I need) This sounds similar to the previous two cases too. > 3. Many types a stair step adaptive approaches in experiments ( http://en.wikipedia.org/wiki/Psychophysics#Staircase_procedures) I've never implemented one of these so I can't really comment, but in this case don't you have to get input from a human on every pass through the loop? Hardly a case where the numpy overhead is going to be crucial, right? Again, I'm not trying to nitpick your examples, I'm just trying to get a concrete idea of what parts of your proposal provide what benefits. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/efcc5cd9/attachment-0001.html> From python at 2sn.net Fri Jan 9 13:25:07 2015 From: python at 2sn.net (Alexander Heger) Date: Fri, 9 Jan 2015 23:25:07 +1100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <1556539310.4083422.1420781894547.JavaMail.yahoo@jws100143.mail.ne1.yahoo.com> References: <20150107160925.GH19387@ando.pearwood.info> <1556539310.4083422.1420781894547.JavaMail.yahoo@jws100143.mail.ne1.yahoo.com> Message-ID: <CAN3CYHw3xZ05MJ0MZ67snH5MkeyLy1MzFgt_inXmoc1DTbDZgg@mail.gmail.com> > def make_nan(payload=0, sign=False, quiet=True): > payload |= (0x7ff0000000000000 | > (sign * 0x8000000000000000) | > (quiet * 0x0008000000000000)) > > return struct.unpack('d', struct.pack('Q', payload))[0] > > And likewise, you could write a `parse_nan` that returns `payload, sign, quiet`. And you'd probably want to add some parameter validation (and don't forget that payload 0 and quiet-bit 0 means it's not a NaN, it's an inf). so, payload default should not be 0 but, e.g., 1. From ncoghlan at gmail.com Fri Jan 9 13:34:46 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 9 Jan 2015 22:34:46 +1000 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> Message-ID: <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> On 9 January 2015 at 21:58, Alexander Heger <python at 2sn.net> wrote: > here another odd issue related to NaN arithmetics: > (2) >>>> x = (nan,) >>>> x is x > True >>>> x == x > True > > ... hmm ... apparently the comparison uses ID before using == ? for > tuple elements? > I understand why they all give the results they do, however, I think > the second comparison in (2) should give False There's an inherent conflict between the mathematical notion of NaN and the software engineering notion of object identity. For containers in Python, it's the latter that prevails, in order to preserve the invariant that "x is y" implies "x in (y,)" for arbitrary containers and objects, even if x and y happen to be distinct references to the exact same NaN object. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From python at 2sn.net Fri Jan 9 14:03:47 2015 From: python at 2sn.net (Alexander Heger) Date: Sat, 10 Jan 2015 00:03:47 +1100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> Message-ID: <CAN3CYHwixhXhsqtxxWtjuQP6=dWDh9hKgJ4uUxkq93jM8JV=rg@mail.gmail.com> > There's an inherent conflict between the mathematical notion of NaN > and the software engineering notion of object identity. For containers > in Python, it's the latter that prevails, in order to preserve the > invariant that "x is y" implies "x in (y,)" for arbitrary containers > and objects, even if x and y happen to be distinct references to the > exact same NaN object. I suppose in the end it is not advisable to use "is" to compare numerical variables in the first place as you can run into the cached values, >>> x = 256 >>> y = 256 >>> x is y True >>> x = 257 >>> y = 257 >>> x is y False In most cases of numerical applications people will use numpy and that behaves the expected way, the issue is if a python list behaves different from a numpy array. >>> import numpy as np >>> x = np.tile(np.nan, 3) >>> x == x array([False, False, False], dtype=bool) >>> x is x True >>> x = [float('nan')] * 3 >>> x == x True >>> x is x True -Alexander From python at 2sn.net Fri Jan 9 14:05:44 2015 From: python at 2sn.net (Alexander Heger) Date: Sat, 10 Jan 2015 00:05:44 +1100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAPTjJmqD=wf9TSHv1N5MWKUUTkyFM0OSsu-DHX7B-f-BURRgrQ@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> <CAPTjJmqD=wf9TSHv1N5MWKUUTkyFM0OSsu-DHX7B-f-BURRgrQ@mail.gmail.com> Message-ID: <CAN3CYHw+-ifrwY8dnwFwotjiZ6efsPwYtT9oGmFSweXwcZ89qQ@mail.gmail.com> Dear Chris, thanks for answering my questions. >> I understand why they all give the results they do, however, I think >> the second comparison in (2) should give False and, more importantly, >> the first comparison of (4) should be True - maths.nan should be a >> singleton - even if we allow to float.make_nan() to produce NaNs with >> different payloads - but maths.nan could be just one thing, one >> instance, the numerical counterpart to None. > > Why should math.nan be a singleton? Consider: > >>>> float("1.0") is float("1.0") > False >>>> (float("1.0"),) == (float("1.0"),) > True > > Even with integers, although CPython caches some small ones: > >>>> int("12345") is int("12345") > False >>>> (int("12345"),) == (int("12345"),) > True > > (And CPython also caches literals, so both these tests have to use > explicit constructor calls.) > > I think we can all agree that the number 12345 is unique - there > aren't two such numbers. So if we're allowed to have two int objects > representing the exact same number, why should NaN (which represents > the vastly infinite space of non-numbers) be unique? I think I was not thinking clearly. Here, if float('nan') would return math.nan it would it would, by definition, always be the same object instead of creating new ones (with different payloads). An issue may be if the value comes from elsewhere, e.g., was pickled. I am now wondering what about float("-nan")? Do you want to introduce a syntax for signaling nans and such with payload and signs float("-nans123")? -Alexander From abarnert at yahoo.com Fri Jan 9 16:32:52 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 9 Jan 2015 07:32:52 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAN3CYHw3xZ05MJ0MZ67snH5MkeyLy1MzFgt_inXmoc1DTbDZgg@mail.gmail.com> References: <20150107160925.GH19387@ando.pearwood.info> <1556539310.4083422.1420781894547.JavaMail.yahoo@jws100143.mail.ne1.yahoo.com> <CAN3CYHw3xZ05MJ0MZ67snH5MkeyLy1MzFgt_inXmoc1DTbDZgg@mail.gmail.com> Message-ID: <A7F0A109-927E-4ED2-AF70-6ED020354AE3@yahoo.com> On Jan 9, 2015, at 4:25, Alexander Heger <python at 2sn.net> wrote: >> def make_nan(payload=0, sign=False, quiet=True): >> payload |= (0x7ff0000000000000 | >> (sign * 0x8000000000000000) | >> (quiet * 0x0008000000000000)) >> >> return struct.unpack('d', struct.pack('Q', payload))[0] >> >> And likewise, you could write a `parse_nan` that returns `payload, sign, quiet`. And you'd probably want to add some parameter validation (and don't forget that payload 0 and quiet-bit 0 means it's not a NaN, it's an inf). > > so, payload default should not be 0 but, e.g., 1. I don't think so. The NaN you get from float("nan") on most platforms is positive, quiet, payload 0 (matching the defaults above). The NaN you get from np.nan is usually either the same, or the negative version of it. Also, any library that defines meanings for NaN payloads is presumably going to define one for 1 (I'm pretty sure SANE did), and it's unlikely to be anything generic. And on platforms where the native C lib displays detailed NaN information, quiet 0 will just be NaN or qNaN or similar, but quiet 1 will be NaN1 or qNaN(1) or similar. So, 1 is not a good default. Maybe the obvious answer is to not provide a default value at all; if you want quiet 0 you have to make_nan(0), not just make_nan, and that will raise if you happen to be on an old reversed-quiet-bit platform, while if you want default NaN you just use float.nan instead of float.make_nan. You could even to do as SANE did (or at least as I remember it) and not allow 0 at all; if you want quiet 0 you use float.nan. Or, alternatively, make the default None, which uses the right default payload for your platform. Speaking of which, IIRC, the default payload on MIPS machines (which had reversed quiet bit) was -1. Which implies that it might be handy to take negative payloads and complement off 1<<51, because if anyone has any use for such payloads they may be thinking of them as negatives, not as huge positives. But anyway, I think all of this shows why we probably don't want to add make_nan today, but how easily we (or even a third-party lib) can add it later, once someone has a platform/library that they need it for so we can paint the bikeshed to match the house. And my original point: adding float.nan to 3.5 doesn't prevent a more detailed NaN API in the future. From guido at python.org Fri Jan 9 18:10:28 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Jan 2015 09:10:28 -0800 Subject: [Python-ideas] Need help converting HTML doc to ReST Message-ID: <CAP7+vJ+qgD_gTeosZKcNjYDwkMp58wVQzrFgPfU2tqwzmP0GsQ@mail.gmail.com> I'd like to convert my document "The Theory of TypeHinting" to ReST so I can include it in PEP 483. Can someone help me with the conversion? The source is https://quip.com/r69HA9GhGa7J and the destination is http://legacy.python.org/dev/peps/pep-0483/ (or, more correctly, https://hg.python.org/peps/file/7cbb166b30fd/pep-0483.txt). Thanks!! -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/fafaea81/attachment-0001.html> From chris.barker at noaa.gov Fri Jan 9 18:21:25 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 9 Jan 2015 09:21:25 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> Message-ID: <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> On Thu, Jan 8, 2015 at 7:33 PM, Nathaniel Smith <njs at pobox.com> wrote: > On 9 Jan 2015 03:02, "Neil Girdhar" <mistersheik at gmail.com> wrote: > > > > I agree with everyone above. At first I was +1 on this proposal, but > why not suggest to the numpy people that arange and linespace should return > Sequence objects rather than numpy arrays? > numpy arrays are a python sequence -- what do you mean here? Did you mean iterator? This whole conversation made me think a bit about numpy and iterators -- python has been moving toward more use of iterators, maybe numpy should do the same? But as I think about, it's actually a totally different model -- py3 made range() an iterator, because it is most often used that way -- i.e. in a for loop or list comporehension or genator expression. when you really want the sequence, you wrap it in a list(). But numpy is, at it's core, about arrays, and the iteration happens INSIDE the array object. e.g. to multiply all the elements of an array by a number you do: new_arr = arr * x then the actual looping(iteration) happens inside numpy, with C data types, at C speed. turning that into: new_arr = [i*x for i in arr] would push all the work back out into python, killing the point of numpy. In fact, killing BOTH points of numpy: 1) performance 2) clean readable array expressions -- that is: c = np.sqrt(a**2 + b**2) rather than: c = [ math.sqrt(x) for x in (x+y for x, y in zip( (x**2 for x in a), (x**2 for x in b) ) ) ] OK, there may be a more readable way to write that... Anyway, numpy is about arrays, so linspace, arange, etc create arrays. There may well be a good reason to make a numpy iterator version of these, for when you HAVE to loop, but that wouldn't help folks that aren't using numpy anyway. but sequences (iterators) of ranges of floating point numbers (and other "numeric-like" objects) is a generally useful thing for all users of python, and not entirely trivial to do right -- hence this conversation. BTW, here is an example of the OP's point about the performance hit of pulling numpy integares (or floats) out of numpy array objects: using np.arange() as the source in a list comp: In [39]: timeit a = [x**2 for x in np.arange(1000)] 1000 loops, best of 3: 907 ?s per loop compared to python xrange() - sorry, on py2 herre... In [45]: In [40]: timeit a = [x**2 for x in xrange(1000)] 10000 loops, best of 3: 74.9 ?s per loop factor of 12 or so. but really, for the most part, if you are using numpy, you're going to use numpy: In [43]: timeit a = np.arange(1000)**2 100000 loops, best of 3: 3.99 ?s per loop another factor of 18... So again, we don't ned this because numpy linspace and arrage are "slow" but because it's useful when you are not using numpy. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/b42e6790/attachment.html> From rosuav at gmail.com Fri Jan 9 18:24:48 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 10 Jan 2015 04:24:48 +1100 Subject: [Python-ideas] Need help converting HTML doc to ReST In-Reply-To: <CAP7+vJ+qgD_gTeosZKcNjYDwkMp58wVQzrFgPfU2tqwzmP0GsQ@mail.gmail.com> References: <CAP7+vJ+qgD_gTeosZKcNjYDwkMp58wVQzrFgPfU2tqwzmP0GsQ@mail.gmail.com> Message-ID: <CAPTjJmpDKkFxAPeF7wmMs17K1erdpkJNa444k6GesMG3zQ1--A@mail.gmail.com> On Sat, Jan 10, 2015 at 4:10 AM, Guido van Rossum <guido at python.org> wrote: > I'd like to convert my document "The Theory of TypeHinting" to ReST so I can > include it in PEP 483. Can someone help me with the conversion? The source > is https://quip.com/r69HA9GhGa7J and the destination is > http://legacy.python.org/dev/peps/pep-0483/ (or, more correctly, > https://hg.python.org/peps/file/7cbb166b30fd/pep-0483.txt). > > Thanks!! The Debian repositories include a utility called pandoc which can do that conversion. I don't know if it's perfect or not, but it certainly looks pretty reasonable. Footnotes need to be transformed a bit, which can be done manually. Shall I put the content straight into the peps repo? ChrisA From rosuav at gmail.com Fri Jan 9 18:26:31 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 10 Jan 2015 04:26:31 +1100 Subject: [Python-ideas] Float range class In-Reply-To: <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> Message-ID: <CAPTjJmphDandSE1vBx4BQHf2HjQNNH2DDdE5q=UbRCaC93DpLg@mail.gmail.com> On Sat, Jan 10, 2015 at 4:21 AM, Chris Barker <chris.barker at noaa.gov> wrote: > numpy arrays are a python sequence -- what do you mean here? Did you mean > iterator? It's a specific protocol: https://docs.python.org/3/glossary.html#term-sequence ChrisA From donald at stufft.io Fri Jan 9 18:33:50 2015 From: donald at stufft.io (Donald Stufft) Date: Fri, 9 Jan 2015 12:33:50 -0500 Subject: [Python-ideas] Need help converting HTML doc to ReST In-Reply-To: <CAP7+vJ+qgD_gTeosZKcNjYDwkMp58wVQzrFgPfU2tqwzmP0GsQ@mail.gmail.com> References: <CAP7+vJ+qgD_gTeosZKcNjYDwkMp58wVQzrFgPfU2tqwzmP0GsQ@mail.gmail.com> Message-ID: <723A22AB-85AD-4CF5-B833-D95554B97B98@stufft.io> > On Jan 9, 2015, at 12:10 PM, Guido van Rossum <guido at python.org> wrote: > > I'd like to convert my document "The Theory of TypeHinting" to ReST so I can include it in PEP 483. Can someone help me with the conversion? The source is https://quip.com/r69HA9GhGa7J <https://quip.com/r69HA9GhGa7J> and the destination is http://legacy.python.org/dev/peps/pep-0483/ <http://legacy.python.org/dev/peps/pep-0483/> (or, more correctly, https://hg.python.org/peps/file/7cbb166b30fd/pep-0483.txt <https://hg.python.org/peps/file/7cbb166b30fd/pep-0483.txt>). > > Thanks!! > I ran this through pandoc real quick and cleaned it up a bit: https://gist.github.com/dstufft/d4fc00108b323ffa8034 <https://gist.github.com/dstufft/d4fc00108b323ffa8034>. I didn?t read through it or anything though just ran pandoc on it and killed the leftover raw html tidbits. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/6951a7be/attachment-0001.html> From guido at python.org Fri Jan 9 18:40:41 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Jan 2015 09:40:41 -0800 Subject: [Python-ideas] Need help converting HTML doc to ReST In-Reply-To: <723A22AB-85AD-4CF5-B833-D95554B97B98@stufft.io> References: <CAP7+vJ+qgD_gTeosZKcNjYDwkMp58wVQzrFgPfU2tqwzmP0GsQ@mail.gmail.com> <723A22AB-85AD-4CF5-B833-D95554B97B98@stufft.io> Message-ID: <CAP7+vJL0HqMKvnOk+qbpnyUoC0Kw4_s_4+_T3duju6g2xvt9Pw@mail.gmail.com> Thanks! I'll copy that in the PEP and make sure it generates correctly. On Fri, Jan 9, 2015 at 9:33 AM, Donald Stufft <donald at stufft.io> wrote: > > On Jan 9, 2015, at 12:10 PM, Guido van Rossum <guido at python.org> wrote: > > I'd like to convert my document "The Theory of TypeHinting" to ReST so I > can include it in PEP 483. Can someone help me with the conversion? The > source is https://quip.com/r69HA9GhGa7J and the destination is > http://legacy.python.org/dev/peps/pep-0483/ (or, more correctly, > https://hg.python.org/peps/file/7cbb166b30fd/pep-0483.txt). > > Thanks!! > > > I ran this through pandoc real quick and cleaned it up a bit: > https://gist.github.com/dstufft/d4fc00108b323ffa8034. I didn?t read > through it or anything though just ran pandoc on it and killed the leftover > raw html tidbits. > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/0df8b7db/attachment.html> From guido at python.org Fri Jan 9 18:46:17 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Jan 2015 09:46:17 -0800 Subject: [Python-ideas] Need help converting HTML doc to ReST In-Reply-To: <CAPTjJmpDKkFxAPeF7wmMs17K1erdpkJNa444k6GesMG3zQ1--A@mail.gmail.com> References: <CAP7+vJ+qgD_gTeosZKcNjYDwkMp58wVQzrFgPfU2tqwzmP0GsQ@mail.gmail.com> <CAPTjJmpDKkFxAPeF7wmMs17K1erdpkJNa444k6GesMG3zQ1--A@mail.gmail.com> Message-ID: <CAP7+vJ+GVjU2qJfU85My5_KCxgH6oEByZmpLi-z-iuWBXB5xaA@mail.gmail.com> Chris, would you be so kind? I tried to work from Donald's gist, but there are some funny characters in it. I'd also like to use the Open Publication License (see PEP 1) rather than placing it in the public domain. You can go straight to the peps repo, I can fix things up later once I have time to review it. Thank you so much!! On Fri, Jan 9, 2015 at 9:24 AM, Chris Angelico <rosuav at gmail.com> wrote: > On Sat, Jan 10, 2015 at 4:10 AM, Guido van Rossum <guido at python.org> > wrote: > > I'd like to convert my document "The Theory of TypeHinting" to ReST so I > can > > include it in PEP 483. Can someone help me with the conversion? The > source > > is https://quip.com/r69HA9GhGa7J and the destination is > > http://legacy.python.org/dev/peps/pep-0483/ (or, more correctly, > > https://hg.python.org/peps/file/7cbb166b30fd/pep-0483.txt). > > > > Thanks!! > > The Debian repositories include a utility called pandoc which can do > that conversion. I don't know if it's perfect or not, but it certainly > looks pretty reasonable. Footnotes need to be transformed a bit, which > can be done manually. Shall I put the content straight into the peps > repo? > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/acf08bf4/attachment.html> From rosuav at gmail.com Fri Jan 9 18:54:07 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 10 Jan 2015 04:54:07 +1100 Subject: [Python-ideas] Need help converting HTML doc to ReST In-Reply-To: <CAP7+vJ+GVjU2qJfU85My5_KCxgH6oEByZmpLi-z-iuWBXB5xaA@mail.gmail.com> References: <CAP7+vJ+qgD_gTeosZKcNjYDwkMp58wVQzrFgPfU2tqwzmP0GsQ@mail.gmail.com> <CAPTjJmpDKkFxAPeF7wmMs17K1erdpkJNa444k6GesMG3zQ1--A@mail.gmail.com> <CAP7+vJ+GVjU2qJfU85My5_KCxgH6oEByZmpLi-z-iuWBXB5xaA@mail.gmail.com> Message-ID: <CAPTjJmohYE5bPmTbivx1+Va6gTJ5OHYp8wQOy57zmB3nZ+oY4g@mail.gmail.com> On Sat, Jan 10, 2015 at 4:46 AM, Guido van Rossum <guido at python.org> wrote: > Chris, would you be so kind? I tried to work from Donald's gist, but there > are some funny characters in it. > > I'd also like to use the Open Publication License (see PEP 1) rather than > placing it in the public domain. > > You can go straight to the peps repo, I can fix things up later once I have > time to review it. > > Thank you so much!! No probs, will do. Not sure what the "funny characters" are; I can see a U+201C and U+201D double quote pair, possibly there's an encoding problem. I'll make sure it's saved as UTF-8. ChrisA From rosuav at gmail.com Fri Jan 9 19:30:02 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 10 Jan 2015 05:30:02 +1100 Subject: [Python-ideas] Need help converting HTML doc to ReST In-Reply-To: <CAP7+vJ+GVjU2qJfU85My5_KCxgH6oEByZmpLi-z-iuWBXB5xaA@mail.gmail.com> References: <CAP7+vJ+qgD_gTeosZKcNjYDwkMp58wVQzrFgPfU2tqwzmP0GsQ@mail.gmail.com> <CAPTjJmpDKkFxAPeF7wmMs17K1erdpkJNa444k6GesMG3zQ1--A@mail.gmail.com> <CAP7+vJ+GVjU2qJfU85My5_KCxgH6oEByZmpLi-z-iuWBXB5xaA@mail.gmail.com> Message-ID: <CAPTjJmq2NbOBaxu66zdnq=aa1e8WmzrHUDHezx=C3QwzcyxWbg@mail.gmail.com> On Sat, Jan 10, 2015 at 4:46 AM, Guido van Rossum <guido at python.org> wrote: > Chris, would you be so kind? I tried to work from Donald's gist, but there > are some funny characters in it. > > I'd also like to use the Open Publication License (see PEP 1) rather than > placing it in the public domain. > > You can go straight to the peps repo, I can fix things up later once I have > time to review it. > > Thank you so much!! Text imported. It builds cleanly, and it looks roughly like the original page, but I suspect there's some places where other forms of formatting will be better suited (eg nested lists are used to indent code blocks). I've left the license as it is (Creative Commons) for the moment; does "like to use" mean that you would if you could, or you are hereby relicensing it? Quotes, dashes, etc have all been ASCIIfied. ChrisA From guido at python.org Fri Jan 9 19:35:20 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Jan 2015 10:35:20 -0800 Subject: [Python-ideas] Need help converting HTML doc to ReST In-Reply-To: <CAPTjJmq2NbOBaxu66zdnq=aa1e8WmzrHUDHezx=C3QwzcyxWbg@mail.gmail.com> References: <CAP7+vJ+qgD_gTeosZKcNjYDwkMp58wVQzrFgPfU2tqwzmP0GsQ@mail.gmail.com> <CAPTjJmpDKkFxAPeF7wmMs17K1erdpkJNa444k6GesMG3zQ1--A@mail.gmail.com> <CAP7+vJ+GVjU2qJfU85My5_KCxgH6oEByZmpLi-z-iuWBXB5xaA@mail.gmail.com> <CAPTjJmq2NbOBaxu66zdnq=aa1e8WmzrHUDHezx=C3QwzcyxWbg@mail.gmail.com> Message-ID: <CAP7+vJJ-FFhMGmGownF82qFK_tJndmf7-uY_3D-X12JQD7T0VQ@mail.gmail.com> Thank you so much!! It is already live on legacy.python.org/dev/peps/ (www seems to lag behind or require a manual kick). In PEP 1 there are only two options for PEP copyrights: public domain or open publication license. For this particular document I'd rather not do public domain (I've already received a request for republication) so I'll convert it to Open Publication License. On Fri, Jan 9, 2015 at 10:30 AM, Chris Angelico <rosuav at gmail.com> wrote: > On Sat, Jan 10, 2015 at 4:46 AM, Guido van Rossum <guido at python.org> > wrote: > > Chris, would you be so kind? I tried to work from Donald's gist, but > there > > are some funny characters in it. > > > > I'd also like to use the Open Publication License (see PEP 1) rather than > > placing it in the public domain. > > > > You can go straight to the peps repo, I can fix things up later once I > have > > time to review it. > > > > Thank you so much!! > > Text imported. It builds cleanly, and it looks roughly like the > original page, but I suspect there's some places where other forms of > formatting will be better suited (eg nested lists are used to indent > code blocks). I've left the license as it is (Creative Commons) for > the moment; does "like to use" mean that you would if you could, or > you are hereby relicensing it? > > Quotes, dashes, etc have all been ASCIIfied. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/699ae2fb/attachment-0001.html> From rosuav at gmail.com Fri Jan 9 19:41:26 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 10 Jan 2015 05:41:26 +1100 Subject: [Python-ideas] Need help converting HTML doc to ReST In-Reply-To: <CAP7+vJJ-FFhMGmGownF82qFK_tJndmf7-uY_3D-X12JQD7T0VQ@mail.gmail.com> References: <CAP7+vJ+qgD_gTeosZKcNjYDwkMp58wVQzrFgPfU2tqwzmP0GsQ@mail.gmail.com> <CAPTjJmpDKkFxAPeF7wmMs17K1erdpkJNa444k6GesMG3zQ1--A@mail.gmail.com> <CAP7+vJ+GVjU2qJfU85My5_KCxgH6oEByZmpLi-z-iuWBXB5xaA@mail.gmail.com> <CAPTjJmq2NbOBaxu66zdnq=aa1e8WmzrHUDHezx=C3QwzcyxWbg@mail.gmail.com> <CAP7+vJJ-FFhMGmGownF82qFK_tJndmf7-uY_3D-X12JQD7T0VQ@mail.gmail.com> Message-ID: <CAPTjJmp-99aaJENiYe+2ug3DQLwoLUiNpKBK_yeyVYXqCx5TUg@mail.gmail.com> On Sat, Jan 10, 2015 at 5:35 AM, Guido van Rossum <guido at python.org> wrote: > Thank you so much!! It is already live on legacy.python.org/dev/peps/ (www > seems to lag behind or require a manual kick). > > In PEP 1 there are only two options for PEP copyrights: public domain or > open publication license. For this particular document I'd rather not do > public domain (I've already received a request for republication) so I'll > convert it to Open Publication License. Done. To make sure I got the text and formatting correct, I went looking for any other RST PEP using that license... and there is exactly one (plus one more in text/plain). So the copyright info is lifted from PEP 437. :) ChrisA From guido at python.org Fri Jan 9 19:58:46 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Jan 2015 10:58:46 -0800 Subject: [Python-ideas] Need help converting HTML doc to ReST In-Reply-To: <CAPTjJmp-99aaJENiYe+2ug3DQLwoLUiNpKBK_yeyVYXqCx5TUg@mail.gmail.com> References: <CAP7+vJ+qgD_gTeosZKcNjYDwkMp58wVQzrFgPfU2tqwzmP0GsQ@mail.gmail.com> <CAPTjJmpDKkFxAPeF7wmMs17K1erdpkJNa444k6GesMG3zQ1--A@mail.gmail.com> <CAP7+vJ+GVjU2qJfU85My5_KCxgH6oEByZmpLi-z-iuWBXB5xaA@mail.gmail.com> <CAPTjJmq2NbOBaxu66zdnq=aa1e8WmzrHUDHezx=C3QwzcyxWbg@mail.gmail.com> <CAP7+vJJ-FFhMGmGownF82qFK_tJndmf7-uY_3D-X12JQD7T0VQ@mail.gmail.com> <CAPTjJmp-99aaJENiYe+2ug3DQLwoLUiNpKBK_yeyVYXqCx5TUg@mail.gmail.com> Message-ID: <CAP7+vJJpJErkMYyZw=d-aYEEVujOcyuH64m2xdNuQj_7R6uupA@mail.gmail.com> Sounds fine. I've made a few superficial changes -- thanks for getting this out so quickly! Now it's on to PEP 484, which will lay out the actual specification of the typing module and address many details. This may take another week (but folks interested in the progress can peek at the github repo referenced from the PEP 484 stub :-). On Fri, Jan 9, 2015 at 10:41 AM, Chris Angelico <rosuav at gmail.com> wrote: > On Sat, Jan 10, 2015 at 5:35 AM, Guido van Rossum <guido at python.org> > wrote: > > Thank you so much!! It is already live on legacy.python.org/dev/peps/ > (www > > seems to lag behind or require a manual kick). > > > > In PEP 1 there are only two options for PEP copyrights: public domain or > > open publication license. For this particular document I'd rather not do > > public domain (I've already received a request for republication) so I'll > > convert it to Open Publication License. > > Done. To make sure I got the text and formatting correct, I went > looking for any other RST PEP using that license... and there is > exactly one (plus one more in text/plain). So the copyright info is > lifted from PEP 437. :) > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/15c257f1/attachment.html> From mistersheik at gmail.com Fri Jan 9 21:37:41 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Fri, 9 Jan 2015 15:37:41 -0500 Subject: [Python-ideas] Float range class In-Reply-To: <CAPTjJmphDandSE1vBx4BQHf2HjQNNH2DDdE5q=UbRCaC93DpLg@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> <CAPTjJmphDandSE1vBx4BQHf2HjQNNH2DDdE5q=UbRCaC93DpLg@mail.gmail.com> Message-ID: <CAA68w_n7BkZWOzxU0idCMzK=M6LVLf-O6PEBUZWVZq8ELRsDPg@mail.gmail.com> I mean a class that implements the Sequence protocol lazily ? just like range. It should use multiplication to avoid precision errors. On Fri, Jan 9, 2015 at 12:26 PM, Chris Angelico <rosuav at gmail.com> wrote: > On Sat, Jan 10, 2015 at 4:21 AM, Chris Barker <chris.barker at noaa.gov> > wrote: > > numpy arrays are a python sequence -- what do you mean here? Did you mean > > iterator? > > It's a specific protocol: > > https://docs.python.org/3/glossary.html#term-sequence > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/H69Z7qPe0Dg/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/39673ab7/attachment.html> From abarnert at yahoo.com Fri Jan 9 21:58:43 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 9 Jan 2015 12:58:43 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> Message-ID: <0D487B6F-26E0-4A11-AD59-1982365FFC97@yahoo.com> On Jan 9, 2015, at 9:21, Chris Barker <chris.barker at noaa.gov> wrote: > On Thu, Jan 8, 2015 at 7:33 PM, Nathaniel Smith <njs at pobox.com> wrote: >> On 9 Jan 2015 03:02, "Neil Girdhar" <mistersheik at gmail.com> wrote: >> > >> > I agree with everyone above. At first I was +1 on this proposal, but why not suggest to the numpy people that arange and linespace should return Sequence objects rather than numpy arrays? >> > numpy arrays are a python sequence -- what do you mean here? Did you mean iterator? > > This whole conversation made me think a bit about numpy and iterators -- python has been moving toward more use of iterators, maybe numpy should do the same? > > But as I think about, it's actually a totally different model -- py3 made range() an iterator, because it is most often used that way -- i.e. in a for loop or list comporehension or genator expression. when you really want the sequence, you wrap it in a list(). No, py3 made range a lazy sequence-like view, not an iterator. It has indexing, __contains__, etc. People make the same mistake about dictionary views. dict.keys and friends return sequence-like or set-like views, not iterators. (That was backported to 2.7 as dict.viewkeys and friends, but people using 2.x still don't seem to know it exists.) There's no reason numpy _couldn't_ add various kinds of lazy views. It already has arrays that borrow storage from slices of other arrays, and things like scipy,sparse that generate the values on the fly. You could easily write a range class that generates values on the fly; you could even create lazy arrays by arithmetic (an a**2 that didn't store anything, but referenced a and generated each value on demand by squaring the value from a). The reason numpy doesn't do this is that it's usually significantly faster to precompute everything, keeping each inner loop as small as possible, and keeping each function dealing only with one kind of thing (so there's no need for costly branching anywhere), despite the waste of storage (and potential cache and VM effects). Not _always_ faster, so there might be some benefits from adding laziness all over the place, but presumably often enough that's it hasn't been worth anyone's effort to add it and see. Also, the waste of storage usually comes down to peak memory being at worst 2x what you'd want (e.g., in newarr = arr * x, presumably each of the three has the same size and dtype, so having x by lazy would only reduce peak memory by 1/3rd); it's not like in Python iterator code, where you might have no in-memory sequences at all, and adding one for x might increase your storage 500000x (and, more to the point, make it linear instead of constant). > But numpy is, at it's core, about arrays, and the iteration happens INSIDE the array object. e.g. to multiply all the elements of an array by a number you do: > > new_arr = arr * x > > then the actual looping(iteration) happens inside numpy, with C data types, at C speed. > > turning that into: > > new_arr = [i*x for i in arr] > > would push all the work back out into python, killing the point of numpy. Right, but there's no reason numpy couldn't implement an arr * x for a special lazy view type x. Using Python iterators would almost certainly not work, because accessing each value is done by calling a Python function, and they can only be accessed in order, and so on. But if x were a custom type whose values could be generated random-access by a native numpy expression, that wouldn't be a problem. Again, the problem is that someone would have to write the code to implement that type, to multiply that type with itself or with an array, etc. Unless there's a clear win, nobody's going to write all that. > In fact, killing BOTH points of numpy: > > 1) performance > > 2) clean readable array expressions -- that is: > > c = np.sqrt(a**2 + b**2) > > rather than: > > c = [ math.sqrt(x) for x in (x+y for x, y in zip( (x**2 for x in a), (x**2 for x in b) ) ) ] > > OK, there may be a more readable way to write that... > > Anyway, numpy is about arrays, so linspace, arange, etc create arrays. > > There may well be a good reason to make a numpy iterator version of these, for when you HAVE to loop, but that wouldn't help folks that aren't using numpy anyway. > > but sequences (iterators) of ranges of floating point numbers (and other "numeric-like" objects) is a generally useful thing for all users of python, and not entirely trivial to do right -- hence this conversation. Again, this is misleading, and missed an important point: arange is actually very easy to get right, but hard to _use_ properly. A half-open range of values from 0 to .9 by .3 is going to include a number just under 0.9 if properly implemented. However you slice it, .3*3<.9, .3+.3+.3<.9, etc., so that number belongs in the range. The problem is on the user's side, not the implementation's--using arange is generally effectively the same mistake as using ==. You can use arange safely if you think through the FP issues properly (e.g., if you know that epsilon<<step, you can pass stop-epsilon instead of stop), but it's up to the user to do that thinking. linspace is harder to get right, but very easy to use properly. There's no room for rounding issues to affect things on the user side; the worst that happens is that all your values are off by up to the input error e, which is generally acceptable (as opposed to having the wrong number of values, which rarely is). And here, the implementation matters; a naive implementation will return values off by more than e. So, having linspace in the stdlib sounds good--not just to avoid people implementing it wrong, but to avoid people implementing and/or using arange when they should be using linspace. In fact, maybe even make range's TypeError message reference linspace when given floats, as sum does for str.join? In the less common cases where arange really was what you wanted, and you know how to use it, you almost certainly can implement it yourself. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/c43ded08/attachment-0001.html> From ethan at stoneleaf.us Fri Jan 9 22:22:42 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 09 Jan 2015 13:22:42 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <0D487B6F-26E0-4A11-AD59-1982365FFC97@yahoo.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> <0D487B6F-26E0-4A11-AD59-1982365FFC97@yahoo.com> Message-ID: <54B046A2.9020309@stoneleaf.us> On 01/09/2015 12:58 PM, Andrew Barnert wrote: > arange is actually very easy to get right, but hard to _use_ properly. A half-open range of values from 0 to .9 by .3 > is going to include a number just under 0.9 if properly implemented. However you slice it, .3*3<.9, .3+.3+.3<.9, etc., > so that number belongs in the range. As a non-maths person (no degree, passingly familiar with algebra, light trigonometry, what little I remember from calculus) can you explain that to me? Why does a number slightly less than .9 /have/ to be in the results for a "properly implemented" frange? def frange(start, stop, step): count, remainder = divmod(stop - start, step) count = int(count) if remainder > 0.00000001: count += 1 for i in range(count): yield start + i * step list(frange(0.3, 0.9, 0.3)) [0.3, 0.6] list(frange(0.3, 1, 0.3)) [0.3, 0.6, 0.89999999999999] -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/1600d778/attachment.sig> From guido at python.org Fri Jan 9 22:24:50 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Jan 2015 13:24:50 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <0D487B6F-26E0-4A11-AD59-1982365FFC97@yahoo.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> <0D487B6F-26E0-4A11-AD59-1982365FFC97@yahoo.com> Message-ID: <CAP7+vJJ0P4cKFf2sCThWCL_e=CJaMxWt0tK+wpcHdmux0eNvkw@mail.gmail.com> On Fri, Jan 9, 2015 at 12:58 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > [...] > arange is actually very easy to get right, but hard to _use_ properly. A > half-open range of values from 0 to .9 by .3 is going to include a number > just under 0.9 if properly implemented. However you slice it, .3*3<.9, > .3+.3+.3<.9, etc., so that number belongs in the range. [...] > I don't know if this some kind of fp in-joke that I'm missing, but I find that the wording "if properly implemented" comes on a little strong. It depends on behavior of IEEE binary fp that's far from obvious -- I observe that .3*4 > 1.2, for example. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/4fcd1118/attachment.html> From abarnert at yahoo.com Fri Jan 9 23:06:40 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 9 Jan 2015 14:06:40 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <CAP7+vJJ0P4cKFf2sCThWCL_e=CJaMxWt0tK+wpcHdmux0eNvkw@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> <0D487B6F-26E0-4A11-AD59-1982365FFC97@yahoo.com> <CAP7+vJJ0P4cKFf2sCThWCL_e=CJaMxWt0tK+wpcHdmux0eNvkw@mail.gmail.com> Message-ID: <5797D06D-AE5A-486B-962C-1B37EFDDF9CA@yahoo.com> On Jan 9, 2015, at 13:24, Guido van Rossum <guido at python.org> wrote: > On Fri, Jan 9, 2015 at 12:58 PM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: >> [...] >> arange is actually very easy to get right, but hard to _use_ properly. A half-open range of values from 0 to .9 by .3 is going to include a number just under 0.9 if properly implemented. However you slice it, .3*3<.9, .3+.3+.3<.9, etc., so that number belongs in the range. [...] > > I don't know if this some kind of fp in-joke that I'm missing, but I find that the wording "if properly implemented" comes on a little strong. It depends on behavior of IEEE binary fp that's far from obvious -- I observe that .3*4 > 1.2, for example. OK, you're right; that isn't really stated well. If properly implemented _with IEEE binary doubles_, .3*3 < .9. If properly implemented with a _different_ inexact float representation, that one might be fine--and a different one might be trouble. But regardless, the mistake is not in the implementation of arange, it's in the user's assumption that .3*3 == .9 (or that 0+.3+.3+.3 == .9, or whatever, depending on how the stop was chosen). That may be true or false depending on your FP implementation, and therefore it's a bad assumption. And the fact that the == is inside your brain rather than in your code makes the red flag harder to spot. Of course you can solve this the same way you solve any FP comparison issue: by picking an appropriate epsilon. For example, given that e << step, you can just count to .6+.15 or .9-.15 and be absolutely sure that something close to .6 will appear but something close to .9 will not. (And similarly for counting backward to 1.2 by .3.) But rethinking your problem in terms of linspace is much harder to get wrong and requires much less understanding of how FP works. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/2f712343/attachment.html> From abarnert at yahoo.com Fri Jan 9 23:17:20 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 9 Jan 2015 14:17:20 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <54B046A2.9020309@stoneleaf.us> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> <0D487B6F-26E0-4A11-AD59-1982365FFC97@yahoo.com> <54B046A2.9020309@stoneleaf.us> Message-ID: <F246BB7A-9D1E-460B-849D-DAD38ED8A2AF@yahoo.com> On Jan 9, 2015, at 13:22, Ethan Furman <ethan at stoneleaf.us> wrote: > On 01/09/2015 12:58 PM, Andrew Barnert wrote: > >> arange is actually very easy to get right, but hard to _use_ properly. A half-open range of values from 0 to .9 by .3 >> is going to include a number just under 0.9 if properly implemented. However you slice it, .3*3<.9, .3+.3+.3<.9, etc., >> so that number belongs in the range. > > As a non-maths person (no degree, passingly familiar with algebra, light trigonometry, what little I remember from > calculus) can you explain that to me? Why does a number slightly less than .9 /have/ to be in the results for a > "properly implemented" frange? > > def frange(start, stop, step): > count, remainder = divmod(stop - start, step) > count = int(count) > if remainder > 0.00000001: > count += 1 > for i in range(count): > yield start + i * step Try it with step=1e-12. This is equivalent on designing float.__eq__ as something like abs(self-other) < 0.00000001. At first glance it seems like that makes all floating point comparison problems go away, until you try it with small floats and suddenly 9e-12 - 1e-12 == 4e-12. The right epsilon to use depends on the specific problem space, so it can't be hardcoded into a general-purpose function. From ethan at stoneleaf.us Fri Jan 9 23:34:11 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 09 Jan 2015 14:34:11 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <F246BB7A-9D1E-460B-849D-DAD38ED8A2AF@yahoo.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> <0D487B6F-26E0-4A11-AD59-1982365FFC97@yahoo.com> <54B046A2.9020309@stoneleaf.us> <F246BB7A-9D1E-460B-849D-DAD38ED8A2AF@yahoo.com> Message-ID: <54B05763.9070105@stoneleaf.us> On 01/09/2015 02:17 PM, Andrew Barnert wrote: > The right epsilon to use depends on the specific problem space, so it can't be hardcoded into a general-purpose function. Which is why if this goes forward we need to be able to specify the epsilon, possibly with a default epsilon (unless that's just a bad idea). -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/d6d522f2/attachment.sig> From guido at python.org Fri Jan 9 23:36:59 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Jan 2015 14:36:59 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <5797D06D-AE5A-486B-962C-1B37EFDDF9CA@yahoo.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> <0D487B6F-26E0-4A11-AD59-1982365FFC97@yahoo.com> <CAP7+vJJ0P4cKFf2sCThWCL_e=CJaMxWt0tK+wpcHdmux0eNvkw@mail.gmail.com> <5797D06D-AE5A-486B-962C-1B37EFDDF9CA@yahoo.com> Message-ID: <CAP7+vJ+R-+fzOg26yZWnaTEmWU7QWVQ31D_AMWvVi0BC9LEaXA@mail.gmail.com> On Fri, Jan 9, 2015 at 2:06 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > On Jan 9, 2015, at 13:24, Guido van Rossum <guido at python.org> wrote: > > On Fri, Jan 9, 2015 at 12:58 PM, Andrew Barnert < > abarnert at yahoo.com.dmarc.invalid> wrote: > >> [...] >> arange is actually very easy to get right, but hard to _use_ properly. A >> half-open range of values from 0 to .9 by .3 is going to include a number >> just under 0.9 if properly implemented. However you slice it, .3*3<.9, >> .3+.3+.3<.9, etc., so that number belongs in the range. [...] >> > > I don't know if this some kind of fp in-joke that I'm missing, but I find > that the wording "if properly implemented" comes on a little strong. It > depends on behavior of IEEE binary fp that's far from obvious -- I observe > that .3*4 > 1.2, for example. > > > OK, you're right; that isn't really stated well. > > If properly implemented _with IEEE binary doubles_, .3*3 < .9. If properly > implemented with a _different_ inexact float representation, that one might > be fine--and a different one might be trouble. > > But regardless, the mistake is not in the implementation of arange, it's > in the user's assumption that .3*3 == .9 (or that 0+.3+.3+.3 == .9, or > whatever, depending on how the stop was chosen). That may be true or false > depending on your FP implementation, and therefore it's a bad > assumption. And the fact that the == is inside your brain rather than in > your code makes the red flag harder to spot. > > Of course you can solve this the same way you solve any FP comparison > issue: by picking an appropriate epsilon. For example, given that e << > step, you can just count to .6+.15 or .9-.15 and be absolutely sure that > something close to .6 will appear but something close to .9 will not. (And > similarly for counting backward to 1.2 by .3.) > > But rethinking your problem in terms of linspace is much harder to get > wrong and requires much less understanding of how FP works. > This is a beautiful illustration of how hard interface design is. Python's use of half-open intervals for range() works really well in combination with slicing (there's a Dijkstra note about this somewhere <http://www.cs.utexas.edu/users/EWD/ewd08xx/EWD831.PDF>). But extending it into the world of floating point, it suddenly becomes a liability, and the correct response is not to patch things up with epsilons but to redefine the interface to talk about dividing a given interval up in a given number of evenly-sized sub-intervals. (Although the actual signature of linspace gives me a headache. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/71ca303a/attachment.html> From chris.barker at noaa.gov Fri Jan 9 23:37:56 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 9 Jan 2015 14:37:56 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <F246BB7A-9D1E-460B-849D-DAD38ED8A2AF@yahoo.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> <0D487B6F-26E0-4A11-AD59-1982365FFC97@yahoo.com> <54B046A2.9020309@stoneleaf.us> <F246BB7A-9D1E-460B-849D-DAD38ED8A2AF@yahoo.com> Message-ID: <CALGmxEJtJRRz_sPZQPnGdGCxt0V9rpfw_E60s4z6oU9zPSj3iw@mail.gmail.com> On Fri, Jan 9, 2015 at 2:17 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > This is equivalent on designing float.__eq__ as something like > abs(self-other) < 0.00000001. At first glance it seems like that makes all > floating point comparison problems go away, until you try it with small > floats and suddenly 9e-12 - 1e-12 == 4e-12. > > The right epsilon to use depends on the specific problem space, so it > can't be hardcoded into a general-purpose function. You could scale the epsilon with step, and get closer to general purpose. but then you STILL might not get exactly what was expected. Making an fp range an attractive nuisance. And I still think most people actually want linspace(), anyway... Note that a closed interval version with a specified delta could be handy, but not hard to just compute the number of elements if you want. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/a25bcbb5/attachment.html> From chris.barker at noaa.gov Fri Jan 9 23:39:30 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 9 Jan 2015 14:39:30 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <CAP7+vJJ0P4cKFf2sCThWCL_e=CJaMxWt0tK+wpcHdmux0eNvkw@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> <0D487B6F-26E0-4A11-AD59-1982365FFC97@yahoo.com> <CAP7+vJJ0P4cKFf2sCThWCL_e=CJaMxWt0tK+wpcHdmux0eNvkw@mail.gmail.com> Message-ID: <CALGmxEJE90ReEZMODzAkk0mRoYizSMMp+U2xCOiB2=7Xz5ZJzQ@mail.gmail.com> On Fri, Jan 9, 2015 at 1:24 PM, Guido van Rossum <guido at python.org> wrote: > On Fri, Jan 9, 2015 at 12:58 PM, Andrew Barnert < > abarnert at yahoo.com.dmarc.invalid> wrote: > >> [...] >> arange is actually very easy to get right, but hard to _use_ properly. A >> half-open range of values from 0 to .9 by .3 is going to include a number >> just under 0.9 if properly implemented. However you slice it, .3*3<.9, >> .3+.3+.3<.9, etc., so that number belongs in the range. [...] >> > > I don't know if this some kind of fp in-joke that I'm missing, but I find > that the wording "if properly implemented" comes on a little strong. It > depends on behavior of IEEE binary fp that's far from obvious -- I observe > that .3*4 > 1.2, for example. > That's the entire point here. See Warren's examples from the other day. And I went and got all sidetracked, so I think my key point was lost, and I also think I agree with Andrew, despite my notes being apparently unclear. So I'm going to try to make a couple clear, simple suggestions: It would be good to have something that provides ranges of floating point numbers (and maybe other number-like things, like rational, and Decimals, and datetimes) But we should NOT have a floating point range(), rather a floating point linspace, or similar. Andrew wrote: > arange is actually very easy to get right, but hard to _use_ properly. agreed -- I wasn't being clear, but that's what I meant -- my point was that EITHER a range() for FP OR a self-implemented version is all too likely to get the a different result than the user has in mind. whether than measn you implimented if poorly or used it poorly is kin dof irelevant, but your way of expressing it makes it more clear why we dont' want the "attractive nuisance" of an fp range(). > linspace is harder to get right, but very easy to use properly. yes, exactly. Also, we need to keep in mind common use cases -- range() is exactly the right thing for teh common use cases of compatibility with sequence indexing -- which is irrelvent to FP use. Either the user cares about the delta, or the number of elements, and they either care about the end point or not. range() is oriented to caring about the number elements, secondarily about the delta, thirdly about the about the end point. and these all can be inferred from each-other pretty easily with integers. But the end points and number of elements can not be as easily inferred for FP. linspace() is oriented to caring primarily about the end points, secondarily about the number of elements, only thirdly about the delta. I would argue that this use case is more common for FP use, AND that inferring the delta from linspace is less error prone than inferring the end points from range(), and thus linspace is the "right" thing to implement for FP. And probably for the other non-integer types as well. So, having linspace in the stdlib sounds good--not just to avoid people > implementing it wrong, but to avoid people implementing and/or using arange > when they should be using linspace. > yes -- so much better said! > In fact, maybe even make range's TypeError message reference linspace when > given floats, as sum does for str.join? > In the less common cases where arange really was what you wanted, and you > know how to use it, you almost certainly can implement it yourself. I like that too. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/bef58bb7/attachment-0001.html> From yawar.amin at gmail.com Sat Jan 10 01:48:57 2015 From: yawar.amin at gmail.com (Yawar Amin) Date: Fri, 09 Jan 2015 19:48:57 -0500 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <F86DAA65-1E17-4BDE-9D21-D54AF9567857@yahoo.com> References: <54AA093A.2050508@gmail.com> <3002533B-C171-49E0-B3DA-CFB08345D370@yahoo.com> <54AB4178.3050508@gmail.com> <97D50920-B7D2-4125-9389-224C4CBC83CA@yahoo.com> <54AC952E.5060301@gmail.com> <74164843-0774-441F-A898-4776E15EFC8D@yahoo.com> <8C6A6BAA-197B-4796-9163-6EE3AE3E1553@gmail.com> <54ADD7C9.9030709@gmail.com> <F86DAA65-1E17-4BDE-9D21-D54AF9567857@yahoo.com> Message-ID: <54B076F9.6060308@gmail.com> Hi Andrew, On 2015-01-08 10:32, Andrew Barnert wrote: > [...] > Haskell's do statement has a nice way out of that (if you really want > to turn off Pythonistas by using the word "monad").... > Another trick that might be worth playing with explicit native globals > and locals for your functions (e.g., by constructing a new > types.FunctionType),... Thanks for the ideas. I have to admit I didn't really think deeply about most of them, but the above two resonated and gave me a couple of ideas myself. I was able to refine my continuations idea, using monadic sequencing of effects and defining a new function-like type, so that at this point I can do this: actions = { "hello": print_("Hello, World!", ?: import_("math", ? m: print_("The area of my circle is:", ?: print_(m.pi * 5 * 5)))), "goodbye": print_("Goodbye, Cruel World!", ?: try_(?: 1 / 0, ?: print_("Danger, Will Robinson!"))) } actions["hello"]() actions["goodbye"]() For now, I'm satisfied that this is the best I can do towards resolving the duality of statements and expressions. As you and others have pointed out, this duality is intentional and is meant to guide the programmer into a certain mindset, which is why fighting against it is so hard. But often when you're just trying to express the ideas in your head, sometimes you just _have_ to get something out in a certain way. So I think there's utility in my approach. As you say, this isn't really a relevant 'Python idea' for this mailing list, so this discussion isn't really going to go anywhere. Anyway, thanks for your input, it's appreciated. Regards, Yawar -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 834 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/c2173ce9/attachment.sig> From abarnert at yahoo.com Sat Jan 10 04:29:08 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 9 Jan 2015 19:29:08 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <CAP7+vJ+R-+fzOg26yZWnaTEmWU7QWVQ31D_AMWvVi0BC9LEaXA@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> <0D487B6F-26E0-4A11-AD59-1982365FFC97@yahoo.com> <CAP7+vJJ0P4cKFf2sCThWCL_e=CJaMxWt0tK+wpcHdmux0eNvkw@mail.gmail.com> <5797D06D-AE5A-486B-962C-1B37EFDDF9CA@yahoo.com> <CAP7+vJ+R-+fzOg26yZWnaTEmWU7QWVQ31D_AMWvVi0BC9LEaXA@mail.gmail.com> Message-ID: <01DACEF7-CA59-48AB-9DE5-C3A4ADE69DE6@yahoo.com> On Jan 9, 2015, at 14:36, Guido van Rossum <guido at python.org> wrote: > (Although the actual signature of linspace gives me a headache. :-) I'm assuming you're not talking about the rested and dtype args (which the stdlib wouldn't need), but rather the fact that endpoint changes the meaning of num (you effectively generate num+1 points and discard the last, instead of generating num points). I've stumbled over that before, and helped other people with it. But the other way around would be confusing just as often. And the confusion wouldn't be flagged as quickly as it is in numpy (broadcasting mismatched arrays is an error; zipping mismatched sequences is not). And of course it would be a gratuitous incompatibility between the stdlib and numpy. All of this implies that there's probably some better way to specify half-open ranges for linspace that nobody has thought of yet. Hopefully someone can think of it now. :) Or, failing that, maybe we just don't need half-open linspace in the stdlib. You can always do linspace(0, 1, 5+1)[:-1] or, if you want the other, linspace(0, 1, 5)[:-1]. Either way, what you're asking for is explicit and obvious. And if linspace is a sequence-like view akin to range, this doesn't cost anything. And it's even briefer than using the endpoint argument (especially since in real life you almost always use it as a keyword arg). Since Chris seems to have a better intuition than me on how you're usually thinking when you need linspace, hopefully he can chime in on whether this would be intuitive enough, or if we need an actual solution. From guido at python.org Sat Jan 10 06:13:17 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Jan 2015 21:13:17 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <01DACEF7-CA59-48AB-9DE5-C3A4ADE69DE6@yahoo.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> <0D487B6F-26E0-4A11-AD59-1982365FFC97@yahoo.com> <CAP7+vJJ0P4cKFf2sCThWCL_e=CJaMxWt0tK+wpcHdmux0eNvkw@mail.gmail.com> <5797D06D-AE5A-486B-962C-1B37EFDDF9CA@yahoo.com> <CAP7+vJ+R-+fzOg26yZWnaTEmWU7QWVQ31D_AMWvVi0BC9LEaXA@mail.gmail.com> <01DACEF7-CA59-48AB-9DE5-C3A4ADE69DE6@yahoo.com> Message-ID: <CAP7+vJLK3t=otHBpcWwz2JBy49f=y_dK_qzZYwwv-qNpinnFBQ@mail.gmail.com> On Fri, Jan 9, 2015 at 7:29 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > On Jan 9, 2015, at 14:36, Guido van Rossum <guido at python.org> wrote: > > > (Although the actual signature of linspace gives me a headache. :-) > > I'm assuming you're not talking about the rested and dtype args (which the > stdlib wouldn't need), but rather the fact that endpoint changes the > meaning of num (you effectively generate num+1 points and discard the last, > instead of generating num points). > All of the above, plus the default to n=50. :-) Oh, and the name is odd-sounding for someone not from your world. > I've stumbled over that before, and helped other people with it. But the > other way around would be confusing just as often. And the confusion > wouldn't be flagged as quickly as it is in numpy (broadcasting mismatched > arrays is an error; zipping mismatched sequences is not). And of course it > would be a gratuitous incompatibility between the stdlib and numpy. > > All of this implies that there's probably some better way to specify > half-open ranges for linspace that nobody has thought of yet. Hopefully > someone can think of it now. :) > > Or, failing that, maybe we just don't need half-open linspace in the > stdlib. You can always do linspace(0, 1, 5+1)[:-1] or, if you want the > other, linspace(0, 1, 5)[:-1]. Either way, what you're asking for is > explicit and obvious. And if linspace is a sequence-like view akin to > range, this doesn't cost anything. And it's even briefer than using the > endpoint argument (especially since in real life you almost always use it > as a keyword arg). > > Since Chris seems to have a better intuition than me on how you're usually > thinking when you need linspace, hopefully he can chime in on whether this > would be intuitive enough, or if we need an actual solution. I'm guessing that in terms of implementation you might start with this: def linspace(start, stop, num): return [(stop*i + start*(num-i)) / num for i in range(num+1)] and then refine as follows: - make it a lazy sequence, with slicing ability, etc. - numeric tricks to avoid intermediate overflows or numeric instabilities, plus optimizations - while still returning exactly the endpoints at the extremes(*) - and better error checking (num must be an integer > 0) - and support for start/stop being other numeric types (e.g. complex, Fraction, Decimal) Then bikeshed about the module it should live in. At least you can shut down the bikeshed about the name with "that's what numpy calls it", and ditto about the argument order. Just don't add any of the other features of the numpy version. :-) (*) I was going to give the equally naive definition [start + i*step for i in range(num+1)] where step is (stop-start)/num, but that would give away my naivet? even quicker. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/33271f20/attachment.html> From abarnert at yahoo.com Sat Jan 10 08:03:59 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 9 Jan 2015 23:03:59 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <CAP7+vJLK3t=otHBpcWwz2JBy49f=y_dK_qzZYwwv-qNpinnFBQ@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> <0D487B6F-26E0-4A11-AD59-1982365FFC97@yahoo.com> <CAP7+vJJ0P4cKFf2sCThWCL_e=CJaMxWt0tK+wpcHdmux0eNvkw@mail.gmail.com> <5797D06D-AE5A-486B-962C-1B37EFDDF9CA@yahoo.com> <CAP7+vJ+R-+fzOg26yZWnaTEmWU7QWVQ31D_AMWvVi0BC9LEaXA@mail.gmail.com> <01DACEF7-CA59-48AB-9DE5-C3A4ADE69DE6@yahoo.com> <CAP7+vJLK3t=otHBpcWwz2JBy49f=y_dK_qzZYwwv-qNpinnFBQ@mail.gmail.com> Message-ID: <C2DB1DCE-002D-4457-AA1A-95EFB54D9613@yahoo.com> On Jan 9, 2015, at 21:13, Guido van Rossum <guido at python.org> wrote: > On Fri, Jan 9, 2015 at 7:29 PM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: >> On Jan 9, 2015, at 14:36, Guido van Rossum <guido at python.org> wrote: >> >> > (Although the actual signature of linspace gives me a headache. :-) >> >> I'm assuming you're not talking about the rested and dtype args (which the stdlib wouldn't need), but rather the fact that endpoint changes the meaning of num (you effectively generate num+1 points and discard the last, instead of generating num points). > > All of the above, plus the default to n=50. :-) Oh yeah, that too. I see that you fixed that by just not having a default on count, which seems like the best answer. So, assuming we can get rid of endpoint (either my way or a better way) are you happy with the signature now? > Oh, and the name is odd-sounding for someone not from your world. I move around from world to world; it's weird to me whenever I come back to numpy, unless I also need logspace... > I'm guessing that in terms of implementation you might start with this: > > def linspace(start, stop, num): > return [(stop*i + start*(num-i)) / num for i in range(num+1)] That gives a closed range of num+1 values. I think you want a closed range of num values (or maybe a half-open range of the first num of num+1 values). So you want to replace every num with num-1. > and then refine as follows: > > - make it a lazy sequence, with slicing ability, etc. If you don't need O(1) __contains__ and friends (and I don't think you do) that's maybe 5 lines of code with the ABC. > - numeric tricks to avoid intermediate overflows or numeric instabilities, plus optimizations I'll leave this to others. (If no one steps up, I can look at the numpy code, but I'm sure this list has people who know it off the top of their heads.) > - while still returning exactly the endpoints at the extremes(*) Likewise. > - and better error checking (num must be an integer > 0) Is there anything else that needs to be checked? In particular, start and stop have to be something that can be multiplier and divided by integers, and added to itself, but that's all kinds of types (numbers, numpy arrays of numbers, timedeltas, etc.), so I'm not sure you want to precheck anything there. > - and support for start/stop being other numeric types (e.g. complex, Fraction, Decimal) That already works out of the box for your implementation, and for the obvious conversion to a lazy sequence. It won't work with datetimes because you can't multiply those by integers (you'd need to switch to the naive implementation for those), but it will work with timedeltas and all kinds of other types you'd expect to work. In fact, rather than just say how easy it is, give me 10 minutes to code it... https://github.com/abarnert/linspace > Then bikeshed about the module it should live in. At least you can shut down the bikeshed about the name with "that's what numpy calls it", and ditto about the argument order. Just don't add any of the other features of the numpy version. :-) > > (*) I was going to give the equally naive definition [start + i*step for i in range(num+1)] where step is (stop-start)/num, but that would give away my naivet? even quicker. :-) > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150109/1cd6d062/attachment-0001.html> From alexander.belopolsky at gmail.com Sat Jan 10 08:43:06 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Sat, 10 Jan 2015 02:43:06 -0500 Subject: [Python-ideas] Float range class In-Reply-To: <CAP7+vJLK3t=otHBpcWwz2JBy49f=y_dK_qzZYwwv-qNpinnFBQ@mail.gmail.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> <0D487B6F-26E0-4A11-AD59-1982365FFC97@yahoo.com> <CAP7+vJJ0P4cKFf2sCThWCL_e=CJaMxWt0tK+wpcHdmux0eNvkw@mail.gmail.com> <5797D06D-AE5A-486B-962C-1B37EFDDF9CA@yahoo.com> <CAP7+vJ+R-+fzOg26yZWnaTEmWU7QWVQ31D_AMWvVi0BC9LEaXA@mail.gmail.com> <01DACEF7-CA59-48AB-9DE5-C3A4ADE69DE6@yahoo.com> <CAP7+vJLK3t=otHBpcWwz2JBy49f=y_dK_qzZYwwv-qNpinnFBQ@mail.gmail.com> Message-ID: <07FCEC90-5137-462A-BF3A-5F83B2D9FA0E@gmail.com> > On Jan 10, 2015, at 12:13 AM, Guido van Rossum <guido at python.org> wrote: > > Oh, and the name is odd-sounding for someone not from your world. The name actually came from MATLAB [1] where the signature and the definition actually make more sense. (Note that MATLAB uses 1-based indexing and closed interval slicing, though.) I always thought that "grid" would be a better name. [1] http://www.mathworks.com/help/matlab/ref/linspace.html From steve at pearwood.info Sat Jan 10 09:33:05 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 10 Jan 2015 19:33:05 +1100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <1556539310.4083422.1420781894547.JavaMail.yahoo@jws100143.mail.ne1.yahoo.com> References: <20150107160925.GH19387@ando.pearwood.info> <1556539310.4083422.1420781894547.JavaMail.yahoo@jws100143.mail.ne1.yahoo.com> Message-ID: <20150110083305.GT19387@ando.pearwood.info> On Fri, Jan 09, 2015 at 05:38:14AM +0000, Andrew Barnert wrote: > On Wednesday, January 7, 2015 8:10 AM, Steven D'Aprano <steve at pearwood.info> wrote: > > Python today doesn't give us any simple way to access all those > > different float NANs, or offer any interpretation of what they might > > mean, but it might some day. I am strongly opposed to anything which > > might pre-empt that or discourage future advances. > > I don't think there's really an issue here; the only thing this > preempts is using the plain name `float.nan` for such future advances. Okay, okay, I admit defeat :-) I must say, compared to some years ago when the prevailing attitude about NANs was closer to "if Python supports them at all you're lucky", I am glad to see better support for NANs. If that means baking in the idea that there is "a NAN", I can live with that. > What's wrong with, say, `float.make_nan(payload=30, quiet=False, > sign=True)`? Given that the former is almost certainly going to be > used far more often than the latter, the extra characters to type and > read for the uncommon case don't seem like a serious problem. Or float('-snan30'). > Since you brought up SANE, from what I remember (I tried to dig up the > Apple Numerics Manual or the semi-public SANE 68k assembly source, but > I give up?), this is roughly equivalent to what SANE did. There was a > function to return a NaN with a code from 0-255, with meanings like > "square root of a negative" or "division by zero" or "invalid NaN > code"?the last being used if you tried to call it with 0 as the code. The payloads SANE used (adjusted to Python syntax where appropriate): Name Payload Example ============= ======= ========= NANSQRT 1 sqrt(-1) NANADD 2 +INF + (-INF) NANDIV 4 0/0 NANMUL 8 0*INF NANREM 9 x % 0 NANASCBIN 17 float("foo") NANCOMP 20 convert COMP NAN to float NANZERO 21 create NAN with 0 payload NANTRIG 33 invalid arg to trig functions NANINVTRIG 34 asin(2.0) NANLOG 36 log(-1) NANPOWER 37 invalid arg to x**y NANFINAN 38 invalid arg to finance funcs > If you wanted the all-zeros NaN, that was a separate function. I don't think there is an all-zeroes NAN for single and double format. I believe they would be +INF or -INF depending on the sign bit. There was a 64-bit "comp" data type which had a single NAN value, perhaps that is what you are thinking of. comp was integer valued, and had no INFs. > Meanwhile, if you only care about platforms that use x86/68k-style > representations for IEEE-754 (unfortunately, pre-2008 the bits weren't > standardized, so at least MIPS did things differently), `make_nan` is > pretty easy. Something like: > > def make_nan(payload=0, sign=False, quiet=True): > payload |= (0x7ff0000000000000 | > (sign * 0x8000000000000000) | > (quiet * 0x0008000000000000)) > > return struct.unpack('d', struct.pack('Q', payload))[0] Nice! > And likewise, you could write a `parse_nan` that returns `payload, > sign, quiet`. And you'd probably want to add some parameter validation > (and don't forget that payload 0 and quiet-bit 0 means it's not a NaN, > it's an inf). -- Steve From abarnert at yahoo.com Sat Jan 10 09:37:40 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 10 Jan 2015 00:37:40 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <C2DB1DCE-002D-4457-AA1A-95EFB54D9613@yahoo.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> <0D487B6F-26E0-4A11-AD59-1982365FFC97@yahoo.com> <CAP7+vJJ0P4cKFf2sCThWCL_e=CJaMxWt0tK+wpcHdmux0eNvkw@mail.gmail.com> <5797D06D-AE5A-486B-962C-1B37EFDDF9CA@yahoo.com> <CAP7+vJ+R-+fzOg26yZWnaTEmWU7QWVQ31D_AMWvVi0BC9LEaXA@mail.gmail.com> <01DACEF7-CA59-48AB-9DE5-C3A4ADE69DE6@yahoo.com> <CAP7+vJLK3t=otHBpcWwz2JBy49f=y_dK_qzZYwwv-qNpinnFBQ@mail.gmail.com> <C2DB1DCE-002D-4457-AA1A-95EFB54D9613@yahoo.com> Message-ID: <EC9B82EB-321C-485B-AD77-B317F859001A@yahoo.com> On Jan 9, 2015, at 23:03, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: > On Jan 9, 2015, at 21:13, Guido van Rossum <guido at python.org> wrote: > >> - numeric tricks to avoid intermediate overflows or numeric instabilities, plus optimizations > > I'll leave this to others. (If no one steps up, I can look at the numpy code, but I'm sure this list has people who know it off the top of their heads.) Actually, here's what numpy does: step = (stop-start)/float((num-1)) y = arange(0, num) * step + start y[-1] = stop De-vectorize that, and it's exactly your too-naive algorithm (except for the off-by-1), with a simple fix for inexact stop. From a quick check, the timing is almost identical (6.89us vs. 6.49us to enumerate linspace(1, 2, 5000). It doesn't _normally_ accumulate rounding errors. For example, whether I use the same test, or much smaller numbers, every 5th element, and sometimes the one immediately after, is off by one digit, but the intervening elements are identical. However, if you get down to denormal values, then the underflow _does_ accumulate. For example, linspace(0, 2e-323, 10) should give (0, 0, 5e-324, 5e-324. ...), and the smart version does that, but the naive version--and numpy--gives all 0s. And if you build a simple 4.2 fixed-point type and ask for linspace(0, .1, 100), again you get all 0s. On the other hand, the naive version avoids overflow problems at the other end of the scale, so maybe that's a wash? And, since you no longer need to multiply start or stop by an integer, only stop-start, it works for types like datetime. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150110/24d13190/attachment.html> From abarnert at yahoo.com Sat Jan 10 10:09:22 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 10 Jan 2015 01:09:22 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <C2DB1DCE-002D-4457-AA1A-95EFB54D9613@yahoo.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> <0D487B6F-26E0-4A11-AD59-1982365FFC97@yahoo.com> <CAP7+vJJ0P4cKFf2sCThWCL_e=CJaMxWt0tK+wpcHdmux0eNvkw@mail.gmail.com> <5797D06D-AE5A-486B-962C-1B37EFDDF9CA@yahoo.com> <CAP7+vJ+R-+fzOg26yZWnaTEmWU7QWVQ31D_AMWvVi0BC9LEaXA@mail.gmail.com> <01DACEF7-CA59-48AB-9DE5-C3A4ADE69DE6@yahoo.com> <CAP7+vJLK3t=otHBpcWwz2JBy49f=y_dK_qzZYwwv-qNpinnFBQ@mail.gmail.com> <C2DB1DCE-002D-4457-AA1A-95EFB54D9613@yahoo.com> Message-ID: <0F378993-FC43-4293-854F-0D2F25167DB0@yahoo.com> On Jan 9, 2015, at 23:03, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: > On Jan 9, 2015, at 21:13, Guido van Rossum <guido at python.org> wrote: > >> On Fri, Jan 9, 2015 at 7:29 PM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: >>> On Jan 9, 2015, at 14:36, Guido van Rossum <guido at python.org> wrote: >> - make it a lazy sequence, with slicing ability, etc. > > If you don't need O(1) __contains__ and friends (and I don't think you do) that's maybe 5 lines of code with the ABC. Actually, one problem here: computing the linspace corresponding to a slice of another linspace is not guaranteed to give you the same values. For example, using the second algorithm, lisf(linspace(0, 1, 11)[::3]) != list(linspace(0, .9, 4)). Using the first algorithm, list(linspace(-2, 2, 20))[1::2] != list(linspace(-1.7894736842105263, 2, 10). So, does this mean a slice of a linspace should be a list? A linspace whose values may not compare equal? A special linspace-slice-view object? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150110/7458c148/attachment-0001.html> From abarnert at yahoo.com Sat Jan 10 10:35:36 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 10 Jan 2015 01:35:36 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <20150110083305.GT19387@ando.pearwood.info> References: <20150107160925.GH19387@ando.pearwood.info> <1556539310.4083422.1420781894547.JavaMail.yahoo@jws100143.mail.ne1.yahoo.com> <20150110083305.GT19387@ando.pearwood.info> Message-ID: <7146EB52-B12F-4A5D-8604-1F618F44846F@yahoo.com> On Jan 10, 2015, at 0:33, Steven D'Aprano <steve at pearwood.info> wrote: > On Fri, Jan 09, 2015 at 05:38:14AM +0000, Andrew Barnert wrote: >> On Wednesday, January 7, 2015 8:10 AM, Steven D'Aprano <steve at pearwood.info> wrote: >>> Python today doesn't give us any simple way to access all those >>> different float NANs, or offer any interpretation of what they might >>> mean, but it might some day. I am strongly opposed to anything which >>> might pre-empt that or discourage future advances. >> >> I don't think there's really an issue here; the only thing this >> preempts is using the plain name `float.nan` for such future advances. > > Okay, okay, I admit defeat :-) > > I must say, compared to some years ago when the prevailing attitude > about NANs was closer to "if Python supports them at all you're lucky", > I am glad to see better support for NANs. If that means baking in the > idea that there is "a NAN", I can live with that. > >> What's wrong with, say, `float.make_nan(payload=30, quiet=False, >> sign=True)`? Given that the former is almost certainly going to be >> used far more often than the latter, the extra characters to type and >> read for the uncommon case don't seem like a serious problem. > > Or float('-snan30'). > > >> Since you brought up SANE, from what I remember (I tried to dig up the >> Apple Numerics Manual or the semi-public SANE 68k assembly source, but >> I give up?), this is roughly equivalent to what SANE did. There was a >> function to return a NaN with a code from 0-255, with meanings like >> "square root of a negative" or "division by zero" or "invalid NaN >> code"?the last being used if you tried to call it with 0 as the code. > > The payloads SANE used (adjusted to Python syntax where appropriate): > > > Name Payload Example > ============= ======= ========= > NANSQRT 1 sqrt(-1) > NANADD 2 +INF + (-INF) > NANDIV 4 0/0 > NANMUL 8 0*INF > NANREM 9 x % 0 > NANASCBIN 17 float("foo") > NANCOMP 20 convert COMP NAN to float > NANZERO 21 create NAN with 0 payload > NANTRIG 33 invalid arg to trig functions > NANINVTRIG 34 asin(2.0) > NANLOG 36 log(-1) > NANPOWER 37 invalid arg to x**y > NANFINAN 38 invalid arg to finance funcs Out of curiosity, where did you find this? Do you still have the Apple Numerics Manual in a box with all the Inside Mac and HIG books, or is it online somewhere and I just couldn't find it? >> If you wanted the all-zeros NaN, that was a separate function. > > I don't think there is an all-zeroes NAN for single and double format. I > believe they would be +INF or -INF depending on the sign bit. According to the 2008 standard, the quiet bit is part of the bits that distinguish nan from inf, and that bit is set for quiet; therefore, a qnan with payload 0 exists, and is not an inf. Pre-2008, that was already the way most platforms worked, but not all. I remember some code I had to deal with that had "ifdef MIPS 1s-complement the payload" all over the place (in fact, too all over the place; when reading a float out of storage it would flip the payload twice, giving you -inf where you should have has nan, which was the bug I has to fix that forced me to learn all this stuff in the first place?). From what I remember, the original spec didn't even require that the quiet/signaling bit has to count as part of the bits that separate nan from inf, allowed platforms to reserve any bits for internal use as long as 9 contiguous bits were available for library payloads, and all kinds of other craziness, but I never dealt with anything more deviant than MIPS doing the quiet/signaling bit backward. From ncoghlan at gmail.com Sat Jan 10 11:11:18 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 10 Jan 2015 20:11:18 +1000 Subject: [Python-ideas] Need help converting HTML doc to ReST In-Reply-To: <CAP7+vJJ-FFhMGmGownF82qFK_tJndmf7-uY_3D-X12JQD7T0VQ@mail.gmail.com> References: <CAP7+vJ+qgD_gTeosZKcNjYDwkMp58wVQzrFgPfU2tqwzmP0GsQ@mail.gmail.com> <CAPTjJmpDKkFxAPeF7wmMs17K1erdpkJNa444k6GesMG3zQ1--A@mail.gmail.com> <CAP7+vJ+GVjU2qJfU85My5_KCxgH6oEByZmpLi-z-iuWBXB5xaA@mail.gmail.com> <CAPTjJmq2NbOBaxu66zdnq=aa1e8WmzrHUDHezx=C3QwzcyxWbg@mail.gmail.com> <CAP7+vJJ-FFhMGmGownF82qFK_tJndmf7-uY_3D-X12JQD7T0VQ@mail.gmail.com> Message-ID: <CADiSq7dJWm=V=yQEJZAmubKUR5O2x=K5mucErCEEiyuzvXM=Tg@mail.gmail.com> On 10 January 2015 at 04:35, Guido van Rossum <guido at python.org> wrote: > Thank you so much!! It is already live on legacy.python.org/dev/peps/ (www > seems to lag behind or require a manual kick). I haven't dug into the details to confirm it yet (not enough roundtuits), but my current theory for the lag with new PEPs is that the 404 response is getting cached somewhere, and taking ~45 minutes to expire. That theory came from a commit where I both updated an existing PEP and added a new one, and the updates came through quickly, but the new one took nearly an hour to become available. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Sat Jan 10 11:12:45 2015 From: donald at stufft.io (Donald Stufft) Date: Sat, 10 Jan 2015 05:12:45 -0500 Subject: [Python-ideas] Need help converting HTML doc to ReST In-Reply-To: <CADiSq7dJWm=V=yQEJZAmubKUR5O2x=K5mucErCEEiyuzvXM=Tg@mail.gmail.com> References: <CAP7+vJ+qgD_gTeosZKcNjYDwkMp58wVQzrFgPfU2tqwzmP0GsQ@mail.gmail.com> <CAPTjJmpDKkFxAPeF7wmMs17K1erdpkJNa444k6GesMG3zQ1--A@mail.gmail.com> <CAP7+vJ+GVjU2qJfU85My5_KCxgH6oEByZmpLi-z-iuWBXB5xaA@mail.gmail.com> <CAPTjJmq2NbOBaxu66zdnq=aa1e8WmzrHUDHezx=C3QwzcyxWbg@mail.gmail.com> <CAP7+vJJ-FFhMGmGownF82qFK_tJndmf7-uY_3D-X12JQD7T0VQ@mail.gmail.com> <CADiSq7dJWm=V=yQEJZAmubKUR5O2x=K5mucErCEEiyuzvXM=Tg@mail.gmail.com> Message-ID: <75A1368C-502B-4C0E-A5D6-EA684B4AD1B6@stufft.io> > On Jan 10, 2015, at 5:11 AM, Nick Coghlan <ncoghlan at gmail.com> wrote: > > On 10 January 2015 at 04:35, Guido van Rossum <guido at python.org> wrote: >> Thank you so much!! It is already live on legacy.python.org/dev/peps/ (www >> seems to lag behind or require a manual kick). > > I haven't dug into the details to confirm it yet (not enough > roundtuits), but my current theory for the lag with new PEPs is that > the 404 response is getting cached somewhere, and taking ~45 minutes > to expire. That theory came from a commit where I both updated an > existing PEP and added a new one, and the updates came through > quickly, but the new one took nearly an hour to become available. www.python.org is behind Fastly so it?s probably getting cached there. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From mal at egenix.com Sat Jan 10 11:17:21 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 10 Jan 2015 11:17:21 +0100 Subject: [Python-ideas] Float range class In-Reply-To: <EC9B82EB-321C-485B-AD77-B317F859001A@yahoo.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> <0D487B6F-26E0-4A11-AD59-1982365FFC97@yahoo.com> <CAP7+vJJ0P4cKFf2sCThWCL_e=CJaMxWt0tK+wpcHdmux0eNvkw@mail.gmail.com> <5797D06D-AE5A-486B-962C-1B37EFDDF9CA@yahoo.com> <CAP7+vJ+R-+fzOg26yZWnaTEmWU7QWVQ31D_AMWvVi0BC9LEaXA@mail.gmail.com> <01DACEF7-CA59-48AB-9DE5-C3A4ADE69DE6@yahoo.com> <CAP7+vJLK3t=otHBpcWwz2JBy49f=y_dK_qzZYwwv-qNpinnFBQ@mail.gmail.com> <C2DB1DCE-002D-4457-AA1A-95EFB54D9613@yahoo.com> <EC9B82EB-321C-485B-AD77-B317F859001A@yahoo.com> Message-ID: <54B0FC31.2000307@egenix.com> On 10.01.2015 09:37, Andrew Barnert wrote: > On Jan 9, 2015, at 23:03, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: > >> On Jan 9, 2015, at 21:13, Guido van Rossum <guido at python.org> wrote: >> >>> - numeric tricks to avoid intermediate overflows or numeric instabilities, plus optimizations >> >> I'll leave this to others. (If no one steps up, I can look at the numpy code, but I'm sure this list has people who know it off the top of their heads.) > > Actually, here's what numpy does: > > step = (stop-start)/float((num-1)) > y = arange(0, num) * step + start > y[-1] = stop > > De-vectorize that, and it's exactly your too-naive algorithm (except for the off-by-1), with a simple fix for inexact stop. > > From a quick check, the timing is almost identical (6.89us vs. 6.49us to enumerate linspace(1, 2, 5000). > > It doesn't _normally_ accumulate rounding errors. For example, whether I use the same test, or much smaller numbers, every 5th element, and sometimes the one immediately after, is off by one digit, but the intervening elements are identical. However, if you get down to denormal values, then the underflow _does_ accumulate. For example, linspace(0, 2e-323, 10) should give (0, 0, 5e-324, 5e-324. ...), and the smart version does that, but the naive version--and numpy--gives all 0s. And if you build a simple 4.2 fixed-point type and ask for linspace(0, .1, 100), again you get all 0s. On the other hand, the naive version avoids overflow problems at the other end of the scale, so maybe that's a wash? Hmm, I guess that depends on what floats you "normally" use ;-) You can't even represent 0.1 accurately in IEEE floats: http://grouper.ieee.org/groups/754/faq.html#binary-decimal Since we're not after performance here (use numpy for that), I think it's more important to get as accurate as possible and not have the last interval be larger or smaller than the rest, or having the interval points further right in the list be off of the exact value by an increasing amount (with positive or negative delta). The version I posted does not have these problems. It also has the usual fp rounding issues per interval value, but those errors don't accumulate, since you're not using the same delta across the whole range. I'm sure it can be improved, but it's certainly a more accurate approach than the naive "accumulate rounding errors and then fix the end point" approach :-) def naive_frange(start, stop, steps): start = float(start) stop = float(stop) steps = int(steps) delta = (stop - start) / steps for i in range(steps + 1): yield start + i * delta def frange(start, stop, steps): start = float(start) stop = float(stop) steps = int(steps) for i in range(steps + 1): yield ((steps - i) * start + i * stop) / steps Example: naive_frange: [0.0, 0.1, 0.2, 0.30000000000000004, 0.4, 0.5, 0.6000000000000001, 0.7000000000000001, 0.8, 0.9, 1.0] frange: [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0] -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 10 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From steve at pearwood.info Sat Jan 10 11:23:38 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 10 Jan 2015 21:23:38 +1100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <7146EB52-B12F-4A5D-8604-1F618F44846F@yahoo.com> References: <20150107160925.GH19387@ando.pearwood.info> <1556539310.4083422.1420781894547.JavaMail.yahoo@jws100143.mail.ne1.yahoo.com> <20150110083305.GT19387@ando.pearwood.info> <7146EB52-B12F-4A5D-8604-1F618F44846F@yahoo.com> Message-ID: <20150110102337.GV19387@ando.pearwood.info> On Sat, Jan 10, 2015 at 01:35:36AM -0800, Andrew Barnert wrote: > On Jan 10, 2015, at 0:33, Steven D'Aprano <steve at pearwood.info> wrote: > > The payloads SANE used (adjusted to Python syntax where appropriate): > > > > > > Name Payload Example > > ============= ======= ========= > > NANSQRT 1 sqrt(-1) > > NANADD 2 +INF + (-INF) [...] > Out of curiosity, where did you find this? Do you still have the Apple > Numerics Manual in a box with all the Inside Mac and HIG books, or is > it online somewhere and I just couldn't find it? I have the Numerics Manual on my book shelf, not quite in arms reach but close. I came across a PDF version on Apple's website a long time ago, but lost it, and it no longer appears to be available. > >> If you wanted the all-zeros NaN, that was a separate function. > > > > I don't think there is an all-zeroes NAN for single and double format. I > > believe they would be +INF or -INF depending on the sign bit. > > According to the 2008 standard, the quiet bit is part of the bits that > distinguish nan from inf, and that bit is set for quiet; therefore, a > qnan with payload 0 exists, and is not an inf. Ah, well if you count the quiet bit as separate from the payload, that makes sense. I'm used to thinking of the flag that distinguishes quiet from signalling as part of the payload. The Apple Numerics Manual gives the following three fields for 64-bit doubles: Bit 1: sign bit s Bits 2-12: biased exponent e Bit2 13-64: fraction f 0 < e < 2047, any f: normalised e = 0, f!= 0: denormalised e = 0, f = 0: zero e = 2047, f = 0: infinity e = 2047, f != 0: NAN so I'm used to thinking of the whole 52 bits of f as the payload. For the longest time, I assumed that the sign bit distinguished signalling from quiet NANs, which would leave all 52 bits of f for payload, rather than the most significant bit of f. As it stands now, there are two of each NAN, one with the sign bit set and one without, and the standard doesn't specify any meaning to that. > Pre-2008, that was already the way most platforms worked, but not all. > I remember some code I had to deal with that had "ifdef MIPS > 1s-complement the payload" all over the place (in fact, too all over > the place; when reading a float out of storage it would flip the > payload twice, giving you -inf where you should have has nan, which > was the bug I has to fix that forced me to learn all this stuff in the > first place?). From what I remember, the original spec didn't even > require that the quiet/signaling bit has to count as part of the bits > that separate nan from inf, allowed platforms to reserve any bits for > internal use as long as 9 contiguous bits were available for library > payloads, and all kinds of other craziness, but I never dealt with > anything more deviant than MIPS doing the quiet/signaling bit > backward. Thanks for that bit of history. Are there now any restrictions or standards for what the NAN payloads mean? -- Steve From abarnert at yahoo.com Sat Jan 10 11:47:43 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 10 Jan 2015 02:47:43 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <54B0FC31.2000307@egenix.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> <0D487B6F-26E0-4A11-AD59-1982365FFC97@yahoo.com> <CAP7+vJJ0P4cKFf2sCThWCL_e=CJaMxWt0tK+wpcHdmux0eNvkw@mail.gmail.com> <5797D06D-AE5A-486B-962C-1B37EFDDF9CA@yahoo.com> <CAP7+vJ+R-+fzOg26yZWnaTEmWU7QWVQ31D_AMWvVi0BC9LEaXA@mail.gmail.com> <01DACEF7-CA59-48AB-9DE5-C3A4ADE69DE6@yahoo.com> <CAP7+vJLK3t=otHBpcWwz2JBy49f=y_dK_qzZYwwv-qNpinnFBQ@mail.gmail.com> <C2DB1DCE-002D-4457-AA1A-95EFB54D9613@yahoo.com> <EC9B82EB-321C-485B-AD77-B317F859001A@yahoo.com> <54B0FC31.2000307@egenix.com> Message-ID: <6D538ABE-B688-4D97-B8C5-175A2016ECF2@yahoo.com> On Jan 10, 2015, at 2:17, "M.-A. Lemburg" <mal at egenix.com> wrote: > On 10.01.2015 09:37, Andrew Barnert wrote: >> On Jan 9, 2015, at 23:03, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: >> >>> On Jan 9, 2015, at 21:13, Guido van Rossum <guido at python.org> wrote: >>> >>>> - numeric tricks to avoid intermediate overflows or numeric instabilities, plus optimizations >>> >>> I'll leave this to others. (If no one steps up, I can look at the numpy code, but I'm sure this list has people who know it off the top of their heads.) >> >> Actually, here's what numpy does: >> >> step = (stop-start)/float((num-1)) >> y = arange(0, num) * step + start >> y[-1] = stop >> >> De-vectorize that, and it's exactly your too-naive algorithm (except for the off-by-1), with a simple fix for inexact stop. >> >> From a quick check, the timing is almost identical (6.89us vs. 6.49us to enumerate linspace(1, 2, 5000). >> >> It doesn't _normally_ accumulate rounding errors. For example, whether I use the same test, or much smaller numbers, every 5th element, and sometimes the one immediately after, is off by one digit, but the intervening elements are identical. However, if you get down to denormal values, then the underflow _does_ accumulate. For example, linspace(0, 2e-323, 10) should give (0, 0, 5e-324, 5e-324. ...), and the smart version does that, but the naive version--and numpy--gives all 0s. And if you build a simple 4.2 fixed-point type and ask for linspace(0, .1, 100), again you get all 0s. On the other hand, the naive version avoids overflow problems at the other end of the scale, so maybe that's a wash? > > Hmm, I guess that depends on what floats you "normally" use ;-) > You can't even represent 0.1 accurately in IEEE floats: Well, sure, but a very small linspace having spurious underflows to 0 (with the naive version) or a very large one having spurious overflows to inf (with the clever one) is more serious a problem than 0.1 being off by part of one digit, isn't it? > http://grouper.ieee.org/groups/754/faq.html#binary-decimal > > Since we're not after performance here (use numpy for that), I > think it's more important to get as accurate as possible and > not have the last interval be larger or smaller than the rest, > or having the interval points further right in the list be off > of the exact value by an increasing amount (with positive or > negative delta). Sure, but all of that is true of both versions. Neither one accumulates errors, or has interval points further right in the list be off by an increasing amount (except in the case of very small/large, respectively, values, for which they both have that problem). Both versions have errors of up to one digit, distributed evenly throughout the range. So, I don't think your criteria help to judge, because they judge both versions to be equally good/bad. The naive version happens to give exactly the same errors as np.linspace on most platforms. And the fact that it's used by numpy itself seems to imply that people have thought this through and chosen that implementation, so it's probably worth looking into why they did so. On the other hand, the clever version gives nicer-looking results on some common values (as your example shows). I can find cases where the naive version looks nicer, but they're clearly not going to come up as often. The naive version works with types like datetime. I don't know if any of those considerations are important enough to "win" here. > The version I posted does not have these problems. It also has > the usual fp rounding issues per interval value, but those errors > don't accumulate, since you're not using the same delta > across the whole range. > > I'm sure it can be improved, but it's certainly a more accurate > approach than the naive "accumulate rounding errors and then fix > the end point" approach :-) You'd think so, and yet tests don't seem to bear that out. The reason for fixing the endpoint isn't to deal with accumulate errors (which, again, don't exist), just to deal with the possible single-digit error. > def naive_frange(start, stop, steps): > start = float(start) > stop = float(stop) > steps = int(steps) > delta = (stop - start) / steps > for i in range(steps + 1): > yield start + i * delta > > def frange(start, stop, steps): > start = float(start) > stop = float(stop) > steps = int(steps) > for i in range(steps + 1): > yield ((steps - i) * start + i * stop) / steps You've got an off-by-one error in both versions. If you ask for 10 steps, you get 11 steps. (Guido's two versions had the exact same error--in fact, yours are just slightly rearranged, more verbose, iterator versions of his list comps.) Also, the conversions are unnecessary for the case you're trying to solve (pass in integers and the /steps ensures that they'll be converted to float automatically), and harmful in cases you haven't thought of (your algorithm would work correctly for complex or np.float32, but the conversions prevent it from being used). > Example: > > naive_frange: [0.0, 0.1, 0.2, 0.30000000000000004, 0.4, 0.5, 0.6000000000000001, 0.7000000000000001, > 0.8, 0.9, 1.0] > > frange: [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0] Your own example demonstrates that the naive version _doesn't_ accumulate error. Otherwise, 0.9 would be farther off than 0.7, and it's not. And again, look at my example. With 5000 numbers from 1 to 2, most are correct, but when index%5==0 they're off by one digit, and when index%5==1 they're sometimes off by one digit. The errors are almost perfectly distributed evenly through the range, being no larger or more common at the high end than the low. So, unless you can find some examples that I can't (and explain why numpy would go with an erroneous implementation), I think you may be trying to fix a problem that doesn't exist (as both Guido and I did before you, and in exactly the same way). From rosuav at gmail.com Sat Jan 10 11:54:30 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 10 Jan 2015 21:54:30 +1100 Subject: [Python-ideas] Need help converting HTML doc to ReST In-Reply-To: <CADiSq7dJWm=V=yQEJZAmubKUR5O2x=K5mucErCEEiyuzvXM=Tg@mail.gmail.com> References: <CAP7+vJ+qgD_gTeosZKcNjYDwkMp58wVQzrFgPfU2tqwzmP0GsQ@mail.gmail.com> <CAPTjJmpDKkFxAPeF7wmMs17K1erdpkJNa444k6GesMG3zQ1--A@mail.gmail.com> <CAP7+vJ+GVjU2qJfU85My5_KCxgH6oEByZmpLi-z-iuWBXB5xaA@mail.gmail.com> <CAPTjJmq2NbOBaxu66zdnq=aa1e8WmzrHUDHezx=C3QwzcyxWbg@mail.gmail.com> <CAP7+vJJ-FFhMGmGownF82qFK_tJndmf7-uY_3D-X12JQD7T0VQ@mail.gmail.com> <CADiSq7dJWm=V=yQEJZAmubKUR5O2x=K5mucErCEEiyuzvXM=Tg@mail.gmail.com> Message-ID: <CAPTjJmpfESGyb7=gnPvasQAfJ6FBOxGA7bq95Bpy8dizTWvUSQ@mail.gmail.com> On Sat, Jan 10, 2015 at 9:11 PM, Nick Coghlan <ncoghlan at gmail.com> wrote: > On 10 January 2015 at 04:35, Guido van Rossum <guido at python.org> wrote: >> Thank you so much!! It is already live on legacy.python.org/dev/peps/ (www >> seems to lag behind or require a manual kick). > > I haven't dug into the details to confirm it yet (not enough > roundtuits), but my current theory for the lag with new PEPs is that > the 404 response is getting cached somewhere, and taking ~45 minutes > to expire. That theory came from a commit where I both updated an > existing PEP and added a new one, and the updates came through > quickly, but the new one took nearly an hour to become available. In this case, there was a stub PEP before the change went through. After the first change, I pushed a separate commit that updated the copyright/license; both changes were separately visible on Legacy while the primary site still showed the stub. ChrisA From abarnert at yahoo.com Sat Jan 10 12:18:55 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 10 Jan 2015 03:18:55 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <20150110102337.GV19387@ando.pearwood.info> References: <20150107160925.GH19387@ando.pearwood.info> <1556539310.4083422.1420781894547.JavaMail.yahoo@jws100143.mail.ne1.yahoo.com> <20150110083305.GT19387@ando.pearwood.info> <7146EB52-B12F-4A5D-8604-1F618F44846F@yahoo.com> <20150110102337.GV19387@ando.pearwood.info> Message-ID: <AB38F31B-11EF-46A1-B212-3CE2DBB445ED@yahoo.com> On Jan 10, 2015, at 2:23, Steven D'Aprano <steve at pearwood.info> wrote: > As it stands now, there are two of each NAN, one with the sign bit set > and one without, and the standard doesn't specify any meaning to that. Yeah, I think the current version says that implementations must ignore the sign bit, and users should ignore it. (I don't have the spec in front of me right now, so I'm going by memory here.) > Are there now any restrictions or standards for what the NAN payloads > mean? IIRC, the short version is as few restrictions as possible on the user, and whatever restrictions are needed on implementations to make that true. Something like: * Implementations shouldn't assign any meaning to the payload beyond the quiet bit. * NaN-on-number ops should preserve the entire NaN payload. (I forget what NaN-on-NaN ops do, but I assume it's preserving one or the other.) * If quiet bit is used, it must be x86/etc.-style (set means quiet). * Signaling-to-quiet conversions should leave the other bits alone. From mal at egenix.com Sat Jan 10 14:26:25 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 10 Jan 2015 14:26:25 +0100 Subject: [Python-ideas] Float range class In-Reply-To: <6D538ABE-B688-4D97-B8C5-175A2016ECF2@yahoo.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> <0D487B6F-26E0-4A11-AD59-1982365FFC97@yahoo.com> <CAP7+vJJ0P4cKFf2sCThWCL_e=CJaMxWt0tK+wpcHdmux0eNvkw@mail.gmail.com> <5797D06D-AE5A-486B-962C-1B37EFDDF9CA@yahoo.com> <CAP7+vJ+R-+fzOg26yZWnaTEmWU7QWVQ31D_AMWvVi0BC9LEaXA@mail.gmail.com> <01DACEF7-CA59-48AB-9DE5-C3A4ADE69DE6@yahoo.com> <CAP7+vJLK3t=otHBpcWwz2JBy49f=y_dK_qzZYwwv-qNpinnFBQ@mail.gmail.com> <C2DB1DCE-002D-4457-AA1A-95EFB54D9613@yahoo.com> <EC9B82EB-321C-485B-AD77-B317F859001A@yahoo.com> <54B0FC31.2000307@egenix.com> <6D538ABE-B688-4D97-B8C5-175A2016ECF2@yahoo.co m> Message-ID: <54B12881.9030306@egenix.com> On 10.01.2015 11:47, Andrew Barnert wrote: > On Jan 10, 2015, at 2:17, "M.-A. Lemburg" <mal at egenix.com> wrote: >> Example: >> >> naive_frange: [0.0, 0.1, 0.2, 0.30000000000000004, 0.4, 0.5, 0.6000000000000001, 0.7000000000000001, >> 0.8, 0.9, 1.0] >> >> frange: [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0] > > Your own example demonstrates that the naive version _doesn't_ accumulate error. Otherwise, 0.9 would be farther off than 0.7, and it's not. You're right. The accumulation only happens when using a different naive algorithm: one using addition for calculating the interval points. def naive_add_frange(start, stop, steps): start = float(start) stop = float(stop) steps = int(steps) delta = (stop - start) / steps x = start for i in range(steps + 1): yield x x += delta compared with the multiplication based naive algorithm (this is the same as naive_frange used in the above example): def naive_mul_frange(start, stop, steps): start = float(start) stop = float(stop) steps = int(steps) delta = (stop - start) / steps for i in range(steps + 1): yield start + i * delta With multiplication, the relative errors of both arguments add up. Since the relative errors of both arguments stay (mostly) the same, there's no accumulation, unlike for the addition based variant, where the error adds up in every step (and the relative error can increase even more depending on the value of delta compared to x). Still I find the example results for frange() more in line with what a user would expect. Note: The frange() implementations include the right side of the interval unlike range(). frange() steps refers to the number of steps between the left and right side. Since the right side is included, you get steps + 1 elements in the result. Whether that's what a user would expect or not, is another discussion. Here's the motivation for the frange() implementation: A typical use case is creating values for an axis in a chart. Say your axis goes from 0.0 to 1.0 and you want to split it up in ten intervals, so you write frange(0.0, 1.0, 10). PS: The type conversions in the example implementations are just to signal which types I'm referring to. You can drop those and then get all sorts of wonderful wanted and unwanted effects :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 10 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From njs at pobox.com Sat Jan 10 16:48:14 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 10 Jan 2015 15:48:14 +0000 Subject: [Python-ideas] Float range class In-Reply-To: <EC9B82EB-321C-485B-AD77-B317F859001A@yahoo.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> <0D487B6F-26E0-4A11-AD59-1982365FFC97@yahoo.com> <CAP7+vJJ0P4cKFf2sCThWCL_e=CJaMxWt0tK+wpcHdmux0eNvkw@mail.gmail.com> <5797D06D-AE5A-486B-962C-1B37EFDDF9CA@yahoo.com> <CAP7+vJ+R-+fzOg26yZWnaTEmWU7QWVQ31D_AMWvVi0BC9LEaXA@mail.gmail.com> <01DACEF7-CA59-48AB-9DE5-C3A4ADE69DE6@yahoo.com> <CAP7+vJLK3t=otHBpcWwz2JBy49f=y_dK_qzZYwwv-qNpinnFBQ@mail.gmail.com> <C2DB1DCE-002D-4457-AA1A-95EFB54D9613@yahoo.com> <EC9B82EB-321C-485B-AD77-B317F859001A@yahoo.com> Message-ID: <CAPJVwBnb30_YVNb0bw7NT69WYn6BexyyCJ=48sL750DRRW5RYA@mail.gmail.com> On Sat, Jan 10, 2015 at 8:37 AM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: > > Actually, here's what numpy does: > > step = (stop-start)/float((num-1)) > y = arange(0, num) * step + start > y[-1] = stop > > De-vectorize that, and it's exactly your too-naive algorithm (except for the > off-by-1), with a simple fix for inexact stop. Numpy in general doesn't spend a lot of time worrying about 1 ulp errors versus 2 ulp errors or whatever. We try to avoid using particularly buggy platform transcendental functions, but otherwise, well, real computations accumulate small rounding errors in dozens of places, so worrying about 1 ulp here or there is usually a waste of time. We even use the naive summation algorithm in many cases (though we recently started upgrading to pairwise summation whenever it's cheap to do so), and don't even have an implementation of Kahan summation, never mind something as fancy as math.fsum. We'll probably get around to it eventually, but it's not a high priority. Basically what I'm saying is that numpy is not necessarily an authority on the perfect high-precision implementation of little numeric cores :-). I can say that AFAIK we've never gotten any bug reports about rounding errors in linspace. > From a quick check, the timing is almost identical (6.89us vs. 6.49us to > enumerate linspace(1, 2, 5000). > > It doesn't _normally_ accumulate rounding errors. For example, whether I use > the same test, or much smaller numbers, every 5th element, and sometimes the > one immediately after, is off by one digit, but the intervening elements are > identical. However, if you get down to denormal values, then the underflow > _does_ accumulate. For example, linspace(0, 2e-323, 10) should give (0, 0, > 5e-324, 5e-324. ...), and the smart version does that, but the naive > version--and numpy--gives all 0s. And if you build a simple 4.2 fixed-point > type and ask for linspace(0, .1, 100), again you get all 0s. On the other > hand, the naive version avoids overflow problems at the other end of the > scale, so maybe that's a wash? Okay, I guess I spoke too soon :-) https://github.com/numpy/numpy/issues/5437 -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From guido at python.org Sat Jan 10 18:57:38 2015 From: guido at python.org (Guido van Rossum) Date: Sat, 10 Jan 2015 09:57:38 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <C2DB1DCE-002D-4457-AA1A-95EFB54D9613@yahoo.com> References: <CAFpSVpKpFaVue_1MtLU9UZRRNKhtUEBNVA2w90pM9hEK+846Rg@mail.gmail.com> <CALGmxEKMGapq3TgtQhMePEka6gm_t0TP=eSDzk3fBHVC2rgNwA@mail.gmail.com> <CAFpSVpLPPBd0T3mu-d7bRH3YNKx5WepAqGb-vdoP0_mZgvUwCw@mail.gmail.com> <CALGmxEJvPYr5w3Scp=4vqt-X4jJ2GQmOfRVSTikqugtLC6GcuA@mail.gmail.com> <EDFEA61E-EBB9-4438-87B4-C4811F51E6B0@yahoo.com> <CALGmxELAjWQMvQrviwZaz0yYGOCwdXvcnTN7cCtgCGiE9J0mrg@mail.gmail.com> <7612851a-e55f-4f2c-9cd8-aadaee42b69a@googlegroups.com> <CAPJVwBnTRsExEVunAh-j4c5qJbnPS7XbAYKgZqwNKCbMP58UKA@mail.gmail.com> <CALGmxE+Gmz6LwaDA+pQYyaoaQ-Mj=Hs=oX1QB5e9m=YZhOVn-w@mail.gmail.com> <0D487B6F-26E0-4A11-AD59-1982365FFC97@yahoo.com> <CAP7+vJJ0P4cKFf2sCThWCL_e=CJaMxWt0tK+wpcHdmux0eNvkw@mail.gmail.com> <5797D06D-AE5A-486B-962C-1B37EFDDF9CA@yahoo.com> <CAP7+vJ+R-+fzOg26yZWnaTEmWU7QWVQ31D_AMWvVi0BC9LEaXA@mail.gmail.com> <01DACEF7-CA59-48AB-9DE5-C3A4ADE69DE6@yahoo.com> <CAP7+vJLK3t=otHBpcWwz2JBy49f=y_dK_qzZYwwv-qNpinnFBQ@mail.gmail.com> <C2DB1DCE-002D-4457-AA1A-95EFB54D9613@yahoo.com> Message-ID: <CAP7+vJJ6qhoWCLi8e3er1bKSGCnYJfzpLpyLgH3RfnYQXhAVXg@mail.gmail.com> On Fri, Jan 9, 2015 at 11:03 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > On Jan 9, 2015, at 21:13, Guido van Rossum <guido at python.org> wrote: > > On Fri, Jan 9, 2015 at 7:29 PM, Andrew Barnert < > abarnert at yahoo.com.dmarc.invalid> wrote: > >> On Jan 9, 2015, at 14:36, Guido van Rossum <guido at python.org> wrote: >> >> > (Although the actual signature of linspace gives me a headache. :-) >> >> I'm assuming you're not talking about the rested and dtype args (which >> the stdlib wouldn't need), but rather the fact that endpoint changes the >> meaning of num (you effectively generate num+1 points and discard the last, >> instead of generating num points). >> > > All of the above, plus the default to n=50. :-) > > > Oh yeah, that too. I see that you fixed that by just not having a default > on count, which seems like the best answer. > > So, assuming we can get rid of endpoint (either my way or a better way) > are you happy with the signature now? > No, the name still blows, we don't know which module to put it in, and there's the discussion about how many values to return. > > Oh, and the name is odd-sounding for someone not from your world. > > > I move around from world to world; it's weird to me whenever I come back > to numpy, unless I also need logspace... > > I'm guessing that in terms of implementation you might start with this: > > def linspace(start, stop, num): > return [(stop*i + start*(num-i)) / num for i in range(num+1)] > > > That gives a closed range of num+1 values. I think you want a closed range > of num values (or maybe a half-open range of the first num of num+1 > values). So you want to replace every num with num-1. > Not really. I think of this as splitting the range into `num` equal subranges and returning all the points including the endpoints. I think it would look really weird if you wanted to split [0, 1] into 10 equal sections (or 0.1 each) and you had to say 11. > and then refine as follows: > > - make it a lazy sequence, with slicing ability, etc. > > > If you don't need O(1) __contains__ and friends (and I don't think you > do) that's maybe 5 lines of code with the ABC. > If it's in the stdlib it should be modeled after range(), which is significantly more sophisticated. OTOH if it's just a one-liner recipe in the docs returning a list is fine. OT3H __contains__ seems a really bad idea, due to general issues with float equality. > > - numeric tricks to avoid intermediate overflows or numeric instabilities, > plus optimizations > > > I'll leave this to others. (If no one steps up, I can look at the numpy > code, but I'm sure this list has people who know it off the top of their > heads.) > > - while still returning exactly the endpoints at the extremes(*) > > > Likewise. > > - and better error checking (num must be an integer > 0) > > > Is there anything else that needs to be checked? In particular, start and > stop have to be something that can be multiplier and divided by integers, > and added to itself, but that's all kinds of types (numbers, numpy arrays > of numbers, timedeltas, etc.), so I'm not sure you want to precheck > anything there. > Fair enough. > > - and support for start/stop being other numeric types (e.g. complex, > Fraction, Decimal) > > > That already works out of the box for your implementation, and for the > obvious conversion to a lazy sequence. > > It won't work with datetimes because you can't multiply those by integers > (you'd need to switch to the naive implementation for those), but it will > work with timedeltas and all kinds of other types you'd expect to work. > > In fact, rather than just say how easy it is, give me 10 minutes to code > it... > > https://github.com/abarnert/linspace > I just see more opportunity for bikeshedding. :-( > > Then bikeshed about the module it should live in. At least you can shut > down the bikeshed about the name with "that's what numpy calls it", and > ditto about the argument order. Just don't add any of the other features of > the numpy version. :-) > > (*) I was going to give the equally naive definition [start + i*step for i > in range(num+1)] where step is (stop-start)/num, but that would give away > my naivet? even quicker. :-) > > Honestly, I think this is better off as a set of recipies (starting with the most naive of all and then discussing various refinements for better accuracy or speed or support of e.g. datetime), with a hefty section on the off-by-one issue, rather than a stdlib function. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150110/bf5241f3/attachment-0001.html> From abarnert at yahoo.com Sat Jan 10 20:30:18 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 10 Jan 2015 19:30:18 +0000 (UTC) Subject: [Python-ideas] Float range class In-Reply-To: <CAP7+vJJ6qhoWCLi8e3er1bKSGCnYJfzpLpyLgH3RfnYQXhAVXg@mail.gmail.com> References: <CAP7+vJJ6qhoWCLi8e3er1bKSGCnYJfzpLpyLgH3RfnYQXhAVXg@mail.gmail.com> Message-ID: <528318575.153312.1420918218746.JavaMail.yahoo@jws10099.mail.ne1.yahoo.com> On Saturday, January 10, 2015 9:59 AM, Guido van Rossum <guido at python.org> wrote: >On Fri, Jan 9, 2015 at 11:03 PM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: > >>On Fri, Jan 9, 2015 at 7:29 PM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: >>That gives a closed range of num+1 values. I think you want a closed range of num values (or maybe a half-open range of the first num of num+1 values). So you want to replace every num with num-1. > >Not really. I think of this as splitting the range into `num` equal subranges and returning all the points including the endpoints. I think it would look really weird if you wanted to split [0, 1] into 10 equal sections (or 0.1 each) and you had to say 11. Maybe I'm just expecting it to work the same way as NumPy, and have lost whatever original intuition I might have had on this, and I suspect the NumPy designers likewise just copied from MATLAB rather than rethinking it from first principles, so I'll accept that your way seems more intuitive. But that just means we have to choose between more intuitive vs. more NumPy-like, which seems like yet another opportunity for a bikeshedding argument (and without the NumPy-equivalent API, it reopens the question of whether to use the same name as NumPy, too), so I think you're right in the end; there are too many questions without obvious answers, so maybe it's better off as a recipe. >>If you don't need O(1) __contains__ and friends (and I don't think you do) that's maybe 5 lines of code with the ABC. > >If it's in the stdlib it should be modeled after range(), which is significantly more sophisticated. > >OTOH if it's just a one-liner recipe in the docs returning a list is fine. > >OT3H __contains__ seems a really bad idea, due to general issues with float equality. Exactly, and the same for index and count. If you don't have those, you're not a Sequence. If you have those but they just rely on the linear fallback from the ABC it may not look quite as bad (e.g., they don't show up in the "defined here" part of help), but I'm not sure how good an argument that is for doing it that way when we could do it in O(1) time, and range does so. Of course it could also just be a sequence-like view, not an actual Sequence view, which has its own pros and cons. More bikeshedding... >>In fact, rather than just say how easy it is, give me 10 minutes to code it... >> >>https://github.com/abarnert/linspace > >I just see more opportunity for bikeshedding. :-( >Honestly, I think this is better off as a set of recipies (starting with the most naive of all and then discussing various refinements for better accuracy or speed or support of e.g. datetime), with a hefty section on the off-by-one issue, rather than a stdlib function. Yeah, you've convinced me. Forgive me for further bikeshedding questions, but? should these recipes go in the library docs? (If so, where?) In a HOWTO? Outside the official docs (ActiveState, the wiki, ??) with a reference to them somewhere? From guido at python.org Sat Jan 10 20:50:09 2015 From: guido at python.org (Guido van Rossum) Date: Sat, 10 Jan 2015 11:50:09 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <528318575.153312.1420918218746.JavaMail.yahoo@jws10099.mail.ne1.yahoo.com> References: <CAP7+vJJ6qhoWCLi8e3er1bKSGCnYJfzpLpyLgH3RfnYQXhAVXg@mail.gmail.com> <528318575.153312.1420918218746.JavaMail.yahoo@jws10099.mail.ne1.yahoo.com> Message-ID: <CAP7+vJKQVmzc_+NvbFni-SawJdoMrVNBu8qyDtFVJCozP+3_gQ@mail.gmail.com> I suppose the briefest of recipes could go in the range() docs, in a section "what do do for floats". More extended recipes should probably go on the activestate recipes site. On Sat, Jan 10, 2015 at 11:30 AM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > On Saturday, January 10, 2015 9:59 AM, Guido van Rossum <guido at python.org> > wrote: > > >On Fri, Jan 9, 2015 at 11:03 PM, Andrew Barnert > <abarnert at yahoo.com.dmarc.invalid> wrote: > > > >>On Fri, Jan 9, 2015 at 7:29 PM, Andrew Barnert > <abarnert at yahoo.com.dmarc.invalid> wrote: > >>That gives a closed range of num+1 values. I think you want a closed > range of num values (or maybe a half-open range of the first num of num+1 > values). So you want to replace every num with num-1. > > > >Not really. I think of this as splitting the range into `num` equal > subranges and returning all the points including the endpoints. I think it > would look really weird if you wanted to split [0, 1] into 10 equal > sections (or 0.1 each) and you had to say 11. > > > Maybe I'm just expecting it to work the same way as NumPy, and have lost > whatever original intuition I might have had on this, and I suspect the > NumPy designers likewise just copied from MATLAB rather than rethinking it > from first principles, so I'll accept that your way seems more intuitive. > But that just means we have to choose between more intuitive vs. more > NumPy-like, which seems like yet another opportunity for a bikeshedding > argument (and without the NumPy-equivalent API, it reopens the question of > whether to use the same name as NumPy, too), so I think you're right in the > end; there are too many questions without obvious answers, so maybe it's > better off as a recipe. > > >>If you don't need O(1) __contains__ and friends (and I don't think you > do) that's maybe 5 lines of code with the ABC. > > > >If it's in the stdlib it should be modeled after range(), which is > significantly more sophisticated. > > > >OTOH if it's just a one-liner recipe in the docs returning a list is fine. > > > >OT3H __contains__ seems a really bad idea, due to general issues with > float equality. > > > Exactly, and the same for index and count. If you don't have those, you're > not a Sequence. If you have those but they just rely on the linear fallback > from the ABC it may not look quite as bad (e.g., they don't show up in the > "defined here" part of help), but I'm not sure how good an argument that is > for doing it that way when we could do it in O(1) time, and range does so. > Of course it could also just be a sequence-like view, not an actual > Sequence view, which has its own pros and cons. More bikeshedding... > > >>In fact, rather than just say how easy it is, give me 10 minutes to code > it... > >> > >>https://github.com/abarnert/linspace > > > >I just see more opportunity for bikeshedding. :-( > > > >Honestly, I think this is better off as a set of recipies (starting with > the most naive of all and then discussing various refinements for better > accuracy or speed or support of e.g. datetime), with a hefty section on > the off-by-one issue, rather than a stdlib function. > > > Yeah, you've convinced me. > > > > Forgive me for further bikeshedding questions, but? should these recipes > go in the library docs? (If so, where?) In a HOWTO? Outside the official > docs (ActiveState, the wiki, ??) with a reference to them somewhere? > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150110/9c5643ac/attachment.html> From brett at python.org Sat Jan 10 21:56:21 2015 From: brett at python.org (Brett Cannon) Date: Sat, 10 Jan 2015 20:56:21 +0000 Subject: [Python-ideas] math.inf and math.nan constants References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CAGu0Ansg=r5eTBWOsRztYyBVPYzBJvT9gUU6x9a45-jYPAbSoA@mail.gmail.com> <CAPTjJmqf2s2U7DsjkpyenVHHb8QBbzJUOiLE2cw1xyTGF+_SfA@mail.gmail.com> <CAP7+vJLuoR2d3+ndpz=8cu4=PLPaJqd5AteZe75wZfF41AbExw@mail.gmail.com> Message-ID: <CAP1=2W6asGYzJqys7QONp4F9cbF_3prTgRueNJT7PesqWpi0zQ@mail.gmail.com> On Wed Jan 07 2015 at 5:20:04 PM Guido van Rossum <guido at python.org> wrote: > On Wed, Jan 7, 2015 at 2:13 PM, Chris Angelico <rosuav at gmail.com> wrote: > >> On Thu, Jan 8, 2015 at 6:02 AM, Bruce Leban <bruce at leban.us> wrote: >> > There is a problem here but that's not it. The problem is that >> > >> > float("nan") is float("nan") => False >> > math.nan is math.nan => True >> > >> > >> > So is this an attractive nuisance that will make people think they can >> > should use x is math.nan instead of math.isnan(x)? Hopefully not but >> > something to consider. Pylint should flag this usage. >> >> I don't think that's a problem. Using 'is' to test floating-point >> values is already broken: > > [...] >> > > Correct. And the first half of the thread was comparing the convenience of > defining a nan constant at the top of your module using "nan = > float('nan')" with "from math import nan". That would have the exact same > 'is' problem. > > There is no attractive nuisance here except that this thread itself is > wasting everybody's time. :- > ) > Exactly. Someone should file a bug to track this and then nosy mark.dickinson, rhettinger, stutzbach as specified in the experts index. Then the work can be tracked and be seen done. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150110/37c9e1d9/attachment-0001.html> From ethan at stoneleaf.us Sat Jan 10 22:48:32 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 10 Jan 2015 13:48:32 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <CAP7+vJKQVmzc_+NvbFni-SawJdoMrVNBu8qyDtFVJCozP+3_gQ@mail.gmail.com> References: <CAP7+vJJ6qhoWCLi8e3er1bKSGCnYJfzpLpyLgH3RfnYQXhAVXg@mail.gmail.com> <528318575.153312.1420918218746.JavaMail.yahoo@jws10099.mail.ne1.yahoo.com> <CAP7+vJKQVmzc_+NvbFni-SawJdoMrVNBu8qyDtFVJCozP+3_gQ@mail.gmail.com> Message-ID: <54B19E30.9070706@stoneleaf.us> On 01/10/2015 11:50 AM, Guido van Rossum wrote: > > I suppose the briefest of recipes could go in the range() docs, in a section "what do do for floats". More extended > recipes should probably go on the activestate recipes site. The range docs do seem to be the obvious place for range recipes. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150110/b665a6e0/attachment.sig> From python at 2sn.net Sun Jan 11 00:22:48 2015 From: python at 2sn.net (Alexander Heger) Date: Sun, 11 Jan 2015 10:22:48 +1100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <7146EB52-B12F-4A5D-8604-1F618F44846F@yahoo.com> References: <20150107160925.GH19387@ando.pearwood.info> <1556539310.4083422.1420781894547.JavaMail.yahoo@jws100143.mail.ne1.yahoo.com> <20150110083305.GT19387@ando.pearwood.info> <7146EB52-B12F-4A5D-8604-1F618F44846F@yahoo.com> Message-ID: <CAN3CYHxg7dnqGQ6=o9qKd9yzHxNuug8p0eQ=y=HX2Acv4C4SGw@mail.gmail.com> > According to the 2008 standard, the quiet bit is part of the bits that distinguish nan from inf, and that bit is set for quiet; therefore, a qnan with payload 0 exists, and is not an inf. yes, the default payload is 0 for the quiet NaNs. I suppose the routine can throw an exception if one tries to make a signaling NaN with payload 0. Maybe setting default payload to 1 was not a good suggestion. On could set it to 1 if someone creates a signaling NaN w/o specifying payload (instead)...but, then, if someone goes all the way specifying a signaling NaN, they should know what they are doing ... I also have not been able to find any documentation on specific NaN payload values/constants online so far. ... I find it somewhat a surprise this is not well documented ... -Alexander From ethan at stoneleaf.us Sun Jan 11 01:06:05 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 10 Jan 2015 16:06:05 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAP1=2W6asGYzJqys7QONp4F9cbF_3prTgRueNJT7PesqWpi0zQ@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CAGu0Ansg=r5eTBWOsRztYyBVPYzBJvT9gUU6x9a45-jYPAbSoA@mail.gmail.com> <CAPTjJmqf2s2U7DsjkpyenVHHb8QBbzJUOiLE2cw1xyTGF+_SfA@mail.gmail.com> <CAP7+vJLuoR2d3+ndpz=8cu4=PLPaJqd5AteZe75wZfF41AbExw@mail.gmail.com> <CAP1=2W6asGYzJqys7QONp4F9cbF_3prTgRueNJT7PesqWpi0zQ@mail.gmail.com> Message-ID: <54B1BE6D.8050304@stoneleaf.us> On 01/10/2015 12:56 PM, Brett Cannon wrote: > > Exactly. Someone should file a bug to track this and then nosy mark.dickinson, rhettinger, stutzbach as specified in the > experts index. Then the work can be tracked and be seen done. http://bugs.python.org/issue23185 Was already filed, but other appropriate parties added. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150110/3fa7c278/attachment.sig> From abarnert at yahoo.com Sun Jan 11 05:02:15 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 10 Jan 2015 20:02:15 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAN3CYHxg7dnqGQ6=o9qKd9yzHxNuug8p0eQ=y=HX2Acv4C4SGw@mail.gmail.com> References: <20150107160925.GH19387@ando.pearwood.info> <1556539310.4083422.1420781894547.JavaMail.yahoo@jws100143.mail.ne1.yahoo.com> <20150110083305.GT19387@ando.pearwood.info> <7146EB52-B12F-4A5D-8604-1F618F44846F@yahoo.com> <CAN3CYHxg7dnqGQ6=o9qKd9yzHxNuug8p0eQ=y=HX2Acv4C4SGw@mail.gmail.com> Message-ID: <5214B397-3617-419D-85F1-510686E491CF@yahoo.com> On Jan 10, 2015, at 15:22, Alexander Heger <python at 2sn.net> wrote: >> According to the 2008 standard, the quiet bit is part of the bits that distinguish nan from inf, and that bit is set for quiet; therefore, a qnan with payload 0 exists, and is not an inf. > > yes, the default payload is 0 for the quiet NaNs. I suppose the > routine can throw an exception if one tries to make a signaling NaN > with payload 0. Maybe setting default payload to 1 was not a good > suggestion. On could set it to 1 if someone creates a signaling NaN > w/o specifying payload (instead)...but, then, if someone goes all the > way specifying a signaling NaN, they should know what they are doing Well, my code wasn't intended as a stdlib proposal; if you copy it into your app, hopefully it's because you have a need for it and therefore know what you're doing. :) > I also have not been able to find any documentation on specific NaN > payload values/constants online so far. ... I find it somewhat a > surprise this is not well documented ... I don't know if anyone ever tried to standardize a set of meanings for the payload (I can imagine Apple pushing for SANE's codes, or NASA/JPL trying to get vendors to agree on something), but if so, they apparently never succeeded, because the latest standard (2008) leaves all the bits to user/library code and says the implementation should just preserve and ignore them. There seem to be a lot more blogs and such by committee members and participants nowadays, so there's a good chance that you can find some of the history if you're interested. If you're just looking for something to use as a guide for how to assign payloads in your own project, the list of SANE codes earlier in this thread is probably not a bad place to start. (And you've still got 44 bits left if you want to track the module or something else along with the reason code.) From chris.barker at noaa.gov Mon Jan 12 00:55:56 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Sun, 11 Jan 2015 15:55:56 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAN3CYHwixhXhsqtxxWtjuQP6=dWDh9hKgJ4uUxkq93jM8JV=rg@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> <CAN3CYHwixhXhsqtxxWtjuQP6=dWDh9hKgJ4uUxkq93jM8JV=rg@mail.gmail.com> Message-ID: <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> On Fri, Jan 9, 2015 at 5:03 AM, Alexander Heger <python at 2sn.net> wrote: > I suppose in the end it is not advisable to use "is" to compare > numerical variables in the first place AFAIU, you only want to use "is" for singletons (None) or if you really want to know if two names reference the same object, which is very unlikely need at runtime, but handy for debugging. Making math.nan a singleton is a bad idea for many reasons already mentioned. Though it would be pretty cool if you could override "is" to make: x is math.nan be: math.isnan(x) and maybe the same thing for any arbitrary floats that may have nan values: x is y becomes: True if math.isnan(x) and math.isnan(y) else (id(x) == id(y)) (or something like that...) Is that even possible to override is that way? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150111/6b453353/attachment.html> From rosuav at gmail.com Mon Jan 12 02:05:11 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 12 Jan 2015 12:05:11 +1100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> <CAN3CYHwixhXhsqtxxWtjuQP6=dWDh9hKgJ4uUxkq93jM8JV=rg@mail.gmail.com> <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> Message-ID: <CAPTjJmp_OfG7P90Pujs4A_z7rtJ5fL8aZvLAGzNoXyDj2LjcGA@mail.gmail.com> On Mon, Jan 12, 2015 at 10:55 AM, Chris Barker <chris.barker at noaa.gov> wrote: > Though it would be pretty cool if you could override "is" to make: > > x is math.nan > > be: > > math.isnan(x) No, it wouldn't! It'd be cute, it'd read really nicely the first time, but breaking the definition of object identity is a Bad Thing. For instance, imagine this: x = 0*1e400 y = 1e400-1e400 z = math.nan print(x is y, y is z, x is z) If you hide the first three lines and look only at the last one, you would expect that "False True True" is an impossible result. But that's exactly what redefining "is" could do. ChrisA From stephen at xemacs.org Mon Jan 12 02:31:30 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 12 Jan 2015 10:31:30 +0900 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> <CAN3CYHwixhXhsqtxxWtjuQP6=dWDh9hKgJ4uUxkq93jM8JV=rg@mail.gmail.com> <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> Message-ID: <8761cczswd.fsf@uwakimon.sk.tsukuba.ac.jp> Chris Barker writes: > Making math.nan a singleton is a bad idea for many reasons already > mentioned. But it's not a singleton, not even in the class of NaNs. It's a named constant, like zero or pi. True, the name is ambiguous in the sense that it's easy to forget that (unlike zero and pi) this NaN-with-a-name is not otherwise special in any NaN-y way you can depend on, and that's slightly dangerous. I dislike it -- AFAICS math.nan is pretty useless: if you need a particular NaN repeatedly, I still don't see why "nan = float('nan')" is unsatisfactory, and I worry that multiple package writers might use it as "the NaN to return for undefined operations", competing for the role of "my NaN" (cf the idea suggested that NaN payload could be used to track the module that produced it). That's different from math.inf, which seems like a reasonable addition with no real downside (although similarly little upside AFAICS). But I have to give the advocates this point: I think it's much less likely that people will write "x is math.nan" with poor results than that they will write "x == math.pi",and it's the same group that would likely fall prey to both. > Though it would be pretty cool if you could override "is" to make: > > x is math.nan > > be: > > math.isnan(x) In Python we spell that "isinstance", not "is". From jeanpierreda at gmail.com Mon Jan 12 05:44:18 2015 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sun, 11 Jan 2015 20:44:18 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> <CAN3CYHwixhXhsqtxxWtjuQP6=dWDh9hKgJ4uUxkq93jM8JV=rg@mail.gmail.com> <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> Message-ID: <CABicbJJZ-4ygMnRO30OUhD2Dqs32Esy6xVf5stj+Ti9sXYPVKQ@mail.gmail.com> On Sun, Jan 11, 2015 at 3:55 PM, Chris Barker <chris.barker at noaa.gov> wrote: > Though it would be pretty cool if you could override "is" to make: > > x is math.nan > > be: > > math.isnan(x) > > and maybe the same thing for any arbitrary floats that may have nan values: > > x is y > > becomes: > > True if math.isnan(x) and math.isnan(y) else (id(x) == id(y)) > > (or something like that...) > > Is that even possible to override is that way? Not in Python. Anyway, a huge motivation for "is" is to know that if x is y, then foo(x) is exactly equivalent to foo(y). That would no longer be the if all nans were "identical", since struct.pack can give different values for different nans. -- Devin From chris.barker at noaa.gov Mon Jan 12 06:23:40 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Sun, 11 Jan 2015 21:23:40 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <8761cczswd.fsf@uwakimon.sk.tsukuba.ac.jp> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> <CAN3CYHwixhXhsqtxxWtjuQP6=dWDh9hKgJ4uUxkq93jM8JV=rg@mail.gmail.com> <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> <8761cczswd.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <-3824075524171795640@unknownmsgid> > But it's not a singleton, not even in the class of NaNs. It's a named > constant, like zero or pi. Right, it's a particular value of a float -- more that that, it's a bunch if values, so even less a singleton than pi, even. >> math.isnan(x) > > In Python we spell that "isinstance", not "is". Except that nan is not a type. In way, it's like how there are two bit patterns for the value 0.0 in floats (-0.0 and 0.0). But in that case, -0.0 == 0.0 is True. Whereas NaN == NaN is false. hence the need for math.isnan. Honestly, I wasn't really thinking it would be a good idea, but it did seem like it could make it a pseudo-singleton, and be a, as Chris A. puts it, "cute" way to express it. Not cute enough to create a weird exception to an otherwise clearly defined operator, however. -CHB From jdhardy at gmail.com Mon Jan 12 12:16:08 2015 From: jdhardy at gmail.com (Jeff Hardy) Date: Mon, 12 Jan 2015 11:16:08 +0000 Subject: [Python-ideas] python on mobile In-Reply-To: <CADjSo4QnhX4bSNuiBExNx7pMyffzBT6J0pCZGfvk-2cv_hgM1Q@mail.gmail.com> References: <CADjSo4TssZHGiOCm110co-BQxLAXZpCZbT1T=zdd8eBavnyVzQ@mail.gmail.com> <CAJxq848RyRhbeD4kHssFo1jHUtVX5N1_G=RB1rB+DkmDaxxA2A@mail.gmail.com> <CADiSq7enX0nmNd2AEHGwCgdDsgaXWzdz0A1w5iU_m1V+JdHe6g@mail.gmail.com> <CAJxq84-EeN0v49sHpPYZStJJ_615quqMYmmq2OPvhcaae6Hfiw@mail.gmail.com> <CADjSo4Seed-B0FmTUikrtvDux_LprdvedVrR1-RvQS8fPE=LiQ@mail.gmail.com> <87k31833vw.fsf@uwakimon.sk.tsukuba.ac.jp> <CAJxq848T09_c-jrHhtyyWUBFk7By7xtUphCExom7wNd36ZCtow@mail.gmail.com> <CADiSq7ewkZwRZhkeyuJC32KeaaPuq9FTxQXHiY5uhOZBA+Un=g@mail.gmail.com> <CAF7AXFFh0voFPLPtYxv0qC8yDXmxdBn5K-PT_5oZSXC8RT7jLg@mail.gmail.com> <CADjSo4QnhX4bSNuiBExNx7pMyffzBT6J0pCZGfvk-2cv_hgM1Q@mail.gmail.com> Message-ID: <CAF7AXFEncW5sfZFCc=Jux1UPCS2iKDQoPdQk6c4x_N3YUeO6bQ@mail.gmail.com> On Mon, Jan 5, 2015 at 4:28 PM, Fetchinson . <fetchinson at googlemail.com> wrote: >> So +1 on mobile-sig from me. I'll even volunteer to get it set up & >> administer it > > +1 Apologies for the delay. I've sent a request to meta-sig to create mobile-sig. Anyone interested should chime in over there (https://mail.python.org/mailman/listinfo/meta-sig). - Jeff From stephen at xemacs.org Mon Jan 12 17:05:41 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 13 Jan 2015 01:05:41 +0900 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <-3824075524171795640@unknownmsgid> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> <CAN3CYHwixhXhsqtxxWtjuQP6=dWDh9hKgJ4uUxkq93jM8JV=rg@mail.gmail.com> <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> <8761cczswd.fsf@uwakimon.sk.tsukuba.ac.jp> <-3824075524171795640@unknownmsgid> Message-ID: <87387gyofe.fsf@uwakimon.sk.tsukuba.ac.jp> Chris Barker - NOAA Federal writes: > > But it's not a singleton, not even in the class of NaNs. It's a named > > constant, like zero or pi. > > Right, it's a particular value of a float -- more that that, it's a > bunch if values, so even less a singleton than pi, even. AFAIK, the proposed math.nan is *not* a bunch of values. It's a particular float value that happens to be a NaN, just as math.pi is a particular float value that happens not to be a NaN. The use cases are very different, however. math.pi is used when you want a value that is a close IEEE 754 float approximation to mathematical pi in concrete computations, while math.nan is used when you want a value that "takes you out" of the world of concrete computation and can be interpreted as an "error value". IEEE 754 provides a huge number of NaN values but only defines that "error value" role for them (AIUI, actually four such roles, the combinations of positive vs negative and signaling vs quiet). > >> math.isnan(x) > > > > In Python we spell [the operator form of] that "isinstance", not "is". > > Except that nan is not a type. True, "NaN" is not a type currently implemented in Python, but it could be a subtype of float. I imagine getting the Python class definition right would be a bit fiddly and not at all useful, though. Compare the "bool" type as a subtype of "int". > In way, it's like how there are two bit patterns for the value 0.0 in > floats (-0.0 and 0.0). As far as "typeness" goes, that analogy is exact. Both "Zero" and "NaN" are (non-singleton) subtypes of (IEEE 754) float. You could say that "Pi" is a singleton containing only math.pi, too, but that's not very useful as a description. From breamoreboy at yahoo.co.uk Mon Jan 12 17:44:29 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Mon, 12 Jan 2015 16:44:29 +0000 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <87387gyofe.fsf@uwakimon.sk.tsukuba.ac.jp> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> <CAN3CYHwixhXhsqtxxWtjuQP6=dWDh9hKgJ4uUxkq93jM8JV=rg@mail.gmail.com> <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> <8761cczswd.fsf@uwakimon.sk.tsukuba.ac.jp> <-3824075524171795640@unknownmsgid> <87387gyofe.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <m90tle$sdo$1@ger.gmane.org> On 12/01/2015 16:05, Stephen J. Turnbull wrote: > Chris Barker - NOAA Federal writes: > > > > But it's not a singleton, not even in the class of NaNs. It's a named > > > constant, like zero or pi. > > > > Right, it's a particular value of a float -- more that that, it's a > > bunch if values, so even less a singleton than pi, even. > > AFAIK, the proposed math.nan is *not* a bunch of values. It's a > particular float value that happens to be a NaN, just as math.pi is a > particular float value that happens not to be a NaN. > The implementation is here https://hg.python.org/cpython/rev/cf4bf577749c so as "Although practicality beats purity." can we move on now please? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From chris.barker at noaa.gov Mon Jan 12 17:54:00 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 12 Jan 2015 08:54:00 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <87387gyofe.fsf@uwakimon.sk.tsukuba.ac.jp> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> <CAN3CYHwixhXhsqtxxWtjuQP6=dWDh9hKgJ4uUxkq93jM8JV=rg@mail.gmail.com> <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> <8761cczswd.fsf@uwakimon.sk.tsukuba.ac.jp> <-3824075524171795640@unknownmsgid> <87387gyofe.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <CALGmxEJ8h28B8JymErZ1aRQ5+5P-pikUBLKFSOTwWHfRYtkB8g@mail.gmail.com> TL;DR: Mark Lawrence wrote: > as "Although practicality beats purity." can we move on now please? Absolutely. Adding math.nan and math.inf is great, nothing else to do. Feel free to read more below, if you're as obsessed by this as I am..... On Mon, Jan 12, 2015 at 8:05 AM, Stephen J. Turnbull <stephen at xemacs.org> wrote: > AFAIK, the proposed math.nan is *not* a bunch of values. correct -- math.nan is a particular one of those values. The thing is, in common usage, one often needs to know if a given float is one of the nan values -- it is very rare that anyone knows or cares which one. So you need a good way to ask that question, and: x == math.nan does not work, due to how IEEE defined NaN. So we have math.isnan(x), which is the right thing to do, and the most common question people would ask. Being able to do x is math.nan, and have that give the answer most commonly wanted would be a nice way to spell, but, again, not worth breaking the now clear definition of "is". So -- sorry to bring it up -- I was just enjoying the what if game (and genuinely curious about whether it was possible in the current python implementation to overload is like that. I take it the answer is: "no, and that's a good thing." > particular float value that happens to be a NaN, just as math.pi is a > particular float value that happens not to be a NaN. The use cases > are very different, however. math.pi is used when you want a value > that is a close IEEE 754 float approximation to mathematical pi in > concrete computations, while math.nan is used when you want a value > that "takes you out" of the world of concrete computation and can be > interpreted as an "error value". sure, but the bigger difference is that math.pi == math.pi returns True and x == math.pi returns True is x has the exact same values as math.pi math.isnan(x) is, in more analogous to a < x < b i.e., a specific range of values. Anyway, I'm done. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150112/64b1f572/attachment.html> From guido at python.org Mon Jan 12 19:05:03 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Jan 2015 10:05:03 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CALGmxEJ8h28B8JymErZ1aRQ5+5P-pikUBLKFSOTwWHfRYtkB8g@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> <CAN3CYHwixhXhsqtxxWtjuQP6=dWDh9hKgJ4uUxkq93jM8JV=rg@mail.gmail.com> <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> <8761cczswd.fsf@uwakimon.sk.tsukuba.ac.jp> <-3824075524171795640@unknownmsgid> <87387gyofe.fsf@uwakimon.sk.tsukuba.ac.jp> <CALGmxEJ8h28B8JymErZ1aRQ5+5P-pikUBLKFSOTwWHfRYtkB8g@mail.gmail.com> Message-ID: <CAP7+vJLvFtTQaXtsHOV=_EJA0H62Jqj77TPNbupWMdDOrE8rxg@mail.gmail.com> On Mon, Jan 12, 2015 at 8:54 AM, Chris Barker <chris.barker at noaa.gov> wrote: > > Being able to do x is math.nan, and have that give the answer most > commonly wanted would be a nice way to spell, but, again, not worth > breaking the now clear definition of "is". > While this discussion about what the meaning of "is" is is a funny in-joke for those who remember the Clinton impeachment trials, that you even considered it (and posted about it!) makes me shudder -- Python's "is" operator is so closely tied to object identity that to propose overriding it as a kind of "category" test makes me wonder if you understand Python at all. Proposing to change "==" to make all NaNs appear equal would be less earth-shattering -- that might go against all IEEE fp standards, but at least Python's "==" operator is intended for that purpose. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150112/67fe8d93/attachment-0001.html> From python at mrabarnett.plus.com Mon Jan 12 20:33:24 2015 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 12 Jan 2015 19:33:24 +0000 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAP7+vJLvFtTQaXtsHOV=_EJA0H62Jqj77TPNbupWMdDOrE8rxg@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> <CAN3CYHwixhXhsqtxxWtjuQP6=dWDh9hKgJ4uUxkq93jM8JV=rg@mail.gmail.com> <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> <8761cczswd.fsf@uwakimon.sk.tsukuba.ac.jp> <-3824075524171795640@unknownmsgid> <87387gyofe.fsf@uwakimon.sk.tsukuba.ac.jp> <CALGmxEJ8h28B8JymErZ1aRQ5+5P-pikUBLKFSOTwWHfRYtkB8g@mail.gmail.com> <CAP7+vJLvFtTQaXtsHOV=_EJA0H62Jqj77TPNbupWMdDOrE8rxg@mail.gmail.com> Message-ID: <54B42184.4050702@mrabarnett.plus.com> On 2015-01-12 18:05, Guido van Rossum wrote: > On Mon, Jan 12, 2015 at 8:54 AM, Chris Barker <chris.barker at noaa.gov > <mailto:chris.barker at noaa.gov>> wrote: > > Being able to do x is math.nan, and have that give the answer most > commonly wanted would be a nice way to spell, but, again, not worth > breaking the now clear definition of "is". > > > While this discussion about what the meaning of "is" is is a funny > in-joke for those who remember the Clinton impeachment trials, that you > even considered it (and posted about it!) makes me shudder -- Python's > "is" operator is so closely tied to object identity that to propose > overriding it as a kind of "category" test makes me wonder if you > understand Python at all. Proposing to change "==" to make all NaNs > appear equal would be less earth-shattering -- that might go against all > IEEE fp standards, but at least Python's "==" operator is intended for > that purpose. > I was going to say that calling it "isnan" is in keeping with "isdigit", etc, but those are instance methods! So, why is it "math.isnan(x)" and not "x.isnan()"? From zachary.ware+pyideas at gmail.com Mon Jan 12 20:40:20 2015 From: zachary.ware+pyideas at gmail.com (Zachary Ware) Date: Mon, 12 Jan 2015 13:40:20 -0600 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <54B42184.4050702@mrabarnett.plus.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> <CAN3CYHwixhXhsqtxxWtjuQP6=dWDh9hKgJ4uUxkq93jM8JV=rg@mail.gmail.com> <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> <8761cczswd.fsf@uwakimon.sk.tsukuba.ac.jp> <-3824075524171795640@unknownmsgid> <87387gyofe.fsf@uwakimon.sk.tsukuba.ac.jp> <CALGmxEJ8h28B8JymErZ1aRQ5+5P-pikUBLKFSOTwWHfRYtkB8g@mail.gmail.com> <CAP7+vJLvFtTQaXtsHOV=_EJA0H62Jqj77TPNbupWMdDOrE8rxg@mail.gmail.com> <54B42184.4050702@mrabarnett.plus.com> Message-ID: <CAKJDb-NfTzNyax4XkhFFBbHA12LGn=DCC3fsNFK9LMRd_pu9aw@mail.gmail.com> On Mon, Jan 12, 2015 at 1:33 PM, MRAB <python at mrabarnett.plus.com> wrote: > I was going to say that calling it "isnan" is in keeping with > "isdigit", etc, but those are instance methods! > > So, why is it "math.isnan(x)" and not "x.isnan()"? I'm a little curious about this myself, since float.is_nan(), float.is_inf(), and float.is_finite() are all implemented, but #if 0'd out. -- Zach From guido at python.org Mon Jan 12 20:55:46 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Jan 2015 11:55:46 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAKJDb-NfTzNyax4XkhFFBbHA12LGn=DCC3fsNFK9LMRd_pu9aw@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> <CAN3CYHwixhXhsqtxxWtjuQP6=dWDh9hKgJ4uUxkq93jM8JV=rg@mail.gmail.com> <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> <8761cczswd.fsf@uwakimon.sk.tsukuba.ac.jp> <-3824075524171795640@unknownmsgid> <87387gyofe.fsf@uwakimon.sk.tsukuba.ac.jp> <CALGmxEJ8h28B8JymErZ1aRQ5+5P-pikUBLKFSOTwWHfRYtkB8g@mail.gmail.com> <CAP7+vJLvFtTQaXtsHOV=_EJA0H62Jqj77TPNbupWMdDOrE8rxg@mail.gmail.com> <54B42184.4050702@mrabarnett.plus.com> <CAKJDb-NfTzNyax4XkhFFBbHA12LGn=DCC3fsNFK9LMRd_pu9aw@mail.gmail.com> Message-ID: <CAP7+vJLuHLVckwbsTrBTXhNCO0xyBDoRD_-rtROQ-m99Nsg3Tw@mail.gmail.com> On Mon, Jan 12, 2015 at 11:40 AM, Zachary Ware < zachary.ware+pyideas at gmail.com> wrote: > On Mon, Jan 12, 2015 at 1:33 PM, MRAB <python at mrabarnett.plus.com> wrote: > > I was going to say that calling it "isnan" is in keeping with > > "isdigit", etc, but those are instance methods! > > > > So, why is it "math.isnan(x)" and not "x.isnan()"? > > I'm a little curious about this myself, since float.is_nan(), > float.is_inf(), and float.is_finite() are all implemented, but #if 0'd > out. > Heh. I don't recall how that was decided. If you really want to know, try tracking down the author (hg blame gives me unhelpfully that it was some svn merge in 2008). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150112/32603eb7/attachment.html> From chris.barker at noaa.gov Mon Jan 12 18:02:17 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 12 Jan 2015 09:02:17 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? Message-ID: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> Now that we're talking about floating point conveniences (math.nan, linspace): What about putting an almost_equal(x,y,tol=1e14) (or close(), or...) in the math module. AFAICT, in the standard library, there is only: unittest.TestCase.assertAlmostEqual but it: A) is buried in the unittest.TestCase class B) is an assertion, so you can't use it as a general test (easily) C) uses number of decimal digits or an absolute delta, but does not provide a significant figures comparison, which is most likely what's wanted (and a bit harder to write yourself) numpy provides allclose() (and isclose() ), which is pretty much what I'm suggesting. Anyone else think this would be a good idea to add to the stdlib? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150112/790b8a28/attachment.html> From dickinsm at gmail.com Mon Jan 12 20:59:11 2015 From: dickinsm at gmail.com (Mark Dickinson) Date: Mon, 12 Jan 2015 19:59:11 +0000 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAP7+vJLuHLVckwbsTrBTXhNCO0xyBDoRD_-rtROQ-m99Nsg3Tw@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> <CAN3CYHwixhXhsqtxxWtjuQP6=dWDh9hKgJ4uUxkq93jM8JV=rg@mail.gmail.com> <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> <8761cczswd.fsf@uwakimon.sk.tsukuba.ac.jp> <-3824075524171795640@unknownmsgid> <87387gyofe.fsf@uwakimon.sk.tsukuba.ac.jp> <CALGmxEJ8h28B8JymErZ1aRQ5+5P-pikUBLKFSOTwWHfRYtkB8g@mail.gmail.com> <CAP7+vJLvFtTQaXtsHOV=_EJA0H62Jqj77TPNbupWMdDOrE8rxg@mail.gmail.com> <54B42184.4050702@mrabarnett.plus.com> <CAKJDb-NfTzNyax4XkhFFBbHA12LGn=DCC3fsNFK9LMRd_pu9aw@mail.gmail.com> <CAP7+vJLuHLVckwbsTrBTXhNCO0xyBDoRD_-rtROQ-m99Nsg3Tw@mail.gmail.com> Message-ID: <CAAu3qLXaOueYq9KH56tdV_hVqYPgS7Qp2e8DincUvTYVTc7JuQ@mail.gmail.com> On Mon, Jan 12, 2015 at 7:55 PM, Guido van Rossum <guido at python.org> wrote: > Heh. I don't recall how that was decided. If you really want to know, try > tracking down the author (hg blame gives me unhelpfully that it was some > svn merge in 2008). > My memory tells me that it was Christian Heimes. I'm not sure how much I trust it, though. Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150112/2d5070a6/attachment-0001.html> From guido at python.org Mon Jan 12 21:04:47 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Jan 2015 12:04:47 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> Message-ID: <CAP7+vJL-SWXr01GQRGJsqWmyrN9s+KY6+EqfEcQ=f9PbPswJjQ@mail.gmail.com> So is 1e-100 close to 1e-50 with the default tolerance? What about 1e+100 and (1e+100 plus one ulp)? On Mon, Jan 12, 2015 at 9:02 AM, Chris Barker <chris.barker at noaa.gov> wrote: > Now that we're talking about floating point conveniences (math.nan, > linspace): > > What about putting an > > almost_equal(x,y,tol=1e14) > > (or close(), or...) in the math module. > > AFAICT, in the standard library, there is only: > > unittest.TestCase.assertAlmostEqual > > but it: > > A) is buried in the unittest.TestCase class > > B) is an assertion, so you can't use it as a general test (easily) > > C) uses number of decimal digits or an absolute delta, but does not > provide a significant figures comparison, which is most likely what's > wanted (and a bit harder to write yourself) > > numpy provides allclose() (and isclose() ), which is pretty much what I'm > suggesting. > > Anyone else think this would be a good idea to add to the stdlib? > > -Chris > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150112/462f2540/attachment.html> From guido at python.org Mon Jan 12 21:05:42 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Jan 2015 12:05:42 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAAu3qLXaOueYq9KH56tdV_hVqYPgS7Qp2e8DincUvTYVTc7JuQ@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> <CAN3CYHwixhXhsqtxxWtjuQP6=dWDh9hKgJ4uUxkq93jM8JV=rg@mail.gmail.com> <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> <8761cczswd.fsf@uwakimon.sk.tsukuba.ac.jp> <-3824075524171795640@unknownmsgid> <87387gyofe.fsf@uwakimon.sk.tsukuba.ac.jp> <CALGmxEJ8h28B8JymErZ1aRQ5+5P-pikUBLKFSOTwWHfRYtkB8g@mail.gmail.com> <CAP7+vJLvFtTQaXtsHOV=_EJA0H62Jqj77TPNbupWMdDOrE8rxg@mail.gmail.com> <54B42184.4050702@mrabarnett.plus.com> <CAKJDb-NfTzNyax4XkhFFBbHA12LGn=DCC3fsNFK9LMRd_pu9aw@mail.gmail.com> <CAP7+vJLuHLVckwbsTrBTXhNCO0xyBDoRD_-rtROQ-m99Nsg3Tw@mail.gmail.com> <CAAu3qLXaOueYq9KH56tdV_hVqYPgS7Qp2e8DincUvTYVTc7JuQ@mail.gmail.com> Message-ID: <CAP7+vJLM_h+WySiWi5QCxmK4AbGMnT5nte1=bAxuPCBE+LxbpQ@mail.gmail.com> Christian is the author of the merges for sure. Christian, do you remember why float.is_finite and friends are commented out, in favor of math.isnan()? On Mon, Jan 12, 2015 at 11:59 AM, Mark Dickinson <dickinsm at gmail.com> wrote: > On Mon, Jan 12, 2015 at 7:55 PM, Guido van Rossum <guido at python.org> > wrote: > >> Heh. I don't recall how that was decided. If you really want to know, try >> tracking down the author (hg blame gives me unhelpfully that it was some >> svn merge in 2008). >> > > My memory tells me that it was Christian Heimes. I'm not sure how much I > trust it, though. > > Mark > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150112/a37b7342/attachment.html> From zachary.ware+pyideas at gmail.com Mon Jan 12 21:11:41 2015 From: zachary.ware+pyideas at gmail.com (Zachary Ware) Date: Mon, 12 Jan 2015 14:11:41 -0600 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAP7+vJLM_h+WySiWi5QCxmK4AbGMnT5nte1=bAxuPCBE+LxbpQ@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> <CAN3CYHwixhXhsqtxxWtjuQP6=dWDh9hKgJ4uUxkq93jM8JV=rg@mail.gmail.com> <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> <8761cczswd.fsf@uwakimon.sk.tsukuba.ac.jp> <-3824075524171795640@unknownmsgid> <87387gyofe.fsf@uwakimon.sk.tsukuba.ac.jp> <CALGmxEJ8h28B8JymErZ1aRQ5+5P-pikUBLKFSOTwWHfRYtkB8g@mail.gmail.com> <CAP7+vJLvFtTQaXtsHOV=_EJA0H62Jqj77TPNbupWMdDOrE8rxg@mail.gmail.com> <54B42184.4050702@mrabarnett.plus.com> <CAKJDb-NfTzNyax4XkhFFBbHA12LGn=DCC3fsNFK9LMRd_pu9aw@mail.gmail.com> <CAP7+vJLuHLVckwbsTrBTXhNCO0xyBDoRD_-rtROQ-m99Nsg3Tw@mail.gmail.com> <CAAu3qLXaOueYq9KH56tdV_hVqYPgS7Qp2e8DincUvTYVTc7JuQ@mail.gmail.com> <CAP7+vJLM_h+WySiWi5QCxmK4AbGMnT5nte1=bAxuPCBE+LxbpQ@mail.gmail.com> Message-ID: <CAKJDb-OsKC-4JOZWP7hX-5mbGTHHkBLvWrRPXB0=H9yZ5-baaw@mail.gmail.com> On Mon, Jan 12, 2015 at 2:05 PM, Guido van Rossum <guido at python.org> wrote: > Christian is the author of the merges for sure. Christian, do you remember > why float.is_finite and friends are commented out, in favor of math.isnan()? Here's the actual commit that added float.is_{nan,inf,finite}, already commented out: http://svn.python.org/view?view=revision&revision=62380 """ Author: christian.heimes Date: Fri Apr 18 23:13:07 2008 UTC (6 years, 8 months ago) Changed paths: 25 Log Message: I finally got the time to update and merge Mark's and my trunk-math branch. The patch is collaborated work of Mark Dickinson and me. It was mostly done a few months ago. The patch fixes a lot of loose ends and edge cases related to operations with NaN, INF, very small values and complex math. The patch also adds acosh, asinh, atanh, log1p and copysign to all platforms. Finally it fixes differences between platforms like different results or exceptions for edge cases. Have fun :) """ -- Zach From solipsis at pitrou.net Mon Jan 12 22:00:00 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 12 Jan 2015 22:00:00 +0100 Subject: [Python-ideas] Way to check for floating point "closeness"? References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <CAP7+vJL-SWXr01GQRGJsqWmyrN9s+KY6+EqfEcQ=f9PbPswJjQ@mail.gmail.com> Message-ID: <20150112220000.706875b9@fsol> On Mon, 12 Jan 2015 12:04:47 -0800 Guido van Rossum <guido at python.org> wrote: > So is 1e-100 close to 1e-50 with the default tolerance? What about 1e+100 > and (1e+100 plus one ulp)? Indeed, there are different ways to express such a requirement. Numpy uses a combination of relative and absolute difference (see http://docs.scipy.org/doc/numpy/reference/generated/numpy.isclose.html#numpy.isclose), but you may also want a maximum number of ulps differing. You also need special code for infinities and the like. (and in some cases you may want to distinguish positive and negative zeros, even though they are infinitely close :-)) Regards Antoine. From njs at pobox.com Mon Jan 12 22:03:04 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 12 Jan 2015 21:03:04 +0000 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> Message-ID: <CAPJVwBnGnx+dFw9qE6t-5PqfskSAFBxjNAVLGZRhYeM=b7qtPw@mail.gmail.com> On Mon, Jan 12, 2015 at 5:02 PM, Chris Barker <chris.barker at noaa.gov> wrote: > Now that we're talking about floating point conveniences (math.nan, > linspace): > > What about putting an > > almost_equal(x,y,tol=1e14) > > (or close(), or...) in the math module. > > AFAICT, in the standard library, there is only: > > unittest.TestCase.assertAlmostEqual > > but it: > > A) is buried in the unittest.TestCase class > > B) is an assertion, so you can't use it as a general test (easily) > > C) uses number of decimal digits or an absolute delta, but does not provide > a significant figures comparison, which is most likely what's wanted (and a > bit harder to write yourself) > > numpy provides allclose() (and isclose() ), which is pretty much what I'm > suggesting. Unfortunately this opens a tremendous can of worms. Numpy actually provides: allclose/assert_allclose -- absolute + relative tolerance, not symmetric assert_approx_equal -- agreement to n digits (don't use this) assert_array_max_ulp, assert_array_almost_equal_nulp -- tolerance based on counting ulps. Something like a relative tolerance of n*2^-53 for doubles, but it depends on datatype bits and acts differently near zero. I have no idea what the difference between these two functions is. I've recently argued that allclose should actually act slightly differently: http://mail.scipy.org/pipermail/numpy-discussion/2014-July/070639.html Boost has thought about this a lot and advocates a slightly different definition (actually, two slightly different definitions) from any of the above: http://www.boost.org/doc/libs/1_34_0/libs/test/doc/components/test_tools/floating_point_comparison.html -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From chris.barker at noaa.gov Mon Jan 12 23:23:11 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 12 Jan 2015 14:23:11 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CAP7+vJL-SWXr01GQRGJsqWmyrN9s+KY6+EqfEcQ=f9PbPswJjQ@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <CAP7+vJL-SWXr01GQRGJsqWmyrN9s+KY6+EqfEcQ=f9PbPswJjQ@mail.gmail.com> Message-ID: <CALGmxEKs_k7wRks+HhBQqvNkq9TYzmo5CRJgr_pGFkQL0Q06Hg@mail.gmail.com> On Mon, Jan 12, 2015 at 12:04 PM, Guido van Rossum <guido at python.org> wrote: > So is 1e-100 close to 1e-50 with the default tolerance? What about 1e+100 > and (1e+100 plus one ulp)? > sorry I should have not specified a specific default, and/or explained more what I think it should mean. But I thought the first question was "might this be a good idea for the standard library" -- and only if so, then we can work out how to do it. But anyway, the while point is that it would be some version "relative error", NOT an absolute tolerance -- that is pretty easy to write. See Nathaniel's note for the too many options already in numpy. Nathaniel Smith wrote: Unfortunately this opens a tremendous can of worms. Well, yes, but something generally useful for many cases would still be nice. Boost has thought about this a lot and advocates a slightly different > definition (actually, two slightly different definitions) from any of > the above: > > http://www.boost.org/doc/libs/1_34_0/libs/test/doc/components/test_tools/floating_point_comparison.html > It looks like they start with two, but end up with "the implementation is using modified version of the equations (1) and (2) where all underflow, overflow conditions could be guarded safely" That looks good to me, and the origins in Knuth are a good sign. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150112/bc6c88cd/attachment.html> From guido at python.org Mon Jan 12 23:33:59 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Jan 2015 14:33:59 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CALGmxEKs_k7wRks+HhBQqvNkq9TYzmo5CRJgr_pGFkQL0Q06Hg@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <CAP7+vJL-SWXr01GQRGJsqWmyrN9s+KY6+EqfEcQ=f9PbPswJjQ@mail.gmail.com> <CALGmxEKs_k7wRks+HhBQqvNkq9TYzmo5CRJgr_pGFkQL0Q06Hg@mail.gmail.com> Message-ID: <CAP7+vJLtNO3+qv_VXezcFDpfAywgjA5pU+BJV32q2yDLJ72FyA@mail.gmail.com> I'm guessing this is also something that can go into math. (However, what about complex?) On Mon, Jan 12, 2015 at 2:23 PM, Chris Barker <chris.barker at noaa.gov> wrote: > On Mon, Jan 12, 2015 at 12:04 PM, Guido van Rossum <guido at python.org> > wrote: > >> So is 1e-100 close to 1e-50 with the default tolerance? What about 1e+100 >> and (1e+100 plus one ulp)? >> > > sorry I should have not specified a specific default, and/or explained > more what I think it should mean. > > But I thought the first question was "might this be a good idea for the > standard library" -- and only if so, then we can work out how to do it. > > But anyway, the while point is that it would be some version "relative > error", NOT an absolute tolerance -- that is pretty easy to write. > > See Nathaniel's note for the too many options already in numpy. > > Nathaniel Smith wrote: > > Unfortunately this opens a tremendous can of worms. > > > Well, yes, but something generally useful for many cases would still be > nice. > > Boost has thought about this a lot and advocates a slightly different >> definition (actually, two slightly different definitions) from any of >> the above: >> >> http://www.boost.org/doc/libs/1_34_0/libs/test/doc/components/test_tools/floating_point_comparison.html >> > > > It looks like they start with two, but end up with > > "the implementation is using modified version of the equations (1) and (2) > where all underflow, overflow conditions could be guarded safely" > > That looks good to me, and the origins in Knuth are a good sign. > > -Chris > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150112/14768ece/attachment.html> From abarnert at yahoo.com Mon Jan 12 23:44:12 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 12 Jan 2015 14:44:12 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <54B19E30.9070706@stoneleaf.us> References: <CAP7+vJJ6qhoWCLi8e3er1bKSGCnYJfzpLpyLgH3RfnYQXhAVXg@mail.gmail.com> <528318575.153312.1420918218746.JavaMail.yahoo@jws10099.mail.ne1.yahoo.com> <CAP7+vJKQVmzc_+NvbFni-SawJdoMrVNBu8qyDtFVJCozP+3_gQ@mail.gmail.com> <54B19E30.9070706@stoneleaf.us> Message-ID: <EF17D2ED-4FDC-4CDD-81FD-220A7EBC3CE3@yahoo.com> On Jan 10, 2015, at 13:48, Ethan Furman <ethan at stoneleaf.us> wrote: > On 01/10/2015 11:50 AM, Guido van Rossum wrote: >> >> I suppose the briefest of recipes could go in the range() docs, in a section "what do do for floats". More extended >> recipes should probably go on the activestate recipes site. > > The range docs do seem to be the obvious place for range recipes. I've created issue #23226, with a patch (the whole thing fits into one paragraph and a short example, so I didn't think a subsection was necessary, but that can easily be changed), and a link to a new ActiveState recipe ( that contains the same simple class that turns the NumPy algorithm into a lazy sequence. From abarnert at yahoo.com Mon Jan 12 23:51:26 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 12 Jan 2015 14:51:26 -0800 Subject: [Python-ideas] Float range class In-Reply-To: <EF17D2ED-4FDC-4CDD-81FD-220A7EBC3CE3@yahoo.com> References: <CAP7+vJJ6qhoWCLi8e3er1bKSGCnYJfzpLpyLgH3RfnYQXhAVXg@mail.gmail.com> <528318575.153312.1420918218746.JavaMail.yahoo@jws10099.mail.ne1.yahoo.com> <CAP7+vJKQVmzc_+NvbFni-SawJdoMrVNBu8qyDtFVJCozP+3_gQ@mail.gmail.com> <54B19E30.9070706@stoneleaf.us> <EF17D2ED-4FDC-4CDD-81FD-220A7EBC3CE3@yahoo.com> Message-ID: <C48EA2F0-5733-445B-ADA0-87F9EE12A282@yahoo.com> Sorry, hit Send accidentally. The ActiveState recipe referenced is #579000; it's the same as the class I posted on GitHub, but with only the NumPy implementation, rather than both that and Guido's version. It also links to the two earlier recipes I could find, and to NumPy docs and code, and the discussion section covers (I hope) all the caveats and bikeshedding issues that came up in this thread; the goal is that anyone following a link from the docs will find everything they might need to know about. Sent from a random iPhone On Jan 12, 2015, at 14:44, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: > On Jan 10, 2015, at 13:48, Ethan Furman <ethan at stoneleaf.us> wrote: > >> On 01/10/2015 11:50 AM, Guido van Rossum wrote: >>> >>> I suppose the briefest of recipes could go in the range() docs, in a section "what do do for floats". More extended >>> recipes should probably go on the activestate recipes site. >> >> The range docs do seem to be the obvious place for range recipes. > > I've created issue #23226, with a patch (the whole thing fits into one paragraph and a short example, so I didn't think a subsection was necessary, but that can easily be changed), and a link to a new ActiveState recipe ( that contains the same simple class that turns the NumPy algorithm into a lazy sequence. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From guido at python.org Mon Jan 12 23:56:16 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Jan 2015 14:56:16 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CALGmxEJ8PPWdoV6kRy3RY5x-ddLysfmAegVoE+MP8u22sdx-tw@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <CAP7+vJL-SWXr01GQRGJsqWmyrN9s+KY6+EqfEcQ=f9PbPswJjQ@mail.gmail.com> <CALGmxEKs_k7wRks+HhBQqvNkq9TYzmo5CRJgr_pGFkQL0Q06Hg@mail.gmail.com> <CAP7+vJLtNO3+qv_VXezcFDpfAywgjA5pU+BJV32q2yDLJ72FyA@mail.gmail.com> <CALGmxEJ8PPWdoV6kRy3RY5x-ddLysfmAegVoE+MP8u22sdx-tw@mail.gmail.com> Message-ID: <CAP7+vJ+MEZxPOcvH3MBfBNpD0sVgZPRrJWN++ahZyg_fnO_VVw@mail.gmail.com> But should probably use complex abs(). Adding it to cmath makes sense (if adding it to math happens). On Mon, Jan 12, 2015 at 2:39 PM, Chris Barker <chris.barker at noaa.gov> wrote: > On Mon, Jan 12, 2015 at 2:33 PM, Guido van Rossum <guido at python.org> > wrote: > >> I'm guessing this is also something that can go into math. >> > > makes sense to me. > > >> (However, what about complex?) >> > > first blush: > > is_close(x.real) and is_close(x.imag) > > -Chris > > > > >> On Mon, Jan 12, 2015 at 2:23 PM, Chris Barker <chris.barker at noaa.gov> >> wrote: >> >>> On Mon, Jan 12, 2015 at 12:04 PM, Guido van Rossum <guido at python.org> >>> wrote: >>> >>>> So is 1e-100 close to 1e-50 with the default tolerance? What about >>>> 1e+100 and (1e+100 plus one ulp)? >>>> >>> >>> sorry I should have not specified a specific default, and/or explained >>> more what I think it should mean. >>> >>> But I thought the first question was "might this be a good idea for the >>> standard library" -- and only if so, then we can work out how to do it. >>> >>> But anyway, the while point is that it would be some version "relative >>> error", NOT an absolute tolerance -- that is pretty easy to write. >>> >>> See Nathaniel's note for the too many options already in numpy. >>> >>> Nathaniel Smith wrote: >>> >>> Unfortunately this opens a tremendous can of worms. >>> >>> >>> Well, yes, but something generally useful for many cases would still be >>> nice. >>> >>> Boost has thought about this a lot and advocates a slightly different >>>> definition (actually, two slightly different definitions) from any of >>>> the above: >>>> >>>> http://www.boost.org/doc/libs/1_34_0/libs/test/doc/components/test_tools/floating_point_comparison.html >>>> >>> >>> >>> It looks like they start with two, but end up with >>> >>> "the implementation is using modified version of the equations (1) and >>> (2) where all underflow, overflow conditions could be guarded safely" >>> >>> That looks good to me, and the origins in Knuth are a good sign. >>> >>> -Chris >>> >>> >>> -- >>> >>> Christopher Barker, Ph.D. >>> Oceanographer >>> >>> Emergency Response Division >>> NOAA/NOS/OR&R (206) 526-6959 voice >>> 7600 Sand Point Way NE (206) 526-6329 fax >>> Seattle, WA 98115 (206) 526-6317 main reception >>> >>> Chris.Barker at noaa.gov >>> >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150112/e4bb3f80/attachment.html> From chris.barker at noaa.gov Tue Jan 13 00:34:09 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 12 Jan 2015 15:34:09 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CAP7+vJ+MEZxPOcvH3MBfBNpD0sVgZPRrJWN++ahZyg_fnO_VVw@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <CAP7+vJL-SWXr01GQRGJsqWmyrN9s+KY6+EqfEcQ=f9PbPswJjQ@mail.gmail.com> <CALGmxEKs_k7wRks+HhBQqvNkq9TYzmo5CRJgr_pGFkQL0Q06Hg@mail.gmail.com> <CAP7+vJLtNO3+qv_VXezcFDpfAywgjA5pU+BJV32q2yDLJ72FyA@mail.gmail.com> <CALGmxEJ8PPWdoV6kRy3RY5x-ddLysfmAegVoE+MP8u22sdx-tw@mail.gmail.com> <CAP7+vJ+MEZxPOcvH3MBfBNpD0sVgZPRrJWN++ahZyg_fnO_VVw@mail.gmail.com> Message-ID: <CALGmxEKP2auDojROYz=Lc-gVG72xxqv88PMahvE-iw1mc3_HSA@mail.gmail.com> On Mon, Jan 12, 2015 at 2:56 PM, Guido van Rossum <guido at python.org> wrote: > But should probably use complex abs(). > maybe, it does add complication -- I'd need to look more closely at the Boost code. And I'm trying to think if there are any numerical FP issues that would arise from that -- I'm no expert. Adding it to cmath makes sense (if adding it to math happens). > sure. And I hate to bring this up, but any reason for a cmath.nan and cmath.inf ? I wrote: > It looks like they start with two, but end up with > "the implementation is using modified version of the equations (1) and (2) > where all underflow, overflow conditions could be guarded safely" oops, they do indeed keep both versions, a "weak" and "strong" one. essentially, "is the error small enough relative to both input values, or only one of them. Fairly subtle difference, but I like the "strong" definition, as it gives the same result regardless of input order. though it does guarantee a bit of maybe unnecessary computation. I'm going to play with code a bit now. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150112/a669966e/attachment.html> From jeanpierreda at gmail.com Tue Jan 13 00:39:11 2015 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Mon, 12 Jan 2015 15:39:11 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAKJDb-NfTzNyax4XkhFFBbHA12LGn=DCC3fsNFK9LMRd_pu9aw@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> <CAN3CYHwixhXhsqtxxWtjuQP6=dWDh9hKgJ4uUxkq93jM8JV=rg@mail.gmail.com> <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> <8761cczswd.fsf@uwakimon.sk.tsukuba.ac.jp> <-3824075524171795640@unknownmsgid> <87387gyofe.fsf@uwakimon.sk.tsukuba.ac.jp> <CALGmxEJ8h28B8JymErZ1aRQ5+5P-pikUBLKFSOTwWHfRYtkB8g@mail.gmail.com> <CAP7+vJLvFtTQaXtsHOV=_EJA0H62Jqj77TPNbupWMdDOrE8rxg@mail.gmail.com> <54B42184.4050702@mrabarnett.plus.com> <CAKJDb-NfTzNyax4XkhFFBbHA12LGn=DCC3fsNFK9LMRd_pu9aw@mail.gmail.com> Message-ID: <CABicbJJc12RyiR_Ai6AGk_sJOiY5UTATnVjtUoE_VKZUL0o4Bg@mail.gmail.com> On Mon, Jan 12, 2015 at 11:40 AM, Zachary Ware <zachary.ware+pyideas at gmail.com> wrote: > On Mon, Jan 12, 2015 at 1:33 PM, MRAB <python at mrabarnett.plus.com> wrote: >> I was going to say that calling it "isnan" is in keeping with >> "isdigit", etc, but those are instance methods! >> >> So, why is it "math.isnan(x)" and not "x.isnan()"? > > I'm a little curious about this myself, since float.is_nan(), > float.is_inf(), and float.is_finite() are all implemented, but #if 0'd > out. I'm not Christian, but adding methods to numbers can be annoying because you may also have to add it to ints, rationals, decimals, the numeric ABCs... At the very least, to ints, anyway. -- Devin From chris.barker at noaa.gov Tue Jan 13 00:43:41 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 12 Jan 2015 15:43:41 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CABicbJJc12RyiR_Ai6AGk_sJOiY5UTATnVjtUoE_VKZUL0o4Bg@mail.gmail.com> References: <CAFpSVpJ1JZ0A14yH35Lup8uO6=cNnisNA0Cmrj286Jv89Zvk6w@mail.gmail.com> <20150107160925.GH19387@ando.pearwood.info> <CALGmxE+mn1v7NMF_gWdxCFRmxcT_urFVnccoo7_K3=SJGTRuNg@mail.gmail.com> <CABicbJ+E48zkKbFTj85=vn=exLT3sWmv2vH8tHW14N-qzX8WSQ@mail.gmail.com> <CAN3CYHxC9vORkHjnyj4pYdg97S_EbkpcrFYFsMoJJsBfvJBpOQ@mail.gmail.com> <20150107232304.GM19387@ando.pearwood.info> <CAN3CYHxb2pcSSFyG7NTbWR9DwXoj-YcJvOvx-+ubiqgdSn3kgQ@mail.gmail.com> <CAN3CYHzR42Jo9HiKtSnWW-E7e8ofmfrgXY3sh7q-TMseM_+PiQ@mail.gmail.com> <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> <CAN3CYHwixhXhsqtxxWtjuQP6=dWDh9hKgJ4uUxkq93jM8JV=rg@mail.gmail.com> <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> <8761cczswd.fsf@uwakimon.sk.tsukuba.ac.jp> <-3824075524171795640@unknownmsgid> <87387gyofe.fsf@uwakimon.sk.tsukuba.ac.jp> <CALGmxEJ8h28B8JymErZ1aRQ5+5P-pikUBLKFSOTwWHfRYtkB8g@mail.gmail.com> <CAP7+vJLvFtTQaXtsHOV=_EJA0H62Jqj77TPNbupWMdDOrE8rxg@mail.gmail.com> <54B42184.4050702@mrabarnett.plus.com> <CAKJDb-NfTzNyax4XkhFFBbHA12LGn=DCC3fsNFK9LMRd_pu9aw@mail.gmail.com> <CABicbJJc12RyiR_Ai6AGk_sJOiY5UTATnVjtUoE_VKZUL0o4Bg@mail.gmail.com> Message-ID: <CALGmxELMjCCiOBcGB8H0+_oqK9AJ11wWT9QbfhyZsxi=kWDnAw@mail.gmail.com> On Mon, Jan 12, 2015 at 3:39 PM, Devin Jeanpierre <jeanpierreda at gmail.com> wrote: > >> So, why is it "math.isnan(x)" and not "x.isnan()"? > > > > I'm a little curious about this myself, since float.is_nan(), > > float.is_inf(), and float.is_finite() are all implemented, but #if 0'd > > out. > > I'm not Christian, but adding methods to numbers can be annoying > because you may also have to add it to ints, rationals, decimals, the > numeric ABCs... At the very least, to ints, anyway. and there are any number of other things in math that _could_ be methods.... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150112/6cac26d1/attachment-0001.html> From jeanpierreda at gmail.com Tue Jan 13 00:48:11 2015 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Mon, 12 Jan 2015 15:48:11 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CALGmxEKP2auDojROYz=Lc-gVG72xxqv88PMahvE-iw1mc3_HSA@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <CAP7+vJL-SWXr01GQRGJsqWmyrN9s+KY6+EqfEcQ=f9PbPswJjQ@mail.gmail.com> <CALGmxEKs_k7wRks+HhBQqvNkq9TYzmo5CRJgr_pGFkQL0Q06Hg@mail.gmail.com> <CAP7+vJLtNO3+qv_VXezcFDpfAywgjA5pU+BJV32q2yDLJ72FyA@mail.gmail.com> <CALGmxEJ8PPWdoV6kRy3RY5x-ddLysfmAegVoE+MP8u22sdx-tw@mail.gmail.com> <CAP7+vJ+MEZxPOcvH3MBfBNpD0sVgZPRrJWN++ahZyg_fnO_VVw@mail.gmail.com> <CALGmxEKP2auDojROYz=Lc-gVG72xxqv88PMahvE-iw1mc3_HSA@mail.gmail.com> Message-ID: <CABicbJL-RyNxtSXmq=iVm9spvuwAZ5DCAsEXWPvxdz4fr64dnA@mail.gmail.com> On Mon, Jan 12, 2015 at 3:34 PM, Chris Barker <chris.barker at noaa.gov> wrote: > And I hate to bring this up, but any reason for a cmath.nan and cmath.inf ? One constant would be weird. cmath has more than one or two infinitely far away points, it has eight. (Maybe nan is more obviously nan+nanj though.) Tangent: This bothered me for like three seconds. The complex plane conventionally has only one point at infinity. I guess that would be useless for knowing how you overflowed, though. -- Devin From jeanpierreda at gmail.com Tue Jan 13 00:51:31 2015 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Mon, 12 Jan 2015 15:51:31 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CABicbJL-RyNxtSXmq=iVm9spvuwAZ5DCAsEXWPvxdz4fr64dnA@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <CAP7+vJL-SWXr01GQRGJsqWmyrN9s+KY6+EqfEcQ=f9PbPswJjQ@mail.gmail.com> <CALGmxEKs_k7wRks+HhBQqvNkq9TYzmo5CRJgr_pGFkQL0Q06Hg@mail.gmail.com> <CAP7+vJLtNO3+qv_VXezcFDpfAywgjA5pU+BJV32q2yDLJ72FyA@mail.gmail.com> <CALGmxEJ8PPWdoV6kRy3RY5x-ddLysfmAegVoE+MP8u22sdx-tw@mail.gmail.com> <CAP7+vJ+MEZxPOcvH3MBfBNpD0sVgZPRrJWN++ahZyg_fnO_VVw@mail.gmail.com> <CALGmxEKP2auDojROYz=Lc-gVG72xxqv88PMahvE-iw1mc3_HSA@mail.gmail.com> <CABicbJL-RyNxtSXmq=iVm9spvuwAZ5DCAsEXWPvxdz4fr64dnA@mail.gmail.com> Message-ID: <CABicbJJwTTBgp1AS0B0JhsrRd9yYeZ1M=QJbSo4kRUoA6kct=w@mail.gmail.com> On Mon, Jan 12, 2015 at 3:48 PM, Devin Jeanpierre <jeanpierreda at gmail.com> wrote: > On Mon, Jan 12, 2015 at 3:34 PM, Chris Barker <chris.barker at noaa.gov> wrote: >> And I hate to bring this up, but any reason for a cmath.nan and cmath.inf ? > > One constant would be weird. cmath has more than one or two infinitely > far away points, it has eight. (Maybe nan is more obviously nan+nanj > though.) Er, it has infinitely many, of course. I didn't think that through. :) -- Devin > Tangent: This bothered me for like three seconds. The complex plane > conventionally has only one point at infinity. I guess that would be > useless for knowing how you overflowed, though. > > -- Devin From abarnert at yahoo.com Tue Jan 13 01:26:05 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 12 Jan 2015 16:26:05 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CABicbJJwTTBgp1AS0B0JhsrRd9yYeZ1M=QJbSo4kRUoA6kct=w@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <CAP7+vJL-SWXr01GQRGJsqWmyrN9s+KY6+EqfEcQ=f9PbPswJjQ@mail.gmail.com> <CALGmxEKs_k7wRks+HhBQqvNkq9TYzmo5CRJgr_pGFkQL0Q06Hg@mail.gmail.com> <CAP7+vJLtNO3+qv_VXezcFDpfAywgjA5pU+BJV32q2yDLJ72FyA@mail.gmail.com> <CALGmxEJ8PPWdoV6kRy3RY5x-ddLysfmAegVoE+MP8u22sdx-tw@mail.gmail.com> <CAP7+vJ+MEZxPOcvH3MBfBNpD0sVgZPRrJWN++ahZyg_fnO_VVw@mail.gmail.com> <CALGmxEKP2auDojROYz=Lc-gVG72xxqv88PMahvE-iw1mc3_HSA@mail.gmail.com> <CABicbJL-RyNxtSXmq=iVm9spvuwAZ5DCAsEXWPvxdz4fr64dnA@mail.gmail.com> <CABicbJJwTTBgp1AS0B0JhsrRd9yYeZ1M=QJbSo4kRUoA6kct=w@mail.gmail.com> Message-ID: <1E935C35-BEDE-4865-80B1-AF18E80AC654@yahoo.com> On Jan 12, 2015, at 15:51, Devin Jeanpierre <jeanpierreda at gmail.com> wrote: > On Mon, Jan 12, 2015 at 3:48 PM, Devin Jeanpierre > <jeanpierreda at gmail.com> wrote: >> On Mon, Jan 12, 2015 at 3:34 PM, Chris Barker <chris.barker at noaa.gov> wrote: >>> And I hate to bring this up, but any reason for a cmath.nan and cmath.inf ? >> >> One constant would be weird. cmath has more than one or two infinitely >> far away points, it has eight. (Maybe nan is more obviously nan+nanj >> though.) Well, something like nan+1j or 1+nanj is also sort of nan (as cmath.isnan reflects; also, operations like addition or conjugate that treat real and imag separately properly preserve the nanness or numberness of each part, but those that have to combine them like multiplication convert both to nan). In fact, complex(float('nan')) is nan+0j, not nan+nanj. I don't know if that argues for or against having a cmath.nan. But I think the unavoidable ambiguity of cmath.inf argues against having cmath.nan. > Er, it has infinitely many, of course. I didn't think that through. :) Nah, only 4 * the perfectly finite number of float values. :) From alexander.belopolsky at gmail.com Tue Jan 13 01:52:31 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 12 Jan 2015 19:52:31 -0500 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <1E935C35-BEDE-4865-80B1-AF18E80AC654@yahoo.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <CAP7+vJL-SWXr01GQRGJsqWmyrN9s+KY6+EqfEcQ=f9PbPswJjQ@mail.gmail.com> <CALGmxEKs_k7wRks+HhBQqvNkq9TYzmo5CRJgr_pGFkQL0Q06Hg@mail.gmail.com> <CAP7+vJLtNO3+qv_VXezcFDpfAywgjA5pU+BJV32q2yDLJ72FyA@mail.gmail.com> <CALGmxEJ8PPWdoV6kRy3RY5x-ddLysfmAegVoE+MP8u22sdx-tw@mail.gmail.com> <CAP7+vJ+MEZxPOcvH3MBfBNpD0sVgZPRrJWN++ahZyg_fnO_VVw@mail.gmail.com> <CALGmxEKP2auDojROYz=Lc-gVG72xxqv88PMahvE-iw1mc3_HSA@mail.gmail.com> <CABicbJL-RyNxtSXmq=iVm9spvuwAZ5DCAsEXWPvxdz4fr64dnA@mail.gmail.com> <CABicbJJwTTBgp1AS0B0JhsrRd9yYeZ1M=QJbSo4kRUoA6kct=w@mail.gmail.com> <1E935C35-BEDE-4865-80B1-AF18E80AC654@yahoo.com> Message-ID: <CAP7h-xZi2cC=d7moYPFpgCcU0xio2QbGWhGG5Anv6mgfqAF8Yg@mail.gmail.com> On Mon, Jan 12, 2015 at 7:26 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > But I think the unavoidable ambiguity of cmath.inf argues against having > cmath.nan. I think nan and inf constants should follow the precedent established by e and pi and be float in both math and cmath: >>> type(cmath.pi) <type 'float'> >>> type(cmath.e) <type 'float'> Once you import float nan or inf, it is obvious how to build an complex value containing them using x + y*1j expression, but having to import nan and inf separately when you import everything else from cmath is an annoyance. BTW, as we are discussing math, cmath and non-finite floats, maybe we could clean up some inconsistencies. For example >>> cmath.exp(inf*1j) (nan+nanj) but >>> cmath.sin(inf) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: math domain error >>> cmath.cos(inf) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: math domain error -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150112/09837605/attachment.html> From chris.barker at noaa.gov Tue Jan 13 02:18:45 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 12 Jan 2015 17:18:45 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CALGmxEKP2auDojROYz=Lc-gVG72xxqv88PMahvE-iw1mc3_HSA@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <CAP7+vJL-SWXr01GQRGJsqWmyrN9s+KY6+EqfEcQ=f9PbPswJjQ@mail.gmail.com> <CALGmxEKs_k7wRks+HhBQqvNkq9TYzmo5CRJgr_pGFkQL0Q06Hg@mail.gmail.com> <CAP7+vJLtNO3+qv_VXezcFDpfAywgjA5pU+BJV32q2yDLJ72FyA@mail.gmail.com> <CALGmxEJ8PPWdoV6kRy3RY5x-ddLysfmAegVoE+MP8u22sdx-tw@mail.gmail.com> <CAP7+vJ+MEZxPOcvH3MBfBNpD0sVgZPRrJWN++ahZyg_fnO_VVw@mail.gmail.com> <CALGmxEKP2auDojROYz=Lc-gVG72xxqv88PMahvE-iw1mc3_HSA@mail.gmail.com> Message-ID: <CALGmxEK85cr7tG5hU6TFoBPdFEL-J8H9rrGRr-6HG8EqHKmhoA@mail.gmail.com> OK, I tossed together an example implementation: https://gist.github.com/PythonCHB/6e9ef7732a9074d9337a Simple enough, and seems to work fine for complex, too, out of the box (just using plain old abs). I haven't tried it, but it should work for integers (and decimal?), too, though less need for that anyway. The Boost docs talk about overflow issues, but I can't trigger them, nor see where there would show up, except for if the two values tested were REALLY far apart, which isn't really the use case we're worried about. But I could very well be missing something, I'm no FP expert. It would probably be good if someone that understands these things better than me came up with some near-pathological test cases. Nathaniel, thanks for the pointer to the Boost docs -- I like this much more than numpy's weird atol+rtol approach. -Chris On Mon, Jan 12, 2015 at 3:34 PM, Chris Barker <chris.barker at noaa.gov> wrote: > > On Mon, Jan 12, 2015 at 2:56 PM, Guido van Rossum <guido at python.org> > wrote: > >> But should probably use complex abs(). >> > > maybe, it does add complication -- I'd need to look more closely at the > Boost code. And I'm trying to think if there are any numerical FP issues > that would arise from that -- I'm no expert. > > Adding it to cmath makes sense (if adding it to math happens). >> > > sure. > > And I hate to bring this up, but any reason for a cmath.nan and cmath.inf ? > > > I wrote: > >> It looks like they start with two, but end up with >> "the implementation is using modified version of the equations (1) and >> (2) where all underflow, overflow conditions could be guarded safely" > > > oops, they do indeed keep both versions, a "weak" and "strong" one. > > essentially, "is the error small enough relative to both input values, or > only one of them. Fairly subtle difference, but I like the "strong" > definition, as it gives the same result regardless of input order. though > it does guarantee a bit of maybe unnecessary computation. > > I'm going to play with code a bit now. > > -Chris > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150112/7614ab0e/attachment-0001.html> From chris.barker at noaa.gov Tue Jan 13 02:22:09 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 12 Jan 2015 17:22:09 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <1E935C35-BEDE-4865-80B1-AF18E80AC654@yahoo.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <CAP7+vJL-SWXr01GQRGJsqWmyrN9s+KY6+EqfEcQ=f9PbPswJjQ@mail.gmail.com> <CALGmxEKs_k7wRks+HhBQqvNkq9TYzmo5CRJgr_pGFkQL0Q06Hg@mail.gmail.com> <CAP7+vJLtNO3+qv_VXezcFDpfAywgjA5pU+BJV32q2yDLJ72FyA@mail.gmail.com> <CALGmxEJ8PPWdoV6kRy3RY5x-ddLysfmAegVoE+MP8u22sdx-tw@mail.gmail.com> <CAP7+vJ+MEZxPOcvH3MBfBNpD0sVgZPRrJWN++ahZyg_fnO_VVw@mail.gmail.com> <CALGmxEKP2auDojROYz=Lc-gVG72xxqv88PMahvE-iw1mc3_HSA@mail.gmail.com> <CABicbJL-RyNxtSXmq=iVm9spvuwAZ5DCAsEXWPvxdz4fr64dnA@mail.gmail.com> <CABicbJJwTTBgp1AS0B0JhsrRd9yYeZ1M=QJbSo4kRUoA6kct=w@mail.gmail.com> <1E935C35-BEDE-4865-80B1-AF18E80AC654@yahoo.com> Message-ID: <CALGmxEJe91FoJX7zF6+b4J7L8BJkWQGcVw6_0c3ZhRCK_J7eiQ@mail.gmail.com> On Mon, Jan 12, 2015 at 4:26 PM, Andrew Barnert <abarnert at yahoo.com> wrote: > But I think the unavoidable ambiguity of cmath.inf argues against having > cmath.nan. > agreed. we already need to spell -inf: -math.inf, so the four possible complex infs can be built up similarly. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150112/2c7060e1/attachment.html> From steve at pearwood.info Tue Jan 13 02:34:53 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 13 Jan 2015 12:34:53 +1100 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> Message-ID: <20150113013452.GE10173@ando.pearwood.info> On Mon, Jan 12, 2015 at 09:02:17AM -0800, Chris Barker wrote: > Now that we're talking about floating point conveniences (math.nan, > linspace): > > What about putting an > > almost_equal(x,y,tol=1e14) [...] > Anyone else think this would be a good idea to add to the stdlib? I do, and I have already done so! It's an implementation detail of the statistics module (to be specific, its test suite), but it covers both relative and absolute error tolerances and handles infinities and NANs. https://hg.python.org/cpython/file/1b145e8ae4be/Lib/test/test_statistics.py#l41 The default tolerances are more or less plucked out of thin air and probably should be discussed. Ideally it should also handle ULP comparisons: https://randomascii.wordpress.com/2012/02/25/comparing-floating-point-numbers-2012-edition/ Unfortunately a naive ULP comparison has trouble with NANs, INFs, and numbers close to zero, especially if they have opposite signs. The smallest representable denormalised floats larger, and smaller, than zero are: 5e-324 -5e-324 These are the smallest magnitude floats apart from zero, so we might hope that they are considered "close together", but they actually differ by 9223372036854775808 ULP. Ouch. I have some ideas for dealing with that, and if anyone is interested I'm happy to talk about it, but they're not ready for production yet. I think that the Bruce Dawson is right. Floating point comparisons are hard, really hard. I know that I've still got a lot to learn about it. I can think of at least five different ways to compare floats for equality, and they all have their uses: - exact equality using == - absolute error tolerances - relative error tolerances - ULP comparisons - the method unittest uses, using round() I'm explicitly including == because it is a floating point superstition that one should never under any circumstances compare floats for exact equality. As general advice, "don't use == unless you know what you are doing" is quite reasonable, but it's the "never use" that turns it into superstition. As Bruce Dawson says, "Floating-point numbers aren?t cursed", and throwing epsilons into a problem where no epsilon is needed is a bad idea. https://randomascii.wordpress.com/2012/06/26/doubles-are-not-floats-so-dont-compare-them/ Aside: I'm reminded of APL, which mandates fuzzy equality (i.e. with a tolerance) of floating point numbers: In an early talk Ken [Iverson] was explaining the advantages of tolerant comparison. A member of the audience asked incredulously, ?Surely you don?t mean that when A=B and B=C, A may not equal C?? Without skipping a beat, Ken replied, ?Any carpenter knows that!? and went on to the next question. - Paul Berry -- Steve From steve at pearwood.info Tue Jan 13 02:49:13 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 13 Jan 2015 12:49:13 +1100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAKJDb-NfTzNyax4XkhFFBbHA12LGn=DCC3fsNFK9LMRd_pu9aw@mail.gmail.com> References: <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> <CAN3CYHwixhXhsqtxxWtjuQP6=dWDh9hKgJ4uUxkq93jM8JV=rg@mail.gmail.com> <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> <8761cczswd.fsf@uwakimon.sk.tsukuba.ac.jp> <-3824075524171795640@unknownmsgid> <87387gyofe.fsf@uwakimon.sk.tsukuba.ac.jp> <CALGmxEJ8h28B8JymErZ1aRQ5+5P-pikUBLKFSOTwWHfRYtkB8g@mail.gmail.com> <CAP7+vJLvFtTQaXtsHOV=_EJA0H62Jqj77TPNbupWMdDOrE8rxg@mail.gmail.com> <54B42184.4050702@mrabarnett.plus.com> <CAKJDb-NfTzNyax4XkhFFBbHA12LGn=DCC3fsNFK9LMRd_pu9aw@mail.gmail.com> Message-ID: <20150113014912.GF10173@ando.pearwood.info> On Mon, Jan 12, 2015 at 01:40:20PM -0600, Zachary Ware wrote: > On Mon, Jan 12, 2015 at 1:33 PM, MRAB <python at mrabarnett.plus.com> wrote: > > I was going to say that calling it "isnan" is in keeping with > > "isdigit", etc, but those are instance methods! > > > > So, why is it "math.isnan(x)" and not "x.isnan()"? > > I'm a little curious about this myself, since float.is_nan(), > float.is_inf(), and float.is_finite() are all implemented, but #if 0'd > out. Can we re-start this issue? http://bugs.python.org/issue18842 I have a lot of code that takes either a decimal or a float, and I currently have to write things like: if isinstance(x, Decimal): isnan = x.is_nan() else: isnan = math.isnan(x) which is just ick. If you're wondering why I don't just unconditionally call math.isnan, consider this: py> from decimal import Decimal py> x = Decimal('snan') py> float(x) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: cannot convert signaling NaN to float -- Steve From guido at python.org Tue Jan 13 03:12:05 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Jan 2015 18:12:05 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <20150113014912.GF10173@ando.pearwood.info> References: <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> <CAN3CYHwixhXhsqtxxWtjuQP6=dWDh9hKgJ4uUxkq93jM8JV=rg@mail.gmail.com> <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> <8761cczswd.fsf@uwakimon.sk.tsukuba.ac.jp> <-3824075524171795640@unknownmsgid> <87387gyofe.fsf@uwakimon.sk.tsukuba.ac.jp> <CALGmxEJ8h28B8JymErZ1aRQ5+5P-pikUBLKFSOTwWHfRYtkB8g@mail.gmail.com> <CAP7+vJLvFtTQaXtsHOV=_EJA0H62Jqj77TPNbupWMdDOrE8rxg@mail.gmail.com> <54B42184.4050702@mrabarnett.plus.com> <CAKJDb-NfTzNyax4XkhFFBbHA12LGn=DCC3fsNFK9LMRd_pu9aw@mail.gmail.com> <20150113014912.GF10173@ando.pearwood.info> Message-ID: <CAP7+vJJ28v++cw85NWubg8HzkRRwfZv7zTP4FVchkkLs_sWN1A@mail.gmail.com> But what other methods from Decimal would we have to add to float (and to int, etc.)? IIRC there are just too many differences between Decimal and float to ever hope for equivalence. (How do you spell sin() of a Decimal?) On Mon, Jan 12, 2015 at 5:49 PM, Steven D'Aprano <steve at pearwood.info> wrote: > On Mon, Jan 12, 2015 at 01:40:20PM -0600, Zachary Ware wrote: > > On Mon, Jan 12, 2015 at 1:33 PM, MRAB <python at mrabarnett.plus.com> > wrote: > > > I was going to say that calling it "isnan" is in keeping with > > > "isdigit", etc, but those are instance methods! > > > > > > So, why is it "math.isnan(x)" and not "x.isnan()"? > > > > I'm a little curious about this myself, since float.is_nan(), > > float.is_inf(), and float.is_finite() are all implemented, but #if 0'd > > out. > > Can we re-start this issue? > > http://bugs.python.org/issue18842 > > > I have a lot of code that takes either a decimal or a float, and I > currently have to write things like: > > if isinstance(x, Decimal): > isnan = x.is_nan() > else: > isnan = math.isnan(x) > > which is just ick. > > > If you're wondering why I don't just unconditionally call math.isnan, > consider this: > > py> from decimal import Decimal > py> x = Decimal('snan') > py> float(x) > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > ValueError: cannot convert signaling NaN to float > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150112/165c931c/attachment.html> From rosuav at gmail.com Tue Jan 13 03:29:20 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 13 Jan 2015 13:29:20 +1100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAP7+vJJ28v++cw85NWubg8HzkRRwfZv7zTP4FVchkkLs_sWN1A@mail.gmail.com> References: <CADiSq7dd7Ky_G12-gp9NDRuG08B_JDO1M1n_pcRgH1x5zhFtXA@mail.gmail.com> <CAN3CYHwixhXhsqtxxWtjuQP6=dWDh9hKgJ4uUxkq93jM8JV=rg@mail.gmail.com> <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> <8761cczswd.fsf@uwakimon.sk.tsukuba.ac.jp> <-3824075524171795640@unknownmsgid> <87387gyofe.fsf@uwakimon.sk.tsukuba.ac.jp> <CALGmxEJ8h28B8JymErZ1aRQ5+5P-pikUBLKFSOTwWHfRYtkB8g@mail.gmail.com> <CAP7+vJLvFtTQaXtsHOV=_EJA0H62Jqj77TPNbupWMdDOrE8rxg@mail.gmail.com> <54B42184.4050702@mrabarnett.plus.com> <CAKJDb-NfTzNyax4XkhFFBbHA12LGn=DCC3fsNFK9LMRd_pu9aw@mail.gmail.com> <20150113014912.GF10173@ando.pearwood.info> <CAP7+vJJ28v++cw85NWubg8HzkRRwfZv7zTP4FVchkkLs_sWN1A@mail.gmail.com> Message-ID: <CAPTjJmqACE3G93_pr7iTTUi1N-Z-cPcTjrVDTQL4anb=zBqGMw@mail.gmail.com> On Tue, Jan 13, 2015 at 1:12 PM, Guido van Rossum <guido at python.org> wrote: > But what other methods from Decimal would we have to add to float (and to > int, etc.)? > > IIRC there are just too many differences between Decimal and float to ever > hope for equivalence. (How do you spell sin() of a Decimal?) These kinds of features should be modular (in the Mark Rosewater sense of the word). Adding float.is_nan is useful whether or not you can calculate the sine of a Decimal (and vice versa). It's currently possible to ask the question "Are you NaN?" of both floats and Decimals, and this simply adds compatible spellings for it. I'm +0 on it, but only because I don't use Decimal NaN in my code. If I did, I'd probably be a strong +1. ChrisA From njs at pobox.com Tue Jan 13 03:32:15 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 13 Jan 2015 02:32:15 +0000 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CALGmxEK85cr7tG5hU6TFoBPdFEL-J8H9rrGRr-6HG8EqHKmhoA@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <CAP7+vJL-SWXr01GQRGJsqWmyrN9s+KY6+EqfEcQ=f9PbPswJjQ@mail.gmail.com> <CALGmxEKs_k7wRks+HhBQqvNkq9TYzmo5CRJgr_pGFkQL0Q06Hg@mail.gmail.com> <CAP7+vJLtNO3+qv_VXezcFDpfAywgjA5pU+BJV32q2yDLJ72FyA@mail.gmail.com> <CALGmxEJ8PPWdoV6kRy3RY5x-ddLysfmAegVoE+MP8u22sdx-tw@mail.gmail.com> <CAP7+vJ+MEZxPOcvH3MBfBNpD0sVgZPRrJWN++ahZyg_fnO_VVw@mail.gmail.com> <CALGmxEKP2auDojROYz=Lc-gVG72xxqv88PMahvE-iw1mc3_HSA@mail.gmail.com> <CALGmxEK85cr7tG5hU6TFoBPdFEL-J8H9rrGRr-6HG8EqHKmhoA@mail.gmail.com> Message-ID: <CAPJVwBmOMrdataFhhtCSiY=4Au-9=wa=fpKhz4j9cdeUPkCnvw@mail.gmail.com> Ok, but you're missing a huge issue, which is that according to the rule you suggest, 0.0 is not close to any non-zero value. Handing such a function to users and claiming it's the one "is close" predicate they need is going to produce a lot of sad and confused users. I don't want to step on your positive let's-fix-it attitude, I think that's great, but... I do think you are really underestimating the complexity of the problem. The right way to handle comparison with zero, or comparison between epsilon and negative epsilon, etc, really depends on the context. For common unit test cases it may even make sense to prefer a non-symmetric predicate -- e.g. if you're expecting 0 then 1e-100 is usually fine, but if you're expecting 1e-100 then getting 0 might be a huge problem. -n On 13 Jan 2015 01:19, "Chris Barker" <chris.barker at noaa.gov> wrote: > OK, > > I tossed together an example implementation: > > https://gist.github.com/PythonCHB/6e9ef7732a9074d9337a > > Simple enough, and seems to work fine for complex, too, out of the box > (just using plain old abs). I haven't tried it, but it should work for > integers (and decimal?), too, though less need for that anyway. > > The Boost docs talk about overflow issues, but I can't trigger them, nor > see where there would show up, except for if the two values tested were > REALLY far apart, which isn't really the use case we're worried about. But > I could very well be missing something, I'm no FP expert. > > It would probably be good if someone that understands these things better > than me came up with some near-pathological test cases. > > Nathaniel, thanks for the pointer to the Boost docs -- I like this much > more than numpy's weird atol+rtol approach. > > -Chris > > > On Mon, Jan 12, 2015 at 3:34 PM, Chris Barker <chris.barker at noaa.gov> > wrote: > >> >> On Mon, Jan 12, 2015 at 2:56 PM, Guido van Rossum <guido at python.org> >> wrote: >> >>> But should probably use complex abs(). >>> >> >> maybe, it does add complication -- I'd need to look more closely at the >> Boost code. And I'm trying to think if there are any numerical FP issues >> that would arise from that -- I'm no expert. >> >> Adding it to cmath makes sense (if adding it to math happens). >>> >> >> sure. >> >> And I hate to bring this up, but any reason for a cmath.nan and cmath.inf >> ? >> >> >> I wrote: >> >>> It looks like they start with two, but end up with >>> "the implementation is using modified version of the equations (1) and >>> (2) where all underflow, overflow conditions could be guarded safely" >> >> >> oops, they do indeed keep both versions, a "weak" and "strong" one. >> >> essentially, "is the error small enough relative to both input values, or >> only one of them. Fairly subtle difference, but I like the "strong" >> definition, as it gives the same result regardless of input order. though >> it does guarantee a bit of maybe unnecessary computation. >> >> I'm going to play with code a bit now. >> >> -Chris >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.Barker at noaa.gov >> > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150113/1d8d546b/attachment.html> From steve at pearwood.info Tue Jan 13 03:39:11 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 13 Jan 2015 13:39:11 +1100 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAP7+vJJ28v++cw85NWubg8HzkRRwfZv7zTP4FVchkkLs_sWN1A@mail.gmail.com> References: <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> <8761cczswd.fsf@uwakimon.sk.tsukuba.ac.jp> <-3824075524171795640@unknownmsgid> <87387gyofe.fsf@uwakimon.sk.tsukuba.ac.jp> <CALGmxEJ8h28B8JymErZ1aRQ5+5P-pikUBLKFSOTwWHfRYtkB8g@mail.gmail.com> <CAP7+vJLvFtTQaXtsHOV=_EJA0H62Jqj77TPNbupWMdDOrE8rxg@mail.gmail.com> <54B42184.4050702@mrabarnett.plus.com> <CAKJDb-NfTzNyax4XkhFFBbHA12LGn=DCC3fsNFK9LMRd_pu9aw@mail.gmail.com> <20150113014912.GF10173@ando.pearwood.info> <CAP7+vJJ28v++cw85NWubg8HzkRRwfZv7zTP4FVchkkLs_sWN1A@mail.gmail.com> Message-ID: <20150113023911.GG10173@ando.pearwood.info> On Mon, Jan 12, 2015 at 06:12:05PM -0800, Guido van Rossum wrote: > But what other methods from Decimal would we have to add to float (and to > int, etc.)? > > IIRC there are just too many differences between Decimal and float to ever > hope for equivalence. (How do you spell sin() of a Decimal?) Ideally you would spell it num.sin(), if anyone ever gets around to writing a sin routine for Decimals that doesn't go through conversion of floats first. But I'm not asking for Decimal to support every function and method that floats do. I'm asking that where Decimal and float both support the same function, that be a method rather than a function for one and a method for the other. I'm very glad that Python doesn't *require* object-oriented code for all things, as Java does, but there are cases where mixing procedural and OO interfaces is a nuisance. Now that Python has a full numeric tower, having the same function be a method for some numbers and a function for others makes it difficult to write type-agnostic code. As an alternative to float methods, perhaps the math module could be more polymorphic, so we could write things like (say): math.sqrt(some_decimal) and get a Decimal result instead of a float. I don't particularly care whether I call a function or a method, so long as I don't have to call isinstance() first. Steve > On Mon, Jan 12, 2015 at 5:49 PM, Steven D'Aprano <steve at pearwood.info> > wrote: > > > On Mon, Jan 12, 2015 at 01:40:20PM -0600, Zachary Ware wrote: > > > On Mon, Jan 12, 2015 at 1:33 PM, MRAB <python at mrabarnett.plus.com> > > wrote: > > > > I was going to say that calling it "isnan" is in keeping with > > > > "isdigit", etc, but those are instance methods! > > > > > > > > So, why is it "math.isnan(x)" and not "x.isnan()"? > > > > > > I'm a little curious about this myself, since float.is_nan(), > > > float.is_inf(), and float.is_finite() are all implemented, but #if 0'd > > > out. > > > > Can we re-start this issue? > > > > http://bugs.python.org/issue18842 > > > > > > I have a lot of code that takes either a decimal or a float, and I > > currently have to write things like: > > > > if isinstance(x, Decimal): > > isnan = x.is_nan() > > else: > > isnan = math.isnan(x) > > > > which is just ick. > > > > > > If you're wondering why I don't just unconditionally call math.isnan, > > consider this: > > > > py> from decimal import Decimal > > py> x = Decimal('snan') > > py> float(x) > > Traceback (most recent call last): > > File "<stdin>", line 1, in <module> > > ValueError: cannot convert signaling NaN to float > > > > > > -- > > Steve > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > > -- > --Guido van Rossum (python.org/~guido) From alexander.belopolsky at gmail.com Tue Jan 13 03:52:28 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 12 Jan 2015 21:52:28 -0500 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CAPJVwBmOMrdataFhhtCSiY=4Au-9=wa=fpKhz4j9cdeUPkCnvw@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <CAP7+vJL-SWXr01GQRGJsqWmyrN9s+KY6+EqfEcQ=f9PbPswJjQ@mail.gmail.com> <CALGmxEKs_k7wRks+HhBQqvNkq9TYzmo5CRJgr_pGFkQL0Q06Hg@mail.gmail.com> <CAP7+vJLtNO3+qv_VXezcFDpfAywgjA5pU+BJV32q2yDLJ72FyA@mail.gmail.com> <CALGmxEJ8PPWdoV6kRy3RY5x-ddLysfmAegVoE+MP8u22sdx-tw@mail.gmail.com> <CAP7+vJ+MEZxPOcvH3MBfBNpD0sVgZPRrJWN++ahZyg_fnO_VVw@mail.gmail.com> <CALGmxEKP2auDojROYz=Lc-gVG72xxqv88PMahvE-iw1mc3_HSA@mail.gmail.com> <CALGmxEK85cr7tG5hU6TFoBPdFEL-J8H9rrGRr-6HG8EqHKmhoA@mail.gmail.com> <CAPJVwBmOMrdataFhhtCSiY=4Au-9=wa=fpKhz4j9cdeUPkCnvw@mail.gmail.com> Message-ID: <CAP7h-xZCWXvyvLeTmzi+iLeqHd3=n+XQE2O74ENV1eZyVTJ+sw@mail.gmail.com> On Mon, Jan 12, 2015 at 9:32 PM, Nathaniel Smith <njs at pobox.com> wrote: > I don't want to step on your positive let's-fix-it attitude, I think > that's great, but... I do think you are really underestimating the > complexity of the problem. Didn't IEEE 754 solve this problem for us? In IEEE 754 the bijection between floats and integers defined by reading float bits as a 64-bit unsigned integer preserves the total ordering and therefore the "closeness". For finite floats that are not too far apart the difference between the corresponding ints is the same as the difference in ulps. I think something like abs(x.view(int) - y.view(int)) < ulps would be a reasonable implementation for is_close(x, y). -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150112/03d58ff9/attachment-0001.html> From casevh at gmail.com Tue Jan 13 03:57:55 2015 From: casevh at gmail.com (Case Van Horsen) Date: Mon, 12 Jan 2015 18:57:55 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <20150113023911.GG10173@ando.pearwood.info> References: <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> <8761cczswd.fsf@uwakimon.sk.tsukuba.ac.jp> <-3824075524171795640@unknownmsgid> <87387gyofe.fsf@uwakimon.sk.tsukuba.ac.jp> <CALGmxEJ8h28B8JymErZ1aRQ5+5P-pikUBLKFSOTwWHfRYtkB8g@mail.gmail.com> <CAP7+vJLvFtTQaXtsHOV=_EJA0H62Jqj77TPNbupWMdDOrE8rxg@mail.gmail.com> <54B42184.4050702@mrabarnett.plus.com> <CAKJDb-NfTzNyax4XkhFFBbHA12LGn=DCC3fsNFK9LMRd_pu9aw@mail.gmail.com> <20150113014912.GF10173@ando.pearwood.info> <CAP7+vJJ28v++cw85NWubg8HzkRRwfZv7zTP4FVchkkLs_sWN1A@mail.gmail.com> <20150113023911.GG10173@ando.pearwood.info> Message-ID: <CANerV6=jfZDn3dXsd37A6Yh8k+xMb_KWmCjG3fjL0PQb8xmXYw@mail.gmail.com> On Mon, Jan 12, 2015 at 6:39 PM, Steven D'Aprano <steve at pearwood.info> wrote: > > I'm very glad that Python doesn't *require* object-oriented code for all > things, as Java does, but there are cases where mixing procedural and OO > interfaces is a nuisance. Now that Python has a full numeric tower, > having the same function be a method for some numbers and a function for > others makes it difficult to write type-agnostic code. > > As an alternative to float methods, perhaps the math module could be > more polymorphic, so we could write things like (say): > > math.sqrt(some_decimal) > > and get a Decimal result instead of a float. I don't particularly care > whether I call a function or a method, so long as I don't have to call > isinstance() first. > I proposed something similar in https://mail.python.org/pipermail/python-ideas/2011-November/012844.html Summary: math.trunc, math.ceil, and math.floor currently look for __trunc__, __ceil__, and __floor__ methods on the argument. This could extended to add a check for __sqrt__, __is_nan__, __sin__, etc. special methods. casevh From alexander.belopolsky at gmail.com Tue Jan 13 04:02:12 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 12 Jan 2015 22:02:12 -0500 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <20150113023911.GG10173@ando.pearwood.info> References: <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> <8761cczswd.fsf@uwakimon.sk.tsukuba.ac.jp> <-3824075524171795640@unknownmsgid> <87387gyofe.fsf@uwakimon.sk.tsukuba.ac.jp> <CALGmxEJ8h28B8JymErZ1aRQ5+5P-pikUBLKFSOTwWHfRYtkB8g@mail.gmail.com> <CAP7+vJLvFtTQaXtsHOV=_EJA0H62Jqj77TPNbupWMdDOrE8rxg@mail.gmail.com> <54B42184.4050702@mrabarnett.plus.com> <CAKJDb-NfTzNyax4XkhFFBbHA12LGn=DCC3fsNFK9LMRd_pu9aw@mail.gmail.com> <20150113014912.GF10173@ando.pearwood.info> <CAP7+vJJ28v++cw85NWubg8HzkRRwfZv7zTP4FVchkkLs_sWN1A@mail.gmail.com> <20150113023911.GG10173@ando.pearwood.info> Message-ID: <CAP7h-xZ0gNXvjz=Zrav_iKc3ORY8mX5-c1WvYf7noz2i2kRL-g@mail.gmail.com> On Mon, Jan 12, 2015 at 9:39 PM, Steven D'Aprano <steve at pearwood.info> wrote: > As an alternative to float methods, perhaps the math module could be > more polymorphic, so we could write things like (say): > > math.sqrt(some_decimal) > Something like this is trivial to implement using the singledispatch module, but when you are faced with the functions of two arguments such as pow, things get hairier. What type should pow(2.5, Decimal(2)) return? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150112/c7b3fb12/attachment.html> From njs at pobox.com Tue Jan 13 04:04:19 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 13 Jan 2015 03:04:19 +0000 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CAP7h-xZCWXvyvLeTmzi+iLeqHd3=n+XQE2O74ENV1eZyVTJ+sw@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <CAP7+vJL-SWXr01GQRGJsqWmyrN9s+KY6+EqfEcQ=f9PbPswJjQ@mail.gmail.com> <CALGmxEKs_k7wRks+HhBQqvNkq9TYzmo5CRJgr_pGFkQL0Q06Hg@mail.gmail.com> <CAP7+vJLtNO3+qv_VXezcFDpfAywgjA5pU+BJV32q2yDLJ72FyA@mail.gmail.com> <CALGmxEJ8PPWdoV6kRy3RY5x-ddLysfmAegVoE+MP8u22sdx-tw@mail.gmail.com> <CAP7+vJ+MEZxPOcvH3MBfBNpD0sVgZPRrJWN++ahZyg_fnO_VVw@mail.gmail.com> <CALGmxEKP2auDojROYz=Lc-gVG72xxqv88PMahvE-iw1mc3_HSA@mail.gmail.com> <CALGmxEK85cr7tG5hU6TFoBPdFEL-J8H9rrGRr-6HG8EqHKmhoA@mail.gmail.com> <CAPJVwBmOMrdataFhhtCSiY=4Au-9=wa=fpKhz4j9cdeUPkCnvw@mail.gmail.com> <CAP7h-xZCWXvyvLeTmzi+iLeqHd3=n+XQE2O74ENV1eZyVTJ+sw@mail.gmail.com> Message-ID: <CAPJVwBmqBjwxREvOw3XS03yf1rRegFaRJNJ1e61D83qPTai_JQ@mail.gmail.com> On 13 Jan 2015 02:52, "Alexander Belopolsky" <alexander.belopolsky at gmail.com> wrote: > > On Mon, Jan 12, 2015 at 9:32 PM, Nathaniel Smith <njs at pobox.com> wrote: >> >> I don't want to step on your positive let's-fix-it attitude, I think that's great, but... I do think you are really underestimating the complexity of the problem. > > > Didn't IEEE 754 solve this problem for us? In IEEE 754 the bijection between floats and integers defined by reading float bits as a 64-bit unsigned integer preserves the total ordering and therefore the "closeness". For finite floats that are not too far apart the difference between the corresponding ints is the same as the difference in ulps. > > I think something like abs(x.view(int) - y.view(int)) < ulps would be a reasonable implementation for is_close(x, y). This also has potentially non-intuitive behaviour near zero, and at the least you need to do some thinking about how to handle nans and infs. This seems to be a good article that covers a lot of these kinds of subtleties: https://randomascii.wordpress.com/2012/02/25/comparing-floating-point-numbers-2012-edition/ (Ulps also require a lot more user sophistication to understand at all, as compared to absolute or relative tolerances.) It does seem like adding some way to compute ulp differences to the math module might be useful -- regardless of whether it's a panacea, it is a useful option to have, and AFAIK there is no reasonable way to do the computation in pure python currently. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150113/becdd464/attachment.html> From casevh at gmail.com Tue Jan 13 05:05:44 2015 From: casevh at gmail.com (Case Van Horsen) Date: Mon, 12 Jan 2015 20:05:44 -0800 Subject: [Python-ideas] math.inf and math.nan constants In-Reply-To: <CAP7h-xZ0gNXvjz=Zrav_iKc3ORY8mX5-c1WvYf7noz2i2kRL-g@mail.gmail.com> References: <CALGmxE+Xvwzyu_b_C=PKP+hMUbF93TRMKyxFoL-FSQLagSB3Jg@mail.gmail.com> <8761cczswd.fsf@uwakimon.sk.tsukuba.ac.jp> <-3824075524171795640@unknownmsgid> <87387gyofe.fsf@uwakimon.sk.tsukuba.ac.jp> <CALGmxEJ8h28B8JymErZ1aRQ5+5P-pikUBLKFSOTwWHfRYtkB8g@mail.gmail.com> <CAP7+vJLvFtTQaXtsHOV=_EJA0H62Jqj77TPNbupWMdDOrE8rxg@mail.gmail.com> <54B42184.4050702@mrabarnett.plus.com> <CAKJDb-NfTzNyax4XkhFFBbHA12LGn=DCC3fsNFK9LMRd_pu9aw@mail.gmail.com> <20150113014912.GF10173@ando.pearwood.info> <CAP7+vJJ28v++cw85NWubg8HzkRRwfZv7zTP4FVchkkLs_sWN1A@mail.gmail.com> <20150113023911.GG10173@ando.pearwood.info> <CAP7h-xZ0gNXvjz=Zrav_iKc3ORY8mX5-c1WvYf7noz2i2kRL-g@mail.gmail.com> Message-ID: <CANerV6k_3NEpeztkyK6ue16W4qfKXAKJsZb7nns5J4WYJhLHFA@mail.gmail.com> On Mon, Jan 12, 2015 at 7:02 PM, Alexander Belopolsky <alexander.belopolsky at gmail.com> wrote: > > On Mon, Jan 12, 2015 at 9:39 PM, Steven D'Aprano <steve at pearwood.info> > wrote: >> >> As an alternative to float methods, perhaps the math module could be >> more polymorphic, so we could write things like (say): >> >> math.sqrt(some_decimal) > > > Something like this is trivial to implement using the singledispatch module, > but when you are faced with the functions of two arguments such as pow, > things get hairier. What type should pow(2.5, Decimal(2)) return? > I've thought about about the following for pow(x,y), but it would break too much existing code (i.e. when x and y are integers). if isinstance(x, float) and isinstance(y, float): return the float x**y try: return x.__pow__(x,y): except TypeError: pass try: return y.__pow__(x,y) except TypeError: pass return float(x)**float(y) It would be nice we could distinguish "real" numbers where a real number uses a radix, a significand, and an exponent to represent a number. A float uses a radix of 2, a significant of 53 bits, and limited exponent range. A Decimal uses a radix of 10, a significand of arbitrary precision, and a (usually) wide exponent range. Both float and Decimal would be instances of real numbers. Then the following might work: if isinstance(x, real) and isinstance(y, real): try: return x.__pow__(x,y): except TypeError: pass try: return y.__pow__(x,y) except TypeError: pass return float(x)**float(y) casevh > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From chris.barker at noaa.gov Tue Jan 13 08:12:12 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 12 Jan 2015 23:12:12 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CAPJVwBmOMrdataFhhtCSiY=4Au-9=wa=fpKhz4j9cdeUPkCnvw@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <CAP7+vJL-SWXr01GQRGJsqWmyrN9s+KY6+EqfEcQ=f9PbPswJjQ@mail.gmail.com> <CALGmxEKs_k7wRks+HhBQqvNkq9TYzmo5CRJgr_pGFkQL0Q06Hg@mail.gmail.com> <CAP7+vJLtNO3+qv_VXezcFDpfAywgjA5pU+BJV32q2yDLJ72FyA@mail.gmail.com> <CALGmxEJ8PPWdoV6kRy3RY5x-ddLysfmAegVoE+MP8u22sdx-tw@mail.gmail.com> <CAP7+vJ+MEZxPOcvH3MBfBNpD0sVgZPRrJWN++ahZyg_fnO_VVw@mail.gmail.com> <CALGmxEKP2auDojROYz=Lc-gVG72xxqv88PMahvE-iw1mc3_HSA@mail.gmail.com> <CALGmxEK85cr7tG5hU6TFoBPdFEL-J8H9rrGRr-6HG8EqHKmhoA@mail.gmail.com> <CAPJVwBmOMrdataFhhtCSiY=4Au-9=wa=fpKhz4j9cdeUPkCnvw@mail.gmail.com> Message-ID: <CALGmxELx_P5Y-peUA83AK6CCgvyQf9gLggiskBWNOJpwHBEeag@mail.gmail.com> On Mon, Jan 12, 2015 at 6:32 PM, Nathaniel Smith <njs at pobox.com> wrote: > Ok, but you're missing a huge issue, which is that according to the rule > you suggest, 0.0 is not close to any non-zero value. > Well, I did say I just threw something together (it's a gist, not a pull request!). and asked for test cases that would push it. At best it could be a starting point for experimentation and discussion - though Steven's code is probably a much better place to start. Near zero clearly is one case to be considered. Perhaps that's why numpy used a relative AND absolute tolerance together -- I'll need to take a closer look at that. > Handing such a function to users and claiming it's the one "is close" > predicate they need is going to produce a lot of sad and confused users. > I don't think we'll be able to come up with anything that is the "one" predicate anyone will even need. But I do think that it would be good to provide something that is useful for common cases, and more thought out than most folks will bother to write on their own. > I don't want to step on your positive let's-fix-it attitude, I think > that's great, but... I do think you are really underestimating the > complexity of the problem. > I really didn't think I was done ;-) > The right way to handle comparison with zero, or comparison between > epsilon and negative epsilon, etc, really depends on the context. For > common unit test cases it may even make sense to prefer a non-symmetric > predicate -- e.g. if you're expecting 0 then 1e-100 is usually fine, but if > you're expecting 1e-100 then getting 0 might be a huge problem. > Sure, but no matter how you slice it, users will need to think. In fact, if you are expecting zero, you can write that test yourself pretty easily. Though that may be an example of why you want to provide an absolute tolerance method as well. It would be more appropriate if you are expecting zero. >From Bruce Dawson's article: "Zero is a huge nuisance. Avoid it if possible." I don't love it, but numpy's allclose() has worked fine for me, and many other people, certainly well enough that I haven't bothered to write my own. Also, thinking a bit about complex -- complex abs() is easy to think about, but maybe not as rigorous as both components being individually. For example: 1e100+0.1j and 1e100+10j -- should those be "close"? They are in the absolute value sense, but the imaginary part is not in the least. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150112/9a6d0112/attachment-0001.html> From storchaka at gmail.com Tue Jan 13 08:57:30 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 13 Jan 2015 09:57:30 +0200 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CAP7h-xZi2cC=d7moYPFpgCcU0xio2QbGWhGG5Anv6mgfqAF8Yg@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <CAP7+vJL-SWXr01GQRGJsqWmyrN9s+KY6+EqfEcQ=f9PbPswJjQ@mail.gmail.com> <CALGmxEKs_k7wRks+HhBQqvNkq9TYzmo5CRJgr_pGFkQL0Q06Hg@mail.gmail.com> <CAP7+vJLtNO3+qv_VXezcFDpfAywgjA5pU+BJV32q2yDLJ72FyA@mail.gmail.com> <CALGmxEJ8PPWdoV6kRy3RY5x-ddLysfmAegVoE+MP8u22sdx-tw@mail.gmail.com> <CAP7+vJ+MEZxPOcvH3MBfBNpD0sVgZPRrJWN++ahZyg_fnO_VVw@mail.gmail.com> <CALGmxEKP2auDojROYz=Lc-gVG72xxqv88PMahvE-iw1mc3_HSA@mail.gmail.com> <CABicbJL-RyNxtSXmq=iVm9spvuwAZ5DCAsEXWPvxdz4fr64dnA@mail.gmail.com> <CABicbJJwTTBgp1AS0B0JhsrRd9yYeZ1M=QJbSo4kRUoA6kct=w@mail.gmail.com> <1E935C35-BEDE-4865-80B1-AF18E80AC654@yahoo.com> <CAP7h-xZi2cC=d7moYPFpgCcU0xio2QbGWhGG5Anv6mgfqAF8Yg@mail.gmail.com> Message-ID: <m92j5a$qnv$1@ger.gmane.org> On 13.01.15 02:52, Alexander Belopolsky wrote: > Once you import float nan or inf, it is obvious how to build an complex > value containing them using x + y*1j expression, but having to import > nan and inf separately when you import everything else from cmath is an > annoyance. Note that x + y*1j is not equivalent to complex(x, y). >>> complex(0, math.inf) infj >>> math.inf*1j (nan+infj) It's because 1j == 0 + 1j and 0*inf is nan. > For example > > >>> cmath.exp(inf*1j) > (nan+nanj) >>> cmath.exp(complex(0, math.inf)) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: math domain error From dickinsm at gmail.com Tue Jan 13 10:28:39 2015 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 13 Jan 2015 09:28:39 +0000 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <20150113013452.GE10173@ando.pearwood.info> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> Message-ID: <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> On Tue, Jan 13, 2015 at 1:34 AM, Steven D'Aprano <steve at pearwood.info> wrote: > Unfortunately a naive ULP comparison has trouble with NANs, INFs, and > numbers close to zero, especially if they have opposite signs. The > smallest representable denormalised floats larger, and smaller, than > zero are: > > 5e-324 > -5e-324 > > These are the smallest magnitude floats apart from zero, so we might > hope that they are considered "close together", but they actually differ > by 9223372036854775808 ULP. Ouch. > Only with a naive (i.e., wrong :-) implementation. Those two floats differ by precisely 2 units in the last place, and any correct implementation should report that. It's not hard to write code that deals correctly with opposite signs. Here's a simple difference_in_ulps function that correctly reports the number of ulps difference between any two finite floats. >>> import struct >>> def to_ulps(x): ... n = struct.unpack('<q', struct.pack('<d', x))[0] ... return -(n + 2**63) if n < 0 else n ... >>> def difference_in_ulps(x, y): ... return abs(to_ulps(x) - to_ulps(y)) ... >>> difference_in_ulps(-5e-324, 5e-324) 2 This is almost exactly what's in Lib/test/test_math.py already, except that the function there is better documented and uses "~(n + 2**63)" instead of "-(n + 2**63)" in the negative n correction branch, which has the effect of regarding 0.0 and -0.0 as 1 ulp apart. Comparing by ulps was what I needed for testing library-quality functions for the math and cmath modules; I doubt that it's what's needed for most comparison tasks. I'd expect the suggested combination of relative error and absolute error to be more appropriate most of the time. -- Mark > > I have some ideas for dealing with that, and if anyone is interested I'm > happy to talk about it, but they're not ready for production yet. > > I think that the Bruce Dawson is right. Floating point comparisons are > hard, really hard. I know that I've still got a lot to learn about it. I > can think of at least five different ways to compare floats for > equality, and they all have their uses: > > - exact equality using == > - absolute error tolerances > - relative error tolerances > - ULP comparisons > - the method unittest uses, using round() > > > I'm explicitly including == because it is a floating point superstition > that one should never under any circumstances compare floats for exact > equality. As general advice, "don't use == unless you know what you are > doing" is quite reasonable, but it's the "never use" that turns it into > superstition. As Bruce Dawson says, "Floating-point numbers aren?t > cursed", and throwing epsilons into a problem where no epsilon is needed > is a bad idea. > > > https://randomascii.wordpress.com/2012/06/26/doubles-are-not-floats-so-dont-compare-them/ > > > Aside: I'm reminded of APL, which mandates fuzzy equality (i.e. with a > tolerance) of floating point numbers: > > In an early talk Ken [Iverson] was explaining the advantages > of tolerant comparison. A member of the audience asked > incredulously, ?Surely you don?t mean that when A=B and B=C, > A may not equal C?? Without skipping a beat, Ken replied, > ?Any carpenter knows that!? and went on to the next question. > - Paul Berry > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150113/d978b88a/attachment.html> From dickinsm at gmail.com Tue Jan 13 10:54:04 2015 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 13 Jan 2015 09:54:04 +0000 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CAP7h-xZi2cC=d7moYPFpgCcU0xio2QbGWhGG5Anv6mgfqAF8Yg@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <CAP7+vJL-SWXr01GQRGJsqWmyrN9s+KY6+EqfEcQ=f9PbPswJjQ@mail.gmail.com> <CALGmxEKs_k7wRks+HhBQqvNkq9TYzmo5CRJgr_pGFkQL0Q06Hg@mail.gmail.com> <CAP7+vJLtNO3+qv_VXezcFDpfAywgjA5pU+BJV32q2yDLJ72FyA@mail.gmail.com> <CALGmxEJ8PPWdoV6kRy3RY5x-ddLysfmAegVoE+MP8u22sdx-tw@mail.gmail.com> <CAP7+vJ+MEZxPOcvH3MBfBNpD0sVgZPRrJWN++ahZyg_fnO_VVw@mail.gmail.com> <CALGmxEKP2auDojROYz=Lc-gVG72xxqv88PMahvE-iw1mc3_HSA@mail.gmail.com> <CABicbJL-RyNxtSXmq=iVm9spvuwAZ5DCAsEXWPvxdz4fr64dnA@mail.gmail.com> <CABicbJJwTTBgp1AS0B0JhsrRd9yYeZ1M=QJbSo4kRUoA6kct=w@mail.gmail.com> <1E935C35-BEDE-4865-80B1-AF18E80AC654@yahoo.com> <CAP7h-xZi2cC=d7moYPFpgCcU0xio2QbGWhGG5Anv6mgfqAF8Yg@mail.gmail.com> Message-ID: <CAAu3qLUkK+ECTs1XkdX-DfQjXXsujJjDTSBiM+1eBAWt40W2vg@mail.gmail.com> [Responding to everyone this time; apologies to Alexander.] On Tue, Jan 13, 2015 at 12:52 AM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > BTW, as we are discussing math, cmath and non-finite floats, maybe we > could clean up some inconsistencies. > > For example > > >>> cmath.exp(inf*1j) > (nan+nanj) > > but > > >>> cmath.sin(inf) > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > ValueError: math domain error > >>> cmath.cos(inf) > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > ValueError: math domain error > There's no inconsistency here: those are all correct. Note that "inf * 1j" is *not* complex(0, inf): it's a complex nan (complex(nan, inf), to be precise), and exp applied to that correctly gives another complex nan, with no exception. You'd see the same with cmath.sin(inf*1j). Note that cmath.exp(complex(0, float('inf'))) *does* give a ValueError. -- Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150113/0f313ab3/attachment.html> From steve at pearwood.info Tue Jan 13 13:06:13 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 13 Jan 2015 23:06:13 +1100 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> Message-ID: <20150113120613.GJ10173@ando.pearwood.info> On Tue, Jan 13, 2015 at 09:28:39AM +0000, Mark Dickinson wrote: > On Tue, Jan 13, 2015 at 1:34 AM, Steven D'Aprano <steve at pearwood.info> > wrote: > > > Unfortunately a naive ULP comparison has trouble with NANs, INFs, and > > numbers close to zero, especially if they have opposite signs. The > > smallest representable denormalised floats larger, and smaller, than > > zero are: > > > > 5e-324 > > -5e-324 > > > > These are the smallest magnitude floats apart from zero, so we might > > hope that they are considered "close together", but they actually differ > > by 9223372036854775808 ULP. Ouch. > > > > Only with a naive (i.e., wrong :-) implementation. Well, I said I was still learning :-) In my defence, I was influenced by Bruce Dawson who said you can't do ulps comparisons if the signs are not the same. > Those two floats differ > by precisely 2 units in the last place, and any correct implementation > should report that. It's not hard to write code that deals correctly with > opposite signs. How do you deal with cases where one number is normalised and the other is denormalised? E.g. I think most people would consider these two numbers to be close together: py> to_ulps(2.2250738585072014e-308) - to_ulps(1.1125369292536007e-308) 2251799813685248 -- Steve From chris.barker at noaa.gov Tue Jan 13 16:53:37 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 13 Jan 2015 07:53:37 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> Message-ID: <-8174570845814181397@unknownmsgid> > On Jan 13, 2015, at 1:29 AM, Mark Dickinson <dickinsm at gmail.com> wrote: > > > Comparing by ulps was what I needed for testing library-quality functions for the math and cmath modules; I doubt that it's what's needed for most comparison tasks. That's the conclusion I was coming to. Ulps are likely to be the right way to do it if your trying to understand/test the accuracy of an algorithm, but not for general "did I get a close enough result". And it would be a lot harder to understand for most of us. As for comparing to zero -- in reading about this, it seems there simply is no general solution -- only the user knows what they want. So the only thing to do is a big warning in the docs about it, and providing an absolute tolerance option. Should that a separate function or a flag? This is actually a harder problem for numpy, as it's an array function, so you need to have the same function/parameters for every value in the array, some of which may be near zero. I haven't thought it out yet, but maybe we could specify an absolute tolerance near zero, and a relative tolerance elsewhere, both at once. Tricky to document, even if possible. > I'd expect the suggested combination of relative error and absolute error to be more appropriate most of the time. And most of the time is what we are going for. -Chris From ron3200 at gmail.com Wed Jan 14 03:57:38 2015 From: ron3200 at gmail.com (Ron Adam) Date: Tue, 13 Jan 2015 20:57:38 -0600 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <-8174570845814181397@unknownmsgid> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> Message-ID: <m94lv3$h9d$1@ger.gmane.org> On 01/13/2015 09:53 AM, Chris Barker - NOAA Federal wrote: >> >On Jan 13, 2015, at 1:29 AM, Mark Dickinson<dickinsm at gmail.com> wrote: >> > >> > >> >Comparing by ulps was what I needed for testing library-quality functions for the math and cmath modules; I doubt that it's what's needed for most comparison tasks. > That's the conclusion I was coming to. Ulps are likely to be the right > way to do it if your trying to understand/test the accuracy of an > algorithm, but not for general "did I get a close enough result". And > it would be a lot harder to understand for most of us. > > As for comparing to zero -- in reading about this, it seems there > simply is no general solution -- only the user knows what they want. > So the only thing to do is a big warning in the docs about it, and > providing an absolute tolerance option. > > Should that a separate function or a flag? > > This is actually a harder problem for numpy, as it's an array > function, so you need to have the same function/parameters for every > value in the array, some of which may be near zero. > > I haven't thought it out yet, but maybe we could specify an absolute > tolerance near zero, and a relative tolerance elsewhere, both at once. > Tricky to document, even if possible. Doesn't this problem come up at any boundary comparison, and not just zero? So isn't the issue about any n distance from any floating point number that is less than 1 ulp? And in that regard, comparison to zero is no different than any comparison to any other floating point value? Just trying to follow along, -Ron >> > I'd expect the suggested combination of relative error and absolute error to be more appropriate most of the time. > And most of the time is what we are going for. From chris.barker at noaa.gov Wed Jan 14 04:25:26 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 13 Jan 2015 19:25:26 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <m94lv3$h9d$1@ger.gmane.org> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> Message-ID: <6011660376447291684@unknownmsgid> > On Jan 13, 2015, at 6:58 PM, Ron Adam <ron3200 at gmail.com> wrote: >> maybe we could specify an absolute >> tolerance near zero, and a relative tolerance elsewhere, both at once. >> Tricky to document, even if possible. > > Doesn't this problem come up at any boundary comparison, and not just zero? Zero is special because you lose the ability to use a relative tolerance. Everything is huge compared to zero. > So isn't the issue about any n distance from any floating point number that is less than 1 ulp? I'm still a bit fuzzy on Ulps, but it seems the goal here is to define a tolerance larger than an ulp. This is for the use case where we expect multiple rounding errors -- many more than one ulp, That's why I think the use case for ulp comparisons is more about assessment of accuracy of algorithms than "did I introduce a big old bug?" or, "is this computed value close enough to what I measured?" -Chris From ron3200 at gmail.com Wed Jan 14 06:12:44 2015 From: ron3200 at gmail.com (Ron Adam) Date: Tue, 13 Jan 2015 23:12:44 -0600 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <6011660376447291684@unknownmsgid> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <6011660376447291684@unknownmsgid> Message-ID: <m94tsd$uav$1@ger.gmane.org> On 01/13/2015 09:25 PM, Chris Barker - NOAA Federal wrote: >> >On Jan 13, 2015, at 6:58 PM, Ron Adam<ron3200 at gmail.com> wrote: >>> >> maybe we could specify an absolute >>> >>tolerance near zero, and a relative tolerance elsewhere, both at once. >>> >>Tricky to document, even if possible. >>Doesn't this problem come up at any boundary comparison, and not just zero? > Zero is special because you lose the ability to use a relative > tolerance. Everything is huge compared to zero. After I posted I realised that when you compare anything you subtract what you are comparing to, and if it's equal to zero, then it's equal to what you are comparing to. So testing against zero is fundamental to all comparisons, is this is correct? Wouldn't a relative tolerance be set relative to some value that is *not* zero? And then used when comparing any close values, including zero. >> >So isn't the issue about any n distance from any floating point number that is less than 1 ulp? > I'm still a bit fuzzy on Ulps, but it seems the goal here is to define > a tolerance larger than an ulp. This is for the use case where we > expect multiple rounding errors -- many more than one ulp, > That's why I think the use case for ulp comparisons is more about > assessment of accuracy of algorithms than "did I introduce a big old > bug?" or, "is this computed value close enough to what I measured?" I haven't looked into the finer aspects of ulps myself. It seems to me ulps only matter if the exponent part of two floating point numbers are equal and the value part is within 1 (or a small few) ulps of each other. Then there may be problems determining if they are equal, or one is greater or less than the other. And only if there is no greater tolerance value set. A couple of thoughts come to mind. I'm not sure if they are relevant though. I think for math algorithms that are hard coded in a program, it isn't much of an issue as the author would have a feel for the size of a delta if needed, and output of an appropriate number of significant digits. and can calculate the error range if needed as well. That probably fits most situations and is what is typically done. It seems to me, that automated tracking and/or use of these things may be wanted with equation solvers. The solver would determine the delta and significant digits from it's initial data. That sounds like it could get very complex, but maybe making this easier to do is the point? Cheers, Ron From abarnert at yahoo.com Wed Jan 14 07:10:39 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 13 Jan 2015 22:10:39 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <m94tsd$uav$1@ger.gmane.org> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <6011660376447291684@unknownmsgid> <m94tsd$uav$1@ger.gmane.org> Message-ID: <09BB22C6-168A-43ED-9BF5-F731FDEF5240@yahoo.com> On Jan 13, 2015, at 21:12, Ron Adam <ron3200 at gmail.com> wrote: > On 01/13/2015 09:25 PM, Chris Barker - NOAA Federal wrote: >>> >On Jan 13, 2015, at 6:58 PM, Ron Adam<ron3200 at gmail.com> wrote: > >>>> >> maybe we could specify an absolute >>>> >>tolerance near zero, and a relative tolerance elsewhere, both at once. >>>> >>Tricky to document, even if possible. > >>> Doesn't this problem come up at any boundary comparison, and not just zero? > >> Zero is special because you lose the ability to use a relative >> tolerance. Everything is huge compared to zero. > > After I posted I realised that when you compare anything you subtract what you are comparing to, and if it's equal to zero, then it's equal to what you are comparing to. So testing against zero is fundamental to all comparisons, is this is correct? > > Wouldn't a relative tolerance be set relative to some value that is *not* zero? And then used when comparing any close values, including zero. I think you're missing the meanings of these terms. Absolute tolerance means a fixed tolerance, like 1e-5; relative tolerance means you pick a tolerance that's relative to the values being compared--think of it as a percentage, say, 0.01%. Plenty of values are within +/- 1e-5 of 0. But the only value within +/- 0.01% of 0 is 0 itself. Another way to look at it: x is close to y with absolute tolerance 1e-5 if abs(x-y) < 1e-5. x is close to y with relative tolerance 1e-5 if abs(x-y)/y < 1e-5. So, no value is ever close to 0 within any relative tolerance. >>> >So isn't the issue about any n distance from any floating point number that is less than 1 ulp? >> I'm still a bit fuzzy on Ulps, but it seems the goal here is to define >> a tolerance larger than an ulp. This is for the use case where we >> expect multiple rounding errors -- many more than one ulp, >> That's why I think the use case for ulp comparisons is more about >> assessment of accuracy of algorithms than "did I introduce a big old >> bug?" or, "is this computed value close enough to what I measured?" > > I haven't looked into the finer aspects of ulps myself. It seems to me ulps > only matter if the exponent part of two floating point numbers are equal > and the value part is within 1 (or a small few) ulps of each other. Then > there may be problems determining if they are equal, or one is greater or > less than the other. And only if there is no greater tolerance value set. No, this is wrong. First, 1.1e1 and 1.0e10 (binary) are only off by 1 ulp even though they have different exponents. Second, there's never any problem determining if two finite numbers are equal or which one is greater. The issue is determining whether two numbers +/- their error bars are too close to call vs. unambiguously greater or lesser. For example, if I get 1.0e1 and 1.1e1 (assuming 1 bit mantissa for ease of discussion), the latter is clearly greater--but if I have 2 ulp of error, or an absolute error of 1e1, or a relative error of 50%, the fact that the latter is greater is irrelevant--each value is within the other value's error range. But if I get 1.0e1 and 1.0e10 with the same error, then I can say that the latter is unambiguously greater. I think Chris is right that ulp comparisons usually only come up in testing an algorithm. You have to actually do an error analysis, and you have to have input data with a precision specified in ulp, or you're not going to get a tolerance in ulp. When you want to verify that you've correctly implemented an algorithm that guarantees to multiply input ulp by no more than 4, you can feed in numbers with +/- 1 ulp error (just typing in decimal numbers does that) and verify that the results are within +/- 4 ulp. But when you have real data, or inherent rounding issues, or an error analysis that's partly made up of rules of thumb and winging it, you're almost always going to end up with absolute or relative error instead. (Or, occasionally, something more complicated that you have to code up manually, like logarithmic relative error.) > > A couple of thoughts come to mind. I'm not sure if they are relevant though. > > I think for math algorithms that are hard coded in a program, it isn't much of an issue as the author would have a feel for the size of a delta if needed, and output of an appropriate number of significant digits. and can calculate the error range if needed as well. That probably fits most situations and is what is typically done. > > It seems to me, that automated tracking and/or use of these things may be wanted with equation solvers. The solver would determine the delta and significant digits from it's initial data. That sounds like it could get very complex, but maybe making this easier to do is the point? > > Cheers, > Ron > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From ron3200 at gmail.com Wed Jan 14 10:08:14 2015 From: ron3200 at gmail.com (Ron Adam) Date: Wed, 14 Jan 2015 03:08:14 -0600 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <09BB22C6-168A-43ED-9BF5-F731FDEF5240@yahoo.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <6011660376447291684@unknownmsgid> <m94tsd$uav$1@ger.gmane.org> <09BB22C6-168A-43ED-9BF5-F731FDEF5240@yahoo.com> Message-ID: <m95blv$bu6$1@ger.gmane.org> On 01/14/2015 12:10 AM, Andrew Barnert wrote: > On Jan 13, 2015, at 21:12, Ron Adam<ron3200 at gmail.com> wrote: > >> >On 01/13/2015 09:25 PM, Chris Barker - NOAA Federal wrote: >>>>> >>> >On Jan 13, 2015, at 6:58 PM, Ron Adam<ron3200 at gmail.com> wrote: >> > >>>>>>> >>>> >> maybe we could specify an absolute >>>>>>> >>>> >>tolerance near zero, and a relative tolerance elsewhere, both at once. >>>>>>> >>>> >>Tricky to document, even if possible. >> > >>>> >>>Doesn't this problem come up at any boundary comparison, and not just zero? >> > >>> >>Zero is special because you lose the ability to use a relative >>> >>tolerance. Everything is huge compared to zero. >> > >> >After I posted I realised that when you compare anything you subtract what you are comparing to, and if it's equal to zero, then it's equal to what you are comparing to. So testing against zero is fundamental to all comparisons, is this is correct? >> > >> >Wouldn't a relative tolerance be set relative to some value that is*not* zero? And then used when comparing any close values, including zero. > I think you're missing the meanings of these terms. > > Absolute tolerance means a fixed tolerance, like 1e-5; relative tolerance means you pick a tolerance that's relative to the values being compared--think of it as a percentage, say, 0.01%. Plenty of values are within ? 1e-5 of 0. But the only value within ? 0.01% of 0 is 0 itself. > > Another way to look at it: x is close to y with absolute tolerance 1e-5 if abs(x-y) < 1e-5. x is close to y with relative tolerance 1e-5 if abs(x-y)/y < 1e-5. So, no value is ever close to 0 within any relative tolerance. > >>>>> >>> >So isn't the issue about any n distance from any floating point number that is less than 1 ulp? >>> >>I'm still a bit fuzzy on Ulps, but it seems the goal here is to define >>> >>a tolerance larger than an ulp. This is for the use case where we >>> >>expect multiple rounding errors -- many more than one ulp, >>> >>That's why I think the use case for ulp comparisons is more about >>> >>assessment of accuracy of algorithms than "did I introduce a big old >>> >>bug?" or, "is this computed value close enough to what I measured?" >> > >> >I haven't looked into the finer aspects of ulps myself. It seems to me ulps >> >only matter if the exponent part of two floating point numbers are equal >> >and the value part is within 1 (or a small few) ulps of each other. Then >> >there may be problems determining if they are equal, or one is greater or >> >less than the other. And only if there is no greater tolerance value set. > No, this is wrong. > > First, 1.1e1 and 1.0e10 (binary) are only off by 1 ulp even though they have different exponents. I see your point about the exponent. To be clear, Are we referring to significant digits here, or ... 1.0000...1e1 and 1.0000...0e10 Where the ... is a lot of zero's to the max digits pythons floating point can handle on what ever platform it's running on. What I've read indicates ULP usually refers to the limits of the implementation/device. Significant digits has more to do with error of measurements, (and estimates), while ULPs is about accuracy limits of the hardware/software ability to calculate. > Second, there's never any problem determining if two finite numbers are > equal or which one is greater. The issue is determining whether two > numbers ? their error bars are too close to call vs. unambiguously > greater or lesser. For example, if I get 1.0e1 and 1.1e1 (assuming 1 bit > mantissa for ease of discussion), the latter is clearly greater--but if > I have 2 ulp of error, or an absolute error of 1e1, or a relative error > of 50%, the fact that the latter is greater is irrelevant--each value is > within the other value's error range. You would use the larger of the three. And possibly give a warning if the 2 ulp error is the largest. (if the application is set to do so.) I presuming the 2 ulp is twice the limit of the floating point precision here. 50% accuracy of data 1e1 limit of significant digits/measurement 2 ulp twice floating point unit of least precision > But if I get 1.0e1 and 1.0e10 with > the same error, then I can say that the latter is unambiguously > greater. Yes, This was the point I was alluding to earlier. > I think Chris is right that ulp comparisons usually only come up in > testing an algorithm. You have to actually do an error analysis, and you > have to have input data with a precision specified in ulp, or you're not > going to get a tolerance in ulp. When you want to verify that you've > correctly implemented an algorithm that guarantees to multiply input ulp by > no more than 4, you can feed in numbers with ? 1 ulp error (just typing in > decimal numbers does that) and verify that the results are within ? 4 ulp. > But when you have real data, or inherent rounding issues, or an error > analysis that's partly made up of rules of thumb and winging it, you're > almost always going to end up with absolute or relative error instead. (Or, > occasionally, something more complicated that you have to code up man > ually, like logarithmic relative error.) If the algorithm doesn't track error accumulation, then yes. This is interesting but I'm going to search for some examples of how to use some of this. I'm not sure I can add to the conversation much, but thanks for taking the time to explain some of it. Cheers, Ron From steve at pearwood.info Wed Jan 14 11:39:20 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 14 Jan 2015 21:39:20 +1100 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <m94lv3$h9d$1@ger.gmane.org> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> Message-ID: <20150114103919.GO10173@ando.pearwood.info> On Tue, Jan 13, 2015 at 08:57:38PM -0600, Ron Adam wrote: > > On 01/13/2015 09:53 AM, Chris Barker - NOAA Federal wrote: [...] > >I haven't thought it out yet, but maybe we could specify an absolute > >tolerance near zero, and a relative tolerance elsewhere, both at once. > >Tricky to document, even if possible. > > Doesn't this problem come up at any boundary comparison, and not just zero? No. A quick refresher on error tolerances... Suppose you have a value which should be exactly 0.5, but you calculate it as 0.51. Then the absolute error is 0.51-0.5 = 0.01, and the relative error is 0.01/0.5 = 0.02 (or 2%). But consider two values, 0.0 and 0.1. Then the absolute error is 0.1-0.0 = 0.1, and the relative error is 0.1/0.0 which is infinite. So 0.0 and -0.0 are problematic when dealing with relative errors. > So isn't the issue about any n distance from any floating point number that > is less than 1 ulp? And in that regard, comparison to zero is no different > than any comparison to any other floating point value? No. 1 ULP (Unit In Last Place) is the smallest possible difference between two floats. A difference of 0 ULP means the two floats are exactly equal. How it works: in mathematics, real numbers are continuous, but floats are not. There are only 2**64 floats in Python (a C double), less if you ignore the NANs and INFs, which means we can conveniently enumerate them from -(2**64) to (2**64-1), based on the internal structure of a float. So if you convert two floats into this enumerated integer value (which is equivalent to doing a type-cast from a C double to a C long) and subtract the two ints, this gives you a measure of how far apart they are. (As Mark mentioned earlier, you have to make allowance for negative floats, also INF and NANs are problematic too.) If two values are exactly equal, their "distance apart" in ULP will be zero. A distance of 1 ULP means they are consecutive floats, they cannot possibly be any closer without being equal. A distance of 2 ULP means there is only a single float separating them, and so on. Note that ULP do not directly correspond to a numeric tolerance. For example, these pairs of values are each 1 ULP apart: 0.0 and 5e-324 1.0 and 1.0000000000000002 1e300 and 1.0000000000000002e+300 So in these three cases, 1 ULP represents numeric differences of: 0.00000000000000000000...00005 0.0000000000000002 2000000000000000000000...000.0 respectively. > Just trying to follow along, A good resource is Bruce Dawson's blog RandomASCII, if you don't mind the focus on C++. Start here: https://randomascii.wordpress.com/2012/02/25/comparing-floating-point-numbers-2012-edition/ -- Steve From abarnert at yahoo.com Wed Jan 14 11:44:27 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 14 Jan 2015 02:44:27 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <m95blv$bu6$1@ger.gmane.org> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <6011660376447291684@unknownmsgid> <m94tsd$uav$1@ger.gmane.org> <09BB22C6-168A-43ED-9BF5-F731FDEF5240@yahoo.com> <m95blv$bu6$1@ger.gmane.org> Message-ID: <8257D0DC-151D-44B5-897F-A9E940DD099D@yahoo.com> On Jan 14, 2015, at 1:08, Ron Adam <ron3200 at gmail.com> wrote: > On 01/14/2015 12:10 AM, Andrew Barnert wrote: >> On Jan 13, 2015, at 21:12, Ron Adam<ron3200 at gmail.com> wrote: >> >>> >On 01/13/2015 09:25 PM, Chris Barker - NOAA Federal wrote: >>>>>> >>> >On Jan 13, 2015, at 6:58 PM, Ron Adam<ron3200 at gmail.com> wrote: >>> > >>>>>>>> >>>> >> maybe we could specify an absolute >>>>>>>> >>>> >>tolerance near zero, and a relative tolerance elsewhere, both at once. >>>>>>>> >>>> >>Tricky to document, even if possible. >>> > >>>>> >>>Doesn't this problem come up at any boundary comparison, and not just zero? >>> > >>>> >>Zero is special because you lose the ability to use a relative >>>> >>tolerance. Everything is huge compared to zero. >>> > >>> >After I posted I realised that when you compare anything you subtract what you are comparing to, and if it's equal to zero, then it's equal to what you are comparing to. So testing against zero is fundamental to all comparisons, is this is correct? >>> > >>> >Wouldn't a relative tolerance be set relative to some value that is*not* zero? And then used when comparing any close values, including zero. >> I think you're missing the meanings of these terms. >> >> Absolute tolerance means a fixed tolerance, like 1e-5; relative tolerance means you pick a tolerance that's relative to the values being compared--think of it as a percentage, say, 0.01%. Plenty of values are within ? 1e-5 of 0. But the only value within ? 0.01% of 0 is 0 itself. >> >> Another way to look at it: x is close to y with absolute tolerance 1e-5 if abs(x-y) < 1e-5. x is close to y with relative tolerance 1e-5 if abs(x-y)/y < 1e-5. So, no value is ever close to 0 within any relative tolerance. >> >>>>>> >>> >So isn't the issue about any n distance from any floating point number that is less than 1 ulp? >>>> >>I'm still a bit fuzzy on Ulps, but it seems the goal here is to define >>>> >>a tolerance larger than an ulp. This is for the use case where we >>>> >>expect multiple rounding errors -- many more than one ulp, >>>> >>That's why I think the use case for ulp comparisons is more about >>>> >>assessment of accuracy of algorithms than "did I introduce a big old >>>> >>bug?" or, "is this computed value close enough to what I measured?" >>> > >>> >I haven't looked into the finer aspects of ulps myself. It seems to me ulps >>> >only matter if the exponent part of two floating point numbers are equal >>> >and the value part is within 1 (or a small few) ulps of each other. Then >>> >there may be problems determining if they are equal, or one is greater or >>> >less than the other. And only if there is no greater tolerance value set. >> No, this is wrong. >> >> First, 1.1e1 and 1.0e10 (binary) are only off by 1 ulp even though they have different exponents. > > I see your point about the exponent. > > To be clear, Are we referring to significant digits here, or ... > > 1.0000...1e1 and 1.0000...0e10 No, those are pretty far apart. We're referring to 1.1111?1e1 and 1.0000?e10 I think your confusion here is entirely my fault. For simplicity, it's often helpful to look at tiny float representations--e.g., a 4-bit float with 1 sign bit, 1 mantissa bit, and 2 exponent bits (that, if both 1, mean inf/nan), because writing 51 0's tends to obscure what you're looking at. But it's pretty stupid to do that without mentioning that you're doing so, or how it extends to larger representations if non-obvious, and I think I was just that stupid. > What I've read indicates ULP usually refers to the limits of the implementation/device. Yes, it means "Unit of Least Precision" or "Unit in Least Place". There are a few ways to define this, but one definition is: Ignoring zeroes and denormals, two numbers are 1 ulp apart if they're finite and have the same sign and either (a) they have the same exponent and a mantissa that differs by one, or (b) they have an exponent that differs by one, the smaller one has the max mantissa and the larger the min mantissa. For zeroes, you can either define pos and neg zero as 1 ulp from each other and from the smallest denormal of the same sign, or as 0 ulp from each other and both 1 ulp from the smallest denormal of either sign. > Significant digits has more to do with error of measurements, (and estimates), while ULPs is about accuracy limits of the hardware/software ability to calculate. And, importantly, to represent your values in the first place. If you have a value that's, say, exactly 0.3 (or, for that matter, 0.3 to 28 significant digits), +/- 1 ulp is larger than your measurement error, but it's the minimum error range you can store.l >> Second, there's never any problem determining if two finite numbers are >> equal or which one is greater. The issue is determining whether two >> numbers ? their error bars are too close to call vs. unambiguously >> greater or lesser. For example, if I get 1.0e1 and 1.1e1 (assuming 1 bit >> mantissa for ease of discussion), the latter is clearly greater--but if >> I have 2 ulp of error, or an absolute error of 1e1, or a relative error >> of 50%, the fact that the latter is greater is irrelevant--each value is >> within the other value's error range. > > You would use the larger of the three. And possibly give a warning if the 2 ulp error is the largest. (if the application is set to do so.) You use whichever is/are relevant to your error analysis and ignore the others. (Technically I guess you could say you're just using 0 as the error for the two you don't care about, and then it's guaranteed that the one you do care about is largest, but I don't think that's the way you'd normally think about it.) Also, you have to be careful about how that extends to inequality. > I presuming the 2 ulp is twice the limit of the floating point precision here. Yes, 2 ulp means off by 2 units of least precision. Of course for binary, that actually means off by 1 in the unit of penultimate precision. > 50% accuracy of data > 1e1 limit of significant digits/measurement > 2 ulp twice floating point unit of least precision > >> But if I get 1.0e1 and 1.0e10 with >> the same error, then I can say that the latter is unambiguously >> greater. > > Yes, This was the point I was alluding to earlier. > > >> I think Chris is right that ulp comparisons usually only come up in >> testing an algorithm. You have to actually do an error analysis, and you >> have to have input data with a precision specified in ulp, or you're not >> going to get a tolerance in ulp. When you want to verify that you've >> correctly implemented an algorithm that guarantees to multiply input ulp by >> no more than 4, you can feed in numbers with ? 1 ulp error (just typing in >> decimal numbers does that) and verify that the results are within ? 4 ulp. >> But when you have real data, or inherent rounding issues, or an error >> analysis that's partly made up of rules of thumb and winging it, you're >> almost always going to end up with absolute or relative error instead. (Or, >> occasionally, something more complicated that you have to code up man >> ually, like logarithmic relative error.) > > If the algorithm doesn't track error accumulation, then yes. > > This is interesting but I'm going to search for some examples of how to use some of this. I'm not sure I can add to the conversation much, but thanks for taking the time to explain some of it. > > Cheers, > Ron > > > > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From chris.barker at noaa.gov Wed Jan 14 17:44:24 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 14 Jan 2015 08:44:24 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <m95blv$bu6$1@ger.gmane.org> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <6011660376447291684@unknownmsgid> <m94tsd$uav$1@ger.gmane.org> <09BB22C6-168A-43ED-9BF5-F731FDEF5240@yahoo.com> <m95blv$bu6$1@ger.gmane.org> Message-ID: <CALGmxEJ6Z6SJsxkV2d5c2Vxav_ROUA=CbT2Wu+MJY8cf0L5qAA@mail.gmail.com> On Wed, Jan 14, 2015 at 1:08 AM, Ron Adam <ron3200 at gmail.com> wrote: > Significant digits has more to do with error of measurements, (and > estimates), right -- and relative tolerance is kinda-sorta like significant digits. i.e a relative tolerance of 1e-4 is like saying the same to 4 (decimal) significant digits. which brings up the issue with 0.0 -- how many significant digits does 0.0 have? does 0.000001234 have the same significant digits as 0.0 ? -- its not really defined. whereas it's fairly straightforward to say that: 0.000001234 and 0.000001233 are the same to 3 significant digits, but differ in the forth. and note that: In [46]: u, v = 0.000001234, 0.000001233 In [47]: err = abs(u-v) In [48]: err Out[48]: 9.999999999998634e-10 so the absolute error is less than 1e-9 == pretty "small" -- but is that what we generally want? no. In [49]: err <= 1e-3*abs(u) Out[49]: True but the error is less than a relative tolerance of 1e-3 (three sig figs) In [50]: err <= 1e-4*abs(v) Out[50]: False and greater than a relative tolerance of 1e-4 (not four sig figs) Again, this all works great when you are away from zero (and maybe need to handle NaN and inf carefully too), but if we simply put this in the stdlib, then folks may use it with a zero value, and not get what they expect. My thinking now: set a "zero_tolerance",which will default to the relative tolerance, but be user-settable. If one of the input values is zero, then use the zero_tolerance as an absolute tolerance, if not, then use a relative tolerance. I think this would make for fewer surprises, and make it easier to use the same function call for a wide range of values, some of which may be zero. What I haven't figure out yet is how(or if) to make sure that the transition is continuous -- do we only use zero_tol if one of the values is exactly zero? of if one or both of the values is less than zero_tol? It seems that if you say, for instance that: 1e-12 is "close" to zero, then 1e-12 should also be "close" to any value less than 1e-12. But if 1e-12 is "close" to 1e-14 (for example), then 1e-12 should probably be "close" to 1.00000000001 also, but it wouldn't be, if we did an abrupt change to relative tolerance for any value >= the zero_tolerance. So more to think out here -- feel free to chime in. I've been playing with this gist, though it's only a few real lines of code anyway: https://gist.github.com/PythonCHB/6e9ef7732a9074d9337a > while ULPs is about accuracy limits of the hardware/software ability to > calculate. > > > Second, there's never any problem determining if two finite numbers are >> equal or which one is greater. The issue is determining whether two >> numbers ? their error bars are too close to call vs. unambiguously >> greater or lesser. For example, if I get 1.0e1 and 1.1e1 (assuming 1 bit >> mantissa for ease of discussion), the latter is clearly greater--but if >> I have 2 ulp of error, or an absolute error of 1e1, or a relative error >> of 50%, the fact that the latter is greater is irrelevant--each value is >> within the other value's error range. >> > > You would use the larger of the three. And possibly give a warning if the > 2 ulp error is the largest. (if the application is set to do so.) > > I presuming the 2 ulp is twice the limit of the floating point precision > here. > > 50% accuracy of data > 1e1 limit of significant digits/measurement > 2 ulp twice floating point unit of least precision > > But if I get 1.0e1 and 1.0e10 with >> the same error, then I can say that the latter is unambiguously >> greater. >> > > Yes, This was the point I was alluding to earlier. > > > I think Chris is right that ulp comparisons usually only come up in >> testing an algorithm. You have to actually do an error analysis, and you >> have to have input data with a precision specified in ulp, or you're not >> going to get a tolerance in ulp. When you want to verify that you've >> correctly implemented an algorithm that guarantees to multiply input ulp >> by >> no more than 4, you can feed in numbers with ? 1 ulp error (just typing in >> decimal numbers does that) and verify that the results are within ? 4 ulp. >> But when you have real data, or inherent rounding issues, or an error >> analysis that's partly made up of rules of thumb and winging it, you're >> almost always going to end up with absolute or relative error instead. >> (Or, >> occasionally, something more complicated that you have to code up man >> ually, like logarithmic relative error.) >> > > If the algorithm doesn't track error accumulation, then yes. > > This is interesting but I'm going to search for some examples of how to > use some of this. I'm not sure I can add to the conversation much, but > thanks for taking the time to explain some of it. > > > Cheers, > Ron > > > > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150114/46d0b59f/attachment.html> From ron3200 at gmail.com Wed Jan 14 21:04:43 2015 From: ron3200 at gmail.com (Ron Adam) Date: Wed, 14 Jan 2015 14:04:43 -0600 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CALGmxEJ6Z6SJsxkV2d5c2Vxav_ROUA=CbT2Wu+MJY8cf0L5qAA@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <6011660376447291684@unknownmsgid> <m94tsd$uav$1@ger.gmane.org> <09BB22C6-168A-43ED-9BF5-F731FDEF5240@yahoo.com> <m95blv$bu6$1@ger.gmane.org> <CALGmxEJ6Z6SJsxkV2d5c2Vxav_ROUA=CbT2Wu+MJY8cf0L5qAA@mail.gmail.com> Message-ID: <m96i4s$qf9$1@ger.gmane.org> On 01/14/2015 10:44 AM, Chris Barker wrote: > On Wed, Jan 14, 2015 at 1:08 AM, Ron Adam > <ron3200 at gmail.com > <mailto:ron3200 at gmail.com>> wrote: > > Significant digits has more to do with error of measurements, (and > estimates), > > > right -- and relative tolerance is kinda-sorta like significant digits. i.e > a relative tolerance of 1e-4 is like saying the same to 4 (decimal) > significant digits. > > which brings up the issue with 0.0 -- how many significant digits does 0.0 > have? It has as many significant places as your data is measured or estimated to. If your data is entered with four places, then 0.0 is actually 0.0000. And a value displayed as 1.1 that was entered as 1.1000 would still have 4 significant places. > does 0.000001234 have the same significant digits as 0.0 ? -- its not > really defined. Well, if the significant digits/places was 4 places, those would be considered equal. Where 0,000001234 is a value after a calculation is done. And the 0,0 is actually 0.0000. If the significant places used as actually 9, then the 0,0 would be 0,000000000. Python (and most languages) truncate the zero's to the right. So you can't go by what is displayed, it needs to be explicitly set. Then you would use that to determine the error range of two numbers, rounding to significant digits and calculating your +- error. Percentage error ranges are something completely different. You need to calculate that separately. And set it based on your data. And also take into account significant digits. And taking into account ULPs is another thing. If ULPs become larger than significant digits or a relative error, then maybe a warning (or error) needs to be issued. Then you can use an alternative algorithm, or modify the algorithm to avoid the warning or error, and get a valid result. And possibly python should do none of these by default so they are always explicitly ask for behaviours. > whereas it's fairly straightforward to say that: > > 0.000001234 and 0.000001233 > > are the same to 3 significant digits, but differ in the forth. > > and note that: > > In [46]: u, v = 0.000001234, 0.000001233 > > In [47]: err = abs(u-v) > > In [48]: err > Out[48]: 9.999999999998634e-10 > > so the absolute error is less than 1e-9 == pretty "small" -- but is that > what we generally want? no. I think it depends if you are trying to measure the error in your calculation, the error of your answer based on your data accuracy, or the error of the floating point implementation. Each is a different problem. A calculation may use constants that are less precise than what is possible by floating point, but more precise than the data. So in this case, it has a limit above what is introduced by ULPs. > In [49]: err <= 1e-3*abs(u) > Out[49]: True > > but the error is less than a relative tolerance of 1e-3 (three sig figs) > > In [50]: err <= 1e-4*abs(v) > Out[50]: False > > and greater than a relative tolerance of 1e-4 (not four sig figs) If python worked more like a calculator and you actually set the significant figures, this would be relevant. But since it doesn't, you need to track significant figures separately and use that value, not just use the represented value which may have truncated zero's. Then zero is not different than other numbers. > Again, this all works great when you are away from zero (and maybe need to > handle NaN and inf carefully too), but if we simply put this in the stdlib, > then folks may use it with a zero value, and not get what they expect. > > My thinking now: > > set a "zero_tolerance",which will default to the relative tolerance, but be > user-settable. What do you think of having a calculator mode or module, which would display specified significant digits in the output, but keep the entire number internally? import calc calc.fix(4) calc.print(epxr) And possibly have other general features that can be useful in other more specialised modules. > If one of the input values is zero, then use the zero_tolerance as an > absolute tolerance, if not, then use a relative tolerance. I think this > would make for fewer surprises, and make it easier to use the same function > call for a wide range of values, some of which may be zero. > > What I haven't figure out yet is how(or if) to make sure that the > transition is continuous -- do we only use zero_tol if one of the values is > exactly zero? of if one or both of the values is less than zero_tol? It > seems that if you say, for instance that: > > 1e-12 is "close" to zero, then 1e-12 should also be "close" to any value > less than 1e-12. But if 1e-12 is "close" to 1e-14 (for example), then 1e-12 > should probably be "close" to 1.00000000001 also, but it wouldn't be, if we > did an abrupt change to relative tolerance for any value >= the zero_tolerance. > > So more to think out here -- feel free to chime in. On the parts I think I know.. OK. ;-) > I've been playing with this gist, though it's only a few real lines of > code anyway: > > https://gist.github.com/PythonCHB/6e9ef7732a9074d9337a I'll take a look later today. Cheers, Ron From ethan at stoneleaf.us Wed Jan 14 21:38:29 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 14 Jan 2015 12:38:29 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <m96i4s$qf9$1@ger.gmane.org> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <6011660376447291684@unknownmsgid> <m94tsd$uav$1@ger.gmane.org> <09BB22C6-168A-43ED-9BF5-F731FDEF5240@yahoo.com> <m95blv$bu6$1@ger.gmane.org> <CALGmxEJ6Z6SJsxkV2d5c2Vxav_ROUA=CbT2Wu+MJY8cf0L5qAA@mail.gmail.com> <m96i4s$qf9$1@ger.gmane.org> Message-ID: <54B6D3C5.3020507@stoneleaf.us> On 01/14/2015 12:04 PM, Ron Adam wrote: > On 01/14/2015 10:44 AM, Chris Barker wrote: >> On Wed, Jan 14, 2015 at 1:08 AM, Ron Adam wrote: >> >>> Significant digits has more to do with error of measurements, (and >>> estimates), >> >> right -- and relative tolerance is kinda-sorta like significant digits. i.e >> a relative tolerance of 1e-4 is like saying the same to 4 (decimal) >> significant digits. >> >> which brings up the issue with 0.0 -- how many significant digits does 0.0 >> have? > > It has as many significant places as your data is measured or estimated to. If your data is entered with four places, > then 0.0 is actually 0.0000. And a value displayed as 1.1 that was entered as 1.1000 would still have 4 significant places. > >> does 0.000001234 have the same significant digits as 0.0 ? -- its not >> really defined. > > Well, if the significant digits/places was 4 places, those would be considered equal. Where 0,000001234 is a value > after a calculation is done. And the 0,0 is actually 0.0000. If the significant places used as actually 9, then the > 0,0 would be 0,000000000. If 0.000001234 has four significant digits, those digits are the ending 1234, not the first four zeros. The leading zeros would only be significant if there was a leadinger non-zero ;) -- and then the zeros would no longer be leading. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150114/fa5f889f/attachment.sig> From alan.cristh at gmail.com Wed Jan 14 21:40:40 2015 From: alan.cristh at gmail.com (Alan Cristhian Ruiz) Date: Wed, 14 Jan 2015 17:40:40 -0300 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <m96i4s$qf9$1@ger.gmane.org> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <6011660376447291684@unknownmsgid> <m94tsd$uav$1@ger.gmane.org> <09BB22C6-168A-43ED-9BF5-F731FDEF5240@yahoo.com> <m95blv$bu6$1@ger.gmane.org> <CALGmxEJ6Z6SJsxkV2d5c2Vxav_ROUA=CbT2Wu+MJY8cf0L5qAA@mail.gmail.com> <m96i4s$qf9$1@ger.gmane.org> Message-ID: <54B6D448.20106@gmail.com> El 14/01/15 a las 17:04, Ron Adam escribi?: > which brings up the issue with 0.0 -- how many significant digits does > 0.0 > have? When a number is not expressed in scientific notation<http://mathworld.wolfram.com/ScientificNotation.html>, the amount of significant digits is infinite. You can explicitly define the amount of significant figures in scientific notation. For example, 0.0000e0 contain four significant digits, 2.00 contain infinite, 0.00300e0 contain three. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150114/91fdac13/attachment.html> From ron3200 at gmail.com Wed Jan 14 22:18:17 2015 From: ron3200 at gmail.com (Ron Adam) Date: Wed, 14 Jan 2015 15:18:17 -0600 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <20150114103919.GO10173@ando.pearwood.info> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <20150114103919.GO10173@ando.pearwood.info> Message-ID: <m96meq$7j0$1@ger.gmane.org> On 01/14/2015 04:39 AM, Steven D'Aprano wrote: > On Tue, Jan 13, 2015 at 08:57:38PM -0600, Ron Adam wrote: >> > >> >On 01/13/2015 09:53 AM, Chris Barker - NOAA Federal wrote: > [...] >>> > >I haven't thought it out yet, but maybe we could specify an absolute >>> > >tolerance near zero, and a relative tolerance elsewhere, both at once. >>> > >Tricky to document, even if possible. >> > >> >Doesn't this problem come up at any boundary comparison, and not just zero? > No. > > A quick refresher on error tolerances... > > Suppose you have a value which should be exactly 0.5, but you calculate > it as 0.51. Then the absolute error is 0.51-0.5 = 0.01, and the relative > error is 0.01/0.5 = 0.02 (or 2%). > > But consider two values, 0.0 and 0.1. Then the absolute error is 0.1-0.0 > = 0.1, and the relative error is 0.1/0.0 which is infinite. > > So 0.0 and -0.0 are problematic when dealing with relative errors. I'm not sure why, but it seems like something here is out of place to me. Either it's not something that would come up, or it's something you would expect and handle differently. Or it's would be an error, signalling you need to handle it differently. >> >So isn't the issue about any n distance from any floating point number that >> >is less than 1 ulp? And in that regard, comparison to zero is no different >> >than any comparison to any other floating point value? > No. 1 ULP (Unit In Last Place) is the smallest possible difference > between two floats. A difference of 0 ULP means the two floats are > exactly equal. A difference of 0 ULP means they *may* be exactly equal. The calculation/representation just can't resolve to a finer amount, so two figures less than 1 ULP apart can get the same floating point representation. The two numbers still have a +- error. Possibly +-.5 ULP. ie... potentially 1 ULP apart. > How it works: in mathematics, real numbers are continuous, but floats > are not. There are only 2**64 floats in Python (a C double), less if you > ignore the NANs and INFs, which means we can conveniently enumerate them > from -(2**64) to (2**64-1), based on the internal structure of a float. > So if you convert two floats into this enumerated integer value (which > is equivalent to doing a type-cast from a C double to a C long) and > subtract the two ints, this gives you a measure of how far apart they > are. (As Mark mentioned earlier, you have to make allowance for negative > floats, also INF and NANs are problematic too.) > > If two values are exactly equal, their "distance apart" in ULP will be > zero. A distance of 1 ULP means they are consecutive floats, they cannot > possibly be any closer without being equal. A distance of 2 ULP means > there is only a single float separating them, and so on. > > Note that ULP do not directly correspond to a numeric tolerance. For > example, these pairs of values are each 1 ULP apart: > > 0.0 and 5e-324 > > 1.0 and 1.0000000000000002 > > 1e300 and 1.0000000000000002e+300 > > > So in these three cases, 1 ULP represents numeric differences of: > > 0.00000000000000000000...00005 > 0.0000000000000002 > 2000000000000000000000...000.0 > > respectively. > > >> >Just trying to follow along, > A good resource is Bruce Dawson's blog RandomASCII, if you don't mind > the focus on C++. Start here: > > https://randomascii.wordpress.com/2012/02/25/comparing-floating-point-numbers-2012-edition/ Thanks, :) Ron From ron3200 at gmail.com Wed Jan 14 22:26:07 2015 From: ron3200 at gmail.com (Ron Adam) Date: Wed, 14 Jan 2015 15:26:07 -0600 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <54B6D448.20106@gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <6011660376447291684@unknownmsgid> <m94tsd$uav$1@ger.gmane.org> <09BB22C6-168A-43ED-9BF5-F731FDEF5240@yahoo.com> <m95blv$bu6$1@ger.gmane.org> <CALGmxEJ6Z6SJsxkV2d5c2Vxav_ROUA=CbT2Wu+MJY8cf0L5qAA@mail.gmail.com> <m96i4s$qf9$1@ger.gmane.org> <54B6D448.20106@gmail.com> Message-ID: <m96mtg$fed$1@ger.gmane.org> On 01/14/2015 02:40 PM, Alan Cristhian Ruiz wrote: > El 14/01/15 a las 17:04, Ron Adam escribi?: >> which brings up the issue with 0.0 -- how many significant digits does 0.0 >> have? (as displayed by python) My point is it needs to track it separately, and not use pythons displayed value, or the internal representation. > When a number is not expressed in scientific > notation<http://mathworld.wolfram.com/ScientificNotation.html>, the amount > of significant digits is infinite. > > You can explicitly define the amount of significant figures in scientific > notation. For example, > 0.0000e0 contain four significant digits, 2.00 contain infinite, 0.00300e0 > contain three. >>> 0.0000e0 0.0 See what I mean? Cheers, Ron From abarnert at yahoo.com Wed Jan 14 22:33:18 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 14 Jan 2015 13:33:18 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <m96i4s$qf9$1@ger.gmane.org> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <6011660376447291684@unknownmsgid> <m94tsd$uav$1@ger.gmane.org> <09BB22C6-168A-43ED-9BF5-F731FDEF5240@yahoo.com> <m95blv$bu6$1@ger.gmane.org> <CALGmxEJ6Z6SJsxkV2d5c2Vxav_ROUA=CbT2Wu+MJY8cf0L5qAA@mail.gmail.com> <m96i4s$qf9$1@ger.gmane.org> Message-ID: <A9244D96-CFAF-4151-9D9F-49F85DC40B47@yahoo.com> On Jan 14, 2015, at 12:04, Ron Adam <ron3200 at gmail.com> wrote: > What do you think of having a calculator mode or module, which would display specified significant digits in the output, but keep the entire number internally? > > import calc > calc.fix(4) > calc.print(epxr) > > And possibly have other general features that can be useful in other more specialised modules. First, why use a "calc" mode instead of just format('.4')? Also, this only works when the number of significant digits is small enough to fit in 52 bits; beyond that, it's misleading. Most calculators do something like show 8 decimal digits and store 10 decimal digits, which doesn't have this problem. And in fact, I think in any case where format isn't sufficient, the decimal module is what you need. (In fact, even with 4 digits it's misleading; two different values can both print identically but there's no function to check that except to stringify them and compare the strings. Calculators that display 8 digits but use 10 for intermediate results usually do comparisons on the 8 digits.) At any rate, switching to decimal makes it easier to think about and see the rounding errors, and eliminates any additional errors caused by going back and forth to decimal strings (and notice inputs are often decimal strings, with precision represented in decimal significant figures). But ultimately you still have the same problems of tracking the accumulation of error and determining the appropriate relative or absolute (or ulp or log2-relative or whatever) tolerance for equality and inequality comparisons. From steve at pearwood.info Thu Jan 15 01:04:17 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 15 Jan 2015 11:04:17 +1100 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <m96meq$7j0$1@ger.gmane.org> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <20150114103919.GO10173@ando.pearwood.info> <m96meq$7j0$1@ger.gmane.org> Message-ID: <20150115000416.GP10173@ando.pearwood.info> On Wed, Jan 14, 2015 at 03:18:17PM -0600, Ron Adam wrote: > > > On 01/14/2015 04:39 AM, Steven D'Aprano wrote: [...] > >Suppose you have a value which should be exactly 0.5, but you calculate > >it as 0.51. Then the absolute error is 0.51-0.5 = 0.01, and the relative > >error is 0.01/0.5 = 0.02 (or 2%). > > > >But consider two values, 0.0 and 0.1. Then the absolute error is 0.1-0.0 > >= 0.1, and the relative error is 0.1/0.0 which is infinite. > > > >So 0.0 and -0.0 are problematic when dealing with relative errors. > > I'm not sure why, but it seems like something here is out of place to me. > Either it's not something that would come up, or it's something you would > expect and handle differently. Or it's would be an error, signalling you > need to handle it differently. You're going to have to explain what you think is "out of place". Of course doing division by zero is an error. The whole point of this discussion is that you cannot talk about errors relative to zero. At zero, you can still have an absolute error, but you cannot have a relative error. The definition of the error between two numeric values: def abserr(a, b): return abs(a - b) def relerr(a, b): return abs(a - b)/min(abs(a), abs(b)) If either a or b is zero, relarr() fails. This is a mathematical limitation, not a bug to be fixed. > >>>So isn't the issue about any n distance from any floating point number > >>that > >>>is less than 1 ulp? And in that regard, comparison to zero is no > >>different > >>>than any comparison to any other floating point value? > >No. 1 ULP (Unit In Last Place) is the smallest possible difference > >between two floats. A difference of 0 ULP means the two floats are > >exactly equal. > > A difference of 0 ULP means they *may* be exactly equal. The > calculation/representation just can't resolve to a finer amount, so two > figures less than 1 ULP apart can get the same floating point > representation. The two numbers still have a +- error. Possibly +-.5 ULP. > ie... potentially 1 ULP apart. I'm referring specifically to errors between two floats, not between a float and a theoretical exact real number. Sorry if that was not clear. E.g. there are two consecutive floats, 5e-324 and 1e-323, which are 1 ULP apart. But there is no such thing as a float for 8e-324. If you have some pencil and paper calculation which comes to 8e-324, then the error between that real value and either of those two floats is about 0.5 ULP. -- Steve From steve at pearwood.info Thu Jan 15 01:18:18 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 15 Jan 2015 11:18:18 +1100 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <8257D0DC-151D-44B5-897F-A9E940DD099D@yahoo.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <6011660376447291684@unknownmsgid> <m94tsd$uav$1@ger.gmane.org> <09BB22C6-168A-43ED-9BF5-F731FDEF5240@yahoo.com> <m95blv$bu6$1@ger.gmane.org> <8257D0DC-151D-44B5-897F-A9E940DD099D@yahoo.com> Message-ID: <20150115001818.GQ10173@ando.pearwood.info> On Wed, Jan 14, 2015 at 02:44:27AM -0800, Andrew Barnert wrote: > I think your confusion here is entirely my fault. For simplicity, it's > often helpful to look at tiny float representations--e.g., a 4-bit > float with 1 sign bit, 1 mantissa bit, and 2 exponent bits (that, if > both 1, mean inf/nan), Since this is python-ideas, how about a minifloat type? I think a 4-bit float is a bit *too* mini, 8-bit sounds about right. (I know, I know, it should be a third-party library, not a built-in :-) -- Steven From ethan at stoneleaf.us Thu Jan 15 01:38:04 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 14 Jan 2015 16:38:04 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <20150115001818.GQ10173@ando.pearwood.info> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <6011660376447291684@unknownmsgid> <m94tsd$uav$1@ger.gmane.org> <09BB22C6-168A-43ED-9BF5-F731FDEF5240@yahoo.com> <m95blv$bu6$1@ger.gmane.org> <8257D0DC-151D-44B5-897F-A9E940DD099D@yahoo.com> <20150115001818.GQ10173@ando.pearwood.info> Message-ID: <54B70BEC.5020605@stoneleaf.us> On 01/14/2015 04:18 PM, Steven D'Aprano wrote: > On Wed, Jan 14, 2015 at 02:44:27AM -0800, Andrew Barnert wrote: > >> I think your confusion here is entirely my fault. For simplicity, it's >> often helpful to look at tiny float representations--e.g., a 4-bit >> float with 1 sign bit, 1 mantissa bit, and 2 exponent bits (that, if >> both 1, mean inf/nan), > > Since this is python-ideas, how about a minifloat type? > > I think a 4-bit float is a bit *too* mini, 8-bit sounds about right. > > > (I know, I know, it should be a third-party library, not a built-in :-) Speaking of which: https://pypi.python.org/pypi/bigfloat/0.3.0 I know it's /called/ bigfloat, but it /looks/ like you can specify however many/few bits you want. ;) -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150114/248b28cb/attachment.sig> From chris.barker at noaa.gov Thu Jan 15 01:48:35 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 14 Jan 2015 16:48:35 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <3b503462-8868-4c02-9988-bd53c66fe271@googlegroups.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <3b503462-8868-4c02-9988-bd53c66fe271@googlegroups.com> Message-ID: <CALGmxEJet44RNdD6xD_bYPcUniwOUTX6j7pL79SjSSHEhhRZ5Q@mail.gmail.com> On Wed, Jan 14, 2015 at 4:23 PM, Neil Girdhar <mistersheik at gmail.com> wrote: > You might be interested in my question: > > > http://stackoverflow.com/questions/4028889/floating-point-equality-in-python > nothing new there, I'm afaid -- and no one seemed to have brought up the issue with zero. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150114/bb8fd890/attachment.html> From chris.barker at noaa.gov Thu Jan 15 01:43:44 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 14 Jan 2015 16:43:44 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <54B7043A.1050303@gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <6011660376447291684@unknownmsgid> <m94tsd$uav$1@ger.gmane.org> <94278295727731205@unknownmsgid> <54B7043A.1050303@gmail.com> Message-ID: <CALGmxELT2n9R2D+PjmUmW9sbivwYD9BbPYS1ZhJVJojC58YZvg@mail.gmail.com> On Wed, Jan 14, 2015 at 4:05 PM, Ron Adam <ron3200 at gmail.com> wrote: > > Did you mean to take this off line? > no I really hate mailing lists that don't default to reply to the list -- drives me crazy... Bringing it back on list. If I'm checking the tolerance of a resistor, I would measure it's > resistance and calculate that against it's rated value. > > abs(rated-resistance - actual_resistance) / rated_resistance < .05 > > If they are equal, then I get 0.0. If it's 0, I get 1.0, or 100%. > well, I suspect that you never need to check a resistor that is rated for zero ohms. and if you get one that measures out to zero ohms, it probably is 100% error, or, in any case not "close" enough. so that's a particular use case, and I think my idea for a zero_tolerance, which is this case you'd set to zero, would support that. But the goal here is a method that will work most of the time, without a priory knowledge of how large the inputs are likely to be. If you know that, then you can use an absolute tolerance. You might get that by your computation above, but you could pass that in to a "close" function -- or not bother to write a close function, 'cause that's easy anyway. I would never be comparing a resistor to a rating of 0, and expect any > reasonable answer. Can you think of a real case which this come up? I > think it's in the realm of just don't do that. > Not in that case, of course, no. But in the general case of checking whether a computation produces about the same answer as expected (or as it did before you refactored the code), then zero mught cro up. But you have a point - maybe we could have a is_close function with a nice warning in the docs "don't expect this to work with zeros", and still have somethign useful. Would raising a ValueError on zero input be too pedantic? > It might be a case where you need more than two values. You just can't > assume a relative size with only two numbers. > the assumption is relative to the size of the input numbers -- which is mostly fine, but another way to deal with zero is to (optional) pass in a "relative to this" parameter -- but that essentially reverts to absolute tolerance anyway. Hmm -- new thought -- zero_tolerance can be the Minimum value that you use for the relative tolerance comparison -- that may be easy and at least give a continuous result... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150114/88ea2af4/attachment-0001.html> From abarnert at yahoo.com Thu Jan 15 03:00:53 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 15 Jan 2015 02:00:53 +0000 (UTC) Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <20150115001818.GQ10173@ando.pearwood.info> References: <20150115001818.GQ10173@ando.pearwood.info> Message-ID: <534505580.812224.1421287253115.JavaMail.yahoo@jws100211.mail.ne1.yahoo.com> On Wednesday, January 14, 2015 4:18 PM, Steven D'Aprano <steve at pearwood.info> wrote: >On Wed, Jan 14, 2015 at 02:44:27AM -0800, Andrew Barnert wrote: > >> I think your confusion here is entirely my fault. For simplicity, it's >> often helpful to look at tiny float representations--e.g., a 4-bit >> float with 1 sign bit, 1 mantissa bit, and 2 exponent bits (that, if >> both 1, mean inf/nan), > >Since this is python-ideas, how about a minifloat type? > >I think a 4-bit float is a bit *too* mini, 8-bit sounds about right. Anything up to 5 or 6 bits, you can fit a table of all the values on your monitor at decent font size. 8 bits is a bit too much for that. Which reminds me, I found an old blog post where I used such a 6-bit (1+3+2) format to demonstrate all the details of IEEE binary floats, including exactly such a table. It's now back online (http://stupidpythonideas.blogspot.com/2015/01/ieee-floats-and-python.html?view=sidebar) if anyone's interested. I had a Binary6 class to go with it?which didn't have the complete float interface, but did have a lot of handy optionals like as_tuple and next_plus, as Decimal does?but I can no longer find that, so I stripped out that section of the blog post. But it still shows how to fiddle with the bits manually using struct plus integer operations or a library like bitstring. >(I know, I know, it should be a third-party library, not a built-in :-) I don't think it would be very useful as a third-party lib either. But an arbitrary-precision IEEE 754-2008 Binary type, with all the optional features that aren't part of float, akin to decimal.Decimal?that might be useful. And while you're at it (you were volunteering, right?), want to go over 754-2008 vs. 854-1987 and see if there's anything worth adding to Decimal? :) From ron3200 at gmail.com Thu Jan 15 03:13:42 2015 From: ron3200 at gmail.com (Ron Adam) Date: Wed, 14 Jan 2015 20:13:42 -0600 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <20150115000416.GP10173@ando.pearwood.info> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <20150114103919.GO10173@ando.pearwood.info> <m96meq$7j0$1@ger.gmane.org> <20150115000416.GP10173@ando.pearwood.info> Message-ID: <m977on$can$1@ger.gmane.org> On 01/14/2015 06:04 PM, Steven D'Aprano wrote: > On Wed, Jan 14, 2015 at 03:18:17PM -0600, Ron Adam wrote: >> > >> > >> >On 01/14/2015 04:39 AM, Steven D'Aprano wrote: > [...] >>> > >Suppose you have a value which should be exactly 0.5, but you calculate >>> > >it as 0.51. Then the absolute error is 0.51-0.5 = 0.01, and the relative >>> > >error is 0.01/0.5 = 0.02 (or 2%). >>> > > >>> > >But consider two values, 0.0 and 0.1. Then the absolute error is 0.1-0.0 >>> > >= 0.1, and the relative error is 0.1/0.0 which is infinite. >>> > > >>> > >So 0.0 and -0.0 are problematic when dealing with relative errors. >> > >> >I'm not sure why, but it seems like something here is out of place to me. >> >Either it's not something that would come up, or it's something you would >> >expect and handle differently. Or it's would be an error, signalling you >> >need to handle it differently. > You're going to have to explain what you think is "out of place". > > Of course doing division by zero is an error. The whole point of this > discussion is that you cannot talk about errors relative to zero. At > zero, you can still have an absolute error, but you cannot have a > relative error. The definition of the error between two numeric values: > > def abserr(a, b): > return abs(a - b) > > def relerr(a, b): > return abs(a - b)/min(abs(a), abs(b)) > > > If either a or b is zero, relarr() fails. This is a mathematical > limitation, not a bug to be fixed. I would probably write like this. def relerr(actual, expected): return abs(actual - expected)/expected Where expected is a non-zero reference value. it needs a third value. In the above the expected value is the scaler. And then use it this way to measure a resistors tolerance. if relerr(218.345, 220) <= 0.05: print("Good") else: print("Bad") Note that you would never compare to an expected value of zero. relerr(a - b, expected_feet) < tolerance # relative feet from b relerr(a - 0, expected_feet) < tolerance # relative feet from zero relerr(a - b, ulp) # percentage of ulp's What Chris is looking for is a way to get a closeness function that works most of the time. (He posted while I'm writing this.) But I don't see how you can do that without specifying some scaler to give it context, and a tolerance. def is_close(a, b, unit, good): return (abs(a - b) / unit) <= good is_close(218.345, 220, 1, .05) # OHMs is_close(a, b, ULP, 2) # ULPs is_close(a, b, AU, .001) # astronomical units I don't see anyway to generalise those with just a function. By using objects we can do a bit more. I seem to recall coming across measurement objects some place. They keep a bit more context with them. Cheers, Ron From abarnert at yahoo.com Thu Jan 15 03:27:43 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 14 Jan 2015 18:27:43 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <m977on$can$1@ger.gmane.org> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <20150114103919.GO10173@ando.pearwood.info> <m96meq$7j0$1@ger.gmane.org> <20150115000416.GP10173@ando.pearwood.info> <m977on$can$1@ger.gmane.org> Message-ID: <64FE680E-03D7-4265-A222-80945AADD761@yahoo.com> On Jan 14, 2015, at 18:13, Ron Adam <ron3200 at gmail.com> wrote: > By using objects we can do a bit more. I seem to recall coming across measurement objects some place. They keep a bit more context with them. You can use an interval class to represent your measurements with error bars, in which case many problems either go away, or become too obvious to ignore. In particular, for simple cases at least, you get error analysis for free, and you can just check whether the final interval includes the expected value (or overlaps the expected interval, if there are error bars on the expected result too). However, finding an interval math library that handles all of the mathematical operations you want for a particular application may not be easy. You might have to ctypes or wrap up a C or C++ library instead. (I'm just guessing, but I suspect more people have written wrappers for such libs to work with numpy/scipy than with native Python scalars.) From mistersheik at gmail.com Thu Jan 15 01:54:15 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Wed, 14 Jan 2015 16:54:15 -0800 (PST) Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <20150115001818.GQ10173@ando.pearwood.info> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <6011660376447291684@unknownmsgid> <m94tsd$uav$1@ger.gmane.org> <09BB22C6-168A-43ED-9BF5-F731FDEF5240@yahoo.com> <m95blv$bu6$1@ger.gmane.org> <8257D0DC-151D-44B5-897F-A9E940DD099D@yahoo.com> <20150115001818.GQ10173@ando.pearwood.info> Message-ID: <557bd10a-8704-473f-b66b-bb06080e099e@googlegroups.com> There's already float16, float32, float64, and float128 in numpy. That would be the best place to add float8 if you want it. Best, Neil On Wednesday, January 14, 2015 at 7:19:01 PM UTC-5, Steven D'Aprano wrote: > > On Wed, Jan 14, 2015 at 02:44:27AM -0800, Andrew Barnert wrote: > > > I think your confusion here is entirely my fault. For simplicity, it's > > often helpful to look at tiny float representations--e.g., a 4-bit > > float with 1 sign bit, 1 mantissa bit, and 2 exponent bits (that, if > > both 1, mean inf/nan), > > Since this is python-ideas, how about a minifloat type? > > I think a 4-bit float is a bit *too* mini, 8-bit sounds about right. > > > (I know, I know, it should be a third-party library, not a built-in :-) > > > -- > Steven > _______________________________________________ > Python-ideas mailing list > Python... at python.org <javascript:> > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150114/6235e90c/attachment.html> From abarnert at yahoo.com Thu Jan 15 04:38:36 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 14 Jan 2015 19:38:36 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <64FE680E-03D7-4265-A222-80945AADD761@yahoo.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <20150114103919.GO10173@ando.pearwood.info> <m96meq$7j0$1@ger.gmane.org> <20150115000416.GP10173@ando.pearwood.info> <m977on$can$1@ger.gmane.org> <64FE680E-03D7-4265-A222-80945AADD761@yahoo.com> Message-ID: <7BF7BF34-D2EA-41B0-8D5E-CA8F9E920A60@yahoo.com> On Jan 14, 2015, at 18:27, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: > However, finding an interval math library that handles all of the mathematical operations you want for a particular application may not be easy. You might have to ctypes or wrap up a C or C++ library instead. (I'm just guessing, but I suspect more people have written wrappers for such libs to work with numpy/scipy than with native Python scalars.) I found a few things on PyPI that look promising, but not a single one of them installs and runs for me on either 2.7 or 3.4. But I was able to fix up the first one pretty quickly; see https://github.com/abarnert/pyinterval for a fork. You'll need to install crlibm first, but that's trivial on at least Mac/Homebrew and Fedora, so I'm guessing most other *nix systems. One big thing it's missing is pow (and, in fact, all bivariate functions). Of course you can always define it as exp(log(base) * exponent), but IIRC that gives much wider error bars than a proper interval power function. From chris.barker at noaa.gov Thu Jan 15 06:19:47 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Wed, 14 Jan 2015 21:19:47 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <m977on$can$1@ger.gmane.org> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <20150114103919.GO10173@ando.pearwood.info> <m96meq$7j0$1@ger.gmane.org> <20150115000416.GP10173@ando.pearwood.info> <m977on$can$1@ger.gmane.org> Message-ID: <5021473497021594897@unknownmsgid> On Jan 14, 2015, at 6:14 PM, Ron Adam <ron3200 at gmail.com> wrote: > > I would probably write like this. > > def relerr(actual, expected): > return abs(actual - expected)/expected > > Where expected is a non-zero reference value. Hmm, there is something to be said about making it clear which is "expected" -- then that value clearly provides the scale. And you can document that it can't be zero. And raise an Exception if it does. > it needs a third value. Well, a tolerance, yes. > And then use it this way to measure a resistors tolerance. > > if relerr(218.345, 220) <= 0.05: > print("Good") > else: > print("Bad") I don't see the advantage at all of this over a simple function API: Is_close(actual, expected, tolerance) (Probably with a default tolerance) > Note that you would never compare to an expected value of zero. Yup --I like how you can make that clear. Though sometimes you really do expect zero. Maybe no need for the same API for that, or an API at all. > relerr(a - b, expected_feet) < tolerance # relative feet from b I'm still confused why bother with the relerr function -- why bother? > relerr(a - b, ulp) # percentage of ulp's I don't think you can define an ulp like that. > What Chris is looking for is a way to get a closeness function that works most of the time. Exactly. > But I don't see how you can do that without specifying some scaler to give it context, and a tolerance. Yes, you need a tolerance, sure. And the context comes from the values you are comparing. The only problem is what to do with zero. (And maybe really near zero). > def is_close(a, b, unit, good): > return (abs(a - b) / unit) <= good > > is_close(218.345, 220, 1, .05) # OHMs > > is_close(a, b, ULP, 2) # ULPs > > is_close(a, b, AU, .001) # astronomical units I don't get why it helps to introduce units here. I don't think that's point (this would all be equally valid with unit less quantities). And if you want a scale for relative tolerance that is independent of the values you are checking, that's essentially an absolute tolerance. I'm not sure it's worth burying that simple calculation in a function. > I don't see anyway to generalise those with just a function. Didn't just do that? Except for the Ulps, not sure what to do there, an ulp is a different way to define a delta. Unless you mean units aware, but that's really not what this all is about. > By using objects we can do a bit more. I seem to recall coming across measurement objects some place. They keep a bit more context with them. Sure, there are various unit/physical quantities objects out there, many numpy-based. But I think is orthogonal to comparing floating point numbers. And a sime floating point comparison function _may_ belong the standard library, but not a more complex physical quantity system -Chris From ron3200 at gmail.com Thu Jan 15 08:24:17 2015 From: ron3200 at gmail.com (Ron Adam) Date: Thu, 15 Jan 2015 01:24:17 -0600 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <5021473497021594897@unknownmsgid> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <20150114103919.GO10173@ando.pearwood.info> <m96meq$7j0$1@ger.gmane.org> <20150115000416.GP10173@ando.pearwood.info> <m977on$can$1@ger.gmane.org> <5021473497021594897@unknownmsgid> Message-ID: <m97pv1$8fa$1@ger.gmane.org> On 01/14/2015 11:19 PM, Chris Barker - NOAA Federal wrote: >> >What Chris is looking for is a way to get a closeness function that works most of the time. > Exactly. > >> >But I don't see how you can do that without specifying some scaler to give it context, and a tolerance. > Yes, you need a tolerance, sure. And the context comes from the values > you are comparing. The only problem is what to do with zero. (And > maybe really near zero). This is where a good but realistic example of the issue would help. Something you may actually see in a specific use case. >> > def is_close(a, b, unit, good): >> > return (abs(a - b) / unit) <= good But I also think of these sort of things building as building blocks for those more complex systems. So it may help to consider how they may use these blocks. >> > >> > is_close(218.345, 220, 1, .05) # OHMs >> > >> > is_close(a, b, ULP, 2) # ULPs >> > >> > is_close(a, b, AU, .001) # astronomical units > I don't get why it helps to introduce units here. I don't think that's > point (this would all be equally valid with unit less quantities). > And if you want a scale for relative tolerance that is independent of > the values you are checking, that's essentially an absolute tolerance. > I'm not sure it's worth burying that simple calculation in a function. I agree. The returned value I was thinking of should have been (abs(a - b) / b) / units <= good Or something like that.. The units here just scales the result. Without that you need to either scale the inputs, or scale the tolerance. In most cases when you are entering a tolerance, you will also be working with some multiple of units. But it seems that can confuse as much as it helps. In any case, I was just trying to come up with some practical example, and at the same time was thinking about how different units of measure can effect it. >>> By using objects we can do a bit more. I seem to recall coming >>> across measurement objects some place. They keep a bit more context with them. > Sure, there are various unit/physical quantities objects out there, many > numpy-based. But I think is orthogonal to comparing floating point > numbers. > > > And a sime floating point comparison function_may_ belong the > standard library, but not a more complex physical quantity system Yes, for the most part I agree, but think such functions would be useful in those more complex systems as well. Ron From steve at pearwood.info Thu Jan 15 08:29:28 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 15 Jan 2015 18:29:28 +1100 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <m977on$can$1@ger.gmane.org> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <20150114103919.GO10173@ando.pearwood.info> <m96meq$7j0$1@ger.gmane.org> <20150115000416.GP10173@ando.pearwood.info> <m977on$can$1@ger.gmane.org> Message-ID: <20150115072927.GA16607@ando.pearwood.info> On Wed, Jan 14, 2015 at 08:13:42PM -0600, Ron Adam wrote: > > > On 01/14/2015 06:04 PM, Steven D'Aprano wrote: [...] > >Of course doing division by zero is an error. The whole point of this > >discussion is that you cannot talk about errors relative to zero. At > >zero, you can still have an absolute error, but you cannot have a > >relative error. The definition of the error between two numeric values: > > > >def abserr(a, b): > > return abs(a - b) > > > >def relerr(a, b): > > return abs(a - b)/min(abs(a), abs(b)) > > > > > >If either a or b is zero, relarr() fails. This is a mathematical > >limitation, not a bug to be fixed. > > > I would probably write like this. > > def relerr(actual, expected): > return abs(actual - expected)/expected > > Where expected is a non-zero reference value. > > it needs a third value. No the function doesn't require a third value. It just returns the relative error between two values. What you do with that error is up to you: - compare it to an acceptable amount of error, as you do below - print it - add it to a list of error amounts etc. Of course, you could have other functions which do more, like decide whether an error is "small enough", but that is out of scope for a function which only calculates the error. > In the above the expected value is the scaler. > And then use it this way to measure a resistors tolerance. > > if relerr(218.345, 220) <= 0.05: > print("Good") > else: > print("Bad") # using your implementation if relerr(actual=200.0, expected=-5.0) < 0.05: print("oops") Errors, by definition, are non-negative. You have to take care to ensure that the error value returned is non-negative. The question of which to use as the denominator is more subtle. Like you, I used to think that you should choose ahead of time which value was expected and which was actual, and divide by the actual. Or should that be the expected? I could never decide which I wanted: error relative to the expected, or error relative to the actual? And then I could never remember which order the two arguments went. Finally I read Bruce Dawson (I've already linked to his blog three or four times) and realised that he is correct and I was wrong. Error calculations should be symmetrical, so that error(a, b) == error(b, a) regardless of whether you have absolute or relative error. Furthermore, for safety you normally want the larger estimate of error, not the smaller: given the choice between (abs(a - b))/abs(a) versus (abs(a - b))/abs(b) you want the *larger* error estimate, which means the *smaller* denominator. That's the conservative way of doing it. A concrete example: given a=5 and b=7, we have: absolute error = 2 relative error (calculated relative to a) = 0.4 relative error (calculated relative to b) = 0.286 That is, b is off by 40% relative to a; or a is off by 28.6% relative to b. Or another way to put it, given that a is the "true" value, b is 40% too big; or if you prefer, 28.6% of b is in error. Whew! Percentages are hard! *wink* The conservative, "safe" way to handle this is to just treat the error function as symmetrical and always report the larger of the two relative errors (excluding the case where the denominator is 0, in which case the relative error is either 100% or it doesn't exist). Worst case, you may reject some values which you should accept, but you will never accept any values that you should reject. > Note that you would never compare to an expected value of zero. You *cannot* compare to an expected value of zero, but you certainly can be in a situation where you would like to: math.sin(math.pi) should return 0.0, but doesn't, it returns 1.2246063538223773e-16 instead. What is the relative error of the sin function at x = math.pi? > relerr(a - b, expected_feet) < tolerance # relative feet from b > relerr(a - 0, expected_feet) < tolerance # relative feet from zero > relerr(a - b, ulp) # percentage of ulp's I don't understand what you think these three examples are showing. > What Chris is looking for is a way to get a closeness function that works > most of the time. (He posted while I'm writing this.) I think the function I have in the statistics test suite is that function. I would like to see ULP calculations offered as well, but Mark thinks that's unnecessary and I'm not going to go to the battlements to fight for ULPs. > But I don't see how you can do that without specifying some scaler to give > it context, and a tolerance. By the way, it is spelled "scalar", and all that means is that it is a number, like 23, not a vector or array or matrix. > def is_close(a, b, unit, good): > return (abs(a - b) / unit) <= good Take a look at the statistics test suite. I'll be the first to admit that the error tolerances are plucked from thin air, based on what I think are "close enough", but they show how such a function might work: * you provide two values, and at least one of an absolute error tolerance and a relative error; * if the error is less than the error(s) you provided, the test passes, otherwise it fails; * NANs and INFs are handled apprpriately. > is_close(218.345, 220, 1, .05) # OHMs > is_close(a, b, ULP, 2) # ULPs > is_close(a, b, AU, .001) # astronomical units > > > I don't see anyway to generalise those with just a function. Generalise in what way? > By using objects we can do a bit more. I seem to recall coming across > measurement objects some place. They keep a bit more context with them. A full system of <value + unit> arithmetic is a *much* bigger problem than just calculating error estimates correctly, and should be a third-party library before even considering it for the std lib. -- Steve From mistersheik at gmail.com Thu Jan 15 01:22:05 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Wed, 14 Jan 2015 16:22:05 -0800 (PST) Subject: [Python-ideas] What's going on with PEP 448 - Additional Unpacking Generalizations ? Message-ID: <f97552ab-bfb8-434b-bc68-5120d394682e@googlegroups.com> It says "standards track", but it doesn't seem to work in Python 3.4. I am really looking forward to this! Any chance we can get a from __future__ for inclusion in 3.4? Or maybe it can make it into 3.5? Best, Neil -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150114/4dd736e5/attachment-0001.html> From rosuav at gmail.com Thu Jan 15 12:20:01 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 15 Jan 2015 22:20:01 +1100 Subject: [Python-ideas] What's going on with PEP 448 - Additional Unpacking Generalizations ? In-Reply-To: <f97552ab-bfb8-434b-bc68-5120d394682e@googlegroups.com> References: <f97552ab-bfb8-434b-bc68-5120d394682e@googlegroups.com> Message-ID: <CAPTjJmpGkgh0onpJVBYB4KXOi-n8HRDx2R55XJFteO=ty3FRVQ@mail.gmail.com> On Thu, Jan 15, 2015 at 11:22 AM, Neil Girdhar <mistersheik at gmail.com> wrote: > It says "standards track", but it doesn't seem to work in Python 3.4. I am > really looking forward to this! Any chance we can get a from __future__ for > inclusion in 3.4? Or maybe it can make it into 3.5? It also says "Status: Draft", which means it hasn't (yet) been accepted as part of the language. That means it's definitely not going to get into 3.4; it might be able to get into 3.5 if it gets approved very shortly, but the time frame is narrowing. Last chance to get it into 3.5 would be Beta 1, scheduled for May 24th (see PEP 478), and ideally it should get into an alpha before that. Would be nice to actually have it land somewhere. Seems it's been proposed for every Python version since 3.0. ChrisA From wolfgang.maier at biologie.uni-freiburg.de Thu Jan 15 15:31:31 2015 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Thu, 15 Jan 2015 15:31:31 +0100 Subject: [Python-ideas] pep 397 In-Reply-To: <CAPTjJmrGwWr7G84S_9maK6Su9GudAHwLF7Omu-v7ha0SVyLkbw@mail.gmail.com> References: <CACPPHzsuATV4Gkn2fCE-P3Tr4frjQ9TuEATU5cg6q3vzR9pS0w@mail.gmail.com> <CAPTjJmoav+k8xa3u2LU84c8_UXMkEerFmMSnhgwkM=g2vtG4AQ@mail.gmail.com> <CACPPHzuMqWfFqOFdRQbqpXMJLr078asqeBQv4k2Fn=S9KWmyEg@mail.gmail.com> <CAPTjJmrGwWr7G84S_9maK6Su9GudAHwLF7Omu-v7ha0SVyLkbw@mail.gmail.com> Message-ID: <m98j03$u3g$1@ger.gmane.org> On 01.01.2015 14:27, Chris Angelico wrote: > On Thu, Jan 1, 2015 at 11:47 PM, Liam Marsh <liam.marsh.home at gmail.com> wrote: >> "I suspect you don't >> need any changes to the language to make this work" >> >> - you are right, this problem does not affect the language itself, but the >> idlelib library.... >> - I was thinking of an "pyidle" executable next to the "py" and "pyw" >> executables. >> I am also trying to create a small script to do this... > > A small script ought to be able to handle it. All it needs to do is: > > 1) Read in the first line of the file. > 2) See if it can be parsed according to PEP 397 > 3) Invoke IDLE. > Yes and no. It is certainly doable, but it also means coding things twice since the functionality already exists in the launcher. What would be good is if the launcher had an additional flag telling it to not actually start python, but just report the path to it, something like: $ py --dry-run my_script C:\Python34\python.exe From rosuav at gmail.com Thu Jan 15 15:47:46 2015 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 16 Jan 2015 01:47:46 +1100 Subject: [Python-ideas] pep 397 In-Reply-To: <m98j03$u3g$1@ger.gmane.org> References: <CACPPHzsuATV4Gkn2fCE-P3Tr4frjQ9TuEATU5cg6q3vzR9pS0w@mail.gmail.com> <CAPTjJmoav+k8xa3u2LU84c8_UXMkEerFmMSnhgwkM=g2vtG4AQ@mail.gmail.com> <CACPPHzuMqWfFqOFdRQbqpXMJLr078asqeBQv4k2Fn=S9KWmyEg@mail.gmail.com> <CAPTjJmrGwWr7G84S_9maK6Su9GudAHwLF7Omu-v7ha0SVyLkbw@mail.gmail.com> <m98j03$u3g$1@ger.gmane.org> Message-ID: <CAPTjJmoXz75TDHECBxOK-GoZLrEJYXSn0faKUHx4z-1CMrKjVQ@mail.gmail.com> On Fri, Jan 16, 2015 at 1:31 AM, Wolfgang Maier <wolfgang.maier at biologie.uni-freiburg.de> wrote: > What would be good is if the launcher had an additional flag telling it to > not actually start python, but just report the path to it, something like: > > $ py --dry-run my_script > > C:\Python34\python.exe Ooh that would be a nice feature. I'd be inclined to call it --query rather than --dry-run, but that's just bike-shedding. Any other programs could make use of this feature too - maybe they want to run a wrapper that ultimately invokes some script, so the wrapper gets written to be 2.7 and 3.3+ compatible, and then design it to be run using the target script's interpreter. ChrisA From skip.montanaro at gmail.com Thu Jan 15 15:52:46 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Thu, 15 Jan 2015 08:52:46 -0600 Subject: [Python-ideas] pep 397 In-Reply-To: <m98j03$u3g$1@ger.gmane.org> References: <CACPPHzsuATV4Gkn2fCE-P3Tr4frjQ9TuEATU5cg6q3vzR9pS0w@mail.gmail.com> <CAPTjJmoav+k8xa3u2LU84c8_UXMkEerFmMSnhgwkM=g2vtG4AQ@mail.gmail.com> <CACPPHzuMqWfFqOFdRQbqpXMJLr078asqeBQv4k2Fn=S9KWmyEg@mail.gmail.com> <CAPTjJmrGwWr7G84S_9maK6Su9GudAHwLF7Omu-v7ha0SVyLkbw@mail.gmail.com> <m98j03$u3g$1@ger.gmane.org> Message-ID: <CANc-5Uw=yHDtmGDozD5iMWz0S3K92nZx1qUXykiajJZspm2WiQ@mail.gmail.com> On Thu, Jan 15, 2015 at 8:31 AM, Wolfgang Maier <wolfgang.maier at biologie.uni-freiburg.de> wrote: > > What would be good is if the launcher had an additional flag telling it to not actually start python, but just report the path to it, something like: > > $ py --dry-run my_script > > C:\Python34\python.exe How about: % python -c 'import sys ; print sys.executable' /opt/local/bin/python ? Skip From rosuav at gmail.com Thu Jan 15 15:54:37 2015 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 16 Jan 2015 01:54:37 +1100 Subject: [Python-ideas] pep 397 In-Reply-To: <CANc-5Uw=yHDtmGDozD5iMWz0S3K92nZx1qUXykiajJZspm2WiQ@mail.gmail.com> References: <CACPPHzsuATV4Gkn2fCE-P3Tr4frjQ9TuEATU5cg6q3vzR9pS0w@mail.gmail.com> <CAPTjJmoav+k8xa3u2LU84c8_UXMkEerFmMSnhgwkM=g2vtG4AQ@mail.gmail.com> <CACPPHzuMqWfFqOFdRQbqpXMJLr078asqeBQv4k2Fn=S9KWmyEg@mail.gmail.com> <CAPTjJmrGwWr7G84S_9maK6Su9GudAHwLF7Omu-v7ha0SVyLkbw@mail.gmail.com> <m98j03$u3g$1@ger.gmane.org> <CANc-5Uw=yHDtmGDozD5iMWz0S3K92nZx1qUXykiajJZspm2WiQ@mail.gmail.com> Message-ID: <CAPTjJmo0y_dS3yAwAd4xH7jv41qZOVKzuj4zX__Hxb7k6SzLDw@mail.gmail.com> On Fri, Jan 16, 2015 at 1:52 AM, Skip Montanaro <skip.montanaro at gmail.com> wrote: > On Thu, Jan 15, 2015 at 8:31 AM, Wolfgang Maier > <wolfgang.maier at biologie.uni-freiburg.de> wrote: >> >> What would be good is if the launcher had an additional flag telling it to not actually start python, but just report the path to it, something like: >> >> $ py --dry-run my_script >> >> C:\Python34\python.exe > > How about: > > % python -c 'import sys ; print sys.executable' > /opt/local/bin/python It has to be able to be controlled by a shebang in the script itself. ChrisA From wolfgang.maier at biologie.uni-freiburg.de Thu Jan 15 15:55:41 2015 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Thu, 15 Jan 2015 15:55:41 +0100 Subject: [Python-ideas] pep 397 In-Reply-To: <CAPTjJmoXz75TDHECBxOK-GoZLrEJYXSn0faKUHx4z-1CMrKjVQ@mail.gmail.com> References: <CACPPHzsuATV4Gkn2fCE-P3Tr4frjQ9TuEATU5cg6q3vzR9pS0w@mail.gmail.com> <CAPTjJmoav+k8xa3u2LU84c8_UXMkEerFmMSnhgwkM=g2vtG4AQ@mail.gmail.com> <CACPPHzuMqWfFqOFdRQbqpXMJLr078asqeBQv4k2Fn=S9KWmyEg@mail.gmail.com> <CAPTjJmrGwWr7G84S_9maK6Su9GudAHwLF7Omu-v7ha0SVyLkbw@mail.gmail.com> <m98j03$u3g$1@ger.gmane.org> <CAPTjJmoXz75TDHECBxOK-GoZLrEJYXSn0faKUHx4z-1CMrKjVQ@mail.gmail.com> Message-ID: <m98kdd$kjo$1@ger.gmane.org> On 15.01.2015 15:47, Chris Angelico wrote: > On Fri, Jan 16, 2015 at 1:31 AM, Wolfgang Maier > <wolfgang.maier at biologie.uni-freiburg.de> wrote: >> What would be good is if the launcher had an additional flag telling it to >> not actually start python, but just report the path to it, something like: >> >> $ py --dry-run my_script >> >> C:\Python34\python.exe > > Ooh that would be a nice feature. I'd be inclined to call it --query > rather than --dry-run, but that's just bike-shedding. Any other > programs could make use of this feature too - maybe they want to run a > wrapper that ultimately invokes some script, so the wrapper gets > written to be 2.7 and 3.3+ compatible, and then design it to be run > using the target script's interpreter. > Right, that was what I was thinking of, too. From wolfgang.maier at biologie.uni-freiburg.de Thu Jan 15 15:58:28 2015 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Thu, 15 Jan 2015 15:58:28 +0100 Subject: [Python-ideas] pep 397 In-Reply-To: <CANc-5Uw=yHDtmGDozD5iMWz0S3K92nZx1qUXykiajJZspm2WiQ@mail.gmail.com> References: <CACPPHzsuATV4Gkn2fCE-P3Tr4frjQ9TuEATU5cg6q3vzR9pS0w@mail.gmail.com> <CAPTjJmoav+k8xa3u2LU84c8_UXMkEerFmMSnhgwkM=g2vtG4AQ@mail.gmail.com> <CACPPHzuMqWfFqOFdRQbqpXMJLr078asqeBQv4k2Fn=S9KWmyEg@mail.gmail.com> <CAPTjJmrGwWr7G84S_9maK6Su9GudAHwLF7Omu-v7ha0SVyLkbw@mail.gmail.com> <m98j03$u3g$1@ger.gmane.org> <CANc-5Uw=yHDtmGDozD5iMWz0S3K92nZx1qUXykiajJZspm2WiQ@mail.gmail.com> Message-ID: <m98kik$kjo$2@ger.gmane.org> On 15.01.2015 15:52, Skip Montanaro wrote: > On Thu, Jan 15, 2015 at 8:31 AM, Wolfgang Maier > <wolfgang.maier at biologie.uni-freiburg.de> wrote: >> >> What would be good is if the launcher had an additional flag telling it to not actually start python, but just report the path to it, something like: >> >> $ py --dry-run my_script >> >> C:\Python34\python.exe > > How about: > > % python -c 'import sys ; print sys.executable' > /opt/local/bin/python > > ? What is of interest here is not what is hiding behind "python". It is about finding out which python would have been selected by the Python launcher based on the shebang line of the script. Wolfgang From steve at pearwood.info Thu Jan 15 16:54:06 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 16 Jan 2015 02:54:06 +1100 Subject: [Python-ideas] pep 397 In-Reply-To: <CAPTjJmoXz75TDHECBxOK-GoZLrEJYXSn0faKUHx4z-1CMrKjVQ@mail.gmail.com> References: <CACPPHzsuATV4Gkn2fCE-P3Tr4frjQ9TuEATU5cg6q3vzR9pS0w@mail.gmail.com> <CAPTjJmoav+k8xa3u2LU84c8_UXMkEerFmMSnhgwkM=g2vtG4AQ@mail.gmail.com> <CACPPHzuMqWfFqOFdRQbqpXMJLr078asqeBQv4k2Fn=S9KWmyEg@mail.gmail.com> <CAPTjJmrGwWr7G84S_9maK6Su9GudAHwLF7Omu-v7ha0SVyLkbw@mail.gmail.com> <m98j03$u3g$1@ger.gmane.org> <CAPTjJmoXz75TDHECBxOK-GoZLrEJYXSn0faKUHx4z-1CMrKjVQ@mail.gmail.com> Message-ID: <20150115155405.GA18556@ando.pearwood.info> On Fri, Jan 16, 2015 at 01:47:46AM +1100, Chris Angelico wrote: > On Fri, Jan 16, 2015 at 1:31 AM, Wolfgang Maier > <wolfgang.maier at biologie.uni-freiburg.de> wrote: > > What would be good is if the launcher had an additional flag telling it to > > not actually start python, but just report the path to it, something like: > > > > $ py --dry-run my_script > > > > C:\Python34\python.exe > > Ooh that would be a nice feature. I'd be inclined to call it --query > rather than --dry-run, but that's just bike-shedding. Sometimes bike-shedding has a bad reputation. Option names should reflect what they do. --dry-run suggests that it will run the script except for anything destructive (e.g. writing or deleting files, printing a 300 page report). --query is better, but query what? This isn't a common command that needs to be short even at the expense of being cryptic, so how about a self-documenting switch like: --which-executable --which-python and -w for those who insist on having short cryptic switches :-) -- Steven From rosuav at gmail.com Thu Jan 15 16:58:45 2015 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 16 Jan 2015 02:58:45 +1100 Subject: [Python-ideas] pep 397 In-Reply-To: <20150115155405.GA18556@ando.pearwood.info> References: <CACPPHzsuATV4Gkn2fCE-P3Tr4frjQ9TuEATU5cg6q3vzR9pS0w@mail.gmail.com> <CAPTjJmoav+k8xa3u2LU84c8_UXMkEerFmMSnhgwkM=g2vtG4AQ@mail.gmail.com> <CACPPHzuMqWfFqOFdRQbqpXMJLr078asqeBQv4k2Fn=S9KWmyEg@mail.gmail.com> <CAPTjJmrGwWr7G84S_9maK6Su9GudAHwLF7Omu-v7ha0SVyLkbw@mail.gmail.com> <m98j03$u3g$1@ger.gmane.org> <CAPTjJmoXz75TDHECBxOK-GoZLrEJYXSn0faKUHx4z-1CMrKjVQ@mail.gmail.com> <20150115155405.GA18556@ando.pearwood.info> Message-ID: <CAPTjJmq3owuGFf40Zm+JvgwhNrNtt++_KLipN50Bo-mqMaKCqA@mail.gmail.com> On Fri, Jan 16, 2015 at 2:54 AM, Steven D'Aprano <steve at pearwood.info> wrote: >> Ooh that would be a nice feature. I'd be inclined to call it --query >> rather than --dry-run, but that's just bike-shedding. > > Sometimes bike-shedding has a bad reputation. Option names should > reflect what they do. --dry-run suggests that it will run the script > except for anything destructive (e.g. writing or deleting files, > printing a 300 page report). --query is better, but query what? "Just" bike-shedding means that I support the feature, and think it's a great idea, but all it needs is a different name. And you're right, --which-executable would be better than --query. ChrisA From chris.barker at noaa.gov Thu Jan 15 17:06:31 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 15 Jan 2015 08:06:31 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <m97pv1$8fa$1@ger.gmane.org> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <20150114103919.GO10173@ando.pearwood.info> <m96meq$7j0$1@ger.gmane.org> <20150115000416.GP10173@ando.pearwood.info> <m977on$can$1@ger.gmane.org> <5021473497021594897@unknownmsgid> <m97pv1$8fa$1@ger.gmane.org> Message-ID: <-2842639290373140778@unknownmsgid> > On Jan 14, 2015, at 11:24 PM, Ron Adam <ron3200 at gmail.com> wrote: > > (abs(a - b) / b) / units <= good If units and good are positive, and b is negative, this fails - got to use abs everywhere. But I'm sure both Steven and my code do that. > Without that you need to either scale the inputs, or scale the tolerance. That's why we want a relative tolerance -- that scales the tolerance for you -- that's the whole point. > In most cases when you are entering a tolerance, you will also be working with some multiple of units. But it seems that can confuse as much as it helps. Again, relative tolerance takes care of that for you- two values will have the same relative error whether they are stored in angstroms or meters -- the exponent will be wildly different, but the mantissas will be the same, so the relative error doesn't change. If you want to compare values In Meters According to a tolerance in angstroms, then you a) want absolute tolerance, and b) want a when units system. > In any case, I was just trying to come up with some practical example, and at the same time was thinking about how different units of measure can effect it. Is_close(computed_value, expected_value, tolerance) Really does what you want, provided that the inputs are in the same units, and neither is zero. Tolerance is unitless. I'll take a look at Steven's code it sound like it's very much like where I was trying to go. -Chris > > >>>> By using objects we can do a bit more. I seem to recall coming >>>> across measurement objects some place. They keep a bit more context with them. > >> Sure, there are various unit/physical quantities objects out there, many >> numpy-based. But I think is orthogonal to comparing floating point >> numbers. > >> >> And a sime floating point comparison function_may_ belong the >> standard library, but not a more complex physical quantity system > > Yes, for the most part I agree, but think such functions would be useful in those more complex systems as well. > > Ron > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From chris.barker at noaa.gov Thu Jan 15 17:52:36 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 15 Jan 2015 08:52:36 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <20150115072927.GA16607@ando.pearwood.info> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <20150114103919.GO10173@ando.pearwood.info> <m96meq$7j0$1@ger.gmane.org> <20150115000416.GP10173@ando.pearwood.info> <m977on$can$1@ger.gmane.org> <20150115072927.GA16607@ando.pearwood.info> Message-ID: <CALGmxEK4XhRarVNS9PrB6z7x5dmC+BAnUj4UzO3k6KhNGEjCqg@mail.gmail.com> On Wed, Jan 14, 2015 at 11:29 PM, Steven D'Aprano <steve at pearwood.info> wrote: > The question of which to use as the denominator is more subtle. Like > you, I used to think that you should choose ahead of time which value > was expected and which was actual, and divide by the actual. Or should > that be the expected? I could never decide which I wanted: error > relative to the expected, or error relative to the actual? And then I > could never remember which order the two arguments went. > I'm on the fence about this -- it seems clear to me that if the user has specified an "expected" value, that tolerance would clearly be based on that magnitude. If nothing else, that is because you would more likely have many computed values for each expected value than the other way around. And I was thinking that calling the arguments something like "actual" and "expected" would make that clear in the API, and would certainly be documentable. But the fact that it was never clear to you , even as you were writing the code, is good evidence that it wouldn't be clear to everyone ;-) Error > calculations should be symmetrical, so that > > error(a, b) == error(b, a) > That does make it simpler to think about and reason about, and makes the use-cased more universal (or at least appear more universal) "are these two values close" is a simple question to ask, if not to answer. regardless of whether you have absolute or relative error. Furthermore, > for safety you normally want the larger estimate of error, not the > smaller: given the choice between > > (abs(a - b))/abs(a) > > versus > > (abs(a - b))/abs(b) > > > you want the *larger* error estimate, which means the *smaller* > denominator. That's the conservative way of doing it. > Which is what the Boost "strong" method does -- rather than compute teh max and use that, it computes both and does an "and" check -- but same result. A concrete example: given a=5 and b=7, we have: > > absolute error = 2 > relative error (calculated relative to a) = 0.4 > relative error (calculated relative to b) = 0.286 > > That is, b is off by 40% relative to a; or a is off by 28.6% relative to > b. Or another way to put it, given that a is the "true" value, b is 40% > too big; or if you prefer, 28.6% of b is in error. > The think is, in general, we use this to test for small errors, with low tolerance. Which value you use to scale only makes a big difference if the values are far apart, in which case the error will be larger than the tolerance anyway. In your above example, if the tolerance is, say, 1%, then if makes no difference which you use -- you are way off anyway. And in the common use cases, comparing a double precision floating point calculation, tolerances are more likely to be around 1e-12, not 1e-2 anyway! So I think that which relative tolerance you use makes little difference in practice, but it might as well be robust and symmetrical. (another option is to use the average of the two values to scale the tolerance, but why bother?) > Note that you would never compare to an expected value of zero. > > You *cannot* compare to an expected value of zero, but you certainly can > be in a situation where you would like to: math.sin(math.pi) should > return 0.0, but doesn't, it returns 1.2246063538223773e-16 instead. What > is the relative error of the sin function at x = math.pi? > there isn't one -- that's the whole point -- but there is an absolute error, so that's what you should check. We all agree a relative error involving zero is not defined / possible. So the question is what to do? 1) Raise a ValueError 2) Let it return "not close" regardless of the other input -- that's mathematically correct, nothing is relatively close to zero. 3) Automagically switch to an absolute tolerance near zero -- user specified what it should be. It seems the implementations (Boost's, for instance) I've seen simply do (2). But if the point of putting this in the standard library is that people will have something that can be used for common use cases without thinking about it, I think maybe (1) or (3) would be better. Probably (3), as raising an Exception would make a mess of this if it were inside a comprehension or something. > What Chris is looking for is a way to get a closeness function that works > > most of the time. (He posted while I'm writing this.) > > I think the function I have in the statistics test suite is that > function. I'll take a look -- it does sound like you've already done pretty much what I have in mind. > I would like to see ULP calculations offered as well, but > Mark thinks that's unnecessary and I'm not going to go to the > battlements to fight for ULPs. I suppose it could be added later -- I agree that it could be pretty useful, but that it's also much harder to wrap your brain around, and really for a different use-case. > * you provide two values, and at least one of an absolute error > tolerance and a relative error; > * if the error is less than the error(s) you provided, the test > passes, otherwise it fails; > * NANs and INFs are handled appropriately. Is this different than the numpy implementation: http://docs.scipy.org/doc/numpy/reference/generated/numpy.allclose.html In that (according to the docs, I haven't looked at the code): """ The relative difference (*rtol* * abs(*b*)) and the absolute difference *atol* are added together to compare against the absolute difference between *a *and *b*. """ I think it should either be: - you specify atol or rtol, but only one is used. or - some way to transition from a relative tolerance to an absolute on near zero -- I a haven't figured out if that can be done smoothly yet. [Also, it looks like numpy computes the tolerance from the second input, rather than looking at both, resulting in an asymetric result -- discussed above.) I've always thought the numpy approach is weird, but now that I think about it, it would be really horrible (with defaults) for small numbers: rtol defaults to 1e-5, atol to 1e-8 -- too big, I think, but not the point here. In [23]: a, b = 1.1, 1.2 In [24]: np.allclose(a,b) Out[24]: False ## that's good there are pretty far apart In [27]: a, b = 1.1e15, 1.2e15 In [28]: np.allclose(a,b) Out[28]: False # same thing for large values -- still good. In [25]: a, b = 1.1e-15, 1.2e-15 In [26]: np.allclose(a,b) Out[26]: True OOPS! this is NOT what most people would expect!! In [30]: np.allclose(a,b, atol=0.0) Out[30]: False There we go. But with a default atol as large as 1e-8, this is a rally bad idea! I can only imagine whoever wrote this was thinking about really large values, but not really small values... (I think this has been brought up in the numpy community, but I'll make sure) -Chris > > > is_close(218.345, 220, 1, .05) # OHMs > > is_close(a, b, ULP, 2) # ULPs > > is_close(a, b, AU, .001) # astronomical units > > > > > > I don't see anyway to generalise those with just a function. > > Generalise in what way? > > > > By using objects we can do a bit more. I seem to recall coming across > > measurement objects some place. They keep a bit more context with them. > > A full system of <value + unit> arithmetic is a *much* bigger problem > than just calculating error estimates correctly, and should be a > third-party library before even considering it for the std lib. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150115/3a8c4a1c/attachment-0001.html> From ron3200 at gmail.com Thu Jan 15 18:42:18 2015 From: ron3200 at gmail.com (Ron Adam) Date: Thu, 15 Jan 2015 11:42:18 -0600 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <20150115072927.GA16607@ando.pearwood.info> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <20150114103919.GO10173@ando.pearwood.info> <m96meq$7j0$1@ger.gmane.org> <20150115000416.GP10173@ando.pearwood.info> <m977on$can$1@ger.gmane.org> <20150115072927.GA16607@ando.pearwood.info> Message-ID: <m98u5r$22t$1@ger.gmane.org> On 01/15/2015 01:29 AM, Steven D'Aprano wrote: > On Wed, Jan 14, 2015 at 08:13:42PM -0600, Ron Adam wrote: > The question of which to use as the denominator is more subtle. Like > you, I used to think that you should choose ahead of time which value > was expected and which was actual, and divide by the actual. Or should > that be the expected? I could never decide which I wanted: error > relative to the expected, or error relative to the actual? And then I > could never remember which order the two arguments went. > > Finally I read Bruce Dawson (I've already linked to his blog three or > four times) and realised that he is correct and I was wrong. Error > calculations should be symmetrical, so that > > error(a, b) == error(b, a) > > regardless of whether you have absolute or relative error. Furthermore, > for safety you normally want the larger estimate of error, not the > smaller: given the choice between > > (abs(a - b))/abs(a) > > versus > > (abs(a - b))/abs(b) > > > you want the *larger* error estimate, which means the *smaller* > denominator. That's the conservative way of doing it. > > A concrete example: given a=5 and b=7, we have: > > absolute error = 2 > relative error (calculated relative to a) = 0.4 > relative error (calculated relative to b) = 0.286 > > That is, b is off by 40% relative to a; or a is off by 28.6% relative to > b. Or another way to put it, given that a is the "true" value, b is 40% > too big; or if you prefer, 28.6% of b is in error. > > Whew! Percentages are hard! *wink* Ewww the P word. :) > The conservative, "safe" way to handle this is to just treat the error > function as symmetrical What if we are not concerned with the location two points are relative to zero? Or if the numbers straddle zero? and always report the larger of the two relative > errors (excluding the case where the denominator is 0, in which case > the relative error is either 100% or it doesn't exist). Worst case, you > may reject some values which you should accept, but you will never > accept any values that you should reject. Consider two points that are a constant distance apart, but moving relative to zero. Their closeness doesn't change, but the relative error in respect to each other (and zero) does change. There is an implicit assumption that the number system used and the origin the numbers are measured from are chosen and relate to each other in some expected way. When ever you supply all the numbers, like in a test, it's not a problem, you just give good numbers. >> Note that you would never compare to an expected value of zero. > > You *cannot* compare to an expected value of zero, but you certainly can > be in a situation where you would like to: math.sin(math.pi) should > return 0.0, but doesn't, it returns 1.2246063538223773e-16 instead. What > is the relative error of the sin function at x = math.pi? > > >> relerr(a - b, expected_feet) < tolerance # relative feet from b >> relerr(a - 0, expected_feet) < tolerance # relative feet from zero >> relerr(a - b, ulp) # percentage of ulp's > > I don't understand what you think these three examples are showing. A percentage of an expected distance. Error of two points compared to a specific distance. >>> relerr(5 - -5, 10) 0.0 I think unless you use decimal, the ulp example will either be zero or some large multiple of ulp. > Take a look at the statistics test suite. I definitely will. :-) > I'll be the first to admit > that the error tolerances are plucked from thin air, based on what I > think are "close enough", but they show how such a function might work: > > * you provide two values, and at least one of an absolute error > tolerance and a relative error; > * if the error is less than the error(s) you provided, the test > passes, otherwise it fails; > * NANs and INFs are handled apprpriately. > > >> is_close(218.345, 220, 1, .05) # OHMs >> is_close(a, b, ULP, 2) # ULPs >> is_close(a, b, AU, .001) # astronomical units >> >> >> I don't see anyway to generalise those with just a function. > > Generalise in what way? I meant a function that would work in many places without giving some sort size and tolerance hints. Given two floating point numbers and noting else, I don't think you can tell if they represent something that is close without assuming some sort of context. At best, you need to assume the distance from zero and the numbers used are chosen to give a meaningful return value. While that can sometimes work, I don't think you can depend on it. >> By using objects we can do a bit more. I seem to recall coming across >> measurement objects some place. They keep a bit more context with them. > > A full system of <value + unit> arithmetic is a *much* bigger problem > than just calculating error estimates correctly, and should be a > third-party library before even considering it for the std lib. Yes, I agree. There are a few of them out there already. Cheers, Ron From mistersheik at gmail.com Thu Jan 15 01:23:50 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Wed, 14 Jan 2015 16:23:50 -0800 (PST) Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> Message-ID: <3b503462-8868-4c02-9988-bd53c66fe271@googlegroups.com> You might be interested in my question: http://stackoverflow.com/questions/4028889/floating-point-equality-in-python On Monday, January 12, 2015 at 2:58:30 PM UTC-5, Chris Barker wrote: > > Now that we're talking about floating point conveniences (math.nan, > linspace): > > What about putting an > > almost_equal(x,y,tol=1e14) > > (or close(), or...) in the math module. > > AFAICT, in the standard library, there is only: > > unittest.TestCase.assertAlmostEqual > > but it: > > A) is buried in the unittest.TestCase class > > B) is an assertion, so you can't use it as a general test (easily) > > C) uses number of decimal digits or an absolute delta, but does not > provide a significant figures comparison, which is most likely what's > wanted (and a bit harder to write yourself) > > numpy provides allclose() (and isclose() ), which is pretty much what I'm > suggesting. > > Anyone else think this would be a good idea to add to the stdlib? > > -Chris > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.... at noaa.gov <javascript:> > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150114/cfa444cb/attachment.html> From guido at python.org Thu Jan 15 19:04:26 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 15 Jan 2015 10:04:26 -0800 Subject: [Python-ideas] What's going on with PEP 448 - Additional Unpacking Generalizations ? In-Reply-To: <CAPTjJmpGkgh0onpJVBYB4KXOi-n8HRDx2R55XJFteO=ty3FRVQ@mail.gmail.com> References: <f97552ab-bfb8-434b-bc68-5120d394682e@googlegroups.com> <CAPTjJmpGkgh0onpJVBYB4KXOi-n8HRDx2R55XJFteO=ty3FRVQ@mail.gmail.com> Message-ID: <CAP7+vJKKmwuWq+4Ps2drFuQA5ParxdPKe_XDuAFATiQrZ6TZMQ@mail.gmail.com> I see no big problem with it from a language design perspective, but also no reason to commit huge resources to it -- it's all very much in the "nice to have" category. Unless you find a champion who wants to review the (old) patch and so on, what's the point of accepting the PEP? On Jan 15, 2015 3:28 AM, "Chris Angelico" <rosuav at gmail.com> wrote: > On Thu, Jan 15, 2015 at 11:22 AM, Neil Girdhar <mistersheik at gmail.com> > wrote: > > It says "standards track", but it doesn't seem to work in Python 3.4. I > am > > really looking forward to this! Any chance we can get a from __future__ > for > > inclusion in 3.4? Or maybe it can make it into 3.5? > > It also says "Status: Draft", which means it hasn't (yet) been > accepted as part of the language. That means it's definitely not going > to get into 3.4; it might be able to get into 3.5 if it gets approved > very shortly, but the time frame is narrowing. Last chance to get it > into 3.5 would be Beta 1, scheduled for May 24th (see PEP 478), and > ideally it should get into an alpha before that. > > Would be nice to actually have it land somewhere. Seems it's been > proposed for every Python version since 3.0. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150115/b9bb9199/attachment.html> From chris.barker at noaa.gov Thu Jan 15 20:45:22 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 15 Jan 2015 11:45:22 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <m98u5r$22t$1@ger.gmane.org> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <20150114103919.GO10173@ando.pearwood.info> <m96meq$7j0$1@ger.gmane.org> <20150115000416.GP10173@ando.pearwood.info> <m977on$can$1@ger.gmane.org> <20150115072927.GA16607@ando.pearwood.info> <m98u5r$22t$1@ger.gmane.org> Message-ID: <CALGmxE+NA6AwjBaHot1qcrvPv6zh2v81FA_qbxVZUb9HaUJYkg@mail.gmail.com> On Thu, Jan 15, 2015 at 9:42 AM, Ron Adam <ron3200 at gmail.com> wrote: > The conservative, "safe" way to handle this is to just treat the error >> function as symmetrical >> > What if we are not concerned with the location two points are relative to > zero? not sure what location two points are relative to zero means. Or if the numbers straddle zero? Good catch -- at least if the numbers are symmetric to zero -- i.e. a == -b, then they will never be "close" according to our working definition of relative tolerance: a = -b err == 2*a abs_tol = rel_tol * a #rel_tol is small and positive abs_tol is always smaller than a, so always smaller than the error, so never "close". In fact, this is true for any numbers that straddle zero: err == abs(a,b) abs_tol = rel_tol* max(abs(a), abs(b)) abs_tol <= err # provided 0< rel_tol < 1 So I was thinking the issues were near zero, but they actually are for any time: a <= 0 <= b (or b <= 0 <= a ) that is, any time they straddle or include zero. In this case, you need an absolute tolerance defined. Still figuring out how to use that and provide a smooth transition... Steven, does your code address this somehow? -Chris > > and always report the larger of the two relative > >> errors (excluding the case where the denominator is 0, in which case >> the relative error is either 100% or it doesn't exist). Worst case, you >> may reject some values which you should accept, but you will never >> accept any values that you should reject. >> > > Consider two points that are a constant distance apart, but moving > relative to zero. Their closeness doesn't change, but the relative error > in respect to each other (and zero) does change. > > There is an implicit assumption that the number system used and the origin > the numbers are measured from are chosen and relate to each other in some > expected way. > > When ever you supply all the numbers, like in a test, it's not a problem, > you just give good numbers. > > > Note that you would never compare to an expected value of zero. >>> >> >> You *cannot* compare to an expected value of zero, but you certainly can >> be in a situation where you would like to: math.sin(math.pi) should >> return 0.0, but doesn't, it returns 1.2246063538223773e-16 instead. What >> is the relative error of the sin function at x = math.pi? >> >> >> relerr(a - b, expected_feet) < tolerance # relative feet from b >>> relerr(a - 0, expected_feet) < tolerance # relative feet from zero >>> relerr(a - b, ulp) # percentage of ulp's >>> >> >> I don't understand what you think these three examples are showing. >> > > A percentage of an expected distance. > > Error of two points compared to a specific distance. > > >>> relerr(5 - -5, 10) > 0.0 > > I think unless you use decimal, the ulp example will either be zero or > some large multiple of ulp. > > > Take a look at the statistics test suite. >> > > I definitely will. :-) > > I'll be the first to admit >> that the error tolerances are plucked from thin air, based on what I >> think are "close enough", but they show how such a function might work: >> >> * you provide two values, and at least one of an absolute error >> tolerance and a relative error; >> * if the error is less than the error(s) you provided, the test >> passes, otherwise it fails; >> * NANs and INFs are handled apprpriately. >> >> >> is_close(218.345, 220, 1, .05) # OHMs >>> is_close(a, b, ULP, 2) # ULPs >>> is_close(a, b, AU, .001) # astronomical units >>> >>> >>> I don't see anyway to generalise those with just a function. >>> >> >> Generalise in what way? >> > > I meant a function that would work in many places without giving some sort > size and tolerance hints. > > Given two floating point numbers and noting else, I don't think you can > tell if they represent something that is close without assuming some sort > of context. At best, you need to assume the distance from zero and the > numbers used are chosen to give a meaningful return value. While that can > sometimes work, I don't think you can depend on it. > > > By using objects we can do a bit more. I seem to recall coming across >>> measurement objects some place. They keep a bit more context with them. >>> >> >> A full system of <value + unit> arithmetic is a *much* bigger problem >> than just calculating error estimates correctly, and should be a >> third-party library before even considering it for the std lib. >> > > Yes, I agree. There are a few of them out there already. > > Cheers, > Ron > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150115/b317d77d/attachment-0001.html> From ron3200 at gmail.com Thu Jan 15 22:47:43 2015 From: ron3200 at gmail.com (Ron Adam) Date: Thu, 15 Jan 2015 15:47:43 -0600 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CALGmxE+NA6AwjBaHot1qcrvPv6zh2v81FA_qbxVZUb9HaUJYkg@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <20150114103919.GO10173@ando.pearwood.info> <m96meq$7j0$1@ger.gmane.org> <20150115000416.GP10173@ando.pearwood.info> <m977on$can$1@ger.gmane.org> <20150115072927.GA16607@ando.pearwood.info> <m98u5r$22t$1@ger.gmane.org> <CALGmxE+NA6AwjBaHot1qcrvPv6zh2v81FA_qbxVZUb9HaUJYkg@mail.gmail.com> Message-ID: <m99ci0$jhl$1@ger.gmane.org> On 01/15/2015 01:45 PM, Chris Barker wrote: > On Thu, Jan 15, 2015 at 9:42 AM, Ron Adam > <ron3200 at gmail.com > <mailto:ron3200 at gmail.com>> wrote: > > The conservative, "safe" way to handle this is to just treat the error > function as symmetrical > > What if we are not concerned with the location two points are relative > to zero? > > > not sure what location two points are relative to zero means. A number relative to zero is it's value. A number relative to another number is it's distance from the other number. If we divide that from one of the numbers, we are getting the ratio of their distance from each to the distance they are from zero. Seven suggest using the closer one as the divisor so you get a bigger result, but I prefer the larger one so I get a result between 0 and 1. A strategy for numbers of different signs may be to swap the sign on the smaller and add the smaller to the larger... err.. that isn't clear at all. How about this? def is_close(x, y, delta=.001): """ Check is x and y are close to each other. """ if x == y: return True # Can't get smaller than this. if x == 0 or y == 0: return False # Nothing is close to zero. if abs(x) < abs(y): # Make x the larger one. x, y = y, x if x * y < 0: # Signs differ. x = x - 2.0 * y # Move x and y same distance. y = -y return (x - y)/float(x) <= delta Cheers, Ron From mistersheik at gmail.com Thu Jan 15 22:52:21 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 15 Jan 2015 16:52:21 -0500 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CALGmxEJet44RNdD6xD_bYPcUniwOUTX6j7pL79SjSSHEhhRZ5Q@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <3b503462-8868-4c02-9988-bd53c66fe271@googlegroups.com> <CALGmxEJet44RNdD6xD_bYPcUniwOUTX6j7pL79SjSSHEhhRZ5Q@mail.gmail.com> Message-ID: <CAA68w_kx1kyVnoPajUGGsJgCv=cQRwQ56RdFuBLZW15C676-Qw@mail.gmail.com> The point is that this function is already in Python and if you want to do something different, you should have a really good reason to do it differently. If you were to add a function to math, say math.close, it should work like numpy.allclose in my opinion. For reference, numpy does this: absolute(*a* - *b*) <= (*atol* + *rtol* * absolute(*b*)) where atol is an absolute tolerance and rtol is a relative tolerance (relative to the actual value b). This subsumes most of the proposals here. Best, Neil On Wed, Jan 14, 2015 at 7:48 PM, Chris Barker <chris.barker at noaa.gov> wrote: > On Wed, Jan 14, 2015 at 4:23 PM, Neil Girdhar <mistersheik at gmail.com> > wrote: > >> You might be interested in my question: >> >> >> http://stackoverflow.com/questions/4028889/floating-point-equality-in-python >> > > nothing new there, I'm afaid -- and no one seemed to have brought up the > issue with zero. > > -Chris > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150115/32cb2085/attachment.html> From mistersheik at gmail.com Thu Jan 15 22:53:08 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 15 Jan 2015 16:53:08 -0500 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CAA68w_kx1kyVnoPajUGGsJgCv=cQRwQ56RdFuBLZW15C676-Qw@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <3b503462-8868-4c02-9988-bd53c66fe271@googlegroups.com> <CALGmxEJet44RNdD6xD_bYPcUniwOUTX6j7pL79SjSSHEhhRZ5Q@mail.gmail.com> <CAA68w_kx1kyVnoPajUGGsJgCv=cQRwQ56RdFuBLZW15C676-Qw@mail.gmail.com> Message-ID: <CAA68w_kNru8kykrtBSdKwCjYghTZSrTGaeKco9aC0uNztrUqzw@mail.gmail.com> Note also that it does something reasonable with zero, which most of the proposals here do not do. On Thu, Jan 15, 2015 at 4:52 PM, Neil Girdhar <mistersheik at gmail.com> wrote: > The point is that this function is already in Python and if you want to do > something different, you should have a really good reason to do it > differently. If you were to add a function to math, say math.close, it > should work like numpy.allclose in my opinion. > > For reference, numpy does this: > > absolute(*a* - *b*) <= (*atol* + *rtol* * absolute(*b*)) > > where atol is an absolute tolerance and rtol is a relative tolerance > (relative to the actual value b). This subsumes most of the proposals here. > > Best, > > Neil > > On Wed, Jan 14, 2015 at 7:48 PM, Chris Barker <chris.barker at noaa.gov> > wrote: > >> On Wed, Jan 14, 2015 at 4:23 PM, Neil Girdhar <mistersheik at gmail.com> >> wrote: >> >>> You might be interested in my question: >>> >>> >>> http://stackoverflow.com/questions/4028889/floating-point-equality-in-python >>> >> >> nothing new there, I'm afaid -- and no one seemed to have brought up the >> issue with zero. >> >> -Chris >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.Barker at noaa.gov >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150115/c3520bd3/attachment.html> From ron3200 at gmail.com Thu Jan 15 23:06:15 2015 From: ron3200 at gmail.com (Ron Adam) Date: Thu, 15 Jan 2015 16:06:15 -0600 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <m99ci0$jhl$1@ger.gmane.org> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <20150114103919.GO10173@ando.pearwood.info> <m96meq$7j0$1@ger.gmane.org> <20150115000416.GP10173@ando.pearwood.info> <m977on$can$1@ger.gmane.org> <20150115072927.GA16607@ando.pearwood.info> <m98u5r$22t$1@ger.gmane.org> <CALGmxE+NA6AwjBaHot1qcrvPv6zh2v81FA_qbxVZUb9HaUJYkg@mail.gmail.com> <m99ci0$jhl$1@ger.gmane.org> Message-ID: <m99dko$4hr$1@ger.gmane.org> On 01/15/2015 03:47 PM, Ron Adam wrote: > How about this? > > > def is_close(x, y, delta=.001): > """ Check is x and y are close to each other. """ > if x == y: > return True # Can't get smaller than this. > if x == 0 or y == 0: > return False # Nothing is close to zero. > if abs(x) < abs(y): # Make x the larger one. > x, y = y, x > if x * y < 0: # Signs differ. > x = x - 2.0 * y # Move x and y same distance. > y = -y > return (x - y)/float(x) <= delta Remove the check of one being zero, it isn't needed. def is_close(x, y, delta=.001): """ Check is x and y are close to each other. """ if x == y: return True # Can't get smaller than this. if abs(x) < abs(y): # Make x the larger one. x, y = y, x if x * y < 0: # Signs differ. x = x - 2.0 * y # Move x and y same distance. y = -y return (x - y)/float(x) <= delta If one of them is zero, then you get a ratio of one. So a delta of 1 would mean everything is close. -Ron From mistersheik at gmail.com Thu Jan 15 23:10:55 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 15 Jan 2015 17:10:55 -0500 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <m99dko$4hr$1@ger.gmane.org> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <20150114103919.GO10173@ando.pearwood.info> <m96meq$7j0$1@ger.gmane.org> <20150115000416.GP10173@ando.pearwood.info> <m977on$can$1@ger.gmane.org> <20150115072927.GA16607@ando.pearwood.info> <m98u5r$22t$1@ger.gmane.org> <CALGmxE+NA6AwjBaHot1qcrvPv6zh2v81FA_qbxVZUb9HaUJYkg@mail.gmail.com> <m99ci0$jhl$1@ger.gmane.org> <m99dko$4hr$1@ger.gmane.org> Message-ID: <CAA68w_mYH__g5n6c19eM=cb+b2BD+Jpbg9wGGfRFipgKuB=Bpw@mail.gmail.com> The problem with this is that if your "actual" is say, 5, then a large enough "estimate" will always be close. On Thu, Jan 15, 2015 at 5:06 PM, Ron Adam <ron3200 at gmail.com> wrote: > > > On 01/15/2015 03:47 PM, Ron Adam wrote: > >> How about this? >> >> >> def is_close(x, y, delta=.001): >> """ Check is x and y are close to each other. """ >> if x == y: >> return True # Can't get smaller than this. >> if x == 0 or y == 0: >> return False # Nothing is close to zero. >> if abs(x) < abs(y): # Make x the larger one. >> x, y = y, x >> if x * y < 0: # Signs differ. >> x = x - 2.0 * y # Move x and y same distance. >> y = -y >> return (x - y)/float(x) <= delta >> > > > Remove the check of one being zero, it isn't needed. > > def is_close(x, y, delta=.001): > """ Check is x and y are close to each other. """ > if x == y: > return True # Can't get smaller than this. > if abs(x) < abs(y): # Make x the larger one. > x, y = y, x > if x * y < 0: # Signs differ. > x = x - 2.0 * y # Move x and y same distance. > y = -y > return (x - y)/float(x) <= delta > > > If one of them is zero, then you get a ratio of one. So a delta of 1 would > mean everything is close. > > -Ron > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/python-ideas/zpfXmlTOp2I/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150115/051532af/attachment.html> From chris.barker at noaa.gov Thu Jan 15 23:50:14 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 15 Jan 2015 14:50:14 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CAA68w_kx1kyVnoPajUGGsJgCv=cQRwQ56RdFuBLZW15C676-Qw@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <3b503462-8868-4c02-9988-bd53c66fe271@googlegroups.com> <CALGmxEJet44RNdD6xD_bYPcUniwOUTX6j7pL79SjSSHEhhRZ5Q@mail.gmail.com> <CAA68w_kx1kyVnoPajUGGsJgCv=cQRwQ56RdFuBLZW15C676-Qw@mail.gmail.com> Message-ID: <CALGmxEJSZMUyqicL2ut9RhB2w5pfraw-L=ubGDX1ttk7ctm8Ww@mail.gmail.com> On Thu, Jan 15, 2015 at 1:52 PM, Neil Girdhar <mistersheik at gmail.com> wrote: > The point is that this function is already in Python > I don't think somethign being in an external package means that we have to do it the same way in teh stdlib -- even a widely used and well regarded package like numpy. And I say this as someone that has "import numpy" in maybe 90% of my python files. Maybe we should be careful to give it a very distinct name, however, to avoid confusion. > and if you want to do something different, you should have a really good > reason to do it differently. > I'm not sure I agree, but we do in this case anyway. The truth is, while really smart people wrote numpy, many of the algorithms in there did not go through nearly the level of review currently required for the python standard library > If you were to add a function to math, say math.close, it should work > like numpy.allclose in my opinion. > > For reference, numpy does this: > > absolute(*a* - *b*) <= (*atol* + *rtol* * absolute(*b*)) > > where atol is an absolute tolerance and rtol is a relative tolerance > (relative to the actual value b). This subsumes most of the proposals here. > adding atol in there "takes care of" the near zero and straddleing zero issue ( I suspect that's why it's done that way), but it is fatally wrong for values much less than 1.0 -- the atol totally overwhelms the rtol. See my post earlier today. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150115/45298e54/attachment.html> From mistersheik at gmail.com Fri Jan 16 00:31:31 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 15 Jan 2015 18:31:31 -0500 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CALGmxEJSZMUyqicL2ut9RhB2w5pfraw-L=ubGDX1ttk7ctm8Ww@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <3b503462-8868-4c02-9988-bd53c66fe271@googlegroups.com> <CALGmxEJet44RNdD6xD_bYPcUniwOUTX6j7pL79SjSSHEhhRZ5Q@mail.gmail.com> <CAA68w_kx1kyVnoPajUGGsJgCv=cQRwQ56RdFuBLZW15C676-Qw@mail.gmail.com> <CALGmxEJSZMUyqicL2ut9RhB2w5pfraw-L=ubGDX1ttk7ctm8Ww@mail.gmail.com> Message-ID: <CAA68w_=vPaL8y7aS3szELshU_m6Mq1Jg777-gKihj-XrC5+0vQ@mail.gmail.com> You can always disable atol by setting atol to zero. I really don't see what's wrong with their implementation. On Thu, Jan 15, 2015 at 5:50 PM, Chris Barker <chris.barker at noaa.gov> wrote: > On Thu, Jan 15, 2015 at 1:52 PM, Neil Girdhar <mistersheik at gmail.com> > wrote: > >> The point is that this function is already in Python >> > > I don't think somethign being in an external package means that we have > to do it the same way in teh stdlib -- even a widely used and well regarded > package like numpy. And I say this as someone that has "import numpy" in > maybe 90% of my python files. > > Maybe we should be careful to give it a very distinct name, however, to > avoid confusion. > > >> and if you want to do something different, you should have a really good >> reason to do it differently. >> > > I'm not sure I agree, but we do in this case anyway. The truth is, while > really smart people wrote numpy, many of the algorithms in there did not go > through nearly the level of review currently required for the python > standard library > > >> If you were to add a function to math, say math.close, it should work >> like numpy.allclose in my opinion. >> >> For reference, numpy does this: >> >> absolute(*a* - *b*) <= (*atol* + *rtol* * absolute(*b*)) >> >> where atol is an absolute tolerance and rtol is a relative tolerance >> (relative to the actual value b). This subsumes most of the proposals here. >> > > adding atol in there "takes care of" the near zero and straddleing zero > issue ( I suspect that's why it's done that way), but it is fatally wrong > for values much less than 1.0 -- the atol totally overwhelms the rtol. > > See my post earlier today. > > -Chris > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150115/4b22051f/attachment-0001.html> From chris.barker at noaa.gov Fri Jan 16 00:36:11 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 15 Jan 2015 15:36:11 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CAA68w_=vPaL8y7aS3szELshU_m6Mq1Jg777-gKihj-XrC5+0vQ@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <3b503462-8868-4c02-9988-bd53c66fe271@googlegroups.com> <CALGmxEJet44RNdD6xD_bYPcUniwOUTX6j7pL79SjSSHEhhRZ5Q@mail.gmail.com> <CAA68w_kx1kyVnoPajUGGsJgCv=cQRwQ56RdFuBLZW15C676-Qw@mail.gmail.com> <CALGmxEJSZMUyqicL2ut9RhB2w5pfraw-L=ubGDX1ttk7ctm8Ww@mail.gmail.com> <CAA68w_=vPaL8y7aS3szELshU_m6Mq1Jg777-gKihj-XrC5+0vQ@mail.gmail.com> Message-ID: <CALGmxE+7-2Qx2FoA_xPXAW5b612-Q3o7LyQPDUSL8LT-MHgsFg@mail.gmail.com> On Thu, Jan 15, 2015 at 3:31 PM, Neil Girdhar <mistersheik at gmail.com> wrote: > You can always disable atol by setting atol to zero. I really don't see > what's wrong with their implementation. > 1) the default should be zero in that case having a default close to the rtol default is asking for trouble. 2) if the user has both large and small numbers, there IS no appropriate value for a_tol for all of them. They simply should not be mixed in that way. -Chris > > On Thu, Jan 15, 2015 at 5:50 PM, Chris Barker <chris.barker at noaa.gov> > wrote: > >> On Thu, Jan 15, 2015 at 1:52 PM, Neil Girdhar <mistersheik at gmail.com> >> wrote: >> >>> The point is that this function is already in Python >>> >> >> I don't think somethign being in an external package means that we have >> to do it the same way in teh stdlib -- even a widely used and well regarded >> package like numpy. And I say this as someone that has "import numpy" in >> maybe 90% of my python files. >> >> Maybe we should be careful to give it a very distinct name, however, to >> avoid confusion. >> >> >>> and if you want to do something different, you should have a really good >>> reason to do it differently. >>> >> >> I'm not sure I agree, but we do in this case anyway. The truth is, while >> really smart people wrote numpy, many of the algorithms in there did not go >> through nearly the level of review currently required for the python >> standard library >> >> >>> If you were to add a function to math, say math.close, it should work >>> like numpy.allclose in my opinion. >>> >>> For reference, numpy does this: >>> >>> absolute(*a* - *b*) <= (*atol* + *rtol* * absolute(*b*)) >>> >>> where atol is an absolute tolerance and rtol is a relative tolerance >>> (relative to the actual value b). This subsumes most of the proposals here. >>> >> >> adding atol in there "takes care of" the near zero and straddleing zero >> issue ( I suspect that's why it's done that way), but it is fatally wrong >> for values much less than 1.0 -- the atol totally overwhelms the rtol. >> >> See my post earlier today. >> >> -Chris >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.Barker at noaa.gov >> > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150115/f0a75c14/attachment.html> From ron3200 at gmail.com Fri Jan 16 00:45:43 2015 From: ron3200 at gmail.com (Ron Adam) Date: Thu, 15 Jan 2015 17:45:43 -0600 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CAA68w_mYH__g5n6c19eM=cb+b2BD+Jpbg9wGGfRFipgKuB=Bpw@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <20150114103919.GO10173@ando.pearwood.info> <m96meq$7j0$1@ger.gmane.org> <20150115000416.GP10173@ando.pearwood.info> <m977on$can$1@ger.gmane.org> <20150115072927.GA16607@ando.pearwood.info> <m98u5r$22t$1@ger.gmane.org> <CALGmxE+NA6AwjBaHot1qcrvPv6zh2v81FA_qbxVZUb9HaUJYkg@mail.gmail.com> <m99ci0$jhl$1@ger.gmane.org> <m99dko$4hr$1@ger.gmane.org> <CAA68w_mYH__g5n6c19eM=cb+b2BD+Jpbg9wGGfRFipgKuB=Bpw@mail.gmail.com> Message-ID: <m99jf7$tge$1@ger.gmane.org> On 01/15/2015 04:10 PM, Neil Girdhar wrote: > The problem with this is that if your "actual" is say, 5, then a large > enough "estimate" will always be close. def is_close(x, y, delta=.001): """ Check is x and y are close to each other. """ if x == y: return True # Can't get smaller than this. if abs(x) < abs(y): # Make x the larger one. x, y = y, x if x * y < 0: # Signs differ. x = x - 2.0 * y # Move x and y same distance. y = -y return (x - y)/float(x) <= delta Do you mean the y value? Some tests. is_close(0, 0, delta=0.001) --> True is_close(0, 1, delta=0.001) --> False is_close(1, 0, delta=0.001) --> False is_close(1000, 999, delta=0.001) --> True is_close(999, 1000, delta=0.001) --> True is_close(100, 95, delta=0.001) --> False is_close(95, 100, delta=0.001) --> False is_close(inf, 1, delta=0.001) --> False is_close(5, 3, delta=0.001) --> False is_close(5, 4.999, delta=0.001) --> True is_close(0, 0, delta=0.2) --> True is_close(0, 1, delta=0.2) --> False is_close(1, 0, delta=0.2) --> False is_close(1000, 999, delta=0.2) --> True is_close(999, 1000, delta=0.2) --> True is_close(100, 95, delta=0.2) --> True is_close(95, 100, delta=0.2) --> True is_close(inf, 1, delta=0.2) --> False is_close(5, 3, delta=0.2) --> False is_close(5, 4.999, delta=0.2) --> True Or do you mean this? is_close(0, 0, delta=5) --> True is_close(0, 1, delta=5) --> True is_close(1, 0, delta=5) --> True is_close(1000, 999, delta=5) --> True is_close(999, 1000, delta=5) --> True is_close(100, 95, delta=5) --> TrueBut it's probably the best you can do without adding a reference length. is_close(95, 100, delta=5) --> True is_close(inf, 1, delta=5) --> False is_close(5, 3, delta=5) --> True is_close(5, 4.999, delta=5) --> True A 1 is 100 percent the distance of the larger to 0. So every thing except infinity is smaller. A 5 is 500 percent, which is harmless as it's the same as 100 percent. The smaller of the two will be located from the larger amount and zero. It doesn't check for a negative delta, that should probably be an error. Cheers, Ron From mistersheik at gmail.com Fri Jan 16 00:47:09 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 15 Jan 2015 18:47:09 -0500 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CALGmxE+7-2Qx2FoA_xPXAW5b612-Q3o7LyQPDUSL8LT-MHgsFg@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <3b503462-8868-4c02-9988-bd53c66fe271@googlegroups.com> <CALGmxEJet44RNdD6xD_bYPcUniwOUTX6j7pL79SjSSHEhhRZ5Q@mail.gmail.com> <CAA68w_kx1kyVnoPajUGGsJgCv=cQRwQ56RdFuBLZW15C676-Qw@mail.gmail.com> <CALGmxEJSZMUyqicL2ut9RhB2w5pfraw-L=ubGDX1ttk7ctm8Ww@mail.gmail.com> <CAA68w_=vPaL8y7aS3szELshU_m6Mq1Jg777-gKihj-XrC5+0vQ@mail.gmail.com> <CALGmxE+7-2Qx2FoA_xPXAW5b612-Q3o7LyQPDUSL8LT-MHgsFg@mail.gmail.com> Message-ID: <CAA68w_kSiz2HoYJNuD2NXAvniKyQ1oFOFPgHoiqzajPtM33XUw@mail.gmail.com> On Thu, Jan 15, 2015 at 6:36 PM, Chris Barker <chris.barker at noaa.gov> wrote: > On Thu, Jan 15, 2015 at 3:31 PM, Neil Girdhar <mistersheik at gmail.com> > wrote: > >> You can always disable atol by setting atol to zero. I really don't see >> what's wrong with their implementation. >> > > 1) the default should be zero in that case having a default close to the > rtol default is asking for trouble. > It's not that close: rtol defaults to 1000 times bigger than atol. > > 2) if the user has both large and small numbers, there IS no appropriate > value for a_tol for all of them. > The relative error is not symmetric, so it's not about having "large and small numbers". Your estimate should not affect the error you tolerate. > > They simply should not be mixed in that way. > > -Chris > > > > > >> >> On Thu, Jan 15, 2015 at 5:50 PM, Chris Barker <chris.barker at noaa.gov> >> wrote: >> >>> On Thu, Jan 15, 2015 at 1:52 PM, Neil Girdhar <mistersheik at gmail.com> >>> wrote: >>> >>>> The point is that this function is already in Python >>>> >>> >>> I don't think somethign being in an external package means that we have >>> to do it the same way in teh stdlib -- even a widely used and well regarded >>> package like numpy. And I say this as someone that has "import numpy" in >>> maybe 90% of my python files. >>> >>> Maybe we should be careful to give it a very distinct name, however, to >>> avoid confusion. >>> >>> >>>> and if you want to do something different, you should have a really >>>> good reason to do it differently. >>>> >>> >>> I'm not sure I agree, but we do in this case anyway. The truth is, while >>> really smart people wrote numpy, many of the algorithms in there did not go >>> through nearly the level of review currently required for the python >>> standard library >>> >>> >>>> If you were to add a function to math, say math.close, it should work >>>> like numpy.allclose in my opinion. >>>> >>>> For reference, numpy does this: >>>> >>>> absolute(*a* - *b*) <= (*atol* + *rtol* * absolute(*b*)) >>>> >>>> where atol is an absolute tolerance and rtol is a relative tolerance >>>> (relative to the actual value b). This subsumes most of the proposals here. >>>> >>> >>> adding atol in there "takes care of" the near zero and straddleing zero >>> issue ( I suspect that's why it's done that way), but it is fatally wrong >>> for values much less than 1.0 -- the atol totally overwhelms the rtol. >>> >>> See my post earlier today. >>> >>> -Chris >>> >>> >>> -- >>> >>> Christopher Barker, Ph.D. >>> Oceanographer >>> >>> Emergency Response Division >>> NOAA/NOS/OR&R (206) 526-6959 voice >>> 7600 Sand Point Way NE (206) 526-6329 fax >>> Seattle, WA 98115 (206) 526-6317 main reception >>> >>> Chris.Barker at noaa.gov >>> >> >> > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150115/934f2fe9/attachment-0001.html> From chris.barker at noaa.gov Fri Jan 16 00:47:25 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 15 Jan 2015 15:47:25 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CAA68w_=vPaL8y7aS3szELshU_m6Mq1Jg777-gKihj-XrC5+0vQ@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <3b503462-8868-4c02-9988-bd53c66fe271@googlegroups.com> <CALGmxEJet44RNdD6xD_bYPcUniwOUTX6j7pL79SjSSHEhhRZ5Q@mail.gmail.com> <CAA68w_kx1kyVnoPajUGGsJgCv=cQRwQ56RdFuBLZW15C676-Qw@mail.gmail.com> <CALGmxEJSZMUyqicL2ut9RhB2w5pfraw-L=ubGDX1ttk7ctm8Ww@mail.gmail.com> <CAA68w_=vPaL8y7aS3szELshU_m6Mq1Jg777-gKihj-XrC5+0vQ@mail.gmail.com> Message-ID: <6473609160848220685@unknownmsgid> On Jan 15, 2015, at 3:31 PM, Neil Girdhar <mistersheik at gmail.com> wrote: absolute(*a* - *b*) <= (*atol* + *rtol* * absolute(*b*)) Oh, and if the numbers are small, then adding the absolute tolerance changes the tolerance significantly -- so you don't get what you expect there, either. Chris >> where atol is an absolute tolerance and rtol is a relative tolerance >> (relative to the actual value b). This subsumes most of the proposals here. >> > > adding atol in there "takes care of" the near zero and straddleing zero > issue ( I suspect that's why it's done that way), but it is fatally wrong > for values much less than 1.0 -- the atol totally overwhelms the rtol. > > See my post earlier today. > > -Chris > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150115/916727fd/attachment.html> From mistersheik at gmail.com Fri Jan 16 00:58:04 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 15 Jan 2015 18:58:04 -0500 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <6473609160848220685@unknownmsgid> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <3b503462-8868-4c02-9988-bd53c66fe271@googlegroups.com> <CALGmxEJet44RNdD6xD_bYPcUniwOUTX6j7pL79SjSSHEhhRZ5Q@mail.gmail.com> <CAA68w_kx1kyVnoPajUGGsJgCv=cQRwQ56RdFuBLZW15C676-Qw@mail.gmail.com> <CALGmxEJSZMUyqicL2ut9RhB2w5pfraw-L=ubGDX1ttk7ctm8Ww@mail.gmail.com> <CAA68w_=vPaL8y7aS3szELshU_m6Mq1Jg777-gKihj-XrC5+0vQ@mail.gmail.com> <6473609160848220685@unknownmsgid> Message-ID: <CAA68w_kktJGaj=Z-9NGfOq7B_choKk6b-t5tkb4MrKUAkbXAtw@mail.gmail.com> You almost always want to use either an absolute tolerance or a relative tolerance. Here's why: Consider your estimated value a and your actual value b. If the estimate is taken to be the mean of a standard Gaussian distribution (the minimum assumptive distribution for a given mean and variance over the reals), then using an absolute tolerance is equivalent to verifying that the probability of observing b is within a interval with sufficient probability. Similarly, if the estimate is taken to be the mean of an standard exponential distribution (the minimum assumptive distribution for a given mean over the positive reals), then using a relative tolerance is equivalent to verifying the same thing. You almost always want one or the other. The symmetric error that people are proposing in this thread has no intuitive meaning to me. Best, Neil On Thu, Jan 15, 2015 at 6:47 PM, Chris Barker - NOAA Federal < chris.barker at noaa.gov> wrote: > > > On Jan 15, 2015, at 3:31 PM, Neil Girdhar <mistersheik at gmail.com> wrote: > absolute(*a* - *b*) <= (*atol* + *rtol* * absolute(*b*)) > Oh, and if the numbers are small, then adding the absolute tolerance > changes the tolerance significantly -- so you don't get what you expect > there, either. > > Chris > > > > >>> where atol is an absolute tolerance and rtol is a relative tolerance >>> (relative to the actual value b). This subsumes most of the proposals here. >>> >> >> adding atol in there "takes care of" the near zero and straddleing zero >> issue ( I suspect that's why it's done that way), but it is fatally wrong >> for values much less than 1.0 -- the atol totally overwhelms the rtol. >> >> See my post earlier today. >> >> -Chris >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.Barker at noaa.gov >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150115/17be945f/attachment.html> From chris.barker at noaa.gov Fri Jan 16 01:14:33 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 15 Jan 2015 16:14:33 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CAA68w_kSiz2HoYJNuD2NXAvniKyQ1oFOFPgHoiqzajPtM33XUw@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <3b503462-8868-4c02-9988-bd53c66fe271@googlegroups.com> <CALGmxEJet44RNdD6xD_bYPcUniwOUTX6j7pL79SjSSHEhhRZ5Q@mail.gmail.com> <CAA68w_kx1kyVnoPajUGGsJgCv=cQRwQ56RdFuBLZW15C676-Qw@mail.gmail.com> <CALGmxEJSZMUyqicL2ut9RhB2w5pfraw-L=ubGDX1ttk7ctm8Ww@mail.gmail.com> <CAA68w_=vPaL8y7aS3szELshU_m6Mq1Jg777-gKihj-XrC5+0vQ@mail.gmail.com> <CALGmxE+7-2Qx2FoA_xPXAW5b612-Q3o7LyQPDUSL8LT-MHgsFg@mail.gmail.com> <CAA68w_kSiz2HoYJNuD2NXAvniKyQ1oFOFPgHoiqzajPtM33XUw@mail.gmail.com> Message-ID: <7410666359556586725@unknownmsgid> On Jan 15, 2015, at 3:47 PM, Neil Girdhar <mistersheik at gmail.com> wrote: On Thu, Jan 15, 2015 at 6:36 PM, Chris Barker <chris.barker at noaa.gov> wrote: > On Thu, Jan 15, 2015 at 3:31 PM, Neil Girdhar <mistersheik at gmail.com> > wrote: > >> You can always disable atol by setting atol to zero. I really don't see >> what's wrong with their implementation. >> > > 1) the default should be zero in that case having a default close to the > rtol default is asking for trouble. > It's not that close: rtol defaults to 1000 times bigger than atol. That's pretty huge if your values are on order of 1e-100 ;-) Which is my point -- this approach only makes sense for values over order 1. Then the scaled relative error will be a lot larger, so atol sets a sort of lower bound on the error. But if the magnitude of your values is small, then the scaled value becomes small, and the atol overwhelms it. This kind of "switch to an absolute tolerance when close to zero" behavior is what I've been looking for, but I don't like how numpy does it. -Chris -CHB > 2) if the user has both large and small numbers, there IS no appropriate > value for a_tol for all of them. > The relative error is not symmetric, so it's not about having "large and small numbers". Your estimate should not affect the error you tolerate. > > They simply should not be mixed in that way. > > -Chris > > > > > >> >> On Thu, Jan 15, 2015 at 5:50 PM, Chris Barker <chris.barker at noaa.gov> >> wrote: >> >>> On Thu, Jan 15, 2015 at 1:52 PM, Neil Girdhar <mistersheik at gmail.com> >>> wrote: >>> >>>> The point is that this function is already in Python >>>> >>> >>> I don't think somethign being in an external package means that we have >>> to do it the same way in teh stdlib -- even a widely used and well regarded >>> package like numpy. And I say this as someone that has "import numpy" in >>> maybe 90% of my python files. >>> >>> Maybe we should be careful to give it a very distinct name, however, to >>> avoid confusion. >>> >>> >>>> and if you want to do something different, you should have a really >>>> good reason to do it differently. >>>> >>> >>> I'm not sure I agree, but we do in this case anyway. The truth is, while >>> really smart people wrote numpy, many of the algorithms in there did not go >>> through nearly the level of review currently required for the python >>> standard library >>> >>> >>>> If you were to add a function to math, say math.close, it should work >>>> like numpy.allclose in my opinion. >>>> >>>> For reference, numpy does this: >>>> >>>> absolute(*a* - *b*) <= (*atol* + *rtol* * absolute(*b*)) >>>> >>>> where atol is an absolute tolerance and rtol is a relative tolerance >>>> (relative to the actual value b). This subsumes most of the proposals here. >>>> >>> >>> adding atol in there "takes care of" the near zero and straddleing zero >>> issue ( I suspect that's why it's done that way), but it is fatally wrong >>> for values much less than 1.0 -- the atol totally overwhelms the rtol. >>> >>> See my post earlier today. >>> >>> -Chris >>> >>> >>> -- >>> >>> Christopher Barker, Ph.D. >>> Oceanographer >>> >>> Emergency Response Division >>> NOAA/NOS/OR&R (206) 526-6959 voice >>> 7600 Sand Point Way NE (206) 526-6329 fax >>> Seattle, WA 98115 (206) 526-6317 main reception >>> >>> Chris.Barker at noaa.gov >>> >> >> > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150115/14dc45b7/attachment-0001.html> From ron3200 at gmail.com Fri Jan 16 01:21:55 2015 From: ron3200 at gmail.com (Ron Adam) Date: Thu, 15 Jan 2015 18:21:55 -0600 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CAA68w_mYH__g5n6c19eM=cb+b2BD+Jpbg9wGGfRFipgKuB=Bpw@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <20150114103919.GO10173@ando.pearwood.info> <m96meq$7j0$1@ger.gmane.org> <20150115000416.GP10173@ando.pearwood.info> <m977on$can$1@ger.gmane.org> <20150115072927.GA16607@ando.pearwood.info> <m98u5r$22t$1@ger.gmane.org> <CALGmxE+NA6AwjBaHot1qcrvPv6zh2v81FA_qbxVZUb9HaUJYkg@mail.gmail.com> <m99ci0$jhl$1@ger.gmane.org> <m99dko$4hr$1@ger.gmane.org> <CAA68w_mYH__g5n6c19eM=cb+b2BD+Jpbg9wGGfRFipgKuB=Bpw@mail.gmail.com> Message-ID: <m99lj4$s63$1@ger.gmane.org> On 01/15/2015 04:10 PM, Neil Girdhar wrote: > The problem with this is that if your "actual" is say, 5, then a large > enough "estimate" will always be close. Actually the problem is with numbers of different signs. The idea of normalising them in some way only works partially, and it doesn't fit mathematically. It seems to me if the signs are different, then relatively speaking, they are always far apart. Ron From chris.barker at noaa.gov Fri Jan 16 01:23:14 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 15 Jan 2015 16:23:14 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CAA68w_kktJGaj=Z-9NGfOq7B_choKk6b-t5tkb4MrKUAkbXAtw@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <3b503462-8868-4c02-9988-bd53c66fe271@googlegroups.com> <CALGmxEJet44RNdD6xD_bYPcUniwOUTX6j7pL79SjSSHEhhRZ5Q@mail.gmail.com> <CAA68w_kx1kyVnoPajUGGsJgCv=cQRwQ56RdFuBLZW15C676-Qw@mail.gmail.com> <CALGmxEJSZMUyqicL2ut9RhB2w5pfraw-L=ubGDX1ttk7ctm8Ww@mail.gmail.com> <CAA68w_=vPaL8y7aS3szELshU_m6Mq1Jg777-gKihj-XrC5+0vQ@mail.gmail.com> <6473609160848220685@unknownmsgid> <CAA68w_kktJGaj=Z-9NGfOq7B_choKk6b-t5tkb4MrKUAkbXAtw@mail.gmail.com> Message-ID: <2005763677511755342@unknownmsgid> On Jan 15, 2015, at 3:58 PM, Neil Girdhar <mistersheik at gmail.com> wrote: You almost always want to use either an absolute tolerance or a relative tolerance. Exactly -- only the user knows what is wanted. The trick is that relative tolerance is the most-of the time choice for float compares, but it's not appropriate (or even possible) for zero. So here we are grappling for a way to have sensible behavior at zero with otherwise relative tolerance. It's probably not possible to do so in the general case. So we should probably simply require the user to specify either and absolute or relative tolerance and be done with it. Nothing will be relatively close to zero except zero, which is fine, but should be mentioned in the docs. It sounds like this is what Steven's code does. I just haven't figured out how to look at it on a phone. -Chris Here's why: Consider your estimated value a and your actual value b. If the estimate is taken to be the mean of a standard Gaussian distribution (the minimum assumptive distribution for a given mean and variance over the reals), then using an absolute tolerance is equivalent to verifying that the probability of observing b is within a interval with sufficient probability. Similarly, if the estimate is taken to be the mean of an standard exponential distribution (the minimum assumptive distribution for a given mean over the positive reals), then using a relative tolerance is equivalent to verifying the same thing. You almost always want one or the other. The symmetric error that people are proposing in this thread has no intuitive meaning to me. Best, Neil On Thu, Jan 15, 2015 at 6:47 PM, Chris Barker - NOAA Federal < chris.barker at noaa.gov> wrote: > > > On Jan 15, 2015, at 3:31 PM, Neil Girdhar <mistersheik at gmail.com> wrote: > absolute(*a* - *b*) <= (*atol* + *rtol* * absolute(*b*)) > Oh, and if the numbers are small, then adding the absolute tolerance > changes the tolerance significantly -- so you don't get what you expect > there, either. > > Chris > > > > >>> where atol is an absolute tolerance and rtol is a relative tolerance >>> (relative to the actual value b). This subsumes most of the proposals here. >>> >> >> adding atol in there "takes care of" the near zero and straddleing zero >> issue ( I suspect that's why it's done that way), but it is fatally wrong >> for values much less than 1.0 -- the atol totally overwhelms the rtol. >> >> See my post earlier today. >> >> -Chris >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.Barker at noaa.gov >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150115/b361a021/attachment.html> From chris.barker at noaa.gov Fri Jan 16 01:25:27 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 15 Jan 2015 16:25:27 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <m99lj4$s63$1@ger.gmane.org> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <20150113013452.GE10173@ando.pearwood.info> <CAAu3qLW0-RSYzLghMuCgk=J4Tg65HSwgR=QSt6f+NMg80-MYtg@mail.gmail.com> <-8174570845814181397@unknownmsgid> <m94lv3$h9d$1@ger.gmane.org> <20150114103919.GO10173@ando.pearwood.info> <m96meq$7j0$1@ger.gmane.org> <20150115000416.GP10173@ando.pearwood.info> <m977on$can$1@ger.gmane.org> <20150115072927.GA16607@ando.pearwood.info> <m98u5r$22t$1@ger.gmane.org> <CALGmxE+NA6AwjBaHot1qcrvPv6zh2v81FA_qbxVZUb9HaUJYkg@mail.gmail.com> <m99ci0$jhl$1@ger.gmane.org> <m99dko$4hr$1@ger.gmane.org> <CAA68w_mYH__g5n6c19eM=cb+b2BD+Jpbg9wGGfRFipgKuB=Bpw@mail.gmail.com> <m99lj4$s63$1@ger.gmane.org> Message-ID: <-4541968182008671162@unknownmsgid> > On Jan 15, 2015, at 4:22 PM, Ron Adam <ron3200 at gmail.com> wrote > > Actually the problem is with numbers of different signs. The idea of normalising them in some way only works partially, and it doesn't fit mathematically. > > It seems to me if the signs are different, then relatively speaking, they are always far apart. Just like any number is relatively far from zero. -Chris > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From mistersheik at gmail.com Fri Jan 16 01:46:46 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 15 Jan 2015 19:46:46 -0500 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <2005763677511755342@unknownmsgid> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <3b503462-8868-4c02-9988-bd53c66fe271@googlegroups.com> <CALGmxEJet44RNdD6xD_bYPcUniwOUTX6j7pL79SjSSHEhhRZ5Q@mail.gmail.com> <CAA68w_kx1kyVnoPajUGGsJgCv=cQRwQ56RdFuBLZW15C676-Qw@mail.gmail.com> <CALGmxEJSZMUyqicL2ut9RhB2w5pfraw-L=ubGDX1ttk7ctm8Ww@mail.gmail.com> <CAA68w_=vPaL8y7aS3szELshU_m6Mq1Jg777-gKihj-XrC5+0vQ@mail.gmail.com> <6473609160848220685@unknownmsgid> <CAA68w_kktJGaj=Z-9NGfOq7B_choKk6b-t5tkb4MrKUAkbXAtw@mail.gmail.com> <2005763677511755342@unknownmsgid> Message-ID: <CAA68w_kRY7dEOyrMyUayfw4_xqzKUh-6pRhrjGrf9Gme_Nf8_w@mail.gmail.com> I'm not sure what Steven's code is, but his hypothesis: "Furthermore, for safety you normally want the larger estimate of error, not the smaller?" does not apply to the usual case where you have an actual value and an estimate. He seems to be describing a symmetric error function, which is not intuitive to me. Best, Neil On Thu, Jan 15, 2015 at 7:23 PM, Chris Barker - NOAA Federal < chris.barker at noaa.gov> wrote: > On Jan 15, 2015, at 3:58 PM, Neil Girdhar <mistersheik at gmail.com> wrote: > > You almost always want to use either an absolute tolerance or a relative > tolerance. > > > Exactly -- only the user knows what is wanted. The trick is that relative > tolerance is the most-of the time choice for float compares, but it's not > appropriate (or even possible) for zero. > > So here we are grappling for a way to have sensible behavior at zero with > otherwise relative tolerance. It's probably not possible to do so in the > general case. > > So we should probably simply require the user to specify either and > absolute or relative tolerance and be done with it. > > Nothing will be relatively close to zero except zero, which is fine, but > should be mentioned in the docs. > > It sounds like this is what Steven's code does. I just haven't figured out > how to look at it on a phone. > > -Chris > > Here's why: Consider your estimated value a and your actual value b. If > the estimate is taken to be the mean of a standard Gaussian distribution > (the minimum assumptive distribution for a given mean and variance over the > reals), then using an absolute tolerance is equivalent to verifying that > the probability of observing b is within a interval with sufficient > probability. Similarly, if the estimate is taken to be the mean of an > standard exponential distribution (the minimum assumptive distribution for > a given mean over the positive reals), then using a relative tolerance is > equivalent to verifying the same thing. > > You almost always want one or the other. The symmetric error that people > are proposing in this thread has no intuitive meaning to me. > > Best, > > Neil > > On Thu, Jan 15, 2015 at 6:47 PM, Chris Barker - NOAA Federal < > chris.barker at noaa.gov> wrote: > >> >> >> On Jan 15, 2015, at 3:31 PM, Neil Girdhar <mistersheik at gmail.com> wrote: >> absolute(*a* - *b*) <= (*atol* + *rtol* * absolute(*b*)) >> Oh, and if the numbers are small, then adding the absolute tolerance >> changes the tolerance significantly -- so you don't get what you expect >> there, either. >> >> Chris >> >> >> >> >>>> where atol is an absolute tolerance and rtol is a relative tolerance >>>> (relative to the actual value b). This subsumes most of the proposals here. >>>> >>> >>> adding atol in there "takes care of" the near zero and straddleing zero >>> issue ( I suspect that's why it's done that way), but it is fatally wrong >>> for values much less than 1.0 -- the atol totally overwhelms the rtol. >>> >>> See my post earlier today. >>> >>> -Chris >>> >>> >>> -- >>> >>> Christopher Barker, Ph.D. >>> Oceanographer >>> >>> Emergency Response Division >>> NOAA/NOS/OR&R (206) 526-6959 voice >>> 7600 Sand Point Way NE (206) 526-6329 fax >>> Seattle, WA 98115 (206) 526-6317 main reception >>> >>> Chris.Barker at noaa.gov >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150115/9121354e/attachment-0001.html> From stephen at xemacs.org Fri Jan 16 04:28:43 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 16 Jan 2015 12:28:43 +0900 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CAA68w_kktJGaj=Z-9NGfOq7B_choKk6b-t5tkb4MrKUAkbXAtw@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <3b503462-8868-4c02-9988-bd53c66fe271@googlegroups.com> <CALGmxEJet44RNdD6xD_bYPcUniwOUTX6j7pL79SjSSHEhhRZ5Q@mail.gmail.com> <CAA68w_kx1kyVnoPajUGGsJgCv=cQRwQ56RdFuBLZW15C676-Qw@mail.gmail.com> <CALGmxEJSZMUyqicL2ut9RhB2w5pfraw-L=ubGDX1ttk7ctm8Ww@mail.gmail.com> <CAA68w_=vPaL8y7aS3szELshU_m6Mq1Jg777-gKihj-XrC5+0vQ@mail.gmail.com> <6473609160848220685@unknownmsgid> <CAA68w_kktJGaj=Z-9NGfOq7B_choKk6b-t5tkb4MrKUAkbXAtw@mail.gmail.com> Message-ID: <871tmvxv2s.fsf@uwakimon.sk.tsukuba.ac.jp> Neil Girdhar writes: > The symmetric error that people are proposing in this thread has no > intuitive meaning to me. There are many applications where the goal is to match two values, neither of which is the obvious standard (eg, statistical tests comparing populations, or even electrical circuits, where it may be important that two components be matched to within 1%, although the absolute value might be allowed to vary by up to 10%). Symmetric error is appropriate for those applications. Symmetric error may be less appropriate for applications where you want to hit an absolute value, but it's (provably) not too bad. By "provably not too bad" I mean that if you take the word "close" as a qualitative predicate, then although you can make the "distance" explode by taking the "actual" to be an order of magnitude distant in absolute units, you'll still judge it "not close" (just more so, but "more so" is meaningless in this qualitative context). On the other hand, for values that *are* close (with reasonable tolerances) it doesn't much matter which value you choose as the standard: "most" of the time you will get the "right" answer (and as the tolerance gets tighter, "most" tends to a limit of 100%). The generic "are_close()" function should be symmetric. I suppose it might also to useful to have an "is_close_to()" function that is asymmetric. From mistersheik at gmail.com Fri Jan 16 04:46:05 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 15 Jan 2015 22:46:05 -0500 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <871tmvxv2s.fsf@uwakimon.sk.tsukuba.ac.jp> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <3b503462-8868-4c02-9988-bd53c66fe271@googlegroups.com> <CALGmxEJet44RNdD6xD_bYPcUniwOUTX6j7pL79SjSSHEhhRZ5Q@mail.gmail.com> <CAA68w_kx1kyVnoPajUGGsJgCv=cQRwQ56RdFuBLZW15C676-Qw@mail.gmail.com> <CALGmxEJSZMUyqicL2ut9RhB2w5pfraw-L=ubGDX1ttk7ctm8Ww@mail.gmail.com> <CAA68w_=vPaL8y7aS3szELshU_m6Mq1Jg777-gKihj-XrC5+0vQ@mail.gmail.com> <6473609160848220685@unknownmsgid> <CAA68w_kktJGaj=Z-9NGfOq7B_choKk6b-t5tkb4MrKUAkbXAtw@mail.gmail.com> <871tmvxv2s.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <CAA68w_nRoS1Kk-Kthfy+X1gEpPF2rLA7Ck_-V0tJTy_W5i2JgQ@mail.gmail.com> On Thu, Jan 15, 2015 at 10:28 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote: > Neil Girdhar writes: > > > The symmetric error that people are proposing in this thread has no > > intuitive meaning to me. > > There are many applications where the goal is to match two values, > neither of which is the obvious standard (eg, statistical tests > comparing populations, No, if you're trying to answer the question whether two things belong to the same population as opposed to another, you should infer the population statistics based on a and b and a your estimated overall population statistics and then calculate cross entropies. Using some symmetric cross relative error has no meaning. > or even electrical circuits, where it may be > important that two components be matched to within 1%, although the > absolute value might be allowed to vary by up to 10%). Symmetric > error is appropriate for those applications. Symmetric error may be > less appropriate for applications where you want to hit an absolute > value, but it's (provably) not too bad. > > By "provably not too bad" I mean that if you take the word "close" as > a qualitative predicate, then although you can make the "distance" > explode by taking the "actual" to be an order of magnitude distant in > absolute units, you'll still judge it "not close" (just more so, but > "more so" is meaningless in this qualitative context). On the other > hand, for values that *are* close (with reasonable tolerances) it > doesn't much matter which value you choose as the standard: "most" of > the time you will get the "right" answer (and as the tolerance gets > tighter, "most" tends to a limit of 100%). > In statistics and machine learning at least many people have argued that the cross entropy error is the most reasonable loss function. When you have an observed value and estimated value, the right way of comparing them is a cross entropy error, and that's what absolute error and relative error are doing. They correspond to cross entropies of the minimum assumptive distributions over the reals and positive reals. I think the numpy.allclose function almost always gives you what you want when you have an actual and an estimated value, which is the more usual case. > > The generic "are_close()" function should be symmetric. I suppose it > might also to useful to have an "is_close_to()" function that is > asymmetric. > I disagree. Since the usual case is to have an observed and estimated value, then the close function should not be symmetric. Either you should have two functions: relative error and absolute error, or you should combine them like numpy did. Best, Neil -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150115/57321ba6/attachment.html> From mistersheik at gmail.com Fri Jan 16 16:13:53 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Fri, 16 Jan 2015 10:13:53 -0500 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CAA68w_nRoS1Kk-Kthfy+X1gEpPF2rLA7Ck_-V0tJTy_W5i2JgQ@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <3b503462-8868-4c02-9988-bd53c66fe271@googlegroups.com> <CALGmxEJet44RNdD6xD_bYPcUniwOUTX6j7pL79SjSSHEhhRZ5Q@mail.gmail.com> <CAA68w_kx1kyVnoPajUGGsJgCv=cQRwQ56RdFuBLZW15C676-Qw@mail.gmail.com> <CALGmxEJSZMUyqicL2ut9RhB2w5pfraw-L=ubGDX1ttk7ctm8Ww@mail.gmail.com> <CAA68w_=vPaL8y7aS3szELshU_m6Mq1Jg777-gKihj-XrC5+0vQ@mail.gmail.com> <6473609160848220685@unknownmsgid> <CAA68w_kktJGaj=Z-9NGfOq7B_choKk6b-t5tkb4MrKUAkbXAtw@mail.gmail.com> <871tmvxv2s.fsf@uwakimon.sk.tsukuba.ac.jp> <CAA68w_nRoS1Kk-Kthfy+X1gEpPF2rLA7Ck_-V0tJTy_W5i2JgQ@mail.gmail.com> Message-ID: <CAA68w_k4wUPs8A2HgH5M0Jz-4M6L90KY4rpo_VDxGbV7bA5BZA@mail.gmail.com> Actually, I was wrong about the exponential distribution's KL divergence. It's the relative error (b-a)/b plus another term: log(b/a) ? so I guess I don't see what relative error means except as a heuristic. Anyway, even if your symmetric error makes sense to you, does anyone already use it? If it were up to me, relative error would be b-a/b + log(b/a), but since no one uses that, I think it's a bad idea. On Thu, Jan 15, 2015 at 10:46 PM, Neil Girdhar <mistersheik at gmail.com> wrote: > > > On Thu, Jan 15, 2015 at 10:28 PM, Stephen J. Turnbull <stephen at xemacs.org> > wrote: > >> Neil Girdhar writes: >> >> > The symmetric error that people are proposing in this thread has no >> > intuitive meaning to me. >> >> There are many applications where the goal is to match two values, >> neither of which is the obvious standard (eg, statistical tests >> comparing populations, > > > No, if you're trying to answer the question whether two things belong to > the same population as opposed to another, you should infer the population > statistics based on a and b and a your estimated overall population > statistics and then calculate cross entropies. Using some symmetric cross > relative error has no meaning. > > >> or even electrical circuits, where it may be >> important that two components be matched to within 1%, although the >> absolute value might be allowed to vary by up to 10%). Symmetric >> error is appropriate for those applications. Symmetric error may be >> less appropriate for applications where you want to hit an absolute >> value, but it's (provably) not too bad. >> >> By "provably not too bad" I mean that if you take the word "close" as >> a qualitative predicate, then although you can make the "distance" >> explode by taking the "actual" to be an order of magnitude distant in >> absolute units, you'll still judge it "not close" (just more so, but >> "more so" is meaningless in this qualitative context). On the other >> hand, for values that *are* close (with reasonable tolerances) it >> doesn't much matter which value you choose as the standard: "most" of >> the time you will get the "right" answer (and as the tolerance gets >> tighter, "most" tends to a limit of 100%). >> > > In statistics and machine learning at least many people have argued that > the cross entropy error is the most reasonable loss function. When you > have an observed value and estimated value, the right way of comparing them > is a cross entropy error, and that's what absolute error and relative error > are doing. They correspond to cross entropies of the minimum assumptive > distributions over the reals and positive reals. > > I think the numpy.allclose function almost always gives you what you want > when you have an actual and an estimated value, which is the more usual > case. > > >> >> The generic "are_close()" function should be symmetric. I suppose it >> might also to useful to have an "is_close_to()" function that is >> asymmetric. >> > > I disagree. Since the usual case is to have an observed and estimated > value, then the close function should not be symmetric. Either you should > have two functions: relative error and absolute error, or you should > combine them like numpy did. > > Best, > > Neil > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150116/79b2bd57/attachment.html> From guido at python.org Fri Jan 16 18:17:36 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Jan 2015 09:17:36 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round Message-ID: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> After a long editing process we've got PEP 484 <https://www.python.org/dev/peps/pep-0484/> (Type Hints) ready for your review. This is by no means final, and several areas are either open (to be resolved in a later draft) or postponed (to a different PEP altogether). But there's enough meat that I think we can start having the discussion. Please also see PEP 483 <https://www.python.org/dev/peps/pep-0483/> (The Theory of Type Hint; copied and reformatted from the original Quip document that I posted just before last Christmas) and PEP 482 <https://www.python.org/dev/peps/pep-0482/> (Literature Overview for Type Hints, by ?ukasz). Those are informational PEPs though; the actual spec is focused in PEP 484 (the only one on the Standards Track). As I said earlier, I hope to have a rough consensus before PyCon <https://us.pycon.org/2015/> and working code (just the typing.py module <https://github.com/ambv/typehinting/tree/master/prototyping>, really) committed to CPython before the last 3.5 alpha <https://www.python.org/dev/peps/pep-0478/>. Here is the raw text of PEP 484. Fire away!! PEP: 484 Title: Type Hints Version: $Revision$ Last-Modified: $Date$ Author: Guido van Rossum <guido at python.org>, Jukka Lehtosalo < jukka.lehtosalo at iki.fi>, ?ukasz Langa <lukasz at langa.pl> Discussions-To: Python-Dev <python-dev at python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 29-Sep-2014 Post-History: 16-Jan-2015 Resolution: Abstract ======== This PEP introduces a standard syntax for type hints using annotations on function definitions. The proposal is strongly inspired by mypy [mypy]_. The theory behind type hints and gradual typing is explained in PEP 483. Rationale and Goals =================== PEP 3107 added support for arbitrary annotations on parts of a function definition. Although no meaning was assigned to annotations then, there has always been an implicit goal to use them for type hinting, which is listed as the first possible use case in said PEP. This PEP aims to provide a standard syntax for type annotations, opening up Python code to easier static analysis and refactoring, potential runtime type checking, and performance optimizations utilizing type information. Type Definition Syntax ====================== The syntax leverages PEP 3107-style annotations with a number of extensions described in sections below. In its basic form, type hinting is used by filling function annotations with classes:: def greeting(name: str) -> str: return 'Hello ' + name This denotes that the expected type of the ``name`` argument is ``str``. Analogically, the expected return type is ``str``. Subclasses of a specified argument type are also accepted as valid types for that argument. Abstract base classes, types available in the ``types`` module, and user-defined classes may be used as type hints as well. Annotations must be valid expressions that evaluate without raising exceptions at the time the function is defined. In addition, the needs of static analysis require that annotations must be simple enough to be interpreted by static analysis tools. (This is an intentionally somewhat vague requirement.) .. FIXME: Define rigorously what is/isn't supported. When used as an annotation, the expression ``None`` is considered equivalent to ``NoneType`` (i.e., ``type(None)`` for type hinting purposes. Type aliases are also valid type hints:: integer = int def retry(url: str, retry_count: integer): ... New names that are added to support features described in following sections are available in the ``typing`` package. Callbacks --------- Frameworks expecting callback functions of specific signatures might be type hinted using ``Callable[[Arg1Type, Arg2Type], ReturnType]``. Examples:: from typing import Any, AnyArgs, Callable def feeder(get_next_item: Callable[[], Item]): ... def async_query(on_success: Callable[[int], None], on_error: Callable[[int, Exception], None]): ... def partial(func: Callable[AnyArgs, Any], *args): ... Since using callbacks with keyword arguments is not perceived as a common use case, there is currently no support for specifying keyword arguments with ``Callable``. Generics -------- Since type information about objects kept in containers cannot be statically inferred in a generic way, abstract base classes have been extended to support subscription to denote expected types for container elements. Example:: from typing import Mapping, Set def notify_by_email(employees: Set[Employee], overrides: Mapping[str, str]): ... Generics can be parametrized by using a new factory available in ``typing`` called ``TypeVar``. Example:: from typing import Sequence, TypeVar T = TypeVar('T') # Declare type variable def first(l: Sequence[T]) -> T: # Generic function return l[0] In this case the contract is that the returning value is consistent with the elements held by the collection. ``TypeVar`` supports constraining parametric types to classes with any of the specified bases. Example:: from typing import Iterable X = TypeVar('X') Y = TypeVar('Y', Iterable[X]) def filter(rule: Callable[[X], bool], input: Y) -> Y: ... .. FIXME: Add an example with multiple bases defined. In the example above we specify that ``Y`` can be any subclass of Iterable with elements of type ``X``, as long as the return type of ``filter()`` will be the same as the type of the ``input`` argument. .. FIXME: Explain more about how this works. Forward references ------------------ When a type hint contains names that have not been defined yet, that definition may be expressed as a string, to be resolved later. For example, instead of writing:: def notify_by_email(employees: Set[Employee]): ... one might write:: def notify_by_email(employees: 'Set[Employee]'): ... .. FIXME: Rigorously define this. Defend it, or find an alternative. Union types ----------- Since accepting a small, limited set of expected types for a single argument is common, there is a new special factory called ``Union``. Example:: from typing import Union def handle_employees(e: Union[Employee, Sequence[Employee]]): if isinstance(e, Employee): e = [e] ... A type factored by ``Union[T1, T2, ...]`` responds ``True`` to ``issubclass`` checks for ``T1`` and any of its subclasses, ``T2`` and any of its subclasses, and so on. One common case of union types are *optional* types. By default, ``None`` is an invalid value for any type, unless a default value of ``None`` has been provided in the function definition. Examples:: def handle_employee(e: Union[Employee, None]): ... As a shorthand for ``Union[T1, None]`` you can write ``Optional[T1]``; for example, the above is equivalent to:: from typing import Optional def handle_employee(e: Optional[Employee]): ... An optional type is also automatically assumed when the default value is ``None``, for example:: def handle_employee(e: Employee = None): ... This is equivalent to:: def handle_employee(e: Optional[Employee] = None): ... .. FIXME: Is this really a good idea? A special kind of union type is ``Any``, a class that responds ``True`` to ``issubclass`` of any class. This lets the user explicitly state that there are no constraints on the type of a specific argument or return value. Platform-specific type checking ------------------------------- In some cases the typing information will depend on the platform that the program is being executed on. To enable specifying those differences, simple conditionals can be used:: from typing import PY2, WINDOWS if PY2: text = unicode else: text = str def f() -> text: ... if WINDOWS: loop = ProactorEventLoop else: loop = UnixSelectorEventLoop Arbitrary literals defined in the form of ``NAME = True`` will also be accepted by the type checker to differentiate type resolution:: DEBUG = False ... if DEBUG: class Tracer: <verbose implementation> else: class Tracer: <dummy implementation> For the purposes of type hinting, the type checker assumes ``__debug__`` is set to ``True``, in other words the ``-O`` command-line option is not used while type checking. Compatibility with other uses of function annotations ----------------------------------------------------- A number of existing or potential use cases for function annotations exist, which are incompatible with type hinting. These may confuse a static type checker. However, since type hinting annotations have no run time behavior (other than evaluation of the annotation expression and storing annotations in the ``__annotations__`` attribute of the function object), this does not make the program incorrect -- it just makes it issue warnings when a static analyzer is used. To mark portions of the program that should not be covered by type hinting, use the following: * a ``@no_type_checks`` decorator on classes and functions * a ``# type: ignore`` comment on arbitrary lines .. FIXME: should we have a module-wide comment as well? Type Hints on Local and Global Variables ======================================== No first-class syntax support for explicitly marking variables as being of a specific type is added by this PEP. To help with type inference in complex cases, a comment of the following format may be used:: x = [] # type: List[Employee] In the case where type information for a local variable is needed before if was declared, an ``Undefined`` placeholder might be used:: from typing import Undefined x = Undefined # type: List[Employee] y = Undefined(int) If type hinting proves useful in general, a syntax for typing variables may be provided in a future Python version. Explicit raised exceptions ========================== No support for listing explicitly raised exceptions is being defined by this PEP. Currently the only known use case for this feature is documentational, in which case the recommendation is to put this information in a docstring. The ``typing`` package ====================== To open the usage of static type checking to Python 3.5 as well as older versions, a uniform namespace is required. For this purpose, a new package in the standard library is introduced called ``typing``. It holds a set of classes representing builtin types with generics, namely: * Dict, used as ``Dict[key_type, value_type]`` * List, used as ``List[element_type]`` * Set, used as ``Set[element_type]``. See remark for ``AbstractSet`` below. * FrozenSet, used as ``FrozenSet[element_type]`` * Tuple, used as ``Tuple[index0_type, index1_type, ...]``. Arbitrary-length tuples might be expressed using ellipsis, in which case the following arguments are considered the same type as the last defined type on the tuple. It also introduces factories and helper members needed to express generics and union types: * Any, used as ``def get(key: str) -> Any: ...`` * Union, used as ``Union[Type1, Type2, Type3]`` * TypeVar, used as ``X = TypeVar('X', Type1, Type2, Type3)`` or simply ``Y = TypeVar('Y')`` * Undefined, used as ``local_variable = Undefined # type: List[int]`` or ``local_variable = Undefined(List[int])`` (the latter being slower during runtime) * Callable, used as ``Callable[[Arg1Type, Arg2Type], ReturnType]`` * AnyArgs, used as ``Callable[AnyArgs, ReturnType]`` * AnyStr, equivalent to ``TypeVar('AnyStr', str, bytes)`` All abstract base classes available in ``collections.abc`` are importable from the ``typing`` package, with added generics support: * ByteString * Callable * Container * Hashable * ItemsView * Iterable * Iterator * KeysView * Mapping * MappingView * MutableMapping * MutableSequence * MutableSet * Sequence * Set as ``AbstractSet``. This name change was required because ``Set`` in the ``typing`` module means ``set()`` with generics. * Sized * ValuesView * Mapping The library includes literals for platform-specific type hinting: * PY2 * PY3, equivalent to ``not PY2`` * WINDOWS * UNIXOID, equivalent to ``not WINDOWS`` The following types are available in the ``typing.io`` module: * IO * BinaryIO * TextIO The following types are provided by the ``typing.re`` module: * Match and Pattern, types of ``re.match()`` and ``re.compile()`` results As a convenience measure, types from ``typing.io`` and ``typing.re`` are also available in ``typing`` (quoting Guido, "There's a reason those modules have two-letter names."). The place of the ``typing`` module in the standard library ---------------------------------------------------------- .. FIXME: complete this section Usage Patterns ============== The main use case of type hinting is static analysis using an external tool without executing the analyzed program. Existing tools used for that purpose like ``pyflakes`` [pyflakes]_ or ``pylint`` [pylint]_ might be extended to support type checking. New tools, like mypy's ``mypy -S`` mode, can be adopted specifically for this purpose. Type checking based on type hints is understood as a best-effort mechanism. In other words, whenever types are not annotated and cannot be inferred, the type checker considers such code valid. Type errors are only reported in case of explicit or inferred conflict. Moreover, as a mechanism that is not tied to execution of the code, it does not affect runtime behaviour. In other words, even in the case of a typing error, the program will continue running. The implementation of a type checker, whether linting source files or enforcing type information during runtime, is out of scope for this PEP. .. FIXME: Describe stub modules. .. FIXME: Describe run-time behavior of generic types. Existing Approaches =================== PEP 482 lists existing approaches in Python and other languages. Is type hinting Pythonic? ========================= Type annotations provide important documentation for how a unit of code should be used. Programmers should therefore provide type hints on public APIs, namely argument and return types on functions and methods considered public. However, because types of local and global variables can be often inferred, they are rarely necessary. The kind of information that type hints hold has always been possible to achieve by means of docstrings. In fact, a number of formalized mini-languages for describing accepted arguments have evolved. Moving this information to the function declaration makes it more visible and easier to access both at runtime and by static analysis. Adding to that the notion that ?explicit is better than implicit?, type hints are indeed *Pythonic*. Acknowledgements ================ This document could not be completed without valuable input, encouragement and advice from Jim Baker, Jeremy Siek, Michael Matson Vitousek, Andrey Vlasovskikh, and Radomir Dopieralski. Influences include existing languages, libraries and frameworks mentioned in PEP 482. Many thanks to their creators, in alphabetical order: Stefan Behnel, William Edwards, Greg Ewing, Larry Hastings, Anders Hejlsberg, Alok Menghrajani, Travis E. Oliphant, Joe Pamer, Raoul-Gabriel Urma, and Julien Verlaguet. References ========== .. [mypy] http://mypy-lang.org .. [pyflakes] https://github.com/pyflakes/pyflakes/ .. [pylint] http://www.pylint.org Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150116/cb2b7c9a/attachment-0001.html> From guido at python.org Fri Jan 16 18:30:58 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Jan 2015 09:30:58 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> Message-ID: <CAP7+vJL5--BU0WkKU=dXxDkvX70ts02Qh4H3R31ctt6+XhSOLg@mail.gmail.com> I forgot to mention the tracker! A lot of the recent discussions regarding PEP 484 have actually been held in an issue tracker on GitHub: https://github.com/ambv/typehinting/issues . We intend to keep this tracker around forever, as a collection of artifacts that may help understanding various decisions. We also hope that continued discussion will happen in this tracker (both in existing issues and in new ones -- anyone with a github account can open a new issue). So when should you use the tracker and when should you post to python-ideas? That's really up to you. The tracker is probably a better place to have detailed discussions about type system esoterica; python-ideas is probably a better place to ask for clarification. When in doubt, try python-ideas; and I'd prefer it if the tracker didn't get flooded with new issues about typos and such (you can send those to me directly). Note that the GitHub project also contains the history of the PEP, as well as an experimental typing.py module. The latter is far from finished and hasn't been updated to match the PEP yet, so read it at your own risk. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150116/5d31fd24/attachment.html> From ron3200 at gmail.com Fri Jan 16 20:26:45 2015 From: ron3200 at gmail.com (Ron Adam) Date: Fri, 16 Jan 2015 13:26:45 -0600 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CAA68w_k4wUPs8A2HgH5M0Jz-4M6L90KY4rpo_VDxGbV7bA5BZA@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <3b503462-8868-4c02-9988-bd53c66fe271@googlegroups.com> <CALGmxEJet44RNdD6xD_bYPcUniwOUTX6j7pL79SjSSHEhhRZ5Q@mail.gmail.com> <CAA68w_kx1kyVnoPajUGGsJgCv=cQRwQ56RdFuBLZW15C676-Qw@mail.gmail.com> <CALGmxEJSZMUyqicL2ut9RhB2w5pfraw-L=ubGDX1ttk7ctm8Ww@mail.gmail.com> <CAA68w_=vPaL8y7aS3szELshU_m6Mq1Jg777-gKihj-XrC5+0vQ@mail.gmail.com> <6473609160848220685@unknownmsgid> <CAA68w_kktJGaj=Z-9NGfOq7B_choKk6b-t5tkb4MrKUAkbXAtw@mail.gmail.com> <871tmvxv2s.fsf@uwakimon.sk.tsukuba.ac.jp> <CAA68w_nRoS1Kk-Kthfy+X1gEpPF2rLA7Ck_-V0tJTy_W5i2JgQ@mail.gmail.com> <CAA68w_k4wUPs8A2HgH5M0Jz-4M6L90KY4rpo_VDxGbV7bA5BZA@mail.gmail.com> Message-ID: <m9bolm$65u$1@ger.gmane.org> On 01/16/2015 09:13 AM, Neil Girdhar wrote: > Actually, I was wrong about the exponential distribution's KL divergence. > It's the relative error (b-a)/b plus another term: log(b/a) ? so I guess I > don't see what relative error means except as a heuristic. > Anyway, even if your symmetric error makes sense to you, does anyone > already use it? If it were up to me, relative error would be b-a/b + > log(b/a), but since no one uses that, I think it's a bad idea. http://en.wikipedia.org/wiki/Approximation_error In this case, we are only confirming a single value is within an already determined range. In most cases that is enough to confirm the coding of the algorithm is correct, where as the more stringent methods you are thinking of is used to confirm the behaviour of the algorithm is valid. Cheers, Ron From stefan_ml at behnel.de Fri Jan 16 20:38:50 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 16 Jan 2015 20:38:50 +0100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> Message-ID: <m9bpcb$eq7$1@ger.gmane.org> Guido van Rossum schrieb am 16.01.2015 um 18:17: > After a long editing process we've got PEP 484 > <https://www.python.org/dev/peps/pep-0484/> (Type Hints) ready for your > review. I can't see it mention interoperability with other usages of function annotations. I think the idea was that in this case, the type hints must be explicitly declared as such, e.g. def func(x: {'type': str, 'wobble': 'off', 'ctype': 'std::string[utf8]'}) -> {'type': str, 'wobble': 'sure'}: return x + 'huhu' Stefan From guido at python.org Fri Jan 16 20:45:01 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Jan 2015 11:45:01 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <m9bpcb$eq7$1@ger.gmane.org> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> Message-ID: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> That's discussed in this section: https://www.python.org/dev/peps/pep-0484/#compatibility-with-other-uses-of-function-annotations . The dict notation (which was proposed in an earlier thread on python-ideas) looks too messy to seriously support. On Fri, Jan 16, 2015 at 11:38 AM, Stefan Behnel <stefan_ml at behnel.de> wrote: > Guido van Rossum schrieb am 16.01.2015 um 18:17: > > After a long editing process we've got PEP 484 > > <https://www.python.org/dev/peps/pep-0484/> (Type Hints) ready for your > > review. > > I can't see it mention interoperability with other usages of function > annotations. I think the idea was that in this case, the type hints must be > explicitly declared as such, e.g. > > def func(x: {'type': str, 'wobble': 'off', > 'ctype': 'std::string[utf8]'}) -> > {'type': str, 'wobble': 'sure'}: > return x + 'huhu' > > Stefan > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150116/1c54410d/attachment.html> From stefan_ml at behnel.de Fri Jan 16 20:56:47 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 16 Jan 2015 20:56:47 +0100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> Message-ID: <m9bqe0$2ub$1@ger.gmane.org> Guido van Rossum schrieb am 16.01.2015 um 20:45: > On Fri, Jan 16, 2015 at 11:38 AM, Stefan Behnel wrote: > >> Guido van Rossum schrieb am 16.01.2015 um 18:17: >>> After a long editing process we've got PEP 484 >>> <https://www.python.org/dev/peps/pep-0484/> (Type Hints) ready for your >>> review. >> >> I can't see it mention interoperability with other usages of function >> annotations. I think the idea was that in this case, the type hints must be >> explicitly declared as such, e.g. >> >> def func(x: {'type': str, 'wobble': 'off', >> 'ctype': 'std::string[utf8]'}) -> >> {'type': str, 'wobble': 'sure'}: >> return x + 'huhu' > > That's discussed in this section: > https://www.python.org/dev/peps/pep-0484/#compatibility-with-other-uses-of-function-annotations > . The dict notation (which was proposed in an earlier thread on > python-ideas) looks too messy to seriously support. So, what you're saying is: either Python type hints are used exclusively, and no other annotations, or the user must add magic comments in all places where annotations are used, so that each tool that processes a subset of them knows which to process and which to ignore. And mixing them in one function will still be completely impossible. I don't see how that's less messy than explicit naming of the type of annotation. It's just less generally usable. Also, how would you support different type systems, such as C/C++ in Cython or Java types in Jython? In both cases, "int" or "float" would mean something different than the Python types "int" and "float", for example. Stefan From guido at python.org Fri Jan 16 21:08:05 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Jan 2015 12:08:05 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <m9bqe0$2ub$1@ger.gmane.org> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> Message-ID: <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> Yes, the PEP tries to gently push an agenda where annotations are only used for Python type hints. Note that this was the original idea when PEP 3107 was created, even if it doesn't say so. Jim Baker from the Jython team is totally on board with the proposal, he just wants to have a way to declare that e.g. java.Util.ArrayList is equivalent to typing.List. This can be done through a stub file. Read https://github.com/ambv/typehinting/issues/7 for details. Also note that Jim co-write a paper on the use of gradual typing for Jython, referenced in PEP 482 (the literature overview) -- see https://www.python.org/dev/peps/pep-0482/#reticulated-python. I don't know what to answer for Cython -- doesn't it have its own syntax for declaring C types? Or is it now also using annotations? If Cython uses annotations with different meanings, perhaps for Cython files the Python annotations could be put in stub files. Stub files take precedence over regular modules when the static checker runs. (Note that the PEP doesn't yet describe stubs, simply because we ran out of time this week to write it up. They are implemented by mypy. See https://github.com/ambv/typehinting/issues/22 . On Fri, Jan 16, 2015 at 11:56 AM, Stefan Behnel <stefan_ml at behnel.de> wrote: > Guido van Rossum schrieb am 16.01.2015 um 20:45: > > On Fri, Jan 16, 2015 at 11:38 AM, Stefan Behnel wrote: > > > >> Guido van Rossum schrieb am 16.01.2015 um 18:17: > >>> After a long editing process we've got PEP 484 > >>> <https://www.python.org/dev/peps/pep-0484/> (Type Hints) ready for > your > >>> review. > >> > >> I can't see it mention interoperability with other usages of function > >> annotations. I think the idea was that in this case, the type hints > must be > >> explicitly declared as such, e.g. > >> > >> def func(x: {'type': str, 'wobble': 'off', > >> 'ctype': 'std::string[utf8]'}) -> > >> {'type': str, 'wobble': 'sure'}: > >> return x + 'huhu' > > > > That's discussed in this section: > > > https://www.python.org/dev/peps/pep-0484/#compatibility-with-other-uses-of-function-annotations > > . The dict notation (which was proposed in an earlier thread on > > python-ideas) looks too messy to seriously support. > > So, what you're saying is: either Python type hints are used exclusively, > and no other annotations, or the user must add magic comments in all places > where annotations are used, so that each tool that processes a subset of > them knows which to process and which to ignore. And mixing them in one > function will still be completely impossible. > > I don't see how that's less messy than explicit naming of the type of > annotation. It's just less generally usable. > > Also, how would you support different type systems, such as C/C++ in Cython > or Java types in Jython? In both cases, "int" or "float" would mean > something different than the Python types "int" and "float", for example. > > Stefan > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150116/b657680c/attachment.html> From abarnert at yahoo.com Fri Jan 16 21:14:38 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 16 Jan 2015 12:14:38 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> Message-ID: <4E76D308-4378-4895-8A0D-916706A24A9F@yahoo.com> One quick clarification question: Can strings-as-forward-declarations be used inside non-string type annotations--e.g.. Set['Employee'] instead of 'Set[Employee]'? If so, do they have identical meaning? I can imagine, say, a refactoring tool that didn't understand the full Python+MyPy syntax being able to do things with the former that it couldn't with the latter. And I think it might look more natural for simple recursive types. But those are just vague feelings, so don't take this as a suggestion. Sent from a random iPhone On Jan 16, 2015, at 9:17, Guido van Rossum <guido at python.org> wrote: > After a long editing process we've got PEP 484 (Type Hints) ready for your review. This is by no means final, and several areas are either open (to be resolved in a later draft) or postponed (to a different PEP altogether). But there's enough meat that I think we can start having the discussion. Please also see PEP 483 (The Theory of Type Hint; copied and reformatted from the original Quip document that I posted just before last Christmas) and PEP 482 (Literature Overview for Type Hints, by ?ukasz). Those are informational PEPs though; the actual spec is focused in PEP 484 (the only one on the Standards Track). > > As I said earlier, I hope to have a rough consensus before PyCon and working code (just the typing.py module, really) committed to CPython before the last 3.5 alpha. > > Here is the raw text of PEP 484. Fire away!! > > PEP: 484 > Title: Type Hints > Version: $Revision$ > Last-Modified: $Date$ > Author: Guido van Rossum <guido at python.org>, Jukka Lehtosalo <jukka.lehtosalo at iki.fi>, ?ukasz Langa <lukasz at langa.pl> > Discussions-To: Python-Dev <python-dev at python.org> > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 29-Sep-2014 > Post-History: 16-Jan-2015 > Resolution: > > > Abstract > ======== > > This PEP introduces a standard syntax for type hints using annotations > on function definitions. > > The proposal is strongly inspired by mypy [mypy]_. > > The theory behind type hints and gradual typing is explained in PEP 483. > > > Rationale and Goals > =================== > > PEP 3107 added support for arbitrary annotations on parts of a function > definition. Although no meaning was assigned to annotations then, there > has always been an implicit goal to use them for type hinting, which is > listed as the first possible use case in said PEP. > > This PEP aims to provide a standard syntax for type annotations, opening > up Python code to easier static analysis and refactoring, potential > runtime type checking, and performance optimizations utilizing type > information. > > > Type Definition Syntax > ====================== > > The syntax leverages PEP 3107-style annotations with a number of > extensions described in sections below. In its basic form, type hinting > is used by filling function annotations with classes:: > > def greeting(name: str) -> str: > return 'Hello ' + name > > This denotes that the expected type of the ``name`` argument is ``str``. > Analogically, the expected return type is ``str``. Subclasses of > a specified argument type are also accepted as valid types for that > argument. > > Abstract base classes, types available in the ``types`` module, and > user-defined classes may be used as type hints as well. Annotations > must be valid expressions that evaluate without raising exceptions at > the time the function is defined. In addition, the needs of static > analysis require that annotations must be simple enough to be > interpreted by static analysis tools. (This is an intentionally > somewhat vague requirement.) > > .. FIXME: Define rigorously what is/isn't supported. > > When used as an annotation, the expression ``None`` is considered > equivalent to ``NoneType`` (i.e., ``type(None)`` for type hinting > purposes. > > Type aliases are also valid type hints:: > > integer = int > > def retry(url: str, retry_count: integer): ... > > New names that are added to support features described in following > sections are available in the ``typing`` package. > > > Callbacks > --------- > > Frameworks expecting callback functions of specific signatures might be > type hinted using ``Callable[[Arg1Type, Arg2Type], ReturnType]``. > Examples:: > > from typing import Any, AnyArgs, Callable > > def feeder(get_next_item: Callable[[], Item]): ... > > def async_query(on_success: Callable[[int], None], on_error: Callable[[int, Exception], None]): ... > > def partial(func: Callable[AnyArgs, Any], *args): ... > > Since using callbacks with keyword arguments is not perceived as > a common use case, there is currently no support for specifying keyword > arguments with ``Callable``. > > > Generics > -------- > > Since type information about objects kept in containers cannot be > statically inferred in a generic way, abstract base classes have been > extended to support subscription to denote expected types for container > elements. Example:: > > from typing import Mapping, Set > > def notify_by_email(employees: Set[Employee], overrides: Mapping[str, str]): ... > > Generics can be parametrized by using a new factory available in > ``typing`` called ``TypeVar``. Example:: > > from typing import Sequence, TypeVar > > T = TypeVar('T') # Declare type variable > > def first(l: Sequence[T]) -> T: # Generic function > return l[0] > > In this case the contract is that the returning value is consistent with > the elements held by the collection. > > ``TypeVar`` supports constraining parametric types to classes with any of > the specified bases. Example:: > > from typing import Iterable > > X = TypeVar('X') > Y = TypeVar('Y', Iterable[X]) > > def filter(rule: Callable[[X], bool], input: Y) -> Y: > ... > > .. FIXME: Add an example with multiple bases defined. > > In the example above we specify that ``Y`` can be any subclass of > Iterable with elements of type ``X``, as long as the return type of > ``filter()`` will be the same as the type of the ``input`` > argument. > > .. FIXME: Explain more about how this works. > > > Forward references > ------------------ > > When a type hint contains names that have not been defined yet, that > definition may be expressed as a string, to be resolved later. For > example, instead of writing:: > > def notify_by_email(employees: Set[Employee]): ... > > one might write:: > > def notify_by_email(employees: 'Set[Employee]'): ... > > .. FIXME: Rigorously define this. Defend it, or find an alternative. > > > Union types > ----------- > > Since accepting a small, limited set of expected types for a single > argument is common, there is a new special factory called ``Union``. > Example:: > > from typing import Union > > def handle_employees(e: Union[Employee, Sequence[Employee]]): > if isinstance(e, Employee): > e = [e] > ... > > A type factored by ``Union[T1, T2, ...]`` responds ``True`` to > ``issubclass`` checks for ``T1`` and any of its subclasses, ``T2`` and > any of its subclasses, and so on. > > One common case of union types are *optional* types. By default, > ``None`` is an invalid value for any type, unless a default value of > ``None`` has been provided in the function definition. Examples:: > > def handle_employee(e: Union[Employee, None]): ... > > As a shorthand for ``Union[T1, None]`` you can write ``Optional[T1]``; > for example, the above is equivalent to:: > > from typing import Optional > > def handle_employee(e: Optional[Employee]): ... > > An optional type is also automatically assumed when the default value is > ``None``, for example:: > > def handle_employee(e: Employee = None): ... > > This is equivalent to:: > > def handle_employee(e: Optional[Employee] = None): ... > > .. FIXME: Is this really a good idea? > > A special kind of union type is ``Any``, a class that responds > ``True`` to ``issubclass`` of any class. This lets the user > explicitly state that there are no constraints on the type of a > specific argument or return value. > > > Platform-specific type checking > ------------------------------- > > In some cases the typing information will depend on the platform that > the program is being executed on. To enable specifying those > differences, simple conditionals can be used:: > > from typing import PY2, WINDOWS > > if PY2: > text = unicode > else: > text = str > > def f() -> text: ... > > if WINDOWS: > loop = ProactorEventLoop > else: > loop = UnixSelectorEventLoop > > Arbitrary literals defined in the form of ``NAME = True`` will also be > accepted by the type checker to differentiate type resolution:: > > DEBUG = False > ... > if DEBUG: > class Tracer: > <verbose implementation> > else: > class Tracer: > <dummy implementation> > > For the purposes of type hinting, the type checker assumes ``__debug__`` > is set to ``True``, in other words the ``-O`` command-line option is not > used while type checking. > > > Compatibility with other uses of function annotations > ----------------------------------------------------- > > A number of existing or potential use cases for function annotations > exist, which are incompatible with type hinting. These may confuse a > static type checker. However, since type hinting annotations have no > run time behavior (other than evaluation of the annotation expression > and storing annotations in the ``__annotations__`` attribute of the > function object), this does not make the program incorrect -- it just > makes it issue warnings when a static analyzer is used. > > To mark portions of the program that should not be covered by type > hinting, use the following: > > * a ``@no_type_checks`` decorator on classes and functions > > * a ``# type: ignore`` comment on arbitrary lines > > .. FIXME: should we have a module-wide comment as well? > > > Type Hints on Local and Global Variables > ======================================== > > No first-class syntax support for explicitly marking variables as being > of a specific type is added by this PEP. To help with type inference in > complex cases, a comment of the following format may be used:: > > x = [] # type: List[Employee] > > In the case where type information for a local variable is needed before > if was declared, an ``Undefined`` placeholder might be used:: > > from typing import Undefined > > x = Undefined # type: List[Employee] > y = Undefined(int) > > If type hinting proves useful in general, a syntax for typing variables > may be provided in a future Python version. > > > Explicit raised exceptions > ========================== > > No support for listing explicitly raised exceptions is being defined by > this PEP. Currently the only known use case for this feature is > documentational, in which case the recommendation is to put this > information in a docstring. > > > The ``typing`` package > ====================== > > To open the usage of static type checking to Python 3.5 as well as older > versions, a uniform namespace is required. For this purpose, a new > package in the standard library is introduced called ``typing``. It > holds a set of classes representing builtin types with generics, namely: > > * Dict, used as ``Dict[key_type, value_type]`` > > * List, used as ``List[element_type]`` > > * Set, used as ``Set[element_type]``. See remark for ``AbstractSet`` > below. > > * FrozenSet, used as ``FrozenSet[element_type]`` > > * Tuple, used as ``Tuple[index0_type, index1_type, ...]``. > Arbitrary-length tuples might be expressed using ellipsis, in which > case the following arguments are considered the same type as the last > defined type on the tuple. > > It also introduces factories and helper members needed to express > generics and union types: > > * Any, used as ``def get(key: str) -> Any: ...`` > > * Union, used as ``Union[Type1, Type2, Type3]`` > > * TypeVar, used as ``X = TypeVar('X', Type1, Type2, Type3)`` or simply > ``Y = TypeVar('Y')`` > > * Undefined, used as ``local_variable = Undefined # type: List[int]`` or > ``local_variable = Undefined(List[int])`` (the latter being slower > during runtime) > > * Callable, used as ``Callable[[Arg1Type, Arg2Type], ReturnType]`` > > * AnyArgs, used as ``Callable[AnyArgs, ReturnType]`` > > * AnyStr, equivalent to ``TypeVar('AnyStr', str, bytes)`` > > All abstract base classes available in ``collections.abc`` are > importable from the ``typing`` package, with added generics support: > > * ByteString > > * Callable > > * Container > > * Hashable > > * ItemsView > > * Iterable > > * Iterator > > * KeysView > > * Mapping > > * MappingView > > * MutableMapping > > * MutableSequence > > * MutableSet > > * Sequence > > * Set as ``AbstractSet``. This name change was required because ``Set`` > in the ``typing`` module means ``set()`` with generics. > > * Sized > > * ValuesView > > * Mapping > > The library includes literals for platform-specific type hinting: > > * PY2 > > * PY3, equivalent to ``not PY2`` > > * WINDOWS > > * UNIXOID, equivalent to ``not WINDOWS`` > > The following types are available in the ``typing.io`` module: > > * IO > > * BinaryIO > > * TextIO > > The following types are provided by the ``typing.re`` module: > > * Match and Pattern, types of ``re.match()`` and ``re.compile()`` > results > > As a convenience measure, types from ``typing.io`` and ``typing.re`` are > also available in ``typing`` (quoting Guido, "There's a reason those > modules have two-letter names."). > > > The place of the ``typing`` module in the standard library > ---------------------------------------------------------- > > .. FIXME: complete this section > > > Usage Patterns > ============== > > The main use case of type hinting is static analysis using an external > tool without executing the analyzed program. Existing tools used for > that purpose like ``pyflakes`` [pyflakes]_ or ``pylint`` [pylint]_ > might be extended to support type checking. New tools, like mypy's > ``mypy -S`` mode, can be adopted specifically for this purpose. > > Type checking based on type hints is understood as a best-effort > mechanism. In other words, whenever types are not annotated and cannot > be inferred, the type checker considers such code valid. Type errors > are only reported in case of explicit or inferred conflict. Moreover, > as a mechanism that is not tied to execution of the code, it does not > affect runtime behaviour. In other words, even in the case of a typing > error, the program will continue running. > > The implementation of a type checker, whether linting source files or > enforcing type information during runtime, is out of scope for this PEP. > > .. FIXME: Describe stub modules. > > .. FIXME: Describe run-time behavior of generic types. > > > Existing Approaches > =================== > > PEP 482 lists existing approaches in Python and other languages. > > > Is type hinting Pythonic? > ========================= > > Type annotations provide important documentation for how a unit of code > should be used. Programmers should therefore provide type hints on > public APIs, namely argument and return types on functions and methods > considered public. However, because types of local and global variables > can be often inferred, they are rarely necessary. > > The kind of information that type hints hold has always been possible to > achieve by means of docstrings. In fact, a number of formalized > mini-languages for describing accepted arguments have evolved. Moving > this information to the function declaration makes it more visible and > easier to access both at runtime and by static analysis. Adding to that > the notion that ?explicit is better than implicit?, type hints are > indeed *Pythonic*. > > > Acknowledgements > ================ > > This document could not be completed without valuable input, > encouragement and advice from Jim Baker, Jeremy Siek, Michael Matson > Vitousek, Andrey Vlasovskikh, and Radomir Dopieralski. > > Influences include existing languages, libraries and frameworks > mentioned in PEP 482. Many thanks to their creators, in alphabetical > order: Stefan Behnel, William Edwards, Greg Ewing, Larry Hastings, > Anders Hejlsberg, Alok Menghrajani, Travis E. Oliphant, Joe Pamer, > Raoul-Gabriel Urma, and Julien Verlaguet. > > > References > ========== > > .. [mypy] > http://mypy-lang.org > > .. [pyflakes] > https://github.com/pyflakes/pyflakes/ > > .. [pylint] > http://www.pylint.org > > > Copyright > ========= > > This document has been placed in the public domain. > > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150116/ef78852d/attachment-0001.html> From brakhane at googlemail.com Fri Jan 16 21:34:59 2015 From: brakhane at googlemail.com (Dennis Brakhane) Date: Fri, 16 Jan 2015 21:34:59 +0100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> Message-ID: <54B975F3.4000104@gmail.com> Am 16.01.2015 um 20:45 schrieb Guido van Rossum: > That's discussed in this section: > https://www.python.org/dev/peps/pep-0484/#compatibility-with-other-uses-of-function-annotations > . I still feel that saying "if you use function annotations, that function cannot be statically checked" is bad. If type hints are successful, many projects will adopt them, especially big projects and/or projects in corporations. Probably many will require that all code must pass the type checker, and (at least) public functions be type annotated. Let's suppose that indeed happens. If I now were writing a new open source Python library, I would want that my library could be used in big projects, and therefore would have an incentive to type annotate my library. If I don't, a big project might choose another lib instead that does. So in that future, a library that wants to be successful basically can't use type annotations, even if they would have been an elegant solution to a particular problem. That feels somewhat wrong to me. A possible work around using skeletons seems ugly, you would have to edit two different files and keep them in sync. I feel there should be another way to make type annotations and "creative" annotations coexist. > The dict notation (which was proposed in an earlier thread on > python-ideas) looks too messy to seriously support. How about using a tuple? I'll admit that I've proposed it before, but got no reaction. If an annotation is a tuple, the type checker would check each element for known values, and every other thing that uses __annotations__ could do the same. Stefan's example could look something like def func(x: (str, wobble(False), ctype('std::string[utf8]'))) -> (str, wobble(True)) It's still not pretty, but it has the advantage that it is only needed for when more than one annotation is used. It has the disadvantage of preventing (int,str) being an alias for Tuple[int,str]. Actually, I don't care that much what particular solution is used to allow type annotations and non-type annotations to coexist, but I find it important that is possible. From encukou at gmail.com Fri Jan 16 22:08:12 2015 From: encukou at gmail.com (Petr Viktorin) Date: Fri, 16 Jan 2015 22:08:12 +0100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> Message-ID: <CA+=+wqCbBm77f3uYnXmp8sobu+6gkMZU+_wcWY=mwHjELGwU9w@mail.gmail.com> "On Fri, Jan 16, 2015 at 6:17 PM, Guido van Rossum <guido at python.org> wrote: > After a long editing process we've got PEP 484 (Type Hints) ready for your > review. This is by no means final, and several areas are either open (to be > resolved in a later draft) or postponed (to a different PEP altogether). But > there's enough meat that I think we can start having the discussion. Please > also see PEP 483 (The Theory of Type Hint; copied and reformatted from the > original Quip document that I posted just before last Christmas) and PEP 482 > (Literature Overview for Type Hints, by ?ukasz). Those are informational > PEPs though; the actual spec is focused in PEP 484 (the only one on the > Standards Track). > > As I said earlier, I hope to have a rough consensus before PyCon and working > code (just the typing.py module, really) committed to CPython before the > last 3.5 alpha. > > Here is the raw text of PEP 484. Fire away!! [...] > * Tuple, used as ``Tuple[index0_type, index1_type, ...]``. > Arbitrary-length tuples might be expressed using ellipsis, in which > case the following arguments are considered the same type as the last > defined type on the tuple. How would Tuple[...] or Tuple[t1, ..., t2] work? The PEP should say such shenanigans are undefined, rather than leave it to interpretation. Also, does the empty tuple match Tuple[int, ...]? [...] > * WINDOWS > > * UNIXOID, equivalent to ``not WINDOWS`` I wonder what the use cases are here. Is this useful, as a dichotomy, outside of asyncio (or other low-level stdlib code)? From guido at python.org Fri Jan 16 22:30:49 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Jan 2015 13:30:49 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAN8d9gnvftkuaX56k7RA_G_xh_iY5vTJd+0+aW6O_v_y=XmtMQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <CAN8d9gnvftkuaX56k7RA_G_xh_iY5vTJd+0+aW6O_v_y=XmtMQ@mail.gmail.com> Message-ID: <CAP7+vJL5Ap22XKgOTu13cNZXOG0TYG+BsQUSdx7=RKCHv5ydBg@mail.gmail.com> On Fri, Jan 16, 2015 at 12:05 PM, Philipp A. <flying-sheep at web.de> wrote: > hi, nice work! > > I like most of the ideas, except the semantic comments and some specific > abstract classes. > > so criticism first: > > 1. i don?t think comments should ever have an impact on the code. > They don't. They also change the errors emitted by the static checker (which is not invoked when you run the code -- it is like a linter). > i?d prefer ?assert isinstance(...)? for type annotations on variables. > But that *would* affect runtime, by slowing down execution. Also for generic types the type checking could be either very slow (if checking List[int] would check every item of a list) or always fail (if a type variable is involved). See discussion at https://github.com/ambv/typehinting/issues/35 > > 2. also i don?t see why Dict, (non-abstract) Set, and List exist as long > as we have Mapping and so on. Tuple is different because of its ?Multiple > return value? status and status as ?is treated as Union[Class1, Class2] > when used with instanceof(), except and friends?. > > but checking for instanceof(..., list) is unpytonic! imho List and Dict > really go against python?s data model and protocol-based nature. > > i?d be for consistency: Set and MutableSet, Sequence and MutableSequence, > Mapping and MutableMapping. no ?abstract? (because all of this is > abstract), no ?frozen? (because frozen sets are just immutable sets and > immutable behavior is a subset of the corresponding mutable behavior). > But there is a lot of code that really needs the concrete types. E.g. list+list returns a list, but sequence+sequence may fail (as list+tuple exemplifies). Or list.sort. Etc. > Questions: > > 1. will we be able to use Union[...] in places like an except clause, > where tuples are used to mean the same right now? would make sense imho > No. Remember TOOWTDI. (Also the PEP intentionally stays far away from exception handling.) > 2. what are ItemsView and KeysView for? what makes them different from > FrozenSet or Sequence? > They exist in collections.abc. Note that KeysView and ItemsView are (abstract) Sets, but ValuesView is not (you can have multiple equal values). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150116/0f74d4a0/attachment.html> From guido at python.org Fri Jan 16 22:34:36 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Jan 2015 13:34:36 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <4E76D308-4378-4895-8A0D-916706A24A9F@yahoo.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <4E76D308-4378-4895-8A0D-916706A24A9F@yahoo.com> Message-ID: <CAP7+vJKBfmkSx4JHyYvghABNKm_MKGoPhwaJHhkdHQxvCGz3kQ@mail.gmail.com> On Fri, Jan 16, 2015 at 12:14 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > One quick clarification question: > > Can strings-as-forward-declarations be used inside non-string type > annotations--e.g.. Set['Employee'] instead of 'Set[Employee]'? If so, do > they have identical meaning? > Yes. There's a FIXME in the PEP source to define forward references rigorously. See also https://github.com/ambv/typehinting/issues/34 > > I can imagine, say, a refactoring tool that didn't understand the full > Python+MyPy syntax being able to do things with the former that it couldn't > with the latter. And I think it might look more natural for simple > recursive types. But those are just vague feelings, so don't take this as a > suggestion. > Well, the most common use case is where you are referencing the current class (which is in the process of being defined but isn't actually created until the entire class body is finished). E.g. class Node(Generic[T]): def add_left(self, node: 'Node[T]'): ... -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150116/2e565c6c/attachment-0001.html> From guido at python.org Fri Jan 16 22:40:15 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Jan 2015 13:40:15 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <54B975F3.4000104@gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> Message-ID: <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> On Fri, Jan 16, 2015 at 12:34 PM, Dennis Brakhane <brakhane at googlemail.com> wrote: > Am 16.01.2015 um 20:45 schrieb Guido van Rossum: > > That's discussed in this section: > > > https://www.python.org/dev/peps/pep-0484/#compatibility-with-other-uses-of-function-annotations > > . > I still feel that saying "if you use function annotations, that function > cannot be statically checked" is bad. > > If type hints are successful, many projects will adopt them, especially > big projects and/or projects in corporations. Probably many will require > that all code must pass the type checker, and (at least) public > functions be type annotated. > > Let's suppose that indeed happens. If I now were writing a new open > source Python library, I would want that my library could be used in big > projects, and therefore would have an incentive to type annotate my > library. If I don't, a big project might choose another lib instead that > does. > > So in that future, a library that wants to be successful basically can't > use type annotations, even if they would have been an elegant solution > to a particular problem. That feels somewhat wrong to me. > > A possible work around using skeletons seems ugly, you would have to > edit two different files and keep them in sync. > > I feel there should be another way to make type annotations and > "creative" annotations coexist. > The cleanest way to do this would be to use decorators for the non-type-hint use case. > > > > The dict notation (which was proposed in an earlier thread on > > python-ideas) looks too messy to seriously support. > How about using a tuple? I'll admit that I've proposed it before, but > got no reaction. > > If an annotation is a tuple, the type checker would check each element > for known values, and every other thing that uses __annotations__ could > do the same. Stefan's example could look something like > > def func(x: (str, wobble(False), ctype('std::string[utf8]'))) -> (str, > wobble(True)) > > It's still not pretty, but it has the advantage that it is only needed > for when more than one annotation is used. > It's very ambiguous though. Especially since plain strings are taken to be type annotations (used for forward references). > It has the disadvantage of preventing (int,str) being an alias for > Tuple[int,str]. > But we're not proposing that (overloading native container syntax for types was proposed and rejected early on). > > Actually, I don't care that much what particular solution is used to > allow type annotations and non-type annotations to coexist, but I find > it important that is possible. > Decorators. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150116/c0638ec5/attachment.html> From brakhane at googlemail.com Fri Jan 16 22:39:51 2015 From: brakhane at googlemail.com (Dennis Brakhane) Date: Fri, 16 Jan 2015 22:39:51 +0100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> Message-ID: <54B98527.2010508@gmail.com> > > Forward references <https://www.python.org/dev/peps/pep-0484/#id9> > > When a type hint contains names that have not been defined yet, that > definition may be expressed as a string, to be resolved later. Is it intentionally undefined when exactly "later" is? Silly example: from typing import Set def foo(bar: 'Set[Employee]') -> None: ... class Employee: ... def foo2(bar: Set[Employee]) -> None: ... class Set(Generic[T]): # oops, I didn't pay attention and redefined Set ... A conforming type checker could interpret the type of foo in two ways so that foo either takes a normal set (resolved right after Employee was defined) or an instance of the new class Set (what Set refers to at the end of the file). In the first case, the signatures of foo and foo2 are the same, in the second one, foo2(bar: Set[Employee]) and foo2(bar: 'Set[Employee]') mean different things. Of course, a good style checker should have slapped the user for redefining Set, so it's more of a academic problem and leaving "later" vague doesn't matter in practise, but I'd still document the fact that conforming implementations are free to choose what later means. From guido at python.org Fri Jan 16 22:47:53 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Jan 2015 13:47:53 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CA+=+wqCbBm77f3uYnXmp8sobu+6gkMZU+_wcWY=mwHjELGwU9w@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <CA+=+wqCbBm77f3uYnXmp8sobu+6gkMZU+_wcWY=mwHjELGwU9w@mail.gmail.com> Message-ID: <CAP7+vJL6u+v0tuDF-dv08F-Tj1KT+L_Xo7b5W0naFYR0Gpd+=A@mail.gmail.com> On Fri, Jan 16, 2015 at 1:08 PM, Petr Viktorin <encukou at gmail.com> wrote: > "On Fri, Jan 16, 2015 at 6:17 PM, Guido van Rossum <guido at python.org> > wrote: > [...] > > * Tuple, used as ``Tuple[index0_type, index1_type, ...]``. > > Arbitrary-length tuples might be expressed using ellipsis, in which > > case the following arguments are considered the same type as the last > > defined type on the tuple. > > How would Tuple[...] or Tuple[t1, ..., t2] work? The PEP should say > such shenanigans are undefined, rather than leave it to > interpretation. > Good point. This needs to be elaborated. See https://github.com/ambv/typehinting/issues/30. (Wanna submit a pull request?) > Also, does the empty tuple match Tuple[int, ...]? > I think it probably does. At least the corresponding thing for lists works in mypy: x = [] # type: List[int] So I think we should allow a similar thing for tuples: x = () # type: Tuple[int, ...] > [...] > > * WINDOWS > > > > * UNIXOID, equivalent to ``not WINDOWS`` > > I wonder what the use cases are here. Is this useful, as a dichotomy, > outside of asyncio (or other low-level stdlib code)? > The stdlib is full of things that work differently on Windows. Consequently application code will also often contain things like if <windows>: <do it the Windows way> else: <do it the other way> Typically each branch can only be type-checked if the stdlib for that particular platform is accessible. So it makes sense to standardize on a way of spelling the test in a way that static checkers can understand. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150116/a94ca1f7/attachment.html> From guido at python.org Fri Jan 16 22:51:31 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Jan 2015 13:51:31 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <54B98527.2010508@gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <54B98527.2010508@gmail.com> Message-ID: <CAP7+vJ+9MJAiFe_yLe9Z0=pz3JeNmWd7tSCgZP0Bvd=B-fW5+A@mail.gmail.com> On Fri, Jan 16, 2015 at 1:39 PM, Dennis Brakhane <brakhane at googlemail.com> wrote: > > > > Forward references <https://www.python.org/dev/peps/pep-0484/#id9> > > > > When a type hint contains names that have not been defined yet, that > > definition may be expressed as a string, to be resolved later. > > Is it intentionally undefined when exactly "later" is? > I doubt it is. But specifying the exact semantics of every nit of the static type checking algorithm is beyond the scope of the PEP -- it would be as complex as the implementation of mypy. The intention is that this is a forward reference to something defined at some later point in the same file. > Silly example: > > from typing import Set > > def foo(bar: 'Set[Employee]') -> None: > ... > > class Employee: > ... > > def foo2(bar: Set[Employee]) -> None: > ... > > class Set(Generic[T]): # oops, I didn't pay attention and redefined Set > ... > I think a static checker should always warn about such redefinitions. > > > A conforming type checker could interpret the type of foo in two ways so > that foo either takes a normal set (resolved right after Employee was > defined) or an instance of the new class Set (what Set refers to at the > end of the file). > > In the first case, the signatures of foo and foo2 are the same, in the > second one, foo2(bar: Set[Employee]) and foo2(bar: 'Set[Employee]') mean > different things. > > Of course, a good style checker should have slapped the user for > redefining Set, so it's more of a academic problem and leaving "later" > vague doesn't matter in practise, but I'd still document the fact that > conforming implementations are free to choose what later means. > > > Right. This doesn't really bother me much. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150116/1f9a43dd/attachment-0001.html> From edk141 at gmail.com Fri Jan 16 23:00:52 2015 From: edk141 at gmail.com (Ed Kellett) Date: Fri, 16 Jan 2015 22:00:52 +0000 Subject: [Python-ideas] Better comand line version of python -c References: <CABmzr0iT675COwNEnKj+omFV081X_H__RGwmqhaMWixY=9ia6w@mail.gmail.com> <1628625043.4104139.1420791882196.JavaMail.yahoo@jws10043.mail.ne1.yahoo.com> Message-ID: <CABmzr0hZ-ogDTWparCadLw736E2TJULpHwyyx3UE2rXY3Ar1Kw@mail.gmail.com> On Fri Jan 09 2015 at 8:24:43 AM Andrew Barnert <abarnert at yahoo.com> wrote: > ... Sorry about leaving it so long to reply to this. It's occurred to me that using a daemon for the purpose of `_` is almost useless. One of the things that's been discussed is processing by iterators, which is obviously a good thing because it lets you process files that are much larger than your system memory, but an iterator that's already been iterated is no use as _. Either I'm missing something fundamental, or there's simply nothing we can do about this: caching the iterated results for later use would be non-trivial to get right, and use impossibly large amounts of memory. I still think this idea deserves some serious thought, but I think this particular feature just can't work. Ed Kellett -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150116/e18e6ce5/attachment.html> From mistersheik at gmail.com Fri Jan 16 23:20:03 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Fri, 16 Jan 2015 17:20:03 -0500 Subject: [Python-ideas] Float range class In-Reply-To: <C48EA2F0-5733-445B-ADA0-87F9EE12A282@yahoo.com> References: <CAP7+vJJ6qhoWCLi8e3er1bKSGCnYJfzpLpyLgH3RfnYQXhAVXg@mail.gmail.com> <528318575.153312.1420918218746.JavaMail.yahoo@jws10099.mail.ne1.yahoo.com> <CAP7+vJKQVmzc_+NvbFni-SawJdoMrVNBu8qyDtFVJCozP+3_gQ@mail.gmail.com> <54B19E30.9070706@stoneleaf.us> <EF17D2ED-4FDC-4CDD-81FD-220A7EBC3CE3@yahoo.com> <C48EA2F0-5733-445B-ADA0-87F9EE12A282@yahoo.com> Message-ID: <CAA68w_mzYq1-Y3mzxWb1PuvWahnNx=iESEhOEuH1sRyQ_K_Tig@mail.gmail.com> This looks really good. If you want to be consistent with range, line 21 should be: if i >= self.num or i < 0: because i could have been less than -self.num, which is an index error. I think you should support num == 0 and num == 1 because numpy's linspace supports it. Just set step to None in those cases and make sure you return start when indexing at 0 for num == 1. I think the rest of your code will work. Also, you should put the code through autopep8 as there are some style "problems" :) Best, Neil On Mon, Jan 12, 2015 at 5:51 PM, 'Andrew Barnert' via python-ideas < python-ideas at googlegroups.com> wrote: > Sorry, hit Send accidentally. > > The ActiveState recipe referenced is #579000; it's the same as the class I > posted on GitHub, but with only the NumPy implementation, rather than both > that and Guido's version. It also links to the two earlier recipes I could > find, and to NumPy docs and code, and the discussion section covers (I > hope) all the caveats and bikeshedding issues that came up in this thread; > the goal is that anyone following a link from the docs will find everything > they might need to know about. > > Sent from a random iPhone > > On Jan 12, 2015, at 14:44, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> > wrote: > > > On Jan 10, 2015, at 13:48, Ethan Furman <ethan at stoneleaf.us> wrote: > > > >> On 01/10/2015 11:50 AM, Guido van Rossum wrote: > >>> > >>> I suppose the briefest of recipes could go in the range() docs, in a > section "what do do for floats". More extended > >>> recipes should probably go on the activestate recipes site. > >> > >> The range docs do seem to be the obvious place for range recipes. > > > > I've created issue #23226, with a patch (the whole thing fits into one > paragraph and a short example, so I didn't think a subsection was > necessary, but that can easily be changed), and a link to a new ActiveState > recipe ( that contains the same simple class that turns the NumPy algorithm > into a lazy sequence. > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/H69Z7qPe0Dg/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150116/14414964/attachment.html> From encukou at gmail.com Fri Jan 16 23:37:01 2015 From: encukou at gmail.com (Petr Viktorin) Date: Fri, 16 Jan 2015 23:37:01 +0100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJL6u+v0tuDF-dv08F-Tj1KT+L_Xo7b5W0naFYR0Gpd+=A@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <CA+=+wqCbBm77f3uYnXmp8sobu+6gkMZU+_wcWY=mwHjELGwU9w@mail.gmail.com> <CAP7+vJL6u+v0tuDF-dv08F-Tj1KT+L_Xo7b5W0naFYR0Gpd+=A@mail.gmail.com> Message-ID: <CA+=+wqBofgoaOv-soahHv12VvooDXKGxRxhJ8JPTKYcC0BjovA@mail.gmail.com> On Fri, Jan 16, 2015 at 10:47 PM, Guido van Rossum <guido at python.org> wrote: > On Fri, Jan 16, 2015 at 1:08 PM, Petr Viktorin <encukou at gmail.com> wrote: >> >> "On Fri, Jan 16, 2015 at 6:17 PM, Guido van Rossum <guido at python.org> >> wrote: >> [...] >> > * Tuple, used as ``Tuple[index0_type, index1_type, ...]``. >> > Arbitrary-length tuples might be expressed using ellipsis, in which >> > case the following arguments are considered the same type as the last >> > defined type on the tuple. >> >> How would Tuple[...] or Tuple[t1, ..., t2] work? The PEP should say >> such shenanigans are undefined, rather than leave it to >> interpretation. > > > Good point. This needs to be elaborated. See > https://github.com/ambv/typehinting/issues/30. (Wanna submit a pull > request?) Yes. >> [...] >> > * WINDOWS >> > >> > * UNIXOID, equivalent to ``not WINDOWS`` >> >> I wonder what the use cases are here. Is this useful, as a dichotomy, >> outside of asyncio (or other low-level stdlib code)? > > > The stdlib is full of things that work differently on Windows. Consequently > application code will also often contain things like > > if <windows>: > <do it the Windows way> > else: > <do it the other way> > > Typically each branch can only be type-checked if the stdlib for that > particular platform is accessible. So it makes sense to standardize on a way > of spelling the test in a way that static checkers can understand. So "unixoid" just means "not Windows"? Any new platform Python will be ported to (looking at Micropython for example) is expected to be "unixoid"? Can it just be spelled ``not WINDOWS``? From rosuav at gmail.com Sat Jan 17 00:58:40 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 17 Jan 2015 10:58:40 +1100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CA+=+wqBofgoaOv-soahHv12VvooDXKGxRxhJ8JPTKYcC0BjovA@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <CA+=+wqCbBm77f3uYnXmp8sobu+6gkMZU+_wcWY=mwHjELGwU9w@mail.gmail.com> <CAP7+vJL6u+v0tuDF-dv08F-Tj1KT+L_Xo7b5W0naFYR0Gpd+=A@mail.gmail.com> <CA+=+wqBofgoaOv-soahHv12VvooDXKGxRxhJ8JPTKYcC0BjovA@mail.gmail.com> Message-ID: <CAPTjJmpRJKw4j6j-A9NUx8t4fhUeyg5a12iyPZCuoxjQj8S8VQ@mail.gmail.com> On Sat, Jan 17, 2015 at 9:37 AM, Petr Viktorin <encukou at gmail.com> wrote: > So "unixoid" just means "not Windows"? > Any new platform Python will be ported to (looking at Micropython for > example) is expected to be "unixoid"? > Can it just be spelled ``not WINDOWS``? I've only just now figured out that the "oid" here is the suffix meaning "similar/related to", and has nothing to do with object IDs or other uses of the term OID. Could it be spelled POSIXISH or UNIXISH or something? The -ish suffix doesn't collide with as many computer-related concepts. ChrisA From abarnert at yahoo.com Sat Jan 17 01:27:32 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 16 Jan 2015 16:27:32 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJKBfmkSx4JHyYvghABNKm_MKGoPhwaJHhkdHQxvCGz3kQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <4E76D308-4378-4895-8A0D-916706A24A9F@yahoo.com> <CAP7+vJKBfmkSx4JHyYvghABNKm_MKGoPhwaJHhkdHQxvCGz3kQ@mail.gmail.com> Message-ID: <AE63244A-1688-4357-A382-DC43F165276E@yahoo.com> On Jan 16, 2015, at 13:34, Guido van Rossum <guido at python.org> wrote: > On Fri, Jan 16, 2015 at 12:14 PM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: >> One quick clarification question: >> >> Can strings-as-forward-declarations be used inside non-string type annotations--e.g.. Set['Employee'] instead of 'Set[Employee]'? If so, do they have identical meaning? > > Yes. There's a FIXME in the PEP source to define forward references rigorously. See also https://github.com/ambv/typehinting/issues/34 OK, thanks. I see that MyPy already supports this (List['A'] is valid per Jukka's comment, and from a quick test it seems to be identical to 'List[A]'). >> I can imagine, say, a refactoring tool that didn't understand the full Python+MyPy syntax being able to do things with the former that it couldn't with the latter. And I think it might look more natural for simple recursive types. But those are just vague feelings, so don't take this as a suggestion. > > Well, the most common use case is where you are referencing the current class (which is in the process of being defined but isn't actually created until the entire class body is finished). E.g. > > class Node(Generic[T]): > def add_left(self, node: 'Node[T]'): ... Sure, but in that case you're using a generic class being defined, with a declared typevar as a parameter. It's the other way around that's interesting--when the class being defined is a parameter to a declared generic type, like: class Node: def descendants(self) -> Set['Node']: ... -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150116/c025d065/attachment.html> From ethan at stoneleaf.us Sat Jan 17 01:38:39 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 16 Jan 2015 16:38:39 -0800 Subject: [Python-ideas] Better comand line version of python -c In-Reply-To: <CABmzr0hZ-ogDTWparCadLw736E2TJULpHwyyx3UE2rXY3Ar1Kw@mail.gmail.com> References: <CABmzr0iT675COwNEnKj+omFV081X_H__RGwmqhaMWixY=9ia6w@mail.gmail.com> <1628625043.4104139.1420791882196.JavaMail.yahoo@jws10043.mail.ne1.yahoo.com> <CABmzr0hZ-ogDTWparCadLw736E2TJULpHwyyx3UE2rXY3Ar1Kw@mail.gmail.com> Message-ID: <54B9AF0F.7070707@stoneleaf.us> On 01/16/2015 02:00 PM, Ed Kellett wrote: > > It's occurred to me that using a daemon for the purpose of `_` is almost useless. One of the things that's been > discussed is processing by iterators Just because '_' isn't useful in the iterator case doesn't mean it's also useless for other situations. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150116/8029b70e/attachment.sig> From edk141 at gmail.com Sat Jan 17 01:47:56 2015 From: edk141 at gmail.com (Ed Kellett) Date: Sat, 17 Jan 2015 00:47:56 +0000 Subject: [Python-ideas] Better comand line version of python -c References: <CABmzr0iT675COwNEnKj+omFV081X_H__RGwmqhaMWixY=9ia6w@mail.gmail.com> <1628625043.4104139.1420791882196.JavaMail.yahoo@jws10043.mail.ne1.yahoo.com> <CABmzr0hZ-ogDTWparCadLw736E2TJULpHwyyx3UE2rXY3Ar1Kw@mail.gmail.com> <54B9AF0F.7070707@stoneleaf.us> Message-ID: <CABmzr0hEF7aGcH__GAZkjX9x0fv0Ce1KcJtfBknG83pROaSSvw@mail.gmail.com> Okay, maybe "useless" is a bit much, but the cases where _ would be most useful and those where you're likely to want to use iterators: you want to avoid processing a whole pile of data again. Also, the interface more or less has to make a choice between using iterators and lists, I think (at least for the "map python over this input data" case) - so either the default wouldn't work for large datasets, or _ wouldn't be useful. Ed Kellett On Sat Jan 17 2015 at 00:39:25 Ethan Furman <ethan at stoneleaf.us> wrote: > On 01/16/2015 02:00 PM, Ed Kellett wrote: > > > > It's occurred to me that using a daemon for the purpose of `_` is almost > useless. One of the things that's been > > discussed is processing by iterators > > Just because '_' isn't useful in the iterator case doesn't mean it's also > useless for other situations. > > -- > ~Ethan~ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150117/03795698/attachment.html> From abarnert at yahoo.com Sat Jan 17 01:49:19 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 16 Jan 2015 16:49:19 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJL6u+v0tuDF-dv08F-Tj1KT+L_Xo7b5W0naFYR0Gpd+=A@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <CA+=+wqCbBm77f3uYnXmp8sobu+6gkMZU+_wcWY=mwHjELGwU9w@mail.gmail.com> <CAP7+vJL6u+v0tuDF-dv08F-Tj1KT+L_Xo7b5W0naFYR0Gpd+=A@mail.gmail.com> Message-ID: <27C6533C-432B-4DBD-9AA6-CAF628B02F72@yahoo.com> On Jan 16, 2015, at 13:47, Guido van Rossum <guido at python.org> wrote: > On Fri, Jan 16, 2015 at 1:08 PM, Petr Viktorin <encukou at gmail.com> wrote: >> "On Fri, Jan 16, 2015 at 6:17 PM, Guido van Rossum <guido at python.org> wrote: >> [...] >> > * WINDOWS >> > >> > * UNIXOID, equivalent to ``not WINDOWS`` >> >> I wonder what the use cases are here. Is this useful, as a dichotomy, >> outside of asyncio (or other low-level stdlib code)? > > The stdlib is full of things that work differently on Windows. The stdlib is also full of things that work one way on Windows, another way on Unix, and another way (or, often, not at all) on (all, or specific) other platforms, I realize nobody is building Python 3.5 for classic Mac or Symbian S60 or MicroVMS, but do we need to bake in as a permanent assumption that Python will never again run on a non-Windows non-POSIX-ish system? You already have both WINDOWS and UNIXOID; why not just remove the statement that one is the negation of the other? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150116/9b1b5c26/attachment.html> From guido at python.org Sat Jan 17 02:21:46 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Jan 2015 17:21:46 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAPTjJmpRJKw4j6j-A9NUx8t4fhUeyg5a12iyPZCuoxjQj8S8VQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <CA+=+wqCbBm77f3uYnXmp8sobu+6gkMZU+_wcWY=mwHjELGwU9w@mail.gmail.com> <CAP7+vJL6u+v0tuDF-dv08F-Tj1KT+L_Xo7b5W0naFYR0Gpd+=A@mail.gmail.com> <CA+=+wqBofgoaOv-soahHv12VvooDXKGxRxhJ8JPTKYcC0BjovA@mail.gmail.com> <CAPTjJmpRJKw4j6j-A9NUx8t4fhUeyg5a12iyPZCuoxjQj8S8VQ@mail.gmail.com> Message-ID: <CAP7+vJJhPkX+RJx4yugOi_t1evav4P3B0ArQJ=wph5A7a02t6Q@mail.gmail.com> On Fri, Jan 16, 2015 at 3:58 PM, Chris Angelico <rosuav at gmail.com> wrote: > On Sat, Jan 17, 2015 at 9:37 AM, Petr Viktorin <encukou at gmail.com> wrote: > > So "unixoid" just means "not Windows"? > > Any new platform Python will be ported to (looking at Micropython for > > example) is expected to be "unixoid"? > > Can it just be spelled ``not WINDOWS``? > > I've only just now figured out that the "oid" here is the suffix > meaning "similar/related to", and has nothing to do with object IDs or > other uses of the term OID. Could it be spelled POSIXISH or UNIXISH or > something? The -ish suffix doesn't collide with as many > computer-related concepts. > The current plan is to spell it POSIX. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150116/5ef8a2ea/attachment-0001.html> From guido at python.org Sat Jan 17 02:24:44 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Jan 2015 17:24:44 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <27C6533C-432B-4DBD-9AA6-CAF628B02F72@yahoo.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <CA+=+wqCbBm77f3uYnXmp8sobu+6gkMZU+_wcWY=mwHjELGwU9w@mail.gmail.com> <CAP7+vJL6u+v0tuDF-dv08F-Tj1KT+L_Xo7b5W0naFYR0Gpd+=A@mail.gmail.com> <27C6533C-432B-4DBD-9AA6-CAF628B02F72@yahoo.com> Message-ID: <CAP7+vJJ_AJ_c2bXAXNUgdedo+4W-EvAXuTT3CKswcBmm+_qVEA@mail.gmail.com> On Fri, Jan 16, 2015 at 4:49 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > On Jan 16, 2015, at 13:47, Guido van Rossum <guido at python.org> wrote: > > On Fri, Jan 16, 2015 at 1:08 PM, Petr Viktorin <encukou at gmail.com> wrote: > >> "On Fri, Jan 16, 2015 at 6:17 PM, Guido van Rossum <guido at python.org> >> wrote: >> > [...] >> > * WINDOWS >> > >> > * UNIXOID, equivalent to ``not WINDOWS`` >> >> I wonder what the use cases are here. Is this useful, as a dichotomy, >> outside of asyncio (or other low-level stdlib code)? >> > > The stdlib is full of things that work differently on Windows. > > > The stdlib is also full of things that work one way on Windows, another > way on Unix, and another way (or, often, not at all) on (all, or specific) > other platforms, > > I realize nobody is building Python 3.5 for classic Mac or Symbian S60 > or MicroVMS, but do we need to bake in as a permanent assumption that > Python will never again run on a non-Windows non-POSIX-ish system? > > You already have both WINDOWS and UNIXOID; why not just remove the > statement that one is the negation of the other? > Because that's going to be the assumption that people will make anyway. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150116/4ba46582/attachment.html> From tjreedy at udel.edu Sat Jan 17 02:45:16 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 16 Jan 2015 20:45:16 -0500 Subject: [Python-ideas] pep 397 In-Reply-To: <CACPPHzsuATV4Gkn2fCE-P3Tr4frjQ9TuEATU5cg6q3vzR9pS0w@mail.gmail.com> References: <CACPPHzsuATV4Gkn2fCE-P3Tr4frjQ9TuEATU5cg6q3vzR9pS0w@mail.gmail.com> Message-ID: <m9cerb$tst$1@ger.gmane.org> On 12/30/2014 3:31 PM, Liam Marsh wrote: > /the pep 397 says that any python script is able to choose the language > version which will run it, between all the versions installed on the > computer, using on windows a launcher in the "C:\windows" folder./ > > can the idle version be chosen like this too, or can the idle "run" > command do it? Question about using python, including Idle, should be directed to python-list. Quick answers to what seem to be separate questions. 1. At the command line, one select the version of a module to run by selecting the version of python to run. Idle is no different from any other module intended to be directly run. For me, C:\Users\Terry>py -2 -m idlelib.idle # starts Idle in 2.7.9 C:\Users\Terry>py -3 -m idlelib # starts Idle in 3.4.2 On Windows, I pin the Start -> Pythonx.y -> Idle x.y icons to my task bar so I can select either python + idle combination with a mouse click. 2. Selecting F5 or Run-module in an Idle editor runs the editor buffer contents with the same version of python as is running Idle. I have considered adding 'Run with x.y' options according to what is installed, but that is pie-in-the-sky at the moment. -- Terry Jan Reedy From cfkaran2 at gmail.com Sat Jan 17 02:57:41 2015 From: cfkaran2 at gmail.com (Cem Karan) Date: Fri, 16 Jan 2015 20:57:41 -0500 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> Message-ID: <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> On Jan 16, 2015, at 4:40 PM, Guido van Rossum <guido at python.org> wrote: > On Fri, Jan 16, 2015 at 12:34 PM, Dennis Brakhane <brakhane at googlemail.com> wrote: > Am 16.01.2015 um 20:45 schrieb Guido van Rossum: > > That's discussed in this section: > > https://www.python.org/dev/peps/pep-0484/#compatibility-with-other-uses-of-function-annotations > > . > I still feel that saying "if you use function annotations, that function > cannot be statically checked" is bad. > > If type hints are successful, many projects will adopt them, especially > big projects and/or projects in corporations. Probably many will require > that all code must pass the type checker, and (at least) public > functions be type annotated. > > Let's suppose that indeed happens. If I now were writing a new open > source Python library, I would want that my library could be used in big > projects, and therefore would have an incentive to type annotate my > library. If I don't, a big project might choose another lib instead that > does. > > So in that future, a library that wants to be successful basically can't > use type annotations, even if they would have been an elegant solution > to a particular problem. That feels somewhat wrong to me. > > A possible work around using skeletons seems ugly, you would have to > edit two different files and keep them in sync. > > I feel there should be another way to make type annotations and > "creative" annotations coexist. > > The cleanest way to do this would be to use decorators for the non-type-hint use case. > > > > > The dict notation (which was proposed in an earlier thread on > > python-ideas) looks too messy to seriously support. > How about using a tuple? I'll admit that I've proposed it before, but > got no reaction. > > If an annotation is a tuple, the type checker would check each element > for known values, and every other thing that uses __annotations__ could > do the same. Stefan's example could look something like > > def func(x: (str, wobble(False), ctype('std::string[utf8]'))) -> (str, > wobble(True)) > > It's still not pretty, but it has the advantage that it is only needed > for when more than one annotation is used. > > It's very ambiguous though. Especially since plain strings are taken to be type annotations (used for forward references). > > It has the disadvantage of preventing (int,str) being an alias for > Tuple[int,str]. > > But we're not proposing that (overloading native container syntax for types was proposed and rejected early on). > > Actually, I don't care that much what particular solution is used to > allow type annotations and non-type annotations to coexist, but I find > it important that is possible. > > Decorators. If we're using decorators for the annotations, why not use them for the typing as well? E.g., @typing.type(a, int) @doc.doc(a, "some doc") def f(a): pass This would be the equivalent of: def f(a: {"typing.type": int, "doc.doc": "some doc"}): pass The whole dictionary idea will be hidden, and annotations can be used for other purposes while still supporting typing information. I'll admit that stacking the decorators means the code could get very deep, but it still feels like the least bad thing to do. Thanks, Cem Karan From guido at python.org Sat Jan 17 03:11:14 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Jan 2015 18:11:14 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> Message-ID: <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> On Fri, Jan 16, 2015 at 5:57 PM, Cem Karan <cfkaran2 at gmail.com> wrote: > > If we're using decorators for the annotations, why not use them for the > typing as well? Because annotations were intended for type hints. > E.g., > > @typing.type(a, int) > @doc.doc(a, "some doc") > def f(a): > pass > > This would be the equivalent of: > > def f(a: {"typing.type": int, "doc.doc": "some doc"}): > pass > > The whole dictionary idea will be hidden, and annotations can be used for > other purposes while still supporting typing information. I'll admit that > stacking the decorators means the code could get very deep, but it still > feels like the least bad thing to do. > You're taking this "nobody should be more equal than others" thing too far. :-) We're trying to come up with a standard way of doing one very important thing. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150116/bd0ad3a2/attachment-0001.html> From flying-sheep at web.de Fri Jan 16 21:05:44 2015 From: flying-sheep at web.de (Philipp A.) Date: Fri, 16 Jan 2015 20:05:44 +0000 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> Message-ID: <CAN8d9gnvftkuaX56k7RA_G_xh_iY5vTJd+0+aW6O_v_y=XmtMQ@mail.gmail.com> hi, nice work! I like most of the ideas, except the semantic comments and some specific abstract classes. so criticism first: 1. i don?t think comments should ever have an impact on the code. i?d prefer ?assert isinstance(...)? for type annotations on variables. 2. also i don?t see why Dict, (non-abstract) Set, and List exist as long as we have Mapping and so on. Tuple is different because of its ?Multiple return value? status and status as ?is treated as Union[Class1, Class2] when used with instanceof(), except and friends?. but checking for instanceof(..., list) is unpytonic! imho List and Dict really go against python?s data model and protocol-based nature. i?d be for consistency: Set and MutableSet, Sequence and MutableSequence, Mapping and MutableMapping. no ?abstract? (because all of this is abstract), no ?frozen? (because frozen sets are just immutable sets and immutable behavior is a subset of the corresponding mutable behavior). Questions: 1. will we be able to use Union[...] in places like an except clause, where tuples are used to mean the same right now? would make sense imho 2. what are ItemsView and KeysView for? what makes them different from FrozenSet or Sequence? Guido van Rossum <guido at python.org> schrieb am Fri Jan 16 2015 at 18:19:24: > After a long editing process we've got PEP 484 > <https://www.python.org/dev/peps/pep-0484/> (Type Hints) ready for your > review. This is by no means final, and several areas are either open (to be > resolved in a later draft) or postponed (to a different PEP altogether). > But there's enough meat that I think we can start having the discussion. > Please also see PEP 483 <https://www.python.org/dev/peps/pep-0483/> (The > Theory of Type Hint; copied and reformatted from the original Quip document > that I posted just before last Christmas) and PEP 482 > <https://www.python.org/dev/peps/pep-0482/> (Literature Overview for Type > Hints, by ?ukasz). Those are informational PEPs though; the actual spec is > focused in PEP 484 (the only one on the Standards Track). > > As I said earlier, I hope to have a rough consensus before PyCon > <https://us.pycon.org/2015/> and working code (just the typing.py module > <https://github.com/ambv/typehinting/tree/master/prototyping>, really) > committed to CPython before the last 3.5 alpha > <https://www.python.org/dev/peps/pep-0478/>. > > Here is the raw text of PEP 484. Fire away!! > > PEP: 484 > Title: Type Hints > Version: $Revision$ > Last-Modified: $Date$ > Author: Guido van Rossum <guido at python.org>, Jukka Lehtosalo < > jukka.lehtosalo at iki.fi>, ?ukasz Langa <lukasz at langa.pl> > Discussions-To: Python-Dev <python-dev at python.org> > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 29-Sep-2014 > Post-History: 16-Jan-2015 > Resolution: > > > Abstract > ======== > > This PEP introduces a standard syntax for type hints using annotations > on function definitions. > > The proposal is strongly inspired by mypy [mypy]_. > > The theory behind type hints and gradual typing is explained in PEP 483. > > > Rationale and Goals > =================== > > PEP 3107 added support for arbitrary annotations on parts of a function > definition. Although no meaning was assigned to annotations then, there > has always been an implicit goal to use them for type hinting, which is > listed as the first possible use case in said PEP. > > This PEP aims to provide a standard syntax for type annotations, opening > up Python code to easier static analysis and refactoring, potential > runtime type checking, and performance optimizations utilizing type > information. > > > Type Definition Syntax > ====================== > > The syntax leverages PEP 3107-style annotations with a number of > extensions described in sections below. In its basic form, type hinting > is used by filling function annotations with classes:: > > def greeting(name: str) -> str: > return 'Hello ' + name > > This denotes that the expected type of the ``name`` argument is ``str``. > Analogically, the expected return type is ``str``. Subclasses of > a specified argument type are also accepted as valid types for that > argument. > > Abstract base classes, types available in the ``types`` module, and > user-defined classes may be used as type hints as well. Annotations > must be valid expressions that evaluate without raising exceptions at > the time the function is defined. In addition, the needs of static > analysis require that annotations must be simple enough to be > interpreted by static analysis tools. (This is an intentionally > somewhat vague requirement.) > > .. FIXME: Define rigorously what is/isn't supported. > > When used as an annotation, the expression ``None`` is considered > equivalent to ``NoneType`` (i.e., ``type(None)`` for type hinting > purposes. > > Type aliases are also valid type hints:: > > integer = int > > def retry(url: str, retry_count: integer): ... > > New names that are added to support features described in following > sections are available in the ``typing`` package. > > > Callbacks > --------- > > Frameworks expecting callback functions of specific signatures might be > type hinted using ``Callable[[Arg1Type, Arg2Type], ReturnType]``. > Examples:: > > from typing import Any, AnyArgs, Callable > > def feeder(get_next_item: Callable[[], Item]): ... > > def async_query(on_success: Callable[[int], None], on_error: > Callable[[int, Exception], None]): ... > > def partial(func: Callable[AnyArgs, Any], *args): ... > > Since using callbacks with keyword arguments is not perceived as > a common use case, there is currently no support for specifying keyword > arguments with ``Callable``. > > > Generics > -------- > > Since type information about objects kept in containers cannot be > statically inferred in a generic way, abstract base classes have been > extended to support subscription to denote expected types for container > elements. Example:: > > from typing import Mapping, Set > > def notify_by_email(employees: Set[Employee], overrides: Mapping[str, > str]): ... > > Generics can be parametrized by using a new factory available in > ``typing`` called ``TypeVar``. Example:: > > from typing import Sequence, TypeVar > > T = TypeVar('T') # Declare type variable > > def first(l: Sequence[T]) -> T: # Generic function > return l[0] > > In this case the contract is that the returning value is consistent with > the elements held by the collection. > > ``TypeVar`` supports constraining parametric types to classes with any of > the specified bases. Example:: > > from typing import Iterable > > X = TypeVar('X') > Y = TypeVar('Y', Iterable[X]) > > def filter(rule: Callable[[X], bool], input: Y) -> Y: > ... > > .. FIXME: Add an example with multiple bases defined. > > In the example above we specify that ``Y`` can be any subclass of > Iterable with elements of type ``X``, as long as the return type of > ``filter()`` will be the same as the type of the ``input`` > argument. > > .. FIXME: Explain more about how this works. > > > Forward references > ------------------ > > When a type hint contains names that have not been defined yet, that > definition may be expressed as a string, to be resolved later. For > example, instead of writing:: > > def notify_by_email(employees: Set[Employee]): ... > > one might write:: > > def notify_by_email(employees: 'Set[Employee]'): ... > > .. FIXME: Rigorously define this. Defend it, or find an alternative. > > > Union types > ----------- > > Since accepting a small, limited set of expected types for a single > argument is common, there is a new special factory called ``Union``. > Example:: > > from typing import Union > > def handle_employees(e: Union[Employee, Sequence[Employee]]): > if isinstance(e, Employee): > e = [e] > ... > > A type factored by ``Union[T1, T2, ...]`` responds ``True`` to > ``issubclass`` checks for ``T1`` and any of its subclasses, ``T2`` and > any of its subclasses, and so on. > > One common case of union types are *optional* types. By default, > ``None`` is an invalid value for any type, unless a default value of > ``None`` has been provided in the function definition. Examples:: > > def handle_employee(e: Union[Employee, None]): ... > > As a shorthand for ``Union[T1, None]`` you can write ``Optional[T1]``; > for example, the above is equivalent to:: > > from typing import Optional > > def handle_employee(e: Optional[Employee]): ... > > An optional type is also automatically assumed when the default value is > ``None``, for example:: > > def handle_employee(e: Employee = None): ... > > This is equivalent to:: > > def handle_employee(e: Optional[Employee] = None): ... > > .. FIXME: Is this really a good idea? > > A special kind of union type is ``Any``, a class that responds > ``True`` to ``issubclass`` of any class. This lets the user > explicitly state that there are no constraints on the type of a > specific argument or return value. > > > Platform-specific type checking > ------------------------------- > > In some cases the typing information will depend on the platform that > the program is being executed on. To enable specifying those > differences, simple conditionals can be used:: > > from typing import PY2, WINDOWS > > if PY2: > text = unicode > else: > text = str > > def f() -> text: ... > > if WINDOWS: > loop = ProactorEventLoop > else: > loop = UnixSelectorEventLoop > > Arbitrary literals defined in the form of ``NAME = True`` will also be > accepted by the type checker to differentiate type resolution:: > > DEBUG = False > ... > if DEBUG: > class Tracer: > <verbose implementation> > else: > class Tracer: > <dummy implementation> > > For the purposes of type hinting, the type checker assumes ``__debug__`` > is set to ``True``, in other words the ``-O`` command-line option is not > used while type checking. > > > Compatibility with other uses of function annotations > ----------------------------------------------------- > > A number of existing or potential use cases for function annotations > exist, which are incompatible with type hinting. These may confuse a > static type checker. However, since type hinting annotations have no > run time behavior (other than evaluation of the annotation expression > and storing annotations in the ``__annotations__`` attribute of the > function object), this does not make the program incorrect -- it just > makes it issue warnings when a static analyzer is used. > > To mark portions of the program that should not be covered by type > hinting, use the following: > > * a ``@no_type_checks`` decorator on classes and functions > > * a ``# type: ignore`` comment on arbitrary lines > > .. FIXME: should we have a module-wide comment as well? > > > Type Hints on Local and Global Variables > ======================================== > > No first-class syntax support for explicitly marking variables as being > of a specific type is added by this PEP. To help with type inference in > complex cases, a comment of the following format may be used:: > > x = [] # type: List[Employee] > > In the case where type information for a local variable is needed before > if was declared, an ``Undefined`` placeholder might be used:: > > from typing import Undefined > > x = Undefined # type: List[Employee] > y = Undefined(int) > > If type hinting proves useful in general, a syntax for typing variables > may be provided in a future Python version. > > > Explicit raised exceptions > ========================== > > No support for listing explicitly raised exceptions is being defined by > this PEP. Currently the only known use case for this feature is > documentational, in which case the recommendation is to put this > information in a docstring. > > > The ``typing`` package > ====================== > > To open the usage of static type checking to Python 3.5 as well as older > versions, a uniform namespace is required. For this purpose, a new > package in the standard library is introduced called ``typing``. It > holds a set of classes representing builtin types with generics, namely: > > * Dict, used as ``Dict[key_type, value_type]`` > > * List, used as ``List[element_type]`` > > * Set, used as ``Set[element_type]``. See remark for ``AbstractSet`` > below. > > * FrozenSet, used as ``FrozenSet[element_type]`` > > * Tuple, used as ``Tuple[index0_type, index1_type, ...]``. > Arbitrary-length tuples might be expressed using ellipsis, in which > case the following arguments are considered the same type as the last > defined type on the tuple. > > It also introduces factories and helper members needed to express > generics and union types: > > * Any, used as ``def get(key: str) -> Any: ...`` > > * Union, used as ``Union[Type1, Type2, Type3]`` > > * TypeVar, used as ``X = TypeVar('X', Type1, Type2, Type3)`` or simply > ``Y = TypeVar('Y')`` > > * Undefined, used as ``local_variable = Undefined # type: List[int]`` or > ``local_variable = Undefined(List[int])`` (the latter being slower > during runtime) > > * Callable, used as ``Callable[[Arg1Type, Arg2Type], ReturnType]`` > > * AnyArgs, used as ``Callable[AnyArgs, ReturnType]`` > > * AnyStr, equivalent to ``TypeVar('AnyStr', str, bytes)`` > > All abstract base classes available in ``collections.abc`` are > importable from the ``typing`` package, with added generics support: > > * ByteString > > * Callable > > * Container > > * Hashable > > * ItemsView > > * Iterable > > * Iterator > > * KeysView > > * Mapping > > * MappingView > > * MutableMapping > > * MutableSequence > > * MutableSet > > * Sequence > > * Set as ``AbstractSet``. This name change was required because ``Set`` > in the ``typing`` module means ``set()`` with generics. > > * Sized > > * ValuesView > > * Mapping > > The library includes literals for platform-specific type hinting: > > * PY2 > > * PY3, equivalent to ``not PY2`` > > * WINDOWS > > * UNIXOID, equivalent to ``not WINDOWS`` > > The following types are available in the ``typing.io`` module: > > * IO > > * BinaryIO > > * TextIO > > The following types are provided by the ``typing.re`` module: > > * Match and Pattern, types of ``re.match()`` and ``re.compile()`` > results > > As a convenience measure, types from ``typing.io`` and ``typing.re`` are > also available in ``typing`` (quoting Guido, "There's a reason those > modules have two-letter names."). > > > The place of the ``typing`` module in the standard library > ---------------------------------------------------------- > > .. FIXME: complete this section > > > Usage Patterns > ============== > > The main use case of type hinting is static analysis using an external > tool without executing the analyzed program. Existing tools used for > that purpose like ``pyflakes`` [pyflakes]_ or ``pylint`` [pylint]_ > might be extended to support type checking. New tools, like mypy's > ``mypy -S`` mode, can be adopted specifically for this purpose. > > Type checking based on type hints is understood as a best-effort > mechanism. In other words, whenever types are not annotated and cannot > be inferred, the type checker considers such code valid. Type errors > are only reported in case of explicit or inferred conflict. Moreover, > as a mechanism that is not tied to execution of the code, it does not > affect runtime behaviour. In other words, even in the case of a typing > error, the program will continue running. > > The implementation of a type checker, whether linting source files or > enforcing type information during runtime, is out of scope for this PEP. > > .. FIXME: Describe stub modules. > > .. FIXME: Describe run-time behavior of generic types. > > > Existing Approaches > =================== > > PEP 482 lists existing approaches in Python and other languages. > > > Is type hinting Pythonic? > ========================= > > Type annotations provide important documentation for how a unit of code > should be used. Programmers should therefore provide type hints on > public APIs, namely argument and return types on functions and methods > considered public. However, because types of local and global variables > can be often inferred, they are rarely necessary. > > The kind of information that type hints hold has always been possible to > achieve by means of docstrings. In fact, a number of formalized > mini-languages for describing accepted arguments have evolved. Moving > this information to the function declaration makes it more visible and > easier to access both at runtime and by static analysis. Adding to that > the notion that ?explicit is better than implicit?, type hints are > indeed *Pythonic*. > > > Acknowledgements > ================ > > This document could not be completed without valuable input, > encouragement and advice from Jim Baker, Jeremy Siek, Michael Matson > Vitousek, Andrey Vlasovskikh, and Radomir Dopieralski. > > Influences include existing languages, libraries and frameworks > mentioned in PEP 482. Many thanks to their creators, in alphabetical > order: Stefan Behnel, William Edwards, Greg Ewing, Larry Hastings, > Anders Hejlsberg, Alok Menghrajani, Travis E. Oliphant, Joe Pamer, > Raoul-Gabriel Urma, and Julien Verlaguet. > > > References > ========== > > .. [mypy] > http://mypy-lang.org > > .. [pyflakes] > https://github.com/pyflakes/pyflakes/ > > .. [pylint] > http://www.pylint.org > > > Copyright > ========= > > This document has been placed in the public domain. > > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150116/6a09da7c/attachment-0001.html> From tjreedy at udel.edu Sat Jan 17 05:09:16 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 16 Jan 2015 23:09:16 -0500 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> Message-ID: <m9cn9c$i3b$1@ger.gmane.org> On 1/12/2015 12:02 PM, Chris Barker wrote: > Now that we're talking about floating point conveniences (math.nan, > linspace): > > What about putting an > > almost_equal(x,y,tol=1e14) > > (or close(), or...) in the math module. > > AFAICT, in the standard library, there is only: > > unittest.TestCase.assertAlmostEqual As near as I can tell, assertAlmostEqual(first, second, places=7, msg=None, delta=None) does one of two possible absolute difference checks (round(first, places) - round(second, places)) == 0.0 abs(first - second) < = delta There has been discussion of making it more complicated, but I remember that being rejected because other tests require more thought and can be implemented with the above anyway. > but it: > > A) is buried in the unittest.TestCase class > > B) is an assertion, so you can't use it as a general test (easily) > > C) uses number of decimal digits or an absolute delta, but does not > provide a significant figures comparison, which is most likely what's > wanted (and a bit harder to write yourself) assertAlmostEqual((a-b)/d, 0, delta = tol) where d is a, b, and (a+b)/2 as one thinks is appropriate. > numpy provides allclose() According to Neil Girdhar, absolute(/a/ - /b/) <= (/atol/ + /rtol/ * absolute(/b/)) which I presume means, in Python, abs(a-b) <= atol + rtol * abs(b) where atol and rtol are assume >= 0.0 > (and isclose() ), which is pretty much what I'm suggesting. > > Anyone else think this would be a good idea to add to the stdlib? I am somewhat skeptical as there is no universal right answer (see below). But I think allclose is about as good as we could get, maybe with some special casing added. The discussion on the thread seems mostly divorced from the multiple use cases. What do each of a and b represent? Different numbers? or successive approximations of the same number? Are they theoretical 'exact' numbers, approximations of theoretical numbers, or calculations from measurements with error? If the latter, how big is the error? And why are we asking anyway? We are usually asking 'close enough' for some purpose, but purposes differ. Consider the problem of finding the (unique) 0 crossing (root) of a monotonic function f. One is looking for a value x* such that f(x*) is 'near' 0. (Someone claimed that 'nothing is close to zero'. This is nonsensical both in applied math and everyday life.) A standard approach is to compute successive approximations until one finds such a point. But unthinking application of such a method may not get one the desired result. The following are two examples with opposite problems. I once had the job of writing code to find the root of a monotonic function whose parameters were calculated from experimental data (multiple counts). Fortunately, it was known that the unique root (a probability) had to be between 0.0 and 1.0 (endpoints excluded). The function was often kinked (L-shaped) so that the slope at the root could be arbitrarily large. That meant that for a small enough absolute tolerance, there might be no float x such that x would be even 'near' 0, let alone equal to 0. And at other values, the slope could be so close to 0 as to make standard Newton iteration useless. I ended up avoiding float comparison by simply doing 20 steps of binary search, starting with 0 and 1 as the initial 'approximations', and stopping. The resulting 7 or 8 decimal digit approximation was more than good enough for realistic data. On the other hand, consider f(x) = x**9, Because f is so flat at its root (0) f(float) evaluates to -+0.0 for floats in a range of at least -1e-36, 1e-36. Is any number in this range 'good enough' as the root? Or is more work needed to find a value near the middle of the '= 0' range? -- Terry Jan Reedy From steve at pearwood.info Sat Jan 17 06:49:26 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 17 Jan 2015 16:49:26 +1100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> Message-ID: <20150117054925.GH18556@ando.pearwood.info> Very nice! Some comments/questions below. On Fri, Jan 16, 2015 at 09:17:36AM -0800, Guido van Rossum wrote: > Type Definition Syntax > ====================== [...] > An optional type is also automatically assumed when the default value is > ``None``, for example:: > > def handle_employee(e: Employee = None): ... > > This is equivalent to:: > > def handle_employee(e: Optional[Employee] = None): ... > > .. FIXME: Is this really a good idea? I think it is. It seems to me to be a form of type inference: you are declaring that e is an Employee, but inferring that it could also be None. Either the type system reports this as an error, or it infers that the type is intended to be optional. Given how common it is for arguments to be "X or None" having shorthand for that is a good thing. > Platform-specific type checking > ------------------------------- > > In some cases the typing information will depend on the platform that > the program is being executed on. To enable specifying those > differences, simple conditionals can be used:: Since CPython isn't going to do type-checking itself, instead it will leave it up to external tools. Given that, I assume that you are specifying the minimum requirements for such a tool, rather than a declaration of what some specific tool will do. Correct? Hence, type-checkers should be able to deal with "simple conditionals" as in the examples given, and are not required to emulate a full Python interpreter in order to make sense of arbitrarily complex conditionals like: if (PY2 and WINDOWS) or (PY3 and POSIX): ... But a sophisticated type-checker may offer more than the minimum. Is this correct? There are a number of places where the PEP mentions "the type checker", e.g. the next paragraph. I think the PEP should clarify what exactly is meant by that. > Arbitrary literals defined in the form of ``NAME = True`` will also be > accepted by the type checker to differentiate type resolution:: As above. How arbitrary are the literals? Only bool flags? If so, the PEP should say "Boolean literals" rather than "arbitrary literals". > Compatibility with other uses of function annotations > ----------------------------------------------------- > > A number of existing or potential use cases for function annotations > exist, which are incompatible with type hinting. These may confuse a > static type checker. However, since type hinting annotations have no > run time behavior (other than evaluation of the annotation expression > and storing annotations in the ``__annotations__`` attribute of the > function object), this does not make the program incorrect -- it just > makes it issue warnings when a static analyzer is used. I understood from early discussions that type checking would only be enabled if you imported `typing`. If so, then the easy way to disable type checks for a whole module was not to import anything from `typing`. > To mark portions of the program that should not be covered by type > hinting, use the following: > > * a ``@no_type_checks`` decorator on classes and functions > > * a ``# type: ignore`` comment on arbitrary lines > > .. FIXME: should we have a module-wide comment as well? Rather than have to comment each and every line, can we marking entire sections with a directive? # type: off ... ... # type: on ... This will still allow `type: ignore` if you want to skip a single line, and allow module-wide control by simply starting your module with `type: off` and not turning it back on. Where will the @no_type_checks decorator live? In the typing module or as a builtin? I see from below that directives of the form `type: spam` are allowed. That could clash with `type: ignore` etc. Although it is not PEP 8 compliant, people might have classes called "ignore", "on", "off". How about we require keywords to the type directive to have a leading + sign? # type: +ignore # Keyword "ignore", don't type check this line. # type: ignore # Declare the type is class "ignore". Otherwise there is going to be the risk of clashes. > Type Hints on Local and Global Variables > ======================================== > > No first-class syntax support for explicitly marking variables as being > of a specific type is added by this PEP. To help with type inference in > complex cases, a comment of the following format may be used:: > > x = [] # type: List[Employee] Without the type declaration, x would be inferred to be of type List[Any]. Is that correct? Perhaps that should go into the PEP. > In the case where type information for a local variable is needed before > if was declared, an ``Undefined`` placeholder might be used:: ^^^^^ Typo, should read "it was declared". -- Steven From steve at pearwood.info Sat Jan 17 07:06:59 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 17 Jan 2015 17:06:59 +1100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAN8d9gnvftkuaX56k7RA_G_xh_iY5vTJd+0+aW6O_v_y=XmtMQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <CAN8d9gnvftkuaX56k7RA_G_xh_iY5vTJd+0+aW6O_v_y=XmtMQ@mail.gmail.com> Message-ID: <20150117060658.GI18556@ando.pearwood.info> On Fri, Jan 16, 2015 at 08:05:44PM +0000, Philipp A. wrote: > 1. i don?t think comments should ever have an impact on the code. > i?d prefer ?assert isinstance(...)? for type annotations on variables. But that isn't a compile-time type declaration, that is a runtime type check. The idea is to avoid runtime type checks by having assurance that by the time your code runs, the variable cannot be anything but the appropriate type. (The truly paranoid may choose to use both, but I expect most people won't.) Type declarations of any form are a kind of comment. They don't have a runtime effect (except incidentally), although they have a compile-time effect, and they are written for the human reader as much as for the compiler. So in this case, rather than inventing new syntax for compiler instructions, we're just using comments of a particular form which the type-checker can understand. This is an old and conventional choice, e.g. Pascal (and I expect many other languages) used comments of the form {$ ... } as directives to the compiler. [...] > i?d be for consistency: Set and MutableSet, Sequence and MutableSequence, > Mapping and MutableMapping. no ?abstract? (because all of this is > abstract), no ?frozen? (because frozen sets are just immutable sets and > immutable behavior is a subset of the corresponding mutable behavior). frozenset is not a subclass of set. It cannot be, without violating the Liskov Substitution Principle: there are things you can do with sets which you cannot do with frozensets, hence you cannot substitute frozensets for sets. -- Steven From steve at pearwood.info Sat Jan 17 07:51:19 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 17 Jan 2015 17:51:19 +1100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> Message-ID: <20150117065119.GJ18556@ando.pearwood.info> On Fri, Jan 16, 2015 at 08:57:41PM -0500, Cem Karan wrote: > If we're using decorators for the annotations, why not use them for > the typing as well? E.g., > > @typing.type(a, int) > @doc.doc(a, "some doc") > def f(a): > pass > > This would be the equivalent of: > > def f(a: {"typing.type": int, "doc.doc": "some doc"}): > pass Decorators are a second-class solution to the annotation problem, because they require you to repeat the variable name in at least two places. In the past I have strongly defended the idea that type-hinting annotations need to co-exist with other uses for annotations. By this I mean that there should be a simple way for the author of a module who uses annotations for something else to flag that module, or parts of the module, so that the type-checker skips it. There is no good way to have multiple uses of annotations be used in the same function. You have suggested using a dict, but what of the module that wants to give a completely different meaning to dicts as annotations? Your interpetation of annotations is not compatible with that module. Like multiple inheritence, multiple use of annotations is only possible if all parties cooperate and agree on semantics. Besides, the more information you try to squeeze into the function parameter list, the more unwieldy, unreadable and ugly it gets. Type-hinting was Guido's original motivation for introducing annotations, and there is no other use of annotations which has become popular or widespread enough to justify calling it a "standard use". So in the absense of any other standard use of annotations, I am comfortable with giving type-hints special status: - whether type-checking is enabled by default or not, we understand that annotations are primarily for type-hinting, and any annotations are to be understood as type-hints by default; - other uses for annotations are permitted, but it is up to the user to flag the module/class/function so the type-checker skips it; - no provision is made for mixing type-hints and arbitrary other uses in the same annotation (in other words, if you want to use annotations for type-hinting and Foo *at the same time*, you're on your own); - but of course, nothing stops you from creating your own custom annotation scheme that includes Foo + type-hints. If some other use of annotations becomes wildly successful, then it may be worth rethinking the special status of type-hinting in the future. If you want to interoperate with type-hinting, decorators may be a second-class solution, but they are a solution. -- Steve From stefan_ml at behnel.de Sat Jan 17 10:43:28 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 17 Jan 2015 10:43:28 +0100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> Message-ID: <m9das1$vtc$1@ger.gmane.org> Guido van Rossum schrieb am 16.01.2015 um 21:08: > I don't know what to answer for Cython -- doesn't it have its own syntax > for declaring C types? Or is it now also using annotations? It can, but it's optional. When I added support for it, I took care not to enable it by default, so that it wouldn't accidentally interfere with other usages of annotations. The current proposal seems less generous in this regard, and the way I see it, the only gain from Cython's point of view is its very limited support for container item type declarations and function signatures of callable objects. Both are restricted to Python object types and even imply slightly different semantics than Cython (I didn't see a notion of "exact types" as opposed to "type or subtype"). If there was at least a notion of "unknown types" that tools like static type checkers MUST ignore (couldn't find that in the PEP either), then Cython users could mix this with Cython's Python level type mapping through the "cython" magic module, e.g. something like "cython.struct(x=cython.int, y=cython.p_char)". OTOH, Cython could easily understand the proposed type declarations and apply runtime type checks for them to compiled Python code. Not sure if that would be considered an interesting feature, though. Stefan From stefan_ml at behnel.de Sat Jan 17 10:58:01 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 17 Jan 2015 10:58:01 +0100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <m9das1$vtc$1@ger.gmane.org> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> Message-ID: <m9dbna$cnn$1@ger.gmane.org> Stefan Behnel schrieb am 17.01.2015 um 10:43: > Guido van Rossum schrieb am 16.01.2015 um 21:08: >> I don't know what to answer for Cython -- doesn't it have its own syntax >> for declaring C types? Or is it now also using annotations? > > It can, but it's optional. When I added support for it, I took care not to > enable it by default, so that it wouldn't accidentally interfere with other > usages of annotations. > > The current proposal seems less generous in this regard, and the way I see > it, the only gain from Cython's point of view is its very limited support > for container item type declarations and function signatures of callable > objects. Both are restricted to Python object types and even imply slightly > different semantics than Cython (I didn't see a notion of "exact types" as > opposed to "type or subtype"). > > If there was at least a notion of "unknown types" that tools like static > type checkers MUST ignore (couldn't find that in the PEP either), then > Cython users could mix this with Cython's Python level type mapping through > the "cython" magic module, e.g. something like "cython.struct(x=cython.int, > y=cython.p_char)". BTW, I think namespaces naturally fix this issue. If all of the proposed type system lives either in the builtin namespace or in the "typing" module, then annotations and type hints from any other module namespace MUST simply be ignored by tools that don't understand them, and are also easy to ignore. Then you could write "x: typing.List[cython.int]", and static type checkers could still validate that all input for "x" is a list (of some unknown item type), and Cython could enforce that anything coming out of such a list at runtime can coerce to a C "int". Stefan From steve at pearwood.info Sat Jan 17 11:01:18 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 17 Jan 2015 21:01:18 +1100 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <m9cn9c$i3b$1@ger.gmane.org> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <m9cn9c$i3b$1@ger.gmane.org> Message-ID: <20150117100117.GK18556@ando.pearwood.info> On Fri, Jan 16, 2015 at 11:09:16PM -0500, Terry Reedy wrote: > On 1/12/2015 12:02 PM, Chris Barker wrote: > >but it: > > > >A) is buried in the unittest.TestCase class > > > >B) is an assertion, so you can't use it as a general test (easily) > > > >C) uses number of decimal digits or an absolute delta, but does not > >provide a significant figures comparison, which is most likely what's > >wanted (and a bit harder to write yourself) > > assertAlmostEqual((a-b)/d, 0, delta = tol) > where d is a, b, and (a+b)/2 as one thinks is appropriate. That does nothing to solve problems A) and B), and I'm dubious that it provides a "significant figures comparison" (whatever that means, I'm pretty sure it doesn't mean "relative error", which is what you're calculating in a round-about fashion). > >numpy provides allclose() > > According to Neil Girdhar, > absolute(/a/ - /b/) <= (/atol/ + /rtol/ * absolute(/b/)) > which I presume means, in Python, > abs(a-b) <= atol + rtol * abs(b) > where atol and rtol are assume >= 0.0 Adding the error tolerances together is a dubious thing to do. I don't understand the reasoning between that. Given two values a and b, there are two definitions of the error between them: absolute = abs(a - b) relative = abs(a - b)/abs(b) [Note: I'm sticking to numpy's unconditional use of "b" for the denominator, which is not symmetric. In actuality, I would use min(abs(a), abs(b)) for the denominator.] In the case where we only specify absolute or relative tolerance, it is obvious what to do: calculate the appropriate error, and if it is less than the given tolerance, return True: def approx_equal(a, b, allowed_absolute_error, allowed_relative_error): # For simplicity, ignore NANs, INFs, and assume b != 0 actual_error = abs(a - b) if allowed_absolute_error is None: # Only perform a check on relative error. return actual_error <= allowed_relative_error*abs(b) elif allowed_relative_error is None: # Only perform a check on absolute error. return actual_error <= allowed_absolute_error else: # We have specified *both* abs and rel error. How should we handle the third case? Two obvious ways come to mind: require that *both* individual tests pass: return (actual_error <= allowed_relative_error*abs(b) and actual_error <= allowed_absolute_error) # equivalent to: # actual_error <= max(allowed_relative_error*abs(b), # allowed_absolute_error) or require that *either* test pass: return (actual_relative_error <= allowed_relative_error or actual_absolute_error <= allowed_absolute_error) # equivalent to: # actual_error <= min( ... ) But what numpy does is to add the tolerances together, that is, it uses *twice* the average of them, equivalent to this: allowed_error = ( allowed_absolute_error + allowed_relative_error*abs(b) ) return actual_absolute_error <= allowed_error This means that numpy will claim that two numbers are close even though *both* the absolute and relative error tests fail: py> numpy.allclose([1.2], [1.0], 0.0, 0.1) # Fails absolute error test. False py> numpy.allclose([1.2], [1.0], 0.1, 0.0) # Fails relative error test. False py> numpy.allclose([1.2], [1.0], 0.1, 0.1) # Passes! True I cannot think of a good justification for that. Either I am missing something, or this goes to show that numpy can mess up even something as simple and straightforward as an error calculation. If I'm right, that's further evidence that getting this "right" and putting it in the standard library is a good thing to do. [...] > Consider the problem of finding the (unique) 0 crossing (root) of a > monotonic function f. One is looking for a value x* such that f(x*) is > 'near' 0. (Someone claimed that 'nothing is close to zero'. This is > nonsensical both in applied math and everyday life.) It isn't nonsensical, it just needs to be understood in context of relative errors. All non-zero numbers are infinitely far from zero in terms of relative error. > A standard > approach is to compute successive approximations until one finds such a > point. But unthinking application of such a method may not get one the > desired result. The following are two examples with opposite problems. [snip anecdote] I didn't think that the well-known difficulties in root-finding has anything to do with the usefulness of a standard way to compare numbers for approximate equality. -- Steven From stefan_ml at behnel.de Sat Jan 17 11:44:13 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 17 Jan 2015 11:44:13 +0100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <m9dbna$cnn$1@ger.gmane.org> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <m9dbna$cnn$1@ger.gmane.org> Message-ID: <m9dedu$stk$1@ger.gmane.org> Stefan Behnel schrieb am 17.01.2015 um 10:58: > Stefan Behnel schrieb am 17.01.2015 um 10:43: >> Guido van Rossum schrieb am 16.01.2015 um 21:08: >>> I don't know what to answer for Cython -- doesn't it have its own syntax >>> for declaring C types? Or is it now also using annotations? >> >> It can, but it's optional. When I added support for it, I took care not to >> enable it by default, so that it wouldn't accidentally interfere with other >> usages of annotations. >> >> The current proposal seems less generous in this regard, and the way I see >> it, the only gain from Cython's point of view is its very limited support >> for container item type declarations and function signatures of callable >> objects. Both are restricted to Python object types and even imply slightly >> different semantics than Cython (I didn't see a notion of "exact types" as >> opposed to "type or subtype"). >> >> If there was at least a notion of "unknown types" that tools like static >> type checkers MUST ignore (couldn't find that in the PEP either), then >> Cython users could mix this with Cython's Python level type mapping through >> the "cython" magic module, e.g. something like "cython.struct(x=cython.int, >> y=cython.p_char)". > > BTW, I think namespaces naturally fix this issue. If all of the proposed > type system lives either in the builtin namespace or in the "typing" > module, then annotations and type hints from any other module namespace > MUST simply be ignored by tools that don't understand them, and are also > easy to ignore. Then you could write "x: typing.List[cython.int]", and > static type checkers could still validate that all input for "x" is a list > (of some unknown item type), and Cython could enforce that anything coming > out of such a list at runtime can coerce to a C "int". https://github.com/ambv/typehinting/pull/44 Stefan From flying-sheep at web.de Sat Jan 17 12:41:46 2015 From: flying-sheep at web.de (Philipp A.) Date: Sat, 17 Jan 2015 11:41:46 +0000 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <CAN8d9gnvftkuaX56k7RA_G_xh_iY5vTJd+0+aW6O_v_y=XmtMQ@mail.gmail.com> <CAP7+vJL5Ap22XKgOTu13cNZXOG0TYG+BsQUSdx7=RKCHv5ydBg@mail.gmail.com> Message-ID: <CAN8d9gnQHgeeDt-2BPcn_RFfsKomzeXUkd=7o5SnN6WT4rHfBQ@mail.gmail.com> Guido van Rossum <guido at python.org> schrieb am Fri Jan 16 2015 at 22:31:11: > i don?t think comments should ever have an impact on the code. >> > > They don't. They also change the errors emitted by the static checker > (which is not invoked when you run the code -- it is like a linter). > still, comments are in my eyes purely documentational. the only exception in python i can think of, doctest, only exists to ensure that no examples you give start breaking, i.e. doctests are ran to keep the *comments* up to date, not the code. and annotations are a syntax-level element. using comments forces the linter to use an alternative AST with comments preserved, not the AST CPython gives you for free. and i?m looking into the future: python interpreters/VMs could choose to optimize e.g. List[double] based on type annotations, instead of just linting. in which case comments would become part of the code. > i?d prefer ?assert isinstance(...)? for type annotations on variables. > > But that *would* affect runtime, by slowing down execution. Also for > generic types the type checking could be either very slow (if checking > List[int] would check every item of a list) or always fail (if a type > variable is involved). > > See discussion at https://github.com/ambv/typehinting/issues/35 > optimized package builds strip out assertions, no? and if course isinstance([1], List[int]) shouldn?t fail. that would be very surprising! i?ll check out the issue. > But there is a lot of code that really needs the concrete types. E.g. > list+list returns a list, but sequence+sequence may fail (as list+tuple > exemplifies). Or list.sort. Etc. > i commented in the issue: we could specify that behavior in new ABCs or even add it as mixin methods. nothing prevents MutableSequence, which has .reverse, from having .sort, as well. will we be able to use Union[...] in places like an except clause, where >> tuples are used to mean the same right now? would make sense imho >> > > No. Remember TOOWTDI. (Also the PEP intentionally stays far away from > exception handling.) > yeah, i just though that isinstance(foo, (str, bytes)) basically means ?is foo an instance of either str or bytes?, or in other words ?is foo an instance of the union type str|bytes?. the one way to do it would have been Union[str, bytes] if python had Union from the beginning, i?m sure! what are ItemsView and KeysView for? what makes them different from >> FrozenSet or Sequence? >> > > They exist in collections.abc. Note that KeysView and ItemsView are > (abstract) Sets, but ValuesView is not (you can have multiple equal values). > ah, get it! they provide mixins powered by suppying a to-be-wrapped Mapping instance to MappingView?s __init__! this should be documented, else nobody knows *how* they mix in their methods. > --Guido van Rossum (python.org/~guido) > thanks for the explanations, and as always: thanks for this most beautiful language :D -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150117/9755174a/attachment.html> From brakhane at googlemail.com Sat Jan 17 17:14:04 2015 From: brakhane at googlemail.com (Dennis Brakhane) Date: Sat, 17 Jan 2015 17:14:04 +0100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> Message-ID: <54BA8A4C.5060508@googlemail.com> The PEP does not mention MyPy style casts. Is this intentional? If yes, are casts not supported at all, or is there just no cast in typing and people should use type: comments instead? x = [1,2,3] # type: Iterable[int] y = x # type: Sequence[int] From ethan at stoneleaf.us Sat Jan 17 18:08:24 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 17 Jan 2015 09:08:24 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <m9dbna$cnn$1@ger.gmane.org> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <m9dbna$cnn$1@ger.gmane.org> Message-ID: <54BA9708.6080608@stoneleaf.us> On 01/17/2015 01:58 AM, Stefan Behnel wrote: > Stefan Behnel schrieb am 17.01.2015 um 10:43: >> Guido van Rossum schrieb am 16.01.2015 um 21:08: >>> >>> I don't know what to answer for Cython -- doesn't it have its own syntax >>> for declaring C types? Or is it now also using annotations? >> >> It can, but it's optional. When I added support for it, I took care not to >> enable it by default, so that it wouldn't accidentally interfere with other >> usages of annotations. >> >> The current proposal seems less generous in this regard, and the way I see >> it, the only gain from Cython's point of view is its very limited support >> for container item type declarations and function signatures of callable >> objects. Both are restricted to Python object types and even imply slightly >> different semantics than Cython (I didn't see a notion of "exact types" as >> opposed to "type or subtype"). >> >> If there was at least a notion of "unknown types" that tools like static >> type checkers MUST ignore (couldn't find that in the PEP either), then >> Cython users could mix this with Cython's Python level type mapping through >> the "cython" magic module, e.g. something like "cython.struct(x=cython.int, >> y=cython.p_char)". > > BTW, I think namespaces naturally fix this issue. If all of the proposed > type system lives either in the builtin namespace or in the "typing" > module, then annotations and type hints from any other module namespace > MUST simply be ignored by tools that don't understand them, and are also > easy to ignore. Then you could write "x: typing.List[cython.int]", and > static type checkers could still validate that all input for "x" is a list > (of some unknown item type), and Cython could enforce that anything coming > out of such a list at runtime can coerce to a C "int". +1 -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150117/0a269e62/attachment.sig> From tjreedy at udel.edu Sun Jan 18 00:36:26 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 17 Jan 2015 18:36:26 -0500 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> Message-ID: <m9erlr$smk$1@ger.gmane.org> On 1/16/2015 4:40 PM, Guido van Rossum wrote: > On Fri, Jan 16, 2015 at 12:34 PM, Dennis Brakhane > I feel there should be another way to make type annotations and > "creative" annotations coexist. > > The cleanest way to do this would be to use decorators for the > non-type-hint use case. I have come to agree with this as the least bad solution. From a runtime perspective, annotation syntax is merely a nice way to attach to a function an attribute whose value is a dict whose keys are function and parameter names. Decorators can do the same (and have been able to do so since introduced). Repeating (and quoting) names in the decorator is a nuisance, but as a reader, I would prefer that to having multiple annotation systems mixed together. The decorator solution, with custom attribute names, scales better to multiple annotation systems. -- Terry Jan Reedy From tjreedy at udel.edu Sun Jan 18 02:18:36 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 17 Jan 2015 20:18:36 -0500 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> Message-ID: <m9f1ld$hjb$1@ger.gmane.org> On 1/16/2015 12:17 PM, Guido van Rossum wrote: > The following types are available in the ``typing.io > <http://typing.io>`` module: > The following types are provided by the ``typing.re <http://typing.re>`` > module: > As a convenience measure, types from ``typing.io <http://typing.io>`` > and ``typing.re <http://typing.re>`` are Something is over-eager in seeing urls and adding http: web links. -- Terry Jan Reedy From cfkaran2 at gmail.com Sun Jan 18 03:09:01 2015 From: cfkaran2 at gmail.com (Cem Karan) Date: Sat, 17 Jan 2015 21:09:01 -0500 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> Message-ID: <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> On Jan 16, 2015, at 9:11 PM, Guido van Rossum <guido at python.org> wrote: > On Fri, Jan 16, 2015 at 5:57 PM, Cem Karan <cfkaran2 at gmail.com> wrote: > >> If we're using decorators for the annotations, why not use them for the typing as well? > > Because annotations were intended for type hints. > >> E.g., >> >> @typing.type(a, int) >> @doc.doc(a, "some doc") >> def f(a): >> pass >> >> This would be the equivalent of: >> >> def f(a: {"typing.type": int, "doc.doc": "some doc"}): >> pass >> >> The whole dictionary idea will be hidden, and annotations can be used for other purposes while still supporting typing information. I'll admit that stacking the decorators means the code could get very deep, but it still feels like the least bad thing to do. > > You're taking this "nobody should be more equal than others" thing too far. :-) We're trying to come up with a standard way of doing one very important thing. I understand what you're saying, but I feel like annotations have the capability to be MUCH more useful than just as a place to hold type hints. My example is fairly small, but it feels cleaner than some of the other solutions (e.g., Sphinx's :param: & :type: statements, see http://sphinx-doc.org/domains.html#info-field-lists). I just don't want to close the door to other uses, which is what will happen; as others have said, if type hinting becomes the officially blessed use for annotations, then other uses will be effectively deprecated, even if it is technically possible to continue using them for other purposes. Here is another way (this is a hack and requires more thought): """ # In the typing library class Type(object): def __init__(self, t): self._t = t @property def t(self): return self._t # End of class Type # In use: def foo(a: Type(int)): pass """ I haven't thought too much more about it, but I'm sure someone else can come up with a better way of doing it. The basic idea is that the name of the class and the fact that the argument is the type in question is always known; a static analyzer can scan for it fairly easily. If the type of the annotation is not Type, then you know something else is going on, and the analyzer can skip it. This avoids the question of 'is this dict the type, or is there something inside of it that also includes the type?' This can be further extended using decorators that generate dynamic checking code in debug mode, and are no-ops in normal mode (accessible by a command-line switch or something). I'm not sure at them moment how to handle having both Type objects and other information. I suspect having a catch-all *args, **kwargs in the init is going to be the least bad solution, but other ideas are welcome! Thanks, Cem Karan From ben+python at benfinney.id.au Sun Jan 18 03:27:08 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 18 Jan 2015 13:27:08 +1100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> Message-ID: <851tmsu8lf.fsf@benfinney.id.au> Cem Karan <cfkaran2 at gmail.com> writes: > [?] I feel like annotations have the capability to be MUCH more useful > than just as a place to hold type hints. They have that capability. How long do you propose we wait for this capability to be realised? Function annotations have been part of Python since version 3.0, and Guido (IIUC) is deciding that the wait is over: we already have the best use of function annotations we're going to get. -- \ ?I went to the hardware store and bought some used paint. It | `\ was in the shape of a house.? ?Steven Wright | _o__) | Ben Finney From guido at python.org Sun Jan 18 04:51:15 2015 From: guido at python.org (Guido van Rossum) Date: Sat, 17 Jan 2015 19:51:15 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <20150117054925.GH18556@ando.pearwood.info> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <20150117054925.GH18556@ando.pearwood.info> Message-ID: <CAP7+vJKi-q7xszSKXtOizGZL9e+nM8SxVeWCA_yD_97CN6-9hw@mail.gmail.com> On Fri, Jan 16, 2015 at 9:49 PM, Steven D'Aprano <steve at pearwood.info> wrote: > > On Fri, Jan 16, 2015 at 09:17:36AM -0800, Guido van Rossum wrote: > > > Type Definition Syntax > > ====================== > [...] > > An optional type is also automatically assumed when the default value is > > ``None``, for example:: > > > > def handle_employee(e: Employee = None): ... > > > > This is equivalent to:: > > > > def handle_employee(e: Optional[Employee] = None): ... > > > > .. FIXME: Is this really a good idea? > > I think it is. It seems to me to be a form of type inference: you are > declaring that e is an Employee, but inferring that it could also be > None. Either the type system reports this as an error, or it infers that > the type is intended to be optional. Given how common it is for > arguments to be "X or None" having shorthand for that is a good thing. > Agreed. I'm removing the FIXME. > > Platform-specific type checking > > ------------------------------- > > > > In some cases the typing information will depend on the platform that > > the program is being executed on. To enable specifying those > > differences, simple conditionals can be used:: > > Since CPython isn't going to do type-checking itself, instead it > will leave it up to external tools. Given that, I > assume that you are specifying the minimum requirements for such a > tool, rather than a declaration of what some specific tool will do. > Correct? > Yes. A type checker is an immensely complex tool, and there are many gray areas where the PEP just can't hope to be rigorous. In the end the use of the type checker is meant as a linter, not as the rigorous definition of the language (which can do things no type checker can understand). > Hence, type-checkers should be able to deal with "simple conditionals" > as in the examples given, and are not required to emulate a full Python > interpreter in order to make sense of arbitrarily complex conditionals > like: > > if (PY2 and WINDOWS) or (PY3 and POSIX): ... > That already feels too complex -- not because the checker couldn't figure out the value, but because I don't want to support complex expressions in other places (before you know it, you're using conditionals in argument annotations, and I really don't want to go there). > But a sophisticated type-checker may offer more than the minimum. > Is this correct? > Sure. > There are a number of places where the PEP mentions "the type checker", > e.g. the next paragraph. I think the PEP should clarify what exactly is > meant by that. > You're right. There's some language in the "Usage Patterns" sections but it makes more sense to explain this upfront. (Also that section hints at runtime checking, which really should be considered quite out of scope.) > > Arbitrary literals defined in the form of ``NAME = True`` will also be > > accepted by the type checker to differentiate type resolution:: > > As above. How arbitrary are the literals? Only bool flags? If so, the > PEP should say "Boolean literals" rather than "arbitrary literals". > Agreed. I think we need to work with Jukka on a spec of what the minimum complexity is that the checker should support for this purpose (even though in other contexts it of course must understand non-constant expressions, since it must type-check them :-). > > > > Compatibility with other uses of function annotations > > ----------------------------------------------------- > > > > A number of existing or potential use cases for function annotations > > exist, which are incompatible with type hinting. These may confuse a > > static type checker. However, since type hinting annotations have no > > run time behavior (other than evaluation of the annotation expression > > and storing annotations in the ``__annotations__`` attribute of the > > function object), this does not make the program incorrect -- it just > > makes it issue warnings when a static analyzer is used. > > I understood from early discussions that type checking would only be > enabled if you imported `typing`. If so, then the easy way to disable > type checks for a whole module was not to import anything from `typing`. > I changed my mind about that (and I don't think mypy has ever strictly followed this rule). The reason is that it's quite sensible to use built-in types (e.g. int) and user-defined classes as type annotations and expect the checker to do its thing. It's only when you need things like generic classes, unions etc. that you must import typing. > > To mark portions of the program that should not be covered by type > > hinting, use the following: > > > > * a ``@no_type_checks`` decorator on classes and functions > > > > * a ``# type: ignore`` comment on arbitrary lines > > > > .. FIXME: should we have a module-wide comment as well? > > Rather than have to comment each and every line, can we marking entire > sections with a directive? > > # type: off > ... > ... > # type: on > ... > > > This will still allow `type: ignore` if you want to skip a single line, > and allow module-wide control by simply starting your module with > `type: off` and not turning it back on. > We don't have the definitive design. See https://github.com/ambv/typehinting/issues/16 and https://github.com/ambv/typehinting/issues/35 . > Where will the @no_type_checks decorator live? In the typing module or > as a builtin? > In the typing module. > I see from below that directives of the form `type: spam` are allowed. > That could clash with `type: ignore` etc. Although it is not PEP 8 > compliant, people might have classes called "ignore", "on", "off". How > about we require keywords to the type directive to have a leading + > sign? > > # type: +ignore # Keyword "ignore", don't type check this line. > # type: ignore # Declare the type is class "ignore". > I'm not keen on the special meaning of +. We might using "# typing: ..." for directives, or we might just use some all_caps names, e.g. "# type: IGNORE". > Otherwise there is going to be the risk of clashes. > I think the risk of clashes can be made sufficiently low that we can say "tough luck". > > > > Type Hints on Local and Global Variables > > ======================================== > > > > No first-class syntax support for explicitly marking variables as being > > of a specific type is added by this PEP. To help with type inference in > > complex cases, a comment of the following format may be used:: > > > > x = [] # type: List[Employee] > > Without the type declaration, x would be inferred to be of type > List[Any]. Is that correct? Perhaps that should go into the PEP. > Correct. (What else could it be? Although I think that if x has been given a type earlier, the [] is checked against that type.) If you think the PEP is insufficiently clear, could you submit a PR? (BTW, to anyone else considering PRs against the GitHub repo, please create a regular issue when you want to change the proposed behavior or syntax. PRs are good for clarifications, typos etc.) > > In the case where type information for a local variable is needed before > > if was declared, an ``Undefined`` placeholder might be used:: > ^^^^^ > > Typo, should read "it was declared". > Will fix. (I also think it should be "it is declared".) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150117/539c1503/attachment-0001.html> From guido at python.org Sun Jan 18 05:11:34 2015 From: guido at python.org (Guido van Rossum) Date: Sat, 17 Jan 2015 20:11:34 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <m9das1$vtc$1@ger.gmane.org> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> Message-ID: <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> On Sat, Jan 17, 2015 at 1:43 AM, Stefan Behnel <stefan_ml at behnel.de> wrote: > Guido van Rossum schrieb am 16.01.2015 um 21:08: > > I don't know what to answer for Cython -- doesn't it have its own syntax > > for declaring C types? Or is it now also using annotations? > > It can, but it's optional. When I added support for it, I took care not to > enable it by default, so that it wouldn't accidentally interfere with other > usages of annotations. > > The current proposal seems less generous in this regard, and the way I see > it, the only gain from Cython's point of view is its very limited support > for container item type declarations and function signatures of callable > objects. Both are restricted to Python object types and even imply slightly > different semantics than Cython (I didn't see a notion of "exact types" as > opposed to "type or subtype"). > Actually, while the PEP is still lacking in clarity and we're still having some discussions, type variables will most likely be "invariant" by default, which I think is the "exact types" notion you are after. There's lots of discussion in https://github.com/ambv/typehinting/issues/2 (you might want to read this from the end :-). > If there was at least a notion of "unknown types" that tools like static > type checkers MUST ignore (couldn't find that in the PEP either), then > Cython users could mix this with Cython's Python level type mapping through > the "cython" magic module, e.g. something like "cython.struct(x=cython.int > , > y=cython.p_char)". > Perhaps you could supply a stub for the cython module that declares everything in it as type 'Any'? That should shut the type checker up. > OTOH, Cython could easily understand the proposed type declarations and > apply runtime type checks for them to compiled Python code. Not sure if > that would be considered an interesting feature, though. > You should ask Cython's users what they think. For me personally, it has taken me a long time to make peace with the idea that the static type checker runs as a separate program, like a linter, and has no effect on the runtime behavior of the program. But now that I have made that peace, I feel it's easier to make progress. I no longer feel the pressure to design a type system that can describe everything Python can do (which is impossible). It's enough to design a type system that can describe most of what most programmers do with Python most of the time (when they're not feeling too hacky). As long as there's an escape to say "don't worry about this part," such a type checker can be a very useful tool (like a linter, but much better; it can also be used in an IDE). (And in fact there are several escape hatches. You could choose not to run the checker at all, or you could ignore some or all of its advice. Or you can leave out annotations from parts of your program. Or you can use an explicit 'Any' annotation. ('Any' is not the same as 'object'!) Or you can use a decorator or a magic comment to disable checking of code that uses annotations for other purposes.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150117/0d09fbcf/attachment.html> From guido at python.org Sun Jan 18 05:14:01 2015 From: guido at python.org (Guido van Rossum) Date: Sat, 17 Jan 2015 20:14:01 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <m9dbna$cnn$1@ger.gmane.org> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <m9dbna$cnn$1@ger.gmane.org> Message-ID: <CAP7+vJJ87rLu8VUGCDXORndiotwYM-B9K9tS9xULo4FCFVmumA@mail.gmail.com> On Sat, Jan 17, 2015 at 1:58 AM, Stefan Behnel <stefan_ml at behnel.de> wrote: > BTW, I think namespaces naturally fix this issue. If all of the proposed > type system lives either in the builtin namespace or in the "typing" > module, then annotations and type hints from any other module namespace > MUST simply be ignored by tools that don't understand them, and are also > easy to ignore. Then you could write "x: typing.List[cython.int]", and > static type checkers could still validate that all input for "x" is a list > (of some unknown item type), and Cython could enforce that anything coming > out of such a list at runtime can coerce to a C "int". > As I responded to your previous message, I think this can indeed be solved very elegantly by providing a stub module for cython that equates cython.int with typing.Any. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150117/e8e4c2a6/attachment.html> From guido at python.org Sun Jan 18 05:17:23 2015 From: guido at python.org (Guido van Rossum) Date: Sat, 17 Jan 2015 20:17:23 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <54BA8A4C.5060508@googlemail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <54BA8A4C.5060508@googlemail.com> Message-ID: <CAP7+vJLWXs64_V+yLf6L=XqDNd+j6vS9jg-kpefvEzBQp8Ujkg@mail.gmail.com> On Sat, Jan 17, 2015 at 8:14 AM, Dennis Brakhane <brakhane at googlemail.com> wrote: > The PEP does not mention MyPy style casts. Is this intentional? > No, we ran out of time for the first draft (and ?ukasz needed some pushing to like them :-). The conclusion in https://github.com/ambv/typehinting/issues/15 is that we will add it. > If yes, are casts not supported at all, or is there just no cast in typing > and people should use type: comments instead? > > x = [1,2,3] # type: Iterable[int] > y = x # type: Sequence[int] > > The issue I linked to also explains the difference (it's subtle!). If someone wants to give describing cast() in the PEP a shot, send a PR! -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150117/e5af6665/attachment.html> From chris.barker at noaa.gov Sun Jan 18 05:18:04 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Sat, 17 Jan 2015 20:18:04 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <20150117100117.GK18556@ando.pearwood.info> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <m9cn9c$i3b$1@ger.gmane.org> <20150117100117.GK18556@ando.pearwood.info> Message-ID: <-5782896342222828374@unknownmsgid> > provides a "significant figures comparison" (whatever that means, Poor choice of words on my part -- that is not a clearly defined term. I meant relative difference comparison, which does, more or less, indicate that two values are the same to within N significant figures. If you set the relative tolerance to 1e-6 then the values are "close" to about the first 6 decimal digits. >> According to Neil Girdhar, >> absolute(/a/ - /b/) <= (/atol/ + /rtol/ * absolute(/b/)) >> which I presume means, in Python, >> abs(a-b) <= atol + rtol * abs(b) >> where atol and rtol are assume >= 0.0 > > Adding the error tolerances together is a dubious thing to do. I don't > understand the reasoning between that. It ends up making the absolute tolerance essentially the "minimum tolerance". If the scaled relative tolerance is much larger than the absolute tolerance, it makes no difference. Likewise the other way around for small relative tolerance. It does change the result when the two are about the same magnitude, but this is all an order of magnitude thing anyway -- a factor of two doesn't really change the result. On a phone right now, but I guess that it was implemented that way to make it easy to vectorize with numpy arrays. If checks are a pain to vectorize. I think it's an OK way to do it , but very bad idea for the absolute tolerance to default to anything but zero. And really, it makes more sense to specify either absolute or relative, rather that mixing them. The numpy approach goes to heck for values smaller than zero. Essentially you need to know the scale to set the absolute tolerance anyway, killing the point of a relative tolerance. > I cannot think of a good justification for that. Either I am missing > something, See above > or this goes to show that numpy can mess up even something as > simple and straightforward as an error calculation. I'm not going to say that it messed up--but I certainly don't think that we should feel any obligation to do it the same way. > If I'm right, that's > further evidence that getting this "right" and putting it in the > standard library is a good thing to do. I do think this takes some thought, so good to put it in. But this thread also makes it clear that there a lot of ways to define "close", and which is best is use-case dependent. I think a straightforward relative tolerance would be usefull enough in many cases, and maybe an absolute one would be good for completeness. But I can see the argument that if there isn't one available, then folks are forced to think for themselves about what "closeness" means in their case, and be les likely to simply use the one provided when it might not be appropriate. (I need to go back and check some of my tests, for instance, I'm sure I didn't always think about the numpy abs_ tol parameter carefully!) > I didn't think that the well-known difficulties in root-finding has > anything to do with the usefulness of a standard way to compare numbers > for approximate equality. Agreed -- termination criteria of numerical methods is not the use case for a generic is_close implementation. That clearly requires a use-case specific criteria. From chris.barker at noaa.gov Sun Jan 18 06:37:36 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Sat, 17 Jan 2015 21:37:36 -0800 Subject: [Python-ideas] Fwd: Way to check for floating point "closeness"? In-Reply-To: <CALGmxE+B5A8E3djzEmv_WwQa2gMMnKhhaX325xyq83mNpFRJvA@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <m9cn9c$i3b$1@ger.gmane.org> <CALGmxE+B5A8E3djzEmv_WwQa2gMMnKhhaX325xyq83mNpFRJvA@mail.gmail.com> Message-ID: <CALGmxEL9vFYmmw=mkDpLR+UmBGcmHCQ_tAPburGu0ak_Ymrfkg@mail.gmail.com> oops, forgot ot reply-all ---------- Forwarded message ---------- From: Chris Barker <chris.barker at noaa.gov> Date: Sat, Jan 17, 2015 at 8:38 PM Subject: Re: [Python-ideas] Way to check for floating point "closeness"? To: Terry Reedy <tjreedy at udel.edu> On Fri, Jan 16, 2015 at 8:09 PM, Terry Reedy <tjreedy at udel.edu> wrote: > As near as I can tell, > assertAlmostEqual(first, second, places=7, msg=None, delta=None) > does one of two possible absolute difference checks > (round(first, places) - round(second, places)) == 0.0 > abs(first - second) < = delta > > There has been discussion of making it more complicated, but I remember > that being rejected because other tests require more thought and can be > implemented with the above anyway. > anything could be implemented with a simple absolute tolerance method, and for that matter, that is simple enough that there's not much point in having a built in function for it.anted (and a bit harder to write yourself) > > assertAlmostEqual((a-b)/d, 0, delta = tol) > where d is a, b, and (a+b)/2 as one thinks is appropriate. > while yopu're at it, why not jsut write your own assertion: assert (a-b)/b <= tol that function buys you nothing but potential confusion. > The discussion on the thread seems mostly divorced from the multiple use > cases. What do each of a and b represent? Different numbers? or > successive approximations of the same number? Are they theoretical 'exact' > numbers, approximations of theoretical numbers, or calculations from > measurements with error? If the latter, how big is the error? And why are > we asking anyway? We are usually asking 'close enough' for some purpose, > but purposes differ. > yes, they sure do -- but anything in the standard library would be for general use, where the details don't matter too much -- if you are doing something really specific, write your own test. > (Someone claimed that 'nothing is close to zero'. This is nonsensical > both in applied math and everyday life.) I'm pretty sure someone (more than one of use) asserted that "nothing is *relatively* close to zero -- very different. And indeed, that is still a primary stumbling block (at least for me) to putting a relative tolerance function into the stdlib: it will work fine in most cases, and be easy to understand and reason about, but it will not work if one of the input values is zero, and casual users may get tripped up by that. But at this point, i think that just needs note in the docs -- after all, they are likely to get tripped up by that if they write the code themselves anyway. And I really wanted a way to have a default behavior that would do a reasonable transition to an absolute tolerance near zero, but I no longer thing that's possible. (numpy's implimentaion kind of does that, but it is really wrong for small numbers, and if you made the default min_tolerance the smallest possible representable number, it really wouldn't be useful. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150117/d646510d/attachment.html> From cfkaran2 at gmail.com Sun Jan 18 14:37:09 2015 From: cfkaran2 at gmail.com (Cem Karan) Date: Sun, 18 Jan 2015 08:37:09 -0500 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <851tmsu8lf.fsf@benfinney.id.au> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> Message-ID: <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> On Jan 17, 2015, at 9:27 PM, Ben Finney <ben+python at benfinney.id.au> wrote: > Cem Karan <cfkaran2 at gmail.com> writes: > >> [?] I feel like annotations have the capability to be MUCH more useful >> than just as a place to hold type hints. > > They have that capability. How long do you propose we wait for this > capability to be realised? > > Function annotations have been part of Python since version 3.0, and > Guido (IIUC) is deciding that the wait is over: we already have the best > use of function annotations we're going to get. I understand what you're saying, and you're right that they are not used much. I only learned about them after I started playing with Python 3.3. But then I'm the bleeding edge for my group at work; everyone else is down in the python 2 world somewhere (some of them are WAY down there somewhere). Changing a language takes time. Do you program in C? Do you still use pthreads? Why? C11 specifies the thread.h header, you should switch to C11 threads right now! Except... many compilers still don't fully support C11, so we're coding to an earlier standard. There is a lot of code that already works as it is. Etc., etc. etc... When a language changes, there is a LOT of catching up to do. First, tools need to support the new facilities and then programmers need to become aware of new facilities, and they have to realize there is a legitimate use for those facilities. Most people I know are unaware that annotations exist; that is an argument for making the change to supporting typing only, simply because there will be fewer people that are affected. At the same time, if someone has a really good idea that could be layered into annotations WELL, they will be hampered by a proposal that locks them out. If typing is done via a Typing class like I mentioned in my last email, then there is a very simple way of determining if we're looking at a type or at something else. There may be other, better ways of layering it in that don't lock other uses out, and which are better suited for static analysis. I'm just asking for a very simple, very clear method of distinguishing uses when parsing code. Types can get complicated quickly, and ad-hoc rules to determine if we're looking at a type, or at something else, can get us in trouble. Simply turning it on or off for a chunk of code also sucks; I LIKE types, so I'd like to put them into my code, but if I have a way of adding documentation to my code as well, then I'd like to do that in addition to the types. There may be other uses of annotations that programmers haven't yet thought of, simply because they don't know that annotations exist. Locking out those uses would suck. Does all this make sense? Thanks, Cem Karan From rosuav at gmail.com Sun Jan 18 15:16:05 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 19 Jan 2015 01:16:05 +1100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> Message-ID: <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> On Mon, Jan 19, 2015 at 12:37 AM, Cem Karan <cfkaran2 at gmail.com> wrote: > There may be other uses of annotations that programmers haven't yet thought of, simply because they don't know that annotations exist. Locking out those uses would suck. I see a lot of vague "but what if there's something else", and not a lot of concrete "here's a use of annotations that would be locked out". Does anyone actually have another use case that would be seriously harmed by this kind of conflict? Remember, you can simply not use type hints for the function(s) that use other annotations; the conflict is only if you're trying to use both on the same function. ChrisA From kaiser.yann at gmail.com Sun Jan 18 16:32:39 2015 From: kaiser.yann at gmail.com (Yann Kaiser) Date: Sun, 18 Jan 2015 15:32:39 +0000 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> Message-ID: <CANUJvPUDtskLAvE8a3QbqZHjy=ngiQjRpz49RVrctw9vxeDo2w@mail.gmail.com> On Sun Jan 18 2015 at 3:16:47 PM Chris Angelico <rosuav at gmail.com> wrote: > On Mon, Jan 19, 2015 at 12:37 AM, Cem Karan <cfkaran2 at gmail.com> wrote: > > There may be other uses of annotations that programmers haven't yet > thought of, simply because they don't know that annotations exist. Locking > out those uses would suck. > > I see a lot of vague "but what if there's something else", and not a > lot of concrete "here's a use of annotations that would be locked > out". Does anyone actually have another use case that would be > seriously harmed by this kind of conflict? I use annotations in clize, a CLI argument parser, to specify how functions parameters should translate to CLI parameters. This can range from specifying coercion functions/callables for the argument values(ie. int), which could easily be mixed up with typing information as specified here, to specifying aliases for named parameters, marking parameters as undocumented, or simply overriding the translation process altogether for that parameter. Now I could simply instruct users to never use annotations and always use a decorator instead (in fact, I recommend it when Python 2.x compatibility is wished) and have that decorator dump this information elsewhere than in f.__annotations__, but that begs the question of why it isn't as worthy of benefiting from annotation syntax[1] as type-checking. If anything, this is a use of annotations at run-time versus a use at dev-time in an offline fashion. I also find it a bit hasty to proclaim no one has found other viable uses for annotations when Python 3 adoption and thus liberty to use annotations still remains poor. > Remember, you can simply > not use type hints for the function(s) that use other annotations; the > conflict is only if you're trying to use both on the same function. > If typing is to become the default interpretation of annotations, I fear that this will really just turn into "WTF, your annotations aren't type info, fix it", especially given the recursive nature of those checks. Why can't it be explicit that a function's annotations are type info that should be checked? "typing.check(func)", "@typing.check def func(...):", "import typing; typing.check_all()", "import typing" not followed by "typing.nevermind()", anything? [1] I guess I could write a decorator that moves annotations out of __annotations__ into another dict, but I'm not sure that would be relevant at all to a *static* analyzer which wouldn't have to care what dict things end up in. Again, illustrates the weirdness of having this run-time feature be reserved for static analysis. > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150118/a02ae7fe/attachment.html> From abarnert at yahoo.com Sat Jan 17 05:54:54 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 16 Jan 2015 20:54:54 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAN8d9gnvftkuaX56k7RA_G_xh_iY5vTJd+0+aW6O_v_y=XmtMQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <CAN8d9gnvftkuaX56k7RA_G_xh_iY5vTJd+0+aW6O_v_y=XmtMQ@mail.gmail.com> Message-ID: <E0B6BF18-C36C-49CD-89EF-7035C752691E@yahoo.com> On Jan 16, 2015, at 12:05, "Philipp A." <flying-sheep at web.de> wrote: > hi, nice work! > > I like most of the ideas, except the semantic comments and some specific abstract classes. > > so criticism first: > > 1. i don?t think comments should ever have an impact on the code. > i?d prefer ?assert isinstance(...)? for type annotations on variables. The PEP already acknowledges that the comment solution isn't ideal, but says "If type hinting proves useful in general, a syntax for typing variables may be provided in a future Python version." And it also provides Undefined as an optional alternative way to type variables. But, more importantly, I think you've missed the key point that type annotations are not supposed to affect runtime or compile time at all; they're only there for a separate static type checker. (That could change in the future--at that point, I assume either Guido will come up with a syntax for variable types, or drop explicit variable types.) So this is akin to comments for a linker, docs generator, or test generator. Asserts, on the other hand, would be tested at runtime, possibly breaking working code and likely slowing things down. > 2. also i don?t see why Dict, (non-abstract) Set, and List exist as long as we have Mapping and so on. Tuple is different because of its ?Multiple return value? status and status as ?is treated as Union[Class1, Class2] when used with instanceof(), except and friends?. > > but checking for instanceof(..., list) is unpytonic! imho List and Dict really go against python?s data model and protocol-based nature. This was argued in depth when the idea was first proposed. For the record, I agree that List and friends are more often going to be an attractive nuisance that people will use when they should have used Iterable or Sequence, but if you're going to reopen that argument, you'll want to read the old threads first. Also, that's clearly not just a request for clarification, so I think it belongs on the tracker, where the argument can be preserved permanently in a better format than list archives.l > i?d be for consistency: Set and MutableSet, Sequence and MutableSequence, Mapping and MutableMapping. no ?abstract? (because all of this is abstract), no ?frozen? (because frozen sets are just immutable sets and immutable behavior is a subset of the corresponding mutable behavior). The builtin concrete type frozenset has existed for years. It's an instantiation of the abstract Set type, but (unlike set) not of the MutableSet type. You could argue that they should have been named more consistently, but only if you want to go back to whenever sets.ImmutableSet was renamed frozenset (2.5?) or when the ABCs were first added (2.6/3.0). > Questions: > > 1. will we be able to use Union[...] in places like an except clause, where tuples are used to mean the same right now? would make sense imho Since the elements of a Union are all defined to be subclasses of the Union, it seems like this would automatically work in except clauses, isinstance and issubclass calls, and anywhere else that uses issubclass. But why would you want to? > 2. what are ItemsView and KeysView for? what makes them different from FrozenSet or Sequence? They've been in the collections module since 3.0. The documentation already explains why they exist and how they're different from Set. (Actually, I think it's explained more in the docs for the concrete classes associated with dict than in the ABCs, but it's there.) (You actually asked about FrozenSet and Sequence, not AbstractSet/Set. What makes them different from FrozenSet is that's a generic for the concrete type frozenset; what makes them different from Sequence is that they're not indexable, which is the fundamental thing sequences do.) > > Guido van Rossum <guido at python.org> schrieb am Fri Jan 16 2015 at 18:19:24: >> After a long editing process we've got PEP 484 (Type Hints) ready for your review. This is by no means final, and several areas are either open (to be resolved in a later draft) or postponed (to a different PEP altogether). But there's enough meat that I think we can start having the discussion. Please also see PEP 483 (The Theory of Type Hint; copied and reformatted from the original Quip document that I posted just before last Christmas) and PEP 482 (Literature Overview for Type Hints, by ?ukasz). Those are informational PEPs though; the actual spec is focused in PEP 484 (the only one on the Standards Track). >> >> As I said earlier, I hope to have a rough consensus before PyCon and working code (just the typing.py module, really) committed to CPython before the last 3.5 alpha. >> >> Here is the raw text of PEP 484. Fire away!! >> >> PEP: 484 >> Title: Type Hints >> Version: $Revision$ >> Last-Modified: $Date$ >> Author: Guido van Rossum <guido at python.org>, Jukka Lehtosalo <jukka.lehtosalo at iki.fi>, ?ukasz Langa <lukasz at langa.pl> >> Discussions-To: Python-Dev <python-dev at python.org> >> Status: Draft >> Type: Standards Track >> Content-Type: text/x-rst >> Created: 29-Sep-2014 >> Post-History: 16-Jan-2015 >> Resolution: >> >> >> Abstract >> ======== >> >> This PEP introduces a standard syntax for type hints using annotations >> on function definitions. >> >> The proposal is strongly inspired by mypy [mypy]_. >> >> The theory behind type hints and gradual typing is explained in PEP 483. >> >> >> Rationale and Goals >> =================== >> >> PEP 3107 added support for arbitrary annotations on parts of a function >> definition. Although no meaning was assigned to annotations then, there >> has always been an implicit goal to use them for type hinting, which is >> listed as the first possible use case in said PEP. >> >> This PEP aims to provide a standard syntax for type annotations, opening >> up Python code to easier static analysis and refactoring, potential >> runtime type checking, and performance optimizations utilizing type >> information. >> >> >> Type Definition Syntax >> ====================== >> >> The syntax leverages PEP 3107-style annotations with a number of >> extensions described in sections below. In its basic form, type hinting >> is used by filling function annotations with classes:: >> >> def greeting(name: str) -> str: >> return 'Hello ' + name >> >> This denotes that the expected type of the ``name`` argument is ``str``. >> Analogically, the expected return type is ``str``. Subclasses of >> a specified argument type are also accepted as valid types for that >> argument. >> >> Abstract base classes, types available in the ``types`` module, and >> user-defined classes may be used as type hints as well. Annotations >> must be valid expressions that evaluate without raising exceptions at >> the time the function is defined. In addition, the needs of static >> analysis require that annotations must be simple enough to be >> interpreted by static analysis tools. (This is an intentionally >> somewhat vague requirement.) >> >> .. FIXME: Define rigorously what is/isn't supported. >> >> When used as an annotation, the expression ``None`` is considered >> equivalent to ``NoneType`` (i.e., ``type(None)`` for type hinting >> purposes. >> >> Type aliases are also valid type hints:: >> >> integer = int >> >> def retry(url: str, retry_count: integer): ... >> >> New names that are added to support features described in following >> sections are available in the ``typing`` package. >> >> >> Callbacks >> --------- >> >> Frameworks expecting callback functions of specific signatures might be >> type hinted using ``Callable[[Arg1Type, Arg2Type], ReturnType]``. >> Examples:: >> >> from typing import Any, AnyArgs, Callable >> >> def feeder(get_next_item: Callable[[], Item]): ... >> >> def async_query(on_success: Callable[[int], None], on_error: Callable[[int, Exception], None]): ... >> >> def partial(func: Callable[AnyArgs, Any], *args): ... >> >> Since using callbacks with keyword arguments is not perceived as >> a common use case, there is currently no support for specifying keyword >> arguments with ``Callable``. >> >> >> Generics >> -------- >> >> Since type information about objects kept in containers cannot be >> statically inferred in a generic way, abstract base classes have been >> extended to support subscription to denote expected types for container >> elements. Example:: >> >> from typing import Mapping, Set >> >> def notify_by_email(employees: Set[Employee], overrides: Mapping[str, str]): ... >> >> Generics can be parametrized by using a new factory available in >> ``typing`` called ``TypeVar``. Example:: >> >> from typing import Sequence, TypeVar >> >> T = TypeVar('T') # Declare type variable >> >> def first(l: Sequence[T]) -> T: # Generic function >> return l[0] >> >> In this case the contract is that the returning value is consistent with >> the elements held by the collection. >> >> ``TypeVar`` supports constraining parametric types to classes with any of >> the specified bases. Example:: >> >> from typing import Iterable >> >> X = TypeVar('X') >> Y = TypeVar('Y', Iterable[X]) >> >> def filter(rule: Callable[[X], bool], input: Y) -> Y: >> ... >> >> .. FIXME: Add an example with multiple bases defined. >> >> In the example above we specify that ``Y`` can be any subclass of >> Iterable with elements of type ``X``, as long as the return type of >> ``filter()`` will be the same as the type of the ``input`` >> argument. >> >> .. FIXME: Explain more about how this works. >> >> >> Forward references >> ------------------ >> >> When a type hint contains names that have not been defined yet, that >> definition may be expressed as a string, to be resolved later. For >> example, instead of writing:: >> >> def notify_by_email(employees: Set[Employee]): ... >> >> one might write:: >> >> def notify_by_email(employees: 'Set[Employee]'): ... >> >> .. FIXME: Rigorously define this. Defend it, or find an alternative. >> >> >> Union types >> ----------- >> >> Since accepting a small, limited set of expected types for a single >> argument is common, there is a new special factory called ``Union``. >> Example:: >> >> from typing import Union >> >> def handle_employees(e: Union[Employee, Sequence[Employee]]): >> if isinstance(e, Employee): >> e = [e] >> ... >> >> A type factored by ``Union[T1, T2, ...]`` responds ``True`` to >> ``issubclass`` checks for ``T1`` and any of its subclasses, ``T2`` and >> any of its subclasses, and so on. >> >> One common case of union types are *optional* types. By default, >> ``None`` is an invalid value for any type, unless a default value of >> ``None`` has been provided in the function definition. Examples:: >> >> def handle_employee(e: Union[Employee, None]): ... >> >> As a shorthand for ``Union[T1, None]`` you can write ``Optional[T1]``; >> for example, the above is equivalent to:: >> >> from typing import Optional >> >> def handle_employee(e: Optional[Employee]): ... >> >> An optional type is also automatically assumed when the default value is >> ``None``, for example:: >> >> def handle_employee(e: Employee = None): ... >> >> This is equivalent to:: >> >> def handle_employee(e: Optional[Employee] = None): ... >> >> .. FIXME: Is this really a good idea? >> >> A special kind of union type is ``Any``, a class that responds >> ``True`` to ``issubclass`` of any class. This lets the user >> explicitly state that there are no constraints on the type of a >> specific argument or return value. >> >> >> Platform-specific type checking >> ------------------------------- >> >> In some cases the typing information will depend on the platform that >> the program is being executed on. To enable specifying those >> differences, simple conditionals can be used:: >> >> from typing import PY2, WINDOWS >> >> if PY2: >> text = unicode >> else: >> text = str >> >> def f() -> text: ... >> >> if WINDOWS: >> loop = ProactorEventLoop >> else: >> loop = UnixSelectorEventLoop >> >> Arbitrary literals defined in the form of ``NAME = True`` will also be >> accepted by the type checker to differentiate type resolution:: >> >> DEBUG = False >> ... >> if DEBUG: >> class Tracer: >> <verbose implementation> >> else: >> class Tracer: >> <dummy implementation> >> >> For the purposes of type hinting, the type checker assumes ``__debug__`` >> is set to ``True``, in other words the ``-O`` command-line option is not >> used while type checking. >> >> >> Compatibility with other uses of function annotations >> ----------------------------------------------------- >> >> A number of existing or potential use cases for function annotations >> exist, which are incompatible with type hinting. These may confuse a >> static type checker. However, since type hinting annotations have no >> run time behavior (other than evaluation of the annotation expression >> and storing annotations in the ``__annotations__`` attribute of the >> function object), this does not make the program incorrect -- it just >> makes it issue warnings when a static analyzer is used. >> >> To mark portions of the program that should not be covered by type >> hinting, use the following: >> >> * a ``@no_type_checks`` decorator on classes and functions >> >> * a ``# type: ignore`` comment on arbitrary lines >> >> .. FIXME: should we have a module-wide comment as well? >> >> >> Type Hints on Local and Global Variables >> ======================================== >> >> No first-class syntax support for explicitly marking variables as being >> of a specific type is added by this PEP. To help with type inference in >> complex cases, a comment of the following format may be used:: >> >> x = [] # type: List[Employee] >> >> In the case where type information for a local variable is needed before >> if was declared, an ``Undefined`` placeholder might be used:: >> >> from typing import Undefined >> >> x = Undefined # type: List[Employee] >> y = Undefined(int) >> >> If type hinting proves useful in general, a syntax for typing variables >> may be provided in a future Python version. >> >> >> Explicit raised exceptions >> ========================== >> >> No support for listing explicitly raised exceptions is being defined by >> this PEP. Currently the only known use case for this feature is >> documentational, in which case the recommendation is to put this >> information in a docstring. >> >> >> The ``typing`` package >> ====================== >> >> To open the usage of static type checking to Python 3.5 as well as older >> versions, a uniform namespace is required. For this purpose, a new >> package in the standard library is introduced called ``typing``. It >> holds a set of classes representing builtin types with generics, namely: >> >> * Dict, used as ``Dict[key_type, value_type]`` >> >> * List, used as ``List[element_type]`` >> >> * Set, used as ``Set[element_type]``. See remark for ``AbstractSet`` >> below. >> >> * FrozenSet, used as ``FrozenSet[element_type]`` >> >> * Tuple, used as ``Tuple[index0_type, index1_type, ...]``. >> Arbitrary-length tuples might be expressed using ellipsis, in which >> case the following arguments are considered the same type as the last >> defined type on the tuple. >> >> It also introduces factories and helper members needed to express >> generics and union types: >> >> * Any, used as ``def get(key: str) -> Any: ...`` >> >> * Union, used as ``Union[Type1, Type2, Type3]`` >> >> * TypeVar, used as ``X = TypeVar('X', Type1, Type2, Type3)`` or simply >> ``Y = TypeVar('Y')`` >> >> * Undefined, used as ``local_variable = Undefined # type: List[int]`` or >> ``local_variable = Undefined(List[int])`` (the latter being slower >> during runtime) >> >> * Callable, used as ``Callable[[Arg1Type, Arg2Type], ReturnType]`` >> >> * AnyArgs, used as ``Callable[AnyArgs, ReturnType]`` >> >> * AnyStr, equivalent to ``TypeVar('AnyStr', str, bytes)`` >> >> All abstract base classes available in ``collections.abc`` are >> importable from the ``typing`` package, with added generics support: >> >> * ByteString >> >> * Callable >> >> * Container >> >> * Hashable >> >> * ItemsView >> >> * Iterable >> >> * Iterator >> >> * KeysView >> >> * Mapping >> >> * MappingView >> >> * MutableMapping >> >> * MutableSequence >> >> * MutableSet >> >> * Sequence >> >> * Set as ``AbstractSet``. This name change was required because ``Set`` >> in the ``typing`` module means ``set()`` with generics. >> >> * Sized >> >> * ValuesView >> >> * Mapping >> >> The library includes literals for platform-specific type hinting: >> >> * PY2 >> >> * PY3, equivalent to ``not PY2`` >> >> * WINDOWS >> >> * UNIXOID, equivalent to ``not WINDOWS`` >> >> The following types are available in the ``typing.io`` module: >> >> * IO >> >> * BinaryIO >> >> * TextIO >> >> The following types are provided by the ``typing.re`` module: >> >> * Match and Pattern, types of ``re.match()`` and ``re.compile()`` >> results >> >> As a convenience measure, types from ``typing.io`` and ``typing.re`` are >> also available in ``typing`` (quoting Guido, "There's a reason those >> modules have two-letter names."). >> >> >> The place of the ``typing`` module in the standard library >> ---------------------------------------------------------- >> >> .. FIXME: complete this section >> >> >> Usage Patterns >> ============== >> >> The main use case of type hinting is static analysis using an external >> tool without executing the analyzed program. Existing tools used for >> that purpose like ``pyflakes`` [pyflakes]_ or ``pylint`` [pylint]_ >> might be extended to support type checking. New tools, like mypy's >> ``mypy -S`` mode, can be adopted specifically for this purpose. >> >> Type checking based on type hints is understood as a best-effort >> mechanism. In other words, whenever types are not annotated and cannot >> be inferred, the type checker considers such code valid. Type errors >> are only reported in case of explicit or inferred conflict. Moreover, >> as a mechanism that is not tied to execution of the code, it does not >> affect runtime behaviour. In other words, even in the case of a typing >> error, the program will continue running. >> >> The implementation of a type checker, whether linting source files or >> enforcing type information during runtime, is out of scope for this PEP. >> >> .. FIXME: Describe stub modules. >> >> .. FIXME: Describe run-time behavior of generic types. >> >> >> Existing Approaches >> =================== >> >> PEP 482 lists existing approaches in Python and other languages. >> >> >> Is type hinting Pythonic? >> ========================= >> >> Type annotations provide important documentation for how a unit of code >> should be used. Programmers should therefore provide type hints on >> public APIs, namely argument and return types on functions and methods >> considered public. However, because types of local and global variables >> can be often inferred, they are rarely necessary. >> >> The kind of information that type hints hold has always been possible to >> achieve by means of docstrings. In fact, a number of formalized >> mini-languages for describing accepted arguments have evolved. Moving >> this information to the function declaration makes it more visible and >> easier to access both at runtime and by static analysis. Adding to that >> the notion that ?explicit is better than implicit?, type hints are >> indeed *Pythonic*. >> >> >> Acknowledgements >> ================ >> >> This document could not be completed without valuable input, >> encouragement and advice from Jim Baker, Jeremy Siek, Michael Matson >> Vitousek, Andrey Vlasovskikh, and Radomir Dopieralski. >> >> Influences include existing languages, libraries and frameworks >> mentioned in PEP 482. Many thanks to their creators, in alphabetical >> order: Stefan Behnel, William Edwards, Greg Ewing, Larry Hastings, >> Anders Hejlsberg, Alok Menghrajani, Travis E. Oliphant, Joe Pamer, >> Raoul-Gabriel Urma, and Julien Verlaguet. >> >> >> References >> ========== >> >> .. [mypy] >> http://mypy-lang.org >> >> .. [pyflakes] >> https://github.com/pyflakes/pyflakes/ >> >> .. [pylint] >> http://www.pylint.org >> >> >> Copyright >> ========= >> >> This document has been placed in the public domain. >> >> >> >> .. >> Local Variables: >> mode: indented-text >> indent-tabs-mode: nil >> sentence-end-double-space: t >> fill-column: 70 >> coding: utf-8 >> End: >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150116/5ef53b89/attachment-0001.html> From chris.barker at noaa.gov Sun Jan 18 19:28:48 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Sun, 18 Jan 2015 10:28:48 -0800 Subject: [Python-ideas] Way to check for floating point "closeness"? In-Reply-To: <20150117100117.GK18556@ando.pearwood.info> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <m9cn9c$i3b$1@ger.gmane.org> <20150117100117.GK18556@ando.pearwood.info> Message-ID: <CALGmxEKPfMdNfzSxSt=RG7Xv29SZkGqcPQVd49Qt61GLma6=Zw@mail.gmail.com> OK, I FINALLY got a chance to look at Steven's code in the statistic module tests. Not much code there, this really isn't hat big a deal. It does check for NaN, and inf and all that, so that's good. It is also symmetric with respect to x and y -- using the maximum of the two to compute the relative error -- I think that's good. (This is essentially the same as Boosts "strong" method -- though implemented a tiny bit differently). Here is the key definition: def approx_equal(x, y, tol=1e-12, rel=1e-7): ... x is approximately equal to y if the difference between them is less than an absolute error tol or a relative error rel, whichever is bigger. ... This is a lot like the numpy code, actually, except it does a max test, rather than adding the absolute and relative tolerances together. I think this is a better way to go than numpy's but there is little practical difference. However, it suffers from the same issue -- "tol" is essentially a minimum error that is considered acceptable. This is nice, as it it will allow zero to be passed in, and if the other input is within tol of zero, it will be considered approximately equal. However, for very small numbers (less that the absolute tolerance), then they will always be considered approximately equal: In [18]: approx_equal(1.0e-14, 2.0e-14) Out[18]: True off by a factor of 2 In [19]: approx_equal(1.0e-20, 2.0e-25) Out[19]: True oops! way off! This is with the defaults of course, and all you need to do is set teh tol much lower: In [20]: approx_equal(1.0e-20, 2.0e-25, tol=1e-25) Out[20]: False This is less fatal than with numpy, as with numpy you are processing a whole array of numbers with the same tolerances, and they may not be all of the same magnitude. But I think think it's trap for users. My proposal: Allow either an absolute or relative tolerance, but not try to do both in one call. or If you really want the ability to do both at once (i.e. set a minimum for the zero case), then: - make the default absolute tolerance zero -- fewer surprises that way - document the absolute tolerance as a mimimum error (difference), and specifically mention the zero case in the docs. Otherwise, go with Steven's code, and put it in the math module. Also -- there was some talk of what do do with complex -- I say two complex numbers are approx_equal if approx_equal(z1.real, z2.real) and approx_equal(z1.imag, z2.imag) -- that is more rigorous a test than using the complex abs value of the difference. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150118/bf31a142/attachment.html> From ron3200 at gmail.com Sun Jan 18 20:27:40 2015 From: ron3200 at gmail.com (Ron Adam) Date: Sun, 18 Jan 2015 13:27:40 -0600 Subject: [Python-ideas] Fwd: Way to check for floating point "closeness"? In-Reply-To: <CALGmxEL9vFYmmw=mkDpLR+UmBGcmHCQ_tAPburGu0ak_Ymrfkg@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <m9cn9c$i3b$1@ger.gmane.org> <CALGmxE+B5A8E3djzEmv_WwQa2gMMnKhhaX325xyq83mNpFRJvA@mail.gmail.com> <CALGmxEL9vFYmmw=mkDpLR+UmBGcmHCQ_tAPburGu0ak_Ymrfkg@mail.gmail.com> Message-ID: <m9h1fe$4lq$1@ger.gmane.org> On 01/17/2015 11:37 PM, Chris Barker wrote: > (Someone claimed that 'nothing is close to zero'. This is > nonsensical both in applied math and everyday life.) > > > I'm pretty sure someone (more than one of use) asserted that "nothing is > *relatively* close to zero -- very different. Yes, that is the case. > And I really wanted a way to have a default behavior that would do a > reasonable transition to an absolute tolerance near zero, but I no longer > thing that's possible. (numpy's implimentaion kind of does that, but it is > really wrong for small numbers, and if you made the default min_tolerance > the smallest possible representable number, it really wouldn't be useful. I'm going to try to summarise what I got out of this discussion. Maybe it will help bring some focus to the topic. I think there are two case's to consider. # The most common case. rel_is_good(actual, expected, delta) # value +- %delta. # Testing for possible equivalence? rel_is_close(value1, value2, delta) # %delta close to each other. I don't think they are quite the same thing. rel_is_good(9, 10, .1) --> True rel_is_good(10, 9, .1) --> False rel_is_close(9, 10, .1) --> True rel_is_close(10, 9, .1) --> True In the "is close" case, it shouldn't matter what order the arguments are given. The delta is the distance from the larger number the smaller number is. (of the same sign) So when calculating the relative error from two values, you want it to be consistent with the rel_is_close function. rel_is_close(a, b, delta) <---> rel_err(a, b) <= delta And you should not use the rel_err function in the rel_is_good function. The next issue is, where does the numeric accuracy of the data, significant digits, and the languages accuracy (ULPs), come into the picture. My intuition.. I need to test the idea to make a firmer claim.. is that in the case of is_good, you want to exclude the uncertain parts, but with is_close, you want to include the uncertain parts. Two values "are close" if you can't tell one from the other with certainty. The is_close range includes any uncertainty. A value is good if it's within a range with certainty. And this excludes any uncertainty. This is where taking in consideration of an absolute delta comes in. The minimum range for both is the uncertainty of the data. But is_close and is_good do different things with it. Of course all of this only applies if you agree with these definitions of is_close, and is_good. ;) Cheers, Ron From guido at python.org Sun Jan 18 20:34:03 2015 From: guido at python.org (Guido van Rossum) Date: Sun, 18 Jan 2015 11:34:03 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> Message-ID: <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> On Sun, Jan 18, 2015 at 5:37 AM, Cem Karan <cfkaran2 at gmail.com> wrote: > > On Jan 17, 2015, at 9:27 PM, Ben Finney <ben+python at benfinney.id.au> > wrote: > > > Cem Karan <cfkaran2 at gmail.com> writes: > > > >> [?] I feel like annotations have the capability to be MUCH more useful > >> than just as a place to hold type hints. > > > > They have that capability. How long do you propose we wait for this > > capability to be realised? > > > > Function annotations have been part of Python since version 3.0, and > > Guido (IIUC) is deciding that the wait is over: we already have the best > > use of function annotations we're going to get. > > I understand what you're saying, and you're right that they are not used > much. I only learned about them after I started playing with Python 3.3. > But then I'm the bleeding edge for my group at work; everyone else is down > in the python 2 world somewhere (some of them are WAY down there somewhere). > > Changing a language takes time. Do you program in C? Do you still use > pthreads? Why? C11 specifies the thread.h header, you should switch to C11 > threads right now! Except... many compilers still don't fully support C11, > so we're coding to an earlier standard. There is a lot of code that > already works as it is. Etc., etc. etc... > > When a language changes, there is a LOT of catching up to do. First, > tools need to support the new facilities and then programmers need to > become aware of new facilities, and they have to realize there is a > legitimate use for those facilities. Most people I know are unaware that > annotations exist; that is an argument for making the change to supporting > typing only, simply because there will be fewer people that are affected. > At the same time, if someone has a really good idea that could be layered > into annotations WELL, they will be hampered by a proposal that locks them > out. If typing is done via a Typing class like I mentioned in my last > email, then there is a very simple way of determining if we're looking at a > type or at something else. There may be other, better ways of layering it > in that don't lock other uses out, and which are better suited for static > analysis. > > I'm just asking for a very simple, very clear method of distinguishing > uses when parsing code. Types can get complicated quickly, and ad-hoc > rules to determine if we're looking at a type, or at something else, can > get us in trouble. Simply turning it on or off for a chunk of code also > sucks; I LIKE types, so I'd like to put them into my code, but if I have a > way of adding documentation to my code as well, then I'd like to do that in > addition to the types. There may be other uses of annotations that > programmers haven't yet thought of, simply because they don't know that > annotations exist. Locking out those uses would suck. > > Does all this make sense? > I hear you, and I'm not buying it. I've thought about this a lot (more than you, I'm sure) and I'm putting a line in the sand: annotations are for types, and all other uses will eventually have to find some other way (most likely decorators). (The "type: OFF" comments are a transitional measure to avoid breaking other uses without warning in 3.5.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150118/2a94c63c/attachment-0001.html> From chris.barker at noaa.gov Sun Jan 18 20:39:52 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Sun, 18 Jan 2015 11:39:52 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> Message-ID: <CALGmxELGJr9+RMOeo71BRbDZ8r-PFUGh0+5jwy3YyEk=DcUpSg@mail.gmail.com> On Fri, Jan 16, 2015 at 12:08 PM, Guido van Rossum <guido at python.org> wrote: > Yes, the PEP tries to gently push an agenda where annotations are only > used for Python type hints. > Type hints, as opposed to arbitrary other uses, or type _hints_ as aopposed to run-time of compile time static typing? Cython, providing both of those, for example. I don't know what to answer for Cython -- doesn't it have its own syntax > for declaring C types? > yes: cdef int i or: def func( int i ). There is also a decorator-based system, for "pure python But I think it would be nice it it could use the same syntax as other systems, particularly if there is an official one. > If Cython uses annotations with different meanings, perhaps for Cython > files the Python annotations could be put in stub files. > I think we're good there. But I think Stephan's issues is that currently cython allows both C and Python type names: cdef int i cdef list a_list but where the names conflict, the C name is assumed (int, float, others?) Given C's typedef, I'd say we should just have Cython use unique names -- i.e. c_int -> int, etc. But that all requires that no one decides to write a new python type called c_int. Though I'm not sure it's really a problem to reserve names for Cython. Anyway, I guess it's unclear to me whether the intent is to create a system that can be used for a static/run-time type checker like Cython. There have been a fair number of comments alone the lines of "this is only for lint-like static analysis". But I'd really love to see this move toward full functionailty. In fact, after reading the PEPs, I still don't really see the point of static type analysis that does not lead to compile time or run-time type checking. Maybe someone could provide the motivation for that, oradd alink to the PEP where that is. My question: I see two clear advantages to static typing: type safety and performance. As a proponent of Python (and thus dynamic typing), I have argued a lot that static type checking does indeed catch a lot of bugs, but they are generally shallow bugs -- easy to test for, and easy to understand and fix -- thus I don't much need that. However, I can see how it's a "good thing", and maybe important for "Enterprise" systems. On the other hand, dynamic typing does impact performance in a way that effects my code -- so I tun to Cython (and sometimes C, C++, Fortran) to address those issues. (maybe JIT compilation will one day meet this need, but it hasn't yet). So if there is type specifying, I want i to solve these problems. But where I'm lost is the utility of static analysis that is not used to either provide run-time type checking OR compile optimized code (either JIT or ahead-of-time). It seems that your entire system would have to be fully type-hinted to get the type safety. But once you've done that, why not just use a static language? On the other hand, if run-time type checking is provided, then you could use that at the boundaries of your code base (kind of like Cython does) and get both the safety and performance benifits. I've read the MyPY FAQ, which does address these issues to some extent, but doesn't really make a compelling case to me, anyway. I guess the gist here is that I think if Python is going to have an official type hinting system, it should leave the door open to compile time and run-time type checking. A good test case might be: can it be used for Cython? But this thread seems to be indicating that that is not the goal, and indeed, that it's almost an anti-goal. I hope I'm wrong about that. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150118/1c635762/attachment.html> From chris.barker at noaa.gov Sun Jan 18 21:14:04 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Sun, 18 Jan 2015 12:14:04 -0800 Subject: [Python-ideas] Fwd: Way to check for floating point "closeness"? In-Reply-To: <m9h1fe$4lq$1@ger.gmane.org> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <m9cn9c$i3b$1@ger.gmane.org> <CALGmxE+B5A8E3djzEmv_WwQa2gMMnKhhaX325xyq83mNpFRJvA@mail.gmail.com> <CALGmxEL9vFYmmw=mkDpLR+UmBGcmHCQ_tAPburGu0ak_Ymrfkg@mail.gmail.com> <m9h1fe$4lq$1@ger.gmane.org> Message-ID: <CALGmxELNDKWMRSiiJy_1u89jgYwCSZZAEDHQ3Rkp63J3HAi86g@mail.gmail.com> On Sun, Jan 18, 2015 at 11:27 AM, Ron Adam <ron3200 at gmail.com> wrote: I'm going to try to summarise what I got out of this discussion. Maybe it > will help bring some focus to the topic. > > I think there are two case's to consider. > > # The most common case. > why do you think this is the most common case? > rel_is_good(actual, expected, delta) # value +- %delta. > > # Testing for possible equivalence? > rel_is_close(value1, value2, delta) # %delta close to each other. > > I don't think they are quite the same thing. > > rel_is_good(9, 10, .1) --> True > rel_is_good(10, 9, .1) --> False > > rel_is_close(9, 10, .1) --> True > rel_is_close(10, 9, .1) --> True > agreed -- they are not the same thing. But I'm not convinced that they are all that different from a practical perspective -- 0.1 is a very large relative tolerance (I wouldn't cal it delta for a relative measure) if you use a more-common, much smaller tolerance (1e-8 -- 1e-12 maybe?) then the difference between these becomes pretty minimal. And for the most part, this kind of testing is looking for an "approximation" -- so you can't get really upset about exactly where the cut-off is. Though, given my thoughts on that, I suppose if other people want to be able to clearly specify which of the two values should be used to scale "relative", then it won't make much difference anyway. The next issue is, where does the numeric accuracy of the data, significant > digits, and the languages accuracy (ULPs), come into the picture. > > My intuition.. I need to test the idea to make a firmer claim.. is that in > the case of is_good, you want to exclude the uncertain parts, but with > is_close, you want to include the uncertain parts. > I think this is pretty irrelevant -- you can't do better than the precession of the data type, doesn't make a difference which definition you are using -- they only change which value is used to scale relative. There are cases where ULPs, etc are key -- those are for testing precision of algorithms, etc, and I think a special use case that would require a different function -- no need to try to cram it all into one function. > Two values "are close" if you can't tell one from the other with > certainty. The is_close range includes any uncertainty. > I think "uncertainly" is entirely use-case dependent -- floating point calculations are not uncertain -- with a defined rounding procedure, etc, they are completely deterministic. IT can often make things easier to think about if you think of the errors as random, but they are, in fact, not random. And if you care about the precision down to close to limits of representation, then you care about ULPS, etc, and you'd better be reasoning about the errors carefully. This is where taking in consideration of an absolute delta comes in. The > minimum range for both is the uncertainty of the data. But is_close and > is_good do different things with it. > an absolute delta is simply a different use-case -- sometimes you know exactly how much difference you care about, and sometimes you only care that the numbers are relatively close (or, in any case, you need to test a wide variety of magnitudes of numbers, so don't want to have to calculate the absolute delta for each one). This is akin to saying that the numbers are the same to a certain number of significant figures -- a common and usefull way to think about it. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150118/26836d1f/attachment-0001.html> From abarnert at yahoo.com Sun Jan 18 21:29:01 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 18 Jan 2015 12:29:01 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CANUJvPUDtskLAvE8a3QbqZHjy=ngiQjRpz49RVrctw9vxeDo2w@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <CANUJvPUDtskLAvE8a3QbqZHjy=ngiQjRpz49RVrctw9vxeDo2w@mail.gmail.com> Message-ID: <05BDAF49-2478-4BA8-99C9-4E2AC6DE6641@yahoo.com> On Jan 18, 2015, at 7:32, Yann Kaiser <kaiser.yann at gmail.com> wrote: > On Sun Jan 18 2015 at 3:16:47 PM Chris Angelico <rosuav at gmail.com> wrote: >> On Mon, Jan 19, 2015 at 12:37 AM, Cem Karan <cfkaran2 at gmail.com> wrote: >> > There may be other uses of annotations that programmers haven't yet thought of, simply because they don't know that annotations exist. Locking out those uses would suck. >> >> I see a lot of vague "but what if there's something else", and not a >> lot of concrete "here's a use of annotations that would be locked >> out". Does anyone actually have another use case that would be >> seriously harmed by this kind of conflict? > > I use annotations in clize, a CLI argument parser, to specify how functions parameters should translate to CLI parameters. This can range from specifying coercion functions/callables for the argument values(ie. int), which could easily be mixed up with typing information as specified here, to specifying aliases for named parameters, marking parameters as undocumented, or simply overriding the translation process altogether for that parameter. And I can't imagine that you'd ever want to use static types and clize annotations on the same function, given that the whole point of clize is that the functions will be called dynamically by some dispatcher with arguments from the command line. So, this is a perfect example for why we need a way to mark a function as "these annotations don't mean types". Which is already part of the PEP. (It's also, maybe not coincidentally, another example of the fact that most existing uses of annotations that aren't types are "type-like"--e.g., C types, conversion functions, etc.) But it's not an example for why we need to mix two kinds of annotations in the same function, or where it would be unclear which annotations you're using, which is what Chris was asking for. > Now I could simply instruct users to never use annotations and always use a decorator instead (in fact, I recommend it when Python 2.x compatibility is wished) and have that decorator dump this information elsewhere than in f.__annotations__, but that begs the question of why it isn't as worthy of benefiting from annotation syntax[1] as type-checking. If anything, this is a use of annotations at run-time versus a use at dev-time in an offline fashion. The obvious answer is to tell users to decorate their CLI functions with @no_type_check, or just not type-check scripts made up of clize functions. I'm willing to bet, without searching, that clize already uses a decorator for cli functions. This implies a possible useful extension to the PEP: a way to declare decorators as implying no_type_check. Then your users wouldn't have to do anything; you'd just declare @clize as meaning no type checking, and their existing code would be type-checkable with no changes. > I also find it a bit hasty to proclaim no one has found other viable uses for annotations when Python 3 adoption and thus liberty to use annotations still remains poor. This is an argument for not evolving Python 3 at all until everyone switches. And it's not a very good argument, because the only reason for anyone to switch to Python 3 is that it's evolved features that they want. It's also neglecting the fact that Python 3 adoption really isn't that poor. In 2012, many libraries still didn't work in 3.x, any novice question on StackOverflow could be safely assumed to be 2.x-specific, and 2.7 was the obvious and most popular choice for new projects. In 2015, none of that is true anymore. Someone who primarily works on existing applications may not see that, because for an existing application, the cost of moving to 3.x is higher and the benefit lower (you can't design your app to take advantage of new features when it's already designed). But then someone who primarily works on existing applications also has less useful input into the way the language should change. Cem made this even more explicit by bringing up C11: if some code can't assume 2011 for C, no code should assume 2010 for Python. That's ridiculous. C is one of the most stable and conservative languages that people are still using. It also tends to take years for new C features to make it into implementations, because of the way C is designed (a committee works out a definition down to the smallest details, votes on it, publishes it, and then compilers implement it), whereas in Python--as with most other languages--it takes a single release cycle (under 18 months) for every new feature to make it into the implementation(s) that 90% of people are using. A better comparison would be Java 7, ECMAScript 5, or Ruby 1.9. These were all relatively big changes to their languages that happened within a year of Python 3. And nobody would say you can't judge the new features in those languages yet. >> Remember, you can simply >> not use type hints for the function(s) that use other annotations; the >> conflict is only if you're trying to use both on the same function. > > If typing is to become the default interpretation of annotations, I fear that this will really just turn into "WTF, your annotations aren't type info, fix it", especially given the recursive nature of those checks. Why do you fear that? Default just means default. Escape mechanisms that are only meant to be used in rare cases are usually hidden away and ugly; those that are simple and obvious are meant to be used. (At least in Python; not every language is as nice...) > Why can't it be explicit that a function's annotations are type info that should be checked? Because it's the less common uses that should be explicitly marked. There's cognitive value in marking something uncommon and none in marking something common. Think of how much boilerplate you're suggesting be added to 80% of code, all for the benefit of saving (at most) the same amount of boilerplate for 20%. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150118/c9f00636/attachment.html> From stefan_ml at behnel.de Sun Jan 18 21:36:34 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 18 Jan 2015 21:36:34 +0100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> Message-ID: <m9h5gk$2a6$1@ger.gmane.org> Guido van Rossum schrieb am 18.01.2015 um 05:11: > On Sat, Jan 17, 2015 at 1:43 AM, Stefan Behnel wrote: >> Guido van Rossum schrieb am 16.01.2015 um 21:08: >>> I don't know what to answer for Cython -- doesn't it have its own syntax >>> for declaring C types? Or is it now also using annotations? >> >> It can, but it's optional. When I added support for it, I took care not to >> enable it by default, so that it wouldn't accidentally interfere with other >> usages of annotations. >> >> The current proposal seems less generous in this regard, and the way I see >> it, the only gain from Cython's point of view is its very limited support >> for container item type declarations and function signatures of callable >> objects. Both are restricted to Python object types and even imply slightly >> different semantics than Cython (I didn't see a notion of "exact types" as >> opposed to "type or subtype"). > > Actually, while the PEP is still lacking in clarity and we're still having > some discussions, type variables will most likely be "invariant" by > default, which I think is the "exact types" notion you are after. There's > lots of discussion in https://github.com/ambv/typehinting/issues/2 (you > might want to read this from the end :-). No, AFAICT, it's actually not what I meant. The discussion above seems to refer only to the case where a container like "List[X]" may or may not accept subtypes of *X*. For Cython, what matters is whether "list" (or "List[X]") refers to exactly a "list" or allows subtypes of the "list" container type. If the type annotation allows subtypes of builtins, it's useless for Cython, as it does not allow it to generate better code than it does anyway. If it means "exactly list and no subtypes of list", Cython can use it to avoid generating generic fallback code. That's why Cython's current type system enforces exact types when builtin Python types are declared. (You may call it use case optimised: declare only types that help.) And it's one of the reasons why the proposed type system representation is of so little value for Cython. It's simply not intended for anything but type checking. Note that I'm not proposing Cython's semantics for a type system representation that is designed only for type checking, but I guess it would be nice to at least be able to explicitly express somehow that subtypes should be disallowed. >> If there was at least a notion of "unknown types" that tools like static >> type checkers MUST ignore (couldn't find that in the PEP either), then >> Cython users could mix this with Cython's Python level type mapping through >> the "cython" magic module, e.g. something like "cython.struct(x=cython.int >> , y=cython.p_char)". > > Perhaps you could supply a stub for the cython module that declares > everything in it as type 'Any'? That should shut the type checker up. Yes, I guess that would allow Cython's extensions to Python's type system to be ignored. We could even try to be a little smarter and map some C parts of Cython's type system down to more generic types in Python's type system (e.g. int32->int). That would at least give us a minimal baseline level of "interoperability" (as in "it doesn't break and isn't completely useless"). Regarding non-typish declarations (like the contrived doc() example in my pull request), will there be a way (maybe in that "stub" mechanism) to tell type analysis tools that whatever they are looking for, they are not going to find it in this specific kind of annotation, as it's "not a type"? That seems sufficiently different (semantically) from "this represents any type". If it's really left to users to do that at a per-annotation basis, it sounds cumbersome enough to make the real intention appear as saying "annotations are for types only". Stefan From abarnert at yahoo.com Sun Jan 18 21:46:35 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 18 Jan 2015 12:46:35 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CALGmxELGJr9+RMOeo71BRbDZ8r-PFUGh0+5jwy3YyEk=DcUpSg@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <CALGmxELGJr9+RMOeo71BRbDZ8r-PFUGh0+5jwy3YyEk=DcUpSg@mail.gmail.com> Message-ID: <092AFF82-55B9-41C1-AFE7-615661ECE6EA@yahoo.com> On Jan 18, 2015, at 11:39, Chris Barker <chris.barker at noaa.gov> wrote: > I guess the gist here is that I think if Python is going to have an official type hinting system, it should leave the door open to compile time and run-time type checking. A good test case might be: can it be used for Cython? But this thread seems to be indicating that that is not the goal, and indeed, that it's almost an anti-goal. I hope I'm wrong about that. The impression I get--and I hope this isn't just wishful thinking--is that this is an anti-goal for v1 (Python 3.5), but it's very much a goal for v2 (some future version). The issue is that it's a vague and ill-defined goal at the moment. So, there's a tactical argument that explicitly putting all of that off the table, and designing the simplest thing that could work for some purpose (a static type checker) now, is a better way to get to that long-term goal. Meanwhile, the door is open for people to experiment with fitting Cython C types and ORM column types and all kinds of other things into the system, or using it for hinting a static or JIT optimizer, or anything else. Most likely people who try that will find big gaps that they need filled--and that's exactly how we're going to find out what's needed for the v2 type annotation system. From abarnert at yahoo.com Sun Jan 18 22:02:02 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 18 Jan 2015 13:02:02 -0800 Subject: [Python-ideas] Fwd: Way to check for floating point "closeness"? In-Reply-To: <CALGmxELNDKWMRSiiJy_1u89jgYwCSZZAEDHQ3Rkp63J3HAi86g@mail.gmail.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <m9cn9c$i3b$1@ger.gmane.org> <CALGmxE+B5A8E3djzEmv_WwQa2gMMnKhhaX325xyq83mNpFRJvA@mail.gmail.com> <CALGmxEL9vFYmmw=mkDpLR+UmBGcmHCQ_tAPburGu0ak_Ymrfkg@mail.gmail.com> <m9h1fe$4lq$1@ger.gmane.org> <CALGmxELNDKWMRSiiJy_1u89jgYwCSZZAEDHQ3Rkp63J3HAi86g@mail.gmail.com> Message-ID: <45A5BA30-3343-4FC3-A7B0-94B9FCB8A8C5@yahoo.com> On Jan 18, 2015, at 12:14, Chris Barker <chris.barker at noaa.gov> wrote: > There are cases where ULPs, etc are key -- those are for testing precision of algorithms, etc, and I think a special use case that would require a different function -- no need to try to cram it all into one function. I think the right answer there is to add a float_difference that follows the same rules as the IEEE/C/POSIX nextafter function and friends. (And also maybe to add nextafter and friends--note that decimal already has similar functions, but named with underscores.) With that, it's trivial to add ulps-based tests; without it, it's very hard. And then ulps can be removed from the discussion on absolute and relative error. From mertz at gnosis.cx Sun Jan 18 23:18:02 2015 From: mertz at gnosis.cx (David Mertz) Date: Sun, 18 Jan 2015 14:18:02 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <092AFF82-55B9-41C1-AFE7-615661ECE6EA@yahoo.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <CALGmxELGJr9+RMOeo71BRbDZ8r-PFUGh0+5jwy3YyEk=DcUpSg@mail.gmail.com> <092AFF82-55B9-41C1-AFE7-615661ECE6EA@yahoo.com> Message-ID: <CAEbHw4ZKH_BeG6USdQcWdEHTp2pdcKBqAp=U8xy5Bhm7FhkPNg@mail.gmail.com> It occurs to me that it wouldn't be that difficult to create a tool called 'python-annotations-to-cython' (maybe a shorter name like 'py2pyx') that would take whatever type annotation information was contained in a .py file and generate a .pyx file with cython types added in. On Sun, Jan 18, 2015 at 12:46 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > On Jan 18, 2015, at 11:39, Chris Barker <chris.barker at noaa.gov> wrote: > > > I guess the gist here is that I think if Python is going to have an > official type hinting system, it should leave the door open to compile time > and run-time type checking. A good test case might be: can it be used for > Cython? But this thread seems to be indicating that that is not the goal, > and indeed, that it's almost an anti-goal. I hope I'm wrong about that. > > The impression I get--and I hope this isn't just wishful thinking--is that > this is an anti-goal for v1 (Python 3.5), but it's very much a goal for v2 > (some future version). > > The issue is that it's a vague and ill-defined goal at the moment. So, > there's a tactical argument that explicitly putting all of that off the > table, and designing the simplest thing that could work for some purpose (a > static type checker) now, is a better way to get to that long-term goal. > > Meanwhile, the door is open for people to experiment with fitting Cython C > types and ORM column types and all kinds of other things into the system, > or using it for hinting a static or JIT optimizer, or anything else. Most > likely people who try that will find big gaps that they need filled--and > that's exactly how we're going to find out what's needed for the v2 type > annotation system. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150118/cfc84785/attachment.html> From cfkaran2 at gmail.com Sun Jan 18 23:24:51 2015 From: cfkaran2 at gmail.com (Cem Karan) Date: Sun, 18 Jan 2015 17:24:51 -0500 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> Message-ID: <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> On Jan 18, 2015, at 9:16 AM, Chris Angelico <rosuav at gmail.com> wrote: > On Mon, Jan 19, 2015 at 12:37 AM, Cem Karan <cfkaran2 at gmail.com> wrote: >> There may be other uses of annotations that programmers haven't yet thought of, simply because they don't know that annotations exist. Locking out those uses would suck. > > I see a lot of vague "but what if there's something else", and not a > lot of concrete "here's a use of annotations that would be locked > out". Does anyone actually have another use case that would be > seriously harmed by this kind of conflict? Remember, you can simply > not use type hints for the function(s) that use other annotations; the > conflict is only if you're trying to use both on the same function. def foo(a: {Type(int), Doc("some doc")}, b): pass vs. def foo(a, b): """ :param a: some doc :type a: int """ pass With annotations, the documentation lives with the argument. If typing rules all, then you might accidentally have the following: def foo(a: int, b): """ :param a: some doc :type a: str <- Whoops! Type conflict! """ pass Or the ugly but workable: def foo(a: int, b): """ :param a: some doc """ pass I'm arguing that with annotations we can put a lot more information in about a given function argument/return value than just typing. Typing and documentation are two things that I can think of off the top of my head. I can't think of others off of the top of my head, but I suspect that is due to my lack of imagination and not the lack of potential. Thanks, Cem Karan From robertc at robertcollins.net Mon Jan 19 00:41:25 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 19 Jan 2015 12:41:25 +1300 Subject: [Python-ideas] Extending traceback to (optionally) format and show locals. In-Reply-To: <1417383024.1352065.197040833.4FFABE62@webmail.messagingengine.com> References: <CAJ3HoZ3it8x2KTyuOS5fz7zdDbjfi0hSat=VNc-NvC8BFa=WwA@mail.gmail.com> <A02DE549-9800-41B9-938E-132D570BE226@yahoo.com> <CAJ3HoZ1A2SDN5NMKRrCjOV35eNFZmWs_d8SzqTt8A_XNfuP2dw@mail.gmail.com> <1417383024.1352065.197040833.4FFABE62@webmail.messagingengine.com> Message-ID: <CAJ3HoZ0_z3ftb0DqrvAdShFqgZSzOPsB-11RV41S2eDhX1PW-w@mail.gmail.com> On 1 December 2014 at 10:30, <random832 at fastmail.us> wrote: > > > On Thu, Nov 27, 2014, at 19:48, Robert Collins wrote: >> On 27 November 2014 at 14:12, Andrew Barnert <abarnert at yahoo.com> wrote: >> > On Nov 26, 2014, at 15:45, Robert Collins <robertc at robertcollins.net> wrote: >> >> >> I'm sure there is code out there that depends on the quadruple nature >> >> of extract_stack though, so I think we need to preserve that. Three >> >> strategies occured to me; one is to have parallel functions, one >> >> quadruple, one quintuple. A second one is to have the return value of >> >> extract_stack be a quintuple when a new keyword parameter >> >> include_locals is included. Lastly, and this is my preferred one, if >> >> we return a tuple subclass with an attribute containing a dict with >> >> the rendered data on the locals; this can be present but None, or even >> >> just absent when extract_stack was not asked to include locals. >> > >> > There are lots of other cases in the stdlib where something is usable as a tuple of n fields or as a structseq/namedtuple of >n fields: stat results, struct_tm, etc. So, why not do the same thing here? >> >> Because backwards compatibility. Moving to a namedtuple is fine - >> changing the length of the tuple is a problem. > > Er, but what is being suggested is to do the same backwards-compatible > thing: move to a namedtuple-like object with extra non-tuple fields, > just like those others. I'm confused as to what is the conflict here. The thing I was missing is that Andrew was referring to a C only API - AFAICT there is no Python equivalent to PyStructSequence (other than implementing __len__ etc oneself - which is fine, but its not structseq then, AIUI. NamedTuple would imply changing the length - and there's no reason to reimplement traceback as C, so I'd rather not do that. Anyhow, looks like there is a strong desire for a fresh API anyway in 17911, so I'm just going to do that. -Rob -- Robert Collins <rbtcollins at hp.com> Distinguished Technologist HP Converged Cloud From cfkaran2 at gmail.com Mon Jan 19 00:47:09 2015 From: cfkaran2 at gmail.com (Cem Karan) Date: Sun, 18 Jan 2015 18:47:09 -0500 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> Message-ID: <91CF5079-427D-407E-975D-B5C3F6201BE0@gmail.com> On Jan 18, 2015, at 2:34 PM, Guido van Rossum <guido at python.org> wrote: > On Sun, Jan 18, 2015 at 5:37 AM, Cem Karan <cfkaran2 at gmail.com> wrote: > >> On Jan 17, 2015, at 9:27 PM, Ben Finney <ben+python at benfinney.id.au> wrote: >> >> > Cem Karan <cfkaran2 at gmail.com> writes: >> > >> >> [?] I feel like annotations have the capability to be MUCH more useful >> >> than just as a place to hold type hints. >> > >> > They have that capability. How long do you propose we wait for this >> > capability to be realised? >> > >> > Function annotations have been part of Python since version 3.0, and >> > Guido (IIUC) is deciding that the wait is over: we already have the best >> > use of function annotations we're going to get. >> >> I understand what you're saying, and you're right that they are not used much. I only learned about them after I started playing with Python 3.3. But then I'm the bleeding edge for my group at work; everyone else is down in the python 2 world somewhere (some of them are WAY down there somewhere). >> >> Changing a language takes time. Do you program in C? Do you still use pthreads? Why? C11 specifies the thread.h header, you should switch to C11 threads right now! Except... many compilers still don't fully support C11, so we're coding to an earlier standard. There is a lot of code that already works as it is. Etc., etc. etc... >> >> When a language changes, there is a LOT of catching up to do. First, tools need to support the new facilities and then programmers need to become aware of new facilities, and they have to realize there is a legitimate use for those facilities. Most people I know are unaware that annotations exist; that is an argument for making the change to supporting typing only, simply because there will be fewer people that are affected. At the same time, if someone has a really good idea that could be layered into annotations WELL, they will be hampered by a proposal that locks them out. If typing is done via a Typing class like I mentioned in my last email, then there is a very simple way of determining if we're looking at a type or at something else. There may be other, better ways of layering it in that don't lock other uses out, and which are better suited for static analysis. >> >> I'm just asking for a very simple, very clear method of distinguishing uses when parsing code. Types can get complicated quickly, and ad-hoc rules to determine if we're looking at a type, or at something else, can get us in trouble. Simply turning it on or off for a chunk of code also sucks; I LIKE types, so I'd like to put them into my code, but if I have a way of adding documentation to my code as well, then I'd like to do that in addition to the types. There may be other uses of annotations that programmers haven't yet thought of, simply because they don't know that annotations exist. Locking out those uses would suck. >> >> Does all this make sense? > > I hear you, and I'm not buying it. I've thought about this a lot (more than you, I'm sure) and I'm putting a line in the sand: annotations are for types, and all other uses will eventually have to find some other way (most likely decorators). (The "type: OFF" comments are a transitional measure to avoid breaking other uses without warning in 3.5.) As the BDFL, that is your right. But I would personally rather see this: @functools.lru_cache() def foo(a: {Type(int), Doc("some doc"), Something_else(4,5,6)}): """ Some block of documentation for the function as a whole that I'm too lazy to write out, so I'm just going to copy in a paragraph of `Lorem ipsum <http://en.wikipedia.org/wiki/Ipse_lorum>`_ to add as filler to break up the mental flow a bit. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. """ pass Then something like this: @functools.lru_cache() @Something_else(a, 4, 5, 6) def foo(a: int): """ Some block of documentation for the function as a whole that I'm too lazy to write out, so I'm just going to copy in a paragraph of `Lorem ipsum <http://en.wikipedia.org/wiki/Ipse_lorum>`_ to add as filler to break up the mental flow a bit. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. :param a: some doc """ pass Finally, consider what decorators are NOT required to do; they are not required to return the original object. That means that ordering can potentially matter: @Doc(a, "some doc") # Documenting the argument of the wrapped function @functools.lru_cache() # Creating a new function that wraps the original function @Something_else(a, 4, 5, 6) # Doing something to the argument of the original function def foo(a: int): pass versus @functools.lru_cache() # Wrapping the function @Something_else(a, 4, 5, 6) # Doing something to the argument of the original function @Doc(a, "some doc") # Documenting the argument of the function itself. def foo(a: int): pass So what do automatic documentation generators do in this case? More than likely, we want the second case, which means that someone needs to tell programmers what order to use the decorators in. What if Something_else returns a wrapped version of the function? You need to do this: @functools.lru_cache() # Wrapping the modified function @Something_else(a, 4, 5, 6) # Doing something to the argument of the modified function @Something_else(b, 4, 5, 6) # Doing something to the argument of the original function @Doc(a, "some doc") # Documenting the argument of the function itself. @Doc(b, "some other doc") # Documenting the argument of the function itself. def foo(a: int, b: str): pass instead of this: @functools.lru_cache() # Wrapping the modified, modified function @Doc(a, "some doc") # Documenting the argument of the modified version of the modified function. @Something_else(a, 4, 5, 6) # Doing something to the argument of the modified function @Doc(b, "some other doc") # Documenting the argument of the modified function. @Something_else(b, 4, 5, 6) # Doing something to the argument of the original function def foo(a: int, b: str): pass This starts to feel unnatural, and is definitely confusing. We can then put in place a whole bunch of rules regarding decorators, but then we've got two problems; forcing annotations to be for types only, and forcing new rules on decorators that weren't there in the first place. I know that something like "def foo(a: {Type(int), Doc("some doc"), Something_else(4,5,6)})" isn't pretty. But it is order-independent, it operates on the original function only, and it is pretty easy to parse, even for a static analyzer, AND there is no question that Type() means you want a type check done. Finally, annotations will still be open for other uses. Thanks, Cem Karan From abarnert at yahoo.com Mon Jan 19 01:08:37 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 18 Jan 2015 16:08:37 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> Message-ID: <3E9464FE-7D9A-43C6-A830-145A8A1D6474@yahoo.com> On Jan 18, 2015, at 14:24, Cem Karan <cfkaran2 at gmail.com> wrote: > On Jan 18, 2015, at 9:16 AM, Chris Angelico <rosuav at gmail.com> wrote: > >> On Mon, Jan 19, 2015 at 12:37 AM, Cem Karan <cfkaran2 at gmail.com> wrote: >>> There may be other uses of annotations that programmers haven't yet thought of, simply because they don't know that annotations exist. Locking out those uses would suck. >> >> I see a lot of vague "but what if there's something else", and not a >> lot of concrete "here's a use of annotations that would be locked >> out". Does anyone actually have another use case that would be >> seriously harmed by this kind of conflict? Remember, you can simply >> not use type hints for the function(s) that use other annotations; the >> conflict is only if you're trying to use both on the same function. > > def foo(a: {Type(int), Doc("some doc")}, b): > pass > > vs. > > def foo(a, b): > """ > :param a: some doc > :type a: int > """ > pass > > With annotations, the documentation lives with the argument. If typing rules all, then you might accidentally have the following: > > def foo(a: int, b): > """ > :param a: some doc > :type a: str <- Whoops! Type conflict! > """ > pass > > Or the ugly but workable: > > def foo(a: int, b): > """ > :param a: some doc > """ > pass What's ugly about that? Besides being consistent with a wide variety of other languages, it puts signature information in the signature and documentation in the docstring, which seems like the obvious right place for everything. > I'm arguing that with annotations we can put a lot more information in about a given function argument/return value than just typing. Denser doesn't always mean better. One piece of simple information is easy to read; a lot more information implies a lot more mental processing. Plus, you're ignoring the fact that you're substantially increasing boilerplate, to the point where I think you're not even increasing brevity and density in the first place. For the (hopefully common) case of simple types, having to stick the type in a constructor and then stick that constructor in a set vastly increases how much space it takes to represent the type, and how much mental effort it takes to process it. Just compare: def foo(a: int, b): def foo(a: {Type(int)}, b): That's 160% boilerplate, and it includes two levels of punctuated syntax that pushes the limits of what you can read and process without having to think it through--and that's the simplest possible case. If we did things your way, I think people would avoid annotating simple cases, and clamor for a way to hide other people's annotations when viewing help. > Typing and documentation are two things that I can think of off the top of my head. I can't think of others off of the top of my head, but I suspect that is due to my lack of imagination and not the lack of potential. Don't sell yourself short. They're the exact same two things that Collin Winter and the collective members of the python-3000 list came up with back in mid-2006 on the way to PEP 3107. And the same two things people have come up with in the nearly decade since. So either there is no obvious wider potential, or the whole Python community is short on imagination. Obviously PEP 484's type hints are a narrower subset of all the potential kinds of typing information suggested in PEP 3107, but if the question is how to unify multiple different typing systems, I think it's pretty clear that a general "lots of separate things in parallel" solution isn't going to be the answer. (Would anyone want to see {typing.Type(int), cython.Type(cython.int32), xrpc.Type(xrpc.JSONNumber)} as an annotation?) From cfkaran2 at gmail.com Mon Jan 19 02:25:52 2015 From: cfkaran2 at gmail.com (Cem Karan) Date: Sun, 18 Jan 2015 20:25:52 -0500 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <3E9464FE-7D9A-43C6-A830-145A8A1D6474@yahoo.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <3E9464FE-7D9A-43C6-A830-145A8A1D6474@yahoo.com> Message-ID: <4177391A-E1F7-4AD8-A56E-EED92A6F67E8@gmail.com> On Jan 18, 2015, at 7:08 PM, Andrew Barnert <abarnert at yahoo.com> wrote: > On Jan 18, 2015, at 14:24, Cem Karan <cfkaran2 at gmail.com> wrote: > >> On Jan 18, 2015, at 9:16 AM, Chris Angelico <rosuav at gmail.com> wrote: >> >>> On Mon, Jan 19, 2015 at 12:37 AM, Cem Karan <cfkaran2 at gmail.com> wrote: >>>> There may be other uses of annotations that programmers haven't yet thought of, simply because they don't know that annotations exist. Locking out those uses would suck. >>> >>> I see a lot of vague "but what if there's something else", and not a >>> lot of concrete "here's a use of annotations that would be locked >>> out". Does anyone actually have another use case that would be >>> seriously harmed by this kind of conflict? Remember, you can simply >>> not use type hints for the function(s) that use other annotations; the >>> conflict is only if you're trying to use both on the same function. >> >> def foo(a: {Type(int), Doc("some doc")}, b): >> pass >> >> vs. >> >> def foo(a, b): >> """ >> :param a: some doc >> :type a: int >> """ >> pass >> >> With annotations, the documentation lives with the argument. If typing rules all, then you might accidentally have the following: >> >> def foo(a: int, b): >> """ >> :param a: some doc >> :type a: str <- Whoops! Type conflict! >> """ >> pass >> >> Or the ugly but workable: >> >> def foo(a: int, b): >> """ >> :param a: some doc >> """ >> pass > > What's ugly about that? Besides being consistent with a wide variety of other languages, it puts signature information in the signature and documentation in the docstring, which seems like the obvious right place for everything. Ugly in the sense that information is spread out. That said, you are right that it is more consistent with what other languages do. >> I'm arguing that with annotations we can put a lot more information in about a given function argument/return value than just typing. > > Denser doesn't always mean better. One piece of simple information is easy to read; a lot more information implies a lot more mental processing. You're right about that, it can be more difficult. However, hunting for information can also be difficult. > Plus, you're ignoring the fact that you're substantially increasing boilerplate, to the point where I think you're not even increasing brevity and density in the first place. For the (hopefully common) case of simple types, having to stick the type in a constructor and then stick that constructor in a set vastly increases how much space it takes to represent the type, and how much mental effort it takes to process it. Just compare: > > def foo(a: int, b): > > def foo(a: {Type(int)}, b): > > That's 160% boilerplate, and it includes two levels of punctuated syntax that pushes the limits of what you can read and process without having to think it through--and that's the simplest possible case. If we did things your way, I think people would avoid annotating simple cases, and clamor for a way to hide other people's annotations when viewing help. You're right about that. It IS a lot uglier. But like I said, its one saving grace is that it is clear what the annotation is; it is a type, and it is intended for type checking. There isn't any guessing, and there is very fine-grained control over where and when type checking is done. As an alternative, if all basic types had equivalents in the typing library, you could skip the container above. In short: from typing import Int def foo(a: Int, b): # Simple type checking. def foo(a: {Int, Doc("some doc")}, b): # Something more clever. This reduces the boilerplate while still making it possible to determine if we're looking at a type check or something else. >> Typing and documentation are two things that I can think of off the top of my head. I can't think of others off of the top of my head, but I suspect that is due to my lack of imagination and not the lack of potential. > > Don't sell yourself short. They're the exact same two things that Collin Winter and the collective members of the python-3000 list came up with back in mid-2006 on the way to PEP 3107. And the same two things people have come up with in the nearly decade since. So either there is no obvious wider potential, or the whole Python community is short on imagination. > > Obviously PEP 484's type hints are a narrower subset of all the potential kinds of typing information suggested in PEP 3107, but if the question is how to unify multiple different typing systems, I think it's pretty clear that a general "lots of separate things in parallel" solution isn't going to be the answer. (Would anyone want to see {typing.Type(int), cython.Type(cython.int32), xrpc.Type(xrpc.JSONNumber)} as an annotation?) Only in sort of morbid fascination, kind of like the winners of the obfuscated C code contest... Thanks, Cem Karan From steve at pearwood.info Mon Jan 19 02:58:39 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 19 Jan 2015 12:58:39 +1100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> Message-ID: <20150119015839.GT18556@ando.pearwood.info> On Sun, Jan 18, 2015 at 05:24:51PM -0500, Cem Karan wrote: > def foo(a: {Type(int), Doc("some doc")}, b): > pass I dislike this, because there is too much in the function signature. I could live with this: def foo(a:int, b): pass or *possibly* this: def foo(a:"some doc", b): pass but not both together. Combining the two doesn't look too bad for a toy example like made-up "foo", but think about a real-world example: # from the statistics module def median_grouped( data: {Type(Iterable[Real]), Doc("grouped continuous data")}, interval: {Type(Real), Doc("class interval")} = 1 ) -> {Type(Real), Doc( "50th percentile (median) of grouped continuous data")}: pass That may be machine-readable, but it is too complicated for me to read, and I wrote it! And that's only a simple function with a mere two parameters and simple one-liner docs. Having written that, I still need to write a doc string, and almost certainly I have to duplicate the annotation docs inside the docstring. So it isn't really saving me much, if anything, over this: @Doc({"data": "grouped continuous data", "interval": "class interval", "return": "50th percentile (median) of grouped continuous data" }) def median_grouped(data:Iterable[Real], interval:Type(Real)=1)->Real: pass It's not *ideal* to have to duplicate the parameter names in the decorator, but except in the most trivial toy cases, I can't have it all: - type annotations - and document annotations - without being too verbose or complex - and still easy to read and write and maintain Something has to give, and since type-hinting is deemed to be the primary use for annotations, it has to be the document annotations. You can still have both type-hints and docs, you just can't use annotations for them both. (But see below.) - If you want to use annotations for documentation, then you cannot use them for type-hints at the same time; - if you want to use annotations for type-hints, then you need another method of dealing with the documentation. Since you almost certainly will need to write a docstring anyway, the best solution in my mind is to move the documentation into the docstring: def median_grouped(data:Iterable[Real], interval:Type(Real)=1)->Real: """Return the 50th percentile (median) of grouped continuous data. :param a: grouped continuous data :param interval: class interval more documentation and docstrings follows... """ > With annotations, the documentation lives with the argument. If > typing rules all, then you might accidentally have the following: > > def foo(a: int, b): > """ > :param a: some doc > :type a: str <- Whoops! Type conflict! > """ > pass And that is best solved by having your linter understand the convention for docstrings and flag the error. > I'm arguing that with annotations we can put a lot more information in > about a given function argument/return value than just typing. Of course you can. But I think you shouldn't. You are absolutely free to disagree. Remember, use of the type-checker is optional, and the type-checker itself will be a third-party tool. Tools actually. I expect there will be much competition to develop the best, most powerful type-checker. There is nothing stopping people (including yourself) writing a type-checker which supports your multiple-annotations-in-a-set idea. If it proves itself as a viable, and popular, alternative, then you can come back and try to convince Guido to give it his blessing as officially sanctioned (at which point all the other type-checkers will have to support sets of annotations as well). I *do* believe that it is technically possible to have the type-checker support your {Type(...), Spam(...)} idea, but I *don't* believe that it will be practical or popular. I think it is a case of YAGNI combined with EIYDNIYSDI (Even If You Do Need It You Shouldn't Do It). But I might be wrong, and I encourage you to prove me wrong the only way that really matters: with working code that gets used in the real world. -- Steven From chris.barker at noaa.gov Mon Jan 19 03:09:06 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Sun, 18 Jan 2015 18:09:06 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAEbHw4ZKH_BeG6USdQcWdEHTp2pdcKBqAp=U8xy5Bhm7FhkPNg@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <CALGmxELGJr9+RMOeo71BRbDZ8r-PFUGh0+5jwy3YyEk=DcUpSg@mail.gmail.com> <092AFF82-55B9-41C1-AFE7-615661ECE6EA@yahoo.com> <CAEbHw4ZKH_BeG6USdQcWdEHTp2pdcKBqAp=U8xy5Bhm7FhkPNg@mail.gmail.com> Message-ID: <-6032810226209247011@unknownmsgid> On Jan 18, 2015, at 2:18 PM, David Mertz <mertz at gnosis.cx> wrote: It occurs to me that it wouldn't be that difficult to create a tool called 'python-annotations-to-cython' (maybe a shorter name like 'py2pyx') that would take whatever type annotation information was contained in a .py file and generate a .pyx file with cython types added in. It shouldn't be...and might be a good test of the typing system. Good idea. -Chris On Sun, Jan 18, 2015 at 12:46 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > On Jan 18, 2015, at 11:39, Chris Barker <chris.barker at noaa.gov> wrote: > > > I guess the gist here is that I think if Python is going to have an > official type hinting system, it should leave the door open to compile time > and run-time type checking. A good test case might be: can it be used for > Cython? But this thread seems to be indicating that that is not the goal, > and indeed, that it's almost an anti-goal. I hope I'm wrong about that. > > The impression I get--and I hope this isn't just wishful thinking--is that > this is an anti-goal for v1 (Python 3.5), but it's very much a goal for v2 > (some future version). > > The issue is that it's a vague and ill-defined goal at the moment. So, > there's a tactical argument that explicitly putting all of that off the > table, and designing the simplest thing that could work for some purpose (a > static type checker) now, is a better way to get to that long-term goal. > > Meanwhile, the door is open for people to experiment with fitting Cython C > types and ORM column types and all kinds of other things into the system, > or using it for hinting a static or JIT optimizer, or anything else. Most > likely people who try that will find big gaps that they need filled--and > that's exactly how we're going to find out what's needed for the v2 type > annotation system. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150118/d48c6b64/attachment.html> From chris.barker at noaa.gov Mon Jan 19 03:16:02 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Sun, 18 Jan 2015 18:16:02 -0800 Subject: [Python-ideas] Fwd: Way to check for floating point "closeness"? In-Reply-To: <45A5BA30-3343-4FC3-A7B0-94B9FCB8A8C5@yahoo.com> References: <CALGmxEKvomxhGyxOErV9WPHTdVxZ+VtTG1MUsqQgVJuMS8yQ=A@mail.gmail.com> <m9cn9c$i3b$1@ger.gmane.org> <CALGmxE+B5A8E3djzEmv_WwQa2gMMnKhhaX325xyq83mNpFRJvA@mail.gmail.com> <CALGmxEL9vFYmmw=mkDpLR+UmBGcmHCQ_tAPburGu0ak_Ymrfkg@mail.gmail.com> <m9h1fe$4lq$1@ger.gmane.org> <CALGmxELNDKWMRSiiJy_1u89jgYwCSZZAEDHQ3Rkp63J3HAi86g@mail.gmail.com> <45A5BA30-3343-4FC3-A7B0-94B9FCB8A8C5@yahoo.com> Message-ID: <-8076813069332448647@unknownmsgid> > On Jan 18, 2015, at 1:02 PM, Andrew Barnert <abarnert at yahoo.com> wrote:. > > I think the right answer there is to add a float_difference that follows the same rules as the IEEE/C/POSIX nextafter function and friends. (And also maybe to add nextafter and friends--note that decimal already has similar functions, but named with underscores.) With that, it's trivial to add ulps-based tests; without it, it's very hard. > > And then ulps can be removed from the discussion on absolute and relative error. Sounds good to me. -CHB From guido at python.org Mon Jan 19 04:28:35 2015 From: guido at python.org (Guido van Rossum) Date: Sun, 18 Jan 2015 19:28:35 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CALGmxELGJr9+RMOeo71BRbDZ8r-PFUGh0+5jwy3YyEk=DcUpSg@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <CALGmxELGJr9+RMOeo71BRbDZ8r-PFUGh0+5jwy3YyEk=DcUpSg@mail.gmail.com> Message-ID: <CAP7+vJLyMHw2Xzy6Bum5b9=T+pPaMZpcSLVzj5R+j3+DZgm9gg@mail.gmail.com> On Sun, Jan 18, 2015 at 11:39 AM, Chris Barker <chris.barker at noaa.gov> wrote: > On Fri, Jan 16, 2015 at 12:08 PM, Guido van Rossum <guido at python.org> > wrote: > >> Yes, the PEP tries to gently push an agenda where annotations are only >> used for Python type hints. >> > > Type hints, as opposed to arbitrary other uses, or type _hints_ as > aopposed to run-time of compile time static typing? > Um, what did you mean by "run-time of compile time static typing"? I'm guessing "or" instead of "of". With that assumption, I think the answer is both. I want annotations to be (eventually) reserved for a standard type notation. The PEP focuses on the notation. The (by far largest) use case I see for the notation is running a separate off-line type checker, which acts like a linter but with much more understanding of types. (But not perfect!) A second use case (for the same notation!) is equally off-line, to provide information an IDE needs so it can give accurate suggestions etc. Some use cases that people (including myself) have often assumed but which I now find mostly anti-ideas: efficient code generation; run-time type checking; exact type checking by the Python interpreter when it compiles and loads modules. > Cython, providing both of those, for example. > > I don't know what to answer for Cython -- doesn't it have its own syntax >> for declaring C types? >> > > yes: > > cdef int i > > or: > > def func( int i ). > > There is also a decorator-based system, for "pure python > > But I think it would be nice it it could use the same syntax as other > systems, particularly if there is an official one. > I think it would be nice too. But what I hear from Stefan is that Cython's requirements are not a good fit for the proposal. And I believe it makes more sense to let Cython continue to do its own thing rather than trying to either expand the proposal so Cython can use it without change, or allow for alternate type notations so Cython can do its own thing. The latter however sounds closer to what's possible. > If Cython uses annotations with different meanings, perhaps for Cython >> files the Python annotations could be put in stub files. >> > > I think we're good there. > > But I think Stephan's issues is that currently cython allows both C and > Python type names: > > cdef int i > cdef list a_list > > but where the names conflict, the C name is assumed (int, float, others?) > > Given C's typedef, I'd say we should just have Cython use unique names -- > i.e. c_int -> int, etc. But that all requires that no one decides to write > a new python type called c_int. Though I'm not sure it's really a problem > to reserve names for Cython. > Everything in the proposal has a namespace (either 'typing' or 'builtins') and the checker is expected to understand and follow imports properly. So I don't see a problem here. > Anyway, I guess it's unclear to me whether the intent is to create a > system that can be used for a static/run-time type checker like Cython. > That is not the intent. (Though static and run-time seem two very different forms of type-checking. Apparently Cython does both?) > There have been a fair number of comments alone the lines of "this is only > for lint-like static analysis". But I'd really love to see this move toward > full functionailty. > Once this proposal is in place you may try to adapt it for that purpose. But I'm not holding my breath, and I think we're better off with an imperfect proposal that can make it into 3.5 than with waiting until the ideal all-encompassing proposal is written. > In fact, after reading the PEPs, I still don't really see the point of > static type analysis that does not lead to compile time or run-time type > checking. Maybe someone could provide the motivation for that, oradd alink > to the PEP where that is. > > My question: > > I see two clear advantages to static typing: type safety and performance. > But in the case of Python, the performance thing is entirely unproven, and the focus on it is a hindrance to trying to get any type system at all introduced. > As a proponent of Python (and thus dynamic typing), I have argued a lot > that static type checking does indeed catch a lot of bugs, but they are > generally shallow bugs -- easy to test for, and easy to understand and fix > -- thus I don't much need that. However, I can see how it's a "good thing", > and maybe important for "Enterprise" systems. > Nobody writes enough tests though, and type checking catches a different category of issues (shallow or not) than tests. > On the other hand, dynamic typing does impact performance in a way that > effects my code -- so I tun to Cython (and sometimes C, C++, Fortran) to > address those issues. (maybe JIT compilation will one day meet this need, > but it hasn't yet). > > So if there is type specifying, I want i to solve these problems. > How? > But where I'm lost is the utility of static analysis that is not used to > either provide run-time type checking OR compile optimized code (either JIT > or ahead-of-time). It seems that your entire system would have to be fully > type-hinted to get the type safety. But once you've done that, why not just > use a static language? > This is the problem of the traditional focus on type-safety. The traditional idea is that if you can't prove that the entire program is type-safe you should not run any of it (because the entire program is "broken"), and type-checking just part of the program is useless because you still don't know whether the program is "correct". (This is also often combined with the inability to generate code for an incorrect program.) However, even "correct" programs often contain bugs, because the correctness of a program depends on more than its type-safety. (Almost every program has some constraints that cannot be expressed in the language's type system.) However, gradual typing, for dynamically-typed languages, takes a different position. It claims that if you can type-check *some* of your program, the added type safety for that part helps find bugs. It's not a black or white situation. Just like a linter will find some bugs (after you wade through the false positives, tuning the linter options locally and globally to get a decent yield), a type checker of the kind proposed here will find some bugs (after you spend some time adding type annotations to part of the program). This approach works much better for dynamic languages than for static languages, because a dynamic language can still run a program that doesn't type-check correctly. (And for a variety of reasons, the program may still work well enough.) On the other hand, if run-time type checking is provided, then you could > use that at the boundaries of your code base (kind of like Cython does) and > get both the safety and performance benifits. > But it sounds like it is Cython's job to do the checking at these boundaries. Isn't this similar to the way Python extensions written in C or C++ have to do type checking at the boundary between Python code and the extension? The specification of these checks should be defined by Cython. > I've read the MyPY FAQ, which does address these issues to some extent, > but doesn't really make a compelling case to me, anyway. > It did to me. :-) > > I guess the gist here is that I think if Python is going to have an > official type hinting system, it should leave the door open to compile time > and run-time type checking. A good test case might be: can it be used for > Cython? But this thread seems to be indicating that that is not the goal, > and indeed, that it's almost an anti-goal. I hope I'm wrong about that. > TBH I think the question of whether it can be used for Cython is still very much open. Why couldn't Cython take code that says a certain parameter has type List[int] and generate code from it? (Though no matter how you express this, I don't see how this could help much, since the input, being generated by regular Python code, would have to be a builtins.list object full of builtins.int objects. At that point there isn't much that Cython can do to speed things up.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150118/ff3136d8/attachment-0001.html> From guido at python.org Mon Jan 19 04:40:38 2015 From: guido at python.org (Guido van Rossum) Date: Sun, 18 Jan 2015 19:40:38 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <m9h5gk$2a6$1@ger.gmane.org> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> Message-ID: <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> On Sun, Jan 18, 2015 at 12:36 PM, Stefan Behnel <stefan_ml at behnel.de> wrote: > Guido van Rossum schrieb am 18.01.2015 um 05:11: > > On Sat, Jan 17, 2015 at 1:43 AM, Stefan Behnel wrote: > >> Guido van Rossum schrieb am 16.01.2015 um 21:08: > >>> I don't know what to answer for Cython -- doesn't it have its own > syntax > >>> for declaring C types? Or is it now also using annotations? > >> > >> It can, but it's optional. When I added support for it, I took care not > to > >> enable it by default, so that it wouldn't accidentally interfere with > other > >> usages of annotations. > >> > >> The current proposal seems less generous in this regard, and the way I > see > >> it, the only gain from Cython's point of view is its very limited > support > >> for container item type declarations and function signatures of callable > >> objects. Both are restricted to Python object types and even imply > slightly > >> different semantics than Cython (I didn't see a notion of "exact types" > as > >> opposed to "type or subtype"). > > > > Actually, while the PEP is still lacking in clarity and we're still > having > > some discussions, type variables will most likely be "invariant" by > > default, which I think is the "exact types" notion you are after. There's > > lots of discussion in https://github.com/ambv/typehinting/issues/2 (you > > might want to read this from the end :-). > > No, AFAICT, it's actually not what I meant. The discussion above seems to > refer only to the case where a container like "List[X]" may or may not > accept subtypes of *X*. For Cython, what matters is whether "list" (or > "List[X]") refers to exactly a "list" or allows subtypes of the "list" > container type. If the type annotation allows subtypes of builtins, it's > useless for Cython, as it does not allow it to generate better code than it > does anyway. If it means "exactly list and no subtypes of list", Cython can > use it to avoid generating generic fallback code. Can you explain how this works? How does Cython manage to use the knowledge that the argument's __class__ is equal to builtins.list to generate more efficient code? Also, don't you have to insert dynamic checks to ensure the caller passes exactly a list anyway? Very few people subclass builtins.list (usually it's much simpler to subclass collections.abc.MutableSequence). So perhaps Cython could just assume that typing.List means builtins.list and (dynamically) reject calls that pass a subclass of builtins.list? > That's why Cython's > current type system enforces exact types when builtin Python types are > declared. (You may call it use case optimised: declare only types that > help.) And it's one of the reasons why the proposed type system > representation is of so little value for Cython. It's simply not intended > for anything but type checking. > > Note that I'm not proposing Cython's semantics for a type system > representation that is designed only for type checking, but I guess it > would be nice to at least be able to explicitly express somehow that > subtypes should be disallowed. > Can you come up with a specific proposal? Maybe we're not as far apart as we think. :-) > >> If there was at least a notion of "unknown types" that tools like static > >> type checkers MUST ignore (couldn't find that in the PEP either), then > >> Cython users could mix this with Cython's Python level type mapping > through > >> the "cython" magic module, e.g. something like "cython.struct(x= > cython.int > >> , y=cython.p_char)". > > > > Perhaps you could supply a stub for the cython module that declares > > everything in it as type 'Any'? That should shut the type checker up. > > Yes, I guess that would allow Cython's extensions to Python's type system > to be ignored. We could even try to be a little smarter and map some C > parts of Cython's type system down to more generic types in Python's type > system (e.g. int32->int). That would at least give us a minimal baseline > level of "interoperability" (as in "it doesn't break and isn't completely > useless"). > Right. I still don't understand enough about Cython's type system to be able to make useful suggestions myself -- I am really hoping that you'll take the time to understand PEP 484 well enough so you can help guide it towards something that would benefit Cython without giving up the existing goals (which are similar to what is done e.g. in Hack and Typescript -- see PEP 482). > Regarding non-typish declarations (like the contrived doc() example in my > pull request), will there be a way (maybe in that "stub" mechanism) to tell > type analysis tools that whatever they are looking for, they are not going > to find it in this specific kind of annotation, as it's "not a type"? That > seems sufficiently different (semantically) from "this represents any > type". If it's really left to users to do that at a per-annotation basis, > it sounds cumbersome enough to make the real intention appear as saying > "annotations are for types only". > We'll start out with a way using magic comments (maybe # type: OFF|ON) to disable the interpretation of annotations as types by the type checker during specific sections of a module. The checker will then assume no annotations, and in that case it will assume every argument and return value has the special type 'Any'. (Please do read up in PEP 483 about the important difference between Any and object.) We'll also have a decorator to disable type checking per function or class. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150118/91842fbc/attachment.html> From tjreedy at udel.edu Mon Jan 19 07:06:59 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 19 Jan 2015 01:06:59 -0500 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> Message-ID: <m9i6u4$lf$1@ger.gmane.org> On 1/18/2015 10:40 PM, Guido van Rossum wrote: > On Sun, Jan 18, 2015 at 12:36 PM, Stefan Behnel > does anyway. If it means "exactly list and no subtypes of list", > Cython can > use it to avoid generating generic fallback code. > > > Can you explain how this works? How does Cython manage to use the > knowledge that the argument's __class__ is equal to builtins.list to > generate more efficient code? The obvious part is that Cython will directly access the C array of pointers .. and *assume* that each pointer points to a member of the sequence, in order. The less obvious part is how a subclass can break that assumption. Suppose, for instance, one wants a list of pairs, without the space overhead of a concrete tuple object for each pair. Here is the beginning of a list subclass solution (not necessarily the best approach, but a possible one). class PairList(list): def __getitem__(self, i): g = super().__getitem__ return g(2*i), g(2*i+1) pl = PairList([1,2,3,4]) print(pl[0], pl[1]) >>> (1, 2) (3, 4) For Cython to execute this correctly, it would need the subclass fallback code that would go through the custom .__getitem__. Storing a triangular matrix in a list, with __getitem__ calculating the linear index from an input row, col pair, is another possible example. > Very few people subclass builtins.list ... > So perhaps Cython could just assume that typing.List means > builtins.list and (dynamically) reject calls that pass a subclass > of builtins.list? Identifying a triangular matrix as a List might not be too useful in any case. ... > would be nice to at least be able to explicitly express somehow > that subtypes should be disallowed. > Can you come up with a specific proposal? Maybe we're not as far > apart as we think. :-) Is the use of concrete types like list disallowed? --- My memory of the pre-3.0 discussion is that we did not add something like typing then because we did not know how to write it. You suggested (something like) "let people experiment and we can adopt and adapt the best result." Now is the anticipated future. I agree that mypy seems to be the best starting point of something both usable and useful that we are likely to get. It would be nice to have adapted version in an early alpha. I presume you plan to label typing 'experimental', as I believe asyncio was. -- Terry Jan Reedy From chris.barker at noaa.gov Mon Jan 19 07:32:14 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Sun, 18 Jan 2015 22:32:14 -0800 Subject: [Python-ideas] Floating point "closeness" Proposal Outline Message-ID: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> OK folks, There has been a lot of chatter about this, which I think has served to provide some clarity, at least to me. However, I'm concerned that the upshot, at least for folks not deep into the discussion, will be: clearly there are too many use-case specific details to put any one thing in the std lib. But I still think we can provide something that is useful for most use-cases, and would like to propose what that is, and what the decision points are: A function for the math module, called somethign like "is_close", "approx_equal", etc. It will compute a relative tolerance, with a default maybe around 1-e12, with the user able to specify the tolerance they want. Optionally, the user can specify an "minimum absolute tolerance", it will default to zero, but can be set so that comparisons to zero can be handled gracefully. The relative tolerance will be computed from the smallest of the two input values, so as to get symmetry : is_close(a,b) == is_close(b,a). (this is the Boost "strong" definition, and what is used by Steven D'Aprano's code in the statistics test module) Alternatively, the relative error could be computed against a particular one of the input values (the second one?). This would be asymmetric, but be more clear exactly how "relative" is defined, and be closer to what people may expect when using it as a "actual vs expected" test. --- "expected" would be the scaling value. If the tolerance is small, it makes very little difference anyway, so I'm happy with whatever consensus moves us to. Note that if we go this way, then the parameter names should make it at least a little more clear -- maybe "actual" and "expected", rather than x and y or a and b or... and the function name should be something like is_close_to, rather than just is_close. It will be designed for floating point numbers, and handle inf, -inf, and NaN "properly". But is will also work with other numeric types, to the extent that duck typing "just works" (i.e. division and comparisons all work). complex numbers will be handled by: is_close(x.real, y.real) and is_close(x.imag, y.imag) (but i haven't written any code for that yet) It will not do a simple absolute comparison -- that is the job of a different function, or, better yet, folks just write it themselves: abs(x - y) <= delta really isn't much harder to write than a function call: absolute_diff(x,y,delta) Here is a gist with a sample implementation: https://gist.github.com/PythonCHB/6e9ef7732a9074d9337a I need to add more tests, and make the test proper unit tests, but it's a start. I also need to see how it does with other data types than float -- hopefully, it will "just work" with the core set. I hope we can come to some consensus that something like this is the way to go. -Chris On Sun, Jan 18, 2015 at 11:27 AM, Ron Adam <ron3200 at gmail.com> wrote: > > > On 01/17/2015 11:37 PM, Chris Barker wrote: > >> (Someone claimed that 'nothing is close to zero'. This is >> nonsensical both in applied math and everyday life.) >> >> >> I'm pretty sure someone (more than one of use) asserted that "nothing is >> *relatively* close to zero -- very different. >> > > Yes, that is the case. > > > And I really wanted a way to have a default behavior that would do a >> reasonable transition to an absolute tolerance near zero, but I no longer >> thing that's possible. (numpy's implimentaion kind of does that, but it is >> really wrong for small numbers, and if you made the default min_tolerance >> the smallest possible representable number, it really wouldn't be useful. >> > > I'm going to try to summarise what I got out of this discussion. Maybe it > will help bring some focus to the topic. > > I think there are two case's to consider. > > # The most common case. > rel_is_good(actual, expected, delta) # value +- %delta. > > # Testing for possible equivalence? > rel_is_close(value1, value2, delta) # %delta close to each other. > > I don't think they are quite the same thing. > > rel_is_good(9, 10, .1) --> True > rel_is_good(10, 9, .1) --> False > > rel_is_close(9, 10, .1) --> True > rel_is_close(10, 9, .1) --> True > > > In the "is close" case, it shouldn't matter what order the arguments are > given. The delta is the distance from the larger number the smaller number > is. (of the same sign) > > So when calculating the relative error from two values, you want it to be > consistent with the rel_is_close function. > > rel_is_close(a, b, delta) <---> rel_err(a, b) <= delta > > And you should not use the rel_err function in the rel_is_good function. > > > > The next issue is, where does the numeric accuracy of the data, > significant digits, and the languages accuracy (ULPs), come into the > picture. > > My intuition.. I need to test the idea to make a firmer claim.. is that in > the case of is_good, you want to exclude the uncertain parts, but with > is_close, you want to include the uncertain parts. > > Two values "are close" if you can't tell one from the other with > certainty. The is_close range includes any uncertainty. > > A value is good if it's within a range with certainty. And this excludes > any uncertainty. > > This is where taking in consideration of an absolute delta comes in. The > minimum range for both is the uncertainty of the data. But is_close and > is_good do different things with it. > > Of course all of this only applies if you agree with these definitions of > is_close, and is_good. ;) > > Cheers, > Ron > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150118/e2cf0208/attachment-0001.html> From chris.barker at noaa.gov Mon Jan 19 08:00:54 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Sun, 18 Jan 2015 23:00:54 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJLyMHw2Xzy6Bum5b9=T+pPaMZpcSLVzj5R+j3+DZgm9gg@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <CALGmxELGJr9+RMOeo71BRbDZ8r-PFUGh0+5jwy3YyEk=DcUpSg@mail.gmail.com> <CAP7+vJLyMHw2Xzy6Bum5b9=T+pPaMZpcSLVzj5R+j3+DZgm9gg@mail.gmail.com> Message-ID: <CALGmxELbnwQTHYrBCnF-PmR-HHdX5ezeK3FyfskknLA3yEGuDg@mail.gmail.com> On Sun, Jan 18, 2015 at 7:28 PM, Guido van Rossum <guido at python.org> wrote: > Um, what did you mean by "run-time of compile time static typing"? I'm > guessing "or" instead of "of". With that assumption, I think the answer is > both. > sorry for the typo -- yes that was supposed to be "or" I want annotations to be (eventually) reserved for a standard type > notation. The PEP focuses on the notation. The (by far largest) use case I > see for the notation is running a separate off-line type checker, which > acts like a linter but with much more understanding of types. (But not > perfect!) A second use case (for the same notation!) is equally off-line, > to provide information an IDE needs so it can give accurate suggestions etc. > both good things, yes. > Some use cases that people (including myself) have often assumed but which > I now find mostly anti-ideas: efficient code generation; run-time type > checking; exact type checking by the Python interpreter when it compiles > and loads modules. > I guess that's what I"ve pickedup on -- "anti-ideas"? but these are key for some of us... > I think it would be nice too. But what I hear from Stefan is that Cython's > requirements are not a good fit for the proposal. And I believe it makes > more sense to let Cython continue to do its own thing rather than trying to > either expand the proposal so Cython can use it without change, or allow > for alternate type notations so Cython can do its own thing. The latter > however sounds closer to what's possible. > maybe that will have to be the way to go -- but if still seems like we are building a too-limited system right up front -- that may be necessary to keep it from being too ugly. You've certainly been thinking about this a long time. > Everything in the proposal has a namespace (either 'typing' or 'builtins') > and the checker is expected to understand and follow imports properly. So I > don't see a problem here. > OK -- that could be very helpful. > Anyway, I guess it's unclear to me whether the intent is to create a >> system that can be used for a static/run-time type checker like Cython. >> > > That is not the intent. (Though static and run-time seem two very > different forms of type-checking. Apparently Cython does both?) > well, if you typedef a name: cdef int i then Cython generates static C code that uses i as an int (C int in this case). and if you typedef a function's parameters: def func(int i): then Cython generates code that tries to convert the argument to that type at run-time, raising an exception if it can't -- that's what i mean by run time type checking (I guess it's run-time type conversion, but it has to check in order to convert) There have been a fair number of comments alone the lines of "this is only >> for lint-like static analysis". But I'd really love to see this move toward >> full functionailty. >> > > Once this proposal is in place you may try to adapt it for that purpose. > But I'm not holding my breath, and I think we're better off with an > imperfect proposal that can make it into 3.5 than with waiting until the > ideal all-encompassing proposal is written. > makes sense, but hopefully it will be an imperfect proposal that can evolve inot a more-perfect one in the future. > But in the case of Python, the performance thing is entirely unproven, and > the focus on it is a hindrance to trying to get any type system at all > introduced. > Cython makes all the difference in the world for certain types of code. How is that unproven? > Nobody writes enough tests though, and type checking catches a different > category of issues (shallow or not) than tests. > well, hard to argue that catching more bugs isn't a good thing. On the other hand, dynamic typing does impact performance in a way that >> effects my code -- so I tun to Cython (and sometimes C, C++, Fortran) to >> address those issues. (maybe JIT compilation will one day meet this need, >> but it hasn't yet). >> >> So if there is type specifying, I want i to solve these problems. >> > > How? > I meant that the new type specifying system should be usable by a static or JIT compiler to generate more optimized code -- i.e. Cython (or numba, or....) But where I'm lost is the utility of static analysis that is not used to >> either provide run-time type checking OR compile optimized code (either JIT >> or ahead-of-time). It seems that your entire system would have to be fully >> type-hinted to get the type safety. But once you've done that, why not just >> use a static language? >> > > However, gradual typing, for dynamically-typed languages, takes a > different position. It claims that if you can type-check *some* of your > program, the added type safety for that part helps find bugs. It's not a > black or white situation. Just like a linter will find some bugs (after you > wade through the false positives, tuning the linter options locally and > globally to get a decent yield), a type checker of the kind proposed here > will find some bugs (after you spend some time adding type annotations to > part of the program). > Again, hard to argue that catching more bugs isn't a good thing. But thanks, this makes it more clear to me what' meant by 'gradual typing" I'm not sure that will satisfy the "enterprise" folks that really want static typing checking, but they can keep using Java, I guess. I do wonder how much this will "infect" Python, and folks will add all this completely unnecessary type hinting that will just make it harder to use libraries off the shelf. On the other hand, if run-time type checking is provided, then you could > use that at the boundaries of your code base (kind of like Cython does) and > get both the safety and performance benifits. > > But it sounds like it is Cython's job to do the checking at these > boundaries. Isn't this similar to the way Python extensions written in C or > C++ have to do type checking at the boundary between Python code and the > extension? > exactly -- all cython really does is write a C extension for you -- auto-generating all that boilerplate. > The specification of these checks should be defined by Cython. > Sure, but they are almost always a direct result of type, so all I do when I write cython is specify type. And as you point out about gradual typing, I don't need to specify type at all, Cython is perfectly happy working with python objects and leaving the type checking to run time. But just as you can now run any (most, anyway...) python code through Cython, or would be great to be able to run any type annotated code through Cython, and hopefully get some benefit from the type annotations, and not have to essentially add redundant information. > I've read the MyPY FAQ, which does address these issues to some extent, >> but doesn't really make a compelling case to me, anyway. >> > > It did to me. :-) > I'm getting close with your additional notes... I guess the gist here is that I think if Python is going to have an >> official type hinting system, it should leave the door open to compile time >> and run-time type checking. A good test case might be: can it be used for >> Cython? But this thread seems to be indicating that that is not the goal, >> and indeed, that it's almost an anti-goal. I hope I'm wrong about that. >> > > TBH I think the question of whether it can be used for Cython is still > very much open. Why couldn't Cython take code that says a certain parameter > has type List[int] and generate code from it? > it probably could -- I think it's a good idea for us to start playing with that sort of thing. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150118/ff5ca54b/attachment.html> From chris.barker at noaa.gov Mon Jan 19 08:04:51 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Sun, 18 Jan 2015 23:04:51 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <m9i6u4$lf$1@ger.gmane.org> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9i6u4$lf$1@ger.gmane.org> Message-ID: <CALGmxEKTVZS2QXvkd=oki==c_pccT3ch8WtRymx684Zbh6815A@mail.gmail.com> On Sun, Jan 18, 2015 at 10:06 PM, Terry Reedy <tjreedy at udel.edu> wrote: > Storing a triangular matrix in a list, with __getitem__ calculating the > linear index from an input row, col pair, is another possible example. > well, I prefer do do this sort of thing with a "has a" relationship, anyway, which would work better. Particularly if you are changing the API of get_item.... So we may be OK. > > Very few people subclass builtins.list ... > > So perhaps Cython could just assume that typing.List means > > builtins.list and (dynamically) reject calls that pass a subclass > > of builtins.list? > > Identifying a triangular matrix as a List might not be too useful in any > case. > exactly. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150118/ba472882/attachment-0001.html> From greg.ewing at canterbury.ac.nz Mon Jan 19 01:01:18 2015 From: greg.ewing at canterbury.ac.nz (Greg) Date: Mon, 19 Jan 2015 13:01:18 +1300 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> Message-ID: <54BC494E.3000609@canterbury.ac.nz> On 19/01/2015 8:34 a.m., Guido van Rossum wrote: > I'm putting a line in the sand: annotations are > for types Do you mean *static* types, or types in general? If they're only for static type checking, this seems a waste of a facility that evaluates things at run time. Moreover, evaluating them at run time is actually counterproductive, since it makes dealing with things like forward references unnecessarily awkward. It also introduces useless runtime overhead. And they only address part of the problem, since they only apply to functions and not other things we might want to specify the type of. If you've decided that static type checking is important enough to officially support, how about designing a new language feature specifically for that? We could, for example, borrow Haskell's :: symbol: class Duck: num_feathers :: int = 4200 def quack(volume :: float) ... The type expressions following :: would have the form of Python expressions, but they would not be evaluated. Introspecting on them at runtime could be supported, but you would get an AST instead of a value. -- Greg From stephen at xemacs.org Mon Jan 19 08:51:11 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 19 Jan 2015 16:51:11 +0900 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> Message-ID: <87lhkzw6mo.fsf@uwakimon.sk.tsukuba.ac.jp> Chris Barker writes: > But I still think we can provide something that is useful for most > use-cases, and would like to propose what that is, and what the > decision points are: It would be useful, that is, if the user knows what she is doing. The problem exposed by the plethora of use cases is that in many cases users don't know what they're doing, and it's *not* just a matter of forgetting to take the absolute value. As you say about absolute error: > It will not do a simple absolute comparison -- that is the job of a > different function, or, better yet, folks just write it themselves: > > abs(x - y) <= delta > > really isn't much harder to write than a function call: > > absolute_diff(x,y,delta) for relative error abs(x - y)/min(abs(x), abs(y)) really isn't that hard to write -- if you know that's what you want. But Neil and Ron *don't* want that: they know which value is correct and that it's never zero. > It will be designed for floating point numbers, and handle inf, > -inf, and NaN "properly". Which means what? Something different from what a conforming IEEE 754 implementation would do? The farther we get into this, the more handwaving is being done by advocates. Despite what I wrote above, it's not trivial to write a correct symmetric relative error: some such function would be useful. But I think that it should have a name that (at least to some extent) indicates what it actually does (unlike is_close(), which users are likely to assume means what they think it means because they haven't thought about the alternatives). Steve From toddrjen at gmail.com Mon Jan 19 09:59:58 2015 From: toddrjen at gmail.com (Todd) Date: Mon, 19 Jan 2015 09:59:58 +0100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> Message-ID: <CAFpSVpKcSpw5Z=mOX=XRv0zR3xP8CL6sUeFPKjXzvSSbc-Th5g@mail.gmail.com> On Jan 16, 2015 6:18 PM, "Guido van Rossum" <guido at python.org> wrote: > > <snip> I apologize if I missed this, but is there any support for duck-type hinting, that is somehow defining a type hint by what attributes or methods it has rather than by what it is a subclass of? Or is this something that is being intentionally avoided? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/da58b906/attachment.html> From abarnert at yahoo.com Mon Jan 19 10:19:04 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 19 Jan 2015 01:19:04 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAFpSVpKcSpw5Z=mOX=XRv0zR3xP8CL6sUeFPKjXzvSSbc-Th5g@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <CAFpSVpKcSpw5Z=mOX=XRv0zR3xP8CL6sUeFPKjXzvSSbc-Th5g@mail.gmail.com> Message-ID: <3E876543-26BE-4A67-B55E-5BFCB512B992@yahoo.com> On Jan 19, 2015, at 0:59, Todd <toddrjen at gmail.com> wrote: > On Jan 16, 2015 6:18 PM, "Guido van Rossum" <guido at python.org> wrote: > > > > <snip> > > I apologize if I missed this, but is there any support for duck-type hinting, that is somehow defining a type hint by what attributes or methods it has rather than by what it is a subclass of? Or is this something that is being intentionally avoided? > I think you're asking about what's called the "protocols" sub-feature, which has been put off until post-3.5, and probably a separate PEP. See issue #11 on the tracker. As I understand it, the issue is unifying static typing protocols with ABCs that do the same thing with subclass hooks (and possibly other things like Zope's more complex feature). -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/ca3a5ebc/attachment.html> From cfkaran2 at gmail.com Mon Jan 19 14:27:37 2015 From: cfkaran2 at gmail.com (Cem Karan) Date: Mon, 19 Jan 2015 08:27:37 -0500 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <20150119015839.GT18556@ando.pearwood.info> References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> Message-ID: <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> On Jan 18, 2015, at 8:58 PM, Steven D'Aprano <steve at pearwood.info> wrote: > On Sun, Jan 18, 2015 at 05:24:51PM -0500, Cem Karan wrote: > >> def foo(a: {Type(int), Doc("some doc")}, b): >> pass > > I dislike this, because there is too much in the function signature. I > could live with this: > > def foo(a:int, b): pass > > or *possibly* this: > > def foo(a:"some doc", b): pass > > but not both together. Combining the two doesn't look too bad for a toy > example like made-up "foo", but think about a real-world example: > > > # from the statistics module > def median_grouped( > data: {Type(Iterable[Real]), Doc("grouped continuous data")}, > interval: {Type(Real), Doc("class interval")} = 1 > ) -> {Type(Real), Doc( > "50th percentile (median) of grouped continuous data")}: > pass > > > That may be machine-readable, but it is too complicated for me to read, > and I wrote it! And that's only a simple function with a mere two > parameters and simple one-liner docs. > > Having written that, I still need to write a doc string, and almost > certainly I have to duplicate the annotation docs inside the docstring. > So it isn't really saving me much, if anything, over this: > > > @Doc({"data": "grouped continuous data", > "interval": "class interval", > "return": "50th percentile (median) of grouped continuous data" > }) > def median_grouped(data:Iterable[Real], interval:Type(Real)=1)->Real: > pass I see your point, that was not fun to read. > It's not *ideal* to have to duplicate the parameter names in the > decorator, but except in the most trivial toy cases, I can't have it > all: > > - type annotations > - and document annotations > - without being too verbose or complex > - and still easy to read and write and maintain > > Something has to give, and since type-hinting is deemed to be the > primary use for annotations, it has to be the document annotations. You > can still have both type-hints and docs, you just can't use annotations > for them both. (But see below.) > > - If you want to use annotations for documentation, then you > cannot use them for type-hints at the same time; > - if you want to use annotations for type-hints, then you > need another method of dealing with the documentation. > > Since you almost certainly will need to write a docstring anyway, the > best solution in my mind is to move the documentation into the > docstring: > > def median_grouped(data:Iterable[Real], interval:Type(Real)=1)->Real: > """Return the 50th percentile (median) of grouped continuous data. > > :param a: grouped continuous data > :param interval: class interval > > more documentation and docstrings follows... > """ OK, so should there be a convention for writing docstrings? >> With annotations, the documentation lives with the argument. If >> typing rules all, then you might accidentally have the following: >> >> def foo(a: int, b): >> """ >> :param a: some doc >> :type a: str <- Whoops! Type conflict! >> """ >> pass > > And that is best solved by having your linter understand the convention > for docstrings and flag the error. That implies a convention for docstrings is a good idea... >> I'm arguing that with annotations we can put a lot more information in >> about a given function argument/return value than just typing. > > Of course you can. But I think you shouldn't. > > You are absolutely free to disagree. Remember, use of the type-checker > is optional, and the type-checker itself will be a third-party tool. > Tools actually. I expect there will be much competition to develop the > best, most powerful type-checker. > > There is nothing stopping people (including yourself) writing a > type-checker which supports your multiple-annotations-in-a-set idea. If > it proves itself as a viable, and popular, alternative, then you can > come back and try to convince Guido to give it his blessing as > officially sanctioned (at which point all the other type-checkers will > have to support sets of annotations as well). > > I *do* believe that it is technically possible to have the type-checker > support your {Type(...), Spam(...)} idea, but I *don't* believe that it > will be practical or popular. I think it is a case of YAGNI combined > with EIYDNIYSDI (Even If You Do Need It You Shouldn't Do It). But I > might be wrong, and I encourage you to prove me wrong the only way that > really matters: with working code that gets used in the real world. You're right. I'll have to work on it. Thanks, Cem Karan From lkb.teichmann at gmail.com Mon Jan 19 15:26:30 2015 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Mon, 19 Jan 2015 15:26:30 +0100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round Message-ID: <CAK9R32T8f8UahgfFSVmb9EPcx3rQzU+VPKbYE0u3oy-d1UXuAA@mail.gmail.com> Hi everybody, I'm writing something like remote procedure calls, so type hinting would be very interesting to me - I could send out the function parameter types to the remote side, which could already check the type of the parameters before even sending out a request. I would however be interested in even more detailed information. Like the range in which a number should lie - so giving a maximum and a minimum value for an int. Would it be possible with a PEP 484 system to add information like that? For example by inheriting from int? Greetings Martin Teichmann From guido at python.org Mon Jan 19 17:15:54 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 19 Jan 2015 08:15:54 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <m9i6u4$lf$1@ger.gmane.org> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9i6u4$lf$1@ger.gmane.org> Message-ID: <CAP7+vJK3pWPxC7kra3d_+FGVYrna-CTr9LkbghzAXFwJDZRaSw@mail.gmail.com> On Sun, Jan 18, 2015 at 10:06 PM, Terry Reedy <tjreedy at udel.edu> wrote: > On 1/18/2015 10:40 PM, Guido van Rossum wrote: > >> On Sun, Jan 18, 2015 at 12:36 PM, Stefan Behnel >> > > does anyway. If it means "exactly list and no subtypes of list", >> Cython can >> use it to avoid generating generic fallback code. >> >> >> Can you explain how this works? How does Cython manage to use the >> knowledge that the argument's __class__ is equal to builtins.list to >> generate more efficient code? >> > > The obvious part is that Cython will directly access the C array of > pointers .. and *assume* that each pointer points to a member of the > sequence, in order. The less obvious part is how a subclass can break that > assumption. > > Suppose, for instance, one wants a list of pairs, without the space > overhead of a concrete tuple object for each pair. Here is the beginning > of a list subclass solution (not necessarily the best approach, but a > possible one). > > class PairList(list): > def __getitem__(self, i): > g = super().__getitem__ > return g(2*i), g(2*i+1) > > pl = PairList([1,2,3,4]) > print(pl[0], pl[1]) > >>> > (1, 2) (3, 4) > > For Cython to execute this correctly, it would need the subclass fallback > code that would go through the custom .__getitem__. > > Storing a triangular matrix in a list, with __getitem__ calculating the > linear index from an input row, col pair, is another possible example. > But these are toy examples. For "real" code, you have to override a lot more methods, and the usual implementation approach is not to subclass list but to subclass collections.abc.MutableSequence. Then the problem goes away, in some sense, because MutableSequence isn't compatible with typing.List anyway. > > Very few people subclass builtins.list ... > > So perhaps Cython could just assume that typing.List means > > builtins.list and (dynamically) reject calls that pass a subclass > > of builtins.list? > > Identifying a triangular matrix as a List might not be too useful in any > case. > ... > > > would be nice to at least be able to explicitly express somehow > > that subtypes should be disallowed. > > > Can you come up with a specific proposal? Maybe we're not as far > > apart as we think. :-) > > Is the use of concrete types like list disallowed? > You mean as an argument type hint? It is allowed, but we're not mutating the builtins.list to allow subscripting as a class, so you could only use e.g. "def foo(a: list) -> list:", you couldn't use list[int]. And it would still mean "allow subclasses". > --- > My memory of the pre-3.0 discussion is that we did not add something like > typing then because we did not know how to write it. You suggested > (something like) "let people experiment and we can adopt and adapt the best > result." Now is the anticipated future. I agree that mypy seems to be the > best starting point of something both usable and useful that we are likely > to get. It would be nice to have adapted version in an early alpha. I > presume you plan to label typing 'experimental', as I believe asyncio was. Yes. (Actually, provisional is the term. But yes.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/1b91dac6/attachment.html> From guido at python.org Mon Jan 19 17:23:47 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 19 Jan 2015 08:23:47 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <54BC494E.3000609@canterbury.ac.nz> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> <54BC494E.3000609@canterbury.ac.nz> Message-ID: <CAP7+vJKMDXDorA42UL6dkChN_aMcBCDWEGLZnJcA_GgCg2Dmqw@mail.gmail.com> On Sun, Jan 18, 2015 at 4:01 PM, Greg <greg.ewing at canterbury.ac.nz> wrote: > On 19/01/2015 8:34 a.m., Guido van Rossum wrote: > >> I'm putting a line in the sand: annotations are >> for types >> > > Do you mean *static* types, or types in general? > > If they're only for static type checking, this seems a waste of a > facility that evaluates things at run time. Moreover, evaluating > them at run time is actually counterproductive, since it makes > dealing with things like forward references unnecessarily awkward. > It also introduces useless runtime overhead. And they only address > part of the problem, since they only apply to functions and not > other things we might want to specify the type of. > > If you've decided that static type checking is important > enough to officially support, how about designing a new language > feature specifically for that? > > We could, for example, borrow Haskell's :: symbol: > > class Duck: > > num_feathers :: int = 4200 > > def quack(volume :: float) > ... > > The type expressions following :: would have the form of > Python expressions, but they would not be evaluated. > Introspecting on them at runtime could be supported, but > you would get an AST instead of a value. > There is no time to get such a proposal implemented in time for Python 3.5. Since you are proposing entirely new syntax we could still do this later if we ended up finding that the mypy approach after all is not so useful. And abandoning the mypy-based proposal is easy -- the typing module will be provisional in 3.5, so in 3.6 we could just delete it from the stdlib if we decided we took a wrong turn. People who really liked it could still install it from PyPI. But I don't expect that to happen. I don't see the mypy approach as a waste of facilities evaluated at run time. I have found it a very liberating experience to realize that the type checker could just be a linter and there is an elegant way to avoid noise from the interaction between code that has type annotations and code that doesn't. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/efdd2fae/attachment-0001.html> From guido at python.org Mon Jan 19 17:28:29 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 19 Jan 2015 08:28:29 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAK9R32T8f8UahgfFSVmb9EPcx3rQzU+VPKbYE0u3oy-d1UXuAA@mail.gmail.com> References: <CAK9R32T8f8UahgfFSVmb9EPcx3rQzU+VPKbYE0u3oy-d1UXuAA@mail.gmail.com> Message-ID: <CAP7+vJLPrOoNpO5XVnMmN0Snm-y7WqKu+2dhNi4UXQPrTiUr+Q@mail.gmail.com> In the current proposal there is no way to teach the type checker such dynamic constraints on the value. You could subclass int and you could implement the constraints dynamically in that class, and you could use that class in an argument spec. (And you could introspect methods at run time and extract the annotation types.) But the static type checker would only check that you're passing instances of that class around, it wouldn't understand the range constraint. On Mon, Jan 19, 2015 at 6:26 AM, Martin Teichmann <lkb.teichmann at gmail.com> wrote: > Hi everybody, > > I'm writing something like remote procedure calls, so type hinting would > be very interesting to me - I could send out the function parameter types > to the remote side, which could already check the type of the parameters > before even sending out a request. > > I would however be interested in even more detailed information. Like > the range in which a number should lie - so giving a maximum and a minimum > value for an int. Would it be possible with a PEP 484 system to add > information > like that? For example by inheriting from int? > > Greetings > > Martin Teichmann > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/45446f70/attachment.html> From ben+python at benfinney.id.au Mon Jan 19 19:51:15 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 20 Jan 2015 05:51:15 +1100 Subject: [Python-ideas] Conventions for Python code documentation (was: PEP 484 (Type Hints) -- first draft round) References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> Message-ID: <85oapur4d8.fsf_-_@benfinney.id.au> Cem Karan <cfkaran2 at gmail.com> writes: > OK, so should there be a convention for writing docstrings? The current convention is to use Sphinx to render a reStructuredText docstring, and write the API of a class or function with reStructuredText field lists. <URL:http://sphinx-doc.org/domains.html#the-python-domain> <URL:http://sphinx-doc.org/domains.html#info-field-lists> Python's own documentation, as well as a large amount of third-party packages, are written this way. -- \ ?It is seldom that liberty of any kind is lost all at once.? | `\ ?David Hume | _o__) | Ben Finney From graffatcolmingov at gmail.com Mon Jan 19 20:25:10 2015 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Mon, 19 Jan 2015 13:25:10 -0600 Subject: [Python-ideas] Conventions for Python code documentation (was: PEP 484 (Type Hints) -- first draft round) In-Reply-To: <85oapur4d8.fsf_-_@benfinney.id.au> References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> <85oapur4d8.fsf_-_@benfinney.id.au> Message-ID: <CAN-Kwu3Nj0=1+6B=uNmJz8D8LORfAAu62GLJ8_R5ZwwKjxUg=Q@mail.gmail.com> There's also PEP 0257 for docstring formatting. On Mon, Jan 19, 2015 at 12:51 PM, Ben Finney <ben+python at benfinney.id.au> wrote: > Cem Karan <cfkaran2 at gmail.com> writes: > >> OK, so should there be a convention for writing docstrings? > > The current convention is to use Sphinx to render a reStructuredText > docstring, and write the API of a class or function with > reStructuredText field lists. > > <URL:http://sphinx-doc.org/domains.html#the-python-domain> > <URL:http://sphinx-doc.org/domains.html#info-field-lists> > > Python's own documentation, as well as a large amount of third-party > packages, are written this way. > > -- > \ ?It is seldom that liberty of any kind is lost all at once.? | > `\ ?David Hume | > _o__) | > Ben Finney > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From ron3200 at gmail.com Mon Jan 19 20:51:00 2015 From: ron3200 at gmail.com (Ron Adam) Date: Mon, 19 Jan 2015 13:51:00 -0600 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> Message-ID: <m9jn75$a3s$1@ger.gmane.org> On 01/19/2015 12:32 AM, Chris Barker wrote: > > Here is a gist with a sample implementation: > > https://gist.github.com/PythonCHB/6e9ef7732a9074d9337a For the most part I think it looks good. Boost describes both a week and strong version, but I didn't see why they choose the strong version. I'm guessing that "strong" indicates it has a higher certainty that the numbers are within the specified tolerance. By using "and", it excludes the case when one is in the range and the other is not. I think this needs to be in the description. Your function can test for absolute distance, relative distance, or the greater of both. You could have the defaults for both be zero, and raise an error if at least one isn't set to something other than zero. > I need to add more tests, and make the test proper unit tests, but it's a > start. > > I also need to see how it does with other data types than float -- > hopefully, it will "just work" with the core set. > > I hope we can come to some consensus that something like this is the way to go. Good examples will help with this. It may also help with choosing a good name. To me, the strong version is an "is-good" test, and the weak version is an "is-close" test. I think it could be important to some people. I like the idea of being able to use these as a teaching tool to demonstrate how our ideas of closeness, equality, and inequality can be subjective. There are two cases... 1: (The weak version is require for this to work.) Two number are definitely equivalent if they are the same. (pretty obvious) Two numbers are definitely not equivalent if they are further apart than the largest error amount. (The larger number better indicates the largeness of the the possible relative error.) And two numbers are close if you can't determine if they are equivalent, or not-equivalent with certainty.* (* "close numbers" may include equivalent numbers if you define it as a set of all definitely not-equivalent numbers.) 2: (The strong version is required for this to work.) A value is good if it's within a valid range with certainty. It is less than the smaller relative range of either number. The smaller number better indicates the magnitude of smallness. So case 1 should be used to test for errors, and case 2 should be used to test for valid ranges. It seems you have the 2nd case in mind, and that's fine. Some of us where thinking of the first case, and possibly switching from one to the other during the discussion which is probably why it got confusing or repetitious at some points. I think both of these are useful, but you definitely need to be clear which one you are implementing, and to document it clearly. Cheers, Ron From guido at python.org Mon Jan 19 20:54:20 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 19 Jan 2015 11:54:20 -0800 Subject: [Python-ideas] Conventions for Python code documentation (was: PEP 484 (Type Hints) -- first draft round) In-Reply-To: <CAN-Kwu3Nj0=1+6B=uNmJz8D8LORfAAu62GLJ8_R5ZwwKjxUg=Q@mail.gmail.com> References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> <85oapur4d8.fsf_-_@benfinney.id.au> <CAN-Kwu3Nj0=1+6B=uNmJz8D8LORfAAu62GLJ8_R5ZwwKjxUg=Q@mail.gmail.com> Message-ID: <CAP7+vJ+=3uebwL=+QOajwYVd=h+RaEFsRy8GqKQnDBFrjt7uog@mail.gmail.com> Unfortunately PEP 257 falls short on specifying how to describe arguments -- it has only one example, it's not normative, and there's not much code that follows the example. The reST conventions are more common, but the stdlib itself rarely uses them. It would be nice to come up with a better convention that takes PEP 484 into account (so doc generators can incorporate the argument type annotations into the generated output, merged with per-argument from the docstring). On Mon, Jan 19, 2015 at 11:25 AM, Ian Cordasco <graffatcolmingov at gmail.com> wrote: > There's also PEP 0257 for docstring formatting. > > On Mon, Jan 19, 2015 at 12:51 PM, Ben Finney <ben+python at benfinney.id.au> > wrote: > > Cem Karan <cfkaran2 at gmail.com> writes: > > > >> OK, so should there be a convention for writing docstrings? > > > > The current convention is to use Sphinx to render a reStructuredText > > docstring, and write the API of a class or function with > > reStructuredText field lists. > > > > <URL:http://sphinx-doc.org/domains.html#the-python-domain> > > <URL:http://sphinx-doc.org/domains.html#info-field-lists> > > > > Python's own documentation, as well as a large amount of third-party > > packages, are written this way. > > > > -- > > \ ?It is seldom that liberty of any kind is lost all at once.? | > > `\ ?David Hume | > > _o__) | > > Ben Finney > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/9a0a2c0b/attachment-0001.html> From chris.barker at noaa.gov Mon Jan 19 20:54:39 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 19 Jan 2015 11:54:39 -0800 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <87lhkzw6mo.fsf@uwakimon.sk.tsukuba.ac.jp> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <87lhkzw6mo.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <CALGmxEJAvRQHxzddrnH9noeRhQswEfJsN2o0qOz7-xv0bb8ATQ@mail.gmail.com> On Sun, Jan 18, 2015 at 11:51 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote: > > But I still think we can provide something that is useful for most > > use-cases, and would like to propose what that is, and what the > > decision points are: > > It would be useful, that is, if the user knows what she is doing. The > problem exposed by the plethora of use cases is that in many cases > users don't know what they're doing, and it's *not* just a matter of > forgetting to take the absolute value. > I was afraid of this -- my take is that EVERY algorithm has corner cases, an special cases, and places where it is appropriate and where it is not. And this list (and all lists...) is full of folks that like to hash this stuff out. So the fact that there has been a long and drawn out discussion does not mean that there is not a generally useful function to be had. And I think, in fact, that such a function does is exist that will work most of teh time, even for users that have not thought it out carefully. We, in fact have a lot of evidence that this is needed: * numpy has close() and all_close -- they are widely used, mostly successfully * Steven took that time to write one (and tests for it) for the statistics module tests. * The unittest module has something -- and a poor one (IMHO) at that -- but is still useful. * Boost has one (not Python, but still....) > It will not do a simple absolute comparison -- that is the job of a > > different function, or, better yet, folks just write it themselves: > > > > abs(x - y) <= delta > > > > really isn't much harder to write than a function call: > > > > absolute_diff(x,y,delta) > > for relative error > > abs(x - y)/min(abs(x), abs(y)) > > really isn't that hard to write -- if you know that's what you want. > that is, I think, just enough harder to think about and write than the absolute comparison method, that it's worth it. And if others think an absolute_close is worth adding, too, I have no problem with that. But Neil and Ron *don't* want that: they know which value is correct > and that it's never zero. > I think my proposal satisfies both Neil and Ron's use case -- or a version of it can, and also Steven's. > > It will be designed for floating point numbers, and handle inf, > > -inf, and NaN "properly". > > Which means what? Something different from what a conforming IEEE 754 > implementation would do? > no, sorry for being lazy in writing, -- I meant it would conform to IEEE 754. i.e. NaN would not be close to anything, including another NaN, and inf and -inf would only be close to themselves. (though I don't know that ieee ever mentions non-exact comparison...) The farther we get into this, the more handwaving is being done by > advocates. Despite what I wrote above, it's not trivial to write a > correct symmetric relative error: some such function would be useful. > if it's not trivial, then all the reason to to put it in the standard lib. > But I think that it should have a name that (at least to some extent) > indicates what it actually does (unlike is_close(), which users are > likely to assume means what they think it means because they haven't > thought about the alternatives). > Indeed -- I think that the name of the function, and it's parameters need to be carefully chosen to best convey what they mean. (and the docstring). What I"d like to do is determine whether there is any hope of coming to a consensus about what a reasonable implementation would look like. If not, then so be it. If so, then I guess I"ll need to write a PEP and then hash out the naming and details. So I think the next step is to hear form the active participants in this thread as to whether they think there's hope, and/or from Guido and other core developers as to whether they're ready to reject it at this point -- I don't want to waste any more of my or anyone else's time. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/47394981/attachment.html> From ethan at stoneleaf.us Mon Jan 19 21:22:17 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 19 Jan 2015 12:22:17 -0800 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <CALGmxEJAvRQHxzddrnH9noeRhQswEfJsN2o0qOz7-xv0bb8ATQ@mail.gmail.com> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <87lhkzw6mo.fsf@uwakimon.sk.tsukuba.ac.jp> <CALGmxEJAvRQHxzddrnH9noeRhQswEfJsN2o0qOz7-xv0bb8ATQ@mail.gmail.com> Message-ID: <54BD6779.4060207@stoneleaf.us> On 01/19/2015 11:54 AM, Chris Barker wrote: > > What I"d like to do is determine whether there is any hope of coming to a consensus about what a reasonable > implementation would look like. If not, then so be it. If so, then I guess I"ll need to write a PEP and then hash out > the naming and details. I think it's worth having, but it will definitely need a PEP. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/10dc831d/attachment.sig> From greg.ewing at canterbury.ac.nz Mon Jan 19 21:28:56 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 20 Jan 2015 09:28:56 +1300 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <m9i6u4$lf$1@ger.gmane.org> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9i6u4$lf$1@ger.gmane.org> Message-ID: <54BD6908.7090701@canterbury.ac.nz> > On 1/18/2015 10:40 PM, Guido van Rossum wrote: > > > Very few people subclass builtins.list ... > > So perhaps Cython could just assume that typing.List means > > builtins.list and (dynamically) reject calls that pass a subclass > > of builtins.list? That's actually a reasonable idea. Anything that accepts something as weird as a trianglular matrix implemented as a list subclass will probably accept any kind of sequence, so you might as well type it as Sequence (or whatever the abstract sequence interface is to be called). -- Greg From jeanpierreda at gmail.com Mon Jan 19 21:30:46 2015 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Mon, 19 Jan 2015 12:30:46 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <54BC494E.3000609@canterbury.ac.nz> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> <54BC494E.3000609@canterbury.ac.nz> Message-ID: <CABicbJLdq519s-HDg5GM6Y1aeTbyhYrL-+fNqsxWbjn2B4C5nw@mail.gmail.com> On Sun, Jan 18, 2015 at 4:01 PM, Greg <greg.ewing at canterbury.ac.nz> wrote: > On 19/01/2015 8:34 a.m., Guido van Rossum wrote: >> >> I'm putting a line in the sand: annotations are >> for types > > > Do you mean *static* types, or types in general? > > If they're only for static type checking, this seems a waste of a > facility that evaluates things at run time. Moreover, evaluating > them at run time is actually counterproductive, since it makes > dealing with things like forward references unnecessarily awkward. > It also introduces useless runtime overhead. And they only address > part of the problem, since they only apply to functions and not > other things we might want to specify the type of. Strong +1. The current proposal is a historical accident, it would not be designed this way if it was designed from scratch. That's not a nice thing to include in the core language. (Although it does happen often, since "practicality beats purity"). I like the :: idea. -- Devin From chris.barker at noaa.gov Mon Jan 19 21:32:21 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 19 Jan 2015 12:32:21 -0800 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <54BD6779.4060207@stoneleaf.us> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <87lhkzw6mo.fsf@uwakimon.sk.tsukuba.ac.jp> <CALGmxEJAvRQHxzddrnH9noeRhQswEfJsN2o0qOz7-xv0bb8ATQ@mail.gmail.com> <54BD6779.4060207@stoneleaf.us> Message-ID: <CALGmxEJGbGCUTNhqJ_yE6M=_YBEvuLu-aLtJckG5SgOjBYnmyA@mail.gmail.com> On Mon, Jan 19, 2015 at 12:22 PM, Ethan Furman <ethan at stoneleaf.us> wrote: > > I think it's worth having, but it will definitely need a PEP. > Fair enough -- thanks. What's the procedure? Do I work up a draft, publish it somewhere temporary, and start hashing it out on this list? or or do I put the draft up with the official PEPs first? -Chris Now to find the time to write it...... -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/08860070/attachment.html> From chris.barker at noaa.gov Mon Jan 19 21:28:37 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 19 Jan 2015 12:28:37 -0800 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <m9jn75$a3s$1@ger.gmane.org> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> Message-ID: <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> On Mon, Jan 19, 2015 at 11:51 AM, Ron Adam <ron3200 at gmail.com> wrote: > Here is a gist with a sample implementation: > > >> https://gist.github.com/PythonCHB/6e9ef7732a9074d9337a >> > > For the most part I think it looks good. > > Boost describes both a week and strong version, but I didn't see why they > choose the strong version. Actually, Boost didn't choose, there is a flag you can set to select which one to use. I don't want to do that, if you really want something special, write it yourself. I did choose the "strong" version -- it just seemed more conservative. Also it's what I think Steven chose for his version (though not with and, but the result is the same) > I'm guessing that "strong" indicates it has a higher certainty that the > numbers are within the specified tolerance. > exactly. > By using "and", it excludes the case when one is in the range and the > other is not. I think this needs to be in the description. > I tried to capture that, but apparently not well ;-) -- I do think a lot of care needs to be taken with the docs. But I still think the in common use case, it doesn't matter -- folks want o know that the values are "approximately equal", so some tolerance (1-e8 or ....), and, in fact, the tolerance itself is likely to be approximate as well (i.e 8 or 9 decimal digits is fine). WE need to be careful in the docs so that people can get the tolernace the want if they do really care, but I suspect that's unlikely to be the case often. Has anyone in this thread, or in any test code you've seen or written, provided a tolerance with more than one significant figure? i.e, I always use 1e-12 or 1e-10, I've never used a tolerance anything like 0.000153427213 As, in the common case, the tolerance is approximate, and usually small, then it doesn't matter which version we use: string, weak, or declared which value to scale to. But I prefer a symmetric version, as I suspect hat will be the least surprising -- it's good to get the same answer every time, even if it is approximate! Your function can test for absolute distance, relative distance, or the > greater of both. You could have the defaults for both be zero, and raise > an error if at least one isn't set to something other than zero. > good point -- I hadn't really thought about it that way, and despite my saying that I didn't want an absolute tolerance function, I have in fact provided one -- if you set the relative "tol" to zero and "min_tol" > zero, you get an absolute tolerance test. I prefer to have a useful default, rather than requireing every use to specify it (though there is something to be said for making people think abou tit!), but you are right, they should probalby be renames somethign like: rel_tol and abs_tol, and the dics can make it clear that if you specify both as greater tha zero, it will return True if either one is satisfied. I hope we can come to some consensus that something like this is the way to >> go. >> > > Good examples will help with this. It may also help with choosing a good > name. > you mean use-case examples? rather than specific value examples? To me, the strong version is an "is-good" test, and the weak version is an > "is-close" test. I think it could be important to some people. > > I like the idea of being able to use these as a teaching tool to > demonstrate how our ideas of closeness, equality, and inequality can be > subjective. > Are you suggesting that we allow a flag for the user to set to choose whether ot use weak or string version? I'd rather not -- I see this is a practical, works most of the time thing, not a teaching tool, or a "provides every use case" tool. > There are two cases... > > 1: (The weak version is require for this to work.) > > Two numbers are definitely not equivalent if they are further apart than > the largest error amount. (The larger number better indicates the > largeness of the the possible relative error.) > > And two numbers are close if you can't determine if they are equivalent, > or not-equivalent with certainty.* > > (* "close numbers" may include equivalent numbers if you define it as a > set of all definitely not-equivalent numbers.) > > 2: (The strong version is required for this to work.) > > A value is good if it's within a valid range with certainty. It is less > than the smaller relative range of either number. The smaller number > better indicates the magnitude of smallness. > > So case 1 should be used to test for errors, and case 2 should be used to > test for valid ranges. > > It seems you have the 2nd case in mind, and that's fine. Some of us where > thinking of the first case, and possibly switching from one to the other > during the discussion which is probably why it got confusing or repetitious > at some points. > yes, I suppose I do -- and again, in the common use case, where the tolerance is also approximate, it really doesn't matter. > I think both of these are useful, but you definitely need to be clear > which one you are implementing, and to document it clearly. yup. Nice to know at least two of us seem to be coming to consensus ;-) -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/8e6be5a8/attachment.html> From rosuav at gmail.com Mon Jan 19 21:45:23 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 20 Jan 2015 07:45:23 +1100 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <CALGmxEJGbGCUTNhqJ_yE6M=_YBEvuLu-aLtJckG5SgOjBYnmyA@mail.gmail.com> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <87lhkzw6mo.fsf@uwakimon.sk.tsukuba.ac.jp> <CALGmxEJAvRQHxzddrnH9noeRhQswEfJsN2o0qOz7-xv0bb8ATQ@mail.gmail.com> <54BD6779.4060207@stoneleaf.us> <CALGmxEJGbGCUTNhqJ_yE6M=_YBEvuLu-aLtJckG5SgOjBYnmyA@mail.gmail.com> Message-ID: <CAPTjJmoNuLgE8=wEP8GCEtzUsxZciVMkNh-UxxJQwJEx-0PCRQ@mail.gmail.com> On Tue, Jan 20, 2015 at 7:32 AM, Chris Barker <chris.barker at noaa.gov> wrote: > On Mon, Jan 19, 2015 at 12:22 PM, Ethan Furman <ethan at stoneleaf.us> wrote: >> >> >> I think it's worth having, but it will definitely need a PEP. > > > Fair enough -- thanks. > > What's the procedure? Do I work up a draft, publish it somewhere temporary, > and start hashing it out on this list? or or do I put the draft up with the > official PEPs first? > > -Chris > > Now to find the time to write it...... I've already mentioned this off-list along with some other details, but if anyone else is curious about the procedure for starting a PEP, it's in PEP 12: https://www.python.org/dev/peps/pep-0012/ ChrisA From g.brandl at gmx.net Mon Jan 19 21:48:53 2015 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 19 Jan 2015 21:48:53 +0100 Subject: [Python-ideas] Conventions for Python code documentation In-Reply-To: <CAP7+vJ+=3uebwL=+QOajwYVd=h+RaEFsRy8GqKQnDBFrjt7uog@mail.gmail.com> References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> <85oapur4d8.fsf_-_@benfinney.id.au> <CAN-Kwu3Nj0=1+6B=uNmJz8D8LORfAAu62GLJ8_R5ZwwKjxUg=Q@mail.gmail.com> <CAP7+vJ+=3uebwL=+QOajwYVd=h+RaEFsRy8GqKQnDBFrjt7uog@mail.gmail.com> Message-ID: <m9jqjl$uei$1@ger.gmane.org> On 01/19/2015 08:54 PM, Guido van Rossum wrote: > Unfortunately PEP 257 falls short on specifying how to describe arguments -- it > has only one example, it's not normative, and there's not much code that follows > the example. The reST conventions are more common, but the stdlib itself rarely > uses them. It would be nice to come up with a better convention that takes PEP > 484 into account (so doc generators can incorporate the argument type > annotations into the generated output, merged with per-argument from the docstring). I'll be happy to implement that. Basically if you use :param x: description :type x: type you could just leave out the "type" line and have it generated from the annotation. I guess it's very similar for Google and numpy style docstrings, which are supported by the (as of 1.3 built-in) Napoleon extension. Georg From abarnert at yahoo.com Mon Jan 19 22:05:51 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 19 Jan 2015 13:05:51 -0800 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> Message-ID: <8E24B81C-A241-4AF4-AFD3-FBD4E32C6B24@yahoo.com> On Jan 19, 2015, at 12:28, Chris Barker <chris.barker at noaa.gov> wrote: > Has anyone in this thread, or in any test code you've seen or written, provided a tolerance with more than one significant figure? i.e, I always use 1e-12 or 1e-10, I've never used a tolerance anything like 0.000153427213 If you're working with measurement error or mechanical tolerance or something similar, you often have something with a non-unit mantissa. For example, the first resistor in the examples for the standard color code chart is 4.7KOhm +/- 10%, so you've got an error of 4.7e2. Also, keep in mind that if you do an error propagation analysis, you're unlikely to get round numbers. Even for a dead-simple case, where you start off with two non-covariant values that are +/- 1.00e-5 and add them, the result is +/- 1.41e-5. The reasons you often only have one significant figure are (a) you're hand-waving the error analysis, or (b) your error ends up a couple orders of magnitude smaller than your values and you've only got a couple significant digits. (1.234e5 +/- 6.789e0 is no more meaningful than 1.234e5 +/- 1e1, after all.) From cfkaran2 at gmail.com Mon Jan 19 22:53:05 2015 From: cfkaran2 at gmail.com (Cem Karan) Date: Mon, 19 Jan 2015 16:53:05 -0500 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CABicbJLdq519s-HDg5GM6Y1aeTbyhYrL-+fNqsxWbjn2B4C5nw@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> <54BC494E.3000609@canterbury.ac.nz> <CABicbJLdq519s-HDg5GM6Y1aeTbyhYrL-+fNqsxWbjn2B4C5nw@mail.gmail.com> Message-ID: <5C091DF5-5ACB-479F-BB2D-302F2A60C3B0@gmail.com> On Jan 19, 2015, at 3:30 PM, Devin Jeanpierre <jeanpierreda at gmail.com> wrote: > On Sun, Jan 18, 2015 at 4:01 PM, Greg <greg.ewing at canterbury.ac.nz> wrote: >> On 19/01/2015 8:34 a.m., Guido van Rossum wrote: >>> >>> I'm putting a line in the sand: annotations are >>> for types >> >> >> Do you mean *static* types, or types in general? >> >> If they're only for static type checking, this seems a waste of a >> facility that evaluates things at run time. Moreover, evaluating >> them at run time is actually counterproductive, since it makes >> dealing with things like forward references unnecessarily awkward. >> It also introduces useless runtime overhead. And they only address >> part of the problem, since they only apply to functions and not >> other things we might want to specify the type of. > > Strong +1. The current proposal is a historical accident, it would not > be designed this way if it was designed from scratch. That's not a > nice thing to include in the core language. (Although it does happen > often, since "practicality beats purity"). > > I like the :: idea. +1 on the :: idea. As for the typing system, I think the proposal is pretty much spot on. I know I've been harping on annotations being forcefully restricted to types only, but the proposal itself (the typing library, etc.) is a really good proposal. Unless Python suddenly stops being Python, it's not too likely that any typing system can handle all possible cases of what we can do in the language (at least, not without becoming insanely unwieldy). Thanks, Cem Karan From guido at python.org Mon Jan 19 23:08:01 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 19 Jan 2015 14:08:01 -0800 Subject: [Python-ideas] Conventions for Python code documentation In-Reply-To: <m9jqjl$uei$1@ger.gmane.org> References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> <85oapur4d8.fsf_-_@benfinney.id.au> <CAN-Kwu3Nj0=1+6B=uNmJz8D8LORfAAu62GLJ8_R5ZwwKjxUg=Q@mail.gmail.com> <CAP7+vJ+=3uebwL=+QOajwYVd=h+RaEFsRy8GqKQnDBFrjt7uog@mail.gmail.com> <m9jqjl$uei$1@ger.gmane.org> Message-ID: <CAP7+vJJec8b1Y2Vd5GWNqOkb-XqaHvaB5e=vxQ1j_sQQzg+dEw@mail.gmail.com> "Napoleon"? On Mon, Jan 19, 2015 at 12:48 PM, Georg Brandl <g.brandl at gmx.net> wrote: > On 01/19/2015 08:54 PM, Guido van Rossum wrote: > > Unfortunately PEP 257 falls short on specifying how to describe > arguments -- it > > has only one example, it's not normative, and there's not much code that > follows > > the example. The reST conventions are more common, but the stdlib itself > rarely > > uses them. It would be nice to come up with a better convention that > takes PEP > > 484 into account (so doc generators can incorporate the argument type > > annotations into the generated output, merged with per-argument from the > docstring). > > I'll be happy to implement that. Basically if you use > > :param x: description > :type x: type > > you could just leave out the "type" line and have it generated from the > annotation. > > I guess it's very similar for Google and numpy style docstrings, which are > supported by the (as of 1.3 built-in) Napoleon extension. > > Georg > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/3c9846f9/attachment.html> From guido at python.org Mon Jan 19 23:13:40 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 19 Jan 2015 14:13:40 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CABicbJLdq519s-HDg5GM6Y1aeTbyhYrL-+fNqsxWbjn2B4C5nw@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> <54BC494E.3000609@canterbury.ac.nz> <CABicbJLdq519s-HDg5GM6Y1aeTbyhYrL-+fNqsxWbjn2B4C5nw@mail.gmail.com> Message-ID: <CAP7+vJKK5FwhtUU=3GG-OFL8hZmo3jPpvAV25wYE+CJUwkTQEA@mail.gmail.com> On Mon, Jan 19, 2015 at 12:30 PM, Devin Jeanpierre <jeanpierreda at gmail.com> wrote: > On Sun, Jan 18, 2015 at 4:01 PM, Greg <greg.ewing at canterbury.ac.nz> wrote: > > Do you mean *static* types, or types in general? > > > > If they're only for static type checking, this seems a waste of a > > facility that evaluates things at run time. Moreover, evaluating > > them at run time is actually counterproductive, since it makes > > dealing with things like forward references unnecessarily awkward. > > It also introduces useless runtime overhead. And they only address > > part of the problem, since they only apply to functions and not > > other things we might want to specify the type of. > > Strong +1. The current proposal is a historical accident, it would not > be designed this way if it was designed from scratch. That's not a > nice thing to include in the core language. (Although it does happen > often, since "practicality beats purity"). > It's stronger than that. I'd say that all of Python is a historical accident. > I like the :: idea. > Blechh. It smells of C++ and syntax hacks. PEP 3107 is over 8 years old (also a historical accident :-). Let's stick with the devil we know. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/ffd4e633/attachment-0001.html> From flying-sheep at web.de Mon Jan 19 23:14:45 2015 From: flying-sheep at web.de (Philipp A.) Date: Mon, 19 Jan 2015 22:14:45 +0000 Subject: [Python-ideas] Conventions for Python code documentation (was: PEP 484 (Type Hints) -- first draft round) References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> <85oapur4d8.fsf_-_@benfinney.id.au> <CAN-Kwu3Nj0=1+6B=uNmJz8D8LORfAAu62GLJ8_R5ZwwKjxUg=Q@mail.gmail.com> <CAP7+vJ+=3uebwL=+QOajwYVd=h+RaEFsRy8GqKQnDBFrjt7uog@mail.gmail.com> Message-ID: <CAN8d9gnN=MGNujsDm_5fqauQekrAOEPRP1jVquo0z6JVmyOokQ@mail.gmail.com> I have am idea: how about including a @paramdoc decorator that accepts one named string argument per parameter and appends information gathered via type annotations and default values to the docstring? E.g. ~~~~ @paramdoc( main_course="Meaty because our society is medieval", side_dish="Optional, meat is enough", __return__="Objective opinion") def eat(main_course: Meat, side_dish=None: Food) -> Opinion: """ Decides how well the meat and side dish taste together """ ... ~~~~ After this, eat.__doc__ would be an object that retains the raw parts as properties, but otherwise is an elongated version of the original docstring. Its fourth and third to last line would read something like this: ~~~~ side_dish (Food; defaults to None): Optional, meat is enough ~~~~ What do you think? Guido van Rossum <guido <guido at python.org>@ <guido at python.org>python.org <guido at python.org>> schrieb am Mo., 19. Jan. 2015 20:56: -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/6f817d5a/attachment.html> From kaiser.yann at gmail.com Mon Jan 19 23:16:38 2015 From: kaiser.yann at gmail.com (Yann Kaiser) Date: Mon, 19 Jan 2015 22:16:38 +0000 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> <54BC494E.3000609@canterbury.ac.nz> <CABicbJLdq519s-HDg5GM6Y1aeTbyhYrL-+fNqsxWbjn2B4C5nw@mail.gmail.com> <5C091DF5-5ACB-479F-BB2D-302F2A60C3B0@gmail.com> Message-ID: <CANUJvPXa8MxvpSOxV_5BuV6n4=rhOGpjnahMrU_8RJ-OFOtq6Q@mail.gmail.com> On Mon Jan 19 2015 at 10:53:52 PM Cem Karan <cfkaran2 at gmail.com> wrote: > > On Jan 19, 2015, at 3:30 PM, Devin Jeanpierre <jeanpierreda at gmail.com> > wrote: > > > On Sun, Jan 18, 2015 at 4:01 PM, Greg <greg.ewing at canterbury.ac.nz> > wrote: > >> On 19/01/2015 8:34 a.m., Guido van Rossum wrote: > >>> > >>> I'm putting a line in the sand: annotations are > >>> for types > >> > >> > >> Do you mean *static* types, or types in general? > >> > >> If they're only for static type checking, this seems a waste of a > >> facility that evaluates things at run time. Moreover, evaluating > >> them at run time is actually counterproductive, since it makes > >> dealing with things like forward references unnecessarily awkward. > >> It also introduces useless runtime overhead. And they only address > >> part of the problem, since they only apply to functions and not > >> other things we might want to specify the type of. > > > > Strong +1. The current proposal is a historical accident, it would not > > be designed this way if it was designed from scratch. That's not a > > nice thing to include in the core language. (Although it does happen > > often, since "practicality beats purity"). > > > > I like the :: idea. > > +1 on the :: idea. > > As for the typing system, I think the proposal is pretty much spot on. I > know I've been harping on annotations being forcefully restricted to types > only, but the proposal itself (the typing library, etc.) is a really good > proposal. Unless Python suddenly stops being Python, it's not too likely > that any typing system can handle all possible cases of what we can do in > the language (at least, not without becoming insanely unwieldy). > The :: idea creates unambiguous space for static/type annotations which wouldn't even have to compile and execute and thus escape less-than-optimal solutions to the forward reference problems, all this without needlessly interfering with existing or potential uses of annotations for run-time purposes, a feature unique(?) to python. +1 on the :: idea. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/47020448/attachment.html> From guido at python.org Mon Jan 19 23:23:00 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 19 Jan 2015 14:23:00 -0800 Subject: [Python-ideas] Conventions for Python code documentation (was: PEP 484 (Type Hints) -- first draft round) In-Reply-To: <CAN8d9gnN=MGNujsDm_5fqauQekrAOEPRP1jVquo0z6JVmyOokQ@mail.gmail.com> References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> <85oapur4d8.fsf_-_@benfinney.id.au> <CAN-Kwu3Nj0=1+6B=uNmJz8D8LORfAAu62GLJ8_R5ZwwKjxUg=Q@mail.gmail.com> <CAP7+vJ+=3uebwL=+QOajwYVd=h+RaEFsRy8GqKQnDBFrjt7uog@mail.gmail.com> <CAN8d9gnN=MGNujsDm_5fqauQekrAOEPRP1jVquo0z6JVmyOokQ@mail.gmail.com> Message-ID: <CAP7+vJJ6NYgwQT9hhddkyVg7D3ZSRVUvgJ4ispQXcMLAvc6VJg@mail.gmail.com> Try to implement it and see how well it works. On Mon, Jan 19, 2015 at 2:14 PM, Philipp A. <flying-sheep at web.de> wrote: > I have am idea: how about including a @paramdoc decorator that accepts one > named string argument per parameter and appends information gathered via > type annotations and default values to the docstring? > > E.g. > > ~~~~ > @paramdoc( > main_course="Meaty because our society is medieval", > side_dish="Optional, meat is enough", > __return__="Objective opinion") > def eat(main_course: Meat, side_dish=None: Food) -> Opinion: > """ > Decides how well the meat and side dish taste together > """ > ... > ~~~~ > > After this, eat.__doc__ would be an object that retains the raw parts as > properties, but otherwise is an elongated version of the original > docstring. Its fourth and third to last line would read something like this: > > ~~~~ > side_dish (Food; defaults to None): > Optional, meat is enough > ~~~~ > > What do you think? > > Guido van Rossum <guido <guido at python.org>@ <guido at python.org>python.org > <guido at python.org>> schrieb am Mo., 19. Jan. 2015 20:56: > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/ac71125b/attachment.html> From jeanpierreda at gmail.com Mon Jan 19 23:26:12 2015 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Mon, 19 Jan 2015 14:26:12 -0800 Subject: [Python-ideas] Conventions for Python code documentation In-Reply-To: <CAP7+vJJec8b1Y2Vd5GWNqOkb-XqaHvaB5e=vxQ1j_sQQzg+dEw@mail.gmail.com> References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> <85oapur4d8.fsf_-_@benfinney.id.au> <CAN-Kwu3Nj0=1+6B=uNmJz8D8LORfAAu62GLJ8_R5ZwwKjxUg=Q@mail.gmail.com> <CAP7+vJ+=3uebwL=+QOajwYVd=h+RaEFsRy8GqKQnDBFrjt7uog@mail.gmail.com> <m9jqjl$uei$1@ger.gmane.org> <CAP7+vJJec8b1Y2Vd5GWNqOkb-XqaHvaB5e=vxQ1j_sQQzg+dEw@mail.gmail.com> Message-ID: <CABicbJKvWaL+E7WQi5Zc7mQ=fm9K2itUpdNM-SVyp_EizR3rdQ@mail.gmail.com> On Mon, Jan 19, 2015 at 2:08 PM, Guido van Rossum <guido at python.org> wrote: > "Napoleon"? https://pypi.python.org/pypi/sphinxcontrib-napoleon -- Devin > On Mon, Jan 19, 2015 at 12:48 PM, Georg Brandl <g.brandl at gmx.net> wrote: >> >> On 01/19/2015 08:54 PM, Guido van Rossum wrote: >> > Unfortunately PEP 257 falls short on specifying how to describe >> > arguments -- it >> > has only one example, it's not normative, and there's not much code that >> > follows >> > the example. The reST conventions are more common, but the stdlib itself >> > rarely >> > uses them. It would be nice to come up with a better convention that >> > takes PEP >> > 484 into account (so doc generators can incorporate the argument type >> > annotations into the generated output, merged with per-argument from the >> > docstring). >> >> I'll be happy to implement that. Basically if you use >> >> :param x: description >> :type x: type >> >> you could just leave out the "type" line and have it generated from the >> annotation. >> >> I guess it's very similar for Google and numpy style docstrings, which are >> supported by the (as of 1.3 built-in) Napoleon extension. >> >> Georg >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From mistersheik at gmail.com Mon Jan 19 21:52:59 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 19 Jan 2015 12:52:59 -0800 (PST) Subject: [Python-ideas] What's going on with PEP 448 - Additional Unpacking Generalizations ? In-Reply-To: <CAP7+vJKKmwuWq+4Ps2drFuQA5ParxdPKe_XDuAFATiQrZ6TZMQ@mail.gmail.com> References: <f97552ab-bfb8-434b-bc68-5120d394682e@googlegroups.com> <CAPTjJmpGkgh0onpJVBYB4KXOi-n8HRDx2R55XJFteO=ty3FRVQ@mail.gmail.com> <CAP7+vJKKmwuWq+4Ps2drFuQA5ParxdPKe_XDuAFATiQrZ6TZMQ@mail.gmail.com> Message-ID: <43272cca-20ea-4337-bd2a-39bdf981048f@googlegroups.com> Is there any way that I can help? On Thursday, January 15, 2015 at 1:05:45 PM UTC-5, Guido van Rossum wrote: > > I see no big problem with it from a language design perspective, but also > no reason to commit huge resources to it -- it's all very much in the "nice > to have" category. Unless you find a champion who wants to review the (old) > patch and so on, what's the point of accepting the PEP? > On Jan 15, 2015 3:28 AM, "Chris Angelico" <ros... at gmail.com <javascript:>> > wrote: > >> On Thu, Jan 15, 2015 at 11:22 AM, Neil Girdhar <miste... at gmail.com >> <javascript:>> wrote: >> > It says "standards track", but it doesn't seem to work in Python 3.4. >> I am >> > really looking forward to this! Any chance we can get a from >> __future__ for >> > inclusion in 3.4? Or maybe it can make it into 3.5? >> >> It also says "Status: Draft", which means it hasn't (yet) been >> accepted as part of the language. That means it's definitely not going >> to get into 3.4; it might be able to get into 3.5 if it gets approved >> very shortly, but the time frame is narrowing. Last chance to get it >> into 3.5 would be Beta 1, scheduled for May 24th (see PEP 478), and >> ideally it should get into an alpha before that. >> >> Would be nice to actually have it land somewhere. Seems it's been >> proposed for every Python version since 3.0. >> >> ChrisA >> _______________________________________________ >> Python-ideas mailing list >> Python... at python.org <javascript:> >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/55ad05c6/attachment.html> From mistersheik at gmail.com Tue Jan 20 00:02:11 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 19 Jan 2015 18:02:11 -0500 Subject: [Python-ideas] What's going on with PEP 448 - Additional Unpacking Generalizations ? In-Reply-To: <CAPTjJmrOCo5pzXCUe_nQ44X80FQMfb15QqiTx6AU14Wx7RDSwA@mail.gmail.com> References: <f97552ab-bfb8-434b-bc68-5120d394682e@googlegroups.com> <CAPTjJmpGkgh0onpJVBYB4KXOi-n8HRDx2R55XJFteO=ty3FRVQ@mail.gmail.com> <CAP7+vJKKmwuWq+4Ps2drFuQA5ParxdPKe_XDuAFATiQrZ6TZMQ@mail.gmail.com> <43272cca-20ea-4337-bd2a-39bdf981048f@googlegroups.com> <CAPTjJmrjEm2GuRC=JNeEb7oRZ6xJAYPVHw9Fp0oAOWePPrRRbA@mail.gmail.com> <CAA68w_=_Gt1CsT_S4SHjK5fKKxSUHCt0J=4zSnjU5jMqQ6L=FA@mail.gmail.com> <CAPTjJmrUu72Z=UBa6ej7+zV34O-hco-XovYrZ77pG8O29nBnwQ@mail.gmail.com> <CAPTjJmrOCo5pzXCUe_nQ44X80FQMfb15QqiTx6AU14Wx7RDSwA@mail.gmail.com> Message-ID: <CAA68w_n6PB_rMVdKxG5v2CcoDSJsSfC3+goCJ6mNFanhAUO7tA@mail.gmail.com> Fixed it on my system! (OS X Yosemite) http://hamelot.co.uk/programming/osx-gcc-dispatch_block_t-has-not-been-declared-invalid-typedef/ Okay, I'll get working on this patch soon. Thanks, Neil On Mon, Jan 19, 2015 at 5:39 PM, Chris Angelico <rosuav at gmail.com> wrote: > On Tue, Jan 20, 2015 at 9:22 AM, Chris Angelico <rosuav at gmail.com> wrote: > > Skipped tests are normal; if you're building on a system that doesn't > > have GDB, it'll skip test_gdb, because there's nothing to test. The > > failing tests, though, are likely to be a problem. What environment > > are you building on? Are you experienced with building Python from > > source? > > FWIW, here's my 'make test' summary. Debian Wheezy, Python 3.5 (tip), > amd64 architecture. > > 379 tests OK. > 1 test altered the execution environment: > test_warnings > 11 tests skipped: > test_devpoll test_gdb test_kqueue test_msilib test_ossaudiodev > test_startfile test_tk test_ttk_guionly test_winreg test_winsound > test_zipfile64 > > ChrisA > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/ef6e51c9/attachment.html> From kaiser.yann at gmail.com Tue Jan 20 00:11:23 2015 From: kaiser.yann at gmail.com (Yann Kaiser) Date: Mon, 19 Jan 2015 23:11:23 +0000 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> <54BC494E.3000609@canterbury.ac.nz> <CABicbJLdq519s-HDg5GM6Y1aeTbyhYrL-+fNqsxWbjn2B4C5nw@mail.gmail.com> <CAP7+vJKK5FwhtUU=3GG-OFL8hZmo3jPpvAV25wYE+CJUwkTQEA@mail.gmail.com> Message-ID: <CANUJvPV4AfSSArs988LnPS_RdDi3Ze0=Hu2j14=CEYAupP2YUw@mail.gmail.com> Before I forget, your typical forward-reference problem will most likely look like this: class X(...): def method(self, other:: X): ... On Mon Jan 19 2015 at 11:15:05 PM Guido van Rossum <guido at python.org> wrote: > On Mon, Jan 19, 2015 at 12:30 PM, Devin Jeanpierre <jeanpierreda at gmail.com > > wrote: > >> On Sun, Jan 18, 2015 at 4:01 PM, Greg <greg.ewing at canterbury.ac.nz> >> wrote: > > >> > Do you mean *static* types, or types in general? >> > >> > If they're only for static type checking, this seems a waste of a >> > facility that evaluates things at run time. Moreover, evaluating >> > them at run time is actually counterproductive, since it makes >> > dealing with things like forward references unnecessarily awkward. >> > It also introduces useless runtime overhead. And they only address >> > part of the problem, since they only apply to functions and not >> > other things we might want to specify the type of. >> >> Strong +1. The current proposal is a historical accident, it would not >> be designed this way if it was designed from scratch. That's not a >> nice thing to include in the core language. (Although it does happen >> often, since "practicality beats purity"). >> > > It's stronger than that. I'd say that all of Python is a historical > accident. > > >> I like the :: idea. >> > > Blechh. It smells of C++ and syntax hacks. PEP 3107 is over 8 years old > (also a historical accident :-). Let's stick with the devil we know. > You know, I feel pretty good when I get to write python helper scripts in Python 3 because I get to write def my_script(address, *, port:'p'=80): ... run(my_script) rather than @kwoargs('port') @annotate(port='p') def my_script(address, port=80): ... run(my_script) With just one parameter to annotate, we go from 19 characters plus imports in Python 2.7, to just 4. In 4 characters, in just as much clarity as with 19, I made "-p 8080" synonymous with "--port 8080". Didn't have to think it through, didn't have to go back and add imports, just had to wish "it would be nice if I could use -p instead". That's what I had in mind when writing this argument parser, when I added annotation support to it, when I made annotations the primary way of specifying anything with it. Annotations have been great, and this little project wouldn't have existed in another programming language. Reserving annotations for typing info, as you suggested is your long-term goal, takes that away and takes eager users of my library on Python 3 halfway back to Python 2. Not cool. I understand, from this thread and previous threads about this PEP, that you intended from the get-go parameter annotations to be used for type hinting and co. This appeared in the annotations PEP as a mere suggestion in a bullet point list, and more importantly did not appear in the docs. If the chain of historical accidents is to continue, maybe it shall include Python opting to spawn a separate kind of annotation better-suited for static analysis alongside the dynamic ones that can be used in all kinds of stuff that look ugly in other languages. If the problem against :: really is "There's no time before 3.5", that's really just "Postpone it until 3.6" waiting to be said. That'll be more time to see if whatever is drafted up is useful to static analysis tools and statically-analysed code bases. It has already waited 24 years after all. As for C++'s infamous "> >" vs ">>", "::" isn't comparable because ": :" does not exist and isn't about to. (Right?) The :: idea otherwise brings advantages to all as outlined before, so I'm a bit frustrated to see it dismissed on mere feeling. -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/1c5632b5/attachment-0001.html> From ron3200 at gmail.com Tue Jan 20 00:16:54 2015 From: ron3200 at gmail.com (Ron Adam) Date: Mon, 19 Jan 2015 17:16:54 -0600 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> Message-ID: <m9k397$eo9$1@ger.gmane.org> On 01/19/2015 02:28 PM, Chris Barker wrote: > On Mon, Jan 19, 2015 at 11:51 AM, Ron Adam <ron3200 at gmail.com > <mailto:ron3200 at gmail.com>> wrote: > > Here is a gist with a sample implementation: > > > https://gist.github.com/__PythonCHB/6e9ef7732a9074d9337a > <https://gist.github.com/PythonCHB/6e9ef7732a9074d9337a> > > > For the most part I think it looks good. > > Boost describes both a week and strong version, but I didn't see why > they choose the strong version. > > > Actually, Boost didn't choose, there is a flag you can set to select which > one to use. I don't want to do that, if you really want something special, > write it yourself. I did choose the "strong" version -- it just seemed more > conservative. Also it's what I think Steven chose for his version (though > not with and, but the result is the same) The two different cases probably should be two different functions, and not use a flag. I'm not suggesting we need both. > As, in the common case, the tolerance is approximate, and usually small, > then it doesn't matter which version we use: string, weak, or declared > which value to scale to. But I prefer a symmetric version, as I suspect hat > will be the least surprising -- it's good to get the same answer every > time, even if it is approximate! Well, approximate-in, approximate-out. Unfortunately that applies to all math. If you use approximations in your math, you will get approximate answers. But many computer programmers like things to be a bit more precise. So I suggest not using the word approximate or estimate in the docs. The calculation isn't an approximation even if the values you supply it are. It actually is a well defined range test. > I hope we can come to some consensus that something like this is > the way to go. > > Good examples will help with this. It may also help with choosing a > good name. > > > you mean use-case examples? rather than specific value examples? Yes, specific values don't indicate how something should be used. > To me, the strong version is an "is-good" test, and the weak version is > an "is-close" test. I think it could be important to some people. > > I like the idea of being able to use these as a teaching tool to > demonstrate how our ideas of closeness, equality, and inequality can be > subjective. > > > Are you suggesting that we allow a flag for the user to set to choose > whether ot use weak or string version? I'd rather not -- I see this is a > practical, works most of the time thing, not a teaching tool, or a > "provides every use case" tool. No flag, just that it needs to be well defined and not mix explanations of use of one with the other. Pick one, and then document how to use it correctly. At some point maybe someone will add the other if it's needed. It is possible to use one for the other if you take the differences into account in the arguments. > There are two cases... > > 1: (The weak version is require for this to work.) > > Two numbers are definitely not equivalent if they are further apart > than the largest error amount. (The larger number better indicates the > largeness of the the possible relative error.) > > And two numbers are close if you can't determine if they are > equivalent, or not-equivalent with certainty.* > > (* "close numbers" may include equivalent numbers if you define it as a > set of all definitely not-equivalent numbers.) > > 2: (The strong version is required for this to work.) > > A value is good if it's within a valid range with certainty. It is > less than the smaller relative range of either number. The smaller > number better indicates the magnitude of smallness. > > So case 1 should be used to test for errors, and case 2 should be used > to test for valid ranges. > > It seems you have the 2nd case in mind, and that's fine. Some of us > where thinking of the first case, and possibly switching from one to > the other during the discussion which is probably why it got confusing > or repetitious at some points. > > > yes, I suppose I do -- and again, in the common use case, where the > tolerance is also approximate, it really doesn't matter. I'm curious to what degree it can matter, given different size values and tolerances? > I think both of these are useful, but you definitely need to be clear > which one you are implementing, and to document it clearly. > > > yup. Cheers, Ron From guido at python.org Tue Jan 20 00:44:11 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 19 Jan 2015 15:44:11 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CANUJvPV4AfSSArs988LnPS_RdDi3Ze0=Hu2j14=CEYAupP2YUw@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> <54BC494E.3000609@canterbury.ac.nz> <CABicbJLdq519s-HDg5GM6Y1aeTbyhYrL-+fNqsxWbjn2B4C5nw@mail.gmail.com> <CAP7+vJKK5FwhtUU=3GG-OFL8hZmo3jPpvAV25wYE+CJUwkTQEA@mail.gmail.com> <CANUJvPV4AfSSArs988LnPS_RdDi3Ze0=Hu2j14=CEYAupP2YUw@mail.gmail.com> Message-ID: <CAP7+vJ+=bDz9SSTkk9GcdVrtg=qv6XVQ4nKXLy1GYxBEzfGkUw@mail.gmail.com> On Mon, Jan 19, 2015 at 3:11 PM, Yann Kaiser <kaiser.yann at gmail.com> wrote: > The :: idea otherwise brings advantages to all as outlined before, so I'm > a bit frustrated to see it dismissed on mere feeling. > I'm sure I can explain rationally why "::" is a bad idea given Python's historical approach to syntax, but I'm not sure that it would sway you, so I'm saving both of us time by just saying it's unPythonic. With that in mind, there is nothing in the current proposal that prevents "::" from being introduced in 3.6 -- the proposal doesn't change any syntax, so if "::" is possible now, it will still be possible in 3.6. So you have until Python 3.6 to implement it. PEP 484 will be provisional, so if we really liked "::" better by that time we can just withdraw PEP 484 at that time. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/d1f749d5/attachment.html> From me+python at ixokai.io Tue Jan 20 00:45:40 2015 From: me+python at ixokai.io (Stephen Hansen) Date: Mon, 19 Jan 2015 15:45:40 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CANUJvPXa8MxvpSOxV_5BuV6n4=rhOGpjnahMrU_8RJ-OFOtq6Q@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> <54BC494E.3000609@canterbury.ac.nz> <CABicbJLdq519s-HDg5GM6Y1aeTbyhYrL-+fNqsxWbjn2B4C5nw@mail.gmail.com> <5C091DF5-5ACB-479F-BB2D-302F2A60C3B0@gmail.com> <CANUJvPXa8MxvpSOxV_5BuV6n4=rhOGpjnahMrU_8RJ-OFOtq6Q@mail.gmail.com> Message-ID: <CAM1gar4mXURzN_KqNFj8b4h2nqpRCoNrnFj_9bgFeq4EfcdjFQ@mail.gmail.com> > > > I like the :: idea. >> >> +1 on the :: idea. >> >> As for the typing system, I think the proposal is pretty much spot on. I >> know I've been harping on annotations being forcefully restricted to types >> only, but the proposal itself (the typing library, etc.) is a really good >> proposal. Unless Python suddenly stops being Python, it's not too likely >> that any typing system can handle all possible cases of what we can do in >> the language (at least, not without becoming insanely unwieldy). >> > > The :: idea creates unambiguous space for static/type annotations which > wouldn't even have to compile and execute and thus escape less-than-optimal > solutions to the forward reference problems, all this without needlessly > interfering with existing or potential uses of annotations for run-time > purposes, a feature unique(?) to python. > > +1 on the :: idea. > Please, no. Type hinting is already making function definitions verbose enough as it is. The very idea that we'd have both ":" and "::" in the language and that somehow this isn't ambiguous is completely obscene to me. There's only so much room in a function definition for information while keeping it clear and readable. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/caf402bc/attachment.html> From chris.barker at noaa.gov Tue Jan 20 01:46:13 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Mon, 19 Jan 2015 16:46:13 -0800 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <8E24B81C-A241-4AF4-AFD3-FBD4E32C6B24@yahoo.com> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> <8E24B81C-A241-4AF4-AFD3-FBD4E32C6B24@yahoo.com> Message-ID: <-9096178132670898911@unknownmsgid> > On Jan 19, 2015, at 1:05 PM, Andrew Barnert <abarnert at yahoo.com> wrote: > > If you're working with measurement error or mechanical tolerance or something similar, you often have something with a non-unit mantissa. For example, the first resistor in the examples for the standard color code chart is 4.7KOhm +/- 10%, so you've got an error of 4.7e2. Sure -- I, at least, really wasn't thinking the use-case would be things like measurement testing. If it were a use-case like that, you would want to use the method where you specify which value the relative error is relative to. But using an absolute error would be easy in that case, too. So if folks think this a common use-case to support, that's the method to use. > Also, keep in mind that if you do an error propagation analysis, you're unlikely to get round numbers. This is very much the case that I am NOT expecting to support. If you can do error propagation analysis, you can (and probably will want to) write your own test criteria code. > The reasons you often only have one significant figure are (a) you're hand-waving the error analysis, Exactly, this is for the "do I have a big ol' bug?" kind of check, not the "is this an accurate algorithm?" check. Clearly time for a PEP -Chris B. From chris.barker at noaa.gov Tue Jan 20 01:55:34 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Mon, 19 Jan 2015 16:55:34 -0800 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <m9k397$eo9$1@ger.gmane.org> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> <m9k397$eo9$1@ger.gmane.org> Message-ID: <-3376261731312819627@unknownmsgid> > On Jan 19, 2015, at 3:17 PM, Ron Adam <ron3200 at gmail.com> wrote: > The two different cases probably should be two different functions, and not use a flag. I'm not suggesting we need both. I agree there--"strong" it is for the initial proposal at least. > Well, approximate-in, approximate-out. Unfortunately that applies to all math > But many computer programmers like things to be a bit more precise. Well, yes and no . Floating point has its limitations, and many programmers simply use double (python float), and hope the results are good enough. What I'm hoping is that this will at least make it more likely folks will apply the "good enough" criteria. Proper floating point error analysis is hard, most of us leave it for the experts, and I'm not suggesting this is useful for them. > > So I suggest not using the word approximate or estimate in the docs. The calculation isn't an approximation even if the values you supply it are. It actually is a well defined range test. > >> I hope we can come to some consensus that something like this is >> the way to go. >> >> Good examples will help with this. It may also help with choosing a >> good name. >> >> >> you mean use-case examples? rather than specific value examples? > > Yes, specific values don't indicate how something should be used. > > >> To me, the strong version is an "is-good" test, and the weak version is >> an "is-close" test. I think it could be important to some people. >> >> I like the idea of being able to use these as a teaching tool to >> demonstrate how our ideas of closeness, equality, and inequality can be >> subjective. >> >> >> Are you suggesting that we allow a flag for the user to set to choose >> whether ot use weak or string version? I'd rather not -- I see this is a >> practical, works most of the time thing, not a teaching tool, or a >> "provides every use case" tool. > > No flag, just that it needs to be well defined and not mix explanations of use of one with the other. Pick one, and then document how to use it correctly. At some point maybe someone will add the other if it's needed. > > It is possible to use one for the other if you take the differences into account in the arguments. > > >> There are two cases... >> >> 1: (The weak version is require for this to work.) >> >> Two numbers are definitely not equivalent if they are further apart >> than the largest error amount. (The larger number better indicates the >> largeness of the the possible relative error.) >> >> And two numbers are close if you can't determine if they are >> equivalent, or not-equivalent with certainty.* >> >> (* "close numbers" may include equivalent numbers if you define it as a >> set of all definitely not-equivalent numbers.) >> >> 2: (The strong version is required for this to work.) >> >> A value is good if it's within a valid range with certainty. It is >> less than the smaller relative range of either number. The smaller >> number better indicates the magnitude of smallness. >> >> So case 1 should be used to test for errors, and case 2 should be used >> to test for valid ranges. >> >> It seems you have the 2nd case in mind, and that's fine. Some of us >> where thinking of the first case, and possibly switching from one to >> the other during the discussion which is probably why it got confusing >> or repetitious at some points. >> >> >> yes, I suppose I do -- and again, in the common use case, where the >> tolerance is also approximate, it really doesn't matter. > > I'm curious to what degree it can matter, given different size values and tolerances? > > >> I think both of these are useful, but you definitely need to be clear >> which one you are implementing, and to document it clearly. >> >> >> yup. > > Cheers, > Ron > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From chris.barker at noaa.gov Tue Jan 20 02:04:56 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Mon, 19 Jan 2015 17:04:56 -0800 Subject: [Python-ideas] Conventions for Python code documentation (was: PEP 484 (Type Hints) -- first draft round) In-Reply-To: <CAN8d9gnN=MGNujsDm_5fqauQekrAOEPRP1jVquo0z6JVmyOokQ@mail.gmail.com> References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> <85oapur4d8.fsf_-_@benfinney.id.au> <CAN-Kwu3Nj0=1+6B=uNmJz8D8LORfAAu62GLJ8_R5ZwwKjxUg=Q@mail.gmail.com> <CAP7+vJ+=3uebwL=+QOajwYVd=h+RaEFsRy8GqKQnDBFrjt7uog@mail.gmail.com> <CAN8d9gnN=MGNujsDm_5fqauQekrAOEPRP1jVquo0z6JVmyOokQ@mail.gmail.com> Message-ID: <3694529471710485200@unknownmsgid> On Jan 19, 2015, at 2:16 PM, "Philipp A." <flying-sheep at web.de> wrote: I have am idea: how about including a @paramdoc decorator I always teach that the @ syntax is a decoration, not a decorator, whereas a decorator is a function that takes a function and returns another function ( usually a customized version of the passed in function). This distinction between decorators and decoration syntax keeps the door open to do just about anything with decorations, but am I the only one that thinks it's a bad idea to have it be for "any old thing we want to hang off a function"? Let's keep decorations for decorators... -Chris that accepts one named string argument per parameter and appends information gathered via type annotations and default values to the docstring? E.g. ~~~~ @paramdoc( main_course="Meaty because our society is medieval", side_dish="Optional, meat is enough", __return__="Objective opinion") def eat(main_course: Meat, side_dish=None: Food) -> Opinion: """ Decides how well the meat and side dish taste together """ ... ~~~~ After this, eat.__doc__ would be an object that retains the raw parts as properties, but otherwise is an elongated version of the original docstring. Its fourth and third to last line would read something like this: ~~~~ side_dish (Food; defaults to None): Optional, meat is enough ~~~~ What do you think? Guido van Rossum <guido <guido at python.org>@ <guido at python.org>python.org <guido at python.org>> schrieb am Mo., 19. Jan. 2015 20:56: _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/8df546cd/attachment.html> From apalala at gmail.com Tue Jan 20 04:21:01 2015 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Mon, 19 Jan 2015 22:51:01 -0430 Subject: [Python-ideas] Conventions for Python code documentation (was: PEP 484 (Type Hints) -- first draft round) In-Reply-To: <3694529471710485200@unknownmsgid> References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> <85oapur4d8.fsf_-_@benfinney.id.au> <CAN-Kwu3Nj0=1+6B=uNmJz8D8LORfAAu62GLJ8_R5ZwwKjxUg=Q@mail.gmail.com> <CAP7+vJ+=3uebwL=+QOajwYVd=h+RaEFsRy8GqKQnDBFrjt7uog@mail.gmail.com> <CAN8d9gnN=MGNujsDm_5fqauQekrAOEPRP1jVquo0z6JVmyOokQ@mail.gmail.com> <3694529471710485200@unknownmsgid> Message-ID: <CAN1YFWt14embm3Z5c4ds0Ndhm7ci=6O84qv=U+S1AmSEH+AWog@mail.gmail.com> On Mon, Jan 19, 2015 at 8:34 PM, Chris Barker - NOAA Federal < chris.barker at noaa.gov> wrote: > Let's keep decorations for decorators... +1 -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/6309f5a0/attachment.html> From mistersheik at gmail.com Tue Jan 20 05:10:35 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 19 Jan 2015 20:10:35 -0800 (PST) Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <-3376261731312819627@unknownmsgid> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> <m9k397$eo9$1@ger.gmane.org> <-3376261731312819627@unknownmsgid> Message-ID: <6ea13fe6-0831-4126-a94b-15b6bd1c83a7@googlegroups.com> If you decide to make a PEP, please list the other algorithms you found and their definitions. Personally, I'm for being consistent with numpy and defining math.isclose similar to numpy.isclose for consistency alone. If you decide to invent a relative error function, my suggestion is: (a-b)/b + log(b/a), which is nonnegative, zero only at equality, and otherwise penalizes positive a for being different than some target positive b. To me, it seems like guessing b using 1.9b is better than guessing it as 0.1b, and so on. This corresponds to exponential KL divergence, which has a clear statistical meaning, but only applies to positive numbers. Best, Neil On Monday, January 19, 2015 at 7:59:34 PM UTC-5, Chris Barker - NOAA Federal wrote: > > > On Jan 19, 2015, at 3:17 PM, Ron Adam <ron... at gmail.com <javascript:>> > wrote: > > The two different cases probably should be two different functions, and > not use a flag. I'm not suggesting we need both. > > I agree there--"strong" it is for the initial proposal at least. > > > Well, approximate-in, approximate-out. Unfortunately that applies to > all math > > > But many computer programmers like things to be a bit more precise. > > Well, yes and no . Floating point has its limitations, and many > programmers simply use double (python float), and hope the results are > good enough. What I'm hoping is that this will at least make it more > likely folks will apply the "good enough" criteria. > > Proper floating point error analysis is hard, most of us leave it for > the experts, and I'm not suggesting this is useful for them. > > > > > > > So I suggest not using the word approximate or estimate in the docs. > The calculation isn't an approximation even if the values you supply it > are. It actually is a well defined range test. > > > >> I hope we can come to some consensus that something like this is > >> the way to go. > >> > >> Good examples will help with this. It may also help with choosing a > >> good name. > >> > >> > >> you mean use-case examples? rather than specific value examples? > > > > Yes, specific values don't indicate how something should be used. > > > > > >> To me, the strong version is an "is-good" test, and the weak version > is > >> an "is-close" test. I think it could be important to some people. > >> > >> I like the idea of being able to use these as a teaching tool to > >> demonstrate how our ideas of closeness, equality, and inequality can > be > >> subjective. > >> > >> > >> Are you suggesting that we allow a flag for the user to set to choose > >> whether ot use weak or string version? I'd rather not -- I see this is > a > >> practical, works most of the time thing, not a teaching tool, or a > >> "provides every use case" tool. > > > > No flag, just that it needs to be well defined and not mix explanations > of use of one with the other. Pick one, and then document how to use it > correctly. At some point maybe someone will add the other if it's needed. > > > > It is possible to use one for the other if you take the differences into > account in the arguments. > > > > > >> There are two cases... > >> > >> 1: (The weak version is require for this to work.) > >> > >> Two numbers are definitely not equivalent if they are further apart > >> than the largest error amount. (The larger number better indicates > the > >> largeness of the the possible relative error.) > >> > >> And two numbers are close if you can't determine if they are > >> equivalent, or not-equivalent with certainty.* > >> > >> (* "close numbers" may include equivalent numbers if you define it > as a > >> set of all definitely not-equivalent numbers.) > >> > >> 2: (The strong version is required for this to work.) > >> > >> A value is good if it's within a valid range with certainty. It is > >> less than the smaller relative range of either number. The smaller > >> number better indicates the magnitude of smallness. > >> > >> So case 1 should be used to test for errors, and case 2 should be > used > >> to test for valid ranges. > >> > >> It seems you have the 2nd case in mind, and that's fine. Some of us > >> where thinking of the first case, and possibly switching from one to > >> the other during the discussion which is probably why it got > confusing > >> or repetitious at some points. > >> > >> > >> yes, I suppose I do -- and again, in the common use case, where the > >> tolerance is also approximate, it really doesn't matter. > > > > I'm curious to what degree it can matter, given different size values > and tolerances? > > > > > >> I think both of these are useful, but you definitely need to be > clear > >> which one you are implementing, and to document it clearly. > >> > >> > >> yup. > > > > Cheers, > > Ron > > > > _______________________________________________ > > Python-ideas mailing list > > Python... at python.org <javascript:> > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python... at python.org <javascript:> > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/e5de9d94/attachment-0001.html> From abarnert at yahoo.com Tue Jan 20 05:37:21 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 19 Jan 2015 20:37:21 -0800 Subject: [Python-ideas] Conventions for Python code documentation (was: PEP 484 (Type Hints) -- first draft round) In-Reply-To: <3694529471710485200@unknownmsgid> References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> <85oapur4d8.fsf_-_@benfinney.id.au> <CAN-Kwu3Nj0=1+6B=uNmJz8D8LORfAAu62GLJ8_R5ZwwKjxUg=Q@mail.gmail.com> <CAP7+vJ+=3uebwL=+QOajwYVd=h+RaEFsRy8GqKQnDBFrjt7uog@mail.gmail.com> <CAN8d9gnN=MGNujsDm_5fqauQekrAOEPRP1jVquo0z6JVmyOokQ@mail.gmail.com> <3694529471710485200@unknownmsgid> Message-ID: <326DA4B5-0015-499D-9C9B-5BA0F4D0449F@yahoo.com> On Jan 19, 2015, at 17:04, Chris Barker - NOAA Federal <chris.barker at noaa.gov> wrote: > On Jan 19, 2015, at 2:16 PM, "Philipp A." <flying-sheep at web.de> wrote: > >> I have am idea: how about including a @paramdoc decorator >> > I always teach that the @ syntax is a decoration, not a decorator, whereas a decorator is a function that takes a function and returns another function ( usually a customized version of the passed in function). This distinction between decorators and decoration syntax keeps the door open to do just about anything with decorations, but am I the only one that thinks it's a bad idea to have it be for "any old thing we want to hang off a function"? But his paramdoc(?) is clearly a function that takes a function and returns a customized version of the function passed in--hence a decorator. And I don't know what distinction you're trying to make. I can guess at a few possibilities, but none of them work. Sure, paramdoc itself is a function that takes some arguments and returns a decorator function, but the same is true for all kinds of things we normally call decorators, like lru_cache and partial. And the function is customized by having a different doc string instead of, say, different parameter annotations or a closure that runs some extra code before the main code, but I don't see how that makes a difference. And the decorator would likely be implemented by mutating and returning the function instead of creating and returning a wrapper, but that's just an implementation detail that's irrelevant to the user (and wraps and many other well-known decorators work that way). > Let's keep decorations for decorators... > > -Chris > > >> that accepts one named string argument per parameter and appends information gathered via type annotations and default values to the docstring? >> >> E.g. >> >> ~~~~ >> @paramdoc( >> main_course="Meaty because our society is medieval", >> side_dish="Optional, meat is enough", >> __return__="Objective opinion") >> def eat(main_course: Meat, side_dish=None: Food) -> Opinion: >> """ >> Decides how well the meat and side dish taste together >> """ >> ... >> ~~~~ >> >> After this, eat.__doc__ would be an object that retains the raw parts as properties, but otherwise is an elongated version of the original docstring. Its fourth and third to last line would read something like this: >> >> ~~~~ >> side_dish (Food; defaults to None): >> Optional, meat is enough >> ~~~~ >> >> What do you think? >> >> Guido van Rossum <guido at python.org> schrieb am Mo., 19. Jan. 2015 20:56: >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/2e86705e/attachment.html> From mistersheik at gmail.com Tue Jan 20 05:29:37 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 19 Jan 2015 20:29:37 -0800 (PST) Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> Message-ID: <c88ee7a0-b0b0-40f8-bb61-527d7b44664a@googlegroups.com> Also for complex numbers, I think comparing the magnitude (distance from the origin, or absolute value) of (x-y) to the size of x or y makes more sense than calling is_close on the real and imaginary parts. What if the real parts are much larger than the imaginary parts, e.g. x=1e5+1e-5j, y=1e5-1e-5j. Do you think x and y are not close? Best, Neil On Monday, January 19, 2015 at 1:33:44 AM UTC-5, Chris Barker wrote: > > OK folks, > > There has been a lot of chatter about this, which I think has served to > provide some clarity, at least to me. However, I'm concerned that the > upshot, at least for folks not deep into the discussion, will be: clearly > there are too many use-case specific details to put any one thing in the > std lib. But I still think we can provide something that is useful for most > use-cases, and would like to propose what that is, and what the decision > points are: > > A function for the math module, called somethign like "is_close", > "approx_equal", etc. It will compute a relative tolerance, with a default > maybe around 1-e12, with the user able to specify the tolerance they want. > > Optionally, the user can specify an "minimum absolute tolerance", it will > default to zero, but can be set so that comparisons to zero can be handled > gracefully. > > The relative tolerance will be computed from the smallest of the two input > values, so as to get symmetry : is_close(a,b) == is_close(b,a). (this is > the Boost "strong" definition, and what is used by Steven D'Aprano's code > in the statistics test module) > > Alternatively, the relative error could be computed against a particular > one of the input values (the second one?). This would be asymmetric, but be > more clear exactly how "relative" is defined, and be closer to what people > may expect when using it as a "actual vs expected" test. --- "expected" > would be the scaling value. If the tolerance is small, it makes very little > difference anyway, so I'm happy with whatever consensus moves us to. Note > that if we go this way, then the parameter names should make it at least a > little more clear -- maybe "actual" and "expected", rather than x and y or > a and b or... and the function name should be something like is_close_to, > rather than just is_close. > > It will be designed for floating point numbers, and handle inf, -inf, and > NaN "properly". But is will also work with other numeric types, to the > extent that duck typing "just works" (i.e. division and comparisons all > work). > > complex numbers will be handled by: > is_close(x.real, y.real) and is_close(x.imag, y.imag) > (but i haven't written any code for that yet) > > It will not do a simple absolute comparison -- that is the job of a > different function, or, better yet, folks just write it themselves: > > abs(x - y) <= delta > > really isn't much harder to write than a function call: > > absolute_diff(x,y,delta) > > Here is a gist with a sample implementation: > > https://gist.github.com/PythonCHB/6e9ef7732a9074d9337a > > I need to add more tests, and make the test proper unit tests, but it's a > start. > > I also need to see how it does with other data types than float -- > hopefully, it will "just work" with the core set. > > I hope we can come to some consensus that something like this is the way > to go. > > -Chris > > > > > > > > > > > > > > > > On Sun, Jan 18, 2015 at 11:27 AM, Ron Adam <ron... at gmail.com <javascript:> > > wrote: > >> >> >> On 01/17/2015 11:37 PM, Chris Barker wrote: >> >>> (Someone claimed that 'nothing is close to zero'. This is >>> nonsensical both in applied math and everyday life.) >>> >>> >>> I'm pretty sure someone (more than one of use) asserted that "nothing is >>> *relatively* close to zero -- very different. >>> >> >> Yes, that is the case. >> >> >> And I really wanted a way to have a default behavior that would do a >>> reasonable transition to an absolute tolerance near zero, but I no longer >>> thing that's possible. (numpy's implimentaion kind of does that, but it >>> is >>> really wrong for small numbers, and if you made the default min_tolerance >>> the smallest possible representable number, it really wouldn't be useful. >>> >> >> I'm going to try to summarise what I got out of this discussion. Maybe >> it will help bring some focus to the topic. >> >> I think there are two case's to consider. >> >> # The most common case. >> rel_is_good(actual, expected, delta) # value +- %delta. >> >> # Testing for possible equivalence? >> rel_is_close(value1, value2, delta) # %delta close to each other. >> >> I don't think they are quite the same thing. >> >> rel_is_good(9, 10, .1) --> True >> rel_is_good(10, 9, .1) --> False >> >> rel_is_close(9, 10, .1) --> True >> rel_is_close(10, 9, .1) --> True >> >> >> In the "is close" case, it shouldn't matter what order the arguments are >> given. The delta is the distance from the larger number the smaller number >> is. (of the same sign) >> >> So when calculating the relative error from two values, you want it to be >> consistent with the rel_is_close function. >> >> rel_is_close(a, b, delta) <---> rel_err(a, b) <= delta >> >> And you should not use the rel_err function in the rel_is_good function. >> >> >> >> The next issue is, where does the numeric accuracy of the data, >> significant digits, and the languages accuracy (ULPs), come into the >> picture. >> >> My intuition.. I need to test the idea to make a firmer claim.. is that >> in the case of is_good, you want to exclude the uncertain parts, but with >> is_close, you want to include the uncertain parts. >> >> Two values "are close" if you can't tell one from the other with >> certainty. The is_close range includes any uncertainty. >> >> A value is good if it's within a range with certainty. And this excludes >> any uncertainty. >> >> This is where taking in consideration of an absolute delta comes in. The >> minimum range for both is the uncertainty of the data. But is_close and >> is_good do different things with it. >> >> Of course all of this only applies if you agree with these definitions of >> is_close, and is_good. ;) >> >> Cheers, >> Ron >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python... at python.org <javascript:> >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.... at noaa.gov <javascript:> > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/9afdef67/attachment-0001.html> From guido at python.org Tue Jan 20 05:42:46 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 19 Jan 2015 20:42:46 -0800 Subject: [Python-ideas] Conventions for Python code documentation (was: PEP 484 (Type Hints) -- first draft round) In-Reply-To: <3694529471710485200@unknownmsgid> References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> <85oapur4d8.fsf_-_@benfinney.id.au> <CAN-Kwu3Nj0=1+6B=uNmJz8D8LORfAAu62GLJ8_R5ZwwKjxUg=Q@mail.gmail.com> <CAP7+vJ+=3uebwL=+QOajwYVd=h+RaEFsRy8GqKQnDBFrjt7uog@mail.gmail.com> <CAN8d9gnN=MGNujsDm_5fqauQekrAOEPRP1jVquo0z6JVmyOokQ@mail.gmail.com> <3694529471710485200@unknownmsgid> Message-ID: <CAP7+vJL4PAZ31=QLBUdjQzLDiik02L5zXocm7CtdmEXem3dJBA@mail.gmail.com> On Mon, Jan 19, 2015 at 5:04 PM, Chris Barker - NOAA Federal < chris.barker at noaa.gov> wrote: > I always teach that the @ syntax is a decoration, not a decorator, whereas > a decorator is a function that takes a function and returns another > function ( usually a customized version of the passed in function). This > distinction between decorators and decoration syntax keeps the door open to > do just about anything with decorations, but am I the only one that thinks > it's a bad idea to have it be for "any old thing we want to hang off a > function"? > I think you're the only one who makes this distinction. In common use "decorator" is used to describe both the syntax and the function invoked by the syntax. "Decoration" is never (well, very rarely) used. And calling any function that takes a function and returns one a decorator feels overreaching -- I'd only call it a decorator if it is intended to use with the decorator syntax. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/aaf89741/attachment.html> From greg.ewing at canterbury.ac.nz Tue Jan 20 07:12:33 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 20 Jan 2015 19:12:33 +1300 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJKK5FwhtUU=3GG-OFL8hZmo3jPpvAV25wYE+CJUwkTQEA@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> <54BC494E.3000609@canterbury.ac.nz> <CABicbJLdq519s-HDg5GM6Y1aeTbyhYrL-+fNqsxWbjn2B4C5nw@mail.gmail.com> <CAP7+vJKK5FwhtUU=3GG-OFL8hZmo3jPpvAV25wYE+CJUwkTQEA@mail.gmail.com> Message-ID: <54BDF1D1.70904@canterbury.ac.nz> Guido van Rossum wrote: > On Mon, Jan 19, 2015 at 12:30 PM, Devin Jeanpierre > <jeanpierreda at gmail.com <mailto:jeanpierreda at gmail.com>> wrote: > > I like the :: idea. > > Blechh. It smells of C++ and syntax hacks. That smells like a knee-jerk reaction to me. Just because you've had bad experiences with it in C++ doesn't mean the symbol itself is inherently evil. But there are alternatives. We could borrow 'as' from a certain other language: num_feathers as int = 0 Or exploit the fact that '{' following an identifier is currently undefined: num_feathers {int} = 0 -- Greg From greg.ewing at canterbury.ac.nz Tue Jan 20 07:22:48 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 20 Jan 2015 19:22:48 +1300 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAM1gar4mXURzN_KqNFj8b4h2nqpRCoNrnFj_9bgFeq4EfcdjFQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> <54BC494E.3000609@canterbury.ac.nz> <CABicbJLdq519s-HDg5GM6Y1aeTbyhYrL-+fNqsxWbjn2B4C5nw@mail.gmail.com> <5C091DF5-5ACB-479F-BB2D-302F2A60C3B0@gmail.com> <CANUJvPXa8MxvpSOxV_5BuV6n4=rhOGpjnahMrU_8RJ-OFOtq6Q@mail.gmail.com> <CAM1gar4mXURzN_KqNFj8b4h2nqpRCoNrnFj_9bgFeq4EfcdjFQ@mail.gmail.com> Message-ID: <54BDF438.5050504@canterbury.ac.nz> Stephen Hansen wrote: > Type hinting is already making function definitions verbose enough as it > is. The very idea that we'd have both ":" and "::" in the language and > that somehow this isn't ambiguous is completely obscene to me. If we were to *really* use :: the way Haskell uses it, then the type information wouldn't be in the function header. It would be on a separate line, much like a decorator: quack :: lambda float, int: AnimalSound def quack(volume, num_times): ... -- Greg From me+python at ixokai.io Tue Jan 20 07:35:45 2015 From: me+python at ixokai.io (Stephen Hansen) Date: Mon, 19 Jan 2015 22:35:45 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <54BDF438.5050504@canterbury.ac.nz> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> <54BC494E.3000609@canterbury.ac.nz> <CABicbJLdq519s-HDg5GM6Y1aeTbyhYrL-+fNqsxWbjn2B4C5nw@mail.gmail.com> <5C091DF5-5ACB-479F-BB2D-302F2A60C3B0@gmail.com> <CANUJvPXa8MxvpSOxV_5BuV6n4=rhOGpjnahMrU_8RJ-OFOtq6Q@mail.gmail.com> <CAM1gar4mXURzN_KqNFj8b4h2nqpRCoNrnFj_9bgFeq4EfcdjFQ@mail.gmail.com> <54BDF438.5050504@canterbury.ac.nz> Message-ID: <CAM1gar44vrW656SjVet00xp1r-KT7WphMRM0pSJmd33nNH1UjA@mail.gmail.com> On Mon, Jan 19, 2015 at 10:22 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote: > Stephen Hansen wrote: > >> Type hinting is already making function definitions verbose enough as it >> is. The very idea that we'd have both ":" and "::" in the language and that >> somehow this isn't ambiguous is completely obscene to me. >> > > If we were to *really* use :: the way Haskell uses it, > then the type information wouldn't be in the function > header. It would be on a separate line, much like a > decorator: > > quack :: lambda float, int: AnimalSound > def quack(volume, num_times): > ... > > That's still verbose, its simply vertically verbose. We have decorators for adding information in the vertical space. Again the duplication of effect seems crazy to me: @foo bar :: something, baz: something def bar(baz): pass This reads like a terrible joke. You have : and :: and there's absolutely nothing intuitive or even suggestive about one verses the other, they're almost identical after all. Then you have decorators and ... whatever you call that line? What's the difference? We already have an approved syntax for tagging information to function arguments and to signify return type. We shouldn't have two. Almost no one uses function annotations, in eight years almost no one has come up with a good general use-case for them besides type hinting, which they were always meant for. Let's use them. I do wish you could annotate assignment statements (say, a: int = 5) as I hate the comment feature fiercely, but we can't all have everything. Worst case scenario, someone wants to use the annotations for something else, doesn't get to use mypy or other competing tool on that file. We don't need two syntaxes for this problem space. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/5f06c168/attachment.html> From tjreedy at udel.edu Tue Jan 20 07:39:02 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 20 Jan 2015 01:39:02 -0500 Subject: [Python-ideas] Conventions for Python code documentation In-Reply-To: <CAP7+vJ+=3uebwL=+QOajwYVd=h+RaEFsRy8GqKQnDBFrjt7uog@mail.gmail.com> References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> <85oapur4d8.fsf_-_@benfinney.id.au> <CAN-Kwu3Nj0=1+6B=uNmJz8D8LORfAAu62GLJ8_R5ZwwKjxUg=Q@mail.gmail.com> <CAP7+vJ+=3uebwL=+QOajwYVd=h+RaEFsRy8GqKQnDBFrjt7uog@mail.gmail.com> Message-ID: <m9kt66$a1n$1@ger.gmane.org> On 1/19/2015 2:54 PM, Guido van Rossum wrote: > Unfortunately PEP 257 falls short on specifying how to describe > arguments -- it has only one example, it's not normative, and there's > not much code that follows the example. The reST conventions are more > common, but the stdlib itself rarely uses them. Because I have not seen such used, except maybe once, I have been meaning to ask if I am allowed to use rst markup in stdlib docstrings? > It would be nice to come > up with a better convention that takes PEP 484 into account (so doc > generators can incorporate the argument type annotations into the > generated output, merged with per-argument from the docstring). My impression is that the plan is that stdlib .py files would not have type annotations, but that they might be in stub files. Still true? Would type hints be added to the signatures in .rst docs? -- Terry Jan Reedy From tjreedy at udel.edu Tue Jan 20 07:50:58 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 20 Jan 2015 01:50:58 -0500 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <CALGmxEJAvRQHxzddrnH9noeRhQswEfJsN2o0qOz7-xv0bb8ATQ@mail.gmail.com> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <87lhkzw6mo.fsf@uwakimon.sk.tsukuba.ac.jp> <CALGmxEJAvRQHxzddrnH9noeRhQswEfJsN2o0qOz7-xv0bb8ATQ@mail.gmail.com> Message-ID: <m9ktsi$k5c$1@ger.gmane.org> On 1/19/2015 2:54 PM, Chris Barker wrote: > But I think that it should have a name that (at least to some extent) > indicates what it actually does (unlike is_close(), which users are > likely to assume means what they think it means because they haven't > thought about the alternatives). > > > Indeed -- I think that the name of the function, and it's parameters > need to be carefully chosen to best convey what they mean. (and the > docstring). If the function only handles 'relatively close', I would be happier with 'relative_close' or 'rel_close' than 'is close' since the latter tends to imply to me absolute closeness. The proposed doc (and docstring) should mention absolute tolerance and 'abs(a-b) < tol' as an alternative, expecially near or at 0. -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Tue Jan 20 07:59:40 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 20 Jan 2015 19:59:40 +1300 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAM1gar44vrW656SjVet00xp1r-KT7WphMRM0pSJmd33nNH1UjA@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> <54BC494E.3000609@canterbury.ac.nz> <CABicbJLdq519s-HDg5GM6Y1aeTbyhYrL-+fNqsxWbjn2B4C5nw@mail.gmail.com> <5C091DF5-5ACB-479F-BB2D-302F2A60C3B0@gmail.com> <CANUJvPXa8MxvpSOxV_5BuV6n4=rhOGpjnahMrU_8RJ-OFOtq6Q@mail.gmail.com> <CAM1gar4mXURzN_KqNFj8b4h2nqpRCoNrnFj_9bgFeq4EfcdjFQ@mail.gmail.com> <54BDF438.5050504@canterbury.ac.nz> <CAM1gar44vrW656SjVet00xp1r-KT7WphMRM0pSJmd33nNH1UjA@mail.gmail.com> Message-ID: <54BDFCDC.9050802@canterbury.ac.nz> Stephen Hansen wrote: > On Mon, Jan 19, 2015 at 10:22 PM, Greg Ewing > <greg.ewing at canterbury.ac.nz <mailto:greg.ewing at canterbury.ac.nz>> wrote: > > quack :: lambda float, int: AnimalSound > def quack(volume, num_times): > ... > > That's still verbose, its simply vertically verbose. We have decorators > for adding information in the vertical space. > > Then you have decorators and > ... whatever you call that line? What's the difference? The difference is that the :: annotations would *not* be evaluated a run time. There's currently no way to get that with a decorator, or anything else in the language, short of abusing comments or string literals. Yet it's what you really want for static type checking. -- Greg From me+python at ixokai.io Tue Jan 20 08:52:01 2015 From: me+python at ixokai.io (Stephen Hansen) Date: Mon, 19 Jan 2015 23:52:01 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <54BDFCDC.9050802@canterbury.ac.nz> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> <54BC494E.3000609@canterbury.ac.nz> <CABicbJLdq519s-HDg5GM6Y1aeTbyhYrL-+fNqsxWbjn2B4C5nw@mail.gmail.com> <5C091DF5-5ACB-479F-BB2D-302F2A60C3B0@gmail.com> <CANUJvPXa8MxvpSOxV_5BuV6n4=rhOGpjnahMrU_8RJ-OFOtq6Q@mail.gmail.com> <CAM1gar4mXURzN_KqNFj8b4h2nqpRCoNrnFj_9bgFeq4EfcdjFQ@mail.gmail.com> <54BDF438.5050504@canterbury.ac.nz> <CAM1gar44vrW656SjVet00xp1r-KT7WphMRM0pSJmd33nNH1UjA@mail.gmail.com> <54BDFCDC.9050802@canterbury.ac.nz> Message-ID: <CAM1gar7ZiZ6jmDkVx7c62zZ+gpn4xJD96mV79JLoi5RekU2GzA@mail.gmail.com> > > The difference is that the :: annotations would *not* be > evaluated a run time. There's currently no way to get > that with a decorator, or anything else in the language, > short of abusing comments or string literals. > Yet it's what you really want for static type checking. We already have : annotations. If you don't want evaluation at run time, "abuse" string literals -- all you're doing is removing the need to put quotes around the argument. Typing two characters is a very poor justification for new syntax. Especially when its at the cost of making the whole thing horribly more complex by having two syntaxes for essentially the same thing ... and especially when they look confusingly similar and there is no intuitive way to make a distinction between them. def foo(bar: int:: whatever): pass Please, no. We really, really, really do not need two kinds of annotations that look essentially identical. def foo(bar: 'int'): pass if what you "really want" for static type checking is no evaluation, is no big deal. x : 'xyz' is how you spell 'not evaluating' and its only one extra character to type. As an aside, I'm not convinced that what you really want for static type checking is no evaluation, since mypy seems seems to work just fine with evaluation or strings either way. -1 on a second form of annotations. Its like giving a dog a hat, you might think it keeps its ears dry but the dog doesn't need it and looks ridiculous. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150119/793c0ce5/attachment.html> From p.f.moore at gmail.com Tue Jan 20 10:36:19 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 20 Jan 2015 09:36:19 +0000 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <6ea13fe6-0831-4126-a94b-15b6bd1c83a7@googlegroups.com> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> <m9k397$eo9$1@ger.gmane.org> <-3376261731312819627@unknownmsgid> <6ea13fe6-0831-4126-a94b-15b6bd1c83a7@googlegroups.com> Message-ID: <CACac1F-4p=vD-VeDG9isyfSc08zSTnaNmxRGTGE9zjq5JChYaA@mail.gmail.com> On 20 January 2015 at 04:10, Neil Girdhar <mistersheik at gmail.com> wrote: > If you decide to make a PEP, please list the other algorithms you found and > their definitions. Personally, I'm for being consistent with numpy and > defining math.isclose similar to numpy.isclose for consistency alone. Good point. Also, in the PEP, cover the use cases being considered as relevant (and just as importantly, those which aren't). As a person who doesn't really do much numerical coding, I see the following cases where I'd even consider using a function like this: 1. Testing when to stop with an iterative algorithm. These would normally be "toy" examples, because if the right answer really mattered to me, I would probably be using a specialised function (from somewhere like numpy or mpmath). 2. Comparing with decimal literals ("is nearly 0.1"). But I hope I would immediately rethink in that case, as I know that the issue here is not about being "nearly equal" but about representability of decimals as floats. If not, this use case counts more as an attractive nuisance than as a valid use case. 3. Dealing with approximate measurements. I can sort of imagine uses for this, but it's not something I've ever needed, so again I'd probably be more likely to reach for a specialist module or some literature rather than just hitting it with a stdlib function. I guess using it (with a full understanding of how it worked) in the internals of the sort of specialist function that I'd expect to use (such as the statistics module that's been mentioned in this thread) is probably the main use case I can see. In summary, I can't really think of a case in my experience as a non-specialist where I'd consider a stdlib function to be the right answer for my use. So spelling out in the PEP those cases where it would be useful is pretty important. Paul From steve at pearwood.info Tue Jan 20 11:40:12 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 20 Jan 2015 21:40:12 +1100 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <6ea13fe6-0831-4126-a94b-15b6bd1c83a7@googlegroups.com> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> <m9k397$eo9$1@ger.gmane.org> <-3376261731312819627@unknownmsgid> <6ea13fe6-0831-4126-a94b-15b6bd1c83a7@googlegroups.com> Message-ID: <20150120104012.GU18556@ando.pearwood.info> On Mon, Jan 19, 2015 at 08:10:35PM -0800, Neil Girdhar wrote: > If you decide to invent a relative error function, The error functions we have been talking about are hardly "invented". They're mathematically simple and obvious, and can be found in just about any undergraduate book on numerical computing: - The absolute error between two quantities a and b is the absolute difference between them, abs(a-b). - The relative error is the difference relative to some denominator d, abs(a-b)/abs(d), typically with d=a or d=b. If you happen to know that b is the correct value, then it is common to choose b as the denominator. If you have no a priori reason to think either a or b is correct, or if you prefer a symmetrical function, a common choice is to use d = min(abs(a), abs(b)). See, for example: http://mathworld.wolfram.com/AbsoluteError.html http://mathworld.wolfram.com/RelativeError.html > my suggestion is: > (a-b)/b + log(b/a), which is nonnegative, zero only at equality, and > otherwise penalizes positive a for being different than some target > positive b. To me, it seems like guessing b using 1.9b is better than > guessing it as 0.1b, and so on. This corresponds to exponential KL > divergence, which has a clear statistical meaning, but only applies to > positive numbers. Do you have a reference or derivation for this? I'm happy to admit that I'm no Knuth or Kahan, but I've read a bit of numerical computing[1] and I've never seen anyone add a log term. I'm not even sure why you would do so. [1] I know just enough to know how much I don't know. -- Steve From steve at pearwood.info Tue Jan 20 11:55:10 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 20 Jan 2015 21:55:10 +1100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <54BDF1D1.70904@canterbury.ac.nz> References: <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> <54BC494E.3000609@canterbury.ac.nz> <CABicbJLdq519s-HDg5GM6Y1aeTbyhYrL-+fNqsxWbjn2B4C5nw@mail.gmail.com> <CAP7+vJKK5FwhtUU=3GG-OFL8hZmo3jPpvAV25wYE+CJUwkTQEA@mail.gmail.com> <54BDF1D1.70904@canterbury.ac.nz> Message-ID: <20150120105510.GV18556@ando.pearwood.info> On Tue, Jan 20, 2015 at 07:12:33PM +1300, Greg Ewing wrote: > But there are alternatives. We could borrow 'as' > from a certain other language: > > num_feathers as int = 0 Which language did you have in mind? Cobra uses that syntax: http://cobra-language.com/trac/cobra/wiki/Variables Even though Cobra is inspired by Python, I think it makes some bad choices, and this is one of them. Python uses "as" for name binding: try: import spam as ham with ham() as eggs: ... except SomeError as err: ... It would be inconsistent and confusing to introduce "as" for type-declarations. > Or exploit the fact that '{' following an identifier > is currently undefined: > > num_feathers {int} = 0 Now you're just making up random syntax :-) -- Steve From mistersheik at gmail.com Tue Jan 20 12:02:50 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 20 Jan 2015 03:02:50 -0800 (PST) Subject: [Python-ideas] What's going on with PEP 448 - Additional Unpacking Generalizations ? In-Reply-To: <CAA68w_n6PB_rMVdKxG5v2CcoDSJsSfC3+goCJ6mNFanhAUO7tA@mail.gmail.com> References: <f97552ab-bfb8-434b-bc68-5120d394682e@googlegroups.com> <CAPTjJmpGkgh0onpJVBYB4KXOi-n8HRDx2R55XJFteO=ty3FRVQ@mail.gmail.com> <CAP7+vJKKmwuWq+4Ps2drFuQA5ParxdPKe_XDuAFATiQrZ6TZMQ@mail.gmail.com> <43272cca-20ea-4337-bd2a-39bdf981048f@googlegroups.com> <CAPTjJmrjEm2GuRC=JNeEb7oRZ6xJAYPVHw9Fp0oAOWePPrRRbA@mail.gmail.com> <CAA68w_=_Gt1CsT_S4SHjK5fKKxSUHCt0J=4zSnjU5jMqQ6L=FA@mail.gmail.com> <CAPTjJmrUu72Z=UBa6ej7+zV34O-hco-XovYrZ77pG8O29nBnwQ@mail.gmail.com> <CAPTjJmrOCo5pzXCUe_nQ44X80FQMfb15QqiTx6AU14Wx7RDSwA@mail.gmail.com> <CAA68w_n6PB_rMVdKxG5v2CcoDSJsSfC3+goCJ6mNFanhAUO7tA@mail.gmail.com> Message-ID: <26294fbf-d883-410c-b7ec-cd82697ddd71@googlegroups.com> I've been working on the patch for PEP 448. Are the restrictions on the order of arguments in a function call partially or fully lifted? Currently, we can do: def f(a, b, c, d): pass f(1, d=4, *[1,2], 2, **{}) After PEP 448 can we do: f(1, d=4, *[2], c=3, **{}) ? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/b6bd9f65/attachment.html> From mistersheik at gmail.com Tue Jan 20 11:58:31 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 20 Jan 2015 05:58:31 -0500 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <20150120104012.GU18556@ando.pearwood.info> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> <m9k397$eo9$1@ger.gmane.org> <-3376261731312819627@unknownmsgid> <6ea13fe6-0831-4126-a94b-15b6bd1c83a7@googlegroups.com> <20150120104012.GU18556@ando.pearwood.info> Message-ID: <CAA68w_kH8SnoWyxDO4FefGakXaZ=VFadpNB6ufUuSKZd6PczFw@mail.gmail.com> On Tue, Jan 20, 2015 at 5:40 AM, Steven D'Aprano <steve at pearwood.info> wrote: > On Mon, Jan 19, 2015 at 08:10:35PM -0800, Neil Girdhar wrote: > > > If you decide to invent a relative error function, > > The error functions we have been talking about are hardly "invented". > They're mathematically simple and obvious, and can be found in just > about any undergraduate book on numerical computing: > > - The absolute error between two quantities a and b is the > absolute difference between them, abs(a-b). > > - The relative error is the difference relative to some > denominator d, abs(a-b)/abs(d), typically with d=a or d=b. > > If you happen to know that b is the correct value, then it is common to > choose b as the denominator. If you have no a priori reason to think > either a or b is correct, or if you prefer a symmetrical function, a > common choice is to use d = min(abs(a), abs(b)). > I'm not against this. What I meant by invention was that it appears that no one defines relative error this way that I could see, which is why I suggested you collect some examples. The mathworld article, numpy, and wikipedia all choose the asymmetric relative error, which is what I prefer as well. > See, for example: > > http://mathworld.wolfram.com/AbsoluteError.html > http://mathworld.wolfram.com/RelativeError.html > > > > my suggestion is: > > (a-b)/b + log(b/a), which is nonnegative, zero only at equality, and > > otherwise penalizes positive a for being different than some target > > positive b. To me, it seems like guessing b using 1.9b is better than > > guessing it as 0.1b, and so on. This corresponds to exponential KL > > divergence, which has a clear statistical meaning, but only applies to > > positive numbers. > > Do you have a reference or derivation for this? I'm happy to admit that > I'm no Knuth or Kahan, but I've read a bit of numerical computing[1] and > I've never seen anyone add a log term. I'm not even sure why you would > do so. > Suppose that you make a necessarily positive real measurement such as a length. Your minimum assumptive distribution ? the maximum entropy distribution that is concordant with your measurement ? is exponential with appropriate mean. Then, given a true belief distribution, you would like to know how similar your approximated belief is to your true belief. The number of bits of information required to correct your approximation is the KL divergence. However, I should not have proposed it. Since no one uses it and it behaves so badly around zero, I don't think it's a good idea for the main use of this function, which is testing. I think I'm sold on the relative error as it's normally defined, but I'd rather we see what all of the other packages have done and why. I don't think the default should be symmetrical since to me the asymmetrical case seems more common. How about: relative_error(true_value, approximate_value, absolute_tolerance=..., relative_tolerance=..., tolerance_relative_to=None) if tolerance_relative_to is None, replace it with true_value (or min(true, approx) if we go with the symmetrical definition). Best, Neil > > > [1] I know just enough to know how much I don't know. > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/VouL2DVqQmw/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/4a06829d/attachment.html> From steve at pearwood.info Tue Jan 20 12:06:27 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 20 Jan 2015 22:06:27 +1100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <54BDFCDC.9050802@canterbury.ac.nz> References: <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> <54BC494E.3000609@canterbury.ac.nz> <CABicbJLdq519s-HDg5GM6Y1aeTbyhYrL-+fNqsxWbjn2B4C5nw@mail.gmail.com> <5C091DF5-5ACB-479F-BB2D-302F2A60C3B0@gmail.com> <CANUJvPXa8MxvpSOxV_5BuV6n4=rhOGpjnahMrU_8RJ-OFOtq6Q@mail.gmail.com> <CAM1gar4mXURzN_KqNFj8b4h2nqpRCoNrnFj_9bgFeq4EfcdjFQ@mail.gmail.com> <54BDF438.5050504@canterbury.ac.nz> <CAM1gar44vrW656SjVet00xp1r-KT7WphMRM0pSJmd33nNH1UjA@mail.gmail.com> <54BDFCDC.9050802@canterbury.ac.nz> Message-ID: <20150120110627.GW18556@ando.pearwood.info> On Tue, Jan 20, 2015 at 07:59:40PM +1300, Greg Ewing wrote: > Stephen Hansen wrote: > >On Mon, Jan 19, 2015 at 10:22 PM, Greg Ewing > ><greg.ewing at canterbury.ac.nz <mailto:greg.ewing at canterbury.ac.nz>> wrote: > > > > quack :: lambda float, int: AnimalSound > > def quack(volume, num_times): > > ... > > > >That's still verbose, its simply vertically verbose. We have decorators > >for adding information in the vertical space. > > > > Then you have decorators and > >... whatever you call that line? What's the difference? > > The difference is that the :: annotations would *not* be > evaluated a run time. There's currently no way to get > that with a decorator, or anything else in the language, > short of abusing comments or string literals. > Yet it's what you really want for static type checking. Then I guess Python won't have static type checking. I think Guido has been very clear that he has little interest at this stage (if ever) about introducing a Java/Pascal/Haskell style statically typed compiler to Python. I seem to recall he expressed considerable skepticism that such a thing was even practical, or useful even if it could be done. I recall Guido expressing doubts that Nuitka's approach of trying to optimize code statically at compile-time will ever beat Cython's and PyPy's JIT approaches. So anyone thinking that this is the first step to turning Python into Haskell is probably mistaken :-) A type-checker that operates similarly to a linter is proven technology, thanks to mypy. So let's not waste too much time talking about syntax for some feature that Python will probably never get, and even if it does get it, it won't be in 3.5. How about we play around with type-checkers for a release or ten before proposing static typing for Python? -- Steve From mistersheik at gmail.com Tue Jan 20 12:08:15 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 20 Jan 2015 06:08:15 -0500 Subject: [Python-ideas] What's going on with PEP 448 - Additional Unpacking Generalizations ? In-Reply-To: <26294fbf-d883-410c-b7ec-cd82697ddd71@googlegroups.com> References: <f97552ab-bfb8-434b-bc68-5120d394682e@googlegroups.com> <CAPTjJmpGkgh0onpJVBYB4KXOi-n8HRDx2R55XJFteO=ty3FRVQ@mail.gmail.com> <CAP7+vJKKmwuWq+4Ps2drFuQA5ParxdPKe_XDuAFATiQrZ6TZMQ@mail.gmail.com> <43272cca-20ea-4337-bd2a-39bdf981048f@googlegroups.com> <CAPTjJmrjEm2GuRC=JNeEb7oRZ6xJAYPVHw9Fp0oAOWePPrRRbA@mail.gmail.com> <CAA68w_=_Gt1CsT_S4SHjK5fKKxSUHCt0J=4zSnjU5jMqQ6L=FA@mail.gmail.com> <CAPTjJmrUu72Z=UBa6ej7+zV34O-hco-XovYrZ77pG8O29nBnwQ@mail.gmail.com> <CAPTjJmrOCo5pzXCUe_nQ44X80FQMfb15QqiTx6AU14Wx7RDSwA@mail.gmail.com> <CAA68w_n6PB_rMVdKxG5v2CcoDSJsSfC3+goCJ6mNFanhAUO7tA@mail.gmail.com> <26294fbf-d883-410c-b7ec-cd82697ddd71@googlegroups.com> Message-ID: <CAA68w_m55rwQXHLXrw8sF7TecZq5i+Kjhv=PhK8Zwkn_6NU1Fg@mail.gmail.com> oops, I mean, currently we can do f(1, d=4, *[1,2], **{}) I'm asking about the others? On Tue, Jan 20, 2015 at 6:02 AM, Neil Girdhar <mistersheik at gmail.com> wrote: > I've been working on the patch for PEP 448. Are the restrictions on the > order of arguments in a function call partially or fully lifted? > > > Currently, we can do: > def f(a, b, c, d): > pass > > f(1, d=4, *[1,2], 2, **{}) > > After PEP 448 can we do: > > f(1, d=4, *[2], c=3, **{}) > > ? > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/J99EFY1D1nI/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/J99EFY1D1nI/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/1c50480d/attachment-0001.html> From mistersheik at gmail.com Tue Jan 20 12:26:01 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 20 Jan 2015 06:26:01 -0500 Subject: [Python-ideas] What's going on with PEP 448 - Additional Unpacking Generalizations ? In-Reply-To: <CAA68w_m55rwQXHLXrw8sF7TecZq5i+Kjhv=PhK8Zwkn_6NU1Fg@mail.gmail.com> References: <f97552ab-bfb8-434b-bc68-5120d394682e@googlegroups.com> <CAPTjJmpGkgh0onpJVBYB4KXOi-n8HRDx2R55XJFteO=ty3FRVQ@mail.gmail.com> <CAP7+vJKKmwuWq+4Ps2drFuQA5ParxdPKe_XDuAFATiQrZ6TZMQ@mail.gmail.com> <43272cca-20ea-4337-bd2a-39bdf981048f@googlegroups.com> <CAPTjJmrjEm2GuRC=JNeEb7oRZ6xJAYPVHw9Fp0oAOWePPrRRbA@mail.gmail.com> <CAA68w_=_Gt1CsT_S4SHjK5fKKxSUHCt0J=4zSnjU5jMqQ6L=FA@mail.gmail.com> <CAPTjJmrUu72Z=UBa6ej7+zV34O-hco-XovYrZ77pG8O29nBnwQ@mail.gmail.com> <CAPTjJmrOCo5pzXCUe_nQ44X80FQMfb15QqiTx6AU14Wx7RDSwA@mail.gmail.com> <CAA68w_n6PB_rMVdKxG5v2CcoDSJsSfC3+goCJ6mNFanhAUO7tA@mail.gmail.com> <26294fbf-d883-410c-b7ec-cd82697ddd71@googlegroups.com> <CAA68w_m55rwQXHLXrw8sF7TecZq5i+Kjhv=PhK8Zwkn_6NU1Fg@mail.gmail.com> Message-ID: <CAA68w_nLvm-Zb5mBhBy_v7GHPffqscNk2NW+Baab-pCcqKpOkg@mail.gmail.com> Also, if after PEP 448, you can do f(*a, *b, *c) and you can always do f(a, b, c, ) then why shouldn't you be able to do f(*a, *b, *c, ) ? On Tue, Jan 20, 2015 at 6:08 AM, Neil Girdhar <mistersheik at gmail.com> wrote: > oops, I mean, currently we can do > > f(1, d=4, *[1,2], **{}) > > I'm asking about the others? > > On Tue, Jan 20, 2015 at 6:02 AM, Neil Girdhar <mistersheik at gmail.com> > wrote: > >> I've been working on the patch for PEP 448. Are the restrictions on the >> order of arguments in a function call partially or fully lifted? >> >> >> Currently, we can do: >> def f(a, b, c, d): >> pass >> >> f(1, d=4, *[1,2], 2, **{}) >> >> After PEP 448 can we do: >> >> f(1, d=4, *[2], c=3, **{}) >> >> ? >> >> -- >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "python-ideas" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/python-ideas/J99EFY1D1nI/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> python-ideas+unsubscribe at googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> -- >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "python-ideas" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/python-ideas/J99EFY1D1nI/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> python-ideas+unsubscribe at googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/41b1ea69/attachment.html> From joshua at landau.ws Tue Jan 20 15:18:51 2015 From: joshua at landau.ws (Joshua Landau) Date: Tue, 20 Jan 2015 14:18:51 +0000 Subject: [Python-ideas] What's going on with PEP 448 - Additional Unpacking Generalizations ? In-Reply-To: <26294fbf-d883-410c-b7ec-cd82697ddd71@googlegroups.com> References: <f97552ab-bfb8-434b-bc68-5120d394682e@googlegroups.com> <CAPTjJmpGkgh0onpJVBYB4KXOi-n8HRDx2R55XJFteO=ty3FRVQ@mail.gmail.com> <CAP7+vJKKmwuWq+4Ps2drFuQA5ParxdPKe_XDuAFATiQrZ6TZMQ@mail.gmail.com> <43272cca-20ea-4337-bd2a-39bdf981048f@googlegroups.com> <CAPTjJmrjEm2GuRC=JNeEb7oRZ6xJAYPVHw9Fp0oAOWePPrRRbA@mail.gmail.com> <CAA68w_=_Gt1CsT_S4SHjK5fKKxSUHCt0J=4zSnjU5jMqQ6L=FA@mail.gmail.com> <CAPTjJmrUu72Z=UBa6ej7+zV34O-hco-XovYrZ77pG8O29nBnwQ@mail.gmail.com> <CAPTjJmrOCo5pzXCUe_nQ44X80FQMfb15QqiTx6AU14Wx7RDSwA@mail.gmail.com> <CAA68w_n6PB_rMVdKxG5v2CcoDSJsSfC3+goCJ6mNFanhAUO7tA@mail.gmail.com> <26294fbf-d883-410c-b7ec-cd82697ddd71@googlegroups.com> Message-ID: <CAN1F8qUzFUSdaKg_WP6d_2BqHGKY1iW_18M+9zPGEM=7kkF5jg@mail.gmail.com> On 20 January 2015 at 11:02, Neil Girdhar <mistersheik at gmail.com> wrote: > I've been working on the patch for PEP 448. Are the restrictions on the > order of arguments in a function call partially or fully lifted? I believe we settled on function( argument or *args, argument or *args, ..., kwargument or *args, kwargument or *args, ..., kwargument or **kwargs, kwargument or **kwargs, ... ) as the most likely candidate. Guido said > Regarding the decision about the allowable syntax for argument lists, > I prefer to keep the existing restriction See https://mail.python.org/pipermail/python-ideas/2013-July/021872.html for the discussion. > Currently, we can do: > def f(a, b, c, d): > pass > > f(1, d=4, *[1,2], 2, **{}) > > After PEP 448 can we do: > > f(1, d=4, *[2], c=3, **{}) > > ? These should be OK, if I've understood the sentiment. From joshua at landau.ws Tue Jan 20 15:22:13 2015 From: joshua at landau.ws (Joshua Landau) Date: Tue, 20 Jan 2015 14:22:13 +0000 Subject: [Python-ideas] What's going on with PEP 448 - Additional Unpacking Generalizations ? In-Reply-To: <CAN1F8qUzFUSdaKg_WP6d_2BqHGKY1iW_18M+9zPGEM=7kkF5jg@mail.gmail.com> References: <f97552ab-bfb8-434b-bc68-5120d394682e@googlegroups.com> <CAPTjJmpGkgh0onpJVBYB4KXOi-n8HRDx2R55XJFteO=ty3FRVQ@mail.gmail.com> <CAP7+vJKKmwuWq+4Ps2drFuQA5ParxdPKe_XDuAFATiQrZ6TZMQ@mail.gmail.com> <43272cca-20ea-4337-bd2a-39bdf981048f@googlegroups.com> <CAPTjJmrjEm2GuRC=JNeEb7oRZ6xJAYPVHw9Fp0oAOWePPrRRbA@mail.gmail.com> <CAA68w_=_Gt1CsT_S4SHjK5fKKxSUHCt0J=4zSnjU5jMqQ6L=FA@mail.gmail.com> <CAPTjJmrUu72Z=UBa6ej7+zV34O-hco-XovYrZ77pG8O29nBnwQ@mail.gmail.com> <CAPTjJmrOCo5pzXCUe_nQ44X80FQMfb15QqiTx6AU14Wx7RDSwA@mail.gmail.com> <CAA68w_n6PB_rMVdKxG5v2CcoDSJsSfC3+goCJ6mNFanhAUO7tA@mail.gmail.com> <26294fbf-d883-410c-b7ec-cd82697ddd71@googlegroups.com> <CAN1F8qUzFUSdaKg_WP6d_2BqHGKY1iW_18M+9zPGEM=7kkF5jg@mail.gmail.com> Message-ID: <CAN1F8qVLEsm6PxwfHAho7v0ktMmK+Ym3V78e+nFhar4j67LuJQ@mail.gmail.com> On 20 January 2015 at 14:18, Joshua Landau <joshua at landau.ws> wrote: > On 20 January 2015 at 11:02, Neil Girdhar <mistersheik at gmail.com> wrote: >> >> f(1, d=4, *[1,2], 2, **{}) > > These should be OK, if I've understood the sentiment. I meant, of course, to reply to your fixed version: >> f(1, d=4, *[1,2], **{}) wrt. >> Also, if after PEP 448, you can do >> >> f(*a, *b, *c) >> >> and you can always do >> >> f(a, >> b, >> c, >> ) >> >> then why shouldn't you be able to do >> >> f(*a, >> *b, >> *c, >> ) I see no reason this shouldn't work. On 20 January 2015 at 14:18, Joshua Landau <joshua at landau.ws> wrote: > On 20 January 2015 at 11:02, Neil Girdhar <mistersheik at gmail.com> wrote: >> I've been working on the patch for PEP 448. Are the restrictions on the >> order of arguments in a function call partially or fully lifted? > > I believe we settled on > > function( > argument or *args, argument or *args, ..., > kwargument or *args, kwargument or *args, ..., > kwargument or **kwargs, kwargument or **kwargs, ... > ) > > as the most likely candidate. > > Guido said >> Regarding the decision about the allowable syntax for argument lists, >> I prefer to keep the existing restriction > > See https://mail.python.org/pipermail/python-ideas/2013-July/021872.html > for the discussion. > >> Currently, we can do: >> def f(a, b, c, d): >> pass >> >> f(1, d=4, *[1,2], 2, **{}) >> >> After PEP 448 can we do: >> >> f(1, d=4, *[2], c=3, **{}) >> >> ? > > These should be OK, if I've understood the sentiment. From mistersheik at gmail.com Tue Jan 20 15:30:48 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 20 Jan 2015 09:30:48 -0500 Subject: [Python-ideas] What's going on with PEP 448 - Additional Unpacking Generalizations ? In-Reply-To: <CAN1F8qVLEsm6PxwfHAho7v0ktMmK+Ym3V78e+nFhar4j67LuJQ@mail.gmail.com> References: <f97552ab-bfb8-434b-bc68-5120d394682e@googlegroups.com> <CAPTjJmpGkgh0onpJVBYB4KXOi-n8HRDx2R55XJFteO=ty3FRVQ@mail.gmail.com> <CAP7+vJKKmwuWq+4Ps2drFuQA5ParxdPKe_XDuAFATiQrZ6TZMQ@mail.gmail.com> <43272cca-20ea-4337-bd2a-39bdf981048f@googlegroups.com> <CAPTjJmrjEm2GuRC=JNeEb7oRZ6xJAYPVHw9Fp0oAOWePPrRRbA@mail.gmail.com> <CAA68w_=_Gt1CsT_S4SHjK5fKKxSUHCt0J=4zSnjU5jMqQ6L=FA@mail.gmail.com> <CAPTjJmrUu72Z=UBa6ej7+zV34O-hco-XovYrZ77pG8O29nBnwQ@mail.gmail.com> <CAPTjJmrOCo5pzXCUe_nQ44X80FQMfb15QqiTx6AU14Wx7RDSwA@mail.gmail.com> <CAA68w_n6PB_rMVdKxG5v2CcoDSJsSfC3+goCJ6mNFanhAUO7tA@mail.gmail.com> <26294fbf-d883-410c-b7ec-cd82697ddd71@googlegroups.com> <CAN1F8qUzFUSdaKg_WP6d_2BqHGKY1iW_18M+9zPGEM=7kkF5jg@mail.gmail.com> <CAN1F8qVLEsm6PxwfHAho7v0ktMmK+Ym3V78e+nFhar4j67LuJQ@mail.gmail.com> Message-ID: <CAA68w_kTY5r6UCHZnYiBnA3ZbncJYA0eytysfo=d+RzgmTht+g@mail.gmail.com> Ok, the final comma means a small change to the grammar. In summary of the other rules: positional argument never follow keyword arguments iterable argument unpacking never follow keyword argument unpacking Neither rule is truly necessary, but ok for now. Best, Neil On Tue, Jan 20, 2015 at 9:22 AM, Joshua Landau <joshua at landau.ws> wrote: > On 20 January 2015 at 14:18, Joshua Landau <joshua at landau.ws> wrote: > > On 20 January 2015 at 11:02, Neil Girdhar <mistersheik at gmail.com> wrote: > >> > >> f(1, d=4, *[1,2], 2, **{}) > > > > These should be OK, if I've understood the sentiment. > > I meant, of course, to reply to your fixed version: > > >> f(1, d=4, *[1,2], **{}) > > wrt. > > >> Also, if after PEP 448, you can do > >> > >> f(*a, *b, *c) > >> > >> and you can always do > >> > >> f(a, > >> b, > >> c, > >> ) > >> > >> then why shouldn't you be able to do > >> > >> f(*a, > >> *b, > >> *c, > >> ) > > I see no reason this shouldn't work. > > On 20 January 2015 at 14:18, Joshua Landau <joshua at landau.ws> wrote: > > On 20 January 2015 at 11:02, Neil Girdhar <mistersheik at gmail.com> wrote: > >> I've been working on the patch for PEP 448. Are the restrictions on the > >> order of arguments in a function call partially or fully lifted? > > > > I believe we settled on > > > > function( > > argument or *args, argument or *args, ..., > > kwargument or *args, kwargument or *args, ..., > > kwargument or **kwargs, kwargument or **kwargs, ... > > ) > > > > as the most likely candidate. > > > > Guido said > >> Regarding the decision about the allowable syntax for argument lists, > >> I prefer to keep the existing restriction > > > > See https://mail.python.org/pipermail/python-ideas/2013-July/021872.html > > for the discussion. > > > >> Currently, we can do: > >> def f(a, b, c, d): > >> pass > >> > >> f(1, d=4, *[1,2], 2, **{}) > >> > >> After PEP 448 can we do: > >> > >> f(1, d=4, *[2], c=3, **{}) > >> > >> ? > > > > These should be OK, if I've understood the sentiment. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/J99EFY1D1nI/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/865bef81/attachment-0001.html> From chris.barker at noaa.gov Tue Jan 20 17:28:10 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 20 Jan 2015 08:28:10 -0800 Subject: [Python-ideas] Conventions for Python code documentation (was: PEP 484 (Type Hints) -- first draft round) In-Reply-To: <CAP7+vJL4PAZ31=QLBUdjQzLDiik02L5zXocm7CtdmEXem3dJBA@mail.gmail.com> References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> <85oapur4d8.fsf_-_@benfinney.id.au> <CAN-Kwu3Nj0=1+6B=uNmJz8D8LORfAAu62GLJ8_R5ZwwKjxUg=Q@mail.gmail.com> <CAP7+vJ+=3uebwL=+QOajwYVd=h+RaEFsRy8GqKQnDBFrjt7uog@mail.gmail.com> <CAN8d9gnN=MGNujsDm_5fqauQekrAOEPRP1jVquo0z6JVmyOokQ@mail.gmail.com> <3694529471710485200@unknownmsgid> <CAP7+vJL4PAZ31=QLBUdjQzLDiik02L5zXocm7CtdmEXem3dJBA@mail.gmail.com> Message-ID: <CALGmxE+-JAzRR0Y-7eeqGebemAD_hd6ZwwOo3Abv3WrRAu=hpg@mail.gmail.com> On Mon, Jan 19, 2015 at 8:42 PM, Guido van Rossum <guido at python.org> wrote: > On Mon, Jan 19, 2015 at 5:04 PM, Chris Barker - NOAA Federal < > chris.barker at noaa.gov> wrote: > >> I always teach that the @ syntax is a decoration, not a decorator, >> whereas a decorator is a function that takes a function and returns another >> function ( usually a customized version of the passed in function). This >> distinction between decorators and decoration syntax keeps the door open to >> do just about anything with decorations, but am I the only one that thinks >> it's a bad idea to have it be for "any old thing we want to hang off a >> function"? >> > > I think you're the only one who makes this distinction. > No, I'm not -- take a look at any number of tutorial sites, and answers for newbies on SO, etc. I used to teach decorators without making the distinction, and got a lot of confused students. maybe the distinction is only useful for pedagogical purposes, but I introduced it here 'cause it's a bit harder to talk about when a "decorator" can be either a function the behaves a certain way or the @something line in the code. > In common use "decorator" is used to describe both the syntax and the > function invoked by the syntax. "Decoration" is never (well, very rarely) > used. And calling any function that takes a function and returns one a > decorator feels overreaching -- I'd only call it a decorator if it is > intended to use with the decorator syntax. > sure -- I'm having a hard time coming up with the word s to describe what I"m talking about as "proper" decorator. The truth is, you can put any callable that takes at least one argument after that @. And I'm not suggesting that python should enforce anything in particular about that. What I am suggesting is that we should not officially suggest that you should use a decorator for any old thing that doesn't fit the usual sprit of decorators. A decorator that simply adds stuff to the docstring feels pretty ugly to me, particular if it goes beyond docs to specifying something important about the typing, etc. Hard to draw a line there, but I think we should keep the spirit of decorators in mind. But enough said. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/dc44f7b9/attachment.html> From guido at python.org Tue Jan 20 17:31:33 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 20 Jan 2015 08:31:33 -0800 Subject: [Python-ideas] Conventions for Python code documentation In-Reply-To: <m9kt66$a1n$1@ger.gmane.org> References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> <85oapur4d8.fsf_-_@benfinney.id.au> <CAN-Kwu3Nj0=1+6B=uNmJz8D8LORfAAu62GLJ8_R5ZwwKjxUg=Q@mail.gmail.com> <CAP7+vJ+=3uebwL=+QOajwYVd=h+RaEFsRy8GqKQnDBFrjt7uog@mail.gmail.com> <m9kt66$a1n$1@ger.gmane.org> Message-ID: <CAP7+vJLr7DjDst5o-Ws1rqnadT4Xpg+mekEgXr88XHvNVkjQbg@mail.gmail.com> On Mon, Jan 19, 2015 at 10:39 PM, Terry Reedy <tjreedy at udel.edu> wrote: > On 1/19/2015 2:54 PM, Guido van Rossum wrote: > >> Unfortunately PEP 257 falls short on specifying how to describe >> arguments -- it has only one example, it's not normative, and there's >> not much code that follows the example. The reST conventions are more >> common, but the stdlib itself rarely uses them. >> > > Because I have not seen such used, except maybe once, I have been meaning > to ask if I am allowed to use rst markup in stdlib docstrings? > I prefer you don't use them. We don't generate docs from the stdlib sources, and if someone generated them locally they would probably be doing themselves a disservice -- the docstrings exist for the benefit of the help() builtin, which AFAIK doesn't understand rst markup. > > It would be nice to come > >> up with a better convention that takes PEP 484 into account (so doc >> generators can incorporate the argument type annotations into the >> generated output, merged with per-argument from the docstring). >> > > My impression is that the plan is that stdlib .py files would not have > type annotations, but that they might be in stub files. Still true? > Correct. The stdlib is a special flower in so many ways... (Stubs are also useful for 3rd party libraries that haven't been updated yet.) > Would type hints be added to the signatures in .rst docs? I think you and Georg may have to decide what to do. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/76949dd1/attachment.html> From guido at python.org Tue Jan 20 17:38:47 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 20 Jan 2015 08:38:47 -0800 Subject: [Python-ideas] What's going on with PEP 448 - Additional Unpacking Generalizations ? In-Reply-To: <CAA68w_kTY5r6UCHZnYiBnA3ZbncJYA0eytysfo=d+RzgmTht+g@mail.gmail.com> References: <f97552ab-bfb8-434b-bc68-5120d394682e@googlegroups.com> <CAPTjJmpGkgh0onpJVBYB4KXOi-n8HRDx2R55XJFteO=ty3FRVQ@mail.gmail.com> <CAP7+vJKKmwuWq+4Ps2drFuQA5ParxdPKe_XDuAFATiQrZ6TZMQ@mail.gmail.com> <43272cca-20ea-4337-bd2a-39bdf981048f@googlegroups.com> <CAPTjJmrjEm2GuRC=JNeEb7oRZ6xJAYPVHw9Fp0oAOWePPrRRbA@mail.gmail.com> <CAA68w_=_Gt1CsT_S4SHjK5fKKxSUHCt0J=4zSnjU5jMqQ6L=FA@mail.gmail.com> <CAPTjJmrUu72Z=UBa6ej7+zV34O-hco-XovYrZ77pG8O29nBnwQ@mail.gmail.com> <CAPTjJmrOCo5pzXCUe_nQ44X80FQMfb15QqiTx6AU14Wx7RDSwA@mail.gmail.com> <CAA68w_n6PB_rMVdKxG5v2CcoDSJsSfC3+goCJ6mNFanhAUO7tA@mail.gmail.com> <26294fbf-d883-410c-b7ec-cd82697ddd71@googlegroups.com> <CAN1F8qUzFUSdaKg_WP6d_2BqHGKY1iW_18M+9zPGEM=7kkF5jg@mail.gmail.com> <CAN1F8qVLEsm6PxwfHAho7v0ktMmK+Ym3V78e+nFhar4j67LuJQ@mail.gmail.com> <CAA68w_kTY5r6UCHZnYiBnA3ZbncJYA0eytysfo=d+RzgmTht+g@mail.gmail.com> Message-ID: <CAP7+vJJ-D-xG9NPDo1TH33YFE8O_aB5hwf5FCdgQnSYzo6OTUw@mail.gmail.com> The PEP hasn't been accepted yet AFAIK... I'm generally okay with allowing multiple *x things (except in an *unpack* position of course) but I still don't think we should be mixing positional and keyword args. So, no f(a, b=2, c), nor f(a, b=2, *c). On Tue, Jan 20, 2015 at 6:30 AM, Neil Girdhar <mistersheik at gmail.com> wrote: > Ok, the final comma means a small change to the grammar. > > In summary of the other rules: > > positional argument never follow keyword arguments > iterable argument unpacking never follow keyword argument unpacking > > Neither rule is truly necessary, but ok for now. > > Best, > > Neil > > > > On Tue, Jan 20, 2015 at 9:22 AM, Joshua Landau <joshua at landau.ws> wrote: > >> On 20 January 2015 at 14:18, Joshua Landau <joshua at landau.ws> wrote: >> > On 20 January 2015 at 11:02, Neil Girdhar <mistersheik at gmail.com> >> wrote: >> >> >> >> f(1, d=4, *[1,2], 2, **{}) >> > >> > These should be OK, if I've understood the sentiment. >> >> I meant, of course, to reply to your fixed version: >> >> >> f(1, d=4, *[1,2], **{}) >> >> wrt. >> >> >> Also, if after PEP 448, you can do >> >> >> >> f(*a, *b, *c) >> >> >> >> and you can always do >> >> >> >> f(a, >> >> b, >> >> c, >> >> ) >> >> >> >> then why shouldn't you be able to do >> >> >> >> f(*a, >> >> *b, >> >> *c, >> >> ) >> >> I see no reason this shouldn't work. >> >> On 20 January 2015 at 14:18, Joshua Landau <joshua at landau.ws> wrote: >> > On 20 January 2015 at 11:02, Neil Girdhar <mistersheik at gmail.com> >> wrote: >> >> I've been working on the patch for PEP 448. Are the restrictions on >> the >> >> order of arguments in a function call partially or fully lifted? >> > >> > I believe we settled on >> > >> > function( >> > argument or *args, argument or *args, ..., >> > kwargument or *args, kwargument or *args, ..., >> > kwargument or **kwargs, kwargument or **kwargs, ... >> > ) >> > >> > as the most likely candidate. >> > >> > Guido said >> >> Regarding the decision about the allowable syntax for argument lists, >> >> I prefer to keep the existing restriction >> > >> > See >> https://mail.python.org/pipermail/python-ideas/2013-July/021872.html >> > for the discussion. >> > >> >> Currently, we can do: >> >> def f(a, b, c, d): >> >> pass >> >> >> >> f(1, d=4, *[1,2], 2, **{}) >> >> >> >> After PEP 448 can we do: >> >> >> >> f(1, d=4, *[2], c=3, **{}) >> >> >> >> ? >> > >> > These should be OK, if I've understood the sentiment. >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> -- >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "python-ideas" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/python-ideas/J99EFY1D1nI/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> python-ideas+unsubscribe at googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/f08fe862/attachment-0001.html> From chris.barker at noaa.gov Tue Jan 20 17:33:46 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 20 Jan 2015 08:33:46 -0800 Subject: [Python-ideas] Conventions for Python code documentation (was: PEP 484 (Type Hints) -- first draft round) In-Reply-To: <CAN8d9gnN=MGNujsDm_5fqauQekrAOEPRP1jVquo0z6JVmyOokQ@mail.gmail.com> References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> <85oapur4d8.fsf_-_@benfinney.id.au> <CAN-Kwu3Nj0=1+6B=uNmJz8D8LORfAAu62GLJ8_R5ZwwKjxUg=Q@mail.gmail.com> <CAP7+vJ+=3uebwL=+QOajwYVd=h+RaEFsRy8GqKQnDBFrjt7uog@mail.gmail.com> <CAN8d9gnN=MGNujsDm_5fqauQekrAOEPRP1jVquo0z6JVmyOokQ@mail.gmail.com> Message-ID: <CALGmxEJ58noM4t4kewXB3zWwNnEd96TDUzM4nevPj91vY5WarQ@mail.gmail.com> On Mon, Jan 19, 2015 at 2:14 PM, Philipp A. <flying-sheep at web.de> wrote: > I have am idea: how about including a @paramdoc decorator that accepts one > named string argument per parameter and appends information gathered via > type annotations and default values to the docstring? > > E.g. > > ~~~~ > @paramdoc( > main_course="Meaty because our society is medieval", > side_dish="Optional, meat is enough", > __return__="Objective opinion") > def eat(main_course: Meat, side_dish=None: Food) -> Opinion: > """ > Decides how well the meat and side dish taste together > """ > ... > aside from my thoughts about this being an odd use of the decorator syntax, I fail to see how this is better than simply putting the information in the docstring itself? You still are putting information in two places, making it easy for it to get out of sync, and I think it actually make sit harder to read -- that could end up being one really big decorator. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/a6f99ea1/attachment.html> From joshua at landau.ws Tue Jan 20 17:47:48 2015 From: joshua at landau.ws (Joshua Landau) Date: Tue, 20 Jan 2015 16:47:48 +0000 Subject: [Python-ideas] What's going on with PEP 448 - Additional Unpacking Generalizations ? In-Reply-To: <CAP7+vJJ-D-xG9NPDo1TH33YFE8O_aB5hwf5FCdgQnSYzo6OTUw@mail.gmail.com> References: <f97552ab-bfb8-434b-bc68-5120d394682e@googlegroups.com> <CAPTjJmpGkgh0onpJVBYB4KXOi-n8HRDx2R55XJFteO=ty3FRVQ@mail.gmail.com> <CAP7+vJKKmwuWq+4Ps2drFuQA5ParxdPKe_XDuAFATiQrZ6TZMQ@mail.gmail.com> <43272cca-20ea-4337-bd2a-39bdf981048f@googlegroups.com> <CAPTjJmrjEm2GuRC=JNeEb7oRZ6xJAYPVHw9Fp0oAOWePPrRRbA@mail.gmail.com> <CAA68w_=_Gt1CsT_S4SHjK5fKKxSUHCt0J=4zSnjU5jMqQ6L=FA@mail.gmail.com> <CAPTjJmrUu72Z=UBa6ej7+zV34O-hco-XovYrZ77pG8O29nBnwQ@mail.gmail.com> <CAPTjJmrOCo5pzXCUe_nQ44X80FQMfb15QqiTx6AU14Wx7RDSwA@mail.gmail.com> <CAA68w_n6PB_rMVdKxG5v2CcoDSJsSfC3+goCJ6mNFanhAUO7tA@mail.gmail.com> <26294fbf-d883-410c-b7ec-cd82697ddd71@googlegroups.com> <CAN1F8qUzFUSdaKg_WP6d_2BqHGKY1iW_18M+9zPGEM=7kkF5jg@mail.gmail.com> <CAN1F8qVLEsm6PxwfHAho7v0ktMmK+Ym3V78e+nFhar4j67LuJQ@mail.gmail.com> <CAA68w_kTY5r6UCHZnYiBnA3ZbncJYA0eytysfo=d+RzgmTht+g@mail.gmail.com> <CAP7+vJJ-D-xG9NPDo1TH33YFE8O_aB5hwf5FCdgQnSYzo6OTUw@mail.gmail.com> Message-ID: <CAN1F8qUXeCPeUknp863XEk=JNAtc+75N5B_r87CA6vy2JtjEzw@mail.gmail.com> On 20 January 2015 at 16:38, Guido van Rossum <guido at python.org> wrote: > The PEP hasn't been accepted yet AFAIK... I'm generally okay with allowing > multiple *x things (except in an *unpack* position of course) but I still > don't think we should be mixing positional and keyword args. So, no f(a, > b=2, c), nor f(a, b=2, *c). > f(a, b=2, *c) is currently legal as both a call and as a definition: a, *c = 1, 2, 3 def f(*args, **kwargs): print(args, kwargs) f(a, b=2, *c) #>>> (1, 2, 3) {'b': 2} def f(a, b=2, *c): print(a, b, c) f(1, 2, 3) #>>> 1 2 (3,) So I imagine that's staying (or, at least, this PEP isn't removing it). I don't think anyone is (yet) arguing for f(a, b=2, c). From mistersheik at gmail.com Tue Jan 20 17:51:08 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 20 Jan 2015 11:51:08 -0500 Subject: [Python-ideas] What's going on with PEP 448 - Additional Unpacking Generalizations ? In-Reply-To: <CAN1F8qUXeCPeUknp863XEk=JNAtc+75N5B_r87CA6vy2JtjEzw@mail.gmail.com> References: <f97552ab-bfb8-434b-bc68-5120d394682e@googlegroups.com> <CAPTjJmpGkgh0onpJVBYB4KXOi-n8HRDx2R55XJFteO=ty3FRVQ@mail.gmail.com> <CAP7+vJKKmwuWq+4Ps2drFuQA5ParxdPKe_XDuAFATiQrZ6TZMQ@mail.gmail.com> <43272cca-20ea-4337-bd2a-39bdf981048f@googlegroups.com> <CAPTjJmrjEm2GuRC=JNeEb7oRZ6xJAYPVHw9Fp0oAOWePPrRRbA@mail.gmail.com> <CAA68w_=_Gt1CsT_S4SHjK5fKKxSUHCt0J=4zSnjU5jMqQ6L=FA@mail.gmail.com> <CAPTjJmrUu72Z=UBa6ej7+zV34O-hco-XovYrZ77pG8O29nBnwQ@mail.gmail.com> <CAPTjJmrOCo5pzXCUe_nQ44X80FQMfb15QqiTx6AU14Wx7RDSwA@mail.gmail.com> <CAA68w_n6PB_rMVdKxG5v2CcoDSJsSfC3+goCJ6mNFanhAUO7tA@mail.gmail.com> <26294fbf-d883-410c-b7ec-cd82697ddd71@googlegroups.com> <CAN1F8qUzFUSdaKg_WP6d_2BqHGKY1iW_18M+9zPGEM=7kkF5jg@mail.gmail.com> <CAN1F8qVLEsm6PxwfHAho7v0ktMmK+Ym3V78e+nFhar4j67LuJQ@mail.gmail.com> <CAA68w_kTY5r6UCHZnYiBnA3ZbncJYA0eytysfo=d+RzgmTht+g@mail.gmail.com> <CAP7+vJJ-D-xG9NPDo1TH33YFE8O_aB5hwf5FCdgQnSYzo6OTUw@mail.gmail.com> <CAN1F8qUXeCPeUknp863XEk=JNAtc+75N5B_r87CA6vy2JtjEzw@mail.gmail.com> Message-ID: <CAA68w_m1Qo0dfQkqpoP=egHC8kpF18a8iT+5mJGm08gabqsXvg@mail.gmail.com> Okay, so: positional arguments neither follow keyword arguments nor keyword argument unpackings; iterable argument unpackings never follow keyword argument unpackings. On Tue, Jan 20, 2015 at 11:47 AM, Joshua Landau <joshua at landau.ws> wrote: > On 20 January 2015 at 16:38, Guido van Rossum <guido at python.org> wrote: > > The PEP hasn't been accepted yet AFAIK... I'm generally okay with > allowing > > multiple *x things (except in an *unpack* position of course) but I still > > don't think we should be mixing positional and keyword args. So, no f(a, > > b=2, c), nor f(a, b=2, *c). > > > > f(a, b=2, *c) is currently legal as both a call and as a definition: > > a, *c = 1, 2, 3 > > def f(*args, **kwargs): > print(args, kwargs) > > f(a, b=2, *c) > #>>> (1, 2, 3) {'b': 2} > > def f(a, b=2, *c): > print(a, b, c) > > f(1, 2, 3) > #>>> 1 2 (3,) > > So I imagine that's staying (or, at least, this PEP isn't removing > it). I don't think anyone is (yet) arguing for f(a, b=2, c). > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/J99EFY1D1nI/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/06a9b430/attachment.html> From guido at python.org Tue Jan 20 18:03:19 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 20 Jan 2015 09:03:19 -0800 Subject: [Python-ideas] What's going on with PEP 448 - Additional Unpacking Generalizations ? In-Reply-To: <CAA68w_m1Qo0dfQkqpoP=egHC8kpF18a8iT+5mJGm08gabqsXvg@mail.gmail.com> References: <f97552ab-bfb8-434b-bc68-5120d394682e@googlegroups.com> <CAPTjJmpGkgh0onpJVBYB4KXOi-n8HRDx2R55XJFteO=ty3FRVQ@mail.gmail.com> <CAP7+vJKKmwuWq+4Ps2drFuQA5ParxdPKe_XDuAFATiQrZ6TZMQ@mail.gmail.com> <43272cca-20ea-4337-bd2a-39bdf981048f@googlegroups.com> <CAPTjJmrjEm2GuRC=JNeEb7oRZ6xJAYPVHw9Fp0oAOWePPrRRbA@mail.gmail.com> <CAA68w_=_Gt1CsT_S4SHjK5fKKxSUHCt0J=4zSnjU5jMqQ6L=FA@mail.gmail.com> <CAPTjJmrUu72Z=UBa6ej7+zV34O-hco-XovYrZ77pG8O29nBnwQ@mail.gmail.com> <CAPTjJmrOCo5pzXCUe_nQ44X80FQMfb15QqiTx6AU14Wx7RDSwA@mail.gmail.com> <CAA68w_n6PB_rMVdKxG5v2CcoDSJsSfC3+goCJ6mNFanhAUO7tA@mail.gmail.com> <26294fbf-d883-410c-b7ec-cd82697ddd71@googlegroups.com> <CAN1F8qUzFUSdaKg_WP6d_2BqHGKY1iW_18M+9zPGEM=7kkF5jg@mail.gmail.com> <CAN1F8qVLEsm6PxwfHAho7v0ktMmK+Ym3V78e+nFhar4j67LuJQ@mail.gmail.com> <CAA68w_kTY5r6UCHZnYiBnA3ZbncJYA0eytysfo=d+RzgmTht+g@mail.gmail.com> <CAP7+vJJ-D-xG9NPDo1TH33YFE8O_aB5hwf5FCdgQnSYzo6OTUw@mail.gmail.com> <CAN1F8qUXeCPeUknp863XEk=JNAtc+75N5B_r87CA6vy2JtjEzw@mail.gmail.com> <CAA68w_m1Qo0dfQkqpoP=egHC8kpF18a8iT+5mJGm08gabqsXvg@mail.gmail.com> Message-ID: <CAP7+vJKBLN0jzQ-yr=qRL97B+1N+y819bW_nXbn+CLdJv4+2Dw@mail.gmail.com> OK. I don't like arg unpackings after keyword args, for the same reason plain positional args aren't allowed after keyword args, but I guess I didn't pay attention when it was introduced, so we're stuck with it now, it's not the end of the world, and at least the definition is clear (collect all positional args first, then handle keyword args). On Tue, Jan 20, 2015 at 8:51 AM, Neil Girdhar <mistersheik at gmail.com> wrote: > > > Okay, so: positional arguments neither follow keyword arguments nor > keyword argument unpackings; iterable argument unpackings never follow > keyword argument unpackings. > > > > On Tue, Jan 20, 2015 at 11:47 AM, Joshua Landau <joshua at landau.ws> wrote: > >> On 20 January 2015 at 16:38, Guido van Rossum <guido at python.org> wrote: >> > The PEP hasn't been accepted yet AFAIK... I'm generally okay with >> allowing >> > multiple *x things (except in an *unpack* position of course) but I >> still >> > don't think we should be mixing positional and keyword args. So, no f(a, >> > b=2, c), nor f(a, b=2, *c). >> > >> >> f(a, b=2, *c) is currently legal as both a call and as a definition: >> >> a, *c = 1, 2, 3 >> >> def f(*args, **kwargs): >> print(args, kwargs) >> >> f(a, b=2, *c) >> #>>> (1, 2, 3) {'b': 2} >> >> def f(a, b=2, *c): >> print(a, b, c) >> >> f(1, 2, 3) >> #>>> 1 2 (3,) >> >> So I imagine that's staying (or, at least, this PEP isn't removing >> it). I don't think anyone is (yet) arguing for f(a, b=2, c). >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> -- >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "python-ideas" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/python-ideas/J99EFY1D1nI/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> python-ideas+unsubscribe at googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/a1da97b6/attachment-0001.html> From chris.barker at noaa.gov Tue Jan 20 17:56:58 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 20 Jan 2015 08:56:58 -0800 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <c88ee7a0-b0b0-40f8-bb61-527d7b44664a@googlegroups.com> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <c88ee7a0-b0b0-40f8-bb61-527d7b44664a@googlegroups.com> Message-ID: <CALGmxEJcN09jDwvrC8v1XigfTmVUpkej8r5GnyFXULSuPb1OJQ@mail.gmail.com> On Mon, Jan 19, 2015 at 8:29 PM, Neil Girdhar <mistersheik at gmail.com> wrote: > Also for complex numbers, I think comparing the magnitude (distance from > the origin, or absolute value) of (x-y) to the size of x or y makes more > sense than calling is_close on the real and imaginary parts. What if the > real parts are much larger than the imaginary parts, e.g. x=1e5+1e-5j, > y=1e5-1e-5j. Do you think x and y are not close? > I've vacillated on this one. Personally I have no use case in mind, so really hard to know what's best. but your example case was exactly what I was thinking -- if the magnitude of one component is very different that the other, then I'm not sure if you would want it to swamp the answer, and requiring both components to be "close" would be the more conservative approach. On the other hand, there are cases where the exact result of a computation on complex numbers is, in fact, a pure real or imaginary number. In these cases, the computed result may be a large value in one component and a tiny value in the other, and you would test against the "actual" value, which would have a zero in there, so the relative error of that component by itself would never be "small". So yes, probably better to use the absolute value(s) to scale relative. But if anyone has some use-cases that would suggest the more strict approach, speak now. -Chris > Best, > > Neil > > > On Monday, January 19, 2015 at 1:33:44 AM UTC-5, Chris Barker wrote: >> >> OK folks, >> >> There has been a lot of chatter about this, which I think has served to >> provide some clarity, at least to me. However, I'm concerned that the >> upshot, at least for folks not deep into the discussion, will be: clearly >> there are too many use-case specific details to put any one thing in the >> std lib. But I still think we can provide something that is useful for most >> use-cases, and would like to propose what that is, and what the decision >> points are: >> >> A function for the math module, called somethign like "is_close", >> "approx_equal", etc. It will compute a relative tolerance, with a default >> maybe around 1-e12, with the user able to specify the tolerance they want. >> >> Optionally, the user can specify an "minimum absolute tolerance", it will >> default to zero, but can be set so that comparisons to zero can be handled >> gracefully. >> >> The relative tolerance will be computed from the smallest of the two >> input values, so as to get symmetry : is_close(a,b) == is_close(b,a). (this >> is the Boost "strong" definition, and what is used by Steven D'Aprano's >> code in the statistics test module) >> >> Alternatively, the relative error could be computed against a particular >> one of the input values (the second one?). This would be asymmetric, but be >> more clear exactly how "relative" is defined, and be closer to what people >> may expect when using it as a "actual vs expected" test. --- "expected" >> would be the scaling value. If the tolerance is small, it makes very little >> difference anyway, so I'm happy with whatever consensus moves us to. Note >> that if we go this way, then the parameter names should make it at least a >> little more clear -- maybe "actual" and "expected", rather than x and y or >> a and b or... and the function name should be something like is_close_to, >> rather than just is_close. >> >> It will be designed for floating point numbers, and handle inf, -inf, and >> NaN "properly". But is will also work with other numeric types, to the >> extent that duck typing "just works" (i.e. division and comparisons all >> work). >> >> complex numbers will be handled by: >> is_close(x.real, y.real) and is_close(x.imag, y.imag) >> (but i haven't written any code for that yet) >> >> It will not do a simple absolute comparison -- that is the job of a >> different function, or, better yet, folks just write it themselves: >> >> abs(x - y) <= delta >> >> really isn't much harder to write than a function call: >> >> absolute_diff(x,y,delta) >> >> Here is a gist with a sample implementation: >> >> https://gist.github.com/PythonCHB/6e9ef7732a9074d9337a >> >> I need to add more tests, and make the test proper unit tests, but it's a >> start. >> >> I also need to see how it does with other data types than float -- >> hopefully, it will "just work" with the core set. >> >> I hope we can come to some consensus that something like this is the way >> to go. >> >> -Chris >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Sun, Jan 18, 2015 at 11:27 AM, Ron Adam <ron... at gmail.com> wrote: >> >>> >>> >>> On 01/17/2015 11:37 PM, Chris Barker wrote: >>> >>>> (Someone claimed that 'nothing is close to zero'. This is >>>> nonsensical both in applied math and everyday life.) >>>> >>>> >>>> I'm pretty sure someone (more than one of use) asserted that "nothing is >>>> *relatively* close to zero -- very different. >>>> >>> >>> Yes, that is the case. >>> >>> >>> And I really wanted a way to have a default behavior that would do a >>>> reasonable transition to an absolute tolerance near zero, but I no >>>> longer >>>> thing that's possible. (numpy's implimentaion kind of does that, but it >>>> is >>>> really wrong for small numbers, and if you made the default >>>> min_tolerance >>>> the smallest possible representable number, it really wouldn't be >>>> useful. >>>> >>> >>> I'm going to try to summarise what I got out of this discussion. Maybe >>> it will help bring some focus to the topic. >>> >>> I think there are two case's to consider. >>> >>> # The most common case. >>> rel_is_good(actual, expected, delta) # value +- %delta. >>> >>> # Testing for possible equivalence? >>> rel_is_close(value1, value2, delta) # %delta close to each other. >>> >>> I don't think they are quite the same thing. >>> >>> rel_is_good(9, 10, .1) --> True >>> rel_is_good(10, 9, .1) --> False >>> >>> rel_is_close(9, 10, .1) --> True >>> rel_is_close(10, 9, .1) --> True >>> >>> >>> In the "is close" case, it shouldn't matter what order the arguments are >>> given. The delta is the distance from the larger number the smaller number >>> is. (of the same sign) >>> >>> So when calculating the relative error from two values, you want it to >>> be consistent with the rel_is_close function. >>> >>> rel_is_close(a, b, delta) <---> rel_err(a, b) <= delta >>> >>> And you should not use the rel_err function in the rel_is_good function. >>> >>> >>> >>> The next issue is, where does the numeric accuracy of the data, >>> significant digits, and the languages accuracy (ULPs), come into the >>> picture. >>> >>> My intuition.. I need to test the idea to make a firmer claim.. is that >>> in the case of is_good, you want to exclude the uncertain parts, but with >>> is_close, you want to include the uncertain parts. >>> >>> Two values "are close" if you can't tell one from the other with >>> certainty. The is_close range includes any uncertainty. >>> >>> A value is good if it's within a range with certainty. And this >>> excludes any uncertainty. >>> >>> This is where taking in consideration of an absolute delta comes in. The >>> minimum range for both is the uncertainty of the data. But is_close and >>> is_good do different things with it. >>> >>> Of course all of this only applies if you agree with these definitions >>> of is_close, and is_good. ;) >>> >>> Cheers, >>> Ron >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python... at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.... at noaa.gov >> > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/905c7451/attachment.html> From joshua at landau.ws Tue Jan 20 18:12:45 2015 From: joshua at landau.ws (Joshua Landau) Date: Tue, 20 Jan 2015 17:12:45 +0000 Subject: [Python-ideas] What's going on with PEP 448 - Additional Unpacking Generalizations ? In-Reply-To: <CAA68w_m1Qo0dfQkqpoP=egHC8kpF18a8iT+5mJGm08gabqsXvg@mail.gmail.com> References: <f97552ab-bfb8-434b-bc68-5120d394682e@googlegroups.com> <CAPTjJmpGkgh0onpJVBYB4KXOi-n8HRDx2R55XJFteO=ty3FRVQ@mail.gmail.com> <CAP7+vJKKmwuWq+4Ps2drFuQA5ParxdPKe_XDuAFATiQrZ6TZMQ@mail.gmail.com> <43272cca-20ea-4337-bd2a-39bdf981048f@googlegroups.com> <CAPTjJmrjEm2GuRC=JNeEb7oRZ6xJAYPVHw9Fp0oAOWePPrRRbA@mail.gmail.com> <CAA68w_=_Gt1CsT_S4SHjK5fKKxSUHCt0J=4zSnjU5jMqQ6L=FA@mail.gmail.com> <CAPTjJmrUu72Z=UBa6ej7+zV34O-hco-XovYrZ77pG8O29nBnwQ@mail.gmail.com> <CAPTjJmrOCo5pzXCUe_nQ44X80FQMfb15QqiTx6AU14Wx7RDSwA@mail.gmail.com> <CAA68w_n6PB_rMVdKxG5v2CcoDSJsSfC3+goCJ6mNFanhAUO7tA@mail.gmail.com> <26294fbf-d883-410c-b7ec-cd82697ddd71@googlegroups.com> <CAN1F8qUzFUSdaKg_WP6d_2BqHGKY1iW_18M+9zPGEM=7kkF5jg@mail.gmail.com> <CAN1F8qVLEsm6PxwfHAho7v0ktMmK+Ym3V78e+nFhar4j67LuJQ@mail.gmail.com> <CAA68w_kTY5r6UCHZnYiBnA3ZbncJYA0eytysfo=d+RzgmTht+g@mail.gmail.com> <CAP7+vJJ-D-xG9NPDo1TH33YFE8O_aB5hwf5FCdgQnSYzo6OTUw@mail.gmail.com> <CAN1F8qUXeCPeUknp863XEk=JNAtc+75N5B_r87CA6vy2JtjEzw@mail.gmail.com> <CAA68w_m1Qo0dfQkqpoP=egHC8kpF18a8iT+5mJGm08gabqsXvg@mail.gmail.com> Message-ID: <CAN1F8qWKtJ0o1ZZoR_MJuD=z_vTZAY9fA1xOC_fyfr2XchvV5Q@mail.gmail.com> On 20 January 2015 at 16:51, Neil Girdhar <mistersheik at gmail.com> wrote: > > Okay, so: positional arguments neither follow keyword arguments nor keyword > argument unpackings; iterable argument unpackings never follow keyword > argument unpackings. Yep. :) Incidentally, this confusion was why I originally suggested the more relaxed rules. It's probably not a problem for the end-user though, because nobody does it. It would probably be fine from an implementation perspective, too, now that I know how ast_for_call works. From leewangzhong+python at gmail.com Tue Jan 20 18:15:44 2015 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Tue, 20 Jan 2015 12:15:44 -0500 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <818732810.4093303.1420793208359.JavaMail.yahoo@jws100159.mail.ne1.yahoo.com> References: <CADiSq7c0hFqm-XE7nx+r_RBLHmR-_zVinoDD6-9ja54M-UGo2w@mail.gmail.com> <818732810.4093303.1420793208359.JavaMail.yahoo@jws100159.mail.ne1.yahoo.com> Message-ID: <CAB_e7izPftcRKF1w-CVhrax=aMYTT1X-DGNR2==s_bYY=yfnCg@mail.gmail.com> On Fri, Jan 9, 2015 at 3:46 AM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: > > I believe the distinction isn't just there to keep the parser simpler, or because Guido wasn't clever enough to figure out how to embed statements in expressions in an indentation-significant language, or because the examples of Ruby and CoffeeScript didn't exist yet and now it's too late for Python; statements are a big part of what makes Python code more readable than those languages, both in-depth and in a quick scan to see the general flow control and state changes. > > But it's hard to explain why. And, even if I manage to explain why _I_ think statements make Python more readable, that's just my opinion; it would be nice to have something that Guido endorsed. > > Also, for people who want to suggest changes to Python (or design their own languages), it would be helpful to understand how the distinction fits in with other things. For example, in a pure functional language, there may be no benefit to having statements. In a language where (almost) everything is an expression, having mutating methods return self instead of None wouldn't be nearly as much of a problem. More generally, expression-ness and not-mutating are deeply connected (e.g., comprehensions aren't designed for mutation, loop statements are), but it's not easy to say how. And so on. Sorry to be late. Will this Guido post suffice as a substitute? http://www.artima.com/weblogs/viewpost.jsp?thread=147358 Yawar might also need this, if he chooses to pursue this path. https://wiki.python.org/moin/AlternateLambdaSyntax From mistersheik at gmail.com Tue Jan 20 18:16:16 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 20 Jan 2015 12:16:16 -0500 Subject: [Python-ideas] What's going on with PEP 448 - Additional Unpacking Generalizations ? In-Reply-To: <CAN1F8qWKtJ0o1ZZoR_MJuD=z_vTZAY9fA1xOC_fyfr2XchvV5Q@mail.gmail.com> References: <f97552ab-bfb8-434b-bc68-5120d394682e@googlegroups.com> <CAPTjJmpGkgh0onpJVBYB4KXOi-n8HRDx2R55XJFteO=ty3FRVQ@mail.gmail.com> <CAP7+vJKKmwuWq+4Ps2drFuQA5ParxdPKe_XDuAFATiQrZ6TZMQ@mail.gmail.com> <43272cca-20ea-4337-bd2a-39bdf981048f@googlegroups.com> <CAPTjJmrjEm2GuRC=JNeEb7oRZ6xJAYPVHw9Fp0oAOWePPrRRbA@mail.gmail.com> <CAA68w_=_Gt1CsT_S4SHjK5fKKxSUHCt0J=4zSnjU5jMqQ6L=FA@mail.gmail.com> <CAPTjJmrUu72Z=UBa6ej7+zV34O-hco-XovYrZ77pG8O29nBnwQ@mail.gmail.com> <CAPTjJmrOCo5pzXCUe_nQ44X80FQMfb15QqiTx6AU14Wx7RDSwA@mail.gmail.com> <CAA68w_n6PB_rMVdKxG5v2CcoDSJsSfC3+goCJ6mNFanhAUO7tA@mail.gmail.com> <26294fbf-d883-410c-b7ec-cd82697ddd71@googlegroups.com> <CAN1F8qUzFUSdaKg_WP6d_2BqHGKY1iW_18M+9zPGEM=7kkF5jg@mail.gmail.com> <CAN1F8qVLEsm6PxwfHAho7v0ktMmK+Ym3V78e+nFhar4j67LuJQ@mail.gmail.com> <CAA68w_kTY5r6UCHZnYiBnA3ZbncJYA0eytysfo=d+RzgmTht+g@mail.gmail.com> <CAP7+vJJ-D-xG9NPDo1TH33YFE8O_aB5hwf5FCdgQnSYzo6OTUw@mail.gmail.com> <CAN1F8qUXeCPeUknp863XEk=JNAtc+75N5B_r87CA6vy2JtjEzw@mail.gmail.com> <CAA68w_m1Qo0dfQkqpoP=egHC8kpF18a8iT+5mJGm08gabqsXvg@mail.gmail.com> <CAN1F8qWKtJ0o1ZZoR_MJuD=z_vTZAY9fA1xOC_fyfr2XchvV5Q@mail.gmail.com> Message-ID: <CAA68w_njMH27nB5aD21gMtb_==4cqatkfYbGUjOSfnQq7iLNHg@mail.gmail.com> Exactly? On Tue, Jan 20, 2015 at 12:12 PM, Joshua Landau <joshua at landau.ws> wrote: > On 20 January 2015 at 16:51, Neil Girdhar <mistersheik at gmail.com> wrote: > > > > Okay, so: positional arguments neither follow keyword arguments nor > keyword > > argument unpackings; iterable argument unpackings never follow keyword > > argument unpackings. > > Yep. :) > > Incidentally, this confusion was why I originally suggested the more > relaxed rules. It's probably not a problem for the end-user though, > because nobody does it. It would probably be fine from an > implementation perspective, too, now that I know how ast_for_call > works. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/J99EFY1D1nI/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/c773bfeb/attachment.html> From chris.barker at noaa.gov Tue Jan 20 18:11:33 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 20 Jan 2015 09:11:33 -0800 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <m9ktsi$k5c$1@ger.gmane.org> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <87lhkzw6mo.fsf@uwakimon.sk.tsukuba.ac.jp> <CALGmxEJAvRQHxzddrnH9noeRhQswEfJsN2o0qOz7-xv0bb8ATQ@mail.gmail.com> <m9ktsi$k5c$1@ger.gmane.org> Message-ID: <CALGmxEKfjLkYRE4_KGVQ1TQn96k10GFPw+ThT_YHe1bNRXzyFw@mail.gmail.com> On Mon, Jan 19, 2015 at 10:50 PM, Terry Reedy <tjreedy at udel.edu> wrote: > If the function only handles 'relatively close', I would be happier with > 'relative_close' or 'rel_close' than 'is close' since the latter tends to > imply to me absolute closeness. > yeah, though I think Ron has persuaded me that we can (and proaly need to, for the near-zero case) have it do both relative and absolute.... The proposed doc (and docstring) should mention absolute tolerance and > 'abs(a-b) < tol' as an alternative, expecially near or at 0. absolutely -- I think the near-zero case is where we need to put absolute in there, and once it's there, why not use it for other use-cases? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/5e216091/attachment.html> From chris.barker at noaa.gov Tue Jan 20 18:23:57 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 20 Jan 2015 09:23:57 -0800 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <CAA68w_kH8SnoWyxDO4FefGakXaZ=VFadpNB6ufUuSKZd6PczFw@mail.gmail.com> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> <m9k397$eo9$1@ger.gmane.org> <-3376261731312819627@unknownmsgid> <6ea13fe6-0831-4126-a94b-15b6bd1c83a7@googlegroups.com> <20150120104012.GU18556@ando.pearwood.info> <CAA68w_kH8SnoWyxDO4FefGakXaZ=VFadpNB6ufUuSKZd6PczFw@mail.gmail.com> Message-ID: <CALGmxELCNFC5OPzAdDgJ5VMw==HTOWpWcF9bO3s2V7RfhxjCsw@mail.gmail.com> On Tue, Jan 20, 2015 at 2:58 AM, Neil Girdhar <mistersheik at gmail.com> wrote: > I'm not against this. What I meant by invention was that it appears that > no one defines relative error this way that I could see, which is why I > suggested you collect some examples. > Did you look at the Boost docs? That's pretty much what they do, and it was apparently derived from Kahan -- can't get a better pedigree than that ;-). To some extent, whether it's symmetric or not depends on what you name it. to me, at least "is_close" implies symmetry -- a is close to be , but b is not close to a? pretty weird. but if you call it is_close_to, then I could see that a is close to be, but b may not be close to a (though that still seems pretty odd). ANyone got a better name? Again, I'm pretty sure that the most common use cases (and the one where folks are likely to use a stdlib function, rather than write their own) are where the tolerance is approximate anyway -- i.e. ten decimal digits or so (or 1e-10) -- these distinctions are irrelevant in that case, but the symmetric versions may make for fewer surprises. The mathworld article, numpy, and wikipedia all choose the asymmetric > relative error, which is what I prefer as well. > See, for example: >> >> http://mathworld.wolfram.com/AbsoluteError.html >> http://mathworld.wolfram.com/RelativeError.html > > thanks for the links. I think I'm sold on the relative error as it's normally defined, but I'd >> rather we see what all of the other packages have done and why. I don't >> think the default should be symmetrical since to me the asymmetrical case >> seems more common. >> > > How about: > > relative_error(true_value, approximate_value, absolute_tolerance=..., > relative_tolerance=..., tolerance_relative_to=None) > Interesting -- numpy, for instance puts the "true value" (the one used to scale) second in the parameter list... one reason that I like symmetric. But using clear names like that does make it more clear that it isn't symmetric. > if tolerance_relative_to is None, replace it with true_value (or min(true, > approx) if we go with the symmetrical definition). > hmm -- I'm a bit wary of too many options, but if we can't decide on the "best" way, then that may be the way to go -- as long as the defaults are reasonable. [1] I know just enough to know how much I don't know. > > me too -- I _barely_ passed a class with Kahan, which really did teach me far more about what I don't know than anything else! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/2ba7db71/attachment-0001.html> From guido at python.org Tue Jan 20 18:25:05 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 20 Jan 2015 09:25:05 -0800 Subject: [Python-ideas] Conventions for Python code documentation (was: PEP 484 (Type Hints) -- first draft round) In-Reply-To: <CALGmxE+-JAzRR0Y-7eeqGebemAD_hd6ZwwOo3Abv3WrRAu=hpg@mail.gmail.com> References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> <85oapur4d8.fsf_-_@benfinney.id.au> <CAN-Kwu3Nj0=1+6B=uNmJz8D8LORfAAu62GLJ8_R5ZwwKjxUg=Q@mail.gmail.com> <CAP7+vJ+=3uebwL=+QOajwYVd=h+RaEFsRy8GqKQnDBFrjt7uog@mail.gmail.com> <CAN8d9gnN=MGNujsDm_5fqauQekrAOEPRP1jVquo0z6JVmyOokQ@mail.gmail.com> <3694529471710485200@unknownmsgid> <CAP7+vJL4PAZ31=QLBUdjQzLDiik02L5zXocm7CtdmEXem3dJBA@mail.gmail.com> <CALGmxE+-JAzRR0Y-7eeqGebemAD_hd6ZwwOo3Abv3WrRAu=hpg@mail.gmail.com> Message-ID: <CAP7+vJKZyUtg-88duCtAJhSqFwjJOL2YznTe+OR+=cf5njmKjg@mail.gmail.com> On Tue, Jan 20, 2015 at 8:28 AM, Chris Barker <chris.barker at noaa.gov> wrote: > On Mon, Jan 19, 2015 at 8:42 PM, Guido van Rossum <guido at python.org> > wrote: > >> On Mon, Jan 19, 2015 at 5:04 PM, Chris Barker - NOAA Federal < >> chris.barker at noaa.gov> wrote: >> >>> I always teach that the @ syntax is a decoration, not a decorator, >>> whereas a decorator is a function that takes a function and returns another >>> function ( usually a customized version of the passed in function). This >>> distinction between decorators and decoration syntax keeps the door open to >>> do just about anything with decorations, but am I the only one that thinks >>> it's a bad idea to have it be for "any old thing we want to hang off a >>> function"? >>> >> >> I think you're the only one who makes this distinction. >> > > No, I'm not -- take a look at any number of tutorial sites, and answers > for newbies on SO, etc. I used to teach decorators without making the > distinction, and got a lot of confused students. maybe the distinction is > only useful for pedagogical purposes, but I introduced it here 'cause it's > a bit harder to talk about when a "decorator" can be either a function the > behaves a certain way or the @something line in the code. > It's a fair cop. > In common use "decorator" is used to describe both the syntax and the >> function invoked by the syntax. "Decoration" is never (well, very rarely) >> used. And calling any function that takes a function and returns one a >> decorator feels overreaching -- I'd only call it a decorator if it is >> intended to use with the decorator syntax. >> > > sure -- I'm having a hard time coming up with the word s to describe what > I"m talking about as "proper" decorator. The truth is, you can put any > callable that takes at least one argument after that @. And I'm not > suggesting that python should enforce anything in particular about that. > What I am suggesting is that we should not officially suggest that you > should use a decorator for any old thing that doesn't fit the usual sprit > of decorators. > Agreed. The 99% case is that a decorator returns a function that (with some qualifications) calls the decorated function. > A decorator that simply adds stuff to the docstring feels pretty ugly to > me, particular if it goes beyond docs to specifying something important > about the typing, etc. > This sounds totally fine to me, honestly. > > Hard to draw a line there, but I think we should keep the spirit of > decorators in mind. > > But enough said. > Amen. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/7919e9d5/attachment.html> From chris.barker at noaa.gov Tue Jan 20 18:32:04 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 20 Jan 2015 09:32:04 -0800 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <6ea13fe6-0831-4126-a94b-15b6bd1c83a7@googlegroups.com> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> <m9k397$eo9$1@ger.gmane.org> <-3376261731312819627@unknownmsgid> <6ea13fe6-0831-4126-a94b-15b6bd1c83a7@googlegroups.com> Message-ID: <CALGmxE+_CJf_Siwm8-WcdPcCcUZF7sxrtEHONjVEMi0eVoXiqA@mail.gmail.com> On Mon, Jan 19, 2015 at 8:10 PM, Neil Girdhar <mistersheik at gmail.com> wrote: > If you decide to make a PEP, please list the other algorithms you found > and their definitions. > sure -- care to contribute ;-) > Personally, I'm for being consistent with numpy and defining math.isclose > similar to numpy.isclose for consistency alone. > This is a good question for the larger, non-numpy group. I'm personally a heavy numpy user, and don't see any reason to be consistent -- and I (and others on this list) have some real issues with the numpy approach. But we shouldn't call it isclose() if it's not the same algorithm... -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/220a2e33/attachment.html> From mistersheik at gmail.com Tue Jan 20 18:39:54 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 20 Jan 2015 12:39:54 -0500 Subject: [Python-ideas] What's going on with PEP 448 - Additional Unpacking Generalizations ? In-Reply-To: <CAP7+vJKBLN0jzQ-yr=qRL97B+1N+y819bW_nXbn+CLdJv4+2Dw@mail.gmail.com> References: <f97552ab-bfb8-434b-bc68-5120d394682e@googlegroups.com> <CAPTjJmpGkgh0onpJVBYB4KXOi-n8HRDx2R55XJFteO=ty3FRVQ@mail.gmail.com> <CAP7+vJKKmwuWq+4Ps2drFuQA5ParxdPKe_XDuAFATiQrZ6TZMQ@mail.gmail.com> <43272cca-20ea-4337-bd2a-39bdf981048f@googlegroups.com> <CAPTjJmrjEm2GuRC=JNeEb7oRZ6xJAYPVHw9Fp0oAOWePPrRRbA@mail.gmail.com> <CAA68w_=_Gt1CsT_S4SHjK5fKKxSUHCt0J=4zSnjU5jMqQ6L=FA@mail.gmail.com> <CAPTjJmrUu72Z=UBa6ej7+zV34O-hco-XovYrZ77pG8O29nBnwQ@mail.gmail.com> <CAPTjJmrOCo5pzXCUe_nQ44X80FQMfb15QqiTx6AU14Wx7RDSwA@mail.gmail.com> <CAA68w_n6PB_rMVdKxG5v2CcoDSJsSfC3+goCJ6mNFanhAUO7tA@mail.gmail.com> <26294fbf-d883-410c-b7ec-cd82697ddd71@googlegroups.com> <CAN1F8qUzFUSdaKg_WP6d_2BqHGKY1iW_18M+9zPGEM=7kkF5jg@mail.gmail.com> <CAN1F8qVLEsm6PxwfHAho7v0ktMmK+Ym3V78e+nFhar4j67LuJQ@mail.gmail.com> <CAA68w_kTY5r6UCHZnYiBnA3ZbncJYA0eytysfo=d+RzgmTht+g@mail.gmail.com> <CAP7+vJJ-D-xG9NPDo1TH33YFE8O_aB5hwf5FCdgQnSYzo6OTUw@mail.gmail.com> <CAN1F8qUXeCPeUknp863XEk=JNAtc+75N5B_r87CA6vy2JtjEzw@mail.gmail.com> <CAA68w_m1Qo0dfQkqpoP=egHC8kpF18a8iT+5mJGm08gabqsXvg@mail.gmail.com> <CAP7+vJKBLN0jzQ-yr=qRL97B+1N+y819bW_nXbn+CLdJv4+2Dw@mail.gmail.com> Message-ID: <CAA68w_=MY+CmqpZ5Ni=FwLFpjAGhnJ_yrZu+wmAW+fS74dD1Gg@mail.gmail.com> I believe I have finally finished the work on the patch for PEP 448 ( http://bugs.python.org/issue2292). How do we get the PEP approved? What else would we need to check it into Python? Best, Neil On Tue, Jan 20, 2015 at 12:03 PM, Guido van Rossum <guido at python.org> wrote: > OK. I don't like arg unpackings after keyword args, for the same reason > plain positional args aren't allowed after keyword args, but I guess I > didn't pay attention when it was introduced, so we're stuck with it now, > it's not the end of the world, and at least the definition is clear > (collect all positional args first, then handle keyword args). > > On Tue, Jan 20, 2015 at 8:51 AM, Neil Girdhar <mistersheik at gmail.com> > wrote: > >> >> >> Okay, so: positional arguments neither follow keyword arguments nor >> keyword argument unpackings; iterable argument unpackings never follow >> keyword argument unpackings. >> >> >> >> On Tue, Jan 20, 2015 at 11:47 AM, Joshua Landau <joshua at landau.ws> wrote: >> >>> On 20 January 2015 at 16:38, Guido van Rossum <guido at python.org> wrote: >>> > The PEP hasn't been accepted yet AFAIK... I'm generally okay with >>> allowing >>> > multiple *x things (except in an *unpack* position of course) but I >>> still >>> > don't think we should be mixing positional and keyword args. So, no >>> f(a, >>> > b=2, c), nor f(a, b=2, *c). >>> > >>> >>> f(a, b=2, *c) is currently legal as both a call and as a definition: >>> >>> a, *c = 1, 2, 3 >>> >>> def f(*args, **kwargs): >>> print(args, kwargs) >>> >>> f(a, b=2, *c) >>> #>>> (1, 2, 3) {'b': 2} >>> >>> def f(a, b=2, *c): >>> print(a, b, c) >>> >>> f(1, 2, 3) >>> #>>> 1 2 (3,) >>> >>> So I imagine that's staying (or, at least, this PEP isn't removing >>> it). I don't think anyone is (yet) arguing for f(a, b=2, c). >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >>> -- >>> >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "python-ideas" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/python-ideas/J99EFY1D1nI/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> python-ideas+unsubscribe at googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/34dee3aa/attachment-0001.html> From guido at python.org Tue Jan 20 18:48:40 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 20 Jan 2015 09:48:40 -0800 Subject: [Python-ideas] What's going on with PEP 448 - Additional Unpacking Generalizations ? In-Reply-To: <CAA68w_=MY+CmqpZ5Ni=FwLFpjAGhnJ_yrZu+wmAW+fS74dD1Gg@mail.gmail.com> References: <f97552ab-bfb8-434b-bc68-5120d394682e@googlegroups.com> <CAPTjJmpGkgh0onpJVBYB4KXOi-n8HRDx2R55XJFteO=ty3FRVQ@mail.gmail.com> <CAP7+vJKKmwuWq+4Ps2drFuQA5ParxdPKe_XDuAFATiQrZ6TZMQ@mail.gmail.com> <43272cca-20ea-4337-bd2a-39bdf981048f@googlegroups.com> <CAPTjJmrjEm2GuRC=JNeEb7oRZ6xJAYPVHw9Fp0oAOWePPrRRbA@mail.gmail.com> <CAA68w_=_Gt1CsT_S4SHjK5fKKxSUHCt0J=4zSnjU5jMqQ6L=FA@mail.gmail.com> <CAPTjJmrUu72Z=UBa6ej7+zV34O-hco-XovYrZ77pG8O29nBnwQ@mail.gmail.com> <CAPTjJmrOCo5pzXCUe_nQ44X80FQMfb15QqiTx6AU14Wx7RDSwA@mail.gmail.com> <CAA68w_n6PB_rMVdKxG5v2CcoDSJsSfC3+goCJ6mNFanhAUO7tA@mail.gmail.com> <26294fbf-d883-410c-b7ec-cd82697ddd71@googlegroups.com> <CAN1F8qUzFUSdaKg_WP6d_2BqHGKY1iW_18M+9zPGEM=7kkF5jg@mail.gmail.com> <CAN1F8qVLEsm6PxwfHAho7v0ktMmK+Ym3V78e+nFhar4j67LuJQ@mail.gmail.com> <CAA68w_kTY5r6UCHZnYiBnA3ZbncJYA0eytysfo=d+RzgmTht+g@mail.gmail.com> <CAP7+vJJ-D-xG9NPDo1TH33YFE8O_aB5hwf5FCdgQnSYzo6OTUw@mail.gmail.com> <CAN1F8qUXeCPeUknp863XEk=JNAtc+75N5B_r87CA6vy2JtjEzw@mail.gmail.com> <CAA68w_m1Qo0dfQkqpoP=egHC8kpF18a8iT+5mJGm08gabqsXvg@mail.gmail.com> <CAP7+vJKBLN0jzQ-yr=qRL97B+1N+y819bW_nXbn+CLdJv4+2Dw@mail.gmail.com> <CAA68w_=MY+CmqpZ5Ni=FwLFpjAGhnJ_yrZu+wmAW+fS74dD1Gg@mail.gmail.com> Message-ID: <CAP7+vJJszvJJ-zNQ5T7rk9+yJ2J4G_2j7TPE-3Re0ABXYe9Xmw@mail.gmail.com> Thanks very much for your work! I am CC'ing python-dev to see if there are any last calls for PEP 448. Assuming no material objection appear to the new syntax and semantics, I can approve the PEP later this week. To get it committed, you need one of the active committers to give you a code review (waiting for me would mean waiting forever). Maybe Antoine, Benjamin or Victor? On Tue, Jan 20, 2015 at 9:39 AM, Neil Girdhar <mistersheik at gmail.com> wrote: > I believe I have finally finished the work on the patch for PEP 448 ( > http://bugs.python.org/issue2292). How do we get the PEP approved? What > else would we need to check it into Python? > > Best, > > Neil > > On Tue, Jan 20, 2015 at 12:03 PM, Guido van Rossum <guido at python.org> > wrote: > >> OK. I don't like arg unpackings after keyword args, for the same reason >> plain positional args aren't allowed after keyword args, but I guess I >> didn't pay attention when it was introduced, so we're stuck with it now, >> it's not the end of the world, and at least the definition is clear >> (collect all positional args first, then handle keyword args). >> >> On Tue, Jan 20, 2015 at 8:51 AM, Neil Girdhar <mistersheik at gmail.com> >> wrote: >> >>> >>> >>> Okay, so: positional arguments neither follow keyword arguments nor >>> keyword argument unpackings; iterable argument unpackings never follow >>> keyword argument unpackings. >>> >>> >>> >>> On Tue, Jan 20, 2015 at 11:47 AM, Joshua Landau <joshua at landau.ws> >>> wrote: >>> >>>> On 20 January 2015 at 16:38, Guido van Rossum <guido at python.org> wrote: >>>> > The PEP hasn't been accepted yet AFAIK... I'm generally okay with >>>> allowing >>>> > multiple *x things (except in an *unpack* position of course) but I >>>> still >>>> > don't think we should be mixing positional and keyword args. So, no >>>> f(a, >>>> > b=2, c), nor f(a, b=2, *c). >>>> > >>>> >>>> f(a, b=2, *c) is currently legal as both a call and as a definition: >>>> >>>> a, *c = 1, 2, 3 >>>> >>>> def f(*args, **kwargs): >>>> print(args, kwargs) >>>> >>>> f(a, b=2, *c) >>>> #>>> (1, 2, 3) {'b': 2} >>>> >>>> def f(a, b=2, *c): >>>> print(a, b, c) >>>> >>>> f(1, 2, 3) >>>> #>>> 1 2 (3,) >>>> >>>> So I imagine that's staying (or, at least, this PEP isn't removing >>>> it). I don't think anyone is (yet) arguing for f(a, b=2, c). >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> >>>> -- >>>> >>>> --- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "python-ideas" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/python-ideas/J99EFY1D1nI/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> python-ideas+unsubscribe at googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/7efd296e/attachment.html> From toddrjen at gmail.com Tue Jan 20 19:38:16 2015 From: toddrjen at gmail.com (Todd) Date: Tue, 20 Jan 2015 19:38:16 +0100 Subject: [Python-ideas] Conventions for Python code documentation In-Reply-To: <CAP7+vJLr7DjDst5o-Ws1rqnadT4Xpg+mekEgXr88XHvNVkjQbg@mail.gmail.com> References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> <85oapur4d8.fsf_-_@benfinney.id.au> <CAN-Kwu3Nj0=1+6B=uNmJz8D8LORfAAu62GLJ8_R5ZwwKjxUg=Q@mail.gmail.com> <CAP7+vJ+=3uebwL=+QOajwYVd=h+RaEFsRy8GqKQnDBFrjt7uog@mail.gmail.com> <m9kt66$a1n$1@ger.gmane.org> <CAP7+vJLr7DjDst5o-Ws1rqnadT4Xpg+mekEgXr88XHvNVkjQbg@mail.gmail.com> Message-ID: <CAFpSVp+iWmzOiybm9oKC-F4KnhoTz+TaSUCvNr=9SMqQjFOetg@mail.gmail.com> On Jan 20, 2015 5:32 PM, "Guido van Rossum" <guido at python.org> wrote: > > On Mon, Jan 19, 2015 at 10:39 PM, Terry Reedy <tjreedy at udel.edu> wrote: >> >> On 1/19/2015 2:54 PM, Guido van Rossum wrote: >>> >>> Unfortunately PEP 257 falls short on specifying how to describe >>> arguments -- it has only one example, it's not normative, and there's >>> not much code that follows the example. The reST conventions are more >>> common, but the stdlib itself rarely uses them. >> >> >> Because I have not seen such used, except maybe once, I have been meaning to ask if I am allowed to use rst markup in stdlib docstrings? > > > I prefer you don't use them. We don't generate docs from the stdlib sources, and if someone generated them locally they would probably be doing themselves a disservice -- the docstrings exist for the benefit of the help() builtin, which AFAIK doesn't understand rst markup. > Might the lack of rst support in help() be something to address either as part of this effort or in parallel with it? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/3b30e368/attachment.html> From guido at python.org Tue Jan 20 19:43:37 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 20 Jan 2015 10:43:37 -0800 Subject: [Python-ideas] Conventions for Python code documentation In-Reply-To: <CAFpSVp+iWmzOiybm9oKC-F4KnhoTz+TaSUCvNr=9SMqQjFOetg@mail.gmail.com> References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> <85oapur4d8.fsf_-_@benfinney.id.au> <CAN-Kwu3Nj0=1+6B=uNmJz8D8LORfAAu62GLJ8_R5ZwwKjxUg=Q@mail.gmail.com> <CAP7+vJ+=3uebwL=+QOajwYVd=h+RaEFsRy8GqKQnDBFrjt7uog@mail.gmail.com> <m9kt66$a1n$1@ger.gmane.org> <CAP7+vJLr7DjDst5o-Ws1rqnadT4Xpg+mekEgXr88XHvNVkjQbg@mail.gmail.com> <CAFpSVp+iWmzOiybm9oKC-F4KnhoTz+TaSUCvNr=9SMqQjFOetg@mail.gmail.com> Message-ID: <CAP7+vJ+jN_1Z3yeyW+wCF0FvecLxtWDEhR4nSmPiKm3oyp7NuQ@mail.gmail.com> IIUC there is no rst support in the stdlib, and including docutils sounds like a bad idea (it's a huge unmaintained codebase). On Tue, Jan 20, 2015 at 10:38 AM, Todd <toddrjen at gmail.com> wrote: > > On Jan 20, 2015 5:32 PM, "Guido van Rossum" <guido at python.org> wrote: > > > > On Mon, Jan 19, 2015 at 10:39 PM, Terry Reedy <tjreedy at udel.edu> wrote: > >> > >> On 1/19/2015 2:54 PM, Guido van Rossum wrote: > >>> > >>> Unfortunately PEP 257 falls short on specifying how to describe > >>> arguments -- it has only one example, it's not normative, and there's > >>> not much code that follows the example. The reST conventions are more > >>> common, but the stdlib itself rarely uses them. > >> > >> > >> Because I have not seen such used, except maybe once, I have been > meaning to ask if I am allowed to use rst markup in stdlib docstrings? > > > > > > I prefer you don't use them. We don't generate docs from the stdlib > sources, and if someone generated them locally they would probably be doing > themselves a disservice -- the docstrings exist for the benefit of the > help() builtin, which AFAIK doesn't understand rst markup. > > > > Might the lack of rst support in help() be something to address either as > part of this effort or in parallel with it? > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/a7f402fd/attachment-0001.html> From njs at pobox.com Tue Jan 20 20:03:49 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 20 Jan 2015 19:03:49 +0000 Subject: [Python-ideas] Conventions for Python code documentation In-Reply-To: <m9kt66$a1n$1@ger.gmane.org> References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> <85oapur4d8.fsf_-_@benfinney.id.au> <CAN-Kwu3Nj0=1+6B=uNmJz8D8LORfAAu62GLJ8_R5ZwwKjxUg=Q@mail.gmail.com> <CAP7+vJ+=3uebwL=+QOajwYVd=h+RaEFsRy8GqKQnDBFrjt7uog@mail.gmail.com> <m9kt66$a1n$1@ger.gmane.org> Message-ID: <CAPJVwBkhodrC7gc9tQAqgio421KCHvJR-mQ7F0r7i4NfynCwyw@mail.gmail.com> On Tue, Jan 20, 2015 at 6:39 AM, Terry Reedy <tjreedy at udel.edu> wrote: > On 1/19/2015 2:54 PM, Guido van Rossum wrote: >> >> Unfortunately PEP 257 falls short on specifying how to describe >> arguments -- it has only one example, it's not normative, and there's >> not much code that follows the example. The reST conventions are more >> common, but the stdlib itself rarely uses them. > > > Because I have not seen such used, except maybe once, I have been meaning to > ask if I am allowed to use rst markup in stdlib docstrings? If you want a standard, then the Google and/or Numpy docstring standards are generally better than raw RST anyway: they're readable by humans -- no-one will necessarily even notice you're using a special format, esp. the Google standard -- and sphinx can parse them too in case that ever becomes important. https://google-styleguide.googlecode.com/svn/trunk/pyguide.html#Comments <- then click the little arrow https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt#sections -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From abarnert at yahoo.com Tue Jan 20 23:57:46 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 20 Jan 2015 14:57:46 -0800 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <CAB_e7izPftcRKF1w-CVhrax=aMYTT1X-DGNR2==s_bYY=yfnCg@mail.gmail.com> References: <CADiSq7c0hFqm-XE7nx+r_RBLHmR-_zVinoDD6-9ja54M-UGo2w@mail.gmail.com> <818732810.4093303.1420793208359.JavaMail.yahoo@jws100159.mail.ne1.yahoo.com> <CAB_e7izPftcRKF1w-CVhrax=aMYTT1X-DGNR2==s_bYY=yfnCg@mail.gmail.com> Message-ID: <3B6C4C96-F318-4582-A4CD-10DF221A4F1D@yahoo.com> On Jan 20, 2015, at 9:15, "Franklin? Lee" <leewangzhong+python at gmail.com> wrote: > On Fri, Jan 9, 2015 at 3:46 AM, Andrew Barnert > <abarnert at yahoo.com.dmarc.invalid> wrote: >> >> I believe the distinction isn't just there to keep the parser simpler, or because Guido wasn't clever enough to figure out how to embed statements in expressions in an indentation-significant language, or because the examples of Ruby and CoffeeScript didn't exist yet and now it's too late for Python; statements are a big part of what makes Python code more readable than those languages, both in-depth and in a quick scan to see the general flow control and state changes. >> >> But it's hard to explain why. And, even if I manage to explain why _I_ think statements make Python more readable, that's just my opinion; it would be nice to have something that Guido endorsed. >> >> Also, for people who want to suggest changes to Python (or design their own languages), it would be helpful to understand how the distinction fits in with other things. For example, in a pure functional language, there may be no benefit to having statements. In a language where (almost) everything is an expression, having mutating methods return self instead of None wouldn't be nearly as much of a problem. More generally, expression-ness and not-mutating are deeply connected (e.g., comprehensions aren't designed for mutation, loop statements are), but it's not easy to say how. And so on. > > Sorry to be late. > > Will this Guido post suffice as a substitute? > http://www.artima.com/weblogs/viewpost.jsp?thread=147358 That's not what I was looking for--but it is a great counter to my dismissal of "just to keep the parser simpler". Guido is of course right. I wasn't thinking about the fact that statement parsing and expression parsing are different modes, with state, and it's a lot harder for a human mind to subconsciously keep a stack of stateful modes in short-term memory than for a program. But I still think Python gains more from the inescapable statement-expression than this. Guido's answer is sufficient in itself, but there's more there even if it weren't true. I suppose the way to answer that would be to come up with a language where complex expressions are also indentation-sensitive (and there's no paren or backslash continuation) and show that if you get rid of the stack of stateful modes, there really is no other problem putting statements inside expressions. That's kind of the opposite of the way people usually try to tackle this problem, but if Guido's right that the usual way is inherently bound to fail... > Yawar might also need this, if he chooses to pursue this path. > https://wiki.python.org/moin/AlternateLambdaSyntax From jeanpierreda at gmail.com Tue Jan 20 23:57:48 2015 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Tue, 20 Jan 2015 14:57:48 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <20150120110627.GW18556@ando.pearwood.info> References: <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> <54BC494E.3000609@canterbury.ac.nz> <CABicbJLdq519s-HDg5GM6Y1aeTbyhYrL-+fNqsxWbjn2B4C5nw@mail.gmail.com> <5C091DF5-5ACB-479F-BB2D-302F2A60C3B0@gmail.com> <CANUJvPXa8MxvpSOxV_5BuV6n4=rhOGpjnahMrU_8RJ-OFOtq6Q@mail.gmail.com> <CAM1gar4mXURzN_KqNFj8b4h2nqpRCoNrnFj_9bgFeq4EfcdjFQ@mail.gmail.com> <54BDF438.5050504@canterbury.ac.nz> <CAM1gar44vrW656SjVet00xp1r-KT7WphMRM0pSJmd33nNH1UjA@mail.gmail.com> <54BDFCDC.9050802@canterbury.ac.nz> <20150120110627.GW18556@ando.pearwood.info> Message-ID: <CABicbJ++udqOoQB-ZnBx9yczzzTUopDWFbPZGZwKX=uEsRi-dQ@mail.gmail.com> On Tue, Jan 20, 2015 at 3:06 AM, Steven D'Aprano <steve at pearwood.info> wrote: > On Tue, Jan 20, 2015 at 07:59:40PM +1300, Greg Ewing wrote: >> Stephen Hansen wrote: >> > Then you have decorators and >> >... whatever you call that line? What's the difference? >> >> The difference is that the :: annotations would *not* be >> evaluated a run time. There's currently no way to get >> that with a decorator, or anything else in the language, >> short of abusing comments or string literals. >> Yet it's what you really want for static type checking. > > Then I guess Python won't have static type checking. > > I think Guido has been very clear that he has little interest at this > stage (if ever) about introducing a Java/Pascal/Haskell style statically > typed compiler to Python. Fortunately, Greg wasn't suggesting that. You seem to be misunderstanding the phrase "static type checker". It means something that checks types, statically. Like mypy. Greg's complaints make sense in that context (and I agree with him.) -- Devin From guido at python.org Wed Jan 21 00:21:50 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 20 Jan 2015 15:21:50 -0800 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <3B6C4C96-F318-4582-A4CD-10DF221A4F1D@yahoo.com> References: <CADiSq7c0hFqm-XE7nx+r_RBLHmR-_zVinoDD6-9ja54M-UGo2w@mail.gmail.com> <818732810.4093303.1420793208359.JavaMail.yahoo@jws100159.mail.ne1.yahoo.com> <CAB_e7izPftcRKF1w-CVhrax=aMYTT1X-DGNR2==s_bYY=yfnCg@mail.gmail.com> <3B6C4C96-F318-4582-A4CD-10DF221A4F1D@yahoo.com> Message-ID: <CAP7+vJKd19Yb5yeVPt38sXHTKVjfPh_JO4PX_dngR50Vck2dNQ@mail.gmail.com> On Tue, Jan 20, 2015 at 2:57 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > On Jan 20, 2015, at 9:15, "Franklin? Lee" <leewangzhong+python at gmail.com> > wrote: > > > On Fri, Jan 9, 2015 at 3:46 AM, Andrew Barnert > > <abarnert at yahoo.com.dmarc.invalid> wrote: > >> > >> I believe the distinction isn't just there to keep the parser simpler, > or because Guido wasn't clever enough to figure out how to embed statements > in expressions in an indentation-significant language, or because the > examples of Ruby and CoffeeScript didn't exist yet and now it's too late > for Python; statements are a big part of what makes Python code more > readable than those languages, both in-depth and in a quick scan to see the > general flow control and state changes. > >> > >> But it's hard to explain why. And, even if I manage to explain why _I_ > think statements make Python more readable, that's just my opinion; it > would be nice to have something that Guido endorsed. > >> > >> Also, for people who want to suggest changes to Python (or design their > own languages), it would be helpful to understand how the distinction fits > in with other things. For example, in a pure functional language, there may > be no benefit to having statements. In a language where (almost) everything > is an expression, having mutating methods return self instead of None > wouldn't be nearly as much of a problem. More generally, expression-ness > and not-mutating are deeply connected (e.g., comprehensions aren't designed > for mutation, loop statements are), but it's not easy to say how. And so on. > > > > Sorry to be late. > > > > Will this Guido post suffice as a substitute? > > http://www.artima.com/weblogs/viewpost.jsp?thread=147358 > > That's not what I was looking for--but it is a great counter to my > dismissal of "just to keep the parser simpler". Guido is of course right. I > wasn't thinking about the fact that statement parsing and expression > parsing are different modes, with state, and it's a lot harder for a human > mind to subconsciously keep a stack of stateful modes in short-term memory > than for a program. > > But I still think Python gains more from the inescapable > statement-expression than this. Guido's answer is sufficient in itself, but > there's more there even if it weren't true. > > I suppose the way to answer that would be to come up with a language where > complex expressions are also indentation-sensitive (and there's no paren or > backslash continuation) and show that if you get rid of the stack of > stateful modes, there really is no other problem putting statements inside > expressions. That's kind of the opposite of the way people usually try to > tackle this problem, but if Guido's right that the usual way is inherently > bound to fail... > I've lost track which side you are now arguing for, but one example of a language that takes indentation in complex expressions to the extreme is Coffeescript. Having had to reverse-engineer a fairly complex piece of Coffeescript recently, I definitely thing they went too far. It seems there is no formal grammar for Coffeescript, just a translator, and even if there were a formal grammar, it would have many odd corners and sharp edges. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/53eb638f/attachment-0001.html> From abarnert at yahoo.com Wed Jan 21 00:23:41 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 20 Jan 2015 15:23:41 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CABicbJ++udqOoQB-ZnBx9yczzzTUopDWFbPZGZwKX=uEsRi-dQ@mail.gmail.com> References: <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> <54BC494E.3000609@canterbury.ac.nz> <CABicbJLdq519s-HDg5GM6Y1aeTbyhYrL-+fNqsxWbjn2B4C5nw@mail.gmail.com> <5C091DF5-5ACB-479F-BB2D-302F2A60C3B0@gmail.com> <CANUJvPXa8MxvpSOxV_5BuV6n4=rhOGpjnahMrU_8RJ-OFOtq6Q@mail.gmail.com> <CAM1gar4mXURzN_KqNFj8b4h2nqpRCoNrnFj_9bgFeq4EfcdjFQ@mail.gmail.com> <54BDF438.5050504@canterbury.ac.nz> <CAM1gar44vrW656SjVet00xp1r-KT7WphMRM0pSJmd33nNH1UjA@mail.gmail.com> <54BDFCDC.9050802@canterbury.ac.nz> <20150120110627.GW18556@ando.pearwood.info> <CABicbJ++udqOoQB-ZnBx9yczzzTUopDWFbPZGZwKX=uEsRi-dQ@mail.gmail.com> Message-ID: <B07BB060-5626-4A20-9C94-E698F3B59D51@yahoo.com> On Jan 20, 2015, at 14:57, Devin Jeanpierre <jeanpierreda at gmail.com> wrote: > On Tue, Jan 20, 2015 at 3:06 AM, Steven D'Aprano <steve at pearwood.info> wrote: >> On Tue, Jan 20, 2015 at 07:59:40PM +1300, Greg Ewing wrote: >>> Stephen Hansen wrote: >>>> Then you have decorators and >>>> ... whatever you call that line? What's the difference? >>> >>> The difference is that the :: annotations would *not* be >>> evaluated a run time. There's currently no way to get >>> that with a decorator, or anything else in the language, >>> short of abusing comments or string literals. >>> Yet it's what you really want for static type checking. >> >> Then I guess Python won't have static type checking. >> >> I think Guido has been very clear that he has little interest at this >> stage (if ever) about introducing a Java/Pascal/Haskell style statically >> typed compiler to Python. > > Fortunately, Greg wasn't suggesting that. You seem to be > misunderstanding the phrase "static type checker". It means something > that checks types, statically. Like mypy. Greg's complaints make sense > in that context (and I agree with him.) As I understand it (and please correct me if I'm wrong), Greg is making a very simple point: If type annotations are only for static type checking, as done by something like MyPy, they, by definition, have no use at runtime. But annotations are about storing information with functions at runtime. So the proposal is inherently storing useless information at runtime. (And it's also preventing anyone else from storing useful information there, but that's not the main issue.) But I think there's a hole in the premise. Python's key strength is good debug-ability and amazingly powerful introspection. (I mean Python's _two_ key strengths are...) Information that has no runtime meaning to the interpreter may still have meaning to the user who's debugging code, or interactively exploring it. The fact that inspect.signature returns the parameter and return types is useful information to that user, even if there really is nothing you can do with it programmatically. You could ask why not just always use strings for the annotations. After all, MyPy obviously already has to be able to handle that, and to a human debugging the code and printing out the annotation there's no real difference. But the reason there is pretty obvious: the quotes are extra characters that get in the way of reading the function definition, and 90% of the time they're unnecessary. Think of being able to use a compile-time expression instead of an equivalent string as syntactic sugar--not necessary, but still nice. From ethan at stoneleaf.us Wed Jan 21 00:27:03 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 20 Jan 2015 15:27:03 -0800 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <CALGmxE+_CJf_Siwm8-WcdPcCcUZF7sxrtEHONjVEMi0eVoXiqA@mail.gmail.com> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> <m9k397$eo9$1@ger.gmane.org> <-3376261731312819627@unknownmsgid> <6ea13fe6-0831-4126-a94b-15b6bd1c83a7@googlegroups.com> <CALGmxE+_CJf_Siwm8-WcdPcCcUZF7sxrtEHONjVEMi0eVoXiqA@mail.gmail.com> Message-ID: <54BEE447.7010001@stoneleaf.us> On 01/20/2015 09:32 AM, Chris Barker wrote: > On Mon, Jan 19, 2015 at 8:10 PM, Neil Girdhar wrote: > >> If you decide to make a PEP, please list the other algorithms you found and their definitions. > > sure -- care to contribute ;-) >> Personally, I'm for being consistent with numpy and defining math.isclose similar to numpy.isclose for consistency >> alone. > > This is a good question for the larger, non-numpy group. I'm personally a heavy numpy user, and don't see any reason to > be consistent -- and I (and others on this list) have some real issues with the numpy approach. But we shouldn't call it > isclose() if it's not the same algorithm... I'm not a numpy nor heavy maths user, and see no reason why we should try to match numpy -- if you want/need numpy, get numpy. There is also no reason to not use the same name, even if it is not the same algorithm -- if the answer we're looking for is "is it close?" then is_close is a fine name (or isclose, similar to the other is... functions and methods Python already has). I think for simplicity' sake, a symmetric function would be better. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/307c2df6/attachment.sig> From eltoder at gmail.com Wed Jan 21 01:20:17 2015 From: eltoder at gmail.com (Eugene Toder) Date: Tue, 20 Jan 2015 19:20:17 -0500 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <B07BB060-5626-4A20-9C94-E698F3B59D51@yahoo.com> References: <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> <54BC494E.3000609@canterbury.ac.nz> <CABicbJLdq519s-HDg5GM6Y1aeTbyhYrL-+fNqsxWbjn2B4C5nw@mail.gmail.com> <5C091DF5-5ACB-479F-BB2D-302F2A60C3B0@gmail.com> <CANUJvPXa8MxvpSOxV_5BuV6n4=rhOGpjnahMrU_8RJ-OFOtq6Q@mail.gmail.com> <CAM1gar4mXURzN_KqNFj8b4h2nqpRCoNrnFj_9bgFeq4EfcdjFQ@mail.gmail.com> <54BDF438.5050504@canterbury.ac.nz> <CAM1gar44vrW656SjVet00xp1r-KT7WphMRM0pSJmd33nNH1UjA@mail.gmail.com> <54BDFCDC.9050802@canterbury.ac.nz> <20150120110627.GW18556@ando.pearwood.info> <CABicbJ++udqOoQB-ZnBx9yczzzTUopDWFbPZGZwKX=uEsRi-dQ@mail.gmail.com> <B07BB060-5626-4A20-9C94-E698F3B59D51@yahoo.com> Message-ID: <CA+KNMzmdL+yhc_7_9NAk70VaGxofqFTZgL1wttFzCP3GhyN-sg@mail.gmail.com> On Tue, Jan 20, 2015 at 6:23 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > As I understand it (and please correct me if I'm wrong), Greg is making a > very simple point: > > If type annotations are only for static type checking, as done by > something like MyPy, they, by definition, have no use at runtime. But > annotations are about storing information with functions at runtime. So the > proposal is inherently storing useless information at runtime. (And it's > also preventing anyone else from storing useful information there, but > that's not the main issue.) > I believe the argument is slightly different. The main issue is not with storing the annotation, but with evaluating it. This evaluation does not help a static type checker in any way, but introduces problems with forward references, and limits the syntax. For example, a type checker could support the common and natural "T1|T2" syntax for the Union type if the interpreter did not attempt to evaluate it. The current solution is to abandon some syntax, and put other things in quotes, which is a bit ugly. The second issue is that the annotations are only supported on the function arguments and return value. There are other things that are useful to annotate, most obviously class attributes and local variables. The current solution is to put these annotations into comments, which is not pretty too. I think these are both good points. They can be addressed by introducing an entirely new syntax, like suggested above, or by changing and extending how the current syntax works. E.g. one can imagine that annotations are not evaluated. They are always stored as strings for the benefit of introspection, but otherwise are not touched by the interpreter. Also, syntax is added to put annotations on other constructs. At least for the 2 cases above it should not be not hard, and we can store these new annotations on the class object and on the function object respectively. Obviously not evaluating annotations is a breaking change, so it can be a non-starter, but it will make things cleaner. Eugene -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/0359b148/attachment.html> From abarnert at yahoo.com Wed Jan 21 00:57:37 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 20 Jan 2015 23:57:37 +0000 (UTC) Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <CAP7+vJKd19Yb5yeVPt38sXHTKVjfPh_JO4PX_dngR50Vck2dNQ@mail.gmail.com> References: <CAP7+vJKd19Yb5yeVPt38sXHTKVjfPh_JO4PX_dngR50Vck2dNQ@mail.gmail.com> Message-ID: <538698398.2487257.1421798257472.JavaMail.yahoo@jws10073.mail.ne1.yahoo.com> On Tuesday, January 20, 2015 3:23 PM, Guido van Rossum <guido at python.org> wrote: >On Tue, Jan 20, 2015 at 2:57 PM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: > >On Jan 20, 2015, at 9:15, "Franklin? Lee" <leewangzhong+python at gmail.com> wrote: >> >>> On Fri, Jan 9, 2015 at 3:46 AM, Andrew Barnert >>> <abarnert at yahoo.com.dmarc.invalid> wrote: >>>> >>>> I believe the distinction isn't just there to keep the parser simpler, or because Guido wasn't clever enough to figure out how to embed statements in expressions in an indentation-significant language, or because the examples of Ruby and CoffeeScript didn't exist yet and now it's too late for Python; statements are a big part of what makes Python code more readable than those languages, both in-depth and in a quick scan to see the general flow control and state changes. >>>> >>>> But it's hard to explain why. And, even if I manage to explain why _I_ think statements make Python more readable, that's just my opinion; it would be nice to have something that Guido endorsed. >>>> >>>> Also, for people who want to suggest changes to Python (or design their own languages), it would be helpful to understand how the distinction fits in with other things. For example, in a pure functional language, there may be no benefit to having statements. In a language where (almost) everything is an expression, having mutating methods return self instead of None wouldn't be nearly as much of a problem. More generally, expression-ness and not-mutating are deeply connected (e.g., comprehensions aren't designed for mutation, loop statements are), but it's not easy to say how. And so on. >>> >>> Sorry to be late. >>> >>> Will this Guido post suffice as a substitute? >>> http://www.artima.com/weblogs/viewpost.jsp?thread=147358 >> >>That's not what I was looking for--but it is a great counter to my dismissal of "just to keep the parser simpler". Guido is of course right. I wasn't thinking about the fact that statement parsing and expression parsing are different modes, with state, and it's a lot harder for a human mind to subconsciously keep a stack of stateful modes in short-term memory than for a program. >> >>But I still think Python gains more from the inescapable statement-expression than this. Guido's answer is sufficient in itself, but there's more there even if it weren't true. >> >>I suppose the way to answer that would be to come up with a language where complex expressions are also indentation-sensitive (and there's no paren or backslash continuation) and show that if you get rid of the stack of stateful modes, there really is no other problem putting statements inside expressions. That's kind of the opposite of the way people usually try to tackle this problem, but if Guido's right that the usual way is inherently bound to fail... >> > >I've lost track which side you are now arguing for, but one example of a language that takes indentation in complex expressions to the extreme is Coffeescript. Having had to reverse-engineer a fairly complex piece of Coffeescript recently, I definitely thing they went too far. It seems there is no formal grammar for Coffeescript, just a translator, and even if there were a formal grammar, it would have many odd corners and sharp edges. I'm arguing the same thing: anyone who wants to embed statements inside expressions needs to understand why statements and expressions are distinct in the first place. Your old post that Franklin linked explains, given that we have that distinction, and significant indentation for statements, why any attempt to embed statements inside expressions is going to be inherently complex, and therefore likely a bad idea. But what I was hoping for was a post where you explained why we have that distinction in the first place. If Ruby and CoffeeScript don't need statements, why does Python? I think there's an answer, and it's part of the reason why code in Python tends to be more readable than in those languages (although, as you pointed out, CoffeeScript has other tangential problems), but I was hoping for _your_ answer. From guido at python.org Wed Jan 21 02:16:12 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 20 Jan 2015 17:16:12 -0800 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <538698398.2487257.1421798257472.JavaMail.yahoo@jws10073.mail.ne1.yahoo.com> References: <CAP7+vJKd19Yb5yeVPt38sXHTKVjfPh_JO4PX_dngR50Vck2dNQ@mail.gmail.com> <538698398.2487257.1421798257472.JavaMail.yahoo@jws10073.mail.ne1.yahoo.com> Message-ID: <CAP7+vJKU2oiyhmtOsM6cA=NUB+CjqLzGehy8uXQT=Gn=0hLc6Q@mail.gmail.com> On Tue, Jan 20, 2015 at 3:57 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > [...] > I'm arguing the same thing: anyone who wants to embed statements inside > expressions needs to understand why statements and expressions are distinct > in the first place. > > Your old post that Franklin linked explains, given that we have that > distinction, and significant indentation for statements, why any attempt to > embed statements inside expressions is going to be inherently complex, and > therefore likely a bad idea. > > But what I was hoping for was a post where you explained why we have that > distinction in the first place. If Ruby and CoffeeScript don't need > statements, why does Python? I think there's an answer, and it's part of > the reason why code in Python tends to be more readable than in those > languages (although, as you pointed out, CoffeeScript has other tangential > problems), but I was hoping for _your_ answer. > Hm... Practically every language I knew before I designed Python had this distinction built right into the grammar and other assumptions: Algol-60, Fortran, Pascal, C, ABC. Even Basic. I was aware of the alternative design choice: Algol-68 had statements-as-expression, and Lisp of course -- but I wasn't a big Lisp fan, and in Algol-68 it was largely a curiosity for people who wanted to write extra-terse code (also, IIRC the prevailing custom was to stick to a more conservative coding style which was derived from Algol-60). So it's hard to say to what extent this was a conscious choice and to what extent it was just tradition. But there's nothing necessarily wrong with tradition (up to a point). I think it still makes sense that statements are laid out vertically while expressions are laid out horizontally. Come to think of it, mathematics uses a similar convention -- a formula is laid out (primarily) horizontally, while a sequence of formulas (like a proof or a set of axioms) is laid out vertically. I think several Zen items apply: readability counts, and flat is better than nested. There is a lot to be said for the readability that is the result of the constraints of the blackboard or the page. (And this reminds me of how infuriating it is to me when this is violated -- e.g. 2up text in a PDF that's too tall to fit on the screen vertically, or when a dumb editor breaks lines but doesn't preserve indentation.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/99f9a619/attachment.html> From jeanpierreda at gmail.com Wed Jan 21 02:27:17 2015 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Tue, 20 Jan 2015 17:27:17 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CA+KNMzmdL+yhc_7_9NAk70VaGxofqFTZgL1wttFzCP3GhyN-sg@mail.gmail.com> References: <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> <54BC494E.3000609@canterbury.ac.nz> <CABicbJLdq519s-HDg5GM6Y1aeTbyhYrL-+fNqsxWbjn2B4C5nw@mail.gmail.com> <5C091DF5-5ACB-479F-BB2D-302F2A60C3B0@gmail.com> <CANUJvPXa8MxvpSOxV_5BuV6n4=rhOGpjnahMrU_8RJ-OFOtq6Q@mail.gmail.com> <CAM1gar4mXURzN_KqNFj8b4h2nqpRCoNrnFj_9bgFeq4EfcdjFQ@mail.gmail.com> <54BDF438.5050504@canterbury.ac.nz> <CAM1gar44vrW656SjVet00xp1r-KT7WphMRM0pSJmd33nNH1UjA@mail.gmail.com> <54BDFCDC.9050802@canterbury.ac.nz> <20150120110627.GW18556@ando.pearwood.info> <CABicbJ++udqOoQB-ZnBx9yczzzTUopDWFbPZGZwKX=uEsRi-dQ@mail.gmail.com> <B07BB060-5626-4A20-9C94-E698F3B59D51@yahoo.com> <CA+KNMzmdL+yhc_7_9NAk70VaGxofqFTZgL1wttFzCP3GhyN-sg@mail.gmail.com> Message-ID: <CABicbJKoUeCHrcqz9wSNwOu2VRu9HvW-OYRrxY13n55aq2vEfg@mail.gmail.com> On Tue, Jan 20, 2015 at 4:20 PM, Eugene Toder <eltoder at gmail.com> wrote: > On Tue, Jan 20, 2015 at 6:23 PM, Andrew Barnert > <abarnert at yahoo.com.dmarc.invalid> wrote: >> >> As I understand it (and please correct me if I'm wrong), Greg is making a >> very simple point: >> >> If type annotations are only for static type checking, as done by >> something like MyPy, they, by definition, have no use at runtime. But >> annotations are about storing information with functions at runtime. So the >> proposal is inherently storing useless information at runtime. (And it's >> also preventing anyone else from storing useful information there, but >> that's not the main issue.) > > I believe the argument is slightly different. The main issue is not with > storing the annotation, but with evaluating it. This evaluation does not > help a static type checker in any way, but introduces problems with forward > references, and limits the syntax. For example, a type checker could support > the common and natural "T1|T2" syntax for the Union type if the interpreter > did not attempt to evaluate it. The current solution is to abandon some > syntax, and put other things in quotes, which is a bit ugly. > > The second issue is that the annotations are only supported on the function > arguments and return value. There are other things that are useful to > annotate, most obviously class attributes and local variables. The current > solution is to put these annotations into comments, which is not pretty too. I would add a third problem: Evaluating at runtime fools people into thinking that this is Python code. In fact, the linter is only able to evaluate very few things, and it might even evaluate them differently to how Python does. This will be a source of confusion and bugs. > I think these are both good points. They can be addressed by introducing an > entirely new syntax, like suggested above, or by changing and extending how > the current syntax works. E.g. one can imagine that annotations are not > evaluated. They are always stored as strings for the benefit of > introspection, but otherwise are not touched by the interpreter. Also, > syntax is added to put annotations on other constructs. At least for the 2 > cases above it should not be not hard, and we can store these new > annotations on the class object and on the function object respectively. > Obviously not evaluating annotations is a breaking change, so it can be a > non-starter, but it will make things cleaner. This is an interesting idea. Since the original stated purpose of annotations is being overridden to be for mypy style type checkers, the normal backwards compatibility rules maybe don't apply as strongly. -- Devin From yawar.amin at gmail.com Wed Jan 21 03:27:37 2015 From: yawar.amin at gmail.com (Yawar Amin) Date: Tue, 20 Jan 2015 21:27:37 -0500 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <CAB_e7izPftcRKF1w-CVhrax=aMYTT1X-DGNR2==s_bYY=yfnCg@mail.gmail.com> References: <CADiSq7c0hFqm-XE7nx+r_RBLHmR-_zVinoDD6-9ja54M-UGo2w@mail.gmail.com> <818732810.4093303.1420793208359.JavaMail.yahoo@jws100159.mail.ne1.yahoo.com> <CAB_e7izPftcRKF1w-CVhrax=aMYTT1X-DGNR2==s_bYY=yfnCg@mail.gmail.com> Message-ID: <54BF0E99.2020800@gmail.com> Hi Franklin. On 2015-01-20 12:15, Franklin? Lee wrote: > [...] > Will this Guido post suffice as a substitute? > http://www.artima.com/weblogs/viewpost.jsp?thread=147358 Language design is not just solving puzzles? I read it very closely before starting this thread :-) To be honest, after reading it I came away thinking that language design/API design/whatever technical design really _can_ be just solving a puzzle--if you encode unseability as one of the parameters of the puzzle. > Yawar might also need this, if he chooses to pursue this path. > https://wiki.python.org/moin/AlternateLambdaSyntax Thanks, I read that pretty closely too. From what I could tell, they are all rather unfeasible for the simple reason that Python doesn't allow statements inside of expressions (nor should it--I believe; it would not be a good fit for the current syntax). The closest I've seen to one of those alternate lamba syntaxes being implemented is in the Mochi language, which compiles down to Python 3 bytecode. Personally, I've gone in a different direction--Google my name + lambdak and you will find it :-) Regards, Yawar -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 834 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/a358a082/attachment.sig> From tjreedy at udel.edu Wed Jan 21 03:57:10 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 20 Jan 2015 21:57:10 -0500 Subject: [Python-ideas] Conventions for Python code documentation In-Reply-To: <CAP7+vJLr7DjDst5o-Ws1rqnadT4Xpg+mekEgXr88XHvNVkjQbg@mail.gmail.com> References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> <85oapur4d8.fsf_-_@benfinney.id.au> <CAN-Kwu3Nj0=1+6B=uNmJz8D8LORfAAu62GLJ8_R5ZwwKjxUg=Q@mail.gmail.com> <CAP7+vJ+=3uebwL=+QOajwYVd=h+RaEFsRy8GqKQnDBFrjt7uog@mail.gmail.com> <m9kt66$a1n$1@ger.gmane.org> <CAP7+vJLr7DjDst5o-Ws1rqnadT4Xpg+mekEgXr88XHvNVkjQbg@mail.gmail.com> Message-ID: <m9n4i7$f75$1@ger.gmane.org> On 1/20/2015 11:31 AM, Guido van Rossum wrote: > On Mon, Jan 19, 2015 at 10:39 PM, Terry Reedy > <tjreedy at udel.edu > Because I have not seen such used, except maybe once, I have been > meaning to ask if I am allowed to use rst markup in stdlib docstrings? > > > I prefer you don't use them. That's easy ;-). > My impression is that the plan is that stdlib .py files would not > have type annotations, but that they might be in stub files. Still > true? > > Correct. The stdlib is a special flower in so many ways... (Stubs are > also useful for 3rd party libraries that haven't been updated yet.) My particular interest here is adding type information to the signature information already displayed in Idle calltips. It would be nice if the typing module (eventually) had a function to do that. It could be used by help(function) also. The point is to help people write correct calls in the first place, by providing easily accessed information, rather than wait for a checker or the runtime to find the error. -- Terry Jan Reedy From ron3200 at gmail.com Wed Jan 21 05:43:58 2015 From: ron3200 at gmail.com (Ron Adam) Date: Tue, 20 Jan 2015 22:43:58 -0600 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <20150120104012.GU18556@ando.pearwood.info> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> <m9k397$eo9$1@ger.gmane.org> <-3376261731312819627@unknownmsgid> <6ea13fe6-0831-4126-a94b-15b6bd1c83a7@googlegroups.com> <20150120104012.GU18556@ando.pearwood.info> Message-ID: <m9naqf$9s1$1@ger.gmane.org> On 01/20/2015 04:40 AM, Steven D'Aprano wrote: > On Mon, Jan 19, 2015 at 08:10:35PM -0800, Neil Girdhar wrote: > >> >If you decide to invent a relative error function, > The error functions we have been talking about are hardly "invented". > They're mathematically simple and obvious, and can be found in just > about any undergraduate book on numerical computing: > > - The absolute error between two quantities a and b is the > absolute difference between them, abs(a-b). > > - The relative error is the difference relative to some > denominator d, abs(a-b)/abs(d), typically with d=a or d=b. > > If you happen to know that b is the correct value, then it is common to > choose b as the denominator. If you have no a priori reason to think > either a or b is correct, or if you prefer a symmetrical function, a > common choice is to use d = min(abs(a), abs(b)). The more we talk about this, the more I'm beginning to dislike the symmetric version. We are trading an explicit (first, second) relationship with an implicit (smaller, larger) relationship. For Python's general use, I don't like that. Sorry. :/ Example... Testing two resistors are within 5% of 10k ohms. is_close(10500, 10000, 0.05) 9500.0<--->10500.0 True is_close(10000, 9500, 0.05) 9025.0<--->9975.0 False The order doesn't matter, but the size has an effect. Using the larger value as the divider can result in false positives on the lower end instead of false negative on the higher end. So I strongly think any such function (in Python) should have meaningful named arguments. It's going to save a lot of explaining down the road. :-) > See, for example: > > http://mathworld.wolfram.com/AbsoluteError.html > http://mathworld.wolfram.com/RelativeError.html > > >> >my suggestion is: >> >(a-b)/b + log(b/a), which is nonnegative, zero only at equality, and >> >otherwise penalizes positive a for being different than some target >> >positive b. To me, it seems like guessing b using 1.9b is better than >> >guessing it as 0.1b, and so on. This corresponds to exponential KL >> >divergence, which has a clear statistical meaning, but only applies to >> >positive numbers. > Do you have a reference or derivation for this? I'm happy to admit that > I'm no Knuth or Kahan, but I've read a bit of numerical computing[1] and > I've never seen anyone add a log term. I'm not even sure why you would > do so. I found this... There is a short paragraph there about measuring error in natural log units. http://people.duke.edu/~rnau/411log.htm This next one has a lot of good info I think is relevant to this topic. http://www2.phy.ilstu.edu/~wenning/slh/ It does have one symmetric formula... relative difference, that uses the average of the two values. It's probably what most people what for comparing two generated data point where neither one can be picked as a reference. > [1] I know just enough to know how much I don't know. I know even less. I'm hoping to know what I don't know soon, but I think I have a lot to learn still. ;-) Cheers, Ron From guido at python.org Wed Jan 21 05:45:41 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 20 Jan 2015 20:45:41 -0800 Subject: [Python-ideas] Conventions for Python code documentation In-Reply-To: <m9n4i7$f75$1@ger.gmane.org> References: <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <54B975F3.4000104@gmail.com> <CAP7+vJJdr-JxjB8KM_j1uD4CO8NUP+oAtmzbO+bkDaUagF5Dig@mail.gmail.com> <B68998E2-0AF8-4D31-862E-06F977B449DA@gmail.com> <CAP7+vJJUNZRXMxSq35_fmsPT58gY41XR9TX0TMHGZ3FQQ=qGYg@mail.gmail.com> <9D7235FC-FF27-44BD-AF10-3D1FC07F540E@gmail.com> <851tmsu8lf.fsf@benfinney.id.au> <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAPTjJmovTA+gV6u0rzbzLoFn_e_5cqPuL7vmzY0uvaX-KJ5cGQ@mail.gmail.com> <4D61794D-74FA-448E-9B50-8E5F198AA759@gmail.com> <20150119015839.GT18556@ando.pearwood.info> <B8D158F7-8DD0-4EFF-8BBF-3A69C671C413@gmail.com> <85oapur4d8.fsf_-_@benfinney.id.au> <CAN-Kwu3Nj0=1+6B=uNmJz8D8LORfAAu62GLJ8_R5ZwwKjxUg=Q@mail.gmail.com> <CAP7+vJ+=3uebwL=+QOajwYVd=h+RaEFsRy8GqKQnDBFrjt7uog@mail.gmail.com> <m9kt66$a1n$1@ger.gmane.org> <CAP7+vJLr7DjDst5o-Ws1rqnadT4Xpg+mekEgXr88XHvNVkjQbg@mail.gmail.com> <m9n4i7$f75$1@ger.gmane.org> Message-ID: <CAP7+vJJaeTG6U3dw9xTcPsthnsP=GEpyNTEoOoH3u6wJ6AkZcQ@mail.gmail.com> On Tue, Jan 20, 2015 at 6:57 PM, Terry Reedy <tjreedy at udel.edu> wrote: > > My particular interest here is adding type information to the signature > information already displayed in Idle calltips. It would be nice if the > typing module (eventually) had a function to do that. It could be used by > help(function) also. The point is to help people write correct calls in > the first place, by providing easily accessed information, rather than wait > for a checker or the runtime to find the error. > That is indeed part of the plan. In first approximation, the repr() of an annotation (if it is one of the classes defined in typing.py, such as Union, Callable, or classes derived from Generic[...]) will be usable to describe the type. E.g. the repr() of Tuple[int, str] is exactly 'Tuple[int, str]'. (This works today in my prototype at https://github.com/ambv/typehinting/tree/master/prototyping .) There are some issues around forward references (not yet implemented) which mean that you probably have to use a simple helper function defined in typing.py to get the annotation, but it should be pretty smooth sailing. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/34e6e56e/attachment.html> From greg.ewing at canterbury.ac.nz Wed Jan 21 06:09:12 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 21 Jan 2015 18:09:12 +1300 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <B07BB060-5626-4A20-9C94-E698F3B59D51@yahoo.com> References: <5665C80B-6E82-4FB8-A579-FE83C206F0DD@gmail.com> <CAP7+vJ+rhotziv6S7U6gJwxRLgMrfFRCCkdvJhqu_YYFsxusDg@mail.gmail.com> <54BC494E.3000609@canterbury.ac.nz> <CABicbJLdq519s-HDg5GM6Y1aeTbyhYrL-+fNqsxWbjn2B4C5nw@mail.gmail.com> <5C091DF5-5ACB-479F-BB2D-302F2A60C3B0@gmail.com> <CANUJvPXa8MxvpSOxV_5BuV6n4=rhOGpjnahMrU_8RJ-OFOtq6Q@mail.gmail.com> <CAM1gar4mXURzN_KqNFj8b4h2nqpRCoNrnFj_9bgFeq4EfcdjFQ@mail.gmail.com> <54BDF438.5050504@canterbury.ac.nz> <CAM1gar44vrW656SjVet00xp1r-KT7WphMRM0pSJmd33nNH1UjA@mail.gmail.com> <54BDFCDC.9050802@canterbury.ac.nz> <20150120110627.GW18556@ando.pearwood.info> <CABicbJ++udqOoQB-ZnBx9yczzzTUopDWFbPZGZwKX=uEsRi-dQ@mail.gmail.com> <B07BB060-5626-4A20-9C94-E698F3B59D51@yahoo.com> Message-ID: <54BF3478.1060107@canterbury.ac.nz> Andrew Barnert wrote: > But I think there's a hole in the premise. Python's key strength is good > debug-ability and amazingly powerful introspection. (I mean Python's _two_ > key strengths are...) Information that has no runtime meaning to the > interpreter may still have meaning to the user who's debugging code, There's nothing wrong with making the annotations *available* at run time. All I'm saying is that *evaluating* them at run time hinders rather than helping for the intended use. -- Greg From mrocklin at gmail.com Wed Jan 21 06:20:43 2015 From: mrocklin at gmail.com (Matthew Rocklin) Date: Tue, 20 Jan 2015 21:20:43 -0800 Subject: [Python-ideas] Keyword/Symbol literals Message-ID: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> It would be nice to have literal keywords/symbols. By this I mean, terms that look like words but are not previously defined variables. *Prior art* Some other languages prefix by colon, :foo, or use a backtick, `foo. julia> :foo :foo julia> typeof(:foo) Symbol user=> :foo :foo user=> (type :foo) clojure.lang.Keyword user=> `foo user/foo user=> (type `foo) clojure.lang.Symbol *Why is this useful?* One use case in NumPy/Pandas use is to specify columns or fields of data without resorting to strings. E.g. df = pandas.load(...) df.sort(:name) *What do people do now?* Currently people use auto-generated attributes df.sort(df.name) or strings df.sort('name') auto-generated attributes work great until you want to use chained expressions df.change_dataframe().sort(some_new_column_not_in_df) strings work but feel unpleasant *Prior discussion?* This is a common language construct so my guess is that it has come up before. Sadly Google searching the terms *keywords* and *symbols* results it a lot of unrelated material. Can anyone point me to previous discussion? There are clearly issues with using :foo in that it overlaps with slice syntax, presumably some other character could be pressed into service if this was found worthwhile. I can come up with more motivating use cases if desired. Best, -Matthew -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/4130e6da/attachment-0001.html> From stefan_ml at behnel.de Wed Jan 21 08:18:29 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 21 Jan 2015 08:18:29 +0100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <54BD6908.7090701@canterbury.ac.nz> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9i6u4$lf$1@ger.gmane.org> <54BD6908.7090701@canterbury.ac.nz> Message-ID: <m9njs6$dgf$1@ger.gmane.org> Greg Ewing schrieb am 19.01.2015 um 21:28: >> On 1/18/2015 10:40 PM, Guido van Rossum wrote: >> >> > Very few people subclass builtins.list ... >> > So perhaps Cython could just assume that typing.List means >> > builtins.list and (dynamically) reject calls that pass a subclass >> > of builtins.list? > > That's actually a reasonable idea. Anything that accepts > something as weird as a trianglular matrix implemented as > a list subclass will probably accept any kind of sequence, > so you might as well type it as Sequence (or whatever the > abstract sequence interface is to be called). Well, that's what Cython currently does, based on its own type declaration semantics for builtin types. I agree that subtyping is rare for lists, but it's quite common for dicts, including defaultdict, OrderedDict, or even YourOwnMissingKeyDictImpl. And there's set and frozenset, so if you type something as "set" (which no-one does, really), it would have to allow frozenset as well in most (read-only) cases (or maybe the other way round). But set and frozenset do not inherit from each other, so instead of typing something as "set", it would have to be the "Set" or "MutableSet" ABC type, which then is uninteresting again for any C level optimisation, as it's way too broad and there's no C-API for it. There's really a visible gap between what's helpful at a Python type checking level and what's a helpful compiler type hint for Cython and CPython. Stefan From stefan_ml at behnel.de Wed Jan 21 08:31:03 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 21 Jan 2015 08:31:03 +0100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> Message-ID: <m9nkjn$p3d$1@ger.gmane.org> Guido van Rossum schrieb am 19.01.2015 um 04:40: > On Sun, Jan 18, 2015 at 12:36 PM, Stefan Behnel wrote: >> would be nice to at least be able to explicitly express somehow that >> subtypes should be disallowed. > > Can you come up with a specific proposal? Maybe we're not as far apart as > we think. :-) Normally, "x: List[int]" would mean that x is either a list or a subtype of list, where each item is an int or a subtype. To disallow subtypes, you could have something like "exactly(List)[int]" or "List[exactly(int)]" or "exactly(List[int])", which would then make it clear that the exact type matters, only for the List in the first case, only for the item type in the second, and (this is not necessarily required) for all types inside of the "exactly()" call in the last case. The return value of "exactly()" could be some kind of type wrapper object which marks the wrapped type construct as allowing "no subtypes". This would keep the normal case of allowing subtypes everywhere normal and simple, but would allow restricting the type to disallow subtypes in the cases where it seems appropriate. Stefan From bruce at leban.us Wed Jan 21 08:36:32 2015 From: bruce at leban.us (Bruce Leban) Date: Tue, 20 Jan 2015 23:36:32 -0800 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> Message-ID: <CAGu0AnvaqNTxWLk_6R9YkvHPWMVxuXAaveeWqv8AKT0jHfx_iw@mail.gmail.com> class Symbol(): def __getattr__(self, var): t = object() object.__setattr__(self, var, t) return t def __setattr__(self, var, val): raise AttributeError() S = Symbol() Now use S.abc and S.xyz etc. They automatically get created as unique objects. Trying to set one of the values raises. With a little more work you can make it pretty printable. Is this a valuable enough feature to build into the language and dedicate a special character to it? I think you'd have to have a lot of use cases and a lot of usage. If it were added I think the best character to use without colliding with anything would be !foo. --- Bruce Check out my new puzzle book: http://Ju.mp/ingToConclusions Get it free here: http://Ju.mp/ingToConclusionsFree (available on iOS) On Tue, Jan 20, 2015 at 9:20 PM, Matthew Rocklin <mrocklin at gmail.com> wrote: > It would be nice to have literal keywords/symbols. By this I mean, terms > that look like words but are not previously defined variables. > > *Prior art* > > Some other languages prefix by colon, :foo, or use a backtick, `foo. > > julia> :foo > :foo > julia> typeof(:foo) > Symbol > > user=> :foo > :foo > user=> (type :foo) > clojure.lang.Keyword > > user=> `foo > user/foo > user=> (type `foo) > clojure.lang.Symbol > > *Why is this useful?* > > One use case in NumPy/Pandas use is to specify columns or fields of data > without resorting to strings. E.g. > > df = pandas.load(...) > df.sort(:name) > > *What do people do now?* > > Currently people use auto-generated attributes > > df.sort(df.name) > > or strings > > df.sort('name') > > auto-generated attributes work great until you want to use chained > expressions > > df.change_dataframe().sort(some_new_column_not_in_df) > > strings work but feel unpleasant > > *Prior discussion?* > > This is a common language construct so my guess is that it has come up > before. Sadly Google searching the terms *keywords* and *symbols* > results it a lot of unrelated material. Can anyone point me to previous > discussion? > > There are clearly issues with using :foo in that it overlaps with slice > syntax, presumably some other character could be pressed into service if > this was found worthwhile. > > I can come up with more motivating use cases if desired. > > Best, > -Matthew > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/dc157ba6/attachment.html> From me+python at ixokai.io Wed Jan 21 08:48:58 2015 From: me+python at ixokai.io (Stephen Hansen) Date: Tue, 20 Jan 2015 23:48:58 -0800 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> Message-ID: <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> > > *What do people do now?* > or strings > > df.sort('name') > ... > strings work but feel unpleasant > Why? What does :foo get you that 'foo' doesn't? One character being saved doesn't seem like a good reason to add syntax. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150120/28be38a9/attachment-0001.html> From toddrjen at gmail.com Wed Jan 21 08:58:51 2015 From: toddrjen at gmail.com (Todd) Date: Wed, 21 Jan 2015 08:58:51 +0100 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> Message-ID: <CAFpSVp+mC50vXdn86_k-DX5LVv_K-9i6jfkrm_3q4xNY_rzOew@mail.gmail.com> On Jan 21, 2015 6:21 AM, "Matthew Rocklin" <mrocklin at gmail.com> wrote: > > It would be nice to have literal keywords/symbols. By this I mean, terms that look like words but are not previously defined variables. > > Prior art > > Some other languages prefix by colon, :foo, or use a backtick, `foo. > > julia> :foo > :foo > julia> typeof(:foo) > Symbol > > user=> :foo > :foo > user=> (type :foo) > clojure.lang.Keyword > > user=> `foo > user/foo > user=> (type `foo) > clojure.lang.Symbol > > Why is this useful? > > One use case in NumPy/Pandas use is to specify columns or fields of data without resorting to strings. E.g. > > df = pandas.load(...) > df.sort(:name) > > What do people do now? > > Currently people use auto-generated attributes > > df.sort(df.name) > > or strings > > df.sort('name') > > auto-generated attributes work great until you want to use chained expressions > > df.change_dataframe().sort(some_new_column_not_in_df) > > strings work but feel unpleasant > > Prior discussion? > > This is a common language construct so my guess is that it has come up before. Sadly Google searching the terms keywords and symbols results it a lot of unrelated material. Can anyone point me to previous discussion? > > There are clearly issues with using :foo in that it overlaps with slice syntax, presumably some other character could be pressed into service if this was found worthwhile. > > I can come up with more motivating use cases if desired. At best this saves one character, may reduce readability, and would require a huge amount of software to support another code path. Is there any advantage to this besides the one character saved? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/b71a8f26/attachment.html> From edk141 at gmail.com Wed Jan 21 10:07:14 2015 From: edk141 at gmail.com (Ed Kellett) Date: Wed, 21 Jan 2015 09:07:14 +0000 Subject: [Python-ideas] Keyword/Symbol literals References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAFpSVp+mC50vXdn86_k-DX5LVv_K-9i6jfkrm_3q4xNY_rzOew@mail.gmail.com> Message-ID: <CABmzr0gx7SKGYH8aB-BBVTaNqNVpOLBBFxU0hrKQFVVZGKPM5Q@mail.gmail.com> On Wed Jan 21 2015 at 07:59:41 Todd <toddrjen at gmail.com> wrote: > At best this saves one character, may reduce readability, and would > require a huge amount of software to support another code path. Is there > any advantage to this besides the one character saved? > If symbols are [a subclass of] str, I don't see why a huge amount of software - or indeed any software - would need to do anything special to support it. (Other than the implementation of the feature itself, obviously). That said, I still don't think it's very useful. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/405c2750/attachment.html> From abarnert at yahoo.com Wed Jan 21 10:13:10 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 21 Jan 2015 01:13:10 -0800 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> Message-ID: <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> On Jan 20, 2015, at 23:48, Stephen Hansen <me+python at ixokai.io> wrote: >> What do people do now? >> or strings >> >> df.sort('name') > > ... > >> strings work but feel unpleasant > > Why? > > What does :foo get you that 'foo' doesn't? > > One character being saved doesn't seem like a good reason to add syntax. Mainly they're used to fill in gaps where a language isn't dynamic enough to just use strings, which Python doesn't need. (Sometimes this is claimed to be a strength rather than a weakness. Arguably, using auto-interned symbols instead of strings for things like attribute or global lookup could be more efficient, but I don't think languages like Ruby have shown any demonstrable gain. And we just had a thread from someone who considered it a "bug" that Python allows setattr(spam, '3', 3) when 3 isn't a valid identifier, which symbols "fix" automatically. But even if these were real advantages, Python obviously isn't going to make such a radical change as banning strings for dynamic access.) A language can also add things on top of the symbol/string distinction (besides auto-interning)--e.g., you can add syntactic sugar for dynamic lookup ("foo.:bar" means "getattr(foo, bar)"), or make undefined variables have the value of their symbol so you can LBYL test them, or all kinds of other things--but again, they're all things I don't think Python wants or needs. And without a vague idea for why Python would want to do any of the things that other languages do with their symbol-vs.-string distinctions, this really does seem to come down to just saving one character. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/f041c849/attachment.html> From masklinn at masklinn.net Wed Jan 21 11:05:04 2015 From: masklinn at masklinn.net (Masklinn) Date: Wed, 21 Jan 2015 11:05:04 +0100 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> Message-ID: <06B4FE82-98CE-4D31-81FA-549440AE8B21@masklinn.net> On 2015-01-21, at 10:13 , Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: > On Jan 20, 2015, at 23:48, Stephen Hansen <me+python at ixokai.io> wrote: > >> What do people do now? >> or strings >> >> df.sort('name') >> >> ... >> >> strings work but feel unpleasant >> >> Why? >> >> What does :foo get you that 'foo' doesn't? >> >> One character being saved doesn't seem like a good reason to add syntax. > > Mainly they're used to fill in gaps where a language isn't dynamic enough to just use strings, which Python doesn't need. That's debatable, various languages have used them for fairly different reasons. AFAIK: * Smalltalk had symbols because its strings are mutable, not internable and O(n) comparison (implementation may vary, Squeak's literal strings are/were immutable but I do not know if they were interned). Symbols are always interned and provide O(1) comparison which is useful in many situations * Ruby inherited behaviour and purpose (the community did later add syntactic sugar on top of symbols, like `&:foo` for {|o| o.foo}) * Erlang uses symbols (which it calls atoms) because its strings are linked lists of integers and thus relatively expensive; atoms are literal and interned with identity (across the process too), historically it expected such a small number of atoms they weren't even GC'd * Lisp's use of symbols comes from the metaprogramming side: how do you represent a symbol (a variable name) in a macro? With a string it's not possible to differentiate actual strings and strings-as-identifiers. So symbols it is. And since symbols were in, cheap and useful they could be exposed to the developer. It's got nothing to do with "dynamic enough", that's just nonsense, it has to do with the semantics of the language and its strings, and the string's suitability to some specific purposes or use cases. Which does not mean symbols should be added to Python (let alone that all languages should have symbols). -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/896606ab/attachment-0001.html> From epsilonmichael at gmail.com Wed Jan 21 11:12:56 2015 From: epsilonmichael at gmail.com (Michael Mitchell) Date: Wed, 21 Jan 2015 02:12:56 -0800 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> Message-ID: <CABvhCyG0xNn3RaaQfjpYUq4TbONQNbCYW7=DFhDC=GgrS9hatg@mail.gmail.com> On Tue, Jan 20, 2015 at 9:20 PM, Matthew Rocklin <mrocklin at gmail.com> wrote: > df.sort(df.name) > > or strings > > df.sort('name') > > auto-generated attributes work great until you want to use chained > expressions > > df.change_dataframe().sort(some_new_column_not_in_df) > I would like to add that Python 3.4 enums would make a better and more idiomatic replacement for auto-generated attributes, e.g. df.columns.name (which may look similar to SQLAlchemy semantics for those familiar). Note how languages such as Ruby, Clojure, and Julia do not have enums and how languages with enums tend to not have symbols. With immutable strings and enums already present, symbols do not add very much value. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/2083db6f/attachment.html> From jsbueno at python.org.br Wed Jan 21 12:27:39 2015 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Wed, 21 Jan 2015 09:27:39 -0200 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <m9naqf$9s1$1@ger.gmane.org> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> <m9k397$eo9$1@ger.gmane.org> <-3376261731312819627@unknownmsgid> <6ea13fe6-0831-4126-a94b-15b6bd1c83a7@googlegroups.com> <20150120104012.GU18556@ando.pearwood.info> <m9naqf$9s1$1@ger.gmane.org> Message-ID: <CAH0mxTQR=sPKULnBpc8ukCZg-A53ynZgBwomjjRv-M0RhDsu6Q@mail.gmail.com> While we are at this, since someone will have to step up to come up with a PEP, I have these two ideas to drop in the brainstorm - as I'd like to the one writting the PEP text to at least have them in mind: 1) Implement a Number class that behaves as a float number, and does the fuzzy comparisons automatically. Justificative: in a whole lot of code, having a "casted" FuzzyFloat(mynumber) to be able to be compared to others with "==" and "<=" would be much more readable than the Javaesque " if is_close(my_number, other) or my_number < other: " (against: "if FuzzyFloat(my_number) <= other :" , or simply "if my_number <= other: " if we are already working with the FuzzyFloat type) It could use a "context" just like "decimal" to specify comparison parameters I made a naive approach to this when these threads started a couple of days ago - of course it is far from a real implementation - https://gist.github.com/jsbueno/1e62be882639a13180f6 (forgive me for not using ABC's there, I just wrote that to have a "feel" on this idea on the interactive console - and it was a nice test-drive) 2) Put a though on how things could work with other numeric types than float and complex - "decimal" comes to mind. js -><- From epsilonmichael at gmail.com Wed Jan 21 12:32:49 2015 From: epsilonmichael at gmail.com (Michael Mitchell) Date: Wed, 21 Jan 2015 03:32:49 -0800 Subject: [Python-ideas] Should bool continue to inherit from int? Message-ID: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> PEP 285 <http://legacy.python.org/dev/peps/pep-0285> provides some justification for why arithmetic operations should be valid on bools and why bool should inherit from int (see points (4) and (6) respectively). Since it's been 12 years (it's possible this has been brought up again between now and then), I thought it could be worthwhile to take another look. I am mostly interested in a resurveying of the questions: 1) Would it still be very inconvenient to implement bools separately from ints? 2) Do most Python users still agree that arithmetic operations should be supported on booleans? Follow-up thought: Something I noticed is that with PEP 484 <https://www.python.org/dev/peps/pep-0484/> (Type Hints) specified as is, there would be no way to statically verify that a function will only operate on ints and not bools. An example would be a function that can only operate on integer values in a JSON dict created by the builtin `json` module (using the default decoder) cannot exist, as that function could operate on the boolean values of the dict. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/b6f5d272/attachment.html> From steve at pearwood.info Wed Jan 21 12:40:39 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 21 Jan 2015 22:40:39 +1100 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> Message-ID: <20150121114038.GZ18556@ando.pearwood.info> On Tue, Jan 20, 2015 at 11:48:58PM -0800, Stephen Hansen wrote: > > > > *What do people do now?* > > or strings > > > > df.sort('name') > ... > > strings work but feel unpleasant > > Why? > > What does :foo get you that 'foo' doesn't? It gets you an atomic data type which isn't a string and isn't an int. The point of data types is what they *don't allow* as much as what they do allow. Symbols is that they are atomic symbolic names, not lists, ints, dicts or strings. 'foo'.upper()[1:] is a meaningful operation. 'foo' is a sequence of characters, so you can slice it, and characters can be uppercased and lowercased and case-folded. Strings have prefixes and suffixes and substrings. If :foo is merely yet another syntax for creating strings, then it is pointless and you should just use a string. But the point of symbols is that they don't support string operations. :foo.upper()[1:] is a meaningless operation, as is :foo + 23. Python already has a number of symbol-like objects in the language: None True and False NotImplemented Ellipsis and possible a few others. With the exception of True and False, most of them have virtually no "interesting" or public methods: e.g. you can convert None into a string, but you can't do None.upper()[1:]. Although, arguably, they are more like Enums than symbols, at least the way Julia uses symbols. http://stackoverflow.com/questions/23480722/what-is-a-symbol-in-julia I'd like to learn more about Julia-style metaprogramming with symbols. The alternative is to treat symbols just as an object with a name and little behaviour (like None), which is not as interesting and more suited to Enums. -- Steve From steve at pearwood.info Wed Jan 21 12:47:28 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 21 Jan 2015 22:47:28 +1100 Subject: [Python-ideas] Should bool continue to inherit from int? In-Reply-To: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> References: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> Message-ID: <20150121114728.GA18556@ando.pearwood.info> On Wed, Jan 21, 2015 at 03:32:49AM -0800, Michael Mitchell wrote: > PEP 285 <http://legacy.python.org/dev/peps/pep-0285> provides some > justification for why arithmetic operations should be valid on bools and > why bool should inherit from int (see points (4) and (6) respectively). > Since it's been 12 years (it's possible this has been brought up again > between now and then), I thought it could be worthwhile to take another > look. > > I am mostly interested in a resurveying of the questions: > 1) Would it still be very inconvenient to implement bools separately from > ints? Yes, it would break a lot of code that for the last twelve years has relied on being able to perform arithmetic on bools. Arguably, Python 3 might have been the opportunity to break that backwards compatibility, but we're up to 3.4 now so we would need a very good reason to do so. > 2) Do most Python users still agree that arithmetic operations should be > supported on booleans? I don't know about "most", but this Python user thinks so. > Follow-up thought: > Something I noticed is that with PEP 484 > <https://www.python.org/dev/peps/pep-0484/> (Type Hints) specified as is, > there would be no way to statically verify that a function will only > operate on ints and not bools. If you want an int, I'm not sure why you would want to exclude bools since they are ints. > An example would be a function that can only > operate on integer values in a JSON dict created by the builtin `json` > module (using the default decoder) cannot exist, as that function could > operate on the boolean values of the dict. I'm afraid I don't quite follow this example. Can you give some sample code (with or without annotations)? -- Steve From steve at pearwood.info Wed Jan 21 12:59:34 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 21 Jan 2015 22:59:34 +1100 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <CAH0mxTQR=sPKULnBpc8ukCZg-A53ynZgBwomjjRv-M0RhDsu6Q@mail.gmail.com> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> <m9k397$eo9$1@ger.gmane.org> <-3376261731312819627@unknownmsgid> <6ea13fe6-0831-4126-a94b-15b6bd1c83a7@googlegroups.com> <20150120104012.GU18556@ando.pearwood.info> <m9naqf$9s1$1@ger.gmane.org> <CAH0mxTQR=sPKULnBpc8ukCZg-A53ynZgBwomjjRv-M0RhDsu6Q@mail.gmail.com> Message-ID: <20150121115934.GB18556@ando.pearwood.info> On Wed, Jan 21, 2015 at 09:27:39AM -0200, Joao S. O. Bueno wrote: > 1) Implement a Number class that behaves as a float number, and does > the fuzzy comparisons automatically. -1 on this. > Justificative: in a whole lot of code, having a "casted" > FuzzyFloat(mynumber) to be able to be compared to others with "==" and "<=" > would be much more readable than the Javaesque > > " if is_close(my_number, other) or my_number < other: " > (against: > "if FuzzyFloat(my_number) <= other :" > , or simply > > "if my_number <= other: " If that sort of comparison is important, a helper function is easy to write. But it really sounds like you want a full interval arithmetic class, which is a lot more than just fuzzy comparisons. There are two problems with overriding == in this way: (1) What do you mean by "is close"? Do you mean that they are within 0.0001 or 0.00000001 or within 10000.0? With a binary operator == there is no easy way to specify an error tolerance, which means you're stuck with using a language-wide default which is very unlikely to be suitable for your application. A context manager is one solution, but that's still awkward. Think about doing: a == b or x == y where the a, b comparison and x, y comparison have different tolerances. (2) Fuzzy equality means that there are situations where: a == b and b == c but a != c The language APL tried to turn this into a feature: The intransitivity of [tolerant] equality is well known in practical situations and can be easily demonstrated by sawing several pieces of wood of equal length. In one case, use the first piece to measure subsequent lengths; in the second case, use the last piece cut to measure the next. Compare the lengths of the two final pieces. ? Richard Lathwell, APL Comparison Tolerance, APL76, 1976 but I'm not so sure. If we introduced a new comparison operator (perhaps the Unicode symbol ? or the ASCII ~= ?) I'd be less concerned but I think overriding == for fuzzy equality is a mistake. -- Steve From jsbueno at python.org.br Wed Jan 21 13:29:43 2015 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Wed, 21 Jan 2015 10:29:43 -0200 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <20150121115934.GB18556@ando.pearwood.info> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> <m9k397$eo9$1@ger.gmane.org> <-3376261731312819627@unknownmsgid> <6ea13fe6-0831-4126-a94b-15b6bd1c83a7@googlegroups.com> <20150120104012.GU18556@ando.pearwood.info> <m9naqf$9s1$1@ger.gmane.org> <CAH0mxTQR=sPKULnBpc8ukCZg-A53ynZgBwomjjRv-M0RhDsu6Q@mail.gmail.com> <20150121115934.GB18556@ando.pearwood.info> Message-ID: <CAH0mxTR3DZHnkqc2UMLojiRymi_CkGT=Y_jSTkJrjqqnMG5n0A@mail.gmail.com> On 21 January 2015 at 09:59, Steven D'Aprano <steve at pearwood.info> wrote: > On Wed, Jan 21, 2015 at 09:27:39AM -0200, Joao S. O. Bueno wrote: > >> 1) Implement a Number class that behaves as a float number, and does >> the fuzzy comparisons automatically. > > -1 on this. > > >> Justificative: in a whole lot of code, having a "casted" >> FuzzyFloat(mynumber) to be able to be compared to others with "==" and "<=" >> would be much more readable than the Javaesque >> >> " if is_close(my_number, other) or my_number < other: " >> (against: >> "if FuzzyFloat(my_number) <= other :" >> , or simply >> >> "if my_number <= other: " > > If that sort of comparison is important, a helper function is easy to > write. But it really sounds like you want a full interval arithmetic > class, which is a lot more than just fuzzy comparisons. > > There are two problems with overriding == in this way: > > (1) What do you mean by "is close"? Do you mean that they are within > 0.0001 or 0.00000001 or within 10000.0? With a binary operator == there > is no easy way to specify an error tolerance, which means you're stuck > with using a language-wide default which is very unlikely to be suitable > for your application. A context manager is one solution, but that's > still awkward. Think about doing: > > a == b or x == y > > where the a, b comparison and x, y comparison have different tolerances. > Well - that would be treated by the fuzzyfloat context -just in the same way contexts are used by decimal.Decimals And of course, if one happens to be comparing numbers with different tolerances, the smaller should rule. (Although when comparing with a non-fuzzy, which has a tolerance of 0, the fuzzy one should be used or there would be no point in having it in the first place. So, yeah, there is a problem there. Requiring both numbers to be Fuzzy might be a way out) > > (2) Fuzzy equality means that there are situations where: > > a == b and b == c > > but > > a != c > > The language APL tried to turn this into a feature: > > The intransitivity of [tolerant] equality is well known in practical > situations and can be easily demonstrated by sawing several pieces > of wood of equal length. In one case, use the first piece to measure > subsequent lengths; in the second case, use the last piece cut to > measure the next. Compare the lengths of the two final pieces. > ? Richard Lathwell, APL Comparison Tolerance, APL76, 1976 Point taken. Anyway, upon reminding how unconfortable ".is_equal" is in source code, I think having an explicit class that can use "==" in this way is not that bad; > > but I'm not so sure. If we introduced a new comparison operator (perhaps > the Unicode symbol ? or the ASCII ~= ?) I'd be less concerned but I > think overriding == for fuzzy equality is a mistake. That is nice. As I said, I'd like people to at least having this in mind. > > -- > Steve > ______________________ From jsbueno at python.org.br Wed Jan 21 13:35:18 2015 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Wed, 21 Jan 2015 10:35:18 -0200 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <CAH0mxTR3DZHnkqc2UMLojiRymi_CkGT=Y_jSTkJrjqqnMG5n0A@mail.gmail.com> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> <m9k397$eo9$1@ger.gmane.org> <-3376261731312819627@unknownmsgid> <6ea13fe6-0831-4126-a94b-15b6bd1c83a7@googlegroups.com> <20150120104012.GU18556@ando.pearwood.info> <m9naqf$9s1$1@ger.gmane.org> <CAH0mxTQR=sPKULnBpc8ukCZg-A53ynZgBwomjjRv-M0RhDsu6Q@mail.gmail.com> <20150121115934.GB18556@ando.pearwood.info> <CAH0mxTR3DZHnkqc2UMLojiRymi_CkGT=Y_jSTkJrjqqnMG5n0A@mail.gmail.com> Message-ID: <CAH0mxTRgfHfrQP=jtcjPh4x_pmmkKrubhQnvC16-RYgWHTShtA@mail.gmail.com> On 21 January 2015 at 10:29, Joao S. O. Bueno <jsbueno at python.org.br> wrote: > On 21 January 2015 at 09:59, Steven D'Aprano <steve at pearwood.info> wrote: >> On Wed, Jan 21, 2015 at 09:27:39AM -0200, Joao S. O. Bueno wrote: ... >> >> (2) Fuzzy equality means that there are situations where: >> >> a == b and b == c >> >> but >> >> a != c >> >> The language APL tried to turn this into a feature: >> >> The intransitivity of [tolerant] equality is well known in practical >> situations and can be easily demonstrated by sawing several pieces >> of wood of equal length. In one case, use the first piece to measure >> subsequent lengths; in the second case, use the last piece cut to >> measure the next. Compare the lengths of the two final pieces. >> ? Richard Lathwell, APL Comparison Tolerance, APL76, 1976 > It just hit me that while this intransivity is strange, mathematically speaking, it _is_ indeed what happens in "the real world" when we are dealing with measurements - this wood piece example being more than enough to show it. Since the whole discussion about floating point closeness is to ease out algorithms that need to be concerned on how numbers behave "out there", maybe this behavior is not that far off, neither that undesirable at all. >> >> -- >> Steve >> ______________________ From jsbueno at python.org.br Wed Jan 21 13:44:39 2015 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Wed, 21 Jan 2015 10:44:39 -0200 Subject: [Python-ideas] Should bool continue to inherit from int? In-Reply-To: <20150121114728.GA18556@ando.pearwood.info> References: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> <20150121114728.GA18556@ando.pearwood.info> Message-ID: <CAH0mxTQ7e9TOfT948PmxvmaiUeoDYW_-HsAb7Pa+pb8T_0UM2A@mail.gmail.com> > Should bool continue to inherit from int? yes. From rosuav at gmail.com Wed Jan 21 14:27:05 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 22 Jan 2015 00:27:05 +1100 Subject: [Python-ideas] Should bool continue to inherit from int? In-Reply-To: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> References: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> Message-ID: <CAPTjJmqC3iDnobQna-6bCbAjPYabEsidpdTj+qzaLxmm+oQT1Q@mail.gmail.com> On Wed, Jan 21, 2015 at 10:32 PM, Michael Mitchell <epsilonmichael at gmail.com> wrote: > PEP 285 provides some justification for why arithmetic operations should be > valid on bools and why bool should inherit from int (see points (4) and (6) > respectively). Since it's been 12 years (it's possible this has been brought > up again between now and then), I thought it could be worthwhile to take > another look. > > I am mostly interested in a resurveying of the questions: > 1) Would it still be very inconvenient to implement bools separately from > ints? > 2) Do most Python users still agree that arithmetic operations should be > supported on booleans? We recently had a thread on python-list about this. There are a number of useful operations that can be done on bools-as-integers (such as summing them to quickly and efficiently figure out how many are True). But let me rephrase your question: "Python's booleans currently support arithmetic operations. Should we no longer support them?" Even leaving aside the massive backward compatibility implications (which would probably mean this kind of removal can't be done until Python 4.0 or 5.0 or who-knows-when), the philosophical question of "why should we NOT support something" needs a strongly-justified answer. Sure, it might make sense to have a non-integer type for the two special objects True and False; but what would the language gain by it? It's quite common for 1 and 0 to *be* the language's True and False values. REXX has no types, but in a boolean context, any value other than 1 or 0 triggers an error; C doesn't have a boolean type, and comparisons return 1 or 0; C++ has a boolean type, but will cheerfully and silently up-cast it to integer if you do arithmetic on it; Pike uses 1 and 0 (parallel to C), and allows any object to declare its own "truthiness" (more-or-less equivalently to Python's __bool__ method). All of them allow you to write stuff like: if (((1==1) + (1==0) + (0==0)) == 2) which will be true (if a little over-parenthesized for some people's liking). These kinds of constructs allow you to do fuzzy matching fairly efficiently - imagine the three conditions are three different search criteria, and instead of simply seeing if the sum is 2, you sort a large number of results by that sum. The more conditions match, the greater the sum. Sure, this isn't a *common* requirement, but what would the language really gain by forcing you to intify with something like "lambda b: 1 if b else 0"? ChrisA From chris.barker at noaa.gov Wed Jan 21 16:50:19 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Wed, 21 Jan 2015 07:50:19 -0800 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <20150121115934.GB18556@ando.pearwood.info> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> <m9k397$eo9$1@ger.gmane.org> <-3376261731312819627@unknownmsgid> <6ea13fe6-0831-4126-a94b-15b6bd1c83a7@googlegroups.com> <20150120104012.GU18556@ando.pearwood.info> <m9naqf$9s1$1@ger.gmane.org> <CAH0mxTQR=sPKULnBpc8ukCZg-A53ynZgBwomjjRv-M0RhDsu6Q@mail.gmail.com> <20150121115934.GB18556@ando.pearwood.info> Message-ID: <207409270096037251@unknownmsgid> > But it really sounds like you want a full interval arithmetic > class, which is a lot more than just fuzzy comparisons. Indeed -- this is way out of scope for the current conversation/PEP. Please start another thread if you think this is worth pushing forward. But as this would be a whole new Numeric-like class, the way forward would probably be to impliment it, put it up on pypi, and if it sees significant use, propose it for inclusion in the stdlib. The current proposal is for a simple, generally useful "fuzzy"comparison. It is quite specifically for use with the existing float object ( and other related types ). PEP on the way -- stay tuned. -Chris > > There are two problems with overriding == in this way: > > (1) What do you mean by "is close"? Do you mean that they are within > 0.0001 or 0.00000001 or within 10000.0? With a binary operator == there > is no easy way to specify an error tolerance, which means you're stuck > with using a language-wide default which is very unlikely to be suitable > for your application. A context manager is one solution, but that's > still awkward. Think about doing: > > a == b or x == y > > where the a, b comparison and x, y comparison have different tolerances. > > > (2) Fuzzy equality means that there are situations where: > > a == b and b == c > > but > > a != c > > The language APL tried to turn this into a feature: > > The intransitivity of [tolerant] equality is well known in practical > situations and can be easily demonstrated by sawing several pieces > of wood of equal length. In one case, use the first piece to measure > subsequent lengths; in the second case, use the last piece cut to > measure the next. Compare the lengths of the two final pieces. > ? Richard Lathwell, APL Comparison Tolerance, APL76, 1976 > > > but I'm not so sure. If we introduced a new comparison operator (perhaps > the Unicode symbol ? or the ASCII ~= ?) I'd be less concerned but I > think overriding == for fuzzy equality is a mistake. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From chris.barker at noaa.gov Wed Jan 21 17:13:58 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 21 Jan 2015 08:13:58 -0800 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <m9naqf$9s1$1@ger.gmane.org> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> <m9k397$eo9$1@ger.gmane.org> <-3376261731312819627@unknownmsgid> <6ea13fe6-0831-4126-a94b-15b6bd1c83a7@googlegroups.com> <20150120104012.GU18556@ando.pearwood.info> <m9naqf$9s1$1@ger.gmane.org> Message-ID: <CALGmxELCS9Wo1eYLB6pBu3fygjDV-TDJSz-iS-uZ_kWLtq_Aeg@mail.gmail.com> On Tue, Jan 20, 2015 at 8:43 PM, Ron Adam <ron3200 at gmail.com> wrote: > The more we talk about this, the more I'm beginning to dislike the > symmetric version. > > We are trading an explicit (first, second) relationship with an implicit > (smaller, larger) relationship. For Python's general use, I don't like > that. Sorry. :/ > I don't think there's really anything more "implicit" about it. And you could use the mean of a and b if you wanted it a bit less "implicit". But your other point is well taken, if you DO have a clearly defined "correct" value, then that is the one to scale the error on. And it's a good idea to provide that for those cases. Care to help me write the PEP ? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/e08bd2bf/attachment-0001.html> From chris.barker at noaa.gov Wed Jan 21 17:07:10 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Wed, 21 Jan 2015 08:07:10 -0800 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <CAH0mxTRgfHfrQP=jtcjPh4x_pmmkKrubhQnvC16-RYgWHTShtA@mail.gmail.com> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> <m9k397$eo9$1@ger.gmane.org> <-3376261731312819627@unknownmsgid> <6ea13fe6-0831-4126-a94b-15b6bd1c83a7@googlegroups.com> <20150120104012.GU18556@ando.pearwood.info> <m9naqf$9s1$1@ger.gmane.org> <CAH0mxTQR=sPKULnBpc8ukCZg-A53ynZgBwomjjRv-M0RhDsu6Q@mail.gmail.com> <20150121115934.GB18556@ando.pearwood.info> <CAH0mxTR3DZHnkqc2UMLojiRymi_CkGT=Y_jSTkJrjqqnMG5n0A@mail.gmail.com> <CAH0mxTRgfHfrQP=jtcjPh4x_pmmkKrubhQnvC16-RYgWHTShtA@mail.gmail.com> Message-ID: <1843083354531868365@unknownmsgid> > It just hit me that while this intransivity is strange, mathematically > speaking, it > _is_ indeed what happens in "the real world" when we are dealing with > measurements > - this wood piece example being more than enough to show it. The wood piece example is analogous to accumulated floating point error. A good carpenter knows not to do that, just as a good programmer knows not to successively add 0.1 floats and expect to get a nice round result. But measurement of error is a key issue here. For the use cases Ron brings up, that is, answering the question " is this measured value within some relative tolerance of another, known, and presumed to be exact, value?" Then yes, the asymmetric approach, where you specify which value is the "known" one, is the way to do it. But is that the use-case that's likely to be common? I think there are a heck of a lot of people that need to check if two values are close to each other for tests and whatnot, than writing python code for a quality assurance system. So what do we need for the common use case? It may not be that different, actually. You probably do often have a computed value that you are checking against a known result. In which case the asymmetric approach is clear and makes sense. Again, for what I think is the MOST common case: " I just want to know if I'm in the ballpark" it doesn't matter which is used. But we should probably call it "is_close_to" or some such, and name the parameters clearly. -Chris From yawar.amin at gmail.com Wed Jan 21 17:18:36 2015 From: yawar.amin at gmail.com (Yawar Amin) Date: Wed, 21 Jan 2015 11:18:36 -0500 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> Message-ID: <E9834011-6DBA-42EE-96D5-AFA16CA47ED9@gmail.com> Hi, On 2015-01-21, at 0:20, Matthew Rocklin <mrocklin at gmail.com> wrote: > [...] > > What do people do now? Currently I use new empty types as symbols. E.g.: class my_symbol: pass x = my_symbol if x == my_symbol: print("yes") Probably there's a nicer way to do it, but the advantage of using types is they are obviously interned. Regards, Yawar -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/56e27764/attachment.html> From boehm.matthew at gmail.com Wed Jan 21 17:38:28 2015 From: boehm.matthew at gmail.com (Matt) Date: Wed, 21 Jan 2015 11:38:28 -0500 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <m9nkjn$p3d$1@ger.gmane.org> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> Message-ID: <CAPRDBu6KLyck3bwODMJX+YupYUYUJ1jSvw-Kzjq02+A=wWH7_g@mail.gmail.com> What do people think about allowing the annotation to be a function that returns the type? Is this too complicated to understand/implement? import typing def Doc(param_doc, param_type=typing.Any): #do something with the documentation for the parameter... return param_type def print_twice(message: Doc("something to print")): print(message) print(message) def Prime(): return Doc("A prime number", int) I've done my best to read through the thread/PEP before replying, but please excuse me if I've missed something and this is a dumb question. If this is out of the question, perhaps people wishing to encode more information into function annotations could (ab)use subscripting. On Wed, Jan 21, 2015 at 2:31 AM, Stefan Behnel <stefan_ml at behnel.de> wrote: > Guido van Rossum schrieb am 19.01.2015 um 04:40: > > On Sun, Jan 18, 2015 at 12:36 PM, Stefan Behnel wrote: > >> would be nice to at least be able to explicitly express somehow that > >> subtypes should be disallowed. > > > > Can you come up with a specific proposal? Maybe we're not as far apart as > > we think. :-) > > Normally, "x: List[int]" would mean that x is either a list or a subtype of > list, where each item is an int or a subtype. To disallow subtypes, you > could have something like "exactly(List)[int]" or "List[exactly(int)]" or > "exactly(List[int])", which would then make it clear that the exact type > matters, only for the List in the first case, only for the item type in the > second, and (this is not necessarily required) for all types inside of the > "exactly()" call in the last case. The return value of "exactly()" could be > some kind of type wrapper object which marks the wrapped type construct as > allowing "no subtypes". > > This would keep the normal case of allowing subtypes everywhere normal and > simple, but would allow restricting the type to disallow subtypes in the > cases where it seems appropriate. > > Stefan > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/3d46e1fc/attachment.html> From skip.montanaro at gmail.com Wed Jan 21 17:45:01 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Wed, 21 Jan 2015 10:45:01 -0600 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAPRDBu6KLyck3bwODMJX+YupYUYUJ1jSvw-Kzjq02+A=wWH7_g@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAPRDBu6KLyck3bwODMJX+YupYUYUJ1jSvw-Kzjq02+A=wWH7_g@mail.gmail.com> Message-ID: <CANc-5UyOUqPhusf6bc4=QnUWnhCacEwf9m4W6G7xqufZhiHWxg@mail.gmail.com> On Wed, Jan 21, 2015 at 10:38 AM, Matt <boehm.matthew at gmail.com> wrote: > What do people think about allowing the annotation to be a function that > returns the type? Is this too complicated to understand/implement? I think that since type hints have to be analyzed at compile time, that execution of arbitrary Python code is a non-starter. Smarter folks than me may well see a way this could happen though. Skip From breamoreboy at yahoo.co.uk Wed Jan 21 18:18:36 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Wed, 21 Jan 2015 17:18:36 +0000 Subject: [Python-ideas] Should bool continue to inherit from int? In-Reply-To: <CAH0mxTQ7e9TOfT948PmxvmaiUeoDYW_-HsAb7Pa+pb8T_0UM2A@mail.gmail.com> References: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> <20150121114728.GA18556@ando.pearwood.info> <CAH0mxTQ7e9TOfT948PmxvmaiUeoDYW_-HsAb7Pa+pb8T_0UM2A@mail.gmail.com> Message-ID: <m9on1i$2uf$1@ger.gmane.org> On 21/01/2015 12:44, Joao S. O. Bueno wrote: >> Should bool continue to inherit from int? > yes. > Except when your (plural) definition of bool differs from mine, as was seen recently over on the main Python mailing list. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From guido at python.org Wed Jan 21 18:42:02 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Jan 2015 09:42:02 -0800 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> Message-ID: <CAP7+vJ+uQ3LfpqmME6f3EEmOxoXkJP8Nv4t73EVzxQwr0hzVkg@mail.gmail.com> Yeah, basically this was a fork in the road roughly 25 years ago that Python didn't take. (I was aware of the idea from Lisp, and decided Python could do with just strings instead.) There's no way adding something like this today will suddenly change everything else that is part of having "symbols"; the proposed feature would be lacking the integration with the internals that makes it work well in some other languages. On Wed, Jan 21, 2015 at 1:13 AM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > On Jan 20, 2015, at 23:48, Stephen Hansen <me+python at ixokai.io> wrote: > > *What do people do now?* >> or strings >> >> df.sort('name') >> > > ... > > >> strings work but feel unpleasant >> > > Why? > > What does :foo get you that 'foo' doesn't? > > One character being saved doesn't seem like a good reason to add syntax. > > > Mainly they're used to fill in gaps where a language isn't dynamic enough > to just use strings, which Python doesn't need. > > (Sometimes this is claimed to be a strength rather than a weakness. Arguably, > using auto-interned symbols instead of strings for things like attribute or > global lookup could be more efficient, but I don't think languages like > Ruby have shown any demonstrable gain. And we just had a thread from > someone who considered it a "bug" that Python allows setattr(spam, '3', 3) > when 3 isn't a valid identifier, which symbols "fix" automatically. But > even if these were real advantages, Python obviously isn't going to make > such a radical change as banning strings for dynamic access.) > > A language can also add things on top of the symbol/string distinction > (besides auto-interning)--e.g., you can add syntactic sugar for dynamic > lookup ("foo.:bar" means "getattr(foo, bar)"), or make undefined > variables have the value of their symbol so you can LBYL test them, or all > kinds of other things--but again, they're all things I don't think Python > wants or needs. > > And without a vague idea for why Python would want to do any of the things > that other languages do with their symbol-vs.-string distinctions, this > really does seem to come down to just saving one character. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/88320917/attachment.html> From python at mrabarnett.plus.com Wed Jan 21 19:12:13 2015 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 21 Jan 2015 18:12:13 +0000 Subject: [Python-ideas] Should bool continue to inherit from int? In-Reply-To: <CAPTjJmqC3iDnobQna-6bCbAjPYabEsidpdTj+qzaLxmm+oQT1Q@mail.gmail.com> References: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> <CAPTjJmqC3iDnobQna-6bCbAjPYabEsidpdTj+qzaLxmm+oQT1Q@mail.gmail.com> Message-ID: <54BFEBFD.5030201@mrabarnett.plus.com> On 2015-01-21 13:27, Chris Angelico wrote: > On Wed, Jan 21, 2015 at 10:32 PM, Michael Mitchell > <epsilonmichael at gmail.com> wrote: >> PEP 285 provides some justification for why arithmetic operations should be >> valid on bools and why bool should inherit from int (see points (4) and (6) >> respectively). Since it's been 12 years (it's possible this has been brought >> up again between now and then), I thought it could be worthwhile to take >> another look. >> >> I am mostly interested in a resurveying of the questions: >> 1) Would it still be very inconvenient to implement bools separately from >> ints? >> 2) Do most Python users still agree that arithmetic operations should be >> supported on booleans? > > We recently had a thread on python-list about this. There are a number > of useful operations that can be done on bools-as-integers (such as > summing them to quickly and efficiently figure out how many are True). > But let me rephrase your question: > > "Python's booleans currently support arithmetic operations. Should we > no longer support them?" > > Even leaving aside the massive backward compatibility implications > (which would probably mean this kind of removal can't be done until > Python 4.0 or 5.0 or who-knows-when), the philosophical question of > "why should we NOT support something" needs a strongly-justified > answer. Sure, it might make sense to have a non-integer type for the > two special objects True and False; but what would the language gain > by it? > > It's quite common for 1 and 0 to *be* the language's True and False > values. REXX has no types, but in a boolean context, any value other > than 1 or 0 triggers an error; C doesn't have a boolean type, and > comparisons return 1 or 0; C++ has a boolean type, but will cheerfully > and silently up-cast it to integer if you do arithmetic on it; Pike > uses 1 and 0 (parallel to C), and allows any object to declare its own > "truthiness" (more-or-less equivalently to Python's __bool__ method). > All of them allow you to write stuff like: > > if (((1==1) + (1==0) + (0==0)) == 2) > > which will be true (if a little over-parenthesized for some people's > liking). These kinds of constructs allow you to do fuzzy matching > fairly efficiently - imagine the three conditions are three different > search criteria, and instead of simply seeing if the sum is 2, you > sort a large number of results by that sum. The more conditions match, > the greater the sum. Sure, this isn't a *common* requirement, but what > would the language really gain by forcing you to intify with something > like "lambda b: 1 if b else 0"? > Pascal has 'ord' for getting the ordinal number from an enumerated type, so perhaps that could be used: ord(False) == 0 ord(True) == 1 Currently it's just the inverse of 'chr'. From ethan at stoneleaf.us Wed Jan 21 19:20:48 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 21 Jan 2015 10:20:48 -0800 Subject: [Python-ideas] Should bool continue to inherit from int? In-Reply-To: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> References: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> Message-ID: <54BFEE00.1060501@stoneleaf.us> On 01/21/2015 03:32 AM, Michael Mitchell wrote: > 1) Would it still be very inconvenient to implement bools separately from ints? Extremely. Lots of code relies on the int properties of bool. > 2) Do most Python users still agree that arithmetic operations should be supported on booleans? I certainly do. It might be nice from an ivory-tower perspective to keep bools separate from ints, but in a Practicality Beats Purity world it is quite handy. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/d932090a/attachment.sig> From boehm.matthew at gmail.com Wed Jan 21 19:27:45 2015 From: boehm.matthew at gmail.com (Matt) Date: Wed, 21 Jan 2015 13:27:45 -0500 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CANc-5UyOUqPhusf6bc4=QnUWnhCacEwf9m4W6G7xqufZhiHWxg@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAPRDBu6KLyck3bwODMJX+YupYUYUJ1jSvw-Kzjq02+A=wWH7_g@mail.gmail.com> <CANc-5UyOUqPhusf6bc4=QnUWnhCacEwf9m4W6G7xqufZhiHWxg@mail.gmail.com> Message-ID: <CAPRDBu41FK2gfVN7HeWnebhbux50_g0A=k_q9NWKNbtSzvySAg@mail.gmail.com> > > I think that since type hints have to be analyzed at compile time, > that execution of arbitrary Python code is a non-starter. >From the way the abstract is worded, it sounds like arbitrary expressions could be used: > Abstract base classes, types available in the ``types`` module, and > user-defined classes may be used as type hints as well. Annotations > must be valid expressions that evaluate without raising exceptions at > the time the function is defined I had assumed that this means that the annotation is any arbitrary python expression that evaluates to one of these things. Even if the syntax is more limited (as I guess it must be), I assume that Union and generics work not because the static analyzers know anything special about Union, but that typing.Union is a python class and could have just as easily been defined outside of the standard library. Even if function calls do not work, I imagine people can somehow abuse generics to capture documentation or other things inside type hints. As an example, let's say a function expects a positive integer. A programmer could achieve this with an assert statement near the top of the function, a decorator, or by making users pass in a PositiveInt object (subclass of int). However, they may prefer to annotate the function as Type[int, Positive]. Perhaps a better example would be something like Type[FilePath, exists, readable] I like the idea of keeping annotation syntax very simple and primarily about static type checking, but if it's true that people can experiment with other uses for function annotations without violating the current syntax, then it sounds like everybody wins. On Wed, Jan 21, 2015 at 11:45 AM, Skip Montanaro <skip.montanaro at gmail.com> wrote: > On Wed, Jan 21, 2015 at 10:38 AM, Matt <boehm.matthew at gmail.com> wrote: > > What do people think about allowing the annotation to be a function that > > returns the type? Is this too complicated to understand/implement? > > I think that since type hints have to be analyzed at compile time, > that execution of arbitrary Python code is a non-starter. Smarter > folks than me may well see a way this could happen though. > > Skip > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/908433bb/attachment-0001.html> From guido at python.org Wed Jan 21 20:11:03 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Jan 2015 11:11:03 -0800 Subject: [Python-ideas] Should bool continue to inherit from int? In-Reply-To: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> References: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> Message-ID: <CAP7+vJKen6W4yopMZPXNM9k9pm5M2OZKQtUugFxwb2JAVa4QsQ@mail.gmail.com> On Wed, Jan 21, 2015 at 3:32 AM, Michael Mitchell <epsilonmichael at gmail.com> wrote: > PEP 285 <http://legacy.python.org/dev/peps/pep-0285> provides some > justification for why arithmetic operations should be valid on bools and > why bool should inherit from int (see points (4) and (6) respectively). > Since it's been 12 years (it's possible this has been brought up again > between now and then), I thought it could be worthwhile to take another > look. > > I am mostly interested in a resurveying of the questions: > 1) Would it still be very inconvenient to implement bools separately from > ints? > 2) Do most Python users still agree that arithmetic operations should be > supported on booleans? > It would be so massively backward compatible that it's not worth thinking about. > Follow-up thought: > Something I noticed is that with PEP 484 > <https://www.python.org/dev/peps/pep-0484/> (Type Hints) specified as is, > there would be no way to statically verify that a function will only > operate on ints and not bools. An example would be a function that can only > operate on integer values in a JSON dict created by the builtin `json` > module (using the default decoder) cannot exist, as that function could > operate on the boolean values of the dict. > Like Steven, I can't completely follow this thought (it looks like you were writing too fast and jumbled a few verbs or negations) but you bring up an interesting point: even though bools are ints, their JSON encodings are different. So I believe you are saying, suppose I have a function that takes an integer argument and then JSON-encodes it and sends it to some server that requires a JSON int. Now suppose you are adding type hints to your code, and you add ": int" to the parameter under discussion. And now suppose you have a faulty caller which calls this function with the argument set to True. This program will type-check correctly, because True is a bool which is a subclass of int, but it will run incorrectly, because (under this assumption) True will be converted to a JSON true value, which the server rejects. TBH I have to think about this more. I think that in the initial version of PEP 484 we won't try to address this (JSON without something that enforces a schema is too dynamic to fit well into any type system), but I've opened an issue to have a further discussion about this: https://github.com/ambv/typehinting/issues/52 -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/1ed74b60/attachment.html> From alexander.belopolsky at gmail.com Wed Jan 21 20:11:36 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 21 Jan 2015 14:11:36 -0500 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <CAP7+vJ+uQ3LfpqmME6f3EEmOxoXkJP8Nv4t73EVzxQwr0hzVkg@mail.gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> <CAP7+vJ+uQ3LfpqmME6f3EEmOxoXkJP8Nv4t73EVzxQwr0hzVkg@mail.gmail.com> Message-ID: <CAP7h-xbb9pCsWsRnCHWRLk2Ukm5p3PmUPW+r=AFvtbLpx17Fuw@mail.gmail.com> On Wed, Jan 21, 2015 at 12:42 PM, Guido van Rossum <guido at python.org> wrote: > There's no way adding something like this today will suddenly change > everything else that is part of having "symbols"; the proposed feature > would be lacking the integration with the internals that makes it work well > in some other languages. Wouldn't it be possible to simply have def Symbol(str): def __new__(cls_, *args): istr = sys.intern(str(*args)) return str.__new__(cls, istr) and have sys.intern() return Symbol instances at some point? I don't see any obvious compatibility problems. How would this be different from adding bool as a subclass of int some time ago? BTW, I think Pandas found [1] the way to have memory-saving string vectors without having a symbol type in the core. [1] http://pandas-docs.github.io/pandas-docs-travis/categorical.html -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/e1f57abe/attachment.html> From guido at python.org Wed Jan 21 20:13:38 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Jan 2015 11:13:38 -0800 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <CAP7h-xbb9pCsWsRnCHWRLk2Ukm5p3PmUPW+r=AFvtbLpx17Fuw@mail.gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> <CAP7+vJ+uQ3LfpqmME6f3EEmOxoXkJP8Nv4t73EVzxQwr0hzVkg@mail.gmail.com> <CAP7h-xbb9pCsWsRnCHWRLk2Ukm5p3PmUPW+r=AFvtbLpx17Fuw@mail.gmail.com> Message-ID: <CAP7+vJK_KqqU8mbuR+14L4dC5M2YmxROhD-UvR=8tJRheq-=HQ@mail.gmail.com> On Wed, Jan 21, 2015 at 11:11 AM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Wed, Jan 21, 2015 at 12:42 PM, Guido van Rossum <guido at python.org> > wrote: > >> There's no way adding something like this today will suddenly change >> everything else that is part of having "symbols"; the proposed feature >> would be lacking the integration with the internals that makes it work well >> in some other languages. > > > Wouldn't it be possible to simply have > > def Symbol(str): > def __new__(cls_, *args): > istr = sys.intern(str(*args)) > return str.__new__(cls, istr) > > and have sys.intern() return Symbol instances at some point? I don't see > any obvious compatibility problems. > I don't see any obvious advantages either. > How would this be different from adding bool as a subclass of int some > time ago? > Lack of a use case? > BTW, I think Pandas found [1] the way to have memory-saving string vectors > without having a symbol type in the core. > > [1] http://pandas-docs.github.io/pandas-docs-travis/categorical.html > Great. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/4f76e07a/attachment.html> From alexander.belopolsky at gmail.com Wed Jan 21 20:18:04 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 21 Jan 2015 14:18:04 -0500 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <CAP7h-xbb9pCsWsRnCHWRLk2Ukm5p3PmUPW+r=AFvtbLpx17Fuw@mail.gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> <CAP7+vJ+uQ3LfpqmME6f3EEmOxoXkJP8Nv4t73EVzxQwr0hzVkg@mail.gmail.com> <CAP7h-xbb9pCsWsRnCHWRLk2Ukm5p3PmUPW+r=AFvtbLpx17Fuw@mail.gmail.com> Message-ID: <CAP7h-xaKyYA-c7WO3ND2j1bYuAerV=EDtS0Y1092y2so16DpsQ@mail.gmail.com> On Wed, Jan 21, 2015 at 2:11 PM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > Wouldn't it be possible to simply have > > def Symbol(str): > Hmm, I should have tested the code before posting. That should have been a class and the logic would not work without modifying sys.intern first, but it does not kill the idea. Why can't we make interned strings a subclass of strings like bool is a subclass of int? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/fd8ba8a2/attachment-0001.html> From guido at python.org Wed Jan 21 20:22:12 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Jan 2015 11:22:12 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAPRDBu41FK2gfVN7HeWnebhbux50_g0A=k_q9NWKNbtSzvySAg@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAPRDBu6KLyck3bwODMJX+YupYUYUJ1jSvw-Kzjq02+A=wWH7_g@mail.gmail.com> <CANc-5UyOUqPhusf6bc4=QnUWnhCacEwf9m4W6G7xqufZhiHWxg@mail.gmail.com> <CAPRDBu41FK2gfVN7HeWnebhbux50_g0A=k_q9NWKNbtSzvySAg@mail.gmail.com> Message-ID: <CAP7+vJL2Tbc5dr1q677TwvnEGE_PCLd2uw-nuOTS=Yt0=ZjS6Q@mail.gmail.com> I need to cut this subthread short, but basically, for a successful type check, the expressions used in annotations *cannot* be arbitrary expressions. The type checker has a special understanding of things defined int 'typing', and it knows e.g. what typing.Union means. It supports simple aliases through assignments, so that you can say e.g. import typing U = typing.Union def foo(U[int, str]) -> str: ... but it does not understand arbitrary call chaines, so that e.g. this does *not* work: import typing def u(a, b): return typing.Union[a, b] def foo(u(int, str)) -> str: ... On Wed, Jan 21, 2015 at 10:27 AM, Matt <boehm.matthew at gmail.com> wrote: > I think that since type hints have to be analyzed at compile time, >> that execution of arbitrary Python code is a non-starter. > > > From the way the abstract is worded, it sounds like arbitrary expressions > could be used: > > >> Abstract base classes, types available in the ``types`` module, and >> user-defined classes may be used as type hints as well. Annotations >> must be valid expressions that evaluate without raising exceptions at >> the time the function is defined > > > I had assumed that this means that the annotation is any arbitrary python > expression that evaluates to one of these things. Even if the syntax is > more limited (as I guess it must be), I assume that Union and generics work > not because the static analyzers know anything special about Union, but > that typing.Union is a python class and could have just as easily been > defined outside of the standard library. Even if function calls do not > work, I imagine people can somehow abuse generics to capture documentation > or other things inside type hints. > > As an example, let's say a function expects a positive integer. A > programmer could achieve this with an assert statement near the top of the > function, a decorator, or by making users pass in a PositiveInt object > (subclass of int). However, they may prefer to annotate the function as > Type[int, Positive]. Perhaps a better example would be something like > Type[FilePath, exists, readable] > > I like the idea of keeping annotation syntax very simple and primarily > about static type checking, but if it's true that people can experiment > with other uses for function annotations without violating the current > syntax, then it sounds like everybody wins. > > > On Wed, Jan 21, 2015 at 11:45 AM, Skip Montanaro <skip.montanaro at gmail.com > > wrote: > >> On Wed, Jan 21, 2015 at 10:38 AM, Matt <boehm.matthew at gmail.com> wrote: >> > What do people think about allowing the annotation to be a function that >> > returns the type? Is this too complicated to understand/implement? >> >> I think that since type hints have to be analyzed at compile time, >> that execution of arbitrary Python code is a non-starter. Smarter >> folks than me may well see a way this could happen though. >> >> Skip >> > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/d7f6b766/attachment.html> From guido at python.org Wed Jan 21 20:23:09 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Jan 2015 11:23:09 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <m9nkjn$p3d$1@ger.gmane.org> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> Message-ID: <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> Let's discuss this idea of "exactly" further in the tracker, where I just created an issue that touches on the same idea: https://github.com/ambv/typehinting/issues/52 On Tue, Jan 20, 2015 at 11:31 PM, Stefan Behnel <stefan_ml at behnel.de> wrote: > Guido van Rossum schrieb am 19.01.2015 um 04:40: > > On Sun, Jan 18, 2015 at 12:36 PM, Stefan Behnel wrote: > >> would be nice to at least be able to explicitly express somehow that > >> subtypes should be disallowed. > > > > Can you come up with a specific proposal? Maybe we're not as far apart as > > we think. :-) > > Normally, "x: List[int]" would mean that x is either a list or a subtype of > list, where each item is an int or a subtype. To disallow subtypes, you > could have something like "exactly(List)[int]" or "List[exactly(int)]" or > "exactly(List[int])", which would then make it clear that the exact type > matters, only for the List in the first case, only for the item type in the > second, and (this is not necessarily required) for all types inside of the > "exactly()" call in the last case. The return value of "exactly()" could be > some kind of type wrapper object which marks the wrapped type construct as > allowing "no subtypes". > > This would keep the normal case of allowing subtypes everywhere normal and > simple, but would allow restricting the type to disallow subtypes in the > cases where it seems appropriate. > > Stefan > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/08c3f501/attachment.html> From alexander.belopolsky at gmail.com Wed Jan 21 20:38:42 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 21 Jan 2015 14:38:42 -0500 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <CAP7+vJK_KqqU8mbuR+14L4dC5M2YmxROhD-UvR=8tJRheq-=HQ@mail.gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> <CAP7+vJ+uQ3LfpqmME6f3EEmOxoXkJP8Nv4t73EVzxQwr0hzVkg@mail.gmail.com> <CAP7h-xbb9pCsWsRnCHWRLk2Ukm5p3PmUPW+r=AFvtbLpx17Fuw@mail.gmail.com> <CAP7+vJK_KqqU8mbuR+14L4dC5M2YmxROhD-UvR=8tJRheq-=HQ@mail.gmail.com> Message-ID: <CAP7h-xYCVx453=o0mPsDFpT+8=_zKCCb4S0yBW=5aOK-7RFxhg@mail.gmail.com> On Wed, Jan 21, 2015 at 2:13 PM, Guido van Rossum <guido at python.org> wrote: > Lack of a use case? My use case comes from interoperability with other languages. I maintain a package which provides an interface [1] between Python and the language [2] called Q. Q has 3 scalar types to represent text data (char, byte and symbol) and the corresponding vector types. When I pass Python strings to Q, I convert them to symbols because that is what users expect most of the time, but as a result, working with char vectors is unnatural. I currently use bytes type for that, but it is really an abuse because bytes should map to byte vectors. I realize that mine is a niche application, but the same problem is likely to arise with other data systems that have "symbol" or categorical type. [1] http://code.kx.com/wiki/Contrib/PyQ [2] http://kx.com/q/d/q.htm -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/7180a118/attachment-0001.html> From alexander.belopolsky at gmail.com Wed Jan 21 20:49:20 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 21 Jan 2015 14:49:20 -0500 Subject: [Python-ideas] Should bool continue to inherit from int? In-Reply-To: <CAP7+vJKen6W4yopMZPXNM9k9pm5M2OZKQtUugFxwb2JAVa4QsQ@mail.gmail.com> References: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> <CAP7+vJKen6W4yopMZPXNM9k9pm5M2OZKQtUugFxwb2JAVa4QsQ@mail.gmail.com> Message-ID: <CAP7h-xapgY1SLqmNhhT4p2E46qib1jpSRzrMtaOGBQs+EhXqDQ@mail.gmail.com> On Wed, Jan 21, 2015 at 2:11 PM, Guido van Rossum <guido at python.org> wrote: > 2) Do most Python users still agree that arithmetic operations should be >> supported on booleans? >> > > It would be so massively backward compatible that it's not worth thinking > about. > The only time I encountered a situation where True + True was not 2 was in numpy: >>> numpy.bool_(True) + True True That was *very* inconvenient! -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/b919eb29/attachment.html> From chris.barker at noaa.gov Wed Jan 21 21:12:10 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 21 Jan 2015 12:12:10 -0800 Subject: [Python-ideas] Should bool continue to inherit from int? In-Reply-To: <CAP7+vJKen6W4yopMZPXNM9k9pm5M2OZKQtUugFxwb2JAVa4QsQ@mail.gmail.com> References: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> <CAP7+vJKen6W4yopMZPXNM9k9pm5M2OZKQtUugFxwb2JAVa4QsQ@mail.gmail.com> Message-ID: <CALGmxEKsjj4m_0LF72Tu+CAvafO1O5rUfxyLXvU94bwyNw1Ptw@mail.gmail.com> On Wed, Jan 21, 2015 at 11:11 AM, Guido van Rossum <guido at python.org> wrote: > So I believe you are saying, suppose I have a function that takes an > integer argument and then JSON-encodes it and sends it to some server that > requires a JSON int. Now suppose you are adding type hints to your code, > and you add ": int" to the parameter under discussion. And now suppose you > have a faulty caller which calls this function with the argument set to > True. This program will type-check correctly, because True is a bool which > is a subclass of int, but it will run incorrectly, because (under this > assumption) True will be converted to a JSON true value, which the server > rejects. > This sounds like it may benefit from the issue under discussion for Cython -- a way to specify an int only, and not allow subclasses. Though I suppose in this case, you may want to allow other subclasses, just not that one. This hypothetical code could also to a cast/conversion to int anyway, which is pretty much how you'd do it with duck typing and no type checking. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/5fb976f8/attachment.html> From guido at python.org Wed Jan 21 21:16:02 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Jan 2015 12:16:02 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> Message-ID: <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> On Tue, Jan 20, 2015 at 9:09 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote: > Andrew Barnert wrote: > >> But I think there's a hole in the premise. Python's key strength is good >> debug-ability and amazingly powerful introspection. (I mean Python's _two_ >> key strengths are...) Information that has no runtime meaning to the >> interpreter may still have meaning to the user who's debugging code, >> > > There's nothing wrong with making the annotations *available* > at run time. All I'm saying is that *evaluating* them at run > time hinders rather than helping for the intended use. That's debatable. While evaluating them at run time causes some issues with forward references (mostly for container-ish class definitions), it also catches many mistakes early, such as not importing the referenced classes. Surely if you are reference a class that's not a built-in you have to import it before you can reference in a type hint -- otherwise how would the type checker know where to look for the class definition? FWIW if you really don't want to have them evaluated at run time, you can just put all type hints in string quotes. mypy will treat them as forward references, but AFAIC it doesn't insist that a forward reference contains at least one symbol that is not defined yet at the place where it occurs. :-) I doubt that this style has many things to recommend it though (see previous paragraph). Let me also use this response to explain why I think "::" is a bad idea: - First and foremost, Python usually (an d intentionally!) uses punctuation in a way derived from or similar to long-standing traditions for written text and basic math (with a few going back to old programming conventions, like using * for multiplication). The single colon used for type annotations is an example of this: it is analogous to the way the colon is often used in prose. Switching to a double colon would make it a magical operator -- especially since a single colon in the same position would have a similar but different meaning. Decorator syntax notwithstanding, magical operators are and should remain rare in Python. Eight years ago we picked the winning syntax for argument annotations. It would be tragic if we now were to replace it with this syntactic hack. - I want the type checker to work for earlier versions of Python 3 as well, so that library maintainers (at least for code that doesn't need to support Python 2) can start including type hints in their code without locking that code in to Python 3.5 -- they can use a backport of the (run time) typing package from PyPI, and the type checker can handle earlier Python versions just fine. (All of this already works for mypy.) - What would you do for function return type hints? "-->" or "->>" or "->:" or "->->" all look completely arbitrary. - Looking ahead, if type hints are successful we might want to add a similar notation to declare the type of variables or class attributes, for example "age: int = 99" or perhaps "var age: int = 99". But if we were to use "::" for type hints in function definitions, for consistency we would have to use "age:: int = 99", which would just perpetuate the syntactic hack. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/5848b4d1/attachment.html> From epsilonmichael at gmail.com Wed Jan 21 21:28:40 2015 From: epsilonmichael at gmail.com (Michael Mitchell) Date: Wed, 21 Jan 2015 12:28:40 -0800 Subject: [Python-ideas] Should bool continue to inherit from int? In-Reply-To: <CAP7+vJKen6W4yopMZPXNM9k9pm5M2OZKQtUugFxwb2JAVa4QsQ@mail.gmail.com> References: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> <CAP7+vJKen6W4yopMZPXNM9k9pm5M2OZKQtUugFxwb2JAVa4QsQ@mail.gmail.com> Message-ID: <CABvhCyGMXELMSJzmuDZg6MOGcQ3aJH_nJxhv6uzDVZTgrPKyhQ@mail.gmail.com> On Wed, Jan 21, 2015 at 11:11 AM, Guido van Rossum <guido at python.org> wrote: > So I believe you are saying, suppose I have a function that takes an > integer argument and then JSON-encodes it and sends it to some server that > requires a JSON int. Now suppose you are adding type hints to your code, > and you add ": int" to the parameter under discussion. And now suppose you > have a faulty caller which calls this function with the argument set to > True. This program will type-check correctly, because True is a bool which > is a subclass of int, but it will run incorrectly, because (under this > assumption) True will be converted to a JSON true value, which the server > rejects. This is much better phrased than what I had written, thanks :) I'll be watching the Github issue. Thank you to everyone that replied. My impression is that the rationale in PEP 285 still holds strong. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/c5cd496e/attachment-0001.html> From alexander.belopolsky at gmail.com Wed Jan 21 21:40:39 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 21 Jan 2015 15:40:39 -0500 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <CALGmxE+LGyJ0jEZYK1JjkEV7opWq5zWsSGNVz8+Ea5OuBW_jtA@mail.gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> <CAP7+vJ+uQ3LfpqmME6f3EEmOxoXkJP8Nv4t73EVzxQwr0hzVkg@mail.gmail.com> <CAP7h-xbb9pCsWsRnCHWRLk2Ukm5p3PmUPW+r=AFvtbLpx17Fuw@mail.gmail.com> <CAP7+vJK_KqqU8mbuR+14L4dC5M2YmxROhD-UvR=8tJRheq-=HQ@mail.gmail.com> <CAP7h-xYCVx453=o0mPsDFpT+8=_zKCCb4S0yBW=5aOK-7RFxhg@mail.gmail.com> <CALGmxE+LGyJ0jEZYK1JjkEV7opWq5zWsSGNVz8+Ea5OuBW_jtA@mail.gmail.com> Message-ID: <CAP7h-xa26zH9wa6m6yST227q65XYHvTGd-c=S=G9Ho5dLFrq_g@mail.gmail.com> On Wed, Jan 21, 2015 at 3:07 PM, Chris Barker <chris.barker at noaa.gov> wrote: > couldn't you create a subclass of str that was exactly the same except > that it was defined as mapping to a Q symbol? I could, but having a dedicated syntax would make things nicer. Here is what I have: ## Pass some python data to q q.symbol = 'abc' q.symbols = ['abc', 'def'] q.char = q('"a"') # no python syntax q.chars = b'abc' q.vector_of_chars = [b'abc', b'def'] q.byte = q('0x01') # no python syntax .. here is what I would rather have q.symbol = `abc q.symbols = [`abc, `def] q.char = q('"a"') # no python syntax q.chars = 'abc' q.vector_of_chars = ['abc', 'def'] q.byte = q('0x01') # no python syntax q.bytes = [b'xyz'] # or bytes([0x78, 0x79, 0x7a]) .. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/203b53c8/attachment.html> From greg.ewing at canterbury.ac.nz Wed Jan 21 21:57:41 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Jan 2015 09:57:41 +1300 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <20150121114038.GZ18556@ando.pearwood.info> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> <20150121114038.GZ18556@ando.pearwood.info> Message-ID: <54C012C5.6040101@canterbury.ac.nz> Steven D'Aprano wrote: > On Tue, Jan 20, 2015 at 11:48:58PM -0800, Stephen Hansen wrote: > >>What does :foo get you that 'foo' doesn't? > > It gets you an atomic data type which isn't a string and isn't an int. If you want that, you're better off creating your own class. Otherwise, there's nothing to distinguish symbols used for one purpose from symbols used for another, which is the same problem recurring at another level. > Python already has a number of symbol-like objects in the language: > > None > True and False > NotImplemented > Ellipsis And it's not hard to create more, especially now that we have Enums as a provided battery. -- Greg From rosuav at gmail.com Wed Jan 21 21:58:18 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 22 Jan 2015 07:58:18 +1100 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <CAP7h-xaKyYA-c7WO3ND2j1bYuAerV=EDtS0Y1092y2so16DpsQ@mail.gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> <CAP7+vJ+uQ3LfpqmME6f3EEmOxoXkJP8Nv4t73EVzxQwr0hzVkg@mail.gmail.com> <CAP7h-xbb9pCsWsRnCHWRLk2Ukm5p3PmUPW+r=AFvtbLpx17Fuw@mail.gmail.com> <CAP7h-xaKyYA-c7WO3ND2j1bYuAerV=EDtS0Y1092y2so16DpsQ@mail.gmail.com> Message-ID: <CAPTjJmoG_DYLXqxJdt8FAfkzaSVcyYRpPb2X6826htrA_zjrLg@mail.gmail.com> On Thu, Jan 22, 2015 at 6:18 AM, Alexander Belopolsky <alexander.belopolsky at gmail.com> wrote: > Why can't we make interned strings a subclass of strings like bool is a > subclass of int? My gut feeling is that an interned string is exactly the same thing as a non-interned string, and that it should be possible for sys.intern() to return the original string unchanged. But on the flip side, if you *know* that two strings are interned, you can compare their identities instead of their values; currently, identical interned strings compare equal quickly, but distinct ones still have to be compared character-for-character: >>> timeit.repeat("x==y","x='a'*65536+'q'; y='a'*65536+'q'") [2.535494999960065, 2.2198410001583397, 2.2210810002870858] >>> timeit.repeat("x==y","import sys; x=sys.intern('a'*65536+'q'); y=sys.intern('a'*65536+'q')") [0.056215000338852406, 0.05387900024652481, 0.04834900004789233] >>> timeit.repeat("x==y","x='a'*65536+'x'; y='a'*65536+'y'") [2.53101500030607, 2.2251750002615154, 2.228022000286728] >>> timeit.repeat("x==y","import sys; x=sys.intern('a'*65536+'x'); y=sys.intern('a'*65536+'y')") [2.583013000432402, 2.2194420001469553, 2.224679999984801] If, in the fourth example, the interpreter knew that both those string objects were interned, it could know that they can't possibly be the same string. So is that distinction worth coding into the class hierarchy? ChrisA From chris.barker at noaa.gov Wed Jan 21 21:07:14 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 21 Jan 2015 12:07:14 -0800 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <CAP7h-xYCVx453=o0mPsDFpT+8=_zKCCb4S0yBW=5aOK-7RFxhg@mail.gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> <CAP7+vJ+uQ3LfpqmME6f3EEmOxoXkJP8Nv4t73EVzxQwr0hzVkg@mail.gmail.com> <CAP7h-xbb9pCsWsRnCHWRLk2Ukm5p3PmUPW+r=AFvtbLpx17Fuw@mail.gmail.com> <CAP7+vJK_KqqU8mbuR+14L4dC5M2YmxROhD-UvR=8tJRheq-=HQ@mail.gmail.com> <CAP7h-xYCVx453=o0mPsDFpT+8=_zKCCb4S0yBW=5aOK-7RFxhg@mail.gmail.com> Message-ID: <CALGmxE+LGyJ0jEZYK1JjkEV7opWq5zWsSGNVz8+Ea5OuBW_jtA@mail.gmail.com> On Wed, Jan 21, 2015 at 11:38 AM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Wed, Jan 21, 2015 at 2:13 PM, Guido van Rossum <guido at python.org> > wrote: > >> Lack of a use case? > > > My use case comes from interoperability with other languages. I maintain > a package which provides an interface [1] between Python and the language > [2] called Q. Q has 3 scalar types to represent text data (char, byte and > symbol) and the corresponding vector types. When I pass Python strings to > Q, I convert them to symbols because that is what users expect most of the > time, but as a result, working with char vectors is unnatural. > couldn't you create a subclass of str that was exactly the same except that it was defined as mapping to a Q symbol? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/a20ac36c/attachment.html> From greg.ewing at canterbury.ac.nz Wed Jan 21 22:09:44 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Jan 2015 10:09:44 +1300 Subject: [Python-ideas] Should bool continue to inherit from int? In-Reply-To: <CAPTjJmqC3iDnobQna-6bCbAjPYabEsidpdTj+qzaLxmm+oQT1Q@mail.gmail.com> References: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> <CAPTjJmqC3iDnobQna-6bCbAjPYabEsidpdTj+qzaLxmm+oQT1Q@mail.gmail.com> Message-ID: <54C01598.3000303@canterbury.ac.nz> Chris Angelico wrote: > All of them allow you to write stuff like: > > if (((1==1) + (1==0) + (0==0)) == 2) Packing a number of booleans into an int is another case where bool-int equivalence comes in handy. I've found myself annoyed by this in Java on more than one occasion recently. (Java neither allows bools to be used as ints nor non-bools to be used directly as truth values.) -- Greg From ckaynor at zindagigames.com Wed Jan 21 22:17:18 2015 From: ckaynor at zindagigames.com (Chris Kaynor) Date: Wed, 21 Jan 2015 13:17:18 -0800 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <CAPTjJmoG_DYLXqxJdt8FAfkzaSVcyYRpPb2X6826htrA_zjrLg@mail.gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> <CAP7+vJ+uQ3LfpqmME6f3EEmOxoXkJP8Nv4t73EVzxQwr0hzVkg@mail.gmail.com> <CAP7h-xbb9pCsWsRnCHWRLk2Ukm5p3PmUPW+r=AFvtbLpx17Fuw@mail.gmail.com> <CAP7h-xaKyYA-c7WO3ND2j1bYuAerV=EDtS0Y1092y2so16DpsQ@mail.gmail.com> <CAPTjJmoG_DYLXqxJdt8FAfkzaSVcyYRpPb2X6826htrA_zjrLg@mail.gmail.com> Message-ID: <CALvWhxtF3aqKR-8FBNZpYLUfpzSENd6KA-HqvH-8oBAxEnsNnQ@mail.gmail.com> On Wed, Jan 21, 2015 at 12:58 PM, Chris Angelico <rosuav at gmail.com> wrote: > On Thu, Jan 22, 2015 at 6:18 AM, Alexander Belopolsky > <alexander.belopolsky at gmail.com> wrote: > > Why can't we make interned strings a subclass of strings like bool is a > > subclass of int? > > My gut feeling is that an interned string is exactly the same thing as > a non-interned string, and that it should be possible for sys.intern() > to return the original string unchanged. I would tend to agree that there is no real difference between an interned and non-interned string in Python. If Python had mutable strings as the base type, that would be different; then having an immutable version of string would be helpful for various reasons. > But on the flip side, if you > *know* that two strings are interned, you can compare their identities > instead of their values; currently, identical interned strings compare > equal quickly, but distinct ones still have to be compared > character-for-character: > > >>> timeit.repeat("x==y","x='a'*65536+'q'; y='a'*65536+'q'") > [2.535494999960065, 2.2198410001583397, 2.2210810002870858] > >>> timeit.repeat("x==y","import sys; x=sys.intern('a'*65536+'q'); y=sys.intern('a'*65536+'q')") > [0.056215000338852406, 0.05387900024652481, 0.04834900004789233] > >>> timeit.repeat("x==y","x='a'*65536+'x'; y='a'*65536+'y'") > [2.53101500030607, 2.2251750002615154, 2.228022000286728] > >>> timeit.repeat("x==y","import sys; x=sys.intern('a'*65536+'x'); y=sys.intern('a'*65536+'y')") > [2.583013000432402, 2.2194420001469553, 2.224679999984801] > > If, in the fourth example, the interpreter knew that both those string > objects were interned, it could know that they can't possibly be the > same string. So is that distinction worth coding into the class > hierarchy? If such an optimization were desirable, it would be better implemented as a flag that is accessible on the string object, but would only be writable internally (no public API for setting*). This would then allow the string compare to add a check to see if both are interned after checking the pointer values and early exit with a false result. There is not real need (that I know of) to have a class distinction to indicate interning vs a non-mutable string. Chris K *Note that hacks similar to the ones that can be used to change the value of small integers in Python may still function. Depending on internal implementation, it could also be viable to instead have the flag actually perform a lookup into the intern table, but that may negative (much of) the performance benefits. From tjreedy at udel.edu Wed Jan 21 22:29:10 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 21 Jan 2015 16:29:10 -0500 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <20150121115934.GB18556@ando.pearwood.info> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> <m9k397$eo9$1@ger.gmane.org> <-3376261731312819627@unknownmsgid> <6ea13fe6-0831-4126-a94b-15b6bd1c83a7@googlegroups.com> <20150120104012.GU18556@ando.pearwood.info> <m9naqf$9s1$1@ger.gmane.org> <CAH0mxTQR=sPKULnBpc8ukCZg-A53ynZgBwomjjRv-M0RhDsu6Q@mail.gmail.com> <20150121115934.GB18556@ando.pearwood.info> Message-ID: <m9p5n7$10g$1@ger.gmane.org> On 1/21/2015 6:59 AM, Steven D'Aprano wrote: > On Wed, Jan 21, 2015 at 09:27:39AM -0200, Joao S. O. Bueno wrote: > >> 1) Implement a Number class that behaves as a float number, and does >> the fuzzy comparisons automatically. > > -1 on this. > > (2) Fuzzy equality means that there are situations where: > > a == b and b == c > > but > > a != c > > The language APL tried to turn this into a feature: > > The intransitivity of [tolerant] equality is well known in practical > situations and can be easily demonstrated by sawing several pieces > of wood of equal length. In one case, use the first piece to measure > subsequent lengths; in the second case, use the last piece cut to > measure the next. Compare the lengths of the two final pieces. > ? Richard Lathwell, APL Comparison Tolerance, APL76, 1976 > > > but I'm not so sure. If we introduced a new comparison operator (perhaps > the Unicode symbol ? or the ASCII ~= ?) I'd be less concerned but I > think overriding == for fuzzy equality is a mistake. The proper operation of sets (and dicts) depends on == being transitive. Letting D0 = Decimal(0), we once had 0.0 == 0 and 0 == D0 but 0.0 != D0 (now fixed) and the result was obscure bugs. One of the design criteria for, I believe, Enums was that == be transitive. -- Terry Jan Reedy From njs at pobox.com Wed Jan 21 22:36:25 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 21 Jan 2015 21:36:25 +0000 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <CAH0mxTQR=sPKULnBpc8ukCZg-A53ynZgBwomjjRv-M0RhDsu6Q@mail.gmail.com> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> <m9k397$eo9$1@ger.gmane.org> <-3376261731312819627@unknownmsgid> <6ea13fe6-0831-4126-a94b-15b6bd1c83a7@googlegroups.com> <20150120104012.GU18556@ando.pearwood.info> <m9naqf$9s1$1@ger.gmane.org> <CAH0mxTQR=sPKULnBpc8ukCZg-A53ynZgBwomjjRv-M0RhDsu6Q@mail.gmail.com> Message-ID: <CAPJVwBn1E0hvyJx41XNaXZde4QrC9+J189igT4W5vy3N_xMmkA@mail.gmail.com> On Wed, Jan 21, 2015 at 11:27 AM, Joao S. O. Bueno <jsbueno at python.org.br> wrote: > 1) Implement a Number class that behaves as a float number, and does > the fuzzy comparisons automatically. > > Justificative: in a whole lot of code, having a "casted" > FuzzyFloat(mynumber) to be able to be compared to others with "==" and "<=" > would be much more readable than the Javaesque > > " if is_close(my_number, other) or my_number < other: " > (against: > "if FuzzyFloat(my_number) <= other :" > , or simply I'm sure this has come up somewhere, but I have to say that I've never in my life seen a situation where an approximate <= was the correct solution to a problem. I've seen lots of exact <= (which you just assume is interchangeable with <) and approximate ==, but never approximate <=. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From ethan at stoneleaf.us Wed Jan 21 22:38:05 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 21 Jan 2015 13:38:05 -0800 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <m9p5n7$10g$1@ger.gmane.org> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> <m9k397$eo9$1@ger.gmane.org> <-3376261731312819627@unknownmsgid> <6ea13fe6-0831-4126-a94b-15b6bd1c83a7@googlegroups.com> <20150120104012.GU18556@ando.pearwood.info> <m9naqf$9s1$1@ger.gmane.org> <CAH0mxTQR=sPKULnBpc8ukCZg-A53ynZgBwomjjRv-M0RhDsu6Q@mail.gmail.com> <20150121115934.GB18556@ando.pearwood.info> <m9p5n7$10g$1@ger.gmane.org> Message-ID: <54C01C3D.1050607@stoneleaf.us> On 01/21/2015 01:29 PM, Terry Reedy wrote: > > The proper operation of sets (and dicts) depends on == being transitive. Letting D0 = Decimal(0), we once had 0.0 == 0 > and 0 == D0 but 0.0 != D0 (now fixed) and the result was obscure bugs. One of the design criteria for, I believe, Enums > was that == be transitive. Yes -- it == transitivity makes life easier. :) -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/920ea2fc/attachment.sig> From greg.ewing at canterbury.ac.nz Wed Jan 21 23:18:28 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Jan 2015 11:18:28 +1300 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> Message-ID: <54C025B4.6020805@canterbury.ac.nz> Guido van Rossum wrote: > That's debatable. While evaluating them at run time causes some issues > with forward references (mostly for container-ish class definitions), it > also catches many mistakes early, such as not importing the referenced > classes. Wouldn't the static type checker catch that even earlier? If you reference something in a type expression that you haven't imported, the static checker won't know what it means, so surely it must complain. > Switching to a double colon > would make it a magical operator -- especially since a single colon in > the same position would have a similar but different meaning. I don't see how it's any more magical than '==' being a distinct token rather than just two '=' tokens in a row, and 'x == 2' having a different meaning from 'x = 2'. > Eight years ago we picked the winning syntax for argument > annotations. It would be tragic if we now were to replace it with this > syntactic hack. I think it's tragic that we seem to have painted ourselves into a corner where we can't give that winning syntax the semantics that would be best for what we've now decided on as the winning use case. How about we just deprecate relying on run-time evaluation of argument annotations? You've pretty much declared that all existing uses of them are deprecated, so it wouldn't cause much more breakage in the long run, and it would give us a way out of the corner in the future if we wanted. -- Greg From greg.ewing at canterbury.ac.nz Wed Jan 21 23:18:35 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Jan 2015 11:18:35 +1300 Subject: [Python-ideas] Should bool continue to inherit from int? In-Reply-To: <CAP7+vJKen6W4yopMZPXNM9k9pm5M2OZKQtUugFxwb2JAVa4QsQ@mail.gmail.com> References: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> <CAP7+vJKen6W4yopMZPXNM9k9pm5M2OZKQtUugFxwb2JAVa4QsQ@mail.gmail.com> Message-ID: <54C025BB.6020207@canterbury.ac.nz> Guido van Rossum wrote: > And now > suppose you have a faulty caller which calls this function with the > argument set to True. This program will type-check correctly, because > True is a bool which is a subclass of int, but it will run incorrectly, > because (under this assumption) True will be converted to a JSON true > value, which the server rejects. This could be regarded as just one of the many things that static type checking can't catch, and needs to be checked at run time. However, with something like the exactly() function suggested earlier, the required type could be described statically. -- Greg From greg.ewing at canterbury.ac.nz Wed Jan 21 23:18:31 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Jan 2015 11:18:31 +1300 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <CAP7h-xaKyYA-c7WO3ND2j1bYuAerV=EDtS0Y1092y2so16DpsQ@mail.gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> <CAP7+vJ+uQ3LfpqmME6f3EEmOxoXkJP8Nv4t73EVzxQwr0hzVkg@mail.gmail.com> <CAP7h-xbb9pCsWsRnCHWRLk2Ukm5p3PmUPW+r=AFvtbLpx17Fuw@mail.gmail.com> <CAP7h-xaKyYA-c7WO3ND2j1bYuAerV=EDtS0Y1092y2so16DpsQ@mail.gmail.com> Message-ID: <54C025B7.3000102@canterbury.ac.nz> Alexander Belopolsky wrote: > > Why can't we make interned > strings a subclass of strings like bool is a subclass of int? Because a string can change from being non-interned to being interned at some point in its life. I suppose its tp_class could be changed at that point, but that would be surprising for an otherwise immutable type. -- Greg From tjreedy at udel.edu Wed Jan 21 23:21:45 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 21 Jan 2015 17:21:45 -0500 Subject: [Python-ideas] Should bool continue to inherit from int? In-Reply-To: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> References: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> Message-ID: <m9p8pp$jhd$1@ger.gmane.org> On 1/21/2015 6:32 AM, Michael Mitchell wrote: > PEP 285 <http://legacy.python.org/dev/peps/pep-0285> provides some > justification for why arithmetic operations should be valid on bools and > why bool should inherit from int (see points (4) and (6) respectively). > Since it's been 12 years (it's possible this has been brought up again > between now and then), I thought it could be worthwhile to take another > look. > > I am mostly interested in a resurveying of the questions: > 1) Would it still be very inconvenient to implement bools separately > from ints? > 2) Do most Python users still agree that arithmetic operations should be > supported on booleans? python-list would be a better place for this discussion. -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Wed Jan 21 23:24:23 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Jan 2015 11:24:23 +1300 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <CAPTjJmoG_DYLXqxJdt8FAfkzaSVcyYRpPb2X6826htrA_zjrLg@mail.gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> <CAP7+vJ+uQ3LfpqmME6f3EEmOxoXkJP8Nv4t73EVzxQwr0hzVkg@mail.gmail.com> <CAP7h-xbb9pCsWsRnCHWRLk2Ukm5p3PmUPW+r=AFvtbLpx17Fuw@mail.gmail.com> <CAP7h-xaKyYA-c7WO3ND2j1bYuAerV=EDtS0Y1092y2so16DpsQ@mail.gmail.com> <CAPTjJmoG_DYLXqxJdt8FAfkzaSVcyYRpPb2X6826htrA_zjrLg@mail.gmail.com> Message-ID: <54C02717.5020705@canterbury.ac.nz> Chris Angelico wrote: > If, in the fourth example, the interpreter knew that both those string > objects were interned, it could know that they can't possibly be the > same string. So is that distinction worth coding into the class > hierarchy? I believe there is a flag in the string object indicating that it is interned, so the interpreter *does* know this. It's just not visible to Python code. Switching the class would achieve the same thing and make it visible, but is there any need for it to be? And even if there is, an is_interned() function would achieve the same purpose without having an immutable type magically changing its class. -- Greg From rosuav at gmail.com Wed Jan 21 23:30:42 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 22 Jan 2015 09:30:42 +1100 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <54C02717.5020705@canterbury.ac.nz> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> <CAP7+vJ+uQ3LfpqmME6f3EEmOxoXkJP8Nv4t73EVzxQwr0hzVkg@mail.gmail.com> <CAP7h-xbb9pCsWsRnCHWRLk2Ukm5p3PmUPW+r=AFvtbLpx17Fuw@mail.gmail.com> <CAP7h-xaKyYA-c7WO3ND2j1bYuAerV=EDtS0Y1092y2so16DpsQ@mail.gmail.com> <CAPTjJmoG_DYLXqxJdt8FAfkzaSVcyYRpPb2X6826htrA_zjrLg@mail.gmail.com> <54C02717.5020705@canterbury.ac.nz> Message-ID: <CAPTjJmo4BaDixXJmzoTpGsbxC6LV-v4v5Zkt5PsGKGa=OggVTA@mail.gmail.com> On Thu, Jan 22, 2015 at 9:24 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote: > Chris Angelico wrote: >> >> If, in the fourth example, the interpreter knew that both those string >> objects were interned, it could know that they can't possibly be the >> same string. So is that distinction worth coding into the class >> hierarchy? > > > I believe there is a flag in the string object indicating > that it is interned, so the interpreter *does* know this. > It's just not visible to Python code. Huh. Okay. It's not made use of for optimizing the == operator, which ought to be easy to do. Or is the cost of checking not worth it? ChrisA From guido at python.org Wed Jan 21 23:30:54 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Jan 2015 14:30:54 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <54C025B4.6020805@canterbury.ac.nz> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> Message-ID: <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> On Wed, Jan 21, 2015 at 2:18 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote: > Guido van Rossum wrote: > >> That's debatable. While evaluating them at run time causes some issues >> with forward references (mostly for container-ish class definitions), it >> also catches many mistakes early, such as not importing the referenced >> classes. >> > > Wouldn't the static type checker catch that even earlier? > If you reference something in a type expression that you > haven't imported, the static checker won't know what it > means, so surely it must complain. > But the static checker is like a linter and it's common to run those infrequently (e.g. only when submitting a diff for review) while running unittests frequently (e.g. whenever the developer has written some new code). Assuming you have fast unittests. > Switching to a double colon would make it a magical operator -- >> especially since a single colon in the same position would have a similar >> but different meaning. >> > > I don't see how it's any more magical than '==' being a > distinct token rather than just two '=' tokens in a row, > and 'x == 2' having a different meaning from 'x = 2'. > It's different because == is used in nearly all languages in common use (certainly all in the TIOBE top 10) while :: would be a brand new invention here (unrelated to e.g. C++'s :: scoping operator). > Eight years ago we picked the winning syntax for argument annotations. It >> would be tragic if we now were to replace it with this syntactic hack. >> > > I think it's tragic that we seem to have painted ourselves > into a corner where we can't give that winning syntax the > semantics that would be best for what we've now decided on > as the winning use case. > To the contrary, I don't think it's tragic at all. It's how most Python evolution happens. mypy wouldn't have happened the way it did if we didn't have annotations: Jukka's original plan was a custom language, which would have made it just one more Ph.D. project without any users. > How about we just deprecate relying on run-time evaluation > of argument annotations? You've pretty much declared that > all existing uses of them are deprecated, so it wouldn't > cause much more breakage in the long run, and it would give > us a way out of the corner in the future if we wanted. I don't see a way to get there from here. Talking about deprecating them is cheap -- actually breaking the code can't happen until 3.6 at the earliest (if we were to issue deprecation warnings by default in 3.5 for every annotation that's not a class or imported from typing), or 3.7 realistically (if in 3.5 we document the deprecation but don't act on it). The provisional status of PEP 484 makes even the more conservative scenario unlikely. And, having coded up much of a possible typing.py module already ( https://github.com/ambv/typehinting/tree/master/prototyping) I really don't see the big problem with run-time evaluation of constraints. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/d70f110d/attachment.html> From abarnert at yahoo.com Thu Jan 22 00:41:42 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 21 Jan 2015 15:41:42 -0800 Subject: [Python-ideas] Floating point "closeness" Proposal Outline In-Reply-To: <CAPJVwBn1E0hvyJx41XNaXZde4QrC9+J189igT4W5vy3N_xMmkA@mail.gmail.com> References: <CALGmxELAWrQygPH7pkC4rje7_nFeMQdwXd_Z5ZX3homYADhh_g@mail.gmail.com> <m9jn75$a3s$1@ger.gmane.org> <CALGmxEJJ570Qk1-OZre4iOOc-hyLANjF30U_Vk=zyURUjoEBrg@mail.gmail.com> <m9k397$eo9$1@ger.gmane.org> <-3376261731312819627@unknownmsgid> <6ea13fe6-0831-4126-a94b-15b6bd1c83a7@googlegroups.com> <20150120104012.GU18556@ando.pearwood.info> <m9naqf$9s1$1@ger.gmane.org> <CAH0mxTQR=sPKULnBpc8ukCZg-A53ynZgBwomjjRv-M0RhDsu6Q@mail.gmail.com> <CAPJVwBn1E0hvyJx41XNaXZde4QrC9+J189igT4W5vy3N_xMmkA@mail.gmail.com> Message-ID: <967DE716-4440-4329-8423-28BE41297F56@yahoo.com> On Jan 21, 2015, at 13:36, Nathaniel Smith <njs at pobox.com> wrote: > On Wed, Jan 21, 2015 at 11:27 AM, Joao S. O. Bueno > <jsbueno at python.org.br> wrote: >> 1) Implement a Number class that behaves as a float number, and does >> the fuzzy comparisons automatically. >> >> Justificative: in a whole lot of code, having a "casted" >> FuzzyFloat(mynumber) to be able to be compared to others with "==" and "<=" >> would be much more readable than the Javaesque >> >> " if is_close(my_number, other) or my_number < other: " >> (against: >> "if FuzzyFloat(my_number) <= other :" >> , or simply > > I'm sure this has come up somewhere, but I have to say that I've never > in my life seen a situation where an approximate <= was the correct > solution to a problem. I've seen lots of exact <= (which you just > assume is interchangeable with <) and approximate ==, but never > approximate <=. Well, when both values are approximate, you can use interval <= to mean "possibly <" (or "not definitely >", which is more often the way you'd write it). But of course this isn't interval <=, it's approximate vs. assumed-to-be-exact, so this suggestion seems at best useless and at worst an attractive nuisance for people who want interval math but don't realize it. From greg.ewing at canterbury.ac.nz Thu Jan 22 01:04:36 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Jan 2015 13:04:36 +1300 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> Message-ID: <54C03E94.9050800@canterbury.ac.nz> On 01/22/2015 11:30 AM, Guido van Rossum wrote: > But the static checker is like a linter and it's common to run those > infrequently (e.g. only when submitting a diff for review) while running > unittests frequently (e.g. whenever the developer has written some new code). > Assuming you have fast unittests. This still doesn't make sense to me. If your unit tests are any good, they'll contain tests that use the types you're supposed to be importing, so they'll fail if you forget to import them. > And, having coded up much of a possible typing.py module already > (https://github.com/ambv/typehinting/tree/master/prototyping) I really don't > see the big problem with run-time evaluation of constraints. Well, having to quote forward-referenced types is a fairly big aesthetic problem for me, even if it's not a practical one. To my mind it's a worse syntactic hack than '::' would be. But it seems you're not bothered by it so much. -- Greg From guido at python.org Thu Jan 22 01:50:16 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Jan 2015 16:50:16 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <54C03E94.9050800@canterbury.ac.nz> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> Message-ID: <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> On Wed, Jan 21, 2015 at 4:04 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote: > On 01/22/2015 11:30 AM, Guido van Rossum wrote: > >> But the static checker is like a linter and it's common to run those >> infrequently (e.g. only when submitting a diff for review) while running >> unittests frequently (e.g. whenever the developer has written some new >> code). >> Assuming you have fast unittests. >> > > This still doesn't make sense to me. If your unit tests are any > good, they'll contain tests that use the types you're supposed to > be importing, so they'll fail if you forget to import them. > Not necessarily. I've seen plenty of code that takes instances a certain class as argument and calls methods of that type but never constructs a new instance nor does any of the other things for which you need the class name, and where the module defining the class is never imported. Such code works fine, but when you want to declare the type you have to import it. > And, having coded up much of a possible typing.py module already >> (https://github.com/ambv/typehinting/tree/master/prototyping) I really >> don't >> see the big problem with run-time evaluation of constraints. >> > > Well, having to quote forward-referenced types is a fairly big > aesthetic problem for me, even if it's not a practical one. To > my mind it's a worse syntactic hack than '::' would be. But it > seems you're not bothered by it so much. You can write a lot of code without ever having to use a forward reference. But "::" would have to be used everywhere. Or perhaps I value the ability to use this with Python 3.4 more than you do. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/2ae0a9b5/attachment.html> From rob.cliffe at btinternet.com Thu Jan 22 02:10:04 2015 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Thu, 22 Jan 2015 01:10:04 +0000 Subject: [Python-ideas] Should bool continue to inherit from int? In-Reply-To: <CAPTjJmqC3iDnobQna-6bCbAjPYabEsidpdTj+qzaLxmm+oQT1Q@mail.gmail.com> References: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> <CAPTjJmqC3iDnobQna-6bCbAjPYabEsidpdTj+qzaLxmm+oQT1Q@mail.gmail.com> Message-ID: <54C04DEC.9090102@btinternet.com> On 21/01/2015 13:27, Chris Angelico wrote: > On Wed, Jan 21, 2015 at 10:32 PM, Michael Mitchell > <epsilonmichael at gmail.com> wrote: >> PEP 285 provides some justification for why arithmetic operations should be >> valid on bools and why bool should inherit from int (see points (4) and (6) >> respectively). Since it's been 12 years (it's possible this has been brought >> up again between now and then), I thought it could be worthwhile to take >> another look. >> >> I am mostly interested in a resurveying of the questions: >> 1) Would it still be very inconvenient to implement bools separately from >> ints? >> 2) Do most Python users still agree that arithmetic operations should be >> supported on booleans? > All of them allow you to write stuff like: > > if (((1==1) + (1==0) + (0==0)) == 2) > > which will be true (if a little over-parenthesized for some people's > liking). May I offer a more realistic example, adapted from a real-life program: ST1 = '20:00' ET1 = '22:00' ST2 = '22:00' ET2 = '06:00' # These variables represent times of day (24-hour clock). # We want to check that [ST1, ET1) and [ST2, ET2) are non-empty, non-overlapping time periods, # (allowing the time to wrap past midnight) as they are in this example # (the time periods are considered to include the *S*tart *T*ime but not the *E*nd *T*ime). # A naive approach might be: if ( (ST1 < ET1 <= ST2 < ET2) or (ET2 <= ST1 < ET1 <= ST2) or (ST2 < ET2 <= ST1 < ET1) or (ET1 <= ST2 < ET2 <= ST1) ): <time intervals are OK> # but this can be coded much more succinctly if at first sight less transparently (and sorry, Chris, without redundant parentheses :-) ) as: if (ST1<ET1) + (ET1 <= ST2) + (ST2<ET2) + (ET2<=ST1) == 3: <time intervals are OK> # Lines 1,2,3,4 in the naive approach correspond exactly # to the cases where only the 4th/3rd/2nd/1st term, respectively, # is False in the succinct approach. So arithmetic on booleans may not be "pure" but it can be handy. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150122/34f49b40/attachment.html> From rosuav at gmail.com Thu Jan 22 02:22:55 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 22 Jan 2015 12:22:55 +1100 Subject: [Python-ideas] Should bool continue to inherit from int? In-Reply-To: <54C04DEC.9090102@btinternet.com> References: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> <CAPTjJmqC3iDnobQna-6bCbAjPYabEsidpdTj+qzaLxmm+oQT1Q@mail.gmail.com> <54C04DEC.9090102@btinternet.com> Message-ID: <CAPTjJmoeWWQO-8AsWDDO6mS35R7i_TAP73L2Bz6BHCz4MPhtHA@mail.gmail.com> On Thu, Jan 22, 2015 at 12:10 PM, Rob Cliffe <rob.cliffe at btinternet.com> wrote: > # but this can be coded much more succinctly if at first sight less > transparently (and sorry, Chris, without redundant parentheses :-) ) as: The extra parens were just because I was listing a bunch of languages, and the exact same statement would work in all of them (although Python would expect a colon at the end, and REXX would be looking for a 'THEN' on that or the next line). :) I wouldn't normally put the extra parens in a Python program. ChrisA From greg.ewing at canterbury.ac.nz Thu Jan 22 02:25:01 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Jan 2015 14:25:01 +1300 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> Message-ID: <54C0516D.5090902@canterbury.ac.nz> On 01/22/2015 01:50 PM, Guido van Rossum wrote: > I've seen plenty of code that takes instances a certain class > as argument and calls methods of that type but never constructs a new instance > nor does any of the other things for which you need the class name, If the only thing the name is needed for is static checking, I don't see what's so bad about not finding out until you actually run the static check. Adding those imports earlier doesn't make the code work any better or show up any more runtime bugs. -- Greg From alexander.belopolsky at gmail.com Thu Jan 22 03:08:06 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 21 Jan 2015 21:08:06 -0500 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <54C025B7.3000102@canterbury.ac.nz> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> <CAP7+vJ+uQ3LfpqmME6f3EEmOxoXkJP8Nv4t73EVzxQwr0hzVkg@mail.gmail.com> <CAP7h-xbb9pCsWsRnCHWRLk2Ukm5p3PmUPW+r=AFvtbLpx17Fuw@mail.gmail.com> <CAP7h-xaKyYA-c7WO3ND2j1bYuAerV=EDtS0Y1092y2so16DpsQ@mail.gmail.com> <54C025B7.3000102@canterbury.ac.nz> Message-ID: <CAP7h-xa8tNU-c98Rqshg3OLCZ1fnWPM6yTYiOj37azSzwFhsmA@mail.gmail.com> On Wed, Jan 21, 2015 at 5:18 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote: > Alexander Belopolsky wrote: > >> >> Why can't we make interned strings a subclass of strings like bool is a >> subclass of int? >> > > Because a string can change from being non-interned to > being interned at some point in its life. I did not know that. Can you show the code that would cause such a change? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/9db6da25/attachment-0001.html> From alexander.belopolsky at gmail.com Thu Jan 22 03:13:20 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 21 Jan 2015 21:13:20 -0500 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <CAP7h-xa8tNU-c98Rqshg3OLCZ1fnWPM6yTYiOj37azSzwFhsmA@mail.gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> <CAP7+vJ+uQ3LfpqmME6f3EEmOxoXkJP8Nv4t73EVzxQwr0hzVkg@mail.gmail.com> <CAP7h-xbb9pCsWsRnCHWRLk2Ukm5p3PmUPW+r=AFvtbLpx17Fuw@mail.gmail.com> <CAP7h-xaKyYA-c7WO3ND2j1bYuAerV=EDtS0Y1092y2so16DpsQ@mail.gmail.com> <54C025B7.3000102@canterbury.ac.nz> <CAP7h-xa8tNU-c98Rqshg3OLCZ1fnWPM6yTYiOj37azSzwFhsmA@mail.gmail.com> Message-ID: <CAP7h-xYdrj4P9DWpoGKs=OoFgJxG80gBUGAWKK6R4CYx5NPOAg@mail.gmail.com> Sorry, I misread what you wrote. I thought you suggested that an interned string can get "uninterned" somehow. Yes, sys.intern(x) will change x's status, but as you suggested, it can change x.__class__ as well. However, with a Symbol class in-place, I don't think there will be a need for such transitions. Even now, I don't see sys.intern() in user code too often. On Wed, Jan 21, 2015 at 9:08 PM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Wed, Jan 21, 2015 at 5:18 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> > wrote: > >> Alexander Belopolsky wrote: >> >>> >>> Why can't we make interned strings a subclass of strings like bool is a >>> subclass of int? >>> >> >> Because a string can change from being non-interned to >> being interned at some point in its life. > > > I did not know that. Can you show the code that would cause such a change? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/44ec0a4c/attachment.html> From epsilonmichael at gmail.com Thu Jan 22 03:13:20 2015 From: epsilonmichael at gmail.com (Michael Mitchell) Date: Wed, 21 Jan 2015 18:13:20 -0800 Subject: [Python-ideas] Should bool continue to inherit from int? In-Reply-To: <54C04DEC.9090102@btinternet.com> References: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> <CAPTjJmqC3iDnobQna-6bCbAjPYabEsidpdTj+qzaLxmm+oQT1Q@mail.gmail.com> <54C04DEC.9090102@btinternet.com> Message-ID: <CABvhCyEa6B2RYgciMaq6gLK9GtLHKzTsEj0yR4c3CfZtm=knjQ@mail.gmail.com> Might not be the best example because there exists a comparably concise way without using addition: if all([ST1 < ET1, ST2 < ET2, ET2 <= ST1 or ET1 <= ST2]): <time intervals are okay> but I do see the point On Wed, Jan 21, 2015 at 5:10 PM, Rob Cliffe <rob.cliffe at btinternet.com> wrote: > On 21/01/2015 13:27, Chris Angelico wrote: > > On Wed, Jan 21, 2015 at 10:32 PM, Michael Mitchell<epsilonmichael at gmail.com> <epsilonmichael at gmail.com> wrote: > > PEP 285 provides some justification for why arithmetic operations should be > valid on bools and why bool should inherit from int (see points (4) and (6) > respectively). Since it's been 12 years (it's possible this has been brought > up again between now and then), I thought it could be worthwhile to take > another look. > > I am mostly interested in a resurveying of the questions: > 1) Would it still be very inconvenient to implement bools separately from > ints? > 2) Do most Python users still agree that arithmetic operations should be > supported on booleans? > > All of them allow you to write stuff like: > > if (((1==1) + (1==0) + (0==0)) == 2) > > which will be true (if a little over-parenthesized for some people's > liking). > > May I offer a more realistic example, adapted from a real-life program: > > ST1 = '20:00' > ET1 = '22:00' > ST2 = '22:00' > ET2 = '06:00' > # These variables represent times of day (24-hour clock). > # We want to check that [ST1, ET1) and [ST2, ET2) are non-empty, > non-overlapping time periods, > # (allowing the time to wrap past midnight) as they are in this example > # (the time periods are considered to include the *S*tart *T*ime but not > the *E*nd *T*ime). > # A naive approach might be: > if ( (ST1 < ET1 <= ST2 < ET2) > or (ET2 <= ST1 < ET1 <= ST2) > or (ST2 < ET2 <= ST1 < ET1) > or (ET1 <= ST2 < ET2 <= ST1) ): > > <time intervals are OK> > > # but this can be coded much more succinctly if at first sight less > transparently (and sorry, Chris, without redundant parentheses :-) ) as: > > if (ST1<ET1) + (ET1 <= ST2) + (ST2<ET2) + (ET2<=ST1) == 3: > > <time intervals are OK> > > # Lines 1,2,3,4 in the naive approach correspond exactly > # to the cases where only the 4th/3rd/2nd/1st term, respectively, > # is False in the succinct approach. > > So arithmetic on booleans may not be "pure" but it can be handy. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/768b41d0/attachment.html> From yawar.amin at gmail.com Thu Jan 22 05:25:21 2015 From: yawar.amin at gmail.com (Yawar Amin) Date: Wed, 21 Jan 2015 23:25:21 -0500 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <CAP7h-xYCVx453=o0mPsDFpT+8=_zKCCb4S0yBW=5aOK-7RFxhg@mail.gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> <CAP7+vJ+uQ3LfpqmME6f3EEmOxoXkJP8Nv4t73EVzxQwr0hzVkg@mail.gmail.com> <CAP7h-xbb9pCsWsRnCHWRLk2Ukm5p3PmUPW+r=AFvtbLpx17Fuw@mail.gmail.com> <CAP7+vJK_KqqU8mbuR+14L4dC5M2YmxROhD-UvR=8tJRheq-=HQ@mail.gmail.com> <CAP7h-xYCVx453=o0mPsDFpT+8=_zKCCb4S0yBW=5aOK-7RFxhg@mail.gmail.com> Message-ID: <54C07BB1.4090606@gmail.com> Hi, On 2015-01-21 14:38, Alexander Belopolsky wrote: > [...] > My use case comes from interoperability with other languages. I > maintain a package which provides an interface [1] between Python and > the language [2] called Q. Q has 3 scalar types to represent text > data (char, byte and symbol) and the corresponding vector types. When > I pass Python strings to Q, I convert them to symbols because that is > what users expect most of the time, but as a result, working with char > vectors is unnatural. I currently use bytes type for that, but it is > really an abuse because bytes should map to byte vectors. Hi, after thinking about it some more I've come to believe new, empty classes are the best way to model symbols in Python (for now at least). My reasoning is: - Symbols are tags (i.e. unique names for things). - In a typed language which has enums or discriminated unions (aka tagged unions), tags are often created from enums. - Enums and discriminated unions are isomorphic to base and inheriting classes. - Therefore, symbols are classes which could optionally inherit from a base class. In your case, you could define a function which dynamically creates these 'symbols' for you and then stashes them somewhere for re-use. Below, I've used the calling module's `globals()` dict, but you could store them away very neatly in some other dict to avoid any possible name conflicts. # In called module: class sym(object): pass def get_sym(sym_name, d): if s in d: return d[sym_name] new_sym = type(sym_name, (sym,), {}) d[sym_name] = new_sym return new_sym # In calling module: def char_to_sym(c): "Map a 'character' to a symbol." assert len(c) == 1 return get_sym(c, globals()) # This could be any dict. def str_to_syms(s): return map(char_to_sym, s) print str_to_syms("Hello, World!") # ==> [<class '__main__.H'>, ...] print char_to_sym("!") == char_to_sym("!") # ==> True Regards, Yawar -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 834 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/9ce7bb91/attachment.sig> From yawar.amin at gmail.com Thu Jan 22 05:29:55 2015 From: yawar.amin at gmail.com (Yawar Amin) Date: Wed, 21 Jan 2015 23:29:55 -0500 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <54C07BB1.4090606@gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> <CAP7+vJ+uQ3LfpqmME6f3EEmOxoXkJP8Nv4t73EVzxQwr0hzVkg@mail.gmail.com> <CAP7h-xbb9pCsWsRnCHWRLk2Ukm5p3PmUPW+r=AFvtbLpx17Fuw@mail.gmail.com> <CAP7+vJK_KqqU8mbuR+14L4dC5M2YmxROhD-UvR=8tJRheq-=HQ@mail.gmail.com> <CAP7h-xYCVx453=o0mPsDFpT+8=_zKCCb4S0yBW=5aOK-7RFxhg@mail.gmail.com> <54C07BB1.4090606@gmail.com> Message-ID: <54C07CC3.1030406@gmail.com> On 2015-01-21 23:25, Yawar Amin wrote: > [...] > > def get_sym(sym_name, d): > if s in d: return d[sym_name] Argh, that should be: if sym_name in d: return d[sym_name] -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 834 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/d72a5416/attachment-0001.sig> From abarnert at yahoo.com Thu Jan 22 05:31:00 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 21 Jan 2015 20:31:00 -0800 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <CAP7h-xYdrj4P9DWpoGKs=OoFgJxG80gBUGAWKK6R4CYx5NPOAg@mail.gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> <CAP7+vJ+uQ3LfpqmME6f3EEmOxoXkJP8Nv4t73EVzxQwr0hzVkg@mail.gmail.com> <CAP7h-xbb9pCsWsRnCHWRLk2Ukm5p3PmUPW+r=AFvtbLpx17Fuw@mail.gmail.com> <CAP7h-xaKyYA-c7WO3ND2j1bYuAerV=EDtS0Y1092y2so16DpsQ@mail.gmail.com> <54C025B7.3000102@canterbury.ac.nz> <CAP7h-xa8tNU-c98Rqshg3OLCZ1fnWPM6yTYiOj37azSzwFhsmA@mail.gmail.com> <CAP7h-xYdrj4P9DWpoGKs=OoFgJxG80gBUGAWKK6R4CYx5NPOAg@mail.gmail.com> Message-ID: <723FA2F9-2523-489A-BA0A-69A24D43AC91@yahoo.com> On Jan 21, 2015, at 18:13, Alexander Belopolsky <alexander.belopolsky at gmail.com> wrote: > Sorry, I misread what you wrote. I thought you suggested that an interned string can get "uninterned" somehow. Yes, sys.intern(x) will change x's status, but as you suggested, it can change x.__class__ as well. However, with a Symbol class in-place, I don't think there will be a need for such transitions. Even now, I don't see sys.intern() in user code too often. Probably because the difference between interned and uninterned strings isn't relevant too often. In Python, there is nothing extra you could do with an interned string or symbol that you can't do with a normal string. So the only conceivable benefit is as a performance optimization. And even that won't come up very often--if your hotspot is string comparisons between static or long-lived strings, most likely you're doing something wrong at a much bigger level (e.g., you've got a long chain of elif comparisons where you should have a dict). It's there on the occasions when it's really needed, but the fact that people rarely use it implies that there isn't a need to expand on it with an interned_string class, or anything else. > On Wed, Jan 21, 2015 at 9:08 PM, Alexander Belopolsky <alexander.belopolsky at gmail.com> wrote: >> >> On Wed, Jan 21, 2015 at 5:18 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote: >>> Alexander Belopolsky wrote: >>>> >>>> Why can't we make interned strings a subclass of strings like bool is a subclass of int? >>> >>> Because a string can change from being non-interned to >>> being interned at some point in its life. >> >> I did not know that. Can you show the code that would cause such a change? > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/3472905a/attachment.html> From eltoder at gmail.com Thu Jan 22 05:39:15 2015 From: eltoder at gmail.com (Eugene Toder) Date: Wed, 21 Jan 2015 23:39:15 -0500 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <54C0516D.5090902@canterbury.ac.nz> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> Message-ID: <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> On Wed, Jan 21, 2015 at 8:25 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote: > On 01/22/2015 01:50 PM, Guido van Rossum wrote: > >> I've seen plenty of code that takes instances a certain class >> as argument and calls methods of that type but never constructs a new >> instance >> nor does any of the other things for which you need the class name, >> > > If the only thing the name is needed for is static checking, > I don't see what's so bad about not finding out until you > actually run the static check. Adding those imports earlier > doesn't make the code work any better or show up any more > runtime bugs. Also, if static type checking succeeds, it will be integrated into many IDEs/editors, so you will see type checker errors even before you get to run the tests. Eugene -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/e95447f1/attachment.html> From guido at python.org Thu Jan 22 05:47:44 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Jan 2015 20:47:44 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> Message-ID: <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> On Wed, Jan 21, 2015 at 8:39 PM, Eugene Toder <eltoder at gmail.com> wrote: > On Wed, Jan 21, 2015 at 8:25 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> > wrote: > >> On 01/22/2015 01:50 PM, Guido van Rossum wrote: >> >>> I've seen plenty of code that takes instances a certain class >>> as argument and calls methods of that type but never constructs a new >>> instance >>> nor does any of the other things for which you need the class name, >>> >> >> If the only thing the name is needed for is static checking, >> I don't see what's so bad about not finding out until you >> actually run the static check. Adding those imports earlier >> doesn't make the code work any better or show up any more >> runtime bugs. > > > > Also, if static type checking succeeds, it will be integrated into many > IDEs/editors, so you will see type checker errors even before you get to > run the tests. > So I think we're all agreed that the current proposal is fine! -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150121/d9f1774e/attachment.html> From storchaka at gmail.com Thu Jan 22 07:35:41 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 22 Jan 2015 08:35:41 +0200 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <CAPTjJmo4BaDixXJmzoTpGsbxC6LV-v4v5Zkt5PsGKGa=OggVTA@mail.gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> <CAP7+vJ+uQ3LfpqmME6f3EEmOxoXkJP8Nv4t73EVzxQwr0hzVkg@mail.gmail.com> <CAP7h-xbb9pCsWsRnCHWRLk2Ukm5p3PmUPW+r=AFvtbLpx17Fuw@mail.gmail.com> <CAP7h-xaKyYA-c7WO3ND2j1bYuAerV=EDtS0Y1092y2so16DpsQ@mail.gmail.com> <CAPTjJmoG_DYLXqxJdt8FAfkzaSVcyYRpPb2X6826htrA_zjrLg@mail.gmail.com> <54C02717.5020705@canterbury.ac.nz> <CAPTjJmo4BaDixXJmzoTpGsbxC6LV-v4v5Zkt5PsGKGa=OggVTA@mail.gmail.com> Message-ID: <m9q5nu$3tf$1@ger.gmane.org> On 22.01.15 00:30, Chris Angelico wrote: > On Thu, Jan 22, 2015 at 9:24 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote: >> I believe there is a flag in the string object indicating >> that it is interned, so the interpreter *does* know this. >> It's just not visible to Python code. > > Huh. Okay. It's not made use of for optimizing the == operator, which > ought to be easy to do. Or is the cost of checking not worth it? The cost of checking is not worth it. Comparing hashes (which are already calculated for interned strings) before comparing contents has almost the same effect and this proposition was tested and rejected. This makes double work in dict lookup which is more common case. https://bugs.python.org/issue16286 From storchaka at gmail.com Thu Jan 22 07:47:15 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 22 Jan 2015 08:47:15 +0200 Subject: [Python-ideas] Keyword/Symbol literals In-Reply-To: <CAP7h-xaKyYA-c7WO3ND2j1bYuAerV=EDtS0Y1092y2so16DpsQ@mail.gmail.com> References: <CAJ8oX-HkT9q-++cY3CmT4Sp6ebJgW9w2AXPXnp744YSj_KMTVg@mail.gmail.com> <CAM1gar5vED9KPkkdWosC69gHtMdJFcZ_8=RuVvVxiM+bChb8RQ@mail.gmail.com> <99879879-CCBC-48E9-91F0-40FCA0540C6B@yahoo.com> <CAP7+vJ+uQ3LfpqmME6f3EEmOxoXkJP8Nv4t73EVzxQwr0hzVkg@mail.gmail.com> <CAP7h-xbb9pCsWsRnCHWRLk2Ukm5p3PmUPW+r=AFvtbLpx17Fuw@mail.gmail.com> <CAP7h-xaKyYA-c7WO3ND2j1bYuAerV=EDtS0Y1092y2so16DpsQ@mail.gmail.com> Message-ID: <m9q6dk$dtv$1@ger.gmane.org> On 21.01.15 21:18, Alexander Belopolsky wrote: > Why can't we make interned > strings a subclass of strings like bool is a subclass of int? Some code (e.g. pickle, copy, lry_cache, re, pprint) relies on exact type and wouldn't work with str subtypes as good as with str. From cfkaran2 at gmail.com Thu Jan 22 12:18:54 2015 From: cfkaran2 at gmail.com (Cem Karan) Date: Thu, 22 Jan 2015 06:18:54 -0500 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> Message-ID: <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> On Jan 21, 2015, at 11:47 PM, Guido van Rossum <guido at python.org> wrote: > On Wed, Jan 21, 2015 at 8:39 PM, Eugene Toder <eltoder at gmail.com> wrote: > On Wed, Jan 21, 2015 at 8:25 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote: > On 01/22/2015 01:50 PM, Guido van Rossum wrote: > I've seen plenty of code that takes instances a certain class > as argument and calls methods of that type but never constructs a new instance > nor does any of the other things for which you need the class name, > > If the only thing the name is needed for is static checking, > I don't see what's so bad about not finding out until you > actually run the static check. Adding those imports earlier > doesn't make the code work any better or show up any more > runtime bugs. > > > Also, if static type checking succeeds, it will be integrated into many IDEs/editors, so you will see type checker errors even before you get to run the tests. > > So I think we're all agreed that the current proposal is fine! *grumble* ;P I know I keep arguing for a slightly different syntax so that its easy to differentiate between type hints and other uses, but I do want to say that I like the proposal. It seems quite practical to me. Would you at least consider adding something to the proposal that makes it clear that a given annotation is a type and not something else? Thanks, Cem Karan From mistersheik at gmail.com Thu Jan 22 09:45:48 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 22 Jan 2015 00:45:48 -0800 (PST) Subject: [Python-ideas] Should bool continue to inherit from int? In-Reply-To: <CAP7h-xapgY1SLqmNhhT4p2E46qib1jpSRzrMtaOGBQs+EhXqDQ@mail.gmail.com> References: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> <CAP7+vJKen6W4yopMZPXNM9k9pm5M2OZKQtUugFxwb2JAVa4QsQ@mail.gmail.com> <CAP7h-xapgY1SLqmNhhT4p2E46qib1jpSRzrMtaOGBQs+EhXqDQ@mail.gmail.com> Message-ID: <a00d200a-b040-4c3e-b0f3-3e8a5997cf1c@googlegroups.com> What did you expect? It can't be 2 since np.bool is free to be a one-bit type. It shouldn't promote. If anything it should probably be False in my opinion just like np.int8(255) + 1 == 0 >From a mathematician's standpoint Booleans are the ring of integers modulo 2. On this proposal, I agree with the people saying it's too backwards incompatible, but from an ivory tower standpoint it would have been better. Best, Neil On Wednesday, January 21, 2015 at 2:50:00 PM UTC-5, Alexander Belopolsky wrote: > > > On Wed, Jan 21, 2015 at 2:11 PM, Guido van Rossum <gu... at python.org > <javascript:>> wrote: > >> 2) Do most Python users still agree that arithmetic operations should be >>> supported on booleans? >>> >> >> It would be so massively backward compatible that it's not worth thinking >> about. >> > > The only time I encountered a situation where True + True was not 2 was in > numpy: > > >>> numpy.bool_(True) + True > True > > That was *very* inconvenient! > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150122/a6e6d90b/attachment.html> From rosuav at gmail.com Thu Jan 22 13:16:51 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 22 Jan 2015 23:16:51 +1100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> Message-ID: <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> On Thu, Jan 22, 2015 at 10:18 PM, Cem Karan <cfkaran2 at gmail.com> wrote: > I know I keep arguing for a slightly different syntax so that its easy to differentiate between type hints and other uses, but I do want to say that I like the proposal. It seems quite practical to me. Would you at least consider adding something to the proposal that makes it clear that a given annotation is a type and not something else? > The absence of a decorator that uses the annotations some other way should make it clear that they're type hints, just as the absence of a "global" statement should make it clear that assigned names are local to a function. ChrisA From edk141 at gmail.com Thu Jan 22 13:32:40 2015 From: edk141 at gmail.com (Ed Kellett) Date: Thu, 22 Jan 2015 12:32:40 +0000 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> Message-ID: <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> On Thu Jan 22 2015 at 12:17:37 Chris Angelico <rosuav at gmail.com> wrote: > The absence of a decorator that uses the annotations some other way > should make it clear that they're type hints, just as the absence of a > "global" statement should make it clear that assigned names are local > to a function. > What about code that already exists? What about modules that want to support Python installations that already exist (hence don't have the `typing` module)? What about module authors who simply don't want to make their code ugly with a no-op decorator just to signify that they're not making it ugly with type annotations? Ed Kellett -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150122/a5fd7528/attachment.html> From rosuav at gmail.com Thu Jan 22 13:37:33 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 22 Jan 2015 23:37:33 +1100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> Message-ID: <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> On Thu, Jan 22, 2015 at 11:32 PM, Ed Kellett <edk141 at gmail.com> wrote: > On Thu Jan 22 2015 at 12:17:37 Chris Angelico <rosuav at gmail.com> wrote: >> >> The absence of a decorator that uses the annotations some other way >> should make it clear that they're type hints, just as the absence of a >> "global" statement should make it clear that assigned names are local >> to a function. > > > What about code that already exists? What about modules that want to support > Python installations that already exist (hence don't have the `typing` > module)? What about module authors who simply don't want to make their code > ugly with a no-op decorator just to signify that they're not making it ugly > with type annotations? If you don't run the type checker, and the annotations will continue to do what they always have done. Seems easy enough. Supporting Python installations that already exist is as simple as either installing a backported typing.py, or not using a feature that didn't exist at that point. Same as the compatibility requirements of using enumerations in earlier versions of Python - you either grab the one off PyPI, or don't use 'em. I'm not seeing a problem here... have I missed something? ChrisA From edk141 at gmail.com Thu Jan 22 13:43:14 2015 From: edk141 at gmail.com (Ed Kellett) Date: Thu, 22 Jan 2015 12:43:14 +0000 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> Message-ID: <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> On Thu Jan 22 2015 at 12:38:17 Chris Angelico <rosuav at gmail.com> wrote: > > If you don't run the type checker, and the annotations will continue > to do what they always have done. Seems easy enough. > > Supporting Python installations that already exist is as simple as > either installing a backported typing.py, or not using a feature that > didn't exist at that point. Same as the compatibility requirements of > using enumerations in earlier versions of Python - you either grab the > one off PyPI, or don't use 'em. I'm not seeing a problem here... have > I missed something? > Put simply, code can't support existing Python installations and future ones at the same time... unless it adds a dependency (is there a dummy typing.py available on PyPI?) or just doesn't use annotations for *anything*. It's kind of like enumerations, except there aren't any proposals threatening to punish you for not using them. Ed Kellett -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150122/c0ee6214/attachment.html> From rosuav at gmail.com Thu Jan 22 13:48:54 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 22 Jan 2015 23:48:54 +1100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> Message-ID: <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> On Thu, Jan 22, 2015 at 11:43 PM, Ed Kellett <edk141 at gmail.com> wrote: > It's kind of like enumerations, except there aren't any proposals > threatening to punish you for not using them. What's threatening to punish you for not using annotations? ChrisA From brett at python.org Thu Jan 22 15:52:34 2015 From: brett at python.org (Brett Cannon) Date: Thu, 22 Jan 2015 14:52:34 +0000 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> Message-ID: <CAP1=2W59Pxr4j_Cx+Cvd+WQQZ4RznOSrEL9q_G2wZcB=F81=wQ@mail.gmail.com> On Thu Jan 22 2015 at 7:49:38 AM Chris Angelico <rosuav at gmail.com> wrote: > On Thu, Jan 22, 2015 at 11:43 PM, Ed Kellett <edk141 at gmail.com> wrote: > > It's kind of like enumerations, except there aren't any proposals > > threatening to punish you for not using them. > > What's threatening to punish you for not using annotations? > If you read one of Ed's earlier emails it sounds like he thinks you need the @no_type_hints decorator even if you don't provide *any* annotations, which is incorrect (the decorator is only to signal that annotations you did specify are not type hints; not specifying any annotations signals the same thing). Heck, the decorator isn't even technically necessary if the linter or IDE you are using that can process the type hints chooses to treat unrecognized annotations as Any instead of throwing an error or something. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150122/ae226d3e/attachment.html> From steve at pearwood.info Thu Jan 22 15:55:33 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 23 Jan 2015 01:55:33 +1100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> References: <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> Message-ID: <20150122145533.GF18556@ando.pearwood.info> On Thu, Jan 22, 2015 at 12:32:40PM +0000, Ed Kellett wrote: > On Thu Jan 22 2015 at 12:17:37 Chris Angelico <rosuav at gmail.com> wrote: > > > The absence of a decorator that uses the annotations some other way > > should make it clear that they're type hints, just as the absence of a > > "global" statement should make it clear that assigned names are local > > to a function. > > > > What about code that already exists? What about them? I have lots of code that already exists. (Some of it even works *wink*) Since that code doesn't use annotations, this proposal will not make one iota of difference to that code. *Literally* nothing will change. Some of my code has annotations. But since I'm not using a PEP 484 type-checker on that code, it too will continue to work completely unchanged. The *only* code that will change is that which *I* decide to start using annotations for type-checking. If I'm using Python 3.4 or older, I go to PyPI and download the typing.py module. If I'm using 3.5, it will be included in the standard library. So I add the annotations to my code: as much, or as little, of the code as I choose. I could annotate as little as one function or as much as the entire application. Even then, I still have choices. I can choose whether to perform type-checking or not: - I can just run my code directly with the standard CPython interpreter (the one you are probably using now), and no type-checks will take place. CPython will not include a type-checker, not in 3.5 and perhaps not ever. - I can run my code with Mypy, and it will perform static type-checks before executing the code. - I can run my code with a static linter or type-checker and not run the code. - Or just let my IDE or editor give me better completion hints and flag errors as I type them. > What about modules that want to > support Python installations that already exist (hence don't have the > `typing` module)? typing.py will be available on PyPI. > What about module authors who simply don't want to make > their code ugly with a no-op decorator just to signify that they're not > making it ugly with type annotations? You don't need to flag functions with a decorator to signify that you aren't using annotations. You just don't use annotations. Nobody (except maybe your boss) will force you to use annotations. Python will continue to work with them or without them. The only people who end up a little worse off are those who already use annotations for something else. But even they won't be *much* worse off, just a little: - Nobody will force them to use a type-checker or mypy, they can just keep using their annotations and *not* use a type-checker. - If they want to interoperate with a type-checker, they will have to take steps to tell the type-checker to skip their annotations. This might be a directive "# typing=OFF" or a "@notyping" decorator. Either way, it won't be onerous. - The worst case is if they want to use their existing annotations AND the new type-hints at the same time, for the same functions. In that case, they will have to find an alternative for their annotations, perhaps moving them into a decorator. -- Steven From brett at python.org Thu Jan 22 15:59:46 2015 From: brett at python.org (Brett Cannon) Date: Thu, 22 Jan 2015 14:59:46 +0000 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> Message-ID: <CAP1=2W6Jw6XzRq=O=RRm5JZ1Nq0MyJraVUMG+BzX2qwAnx3khA@mail.gmail.com> On Thu Jan 22 2015 at 6:19:38 AM Cem Karan <cfkaran2 at gmail.com> wrote: > > On Jan 21, 2015, at 11:47 PM, Guido van Rossum <guido at python.org> wrote: > > > On Wed, Jan 21, 2015 at 8:39 PM, Eugene Toder <eltoder at gmail.com> wrote: > > On Wed, Jan 21, 2015 at 8:25 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> > wrote: > > On 01/22/2015 01:50 PM, Guido van Rossum wrote: > > I've seen plenty of code that takes instances a certain class > > as argument and calls methods of that type but never constructs a new > instance > > nor does any of the other things for which you need the class name, > > > > If the only thing the name is needed for is static checking, > > I don't see what's so bad about not finding out until you > > actually run the static check. Adding those imports earlier > > doesn't make the code work any better or show up any more > > runtime bugs. > > > > > > Also, if static type checking succeeds, it will be integrated into many > IDEs/editors, so you will see type checker errors even before you get to > run the tests. > > > > So I think we're all agreed that the current proposal is fine! > > *grumble* ;P > > I know I keep arguing for a slightly different syntax so that its easy to > differentiate between type hints and other uses, but I do want to say that > I like the proposal. It seems quite practical to me. Would you at least > consider adding something to the proposal that makes it clear that a given > annotation is a type and not something else? > Guido already has: the typing module and its namespace along with the @no_type_hints decorator. Practicality beats Purity in this case and so adding any unnecessary overhead in the specification of the type hints is not going to fly. And your worry about all needing to flag non-type hints for annotations really only comes into play if the tool consuming them errors out or something when it comes across them. If it plays it safe and chooses unrecognized type hints as Any then any alternative use of annotations will just be ignored. I realize you don't like the loss in flexibility, but it is just a fact that function annotations are going to de-facto become type hint annotations in Python's future. They are the best use for annotations that we have come up with as a community and making the barrier to their use as low as possible puts it in the best interest of everyone to tighten the expectation of what annotations represent. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150122/46cc66df/attachment-0001.html> From edk141 at gmail.com Thu Jan 22 16:10:13 2015 From: edk141 at gmail.com (Ed Kellett) Date: Thu, 22 Jan 2015 15:10:13 +0000 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> <CAP1=2W59Pxr4j_Cx+Cvd+WQQZ4RznOSrEL9q_G2wZcB=F81=wQ@mail.gmail.com> Message-ID: <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> On Thu Jan 22 2015 at 14:53:56 Brett Cannon <brett at python.org> wrote: > > If you read one of Ed's earlier emails it sounds like he thinks you need > the @no_type_hints decorator even if you don't provide *any* annotations, > which is incorrect (the decorator is only to signal that annotations you > did specify are not type hints; not specifying any annotations signals the > same thing). Heck, the decorator isn't even technically necessary if the > linter or IDE you are using that can process the type hints chooses to > treat unrecognized annotations as Any instead of throwing an error or > something. > I feel like I've been significantly misunderstood. To be clear, I'm talking about using @no_type_checks or whatever the decorator is called for code I have that *already uses annotations*. Type checking will either fail, or become ubiquitous to the point that passing type checking is a prerequisite to being considered "good" for any Python module. Should the latter happen, I'll have to break my code for past or future Python versions, no? Personally, I'd like to believe that this proposal and the echochamber that's surrounded it has all been just a bad dream, but given that it's almost certainly going to make it into Python at this point, it'd be nice not to have to make good code bad just to maintain compatibility with it. Since type hints already depend on the 'typing' module anyway, what exactly would be wrong with the type checker defaulting off unless it is imported? Ed Kellett -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150122/170f0028/attachment.html> From rosuav at gmail.com Thu Jan 22 16:23:10 2015 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 23 Jan 2015 02:23:10 +1100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> <CAP1=2W59Pxr4j_Cx+Cvd+WQQZ4RznOSrEL9q_G2wZcB=F81=wQ@mail.gmail.com> <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> Message-ID: <CAPTjJmoirqgmDW6q62dNTaWiDpPGkgdft25_zmR66gDk7omZUw@mail.gmail.com> On Fri, Jan 23, 2015 at 2:10 AM, Ed Kellett <edk141 at gmail.com> wrote: > Type checking will either fail, or become ubiquitous to the point that > passing type checking is a prerequisite to being considered "good" for any > Python module. Should the latter happen, I'll have to break my code for past > or future Python versions, no? If that ever does happen, it won't be for a very VERY long time. So you would have the choice between breaking your code for <=3.4 or for >=4.7 or something. Is that really so likely? ChrisA From p.f.moore at gmail.com Thu Jan 22 16:31:33 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 22 Jan 2015 15:31:33 +0000 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> <CAP1=2W59Pxr4j_Cx+Cvd+WQQZ4RznOSrEL9q_G2wZcB=F81=wQ@mail.gmail.com> <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> Message-ID: <CACac1F_-YVsiH85oOU7DVpyHrrgqBK-0D3QihMUbU0zMwzWp9A@mail.gmail.com> On 22 January 2015 at 15:10, Ed Kellett <edk141 at gmail.com> wrote: > Type checking will either fail, or become ubiquitous to the point that > passing type checking is a prerequisite to being considered "good" for any > Python module. Should the latter happen, I'll have to break my code for past > or future Python versions, no? You would never be required to add type hints to your own code. I guess other projects could decide they will only accept contributions that pass the typechecker - but you don't have to contribute to such projects if you don't want to. At least that's my understanding. Personally, I doubt I'll ever use type hints, any more than I use tools like flake8. They don't suit the sort of code I typically write. Paul From guido at python.org Thu Jan 22 21:05:21 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 22 Jan 2015 12:05:21 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> <CAP1=2W59Pxr4j_Cx+Cvd+WQQZ4RznOSrEL9q_G2wZcB=F81=wQ@mail.gmail.com> <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> Message-ID: <CAP7+vJLmN_YwmHOAXzqm5Dga1aQY8ygu6Z86WFaDCM-JLTh6aQ@mail.gmail.com> On Thu, Jan 22, 2015 at 7:10 AM, Ed Kellett <edk141 at gmail.com> wrote: > On Thu Jan 22 2015 at 14:53:56 Brett Cannon <brett at python.org> wrote: >> >> If you read one of Ed's earlier emails it sounds like he thinks you need >> the @no_type_hints decorator even if you don't provide *any* annotations, >> which is incorrect (the decorator is only to signal that annotations you >> did specify are not type hints; not specifying any annotations signals the >> same thing). Heck, the decorator isn't even technically necessary if the >> linter or IDE you are using that can process the type hints chooses to >> treat unrecognized annotations as Any instead of throwing an error or >> something. >> > > I feel like I've been significantly misunderstood. To be clear, I'm > talking about using @no_type_checks or whatever the decorator is called for > code I have that *already uses annotations*. > Or you could just refrain from running the type checker. The checker is not built into CPython. It will be a separate program, run voluntarily, that points out issues, just like a linter. Support you run the pep8 program (a linter mostly for whitespace and naming conventions) and it tells you that your lines are longer than 79 chars. What are you going to do? Reformat your code to fit? You could, but nobody will force you. You could just ignore those warnings, or change pep8's configuration file to disable that message, or just *not run pep8 at all*. SImilar for other linters (e.g. pyflakes, pylint) that point out more substantial issues such as unused or undefined variables. Type checking will either fail, or become ubiquitous to the point that > passing type checking is a prerequisite to being considered "good" for any > Python module. Should the latter happen, I'll have to break my code for > past or future Python versions, no? > I doubt that is how it will work out. There's a reason we call it *Gradual typing*. You as a Python developer (or a team of collaborating developers for a given project) can choose where you think you will benefit from getting parts of your code type-checked, and where you don't think it is worth it. Again, this is the same as for linting tools. When you're writing a half-hour hack to scrape your old mailbox, you're probably not going to care about a type checker (you run your script and fix it until it works, and then you move on). When you're planning to write a 100,000 line application over a year's time with a team of six programmers you might agree that your code follows the strictest style guides, passes a linter with zero warnings, or, in the future, that it type-checks without warnings. (You should probably also agree on unit testing and integration testing. And you need a code review process. And revision control. But I digress.) If you're in the latter situation there are usually some compromises you have to make -- e.g. your team may not all agree on the line length, but it's better to pick one and stick to it than to continually bicker about it. Similar for type hints -- you may have to agree on how important they are for your project and and adjust your habits accordingly. > Personally, I'd like to believe that this proposal and the echochamber > that's surrounded it has all been just a bad dream, but given that it's > almost certainly going to make it into Python at this point, it'd be nice > not to have to make good code bad just to maintain compatibility with it. > This paragraph (and a some words you wrote earlier) feels really offensive to me. I am doing the best I can. I have listened to an *enormous* amount of feedback and if you read back earlier discussions (going back to I believe September or October 2014) or skim the typehinting issue tracker <https://github.com/ambv/typehinting/issues> you'll find that I am taking the feedback very seriously. Having got that off my mind, the vast majority of Python code that's been written does not use annotations at all, so there is no reason to change it at all. For code that does use annotation, *nothing* will break at run time -- the code will work unchanged, without any deprecations or slowdowns. The only situation where there may be a desire to change something (the minimal thing being to add a "# type: OFF" comment at the top of a module) is when a library that uses annotations for non-type-hinting purposes is used by an application (or another library) that wants to use type hints, and the user who is running the type checker cannot live with the noise output spewed by the type checker for that library. Even there, the problem could also be solved by having a way to configure the type checker to ignore annotations in certain packages (or to only check code in specific directories). > Since type hints already depend on the 'typing' module anyway, what > exactly would be wrong with the type checker defaulting off unless it is > imported? > I've considered that, but I find it limiting. It is quite likely that a module can be fully annotated with type hints without ever needing to reference the typing module. For example, all argument types may be simple built-in types (strings and numbers) or user-defined classes. I think it would be unexpected if type hints that were present in a module would be ignored by the type checker unless the typing module was imported. (Also, perhaps ironically, most linters will complain about modules that are imported but not used. :-) But this is an area where we may be able to cater to different preferences by passing the type checker different flags or configurations. So I don't want to make a big thing out of this, and if people think it's useful, this can well be an option. Finally. In the distant future there may be more agreement about the use of annotations for type hints, and users will start asking library authors to change their code. But that's no different than other evolution of the language -- it won't happen overnight, you will get ample warning (DeprecationWarning :-), and the principle of supply and demand will apply. Heck, if you're not using Python 3, you're safe until 2020. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150122/7c73aa98/attachment-0001.html> From abarnert at yahoo.com Thu Jan 22 22:34:57 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 22 Jan 2015 13:34:57 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJLmN_YwmHOAXzqm5Dga1aQY8ygu6Z86WFaDCM-JLTh6aQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> <CAP1=2W59Pxr4j_Cx+Cvd+WQQZ4RznOSrEL9q_G2wZcB=F81=wQ@mail.gmail.com> <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> <CAP7+vJLmN_YwmHOAXzqm5Dga1aQY8ygu6Z86WFaDCM-JLTh6aQ@mail.gmail.com> Message-ID: <57DA5869-3C97-40AC-B1E3-0F0F5A6800E8@yahoo.com> On Jan 22, 2015, at 12:05, Guido van Rossum <guido at python.org> wrote: > But this is an area where we may be able to cater to different preferences by passing the type checker different flags or configurations. So I don't want to make a big thing out of this, and if people think it's useful, this can well be an option. I agree with everything else you said, but I'm not sure I understand this paragraph. The PEP isn't specifying anything about type checkers except how they interpret type annotations. So isn't such a flag just a quality of implementation issue on (hopefully-competing) checkers and IDEs and other tools? Are you just suggesting that such a flag can be added to MyPy (which will likely encourage other checkers to do the same unless nobody finds it useful)? Or are you planning to expand the spec with suggestions for configuration that type checking tools should consider offering? From edk141 at gmail.com Thu Jan 22 22:45:41 2015 From: edk141 at gmail.com (Ed Kellett) Date: Thu, 22 Jan 2015 21:45:41 +0000 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> <CAP1=2W59Pxr4j_Cx+Cvd+WQQZ4RznOSrEL9q_G2wZcB=F81=wQ@mail.gmail.com> <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> <CAP7+vJLmN_YwmHOAXzqm5Dga1aQY8ygu6Z86WFaDCM-JLTh6aQ@mail.gmail.com> Message-ID: <CABmzr0h+RZs=9ri-fiQjrZeXNTnsJP5LcZhZzBqRpdoYoHjW+A@mail.gmail.com> On Thu Jan 22 2015 at 20:05:41 Guido van Rossum <guido at python.org> wrote: > > This paragraph (and a some words you wrote earlier) feels really offensive > to me. I am doing the best I can. I have listened to an *enormous* amount > of feedback and if you read back earlier discussions (going back to I > believe September or October 2014) or skim the typehinting issue tracker > <https://github.com/ambv/typehinting/issues> you'll find that I am taking > the feedback very seriously. > I'm sorry. I don't mean to minimize the work you've put into this, and in any case there was no call to speak so unconstructively. > The only situation where there may be a desire to change something (the > minimal thing being to add a "# type: OFF" comment at the top of a module) > is when a library that uses annotations for non-type-hinting purposes is > used by an application (or another library) that wants to use type hints, > and the user who is running the type checker cannot live with the noise > output spewed by the type checker for that library. > This is more or less the situation I imagined. Or, more problematic - the user running the type checker might assume the library is no good because it sets off the warnings. > Finally. In the distant future there may be more agreement about the use > of annotations for type hints, and users will start asking library authors > to change their code. But that's no different than other evolution of the > language -- it won't happen overnight, you will get ample warning > (DeprecationWarning :-), and the principle of supply and demand will apply. > Heck, if you're not using Python 3, you're safe until 2020. > I understand that it's a long way off, but in my view that'd be a regression: I know not many things are using annotations at the moment, but where they are used they fit really well (extra information for command-line-ifying a function being the one I have most experience with). It seems a shame to throw away something so neat when it feels as if the different use cases could coexist peacefully. Thanks for your time, Ed Kellett -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150122/ef5efcf3/attachment.html> From guido at python.org Fri Jan 23 00:07:04 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 22 Jan 2015 15:07:04 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <57DA5869-3C97-40AC-B1E3-0F0F5A6800E8@yahoo.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> <CAP1=2W59Pxr4j_Cx+Cvd+WQQZ4RznOSrEL9q_G2wZcB=F81=wQ@mail.gmail.com> <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> <CAP7+vJLmN_YwmHOAXzqm5Dga1aQY8ygu6Z86WFaDCM-JLTh6aQ@mail.gmail.com> <57DA5869-3C97-40AC-B1E3-0F0F5A6800E8@yahoo.com> Message-ID: <CAP7+vJ+9uXJdmuAWCjNc3KShOQvrhMgSZ1NaoQcKYr3cZkn31g@mail.gmail.com> On Thu, Jan 22, 2015 at 1:34 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > On Jan 22, 2015, at 12:05, Guido van Rossum <guido at python.org> wrote: > > > But this is an area where we may be able to cater to different > preferences by passing the type checker different flags or configurations. > So I don't want to make a big thing out of this, and if people think it's > useful, this can well be an option. > > I agree with everything else you said, but I'm not sure I understand this > paragraph. The PEP isn't specifying anything about type checkers except how > they interpret type annotations. So isn't such a flag just a quality of > implementation issue on (hopefully-competing) checkers and IDEs and other > tools? > > Are you just suggesting that such a flag can be added to MyPy (which will > likely encourage other checkers to do the same unless nobody finds it > useful)? Or are you planning to expand the spec with suggestions for > configuration that type checking tools should consider offering? I guess it should just be a mypy flag. But I think the PEP should have a few reassuring words for users like Ed who worry about this. See: https://github.com/ambv/typehinting/issues/53 -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150122/21f7bce4/attachment.html> From guido at python.org Fri Jan 23 00:19:37 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 22 Jan 2015 15:19:37 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CABmzr0h+RZs=9ri-fiQjrZeXNTnsJP5LcZhZzBqRpdoYoHjW+A@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> <CAP1=2W59Pxr4j_Cx+Cvd+WQQZ4RznOSrEL9q_G2wZcB=F81=wQ@mail.gmail.com> <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> <CAP7+vJLmN_YwmHOAXzqm5Dga1aQY8ygu6Z86WFaDCM-JLTh6aQ@mail.gmail.com> <CABmzr0h+RZs=9ri-fiQjrZeXNTnsJP5LcZhZzBqRpdoYoHjW+A@mail.gmail.com> Message-ID: <CAP7+vJKctVBZ0xnENZh5-NqJ1YK379nw410PNigG4eBFU8Oc3Q@mail.gmail.com> On Thu, Jan 22, 2015 at 1:45 PM, Ed Kellett <edk141 at gmail.com> wrote: > On Thu Jan 22 2015 at 20:05:41 Guido van Rossum <guido at python.org> wrote: >> >> This paragraph (and a some words you wrote earlier) feels really >> offensive to me. I am doing the best I can. I have listened to an >> *enormous* amount of feedback and if you read back earlier discussions >> (going back to I believe September or October 2014) or skim the typehinting >> issue tracker <https://github.com/ambv/typehinting/issues> you'll find >> that I am taking the feedback very seriously. >> > > I'm sorry. I don't mean to minimize the work you've put into this, and in > any case there was no call to speak so unconstructively. > Apology accepted! > The only situation where there may be a desire to change something (the >> minimal thing being to add a "# type: OFF" comment at the top of a module) >> is when a library that uses annotations for non-type-hinting purposes is >> used by an application (or another library) that wants to use type hints, >> and the user who is running the type checker cannot live with the noise >> output spewed by the type checker for that library. >> > > This is more or less the situation I imagined. Or, more problematic - the > user running the type checker might assume the library is no good because > it sets off the warnings. > Hopefully it'll be a while before users who are that clueless will figure out how to run the type checker. :-) > Finally. In the distant future there may be more agreement about the use >> of annotations for type hints, and users will start asking library authors >> to change their code. But that's no different than other evolution of the >> language -- it won't happen overnight, you will get ample warning >> (DeprecationWarning :-), and the principle of supply and demand will apply. >> Heck, if you're not using Python 3, you're safe until 2020. >> > > I understand that it's a long way off, but in my view that'd be a > regression: I know not many things are using annotations at the moment, but > where they are used they fit really well (extra information for > command-line-ifying a function being the one I have most experience with). > It seems a shame to throw away something so neat when it feels as if the > different use cases could coexist peacefully. > Yes, I am well aware that it's a regression. Regressions are taken seriously; see e.g. PEP 4, which describes the deprecation process for modules. It's much the same for language features. But that doesn't mean we don't sometimes decide to change our mind. (And this isn't limited to major version number transitions like Python 2 -> 3.) This is one of those cases. I expect (or hope?) that at some point in the distant future users who see an annotation will know that it is a type hint, without having to figure out whether may be there is a decorator or some other less-visible mechanism (e.g. a metaclass, shudder) that gives it a different meaning. I believe that this is a more desirable future than one in which type hints have to compete with other meta-information for arguments for their syntactic position. At the same time -- let me emphasize this one more time -- I hope there will never be a time in the future where type hints are mandatory or otherwise always expected to exist. (If you want that you should probably switch to Java sooner rather than later. :-) And lastly, such a future is a long way off, and in 3.5 your code won't break. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150122/eea9d6b7/attachment-0001.html> From chris.barker at noaa.gov Fri Jan 23 01:40:14 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 22 Jan 2015 16:40:14 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality Message-ID: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> Hi folks, After much discussion on this list, I have written up a PEP, and it is ready for review (see below) It is also here: https://www.python.org/dev/peps/pep-0485/ That version is not quite up to date just yet, so please refer to the one enclosed in this email for now. I am managing both the PEP and a sample implementation and tests in gitHub here: https://github.com/PythonCHB/close_pep Please go there if you want to try it out, add some tests, etc. Pull requests welcomed for code, tests, or PEP editing. A quick summary of the decisions I made, and what I think are the open discussion points: The focus is on relative tolerance, but with an optional absolute tolerance, primarily to be used near zero, but it also allows it to be used as a plain absolute difference check. It is using an asymmetric test -- that is, the tolerance is computed relative to one of the arguments. It is perhaps surprising and confusing that you may get a different result if you reverse the arguments, but in this discussion it became clear that there were some use-cases where it was helpful to know exactly what the tolerance is computed relative too, and that in most use cases, if just doesn't matter. I hope this is adequately explained in the PEP. We could add a flag to set a symmetric test (I'd go with what boost calls the "strong" test), but I'd rather not -- it just confuses things, and I expect users will tend to use defaults anyway. It is designed to work mostly with floats, but also supports Integer, Decimal, Fraction, and Complex. I'm not really thrilled with that, though, it turns out to be not quite as easy to duck-type it as I had hoped. To really do it right, there would have to be more switching on type in the code, which I think is ugly to write -- contributions, opinions welcome on this. I used 1e-8 as a default relative tolerance -- arbitrarily because that's about half of the decimal digits in a python float -- suggestions welcome. Other than that, of course, we can bike-shed the names of the function and the parameters. ;-) Fire away! -Chris PEP: 485 Title: A Function for testing approximate equality Version: $Revision$ Last-Modified: $Date$ Author: Christopher Barker <Chris.Barker at noaa.gov> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 20-Jan-2015 Python-Version: 3.5 Post-History: Abstract ======== This PEP proposes the addition of a function to the standard library that determines whether one value is approximately equal or "close" to another value. Rationale ========= Floating point values contain limited precision, which results in their being unable to exactly represent some values, and for error to accumulate with repeated computation. As a result, it is common advice to only use an equality comparison in very specific situations. Often a inequality comparison fits the bill, but there are times (often in testing) where the programmer wants to determine whether a computed value is "close" to an expected value, without requiring them to be exactly equal. This is common enough, particularly in testing, and not always obvious how to do it, so it would be useful addition to the standard library. Existing Implementations ------------------------ The standard library includes the ``unittest.TestCase.assertAlmostEqual`` method, but it: * Is buried in the unittest.TestCase class * Is an assertion, so you can't use it as a general test (easily) * Uses number of decimal digits or an absolute delta, which are particular use cases that don't provide a general relative error. The numpy package has the ``allclose()`` and ``isclose()`` functions. The statistics package tests include an implementation, used for its unit tests. One can also find discussion and sample implementations on Stack Overflow, and other help sites. These existing implementations indicate that this is a common need, and not trivial to write oneself, making it a candidate for the standard library. Proposed Implementation ======================= NOTE: this PEP is the result of an extended discussion on the python-ideas list [1]_. The new function will have the following signature:: is_close_to(actual, expected, tol=1e-8, abs_tol=0.0) ``actual``: is the value that has been computed, measured, etc. ``expected``: is the "known" value. ``tol``: is the relative tolerance -- it is the amount of error allowed, relative to the magnitude of the expected value. ``abs_tol``: is an minimum absolute tolerance level -- useful for comparisons near zero. Modulo error checking, etc, the function will return the result of:: abs(expected-actual) <= max(tol*expected, abs_tol) Handling of non-finite numbers ------------------------------ The IEEE 754 special values of NaN, inf, and -inf will be handled according to IEEE rules. Specifically, NaN is not considered close to any other value, including NaN. inf and -inf are only considered close to themselves. Non-float types --------------- The primary use-case is expected to be floating point numbers. However, users may want to compare other numeric types similarly. In theory, it should work for any type that supports ``abs()``, comparisons, and subtraction. The code will be written and tested to accommodate these types: * ``Decimal``: for Decimal, the tolerance must be set to a Decimal type. * ``int`` * ``Fraction`` * ``complex``: for complex, ``abs(z)`` will be used for scaling and comparison. Behavior near zero ------------------ Relative comparison is problematic if either value is zero. In this case, the difference is relative to zero, and thus will always be smaller than the prescribed tolerance. To handle this case, an optional parameter, ``abs_tol`` (default 0.0) can be used to set a minimum tolerance to be used in the case of very small relative tolerance. That is, the values will be considered close if:: abs(a-b) <= abs(tol*expected) or abs(a-b) <= abs_tol If the user sets the rel_tol parameter to 0.0, then only the absolute tolerance will effect the result, so this function provides an absolute tolerance check as well. A sample implementation is available (as of Jan 22, 2015) on gitHub: https://github.com/PythonCHB/close_pep/blob/master/is_close_to.py Relative Difference =================== There are essentially two ways to think about how close two numbers are to each-other: absolute difference: simply ``abs(a-b)``, and relative difference: ``abs(a-b)/scale_factor`` [2]_. The absolute difference is trivial enough that this proposal focuses on the relative difference. Usually, the scale factor is some function of the values under consideration, for instance: 1) The absolute value of one of the input values 2) The maximum absolute value of the two 3) The minimum absolute value of the two. 4) The arithmetic mean of the two Symmetry -------- A relative comparison can be either symmetric or non-symmetric. For a symmetric algorithm: ``is_close_to(a,b)`` is always equal to ``is_close_to(b,a)`` This is an appealing consistency -- it mirrors the symmetry of equality, and is less likely to confuse people. However, often the question at hand is: "Is this computed or measured value within some tolerance of a known value?" In this case, the user wants the relative tolerance to be specifically scaled against the known value. It is also easier for the user to reason about. This proposal uses this asymmetric test to allow this specific definition of relative tolerance. Example: For the question: "Is the value of a within x% of b?", Using b to scale the percent error clearly defines the result. However, as this approach is not symmetric, a may be within 10% of b, but b is not within x% of a. Consider the case:: a = 9.0 b = 10.0 The difference between a and b is 1.0. 10% of a is 0.9, so b is not within 10% of a. But 10% of b is 1.0, so a is within 10% of b. Casual users might reasonably expect that if a is close to b, then b would also be close to a. However, in the common cases, the tolerance is quite small and often poorly defined, i.e. 1e-8, defined to only one significant figure, so the result will be very similar regardless of the order of the values. And if the user does care about the precise result, s/he can take care to always pass in the two parameters in sorted order. This proposed implementation uses asymmetric criteria with the scaling value clearly identified. Expected Uses ============= The primary expected use case is various forms of testing -- "are the results computed near what I expect as a result?" This sort of test may or may not be part of a formal unit testing suite. The function might be used also to determine if a measured value is within an expected value. Inappropriate uses ------------------ One use case for floating point comparison is testing the accuracy of a numerical algorithm. However, in this case, the numerical analyst ideally would be doing careful error propagation analysis, and should understand exactly what to test for. It is also likely that ULP (Unit in the Last Place) comparison may be called for. While this function may prove useful in such situations, It is not intended to be used in that way. Other Approaches ================ ``unittest.TestCase.assertAlmostEqual`` --------------------------------------- ( https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertAlmostEqual ) Tests that values are approximately (or not approximately) equal by computing the difference, rounding to the given number of decimal places (default 7), and comparing to zero. This method was not selected for this proposal, as the use of decimal digits is a specific, not generally useful or flexible test. numpy ``is_close()`` -------------------- http://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.isclose.html The numpy package provides the vectorized functions is_close() and all_close, for similar use cases as this proposal: ``isclose(a, b, rtol=1e-05, atol=1e-08, equal_nan=False)`` Returns a boolean array where two arrays are element-wise equal within a tolerance. The tolerance values are positive, typically very small numbers. The relative difference (rtol * abs(b)) and the absolute difference atol are added together to compare against the absolute difference between a and b In this approach, the absolute and relative tolerance are added together, rather than the ``or`` method used in this proposal. This is computationally more simple, and if relative tolerance is larger than the absolute tolerance, then the addition will have no effect. But if the absolute and relative tolerances are of similar magnitude, then the allowed difference will be about twice as large as expected. Also, if the value passed in are small compared to the absolute tolerance, then the relative tolerance will be completely swamped, perhaps unexpectedly. This is why, in this proposal, the absolute tolerance defaults to zero -- the user will be required to choose a value appropriate for the values at hand. Boost floating-point comparison ------------------------------- The Boost project ( [3]_ ) provides a floating point comparison function. Is is a symetric approach, with both "weak" (larger of the two relative errors) and "strong" (smaller of the two relative errors) options. It was decided that a method that clearly defined which value was used to scale the relative error would be more appropriate for the standard library. References ========== .. [1] Python-ideas list discussion thread (https://mail.python.org/pipermail/python-ideas/2015-January/030947.html) .. [2] Wikipedaia page on relative difference (http://en.wikipedia.org/wiki/Relative_change_and_difference) .. [3] Boost project floating-point comparison algorithms ( http://www.boost.org/doc/libs/1_35_0/libs/test/doc/components/test_tools/floating_point_comparison.html ) Copyright ========= This document has been placed in the public domain. -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150122/58672dbe/attachment-0001.html> From apalala at gmail.com Fri Jan 23 01:55:11 2015 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Thu, 22 Jan 2015 20:25:11 -0430 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJLmN_YwmHOAXzqm5Dga1aQY8ygu6Z86WFaDCM-JLTh6aQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <m9bpcb$eq7$1@ger.gmane.org> <CAP7+vJLos6pm4iCUV-wmp58BviPMHtVSUsDfjeo8X8NhtEMA5g@mail.gmail.com> <m9bqe0$2ub$1@ger.gmane.org> <CAP7+vJLWt1rME0YdBGTv_+VqQns2E=O9iEysf--2Yo=sUcvsDQ@mail.gmail.com> <m9das1$vtc$1@ger.gmane.org> <CAP7+vJKRkLW=dPw+eV9WUdWt0eUv5dSZ5ZJfKKjO4PSo6nF2gA@mail.gmail.com> <m9h5gk$2a6$1@ger.gmane.org> <CAP7+vJ+Y2hDkeSn6rMmKqk0HZK0sm8cNpZsCw6kazLwuZcByfA@mail.gmail.com> <m9nkjn$p3d$1@ger.gmane.org> <CAP7+vJJ4GoSSdZZi+0MaaBtmXXexsZTQU_8XyG4wmN1qrywK3A@mail.gmail.com> <CAP7+vJKLJpFMFK3BviRsKsi72m2w8X8F68tezm==gww+oYzP_w@mail.gmail.com> <54C025B4.6020805@canterbury.ac.nz> <CAP7+vJKrN4f0__3nFmse+Y-8v2YBoGQWo+aneRzJyuix+kz+Wg@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> <CAP1=2W59Pxr4j_Cx+Cvd+WQQZ4RznOSrEL9q_G2wZcB=F81=wQ@mail.gmail.com> <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> <CAP7+vJLmN_YwmHOAXzqm5Dga1aQY8ygu6Z86WFaDCM-JLTh6aQ@mail.gmail.com> Message-ID: <CAN1YFWvA5fO9X9=xoxu_n6__90fSrog8tbgGbMXsF3cXwHQv+w@mail.gmail.com> On Thu, Jan 22, 2015 at 3:35 PM, Guido van Rossum <guido at python.org> wrote: > The only situation where there may be a desire to change something (the > minimal thing being to add a "# type: OFF" comment at the top of a module) > is when a library that uses annotations for non-type-hinting purposes is > used by an application (or another library) that wants to use type hints, > and the user who is running the type checker cannot live with the noise > output spewed by the type checker for that library. > > Even there, the problem could also be solved by having a way to configure > the type checker to ignore annotations in certain packages (or to only > check code in specific directories). > Having library authors change their code because type hinting was introduced would be unacceptable. The burden must all on whomever wants to benefit from type checking. > > >> Since type hints already depend on the 'typing' module anyway, what >> exactly would be wrong with the type checker defaulting off unless it is >> imported? >> > > I've considered that, but I find it limiting. It is quite likely that a > module can be fully annotated with type hints without ever needing to > reference the typing module. For example, all argument types may be simple > built-in types (strings and numbers) or user-defined classes. I think it > would be unexpected if type hints that were present in a module would be > ignored by the type checker unless the typing module was imported. (Also, > perhaps ironically, most linters will complain about modules that are > imported but not used. :-) > "import typing" is a nice and easy compromise, very much in the pythonic way, in my opinion. If you want your module type checked: import typing. It will not be checked otherwise. It's simple, backwards compatible, very little effort for those wanting to jump into PEP484, and zero pain for those who don't. Cheers, -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150122/4ac9cf7d/attachment.html> From ethan at stoneleaf.us Fri Jan 23 01:59:21 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 22 Jan 2015 16:59:21 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> Message-ID: <54C19CE9.5000200@stoneleaf.us> On 01/22/2015 04:40 PM, Chris Barker wrote: > > After much discussion on this list, I have written up a PEP, and it is ready > for review (see below) Thanks! Very nice. > It is using an asymmetric test Good - Ron convinced me that was the better way > However, as this approach is not symmetric, a may be within 10% of b, > but b is not within x% of a. Consider the case:: Instead of x%, how about 10% ? ;) -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150122/f6546683/attachment.sig> From ethan at stoneleaf.us Fri Jan 23 02:04:07 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 22 Jan 2015 17:04:07 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAN1YFWvA5fO9X9=xoxu_n6__90fSrog8tbgGbMXsF3cXwHQv+w@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> <CAP1=2W59Pxr4j_Cx+Cvd+WQQZ4RznOSrEL9q_G2wZcB=F81=wQ@mail.gmail.com> <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> <CAP7+vJLmN_YwmHOAXzqm5Dga1aQY8ygu6Z86WFaDCM-JLTh6aQ@mail.gmail.com> <CAN1YFWvA5fO9X9=xoxu_n6__90fSrog8tbgGbMXsF3cXwHQv+w@mail.gmail.com> Message-ID: <54C19E07.2030500@stoneleaf.us> On 01/22/2015 04:55 PM, Juancarlo A?ez wrote: > On Thu, Jan 22, 2015 at 3:35 PM, Guido van Rossum wrote: >> >> Even there, the problem could also be solved by having a way to configure >> the type checker to ignore annotations in certain packages (or to only check >> code in specific directories). > > Having library authors change their code because type hinting was introduced would be unacceptable. Annoying, yes. Unacceptable? No. As the language progresses library authors occasionally have to make adjustments (and I am one, so I feel the pain). > The burden must fall on whomever wants to benefit from type checking. Indeed -- if someone does want type checking, but library xyz is using annotations for something else, then a way (provided by the type checker) to skip certain modules would be good. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150122/96c2f7ad/attachment.sig> From apalala at gmail.com Fri Jan 23 02:24:34 2015 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Thu, 22 Jan 2015 20:54:34 -0430 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> Message-ID: <CAN1YFWvjGj=Gb+KH=FGz_25Rik6eeVs5YJGJ-P3E-_ncTusasg@mail.gmail.com> +1 On Thu, Jan 22, 2015 at 8:10 PM, Chris Barker <chris.barker at noaa.gov> wrote: > Hi folks, > > After much discussion on this list, I have written up a PEP, and it is > ready for review (see below) > > It is also here: https://www.python.org/dev/peps/pep-0485/ > > That version is not quite up to date just yet, so please refer to the one > enclosed in this email for now. > > I am managing both the PEP and a sample implementation and tests in gitHub > here: > > https://github.com/PythonCHB/close_pep > > Please go there if you want to try it out, add some tests, etc. Pull > requests welcomed for code, tests, or PEP editing. > > A quick summary of the decisions I made, and what I think are the open > discussion points: > > The focus is on relative tolerance, but with an optional absolute > tolerance, primarily to be used near zero, but it also allows it to be used > as a plain absolute difference check. > > It is using an asymmetric test -- that is, the tolerance is computed > relative to one of the arguments. It is perhaps surprising and confusing > that you may get a different result if you reverse the arguments, but in > this discussion it became clear that there were some use-cases where it was > helpful to know exactly what the tolerance is computed relative too, and > that in most use cases, if just doesn't matter. I hope this is adequately > explained in the PEP. We could add a flag to set a symmetric test (I'd go > with what boost calls the "strong" test), but I'd rather not -- it just > confuses things, and I expect users will tend to use defaults anyway. > > It is designed to work mostly with floats, but also supports Integer, > Decimal, Fraction, and Complex. I'm not really thrilled with that, though, > it turns out to be not quite as easy to duck-type it as I had hoped. To > really do it right, there would have to be more switching on type in the > code, which I think is ugly to write -- contributions, opinions welcome on > this. > > I used 1e-8 as a default relative tolerance -- arbitrarily because that's > about half of the decimal digits in a python float -- suggestions welcome. > > Other than that, of course, we can bike-shed the names of the function and > the parameters. ;-) > > Fire away! > > -Chris > > > PEP: 485 > Title: A Function for testing approximate equality > Version: $Revision$ > Last-Modified: $Date$ > Author: Christopher Barker <Chris.Barker at noaa.gov> > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 20-Jan-2015 > Python-Version: 3.5 > Post-History: > > > Abstract > ======== > > This PEP proposes the addition of a function to the standard library > that determines whether one value is approximately equal or "close" > to another value. > > > Rationale > ========= > > Floating point values contain limited precision, which results in > their being unable to exactly represent some values, and for error to > accumulate with repeated computation. As a result, it is common > advice to only use an equality comparison in very specific situations. > Often a inequality comparison fits the bill, but there are times > (often in testing) where the programmer wants to determine whether a > computed value is "close" to an expected value, without requiring them > to be exactly equal. This is common enough, particularly in testing, > and not always obvious how to do it, so it would be useful addition to > the standard library. > > > Existing Implementations > ------------------------ > > The standard library includes the > ``unittest.TestCase.assertAlmostEqual`` method, but it: > > * Is buried in the unittest.TestCase class > > * Is an assertion, so you can't use it as a general test (easily) > > * Uses number of decimal digits or an absolute delta, which are > particular use cases that don't provide a general relative error. > > The numpy package has the ``allclose()`` and ``isclose()`` functions. > > The statistics package tests include an implementation, used for its > unit tests. > > One can also find discussion and sample implementations on Stack > Overflow, and other help sites. > > These existing implementations indicate that this is a common need, > and not trivial to write oneself, making it a candidate for the > standard library. > > > Proposed Implementation > ======================= > > NOTE: this PEP is the result of an extended discussion on the > python-ideas list [1]_. > > The new function will have the following signature:: > > is_close_to(actual, expected, tol=1e-8, abs_tol=0.0) > > ``actual``: is the value that has been computed, measured, etc. > > ``expected``: is the "known" value. > > ``tol``: is the relative tolerance -- it is the amount of error > allowed, relative to the magnitude of the expected value. > > ``abs_tol``: is an minimum absolute tolerance level -- useful for > comparisons near zero. > > Modulo error checking, etc, the function will return the result of:: > > abs(expected-actual) <= max(tol*expected, abs_tol) > > > Handling of non-finite numbers > ------------------------------ > > The IEEE 754 special values of NaN, inf, and -inf will be handled > according to IEEE rules. Specifically, NaN is not considered close to > any other value, including NaN. inf and -inf are only considered close > to themselves. > > > Non-float types > --------------- > > The primary use-case is expected to be floating point numbers. > However, users may want to compare other numeric types similarly. In > theory, it should work for any type that supports ``abs()``, > comparisons, and subtraction. The code will be written and tested to > accommodate these types: > > * ``Decimal``: for Decimal, the tolerance must be set to a Decimal type. > > * ``int`` > > * ``Fraction`` > > * ``complex``: for complex, ``abs(z)`` will be used for scaling and > comparison. > > > Behavior near zero > ------------------ > > Relative comparison is problematic if either value is zero. In this > case, the difference is relative to zero, and thus will always be > smaller than the prescribed tolerance. To handle this case, an > optional parameter, ``abs_tol`` (default 0.0) can be used to set a > minimum tolerance to be used in the case of very small relative > tolerance. That is, the values will be considered close if:: > > abs(a-b) <= abs(tol*expected) or abs(a-b) <= abs_tol > > If the user sets the rel_tol parameter to 0.0, then only the absolute > tolerance will effect the result, so this function provides an > absolute tolerance check as well. > > A sample implementation is available (as of Jan 22, 2015) on gitHub: > > https://github.com/PythonCHB/close_pep/blob/master/is_close_to.py > > > Relative Difference > =================== > > There are essentially two ways to think about how close two numbers > are to each-other: absolute difference: simply ``abs(a-b)``, and > relative difference: ``abs(a-b)/scale_factor`` [2]_. The absolute > difference is trivial enough that this proposal focuses on the > relative difference. > > Usually, the scale factor is some function of the values under > consideration, for instance: > > 1) The absolute value of one of the input values > > 2) The maximum absolute value of the two > > 3) The minimum absolute value of the two. > > 4) The arithmetic mean of the two > > > Symmetry > -------- > > A relative comparison can be either symmetric or non-symmetric. For a > symmetric algorithm: > > ``is_close_to(a,b)`` is always equal to ``is_close_to(b,a)`` > > This is an appealing consistency -- it mirrors the symmetry of > equality, and is less likely to confuse people. However, often the > question at hand is: > > "Is this computed or measured value within some tolerance of a known > value?" > > In this case, the user wants the relative tolerance to be specifically > scaled against the known value. It is also easier for the user to > reason about. > > This proposal uses this asymmetric test to allow this specific > definition of relative tolerance. > > Example: > > For the question: "Is the value of a within x% of b?", Using b to > scale the percent error clearly defines the result. > > However, as this approach is not symmetric, a may be within 10% of b, > but b is not within x% of a. Consider the case:: > > a = 9.0 > b = 10.0 > > The difference between a and b is 1.0. 10% of a is 0.9, so b is not > within 10% of a. But 10% of b is 1.0, so a is within 10% of b. > > Casual users might reasonably expect that if a is close to b, then b > would also be close to a. However, in the common cases, the tolerance > is quite small and often poorly defined, i.e. 1e-8, defined to only > one significant figure, so the result will be very similar regardless > of the order of the values. And if the user does care about the > precise result, s/he can take care to always pass in the two > parameters in sorted order. > > This proposed implementation uses asymmetric criteria with the scaling > value clearly identified. > > > Expected Uses > ============= > > The primary expected use case is various forms of testing -- "are the > results computed near what I expect as a result?" This sort of test > may or may not be part of a formal unit testing suite. > > The function might be used also to determine if a measured value is > within an expected value. > > > Inappropriate uses > ------------------ > > One use case for floating point comparison is testing the accuracy of > a numerical algorithm. However, in this case, the numerical analyst > ideally would be doing careful error propagation analysis, and should > understand exactly what to test for. It is also likely that ULP (Unit > in the Last Place) comparison may be called for. While this function > may prove useful in such situations, It is not intended to be used in > that way. > > > Other Approaches > ================ > > ``unittest.TestCase.assertAlmostEqual`` > --------------------------------------- > > ( > https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertAlmostEqual > ) > > Tests that values are approximately (or not approximately) equal by > computing the difference, rounding to the given number of decimal > places (default 7), and comparing to zero. > > This method was not selected for this proposal, as the use of decimal > digits is a specific, not generally useful or flexible test. > > numpy ``is_close()`` > -------------------- > > http://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.isclose.html > > The numpy package provides the vectorized functions is_close() and > all_close, for similar use cases as this proposal: > > ``isclose(a, b, rtol=1e-05, atol=1e-08, equal_nan=False)`` > > Returns a boolean array where two arrays are element-wise equal > within a tolerance. > > The tolerance values are positive, typically very small numbers. > The relative difference (rtol * abs(b)) and the absolute > difference atol are added together to compare against the > absolute difference between a and b > > In this approach, the absolute and relative tolerance are added > together, rather than the ``or`` method used in this proposal. This is > computationally more simple, and if relative tolerance is larger than > the absolute tolerance, then the addition will have no effect. But if > the absolute and relative tolerances are of similar magnitude, then > the allowed difference will be about twice as large as expected. > > Also, if the value passed in are small compared to the absolute > tolerance, then the relative tolerance will be completely swamped, > perhaps unexpectedly. > > This is why, in this proposal, the absolute tolerance defaults to zero > -- the user will be required to choose a value appropriate for the > values at hand. > > > Boost floating-point comparison > ------------------------------- > > The Boost project ( [3]_ ) provides a floating point comparison > function. Is is a symetric approach, with both "weak" (larger of the > two relative errors) and "strong" (smaller of the two relative errors) > options. > > It was decided that a method that clearly defined which value was used > to scale the relative error would be more appropriate for the standard > library. > > References > ========== > > .. [1] Python-ideas list discussion thread > ( > https://mail.python.org/pipermail/python-ideas/2015-January/030947.html) > > .. [2] Wikipedaia page on relative difference > (http://en.wikipedia.org/wiki/Relative_change_and_difference) > > .. [3] Boost project floating-point comparison algorithms > ( > http://www.boost.org/doc/libs/1_35_0/libs/test/doc/components/test_tools/floating_point_comparison.html > ) > > > Copyright > ========= > > This document has been placed in the public domain. > > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150122/c57e6a56/attachment-0001.html> From apalala at gmail.com Fri Jan 23 02:27:14 2015 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Thu, 22 Jan 2015 20:57:14 -0430 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <54C19E07.2030500@stoneleaf.us> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> <CAP1=2W59Pxr4j_Cx+Cvd+WQQZ4RznOSrEL9q_G2wZcB=F81=wQ@mail.gmail.com> <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> <CAP7+vJLmN_YwmHOAXzqm5Dga1aQY8ygu6Z86WFaDCM-JLTh6aQ@mail.gmail.com> <CAN1YFWvA5fO9X9=xoxu_n6__90fSrog8tbgGbMXsF3cXwHQv+w@mail.gmail.com> <54C19E07.2030500@stoneleaf.us> Message-ID: <CAN1YFWvOZ=xGyvhWzSB6LXUt8Bs_LhmmqZv1GOL3BOU=_n2uuQ@mail.gmail.com> On Thu, Jan 22, 2015 at 8:34 PM, Ethan Furman <ethan at stoneleaf.us> wrote: > Indeed -- if someone does want type checking, but library xyz is using > annotations for something else, then a way > (provided by the type checker) to skip certain modules would be good. > FWIW, the proposition is to skip every module that doesn't "import typing". Cheers, -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150122/2c1ca6c3/attachment.html> From chris.barker at noaa.gov Fri Jan 23 02:30:25 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 22 Jan 2015 17:30:25 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <54C19CE9.5000200@stoneleaf.us> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <54C19CE9.5000200@stoneleaf.us> Message-ID: <-3534518513457978521@unknownmsgid> > On Jan 22, 2015, at 5:00 PM, Ethan Furman <ethan at stoneleaf.us> wrote: > Instead of x%, how about 10% ? ;) Thanks -- I am really bad at proof reading my own stuff ;-) -Chris From abarnert at yahoo.com Fri Jan 23 04:30:16 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 22 Jan 2015 19:30:16 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> Message-ID: <425E9C2C-706E-4BA2-9D31-8B4539EC9AF4@yahoo.com> Overall I like it, but I'm not sure the help on the tol parameter is clear enough for people who don't already know what they want--in other words, the very people this function should be helping. In my experience, novices understand relative tolerance immediately if you put it in terms of "within X% of expected", but don't always understand it if you put it in terms of "within X * expected" or, worse, "relative to the magnitude of the expected value". Just using % in there somewhere makes people get the concept. Unfortunately, since the API doesn't actually use a percentage--and shouldn't--I'm not sure how to get this across in a one-liner in the help. You can always add something like "(e.g., a relative tolerance of .005 means that the actual value must be within 0.5% of the expected value)", but that's way too verbose. (Also, I should note that the people I've explained this to have mostly been people with a US 1960-1990-style basic math education; I can't be sure that people who learned in another country, or in the post-post-new-math era in the US, etc. will respond the same way, although I do have a bit of anecdotal evidence from helping a few people on forums like StackOverflow that seems to imply they do.) Sent from a random iPhone On Jan 22, 2015, at 16:40, Chris Barker <chris.barker at noaa.gov> wrote: > is the relative tolerance -- it is the amount of error > allowed, relative to the magnitude of the expected value. From tjreedy at udel.edu Fri Jan 23 05:45:57 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 22 Jan 2015 23:45:57 -0500 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJLmN_YwmHOAXzqm5Dga1aQY8ygu6Z86WFaDCM-JLTh6aQ@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> <CAP1=2W59Pxr4j_Cx+Cvd+WQQZ4RznOSrEL9q_G2wZcB=F81=wQ@mail.gmail.com> <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> <CAP7+vJLmN_YwmHOAXzqm5Dga1aQY8ygu6Z86WFaDCM-JLTh6aQ@mail.gmail.com> Message-ID: <m9sjm6$v3u$1@ger.gmane.org> On 1/22/2015 3:05 PM, Guido van Rossum wrote: > Having got that off my mind, the vast majority of Python code that's > been written does not use annotations at all, so there is no reason to > change it at all. For code that does use annotation, *nothing* will > break at run time -- the code will work unchanged, without any > deprecations or slowdowns. > > The only situation where there may be a desire to change something (the > minimal thing being to add a "# type: OFF" comment at the top of a > module) is when a library that uses annotations for non-type-hinting > purposes is used by an application (or another library) that wants to > use type hints, and the user who is running the type checker cannot live > with the noise output spewed by the type checker for that library. I believe the dual-use of annotations problem would be solved if static type checkers gave precedence to annotations in stub files. Putting type hints in separate files is *possible* because static checkers do not need and will not use a runtime .__annotations__ attribute. If the presence of a stub file meant that annotations in the corresponding .py file were ignored by static checkers, then they could still be used in .py files to create .__annotations__ for dynamic use. I think typing should have a make_stub(inpy, outstub) function to make blank stub creation easy. It would copy all def headers and add ':' and '->' where appropriate. It could have an any=??? option which, when true, would add 'any' everywhere. Running typing as a script (-m) could run make_stub. An all-any stub would eliminate the need even for #type:off, and could be added by a user without touching the .py file. I think putting type hints in stub files should be documented as at least on a par, if not encouraged, with putting them in .py files. 1. As with writing tests, only a subset of python programmers will enjoy, be willing to, and be good at writing complex type hints. So stub files might be writtin by someone other than the .py author(s). 2. In some cases, type hints intended to be as accurate as possible will not be pretty -- or simple. In a separate file intended for machine consumption, aesthetics can more easily be ignored in favor of accuracy. 3. Users may want to add or customize annotations (perhaps to narrow the allowed arguments to conform to policy). This is better done in a separate file. -- Terry Jan Reedy From tjreedy at udel.edu Fri Jan 23 05:58:33 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 22 Jan 2015 23:58:33 -0500 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJKctVBZ0xnENZh5-NqJ1YK379nw410PNigG4eBFU8Oc3Q@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> <CAP1=2W59Pxr4j_Cx+Cvd+WQQZ4RznOSrEL9q_G2wZcB=F81=wQ@mail.gmail.com> <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> <CAP7+vJLmN_YwmHOAXzqm5Dga1aQY8ygu6Z86WFaDCM-JLTh6aQ@mail.gmail.com> <CABmzr0h+RZs=9ri-fiQjrZeXNTnsJP5LcZhZzBqRpdoYoHjW+A@mail.gmail.com> <CAP7+vJKctVBZ0xnENZh5-NqJ1YK379nw410PNigG4eBFU8Oc3Q@mail.gmail.com> Message-ID: <m9skdr$9uk$1@ger.gmane.org> On 1/22/2015 6:19 PM, Guido van Rossum wrote: > At the same time -- let me emphasize this one more time -- I hope there > will never be a time in the future where type hints are mandatory or > otherwise always expected to exist. If the PEP does not already say this, it might be helpful to add it -- for those who read it. Probably more important is the doc -- which gets read much more. -- Terry Jan Reedy From guido at python.org Fri Jan 23 06:23:01 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 22 Jan 2015 21:23:01 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <m9sjm6$v3u$1@ger.gmane.org> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> <CAP1=2W59Pxr4j_Cx+Cvd+WQQZ4RznOSrEL9q_G2wZcB=F81=wQ@mail.gmail.com> <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> <CAP7+vJLmN_YwmHOAXzqm5Dga1aQY8ygu6Z86WFaDCM-JLTh6aQ@mail.gmail.com> <m9sjm6$v3u$1@ger.gmane.org> Message-ID: <CAP7+vJ+H7gz4Qeqo0fZ6xW3Hvtf87PDmOmdymyC2pydKKPYaQw@mail.gmail.com> On Thu, Jan 22, 2015 at 8:45 PM, Terry Reedy <tjreedy at udel.edu> wrote: > On 1/22/2015 3:05 PM, Guido van Rossum wrote: > >> Having got that off my mind, the vast majority of Python code that's >> been written does not use annotations at all, so there is no reason to >> change it at all. For code that does use annotation, *nothing* will >> break at run time -- the code will work unchanged, without any >> deprecations or slowdowns. >> >> The only situation where there may be a desire to change something (the >> minimal thing being to add a "# type: OFF" comment at the top of a >> module) is when a library that uses annotations for non-type-hinting >> purposes is used by an application (or another library) that wants to >> use type hints, and the user who is running the type checker cannot live >> with the noise output spewed by the type checker for that library. >> > > I believe the dual-use of annotations problem would be solved if static > type checkers gave precedence to annotations in stub files. Putting type > hints in separate files is *possible* because static checkers do not need > and will not use a runtime .__annotations__ attribute. If the presence of > a stub file meant that annotations in the corresponding .py file were > ignored by static checkers, then they could still be used in .py files to > create .__annotations__ for dynamic use. > Correct -- if a stub exists the checker should not look at the actual source code at all. (The current PEP draft is light on stubs but we know we need to fix this. Patches welcome!) > I think typing should have a make_stub(inpy, outstub) function to make > blank stub creation easy. It would copy all def headers and add ':' and > '->' where appropriate. It could have an any=??? option which, when true, > would add 'any' everywhere. Running typing as a script (-m) could run > make_stub. An all-any stub would eliminate the need even for #type:off, > and could be added by a user without touching the .py file. > That sounds like a useful tool, but why should it be in typing.py? It shouldn't be needed at run time -- if the code is being run, the stub isn't needed, and if the code is being type-checked, the code won't be run so the stub won't be generated. > I think putting type hints in stub files should be documented as at least > on a par, if not encouraged, with putting them in .py files. > On a par. Stubs are essential for extension modules, and nearly essential for existing packages (both 3rd party and the stdlib!). But I don't want people writing new 3.x code having to deal with two files for every module. I've recently had to do this for C++ and the endless back-and-forth between .hpp and .cpp files (and the subtleties of what is allowed/required in each type of file) drove me nuts. > 1. As with writing tests, only a subset of python programmers will enjoy, > be willing to, and be good at writing complex type hints. So stub files > might be writtin by someone other than the .py author(s). > Yes, especially for existing libraries that is very useful. > 2. In some cases, type hints intended to be as accurate as possible will > not be pretty -- or simple. In a separate file intended for machine > consumption, aesthetics can more easily be ignored in favor of accuracy. > Hm, I'm skeptical about that. We're not writing Haskell types here. Most type hints are very bread-and-buttery things. > 3. Users may want to add or customize annotations (perhaps to narrow the > allowed arguments to conform to policy). This is better done in a separate > file. > I don't understand this part. Can you clarify, maybe give an example? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150122/ef36f82a/attachment-0001.html> From chris.barker at noaa.gov Fri Jan 23 05:57:56 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 22 Jan 2015 20:57:56 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <425E9C2C-706E-4BA2-9D31-8B4539EC9AF4@yahoo.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <425E9C2C-706E-4BA2-9D31-8B4539EC9AF4@yahoo.com> Message-ID: <CALGmxEK2p3CzJQPukax6mBe9vOiO7C0p=WEm7Kow0KRf6H6HcQ@mail.gmail.com> Andrew, I totally agree that it's not going to be that clear to folks -- but I'm as stumped as you as to how to make it clear without getting really wordy. Also, I think the percent error use case is infrequent, more likely would be that a relative tolerance of 1e-8 means that the numbers are the same to within about 8 significant decimal figures. After all, not many people think in terms of 0.0000001% Suggestions gladly accepted! -Chris On Thu, Jan 22, 2015 at 7:30 PM, Andrew Barnert <abarnert at yahoo.com> wrote: > Overall I like it, but I'm not sure the help on the tol parameter is clear > enough for people who don't already know what they want--in other words, > the very people this function should be helping. > > In my experience, novices understand relative tolerance immediately if you > put it in terms of "within X% of expected", but don't always understand it > if you put it in terms of "within X * expected" or, worse, "relative to the > magnitude of the expected value". Just using % in there somewhere makes > people get the concept. > > Unfortunately, since the API doesn't actually use a percentage--and > shouldn't--I'm not sure how to get this across in a one-liner in the help. > You can always add something like "(e.g., a relative tolerance of .005 > means that the actual value must be within 0.5% of the expected value)", > but that's way too verbose. > > (Also, I should note that the people I've explained this to have mostly > been people with a US 1960-1990-style basic math education; I can't be sure > that people who learned in another country, or in the post-post-new-math > era in the US, etc. will respond the same way, although I do have a bit of > anecdotal evidence from helping a few people on forums like StackOverflow > that seems to imply they do.) > > Sent from a random iPhone > > On Jan 22, 2015, at 16:40, Chris Barker <chris.barker at noaa.gov> wrote: > > > is the relative tolerance -- it is the amount of error > > allowed, relative to the magnitude of the expected value. > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150122/eb3434dc/attachment.html> From guido at python.org Fri Jan 23 06:54:11 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 22 Jan 2015 21:54:11 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEK2p3CzJQPukax6mBe9vOiO7C0p=WEm7Kow0KRf6H6HcQ@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <425E9C2C-706E-4BA2-9D31-8B4539EC9AF4@yahoo.com> <CALGmxEK2p3CzJQPukax6mBe9vOiO7C0p=WEm7Kow0KRf6H6HcQ@mail.gmail.com> Message-ID: <CAP7+vJLP8Se-VGVBSd3qOERF3ByX4dzcX-7kyP1SyLjCGYyKgQ@mail.gmail.com> I'd use an example with round numbers. "For example, to set a tolerance of 5%, pass tol=0.05. The default tolerance is 1e-8." On Thursday, January 22, 2015, Chris Barker <chris.barker at noaa.gov> wrote: > Andrew, > > I totally agree that it's not going to be that clear to folks -- but I'm > as stumped as you as to how to make it clear without getting really wordy. > > Also, I think the percent error use case is infrequent, more likely would > be that a relative tolerance of 1e-8 means that the numbers are the same to > within about 8 significant decimal figures. After all, not many people > think in terms of 0.0000001% > > Suggestions gladly accepted! > > -Chris > > > > On Thu, Jan 22, 2015 at 7:30 PM, Andrew Barnert <abarnert at yahoo.com > <javascript:_e(%7B%7D,'cvml','abarnert at yahoo.com');>> wrote: > >> Overall I like it, but I'm not sure the help on the tol parameter is >> clear enough for people who don't already know what they want--in other >> words, the very people this function should be helping. >> >> In my experience, novices understand relative tolerance immediately if >> you put it in terms of "within X% of expected", but don't always understand >> it if you put it in terms of "within X * expected" or, worse, "relative to >> the magnitude of the expected value". Just using % in there somewhere makes >> people get the concept. >> >> Unfortunately, since the API doesn't actually use a percentage--and >> shouldn't--I'm not sure how to get this across in a one-liner in the help. >> You can always add something like "(e.g., a relative tolerance of .005 >> means that the actual value must be within 0.5% of the expected value)", >> but that's way too verbose. >> >> (Also, I should note that the people I've explained this to have mostly >> been people with a US 1960-1990-style basic math education; I can't be sure >> that people who learned in another country, or in the post-post-new-math >> era in the US, etc. will respond the same way, although I do have a bit of >> anecdotal evidence from helping a few people on forums like StackOverflow >> that seems to imply they do.) >> >> Sent from a random iPhone >> >> On Jan 22, 2015, at 16:40, Chris Barker <chris.barker at noaa.gov >> <javascript:_e(%7B%7D,'cvml','chris.barker at noaa.gov');>> wrote: >> >> > is the relative tolerance -- it is the amount of error >> > allowed, relative to the magnitude of the expected value. >> > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > <javascript:_e(%7B%7D,'cvml','Chris.Barker at noaa.gov');> > -- --Guido van Rossum (on iPad) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150122/19c365ed/attachment.html> From abarnert at yahoo.com Fri Jan 23 08:12:59 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 22 Jan 2015 23:12:59 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAP7+vJLP8Se-VGVBSd3qOERF3ByX4dzcX-7kyP1SyLjCGYyKgQ@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <425E9C2C-706E-4BA2-9D31-8B4539EC9AF4@yahoo.com> <CALGmxEK2p3CzJQPukax6mBe9vOiO7C0p=WEm7Kow0KRf6H6HcQ@mail.gmail.com> <CAP7+vJLP8Se-VGVBSd3qOERF3ByX4dzcX-7kyP1SyLjCGYyKgQ@mail.gmail.com> Message-ID: <EFC06649-FC50-4A56-A3B1-5768627DF44C@yahoo.com> On Jan 22, 2015, at 21:54, Guido van Rossum <guido at python.org> wrote: > I'd use an example with round numbers. "For example, to set a tolerance of 5%, pass tol=0.05. The default tolerance is 1e-8." Hard to beat that for simplicity. +1 on this wording or something similar instead of the current abstract version. > > On Thursday, January 22, 2015, Chris Barker <chris.barker at noaa.gov> wrote: >> Andrew, >> >> I totally agree that it's not going to be that clear to folks -- but I'm as stumped as you as to how to make it clear without getting really wordy. >> >> Also, I think the percent error use case is infrequent, more likely would be that a relative tolerance of 1e-8 means that the numbers are the same to within about 8 significant decimal figures. After all, not many people think in terms of 0.0000001% >> >> Suggestions gladly accepted! >> >> -Chris >> >> >> >> On Thu, Jan 22, 2015 at 7:30 PM, Andrew Barnert <abarnert at yahoo.com> wrote: >>> Overall I like it, but I'm not sure the help on the tol parameter is clear enough for people who don't already know what they want--in other words, the very people this function should be helping. >>> >>> In my experience, novices understand relative tolerance immediately if you put it in terms of "within X% of expected", but don't always understand it if you put it in terms of "within X * expected" or, worse, "relative to the magnitude of the expected value". Just using % in there somewhere makes people get the concept. >>> >>> Unfortunately, since the API doesn't actually use a percentage--and shouldn't--I'm not sure how to get this across in a one-liner in the help. You can always add something like "(e.g., a relative tolerance of .005 means that the actual value must be within 0.5% of the expected value)", but that's way too verbose. >>> >>> (Also, I should note that the people I've explained this to have mostly been people with a US 1960-1990-style basic math education; I can't be sure that people who learned in another country, or in the post-post-new-math era in the US, etc. will respond the same way, although I do have a bit of anecdotal evidence from helping a few people on forums like StackOverflow that seems to imply they do.) >>> >>> Sent from a random iPhone >>> >>> On Jan 22, 2015, at 16:40, Chris Barker <chris.barker at noaa.gov> wrote: >>> >>> > is the relative tolerance -- it is the amount of error >>> > allowed, relative to the magnitude of the expected value. >> >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.Barker at noaa.gov > > > -- > --Guido van Rossum (on iPad) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150122/6686a0c8/attachment-0001.html> From steve at pearwood.info Fri Jan 23 08:41:44 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 23 Jan 2015 18:41:44 +1100 Subject: [Python-ideas] Should bool continue to inherit from int? In-Reply-To: <54BFEBFD.5030201@mrabarnett.plus.com> References: <CABvhCyFWGv4VHduA-wBKG5ifwzPWeMQSBsDZi-BC9tWchFwCNA@mail.gmail.com> <CAPTjJmqC3iDnobQna-6bCbAjPYabEsidpdTj+qzaLxmm+oQT1Q@mail.gmail.com> <54BFEBFD.5030201@mrabarnett.plus.com> Message-ID: <20150123074144.GH18556@ando.pearwood.info> On Wed, Jan 21, 2015 at 06:12:13PM +0000, MRAB wrote: > Pascal has 'ord' for getting the ordinal number from an enumerated > type, so perhaps that could be used: > > ord(False) == 0 > ord(True) == 1 > > Currently it's just the inverse of 'chr'. In Python, that is spelled int(False) # returns 0 int(True) # returns 1 -- Steve From steve at pearwood.info Fri Jan 23 09:06:49 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 23 Jan 2015 19:06:49 +1100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> Message-ID: <20150123080649.GI18556@ando.pearwood.info> On Thu, Jan 22, 2015 at 04:40:14PM -0800, Chris Barker wrote: > Hi folks, > > After much discussion on this list, I have written up a PEP, and it is > ready for review (see below) I do not agree that it is ready for review. I think you have rushed to decide that this needs a PEP, rushed the preparation of the PEP, and now you have rushed the request for review. What's the hurry? As it stands with the decisions you have made, I cannot support this PEP even though I support the basic idea. -- Steve From p.f.moore at gmail.com Fri Jan 23 09:11:08 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 23 Jan 2015 08:11:08 +0000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> Message-ID: <CACac1F-Zg8RGCoAinqp8_=G+inpdsOic=abBp4Ymq2KDdFjdKw@mail.gmail.com> On 23 January 2015 at 00:40, Chris Barker <chris.barker at noaa.gov> wrote: > Behavior near zero > ------------------ > > Relative comparison is problematic if either value is zero. In this > case, the difference is relative to zero, and thus will always be > smaller than the prescribed tolerance. I'm not sure I follow the specifics but this is saying that everything will be close to zero. Isn't that the wrong way round? I thought the comments in the discussion on the list were saying that the problem with relative tolerance is that *nothing* is close to zero? Paul From ethan at stoneleaf.us Fri Jan 23 09:59:21 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 23 Jan 2015 00:59:21 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150123080649.GI18556@ando.pearwood.info> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <20150123080649.GI18556@ando.pearwood.info> Message-ID: <54C20D69.9080702@stoneleaf.us> On 01/23/2015 12:06 AM, Steven D'Aprano wrote: > On Thu, Jan 22, 2015 at 04:40:14PM -0800, Chris Barker wrote: >> After much discussion on this list, I have written up a PEP, and it is >> ready for review (see below) > > I do not agree that it is ready for review. Why? If it has problems, how will he find out about them unless people read it and offer critiques? Or do you not refer to that process as reviewing? > I think you have rushed to decide that this needs a PEP, He asked if a PEP was needed, and one is. Worst-case scenario we have something to point the next floating-point closeness requester to. > rushed the preparation of the PEP, With over 100 messages to pull from, how was the preparation rushed? He should have taken a month to write it? > and now you have rushed the request for review. Um, what? He should have just sat on it for a couple weeks before asking people to look it over? Asking for a review is not the same as asking for a pronouncement; it's not even on python-dev yet. > What's the hurry? For one, Python 3.5 alpha one is just around the corner, and while there's still time after that the more eyeballs the better; for another, why wait? He has the information he needed, he collected it, made some decisions, and brought it back to the community. Ten days from the first floating point closeness message (14 if you count the float range class thread). A PEP also helps focus the conversation. > As it stands with the decisions you have made, I cannot support this PEP > even though I support the basic idea. Perhaps you feel rushed because you don't like it? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150123/12657878/attachment.sig> From ethan at stoneleaf.us Fri Jan 23 10:09:46 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 23 Jan 2015 01:09:46 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJ+H7gz4Qeqo0fZ6xW3Hvtf87PDmOmdymyC2pydKKPYaQw@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> <CAP1=2W59Pxr4j_Cx+Cvd+WQQZ4RznOSrEL9q_G2wZcB=F81=wQ@mail.gmail.com> <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> <CAP7+vJLmN_YwmHOAXzqm5Dga1aQY8ygu6Z86WFaDCM-JLTh6aQ@mail.gmail.com> <m9sjm6$v3u$1@ger.gmane.org> <CAP7+vJ+H7gz4Qeqo0fZ6xW3Hvtf87PDmOmdymyC2pydKKPYaQw@mail.gmail.com> Message-ID: <54C20FDA.1080907@stoneleaf.us> On 01/22/2015 09:23 PM, Guido van Rossum wrote: > On Thu, Jan 22, 2015 at 8:45 PM, Terry Reedy wrote: >> >> 3. Users may want to add or customize annotations (perhaps to narrow the >> allowed arguments to conform to policy). This is better done in a separate >> file. > > I don't understand this part. Can you clarify, maybe give an example? As a guess, if somebody has a library function that is somewhat generic, but they will only be using it with a specific type, then having a stub file with that more specific information would help catch bugs where a type is being passed in that is not the required specific type, even though it satisfies the more generic type in the library's annotations. Something like: --8< actual_lib.py ------------------------------- def cool_adder_func(a:Number, b:Number): -> Number return a + b --8< --------------------------------------------- --8< overriding stub file ----------------------- def cool_adder_func(a:int, b:int): -> int pass --8< -------------------------------------------- --8< application -------------------------------- ... cool_adder_func(float_var, Decimal_var) --8< -------------------------------------------- The above would be flagged because in this application, using this stub file, cool_adder_func should only take ints. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150123/4f109dd6/attachment.sig> From steve at pearwood.info Fri Jan 23 10:15:57 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 23 Jan 2015 20:15:57 +1100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <54C20D69.9080702@stoneleaf.us> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <20150123080649.GI18556@ando.pearwood.info> <54C20D69.9080702@stoneleaf.us> Message-ID: <20150123091557.GK18556@ando.pearwood.info> On Fri, Jan 23, 2015 at 12:59:21AM -0800, Ethan Furman wrote: > On 01/23/2015 12:06 AM, Steven D'Aprano wrote: > > On Thu, Jan 22, 2015 at 04:40:14PM -0800, Chris Barker wrote: > > >> After much discussion on this list, I have written up a PEP, and it is > >> ready for review (see below) > > > > I do not agree that it is ready for review. > > Why? If it has problems, how will he find out about them unless people read it and offer critiques? Or do you not > refer to that process as reviewing? Ethan, there are factors that you are unaware of because they took place off-list. Since they are private, I will say no more about them except to say that Chris has proceeded as if there is consensus when there actually is not. -- Steven From apalala at gmail.com Fri Jan 23 14:24:09 2015 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Fri, 23 Jan 2015 08:54:09 -0430 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150123080649.GI18556@ando.pearwood.info> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <20150123080649.GI18556@ando.pearwood.info> Message-ID: <CAN1YFWvbNksAzeDE20z-i98+cTDL3jJwxkbOnFh5C3fya7gG3A@mail.gmail.com> On Fri, Jan 23, 2015 at 3:36 AM, Steven D'Aprano <steve at pearwood.info> wrote: > I do not agree that it is ready for review. > > I think you have rushed to decide that this needs a PEP, rushed the > preparation of the PEP, and now you have rushed the request for review. > What's the hurry? > > As it stands with the decisions you have made, I cannot support this PEP > even though I support the basic idea. > Mmmm... That seemed kind of dogmatic... This thread has been going on for long. I prefer the PEP because it is a concrete proposal. Even if it is rejected, the reasons for the rejection will be documented, so people can be referred to the document instead of spinning this wheel again. Cheers, -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150123/3598a079/attachment.html> From fetchinson at googlemail.com Fri Jan 23 14:33:25 2015 From: fetchinson at googlemail.com (Fetchinson .) Date: Fri, 23 Jan 2015 14:33:25 +0100 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJ+H7gz4Qeqo0fZ6xW3Hvtf87PDmOmdymyC2pydKKPYaQw@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> <CAP1=2W59Pxr4j_Cx+Cvd+WQQZ4RznOSrEL9q_G2wZcB=F81=wQ@mail.gmail.com> <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> <CAP7+vJLmN_YwmHOAXzqm5Dga1aQY8ygu6Z86WFaDCM-JLTh6aQ@mail.gmail.com> <m9sjm6$v3u$1@ger.gmane.org> <CAP7+vJ+H7gz4Qeqo0fZ6xW3Hvtf87PDmOmdymyC2pydKKPYaQw@mail.gmail.com> Message-ID: <CADjSo4TX_0jEeUM2z3=Wnc_LLrB99=KhHHuO8qyObBiGCxUF5Q@mail.gmail.com> It looks like type hinting as a feature is applauded by everyone but the actual syntax/implementation is polarizing. Could we have a section in the PEP listing all alternative sytax/implementations with pros and cons? Stuff like: 1. Stub files 2. property-style type hinting, e.g.: @typehints( arg1: int, arg2: str, returns: str ) def func( arg1, arg2 ) return 'hello' 3. docstrings, e.g.: def func( arg1, arg2 ): """type hints: arg1: int, arg2: str, returns: str""" return 'hello' 4. Cobra-inspired type hints, e.g.: def func( arg1, arg2 ): requires: isinstance( arg1, int ) isinstance( arg2, str ) returns: str Cheers, Daniel -- Psss, psss, put it down! - http://www.cafepress.com/putitdown From ncoghlan at gmail.com Fri Jan 23 15:56:27 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 24 Jan 2015 00:56:27 +1000 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <CAP7+vJKU2oiyhmtOsM6cA=NUB+CjqLzGehy8uXQT=Gn=0hLc6Q@mail.gmail.com> References: <CAP7+vJKd19Yb5yeVPt38sXHTKVjfPh_JO4PX_dngR50Vck2dNQ@mail.gmail.com> <538698398.2487257.1421798257472.JavaMail.yahoo@jws10073.mail.ne1.yahoo.com> <CAP7+vJKU2oiyhmtOsM6cA=NUB+CjqLzGehy8uXQT=Gn=0hLc6Q@mail.gmail.com> Message-ID: <CADiSq7f6OscLNqubW_3Db_ka=UrOVn9GMES_f-5kvJuNVUL-Ew@mail.gmail.com> On 21 January 2015 at 11:16, Guido van Rossum <guido at python.org> wrote: > So it's hard to say to what extent this was a conscious choice and to what > extent it was just tradition. But there's nothing necessarily wrong with > tradition (up to a point). I think it still makes sense that statements are > laid out vertically while expressions are laid out horizontally. Come to > think of it, mathematics uses a similar convention -- a formula is laid out > (primarily) horizontally, while a sequence of formulas (like a proof or a > set of axioms) is laid out vertically. I really like that "each statement is like a single step in a mathematical proof" analogy. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From emile at fenx.com Fri Jan 23 15:52:34 2015 From: emile at fenx.com (Emile van Sebille) Date: Fri, 23 Jan 2015 06:52:34 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150123091557.GK18556@ando.pearwood.info> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <20150123080649.GI18556@ando.pearwood.info> <54C20D69.9080702@stoneleaf.us> <20150123091557.GK18556@ando.pearwood.info> Message-ID: <m9tn7l$2mf$1@ger.gmane.org> On 1/23/2015 1:15 AM, Steven D'Aprano wrote: > On Fri, Jan 23, 2015 at 12:59:21AM -0800, Ethan Furman wrote: >> On 01/23/2015 12:06 AM, Steven D'Aprano wrote: >>> On Thu, Jan 22, 2015 at 04:40:14PM -0800, Chris Barker wrote: >> >>>> After much discussion on this list, I have written up a PEP, and it is >>>> ready for review (see below) >>> >>> I do not agree that it is ready for review. >> >> Why? If it has problems, how will he find out about them unless people read it and offer critiques? Or do you not >> refer to that process as reviewing? > > Ethan, there are factors that you are unaware of because they took place > off-list. Since they are private, I will say no more about them except > to say that Chris has proceeded as if there is consensus when there > actually is not. Damn PSU! Just as I start to think that the _ideas_ list is the place to openly discuss ideas. Emile From solipsis at pitrou.net Fri Jan 23 16:36:18 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 23 Jan 2015 16:36:18 +0100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> Message-ID: <20150123163618.3465f819@fsol> On Thu, 22 Jan 2015 16:40:14 -0800 Chris Barker <chris.barker at noaa.gov> wrote: > > Expected Uses > ============= > > The primary expected use case is various forms of testing -- "are the > results computed near what I expect as a result?" This sort of test > may or may not be part of a formal unit testing suite. I don't think the proposal fits the bill. For testing you want a function that is both 1) quite rigorous (i.e. checks equality within a defined number of ulps) 2) handles all special cases in a useful way (i.e. zeros, including distinguishing between positive and negative zeros, infinities, NaNs etc.). As someone who wrote such a function for Numba, what you're proposing would not be a suitable replacement. Regards Antoine. From p.f.moore at gmail.com Fri Jan 23 17:05:05 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 23 Jan 2015 16:05:05 +0000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> Message-ID: <CACac1F8jksJd=z-3u53M1hO=1ou2COpmzDdm4oNu0WwXN9gwCA@mail.gmail.com> On 23 January 2015 at 00:40, Chris Barker <chris.barker at noaa.gov> wrote: > Expected Uses > ============= > > The primary expected use case is various forms of testing -- "are the > results computed near what I expect as a result?" This sort of test > may or may not be part of a formal unit testing suite. > > The function might be used also to determine if a measured value is > within an expected value. This section is very weak. As someone who doesn't do numerically intensive computing I would start with the assumption that people who do would have the appropriate tools in packages like numpy, and they would have the knowledge and understanding to use them properly. So my expectation is that this function is intended specifically for non-specialists like me. Based on that, I can't imagine when I'd use this function. You mention testing, but unittest has a function to do this already. Sure, it's tied tightly to unittest, so it's not useful for something like py.test, but that's because unittest is the stdlib testing framework. If you wanted to make that check more widely available, why not simply make it into a full-fledged function rather than an assertion? And if it's not suitable for that purpose, why does this PEP not propose updating the unittest assertion to use the new function? It can't be right to have 2 *different* "nearly equal" functions in the stdlib. Outside of testing, there seems to be no obvious use for the new function. You mention measured values, but what does that mean? "Measure in the length of the line and type in the result, and I'll confirm if it matches the value calculated"? That seems a bit silly. I'd like to see a couple of substantial, properly explained examples that aren't testing and aren't specialist. My worry is that what this function will *actually* be used for is to allow naive users to gloss over their lack of understanding of floating point: n = 0.0 while not is_close_to(n, 1.0): # Because I don't understand floating point do_something_with(n) n += 0.1 BTW, when writing that I had to keep scrolling up to see which order actual and expected went in. I'd imagine plenty of naive users will assume "it's symmetrical so it shouldn't matter" and get the order wrong. In summary - it looks too much like an attractive nuisance to me, and I don't see enough value in it to counteract that. Paul From guido at python.org Fri Jan 23 17:07:18 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 23 Jan 2015 08:07:18 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150123091557.GK18556@ando.pearwood.info> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <20150123080649.GI18556@ando.pearwood.info> <54C20D69.9080702@stoneleaf.us> <20150123091557.GK18556@ando.pearwood.info> Message-ID: <CAP7+vJ+Ja6Nsv-YHDYjQ6yARrBhe5LkQ7Px+6fpY3BhXQ_Q8HA@mail.gmail.com> On Fri, Jan 23, 2015 at 1:15 AM, Steven D'Aprano <steve at pearwood.info> wrote: > On Fri, Jan 23, 2015 at 12:59:21AM -0800, Ethan Furman wrote: > > On 01/23/2015 12:06 AM, Steven D'Aprano wrote: > > > On Thu, Jan 22, 2015 at 04:40:14PM -0800, Chris Barker wrote: > > > > >> After much discussion on this list, I have written up a PEP, and it is > > >> ready for review (see below) > > > > > > I do not agree that it is ready for review. > > > > Why? If it has problems, how will he find out about them unless people > read it and offer critiques? Or do you not > > refer to that process as reviewing? > > Ethan, there are factors that you are unaware of because they took place > off-list. Since they are private, I will say no more about them except > to say that Chris has proceeded as if there is consensus when there > actually is not. Steven, this appeal to things unmentionable is not an acceptable way to oppose a PEP. In the text you quoted I didn't see Chris claim consensus -- just that he has written up his version. It's ready for review because he wants feedback -- "ready for review" is *not* code for "this is the final word from the community, now the BDFL must speak." Your posts make me worried that we have turned into a political body rather than a group of technical enthusiasts trying to improve the language they all love. I don't think you can reasonably disagree that a PEP is needed -- not with so much discussion and apparently still no agreement. If you oppose the specific proposal, say what you think is wrong with it. If you think it needs more input from other experts, name those experts. If you think it needs more input from a community, name that community. I haven't actually read the PEP, so I don't have an opinion about it (my post last night was just an attempt to reword something quoted in the email thread). I just saw Antoine's response, and at least he talks about the proposal, not the politics around it. But he's awfully vague. We need a concrete counterproposal. Possibly a competing PEP. Anything but references to things that happened off-stage. If you have a personal beef with Chris, this is not the place. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150123/1be56f97/attachment.html> From guido at python.org Fri Jan 23 17:22:45 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 23 Jan 2015 08:22:45 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CADjSo4TX_0jEeUM2z3=Wnc_LLrB99=KhHHuO8qyObBiGCxUF5Q@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> <CAP1=2W59Pxr4j_Cx+Cvd+WQQZ4RznOSrEL9q_G2wZcB=F81=wQ@mail.gmail.com> <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> <CAP7+vJLmN_YwmHOAXzqm5Dga1aQY8ygu6Z86WFaDCM-JLTh6aQ@mail.gmail.com> <m9sjm6$v3u$1@ger.gmane.org> <CAP7+vJ+H7gz4Qeqo0fZ6xW3Hvtf87PDmOmdymyC2pydKKPYaQw@mail.gmail.com> <CADjSo4TX_0jEeUM2z3=Wnc_LLrB99=KhHHuO8qyObBiGCxUF5Q@mail.gmail.com> Message-ID: <CAP7+vJ+7jXMjSNeeErTKgjJm0U9Gc9M5iMLa6+C+T8LZZ4fAiA@mail.gmail.com> Sigh. I guess I'll have to do this. https://github.com/ambv/typehinting/issues/55 On Fri, Jan 23, 2015 at 5:33 AM, Fetchinson . <fetchinson at googlemail.com> wrote: > It looks like type hinting as a feature is applauded by everyone but > the actual syntax/implementation is polarizing. Could we have a > section in the PEP listing all alternative sytax/implementations with > pros and cons? Stuff like: > > 1. Stub files > > > 2. property-style type hinting, e.g.: > > @typehints( arg1: int, arg2: str, returns: str ) > def func( arg1, arg2 ) > return 'hello' > > > 3. docstrings, e.g.: > > def func( arg1, arg2 ): > """type hints: arg1: int, arg2: str, returns: str""" > return 'hello' > > 4. Cobra-inspired type hints, e.g.: > > def func( arg1, arg2 ): > requires: > isinstance( arg1, int ) > isinstance( arg2, str ) > returns: > str > > > Cheers, > Daniel > > > > -- > Psss, psss, put it down! - http://www.cafepress.com/putitdown > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150123/f2c250d4/attachment.html> From chris.barker at noaa.gov Fri Jan 23 17:51:00 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 23 Jan 2015 08:51:00 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150123163618.3465f819@fsol> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <20150123163618.3465f819@fsol> Message-ID: <CALGmxEL8W7ziZkze1hdxLKoCaVrx9h7oSVZwLHxFfDe0sjBmGQ@mail.gmail.com> On Fri, Jan 23, 2015 at 7:36 AM, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Thu, 22 Jan 2015 16:40:14 -0800 > Chris Barker <chris.barker at noaa.gov> wrote: > > > > Expected Uses > > ============= > > > > The primary expected use case is various forms of testing -- "are the > > results computed near what I expect as a result?" This sort of test > > may or may not be part of a formal unit testing suite. > > I don't think the proposal fits the bill. For testing you want a > function that is both 1) quite rigorous (i.e. checks equality within a > defined number of ulps) 2) It depends on what you are testing -- I tried to be explicite that this was not intended for testing the accuracy of numerical algorithms, for instance. Rather, it's best use case is testing to see whether you have introduced a big 'ol bug that completely changed your result -- have you got in the ballark. Something similar is in Boost, is in numpy, and n any number of other places. It is clearly useful. THat doesn't mean it has to go in the stdlib, but it is useful in many cases. As for the ulps test -- can you suggest a way to do that, while also providing a simple definition of tolerance that casual users can understand and use (and have a reasonable default? I know I can't. Note that some of the feedback on the PEP as is is that it's too hard to understand already! (without better docs, anyway) > handles all special cases in a useful way > (i.e. zeros, including distinguishing between positive and negative > zeros, infinities, NaNs etc.). > zero, inf, -inf, NaN are all handles, I think correctly. And if -0.0 is not cloe to 0.0, I dont know what is ;-) (there is a test to make sure that's true actually) If you want to make the distinction between -0.0 and 0.0, then you don't want a "close" or "approximate" test. > As someone who wrote such a function for Numba, what you're proposing > would not be a suitable replacement. > I never expected it would be a replacement for what is needed for a project like numba. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150123/1caef8b4/attachment-0001.html> From ethan at stoneleaf.us Fri Jan 23 18:12:26 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 23 Jan 2015 09:12:26 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150123163618.3465f819@fsol> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <20150123163618.3465f819@fsol> Message-ID: <54C280FA.2000007@stoneleaf.us> On 01/23/2015 07:36 AM, Antoine Pitrou wrote: > On Thu, 22 Jan 2015 16:40:14 -0800 Chris Barker wrote: >> >> Expected Uses >> ============= >> >> The primary expected use case is various forms of testing -- "are the >> results computed near what I expect as a result?" This sort of test >> may or may not be part of a formal unit testing suite. > > I don't think the proposal fits the bill. For testing you want a > function that is both 1) quite rigorous (i.e. checks equality within a > defined number of ulps) 2) handles all special cases in a useful way > (i.e. zeros, including distinguishing between positive and negative > zeros, infinities, NaNs etc.). I disagree -- this function is not meant for mathematicians, but for the non-maths person who needs something that works. Will there be situations where it doesn't work? Certainly. Will they be surprising? Possibly. On the other hand, I was very surprised the first time an bytes object gave me an integer and not a byte. > As someone who wrote such a function for Numba, what you're proposing > would not be a suitable replacement. This isn't for Numba, SciPy, or NumPy. It's to help those who don't use/need those products, but still have some light floating point work to do. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150123/50f93000/attachment.sig> From chris.barker at noaa.gov Fri Jan 23 18:28:14 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 23 Jan 2015 09:28:14 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150123091557.GK18556@ando.pearwood.info> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <20150123080649.GI18556@ando.pearwood.info> <54C20D69.9080702@stoneleaf.us> <20150123091557.GK18556@ando.pearwood.info> Message-ID: <CALGmxEKqJycEsESgcQtNxaAOfjXE8wCGQtF+Bexxe7n2c=ueMQ@mail.gmail.com> On Fri, Jan 23, 2015 at 1:15 AM, Steven D'Aprano <steve at pearwood.info> wrote: > Since they are private, I will say no more about them except > to say that Chris has proceeded as if there is consensus when there > actually is not. no need for mystery here -- I asked off-list for feedback from Steven and a couple others, then posted the PEP without having given them much time to respond. However, I posted the PEP because I wanted review, and we had had enough circular conversations that I thought is was time for a concrete proposal to bash on. I by no means intended to convey the impression that there was consensus reached among anyone in particular. The goal of posting the PEP was to determine if that was so, and if not, to change it to a point where that could happen. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150123/ce689d42/attachment.html> From chris.barker at noaa.gov Fri Jan 23 18:21:27 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 23 Jan 2015 09:21:27 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CACac1F8jksJd=z-3u53M1hO=1ou2COpmzDdm4oNu0WwXN9gwCA@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CACac1F8jksJd=z-3u53M1hO=1ou2COpmzDdm4oNu0WwXN9gwCA@mail.gmail.com> Message-ID: <CALGmxE+syr4suqsJSOziJDOt4hOVAohVx9fds9PLNJZuVPM3nw@mail.gmail.com> On Fri, Jan 23, 2015 at 8:05 AM, Paul Moore <p.f.moore at gmail.com> wrote: > > The primary expected use case is various forms of testing -- "are the > > results computed near what I expect as a result?" This sort of test > > may or may not be part of a formal unit testing suite. > > > > The function might be used also to determine if a measured value is > > within an expected value. > > This section is very weak. I'll see what I can do to strengthen it. > As someone who doesn't do numerically > intensive computing I would start with the assumption that people who > do would have the appropriate tools in packages like numpy, and they > would have the knowledge and understanding to use them properly. So my > expectation is that this function is intended specifically for > non-specialists like me. > Indeed that is the idea (though there are plenty of specialists using numpy as well ;-) ) Based on that, I can't imagine when I'd use this function. You mention > testing, but unittest has a function to do this already. Sure, it's > tied tightly to unittest, so it's not useful for something like > py.test, but that's because unittest is the stdlib testing framework. > If you wanted to make that check more widely available, why not simply > make it into a full-fledged function rather than an assertion? That would be an option, but I don't think the one in unittest is the right test anyway -- its focus on on number of decimal digits after the decimal place is not generally useful. (that would make some sense for the Decimal type...) And if > it's not suitable for that purpose, why does this PEP not propose > updating the unittest assertion to use the new function? well, for backward compatibility reasons, I had just assumed it was off the table -- or a long, painful road anyway. And the unitest is very vested in it's OO structure -- would we want add free-form functions to it? > It can't be > right to have 2 *different* "nearly equal" functions in the stdlib. > Well, they do have a different functionality -- maybe some people really do want the decimal digits thing. I'm not sure we'd want one function with a whole bunch of different ways to call it -- maybe we would, but having different functions seems fine to me. > Outside of testing, there seems to be no obvious use for the new > function. You mention measured values, but what does that mean? > "Measure in the length of the line and type in the result, and I'll > confirm if it matches the value calculated"? That seems a bit silly. > This came up in examples in the discussion thread -- I'd don't think I would use it that way myself, so I'm going to leave it to others to suggest better examples or wording. Otherwise, I'll probably take it out. I'd like to see a couple of substantial, properly explained examples > that aren't testing and aren't specialist. In practice, I think testing is the biggest use case, but not necessarily formal unit testing. That's certainly how I would use it (and the use case that prompted me to start this whole thread to begin with..). I'll look in my code to see if I use it other ways, and I'm open to any other examples anyone might have. But maybe it should be with testing code in that case -- but I don't see any free-form testing utility functions in there now. Maybe it should go in unitest.util ? I'd rather not, but it's just a different import line. > My worry is that what this > function will *actually* be used for is to allow naive users to gloss > over their lack of understanding of floating point: > > n = 0.0 > while not is_close_to(n, 1.0): # Because I don't understand floating > point > do_something_with(n) > n += 0.1 > Is that necessarily worse? it would at least terminate ;-) floating point is a bit of an attractive nuisance anyway. > BTW, when writing that I had to keep scrolling up to see which order > actual and expected went in. I'd imagine plenty of naive users will > assume "it's symmetrical so it shouldn't matter" and get the order > wrong. > Well, I think the biggest real issue about this (other than should it be in the stdlib at all) is the question of a symmetrical vs. symmetrical test. I decided to go (for this draft, anyway) with the asymmetric test, as it is better defined and easier to reason about, and more appropriate for some cases. And the biggest argument for a symmetric test is that it is what people would expect. So I tried to make the parameter names that would make it clear (rather than a,b or x,y) -- I think I failed on that, however -- anyone have a better suggestion for names? It turns out "actual" is far too similar in meaning to "expected". In summary - it looks too much like an attractive nuisance to me, If it's not there, the folks will cobble somethign up themselves (and I'm sure do, all the time). If they know what they are doing, and take care, then great, but if not then they may get something with worse behavior that this. Maybe they will at least understand it better, but I suspect the pitfalls will all still be there in a typical case. And in any case, have to take the time to write it. That's my logic anyway. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150123/46bdc51e/attachment-0001.html> From chris.barker at noaa.gov Fri Jan 23 18:27:23 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 23 Jan 2015 09:27:23 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxE+syr4suqsJSOziJDOt4hOVAohVx9fds9PLNJZuVPM3nw@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CACac1F8jksJd=z-3u53M1hO=1ou2COpmzDdm4oNu0WwXN9gwCA@mail.gmail.com> <CALGmxE+syr4suqsJSOziJDOt4hOVAohVx9fds9PLNJZuVPM3nw@mail.gmail.com> Message-ID: <CALGmxEKbqNR3wXx4Suhzps6WVrj8H71T1uzRBUb4S1hOaK-XKQ@mail.gmail.com> On Fri, Jan 23, 2015 at 9:21 AM, Chris Barker <chris.barker at noaa.gov> wrote: > Indeed that is the idea (though there are plenty of specialists using > numpy as well ;-) ) > uhm, non-specialists, that is. In fact, the one in numpy is more susceptible to miss-use. On the other hand, it's there, and it's useful, and works most of the time. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150123/68ee1d02/attachment.html> From guido at python.org Fri Jan 23 18:41:24 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 23 Jan 2015 09:41:24 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEL8W7ziZkze1hdxLKoCaVrx9h7oSVZwLHxFfDe0sjBmGQ@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <20150123163618.3465f819@fsol> <CALGmxEL8W7ziZkze1hdxLKoCaVrx9h7oSVZwLHxFfDe0sjBmGQ@mail.gmail.com> Message-ID: <CAP7+vJKQO8DfJbQL7O6Vs6pD=y+3ddmDxcK3Mw-BPGYf+ONH=Q@mail.gmail.com> On Fri, Jan 23, 2015 at 8:51 AM, Chris Barker <chris.barker at noaa.gov> wrote: > On Fri, Jan 23, 2015 at 7:36 AM, Antoine Pitrou <solipsis at pitrou.net> > wrote: > >> On Thu, 22 Jan 2015 16:40:14 -0800 >> Chris Barker <chris.barker at noaa.gov> wrote: >> > >> > Expected Uses >> > ============= >> > >> > The primary expected use case is various forms of testing -- "are the >> > results computed near what I expect as a result?" This sort of test >> > may or may not be part of a formal unit testing suite. >> >> I don't think the proposal fits the bill. For testing you want a >> function that is both 1) quite rigorous (i.e. checks equality within a >> defined number of ulps) 2) > > > It depends on what you are testing -- I tried to be explicite that this > was not intended for testing the accuracy of numerical algorithms, for > instance. Rather, it's best use case is testing to see whether you have > introduced a big 'ol bug that completely changed your result -- have you > got in the ballark. > Maybe the confusion here is around the use of "test". To some, that means "unit test" or some other way of testing software. But I hope that's not the main use case. Let's look at Newton's algorithm for computing a square root. It's something like def sqrt(x): new_guess = 1 repeat: guess = new_guess new_guess = avg(guess, x/guess) # Not sure if I've got this right until guess is close enough to new guess return guess This seems a place where a decent "is close enough" definition would help. (Even though this particular algorithm usually converges so rapidly that you can get a result that's correct to within an ulp or so -- other approximations might not.) > Something similar is in Boost, is in numpy, and n any number of other > places. It is clearly useful. THat doesn't mean it has to go in the stdlib, > but it is useful in many cases. > > As for the ulps test -- can you suggest a way to do that, while also > providing a simple definition of tolerance that casual users can understand > and use (and have a reasonable default? I know I can't. > Isn't an ulp just a base-2 way of specifying precision scaled so that 1 ulp is the low bit of the mantissa in IEEE fp? > > Note that some of the feedback on the PEP as is is that it's too hard to > understand already! (without better docs, anyway) > > >> handles all special cases in a useful way >> (i.e. zeros, including distinguishing between positive and negative >> zeros, infinities, NaNs etc.). >> > > zero, inf, -inf, NaN are all handles, I think correctly. And if -0.0 is > not cloe to 0.0, I dont know what is ;-) > > (there is a test to make sure that's true actually) > > If you want to make the distinction between -0.0 and 0.0, then you don't > want a "close" or "approximate" test. > > >> As someone who wrote such a function for Numba, what you're proposing >> would not be a suitable replacement. >> > > I never expected it would be a replacement for what is needed for a > project like numba. > > -Chris > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150123/4dd3f002/attachment.html> From rosuav at gmail.com Fri Jan 23 18:45:35 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 24 Jan 2015 04:45:35 +1100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxE+syr4suqsJSOziJDOt4hOVAohVx9fds9PLNJZuVPM3nw@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CACac1F8jksJd=z-3u53M1hO=1ou2COpmzDdm4oNu0WwXN9gwCA@mail.gmail.com> <CALGmxE+syr4suqsJSOziJDOt4hOVAohVx9fds9PLNJZuVPM3nw@mail.gmail.com> Message-ID: <CAPTjJmpoeSMUwqc26MuJ4k+16egjgsD=nSzD3v6X1n-jENYDMw@mail.gmail.com> On Sat, Jan 24, 2015 at 4:21 AM, Chris Barker <chris.barker at noaa.gov> wrote: >> If you wanted to make that check more widely available, why not simply >> make it into a full-fledged function rather than an assertion? > > > That would be an option, but I don't think the one in unittest is the right > test anyway -- its focus on on number of decimal digits after the decimal > place is not generally useful. (that would make some sense for the Decimal > type...) > >> And if >> it's not suitable for that purpose, why does this PEP not propose >> updating the unittest assertion to use the new function? > > > well, for backward compatibility reasons, I had just assumed it was off the > table -- or a long, painful road anyway. And the unitest is very vested in > it's OO structure -- would we want add free-form functions to it? Updating the assertion to use the new function would be a matter of tweaking the implementation of unittest's assertAlmostEqual() to now call this function and assert that it returns True. The OO structure of unittest wouldn't be affected; just the exact definition of one particular assertion. I'd say that's a point worth mentioning in the PEP. Conceptually, this is going to do the same thing; yes, it's a change of definition, but obviously this won't be done in a point release anyway. It would make reasonable sense to sync them up. Alternatively, if you choose not to have that as part of the proposal, it would be worth adding a word or two of docs to unittest stating that assertAlmostEqual is not the same as is_close_to (and/or add "assertCloseTo" which would use it), as the existing implementation is all about absolute difference. ChrisA From njs at pobox.com Fri Jan 23 18:54:53 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 23 Jan 2015 17:54:53 +0000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAPTjJmpoeSMUwqc26MuJ4k+16egjgsD=nSzD3v6X1n-jENYDMw@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CACac1F8jksJd=z-3u53M1hO=1ou2COpmzDdm4oNu0WwXN9gwCA@mail.gmail.com> <CALGmxE+syr4suqsJSOziJDOt4hOVAohVx9fds9PLNJZuVPM3nw@mail.gmail.com> <CAPTjJmpoeSMUwqc26MuJ4k+16egjgsD=nSzD3v6X1n-jENYDMw@mail.gmail.com> Message-ID: <CAPJVwBn6VMxF8s0KhnJtnKh9Y43-0qMmgPJsYL9UmbbsvRcrVA@mail.gmail.com> On Fri, Jan 23, 2015 at 5:45 PM, Chris Angelico <rosuav at gmail.com> wrote: > Updating the assertion to use the new function would be a matter of > tweaking the implementation of unittest's assertAlmostEqual() to now > call this function and assert that it returns True. The OO structure > of unittest wouldn't be affected; just the exact definition of one > particular assertion. I'd say that's a point worth mentioning in the > PEP. Conceptually, this is going to do the same thing; yes, it's a > change of definition, but obviously this won't be done in a point > release anyway. It would make reasonable sense to sync them up. > Alternatively, if you choose not to have that as part of the proposal, > it would be worth adding a word or two of docs to unittest stating > that assertAlmostEqual is not the same as is_close_to (and/or add > "assertCloseTo" which would use it), as the existing implementation is > all about absolute difference. Yeah, having just taken a quick look at the source, I'd go so far as to say assertAlmostEqual is almost totally broken. I had to read the docs three times to work out that while it sorta sounds like it provides relative tolerances, it actually doesn't at all -- places=3 means something like abs_tol=10**-3. Not really appropriate for numerical work. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From chris.barker at noaa.gov Fri Jan 23 18:55:31 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 23 Jan 2015 09:55:31 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAPTjJmpoeSMUwqc26MuJ4k+16egjgsD=nSzD3v6X1n-jENYDMw@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CACac1F8jksJd=z-3u53M1hO=1ou2COpmzDdm4oNu0WwXN9gwCA@mail.gmail.com> <CALGmxE+syr4suqsJSOziJDOt4hOVAohVx9fds9PLNJZuVPM3nw@mail.gmail.com> <CAPTjJmpoeSMUwqc26MuJ4k+16egjgsD=nSzD3v6X1n-jENYDMw@mail.gmail.com> Message-ID: <CALGmxEKJsLEhz+Mag5nKxuWOJrD6yrPU8PQBbXMxedKcCZXE5g@mail.gmail.com> On Fri, Jan 23, 2015 at 9:45 AM, Chris Angelico <rosuav at gmail.com> wrote: > > And the unitest is very vested in > > it's OO structure -- would we want add free-form functions to it? > > Updating the assertion to use the new function would be a matter of > tweaking the implementation of unittest's assertAlmostEqual() to now > call this function and assert that it returns True. sure -- that's not quite what I meant. I was really addressing the "where would this sit" question. unittest does not currently have any stand-alone utility functions for testing in it. If we put this there, would anyone think to look for it there? > I'd say that's a point worth mentioning in the > PEP. well, whether to change a TestCase assertion or add a new one is a brand new question -- we could add that to this PEP if people think that's a good idea. For my part, I find unittest painful, and use py.test (and sometimes nose) anyway.... > it would be worth adding a word or two of docs to unittest stating > that assertAlmostEqual is not the same as is_close_to (and/or add > "assertCloseTo" which would use it), as the existing implementation is > all about absolute difference. > probably a good idea, yes. I really don't think we want to change assertAlmostEqual -- certainly not anytime soon. It seems like gratuitous backward incompatibility. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150123/4abef1d5/attachment.html> From p.f.moore at gmail.com Fri Jan 23 18:59:12 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 23 Jan 2015 17:59:12 +0000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxE+syr4suqsJSOziJDOt4hOVAohVx9fds9PLNJZuVPM3nw@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CACac1F8jksJd=z-3u53M1hO=1ou2COpmzDdm4oNu0WwXN9gwCA@mail.gmail.com> <CALGmxE+syr4suqsJSOziJDOt4hOVAohVx9fds9PLNJZuVPM3nw@mail.gmail.com> Message-ID: <CACac1F8iMmt-BdSL3vBpHYAQGLu62=X8jO828EWQRBr+=xujcg@mail.gmail.com> On 23 January 2015 at 17:21, Chris Barker <chris.barker at noaa.gov> wrote: >> My worry is that what this >> function will *actually* be used for is to allow naive users to gloss >> over their lack of understanding of floating point: >> >> n = 0.0 >> while not is_close_to(n, 1.0): # Because I don't understand floating >> point >> do_something_with(n) >> n += 0.1 > > Is that necessarily worse? it would at least terminate ;-) floating point is > a bit of an attractive nuisance anyway. Than understanding what you're doing? Yes. But it's sort of my point that fp is prone to people misunderstanding, and it's a shame to give people more opportunities. >> BTW, when writing that I had to keep scrolling up to see which order >> actual and expected went in. I'd imagine plenty of naive users will >> assume "it's symmetrical so it shouldn't matter" and get the order >> wrong. > > > Well, I think the biggest real issue about this (other than should it be in > the stdlib at all) is the question of a symmetrical vs. symmetrical test. I > decided to go (for this draft, anyway) with the asymmetric test, as it is > better defined and easier to reason about, and more appropriate for some > cases. And the biggest argument for a symmetric test is that it is what > people would expect. > > So I tried to make the parameter names that would make it clear (rather than > a,b or x,y) -- I think I failed on that, however -- anyone have a better > suggestion for names? It turns out "actual" is far too similar in meaning to > "expected". Your parameter names and documentation are fine - it's very obvious how to use the function when you look. It's just that you *need* to look because an asymmetric check isn't immediately intuitive. I say "immediately" because when I think about it yes, the question "is a close enough to b?" is actually asymmetric. >> In summary - it looks too much like an attractive nuisance to me, > > If it's not there, the folks will cobble somethign up themselves (and I'm > sure do, all the time). If they know what they are doing, and take care, > then great, but if not then they may get something with worse behavior that > this. Maybe they will at least understand it better, but I suspect the > pitfalls will all still be there in a typical case. And in any case, have to > take the time to write it. That's my logic anyway. Yeah, you have a point. And TBH, I can ignore this function just as easily as I currently ignore cmath.sin, so it's no big deal. Guido's example of Newton iteration is a good use case (although most of the time I'd expect to use a prebuilt function from a module, rather than build it myself with Newton iteration, but maybe that just reflects the fact that I don't do numerical programming). Paul From rosuav at gmail.com Fri Jan 23 19:00:01 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 24 Jan 2015 05:00:01 +1100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEKJsLEhz+Mag5nKxuWOJrD6yrPU8PQBbXMxedKcCZXE5g@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CACac1F8jksJd=z-3u53M1hO=1ou2COpmzDdm4oNu0WwXN9gwCA@mail.gmail.com> <CALGmxE+syr4suqsJSOziJDOt4hOVAohVx9fds9PLNJZuVPM3nw@mail.gmail.com> <CAPTjJmpoeSMUwqc26MuJ4k+16egjgsD=nSzD3v6X1n-jENYDMw@mail.gmail.com> <CALGmxEKJsLEhz+Mag5nKxuWOJrD6yrPU8PQBbXMxedKcCZXE5g@mail.gmail.com> Message-ID: <CAPTjJmqsnW7RFEKFn7nt102Fmj7kTKy2SC_cMJNZ_Oprp1ze8A@mail.gmail.com> On Sat, Jan 24, 2015 at 4:55 AM, Chris Barker <chris.barker at noaa.gov> wrote: > On Fri, Jan 23, 2015 at 9:45 AM, Chris Angelico <rosuav at gmail.com> wrote: >> >> > And the unitest is very vested in >> > it's OO structure -- would we want add free-form functions to it? >> >> Updating the assertion to use the new function would be a matter of >> tweaking the implementation of unittest's assertAlmostEqual() to now >> call this function and assert that it returns True. > > sure -- that's not quite what I meant. I was really addressing the "where > would this sit" question. > > unittest does not currently have any stand-alone utility functions for > testing in it. If we put this there, would anyone think to look for it > there? I agree, this function doesn't belong in unittest. If it's in, say, math, then unittest can call on it from there. ChrisA From guido at python.org Fri Jan 23 19:10:35 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 23 Jan 2015 10:10:35 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CACac1F8iMmt-BdSL3vBpHYAQGLu62=X8jO828EWQRBr+=xujcg@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CACac1F8jksJd=z-3u53M1hO=1ou2COpmzDdm4oNu0WwXN9gwCA@mail.gmail.com> <CALGmxE+syr4suqsJSOziJDOt4hOVAohVx9fds9PLNJZuVPM3nw@mail.gmail.com> <CACac1F8iMmt-BdSL3vBpHYAQGLu62=X8jO828EWQRBr+=xujcg@mail.gmail.com> Message-ID: <CAP7+vJLf6yFF87xjJRL_S9sAyXw0aXyMkXWrOe_Xh_LHZgy8ZQ@mail.gmail.com> On Fri, Jan 23, 2015 at 9:59 AM, Paul Moore <p.f.moore at gmail.com> wrote: > > Guido's example of Newton iteration is a good use case (although most > of the time I'd expect to use a prebuilt function from a module, > rather than build it myself with Newton iteration, but maybe that just > reflects the fact that I don't do numerical programming). > Well duh. Any algorithm that isn't already in the math module would require way too much code. The point of the example is that most people have probably seen that algorithm before, and it's only one simple step, really, so they won't be distracted by trying to understand the algorithm when the point of the example is to show how you would use is_close_to(). (And it's one of the simplest algorithms that gives an *approximation*, not an exact answer, at least not in the mathematical sense, which is also important in this case -- if the algorithm was exact there would be no need to use is_close_to().) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150123/aedc19d4/attachment.html> From p.f.moore at gmail.com Fri Jan 23 19:47:06 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 23 Jan 2015 18:47:06 +0000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAP7+vJLf6yFF87xjJRL_S9sAyXw0aXyMkXWrOe_Xh_LHZgy8ZQ@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CACac1F8jksJd=z-3u53M1hO=1ou2COpmzDdm4oNu0WwXN9gwCA@mail.gmail.com> <CALGmxE+syr4suqsJSOziJDOt4hOVAohVx9fds9PLNJZuVPM3nw@mail.gmail.com> <CACac1F8iMmt-BdSL3vBpHYAQGLu62=X8jO828EWQRBr+=xujcg@mail.gmail.com> <CAP7+vJLf6yFF87xjJRL_S9sAyXw0aXyMkXWrOe_Xh_LHZgy8ZQ@mail.gmail.com> Message-ID: <CACac1F-c+Pzn5hSDuTwBoy2g3DXvAmhtmdpQpJrADoUOre=eAQ@mail.gmail.com> On 23 January 2015 at 18:10, Guido van Rossum <guido at python.org> wrote: > On Fri, Jan 23, 2015 at 9:59 AM, Paul Moore <p.f.moore at gmail.com> wrote: >> >> Guido's example of Newton iteration is a good use case (although most >> of the time I'd expect to use a prebuilt function from a module, >> rather than build it myself with Newton iteration, but maybe that just >> reflects the fact that I don't do numerical programming). > > > Well duh. Any algorithm that isn't already in the math module would require > way too much code. The point of the example is that most people have > probably seen that algorithm before, and it's only one simple step, really, > so they won't be distracted by trying to understand the algorithm when the > point of the example is to show how you would use is_close_to(). (And it's > one of the simplest algorithms that gives an *approximation*, not an exact > answer, at least not in the mathematical sense, which is also important in > this case -- if the algorithm was exact there would be no need to use > is_close_to().) Sorry. What I was trying to say is that if I had a need for say a Bessel function, or numerical integration, or a zero of a function, I'd go hunting for a package that implemented it (something like mpmath, maybe) rather than rolling my own numerical algorithm using is_close_to(). But I do agree, that implementing numerical algorithms is a good use of is_close_to. And your example was fine, it'd make a good addition to use cases in the PEP. (But I wonder - wouldn't it work better with a "symmetrical" close-to function? That's probably a question for Chris.) Paul From solipsis at pitrou.net Fri Jan 23 21:42:42 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 23 Jan 2015 21:42:42 +0100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <20150123163618.3465f819@fsol> <54C280FA.2000007@stoneleaf.us> Message-ID: <20150123214242.201e501a@fsol> On Fri, 23 Jan 2015 09:12:26 -0800 Ethan Furman <ethan at stoneleaf.us> wrote: > On 01/23/2015 07:36 AM, Antoine Pitrou wrote: > > On Thu, 22 Jan 2015 16:40:14 -0800 Chris Barker wrote: > >> > >> Expected Uses > >> ============= > >> > >> The primary expected use case is various forms of testing -- "are the > >> results computed near what I expect as a result?" This sort of test > >> may or may not be part of a formal unit testing suite. > > > > I don't think the proposal fits the bill. For testing you want a > > function that is both 1) quite rigorous (i.e. checks equality within a > > defined number of ulps) 2) handles all special cases in a useful way > > (i.e. zeros, including distinguishing between positive and negative > > zeros, infinities, NaNs etc.). > > I disagree -- this function is not meant for mathematicians, but for the non-maths person who needs something that > works. In which use case would a "non-maths person" (what exactly does that mean?) need "something that works"? I haven't seen any serious analysis of use cases. Guido talks about the Newton algorithm but I can't understand why a "non-maths person" would want to write one implementation of that - apart from recreation or educational purposes, that is. Regards Antoine. From skip.montanaro at gmail.com Fri Jan 23 22:15:39 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Fri, 23 Jan 2015 15:15:39 -0600 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150123214242.201e501a@fsol> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <20150123163618.3465f819@fsol> <54C280FA.2000007@stoneleaf.us> <20150123214242.201e501a@fsol> Message-ID: <CANc-5UxXFVSDmsaXS_rZnZRKc9fBTpGyUvWhV6Pk78uyEocPjQ@mail.gmail.com> On Fri, Jan 23, 2015 at 2:42 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: > I can't understand why a "non-maths person" would want to write one > implementation of that - apart from recreation or educational purposes, > that is. I'll give you a real life example. I never thought of myself as a "maths person," so I guess that makes me a "non-maths person." I am a software engineer. I leave the math to people with PhDs in mathematics, statistics, and engineering. In my day job at a trading firm, I work on automated trading systems. Most models do all their internal calculations using floating point math. At some point though, the desired order prices calculated by the model's indicators need to be converted to actual prices acceptable to the exchange. Floating point numbers being what they are, a computed value will almost never correspond to a valid order price. If your computed price is very close to, but not exactly on a tick boundary and you're not careful, you might erroneously price your order too aggressively or too passively. In these situations you need to recognize when the floating point value you have is within some small tolerance equal to a price on an exact tick boundary. Furthermore, these comparisons need to take into account the different tick sizes of different contracts, The CME's Yen/USD futures contract (6Y) has a tick size (minimum change between two valid prices) of $.000001 while their Euro/USD futures contract (6E) has a tick size of $.0001. In my world, this is done in Python, though the problem arises independent of the language used. It also has nothing to do with the relative sophistication of the math used internal to the model. It is more-or-less just a case of format conversion on output. Skip From solipsis at pitrou.net Fri Jan 23 22:23:41 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 23 Jan 2015 22:23:41 +0100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <20150123163618.3465f819@fsol> <54C280FA.2000007@stoneleaf.us> <20150123214242.201e501a@fsol> <CANc-5UxXFVSDmsaXS_rZnZRKc9fBTpGyUvWhV6Pk78uyEocPjQ@mail.gmail.com> Message-ID: <20150123222341.24a85b0c@fsol> On Fri, 23 Jan 2015 15:15:39 -0600 Skip Montanaro <skip.montanaro at gmail.com> wrote: > In these situations you need to > recognize when the floating point value you have is within some small > tolerance equal to a price on an exact tick boundary. Furthermore, > these comparisons need to take into account the different tick sizes > of different contracts, The CME's Yen/USD futures contract (6Y) has a > tick size (minimum change between two valid prices) of $.000001 while > their Euro/USD futures contract (6E) has a tick size of $.0001. If you have such a precise requirement (the given tick size), you have to roll your own function, there's no point in a stdlib function, right? Regards Antoine. From njs at pobox.com Fri Jan 23 22:30:47 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 23 Jan 2015 21:30:47 +0000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> Message-ID: <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> On Fri, Jan 23, 2015 at 12:40 AM, Chris Barker <chris.barker at noaa.gov> wrote: > Existing Implementations > ------------------------ > > The standard library includes the > ``unittest.TestCase.assertAlmostEqual`` method, but it: > > * Is buried in the unittest.TestCase class > > * Is an assertion, so you can't use it as a general test (easily) > > * Uses number of decimal digits or an absolute delta, which are > particular use cases that don't provide a general relative error. I might phrase this a bit more strongly -- assertAlmostEqual is confusing and broken-by-default for common cases like comparing two small values, or comparing two large values. > The numpy package has the ``allclose()`` and ``isclose()`` functions. > > The statistics package tests include an implementation, used for its > unit tests. > > One can also find discussion and sample implementations on Stack > Overflow, and other help sites. > > These existing implementations indicate that this is a common need, > and not trivial to write oneself, making it a candidate for the > standard library. > > > Proposed Implementation > ======================= > > NOTE: this PEP is the result of an extended discussion on the > python-ideas list [1]_. > > The new function will have the following signature:: > > is_close_to(actual, expected, tol=1e-8, abs_tol=0.0) > > ``actual``: is the value that has been computed, measured, etc. > > ``expected``: is the "known" value. > > ``tol``: is the relative tolerance -- it is the amount of error > allowed, relative to the magnitude of the expected value. > > ``abs_tol``: is an minimum absolute tolerance level -- useful for > comparisons near zero. > > Modulo error checking, etc, the function will return the result of:: > > abs(expected-actual) <= max(tol*expected, abs_tol) So for reference, it looks like the differences from numpy are: 1) kwarg names: "tol" and "abs_tol" versus "atol", "rtol". Numpy's names seem fine to me, but if you want the longer ones then probably "rel_tol", "abs_tol" would be better? 2) use of max() instead of + to combine the relative and absolute tolerance. I understand that you find the + conceptually offensive, but I'm not really sure why -- max() is maybe a bit better, but it seems like much of a muchness to me in practice. (Sure, like you say further down, the total error using + might end up being higher by a factor of two or so -- but either people are specifying the tolerances they want, in which case they can say what they mean either way, or else they're just accepting the defaults, in which case they don't care.) It might be worth switching to + just for compatibility. 3) The default tolerances. Numpy is inconsistent with itself on this point though (allclose vs. assert_allclose), so I wouldn't worry about it too much :-). However, a lot of the benefit of numpy.allclose is that it will do something mostly-reasonable out-of-the-box even if the users haven't thought things through at all. 99% of the benefit of having something like this available is that it makes it easy to write tests, and 99% of the benefit of a test is that it exists and makes sure that your values are not wildly incorrect. So that's nice. BUT if you want that kind of out-of-the-box utility then you need to have some kind of sensible default for comparisons to zero. (I just did a quick look at uses of python code uses of assertAlmostEqual on github, and in my unscientific survey of reading the first page of results, 30.4% of the calls were comparisons against zero. IMO asking all these people to specify tolerances by hand on every call is not very nice.) One option would be to add a zero_tol argument, which is an absolute tolerance that is only applied if expected == 0. [And a nice possible side-effect of this is that numpy could conceivably then add such an argument as well "for compatibility with the stdlib", and possibly use this as a lever to fix it's weird allclose/assert_allclose discrepancy. The main blocker to making them consistent is that there is lots of code in the wild that assumes allclose handles comparisons-to-zeros right, and also lots of code that assumes that assert_allclose is strict with very-small non-zero numbers, and with only rtol and atol you can't get both of these behaviours simultaneously.] > Inappropriate uses > ------------------ > > One use case for floating point comparison is testing the accuracy of > a numerical algorithm. However, in this case, the numerical analyst > ideally would be doing careful error propagation analysis, and should > understand exactly what to test for. It is also likely that ULP (Unit > in the Last Place) comparison may be called for. While this function > may prove useful in such situations, It is not intended to be used in > that way. I'd strongly consider expanding the scope of this PEP a bit so that it's proposing both a relative/absolute-error-based function *and* a ULP-difference function. There was a plausible-looking one using struct posted in the other thread, it would cover a wider variety of cases, and having both functions next to each other in the docs would provide a good opportunity to explain why the differences and which might be preferred in which situation. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From skip.montanaro at gmail.com Fri Jan 23 22:34:57 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Fri, 23 Jan 2015 15:34:57 -0600 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150123222341.24a85b0c@fsol> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <20150123163618.3465f819@fsol> <54C280FA.2000007@stoneleaf.us> <20150123214242.201e501a@fsol> <CANc-5UxXFVSDmsaXS_rZnZRKc9fBTpGyUvWhV6Pk78uyEocPjQ@mail.gmail.com> <20150123222341.24a85b0c@fsol> Message-ID: <CANc-5UytkMcoGB49vQ4g214d0-R0roezGcDNUSphoWNvzANHXQ@mail.gmail.com> On Fri, Jan 23, 2015 at 3:23 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: > If you have such a precise requirement (the given tick size), you have > to roll your own function, there's no point in a stdlib function, right? No, I think Chris's is_close_to would probably do the trick, as the relative tolerance would be some fractional multiple of the tick size. In any case, whether or not I would choose to use this function is beside the point. (It's actually a real, though solved problem in my environment, so modifying code to use it wouldn't be worth the effort or potential sources of bugs at this point.) I was only pointing out that there are valid reasons where such a function might be useful to "non-math people," outside the realm of software testing. Knowing when you need something like this is often only discovered after mistakes are made though. Is a numerical analysis course still commonly taught in Computer Science departments? Skip From njs at pobox.com Fri Jan 23 22:40:48 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 23 Jan 2015 21:40:48 +0000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAP7+vJKQO8DfJbQL7O6Vs6pD=y+3ddmDxcK3Mw-BPGYf+ONH=Q@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <20150123163618.3465f819@fsol> <CALGmxEL8W7ziZkze1hdxLKoCaVrx9h7oSVZwLHxFfDe0sjBmGQ@mail.gmail.com> <CAP7+vJKQO8DfJbQL7O6Vs6pD=y+3ddmDxcK3Mw-BPGYf+ONH=Q@mail.gmail.com> Message-ID: <CAPJVwB=HvsN7kSb5Lz65abaPLC2OOzGNef9sGRHwrZJepHxf-g@mail.gmail.com> On Fri, Jan 23, 2015 at 5:41 PM, Guido van Rossum <guido at python.org> wrote: > Isn't an ulp just a base-2 way of specifying precision scaled so that 1 ulp > is the low bit of the mantissa in IEEE fp? Basically yes, but there are weird subtleties. E.g. 1 ulp remains the same absolute size between 1.0 and 2.0, so the same ulp threshold can vary by a factor of two in relative precision terms. And where you hit the boundary between exponents funny things happen: 2.0 +/- 1 ulp is [2.0 - 2.2e-16, 2.0 + 4.4e-16]. This can matter if you're looking for high precision -- if the value is supposed to be almost 2.0, then you don't want to get penalized for failing to get 2.0 + 2.2e-16, b/c there is no such number, but it might also be unacceptable to get 2 - 4.4e-16, which would be two values off. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From solipsis at pitrou.net Fri Jan 23 22:43:36 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 23 Jan 2015 22:43:36 +0100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <20150123163618.3465f819@fsol> <CALGmxEL8W7ziZkze1hdxLKoCaVrx9h7oSVZwLHxFfDe0sjBmGQ@mail.gmail.com> Message-ID: <20150123224336.17eac995@fsol> On Fri, 23 Jan 2015 08:51:00 -0800 Chris Barker <chris.barker at noaa.gov> wrote: > > As for the ulps test -- can you suggest a way to do that, while also > providing a simple definition of tolerance that casual users can understand > and use (and have a reasonable default? I know I can't. Note that some of > the feedback on the PEP as is is that it's too hard to understand already! > (without better docs, anyway) My approach was roughly: delta = 2 ** (ulps - 53 - 1) * (abs(first) + abs(second)) assertAlmostEqual(first, second, delta=delta) I don't know if it's right in the case of denormals etc. (there's also special code surrounding that to care for zeros, infinities, and NaNs) Regards Antoine. From rosuav at gmail.com Fri Jan 23 22:55:33 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 24 Jan 2015 08:55:33 +1100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> Message-ID: <CAPTjJmp4m7eEyw_7D8qyP0Ehrt+F1RSdFN6oEgXKW3FA3nX2iw@mail.gmail.com> On Sat, Jan 24, 2015 at 8:30 AM, Nathaniel Smith <njs at pobox.com> wrote: > 1) kwarg names: "tol" and "abs_tol" versus "atol", "rtol". Numpy's > names seem fine to me, but if you want the longer ones then probably > "rel_tol", "abs_tol" would be better? Longer names preferable. It was quite a long way into the original thread before I understood what "atol" meant - my brain kept wanting it to be related to the atoi family of functions from C (ASCII to Integer (atoi), ASCII to Long (atol), etc, converting strings to integers). ChrisA From alexander.belopolsky at gmail.com Fri Jan 23 22:56:22 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 23 Jan 2015 16:56:22 -0500 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> Message-ID: <CAP7h-xbStEGkzvFaHaJngkfq7RWKFs3uMRNBtQAwXe3NZVa95g@mail.gmail.com> On Fri, Jan 23, 2015 at 4:30 PM, Nathaniel Smith <njs at pobox.com> wrote: > 1) kwarg names: "tol" and "abs_tol" versus "atol", "rtol". Numpy's > names seem fine to me, but if you want the longer ones then probably > "rel_tol", "abs_tol" would be better? > Many style guides recommend against using _ to separate abbreviated words in variable names, so either relative_/absolute_tolerance or reltol/abstol. OTOH, I don't see any problem with numpy's atol/rtol. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150123/69cd4568/attachment.html> From ron3200 at gmail.com Fri Jan 23 23:06:11 2015 From: ron3200 at gmail.com (Ron Adam) Date: Fri, 23 Jan 2015 16:06:11 -0600 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <CADiSq7f6OscLNqubW_3Db_ka=UrOVn9GMES_f-5kvJuNVUL-Ew@mail.gmail.com> References: <CAP7+vJKd19Yb5yeVPt38sXHTKVjfPh_JO4PX_dngR50Vck2dNQ@mail.gmail.com> <538698398.2487257.1421798257472.JavaMail.yahoo@jws10073.mail.ne1.yahoo.com> <CAP7+vJKU2oiyhmtOsM6cA=NUB+CjqLzGehy8uXQT=Gn=0hLc6Q@mail.gmail.com> <CADiSq7f6OscLNqubW_3Db_ka=UrOVn9GMES_f-5kvJuNVUL-Ew@mail.gmail.com> Message-ID: <m9ugkk$7hq$1@ger.gmane.org> On 01/23/2015 08:56 AM, Nick Coghlan wrote: > On 21 January 2015 at 11:16, Guido van Rossum<guido at python.org> wrote: >> >So it's hard to say to what extent this was a conscious choice and to what >> >extent it was just tradition. But there's nothing necessarily wrong with >> >tradition (up to a point). I think it still makes sense that statements are >> >laid out vertically while expressions are laid out horizontally. Come to >> >think of it, mathematics uses a similar convention -- a formula is laid out >> >(primarily) horizontally, while a sequence of formulas (like a proof or a >> >set of axioms) is laid out vertically. > I really like that "each statement is like a single step in a > mathematical proof" analogy. I agree. There are some other distinctions. When you consider these, you can see how many language designers come to similar solutions. Expressions evaluate in unique name spaces, while statements generally do not. Consider "a + b"; it is evaluated in a private method after the values a and b are passed to it. Statements are used to mutate the current name space, while expressions generally do not. Statements can alter control flow, while expressions generally do not. Having a clear distinction between expressions and statements makes reading and understanding code much easier. I believe Python follows most of these conventions in most places, and when it doesn't, it's usually for a practical reason that are fairly obvious. For example, an "or" expression is a bit of both. Another example of how python chooses a practical alternative is we can nest expressions instead of using "call" statements on separate lines and a stack to hold the augments and return values. That is what python does in the byte code so we don't have to do it in explicit statements. If you factor out all expressions you get byte code. Or if you factor out all statements you get something like lisp. Of course this subject is definitely a very subjective one which relays on agreeing on the general meaning of the above sentences. OR... YMMV. Cheers, Ron From solipsis at pitrou.net Fri Jan 23 23:07:01 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 23 Jan 2015 23:07:01 +0100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CAPTjJmp4m7eEyw_7D8qyP0Ehrt+F1RSdFN6oEgXKW3FA3nX2iw@mail.gmail.com> Message-ID: <20150123230701.3627582f@fsol> On Sat, 24 Jan 2015 08:55:33 +1100 Chris Angelico <rosuav at gmail.com> wrote: > > Longer names preferable. It was quite a long way into the original > thread before I understood what "atol" meant - my brain kept wanting > it to be related to the atoi family of functions from C (ASCII to > Integer (atoi), ASCII to Long (atol), etc, converting strings to > integers). Same thing here, I think "atol" is too cryptic. Regards Antoine. From ethan at stoneleaf.us Fri Jan 23 23:14:29 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 23 Jan 2015 14:14:29 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> Message-ID: <54C2C7C5.2020100@stoneleaf.us> On 01/23/2015 01:30 PM, Nathaniel Smith wrote: > On Fri, Jan 23, 2015 at 12:40 AM, Chris Barker wrote: > So for reference, it looks like the differences from numpy are: > > 1) kwarg names: "tol" and "abs_tol" versus "atol", "rtol". Numpy's > names seem fine to me, but if you want the longer ones then probably > "rel_tol", "abs_tol" would be better? Longer names are good for us non-maths folks. ;) rel_tol and abs_tol look good to me. > 2) use of max() instead of + to combine the relative and absolute > tolerance. I understand that you find the + conceptually offensive, > but I'm not really sure why -- max() is maybe a bit better, but it > seems like much of a muchness to me in practice. (Sure, like you say > further down, the total error using + might end up being higher by a > factor of two or so -- but either people are specifying the tolerances > they want, in which case they can say what they mean either way, or > else they're just accepting the defaults, in which case they don't > care.) It might be worth switching to + just for compatibility. That makes no sense to me. I'm not sure taking the max does either, though, as phrases like "you can be off by 5% or 30 units, whichever is [smaller | greater]" comes to mind. > One option would be to add a zero_tol argument, which is an absolute > tolerance that is only applied if expected == 0. Seems reasonable. > I'd strongly consider expanding the scope of this PEP a bit so that > it's proposing both a relative/absolute-error-based function *and* a > ULP-difference function. There was a plausible-looking one using > struct posted in the other thread, it would cover a wider variety of > cases, and having both functions next to each other in the docs would > provide a good opportunity to explain why the differences and which > might be preferred in which situation. Also seems reasonable. So, in the interest of keeping things insane ;) how about this signature? def close_to(noi, target, min_tol, max_tol, rel_tol, zero_tol): """ returns True if noi is within tolerance of target noi: Number Of Interest - result of calulations target: the number we are trying to get min_tol: used with rel_tol to determine actual tolerance max_tol: used with rel_tol to determine actual tolerance zero_tol: an absolute tolerance if target == 0 (otherwise rel_tol is used an as zero_tol) """ -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150123/9b401dc8/attachment-0001.sig> From abarnert at yahoo.com Sat Jan 24 00:09:32 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 23 Jan 2015 23:09:32 +0000 (UTC) Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> References: <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> Message-ID: <1149153077.281910.1422054572644.JavaMail.yahoo@mail.yahoo.com> On Friday, January 23, 2015 1:31 PM, Nathaniel Smith <njs at pobox.com> wrote: > I'd strongly consider expanding the scope of this PEP a bit so that > it's proposing both a relative/absolute-error-based function *and* a > ULP-difference function. There was a plausible-looking one using > struct posted in the other thread If you're thinking about the post I think you are (mine), I wouldn't suggest using that. The main issue is that was a bits-difference function, not an ulps-difference function--in other words, bits_difference(x, y) is the number of times you have to do y = nexttoward(y, x) to get y == x. In the case where the difference is <= 2, or where x and y are finite numbers with the same sign and exponent, they happen to be the same, but otherwise, they don't. For example, consider x as the 5th largest number with one exponent, and y as the 5th smallest number with the next. They're 10 bits away, but 12.5 ulp(x) away and 7.5 ulp(y) away. Most algorithms that you want to test for ulp difference are specified to be within 0.5, 1, or 2 ulp, and for C lib functions it's always 0.5 or 1 (except pow in certain cases), or to no more than double the ulp difference--but definitely not _all_, so it would be misleading to offer a bits-difference function as an ulps-difference function. Secondarily, even as a bit-difference function, what I posted isn't complete (but I think the version in https://github.com/abarnert/floatextras is), and makes various decisions and assumptions that aren't necessarily the only option. Also, there's nothing else in the stdlib that directly accesses the bits of a float in Python, which seems a little weird. Finally, neither Python nor the C89 standard that CPython implies require that float actually be an IEEE 754-1985 double (much less an IEEE 754-2005 binary64, the later standard I actually have a copy of...). In particular, sys.float_info doesn't assume it. I think if we wanted this, we'd want to implement nexttoward in C (by calling the C99/POSIX2001 function if present, and maybe our own bit-twiddling-IEEE-doubles-in-C implementation for Windows, but it's not there otherwise), then define ulp (in C or Python) in terms of nexttoward, then define ulp_difference(x, y) (ditto) in terms of ulp(y). This does require a bit of care to make sure that, e.g., ulp_difference(float_info.max, inf) comes out as 1 or as an error, whichever one you want, and so on. (That means it also requires deciding what to do for each edge case, since they're not standardized by IEEE 754-1985, IEEE 754-2008, C99, or POSIX2001.) This would work correctly and consistently on almost every *nix platform (even some that don't use IEEE double) and on Windows, and wouldn't exist on platforms where it won't work correctly. Of course other implementations would have to come up with some other compatible implementation, but at least Java has an ulp function, and if .NET doesn't, it can probably make assumptions about the underlying platform. If we also want a bits_difference function in the stdlib (and I'm not sure we do), I'd suggest also writing that in C, by pointer-casting from double to int64_t and using the information in C99 math.h/limits.h (and again maybe special-casing Windows), rather than twiddling IEEE bits in Python. From abarnert at yahoo.com Sat Jan 24 00:11:45 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 23 Jan 2015 23:11:45 +0000 (UTC) Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150123230701.3627582f@fsol> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CAPTjJmp4m7eEyw_7D8qyP0Ehrt+F1RSdFN6oEgXKW3FA3nX2iw@mail.gmail.com> <20150123230701.3627582f@fsol> Message-ID: <2012016916.272168.1422054705165.JavaMail.yahoo@mail.yahoo.com> On Friday, January 23, 2015 2:09 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: > > On Sat, 24 Jan 2015 08:55:33 +1100 > Chris Angelico <rosuav at gmail.com> wrote: >> >> Longer names preferable. It was quite a long way into the original >> thread before I understood what "atol" meant - my brain kept > wanting >> it to be related to the atoi family of functions from C (ASCII to >> Integer (atoi), ASCII to Long (atol), etc, converting strings to >> integers). > > Same thing here, I think "atol" is too cryptic. Agreed, it would take some work to train myself not to see "ASCII string to long integer" whenever I see "atol"... From chris.barker at noaa.gov Sat Jan 24 00:26:04 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Fri, 23 Jan 2015 15:26:04 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <2012016916.272168.1422054705165.JavaMail.yahoo@mail.yahoo.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CAPTjJmp4m7eEyw_7D8qyP0Ehrt+F1RSdFN6oEgXKW3FA3nX2iw@mail.gmail.com> <20150123230701.3627582f@fsol> <2012016916.272168.1422054705165.JavaMail.yahoo@mail.yahoo.com> Message-ID: <-9027946073171701827@unknownmsgid> >>> Longer names preferable. I had a suggestion on github for the same thing -- how about: rel_tolerance and abs_tolerance ? >>> - my brain kept >> wanting >>> it to be related to the atoi family of functions from C (ASCII to >>> Integer (atoi), ASCII to Long (atol), etc, Not all of us are as contaminated by C ;-) in fact, when I see the C functions I first think of tolerances... Long clear names are good. -Chris From chris.barker at noaa.gov Sat Jan 24 00:32:35 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Fri, 23 Jan 2015 15:32:35 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <1149153077.281910.1422054572644.JavaMail.yahoo@mail.yahoo.com> References: <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <1149153077.281910.1422054572644.JavaMail.yahoo@mail.yahoo.com> Message-ID: <-3728738344276704028@unknownmsgid> >> I'd strongly consider expanding the scope of this PEP a bit so that >> it's proposing both a relative/absolute-error-based function *and* a >> ULP-difference function. The next response makes it clear why I think that's out of scope for this proposal -- it is considerably harder for casual users to wrap their brains around, so I think if such a thing exists, it should probably be a different function. Not that two functions can't be in the same PEP. But in any case, I'm not qualified to write it ( certainly not the code, but not really the PEP either) If someone else wants to champion that part, I'm happy to work together however makes sense. -Chris > > If you're thinking about the post I think you are (mine), I wouldn't suggest using that. > > The main issue is that was a bits-difference function, not an ulps-difference function--in other words, bits_difference(x, y) is the number of times you have to do y = nexttoward(y, x) to get y == x. In the case where the difference is <= 2, or where x and y are finite numbers with the same sign and exponent, they happen to be the same, but otherwise, they don't. For example, consider x as the 5th largest number with one exponent, and y as the 5th smallest number with the next. They're 10 bits away, but 12.5 ulp(x) away and 7.5 ulp(y) away. Most algorithms that you want to test for ulp difference are specified to be within 0.5, 1, or 2 ulp, and for C lib functions it's always 0.5 or 1 (except pow in certain cases), or to no more than double the ulp difference--but definitely not _all_, so it would be misleading to offer a bits-difference function as an ulps-difference function. > > Secondarily, even as a bit-difference function, what I posted isn't complete (but I think the version in https://github.com/abarnert/floatextras is), and makes various decisions and assumptions that aren't necessarily the only option. Also, there's nothing else in the stdlib that directly accesses the bits of a float in Python, which seems a little weird. Finally, neither Python nor the C89 standard that CPython implies require that float actually be an IEEE 754-1985 double (much less an IEEE 754-2005 binary64, the later standard I actually have a copy of...). In particular, sys.float_info doesn't assume it. > > I think if we wanted this, we'd want to implement nexttoward in C (by calling the C99/POSIX2001 function if present, and maybe our own bit-twiddling-IEEE-doubles-in-C implementation for Windows, but it's not there otherwise), then define ulp (in C or Python) in terms of nexttoward, then define ulp_difference(x, y) (ditto) in terms of ulp(y). This does require a bit of care to make sure that, e.g., ulp_difference(float_info.max, inf) comes out as 1 or as an error, whichever one you want, and so on. (That means it also requires deciding what to do for each edge case, since they're not standardized by IEEE 754-1985, IEEE 754-2008, C99, or POSIX2001.) This would work correctly and consistently on almost every *nix platform (even some that don't use IEEE double) and on Windows, and wouldn't exist on platforms where it won't work correctly. > > Of course other implementations would have to come up with some other compatible implementation, but at least Java has an ulp function, and if .NET doesn't, it can probably make assumptions about the underlying platform. > > If we also want a bits_difference function in the stdlib (and I'm not sure we do), I'd suggest also writing that in C, by pointer-casting from double to int64_t and using the information in C99 math.h/limits.h (and again maybe special-casing Windows), rather than twiddling IEEE bits in Python. From chris.barker at noaa.gov Sat Jan 24 02:41:17 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Fri, 23 Jan 2015 17:41:17 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> Message-ID: <8835515035530371300@unknownmsgid> > 2) use of max() instead of + to combine the relative and absolute > tolerance. In fact, the code uses "or", but in amounts to the same thing -- if the difference is within either the relative or absolute tolerance, it's "close". > I understand that you find the + conceptually offensive, > but I'm not really sure why -- max() is maybe a bit better, but it > seems like much of a muchness to me in practice. Actually I agree with you here -- I think I've said elsewhere that I expect in practice people will set their tolerance to an order of magnitude, so even a factor of two doesn't much matter. But I see no advantage to doing it that way (except perhaps as a vectorized computation, which this is not) > It might be worth switching to + just for compatibility. Well, the other difference is that numpy's version sets a default non-zero absolute tolerance. I think this is fatally wrong. Way too easy to get something really wrong for small values. Once we've done something incompatible, why not make it cleaner? And I see little reason for compatability for it's own sake. > However, a lot of the benefit of numpy.allclose is that it will do > something mostly-reasonable out-of-the-box even if the users haven't > thought things through at all. 99% of the benefit of having something > like this available is that it makes it easy to write tests, and 99% > of the benefit of a test is that it exists and makes sure that your > values are not wildly incorrect. So that's nice. BUT if you want that > kind of out-of-the-box utility then you need to have some kind of > sensible default for comparisons to zero. I spent some time thinking about this, and my first version did have a default abs_tol to cover the near-zero case. But it would be absolutely the wrong thing for comparing small values. If you can think of defaults and an algorithm that would work well for large and small values and also comparison to zero, I'd be all for it. > (I just did a quick look at uses of python code uses of > assertAlmostEqual on github, and in my unscientific survey of reading > the first page of results, 30.4% of the calls were comparisons against > zero. IMO asking all these people to specify tolerances by hand on > every call is not very nice.) Hmm -- my thinking is that at least those tests would immediately not work, but agreed, nicer for defaults to work for common cases. > One option would be to add a zero_tol argument, which is an absolute > tolerance that is only applied if expected == 0. Here is where I'm not sure: is there only an issue with comparing to exactly zero? Or can vet small numbers under flow and cause the same problem? > [And a nice possible side-effect of this is that numpy could > conceivably then add such an argument as well "for compatibility with > the stdlib", and possibly use this as a lever to fix it's weird > allclose/assert_allclose discrepancy. The main blocker to making them > consistent is that there is lots of code in the wild that assumes > allclose handles comparisons-to-zeros right, and also lots of code > that assumes that assert_allclose is strict with very-small non-zero > numbers, and with only rtol and atol you can't get both of these > behaviours simultaneously.] I'm not sure it's much of an incentive for ghe Stalin, but sure, that would be nice. > I'd strongly consider expanding the scope of this PEP a bit so that > it's proposing both a relative/absolute-error-based function *and* a > ULP-difference function. I responded to this elsewhere. Thanks for your input. -Chris From chris.barker at noaa.gov Sat Jan 24 02:56:14 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Fri, 23 Jan 2015 17:56:14 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CACac1F-c+Pzn5hSDuTwBoy2g3DXvAmhtmdpQpJrADoUOre=eAQ@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CACac1F8jksJd=z-3u53M1hO=1ou2COpmzDdm4oNu0WwXN9gwCA@mail.gmail.com> <CALGmxE+syr4suqsJSOziJDOt4hOVAohVx9fds9PLNJZuVPM3nw@mail.gmail.com> <CACac1F8iMmt-BdSL3vBpHYAQGLu62=X8jO828EWQRBr+=xujcg@mail.gmail.com> <CAP7+vJLf6yFF87xjJRL_S9sAyXw0aXyMkXWrOe_Xh_LHZgy8ZQ@mail.gmail.com> <CACac1F-c+Pzn5hSDuTwBoy2g3DXvAmhtmdpQpJrADoUOre=eAQ@mail.gmail.com> Message-ID: <-4474967552440356036@unknownmsgid> > (But I wonder - wouldn't it work better with > a "symmetrical" close-to function? That's probably a question for > Chris.) Well, you usually use Newton's algorithm to find the zero of a function, so in that case, you'd want an absolute comparison. But it's pretty common to do a simple iterative solution where you check convergence by seeing if the new solution is close to the previous solution, in which case, a symmetric test would probability be better, but the asymmetric one would be fine -- you'd be asking the question -- is the new solution close to the previous one? -Chris From rosuav at gmail.com Sat Jan 24 03:22:52 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 24 Jan 2015 13:22:52 +1100 Subject: [Python-ideas] OT or going meta: How this mailing list is perceived Message-ID: <CAPTjJmoRUxUhD=2Rhb3eA7ipJpmur38Vd2Y6Vct4iFkpqtMd-A@mail.gmail.com> Some of you here probably saw/heard David Beazley at PyCon 2012, but I'm only just now finding out about this thanks to the magic of the internet. The talk is long, but I just want to highlight this one little sequence (about 20 seconds)... and how python-ideas is depicted. http://youtu.be/6_-5XZzJyt0?t=3m40s :) ChrisA From abarnert at yahoo.com Sat Jan 24 03:35:41 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 23 Jan 2015 18:35:41 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <8835515035530371300@unknownmsgid> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <8835515035530371300@unknownmsgid> Message-ID: <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> On Jan 23, 2015, at 17:41, Chris Barker - NOAA Federal <chris.barker at noaa.gov> wrote: >> One option would be to add a zero_tol argument, which is an absolute >> tolerance that is only applied if expected == 0. > > Here is where I'm not sure: is there only an issue with comparing to > exactly zero? Or can vet small numbers under flow and cause the same > problem? Of course they can underflow. But I don't think that's a practical problem except in very rare cases. It means you're explicitly asking for better than +/- 2**min_exp, so it shouldn't be surprising that nothing but an exact match qualifies. Take a concrete example: with a tol of 1e-5, it's only going to underflow if expected is around 1e-320 or below. But since the next smaller and larger numbers (9.95e-321 and 1.0005e-320) aren't within 1e-5, the test gives the right answer despite underflowing. I'd have to think about it a bit to make sure there's no pathological case that doesn't work out that way--but really, if you're checking subnormal numbers for closeness with a general-purpose function, or checking for relative closeness pushing the bounds of 1 ulp without thinking about what that means, I suspect you're already doing something wrong at a much higher level. So, just special-casing 0 should be sufficient. Maybe the answer there is to have an is_close_to_0 function, instead of a parameter that's only useful if expected is 0? But then you might have, say, a comprehension where some of the expected values are 0 and some aren't, so maybe not... From rosuav at gmail.com Sat Jan 24 03:41:14 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 24 Jan 2015 13:41:14 +1100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <8835515035530371300@unknownmsgid> <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> Message-ID: <CAPTjJmqv0pcdGFg-u8HfUBwapa=agEO5tN+YsiVXjSykTD_uDg@mail.gmail.com> On Sat, Jan 24, 2015 at 1:35 PM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: > Maybe the answer there is to have an is_close_to_0 function, instead of a parameter that's only useful if expected is 0? But then you might have, say, a comprehension where some of the expected values are 0 and some aren't, so maybe not... > That's a more philosophical question about API design. It's probably worth mentioning the two options in the PEP - separate function for "close to zero" with these args, or put it all into the one function with those args. ChrisA From abarnert at yahoo.com Sat Jan 24 04:32:20 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 23 Jan 2015 19:32:20 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAPTjJmqv0pcdGFg-u8HfUBwapa=agEO5tN+YsiVXjSykTD_uDg@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <8835515035530371300@unknownmsgid> <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> <CAPTjJmqv0pcdGFg-u8HfUBwapa=agEO5tN+YsiVXjSykTD_uDg@mail.gmail.com> Message-ID: <802F298B-24F7-4629-AA90-BDC95182ED02@yahoo.com> On Jan 23, 2015, at 18:41, Chris Angelico <rosuav at gmail.com> wrote: > On Sat, Jan 24, 2015 at 1:35 PM, Andrew Barnert > <abarnert at yahoo.com.dmarc.invalid> wrote: >> Maybe the answer there is to have an is_close_to_0 function, instead of a parameter that's only useful if expected is 0? But then you might have, say, a comprehension where some of the expected values are 0 and some aren't, so maybe not... > > That's a more philosophical question about API design. It's probably > worth mentioning the two options in the PEP - separate function for > "close to zero" with these args, or put it all into the one function > with those args. Sure. And obviously the PEP has to pick one of the options and make the case for it. I just wanted to make it clear that (I'm pretty sure) it really is only 0 that's special here, not "any subnormal" or "small numbers in some vague sense" or anything like that. The problem is that nothing is within even a ridiculously huge relative tolerance of 0 except 0; plenty of things (or at least as many values as you have reason to expect) are within a reasonable tolerance of a nonzero subnormal. (Also, this is just about zero_tol, not about the wider abs-and-rel issue, which I have nothing to add to.) From rosuav at gmail.com Sat Jan 24 04:34:53 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 24 Jan 2015 14:34:53 +1100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <802F298B-24F7-4629-AA90-BDC95182ED02@yahoo.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <8835515035530371300@unknownmsgid> <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> <CAPTjJmqv0pcdGFg-u8HfUBwapa=agEO5tN+YsiVXjSykTD_uDg@mail.gmail.com> <802F298B-24F7-4629-AA90-BDC95182ED02@yahoo.com> Message-ID: <CAPTjJmqkCeau9RR43E_kfuQJAqt2pA0CQnJM2Nuz7RnfK3sNMA@mail.gmail.com> On Sat, Jan 24, 2015 at 2:32 PM, Andrew Barnert <abarnert at yahoo.com> wrote: >> That's a more philosophical question about API design. It's probably >> worth mentioning the two options in the PEP - separate function for >> "close to zero" with these args, or put it all into the one function >> with those args. > > Sure. And obviously the PEP has to pick one of the options and make the case for it. Right. Pick one, and have a section of "Alternate proposals" or "Rejected sub-proposals" or something, in which the other is explained. If nothing else, they make entertaining reading :) ChrisA From abarnert at yahoo.com Sat Jan 24 04:57:47 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 23 Jan 2015 19:57:47 -0800 Subject: [Python-ideas] math.nexttoward, math.ulp, etc. Message-ID: <72DB0CC0-A376-493F-A9C7-94803EFFA62D@yahoo.com> Some of this has come up multiple times in the discussion of the is_close_to PEP and the math.nan idea, but I think it's actually a separate issue. I think it might be worth adding some of the following to math: * isnormal (like C) * nextafter/nexttoward (like C) and/or nextplus/nextminus (like decimal) * bitdifference (inverse of nexttoward) * ulp (like Java) * signbit (like C) * isqnan, issnan (like decimal) * makenan(payload, quiet=True, sign=False) and nanpayload(f) * as_tuple and from_tuple (like decimal--although of course the tuple isn't quite the same) In some cases, it would be perfectly reasonable for these functions to only be available on, e.g., Windows (maybe even only NT, not CE) plus any platform that confirms to either C99 or POSIX-2001 (which includes almost all non-Windows platforms you're likely to run into nowadays), or only on platforms that use IEEE 754-2008 or Intel-style IEEE 754-1985 (again, almost everything), etc., as appropriate; no need to try to implement them all fully generally from scratch for any C89 platform just in case someone decides to build Python 3.5 for 68k A/UX or something. From stephen at xemacs.org Sat Jan 24 04:59:15 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 24 Jan 2015 12:59:15 +0900 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJ+H7gz4Qeqo0fZ6xW3Hvtf87PDmOmdymyC2pydKKPYaQw@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> <CAP1=2W59Pxr4j_Cx+Cvd+WQQZ4RznOSrEL9q_G2wZcB=F81=wQ@mail.gmail.com> <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> <CAP7+vJLmN_YwmHOAXzqm5Dga1aQY8ygu6Z86WFaDCM-JLTh6aQ@mail.gmail.com> <m9sjm6$v3u$1@ger.gmane.org> <CAP7+vJ+H7gz4Qeqo0fZ6xW3Hvtf87PDmOmdymyC2pydKKPYaQw@mail.gmail.com> Message-ID: <87oapou8vg.fsf@uwakimon.sk.tsukuba.ac.jp> Guido van Rossum writes: > > I think typing should have a make_stub(inpy, outstub) function to make > > blank stub creation easy. [...] > > That sounds like a useful tool, but why should it be in typing.py? It's an obvious place to look for it. You could put it in the checking application, but with multiple applications that would result in code duplication, wouldn't it? > > I think putting type hints in stub files should be documented as at least > > on a par, if not encouraged, with putting them in .py files. > > On a par. Stubs are essential for extension modules, and nearly essential > for existing packages (both 3rd party and the stdlib!). But I don't want > people writing new 3.x code having to deal with two files for every module. I think what Terry is suggesting is that if you want *both* type hints and some other use of annotations, you could get that by putting the type hints in a separate file, and that this should be documented. I agree the arguments for "encouraged" are weak (except in case where multiple forms of annotation are desired), and the annoyance of the C/C++ .c/.h partition is quite discouraging. Note that, IIUC Terry's idea, this would also allow library authors or users to disable type hinting warnings permanently with "touch <stubfile>". I think that's a reasonable burden, especially since it only requires directory write access for the user to disable warnings if they have a package using non-typing annotations. Aside: I can't agree with those who think that many users of typing should have to request type-checking *three* times: once by writing the hint, once by running the checker, and once by importing typing. From chris.barker at noaa.gov Sat Jan 24 05:13:57 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 23 Jan 2015 20:13:57 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <8835515035530371300@unknownmsgid> <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> Message-ID: <CALGmxELqbpOvLjkAgRMxFDwz7v2h=hx+paeYUpq-kCq7FGshUQ@mail.gmail.com> On Fri, Jan 23, 2015 at 6:35 PM, Andrew Barnert <abarnert at yahoo.com> wrote: > Of course they can underflow. But I don't think that's a practical problem > except in very rare cases. It means you're explicitly asking for better > than +/- 2**min_exp, so it shouldn't be surprising that nothing but an > exact match qualifies. > > Take a concrete example: with a tol of > 1e-5, it's only going to underflow if expected is around 1e-320 or below. > But since the next smaller and larger numbers (9.95e-321 and 1.0005e-320) > aren't within 1e-5, the test gives the right answer despite underflowing. > > I'd have to think about it a bit to make sure there's no pathological case > that doesn't work out that way--but really, if you're checking subnormal > numbers for closeness with a general-purpose function, or checking for > relative closeness pushing the bounds of 1 ulp without thinking about what > that means, I suspect you're already doing something wrong at a much higher > level. > > So, just special-casing 0 should be sufficient. > cool, this may be fine then. And makes a lot of sense. Maybe the answer there is to have an is_close_to_0 function, instead of a > parameter that's only useful if expected is 0? But then you might have, > say, a comprehension where some of the expected values are 0 and some > aren't, so maybe not... exactly -- maybe it's because I'm so used to numpy, but expect that folks would want to call this in a comprehension or something where you've got a wide range of numbers, but want to use the same function and parameters, and don't want it to blow up at zero. I'll try adding this tomorrow, and see how it works out. with some tests, etc. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150123/5503aef3/attachment.html> From chris.barker at noaa.gov Sat Jan 24 05:21:29 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 23 Jan 2015 20:21:29 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAPTjJmqv0pcdGFg-u8HfUBwapa=agEO5tN+YsiVXjSykTD_uDg@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <8835515035530371300@unknownmsgid> <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> <CAPTjJmqv0pcdGFg-u8HfUBwapa=agEO5tN+YsiVXjSykTD_uDg@mail.gmail.com> Message-ID: <CALGmxE+GXv_CNUHxQXwavXfz97Niji8CPM4gjqA88_x0GZkoOw@mail.gmail.com> On Fri, Jan 23, 2015 at 6:41 PM, Chris Angelico <rosuav at gmail.com> wrote: > On Sat, Jan 24, 2015 at 1:35 PM, Andrew Barnert > <abarnert at yahoo.com.dmarc.invalid> wrote: > > Maybe the answer there is to have an is_close_to_0 function, instead of > a parameter that's only useful if expected is 0? But then you might have, > say, a comprehension where some of the expected values are 0 and some > aren't, so maybe not... > > > > That's a more philosophical question about API design. It's probably > worth mentioning the two options in the PEP - separate function for > "close to zero" with these args, or put it all into the one function > with those args. > I could mention it -- though I started all this thinking that we should keep relative and absolute tolerance separate, then realized that relative was going to be useless for zero, so added the absolute tolerance to cover that (which I originally named zero_tol, but realized that it really was absolute everywhere...). Now that we're thinking that we can have a switch for the exactly zero case, then there may be no need to have an absoute tolerance parameter, but only relative and zero. Then you'd have a separate function (if you wanted) for absolute tolerance, so it could have a default -- that may be better than requiring the user to set the parameter to get an absolute tolerance test at all. I guess the key question is if someone would want both an relative tolerance and an absolute tolerance, aside from the zero issue. -Chris > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150123/e04e97e0/attachment.html> From guido at python.org Sat Jan 24 05:40:21 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 23 Jan 2015 20:40:21 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <87oapou8vg.fsf@uwakimon.sk.tsukuba.ac.jp> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <54C03E94.9050800@canterbury.ac.nz> <CAP7+vJL8UdZ6fW0X9FAttU8RzxPKRp1_82KQ6ZkVfQh_K6BMfw@mail.gmail.com> <54C0516D.5090902@canterbury.ac.nz> <CA+KNMz=w7TR5GXhGhWJ-Cp8+TsL81-Mim6HXCFLN0U-PoWu3jQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> <CAP1=2W59Pxr4j_Cx+Cvd+WQQZ4RznOSrEL9q_G2wZcB=F81=wQ@mail.gmail.com> <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> <CAP7+vJLmN_YwmHOAXzqm5Dga1aQY8ygu6Z86WFaDCM-JLTh6aQ@mail.gmail.com> <m9sjm6$v3u$1@ger.gmane.org> <CAP7+vJ+H7gz4Qeqo0fZ6xW3Hvtf87PDmOmdymyC2pydKKPYaQw@mail.gmail.com> <87oapou8vg.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <CAP7+vJJqHSGCo_krBtwEhnS_w1NzSRp2mOL5gXZwbQqvJyYQ7A@mail.gmail.com> On Fri, Jan 23, 2015 at 7:59 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote: > Guido van Rossum writes: > > > > I think typing should have a make_stub(inpy, outstub) function to make > > > blank stub creation easy. [...] > > > > That sounds like a useful tool, but why should it be in typing.py? > > It's an obvious place to look for it. You could put it in the > checking application, but with multiple applications that would result > in code duplication, wouldn't it? > Hm. It sounds like a kitchen sink or grab bag approach. Nothing else in typing.py has even the remotest use for I/O or text processing. It would be a whole bunch of code that would have nothing in common with the rest of the stuff in typing.py except the name. If we wanted a calculator app, should it live in math? Or perhaps in decimal? I don't think so. FWIW, I just remembered that there is actually a really cool stub generator that's part of mypy: pinfer <https://github.com/JukkaL/mypy/tree/master/pinfer> (written or at least greatly improved by Dropbox intern Jared Pochtar last summer). It runs your program (typically a unittest suite) and through introspection discovers the argument types of all methods that are called. There's also an old mypy issue <https://github.com/JukkaL/mypy/issues/43> referring to an old mypy wiki page <http://www.mypy-lang.org/wiki/StubGeneratorTool> about stub generation. > > > I think putting type hints in stub files should be documented as at > least > > > on a par, if not encouraged, with putting them in .py files. > > > > On a par. Stubs are essential for extension modules, and nearly > essential > > for existing packages (both 3rd party and the stdlib!). But I don't want > > people writing new 3.x code having to deal with two files for every > module. > > I think what Terry is suggesting is that if you want *both* type hints > and some other use of annotations, you could get that by putting the > type hints in a separate file, and that this should be documented. Yeah, as a transitional measure. Eventually other use of annotations will look weird to most people. > I agree the arguments for "encouraged" are weak (except in case where > multiple forms of annotation are desired), and the annoyance of the > C/C++ .c/.h partition is quite discouraging. > > Note that, IIUC Terry's idea, this would also allow library authors > or users to disable type hinting warnings permanently with "touch > <stubfile>". I think that's a reasonable burden, especially since it > only requires directory write access for the user to disable warnings > if they have a package using non-typing annotations. > But with the current concept of stub files (borrowed from mypy <http://www.mypy-lang.org/wiki/CreatingStubsForPythonModules>) an empty stub file is not enough. The functions and classes defined in the stub tell the type checker what can be imported from the corresponding real module. I think it would be simpler to have some other, out-of-band way to signal that a module is off-limits. (I've proposed a configuration setting or command line flag to the checker.) This reminds me -- if you have a "# type: OFF" comment at the top of a module, I presume the type checker still reads the rest of the file (at the very least looking for "# type: ON"), and it may still record the presence of things defined there, for the benefit of checking that things imported *from* that module at least exist. The way I envisioned "# type: OFF" was that the type checker will ignore annotations, essentially assuming that annotations it sees don't exist. But when it sees the following in file a.py: # type: OFF def foo(a): return a+1 and in file b.py there is this: from a import foo foo(1, 2, 3) foo('hello') It might still complain about the foo(1, 2, 3) call in b.py, because it still knows that a.foo() takes exactly one argument. My assumption so far has been that # type: OFF tells the checker that it should not interpret annotations, because they may be used for other purposes. (This has been the main reason for proposing # type: OFF.) But maybe we want it to shut up more completely? Maybe there should be a way to tell it that the entire *module object* for a has type Any, so that *all* operations on it are allowed (and return Any)? An empty stub doesn't do that -- but a config option like I proposed above could. Or should we allow some special non-Python syntax in stub files to signal the same thing? E.g. * = * # type: Any or * = Undefined(Any) It might not be hard to teach a type checker (or other stub parser) about such a magic pattern (if there was exactly one form), and instead of touch a.pyi we could use 'echo "* = Undefined(Any)" >a.pyi. Sorry for the rambling post. > Aside: I can't agree with those who think that many users of typing > should have to request type-checking *three* times: once by writing > the hint, once by running the checker, and once by importing typing. > Thanks. I doubt that's going to happen. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150123/755b7155/attachment-0001.html> From abarnert at yahoo.com Sat Jan 24 06:02:32 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 24 Jan 2015 05:02:32 +0000 (UTC) Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxE+GXv_CNUHxQXwavXfz97Niji8CPM4gjqA88_x0GZkoOw@mail.gmail.com> References: <CALGmxE+GXv_CNUHxQXwavXfz97Niji8CPM4gjqA88_x0GZkoOw@mail.gmail.com> Message-ID: <780443585.320289.1422075752374.JavaMail.yahoo@mail.yahoo.com> On Friday, January 23, 2015 8:22 PM, Chris Barker <chris.barker at noaa.gov> wrote: >On Fri, Jan 23, 2015 at 6:41 PM, Chris Angelico <rosuav at gmail.com> wrote: > >On Sat, Jan 24, 2015 at 1:35 PM, Andrew Barnert >><abarnert at yahoo.com.dmarc.invalid> wrote: >>> Maybe the answer there is to have an is_close_to_0 function, instead of a parameter that's only useful if expected is 0? But then you might have, say, a comprehension where some of the expected values are 0 and some aren't, so maybe not... >>> >> >>That's a more philosophical question about API design. It's probably >>worth mentioning the two options in the PEP - separate function for >>"close to zero" with these args, or put it all into the one function >>with those args. > >I could mention it -- though I started all this thinking that we should keep relative and absolute tolerance separate, then realized that relative was going to be useless for zero, so added the absolute tolerance to cover that (which I originally named zero_tol, but realized that it really was absolute everywhere...). Now that we're thinking that we can have a switch for the exactly zero case, then there may be no need to have an absoute tolerance parameter, but only relative and zero. > >Then you'd have a separate function (if you wanted) for absolute tolerance, so it could have a default -- that may be better than requiring the user to set the parameter to get an absolute tolerance test at all. > >I guess the key question is if someone would want both an relative tolerance and an absolute tolerance, aside from the zero issue. Which already raises whether they'd want to min, max, average, or sum the two. And frankly I have no idea. That's exactly the question I didn't want to even try to answer, because smarter people than me have already given arguments about this and I don't have anything extra to contribute. But I guess you do have to answer it, since you're writing the PEP. :) From abarnert at yahoo.com Sat Jan 24 06:11:56 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 24 Jan 2015 05:11:56 +0000 (UTC) Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJJqHSGCo_krBtwEhnS_w1NzSRp2mOL5gXZwbQqvJyYQ7A@mail.gmail.com> References: <CAP7+vJJqHSGCo_krBtwEhnS_w1NzSRp2mOL5gXZwbQqvJyYQ7A@mail.gmail.com> Message-ID: <356482138.318695.1422076316714.JavaMail.yahoo@mail.yahoo.com> On Friday, January 23, 2015 8:41 PM, Guido van Rossum <guido at python.org> wrote: >FWIW, I just remembered that there is actually a really cool stub generator that's part of mypy: pinfer (written or at least greatly improved by Dropbox intern Jared Pochtar last summer). It runs your program (typically a unittest suite) and through introspection discovers the argument types of all methods that are called. There's also an old mypy issue referring to an old mypy wiki page about stub generation. That's very cool. And it seems like something that different implementations could usefully compete on. For an extreme example, PyPy has within it an algorithm to statically walk all code paths in your bytecode and infer all types for RPython. Modifying that to handle full Python by falling back to Any instead of raising an exception when it finds a potentially heterogeneous collection or a use of eval or whatever seems like it might be a plausible project. Although it would probably be a whole lot slower than the mypy stub generator, it might also be useful as something you only run once/release. From stephen at xemacs.org Sat Jan 24 06:29:48 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 24 Jan 2015 14:29:48 +0900 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAP7+vJLf6yFF87xjJRL_S9sAyXw0aXyMkXWrOe_Xh_LHZgy8ZQ@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CACac1F8jksJd=z-3u53M1hO=1ou2COpmzDdm4oNu0WwXN9gwCA@mail.gmail.com> <CALGmxE+syr4suqsJSOziJDOt4hOVAohVx9fds9PLNJZuVPM3nw@mail.gmail.com> <CACac1F8iMmt-BdSL3vBpHYAQGLu62=X8jO828EWQRBr+=xujcg@mail.gmail.com> <CAP7+vJLf6yFF87xjJRL_S9sAyXw0aXyMkXWrOe_Xh_LHZgy8ZQ@mail.gmail.com> Message-ID: <87mw58u4oj.fsf@uwakimon.sk.tsukuba.ac.jp> Guido van Rossum writes: > the point of the [Newton's method] example is to show how you would > use is_close_to(). Except that this clearly is a Cauchy test, the algorithm doesn't know the limit. In principle, the appropriate computation would be symmetric. I don't think this is a problem in practice[1], but Skip's "straddling the tick" example is much stronger for an asymmetric comparison function. On the other hand, Skip's case requires an absolute comparison, not a relative one. The whole discussion has been really fast and loose about use cases. People with strong preferences can't seem to wrap their heads around others' use cases, examples poorly matched to the proposals are common, the expertise of the numerical experts seems irrelevant because we *don't* want accuracy even in corner cases, we just want to make it easier for naive users to avoid writing "x == y". ISTM that this PEP can be reduced to We need a floating comparison function that's good enough for government work, to help naive users avoid writing "x == y" for floating point comparisons. There are use cases where one of the values is a known accurate value, so the comparison function is asymmetric. This generally won't get things "too wrong" for symmetric comparisons, except where a relative comparison involves true values near zero. Unfortunately, not much can be done in that case because it requires enough domain knowledge to realize that true values near zero occur and that this is a problem, so use of this function is covered by "consenting adults".[2] And oh yeah, IANAEINA.[3] But for this PEP, I don't need to be. <wink/> Footnotes: [1] I've reconsidered. A priori, I still like symmetric errors better in general, but the target audience for this function isn't going to be reasoning about equivalence classes of IEEE 754 floats. [2] As is all use of floating point. [3] I am not an expert in numerical analysis. Which IIUC applies to the PEP author as well as to this poster. From abarnert at yahoo.com Sat Jan 24 07:06:53 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 24 Jan 2015 06:06:53 +0000 (UTC) Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <m9ugkk$7hq$1@ger.gmane.org> References: <m9ugkk$7hq$1@ger.gmane.org> Message-ID: <1006605403.316470.1422079613279.JavaMail.yahoo@mail.yahoo.com> On Friday, January 23, 2015 2:06 PM, Ron Adam <ron3200 at gmail.com> wrote: > > > > On 01/23/2015 08:56 AM, Nick Coghlan wrote: >> On 21 January 2015 at 11:16, Guido van Rossum<guido at python.org> > wrote: >>> >So it's hard to say to what extent this was a conscious choice > and to what >>> >extent it was just tradition. But there's nothing necessarily > wrong with >>> >tradition (up to a point). I think it still makes sense that > statements are >>> >laid out vertically while expressions are laid out horizontally. > Come to >>> >think of it, mathematics uses a similar convention -- a formula is > laid out >>> >(primarily) horizontally, while a sequence of formulas (like a > proof or a >>> >set of axioms) is laid out vertically. > >> I really like that "each statement is like a single step in a >> mathematical proof" analogy. > > I agree. > > > There are some other distinctions. When you consider these, you can see > how many language designers come to similar solutions. > > > Expressions evaluate in unique name spaces, while statements generally do > not. Consider "a + b"; it is evaluated in a private method after the > values > a and b are passed to it. I don't think that's true. Consider "a[b] = c", which is a statement, but it's evaluated in a private method after the values a, b, and c are passed to it. The fact that it's a.__setitem__ rather than a.__add__ doesn't seem particularly important. I think the key is that __setitem__ doesn't have a return value--like (nearly) everything in Python that mutates state--and therefore there's no expression for it. The question is, what does that buy? > Statements are used to mutate the current name space, while expressions > generally do not. This is closer. I think, more generally, statements are used to mutate the important state that the local function is all about mutating. And the thing that gets mutated is almost always the leftmost thing. That definitely helps in scannability. > Statements can alter control flow, while expressions generally do not. Sure, but I think this part only really helps if control flow is somehow visible. In Python, it is, because control flow almost always means compound statements, which means indentation, and very little else means indentation. > Having a clear distinction between expressions and statements makes reading > and understanding code much easier. Definitely. That's the part I think is key, but am struggling to explain. I think Guido offers a great analogy in mathematical proofs, but the question is to find the actual commonality behind the analogy. After some more thought on this, I think what it comes down to is that (idiomatically-written) Python lets you skim the control flow (because all non-trivial control flow, and very little else, is expressed in terms of indentation) and the state transitions (because each statement generally mutates at most one thing, and it's the leftmost thing), so you can quickly find the part of the code you actually need to read carefully, instead of having to read the whole thing. I've written this idea up in a bit more detail here: http://stupidpythonideas.blogspot.com/2015/01/statements-and-expressions.html > I believe Python follows most of these conventions in most places, and when > it doesn't, it's usually for a practical reason that are fairly obvious. > > For example, an "or" expression is a bit of both. > > Another example of how python chooses a practical alternative is we can > nest expressions instead of using "call" statements on separate lines > and a > stack to hold the augments and return values. That is what python does in > the byte code so we don't have to do it in explicit statements. > > If you factor out all expressions you get byte code. Or if you factor out > all statements you get something like lisp. I don't think either part of that is true. Bytecode is full of things that are expressions?even your paradigm case of an operator expression is handled by an opcode. And conversely, CoffeeScript (if you avoid break/continue/return statements) factors out all statements, and Ruby comes close to doing so, and yet they're really not more Lisp-like than Python in any meaningful way. > Of course this subject is definitely a very subjective one which relays on > agreeing on the general meaning of the above sentences. OR... YMMV. From ncoghlan at gmail.com Sat Jan 24 07:46:18 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 24 Jan 2015 16:46:18 +1000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <54C280FA.2000007@stoneleaf.us> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <20150123163618.3465f819@fsol> <54C280FA.2000007@stoneleaf.us> Message-ID: <CADiSq7e_DoVSkZnuMHjQEBgEV=nTyuSvzUhfBSU2Daj4UCL8JA@mail.gmail.com> On 24 January 2015 at 03:12, Ethan Furman <ethan at stoneleaf.us> wrote: > On 01/23/2015 07:36 AM, Antoine Pitrou wrote: >> On Thu, 22 Jan 2015 16:40:14 -0800 Chris Barker wrote: >>> >>> Expected Uses >>> ============= >>> >>> The primary expected use case is various forms of testing -- "are the >>> results computed near what I expect as a result?" This sort of test >>> may or may not be part of a formal unit testing suite. >> >> I don't think the proposal fits the bill. For testing you want a >> function that is both 1) quite rigorous (i.e. checks equality within a >> defined number of ulps) 2) handles all special cases in a useful way >> (i.e. zeros, including distinguishing between positive and negative >> zeros, infinities, NaNs etc.). > > I disagree -- this function is not meant for mathematicians, but for the non-maths person who needs something that > works. Will there be situations where it doesn't work? Certainly. Will they be surprising? Possibly. Note that the key requirement here should be "provide a binary float comparison function that is significantly less wrong than the current 'a == b'". "a == b" is the competition here, not the more correct versions available in other libraries. As far as semantics go, I would expect the new function to be a near drop-in replacement for https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertAlmostEqual in a testing context. The reason I view the proposal in the PEP as problematic is because it is approaching the problem *like a scientist*, rather than as someone who last studied math in high school. The unittest module definition relies on a very simple set of assumptions: 1. The user understands how arithmetic subtraction works 2. The user understands how decimal rounding works 3. The user understands how absolute deltas work This is a "good enough" answer that handles a wide variety of real world use cases, and is very easy to understand. Most importantly, it provides a hint that when working with floating point numbers, "==" is likely to cause you grief. This simple definition *isn't* really good enough for statistical or scientific use cases, but in those cases you should be using a statistical or scientific computation library with a more sophisticated definition of near equality. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Jan 24 08:27:57 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 24 Jan 2015 17:27:57 +1000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> Message-ID: <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> On 24 January 2015 at 07:30, Nathaniel Smith <njs at pobox.com> wrote: > On Fri, Jan 23, 2015 at 12:40 AM, Chris Barker <chris.barker at noaa.gov> wrote: >> Existing Implementations >> ------------------------ >> >> The standard library includes the >> ``unittest.TestCase.assertAlmostEqual`` method, but it: >> >> * Is buried in the unittest.TestCase class >> >> * Is an assertion, so you can't use it as a general test (easily) >> >> * Uses number of decimal digits or an absolute delta, which are >> particular use cases that don't provide a general relative error. > > I might phrase this a bit more strongly -- assertAlmostEqual is > confusing and broken-by-default for common cases like comparing two > small values, or comparing two large values. I would personally find the PEP more persuasive if it was framed in terms of providing an improved definition of assertAlmostEqual that better handles the limitations of binary floating point dynamic ranges. The fact that unittest.assertAlmostEqual is in the standard library implies that any improvement to it must also be in the standard library, and moving the definition of near equality that unittest uses out to the math module so it is reusable in other contexts makes sense to me, especially if it means being able to share the definition between unittest and the statistics module. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ron3200 at gmail.com Sat Jan 24 08:53:34 2015 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 24 Jan 2015 01:53:34 -0600 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <1006605403.316470.1422079613279.JavaMail.yahoo@mail.yahoo.com> References: <m9ugkk$7hq$1@ger.gmane.org> <1006605403.316470.1422079613279.JavaMail.yahoo@mail.yahoo.com> Message-ID: <m9vj1v$fb7$1@ger.gmane.org> On 01/24/2015 12:06 AM, Andrew Barnert wrote: > On Friday, January 23, 2015 2:06 PM, Ron Adam<ron3200 at gmail.com> > wrote: > >>>>> >>> >>> On 01/23/2015 08:56 AM, Nick Coghlan wrote: >>>>> I really like that "each statement is like a single step in a >>>>> mathematical proof" analogy. >>> >>> I agree. >>> >>> >>> There are some other distinctions. When you consider these, you can >>> see how many language designers come to similar solutions. >>> >>> >>> Expressions evaluate in unique name spaces, while statements >>> generally do not. Consider "a + b"; it is evaluated in a private >>> method after the values a and b are passed to it. > I don't think that's true. Not all the time, which is why I said "generally". :-) > Consider "a[b] = c", which is a statement, > but it's evaluated in a private method after the values a, b, and c are > passed to it. The fact that it's a.__setitem__ rather than a.__add__ > doesn't seem particularly important. I think the key is that __setitem__ > doesn't have a return value--like (nearly) everything in Python that > mutates state--and therefore there's no expression for it. The question > is, what does that buy? Right, and good question, but consider that it's not adding or altering the name space the 'a' object is in. It's a convenience/exception for using objects. If objects were done with closures, then the same modification would be done with an assignment statement. And then the generalisation would be more consistent. But at a cost is other ways. >>> Statements are used to mutate the current name space, while >>> expressions generally do not. > This is closer. I think, more generally, statements are used to mutate > the important state that the local function is all about mutating. And > the thing that gets mutated is almost always the leftmost thing. That > definitely helps in scannability. I'm not quite sure I follow that. I think what you are calling the important state, I think of as the shared state. A value that will be used multiple times in the same frame. Binding a name to a value mutates the names pace, but it does not mutate the name. It's still the same name. Again this can be view multiple way if you consider how it actually works. Some languages use a stack to implement a name. And in a new frame, a new bound value would get pushed on the name stack. I'm not completely sure that python doesn't do something like that in some places to speed thing up. But I don't think so. >>> Statements can alter control flow, while expressions generally do >>> not. > Sure, but I think this part only really helps if control flow is somehow > visible. In Python, it is, because control flow almost always means > compound statements, which means indentation, and very little else means > indentation. You are referring to the visual aspect, while I'm referring to what a statement does. Same thing. ;-) >>> Having a clear distinction between expressions and statements makes >>> reading and understanding code much easier. > Definitely. That's the part I think is key, but am struggling to > explain. > > I think Guido offers a great analogy in mathematical proofs, but the > question is to find the actual commonality behind the analogy. Form follows function... Just one way to look at it. Sometimes what something does can come from the shape it has too. > After some more thought on this, I think what it comes down to is that > (idiomatically-written) Python lets you skim the control flow (because > all non-trivial control flow, and very little else, is expressed in > terms of indentation) and the state transitions (because each statement > generally mutates at most one thing, and it's the leftmost thing), so > you can quickly find the part of the code you actually need to read > carefully, instead of having to read the whole thing. > > I've written this idea up in a bit more detail here: > > http://stupidpythonideas.blogspot.com/2015/01/statements-and-expressions.html Very interesting.. and thanks for the mention. ;-) One of the things that makes a difference is to be able to hold a simplified model in your mind while you are working on it. The separation of statements and expressions definitely helps me with that. If I can easily keep the name space in my mind... or at least the part of it that correspond with the block of code I'm looking at, It really helps me to visualise things in a meaningful way and see what effect the statements will have on the name space. When statements and expressions don't represent what they do in an obvious way, then all bets are off. It becomes a mental stumbling block. Much of this became clear to me when I wrote a simplified script language that adds statements to a mini scheme like language. I did need to use braces for blocks. Even though I didn't need to use visual indentation, I still preferred formatting the programs in a similar style to python. In this mini language, the separation of statements and expressions is even more pronounced because all expressions are s-expressions, and all statements are not expressions. And going one bit further, I used dictionaries for names spaces, like python, but used lists for statement blocks. So statement blocks have order, but name spaces don't. >>> I believe Python follows most of these conventions in most places, >>> and when it doesn't, it's usually for a practical reason that are >>> fairly obvious. >>> >>> For example, an "or" expression is a bit of both. >>> >>> Another example of how python chooses a practical alternative is we >>> can nest expressions instead of using "call" statements on separate >>> lines and a stack to hold the augments and return values. That is >>> what python does in the byte code so we don't have to do it in >>> explicit statements. >>> >>> If you factor out all expressions you get byte code. Or if you >>> factor out all statements you get something like lisp. > I don't think either part of that is true. It wasn't meant to be taken absolutely literally. Which is why I said... "something like". There is quite a bit of wiggle room when it comes to how different people think about things, and what words we use to describe them. Some times the hardest part of a discussion is getting the same mental picture in more than one person. :-) > Bytecode is full of things > that are expressions?even your paradigm case of an operator expression > is handled by an opcode. I see opcodes as being similar to keywords that are used in a statement form. And I view the stack as being a temporary value space. Of course, I know certain combinations of several opcodes together may correspond to a particular python expression, but individually, each byecode along with it's following few values, are byte code statements to me. And even if a single bytecode was the equivalent of a python expression, my point was you can use statements replace statements with expressions. Byte code is a good example of that. But you still have functions calls.. You just can't nest them in the same way you do with python function calls, you must push them on the stack and use CALL "statements" to execute it, and then use another statement to store the value that gets put on the top of the stack. (or push another function on the stack that will use that value...) > And conversely, CoffeeScript (if you avoid > break/continue/return statements) factors out all statements, and Ruby > comes close to doing so, and yet they're really not more Lisp-like than > Python in any meaningful way. See the next paragraph... ;-) >>> Of course this subject is definitely a very subjective one which >>> relays on agreeing on the general meaning of the above sentences. >>> OR... YMMV. From ron3200 at gmail.com Sat Jan 24 10:11:40 2015 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 24 Jan 2015 03:11:40 -0600 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <780443585.320289.1422075752374.JavaMail.yahoo@mail.yahoo.com> References: <CALGmxE+GXv_CNUHxQXwavXfz97Niji8CPM4gjqA88_x0GZkoOw@mail.gmail.com> <780443585.320289.1422075752374.JavaMail.yahoo@mail.yahoo.com> Message-ID: <m9vnkd$epo$1@ger.gmane.org> On 01/23/2015 11:02 PM, Andrew Barnert wrote: >>> I guess the key question is if someone would want both an relative >>> tolerance and an absolute tolerance, aside from the zero issue. > Which already raises whether they'd want to min, max, average, or sum > the two. And frankly I have no idea. Today I experimented with implementing is_close by using a parabola equation. y = a(x-h)**2 + k Note: The close area is outside the curve of the parabola. The distance between the point u and v, correspond to the y value, and the x value corresponds to the relative distance from the vertex. def is_parabola_close(u, v, rtol, atol=0): if u == v: return True if u * v < 0: return False x = (u + v) * .5 y = (1.0/x*rtol) * x**2 + atol return abs(u - v) <= y This line: y = (1.0/x*rtol) * x**2 + atol Reduces to: y = rtol * x + atol Which looks familiar. LOL It turns out the relative distance from the vertex means the x distance corresponds to the focus, and the y distance matches the width, for all values of x and y. I thought this was interesting even though it didn't give the result I visualised. I'm going to add a "size" keyword to the function to make the vertex of the parabola independent from the distance of the two points. ;-) I'm not sure it helps the PEP much though. Cheers, Ron From abarnert at yahoo.com Sat Jan 24 12:06:52 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 24 Jan 2015 03:06:52 -0800 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <m9vj1v$fb7$1@ger.gmane.org> References: <m9ugkk$7hq$1@ger.gmane.org> <1006605403.316470.1422079613279.JavaMail.yahoo@mail.yahoo.com> <m9vj1v$fb7$1@ger.gmane.org> Message-ID: <A5B9D58F-992B-4B88-96C7-2F5C786D276F@yahoo.com> On Jan 23, 2015, at 23:53, Ron Adam <ron3200 at gmail.com> wrote: I fear we're probably getting pretty far off-topic for python-ideas, but I'm going to respond anyway. > On 01/24/2015 12:06 AM, Andrew Barnert wrote: >> On Friday, January 23, 2015 2:06 PM, Ron Adam<ron3200 at gmail.com> >> wrote: >> >>>> >>>> On 01/23/2015 08:56 AM, Nick Coghlan wrote: > >>>>>> I really like that "each statement is like a single step in a >>>>>> mathematical proof" analogy. >>>> >>>> I agree. >>>> >>>> >>>> There are some other distinctions. When you consider these, you can >>>> see how many language designers come to similar solutions. >>>> >>>> >>>> Expressions evaluate in unique name spaces, while statements >>>> generally do not. Consider "a + b"; it is evaluated in a private >>>> method after the values a and b are passed to it. > >> I don't think that's true. > > Not all the time, which is why I said "generally". :-) > >> Consider "a[b] = c", which is a statement, >> but it's evaluated in a private method after the values a, b, and c are >> passed to it. The fact that it's a.__setitem__ rather than a.__add__ >> doesn't seem particularly important. I think the key is that __setitem__ >> doesn't have a return value--like (nearly) everything in Python that >> mutates state--and therefore there's no expression for it. The question >> is, what does that buy? > > Right, and good question, but consider that it's not adding or altering the name space the 'a' object is in. It's a convenience/exception for using objects. I think that's the point. In theory, objects are just syntactic sugar for closures; in practice, objects are an intuitively useful way to represent mutable state. And the fact that things that mutate an object are statements (whether directly, because they're __setattr__ type calls, or indirectly, because they're expressions that return None and can therefore only be used in expression statements) is important in making Python readable in practice. > If objects were done with closures, then the same modification would be done with an assignment statement. And then the generalisation would be more consistent. But at a cost is other ways. > >>>> Statements are used to mutate the current name space, while >>>> expressions generally do not. >> This is closer. I think, more generally, statements are used to mutate >> the important state that the local function is all about mutating. And >> the thing that gets mutated is almost always the leftmost thing. That >> definitely helps in scannability. > > I'm not quite sure I follow that. I think what you are calling the important state, I think of as the shared state. A value that will be used multiple times in the same frame. By "important state", I just mean whatever state the reader is likely to care about. And such state is almost always modified by a statement with some readable identification on the left side--whether it's a global/closure/local assignment, an attribute assignment, an element assignment, and augmented version of any of the above, a method call that doesn't return self (and will therefore only be used in an expression statement). So each statement means (at most) one state transition. And it's almost always the leftmost thing that's affected. And I think that aids readability. > Binding a name to a value mutates the names pace, but it does not mutate the name. It's still the same name. > > Again this can be view multiple way if you consider how it actually works. Some languages use a stack to implement a name. And in a new frame, a new bound value would get pushed on the name stack. I'm not completely sure that python doesn't do something like that in some places to speed thing up. But I don't think so. Sure, pure non-mutating languages can use a stack to implement bindings. But in a mutating language, that doesn't work if you have closures. CTM explains what you get out of mutable state (and what it costs) very nicely. Of course in practice any Python implementation has to be able to detect whether a given scope might have closures referring to it, so it _could_ switch to rebinding using a bindinmg stack, but in practice any Python implementation is likely to convert local accesses to offsets as CPython does, which is a much better optimization (at least for a language where rebinding is idiomatically common) that precludes that option. >>>> Statements can alter control flow, while expressions generally do >>>> not. >> Sure, but I think this part only really helps if control flow is somehow >> visible. In Python, it is, because control flow almost always means >> compound statements, which means indentation, and very little else means >> indentation. > > You are referring to the visual aspect, while I'm referring to what a statement does. Same thing. ;-) Well, I'm highlighting the visual aspect because I think that's important to Python's readability, and to why statements contribute to that readability. If statements and expressions had similar indentation rules (as in CoffeeScript), I don't think Python would get the same benefit from having statements. >>>> Having a clear distinction between expressions and statements makes >>>> reading and understanding code much easier. >> Definitely. That's the part I think is key, but am struggling to >> explain. >> >> I think Guido offers a great analogy in mathematical proofs, but the >> question is to find the actual commonality behind the analogy. > > Form follows function... Just one way to look at it. Sometimes what something does can come from the shape it has too. Yes. But sometimes the shape is limiting, rather than expanding--and yet that limitation itself can be used to add meaning. (See Guido's point about code that fits in a window/screen/page.) So function partly follows form. And mathematical proofs are a great example. The fact that there are a limited number of ways you're allowed to get from the previous statements to the next one makes each step more readable. >> After some more thought on this, I think what it comes down to is that >> (idiomatically-written) Python lets you skim the control flow (because >> all non-trivial control flow, and very little else, is expressed in >> terms of indentation) and the state transitions (because each statement >> generally mutates at most one thing, and it's the leftmost thing), so >> you can quickly find the part of the code you actually need to read >> carefully, instead of having to read the whole thing. >> >> I've written this idea up in a bit more detail here: >> >> http://stupidpythonideas.blogspot.com/2015/01/statements-and-expressions.html > > Very interesting.. and thanks for the mention. ;-) Sure; as I said, your last paragraph (well, the last one I quoted) puts the whole thing I'm trying to answer much more clearly than I've been able to. > One of the things that makes a difference is to be able to hold a simplified model in your mind while you are working on it. Yes! That's something else that was on the tip of my tongue that I couldn't explain clearly. Being able to hold enough of the syntax in your head to parse code subconsciously (which CoffeeScript lacks, as Guido pointed out) is part of it, but you're right, the big deal is being able to hold the entire model in your head. And at a different level, that's what makes scanability of flow control and state changes so important, which I couldn't explain before. That's the model of an imperative-style function that you need to be able to internalize to understand the function holistically. Thanks. > The separation of statements and expressions definitely helps me with that. If I can easily keep the name space in my mind... or at least the part of it that correspond with the block of code I'm looking at, It really helps me to visualise things in a meaningful way and see what effect the statements will have on the name space. Exactly. Which is why the "one obvious mutation per line (usually)" property is so important. But I think describing it purely in terms of the local namespace hides the fact that it applies just as well to OO-style code (where most mutation is to the namespace of self or one of the other parameters) as to traditional structured imperative code. > When statements and expressions don't represent what they do in an obvious way, then all bets are off. It becomes a mental stumbling block. > Much of this became clear to me when I wrote a simplified script language that adds statements to a mini scheme like language. I did need to use braces for blocks. Even though I didn't need to use visual indentation, I still preferred formatting the programs in a similar style to python. In this mini language, the separation of statements and expressions is even more pronounced because all expressions are s-expressions, and all statements are not expressions. > > And going one bit further, I used dictionaries for names spaces, like python, but used lists for statement blocks. So statement blocks have order, but name spaces don't. > > >>>> I believe Python follows most of these conventions in most places, >>>> and when it doesn't, it's usually for a practical reason that are >>>> fairly obvious. >>>> >>>> For example, an "or" expression is a bit of both. >>>> >>>> Another example of how python chooses a practical alternative is we >>>> can nest expressions instead of using "call" statements on separate >>>> lines and a stack to hold the augments and return values. That is >>>> what python does in the byte code so we don't have to do it in >>>> explicit statements. >>>> >>>> If you factor out all expressions you get byte code. Or if you >>>> factor out all statements you get something like lisp. >> I don't think either part of that is true. > > It wasn't meant to be taken absolutely literally. Which is why I said... "something like". OK, but I don't think it's figuratively true in any useful sense either. Languages like Ruby prove that removing statements doesn't have to mean something like Lisp. We aren't stuck with the models of the 60s. So the question of what you give up by factoring out all statements turns out to be more complicated (and more interesting) than it was in those models. > There is quite a bit of wiggle room when it comes to how different people think about things, and what words we use to describe them. Some times the hardest part of a discussion is getting the same mental picture in more than one person. :-) > >> Bytecode is full of things >> that are expressions?even your paradigm case of an operator expression >> is handled by an opcode. > > I see opcodes as being similar to keywords that are used in a statement form. And I view the stack as being a temporary value space. Of course, I know certain combinations of several opcodes together may correspond to a particular python expression, but individually, each byecode along with it's following few values, are byte code statements to me. > > And even if a single bytecode was the equivalent of a python expression, my point was you can use statements replace statements with expressions. I don't know what you meant here. Maybe that a stack machine language has a fixed, non-extensible set of expressions, but its set of statements can be effectively arbitrarily extended with jsr/ret to other bytecode? If so, I'll buy that, but I'm not sure how it's relevant. I don't know of any examples of readable expression-free code that compare to such examples of readable statement-free code as, say, anything written in OCaml, or any Ruby code that sticks to single-return style. Eliminating expressions is clearly a non-starter for a readable language; eliminating statements is actually arguable--and the whole point is to find the arguments against doing so. > Byte code is a good example of that. But you still have functions calls.. You just can't nest them in the same way you do with python function calls, you must push them on the stack and use CALL "statements" to execute it, and then use another statement to store the value that gets put on the top of the stack. (or push another function on the stack that will use that value...) > > >> And conversely, CoffeeScript (if you avoid >> break/continue/return statements) factors out all statements, and Ruby >> comes close to doing so, and yet they're really not more Lisp-like than >> Python in any meaningful way. > > See the next paragraph... ;-) > >>>> Of course this subject is definitely a very subjective one which >>>> relays on agreeing on the general meaning of the above sentences. >>>> OR... YMMV. Sure, but I'm not sure I understand the meaning you're going for. And, more importantly, I think there is an objective sense in which Python uses statements to gain subjective readability, and that objective sense is something we can question and try to answer. From guido at python.org Sat Jan 24 18:33:08 2015 From: guido at python.org (Guido van Rossum) Date: Sat, 24 Jan 2015 09:33:08 -0800 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <A5B9D58F-992B-4B88-96C7-2F5C786D276F@yahoo.com> References: <m9ugkk$7hq$1@ger.gmane.org> <1006605403.316470.1422079613279.JavaMail.yahoo@mail.yahoo.com> <m9vj1v$fb7$1@ger.gmane.org> <A5B9D58F-992B-4B88-96C7-2F5C786D276F@yahoo.com> Message-ID: <CAP7+vJKadAY_RUoLmxJ9pghDq40PO-PVgyM7p8gSxO0kXz-mcg@mail.gmail.com> Can you guys get a room? On Sat, Jan 24, 2015 at 3:06 AM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > On Jan 23, 2015, at 23:53, Ron Adam <ron3200 at gmail.com> wrote: > > I fear we're probably getting pretty far off-topic for python-ideas, but > I'm going to respond anyway. > > > On 01/24/2015 12:06 AM, Andrew Barnert wrote: > >> On Friday, January 23, 2015 2:06 PM, Ron Adam<ron3200 at gmail.com> > >> wrote: > >> > >>>> > >>>> On 01/23/2015 08:56 AM, Nick Coghlan wrote: > > > >>>>>> I really like that "each statement is like a single step in a > >>>>>> mathematical proof" analogy. > >>>> > >>>> I agree. > >>>> > >>>> > >>>> There are some other distinctions. When you consider these, you can > >>>> see how many language designers come to similar solutions. > >>>> > >>>> > >>>> Expressions evaluate in unique name spaces, while statements > >>>> generally do not. Consider "a + b"; it is evaluated in a private > >>>> method after the values a and b are passed to it. > > > >> I don't think that's true. > > > > Not all the time, which is why I said "generally". :-) > > > >> Consider "a[b] = c", which is a statement, > >> but it's evaluated in a private method after the values a, b, and c are > >> passed to it. The fact that it's a.__setitem__ rather than a.__add__ > >> doesn't seem particularly important. I think the key is that __setitem__ > >> doesn't have a return value--like (nearly) everything in Python that > >> mutates state--and therefore there's no expression for it. The question > >> is, what does that buy? > > > > Right, and good question, but consider that it's not adding or altering > the name space the 'a' object is in. It's a convenience/exception for > using objects. > > I think that's the point. In theory, objects are just syntactic sugar for > closures; in practice, objects are an intuitively useful way to represent > mutable state. And the fact that things that mutate an object are > statements (whether directly, because they're __setattr__ type calls, or > indirectly, because they're expressions that return None and can therefore > only be used in expression statements) is important in making Python > readable in practice. > > > If objects were done with closures, then the same modification would be > done with an assignment statement. And then the generalisation would be > more consistent. But at a cost is other ways. > > > >>>> Statements are used to mutate the current name space, while > >>>> expressions generally do not. > >> This is closer. I think, more generally, statements are used to mutate > >> the important state that the local function is all about mutating. And > >> the thing that gets mutated is almost always the leftmost thing. That > >> definitely helps in scannability. > > > > I'm not quite sure I follow that. I think what you are calling the > important state, I think of as the shared state. A value that will be used > multiple times in the same frame. > > By "important state", I just mean whatever state the reader is likely to > care about. And such state is almost always modified by a statement with > some readable identification on the left side--whether it's a > global/closure/local assignment, an attribute assignment, an element > assignment, and augmented version of any of the above, a method call that > doesn't return self (and will therefore only be used in an expression > statement). So each statement means (at most) one state transition. And > it's almost always the leftmost thing that's affected. And I think that > aids readability. > > > Binding a name to a value mutates the names pace, but it does not mutate > the name. It's still the same name. > > > > Again this can be view multiple way if you consider how it actually > works. Some languages use a stack to implement a name. And in a new frame, > a new bound value would get pushed on the name stack. I'm not completely > sure that python doesn't do something like that in some places to speed > thing up. But I don't think so. > > Sure, pure non-mutating languages can use a stack to implement bindings. > But in a mutating language, that doesn't work if you have closures. CTM > explains what you get out of mutable state (and what it costs) very nicely. > Of course in practice any Python implementation has to be able to detect > whether a given scope might have closures referring to it, so it _could_ > switch to rebinding using a bindinmg stack, but in practice any Python > implementation is likely to convert local accesses to offsets as CPython > does, which is a much better optimization (at least for a language where > rebinding is idiomatically common) that precludes that option. > > >>>> Statements can alter control flow, while expressions generally do > >>>> not. > >> Sure, but I think this part only really helps if control flow is somehow > >> visible. In Python, it is, because control flow almost always means > >> compound statements, which means indentation, and very little else means > >> indentation. > > > > You are referring to the visual aspect, while I'm referring to what a > statement does. Same thing. ;-) > > Well, I'm highlighting the visual aspect because I think that's important > to Python's readability, and to why statements contribute to that > readability. If statements and expressions had similar indentation rules > (as in CoffeeScript), I don't think Python would get the same benefit from > having statements. > > >>>> Having a clear distinction between expressions and statements makes > >>>> reading and understanding code much easier. > >> Definitely. That's the part I think is key, but am struggling to > >> explain. > >> > >> I think Guido offers a great analogy in mathematical proofs, but the > >> question is to find the actual commonality behind the analogy. > > > > Form follows function... Just one way to look at it. Sometimes what > something does can come from the shape it has too. > > Yes. But sometimes the shape is limiting, rather than expanding--and yet > that limitation itself can be used to add meaning. (See Guido's point about > code that fits in a window/screen/page.) > So function partly follows form. And mathematical proofs are a great > example. The fact that there are a limited number of ways you're allowed to > get from the previous statements to the next one makes each step more > readable. > > >> After some more thought on this, I think what it comes down to is that > >> (idiomatically-written) Python lets you skim the control flow (because > >> all non-trivial control flow, and very little else, is expressed in > >> terms of indentation) and the state transitions (because each statement > >> generally mutates at most one thing, and it's the leftmost thing), so > >> you can quickly find the part of the code you actually need to read > >> carefully, instead of having to read the whole thing. > >> > >> I've written this idea up in a bit more detail here: > >> > >> > http://stupidpythonideas.blogspot.com/2015/01/statements-and-expressions.html > > > > Very interesting.. and thanks for the mention. ;-) > > Sure; as I said, your last paragraph (well, the last one I quoted) puts > the whole thing I'm trying to answer much more clearly than I've been able > to. > > > One of the things that makes a difference is to be able to hold a > simplified model in your mind while you are working on it. > > Yes! That's something else that was on the tip of my tongue that I > couldn't explain clearly. Being able to hold enough of the syntax in your > head to parse code subconsciously (which CoffeeScript lacks, as Guido > pointed out) is part of it, but you're right, the big deal is being able to > hold the entire model in your head. > > And at a different level, that's what makes scanability of flow control > and state changes so important, which I couldn't explain before. That's the > model of an imperative-style function that you need to be able to > internalize to understand the function holistically. > > Thanks. > > > The separation of statements and expressions definitely helps me with > that. If I can easily keep the name space in my mind... or at least the > part of it that correspond with the block of code I'm looking at, It really > helps me to visualise things in a meaningful way and see what effect the > statements will have on the name space. > > Exactly. Which is why the "one obvious mutation per line (usually)" > property is so important. > > But I think describing it purely in terms of the local namespace hides the > fact that it applies just as well to OO-style code (where most mutation is > to the namespace of self or one of the other parameters) as to traditional > structured imperative code. > > > When statements and expressions don't represent what they do in an > obvious way, then all bets are off. It becomes a mental stumbling block. > > > Much of this became clear to me when I wrote a simplified script > language that adds statements to a mini scheme like language. I did need > to use braces for blocks. Even though I didn't need to use visual > indentation, I still preferred formatting the programs in a similar style > to python. In this mini language, the separation of statements and > expressions is even more pronounced because all expressions are > s-expressions, and all statements are not expressions. > > > > And going one bit further, I used dictionaries for names spaces, like > python, but used lists for statement blocks. So statement blocks have > order, but name spaces don't. > > > > > >>>> I believe Python follows most of these conventions in most places, > >>>> and when it doesn't, it's usually for a practical reason that are > >>>> fairly obvious. > >>>> > >>>> For example, an "or" expression is a bit of both. > >>>> > >>>> Another example of how python chooses a practical alternative is we > >>>> can nest expressions instead of using "call" statements on separate > >>>> lines and a stack to hold the augments and return values. That is > >>>> what python does in the byte code so we don't have to do it in > >>>> explicit statements. > >>>> > >>>> If you factor out all expressions you get byte code. Or if you > >>>> factor out all statements you get something like lisp. > >> I don't think either part of that is true. > > > > It wasn't meant to be taken absolutely literally. Which is why I > said... "something like". > > OK, but I don't think it's figuratively true in any useful sense either. > Languages like Ruby prove that removing statements doesn't have to mean > something like Lisp. We aren't stuck with the models of the 60s. So the > question of what you give up by factoring out all statements turns out to > be more complicated (and more interesting) than it was in those models. > > > There is quite a bit of wiggle room when it comes to how different > people think about things, and what words we use to describe them. Some > times the hardest part of a discussion is getting the same mental picture > in more than one person. :-) > > > >> Bytecode is full of things > >> that are expressions?even your paradigm case of an operator expression > >> is handled by an opcode. > > > > I see opcodes as being similar to keywords that are used in a statement > form. And I view the stack as being a temporary value space. Of course, I > know certain combinations of several opcodes together may correspond to a > particular python expression, but individually, each byecode along with > it's following few values, are byte code statements to me. > > > > And even if a single bytecode was the equivalent of a python expression, > my point was you can use statements replace statements with expressions. > > I don't know what you meant here. Maybe that a stack machine language has > a fixed, non-extensible set of expressions, but its set of statements can > be effectively arbitrarily extended with jsr/ret to other bytecode? If so, > I'll buy that, but I'm not sure how it's relevant. I don't know of any > examples of readable expression-free code that compare to such examples of > readable statement-free code as, say, anything written in OCaml, or any > Ruby code that sticks to single-return style. Eliminating expressions is > clearly a non-starter for a readable language; eliminating statements is > actually arguable--and the whole point is to find the arguments against > doing so. > > > Byte code is a good example of that. But you still have functions > calls.. You just can't nest them in the same way you do with python > function calls, you must push them on the stack and use CALL "statements" > to execute it, and then use another statement to store the value that gets > put on the top of the stack. (or push another function on the stack that > will use that value...) > > > > > >> And conversely, CoffeeScript (if you avoid > >> break/continue/return statements) factors out all statements, and Ruby > >> comes close to doing so, and yet they're really not more Lisp-like than > >> Python in any meaningful way. > > > > See the next paragraph... ;-) > > > >>>> Of course this subject is definitely a very subjective one which > >>>> relays on agreeing on the general meaning of the above sentences. > >>>> OR... YMMV. > > Sure, but I'm not sure I understand the meaning you're going for. > > And, more importantly, I think there is an objective sense in which Python > uses statements to gain subjective readability, and that objective sense is > something we can question and try to answer. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150124/5c7c2850/attachment-0001.html> From ethan at stoneleaf.us Sat Jan 24 20:04:50 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 24 Jan 2015 11:04:50 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <CAP7+vJJqHSGCo_krBtwEhnS_w1NzSRp2mOL5gXZwbQqvJyYQ7A@mail.gmail.com> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> <CAP1=2W59Pxr4j_Cx+Cvd+WQQZ4RznOSrEL9q_G2wZcB=F81=wQ@mail.gmail.com> <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> <CAP7+vJLmN_YwmHOAXzqm5Dga1aQY8ygu6Z86WFaDCM-JLTh6aQ@mail.gmail.com> <m9sjm6$v3u$1@ger.gmane.org> <CAP7+vJ+H7gz4Qeqo0fZ6xW3Hvtf87PDmOmdymyC2pydKKPYaQw@mail.gmail.com> <87oapou8vg.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJqHSGCo_krBtwEhnS_w1NzSRp2mOL5gXZwbQqvJyYQ7A@mail.gmail.com> Message-ID: <54C3ECD2.4070808@stoneleaf.us> On 01/23/2015 08:40 PM, Guido van Rossum wrote: > > This reminds me -- if you have a "# type: OFF" comment at the top of a module, I presume the type checker still reads > the rest of the file (at the very least looking for "# type: ON"), and it may still record the presence of things > defined there, for the benefit of checking that things imported *from* that module at least exist. The way I envisioned > "# type: OFF" was that the type checker will ignore annotations, essentially assuming that annotations it sees don't > exist. But when it sees the following in file a.py: > > # type: OFF > def foo(a): > return a+1 > > and in file b.py there is this: > > from a import foo > foo(1, 2, 3) > foo('hello') > > It might still complain about the foo(1, 2, 3) call in b.py, because it still knows that a.foo() takes exactly one argument. +1 I think "# type: OFF" should just ignore any existing annotations, but still track things such as function parameters -- maybe by assigning Any to all paramaters it finds? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150124/a03c6be6/attachment.sig> From guido at python.org Sat Jan 24 20:40:14 2015 From: guido at python.org (Guido van Rossum) Date: Sat, 24 Jan 2015 11:40:14 -0800 Subject: [Python-ideas] PEP 484 (Type Hints) -- first draft round In-Reply-To: <54C3ECD2.4070808@stoneleaf.us> References: <CAP7+vJ+_gsfBwg1HaJXaFHM9p1smtmATtmjhFVV4WbWEQeEGGQ@mail.gmail.com> <CAP7+vJKZ5de6U6TT5LjzU0f5fybt+Xmv14z_c7t=aCKFOEPXag@mail.gmail.com> <EF7E3A91-CAF4-4C55-B81E-D0F41D2F3A3C@gmail.com> <CAPTjJmp7ZDf9JL6EQnNTf+buSYZKR9FL1Oa2-J-5bwtsf6UqYg@mail.gmail.com> <CABmzr0joeO3iaxvz-5k9zOjoeJD8CDzZafexSLsKuvs1NwtFJQ@mail.gmail.com> <CAPTjJmqQQ+-9UVcDShXCRuh+RkJupVXKgwpMKfifzNwNvqAxdg@mail.gmail.com> <CABmzr0hectyY6FsbjaccqVoJw_2jyGcBYduHKOL9MqjGAdZAyw@mail.gmail.com> <CAPTjJmoWYVKGXpWPt=+0QYSHsUarw+m2cUaMGHiN=2eFNRw0gQ@mail.gmail.com> <CAP1=2W59Pxr4j_Cx+Cvd+WQQZ4RznOSrEL9q_G2wZcB=F81=wQ@mail.gmail.com> <CABmzr0iHbaFvKP3MhFfj-zsqgK3oNV78OT9LBN52dkPS8h6P-A@mail.gmail.com> <CAP7+vJLmN_YwmHOAXzqm5Dga1aQY8ygu6Z86WFaDCM-JLTh6aQ@mail.gmail.com> <m9sjm6$v3u$1@ger.gmane.org> <CAP7+vJ+H7gz4Qeqo0fZ6xW3Hvtf87PDmOmdymyC2pydKKPYaQw@mail.gmail.com> <87oapou8vg.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJqHSGCo_krBtwEhnS_w1NzSRp2mOL5gXZwbQqvJyYQ7A@mail.gmail.com> <54C3ECD2.4070808@stoneleaf.us> Message-ID: <CAP7+vJL0UNP20ta6m4zCqLMLDmrxDa=OcCq_JECwO69=GXn8oQ@mail.gmail.com> On Sat, Jan 24, 2015 at 11:04 AM, Ethan Furman <ethan at stoneleaf.us> wrote: > I think "# type: OFF" should just ignore any existing annotations, but > still track things such as function parameters > -- maybe by assigning Any to all paramaters it finds? > Having slept on it, that's probably best for simple, undecorated functions and methods in vanilla classes. For decorated functions and methods it's more complicated though -- if the type checker doesn't understand the decorator it's not safe to assume that the signature is unchanged. (E.g. @staticmethod, @classmethod, @property all modify the signature -- the type checker should know about those, but many custom decorators do this too.) Maybe for decorated functions and methods the best it can do is to record the existence of the function name (since the decorator syntax guarantees that), giving it the type 'Any'. It may also have to do this for classes if it can't figure out whether there is a custom metaclass or what a class decorator does. But we're now out of the realm of how to disable type checking a whole module, and I don't worry so much about this -- to some extent it's a matter of quality of implementation how well the type checker copes with such modules and at what level it must assume the type 'Any'. For quickly disabling a whole module or package, we still have a choice. Every type checker can have its own configuration mechanism that allows selectively telling the checker to stay away from certain module, and this should be good enough for most users; the PEP shouldn't have to specify anything here except hint at the usefulness of such a feature. The question remains what package owners can do if their package has code that causes the type checker to be too noisy. They can use # type: OFF comments (with the above caveat that there might still be noise at call sites outside the package), or they can include stub modules (which require some thought to get right -- even the pinfer tool often guesses wrong). Do we want to support a "type checker shut up!" file can be present in the package root? This may be handy to avoid noise from e.g. mypy, but I'd imagine that e.g. PyCharm might want to ignore this and still look for useful things to aid its completion feature. Thinking about this more, I think we needn't do anything about this now, and we can see how things work out once 3.5 is in active use. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150124/ab487710/attachment.html> From chris.barker at noaa.gov Sat Jan 24 20:59:46 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Sat, 24 Jan 2015 11:59:46 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <8835515035530371300@unknownmsgid> <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> Message-ID: <CALGmxEJNim=QfVxxVE1JHQ03Z=PrJAub1EWQx2TBOq7xtt-MFg@mail.gmail.com> >> One option would be to add a zero_tol argument, which is an absolute > >> tolerance that is only applied if expected == 0. > OK -- now I know what the problem is here -- I thought I"d explored it already. If you have a tolerance that you use only when expected is zero (or when either is...) then you have teh odd reslut that a samll number will be "close" to zero, but NOT close to a smaller number. I implemented this on a branch in github: https://github.com/PythonCHB/close_pep/tree/zero_tol And you get the odd result: In [9]: is_close_to(1e-9, 0.0) Out[9]: True fine -- the default zero_tol is 1e-8 In [10]: is_close_to(1e-9, 1e-12) Out[10]: False but huh??? 1e-9 is close to zero, but not close to 1e-12???? This is why I dropped the idea before. I'm back to the key point -- relative error compared to zero is not defined -- you need to set an absolute tolerance if you want to test against zero -- and there is no default that makes sense for all (or even most) cases. I'd much rather require people to have to think about what makes sense for their use case than get trapped by a default that's totally inappropriate. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150124/6913f606/attachment.html> From guido at python.org Sat Jan 24 21:21:22 2015 From: guido at python.org (Guido van Rossum) Date: Sat, 24 Jan 2015 12:21:22 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEJNim=QfVxxVE1JHQ03Z=PrJAub1EWQx2TBOq7xtt-MFg@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <8835515035530371300@unknownmsgid> <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> <CALGmxEJNim=QfVxxVE1JHQ03Z=PrJAub1EWQx2TBOq7xtt-MFg@mail.gmail.com> Message-ID: <CAP7+vJJk5t=K+WNkzbKh2yRX4sMz9LRoFHALdhD=LXfrWMR85w@mail.gmail.com> On Sat, Jan 24, 2015 at 11:59 AM, Chris Barker <chris.barker at noaa.gov> wrote: > > I'd much rather require people to have to think about what makes sense for > their use case than get trapped by a default that's totally inappropriate. > And by the time you have thought through your use case you're probably better off just writing abs(x-y) <= eps for some eps that you decide from your use case. The number of messages written to debate this one simple formula make me think it's not so simple after all. So perhaps this is, once again, something that's better off as a recipe? Alternatively, maybe a useful approach to resolving this debate could be to look for places in actual code where people have solved this for their own use case (as in, written by someone else to solve a real problem, not made up as an example). Then if you see a particular pattern occur repeatedly, you might be able to use that evidence to suggest the right helper function that does the whole thing in one call, and by looking at variations you might get a good insight in the needed parameters and defaults. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150124/3b563568/attachment-0001.html> From chris.barker at noaa.gov Sat Jan 24 21:46:30 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Sat, 24 Jan 2015 12:46:30 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAP7+vJJk5t=K+WNkzbKh2yRX4sMz9LRoFHALdhD=LXfrWMR85w@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <8835515035530371300@unknownmsgid> <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> <CALGmxEJNim=QfVxxVE1JHQ03Z=PrJAub1EWQx2TBOq7xtt-MFg@mail.gmail.com> <CAP7+vJJk5t=K+WNkzbKh2yRX4sMz9LRoFHALdhD=LXfrWMR85w@mail.gmail.com> Message-ID: <CALGmxE+ucX=Xm73a=302yquFvz-VNMo_b09GC2J+dc3=FOe73Q@mail.gmail.com> On Sat, Jan 24, 2015 at 12:21 PM, Guido van Rossum <guido at python.org> wrote: > On Sat, Jan 24, 2015 at 11:59 AM, Chris Barker <chris.barker at noaa.gov> > wrote: > >> >> I'd much rather require people to have to think about what makes sense >> for their use case than get trapped by a default that's totally >> inappropriate. >> > > And by the time you have thought through your use case you're probably > better off just writing abs(x-y) <= eps for some eps that you decide from > your use case. > Exactly why I never thought a function to simply do an absolute closeness test wasn't worth it. This is for the case where you want a relative test, but also want something sane for a comparison to zero. If there aren't any zeros in your data, you don't need to think about it. If there are, it will fail, and then you will have to think about it. The number of messages written to debate this one simple formula make me > think it's not so simple after all. > well, we shouldn't underestimate the capacity for bike-shedding. Next I plan to go through full set of messages since I posted the PEP, and tease out the real disagreements/decision points -- I guess I'll see how that pans out. > So perhaps this is, once again, something that's better off as a recipe? > I hope not, but if consensus can't be reached, perhaps so. > Alternatively, maybe a useful approach to resolving this debate could be > to look for places in actual code where people have solved this for their > own use case (as in, written by someone else to solve a real problem, not > made up as an example). > I've got a pile of uses of numpy's all_close (mostly tests) and it would be easy to search for those in other places. Not sure how to find other uses in other people's code though. And, of course, Steven put something similar in the statistics tests. So maybe the idea that this is primarily a new testing utility is the way to go. > > Then if you see a particular pattern occur repeatedly, you might be able > to use that evidence to suggest the right helper function that does the > whole thing in one call, and by looking at variations you might get a good > insight in the needed parameters and defaults. > I have no idea how to find those, but if anyone has suggestions for how to look, or examples in their own code, that would be great. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150124/f6109326/attachment.html> From ncoghlan at gmail.com Sun Jan 25 02:54:14 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 Jan 2015 11:54:14 +1000 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <CAP7+vJKadAY_RUoLmxJ9pghDq40PO-PVgyM7p8gSxO0kXz-mcg@mail.gmail.com> References: <m9ugkk$7hq$1@ger.gmane.org> <1006605403.316470.1422079613279.JavaMail.yahoo@mail.yahoo.com> <m9vj1v$fb7$1@ger.gmane.org> <A5B9D58F-992B-4B88-96C7-2F5C786D276F@yahoo.com> <CAP7+vJKadAY_RUoLmxJ9pghDq40PO-PVgyM7p8gSxO0kXz-mcg@mail.gmail.com> Message-ID: <CADiSq7dh1c5c_GoBkYh6qToqZ4Ko-rhOzt7NdXPLE2GdhoK4XQ@mail.gmail.com> On 25 Jan 2015 03:34, "Guido van Rossum" <guido at python.org> wrote: > > Can you guys get a room? More explicitly - I think we've established we *don't actually know* why the statement/expression distinction improves readability, just that decades of experience suggests it does. Further speculation here is unlikely to bring further clarity, so anyone interested in seeing it pursued to the point of reaching a more definitive conclusion would be well advised to get in touch with folks that actually research programming language readability (e.g. the authors of this paper from a few years ago: http://neverworkintheory.org/2011/10/24/an-empirical-comparison-of-the-accuracy-rates-of-novices-using-the-quorum-perl-and-randomo-programming-languages.html and their more recent follow-up http://neverworkintheory.org/2014/01/29/stefik-siebert-syntax.html) Cheers, Nick. > > On Sat, Jan 24, 2015 at 3:06 AM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: >> >> On Jan 23, 2015, at 23:53, Ron Adam <ron3200 at gmail.com> wrote: >> >> I fear we're probably getting pretty far off-topic for python-ideas, but I'm going to respond anyway. >> >> > On 01/24/2015 12:06 AM, Andrew Barnert wrote: >> >> On Friday, January 23, 2015 2:06 PM, Ron Adam<ron3200 at gmail.com> >> >> wrote: >> >> >> >>>> >> >>>> On 01/23/2015 08:56 AM, Nick Coghlan wrote: >> > >> >>>>>> I really like that "each statement is like a single step in a >> >>>>>> mathematical proof" analogy. >> >>>> >> >>>> I agree. >> >>>> >> >>>> >> >>>> There are some other distinctions. When you consider these, you can >> >>>> see how many language designers come to similar solutions. >> >>>> >> >>>> >> >>>> Expressions evaluate in unique name spaces, while statements >> >>>> generally do not. Consider "a + b"; it is evaluated in a private >> >>>> method after the values a and b are passed to it. >> > >> >> I don't think that's true. >> > >> > Not all the time, which is why I said "generally". :-) >> > >> >> Consider "a[b] = c", which is a statement, >> >> but it's evaluated in a private method after the values a, b, and c are >> >> passed to it. The fact that it's a.__setitem__ rather than a.__add__ >> >> doesn't seem particularly important. I think the key is that __setitem__ >> >> doesn't have a return value--like (nearly) everything in Python that >> >> mutates state--and therefore there's no expression for it. The question >> >> is, what does that buy? >> > >> > Right, and good question, but consider that it's not adding or altering the name space the 'a' object is in. It's a convenience/exception for using objects. >> >> I think that's the point. In theory, objects are just syntactic sugar for closures; in practice, objects are an intuitively useful way to represent mutable state. And the fact that things that mutate an object are statements (whether directly, because they're __setattr__ type calls, or indirectly, because they're expressions that return None and can therefore only be used in expression statements) is important in making Python readable in practice. >> >> > If objects were done with closures, then the same modification would be done with an assignment statement. And then the generalisation would be more consistent. But at a cost is other ways. >> > >> >>>> Statements are used to mutate the current name space, while >> >>>> expressions generally do not. >> >> This is closer. I think, more generally, statements are used to mutate >> >> the important state that the local function is all about mutating. And >> >> the thing that gets mutated is almost always the leftmost thing. That >> >> definitely helps in scannability. >> > >> > I'm not quite sure I follow that. I think what you are calling the important state, I think of as the shared state. A value that will be used multiple times in the same frame. >> >> By "important state", I just mean whatever state the reader is likely to care about. And such state is almost always modified by a statement with some readable identification on the left side--whether it's a global/closure/local assignment, an attribute assignment, an element assignment, and augmented version of any of the above, a method call that doesn't return self (and will therefore only be used in an expression statement). So each statement means (at most) one state transition. And it's almost always the leftmost thing that's affected. And I think that aids readability. >> >> > Binding a name to a value mutates the names pace, but it does not mutate the name. It's still the same name. >> > >> > Again this can be view multiple way if you consider how it actually works. Some languages use a stack to implement a name. And in a new frame, a new bound value would get pushed on the name stack. I'm not completely sure that python doesn't do something like that in some places to speed thing up. But I don't think so. >> >> Sure, pure non-mutating languages can use a stack to implement bindings. But in a mutating language, that doesn't work if you have closures. CTM explains what you get out of mutable state (and what it costs) very nicely. Of course in practice any Python implementation has to be able to detect whether a given scope might have closures referring to it, so it _could_ switch to rebinding using a bindinmg stack, but in practice any Python implementation is likely to convert local accesses to offsets as CPython does, which is a much better optimization (at least for a language where rebinding is idiomatically common) that precludes that option. >> >> >>>> Statements can alter control flow, while expressions generally do >> >>>> not. >> >> Sure, but I think this part only really helps if control flow is somehow >> >> visible. In Python, it is, because control flow almost always means >> >> compound statements, which means indentation, and very little else means >> >> indentation. >> > >> > You are referring to the visual aspect, while I'm referring to what a statement does. Same thing. ;-) >> >> Well, I'm highlighting the visual aspect because I think that's important to Python's readability, and to why statements contribute to that readability. If statements and expressions had similar indentation rules (as in CoffeeScript), I don't think Python would get the same benefit from having statements. >> >> >>>> Having a clear distinction between expressions and statements makes >> >>>> reading and understanding code much easier. >> >> Definitely. That's the part I think is key, but am struggling to >> >> explain. >> >> >> >> I think Guido offers a great analogy in mathematical proofs, but the >> >> question is to find the actual commonality behind the analogy. >> > >> > Form follows function... Just one way to look at it. Sometimes what something does can come from the shape it has too. >> >> Yes. But sometimes the shape is limiting, rather than expanding--and yet that limitation itself can be used to add meaning. (See Guido's point about code that fits in a window/screen/page.) >> So function partly follows form. And mathematical proofs are a great example. The fact that there are a limited number of ways you're allowed to get from the previous statements to the next one makes each step more readable. >> >> >> After some more thought on this, I think what it comes down to is that >> >> (idiomatically-written) Python lets you skim the control flow (because >> >> all non-trivial control flow, and very little else, is expressed in >> >> terms of indentation) and the state transitions (because each statement >> >> generally mutates at most one thing, and it's the leftmost thing), so >> >> you can quickly find the part of the code you actually need to read >> >> carefully, instead of having to read the whole thing. >> >> >> >> I've written this idea up in a bit more detail here: >> >> >> >> http://stupidpythonideas.blogspot.com/2015/01/statements-and-expressions.html >> > >> > Very interesting.. and thanks for the mention. ;-) >> >> Sure; as I said, your last paragraph (well, the last one I quoted) puts the whole thing I'm trying to answer much more clearly than I've been able to. >> >> > One of the things that makes a difference is to be able to hold a simplified model in your mind while you are working on it. >> >> Yes! That's something else that was on the tip of my tongue that I couldn't explain clearly. Being able to hold enough of the syntax in your head to parse code subconsciously (which CoffeeScript lacks, as Guido pointed out) is part of it, but you're right, the big deal is being able to hold the entire model in your head. >> >> And at a different level, that's what makes scanability of flow control and state changes so important, which I couldn't explain before. That's the model of an imperative-style function that you need to be able to internalize to understand the function holistically. >> >> Thanks. >> >> > The separation of statements and expressions definitely helps me with that. If I can easily keep the name space in my mind... or at least the part of it that correspond with the block of code I'm looking at, It really helps me to visualise things in a meaningful way and see what effect the statements will have on the name space. >> >> Exactly. Which is why the "one obvious mutation per line (usually)" property is so important. >> >> But I think describing it purely in terms of the local namespace hides the fact that it applies just as well to OO-style code (where most mutation is to the namespace of self or one of the other parameters) as to traditional structured imperative code. >> >> > When statements and expressions don't represent what they do in an obvious way, then all bets are off. It becomes a mental stumbling block. >> >> > Much of this became clear to me when I wrote a simplified script language that adds statements to a mini scheme like language. I did need to use braces for blocks. Even though I didn't need to use visual indentation, I still preferred formatting the programs in a similar style to python. In this mini language, the separation of statements and expressions is even more pronounced because all expressions are s-expressions, and all statements are not expressions. >> > >> > And going one bit further, I used dictionaries for names spaces, like python, but used lists for statement blocks. So statement blocks have order, but name spaces don't. >> > >> > >> >>>> I believe Python follows most of these conventions in most places, >> >>>> and when it doesn't, it's usually for a practical reason that are >> >>>> fairly obvious. >> >>>> >> >>>> For example, an "or" expression is a bit of both. >> >>>> >> >>>> Another example of how python chooses a practical alternative is we >> >>>> can nest expressions instead of using "call" statements on separate >> >>>> lines and a stack to hold the augments and return values. That is >> >>>> what python does in the byte code so we don't have to do it in >> >>>> explicit statements. >> >>>> >> >>>> If you factor out all expressions you get byte code. Or if you >> >>>> factor out all statements you get something like lisp. >> >> I don't think either part of that is true. >> > >> > It wasn't meant to be taken absolutely literally. Which is why I said... "something like". >> >> OK, but I don't think it's figuratively true in any useful sense either. Languages like Ruby prove that removing statements doesn't have to mean something like Lisp. We aren't stuck with the models of the 60s. So the question of what you give up by factoring out all statements turns out to be more complicated (and more interesting) than it was in those models. >> >> > There is quite a bit of wiggle room when it comes to how different people think about things, and what words we use to describe them. Some times the hardest part of a discussion is getting the same mental picture in more than one person. :-) >> > >> >> Bytecode is full of things >> >> that are expressions?even your paradigm case of an operator expression >> >> is handled by an opcode. >> > >> > I see opcodes as being similar to keywords that are used in a statement form. And I view the stack as being a temporary value space. Of course, I know certain combinations of several opcodes together may correspond to a particular python expression, but individually, each byecode along with it's following few values, are byte code statements to me. >> > >> > And even if a single bytecode was the equivalent of a python expression, my point was you can use statements replace statements with expressions. >> >> I don't know what you meant here. Maybe that a stack machine language has a fixed, non-extensible set of expressions, but its set of statements can be effectively arbitrarily extended with jsr/ret to other bytecode? If so, I'll buy that, but I'm not sure how it's relevant. I don't know of any examples of readable expression-free code that compare to such examples of readable statement-free code as, say, anything written in OCaml, or any Ruby code that sticks to single-return style. Eliminating expressions is clearly a non-starter for a readable language; eliminating statements is actually arguable--and the whole point is to find the arguments against doing so. >> >> > Byte code is a good example of that. But you still have functions calls.. You just can't nest them in the same way you do with python function calls, you must push them on the stack and use CALL "statements" to execute it, and then use another statement to store the value that gets put on the top of the stack. (or push another function on the stack that will use that value...) >> > >> > >> >> And conversely, CoffeeScript (if you avoid >> >> break/continue/return statements) factors out all statements, and Ruby >> >> comes close to doing so, and yet they're really not more Lisp-like than >> >> Python in any meaningful way. >> > >> > See the next paragraph... ;-) >> > >> >>>> Of course this subject is definitely a very subjective one which >> >>>> relays on agreeing on the general meaning of the above sentences. >> >>>> OR... YMMV. >> >> Sure, but I'm not sure I understand the meaning you're going for. >> >> And, more importantly, I think there is an objective sense in which Python uses statements to gain subjective readability, and that objective sense is something we can question and try to answer. >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150125/7e4c067f/attachment-0001.html> From yawar.amin at gmail.com Sun Jan 25 05:23:51 2015 From: yawar.amin at gmail.com (Yawar Amin) Date: Sat, 24 Jan 2015 23:23:51 -0500 Subject: [Python-ideas] Syntax: 'return: ...' expressions In-Reply-To: <CADiSq7dh1c5c_GoBkYh6qToqZ4Ko-rhOzt7NdXPLE2GdhoK4XQ@mail.gmail.com> References: <m9ugkk$7hq$1@ger.gmane.org> <1006605403.316470.1422079613279.JavaMail.yahoo@mail.yahoo.com> <m9vj1v$fb7$1@ger.gmane.org> <A5B9D58F-992B-4B88-96C7-2F5C786D276F@yahoo.com> <CAP7+vJKadAY_RUoLmxJ9pghDq40PO-PVgyM7p8gSxO0kXz-mcg@mail.gmail.com> <CADiSq7dh1c5c_GoBkYh6qToqZ4Ko-rhOzt7NdXPLE2GdhoK4XQ@mail.gmail.com> Message-ID: <54C46FD7.6050305@gmail.com> Hi, On 2015-01-24 20:54, Nick Coghlan wrote: > On 25 Jan 2015 03:34, "Guido van Rossum" <guido at python.org> wrote: > > > > Can you guys get a room? > [...] > definitive conclusion would be well advised to get in touch with folks > that actually research programming language readability (e.g. the > authors of this paper from a few years ago: > http://neverworkintheory.org/2011/10/24/an-empirical-comparison-of-the-accuracy-rates-of-novices-using-the-quorum-perl-and-randomo-programming-languages.html > and their more recent follow-up > http://neverworkintheory.org/2014/01/29/stefik-siebert-syntax.html) And/or comment on Andrew's blog post: > >> > On 01/24/2015 12:06 AM, Andrew Barnert wrote: > [...] > >> >> I've written this idea up in a bit more detail here: > >> >> > >> >> http://stupidpythonideas.blogspot.com/2015/01/statements-and-expressions.html In other words, let's put this thread to bed :-) Regards, Yawar -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 834 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150124/3968c448/attachment.sig> From steve at pearwood.info Sun Jan 25 14:03:27 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 26 Jan 2015 00:03:27 +1100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> Message-ID: <20150125130326.GG20517@ando.pearwood.info> On Sat, Jan 24, 2015 at 05:27:57PM +1000, Nick Coghlan wrote: > I would personally find the PEP more persuasive if it was framed in > terms of providing an improved definition of assertAlmostEqual that > better handles the limitations of binary floating point dynamic > ranges. > > The fact that unittest.assertAlmostEqual is in the standard library > implies that any improvement to it must also be in the standard > library, and moving the definition of near equality that unittest uses > out to the math module so it is reusable in other contexts makes sense > to me, especially if it means being able to share the definition > between unittest and the statistics module. Unfortunately, I don't think we can change assertAlmostEqual. If we change the implementation, tests which were passing may fail, and tests which were failing may pass. The approach I took for test_statistics was to subclass TestCase and add my own approx-equal assertion method: https://hg.python.org/cpython/file/fcab9c106f2f/Lib/test/test_statistics.py#l142 In my tests, I call assertApproxEqual(first, second), with optional values tol (absolute error tolerance) and rel (relative error). If tol and/or rel are not used, the assertion defaults to self.tol or self.rel, which if not set default to 0, which is equivalent to testing exact equality. The actual fuzzy comparison itself is handled by a function approx_equal(x, y, tol, rel). I propose: - somebody other than me should review NumericTestCase.assertApproxEqual and check that it does nothing unreasonable; - the assertApproxEqual method be moved into TestCase, making it available to any user of unittest; - the fuzzy comparison approx_equal can be turned into a static(?) method of TestCase (see below for the advantage of a method rather than a function); - since there are considerable disagreements about the right way to handle a fuzzy comparison when *both* an absolute and relative error are given, people who disagree with the default definition can simply subclass TestCase and redefine the approx_equal method. (Which is much simpler than having to write the whole assertApproxEqual method from scratch.) This gives a workable (although not obvious) solution to Guido's example use-case, which might become: from unittest import TestCase are_close = TestCase.approx_equal def sqrt(x): new_guess = 1 repeat: guess = new_guess new_guess = avg(guess, x/guess) until are_close(guess, new_guess) return guess Note that in this case, at least, we do want a symmetric version of "is_close", since neither guess nor new_guess is "correct", they are both approximations. -- Steven From njs at pobox.com Sun Jan 25 16:32:11 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 25 Jan 2015 15:32:11 +0000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEJNim=QfVxxVE1JHQ03Z=PrJAub1EWQx2TBOq7xtt-MFg@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <8835515035530371300@unknownmsgid> <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> <CALGmxEJNim=QfVxxVE1JHQ03Z=PrJAub1EWQx2TBOq7xtt-MFg@mail.gmail.com> Message-ID: <CAPJVwBnAfPEO=OigEFFFH42D+hVS_8m3L4=ajPPTUxeDMhLHwQ@mail.gmail.com> On Sat, Jan 24, 2015 at 7:59 PM, Chris Barker <chris.barker at noaa.gov> wrote: > >> One option would be to add a zero_tol argument, which is an absolute >> >> >> tolerance that is only applied if expected == 0. > > > OK -- now I know what the problem is here -- I thought I"d explored it > already. > > If you have a tolerance that you use only when expected is zero (or when > either is...) then you have teh odd reslut that a samll number will be > "close" to zero, but NOT close to a smaller number. I implemented this on a > branch in github: > > https://github.com/PythonCHB/close_pep/tree/zero_tol > > And you get the odd result: > > In [9]: is_close_to(1e-9, 0.0) > Out[9]: True > > fine -- the default zero_tol is 1e-8 > > In [10]: is_close_to(1e-9, 1e-12) > Out[10]: False > > but huh??? 1e-9 is close to zero, but not close to 1e-12???? Yes that's.... the idea? :-) If someone says that their expected value is exactly zero, then using relative tolerance just makes no sense at all. If they wanted an exact test they'd have written ==. And this is reasonable, because even if you know that the exact answer is zero, then you can't expect to get that with floating point -- +/-1e-16 or so is often the best you can hope for. But if someone says their expected value is 1e-12, then... well, it's possible that they'd be happy to instead get 0. But likely not. 0 is extremely far from 1e-12 in relative terms, and can easily cause qualitatively different behaviour downstream (e.g. log10(1e-12) == -12, log10(0) == error). The example that came up in the numpy discussion of these defaults is that statsmodels has lots of tests to make sure that their computations of tail value probabilities are correct. These are often tiny (e.g., P(being 6 standard deviations off from the mean of a normal) = 9.9e-10), but emphatically different from zero. So it's definitely safer all around to stick to relative error by default for non-zero expected values. Admittedly I am leaning pretty heavily on the "testing" use case here, but that's because AFAICT that's the overwhelming use case for this kind of functionality. Guido's optimization example is fine, but using a function like this isn't really the most obvious way to do optimization termination (as evidenced by the fact that AFAICT none of scipy's optimizers actually use use a rel_tol+abs_tol comparison on two values -- maybe they should?). And I don't understand Skip's example at all. (I get that you need to quantize the prices and you want to minimize error in doing this, but I don't understand why it matters whether you're within 1% of a breakpoint versus 40% of a breakpoint -- either way you're going to have to round.) > I'd much rather require people to have to think about what makes sense for > their use case than get trapped by a default that's totally inappropriate. But this seems a strange reason to advocate for a default that's totally inappropriate. is_close_to(x, 0.0) simply doesn't mean anything sensible in the current PEP -- even giving an error would be better. -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From chris.barker at noaa.gov Mon Jan 26 01:02:57 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Sun, 25 Jan 2015 16:02:57 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAPJVwBnAfPEO=OigEFFFH42D+hVS_8m3L4=ajPPTUxeDMhLHwQ@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <8835515035530371300@unknownmsgid> <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> <CALGmxEJNim=QfVxxVE1JHQ03Z=PrJAub1EWQx2TBOq7xtt-MFg@mail.gmail.com> <CAPJVwBnAfPEO=OigEFFFH42D+hVS_8m3L4=ajPPTUxeDMhLHwQ@mail.gmail.com> Message-ID: <CALGmxEKKpu_wt+ngkBmcZ=LxbAX6huBh9RwaBiNg6-_7fU6EZQ@mail.gmail.com> On Sun, Jan 25, 2015 at 7:32 AM, Nathaniel Smith <njs at pobox.com> wrote: > > If you have a tolerance that you use only when expected is zero (or when > > either is...) then you have the odd result that a small number will be > > "close" to zero, but NOT close to a smaller number. > > > And you get the odd result: > > > > In [9]: is_close_to(1e-9, 0.0) > > Out[9]: True > > > > fine -- the default zero_tol is 1e-8 > > > > In [10]: is_close_to(1e-9, 1e-12) > > Out[10]: False > > > > but huh??? 1e-9 is close to zero, but not close to 1e-12???? > > Yes that's.... the idea? :-) > > If someone says that their expected value is exactly zero, then using > relative tolerance just makes no sense at all. If they wanted an exact > test they'd have written ==. And this is reasonable, because even if > you know that the exact answer is zero, then you can't expect to get > that with floating point -- +/-1e-16 or so is often the best you can > hope for. > sure -- that's why I (and numpy and Steven's statistics test function) put in an absolute tolerance as well. If you know you testing near near, then you set an abs_tolernace that define what "near zero" or "small" mean in this case. But if someone says their expected value is 1e-12, then... well, it's > possible that they'd be happy to instead get 0. But likely not. 0 is > extremely far from 1e-12 in relative terms, And 1e-12 from zero also, of course. Which is the trick here. Even with an asymmetric test, 0.0 is not relatively close to anything, and nothing is relatively close to zero (as long as the relative tolerance is less than 1 -- which it really should be. So I think we should use the zero_tolerance option if either input is zero, but then we get these continuities. So It seems, if a user wants to use the same parameters to test a bunch of numbers, and some of them may be zero, that they should define what small is to them by setting an abs_tolerance. Though I guess I'd rather a zero_tol that defaulted to non-zero that an abs_tol that did. So we might be able to satisfy your observation that a lot of use cases call for testing against zero. > The example that came up in the numpy > discussion of these defaults is that statsmodels has lots of tests to > make sure that their computations of tail value probabilities are > correct. These are often tiny (e.g., P(being 6 standard deviations off > from the mean of a normal) = 9.9e-10), but emphatically different from > zero. So it's definitely safer all around to stick to relative error > by default for non-zero expected values. > But would you even need to test for zero then in that case? And if so, wouldn't setting abs_tol to what you wanted for "very small" be the right thing to do? I note that Steven's testing code the the stdlib statistics library used a rel_tolerance and abs_tolerance approach as well. I haven't seen any example of special casing zero anywhere. > Admittedly I am leaning pretty heavily on the "testing" use case here, > but that's because AFAICT that's the overwhelming use case for this > kind of functionality. I agree that it is as well -- sure you could use it for a simple recursive solution to an implicit equation, but how may people whip those up, compared to either testing code or writing a custom comparison designed specifically for the case at hand. > I'd much rather require people to have to think about what makes sense for > > their use case than get trapped by a default that's totally > inappropriate. > > But this seems a strange reason to advocate for a default that's > totally inappropriate. is_close_to(x, 0.0) simply doesn't mean > anything sensible in the current PEP -- even giving an error would be > better. > Sure it does -- it means nothing is relatively close to zero -- haven't we all agreed that that's the mathematically correct result? And if you write a test against zero it will reliably fail first time if you haven't set an abs_tolerance. So you will then be forces to decide what "near zero" means to you, and set an appropriate abs_tolerance. I think this points to having a separate function for absolute tolerance compared to zero -- but that's just abs(val) > zero_tolerance, so why bother? Or do you think there are common use cases where you would want purely relative tolerance, down to very close to zero, but want a larger tolerance for zero itself, all in the same comprehension? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150125/d04c6d25/attachment-0001.html> From chris.barker at noaa.gov Mon Jan 26 01:52:41 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Sun, 25 Jan 2015 16:52:41 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEKKpu_wt+ngkBmcZ=LxbAX6huBh9RwaBiNg6-_7fU6EZQ@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <8835515035530371300@unknownmsgid> <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> <CALGmxEJNim=QfVxxVE1JHQ03Z=PrJAub1EWQx2TBOq7xtt-MFg@mail.gmail.com> <CAPJVwBnAfPEO=OigEFFFH42D+hVS_8m3L4=ajPPTUxeDMhLHwQ@mail.gmail.com> <CALGmxEKKpu_wt+ngkBmcZ=LxbAX6huBh9RwaBiNg6-_7fU6EZQ@mail.gmail.com> Message-ID: <CALGmxEJ3Ze_4jm86J1yLL-eoppDWYFvDABiB84bVkFWpREJ-EQ@mail.gmail.com> Hi All, I've gone through all the messages in this thread since I posted the draft PEP. I have updated the code and PEP (on gitHub) with changes that were no brainers or seemed to have clear consensus. The PEP also needs some better motivational text -- I"ll try to get that soon. So I think we're left with only a few key questions: 1) Do we put anything in the stdlib at all? (the big one), which is closely tied to: 2) Do we explicitly cal this a testing utility, which would get reflected in the PEP, and mean that we'd probably want to add a unitest.TestCase assert that uses it. 3) Do we use an asymmetric or symmetric test? Most people seemed to be fine with the asymmetric test, but Steven just proposed the symmetric again. I'll comment on that later. 4) What do we do about tolerance relative to zero? Do we define a specific zero_tolerance parameter? or expect people to set abs_tolerance when they need to test against zero? And what should the default be? Here is my take on those issues: 1) Yes, we put something in. It's quite clear that there is no one solution that works best for every case (without a lot of parameters to set, anyway), but I'm quite sure that what I've proposed, modified with any solutions to the issues above, would be useful in the majority of cases. Let's keep in mind Nick's comment: "Note that the key requirement here should be "provide a binary float comparison function that is significantly less wrong than the current 'a == b'" 2) I do agree that the most frequent use case would be for testing, but that doesn't always mean format unit testing (it could be quick check on the command line, or iPython notebook, or....) and it certainly doesn't mean the unittest package. So I think it's a fine idea to add an assertion to TestCase that uses this, I'd much rather see it as a stand alone function (maybe in the math module). A static method of TestCase would be a compromise -- it's just some extra typing on in import line, but I think it would get lost to folks not using unittest. I note that Guido wrote: "To some, that means "unit test" or some other way of testing software. But I hope that's not the main use case." While Nick wrote: "I would personally find the PEP more persuasive if it was framed in terms of providing an improved definition of assertAlmostEqual that better handles the limitations of binary floating point dynamic ranges" So I'm not sure what to make of that. 3) I prefer the asymmetric test -- I've already given my reasons. But I'm pretty convinced that particularly for use in testing that it really doesn't matter: - relative tolerance tend to be small -- on order of 1e-8 or so. The 10% example I used in the PEP was to keep the math easy -- but it's not a common use case (for tests anyway) - folks tend to specify relative tolerance to an order of magnitude: ie.e 1e-8, not 1.323463e-8 -- if the magnitude of the tolerance is much smaller that its precision, then any of the definitions under consideration are effectively the same. So any of these are worth putting in the stdlib. 4) In my thinking and research, I decided that the (essentially optional) abs_tolerance parameter is the way to handle zero. But if Nathaniel or anyone else has use-cases in mind where that wouldn't work, we could add the zero_tol parameter to handle it instead. But not sure what the default should be -- if we think there is something special enough about order of magnitude 1, the something like 1e-12 would be good, but I'm not so sure. But it would be better to set such a default for zero_tolerance than for abs_tolerance. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150125/6674402d/attachment.html> From njs at pobox.com Mon Jan 26 02:17:02 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 26 Jan 2015 01:17:02 +0000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEKKpu_wt+ngkBmcZ=LxbAX6huBh9RwaBiNg6-_7fU6EZQ@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <8835515035530371300@unknownmsgid> <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> <CALGmxEJNim=QfVxxVE1JHQ03Z=PrJAub1EWQx2TBOq7xtt-MFg@mail.gmail.com> <CAPJVwBnAfPEO=OigEFFFH42D+hVS_8m3L4=ajPPTUxeDMhLHwQ@mail.gmail.com> <CALGmxEKKpu_wt+ngkBmcZ=LxbAX6huBh9RwaBiNg6-_7fU6EZQ@mail.gmail.com> Message-ID: <CAPJVwB=aV3mEfjikfwvA=_a7s2tViVjUiiYy54bHrcsE9pSLVA@mail.gmail.com> On Mon, Jan 26, 2015 at 12:02 AM, Chris Barker <chris.barker at noaa.gov> wrote: > On Sun, Jan 25, 2015 at 7:32 AM, Nathaniel Smith <njs at pobox.com> wrote: >> >> > If you have a tolerance that you use only when expected is zero (or when >> > either is...) then you have the odd result that a small number will be >> > "close" to zero, but NOT close to a smaller number. > > >> >> > And you get the odd result: >> > >> > In [9]: is_close_to(1e-9, 0.0) >> > Out[9]: True >> > >> > fine -- the default zero_tol is 1e-8 >> > >> > In [10]: is_close_to(1e-9, 1e-12) >> > Out[10]: False >> > >> > but huh??? 1e-9 is close to zero, but not close to 1e-12???? >> >> Yes that's.... the idea? :-) >> >> If someone says that their expected value is exactly zero, then using >> relative tolerance just makes no sense at all. If they wanted an exact >> test they'd have written ==. And this is reasonable, because even if >> you know that the exact answer is zero, then you can't expect to get >> that with floating point -- +/-1e-16 or so is often the best you can >> hope for. > > > sure -- that's why I (and numpy and Steven's statistics test function) put > in an absolute tolerance as well. If you know you testing near near, then > you set an abs_tolernace that define what "near zero" or "small" mean in > this case. > >> But if someone says their expected value is 1e-12, then... well, it's >> possible that they'd be happy to instead get 0. But likely not. 0 is >> extremely far from 1e-12 in relative terms, > > And 1e-12 from zero also, of course. Which is the trick here. Even with an > asymmetric test, 0.0 is not relatively close to anything, and nothing is > relatively close to zero (as long as the relative tolerance is less than 1 > -- which it really should be. So I think we should use the zero_tolerance > option if either input is zero, but then we get these continuities. > > So It seems, if a user wants to use the same parameters to test a bunch of > numbers, and some of them may be zero, that they should define what small is > to them by setting an abs_tolerance. > > Though I guess I'd rather a zero_tol that defaulted to non-zero that an > abs_tol that did. So we might be able to satisfy your observation that a lot > of use cases call for testing against zero. Yes, that's the idea -- defaulting rel_tol and zero_tol to non-zero values, and abs_tol to zero, gives you a set of defaults that will just work for people who want to write useful tests without having to constantly be distracted by floating point arcana. This does require that zero_tol is only applied for expected == 0.0, *not* for actual == 0.0, though. If you expected 1e-10 and got 0.0 then this *might* be okay in your particular situation but it really requires the user to think things through; a generic tool should definitely flag this by default. >> The example that came up in the numpy >> discussion of these defaults is that statsmodels has lots of tests to >> make sure that their computations of tail value probabilities are >> correct. These are often tiny (e.g., P(being 6 standard deviations off >> from the mean of a normal) = 9.9e-10), but emphatically different from >> zero. So it's definitely safer all around to stick to relative error >> by default for non-zero expected values. > > > But would you even need to test for zero then in that case? And if so, > wouldn't setting abs_tol to what you wanted for "very small" be the right > thing to do? I note that Steven's testing code the the stdlib statistics > library used a rel_tolerance and abs_tolerance approach as well. I haven't > seen any example of special casing zero anywhere. Right, this example came up when it was discovered that np.allclose() has a non-zero abs_tol by default, and that np.testing.assert_allclose() has a zero abs_tol by default. It's a terrible and accidental API design, but it turns out that people really are intentionally use one or the other depending on whether they expect to be dealing with exact zeros or to be dealing with small-but-non-zero values. The whole motivation for zero_tol is to allow a single set of defaults that satisfies both groups. >> Admittedly I am leaning pretty heavily on the "testing" use case here, >> but that's because AFAICT that's the overwhelming use case for this >> kind of functionality. > > > I agree that it is as well -- sure you could use it for a simple recursive > solution to an implicit equation, but how may people whip those up, compared > to either testing code or writing a custom comparison designed specifically > for the case at hand. > >> > I'd much rather require people to have to think about what makes sense >> > for >> > their use case than get trapped by a default that's totally >> > inappropriate. >> >> But this seems a strange reason to advocate for a default that's >> totally inappropriate. is_close_to(x, 0.0) simply doesn't mean >> anything sensible in the current PEP -- even giving an error would be >> better. > > > Sure it does -- it means nothing is relatively close to zero -- haven't we > all agreed that that's the mathematically correct result? And if you write a > test against zero it will reliably fail first time if you haven't set an > abs_tolerance. So you will then be forces to decide what "near zero" means > to you, and set an appropriate abs_tolerance. Tests against zero won't necessarily fail -- sometimes rounding errors do cancel out, and you do get 0.0 instead of 1e-16 or whatever. At least for some sets of inputs, or until the code gets perturbed/refactored slightly, etc. That's why I said it might actually be better to unconditionally fail when expected==0.0 rather than knowingly perform a misleading operation. My claim wasn't that is_close_to(x, 0.0) provides a mathematically ill-defined result. I agree that that's a reasonable definition of "relatively close to" (though one could make an argument that zero is not relatively close to itself -- after all, abs(actual - expected)/expected is ill-defined). Instead, my point was that if the user is asking "is this close to 0?" instead of "is this exactly equal to zero?" then they probably are expecting that there exist some inputs for which those two questions give different answers. Saying "well TECHNICALLY this is a valid definition of 'close to'" is certainly true but somewhat unkind. > I think this points to having a separate function for absolute tolerance > compared to zero -- but that's just abs(val) > zero_tolerance, so why > bother? Or it could just be the same function :-). Who wants to keep track of two functions that conceptually do the same thing? > Or do you think there are common use cases where you would want purely > relative tolerance, down to very close to zero, but want a larger tolerance > for zero itself, all in the same comprehension? inf = float("inf") for (x, expected) in [ (inf, inf), (100, 1e100), (1, 10), (0, 1), (-1, 0.1), (-100, 1e-100), (-inf, 0), ]: assert is_close_to(10 ** x, expected) Though really what I'm arguing is that all in the same userbase people want relative tolerance down close to zero but a larger tolerance for zero itself. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From chris.barker at noaa.gov Mon Jan 26 02:21:53 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Sun, 25 Jan 2015 17:21:53 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150125130326.GG20517@ando.pearwood.info> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> Message-ID: <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> I think I addressed most of these issues in the summary note I jsut posted, but a few specific comments: Unfortunately, I don't think we can change assertAlmostEqual. If we > change the implementation, tests which were passing may fail, and tests > which were failing may pass. > Agreed -- would have thought that was off the table. And ti's really a different test than the proposal -- it is an absolute_tolerance test, but where the tolerance is specified in number of decimal digits after the decimal point (or an optional specific delta) -- not really that useful , but I guess if you have assertTrue(), then why not? But adding a relative tolerance to unittest makes a lot of sense -- would "assertCloseTo" sound entirely too much like assertAlmostEqual? I think it may be OK if the docs for each pointed to the other. > The actual fuzzy comparison itself is handled by a function > approx_equal(x, y, tol, rel). > NOTE: the difference between this and the my current PEP version is that that absolute tolerance defaults to something other than zero (though it looks like it does default to zero for the assert method), and it is a symmetric test (what Boost calls the "strong" test) - somebody other than me should review NumericTestCase.assertApproxEqual > and check that it does nothing unreasonable; > Well it requires the tolerance values to be set on the instance, and they default to zero. So if we were to add this to unittest.TestCase, would you make those instance attributes of TestCase? If so, we want different defaults -- so people could call the assertion with defaults and get something useful. I suggest the one I put in my proposal, naturally ;-) I looked at the underlying function pretty closely. I don't see anything wrong with it. I did a few things differently: - no need to check for NaN explicitly, the comparisons take care of that anyway. - inspired by the boost approach - I used "and" and "or", rather than calling max() -- same result, slightly better performance. But in the end -- essentially the same as the PEP code, except where it's intended to be different. > - since there are considerable disagreements about the right way to > handle a fuzzy comparison when *both* an absolute and relative error > are given, people who disagree with the default definition can simply > subclass TestCase and redefine the approx_equal method. > > (Which is much simpler than having to write the whole assertApproxEqual > method from scratch.) > what assertApproxEqual does is add the ability to test a while sequence of values -- much like numpy's all_close. Do any of the other TestCase assertions provide that? But you could also add an optional parameter to pass in an alternate comparison function, rather than have it be a method of TestCase. As I said, I think it's better to have it available, and discoverable, for use outside of unitest. Note that in this case, at least, we do want a symmetric version of > "is_close", since neither guess nor new_guess is "correct", they are > both approximations. > true, but you are also asking the question -- is the new_guess much different that guess. Which points to a asymmetric test -- but either would work. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150125/6b980bec/attachment-0001.html> From rosuav at gmail.com Mon Jan 26 02:25:14 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 26 Jan 2015 12:25:14 +1100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEJ3Ze_4jm86J1yLL-eoppDWYFvDABiB84bVkFWpREJ-EQ@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <8835515035530371300@unknownmsgid> <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> <CALGmxEJNim=QfVxxVE1JHQ03Z=PrJAub1EWQx2TBOq7xtt-MFg@mail.gmail.com> <CAPJVwBnAfPEO=OigEFFFH42D+hVS_8m3L4=ajPPTUxeDMhLHwQ@mail.gmail.com> <CALGmxEKKpu_wt+ngkBmcZ=LxbAX6huBh9RwaBiNg6-_7fU6EZQ@mail.gmail.com> <CALGmxEJ3Ze_4jm86J1yLL-eoppDWYFvDABiB84bVkFWpREJ-EQ@mail.gmail.com> Message-ID: <CAPTjJmrHE4i-jZeWdM+TWg_GeKypoO7iyzwvOGDVOjVz01bAHA@mail.gmail.com> On Mon, Jan 26, 2015 at 11:52 AM, Chris Barker <chris.barker at noaa.gov> wrote: > I've gone through all the messages in this thread since I posted the draft > PEP. I have updated the code and PEP (on gitHub) with changes that were no > brainers or seemed to have clear consensus. The PEP also needs some better > motivational text -- I"ll try to get that soon. Worth posting the link again I think: draft PEP is here: https://github.com/PythonCHB/close_pep/blob/master/pep-0485.txt Though I'm not seeing many changes, so maybe I'm looking at the wrong version. ChrisA From abarnert at yahoo.com Mon Jan 26 02:39:00 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 25 Jan 2015 17:39:00 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> Message-ID: <9D7D37F5-D3C2-4080-B020-333F12D53093@yahoo.com> On Jan 25, 2015, at 17:21, Chris Barker <chris.barker at noaa.gov> wrote: >> Unfortunately, I don't think we can change assertAlmostEqual. If we >> change the implementation, tests which were passing may fail, and tests >> which were failing may pass. > > Agreed -- would have thought that was off the table. If assertAlmostEqual is primarily intended for float comparisons, and it really is misleadingly bad for that use, I'd think the right answer is to prove a new, better method under a different name, then deprecate the old one. That means old tests continue to work, but new tests stop being misled into testing the wrong thing, and if you're a few people will even think to redo old tests (and, presumably, discover long-standing bugs--otherwise either their tests cases are no good in the first place, or it's really not misleadingly bad, right?), which seems to be the best outcome you can reasonably hope for. That would imply that assertCloseTo should include the same sequence behavior as assertAlmostEqual, not just assertTrue(close_to), so it really is a drop-in replacement for most users. I suppose making assertAlmostEqual take an optional almost-equality function that defaults to the existing behavior, then deprecating calling it with the default, would have the same effect, but it seems clumsier in the long run. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150125/7f3b9583/attachment.html> From njs at pobox.com Mon Jan 26 02:57:30 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 26 Jan 2015 01:57:30 +0000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <9D7D37F5-D3C2-4080-B020-333F12D53093@yahoo.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <9D7D37F5-D3C2-4080-B020-333F12D53093@yahoo.com> Message-ID: <CAPJVwB=9yJyNkrXzjkeb7Eut2kJ5p0juuxRvqm7RhDmdwBE74w@mail.gmail.com> On Mon, Jan 26, 2015 at 1:39 AM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote: > If assertAlmostEqual is primarily intended for float comparisons, and it > really is misleadingly bad for that use, Since I was complaining about assertAlmostEqual, I should probably say explicitly what I don't like: It has two arguments for specifying precision: "places", which is measured in decimal digits, and "delta", which is an absolute tolerance like the ones we've discussed here. The fact that this pair of options exists, and the fact that relative precision is very very frequently talked about in terms of "significant digits", strongly suggests to me that these are the two settings for relative and absolute tolerances, and setting places=7 means that the numbers should match up to 7 significant digits. However, this is not what places=7. In fact, if 'places' is specified, then it means we set the absolute tolerance to 5 * 10**(-places - 1). So the problems are: - This is confusing/misleading/surprising. - There is no support for relative tolerances, which are almost always what you want -- using the same tolerances for comparing two numbers in the millions and for comparing two numbers in the millionths is going to lead to ugly results. - Because there is no support for relative tolerances, the defaults are wildly inappropriate unless the numbers being compared happen to be of normal size, like say between 10**-2 and 10**2. The docs don't warn about this. And in fact if you want to get a useful result when comparing two large numbers, you actually have to assert that they are the same to some negative number of decimal places. Technically this can be worked out from docstring (it references round(), and round() is documented to accept a negative ndigits argument), but it's very surprising and obscure. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From chris.barker at noaa.gov Mon Jan 26 06:43:27 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Sun, 25 Jan 2015 21:43:27 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <9D7D37F5-D3C2-4080-B020-333F12D53093@yahoo.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <9D7D37F5-D3C2-4080-B020-333F12D53093@yahoo.com> Message-ID: <CALGmxEKmiddBoz8x4r=vzCR0-sZBwGXhJzR3pxgCo=dfa4NsPA@mail.gmail.com> On Sun, Jan 25, 2015 at 5:39 PM, Andrew Barnert <abarnert at yahoo.com> wrote: > On Jan 25, 2015, at 17:21, Chris Barker <chris.barker at noaa.gov> wrote: > > Unfortunately, I don't think we can change assertAlmostEqual. > > Agreed -- would have thought that was off the table. > > > If assertAlmostEqual is primarily intended for float comparisons, > See Nathanials note -- but I think the key thing is that assertAlmostEqual is NOT really intended for float comparisons -- rather, it's intended for decimal numbers that happen to be stored in floats (does it pre-date the Decimal object?) In practical use, most of us are used to thinking in terms of decimal numbers, it's what we all learn in grade school. An in practice, Python, and every other language I've seen, takes decimal literals for floats, and displays floats as decimals. Also in practical use, using floats for decimal numbers works just fine, particularly if you round on output. So I that's the use-case for assertAlmostEqual -- it provides an easy way to check if numbers are almost equal to N decimal places. I guess that's a pretty common use case. So maybe folks would want to keep it around. (by the way -- does it work for Decimal objects -- it really should, it's more appropriate for them anyway!) On the other hand, those of us trained in science and engineering are used to working with scientific or engineering notation -- i.e. x.xxxEn (some number times ten raised to a power)-- a mantissa and an exponent. While we still decimal, it's a better match for floating point numbers. We are also trained to consider "significant figures" -- or the number of digits of accuracy of a value -- measured or computed. 123,000,000 has the same number of significant figures as 0.000123 But they are not of the same value at all. Anyway, a relative comparison provides something akin to a comparison to a certain number of significant figures -- really common and useful, and familiar to at least those trained in most disciplines of science or engineering. Key here is that I'm not suggesting a better or more correct assertAlmostEqual. I'm suggesting something different (arguably more useful). I'm honestly not sure how useful assertAlmostEqual is -- but apparently is was useful enough to put in the first place ,and to get used since then. > That would imply that assertCloseTo should include the same sequence > behavior as assertAlmostEqual, not just assertTrue(close_to), so it really > is a drop-in replacement for most users. > even if it isn't a replacemen -- that sequence behavior is nice -- and Steven has already written it as well. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150125/689a6b93/attachment-0001.html> From steve at pearwood.info Mon Jan 26 06:54:38 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 26 Jan 2015 16:54:38 +1100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAPJVwB=aV3mEfjikfwvA=_a7s2tViVjUiiYy54bHrcsE9pSLVA@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <8835515035530371300@unknownmsgid> <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> <CALGmxEJNim=QfVxxVE1JHQ03Z=PrJAub1EWQx2TBOq7xtt-MFg@mail.gmail.com> <CAPJVwBnAfPEO=OigEFFFH42D+hVS_8m3L4=ajPPTUxeDMhLHwQ@mail.gmail.com> <CALGmxEKKpu_wt+ngkBmcZ=LxbAX6huBh9RwaBiNg6-_7fU6EZQ@mail.gmail.com> <CAPJVwB=aV3mEfjikfwvA=_a7s2tViVjUiiYy54bHrcsE9pSLVA@mail.gmail.com> Message-ID: <20150126055432.GH20517@ando.pearwood.info> On Mon, Jan 26, 2015 at 01:17:02AM +0000, Nathaniel Smith wrote: > On Mon, Jan 26, 2015 at 12:02 AM, Chris Barker <chris.barker at noaa.gov> wrote: [...] > > Though I guess I'd rather a zero_tol that defaulted to non-zero that an > > abs_tol that did. So we might be able to satisfy your observation that a lot > > of use cases call for testing against zero. > > Yes, that's the idea -- defaulting rel_tol and zero_tol to non-zero > values, and abs_tol to zero, gives you a set of defaults that will > just work for people who want to write useful tests without having to > constantly be distracted by floating point arcana. I really don't think that setting one or two error tolerances is "floating point arcana". I don't think that having to explicitly decide on what counts as "close" (as either an absolute difference or a relative difference) is especially onerous: surely anyone writing code will be able to cope with one or two decisions: - close enough means they differ by no more than X - close enough means they differ by no more than X%, expressed as a fraction This isn't ULPs :-) I'm almost inclined to not set any defaults, except perhaps zero for both (in which case "close to" cleanly degrades down to "exactly equal" except slower) and force the user to explicitly choose a value. Arguments in favour of setting some defaults: - People who want a "zero-thought" solution will get one, even if it does the wrong thing for their specific application, but at least they didn't have to think about it. - The defaults might occasionally be appropriate. Arguments against: - There is no default error tolerance we can pick, whether relative or absolute, which will suit everyone all the time. Unless the defaults are appropriate (say) 50% of the time or more, they will just be an attractive nuisance (see zero-thought above). In the statistics tests, I had the opportunity to set my own global defaults, but I don't think I ever actually used them. Maybe I could have picked better defaults? I don't know. I did use defaults per test suite, so that's an argument in favour of having is_close (approx_equal) not use defaults, but assertIsClose (assertApproxEqual) use per-instance defaults. [Context: in the tests, I had an assertApproxEqual method that relied on approx_equal function. The function had defaults, but I never used them. The method defaulted to reading defaults from self.rel and self.tol, and I did use them.] > This does require that zero_tol is only applied for expected == 0.0, > *not* for actual == 0.0, though. If you expected 1e-10 and got 0.0 > then this *might* be okay in your particular situation but it really > requires the user to think things through; a generic tool should > definitely flag this by default. I really think that having three tolerances, once of which is nearly always ignored, is poor API design. The user usually knows when they are comparing against an expected value of zero and can set an absolute error tolerance. How about this? - Absolute tolerance defaults to zero (which is equivalent to exact equality). - Relative tolerance defaults to something (possibly zero) to be determined after sufficient bike-shedding. - An argument for setting both values to zero by default is that it will make it easy to choose one of "absolute or relative". You just supply a value for the one that you want, and let the other take the default of zero. - At the moment, I'm punting on the behaviour when both abs and rel tolerances are provided. That can be bike-shedded later. Setting both defaults to zero means that the zero-thought version: if is_close(x, y): ... will silently degrade to x == y, which is no worse than what people do now (except slower). We can raise a warning in that case. The only tricky situation might be if you *may* be comparing against zero, but don't know so in advance. There are some solutions to that: - The suggested "zero_tol" parameter, which I dislike. I think it is an ugly and confusing API. - Some versions of is_close may not require any special treatment for zero, depending on how it treats the situation where both abs and rel tolerances are given. Further discussion needed. - Worst case, people write this: if (expected == 0 and is_close(actual, expected, tol=1e-8) or is_close(actual, expected, rel=1e-5)): but I don't think that will come up in practice very often. In the test_statistics module, I had tests that looked like this: for x in [bunch of non-zero values]: y = do_some_calculation(x) self.assertApproxEqual(x, y, rel=0.01) y = do_some_calculation(0) self.assertApproxEqual(0, y, tol=0.000001) which isn't hard to do, so I don't think this is a real issue in practice. I think the case of "my expected value might be zero, but I'm not sure in advance" is rare and unusual enough that we don't need to worry about it. [...] > My claim wasn't that is_close_to(x, 0.0) provides a mathematically > ill-defined result. I agree that that's a reasonable definition of > "relatively close to" (though one could make an argument that zero is > not relatively close to itself -- after all, abs(actual - > expected)/expected is ill-defined). Don't write it that way. Write it this way: abs(actual - expected) <= relative_tolerance*expected Now if expected is zero, the condition is true if and only if actual==expected. It would be bizarre for is_close(a, a) to return False (or worse, raise an exception!) for any finite number. NANs, of course, are allowed to be bizarre. Zero is not :-) > Instead, my point was that if the > user is asking "is this close to 0?" instead of "is this exactly equal > to zero?" then they probably are expecting that there exist some > inputs for which those two questions give different answers. Saying > "well TECHNICALLY this is a valid definition of 'close to'" is > certainly true but somewhat unkind. I agree, but I think this is a symptom of essential complexity in the problem domain. Ultimately, "is close" is ill-defined, and *somebody* has to make the decision what that will be, and that decision won't satisfy everyone always. We can reduce the complexity in one place: * provide sensible default values that work for expected != 0 but only by increasing the complexity elsewhere: * when expected == 0 the intuition that is_close is different from exact equality fails We can get rid of that complexity, but only by adding it back somewhere else: * is_close(x, y, zero_tol=0.1) and is_close(x, y, zero_tol=0.00001) give the same result for all the x,y I tested! that is, zero_tol is nearly always ignored. Since people will often need to think about what they want "is close" to mean no matter what we do, I would prefer not to add the complexity of a third tolerance value. If that means that "zero thought" users end up inadvertently testing for exact equality without realising it, I think that's a price worth paying for a clean API. (As I said earlier, we can raise a warning in that case.) > > I think this points to having a separate function for absolute tolerance > > compared to zero -- but that's just abs(val) > zero_tolerance, so why > > bother? > > Or it could just be the same function :-). Who wants to keep track of > two functions that conceptually do the same thing? Agreed. -- Steven From chris.barker at noaa.gov Mon Jan 26 07:03:11 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Sun, 25 Jan 2015 22:03:11 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAPJVwB=aV3mEfjikfwvA=_a7s2tViVjUiiYy54bHrcsE9pSLVA@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <8835515035530371300@unknownmsgid> <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> <CALGmxEJNim=QfVxxVE1JHQ03Z=PrJAub1EWQx2TBOq7xtt-MFg@mail.gmail.com> <CAPJVwBnAfPEO=OigEFFFH42D+hVS_8m3L4=ajPPTUxeDMhLHwQ@mail.gmail.com> <CALGmxEKKpu_wt+ngkBmcZ=LxbAX6huBh9RwaBiNg6-_7fU6EZQ@mail.gmail.com> <CAPJVwB=aV3mEfjikfwvA=_a7s2tViVjUiiYy54bHrcsE9pSLVA@mail.gmail.com> Message-ID: <CALGmxEJcDRxTRgXQPPAr_jDcyiWM_LGJ9WXg623gQSyp5Ak+0w@mail.gmail.com> On Sun, Jan 25, 2015 at 5:17 PM, Nathaniel Smith <njs at pobox.com> wrote: > > Though I guess I'd rather a zero_tol that defaulted to non-zero that an > > abs_tol that did. So we might be able to satisfy your observation that a > lot > > of use cases call for testing against zero. > > Yes, that's the idea -- defaulting rel_tol and zero_tol to non-zero > values, and abs_tol to zero, gives you a set of defaults that will > just work for people who want to write useful tests without having to > constantly be distracted by floating point arcana. > OK -- I get it now -- this is really about getting a default for a zero tolerance test that does not mess up the relative test -- that may be a way to go. > This does require that zero_tol is only applied for expected == 0.0, > *not* for actual == 0.0, though. If you expected 1e-10 and got 0.0 > then this *might* be okay in your particular situation but it really > requires the user to think things through; a generic tool should > definitely flag this by default. > got it -- if they want hat, they can set the abs_tolerance to what they need. >> The example that came up in the numpy > >> discussion of these defaults is that statsmodels has lots of tests to > >> make sure that their computations of tail value probabilities are > >> correct. These are often tiny (e.g., P(being 6 standard deviations off > >> from the mean of a normal) = 9.9e-10), but emphatically different from > >> zero. So it's definitely safer all around to stick to relative error > >> by default for non-zero expected values. > Exactly why I don't think abs_tolerance should be anything other than 0.0 > > But would you even need to test for zero then in that case? And if so, > > wouldn't setting abs_tol to what you wanted for "very small" be the right > > thing to do? I note that Steven's testing code the the stdlib statistics > > library used a rel_tolerance and abs_tolerance approach as well. I > haven't > > seen any example of special casing zero anywhere. > > Right, this example came up when it was discovered that np.allclose() > has a non-zero abs_tol by default, and that > np.testing.assert_allclose() has a zero abs_tol by default. It's a > terrible and accidental API design, but it turns out that people > really are intentionally use one or the other depending on whether > they expect to be dealing with exact zeros or to be dealing with > small-but-non-zero values. why didn't they just override the defaults? but whatever. The whole motivation for zero_tol is to > allow a single set of defaults that satisfies both groups. > OK -- I'm buying it. However, what is a sensible default for zero_tolerance? I agree it's less critical than for abs_tolerance, but what should it be? Can we safely figure that order of magnitude one is most common, and something in the 1e-8 to 1e-14 range makes sense? I suppose that wouldn't be surprising to most folks. Tests against zero won't necessarily fail -- sometimes rounding errors > do cancel out, and you do get 0.0 instead of 1e-16 or whatever. At > least for some sets of inputs, or until the code gets > perturbed/refactored slightly, etc. That's why I said it might > actually be better to unconditionally fail when expected==0.0 rather > than knowingly perform a misleading operation. > I get it -- seems rare, certainly more rare than the other case, is_close_to passes for small numbers when it really shouldn't. And sure, you could get a pass the first time around, because, indeed you DID get exactly zero -- that should pass. But when you do refactor and introduce a slightly different answer, you'll get a failure then an can figure it out then. Are you actually proposing that the function should raise an Exception if expected == 0.0 and abs_tolerance is also 0.0? (and i guess zero_tolerance if there is one) > Or do you think there are common use cases where you would want purely > > relative tolerance, down to very close to zero, but want a larger > tolerance > > for zero itself, all in the same comprehension? > > inf = float("inf") > for (x, expected) in [ > (inf, inf), > (100, 1e100), > (1, 10), > (0, 1), > (-1, 0.1), > (-100, 1e-100), > (-inf, 0), > ]: > assert is_close_to(10 ** x, expected) > I meant a case that wasn't contrived ;-) > Though really what I'm arguing is that all in the same userbase people > want relative tolerance down close to zero but a larger tolerance for > zero itself. > Absolutely -- and adding a zero_tolerance may be a way to get everyone useful defaults. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150125/b8f4277f/attachment-0001.html> From steve at pearwood.info Mon Jan 26 07:39:41 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 26 Jan 2015 17:39:41 +1100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> Message-ID: <20150126063932.GI20517@ando.pearwood.info> On Sun, Jan 25, 2015 at 05:21:53PM -0800, Chris Barker wrote: [...] > But adding a relative tolerance to unittest makes a lot of sense -- would > "assertCloseTo" sound entirely too much like assertAlmostEqual? I think it > may be OK if the docs for each pointed to the other. CloseTo assumes an asymetric test, which isn't a given :-) I prefer ApproxEqual, although given that it is confusingly similar to AlmostEqual, IsClose would be my second preference. > > The actual fuzzy comparison itself is handled by a function > > approx_equal(x, y, tol, rel). > > > > NOTE: the difference between this and the my current PEP version is that > that absolute tolerance defaults to something other than zero (though it > looks like it does default to zero for the assert method), and it is a > symmetric test (what Boost calls the "strong" test) > > > - somebody other than me should review NumericTestCase.assertApproxEqual > > and check that it does nothing unreasonable; > > > > Well it requires the tolerance values to be set on the instance, and they > default to zero. So if we were to add this to unittest.TestCase, would you > make those instance attributes of TestCase? No, I would modify it to do something like this: if tol is None: tol = getattr(self, "tol", 0.0) # or some other default and similar for rel. I recommend using short names for the two error tolerances, tol and rel, because if people are going to be writing a lot of tests, having to write: self.assertIsClose(x, y, absolute_tolerance=0.001) will get tiresome. > > - since there are considerable disagreements about the right way to > > handle a fuzzy comparison when *both* an absolute and relative error > > are given, people who disagree with the default definition can simply > > subclass TestCase and redefine the approx_equal method. > > > > (Which is much simpler than having to write the whole assertApproxEqual > > method from scratch.) > > > > what assertApproxEqual does is add the ability to test a while sequence of > values -- much like numpy's all_close. Do any of the other TestCase > assertions provide that? I was motivated by assertEqual and the various sequence/list methods. I wanted to compare two lists element-wise using an approximate comparison, e.g.: # func1 and func2 are alternate implementations of the same thing a = [func1(x) for x in values] b = [func2(x) for x in values] self.assertApproxEqual(a, b) It wouldn't be meaningful to compare the two lists for approximate equality as lists, but it does make sense to do an element-by-element comparison. In the event of failure, you want an error message that is more specific than just the two lists being unequal. > But you could also add an optional parameter to pass in an alternate > comparison function, rather than have it be a method of TestCase. As I > said, I think it's better to have it available, and discoverable, for use > outside of unitest. That's an alternative too. I guess it boils down to whether you prefer inheritance or the strategy design pattern :-) I do think there are two distinct use-cases that should be included in the PEP: (1) Unit testing, and a better alternative to assertAlmostEqual. (2) Approximate equality comparisons, as per Guido's example. Note that those two are slightly different: in the unit testing case, you usually have an known expected value (not necessarily mathematically exact, but at least known) while in Guido's example neither value is necessarily better than they other, you just want to stop when they are close enough. Like Nick, I think the first is the more important one. In the second case, anyone writing a numeric algorithm is probably copying an algorithm which already incorporates a fuzzy comparison, or they know enough to write their own. The benefits of a standard solution are convenience and correctness. Assuming unittest provides a well-tested is_close/approx_equal function, why not use it? > > Note that in this case, at least, we do want a symmetric version of > > "is_close", since neither guess nor new_guess is "correct", they are > > both approximations. > > true, but you are also asking the question -- is the new_guess much > different that guess. Which points to a asymmetric test -- but either would > work. I can see we're going to have to argue about the "Close To" versus "Close" distinction :-) I suggest that in the interest of not flooding everyone's inboxes, we take that off-list until we have either a concensus or at least agreement that we cannot reach concensus. -- Steve From p.f.moore at gmail.com Mon Jan 26 08:07:25 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 26 Jan 2015 07:07:25 +0000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150126055432.GH20517@ando.pearwood.info> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <8835515035530371300@unknownmsgid> <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> <CALGmxEJNim=QfVxxVE1JHQ03Z=PrJAub1EWQx2TBOq7xtt-MFg@mail.gmail.com> <CAPJVwBnAfPEO=OigEFFFH42D+hVS_8m3L4=ajPPTUxeDMhLHwQ@mail.gmail.com> <CALGmxEKKpu_wt+ngkBmcZ=LxbAX6huBh9RwaBiNg6-_7fU6EZQ@mail.gmail.com> <CAPJVwB=aV3mEfjikfwvA=_a7s2tViVjUiiYy54bHrcsE9pSLVA@mail.gmail.com> <20150126055432.GH20517@ando.pearwood.info> Message-ID: <CACac1F-=goF824uuDgbAKijv6iajrBOqv8fqvYE-7MYVUtw35A@mail.gmail.com> On 26 January 2015 at 05:54, Steven D'Aprano <steve at pearwood.info> wrote: > I really don't think that setting one or two error tolerances is > "floating point arcana". The hundreds of messages on this topic would tend to imply otherwise :-( And to many users (including me, apparently - I expected the first one to give False), the following is "floating point arcana": >>> 0.1*10 == 1.0 True >>> sum([0.1]*10) 0.9999999999999999 >>> sum([0.1]*10) == 1 False > I don't think that having to explicitly decide > on what counts as "close" (as either an absolute difference or a > relative difference) is especially onerous: surely anyone writing code > will be able to cope with one or two decisions: > > - close enough means they differ by no more than X > > - close enough means they differ by no more than X%, expressed > as a fraction This does seem relatively straightforward, though. Although in the second case you glossed over the question of X% of *what* which is the root of the "comparison to zero" question, and is precisely where the discussion explodes into complexity that I can't follow, so maybe that's precisely the bit of "floating point arcana" that the naive user doesn't catch on to. I'm not saying that you are being naive, rather that readers of the docs (and hence users of the function) will be, and will find it confusing for precisely this reason. > I'm almost inclined to not set any defaults, except perhaps zero for > both (in which case "close to" cleanly degrades down to "exactly equal" > except slower) and force the user to explicitly choose a value. But the function (in the default case) then won't mean "close to" (at least not in any sense that people will expect). Maybe making it mandatory to specify one or the other parameter, and making them keyword-only parameters, would be sufficiently explicit. But see below). > Arguments in favour of setting some defaults: > > - People who want a "zero-thought" solution will get one, even > if it does the wrong thing for their specific application, but > at least they didn't have to think about it. > > - The defaults might occasionally be appropriate. > > > Arguments against: > > - There is no default error tolerance we can pick, whether relative > or absolute, which will suit everyone all the time. Unless the > defaults are appropriate (say) 50% of the time or more, they will > just be an attractive nuisance (see zero-thought above). I'm not sure what you're saying here - by "not setting defaults" do you mean making it mandatory for the user to supply a tolerance, as I suggested above? > I really think that having three tolerances, once of which is nearly > always ignored, is poor API design. The user usually knows when they are > comparing against an expected value of zero and can set an absolute > error tolerance. Agreed. > How about this? > > - Absolute tolerance defaults to zero (which is equivalent to > exact equality). > > - Relative tolerance defaults to something (possibly zero) to be > determined after sufficient bike-shedding. Starting the bike-shedding now, -1 on zero. Having is_close default to something that most users won't think of as behaving like their naive expectation of "is close" (as opposed to "equals") would be confusing. > - An argument for setting both values to zero by default is that > it will make it easy to choose one of "absolute or relative". You > just supply a value for the one that you want, and let the other > take the default of zero. Just make it illegal to set both. What happens when you have both set is another one of the things that triggers discussions that make my head explode. Setting just one implies the other is zero, setting neither implies whatever default is agreed on. > - At the moment, I'm punting on the behaviour when both abs and rel > tolerances are provided. That can be bike-shedded later. Don't allow it, it's too confusing for the target audience. > Setting both defaults to zero means that the zero-thought version: > > if is_close(x, y): ... > > will silently degrade to x == y, which is no worse than what people > do now (except slower). We can raise a warning in that case. It is worse, because it no longer says what it means. > The only tricky situation might be if you *may* be comparing against > zero, but don't know so in advance. This can probably be handled by sufficiently good documentation. Once it was clear to me that this was an asymmetric operation, and that you were comparing whether X is close to a known value Y, I stopped finding the requirement that you know what Y is to make sense of the function odd. Having said that, I don't think the name "is_close" makes the asymmetry clear enough. Maybe "is_close_to" would work better (there's still room for bikeshedding over which of the 2 arguments is implied as the "expected" value in that case)? >> My claim wasn't that is_close_to(x, 0.0) provides a mathematically >> ill-defined result. I agree that that's a reasonable definition of >> "relatively close to" (though one could make an argument that zero is >> not relatively close to itself -- after all, abs(actual - >> expected)/expected is ill-defined). > > Don't write it that way. Write it this way: > > abs(actual - expected) <= relative_tolerance*expected > > Now if expected is zero, the condition is true if and only if > actual==expected. I would call out this edge case explicitly in the documentation. It's a straightforward consequence of the definition, but it *will* be surprising for many users. Personally, I'd not thought of the implication till it was pointed out here. > It would be bizarre for is_close(a, a) to return False (or worse, > raise an exception!) for any finite number. NANs, of course, are > allowed to be bizarre. Zero is not :-) Definitely agreed. >> Instead, my point was that if the >> user is asking "is this close to 0?" instead of "is this exactly equal >> to zero?" then they probably are expecting that there exist some >> inputs for which those two questions give different answers. Saying >> "well TECHNICALLY this is a valid definition of 'close to'" is >> certainly true but somewhat unkind. > > I agree, but I think this is a symptom of essential complexity in the > problem domain. Ultimately, "is close" is ill-defined, and *somebody* > has to make the decision what that will be, and that decision won't > satisfy everyone always. We can reduce the complexity in one place: > > * provide sensible default values that work for expected != 0 > > but only by increasing the complexity elsewhere: > > * when expected == 0 the intuition that is_close is different > from exact equality fails > > We can get rid of that complexity, but only by adding it back somewhere > else: Agreed. The essential complexity here may not seem that complex to specialists, but I can assure you that it is for at least this user :-) Overall, I think it would be better to simplify the proposed function in order to have it better suit the expectations of its intended audience, rather than trying to dump too much functionality in it on the grounds of making it "general". If there's one clear lesson from this thread, it's that floating point closeness can mean a lot of things to people - and overloading one function with all of those meanings doesn't seem like a good way of having a clean design. Paul From p.f.moore at gmail.com Mon Jan 26 08:43:20 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 26 Jan 2015 07:43:20 +0000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150126063932.GI20517@ando.pearwood.info> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> Message-ID: <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> On 26 January 2015 at 06:39, Steven D'Aprano <steve at pearwood.info> wrote: >> > Note that in this case, at least, we do want a symmetric version of >> > "is_close", since neither guess nor new_guess is "correct", they are >> > both approximations. >> >> true, but you are also asking the question -- is the new_guess much >> different that guess. Which points to a asymmetric test -- but either would >> work. > > I can see we're going to have to argue about the "Close To" versus > "Close" distinction :-) > > I suggest that in the interest of not flooding everyone's inboxes, we > take that off-list until we have either a concensus or at least > agreement that we cannot reach concensus. Does it need to go off-list? I'm still unclear about the arguments over asymmetric vs symmetric (I suspect, as you alluded to earlier, that they reflect a more fundamental problem, which is that there are 2 different types of use case with different constraints) so I'd like to at least be aware of the content of any discussion... Paul From guettliml at thomas-guettler.de Mon Jan 26 09:24:07 2015 From: guettliml at thomas-guettler.de (=?UTF-8?B?VGhvbWFzIEfDvHR0bGVy?=) Date: Mon, 26 Jan 2015 09:24:07 +0100 Subject: [Python-ideas] datetime: Support infinity Message-ID: <54C5F9A7.3010209@thomas-guettler.de> Hi, postgreSQL supports infinity for datetime: http://www.postgresql.org/docs/current/static/datatype-datetime.html#AEN6027 {{{ infinity date, timestamp later than all other time stamps -infinity date, timestamp earlier than all other time stamps }}} Mapping this to python is not possible at the moment. See: http://initd.org/psycopg/docs/usage.html#infinite-dates-handling {{{ PostgreSQL can store the representation of an ?infinite? date, timestamp, or interval. Infinite dates are not available to Python, so these objects are mapped to date.max, datetime.max, interval.max. Unfortunately the mapping cannot be bidirectional so these dates will be stored back into the database with their values, such as 9999-12-31. }}} I don't know the internals of the datetime module. I guess it is not possible to support infinity. What do you think? Thomas G?ttler From mal at egenix.com Mon Jan 26 09:36:52 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 26 Jan 2015 09:36:52 +0100 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <54C5F9A7.3010209@thomas-guettler.de> References: <54C5F9A7.3010209@thomas-guettler.de> Message-ID: <54C5FCA4.7090007@egenix.com> On 26.01.2015 09:24, Thomas G?ttler wrote: > Hi, > > postgreSQL supports infinity for datetime: > > http://www.postgresql.org/docs/current/static/datatype-datetime.html#AEN6027 > > {{{ > infinity date, timestamp later than all other time stamps > -infinity date, timestamp earlier than all other time stamps > }}} > > > Mapping this to python is not possible at the moment. > > See: > > http://initd.org/psycopg/docs/usage.html#infinite-dates-handling > > {{{ > PostgreSQL can store the representation of an ?infinite? date, timestamp, or interval. Infinite > dates are not available to Python, so these objects are mapped to date.max, datetime.max, > interval.max. Unfortunately the mapping cannot be bidirectional so these dates will be stored back > into the database with their values, such as 9999-12-31. > }}} > > I don't know the internals of the datetime module. I guess it is not possible to support infinity. > > What do you think? Leaving aside the question of how useful such date/time values would be, you can use the PG work-around for supporting these in Python as well. In mxDateTime, we have special values which can be used for this: >>> mx.DateTime.MaxDateTime <mx.DateTime.DateTime object for '25200470046051299-12-31 00:00:00.00' at 7fd7c4b3b2b8> >>> mx.DateTime.MinDateTime <mx.DateTime.DateTime object for '-25200470046051299-01-01 00:00:00.00' at 7fd7c4b3b390> -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 26 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From solipsis at pitrou.net Mon Jan 26 10:14:23 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 26 Jan 2015 10:14:23 +0100 Subject: [Python-ideas] datetime: Support infinity References: <54C5F9A7.3010209@thomas-guettler.de> Message-ID: <20150126101423.4edd9abd@fsol> On Mon, 26 Jan 2015 09:24:07 +0100 Thomas G?ttler <guettliml at thomas-guettler.de> wrote: > Hi, > > postgreSQL supports infinity for datetime: > > http://www.postgresql.org/docs/current/static/datatype-datetime.html#AEN6027 > > {{{ > infinity date, timestamp later than all other time stamps > -infinity date, timestamp earlier than all other time stamps > }}} > > > Mapping this to python is not possible at the moment. > > See: > > http://initd.org/psycopg/docs/usage.html#infinite-dates-handling > > {{{ > PostgreSQL can store the representation of an ?infinite? date, timestamp, or interval. Infinite dates are not available > to Python, so these objects are mapped to date.max, datetime.max, interval.max. Unfortunately the mapping cannot be > bidirectional so these dates will be stored back into the database with their values, such as 9999-12-31. > }}} Unless someone has a real-world use for the values of date.max, datetime.max, interval.max, I find it rather counter-productive to not store them back as infinities. Adding infinities to the datetime module would probably be possible but someone has to figure out the arithmetic rules. Do we need "not a time" when adding infinity to -infinity? Regards Antoine. From steve at pearwood.info Mon Jan 26 12:44:40 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 26 Jan 2015 22:44:40 +1100 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <20150126101423.4edd9abd@fsol> References: <54C5F9A7.3010209@thomas-guettler.de> <20150126101423.4edd9abd@fsol> Message-ID: <20150126114440.GL20517@ando.pearwood.info> On Mon, Jan 26, 2015 at 10:14:23AM +0100, Antoine Pitrou wrote: > On Mon, 26 Jan 2015 09:24:07 +0100 > Thomas G?ttler <guettliml at thomas-guettler.de> wrote: > > PostgreSQL can store the representation of an ?infinite? date, timestamp, or interval. Infinite dates are not available > > to Python, so these objects are mapped to date.max, datetime.max, interval.max. Unfortunately the mapping cannot be > > bidirectional so these dates will be stored back into the database with their values, such as 9999-12-31. > Unless someone has a real-world use for the values of date.max, > datetime.max, interval.max, I find it rather counter-productive to not > store them back as infinities. That would make them de-facto infinities that weirdly don't look like infinities: py> datetime.date.max datetime.date(9999, 12, 31) If I'm reading this page correctly, PostgreSQL supports a lot of dates that Python doesn't, up to 5874897AD: http://www.postgresql.org/docs/9.4/static/datatype-datetime.html so it might not matter if 9999-12-31 gets turned into infinity rather than treated as a normal date. > Adding infinities to the datetime module would probably be possible but > someone has to figure out the arithmetic rules. Do we need "not a time" > when adding infinity to -infinity? I shouldn't think so. The purpose of NANs in IEEE-754 maths is to allow the programmer to delay dealing with the failed operation until the end of the calculation. I don't think that date calculations tend to be anywhere as complicated as mathematical ones, so it would be acceptable to just raise an exception. Besides, Postgresql doesn't have a "NotATime" value, so if we're trying to match what they do, we don't need one either. -- Steven From solipsis at pitrou.net Mon Jan 26 13:09:08 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 26 Jan 2015 13:09:08 +0100 Subject: [Python-ideas] datetime: Support infinity References: <54C5F9A7.3010209@thomas-guettler.de> <20150126101423.4edd9abd@fsol> <20150126114440.GL20517@ando.pearwood.info> Message-ID: <20150126130908.58b53811@fsol> On Mon, 26 Jan 2015 22:44:40 +1100 Steven D'Aprano <steve at pearwood.info> wrote: > On Mon, Jan 26, 2015 at 10:14:23AM +0100, Antoine Pitrou wrote: > > On Mon, 26 Jan 2015 09:24:07 +0100 > > Thomas G?ttler <guettliml at thomas-guettler.de> wrote: > > > > PostgreSQL can store the representation of an ?infinite? date, timestamp, or interval. Infinite dates are not available > > > to Python, so these objects are mapped to date.max, datetime.max, interval.max. Unfortunately the mapping cannot be > > > bidirectional so these dates will be stored back into the database with their values, such as 9999-12-31. > > > Unless someone has a real-world use for the values of date.max, > > datetime.max, interval.max, I find it rather counter-productive to not > > store them back as infinities. > > That would make them de-facto infinities that weirdly don't look like > infinities: That would only make them infinities in PostgreSQL. That sounds like a reasonable compromise for a probably little-used feature, and uncommon values. > > Adding infinities to the datetime module would probably be possible but > > someone has to figure out the arithmetic rules. Do we need "not a time" > > when adding infinity to -infinity? > > I shouldn't think so. The purpose of NANs in IEEE-754 maths is to allow > the programmer to delay dealing with the failed operation until the end > of the calculation. I don't think that date calculations tend to be > anywhere as complicated as mathematical ones, so it would be acceptable > to just raise an exception. Note Numpy datetimes and timedeltas do have a concept of "NotATime". But, yes, it's an unexpected feature. Regards Antoine. From ncoghlan at gmail.com Mon Jan 26 15:08:01 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Jan 2015 00:08:01 +1000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> Message-ID: <CADiSq7eCJtQroN-nYzdALh6rLU7Z2KyaYBy0gjUQtLPchw580A@mail.gmail.com> On 26 January 2015 at 11:21, Chris Barker <chris.barker at noaa.gov> wrote: > I think I addressed most of these issues in the summary note I jsut posted, > but a few specific comments: > > >> Unfortunately, I don't think we can change assertAlmostEqual. If we >> change the implementation, tests which were passing may fail, and tests >> which were failing may pass. > > > Agreed -- would have thought that was off the table. And ti's really a > different test than the proposal -- it is an absolute_tolerance test, but > where the tolerance is specified in number of decimal digits after the > decimal point (or an optional specific delta) -- not really that useful , > but I guess if you have assertTrue(), then why not? Since Python 3.2, unittest.assertAlmostEqual has also supported a "delta=value" keyword argument to specify an absolute tolerance directly, rather than using a number of decimal places. It's mutually exclusive with the default "places" argument. I may be missing something, but is there a reason that couldn't be adjusted to also accept a "error_ratio" keyword argument that was mutually exclusive with the other two ("places" and "delta")? With the keyword arguments all being about different ways to specify the error tolerance, I think it would be reasonable to leave that implicit rather than mentioning it in each name. The "places=N" argument could also potentially be adjusted to be a shorthand for "delta=10e-N" rather than its current definition. The mutual exclusion between "error_ratio" and "delta" would require some adjustment to handle values that may be near zero, but it's not clear to me that there's a generally applicable answer to how best to handle that, so it seems advisable to avoid trying to guess. Regards, Nick. P.S. I considered suggesting just "ratio", but that's ambiguous in a way that "delta" isn't: "ratio" could be referring to the ratio of the two numbers, rather than the ratio of the error to the magnitude of the expected value. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Jan 26 15:14:15 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Jan 2015 00:14:15 +1000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CADiSq7eCJtQroN-nYzdALh6rLU7Z2KyaYBy0gjUQtLPchw580A@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <CADiSq7eCJtQroN-nYzdALh6rLU7Z2KyaYBy0gjUQtLPchw580A@mail.gmail.com> Message-ID: <CADiSq7eEj4F-Eyh23LT16_9tUfTfJM3yKWR++wdAFBDP5mN7ZQ@mail.gmail.com> On 27 January 2015 at 00:08, Nick Coghlan <ncoghlan at gmail.com> wrote: > The "places=N" argument could also potentially be adjusted to be a > shorthand for "delta=10e-N" rather than its current definition. And for folks worried about the slight backwards compatibility break here: "the unittest.TestCase.assertAlmostEqual fuzzy equality function is now slightly less fuzzy if you're using the 'places' argument" is well within the scope of the kind of thing we're prepared to cover in the "Porting Notes" section of the What's New docs. The impact of such a change could also be mitigated by changing the default value so that the default behaviour of the new definition resulted in approximately the same absolute delta as the old definition. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris.barker at noaa.gov Mon Jan 26 17:31:49 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Mon, 26 Jan 2015 08:31:49 -0800 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <54C5F9A7.3010209@thomas-guettler.de> References: <54C5F9A7.3010209@thomas-guettler.de> Message-ID: <3114096308045821944@unknownmsgid> Postgres asside, a infinity and minus infinity datetime would be generally useful, Useful enough that I wrote them for my code base. And early the postgress folks think it's a useful construct. So +1 on adding this -- and not just hacked into the postgress adapter. My use case is thus: We have a model that does a computation through time. Various objects can be active for a given time period. We use datetimes to store the active_start and active_stop. For objects that are always active up to a particular stop time, or become active at a particular time, and then stay active forever, we needed a way to express -infTime and infTime. It probably would have mostly worked to use the min and max datetime values, but then we lose information to pass back to the user. We also convert to-from numpy datetimes, which have a different range. I have a sample implementation if anyone wants to look amateur it, though I don't think it does reflected comparisons correctly. it's really only just good enough for our use case. -Chris > On Jan 26, 2015, at 12:31 AM, "Thomas G?ttler" <guettliml at thomas-guettler.de> wrote: > > Hi, > > postgreSQL supports infinity for datetime: > > http://www.postgresql.org/docs/current/static/datatype-datetime.html#AEN6027 > > {{{ > infinity date, timestamp later than all other time stamps > -infinity date, timestamp earlier than all other time stamps > }}} > > > Mapping this to python is not possible at the moment. > > See: > > http://initd.org/psycopg/docs/usage.html#infinite-dates-handling > > {{{ > PostgreSQL can store the representation of an ?infinite? date, timestamp, or interval. Infinite dates are not available to Python, so these objects are mapped to date.max, datetime.max, interval.max. Unfortunately the mapping cannot be bidirectional so these dates will be stored back into the database with their values, such as 9999-12-31. > }}} > > I don't know the internals of the datetime module. I guess it is not possible to support infinity. > > What do you think? > > Thomas G?ttler > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From chris.barker at noaa.gov Mon Jan 26 18:33:02 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 26 Jan 2015 09:33:02 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150126063932.GI20517@ando.pearwood.info> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> Message-ID: <CALGmxEJi_eAOZG_aktG+7jjdhEyH79rzYg2Sq15UoMgsQNHiOA@mail.gmail.com> On Sun, Jan 25, 2015 at 10:39 PM, Steven D'Aprano <steve at pearwood.info> wrote: > On Sun, Jan 25, 2015 at 05:21:53PM -0800, Chris Barker wrote: > > But adding a relative tolerance to unittest makes a lot of sense -- would > > "assertCloseTo" sound entirely too much like assertAlmostEqual? I think > it > > may be OK if the docs for each pointed to the other. > > CloseTo assumes an asymetric test, which isn't a given :-) > > I prefer ApproxEqual, although given that it is confusingly similar to > AlmostEqual, IsClose would be my second preference. indeed -- I did add the "to" to imply the asymmetric test -- so I say if we go with asymmetric test then IsClose and the IsCloseTo if we go with the asymmetric test. > Well it requires the tolerance values to be set on the instance, and they > > default to zero. So if we were to add this to unittest.TestCase, would > you > > make those instance attributes of TestCase? > > No, I would modify it to do something like this: > > if tol is None: > tol = getattr(self, "tol", 0.0) # or some other default > > and similar for rel. > so the attributes would not be there by default but users could add them if they want: class my_tst(unitest.TestCase): tol = 1e-8 That would work, but seems like a pretty unclear API to me -- is there a precedent in unitest for this already? But I'll leave further discussion on that to other -- I don't like the UnitTest API anyway ;-) I recommend using short names for the two error tolerances, tol and rel, > because if people are going to be writing a lot of tests, having to > write: > > self.assertIsClose(x, y, absolute_tolerance=0.001) > > will get tiresome. Isn't that why you set an attribute on your class? but if short, at least rel_tol and abs_tol a plain "tol" could be too confusing (even though I did that in my first draft...) My current draft has rel_tolerance and abs_toelraance -- perhaps a bit too long to type often, but a few people asked for longer, more descriptive names. > > - since there are considerable disagreements about the right way to > > > handle a fuzzy comparison when *both* an absolute and relative error > > > are given, Is there? In this discussion , no one had any issue with the proposed approach: result = difference <= rel_tolerance*scaling_value or difference <= abs_tolerance The only issue brought up is that we might want to do it the numpy way for the sake of compatibility with numpy. That's why I didn't add it to my list of issues to resolve. > I was motivated by assertEqual and the various sequence/list methods. yup -- good to keep that trend going. > But you could also add an optional parameter to pass in an alternate > > comparison function, rather than have it be a method of TestCase. As I > > said, I think it's better to have it available, and discoverable, for use > > outside of unitest. > > That's an alternative too. I guess it boils down to whether you prefer > inheritance or the strategy design pattern :-) > Now that I think about it -- we could easily do both. Define a math.is_close_to() in TestCase: @staticmethod def is_close_to(*args, **kwargs): return math.isclose_to(*args, **kwargs) best of both worlds. I"ve got my stand alone function outside unittest, and folks can still override TestCase.is_close_to if they want. I do think there are two distinct use-cases that should be included in > the PEP: > > (1) Unit testing, and a better alternative to assertAlmostEqual. > > (2) Approximate equality comparisons, as per Guido's example. > > Note that those two are slightly different: in the unit testing case, > you usually have an known expected value (not necessarily mathematically > exact, but at least known) while in Guido's example neither value is > necessarily better than they other, you just want to stop when they are > close enough. > yes, but in an iterative solution you generally compute a solution, then use that to compute a new solution, and you want to know if the new one is significantly different than the previous -- so an asymmetric test does make some sense. But again either work work, and pretty much the same. Example forthcoming.... Like Nick, I think the first is the more important one. In the second > case, anyone writing a numeric algorithm is probably copying an > algorithm which already incorporates a fuzzy comparison, or they know > enough to write their own. The benefits of a standard solution are > convenience and correctness. Assuming unittest provides a well-tested > is_close/approx_equal function, why not use it? Exactly. I can see we're going to have to argue about the "Close To" versus > "Close" distinction :-) > I think we both understand and agree on the distinction. My take is: - Either will work fine in most instances - The asymmetric one is a bit clearer and maybe better for the testing use-case. - I'd be perfectly happy with either on in the standard library Maybe not consensus, but the majority on this thread seem to prefer the asymmetric test. We could, of course add a flag to turn on the symmetric test (probably the Boost "strong" case), but I'd rather not have more flags, and as you indicate above, the people for whom it matters will probably write their own comparison criteria anyway. It looks like we need to add a bunch of tet to the PEP about incorporating this into unitest -- I'd love it if someone else wrote that -- I'm not much of a unitest user anyway. Pull requests accepted: https://github.com/PythonCHB/close_pep -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150126/0b382b14/attachment-0001.html> From ron3200 at gmail.com Mon Jan 26 19:01:18 2015 From: ron3200 at gmail.com (Ron Adam) Date: Mon, 26 Jan 2015 12:01:18 -0600 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150126063932.GI20517@ando.pearwood.info> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> Message-ID: <ma5vdf$nqu$1@ger.gmane.org> On 01/26/2015 12:39 AM, Steven D'Aprano wrote: >>> > >Note that in this case, at least, we do want a symmetric version of >>> > >"is_close", since neither guess nor new_guess is "correct", they are >>> > >both approximations. >> > >> >true, but you are also asking the question -- is the new_guess much >> >different that guess. Which points to a asymmetric test -- but either would >> >work. > I can see we're going to have to argue about the "Close To" versus > "Close" distinction:-) > > I suggest that in the interest of not flooding everyone's inboxes, we > take that off-list until we have either a concensus or at least > agreement that we cannot reach concensus. Is there any reason why someone would not want to use an asymmetric version in a symmetric way? Cheers, Ron From chris.barker at noaa.gov Mon Jan 26 19:23:20 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 26 Jan 2015 10:23:20 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <ma5vdf$nqu$1@ger.gmane.org> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <ma5vdf$nqu$1@ger.gmane.org> Message-ID: <CALGmxEJVXtfmemcXpdQAmrSGCdjpWguWb325p7XiBHjY2eHGjQ@mail.gmail.com> On Mon, Jan 26, 2015 at 10:01 AM, Ron Adam <ron3200 at gmail.com> wrote: > Is there any reason why someone would not want to use an asymmetric > version in a symmetric way? > You mean they pre-sort the arguments the way they want? Sure, you could do that. Though if it were me, I'd probably just re-write the test myself. But throughout all this -- I haven't thought of a single case where I would prefer a symmetric test and it would matter. It certainly appeals to me as clean, and more similar to equal, etc, but I haven't come up with a practical reason to prefer it. Can anyone? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150126/c9dec8b0/attachment.html> From solipsis at pitrou.net Mon Jan 26 19:31:44 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 26 Jan 2015 19:31:44 +0100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <ma5vdf$nqu$1@ger.gmane.org> <CALGmxEJVXtfmemcXpdQAmrSGCdjpWguWb325p7XiBHjY2eHGjQ@mail.gmail.com> Message-ID: <20150126193144.504e926f@fsol> On Mon, 26 Jan 2015 10:23:20 -0800 Chris Barker <chris.barker at noaa.gov> wrote: > On Mon, Jan 26, 2015 at 10:01 AM, Ron Adam <ron3200 at gmail.com> wrote: > > > Is there any reason why someone would not want to use an asymmetric > > version in a symmetric way? > > > > You mean they pre-sort the arguments the way they want? > > Sure, you could do that. Though if it were me, I'd probably just re-write > the test myself. > > But throughout all this -- I haven't thought of a single case where I would > prefer a symmetric test and it would matter. But then it begs the question: are there cases where an asymmetric test matters? Simplicity has its virtues. Regards Antoine. From chris.barker at noaa.gov Mon Jan 26 20:26:59 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 26 Jan 2015 11:26:59 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150126193144.504e926f@fsol> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <ma5vdf$nqu$1@ger.gmane.org> <CALGmxEJVXtfmemcXpdQAmrSGCdjpWguWb325p7XiBHjY2eHGjQ@mail.gmail.com> <20150126193144.504e926f@fsol> Message-ID: <CALGmxE+1L9sTJ92XZ0tFOKSxWcZBTBx9dPfaJ9iJV5-+B-r89g@mail.gmail.com> On Mon, Jan 26, 2015 at 10:31 AM, Antoine Pitrou <solipsis at pitrou.net> wrote: > > But throughout all this -- I haven't thought of a single case where I > would > > prefer a symmetric test and it would matter. > > But then it begs the question: are there cases where an asymmetric test > matters? Yes -- see the example in the PEP and on this list. If you have a known value, and you want to know whether the value at hand is within some error of the known value, then you want an asymmetric test -- and if you have errors on order of 10% -- it does make a difference. Granted, I don't expect this use case to be common, but it was brought up on this list. > Simplicity has its virtues. But which is more simple? I came to the conclusion that the asymmetric was simpler to explain and reason about in any case. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150126/be6a58fc/attachment.html> From solipsis at pitrou.net Mon Jan 26 20:37:23 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 26 Jan 2015 20:37:23 +0100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <ma5vdf$nqu$1@ger.gmane.org> <CALGmxEJVXtfmemcXpdQAmrSGCdjpWguWb325p7XiBHjY2eHGjQ@mail.gmail.com> <20150126193144.504e926f@fsol> <CALGmxE+1L9sTJ92XZ0tFOKSxWcZBTBx9dPfaJ9iJV5-+B-r89g@mail.gmail.com> Message-ID: <20150126203723.0b7c7a77@fsol> On Mon, 26 Jan 2015 11:26:59 -0800 Chris Barker <chris.barker at noaa.gov> wrote: > > > Simplicity has its virtues. > > But which is more simple? I came to the conclusion that the asymmetric was > simpler to explain and reason about in any case. Symmetricity removes any uncertainty as to parameter order. Regards Antoine. From chris.barker at noaa.gov Mon Jan 26 20:42:53 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 26 Jan 2015 11:42:53 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEJi_eAOZG_aktG+7jjdhEyH79rzYg2Sq15UoMgsQNHiOA@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CALGmxEJi_eAOZG_aktG+7jjdhEyH79rzYg2Sq15UoMgsQNHiOA@mail.gmail.com> Message-ID: <CALGmxEJPWv4-AEd0Ws-N4efzA9nZPrCFamdfhvZrpHyPOF6SUg@mail.gmail.com> On Mon, Jan 26, 2015 at 9:33 AM, Chris Barker <chris.barker at noaa.gov> wrote: > yes, but in an iterative solution you generally compute a solution, then > use that to compute a new solution, and you want to know if the new one is > significantly different than the previous -- so an asymmetric test does > make some sense. But again either work work, and pretty much the same. > > Example forthcoming.... > OK -- here is a real-life application -- probably a bit too much (and too specialized) for the PEP, but for the record: Waves in the ocean have a relationship between the wave length, wave period, water depth (and the acceleration of gravity), known as the dispersion relationship. So-called because the wave speed is a function of the frequency and wave length, and so this relation also predicts the wave speed -- and waves of different frequencies move at different speeds -- os "disperse" as the move from the location they were generated -- but I digress.... The relationship is: omega**2 = g*k*tanh(k*h) where omega is the wave frequence, g is the accelartion of gravity, k is the wave number (2*pi/wave_length), and h is the water depth. In the usual case, the frequencey(omega) is known, and you want the wave number. As the wave number appears in two places, there is no direct solution, so a numerical method must be used. The simplist iterative solution is something like this: recast the equation as: k_2 = omega**2 / (g * tanh(k_1 * h)) - guess a k_1 - compute k_2 - check if k_2 is close to k_1 - if not, set k_1 to k_2 repeat. Here is the code for that: def dispersion(omega, h, g=9.806): "compute the dispersion relation" k_1 = 10.0 # initial guess while True: k_2 = omega**2 / (g * math.tanh(k_1 * h)) if is_close_to(k_2, k_1, tol=1e-5): break k_1 = k_2 return k_1 note that I've provided g to only four significant figures, so I set the tolerance to 1e-5 -- no need for more precision than that. Granted, there are better ways to do this that run faster, and I'm sure there is code out there to do it already (well, I know there is, I wrote some of it...). In fact, I had code very much like this that I used in grad school (not Python, though). Since then, I needed this a lot, and took the time to write up a faster-converging method using Newton's method in C. But frankly this works just fine, and any of the proposals on the table for is_closr_to would work fine with it as well. If I were a student and needed this, and is_close_to ws in the stdlib -- I'd probably use it. In fact, you write a generic iteration routine: def iterate(func, x_initial, *args): """ iterate to find a solution to the function passed in func should be a function that takes x as a first argument, and computes an approximation to x as a result. x_initial is an initial guess for the unknown value """ x_1 = x_initial while True: x_2 = func(x_1, *args) if is_close_to(x_2, x_1): break x_1 = x_2 return x_2 Where you can pass in the function you want to iterate over. Here it is for the above: def disp(k, omega, h, g=9.806): """ the linear wave dispersion relationship k as a function of k, omega, h, g """ return omega**2 / (g * math.tanh(k * h)) and I can then call it like this: k2 = iterate(disp, 10, omega, h ) Code on gitHub if you care. https://github.com/PythonCHB/close_pep/blob/master/iteration_example.py -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150126/41d048a8/attachment-0001.html> From eric at trueblade.com Mon Jan 26 20:58:17 2015 From: eric at trueblade.com (Eric V. Smith) Date: Mon, 26 Jan 2015 14:58:17 -0500 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEJi_eAOZG_aktG+7jjdhEyH79rzYg2Sq15UoMgsQNHiOA@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CALGmxEJi_eAOZG_aktG+7jjdhEyH79rzYg2Sq15UoMgsQNHiOA@mail.gmail.com> Message-ID: <54C69C59.20702@trueblade.com> On 01/26/2015 12:33 PM, Chris Barker wrote: > indeed -- I did add the "to" to imply the asymmetric test -- so I say if > we go with asymmetric test then IsClose and the IsCloseTo if we go with > the asymmetric test. Because there's not enough bike-shedding: I think: symmetric = AreClose assymetric = IsCloseTo make the distinction clearer. Eric. From ron3200 at gmail.com Mon Jan 26 21:06:29 2015 From: ron3200 at gmail.com (Ron Adam) Date: Mon, 26 Jan 2015 14:06:29 -0600 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150126203723.0b7c7a77@fsol> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <ma5vdf$nqu$1@ger.gmane.org> <CALGmxEJVXtfmemcXpdQAmrSGCdjpWguWb325p7XiBHjY2eHGjQ@mail.gmail.com> <20150126193144.504e926f@fsol> <CALGmxE+1L9sTJ92XZ0tFOKSxWcZBTBx9dPfaJ9iJV5-+B-r89g@mail.gmail.com> <20150126203723.0b7c7a77@fsol> Message-ID: <ma66o6$mst$1@ger.gmane.org> On 01/26/2015 01:37 PM, Antoine Pitrou wrote: > On Mon, 26 Jan 2015 11:26:59 -0800 > Chris Barker<chris.barker at noaa.gov> wrote: >> > >>> > >Simplicity has its virtues. >> > >> >But which is more simple? I came to the conclusion that the asymmetric was >> >simpler to explain and reason about in any case. > Symmetricity removes any uncertainty as to parameter order. Correct, it uses either the closer or further from zero value. But it seems to me it's not better than just using the asymmetric one as if it was a symmetric function. The difference is you have both options with the asymmetric version, including alternating the order if you want. Or, a symmetric version can use an average, in which case it can't be used for anything other than relative distance comparisons. (Not relative value comparisons.) Ron From ethan at stoneleaf.us Mon Jan 26 21:14:04 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 26 Jan 2015 12:14:04 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150126203723.0b7c7a77@fsol> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <ma5vdf$nqu$1@ger.gmane.org> <CALGmxEJVXtfmemcXpdQAmrSGCdjpWguWb325p7XiBHjY2eHGjQ@mail.gmail.com> <20150126193144.504e926f@fsol> <CALGmxE+1L9sTJ92XZ0tFOKSxWcZBTBx9dPfaJ9iJV5-+B-r89g@mail.gmail.com> <20150126203723.0b7c7a77@fsol> Message-ID: <54C6A00C.2060909@stoneleaf.us> On 01/26/2015 11:37 AM, Antoine Pitrou wrote: > On Mon, 26 Jan 2015 11:26:59 -0800 > Chris Barker <chris.barker at noaa.gov> wrote: >> >>> Simplicity has its virtues. >> >> But which is more simple? I came to the conclusion that the asymmetric was >> simpler to explain and reason about in any case. > > Symmetricity removes any uncertainty as to parameter order. Yes, and nothing is simpler than getting wrong answers -- I can generate those all day long without even trying! ;) One example of when it matters: resistors should have a value of 10, but can be off by 2.5% How would an easy symmetric test handle that? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150126/7e4413a4/attachment.sig> From p.f.moore at gmail.com Mon Jan 26 21:52:04 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 26 Jan 2015 20:52:04 +0000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEJPWv4-AEd0Ws-N4efzA9nZPrCFamdfhvZrpHyPOF6SUg@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CALGmxEJi_eAOZG_aktG+7jjdhEyH79rzYg2Sq15UoMgsQNHiOA@mail.gmail.com> <CALGmxEJPWv4-AEd0Ws-N4efzA9nZPrCFamdfhvZrpHyPOF6SUg@mail.gmail.com> Message-ID: <CACac1F8WzH1UcRSN89KJRhsFguEz1nwPPoaKCm=-2UCbLYiwvw@mail.gmail.com> On 26 January 2015 at 19:42, Chris Barker <chris.barker at noaa.gov> wrote: > OK -- here is a real-life application [details snipped] > The simplist iterative solution is something like this: recast the > equation as: > > k_2 = omega**2 / (g * tanh(k_1 * h)) > > - guess a k_1 > - compute k_2 > - check if k_2 is close to k_1 > - if not, set k_1 to k_2 repeat. [more details snipped] > In fact, you write a generic iteration routine: > > def iterate(func, x_initial, *args): > """ > iterate to find a solution to the function passed in > > func should be a function that takes x as a first argument, > and computes an approximation to x as a result. > > x_initial is an initial guess for the unknown value > """ > x_1 = x_initial > while True: > x_2 = func(x_1, *args) > if is_close_to(x_2, x_1): > break > x_1 = x_2 > return x_2 The point I was trying to make earlier (which came across as "nobody should be coding their own Newton iterations") is essentially this - surely iterating a relation until the value you need converges is a common operation, and libraries exist that do this for you? And those libraries would already have their own "is the result close enough" calculation (possibly tunable via arguments to the routine). Do people actually write the low-level algorithms by hand, so that a building block like is_close is worthwhile? The lack of people coming up with use cases suggests to me that maybe it isn't. FWIW, a quick google search came up with scipy.optimize.fixed_point, which is, AFAICT, the equivalent of your iterate(). (You probably know better than I do if that's the case). Paul From solipsis at pitrou.net Mon Jan 26 22:08:55 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 26 Jan 2015 22:08:55 +0100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <ma5vdf$nqu$1@ger.gmane.org> <CALGmxEJVXtfmemcXpdQAmrSGCdjpWguWb325p7XiBHjY2eHGjQ@mail.gmail.com> <20150126193144.504e926f@fsol> <CALGmxE+1L9sTJ92XZ0tFOKSxWcZBTBx9dPfaJ9iJV5-+B-r89g@mail.gmail.com> <20150126203723.0b7c7a77@fsol> <54C6A00C.2060909@stoneleaf.us> Message-ID: <20150126220855.63b09608@fsol> On Mon, 26 Jan 2015 12:14:04 -0800 Ethan Furman <ethan at stoneleaf.us> wrote: > On 01/26/2015 11:37 AM, Antoine Pitrou wrote: > > On Mon, 26 Jan 2015 11:26:59 -0800 > > Chris Barker <chris.barker at noaa.gov> wrote: > >> > >>> Simplicity has its virtues. > >> > >> But which is more simple? I came to the conclusion that the asymmetric was > >> simpler to explain and reason about in any case. > > > > Symmetricity removes any uncertainty as to parameter order. > > Yes, and nothing is simpler than getting wrong answers -- I can generate those all day long without even trying! ;) > > One example of when it matters: > > resistors should have a value of 10, but can be off by 2.5% This is the kind of test that's extremely easy to write by hand. I don't know why you would need the stdlib's help for that. Regards Antoine. From random832 at fastmail.us Tue Jan 27 00:36:38 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Mon, 26 Jan 2015 18:36:38 -0500 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <20150126114440.GL20517@ando.pearwood.info> References: <54C5F9A7.3010209@thomas-guettler.de> <20150126101423.4edd9abd@fsol> <20150126114440.GL20517@ando.pearwood.info> Message-ID: <1422315398.2846106.219242229.1609034D@webmail.messagingengine.com> On Mon, Jan 26, 2015, at 06:44, Steven D'Aprano wrote: > If I'm reading this page correctly, PostgreSQL supports a lot of dates > that Python doesn't, up to 5874897AD: Why doesn't Python support these? You know, as long as we're talking about expanding the range of datetime. From alexander.belopolsky at gmail.com Tue Jan 27 00:49:16 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 26 Jan 2015 18:49:16 -0500 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <1422315398.2846106.219242229.1609034D@webmail.messagingengine.com> References: <54C5F9A7.3010209@thomas-guettler.de> <20150126101423.4edd9abd@fsol> <20150126114440.GL20517@ando.pearwood.info> <1422315398.2846106.219242229.1609034D@webmail.messagingengine.com> Message-ID: <CAP7h-xbgZ6kY7XYchFaoN9J80RtZxGj4fayNMyOVCbGzzo7Z3Q@mail.gmail.com> On Mon, Jan 26, 2015 at 6:36 PM, <random832 at fastmail.us> wrote: > On Mon, Jan 26, 2015, at 06:44, Steven D'Aprano wrote: > > If I'm reading this page correctly, PostgreSQL supports a lot of dates > > that Python doesn't, up to 5874897AD: > > Why doesn't Python support these? YAGNI. 4-digit year is enough for all practical purposes. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150126/9c00fd9b/attachment.html> From chris.barker at noaa.gov Tue Jan 27 01:16:28 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 26 Jan 2015 16:16:28 -0800 Subject: [Python-ideas] Fwd: PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxE+jDF23HP+RXtTqbmCX6v7EhWBK1znn84aHNFhqRt=fJA@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <8835515035530371300@unknownmsgid> <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> <CALGmxEJNim=QfVxxVE1JHQ03Z=PrJAub1EWQx2TBOq7xtt-MFg@mail.gmail.com> <CAPJVwBnAfPEO=OigEFFFH42D+hVS_8m3L4=ajPPTUxeDMhLHwQ@mail.gmail.com> <CALGmxEKKpu_wt+ngkBmcZ=LxbAX6huBh9RwaBiNg6-_7fU6EZQ@mail.gmail.com> <CAPJVwB=aV3mEfjikfwvA=_a7s2tViVjUiiYy54bHrcsE9pSLVA@mail.gmail.com> <20150126055432.GH20517@ando.pearwood.info> <CACac1F-=goF824uuDgbAKijv6iajrBOqv8fqvYE-7MYVUtw35A@mail.gmail.com> <CALGmxE+jDF23HP+RXtTqbmCX6v7EhWBK1znn84aHNFhqRt=fJA@mail.gmail.com> Message-ID: <CALGmxEKZ1sG5F7AZL6wy3px9M6SuFCTaR1oMftmG-_WrSeVYCQ@mail.gmail.com> OOPS, sent to only Paul by mistake the first time. -Chris On Sun, Jan 25, 2015 at 11:07 PM, Paul Moore <p.f.moore at gmail.com> wrote: > And to many users (including me, apparently - I expected the first one > to give False), the following is "floating point arcana": > > >>> 0.1*10 == 1.0 > True > >>> sum([0.1]*10) > 0.9999999999999999 > >>> sum([0.1]*10) == 1 > False Any of the approaches on the table will do something reasonable in this case: In [4]: is_close_to.is_close_to(sum([0.1]*10), 1) testing: 0.9999999999999999 1 Out[4]: True Note that the 1e-8 d3fault I chose (which I am not committed to) is not ENTIRELY arbitrary -- it's about half the digits carried by a python float (double) -- essentially saying the values are close to about half of the precision available. And we are constrained here, the options are between 0.1 (which would be crazy, if you ask me!) and 1e-14 -- any larger an it would meaningless, and any smaller, and it would surpass the precision of a python float. PIcking a default near the middle of that range seems quite sane to me. This is quite different than setting a value for an absolute tolerance -- saying something is close to another number if the difference is less than 1e-8 would be wildly inappropriate when the smallest numbers a float can hold are on order of 1e-300! > This does seem relatively straightforward, though. Although in the > second case you glossed over the question of X% of *what* which is the > root of the "comparison to zero" question, and is precisely where the > discussion explodes into complexity that I can't follow, so maybe > that's precisely the bit of "floating point arcana" that the naive > user doesn't catch on to. arcana, maybe, not it's not a floating point issue -- X% of zero is zero absolutely precisely. But back to a point made earlier -- the idea here is to provide something better than naive use of x == y for floating point. A default for the relative tolerance provides that. There is no sane default for absolute tolerance, so I dont think we should set one. Note that the zero_tolerance idea provides a way to let is_close_to(something, 0.0) work out of the box, but I'm not sure we can come up with a sane default for that either -- in which case, what's the point? > I'm not sure what you're saying here - by "not setting defaults" do > you mean making it mandatory for the user to supply a tolerance, as I > suggested above? I for one, think making it mandatory to set one would be better than just letting the zeros get used. I've used the numpy version of this a lot for tests, , and my work flow is usually: write a test with the defaults if it pases, I'm done. If it fails, then I look and see if my code is broken, or if I can accept a larger tolerance. So I'm quite happy to have a default. > I really think that having three tolerances, once of which is nearly > > always ignored, is poor API design. The user usually knows when they are > > comparing against an expected value of zero and can set an absolute > > error tolerance. > > Agreed. > also agreed -- Nathanial -- can you live with this? Note that Nathaniel found a LOT of examples of people using assertAlmostEqual to compare to zero -- I think that's why he thinks it's important to have defaults that do something sane for that case. However, that is an absolute comparison function -- inherently different anyway -- it's also tied to "number of digits after the decimal place", so only appropriate for values near 1 anyway -- so you can define a sensible default there. not so here. > - Absolute tolerance defaults to zero (which is equivalent to > > exact equality). > yup > > - Relative tolerance defaults to something (possibly zero) to be > > determined after sufficient bike-shedding. > > Starting the bike-shedding now, -1 on zero. Having is_close default to > something that most users won't think of as behaving like their naive > expectation of "is close" (as opposed to "equals") would be confusing. 1e-8 -- but you already know that ;-) -- anything between 1e-8 and 1e-12 would be fine with me. > Just make it illegal to set both. What happens when you have both set > is another one of the things that triggers discussions that make my > head explode. sorry about the head -- but setting both is a pretty useful use-case -- you have a bunch of values you want to do a realive test on. Some of them maybe exactly zero (note -- it could be either expected or actual) -- and you know what "close to zero" means to you. so you set that for abs_tolerance. Done. And I actually didn't think anyone objected to that approach -- though maybe the exploded heads weren't able to write ;-) In fact, an absolute tolerance is so easy that I wouldn't write a function for it: abs(expected - actual) <= abs_tolerance I ended up adding it to my version to deal with the zero case. > Having said that, I don't think the name "is_close" makes the > asymmetry clear enough. Maybe "is_close_to" would work better (there's > still room for bikeshedding over which of the 2 arguments is implied > as the "expected" value in that case)? I now have "expected" and "actual", but I think those makes are too unclear -- I like "expected" -- anyone have a better idea for the other one? > > abs(actual - expected) <= relative_tolerance*expected > > > > Now if expected is zero, the condition is true if and only if > > actual==expected. > > I would call out this edge case explicitly in the documentation. It is called out, but I guess I need to make that clearer. Overall, I think it would be better to simplify the proposed function > in order to have it better suit the expectations of its intended > audience, rather than trying to dump too much functionality in it on > the grounds of making it "general". right -- that is why I've been resisting adding flags for all the various options -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150126/fa7cfbe7/attachment.html> From njs at pobox.com Tue Jan 27 01:42:14 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 27 Jan 2015 00:42:14 +0000 Subject: [Python-ideas] Fwd: PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEKZ1sG5F7AZL6wy3px9M6SuFCTaR1oMftmG-_WrSeVYCQ@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <8835515035530371300@unknownmsgid> <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> <CALGmxEJNim=QfVxxVE1JHQ03Z=PrJAub1EWQx2TBOq7xtt-MFg@mail.gmail.com> <CAPJVwBnAfPEO=OigEFFFH42D+hVS_8m3L4=ajPPTUxeDMhLHwQ@mail.gmail.com> <CALGmxEKKpu_wt+ngkBmcZ=LxbAX6huBh9RwaBiNg6-_7fU6EZQ@mail.gmail.com> <CAPJVwB=aV3mEfjikfwvA=_a7s2tViVjUiiYy54bHrcsE9pSLVA@mail.gmail.com> <20150126055432.GH20517@ando.pearwood.info> <CACac1F-=goF824uuDgbAKijv6iajrBOqv8fqvYE-7MYVUtw35A@mail.gmail.com> <CALGmxE+jDF23HP+RXtTqbmCX6v7EhWBK1znn84aHNFhqRt=fJA@mail.gmail.com> <CALGmxEKZ1sG5F7AZL6wy3px9M6SuFCTaR1oMftmG-_WrSeVYCQ@mail.gmail.com> Message-ID: <CAPJVwB=RYU2sHWsvi07owmqfhfBGrOFHqB6-GzeyhJMb5gvx0Q@mail.gmail.com> On Tue, Jan 27, 2015 at 12:16 AM, Chris Barker <chris.barker at noaa.gov> wrote: > On Sun, Jan 25, 2015 at 11:07 PM, Paul Moore <p.f.moore at gmail.com> wrote: >> >> I'm not sure what you're saying here - by "not setting defaults" do >> you mean making it mandatory for the user to supply a tolerance, as I >> suggested above? > > I for one, think making it mandatory to set one would be better than just > letting the zeros get used. > > I've used the numpy version of this a lot for tests, , and my work flow is > usually: > > write a test with the defaults > > if it pases, I'm done. > > If it fails, then I look and see if my code is broken, or if I can accept a > larger tolerance. > > So I'm quite happy to have a default. > >> > I really think that having three tolerances, once of which is nearly >> > always ignored, is poor API design. The user usually knows when they are >> > comparing against an expected value of zero and can set an absolute >> > error tolerance. >> >> Agreed. > > > also agreed -- Nathanial -- can you live with this? I can live with it, but I'd ignore the function and use allclose instead :-). In your workflow above, I'm guessing that with allclose then it's <1% of the time that you have a failure due to too-restrictive default tolerances, where you then have to switch from thinking about your problem to thinking about floating point details. With no absolute tolerance, this rises to ~30% (assuming my super quick and dirty github statistics are typical). That's a lot of workflow disruption. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From chris.barker at noaa.gov Tue Jan 27 01:46:35 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 26 Jan 2015 16:46:35 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CACac1F8hyiM+jcnZ28fc5COTfAJUCkL54qd8YQ2rEEqTHsDGig@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <8835515035530371300@unknownmsgid> <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> <CALGmxEJNim=QfVxxVE1JHQ03Z=PrJAub1EWQx2TBOq7xtt-MFg@mail.gmail.com> <CAPJVwBnAfPEO=OigEFFFH42D+hVS_8m3L4=ajPPTUxeDMhLHwQ@mail.gmail.com> <CALGmxEKKpu_wt+ngkBmcZ=LxbAX6huBh9RwaBiNg6-_7fU6EZQ@mail.gmail.com> <CAPJVwB=aV3mEfjikfwvA=_a7s2tViVjUiiYy54bHrcsE9pSLVA@mail.gmail.com> <20150126055432.GH20517@ando.pearwood.info> <CACac1F-=goF824uuDgbAKijv6iajrBOqv8fqvYE-7MYVUtw35A@mail.gmail.com> <CALGmxE+jDF23HP+RXtTqbmCX6v7EhWBK1znn84aHNFhqRt=fJA@mail.gmail.com> <CACac1F8hyiM+jcnZ28fc5COTfAJUCkL54qd8YQ2rEEqTHsDGig@mail.gmail.com> Message-ID: <CALGmxELba7+NvzvS5JXcLoikrUM96PBf_7+u=vOJC42ykc4=AA@mail.gmail.com> Sorry, This slipped off list -- bringin it back. On Mon, Jan 26, 2015 at 12:40 PM, Paul Moore <p.f.moore at gmail.com> wrote: > > Any of the approaches on the table will do something reasonable in this > > case: > > > > In [4]: is_close_to.is_close_to(sum([0.1]*10), 1) > > testing: 0.9999999999999999 1 > > Out[4]: True > > Yes, but that's not my point. I was responding to Steven's comment > that having 2 different types of tolerance isn't "arcana", by pointing > out that I find even stuff as simple as multiplication vs cumulative > addition confusing. And I should note that I was (many years ago!) a > maths graduate and did some numerical maths courses, so this stuff > isn't completely unknown to me. Right it can be arcane -- which is why I want this function, and why we want it to do something "sane" most of the time, be default. > Note that the 1e-8 default I chose (which I am not committed to) is not > > ENTIRELY arbitrary -- it's about half the digits carried by a python > float > > (double) -- essentially saying the values are close to about half of the > > precision available. And we are constrained here, the options are between > > 0.1 (which would be crazy, if you ask me!) and 1e-14 -- any larger an it > > would meaningless, and any smaller, and it would surpass the precision > of a > > python float. PIcking a default near the middle of that range seems quite > > sane to me. > > Sorry, that means nothing to me. Head exploding time again :-) Darn -- I'll try again -- with a relative tolerence, two values are only going to be close if their exponent is within one of each-other. So what you are setting is how many digits of the mantisa you care about. a toleranc eof 0.1 would be about one digit, and a tolerance of 1e-15 would be 15 digits. Python floats carry about 15 digits -- so the relative tolerance has to be betwwen 1e-1 and 1e-15 -- nothign else is useful or makes sense. So I put it in the middle: 1e-8 > > This is quite different than setting a value for an absolute tolerance -- > > saying something is close to another number if the difference is less > than > > 1e-8 would be wildly inappropriate when the smallest numbers a float can > > hold are on order of 1e-300! > > On the other hand, I find this completely obvious. (Well, mostly - > don't the gaps between the representable floats increase as the > magnitude gets bigger, so an absolute tolerance of 1e-8 might be > entirely reasonable when the numbers are sufficiently high? sure it would -- that's the point -- what makes sense as an absolute tolerance depends entirely on the magnitude of the numbers -- since we don't know the magnitude of the numbers someone may use, we can't set a reasonable default. > arcana, maybe, not it's not a floating point issue -- X% of zero is zero > > absolutely precisely. > > But the "arcana" I was talking about is that a relative error of X% > could be X% of the value under test, of the expected value, of their > average, or something else. Ahh! -- which is exactly the point I think some of us are making -- defining X% error relative to the "expected" value is the simplest and most straightforward to explain. That's the primary reason I prefer it. And only *one* of those values is zero, so > whether X% is a useful value is entirely dependent on the definition. > not sure what you meant here, but actually relative error goes to heck if either value is zero, and with any of the definitions we are working with. So X% is useful for any value except if one of the values is zero. > And how relative errors are defined *is* floating point arcana (I can > picture the text book page now, and it wasn't simple...) > semantics here -- defining a realtive error can be done with pure real numbers -- computing it can get complex with floating point. > But back to a point made earlier -- the idea here is to provide something > > better than naive use of > > > > x == y > <snip> > I still wonder whether "naive use of equality" is much of a target, > though. There are only two use cases that have been mentioned so far. > Testing is not about equality, because we're replacing > assertAlmostEqual. And when someone is doing an iterative algorithm, > they are looking for convergence, i.e. within a given range of the > answer. So neither use case would be using an equality test anyway. > well, the secondary target is a better (or more flexible) assertAlmostEqual. It is not suitable for arbitrarily large or small numbers, and particularly not numbers with a range of magnitudes -- a relative difference test is much needed. I'm not sure I follow your point, but I will say that if Nathaniel has > seen a lot of use cases for assertAlmostEqual that can't be easily > handled with the new function, then something is badly wrong. Well, I"m not suggesting that we replace assertAlmostEqual -- but rather augment it. IN fact, assertAlmostEqual is actually a an absolute tolerance test (expressed in terms f decimal places). That is the right thing, and the only right thing to use when you want to compare to zero. What I'm proposing a relative tolerance test, which is not the right thing to use for comparing to zero, but is the right thing to use when comparing numbers of varying magnitude. > There > aren't enough good use cases that we can reasonably decide to reject > any of them as out of scope, I've lost track of what we might be rejecting. The whole symmetric -- non symmetric argument really is bike shedding -- in the context of "better than ==" or "different but as good as assetAlmostEqual" -- any of them are just fine. so really all wer aare left with is defaults -- also bike-shedding, except for the default for the zero test, and there are really two options there: use 0.0 for abs_tolerance, and have it fail for any test against zero unless the user specifies something approporate for their use case. or use a SOME default, either for abs_tolerance or zero_tolerance, and make an assumption about the ofder of magnitide of the lielky results, so that it will "jsut work" for tests against zero. Maybe something small relative to one (like 1e-8) would be OK, but that concerns me -- I think you'd get false positives for small numbers which is worse that false negatives for all comparisons to zero. > 1e-8 -- but you already know that ;-) -- anything between 1e-8 and 1e-12 > > would be fine with me. > > TBH all I care about in this context is that there must be 2 values x > and y for which is_close(x,y) == True and x != y. everything on the table will do that. I'm tempted to > strengthen that to "for all y there must be at least 1 x such that..." > but it's possible that's *too* strong and it can't be achieved. > I think we could say that for all y except 0.0 -- and even zero if an abs_tolerance is greater than zero is set. > Basically, the default behaviour needs to encompass what I believe is > most people's intuition - that "close" is a proper superset of > "equal". > A good reason not to have all defaults be zero -- I don't think we need a function that doesn't work at all with default values. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150126/4a831487/attachment.html> From chris.barker at noaa.gov Tue Jan 27 02:01:24 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 26 Jan 2015 17:01:24 -0800 Subject: [Python-ideas] Fwd: PEP 485: A Function for testing approximate equality In-Reply-To: <CAPJVwB=RYU2sHWsvi07owmqfhfBGrOFHqB6-GzeyhJMb5gvx0Q@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <8835515035530371300@unknownmsgid> <A824502E-A552-4760-9F70-6087B19C228A@yahoo.com> <CALGmxEJNim=QfVxxVE1JHQ03Z=PrJAub1EWQx2TBOq7xtt-MFg@mail.gmail.com> <CAPJVwBnAfPEO=OigEFFFH42D+hVS_8m3L4=ajPPTUxeDMhLHwQ@mail.gmail.com> <CALGmxEKKpu_wt+ngkBmcZ=LxbAX6huBh9RwaBiNg6-_7fU6EZQ@mail.gmail.com> <CAPJVwB=aV3mEfjikfwvA=_a7s2tViVjUiiYy54bHrcsE9pSLVA@mail.gmail.com> <20150126055432.GH20517@ando.pearwood.info> <CACac1F-=goF824uuDgbAKijv6iajrBOqv8fqvYE-7MYVUtw35A@mail.gmail.com> <CALGmxE+jDF23HP+RXtTqbmCX6v7EhWBK1znn84aHNFhqRt=fJA@mail.gmail.com> <CALGmxEKZ1sG5F7AZL6wy3px9M6SuFCTaR1oMftmG-_WrSeVYCQ@mail.gmail.com> <CAPJVwB=RYU2sHWsvi07owmqfhfBGrOFHqB6-GzeyhJMb5gvx0Q@mail.gmail.com> Message-ID: <CALGmxEJ6SJ4r4y7hATBZvM5wRfaJ5u_b+VuHcnUH9PRggabeyg@mail.gmail.com> On Mon, Jan 26, 2015 at 4:42 PM, Nathaniel Smith <njs at pobox.com> wrote: > >> > I really think that having three tolerances, once of which is nearly > >> > always ignored, is poor API design. The user usually knows when they > are > >> > comparing against an expected value of zero and can set an absolute > >> > error tolerance. > >> > >> Agreed. > > > > > > also agreed -- Nathanial -- can you live with this? > > I can live with it, but I'd ignore the function and use allclose instead > :-). > why not just set an appropriate abs_tolerance? > In your workflow above, I'm guessing that with allclose then it's <1% > of the time that you have a failure due to too-restrictive default > tolerances, where you then have to switch from thinking about your > problem to thinking about floating point details. With no absolute > tolerance, this rises to ~30% (assuming my super quick and dirty > github statistics are typical). That's a lot of workflow disruption. well they by definition aren't typical, as assertAlmostEqual isn't the same function -- it is only useful for an absolute tolerance for values near magnitude 1 -- so anyone that needed a relative tolerance (which I"m trying to provide) wouldn't have used it. But your point about numpy.allclose is taken -- it sets a non-zero default abs_tol ( atol=1e-08 ) -- and apparently that's worked well for you and others. Honestly, I've been using numpy's allclose for ages without thinking about abs_tol -- but after all this, I need to go back an look at all my tests -- I"m not at all sure that that was always appropriate! I guess the trade-offs for a default are: What are the most common cases? - so here -- do people only rarely work with numbers much smaller than 1? (i.e. less than 1e-4 or so?) How disastrous is it to have an inappropriate default? - Here is where I think it's a bad idea to have default: A) If there is a default: Someone writes a test where they compare their very small value to 0.0, and there is a default, the test will pass, even if the very small value really is far away from zero for their use-case. The test passes, they are happy, even though the result they are testing are totally bogus -- they very well might not notice. B) If there is no (non-zero) default: Someone writes a test where they compare their very small value to 0.0, the test will fail, even if the very small value really is close to zero for their use case. The test has failed, so they will examine the results, determine that it's as good as it can get, adjust the abs_tolerance value in the test, and away they go. Maybe B is rare -- I have no idea, but it does seem a substantially worse outcome -- the attractive nuisance referred to earlier. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150126/c8bb1f71/attachment-0001.html> From chris.barker at noaa.gov Tue Jan 27 02:10:44 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 26 Jan 2015 17:10:44 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CACac1F8WzH1UcRSN89KJRhsFguEz1nwPPoaKCm=-2UCbLYiwvw@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CALGmxEJi_eAOZG_aktG+7jjdhEyH79rzYg2Sq15UoMgsQNHiOA@mail.gmail.com> <CALGmxEJPWv4-AEd0Ws-N4efzA9nZPrCFamdfhvZrpHyPOF6SUg@mail.gmail.com> <CACac1F8WzH1UcRSN89KJRhsFguEz1nwPPoaKCm=-2UCbLYiwvw@mail.gmail.com> Message-ID: <CALGmxEJC+AycT+B2T0zDRfwqSFvNSty9b3FRO+y-U2jhLJd5Hw@mail.gmail.com> On Mon, Jan 26, 2015 at 12:52 PM, Paul Moore <p.f.moore at gmail.com> wrote: > [details snipped] > > The point I was trying to make earlier (which came across as "nobody > should be coding their own Newton iterations") well, some people should ;-) > is essentially this - > surely iterating a relation until the value you need converges is a > common operation, and libraries exist that do this for you? sure -- > FWIW, a quick google search came up with scipy.optimize.fixed_point, > which is, AFAICT, the equivalent of your iterate(). (You probably know > better than I do if that's the case). > Sure -- scipy.optimize is full of all sorts of optimizers suited to various problems such as these. However, sometimes you have your particular problem and you just want to get an answer without bringing the while dependency of scipy (or whatever) to bear, and figure out how to use the darn thing. -- and which one to use. I really did write that function in grad school -- despite having MATLAB and its optimization package (not to mention the Fortran minpack) available. That being said .. if if we want to say the primary use case is testing that's fine with me -- just please not buried in unittest.TestCase somewhere. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150126/a141239a/attachment.html> From yawar.amin at gmail.com Tue Jan 27 03:59:56 2015 From: yawar.amin at gmail.com (Yawar Amin) Date: Mon, 26 Jan 2015 21:59:56 -0500 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <54C5F9A7.3010209@thomas-guettler.de> References: <54C5F9A7.3010209@thomas-guettler.de> Message-ID: <54C6FF2C.3040703@gmail.com> Hi Thomas, On 2015-01-26 03:24, Thomas G?ttler wrote: > [...] > I don't know the internals of the datetime module. I guess it is not > possible to support infinity. > > What do you think? I think it's a great idea to support infinity dates in Python (in fact, in any modern date handling library). If I read you right, the problem with mapping back to Postgres is that infinity dates are being represented in-band in Python, so they get mapped to non-infinite dates in Postgres. The solution is to represent infinity dates as out-of-band values. My take on this is to represent them as subtypes of datetime and handle those subtypes appropriately. A rough (but working) sketch of the idea: import datetime as dt class datetime_ext(dt.datetime): def __new__(cls, *args, **kwargs): return dt.datetime.__new__(cls, *args, **kwargs) # Getting the infinity dates. @staticmethod def pos_inf(): return datetime_inf(1) @staticmethod def neg_inf(): return datetime_inf(-1) # More fun names. @staticmethod def big_bang(): return datetime_ext.neg_inf() @staticmethod def big_crunch(): return datetime_ext.pos_inf() # Now the fun part. def _are_we_inf(self, other): return ( isinstance(self, datetime_inf), isinstance(other, datetime_inf) ) def __lt__(self, other): awi = self._are_we_inf(other) if awi == (True, True): return self.sign < other.sign elif awi == (False, True): return other.sign > 0 elif awi == (True, False): return self.sign < 0 else: return self < other def __gt__(self, other): awi = self._are_we_inf(other) if awi == (True, True): return self.sign > other.sign elif awi == (False, True): return other.sign < 0 elif awi == (True, False): return self.sign > 0 else: return self > other def __eq__(self, other): awi = self._are_we_inf(other) if awi == (True, True): return self.sign == other.sign elif awi == (False, False): return self == other else: return False def __ne__(self, other): return not (self == other) def __le__(self, other): return (self < other) or (self == other) def __ge__(self, other): return (self > other) or (self == other) class datetime_inf(datetime_ext): def __new__(cls, sign): # The numbers passed in to the constructor are meaningless; they # should never be used. retval = datetime_ext.__new__(cls, 1, 1, 1) retval.sign = sign return retval def _raise_err(self): raise TypeError("Infinity date") def date(self): return datetime_inf(self.sign) def time(self): _raise_err() def timetz(self): _raise_err() def replace(self): _raise_err() def astimezone(self): _raise_err() # ... all the others ... # All True: print(datetime_ext.big_bang() == datetime_ext.big_bang()) print(datetime_ext.big_crunch() == datetime_ext.big_crunch()) print(datetime_ext.big_bang() < datetime_ext.big_crunch()) print(datetime_ext.big_crunch() > datetime_ext.big_bang()) print(datetime_ext.big_bang() < dt.datetime.today()) print(datetime_ext.big_crunch() > dt.datetime.today()) # Added bonuses (boni?): print(dt.datetime.max < datetime_ext.big_crunch()) print(dt.datetime.min > datetime_ext.big_bang()) So the basic idea here is to override the comparison methods to treat the datetime_inf type specially. The implementation of datetime_inf isn't particularly elegant, but I think it works for its intended purpose. Now with these, I'm pretty sure you can write an InfDateWrapper class that checks for datetime_ext.big_bang() and datetime_ext.big_crunch() and returns the corresponding Postgres '-infinity::date' and 'infinity::date'; and that the mapping will be fully bidirectional. And as I said, I believe something like this should be added to the Python stdlib. If no one has a better implementation I'll start working on a proposal. Regards, Yawar -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 834 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150126/b11b5ea7/attachment.sig> From steve at pearwood.info Tue Jan 27 04:08:00 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 27 Jan 2015 14:08:00 +1100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> Message-ID: <20150127030800.GQ20517@ando.pearwood.info> On Mon, Jan 26, 2015 at 07:43:20AM +0000, Paul Moore wrote: > On 26 January 2015 at 06:39, Steven D'Aprano <steve at pearwood.info> wrote: > > I suggest that in the interest of not flooding everyone's inboxes, we > > take that off-list until we have either a concensus or at least > > agreement that we cannot reach concensus. > > Does it need to go off-list? And now you know why there are hundreds of messages in this thread ;-) No, it doesn't need to go off-list, but I'm suffering badly from email fatigue, not just because of this thread but it is one of the major causes, and I'm sure I'm not the only one. > I'm still unclear about the arguments > over asymmetric vs symmetric (I suspect, as you alluded to earlier, > that they reflect a more fundamental problem, which is that there are > 2 different types of use case with different constraints) so I'd like > to at least be aware of the content of any discussion... Symmetry and asymmetry of "close to" is a side-effect of the way you calculate the fuzzy comparison. In real life, "close to" is always symmetric because distance is the same whether you measure from A to B or from B to A. The distance between two numbers is their difference, which is another way of saying the error between them: delta = abs(x - y) (delta being the traditional name for this quantity in mathematics), and obviously delta doesn't depend on the order of x and y. But if we express that difference as a fraction of some base value, i.e. as a relative error, the result depends on which base value you choose: delta/x != delta/y so suddenly we introduce an asymmetry which doesn't reflect any physical difference. The error between x and y is the same whichever way you measure, but that error might be 10% of x and 12.5% of y (say). What *fundamentally* matters is the actual error, delta. But to decide whether any specific value for delta is too much or not, you need to pick a maximum acceptable delta, and that depends on context: a maximum acceptable delta of 0.0001 is probably too big if your x and y are around a billionth, and way too small if they are around a billion. Hence we often prefer to work with relative tolerances ("give or take 1%") since that automatically scales with the size of x and y, but that introduces an asymmetry. Asymmetry is bad, because it is rather surprising and counter-intuitive that "x is close to y", but "y is not close to x". It may also be bad in a practical sense, because people will forget which order they need to give x and y and will give them in the wrong order. I started off with an approx_equal function in test_statistics that was symmetric, and I could never remember which way the arguments went. (We can mitigate against the practical failure with explicit argument names "actual" and "expected" instead of generic ones. But who wants to be using keyword arguments for this all the time?) Example: suppose the user supplies a relative tolerance of 0.01 ("plus or minus one percent"), with x=100.0 and y=99.0. Then delta = 1.0. Is that close? If we use x as the base: 1 <= 0.01*100 returns True, but if we use y as the base: 1 <= 0.01*99 returns False. Instead, Bruce Dawson recommends using the larger of x and y: https://randomascii.wordpress.com/2012/02/25/comparing-floating-point-numbers-2012-edition/ Quote: To compare f1 and f2 calculate diff = fabs(f1-f2). If diff is smaller than n% of max(abs(f1),abs(f2)) then f1 and f2 can be considered equal. This is especially appropriate when you just want to know whether x and y differ from each other by an acceptibly small amount, without specifying which is the "true" value and which the "true value plus or minus some error". Other alternatives would be to take the smaller, or the average, of x and y. Time permitting, over the next day or so I'll draw up some diagrams to show how each of these tactics change what counts as close or not close. -- Steve From chris.barker at noaa.gov Tue Jan 27 05:25:20 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 26 Jan 2015 20:25:20 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150127030800.GQ20517@ando.pearwood.info> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> Message-ID: <CALGmxEKZjkz_iRqte=n3y0kQ6FyC5-30-chm_LAcWFY-C1R0cQ@mail.gmail.com> On Mon, Jan 26, 2015 at 7:08 PM, Steven D'Aprano <steve at pearwood.info> wrote: > > No, it doesn't need to go off-list, but I'm suffering badly from email > fatigue, not just because of this thread but it is one of the major > causes, and I'm sure I'm not the only one. > > nope :-) But one reason to take it off-list is that these very long, if not endless, circular conversations gives the impression that it really matters which specif choices are made, and that we will never come to a consensus about it, so this should not go in the stdlib. However, I'm pretty sure that we're down to details, that while interesting, don't really matter -- whether we use a asymmetric or symmetric test, weak or string version, we'll get something that will work better than == or assertAlmostEqual, and it will do the right thing in the vast majority of cases. I could live with, and indeed be happy with, any of the solutions on the table. My take from this thread is that most people converged on the asymmetric option as the better choice, but Steven feels strongly that the symetric option is the way to go. I don't know if this is a stopper for anyone, though. Is there anyone that could only live with one of the options? (by live with, I mean think that we'd be better off with nothing in the standard lib that one of these options) Please speak. The other issue is whether to have a default that will return True for at least common uses of comparison to zero. - I think it's better to be safe than sorry, and not let folks accidentally think they have a value close to zero that isn't really. - Nathaniel thinks that it's better to provide a default that will give an answer for is close to zero that will at least work for common cases. I could live with what Nathaniel Proposes, and I believe he said he could live with what I propose -- so this is not a stopper. However, someone's going to need to come up with what that default value -- part of why I think it should be zero is that I have no idea if a small-compared-to-one default is reasonable. I think that's it. Perhaps folks could focus now on issues that they think are show stoppers. Or bike-shed the parameter names and stuff, if you really want to paint a bike shed ;-) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150126/1a24431b/attachment-0001.html> From rosuav at gmail.com Tue Jan 27 06:30:24 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 27 Jan 2015 16:30:24 +1100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEKZjkz_iRqte=n3y0kQ6FyC5-30-chm_LAcWFY-C1R0cQ@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEKZjkz_iRqte=n3y0kQ6FyC5-30-chm_LAcWFY-C1R0cQ@mail.gmail.com> Message-ID: <CAPTjJmqFeKvri7jeFU_NAofrih1Z6hieuuNLs+dsdLFcKmxB=Q@mail.gmail.com> On Tue, Jan 27, 2015 at 3:25 PM, Chris Barker <chris.barker at noaa.gov> wrote: > But one reason to take it off-list is that these very long, if not endless, > circular conversations gives the impression that it really matters which > specif choices are made, and that we will never come to a consensus about > it, so this should not go in the stdlib. I would suggest that, as PEP author, you guide the conversation a bit by asking specific questions that you want to answer in the PEP. Steven's not the only one who's been seeing an awful lot of emails lately; I've just been skimming them, myself. But you could reboot the conversation by starting up some very specific discussion on points that will actually affect the PEP, and then things will hopefully be fresh and interesting again :) ChrisA From chris.barker at noaa.gov Tue Jan 27 07:24:07 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 26 Jan 2015 22:24:07 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150127030800.GQ20517@ando.pearwood.info> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> Message-ID: <CALGmxEJ0S4hfC=MYqAz0PXFuqwV+wyu3jfPGrzkxSKeZL7oGAA@mail.gmail.com> TL;DR -- I can live with, and would indeed be happy with, either a symmetric or asymmetric test -- because most of the use-cases it just doesn't matter. But we have to pick one, so if you're interested -- read on --- On Mon, Jan 26, 2015 at 7:08 PM, Steven D'Aprano <steve at pearwood.info> wrote: > > I'm still unclear about the arguments > > over asymmetric vs symmetric (I suspect, as you alluded to earlier, > > that they reflect a more fundamental problem, which is that there are > > 2 different types of use case with different constraints) so I'd like > > to at least be aware of the content of any discussion... > Actually, I think this is exactly true: If you ask the question: "are these to values close to each-other?" then the symmetric test makes the most sense -- obviously if a is close to b then b is close to a. whereas: If you ask the question: Is this value within a defined relative difference of an expected value? (i.e 10% of b) then you want an asymmetric test -- you clearly are defining "relative" to the known value. However, and I've tried to say this multiple times -- and no one disagreed (or agreed) yet: I don't care much which we use, because IT DOES NOT MATTER in most cases which you choose. Why is that? The most common use cases for this kind of thing are a general check of "is my computed number in the right ballpark?" And I've never seen anyone define "ballpark" to a high degree of precision -- usually you are choosing between say 1e-8 and 1e-9 (more of less 8 or 9 significant figures) This is why the 10% example we keep throwing around is a bit deceiving -- it makes the asymmetry seem far more important than it is. Remember that we are talking about: abs(a-b) <= tol*abs(b) vs abs(a-b) <= tol*abs(a) tol*abs(something) defines the absolute difference that can be tolerated. The difference between the two methods is tol*abs(a-b). In the "is 9 withing 10% of 10" example, that's the difference between "tolerating" .9 or 1 as a difference -- seems pretty significant. But if you have a more realistic tolerance, like 1e-8, then you are talking about a difference in absolute tolerance of around 1e-8 -- tiny. So you'll still get: 9.9999999 is close to 10, but 10 is not close to 9.9999999, but if you tack on even an extra 1e8 on there, you get it close both ways: In [45]: is_close_to(10, 9.99999991) Out[45]: True In [46]: is_close_to(9.99999991, 10) Out[46]: True Same if you go down a bit: In [47]: is_close_to(9.9999998, 10) testing: 9.9999998 10 Out[47]: False In [48]: is_close_to(10, 9.9999998) testing: 10 9.9999998 Out[48]: False So there is this tiny range of values for which is it asymmetric. Yes, it is still asymmetric, but remember that the usual use case is someone choosing between a rel_tolerance of 1e-8 or 1e-9, not 1e-8 or 1.00000001e-8, so within the precision of the specified tolerance -- they are the same. OK -- but we need to choose one (or set a flag for selecting one -- but the point of this is to have something people can just use) So -- there are some use cases where people may want to be testing against a specific value -- is the measured resistance within 1% of the nominal value? (is anyone ever going to write resistor testing code in Python???). In this case, they really want the symmetric test, and there is no way to simulate it with an asymmetric test. Granted, I think the use-case is rare that it would matter, but what is very common is testing a computed value against an expected value -- so the asymmetric case makes more sense there, too, or is at least easier to explain. So what I haven't seen yet is an example use case where you really need the symmetric case -- i.e. it matters that is_close(a,b) is guaranteed to be the same as is_close(b,a). Does anyone have a use-case?? Note: I took a look at the tests for the Statistics module -- as far as I could tell, all but one were comparing a computed value to an expected one -- in fact, I even see: self.assertApproxEqual(actual, expected) Happens to use the same names I used for the parameters ;-) The one exception is testing against math.fsum, where it's testing if two different implementations get (almost) the same result -- that arguable want's a symmetric test, though I can't imagine it would make a real difference: rel_tol set to 1e-16 (by the way, a _very_ small tolerance for a python float! -- this may be where an ULPS check would make sense) And I don't see a tolerance ever specified with more than one significant figure. And I see a lot of 1e-8 (though not all, but any means), so maybe that's a good default. Asymmetry is bad, because it is rather surprising and counter-intuitive > that "x is close to y", but "y is not close to x". It may also be bad in > a practical sense, because people will forget which order they need to > give x and y and will give them in the wrong order. I started off with > an approx_equal function in test_statistics that was symmetric, and I > could never remember which way the arguments went. > My point is that it very rarely matters which order you give them in anyway. So I agree that asymmetry is esthetically "bad", but I'm still looking for a practical example where it matters -- again, for the fairly casual user. Instead, Bruce Dawson recommends using the larger of x and y: > > > https://randomascii.wordpress.com/2012/02/25/comparing-floating-point-numbers-2012-edition/ > > Quote: > > To compare f1 and f2 calculate diff = fabs(f1-f2). If diff is > smaller than n% of max(abs(f1),abs(f2)) then f1 and f2 can be > considered equal. > Sure -- but he then jumps right to the whole ULPS thing -- having not explained why that particular definition is best -- in fact, I'd probably go with: n% of min(abs(f1),abs(f2)) -- it's a bit more rigorous -- this is the Boost "strong" test. But again, these are really subtle differences in results, and if you know your allowed error that accurately, you probably should be doing the ULPS thing anyway. In fact, the only use cases I can imagine, or anyone has brought up, for using a tolerance as high as 1% or 10% is for the case when you are testing against a known value, and the asymmetric case makes more sense. > Time permitting, over the next day or so I'll draw up some diagrams to > show how each of these tactics change what counts as close or not close. > I'm not sure we need a whole lot more explanation (maybe some folks do). But I think we do need one of either: Uses cases for when it's important to have a symmetric test. and/or Pronouncements (from anyone) that s/he "can't live with" one or the other Again -- "can't live with" means you think it's better to have nothing in the std lib. I took the time to write the PEP, and I'd like to see this through -- but we need to pick something -- and any of the three options on the table are fine with me. Three options: - The asymmetric test in the PEP - The Boost "strong" test (max rel error) - The Boost "weak" test (min rel error) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150126/59bd1470/attachment.html> From chris.barker at noaa.gov Tue Jan 27 07:35:01 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 26 Jan 2015 22:35:01 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAPTjJmqFeKvri7jeFU_NAofrih1Z6hieuuNLs+dsdLFcKmxB=Q@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEKZjkz_iRqte=n3y0kQ6FyC5-30-chm_LAcWFY-C1R0cQ@mail.gmail.com> <CAPTjJmqFeKvri7jeFU_NAofrih1Z6hieuuNLs+dsdLFcKmxB=Q@mail.gmail.com> Message-ID: <CALGmxEJ8Y6t0DQMe82gBWxxybEnjvN0dhbNhyQSWuYMU8cQpxw@mail.gmail.com> On Mon, Jan 26, 2015 at 9:30 PM, Chris Angelico <rosuav at gmail.com> wrote: > I would suggest that, as PEP author, you guide the conversation a bit > by asking specific questions that you want to answer in the PEP. > I thought I did that, actually, in that email -- I guess I wasn't very clear. Here they are -- and please express not just your preference, but a clear statement about what would be acceptable or not acceptable. A) Which test do we use: 1) The asymmetric test 2) The "strong" test (minimum relative tolerance) 3) The "weak" test (maximum relative tolerance) B) Do we provide a non-zero default for the absolute tolerance? If so what should the value be? Remember that this serves primarily to provide a check against zero. I think that's it for the technical decisions. I would also appreciate suggestions for paramater names --at least if we go with the asymetric test: "actual" and "expected" is a bit confusing -- I like "expected", but we need something better than "actual" for the don't-know-it's-right one. I also would really appreciate someone working out the details and contributing text for the inclusion of this in unittest.TestCase -- unittest is really not my thing. I've pushed some changes to gitHub (sorry, forgot to push yesterday), and once the dust settles I'll incorporate as many of the suggestions in the PEP text as I can. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150126/449ddf51/attachment-0001.html> From p.f.moore at gmail.com Tue Jan 27 10:13:23 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 27 Jan 2015 09:13:23 +0000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEKZjkz_iRqte=n3y0kQ6FyC5-30-chm_LAcWFY-C1R0cQ@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEKZjkz_iRqte=n3y0kQ6FyC5-30-chm_LAcWFY-C1R0cQ@mail.gmail.com> Message-ID: <CACac1F9zOOfgvRz3VdyLvs26wyz2BDmJ-n2WLuLRNbJQxheadA@mail.gmail.com> On 27 January 2015 at 04:25, Chris Barker <chris.barker at noaa.gov> wrote: > But one reason to take it off-list is that these very long, if not endless, > circular conversations gives the impression that it really matters which > specif choices are made, and that we will never come to a consensus about > it, so this should not go in the stdlib. Hmm. That's a reason for writing a PEP, and getting feedback. If people feel strongly enough about the colour of the bikeshed to vote against the PEP, then it matters, otherwise not. If you believe it's only details by now and people can live with most of the options, maybe it's time to declare the PEP as complete, and post to python-dev for final comments before asking for a pronouncement? > However, I'm pretty sure that we're down to details, that while interesting, > don't really matter -- whether we use a asymmetric or symmetric test, weak > or string version, we'll get something that will work better than == or > assertAlmostEqual, and it will do the right thing in the vast majority of > cases. Personally, I'd agree with this. > I could live with, and indeed be happy with, any of the solutions on the > table. My take from this thread is that most people converged on the > asymmetric option as the better choice, but Steven feels strongly that the > symetric option is the way to go. I don't know if this is a stopper for > anyone, though. > > Is there anyone that could only live with one of the options? > > (by live with, I mean think that we'd be better off with nothing in the > standard lib that one of these options) > > Please speak. Nope, I could manage with any of the options. (Disclaimer: I'm not sure I've ever needed a function like this in practice, so my opinion is of limited relevance. My concerns over details are more about whether I'd find the function intuitive enough that I'd think of using it instead of a quick abs(a-b)<1e-8 in toy examples). > The other issue is whether to have a default that will return True for at > least common uses of comparison to zero. > > - I think it's better to be safe than sorry, and not let folks accidentally > think they have a value close to zero that isn't really. > > - Nathaniel thinks that it's better to provide a default that will give an > answer for is close to zero that will at least work for common cases. > > I could live with what Nathaniel Proposes, and I believe he said he could > live with what I propose -- so this is not a stopper. However, someone's > going to need to come up with what that default value -- part of why I think > it should be zero is that I have no idea if a small-compared-to-one default > is reasonable. As Nathaniel has come up with a lot of real-world cases where comparing to zero is the issue, I think the new function needs to handle that case properly. Deliberately choosing defaults that don't suit a significant use case seems like a mistake. And saying that users need to make an explicit choice each time seems to be to be counter to the whole idea of a function which is designed to be used by people who *don't* have the experience to make such choices... Paul From p.f.moore at gmail.com Tue Jan 27 10:49:20 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 27 Jan 2015 09:49:20 +0000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEJ0S4hfC=MYqAz0PXFuqwV+wyu3jfPGrzkxSKeZL7oGAA@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEJ0S4hfC=MYqAz0PXFuqwV+wyu3jfPGrzkxSKeZL7oGAA@mail.gmail.com> Message-ID: <CACac1F-woshMEuTxcbMFK3ytV9zdcX4re5J8yoL38TQ-Ft8yzQ@mail.gmail.com> On 27 January 2015 at 06:24, Chris Barker <chris.barker at noaa.gov> wrote: > If you ask the question: "are these to values close to each-other?" then the > symmetric test makes the most sense -- obviously if a is close to b then b > is close to a. > > whereas: > > If you ask the question: Is this value within a defined relative difference > of an expected value? (i.e 10% of b) then you want an asymmetric test -- you > clearly are defining "relative" to the known value. > > However, and I've tried to say this multiple times -- and no one disagreed > (or agreed) yet: > > I don't care much which we use, because > > IT DOES NOT MATTER in most cases which you choose. All of that makes sense to me (I'm leaving out the detail of "it does not matter" as I'm sort of trusting you guys with the expertise to tell me that ;-)) But what *does* matter to me is usability and how the behaviour matches people's intuition. Not because the mathematical results will differ, but because it makes it easy for people to *think* about what they are doing, and whether it's OK. I would say that the correct approach is to make the default case as easy to use as possible. For that, a symmetrical are_close(a,b) is a no-brainer IMO. (Of course it has to work when either of a and b is zero). It works either way - if one value is a known "target", or if both values are approximations (e.g. when looking at convergence). Once we have that as a basis, look at how people might want to tweak it: are_close(a, b, within_abs=1e-8) # Within a specific distance of each other (absolute tolerance) are_close(a, b, within_rel=0.1) # Within 10% of each other In the relative case, I'd like "the experts" to decide for me what precisely "within 10% of each other" means (document the details, obviously, but don't bother me with them unless I go looking for them). In either case, I'd be happy to assume that if you change the defaults, you understand the implications (they can be explained in the documentation) such as relative tolerances being unstable near zero. I don't think it's a problem that the default behaviour can't be expressed in terms of explicit settings for the tolerance arguments (it's a wart, and could be annoying, but it's not a showstopper for me - allow setting both explicitly to None to mean "default" if it matters that much). That's it. Anyone wanting to specify both parameters together, or wanting the defaults to still apply "as well as" an explicitly specified tolerance, is deemed an "expert" and should be looking for a more specialised function (or writing their own). Paul From guettliml at thomas-guettler.de Tue Jan 27 11:42:58 2015 From: guettliml at thomas-guettler.de (=?windows-1252?Q?Thomas_G=FCttler?=) Date: Tue, 27 Jan 2015 11:42:58 +0100 Subject: [Python-ideas] datetime: Support infinity. Proposal In-Reply-To: <54C6FF2C.3040703@gmail.com> References: <54C5F9A7.3010209@thomas-guettler.de> <54C6FF2C.3040703@gmail.com> Message-ID: <54C76BB2.5000700@thomas-guettler.de> ... > > And as I said, I believe something like this should be added to the > Python stdlib. If no one has a better implementation I'll start working > on a proposal. Great! At the moment I have no better implementation. We have a dirty work around in our code base, and a solution in Python stdlib would be very good. Up to now we must use Python 2.7, so a backport (like subprocess32) would be needed. Is this possible? Please CC your proposal to me. Regards, Thomas G?ttler From ncoghlan at gmail.com Tue Jan 27 15:28:18 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Jan 2015 00:28:18 +1000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CACac1F-woshMEuTxcbMFK3ytV9zdcX4re5J8yoL38TQ-Ft8yzQ@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEJ0S4hfC=MYqAz0PXFuqwV+wyu3jfPGrzkxSKeZL7oGAA@mail.gmail.com> <CACac1F-woshMEuTxcbMFK3ytV9zdcX4re5J8yoL38TQ-Ft8yzQ@mail.gmail.com> Message-ID: <CADiSq7d0C_6tU5hM7ZEwCEuwZ023NzO_euXYw66m4=fTwc-TKg@mail.gmail.com> On 27 January 2015 at 19:49, Paul Moore <p.f.moore at gmail.com> wrote: > On 27 January 2015 at 06:24, Chris Barker <chris.barker at noaa.gov> wrote: >> If you ask the question: "are these to values close to each-other?" then the >> symmetric test makes the most sense -- obviously if a is close to b then b >> is close to a. >> >> whereas: >> >> If you ask the question: Is this value within a defined relative difference >> of an expected value? (i.e 10% of b) then you want an asymmetric test -- you >> clearly are defining "relative" to the known value. >> >> However, and I've tried to say this multiple times -- and no one disagreed >> (or agreed) yet: >> >> I don't care much which we use, because >> >> IT DOES NOT MATTER in most cases which you choose. > > All of that makes sense to me (I'm leaving out the detail of "it does > not matter" as I'm sort of trusting you guys with the expertise to > tell me that ;-)) > > But what *does* matter to me is usability and how the behaviour > matches people's intuition. Not because the mathematical results will > differ, but because it makes it easy for people to *think* about what > they are doing, and whether it's OK. > > I would say that the correct approach is to make the default case as > easy to use as possible. For that, a symmetrical are_close(a,b) is a > no-brainer IMO. (Of course it has to work when either of a and b is > zero). It works either way - if one value is a known "target", or if > both values are approximations (e.g. when looking at convergence). Translate that into explicit English and I'm not sure a symmetric definition reads more clearly: "a and b are close to each other" "a is close to b" "b is close to a" Given that the "is close to" formulation also simplifies the calculation of a relative tolerance (it's always relative to the right hand operand), it has quite a bit to recommend it. > Once we have that as a basis, look at how people might want to tweak it: > > are_close(a, b, within_abs=1e-8) # Within a specific distance of each > other (absolute tolerance) > are_close(a, b, within_rel=0.1) # Within 10% of each other > > In the relative case, I'd like "the experts" to decide for me what > precisely "within 10% of each other" means (document the details, > obviously, but don't bother me with them unless I go looking for > them). > > In either case, I'd be happy to assume that if you change the > defaults, you understand the implications (they can be explained in > the documentation) such as relative tolerances being unstable near > zero. I don't think it's a problem that the default behaviour can't be > expressed in terms of explicit settings for the tolerance arguments > (it's a wart, and could be annoying, but it's not a showstopper for me > - allow setting both explicitly to None to mean "default" if it > matters that much). With an asymmetric comparison, another alternative would be to have an explicit threshold value for the reference where it switched from relative to absolute tolerance checking. That is: def is_close_to(value, reference, *, error_ratio=1e-8, near_zero_threshold=1e-6, near_zero_tolerance=1e-14): """Check if the given value is close to a reference value In most cases, the two values are close if 'abs(value-reference) < reference*error_ratio' If abs(reference) < near_zero_threshold, or near_zero_threshold is None, the values are close if 'abs(value-reference) < near_zero_tolerance' """ Setting near_zero_threshold to 0 would force a relative comparison (even near zero), while setting it to None would force an absolute one (even far away from zero). If you look at the default values, this is actually a very similar definition to the one Chris has in PEP 485, as the default near zero threshold is the default error ratio multiplied by the default near zero tolerance, although I'm not sure as to the suitability of those numbers. The difference is that this takes the cutoff point between using a relative error definition (to handle the dynamic range issues of a floating point representation) and an absolute error definition (to handle the instability of relative difference near zero) and *gives it a name*, rather than deriving it from a confusing combination of the reference value, the error ratio and the near zero tolerance. > That's it. Anyone wanting to specify both parameters together, or > wanting the defaults to still apply "as well as" an explicitly > specified tolerance, is deemed an "expert" and should be looking for a > more specialised function (or writing their own). I believe breaking out the cutoff point as a separately named parameter makes the algorithm easy enough to explain that restricting it isn't necessary. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Tue Jan 27 16:45:54 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 27 Jan 2015 15:45:54 +0000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CADiSq7d0C_6tU5hM7ZEwCEuwZ023NzO_euXYw66m4=fTwc-TKg@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEJ0S4hfC=MYqAz0PXFuqwV+wyu3jfPGrzkxSKeZL7oGAA@mail.gmail.com> <CACac1F-woshMEuTxcbMFK3ytV9zdcX4re5J8yoL38TQ-Ft8yzQ@mail.gmail.com> <CADiSq7d0C_6tU5hM7ZEwCEuwZ023NzO_euXYw66m4=fTwc-TKg@mail.gmail.com> Message-ID: <CACac1F8YnQQ_HnOx7BEBQ7WkPGqaPJ-FwotqAM+RUXjjVjmd-w@mail.gmail.com> On 27 January 2015 at 14:28, Nick Coghlan <ncoghlan at gmail.com> wrote: > Translate that into explicit English and I'm not sure a symmetric > definition reads more clearly: > > "a and b are close to each other" > "a is close to b" > "b is close to a" However, in programming terms, are_close(a, b) is_close_to(a, b) is_close_to(b, a) the latter two have the "which is the target" issue. And yes, real code will have more obvious argument names. It's not a huge deal, I agree. I'm just saying that the first form takes less mental effort to parse while reading through a block of code. Enough said. It's not a big deal, someone (not me) ultimately needs to make the decision. I've explained my view so I'll stop. > Given that the "is close to" formulation also simplifies the > calculation of a relative tolerance (it's always relative to the right > hand operand), it has quite a bit to recommend it. Agreed. It's a trade-off, and my expectation is that most code will simply use the defaults, so making that read better is a good choice. If you believe that most people will explicitly set a tolerance of some form, the asymmetric choice may well be better. > With an asymmetric comparison, another alternative would be to have an > explicit threshold value for the reference where it switched from > relative to absolute tolerance checking. That is: > > def is_close_to(value, reference, *, error_ratio=1e-8, > near_zero_threshold=1e-6, near_zero_tolerance=1e-14): > """Check if the given value is close to a reference value > > In most cases, the two values are close if > 'abs(value-reference) < reference*error_ratio' > If abs(reference) < near_zero_threshold, or > near_zero_threshold is None, > the values are close if 'abs(value-reference) < near_zero_tolerance' > """ Eep. All I can say is that I never expect to write code where I'd even consider changing the parameters as documented there. I don't think I could understand the implications well enough to trust my judgement. Remember, my intuition hits its limit at "within 1 millionth of a percent of each other" (that's the 1e-8) or "numbers under 1e-6 have to differ by no more than 1e-14" (the other two). And I'd punt on what might happen if both conditions apply. > Setting near_zero_threshold to 0 would force a relative comparison > (even near zero), while setting it to None would force an absolute one > (even far away from zero). The latter choice would make the name "near_zero_tolerance" a pretty odd thing to see... > If you look at the default values, this is actually a very similar > definition to the one Chris has in PEP 485, as the default near zero > threshold is the default error ratio multiplied by the default near > zero tolerance, although I'm not sure as to the suitability of those > numbers. > > The difference is that this takes the cutoff point between using a > relative error definition (to handle the dynamic range issues of a > floating point representation) and an absolute error definition (to > handle the instability of relative difference near zero) and *gives it > a name*, rather than deriving it from a confusing combination of the > reference value, the error ratio and the near zero tolerance. > >> That's it. Anyone wanting to specify both parameters together, or >> wanting the defaults to still apply "as well as" an explicitly >> specified tolerance, is deemed an "expert" and should be looking for a >> more specialised function (or writing their own). > > I believe breaking out the cutoff point as a separately named > parameter makes the algorithm easy enough to explain that restricting > it isn't necessary. I'd love to see the proposed documentation, as I think it would probably read as "complicated stuff, leave well alone" to most people. But I *am* assuming a target audience that currently uses "abs(x-y)<1e-8" [1], and unittest's assertAlmostEqual, and doesn't think they need anything different. The rules are different if the target audience is assumed to know more than that. Paul [1] Someone, it may have been Chris or it may have been someone else, used that snippet, and I've seen 1e-8 turn up elsewhere in books on numerical algorithms. I'm not sure why people choose 1e-8 (precision of a C float?), and how it relates to the 1e-6, 1e-8 and 1e-14 you chose for your definition. It feels like the new function may be a lot stricter than the code people naively (or otherwise) write today. Is that fair, or am I reading too much into some arbitrary numbers? (Note - I won't understand the explanation, I'm happy with just "that's a good point" or "no, the numbers ultimately chosen will be fine" :-)) From ron3200 at gmail.com Tue Jan 27 16:56:28 2015 From: ron3200 at gmail.com (Ron Adam) Date: Tue, 27 Jan 2015 09:56:28 -0600 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150127030800.GQ20517@ando.pearwood.info> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> Message-ID: <ma8cfc$u75$1@ger.gmane.org> On 01/26/2015 09:08 PM, Steven D'Aprano wrote: > Asymmetry is bad, because it is rather surprising and counter-intuitive > that "x is close to y", but "y is not close to x". It may also be bad in > a practical sense, because people will forget which order they need to > give x and y and will give them in the wrong order. I started off with > an approx_equal function in test_statistics that was symmetric, and I > could never remember which way the arguments went. Would it help if it was consistent order wise with other python functions such as is_instance? Ron From p.f.moore at gmail.com Tue Jan 27 17:07:02 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 27 Jan 2015 16:07:02 +0000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <ma8cfc$u75$1@ger.gmane.org> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <ma8cfc$u75$1@ger.gmane.org> Message-ID: <CACac1F_-FOgXbLJZdmXXEmDd2cYF7dHtgwF0x9z4ERa6EZe48g@mail.gmail.com> On 27 January 2015 at 15:56, Ron Adam <ron3200 at gmail.com> wrote: > Would it help if it was consistent order wise with other python functions > such as is_instance? /me tries to remember the order of arguments for isinstance and fails :-) Paul From breamoreboy at yahoo.co.uk Tue Jan 27 17:19:01 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Tue, 27 Jan 2015 16:19:01 +0000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CACac1F_-FOgXbLJZdmXXEmDd2cYF7dHtgwF0x9z4ERa6EZe48g@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <ma8cfc$u75$1@ger.gmane.org> <CACac1F_-FOgXbLJZdmXXEmDd2cYF7dHtgwF0x9z4ERa6EZe48g@mail.gmail.com> Message-ID: <ma8dpu$t7c$1@ger.gmane.org> On 27/01/2015 16:07, Paul Moore wrote: > On 27 January 2015 at 15:56, Ron Adam <ron3200 at gmail.com> wrote: >> Would it help if it was consistent order wise with other python functions >> such as is_instance? > > /me tries to remember the order of arguments for isinstance and fails :-) > Paul > Can you remember how to do this? >>> help(isinstance) Help on built-in function isinstance in module builtins: isinstance(obj, class_or_tuple, /) Return whether an object is an instance of a class or of a subclass thereof. A tuple, as in ``isinstance(x, (A, B, ...))``, may be given as the target to check against. This is equivalent to ``isinstance(x, A) or isinstance(x, B) or ...`` etc. Or do you need help :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From guido at python.org Tue Jan 27 17:20:47 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 27 Jan 2015 08:20:47 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEJ8Y6t0DQMe82gBWxxybEnjvN0dhbNhyQSWuYMU8cQpxw@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEKZjkz_iRqte=n3y0kQ6FyC5-30-chm_LAcWFY-C1R0cQ@mail.gmail.com> <CAPTjJmqFeKvri7jeFU_NAofrih1Z6hieuuNLs+dsdLFcKmxB=Q@mail.gmail.com> <CALGmxEJ8Y6t0DQMe82gBWxxybEnjvN0dhbNhyQSWuYMU8cQpxw@mail.gmail.com> Message-ID: <CAP7+vJ+LdEob5RO2J+D6QjfgB-wT68mDXk3iHDXtqbgXGVjP=w@mail.gmail.com> On Mon, Jan 26, 2015 at 10:35 PM, Chris Barker <chris.barker at noaa.gov> wrote: > On Mon, Jan 26, 2015 at 9:30 PM, Chris Angelico <rosuav at gmail.com> wrote: > >> I would suggest that, as PEP author, you guide the conversation a bit >> by asking specific questions that you want to answer in the PEP. >> > > I thought I did that, actually, in that email -- I guess I wasn't very > clear. > > Here they are -- and please express not just your preference, but a clear > statement about what would be acceptable or not acceptable. > > A) Which test do we use: > 1) The asymmetric test > 2) The "strong" test (minimum relative tolerance) > 3) The "weak" test (maximum relative tolerance) > The problem with this question is that, while it's easy to come up with examples where it may matter (e.g. 95 is within 5% of 100, but 100 is not within %5 of 95), in practice the tolerance is more likely to be 1e-8, in which case it doesn't matter. > B) Do we provide a non-zero default for the absolute tolerance? If so what > should the value be? Remember that this serves primarily to provide a check > against zero. > It feels like absolute tolerance is a completely different test. And it is a much simpler test for which w don't need a helper function -- it's just abs(x) < tolerance. When does a program need *both* absolute and relative tolerance in a single test? > I think that's it for the technical decisions. > > I would also appreciate suggestions for paramater names --at least if we > go with the asymetric test: "actual" and "expected" is a bit confusing -- I > like "expected", but we need something better than "actual" for the > don't-know-it's-right one. > > > I also would really appreciate someone working out the details and > contributing text for the inclusion of this in unittest.TestCase -- > unittest is really not my thing. > > I've pushed some changes to gitHub (sorry, forgot to push yesterday), and > once the dust settles I'll incorporate as many of the suggestions in the > PEP text as I can. > I still think this is better off as a recipe than as a helper function. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150127/d42832c5/attachment.html> From p.f.moore at gmail.com Tue Jan 27 17:31:26 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 27 Jan 2015 16:31:26 +0000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <ma8dpu$t7c$1@ger.gmane.org> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <ma8cfc$u75$1@ger.gmane.org> <CACac1F_-FOgXbLJZdmXXEmDd2cYF7dHtgwF0x9z4ERa6EZe48g@mail.gmail.com> <ma8dpu$t7c$1@ger.gmane.org> Message-ID: <CACac1F8GornWYky+OA0-cKa31eMjagTgF84TUoMua0QsXyG6Qw@mail.gmail.com> On 27 January 2015 at 16:19, Mark Lawrence <breamoreboy at yahoo.co.uk> wrote: > On 27/01/2015 16:07, Paul Moore wrote: >> >> On 27 January 2015 at 15:56, Ron Adam <ron3200 at gmail.com> wrote: >>> >>> Would it help if it was consistent order wise with other python functions >>> such as is_instance? >> >> >> /me tries to remember the order of arguments for isinstance and fails :-) >> Paul >> > > Can you remember how to do this? > >>>> help(isinstance) Of course, but (a) it was a joke, and (b) I'm not always at a Python prompt when reading (or writing!) Python code. Apologies for being facetious, though. Paul From chris.barker at noaa.gov Tue Jan 27 18:07:16 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 27 Jan 2015 09:07:16 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAP7+vJ+LdEob5RO2J+D6QjfgB-wT68mDXk3iHDXtqbgXGVjP=w@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEKZjkz_iRqte=n3y0kQ6FyC5-30-chm_LAcWFY-C1R0cQ@mail.gmail.com> <CAPTjJmqFeKvri7jeFU_NAofrih1Z6hieuuNLs+dsdLFcKmxB=Q@mail.gmail.com> <CALGmxEJ8Y6t0DQMe82gBWxxybEnjvN0dhbNhyQSWuYMU8cQpxw@mail.gmail.com> <CAP7+vJ+LdEob5RO2J+D6QjfgB-wT68mDXk3iHDXtqbgXGVjP=w@mail.gmail.com> Message-ID: <CALGmxE+gi=KLQU8ggBafMw20dB6bfMrXHEmdxX8QNFRwKAC_MA@mail.gmail.com> On Tue, Jan 27, 2015 at 8:20 AM, Guido van Rossum <guido at python.org> wrote: > A) Which test do we use: >> 1) The asymmetric test >> 2) The "strong" test (minimum relative tolerance) >> 3) The "weak" test (maximum relative tolerance) >> > > The problem with this question is that, while it's easy to come up with > examples where it may matter (e.g. 95 is within 5% of 100, but 100 is not > within %5 of 95), in practice the tolerance is more likely to be 1e-8, in > which case it doesn't matter. > Exactly why I'm happy with any of them. I'm trying to suss out whether anyone else has a reason to reject one or the others. If no one does, then we can just pick one. > B) Do we provide a non-zero default for the absolute tolerance? If so what >> should the value be? Remember that this serves primarily to provide a check >> against zero. >> > > It feels like absolute tolerance is a completely different test. And it is > a much simpler test for which w don't need a helper function -- it's just > abs(x) < tolerance. > > When does a program need *both* absolute and relative tolerance in a > single test? > Because we want it to be able to do something sane when comparing to zero -- the abs_tolerance allows you to set a minimum tolerance that will do something reasonable near zero (we could use a zero_tolerance, as Nathaniel has suggested, instead, but that creates the incontinuity that , for example, 1e-12 is close to zero, but it is not close to 1e-100 -- I think that's a bad idea for a call with the same arguments). I spend a good while thinking about this and playing with it, and it became clear to me that this is the best way to go for a not-to-surprising result. And it's consistent with what numpy and Steven's statistics test code does. Still TBD is what the default should be, though. > I still think this is better off as a recipe than as a helper function. >> > Are you prepared to reject the PEP? I'd prefer to give it this one last shot at determining if there really is no way to get consensus on a good-enough solution. I suspect there is a lot of bike shedding here -- people have ideas about what is best, and want to understand and talk about it, but that doesn't mean that they wouldn't rather see something else that nothing. -- that's certainly the case for me (both the bike shedding and the desire to see something ;-) ) Evidence: The numpy version has its faults -- but it's widely used. assertAlmost Equal has even more faults (limitations, anyway) but it's also widely used. Boost has something in it, even though it's a one-liner. Clearly this is a useful functionality to have available. ChrisA is right -- I have not done a good job at steering the group toward a consensus. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150127/f42cdb64/attachment-0001.html> From schesis at gmail.com Tue Jan 27 18:42:27 2015 From: schesis at gmail.com (Zero Piraeus) Date: Tue, 27 Jan 2015 14:42:27 -0300 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEJ0S4hfC=MYqAz0PXFuqwV+wyu3jfPGrzkxSKeZL7oGAA@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEJ0S4hfC=MYqAz0PXFuqwV+wyu3jfPGrzkxSKeZL7oGAA@mail.gmail.com> Message-ID: <20150127174227.GA14271@piedra> : Disclaimer: I haven't read all of this thread, and what I have read I've sometimes skimmed (and apparently some of the discusion was offlist anyway). On Mon, Jan 26, 2015 at 10:24:07PM -0800, Chris Barker wrote: > > So what I haven't seen yet is an example use case where you really > need the symmetric case -- i.e. it matters that is_close(a,b) is > guaranteed to be the same as is_close(b,a). > > Does anyone have a use-case?? Once this exists, at some point somebody's going to write: def near_miss(data, tol): pairs = itertools.combinations(data.values(), 2) return any(is_close(a, b, tol) for a, b in pairs) and then (if it's asymmetric) be very surprised when this: example = {'A': 0.01, 'B': 2.34, 'C': 5.67, 'D': 5.68, 'E': 9.99} near_miss(example, 1/568) returns True half the time and False the other half. That's going to be a really nasty heisenbug when it crops up in real code, especially since it's only going to change for each invocation of the interpreter. I don't really buy the idea that it'll almost always be used with tolerances of 1e-8 etc. If it goes in the stdlib, it'll be used in ways no-one here anticipates (and which are not, intuitively, "wrong"). -[]z. -- Zero Piraeus: absit invidia http://etiol.net/pubkey.asc From alexander.belopolsky at gmail.com Tue Jan 27 18:57:03 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 27 Jan 2015 12:57:03 -0500 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAP7+vJ+LdEob5RO2J+D6QjfgB-wT68mDXk3iHDXtqbgXGVjP=w@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEKZjkz_iRqte=n3y0kQ6FyC5-30-chm_LAcWFY-C1R0cQ@mail.gmail.com> <CAPTjJmqFeKvri7jeFU_NAofrih1Z6hieuuNLs+dsdLFcKmxB=Q@mail.gmail.com> <CALGmxEJ8Y6t0DQMe82gBWxxybEnjvN0dhbNhyQSWuYMU8cQpxw@mail.gmail.com> <CAP7+vJ+LdEob5RO2J+D6QjfgB-wT68mDXk3iHDXtqbgXGVjP=w@mail.gmail.com> Message-ID: <CAP7h-xZZQc5pGqGdQjZojCd-FZke9+eY9WsdtUZukQuqsgg=vg@mail.gmail.com> On Tue, Jan 27, 2015 at 11:20 AM, Guido van Rossum <guido at python.org> wrote: > It feels like absolute tolerance is a completely different test. And it is > a much simpler test for which w don't need a helper function -- it's just > abs(x) < tolerance. +1 Just as a point of reference, APL and its derivatives use tolerant comparison in the default equal (=) operator. The definition that they use for finite x and y is simply x = y <=> abs(x-y) <= tol * max(abs(x), abs(y)) The tolerance can be adjusted globally or in some languages (such as J [1]) in the expression using additional syntax. In J, the default tolerance is 2**-44, which is about 5.7e-14. APL restricts the range of tolerance values to 0 through 2**-32. I would be +0 on adding something like def tolerant_equal(x, y, tol=2**-44): return abs(x-y) <= tol * max(abs(x), abs(y)) (name subject to bikeshedding) to math, but -1 on anything more complicated. I would rather see if tolerant_equal(x, y) or abs(x-y) <= 1e-10: .. than if tolerant_equal(x, y, atol=1e-10): .. [1] http://www.jsoftware.com/help/dictionary/d000.htm [2] http://www.dyalog.com/uploads/documents/Papers/tolerant_comparison/tolerant_comparison.htm -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150127/d403a04c/attachment.html> From njs at pobox.com Tue Jan 27 19:05:02 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 27 Jan 2015 18:05:02 +0000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAP7+vJ+LdEob5RO2J+D6QjfgB-wT68mDXk3iHDXtqbgXGVjP=w@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEKZjkz_iRqte=n3y0kQ6FyC5-30-chm_LAcWFY-C1R0cQ@mail.gmail.com> <CAPTjJmqFeKvri7jeFU_NAofrih1Z6hieuuNLs+dsdLFcKmxB=Q@mail.gmail.com> <CALGmxEJ8Y6t0DQMe82gBWxxybEnjvN0dhbNhyQSWuYMU8cQpxw@mail.gmail.com> <CAP7+vJ+LdEob5RO2J+D6QjfgB-wT68mDXk3iHDXtqbgXGVjP=w@mail.gmail.com> Message-ID: <CAPJVwB=aOk-ZaSE4Um=GyBhyUabwOPhgs-BKV+4tvxEfY4g-SQ@mail.gmail.com> On Tue, Jan 27, 2015 at 4:20 PM, Guido van Rossum <guido at python.org> wrote: > When does a program need *both* absolute and relative tolerance in a single > test? It's reasonably common in numpy, where the equivalent function is vectorized to do multiple checks at once, e.g. here are some tests that mix zeros and non-zeros in a single call: https://github.com/mnick/scikit-tensor/blob/b067ed02ef4b1e3f1cb24004b2f20af5bb55c27a/sktensor/tests/test_utils.py#L12 https://github.com/ricklupton/whales/blob/6bc722c1e2dbadb494fa3d4e83e1cb64154a22e7/tests/test_utils.py#L8 The equivalent would be writing assert all([math.is_close_to(a, e) for (a, e) in zip(actual, expected)]) where 'expected' has a mix of zeros and non-zeros. >> I think that's it for the technical decisions. >> >> I would also appreciate suggestions for paramater names --at least if we >> go with the asymetric test: "actual" and "expected" is a bit confusing -- I >> like "expected", but we need something better than "actual" for the >> don't-know-it's-right one. >> >> >> I also would really appreciate someone working out the details and >> contributing text for the inclusion of this in unittest.TestCase -- unittest >> is really not my thing. >> >> I've pushed some changes to gitHub (sorry, forgot to push yesterday), and >> once the dust settles I'll incorporate as many of the suggestions in the PEP >> text as I can. > > > I still think this is better off as a recipe than as a helper function. I don't find it super compelling either way -- at the end of the day, if 'math' doesn't provide this then many people will use libraries that will or else write their own. I guess I don't have a good sense of what the audience for 'math' is these days -- I'm sure it has one, but aside from tiny one-off usages I'm not sure what it is. None of the production numerical code I see even bothers importing it. It's possible that it largely serves a kind of pedagogical role? high schoolers learning to program etc.? in which case there might be some more benefit in having a well-documented discussion of the issues here in the stdlib docs. The most compelling argument I see is that if we care about unittest, then it would be good to have a better alternative to assertAlmostEqual. (...I don't know anyone who uses the unittest API directly these days, though!) -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From guido at python.org Tue Jan 27 19:37:38 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 27 Jan 2015 10:37:38 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxE+gi=KLQU8ggBafMw20dB6bfMrXHEmdxX8QNFRwKAC_MA@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEKZjkz_iRqte=n3y0kQ6FyC5-30-chm_LAcWFY-C1R0cQ@mail.gmail.com> <CAPTjJmqFeKvri7jeFU_NAofrih1Z6hieuuNLs+dsdLFcKmxB=Q@mail.gmail.com> <CALGmxEJ8Y6t0DQMe82gBWxxybEnjvN0dhbNhyQSWuYMU8cQpxw@mail.gmail.com> <CAP7+vJ+LdEob5RO2J+D6QjfgB-wT68mDXk3iHDXtqbgXGVjP=w@mail.gmail.com> <CALGmxE+gi=KLQU8ggBafMw20dB6bfMrXHEmdxX8QNFRwKAC_MA@mail.gmail.com> Message-ID: <CAP7+vJ+8UrrHiGs7=Mogw2i6s0JS94UvfSSD-ZMy2xf2GrdU1Q@mail.gmail.com> On Tue, Jan 27, 2015 at 9:07 AM, Chris Barker <chris.barker at noaa.gov> wrote: > On Tue, Jan 27, 2015 at 8:20 AM, Guido van Rossum <guido at python.org> > wrote: > >> A) Which test do we use: >>> 1) The asymmetric test >>> 2) The "strong" test (minimum relative tolerance) >>> 3) The "weak" test (maximum relative tolerance) >>> >> >> The problem with this question is that, while it's easy to come up with >> examples where it may matter (e.g. 95 is within 5% of 100, but 100 is not >> within %5 of 95), in practice the tolerance is more likely to be 1e-8, in >> which case it doesn't matter. >> > > Exactly why I'm happy with any of them. I'm trying to suss out whether > anyone else has a reason to reject one or the others. If no one does, then > we can just pick one. > By now can't you summarize the reasons that others have brought up? > > > >> B) Do we provide a non-zero default for the absolute tolerance? If so >>> what should the value be? Remember that this serves primarily to provide a >>> check against zero. >>> >> >> It feels like absolute tolerance is a completely different test. And it >> is a much simpler test for which w don't need a helper function -- it's >> just abs(x) < tolerance. >> >> When does a program need *both* absolute and relative tolerance in a >> single test? >> > > Because we want it to be able to do something sane when comparing to zero > -- the abs_tolerance allows you to set a minimum tolerance that will do > something reasonable near zero (we could use a zero_tolerance, as Nathaniel > has suggested, instead, but that creates the incontinuity that , for > example, 1e-12 is close to zero, but it is not close to 1e-100 -- I think > that's a bad idea for a call with the same arguments). I spend a good while > thinking about this and playing with it, and it became clear to me that > this is the best way to go for a not-to-surprising result. And it's > consistent with what numpy and Steven's statistics test code does. > I don't think you can have this always be sane. For someone who for whatever reason is manipulating quantities that are in the range of 1e-100, 1e-12 is about as large as infinity. I think my reasoning comes down to the same rule I often use to decide whether we need one function or two -- if in every use case you always know whether you need version A or version B, then it's better to have two functions rather than a single one with a flag to request A or B. And isn't it the case that whenever you are comparing to zero, you *know* that you are comparing to zero, and you *must* specify an absolute tolerance (otherwise it's not a use case at all)? IIUC, numpy doesn't design APIs this way. They like to have swiss army knives that can do lots of different things, with lots of flags to request variant behavior. (I learned this from the discussion about linspace().) But IMO that goes back to an earlier, murkier tradition -- I recall using Fortran plot functions in the '80s that had 17 parameters. The reason for that style was that there was no concept of modules or packages, and hence there was only one namespace, shared between all possible libraries. So a new library would claim only a single name in the namespace and hook all sorts of functionality onto that single name. We don't have that problem in Python and hence I prefer clarity in functionality -- different functions for different behaviors, basically. (Maybe this should become the 20th line of the Zen of Python. :-) > Still TBD is what the default should be, though. > > >> I still think this is better off as a recipe than as a helper function. >>> >> > Are you prepared to reject the PEP? I'd prefer to give it this one last > shot at determining if there really is no way to get consensus on a > good-enough solution. I suspect there is a lot of bike shedding here -- > people have ideas about what is best, and want to understand and talk about > it, but that doesn't mean that they wouldn't rather see something else that > nothing. -- that's certainly the case for me (both the bike shedding and > the desire to see something ;-) ) > Yes, I am prepared to reject the PEP. In fact any PEP is rejected by default if no consensus is obtained. > > Evidence: The numpy version has its faults -- but it's widely used. > assertAlmost Equal has even more faults (limitations, anyway) but it's also > widely used. Boost has something in it, even though it's a one-liner. > Clearly this is a useful functionality to have available. > IIUC Boost's approach is better than numpy. It has separate functions for is_close (relative) and is_small (absolute), because they are separate use cases. Its is_close uses the symmetric version (though there is a flag to choose between weak and strong, which sounds like overkill). > ChrisA is right -- I have not done a good job at steering the group toward > a consensus. > It's not too late! -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150127/553c2bcf/attachment-0001.html> From ron3200 at gmail.com Tue Jan 27 21:51:19 2015 From: ron3200 at gmail.com (Ron Adam) Date: Tue, 27 Jan 2015 14:51:19 -0600 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CACac1F_-FOgXbLJZdmXXEmDd2cYF7dHtgwF0x9z4ERa6EZe48g@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <ma8cfc$u75$1@ger.gmane.org> <CACac1F_-FOgXbLJZdmXXEmDd2cYF7dHtgwF0x9z4ERa6EZe48g@mail.gmail.com> Message-ID: <ma8to8$5gu$1@ger.gmane.org> On 01/27/2015 10:07 AM, Paul Moore wrote: > /me tries to remember the order of arguments for isinstance and fails:-) I saw where you said this was a bit of a joke. But yes, sometimes it's not all that obvious what order things should be in. Which was your point, and people new to python do have to look up what order the isinstance arguments are in. I'm just wondering if there is a rule of thumb that we can apply that works for other python functions. Cheers, Ron From chris.barker at noaa.gov Tue Jan 27 22:10:29 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 27 Jan 2015 13:10:29 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAPJVwB=aOk-ZaSE4Um=GyBhyUabwOPhgs-BKV+4tvxEfY4g-SQ@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEKZjkz_iRqte=n3y0kQ6FyC5-30-chm_LAcWFY-C1R0cQ@mail.gmail.com> <CAPTjJmqFeKvri7jeFU_NAofrih1Z6hieuuNLs+dsdLFcKmxB=Q@mail.gmail.com> <CALGmxEJ8Y6t0DQMe82gBWxxybEnjvN0dhbNhyQSWuYMU8cQpxw@mail.gmail.com> <CAP7+vJ+LdEob5RO2J+D6QjfgB-wT68mDXk3iHDXtqbgXGVjP=w@mail.gmail.com> <CAPJVwB=aOk-ZaSE4Um=GyBhyUabwOPhgs-BKV+4tvxEfY4g-SQ@mail.gmail.com> Message-ID: <CALGmxE+8A3GAaCuyqpEeJp0daFbR-7=bRiH64X=0fYjpD_pktQ@mail.gmail.com> On Tue, Jan 27, 2015 at 10:05 AM, Nathaniel Smith <njs at pobox.com> wrote: > On Tue, Jan 27, 2015 at 4:20 PM, Guido van Rossum <guido at python.org> > wrote: > > When does a program need *both* absolute and relative tolerance in a > single > > test? > > It's reasonably common in numpy, where the equivalent function is > vectorized to do multiple checks at once, exactly -- as do a buch of the the unittest assertXXXX methods or if folks write their own equivalent in a comprehension. > if 'math' doesn't provide this then many people will use libraries > that will or else write their own. sure -- that's what been done for ages... you could say that about anything new being proposed for the stdlib. > I guess I don't have a good sense > of what the audience for 'math' is these days -- I'm sure it has one, > but aside from tiny one-off usages I'm not sure what it is. None of > the production numerical code I see even bothers importing it. Well, I'm a heavy numpy user as well, but I still use the math module when I have something simple to calculate (actually, not so much simple as small -- if I'm not working with a lot of numbers) or if I don't want the numpy dependency. A lot of people do at least some math with python -- I have no idea how many. And the statistics package was recently added -- I would have thought that would be next to useless without numpy, but shows you what I know. > It's > possible that it largely serves a kind of pedagogical role? high > schoolers learning to program etc.? anyone learning python may need a bit of math -- there's some pretty basic stuff in there. The most compelling argument I see is that if we care about unittest, > then it would be good to have a better alternative to > assertAlmostEqual. (...I don't know anyone who uses the unittest API > directly these days, though!) > I agree -- adding it to unitest would be a very good idea. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150127/d0cd8650/attachment.html> From chris.barker at noaa.gov Tue Jan 27 22:18:08 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 27 Jan 2015 13:18:08 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAP7+vJ+8UrrHiGs7=Mogw2i6s0JS94UvfSSD-ZMy2xf2GrdU1Q@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEKZjkz_iRqte=n3y0kQ6FyC5-30-chm_LAcWFY-C1R0cQ@mail.gmail.com> <CAPTjJmqFeKvri7jeFU_NAofrih1Z6hieuuNLs+dsdLFcKmxB=Q@mail.gmail.com> <CALGmxEJ8Y6t0DQMe82gBWxxybEnjvN0dhbNhyQSWuYMU8cQpxw@mail.gmail.com> <CAP7+vJ+LdEob5RO2J+D6QjfgB-wT68mDXk3iHDXtqbgXGVjP=w@mail.gmail.com> <CALGmxE+gi=KLQU8ggBafMw20dB6bfMrXHEmdxX8QNFRwKAC_MA@mail.gmail.com> <CAP7+vJ+8UrrHiGs7=Mogw2i6s0JS94UvfSSD-ZMy2xf2GrdU1Q@mail.gmail.com> Message-ID: <CALGmxELxomDNmdV7-B9=LSQrfceVvc9b1n_p0yrR08ffN65r4Q@mail.gmail.com> On Tue, Jan 27, 2015 at 10:37 AM, Guido van Rossum <guido at python.org> wrote: > By now can't you summarize the reasons that others have brought up? > >> >> I can try ;-) --probably not until this evening though. > Because we want it to be able to do something sane when comparing to zero -- <snip> I don't think you can have this always be sane. For someone who for > whatever reason is manipulating quantities that are in the range of 1e-100, > 1e-12 is about as large as infinity. > Exactly why I favor having the abs_tolerance default to zero. > I think my reasoning comes down to the same rule I often use to decide > whether we need one function or two -- if in every use case you always know > whether you need version A or version B, then it's better to have two > functions rather than a single one with a flag to request A or B. > > And isn't it the case that whenever you are comparing to zero, you *know* > that you are comparing to zero, and you *must* specify an absolute > tolerance (otherwise it's not a use case at all)? > I really appreciate this API design approach, and in this case I started out with that idea. But I think this is likely to be used where you need to test a bunch of values with single function/set of parameters. In TestCase.assertIsCloseTo, as well as home grown loops and comprehensions. IIUC, numpy doesn't design APIs this way. > I'm not sure numpy's API is exactly designed at all ;-) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150127/dd42b0a3/attachment.html> From chris.barker at noaa.gov Tue Jan 27 22:31:25 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 27 Jan 2015 13:31:25 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150127174227.GA14271@piedra> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEJ0S4hfC=MYqAz0PXFuqwV+wyu3jfPGrzkxSKeZL7oGAA@mail.gmail.com> <20150127174227.GA14271@piedra> Message-ID: <CALGmxEKan04wZyvUGJpe72+7yQkTUriKMyVVWfG-3t_g+SKVHA@mail.gmail.com> On Tue, Jan 27, 2015 at 9:42 AM, Zero Piraeus <schesis at gmail.com> wrote: > Disclaimer: I haven't read all of this thread, and what I have read I've > sometimes skimmed (and apparently some of the discussion was off list > anyway). Nothing substantial was off-list -- thank god, there's been plenty on list! > Once this exists, at some point somebody's going to write: > > def near_miss(data, tol): > pairs = itertools.combinations(data.values(), 2) > return any(is_close(a, b, tol) for a, b in pairs) > > and then (if it's asymmetric) be very surprised when this: > > example = {'A': 0.01, 'B': 2.34, 'C': 5.67, 'D': 5.68, 'E': 9.99} > > near_miss(example, 1/568) > > returns True half the time and False the other half. huh? itertools combinations returns a deterministic result, yes? But that is indeed a case where the question really is "are these two values close to each-other", with no implied order. IS that a lilkey real world use-case? I have no idea. > That's going to be > a really nasty heisenbug when it crops up in real code, especially since > it's only going to change for each invocation of the interpreter. > I still don't follow you here -- the asymmetric test is order dependent, it's not random. > I don't really buy the idea that it'll almost always be used with > tolerances of 1e-8 etc. If it goes in the stdlib, it'll be used in ways > no-one here anticipates (and which are not, intuitively, "wrong"). > Indeed -- absolutely the case -- but all we can do is document the behavior. On the face of it, a symmetric approach seems less surprising, but really the surprize is simply different: In the asymmetric case, the results may depend on the order of the arguments. In the symmetric case, the results may depend on whether the actual value is less than or greater than the expected value. I hark back to the point Steven made: what really matter is the difference between the values. if someone is asking: Is this value within 10% of the value 10 -- they are expecting that the difference used will be 10% of ten, or 1.0 -- so any value between 9 and 11 is "close". But with a symmetric test -- the actual difference accepted will be a function of whether the tested value is less than or greater than the expected value -- that could be equally or more surprising than the asymmetry. I'm still looking for a case where a user would likely pass the same value into the function in a different order -- wouldn't s/he pick an order (maybe arbitratily) and use that? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150127/8d5cf2e0/attachment-0001.html> From p.f.moore at gmail.com Tue Jan 27 22:52:16 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 27 Jan 2015 21:52:16 +0000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEKan04wZyvUGJpe72+7yQkTUriKMyVVWfG-3t_g+SKVHA@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEJ0S4hfC=MYqAz0PXFuqwV+wyu3jfPGrzkxSKeZL7oGAA@mail.gmail.com> <20150127174227.GA14271@piedra> <CALGmxEKan04wZyvUGJpe72+7yQkTUriKMyVVWfG-3t_g+SKVHA@mail.gmail.com> Message-ID: <CACac1F8OdXrLbWhvnFZWV_2dT8Zwrh1tss3oLxArk0J4v-H2bA@mail.gmail.com> On 27 January 2015 at 21:31, Chris Barker <chris.barker at noaa.gov> wrote: >> That's going to be >> a really nasty heisenbug when it crops up in real code, especially since >> it's only going to change for each invocation of the interpreter. > > > I still don't follow you here -- the asymmetric test is order dependent, > it's not random. The order of dictionary iteration is arbitrary. Thanks to hash randomization, the order will differ each time the program is run. Paul From guido at python.org Tue Jan 27 23:00:17 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 27 Jan 2015 14:00:17 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxELxomDNmdV7-B9=LSQrfceVvc9b1n_p0yrR08ffN65r4Q@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEKZjkz_iRqte=n3y0kQ6FyC5-30-chm_LAcWFY-C1R0cQ@mail.gmail.com> <CAPTjJmqFeKvri7jeFU_NAofrih1Z6hieuuNLs+dsdLFcKmxB=Q@mail.gmail.com> <CALGmxEJ8Y6t0DQMe82gBWxxybEnjvN0dhbNhyQSWuYMU8cQpxw@mail.gmail.com> <CAP7+vJ+LdEob5RO2J+D6QjfgB-wT68mDXk3iHDXtqbgXGVjP=w@mail.gmail.com> <CALGmxE+gi=KLQU8ggBafMw20dB6bfMrXHEmdxX8QNFRwKAC_MA@mail.gmail.com> <CAP7+vJ+8UrrHiGs7=Mogw2i6s0JS94UvfSSD-ZMy2xf2GrdU1Q@mail.gmail.com> <CALGmxELxomDNmdV7-B9=LSQrfceVvc9b1n_p0yrR08ffN65r4Q@mail.gmail.com> Message-ID: <CAP7+vJJbsFWQPTxDfSBEdREVyciJiKDWziF82y1HT4-DFwnUVQ@mail.gmail.com> On Tue, Jan 27, 2015 at 1:18 PM, Chris Barker <chris.barker at noaa.gov> wrote: > On Tue, Jan 27, 2015 at 10:37 AM, Guido van Rossum <guido at python.org> > wrote: > I think my reasoning comes down to the same rule I often use to decide >> whether we need one function or two -- if in every use case you always know >> whether you need version A or version B, then it's better to have two >> functions rather than a single one with a flag to request A or B. >> >> And isn't it the case that whenever you are comparing to zero, you *know* >> that you are comparing to zero, and you *must* specify an absolute >> tolerance (otherwise it's not a use case at all)? >> > > I really appreciate this API design approach, and in this case I started > out with that idea. But I think this is likely to be used where you need to > test a bunch of values with single function/set of parameters. In > TestCase.assertIsCloseTo, as well as home grown loops and comprehensions. > I assume you mean assertNotAlmostEqual <https://hg.python.org/cpython/file/94d8524086bd/Lib/unittest/case.py#l525>. This is actually the same misguided two-for-one design. You *must* specify exactly one of delta or places, and the code takes a different path. Also, it looks like both are actually absolute tolerance. So what argument are you making here? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150127/d3adaa40/attachment.html> From chris.barker at noaa.gov Tue Jan 27 23:00:05 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 27 Jan 2015 14:00:05 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CACac1F8OdXrLbWhvnFZWV_2dT8Zwrh1tss3oLxArk0J4v-H2bA@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEJ0S4hfC=MYqAz0PXFuqwV+wyu3jfPGrzkxSKeZL7oGAA@mail.gmail.com> <20150127174227.GA14271@piedra> <CALGmxEKan04wZyvUGJpe72+7yQkTUriKMyVVWfG-3t_g+SKVHA@mail.gmail.com> <CACac1F8OdXrLbWhvnFZWV_2dT8Zwrh1tss3oLxArk0J4v-H2bA@mail.gmail.com> Message-ID: <CALGmxE+uLtEQ-RwCVfe3Lqv__JF76SXvRP3po8Sp96FoKXfPhQ@mail.gmail.com> On Tue, Jan 27, 2015 at 1:52 PM, Paul Moore <p.f.moore at gmail.com> wrote: > The order of dictionary iteration is arbitrary. Thanks to hash > randomization, the order will differ each time the program is run. > duh, of course!. But question remains -- legitimate use case, or pathological example? This kind of points to a is_close_to() and a are are_close() function (better than a flag, yes?) but that really does seem like overkill. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150127/ffda588a/attachment.html> From chris.barker at noaa.gov Tue Jan 27 23:14:14 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 27 Jan 2015 14:14:14 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAP7+vJJbsFWQPTxDfSBEdREVyciJiKDWziF82y1HT4-DFwnUVQ@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEKZjkz_iRqte=n3y0kQ6FyC5-30-chm_LAcWFY-C1R0cQ@mail.gmail.com> <CAPTjJmqFeKvri7jeFU_NAofrih1Z6hieuuNLs+dsdLFcKmxB=Q@mail.gmail.com> <CALGmxEJ8Y6t0DQMe82gBWxxybEnjvN0dhbNhyQSWuYMU8cQpxw@mail.gmail.com> <CAP7+vJ+LdEob5RO2J+D6QjfgB-wT68mDXk3iHDXtqbgXGVjP=w@mail.gmail.com> <CALGmxE+gi=KLQU8ggBafMw20dB6bfMrXHEmdxX8QNFRwKAC_MA@mail.gmail.com> <CAP7+vJ+8UrrHiGs7=Mogw2i6s0JS94UvfSSD-ZMy2xf2GrdU1Q@mail.gmail.com> <CALGmxELxomDNmdV7-B9=LSQrfceVvc9b1n_p0yrR08ffN65r4Q@mail.gmail.com> <CAP7+vJJbsFWQPTxDfSBEdREVyciJiKDWziF82y1HT4-DFwnUVQ@mail.gmail.com> Message-ID: <CALGmxEKqu_L=FBCDaAXF1v1-K1-v6GC=E84bqTSsFaBWKR6rCw@mail.gmail.com> On Tue, Jan 27, 2015 at 2:00 PM, Guido van Rossum <guido at python.org> wrote: > I really appreciate this API design approach, and in this case I started >> out with that idea. But I think this is likely to be used where you need to >> test a bunch of values with single function/set of parameters. In >> TestCase.assertIsCloseTo, as well as home grown loops and comprehensions. >> > > I assume you mean assertNotAlmostEqual > <https://hg.python.org/cpython/file/94d8524086bd/Lib/unittest/case.py#l525>. > > Sorry -- I meant a Hypothetical assertIsCLoseTo -- which would wrap the proposed is_close_to(), but apply it to a entire sequence, like assertNotAlmostEqual <https://hg.python.org/cpython/file/94d8524086bd/Lib/unittest/case.py#l525>. does. > This is actually the same misguided two-for-one design. You *must* specify > exactly one of delta or places, and the code takes a different path. Also, > it looks like both are actually absolute tolerance. > Yes, both are an absolute tolerance -- which , I think makes it a slightly less misguided design -- you aren't selecting two different functionalities, just two ways to spell the tolerance. So what argument are you making here? The point was that if the user applies one set of parameters to a sequence of values that may have some zeros in it, -- we need it in one function. Nathaniel has found that to be a fairly common use-case for the numpy allclose() -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150127/79a85e65/attachment-0001.html> From guido at python.org Tue Jan 27 23:35:50 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 27 Jan 2015 14:35:50 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEKqu_L=FBCDaAXF1v1-K1-v6GC=E84bqTSsFaBWKR6rCw@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEKZjkz_iRqte=n3y0kQ6FyC5-30-chm_LAcWFY-C1R0cQ@mail.gmail.com> <CAPTjJmqFeKvri7jeFU_NAofrih1Z6hieuuNLs+dsdLFcKmxB=Q@mail.gmail.com> <CALGmxEJ8Y6t0DQMe82gBWxxybEnjvN0dhbNhyQSWuYMU8cQpxw@mail.gmail.com> <CAP7+vJ+LdEob5RO2J+D6QjfgB-wT68mDXk3iHDXtqbgXGVjP=w@mail.gmail.com> <CALGmxE+gi=KLQU8ggBafMw20dB6bfMrXHEmdxX8QNFRwKAC_MA@mail.gmail.com> <CAP7+vJ+8UrrHiGs7=Mogw2i6s0JS94UvfSSD-ZMy2xf2GrdU1Q@mail.gmail.com> <CALGmxELxomDNmdV7-B9=LSQrfceVvc9b1n_p0yrR08ffN65r4Q@mail.gmail.com> <CAP7+vJJbsFWQPTxDfSBEdREVyciJiKDWziF82y1HT4-DFwnUVQ@mail.gmail.com> <CALGmxEKqu_L=FBCDaAXF1v1-K1-v6GC=E84bqTSsFaBWKR6rCw@mail.gmail.com> Message-ID: <CAP7+vJLhzz0d5JO6J13zYMo4tFct3Gw_1f+9W1jkeaJUcfeoNg@mail.gmail.com> I'm still confused. We're talking about a function that compares two simple values, right? Where does the sequence of values come from? Numpy users do everything by the array, but they already have an isclose(). Or perhaps you're talking about assertApproxEqual in test_statistics.py? That has so many ways to specify the relative and absolute tolerance that I give up understanding it. The docstring doesn't give any clarity -- it simply describes the behaviors, it doesn't say when you should be using both. "[...] a naive implementation of relative error testing can run into trouble around zero" (followed by a single example) doesn't really help me. On Tue, Jan 27, 2015 at 2:14 PM, Chris Barker <chris.barker at noaa.gov> wrote: > On Tue, Jan 27, 2015 at 2:00 PM, Guido van Rossum <guido at python.org> > wrote: > > >> I really appreciate this API design approach, and in this case I started >>> out with that idea. But I think this is likely to be used where you need to >>> test a bunch of values with single function/set of parameters. In >>> TestCase.assertIsCloseTo, as well as home grown loops and comprehensions. >>> >> >> I assume you mean assertNotAlmostEqual >> <https://hg.python.org/cpython/file/94d8524086bd/Lib/unittest/case.py#l525>. >> >> > > Sorry -- I meant a Hypothetical assertIsCLoseTo -- which would wrap the > proposed is_close_to(), but apply it to a entire sequence, like > assertNotAlmostEqual > <https://hg.python.org/cpython/file/94d8524086bd/Lib/unittest/case.py#l525>. > does. > > >> This is actually the same misguided two-for-one design. You *must* >> specify exactly one of delta or places, and the code takes a different >> path. Also, it looks like both are actually absolute tolerance. >> > > Yes, both are an absolute tolerance -- which , I think makes it a slightly > less misguided design -- you aren't selecting two different > functionalities, just two ways to spell the tolerance. > > So what argument are you making here? > > > The point was that if the user applies one set of parameters to a sequence > of values that may have some zeros in it, -- we need it in one function. > Nathaniel has found that to be a fairly common use-case for the numpy > allclose() > > -Chris > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150127/db7c7c5e/attachment.html> From ben+python at benfinney.id.au Tue Jan 27 23:42:16 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 28 Jan 2015 09:42:16 +1100 Subject: [Python-ideas] datetime: Support infinity References: <54C5F9A7.3010209@thomas-guettler.de> <20150126101423.4edd9abd@fsol> Message-ID: <85iofrlubb.fsf@benfinney.id.au> Antoine Pitrou <solipsis at pitrou.net> writes: > Unless someone has a real-world use for the values of date.max, > datetime.max, interval.max, I find it rather counter-productive to not > store them back as infinities. The values are useful to know exactly how far in the future one can store date values in Python. It is important that this be discoverable programmatically because the maximum values might change in future versions of the ?datetime? library. Because it remains true (and presumably will continue to be true) that the maximum value is *not* infinity, it is important that the following real-world cases are distinct: * Query the ?datetime? library for the largest (furthest-in-the-future) value that it can handle. * Store a value explicitly meaning ?infinitely far in the future? or ?infinitely far in the past? compared to any actual date or datetime value. I have needed both, sometimes in the same program. An example is to be able to represent the timestamp field of a journal entry that is still being composed and has not yet gained a timestamp. ?None? is not sufficient, because it must be ?in the future? compared to any actual point in time, even greater than ?datetime.date.max?. To use the same value attempting to represent both ?maximum? and ?infinite? is a fragile, and needlessly confusing, hack. Having distinct ?infinitely far in the future? and ?infinitely far in the past? values, that are *not* themselves particular points on the timeline, would be a valuable addition for this reason. > Adding infinities to the datetime module would probably be possible but > someone has to figure out the arithmetic rules. Do we need "not a time" > when adding infinity to -infinity? Adding dates (or datetimes) is not a valid operation today, so I don't see why we would need to change behaviour there. -- \ ?This world in arms is not spending money alone. It is spending | `\ the sweat of its laborers, the genius of its scientists, the | _o__) hopes of its children.? ?Dwight Eisenhower, 1953-04-16 | Ben Finney From solipsis at pitrou.net Tue Jan 27 23:47:12 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 27 Jan 2015 23:47:12 +0100 Subject: [Python-ideas] datetime: Support infinity References: <54C5F9A7.3010209@thomas-guettler.de> <20150126101423.4edd9abd@fsol> <85iofrlubb.fsf@benfinney.id.au> Message-ID: <20150127234712.2bbde6f2@fsol> On Wed, 28 Jan 2015 09:42:16 +1100 Ben Finney <ben+python at benfinney.id.au> wrote: > Antoine Pitrou <solipsis at pitrou.net> > writes: > > > Unless someone has a real-world use for the values of date.max, > > datetime.max, interval.max, I find it rather counter-productive to not > > store them back as infinities. > > The values are useful to know exactly how far in the future one can > store date values in Python. It is important that this be discoverable > programmatically because the maximum values might change in future > versions of the ?datetime? library. So what? Again, the values wouldn't change in Python. It would only be the PostgreSQL mapping that would change. > > Adding infinities to the datetime module would probably be possible but > > someone has to figure out the arithmetic rules. Do we need "not a time" > > when adding infinity to -infinity? > > Adding dates (or datetimes) is not a valid operation today, so I don't > see why we would need to change behaviour there. We're talking about both datetimes and timestamps here. Please follow. Regards Antoine. From alexander.belopolsky at gmail.com Tue Jan 27 23:59:13 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 27 Jan 2015 17:59:13 -0500 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <20150127234712.2bbde6f2@fsol> References: <54C5F9A7.3010209@thomas-guettler.de> <20150126101423.4edd9abd@fsol> <85iofrlubb.fsf@benfinney.id.au> <20150127234712.2bbde6f2@fsol> Message-ID: <CAP7h-xYP6LPOyM=esEeEJA6EMNnbS4WQ5+5OJ6LY4aK5C9pfmQ@mail.gmail.com> On Tue, Jan 27, 2015 at 5:47 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: > > Adding dates (or datetimes) is not a valid operation today, so I don't > > see why we would need to change behaviour there. > > We're talking about both datetimes and timestamps here. Please follow. I think Antoine meant "timedeltas". Having a date infinity necessitates an infinite timedelta to represent a difference between a finite and an infinite date. While we cannot add dates, it is still a valid question what the result of subtracting an infinite date from itself should be. If we make it timedelta(0) then date.inf is not that different from date.max. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150127/115336a4/attachment-0001.html> From chris.barker at noaa.gov Wed Jan 28 00:00:27 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 27 Jan 2015 15:00:27 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAP7+vJLhzz0d5JO6J13zYMo4tFct3Gw_1f+9W1jkeaJUcfeoNg@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEKZjkz_iRqte=n3y0kQ6FyC5-30-chm_LAcWFY-C1R0cQ@mail.gmail.com> <CAPTjJmqFeKvri7jeFU_NAofrih1Z6hieuuNLs+dsdLFcKmxB=Q@mail.gmail.com> <CALGmxEJ8Y6t0DQMe82gBWxxybEnjvN0dhbNhyQSWuYMU8cQpxw@mail.gmail.com> <CAP7+vJ+LdEob5RO2J+D6QjfgB-wT68mDXk3iHDXtqbgXGVjP=w@mail.gmail.com> <CALGmxE+gi=KLQU8ggBafMw20dB6bfMrXHEmdxX8QNFRwKAC_MA@mail.gmail.com> <CAP7+vJ+8UrrHiGs7=Mogw2i6s0JS94UvfSSD-ZMy2xf2GrdU1Q@mail.gmail.com> <CALGmxELxomDNmdV7-B9=LSQrfceVvc9b1n_p0yrR08ffN65r4Q@mail.gmail.com> <CAP7+vJJbsFWQPTxDfSBEdREVyciJiKDWziF82y1HT4-DFwnUVQ@mail.gmail.com> <CALGmxEKqu_L=FBCDaAXF1v1-K1-v6GC=E84bqTSsFaBWKR6rCw@mail.gmail.com> <CAP7+vJLhzz0d5JO6J13zYMo4tFct3Gw_1f+9W1jkeaJUcfeoNg@mail.gmail.com> Message-ID: <CALGmxEJGTkHJfux8xLZOBfxtRcDxKcn_J5qseqzouLf3XONYvA@mail.gmail.com> On Tue, Jan 27, 2015 at 2:35 PM, Guido van Rossum <guido at python.org> wrote: > I'm still confused. We're talking about a function that compares two > simple values, right? Where does the sequence of values come from? > it comes from wherever, but the idea is that the function itself is called in a loop or comprehension, we could require something like: [ is_close_to(i) for i in seq if abs(i) > abs_tol else abs(i) <= abs_tol] but I prefer: [ is_close_to(i, abs_tolerance=abs_tol) for i in seq ] And it gets worse for the TestCase.assert** - those are designed to act on a sequence, so you would need to specify both your relative tolerance and absolute tolerance anyway, if your sequence might have zeros in it. > Or perhaps you're talking about assertApproxEqual in test_statistics.py? > That has so many ways to specify the relative and absolute tolerance that I > give up understanding it. > Something like that, yes, but with a cleaner API and better docs ;-) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150127/da215175/attachment.html> From solipsis at pitrou.net Wed Jan 28 00:02:09 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 28 Jan 2015 00:02:09 +0100 Subject: [Python-ideas] datetime: Support infinity References: <54C5F9A7.3010209@thomas-guettler.de> <20150126101423.4edd9abd@fsol> <85iofrlubb.fsf@benfinney.id.au> <20150127234712.2bbde6f2@fsol> <CAP7h-xYP6LPOyM=esEeEJA6EMNnbS4WQ5+5OJ6LY4aK5C9pfmQ@mail.gmail.com> Message-ID: <20150128000209.6f8cdad6@fsol> On Tue, 27 Jan 2015 17:59:13 -0500 Alexander Belopolsky <alexander.belopolsky at gmail.com> wrote: > On Tue, Jan 27, 2015 at 5:47 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: > > > > Adding dates (or datetimes) is not a valid operation today, so I don't > > > see why we would need to change behaviour there. > > > > We're talking about both datetimes and timestamps here. Please follow. > > > I think Antoine meant "timedeltas". Yup, indeed. My bad. And for the record: >>> import numpy as np >>> np.timedelta64(0) / np.timedelta64(0) __main__:1: RuntimeWarning: invalid value encountered in true_divide nan >>> np.timedelta64(0) / np.timedelta64(0) == np.timedelta64('nat') True >>> type(np.timedelta64('nat')) <class 'numpy.timedelta64'> Regards Antoine. From ben+python at benfinney.id.au Wed Jan 28 00:02:47 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 28 Jan 2015 10:02:47 +1100 Subject: [Python-ideas] datetime: Support infinity References: <54C5F9A7.3010209@thomas-guettler.de> <20150126101423.4edd9abd@fsol> <85iofrlubb.fsf@benfinney.id.au> <20150127234712.2bbde6f2@fsol> Message-ID: <85egqfltd4.fsf@benfinney.id.au> Antoine Pitrou <solipsis at pitrou.net> writes: > On Wed, 28 Jan 2015 09:42:16 +1100 > Ben Finney <ben+python at benfinney.id.au> wrote: > > The [minimum and maximum] values are useful to know exactly how far > > in the future [and past] one can store date values in Python. It is > > important that this be discoverable programmatically because the > > maximum [or minimum] values might change in future versions of the > > ?datetime? library. > > So what? Again, the values wouldn't change in Python. It would only be > the PostgreSQL mapping that would change. That's the point, though. We should at least allow for ?datetime.date.max? and ?datetime.date.infinitely_far_in_the_future? to be distinct, and allow for the former to change value in a backward-compatible way. So the maxima and minima should map to their specific points in time in both directions between PostgreSQL and Python. ?Maximum? does not mean ?infinite?, and conflating them robs us not only of the backward-compatible ability to change the maximum in future, but also of the ability to later add a distinct value to represent infinity. -- \ ?You don't need a book of any description to help you have some | `\ kind of moral awareness.? ?Dr. Francesca Stavrakoloulou, bible | _o__) scholar, 2011-05-08 | Ben Finney From chris.barker at noaa.gov Wed Jan 28 00:04:50 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 27 Jan 2015 15:04:50 -0800 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <20150127234712.2bbde6f2@fsol> References: <54C5F9A7.3010209@thomas-guettler.de> <20150126101423.4edd9abd@fsol> <85iofrlubb.fsf@benfinney.id.au> <20150127234712.2bbde6f2@fsol> Message-ID: <CALGmxEKjr1wSH5cKO1_3xzYbVbfBJhDP=tNYL3mTZe_PzFk6eA@mail.gmail.com> On Tue, Jan 27, 2015 at 2:47 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Wed, 28 Jan 2015 09:42:16 +1100 > Ben Finney <ben+python at benfinney.id.au> wrote: > > So what? Again, the values wouldn't change in Python. It would only be > the PostgreSQL mapping that would change. > infinity times would be useful outside of the PostgreSQL mapping. > > Adding infinities to the datetime module would probably be possible but > > > someone has to figure out the arithmetic rules. Do we need "not a time" > > > when adding infinity to -infinity? > not_a_timedelta, I suppose, or simply disallow it. But if you do allow it -- then following the rules for floating point inf, -inf, and NaN would make sense. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150127/4ab830bb/attachment.html> From schesis at gmail.com Wed Jan 28 01:38:04 2015 From: schesis at gmail.com (Zero Piraeus) Date: Tue, 27 Jan 2015 21:38:04 -0300 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxE+uLtEQ-RwCVfe3Lqv__JF76SXvRP3po8Sp96FoKXfPhQ@mail.gmail.com> <CALGmxEKan04wZyvUGJpe72+7yQkTUriKMyVVWfG-3t_g+SKVHA@mail.gmail.com> Message-ID: <20150128003804.GA25474@piedra> : On Tue, Jan 27, 2015 at 01:31:25PM -0800, Chris Barker wrote: > I'm still looking for a case where a user would likely pass the same > value into the function in a different order -- wouldn't s/he pick an > order (maybe arbitratily) and use that? Sometimes you *can't* pick an order (as in my dict example), and sometimes your data picks its own order (for example, using all_close() or similar to check the consistency of experimental results). On Tue, Jan 27, 2015 at 02:00:05PM -0800, Chris Barker wrote: > On Tue, Jan 27, 2015 at 1:52 PM, Paul Moore <p.f.moore at gmail.com> wrote: > > > The order of dictionary iteration is arbitrary. Thanks to hash > > randomization, the order will differ each time the program is run. > > duh, of course!. > > But question remains -- legitimate use case, or pathological example? > > This kind of points to a is_close_to() and a are are_close() function > (better than a flag, yes?) but that really does seem like overkill. Note that my example was an any_close() function, not an all_close() one. I doubt anyone's seriously going to suggest adding *three* new functions, so anyone who wants something like that will by necessity end up rolling their own. I admit I threw in the unpredictable iteration order of dictionaries to make the result as surprising as I could. There was a point to that ... Some behaviour in Python is surprising, often (as with dict iteration) necessarily so. However, the more surprising behaviour there is in the language, the more likely it is that two instances of that behaviour will interact in ways that are *especially* confusing and hard to debug. However, if you're not convinced by my dictionary shenanigans, here's something more straightforward: def any_close(iterable, tol): pairs = itertools.combinations(iterable, 2) return any(is_close_to(a, b, tol) for a, b in pairs) Notice that results = [2.34, 5.68, 9.99, 5.67, 0.01] ac1 = any_close(results, 1/568) results.sort() ac2 = any_close(results, 1/568) will result in ac1 and ac2 being different. A workflow along the lines of: User 1: - generate data - check sanity [all_close(), any_close(), etc] - normalize [sort] - send to User #2 [or persistent storage] User 2: - recieve from User #1 [or persistent storage] - check sanity [as above] - process data ... is pretty reasonable IMO, and would be adversely affected by the behaviour described above. You might fix that by having any_close() work on a sorted copy of the iterable, or by normalizing before sanity-checking, but that's not necessarily obvious until you get bitten, and it's not that hard to imagine scenarios where neither of those fixes are practical. -[]z. -- Zero Piraeus: post scriptum http://etiol.net/pubkey.asc From thomas at kluyver.me.uk Wed Jan 28 01:43:31 2015 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Tue, 27 Jan 2015 16:43:31 -0800 Subject: [Python-ideas] Adding a subprocess.CompletedProcess class Message-ID: <CAOvn4qgTKVTbcY+BN_Z7-v8bMRprn=2FSqcnMuw9j+NT-cegPA@mail.gmail.com> The subprocess module provides some nice tools to control the details of running a process, but it's still rather awkward for common use cases where you want to execute a command in one go. * There are three high-level functions: call, check_call and check_output, which all do very similar things with different return/raise behaviours * Their naming is not very clear (check_output doesn't check the output, it checks the return code and captures the output) * You can't use any of them if you want stdout and stderr separately. * You can get stdout and returncode from check_output, but it's not exactly obvious: try: stdout = check_output(...) returncode = 0 except CalledProcessError as e: stdout = e.output returncode = e.returncode I think that what these are lacking is a good way to represent a process that has already finished (as opposed to Popen, which is mostly designed to handle a running process). So I would: 1. Add a CompletedProcess class: * Attributes stdout and stderr are bytes if the relevant stream was piped, None otherwise, like the return value of Popen.communicate() * Attribute returncode is the exit status * ? Attribute cmd is the list of arguments the process ran with (not sure if this should be there or not) * Method cp.check_returncode() raises CalledProcessError if returncode != 0, inspired by requests' Response.raise_for_status() 2. Add a run() function - like call/check_call/check_output, but returns a CompletedProcess instance 3. Deprecate call/check_call/check_output, but leave them around indefinitely, since lots of existing code relies on them. Thanks, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150127/04dcbfa2/attachment.html> From steve at pearwood.info Wed Jan 28 01:46:38 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 28 Jan 2015 11:46:38 +1100 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <CAP7h-xYP6LPOyM=esEeEJA6EMNnbS4WQ5+5OJ6LY4aK5C9pfmQ@mail.gmail.com> References: <54C5F9A7.3010209@thomas-guettler.de> <20150126101423.4edd9abd@fsol> <85iofrlubb.fsf@benfinney.id.au> <20150127234712.2bbde6f2@fsol> <CAP7h-xYP6LPOyM=esEeEJA6EMNnbS4WQ5+5OJ6LY4aK5C9pfmQ@mail.gmail.com> Message-ID: <20150128004638.GU20517@ando.pearwood.info> On Tue, Jan 27, 2015 at 05:59:13PM -0500, Alexander Belopolsky wrote: > On Tue, Jan 27, 2015 at 5:47 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: > > > > Adding dates (or datetimes) is not a valid operation today, so I don't > > > see why we would need to change behaviour there. > > > > We're talking about both datetimes and timestamps here. Please follow. > > > I think Antoine meant "timedeltas". Having a date infinity necessitates an > infinite timedelta to represent a difference between a finite and an > infinite date. > > While we cannot add dates, it is still a valid question what the result of > subtracting an infinite date from itself should be. If we make it > timedelta(0) then date.inf is not that different from date.max. The obvious question is "What does Postgresql do?" Same for use-cases, which Antoine asked for. The primary use-case is to improve compatibility with Postgresql while still treating the full range of dates up to and including date.max as regular dates. I'm not convinced that numpy is the right library to follow here. Unless there are cases where you would perform calculations with Not-A-Time, an exception would probably be better suited. -- Steve From phd at phdru.name Wed Jan 28 01:57:26 2015 From: phd at phdru.name (Oleg Broytman) Date: Wed, 28 Jan 2015 01:57:26 +0100 Subject: [Python-ideas] Adding a subprocess.CompletedProcess class In-Reply-To: <CAOvn4qgTKVTbcY+BN_Z7-v8bMRprn=2FSqcnMuw9j+NT-cegPA@mail.gmail.com> References: <CAOvn4qgTKVTbcY+BN_Z7-v8bMRprn=2FSqcnMuw9j+NT-cegPA@mail.gmail.com> Message-ID: <20150128005726.GA17684@phdru.name> Hi! On Tue, Jan 27, 2015 at 04:43:31PM -0800, Thomas Kluyver <thomas at kluyver.me.uk> wrote: > The subprocess module provides some nice tools to control the details of > running a process, but it's still rather awkward for common use cases where > you want to execute a command in one go. Have you ever looked at wrappers like these: https://executor.readthedocs.org/en/latest/ https://sarge.readthedocs.org/en/latest/ https://github.com/kennethreitz/envoy https://amoffat.github.io/sh/ ??? Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From thomas at kluyver.me.uk Wed Jan 28 02:13:45 2015 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Tue, 27 Jan 2015 17:13:45 -0800 Subject: [Python-ideas] Adding a subprocess.CompletedProcess class In-Reply-To: <20150128005726.GA17684@phdru.name> References: <CAOvn4qgTKVTbcY+BN_Z7-v8bMRprn=2FSqcnMuw9j+NT-cegPA@mail.gmail.com> <20150128005726.GA17684@phdru.name> Message-ID: <CAOvn4qgkGHANd5KOptswFNQWVXdC6QEYgC0NH41N+1_8soAC5Q@mail.gmail.com> On 27 January 2015 at 16:57, Oleg Broytman <phd at phdru.name> wrote: > Have you ever looked at wrappers like these: > > https://executor.readthedocs.org/en/latest/ > https://sarge.readthedocs.org/en/latest/ > https://github.com/kennethreitz/envoy > https://amoffat.github.io/sh/ > I was aware of three of them (sarge, envoy, sh). But the friction isn't enough to push me to add another dependency: I invariably use subprocess and put up with writing somewhat awkward code. Envoy and sarge appear to implement something very similar to my proposal, but as part of a more ambitious goal, namely shell-style pipelines handled by Python. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150127/5b881d5f/attachment.html> From solipsis at pitrou.net Wed Jan 28 02:25:13 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 28 Jan 2015 02:25:13 +0100 Subject: [Python-ideas] datetime: Support infinity References: <54C5F9A7.3010209@thomas-guettler.de> <20150126101423.4edd9abd@fsol> <85iofrlubb.fsf@benfinney.id.au> <20150127234712.2bbde6f2@fsol> <CAP7h-xYP6LPOyM=esEeEJA6EMNnbS4WQ5+5OJ6LY4aK5C9pfmQ@mail.gmail.com> <20150128004638.GU20517@ando.pearwood.info> Message-ID: <20150128022513.7680e7a7@fsol> On Wed, 28 Jan 2015 11:46:38 +1100 Steven D'Aprano <steve at pearwood.info> wrote: > > The obvious question is "What does Postgresql do?" > > Same for use-cases, which Antoine asked for. The primary use-case is to > improve compatibility with Postgresql while still treating the full > range of dates up to and including date.max as regular dates. > > I'm not convinced that numpy is the right library to follow here. Unless > there are cases where you would perform calculations with Not-A-Time, an > exception would probably be better suited. I should add that I mentioned Numpy as a datapoint, but knowledgeable people I talked to generally agreed that Numpy's datetime64/timedelta64 design isn't very good (actually, it has an even wonkier feature that's only exposed internally AFAIK). So "Not-A-Time" need not necessarily be implemented in the stdlib ;) Regards Antoine. From phd at phdru.name Wed Jan 28 02:34:49 2015 From: phd at phdru.name (Oleg Broytman) Date: Wed, 28 Jan 2015 02:34:49 +0100 Subject: [Python-ideas] Adding a subprocess.CompletedProcess class In-Reply-To: <CAOvn4qgkGHANd5KOptswFNQWVXdC6QEYgC0NH41N+1_8soAC5Q@mail.gmail.com> References: <CAOvn4qgTKVTbcY+BN_Z7-v8bMRprn=2FSqcnMuw9j+NT-cegPA@mail.gmail.com> <20150128005726.GA17684@phdru.name> <CAOvn4qgkGHANd5KOptswFNQWVXdC6QEYgC0NH41N+1_8soAC5Q@mail.gmail.com> Message-ID: <20150128013449.GA19071@phdru.name> On Tue, Jan 27, 2015 at 05:13:45PM -0800, Thomas Kluyver <thomas at kluyver.me.uk> wrote: > On 27 January 2015 at 16:57, Oleg Broytman <phd at phdru.name> wrote: > > > Have you ever looked at wrappers like these: > > > > https://executor.readthedocs.org/en/latest/ > > https://sarge.readthedocs.org/en/latest/ > > https://github.com/kennethreitz/envoy > > https://amoffat.github.io/sh/ > > > > I was aware of three of them (sarge, envoy, sh). > But the friction isn't > enough to push me to add another dependency: I invariably use subprocess > and put up with writing somewhat awkward code. The worst outcome of included batteries: people are always trying to push all possible batteries into stdlib. :-( > Envoy and sarge appear to implement something very similar to my proposal, > but as part of a more ambitious goal, namely shell-style pipelines handled > by Python. Something like these? https://pypi.python.org/pypi/iterpipes https://github.com/kelleyk/py3k-iterpipes https://github.com/airekans/Pypiep http://plumbum.readthedocs.org/en/latest/ https://pypi.python.org/pypi/popen https://pypi.python.org/pypi/shell.py https://seveas.github.io/whelk/ > Thomas Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From thomas at kluyver.me.uk Wed Jan 28 02:56:32 2015 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Tue, 27 Jan 2015 17:56:32 -0800 Subject: [Python-ideas] Adding a subprocess.CompletedProcess class In-Reply-To: <20150128013449.GA19071@phdru.name> References: <CAOvn4qgTKVTbcY+BN_Z7-v8bMRprn=2FSqcnMuw9j+NT-cegPA@mail.gmail.com> <20150128005726.GA17684@phdru.name> <CAOvn4qgkGHANd5KOptswFNQWVXdC6QEYgC0NH41N+1_8soAC5Q@mail.gmail.com> <20150128013449.GA19071@phdru.name> Message-ID: <CAOvn4qhkOsfQ2MJPv3kkiPBbezfhpMUwMJN228_Tfh1ssLhQYg@mail.gmail.com> On 27 January 2015 at 17:34, Oleg Broytman <phd at phdru.name> wrote: > The worst outcome of included batteries: people are always trying to > push all possible batteries into stdlib. :-( > Or perhaps the problem is that people like me keep using lower quality batteries simply because those are the ones included. subprocess is at that point where it's good enough that I don't want to learn and depend on something else, but bad enough that I don't enjoy using it. None of the wrappers you linked to before have gained the kind of traction that requests has for HTTP, and I'm sceptical that they ever would. > Something like these? > > https://pypi.python.org/pypi/iterpipes > https://github.com/kelleyk/py3k-iterpipes > https://github.com/airekans/Pypiep > http://plumbum.readthedocs.org/en/latest/ > https://pypi.python.org/pypi/popen > https://pypi.python.org/pypi/shell.py > https://seveas.github.io/whelk/ > To be clear, this is *not* what I'm suggesting go into subprocess. That would make a much bigger, more complex battery. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150127/65e0c878/attachment.html> From yawar.amin at gmail.com Wed Jan 28 04:24:10 2015 From: yawar.amin at gmail.com (Yawar Amin) Date: Tue, 27 Jan 2015 22:24:10 -0500 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <CALGmxEKjr1wSH5cKO1_3xzYbVbfBJhDP=tNYL3mTZe_PzFk6eA@mail.gmail.com> References: <54C5F9A7.3010209@thomas-guettler.de> <20150126101423.4edd9abd@fsol> <85iofrlubb.fsf@benfinney.id.au> <20150127234712.2bbde6f2@fsol> <CALGmxEKjr1wSH5cKO1_3xzYbVbfBJhDP=tNYL3mTZe_PzFk6eA@mail.gmail.com> Message-ID: <54C8565A.2060705@gmail.com> Hi, On 2015-01-27 18:04, Chris Barker wrote: > On Tue, Jan 27, 2015 at 2:47 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: > [...] > > > Adding infinities to the datetime module would probably be > > > possible but > > > someone has to figure out the arithmetic rules. Do we need > > > "not a time" > > > when adding infinity to -infinity? > not_a_timedelta, I suppose, or simply disallow it. > > But if you do allow it -- then following the rules for floating point > inf, -inf, and NaN would make sense. Adding datetime.neg_inf() and datetime.pos_inf() should return datetime.now(). Regards, Yawar -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 834 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150127/df66d5d0/attachment.sig> From yawar.amin at gmail.com Wed Jan 28 04:29:12 2015 From: yawar.amin at gmail.com (Yawar Amin) Date: Tue, 27 Jan 2015 22:29:12 -0500 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <3114096308045821944@unknownmsgid> References: <54C5F9A7.3010209@thomas-guettler.de> <3114096308045821944@unknownmsgid> Message-ID: <54C85788.3090705@gmail.com> Hi Chris, On 2015-01-26 11:31, Chris Barker - NOAA Federal wrote: > [...] > I have a sample implementation if anyone wants to look amateur it, > though I don't think it does reflected comparisons correctly. it's > really only just good enough for our use case. I'd like to take a look. Can you send it to me, or post it somewhere? Regards, Yawar From alexander.belopolsky at gmail.com Wed Jan 28 05:00:23 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 27 Jan 2015 23:00:23 -0500 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <54C8565A.2060705@gmail.com> References: <54C5F9A7.3010209@thomas-guettler.de> <20150126101423.4edd9abd@fsol> <85iofrlubb.fsf@benfinney.id.au> <20150127234712.2bbde6f2@fsol> <CALGmxEKjr1wSH5cKO1_3xzYbVbfBJhDP=tNYL3mTZe_PzFk6eA@mail.gmail.com> <54C8565A.2060705@gmail.com> Message-ID: <CAP7h-xbpg4Q_ccPWnXKO005LZP9DMJtrT=vi15FUYgV8SjheRg@mail.gmail.com> On Tue, Jan 27, 2015 at 10:24 PM, Yawar Amin <yawar.amin at gmail.com> wrote: > Adding datetime.neg_inf() and datetime.pos_inf() should return > datetime.now(). > Nah, it should return datetime.utcfromtimestamp(random.randint(-2**32, 2**32)). Anything else would be a heresy! -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150127/0fe68514/attachment.html> From chris.barker at noaa.gov Wed Jan 28 06:17:17 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 27 Jan 2015 21:17:17 -0800 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <54C85788.3090705@gmail.com> References: <54C5F9A7.3010209@thomas-guettler.de> <3114096308045821944@unknownmsgid> <54C85788.3090705@gmail.com> Message-ID: <CALGmxE+o7sakjvys1KZPrpHRKpfe7ijTvOBaQ6WkieC3O-z5=Q@mail.gmail.com> On Tue, Jan 27, 2015 at 7:29 PM, Yawar Amin <yawar.amin at gmail.com> wrote: > On 2015-01-26 11:31, Chris Barker - NOAA Federal wrote: > > I have a sample implementation if anyone wants to look at it, > > though I don't think it does reflected comparisons correctly. it's > > really only just good enough for our use case. > > I'd like to take a look. Can you send it to me, or post it somewhere? > Sure, here you go: https://github.com/NOAA-ORR-ERD/PyGnome/blob/master/py_gnome/gnome/utilities/inf_datetime.py -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150127/b6ee2e88/attachment.html> From chris.barker at noaa.gov Wed Jan 28 07:31:01 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 27 Jan 2015 22:31:01 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150128003804.GA25474@piedra> References: <CALGmxE+uLtEQ-RwCVfe3Lqv__JF76SXvRP3po8Sp96FoKXfPhQ@mail.gmail.com> <CALGmxEKan04wZyvUGJpe72+7yQkTUriKMyVVWfG-3t_g+SKVHA@mail.gmail.com> <20150128003804.GA25474@piedra> Message-ID: <CALGmxEJ--kPkr-qSLJXuOvk4vbm4qCkc92Mz0r0SkPbLv+vUzQ@mail.gmail.com> On Tue, Jan 27, 2015 at 4:38 PM, Zero Piraeus <schesis at gmail.com> wrote: <snip> > Note that my example was an any_close() function, not an all_close() > one. I doubt anyone's seriously going to suggest adding *three* new > functions, so anyone who wants something like that will by necessity > end up rolling their own. > OK -- that is pretty compelling --- I have been looking for use-cases for where having a symmetrical test would be a clear advantage, and this is one. But this is where I'm unclear -- is this "any_close" function something you think is a real use case -- i.e. you have needed such a thing, or think there is a strong chance that you will. Or just something that _could_ be done. Granted, I think your point is that if it _could_ be done, there is a good chance that _someone_ will do it -- and with that in mind we want as few surprising behaviors in the standard lib as possible. But it would be even more compelling if it were a real use case ;-) Anyway, it seems that Steven is going to write up something to clarify the issues, and I'll try to write the various options up in the PEP, and then we can suss out which options are acceptable to most folks. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150127/0a8223ed/attachment-0001.html> From greg at krypto.org Wed Jan 28 08:34:41 2015 From: greg at krypto.org (Gregory P. Smith) Date: Wed, 28 Jan 2015 07:34:41 +0000 Subject: [Python-ideas] Adding a subprocess.CompletedProcess class References: <CAOvn4qgTKVTbcY+BN_Z7-v8bMRprn=2FSqcnMuw9j+NT-cegPA@mail.gmail.com> Message-ID: <CAGE7PNKQ0QpZ-3hab4xJppknhCOOCJXuRWadsKGeGFz-BbJg_A@mail.gmail.com> On Tue Jan 27 2015 at 4:44:32 PM Thomas Kluyver <thomas at kluyver.me.uk> wrote: > The subprocess module provides some nice tools to control the details of > running a process, but it's still rather awkward for common use cases where > you want to execute a command in one go. > > * There are three high-level functions: call, check_call and check_output, > which all do very similar things with different return/raise behaviours > * Their naming is not very clear (check_output doesn't check the output, > it checks the return code and captures the output) > * You can't use any of them if you want stdout and stderr separately. > * You can get stdout and returncode from check_output, but it's not > exactly obvious: > > try: > stdout = check_output(...) > returncode = 0 > except CalledProcessError as e: > stdout = e.output > returncode = e.returncode > > I think that what these are lacking is a good way to represent a process > that has already finished (as opposed to Popen, which is mostly designed to > handle a running process). So I would: > > 1. Add a CompletedProcess class: > * Attributes stdout and stderr are bytes if the relevant stream was piped, > None otherwise, like the return value of Popen.communicate() > * Attribute returncode is the exit status > * ? Attribute cmd is the list of arguments the process ran with (not sure > if this should be there or not) > * Method cp.check_returncode() raises CalledProcessError if returncode != > 0, inspired by requests' Response.raise_for_status() > > 2. Add a run() function - like call/check_call/check_output, but returns a > CompletedProcess instance > I like #1 and #2 here. 3. Deprecate call/check_call/check_output, but leave them around > indefinitely, since lots of existing code relies on them. > We need to keep those. They are too widely used and are the long term stable API for 2.7. They are useful for many simple cases which they were designed for. -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150128/8628fa4e/attachment.html> From fetchinson at googlemail.com Wed Jan 28 09:25:35 2015 From: fetchinson at googlemail.com (Fetchinson .) Date: Wed, 28 Jan 2015 09:25:35 +0100 Subject: [Python-ideas] python on mobile In-Reply-To: <CAF7AXFEncW5sfZFCc=Jux1UPCS2iKDQoPdQk6c4x_N3YUeO6bQ@mail.gmail.com> References: <CADjSo4TssZHGiOCm110co-BQxLAXZpCZbT1T=zdd8eBavnyVzQ@mail.gmail.com> <CAJxq848RyRhbeD4kHssFo1jHUtVX5N1_G=RB1rB+DkmDaxxA2A@mail.gmail.com> <CADiSq7enX0nmNd2AEHGwCgdDsgaXWzdz0A1w5iU_m1V+JdHe6g@mail.gmail.com> <CAJxq84-EeN0v49sHpPYZStJJ_615quqMYmmq2OPvhcaae6Hfiw@mail.gmail.com> <CADjSo4Seed-B0FmTUikrtvDux_LprdvedVrR1-RvQS8fPE=LiQ@mail.gmail.com> <87k31833vw.fsf@uwakimon.sk.tsukuba.ac.jp> <CAJxq848T09_c-jrHhtyyWUBFk7By7xtUphCExom7wNd36ZCtow@mail.gmail.com> <CADiSq7ewkZwRZhkeyuJC32KeaaPuq9FTxQXHiY5uhOZBA+Un=g@mail.gmail.com> <CAF7AXFFh0voFPLPtYxv0qC8yDXmxdBn5K-PT_5oZSXC8RT7jLg@mail.gmail.com> <CADjSo4QnhX4bSNuiBExNx7pMyffzBT6J0pCZGfvk-2cv_hgM1Q@mail.gmail.com> <CAF7AXFEncW5sfZFCc=Jux1UPCS2iKDQoPdQk6c4x_N3YUeO6bQ@mail.gmail.com> Message-ID: <CADjSo4RfDP_aLvPHWg2-qunWxEm0Gq7ATPW1OfvyUStyNYm7yA@mail.gmail.com> On 1/12/15, Jeff Hardy <jdhardy at gmail.com> wrote: > On Mon, Jan 5, 2015 at 4:28 PM, Fetchinson . <fetchinson at googlemail.com> > wrote: >>> So +1 on mobile-sig from me. I'll even volunteer to get it set up & >>> administer it >> >> +1 > > Apologies for the delay. I've sent a request to meta-sig to create > mobile-sig. Anyone interested should chime in over there > (https://mail.python.org/mailman/listinfo/meta-sig). The new mobile-sig mailing list is alive: https://mail.python.org/pipermail/mobile-sig/2015-January/thread.html Thanks a lot to our postmaster! Cheers, Daniel -- Psss, psss, put it down! - http://www.cafepress.com/putitdown From ncoghlan at gmail.com Wed Jan 28 09:36:55 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Jan 2015 18:36:55 +1000 Subject: [Python-ideas] Adding a subprocess.CompletedProcess class In-Reply-To: <CAGE7PNKQ0QpZ-3hab4xJppknhCOOCJXuRWadsKGeGFz-BbJg_A@mail.gmail.com> References: <CAOvn4qgTKVTbcY+BN_Z7-v8bMRprn=2FSqcnMuw9j+NT-cegPA@mail.gmail.com> <CAGE7PNKQ0QpZ-3hab4xJppknhCOOCJXuRWadsKGeGFz-BbJg_A@mail.gmail.com> Message-ID: <CADiSq7dAYUudHbDzRuVxmL6SsT4i8_50jvGNPf0G2Jet7MS8Dg@mail.gmail.com> On 28 January 2015 at 17:34, Gregory P. Smith <greg at krypto.org> wrote: > On Tue Jan 27 2015 at 4:44:32 PM Thomas Kluyver <thomas at kluyver.me.uk> > wrote: >> >> The subprocess module provides some nice tools to control the details of >> running a process, but it's still rather awkward for common use cases where >> you want to execute a command in one go. >> >> * There are three high-level functions: call, check_call and check_output, >> which all do very similar things with different return/raise behaviours >> * Their naming is not very clear (check_output doesn't check the output, >> it checks the return code and captures the output) >> * You can't use any of them if you want stdout and stderr separately. >> * You can get stdout and returncode from check_output, but it's not >> exactly obvious: >> >> try: >> stdout = check_output(...) >> returncode = 0 >> except CalledProcessError as e: >> stdout = e.output >> returncode = e.returncode >> >> I think that what these are lacking is a good way to represent a process >> that has already finished (as opposed to Popen, which is mostly designed to >> handle a running process). So I would: >> >> 1. Add a CompletedProcess class: >> * Attributes stdout and stderr are bytes if the relevant stream was piped, >> None otherwise, like the return value of Popen.communicate() >> * Attribute returncode is the exit status >> * ? Attribute cmd is the list of arguments the process ran with (not sure >> if this should be there or not) >> * Method cp.check_returncode() raises CalledProcessError if returncode != >> 0, inspired by requests' Response.raise_for_status() >> >> 2. Add a run() function - like call/check_call/check_output, but returns a >> CompletedProcess instance > > > I like #1 and #2 here. > >> 3. Deprecate call/check_call/check_output, but leave them around >> indefinitely, since lots of existing code relies on them. > > > We need to keep those. They are too widely used and are the long term stable > API for 2.7. They are useful for many simple cases which they were designed > for. I'm with Greg here - the proposed "run()" API sounds like a nice improvement to me, but I don't think it makes sense to deprecate the existing use case specific variants. In particular, check_call() and check_output() aren't easily replaced with equivalent expressions based on run() since you need to make a separate call to cp.check_returncode(). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From barry at python.org Wed Jan 28 17:06:16 2015 From: barry at python.org (Barry Warsaw) Date: Wed, 28 Jan 2015 11:06:16 -0500 Subject: [Python-ideas] Adding a subprocess.CompletedProcess class References: <CAOvn4qgTKVTbcY+BN_Z7-v8bMRprn=2FSqcnMuw9j+NT-cegPA@mail.gmail.com> <CAGE7PNKQ0QpZ-3hab4xJppknhCOOCJXuRWadsKGeGFz-BbJg_A@mail.gmail.com> Message-ID: <20150128110616.56b7c933@anarchist.wooz.org> On Jan 28, 2015, at 07:34 AM, Gregory P. Smith wrote: >On Tue Jan 27 2015 at 4:44:32 PM Thomas Kluyver ><thomas at kluyver.me.uk> wrote: > >> 1. Add a CompletedProcess class: >> * Attributes stdout and stderr are bytes if the relevant stream was piped, >> None otherwise, like the return value of Popen.communicate() Note that in Python 3, if universal_newlines=True is given, the output will be a str, not bytes. (The input is also switched from bytes to str, but that seems less common, at least in my use cases.) You'll probably build the new APIs on the existing ones, so I don't expect that to change. I just wanted to point that out. >> * Attribute returncode is the exit status >> * ? Attribute cmd is the list of arguments the process ran with (not sure >> if this should be there or not) >> * Method cp.check_returncode() raises CalledProcessError if returncode != >> 0, inspired by requests' Response.raise_for_status() >> >> 2. Add a run() function - like call/check_call/check_output, but returns a >> CompletedProcess instance > >I like #1 and #2 here. > >3. Deprecate call/check_call/check_output, but leave them around >> indefinitely, since lots of existing code relies on them. >> >We need to keep those. They are too widely used and are the long term >stable API for 2.7. They are useful for many simple cases which they were >designed for. Agreed that this seems like a nice improvement, but that we need to keep the old APIs around probably forever. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150128/3a6889ac/attachment.sig> From thomas at kluyver.me.uk Wed Jan 28 18:43:55 2015 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Wed, 28 Jan 2015 09:43:55 -0800 Subject: [Python-ideas] Adding a subprocess.CompletedProcess class In-Reply-To: <CADiSq7dAYUudHbDzRuVxmL6SsT4i8_50jvGNPf0G2Jet7MS8Dg@mail.gmail.com> References: <CAOvn4qgTKVTbcY+BN_Z7-v8bMRprn=2FSqcnMuw9j+NT-cegPA@mail.gmail.com> <CAGE7PNKQ0QpZ-3hab4xJppknhCOOCJXuRWadsKGeGFz-BbJg_A@mail.gmail.com> <CADiSq7dAYUudHbDzRuVxmL6SsT4i8_50jvGNPf0G2Jet7MS8Dg@mail.gmail.com> Message-ID: <CAOvn4qhH2Et49NgMMED3DQ-9GuX1k8cELGEXyegh64ezMHoqfw@mail.gmail.com> On 28 January 2015 at 00:36, Nick Coghlan <ncoghlan at gmail.com> wrote: > I'm with Greg here - the proposed "run()" API sounds like a nice > improvement to me, but I don't think it makes sense to deprecate the > existing use case specific variants. In particular, check_call() and > check_output() aren't easily replaced with equivalent expressions > based on run() since you need to make a separate call to > cp.check_returncode(). > Good point. Perhaps it would be more useful for run() to take a kwarg check_returncode, which if True would cause it to raise on a non-zero exit code? That way, any of the function trio could be replaced by use of run(). Barry: > Note that in Python 3, if universal_newlines=True is given, the output will be > a str, not bytes. Ugh, yes. I would follow that for consistency, though I'd love to come up with a way to clean that up too. The fact that 'decode the output' is spelled 'universal_newlines' is one of my least favourite things about subprocess. >>We need to keep those. They are too widely used and are the long term >>stable API for 2.7. They are useful for many simple cases which they were >>designed for. > > Agreed that this seems like a nice improvement, but that we need to keep the > old APIs around probably forever. Agreed. I would however like to de-emphasise them in the docs. It seems like there's some level of interest in this; shall I work on a patch? Thanks, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150128/6d25dcb8/attachment.html> From ethan at stoneleaf.us Wed Jan 28 19:02:30 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 28 Jan 2015 10:02:30 -0800 Subject: [Python-ideas] Adding a subprocess.CompletedProcess class In-Reply-To: <CAOvn4qhH2Et49NgMMED3DQ-9GuX1k8cELGEXyegh64ezMHoqfw@mail.gmail.com> References: <CAOvn4qgTKVTbcY+BN_Z7-v8bMRprn=2FSqcnMuw9j+NT-cegPA@mail.gmail.com> <CAGE7PNKQ0QpZ-3hab4xJppknhCOOCJXuRWadsKGeGFz-BbJg_A@mail.gmail.com> <CADiSq7dAYUudHbDzRuVxmL6SsT4i8_50jvGNPf0G2Jet7MS8Dg@mail.gmail.com> <CAOvn4qhH2Et49NgMMED3DQ-9GuX1k8cELGEXyegh64ezMHoqfw@mail.gmail.com> Message-ID: <54C92436.70404@stoneleaf.us> On 01/28/2015 09:43 AM, Thomas Kluyver wrote: > > It seems like there's some level of interest in this; shall I work on a patch? That would be great! -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150128/98b44651/attachment.sig> From thomas at kluyver.me.uk Wed Jan 28 21:27:34 2015 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Wed, 28 Jan 2015 12:27:34 -0800 Subject: [Python-ideas] Adding a subprocess.CompletedProcess class In-Reply-To: <54C92436.70404@stoneleaf.us> References: <CAOvn4qgTKVTbcY+BN_Z7-v8bMRprn=2FSqcnMuw9j+NT-cegPA@mail.gmail.com> <CAGE7PNKQ0QpZ-3hab4xJppknhCOOCJXuRWadsKGeGFz-BbJg_A@mail.gmail.com> <CADiSq7dAYUudHbDzRuVxmL6SsT4i8_50jvGNPf0G2Jet7MS8Dg@mail.gmail.com> <CAOvn4qhH2Et49NgMMED3DQ-9GuX1k8cELGEXyegh64ezMHoqfw@mail.gmail.com> <54C92436.70404@stoneleaf.us> Message-ID: <CAOvn4qixL1HcWQBt6OMuxD2aL3gFaL5UUh2aLc4UX59u2CUnYA@mail.gmail.com> On 28 January 2015 at 10:02, Ethan Furman <ethan at stoneleaf.us> wrote: > > It seems like there's some level of interest in this; shall I work on a > patch? > > That would be great! First pass at a patch attached. I also ended up adding stderr attributes to CalledProcessError and TimeoutExceeded, so information is not thrown away. Things I wasn't sure about: 1. Should stdout/stderr be captured by default? I opted not to, for consistency with Popen and with shells. 2. I gave run() a check_returncode parameter, but it feels quite a long name for a parameter. Is 'check' clear enough to use as the parameter name? 3. Popen has an 'args' attribute, while CalledProcessError and TimeoutExpired have 'cmd'. CompletedProcess sits between those cases, so which name should it use? For now, it's args. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150128/d08fd67f/attachment-0001.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: subprocess_run.patch Type: text/x-patch Size: 27086 bytes Desc: not available URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150128/d08fd67f/attachment-0001.bin> From ethan at stoneleaf.us Wed Jan 28 21:35:05 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 28 Jan 2015 12:35:05 -0800 Subject: [Python-ideas] Adding a subprocess.CompletedProcess class In-Reply-To: <CAOvn4qixL1HcWQBt6OMuxD2aL3gFaL5UUh2aLc4UX59u2CUnYA@mail.gmail.com> References: <CAOvn4qgTKVTbcY+BN_Z7-v8bMRprn=2FSqcnMuw9j+NT-cegPA@mail.gmail.com> <CAGE7PNKQ0QpZ-3hab4xJppknhCOOCJXuRWadsKGeGFz-BbJg_A@mail.gmail.com> <CADiSq7dAYUudHbDzRuVxmL6SsT4i8_50jvGNPf0G2Jet7MS8Dg@mail.gmail.com> <CAOvn4qhH2Et49NgMMED3DQ-9GuX1k8cELGEXyegh64ezMHoqfw@mail.gmail.com> <54C92436.70404@stoneleaf.us> <CAOvn4qixL1HcWQBt6OMuxD2aL3gFaL5UUh2aLc4UX59u2CUnYA@mail.gmail.com> Message-ID: <54C947F9.7070003@stoneleaf.us> On 01/28/2015 12:27 PM, Thomas Kluyver wrote: > On 28 January 2015 at 10:02, Ethan Furman <ethan at stoneleaf.us <mailto:ethan at stoneleaf.us>> wrote: > > > It seems like there's some level of interest in this; shall I work on a patch? > > That would be great! > > > First pass at a patch attached. I also ended up adding stderr attributes to CalledProcessError and TimeoutExceeded, so > information is not thrown away. > > Things I wasn't sure about: > > 1. Should stdout/stderr be captured by default? I opted not to, for consistency with Popen and with shells. > 2. I gave run() a check_returncode parameter, but it feels quite a long name for a parameter. Is 'check' clear enough to > use as the parameter name? > 3. Popen has an 'args' attribute, while CalledProcessError and TimeoutExpired have 'cmd'. CompletedProcess sits between > those cases, so which name should it use? For now, it's args. At this point, create an issue on the bug tracker and attach your patch there. Much easier to keep track of that way. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150128/0021be11/attachment.sig> From barry at python.org Wed Jan 28 23:13:18 2015 From: barry at python.org (Barry Warsaw) Date: Wed, 28 Jan 2015 17:13:18 -0500 Subject: [Python-ideas] Adding a subprocess.CompletedProcess class References: <CAOvn4qgTKVTbcY+BN_Z7-v8bMRprn=2FSqcnMuw9j+NT-cegPA@mail.gmail.com> <CAGE7PNKQ0QpZ-3hab4xJppknhCOOCJXuRWadsKGeGFz-BbJg_A@mail.gmail.com> <CADiSq7dAYUudHbDzRuVxmL6SsT4i8_50jvGNPf0G2Jet7MS8Dg@mail.gmail.com> <CAOvn4qhH2Et49NgMMED3DQ-9GuX1k8cELGEXyegh64ezMHoqfw@mail.gmail.com> Message-ID: <20150128171318.22a829eb@anarchist.wooz.org> On Jan 28, 2015, at 09:43 AM, Thomas Kluyver wrote: >Ugh, yes. I would follow that for consistency, though I'd love to come up >with a way to clean that up too. The fact that 'decode the output' is >spelled 'universal_newlines' is one of my least favourite things about >subprocess. Not me; it's one of those sekrits that ensures Python 3 porters job security. ;) Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150128/93ff6e0a/attachment.sig> From thomas at kluyver.me.uk Wed Jan 28 23:15:12 2015 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Wed, 28 Jan 2015 14:15:12 -0800 Subject: [Python-ideas] Adding a subprocess.CompletedProcess class In-Reply-To: <54C947F9.7070003@stoneleaf.us> References: <CAOvn4qgTKVTbcY+BN_Z7-v8bMRprn=2FSqcnMuw9j+NT-cegPA@mail.gmail.com> <CAGE7PNKQ0QpZ-3hab4xJppknhCOOCJXuRWadsKGeGFz-BbJg_A@mail.gmail.com> <CADiSq7dAYUudHbDzRuVxmL6SsT4i8_50jvGNPf0G2Jet7MS8Dg@mail.gmail.com> <CAOvn4qhH2Et49NgMMED3DQ-9GuX1k8cELGEXyegh64ezMHoqfw@mail.gmail.com> <54C92436.70404@stoneleaf.us> <CAOvn4qixL1HcWQBt6OMuxD2aL3gFaL5UUh2aLc4UX59u2CUnYA@mail.gmail.com> <54C947F9.7070003@stoneleaf.us> Message-ID: <CAOvn4qgModFCpvoHRUZBQLjnz8jm=0LxXGcN9FLaubbU0AJk+A@mail.gmail.com> On 28 January 2015 at 12:35, Ethan Furman <ethan at stoneleaf.us> wrote: > At this point, create an issue on the bug tracker and attach your patch > there. Much easier to keep track of that way. Done, thanks: http://bugs.python.org/issue23342 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150128/cd4ef440/attachment.html> From yawar.amin at gmail.com Thu Jan 29 03:55:58 2015 From: yawar.amin at gmail.com (Yawar Amin) Date: Wed, 28 Jan 2015 21:55:58 -0500 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <CALGmxE+o7sakjvys1KZPrpHRKpfe7ijTvOBaQ6WkieC3O-z5=Q@mail.gmail.com> References: <54C5F9A7.3010209@thomas-guettler.de> <3114096308045821944@unknownmsgid> <54C85788.3090705@gmail.com> <CALGmxE+o7sakjvys1KZPrpHRKpfe7ijTvOBaQ6WkieC3O-z5=Q@mail.gmail.com> Message-ID: <54C9A13E.6050003@gmail.com> On 2015-01-28 00:17, Chris Barker wrote: > [...] > Sure, here you go: > > https://github.com/NOAA-ORR-ERD/PyGnome/blob/master/py_gnome/gnome/utilities/inf_datetime.py Cool, the comparison operator logic looks very similar to mine: https://groups.google.com/d/msg/python-ideas/G3jeWoa6h14/ELpDLFu28QcJ I believe the ideal design is the following type hierarchy: datetime_base datetime_neg_inf datetime_pos_inf datetime But in the interests of pragmatism, I think this one can be made to work: datetime datetime_neg_inf datetime_pos_inf So we'd keep using the existing datetime type, and whenever we need infinity datetimes we'd just pass in the appropriate subtypes. The advantage to this is that we don't have to switch everything over to using a new derived type--just use the normal datetime for the majority of cases where we don't need inifinity dates. Regards, Yawar -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 834 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150128/4e318ecf/attachment.sig> From Nikolaus at rath.org Thu Jan 29 06:49:00 2015 From: Nikolaus at rath.org (Nikolaus Rath) Date: Wed, 28 Jan 2015 21:49:00 -0800 Subject: [Python-ideas] "atexit" equivalent for functions in contextlib Message-ID: <87bnlirvar.fsf@vostro.rath.org> Hello, The other day, I was looking for an "atexit" equivalent at the function level. I was hoping to replace code like this: def my_function(bla, fasel): with ExitStack() as cm: ... cm.push(...) ... with something like: @handle_on_return def my_function(bla, fasel, on_return): ... on_return.push(...) ... It seems that contextlib *almost* offers a way to do this with ExitStack and ContextDecorator - but as far as I can tell the final piece is missing, because ContextDecorator does not offer a way to pass the context manager to the decorated function. However, the following decorator does the job: def handle_on_return(fn): @functools.wraps(fn) def wrapper(*a, **kw): with contextlib.ExitStack() as on_return: kw['on_return'] = on_return return fn(*a, **kw) return wrapper It's not a lot of code, but my feeling is that not anyone who might be interested in an "on_return" functionality would be able to come up with this right away. Would it make sense to add something along these lines to contextlib? Maybe instead of a new decorator, ContextDecorator could also take an additional keyword argument that tells it to pass the context manager to the decorated function? Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From chris.barker at noaa.gov Thu Jan 29 08:04:10 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 28 Jan 2015 23:04:10 -0800 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <54C9A13E.6050003@gmail.com> References: <54C5F9A7.3010209@thomas-guettler.de> <3114096308045821944@unknownmsgid> <54C85788.3090705@gmail.com> <CALGmxE+o7sakjvys1KZPrpHRKpfe7ijTvOBaQ6WkieC3O-z5=Q@mail.gmail.com> <54C9A13E.6050003@gmail.com> Message-ID: <CALGmxELXfjftwP_qQfRyHfcCE=e-Jae8T68i6LYc9ZCHAxuXSA@mail.gmail.com> On Wed, Jan 28, 2015 at 6:55 PM, Yawar Amin <yawar.amin at gmail.com> wrote: > Cool, the comparison operator logic looks very similar to mine: > > https://groups.google.com/d/msg/python-ideas/G3jeWoa6h14/ELpDLFu28QcJ > > Did you get it to work both ways? i.e. datetime.neg_inf < datetime.now() and datetime.now() > datetime.neg_inf I had trouble with that, because the regular datetime doesn't know how to compare itself to a neg_inf object. > I believe the ideal design is the following type hierarchy: > > datetime_base > datetime_neg_inf > datetime_pos_inf > datetime > > But in the interests of pragmatism, I think this one can be made to > work: > > datetime > datetime_neg_inf > datetime_pos_inf > yup -- should work. and you could add class methods to the datetime object, so you could do: datetime.neg_inf() and datetime.pos_inf() > The > advantage to this is that we don't have to switch everything over to > using a new derived type--just use the normal datetime for the majority > of cases where we don't need inifinity dates. > And it could be a third part package, too. -- particularly good for backward compatibility. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150128/7e6ed455/attachment-0001.html> From ncoghlan at gmail.com Thu Jan 29 12:47:17 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Jan 2015 21:47:17 +1000 Subject: [Python-ideas] Adding a subprocess.CompletedProcess class In-Reply-To: <20150128171318.22a829eb@anarchist.wooz.org> References: <CAOvn4qgTKVTbcY+BN_Z7-v8bMRprn=2FSqcnMuw9j+NT-cegPA@mail.gmail.com> <CAGE7PNKQ0QpZ-3hab4xJppknhCOOCJXuRWadsKGeGFz-BbJg_A@mail.gmail.com> <CADiSq7dAYUudHbDzRuVxmL6SsT4i8_50jvGNPf0G2Jet7MS8Dg@mail.gmail.com> <CAOvn4qhH2Et49NgMMED3DQ-9GuX1k8cELGEXyegh64ezMHoqfw@mail.gmail.com> <20150128171318.22a829eb@anarchist.wooz.org> Message-ID: <CADiSq7etZdNe9mgjpfXO8b=r5981-vUxP3t=Kn+62n-SwW+b1w@mail.gmail.com> On 29 January 2015 at 08:13, Barry Warsaw <barry at python.org> wrote: > On Jan 28, 2015, at 09:43 AM, Thomas Kluyver wrote: > >>Ugh, yes. I would follow that for consistency, though I'd love to come up >>with a way to clean that up too. The fact that 'decode the output' is >>spelled 'universal_newlines' is one of my least favourite things about >>subprocess. > > Not me; it's one of those sekrits that ensures Python 3 porters job > security. ;) I still suspect we should be offering a simpler way to decouple the creation of the pipes from the subprocess call, but I have no idea what that API should look like, and as ugly and unintuitive as it is, the existing universal_newlines trick does address all of my *actual* use cases :P Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Jan 29 12:49:58 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Jan 2015 21:49:58 +1000 Subject: [Python-ideas] "atexit" equivalent for functions in contextlib In-Reply-To: <87bnlirvar.fsf@vostro.rath.org> References: <87bnlirvar.fsf@vostro.rath.org> Message-ID: <CADiSq7eysQvKV=zirToUJt2oVWYuz21xiE+duRt+ZDv_9a8bJA@mail.gmail.com> On 29 January 2015 at 15:49, Nikolaus Rath <Nikolaus at rath.org> wrote: > It's not a lot of code, but my feeling is that not anyone who might be > interested in an "on_return" functionality would be able to come up with > this right away. What does it gain you other than saving a level of indentation over the form with an explicit context manager? The version with the magic hidden wrapper will also be slower than the explicit form, and will break parameter prompting in most IDEs. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve at pearwood.info Thu Jan 29 13:46:02 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 29 Jan 2015 23:46:02 +1100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAP7+vJLhzz0d5JO6J13zYMo4tFct3Gw_1f+9W1jkeaJUcfeoNg@mail.gmail.com> References: <CALGmxEKZjkz_iRqte=n3y0kQ6FyC5-30-chm_LAcWFY-C1R0cQ@mail.gmail.com> <CAPTjJmqFeKvri7jeFU_NAofrih1Z6hieuuNLs+dsdLFcKmxB=Q@mail.gmail.com> <CALGmxEJ8Y6t0DQMe82gBWxxybEnjvN0dhbNhyQSWuYMU8cQpxw@mail.gmail.com> <CAP7+vJ+LdEob5RO2J+D6QjfgB-wT68mDXk3iHDXtqbgXGVjP=w@mail.gmail.com> <CALGmxE+gi=KLQU8ggBafMw20dB6bfMrXHEmdxX8QNFRwKAC_MA@mail.gmail.com> <CAP7+vJ+8UrrHiGs7=Mogw2i6s0JS94UvfSSD-ZMy2xf2GrdU1Q@mail.gmail.com> <CALGmxELxomDNmdV7-B9=LSQrfceVvc9b1n_p0yrR08ffN65r4Q@mail.gmail.com> <CAP7+vJJbsFWQPTxDfSBEdREVyciJiKDWziF82y1HT4-DFwnUVQ@mail.gmail.com> <CALGmxEKqu_L=FBCDaAXF1v1-K1-v6GC=E84bqTSsFaBWKR6rCw@mail.gmail.com> <CAP7+vJLhzz0d5JO6J13zYMo4tFct3Gw_1f+9W1jkeaJUcfeoNg@mail.gmail.com> Message-ID: <20150129124601.GD2498@ando.pearwood.info> On Tue, Jan 27, 2015 at 02:35:50PM -0800, Guido van Rossum wrote: > I'm still confused. We're talking about a function that compares two simple > values, right? Where does the sequence of values come from? Numpy users do > everything by the array, but they already have an isclose(). Or perhaps > you're talking about assertApproxEqual in test_statistics.py? That has so > many ways to specify the relative and absolute tolerance that I give up > understanding it. Those ways evolved from my actual use of the function. What I found in practice is that within each TestCase, most of the tests used the same values for error tolerences. At the very least, I was aiming for all the tests to use the same error tolerance, but in practice I never quite achieved that. From time to time I would have a particularly difficult calculation that just wouldn't meet the desired tolerance, and I had to accept a lower tolerance. Most individual test methods within a single TestCase used the same error tolerances, which I set in the setUp method as self.rel for relative error (or self.tol for absolute), and wrote: self.assertApproxEqual(x, y, rel=self.rel) over and over again, occasionally overriding that value: self.assertApproxEqual(x, y, rel=1e-6) Since most of the time, the method was taken the tolerances from self, I reasoned that I should just make the default "take the tolerances from self" and be done with it. So that's how the method evolved. Most of those tests were for functions that didn't end up in the std lib, so I'm not surprised that this was not so clear. If I remember correctly, at the time I was also reading a lot about OOP design principles (as well as stats text books) and I think the use of instance attributes for the default error tolerances was probably influenced by that. -- Steve From jmcs at jsantos.eu Thu Jan 29 13:54:57 2015 From: jmcs at jsantos.eu (=?UTF-8?B?Sm/Do28gU2FudG9z?=) Date: Thu, 29 Jan 2015 13:54:57 +0100 Subject: [Python-ideas] Adding a subprocess.CompletedProcess class In-Reply-To: <CADiSq7etZdNe9mgjpfXO8b=r5981-vUxP3t=Kn+62n-SwW+b1w@mail.gmail.com> References: <CAOvn4qgTKVTbcY+BN_Z7-v8bMRprn=2FSqcnMuw9j+NT-cegPA@mail.gmail.com> <CAGE7PNKQ0QpZ-3hab4xJppknhCOOCJXuRWadsKGeGFz-BbJg_A@mail.gmail.com> <CADiSq7dAYUudHbDzRuVxmL6SsT4i8_50jvGNPf0G2Jet7MS8Dg@mail.gmail.com> <CAOvn4qhH2Et49NgMMED3DQ-9GuX1k8cELGEXyegh64ezMHoqfw@mail.gmail.com> <20150128171318.22a829eb@anarchist.wooz.org> <CADiSq7etZdNe9mgjpfXO8b=r5981-vUxP3t=Kn+62n-SwW+b1w@mail.gmail.com> Message-ID: <CAH_XWH3T6JyAOUj_JM+EiEKZpOGw8tSsf_7EUjeWcMZqwQjksg@mail.gmail.com> Maybe subprocess could take a page out of pathlib and use | (__or__) for piping. On 29 January 2015 at 12:47, Nick Coghlan <ncoghlan at gmail.com> wrote: > On 29 January 2015 at 08:13, Barry Warsaw <barry at python.org> wrote: > > On Jan 28, 2015, at 09:43 AM, Thomas Kluyver wrote: > > > >>Ugh, yes. I would follow that for consistency, though I'd love to come up > >>with a way to clean that up too. The fact that 'decode the output' is > >>spelled 'universal_newlines' is one of my least favourite things about > >>subprocess. > > > > Not me; it's one of those sekrits that ensures Python 3 porters job > > security. ;) > > I still suspect we should be offering a simpler way to decouple the > creation of the pipes from the subprocess call, but I have no idea > what that API should look like, and as ugly and unintuitive as it is, > the existing universal_newlines trick does address all of my *actual* > use cases :P > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150129/70057fc4/attachment.html> From steve at pearwood.info Thu Jan 29 14:34:00 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 30 Jan 2015 00:34:00 +1100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEJC+AycT+B2T0zDRfwqSFvNSty9b3FRO+y-U2jhLJd5Hw@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CALGmxEJi_eAOZG_aktG+7jjdhEyH79rzYg2Sq15UoMgsQNHiOA@mail.gmail.com> <CALGmxEJPWv4-AEd0Ws-N4efzA9nZPrCFamdfhvZrpHyPOF6SUg@mail.gmail.com> <CACac1F8WzH1UcRSN89KJRhsFguEz1nwPPoaKCm=-2UCbLYiwvw@mail.gmail.com> <CALGmxEJC+AycT+B2T0zDRfwqSFvNSty9b3FRO+y-U2jhLJd5Hw@mail.gmail.com> Message-ID: <20150129133359.GE2498@ando.pearwood.info> On Mon, Jan 26, 2015 at 05:10:44PM -0800, Chris Barker wrote: > That being said .. if if we want to say the primary use case is testing > that's fine with me -- just please not buried in unittest.TestCase > somewhere. Apart from doctest, which I think is completely inappropriate, where else would you put this for testing? For testing, I think it needs to be a TestCase.assert* method. Otherwise you have people writing self.assertTrue(close_enough(a, b)) I know this because that's exactly how my assertApproxEqual test assertion started. I'd write tests like that, they would fail, and I'd have no idea why. Fast forward past a few code iterations, and I had an assertion method which give me some useful diagnostics when it failed, e.g.: AssertionError: 20.666666666666668 != 20.66666667 values differ by more than tol=0 and rel=1e-12 -> absolute error = 3.33333360913457e-09 -> relative error = 1.612903358998517e-10 -- Steve From p.f.moore at gmail.com Thu Jan 29 15:13:55 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 29 Jan 2015 14:13:55 +0000 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150129133359.GE2498@ando.pearwood.info> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CALGmxEJi_eAOZG_aktG+7jjdhEyH79rzYg2Sq15UoMgsQNHiOA@mail.gmail.com> <CALGmxEJPWv4-AEd0Ws-N4efzA9nZPrCFamdfhvZrpHyPOF6SUg@mail.gmail.com> <CACac1F8WzH1UcRSN89KJRhsFguEz1nwPPoaKCm=-2UCbLYiwvw@mail.gmail.com> <CALGmxEJC+AycT+B2T0zDRfwqSFvNSty9b3FRO+y-U2jhLJd5Hw@mail.gmail.com> <20150129133359.GE2498@ando.pearwood.info> Message-ID: <CACac1F_aHTfOg27Tf2zrP8SJ9MH9ALz0x56BxXi=aWCUeVqdYA@mail.gmail.com> On 29 January 2015 at 13:34, Steven D'Aprano <steve at pearwood.info> wrote: > For testing, I think it needs to be a TestCase.assert* method. Otherwise > you have people writing > > self.assertTrue(close_enough(a, b)) The only downside is that this doesn't consider other test frameworks like nose and py.test. For those frameworks, you need a standalone function. *If* the intention is to expose a function that people can use for testing, I think it needs to be a standalone close_enough function plus a unittest assert method that uses that function. But I'm fairly sure by now that Guido's right and this should just be a recipe. I can write my own close_enough function now, with the information from this thread. The only bit I'd struggle over is what tolerance to use, and it doesn't look like anyone has a particularly good universal answer for that :-) Paul From steve at pearwood.info Thu Jan 29 15:51:56 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 30 Jan 2015 01:51:56 +1100 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAP7h-xZZQc5pGqGdQjZojCd-FZke9+eY9WsdtUZukQuqsgg=vg@mail.gmail.com> References: <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEKZjkz_iRqte=n3y0kQ6FyC5-30-chm_LAcWFY-C1R0cQ@mail.gmail.com> <CAPTjJmqFeKvri7jeFU_NAofrih1Z6hieuuNLs+dsdLFcKmxB=Q@mail.gmail.com> <CALGmxEJ8Y6t0DQMe82gBWxxybEnjvN0dhbNhyQSWuYMU8cQpxw@mail.gmail.com> <CAP7+vJ+LdEob5RO2J+D6QjfgB-wT68mDXk3iHDXtqbgXGVjP=w@mail.gmail.com> <CAP7h-xZZQc5pGqGdQjZojCd-FZke9+eY9WsdtUZukQuqsgg=vg@mail.gmail.com> Message-ID: <20150129145155.GF2498@ando.pearwood.info> On Tue, Jan 27, 2015 at 12:57:03PM -0500, Alexander Belopolsky wrote: > Just as a point of reference, APL and its derivatives use tolerant > comparison in the default equal (=) operator. The definition that they use > for finite x and y is simply > > x = y <=> abs(x-y) <= tol * max(abs(x), abs(y)) That's essentially the same as Bruce Dawson recommends: https://randomascii.wordpress.com/2012/02/25/comparing-floating-point-numbers-2012-edition/ and also the approx_equal function in test_statistics, except that also uses an absolute error if given. Guido has asked what the use case for giving both relative and absolute error. I'll quote Dawson from the link above: [quote] The most generic answer to this quandary is to use a mixture of absolute and relative epsilons. If the two numbers being compared are extremely close ? whatever that means ? then treat them as equal, regardless of their relative values. This technique is necessary any time you are expecting an answer of zero due to subtraction. The value of the absolute epsilon should be based on the magnitude of the numbers being subtracted ? it should be something like maxInput * FLT_EPSILON. Unfortunately this means that it is dependent on the algorithm and the inputs. Charming. The ULPs based technique also breaks down near zero for the technical reasons discussed just below the definition of AlmostEqualUlps. Doing a floating-point absolute epsilon check first, and then treating all other different-signed numbers as being non-equal is the simpler and safer thing to do. Here is some possible code for doing this, both for relative epsilon and for ULPs based comparison, with an absolute epsilon ?safety net? to handle the near-zero case: [...] [end quote] -- Steven From chris.barker at noaa.gov Thu Jan 29 16:49:17 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 29 Jan 2015 07:49:17 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150129133359.GE2498@ando.pearwood.info> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CALGmxEJi_eAOZG_aktG+7jjdhEyH79rzYg2Sq15UoMgsQNHiOA@mail.gmail.com> <CALGmxEJPWv4-AEd0Ws-N4efzA9nZPrCFamdfhvZrpHyPOF6SUg@mail.gmail.com> <CACac1F8WzH1UcRSN89KJRhsFguEz1nwPPoaKCm=-2UCbLYiwvw@mail.gmail.com> <CALGmxEJC+AycT+B2T0zDRfwqSFvNSty9b3FRO+y-U2jhLJd5Hw@mail.gmail.com> <20150129133359.GE2498@ando.pearwood.info> Message-ID: <-3165626588370158743@unknownmsgid> >> On Mon, Jan 26, 2015 at 05:10:44PM -0800, Chris Barker wrote: >> >> That being said .. if if we want to say the primary use case is testing >> that's fine with me -- just please not buried in unittest.TestCase >> somewhere. > > Apart from doctest, which I think is completely inappropriate, where > else would you put this for testing? math would be a good place. An awful lot of testing is done outside of the unittest module, and indeed, outside of any formal testing at all: command line, quick if __ name__ ... Stanzas, etc. > For testing, I think it needs to be a TestCase.assert* method. I do think that's a good idea, but it should call the math function. Pretty much like you have in the statistics tests. Though I'm not the one to write that part of the PEP -- I already don't like the unittest API ;-) -Chris > Otherwise > you have people writing > > self.assertTrue(close_enough(a, b)) > > I know this because that's exactly how my assertApproxEqual test > assertion started. I'd write tests like that, they would fail, and I'd > have no idea why. Fast forward past a few code iterations, and I had an > assertion method which give me some useful diagnostics when it failed, > e.g.: > > AssertionError: 20.666666666666668 != 20.66666667 > values differ by more than tol=0 and rel=1e-12 > -> absolute error = 3.33333360913457e-09 > -> relative error = 1.612903358998517e-10 > > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From chris.barker at noaa.gov Thu Jan 29 17:01:52 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 29 Jan 2015 08:01:52 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <20150129145155.GF2498@ando.pearwood.info> References: <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEKZjkz_iRqte=n3y0kQ6FyC5-30-chm_LAcWFY-C1R0cQ@mail.gmail.com> <CAPTjJmqFeKvri7jeFU_NAofrih1Z6hieuuNLs+dsdLFcKmxB=Q@mail.gmail.com> <CALGmxEJ8Y6t0DQMe82gBWxxybEnjvN0dhbNhyQSWuYMU8cQpxw@mail.gmail.com> <CAP7+vJ+LdEob5RO2J+D6QjfgB-wT68mDXk3iHDXtqbgXGVjP=w@mail.gmail.com> <CAP7h-xZZQc5pGqGdQjZojCd-FZke9+eY9WsdtUZukQuqsgg=vg@mail.gmail.com> <20150129145155.GF2498@ando.pearwood.info> Message-ID: <1398098206023325221@unknownmsgid> > On Jan 29, 2015, at 6:52 AM, Steven D'Aprano > That's essentially the same as Bruce Dawson recommends: If anyone's interested in playing with this, I updated my sample implementation with a flag that let's you select which method it will use. Four! options. Note that I not suggesting such a flag be there in an std lib function, but it makes it easy to experiment. https://github.com/PythonCHB/close_pep/blob/master/is_close_to.py Note that I found that every single one of my test cases passed with all four methods, except one designed specifically to demonstrate the asymmetry of the asymmetric test. Which was my point about it not mattering much in practice. I still need to add a bunch more to the PEP by way of explanation, then I hope to be able to gauge where people are at. -Chris From chris.barker at noaa.gov Thu Jan 29 17:12:45 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 29 Jan 2015 08:12:45 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CACac1F_aHTfOg27Tf2zrP8SJ9MH9ALz0x56BxXi=aWCUeVqdYA@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CAPJVwBkiU8zbYfTwmoZL80mAZ5guJGcgziZdTOQL8JuA_3s4WA@mail.gmail.com> <CADiSq7eokB6tFTAt8J710NPKuvzdgu7Cg4ewg0fgZ4czDxNtpg@mail.gmail.com> <20150125130326.GG20517@ando.pearwood.info> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CALGmxEJi_eAOZG_aktG+7jjdhEyH79rzYg2Sq15UoMgsQNHiOA@mail.gmail.com> <CALGmxEJPWv4-AEd0Ws-N4efzA9nZPrCFamdfhvZrpHyPOF6SUg@mail.gmail.com> <CACac1F8WzH1UcRSN89KJRhsFguEz1nwPPoaKCm=-2UCbLYiwvw@mail.gmail.com> <CALGmxEJC+AycT+B2T0zDRfwqSFvNSty9b3FRO+y-U2jhLJd5Hw@mail.gmail.com> <20150129133359.GE2498@ando.pearwood.info> <CACac1F_aHTfOg27Tf2zrP8SJ9MH9ALz0x56BxXi=aWCUeVqdYA@mail.gmail.com> Message-ID: <4488140366426996585@unknownmsgid> > But I'm fairly sure by now that Guido's right and this should just be > a recipe. I think the long thread here is more a reflection that it's an interesting problem, and frankly, simple enough that almost anyone has something to say about it. But the fact that there is no one way to do it that is perfect for all cases doesn't mean there isn't one way to do it that is useful in the vast majority of cases. I'm quite sure that most of the options on the table fit that bill, in fact. And I'm optimistic that we can converge on an option that most everyone on this thread can live with. > I can write my own close_enough function now, with the > information from this thread. Sure -- but the idea is that you shouldn't have to. Particularly for unittest.TestCase or quick command line checks. -Chris > The only bit I'd struggle over is what > tolerance to use, and it doesn't look like anyone has a particularly > good universal answer for that :-) > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From random832 at fastmail.us Thu Jan 29 19:10:37 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Thu, 29 Jan 2015 13:10:37 -0500 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <CALGmxELXfjftwP_qQfRyHfcCE=e-Jae8T68i6LYc9ZCHAxuXSA@mail.gmail.com> References: <54C5F9A7.3010209@thomas-guettler.de> <3114096308045821944@unknownmsgid> <54C85788.3090705@gmail.com> <CALGmxE+o7sakjvys1KZPrpHRKpfe7ijTvOBaQ6WkieC3O-z5=Q@mail.gmail.com> <54C9A13E.6050003@gmail.com> <CALGmxELXfjftwP_qQfRyHfcCE=e-Jae8T68i6LYc9ZCHAxuXSA@mail.gmail.com> Message-ID: <1422555037.834514.220602189.7E8A2FC9@webmail.messagingengine.com> On Thu, Jan 29, 2015, at 02:04, Chris Barker wrote: > I had trouble with that, because the regular datetime doesn't know how to > compare itself to a neg_inf object. Works fine on python 3.3. Python 2 raises exception from __lt__, python 3 returns NotImplemented. From ethan at stoneleaf.us Thu Jan 29 19:45:02 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 29 Jan 2015 10:45:02 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CAP7+vJLhzz0d5JO6J13zYMo4tFct3Gw_1f+9W1jkeaJUcfeoNg@mail.gmail.com> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEKZjkz_iRqte=n3y0kQ6FyC5-30-chm_LAcWFY-C1R0cQ@mail.gmail.com> <CAPTjJmqFeKvri7jeFU_NAofrih1Z6hieuuNLs+dsdLFcKmxB=Q@mail.gmail.com> <CALGmxEJ8Y6t0DQMe82gBWxxybEnjvN0dhbNhyQSWuYMU8cQpxw@mail.gmail.com> <CAP7+vJ+LdEob5RO2J+D6QjfgB-wT68mDXk3iHDXtqbgXGVjP=w@mail.gmail.com> <CALGmxE+gi=KLQU8ggBafMw20dB6bfMrXHEmdxX8QNFRwKAC_MA@mail.gmail.com> <CAP7+vJ+8UrrHiGs7=Mogw2i6s0JS94UvfSSD-ZMy2xf2GrdU1Q@mail.gmail.com> <CALGmxELxomDNmdV7-B9=LSQrfceVvc9b1n_p0yrR08ffN65r4Q@mail.gmail.com> <CAP7+vJJbsFWQPTxDfSBEdREVyciJiKDWziF82y1HT4-DFwnUVQ@mail.gmail.com> <CALGmxEKqu_L=FBCDaAXF1v1-K1-v6GC=E84bqTSsFaBWKR6rCw@mail.gmail.com> <CAP7+vJLhzz0d5JO6J13zYMo4tFct3Gw_1f+9W1jkeaJUcfeoNg@mail.gmail.com> Message-ID: <54CA7FAE.1010304@stoneleaf.us> On 01/27/2015 02:35 PM, Guido van Rossum wrote: > > I'm still confused. We're talking about a function that compares two simple values, right? Where does the sequence of > values come from? I think from uses involving list comps and such. But I could be wrong. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150129/325340db/attachment-0001.sig> From chris.barker at noaa.gov Thu Jan 29 19:53:17 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 29 Jan 2015 10:53:17 -0800 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <54CA7FAE.1010304@stoneleaf.us> References: <CALGmxELKJ7Lyahk__97+ANf_A+jf_cw3d1X0vt1j9NbyEgNs0Q@mail.gmail.com> <CALGmxEJ+GxY8vjJ-ciS00WwhcFNd1xd1A+6XTiZm9aSozjSUow@mail.gmail.com> <20150126063932.GI20517@ando.pearwood.info> <CACac1F9LRN+hRv066DxTsjhwwFYCx0tvngsO46hBRGkc-=_zug@mail.gmail.com> <20150127030800.GQ20517@ando.pearwood.info> <CALGmxEKZjkz_iRqte=n3y0kQ6FyC5-30-chm_LAcWFY-C1R0cQ@mail.gmail.com> <CAPTjJmqFeKvri7jeFU_NAofrih1Z6hieuuNLs+dsdLFcKmxB=Q@mail.gmail.com> <CALGmxEJ8Y6t0DQMe82gBWxxybEnjvN0dhbNhyQSWuYMU8cQpxw@mail.gmail.com> <CAP7+vJ+LdEob5RO2J+D6QjfgB-wT68mDXk3iHDXtqbgXGVjP=w@mail.gmail.com> <CALGmxE+gi=KLQU8ggBafMw20dB6bfMrXHEmdxX8QNFRwKAC_MA@mail.gmail.com> <CAP7+vJ+8UrrHiGs7=Mogw2i6s0JS94UvfSSD-ZMy2xf2GrdU1Q@mail.gmail.com> <CALGmxELxomDNmdV7-B9=LSQrfceVvc9b1n_p0yrR08ffN65r4Q@mail.gmail.com> <CAP7+vJJbsFWQPTxDfSBEdREVyciJiKDWziF82y1HT4-DFwnUVQ@mail.gmail.com> <CALGmxEKqu_L=FBCDaAXF1v1-K1-v6GC=E84bqTSsFaBWKR6rCw@mail.gmail.com> <CAP7+vJLhzz0d5JO6J13zYMo4tFct3Gw_1f+9W1jkeaJUcfeoNg@mail.gmail.com> <54CA7FAE.1010304@stoneleaf.us> Message-ID: <CALGmxEK1zE61=XY6NkM_OnvuT1mLQ_C5Donxv8GBtYUbTixvsQ@mail.gmail.com> On Thu, Jan 29, 2015 at 10:45 AM, Ethan Furman <ethan at stoneleaf.us> wrote: > On 01/27/2015 02:35 PM, Guido van Rossum wrote: > > > > I'm still confused. We're talking about a function that compares two > simple values, right? Where does the sequence of > > values come from? > > I think from uses involving list comps and such. But I could be wrong. > Yes, that's it -- as well as the sequence-compatible asserts in unittest. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150129/3ea7d297/attachment.html> From chris.barker at noaa.gov Thu Jan 29 20:09:04 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 29 Jan 2015 11:09:04 -0800 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <1422555037.834514.220602189.7E8A2FC9@webmail.messagingengine.com> References: <54C5F9A7.3010209@thomas-guettler.de> <3114096308045821944@unknownmsgid> <54C85788.3090705@gmail.com> <CALGmxE+o7sakjvys1KZPrpHRKpfe7ijTvOBaQ6WkieC3O-z5=Q@mail.gmail.com> <54C9A13E.6050003@gmail.com> <CALGmxELXfjftwP_qQfRyHfcCE=e-Jae8T68i6LYc9ZCHAxuXSA@mail.gmail.com> <1422555037.834514.220602189.7E8A2FC9@webmail.messagingengine.com> Message-ID: <CALGmxEJZttA6vFo0FJZJHz=xhdm2DWA6f+hbEtj5tFLg5+OLxw@mail.gmail.com> On Thu, Jan 29, 2015 at 10:10 AM, <random832 at fastmail.us> wrote: > Works fine on python 3.3. Good. > Python 2 raises exception from __lt__, yeah, that's what I got -- we're on 2 for now. You'd think there wold be a way to do it -- but I don't see a "reflected" versions of the comparison operators. But for anything new, it'd be in 3.5+ anyway, so sounds like we're good. Though it would be nice to have it as a recipe for older versions. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150129/2403e503/attachment.html> From ethan at stoneleaf.us Thu Jan 29 20:14:22 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 29 Jan 2015 11:14:22 -0800 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <CALGmxEJZttA6vFo0FJZJHz=xhdm2DWA6f+hbEtj5tFLg5+OLxw@mail.gmail.com> References: <54C5F9A7.3010209@thomas-guettler.de> <3114096308045821944@unknownmsgid> <54C85788.3090705@gmail.com> <CALGmxE+o7sakjvys1KZPrpHRKpfe7ijTvOBaQ6WkieC3O-z5=Q@mail.gmail.com> <54C9A13E.6050003@gmail.com> <CALGmxELXfjftwP_qQfRyHfcCE=e-Jae8T68i6LYc9ZCHAxuXSA@mail.gmail.com> <1422555037.834514.220602189.7E8A2FC9@webmail.messagingengine.com> <CALGmxEJZttA6vFo0FJZJHz=xhdm2DWA6f+hbEtj5tFLg5+OLxw@mail.gmail.com> Message-ID: <54CA868E.5050207@stoneleaf.us> On 01/29/2015 11:09 AM, Chris Barker wrote: > On Thu, Jan 29, 2015 at 10:10 AM, random832 wrote: >> >> Python 2 raises exception from __lt__, > > yeah, that's what I got -- we're on 2 for now. You'd think there wold > be a way to do it -- but I don't see a "reflected" versions of the > comparison operators. Raising an exception was a bug. The comparison operators are their own reflection. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150129/8bc9437e/attachment.sig> From chris.barker at noaa.gov Thu Jan 29 20:18:40 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 29 Jan 2015 11:18:40 -0800 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <54CA868E.5050207@stoneleaf.us> References: <54C5F9A7.3010209@thomas-guettler.de> <3114096308045821944@unknownmsgid> <54C85788.3090705@gmail.com> <CALGmxE+o7sakjvys1KZPrpHRKpfe7ijTvOBaQ6WkieC3O-z5=Q@mail.gmail.com> <54C9A13E.6050003@gmail.com> <CALGmxELXfjftwP_qQfRyHfcCE=e-Jae8T68i6LYc9ZCHAxuXSA@mail.gmail.com> <1422555037.834514.220602189.7E8A2FC9@webmail.messagingengine.com> <CALGmxEJZttA6vFo0FJZJHz=xhdm2DWA6f+hbEtj5tFLg5+OLxw@mail.gmail.com> <54CA868E.5050207@stoneleaf.us> Message-ID: <CALGmxEKLOo0RvWTYfa4vY2ros=S1DGfxAeExiigo5E9KzitU=A@mail.gmail.com> On Thu, Jan 29, 2015 at 11:14 AM, Ethan Furman <ethan at stoneleaf.us> wrote: > >> Python 2 raises exception from __lt__, > > > > yeah, that's what I got -- we're on 2 for now. You'd think there wold > > be a way to do it -- but I don't see a "reflected" versions of the > > comparison operators. > > Raising an exception was a bug. The comparison operators are their own > reflection. > So does that mean it could be fixed in 2.* ? (or already has been?) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150129/b228536f/attachment.html> From yawar.amin at gmail.com Thu Jan 29 20:19:48 2015 From: yawar.amin at gmail.com (Yawar Amin) Date: Thu, 29 Jan 2015 14:19:48 -0500 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <CALGmxELXfjftwP_qQfRyHfcCE=e-Jae8T68i6LYc9ZCHAxuXSA@mail.gmail.com> References: <54C5F9A7.3010209@thomas-guettler.de> <3114096308045821944@unknownmsgid> <54C85788.3090705@gmail.com> <CALGmxE+o7sakjvys1KZPrpHRKpfe7ijTvOBaQ6WkieC3O-z5=Q@mail.gmail.com> <54C9A13E.6050003@gmail.com> <CALGmxELXfjftwP_qQfRyHfcCE=e-Jae8T68i6LYc9ZCHAxuXSA@mail.gmail.com> Message-ID: <D50BDCAB-EDEE-4D5F-BB16-F40E27FA92AA@gmail.com> Hi Chris, sorry for the top-posting; replying from my phone. My test implementation works for me on Python 2.7, see http://repl.it/9Wz Notice especially the last few assertions, where normal datetime objects are on the LHS. Regards, Yawar On 2015-01-29, at 2:04, Chris Barker <chris.barker at noaa.gov> wrote: > On Wed, Jan 28, 2015 at 6:55 PM, Yawar Amin <yawar.amin at gmail.com> wrote: >> Cool, the comparison operator logic looks very similar to mine: >> >> https://groups.google.com/d/msg/python-ideas/G3jeWoa6h14/ELpDLFu28QcJ > > Did you get it to work both ways? i.e. > > datetime.neg_inf < datetime.now() > > and > > datetime.now() > datetime.neg_inf > > I had trouble with that, because the regular datetime doesn't know how to compare itself to a neg_inf object. > >> I believe the ideal design is the following type hierarchy: >> >> datetime_base >> datetime_neg_inf >> datetime_pos_inf >> datetime >> >> But in the interests of pragmatism, I think this one can be made to >> work: >> >> datetime >> datetime_neg_inf >> datetime_pos_inf > > yup -- should work. > > and you could add class methods to the datetime object, so you could do: > > datetime.neg_inf() > and > datetime.pos_inf() > >> The >> advantage to this is that we don't have to switch everything over to >> using a new derived type--just use the normal datetime for the majority >> of cases where we don't need inifinity dates. > > And it could be a third part package, too. -- particularly good for backward compatibility. > > -Chris > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150129/0c9636fa/attachment-0001.html> From random832 at fastmail.us Thu Jan 29 20:23:18 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Thu, 29 Jan 2015 14:23:18 -0500 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <54CA868E.5050207@stoneleaf.us> References: <54C5F9A7.3010209@thomas-guettler.de> <3114096308045821944@unknownmsgid> <54C85788.3090705@gmail.com> <CALGmxE+o7sakjvys1KZPrpHRKpfe7ijTvOBaQ6WkieC3O-z5=Q@mail.gmail.com> <54C9A13E.6050003@gmail.com> <CALGmxELXfjftwP_qQfRyHfcCE=e-Jae8T68i6LYc9ZCHAxuXSA@mail.gmail.com> <1422555037.834514.220602189.7E8A2FC9@webmail.messagingengine.com> <CALGmxEJZttA6vFo0FJZJHz=xhdm2DWA6f+hbEtj5tFLg5+OLxw@mail.gmail.com> <54CA868E.5050207@stoneleaf.us> Message-ID: <1422559398.862584.220636205.13521E97@webmail.messagingengine.com> On Thu, Jan 29, 2015, at 14:14, Ethan Furman wrote: > On 01/29/2015 11:09 AM, Chris Barker wrote: > > On Thu, Jan 29, 2015 at 10:10 AM, random832 wrote: > >> > >> Python 2 raises exception from __lt__, > > > > yeah, that's what I got -- we're on 2 for now. You'd think there wold > > be a way to do it -- but I don't see a "reflected" versions of the > > comparison operators. > > Raising an exception was a bug. The comparison operators are their own > reflection. Or, more precisely, they are *each other's* reflection (__gt__ being the reflection of __lt__). This works fine in 2.7 for types that properly return NotImplemented. This is just a bug in Python 2's datetime module. From skip.montanaro at gmail.com Thu Jan 29 20:22:48 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Thu, 29 Jan 2015 13:22:48 -0600 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <CALGmxEKLOo0RvWTYfa4vY2ros=S1DGfxAeExiigo5E9KzitU=A@mail.gmail.com> References: <54C5F9A7.3010209@thomas-guettler.de> <3114096308045821944@unknownmsgid> <54C85788.3090705@gmail.com> <CALGmxE+o7sakjvys1KZPrpHRKpfe7ijTvOBaQ6WkieC3O-z5=Q@mail.gmail.com> <54C9A13E.6050003@gmail.com> <CALGmxELXfjftwP_qQfRyHfcCE=e-Jae8T68i6LYc9ZCHAxuXSA@mail.gmail.com> <1422555037.834514.220602189.7E8A2FC9@webmail.messagingengine.com> <CALGmxEJZttA6vFo0FJZJHz=xhdm2DWA6f+hbEtj5tFLg5+OLxw@mail.gmail.com> <54CA868E.5050207@stoneleaf.us> <CALGmxEKLOo0RvWTYfa4vY2ros=S1DGfxAeExiigo5E9KzitU=A@mail.gmail.com> Message-ID: <CANc-5UwZ2ofzdKtqDQa32bxeo1F2h0bkKsQ_c-Ht=-v_Hmp+OA@mail.gmail.com> On Thu, Jan 29, 2015 at 1:18 PM, Chris Barker <chris.barker at noaa.gov> wrote: > So does that mean it could be fixed in 2.* ? (or already has been?) That it hasn't been fixed by now suggests that the core developers felt it would break more than it would repair. Might be worth scanning the relevant tracker issue(s) or searching the python-dev list archives. Skip -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150129/94363717/attachment.html> From random832 at fastmail.us Thu Jan 29 20:29:23 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Thu, 29 Jan 2015 14:29:23 -0500 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <D50BDCAB-EDEE-4D5F-BB16-F40E27FA92AA@gmail.com> References: <54C5F9A7.3010209@thomas-guettler.de> <3114096308045821944@unknownmsgid> <54C85788.3090705@gmail.com> <CALGmxE+o7sakjvys1KZPrpHRKpfe7ijTvOBaQ6WkieC3O-z5=Q@mail.gmail.com> <54C9A13E.6050003@gmail.com> <CALGmxELXfjftwP_qQfRyHfcCE=e-Jae8T68i6LYc9ZCHAxuXSA@mail.gmail.com> <D50BDCAB-EDEE-4D5F-BB16-F40E27FA92AA@gmail.com> Message-ID: <1422559763.864637.220641317.739F5AB1@webmail.messagingengine.com> On Thu, Jan 29, 2015, at 14:19, Yawar Amin wrote: > Hi Chris, sorry for the top-posting; replying from my phone. > > My test implementation works for me on Python 2.7, see http://repl.it/9Wz > > Notice especially the last few assertions, where normal datetime objects > are on the LHS. How'd you do that? Does python always know to use the derived class's operators when working with a base class and a derived class? From ethan at stoneleaf.us Thu Jan 29 20:31:53 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 29 Jan 2015 11:31:53 -0800 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <1422559398.862584.220636205.13521E97@webmail.messagingengine.com> References: <54C5F9A7.3010209@thomas-guettler.de> <3114096308045821944@unknownmsgid> <54C85788.3090705@gmail.com> <CALGmxE+o7sakjvys1KZPrpHRKpfe7ijTvOBaQ6WkieC3O-z5=Q@mail.gmail.com> <54C9A13E.6050003@gmail.com> <CALGmxELXfjftwP_qQfRyHfcCE=e-Jae8T68i6LYc9ZCHAxuXSA@mail.gmail.com> <1422555037.834514.220602189.7E8A2FC9@webmail.messagingengine.com> <CALGmxEJZttA6vFo0FJZJHz=xhdm2DWA6f+hbEtj5tFLg5+OLxw@mail.gmail.com> <54CA868E.5050207@stoneleaf.us> <1422559398.862584.220636205.13521E97@webmail.messagingengine.com> Message-ID: <54CA8AA9.6070600@stoneleaf.us> On 01/29/2015 11:23 AM, random832 at fastmail.us wrote: > On Thu, Jan 29, 2015, at 14:14, Ethan Furman wrote: >> On 01/29/2015 11:09 AM, Chris Barker wrote: >>> On Thu, Jan 29, 2015 at 10:10 AM, random832 wrote: >>>> >>>> Python 2 raises exception from __lt__, >>> >>> yeah, that's what I got -- we're on 2 for now. You'd think there wold >>> be a way to do it -- but I don't see a "reflected" versions of the >>> comparison operators. >> >> Raising an exception was a bug. The comparison operators are their own >> reflection. > > Or, more precisely, they are *each other's* reflection (__gt__ being the > reflection of __lt__). Yes, much better phrasing, thank you. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150129/d6e02e10/attachment.sig> From chris.barker at noaa.gov Thu Jan 29 20:35:01 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 29 Jan 2015 11:35:01 -0800 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <D50BDCAB-EDEE-4D5F-BB16-F40E27FA92AA@gmail.com> References: <54C5F9A7.3010209@thomas-guettler.de> <3114096308045821944@unknownmsgid> <54C85788.3090705@gmail.com> <CALGmxE+o7sakjvys1KZPrpHRKpfe7ijTvOBaQ6WkieC3O-z5=Q@mail.gmail.com> <54C9A13E.6050003@gmail.com> <CALGmxELXfjftwP_qQfRyHfcCE=e-Jae8T68i6LYc9ZCHAxuXSA@mail.gmail.com> <D50BDCAB-EDEE-4D5F-BB16-F40E27FA92AA@gmail.com> Message-ID: <CALGmxEJo-ToHFkvEHiLDPtH=NsJtfKmD4GNOGNi91X5GCDYELA@mail.gmail.com> On Thu, Jan 29, 2015 at 11:19 AM, Yawar Amin <yawar.amin at gmail.com> wrote: > My test implementation works for me on Python 2.7, see http://repl.it/9Wz > cool site! And yes, it does appear to work. However the subclassing approach is really weird, as the infinite datetimes have a year, month, day, etc -- that strikes me as ripe for error. That's why I didn't subclass, but just had them e independent. Another option would be to override just about everything... -Chris > Notice especially the last few assertions, where normal datetime objects > are on the LHS. > > Regards, > > Yawar > > On 2015-01-29, at 2:04, Chris Barker <chris.barker at noaa.gov> wrote: > > On Wed, Jan 28, 2015 at 6:55 PM, Yawar Amin <yawar.amin at gmail.com> wrote: > >> Cool, the comparison operator logic looks very similar to mine: >> >> https://groups.google.com/d/msg/python-ideas/G3jeWoa6h14/ELpDLFu28QcJ >> >> > Did you get it to work both ways? i.e. > > datetime.neg_inf < datetime.now() > > and > > datetime.now() > datetime.neg_inf > > I had trouble with that, because the regular datetime doesn't know how to > compare itself to a neg_inf object. > > >> I believe the ideal design is the following type hierarchy: >> >> datetime_base >> datetime_neg_inf >> datetime_pos_inf >> datetime >> >> But in the interests of pragmatism, I think this one can be made to >> work: >> >> datetime >> datetime_neg_inf >> datetime_pos_inf >> > > yup -- should work. > > and you could add class methods to the datetime object, so you could do: > > datetime.neg_inf() > and > datetime.pos_inf() > > >> The >> > advantage to this is that we don't have to switch everything over to >> using a new derived type--just use the normal datetime for the majority >> of cases where we don't need inifinity dates. >> > > And it could be a third part package, too. -- particularly good for > backward compatibility. > > -Chris > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150129/7e1796e7/attachment-0001.html> From ethan at stoneleaf.us Thu Jan 29 20:36:17 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 29 Jan 2015 11:36:17 -0800 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <1422559763.864637.220641317.739F5AB1@webmail.messagingengine.com> References: <54C5F9A7.3010209@thomas-guettler.de> <3114096308045821944@unknownmsgid> <54C85788.3090705@gmail.com> <CALGmxE+o7sakjvys1KZPrpHRKpfe7ijTvOBaQ6WkieC3O-z5=Q@mail.gmail.com> <54C9A13E.6050003@gmail.com> <CALGmxELXfjftwP_qQfRyHfcCE=e-Jae8T68i6LYc9ZCHAxuXSA@mail.gmail.com> <D50BDCAB-EDEE-4D5F-BB16-F40E27FA92AA@gmail.com> <1422559763.864637.220641317.739F5AB1@webmail.messagingengine.com> Message-ID: <54CA8BB1.2090701@stoneleaf.us> On 01/29/2015 11:29 AM, random832 at fastmail.us wrote: > On Thu, Jan 29, 2015, at 14:19, Yawar Amin wrote: >> Hi Chris, sorry for the top-posting; replying from my phone. >> >> My test implementation works for me on Python 2.7, see http://repl.it/9Wz >> >> Notice especially the last few assertions, where normal datetime objects >> are on the LHS. > > How'd you do that? Does python always know to use the derived class's > operators when working with a base class and a derived class? I can't find the reference, but yes. So the broken datetime behavior would only be noticed by a non-datetime derived class. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150129/73cabb97/attachment.sig> From chris.barker at noaa.gov Thu Jan 29 20:39:41 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 29 Jan 2015 11:39:41 -0800 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <54CA8BB1.2090701@stoneleaf.us> References: <54C5F9A7.3010209@thomas-guettler.de> <3114096308045821944@unknownmsgid> <54C85788.3090705@gmail.com> <CALGmxE+o7sakjvys1KZPrpHRKpfe7ijTvOBaQ6WkieC3O-z5=Q@mail.gmail.com> <54C9A13E.6050003@gmail.com> <CALGmxELXfjftwP_qQfRyHfcCE=e-Jae8T68i6LYc9ZCHAxuXSA@mail.gmail.com> <D50BDCAB-EDEE-4D5F-BB16-F40E27FA92AA@gmail.com> <1422559763.864637.220641317.739F5AB1@webmail.messagingengine.com> <54CA8BB1.2090701@stoneleaf.us> Message-ID: <CALGmxELWfurnLtuEFGRhanQRR9sn9t6wVo=1zpMx06ZhMDzpvA@mail.gmail.com> On Thu, Jan 29, 2015 at 11:36 AM, Ethan Furman <ethan at stoneleaf.us> wrote: > > How'd you do that? Does python always know to use the derived class's > > operators when working with a base class and a derived class? > > I can't find the reference, but yes. > > So the broken datetime behavior would only be noticed by a non-datetime > derived class. > got it - that's why my version didn't work. So this is a good reason to derive from datetime. but I'd make sure to override everything that doesn't make sense. -Chris > -- > ~Ethan~ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150129/a9b91277/attachment.html> From yawar.amin at gmail.com Thu Jan 29 20:55:00 2015 From: yawar.amin at gmail.com (Yawar Amin) Date: Thu, 29 Jan 2015 14:55:00 -0500 Subject: [Python-ideas] datetime: Support infinity In-Reply-To: <CALGmxEJo-ToHFkvEHiLDPtH=NsJtfKmD4GNOGNi91X5GCDYELA@mail.gmail.com> References: <54C5F9A7.3010209@thomas-guettler.de> <3114096308045821944@unknownmsgid> <54C85788.3090705@gmail.com> <CALGmxE+o7sakjvys1KZPrpHRKpfe7ijTvOBaQ6WkieC3O-z5=Q@mail.gmail.com> <54C9A13E.6050003@gmail.com> <CALGmxELXfjftwP_qQfRyHfcCE=e-Jae8T68i6LYc9ZCHAxuXSA@mail.gmail.com> <D50BDCAB-EDEE-4D5F-BB16-F40E27FA92AA@gmail.com> <CALGmxEJo-ToHFkvEHiLDPtH=NsJtfKmD4GNOGNi91X5GCDYELA@mail.gmail.com> Message-ID: <4CB153EF-656A-40BF-96E4-E4BE6704854D@gmail.com> Hi Chris, agreed, it's not great to have years, months and dates inside the infinity datetimes. That's why ideally datetime, datetime_neg_inf and datetime_pos_inf should have been sibling types deriving from datetime_base. I'll try to explore making this change in the stdlib ad seeing if that breaks th vs really badly. On 2015-01-29, at 14:35, Chris Barker <chris.barker at noaa.gov> wrote: > On Thu, Jan 29, 2015 at 11:19 AM, Yawar Amin <yawar.amin at gmail.com> wrote: >> My test implementation works for me on Python 2.7, see http://repl.it/9Wz > > cool site! > > And yes, it does appear to work. > > However the subclassing approach is really weird, as the infinite datetimes have a year, month, day, etc -- that strikes me as ripe for error. That's why I didn't subclass, but just had them e independent. > > Another option would be to override just about everything... > > -Chris > > >> Notice especially the last few assertions, where normal datetime objects are on the LHS. >> >> Regards, >> >> Yawar >> >> On 2015-01-29, at 2:04, Chris Barker <chris.barker at noaa.gov> wrote: >> >>> On Wed, Jan 28, 2015 at 6:55 PM, Yawar Amin <yawar.amin at gmail.com> wrote: >>>> Cool, the comparison operator logic looks very similar to mine: >>>> >>>> https://groups.google.com/d/msg/python-ideas/G3jeWoa6h14/ELpDLFu28QcJ >>> >>> Did you get it to work both ways? i.e. >>> >>> datetime.neg_inf < datetime.now() >>> >>> and >>> >>> datetime.now() > datetime.neg_inf >>> >>> I had trouble with that, because the regular datetime doesn't know how to compare itself to a neg_inf object. >>> >>>> I believe the ideal design is the following type hierarchy: >>>> >>>> datetime_base >>>> datetime_neg_inf >>>> datetime_pos_inf >>>> datetime >>>> >>>> But in the interests of pragmatism, I think this one can be made to >>>> work: >>>> >>>> datetime >>>> datetime_neg_inf >>>> datetime_pos_inf >>> >>> yup -- should work. >>> >>> and you could add class methods to the datetime object, so you could do: >>> >>> datetime.neg_inf() >>> and >>> datetime.pos_inf() >>> >>>> The >>>> advantage to this is that we don't have to switch everything over to >>>> using a new derived type--just use the normal datetime for the majority >>>> of cases where we don't need inifinity dates. >>> >>> And it could be a third part package, too. -- particularly good for backward compatibility. >>> >>> -Chris >>> >>> >>> -- >>> >>> Christopher Barker, Ph.D. >>> Oceanographer >>> >>> Emergency Response Division >>> NOAA/NOS/OR&R (206) 526-6959 voice >>> 7600 Sand Point Way NE (206) 526-6329 fax >>> Seattle, WA 98115 (206) 526-6317 main reception >>> >>> Chris.Barker at noaa.gov > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150129/bc476321/attachment.html> From thomas at kluyver.me.uk Thu Jan 29 22:09:14 2015 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Thu, 29 Jan 2015 13:09:14 -0800 Subject: [Python-ideas] Adding a subprocess.CompletedProcess class In-Reply-To: <CADiSq7etZdNe9mgjpfXO8b=r5981-vUxP3t=Kn+62n-SwW+b1w@mail.gmail.com> References: <CAOvn4qgTKVTbcY+BN_Z7-v8bMRprn=2FSqcnMuw9j+NT-cegPA@mail.gmail.com> <CAGE7PNKQ0QpZ-3hab4xJppknhCOOCJXuRWadsKGeGFz-BbJg_A@mail.gmail.com> <CADiSq7dAYUudHbDzRuVxmL6SsT4i8_50jvGNPf0G2Jet7MS8Dg@mail.gmail.com> <CAOvn4qhH2Et49NgMMED3DQ-9GuX1k8cELGEXyegh64ezMHoqfw@mail.gmail.com> <20150128171318.22a829eb@anarchist.wooz.org> <CADiSq7etZdNe9mgjpfXO8b=r5981-vUxP3t=Kn+62n-SwW+b1w@mail.gmail.com> Message-ID: <CAOvn4qiH_NScSjg0W+FXSO21jufu3upHLFcaTeqxiR85-wakxQ@mail.gmail.com> On 29 January 2015 at 03:47, Nick Coghlan <ncoghlan at gmail.com> wrote: > I still suspect we should be offering a simpler way to decouple the > creation of the pipes from the subprocess call, but I have no idea > what that API should look like, > Presumably that would need some kind of object representing a not-yet-started process. Technically, that could be Popen, but for backwards compatibility the Popen constructor needs to start the process, and p = Popen(..., start=False) seems inelegant. Let's imagine it's a new class called Command. Then you could start coming up with interfaces like: c = subprocess.Command(...) c.stdout = fileobj c.stderr = fileobj2 # Or c.capture('combined') # sets stdout=PIPE and stderr=STDOUT # Maybe get into operator overloading? pipeline = c | c2 # Could this work? Would probably require threading c.stdout = BytesIO() c.stderr_decoded = StringIO() # When you've finished setting things up c.run() # returns a Popen instance N.B. This is 'thinking aloud', not any kind of proposal - I'm not convinced by any of that API myself. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150129/c7c60d97/attachment-0001.html> From Nikolaus at rath.org Fri Jan 30 03:28:34 2015 From: Nikolaus at rath.org (Nikolaus Rath) Date: Thu, 29 Jan 2015 18:28:34 -0800 Subject: [Python-ideas] "atexit" equivalent for functions in contextlib In-Reply-To: <CADiSq7eysQvKV=zirToUJt2oVWYuz21xiE+duRt+ZDv_9a8bJA@mail.gmail.com> (Nick Coghlan's message of "Thu, 29 Jan 2015 21:49:58 +1000") References: <87bnlirvar.fsf@vostro.rath.org> <CADiSq7eysQvKV=zirToUJt2oVWYuz21xiE+duRt+ZDv_9a8bJA@mail.gmail.com> Message-ID: <877fw5roh9.fsf@vostro.rath.org> Nick Coghlan <ncoghlan-Re5JQEeQqe8AvxtiuMwx3w at public.gmane.org> writes: > On 29 January 2015 at 15:49, Nikolaus Rath <Nikolaus-BTH8mxji4b0 at public.gmane.org> wrote: >> It's not a lot of code, but my feeling is that not anyone who might be >> interested in an "on_return" functionality would be able to come up with >> this right away. > > What does it gain you other than saving a level of indentation over > the form with an explicit context manager? Nothing. But if that's not a good enough reason, why do we have contextlib.ContextDecorator? Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From schesis at gmail.com Sat Jan 31 18:17:32 2015 From: schesis at gmail.com (Zero Piraeus) Date: Sat, 31 Jan 2015 14:17:32 -0300 Subject: [Python-ideas] PEP 485: A Function for testing approximate equality In-Reply-To: <CALGmxEJ--kPkr-qSLJXuOvk4vbm4qCkc92Mz0r0SkPbLv+vUzQ@mail.gmail.com> References: <CALGmxE+uLtEQ-RwCVfe3Lqv__JF76SXvRP3po8Sp96FoKXfPhQ@mail.gmail.com> <CALGmxEKan04wZyvUGJpe72+7yQkTUriKMyVVWfG-3t_g+SKVHA@mail.gmail.com> <20150128003804.GA25474@piedra> <CALGmxEJ--kPkr-qSLJXuOvk4vbm4qCkc92Mz0r0SkPbLv+vUzQ@mail.gmail.com> Message-ID: <20150131171732.GA5058@piedra> : On Tue, Jan 27, 2015 at 10:31:01PM -0800, Chris Barker wrote: > On Tue, Jan 27, 2015 at 4:38 PM, Zero Piraeus <schesis at gmail.com> wrote: > > Note that my example was an any_close() function, not an all_close() > > one. I doubt anyone's seriously going to suggest adding *three* new > > functions, so anyone who wants something like that will by necessity > > end up rolling their own. > > OK -- that is pretty compelling --- I have been looking for use-cases > for where having a symmetrical test would be a clear advantage, and > this is one. > > But this is where I'm unclear -- is this "any_close" function > something you think is a real use case -- i.e. you have needed such a > thing, or think there is a strong chance that you will. Or just > something that _could_ be done. I have needed an any_close() function in the past, but used absolute tolerances, so it's not particularly relevant to the discussion (except maybe that if is_close() existed, I might have used it just because it was there without really thinking things through). > Granted, I think your point is that if it _could_ be done, there is a > good chance that _someone_ will do it -- and with that in mind we want > as few surprising behaviors in the standard lib as possible. Yep. -[]z. -- Zero Piraeus: pons asinorum http://etiol.net/pubkey.asc